Table of Contents
The official documentation for installation of ONAP with OOM / Kubernetes is located in readthedocs here (TBD). The supported versions of Kubernetes is as follows:
Software Requirements
...
1.8.5-
| Jira | ||||||
|---|---|---|---|---|---|---|
|
...
20180321: use 1.12 for now
Hardware Requirements
...
63G at startup
70G after 48h
Recommend 128G class VM
75G at startup
100G after a week
Recommend
120-160G HD
4 min
60 peak at startup
recommended 16-32 vCores
...
see list on PortProfile
Recommend
0.0.0.0/0 (all open)
Quickstart
Beijing ONAP on Azure ARM template to full OOM install - see ONAP on Kubernetes on Microsoft Azure#Quickstart
Amsterdam ONAP on Intel Openstack to full OOM install - see ONAP on Kubernetes on OpenStack
Installation Flow
1) To start - pick your infrastructure - ONAP supports deployment to all the major public clouds such as:
2) On your VM in the above infrastructure - create a Kubernetes environment (a one time activity) follow one of these guides:
- Setting up Kubernetes with Rancher (recommended)
- Setting up Kubernetes with Kubeadm
- Setting up Kubernetes with Cloudify
3) Install ONAP - use the cd.sh script provided as a patch in review
Known Issues
Beijing/master branch
These are current failures for individual ONAP compoents that have workarounds or can be ignored - as a service to everyone who asks - last update 20180314
AAF has a failing container -
| Jira | ||||||
|---|---|---|---|---|---|---|
|
SDC has an internal dependency issue and will need a coordinated restart -
| Jira | ||||||
|---|---|---|---|---|---|---|
|
SDNC has a timing issue and will need a restart -
| Jira | ||||||
|---|---|---|---|---|---|---|
|
(vCPE/vVOLTE use case only) VFC has a failing container -
| Jira | ||||||
|---|---|---|---|---|---|---|
|
(vCPE/vVOLTE use case only) VNFSDK has a failing container -
| Jira | ||||||
|---|---|---|---|---|---|---|
|
Details
| Table of Contents |
|---|
Installation templates for Kubernetes and deploying ONAP via OOM for the master and amsterdam branches are in the following locations
| Jira | ||||||
|---|---|---|---|---|---|---|
|
Kubernetes devops / workarounds / troubleshooting content restored from version 378 - into the page ONAP on Kubernetes on Rancher
Example End to End Kubernetes based ONAP install and deployment
Scenario: installing Rancher on clean Ubuntu 16.04 64g VM (single collocated server/host) and the master branch of onap via OOM deployment (2 scripts)
1 hour video of automated installation on an AWS EC2 spot instance
| View file | ||||
|---|---|---|---|---|
|
Note: amsterdam will require a different onap-parameters.yaml
| Code Block |
|---|
Either
git clone https://gerrit.onap.org/r/oom
cd oom
git pull https://gerrit.onap.org/r/oom refs/changes/19/32019/6
cd install/rancher
./oom_rancher_install.sh -b master -s cd.onap.info -e onap
or just run the WIP oom_entrypoint.sh script (And swap out your own onap-parameters.yaml)
use the latest on https://jira.onap.org/browse/OOM-710
wget https://jira.onap.org/secure/attachment/LATEST_ID/oom_entrypoint.sh
chmod 777 oom_entrypoint.sh
./oom_entrypoint.sh -b master -s your_ip -e onap
# refer to this script that runs oom_rancher_setup.sh and cd.sh in sequence |
clone continuous deployment script - until it is merged
| Code Block |
|---|
wget https://jira.onap.org/secure/attachment/ID/cd.sh
chmod 777 cd.sh
wget https://jira.onap.org/secure/attachment/ID/aaiapisimpledemoopenecomporg.cer
wget https://jira.onap.org/secure/attachment/ID/onap-parameters.yaml
wget https://jira.onap.org/secure/attachment/ID/aai-cloud-region-put.json
./cd.sh -b master -e onap
# wait about 25-60 min depending on the speed of your network pulling docker images |
Results
| Code Block |
|---|
obrienbiometrics:azure michaelobrien$ ssh ubuntu@cd.onap.info
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-1041-aws x86_64)
ubuntu@ip-172-31-95-122:~$ sudo su -
root@ip-172-31-95-122:~# wget https://jira.onap.org/secure/attachment/ID/oom_rancher_setup.sh
--2018-02-27 15:51:09-- https://jira.onap.org/secure/attachment/ID/oom_rancher_setup.sh
Saving to: ‘oom_rancher_setup.sh’
oom_rancher_setup.sh 100%[=========================================================================================>] 6.68K --.-KB/s in 0s
root@ip-172-31-95-122:~# chmod 777 oom_rancher_setup.sh
root@ip-172-31-95-122:~# wget https://jira.onap.org/secure/attachment/ID/cd.sh
root@ip-172-31-95-122:~# wget https://jira.onap.org/secure/attachment/ID/aaiapisimpledemoopenecomporg.cer
root@ip-172-31-95-122:~# wget https://jira.onap.org/secure/attachment/ID/onap-parameters.yaml
root@ip-172-31-95-122:~# wget https://jira.onap.org/secure/attachment/ID/aai-cloud-region-put.json
root@ip-172-31-95-122:~# ./oom_rancher_setup.sh -b master -s cd.onap.info -e onap
Installing on cd.onap.info for master: Rancher: 1.6.14 Kubectl: 1.8.6 Helm: 2.6.1 Docker: 17.03
+ sh -c docker version
Client: Version: 17.03.2-ce
Status: Downloaded newer image for rancher/server:v1.6.14
Saving to: ‘helm-v2.6.1-linux-amd64.tar.gz’
--2018-02-27 15:52:46-- https://releases.rancher.com/cli/v0.6.7/rancher-linux-amd64-v0.6.7.tar.gz
Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/universe amd64 jq amd64 1.5+dfsg-1 [144 kB]
Setting up jq (1.5+dfsg-1) ...
publicValue: 896EC0CC76D9A171BFF0 secretValue: wH1x1hf5oJe9FSPLex1rgTkk5oabtRMrJGb9MJ6x
ID NAME ORCHESTRATION STATE CREATED
1a5 Default cattle active 2018-02-27T15:53:04Z
Creating kubernetes environment named onap
env id: 1a7
you should see an additional kubernetes environment
ID NAME ORCHESTRATION STATE CREATED
1a5 Default cattle active 2018-02-27T15:53:04Z
1a7 onap kubernetes active 2018-02-27T15:53:48Z
Registering host for image: rancher/agent:v1.2.9 url: http://127.0.0.1:8880/v2-beta/scripts/9E6292FC1B3C65923DD2:1514678400000:Ri3dGEnzBp03z2DVtDrZ4nCQ0XE registrationToken: 9E6292FC1B3C65923DD2:1514678400000:Ri3dGEnzBp03z2DVtDrZ4nCQ0XE
Status: Downloaded newer image for rancher/agent:v1.2.9
INFO: Running Agent Registration Process, CATTLE_URL=http://cd.onap.info:8880/v1
INFO: Attempting to connect to: http://cd.onap.info:8880/v1
INFO: http://cd.onap.info:8880/v1 is accessible
INFO: Launched Rancher Agent: 2a4a71b60657b964701019de36c6045f842d7844057433362f715e201751cf6f
waiting 7 min for host registration to finish
KUBECTL_TOKEN base64 encoded: QmFzaWMgT0RrMlJVTXdRME0zTmtRNVFURTNNVUpHUmpBNmQwZ3hlREZvWmpWdlNtVTVSbE5RVEdWNE1YSm5WR3RyTlc5aFluUlNUWEpLUjJJNVRVbzJlQT09
Verify all pods up on the kubernetes system - will return localhost:8080 until a host is added
kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system heapster-76b8cd7b5-bjckj 1/1 Running 0 6m
kube-system kube-dns-5d7b4487c9-zz9cp 3/3 Running 0 6m
kube-system kubernetes-dashboard-f9577fffd-tcxld 1/1 Running 0 6m
kube-system monitoring-grafana-997796fcf-n4sw5 1/1 Running 0 6m
kube-system monitoring-influxdb-56fdcd96b-h2jr8 1/1 Running 0 6m
kube-system tiller-deploy-cc96d4f6b-6nlgh 1/1 Running 0 6m
172-31-95-122:~# nohup ./cd.sh -b master &
[1] 21891
root@ip-172-31-95-122:~# nohup: ignoring input and appending output to 'nohup.out'
root@ip-172-31-95-122:~# tail -f nohup.out
Tue Feb 27 16:43:00 UTC 2018
verify onap-config is 0/1 not 1/1 - as in completed - an error pod - means you are missing onap-parameters.yaml or values are not set in it.
waiting for config pod to complete
pre pull docker images - 15+ min
docker login -u docker -p docker nexus3.onap.org:10001
Login Succeeded
docker pull oomk8s/readiness-check:1.1.0
Status: Downloaded newer image for nexus3.onap.org:10001/openecomp/appc-image:v1.2.0
waiting for last pull
start onap pods
********** Creating deployments for policy **********
Creating namespace **********
namespace "onap-policy" created
Creating service account **********
clusterrolebinding "onap-policy-admin-binding" created
Creating registry secret **********
secret "onap-docker-registry-key" created
Creating deployments and services **********
NAME: onap-policy
LAST DEPLOYED: Tue Feb 27 17:20:31 2018
NAMESPACE: onap
STATUS: DEPLOYED
==> v1/ConfigMap
NAME DATA AGE
policy-dep-pe-configmap 13 1s
policy-dep-drools-settings-configmap 1 1s
policy-sdk-log-configmap 1 1s
policy-filebeat-configmap 1 1s
policy-pdp-log-configmap 1 1s
policy-drools-log-configmap 1 1s
policy-pap-log-configmap 1 1s
....
8 pending > 0 at the 88th 15 sec interval
onap-aaf aaf-6c64db8fdd-glxxv 0/1 Running 0 25m
onap-sdnc dmaap-listener-579cdfd5cb-wd9gd 0/1 Init:0/1 2 26m
onap-sdnc sdnc-0 0/2 Init:0/1 2 26m
onap-sdnc sdnc-dbhost-0 0/2 Pending 0 26m
onap-sdnc sdnc-dgbuilder-794d686f78-tgq5t 0/1 Init:0/1 2 26m
onap-sdnc sdnc-portal-69b79b6646-g7zz2 0/1 Init:0/1 2 26m
onap-sdnc ueb-listener-6ff56545b6-754v7 0/1 Init:0/1 2 26m
onap-vfc vfc-ztevnfmdriver-585d8db4f7-q24z6 0/1 ImagePullBackOff 0 25m
Basic Policy Health Check | PASS |
root@ip-172-31-95-122:~# kubectl get pods --all-namespaces | grep onap-policy
onap-policy brmsgw-869dc8888c-htkkz 1/1 Running 0 1h
onap-policy drools-5f568559b-vgtd9 2/2 Running 0 1h
onap-policy mariadb-7c66956bf-tkszv 1/1 Running 0 1h
onap-policy nexus-69f7497db5-rnmt9 1/1 Running 0 1h
onap-policy pap-7fbff7db64-fpqgj 2/2 Running 0 1h
onap-policy pdp-55c454589-nzfcf 2/2 Running 0 1h
root@ip-172-31-95-122:~# kubectl get services --all-namespaces | grep onap-policy
onap-policy brmsgw NodePort 10.43.5.29 <none> 9989:30216/TCP 1h
onap-policy drools NodePort 10.43.96.150 <none> 6969:30217/TCP,9696:30221/TCP 1h
onap-policy mariadb ClusterIP None <none> 3306/TCP 1h
onap-policy nexus ClusterIP None <none> 8081/TCP 1h
onap-policy pap NodePort 10.43.125.137 <none> 8443:30219/TCP,9091:30218/TCP 1h
onap-policy pdp NodePort 10.43.130.155 <none> 8081:30220/TCP 1h
root@ip-172-31-95-122:~# curl http://cd.onap.info:30220/pdp/swagger-ui.html#/policy-engine-services
<div id="swagger-ui-container" class="swagger-ui-wrap"></div>
root@ip-172-31-95-122:~#
# will post a curl REST get/post in a minute |
...
Manual Quickstart Installation - Please use script instead
(Manual instructions) - try to use the script at Setting up Kubernetes with Rancher via https://gerrit.onap.org/r/#/c/32019
ONAP Minimum R1 Installation Helm Apps
oom/kubernetes/oneclick/setenv.bash maybe updated to the following reduce app set.
...
HELM_APPS=('mso' 'message-router' 'sdnc' 'vid' 'robot' 'portal' 'policy' 'appc' 'aai' 'sdc' 'log')#HELM_APPS=('consul' 'msb' 'mso' 'message-router' 'sdnc' 'vid' 'robot' 'portal' 'policy' 'appc' 'aai' 'sdc' 'dcaegen2' 'log' 'cli' 'multicloud' 'clamp' 'vnfsdk' 'uui' 'aaf' 'vfc' 'kube2msb' 'esr')
is located in Read the Docs:
- OOM User Guide — onap master documentation
- OOM Quick Start Guide — onap master documentation)
- OOM Cloud Setup Guide — onap master documentation
1) install rancher, clone oom, run config-init pod, run one or all onap components
...
*****************
Note: uninstall docker if already installed - as Kubernetes 1.8 under rancher supports 17.03.2 as of 20180124
|
*****************
Install Rancher
ONAP deployment in kubernetes is modelled in the oom project as a 1:1 set of service:pod sets (1 pod per docker container). The fastest way to get ONAP Kubernetes up is via Rancher on any bare metal or VM that supports a clean Ubuntu 16.04 install and more than 60G ram.
TODO: REMOVE from table cell - wrapping is not working
(on each host) add to your /etc/hosts to point your ip to your hostname (add your hostname to the end). Add entries for all other hosts in your cluster.
For example on openlab - you will need to add the name of your host before you install docker - to avoid the error below
sudo: unable to resolve host onap-oom
|
Open Ports
On most hosts like openstack or EC2 you can open all the ports or they are open by default - on some environments like Rackspace VM's you need to open them
|
Fix virtual memory allocation (to allow onap-log:elasticsearch to come up under Rancher 1.6.11+)
|
clone oom (scp your onap_rsa private key first - or clone anon - Ideally you get a full gerrit account and join the community)
see ssh/http/http access links below
https://gerrit.onap.org/r/#/admin/projects/oom
|
(on each host (server and client(s) which may be the same machine)) Install only the 17.03.2 version of Docker (the only version that works with Kubernetes in Rancher 1.6.13+)
Install Docker
|
Pre pull docker images the first time you install onap. Currently the pre-pull will take 16-180 min depending on your network. Pre pulling the images will allow the entire ONAP to start in 3-8 min instead of up to 3 hours.
OOM-328 - Preload docker images script before createAll.bash will allow 7 min startup REOPENED
Use script above in oom/kubernetes/config once it is merged.
https://git.onap.org/oom/tree/kubernetes/config/prepull_docker.sh
|
To monitor when prepull is finished see section: Prepulldockerimages. It is advised to wait until pre pull has finished before continuing.
(on the master only) Install rancher (Optional: use 8880 instead of 8080 if there is a conflict) - note there may be issues with the dns pod in Rancher after a reboot or when running clustered hosts - a clean system will be OK -
|
In Rancher UI - dont use (http://127.0.0.1:8880) - use the real IP address - so the client configs are populated correctly with callbacks
You must deactivate the default CATTLE environment - by adding a KUBERNETES environment - and Deactivating the older default CATTLE one - your added hosts will attach to the default
- Default → Manage Environments
- Select "Add Environment" button
- Give the Environment a name and description, then select Kubernetes as the Environment Template
- Hit the "Create" button. This will create the environment and bring you back to the Manage Environments view
- At the far right column of the Default Environment row, left-click the menu ( looks like 3 stacked dots ), and select Deactivate. This will make your new Kubernetes environment the new default.
Register your host
Register your host(s) - run following on each host (including the master if you are collocating the master/host on a single machine/vm)
For each host, In Rancher > Infrastructure > Hosts. Select "Add Host"
The first time you add a host - you will be presented with a screen containing the routable IP - hit save only on a routable IP.
Enter IP of host: (if you launched racher with 127.0.0.1/localhost - otherwise keep it empty - it will autopopulate the registration with the real IP)
Copy command to register host with Rancher,
Execute command on each host, for example:
|
wait for kubernetes menu to populate with the CLI
Install Kubectl
The following will install kubectl (for Kubernetes 1.8.6 ) https://github.com/kubernetes/kubernetes/issues/57528 on a linux host. Once configured, this client tool will provide management of a Kubernetes cluster.
|
Paste kubectl config from Rancher (you will see the CLI menu in Rancher / Kubernetes after the k8s pods are up on your host)
Click on "Generate Config" to get your content to add into .kube/config
Verify that Kubernetes config is good
|
Install Helm
The following will install Helm - currently 2.8.0 on a linux host. Helm is used by OOM for package and configuration management.
https://lists.onap.org/pipermail/onap-discuss/2018-January/007674.html
Prerequisite: Install Kubectl (2.6.1 comes with Rancher 1.6.14 - for master)
|
...