Table of Contents
The official documentation for installation of ONAP with OOM / Kubernetes is located in readthedocs here (TBD). The supported versions of Kubernetes is as follows:
...
If you're looking for instructions on how to create a Kubernetes environment (a one time activity) follow one of these guides:
- Setting up Kubernetes with Rancher
- Setting up Kubernetes with Kubeadm
- Setting up Kubernetes with Cloudify
There are many public cloud systems that could be used as well such as:
The following content will be reviewed, moved or eliminated. Use with caution.
Quickstart Guide
Advanced Kubernetes Installations
Children Display |
---|
Table of Contents |
---|
ONAP Installation
Automated Installation
...
(Manual instructions)
ONAP Minimum R1 Installation Helm Apps
oom/kubernetes/oneclick/setenv.bash maybe updated to the following reduce app set.
...
HELM_APPS=('mso' 'message-router' 'sdnc' 'vid' 'robot' 'portal' 'policy' 'appc' 'aai' 'sdc' 'log') #HELM_APPS=('consul' 'msb' 'mso' 'message-router' 'sdnc' 'vid' 'robot' 'portal' 'policy' 'appc' 'aai' 'sdc' 'dcaegen2' 'log' 'cli' 'multicloud' 'clamp' 'vnfsdk' 'uui' 'aaf' 'vfc' 'kube2msb' 'esr')
1) install rancher, clone oom, run config-init pod, run one or all onap components
*****************
Note: uninstall docker if already installed - as Kubernetes 1.8 under rancher supports 17.03.2 as of 20180124
...
% sudo apt-get remove docker-engine
*****************
Install Rancher
ONAP deployment in kubernetes is modelled in the oom project as a 1:1 set of service:pod sets (1 pod per docker container). The fastest way to get ONAP Kubernetes up is via Rancher on any bare metal or VM that supports a clean Ubuntu 16.04 install and more than 55G ram.
TODO: REMOVE from table cell - wrapping is not working
(on each host) add to your /etc/hosts to point your ip to your hostname (add your hostname to the end). Add entries for all other hosts in your cluster.
For example on openlab - you will need to add the name of your host before you install docker - to avoid the error below
sudo: unable to resolve host onap-oom
...
language | bash |
---|
sudo vi /etc/hosts <your-ip> <your-hostname>
Open Ports
On most hosts like openstack or EC2 you can open all the ports or they are open by default - on some environments like Rackspace VM's you need to open them
...
sudo iptables -I INPUT 1 -p tcp --dport 8880 -j ACCEPT iptables-save > /etc/iptables.rules
Fix virtual memory allocation (to allow onap-log:elasticsearch to come up under Rancher 1.6.11)
...
sudo sysctl -w vm.max_map_count=262144
clone oom (scp your onap_rsa private key first - or clone anon - Ideally you get a full gerrit account and join the community)
see ssh/http/http access links below
https://gerrit.onap.org/r/#/admin/projects/oom
...
git clone http://gerrit.onap.org/r/oom
(on each host (server and client(s) which may be the same machine)) Install only the 17.03.2 version of Docker (the only version that works with Kubernetes in Rancher 1.6.13+)
Install Docker
...
# for root just run the following line and skip to next section curl https://releases.rancher.com/install-docker/17.03.sh | sh # when running as non-root (ubuntu) run the following and logout/log back in sudo usermod -aG docker ubuntu
Pre pull docker images the first time you install onap. Currently the pre-pull will take 16-180 min depending on your network. Pre pulling the images will allow the entire ONAP to start in 3-8 min instead of up to 3 hours.
Jira | ||||||
---|---|---|---|---|---|---|
|
Use script above in oom/kubernetes/config once it is merged.
https://git.onap.org/oom/tree/kubernetes/config/prepull_docker.sh
...
cp oom/kubernetes/config/prepull_docker.sh . chmod 777 prepull_docker.sh nohup ./prepull_docker.sh > prepull.log &
To monitor when prepull is finished see section: Prepulldockerimages. It is advised to wait until pre pull has finished before continuing.
(on the master only) Install rancher (Optional: use 8880 instead of 8080 if there is a conflict) - note there may be issues with the dns pod in Rancher after a reboot or when running clustered hosts - a clean system will be OK -
Jira server ONAP JIRA serverId 425b2b0a-557c-3c0c-b515-579789cceedb key OOM-236
Jira | ||||||
---|---|---|---|---|---|---|
|
is located in Read the Docs:
- OOM User Guide — onap master documentation
- OOM Quick Start Guide — onap master documentation)
- OOM Cloud Setup Guide — onap master documentation
...
docker run -d --restart=unless-stopped -p 8880:8080 --name rancher-server rancher/server:v1.6.14
In Rancher UI - dont use (http://127.0.0.1:8880) - use the real IP address - so the client configs are populated correctly with callbacks
You must deactivate the default CATTLE environment - by adding a KUBERNETES environment - and Deactivating the older default CATTLE one - your added hosts will attach to the default
Default → Manage Environments
Select "Add Environment" button
Give the Environment a name and description, then select Kubernetes as the Environment Template
Hit the "Create" button. This will create the environment and bring you back to the Manage Environments view
At the far right column of the Default Environment row, left-click the menu ( looks like 3 stacked dots ), and select Deactivate. This will make your new Kubernetes environment the new default.
Register your host
Register your host(s) - run following on each host (including the master if you are collocating the master/host on a single machine/vm)
For each host, In Rancher > Infrastructure > Hosts. Select "Add Host"
The first time you add a host - you will be presented with a screen containing the routable IP - hit save only on a routable IP.
Enter IP of host: (if you launched racher with 127.0.0.1/localhost - otherwise keep it empty - it will autopopulate the registration with the real IP)
Copy command to register host with Rancher,
Execute command on each host, for example:
...
sudo docker run --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.9 http://rackspace.onap.info:8880/v1/scripts/CDE31E5CDE3217328B2D:1514678400000:xLr2ySIppAaEZYWtTVa5V9ZGc
wait for kubernetes menu to populate with the CLI
Install Kubectl
The following will install kubectl (for Kubernetes 1.9.2 ) https://github.com/kubernetes/kubernetes/issues/57528 on a linux host. Once configured, this client tool will provide management of a Kubernetes cluster.
...
curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.9.2/bin/linux/amd64/kubectl chmod +x ./kubectl sudo mv ./kubectl /usr/local/bin/kubectl mkdir ~/.kube vi ~/.kube/config
Paste kubectl config from Rancher (you will see the CLI menu in Rancher / Kubernetes after the k8s pods are up on your host)
Click on "Generate Config" to get your content to add into .kube/config
Verify that Kubernetes config is good
...
root@obrien-kube11-1:~# kubectl cluster-info Kubernetes master is running at .... Heapster is running at.... KubeDNS is running at .... kubernetes-dashboard is running at ... monitoring-grafana is running at .... monitoring-influxdb is running at ... tiller-deploy is running at....
Install Helm
The following will install Helm - currently 2.8.0 on a linux host. Helm is used by OOM for package and configuration management.
https://lists.onap.org/pipermail/onap-discuss/2018-January/007674.html
Prerequisite: Install Kubectl
...
wget http://storage.googleapis.com/kubernetes-helm/helm-v2.8.0-linux-amd64.tar.gz tar -zxvf helm-v2.8.0-linux-amd64.tar.gz sudo mv linux-amd64/helm /usr/local/bin/helm # verify version helm version # Rancher 1.6.14 installs 2.6.2 - upgrade to 2.8.0 - you will need to upgrade helm on the server to the version to level of client helm init --upgrade
NOTE: If helm version takes long time and eventually errors out, this is most likely because incoming access to port 10250 (exposed by kubelet) is blocked by firewall. Make sure to configure firewall accordingly
Undercloud done - move to ONAP Installation
You can install OOM manually below or run the cd.sh below or attached to the top of this page - Install/RefreshOOM
https://github.com/obrienlabs/onap-root/blob/master/cd.sh
manually.....
Wait until all the hosts show green in rancher,
Then we are ready to configure and deploy onap environment in kubernetes. These scripts are found in the folders:
oom/kubernetes/oneclick
oom/kubernetes/config
First source oom/kubernetes/oneclick/setenv.bash. This will set your helm list of components to start/delete
...
cd ~/oom/kubernetes/oneclick/ source setenv.bash
Seconds we need configure the onap before deployment. This is a onetime operation that spawns temporality config pod. This mounts the volume /dockerdata/ contained in the pod config-init and also creates the directory “/dockerdata-nfs” on the kubernetes node. This mount is required for all other ONAP pods to function.
Note: the pod will stop after NFS creation - this is normal.
https://git.onap.org/oom/tree/kubernetes/config/onap-parameters-sample.yaml
...
cd ~/oom/kubernetes/config # edit or copy the config for MSO data vi onap-parameters.yaml # or cp onap-parameters-sample.yaml onap-parameters.yaml # run the config pod creation % ./createConfig.sh -n onap
**** Creating configuration for ONAP instance: onap
namespace "onap" created
pod "config-init" created
**** Done ****
Wait for the config-init pod is gone before trying to bring up a component or all of ONAP - around 60 sec (up to 10 min) - see https://wiki.onap.org/display/DW/ONAP+on+Kubernetes#ONAPonKubernetes-Waitingforconfig-initcontainertofinish-20sec
root@ip-172-31-93-122:~/oom_20170908/oom/kubernetes/config# kubectl get pods --all-namespaces -a
onap config 0/1 Completed 0 1m
Note: When using the -a option the config container will show up with the status, however when not used with the -a flag, it will not be present
Cluster Configuration (optional - do not use if your server/client are co-located)
3. Share the /dockerdata-nfs Folder between Kubernetes Nodes
...
Don't run all the pods unless you have at least 52G allocated - if you have a laptop/VM with 16G - then you can only run enough pods to fit in around 11G
...
% cd ../oneclick % vi createAll.bash % ./createAll.bash -n onap -a robot|appc|aai
(to bring up a single service at a time)
Use the default "onap" namespace if you want to run robot tests out of the box - as in "onap-robot"
Bring up core components
...
root@kos1001:~/oom1004/oom/kubernetes/oneclick# cat setenv.bash #HELM_APPS=('consul' 'msb' 'mso' 'message-router' 'sdnc' 'vid' 'robot' 'portal' 'policy' 'appc' 'aai' 'sdc' 'dcaegen2' 'log' 'cli' 'multicloud' 'clamp' 'vnfsdk' 'kube2msb' 'aaf' 'vfc') HELM_APPS=('consul' 'msb' 'mso' 'message-router' 'sdnc' 'vid' 'robot' 'portal' 'policy' 'appc' 'aai' 'sdc' 'log' 'kube2msb') # pods with the ELK filebeat container for capturing logs root@kos1001:~/oom1004/oom/kubernetes/oneclick# kubectl get pods --all-namespaces -a | grep 2/2 onap-aai aai-resources-338473047-8k6vr 2/2 Running 0 1h onap-aai aai-traversal-2033243133-6cr9v 2/2 Running 0 1h onap-aai model-loader-service-3356570452-25fjp 2/2 Running 0 1h onap-aai search-data-service-2366687049-jt0nb 2/2 Running 0 1h onap-aai sparky-be-3141964573-f2mhr 2/2 Running 0 1h onap-appc appc-1335254431-v1pcs 2/2 Running 0 1h onap-mso mso-3911927766-bmww7 2/2 Running 0 1h onap-policy drools-2302173499-t0zmt 2/2 Running 0 1h onap-policy pap-1954142582-vsrld 2/2 Running 0 1h onap-policy pdp-4137191120-qgqnj 2/2 Running 0 1h onap-portal portalapps-4168271938-4kp32 2/2 Running 0 1h onap-portal portaldb-2821262885-0t32z 2/2 Running 0 1h onap-sdc sdc-be-2986438255-sdqj6 2/2 Running 0 1h onap-sdc sdc-fe-1573125197-7j3gp 2/2 Running 0 1h onap-sdnc sdnc-3858151307-w9h7j 2/2 Running 0 1h onap-vid vid-server-1837290631-x4ttc 2/2 Running 0 1h
Only if you have >52G run the following (all namespaces)
...
% ./createAll.bash -n onap
ONAP is OK if everything is 1/1 or 2/2 in the following
...
% kubectl get pods --all-namespaces
Run the ONAP portal via instructions at RunningONAPusingthevnc-portal
Wait until the containers are all up
check AAI endpoints
root@ip-172-31-93-160:/dockerdata-nfs/onap/robot# kubectl -n onap-aai exec -it aai-service-3321436576-2snd6 bash
root@aai-service-3321436576-2snd6:/# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 15:50 ? 00:00:00 /usr/local/sbin/haproxy-systemd-
root 7 1 0 15:50 ? 00:00:00 /usr/local/sbin/haproxy-master
root@ip-172-31-93-160:/dockerdata-nfs/onap/robot# curl https://127.0.0.1:30233/aai/v11/service-design-and-creation/models
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
Run Health Check
...
Run Initial healthcheck directly on the host Initialize robot cd ~/oom/kubernetes/robot root@ip-172-31-83-168:~/oom/kubernetes/robot# ./demo-k8s.sh init_robot # password for test:test then health root@ip-172-31-83-168:~/oom/kubernetes/robot# ./ete-k8s.sh health
...
When running the non-root ubuntu and jenkins users - the NFS share needs its permissions upgraded in order for a delete to occur on VM reset
...
ubuntu@ip-172-31-85-6:~$ sudo chmod 777 -R /dockerdata-nfs/
Ports
List of Containers
Total pods is 84 and 16 filebeat containers
Docker container list - may not be fully up to date: https://git.onap.org/integration/tree/packaging/docker/docker-images.csv
OOM Pod Init Dependencies
...
The diagram above describes the init dependencies for the ONAP pods when first deploying OOM through Kubernetes.
Kubernetes DevOps
...
https://kubernetes.io/docs/user-guide/kubectl-cheatsheet/
Questions
https://lists.onap.org/pipermail/onap-discuss/2017-July/002084.html
Links
https://kubernetes.io/docs/user-guide/kubectl-cheatsheet/
Please help out our OPNFV friends
https://wiki.opnfv.org/pages/viewpage.action?pageId=12389095
Of interest
https://github.com/cncf/cross-cloud/
https://en.wikipedia.org/wiki/Fiddler_(software) - thanks Rahul
http://www.opencontrail.org/opencontrail-quick-start-guide/
https://github.com/prometheus/prometheus
http://zipkin.io/pages/quickstart
http://cloudify.co/2017/09/27/model-driven-onap-operations-manager-oom-boarding-tosca-cloudify/
Reference Reviews
https://gerrit.onap.org/r/#/c/6179/
https://gerrit.onap.org/r/#/c/9849/
...