ONAP on Kubernetes

Warning: Draft Content

This wiki is under construction

References

OOM-1 - Getting issue details... STATUS

OOM-3 - Getting issue details... STATUS

OOM-4 - Getting issue details... STATUS

OOM-5 - Getting issue details... STATUS

The OOM (ONAP Operation Manager) project has pushed Kubernetes based deployment code to the oom repository. This page details on getting ONAP running on Kubernetes for various environments.

Note: currently there are no DCAE containers running yet (we are missing 6 yaml files (1 for the controller and 5 for the collector,staging,3-cdap pods)) - therefore DMaaP, VES collectors and APPC actions as the result of policy actions (closed loop) - will not function yet.

Undercloud Installation

We need a kubernetes installation with the proper architecture components running. This architecture can be provided by vendors like Redhat or Rancher

https://kubernetes.io/docs/concepts/overview/components/

There are several options

OS	VIM	Description	Status	Links
OSX Linux	CoreOS	On Vagrant (Thanks Yves)	in-progress	https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant-single.html Implement OSX fix for Vagrant 1.9.6 https://github.com/mitchellh/vagrant/issues/7747 Avoid the kubectl lock https://github.com/coreos/coreos-kubernetes/issues/886 Nexus auth issues TBD
	OSX	MInikube on VMWare Fusion	minikube VM not restartable	https://github.com/kubernetes/minikube
	RHEL 7.3	Redhat Kubernetes	services deploy, but pod IP's not reachable, likely my missing 2 networks (public, onap_oam)	https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html-single/getting_started_with_kubernetes/
	Ubuntu 16.04	Rancher	Issues registering with controller rest endpoint	http://rancher.com/docs/rancher/v1.5/en/quick-start-guide/

Kubernetes specific config

Dashboard

start the dashboard at http://localhost:8001/ui

kubectl proxy &

Nexus Docker repo Credentials

Checking out use of a kubectl secret in the yaml files via - https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

CoreOS on Vagrant

(Yves alerted me to this)

https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant-single.html

Implement OSX fix for Vagrant 1.9.6 https://github.com/mitchellh/vagrant/issues/7747

Adjust the VagrantFile for your system

NODE_VCPUS = 1

NODE_MEMORY_SIZE = 2048

to (for a 5820K on 64G for example)

NODE_VCPUS = 8

NODE_MEMORY_SIZE = 32768

curl -O https://storage.googleapis.com/kubernetes-release/release/v1.6.1/bin/darwin/amd64/kubectl

chmod +x kubectl

skipped (mv kubectl /usr/local/bin/kubectl) - already there

ls /usr/local/bin/kubectl

git clone https://github.com/coreos/coreos-kubernetes.git

cd coreos-kubernetes/single-node/

vagrant box update

sudo ln -sf /usr/local/bin/openssl /opt/vagrant/embedded/bin/openssl

vagrant up

Wait at least 5 min (Yves is good)

(rerun from here)

export KUBECONFIG="${KUBECONFIG}:$(pwd)/kubeconfig"

kubectl config use-context vagrant-single

obrienbiometrics:single-node michaelobrien$ export KUBECONFIG="${KUBECONFIG}:$(pwd)/kubeconfig"

obrienbiometrics:single-node michaelobrien$ kubectl config use-context vagrant-single

Switched to context "vagrant-single".

obrienbiometrics:single-node michaelobrien$ kubectl proxy &

[1] 4079

obrienbiometrics:single-node michaelobrien$ Starting to serve on 127.0.0.1:8001

goto

http://localhost:8001/ui

$ kubectl get nodes

$ kubectl get service --all-namespaces

$ kubectl cluster-info

git clone ssh://michaelobrien@gerrit.onap.org:29418/oom

cd oom/kubernetes/oneclick/

obrienbiometrics:oneclick michaelobrien$ ./createAll.bash -n onap

**** Done ****obrienbiometrics:oneclick michaelobrien$ kubectl get service --all-namespaces

...

onap-vid vid-server 10.3.0.31 <nodes> 8080:30200/TCP 32s

obrienbiometrics:oneclick michaelobrien$ kubectl get pods --all-namespaces

NAMESPACE NAME READY STATUS RESTARTS AGE

kube-system heapster-v1.2.0-4088228293-3k7j1 2/2 Running 2 4h

kube-system kube-apiserver-172.17.4.99 1/1 Running 1 4h

kube-system kube-controller-manager-172.17.4.99 1/1 Running 1 4h

kube-system kube-dns-782804071-jg3nl 4/4 Running 4 4h

kube-system kube-dns-autoscaler-2715466192-k45qg 1/1 Running 1 4h

kube-system kube-proxy-172.17.4.99 1/1 Running 1 4h

kube-system kube-scheduler-172.17.4.99 1/1 Running 1 4h

kube-system kubernetes-dashboard-3543765157-qtnnj 1/1 Running 1 4h

onap-aai aai-service-346921785-w3r22 0/1 Init:0/1 0 1m

...

reset

obrienbiometrics:single-node michaelobrien$ rm -rf ~/.vagrant.d/boxes/coreos-alpha/

Ubuntu 16.04 Install Session

Install Rancher

http://rancher.com/docs/rancher/v1.5/en/quick-start-guide/

http://rancher.com/docs/rancher/v1.6/en/installing-rancher/installing-server/#single-container

Install a docker version that Rancher and Kubernetes support which is currently 1.12.3

http://rancher.com/docs/rancher/v1.5/en/hosts/#supported-docker-versions

curl https://releases.rancher.com/install-docker/1.12.sh \| sh docker run -d --restart=unless-stopped -p 8880:8080 rancher/server:stable

Verify your Rancher admin console is up on the external port you configured above

Wait for the docker container to finish DB startup

Having issues registering a combined single VM (controller + host) - moving on to using 2 VM's

http://rancher.com/docs/rancher/v1.6/en/hosts/

ONAP Installation

Clone

Install the latest version of the OOM (ONAP Operations Manager) project repo - specifically the ONAP on Kubernetes work just uploaded June 2017

https://gerrit.onap.org/r/gitweb?p=oom.git

git clone ssh://michaelobrien@gerrit.onap.org:29418/oom

cd oom/kubernetes/oneclick

OSX

Minicube (not in use)

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl

chmod +x ./kubectl

sudo mv ./kubectl /usr/local/bin/kubectl

kubectl cluster-info

kubectl completion -h

brew install bash-completion

curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.19.0/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

minikube start --vm-driver=vmwarefusion

kubectl run hello-minikube --image=gcr.io/google_containers/echoserver:1.4 --port=8080

kubectl expose deployment hello-minikube --type=NodePort

kubectl get pod

curl $(minikube service hello-minikube --url)

minikube stop

Redhat 7.3

Running onap kubernetes services in a single VM using Redhat Kubernetes for 7.3

Redhat provides 2 docker containers for the scheduler and nbi components and spins up 2 (# is scalable) pod containers for use by onap.

[root@obrien-mbp oneclick]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ee02bbab2037 rhel7/kubernetes-scheduler "/usr/bin/kube-schedu" 40 hours ago Up 40 hours k8s_kube-scheduler.4e069b78_kube-scheduler-127.0.0.1_default_78147ee23cd674839c926daaa58595e5_f4ada53e
f5031b2923ca rhel7/kubernetes-apiserver "/usr/bin/kube-apiser" 40 hours ago Up 40 hours k8s_kube-apiserver.41e368d_kube-apiserver-127.0.0.1_default_ab6617fd8366917b3d6b8c7bb6cbcfcf_8d671f6c
4c5e96ea1074 registry.access.redhat.com/rhel7/pod-infrastructure:latest "/pod" 40 hours ago Up 40 hours k8s_POD.ae8ee9ac_kube-scheduler-127.0.0.1_default_78147ee23cd674839c926daaa58595e5_0ce93fa0
3316c73036fc registry.access.redhat.com/rhel7/pod-infrastructure:latest "/pod" 40 hours ago Up 40 hours k8s_POD.ae8ee9ac_kube-apiserver-127.0.0.1_default_ab6617fd8366917b3d6b8c7bb6cbcfcf_8c0dda0f

Kubernetes setup

Uninstall docker-se (we installed earlier)

Follow https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html-single/getting_started_with_kubernetes/

subscription-manager repos --enable=rhel-7-server-optional-rpms
sudo yum remove docker-ce
sudo yum remove docker-ce-selinux
yum install docker kubernetes-client kubernetes-node etcd
docker ps
systemctl disable firewalld
systemctl stop firewalld
yum install docker-distribution
systemctl start docker-distribution
systemctl enable docker-distribution
systemctl is-active docker-distribution
docker images
systemctl start docker
docker ps
docker images
docker pull registry.access.redhat.com/rhel7/kubernetes-apiserver
docker pull registry.access.redhat.com/rhel7/kubernetes-controller-mgr
docker pull registry.access.redhat.com/rhel7/kubernetes-scheduler
mkdir /etc/kubernetes/manifests
vi /etc/kubernetes/manifests/apiserver-pod.json
vi /etc/kubernetes/manifests/controller-mgr-pod.json
vi /etc/kubernetes/manifests/scheduler-pod.json
vi /etc/kubernetes/kubelet
vi kubestart.sh
chmod 777 kubestart.sh

[root@obrien-mbp opt]# ./kubestart.sh

[root@obrien-mbp opt]# ss -tulnp | grep -E "(kube)|(etcd)"

[root@obrien-mbp opt]# curl -s -L http://localhost:2379/version
{"etcdserver":"3.1.7","etcdcluster":"3.1.0"}[root@obrien-mbp opt]#

Provision

Manually

Start a service

In this case robot - to check your Kubernetes installation.

[root@obrien-mbp oneclick]# ./createAll.bash -n onap -a robot

********** Creating up ONAP: robot

Creating namespaces **********
namespace "onap-robot" created

Creating services **********
service "robot" created

********** Creating deployments for robot **********

Robot....
deployment "robot" created
**** Done ****[root@obrien-mbp oneclick]# kubectl cluster-info
Kubernetes master is running at http://localhost:8080

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Kubernetes Rest api

http://localhost:8080

{ "paths": [ "/api", "/api/v1", ... ] }

Pod List

In verification

[root@obrien-mbp oneclick]# ./createAll.bash -n onap

[root@obrien-mbp oneclick]# kubectl get service --all-namespaces=true

NAMESPACE             NAME                   CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                      AGE

default               kubernetes             10.254.0.1       <none>        443/TCP                                                                      5h

onap-aai              aai-service            10.254.172.161   <nodes>       8443:30233/TCP,8080:30232/TCP                                                1m

...

onap-vid              vid-server             10.254.83.194    <nodes>       8080:30200/TCP                                                               1m

Troubleshooting

Docker Nexus Config