You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 73 Next »

Warning: Draft Content

This wiki is under construction - this means that content here may be not fully specified or missing.

TODO: determine/fix containers not ready, get DCAE yamls working, fix health tracking issues for healing


The OOM (ONAP Operation Manager) project has pushed Kubernetes based deployment code to the oom repository.  This page details getting ONAP running (specifically the vFirewall demo) on Kubernetes for various virtual and native environments.

Undercloud Installation

We need a kubernetes installation either a base installation or with a thin API wrapper like Rancher or Redhat

There are several options - currently Rancher is a focus as a thin wrapper on Kubernetes.

OSVIMDescriptionStatusNodesLinks

Ubuntu 16.04.2

!Redhat

Bare Metal

VMWare

Rancher

Recommended approach

Issue with kubernetes support only in 1.12 (obsolete docker-machine) on OSX

1-4http://rancher.com/docs/rancher/v1.6/en/quick-start-guide/

AWS EC2EC2 VM's (not ECS or EBS PaaS)Interesting - in the queue...x
LinuxBare Metal or VMKubernetes Directly on Ubuntu 16 (no Rancher)In progress1

https://kubernetes.io/docs/setup/independent/install-kubeadm/

https://lukemarsden.github.io/docs/getting-started-guides/kubeadm/

https://kubernetes.io/docs/getting-started-guides/scratch/

OSX

Linux

CoreOSOn Vagrant (Thanks Yves)

Issue: the coreos VM 19G size is insufficient

1

https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant-single.html

Implement OSX fix for Vagrant 1.9.6 https://github.com/mitchellh/vagrant/issues/7747

Avoid the kubectl lock https://github.com/coreos/coreos-kubernetes/issues/886


OSXMInikube on VMWare Fusion VMminikube VM not restartable1https://github.com/kubernetes/minikube
RHEL 7.3VMWare VMRedhat Kubernetes

services deploy, fix kubectl exec

1https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html-single/getting_started_with_kubernetes/

ONAP Installation

Quickstart Installation

ONAP deployment in kubernetes is modelled in the oom project as a 1:1 set of service:pod sets (1 pod per docker container).  The fastest way to get ONAP Kubernetes up is via Rancher.

Primary platform is virtual Ubuntu 16.04 VMs on VMWare Workstation 12.5 on a up to two separate 64Gb/6-core 5820K Windows 10 systems.

Secondary platform is bare-metal 3 NUCs (i7/i5 with 16G each) 

Install only the 1.12.x (currently 1.12.6) version of Docker (the only version that works with Kubernetes in Rancher 1.6)
curl https://releases.rancher.com/install-docker/1.12.sh | sh

Install rancher (use 8880 instead of 8080)
sudo docker run -d --restart=unless-stopped -p 8880:8080 rancher/server

In Rancher UI (http://127.0.0.1:8880) , Set IP name of master node in config, create a new onap environment as Kubernetes (will setup kube containers), stop default environment

register your host(s) - run following on each host (get from "add host" menu)

sudo docker run --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.2 http://192.168.163.131:8880/v1/scripts/BBD465D9B24E94F5FBFD:1483142400000:IDaNFrug38QsjZcu6rXh8TwqA4
install kubectl
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

paste kubectl config from rancher

mkdir ~/.kube

vi ~/.kube/config

clone oom (scp your onap_rsa private key first)

git clone ssh://michaelobrien@gerrit.onap.org:29418/oom

fix nexus3 security temporarily for OOM-3 - Getting issue details... STATUS

Wait until all the hosts show green in rancher, then run the script that wrapps all the kubectl commands

cd oom/kubernetes/oneclick
vi createAll.bash 

./createAll.bash -n onap

Wait until the containers are all up - you should see...

Three host Kubernetes cluster in Rancher

In this case 3 Intel NUCs running Ubuntu 16.04.2 natively

Target Deployment State

root@obriensystemsucont0:~/onap/oom/kubernetes/oneclick# kubectl get pods --all-namespaces -o wide

below any colored container has issues getting to running state.

NAMESPACE

master:20170705

NAMEREADY STATUS

RESTARTS

(in 14h)

HostNotes
onap-aai aai-service-346921785-624ss1/1Running01
onap-aai hbase-139474849-7fg0s1/1Running02
onap-aai model-loader-service-1795708961-wg19w0/1Init:1/2 822
onap-appcappc-2044062043-bx6tc1/1Running01
onap-appcappc-dbhost-2039492951-jslts1/1Running02
onap-appcappc-dgbuilder-2934720673-mcp7c1/1Running02
onap-dcaenot yet pushed



Note: currently there are no DCAE containers running yet (we are missing 6 yaml files (1 for the controller and 5 for the collector,staging,3-cdap pods)) - therefore DMaaP, VES collectors and APPC actions as the result of policy actions (closed loop) - will not function yet.
onap-dcae-cdapnot yet pushed




onap-dcae-stgnot yet pushed




onap-dcae-collnot yet pushed




onap-message-routerdmaap-3842712241-gtdkp0/1CrashLoopBackOff1641
onap-message-routerglobal-kafka-89365896-5fnq91/1Running02
onap-message-routerzookeeper-1406540368-jdscq1/1Running01
onap-msomariadb-2638235337-758zr1/1Running01
onap-msomso-3192832250-fq6pn0/1CrashLoopBackOff1672
onap-policy brmsgw-568914601-d5z710/1Init:0/1 821
onap-policy drools-1450928085-099m2 0/1Init:0/1 821
onap-policy mariadb-2932363958-0l05g1/1Running 00
onap-policy nexus-871440171-tqq4z0/1Running 02
onap-policy pap-2218784661-xlj0n1/1Running 01
onap-policy pdp-1677094700-75wpj0/1Init:0/1822
onap-policy pypdp-3209460526-bwm6b0/1Init:0/1822
onap-portal portalapps-1708810953-trz470/1Init:CrashLoopBackOff1632Initial dockerhub mariadb download issue - fixed
onap-portal portaldb-3652211058-vsg8r1/1Running00
onap-portal vnc-portal-948446550-76kj70/1Init:0/5821
onap-robot robot-964706867-czr051/1Running 02
onap-sdcsdc-be-2426613560-jv8sk 0/1Init:0/2822
onap-sdcsdc-cs-2080334320-95dq80/1CrashLoopBackOff1632
onap-sdcsdc-es-3272676451-skf7z1/1Running01
onap-sdcsdc-fe-931927019-nt94t 0/1Init:0/1821
onap-sdcsdc-kb-3337231379-8m8wx0/1Init:0/1821
onap-sdncsdnc-1788655913-vvxlj 1/1Running00
onap-sdncsdnc-dbhost-240465348-kv8vf1/1Running00
onap-sdncsdnc-dgbuilder-4164493163-cp6rx1/1Running00
onap-sdncsdnc-portal-2324831407-50811 0/1Running25=vm
0=nuc
1
onap-vid vid-mariadb-4268497828-81hm00/1CrashLoopBackOff 1692
onap-vid vid-server-2331936551-6gxsp0/1  Init:0/1821

I get the same container issues on 3 different deployments (virtual Ubuntu on 2 separate VMWare based machines, and a 3 node NUC cluster).  For example the APPC 3-pod service is running fine.

But the 2-pod VID service has failed startup.

Cloning details

Install the latest version of the OOM (ONAP Operations Manager) project repo - specifically the ONAP on Kubernetes work just uploaded June 2017

https://gerrit.onap.org/r/gitweb?p=oom.git

git clone ssh://yourgerrituserid@gerrit.onap.org:29418/oom

cd oom/kubernetes/oneclick

Versions

oom : master (1.1.0-SNAPSHOT)

onap deployments: 1.0.0

Rancher environment for Kubernetes

Adding hosts to the Kubernetes environment will kick in k8s containers

Rancher kubectl config

Nexus3 security settings

Fix nexus3 security for each namespace

in createAll.bash add the following two lines just before namespace creation - to create a secret and attach it to the namespace (thanks to Jason Hunt of IBM last friday to helping us attach it - when we were all getting our pods to come up).  A better fix for the future will be to pass these in as parameters from a prod/stage/dev ecosystem config.

create_namespace() {
  kubectl create namespace $1-$2
+  kubectl --namespace $1-$2 create secret docker-registry regsecret --docker-server=nexus3.onap.org:10001 --docker-username=docker --docker-password=docker --docker-email=email@email.com
+  kubectl --namespace $1-$2 patch serviceaccount default -p '{"imagePullSecrets": [{"name": "regsecret"}]}'
}


Monitor Container Deployment

first verify your kubernetes system is up

Kubernetes specific config

https://kubernetes.io/docs/user-guide/kubectl-cheatsheet/

Dashboard

start the dashboard at http://localhost:8001/ui

kubectl proxy &

Nexus Docker repo Credentials

Checking out use of a kubectl secret in the yaml files via - https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

SSH into ONAP containers

Normally I would via https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/

kubectl exec -it robot -- /bin/bash

The pod id should be sufficient

root@obriensystemsucont0:~/onap/oom/kubernetes/oneclick# kubectl describe node obriensystemsucont0 | grep robot

  Namespace            Name                        CPU Requests    CPU Limits    Memory Requests    Memory Limits

  ---------            ----                        ------------    ----------    ---------------    -------------
  onap-robot            robot-964706867-95hjd                0 (0%)        0 (0%)        0 (0%)        0 (0%)
root@obriensystemsucont0:~/onap/oom/kubernetes/oneclick# kubectl exec -it robot-964706867-95hjd /bin/bash
Error from server (NotFound): pods "robot-964706867-95hjd" not found
root@obriensystemsucont0:~/onap/oom/kubernetes/oneclick# kubectl exec -it robot-964706867 /bin/bash
Error from server (NotFound): pods "robot-964706867" not found
root@obriensystemsucont0:~/onap/oom/kubernetes/oneclick# kubectl exec -it robot /bin/bash
Error from server (NotFound): pods "robot" not found
root@obriensystemsucont0:~/onap/oom/kubernetes/oneclick# kubectl exec -it onap-robot /bin/bash

https://jira.onap.org/browse/OOM-47

in queue....

Running ONAP Portal UI Operations

see Installing and Running the ONAP Demos

In queue.....

Kubernetes Installation Options

Rancher on Ubuntu 16.04

Install Rancher

http://rancher.com/docs/rancher/v1.6/en/quick-start-guide/

http://rancher.com/docs/rancher/v1.6/en/installing-rancher/installing-server/#single-container

Install a docker version that Rancher and Kubernetes support which is currently 1.12.6

http://rancher.com/docs/rancher/v1.5/en/hosts/#supported-docker-versions

curl https://releases.rancher.com/install-docker/1.12.sh | sh
docker run -d --restart=unless-stopped -p 8880:8080 rancher/server:stable


Verify your Rancher admin console is up on the external port you configured above

Wait for the docker container to finish DB startup


http://rancher.com/docs/rancher/v1.6/en/hosts/

Registering Hosts in Rancher

Having issues registering a combined single VM (controller + host) - use your real IP not localhost

In settings | Host Configuration | set your IP

[root@obrien-b2 etcd]# sudo docker run -e CATTLE_AGENT_IP="192.168.163.128"  --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.2 http://192.168.163.128:8080/v1/scripts/A9487FC88388CC31FB76:1483142400000:IypSDQCtA4SwkRnthKqH53Vxoo
INFO: Launched Rancher Agent: 1130bdae106396623a01e34a54f72627da2673e466fc78229688330f597ea247

See your host registered

Troubleshooting

Docker Nexus Config

OOM-3 - Getting issue details... STATUS

Out of the box we cant pull images - currently working on a config step around https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

kubectl create secret docker-registry regsecret --docker-server=nexus3.onap.org:10001 --docker-username=docker --docker-password=docker --docker-email=frank.obrien@amdocs.com

      imagePullSecrets:

       - name: regsecret





Failed to pull image "nexus3.onap.org:10001/openecomp/testsuite:1.0-STAGING-latest": image pull failed for nexus3.onap.org:10001/openecomp/testsuite:1.0-STAGING-latest, this may be because there are no credentials on this request. details: (unauthorized: authentication required)
kubelet 172.17.4.99

OOM Repo changes

20170629: fix on 20170626 on a hardcoded proxy - (for those who run outside the firewall) - https://gerrit.onap.org/r/gitweb?p=oom.git;a=commitdiff;h=131c2a42541fb807f395fe1f39a8482a53f92c60


Deprecated Kubernetes Installation Options

Bare RHEL 7.3 VM - Multi Node Cluster

In progress as of 20170701

https://kubernetes.io/docs/getting-started-guides/scratch/

https://github.com/kubernetes/kubernetes/releases/latest

https://github.com/kubernetes/kubernetes/releases/tag/v1.7.0

https://github.com/kubernetes/kubernetes/releases/download/v1.7.0/kubernetes.tar.gz

tar -xvf kubernetes.tar

optional build from source
cd kubernetes/

vi Vagrantfile
cat README.md
ls client/
git clone https://github.com/kubernetes/kubernetes
systemctl start docker
docker ps
cd kubernetes/
make quick-release

go directly to binaries

/run/media/root/sec/onap_kub/kubernetes/cluster

./get-kube-binaries.sh

export Path=/run/media/root/sec/onap_kub/kubernetes/client/bin:$PATH

[root@obrien-b2 server]# pwd

/run/media/root/sec/onap_kub/kubernetes/server

kubernetes-manifests.tar.gz  kubernetes-salt.tar.gz  kubernetes-server-linux-amd64.tar.gz  README

tar -xvf kubernetes-server-linux-amd64.tar.gz

/run/media/root/sec/onap_kub/kubernetes/server/kubernetes/server/bin


build images

[root@obrien-b2 etcd]# make  


[root@obrien-b2 etcd]# pwd
/run/media/root/sec/onap_kub/kubernetes/cluster/images/etcd

    /go/src/github.com/golang/glog (from $GOPATH)
src/k8s.io/kubernetes/cluster/images/etcd/attachlease/attachlease.go:26:2: cannot find package "golang.org/x/net/context" in any of:
    /usr/local/go/src/golang.org/x/net/context (from $GOROOT)
    /go/src/golang.org/x/net/context (from $GOPATH)


(go lang required - adjust google docs)

https://golang.org/doc/install?download=go1.8.3.linux-amd64.tar.gz



CoreOS on Vagrant on RHEL/OSX

(Yves alerted me to this) - currently blocked by the 19g VM size (changing the HD of the VM is unsupported in the VirtualBox driver)

https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant-single.html

Implement OSX fix for Vagrant 1.9.6 https://github.com/mitchellh/vagrant/issues/7747


Adjust the VagrantFile for your system

NODE_VCPUS = 1

NODE_MEMORY_SIZE = 2048


to (for a 5820K on 64G for example)

NODE_VCPUS = 8

NODE_MEMORY_SIZE = 32768

curl -O https://storage.googleapis.com/kubernetes-release/release/v1.6.1/bin/darwin/amd64/kubectl

chmod +x kubectl

skipped (mv kubectl /usr/local/bin/kubectl) - already there

ls /usr/local/bin/kubectl 

git clone https://github.com/coreos/coreos-kubernetes.git

cd coreos-kubernetes/single-node/

vagrant box update

sudo ln -sf /usr/local/bin/openssl /opt/vagrant/embedded/bin/openssl

vagrant up

Wait at least 5 min (Yves is good)

(rerun from here)

export KUBECONFIG="${KUBECONFIG}:$(pwd)/kubeconfig"

kubectl config use-context vagrant-single

obrienbiometrics:single-node michaelobrien$ export KUBECONFIG="${KUBECONFIG}:$(pwd)/kubeconfig"

obrienbiometrics:single-node michaelobrien$ kubectl config use-context vagrant-single

Switched to context "vagrant-single".

obrienbiometrics:single-node michaelobrien$ kubectl proxy &

[1] 4079

obrienbiometrics:single-node michaelobrien$ Starting to serve on 127.0.0.1:8001

goto

http://localhost:8001/ui


$ kubectl get nodes

$ kubectl get service --all-namespaces

$ kubectl cluster-info

git clone ssh://michaelobrien@gerrit.onap.org:29418/oom

cd oom/kubernetes/oneclick/

obrienbiometrics:oneclick michaelobrien$ ./createAll.bash -n onap

**** Done ****obrienbiometrics:oneclick michaelobrien$ kubectl get service --all-namespaces

...

onap-vid              vid-server             10.3.0.31    <nodes>       8080:30200/TCP                                                               32s

obrienbiometrics:oneclick michaelobrien$ kubectl get pods --all-namespaces

NAMESPACE             NAME                                    READY     STATUS              RESTARTS   AGE

kube-system           heapster-v1.2.0-4088228293-3k7j1        2/2       Running             2          4h

kube-system           kube-apiserver-172.17.4.99              1/1       Running             1          4h

kube-system           kube-controller-manager-172.17.4.99     1/1       Running             1          4h

kube-system           kube-dns-782804071-jg3nl                4/4       Running             4          4h

kube-system           kube-dns-autoscaler-2715466192-k45qg    1/1       Running             1          4h

kube-system           kube-proxy-172.17.4.99                  1/1       Running             1          4h

kube-system           kube-scheduler-172.17.4.99              1/1       Running             1          4h

kube-system           kubernetes-dashboard-3543765157-qtnnj   1/1       Running             1          4h

onap-aai              aai-service-346921785-w3r22             0/1       Init:0/1            0          1m

...

reset

obrienbiometrics:single-node michaelobrien$ rm -rf ~/.vagrant.d/boxes/coreos-alpha/


OSX Minikube

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl

 chmod +x ./kubectl 

sudo mv ./kubectl /usr/local/bin/kubectl

kubectl cluster-info

kubectl completion -h

brew install bash-completion

curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.19.0/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

minikube start --vm-driver=vmwarefusion

kubectl run hello-minikube --image=gcr.io/google_containers/echoserver:1.4 --port=8080

kubectl expose deployment hello-minikube --type=NodePort

kubectl get pod

curl $(minikube service hello-minikube --url)

minikube stop

When upgrading from 0.19 to 0.20 - do a minikube delete

RHEL Kubernetes - Redhat 7.3 Enterprise Linux Host

Running onap kubernetes services in a single VM using Redhat Kubernetes for 7.3

Redhat provides 2 docker containers for the scheduler and nbi components and spins up 2 (# is scalable) pod containers for use by onap.

[root@obrien-mbp oneclick]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ee02bbab2037 rhel7/kubernetes-scheduler "/usr/bin/kube-schedu" 40 hours ago Up 40 hours k8s_kube-scheduler.4e069b78_kube-scheduler-127.0.0.1_default_78147ee23cd674839c926daaa58595e5_f4ada53e
f5031b2923ca rhel7/kubernetes-apiserver "/usr/bin/kube-apiser" 40 hours ago Up 40 hours k8s_kube-apiserver.41e368d_kube-apiserver-127.0.0.1_default_ab6617fd8366917b3d6b8c7bb6cbcfcf_8d671f6c
4c5e96ea1074 registry.access.redhat.com/rhel7/pod-infrastructure:latest "/pod" 40 hours ago Up 40 hours k8s_POD.ae8ee9ac_kube-scheduler-127.0.0.1_default_78147ee23cd674839c926daaa58595e5_0ce93fa0
3316c73036fc registry.access.redhat.com/rhel7/pod-infrastructure:latest "/pod" 40 hours ago Up 40 hours k8s_POD.ae8ee9ac_kube-apiserver-127.0.0.1_default_ab6617fd8366917b3d6b8c7bb6cbcfcf_8c0dda0f


Kubernetes setup

Uninstall docker-se (we installed earlier)

Follow https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html-single/getting_started_with_kubernetes/

subscription-manager repos --enable=rhel-7-server-optional-rpms
sudo yum remove docker-ce
sudo yum remove docker-ce-selinux
yum install docker kubernetes-client kubernetes-node etcd
docker ps
systemctl disable firewalld
systemctl stop firewalld
yum install docker-distribution
systemctl start docker-distribution
systemctl enable docker-distribution
systemctl is-active docker-distribution
docker images
systemctl start docker
docker ps
docker images
docker pull registry.access.redhat.com/rhel7/kubernetes-apiserver
docker pull registry.access.redhat.com/rhel7/kubernetes-controller-mgr
docker pull registry.access.redhat.com/rhel7/kubernetes-scheduler
mkdir /etc/kubernetes/manifests
vi /etc/kubernetes/manifests/apiserver-pod.json
vi /etc/kubernetes/manifests/controller-mgr-pod.json
vi /etc/kubernetes/manifests/scheduler-pod.json
vi /etc/kubernetes/kubelet 
vi kubestart.sh
chmod 777 kubestart.sh

[root@obrien-mbp opt]# ./kubestart.sh 

[root@obrien-mbp opt]# ss -tulnp | grep -E "(kube)|(etcd)"


[root@obrien-mbp opt]# curl -s -L http://localhost:2379/version
{"etcdserver":"3.1.7","etcdcluster":"3.1.0"}[root@obrien-mbp opt]#

References

OOM-1 - Getting issue details... STATUS

Key Summary T Created Updated Due Assignee Reporter P Status Resolution
Loading...
Refresh

https://kubernetes.io/docs/user-guide/kubectl-cheatsheet/

  • No labels