Table of Contents
| Note | ||
|---|---|---|
| ||
This wiki is under construction - this means that content here may be not fully specified or missing. TODO: determine/fix containers not ready, get DCAE yamls working, fix health tracking issues for healing |
References
| Jira | ||||||
|---|---|---|---|---|---|---|
|
| Jira | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
The OOM (ONAP Operation Manager) project has pushed Kubernetes based deployment code to the oom repository. This page details getting ONAP running (specifically the vFirewall demo) on Kubernetes for various virtual and native environments.
Undercloud Installation
We need a kubernetes installation either a base installation or with a thin API wrapper like Rancher or Redhat
There are several options - currently Rancher is a focus as a thin wrapper on Kubernetes.
...
Ubuntu 16.04.2
!Redhat
...
Bare Metal
VMWare
...
Recommended approach
Issue with kubernetes support only in 1.12 (obsolete docker-machine) on OSX
...
https://kubernetes.io/docs/setup/independent/install-kubeadm/
https://lukemarsden.github.io/docs/getting-started-guides/kubeadm/
https://kubernetes.io/docs/getting-started-guides/scratch/
...
OSX
Linux
...
Issue: the coreos VM 19G size is insufficient
...
https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant-single.html
Implement OSX fix for Vagrant 1.9.6 https://github.com/mitchellh/vagrant/issues/7747
Avoid the kubectl lock https://github.com/coreos/coreos-kubernetes/issues/886
...
services deploy, fix kubectl exec
...
ONAP Installation
Quickstart Installation
ONAP deployment in kubernetes is modelled in the oom project as a 1:1 set of service:pod sets (1 pod per docker container). The fastest way to get ONAP Kubernetes up is via Rancher.
Primary platform is virtual Ubuntu 16.04 VMs on VMWare Workstation 12.5 on a up to two separate 64Gb/6-core 5820K Windows 10 systems.
Secondary platform is bare-metal 3 NUCs (i7/i5 with 16G each)
...
Install only the 1.12.x (currently 1.12.6) version of Docker (the only version that works with Kubernetes in Rancher 1.6)
curl https://releases.rancher.com/install-docker/1.12.sh | sh
Install rancher (use 8880 instead of 8080)
sudo docker run -d --restart=unless-stopped -p 8880:8080 rancher/server
In Rancher UI (http://127.0.0.1:8880) , Set IP name of master node in config, create a new onap environment as Kubernetes (will setup kube containers), stop default environment
register your host(s) - run following on each host (get from "add host" menu)
sudo docker run --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.2 http://192.168.163.131:8880/v1/scripts/BBD465D9B24E94F5FBFD:1483142400000:IDaNFrug38QsjZcu6rXh8TwqA4
install kubectl
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
paste kubectl config from rancher
mkdir ~/.kube
vi ~/.kube/config
clone oom (scp your onap_rsa private key first)
git clone ssh://michaelobrien@gerrit.onap.org:29418/oom
fix nexus3 security temporarily for
| Jira | ||||||
|---|---|---|---|---|---|---|
|
Wait until all the hosts show green in rancher, then run the script that wrapps all the kubectl commands
cd oom/kubernetes/oneclick
vi createAll.bash
./createAll.bash -n onap
Wait until the containers are all up - you should see...
Three host Kubernetes cluster in Rancher
In this case 3 Intel NUCs running Ubuntu 16.04.2 natively
Target Deployment State
root@obriensystemsucont0:~/onap/oom/kubernetes/oneclick# kubectl get pods --all-namespaces -o wide
below any colored container has issues getting to running state.
...
NAMESPACE
master:20170705
...
RESTARTS
(in 14h)
...
I get the same container issues on 3 different deployments (virtual Ubuntu on 2 separate VMWare based machines, and a 3 node NUC cluster). For example the APPC 3-pod service is running fine.
But the 2-pod VID service has failed startup.
Cloning details
Install the latest version of the OOM (ONAP Operations Manager) project repo - specifically the ONAP on Kubernetes work just uploaded June 2017
https://gerrit.onap.org/r/gitweb?p=oom.git
...
git clone ssh://yourgerrituserid@gerrit.onap.org:29418/oom
cd oom/kubernetes/oneclick
Versions
oom : master (1.1.0-SNAPSHOT)
onap deployments: 1.0.0
Rancher environment for Kubernetes
Adding hosts to the Kubernetes environment will kick in k8s containers
Rancher kubectl config
Nexus3 security settings
Fix nexus3 security for each namespace
in createAll.bash add the following two lines just before namespace creation - to create a secret and attach it to the namespace (thanks to Jason Hunt of IBM last friday to helping us attach it - when we were all getting our pods to come up). A better fix for the future will be to pass these in as parameters from a prod/stage/dev ecosystem config.
...
create_namespace() {
kubectl create namespace $1-$2
+ kubectl --namespace $1-$2 create secret docker-registry regsecret --docker-server=nexus3.onap.org:10001 --docker-username=docker --docker-password=docker --docker-email=email@email.com
+ kubectl --namespace $1-$2 patch serviceaccount default -p '{"imagePullSecrets": [{"name": "regsecret"}]}'
}
Monitor Container Deployment
first verify your kubernetes system is up
Kubernetes specific config
https://kubernetes.io/docs/user-guide/kubectl-cheatsheet/
Dashboard
start the dashboard at http://localhost:8001/ui
...
Nexus Docker repo Credentials
Checking out use of a kubectl secret in the yaml files via - https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
SSH into ONAP containers
Normally I would via https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/
...
kubectl exec -it robot -- /bin/bashThe pod id should be sufficient
...
root@obriensystemsucont0:~/onap/oom/kubernetes/oneclick# kubectl describe node obriensystemsucont0 | grep robot
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
onap-robot robot-964706867-95hjd 0 (0%) 0 (0%) 0 (0%) 0 (0%)
root@obriensystemsucont0:~/onap/oom/kubernetes/oneclick# kubectl exec -it robot-964706867-95hjd /bin/bash
Error from server (NotFound): pods "robot-964706867-95hjd" not found
root@obriensystemsucont0:~/onap/oom/kubernetes/oneclick# kubectl exec -it robot-964706867 /bin/bash
Error from server (NotFound): pods "robot-964706867" not found
root@obriensystemsucont0:~/onap/oom/kubernetes/oneclick# kubectl exec -it robot /bin/bash
Error from server (NotFound): pods "robot" not found
root@obriensystemsucont0:~/onap/oom/kubernetes/oneclick# kubectl exec -it onap-robot /bin/bash
https://jira.onap.org/browse/OOM-47
in queue....
Running ONAP Portal UI Operations
see Installing and Running the ONAP Demos
In queue.....
Kubernetes Installation Options
Rancher on Ubuntu 16.04
Install Rancher
http://rancher.com/docs/rancher/v1.6/en/quick-start-guide/
http://rancher.com/docs/rancher/v1.6/en/installing-rancher/installing-server/#single-container
Install a docker version that Rancher and Kubernetes support which is currently 1.12.6
http://rancher.com/docs/rancher/v1.5/en/hosts/#supported-docker-versions
...
curl https://releases.rancher.com/install-docker/1.12.sh | sh
docker run -d --restart=unless-stopped -p 8880:8080 rancher/server:stable
...
The official documentation for installation of ONAP with OOM / Kubernetes is located in Read the Docs:
- OOM User Guide — onap master documentation
- OOM Quick Start Guide — onap master documentation)
- OOM Cloud Setup Guide — onap master documentation
Wait for the docker container to finish DB startup
http://rancher.com/docs/rancher/v1.6/en/hosts/
Registering Hosts in Rancher
Having issues registering a combined single VM (controller + host) - use your real IP not localhost
In settings | Host Configuration | set your IP
...
See your host registered
Troubleshooting
Docker Nexus Config
| Jira | ||||||
|---|---|---|---|---|---|---|
|
Out of the box we cant pull images - currently working on a config step around https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
...
imagePullSecrets:
- name: regsecret
...
OOM Repo changes
20170629: fix on 20170626 on a hardcoded proxy - (for those who run outside the firewall) - https://gerrit.onap.org/r/gitweb?p=oom.git;a=commitdiff;h=131c2a42541fb807f395fe1f39a8482a53f92c60
Deprecated Kubernetes Installation Options
Bare RHEL 7.3 VM - Multi Node Cluster
In progress as of 20170701
https://kubernetes.io/docs/getting-started-guides/scratch/
https://github.com/kubernetes/kubernetes/releases/latest
https://github.com/kubernetes/kubernetes/releases/tag/v1.7.0
https://github.com/kubernetes/kubernetes/releases/download/v1.7.0/kubernetes.tar.gz
tar -xvf kubernetes.tar
optional build from source
cd kubernetes/
vi Vagrantfile
cat README.md
ls client/
git clone https://github.com/kubernetes/kubernetes
systemctl start docker
docker ps
cd kubernetes/
make quick-release
go directly to binaries
/run/media/root/sec/onap_kub/kubernetes/cluster
./get-kube-binaries.sh
export Path=/run/media/root/sec/onap_kub/kubernetes/client/bin:$PATH
[root@obrien-b2 server]# pwd
/run/media/root/sec/onap_kub/kubernetes/server
kubernetes-manifests.tar.gz kubernetes-salt.tar.gz kubernetes-server-linux-amd64.tar.gz README
tar -xvf kubernetes-server-linux-amd64.tar.gz
/run/media/root/sec/onap_kub/kubernetes/server/kubernetes/server/bin
build images
[root@obrien-b2 etcd]# make
...
(go lang required - adjust google docs)
https://golang.org/doc/install?download=go1.8.3.linux-amd64.tar.gz
...
(Yves alerted me to this) - currently blocked by the 19g VM size (changing the HD of the VM is unsupported in the VirtualBox driver)
https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant-single.html
Implement OSX fix for Vagrant 1.9.6 https://github.com/mitchellh/vagrant/issues/7747
Adjust the VagrantFile for your system
NODE_VCPUS = 1
NODE_MEMORY_SIZE = 2048
to (for a 5820K on 64G for example)
NODE_VCPUS = 8
NODE_MEMORY_SIZE = 32768
curl -O https://storage.googleapis.com/kubernetes-release/release/v1.6.1/bin/darwin/amd64/kubectl
chmod +x kubectl
skipped (mv kubectl /usr/local/bin/kubectl) - already there
ls /usr/local/bin/kubectl
git clone https://github.com/coreos/coreos-kubernetes.git
cd coreos-kubernetes/single-node/
vagrant box update
sudo ln -sf /usr/local/bin/openssl /opt/vagrant/embedded/bin/openssl
vagrant up
Wait at least 5 min (Yves is good)
(rerun from here)
export KUBECONFIG="${KUBECONFIG}:$(pwd)/kubeconfig"
kubectl config use-context vagrant-single
obrienbiometrics:single-node michaelobrien$ export KUBECONFIG="${KUBECONFIG}:$(pwd)/kubeconfig"
obrienbiometrics:single-node michaelobrien$ kubectl config use-context vagrant-single
Switched to context "vagrant-single".
obrienbiometrics:single-node michaelobrien$ kubectl proxy &
[1] 4079
obrienbiometrics:single-node michaelobrien$ Starting to serve on 127.0.0.1:8001
goto
$ kubectl get nodes
$ kubectl get service --all-namespaces
$ kubectl cluster-info
git clone ssh://michaelobrien@gerrit.onap.org:29418/oom
cd oom/kubernetes/oneclick/
obrienbiometrics:oneclick michaelobrien$ ./createAll.bash -n onap
**** Done ****obrienbiometrics:oneclick michaelobrien$ kubectl get service --all-namespaces
...
onap-vid vid-server 10.3.0.31 <nodes> 8080:30200/TCP 32s
obrienbiometrics:oneclick michaelobrien$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system heapster-v1.2.0-4088228293-3k7j1 2/2 Running 2 4h
kube-system kube-apiserver-172.17.4.99 1/1 Running 1 4h
kube-system kube-controller-manager-172.17.4.99 1/1 Running 1 4h
kube-system kube-dns-782804071-jg3nl 4/4 Running 4 4h
kube-system kube-dns-autoscaler-2715466192-k45qg 1/1 Running 1 4h
kube-system kube-proxy-172.17.4.99 1/1 Running 1 4h
kube-system kube-scheduler-172.17.4.99 1/1 Running 1 4h
kube-system kubernetes-dashboard-3543765157-qtnnj 1/1 Running 1 4h
onap-aai aai-service-346921785-w3r22 0/1 Init:0/1 0 1m
...
reset
obrienbiometrics:single-node michaelobrien$ rm -rf ~/.vagrant.d/boxes/coreos-alpha/
...
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
kubectl cluster-info
kubectl completion -h
brew install bash-completion
curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.19.0/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
minikube start --vm-driver=vmwarefusion
kubectl run hello-minikube --image=gcr.io/google_containers/echoserver:1.4 --port=8080
kubectl expose deployment hello-minikube --type=NodePort
kubectl get pod
curl $(minikube service hello-minikube --url)
minikube stop
When upgrading from 0.19 to 0.20 - do a minikube delete
RHEL Kubernetes - Redhat 7.3 Enterprise Linux Host
Running onap kubernetes services in a single VM using Redhat Kubernetes for 7.3
Redhat provides 2 docker containers for the scheduler and nbi components and spins up 2 (# is scalable) pod containers for use by onap.
...
[root@obrien-mbp oneclick]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ee02bbab2037 rhel7/kubernetes-scheduler "/usr/bin/kube-schedu" 40 hours ago Up 40 hours k8s_kube-scheduler.4e069b78_kube-scheduler-127.0.0.1_default_78147ee23cd674839c926daaa58595e5_f4ada53e
f5031b2923ca rhel7/kubernetes-apiserver "/usr/bin/kube-apiser" 40 hours ago Up 40 hours k8s_kube-apiserver.41e368d_kube-apiserver-127.0.0.1_default_ab6617fd8366917b3d6b8c7bb6cbcfcf_8d671f6c
4c5e96ea1074 registry.access.redhat.com/rhel7/pod-infrastructure:latest "/pod" 40 hours ago Up 40 hours k8s_POD.ae8ee9ac_kube-scheduler-127.0.0.1_default_78147ee23cd674839c926daaa58595e5_0ce93fa0
3316c73036fc registry.access.redhat.com/rhel7/pod-infrastructure:latest "/pod" 40 hours ago Up 40 hours k8s_POD.ae8ee9ac_kube-apiserver-127.0.0.1_default_ab6617fd8366917b3d6b8c7bb6cbcfcf_8c0dda0f
Kubernetes setup
Uninstall docker-se (we installed earlier)
subscription-manager repos --enable=rhel-7-server-optional-rpms
sudo yum remove docker-ce
sudo yum remove docker-ce-selinux
yum install docker kubernetes-client kubernetes-node etcd
docker ps
systemctl disable firewalld
systemctl stop firewalld
yum install docker-distribution
systemctl start docker-distribution
systemctl enable docker-distribution
systemctl is-active docker-distribution
docker images
systemctl start docker
docker ps
docker images
docker pull registry.access.redhat.com/rhel7/kubernetes-apiserver
docker pull registry.access.redhat.com/rhel7/kubernetes-controller-mgr
docker pull registry.access.redhat.com/rhel7/kubernetes-scheduler
mkdir /etc/kubernetes/manifests
vi /etc/kubernetes/manifests/apiserver-pod.json
vi /etc/kubernetes/manifests/controller-mgr-pod.json
vi /etc/kubernetes/manifests/scheduler-pod.json
vi /etc/kubernetes/kubelet
vi kubestart.sh
chmod 777 kubestart.sh
[root@obrien-mbp opt]# ./kubestart.sh
[root@obrien-mbp opt]# ss -tulnp | grep -E "(kube)|(etcd)"
...