Table of Contents
Note | ||
---|---|---|
| ||
This wiki is under construction |
References
Jira | ||||||
---|---|---|---|---|---|---|
|
The OOM (ONAP Operation Manager) project has pushed Kubernetes based deployment code to the oom repository. This page details on getting ONAP running on Kubernetes for various environments.
Note: currently there is no DCAE containers running yet therefore DMaaP, VES collectors and APPC actions as the result of policy actions (closed loop) - will not function yet.
Undercloud Installation
We need a kubernetes installation with the proper architecture components running. This architecture can be provided by vendors like Redhat or Rancher
https://kubernetes.io/docs/concepts/overview/components/
There are several options
...
https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant-single.html
Implement OSX fix for Vagrant 1.9.6 https://github.com/mitchellh/vagrant/issues/7747
Avoid the kubectl lock https://github.com/coreos/coreos-kubernetes/issues/886
CoreOS on Vagrant
(Yves alerted me to this)
https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant-single.html
Implement OSX fix for Vagrant 1.9.6 https://github.com/mitchellh/vagrant/issues/7747
curl -O https://storage.googleapis.com/kubernetes-release/release/v1.6.1/bin/darwin/amd64/kubectl
chmod +x kubectl
skipped (mv kubectl /usr/local/bin/kubectl) - already there
ls /usr/local/bin/kubectl
git clone https://github.com/coreos/coreos-kubernetes.git
cd coreos-kubernetes/single-node/
vagrant box update
sudo ln -sf /usr/local/bin/openssl /opt/vagrant/embedded/bin/openssl
vagrant up
Wait at least 5 min (Yves is good)
export KUBECONFIG="${KUBECONFIG}:$(pwd)/kubeconfig"
kubectl config use-context vagrant-single
obrienbiometrics:single-node michaelobrien$ export KUBECONFIG="${KUBECONFIG}:$(pwd)/kubeconfig"
obrienbiometrics:single-node michaelobrien$ kubectl config use-context vagrant-single
Switched to context "vagrant-single".
$ kubectl get nodes
NAME STATUS AGE VERSION
172.17.4.99 Ready 4h v1.5.4+coreos.0
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system heapster-v1.2.0-4088228293-3k7j1 2/2 Running 2 4h
kube-system kube-apiserver-172.17.4.99 1/1 Running 1 4h
kube-system kube-controller-manager-172.17.4.99 1/1 Running 1 4h
kube-system kube-dns-782804071-jg3nl 4/4 Running 4 4h
kube-system kube-dns-autoscaler-2715466192-k45qg 1/1 Running 1 4h
kube-system kube-proxy-172.17.4.99 1/1 Running 1 4h
kube-system kube-scheduler-172.17.4.99 1/1 Running 1 4h
kube-system kubernetes-dashboard-3543765157-qtnnj 1/1 Running 1 4h
$ kubectl get service --all-namespaces
NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes 10.3.0.1 <none> 443/TCP 4h
kube-system heapster 10.3.0.95 <none> 80/TCP 4h
kube-system kube-dns 10.3.0.10 <none> 53/UDP,53/TCP 4h
kube-system kubernetes-dashboard 10.3.0.66 <none> 80/TCP 4h
$ kubectl cluster-info
Kubernetes master is running at https://172.17.4.99:443
Heapster is running at https://172.17.4.99:443/api/v1/proxy/namespaces/kube-system/services/heapster
KubeDNS is running at https://172.17.4.99:443/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://172.17.4.99:443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
git clone ssh://michaelobrien@gerrit.onap.org:29418/oom
cd oom/kubernetes/oneclick/
./createAll.bash -n onap
kubectl cluster-info
Ubuntu 16.04 Install Session
Install Rancher
http://rancher.com/docs/rancher/v1.5/en/quick-start-guide/
http://rancher.com/docs/rancher/v1.6/en/installing-rancher/installing-server/#single-container
Install a docker version that Rancher and Kubernetes support which is currently 1.12.3
http://rancher.com/docs/rancher/v1.5/en/hosts/#supported-docker-versions
...
curl https://releases.rancher.com/install-docker/1.12.sh | sh
docker run -d --restart=unless-stopped -p 8880:8080 rancher/server:stable
Verify your Rancher admin console is up on the external port you configured above
Wait for the docker container to finish DB startup
Having issues registering a combined single VM (controller + host) - moving on to using 2 VM's
http://rancher.com/docs/rancher/v1.6/en/hosts/
ONAP Installation
Clone
Install the latest version of the OOM (ONAP Operations Manager) project repo - specifically the ONAP on Kubernetes work just uploaded June 2017
https://gerrit.onap.org/r/gitweb?p=oom.git
...
git clone ssh://michaelobrien@gerrit.onap.org:29418/oom
cd oom/kubernetes/oneclick
OSX
Minicube (not in use)
...
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
kubectl cluster-info
kubectl completion -h
brew install bash-completion
curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.19.0/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
minikube start --vm-driver=vmwarefusion
kubectl run hello-minikube --image=gcr.io/google_containers/echoserver:1.4 --port=8080
kubectl expose deployment hello-minikube --type=NodePort
kubectl get pod
curl $(minikube service hello-minikube --url)
minikube stop
Redhat 7.3
Running onap kubernetes services in a single VM using Redhat Kubernetes for 7.3
Redhat provides 2 docker containers for the scheduler and nbi components and spins up 2 (# is scalable) pod containers for use by onap.
...
[root@obrien-mbp oneclick]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ee02bbab2037 rhel7/kubernetes-scheduler "/usr/bin/kube-schedu" 40 hours ago Up 40 hours k8s_kube-scheduler.4e069b78_kube-scheduler-127.0.0.1_default_78147ee23cd674839c926daaa58595e5_f4ada53e
f5031b2923ca rhel7/kubernetes-apiserver "/usr/bin/kube-apiser" 40 hours ago Up 40 hours k8s_kube-apiserver.41e368d_kube-apiserver-127.0.0.1_default_ab6617fd8366917b3d6b8c7bb6cbcfcf_8d671f6c
4c5e96ea1074 registry.access.redhat.com/rhel7/pod-infrastructure:latest "/pod" 40 hours ago Up 40 hours k8s_POD.ae8ee9ac_kube-scheduler-127.0.0.1_default_78147ee23cd674839c926daaa58595e5_0ce93fa0
3316c73036fc registry.access.redhat.com/rhel7/pod-infrastructure:latest "/pod" 40 hours ago Up 40 hours k8s_POD.ae8ee9ac_kube-apiserver-127.0.0.1_default_ab6617fd8366917b3d6b8c7bb6cbcfcf_8c0dda0f
Kubernetes setup
...
Uninstall docker-se (we installed earlier)
subscription-manager repos --enable=rhel-7-server-optional-rpms
sudo yum remove docker-ce
sudo yum remove docker-ce-selinux
yum install docker kubernetes-client kubernetes-node etcd
docker ps
systemctl disable firewalld
systemctl stop firewalld
yum install docker-distribution
systemctl start docker-distribution
systemctl enable docker-distribution
systemctl is-active docker-distribution
docker images
systemctl start docker
docker ps
docker images
docker pull registry.access.redhat.com/rhel7/kubernetes-apiserver
docker pull registry.access.redhat.com/rhel7/kubernetes-controller-mgr
docker pull registry.access.redhat.com/rhel7/kubernetes-scheduler
mkdir /etc/kubernetes/manifests
vi /etc/kubernetes/manifests/apiserver-pod.json
vi /etc/kubernetes/manifests/controller-mgr-pod.json
vi /etc/kubernetes/manifests/scheduler-pod.json
vi /etc/kubernetes/kubelet
vi kubestart.sh
chmod 777 kubestart.sh
[root@obrien-mbp opt]# ./kubestart.sh
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
active
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
active
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
active
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
active
? docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/docker.service.d
??flannel.conf
Active: active (running) since Tue 2017-06-27 14:26:09 PDT; 196ms ago
Docs: http://docs.docker.com
Main PID: 90732 (dockerd-current)
Memory: 14.6M
CGroup: /system.slice/docker.service
??90732 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-p...
??90740 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 -...
Jun 27 14:26:08 obrien-mbp.onap.org dockerd-current[90732]: time="2017-06-27T14:26:08.923309259-07:00" level=info msg="[graphdriver] using prior storage driver \"overlay\""
Jun 27 14:26:08 obrien-mbp.onap.org dockerd-current[90732]: time="2017-06-27T14:26:08.932489041-07:00" level=info msg="Graph migration to content-addressability... seconds"
Jun 27 14:26:08 obrien-mbp.onap.org dockerd-current[90732]: time="2017-06-27T14:26:08.933459958-07:00" level=info msg="Loading containers: start."
Jun 27 14:26:08 obrien-mbp.onap.org dockerd-current[90732]: .time="2017-06-27T14:26:08.944192986-07:00" level=info msg="Firewalld running: false"
Jun 27 14:26:09 obrien-mbp.onap.org dockerd-current[90732]: time="2017-06-27T14:26:09.063447350-07:00" level=info msg="Default bridge (docker0) is assigned with... address"
Jun 27 14:26:09 obrien-mbp.onap.org dockerd-current[90732]: time="2017-06-27T14:26:09.106280840-07:00" level=info msg="Loading containers: done."
Jun 27 14:26:09 obrien-mbp.onap.org dockerd-current[90732]: time="2017-06-27T14:26:09.106331336-07:00" level=info msg="Daemon has completed initialization"
Jun 27 14:26:09 obrien-mbp.onap.org dockerd-current[90732]: time="2017-06-27T14:26:09.106343838-07:00" level=info msg="Docker daemon" commit="1398f24/1.12.6" gr...on=1.12.6
Jun 27 14:26:09 obrien-mbp.onap.org dockerd-current[90732]: time="2017-06-27T14:26:09.116502253-07:00" level=info msg="API listen on /var/run/docker.sock"
Jun 27 14:26:09 obrien-mbp.onap.org systemd[1]: Started Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.
? kube-proxy.service - Kubernetes Kube-Proxy Server
Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2017-06-27 14:26:09 PDT; 165ms ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 90864 (kube-proxy)
Memory: 14.5M
CGroup: /system.slice/kube-proxy.service
??90864 /usr/bin/kube-proxy --logtostderr=true --v=0 --master=http://127.0.0.1:8080
Jun 27 14:26:09 obrien-mbp.onap.org systemd[1]: Started Kubernetes Kube-Proxy Server.
Jun 27 14:26:09 obrien-mbp.onap.org systemd[1]: Starting Kubernetes Kube-Proxy Server...
? kubelet.service - Kubernetes Kubelet Server
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2017-06-27 14:26:09 PDT; 151ms ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 90917 (kubelet)
Memory: 13.9M
CGroup: /system.slice/kubelet.service
??90917 /usr/bin/kubelet --logtostderr=true --v=0 --api-servers=http://127.0.0.1:8080 --address=127.0.0.1 --hostname-override=127.0.0.1 --allow-privileged=fal...
Jun 27 14:26:09 obrien-mbp.onap.org systemd[1]: Started Kubernetes Kubelet Server.
Jun 27 14:26:09 obrien-mbp.onap.org systemd[1]: Starting Kubernetes Kubelet Server...
Jun 27 14:26:09 obrien-mbp.onap.org kubelet[90917]: Flag --api-servers has been deprecated, Use --kubeconfig instead. Will be removed in a future version.
Jun 27 14:26:09 obrien-mbp.onap.org kubelet[90917]: Flag --config has been deprecated, Use --pod-manifest-path instead. Will be removed in a future version.
Jun 27 14:26:09 obrien-mbp.onap.org kubelet[90917]: I0627 14:26:09.697783 90917 feature_gate.go:181] feature gates: map[]
Jun 27 14:26:09 obrien-mbp.onap.org kubelet[90917]: W0627 14:26:09.698143 90917 server.go:605] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: sta... instead.
Jun 27 14:26:09 obrien-mbp.onap.org kubelet[90917]: I0627 14:26:09.698654 90917 docker.go:356] Connecting to docker on unix:///var/run/docker.sock
Jun 27 14:26:09 obrien-mbp.onap.org kubelet[90917]: I0627 14:26:09.698667 90917 docker.go:376] Start docker client with request timeout=2m0s
Jun 27 14:26:09 obrien-mbp.onap.org kubelet[90917]: E0627 14:26:09.702077 90917 cni.go:163] error updating cni config: No networks found in /etc/cni/net.d
Jun 27 14:26:09 obrien-mbp.onap.org kubelet[90917]: I0627 14:26:09.711775 90917 manager.go:143] cAdvisor running in container: "/system.slice/kubelet.service"
Hint: Some lines were ellipsized, use -l to show in full.
[root@obrien-mbp opt]# ss -tulnp | grep -E "(kube)|(etcd)"
tcp LISTEN 0 128 127.0.0.1:2379 *:* users:(("etcd",pid=90567,fd=6))
tcp LISTEN 0 128 127.0.0.1:2380 *:* users:(("etcd",pid=90567,fd=5))
tcp LISTEN 0 128 127.0.0.1:10255 *:* users:(("kubelet",pid=90917,fd=9))
tcp LISTEN 0 128 127.0.0.1:10248 *:* users:(("kubelet",pid=90917,fd=8))
tcp LISTEN 0 128 127.0.0.1:10249 *:* users:(("kube-proxy",pid=90864,fd=9))
tcp LISTEN 0 128 127.0.0.1:10250 *:* users:(("kubelet",pid=90917,fd=14))
tcp LISTEN 0 128 :::10251 :::* users:(("kube-scheduler",pid=91175,fd=8))
tcp LISTEN 0 128 :::6443 :::* users:(("kube-apiserver",pid=91126,fd=42))
tcp LISTEN 0 128 :::8080 :::* users:(("kube-apiserver",pid=91126,fd=43))
tcp LISTEN 0 128 :::4194 :::* users:(("kubelet",pid=90917,fd=13))
[root@obrien-mbp opt]# curl -s -L http://localhost:2379/version
{"etcdserver":"3.1.7","etcdcluster":"3.1.0"}[root@obrien-mbp opt]#
Provision
Manually
Start a service
In this case robot - to check your Kubernetes installation.
[root@obrien-mbp oneclick]# ./createAll.bash -n onap -a robot
********** Creating up ONAP: robot
Creating namespaces **********
namespace "onap-robot" created
Creating services **********
service "robot" created
********** Creating deployments for robot **********
Robot....
deployment "robot" created
**** Done ****[root@obrien-mbp oneclick]# kubectl cluster-info
Kubernetes master is running at http://localhost:8080
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The official documentation for installation of ONAP with OOM / Kubernetes is located in Read the Docs:
- OOM User Guide — onap master documentation
- OOM Quick Start Guide — onap master documentation)
- OOM Cloud Setup Guide — onap master documentation
Kubernetes Rest api
...
{
"paths": [
"/api",
"/api/v1",
"/apis",
"/apis/apps",
"/apis/apps/v1beta1",
"/apis/authentication.k8s.io",
"/apis/authentication.k8s.io/v1beta1",
"/apis/authorization.k8s.io",
"/apis/authorization.k8s.io/v1beta1",
"/apis/autoscaling",
"/apis/autoscaling/v1",
"/apis/batch",
"/apis/batch/v1",
"/apis/batch/v2alpha1",
"/apis/certificates.k8s.io",
"/apis/certificates.k8s.io/v1alpha1",
"/apis/extensions",
"/apis/extensions/v1beta1",
"/apis/policy",
"/apis/policy/v1beta1",
"/apis/rbac.authorization.k8s.io",
"/apis/rbac.authorization.k8s.io/v1alpha1",
"/apis/storage.k8s.io",
"/apis/storage.k8s.io/v1beta1",
"/healthz",
"/healthz/ping",
"/healthz/poststarthook/bootstrap-controller",
"/healthz/poststarthook/extensions/third-party-resources",
"/healthz/poststarthook/rbac/bootstrap-roles",
"/logs",
"/metrics",
"/swaggerapi/",
"/ui/",
"/version"
]
}
Pod List
In verification
[root@obrien-mbp oneclick]# ./createAll.bash -n onap
[root@obrien-mbp oneclick]# kubectl get service --all-namespaces=true
NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes 10.254.0.1 <none> 443/TCP 5h
onap-aai aai-service 10.254.172.161 <nodes> 8443:30233/TCP,8080:30232/TCP 1m
onap-aai hbase None <none> 8020/TCP 1m
onap-aai model-loader-service 10.254.138.128 <nodes> 8443:30229/TCP,8080:30210/TCP 1m
onap-appc dbhost None <none> 3306/TCP 1m
onap-appc dgbuilder 10.254.89.166 <nodes> 3000:30228/TCP 1m
onap-appc sdnctldb01 None <none> 3306/TCP 1m
onap-appc sdnctldb02 None <none> 3306/TCP 1m
onap-appc sdnhost 10.254.68.235 <nodes> 8282:30230/TCP,1830:30231/TCP 1m
onap-message-router dmaap 10.254.149.48 <nodes> 3904:30227/TCP,3905:30226/TCP 1m
onap-message-router global-kafka None <none> 9092/TCP 1m
onap-message-router zookeeper None <none> 2181/TCP 1m
onap-mso mariadb 10.254.254.189 <nodes> 3306:30252/TCP 1m
onap-mso mso 10.254.32.65 <nodes> 8080:30223/TCP,3904:30225/TCP,3905:30224/TCP,9990:30222/TCP,8787:30250/TCP 1m
onap-policy brmsgw 10.254.207.78 <nodes> 9989:30216/TCP 1m
onap-policy drools 10.254.114.125 <nodes> 6969:30217/TCP 1m
onap-policy mariadb None <none> 3306/TCP 1m
onap-policy nexus None <none> 8081/TCP 1m
onap-policy pap 10.254.126.231 <nodes> 8443:30219/TCP,9091:30218/TCP 1m
onap-policy pdp 10.254.244.112 <nodes> 8081:30220/TCP 1m
onap-policy pypdp 10.254.176.200 <nodes> 8480:30221/TCP 1m
onap-portal portalapps 10.254.198.12 <nodes> 8006:30213/TCP,8010:30214/TCP,8989:30215/TCP 1m
onap-portal portaldb None <none> 3306/TCP 1m
onap-portal vnc-portal 10.254.0.129 <nodes> 6080:30211/TCP,5900:30212/TCP 1m
onap-robot robot 10.254.255.97 <nodes> 88:30209/TCP 5h
onap-sdc sdc-be 10.254.109.11 <nodes> 8443:30204/TCP,8080:30205/TCP 1m
onap-sdc sdc-cs None <none> 9042/TCP,9160/TCP 1m
onap-sdc sdc-es None <none> 9200/TCP,9300/TCP 1m
onap-sdc sdc-fe 10.254.219.219 <nodes> 9443:30207/TCP,8181:30206/TCP 1m
onap-sdc sdc-kb None <none> 5601/TCP 1m
onap-sdnc dbhost None <none> 3306/TCP 1m
onap-sdnc sdnc-dgbuilder 10.254.79.74 <nodes> 3000:30203/TCP 1m
onap-sdnc sdnc-portal 10.254.162.50 <nodes> 8843:30201/TCP 1m
onap-sdnc sdnctldb01 None <none> 3306/TCP 1m
onap-sdnc sdnctldb02 None <none> 3306/TCP 1m
onap-sdnc sdnhost 10.254.233.66 <nodes> 8282:30202/TCP 1m
onap-vid vid-mariadb None <none> 3306/TCP 1m
onap-vid vid-server 10.254.83.194 <nodes> 8080:30200/TCP 1m
Troubleshooting
...