Table of Contents
Note | ||
---|---|---|
| ||
This wiki is under construction |
References
Jira | ||||||
---|---|---|---|---|---|---|
|
The OOM (ONAP Operation Manager) project has pushed Kubernetes based deployment code to the oom repository. This page details on getting ONAP running on Kubernetes for various environments.
Undercloud Installation
We need a kubernetes installation with the proper architecture components running. This architecture can be provided by vendors like Redhat or Rancher
https://kubernetes.io/docs/concepts/overview/components/
There are several options
...
https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant-single.html
Implement OSX fix for Vagrant 1.9.6 https://github.com/mitchellh/vagrant/issues/7747
Ubuntu 16.04 Install Session
Install Rancher
http://rancher.com/docs/rancher/v1.5/en/quick-start-guide/
http://rancher.com/docs/rancher/v1.6/en/installing-rancher/installing-server/#single-container
Install a docker version that Rancher and Kubernetes support which is currently 1.12.3
http://rancher.com/docs/rancher/v1.5/en/hosts/#supported-docker-versions
...
curl https://releases.rancher.com/install-docker/1.12.sh | sh
docker run -d --restart=unless-stopped -p 8880:8080 rancher/server:stable
Verify your Rancher admin console is up on the external port you configured above
Wait for the docker container to finish DB startup
Having issues registering a combined single VM (controller + host) - moving on to using 2 VM's
http://rancher.com/docs/rancher/v1.6/en/hosts/
ONAP Installation
Clone
Install the latest version of the OOM (ONAP Operations Manager) project repo - specifically the ONAP on Kubernetes work just uploaded June 2017
https://gerrit.onap.org/r/gitweb?p=oom.git
...
git clone ssh://michaelobrien@gerrit.onap.org:29418/oom
cd oom/kubernetes/oneclick
OSX
Minicube (not in use)
...
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
kubectl cluster-info
kubectl completion -h
brew install bash-completion
curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.19.0/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
minikube start --vm-driver=vmwarefusion
kubectl run hello-minikube --image=gcr.io/google_containers/echoserver:1.4 --port=8080
kubectl expose deployment hello-minikube --type=NodePort
kubectl get pod
curl $(minikube service hello-minikube --url)
minikube stop
Redhat 7.3
Running onap kubernetes services in a single VM using Redhat Kubernetes for 7.3
Redhat provides 2 docker containers for the scheduler and nbi components and spins up 2 (# is scalable) pod containers for use by onap.
...
[root@obrien-mbp oneclick]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ee02bbab2037 rhel7/kubernetes-scheduler "/usr/bin/kube-schedu" 40 hours ago Up 40 hours k8s_kube-scheduler.4e069b78_kube-scheduler-127.0.0.1_default_78147ee23cd674839c926daaa58595e5_f4ada53e
f5031b2923ca rhel7/kubernetes-apiserver "/usr/bin/kube-apiser" 40 hours ago Up 40 hours k8s_kube-apiserver.41e368d_kube-apiserver-127.0.0.1_default_ab6617fd8366917b3d6b8c7bb6cbcfcf_8d671f6c
4c5e96ea1074 registry.access.redhat.com/rhel7/pod-infrastructure:latest "/pod" 40 hours ago Up 40 hours k8s_POD.ae8ee9ac_kube-scheduler-127.0.0.1_default_78147ee23cd674839c926daaa58595e5_0ce93fa0
3316c73036fc registry.access.redhat.com/rhel7/pod-infrastructure:latest "/pod" 40 hours ago Up 40 hours k8s_POD.ae8ee9ac_kube-apiserver-127.0.0.1_default_ab6617fd8366917b3d6b8c7bb6cbcfcf_8c0dda0f
Kubernetes setup
...
Uninstall docker-se (we installed earlier)
subscription-manager repos --enable=rhel-7-server-optional-rpms
sudo yum remove docker-ce
sudo yum remove docker-ce-selinux
yum install docker kubernetes-client kubernetes-node etcd
docker ps
systemctl disable firewalld
systemctl stop firewalld
yum install docker-distribution
systemctl start docker-distribution
systemctl enable docker-distribution
systemctl is-active docker-distribution
docker images
systemctl start docker
docker ps
docker images
docker pull registry.access.redhat.com/rhel7/kubernetes-apiserver
docker pull registry.access.redhat.com/rhel7/kubernetes-controller-mgr
docker pull registry.access.redhat.com/rhel7/kubernetes-scheduler
mkdir /etc/kubernetes/manifests
vi /etc/kubernetes/manifests/apiserver-pod.json
vi /etc/kubernetes/manifests/controller-mgr-pod.json
vi /etc/kubernetes/manifests/scheduler-pod.json
vi /etc/kubernetes/kubelet
vi kubestart.sh
chmod 777 kubestart.sh
[root@obrien-mbp opt]# ./kubestart.sh
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
active
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
active
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
active
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
active
? docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/docker.service.d
??flannel.conf
Active: active (running) since Tue 2017-06-27 14:26:09 PDT; 196ms ago
Docs: http://docs.docker.com
Main PID: 90732 (dockerd-current)
Memory: 14.6M
CGroup: /system.slice/docker.service
??90732 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-p...
??90740 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 -...
Jun 27 14:26:08 obrien-mbp.onap.org dockerd-current[90732]: time="2017-06-27T14:26:08.923309259-07:00" level=info msg="[graphdriver] using prior storage driver \"overlay\""
Jun 27 14:26:08 obrien-mbp.onap.org dockerd-current[90732]: time="2017-06-27T14:26:08.932489041-07:00" level=info msg="Graph migration to content-addressability... seconds"
Jun 27 14:26:08 obrien-mbp.onap.org dockerd-current[90732]: time="2017-06-27T14:26:08.933459958-07:00" level=info msg="Loading containers: start."
Jun 27 14:26:08 obrien-mbp.onap.org dockerd-current[90732]: .time="2017-06-27T14:26:08.944192986-07:00" level=info msg="Firewalld running: false"
Jun 27 14:26:09 obrien-mbp.onap.org dockerd-current[90732]: time="2017-06-27T14:26:09.063447350-07:00" level=info msg="Default bridge (docker0) is assigned with... address"
Jun 27 14:26:09 obrien-mbp.onap.org dockerd-current[90732]: time="2017-06-27T14:26:09.106280840-07:00" level=info msg="Loading containers: done."
Jun 27 14:26:09 obrien-mbp.onap.org dockerd-current[90732]: time="2017-06-27T14:26:09.106331336-07:00" level=info msg="Daemon has completed initialization"
Jun 27 14:26:09 obrien-mbp.onap.org dockerd-current[90732]: time="2017-06-27T14:26:09.106343838-07:00" level=info msg="Docker daemon" commit="1398f24/1.12.6" gr...on=1.12.6
Jun 27 14:26:09 obrien-mbp.onap.org dockerd-current[90732]: time="2017-06-27T14:26:09.116502253-07:00" level=info msg="API listen on /var/run/docker.sock"
Jun 27 14:26:09 obrien-mbp.onap.org systemd[1]: Started Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.
? kube-proxy.service - Kubernetes Kube-Proxy Server
Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2017-06-27 14:26:09 PDT; 165ms ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 90864 (kube-proxy)
Memory: 14.5M
CGroup: /system.slice/kube-proxy.service
??90864 /usr/bin/kube-proxy --logtostderr=true --v=0 --master=http://127.0.0.1:8080
Jun 27 14:26:09 obrien-mbp.onap.org systemd[1]: Started Kubernetes Kube-Proxy Server.
Jun 27 14:26:09 obrien-mbp.onap.org systemd[1]: Starting Kubernetes Kube-Proxy Server...
? kubelet.service - Kubernetes Kubelet Server
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2017-06-27 14:26:09 PDT; 151ms ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 90917 (kubelet)
Memory: 13.9M
CGroup: /system.slice/kubelet.service
??90917 /usr/bin/kubelet --logtostderr=true --v=0 --api-servers=http://127.0.0.1:8080 --address=127.0.0.1 --hostname-override=127.0.0.1 --allow-privileged=fal...
Jun 27 14:26:09 obrien-mbp.onap.org systemd[1]: Started Kubernetes Kubelet Server.
Jun 27 14:26:09 obrien-mbp.onap.org systemd[1]: Starting Kubernetes Kubelet Server...
Jun 27 14:26:09 obrien-mbp.onap.org kubelet[90917]: Flag --api-servers has been deprecated, Use --kubeconfig instead. Will be removed in a future version.
Jun 27 14:26:09 obrien-mbp.onap.org kubelet[90917]: Flag --config has been deprecated, Use --pod-manifest-path instead. Will be removed in a future version.
Jun 27 14:26:09 obrien-mbp.onap.org kubelet[90917]: I0627 14:26:09.697783 90917 feature_gate.go:181] feature gates: map[]
Jun 27 14:26:09 obrien-mbp.onap.org kubelet[90917]: W0627 14:26:09.698143 90917 server.go:605] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: sta... instead.
Jun 27 14:26:09 obrien-mbp.onap.org kubelet[90917]: I0627 14:26:09.698654 90917 docker.go:356] Connecting to docker on unix:///var/run/docker.sock
Jun 27 14:26:09 obrien-mbp.onap.org kubelet[90917]: I0627 14:26:09.698667 90917 docker.go:376] Start docker client with request timeout=2m0s
Jun 27 14:26:09 obrien-mbp.onap.org kubelet[90917]: E0627 14:26:09.702077 90917 cni.go:163] error updating cni config: No networks found in /etc/cni/net.d
Jun 27 14:26:09 obrien-mbp.onap.org kubelet[90917]: I0627 14:26:09.711775 90917 manager.go:143] cAdvisor running in container: "/system.slice/kubelet.service"
Hint: Some lines were ellipsized, use -l to show in full.
[root@obrien-mbp opt]# ss -tulnp | grep -E "(kube)|(etcd)"
tcp LISTEN 0 128 127.0.0.1:2379 *:* users:(("etcd",pid=90567,fd=6))
tcp LISTEN 0 128 127.0.0.1:2380 *:* users:(("etcd",pid=90567,fd=5))
tcp LISTEN 0 128 127.0.0.1:10255 *:* users:(("kubelet",pid=90917,fd=9))
tcp LISTEN 0 128 127.0.0.1:10248 *:* users:(("kubelet",pid=90917,fd=8))
tcp LISTEN 0 128 127.0.0.1:10249 *:* users:(("kube-proxy",pid=90864,fd=9))
tcp LISTEN 0 128 127.0.0.1:10250 *:* users:(("kubelet",pid=90917,fd=14))
tcp LISTEN 0 128 :::10251 :::* users:(("kube-scheduler",pid=91175,fd=8))
tcp LISTEN 0 128 :::6443 :::* users:(("kube-apiserver",pid=91126,fd=42))
tcp LISTEN 0 128 :::8080 :::* users:(("kube-apiserver",pid=91126,fd=43))
tcp LISTEN 0 128 :::4194 :::* users:(("kubelet",pid=90917,fd=13))
[root@obrien-mbp opt]# curl -s -L http://localhost:2379/version
{"etcdserver":"3.1.7","etcdcluster":"3.1.0"}[root@obrien-mbp opt]#
Provision
Manually
Start a service
In this case robot - to check your Kubernetes installation.
[root@obrien-mbp oneclick]# ./createAll.bash -n onap -a robot
********** Creating up ONAP: robot
Creating namespaces **********
namespace "onap-robot" created
Creating services **********
service "robot" created
********** Creating deployments for robot **********
Robot....
deployment "robot" created
**** Done ****[root@obrien-mbp oneclick]# kubectl cluster-info
Kubernetes master is running at http://localhost:8080
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The official documentation for installation of ONAP with OOM / Kubernetes is located in Read the Docs:
- OOM User Guide — onap master documentation
- OOM Quick Start Guide — onap master documentation)
- OOM Cloud Setup Guide — onap master documentation
Kubernetes Rest api
...
{
"paths": [
"/api",
"/api/v1",
"/apis",
"/apis/apps",
"/apis/apps/v1beta1",
"/apis/authentication.k8s.io",
"/apis/authentication.k8s.io/v1beta1",
"/apis/authorization.k8s.io",
"/apis/authorization.k8s.io/v1beta1",
"/apis/autoscaling",
"/apis/autoscaling/v1",
"/apis/batch",
"/apis/batch/v1",
"/apis/batch/v2alpha1",
"/apis/certificates.k8s.io",
"/apis/certificates.k8s.io/v1alpha1",
"/apis/extensions",
"/apis/extensions/v1beta1",
"/apis/policy",
"/apis/policy/v1beta1",
"/apis/rbac.authorization.k8s.io",
"/apis/rbac.authorization.k8s.io/v1alpha1",
"/apis/storage.k8s.io",
"/apis/storage.k8s.io/v1beta1",
"/healthz",
"/healthz/ping",
"/healthz/poststarthook/bootstrap-controller",
"/healthz/poststarthook/extensions/third-party-resources",
"/healthz/poststarthook/rbac/bootstrap-roles",
"/logs",
"/metrics",
"/swaggerapi/",
"/ui/",
"/version"
]
}
Pod List
In verification
[root@obrien-mbp oneclick]# ./createAll.bash -n onap
[root@obrien-mbp oneclick]# kubectl get service --all-namespaces=true
NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes 10.254.0.1 <none> 443/TCP 5h
onap-aai aai-service 10.254.172.161 <nodes> 8443:30233/TCP,8080:30232/TCP 1m
onap-aai hbase None <none> 8020/TCP 1m
onap-aai model-loader-service 10.254.138.128 <nodes> 8443:30229/TCP,8080:30210/TCP 1m
onap-appc dbhost None <none> 3306/TCP 1m
onap-appc dgbuilder 10.254.89.166 <nodes> 3000:30228/TCP 1m
onap-appc sdnctldb01 None <none> 3306/TCP 1m
onap-appc sdnctldb02 None <none> 3306/TCP 1m
onap-appc sdnhost 10.254.68.235 <nodes> 8282:30230/TCP,1830:30231/TCP 1m
onap-message-router dmaap 10.254.149.48 <nodes> 3904:30227/TCP,3905:30226/TCP 1m
onap-message-router global-kafka None <none> 9092/TCP 1m
onap-message-router zookeeper None <none> 2181/TCP 1m
onap-mso mariadb 10.254.254.189 <nodes> 3306:30252/TCP 1m
onap-mso mso 10.254.32.65 <nodes> 8080:30223/TCP,3904:30225/TCP,3905:30224/TCP,9990:30222/TCP,8787:30250/TCP 1m
onap-policy brmsgw 10.254.207.78 <nodes> 9989:30216/TCP 1m
onap-policy drools 10.254.114.125 <nodes> 6969:30217/TCP 1m
onap-policy mariadb None <none> 3306/TCP 1m
onap-policy nexus None <none> 8081/TCP 1m
onap-policy pap 10.254.126.231 <nodes> 8443:30219/TCP,9091:30218/TCP 1m
onap-policy pdp 10.254.244.112 <nodes> 8081:30220/TCP 1m
onap-policy pypdp 10.254.176.200 <nodes> 8480:30221/TCP 1m
onap-portal portalapps 10.254.198.12 <nodes> 8006:30213/TCP,8010:30214/TCP,8989:30215/TCP 1m
onap-portal portaldb None <none> 3306/TCP 1m
onap-portal vnc-portal 10.254.0.129 <nodes> 6080:30211/TCP,5900:30212/TCP 1m
onap-robot robot 10.254.255.97 <nodes> 88:30209/TCP 5h
onap-sdc sdc-be 10.254.109.11 <nodes> 8443:30204/TCP,8080:30205/TCP 1m
onap-sdc sdc-cs None <none> 9042/TCP,9160/TCP 1m
onap-sdc sdc-es None <none> 9200/TCP,9300/TCP 1m
onap-sdc sdc-fe 10.254.219.219 <nodes> 9443:30207/TCP,8181:30206/TCP 1m
onap-sdc sdc-kb None <none> 5601/TCP 1m
onap-sdnc dbhost None <none> 3306/TCP 1m
onap-sdnc sdnc-dgbuilder 10.254.79.74 <nodes> 3000:30203/TCP 1m
onap-sdnc sdnc-portal 10.254.162.50 <nodes> 8843:30201/TCP 1m
onap-sdnc sdnctldb01 None <none> 3306/TCP 1m
onap-sdnc sdnctldb02 None <none> 3306/TCP 1m
onap-sdnc sdnhost 10.254.233.66 <nodes> 8282:30202/TCP 1m
onap-vid vid-mariadb None <none> 3306/TCP 1m
onap-vid vid-server 10.254.83.194 <nodes> 8080:30200/TCP 1m
Troubleshooting
...