...
root@obriensystemsucont0:~/onap/oom/kubernetes/oneclick# kubectl get pods --all-namespaces -o wide
issue may be the fact we need to initialize mounts for pods in the config pod pod-config-init.yaml - rerunning stack
to update on 5820k 4.1GHz 12 vCores 48g Ubuntu 16.04.2 VM on 64g host to update on 5820k 4.1GHz 12 vCores 48g Ubuntu 16.04.2 VM on 64g host root@obriensystemskub0:~/oom/kubernetes/oneclick# kubectl get pods --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE kube-system heapster-859001963-bmlff 1/1 Running 5 43m 10.42.143.118 obriensystemskub0 kube-system kube-dns-1759312207-0x1xx 3/3 Running 8 43m 10.42.246.144 obriensystemskub0 kube-system kubernetes-dashboard-2463885659-jl5jf 1/1 Running 5 43m 10.42.117.156 obriensystemskub0 kube-system monitoring-grafana-1177217109-7gkl6 1/1 Running 4 43m 10.42.79.40 obriensystemskub0 kube-system monitoring-influxdb-1954867534-8nr2q 1/1 Running 5 43m 10.42.146.215 obriensystemskub0 kube-system tiller-deploy-1933461550-w77c5 1/1 Running 4 43m 10.42.1.66 obriensystemskub0 onap-aai aai-service-301900780-wp3w1 1/1 Running 0 25m 10.42.104.101 obriensystemskub0 onap-aai hbase-2985919495-zfs2c 1/1 Running 0 25m 10.42.208.135 obriensystemskub0 onap-aai model-loader-service-2352751609-4qb0x 1/1 Running 0 25m 10.42.25.139 obriensystemskub0 onap-appc appc-4266112350-gscxh 1/1 Running 0 25m 10.42.90.128 obriensystemskub0 onap-appc appc-dbhost-981835105-lp6tn 1/1 Running 0 25m 10.42.201.58 obriensystemskub0 onap-appc appc-dgbuilder-939982213-41znl 1/1 Running 0 25m 10.42.30.127 obriensystemskub0 onap-message-router dmaap-1381770224-c5xp8 1/1 Running 0 25m 10.42.133.232 obriensystemskub0 onap-message-router global-kafka-3488253347-zt8x9 1/1 Running 0 25m 10.42.235.227 obriensystemskub0 onap-message-router zookeeper-3757672320-bxkvs 1/1 Running 0 25m 10.42.14.4 obriensystemskub0 onap-mso mariadb-2610811658-r22z9 1/1 Running 0 25m 10.42.46.110 obriensystemskub0 onap-mso mso-2217182437-1r8fm 1/1 Running 0 25m 10.42.120.204 obriensystemskub0 onap-policy brmsgw-554754608-gssf8 1/1 Running 0 25m 10.42.84.128 obriensystemskub0 onap-policy drools-1184532483-kg8sr 1/1 Running 0 25m 10.42.62.198 obriensystemskub0 onap-policy mariadb-546348828-1ck21 1/1 Running 0 25m 10.42.118.120 obriensystemskub0 onap-policy nexus-2933631225-s1qjz 1/1 Running 0 25m 10.42.73.217 obriensystemskub0 onap-policy pap-235069217-qdf2r 1/1 Running 0 25m 10.42.157.211 obriensystemskub0 onap-policy pdp-819476266-zvncc 1/1 Running 0 25m 10.42.38.47 obriensystemskub0 onap-policy pypdp-3646772508-n801j 1/1 Running 0 25m 10.42.244.206 obriensystemskub0 onap-portal portalapps-157357486-gjnnc 1/1 Running 0 25m 10.42.83.144 obriensystemskub0 onap-portal portaldb-351714684-1n956 1/1 Running 0 25m 10.42.8.80 obriensystemskub0 onap-portal vnc-portal-1027553126-h6dhd 1/1 Running 0 25m 10.42.129.60 obriensystemskub0 onap-robot robot-44708506-t10kk 1/1 Running 0 31m 10.42.185.118 obriensystemskub0 onap-sdc sdc-be-4018435632-3k6k2 1/1 Running 0 25m 10.42.246.193 obriensystemskub0 onap-sdc sdc-cs-2973656688-kktn8 1/1 Running 0 25m 10.42.240.176 obriensystemskub0 onap-sdc sdc-es-2628312921-bg0dg 1/1 Running 0 25m 10.42.67.214 obriensystemskub0 onap-sdc sdc-fe-4051669116-3b9bh 1/1 Running 0 25m 10.42.42.203 obriensystemskub0 onap-sdc sdc-kb-4011398457-fgpkl 1/1 Running 0 25m 10.42.47.218 obriensystemskub0 onap-sdnc sdnc-1672832555-1h4s7 1/1 Running 0 25m 10.42.120.148 obriensystemskub0 onap-sdnc sdnc-dbhost-2119410126-48mt9 1/1 Running 0 25m 10.42.133.166 obriensystemskub0 onap-sdnc sdnc-dgbuilder-730191098-gj6g9 1/1 Running 0 25m 10.42.154.99 obriensystemskub0 onap-sdnc sdnc-portal-3375812606-01s1d 0/1 Running 0 25m 10.42.105.164 obriensystemskub0 onap-vid vid-mariadb-1357170716-vnmhr 1/1 Running 0 28m 10.42.218.225 obriensystemskub0 onap-vid vid-server-248645937-m67r9 1/1 Running 0 28m 10.42.227.81 obriensystemskub0 root@obriensystemskub0:~/oom/kubernetes/oneclick# kubectl get services --all-namespaces -o wide NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR default kubernetes 10.43.0.1 <none> 443/TCP 45m <none> kube-system heapster 10.43.39.217 <none> 80/TCP 45m k8s-app=heapster kube-system kube-dns 10.43.0.10 <none> 53/UDP,53/TCP 45m k8s-app=kube-dns kube-system kubernetes-dashboard 10.43.106.248 <none> 9090/TCP 45m k8s-app=kubernetes-dashboard kube-system monitoring-grafana 10.43.18.184 <none> 80/TCP 45m k8s-app=grafana kube-system monitoring-influxdb 10.43.58.26 <none> 8086/TCP 45m k8s-app=influxdb kube-system tiller-deploy 10.43.235.104 <none> 44134/TCP 45m app=helm,name=tiller onap-aai aai-service 10.43.181.245 <nodes> 8443:30233/TCP,8080:30232/TCP 27m app=aai-service onap-aai hbase None <none> 8020/TCP 27m app=hbase onap-aai model-loader-service 10.43.74.55 <nodes> 8443:30229/TCP,8080:30210/TCP 27m app=model-loader-service onap-appc dbhost None <none> 3306/TCP 27m app=appc-dbhost onap-appc dgbuilder 10.43.146.180 <nodes> 3000:30228/TCP 27m app=appc-dgbuilder onap-appc sdnctldb01 None <none> 3306/TCP 27m app=appc-dbhost onap-appc sdnctldb02 None <none> 3306/TCP 27m app=appc-dbhost onap-appc sdnhost 10.43.196.103 <nodes> 8282:30230/TCP,1830:30231/TCP 27m app=appc onap-message-router dmaap 10.43.95.58 <nodes> 3904:30227/TCP,3905:30226/TCP 27m app=dmaap onap-message-router global-kafka None <none> 9092/TCP 27m app=global-kafka onap-message-router zookeeper None <none> 2181/TCP 27m app=zookeeper onap-mso mariadb 10.43.5.166 <nodes> 3306:30252/TCP 27m app=mariadb onap-mso mso 10.43.156.155 <nodes> 8080:30223/TCP,3904:30225/TCP,3905:30224/TCP,9990:30222/TCP,8787:30250/TCP 27m app=mso onap-policy brmsgw 10.43.81.249 <nodes> 9989:30216/TCP 27m app=brmsgw onap-policy drools 10.43.154.250 <nodes> 6969:30217/TCP 27m app=drools onap-policy mariadb None <none> 3306/TCP 27m app=mariadb onap-policy nexus None <none> 8081/TCP 27m app=nexus onap-policy pap 10.43.37.182 <nodes> 8443:30219/TCP,9091:30218/TCP 27m app=pap onap-policy pdp 10.43.123.239 <nodes> 8081:30220/TCP 27m app=pdp onap-policy pypdp 10.43.226.208 <nodes> 8480:30221/TCP 27m app=pypdp onap-portal portalapps 10.43.225.107 <nodes> 8006:30213/TCP,8010:30214/TCP,8989:30215/TCP 27m app=portalapps onap-portal portaldb None <none> 3306/TCP 27m app=portaldb onap-portal vnc-portal 10.43.216.210 <nodes> 6080:30211/TCP,5900:30212/TCP 27m app=vnc-portal onap-robot robot 10.43.52.131 <nodes> 88:30209/TCP 34m app=robot onap-sdc sdc-be 10.43.123.5 <nodes> 8443:30204/TCP,8080:30205/TCP 27m app=sdc-be onap-sdc sdc-cs None <none> 9042/TCP,9160/TCP 27m app=sdc-cs onap-sdc sdc-es None <none> 9200/TCP,9300/TCP 27m app=sdc-es onap-sdc sdc-fe 10.43.233.17 <nodes> 9443:30207/TCP,8181:30206/TCP 27m app=sdc-fe onap-sdc sdc-kb None <none> 5601/TCP 27m app=sdc-kb onap-sdnc dbhost None <none> 3306/TCP 27m app=sdnc-dbhost onap-sdnc sdnc-dgbuilder 10.43.253.47 <nodes> 3000:30203/TCP 27m app=sdnc-dgbuilder onap-sdnc sdnc-portal 10.43.248.245 <nodes> 8843:30201/TCP 27m app=sdnc-portal onap-sdnc sdnctldb01 None <none> 3306/TCP 27m app=sdnc-dbhost onap-sdnc sdnctldb02 None <none> 3306/TCP 27m app=sdnc-dbhost onap-sdnc sdnhost 10.43.51.170 <nodes> 8282:30202/TCP 27m app=sdnc onap-vid vid-mariadb None <none> 3306/TCP 31m app=vid-mariadb onap-vid vid-server 10.43.54.34 <nodes> 8080:30200/TCP 31m app=vid-server sdnc-portal is still downloading node packages root@obriensystemskub0:~/oom/kubernetes/oneclick# kubectl --namespace onap-sdnc logs -f sdnc-portal-3375812606-01s1d npm http GET https://registry.npmjs.org/minimist/0.0.8 |
|---|
...
NAMESPACE master:20170715 | NAME | READY | STATUS | RESTARTS (in 14h) | Host | Start time | Notes |
|---|---|---|---|---|---|---|---|
| onap-aai | aai-service-346921785-624ss | 1/1 | Running | 0 | 1 | ||
| onap-aai | hbase-139474849-7fg0s | 1/1 | Running | 0 | 2 | ||
| onap-aai | model-loader-service-1795708961-wg19w | 0/1 | Init:1/2 | 82 | 2 | ||
| onap-appc | appc-2044062043-bx6tc | 1/1 | Running | 0 | 1 | ||
| onap-appc | appc-dbhost-2039492951-jslts | 1/1 | Running | 0 | 2 | ||
| onap-appc | appc-dgbuilder-2934720673-mcp7c | 1/1 | Running | 0 | 2 | ||
| onap-dcae | not yet pushed | Note: currently there are no DCAE containers running yet (we are missing 6 yaml files (1 for the controller and 5 for the collector,staging,3-cdap pods)) - therefore DMaaP, VES collectors and APPC actions as the result of policy actions (closed loop) - will not function yet. | |||||
| onap-dcae-cdap | not yet pushed | ||||||
| onap-dcae-stg | not yet pushed | ||||||
| onap-dcae-coll | not yet pushed | ||||||
| onap-message-router | dmaap-3842712241-gtdkp | 0/1 | CrashLoopBackOff | 164 | 1 | ||
| onap-message-router | global-kafka-89365896-5fnq9 | 1/1 | Running | 0 | 2 | ||
| onap-message-router | zookeeper-1406540368-jdscq | 1/1 | Running | 0 | 1 | ||
| onap-mso | mariadb-2638235337-758zr | 1/1 | Running | 0 | 1 | ||
| onap-mso | mso-3192832250-fq6pn | 0/1 | CrashLoopBackOff | 167 | 2 | fixed by config-init and resolv.conf | |
| onap-policy | brmsgw-568914601-d5z71 | 0/1 | Init:0/1 | 82 | 1 | fixed by config-init and resolv.conf | |
| onap-policy | drools-1450928085-099m2 | 0/1 | Init:0/1 | 82 | 1 | 45m | fixed by config-init and resolv.conf |
| onap-policy | mariadb-2932363958-0l05g | 1/1 | Running | 0 | 0 | ||
| onap-policy | nexus-871440171-tqq4z | 0/1 | Running | 0 | 2 | ||
| onap-policy | pap-2218784661-xlj0n | 1/1 | Running | 0 | 1 | ||
| onap-policy | pdp-1677094700-75wpj | 0/1 | Init:0/1 | 82 | 2 | fixed by config-init and resolv.conf | |
| onap-policy | pypdp-3209460526-bwm6b | 0/1 | Init:0/1 | 82 | 2 | fixed by config-init and resolv.conf | |
| onap-portal | portalapps-1708810953-trz47 | 0/1 | Init:CrashLoopBackOff | 163 | 2 | Initial dockerhub mariadb download issue - fixed | |
| onap-portal | portaldb-3652211058-vsg8r | 1/1 | Running | 0 | 0 | ||
| onap-portal | vnc-portal-948446550-76kj7 | 0/1 | Init:0/5 | 82 | 1 | fixed by config-init and resolv.conf | |
| onap-robot | robot-964706867-czr05 | 1/1 | Running | 0 | 2 | ||
| onap-sdc | sdc-be-2426613560-jv8sk | 0/1 | Init:0/2 | 82 | 2 | fixed by config-init and resolv.conf | |
| onap-sdc | sdc-cs-2080334320-95dq8 | 0/1 | CrashLoopBackOff | 163 | 2 | fixed by config-init and resolv.conf | |
| onap-sdc | sdc-es-3272676451-skf7z | 1/1 | Running | 0 | 1 | ||
| onap-sdc | sdc-fe-931927019-nt94t | 0/1 | Init:0/1 | 82 | 1 | fixed by config-init and resolv.conf | |
| onap-sdc | sdc-kb-3337231379-8m8wx | 0/1 | Init:0/1 | 82 | 1 | fixed by config-init and resolv.conf | |
| onap-sdnc | sdnc-1788655913-vvxlj | 1/1 | Running | 0 | 0 | ||
| onap-sdnc | sdnc-dbhost-240465348-kv8vf | 1/1 | Running | 0 | 0 | ||
| onap-sdnc | sdnc-dgbuilder-4164493163-cp6rx | 1/1 | Running | 0 | 0 | ||
| onap-sdnc | sdnc-portal-2324831407-50811 | 0/1 | Running | 3=vm 0=nuc | 1 | root@obriensystemskub0:~/oom/kubernetes/oneclick# kubectl --namespace onap-sdnc logs -f sdnc-portal-3375812606-01s1d | grep ERR npm ERR! fetch failed https://registry.npmjs.org/is-utf8/-/is-utf8-0.2.1.tgz | |
| onap-vid | vid-mariadb-4268497828-81hm0 | 0/1 | CrashLoopBackOff | 169 | 2 | fixed by config-init and resolv.conf | |
| onap-vid | vid-server-2331936551-6gxsp | 0/1 | Init:0/1 | 82 | 1 | fixed by config-init and resolv.conf |
I get the same container issues on 3 different deployments (virtual Ubuntu on 2 separate VMWare based machines, and a 3 node NUC cluster). For example the APPC 3-pod service is running fine.
But the 2-pod VID service has failed startup.
| and resolv.conf |
Run VID if required to verify that config-init has mounted properly (and to verify resolv.conf)Later VID is fixed by running the config-init pod first and also adding to resolv.conf
Cloning details
Install the latest version of the OOM (ONAP Operations Manager) project repo - specifically the ONAP on Kubernetes work just uploaded June 2017
...
Rancher environment for Kubernetes
setup a separate onap kubernetes environment and disable the exising default environment.
Adding hosts to the Kubernetes environment will kick in k8s containers
Rancher kubectl config
To be able to run the kubectl scripts - install kubectl
Nexus3 security settings
Fix nexus3 security for each namespace
...
first verify your kubernetes system is up
Then wait 25-45 min for all pods to attain 1/1 state
Kubernetes specific config
https://kubernetes.io/docs/user-guide/kubectl-cheatsheet/
Dashboard
start the dashboard at http://localhost:8001/ui
...
Nexus Docker repo Credentials
...
Get the mapped external port by checking the service in kubernetes - here 30200 for VID on a particular node in our cluster.
or run a kube
fix /etc/hosts as usual
192.168.163.132 portal.api.simpledemo.openecomp.org 192.168.163.132 sdc.api.simpledemo.openecomp.org 192.168.163.132 policy.api.simpledemo.openecomp.org 192.168.163.132 vid.api.simpledemo.openecomp.org |
|---|
...