ubuntu@a-rke:~$ sudo rke up
INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [rke.onap.cloud]
FATA[0000] Unsupported Docker version found [18.06.3-ce], supported versions are [1.11.x 1.12.x 1.13.x 17.03.x]
DI 20190225-2: RKE upgrade from 0.15 to 0.16 - not working
Does rke remove, regenerate the yaml (or hand upgrade the versions) then rke up
Code Block
theme
Midnight
ubuntu@a-rke:~$ sudo rke remove
Are you sure you want to remove Kubernetes cluster [y/n]: y
INFO[0002] Tearing down Kubernetes cluster
INFO[0002] [dialer] Setup tunnel for host [rke.onap.cloud]
INFO[0002] [worker] Tearing down Worker Plane..
INFO[0002] [remove/kubelet] Successfully removed container on host [rke.onap.cloud]
INFO[0003] [remove/kube-proxy] Successfully removed container on host [rke.onap.cloud]
INFO[0003] [remove/service-sidekick] Successfully removed container on host [rke.onap.cloud]
INFO[0003] [worker] Successfully tore down Worker Plane..
INFO[0003] [controlplane] Tearing down the Controller Plane..
INFO[0003] [remove/kube-apiserver] Successfully removed container on host [rke.onap.cloud]
INFO[0003] [remove/kube-controller-manager] Successfully removed container on host [rke.onap.cloud]
INFO[0004] [remove/kube-scheduler] Successfully removed container on host [rke.onap.cloud]
INFO[0004] [controlplane] Host [rke.onap.cloud] is already a worker host, skipping delete kubelet and kubeproxy.
INFO[0004] [controlplane] Successfully tore down Controller Plane..
INFO[0004] [etcd] Tearing down etcd plane..
INFO[0004] [remove/etcd] Successfully removed container on host [rke.onap.cloud]
INFO[0004] [etcd] Successfully tore down etcd plane..
INFO[0004] [hosts] Cleaning up host [rke.onap.cloud]
INFO[0004] [hosts] Cleaning up host [rke.onap.cloud]
INFO[0004] [hosts] Running cleaner container on host [rke.onap.cloud]
INFO[0005] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud]
INFO[0005] [hosts] Removing cleaner container on host [rke.onap.cloud]
INFO[0005] [hosts] Removing dead container logs on host [rke.onap.cloud]
INFO[0006] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud]
INFO[0006] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud]
INFO[0006] [hosts] Successfully cleaned up host [rke.onap.cloud]
INFO[0006] [hosts] Cleaning up host [rke.onap.cloud]
INFO[0006] [hosts] Cleaning up host [rke.onap.cloud]
INFO[0006] [hosts] Running cleaner container on host [rke.onap.cloud]
INFO[0007] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud]
INFO[0008] [hosts] Removing cleaner container on host [rke.onap.cloud]
INFO[0008] [hosts] Removing dead container logs on host [rke.onap.cloud]
INFO[0008] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud]
INFO[0009] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud]
INFO[0009] [hosts] Successfully cleaned up host [rke.onap.cloud]
INFO[0009] [hosts] Cleaning up host [rke.onap.cloud]
INFO[0009] [hosts] Cleaning up host [rke.onap.cloud]
INFO[0009] [hosts] Running cleaner container on host [rke.onap.cloud]
INFO[0010] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud]
INFO[0010] [hosts] Removing cleaner container on host [rke.onap.cloud]
INFO[0010] [hosts] Removing dead container logs on host [rke.onap.cloud]
INFO[0011] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud]
INFO[0011] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud]
INFO[0011] [hosts] Successfully cleaned up host [rke.onap.cloud]
INFO[0011] Removing local admin Kubeconfig: ./kube_config_cluster.yml
INFO[0011] Local admin Kubeconfig removed successfully
INFO[0011] Cluster removed successfully
ubuntu@a-rke:~$ rke config --name cluster.ym
ubuntu@a-rke:~$ sudo rke up
INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [rke.onap.cloud]
INFO[0000] [network] Deploying port listener containers
INFO[0001] [network] Successfully started [rke-etcd-port-listener] container on host [rke.onap.cloud]
INFO[0001] [network] Successfully started [rke-cp-port-listener] container on host [rke.onap.cloud]
INFO[0002] [network] Successfully started [rke-worker-port-listener] container on host [rke.onap.cloud]
INFO[0002] [network] Port listener containers deployed successfully
INFO[0002] [network] Running control plane -> etcd port checks
INFO[0003] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud]
INFO[0003] [network] Running control plane -> worker port checks
INFO[0004] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud]
INFO[0004] [network] Running workers -> control plane port checks
INFO[0005] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud]
INFO[0005] [network] Checking KubeAPI port Control Plane hosts
INFO[0005] [network] Removing port listener containers
INFO[0005] [remove/rke-etcd-port-listener] Successfully removed container on host [rke.onap.cloud]
INFO[0006] [remove/rke-cp-port-listener] Successfully removed container on host [rke.onap.cloud]
INFO[0006] [remove/rke-worker-port-listener] Successfully removed container on host [rke.onap.cloud]
INFO[0006] [network] Port listener containers removed successfully
INFO[0006] [certificates] Attempting to recover certificates from backup on [etcd,controlPlane] hosts
INFO[0007] [certificates] No Certificate backup found on [etcd,controlPlane] hosts
INFO[0007] [certificates] Generating CA kubernetes certificates
INFO[0007] [certificates] Generating Kubernetes API server certficates
INFO[0008] [certificates] Generating Kube Controller certificates
INFO[0008] [certificates] Generating Kube Scheduler certificates
INFO[0008] [certificates] Generating Kube Proxy certificates
INFO[0009] [certificates] Generating Node certificate
INFO[0009] [certificates] Generating admin certificates and kubeconfig
INFO[0009] [certificates] Generating etcd-rke.onap.cloud certificate and key
INFO[0009] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates
INFO[0009] [certificates] Generating Kubernetes API server proxy client certificates
INFO[0010] [certificates] Temporarily saving certs to [etcd,controlPlane] hosts
INFO[0016] [certificates] Saved certs to [etcd,controlPlane] hosts
INFO[0016] [reconcile] Reconciling cluster state
INFO[0016] [reconcile] This is newly generated cluster
INFO[0016] [certificates] Deploying kubernetes certificates to Cluster nodes
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
INFO[0022] [certificates] Successfully deployed kubernetes certificates to Cluster nodes
INFO[0022] Pre-pulling kubernetes images
INFO[0022] Kubernetes images pulled successfully
INFO[0022] [etcd] Building up etcd plane..
INFO[0023] [etcd] Successfully started [etcd] container on host [rke.onap.cloud]
INFO[0023] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [rke.onap.cloud]
INFO[0028] [certificates] Successfully started [rke-bundle-cert] container on host [rke.onap.cloud]
INFO[0029] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [rke.onap.cloud]
INFO[0029] [etcd] Successfully started [rke-log-linker] container on host [rke.onap.cloud]
INFO[0030] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud]
INFO[0030] [etcd] Successfully started etcd plane..
INFO[0030] [controlplane] Building up Controller Plane..
INFO[0031] [controlplane] Successfully started [kube-apiserver] container on host [rke.onap.cloud]
INFO[0031] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [rke.onap.cloud]
INFO[0045] [healthcheck] service [kube-apiserver] on host [rke.onap.cloud] is healthy
INFO[0046] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud]
INFO[0046] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud]
INFO[0047] [controlplane] Successfully started [kube-controller-manager] container on host [rke.onap.cloud]
INFO[0047] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [rke.onap.cloud]
INFO[0052] [healthcheck] service [kube-controller-manager] on host [rke.onap.cloud] is healthy
INFO[0053] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud]
INFO[0053] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud]
INFO[0054] [controlplane] Successfully started [kube-scheduler] container on host [rke.onap.cloud]
INFO[0054] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [rke.onap.cloud]
INFO[0059] [healthcheck] service [kube-scheduler] on host [rke.onap.cloud] is healthy
INFO[0060] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud]
INFO[0060] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud]
INFO[0060] [controlplane] Successfully started Controller Plane..
INFO[0060] [authz] Creating rke-job-deployer ServiceAccount
INFO[0060] [authz] rke-job-deployer ServiceAccount created successfully
INFO[0060] [authz] Creating system:node ClusterRoleBinding
INFO[0060] [authz] system:node ClusterRoleBinding created successfully
INFO[0060] [certificates] Save kubernetes certificates as secrets
INFO[0060] [certificates] Successfully saved certificates as kubernetes secret [k8s-certs]
INFO[0060] [state] Saving cluster state to Kubernetes
INFO[0061] [state] Successfully Saved cluster state to Kubernetes ConfigMap: cluster-state
INFO[0061] [state] Saving cluster state to cluster nodes
INFO[0061] [state] Successfully started [cluster-state-deployer] container on host [rke.onap.cloud]
INFO[0062] [remove/cluster-state-deployer] Successfully removed container on host [rke.onap.cloud]
INFO[0062] [worker] Building up Worker Plane..
INFO[0062] [remove/service-sidekick] Successfully removed container on host [rke.onap.cloud]
INFO[0063] [worker] Successfully started [kubelet] container on host [rke.onap.cloud]
INFO[0063] [healthcheck] Start Healthcheck on service [kubelet] on host [rke.onap.cloud]
INFO[0068] [healthcheck] service [kubelet] on host [rke.onap.cloud] is healthy
INFO[0069] [worker] Successfully started [rke-log-linker] container on host [rke.onap.cloud]
INFO[0070] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud]
INFO[0070] [worker] Successfully started [kube-proxy] container on host [rke.onap.cloud]
INFO[0070] [healthcheck] Start Healthcheck on service [kube-proxy] on host [rke.onap.cloud]
INFO[0076] [healthcheck] service [kube-proxy] on host [rke.onap.cloud] is healthy
INFO[0076] [worker] Successfully started [rke-log-linker] container on host [rke.onap.cloud]
INFO[0077] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud]
INFO[0077] [worker] Successfully started Worker Plane..
INFO[0077] [sync] Syncing nodes Labels and Taints
INFO[0077] [sync] Successfully synced nodes Labels and Taints
INFO[0077] [network] Setting up network plugin: canal
INFO[0077] [addons] Saving addon ConfigMap to Kubernetes
INFO[0077] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-network-plugin
INFO[0077] [addons] Executing deploy job..
INFO[0082] [addons] Setting up KubeDNS
INFO[0082] [addons] Saving addon ConfigMap to Kubernetes
INFO[0082] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-kubedns-addon
INFO[0082] [addons] Executing deploy job..
INFO[0087] [addons] KubeDNS deployed successfully..
INFO[0087] [addons] Setting up Metrics Server
INFO[0087] [addons] Saving addon ConfigMap to Kubernetes
INFO[0087] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-metrics-addon
INFO[0087] [addons] Executing deploy job..
INFO[0092] [addons] KubeDNS deployed successfully..
INFO[0092] [ingress] Setting up nginx ingress controller
INFO[0092] [addons] Saving addon ConfigMap to Kubernetes
INFO[0092] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-ingress-controller
INFO[0092] [addons] Executing deploy job..
INFO[0097] [ingress] ingress controller nginx is successfully deployed
INFO[0097] [addons] Setting up user addons
INFO[0097] [addons] Checking for included user addons
WARN[0097] [addons] Unable to determine if is a file path or url, skipping
INFO[0097] [addons] Deploying rke-user-includes-addons
INFO[0097] [addons] Saving addon ConfigMap to Kubernetes
INFO[0097] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-user-includes-addons
INFO[0097] [addons] Executing deploy job..
WARN[0128] Failed to deploy addon execute job [rke-user-includes-addons]: Failed to get job complete status: <nil>
INFO[0128] Finished building Kubernetes cluster successfully
ubuntu@a-rke:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ec26c4bd24b5 846921f0fe0e "/server" 10 minutes ago Up 10 minutes k8s_default-http-backend_default-http-backend-797c5bc547-45msr_ingress-nginx_0eddfe19-394e-11e9-b708-000d3a0e23f3_0
f8d5db205e14 8a7739f672b4 "/sidecar --v=2 --lo…" 10 minutes ago Up 10 minutes k8s_sidecar_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
490461545ae4 rancher/metrics-server-amd64 "/metrics-server --s…" 10 minutes ago Up 10 minutes k8s_metrics-server_metrics-server-97bc649d5-q84tz_kube-system_0c566ec8-394e-11e9-b708-000d3a0e23f3_0
aaf03b62bd41 6816817d9dce "/dnsmasq-nanny -v=2…" 10 minutes ago Up 10 minutes k8s_dnsmasq_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
58ec007db72f 55ffe31ac578 "/kube-dns --domain=…" 10 minutes ago Up 10 minutes k8s_kubedns_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
0a95c06f6aa6 e183460c484d "/cluster-proportion…" 10 minutes ago Up 10 minutes k8s_autoscaler_kube-dns-autoscaler-5db9bbb766-6slz7_kube-system_08b5495c-394e-11e9-b708-000d3a0e23f3_0
968a7c99b210 rancher/pause-amd64:3.1 "/pause" 10 minutes ago Up 10 minutes k8s_POD_default-http-backend-797c5bc547-45msr_ingress-nginx_0eddfe19-394e-11e9-b708-000d3a0e23f3_0
69969b331e49 rancher/pause-amd64:3.1 "/pause" 10 minutes ago Up 10 minutes k8s_POD_metrics-server-97bc649d5-q84tz_kube-system_0c566ec8-394e-11e9-b708-000d3a0e23f3_0
baa5f03c16ff rancher/pause-amd64:3.1 "/pause" 10 minutes ago Up 10 minutes k8s_POD_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
82b2a9f640cb rancher/pause-amd64:3.1 "/pause" 10 minutes ago Up 10 minutes k8s_POD_kube-dns-autoscaler-5db9bbb766-6slz7_kube-system_08b5495c-394e-11e9-b708-000d3a0e23f3_0
953a4d4be0c1 df4469c42185 "/usr/bin/dumb-init …" 10 minutes ago Up 10 minutes k8s_nginx-ingress-controller_nginx-ingress-controller-dfhp8_ingress-nginx_0ed3bdbf-394e-11e9-b708-000d3a0e23f3_0
cce552840749 rancher/pause-amd64:3.1 "/pause" 10 minutes ago Up 10 minutes k8s_POD_nginx-ingress-controller-dfhp8_ingress-nginx_0ed3bdbf-394e-11e9-b708-000d3a0e23f3_0
baa65f9c6f97 f0fad859c909 "/opt/bin/flanneld -…" 10 minutes ago Up 10 minutes k8s_kube-flannel_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
1736ce68f41a 9f355e076ea7 "/install-cni.sh" 10 minutes ago Up 10 minutes k8s_install-cni_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
615d3f702ee7 7eca10056c8e "start_runit" 10 minutes ago Up 10 minutes k8s_calico-node_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
1c4a702f0f18 rancher/pause-amd64:3.1 "/pause" 10 minutes ago Up 10 minutes k8s_POD_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
0da1cada08e1 rancher/hyperkube:v1.11.6-rancher1 "/opt/rke-tools/entr…" 10 minutes ago Up 10 minutes kube-proxy
57f44998f34a rancher/hyperkube:v1.11.6-rancher1 "/opt/rke-tools/entr…" 11 minutes ago Up 11 minutes kubelet
50f424c4daec rancher/hyperkube:v1.11.6-rancher1 "/opt/rke-tools/entr…" 11 minutes ago Up 11 minutes kube-scheduler
502d327912d9 rancher/hyperkube:v1.11.6-rancher1 "/opt/rke-tools/entr…" 11 minutes ago Up 11 minutes kube-controller-manager
9fc706bbf3a5 rancher/hyperkube:v1.11.6-rancher1 "/opt/rke-tools/entr…" 11 minutes ago Up 11 minutes kube-apiserver
2e7630c2047c rancher/coreos-etcd:v3.2.18 "/usr/local/bin/etcd…" 11 minutes ago Up 11 minutes etcd
fef566337eb6 rancher/rke-tools:v0.1.15 "/opt/rke-tools/rke-…" 26 minutes ago Up 26 minutes etcd-rolling-snapshots
amdocs@obriensystemsu0:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx default-http-backend-797c5bc547-m8hbx 1/1 Running 0 1h
ingress-nginx nginx-ingress-controller-2v7w7 1/1 Running 0 1h
kube-system canal-thmfg 3/3 Running 0 1h
kube-system kube-dns-7588d5b5f5-j66s8 3/3 Running 0 1h
kube-system kube-dns-autoscaler-5db9bbb766-rg5n8 1/1 Running 0 1h
kube-system metrics-server-97bc649d5-jd2rr 1/1 Running 0 1h
kube-system rke-ingress-controller-deploy-job-znp9n 0/1 Completed 0 1h
kube-system rke-kubedns-addon-deploy-job-dzxsj 0/1 Completed 0 1h
kube-system rke-metrics-addon-deploy-job-gpm4j 0/1 Completed 0 1h
kube-system rke-network-plugin-deploy-job-kqdds 0/1 Completed 0 1h
kube-system tiller-deploy-69458576b-khgr5 1/1 Running 0 1h
DI 20190226-1: RKE up segmentation fault on 0.1.16 - use correct user
Code Block
theme
Midnight
amdocs@obriensystemsu0:~$ sudo rke up
Segmentation fault (core dumped)
# issue was I was using ubuntu as the yml user not amdocs in this case for a particular VM
ubuntu@a-rke:~$ sudo rke up
INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [rke.onap.cloud]
FATA[0000] Unsupported Docker version found [18.06.3-ce], supported versions are [1.11.x 1.12.x 1.13.x 17.03.x]
DI 20190225-2: RKE upgrade from 0.15 to 0.16 - not working
Does rke remove, regenerate the yaml (or hand upgrade the versions) then rke up
Code Block
theme
Midnight
ubuntu@a-rke:~$ sudo rke remove
Are you sure you want to remove Kubernetes cluster [y/n]: y
INFO[0002] Tearing down Kubernetes cluster
INFO[0002] [dialer] Setup tunnel for host [rke.onap.cloud]
INFO[0002] [worker] Tearing down Worker Plane..
INFO[0002] [remove/kubelet] Successfully removed container on host [rke.onap.cloud]
INFO[0003] [remove/kube-proxy] Successfully removed container on host [rke.onap.cloud]
INFO[0003] [remove/service-sidekick] Successfully removed container on host [rke.onap.cloud]
INFO[0003] [worker] Successfully tore down Worker Plane..
INFO[0003] [controlplane] Tearing down the Controller Plane..
INFO[0003] [remove/kube-apiserver] Successfully removed container on host [rke.onap.cloud]
INFO[0003] [remove/kube-controller-manager] Successfully removed container on host [rke.onap.cloud]
INFO[0004] [remove/kube-scheduler] Successfully removed container on host [rke.onap.cloud]
INFO[0004] [controlplane] Host [rke.onap.cloud] is already a worker host, skipping delete kubelet and kubeproxy.
INFO[0004] [controlplane] Successfully tore down Controller Plane..
INFO[0004] [etcd] Tearing down etcd plane..
INFO[0004] [remove/etcd] Successfully removed container on host [rke.onap.cloud]
INFO[0004] [etcd] Successfully tore down etcd plane..
INFO[0004] [hosts] Cleaning up host [rke.onap.cloud]
INFO[0004] [hosts] Cleaning up host [rke.onap.cloud]
INFO[0004] [hosts] Running cleaner container on host [rke.onap.cloud]
INFO[0005] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud]
INFO[0005] [hosts] Removing cleaner container on host [rke.onap.cloud]
INFO[0005] [hosts] Removing dead container logs on host [rke.onap.cloud]
INFO[0006] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud]
INFO[0006] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud]
INFO[0006] [hosts] Successfully cleaned up host [rke.onap.cloud]
INFO[0006] [hosts] Cleaning up host [rke.onap.cloud]
INFO[0006] [hosts] Cleaning up host [rke.onap.cloud]
INFO[0006] [hosts] Running cleaner container on host [rke.onap.cloud]
INFO[0007] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud]
INFO[0008] [hosts] Removing cleaner container on host [rke.onap.cloud]
INFO[0008] [hosts] Removing dead container logs on host [rke.onap.cloud]
INFO[0008] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud]
INFO[0009] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud]
INFO[0009] [hosts] Successfully cleaned up host [rke.onap.cloud]
INFO[0009] [hosts] Cleaning up host [rke.onap.cloud]
INFO[0009] [hosts] Cleaning up host [rke.onap.cloud]
INFO[0009] [hosts] Running cleaner container on host [rke.onap.cloud]
INFO[0010] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud]
INFO[0010] [hosts] Removing cleaner container on host [rke.onap.cloud]
INFO[0010] [hosts] Removing dead container logs on host [rke.onap.cloud]
INFO[0011] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud]
INFO[0011] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud]
INFO[0011] [hosts] Successfully cleaned up host [rke.onap.cloud]
INFO[0011] Removing local admin Kubeconfig: ./kube_config_cluster.yml
INFO[0011] Local admin Kubeconfig removed successfully
INFO[0011] Cluster removed successfully
ubuntu@a-rke:~$ rke config --name cluster.ym
ubuntu@a-rke:~$ sudo rke up
INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [rke.onap.cloud]
INFO[0000] [network] Deploying port listener containers
INFO[0001] [network] Successfully started [rke-etcd-port-listener] container on host [rke.onap.cloud]
INFO[0001] [network] Successfully started [rke-cp-port-listener] container on host [rke.onap.cloud]
INFO[0002] [network] Successfully started [rke-worker-port-listener] container on host [rke.onap.cloud]
INFO[0002] [network] Port listener containers deployed successfully
INFO[0002] [network] Running control plane -> etcd port checks
INFO[0003] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud]
INFO[0003] [network] Running control plane -> worker port checks
INFO[0004] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud]
INFO[0004] [network] Running workers -> control plane port checks
INFO[0005] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud]
INFO[0005] [network] Checking KubeAPI port Control Plane hosts
INFO[0005] [network] Removing port listener containers
INFO[0005] [remove/rke-etcd-port-listener] Successfully removed container on host [rke.onap.cloud]
INFO[0006] [remove/rke-cp-port-listener] Successfully removed container on host [rke.onap.cloud]
INFO[0006] [remove/rke-worker-port-listener] Successfully removed container on host [rke.onap.cloud]
INFO[0006] [network] Port listener containers removed successfully
INFO[0006] [certificates] Attempting to recover certificates from backup on [etcd,controlPlane] hosts
INFO[0007] [certificates] No Certificate backup found on [etcd,controlPlane] hosts
INFO[0007] [certificates] Generating CA kubernetes certificates
INFO[0007] [certificates] Generating Kubernetes API server certficates
INFO[0008] [certificates] Generating Kube Controller certificates
INFO[0008] [certificates] Generating Kube Scheduler certificates
INFO[0008] [certificates] Generating Kube Proxy certificates
INFO[0009] [certificates] Generating Node certificate
INFO[0009] [certificates] Generating admin certificates and kubeconfig
INFO[0009] [certificates] Generating etcd-rke.onap.cloud certificate and key
INFO[0009] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates
INFO[0009] [certificates] Generating Kubernetes API server proxy client certificates
INFO[0010] [certificates] Temporarily saving certs to [etcd,controlPlane] hosts
INFO[0016] [certificates] Saved certs to [etcd,controlPlane] hosts
INFO[0016] [reconcile] Reconciling cluster state
INFO[0016] [reconcile] This is newly generated cluster
INFO[0016] [certificates] Deploying kubernetes certificates to Cluster nodes
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
INFO[0022] [certificates] Successfully deployed kubernetes certificates to Cluster nodes
INFO[0022] Pre-pulling kubernetes images
INFO[0022] Kubernetes images pulled successfully
INFO[0022] [etcd] Building up etcd plane..
INFO[0023] [etcd] Successfully started [etcd] container on host [rke.onap.cloud]
INFO[0023] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [rke.onap.cloud]
INFO[0028] [certificates] Successfully started [rke-bundle-cert] container on host [rke.onap.cloud]
INFO[0029] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [rke.onap.cloud]
INFO[0029] [etcd] Successfully started [rke-log-linker] container on host [rke.onap.cloud]
INFO[0030] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud]
INFO[0030] [etcd] Successfully started etcd plane..
INFO[0030] [controlplane] Building up Controller Plane..
INFO[0031] [controlplane] Successfully started [kube-apiserver] container on host [rke.onap.cloud]
INFO[0031] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [rke.onap.cloud]
INFO[0045] [healthcheck] service [kube-apiserver] on host [rke.onap.cloud] is healthy
INFO[0046] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud]
INFO[0046] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud]
INFO[0047] [controlplane] Successfully started [kube-controller-manager] container on host [rke.onap.cloud]
INFO[0047] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [rke.onap.cloud]
INFO[0052] [healthcheck] service [kube-controller-manager] on host [rke.onap.cloud] is healthy
INFO[0053] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud]
INFO[0053] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud]
INFO[0054] [controlplane] Successfully started [kube-scheduler] container on host [rke.onap.cloud]
INFO[0054] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [rke.onap.cloud]
INFO[0059] [healthcheck] service [kube-scheduler] on host [rke.onap.cloud] is healthy
INFO[0060] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud]
INFO[0060] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud]
INFO[0060] [controlplane] Successfully started Controller Plane..
INFO[0060] [authz] Creating rke-job-deployer ServiceAccount
INFO[0060] [authz] rke-job-deployer ServiceAccount created successfully
INFO[0060] [authz] Creating system:node ClusterRoleBinding
INFO[0060] [authz] system:node ClusterRoleBinding created successfully
INFO[0060] [certificates] Save kubernetes certificates as secrets
INFO[0060] [certificates] Successfully saved certificates as kubernetes secret [k8s-certs]
INFO[0060] [state] Saving cluster state to Kubernetes
INFO[0061] [state] Successfully Saved cluster state to Kubernetes ConfigMap: cluster-state
INFO[0061] [state] Saving cluster state to cluster nodes
INFO[0061] [state] Successfully started [cluster-state-deployer] container on host [rke.onap.cloud]
INFO[0062] [remove/cluster-state-deployer] Successfully removed container on host [rke.onap.cloud]
INFO[0062] [worker] Building up Worker Plane..
INFO[0062] [remove/service-sidekick] Successfully removed container on host [rke.onap.cloud]
INFO[0063] [worker] Successfully started [kubelet] container on host [rke.onap.cloud]
INFO[0063] [healthcheck] Start Healthcheck on service [kubelet] on host [rke.onap.cloud]
INFO[0068] [healthcheck] service [kubelet] on host [rke.onap.cloud] is healthy
INFO[0069] [worker] Successfully started [rke-log-linker] container on host [rke.onap.cloud]
INFO[0070] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud]
INFO[0070] [worker] Successfully started [kube-proxy] container on host [rke.onap.cloud]
INFO[0070] [healthcheck] Start Healthcheck on service [kube-proxy] on host [rke.onap.cloud]
INFO[0076] [healthcheck] service [kube-proxy] on host [rke.onap.cloud] is healthy
INFO[0076] [worker] Successfully started [rke-log-linker] container on host [rke.onap.cloud]
INFO[0077] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud]
INFO[0077] [worker] Successfully started Worker Plane..
INFO[0077] [sync] Syncing nodes Labels and Taints
INFO[0077] [sync] Successfully synced nodes Labels and Taints
INFO[0077] [network] Setting up network plugin: canal
INFO[0077] [addons] Saving addon ConfigMap to Kubernetes
INFO[0077] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-network-plugin
INFO[0077] [addons] Executing deploy job..
INFO[0082] [addons] Setting up KubeDNS
INFO[0082][addons]SavingaddonConfigMaptoKubernetes
INFO[0082] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-kubedns-addon
INFO[0082] [addons] Executing deploy job.. 1/1 Running 029m
INFO[0087][addons]KubeDNSdeployedsuccessfully10.42.4.8
INFO[0087][addons] Setting up Metrics Server 18.188.214.137 <none>
INFO[0087] [addons] Saving addon ConfigMap to Kubernetes
INFO[0087] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-metrics-addon
INFO[0087] [addons] Executing deploy job..<none>
onap onap-dmaap-dmaap-bc-745995bf74-m6hhq 0/1
INFO[0092] [addons] KubeDNS deployed successfully.. Init:0/2
INFO[0092][ingress]Settingup2nginxingresscontroller
INFO[0092][addons]SavingaddonConfigMaptoKubernetes29m
INFO[0092][addons]SuccessfullySaved addon to Kubernetes ConfigMap: rke-ingress-controller
INFO[0092] [addons] Executing deploy job.. 10.42.4.12 18.188.214.137 <none><none>INFO[0097] [ingress] ingress controller nginx is successfully deployed
INFO[0097] [addons] Setting up user addonsonap onap-dmaap-dmaap-bc-post-install-6ff4j
INFO[0097][addons]Checkingforincluded1/1useraddons
WARN[0097] [addons] Unable to determine ifRunning isafilepathorurl,skipping
INFO[0097] [addons] Deploying rke-user-includes-addons
INFO[0097] [addons] Saving addon ConfigMap to Kubernetes
INFO[0097] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-user-includes-addons
INFO[0097] [addons] Executing deploy job.. 0 29m 10.42.4.9 18.188.214.137 <none>
WARN[0128] Failed to deploy addon execute job [rke-user-includes-addons]: Failed to get job complete status: <nil>
INFO[0128] Finished building Kubernetes cluster successfully
ubuntu@a-rke:~$ sudo docker ps
CONTAINER ID<none>
onap onap-dmaap-dmaap-dr-db-0 IMAGE1/1Running0COMMAND29m10.42.4.10CREATED18.188.214.137<none>STATUS<none>
onap PORTSonap-dmaap-dmaap-dr-db-1 NAMES
ec26c4bd24b5846921f0fe0e1/1Running"/server"124m 10.42.5.15minutesago18.220.70.253Up10minutes<none><none>
onap k8s_default-http-backend_default-http-backend-797c5bc547-45msr_ingress-nginx_0eddfe19-394e-11e9-b708-000d3a0e23f3_0
f8d5db205e14onap-dmaap-dmaap-dr-node-0 8a7739f672b42/2Running"/sidecar--v=2--lo…"10minutesago0Up10minutes29m10.42.3.113.17.76.33<none>k8s_sidecar_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
490461545ae4<none>
onaprancher/metrics-server-amd64"/metrics-server --s…"onap-dmaap-dmaap-dr-prov-fbf6c94f5-v9bmq 10minutesagoUp10minutes2/2Running0k8s_metrics-server_metrics-server-97bc649d5-q84tz_kube-system_0c566ec8-394e-11e9-b708-000d3a0e23f3_0
aaf03b62bd4129m10.42.5.106816817d9dce18.220.70.253<none><none>
onap"/dnsmasq-nanny-v=2…"10minutes ago onap-dmaap-message-router-0Up10minutes1/1k8s_dnsmasq_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
58ec007db72fRunning55ffe31ac578029m10.42.4.14 "/kube-dns--domain=…"10 minutes ago 18.188.214.137 <none>Up10minutes<none>
onaponap-dmaap-message-router-kafka-0k8s_kubedns_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
0a95c06f6aa6e183460c484d1/1Running1"/cluster-proportion…"10minutesago29mUp 10 minutes.42.5.13 18.220.70.253<none><none>
onapk8s_autoscaler_kube-dns-autoscaler-5db9bbb766-6slz7_kube-system_08b5495c-394e-11e9-b708-000d3a0e23f3_0
968a7c99b210onap-dmaap-message-router-kafka-1 rancher/pause-amd64:3.1"1/pause"1 Running110minutesagoUp1029mminutes10.42.3.133.17.76.33<none>k8s_POD_default-http-backend-797c5bc547-45msr_ingress-nginx_0eddfe19-394e-11e9-b708-000d3a0e23f3_0
69969b331e49 <none>
onaprancher/pause-amd64:3.1onap-dmaap-message-router-kafka-2"/pause"1/110minutesagoRunningUp10minutes029m10.42.4.15k8s_POD_metrics-server-97bc649d5-q84tz_kube-system_0c566ec8-394e-11e9-b708-000d3a0e23f3_0
baa5f03c16ff 18.188.214.137<none>rancher/pause-amd64:3.1<none>
onap"/pause"onap-dmaap-message-router-mirrormaker-8587c4c9cf-lfnd80/1CrashLoopBackOff910minutesagoUp1029mminutes10.42.4.718.188.214.137<none> k8s_POD_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
82b2a9f640cb<none>
onap rancher/pause-amd64:3.1onap-dmaap-message-router-zookeeper-0"/pause"1/110Running minutesagoUp10minutes029m10.42.5.1418.220.70.253 k8s_POD_kube-dns-autoscaler-5db9bbb766-6slz7_kube-system_08b5495c-394e-11e9-b708-000d3a0e23f3_0
953a4d4be0c1<none> df4469c42185<none>
onaponap-dmaap-message-router-zookeeper-1"/usr/bin/dumb-init…"10minutesago1/1Up10minutesRunning029mk8s_nginx-ingress-controller_nginx-ingress-controller-dfhp8_ingress-nginx_0ed3bdbf-394e-11e9-b708-000d3a0e23f3_0
cce55284074910.42.4.13 rancher/pause-amd64:3.118.188.214.137<none>"/pause"<none>
onaponap-dmaap-message-router-zookeeper-210minutesagoUp10minutes1/1Running0k8s_POD_nginx-ingress-controller-dfhp8_ingress-nginx_0ed3bdbf-394e-11e9-b708-000d3a0e23f3_0
baa65f9c6f9729mf0fad859c90910.42.3.123.17.76.33<none>"/opt/bin/flanneld-…"<none>
onap10minutesagoUp10 minutes onap-nfs-provisioner-nfs-provisioner-57c999dc57-mdcw51/1Running0 k8s_kube-flannel_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
1736ce68f41a24m9f355e076ea710.42.3.153.17.76.33<none>"/install-cni.sh"<none>
onap10minutesago Up 10 minutesonap-robot-robot-677bdbb454-zj9jk 1/1 k8s_install-cni_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
615d3f702ee7Running 7eca10056c8e024m10.42.5.16"start_runit"18.220.70.253<none>10minutesago<none>
onapUp10minutesonap-so-so-8569947cbd-jn5x4k8s_calico-node_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
1c4a702f0f18rancher/pause-amd64:3.0/1 Init:0/1"/pause"113m 10.42.4.19minutesago18.188.214.137Up10minutes<none><none>
onap k8s_POD_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
0da1cada08e1 rancher/hyperkube:v1.11.6-rancher1 "/opt/rke-tools/entr…" 10 minutes agoonap-so-so-bpmn-infra-78c8fd665d-b47qn Up10minutes0/1Init:0/11kube-proxy
57f44998f34a13mrancher/hyperkube:v1.11.6-rancher110.42.3.16"/opt/rke-tools/entr…"11 minutes ago 3.17.76.33 Up<none>11minutes<none>
onapkubelet
50f424c4daeconap-so-so-catalog-db-adapter-565f9767ff-lvbgxrancher/hyperkube:v1.11.6-rancher1"/opt/rke-tools/entr…"0/111minutesagoInit:0/1Up11minutes113m10.42.3.173.17.76.33kube-scheduler
502d327912d9<none> rancher/hyperkube:v1.11.6-rancher1"/opt/rke-tools/entr…"11minutesago<none>
onapUp11minutesonap-so-so-mariadb-config-job-d9sdbkube-controller-manager
9fc706bbf3a50/1Init:0/2rancher/hyperkube:v1.11.6-rancher1"/opt/rke-tools/entr…"11minutesago0Up11minutes3m37s10.42.3.203.17.76.33<none>kube-apiserver
2e7630c2047c<none>
onaprancher/coreos-etcd:v3.2.18onap-so-so-mariadb-config-job-rkqdl"/usr/local/bin/etcd…"11minutesagoUp11minutes0/1Init:Error0etcd
fef566337eb613mrancher/rke-tools:v0.1.1510.42.5.1918.220.70.253 "/opt/rke-tools/rke-…"<none> 26minutesago<none>
onapUp26minutesonap-so-so-monitoring-69b9fdd94c-dks4vetcd-rolling-snapshots
amdocs@obriensystemsu0:~$ kubectl get pods --all-namespaces
NAMESPACE 1/1 RunningNAME013m10.42.4.1818.188.214.137<none>READYSTATUS<none>
onapRESTARTSAGE
ingress-nginxdefault-http-backend-797c5bc547-m8hbxonap-so-so-openstack-adapter-5f9cf896d7-mgbdd1/1Running0/1Init:0/11h
ingress-nginx1nginx-ingress-controller-2v7w713m1/1 10.42.4.17Running18.188.214.137<none>01h<none>kube-systemonapcanal-thmfgonap-so-so-request-db-adapter-5c9bfd7b57-2krnp0/1 3/3Init:0/1Running1013m1h
kube-system kube-dns-7588d5b5f5-j66s810.42.3.18 3.17.76.33<none>3/3Running<none>
onap0onap-so-so-sdc-controller-6fb5cf5775-bsxhm1h
kube-systemkube-dns-autoscaler-5db9bbb766-rg5n810/1 Init:0/1Running011h
kube-system13mmetrics-server-97bc649d5-jd2rr10.42.4.20 1/118.188.214.137 <none>Running0<none>
onap1h
kube-system rke-ingress-controller-deploy-job-znp9n 0/1onap-so-so-sdnc-adapter-8555689c75-r6vkb 1/1Completed0Running1h
kube-system0rke-kubedns-addon-deploy-job-dzxsj0/113m10.42.5.18Completed018.220.70.253<none>1h
kube-systemrke-metrics-addon-deploy-job-gpm4j<none>
onap0/1 Completedonap-so-so-vfc-adapter-68fccc8bb8-c56t2 01h
kube-system rke-network-plugin-deploy-job-kqdds0/1 Init:0/1 Completed011h
kube-system13mtiller-deploy-69458576b-khgr510.42.4.2118.188.214.137<none>1/1Running<none>
onap01h
DI 20190226-1: RKE up segmentation fault on 0.1.16 - use correct user
Code Block
theme
Midnight
amdocs@obriensystemsu0:~$ sudo rke up
Segmentation fault (core dumped)
# issue was I was using ubuntu as the yml user not amdocs in this case for a particular VM
DI 20190227-1: Verify no 110 pod limit per VM
DI 20190228-1: deploy casablanca MR to RKE under K8S 1.11.6, Docker 18.06, Helm 2.12.3
...
theme
Midnight
...
onap-so-so-vnfm-adapter-65c4c5944b-72nlf 1/1 Running 0 13m 10.42.3.19 3.17.76.33 <none> <none>
# on worker nodes only
# nfs client
DI 20190507: ARM support using RKE 0.2.1 ARM friendly install