ubuntu@rke:~$ kubectl get pods -n onap -o wide | grep -v Runn | grep -v Completed NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES dev-aaf-aaf-sms-5974b5cc94-ckrgl 0/1 Init:0/1 92 17h 192.170.4.6 k8s-slave-6 dev-aaf-aaf-sms-preload-rg92x 0/1 Init:0/1 103 17h 192.170.0.3 k8s-slave-3 dev-aaf-aaf-sms-vault-0 0/2 CrashLoopBackOff 412 17h 192.170.6.21 k8s-slave-2 dev-aaf-aaf-sshsm-testca-fb8tk 0/1 Init:Error 0 17h 192.170.4.5 k8s-slave-6 dev-consul-consul-9dfb9876c-jtq2m 0/1 CrashLoopBackOff 206 17h 192.170.5.9 k8s-slave-1 dev-consul-consul-server-0 0/1 CrashLoopBackOff 207 17h 192.170.5.8 k8s-slave-1 dev-dcaegen2-dcae-bootstrap-7998db797b-xkhfh 0/1 Init:0/1 0 17h 192.170.5.10 k8s-slave-1 dev-dcaegen2-dcae-config-binding-service-5c7564f46c-6284g 0/2 Init:0/1 68 17h 192.170.2.8 k8s-slave-5 dev-dcaegen2-dcae-deployment-handler-757f587777-c68rm 0/2 Init:0/3 0 17h 192.170.4.11 k8s-slave-6 dev-dcaegen2-dcae-policy-handler-7b4cb7ddc-j5wt5 0/2 Init:0/3 22 17h 192.170.5.13 k8s-slave-1 dev-dcaegen2-dcae-servicechange-handler-cfb88cd89-wmvgq 0/1 Init:0/1 0 17h 192.170.6.12 k8s-slave-2 ubuntu@rke:~$ kubectl describe pod -n onap dev-consul-consul-server-0 Name: dev-consul-consul-server-0 Namespace: onap Priority: 0 PriorityClassName: Node: k8s-slave-1/10.0.0.3 Start Time: Mon, 12 Aug 2019 15:31:12 +0000 Labels: app=consul-server controller-revision-hash=dev-consul-consul-server-7b45dd8569 release=dev-consul statefulset.kubernetes.io/pod-name=dev-consul-consul-server-0 Annotations: cni.projectcalico.org/podIP: 192.170.5.8/32 Status: Running IP: 192.170.5.8 Controlled By: StatefulSet/dev-consul-consul-server Containers: consul-server: Container ID: docker://c039f23624df726ca9341a548574d51f6be9dc66abcf8840df3c632698818e7f Image: docker.io/consul:1.0.6 Image ID: docker-pullable://consul@sha256:6ffe55dcc1000126a6e874b298fe1f1b87f556fb344781af60681932e408ec6a Ports: 8301/TCP, 8500/TCP Host Ports: 0/TCP, 0/TCP Command: /usr/local/bin/docker-entrypoint.sh Args: agent -bootstrap-expect=3 -enable-script-checks -retry-join=dev-consul-consul-server-0.consul-server -retry-join=dev-consul-consul-server-1.consul-server -retry-join=dev-consul-consul-server-2.consul-server -client=0.0.0.0 -server -ui State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Tue, 13 Aug 2019 08:55:23 +0000 Finished: Tue, 13 Aug 2019 08:55:23 +0000 Ready: False Restart Count: 207 Liveness: tcp-socket :8301 delay=10s timeout=1s period=5s #success=1 #failure=3 Readiness: tcp-socket :8301 delay=10s timeout=1s period=5s #success=1 #failure=3 Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-6k9w4 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: default-token-6k9w4: Type: Secret (a volume populated by a Secret) SecretName: default-token-6k9w4 Optional: false QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning BackOff 3m35s (x4958 over 17h) kubelet, k8s-slave-1 Back-off restarting failed container ubuntu@rke:~$ kubectl logs -n onap dev-consul-consul-server-0 ==> No private IPv4 address found ubuntu@rke:~$ ubuntu@rke:~$ ubuntu@rke:~$ --------------------------------------------------------------------------------------------------------------- Another POD Details --------------------------------------------------------------------------------------------------------------- ubuntu@rke:~$ ubuntu@rke:~$ ubuntu@rke:~$ kubectl describe pod -n onap dev-consul-consul-9dfb9876c-jtq2m Name: dev-consul-consul-9dfb9876c-jtq2m Namespace: onap Priority: 0 PriorityClassName: Node: k8s-slave-1/10.0.0.3 Start Time: Mon, 12 Aug 2019 15:31:13 +0000 Labels: app=consul pod-template-hash=9dfb9876c release=dev-consul Annotations: cni.projectcalico.org/podIP: 192.170.5.9/32 Status: Running IP: 192.170.5.9 Controlled By: ReplicaSet/dev-consul-consul-9dfb9876c Containers: consul: Container ID: docker://c18e8b79c7ef790a59ad54f56cd8ad3c3a9dedd740bbd83b4b641966439e15a0 Image: docker.io/oomk8s/consul:1.0.0 Image ID: docker-pullable://oomk8s/consul@sha256:ba60171909c74af5e4204974bf9940676c51b66d8e570547e009d3f172ab1341 Port: Host Port: Command: /bin/sh -c apk update && apk add jq cp /tmp/consul/config/* /consul/config /usr/local/bin/docker-entrypoint.sh agent -client 0.0.0.0 -enable-script-checks -retry-join consul-server State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Tue, 13 Aug 2019 08:56:59 +0000 Finished: Tue, 13 Aug 2019 08:57:03 +0000 Ready: False Restart Count: 206 Environment: SDNC_ODL_COUNT: 1 SDNC_IS_PRIMARY_CLUSTER: true Mounts: /consul/certs from consul-agent-certs-config (rw) /consul/scripts from consul-agent-scripts-config (rw) /tmp/consul/config from consul-agent-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-6k9w4 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: consul-agent-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-consul-consul-configmap Optional: false consul-agent-scripts-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-consul-consul-scripts-configmap Optional: false consul-agent-certs-config: Type: Secret (a volume populated by a Secret) SecretName: dev-consul-consul-certs-secret Optional: false default-token-6k9w4: Type: Secret (a volume populated by a Secret) SecretName: default-token-6k9w4 Optional: false QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulled 20m (x204 over 17h) kubelet, k8s-slave-1 Container image "docker.io/oomk8s/consul:1.0.0" already present on machine Warning BackOff 28s (x4756 over 17h) kubelet, k8s-slave-1 Back-off restarting failed container ubuntu@rke:~$ kubectl logs -n onap dev-consul-consul-9dfb9876c-jtq2m fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz v3.7.3-118-gea5be313d9 [http://dl-cdn.alpinelinux.org/alpine/v3.7/main] v3.7.3-78-g634dbf6dd0 [http://dl-cdn.alpinelinux.org/alpine/v3.7/community] OK: 9061 distinct packages available (1/2) Installing oniguruma (6.6.1-r0) (2/2) Installing jq (1.5-r5) Executing busybox-1.27.2-r7.trigger OK: 7 MiB in 19 packages ==> No private IPv4 address found ubuntu@rke:~$