About the SDN-C Clustering

The following diagram illustrates the desired SDN-C cluster deployment (as described in  SDNC-163 - Getting issue details... STATUS ) from POD and SERVICE view, which will have:

  • 2 DB pods
  • 3 sdnc (ODL) pods
  • 1 admin (portal) pod
  • 1 dgbuilder pod

Pod, service, deploy kind

SDN-C DB Clustering details

See details from SDN-C DB (MySQL) Clustered Deployment.

Deployment Setup Steps

You can use Rancher or kubeadm to deploy your Kubernetes cluster:


Additional Information

SDN-C Startup Order

sdnc pods start up order

Troubleshoot SDN-C pods are not start up in order

Usually, this is caused by init-container malfuntion. You can troubleshoot it by the following steps:

  • Do a describe of the pod using command kubectl describe pod <pod-name>, to ensure the Init Container section is presented as your defined in your yaml template.
  • If it does not exist, it is possible that your Kubernetes version supports a different format for the init-containers:

    init-containers syntax supported by Kubernetes 1.5, 1.6 and 1.7 and not supported by Kubernetes 1.8 and greaterinitContainers syntax supported from Kubernetes 1.6 and greater

    Init container is defined under

    spec.template.metadata.annotations."pod.beta.kubernetes.io/init-containers

    An example of init container by beta annotation
    apiVersion: apps/v1beta1
    kind: StatefulSet
    metadata:
      name: sdnc
      ...
    spec:
      ...
      template:
        metadata:
          ...
          annotations:
            pod.beta.kubernetes.io/init-containers: '[
              {
                  "args": [
                      "--container-name",
                      "sdnc-db-container"
                  ],
                  "command": [
                      "/root/ready.py"
                  ],
                  "env": [
                      {
                          "name": "NAMESPACE",
                          "valueFrom": {
                              "fieldRef": {
                                  "apiVersion": "v1",
                                  "fieldPath": "metadata.namespace"
                              }
                          }
                      }
                  ],
                  "image": "{{ .Values.image.readiness }}",
                  "imagePullPolicy": "{{ .Values.pullPolicy }}",
                  "name": "sdnc-readiness"
              }
              ]'
        spec:
          containers:
          ...

    Init container is defined under spec.template.spec.initContainers

    An example of init container by spec.initContainers
    apiVersion: apps/v1beta1
    kind: StatefulSet
    metadata:
      name: sdnc
      ...
    spec:
      ...
      template:
        ...
        spec:
          initContainers:
          - command:
            - /root/ready.py
            - "--container-name"
            - "sdnc-db-container"
            env:
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            image: "{{ .Values.image.readiness }}"
            imagePullPolicy: {{ .Values.pullPolicy }}
            name: sdnc-readiness
          containers:
          ...

Restart Dead Instance (start a new one in its place)

When a pod is dead, Kubernetes automatically starts a new one to replace the dead pod.

Examples:

  • Pod-level restart:

Description of Initial sdnc-0Description of restarted sdnc-0

ubuntu@sdnc-k8s:~/oom/kubernetes/config$ kubectl describe pod sdnc-0 -n onap-sdnc
Name: sdnc-0
Namespace: onap-sdnc
Node: sdnc-k8s-2/10.147.90.6
Start Time: Thu, 23 Nov 2017 22:37:00 +0000
Labels: app=sdnc
controller-revision-hash=sdnc-2443484326
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"onap-sdnc","name":"sdnc","uid":"a576a1a3-d09f-11e7-957f-0269cb13eff1","apiVersio...
pod.alpha.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:37:08Z","finishedAt":"2017-11-23T22:37:53Z...
pod.alpha.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NA...
pod.beta.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:37:08Z","finishedAt":"2017-11-23T22:37:53Z"...
pod.beta.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NAM...
Status: Running
IP: 10.42.107.61
Created By: StatefulSet/sdnc
Controlled By: StatefulSet/sdnc

...

ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ kubectl describe pod sdnc-0 -n onap-sdnc
Name: sdnc-0
Namespace: onap-sdnc
Node: sdnc-k8s-2/10.147.90.6
Start Time: Thu, 23 Nov 2017 22:54:10 +0000
Labels: app=sdnc
controller-revision-hash=sdnc-2443484326
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"onap-sdnc","name":"sdnc","uid":"a576a1a3-d09f-11e7-957f-0269cb13eff1","apiVersio...
pod.alpha.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:54:15Z","finishedAt":"2017-11-23T22:54:15Z...
pod.alpha.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NA...
pod.beta.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:54:15Z","finishedAt":"2017-11-23T22:54:15Z"...
pod.beta.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NAM...
Status: Running
IP: 10.42.185.227
Created By: StatefulSet/sdnc
Controlled By: StatefulSet/sdnc

...

  • Container-level restart:

ubuntu@sdnc-k8s-2:~$ kubectl describe pod sdnc-0 -n onap-sdnc
Name: sdnc-0
Namespace: onap-sdnc
Node: sdnc-k8s-2/10.147.90.6
Start Time: Fri, 24 Nov 2017 16:41:42 +0000
Labels: app=sdnc

...

Containers:
sdnc-controller-container:
Container ID: docker://d3cd42925952d43fa5d6cca359c52d7033fa8bdf5e31881eea424664dde0f634
Image: nexus3.onap.org:10001/onap/sdnc-image:1.2-STAGING-latest
Image ID: docker-pullable://nexus3.onap.org:10001/onap/sdnc-image@sha256:87171a78f8f3090a5f31b63bfead858ec80d220f3ddb4a08e5c98d02cc5edeca
Ports: 8181/TCP, 8101/TCP
Command:
/opt/onap/sdnc/bin/startODL.sh
State: Running
Started: Fri, 24 Nov 2017 17:37:41 +0000
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 24 Nov 2017 16:42:15 +0000
Finished: Fri, 24 Nov 2017 17:37:37 +0000
Ready: True
Restart Count: 1

...

Source Code

The source code is currently temporary shared through gerrit topic SDNC-163 until it is commited into gerrit.


  • No labels