About the SDN-C Clustering
The following diagram illustrates the desired SDN-C cluster deployment (as described in
Jira |
---|
server | ONAP JIRA |
---|
columns | key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution |
---|
serverId | 425b2b0a-557c-3c0c-b515-579789cceedb |
---|
key | SDNC-163 |
---|
|
) from POD and SERVICE view, which will have:
- 2 DB pods
- 3 sdnc (ODL) pods
- 1 admin (portal) pod
- 1 dgbuilder pod
Gliffy Diagram |
---|
| |
---|
name | Pod, service, deploy kind |
---|
|
SDN-C DB Clustering details
See details from SDN-C DB (MySQL) Clustered Deployment.
Deployment Setup Steps
Deployment Setup Steps with Rancher
SDN-C Startup Order
Gliffy Diagram |
---|
| |
---|
name | sdnc pods start up order |
---|
|
Troubleshoot SDN-C pods are not start up in order
Usually, this is caused by init-container malfuntion. You can troubleshoot it by the following steps:
Restart Dead Instance (start a new one in its place)
When a pod is dead, Kubernetes automatically starts a new one to replace the dead pod.
Examples:
Expand |
---|
title | Example of restarting dead instance by deleting pod |
---|
|
Expand |
---|
title | sdnc-0 pod runs fine at the beginning |
---|
| ![](http://d2athenaconf:8090/confluence/download/attachments/12356276/image2017-11-23%2018%3A28%3A26.png?version=1&modificationDate=1511479706000&api=v2)
|
Expand |
---|
title | manually delete the sdnc-0 pod |
---|
| ![](http://d2athenaconf:8090/confluence/download/attachments/12356276/image2017-11-24%209%3A35%3A27.png?version=1&modificationDate=1511534128000&api=v2)
|
Expand |
---|
title | once the sdnc-0 pod is fully terminated, a new sdnc-0 pod will be initialized |
---|
| ![](http://d2athenaconf:8090/confluence/download/attachments/12356276/image2017-11-24%209%3A37%3A40.png?version=1&modificationDate=1511534261000&api=v2)
|
Expand |
---|
title | The new sdnc-0 pod will be brought to running status to replace the initial sdnc-0 |
---|
| ![](http://d2athenaconf:8090/confluence/download/attachments/12356276/image2017-11-24%209%3A42%3A13.png?version=1&modificationDate=1511534533000&api=v2)
|
Expand |
---|
title | The new and initial sdnc-0 pod has the same name (although with different internal IP) |
---|
| Description of Initial sdnc-0 | Description of restarted sdnc-0 |
---|
ubuntu@sdnc-k8s:~/oom/kubernetes/config$ kubectl describe pod sdnc-0 -n onap-sdnc Name: sdnc-0 Namespace: onap-sdnc Node: sdnc-k8s-2/10.147.90.6 Start Time: Thu, 23 Nov 2017 22:37:00 +0000 Labels: app=sdnc controller-revision-hash=sdnc-2443484326 Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"onap-sdnc","name":"sdnc","uid":"a576a1a3-d09f-11e7-957f-0269cb13eff1","apiVersio... pod.alpha.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:37:08Z","finishedAt":"2017-11-23T22:37:53Z... pod.alpha.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NA... pod.beta.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:37:08Z","finishedAt":"2017-11-23T22:37:53Z"... pod.beta.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NAM... Status: Running IP: 10.42.107.61 Created By: StatefulSet/sdnc Controlled By: StatefulSet/sdnc ... | ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ kubectl describe pod sdnc-0 -n onap-sdnc Name: sdnc-0 Namespace: onap-sdnc Node: sdnc-k8s-2/10.147.90.6 Start Time: Thu, 23 Nov 2017 22:54:10 +0000 Labels: app=sdnc controller-revision-hash=sdnc-2443484326 Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"onap-sdnc","name":"sdnc","uid":"a576a1a3-d09f-11e7-957f-0269cb13eff1","apiVersio... pod.alpha.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:54:15Z","finishedAt":"2017-11-23T22:54:15Z... pod.alpha.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NA... pod.beta.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:54:15Z","finishedAt":"2017-11-23T22:54:15Z"... pod.beta.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NAM... Status: Running IP: 10.42.185.227 Created By: StatefulSet/sdnc Controlled By: StatefulSet/sdnc ... |
|
|
Expand |
---|
title | Example of restarting dead instance by stopping karaf inside SDNC pod |
---|
|
Expand |
---|
title | Pod sdnc-0 is in Running STATUS with 0 RESTARTS |
---|
| ![](http://d2athenaconf:8090/confluence/download/attachments/12356276/image2017-11-24%2013%3A26%3A26.png?version=1&modificationDate=1511547987000&api=v2)
|
Expand |
---|
title | Enter pod container and stop Karaf |
---|
| ![](http://d2athenaconf:8090/confluence/download/attachments/12356276/image2017-11-24%2013%3A29%3A1.png?version=1&modificationDate=1511548142000&api=v2)
|
Expand |
---|
title | Kubernetes discovers pod process stopped and restarts the container |
---|
| ![](http://d2athenaconf:8090/confluence/download/attachments/12356276/image2017-11-24%2013%3A31%3A19.png?version=1&modificationDate=1511548279000&api=v2)
|
Expand |
---|
title | sdnc-0 pod will be brought up with the restart container and running with 2 out of 2 container READY |
---|
| ![](http://d2athenaconf:8090/confluence/download/attachments/12356276/image2017-11-24%2013%3A34%3A5.png?version=1&modificationDate=1511548446000&api=v2)
|
|
Source Code
The source code is currently temporary shared through gerrit topic SDNC-163 until it is commited into gerrit.