Page History

...

curl -u admin:Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U -H 'Accept: application/json' -X POST 'http://${kubernetes-master-api

#

Purpose

Command and Examples

0

Set the OOM Kubernetes config environment

(If you have set the OOM Kubernetes config enviorment in the same terminal, you can skip this step)

cd {$OOM}/kubernetes/oneclick

source setenv.bash

1

Run the createAll script to deploy the SDN-C appilication

cd {$OOM}/kubernetes/oneclick

./createAll.bash -n onap -a sdnc

Expand

title	Example of output

********** Creating instance 1 of ONAP with port range 30200 and 30399

********** Creating ONAP:

********** Creating deployments for sdnc **********

Creating namespace **********
namespace "onap-sdnc" created

Creating service account **********
clusterrolebinding "onap-sdnc-admin-binding" created

Creating registry secret **********
secret "onap-docker-registry-key" created

Creating deployments and services **********
NAME: onap-sdnc
LAST DEPLOYED: Thu Nov 23 20:13:32 2017
NAMESPACE: onap
STATUS: DEPLOYED

RESOURCES:
==> v1/PersistentVolume
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
onap-sdnc-db 2Gi RWX Retain Bound onap-sdnc/sdnc-db 1s

==> v1/PersistentVolumeClaim
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
sdnc-db Bound onap-sdnc-db 2Gi RWX 1s

==> v1/Service
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dbhost None <none> 3306/TCP 1s
sdnctldb01 None <none> 3306/TCP 1s
sdnctldb02 None <none> 3306/TCP 1s
sdnc-dgbuilder 10.43.97.219 <nodes> 3000:30203/TCP 1s
sdnhost 10.43.99.163 <nodes> 8282:30202/TCP,8201:30208/TCP 1s
sdnc-portal 10.43.72.72 <nodes> 8843:30201/TCP 1s

==> extensions/v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
sdnc-dgbuilder 1 1 1 0 1s

==> apps/v1beta1/StatefulSet
NAME DESIRED CURRENT AGE
sdnc-dbhost 2 1 1s
sdnc 3 3 1s
sdnc-portal 2 2 1s

**** Done ****

Ensure that the SDN-C appication has started

Use the kubectl get pods command to monitor the SDN-C startup; you should observe:

sdnc-dbhost-0 pod starts and gets into Running STATUS first,
- while
  - sdnc-dbhost-1 pod does not exist and
  - sdnc, sdnc-dgbuilder and sdnc-portal pods are staying in Init:0/1 STATUS
once sdnc-dbhost-0 pod is fully started with the READY "1/1",
- sdnc-dbhost-1 will be starting from ContainerCreating STATUS and runs up to Running STATUS
once sdnc-dbhost-1 pod is in RunningSTATUS,
- sdnc pods will be starting from PodInitializing STATUS and end up with Running STATUS in parallel
- while
  - sdnc-dgbuilder and sdnc-portal pods are staying in Init:0/1 STATUS
once sdnc pods are (is) in Running STATUS,
- sdnc-dgbuilder and sdnc-portal will be starting from PodInitializing STATUS and end up with Running STATUS in parllel

Expand

title	Example of start up status changes

2

Validate that all SDN-C pods and services are created properly

helm ls --all

Expand

title	Example of SDNC release

ubuntu@sdnc-k8s:~$ helm ls --all
NAME REVISION UPDATED STATUS CHART NAMESPACE
onap-config 1 Tue Nov 21 17:07:13 2017 DEPLOYED config-1.1.0 onap
onap-sdnc 1 Thu Nov 23 20:13:32 2017 DEPLOYED sdnc-0.1.0 onap
ubuntu@sdnc-k8s:~$

kubectl get namespaces

Expand

title	Example of SDNC namespace

ubuntu@sdnc-k8s:~$ kubectl get namespaces
NAME STATUS AGE
default Active 15d
kube-public Active 15d
kube-system Active 15d
onap Active 2d
onap-sdnc Active 12m
ubuntu@sdnc-k8s:~$

kubectl get deployment --all-namespaces

Expand

title	Example of SDNC deployment

ubuntu@sdnc-k8s-2:~$ kubectl get deployment --all-namespaces
NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kube-system heapster 1 1 1 1 15d
kube-system kube-dns 1 1 1 1 15d
kube-system kubernetes-dashboard 1 1 1 1 15d
kube-system monitoring-grafana 1 1 1 1 15d
kube-system monitoring-influxdb 1 1 1 1 15d
kube-system tiller-deploy 1 1 1 1 15d
onap-sdnc sdnc-dgbuilder 1 1 1 0 26m
ubuntu@sdnc-k8s-2:~$

kubectl get clusterrolebinding --all-namespaces

Expand

title	Example of SDNC cluster role binding

ubuntu@sdnc-k8s:~$ kubectl get clusterrolebinding --all-namespaces
NAMESPACE NAME AGE
addons-binding 15d
onap-sdnc-admin-binding 13m
ubuntu@sdnc-k8s:~$

kubectl get serviceaccounts --all-namespaces

Expand

title	Example of SDNC service account

ubuntu@sdnc-k8s:~$ kubectl get serviceaccounts --all-namespaces
NAMESPACE NAME SECRETS AGE
default default 1 15d
kube-public default 1 15d
kube-system default 1 15d
kube-system io-rancher-system 1 15d
onap default 1 2d
onap-sdnc default 1 14m
ubuntu@sdnc-k8s:~$

kubectl get service --all-namespaces

Expand

title	Example of all SDNC services

ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ kubectl get service --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 15d
kube-system heapster ClusterIP 10.43.210.11 <none> 80/TCP 15d
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP 15d
kube-system kubernetes-dashboard ClusterIP 10.43.196.205 <none> 9090/TCP 15d
kube-system monitoring-grafana ClusterIP 10.43.90.8 <none> 80/TCP 15d
kube-system monitoring-influxdb ClusterIP 10.43.52.1 <none> 8086/TCP 15d
kube-system tiller-deploy ClusterIP 10.43.106.73 <none> 44134/TCP 15d
onap-sdnc dbhost ClusterIP None <none> 3306/TCP 17m
onap-sdnc sdnc-dgbuilder NodePort 10.43.97.219 <none> 3000:30203/TCP 17m
onap-sdnc sdnc-portal NodePort 10.43.72.72 <none> 8843:30201/TCP 17m
onap-sdnc sdnctldb01 ClusterIP None <none> 3306/TCP 17m
onap-sdnc sdnctldb02 ClusterIP None <none> 3306/TCP 17m
onap-sdnc sdnhost NodePort 10.43.99.163 <none> 8282:30202/TCP,8201:30208/TCP 17m
ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$

kubectl get pods --all-namespaces -a

Expand

title	Example of all SDNC pods

ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ kubectl get pods --all-namespaces -a
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system heapster-4285517626-32km8 1/1 Running 0 15d
kube-system kube-dns-638003847-vqz8t 3/3 Running 0 15d
kube-system kubernetes-dashboard-716739405-tnxj6 1/1 Running 0 15d
kube-system monitoring-grafana-2360823841-qfhzm 1/1 Running 0 15d
kube-system monitoring-influxdb-2323019309-41q0l 1/1 Running 0 15d
kube-system tiller-deploy-737598192-5663c 1/1 Running 0 15d
onap config 0/1 Completed 0 2d
onap-sdnc sdnc-0 2/2 Running 0 17m
onap-sdnc sdnc-1 0/2 CrashLoopBackOff 16 17m
onap-sdnc sdnc-2 2/2 Running 0 17m
onap-sdnc sdnc-dbhost-0 1/1 Running 0 17m
onap-sdnc sdnc-dbhost-1 1/1 Running 0 16m
onap-sdnc sdnc-dgbuilder-356329770-cpfzj 0/1 Running 6 17m
onap-sdnc sdnc-portal-0 0/1 Running 6 17m
onap-sdnc sdnc-portal-1 0/1 CrashLoopBackOff 7 17m
ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$

docker ps |grep sdnc

Expand

title	Example of SDNC docker container

$ docker ps |grep sdnc |wc -l
14
$ docker ps |grep sdnc

9a1fc91b6dcc docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942 "filebeat -e" 19 minutes ago Up 19 minutes k8s_filebeat-onap_sdnc-2_onap-sdnc_c7feb9d1-d08a-11e7-957f-0269cb13eff1_0
9dbaa04e160c docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942 "filebeat -e" 19 minutes ago Up 19 minutes k8s_filebeat-onap_sdnc-0_onap-sdnc_c7f28480-d08a-11e7-957f-0269cb13eff1_0
fca36e9b5353 nexus3.onap.org:10001/onap/sdnc-image@sha256:1049151464b3e60d9a553bc2f3bdaf79555839217f0557652e982ca99398375a "/opt/onap/sdnc/bin/s" 19 minutes ago Up 19 minutes k8s_sdnc-controller-container_sdnc-2_onap-sdnc_c7feb9d1-d08a-11e7-957f-0269cb13eff1_0
00efa164a58a nexus3.onap.org:10001/onap/sdnc-image@sha256:1049151464b3e60d9a553bc2f3bdaf79555839217f0557652e982ca99398375a "/opt/onap/sdnc/bin/s" 19 minutes ago Up 19 minutes k8s_sdnc-controller-container_sdnc-0_onap-sdnc_c7f28480-d08a-11e7-957f-0269cb13eff1_0
4a2769dfee37 mysql/mysql-server@sha256:720f301388709af2c84ee09ba51340d09d1e9f7ba45f19719b5b18b5fa696771 "/entrypoint.sh mysql" 19 minutes ago Up 19 minutes (healthy) k8s_sdnc-db-container_sdnc-dbhost-0_onap-sdnc_c7e050b3-d08a-11e7-957f-0269cb13eff1_0

8b4a21cb2bd2 mysql/mysql-server@sha256:720f301388709af2c84ee09ba51340d09d1e9f7ba45f19719b5b18b5fa696771 "/entrypoint.sh mysql" 19 minutes ago Up 19 minutes (healthy) k8s_sdnc-db-container_sdnc-dbhost-1_onap-sdnc_cde0fde0-d08a-11e7-957f-0269cb13eff1_0

04904cb18336 gcr.io/google_containers/pause-amd64:3.0 "/pause" 19 minutes ago Up 19 minutes k8s_POD_sdnc-portal-0_onap-sdnc_c810f3e4-d08a-11e7-957f-0269cb13eff1_0
e89a3e28505a gcr.io/google_containers/pause-amd64:3.0 "/pause" 19 minutes ago Up 19 minutes k8s_POD_sdnc-2_onap-sdnc_c7feb9d1-d08a-11e7-957f-0269cb13eff1_0
7d2c4ac066f4 gcr.io/google_containers/pause-amd64:3.0 "/pause" 19 minutes ago Up 19 minutes k8s_POD_sdnc-dgbuilder-356329770-cpfzj_onap-sdnc_c7dd4b22-d08a-11e7-957f-0269cb13eff1_0
660ee9119001 gcr.io/google_containers/pause-amd64:3.0 "/pause" 19 minutes ago Up 19 minutes k8s_POD_sdnc-0_onap-sdnc_c7f28480-d08a-11e7-957f-0269cb13eff1_0
fa0e59e1a7c7 gcr.io/google_containers/pause-amd64:3.0 "/pause" 19 minutes ago Up 19 minutes k8s_POD_sdnc-dbhost-0_onap-sdnc_c7e050b3-d08a-11e7-957f-0269cb13eff1_0

24b6d3eb3020 gcr.io/google_containers/pause-amd64:3.0 "/pause" 19 minutes ago Up 19 minutes k8s_POD_sdnc-dbhost-1_onap-sdnc_cde0fde0-d08a-11e7-957f-0269cb13eff1_0
a6e6445b87eb gcr.io/google_containers/pause-amd64:3.0 "/pause" 19 minutes ago Up 19 minutes k8s_POD_sdnc-1_onap-sdnc_c7f6b06f-d08a-11e7-957f-0269cb13eff1_0
1d00b4bb46c0 gcr.io/google_containers/pause-amd64:3.0 "/pause" 19 minutes ago Up 19 minutes k8s_POD_sdnc-portal-1_onap-sdnc_c813c667-d08a-11e7-957f-0269cb13eff1_0

$

In the above example, it missed sdnc-1 container as I had a failure of it due to directory sharing.

3

Validate that the SDN-C bundlers are up

Expand

title	Enter pod container with 2 options

Expand

title	Option 1: through pod name from anywhere

Use command

kubectl exec -it <POD_NAME_WITH_NAME_SPACE> bash

Expand

title	Example

Expand

title	Option 2: through docker container ID from where the container is

Use command

docker exec -it <DOCKER_CONTAINER_ID> bash

Expand

title	Example

Expand

title	Check SDNC bundles in ODL client

Expand

title	Enter ODL client

Expand

title	Check SDNC bundlers

4

Validate that the SDN-C APIs are shown on the ODL RestConf page

Access the ODL RestConf page from the following URL:

http://<Kubernetes-Master-Node-IP>:30202/apidoc/explorer/index.html

Expand

title	Example of SDNC APIs in ODL RestConf page

5

Validate the SDN-C ODL cluster

Goal:

Verify if the SDN-C ODL-Cluster is running properly

Prerequisites

This test is on one of your Kubernetes nodes
Make sure python-pycurl is installed
- If not, for Ubuntu use "apt-get install python-pycurl" to install it

Use ODL intergration tool to monitor ODL cluster

Clone ODL Integration-test project

git clone https://github.com/opendaylight/integration-test.git

Enter cluster-montor folder

cd integration-test/tools/clustering/cluster-monitor

Create cluster-monitor.bash script

vi cluster-monitor.bash

Code Block

language	bash
title	Content of cluster-monitor.bash
linenumbers	true
collapse	true

########################################################################################
# This script wraps ODL monitor.py and dynamicly picks up ODL clustered IP of the sdnc #
# clustered pods and update the IP address in the cluster.json which feed the ODL      #
# monitor.py.                                                                          #
# This script also changed the username and password in the cluster.json file.         #
#                                                                                      #
# If the sdnc pods IP address is re-assigned, the running session of this script       #
# should be restarted.                                                                 #
#                                                                                      #
# To run it, just enter the following command:                                         #
#    ./cluster-monitor.bash                                                              #
########################################################################################
#!/bin/bash

# get IPs string by using kubectl
ips_string=$(kubectl get pods --all-namespaces -o wide | grep 'sdnc-[0-9]' | awk '{print $7}')
ip_list=($(echo ${ips_string} | tr ' ' '\n'))

# loop and replace existing IP
for ((i=0;i<=2;i++));
do
   if [ "${ip_list[$i]}" == "<none>" ]; then
     echo "Ip of deleted pod is not ready yet"
     exit 1;
   fi

   let "j=$i+4"
   sed -i -r "${j}s/(\b[0-9]{1,3}\.){3}[0-9]{1,3}\b"/${ip_list[$i]}/ cluster.json
done

# replace port, username and password
sed -i  's/8181/8080/g' cluster.json
sed -i 's/username/admin/' cluster.json
sed -i 's/password/admin/' cluster.json

# start monitoring
python monitor.py

This script is used to fetch all the IPs of SDN-C pods and automatically update cluster.json file

Start cluster montor UI

./cluster-monitor.bash

Note:

If applications inside any of these three SDNC pods are not fully started, this script won't be executed successfully due to the issues such as connection error, value error and etc.

Otherwise, you should see the monitoring UI as the following:

Use testCluster RPC to test SDN-C load sharing

The testCluster-bundle.zip provides a testBundle which offers a testCluster API to help with validating SDN-C RPC load sharing in the deployed SDN-C cluster.

It's just as easy as to do the following:

download testCluster-bundle.zip

from the attachmenttestCluster api can be accessed from ODL RestConf page, postman or curl command:

(by clicking on the hyplinked text), and place it to the sdnc-deploy hostPath which has been defined in .spec.volumes of sdnc-statefulset.yaml file

As this hostPath is mounted as ODL's deploy directory, testBundle will be automatically loaded by ODL and the testCluster API will be availble to use.

unzip testCluster-bundle.zip, If unzip command is not installed, install it by using "sudo apt install unzip" and then "unzip testCluster-bundle.zip".
Expand
title

curl command

Here's an example

ubuntu@sdnc-k8s-1:~/cluster/deploy$ unzip testCluster-0.2.0.zip

The program 'unzip' is currently not installed. You can install it by typing:
sudo apt install unzip
ubuntu@sdnc-k8s-1:~/cluster/deploy$ sudo apt install unzip
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
zip
The following NEW packages will be installed:
unzip
0 upgraded, 1 newly installed, 0 to remove and 117 not upgraded.
Need to get 158 kB of archives.
After this operation, 530 kB of additional disk space will be used.
Get:1 http://nova.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 unzip amd64 6.0-20ubuntu1 [158 kB]
Fetched 158 kB in 0s (230 kB/s)
Selecting previously unselected package unzip.
(Reading database ... 93492 files and directories currently installed.)
Preparing to unpack .../unzip_6.0-20ubuntu1_amd64.deb ...
Unpacking unzip (6.0-20ubuntu1) ...
Processing triggers for mime-support (3.59ubuntu1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up unzip (6.0-20ubuntu1) ...
ubuntu@sdnc-k8s-1:~/cluster/deploy$ unzip testCluster-0.2.0.zip
Archive: testCluster-0.2.0.zip
inflating: testDataBroker-api-0.2.0-SNAPSHOT.jar
inflating: testDataBroker-features-0.2.0-SNAPSHOT.jar
inflating: testDataBroker-impl-0.2.0-SNAPSHOT.jar
ubuntu@sdnc-k8s-1:~/cluster/deploy$

As this hostPath is mounted as ODL's deploy directory, once the zip file is unzipped, the testBundle will be automatically loaded by ODL and the testCluster API will be availble as ODL RestConf API.

testCluster API is accessible from

Expand
title ODL RestConf API Documentation
Image Added

Expand

title	Postman

Image Added

Code Block

language	actionscript3
title	Example of postman code snippets
linenumbers	true
collapse	true

POST /restconf/operations/testCluster:who-am-i HTTP/1.1
Host: ${KUBERNETES MASTER VM IP}:30202
Accept: application/json
Authorization: Basic YWRtaW46S3A4Yko0U1hzek0wV1hsaGFrM2VIbGNzZTJnQXc4NHZhb0dHbUp2VXkyVQ==
Cache-Control: no-cache
Postman-Token: 9683538b-de47-dec8-3e88-c491be9dd6ef

Expand

title	Curl command

curl -u admin:Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U -H 'Accept: application/json' -X POST 'http://${KUBERNETES MASTER VM IP}:30202/restconf/operations/testCluster:who-am-i'

- Expand
  title An example of testCluster API response
  { "output": { "node": "sdnc-2" } }

Space shortcuts

Page tree

Versions Compared

Old Version 18

New Version 19

Key

Set the OOM Kubernetes config environment

Run the createAll script to deploy the SDN-C appilication

Validate that all SDN-C pods and services are created properly

Validate that the SDN-C bundlers are up

Validate that the SDN-C APIs are shown on the ODL RestConf page

Validate the SDN-C ODL cluster

Goal:

Prerequisites

Use ODL intergration tool to monitor ODL cluster

Use testCluster RPC to test SDN-C load sharing

Undeploy the SDN-C Application

Set the OOM Kubernetes config environment

Run the deleteAll script to delete all SDN-C pods and services

Validate that all SDN-C pods and servers are cleaned up

Remove the ONAP Config

Set the OOM Kubernetes config environment

Remove the ONAP config and any deployed applications in one shot

Manually clean up

Delete the shared folder

Scripts