Info |
---|
Steps described in this page are run by "ubuntu", a non-root user. |
Table of Contents | ||
---|---|---|
|
...
Clone the OOM project from ONAP gerrit
Run the following command on master node to clone OOM project from ONAP gerrit at any directory you prefer, and that directory will be referred as "{$OOM}" in this page.
git clone http://gerrit.onap.org/r/oom
SDN-C Cluster Deployment
Configure
...
SDN-C Cluster Deployment
We are using Kubernetes replicas to achieve the SDNC SDN-C cluster deployment (see details from 4. Deploy/Undeploy the About SDN-C ClusterClustering for the desired goal).
This only needs to be done one time and, at the moment, all modifications are done manually (they can be automated via scripting in the future when the needs come up).
Edit SDNC Templates
The following is the list of SDNC deployment templates which need to be modified for an SDN-C cluster deployment:
...
db-deployment.yaml
...
Field | New value | Old value | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
.apiVersion | apps/v1beta1 1 | extensions/v1beta1 | ||||||||||||||
.kind | StatefulSet | Deployment | ||||||||||||||
.spec.serviceName | "dbhost" 2 | N/A | ||||||||||||||
.spec.replicas | 2 | N/A | ||||||||||||||
.spec.initContainers |
| N/A | ||||||||||||||
.spec.containers |
|
| ||||||||||||||
.spec.volumes |
|
|
...
sdnc-deployment.yaml
...
Field | New value | Old value |
---|---|---|
.apiVersion | apps/v1beta1 1 | extensions/v1beta1 |
.kind | StatefulSet | Deployment |
.spec.serviceName | "sdnhostcluster" 2 | N/A |
.spec.replicas | 3 | N/A |
.spec.podManagementPolicy | "Parallel" 3 | N/A |
.spec.containers.ports 4 | - containerPort: 2550 | N/A |
.spec.containers.volumeMounts 5.a | - mountPath: /opt/onap/sdnc/bin/startODL.sh | N/A |
.spec.volumes 5.b | - name: sdnc-deploy | N/A |
Get New startODL.sh Script From Gerrit Topic SDNC-163
Info |
---|
The source of the new startODL.sh script, gerrit change 25475, has been merged into sdnc/oam project on December 15th, 2017. |
Do the following to get the new startODL.sh script which provides the configuration of ODL clustering for SDN-C cluster.
# | Purpose | Command Examples |
---|---|---|
1 | Get the shared new startODL.sh script content | Go to gerrit change 25475 click on installation/sdnc/src/main/scripts/startODL.sh under the Files section to view the text of the script. click on the Download button () to download the startODL_new.sh.zip file and extract open the sh file inside the zip file, rename it to "startODL.sh". |
2 | Create new startODL.sh on the Kubernetes node VM | mkdir -p /dockerdata-nfs/cluster/script vi /dockerdata-nfs/cluster/script/startODL.sh paste the copied content from step 1 to this file |
3 | Give execution permission to the new startODL.sh script | chmod 777 /dockerdata-nfs/cluster/script/startODL.sh
|
Get SDN-C Cluster Templates From Gerrit Topic SDNC-163
Info |
---|
The source of the templates, gerrit change 25467, has been merged into sdnc/oam project on December 20th, 2017. Skip step 1 and 2 if your cloned OOM project includes this change. Skip step 3 if you skipped previous section (adding startODL.sh script). Skip step 4, if you dont want to add/deploy extra features/bundles/packages. Step 5 in important. It determines number of sdnc and db pods. |
# | Purpose | Command and Examples | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Get the shared templates code git fetch command | Go to gerrit change 25467 Click Download downward arrow, From the right bottom corner drop list, select anonymous http, Click the click board in the same line as Checkout to get (copy to clipboard) the git commands (which includes the git fetch and checkout commands).
| ||||||||||||
2 | Fetch the shared template to the oom directory on the Kubernetes node VM | cd {$OOM} Execute the git command from step 1. | ||||||||||||
3 | Link the new startODL.sh |
vi kubernetes/sdnc/templates/sdnc-statefulset.yaml Make the following changes:
| ||||||||||||
4 | Link the ODL deploy directory |
ODL automatically install bundles/pacakges that are put under deploy directory, this mount point provides capability to drop a bundle/package in the Kubernetes node at /dockerdata-nfs/cluster/deploy directory and it will automativally be installed in the sdnc pods (under opt/opendaylight/current/deploy directory). vi kubernetes/sdnc/templates/sdnc-statefulset.yaml Make the following changes:
| ||||||||||||
5 | Enable cluster configuration | vi kubernetes/sdnc/values.yaml Change the following fields with the new value:
|
Make nfs-provisioner Pod Runs On Node Where NFS Server Runs
Info |
---|
Skip the following section, if any of thew following coditions matches.
|
Verify (from Master node)
# | Purpose | Command and Example | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Find the node name |
kubectl get node
| ||||||||||
2 | Set label on the node | kubectl label nodes <NODE_NAME_FROM_LAST_STEP> disktype=ssd
| ||||||||||
3 | Check the label has been set on the node | kubectl get node --show-labels
| ||||||||||
4 | Update nfs-provisioner pod template to force it running on the nfs server node | In nfs-provisoner-deployment.yaml file, add “spec.template.spec.nodeSelector” for pod “nfs-provisioner”
|
Create the ONAP Config
Setup onap-parameters.yaml file
The following commands must run on master node before creating ONAP configuration.
cd {$OOM}/kubernetes/config
cp onap-parameters-sample.yaml onap-parameters.yaml
Run createConfig
Info | ||
---|---|---|
| ||
You can skip the steps in this section by following instruction in autoCreateOnapConfig of Scripts section to
|
# | Purpose | Command and Examples | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | Set the OOM Kubernetes config environment | cd {$OOM}/kubernetes/oneclick source setenv.bash | ||||||||||||||||||
1 | Run the createConfig script to create the ONAP config | cd {$OOM}/kubernetes/config
| ||||||||||||||||||
2 | Wait for the config-init container to finish | Use the following command to monitor onap config init intil it reaches to Completed STATUS:
| ||||||||||||||||||
Additional checks for config-init |
|
Deploy the SDN-C Application
Info | ||
---|---|---|
| ||
You can skip the steps in this section by following instruction in autoDeploySdnc of Scripts section to
|
Execute the followings on the master node.
# | Purpose | Command and Examples | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | Set the OOM Kubernetes config environment(If you have set the OOM Kubernetes config enviorment in the same terminal, you can skip this step) | cd {$OOM}/kubernetes/oneclick source setenv.bash | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
1 | Run the createAll script to deploy the SDN-C appilication | cd {$OOM}/kubernetes/oneclick ./createAll.bash -n onap -a sdnc
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||
Ensure that the SDN-C appication has started | Use the kubectl get pods command to monitor the SDN-C startup; you should observe:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
2 | Validate that all SDN-C pods and services are created properly | helm ls --all
kubectl get namespace
kubectl get deployment --all-namespaces
kubectl get clusterrolebinding --all-namespaces
kubectl get serviceaccounts --all-namespaces
kubectl get service -n onap-sdnc
kubectl get pods --all-namespaces -a
docker ps |grep sdnc
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||
3 | Validate that the SDN-C bundlers are up |
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||
4 | Validate that the SDN-C APIs are shown on the ODL RestConf page | Access the ODL RestConf page from the following URL:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||
5 | Validate the SDN-C ODL cluster | Goal:Verify if the SDN-C ODL-Cluster is running properly Prerequisites
Use ODL intergration tool to monitor ODL cluster
Use testCluster RPC to test SDN-C load sharingThe testCluster-bundle.zip provides a testBundle which offers a testCluster API to help with validating SDN-C RPC load sharing in the deployed SDN-C cluster. It's just as easy as to do the following:
|
Undeploy the SDN-C Application
Info | ||
---|---|---|
| ||
You can skip the steps in this section by following instruction in autoCleanSdnc of Scripts section to
|
# | Purpose | Command and Examples |
---|---|---|
0 |
(If you have set OOM kubernetes config enviorment in the same terminal, you can skip this step) |
...
web-deployment.yaml
...
Field | New value | Old value |
---|---|---|
.apiVersion | apps/v1beta1 1 | extensions/v1beta1 |
.kind | StatefulSet | Deployment |
.spec.serviceName | "sdnc-portal" 2 | N/A |
.spec.replicas | 3 | N/A |
.spec.podManagementPolicy | "Parellel" 3 | N/A |
...
Changed from Static volume mounts (PV and PVCs) to Dynamic volume mounts
New Value | Old Value | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
...
all-services.yaml
...
- Add a headless service of sdnc 6
Code Block |
---|
---
apiVersion: v1
kind: Service
metadata:
name: sdnhostcluster
namespace: onap-sdnc
labels:
app: sdnc
annotations:
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
ports:
- name: "sdnc-cluster-port"
port: 2550
clusterIP: None
selector:
app: sdnc
sessionAffinity: None
type: ClusterIP |
...
- Expose 8080 port for using jolokia
(Find sdnhost service and add expose one more port under .spec.ports)
Code Block |
---|
- name: "sdnc-jolokia-port-8080"
port: 9090
targetPort: 8080
nodePort: {{ .Values.nodePortPrefix }}00 |
Notes:
- Use .apiVersion "apps/v1beta1" for the Kubernetes version before 1.8.0; otherwise, use .apiVersion "apps/v1beta2"
Check the Kubernetes version using the command "kubectl version"
Expand title Example of kubernetes version 1.7.7 Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.2", GitCommit:"bdaeafa71f6c7c04636251031f93464384d54963", GitTreeState:"clean", BuildDate:"2017-10-24T19:48:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.7-rancher1", GitCommit:"a1ea37c6f6d21f315a07631b17b9537881e1986a", GitTreeState:"clean", BuildDate:"2017-10-02T21:33:08Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}Expand title Example of kubernetes version 1.8.3 Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.3", GitCommit:"f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd", GitTreeState:"clean", BuildDate:"2017-11-08T18:39:33Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.3-rancher1", GitCommit:"beb8311a9f114ba92558d8d771a81b7fb38422ae", GitTreeState:"clean", BuildDate:"2017-11-14T00:54:19Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
- The value must align with the associated service name in the all_services.yaml file under the same directory.
- By default, .spec.podManagementPolicy has the value "OrderReady".
- With the value "OrderReady", the Kubernetes pod management tells the StatefulSet controller to respect the ordering guarantees, waiting for a Pod to become Running and Ready or completely terminate and then launching or terminating another Pod.
- With the value "Parallel", the Kubernetes pod management tells the StatefulSet controller to launch or terminate all Pods in parallel, not waiting for Pods to become Running and Ready or completely terminated prior to launching or terminating another Pod.
- Export 2 new ports for
- SDNC cluster (2550)
- Jolokia (8080)
- Since StartODL.sh has to be changed to enable clusters to function, two paths must be mounted:
- mount /home/ubuntu/cluster/script/cluster-startODL.sh (local) to replace /opt/onap/sdnc/bin/startODL.sh (docker), so that we can use our local updated script with cluster config.
- mount /home/ubuntu/cluster/deploy (local) to /opt/opendaylight/current/deploy (docker), so that we can dynamically deploy test bundles outside pods.
- The newly added headless service is SDNC pods in SDNC cluster to be able to find each other with fixed FQDN directly.
Create New Script: cluster-startODL.sh
This is manual for now, It can be automated when we automate the SDN-C cluster deployment.
Create cluster-startODL.sh under /home/ubuntu/cluster/script/
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
#!/bin/bash
###
# ============LICENSE_START=======================================================
# openECOMP : SDN-C
# ================================================================================
# Copyright (C) 2017 AT&T Intellectual Property. All rights
# reserved.
# ================================================================================
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ============LICENSE_END=========================================================
###
function enable_odl_cluster(){
echo "Installing Opendaylight cluster features"
${ODL_HOME}/bin/client -u karaf feature:install odl-mdsal-clustering
${ODL_HOME}/bin/client -u karaf feature:install odl-jolokia
echo "Update cluster information statically"
hm=$(hostname)
echo "Get current Hostname ${hm}"
#TODO Do naming check
node=($(echo ${hm} | tr '-' '\n'))
node_name=${node[0]}
node_index=${node[1]}
#TODO for dynamic clustering, have to use rest call to Master server
#for getting the real replication number
#sdnhostcluster should be the same as headless service
node_list="${node_name}-0.sdnhostcluster.onap-sdnc.svc.cluster.local";
for ((i=1;i<=2;i++));
do
node_list="${node_list} ${node_name}-$i.sdnhostcluster.onap-sdnc.svc.cluster.local"
done
/opt/opendaylight/current/bin/configure_cluster.sh $((node_index+1)) ${node_list}
}
# Install SDN-C platform components if not already installed and start container
ODL_HOME=${ODL_HOME:-/opt/opendaylight/current}
ODL_ADMIN_PASSWORD=${ODL_ADMIN_PASSWORD:-Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U}
SDNC_HOME=${SDNC_HOME:-/opt/onap/sdnc}
SLEEP_TIME=${SLEEP_TIME:-120}
MYSQL_PASSWD=${MYSQL_PASSWD:-openECOMP1.0}
#
# Wait for database
#
echo "Waiting for mysql"
until mysql -h dbhost -u root -p${MYSQL_PASSWD} mysql &> /dev/null
do
printf "."
sleep 1
done
echo -e "\nmysql ready"
if [ ! -f ${SDNC_HOME}/.installed ]
then
echo "Installing SDN-C database"
${SDNC_HOME}/bin/installSdncDb.sh
echo "Starting OpenDaylight"
${ODL_HOME}/bin/start
echo "Waiting ${SLEEP_TIME} seconds for OpenDaylight to initialize"
sleep ${SLEEP_TIME}
echo "Installing SDN-C platform features"
${SDNC_HOME}/bin/installFeatures.sh
if [ -x ${SDNC_HOME}/svclogic/bin/install.sh ]
then
echo "Installing directed graphs"
${SDNC_HOME}/svclogic/bin/install.sh
fi
enable_odl_cluster
echo "Restarting OpenDaylight"
${ODL_HOME}/bin/stop
echo "Installed at `date`" > ${SDNC_HOME}/.installed
fi
exec ${ODL_HOME}/bin/karaf |
Create the ONAP Config
# | Purpose | Command and Examples | |||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0.1 | (Only Once) Create the ONAP config using a sample YAML file | cd {$OOM}/kubernetes/config cp onap-parameters-sample.yaml onap-parameters.yaml | |||||||||||||||||||
0 | Set the OOM Kubernetes config environment | cd {$OOM}/kubernetes/oneclick source setenv.bash | |||||||||||||||||||
1 |
| createConfig script to create the ONAP configcd {$OOM}/kubernetes/config
| ./ | createConfigdeleteAll. | sh bash -n onap -a sdnc
| createConfig
| Creating configuration for ONAP instance: onap
| created
| **** Done
| 2 | Wait for the config-init container to finish | ||||||||||
2 |
| docker ps |grep sdnc
kubectl get pods | kubctl get pod --all-namespaces -a
| final output||||||||||||||||||
The final output should be shown as the the following with onap config in Completed STATUS: |
| Additional checks for config-init | helm | helm ls --all
helm status onap-config Expand |
LAST DEPLOYED: Tue Nov 21 17:07:13 2017 RESOURCES: NAME READY STATUS RESTARTS AGE
kubectl get service --all-namespaces
NAME STATUS AGE |
Deploy the SDN-C Application
kubectl get serviceaccounts --all-namespaces | |||||||||||||||||||||||||||||||||||||||||||||||||||
# | Purpose | Command and Examples | |||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | Set the OOM Kubernetes config environment(If you have set the OOM Kubernetes config enviorment in the same terminal, you can skip this step) | cd {$OOM}/kubernetes/oneclick source setenv.bash | |||||||||||||||||||||||||||||||||||||||||||||||||
1 | Run the createAll script to deploy the SDN-C appilication |
| createAll outputEnsure that the SDN-C appication has started | Use the kubectl get pods command to monitor the SDN-C startup; you should observe:
| 2 | Validate that all SDN-C pods and services are created properly |
Expand | ||
---|---|---|
| ||
ubuntu@sdnc-k8s:~$ helm ls --all |
kubectl get namespaces
Expand | ||
---|---|---|
| ||
ubuntu@sdnc-k8s:~$ kubectl get namespaces |
kubectl get deployment --all-namespaces
Expand | ||
---|---|---|
| ||
ubuntu@sdnc-k8s-2:~$ kubectl get deployment --all-namespaces |
kubectl get clusterrolebinding --all-namespaces
Expand | ||
---|---|---|
| ||
ubuntu@sdnc-k8s:~$ kubectl get clusterrolebinding --all-namespaces |
kubectl get serviceaccounts --all-namespaces
Expand | ||
---|---|---|
| ||
ubuntu@sdnc-k8s:~$ kubectl get serviceaccounts --all-namespaces |
kubectl get service --all-namespaces
Expand | ||
---|---|---|
| ||
ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ kubectl get service --all-namespaces |
kubectl get pods --all-namespaces -a
Expand | ||
---|---|---|
| ||
ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ kubectl get pods --all-namespaces -a |
docker ps |grep sdnc
title | Example of SDNC docker container |
---|
$ docker ps |grep sdnc |wc -l
14
$ docker ps |grep sdnc
9a1fc91b6dcc docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942 "filebeat -e" 19 minutes ago Up 19 minutes k8s_filebeat-onap_sdnc-2_onap-sdnc_c7feb9d1-d08a-11e7-957f-0269cb13eff1_0
9dbaa04e160c docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942 "filebeat -e" 19 minutes ago Up 19 minutes k8s_filebeat-onap_sdnc-0_onap-sdnc_c7f28480-d08a-11e7-957f-0269cb13eff1_0
fca36e9b5353 nexus3.onap.org:10001/onap/sdnc-image@sha256:1049151464b3e60d9a553bc2f3bdaf79555839217f0557652e982ca99398375a "/opt/onap/sdnc/bin/s" 19 minutes ago Up 19 minutes k8s_sdnc-controller-container_sdnc-2_onap-sdnc_c7feb9d1-d08a-11e7-957f-0269cb13eff1_0
00efa164a58a nexus3.onap.org:10001/onap/sdnc-image@sha256:1049151464b3e60d9a553bc2f3bdaf79555839217f0557652e982ca99398375a "/opt/onap/sdnc/bin/s" 19 minutes ago Up 19 minutes k8s_sdnc-controller-container_sdnc-0_onap-sdnc_c7f28480-d08a-11e7-957f-0269cb13eff1_0
4a2769dfee37 mysql/mysql-server@sha256:720f301388709af2c84ee09ba51340d09d1e9f7ba45f19719b5b18b5fa696771 "/entrypoint.sh mysql" 19 minutes ago Up 19 minutes (healthy) k8s_sdnc-db-container_sdnc-dbhost-0_onap-sdnc_c7e050b3-d08a-11e7-957f-0269cb13eff1_0
8b4a21cb2bd2 mysql/mysql-server@sha256:720f301388709af2c84ee09ba51340d09d1e9f7ba45f19719b5b18b5fa696771 "/entrypoint.sh mysql" 19 minutes ago Up 19 minutes (healthy) k8s_sdnc-db-container_sdnc-dbhost-1_onap-sdnc_cde0fde0-d08a-11e7-957f-0269cb13eff1_0
04904cb18336 gcr.io/google_containers/pause-amd64:3.0 "/pause" 19 minutes ago Up 19 minutes k8s_POD_sdnc-portal-0_onap-sdnc_c810f3e4-d08a-11e7-957f-0269cb13eff1_0
e89a3e28505a gcr.io/google_containers/pause-amd64:3.0 "/pause" 19 minutes ago Up 19 minutes k8s_POD_sdnc-2_onap-sdnc_c7feb9d1-d08a-11e7-957f-0269cb13eff1_0
7d2c4ac066f4 gcr.io/google_containers/pause-amd64:3.0 "/pause" 19 minutes ago Up 19 minutes k8s_POD_sdnc-dgbuilder-356329770-cpfzj_onap-sdnc_c7dd4b22-d08a-11e7-957f-0269cb13eff1_0
660ee9119001 gcr.io/google_containers/pause-amd64:3.0 "/pause" 19 minutes ago Up 19 minutes k8s_POD_sdnc-0_onap-sdnc_c7f28480-d08a-11e7-957f-0269cb13eff1_0
fa0e59e1a7c7 gcr.io/google_containers/pause-amd64:3.0 "/pause" 19 minutes ago Up 19 minutes k8s_POD_sdnc-dbhost-0_onap-sdnc_c7e050b3-d08a-11e7-957f-0269cb13eff1_0
a6e6445b87eb gcr.io/google_containers/pause-amd64:3.0 "/pause" 19 minutes ago Up 19 minutes k8s_POD_sdnc-1_onap-sdnc_c7f6b06f-d08a-11e7-957f-0269cb13eff1_0
1d00b4bb46c0 gcr.io/google_containers/pause-amd64:3.0 "/pause" 19 minutes ago Up 19 minutes k8s_POD_sdnc-portal-1_onap-sdnc_c813c667-d08a-11e7-957f-0269cb13eff1_0
$
In the above example, it missed sdnc-1 container as I had a failure of it due to directory sharing.
Validate that the SDN-C bundlers are up
Expand | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||
|
Expand | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||
|
Validate that the SDN-C APIs are shown on the ODL RestConf page
Access the ODL RestConf page from the following URL:
http://<Kubernetes-Master-Node-IP>:30202/apidoc/explorer/index.html
Expand | ||
---|---|---|
| ||
Validate the SDN-C ODL cluster
Goal:
Verify if the SDNC ODL-Cluster is running properly
Prerequisites
- This test is on one of your Kubernetes nodes
- Make sure python-pycurl is installed
- If not, for Ubuntu use "apt-get install python-pycurl" to install it
Use ODL intergration tool to monitor ODL cluster
Clone ODL Integration-test project | git clone https://github.com/opendaylight/integration-test.git | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Enter cluster-montor folder | cd integration-test/tools/clustering/cluster-monitor | |||||||||||
Create update-cluster.bash script | vi update-cluster.sh
This script is used to fetch all the IPs of SDN-C pods and automatically update cluster.json file | |||||||||||
Start cluster montor UI | ./update-cluster.bash Note: If applications inside any of these three SDNC pods are not fully started, this script won't be executed successfully due to the issues such as connection error, value error and etc. Otherwise, you should see the monitoring UI as the following: |
Use testCluster RPC to test SDN-C load sharing
The testCluster-bundle.zip provides a testBundle which offers a testCluster API to help with validating SDN-C RPC load sharing in the deployed SDN-C cluster.
It's just as easy as to do the following:
download testCluster-bundle.zipfrom the attachment, and place it to the sdnc-deployhostPath which has been defined in .spec.volumes of sdnc-deployment.yaml file- As this hostPath is mounted as ODL's deploy directory, testBundle will be automatically loaded by ODL and the testCluster API will be availble to use.
testCluster api can be accessed from ODL RestConf page, postman or curl command:
Expand title curl command curl -u admin:Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U -H 'Accept: application/json' -X POST 'http://${kubernetes-master-api}:30202/restconf/operations/testCluster:who-am-i'
Expand title An example of testCluster API response { "output": { "node": "sdnc-2" } }
Undeploy the SDN-C Application
...
Aet the OOM Kubernetes config environment
(If you have set OOM kubernetes config enviorment in the same terminal, you can skip this step)
...
cd {$OOM}/kubernetes/oneclick
source setenv.bash
...
Run the deleteAll script to delete all SDN-C pods and services
...
./deleteAll.bash -n onap -a sdnc
Expand | ||
---|---|---|
| ||
********** Cleaning up ONAP: Waiting for namespaces termination... ********** Gone ********** |
...
Validate that all SDN-C pods and servers are cleaned up
...
docker ps |grep sdnc
Expand | ||
---|---|---|
| ||
ubuntu@sdnc-k8s:~$ docker ps |grep sdnc |
kubectl get pods --all-namespaces -a
Expand | ||
---|---|---|
| ||
ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ kubectl get pods --all-namespaces -a |
kubectl get service --all-namespaces
Expand | ||
---|---|---|
| ||
ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ kubectl get service --all-namespaces |
kubectl get serviceaccounts --all-namespaces
Expand | ||
---|---|---|
| ||
ubuntu@sdnc-k8s:~$ kubectl get serviceaccounts --all-namespaces |
kubectl get clusterrolebinding --all-namespaces
Expand | ||
---|---|---|
| ||
ubuntu@sdnc-k8s:~$ kubectl get clusterrolebinding --all-namespaces |
kubectl get deployment --all-namespaces
Expand | ||
---|---|---|
| ||
ubuntu@sdnc-k8s:~$ kubectl get deployment --all-namespaces NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE kube-system heapster 1 1 1 1 2d kube-system kube-dns 1 1 1 1 2d kube-system kubernetes-dashboard 1 1 1 1 2d kube-system monitoring-grafana 1 1 1 1 2d kube-system monitoring-influxdb 1 1 1 1 2d kube-system tiller-deploy 1 1 1 1 2d ubuntu@sdnc-k8s:~$ |
kubectl get namespaces
Expand | ||
---|---|---|
| ||
ubuntu@sdnc-k8s:~$ kubectl get namespaces |
helm ls --all
Expand | ||
---|---|---|
| ||
ubuntu@sdnc-k8s:~$ helm ls --all |
Remove the ONAP Config
...
Set the OOM Kubernetes config environment
(If you have set OOM kubernetes config enviorment in the same terminal, you can skip this step)
...
cd {$OOM}/kubernetes/oneclick
source setenv.bash
...
Remove the ONAP config and any deployed applications in one shot
...
./deleteAll.bash -n onap
Expand | ||
---|---|---|
| ||
ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ ./deleteAll.bash -n onap
********** Cleaning up ONAP: Error: release: not found Error from server (NotFound): namespaces "onap-consul" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-consul-admin-binding" not found Service account onap-consul-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-msb" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-msb-admin-binding" not found Service account onap-msb-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-mso" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-mso-admin-binding" not found Service account onap-mso-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-message-router" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-message-router-admin-binding" not found Service account onap-message-router-admin-binding deleted.ls
release "onap-sdnc" deleted namespace "onap-sdnc" deleted clusterrolebinding "onap-sdnc-admin-binding" deleted Service account onap-sdnc-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-vid" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-vid-admin-binding" not found Service account onap-vid-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-robot" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-robot-admin-binding" not found Service account onap-robot-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-portal" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-portal-admin-binding" not found Service account onap-portal-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-policy" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-policy-admin-binding" not found Service account onap-policy-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-appc" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-appc-admin-binding" not found Service account onap-appc-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-aai" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-aai-admin-binding" not found Service account onap-aai-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-sdc" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-sdc-admin-binding" not found Service account onap-sdc-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-dcaegen2" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-dcaegen2-admin-binding" not found Service account onap-dcaegen2-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-log" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-log-admin-binding" not found Service account onap-log-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-cli" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-cli-admin-binding" not found Service account onap-cli-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-multicloud" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-multicloud-admin-binding" not found Service account onap-multicloud-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-clamp" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-clamp-admin-binding" not found Service account onap-clamp-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-vnfsdk" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-vnfsdk-admin-binding" not found Service account onap-vnfsdk-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-uui" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-uui-admin-binding" not found Service account onap-uui-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-aaf" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-aaf-admin-binding" not found Service account onap-aaf-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-vfc" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-vfc-admin-binding" not found Service account onap-vfc-admin-binding deleted.
Error: release: not found Error from server (NotFound): namespaces "onap-kube2msb" not found Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "onap-kube2msb-admin-binding" not found Service account onap-kube2msb-admin-binding deleted.
Waiting for namespaces termination...
********** Gone ********** |
...
Manually clean up
This step is to clean up the leftover items which were created by the config/createConfig script but not cleaned up by the oneclick/deleteAll script.
Expand | ||
---|---|---|
| ||
ubuntu@sdnc-k8s:~/oom/kubernetes/config$ ./createConfig.sh -n onap **** Creating configuration for ONAP instance: onap |
...
ONAP serviceaccount | No action needed. It cannot be deleted by a specific command, but will instead be automatically deleted when the namespace is deleted.
| |||||
---|---|---|---|---|---|---|
ONAP namespace |
| |||||
release |
|
...
Delete the shared folder
...
sudo rm -rf /dockerdata-nfs/onap
kubectl get clusterrolebinding --all-namespaces
kubectl get deployment --all-namespaces
kubectl get namespaces
helm ls --all
|
Remove the ONAP Config
Info | ||
---|---|---|
| ||
You can skip the steps in this section by following instruction in autoCleanOnapConfig of Scripts section to
|
# | Purpose | Command and Examples | ||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 |
(If you have set OOM kubernetes config enviorment in the same terminal, you can skip this step) | cd {$OOM}/kubernetes/oneclick source setenv.bash | ||||||||||||||||||||||||||
1 |
| ./deleteAll.bash -n onap
| ||||||||||||||||||||||||||
2 |
This step is to clean up the leftover items which were created by the config/createConfig script but not cleaned up by the oneclick/deleteAll script.
|
| ||||||||||||||||||||||||||
3 |
| sudo rm -rf /dockerdata-nfs/onap |
Scripts
Anchor | ||||
---|---|---|---|---|
|
The following scripts are intended to be placed under directory {$OOM}/kubernetes/oneclick/tools and help to simplify various procedures by automating them
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
########################################################################################
# This script wraps {$OOM}/kubernetes/config/createConfig.sh script #
# and will only terminated when the ONAP configuration is Completed #
# #
# Before using it, do the following to prepare the bash file: #
# 1, cd {$OOM}/kumbernetes/oneclick/tools #
# 2, vi autoCreateOnapConfig.bash #
# 3, paste the full content here to autoCreateOnapConfig.bash file and save the file #
# 4, chmod 777 autoCleanOnapConfig.bash |
Scripts
The following scripts help to simplify various procedures by automating them
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
######################################################################################## # This script replaces {$OOM}/kubernetes/config/createConfig.sh script # # and will only terminated when the ONAP configuration is Completed# # # # # To run it, just enter the following command: # # ./autoCreateOnapConfig.bash <namespace, default is "onap"> # # Before using it, do the following to prepare the bash file: # # 1, cd {$OOM}/kumbernetes/oneclick # ######################################################################################## #!/bin/bash NS=$1 if [[ -z $NS ]] then echo "Namespace is not specified, use onap namespace." NS="onap" fi echo "Create $NS config under config directory..." cd ../../config ./createConfig.sh -n $NS cd - echo "...done : kubectl get namespace ----------------------------------------------- >>>>>>>>>>>>>> k8s namespace" kubectl get # # 2, vi autoCreateOnapConfig.bash # # 3, paste the full content here to autoCreateOnapConfig.bash file and save the file # # 4, chmod 777 autoCreateOnapConfig.bash namespace echo " ----------------------------------------------- >>>>>>>>>>>>>> helm : helm ls --all" helm ls --all echo " ----------------------------------------------- >>>>>>>>>>>>>> pod : kubectl get pods -n $NS -a" kubectl get pods -n $NS -a while true do echo "wait for $NS config pod reach to Completed STATUS" sleep 5 echo "-----------------------------------------------" kubectl get pods -n $NS -a status=`kubectl get pods -n $NS -a |grep config |xargs echo | cut -d' ' -f3` if [ "$status" = "Completed" ] then echo "$NS config is Completed!!!" break fi if [ "$status" = "Error" ] then echo "$NS config is failed with Error!!! POD logs are:" kubectl logs config -n $NS -f break fi done |
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
######################################################################################## # This script wraps {$OOM}/kubernetes/oneclick/deleteAll.sh script along with# # To run it, just enter the following command: # # the following steps to clean up ONAP configure: # # ./autoCreateOnapConfig.bash # # - remove ONAP namespace # ######################################################################################## #!/bin/bash echo "Create ONAP config under config directory..." cd ../config ./createConfig.sh -n onap cd - echo "...done : kubectl get namespace ----------------------------------------------- >>>>>>>>>>>>>> k8s namespace" kubectl get namespace echo " ----------------------------------------------- >>>>>>>>>>>>>> helm : helm ls --all" helm ls --all echo " ----------------------------------------------- >>>>>>>>>>>>>> pod : kubectl get pods --all-namespaces -a" kubectl get pods --all-namespaces -a status=`kubectl get pods --all-namespaces -a |grep onap |xargs echo | cut -d' ' -f4` while true do echo "wait for onap config pod reach to Completed STATUS" sleep 5 echo "-----------------------------------------------" kubectl get pods --all-namespaces -a status=`kubectl get pods --all-namespaces -a |grep onap |xargs echo | cut -d' ' -f4` if [ "$status" = "Completed" ] then echo "onap config is Completed!!!" break fi done | ||||||||
Code Block | ||||||||
| ||||||||
######################################################################################## # This script wraps {$OOM}/kubernetes/oneclick/deleteAll.sh script along with # # - remove ONAP release # # - remove ONAP shared directory # # the following steps to clean up ONAP configure: # # # # - remove ONAP namespace # # Before using it, do -the removefollowing ONAP releaseto prepare the bash file: # # 1, cd {$OOM}/kumbernetes/oneclick/tools # # - remove ONAP shared directory # # 2, vi autoCleanOnapConfig.bash # # # # 3, paste the full content here to autoCleanOnapConfig.bash file and save the file # # 4, chmod 777 autoCleanOnapConfig.bash # # Before using it, do the following to prepare the bash file: # # 1, cd {$OOM}/kumbernetes/oneclick # # To run 2it, vijust autoCleanOnapConfig.bashenter the following command: # # 3, paste the full content here to ./autoCleanOnapConfig.bash file and save the file # # 4, chmod 777 autoCleanOnapConfig.bash # ######################################################################################## # To run it, just enter the following command: # # ./autoCleanOnapConfig.bash # ######################################################################################## #!/bin/bash ./deleteAll.bash -n onap echo "!/bin/bash NS=$1 if [[ -z $NS ]] then echo "Namespace is not specified, use onap namespace." NS="onap" fi echo "Clean up $NS configuration" cd .. ./deleteAll.bash -n $NS cd - echo "---------------------------------------------- Force remove namespace..." kubectl delete namespace $NS echo "...done : kubectl get namespace ----------------------------------------------- >>>>>>>>>>>>>> k8s namespace" kubectl get namespace while [[ ! -z `kubectl get namespace|grep $NS` ]] do echo "Wait for namespace $NS to be deleted ----------------------------------------------- >>>>>>>>>>>>>> k8s namespace" kubectl get namespace sleep 2 done echo "Force delete helm process ..." helm delete $NS-config --purge --debug echo "...done : helm ls --all ----------------------------------------------- Force remove namespace...>>>>>>>>>>>>>> helm" kubectlhelm delete namespace onap echo "...done" kubectl get namespace ls --all echo "ForceRemove delete helm process $NS dockerdata..." helmsudo deleterm onap-configrf --purge --debug/dockerdata-nfs/onap echo "...done" helm ls --all echo "Remove ONAP dockerdata..." sudo rm -rf /dockerdata-nfs/onap echo "...done : ls -altr /dockerdata-nfs ----------------------------------------------- >>>>>>>>>>>>>> /dockerdata-nfs directory" ls -altr /dockerdata-nfs |
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
######################################################################################## # This script wraps {$OOM}/kubernetes/oneclick/createAll.sh script along with # # the following steps to deploy ONAP SDNC application: # # - wait until sdnc-0 is running properly with both (2) containers up # # # # Before using it, do the following to prepare the bash file: # # 1, cd {$OOM}/kumbernetes/oneclick/tools # # 2, vi autoDeploySdnc.bash # # 1, cd {$OOM}/kumbernetes/oneclick # # 3, paste the full content here to autoDeploySdnc.bash file and save the file # # 24, vichmod 777 autoDeploySdnc.bash # # # # 3, paste the full content here to autoDeploySdnc.bash file and save the file # # 4, chmod 777 autoDeploySdnc.bash # # To run it, just enter the following command: # # ./autoDeploySdnc.bash # ######################################################################################## #!/bin/bash cd .. echo "Deploy SDNC..." ./createAll.bash -n onap -a sdnc -n onap -a sdnc returnCode=$? echo "deploy result: $returnCode" cd - if [ "$returnCode" != "0" ] then echo "Abort..." exit fi echo "...done ----------------------------------------------- >>>>>>>>>>>>>> pod : kubectl get pods --all-namespaces -a -o wide" kubectl get pods --all-namespaces -a -o wide status=`kubectl get pods --all-namespaces -a |grep sdnc-0 |xargs echo | cut -d' ' -f3` while true do echo "wait for onap sdnc-0 reachs fully running" sleep 5 echo "-----------------------------------------------" kubectl get pods --all-namespaces -a -o wide status=`kubectl get pods --all-namespaces -a |grep sdnc-0 |xargs echo | cut -d' ' -f3` if [ "$status" = "2/2" ] then echo "onap sdnc-0 is running!!!" break fi done |
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
######################################################################################## # This script wraps {$OOM}/kubernetes/oneclick/deleteAll.sh script along with # # the following steps to un-deploy ONAP SDNC application fully: # # - force remove clusterrolebinding for onap-sdnc # # - force remove namespace for onap-sdnc # # - force remove release for onap-sdnc # # - wait until onap-sdnc namespace is remvoed # # # # Before using it, do the following to prepare the bash file: # # 1, cd {$OOM}/kumbernetes/oneclick /tools # # 2, vi autoCleanSdnc.bash # # 3, paste the full content here to autoCleanSdnc.bash file and save the file # # 4, chmod 777 autoCleanSdnc.bash # # # # To run it, just enter the following command: # # ./autoCleanSdnc.bash # ######################################################################################## #!/bin/bash cd .. ./deleteAll.bash -n onap -a sdnc cd - echo "---------------------------------------------- Remove clusterrolebindnig..." kubectl delete clusterrolebinding onap-sdnc-admin-binding echo "...done : kubectl get clusterrolebinding" kubectl get clusterrolebinding echo "Remove onap-sdnc namespace..." kubectl delete namespaces onap-sdnc echo "...done : kubectl get namespaces" kubectl get namespaces echo "Delete onap-sdnc release..." helm delete onap-sdnc --purge echo "...done: helm ls --all" helm ls --all sdncCount=`kubectl get namespaces | grep onap-sdnc | wc -l` while true do echo "wait for onap-sdnc namespace to be removed" sleep 5 echo "-----------------------------------------------" kubectl get namespaces sdncCount=`kubectl get namespaces | grep onap-sdnc | wc -l` if [ "$sdncCount" = "0" ] then echo "sdnc removed!!!" break fi done |