Table of Contents | ||
---|---|---|
|
SDNC Cluster Deployment
Configure SDNC Cluster Deployment
We are using Kubernetes replicas to achieve the SDNC cluster deployment (see details from 4. Deploy/Undeploy the SDN-C Cluster for the desired goal).
This only needs to be done one time and, at the moment, all modifications are done manually (they can be automated via scripting in the future when the needs come up).
Edit SDNC Templates
The following is the list of SDNC deployment templates which need to be modified for an SDN-C cluster deployment:
# | Template file under {$OOM}/kubernetes/sdnc/templates | Changed/Added fields and values | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | db-deployment.yaml |
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
2 | sdnc-deployment.yaml |
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
3 | web-deployment.yaml |
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
4 | sdnc-pv-pvc.yaml | Changed from Static volume mounts (PV and PVCs) to Dynamic volume mounts
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
5 | all-services.yaml |
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
(Find sdnhost service and add expose one more port under .spec.ports)
|
Notes:
- Use .apiVersion "apps/v1beta1" for the Kubernetes version before 1.8.0; otherwise, use .apiVersion "apps/v1beta2"
Check the Kubernetes version using the command "kubectl version"
Expand title Example of kubernetes version 1.7.7 Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.2", GitCommit:"bdaeafa71f6c7c04636251031f93464384d54963", GitTreeState:"clean", BuildDate:"2017-10-24T19:48:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.7-rancher1", GitCommit:"a1ea37c6f6d21f315a07631b17b9537881e1986a", GitTreeState:"clean", BuildDate:"2017-10-02T21:33:08Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}Expand title Example of kubernetes version 1.8.3 Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.3", GitCommit:"f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd", GitTreeState:"clean", BuildDate:"2017-11-08T18:39:33Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.3-rancher1", GitCommit:"beb8311a9f114ba92558d8d771a81b7fb38422ae", GitTreeState:"clean", BuildDate:"2017-11-14T00:54:19Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
- The value must align with the associated service name in the all_services.yaml file under the same directory.
- By default, .spec.podManagementPolicy has the value "OrderReady".
- With the value "OrderReady", the Kubernetes pod management tells the StatefulSet controller to respect the ordering guarantees, waiting for a Pod to become Running and Ready or completely terminate and then launching or terminating another Pod.
- With the value "Parallel", the Kubernetes pod management tells the StatefulSet controller to launch or terminate all Pods in parallel, not waiting for Pods to become Running and Ready or completely terminated prior to launching or terminating another Pod.
- Export 2 new ports for
- SDNC cluster (2550)
- Jolokia (8080)
- Since StartODL.sh has to be changed to enable clusters to function, two paths must be mounted:
- mount /home/ubuntu/cluster/script/cluster-startODL.sh (local) to replace /opt/onap/sdnc/bin/startODL.sh (docker), so that we can use our local updated script with cluster config.
- mount /home/ubuntu/cluster/deploy (local) to /opt/opendaylight/current/deploy (docker), so that we can dynamically deploy test bundles outside pods.
- The newly added headless service is SDNC pods in SDNC cluster to be able to find each other with fixed FQDN directly.
Create New Script: cluster-startODL.sh
This is manual for now, It can be automated when we automate the SDN-C cluster deployment.
Create cluster-startODL.sh under /home/ubuntu/cluster/script/
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
#!/bin/bash ### # ============LICENSE_START======================================================= # openECOMP : SDN-C # ================================================================================ # Copyright (C) 2017 AT&T Intellectual Property. All rights # reserved. # ================================================================================ # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============LICENSE_END========================================================= ### function enable_odl_cluster(){ echo "Installing Opendaylight cluster features" ${ODL_HOME}/bin/client -u karaf feature:install odl-mdsal-clustering ${ODL_HOME}/bin/client -u karaf feature:install odl-jolokia echo "Update cluster information statically" hm=$(hostname) echo "Get current Hostname ${hm}" #TODO Do naming check node=($(echo ${hm} | tr '-' '\n')) node_name=${node[0]} node_index=${node[1]} #TODO for dynamic clustering, have to use rest call to Master server #for getting the real replication number #sdnhostcluster should be the same as headless service node_list="${node_name}-0.sdnhostcluster.onap-sdnc.svc.cluster.local"; for ((i=1;i<=2;i++)); do node_list="${node_list} ${node_name}-$i.sdnhostcluster.onap-sdnc.svc.cluster.local" done /opt/opendaylight/current/bin/configure_cluster.sh $((node_index+1)) ${node_list} } # Install SDN-C platform components if not already installed and start container ODL_HOME=${ODL_HOME:-/opt/opendaylight/current} ODL_ADMIN_PASSWORD=${ODL_ADMIN_PASSWORD:-Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U} SDNC_HOME=${SDNC_HOME:-/opt/onap/sdnc} SLEEP_TIME=${SLEEP_TIME:-120} MYSQL_PASSWD=${MYSQL_PASSWD:-openECOMP1.0} # # Wait for database # echo "Waiting for mysql" until mysql -h dbhost -u root -p${MYSQL_PASSWD} mysql &> /dev/null do printf "." sleep 1 done echo -e "\nmysql ready" if [ ! -f ${SDNC_HOME}/.installed ] then echo "Installing SDN-C database" ${SDNC_HOME}/bin/installSdncDb.sh echo "Starting OpenDaylight" ${ODL_HOME}/bin/start echo "Waiting ${SLEEP_TIME} seconds for OpenDaylight to initialize" sleep ${SLEEP_TIME} echo "Installing SDN-C platform features" ${SDNC_HOME}/bin/installFeatures.sh if [ -x ${SDNC_HOME}/svclogic/bin/install.sh ] then echo "Installing directed graphs" ${SDNC_HOME}/svclogic/bin/install.sh fi enable_odl_cluster echo "Restarting OpenDaylight" ${ODL_HOME}/bin/stop echo "Installed at `date`" > ${SDNC_HOME}/.installed fi exec ${ODL_HOME}/bin/karaf |
Create the ONAP Config
# | Purpose | Command and Examples | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0.1 | (Only Once) Create the ONAP config using a sample YAML file | cd {$OOM}/kubernetes/config cp onap-parameters-sample.yaml onap-parameters.yaml | ||||||||||||||||||
0 | Set the OOM Kubernetes config environment | cd {$OOM}/kubernetes/oneclick source setenv.bash | ||||||||||||||||||
1 | Run the createConfig script to create the ONAP config | cd {$OOM}/kubernetes/config
| ||||||||||||||||||
2 | Wait for the config-init container to finish | Use the following command to monitor onap config init intil it reaches to Completed STATUS:
| ||||||||||||||||||
Additional checks for config-init |
|
Deploy the SDN-C Application
# | Purpose | Command and Examples | ||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | Set the OOM Kubernetes config environment(If you have set the OOM Kubernetes config enviorment in the same terminal, you can skip this step) | cd {$OOM}/kubernetes/oneclick source setenv.bash | ||||||||||||||||||||||||||||||||||||||||
1 | Run the createAll script to deploy the SDN-C appilication | cd {$OOM}/kubernetes/oneclick ./createAll.bash -n onap -a sdnc
| ||||||||||||||||||||||||||||||||||||||||
Ensure that the SDN-C appication has started | Use the kubectl get pods command to monitor the SDN-C startup; you should observe:
| |||||||||||||||||||||||||||||||||||||||||
2 | Validate that all SDN-C pods and services are created properly | helm ls --all
kubectl get namespaces
kubectl get deployment --all-namespaces
kubectl get clusterrolebinding --all-namespaces
kubectl get serviceaccounts --all-namespaces
kubectl get service --all-namespaces
kubectl get pods --all-namespaces -a
docker ps |grep sdnc
| ||||||||||||||||||||||||||||||||||||||||
3 | Validate that the SDN-C bundlers are up |
| ||||||||||||||||||||||||||||||||||||||||
4 | Validate that the SDN-C APIs are shown on the ODL RestConf page | Access the ODL RestConf page from the following URL:
| ||||||||||||||||||||||||||||||||||||||||
5 | Validate the SDN-C ODL cluster | Goal:Verify if the SDNC ODL-Cluster is running properly Prerequisites
Use ODL intergration tool to monitor ODL cluster
Use testCluster RPC to test SDN-C load sharingThe testCluster-bundle.zip provides a testBundle which offers a testCluster API to help with validating SDN-C RPC load sharing in the deployed SDN-C cluster. It's just as easy as to do the following:
|
Undeploy the SDN-C Application
# | Purpose | Command and Examples | ||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | Aet the OOM Kubernetes config environment(If you have set OOM kubernetes config enviorment in the same terminal, you can skip this step) | cd {$OOM}/kubernetes/oneclick source setenv.bash | ||||||||||||||||||||||||||||||||||||||||
1 | Run the deleteAll script to delete all SDN-C pods and services | ./deleteAll.bash -n onap -a sdnc
| ||||||||||||||||||||||||||||||||||||||||
2 | Validate that all SDN-C pods and servers are cleaned up | docker ps |grep sdnc
kubectl get pods --all-namespaces -a
kubectl get service --all-namespaces
kubectl get serviceaccounts --all-namespaces
kubectl get clusterrolebinding --all-namespaces
kubectl get deployment --all-namespaces
kubectl get namespaces
helm ls --all
|
Remove the ONAP Config
# | Purpose | Command and Examples | ||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | Set the OOM Kubernetes config environment(If you have set OOM kubernetes config enviorment in the same terminal, you can skip this step) | cd {$OOM}/kubernetes/oneclick source setenv.bash | ||||||||||||||||||||||||||
1 | Remove the ONAP config and any deployed applications in one shot | ./deleteAll.bash -n onap
| ||||||||||||||||||||||||||
2 | Manually clean upThis step is to clean up the leftover items which were created by the config/createConfig script but not cleaned up by the oneclick/deleteAll script.
|
| ||||||||||||||||||||||||||
3 | Delete the shared folder | sudo rm -rf /dockerdata-nfs/onap |
Scripts
The following scripts help to simplify various procedures by automating them
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
######################################################################################## # This script replaces {$OOM}/kubernetes/config/createConfig.sh script # # and will only terminated when the ONAP configuration is Completed # # # # Before using it, do the following to prepare the bash file: # # 1, cd {$OOM}/kumbernetes/oneclick # # 2, vi autoCreateOnapConfig.bash # # 3, paste the full content here to autoCreateOnapConfig.bash file and save the file # # 4, chmod 777 autoCreateOnapConfig.bash # # To run it, just enter the following command: # # ./autoCreateOnapConfig.bash # ######################################################################################## #!/bin/bash echo "Create ONAP config under config directory..." cd ../config ./createConfig.sh -n onap cd - echo "...done : kubectl get namespace ----------------------------------------------- >>>>>>>>>>>>>> k8s namespace" kubectl get namespace echo " ----------------------------------------------- >>>>>>>>>>>>>> helm : helm ls --all" helm ls --all echo " ----------------------------------------------- >>>>>>>>>>>>>> pod : kubectl get pods --all-namespaces -a" kubectl get pods --all-namespaces -a status=`kubectl get pods --all-namespaces -a |grep onap |xargs echo | cut -d' ' -f4` while true do echo "wait for onap config pod reach to Completed STATUS" sleep 5 echo "-----------------------------------------------" kubectl get pods --all-namespaces -a status=`kubectl get pods --all-namespaces -a |grep onap |xargs echo | cut -d' ' -f4` if [ "$status" = "Completed" ] then echo "onap config is Completed!!!" break fi done |
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
######################################################################################## # This script wraps {$OOM}/kubernetes/oneclick/deleteAll.sh script along with # # the following steps to clean up ONAP configure: # # - remove ONAP namespace # # - remove ONAP release # # - remove ONAP shared directory # # # # Before using it, do the following to prepare the bash file: # # 1, cd {$OOM}/kumbernetes/oneclick # # 2, vi autoCleanOnapConfig.bash # # 3, paste the full content here to autoCleanOnapConfig.bash file and save the file # # 4, chmod 777 autoCleanOnapConfig.bash # # To run it, just enter the following command: # # ./autoCleanOnapConfig.bash # ######################################################################################## #!/bin/bash ./deleteAll.bash -n onap echo "---------------------------------------------- Force remove namespace..." kubectl delete namespace onap echo "...done" kubectl get namespace echo "Force delete helm process ..." helm delete onap-config --purge --debug echo "...done" helm ls --all echo "Remove ONAP dockerdata..." sudo rm -rf /dockerdata-nfs/onap echo "...done" ls -altr /dockerdata-nfs |
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
######################################################################################## # This script wraps {$OOM}/kubernetes/oneclick/createAll.sh script along with # # the following steps to deploy ONAP SDNC application: # # - wait until sdnc-0 is running properly with both (2) containers up # # # # Before using it, do the following to prepare the bash file: # # 1, cd {$OOM}/kumbernetes/oneclick # # 2, vi autoDeploySdnc.bash # # 3, paste the full content here to autoDeploySdnc.bash file and save the file # # 4, chmod 777 autoDeploySdnc.bash # # To run it, just enter the following command: # # ./autoDeploySdnc.bash # ######################################################################################## #!/bin/bash echo "Deploy SDNC..." ./createAll.bash -n onap -a sdnc echo "...done ----------------------------------------------- >>>>>>>>>>>>>> pod : kubectl get pods --all-namespaces -a" kubectl get pods --all-namespaces -a status=`kubectl get pods --all-namespaces -a |grep sdnc-0 |xargs echo | cut -d' ' -f3` while true do echo "wait for onap sdnc-0 reachs fully running" sleep 5 echo "-----------------------------------------------" kubectl get pods --all-namespaces -a status=`kubectl get pods --all-namespaces -a |grep sdnc-0 |xargs echo | cut -d' ' -f3` if [ "$status" = "2/2" ] then echo "onap sdnc-0 is running!!!" break fi done |
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
######################################################################################## # This script wraps {$OOM}/kubernetes/oneclick/deleteAll.sh script along with # # the following steps to un-deploy ONAP SDNC application fully: # # - force remove clusterrolebinding for onap-sdnc # # - force remove namespace for onap-sdnc # # - force remove release for onap-sdnc # # - wait until onap-sdnc namespace is remvoed # # # # Before using it, do the following to prepare the bash file: # # 1, cd {$OOM}/kumbernetes/oneclick # # 2, vi autoCleanSdnc.bash # # 3, paste the full content here to autoCleanSdnc.bash file and save the file # # 4, chmod 777 autoCleanSdnc.bash # # To run it, just enter the following command: # # ./autoCleanSdnc.bash # ######################################################################################## #!/bin/bash ./deleteAll.bash -n onap -a sdnc echo "---------------------------------------------- Remove clusterrolebindnig..." kubectl delete clusterrolebinding onap-sdnc-admin-binding echo "...done : kubectl get clusterrolebinding" kubectl get clusterrolebinding echo "Remove onap-sdnc namespace..." kubectl delete namespaces onap-sdnc echo "...done : kubectl get namespaces" kubectl get namespaces echo "Delete onap-sdnc release..." helm delete onap-sdnc --purge echo "...done: helm ls --all" helm ls --all sdncCount=`kubectl get namespaces | grep onap-sdnc | wc -l` while true do echo "wait for onap-sdnc namespace to be removed" sleep 5 echo "-----------------------------------------------" kubectl get namespaces sdncCount=`kubectl get namespaces | grep onap-sdnc | wc -l` if [ "$sdncCount" = "0" ] then echo "sdnc removed!!!" break fi done |