Overview
For ONAP SDN-R load and stress test and Proof of concept, June 19 a three node SDN-R cluster is used. Version is El Alto.
Precondition
- Running physical server cluster (The hardware requirements)
- OpenStack installed
- Virtual Machine cloud image Ubuntu 18.04-server-image
Goal of this description is to setup a complete ONAP with focus of SDNC services
Steps
- Installing Docker, Kubernetes and Rancher
- Initiation of HELM
- Downloading OOM charts
- Installation of all OOM based ONAP componnents
- Installation of Device Simulators
Installing Kubernetes and Rancher
Create the Rancher 3 nodes control cluster named as onap-control on OpenStack
The following instructions describe how to create 3 OpenStack VMs to host the Highly-Available Kubernetes Control Plane. ONAP workloads will not be scheduled on these Control Plane nodes.
Launch new VMs in OpenStack.
Select Ubuntu 18.04 as the boot image for the virtual machine without any volume
Select m1.large flavor
Networking
Apply customization script for Control Plane VMs
The script to be copied:
#!/bin/bash DOCKER_VERSION=18.09.5 KUBECTL_VERSION=1.13.5 HELM_VERSION=2.12.3 export DEBIAN_FRONTEND=noninteractive sudo apt-get update curl https://releases.rancher.com/install-docker/$DOCKER_VERSION.sh | sh cat > /etc/docker/daemon.json << EOF { "insecure-registries" : [ "nexus3.onap.org:10001" ], "log-driver": "json-file", "log-opts": { "max-size": "1m", "max-file": "9" }, "mtu": 1450, "ipv6": true, "fixed-cidr-v6": "2001:db8:1::/64", "registry-mirrors": ["https://nexus3.onap.org:10001"] } EOF sudo usermod -aG docker ubuntu systemctl daemon-reload systemctl restart docker apt-mark hold docker-ce IP_ADDR=`ip address |grep ens|grep inet|awk '{print $2}'| awk -F / '{print $1}'` HOSTNAME=`hostname` sudo echo "$IP_ADDR $HOSTNAME" >> /etc/hosts docker login -u docker -p docker nexus3.onap.org:10001 sudo apt-get install make -y wget https://storage.googleapis.com/kubernetes-release/release/v$KUBECTL_VERSION/bin/linux/amd64/kubectl chmod +x kubectl sudo mv ./kubectl /usr/local/bin/kubectl wget http://storage.googleapis.com/kubernetes-helm/helm-v${HELM_VERSION}-linux-amd64.tar.gz tar -zxvf helm-v${HELM_VERSION}-linux-amd64.tar.gz sudo mv linux-amd64/helm /usr/local/bin/helm sudo apt-get update exit 0
This customization script will:
- Install docker and hold the docker version to 18.09.5
- set hosts file with IP and hostname
- install make
- download and install kubectl
- download and install helm
Launched Instances
Create the Kubernetes 12 nodes worker cluster named as onap-k8s on OpenStack cloud
The following instructions describe how to create OpenStack VMs to host the Highly-Available Kubernetes Workers. ONAP workloads will only be scheduled on these nodes.
Launch new VM instances in OpenStack
Select Ubuntu 18.04 as base image
Select Flavor
The size of Kubernetes hosts depend on the size of the ONAP deployment being installed.
If a small subset of ONAP applications are being deployed (i.e. for testing purposes), then 16GB or 32GB may be sufficient.
Networking
Apply customization script for Kubernetes VM(s)
The scrip to be copied:
#!/bin/bash DOCKER_VERSION=18.09.5 KUBECTL_VERSION=1.13.5 export DEBIAN_FRONTEND=noninteractive sudo apt-get update curl https://releases.rancher.com/install-docker/$DOCKER_VERSION.sh | sh cat > /etc/docker/daemon.json << EOF { "insecure-registries" : [ "nexus3.onap.org:10001","10.20.6.10:30000" ], "log-driver": "json-file", "log-opts": { "max-size": "1m", "max-file": "9" }, "mtu": 1450, "ipv6": true, "fixed-cidr-v6": "2001:db8:1::/64", "registry-mirrors": ["https://nexus3.onap.org:10001"] } EOF sudo usermod -aG docker ubuntu systemctl daemon-reload systemctl restart docker apt-mark hold docker-ce IP_ADDR=`ip address |grep ens|grep inet|awk '{print $2}'| awk -F / '{print $1}'` HOSTNAME=`hostname` sudo echo "$IP_ADDR $HOSTNAME" >> /etc/hosts docker login -u docker -p docker nexus3.onap.org:10001 sudo apt-get install make -y wget https://storage.googleapis.com/kubernetes-release/release/v$KUBECTL_VERSION/bin/linux/amd64/kubectl chmod +x kubectl sudo mv ./kubectl /usr/local/bin/kubectl sudo apt-get update exit 0
This customization script will:
- Install docker and hold the docker version to 18.09.5
- insert hostname and IP in the hosts file
- install nfs common
- download and install kubectl
Launched k8s instances
Configure Rancher Kubernetes Engine
Install RKE
Download and install RKE on a VM, desktop or laptop. Binaries can be found here for Linux and Mac: https://github.com/rancher/rke/releases/tag/v0.2.1
Perform this on linux based personal laptop, local machine or a VM which has the connectivity to the created VMs on OpenStack and whose key has been configured during creation of VMs.
In our setup, the key of OpenStack controller has been used to login into VMs, so these steps have been performed on OpenStack controller.
mkdir rke cd rke/ wget https://github.com/rancher/rke/releases/download/v0.2.1/rke_linux-amd64 mv rke_linux-amd64 rke chmod 754 rke
RKE requires a cluster.yml as input. An example file is show below that describes a Kubernetes cluster that will be mapped onto the OpenStack VMs created earlier in this guide.
Example: cluster.yml
The example file:
# An example of an HA Kubernetes cluster for ONAP nodes: - address: 10.31.4.11 port: "22" role: - controlplane - etcd hostname_override: "onap-control-1" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.12 port: "22" role: - controlplane - etcd hostname_override: "onap-control-2" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.13 port: "22" role: - controlplane - etcd hostname_override: "onap-control-3" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.21 port: "22" role: - worker hostname_override: "onap-k8s-1" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.22 port: "22" role: - worker hostname_override: "onap-k8s-2" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.23 port: "22" role: - worker hostname_override: "onap-k8s-3" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.24 port: "22" role: - worker hostname_override: "onap-k8s-4" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.25 port: "22" role: - worker hostname_override: "onap-k8s-5" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.26 port: "22" role: - worker hostname_override: "onap-k8s-6" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.27 port: "22" role: - worker hostname_override: "onap-k8s-7" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.28 port: "22" role: - worker hostname_override: "onap-k8s-8" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" services: kube-api: service_cluster_ip_range: 10.43.0.0/16 pod_security_policy: false always_pull_images: false kube-controller: cluster_cidr: 10.42.0.0/16 service_cluster_ip_range: 10.43.0.0/16 kubelet: cluster_domain: cluster.local cluster_dns_server: 10.43.0.10 fail_swap_on: false network: plugin: canal authentication: strategy: x509 ssh_key_path: "~/.ssh/id_ecdsa" ssh_agent_auth: false authorization: mode: rbac ignore_docker_version: false kubernetes_version: "v1.15.11-rancher1-2" private_registries: - url: nexus3.onap.org:10001 user: docker password: docker is_default: true cluster_name: "onap" restore: restore: false snapshot_name: ""
Prepare cluster.yml
Before this configuration file can be used the IP address must be mapped for each control and worker node in this file.
Run RKE
From within the same directory as the cluster.yml file, simply execute:
native@node1-1:~/rke$ ./rke up
./rke up
The output will look something like:
native@node1-1:~/rke$ ./rke up INFO[0000] Initiating Kubernetes cluster INFO[0000] [certificates] Generating CA kubernetes certificates INFO[0000] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates INFO[0000] [certificates] Generating Kubernetes API server certificates INFO[0000] [certificates] Generating Service account token key INFO[0000] [certificates] Generating Kube Controller certificates INFO[0001] [certificates] Generating Node certificate INFO[0001] [certificates] Generating admin certificates and kubeconfig INFO[0001] [certificates] Generating Kubernetes API server proxy client certificates . . . . . . . . . . . . INFO[0309] [addons] Setting up Metrics Server INFO[0309] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes INFO[0309] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes INFO[0309] [addons] Executing deploy job rke-metrics-addon INFO[0315] [addons] Metrics Server deployed successfully INFO[0315] [ingress] Setting up nginx ingress controller INFO[0315] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes INFO[0316] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes INFO[0316] [addons] Executing deploy job rke-ingress-controller INFO[0322] [ingress] ingress controller nginx deployed successfully INFO[0322] [addons] Setting up user addons INFO[0322] [addons] no user addons defined INFO[0322] Finished building Kubernetes cluster successfully
Validate RKE deployment
copy the file "kube_config_cluster.yml" to onap-control-1 VM.
In the VM/system where the rke was executed. In Winlab it is node1-1 (OpenStack controller node).
example: to perform sftp to connect to onap-control-1 node:
native@node1-1:~/rke$ sftp ubuntu@10.31.3.2 Connected to 10.31.3.2. sftp> put kube_config_cluster.yml sftp> exit These steps can also be done using scp as (assuming that the controller has '.kube' folder): native@node1-1:~/rke$ scp kube_config_cluster.yml ubuntu@10.31.3.2:~/.kube/config
login into onap-control-1 VM and move the "kube_config_cluster.yml" file into .kube/ folder and rename the file to "config.onap":
In "onap-control-1" vm:
ubuntu@onap-control-1:~$ mkdir .kube ubuntu@onap-control-1:~$ mv kube_config_cluster.yml .kube/config ubuntu@onap-control-1:~$ kubectl config set-context --current --namespace=onap
Perform these above operations on all control nodes as well as worker nodes to run kubectl and helm commands
mkdir .kube mv kube_config_cluster.yml .kube/config kubectl config set-context --current --namespace=onap
Verify the kubernetes cluster
ubuntu@onap-control-1:~$ kubectl get nodes -o=wide
Result:
Initialize Kubernetes Cluster for use by Helm
Perform this on onap-control-1 VM only during the first setup.
kubectl -n kube-system create serviceaccount tiller kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller helm init --service-account tiller kubectl -n kube-system rollout status deploy/tiller-deploy
Setting up the NFS share for multinode kubernetes cluster:
Deploying applications to a Kubernetes cluster requires Kubernetes nodes to share a common, distributed filesystem. In this tutorial, we will setup an NFS Master, and configure all Worker nodes a Kubernetes cluster to play the role of NFS slaves.
It is recommneded that a separate VM, outside of the kubernetes cluster, be used. This is to ensure that the NFS Master does not compete for resources with Kubernetes Control Plane or Worker Nodes.
Launch new NFS Server VM instance
Select Ubuntu 18.04 as base image
Select Flavor
Networking
Apply customization script for NFS Server VM
Script to be added:
#!/bin/bash DOCKER_VERSION=18.09.5 export DEBIAN_FRONTEND=noninteractive apt-get update curl https://releases.rancher.com/install-docker/$DOCKER_VERSION.sh | sh cat > /etc/docker/daemon.json << EOF { "insecure-registries" : [ "nexus3.onap.org:10001","10.20.6.10:30000" ], "log-driver": "json-file", "log-opts": { "max-size": "1m", "max-file": "9" }, "mtu": 1450, "ipv6": true, "fixed-cidr-v6": "2001:db8:1::/64", "registry-mirrors": ["https://nexus3.onap.org:10001"] } EOF sudo usermod -aG docker ubuntu systemctl daemon-reload systemctl restart docker apt-mark hold docker-ce IP_ADDR=`ip address |grep ens|grep inet|awk '{print $2}'| awk -F / '{print $1}'` HOSTNAME=`hostname` sudo echo "$IP_ADDR $HOSTNAME" >> /etc/hosts docker login -u docker -p docker nexus3.onap.org:10001 sudo apt-get install make -y # install nfs sudo apt-get install nfs-common -y sudo apt update exit 0
This customization script will:
- Install docker and hold the docker version to 18.09.5
- insert hostname and IP address of the onap-nfs-server in the hosts file
- install nfs server
Resulting example
Configure NFS Share on Master node
Login into onap-nfs-server and perform the below commands
Create a master_nfs_node.sh file as below:
#!/bin/bash usage () { echo "Usage:" echo " ./$(basename $0) node1_ip node2_ip ... nodeN_ip" exit 1 } if [ "$#" -lt 1 ]; then echo "Missing NFS slave nodes" usage fi #Install NFS kernel sudo apt-get update sudo apt-get install -y nfs-kernel-server #Create /dockerdata-nfs and set permissions sudo mkdir -p /dockerdata-nfs sudo chmod 777 -R /dockerdata-nfs sudo chown nobody:nogroup /dockerdata-nfs/ #Update the /etc/exports NFS_EXP="" for i in $@; do NFS_EXP+="$i(rw,sync,no_root_squash,no_subtree_check) " done echo "/dockerdata-nfs "$NFS_EXP | sudo tee -a /etc/exports #Restart the NFS service sudo exportfs -a sudo systemctl restart nfs-kernel-server
Make the above created file as executable and run the script in the onap-nfs-server with the IP's of the worker nodes:
chmod +x master_nfs_node.sh sudo ./master_nfs_node.sh {list kubernetes worker nodes ip} example from the WinLab setup: sudo ./master_nfs_node.sh 10.31.3.39 10.31.3.24 10.31.3.52 10.31.3.8 10.31.3.34 10.31.3.47 10.31.3.15 10.31.3.9 10.31.3.5 10.31.3.21 10.31.3.1 10.31.3.68
Login into each kubernetes worker node, i.e. onap-k8s VMs and perform the below commands
Create a slave_nfs_node.sh file as below:
#!/bin/bash usage () { echo "Usage:" echo " ./$(basename $0) nfs_master_ip" exit 1 } if [ "$#" -ne 1 ]; then echo "Missing NFS mater node" usage fi MASTER_IP=$1 #Install NFS common sudo apt-get update sudo apt-get install -y nfs-common #Create NFS directory sudo mkdir -p /dockerdata-nfs #Mount the remote NFS directory to the local one sudo mount $MASTER_IP:/dockerdata-nfs /dockerdata-nfs/ echo "$MASTER_IP:/dockerdata-nfs /dockerdata-nfs nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0" | sudo tee -a /etc/fstab
Make the above created file as executable and run the script in all the worker nodes:
chmod +x slave_nfs_node.sh sudo ./slave_nfs_node.sh {master nfs node IP address} example from the WinLab setup: sudo ./slave_nfs_node.sh 10.31.3.11
ONAP Installation
Perform the following steps in onap-control-1 VM.
Clone the OOM helm repository
Use the master branch as Dublin branch is not available.
Perform these on the home directory
git clone -b <branch> "https://gerrit.onap.org/r/oom" --recurse-submodules mkdir .helm cp -R ~/oom/kubernetes/helm/plugins/ ~/.helm cd oom/kubernetes/sdnc
Edit the values.yaml file
... # Add sdnrwt as true at the end of the config config: ... sdnrwt: true
Add sdnrwt value under config in the file.
Save the file.
navigate to templates folder:
cd templates/
Edit the statefulset.yaml file
... spec: ... template: ... spec: ... containers: - name: {{ include "common.name" . }} ... ... # add sdnrwt flag set to true under env env: ... - name: SDNRWT value: "{{ .Values.config.sdnrwt}}"
Add SDNRWT environment parameter and its value reference.
Save the file.
Edit the service.yaml file
... spec: type: {{ .Values.service.type }} ports: {{if eq .Values.service.type "NodePort" -}} ... - port: {{ .Values.service.externalPort4 }} targetPort: {{ .Values.service.internalPort4 }} nodePort: {{ .Values.global.nodePortPrefix | default .Values.nodePortPrefix }}{{ .Values.service.nodePort4 }} name: "{{ .Values.service.portName }}-8443" {{- else -}} - port: {{ .Values.service.externalPort }} targetPort: {{ .Values.service.internalPort }} name: {{ .Values.service.portName }} {{- end}} selector: app: {{ include "common.name" . }} release: {{ .Release.Name }} type: NodePort sessionAffinity: ClientIP externalTrafficPolicy: Cluster sessionAffinityConfig: clientIP: timeoutSeconds: 10800 status: loadBalancer: {} ---
Append type NodePort, session affinity, session affinity config and load balancer parameters in service.
Copy override files
cd cp -r ~/oom/kubernetes/onap/resources/overrides . cd overrides/
Edit the onap-all.yaml file
# Copyright © 2019 Amdocs, Bell Canada # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. ################################################################### # This override file enables helm charts for all ONAP applications. ################################################################### cassandra: enabled: true mariadb-galera: enabled: true aaf: enabled: true aai: enabled: true appc: enabled: true cds: enabled: true clamp: enabled: true cli: enabled: true consul: enabled: true contrib: enabled: true dcaegen2: enabled: true dmaap: enabled: true esr: enabled: true log: enabled: true sniro-emulator: enabled: true oof: enabled: true msb: enabled: true multicloud: enabled: true nbi: enabled: true policy: enabled: true pomba: enabled: true portal: enabled: true robot: enabled: true sdc: enabled: true sdnc: enabled: true config: sdnrwt: true so: enabled: true uui: enabled: true vfc: enabled: true vid: enabled: true vnfsdk: enabled: true modeling: enabled: true
Save the file.
Start helm server
go to home directory and start helm server and local repository.
cd helm serve &
Hit on ENTER key to come out of helm serve if it shows some logs
Add helm repository
Note the port number that is listed and use it in the Helm repo add as follows
helm repo add local http://127.0.0.1:8879
Verify helm repository
helm repo list
output:
ubuntu@onap-control-1:~$ helm repo list NAME URL stable https://kubernetes-charts.storage.googleapis.com local http://127.0.0.1:8879 ubuntu@onap-control-1:~$
Make onap helm charts available in local helm repository
cd ~/oom/kubernetes make all; make onap
Do not use sudo to perform the above 'make' operation
This take somewhere around 10-15 minutes of time to make all the helm charts and save the charts into local helm repository
output:
ubuntu@onap-control-1:~$ cd ~/oom/kubernetes/ ubuntu@onap-control-1:~/oom/kubernetes$ make all; make onap [common] make[1]: Entering directory '/home/ubuntu/oom/kubernetes' make[2]: Entering directory '/home/ubuntu/oom/kubernetes/common' [common] make[3]: Entering directory '/home/ubuntu/oom/kubernetes/common' ==> Linting common [INFO] Chart.yaml: icon is recommended 1 chart(s) linted, no failures Successfully packaged chart and saved it to: /home/ubuntu/oom/kubernetes/dist/packages/common-4.0.0.tgz make[3]: Leaving directory '/home/ubuntu/oom/kubernetes/common' ... ... ... [onap] make[1]: Entering directory '/home/ubuntu/oom/kubernetes' Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "local" chart repository ...Successfully got an update from the "stable" chart repository Update Complete. ⎈Happy Helming!⎈ Saving 33 charts Downloading aaf from repo http://127.0.0.1:8879 Downloading aai from repo http://127.0.0.1:8879 Downloading appc from repo http://127.0.0.1:8879 Downloading cassandra from repo http://127.0.0.1:8879 Downloading clamp from repo http://127.0.0.1:8879 Downloading cli from repo http://127.0.0.1:8879 Downloading common from repo http://127.0.0.1:8879 Downloading consul from repo http://127.0.0.1:8879 Downloading contrib from repo http://127.0.0.1:8879 Downloading dcaegen2 from repo http://127.0.0.1:8879 Downloading dmaap from repo http://127.0.0.1:8879 Downloading esr from repo http://127.0.0.1:8879 Downloading log from repo http://127.0.0.1:8879 Downloading sniro-emulator from repo http://127.0.0.1:8879 Downloading mariadb-galera from repo http://127.0.0.1:8879 Downloading msb from repo http://127.0.0.1:8879 Downloading multicloud from repo http://127.0.0.1:8879 Downloading nbi from repo http://127.0.0.1:8879 Downloading nfs-provisioner from repo http://127.0.0.1:8879 Downloading pnda from repo http://127.0.0.1:8879 Downloading policy from repo http://127.0.0.1:8879 Downloading pomba from repo http://127.0.0.1:8879 Downloading portal from repo http://127.0.0.1:8879 Downloading oof from repo http://127.0.0.1:8879 Downloading robot from repo http://127.0.0.1:8879 Downloading sdc from repo http://127.0.0.1:8879 Downloading sdnc from repo http://127.0.0.1:8879 Downloading so from repo http://127.0.0.1:8879 Downloading uui from repo http://127.0.0.1:8879 Downloading vfc from repo http://127.0.0.1:8879 Downloading vid from repo http://127.0.0.1:8879 Downloading vnfsdk from repo http://127.0.0.1:8879 Downloading modeling from repo http://127.0.0.1:8879 Deleting outdated charts ==> Linting onap Lint OK 1 chart(s) linted, no failures Successfully packaged chart and saved it to: /home/ubuntu/oom/kubernetes/dist/packages/onap-4.0.0.tgz make[1]: Leaving directory '/home/ubuntu/oom/kubernetes' ubuntu@onap-control-1:~/oom/kubernetes$
Deploy ONAP
The name of the release as 'demo', the namespace as 'onap' and the timeout of 300 seconds to deploy 'dmaap' and 'so' which takes some time to deploy these charts waiting for other components.
Perform these below commands in onap-control-1 where git was cloned.
helm deploy demo local/onap --namespace onap -f ~/overrides/onap-all.yaml -f ~/overrides/environment.yaml --timeout 900
Takes some 60-70 minutes due to added timeout parameter and environment file.
The environment file provides module wise delay required to process the deployment.
The output:
ubuntu@onap-control-1:~/oom/kubernetes$ helm deploy demo local/onap --namespace onap -f ~/overrides/onap-all.yaml --timeout 900 fetching local/onap release "demo" deployed release "demo-aaf" deployed release "demo-aai" deployed release "demo-appc" deployed release "demo-cassandra" deployed release "demo-cds" deployed release "demo-clamp" deployed release "demo-cli" deployed release "demo-consul" deployed release "demo-contrib" deployed release "demo-dcaegen2" deployed release "demo-dmaap" deployed release "demo-esr" deployed release "demo-log" deployed release "demo-mariadb-galera" deployed release "demo-modeling" deployed release "demo-msb" deployed release "demo-multicloud" deployed release "demo-nbi" deployed release "demo-oof" deployed release "demo-policy" deployed release "demo-pomba" deployed release "demo-portal" deployed release "demo-robot" deployed release "demo-sdc" deployed release "demo-sdnc" deployed release "demo-sniro-emulator" deployed release "demo-so" deployed release "demo-uui" deployed release "demo-vfc" deployed release "demo-vid" deployed release "demo-vnfsdk" deployed ubuntu@onap-control-1:~/oom/kubernetes$
Verify the deploy
ubuntu@onap-control-1:~/overrides$ helm ls NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE demo 1 Fri Nov 29 16:26:16 2019 DEPLOYED onap-5.0.0 El Alto onap demo-aaf 1 Fri Nov 29 16:26:17 2019 DEPLOYED aaf-5.0.0 onap demo-aai 1 Fri Nov 29 16:26:28 2019 DEPLOYED aai-5.0.0 onap demo-appc 1 Fri Nov 29 16:27:30 2019 DEPLOYED appc-5.0.0 onap demo-cassandra 1 Fri Nov 29 16:27:41 2019 DEPLOYED cassandra-5.0.0 onap demo-cds 1 Fri Nov 29 16:27:56 2019 DEPLOYED cds-5.0.0 onap demo-clamp 1 Fri Nov 29 16:28:58 2019 DEPLOYED clamp-5.0.0 onap demo-cli 1 Fri Nov 29 16:29:31 2019 DEPLOYED cli-5.0.0 onap demo-consul 1 Fri Nov 29 16:29:45 2019 DEPLOYED consul-5.0.0 onap demo-contrib 1 Fri Nov 29 16:30:04 2019 DEPLOYED contrib-5.0.0 onap demo-dcaegen2 1 Fri Nov 29 16:31:03 2019 DEPLOYED dcaegen2-5.0.0 onap demo-dmaap 1 Fri Nov 29 16:35:47 2019 DEPLOYED dmaap-5.0.0 onap demo-esr 1 Fri Nov 29 16:50:24 2019 DEPLOYED esr-5.0.0 onap demo-log 1 Fri Nov 29 16:50:38 2019 DEPLOYED log-5.0.0 onap demo-mariadb-galera 1 Fri Nov 29 16:51:08 2019 DEPLOYED mariadb-galera-5.0.0 onap demo-modeling 1 Fri Nov 29 16:51:29 2019 DEPLOYED modeling-5.0.0 onap demo-msb 1 Fri Nov 29 16:51:45 2019 DEPLOYED msb-5.0.0 onap demo-multicloud 1 Fri Nov 29 16:52:23 2019 DEPLOYED multicloud-5.0.0 onap demo-nbi 1 Fri Nov 29 16:53:26 2019 DEPLOYED nbi-5.0.0 onap demo-oof 1 Fri Nov 29 16:54:03 2019 DEPLOYED oof-5.0.0 onap demo-policy 1 Fri Nov 29 16:56:27 2019 DEPLOYED policy-5.0.0 onap demo-pomba 1 Fri Nov 29 16:59:13 2019 DEPLOYED pomba-5.0.0 onap demo-portal 1 Fri Nov 29 17:01:44 2019 DEPLOYED portal-5.0.0 onap demo-robot 1 Fri Nov 29 17:03:10 2019 DEPLOYED robot-5.0.0 onap demo-sdc 1 Fri Nov 29 17:03:24 2019 DEPLOYED sdc-5.0.0 onap demo-sdnc 1 Fri Nov 29 17:05:27 2019 DEPLOYED sdnc-5.0.0 onap demo-sniro-emulator 1 Fri Nov 29 17:08:24 2019 DEPLOYED sniro-emulator-5.0.0 onap demo-so 1 Fri Nov 29 17:10:33 2019 DEPLOYED so-5.0.0 onap demo-uui 1 Fri Nov 29 17:24:55 2019 DEPLOYED uui-5.0.0 onap demo-vfc 1 Fri Nov 29 17:24:59 2019 DEPLOYED vfc-5.0.0 onap demo-vid 1 Fri Nov 29 17:26:05 2019 DEPLOYED vid-5.0.0 onap demo-vnfsdk 1 Fri Nov 29 17:26:38 2019 DEPLOYED vnfsdk-5.0.0 onap ubuntu@onap-control-1:~/oom/kubernetes$
In case of failures in deployment
If the deployment of any onap module fails, please go through these steps to redeploy the modules.
In this example, we demonstrate failure of dmaap, which normally occurs due to timeout issues.
Check the failed modules
perform 'helm ls' on the control node.
ubuntu@onap-control-1:~/overrides$ helm ls NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE demo 1 Fri Nov 29 16:26:16 2019 DEPLOYED onap-5.0.0 El Alto onap demo-aaf 1 Fri Nov 29 16:26:17 2019 DEPLOYED aaf-5.0.0 onap demo-aai 1 Fri Nov 29 16:26:28 2019 DEPLOYED aai-5.0.0 onap demo-appc 1 Fri Nov 29 16:27:30 2019 DEPLOYED appc-5.0.0 onap demo-cassandra 1 Fri Nov 29 16:27:41 2019 DEPLOYED cassandra-5.0.0 onap demo-cds 1 Fri Nov 29 16:27:56 2019 DEPLOYED cds-5.0.0 onap demo-clamp 1 Fri Nov 29 16:28:58 2019 DEPLOYED clamp-5.0.0 onap demo-cli 1 Fri Nov 29 16:29:31 2019 DEPLOYED cli-5.0.0 onap demo-consul 1 Fri Nov 29 16:29:45 2019 DEPLOYED consul-5.0.0 onap demo-contrib 1 Fri Nov 29 16:30:04 2019 DEPLOYED contrib-5.0.0 onap demo-dcaegen2 1 Fri Nov 29 16:31:03 2019 DEPLOYED dcaegen2-5.0.0 onap demo-dmaap 1 Mon Dec 2 05:45:05 2019 FAILED dmaap-5.0.0 onap demo-esr 1 Fri Nov 29 16:50:24 2019 DEPLOYED esr-5.0.0 onap demo-log 1 Fri Nov 29 16:50:38 2019 DEPLOYED log-5.0.0 onap demo-mariadb-galera 1 Fri Nov 29 16:51:08 2019 DEPLOYED mariadb-galera-5.0.0 onap demo-modeling 1 Fri Nov 29 16:51:29 2019 DEPLOYED modeling-5.0.0 onap demo-msb 1 Fri Nov 29 16:51:45 2019 DEPLOYED msb-5.0.0 onap demo-multicloud 1 Fri Nov 29 16:52:23 2019 DEPLOYED multicloud-5.0.0 onap demo-nbi 1 Fri Nov 29 16:53:26 2019 DEPLOYED nbi-5.0.0 onap demo-oof 1 Mon Dec 2 07:04:11 2019 DEPLOYED oof-5.0.0 onap demo-policy 1 Fri Nov 29 16:56:27 2019 DEPLOYED policy-5.0.0 onap demo-pomba 1 Fri Nov 29 16:59:13 2019 DEPLOYED pomba-5.0.0 onap demo-portal 1 Fri Nov 29 17:01:44 2019 DEPLOYED portal-5.0.0 onap demo-robot 1 Fri Nov 29 17:03:10 2019 DEPLOYED robot-5.0.0 onap demo-sdc 1 Fri Nov 29 17:03:24 2019 DEPLOYED sdc-5.0.0 onap demo-sdnc 1 Fri Nov 29 17:05:27 2019 DEPLOYED sdnc-5.0.0 onap demo-sniro-emulator 1 Fri Nov 29 17:08:24 2019 DEPLOYED sniro-emulator-5.0.0 onap demo-so 1 Mon Dec 2 08:58:33 2019 DEPLOYED so-5.0.0 onap demo-uui 1 Fri Nov 29 17:24:55 2019 DEPLOYED uui-5.0.0 onap demo-vfc 1 Fri Nov 29 17:24:59 2019 DEPLOYED vfc-5.0.0 onap demo-vid 1 Fri Nov 29 17:26:05 2019 DEPLOYED vid-5.0.0 onap demo-vnfsdk 1 Fri Nov 29 17:26:38 2019 DEPLOYED vnfsdk-5.0.0 onap
Delete the failed module
use the right release name from the name as shown in helm ls.
The below commands will delete all the pods and dependencies.
helm delete demo-dmaap --purge kubectl get persistentvolumeclaims -n onap | grep demo-dmaap | sed -r 's/(^[^ ]+).*/kubectl delete persistentvolumeclaims -n onap \1/' | bash kubectl get persistentvolumes -n onap | grep demo-dmaap | sed -r 's/(^[^ ]+).*/kubectl delete persistentvolumes -n onap \1/' | bash kubectl get secrets -n onap | grep demo-dmaap | sed -r 's/(^[^ ]+).*/kubectl delete secrets -n onap \1/' | bash kubectl get clusterrolebindings -n onap | grep demo-dmaap | sed -r 's/(^[^ ]+).*/kubectl delete clusterrolebindings -n onap \1/' | bash kubectl get jobs -n onap | grep demo-dmaap | sed -r 's/(^[^ ]+).*/kubectl delete jobs -n onap \1/' | bash kubectl get pods -n onap | grep demo-dmaap | sed -r 's/(^[^ ]+).*/kubectl delete pods -n onap \1/' | bash
Delete any residue data from onap-nfs-server.
cd /dockerdata-nfs/ sudo rm -r demo-dmaap/
Reinstall module
Reinstall the deleted module with the same release name as used in the deletion
helm install --namespace onap -n demo-dmaap local/dmaap --timeout 900
You can verify the deployment in a parallel terminal by checking the pods using the command
kubectl get pods -n onap | grep demo-dmaap
This shall show you the status of the upcoming pods during the deployment wait time.
Once this is deployed, you can verify using the "helm ls" command to check all the required modules are up and running.
Undeploy ONAP
For the release name 'demo' and namespace 'onap':
Perform these below commands in onap-control-1 where the helm was deployed.
helm undeploy demo --purge kubectl get persistentvolumeclaims -n onap | grep demo | sed -r 's/(^[^ ]+).*/kubectl delete persistentvolumeclaims -n onap \1/' | bash kubectl get persistentvolumes -n onap | grep demo | sed -r 's/(^[^ ]+).*/kubectl delete persistentvolumes -n onap \1/' | bash kubectl get secrets -n onap | grep demo | sed -r 's/(^[^ ]+).*/kubectl delete secrets -n onap \1/' | bash kubectl get clusterrolebindings -n onap | grep demo | sed -r 's/(^[^ ]+).*/kubectl delete clusterrolebindings -n onap \1/' | bash kubectl get jobs -n onap | grep demo | sed -r 's/(^[^ ]+).*/kubectl delete jobs -n onap \1/' | bash kubectl get pods -n onap | grep demo | sed -r 's/(^[^ ]+).*/kubectl delete pods -n onap \1/' | bash
Perform the below commands in onap-nfs-server.
This is to delete all the network shared data which will not be deleted on purge delete of helm release.
cd /dockerdata-nfs/ sudo rm -r * e.g. ubuntu@onap-nfs-server:~$ cd /dockerdata-nfs/ ubuntu@onap-nfs-server:/dockerdata-nfs$ sudo rm -r *
Access SDN-R
ODLUX-GUI at winlab
The {user} should be replaced by your user id created in orbit-lab.
ssh -A -t {user}@console.sb10.orbit-lab.org -L 31939:localhost:31939 ssh -A -t native@node1-1.sb10.orbit-lab.org -L 31939:localhost:31939 ssh -A -t ubuntu@10.31.3.2 -L 31939:localhost:31939
On your browser:
http://localhost:31939/odlux/index.html
The login credentials are:
username: admin
password: Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U
ODLUX-GUI at your labs
Check the IP on the sdnc service bound to port 8282:
ubuntu@onap-control-1:~$ kubectl get service | grep sdnc pomba-sdncctxbuilder ClusterIP 10.43.145.187 <none> 9530/TCP 2d18h sdnc NodePort 10.43.103.106 <none> 8282:31939/TCP,8202:31510/TCP,8280:30246/TCP,8443:30267/TCP 7m52s sdnc-ansible-server ClusterIP 10.43.113.226 <none> 8000/TCP 7m52s sdnc-cluster ClusterIP None <none> 2550/TCP 7m52s sdnc-dgbuilder NodePort 10.43.62.50 <none> 3000:30203/TCP 7m52s sdnc-dmaap-listener ClusterIP None <none> <none> 7m52s sdnc-portal NodePort 10.43.116.73 <none> 8443:30201/TCP 7m52s sdnc-ueb-listener ClusterIP None <none> <none> 7m52s so-sdnc-adapter ClusterIP 10.43.27.8 <none> 8086/TCP 142m vfc-zte-sdnc-driver ClusterIP 10.43.189.170 <none> 8411/TCP 2d17h
Look for sdnc service only and the port bound to 8282.
In the above result it is 31939.
http://{ip-address}:{sdnc-port-bound-to-8282}/odlux/index.html
The {ip-address} is the IP address of the onap-control-1 machine.
Troubleshooting
Documentation for troubleshooting.
ES-DB index read only – NFS out of disc space
ONAP log (elastic DB) consumes approx 1 GB per day. If disc space crosses a threshold, indindices of SDNC elastic DB are set to read-only.
** FORBIDDEN/12/index read-only / allow delete (api)] **
Housekeeping of elastic db can be done by Curator.
install curator
pip install elasticsearch-curator
#show indices with size curator_cli show_indices --verbose curator_cli show_indices --verbose|wc -l # delete older indices # test with dry run curator --dry-run delete_index.yml # delete! curator delete_index.yml
delete_index.yml
--- actions: 1: action: delete_indices description: >- Delete indices older than 45 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. options: ignore_empty_list: True timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: prefix value: logstash- exclude: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 30 exclude:
configuration of curator:
/home/ubuntu/.curator/curator.yml for curator-cli
/etc/curator/config.yaml for curator
client: hosts: - 127.0.0.1 port: 30254 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False http_auth: timeout: 30 master_only: False logging: loglevel: INFO logfile:
Define as a cron job (e.g. every 24h)
00 6 * * * root curator /home/ubuntu/curator/delete-indices.yml — config /etc/curator/config.yml
Set SDNC elastic DB indices back to read-write
PUT _settings { "index": { "blocks": { "read_only_allow_delete": "false" } } }