You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 85 Next »

Overview

For ONAP SDN-R load and stress test and Proof of concept, June 19 a three node SDN-R cluster is used. Version is El Alto.

Precondition

  • Running physical server cluster (The hardware requirements)
  • OpenStack installed
  • Virtual Machine cloud image Ubuntu 18.04-server-image

Goal of this description is to setup a complete ONAP with focus of SDNC services

Steps

  1. Installing Docker, Kubernetes and Rancher
  2. Initiation of HELM
  3. Downloading OOM charts
  4. Installation of all OOM based ONAP componnents
  5. Installation of Device Simulators


Installing Kubernetes and Rancher

Create the Rancher 3 nodes control cluster named as onap-control on OpenStack

The following instructions describe how to create 3 OpenStack VMs to host the Highly-Available Kubernetes Control Plane. ONAP workloads will not be scheduled on these Control Plane nodes.

Launch new VMs in OpenStack.

Select Ubuntu 18.04 as the boot image for the virtual machine without any volume


Select m1.large flavor

Networking

Apply customization script for Control Plane VMs


The script to be copied:

#!/bin/bash

DOCKER_VERSION=18.09.5
KUBECTL_VERSION=1.13.5 
HELM_VERSION=2.12.3
export DEBIAN_FRONTEND=noninteractive
sudo apt-get update

curl https://releases.rancher.com/install-docker/$DOCKER_VERSION.sh | sh
mkdir -p /etc/systemd/system/docker.service.d/
cat > /etc/systemd/system/docker.service.d/docker.conf << EOF
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry=nexus3.onap.org:10001
EOF

sudo usermod -aG docker ubuntu

systemctl daemon-reload
systemctl restart docker
apt-mark hold docker-ce

IP_ADDR=`ip address |grep ens|grep inet|awk '{print $2}'| awk -F / '{print $1}'`

HOSTNAME=`hostname`

sudo echo "$IP_ADDR $HOSTNAME" >> /etc/hosts

docker login -u docker -p docker nexus3.onap.org:10001

sudo apt-get install make -y

wget https://storage.googleapis.com/kubernetes-release/release/v$KUBECTL_VERSION/bin/linux/amd64/kubectl

chmod +x kubectl

sudo mv ./kubectl /usr/local/bin/kubectl


wget http://storage.googleapis.com/kubernetes-helm/helm-v${HELM_VERSION}-linux-amd64.tar.gz

tar -zxvf helm-v${HELM_VERSION}-linux-amd64.tar.gz

sudo mv linux-amd64/helm /usr/local/bin/helm

sudo apt-get update

exit 0

This customization script will:

  • Install docker and hold the docker version to 18.09.5
  • set hosts file with IP and hostname
  • install make
  • download and install kubectl
  • download and install helm

Launched Instances

Create the Kubernetes 12 nodes worker cluster named as onap-k8s on OpenStack cloud

The following instructions describe how to create OpenStack VMs to host the Highly-Available Kubernetes Workers. ONAP workloads will only be scheduled on these nodes.

Launch new VM instances in OpenStack

Select Ubuntu 18.04 as base image

Select Flavor

The size of Kubernetes hosts depend on the size of the ONAP deployment being installed.

If a small subset of ONAP applications are being deployed (i.e. for testing purposes), then 16GB or 32GB may be sufficient.

Networking

Apply customization script for Kubernetes VM(s)

The scrip to be copied:

#!/bin/bash

DOCKER_VERSION=18.09.5
KUBECTL_VERSION=1.13.5
export DEBIAN_FRONTEND=noninteractive
sudo apt-get update

curl https://releases.rancher.com/install-docker/$DOCKER_VERSION.sh | sh
mkdir -p /etc/systemd/system/docker.service.d/
cat > /etc/systemd/system/docker.service.d/docker.conf << EOF
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry=nexus3.onap.org:10001
EOF

sudo usermod -aG docker ubuntu

systemctl daemon-reload

systemctl restart docker

apt-mark hold docker-ce

IP_ADDR=`ip address |grep ens|grep inet|awk '{print $2}'| awk -F / '{print $1}'`

HOSTNAME=`hostname`

sudo echo "$IP_ADDR $HOSTNAME" >> /etc/hosts

docker login -u docker -p docker nexus3.onap.org:10001

sudo apt-get install make -y

wget https://storage.googleapis.com/kubernetes-release/release/v$KUBECTL_VERSION/bin/linux/amd64/kubectl

chmod +x kubectl

sudo mv ./kubectl /usr/local/bin/kubectl


sudo apt-get update

exit 0

This customization script will:

  • Install docker and hold the docker version to 18.09.5
  • insert hostname and IP in the hosts file
  • install nfs common
  • download and install kubectl


Launched k8s instances

Configure Rancher Kubernetes Engine

Install RKE

Download and install RKE on a VM, desktop or laptop. Binaries can be found here for Linux and Mac: https://github.com/rancher/rke/releases/tag/v0.2.1

Perform this on linux based personal laptop, local machine or a VM which has the connectivity to the created VMs on OpenStack and whose key has been configured during creation of VMs.

In our setup, the key of OpenStack controller has been used to login into VMs, so these steps have been performed on OpenStack controller.

mkdir rke
cd rke/
wget https://github.com/rancher/rke/releases/download/v0.2.1/rke_linux-amd64
mv rke_linux-amd64 rke
chmod 754 rke

RKE requires a cluster.yml as input. An example file is show below that describes a Kubernetes cluster that will be mapped onto the OpenStack VMs created earlier in this guide.

Example: cluster.yml

The example file:

# An example of an HA Kubernetes cluster for ONAP
nodes:
- address: 10.31.3.2
  port: "22"
  role:
  - controlplane
  - etcd
  hostname_override: "onap-control-1"
  user: ubuntu
  ssh_key_path: "~/.ssh/id_rsa"
- address: 10.31.3.3
  port: "22"
  role:
  - controlplane
  - etcd
  hostname_override: "onap-control-2"
  user: ubuntu
  ssh_key_path: "~/.ssh/id_rsa"
- address: 10.31.3.16
  port: "22"
  role:
  - controlplane
  - etcd
  hostname_override: "onap-control-3"
  user: ubuntu
  ssh_key_path: "~/.ssh/id_rsa"
- address: 10.31.3.39
  port: "22"
  role:
  - worker
  hostname_override: "onap-k8s-1"
  user: ubuntu
  ssh_key_path: "~/.ssh/id_rsa"
- address: 10.31.3.24
  port: "22"
  role:
  - worker
  hostname_override: "onap-k8s-2"
  user: ubuntu
  ssh_key_path: "~/.ssh/id_rsa"
- address: 10.31.3.52
  port: "22"
  role:
  - worker
  hostname_override: "onap-k8s-3"
  user: ubuntu
  ssh_key_path: "~/.ssh/id_rsa"
- address: 10.31.3.8
  port: "22"
  role:
  - worker
  hostname_override: "onap-k8s-4"
  user: ubuntu
  ssh_key_path: "~/.ssh/id_rsa"
- address: 10.31.3.34
  port: "22"
  role:
  - worker
  hostname_override: "onap-k8s-5"
  user: ubuntu
  ssh_key_path: "~/.ssh/id_rsa"
- address: 10.31.3.47
  port: "22"
  role:
  - worker
  hostname_override: "onap-k8s-6"
  user: ubuntu
  ssh_key_path: "~/.ssh/id_rsa"
- address: 10.31.3.15
  port: "22"
  role:
  - worker
  hostname_override: "onap-k8s-7"
  user: ubuntu
  ssh_key_path: "~/.ssh/id_rsa"
- address: 10.31.3.9
  port: "22"
  role:
  - worker
  hostname_override: "onap-k8s-8"
  user: ubuntu
  ssh_key_path: "~/.ssh/id_rsa"
- address: 10.31.3.5
  port: "22"
  role:
  - worker
  hostname_override: "onap-k8s-9"
  user: ubuntu
  ssh_key_path: "~/.ssh/id_rsa"
- address: 10.31.3.21
  port: "22"
  role:
  - worker
  hostname_override: "onap-k8s-10"
  user: ubuntu
  ssh_key_path: "~/.ssh/id_rsa"
- address: 10.31.3.1
  port: "22"
  role:
  - worker
  hostname_override: "onap-k8s-11"
  user: ubuntu
  ssh_key_path: "~/.ssh/id_rsa"
- address: 10.31.3.68
  port: "22"
  role:
  - worker
  hostname_override: "onap-k8s-12"
  user: ubuntu
  ssh_key_path: "~/.ssh/id_rsa"
services:
  kube-api:
    service_cluster_ip_range: 10.43.0.0/16
    pod_security_policy: false
    always_pull_images: false
  kube-controller:
    cluster_cidr: 10.42.0.0/16
    service_cluster_ip_range: 10.43.0.0/16
  kubelet:
    cluster_domain: cluster.local
    cluster_dns_server: 10.43.0.10
    fail_swap_on: false
network:
  plugin: canal
authentication:
  strategy: x509
ssh_key_path: "~/.ssh/id_rsa"
ssh_agent_auth: false
authorization:
  mode: rbac
ignore_docker_version: false
kubernetes_version: "v1.13.5-rancher1-2"
private_registries:
- url: nexus3.onap.org:10001
  user: docker
  password: docker
  is_default: true
cluster_name: "onap"
restore:
  restore: false
  snapshot_name: ""

Prepare cluster.yml

Before this configuration file can be used the IP address must be mapped for each control and worker node in this file.

Run RKE

From within the same directory as the cluster.yml file, simply execute:

native@node1-1:~/rke$ ./rke up

./rke up

The output will look something like:

native@node1-1:~/rke$ ./rke up
INFO[0000] Initiating Kubernetes cluster                
INFO[0000] [certificates] Generating CA kubernetes certificates 
INFO[0000] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates 
INFO[0000] [certificates] Generating Kubernetes API server certificates 
INFO[0000] [certificates] Generating Service account token key 
INFO[0000] [certificates] Generating Kube Controller certificates 
INFO[0001] [certificates] Generating Node certificate   
INFO[0001] [certificates] Generating admin certificates and kubeconfig 
INFO[0001] [certificates] Generating Kubernetes API server proxy client certificates
. . . .
. . . .
. . . .
INFO[0309] [addons] Setting up Metrics Server
INFO[0309] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes
INFO[0309] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes
INFO[0309] [addons] Executing deploy job rke-metrics-addon
INFO[0315] [addons] Metrics Server deployed successfully
INFO[0315] [ingress] Setting up nginx ingress controller
INFO[0315] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes
INFO[0316] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes
INFO[0316] [addons] Executing deploy job rke-ingress-controller
INFO[0322] [ingress] ingress controller nginx deployed successfully
INFO[0322] [addons] Setting up user addons
INFO[0322] [addons] no user addons defined
INFO[0322] Finished building Kubernetes cluster successfully

Validate RKE deployment

copy the file "kube_config_cluster.yml" to onap-control-1 VM.

In the VM/system where the rke was executed. In Winlab it is node1-1 (OpenStack controller node).

example: to perform sftp to connect to onap-control-1 node:

native@node1-1:~/rke$ sftp ubuntu@10.31.3.2
Connected to 10.31.3.2.
sftp> put kube_config_cluster.yml
sftp> exit

These steps can also be done using scp as (assuming that the controller has '.kube' folder):
native@node1-1:~/rke$ scp kube_config_cluster.yml ubuntu@10.31.3.2:~/.kube/config

login into onap-control-1 VM and move the "kube_config_cluster.yml" file into .kube/ folder and rename the file to "config.onap":

In "onap-control-1" vm:

ubuntu@onap-control-1:~$ mkdir .kube
ubuntu@onap-control-1:~$ mv kube_config_cluster.yml .kube/config
ubuntu@onap-control-1:~$ kubectl config set-context --current --namespace=onap

Perform these above operations on all control nodes as well as worker nodes to run kubectl and helm commands

mkdir .kube
mv kube_config_cluster.yml .kube/config
kubectl config set-context --current --namespace=onap

Verify the kubernetes cluster

ubuntu@onap-control-1:~$ kubectl get nodes -o=wide

Result:

Initialize Kubernetes Cluster for use by Helm

Perform this on onap-control-1 VM only during the first setup.

kubectl -n kube-system create serviceaccount tiller
kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account tiller
kubectl -n kube-system  rollout status deploy/tiller-deploy

Setting up the NFS share for multinode kubernetes cluster:

Deploying applications to a Kubernetes cluster requires Kubernetes nodes to share a common, distributed filesystem. In this tutorial, we will setup an NFS Master, and configure all Worker nodes a Kubernetes cluster to play the role of NFS slaves.

It is recommneded that a separate VM, outside of the kubernetes cluster, be used. This is to ensure that the NFS Master does not compete for resources with Kubernetes Control Plane or Worker Nodes.

Launch new NFS Server VM instance

Select Ubuntu 18.04 as base image

Select Flavor

Networking

Apply customization script for NFS Server VM

Script to be added:

#!/bin/bash

DOCKER_VERSION=18.09.5

apt-get update

curl https://releases.rancher.com/install-docker/$DOCKER_VERSION.sh | sh
mkdir -p /etc/systemd/system/docker.service.d/
cat > /etc/systemd/system/docker.service.d/docker.conf << EOF
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry=nexus3.onap.org:10001
EOF

sudo usermod -aG docker ubuntu

systemctl daemon-reload
systemctl restart docker
apt-mark hold docker-ce

IP_ADDR=`ip address |grep ens|grep inet|awk '{print $2}'| awk -F / '{print $1}'`

HOSTNAME=`hostname`

sudo echo "$IP_ADDR $HOSTNAME" >> /etc/hosts

docker login -u docker -p docker nexus3.onap.org:10001

sudo apt-get install make -y

# install nfs
sudo apt-get install nfs-common -y

sudo apt update


exit 0

This customization script will:

  • Install docker and hold the docker version to 18.09.5
  • insert hostname and IP address of the onap-nfs-server in the hosts file
  • install nfs server

Resulting example


Configure NFS Share on Master node

Login into onap-nfs-server and perform the below commands

Create a master_nfs_node.sh file as below:

master_nfs_node.sh
#!/bin/bash

usage () {
  echo "Usage:"
  echo "   ./$(basename $0) node1_ip node2_ip ... nodeN_ip"
  exit 1
}

if [ "$#" -lt 1 ]; then
  echo "Missing NFS slave nodes"
  usage
fi

#Install NFS kernel
sudo apt-get update
sudo apt-get install -y nfs-kernel-server

#Create /dockerdata-nfs and set permissions
sudo mkdir -p /dockerdata-nfs
sudo chmod 777 -R /dockerdata-nfs
sudo chown nobody:nogroup /dockerdata-nfs/

#Update the /etc/exports
NFS_EXP=""
for i in $@; do
  NFS_EXP+="$i(rw,sync,no_root_squash,no_subtree_check) "
done
echo "/dockerdata-nfs "$NFS_EXP | sudo tee -a /etc/exports

#Restart the NFS service
sudo exportfs -a
sudo systemctl restart nfs-kernel-server

Make the above created file as executable and run the script in the onap-nfs-server with the IP's of the worker nodes:

chmod +x master_nfs_node.sh
sudo ./master_nfs_node.sh {list kubernetes worker nodes ip}
example from the WinLab setup:
sudo ./master_nfs_node.sh 10.31.3.39 10.31.3.24 10.31.3.52 10.31.3.8 10.31.3.34 10.31.3.47 10.31.3.15 10.31.3.9 10.31.3.5 10.31.3.21 10.31.3.1 10.31.3.68

Login into each kubernetes worker node, i.e. onap-k8s VMs and perform the below commands

Create a slave_nfs_node.sh file as below:

slave_nfs_node.sh
#!/bin/bash

usage () {
  echo "Usage:"
  echo "   ./$(basename $0) nfs_master_ip"
  exit 1
}

if [ "$#" -ne 1 ]; then
  echo "Missing NFS mater node"
  usage
fi

MASTER_IP=$1

#Install NFS common
sudo apt-get update
sudo apt-get install -y nfs-common

#Create NFS directory
sudo mkdir -p /dockerdata-nfs

#Mount the remote NFS directory to the local one
sudo mount $MASTER_IP:/dockerdata-nfs /dockerdata-nfs/
echo "$MASTER_IP:/dockerdata-nfs /dockerdata-nfs  nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0" | sudo tee -a /etc/fstab

Make the above created file as executable and run the script in all the worker nodes:

chmod +x slave_nfs_node.sh
sudo ./slave_nfs_node.sh {master nfs node IP address}
example from the WinLab setup:
sudo ./slave_nfs_node.sh 10.31.3.11

ONAP Installation

Perform the following steps in onap-control-1 VM.

Clone the OOM helm repository

Use the master branch as Dublin branch is not available.

Perform these on the home directory

git clone http://gerrit.onap.org/r/oom --recurse-submodules
mkdir .helm
cp -R ~/oom/kubernetes/helm/plugins/ ~/.helm
cd oom/kubernetes/sdnc

Edit the values.yaml file

...
# Add sdnrwt as true at the end of the config
config:
  ...
  sdnrwt: true

Add sdnrwt value under config in the file.

Save the file.

navigate to templates folder:

cd templates/

Edit the statefulset.yaml file

...
spec:
  ...
  template:
    ...
    spec:
      ...
      containers:
        - name: {{ include "common.name" . }}
          ...
          ...
          # add sdnrwt flag set to true under env
          env:
            ...
            - name: SDNRWT
              value: "{{ .Values.config.sdnrwt}}"

Add SDNRWT environment parameter and its value reference.

Save the file.

Edit the service.yaml file

...
spec:
  type: {{ .Values.service.type }}
  ports:
    {{if eq .Values.service.type "NodePort" -}}
    ...
    - port: {{ .Values.service.externalPort4 }}
      targetPort: {{ .Values.service.internalPort4 }}
      nodePort: {{ .Values.global.nodePortPrefix | default .Values.nodePortPrefix }}{{ .Values.service.nodePort4 }}
      name: "{{ .Values.service.portName }}-8443"
    {{- else -}}
    - port: {{ .Values.service.externalPort }}
      targetPort: {{ .Values.service.internalPort }}
      name: {{ .Values.service.portName }}
    {{- end}}
  selector:
    app: {{ include "common.name" . }}
    release: {{ .Release.Name }}
  type: NodePort
  sessionAffinity: ClientIP
  externalTrafficPolicy: Cluster
  sessionAffinityConfig:
    clientIP:
     timeoutSeconds: 10800
status:
  loadBalancer: {}

---

Append type NodePort,  session affinity, session affinity config and load balancer parameters in service.

Copy override files

cd
cp -r ~/oom/kubernetes/onap/resources/overrides .
cd overrides/

Edit the onap-all.yaml file

# Copyright © 2019 Amdocs, Bell Canada
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

###################################################################
# This override file enables helm charts for all ONAP applications.
###################################################################
cassandra:
  enabled: true
mariadb-galera:
  enabled: true

aaf:
  enabled: true
aai:
  enabled: true
appc:
  enabled: true
cds:
  enabled: true
clamp:
  enabled: true
cli:
  enabled: true
consul:
  enabled: true
contrib:
  enabled: true
dcaegen2:
  enabled: true
dmaap:
  enabled: true
esr:
  enabled: true
log:
  enabled: true
sniro-emulator:
  enabled: true
oof:
  enabled: true
msb:
  enabled: true
multicloud:
  enabled: true
nbi:
  enabled: true
policy:
  enabled: true
pomba:
  enabled: true
portal:
  enabled: true
robot:
  enabled: true
sdc:
  enabled: true
sdnc:
  enabled: true
  config:
    sdnrwt: true
so:
  enabled: true
uui:
  enabled: true
vfc:
  enabled: true
vid:
  enabled: true
vnfsdk:
  enabled: true
modeling:
  enabled: true

Save the file.

Start helm server

go to home directory and start helm server and local repository.

cd
helm serve &

Hit on ENTER key to come out of helm serve if it shows some logs

Add helm repository

Note the port number that is listed and use it in the Helm repo add as follows

helm repo add local http://127.0.0.1:8879

Verify helm repository

helm repo list

output:

ubuntu@onap-control-1:~$ helm repo list
NAME  	URL                                             
stable	https://kubernetes-charts.storage.googleapis.com
local 	http://127.0.0.1:8879                           
ubuntu@onap-control-1:~$

Make onap helm charts available in local helm repository

cd ~/oom/kubernetes
make all; make onap

Do not use sudo to perform the above 'make' operation

This take somewhere around 10-15 minutes of time to make all the helm charts and save the charts into local helm repository

output:

ubuntu@onap-control-1:~$ cd ~/oom/kubernetes/
ubuntu@onap-control-1:~/oom/kubernetes$ make all; make onap

[common]
make[1]: Entering directory '/home/ubuntu/oom/kubernetes'
make[2]: Entering directory '/home/ubuntu/oom/kubernetes/common'

[common]
make[3]: Entering directory '/home/ubuntu/oom/kubernetes/common'
==> Linting common
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, no failures
Successfully packaged chart and saved it to: /home/ubuntu/oom/kubernetes/dist/packages/common-4.0.0.tgz
make[3]: Leaving directory '/home/ubuntu/oom/kubernetes/common'
...
...
...
[onap]
make[1]: Entering directory '/home/ubuntu/oom/kubernetes'
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "local" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 33 charts
Downloading aaf from repo http://127.0.0.1:8879
Downloading aai from repo http://127.0.0.1:8879
Downloading appc from repo http://127.0.0.1:8879
Downloading cassandra from repo http://127.0.0.1:8879
Downloading clamp from repo http://127.0.0.1:8879
Downloading cli from repo http://127.0.0.1:8879
Downloading common from repo http://127.0.0.1:8879
Downloading consul from repo http://127.0.0.1:8879
Downloading contrib from repo http://127.0.0.1:8879
Downloading dcaegen2 from repo http://127.0.0.1:8879
Downloading dmaap from repo http://127.0.0.1:8879
Downloading esr from repo http://127.0.0.1:8879
Downloading log from repo http://127.0.0.1:8879
Downloading sniro-emulator from repo http://127.0.0.1:8879
Downloading mariadb-galera from repo http://127.0.0.1:8879
Downloading msb from repo http://127.0.0.1:8879
Downloading multicloud from repo http://127.0.0.1:8879
Downloading nbi from repo http://127.0.0.1:8879
Downloading nfs-provisioner from repo http://127.0.0.1:8879
Downloading pnda from repo http://127.0.0.1:8879
Downloading policy from repo http://127.0.0.1:8879
Downloading pomba from repo http://127.0.0.1:8879
Downloading portal from repo http://127.0.0.1:8879
Downloading oof from repo http://127.0.0.1:8879
Downloading robot from repo http://127.0.0.1:8879
Downloading sdc from repo http://127.0.0.1:8879
Downloading sdnc from repo http://127.0.0.1:8879
Downloading so from repo http://127.0.0.1:8879
Downloading uui from repo http://127.0.0.1:8879
Downloading vfc from repo http://127.0.0.1:8879
Downloading vid from repo http://127.0.0.1:8879
Downloading vnfsdk from repo http://127.0.0.1:8879
Downloading modeling from repo http://127.0.0.1:8879
Deleting outdated charts
==> Linting onap
Lint OK

1 chart(s) linted, no failures
Successfully packaged chart and saved it to: /home/ubuntu/oom/kubernetes/dist/packages/onap-4.0.0.tgz
make[1]: Leaving directory '/home/ubuntu/oom/kubernetes'
ubuntu@onap-control-1:~/oom/kubernetes$

Deploy ONAP

The name of the release as 'demo', the namespace as 'onap' and the timeout of 300 seconds to deploy 'dmaap' and 'so' which takes some time to deploy these charts waiting for other components.

Perform these below commands in onap-control-1 where git was cloned.

helm deploy demo local/onap --namespace onap -f ~/overrides/onap-all.yaml -f ~/overrides/environment.yaml --timeout 900

Takes some 60-70 minutes due to added timeout parameter and environment file.

The environment file provides module wise delay required to process the deployment.

The output:

ubuntu@onap-control-1:~/oom/kubernetes$ helm deploy demo local/onap --namespace onap -f ~/overrides/onap-all.yaml --timeout 900
fetching local/onap
release "demo" deployed
release "demo-aaf" deployed
release "demo-aai" deployed
release "demo-appc" deployed
release "demo-cassandra" deployed
release "demo-cds" deployed
release "demo-clamp" deployed
release "demo-cli" deployed
release "demo-consul" deployed
release "demo-contrib" deployed
release "demo-dcaegen2" deployed
release "demo-dmaap" deployed
release "demo-esr" deployed
release "demo-log" deployed
release "demo-mariadb-galera" deployed
release "demo-modeling" deployed
release "demo-msb" deployed
release "demo-multicloud" deployed
release "demo-nbi" deployed
release "demo-oof" deployed
release "demo-policy" deployed
release "demo-pomba" deployed
release "demo-portal" deployed
release "demo-robot" deployed
release "demo-sdc" deployed
release "demo-sdnc" deployed
release "demo-sniro-emulator" deployed
release "demo-so" deployed
release "demo-uui" deployed
release "demo-vfc" deployed
release "demo-vid" deployed
release "demo-vnfsdk" deployed
ubuntu@onap-control-1:~/oom/kubernetes$

Verify the deploy

ubuntu@onap-control-1:~/overrides$ helm ls
NAME               	REVISION	UPDATED                 	STATUS  	CHART               	APP VERSION	NAMESPACE
demo               	1       	Fri Nov 29 16:26:16 2019	DEPLOYED	onap-5.0.0          	El Alto    	onap     
demo-aaf           	1       	Fri Nov 29 16:26:17 2019	DEPLOYED	aaf-5.0.0           	           	onap     
demo-aai           	1       	Fri Nov 29 16:26:28 2019	DEPLOYED	aai-5.0.0           	           	onap     
demo-appc          	1       	Fri Nov 29 16:27:30 2019	DEPLOYED	appc-5.0.0          	           	onap     
demo-cassandra     	1       	Fri Nov 29 16:27:41 2019	DEPLOYED	cassandra-5.0.0     	           	onap     
demo-cds           	1       	Fri Nov 29 16:27:56 2019	DEPLOYED	cds-5.0.0           	           	onap     
demo-clamp         	1       	Fri Nov 29 16:28:58 2019	DEPLOYED	clamp-5.0.0         	           	onap     
demo-cli           	1       	Fri Nov 29 16:29:31 2019	DEPLOYED	cli-5.0.0           	           	onap     
demo-consul        	1       	Fri Nov 29 16:29:45 2019	DEPLOYED	consul-5.0.0        	           	onap     
demo-contrib       	1       	Fri Nov 29 16:30:04 2019	DEPLOYED	contrib-5.0.0       	           	onap     
demo-dcaegen2      	1       	Fri Nov 29 16:31:03 2019	DEPLOYED	dcaegen2-5.0.0      	           	onap     
demo-dmaap         	1       	Fri Nov 29 16:35:47 2019	DEPLOYED  	dmaap-5.0.0         	           	onap     
demo-esr           	1       	Fri Nov 29 16:50:24 2019	DEPLOYED	esr-5.0.0           	           	onap     
demo-log           	1       	Fri Nov 29 16:50:38 2019	DEPLOYED	log-5.0.0           	           	onap     
demo-mariadb-galera	1       	Fri Nov 29 16:51:08 2019	DEPLOYED	mariadb-galera-5.0.0	           	onap     
demo-modeling      	1       	Fri Nov 29 16:51:29 2019	DEPLOYED	modeling-5.0.0      	           	onap     
demo-msb           	1       	Fri Nov 29 16:51:45 2019	DEPLOYED	msb-5.0.0           	           	onap     
demo-multicloud    	1       	Fri Nov 29 16:52:23 2019	DEPLOYED	multicloud-5.0.0    	           	onap     
demo-nbi           	1       	Fri Nov 29 16:53:26 2019	DEPLOYED	nbi-5.0.0           	           	onap     
demo-oof           	1       	Fri Nov 29 16:54:03 2019	DEPLOYED	oof-5.0.0           	           	onap     
demo-policy        	1       	Fri Nov 29 16:56:27 2019	DEPLOYED	policy-5.0.0        	           	onap     
demo-pomba         	1       	Fri Nov 29 16:59:13 2019	DEPLOYED	pomba-5.0.0         	           	onap     
demo-portal        	1       	Fri Nov 29 17:01:44 2019	DEPLOYED	portal-5.0.0        	           	onap     
demo-robot         	1       	Fri Nov 29 17:03:10 2019	DEPLOYED	robot-5.0.0         	           	onap     
demo-sdc           	1       	Fri Nov 29 17:03:24 2019	DEPLOYED	sdc-5.0.0           	           	onap     
demo-sdnc          	1       	Fri Nov 29 17:05:27 2019	DEPLOYED	sdnc-5.0.0          	           	onap     
demo-sniro-emulator	1       	Fri Nov 29 17:08:24 2019	DEPLOYED	sniro-emulator-5.0.0	           	onap     
demo-so            	1       	Fri Nov 29 17:10:33 2019	DEPLOYED	so-5.0.0            	           	onap     
demo-uui           	1       	Fri Nov 29 17:24:55 2019	DEPLOYED	uui-5.0.0           	           	onap     
demo-vfc           	1       	Fri Nov 29 17:24:59 2019	DEPLOYED	vfc-5.0.0           	           	onap     
demo-vid           	1       	Fri Nov 29 17:26:05 2019	DEPLOYED	vid-5.0.0           	           	onap     
demo-vnfsdk        	1       	Fri Nov 29 17:26:38 2019	DEPLOYED	vnfsdk-5.0.0        	           	onap     
ubuntu@onap-control-1:~/oom/kubernetes$

In case of failures in deployment

If the deployment of any onap module fails, please go through these steps to redeploy the modules.

In this example, we demonstrate failure of dmaap, which normally occurs due to timeout issues.

Check the failed modules

perform 'helm ls' on the control node.

ubuntu@onap-control-1:~/overrides$ helm ls
NAME               	REVISION	UPDATED                 	STATUS  	CHART               	APP VERSION	NAMESPACE
demo               	1       	Fri Nov 29 16:26:16 2019	DEPLOYED	onap-5.0.0          	El Alto    	onap     
demo-aaf           	1       	Fri Nov 29 16:26:17 2019	DEPLOYED	aaf-5.0.0           	           	onap     
demo-aai           	1       	Fri Nov 29 16:26:28 2019	DEPLOYED	aai-5.0.0           	           	onap     
demo-appc          	1       	Fri Nov 29 16:27:30 2019	DEPLOYED	appc-5.0.0          	           	onap     
demo-cassandra     	1       	Fri Nov 29 16:27:41 2019	DEPLOYED	cassandra-5.0.0     	           	onap     
demo-cds           	1       	Fri Nov 29 16:27:56 2019	DEPLOYED	cds-5.0.0           	           	onap     
demo-clamp         	1       	Fri Nov 29 16:28:58 2019	DEPLOYED	clamp-5.0.0         	           	onap     
demo-cli           	1       	Fri Nov 29 16:29:31 2019	DEPLOYED	cli-5.0.0           	           	onap     
demo-consul        	1       	Fri Nov 29 16:29:45 2019	DEPLOYED	consul-5.0.0        	           	onap     
demo-contrib       	1       	Fri Nov 29 16:30:04 2019	DEPLOYED	contrib-5.0.0       	           	onap     
demo-dcaegen2      	1       	Fri Nov 29 16:31:03 2019	DEPLOYED	dcaegen2-5.0.0      	           	onap     
demo-dmaap         	1       	Mon Dec  2 05:45:05 2019	FAILED  	dmaap-5.0.0         	           	onap     
demo-esr           	1       	Fri Nov 29 16:50:24 2019	DEPLOYED	esr-5.0.0           	           	onap     
demo-log           	1       	Fri Nov 29 16:50:38 2019	DEPLOYED	log-5.0.0           	           	onap     
demo-mariadb-galera	1       	Fri Nov 29 16:51:08 2019	DEPLOYED	mariadb-galera-5.0.0	           	onap     
demo-modeling      	1       	Fri Nov 29 16:51:29 2019	DEPLOYED	modeling-5.0.0      	           	onap     
demo-msb           	1       	Fri Nov 29 16:51:45 2019	DEPLOYED	msb-5.0.0           	           	onap     
demo-multicloud    	1       	Fri Nov 29 16:52:23 2019	DEPLOYED	multicloud-5.0.0    	           	onap     
demo-nbi           	1       	Fri Nov 29 16:53:26 2019	DEPLOYED	nbi-5.0.0           	           	onap     
demo-oof           	1       	Mon Dec  2 07:04:11 2019	DEPLOYED	oof-5.0.0           	           	onap     
demo-policy        	1       	Fri Nov 29 16:56:27 2019	DEPLOYED	policy-5.0.0        	           	onap     
demo-pomba         	1       	Fri Nov 29 16:59:13 2019	DEPLOYED	pomba-5.0.0         	           	onap     
demo-portal        	1       	Fri Nov 29 17:01:44 2019	DEPLOYED	portal-5.0.0        	           	onap     
demo-robot         	1       	Fri Nov 29 17:03:10 2019	DEPLOYED	robot-5.0.0         	           	onap     
demo-sdc           	1       	Fri Nov 29 17:03:24 2019	DEPLOYED	sdc-5.0.0           	           	onap     
demo-sdnc          	1       	Fri Nov 29 17:05:27 2019	DEPLOYED	sdnc-5.0.0          	           	onap     
demo-sniro-emulator	1       	Fri Nov 29 17:08:24 2019	DEPLOYED	sniro-emulator-5.0.0	           	onap     
demo-so            	1       	Mon Dec  2 08:58:33 2019	DEPLOYED	so-5.0.0            	           	onap     
demo-uui           	1       	Fri Nov 29 17:24:55 2019	DEPLOYED	uui-5.0.0           	           	onap     
demo-vfc           	1       	Fri Nov 29 17:24:59 2019	DEPLOYED	vfc-5.0.0           	           	onap     
demo-vid           	1       	Fri Nov 29 17:26:05 2019	DEPLOYED	vid-5.0.0           	           	onap     
demo-vnfsdk        	1       	Fri Nov 29 17:26:38 2019	DEPLOYED	vnfsdk-5.0.0        	           	onap

Delete the failed module

use the right release name from the name as shown in helm ls.

The below commands will delete all the pods and dependencies.

helm delete demo-dmaap --purge
kubectl get persistentvolumeclaims -n onap | grep demo-dmaap | sed -r 's/(^[^ ]+).*/kubectl delete persistentvolumeclaims -n onap \1/' | bash
kubectl get persistentvolumes      -n onap | grep demo-dmaap | sed -r 's/(^[^ ]+).*/kubectl delete persistentvolumes      -n onap \1/' | bash
kubectl get secrets                -n onap | grep demo-dmaap | sed -r 's/(^[^ ]+).*/kubectl delete secrets                -n onap \1/' | bash
kubectl get clusterrolebindings    -n onap | grep demo-dmaap | sed -r 's/(^[^ ]+).*/kubectl delete clusterrolebindings    -n onap \1/' | bash
kubectl get jobs                   -n onap | grep demo-dmaap | sed -r 's/(^[^ ]+).*/kubectl delete jobs                   -n onap \1/' | bash
kubectl get pods                   -n onap | grep demo-dmaap | sed -r 's/(^[^ ]+).*/kubectl delete pods                   -n onap \1/' | bash

Delete any residue data from onap-nfs-server.

cd /dockerdata-nfs/
sudo rm -r demo-dmaap/

Reinstall module

Reinstall the deleted module with the same release name as used in the deletion

helm install --namespace onap -n demo-dmaap local/dmaap --timeout 900

You can verify the deployment in a parallel terminal by checking the pods using the command

kubectl get pods -n onap | grep demo-dmaap

This shall show you the status of the upcoming pods during the deployment wait time.

Once this is deployed, you can verify using the "helm ls" command to check all the required modules are up and running.

Undeploy ONAP

For the release name 'demo' and namespace 'onap':

Perform these below commands in onap-control-1 where the helm was deployed.

helm undeploy demo --purge
kubectl get persistentvolumeclaims -n onap | grep demo | sed -r 's/(^[^ ]+).*/kubectl delete persistentvolumeclaims -n onap \1/' | bash
kubectl get persistentvolumes      -n onap | grep demo | sed -r 's/(^[^ ]+).*/kubectl delete persistentvolumes      -n onap \1/' | bash
kubectl get secrets                -n onap | grep demo | sed -r 's/(^[^ ]+).*/kubectl delete secrets                -n onap \1/' | bash
kubectl get clusterrolebindings    -n onap | grep demo | sed -r 's/(^[^ ]+).*/kubectl delete clusterrolebindings    -n onap \1/' | bash
kubectl get jobs                   -n onap | grep demo | sed -r 's/(^[^ ]+).*/kubectl delete jobs                   -n onap \1/' | bash
kubectl get pods                   -n onap | grep demo | sed -r 's/(^[^ ]+).*/kubectl delete pods                   -n onap \1/' | bash

Perform the below commands in onap-nfs-server.

This is to delete all the network shared data which will not be deleted on purge delete of helm release.

cd /dockerdata-nfs/
sudo rm -r *

e.g.
ubuntu@onap-nfs-server:~$ cd /dockerdata-nfs/
ubuntu@onap-nfs-server:/dockerdata-nfs$ sudo rm -r *

Access SDN-R

ODLUX-GUI  at winlab

The {user} should be replaced by your user id created in orbit-lab.

ssh -A -t {user}@console.sb10.orbit-lab.org -L 31939:localhost:31939 ssh -A -t native@node1-1.sb10.orbit-lab.org -L 31939:localhost:31939 ssh -A -t ubuntu@10.31.3.2 -L 31939:localhost:31939

On your browser:

http://localhost:31939/odlux/index.html

The login credentials are:

username: admin

password: Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U

ODLUX-GUI at your labs

Check the IP on the sdnc service bound to port 8282:

ubuntu@onap-control-1:~$ kubectl get service | grep sdnc
pomba-sdncctxbuilder               ClusterIP      10.43.145.187   <none>                                 9530/TCP                                                      2d18h
sdnc                               NodePort       10.43.103.106   <none>                                 8282:31939/TCP,8202:31510/TCP,8280:30246/TCP,8443:30267/TCP   7m52s
sdnc-ansible-server                ClusterIP      10.43.113.226   <none>                                 8000/TCP                                                      7m52s
sdnc-cluster                       ClusterIP      None            <none>                                 2550/TCP                                                      7m52s
sdnc-dgbuilder                     NodePort       10.43.62.50     <none>                                 3000:30203/TCP                                                7m52s
sdnc-dmaap-listener                ClusterIP      None            <none>                                 <none>                                                        7m52s
sdnc-portal                        NodePort       10.43.116.73    <none>                                 8443:30201/TCP                                                7m52s
sdnc-ueb-listener                  ClusterIP      None            <none>                                 <none>                                                        7m52s
so-sdnc-adapter                    ClusterIP      10.43.27.8      <none>                                 8086/TCP                                                      142m
vfc-zte-sdnc-driver                ClusterIP      10.43.189.170   <none>                                 8411/TCP                                                      2d17h

Look for sdnc service only and the port bound to 8282.

In the above result it is 31939.

http://{ip-address}:{sdnc-port-bound-to-8282}/odlux/index.html

The {ip-address} is the IP address of the onap-control-1 machine.

Troubleshooting

Documentation for troubleshooting.

ES-DB index read only – NFS out of disc space

ONAP log (elastic DB) consumes approx 1 GB per day. If disc space crosses a threshold, indindices of SDNC elastic DB are set to read-only.

** FORBIDDEN/12/index read-only / allow delete (api)] **

Housekeeping of elastic db can be done by Curator.

install curator

pip install elasticsearch-curator
#show indices with size
curator_cli show_indices --verbose
curator_cli show_indices --verbose|wc -l
# delete older indices
# test with dry run
curator --dry-run delete_index.yml
# delete!
curator delete_index.yml

delete_index.yml

---
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 45 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 30
      exclude:

configuration of curator:

/home/ubuntu/.curator/curator.yml  for curator-cli

/etc/curator/config.yaml  for curator

client:
  hosts:
    - 127.0.0.1
  port: 30254
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:

Define as a cron job (e.g. every 24h)

00 6 * * * root curator /home/ubuntu/curator/delete-indices.yml — config /etc/curator/config.yml


Set SDNC elastic DB indices back to read-write

PUT _settings
    {
    "index": {
    "blocks": {
    "read_only_allow_delete": "false"
    }
    }
    }



  • No labels