Table of Contents
Table of Contents exclude Table of Contents
VMware VIO 4.1 Kubernetes Architecture
VMware VIO supports Container Orchestration. Container Orchestration can be used a product or as a service. This page captures the information and steps to deploy ONAP using OOM on VI0 4.1 with Kubernetes
...
using 'OpenStack'
...
as the Cloud Provider
...
.
VMware VIO 4.1 Kubernetes Architecture
Prerequisites
...
- VMware Integrated OpenStack 4.1.
- VMware Integrated OpenStack 4.1 with Kubernetes.
- A SSH Client - (Example: MobaXTerm - Download).
Software Requirements
...
OOM Software Requirements
Release | Kubernetes | Helm | Kubectl | Docker |
---|---|---|---|---|
Amsterdam | 1.7.x | 2.3.x | 1.7.x | 1.12.x |
Beijing/Master | 1.8.5 | 2.7.x | 1.8.5 | 1.12.x |
Minimum Hardware Configuration
...
OOM Hardware Requirements
RAM | HD | vCores | Ports |
---|---|---|---|
128 GB | 160 GB | 16 | 0.0.0.0/0 (All Open) |
Procedure
...
Step 1: Login to VMware Intergrated OpenStack 4.1 with Kubernetes
- Go to your web browser (Firefox/Chrome Recommended).
- Visit https://FQDN/login or https://VIO_K8s_IP_Address/login
Step 2: Create a Cloud Provider
- Go to Cloud Provider and click on Deploy New Provider.
- If you have a JSON File with all the configurations, then click on Choose File and upload the JSON.
Otherwise, click Next. - Give a Provider name and select OpenStack as the Provider type, then click Next.
- We need to provide OpenStack Authentication now to add it as the Cloud Provider.
Give the Keystone Public URL, Username, Password, Project name, Region name, Domain name and upload the CA Certificate.
(CA Certificate is an important step for successful OpenStack Authentication, you can found more details about the certificate on the Page 34 of this PDF.)
Click Next. - Now, we need to select the Image which will be used to deploy the Kubernetes Master and Worker Nodes.
We are selecting Ubuntu 16.04 as the image for our deployment.
Give the Image username and Select an image from the list.
(This list will be fetched from your VMware Integrated Openstack deployment post successful authentication.) - Now, we need to select the Flavor which will be used to deploy the Kubernetes Master and Worker Nodes.
We are selecting m1.oom as the flavor for our deployment. This is a custom flavor created on VMware Integrated OpenStack 4.1.
(It is recommended to create a flavor which meets the minimum hardware configurations as described above in this document.)
Select a flavor from the list. (This list will be fetched from your VMware Integrated Openstack deployment post successful authentication.)
Click Next. - If you are not using NSX-T, then you can click Next.
If you are using NSX-T, then give then Manager address, Username, Password, Tier 0 Router and Transport zone. - Now, we need to select the Security Group which will be applied to all the cluster nodes. All the security groups created by you will be listed here.
Select security group(s) has a list of groups with their names, select them as per your requirements.
Click Next. - Now, we need to select the External Network which will used to expose Kubernetes Cluster API and also for external services. All the external networks created by you will be listed here.
Select external network has a list of external network with their names, select the external network as per your requirement.
(Please make sure that at least two floating IPs are available.)
Click Next. - Now, we need to select the Internal Network, it should be connected to the External Network via a router and should have atleast one subnet. All the internal networks created by you will be listed here.
Select internal network and Select a subnet has a list of internal networks and subnets with their names, select the internal network and subnet as per your requirement.
Click Next. - Now, check the Configuration Summary and verify the information.
If you wish to download the JSON of your configurations, click on Download Provider JSON.
Click Finish to add the Cloud Provider. - The addition of Cloud Provider is pretty quick and it should be added instantly as soon you click Finish. If not then kindly wait for few minutes.
- You can check your Cloud Provider on the page.
It should show the state as ACTIVE, denoting successful addition of Cloud Provider. - You can get additional information regarding a Cloud Provider by clicking on it's name.
Step 3: Create a Kubernetes Cluster
- Go to Clusters and click on Deploy New Cluster.
- If you have a JSON File with all the configurations, then click on Choose File and upload the JSON.
Otherwise, click Next. - Select an Infrastructure Provider, it shows a list of all the Cloud Providers you have added.
Select the one on which you want to deploy your Kubernetes Cluster.
Click Next. - Select an Infrastructure Node Profile, it is the configuration using which your Kubernetes Cluster will be deployed.
If you want to use the default profile, click on the checkbox of Use default node profile.
(You can also add your own profiles from the Cloud Provider section and then select the one on which you want to deploy your Kubernetes Cluster from the list.)
Click Next.
- Now, we need to add Cluster Data.
Give the Cluster Name, Number of Master Nodes, Number of Worker Nodes, DNS Servers and Cluster Type (Shared or Exclusive).
(In an exclusive cluster, multi-tenancy is not supported. In a shared cluster, multi-tenancy is supported and enforced by the Kubernetes namespace.)
Click Next. - Now, we need to add Users & Groups for this cluster.
Select the Users and Group from the list, or we can provide additional users and groups if they are not created.
Click Next. - Now, check the Kubernetes Cluster Deployment Summary and verify the information.
If you wish to download the JSON of your configurations, click on Download Cluster JSON.
Click Finish to start the deployment of Kubernetes Cluster. - The deployment of Kubernetes Cluster takes some time as compared to Cloud Provider.
Please be patient while the Cluster is being deployed. Once deployed, the state of cluster will change from CREATING to READY.
During the deployment process, the cluster state will be CREATING.
After the successful deployment process, the cluster state will be ACTIVE.
During the deployment process if there is an error, the cluster state will be changed to ERROR.
Meanwhile, you can login
Login to your VMware Integrated OpenStack Horizon and check if the correct number of instance to verify that the Kubernetes instances have been deployed or not. If they are there then the deployment is in progress,
if you don't see these instance after 5 mins then there is a possiblity that the deployment has faced some error.
If you want to delete your cluster, the cluster state will be changed to DELETING.
...
- .
Step 4: Login to Kubernetes Master and
...
Workers Nodes
SSH into the VMware Integrated OpenStack with Kubernetes.
Once you are logged into VMware Integrated OpenStack with Kubernetes, use the following command with the same username and password used during SSH:
Code Block theme Midnight #vkube login --insecure
Get the list of all the clusters along with their Cluster ID, use the following command:
Code Block theme Midnight #vkube cluster list --insecure
Make a note of the Cluster ID of the cluster you want to access or know more about.
Get the detailed information about the nodes running on a particular cluster, use the following command:
Code Block theme Midnight #vkube cluster show <Cluster ID> --insecure
Make a note of the IP of the Master and Worker Nodes.
Get a list of all the docker containers running in background.
Code Block theme Midnight #docker ps -a
Get an interactive shell of the docker conatiner with the name app-api.
Code Block theme Midnight #docker exec -it <Container ID or Name> /bin/bash
After this command, you'll get a shell of a running container.
Once we are inside a running container, change the directory and go the cluster location using the following command:
Code Block theme Midnight #cd /var/lib/vrc/terraform/<ClusterID>
Once you are into the correct directory, use the following command to SSH into the Kubernetes Master or Worker Node depending on the IP Address:
Code Block theme Midnight #ssh -i private.key -F ssh-bastion.conf ubuntu@Kubernetes_Host_IP_Address
Once you are inside the Kubernetes Nodes, you can deploy ONAP, or troubleshoot an error as per your requirements.
Step 5: Deploy ONAP
Log in to the Master Node (as described in the steps above) and then follow the steps for ONAP Deployment.
The detailed documentation for the ONAP Deployment can be found on GitHub and ONAP Docs.
Prerequisites:
- VMware Integrated OpenStack 4.1.
- VMware Integrated OpenStack 4.1 with Kubernetes.
- A SSH Client - (Example: MobaXTerm - Download).
Software Requirements:
OOM Software Requirements
...
Release
...
Kubernetes
...
Helm
...
Kubectl
...
Docker
...
Minimum Hardware Configuration:
OOM Hardware Requirements
...
RAM
...
HD
...
vCores
...
Ports
...
Procedure:
Step 1: Login to VMware Intergrated OpenStack 4.1 with Kubernetes
...
Step 2: Create a Cloud Provider
...
Step 3: Create a Kubernetes Cluster
...
Troubleshooting
- During the deployment process if there is an error, the cluster state will be changed to ERROR.
You can delete your cluster in case of an ERROR, the cluster state will be changed to DELETING.
During the creation of your cluster, you can login to your VMware Integrated OpenStack Horizon and check if the correct number of instance have been deployed or not. If they are there then the deployment is in progress else there is a possiblity that the deployment has faced some error.
...
Step 4: Login to Kubernetes Master and Worker Nodes
...
Once you are logged into VMware Integrated OpenStack with Kubernetes, use the following command with the same username and password used during SSH:
...
#vkube login --insecure
Get the list of all the clusters along with their Cluster ID, use the following command:
...
#vkube cluster list --insecure
...
Get the detailed information about the nodes running on a particular cluster, use the following command:
...
#vkube cluster show <Cluster ID> --insecure
Make a note of the IP of the Master and Worker Nodes.
Get a list of all the docker containers running in background.
...
#docker ps -a
Get an interactive shell of the docker container with the name app-api.
...
#docker exec -it <Container ID or Name> /bin/bash
After this command, you'll get a shell of a running container.
Once we are inside a running container, change the directory and go the cluster location using the following command:
...
#cd /var/lib/vrc/terraform/<ClusterID>
Once we are inside a running container, change the directory and go the cluster location using the following command:
...
#cd /var/lib/vrc/terraform/<ClusterID>
...
Once you are inside the Kubernetes Nodes, you can deploy ONAP, or troubleshoot an error as per your requirements.
Step 5: Deploy ONAP
Log in to the Master Node (as described in the steps above) and then follow the steps for ONAP Deployment.
The detailed documentation for the ONAP Deployment can be found on GitHub and ONAP Docs.
Prerequisites
Mgmt IP Address, username and password to create Kubernetes cluster .
K8S Master and K8S Node instances should have a OpenStack Flavor attached as per the requirement below
...
View file | ||||
---|---|---|---|---|
|
View file | ||||
---|---|---|---|---|
|
Steps to create a Kubernetes cluster
Follow the below steps 1 - 4 to create a Kubernetes cluster
Step 5 onwards are needed for getting the Kubernetes Host IP so that user can login to Kubernetes Host and ONAP using OOM can be deployed.
Step-1 - https://MGMT_IP_ADDRESS/LOGIN
Step-2 Create the Cloud Provider before creating a Kubernetes cluster
Cloud Provider creation is a prerequisite to Kubernetes cluster creation. VIO with Kubernetes uses the cloud provider to create the infrastructure required to deploy all your Kubernetes clusters. VMware currently supports 2 options for infrastructure provider. VMware SDDC (vSphere + NSX + VSAN) or OpenStack (i.e. VMware Integrated OpenStack). When choosing the type of provider to create, consider the following:
...
Step-3 Create the Kubernetes cluster
3.1 - Click '+NEW' to create a Kubernetes cluster
3.2 - Click NEXT
3.3 - Select an Infrastructure Provider for creating the Kubernetes
Before you deploy a Kubernetes cluster, you must create the cloud provider. Cloud provides can be SDDC or OpenStack. Select the option as appropriate.
Here, with VIO with Kubernetes, we select 'OpenStack' as cloud provider
3.4 - Select a Node Profile. If you have more than one node profiles, uncheck the box "Use default node profile" to see the list.
3.5 - Provide the Input for the Cluster as indicated in Example Data below
Node Types - A Kubernetes cluster is comprised of two types of nodes. Each node in the VMware Integrated OpenStack with Kubernetes is a VM.
- Master Nodes - A master node provides the Kubernetes API service, scheduler, replicator, and so on. It manages the worker nodes. A cluster with a single master node is valid but has no redundancy.
- Worker Nodes - A worker node hosts your containers. A cluster with a single worker node is valid but has no redundancy.
Cluster Types - VMware Integrated OpenStack with Kubernetes supports two types of clusters.
- Exclusive Cluster - In an exclusive cluster, multi-tenancy is not supported. Any authorized Kubernetes user using the Kubernetes CLI of APIs has namespace management privileges. The exclusive cluster provides a familiar environment for developers who deploy Kubernetes themselves.
- Shared Cluster - In a shared cluster, multi-tenancy is supported and enforced by the Kubernetes namespace. Only a VMware Integrated OpenStack with Kubernetes administrator using the VMware Integrated OpenStack with Kubernetes interface or CLI has namespace management privileges. The shared cluster is an environment where the administrator can manage resource isolation among users.
Example Data:
Cluster Name :testCluster
Number of Master Nodes :1
Number of worker nodes :1
DNS servers : 10.112.64.1
Cluster type: Exclusive Cluster
3.6 - Add Users and Groups for this cluster
Once a Kubernetes cluster is created, you can authorize users or groups for the cluster. The users and groups belong to the SDDC or OpenStack provider where the cluster was created..
In the Configure user and group for cluster dialogue box, check the boxes for users or groups that you want to authorize for access to the cluster.
Or check off the boxes for users or groups that you no longer want to authorize for access to the cluster.
3.7 - Click on FINISH and wait for few minutes for the kubernetes cluster to get created .
Step-4 Verifying the VIO Kubernetes Cluster
If everything in step 3.1 to step 3.6 has been done successfully .The Summary information for the cluster will be filled as given in the example below .
Step-5 How to get Kubernetes host IP Address and login to Kubernetes host .
5.1 - Login via console window to MGMT_IP_ADDRESS . user name and password is same as used in step1 .
5.2 - Once logged into MGMT_IP_ADDRESS session use the command "vkube login --insecure" . use the user name and password same as in step1 .
5.3 - Get the list of clusters using command "vkube cluster list --insecure" . make a note of cluster Id in the output of the command
5.4 - Get the cluster node details using command "vkube cluster show <cluster Id > --insecure " .make note of worker ip address .Worker IP Address KUBERNETES_HOST_IP.
5.5 - Login to app-api docker using docker exec it app-api bash
5.6 - Once inside the app-api docker go to /var/lib/vrc/terraform/<cluserId>
5.7 - ssh to KUBERNETES_HOST_IP_ADDRESS using below command
ssh -i private.key -F ssh.bastion.conf ubuntu@KUBERNETES_HOST_IP_ADDRESS
5.8 - Once inside the KUBERNETES_HOST_IP_ADDRESS , install kubectl and Helm to prepare Kubernetes for OOM installation .
Step-6 Installing kubectl to manage Kubernetes cluster
6.1 - Download the kubectl using below command
curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.7.0/bin/linux/amd64/kubectl
6.2 - Make the kubectl binary executable.
chmod +x ./kubectl
6.3 - Move the kubectl to PATH
sudo mv ./kubectl /usr/local/bin/kubectl
Step-7 Verifying that kubectl config is good
7.1 - On Kubernetes cluster
root@localhost:~# kubectl cluster-info
Kubernetes master is running at ....
Heapster is running at....
KubeDNS is running at ....
kubernetes-dashboard is running at ...
monitoring-grafana is running at ....
monitoring-influxdb is running at ...
tiller-deploy is running at....
7.2 - On client from Where Kubernetes cluster can be managed Remotely
root@localhost:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-29T23:15:59Z", GoVersion:"go1.8.3",
Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.7-rancher1", GitCommit:"a1ea37c6f6d21f315a07631b17b9537881e1986a", GitTreeState:"clean", BuildDate:"2017-10-02T21:33:08Z",GoVersion:"go1.8.3" Compiler:"gc", Platform:"linux/amd64"}
Step-8 Verifying that kube config is good
8.1 - On Kubernetes cluster
root@localhost:~# cat ~/.kube/config
apiVersion: v1
kind: Config
clusters:
- cluster:
api-version: v1
insecure-skip-tls-verify: true
server: "<SERVER_IP_ADDRESS:8080/r/projects/CLUSTER_NAME/kubernetes:SERVER_PORT_NUMBER"
name: "(CLUSTER_NAME)"
contexts:
- context:
cluster: "(CLUSTER_NAME)"
user: "(CLUSTER_NAME)"
name: "(CLUSTER_NAME)"
current-context: "(CLUSTER_NAME)"
users:
- name: "(CLUSTER_NAME/USER_NAME)"
user:
token: "<SECURITY_TOKEN>"
8.2 - On client from where Kubernetes cluster can be managed Remotely
root@localhost:~# cat ~/.kube/config
current-context: default-context
apiVersion: v1
clusters:
- cluster:
api-version: v1
server: https://SERVER_IP_ADDRESS:SERVER_PORT_NUMBER/
insecure-skip-tls-verify: true
name: CLUSTER_NAME
contexts:
- context:
cluster: CLUSTER_NAME
namespace: default
user: user1
name: default-context
users:
- name: user1
user:
username: "<USERNAME>"
password: "<PASSWORD"
kind: Config
preferences:
colors: true
Step-9 Installing Helm
9.1 - Download the helm using below command
wget http://storage.googleapis.com/kubernetes-helm/helm-v2.3.0-linux-amd64.tar.gz
9.2 - Download the helm using below command
tar -zxvf helm-v2.3.0-linux-amd64.tar.gz
9.3 - Move the helm to /usr/local/bin
sudo mv linux-amd64/helm /usr/local/bin/helm
Step-10 Verifying Helm
10.1 - Type the below command
helm help
10.2 - Helm version .
Client: &version.Version{SemVer:"v2.3.0", GitCommit:"d83c245fc324117885ed83afc90ac74afed271b4", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.3.0", GitCommit:"d83c245fc324117885ed83afc90ac74afed271b4", GitTreeState:"clean"}
Step-11 Please follow below instructions for OOM Deployment on VIO 4.0 Kubernetes and rest of the service design creation and instantiation on VIO 4.0 Kubernetes cluster
11.1 - vFW and vDNS service design & creation and service instantiation steps are identical .
11.2 - vFW and vDNS service design & creation and service instantiation steps on VMware Integrated OpenStack(VIO) 4.0 are identical to Vanilla OpenStack .
11.3 - Please refer to below link for OOM deployment and service design creation and instantiation on VIO 4.0 Kubernetes
...