...
Download and install RKE on a VM, desktop or laptopthe onap-control-01 VM. Binaries can be found here for Linux and Mac: https://github.com/rancher/rke/releases/tagdownload/v0v1.2.17/
Perform this on linux based personal laptop, local machine or a on the onap-control-01 VM which has the connectivity key to access the created VMs on OpenStack and whose key has been configured during creation of VMs.
In our setup, the key of OpenStack controller has also been placed in onap-control-01 and used to login into VMs, so these steps have been performed on OpenStack controlleronap-control-01 VM.
| Code Block |
|---|
mkdir rke cd rke/ wget https://github.com/rancher/rke/releases/download/v1.12.67/rke_linux-amd64 mv rke_linux-amd64 rke chmod 754 rke |
RKE requires a cluster.yml as input. An example file is show below that describes a Kubernetes cluster that will be mapped onto the OpenStack VMs created earlier in this guide.
Example: cluster.yml
The example file:
| Code Block |
|---|
# An example of an HA Kubernetes cluster for ONAP nodes: - address: 10.31.4.11 port: "22" role: - controlplane - etcd hostname_override: "onap-control-01" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.12 port: "22" role: - controlplane - etcd hostname_override: "onap-control-02" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.13 port: "22" role: - controlplane - etcd hostname_override: "onap-control-03" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.21 port: "22" role: - worker hostname_override: "onap-workers-01" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.22 port: "22" role: - worker hostname_override: "onap-workers-02" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.23 port: "22" role: - worker hostname_override: "onap-workers-03" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.24 port: "22" role: - worker hostname_override: "onap-workers-04" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.25 port: "22" role: - worker hostname_override: "onap-workers-05" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.26 port: "22" role: - worker hostname_override: "onap-workers-06" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.27 port: "22" role: - worker hostname_override: "onap-workers-07" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" - address: 10.31.4.28 port: "22" role: - worker hostname_override: "onap-workers-08" user: ubuntu ssh_key_path: "~/.ssh/id_ecdsa" services: kube-api: service_cluster_ip_range address: 10.4331.04.0/1629 pod_security_policyport: false"22" always_pull_images: falserole: kube-controller: worker clusterhostname_cidr: 10.42.0.0/16override: "onap-workers-09" user: ubuntu service ssh_cluster_ip_rangekey_path: "~/.ssh/id_ecdsa" - address: 10.4331.04.0/1630 kubeletport: "22" cluster_domain: cluster.localrole: cluster_dns_server: 10.43.0.10- worker fail_swap_on: false network:hostname_override: "onap-workers-10" pluginuser: canalubuntu authentication: strategy: x509 ssh_key_path: "~/.ssh/id_ecdsa" ssh_agent_authservices: false authorization kube-api: mode: rbac ignore_docker_version: false kubernetes_version: "v1.15.11-rancher1-2" private_registries: - url: nexus3.onap.org:10001 user: docker password: docker is_default: true cluster_name: "onap" restore: restore: false snapshot_name: "" |
Prepare cluster.yml
Before this configuration file can be used the IP address must be mapped for each control and worker node in this file.
Run RKE
From within the same directory as the cluster.yml file, simply execute:
native@node1-1:~/rke$ ./rke up
| Code Block |
|---|
./rke up |
The output will look something like:
service_cluster_ip_range: 10.43.0.0/16
pod_security_policy: false
always_pull_images: false
kube-controller:
cluster_cidr: 10.42.0.0/16
service_cluster_ip_range: 10.43.0.0/16
kubelet:
cluster_domain: cluster.local
cluster_dns_server: 10.43.0.10
fail_swap_on: false
network:
plugin: canal
authentication:
strategy: x509
ssh_key_path: "~/.ssh/id_ecdsa"
ssh_agent_auth: false
authorization:
mode: rbac
ignore_docker_version: false
kubernetes_version: "v1.19.9-rancher1-1"
private_registries:
- url: nexus3.onap.org:10001
user: docker
password: docker
is_default: true
cluster_name: "onap"
restore:
restore: false
snapshot_name: "" |
Prepare cluster.yml
Before this configuration file can be used the IP address must be mapped for each control and worker node in this file.
Run RKE
From within the same directory as the cluster.yml file, simply execute:
ubuntu@onap-control-01:~/rke$ ./rke up
| Code Block |
|---|
./rke up |
The output will look something like:
| Code Block |
|---|
ubuntu@onap-control-01:~/rke$ ./rke up
INFO[0000] Running RKE version: v1.2.7
INFO[0000] Initiating Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [10.31.4.11]
INFO[0000] [dialer] Setup tunnel for host [10.31.4.29]
INFO[0000] [dialer] Setup tunnel for host [10.31.4.25]
INFO[0000] [dialer] Setup tunnel for host [10.31.4.12]
INFO[0000] [dialer] Setup tunnel for host [10.31.4.21]
INFO[0000] [dialer] Setup tunnel for host [10.31.4.13]
INFO[0000] [dialer] Setup tunnel for host [10.31.4.23]
INFO[0000] [dialer] Setup tunnel for host [10.31.4.30]
INFO[0000] [dialer] Setup tunnel for host [10.31.4.28]
INFO[0000] [dialer] Setup tunnel for host [10.31.4.27]
INFO[0000] [dialer] Setup tunnel for host [10.31.4.26]
INFO[0000] [dialer] Setup tunnel for host [10.31.4.24]
INFO[0000] [dialer] Setup tunnel for host [10.31.4.22]
WARN[0050] Failed to set up SSH tunneling for host [10.31.4.13]: Can't retrieve Docker Info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
WARN[0050] Failed to set up SSH tunneling for host [10.31.4.12]: Can't retrieve Docker Info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
WARN[0050] Failed to set up SSH tunneling for host [10.31.4.25]: Can't retrieve Docker Info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
WARN[0050] Failed to set up SSH tunneling for host [10.31.4.29]: Can't retrieve Docker Info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
WARN[0050] Failed to set up SSH tunneling for host [10.31.4.21]: Can't retrieve Docker Info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
WARN[0050] Failed to set up SSH tunneling for host [10.31.4.30]: Can't retrieve Docker Info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
WARN[0050] Failed to set up SSH tunneling for host [10.31.4.23]: Can't retrieve Docker Info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
WARN[0050] Failed to set up SSH tunneling for host [10.31.4.27]: Can't retrieve Docker Info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
WARN[0050] Failed to set up SSH tunneling for host [10.31.4.28]: Can't retrieve Docker Info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
WARN[0050] Failed to set up SSH tunneling for host [10.31.4.24]: Can't retrieve Docker Info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
WARN[0050] Removing host [10.31.4.13] from node lists
WARN[0050] Removing host [10.31.4.12] from node lists
WARN[0050] Removing host [10.31.4.25] from node lists
WARN[0050] Removing host [10.31.4.29] from node lists
WARN[0050] Removing host [10.31.4.21] from node lists
WARN[0050] Removing host [10.31.4.30] from node lists
WARN[0050] Removing host [10.31.4.23] from node lists
WARN[0050] Removing host [10.31.4.27] from node lists
WARN[0050] Removing host [10.31.4.28] from node lists
WARN[0050] Removing host [10.31.4.24] from node lists
INFO[0050] Checking if container [cluster-state-deployer] is running on host [10.31.4.26], try #1
INFO[0051] Pulling image [nexus3.onap.org:10001/rancher/rke-tools:v0.1.72] on host [10.31.4.26], try #1
INFO[0057] Image [nexus3.onap.org:10001/rancher/rke-tools:v0.1.72] exists on host [10.31.4.26]
INFO[0057] Starting container [cluster-state-deployer] on host [10.31.4.26], try #1
INFO[0058] [state] Successfully started [cluster-state-deployer] container on host [10.31.4.26]
INFO[0058] Checking if container [cluster-state-deployer] is running on host [10.31.4.11], try #1
INFO[0059] Pulling image [nexus3.onap.org:10001/rancher/rke-tools:v0.1.72] on host [10.31.4.11], try #1
INFO[0064] Image [nexus3.onap.org:10001/rancher/rke-tools:v0.1.72] exists on host [10.31.4.11]
INFO[0065] Starting container [cluster-state-deployer] on host [10.31.4.11], try #1
INFO[0065] [state] Successfully started [cluster-state-deployer] container on host [10.31.4.11]
INFO[0065] Checking if container [cluster-state-deployer] is running on host [10.31.4.22], try #1
INFO[0066] Pulling image [nexus3.onap.org:10001/rancher/rke-tools:v0.1.72] on host [10.31.4.22], try #1
INFO[0070] Image [nexus3.onap.org:10001/rancher/rke-tools:v0.1.72] exists on host [10.31.4.22]
INFO[0071] Starting container [cluster-state-deployer] on host [10.31.4.22], try #1
INFO[0071] [state] Successfully started [cluster-state-deployer] container on host [10.31.4.22]
INFO[0071] [certificates] Generating CA kubernetes certificates
INFO[0072] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates
INFO[0072] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates
INFO[0072] [certificates] Generating Kubernetes API server certificates
INFO[0072] [certificates] Generating Service account token key
INFO[0072] [certificates] Generating Kube Controller certificates
INFO[0072] [certificates] Generating Kube Scheduler certificates
INFO[0072] [certificates] Generating Kube Proxy certificates
INFO[0073] [certificates] Generating Node certificate
INFO[0073] [certificates] Generating admin certificates and kubeconfig
INFO[0073] [certificates] Generating Kubernetes API server proxy client certificates
INFO[0073] [certificates] Generating kube-etcd-10-31-4-11 certificate and key
INFO[0073] Successfully Deployed state file at [./cluster.rkestate]
. . . .
. . . .
. . . .
INFO[0168] [sync] Successfully synced nodes Labels and Taints
INFO[0168] [network] Setting up network plugin: canal
INFO[0168] [addons] Saving ConfigMap for addon rke-network-plugin to Kubernetes
INFO[0168] [addons] Successfully saved ConfigMap for addon rke-network-plugin to Kubernetes
INFO[0168] [addons] Executing deploy job rke-network-plugin
INFO[0178] [addons] Setting up coredns
INFO[0178] [addons] Saving ConfigMap for addon rke-coredns-addon to Kubernetes
INFO[0178] [addons] Successfully saved ConfigMap for addon rke-coredns-addon to Kubernetes
INFO[0178] [addons] Executing deploy job rke-coredns-addon
INFO[0183] [addons] CoreDNS deployed successfully
INFO[0183] [dns] DNS provider coredns deployed successfully
INFO[0183 |
| Code Block |
native@node1-1:~/rke$ ./rke up INFO[0000] Initiating Kubernetes cluster INFO[0000] [certificates] Generating CA kubernetes certificates INFO[0000] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates INFO[0000] [certificates] Generating Kubernetes API server certificates INFO[0000] [certificates] Generating Service account token key INFO[0000] [certificates] Generating Kube Controller certificates INFO[0001] [certificates] Generating Node certificate INFO[0001] [certificates] Generating admin certificates and kubeconfig INFO[0001] [certificates] Generating Kubernetes API server proxy client certificates . . . . . . . . . . . . INFO[0309] [addons] Setting up Metrics Server INFO[03090183] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes INFO[03090183] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes INFO[03090183] [addons] Executing deploy job rke-metrics-addon INFO[03150188] [addons] Metrics Server deployed successfully INFO[03150188] [ingress] Setting up nginx ingress controller INFO[03150188] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes INFO[03160188] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes INFO[03160188] [addons] Executing deploy job rke-ingress-controller INFO[03220199] [ingress] ingress controller nginx deployed successfully INFO[03220199] [addons] Setting up user addons INFO[03220199] [addons] no user addons defined INFO[03220199] Finished building Kubernetes cluster successfully |
...
copy the file "kube_config_cluster.yml" to .kube directory on home of onap-control-1 01 VM.
In the VM/system where the rke was executed. In Winlab it is node1-1 (OpenStack controller node).
example: to perform sftp to connect to onap-control-1 node:
| Code Block |
|---|
ubuntu@onap-control-01:~/rke$ cd
ubuntu@onap-control-01:~$ mkdir .kube
ubuntu@onap-control-01:~$ cp rke/kube_config_cluster.yml .kube/
ubuntu@onap-control-01:~$ cd .kube/
ubuntu@onap-control-01:~/.kube$ ll
total 16
drwxrwxr-x 2 ubuntu ubuntu 4096 Jun 14 15:09 ./
drwxr-xr-x 8 ubuntu ubuntu 4096 Jun 14 15:09 ../
-rw-r----- 1 ubuntu ubuntu 5375 Jun 14 15:09 kube_config_cluster.yml
ubuntu@onap-control-01:~/.kube$ mv |
| Code Block |
native@node1-1:~/rke$ sftp ubuntu@10.31.3.2 Connected to 10.31.3.2. sftp> put kube_config_cluster.yml sftp> exit These steps can also be done using scp as (assuming that the controller has '.kube' folder): native@node1-1:~/rke$ scp kube_config_cluster.yml ubuntu@10.31.3.2:~/.kube/config |
...
config
ubuntu@onap-control-01:~/.kube$ ll
total 16
drwxrwxr-x 2 ubuntu ubuntu 4096 Jun 14 15:10 ./
drwxr-xr-x 8 ubuntu ubuntu 4096 Jun 14 15:09 ../
-rw-r----- 1 ubuntu ubuntu 5375 Jun 14 15:09 config |
In "onap-control-1" vm:
| Code Block |
|---|
ubuntu@onap-control-1:~$ mkdir .kube ubuntu@onap-control-1:~$ mv kube_config_cluster.yml .kube/config ubuntu@onap-control-1:~$ kubectl config set-context --current --namespace=onap |
...