Important: As Windriver is ONAP community lab, the access/privilege for all users are same. Hence do NOT delete any folders/files/vm unless you created it and are sole users (otherwise need to be communicated to the community/discuss-list and PTL before purging it). Other than jumpserver -10.12.5.50 (provided by Stephen) – all other VM’s should be considered volatile and hence any development changes being worked on windriver lab should be backed-up locally.
Redeployment steps
1. After login to Windriver, identify the K8S cluster and corresponding nfs server ip from dashboard - http://10.12.25.2/project/instances/. Typically there are two separate clusters setup, for Frankfurt – following can be used.
Note: The k8s-01 and k8s-02 are k8s node server. The cluster/nfs ip’s would change for every release and deployments will be periodically refreshed (every ~2 weeks)
2. Login using onap_dev key to required NFS server and su as “root”
ssh -i ~/onap_dev ubuntu@10.12.7.110
sudo su -
3. Check if required platform container/pod is running via kubectl
root@onap2-nfs:~# kubectl get pods -n onap | egrep "inventory|dashboard" dev-dcaegen2-dcae-dashboard-f6b57bd89-4fjv9 2/2 Running 0 3d19h dev-dcaegen2-dcae-dashboard-pg-primary-56c878df5d-c8z75 1/1 Running 0 4d19h dev-dcaegen2-dcae-dashboard-pg-replica-8659dd7f8-7m4h8 1/1 Running 0 4d19h dev-dcaegen2-dcae-dashboard-pgpool-6cb5c4dc76-ffzgs 1/1 Running 0 4d19h dev-dcaegen2-dcae-dashboard-pgpool-6cb5c4dc76-w6tl9 1/1 Running 0 4d19h dev-dcaegen2-dcae-inventory-api-6d98b567d4-zx6zq 1/1 Running 0 3d19h
4. All DCAE platform components are packages/deployed via helm as dcaegen2 chart - this can be verified using helm list command to identify how current platform components are deployed
root@onap2-nfs:~# helm list NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE dcae-mongo 1 Fri Jan 24 21:32:23 2020 DEPLOYED dcae-mongo-6.0.0 onap dcae-redis 1 Fri Jan 24 20:48:23 2020 DEPLOYED dcae-redis-6.0.0 onap dev 1 Thu Jan 23 20:36:21 2020 DEPLOYED onap-5.0.0 El Alto onap dev-aaf 1 Thu Jan 23 20:36:22 2020 DEPLOYED aaf-5.0.0 onap dev-consul 1 Thu Jan 23 20:36:23 2020 DEPLOYED consul-5.0.0 onap dev-dcaegen2 2 Fri Jan 24 20:43:07 2020 DEPLOYED dcaegen2-6.0.0 onap dev-dmaap 1 Thu Jan 23 20:36:27 2020 FAILED dmaap-5.0.0 onap dev-msb 1 Thu Jan 23 20:41:29 2020 DEPLOYED msb-5.0.0 onap dev-policy 1 Thu Jan 23 20:41:30 2020 DEPLOYED policy-5.0.0 onap
helm charts for each platform component can be found under ~/oom/kubernetes/dcaegen2/components (maintained in gerrit under OOM repository https://git.onap.org/oom/tree/kubernetes/dcaegen2/components)
root@onap2-nfs:~/oom/kubernetes/dcaegen2/components# ls -lrt total 40 drwxr-xr-x 5 root root 4096 Jan 23 19:03 dcae-bootstrap drwxr-xr-x 5 root root 4096 Jan 23 19:03 dcae-cloudify-manager drwxr-xr-x 5 root root 4096 Jan 23 19:04 dcae-config-binding-service drwxr-xr-x 4 root root 4096 Jan 23 19:04 dcae-healthcheck drwxr-xr-x 5 root root 4096 Jan 23 19:04 dcae-servicechange-handler drwxr-xr-x 5 root root 4096 Jan 23 19:04 dcae-inventory-api drwxr-xr-x 5 root root 4096 Jan 23 19:04 dcae-deployment-handler drwxr-xr-x 5 root root 4096 Jan 23 19:04 dcae-policy-handler drwxr-xr-x 5 root root 4096 Jan 23 19:04 dcae-dashboard drwxr-xr-x 5 root root 4096 Jan 24 21:06 dcae-redis
5. If component is already running, to undeploy a specific component installed under dev-dcaegen2 helm release, edit (or make a copy) the following ~/dcae_component_override.yaml and undeploy it via helm upgrade command.
In the example below, updated both dcae-redis and dcae-dashboard enabled flag to “false”
root@onap2-nfs:~# cat dcae_component_override.yaml dcae-bootstrap: enabled: true dcae-cloudify-manager: enabled: true dcae-config-binding-service: enabled: true dcae-healthcheck: enabled: true dcae-redis: enabled: false dcae-servicechange-handler: enabled: true dcae-inventory-api: enabled: true dcae-deployment-handler: enabled: true dcae-policy-handler: enabled: true dcae-dashboard: enabled: false
Followed by doing helm upgrade with updated yaml to trigger the termination.
helm upgrade --install dev-dcaegen2 local/dcaegen2 --namespace onap -f ~/dcae_component_override.yaml
This will undeploy/terminate dashboard component (redis was undeployed previously)
root@onap2-nfs:~# kubectl get pods -n onap | egrep "inventory|dashboard" dev-dcaegen2-dcae-dashboard-f6b57bd89-4fjv9 0/2 Terminating 0 3d19h dev-dcaegen2-dcae-inventory-api-6d98b567d4-zx6zq 1/1 Running 0 3d19h root@onap2-nfs:~# kubectl get pods -n onap | egrep "inventory|dashboard" dev-dcaegen2-dcae-inventory-api-6d98b567d4-zx6zq 1/1 Running 0 3d19h
More details on helm upgrade and override file update per subcomponent can be found under parent page - https://wiki.onap.org/pages/viewpage.action?pageId=71837415
6. To reinstall new version, goto chart directly, ensure the values.yaml under step #4 is updated to reflect the new version and any required config changes in charts
helm install -n <release name> --namespace onap <chart locations>
Example :
helm install -n dcae-dashboard --namespace onap ./dcae-dashboard
This brings up dashboard pods and Helm list also indicates the new deployment
root@onap2-nfs:~/oom/kubernetes/dcaegen2/components# kubectl get pods -n onap | egrep "inventory|dashboard" dcae-dashboard-dcae-dashboard-6b5c9ccd4c-lx7cw 2/2 Running 0 2m40s dcae-dashboard-dcae-dashboard-pg-primary-d8b649c7-gqrsx 1/1 Running 0 2m40s dcae-dashboard-dcae-dashboard-pg-replica-c4cfbd8fc-tnbpf 1/1 Running 0 2m40s dcae-dashboard-dcae-dashboard-pgpool-7bcf656f68-4555q 1/1 Running 0 2m40s dcae-dashboard-dcae-dashboard-pgpool-7bcf656f68-gqbd5 1/1 Running 0 2m40s dev-dcaegen2-dcae-inventory-api-6d98b567d4-zx6zq 1/1 Running 0 3d19h root@onap2-nfs:~/oom/kubernetes/dcaegen2/components# helm list NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE dcae-dashboard 1 Tue Jan 28 16:01:41 2020 DEPLOYED dcae-dashboard-6.0.0 1.0 onap dcae-mongo 1 Fri Jan 24 21:32:23 2020 DEPLOYED dcae-mongo-6.0.0 onap dcae-redis 1 Fri Jan 24 20:48:23 2020 DEPLOYED dcae-redis-6.0.0 onap dev 1 Thu Jan 23 20:36:21 2020 DEPLOYED onap-5.0.0 El Alto onap dev-aaf 1 Thu Jan 23 20:36:22 2020 DEPLOYED aaf-5.0.0 onap dev-consul 1 Thu Jan 23 20:36:23 2020 DEPLOYED consul-5.0.0 onap dev-dcaegen2 3 Tue Jan 28 15:58:47 2020 DEPLOYED dcaegen2-6.0.0 onap dev-dmaap 1 Thu Jan 23 20:36:27 2020 FAILED dmaap-5.0.0 onap dev-msb 1 Thu Jan 23 20:41:29 2020 DEPLOYED msb-5.0.0 onap dev-policy 1 Thu Jan 23 20:41:30 2020 DEPLOYED policy-5.0.0 onap
7. To delete the installed chart, use helm purge
helm del --purge <release_name>
Example : helm del --purge dcae-dashboard
root@onap2-nfs:~/oom/kubernetes/dcaegen2/components# kubectl get pods -n onap | egrep "inventory|dashboard" dcae-dashboard-dcae-dashboard-6b5c9ccd4c-lx7cw 1/2 Terminating 0 6m55s dcae-dashboard-dcae-dashboard-pg-primary-d8b649c7-gqrsx 0/1 Terminating 0 6m55s dcae-dashboard-dcae-dashboard-pgpool-7bcf656f68-4555q 0/1 Terminating 0 6m55s dev-dcaegen2-dcae-inventory-api-6d98b567d4-zx6zq 1/1 Running 0 3d19h root@onap2-nfs:~/oom/kubernetes/dcaegen2/components# kubectl get pods -n onap | egrep "inventory|dashboard" dev-dcaegen2-dcae-inventory-api-6d98b567d4-zx6zq 1/1 Running 0 3d19h
Note: For redeploy, follow step #6 again.
Development/build of components in Windriver
- Find the k8s node server where POD is running using kubectl
kubectl get pods -n onap -o=wide | grep <podname>
- Login to k8s node server (same key/user) and create a sub-folder on your name
- Check if k8s node server has required s/w such as mvn/java etc
- Clone the repo/patchset from gerrit (there is download option in gerrit, when clicked clicked it will give the exact pull request (or checkout) command)
- Build image (using mvn -s <onap settingfile.xml> clean install <optional docker goals>) to generate docker image. ONAP setting file can be referenced from here - Setting Up Your Development Environment#MavenExamplesettings.xml
Once build is successful and docker image generated, retag it for uniqueness ( nexus3.onap.org:10001/onap/<imagename>:<uniquetag>)
server prep# install mvn: apt install maven # install java8 (java 11 is pre-installed) apt install openjdk-8-jdk-headless # export JAVA_HOME To use java8 export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 # download the setting.xml from https://git.onap.org/oparent/plain/settings.xml and use it in mvn install command # initialize git and pull the specific patchset for test root@onap2-k8s-02:/var/tmp/inv# git init # example with specific inventory patch root@onap2-k8s-02:/var/tmp/inv# git pull "https://sh1986@gerrit.onap.org/r/a/dcaegen2/platform/inventory-api" refs/changes/53/101253/3 # Make other updates/fixes as needed root@onap2-k8s-02:/var/tmp/inv# mvn -s ../onap-settings.xml install docker:build
- For redeploy, you can either update the values.yaml charts (under ~/oom/kubernetes/dcaegen2/components) and follow helm commands above to redeploy, or find the corresponding deployment descriptor (via kubectl) and update the version and save (k8s would automatically terminate current pod and redeploy new based on version provided). Note: Any changes done on helm charts/values.yaml should be submitted into OOM repository (https://git.onap.org/oom/tree/kubernetes/dcaegen2/components) after new images has been released in DCAE.update deploy
##Example of inventory build and using new built image for deployment root@onap2-k8s-02:/var/tmp/inv# docker images|grep inv nexus3.onap.org:10003/onap/org.onap.dcaegen2.platform.inventory-api 3.4.1-SNAPSHOT e4285faa9608 28 seconds ago 124MB nexus3.onap.org:10003/onap/org.onap.dcaegen2.platform.inventory-api 3.4.1-SNAPSHOT-20200214T202415ZZ e4285faa9608 28 seconds ago 124MB nexus3.onap.org:10003/onap/org.onap.dcaegen2.platform.inventory-api latest e4285faa9608 28 seconds ago 124MB 10.12.5.2:5000/onap/org.onap.dcaegen2.platform.inventory-api 3.4.0 088614c21f0f 6 months ago 105MB nexus3.onap.org:10001/onap/org.onap.dcaegen2.platform.inventory-api 3.4.0 088614c21f0f 6 months ago 105MB root@onap2-k8s-02:/var/tmp/inv# ##use: ##nexus3.onap.org:10003/onap/org.onap.dcaegen2.platform.inventory-api:3.4.1-SNAPSHOT-20200214T202415ZZ ##on k8s master find the corresponding deployment root@onap2-nfs:~# kubectl get deployments -n onap -o wide|grep inve dev-dcaegen2-dcae-inventory-api 1/1 1 1 21d dcae-inventory api ##Edit deployement, change: ##image: nexus3.onap.org:10001/onap/org.onap.dcaegen2.platform.inventory-api:3.4.0 ##to ##image: nexus3.onap.org:10003/onap/org.onap.dcaegen2.platform.inventory-api:3.4.1-SNAPSHOT-20200214T202415ZZ ##and imagePullPolicy: Never root@onap2-nfs:~# kubectl edit deployments -n onap dev-dcaegen2-dcae-inventory-api deployment.extensions/dev-dcaegen2-dcae-inventory-api edited