Overview
In the Casablanca release ONAP has become very large with many Helm charts ( ~6 x that of Amsterdam ). Every Helm chart contains some amount of external configuration and, unfortunately, there is a limit to the amount of configuration that can exist in a Helm chart (1 MB). As of Casablanca, we in ONAP have exceeded this limit.
ONAP is installed as an umbrella (ie. parent) chart containing many subcharts, each with configuration. The total amount of configuration that resides in configmaps within K8s (exceeding 1MB) causes an installation of ONAP to fail. To work around this issue, a Helm plugin has been introduced that will install/upgrade ONAP by deploying the parent chart and each subchart within its own Helm "release". It is important to note that all releases must be deployed within the same Kubernetes namespace in order for communication between the components to succeed.
Disclaimer:
The plugins described here were introduced to address config map limitations in the Casablanca release. There are alternative projects, such as https://github.com/roboll/helmfile, that can also solve this problem.
The decision to not use an existing project was due to:
- not wanting to introduce a new project-specific deployment specification this late in the release cycle
- the desire to stay as close to an existing Helm solution as possible (we anticipate significant improvements in Helm 3)
That said, the use of deploy and undeploy plugins can be viewed as a temporary solution. In their current state, they have not been hardened (eg. resilient to networking errors) but are made available to unblock installation of ONAP. To avoid networking errors that can cause some of the sub-charts to fail to deploy, it is recommend that Helm deploy and undeploy commands execute from within the same network (e.g Rancher node or Jumpnode) as the K8s cluster you are deploying to.
Install Helm Plugins (deploy & undeploy)
Clone oom repository
Copy oom/kubernetes/helm/plugins directory into your local ~/.helm/ folder.
# ubuntu sudo cp -R ~/oom/kubernetes/helm/plugins/ ~/.helm # mac sudo cp -R ~/oom/kubernetes/helm/plugins/* ~/.helm/plugins
Verify plugins installed correctly
Execute 'helm' (no arguments) should show both 'deploy' and 'undeploy' in list of available commands.
>sudo helm ....... Usage: Available Commands: dependency manage a chart's dependencies template locally render templates |
|---|
Deploying ONAP
Deploy from public Helm Chart Repository
Example is using one of the public Helm Chart repos https://nexus.onap.org/content/sites/ (ie. staging).
Prerequisite:
- helm repo add staging https://nexus.onap.org/content/sites/oom-helm-staging/
helm deploy demo staging/onap --namespace onap
Deploy from cloned OOM codebase
Prerequisites:
- clone oom repository
- cd oom/kubernetes
- make repo
- make; make onap
- use sudo for helm commands except when running root
sudo helm deploy demo local/onap --namespace onap
or
sudo helm deploy demo ./onap --namespace onap
Results
ONAP pods deployed into the 'onap' Kubernetes namespace. Each application project is installed as a Helm Release that is prefixed by the parent release name (e.g. "demo-")
> helm list |
|---|
Advanced Options
Note that any overrides that can be used using Helm install or update can be applied.
Customizing ONAP deployment using overrides *
helm deploy demo ./onap --namespace onap -f ~/overrides.yaml --set vid.enabled=false
Update (or install) a specific ONAP component *
helm deploy demo-robot ./onap --namespace onap -f ~/overrides.yaml --set vid.enabled=false
* Note that in order for any changes to a Helm Chart to take affect, a make is required (e.g. "make; make onap" or "make robot; make onap")
Helm Deploy plugin logs
ubuntu@a-ld0:~$ sudo ls ~/.helm/plugins/deploy/cache/onap/logs/onap- onap-aaf.log onap-cli.log onap-dmaap.log onap-multicloud.log onap-portal.log onap-sniro-emulator.log onap-vid.log onap-aai.log onap-consul.log onap-esr.log onap-oof.log onap-robot.log onap-so.log onap-vnfsdk.log onap-appc.log onap-contrib.log onap-log.log onap-policy.log onap-sdc.log onap-uui.log onap-vvp.log onap-clamp.log onap-dcaegen2.log onap-msb.log onap-pomba.log onap-sdnc.log onap-vfc.log
Undeploying ONAP
Undeploy entire ONAP deployment
helm undeploy demo --purge
Undeploy specific ONAP component
helm undeploy demo-robot --purge
19 Comments
Brian Freeman
Can I do
helm deploy demo staging/onap --namespace onap -f ~/integration-override.yaml --set aaf.enabled=false
helm deploy demo staging/onap --namespace onap -f ~/integration-override.yaml --set aaf.enabled=true
to upgrade an existing deployment with a new aaf chart/container ?
Andrew Fenner
Hi,
I didn't find that this worked i.e. it didn't delete the deployment and recreate it.
But its easy to work around.
You can do a
helm ls
and find the part of onap you are interested in and then
helm delete <required deployment> --purge
and then
helm deploy demo staging/onap --namespace onap -f ~/integration-override.yaml
will put it back
/Andrew
Kiran Kamineni
How would cross project initContainer dependencies work in this scenario where a pod might be waiting on a job from a different project?
Would we have to name such jobs with hardcoded values instead of using the Release.Name prefix?
Michael O'Brien
Use sudo for helm commands - this page and the following both flag this
https://lists.onap.org/g/onap-discuss/topic/oom_casablanca_install/28513090?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,28513090
https://git.onap.org/logging-analytics/tree/deploy/cd.sh#n139
Cloud Native Deployment#UndercloudInstall-Rancher/Kubernetes
Michael O'Brien
mike notes for helping adjust the RTDs
OOM-1543 - Getting issue details... STATUS
Need to triage the logs for failed deployments
2 issues
sdc does not deploy - how to get the logs
aai deploys but is not listed - so we cannot use the plugin - and the base install/upgrade commands are blocked by the configmap issue
onap onap-aai-aai-resources-f75d854cd-l4spw 2/2 Running 0 22h
hao xing
Hello, I have an openstack environment, I followed the instructions to setup rancher and kubernetes cluster, now I am ready to run helm to install ONAP, do I run helm on the kubernetes node or the rancher VM?
thanks
Brendan Kehoe
Surely you can choose and they can be installed on any machine?
kubectl and helm have been installed (as a convenience) onto the rancher and kubernetes hosts. Typically you would install them both on your PC and remotely connect to the cluster.
https://onap.readthedocs.io/en/latest/submodules/oom.git/docs/oom_setup_kubernetes_rancher.html#configure-kubectl-and-helm
hao xing
Thank you so much.
Now I have helm installed on my local linux box following the instructions from this page "ONAP on kubernetes with rancher" (https://docs.onap.org/en/casablanca/submodules/oom.git/docs/oom_setup_kubernetes_rancher.html#onap-on-kubernetes-with-rancher),
next I run helm to deploy ONAP following this page "OOM quick start guide "(https://docs.onap.org/en/casablanca/submodules/oom.git/docs/oom_quickstart_guide.html#quick-start-label),
it failed, I think I have to modify the values.yaml file for my environment, I am trying to find out more instructions about the OOM configuration, but not very successful. Is there any detailed configuration available?
Brendan Kehoe
What failed and what was failure message?
hao xing
I could only get about 20 pods running, most of them are not coming up, and a lot of failures are similar to this:
Failed to pull image "nexus3.onap.org:10001/onap/so/sdnc-adapter:1.3.2": rpc error: code = Canceled desc = context canceled
Failed to pull image "nexus3.onap.org:10001/mariadb:10.1.11": rpc error: code = Canceled desc = context canceled
Brendan Kehoe
The last manual installation, I think these are the only values I changed in the OOM repository where I ran make all:
kubernetes/onap/values.yaml
- openStackUserName: "vnf_user"
- openStackKeyStoneUrl: "http://1.2.3.4:5000"
- openStackServiceTenantName: "service"
- openStackEncryptedPasswordHere: "c124921a3a0efbe579782cde8227681e"
kubernetes/robot/values.yaml
-openStackKeyStoneUrl: "http://1.2.3.4:5000"
-openStackPassword: "tenantPassword"
-openStackTenantId: "47899782ed714295b1151681fdfd51f5"
-openStackUserName: "tenantUsername"
hao xing
Brendan:
In the onap/values.yaml file, the appc has several config parameters:
appc:
enabled: true
config:
openStackType: OpenStackProvider
openStackName: OpenStack
openStackKeyStoneUrl: http://localhost:8181/apidoc/explorer/index.html
should I change the openStackKeyStoneUrl to "http://1.2.3.4:5000"?
thanks
Brendan Kehoe
These values are probably needed and if they are they have to be real values for your environment. 1.2.3.4 is an example and used to be in the OOM repository onap/values.yaml. You need to find these for your Open Stack installation. For example in Mirantis Open Stack it is under Identity, Projects.
Brendan Kehoe
Also it may be useful for you to clone the integration git repository and run the following script:
https://gerrit.onap.org/r/gitweb?p=integration.git;a=blob;f=version-manifest/src/main/scripts/update-oom-image-versions.sh;hb=HEAD
This updates the Docker image versions in the values.yaml files in OOM to the values in the file supplied. You might want to run this script with the release version manifest:
https://gerrit.onap.org/r/gitweb?p=integration.git;a=blob;f=version-manifest/src/main/resources/docker-manifest.csv;hb=HEAD
The second argument is the OOM repository directory
eg.
./update-oom-image-versions.sh ../resources.docker-manifest.csv /home/ubuntu/oom
Michael O'Brien
As I understand it - the oom repository is the final word - not the other way around - as in the docker manifest is periodically updated to reflect the image tags in oom. I would recommend not changing the tags in the oom repo before deployment. The manifest is good for 90-100% prepull only.
/michael
Michael O'Brien
still a bit in the air about the source of truth - manifest or oom values.yaml - integration team may be overriding the images tags - everyone else is not - perhaps we should
put the decision in TSC-86 - Getting issue details... STATUS
https://lists.onap.org/g/onap-discuss/topic/oom_onap_deployment/28883609?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,28883609
quote:
Brian is right – anything over 2 hours will not restart – bounce your particular pod –set ?.enabled=false/true – with a dockerdata-nfs clean in the middle – and a pv/pvc deletion for some rogue pods that have pvs outside the loop – only if you get a pv exists exception on restart.
https://wiki.onap.org/display/DW/ONAP+Development#ONAPDevelopment-Bounce/Fixafailedcontainer
change –all to a particular pod – or use a –force delete like
kubectl delete pod $ENVIRON-aaf-sms-vault-0 -n $ENVIRON --grace-period=0 --force
https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-RemoveaDeployment
https://git.onap.org/logging-analytics/tree/deploy/cd.sh#n79
The above script with a 3 min timeout between the 30 pod deploys brings up the whole system except some minor issues with a couple pods
For your situation – make sure healthcheck is 50/51 before attempting to use the cluster
3.2.2 ete-k8s.sh health test: 51 critical tests, 7 passed, 44 failed
Should be 50 passed (verified 20181229 for 3.0.0-ONAP (Casablanca manifest)
Guys,
A secondary question – are the values.yaml files in OOM the truth or is the manifest file the truth.
I also need a bit of clarification on the source of truth for the image tags in the values.yamls for the 30 components in OOM
My understanding is that the oom repo is the source of truth – and that the manifest file in the integration repo is 99-100% kept up to date with what is deployed – but the manifest is not the truth- if it is the reverse then we would need to either make sure every deployment (all CD systems including mine) and all developers use a generated values.yaml override – or at least force a change in OOM to a values.yaml everytime the manifest is updated – from reviewing patches between oom and integration it looks like image names in oom are the source.
I would like to nail this down because no one I know of adjusts any of the merged docker image tag names that OOM uses to deploy – only the prepull script makes use of the manifest – as far as I know – if not we need to adjust the documentation so that we are running the same way integration does.
The following assumes the manifest is the same as the values.yamls
https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-Nexus3proxyusageperclusternode
Currently we use the manifest as the docker_prepull.sh target (as it is easier to parse and pull from there instead of mining OOM like the previous iteration did) – however oom when we deploy – will still use what is hardcoded into each values.yaml.
sudo nohup ./docker_prepull.sh -b casablanca -s nexus4.onap.cloud:5000&pulls fromhttps://git.onap.org/logging-analytics/tree/deploy/docker_prepull.sh#n35
If there is a script that we run somewhere that takes the this manifest and overrides all the image tags as a values.yaml overlay before deployment – let us know. The current wiki and readthedocs do not mention this.
Current Process for deployment
https://onap.readthedocs.io/en/latest/submodules/oom.git/docs/oom_quickstart_guide.html
https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-Scriptedundercloud(Helm/Kubernetes/Docker)andONAPinstall-clustered
thanks guys
/michael
From: Dominique Deschênes <dominique.deschenes@gcgenicom.com>
Sent: Thursday, January 3, 2019 2:37 PM
To: Borislav Glozman <Borislav.Glozman@amdocs.com>; bf1936@att.com; onap-discuss@lists.onap.org
Cc: Jacques Faucher <jacques.faucher@gcgenicom.com>; Jasmin Audet <jasmin.audet@gcgenicom.com>; Michael O'Brien <Frank.Obrien@amdocs.com>
Subject: Re[2]: [onap-discuss] OOM ONAP Deployment
Hi,
Michael O'Brien
Tracking issue with the deploy plugin not differentiating between an install and upgrade as of this week - (no script changes - so this is an env issue for now)
Michal Ptacek
we have some challenges with reinstalling deployments:
[root@tomas-infra ~]# helm ls -a
[root@tomas-infra ~]# kubectl get pods -n onap
No resources found.
[root@tomas-infra ~]# kubectl get deployments.extensions --all-namespaces
NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kube-system heapster 1 1 1 1 1h
kube-system kube-dns 1 1 1 1 1h
kube-system kubernetes-dashboard 1 1 1 1 1h
kube-system monitoring-grafana 1 1 1 1 1h
kube-system monitoring-influxdb 1 1 1 1 1h
kube-system tiller-deploy 1 1 1 1 1h
[root@tomas-infra ~]# helm deploy dev local/onap --namespace onap
fetching local/onap
release "dev" deployed
release "dev-sdnc" deployed
dev-sdnc 1 Wed Jan 23 09:53:21 2019 FAILED sdnc-3.0.0 onap
#from kubectl logs tiller-deploy-78db58d887-gthv8 -n kube-system
[tiller] 2019/01/23 14:53:22 warning: Release "dev-sdnc" failed: deployments.extensions "dev-sdnc-sdnc-dgbuilder" already exists
[storage] 2019/01/23 14:53:22 updating release "dev-sdnc.v1"
[tiller] 2019/01/23 14:53:22 failed install perform step: release dev-sdnc failed: deployments.extensions "dev-sdnc-sdnc-dgbuilder" already exists
does anyone know how to clean stack properly ?
anyway it doesn't looks like cleaning problem but some race-conditions during creating (not reproducible on fresh vanilla system)
bao air
Hello, I am a student. I get a problem
My release is deployed, but there is no pods.
I put "helm ls" in but the list is blank. However there is no error.
I don't know why, who can help me.
Thank you very much.