...
OOM provides two mechanims to monitor the real-time health of an ONAP deployment: a Consul GUI for a human operator or downstream monitoring systems and Kubernetes liveness probes that enable automatic healing of failed containers.
Consul
The liveness probes are described in the following section.
OOM deploys a 3 instance Consul server cluster that provides a real-time health monitoring capability for all of the ONAP components. For each of the ONAP components a Consul health check has been created, here is an example from the AAI model loader:
...
The
| Jira | ||||||
|---|---|---|---|---|---|---|
|
...
Heal
Liveness probes are used by the Kubernetes manager to monitor the real-time health of the containers in an ONAP deployment. If the liveness probe fails, Kubernetes will kill off the failed container and start a new container to replace it. The liveness probe have same parameters as the readiness probes described in the Deploy section. For example, to monitor the SDNC component has following liveness probe can be found in the SDNC DB deployment specification:
...
Note that containers are inherently ephemeral so the healing action destroys failed containers and any state information within it. To avoid a loss of state, a persistent volume should be used to store all data that needs to be persisted over the re-creation of a container. Persistent volumes have been created for the database components of each of the projects and the same technique can be used for all persistent state information.
Clustering and Scaling
In-order to avoid the loss of an entire ONAP component by the failure of a single container, OOM enables the creation of clusters of pods hidden behind a load balancer in the form of a service.
Upgrade
OOM uses Helm K8S package manager to deploy ONAP components.
Each component is arranged in a packaging format called a chart – a collection of files that describe a set of k8s resources.
Helm allows for rolling upgrades of the ONAP component deployed.
To upgrade a component Helm release you will need an updated Helm chart. The chart might have modified, deleted or added values, deployment yamls, and more.
To get the release name use:
helm ls |
To easily upgrade the release use:
helm upgrade [RELEASE] [CHART] |
To roll back to a previous release version use:
helm rollback [flags] [RELEASE] [REVISION] |
for example, to upgrade the onap-mso helm release to the latest MSO container release v1.1.2:
- Edit mso valus.yaml which is part of the chart
Change “mso: nexus3.onap.org:10001/openecomp/mso:v1.1.1” to
“mso: nexus3.onap.org:10001/openecomp/mso:v1.1.2”
- From the chart location run:
helm upgrade onap-mso ./
The previous mso pod will be terminated and a new mso pod with an updated mso container will be created.