...
During the Amsterdam release OOM provided a set of capabilities to deploy some or all the ONAP components rapidly and efficiently as a cloud native application with the Kubernetes container orchestration system (note that DCAE is an exception here as DCAE provides its own orchestration system). Each of the components has a deployment specification that describes not only the containers and the container requirements but the relationships or dependencies between the containers. These dependencies dictate the order in-which the containers are started for the first time such that such dependencies are always met without arbitrary sleep times between container startups. For example, the SDC back-end container requires the Elastic-Search, Cassandra and Kibana containers within SDC to be ready and is also dependent on DMaaP (or the message-router) to be ready before becoming fully operational.
Another feature that may assist in achieving a repeatable deployment in the presence of faults that may have reduced the capacity of the cloud is assigning priority to the containers such that mission critical components have the ability to evict less critical components. Kubernetes provides this capability with Pod Priority and Preemption.
Prior to having more advantage carrier grade features available, the ability to at least be able to re-deploy ONAP (or a subset of) reliably provides a level of confidence that should an outage occur the system can be brought back on-line predictably.
Backup and Restore
Here is the deployment specification that describes these dependencies:
| Code Block | ||
|---|---|---|
| ||
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: sdc-be
name: sdc-be
namespace: "{{ .Values.nsPrefix }}-sdc"
spec:
selector:
matchLabels:
app: sdc-be
template:
metadata:
labels:
app: sdc-be
name: sdc-be
annotations:
pod.beta.kubernetes.io/init-containers: '[
{
"args": [
"--container-name",
"sdc-es",
"--container-name",
"sdc-cs",
"--container-name",
"sdc-kb"
],
"command": [
"/root/ready.py"
],
"env": [
{
"name": "NAMESPACE",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.namespace"
}
}
}
],
"image": "{{ .Values.image.readiness }}",
"imagePullPolicy": "{{ .Values.pullPolicy }}",
"name": "sdc-be-readiness"
},
{
"args": [
"--container-name",
"dmaap"
],
"command": [
"/root/ready.py"
],
"env": [
{
"name": "NAMESPACE",
"value": "{{ .Values.nsPrefix }}-message-router"
}
],
"image": "{{ .Values.image.readiness }}",
"imagePullPolicy": "{{ .Values.pullPolicy }}",
"name": "sdc-dmaap-readiness"
}
]'
spec:
containers:
- env:
- name: ENVNAME
value: AUTO
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
image: {{ .Values.image.sdcBackend }}
imagePullPolicy: {{ .Values.pullPolicy }}
name: sdc-be
volumeMounts:
- mountPath: /usr/share/elasticsearch/data/
name: sdc-sdc-es-es
- mountPath: /root/chef-solo/environments/
name: sdc-environments
- mountPath: /etc/localtime
name: sdc-localtime
readOnly: true
- mountPath: /var/lib/jetty/logs
name: sdc-logs
- mountPath: /var/log/onap
name: sdc-logs-2
- mountPath: /tmp/logback.xml
name: sdc-logback
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "export LOG=wait_logback.log; touch $LOG; export SRC=/tmp/logback.xml; export DST=/var/lib/jetty/config/catalog-be/; while [ ! -e $DST ]; do echo 'Waiting for $DST...' >> $LOG; sleep 5; done; sleep 2; /bin/cp -f $SRC $DST; echo 'Done' >> $LOG"]
ports:
- containerPort: 8443
- containerPort: 8080
readinessProbe:
tcpSocket:
port: 8443
initialDelaySeconds: 5
periodSeconds: 10
- image: {{ .Values.image.filebeat }}
imagePullPolicy: {{ .Values.pullPolicy }}
name: filebeat-onap
volumeMounts:
- mountPath: /usr/share/filebeat/filebeat.yml
name: filebeat-conf
- mountPath: /var/log/onap
name: sdc-logs-2
- mountPath: /usr/share/filebeat/data
name: sdc-data-filebeat
volumes:
- name: filebeat-conf
hostPath:
path: /dockerdata-nfs/{{ .Values.nsPrefix }}/log/filebeat/logback/filebeat.yml
- name: sdc-logs-2
emptyDir: {}
- name: sdc-data-filebeat
emptyDir: {}
- name: sdc-logback
hostPath:
path: /dockerdata-nfs/{{ .Values.nsPrefix }}/log/sdc/be/logback.xml
- name: sdc-sdc-es-es
hostPath:
path: /dockerdata-nfs/{{ .Values.nsPrefix }}/sdc/sdc-es/ES
- name: sdc-environments
hostPath:
path: /dockerdata-nfs/{{ .Values.nsPrefix }}/sdc/environments
- name: sdc-localtime
hostPath:
path: /etc/localtime
- name: sdc-logs
hostPath:
path: /dockerdata-nfs/{{ .Values.nsPrefix }}/sdc/logs
imagePullSecrets:
- name: "{{ .Values.nsPrefix }}-docker-registry-key" |
Another feature that may assist in achieving a repeatable deployment in the presence of faults that may have reduced the capacity of the cloud is assigning priority to the containers such that mission critical components have the ability to evict less critical components. Kubernetes provides this capability with Pod Priority and Preemption.
Prior to having more advantage carrier grade features available, the ability to at least be able to re-deploy ONAP (or a subset of) reliably provides a level of confidence that should an outage occur the system can be brought back on-line predictably.
Backup and Restore
A critical factor in being able to recover from an ONAP outage is to ensure that critical state isn't lost after a failure. Much like ephemeral storage on VMs; any state information stored within a container will be lost once the container is restarted - containers are managed as Cattle not Pets. To ensure that critical state information is retained after a failure, the OOM deployment specifications for the ONAP components use the Kubernetes concept of a Persistent Volumes, an external storage facility that has its own lifecycle. The use of a persistent volume is specified in the ONAP deployment specifications. Here is an example from the sdnc db-deployment.yaml:
| Code Block | ||
|---|---|---|
| ||
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: sdnc-dbhost
namespace: "{{ .Values.nsPrefix }}-sdnc"
spec:
selector:
matchLabels:
app: sdnc-dbhost
template:
metadata:
labels:
app: sdnc-dbhost
name: sdnc-dbhost
spec:
containers:
- env:
- name: MYSQL_ROOT_PASSWORD
value: openECOMP1.0
- name: MYSQL_ROOT_HOST
value: '%'
image: {{ .Values.image.mysqlServer }}
imagePullPolicy: {{ .Values.pullPolicy }}
name: sdnc-db-container
volumeMounts:
- mountPath: /etc/localtime
name: localtime
readOnly: true
- mountPath: /var/lib/mysql
name: sdnc-data
ports:
- containerPort: 3306
readinessProbe:
tcpSocket:
port: 3306
initialDelaySeconds: 5
periodSeconds: 10
volumes:
- name: localtime
hostPath:
path: /etc/localtime
- name: sdnc-data
persistentVolumeClaim:
claimName: sdnc-db
imagePullSecrets:
- name: "{{ .Values.nsPrefix }}-docker-registry-key" |
At the bottom of the deployment specification is a list of the volumes, localtime and sdnc-data which uses an external persistentVolumeClaim of sdnc-db. This claim is specified as follows:
| Code Block | ||
|---|---|---|
| ||
apiVersion: v1
kind: PersistentVolume
metadata:
name: "{{ .Values.nsPrefix }}-sdnc-db"
namespace: "{{ .Values.nsPrefix }}-sdnc"
labels:
name: "{{ .Values.nsPrefix }}-sdnc-db"
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
hostPath:
path: /dockerdata-nfs/{{ .Values.nsPrefix }}/sdnc/data
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: sdnc-db
namespace: "{{ .Values.nsPrefix }}-sdnc"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2Gi
selector:
matchLabels:
name: "{{ .Values.nsPrefix }}-sdnc-db" |
A critical factor in being able to recover from an ONAP outage is to ensure that critical state isn't lost after a failure. Much like ephemeral storage on VMs; any state information stored within a container will be lost once the container is restarted - containers are managed as Cattle not Pets. To ensure that critical state information is retained after a failure, the OOM deployment specifications for the ONAP components use the Kubernetes concept of a Persistent Volumes, an external storage facility that has its own lifecycle. Many different types of storage are supported by this capability such as: GCEPersistentDisk, AWSElasticBlockStore, AzureFile, AzureDisk, FC (Fibre Channel), FlexVolume, Flocker, NFS, iSCSI, RBD (Ceph Block Device), CephFS, Cinder (OpenStack block storage), Glusterfs, VsphereVolume, Quobyte Volumes, HostPath (Single node testing only), VMware Photon, Portworx Volumes, ScaleIO Volumes, and StorageOS.
...
The OOM project is not responsible to creating highly available versions of all of the ONAP components but does provide via Kubernetes many built in facilities to build clustered, highly available systems including: Services with load-balancers (including support for External Load Balancers), Ingress Resources, and Replica Sets. Some of the open-source projects that form the basis of ONAP components directly support clustered configurations like ODL with instructions on Setting Up Clustering or MaridDB Getting Started with MariaDB Galera Cluster. .
OOM uses the Kubernetes service abstraction to provide a consistent access point for each of the ONAP components independent of the pod or container architecture of that component. For example, the SDNC component may introduce OpenDaylight clustering as some point and change the number of pods in this component to three or more but this change will be isolated from the other ONAP components by the service abstraction. A service can include a load balancer on its ingress to distribute traffic between the pods and even react to dynamic changes in the number of pods if they are part of a replica set. A replica set is a construct that is used to describe the desired state of the cluster. For example 'replicas: 3' indicates to Kubernetes that a cluster of 3 instances is the desired state. Should one of the members of the cluster fail, a new member will be automatically started to replace it.
...