Work in Progress - 9/9/2020
High Level Overview
- Ensure NFD (Node feature discovery - K8S CNCF project) deploy to Multicloud + K8s Plugin is working and labeling Kubernetes nodes
- Add registrationHandler to K8s Plugin and expand K8s Plugin API to work with ESR VIM registration of K8s Clusters
- Use POST/DELETE/GET of http://{{MSB_IP}}:{{MSB_PORT}}/api/multicloud-k8s/v1/{{cloud-owner}}/{{cloud-region-id}}/registry
- Ensure registrationHandler can get K8s feature labels - (labels discovered and placed by NFD representing hardware features)
- Ensure registrationHandler can populate A&AI cloud-region/tenant/compute flavors based on cluster under registration
- Register K8s cluster internally with the K8s Plugin
- Add Node App in K8s Plugin to retrieve node labels from a cluster
- Identify and Address any gaps in policy and OOF to do match making of CNFs/VNFs using K8s Plugin
Kubernetes clusters using ONAP Multicloud K8s Plugin Project for R4/R5 do not report hardware features to AAA&IAI.
Consequently during the CNF/VNF life cycle CNFs/VNFs that require or recommend specific hardware during instantiation cannot dynamically reach the correct cloud cluster and node that provides the needed hardware capabilities.
This Frankfurt Release epic (
Jira | ||||||
---|---|---|---|---|---|---|
|
Overall Flow Diagram for HPA for K8s
Discovery and Labeling of Node Features
The below figure shows a K8s cluster registering with the K8s plugin, wherein capabilities are transmitted
Discovery of HPA features will be done using the node feature discovery [NFD] addon (https://github.com/kubernetes-sigs/node-feature-discovery) for kubernetes. NFD will be deployed as a daemonSet, with pods on each node and will discover and label hardware enabled hardware features on each kubernetes nodes node with supported features labels (https://github.com/kubernetes-sigs/node-feature-discovery#feature-labels).
For SR-IOV Add Network Controllers Class "0200" to NFD-Worker Config File on deploy to label network cards on nodes
For HPA Features not labeled by NFD, investigate creation of Feature Detector Hooks (User Specific Features)
Registering of K8s Clusters
The GREEN FLOW is Register HPA features to AAI.
Registration of K8s Cluster will happen at the ESR UI (see: https://onap.readthedocs.io/en/latest/guides/onap-user/cloud_site/openstack/index.html - use k8s for cloud-type). ESR will then send a POST request to
http://{{MSB_IP}}:{{MSB_PORT}}/api/v1/{{cloud-owner}}/{{cloud-region-id}}/registry
Multicloud K8s Plugin registrationHandler will receive said POST and register the new Kubernetes cluster with the K8s Plugin using the connectivity code/API.
Note | ||
---|---|---|
| ||
We need to check what is sent from ESR to registery URL. Currently the K8s Plugin uses a kubeconfig file to access different k8s clusters. We will either need to generate a kubeconfig file (does ESR send enough info to do this? If not we might need to extend ESR) or access the cluster using another form of authentication (check k8s code and see if this is possible - auth token perhaps?
|
The registrationHandler will then query the k8s cluster under registration using the Node App or Node Plugin code to get a list of nodes and the labels for each node.
- registrationHandler - onap/multicloud-k8s/src/k8splugin/api/registrationhandler.go
- Registers K8s cloud to K8s plugin using onap/multicloud-k8s/src/k8splugin/internal/connection/connection.go – check into if we need kubeconfig or ESR change
- Requires update of onap/multicloud-k8s/src/k8splugin/api/api.go
- Invokes AAI Client to K8s AAI Module to update AAI with Cloud/Tenant/Flavor information for the cluster
Node App or Node Plugin
- Node App - onap/multicloud-k8s/src/k8splugin/internal/app/node or onap/multicloud-k8s/src/k8splugin/internal/plugin/node.go
- Requests labels from nodes in K8s cluster
Requires update of onap/multicloud-k8s/src/k8splugin/internal/app/client.go extend to node Resource (helm.KubernetesResource does not have resource type)
Reporting of Cluster Info to AA&I
Reporting of HPA features to AA&I is done by the registrationHandler. Data will be munged to Reporting of HPA features will be done in Multicloud K8s Plugin registrationHander by discovering node labels on each node in a cluster and pushing munged cloud-region/tenant/flavors info based on schema for that cluster to A&AI.and reported to AA&I. registrationHandler is planning on using AA&I module under development for Frankfurt to report to AA&I. In this case we will send a request to the AA&I module instructing it to update the cloud-region, create a tenant and create flavors that match the node feature labels in said cloud-region.
Convert node feature label to flavor properties
below table is example how to convert node feature label to flavor properties
node feature label | Openstack flavor properties |
---|---|
feature.node.kubernetes.io/network-sriov.capable=True feature.node.kubernetes.io/network-sriov.configured=True feature.node.kubernetes.io/pci-1200_8086.present=True Comments: The other parameter we can set default because node feature label can not provide. |
We can get the latest mapping table in HPA Policies and Mappings + K8s
Update of CNF Artifacts to Include nodeSelector
CNF CNF and VNF helm charts will be updated with nodeSelector requirements & recommendations in line with node labels created by NFD during feature discovery.
Example Node Features:
Code Block | ||
---|---|---|
| ||
# Hardware Feature Labels: feature.node.kubernetes.io/cpu-cpuid.AESNI=true, feature.node.kubernetes.io/cpu-cpuid.AVX2=true, feature.node.kubernetes.io/cpu-cpuid.AVX=true, feature.node.kubernetes.io/cpu-cpuid.FMA3=true, feature.node.kubernetes.io/cpu-cpuid.IBPB=true, feature.node.kubernetes.io/cpu-cpuid.STIBP=true, feature.node.kubernetes.io/cpu-hardware_multithreading=true, feature.node.kubernetes.io/cpu-pstate.turbo=true, feature.node.kubernetes.io/cpu-rdt.RDTCMT=true, feature.node.kubernetes.io/cpu-rdt.RDTMON=true, feature.node.kubernetes.io/memory-numa=true, feature.node.kubernetes.io/network-sriov.capable=true, feature.node.kubernetes.io/pci-0300_102b.present=true, # GPU Present feature.node.kubernetes.io/pci-0200_8086.present=true, # Network Card Present feature.node.kubernetes.io/storage-nonrotationaldisk=true # Software Feature Labels: feature.node.kubernetes.io/kernel-config.NO_HZ=true, feature.node.kubernetes.io/kernel-config.NO_HZ_FULL=true, feature.node.kubernetes.io/kernel-version.full=3.10.0-957.el7.x86_64, feature.node.kubernetes.io/kernel-version.major=3, feature.node.kubernetes.io/kernel-version.minor=10, feature.node.kubernetes.io/kernel-version.revision=0, feature.node.kubernetes.io/system-os_release.ID=centos, feature.node.kubernetes.io/system-os_release.VERSION_ID.major=7, feature.node.kubernetes.io/system-os_release.VERSION_ID.minor=, feature.node.kubernetes.io/system-os_release.VERSION_ID=7 |
Where needed add resource limits and request to ResourceBundle charts. I.E.:
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
Charts to Update
https://github.com/onap/multicloud-k8s/tree/master/kud/demo/firewall
https://github.com/onap/multicloud-k8s/tree/master/kud/tests/vnfs/edgex/helm/edgex
For CPU Pinning etc - investigate CMK for kubernetes (see links at bottom of page) and specifying nodeSelector of "cmk.intel.com/cmk-node": "true"
Instantiation of K8s HPA
See Deploying vFw and EdgeXFoundry Services on Kubernets Cluster with ONAP for instantiation flow for K8s. See https://docs.onap.org/en/latest/submodules/integration.git/docs/docs_vfwHPA.html?highlight=hpa VFW deployment with HPA for HPA use case.
If we follow the deploying vFW for K8s on ONAP, the only difference in the process for HPA would be the addition of homing (see VFW deployment with HPA docs above) to the SO instantiation request. This enables a call to OOF that uses AA&I and Policy to select the correct cluster.
Policy and OOF will use the same path to discover capabilities of K8s nodes as are done for OpenStack. We are using the same data model (OpenStack cloud-region/tenant/flavors) from AA&I. OOF will use AA&I Policy and OOF can then discover capabilities of K8s nodes and use that information to home particular CNF workloads to nodes which can accelerate their function. Further Multicloud will use labels when instantiating CNF workload to ensure kubernetes places the workloads on the correct node.
Addition of NFD
...
cloud-regions which contain requested features. When K8s plugin instantiates said workloads Kubernetes will read nodeSelector preferences and place each workload on nodes which contain needed features to accelerate their function.
High Level Task Overview
Jira | ||||||
---|---|---|---|---|---|---|
|
- Ensure NFD (Node feature discovery - K8S CNCF project) deploy to Multicloud + K8s Plugin is working and labeling Kubernetes nodes
Jira server ONAP JIRA serverId 425b2b0a-557c-3c0c-b515-579789cceedb key MULTICLOUD-741 - Add registrationHandler to K8s Plugin and expand K8s Plugin API to work with ESR VIM registration of K8s Clusters
Jira server ONAP JIRA serverId 425b2b0a-557c-3c0c-b515-579789cceedb key MULTICLOUD-740 - Use POST/DELETE/GET of http://{{MSB_IP}}:{{MSB_PORT}}/api/multicloud-k8s/v1/{{cloud-owner}}/{{cloud-region-id}}/registry
- Ensure registrationHandler can get K8s feature labels - (labels discovered and placed by NFD representing hardware features)
- Ensure registrationHandler can populate A&AI cloud-region/tenant/compute flavors based on cluster under registration
Jira server ONAP JIRA serverId 425b2b0a-557c-3c0c-b515-579789cceedb key MULTICLOUD-739 - Register K8s cluster internally with the K8s Plugin
- Add Node App in K8s Plugin to retrieve node labels from a cluster
- Identify and Address any gaps in policy and OOF to do match making of CNFs/VNFs using K8s Plugin
- Update CNF packages (Resource Bundles, CSAR) to include nodeSelector and resource requests and limits
- Update docs for HPA and K8s and Multicloud and ESR to reflect any change
Jira server ONAP JIRA serverId 425b2b0a-557c-3c0c-b515-579789cceedb key MULTICLOUD-742
Identify & Address Gaps in AAI/ESR Cloud Registration
Jira | ||||||
---|---|---|---|---|---|---|
|
Identify & Address Gaps to Policy and OOF for K8s HPA Homing
As of now there are not any known gaps in OOF/Policy that would inhibit this from working.
Background Links:
Deploying vFw and EdgeXFoundry Services on Kubernets Cluster with ONAP
vFW K8S examples mapping to AAI
Setting up Closed Loop for K8S vFW - initial pass
https://github.com/intel/CPU-Manager-for-Kubernetes
https://github.com/kubernetes-sigs/node-feature-discovery
https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/
https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/
https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/
https://kubernetes.io/docs/tasks/administer-cluster/extended-resource-node/