Introduction:
In today, many complex applications are consisting in a mixed, complex workload, that is described in many Kubernetes resources, e.g. to be run on a certain cluster, etc. In order to deploy the application, the orchestration task would be requiring dealing with different abstract layers of resources, different templates system mapping, and application packaging. Another challenge is to keep up with changes in the cloud infrastructures features enhancement mapping into abstract resource template.
Application metadata:
In order to describe the containerized application to an orchestrator, there is a need for some metadata to accompany the bundled cloud-native deployment artifacts (e.g. Helm files) and images.
A basic decision that will affect this metadata is what exactly the cloud-native deployment artifacts are, and what implications that has for the way the orchestrator communicates with the cluster.
The primary decision is to require that applications are deployed using Helm v3 or later (ref: https://helm.sh/docs/helm/helm/), and therefore the deployment artifacts are one or more Helm Charts. Furthermore, in order not to limit future choices of tooling, or tooling choices creating dependencies between the orchestrator and the underlying Kubernetes cluster, it is assumed that the interface between the orchestrator and the cluster is the Kubernetes API (ref: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/).
This implies that the orchestrator
- Has an embedded Helm v3 client and is able to use it to deploy the artifacts embedded in the package.
- If it requires any information present in the K8S resource descriptions, it is ably to pre-render the Helm Charts and extract such information.
The most widely used format for describing telco virtualized applications is the ETSI MANO Virtualized Network Function Descriptor (VNFD), ETSI NFV SOL001 specification. This descriptor was created as a vendor neutral way to fully describe a virtualized application, with highly detailed requirements ranging from placement rules, the various virtual components like virtual NICs, CPU and memory requirements, scaling policy, input parameters, and monitoring parameters, etc.
ETSI sought to extend the existing VNFD to cover containerized workloads. It has been pointed out that such efforts have noticeable drawbacks:
- The overall proposed model is essentially duplicating the workload description that K8S (and Helm) provide, but so far with less features.
- The ETSI NFV VNFD (VM and containerized based) definitions could conflict with the Helm Chart definitions, which can cause orchestration confusion and/or failure.
- Non-ETSI-based CNF model and orchestration desire a simplified CNF descriptor
Therefore, this proposal is proposing a new, simple descriptor, the Application Service Descriptor (ASD) with the minimum information for the orchestrator, and pointers to cloud-native artifacts and code (including configuration) required for the LCM implementation. An ASD can describe a complete application / NF, or parts of application / NF.
The ASD allows a clean separation between high-level orchestration, focused on service and resource models, and cloud-native application deployment, implemented via Helm Charts.
Application Service Descriptor model
The tables below summarize the Application Service Descriptor contents.
The overall objective is to keep the items in the descriptor to the bare minimum information, and not duplicate any attributes that might be instead extracted from the Helm Charts. This helps maintain the principle that Helm Charts are the primary deployment artifact for a containerized application and avoids any possible source of error or confusion that such duplication would cause.
Application Service Descriptor (ASD) Information Element (top level)
Attribute | Qualifier | # | Type | Description |
asdId | M | 1 | Identifier | Identifier of this ASD information element. This attribute shall be globally unique. The format will be defined in the data model specification phase. |
asdSchemaVersion | M | 1 | Version | Specifies the version of the ASD’s schema (if we modify an ASD field definition, add/remove field definitions, etc.). |
asdProvider | M | 1 | String | Provider of the AS and of the ASD. |
asdApplicationName | M | 1 | String | Name to identify the Application Service. Invariant for the AS lifetime. |
asdApplicationVersion | M | 1 | Version | Specifies the version of the Application (so, if software, DeploymentArtifacts , ASD values, ... change, this changes). |
asdApplicationInfoName | M | 0..1 | String | Human readable name for the Application service. Can change during the AS lifetime. |
asdInfoDescription | M | 0..1 | String | Human readable description of the AS. Can change during the AS lifetime. |
asdExtCpd | M | 0..N | datatype.ExtCpd | Describes the externally exposed connection points of the application. |
enhancedClusterCapabilities | M | 0..1 | datatype. enhancedClusterCapabilities | A list of expected capabilities of the target Kubernetes cluster to aid placement of the application service on a suitable cluster. |
deploymentItems | M | 1..N | DeploymentItem | Deployment artifacts |
The initial attributes essentially describe the application – a unique identifier, a schema version (that enables versioning the data model of the descriptor itself), and basic metadata, like application name and version and human-readable descriptive fields.
The attribute "asdExtCpd" will be used for exposing endpoints (to enable orchestrators to string together or optimally place linked applications.
The attributes “enhancedClusterCapabilities” provides information which is used to aid placement of the application service on a suitable cluster.
Finally, “deploymentItems” is a list of deployment items, i.e. Helm Charts, that together can deploy an application. The table below shows the information element of these deployment item descriptors.
Deployment Item Information Element
Attribute | Qualifier | # | Content | Description |
deploymentItemId | M | 1 | Identifier | The identifier of this deployment item |
artifactType | M | 1 | String/enum | Specify artifact type. e.g. Helm chart, helmfile, CRD etc. |
artifactId | M | 1 | Identifier (reference to) | Reference to a DeploymentArtifact |
deploymentOrder | M | 0..1 | Integer | Specifies the deployment stage that the DeploymentArtifact belongs to. A lower value specifies that the DeploymentArtifact belongs to an earlier deployment stage, i.e. needs to be installed prior to DeploymentArtifact with higher deploymentOrder values. |
lifecycleParameters | M | 0..1 | List of strings | list of parameters that can be overridden at deployment time (e.g., values for values.yaml in the chart this item references) |
asdExtCpd Information Element
| Attribute | Qualifier | Cardinality | Content | Description |
|---|---|---|---|---|
| description | M | 1 | String | Describes the service exposed. |
| virtualLinkRequirement | M | 1 | String | Refers in an abstract way to the network or multiple networks that the ExtCpd shall be exposed on (ex: OAM, EndUser, backhaul, LI, etc). The intent is to enable a network operator to take decision on to which actual VPN to connect the extCpd to. NOTE 1. |
| interfaceOrder | M | 0..1 | Integer, greater or equal to zero | Mandatory attribute for a secondary network interface (not applicable for a primary network interface). Defines the order in which an the additional/secondary network interface declaration appears in the pod manifest. Note that an SRIOV mated vNIC pair shall be modelled by a single vduCp, and its order value set to the lower value of the two vNIC’s order numbers; the two vNICs are expected to appear in consecutive order on the compute instance, and be attached to same network(s). See NOTE 2 |
| networkInterfaceRealizationRequirements | M | 0..1 | datatype.networkInterfaceRealizationRequirements | Details container implementation specific requirements on the NetworkAttachmentDefinition to . See NOTE 2 & 3. |
| inputParamMappings | M | 0..1 | datatype.extCpd.ParamMappings | Information on what parameters that are required to be provided to the deployment tools for the asdExtCpd instance. |
| resourceMapping | M | 0..1 | String | Kubernetes API resource name for the resource manifest for the service, ingress or pod resource declaring the network interface. Enables, together with knowledge on namespace, the orchestrator to lookup the runtime data related to the extCpd. |
NOTE 1: Corresponds more or less to a virtual_link requirement in ETSI NFV SOL001.
NOTE 2: Applies only for ExtCpds representing secondary network interfaces in a pod.
NOTE 3: Several ExtCpd may refer to same additional network interface requirements.
networkInterfaceRealizationRequirements Information Element
| Attribute | Qualifier | Cardinality | Content | Description |
|---|---|---|---|---|
| trunkMode | M | 0..1 | ”false” | ”true” | If not present or set to”false”, means that this interface shall connect to single network. If set to ”true” then the network interface shall be a trunk interface (connects to multiple VLANS). |
| ipam | M | 0..1 | ”cniManaged” ”userManaged” ”inBand” | The default value (”cniManaged”) means that the CNI specifies how IPAM is done and assigns the IP address to the pod interface. Value ”user” indicates that IPAM is done via the application inside the pod. "inBand" indicates that the application expects to receive the interface configuration through protocols/procedures over the interface itself, such as DHCP, DHCPv6, SLAAC. |
| interfaceType | M | 0..1 | ”kernel” | ”userspaceDpdk” | "memif" | This attribute is applicable for passthrough and memif interfaces. Value default value is ”kernel”. |
| interfaceOption | M | 0..N | e.g. "speed=1G|10G|25G|100G" "nic-type=virtio|i710|mlx-cx5|" | Applicable to Pod network interfaces that directly connect to a physical NIC. The value is a list of verified options for physical NIC caracteristics. |
| interfaceRedundancy | M | 0..1 | "infraProvided“ | "none“ | "matedPair" | Default value is "infra-provided”, which means that the infrastructure is expected to provide network redundancy for the pod interface. Value "none" means that the application has no requirement on network redundancy. Value ”matedPair” means that the Pod asks for a mated pair of non-redundant left/right network attachments (typically SRIOV) and handles redundancy on application level. The same set of networks shall be configured on both interfaces. |
| switchPlane | M | 0..1 | “left” | “right” | Used (only) in conjunction with networkRedundancy "none" when the application requires two independent virtual links that for redundancy reasons have to reside on different switch planes (left or right). |
| redundancyMethod | M | 0..1 | ”activePassiveBond” | ”activeActiveBond” | ”activePassiveL3” | ”activeActiveL3” | Used (only) in conjunction with networkRedundancy "matedPair". "activeActiveBond“: The bonded left/right links must be part of a multi-chassis LAG in active-active mode. | "activePassiveBond": Interfaces bonded in active-passive mode in the application with move of bond MAC address. No specific requirements on DC fabric. | "activePassiveL3": Move of application IP address. | "activeActiveL3": Anycast/ECMP. |
datatype.ExtCpd.ParamMappings Information Element
| Attribute | Qualifier | Cardinality | Content | Description |
|---|---|---|---|---|
| ipAddressParameter | M | 0..1 | String | When present, this attribute specifies the name of the deployment artifact input parameter through which the orchestrator can configure the IP address(es), ipv4 and/or IPv6, for this asdExtCpd. The param name and provided IP address value will be passed to the deployment tool when deploying the DeploymentArtifacts. Note 1 |
| nadName | M | 0..1 | [String]1..N, Note 2 | These attributes specifies, for an asdExtCpd respesenting a secondary network interface, the name(s) of the deployment artifact input parameters through which the orchestrator can configure the corresponding network annotation in the pod manifest with references to the NAD(s) to be used for creating the network interface. |
| nadNamespace | M | 0..1 | String |
Note 1: When the asdExt Cpd represent a networkRedundant/mated-pair of sriov interfaces, there are references to 2 or 3 related NADs needed to be passed, lwhile for other interface types only one NAD reference is needed to be passed.
Note 2: The format of the Content strings is specific for each different orchestration templating technology used (Helm, Teraform, etc.). Currently only a format for use with Helm charts is suggested: ”helmchartname:[subchartname.]0..N[parentparamname.] 0..Nparametername”. Whether the optional parts of the format are present depends on how the parameter is declared in the helm chart.
enhancedClusterCapabilities Information Element
| Attribute | Qualifier | Cardinality | Content | Description |
|---|---|---|---|---|
| Id | M | 1 | String | Asd local unique name for the enhanceClusterCapabilities instance |
| minKernelVersion | M | 1 | String | Describes the minimal required Kernel version, e.g. 4.15.0. Coded as displayed by linux command uname –r |
| requiredKernelModules | M | 0..1 | List of String | Required kernel modules are coded as listed by linux lsmod command, e.g. ip6_tables, cryptd, nf_nat etc. |
| conflictingKernelModules | M | 0..1 | List of String | Kernel modules, which must not be present in the target environment. The kernel modules are coded as listed by linux lsmod command, e.g. ip6_tables, cryptd, nf_nat etc. Example: Linux kernel SCTP module, which would conflict with use of proprietary user space SCTP stack provided by the application. |
requiredCustomResources >kind >apiVersion | M | 0..N | Structure (inlined) | List the required custom resources types in the target environment, identifying each by the "kind" and "apiVersion" field in the K8S resource manifests and in the application. The list shall include those custom resource types which are not delivered with the application. Example: |
| clusterLabels | M | 0..1 | List of String | This attribute allows to associate arbitrary labels to clusters. These can indicate special infrastructure capabilities (e.g., NW acceleration, GPGPU compute, etc.). The intent of these labels is to serve as a set of values that can help in application placement decisions. This can be specified with the attribute -m: Mandatory, means deployment is not attempted if such support is not available in the target system -p: As preference - it means orchestrator will try to select a system with specific requirements, but if not found it will attempt deployment in a system not having such HW. |
| secondaryInterfacePlugin | M | 0..1 | String | The plug-in name / revision of the operator to handle secondary interface (e.g. Multus-CNI, v3.8) |
References:
- Initial proposal and the background information can be found at: Application Service Descriptor (ASD) and packaging Proposals for CNF - Developer Wiki - Confluence (onap.org)