Summary: Edge Scoping
ONAP Component | Life Cycle Phase | Enhancements |
---|---|---|
Multi-Cloud | Deploy | Support Distributed Cloud Infrastructure Capability Discovery (Note 1, Note 2) |
A&AI | Deploy | Support Standardized Distributed Cloud Infrastructure Object Hierarchy & Capability Database (Ref. 1)
|
OOF | Deploy | Execute Distributed Cloud Infrastructure Placement Policies for Optimized Service/VNF Placement across Cloud Regions (Note 3, Note 4) |
SO | Deploy | Extend SO ↔ OOF API to support data opaque to SO (Note 5) Extend SO ↔ MC API to support data opaque to SO (Note 6) |
Assumption for Policy, SO, OOF:
Note 1:
Note 2:
Note 3:
5G CU-UP VNF location to be fixed to a specific physical DC based on 5G DU, bounded by a max distance from 5G DU
Choose optimized cloud region (or instance) for the placement of 5G CU UP for subscriber group based on the above constraints
Note 4:
Note 5:
Note 6:
All (especially the data plane VNFs with fine-grained VNF placement and high performance networking requirements)
Improve "workload deployability" by avoiding exposure of "cloud specific" capabilities to several ONAP components and addressing "separation of concerns"
Support capacity check (besides capability check) for HPA resources
Applicable to all workloads - VM-based or Container-based
Cloud Agnostic Intent (Policy) Execution Workflow - Steps 1- 6
Cloud Agnostic Intent (Policy) Execution Workflow - Step 7
Intent Support
Single realization option per Cloud Region for the specified Intent
The sequence diagram below expands "Multi-Cloud/VNFM Deploy Apps" in Edge Scoping Sequence Diagram
2a) OOF Processing - the fetched Policy (example below) is stored in a local data structure and is available for further use (need OOF code changes).
// //Example 1: vCPE, Burstable QoS //vCPE: Infrastructure Resource Isolation for VNF with Burstable QoS // { "service": "cloudSelectionPolicy", "policyName": "oofMulti-cloudCasablanca.cloudSelectionPolicy_vCPE_VNF", "description": "Cloud Selection Policy for vCPE VNFs", "templateVersion": "0.0.1", "version": "oofMulti-cloudCasablanca", "priority": "3", "riskType": "test", "riskLevel": "2", "guard": "False", "content": { //Note on cloudOwner, cloudRegion usage: e.g. //"cloudOwner": "xyz", //can be a specific cloud owner such as Azure, VMware VIO, Wind River Titanium Cloud etc. //"cloudRegion": "1", //can be a specific cloud region for a cloud owner //If no cloudOwner is specified, policy applies to all cloudOwners //If no cloudRegion is specified, policy applies to all cloudRegions {//new in R3 "cost-intent": "TRUE", }, {//new in R3 "cloudOwner": "VMware VIO", //can be a specific cloud owner such as Azure, VMware VIO, Wind River Titanium Cloud etc. "deployment-intent": { "name": "Infrastructure Resource Isolation for VNF", // realization possible without dedicating CPU and Memory, refer to section on "Cloud Resource Partitioning for Differentiated QoS" // on how this can help in offering tiered services "qosProperty": { {"Burstable QoS": "TRUE", "Burstable QoS Oversubscription Percentage": "25"} } } }, } "resources": ["vgw", "vgmux"], //"vgw" is also interchangeably used as "vg" "applicableResources": "any", "identity": "cloud-atrributes", "policyScope": ["vCPE", "US", "INTERNATIONAL", "ip", "vgw", "vgmux"], "policyType": "AllPolicy" } // //Example 2: //vCPE: Infrastructure Resource Isolation for VNF with Guaranteed QoS // { "service": "cloudSelectionPolicy", "policyName": "oofMulti-cloudCasablanca.cloudSelectionPolicy_vCPE_VNF", "description": "Cloud Selection Policy for vCPE VNFs", "templateVersion": "0.0.1", "version": "oofMulti-cloudCasablanca", "priority": "3", "riskType": "test", "riskLevel": "2", "guard": "False", "content": { //Note on cloudOwner, cloudRegion usage: e.g. //"cloudOwner": "xyz", //can be a specific cloud owner such as Azure, VMware VIO, Wind River Titanium Cloud etc. //"cloudRegion": "1", //can be a specific cloud region for a cloud owner //If no cloudOwner is specified, policy applies to all cloudOwners //If no cloudRegion is specified, policy applies to all cloudRegions {//new in R3 "cost-intent": "TRUE", }, {//new in R3 "deployment-intent": { "name": "Infrastructure Resource Isolation for VNF", // realization possible without dedicating CPU and Memory, refer to section on "Cloud Resource Partitioning for Differentiated QoS" // on how this can help in offering tiered services "qosProperty": { {"Guaranteed QoS": "TRUE"} } } } } "resources": ["vgw", "vgmux"], //"vgw" is also interchangeably used as "vg" "applicableResources": "any", "identity": "cloud-atrributes", "policyScope": ["vCPE", "US", "INTERNATIONAL", "ip", "vgw", "vgmux"], "policyType": "AllPolicy" } // //Example 3: //vDNS: Infrastructure HA for VMs in a VNF & Infrastructure Resource Isolation for VNF with Burstable QoS // { "service": "cloudSelectionPolicy", "policyName": "oofMulti-cloudCasablanca.cloudSelectionPolicy_vDNS_VNF", "description": "Cloud Selection Policy for vDNS VNFs", "templateVersion": "0.0.1", "version": "oofMulti-cloudCasablanca", "priority": "3", "riskType": "test", "riskLevel": "2", "guard": "False", "content": { //Note on cloudOwner, cloudRegion usage: e.g. //"cloudOwner": "xyz", //can be a specific cloud owner such as Azure, VMware VIO, Wind River Titanium Cloud etc. //"cloudRegion": "1", //can be a specific cloud region for a cloud owner //If no cloudOwner is specified, policy applies to all cloudOwners //If no cloudRegion is specified, policy applies to all cloudRegions {//new in R3 "cost-intent": "TRUE", }, {//new in R3 "cloudOwner": "VMware VIO", //can be a specific cloud owner such as Azure, VMware VIO, Wind River Titanium Cloud etc. "deployment-intent": { "name": "Infrastructure Resource Isolation for VNF", // realization possible without dedicating CPU and Memory, refer to section on "Cloud Resource Partitioning for Differentiated QoS" // on how this can help in offering tiered services "qosProperty": { {"Burstable QoS": "TRUE", "Burstable QoS Oversubscription Percentage": "25"} } } }, {//new in R3 "deployment-intent": { "name": "Infrastructure High Availability (HA) for VNF", } }, } "resources": ["vDNS"], "applicableResources": "any", "identity": "cloud-atrributes", "policyScope": ["vDNS", "US", "INTERNATIONAL", "vDNS"], "policyType": "AllPolicy" } // //Example 4: //vDNS: Infrastructure HA for VMs in a VNF & Infrastructure Resource Isolation for VNF with Guaranteed QoS // { "service": "cloudSelectionPolicy", "policyName": "oofMulti-cloudCasablanca.cloudSelectionPolicy_vDNS_VNF", "description": "Cloud Selection Policy for vDNS VNFs", "templateVersion": "0.0.1", "version": "oofMulti-cloudCasablanca", "priority": "3", "riskType": "test", "riskLevel": "2", "guard": "False", "content": { //Note on cloudOwner, cloudRegion usage: e.g. //"cloudOwner": "xyz", //can be a specific cloud owner such as Azure, VMware VIO, Wind River Titanium Cloud etc. //"cloudRegion": "1", //can be a specific cloud region for a cloud owner //If no cloudOwner is specified, policy applies to all cloudOwners //If no cloudRegion is specified, policy applies to all cloudRegions {//new in R3 "cost-intent": "TRUE", }, {//new in R3 "deployment-intent": { "name": "Infrastructure High Availability (HA) for VNF", } }, {//new in R3 "deployment-intent": { "name": "Infrastructure Resource Isolation for VNF", // realization possible without dedicating CPU and Memory, refer to section on "Cloud Resource Partitioning for Differentiated QoS" // on how this can help in offering tiered services "qosProperty": { {"Guaranteed QoS": "TRUE"} } } } } "resources": ["vDNS"], "applicableResources": "any", "identity": "cloud-atrributes", "policyScope": ["vDNS", "US", "INTERNATIONAL", "ip", "vDNS"], "policyType": "AllPolicy" } |
3a) OOF Processing - Perform Cloud Agnostic Capability check for each <cloud owner, cloud region>. OOF will prune any <cloud owner, cloud region> which is not satisfying the standardized capabilities.
4a) OOF Processing
The enhanced OOF ↔ MC capacity check API, described below, is filled based on the enhanced Capacity Check & Cloud Selection Policy for Homing retrieved in step 2) – need OOF code changes.
//flexibility of having cloud type in the new API provides fine grained control, addresses capacity/cost differences across different cloud owners/regions and ensures backward compatibility JSON Schema with Examples: https://gerrit.onap.org/r/#/c/58531/1/usage-examples/oof-mc-r3-interface-examples.py // OOF -> MC API REQUEST JSON SCHEMA { "type" : "object", "properties" : { # vnfc is not used in the OOF->MC path for R3, this is kept to be consistent # with the SO-> MC path "vnfc": {"type": "string"}, # evaluate cloud cost if set # cost is fixed per cloud type for all workloads -- simplifying assumption for R3 # cost specified in the respective plugin through a configuration file "cost-intent" : {"type" : "boolean"}, "deployment-intent": {"type": "object"}, "properties" : { # Azure, K8S, OpenStack, VMware VIO, Wind River Titanium "Cloud Type (Cloud Provider)": {"type", "string"}, "Infrastructure High Availability for VNF": {"type", "boolean"}, "Infrastructure Resource Isolation for VNF": {"type", "string"}, # Infrastructure Resource Isolation for VNF # Only certain pre-defined over-subscription values are allowed to # reflect practical deployment and simplify implementation for R3 "Infrastructure Resource Isolation for VNF - Burstable QoS Oversubscription Percentage": {"type": "int"}, }, # vCPU, Memory, Storage, VIMs - part of R2 capacity check "vCPU": {"type": "number"}, # number of cores "Memory": {"type": "number"}, # size of memory, GB "Storage": {"type": "number"}, # size of storage, GB "VIMs": {"type": "array"}, # VIMs OOF wish to check with }, "required": ["cost-intent", "deployment-intent", "vCPU", "Memory", "Storage", "VIMs"] } // OOF -> MC API RESPONSE JSON SCHEMA { "cloudRegionNetValue": { "type": "array", "items": { "$ref": "#/definitions/xxx" } }, "definitions": { "xxx": { "type": "object", "required": [ "VIM", "netValue" ], "properties": { # VIM id "VIM": { "type": "string", }, # For R3, netValue signifies cost per VIM id # Referring to cost-intent in the API from OOF -> MC # cost is fixed per cloud type for all workloads # cost specified in the respective plugin through a configuration file "netValue": { "type": "number", } } } } } |
// //Policy Relevant to Azure plugins // //fixed workload deployment cost for all workloads (simplifying assumption for R3) { "workloadDeploymentCost": "100", } // //Policy Relevant to Wind River OpenStack plugin // //fixed workload deployment cost for all workloads (simplifying assumption for R3) { "workloadDeploymentCost": "100", (set to higher or lower value for testing) } ... |
5a) MC Processing (need MC code changes)
For each cloud owner
5b) Workload Deployment Cost Policy - Configured by the Operator
The operator/service provider who uses ONAP will choose which VIMs to use and include the appropriate MC plugins in his ONAP deployment. For example, let’s assume they pick private Openstack, private VMWare, and public Azure as the platform to run their services on.
For R3, Workload Deployment Cost Policy can be stored in the form of configuration file(s) in the OOM K8S Persistent Volumes visible to the relevant MC plugin to simplify implementation. Beyond R3, this could be moved to the Policy DB. The details of the configuration are described below.
"Workload Deployment Cost Policy Example" depicted above has an exemplary description of this.
6a) OOF Processing - cloud_net_value input in Multi-objective Optimization (need OOF code changes)
Each service specifies an service-specific objective function that is stored as part of the service-specific policy and is used by OOF to evaluate the candidate <cloud owner, cloud region>. For simplicity of the example, let’s consider service that consists only of one VNF instance. The objective function has two components:
- distance from customer location to the VNF - the service designed assigns a weight for the distance: wd
- the cost of deploying the VNF in a location - the service designer assigns a weight for the cost: wc
OOF optimization function: min (wd*distance + wc*cloud_net_value)
If the service does not care about the cost at all, it would set wc = 0. If the service designer wants to minimize cost, he could set wd=0. Note that candidates that are too far can be eliminated by a distance constraint even before the optimization. For example, if the service has a distance constraint of at most 100 kilometers, then only those <cloud owner, cloud region> within 100 kilometers to the customer location would be considered in the objective function evaluation.
If the service designer wants to trade off between distance and cost, for example, they might set wd = 1, wc = 2. This would mean that one $1 increase in price is as valuable as 2 kilometers in distance.
<cloud owner, cloud region> Candidate 1: $100, 100 kilometers => value: 300
<cloud owner, cloud region> Candidate 2: $150, 80 kilometers => value: 380
<cloud owner, cloud region> Candidate 3: $50, 190 kilometers => value: 290 <- pick this one
7a) MC Processing (need MC code changes)
Policy (Intent) Realization
"Infrastructure Resource Isolation for VNF" – { "qosProperty": { {"Burstable QoS": "TRUE", "Burstable QoS Oversubscription Percentage": "25"} } }
OpenStack-based VMware VIO Cloud realization
7b) Cloud-Agnostic Workload Deployment Policy (Intent)
//Policy Example 1 - VNFC VGW which is part of vCPE service { "Service": "vCPE" "VNFC": "vgw", //"vgw" is also intechangeably used as "vg" { "cloudOwner": "VMware VIO", //can be a specific cloud owner such as Azure, VMware VIO, Wind River Titanium Cloud etc. "deployment-intent": { "name": "Infrastructure Resource Isolation for VNF", // realization possible without dedicating CPU and Memory, refer to section on "Cloud Resource Partitioning for Differentiated QoS" // on how this can help in offering tiered services "qosProperty": { {"Burstable QoS": "TRUE", "Burstable QoS Oversubscription Percentage": "25"} } } } } //Policy Example 2 - VNFC vDNS which is part of vLoadBalancer/vDNS service { "Service": "vDNS", "VNFC": "vDNS", //By default, Policy (Intent) is applicable to all cloud owners/regions unless specified. { "deployment-intent": { "name": "Infrastructure High Availability (HA) for VNF", } }, { "deployment-intent": { "name": "Infrastructure Resource Isolation for VNF", // realization possible without dedicating CPU and Memory, refer to section on "Cloud Resource Partitioning for Differentiated QoS" // on how this can help in offering tiered services "qosProperty": { {"Guaranteed QoS": "TRUE"} } } } } |
Follow ups:
Implementation trade offs for Casablanca (R3) and potential Dublin (R4) plan:
5. VM Instance type/VM Feature Group dollar-cost-based cloud selection
Casablanca Plan
VM Instance type/VM Feature Group dollar-cost-based cloud selection is not mandatory for all Multi-Cloud Plugins
If a Multi-Cloud Plugin does not support cost based on "dollarCostEvaluationVM-Type" and/or "dollarCostEvaluationVM-FeatureGroup"
Dublin & Beyond Potential Plan
6. The workload instance type model (Note 1 below) is not passed from OOF → MC. Rationale below.
Currently, the VNFC model supports only compute/memory/local storage. A comprehensive VNFC model needs to include HW generation (Intel Broadwell/Haswell, ARM etc.), HPA details (SR-IOV etc.) and more.
Casablanca Plan
Since the VNFC model standardization is in progress, VNFC mapping to VM Instance Type (OpenStack Flavor, Azure M4 etc.) happens in the Multi-Cloud plugin through configuration.
Cloud Resource Partitioning for Differentiated QoS (Combined with Previous)
Support three types of slices in the Cloud Infrastructure (Definition Reference: https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/)
Implementation:
References:
Driving Superior Isolation for Tiered Services using Resource Reservation -- Optimization Policies for Residential vCPE
-https://jira.onap.org/browse/OPTFRA-240
Note:
ONAP Component | Life Cycle Phase | Enhancements |
---|---|---|
Policy | Design | Configuration Policies for Guaranteed, Burstable & Best Effort Cloud Infrastructure Resource Slices (this will apply to VMs/Containers also) Placement Policies for Resource Slices
|
Multi-Cloud | Deploy | Resource Slice Capability Discovery |
A&AI | Deploy | Resource Slice Capability per Cloud Region
Resource Slice Type
|
OOF | Deploy | Execute Resource Slice Placement Policies for Optimized Service/VNF Placement across Cloud Regions |
Edge Infrastructure Analytics complementing 5G VNF Analytics
ONAP, as in R2, collects the statistics/alarms/events from workloads (VMs) and take any close loop control actions such as Heal a process, scale-out, restart etc.. In R3, infrastructure related statistics/alarms/events will be collected, generate actionable insights and take life cycle actions on the workloads. Infrastructure statistics normally include performance counters, NIC counters, IPMI information on per physical server node basis. To reduce the load on the ONAP, it is necessary that aggregated (summarized) information is sent to the ONAP from edge-clouds.
As part of this activity, intention is to create aggregation micro-service that collects the data from physical nodes (over collected and other mechanisms), aggregate the information (time based aggregation, threshold based aggregation, silencing etc.,..) based on the configurable rules and export the aggregate data to DCAE. This micro service can be instantiated by ONAP itself - one or more instances for edge-clouds at the ONAP-central itself using OOM, it could be instantiated at the edge-cloud using their own deployment tools or it could be deployed edge service providers at the regional site level.
Impacted projects (development activities)
ONAP Component | Enhancements |
---|---|
Overall |
|
Multi-Cloud |
|
AAI & ESR |
|
PORTAL | ESR portal related changes to take information about the edge-cloud (CA Cert and UN/PWD information) - Future when the edges started to send aggregate data) |
OOF | HPA Enhancements
|
Life Cycle stages related functions
ONAP Component | Life cycle phase | Activities |
---|---|---|
AAI and ESR | Deploy & Run time |
|
AAI and ESR | Run time |
|
Multi-Cloud | Run time |
|
OOF | Run time |
|
High level architecture slides:
ONAP Component | Life cycle phase | Enhancements |
---|---|---|
OOM - ONAP Central | Deploy |
|