Edge Scoping MVP for Casablanca - ONAP Enhancements

Summary: Edge Scoping 

Distributed Edge Cloud Infrastructure Object Hierarchy (Stretch Goal)

Value:

• Fine grained resource management & analytics for Distributed Edge Clouds 
  

References: 

• Infrastructure Modelling: ONAP R3+ Cloud Infrastructure Modeling; Cloud Infrastructure Aggregate Representation Classes

MULTICLOUD-153 - Getting issue details... STATUS

Assumption for Policy, SO, OOF:

• This uses the current Generic VNF workflow in SO

Note 1: 

• Configured Capacity and Utilized (or Currently Used) Capacity are managed by the specific cloud.

Note 2:

• Cloud SW Capability example 
  • Cloud region "x" with SR-IOV, GPU, Min-guarantee support
  • Cloud region "y" with SR-IOV support
• Cloud HW Capability example 
  • Resource cluster "xa" in Cloud region "x" with SR-IOV and GPU support 
  • Resource cluster "xb" in Cloud region "x" with GPU support
  • Resource cluster "ya" in Cloud region "y" with SR-IOV support

Note 3:

• 5G Service/VNF placement example
  • Constraints used by Optimization Framework (OOF)

    • 5G CU-UP VNF location to be fixed to a specific physical DC based on 5G DU, bounded by a max distance from 5G DU

  • Optimization Policy used by OOF

    • Choose optimized cloud region (or instance) for the placement of 5G CU UP for subscriber group based on the above constraints

Note 4:

• For the 5G Service/VNF placement example in Note 3
  • 5G CU-UP VNF preferably maps to a specific Cloud region & Physical DC End Point 

Note 5:

• For the 5G Service/VNF placement example in Note 3
  • OOF will pass the Physical DC End Point to SO as a opaque data

Note 6:

• For the 5G Service/VNF placement example in Note 3
  
  • SO passes the Physical DC End Point to Multi-Cloud as a opaque data, besides the Cloud Region

Cloud-agnostic Placement/Networking & Homing Policies (Phase 1 - Casablanca MVP, Phase 2 - Stretch Goal)

End-to-end use case Applicability:

• All (especially the data plane VNFs with fine-grained VNF placement and high performance networking requirements)

Value:

• Improve "workload deployability" by avoiding exposure of "cloud specific" capabilities to several ONAP components and addressing "separation of concerns" 

• Support capacity check (besides capability check) for HPA resources 

• Applicable to all workloads - VM-based or Container-based

MULTICLOUD-272 - Getting issue details... STATUS

Phase 1 Summary:

• Multi-Cloud Policy Framework
  • Assist OOF in target cloud region selection for VNF placement (aka homing) by summarizing cloud-specific capability, capacity & cost metrics (e.g. Infra HA for VMs in a VNF could have different cost in different clouds)
  • Dynamically modify the cloud specific VNF deployment template based on cloud-specific realization of the specified intent  
    • E.g. Infra HA for VMs within a VNF could have different realizations across different clouds
• Intent Support
  • Single realization option per Cloud Region for the specified Intent
• Major Impact Projects:
  • Multi-Cloud (Highest), OOF
• Minor Impact Projects:
  • A&AI, SO
• End-to-end use case demonstration:
  • vCPE, 5G?

Phase 2 Summary (Build on Phase 1 Work):

• Multi-Cloud Policy Framework
  
  • Dynamically modify the cloud specific VNF deployment template based on cloud-specific realization of the specified intent – Impact to VNF configuration 
    • E.g. High performance Intra-DC data plane networking with several realization choices
• Intent Support
  • Multiple realization options per Cloud Region for the specified Intent
• Major Impact Projects:
  • Multi-Cloud
• Minor Impact Projects:
  
  • OOF, GNF Controller

References: 

 The sequence diagram below expands "Multi-Cloud/VNFM Deploy Apps" in Edge Scoping Sequence Diagram

Cloud Agnostic Intent (Policy) Execution Workflow:

Follow up :

• Policy DB – is there any restriction on json objects store? - Matti to follow up with Ankit
• Current R2 support – single VF module (VNFC) assumption per VNF for vCPE - follow up with Kang Xi to validate
  
  

//Support the current simple capacity check API besides the intent-based framework for backward compatibility. 
//If a cloud region does not support the policy-based interface, it is given a high net value assuming the current capacity api (yes/no) 
//returns an yes. This ensures smooth migration to the new policy-based framework. 
{
"service": "cloudPolicy",
"policyName": "oofMulti-cloudCasablanca.cloudPolicy_vCPE_VNF",
"description": "Cloud Policy for vCPE VNF",
"templateVersion": "0.0.1",
"version": "oofMulti-cloudCasablanca",
"priority": "3",
"riskType": "test",
"riskLevel": "2",
"guard": "False",

"content": 
{
	"cloudOwner" : 
	{
		"owner": "All",
		"intent": 
		{
			"name": "Infrastructure High Availability (HA) for VNF", 
				//realization thru OpenStack-based: anti-affinity, Azure: Fault Domain or
    			//Different anti-affinity models from ETSI -- host-level, rack-level, availability zone level
				//max-count in heat template - scale out factor
				//server-group in heat template - usable thru API and CLI in OpenStack, VMware VIO
		}
		"intent": 
		{
			"name": "Infrastructure Resource Isolation for VNF", 
				// realization possible without dedicating CPU and Memory, refer to section on "Cloud Resource Partitioning for Differentiated QoS" 
				// on how this can help in offering tiered services
			"qosProperty": 
			{
				{"Burstable QoS": "TRUE", "Burstable QoS Oversubscription Percentage": "10"}, {"operator", "OR"}, {"Guaranteed QoS": "TRUE"}
				// VMware VIO - tenant VDC CLI and API - configure the appropriate settings per tenant
				// Burstable QoS is specified through min guarantee (part of flavor metadata -- 
              	// https://docs.openstack.org/horizon/latest/admin/manage-flavors.html
			}
		}
		"cloudCapacityUtilizaitonAttributes" : 
		{
			//current_allocated_capacity is normalized to 1
			//max value for cpu or memory is 1 if usage is greater than equal to limit
			"current_allocated_capacity" :
			{ 
				{"cpu", "memory", "disk"}: "tenant (OpenStack Project or Resource Slice)", 
			},

			// under discussion - elaborate capacity, utilization checks for various objects
			//"current_allocated_capacity" : 
			//{ 
			//	{"cpu", "memory", "network"}: 
			//	{"cloud": {"weight": "0.85", "threshold": "0.9"}, 
			//	{"tenant (resource slice)": {"weight": "0.85", "threshold": "0.9" }, 
			//	{"host aggregate (resource cluster)": {"weight": "0.85", "threshold": "0.9"},
			//},
			//"average_utilization" : { {"cpu", "memory", "network"}: {"cloud": {"weight": "0.13"}, "tenant (resource slice)": {"weight": "0.13"}, 
			//	"host aggregate (resource cluster)": {"weight": "0.13"} }, "time-window": "24", "unit": "hours" },
			//"peak_utilization" : { {"cpu", "memory", "network"}: {"cloud": {"weight": "0.02"}, "tenant (resource slice)": {"weight": "0.02"}, "host 
			//	aggregate (resource cluster)": {"weight": "0.02"} }, "time-window": "24", "unit": "hours" }
			
			//current_allocated_capacity, average_utilization and peak_utilization are normalized to 1
			
			//For a given object such as tenant_cpu, sum of weights across all attributes (current_allocated_capacity, average_utilization & 
			//peak_utilization) must be 1
			//E.g. net_value = cloud_cpu_current_allocated_capacity*0.85 + cloud_cpu_average_utilization*0.13 + cloud_cpu_peak_utilization*.02 + ...
			
			//For a given object such as cloud_cpu, if the current_allocated_capacity "threshold" exceeds the specified value, return "high net 
			//value" 
		}
	}

	//use cloud provider in – <cloud region id, cloud provider> – different cloud providers may need different capacities for the same VNF
	"cloudOwner" : 
	{
		"owner": "Azure",
		"capacityProperty": 
		{ 		 
			//under discussion – "capabilityProperty": {SR_IOV, ...} 
			//under discussion - host network bandwidth
		
			"controller": "multicloud",
			"request": //from R2
			"{\"vCPU\": {\"quantity\": {\"get_param\": \"REQUIRED_VCPU\"}, \"Memory\": {\"quantity\": {\"get_param\": \"REQUIRED_MEM\"}, 	
			\"unit\": 	\"GB\"}, \"Storage\": {\"quantity\": {\"get_param\": \"REQUIRED_DISK\"}, \"unit\": \"GB\"}}"
		}
		
		"owner": "OpenStack",
		"capacityProperty": 
		{ 		 
			//under discussion – "capabilityProperty": {SR_IOV, ...} 
			//under discussion - host network bandwidth
		
			"controller": "multicloud",
			"request": //from R2
			"{\"vCPU\": {\"quantity\": {\"get_param\": \"REQUIRED_VCPU\"}, \"Memory\": {\"quantity\": {\"get_param\": \"REQUIRED_MEM\"}, 	
			\"unit\": 	\"GB\"}, \"Storage\": {\"quantity\": {\"get_param\": \"REQUIRED_DISK\"}, \"unit\": \"GB\"}}"
		}
	}
}

"resources": ["vGMux"], //R2 support status – single VF module assumption per VNF 
"applicableResources": "any",
"identity": "distance-vGMux",
"policyScope": ["vCPE", "US", "INTERNATIONAL", "ip", "vGMux"],
"policyType": "AllPolicy"
}

Private Cloud Setup (e.g. OpenStack)

• Pre-defined (including custom) flavors map to Instance types in Public Clouds
  
  • Pre-defined flavors are created by the Cloud Admin before the Cloud is used by ONAP for workload deployment

 VNFC to Instance Type Mapping

• One or more VNFCs (e.g. vCPE VGW) could map to an instance type
  • Use Case: Residential Broadband vCPE (Approved)
  • https://github.com/onap/demo/blob/a19bd423c1e4d2f5552bab090ba2650c38fa7192/heat/vCPE/vgw/base_vcpe_vgw.yaml

Operator Configuration – Multi-VIM/Cloud Plugin

The operator/service provider who uses ONAP will choose which VIMs to use and include the appropriate MultiVIM plugins in his ONAP deployment. For example, let’s assume they pick private Openstack, private VMWare, and public Azure as the platform to run their services on.

For each MultiVIM plugin, operator configures the following information:

• For each VM flavor, the cost of that VM to the operator. Note that this costs includes the (potentially discounted) list price for the VM, support cost, and operations cost. The last one is definitely operator specific.
• Operator also specify the cost for each feature: HA, etc
• Note that the operator is free to choose what time duration the cost metric is specified for each of the MultiVIM plugins (e.g., cost per hour, cost per month) since they will do it consistently for each of the VIMs. 

OOF → Multi-VIM/Cloud Policy API - Key Processing Steps

For each cloud owner

•  Instance Type Handling
  • Instance Type is passed in the capacity check API from OOF (Discuss) //Note, SO → MC passes OpenStack flavor name in the Heat Template/Env file
  • Convert to appropriate instance type based on intent //e.g. "Infrastructure Resource Isolation for VNF" may result in a different instance type if the cloud owner supports "Burstable QoS"
• Parse OOF → MC Policy API 
• For each cloud region // Public cloud could have different costs in different geographic locations
  • net_value_cost = net_value_cost + cost_instance_type // cost per instance type is based on policy (for R3, it is picked up from Multi Cloud configuration file)
  • net_value_cost = net_value_cost + cost_intent //e.g. "Infrastructure High Availability (HA) for VNF" may have additional cost
  • Capacity Check – Continue R2 Plan
    • Private Clouds (OpenStack based)
      • Capacity check per Tenant (OpenStack Project)
        • returns yes or no 
    • Public Clouds or Other Clouds
      • Capacity check per tenant 
        • return yes always //assumption: public cloud has infinite capacity
  • If infra capacity is finite (Private cloud or Public cloud with reserved instances) // e.g. object of interest is OpenStack Project with a quota (upper limit) for cpu, mem and local disk
    • Option A: (potentially supportable with OpenStack based clouds, other clouds to be investigated)
      
      • normalized_cpu_per_object = (current_cpu_used_object + instance_type_cpu*number_of_instances_of vnfc)/total_cpu_object
        • if normalized_cpu_per_object > 1, capacity check failed
      • normalized_mem_per_object = (current_mem_used_object + instance_type_mem*number_of_instances_of_vnfc)/total_mem_object
        • if normalized_mem_per_object > 1, capacity check failed 
      • normalized_disk_per_object = (current_disk_used_object + instance_type_disk*number_of_instances_of_vnfc)/total_disk_object
        • if normalized_disk_per_object > 1, capacity check failed 
      • net_value_capacity = normalized_infra_capacity = wcpu*normalized_cpu_per_object + wmem*normalized_mem_object + wdisk*normalized_disk_per_object // wcpu, wmem, wdisk are specified in a multi-cloud configuration file; wcpu + wmem + wdisk = 1
    • Option B: (under study)
      • normalized_instances_per_object = (current_used_reserved_instances + number_of_instances_of_vnfc)/total_reserved_instances
      • net_value_capacity = normalized_instances_per_object
• MC → OOF return values per <Cloud Owner, Cloud Region> = {net_value_cost, capacity_check_boolean}
  
  

OOF → Multi-VIM/Cloud Policy API - Other

• Convert to appropriate instance type based on intent //e.g. "Infrastructure Resource Isolation for VNF" may result in a different instance type if the cloud owner supports "Burstable QoS"
  • Example
    • VNFC with "Guaranteed QoS"
      • vCPU (Min/Max) - 16, Mem (Min/Max) - 32GB  
    • Same VNFC with "Burstable QoS", 25% oversubscription
      • vCPU (Min) - 16, Mem (Min) - 32GB
      • vCPU (Max) - 20, Mem (Max) - 40GB
        
        

SO → Multi-VIM/Cloud 

• Get Cloud Region of interest
• Parse Template (e.g. OpenStack Heat Template)
  • For each VNFC, instance type in the template
    
    • Get from A&AI as follows 
      • Key: <VNFC, Instance Type, Cloud Region> in A&AI
      • Value: <Policy JSON - only Intent portion> 
    • Parse Policy JSON
    • Modify template according to Intent - intent examples below
      • "Infrastructure High Availability (HA) for VNF" 
      • "Infrastructure Resource Isolation for VNF"   

OOF Processing - Key Processing Steps

Policy Management

• Store VNFC Policy in A&AI as follows
  • Key: <VNFC (e.g. vGMux), Instance Type, Cloud Region>
  • Value: <Policy JSON - only Intent portion>

Optimization

Each service specifies an service-specific objective function that is stored as part of the service-specific policy and is used by OOF to evaluate the candidates. For simplicity of the example, let’s consider service that consists only of one VNF instance. The objective function has two components:

- distance from customer location to the VNF - the service designed assigns a weight for the distance: wd

- the cost of deploying the VNF in a location - the service designer assigns a weight for the cost: wc

OOF optimizes function: min (wd*distance + wc*cost)

If the service does not care about the cost at all, it would set wc = 0. If the service designer wants to minimize cost, he could set wd=0. Note that candidates that are too far can be eliminated by a distance constraint even before the optimization. For example, if the service has a distance constraint of at most 100 kilometers, then only those cloud regions within 100 kilometers to the customer location would be considered in the objective function evaluation.

If the service designer wants to trade off between distance and cost, for example, they might set wd = 1, wc = 2. This would mean that one $1 increase in price is as valuable as 2 kilometers in distance.

Candidate 1: $100, 100 kilometers => value: 300

Candidate 2: $150, 80 kilometers => value: 380

Candidate 3: $50, 190 kilometers => value: 290  <- pick this one 

Cloud Resource Partitioning for Differentiated QoS (Combined with Previous)

Value:

• Applicable to all use cases
• Casablanca Targets:
  • vCPE (Enable Tiered service offering); 5G Network Slicing (Stretch Goal) 

References:

• https://wiki.onap.org/download/attachments/28377610/ONAP-mc-of-resource-reservation.pptx?version=1&modificationDate=1526771712000&api=v2

Edge Automation Requirement:

Support three types of slices in the Cloud Infrastructure (Definition Reference: https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/)

• Guaranteed Resource Slice (hard isolation) for various infra Resources (CPU/Memory/Network)
  • Max (limit), Min (request) are the same; resource guarantee is "Max"
  • Maps to 5G Applications such as Connected Car which fall in the category of ultra-reliable machine-type communications (ref. 1)
• Burstable Resource Slice (soft isolation) for various infra Resources
  • Min (request) <= Max (limit); resource guarantee is "Min"
  • Maps to Burstable Network Slice such > 1Gbps broadband which fall in the category of extreme mobile broadband (ref. 1)
• Best Effort Resource Slice (no isolation) for various infra Resources
  • No Min (request) ; resource guarantee is "None"
  • Maps to 5G Applications such as IoT which fall in the category of massive machine-type communications (ref. 1)

Implementation:

• Leverage current HPA framework with appropriate extensions

References:

• https://metis-ii.5g-ppp.eu/wp-content/uploads/white_papers/5G-RAN-Architecture-and-Functional-Design.pdf

• Driving Superior Isolation for Tiered Services using Resource Reservation -- Optimization Policies for Residential vCPE
  -https://jira.onap.org/browse/OPTFRA-240

Note:

• Any VMs/Containers which are part of a resource slice will adhere to the specs of the resource slice

Aggregated Infrastructure Telemetry Streams (Aligns with HPA requirements, Combining efforts with HPA)

Value

• Edge Infrastructure Analytics complementing 5G VNF Analytics

MULTICLOUD-254 - Getting issue details... STATUS

ONAP, as in R2, collects the statistics/alarms/events from workloads (VMs) and take any close loop control actions such as Heal a process, scale-out, restart etc.. In R3, infrastructure related statistics/alarms/events will be collected, generate actionable insights and take life cycle actions on the workloads.  Infrastructure statistics normally include performance counters, NIC counters, IPMI information on per physical server node basis.  To reduce the load on the ONAP, it is necessary that aggregated (summarized) information is sent to the ONAP from edge-clouds. 

As part of this activity, intention is to create aggregation micro-service that collects the data from physical nodes (over collected and other mechanisms), aggregate the information (time based aggregation, threshold based aggregation, silencing etc.,..) based on the configurable rules and export the aggregate data to DCAE.  This micro service can be instantiated by ONAP itself - one or more instances for edge-clouds at the ONAP-central itself using OOM, it could be instantiated at the edge-cloud using their own deployment tools or it could be deployed edge service providers at the regional site level.  

Impacted projects (development activities)

Life Cycle stages related functions

ONAP Edge Analytics with DCAE/DMaaP independent of closed loop (Beyond Casablanca)

Value

• 5G Analytics
  
  

Multi-Cloud Deployment in Edge Cloud (Stretch Goal)

MULTICLOUD-262 - Getting issue details... STATUS

Value:

• Multi-Cloud service to assist in central A&AI scaling by caching A&AI data locally and syncing up with A&AI periodically