Versions Compared

Old Version 113

changes.mady.by.user ramki krishnan

Saved on

compared with

New Version Current

changes.mady.by.user ramki krishnan

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Summary: Edge ScopingArchitecture & Work Items#ONAPEdgeMVP 

...

Table of Contents

...


Distributed Edge Cloud Infrastructure Object Hierarchy (Stretch Goal - Beyond Casablanca)

Value:

  • Fine grained resource management & analytics for Distributed Edge Clouds 

...

  • Improve "workload deployability" by avoiding exposure of "cloud specific" capabilities to several ONAP components and addressing "separation of concerns" Support capacity check (besides capability check) for HPA resources 

  • Applicable to all workloads - VM-based or Container-based

Jira
serverONAP JIRA
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyMULTICLOUD-272

Phase 1 (Casablanca MVP) Summary:

  • Multi-Cloud Policy Framework
    • Assist OOF in target cloud region selection for VNF placement (aka homing) by summarizing cloud-specific capability, capacity & cost metrics (e.g. Infra HA for VMs in a VNF could have different cost in different clouds)through intent

      • Cloud Agnostic Intent (Policy) Execution Workflow  - Steps 1- 4

    • Dynamically modify the cloud specific VNF deployment template based on cloud-specific realization of the specified intent  Eintent (e.g. Infra HA for VMs within a VNF could have different realizations across different clouds)

      • Cloud Agnostic Intent (Policy) Execution Workflow  - Step 5

  • Intent Support

    • Single realization option per Cloud Region for the specified Intent

  • Major Impact Projects:
    • Multi-Cloud (Highest), OOF
    Minor Impact Projects:
    • A&AI, SO (Minimal)
  • End-to-end use case demonstration:
    • vCPE , 5G?

Phase 2 Summary (Build on Phase 1 Work):

  • Multi-Cloud Policy Framework
    • Dynamically modify the cloud specific VNF deployment template based on cloud-specific realization of the specified intent – Impact to VNF configuration 
      • E.g. High performance Intra-DC data plane networking with several realization choices
  • Intent Support
    • Multiple realization options per Cloud Region for the specified Intent
  • Major Impact Projects:
    • Multi-Cloud
  • Minor Impact Projects:
    • OOF, GNF Controller

References: 

(warning) The sequence diagram below expands "Multi-Cloud/VNFM Deploy Apps" in Edge Scoping Sequence Diagram

Cloud Agnostic Intent (Policy) Execution Workflow:

Gliffy Diagram
size1200
nameCloud Agnostic Intent Execution Workflow
pagePin30

Follow up :

...

Code Block
languagepy
themeEmacs
titleCloud Policy Example
linenumberstrue
//Support the current simple capacity check API besides the intent-based framework for backward compatibility. 
//If a cloud region does not support the policy-based interface, it is given a high net value assuming the current capacity api (yes/no) 
//returns an yes. This ensures smooth migration to the new policy-based framework. 
{
"service": "cloudPolicy",
"policyName": "oofMulti-cloudCasablanca.cloudPolicy_vCPE_VNF",
"description": "Cloud Policy for vCPE VNF",
"templateVersion": "0.0.1",
"version": "oofMulti-cloudCasablanca",
"priority": "3",
"riskType": "test",
"riskLevel": "2",
"guard": "False",

"content": 
{
	"cloudOwner" : 
	{
		"owner": "All",
		"intent": 
		{
			"name": "Infrastructure High Availability (HA) for VNF", 
				//realization thru OpenStack-based: anti-affinity, Azure: Fault Domain or
    			//Different anti-affinity models from ETSI -- host-level, rack-level, availability zone level
				//max-count in heat template - scale out factor
				//server-group in heat template - usable thru API and CLI in OpenStack, VMware VIO
		}
		"intent": 
		{
			"name": "Infrastructure Resource Isolation for VNF", 
				// realization possible without dedicating CPU and Memory, refer to section on "Cloud Resource Partitioning for Differentiated QoS" 
				// on how this can help in offering tiered services
			"qosProperty": 
			{
				{"Burstable QoS": "TRUE", "Burstable QoS Oversubscription Percentage": "10"}, {"operator", "OR"}, {"Guaranteed QoS": "TRUE"}
				// VMware VIO - tenant VDC CLI and API - configure the appropriate settings per tenant
				// Burstable QoS is specified through min guarantee (part of flavor metadata -- 
              	// https://docs.openstack.org/horizon/latest/admin/manage-flavors.html
			}
		}
		"cloudCapacityUtilizaitonAttributes" : 
		{
			//current_allocated_capacity is normalized to 1
			//max value for cpu or memory is 1 if usage is greater than equal to limit
			"current_allocated_capacity" :
			{ 
				{"cpu", "memory", "disk"}: "tenant (OpenStack Project or Resource Slice)", 
			},

			// under discussion - elaborate capacity, utilization checks for various objects
			//"current_allocated_capacity" : 
			//{ 
			//	{"cpu", "memory", "network"}: 
			//	{"cloud": {"weight": "0.85", "threshold": "0.9"}, 
			//	{"tenant (resource slice)": {"weight": "0.85", "threshold": "0.9" }, 
			//	{"host aggregate (resource cluster)": {"weight": "0.85", "threshold": "0.9"},
			//},
			//"average_utilization" : { {"cpu", "memory", "network"}: {"cloud": {"weight": "0.13"}, "tenant (resource slice)": {"weight": "0.13"}, 
			//	"host aggregate (resource cluster)": {"weight": "0.13"} }, "time-window": "24", "unit": "hours" },
			//"peak_utilization" : { {"cpu", "memory", "network"}: {"cloud": {"weight": "0.02"}, "tenant (resource slice)": {"weight": "0.02"}, "host 
			//	aggregate (resource cluster)": {"weight": "0.02"} }, "time-window": "24", "unit": "hours" }
			
			//current_allocated_capacity, average_utilization and peak_utilization are normalized to 1
			
			//For a given object such as tenant_cpu, sum of weights across all attributes (current_allocated_capacity, average_utilization & 
			//peak_utilization) must be 1
			//E.g. net_value = cloud_cpu_current_allocated_capacity*0.85 + cloud_cpu_average_utilization*0.13 + cloud_cpu_peak_utilization*.02 + ...
			
			//For a given object such as cloud_cpu, if the current_allocated_capacity "threshold" exceeds the specified value, return "high net 
			//value" 
		}
	}

	//use cloud provider in – <cloud region id, cloud provider> – different cloud providers may need different capacities for the same VNF
	"cloudOwner" : 
	{
		"owner": "Azure",
		"capacityProperty": 
		{ 		 
			//under discussion – "capabilityProperty": {SR_IOV, ...} 
			//under discussion - host network bandwidth
		
			"controller": "multicloud",
			"request": //from R2
			"{\"vCPU\": {\"quantity\": {\"get_param\": \"REQUIRED_VCPU\"}, \"Memory\": {\"quantity\": {\"get_param\": \"REQUIRED_MEM\"}, 	
			\"unit\": 	\"GB\"}, \"Storage\": {\"quantity\": {\"get_param\": \"REQUIRED_DISK\"}, \"unit\": \"GB\"}}"
		}
		
		"owner": "OpenStack",
		"capacityProperty": 
		{ 		 
			//under discussion – "capabilityProperty": {SR_IOV, ...} 
			//under discussion - host network bandwidth
		
			"controller": "multicloud",
			"request": //from R2
			"{\"vCPU\": {\"quantity\": {\"get_param\": \"REQUIRED_VCPU\"}, \"Memory\": {\"quantity\": {\"get_param\": \"REQUIRED_MEM\"}, 	
			\"unit\": 	\"GB\"}, \"Storage\": {\"quantity\": {\"get_param\": \"REQUIRED_DISK\"}, \"unit\": \"GB\"}}"
		}
	}
}

"resources": ["vGMux"], //R2 support status – single VF module assumption per VNF 
"applicableResources": "any",
"identity": "distance-vGMux",
"policyScope": ["vCPE", "US", "INTERNATIONAL", "ip", "vGMux"],
"policyType": "AllPolicy"
}

Private Cloud Setup (e.g. OpenStack)

  • Pre-defined (including custom) flavors map to Instance types in Public Clouds
    • Pre-defined flavors are created by the Cloud Admin before the Cloud is used by ONAP for workload deployment

 VNFC to Instance Type Mapping

Operator Configuration – Multi-VIM/Cloud Plugin

The operator/service provider who uses ONAP will choose which VIMs to use and include the appropriate MultiVIM plugins in his ONAP deployment. For example, let’s assume they pick private Openstack, private VMWare, and public Azure as the platform to run their services on.

For each MultiVIM plugin, operator configures the following information:

  • For each VM flavor, the cost of that VM to the operator. Note that this costs includes the (potentially discounted) list price for the VM, support cost, and operations cost. The last one is definitely operator specific.
  • Operator also specify the cost for each feature: HA, etc
  • Note that the operator is free to choose what time duration the cost metric is specified for each of the MultiVIM plugins (e.g., cost per hour, cost per month) since they will do it consistently for each of the VIMs. 

OOF → Multi-VIM/Cloud Policy API - Key Processing Steps

For each cloud owner

...

  • Instance Type is passed in the capacity check API from OOF (Discuss) //Note, SO → MC passes OpenStack flavor name in the Heat Template/Env file
  • Convert to appropriate instance type based on intent //e.g. "Infrastructure Resource Isolation for VNF" may result in a different instance type if the cloud owner supports "Burstable QoS"

...

  • net_value_cost = net_value_cost + cost_instance_type // cost per instance type is based on policy (for R3, it is picked up from Multi Cloud configuration file)
  • net_value_cost = net_value_cost + cost_intent //e.g. "Infrastructure High Availability (HA) for VNF" may have additional cost
  • Capacity Check – Continue R2 Plan
    • Private Clouds (OpenStack based)
      • Capacity check per Tenant (OpenStack Project)
        • returns yes or no 
    • Public Clouds or Other Clouds
      • Capacity check per tenant 
        • return yes always //assumption: public cloud has infinite capacity
  • If infra capacity is finite (Private cloud or Public cloud with reserved instances) // e.g. object of interest is OpenStack Project with a quota (upper limit) for cpu, mem and local disk
    • Option A: (potentially supportable with OpenStack based clouds, other clouds to be investigated)
      • normalized_cpu_per_object = (current_cpu_used_object + instance_type_cpu*number_of_instances_of vnfc)/total_cpu_object
        • if normalized_cpu_per_object > 1, capacity check failed
      • normalized_mem_per_object = (current_mem_used_object + instance_type_mem*number_of_instances_of_vnfc)/total_mem_object
        • if normalized_mem_per_object > 1, capacity check failed 
      • normalized_disk_per_object = (current_disk_used_object + instance_type_disk*number_of_instances_of_vnfc)/total_disk_object
        • if normalized_disk_per_object > 1, capacity check failed 
      • net_value_capacity = normalized_infra_capacity = wcpu*normalized_cpu_per_object + wmem*normalized_mem_object + wdisk*normalized_disk_per_object // wcpu, wmem, wdisk are specified in a multi-cloud configuration file; wcpu + wmem + wdisk = 1
    • Option B: (under study)
      • normalized_instances_per_object = (current_used_reserved_instances + number_of_instances_of_vnfc)/total_reserved_instances
      • net_value_capacity = normalized_instances_per_object

Phase 2 (Casablanca Stretch Goal) Summary (Build on Phase 1 Work):

  • Multi-Cloud Policy Framework
    • Dynamically modify the cloud specific VNF deployment template based on cloud-specific realization of the specified intent – Impact to VNF configuration 
      • E.g. High performance Intra-DC data plane networking with several realization choices
  • Intent Support
    • Multiple realization options per Cloud Region for the specified Intent
  • Major Impact Projects:
    • Multi-Cloud
  • Minor Impact Projects:
    • SO, OOF, GNF Controller
  • Wiki Link:

References: 

(warning) The sequence diagram below expands "Multi-Cloud/VNFM Deploy Apps" in Edge Architecture & Work Items Sequence Diagram

Cloud Agnostic Intent (Policy) Workflow Summary (Phase 1 - Casablanca MVP):

Gliffy Diagram
size1200
nameCloud Agnostic Intent Execution Workflow
pagePin42

Cloud Agnostic Intent (Policy) Workflow Details (Phase 1 - Casablanca MVP):

Private Cloud Setup - OpenStack-based

VNFC to Instance Type Mapping

Step 1. SO → OOF - Get Target <Cloud Owner, Cloud Region> for the Service Instances (no code changes for R3)

Step 2. OOF → Policy - Fetch Cloud Selection Policy for Homing 

2a) OOF Processing - the fetched Policy (example below) is stored in a local data structure and is available for further use (need OOF code changes for R3)

OOF Homing Enhanced Cloud Selection Policy based on Intent -- Schema with Use Case Examples as runnable python code:
Code Block
languagepy
themeEmacs
titleOOF Homing Enhanced Cloud Selection Policy Example (Step 2a)
linenumberstrue
collapsetrue
#
#Spec Reference: https://wiki.onap.org/display/DW/Edge+Scoping+MVP+for+Casablanca+-+ONAP+Enhancements#EdgeScopingMVPforCasablanca-ONAPEnhancements-Cloud-agnosticPlacement/Networking&HomingPolicies(Phase1-CasablancaMVP,Phase2-StretchGoal)
#

from jsonschema import validate

oof_cloud_selection_policy_schema = {
        "service": {"type": "string"},
        "policyName": {"type": "string"},
        "policyDescription": {"type": "string"},
        "templateVersion": {"type": "string"},
        "version": {"type": "string"},
        "priority": {"type": "string"},
        "riskType": {"type": "string"},
        "riskLevel": {"type": "string"},
        "guard": {"type": "string"},

        "content": {
                "type": "object",
                "required": ["cloud-deployment-intent"],
                "properties" : {

                        # VNFC is not used in the OOF->MC path for R3
                        # This is kept to be consistent with the SO-> MC path
            			# As an example, vDNS VNF in ONAP has 3 VNFCs - DNS, Packet Gen & Load Balancer --
						# Each of the VNFCs could have different policies              													      									
                        "vnfc": {"type": "string"},

                        # cloud-specific realization of the specified deployment intent
                        # happens in  multi-cloud in the cloud-specific plugin
                        "cloud-deployment-intent": {
                                "type": "object",
                                "properties" : {

                                        # Cloud Type -- Azure, K8S, OpenStack, VMware VIO, Wind River Titanium
                    					# Optionally Accomodate policies per Cloud Type
                    					"Cloud Type (Cloud Provider)": {"type", "array"},
                                        
                                        "Infrastructure High Availability for VNF": {"type", "boolean"},

                                        "Infrastructure Resource Isolation for VNF": {"type", "string"},

                                        # Infrastructure Resource Isolation for VNF
                                        # Only certain pre-defined over-subscription values are allowed to
                                        # reflect practical deployment and simplify implementation for R3
                                        "Infrastructure Resource Isolation for VNF - Burstable QoS Oversubscription Percentage": {"type": "int"},
                                },
                        },
                },
        },

        "resources": {"type", "array"}, #"vgw" is also interchangeably used as "vg"
        "applicableResources": {"type", "string"},
        "identity": {"type", "string"},
        "policyScope": {"type", "array"},
        "policyType": {"type", "string"}
}

#
#Example 1: vCPE, Burstable QoS
#vCPE: Infrastructure Resource Isolation for VNF with Burstable QoS
#
oof_cloud_selection_policy_instance1 = {
        "service": "cloudSelectionPolicy",
        "policyName": "oofMulti-cloudCasablanca.cloudSelectionPolicy_vCPE_VNF",
        "policyDescription": "Cloud Selection Policy for vCPE VNFs",
        "templateVersion": "0.0.1",
        "version": "oofMulti-cloudCasablanca",
        "priority": "3",
        "riskType": "test",
        "riskLevel": "2",
        "guard": "False",

        "content": {
                "vnfc": "vgw",
                "cloud-deployment-intent": {
                        "Cloud Type (Cloud Provider)": {"VMware VIO"},
            			"Infrastructure Resource Isolation for VNF": "Burstable QoS",
            			"Infrastructure Resource Isolation for VNF - Burstable QoS Oversubscription Percentage": 25,
                },
        },

        "resources": ["vgw"], #"vgw" is also interchangeably used as "vg"
        "applicableResources": "any",
        "identity": "cloud-atrributes",
        "policyScope": ["vCPE", "US", "INTERNATIONAL", "ip", "vgw", "vgmux"],
        "policyType": "AllPolicy"
}

#
#Example 2:
#vCPE: Infrastructure Resource Isolation for VNF with Guaranteed QoS
#
oof_cloud_selection_policy_instance2 = {
        "service": "cloudSelectionPolicy",
        "policyName": "oofMulti-cloudCasablanca.cloudSelectionPolicy_vCPE_VNF",
        "policyDescription": "Cloud Selection Policy for vCPE VNFs",
        "templateVersion": "0.0.1",
        "version": "oofMulti-cloudCasablanca",
        "priority": "3",
        "riskType": "test",
        "riskLevel": "2",
        "guard": "False",

        "content": {
                "vnfc": "vgw",
                "cloud-deployment-intent": {
                 	  	"Infrastructure Resource Isolation for VNF": "Guaranteed QoS",
                },
        },

        "resources": ["vgw"], #"vgw" is also interchangeably used as "vg"
        "applicableResources": "any",
        "identity": "cloud-atrributes",
        "policyScope": ["vCPE", "US", "INTERNATIONAL", "ip", "vgw", "vgmux"],
        "policyType": "AllPolicy"
}

#
#Example 3:
#vDNS: Infrastructure HA for VNF & Infrastructure Resource Isolation for VNF with Burstable QoS
#
oof_cloud_selection_policy_instance3 = {
        "service": "cloudSelectionPolicy",
        "policyName": "oofMulti-cloudCasablanca.cloudSelectionPolicy_vDNS_VNF",
        "policyDescription": "Cloud Selection Policy for vDNS VNFs",
        "templateVersion": "0.0.1",
        "version": "oofMulti-cloudCasablanca",
        "priority": "3",
        "riskType": "test",
        "riskLevel": "2",
        "guard": "False",

        "content": {
                "vnfc": "vdns",
                "cloud-deployment-intent": {
                        "Cloud Type (Cloud Provider)": {"VMware VIO", "Azure"},
                        "Infrastructure High Availability for VNF": True,
                        "Infrastructure Resource Isolation for VNF": "Burstable QoS",
                        "Infrastructure Resource Isolation for VNF - Burstable QoS Oversubscription Percentage": 25,
                },
        },

        "resources": ["vDNS"],
        "applicableResources": "any",
        "identity": "cloud-atrributes",
        "policyScope": ["vDNS", "US", "INTERNATIONAL", "vDNS"],
        "policyType": "AllPolicy"
}

#
# Example 4:
# vDNS: Infrastructure HA for VNF & Infrastructure Resource Isolation for VNF
# with Guaranteed QoS
#
oof_cloud_selection_policy_instance4 = {
        "service": "cloudSelectionPolicy",
        "policyName": "oofMulti-cloudCasablanca.cloudSelectionPolicy_vDNS_VNF",
        "policyDescription": "Cloud Selection Policy for vDNS VNFs",
        "templateVersion": "0.0.1",
        "version": "oofMulti-cloudCasablanca",
        "priority": "3",
        "riskType": "test",
        "riskLevel": "2",
        "guard": "False",

        "content": {
                "vnfc": "vdns",
                "cloud-deployment-intent": {
                        "Infrastructure High Availability for VNF": True,
                        "Infrastructure Resource Isolation for VNF": "Guaranteed QoS",
                },
        },

        "resources": ["vDNS"],
        "applicableResources": "any",
        "identity": "cloud-atrributes",
        "policyScope": ["vDNS", "US", "INTERNATIONAL", "vDNS"],
        "policyType": "AllPolicy"
}

validate(oof_cloud_selection_policy_instance1, oof_cloud_selection_policy_schema)
validate(oof_cloud_selection_policy_instance2, oof_cloud_selection_policy_schema)
validate(oof_cloud_selection_policy_instance3, oof_cloud_selection_policy_schema)
validate(oof_cloud_selection_policy_instance4, oof_cloud_selection_policy_schema)

Step 3. OOF → A&AI - Fetch Cloud-Agnostic (Standardized) Capabilities for the Service Instance (no code changes for R3)

3a) OOF Processing - Perform Cloud Agnostic Capability check for each <cloud owner, cloud region>. OOF will prune any <cloud owner, cloud region> which is not satisfying the standardized capabilities.

Step 4. OOF → SO - Return the target <cloud owner, cloud region> for the Service Instance + deployment-intent per vnfc (code changes in OOF for R3)

OOF ↔ SO API extension - aligned to the OOF/SO API defined by SO Casablanca HPA Design to minimize the terminology set. The data between OOF to SO and SO to MC is identical -- details of the API are in section 5.

Step 5. SO → MC - Deploy VNF template in the target <cloud owner, cloud region> for the Service Instance (code changes in Multi-Cloud for R3)

5) MC Processing (need MC code changes)

  • Parse Template (e.g. OpenStack Heat Template)
    • For each VNFC, instance type in the template
      • Parse Policy JSON coming in the SO ↔ MC directives API
      • Modify template (if needed) according to Intent 
        • Intent examples of interest for R3 
          • "Infrastructure High Availability (HA) for VNF" 
          • "Infrastructure Resource Isolation for VNF"   
            • "Burstable QoS"
          • "Infrastructure Resource Isolation for VNF"   
            • "Guaranteed QoS"

  • Policy (Intent) Realization

    • Determining the flavor (OpenStack-based VIMs) # same logic applies for instance type in Azure
      • Each VNFC uniquely maps to a Flavor - for e.g. VNFC "vgw" maps to "vgw-base", "vDNS" maps to "vDNS-base"
      • Beyond Casablanca
        • VNFC intent to realization mapping happens through A&AI. 
    • "Infrastructure High Availability (HA) for VNF"
      • OpenStack-based Cloud realization 
        • For R3, Host-based anti-affinity using server groups //Beyond R3, Support other anti-affinity models at availability zone level etc. 
        • Implementation Notes: 
          • Instance "count" in heat template specifies VNFC scale out factor 
          • While dynamic injection of server group into heat template is ideal, a simple starting point could be just switching to an alternate heat template which is identical to the deployment template and additionally has server group
      • Azure realization 
        • Availability Set?

    • "Infrastructure Resource Isolation for VNF" – { "qosProperty": { {"Burstable QoS": "TRUE", "Burstable QoS Oversubscription Percentage": "25"} } }

        • Example 
          • VNFC "vgw" with "Guaranteed QoS" 
            • vCPU (Min/Max) - 16, Mem (Min/Max) - 32GB 
            • Maps to "vgw-Guaranteed-QoS" flavor for OpenStack-based VIMs
            • Same VNFC with "Burstable QoS", 25% over-subscription 
              • vCPU (Min) - 16, Mem (Min) - 32GB 
              • vCPU (Max) - 20, Mem (Max) - 40GB  
              • Maps to "vgw-Burstable-QoS-25-percent-oversubscription" flavor for OpenStack-based VIMs
          • VNFC "vDNS" with "Guaranteed QoS" & "Infrastructure High Availability"
            • Maps to "vDNS-Guaranteed-QoS" flavor and "vDNS-infrastructure-high-availability" heat template
        • Only certain pre-defined over-subscription values are allowed to simplify implementation
        • Implementation Notes:
          • While dynamic injection of limit/reservation into flavor is ideal, a simple starting would be to be to switch to a pre-defined flavor in the environment file
            • For aforementioned example
              • Original flavor - "flavor-xyz-no-oversubscription"
              • Modified flavor based on Policy - "flavor-xyz-25-percent-oversubscription" 
    • Implementation Notes:
      • From an implementation stand point, MC would be exposing a Workload Deployment Policy (Intent) API
        • Input : deployment-intent, cloud owner, cloud region, deployment template, deployment environment file, ...
        • Output : Success or Failure with reason, modified deployment template, modified deployment environment file, ...
SO ↔ MC API extension - aligned to the SO/MC API defined by SO Casablanca HPA Design to minimize the terminology set 
(This data is sent from OOF to SO. SO transparently echoes this data to MC)
Code Block
languagepy
themeEmacs
titleSO <-> MC Cloud-Agnostic Workload Deployment Policy API
linenumberstrue
collapsetrue
#
#The same information is opaquely passed from OOF to SO
#

#
#Example 1: vCPE, Burstable QoS
#vCPE: Infrastructure Resource Isolation for VNF with Burstable QoS
#
   "oof_directives":{ 
      "directives":[ 
         { 
            "vnfc_directives":[ 
               { 
                  "vnfc_id":"vgw",
                  "directives":[ 
                     { 
                        "directive_name":"Resource-Isolation-Intent-directive",
                        "attributes":[ 
						   {
                              "attribute_name": "Infrastructure Resource Isolation for VNF",
                              "attribute_value": "Burstable QoS",
                           },
						   {
                              "attribute_name": "Infrastructure Resource Isolation for VNF - Burstable QoS Oversubscription Percentage",
                              "attribute_value": "25",
                           },	    

                        ]
                     },
                  ]
               },
            ]
         },
       ]
     }


#
#Example 2:
#vCPE: Infrastructure Resource Isolation for VNF with Guaranteed QoS
#
   "oof_directives":{ 
      "directives":[ 
         { 
            "vnfc_directives":[ 
               { 
                  "vnfc_id":"vgw",
                  "directives":[ 
                     { 
                        "directive_name":"Resource-Isolation-Intent-directive",
                        "attributes":[ 
						   {
                              "attribute_name": "Infrastructure Resource Isolation for VNF",
                              "attribute_value": "Guaranteed QoS",
                           },
                        ]
                     },
                  ]
               },
            ]
         },
       ]
     }

#
#Example 3:
#vDNS: Infrastructure HA for VNF & Infrastructure Resource Isolation for VNF with Burstable QoS
#
   "oof_directives":{ 
      "directives":[ 
         { 
            "vnfc_directives":[ 
               { 
                  "vnfc_id":"vdns",
                  "directives":[ 
                     { 
                        "directive_name":"Resource-Isolation-Intent-directive",
                        "attributes":[ 
						   {
                              "attribute_name": "Infrastructure Resource Isolation for VNF",
                              "attribute_value": "Burstable QoS",
                           },
						   {
                              "attribute_name": "Infrastructure Resource Isolation for VNF - Burstable QoS Oversubscription Percentage",
                              "attribute_value": "25",
                           },	    
                        ]
                     },
					 {
                        "directive_name":"Infrastructure-HA-Intent-directive",
                        "attributes":[ 
						   {
							  "attribute_name": "Infrastructure High Availability for VNF",                              
                           },
                        ]
					 },
                  ]
               },
            ]
         },
       ]
     }

#
# Example 4:
# vDNS: Infrastructure HA for VNF & Infrastructure Resource Isolation for VNF
# with Guaranteed QoS
#
   "oof_directives":{ 
      "directives":[ 
         { 
            "vnfc_directives":[ 
               { 
                  "vnfc_id":"vdns",
                  "directives":[ 
                     { 
                        "directive_name":"Resource-Isolation-Intent-directive",
                        "attributes":[ 
						   {
                              "attribute_name": "Infrastructure Resource Isolation for VNF",
                              "attribute_value": "Guaranteed QoS",
                           },
                        ]
                     },
					 {
                        "directive_name":"Infrastructure-HA-Intent-directive",
                        "attributes":[ 
						   {
							  "attribute_name": "Infrastructure High Availability for VNF",                              
                           },
                        ]

					 },
                  ]
               },
            ]
         },
       ]
     }

Follow ups:

  • Use Cases for Integration testing
    • vCPE
      • In the current state, this use case cannot support the intent "Infra HA for VMs in a VNF"
      • This use case has been tested in R2 with OOF↔MC capacity check API
    • vDNS 
      • Can support intent "Infra HA for VMs in a VNF" and "Infrastructure Resource Isolation for VNF"
      • Nothing additional needed in OOF or MC
      • Changes needed in SO to call OOF API
        • Marcus from Intel is driving this
  • Policy DB – is there any restriction on the type of json objects that can be stored?
    • Matti to follow up with Ankit

Implementation trade offs for Casablanca (R3) and potential Dublin (R4) plan:

  • Deployment-Intent 
    • 1. "Infrastructure Resource Isolation for VNF" – { "qosProperty": { {"Burstable QoS": "TRUE", "Burstable QoS Oversubscription Percentage": "25"} } }
      • Casablanca Plan
        • Only certain pre-defined over-subscription values are allowed to reflect practical deployment and simplify implementation 
      • Dublin & Beyond Potential Plan
        • Creating instance types on demand for private clouds - to study
    • 2. Cloud-agnostic Workload Deployment Policy (Intent) 
      • Casablanca Plan
        • Cloud-Agnostic Workload Deployment Policy (Intent) can be directly mapped to specific realization (e.g. OpenStack Flavor, Azure Instance Type) to simplify implementation.  
      • Dublin & Beyond Potential Plan
        • VIM Capability Discovery to populate Intent in A&AI aligning taking into account Cloud selection policy based on cost specific to Intent (leverage similarities to HPA label discovery supported since R2)
          • VIM selection – Intent to be populated in A&AI for capability matching 
          • VIM Deployment realization - Intent(s) to specific realization mapping (e.g. OpenStack Flavor, Azure Instance Type) to be populated in A&AI 

Cloud Resource Partitioning for Differentiated QoS (Combined with Previous)

Value:

  • Applicable to all use cases
  • Casablanca Targets:
    • vCPE (Enable Tiered service offering); 5G Network Slicing (Stretch Goal) 

References:

Edge Automation Requirement:

Support three types of slices in the Cloud Infrastructure (Definition Reference: https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/)

  • Guaranteed Resource Slice (hard isolation) for various infra Resources (CPU/Memory/Network)
    • Max (limit), Min (request) are the same; resource guarantee is "Max"
    • Maps to 5G Applications such as Connected Car which fall in the category of ultra-reliable machine-type communications (ref. 1)
  • Burstable Resource Slice (soft isolation) for various infra Resources
    • Min (request) <= Max (limit); resource guarantee is "Min"
    • Maps to Burstable Network Slice such > 1Gbps broadband which fall in the category of extreme mobile broadband (ref. 1)
  • Best Effort Resource Slice (no isolation) for various infra Resources
    • No Min (request) ; resource guarantee is "None"
    • Maps to 5G Applications such as IoT which fall in the category of massive machine-type communications (ref. 1)

Implementation:

  • Leverage current HPA framework with appropriate extensions

References:

Note:

  • Any VMs/Containers which are part of a resource slice will adhere to the specs of the resource slice


ONAP ComponentLife Cycle PhaseEnhancements
PolicyDesign

Configuration Policies for Guaranteed, Burstable & Best Effort Cloud Infrastructure Resource Slices (this will apply to VMs/Containers also)

Placement Policies for Resource Slices

  • Higher (programmable) weight to Cloud Region which supports all three types of resource slices vs only two types of resource slices (Guaranteed/Best Effort)
Multi-CloudDeployResource Slice Capability Discovery
A&AIDeploy

Resource Slice Capability per Cloud Region

  • Guaranteed/Burstable/Best Effort

Resource Slice Type

  • Guaranteed/Burstable/Best Effort
OOFDeploy

Execute Resource Slice Placement Policies for Optimized Service/VNF Placement across Cloud Regions

Aggregated Infrastructure Telemetry Streams (Aligns with HPA requirements, Combining efforts with HPA)

Value

  • Edge Infrastructure Analytics complementing VNF Analytics

  • Increase the accuracy of placement decisions
  • Addresses gap in cloud provider solution – e.g. open source OpenStack does not have a comprehensive telemetry solution

Casablanca MVP

  • HPA metrics visualization
  • End-to-end use cases: vCPE, vDNS

Casablanca Stretch Goal

  • OOF to use aggregated telemetry information for fine-grained optimization

Jira
serverONAP JIRA
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyMULTICLOUD-254


ONAP, as in R2, collects the statistics/alarms/events from workloads (VMs) and take any close loop control actions such as Heal a process, scale-out, restart etc.. In R3 and beyond, infrastructure related statistics/alarms/events will be collected, generate actionable insights and take life cycle actions on the workloads.  Infrastructure statistics normally include performance counters, NIC counters, IPMI information on per physical server node basis.  To reduce the load on the ONAP, it is necessary that aggregated (summarized) information is sent to the ONAP from edge-clouds. 

As part of this activity, intention is to create aggregation micro-service that collects the data from physical nodes (over collected and other mechanisms), aggregate the information (time based aggregation, threshold based aggregation, silencing etc.,..) based on the configurable rules and export the aggregate data to DCAE.  This micro service can be instantiated by ONAP itself - one or more instances for edge-clouds at the ONAP-central itself using OOM, it could be instantiated at the edge-cloud using their own deployment tools or it could be deployed edge service providers at the regional site level.  

In R3, functionality is limited to HPA features and visualization.  R3 stretch goal: It collects information from each compute node for all HPA features and keeps track of health and resource information. It would use this information in placement decisions by OOF for accurate results.

Even though the aggregation service is being developed in Multi-Cloud project, it is expected that this can be deployed at various places. The decision to deploy at various levels can be due to performance and regulatory reasons.  Following deployments are envisaged at this time:

  • At the edge site level.
  • At the regional site level (on behalf of set of edge sites).
  • At the ONAP level (on behalf of set of edge sites)

Impacted projects (development activities)

ONAP ComponentEnhancements
Overall
  • Define models to represent summation information (Alerts/statistics/Events) for various groups
  • Defining various groups such as CPU usage, Memory usage, file descriptor usage, NIC utilization, various HPA features etc...
Multi-Cloud
  • Development activities:
    • Prometheus based monitoring & summation
    • Support for collectd for statistics collection from NFVI nodes.
    • Support for VES agent to send the aggregate data to DCAE (Used when the aggregate service is instantiated outside of ONAP control)
    • Support for DMAAP agent to send the aggregate data to DCAE (Normally used if the aggregate service is instantiated at the ONAP-Central.
    • Provide ability to add new plugins (to collect statistics as well as to export aggregation information)
    • Provide ability to upload the recording and alert rules (on per edge-cloud basis or set of edge-clouds basis)
    • Ability to auto-cleanup of time series DB (based on size allocated for this micro-service)
  • Edge-Cloud registration time (as part of ESR)
    • Check whether registration data indicates whether the aggregation service to be brought up). If so, inform the aggregation micro service to authentication and listen for statistics from that edge-cloud.
  • Run time
    • Collects the information (support for both pull/push).
    • Apply rules
    • Generate alarms
    • Export them via VES or DMAPP or any other plugins in future.
    • Update A&AI HPA (Resources and health)
AAI & ESR
  • ESR Development activities (Need to be done when Edges started to send aggregated data, so future requirement)
    • Enhancements to ESR to indicate whether aggregation service is required for this edge-cloud at the ONAP
    • Enhancements to ESR to indicate Multi-Cloud for Multi-Cloud to listen for connections and statistics requests from the edge-clouds. Information such as CA cert to use to authenticate the remote party or any other UN/PWD method.
  • HPA Enhancements
    • On per cloud-region, provide a a way to indicate whether a given HPA feature (that needs resources) resources are available and if so, the number of resources available.
    • On per cloud-region, provide a way to indicate the health of the HPA feature.
PORTALESR portal related changes to take information about the edge-cloud (CA Cert and UN/PWD information) - Future when the edges started to send aggregate data)
OOF

HPA Enhancements

  • Current HPA filter only consider whether the cloud-region supports HPA capabilities via profiles. Enhancements to consider "availability of resources" and "health of HPA" on the cloud-region.

...

OOF → Multi-VIM/Cloud Policy API - Other

...

  • vCPU (Min/Max) - 16, Mem (Min/Max) - 32GB  

...

SO → Multi-VIM/Cloud 

  • Get Cloud Region of interest
  • Parse Template (e.g. OpenStack Heat Template)
    • For each VNFC, instance type in the template
      • Get from A&AI as follows 
        • Key: <VNFC, Instance Type, Cloud Region> in A&AI
        • Value: <Policy JSON - only Intent portion> 
      • Parse Policy JSON
      • Modify template according to Intent - intent examples below
        • "Infrastructure High Availability (HA) for VNF" 
        • "Infrastructure Resource Isolation for VNF"   

OOF Processing - Key Processing Steps

Policy Management

  • Store VNFC Policy in A&AI as follows
    • Key: <VNFC (e.g. vGMux), Instance Type, Cloud Region>
    • Value: <Policy JSON - only Intent portion>

Optimization

Each service specifies an service-specific objective function that is stored as part of the service-specific policy and is used by OOF to evaluate the candidates. For simplicity of the example, let’s consider service that consists only of one VNF instance. The objective function has two components:

- distance from customer location to the VNF - the service designed assigns a weight for the distance: wd

- the cost of deploying the VNF in a location - the service designer assigns a weight for the cost: wc

OOF optimizes function: min (wd*distance + wc*cost)

If the service does not care about the cost at all, it would set wc = 0. If the service designer wants to minimize cost, he could set wd=0. Note that candidates that are too far can be eliminated by a distance constraint even before the optimization. For example, if the service has a distance constraint of at most 100 kilometers, then only those cloud regions within 100 kilometers to the customer location would be considered in the objective function evaluation.

If the service designer wants to trade off between distance and cost, for example, they might set wd = 1, wc = 2. This would mean that one $1 increase in price is as valuable as 2 kilometers in distance.

Candidate 1: $100, 100 kilometers => value: 300

Candidate 2: $150, 80 kilometers => value: 380

Candidate 3: $50, 190 kilometers => value: 290  <- pick this one 

Cloud Resource Partitioning for Differentiated QoS (Combined with Previous)

Value:

  • Applicable to all use cases
  • Casablanca Targets:
    • vCPE (Enable Tiered service offering); 5G Network Slicing (Stretch Goal) 

References:

Edge Automation Requirement:

Support three types of slices in the Cloud Infrastructure (Definition Reference: https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/)

  • Guaranteed Resource Slice (hard isolation) for various infra Resources (CPU/Memory/Network)
    • Max (limit), Min (request) are the same; resource guarantee is "Max"
    • Maps to 5G Applications such as Connected Car which fall in the category of ultra-reliable machine-type communications (ref. 1)
  • Burstable Resource Slice (soft isolation) for various infra Resources
    • Min (request) <= Max (limit); resource guarantee is "Min"
    • Maps to Burstable Network Slice such > 1Gbps broadband which fall in the category of extreme mobile broadband (ref. 1)
  • Best Effort Resource Slice (no isolation) for various infra Resources
    • No Min (request) ; resource guarantee is "None"
    • Maps to 5G Applications such as IoT which fall in the category of massive machine-type communications (ref. 1)

Implementation:

  • Leverage current HPA framework with appropriate extensions

References:

Note:

  • Any VMs/Containers which are part of a resource slice will adhere to the specs of the resource slice

...

Configuration Policies for Guaranteed, Burstable & Best Effort Cloud Infrastructure Resource Slices (this will apply to VMs/Containers also)

Placement Policies for Resource Slices

  • Higher (programmable) weight to Cloud Region which supports all three types of resource slices vs only two types of resource slices (Guaranteed/Best Effort)

...

Resource Slice Capability per Cloud Region

  • Guaranteed/Burstable/Best Effort

Resource Slice Type

  • Guaranteed/Burstable/Best Effort

...

Execute Resource Slice Placement Policies for Optimized Service/VNF Placement across Cloud Regions

Aggregated Infrastructure Telemetry Streams (Aligns with HPA requirements, Combining efforts with HPA)

Value

  • Edge Infrastructure Analytics complementing 5G VNF Analytics

Jira
serverONAP JIRA
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyMULTICLOUD-254

ONAP, as in R2, collects the statistics/alarms/events from workloads (VMs) and take any close loop control actions such as Heal a process, scale-out, restart etc.. In R3, infrastructure related statistics/alarms/events will be collected, generate actionable insights and take life cycle actions on the workloads.  Infrastructure statistics normally include performance counters, NIC counters, IPMI information on per physical server node basis.  To reduce the load on the ONAP, it is necessary that aggregated (summarized) information is sent to the ONAP from edge-clouds. 

As part of this activity, intention is to create aggregation micro-service that collects the data from physical nodes (over collected and other mechanisms), aggregate the information (time based aggregation, threshold based aggregation, silencing etc.,..) based on the configurable rules and export the aggregate data to DCAE.  This micro service can be instantiated by ONAP itself - one or more instances for edge-clouds at the ONAP-central itself using OOM, it could be instantiated at the edge-cloud using their own deployment tools or it could be deployed edge service providers at the regional site level.  

Impacted projects (development activities)

ONAP ComponentEnhancements
Overall
  • Define models to represent summation information (Alerts/statistics/Events) for various groups
  • Defining various groups such as CPU usage, Memory usage, file descriptor usage, NIC utilization, various HPA features etc...
Multi-Cloud
  • Development activities:
    • Prometheus based monitoring & summation
    • Support for collectd for statistics collection from NFVI nodes.
    • Support for VES agent to send the aggregate data to DCAE (Used when the aggregate service is instantiated outside of ONAP control)
    • Support for DMAAP agent to send the aggregate data to DCAE (Normally used if the aggregate service is instantiated at the ONAP-Central.
    • Provide ability to add new plugins (to collect statistics as well as to export aggregation information)
    • Provide ability to upload the recording and alert rules (on per edge-cloud basis or set of edge-clouds basis)
    • Ability to auto-cleanup of time series DB (based on size allocated for this micro-service)
  • Edge-Cloud registration time (as part of ESR)
    • Check whether registration data indicates whether the aggregation service to be brought up). If so, inform the aggregation micro service to authentication and listen for statistics from that edge-cloud.
  • Run time
    • Collects the information (support for both pull/push).
    • Apply rules
    • Generate alarms
    • Export them via VES or DMAPP or any other plugins in future.
AAI & ESR
  • Development activities
    • Enhancements to ESR to indicate whether aggregation service is required for this edge-cloud at the ONAP.
    • Enhancements to ESR to indicate Multi-Cloud for Multi-Cloud to listen for connections and statistics requests from the edge-clouds. Information such as CA cert to use to authenticate the remote party or any other UN/PWD method.
PORTALESR portal related changes to take information about the edge-cloud (CA Cert and UN/PWD information)
DCAE & DMAPPNone expected??

Life Cycle stages related functions

ONAP ComponentLife cycle phaseActivities
AAI and ESRDeploy & Run time
  • Add/Modify/Delete recording and alerting rules
AAI and ESRRun time
  • Add/Modify/Delete Edge-cloud information



Multi-CloudRun time
  • Get Edge information from A&AI whenever Edge-Cloud is added or removed.
  • Prepare to wait for information from that Edge-cloud
  • Receive information from edge-cloud and put it in the time series DB.
  • Summation based on recording & alerting rules
    • Export information to DCAE via DMAPP or VES
OOFRun time
  • HPA filter changes to accommodate health of the hardware/software on a feature and availability of resources.

High level architecture slides:

View file
namePromethus-aggregation.pdf
height250

ONAP Edge Analytics with DCAE/DMaaP independent of closed loop (Stretch Goal - Beyond Casablanca)

Value

  • 5G Analytics

ONAP ComponentLife cycle phaseEnhancements
OOM - ONAP CentralDeploy
  • Separate ONAP-edge Instance per 'edge domain', (ie., separate from onap-central instance, of course)
    • Note: Independent of any Edge CP's Orchestration components.
  • SP uses a central-OOM with a 'policy' for deployment of an onap-edge instance, e.g., xyz edge provider with abc components, etc.
    • However, onap-edge instance can be 'lighter weight' with subset of components needed (per MVP discussed below)
    • Desirable to managed as a separate K8s cluster (ie., separate from onap-central instance, of course) and, only for onap-edge use, ie., don't use for other 'workloads' like network apps or 3rd party apps
  • Central OOM to deploy the following ONAP edge instance
    • DMaaP with mirror capability


Multi-Cloud Deployment in Edge Cloud (Stretch Goal - Beyond Casablanca)

Jira
serverONAP JIRA
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyMULTICLOUD-262

...