Summary: Edge Architecture & Work Items#ONAPEdgeMVP 




Distributed Edge Cloud Infrastructure Object Hierarchy (Stretch Goal - Beyond Casablanca)

Value:

References: 



ONAP ComponentLife Cycle PhaseEnhancements
Multi-CloudDeploy

Support Distributed Cloud Infrastructure Capability Discovery (Note 1, Note 2)

A&AIDeploy

Support Standardized Distributed Cloud Infrastructure Object Hierarchy & Capability Database (Ref. 1)

  • Loose coupling between HW objects (private cloud) and SW objects (private and public clouds)
  • Includes Standardized Capabilities across clouds & Capabilities unique to certain clouds
  • Note:
    • Multi-Cloud Distributed Cloud Infrastructure Capability Discovery process will populate the aforementioned database
OOFDeploy

Execute Distributed Cloud Infrastructure Placement Policies for Optimized Service/VNF Placement across Cloud Regions (Note 3, Note 4)


SODeploy

Extend SO ↔ OOF API to support data opaque to SO (Note 5)

Extend SO ↔ MC API to support data opaque to SO (Note 6)

Assumption for Policy, SO, OOF:

Note 1: 

Note 2:

Note 3:

Note 4:

Note 5:

Note 6:

Cloud-agnostic Placement/Networking & Homing Policies (Phase 1 - Casablanca MVP, Phase 2 - Stretch Goal)

End-to-end use case Applicability:

Value:


Phase 1 (Casablanca MVP) Summary:

Phase 2 (Casablanca Stretch Goal) Summary (Build on Phase 1 Work):

References: 

(warning) The sequence diagram below expands "Multi-Cloud/VNFM Deploy Apps" in Edge Architecture & Work Items Sequence Diagram

Cloud Agnostic Intent (Policy) Workflow Summary (Phase 1 - Casablanca MVP):


Cloud Agnostic Intent (Policy) Workflow Details (Phase 1 - Casablanca MVP):

Private Cloud Setup - OpenStack-based

VNFC to Instance Type Mapping

Step 1. SO → OOF - Get Target <Cloud Owner, Cloud Region> for the Service Instances (no code changes for R3)

Step 2. OOF → Policy - Fetch Cloud Selection Policy for Homing 

2a) OOF Processing - the fetched Policy (example below) is stored in a local data structure and is available for further use (need OOF code changes for R3)

OOF Homing Enhanced Cloud Selection Policy based on Intent -- Schema with Use Case Examples as runnable python code:
#
#Spec Reference: https://wiki.onap.org/display/DW/Edge+Scoping+MVP+for+Casablanca+-+ONAP+Enhancements#EdgeScopingMVPforCasablanca-ONAPEnhancements-Cloud-agnosticPlacement/Networking&HomingPolicies(Phase1-CasablancaMVP,Phase2-StretchGoal)
#

from jsonschema import validate

oof_cloud_selection_policy_schema = {
        "service": {"type": "string"},
        "policyName": {"type": "string"},
        "policyDescription": {"type": "string"},
        "templateVersion": {"type": "string"},
        "version": {"type": "string"},
        "priority": {"type": "string"},
        "riskType": {"type": "string"},
        "riskLevel": {"type": "string"},
        "guard": {"type": "string"},

        "content": {
                "type": "object",
                "required": ["cloud-deployment-intent"],
                "properties" : {

                        # VNFC is not used in the OOF->MC path for R3
                        # This is kept to be consistent with the SO-> MC path
            			# As an example, vDNS VNF in ONAP has 3 VNFCs - DNS, Packet Gen & Load Balancer --
						# Each of the VNFCs could have different policies              													      									
                        "vnfc": {"type": "string"},

                        # cloud-specific realization of the specified deployment intent
                        # happens in  multi-cloud in the cloud-specific plugin
                        "cloud-deployment-intent": {
                                "type": "object",
                                "properties" : {

                                        # Cloud Type -- Azure, K8S, OpenStack, VMware VIO, Wind River Titanium
                    					# Optionally Accomodate policies per Cloud Type
                    					"Cloud Type (Cloud Provider)": {"type", "array"},
                                        
                                        "Infrastructure High Availability for VNF": {"type", "boolean"},

                                        "Infrastructure Resource Isolation for VNF": {"type", "string"},

                                        # Infrastructure Resource Isolation for VNF
                                        # Only certain pre-defined over-subscription values are allowed to
                                        # reflect practical deployment and simplify implementation for R3
                                        "Infrastructure Resource Isolation for VNF - Burstable QoS Oversubscription Percentage": {"type": "int"},
                                },
                        },
                },
        },

        "resources": {"type", "array"}, #"vgw" is also interchangeably used as "vg"
        "applicableResources": {"type", "string"},
        "identity": {"type", "string"},
        "policyScope": {"type", "array"},
        "policyType": {"type", "string"}
}

#
#Example 1: vCPE, Burstable QoS
#vCPE: Infrastructure Resource Isolation for VNF with Burstable QoS
#
oof_cloud_selection_policy_instance1 = {
        "service": "cloudSelectionPolicy",
        "policyName": "oofMulti-cloudCasablanca.cloudSelectionPolicy_vCPE_VNF",
        "policyDescription": "Cloud Selection Policy for vCPE VNFs",
        "templateVersion": "0.0.1",
        "version": "oofMulti-cloudCasablanca",
        "priority": "3",
        "riskType": "test",
        "riskLevel": "2",
        "guard": "False",

        "content": {
                "vnfc": "vgw",
                "cloud-deployment-intent": {
                        "Cloud Type (Cloud Provider)": {"VMware VIO"},
            			"Infrastructure Resource Isolation for VNF": "Burstable QoS",
            			"Infrastructure Resource Isolation for VNF - Burstable QoS Oversubscription Percentage": 25,
                },
        },

        "resources": ["vgw"], #"vgw" is also interchangeably used as "vg"
        "applicableResources": "any",
        "identity": "cloud-atrributes",
        "policyScope": ["vCPE", "US", "INTERNATIONAL", "ip", "vgw", "vgmux"],
        "policyType": "AllPolicy"
}

#
#Example 2:
#vCPE: Infrastructure Resource Isolation for VNF with Guaranteed QoS
#
oof_cloud_selection_policy_instance2 = {
        "service": "cloudSelectionPolicy",
        "policyName": "oofMulti-cloudCasablanca.cloudSelectionPolicy_vCPE_VNF",
        "policyDescription": "Cloud Selection Policy for vCPE VNFs",
        "templateVersion": "0.0.1",
        "version": "oofMulti-cloudCasablanca",
        "priority": "3",
        "riskType": "test",
        "riskLevel": "2",
        "guard": "False",

        "content": {
                "vnfc": "vgw",
                "cloud-deployment-intent": {
                 	  	"Infrastructure Resource Isolation for VNF": "Guaranteed QoS",
                },
        },

        "resources": ["vgw"], #"vgw" is also interchangeably used as "vg"
        "applicableResources": "any",
        "identity": "cloud-atrributes",
        "policyScope": ["vCPE", "US", "INTERNATIONAL", "ip", "vgw", "vgmux"],
        "policyType": "AllPolicy"
}

#
#Example 3:
#vDNS: Infrastructure HA for VNF & Infrastructure Resource Isolation for VNF with Burstable QoS
#
oof_cloud_selection_policy_instance3 = {
        "service": "cloudSelectionPolicy",
        "policyName": "oofMulti-cloudCasablanca.cloudSelectionPolicy_vDNS_VNF",
        "policyDescription": "Cloud Selection Policy for vDNS VNFs",
        "templateVersion": "0.0.1",
        "version": "oofMulti-cloudCasablanca",
        "priority": "3",
        "riskType": "test",
        "riskLevel": "2",
        "guard": "False",

        "content": {
                "vnfc": "vdns",
                "cloud-deployment-intent": {
                        "Cloud Type (Cloud Provider)": {"VMware VIO", "Azure"},
                        "Infrastructure High Availability for VNF": True,
                        "Infrastructure Resource Isolation for VNF": "Burstable QoS",
                        "Infrastructure Resource Isolation for VNF - Burstable QoS Oversubscription Percentage": 25,
                },
        },

        "resources": ["vDNS"],
        "applicableResources": "any",
        "identity": "cloud-atrributes",
        "policyScope": ["vDNS", "US", "INTERNATIONAL", "vDNS"],
        "policyType": "AllPolicy"
}

#
# Example 4:
# vDNS: Infrastructure HA for VNF & Infrastructure Resource Isolation for VNF
# with Guaranteed QoS
#
oof_cloud_selection_policy_instance4 = {
        "service": "cloudSelectionPolicy",
        "policyName": "oofMulti-cloudCasablanca.cloudSelectionPolicy_vDNS_VNF",
        "policyDescription": "Cloud Selection Policy for vDNS VNFs",
        "templateVersion": "0.0.1",
        "version": "oofMulti-cloudCasablanca",
        "priority": "3",
        "riskType": "test",
        "riskLevel": "2",
        "guard": "False",

        "content": {
                "vnfc": "vdns",
                "cloud-deployment-intent": {
                        "Infrastructure High Availability for VNF": True,
                        "Infrastructure Resource Isolation for VNF": "Guaranteed QoS",
                },
        },

        "resources": ["vDNS"],
        "applicableResources": "any",
        "identity": "cloud-atrributes",
        "policyScope": ["vDNS", "US", "INTERNATIONAL", "vDNS"],
        "policyType": "AllPolicy"
}

validate(oof_cloud_selection_policy_instance1, oof_cloud_selection_policy_schema)
validate(oof_cloud_selection_policy_instance2, oof_cloud_selection_policy_schema)
validate(oof_cloud_selection_policy_instance3, oof_cloud_selection_policy_schema)
validate(oof_cloud_selection_policy_instance4, oof_cloud_selection_policy_schema)

Step 3. OOF → A&AI - Fetch Cloud-Agnostic (Standardized) Capabilities for the Service Instance (no code changes for R3)

3a) OOF Processing - Perform Cloud Agnostic Capability check for each <cloud owner, cloud region>. OOF will prune any <cloud owner, cloud region> which is not satisfying the standardized capabilities.

Step 4. OOF → SO - Return the target <cloud owner, cloud region> for the Service Instance + deployment-intent per vnfc (code changes in OOF for R3)

OOF ↔ SO API extension - aligned to the OOF/SO API defined by SO Casablanca HPA Design to minimize the terminology set. The data between OOF to SO and SO to MC is identical -- details of the API are in section 5.

Step 5. SO → MC - Deploy VNF template in the target <cloud owner, cloud region> for the Service Instance (code changes in Multi-Cloud for R3)

5) MC Processing (need MC code changes)

SO ↔ MC API extension - aligned to the SO/MC API defined by SO Casablanca HPA Design to minimize the terminology set 
(This data is sent from OOF to SO. SO transparently echoes this data to MC)
#
#The same information is opaquely passed from OOF to SO
#

#
#Example 1: vCPE, Burstable QoS
#vCPE: Infrastructure Resource Isolation for VNF with Burstable QoS
#
   "oof_directives":{ 
      "directives":[ 
         { 
            "vnfc_directives":[ 
               { 
                  "vnfc_id":"vgw",
                  "directives":[ 
                     { 
                        "directive_name":"Resource-Isolation-Intent-directive",
                        "attributes":[ 
						   {
                              "attribute_name": "Infrastructure Resource Isolation for VNF",
                              "attribute_value": "Burstable QoS",
                           },
						   {
                              "attribute_name": "Infrastructure Resource Isolation for VNF - Burstable QoS Oversubscription Percentage",
                              "attribute_value": "25",
                           },	    

                        ]
                     },
                  ]
               },
            ]
         },
       ]
     }


#
#Example 2:
#vCPE: Infrastructure Resource Isolation for VNF with Guaranteed QoS
#
   "oof_directives":{ 
      "directives":[ 
         { 
            "vnfc_directives":[ 
               { 
                  "vnfc_id":"vgw",
                  "directives":[ 
                     { 
                        "directive_name":"Resource-Isolation-Intent-directive",
                        "attributes":[ 
						   {
                              "attribute_name": "Infrastructure Resource Isolation for VNF",
                              "attribute_value": "Guaranteed QoS",
                           },
                        ]
                     },
                  ]
               },
            ]
         },
       ]
     }

#
#Example 3:
#vDNS: Infrastructure HA for VNF & Infrastructure Resource Isolation for VNF with Burstable QoS
#
   "oof_directives":{ 
      "directives":[ 
         { 
            "vnfc_directives":[ 
               { 
                  "vnfc_id":"vdns",
                  "directives":[ 
                     { 
                        "directive_name":"Resource-Isolation-Intent-directive",
                        "attributes":[ 
						   {
                              "attribute_name": "Infrastructure Resource Isolation for VNF",
                              "attribute_value": "Burstable QoS",
                           },
						   {
                              "attribute_name": "Infrastructure Resource Isolation for VNF - Burstable QoS Oversubscription Percentage",
                              "attribute_value": "25",
                           },	    
                        ]
                     },
					 {
                        "directive_name":"Infrastructure-HA-Intent-directive",
                        "attributes":[ 
						   {
							  "attribute_name": "Infrastructure High Availability for VNF",                              
                           },
                        ]
					 },
                  ]
               },
            ]
         },
       ]
     }

#
# Example 4:
# vDNS: Infrastructure HA for VNF & Infrastructure Resource Isolation for VNF
# with Guaranteed QoS
#
   "oof_directives":{ 
      "directives":[ 
         { 
            "vnfc_directives":[ 
               { 
                  "vnfc_id":"vdns",
                  "directives":[ 
                     { 
                        "directive_name":"Resource-Isolation-Intent-directive",
                        "attributes":[ 
						   {
                              "attribute_name": "Infrastructure Resource Isolation for VNF",
                              "attribute_value": "Guaranteed QoS",
                           },
                        ]
                     },
					 {
                        "directive_name":"Infrastructure-HA-Intent-directive",
                        "attributes":[ 
						   {
							  "attribute_name": "Infrastructure High Availability for VNF",                              
                           },
                        ]

					 },
                  ]
               },
            ]
         },
       ]
     }

Follow ups:

Implementation trade offs for Casablanca (R3) and potential Dublin (R4) plan:

Cloud Resource Partitioning for Differentiated QoS (Combined with Previous)

Value:

References:

Edge Automation Requirement:

Support three types of slices in the Cloud Infrastructure (Definition Reference: https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/)

Implementation:

References:

Note:


ONAP ComponentLife Cycle PhaseEnhancements
PolicyDesign

Configuration Policies for Guaranteed, Burstable & Best Effort Cloud Infrastructure Resource Slices (this will apply to VMs/Containers also)

Placement Policies for Resource Slices

  • Higher (programmable) weight to Cloud Region which supports all three types of resource slices vs only two types of resource slices (Guaranteed/Best Effort)
Multi-CloudDeployResource Slice Capability Discovery
A&AIDeploy

Resource Slice Capability per Cloud Region

  • Guaranteed/Burstable/Best Effort

Resource Slice Type

  • Guaranteed/Burstable/Best Effort
OOFDeploy

Execute Resource Slice Placement Policies for Optimized Service/VNF Placement across Cloud Regions

Aggregated Infrastructure Telemetry Streams (Aligns with HPA requirements, Combining efforts with HPA)

Value

Casablanca MVP

Casablanca Stretch Goal


ONAP, as in R2, collects the statistics/alarms/events from workloads (VMs) and take any close loop control actions such as Heal a process, scale-out, restart etc.. In R3 and beyond, infrastructure related statistics/alarms/events will be collected, generate actionable insights and take life cycle actions on the workloads.  Infrastructure statistics normally include performance counters, NIC counters, IPMI information on per physical server node basis.  To reduce the load on the ONAP, it is necessary that aggregated (summarized) information is sent to the ONAP from edge-clouds. 

As part of this activity, intention is to create aggregation micro-service that collects the data from physical nodes (over collected and other mechanisms), aggregate the information (time based aggregation, threshold based aggregation, silencing etc.,..) based on the configurable rules and export the aggregate data to DCAE.  This micro service can be instantiated by ONAP itself - one or more instances for edge-clouds at the ONAP-central itself using OOM, it could be instantiated at the edge-cloud using their own deployment tools or it could be deployed edge service providers at the regional site level.  

In R3, functionality is limited to HPA features and visualization.  R3 stretch goal: It collects information from each compute node for all HPA features and keeps track of health and resource information. It would use this information in placement decisions by OOF for accurate results.

Even though the aggregation service is being developed in Multi-Cloud project, it is expected that this can be deployed at various places. The decision to deploy at various levels can be due to performance and regulatory reasons.  Following deployments are envisaged at this time:

Impacted projects (development activities)

ONAP ComponentEnhancements
Overall
  • Define models to represent summation information (Alerts/statistics/Events) for various groups
  • Defining various groups such as CPU usage, Memory usage, file descriptor usage, NIC utilization, various HPA features etc...
Multi-Cloud
  • Development activities:
    • Prometheus based monitoring & summation
    • Support for collectd for statistics collection from NFVI nodes.
    • Support for VES agent to send the aggregate data to DCAE (Used when the aggregate service is instantiated outside of ONAP control)
    • Support for DMAAP agent to send the aggregate data to DCAE (Normally used if the aggregate service is instantiated at the ONAP-Central.
    • Provide ability to add new plugins (to collect statistics as well as to export aggregation information)
    • Provide ability to upload the recording and alert rules (on per edge-cloud basis or set of edge-clouds basis)
    • Ability to auto-cleanup of time series DB (based on size allocated for this micro-service)
  • Edge-Cloud registration time (as part of ESR)
    • Check whether registration data indicates whether the aggregation service to be brought up). If so, inform the aggregation micro service to authentication and listen for statistics from that edge-cloud.
  • Run time
    • Collects the information (support for both pull/push).
    • Apply rules
    • Generate alarms
    • Export them via VES or DMAPP or any other plugins in future.
    • Update A&AI HPA (Resources and health)
AAI & ESR
  • ESR Development activities (Need to be done when Edges started to send aggregated data, so future requirement)
    • Enhancements to ESR to indicate whether aggregation service is required for this edge-cloud at the ONAP
    • Enhancements to ESR to indicate Multi-Cloud for Multi-Cloud to listen for connections and statistics requests from the edge-clouds. Information such as CA cert to use to authenticate the remote party or any other UN/PWD method.
  • HPA Enhancements
    • On per cloud-region, provide a a way to indicate whether a given HPA feature (that needs resources) resources are available and if so, the number of resources available.
    • On per cloud-region, provide a way to indicate the health of the HPA feature.
PORTALESR portal related changes to take information about the edge-cloud (CA Cert and UN/PWD information) - Future when the edges started to send aggregate data)
OOF

HPA Enhancements

  • Current HPA filter only consider whether the cloud-region supports HPA capabilities via profiles. Enhancements to consider "availability of resources" and "health of HPA" on the cloud-region.

Life Cycle stages related functions

ONAP ComponentLife cycle phaseActivities
AAI and ESRDeploy & Run time
  • Add/Modify/Delete recording and alerting rules
AAI and ESRRun time
  • Add/Modify/Delete Edge-cloud information



Multi-CloudRun time
  • Get Edge information from A&AI whenever Edge-Cloud is added or removed.
  • Prepare to wait for information from that Edge-cloud
  • Receive information from edge-cloud and put it in the time series DB.
  • Summation based on recording & alerting rules
    • Export information to DCAE via DMAPP or VES
OOFRun time
  • HPA filter changes to accommodate health of the hardware/software on a feature and availability of resources.

High level architecture slides:

ONAP Edge Analytics with DCAE/DMaaP independent of closed loop (Stretch Goal - Beyond Casablanca)

Value

ONAP ComponentLife cycle phaseEnhancements
OOM - ONAP CentralDeploy
  • Separate ONAP-edge Instance per 'edge domain', (ie., separate from onap-central instance, of course)
    • Note: Independent of any Edge CP's Orchestration components.
  • SP uses a central-OOM with a 'policy' for deployment of an onap-edge instance, e.g., xyz edge provider with abc components, etc.
    • However, onap-edge instance can be 'lighter weight' with subset of components needed (per MVP discussed below)
    • Desirable to managed as a separate K8s cluster (ie., separate from onap-central instance, of course) and, only for onap-edge use, ie., don't use for other 'workloads' like network apps or 3rd party apps
  • Central OOM to deploy the following ONAP edge instance
    • DMaaP with mirror capability


Multi-Cloud Deployment in Edge Cloud (Stretch Goal - Beyond Casablanca)

Value: