General Background

A broad set of transformations are taking place:

• Business transformation: OTT services, faster TTM, Monetization
• Technical transformation: QoE, ULL, SDN/NFV/OMEC integration, Edge Analytics, Big data, Virtualization, Automation, C->E, R->E
• Architectural transformation: 4 views “NORMA-like” Cloud, ECOMP, Flexible architecture (RAN, Core, CDN, Application delivery, Automation, IoT, fog,..)
• Industrial transformation: ICT&E

To efficiently and effectively deploy 5G network supporting ultra low latency and high bandwidth mobile network, we need to deploy variety of applications and workload at the edge and close to the mobile end user devices (UE or IoT).  That would include various virtualized RAN and core network elements, content (video), various applications (AR / VR, industrial automation, connected cars, etc.).  We might deploy near-real time network optimization, customer experience / UE performance enhancement applications at edge.  Edge cloud must support deployment of third party application (e.g. Value added optional services, Marketing, Advertising, etc.).  We must deploy mechanisms to collect real time radio network information, process them in real-time (e.g. Geo Location data), summarize, anonymize, etc. and make them available to third party applications deployed at the edge or central location or outside service provider environment.  Edge data collection could also be used for training machine learning models and fully trained models can be deployed at the edge to support network optimization.

The need

End users and other devices, cyber-physical systems will benefit from a broad set of context information that can enhance and enrich the delivery of a broad set of applications.

Service Deployment Goal

Deliver Application SLAs while minimizing TCO.

Application Profiles

Note 1:  API Details

• e.g., MEC APIs - Location info, Radio control info etc.
• e.g., Cloud APIs - IaaS/PaaS + Context Awareness (time, places, activity, weather etc.)  

Edge Infrastructure

This diverse work load will require somewhat heterogeneous cloud environment, including Graphical Processing Unit, highly programmable network accelerators, etc., in addition to traditional compute, storage, etc.

To support edge deployment, we need:

1)     Rich information / data model to discover and capture hardware resources deployed at the edge and request right type of resource to meet unique application needs.

2)     Must support workload deployment options such as VM, Container (e.g. Kubernetes) on VM or bare metal

3)     Must support a very small foot print to an edge location supporting a metropolitan area with verity of workload deployment

4)     Edge cloud could be on customer premises – Factory automation

5)     Must provide efficient network infrastructure that support slicing and QoS configuration options to meet various mobility services need

6)     Must support policy driven auto recovery / scale up scale down

Edge Infrastructure Profiles

( example based on Akraino Edge Stack..but, need to generalize)  

Edge Infrastructure Profile Summary

ONAP Activity Goal #1: ONAP requires IaaS/PaaS attributes (see ongoing work – Distributed Edge Cloud Infrastructure Enablement in ONAP, 5G Items for Casablanca) from Cloud providers for Infrastructure profiles that allow Distributed, Highly-secure, Config/Cloud-diverse, Capacity-constrained and Peformance/Isolation-aware

• Distributed
  • 1000's of edge locations of varying capacity
  • Casablanca - Implementation
    • 10-100 edge locations (simple starting point)
• Peformance-awareness
  
  • GPU, FPGAs, SR-IOV etc.
  • Casablanca - Implementation
    • SR-IOV desired for Data Plane (5G CU-UP)
    • NIC offload desired for tunnel encap/decap e.g. 5G CU-UP GTP tunnel
• Resource Isolation through fine-grained QoS
  
  • Support both Latency-sensitive and General purpose applications
  • Support ONAP Management plane components in the same cloud with Workloads
  • Casablanca - Implementation
    • Min/Max resource reservation model desired
• Security
  • Workloads are often deployed in external (non-dc-type) locations and need HW security (TPM etc.) 
  • 3rd party applications which need additional HW security (VM, Containers in VM etc.) and SW security (Inter-component TLS etc.)
  • Casablanca - Implementation
    
    • Edge Clouds with private IP addresses, i.e. reachable via private connections 
    • For example, edge cloud in a public cloud provider reachable via AWS direct connect or Azure express route or Google partner interconnect 
• Capacity constraints
  • Very small footprint (few nodes per physical location), Medium footprint (10's of nodes per physical location), Large footprint (100's of nodes per physical location)
  • Casablanca - Deployment
    • Need number of cores per servers; Need storage capacity/pool
• Cloud Diversity

  • Private and Public Cloud Providers

  • Casablanca - Implementation
    • Note: ONAP currently supports private edge clouds based on VMware VIO, Wind River Titanium Cloud, Upstream OpenStack
    • Desire to have at least one Public Cloud Provider (Azure, AWS, GCE etc.) as an Edge Cloud Provider
      • ONAP central instantiates an Edge Cloud instance (blue cloud provider in gliffy) via a IaaS API to cloud provider
      • ONAP central instantiates one or more ONAP edge components as need, e.g. DCAE
      • ONAP central instantiates one or more NFs, e.g. 5G CU-UP/CP
• Configuration Diversity
  • 5G Factory Automation, 5G General Mobility Services etc. – User Plane components (DU, CU-UP, UPF etc.) 

ONAP Edge Automation 

ONAP Activity Goal #2: Define hierarchical ONAP Central/Edge Architecture/functional interactions (API reference points) to support aforementioned Application/Infrastrcuture profile in Any "Cloud" (internal Business Unit or external Partner) at Any "Location" edge, regional or central. 

Details:

• Optimal Distribution of Intelligence and Control, includes distributed data collection and localized processing of intelligence
  • Support for various edge sizes
  • Scaling needs - Hierarchical federation (over and beyond auto scale-out of ONAP services) - Distribution of orchestration, fabric control, stats/faults/log collection and distributed processing of same (Regional Controllers)
  • Optimal placing of edge applications. For example placing edge applications in the best edge(s) considering various constraints (e.g Proximity to end user,  Radio/BW availability, cost,  accelerators availability - HPA,   Geo-affinity regulations, trusted infrastrcture of edge, device characteristics and resource availability to take up load etc...),  Auto creation of constraints is one requirement.
  • Providing contextual information to application services after gathering information from 5G network functions.
• Autonomic Control, Management and Operations of distributed service chains
  • Traffic  steering to the right edge applications (e.g  Programming UE classifier of UPF) and dynamic SFC within VNFCs of edge application.
  • Supporting various workload types (VMs, Containers etc.)
  • Deploying IoT specific infrastructure software in edges such as EdgeXFoundry.
  • Supporting multi-tenancy to place workloads in Edges belonging to various organizations
  • Performance determinism and high throughput edge 
  • Securing confidential information/keys/secrets and detecting any software tampering at edges

Few examples:  on scaling -   OOM based scaling may not be good enough and  there may be a need to  offload some ONAP functionality to regional level as the target number of edge clouds could be in tens of thousands.   Also, to reduce amount of data to central ONAP services for analytics,  there is a need for offloading DCAE functions to regional level, which could involve  identifying real time data sources, collecting and analyzing the data and disseminating output data to central ONAP function.  Controlling fabric (L2/L3 switches in edge-clouds and WAN links) is another function that may require offloading some ONAP SDNC functions to regional sites. 

ONAP Hierarchical (Central/Edge) Architecture

Additional Notes on Gliffy

• Control Loop Subcommittee - useful links:
  • Control Loop Sub Committee
  • Control Loop Sub Committee - Beijing Functional Requirements
  • Control Loop Sub Committee - Beijing Flow Diagrams

• Cloud Provider Business Unit: Provides hosting of Workloads, ie., IaaS/PaaS
  • SP installs and manages ONAP in separate 'Management Cloud' instances
  • SP installs and manages Network Services + 3rd Party Apps in separate 'Services/Apps Cloud' instances
• Cloud Provider Business Unit: Provides SaaS, eg., Analytics/Closed Loop as a Service, LCM of Apps, etc..

• • ONAP Edge may not be needed

 Sequence Diagram (WIP)

ONAP Edge MVP for Casablanca 

 Edge Application (refer to app classification) / Infrastructure (refer to infra profile summary) Requirement – ONAP Project impact

Assuming operational aspects only, ie., in Design/Deploy/Operate

• • Separate instances approach
  • Note: components not mentioned for onap-edge, eg., A&AI, are assume to be present only in onap-central
  • onap-edge Option A (as of current ONAP capabilities) with following components to summarize events/data to onap-central:
    • DCAE (also, to summarize events/data to onap-central)
      • VES Collector and Analytics Microservices (TCA, Holmes etc.)
    • DMaaP (communication for onap-edge DCAE microservices)
      • High speed pub/sub
    • Multi Cloud (to onap-edge DCAE VES Collector)
      • Standardize Infrastructure Events/Alerts/Faults/Metrics and communicate to onap-edge DCAE VES Collector through APIs
  • onap-edge Option B  (Desired?):
    • Option A, plus 
    • DCAE Enhancements 
      • Support New microservice based Apps – Slice monitoring, performance analysis, fault analysis, root cause analysis, centralized SON applications, Optimization (SON Drive Test Minimization etc.), ML methodologies for various apps
      • Note: New DCAE Apps are from the Application profile table
    • Multi Cloud Enhancements
      • Edge Cloud supports Infrastructure Analytics Micro services
      • Multi Cloud standardizes "ONSET" and "ABATE" of Infrastructure Alerts/Faults and communicates them to onap-central's DCAE VES Collector and/or Policy
  • Option C:
    • Option B, plus
    • onap-edge with Static/Dynamic Policy - PDP and limited to only Closed Loop
    • Policy may depend on current deployment state and also might need service context for the service component such as VNFs? So,  other onap components may be involved at the edge? 
• Key Changes needed for Option B:
  • OOM Enhanacements
    • SP uses a central-OOM with a 'policy' for deployment of an onap-edge instance, e.g., xyz edge provider with abc components, etc.
      • However, onap-edge instance can be 'lighter weight' with subset of components needed (per MVP discussed below)
      • Desirable to managed as a separate K8s cluster (ie., separate from onap-central instance, of course) and, only for onap-edge use, ie., don't use for other 'workloads' like network apps or 3rd party apps
  • DCAE Enhancements (described above) 
  • Multi Cloud Enhancements (described above)
  • CLAMP Enhancements in onap-central
    • Deploy a separate Control Loop per Edge DCAE/Multi Cloud
    • Associate the right Edge DCAE during Control Loop deployment
  • ONAP Central↔Edge GW (Can Multi Cloud play the role?)
    
    • GW functionality is part of either Central or Edge ONAP instance
    • Communication across ONAP-central and onap-edge instances or ONAP-central instance to Edge Cloud Provider's Systems
    • Interface with ONAP instances through APIs 
    • Internal ONAP instance communication through API or DMaaP
  • ONAP-central OOF enhancements
    
    • Example: deployment of Network Functions (PNF/VNF) to specific edge cloud
      • Leverage Infrastructure Events/Alerts/Faults/Metrics for aggregate objects (Tenant, Cluster etc.) from (onap edge/central) Multi Cloud or DCAE
      • Leverage Application Events/Alerts/Faults/Metrics for aggregate objects (VNFs etc.) from (onap edge/central) DCAE

 Need to align table with the previous text

Edge Requirement	ONAP role	Projects	What can be done in Casablanca?
Support for large number of Edge sites (Hierarchical Scaling)	Support External controllers that take up the load of ONAP (Identify changes i required in ONAP to support external entities that take up the load off of ONAP) Site specific or regional level external controller support Exteral controllers VNF LCM Controllers (that bringup, terminate, heal, migrate, configure, monitor of workloads) Fabric Controller (that control/configure L2/L3 switches at the Edge) CE and PE controllers (that control WAN connectivity of Edge sites) Distribution of ONAP configuration to Edge sites/regional-sites Policy configuration (for Closed loop control within the site) Policies that help in local optimization (VNF Placement as part of scaling) Support for site/region level for analytics and get hold of aggregate data (API and Model support for remote site) API suppot for sites/regions to send relevant topoloyg information Scaling of ONAP CA (AAF CA or ISTIO CA) to issue Intermediate CA certificates to Edge sites/regional-sties	VNF LCM Controller support: SO, APP-C Fabric and WAN controller support: SDN-C DIstribution of Policies : POLICY Regional Site Analytics support: DCAE Topology support: A&AI	High priority: DCAE and Closed loop control Fabric Control Medium Priority: VNF LCM Controller
Performance (Determinism, Low jitter, Low latency, high throughput)	Support for SRIOV-NIC (Ability for ONAP to take care of unique requirements of VIMs) Support for GPUs and FPGAs (Support on-demand programming. Example: Via Openstack Cyborg)	Multi-Cloud plugins, OOF(?) and SO	SRIOV NIC support: MUST FPGA support: Stretch as OS & K8S don't have FPGA support yet. Openstack Rocky is going to support Cyborg
Constrained Edges	Support for containerized workloads (Network functions) using K8S Unified networking among VM and Container workloads	Multi-Cloud, SO, Modeling	POC approved
Edge Application Provisioning	AF (Application Functions - 5G Terminology)) / User App LCM Proxy (MEC terminology) support APIs for AF registration to ONAP APIs to provide NEF Reachability information to external AFs - To allow External AFs to reach 5G NFs to create traffic rules in UPFs on the sites where edge applications being brought up. To register with the 5G NFs (via NEF) to get hold of contextual information. Acting as proxy to reach 5G NFs from AF. ONAP to provide more constraints in selecting the best region to place edge application (Cost, Latency, Bandwidth etc...)	External API, OOF	MUST. Really need to be driven by 5G team.
Security and Regulations	Constraints to support VNF placement based on data-placement regulations (such as GDPR) Ensuring that secrets and password shared with region-site level external controllers are secured well (Using TPM/SGX) ONAP ensuring that site-sepcific controllers/software are not tampered	OOF AAF (Secret Management Service, CA Service) New project for SW tampering detection and taking actions	VNF placement based on data-placement regulations Architecture and design of secret/password management with edge sites and Software tampering.
Site reachability	ONAP to interact with various site services using private IP addresses (via IPSec tunnels?) - Edge sites/Regional-sties to connect to ONAP IPSec Server.	IPSEC Server (New project?), who would ensure that private IP space is not overlapping across sites/regions.	MUST
VNF image management (Reduce operational expenses)	ONAP to do centralized VNF image management Ability for VNF vendors to provide set of images for their VDUs (One per type of remote site - Openstack based, K8S based, AWS based, Azure based. Ability for VNF vendors to provide artifacts specific to remote site specific controls. ONAP to manage images to sites (Pro-active basis - PUSH , on-demand basis - PULL) Some images to some sites can be pushed on pro-active basis (e.g Hyperclouds) : Push images to sites whenever VNF image is uploaded in ONAP. Also, remove image when the image is removed from the ONAP. Some images or some sites may not take too many images (due to persistent memory limitations or cost) and hence support for pulling images. Support for docker hub based image management for K8S based sites and Support for glance API for Openstack based edge sites : As part of instantiation request, letting the remote sites download the image.	Revive image manager project?	Highly preferred
Manageability - Dynamic Site registration	Currently each site is expected to be registered manually. Need for dynamic registration of edge-site, regional site ONAP to API to Edge/Regional-site registration/De-registration In case of regional-sites, edge sites that it controls. Site status (reach-ability status, Site capacity, current VNFs, Total number of VNFs brought up so far etc...)	ESR	Highly preferred

The context information is ….

What is context?

• User related/pref/prof/…
• Location/trajectory/
• ApplicationStatus
• Device details
• Mobility
• Service characteristics/emergency characterization
• Radio characteristics (eg RSRQ)
• Load ->Thruput guidance.
• Transport
• Monetization/enhancing user experience aspects
• Network/topology
• ……

The collection and processing and delivery mechanism:

Group will show how API exposure/observability:

• An ONAP-based data collection, processing and exposure mechanism  (API engine) can support all the API and exposure capabilities specified in various industry forums & standards bodies: e.g.,  ETSI MEC, and similarly support openfog applications
• Real time data from RAN that identifies congestion and other cell specific information can be provided by DCAE in through various event streaming data (e.g.,  PM Event streaming.) A subset of this is included in ETSI/MEC RNIS.
• Using Acumos, Akraino and other facilities provided by Linux Foundation a powerful  ONAP-based exposure structure can be developed  (e.g., predictions based on Machine intelligence, better optimization algorithms etc)
• ONAP based systems can also coordinate information gathering from a variety of sources, to disseminate and make available to applications.
• ONAP-based exposure supports separations of domains and provides a LCM of  network (e.g., slicing) and applications.

Objective:

This group will address the need of an edge automation environment that will provide context information and other autonomics capabilities to 5G services and 3rd party applications in close proximity of end users (~10s ms). Leveraging ONAP elements (such as DCAE) and identifying real time data sources, fundamental context information and ability to collect, disseminate and store and use standardized APIs or else create new ones will be the task of this group in addition to hosting autonomics capabilities apps/microservices/unikernels. The objective will be to determine the needs of ONAP for supporting this environment with some suggested approach to use-case-body for ONAP-ng (R2or3).

Additional requirements:

bring a set of nonfunctional req’s

• Use information / data model to discover and capture hardware resources deployed at the edge and request right type of resource to meet unique application needs. The information model should be vendor agnostic and should be rich enough to cover new architectural directions within the 5G industry
• Infrastructure management should be technology independent; Must support workload deployment options such as VM, Container (e.g. Kubernetes) on VM or bare metal
• Since Edge clouds are confided to much smaller areas, must support a very small foot print to edge location support a metropolitan area with verity of workload deployment
• Edge cloud could be on customer premises – Factory automation
• Must provide efficient network infrastructure that support slicing and QoS configuration options to meet various mobility services need
• Must support policy driven auto recovery / scale up scale down