Overview

MEF LSO defines a set of specifications and reference points aimed at providing end to end service orchestration across multiple network domains using standardized APIs. One of the reference points in this set is interlude which focuses on providing control related management interactions between service provider and partner (link). The other inter-provider reference point in LSO is SONATA which mainly focuses on the OSS/BSS level business interactions. As ONAP is not specifically on the SONATA level of interactions, the rest of the document focuses mainly on the interlude reference point in MEF and similar specifications in other standard organizations. While interlude is one of the reference point and specification which addresses inter-provider interaction, it is worthwhile to look at the broader scope considering typical operational, business use cases and aspects impacting such interactions in ONAP External API specifically and ONAP in general. MEF Interlude scope is covered in detail in a separate section.

5GExchange (link) as part of 5GPPP is one of the relevant project worth referring which focuses on "cross-domain orchestration of services over multiple administrations or over multi-domain single administrations" (link). 5GEx project defines a multi-domain logical interworking architecture which covers multi-operator interaction and multi-domain interaction within the same operator. As part of the 5GEx project a detailed study is being conducted around inter-domain and inter-provider interactions and results are published here. The 5GEx proposed system consists of multi-domain orchestrator (MdO), domain orchestrators and interactions between MdO (marked as #2 in the diagram below), the interaction between MdO and Domain orchestrators (marked as #3 in the diagram), the interaction between customer and MdO (marked as #1), interaction between Domain Orchestrators and controllers (marked as #5) and interaction between domain orchestrators (marked as #4). Each of these interactions is identified by different types of interfaces. There is also a classification based on business level interactions, management/orchestration level interaction, control level interactions, and data level interaction. Out of these #2 is the one which close matches the MEF interlude reference point, but the scope is slightly different because in 5GEx project business, management/orchestration related interactions are expected to be handled by the same interface (i.e #2). While #3 (between MdO and DO) is quite relevant in the case of External API, this may be an item for future study as domain orchestration concept is currently under discussion in ONAP Tiger team as of September 2018. For the sake of this document, MdO is functionally mapped to the External API component in ONAP as it is providing an end to end service management capability.

Following diagram captures the interface mapping to standards as defined by 5GEx in their functional model found here

Looking at the picture above, it can be observed that 5GEx mostly follows the ETSI MANO specific interfaces for interactions across MdOs or between MdO and DO. But the picture also includes some additional scope as listed below

• Topology distribution across domains for exchanging network topology details that may be used by MLPCE.
• Multi-domain path computation engine interaction across domains for exchanging path specific data
• SLA Management interaction across domains for exchanging business agreement.
• Service Catalogue interactions across domains for exchanging service specifications

Summarizing the scope in5GExproject, its key focus is in virtualized infrastructure with ETSI MANO building blocks with additional scope for exchanging the network topology, network path, business agreement and service specification across different domains. For ONAP Ext-API this may not be quite relevant as it functionsata layer above the NFVO. However, if External API scope is expanded to have cross-layer interaction, i.e MdO of one operator domain interacting with DO of another operator domain 5GEx specific interfaces may be relevant. But what can be learned from the 5GEx project is the concept of SLA Management, Catalogue exchange, Security mechanisms and approaches for supporting use cases such as Slicing.

The topic of inter-provider/inter-domain interaction is being discussed in TMF ODA, ETSI ZSM, but these specifications are still in the early stages of development and may not be relevant in the near future of ONAP development.

Another specification worth referring is ETSI IFA 028 v3.1.1 - wherein MANO architectural options to support multiple administrative domains is being discussed. This specification introduces two concepts - MLPOC (Multiple Logical Point of Contact) andSLPOC (Single Logical Point of Contact) with varying degrees of cross-layer interaction and information abstraction across domains. This specification also defines an Or-Or interface across NFVOs in different administrative domains. Assescribed in the case of 5GEx, ETSI Or-Or level interaction may not be

Subsequent sections in this page cover a comparison of different SDO/OSSP activities around inter-provider APIs.

Multi-domain Interaction

As defined in 5GEx project multi-domain can be multiple network operators or it can be multiple subdomains within a single operator. The scope of interaction might be slightly different within single operator domains and across multiple operators because the latter will be governed by SLAs with strict policies and predefined trust/contract between the two operators. So security and trust are some of the key criteria for interaction across multiple parties. All interaction should be governed, policy controlled based on the trust agreement. Within the same operator domain, there can be multiple administrative domains which can be governed by SLO/OLOs and trust agreement as in the case of inter-provider interaction. But there can also be a model of distributed deployment which may not fit into the purview of multi-domain interaction. For example, geo-redundancy and HA deployments may not be classified as multi-domain interaction, but governed mostly by policies defined between two software components and interaction over internal APIs.

Federation and Delegation

Similar to multi-domain interaction, federation and delegation are two terminologies used for interaction between two logically separated endpoints. While there is no standard terminology defined at the ONAP level, we can assume federation to be the east-west interaction between systems/components at same logical domains- for example between Orchestrators in two administrative domains, or controller in two administrative domains. The delegation terminology can be associated with the interactions between systems/components at different logical domains- for example, an end to end orchestrator interacting with a domain orchestrator. Federation and Delegation can be classified with reference to the diagram above from 5GEx. Interactions marked 2 can be classified in federation and interaction marked 3 and 5 can be classified as delegation. Here the federation is across domains of different operators, whereas delegation is between the same operator domains. Another differentiator is that federation is between logical domains with similar scope whereas delegation is between logical domains with a different scope. The logical separation can be based on the technology abstraction, geographic abstraction or deployment model. One example of the federation model is the interactions in the CCVPN use case an example of the delegation model is interaction possible between the central site and edge site orchestrators in an edge automation use case.

In the ETSIIFA028there are two models of inter-administrative domain interactions -SLPOCand MLPOC. In SLPOC there is a single interaction point between two administrative domains whereas in MLPOC there are multiple interaction points between administrative domains. In simple terms, it is possible based on ETSI MLPOC model for NFVO in one administrative domain to interact with VNFM or VIM in another administrative domain over the ETSI interfaces. Since External API functions at a layer above NFVO, the current scope of interaction is limited to the federation model described above -i.einteraction between External API in two operator domains.

The delegation model support in External API requires further discussion based on specific use cases. There is also discussion around Recursive Orchestration, Orchestration Hierarchy and Domain Orchestration. The delegation model can be considered within the scope of External API once some concrete decision is made by the Architecture team.

Business Agreement

MEF Interlude does not have a specific scope for managing the Business Agreement between SP and Partner, however, the interaction between the parties might be governed and controlled based on the predefined business agreement and associated policy rules, security mechanism. 5GEx document on Business and Economic Layer (link) elaborates this aspect in detail but limits the focus on the SLA between parties. Some interesting aspects to be considered for Interlude are as follows

• Roles of the parties (SP and Partner) which will determine the mode of communication, specific controls required at either end, Policies to be enforced, the direction of communication (Some examples given for the 5G case are - Infrastructure Service Provider, Network Service Provider, Communication Service Provider, Over the top service provider, Exchange point service provider . The roles defined by 5GEx are mostly inspired by those defined by 3GPP 28.801.
• Centralized vs Distributed Interaction: The interaction between parties can be centralized i.e coordinated by an aggregator provider acting as an exchange point between parties or it can be one to one. The aggregator model is better as it will avoid need for multiple business agreements while ensuring centralized enforcement SLA and policies
• Coordination model: Consists of two phases - publishing phase where information is exchanged between parties on the offered services and service composition phase when the actual customer request for a service is forwarded from one party to other. In the case of Ext-API this will be translated to a query on the other parties Service Catalog and initiation of service Configuration/Control Request
• Agreement Push vs Pull Model: In Push model the Business Agreement and policies are predetermined/agreed before any interaction between parties over the inter-provider API where as in pull model business agreement is dynamically decided based on the customer request and required SLA, monitoring levels.

The 5GEx project defines Business Agreement in terms of SLA and the document referred at the beginning of this section also gives a template for defining SLA. For ONAP Ext-API this may not be useful as it currently does not have any referenceable entities for defining policies for interaction between parties, which is quite relevant at the interlude level.

Other relevant SDO References for adapting Business Agreement are as follows

• TMF B2B2X Partnering Step by Step Guide (link) : Lifecycle model for B2B partner management, templates for B2B agreement mostly focused on the business layer, but the operating agreement may be relevant for defining the interlude policies
• Recent ONS presentation on BlockChain based inter-operator agreement implemented based on HyperLedger (link)

For External-API project a new set of APIs needs to be defined for the Business Application layer to push the policies for interacting with the partner. In the absence of this API, it may be assumed that Ext-API will consult the Policy Engine in ONAP for determining the control mechanisms that need to be established before interacting with the partner over the inter provider API.

Cross-layer Interaction

IN MEF LSO , the interlude reference point is between SOF in Service Provider domain and Partner Domain

Security

TBD

Standard APIs

Information/Data Model

There are multiple models found to be relevant for inter-provider API

• MCM aligned E-Line Service Model defined in MEF Interlude Contribution - Access E-Line Service Control Classes - 5th Draft
• Work in progress MEF Services Common Model (link) - Initial Proposal for Work Item
• CFS/RFS being referenced by the TMF 641 (based on SID) (link) - Currently followed by CCVPN use case

The choice of a specific model will depend on the decision of EUAG, TOSCA Task Force in ONAP. From Ext-API point of view it is expected to leverage the CFS/RFS model being referred by the TMF 641 API . In future as Interlude specific model in MEF and MEF Services Common Model matures appropriate mapping can be incorporated to accommodate specific service characteristics to the TMF APIs. In MEF LSO there is also NRM model being used for the Presto interface (derived from ONF). The NRM model is assumed to be out of scope for Ext-API unless there is a cross-layer interaction between SOF in SP domain and ICM in Partner domain is required.

Open Questions

• Do we need to consider intraoperator multi-administrative domain interaction – i.e communication across different instances of domain orchestrators (ONAP or non-ONAP) belonging to the same operator?
• Do we need to limit the scope of Interlude to Service Control, Activation, and Configuration or include Service Order Management?
• Do we need to come up with ONAP specific terminology? Different SDOs follow different terminology e.g. operator interoperability, domain interoperability, administrative system interoperability etc. ?
• Catalogue and Inventory Management – Strategy for 1) onboarding the catalog with service specification across interdomain boundaries 2) Reconciliation and aggregation of inventory at each domain – Pull vs Push model
• Service Model Impact: Service hierarchy in the Service model – i.e Composite or Nested Service, Constituent Service – How the service model is decomposed and distributed to operator and partner domains? Any pattern to follow? Or based on request attributes?
• Cross-layer access requirement for multi-domain interaction for example Orchestration layer of SP need to interact with VIM of Partner for resource instantiation – Is this model valid (MLPOC as per ETSI IFA028 )
• Federation vs Delegation
• Federation Actors and Roles: What type of provider roles we should consider – (NGMN Actor Roles ? 5GEx Actor Roles etc – Infrastructure provider, Connectivity SP, Partner SP, Master/Slave), Do we also need to consider different layers of partners – infrastructure, connectivity etc.
• Use Cases: What use cases we should consider for the interlude specification? Generic Operational use cases (Service activation, query etc)  or Specific Business use case ? (NaaS, NFaaS, Access E-Line etc) – Short term and Long Term Target?
• Consideration for interaction between ONAP and non-ONAP (Legacy) Management system across operator domains
• Need for including the Business layer interactions within the scope of interlude (for example dependency on Service Policy)
• Strategy for closed-loop control (Assurance) – Who will manage? Partner managed or SP managed
• Resiliency requirements for inter-operator management connectivity – Failover mechanisms
• Do we need to consider interdependency of Interlude and Legato/Sonata interface? (for example contract) 

Interlude Scope as per MEF55

Referring to MEF55 document the Interlude reference point is used by Service Orchestration Functionality to request initiation of technical operations or dynamic control behavior associated with a Service with a partner network domain. The dynamic control (Service Control Orchestration) behavior is elaborated in section 8.2.3 of MEF55 as

• Scheduling, assigning and coordinating service control related activities;
• Undertaking necessary tracking of the execution process of service control requests;
• Adding additional information to an existing service control request under execution;
• Modifying information in an existing service control request under execution;
• Modifying the service control request status, and indicating completion of a service control request;
• Canceling a service control request; - Monitoring the jeopardy status of service control requests, and escalating service control requests as necessary;
• Instantiating, when appropriate, an event for the billing system to capture the policy-constrained change.

MEF55 also differentiates Order Fulfillment Orchestration, Service Configuration and Activation, Service Control Orchestration. While Order Fulfillment Orchestration deals with establishing or modifying a service through the ordering process, Service Control permits the service to be dynamically changed within specific bounds described in policies that are established at the time of ordering. After a service is provisioned and established, LSO may enable Service Control to Customers/parties, such as the ability to modify attributes subject to schedule policies and service constraint policies with for example specified ranges of valid values. Service Control relates to capabilities such as turning on or off connections, throttling bandwidth or other QoS characteristics, etc.

So considering the scope in MEF55, Interlude reference point is primarily used for Service Control Orchestration. Here Service Control Orchestration is considered to be an activity enabled after service is being provisioned. So service order management is not defined in the scope of MEF Interlude but expected to be carried over the SONATA interface (Product Order Management) and Legato (Service Order Management, Service Catalog Management).

MEF Interlude Interactions

The interactions factored in the MEF Interlude Reference point are as follows

• Service Provider controls aspects of the Service within the Partner domain (on behalf of the Customer) by requesting changes to dynamic parameters as permitted by service policies.
• Service Provider queries the operational state of the Service.
• Service Provider requests change to the administrative state of a service or service component (e.g. Service Interface)
• Service Provider requests update to defaulted service parameters which are allowed to be customized (policy-controlled)
• Service Provider requests the creation of connectivity between two Service Interfaces as permitted by established business arrangement.
• Service Provider provider queries the Partner's Service Inventory for services provided by the Partner to the Service Provider.
• Service Provider receives Service specific event notifications from the Partner.
• Service Provider receives Service specific performance information from the Partner.
• Service Provider requests test initiation and receive test results from the Partner.

The green one's are interactions currently supported in External API across SP-Partner in the CCVPN use case. In addition to the above-listed capabilities Ext-API also supports following interactions

• Service Provider Queries the Service Catalogue for the offered Services by Partner
• Service Provider places Service Order for a Service offered by Partner

The last two interactions are not specifically scoped as part of Service Control Orchestration in MEF55, but being supported for CCVPN use case in the absence of a SONATA interface at the Business application layer. Additionally, while Interlude scope is limited to Service Configuration and Control, the API used for the interaction across the SP-Partner can be TMF 641 - Service Order Management as this API provides an option to include multiple Service request in a single API call, unlike in TMF 640 wherein each Service request need to be split as separate API invocations. The usage of TMF 641 API also allows to include additional capabilities in future for supporting Service Activation

As per the guidelines in the document here, interlude should be used for

• Service configuration
• Service activation, de-activation, modification, deletion
• Service testing
• Service assurance

Out of this service configuration scope should include

• Setting Schedule for Configuration
• Configuration of User Interfaces
• Configuration of Connection
• Configuration of Connection End Points
• Configuration of Redundancy

Interlude Related Work Items in MEF

Following documents give details of the current work items being developed as part of MEF Interlude.

Note: To be verified by Mehmet Toy

• Interlude Guidelines: Gives the high-level scope of Interlude
• Access E-Line CIR Change Process Flow: Highlights the interactions between SP and Partner for CIR change for a provisioned e-line service over the interlude reference point leveraging the service configuration.
• Access E-Line Service Information Model: Based on MEF MCM (Mef Core Model), with extensions to be used for elastic service levels and service schedule
• Business Requirements and Use Cases for Access E-Line Service Control
• MEF LSO Interlude Activation API

Standardization Work

Opensource (including ONAP) Work

Current capabilities in ONAP External API with respect to Interlude Reference point is elaborated in the presentation by Adrian here. To summarize the capabilities, currently Ext-API supports following types of interactions between Service Provider and Partner (specifically in the context of CCVPN use case, but can be applied in the case of other similar use cases as well

Note: To be verified by Adrian OSullivan

• End to End Service is Designed in SDC on SP and Partner side and independently distributed to the respective runtime environments
• SP or Partner details are prepopulated at respective inventories (A&AI) as customer of the service
• A service order is placed on the SP side ONAP instance using the Ext-API Service Order API (TMF 641) - using the UUI portal
• External API decomposes the service order to individual service order items and passes the service order items as a request for service creation to SO
• SO check the resource requirements - on encountering the SPPartner resource, a new Service Order request is constructed with information available in the SPPartner identifier in the service instantiation request
• SO places an order directly on the Ext-API of the partner and receives a Service Instance identifier in response
• SO keeps polling the status of Service instance until service is created/failed on the partner side
• SO update the SPPartner instance in inventory with the service instance details

Following are some of the shortcomings in the current capabilities

• Services need to be independently designed and distributed on either SP and Partner side
• SP and Partner need to be aware of the Service Specification id to be used while placing the order from either side
• SO from SP side directly invokes Ext-API on the partner side, not following the separation of concerns - i.e ideally the Service Order management should be limited at the Ext-API layer and Ext-API should act like a gateway between SP and Partner those are governed by contracts
• There is no contract or policy based interaction control between SP and Partner
• The scope of interaction is limited (Placing an Order and checking the status of the Order)

While the scope of the current study is limited to short-term Interlude specific capabilities to be supported in Ext-API project and corresponding dependencies, there is also a need to set the long-term scope. Ideally, this requirement should come from the EUAG or the contributors from operator organization. Following are some of the areas identified during the discussions in the Ext-API project. These areas need to be further studied in collaboration with PTLs and EUAG. Note that all these areas are focused on interaction between a Service Provider and Partner(s)

Business Cases

• NFaaS (Network Function as a Service)
• NaaS /SD-WAN
• SlicingaaS
• MVNO Scenario
• Connectivity as a Service 
• NFVIaaS
• Application as a Service (For Edge scenarios) 
  

Operational Use Cases

• Dynamic Service Control
• Query Service State
• Update Service
• Request Connectivity Service (across two Service interfaces)
• Query Service Inventory
• Receive Service Notification
• Receive Service Performance Update
• Initiate Service Test 

Interlude specific considerations

• Layers of interaction, Separation of concerns: Whether to limit the interaction between SP and Partner through a specific layer - for example, Ext-API or support interactions across layer - i.e to support interaction between Ext-API in the Service Provider domain and SO or Controller in the Partner domain. What kind of APIs are used for such interactions?
• Security: Securing the interaction between Service Provider and Partner based on a predefined contract and policy
• Business contract - Policy: Business contract agreed between Service Provider and Partner that will govern the interactions
• SLA Management: The components and method for monitoring and managing SLA including the closed control loop across service provider and partner domains
• Inventory/State Management, Consistency Check, Identity mapping: How inventory across two domains are represented in respective domains, how it is reconciled and how the consistency is checked.
• Interface/API – Reference Specification: What standard APIs to be supported over the Interlude reference point
• Licenses: How licenses are controlled across SP and Partner domains, What are the impacts and associated fulfillment implications due to resource licenses.
• Modelling impact: Interlude impact on Modelling work - especially the representation of Partner resources, reachability information, configuration.
• Integration: Requirements for the integration between SP and Partner components via standard APIs, channels for event and performance data collection, reconciliation
• Interoperability: Interaction between SP and Partner, with either one of them having an ONAP instance
• ONAP Role: Assigning roles - such as master, slave to ONAP considering deployments at SP and Partner domains -specifically for coordination of end to end service orchestration and data collection.

Best Practices and Standard Alignment

• ETSI GR NFV-IFA 028 V3.1.1 (2018-01)
• ETSI ZSM
• MEF LSO Interlude (link)
• Contributions by Mehmet and Jack
• TMF ODA
• ONAP CCVPN Use Case
• 5GPPP 5G-Ex Project

Requirements

Study how we can control modification requested at Interlude level – they are at least 2 subtopics there:

• • Identify where the information could be managed within ONAP (in SDC ? have attribute change value and state change authorized at SOF-SOF interaction level ?)
  • Explore if External API can be used by internal ONAP Components as a proxy to talk to external Partner ( e.g. ONAP 1 SO call ONAP 1 ExtAPI, then ONAP 1 ExtAPI call ONAP 2 ExtAPI ) i.e. ONAP internal components can communicate securely internally, External API act as proxy to forward request ( e.g. Service Order ) to Partner ( maybe a Partners ONAP External API Framework ). 
  • Define the API (resource model) that allow the system to request ‘authorized’ service modification (Policy API ?)

Dependency on other projects