Link to Project Proposal training materials

Project Name:

  • Proposed name for the project: SO
  • Proposed name for the repository: so

Project description:

The SO provides the highest level of service orchestration in the ONAP architecture.  Currently SO is implemented via BPMN flows that operate on Models distributed from SDC that describe the Services and associated VNFs and other Resource components. Cloud orchestration is currently based on HEAT templates.

 In order to support Use Cases 1 and 2 in such a way to promote re-usability within ONAP, the goal of this project is to enhance ONAP’s overall orchestration capabilities by aligning and integrating its imperative workflows with a TOSCA-based declarative execution environment.

Service Orchestrator in ONAP WIKI


Scope:
  • This project proposes an expansion of ONAP SO to include a declarative topologically-driven approach to orchestration.  Specifically this project proposes to enhance ONAP SO run-time orchestration framework to support orchestration driven from declarative models (TOSCA encoded) as well as integrated or independent BPMN imperative processing extensions that themselves could be TOSCA-aware.   

  This is envisioned as a multi-release project, which in its maturity will:

    • Support TOSCA models in native format as distributed by SDC
    • Perform lifecycle operations based on a declarative TOSCA model, including: 
      • Deployment
      • Undeployment
      • Scale (Out, In)
      • Heal
      • Software Upgrade (various forms)

Alternative 1: The following diagram illustrates one option for the internal processing BPMN and TOSCA Orchestrator sub-components within SO.  In this approach a separate off-the-shelf TOSCA Orchestrator is incorporated into SO, with a top level BPMN layer delegating well defined orchestration activities to this native TOSCA Orchestrator (see diagram below).

Alternative 2: Another potential approach is shown below, where the BPMN itself in effect is a TOSCA Orchestrator component. 

It is envisioned that initial Releases of this project would demonstrate both alternative approaches:

               For Alternative 1 approach: 

    • Demonstrate SO BPMN workflows interacting with an off-the-shelf TOSCA Orchestrator to collectively drive orchestration behavior for at least an instantiation use case
    • Demonstrate rainy day handling
    • Accomplish the above in a way that is demonstrably extensible to support lifecycle operations such as Scale-In and Scale-Out. 

               For Alternative 2 approach:

    • Demonstrate SO BPMN workflows to drive TOSCA-aware orchestration behavior for VoLTE use case, dependency to VF-C.
    • Support lifecycle operations such as Scale-In and Scale-Out. 

Architecture Alignment:

  • How does this project fit into the rest of the ONAP Architecture?



    • What other ONAP projects does this project depend on?

      • SDC (true dependency with SDC SDK)

      • AAI (using API interface)

      • SDN-C (using API interface)

      • APP-C (no existing interface yet)

      • DMaaP (included as part of SDC SDK)

      • MSB (no existing interface yet)

      • VF-C (no existing interface yet)

      • Multi-VIM (no existing interface yet)

How does this align with external standards/specifications?

    • TOSCA Simple Profile in YAML Version 1.0
    • OASIS TOSCA Simple Profile for Network Functions Virtualization (NFV) Version 1.0


  • Are there dependencies with other open source projects?
    • ARIA
    • JBOSS
    • Camunda
    • MariaDB
    • EELF

  • Resources:

Other Information:

  • Vendor Neutral
    • if the proposal is coming from an existing proprietary codebase, have you ensured that all proprietary trademarks, logos, product names, etc., have been removed?
             The current SO seed code has been already scanned and cleanup to remove all proprietary trademarks, logos, etc. except openecomp to be replaced by onap
             Subsequent modification to the existing seed code should continue to follow the same scanning and clean up principles.

  • Meets Board policy (including IPR)

Use the above information to create a key project facts section on your project page

Key Project Facts

Project Name:

  • JIRA project name: Master Service Orchestrator (can be renamed as Service Orchestrator)
  • JIRA project prefix: MSO- (can be renamed as SO-)

Repo name: mso (can be renamed as 'so' ?)
Lifecycle State:
Primary Contact:
Project Lead:
mailing list tag [Should match Jira Project Prefix] 
Committers:
Rob Daugherty, rd472p@att.com, AT&T
John Choma, jc1348@att.com, AT&T

DeWayne Filppi dewayne@gigaspaces.com, GigaSpaces/Cloudify

Tal Liron tal@gigaspaces.com, Gigaspaces/Cloudify

Xin Jin saw.jin@huawei.com

Byung-Woo Jun  byung-woo.jun@ericsson.com Ericsson

Chuanyu Chen chenchuanyu@huawei.com

Seshu Kumar, seshu.kumar.m@huawei.com

Joe Zhang, zhang.zhou1@zte.com.cn, ZTE

Maopeng Zhang   zhang.maopeng1@zte.com.cn, ZTE

Lingli Deng denglingli@chinamobile.com CMCC

Chengli Wang wangchengli@chinamobile.com CMCC

Anbing Zhang zhanganbing@chinamobile.com CMCC

Yan Yang yangyanyj@chinamobile.com CMCC

Heliu Zhong  zhongheliu@boco.com.cn

Yuanwei Yang  yangyuanwei@boco.com.cn

Ethan Lynn ethanlynnl@vmware.com VMware


*Link to TSC approval: 
Link to approval of additional submitters: 

  • No labels

47 Comments

  1. Lingli Deng

    Hi, thanks for the proposal. Sorry for missing the drafting session. I have the following comments:

    1, Would you clarify what you mean by "Use AAI and TOSCA node requirements to perform late service/infrastructure binding"?

    2, Would you clarify what you mean by "MSO Catalog representation of the TOSCA"?

    3, The diagram used for "the ONAP Architecture" is not a complete picture only a subset of it.

  2. DeWayne Filppi

    1.  I still haven't updated the text based on recent revisions and this text was removed.  But the idea was that the model could describe a compute resource (for example) by it's capabilities (e.g. RAM/CPU/Storage) but not specify a specific "flavor" or machine spec.  A search of AAI could be made using those requirements to find a suitable host (for example).
    2. Assuming someone had modeled an orchestration in TOSCA, this just means the orchestrator needs to fetch the model.  I was leaving open whether the representation was just the actual YAML text or some database representation.
    3. We've replaced that image as well.  It was pointed out that it also is very EComp oriented. Will update in a few minutes.

  3. Eric Debeau

    Thanks for the proposal. I have a few questions:

    1. Can you clarify the concepts of Global Service Orchestrator and Domain Service Orchestrator and the rationale to introduce such concepts.
    2. Can you indicate how you ARIA component is intended to be used: as a TOSCA parser or as a seed for the workflow engine ?
    3. I imagine that we will use many other open-source components for the Data-base, the server to implement northbound API...

    1. DeWayne Filppi

      Can't speak authoritatively re: 1.  Yes, ARIA is envisioned as the TOSCA workflow engine for the first release, with the caveat that BPMN would be the upstream and downstream workflow mechanism, per the overall architecture.  Re: 3  I'm hoping that there is no direct binding to any internal service (including model storage), rather just protocols on a message bus.  Time will tell.

  4. Eric Debeau

    Why open-o is considered as the seed code for the project ? We never decided such choice during the discussions.

  5. Brian Freeman

    Byung-Woo, How does the APPC/VF-C functions for Heal and Software upgrade fit into SO role in the architecture ? This seems like overlap did you mean to just include the Instantiate/Terminate/Scale functions. Heal and software upgrade wouldn't be done through SO.

    1. DeWayne Filppi

      I'll chime in; Byung-Woo may have a different opinion.  I think the interpretation was that the app controllers fulfill a role similar to VNFM in the ETSI MANO concept; where as the MSO is roughly equivalent to the NFVO in the same.  On the other hand, if app controllers essentially do everything, then the MSO appears to only function as a message queue between BPMN and controllers.

    2. Brian Freeman

      Steve Smokowski correctly reminded me that I was forgetting HEAT stack updates which SO does today. I was thinking of the internal VNF software upgrades/patches that APPC does (e.g. scripted changes from vendor to patch their application not changing the cloud layer). HEAL is still a potential overlap since we have two distinct software upgrade paths (both should be driven by CMO) - one that is at the cloud layer and one that is inside the vnf application/vfc.

      1. DeWayne Filppi

        OK good.  Definitely healing going on at multiple levels potentially.

  6. Rob Daugherty

    MSO is the seed code base, not open-o GSO.  I've removed the reference to GSO.

  7. HuabingZhao

    I have a couple of questions:

    1. Alternative 1 uses an "off the shelf" orchestrator for TOSCA declarative orchestration. Does it mean that the orchestrator is an external component such as Aria and we will not able to modify the source codes of the "off the shelf" orchestrator? Why I have this question is because that I can't see how Aria can support the VoLTE use case which needs interactions with SVNFM.

    2. Assuming SO delegate a TOSCA service template that consists of multiple VNFs connected by VLs, How the "off the shelf" orchestrator update the instance to A&AI after each VNF/VL is created?

    3.From the project description of APPC, It seems that APPC intends to be a VNFM to handle the Lifecycle Management of VNF, including create, update, configure and delete. So what's the relationship of the "off the shelf" orchestrator" and the APPC? Will it delegate the LCM of VNF to APPC?

    1. DeWayne Filppi

      1. Aria is an open source, open governance project, so it is not fixed.  It is extensible via plugins to handle the VoLTE use case. 
      2. Per TOSCA, orchestrations are governed by lifecycle events.  The final event in an "install" workflow would update A&AI.
      3. Per APPC's charter, the TOSCA orchestrator will delegate device configuration.

      1. HuabingZhao

        So in that case, the "install" workflow is a custom imperative workflow written in python, is there any big difference between an imperative workflow written in python and in BPMN or other languages?

        1. DeWayne Filppi

          No.  Python is not a requirement, implicit or otherwise.  Aria happens to be written in Python, but that is an implementation of a specific engine, not a system requirement (implicit or otherwise).  I imagine a Python-based orchestrator would access the BPMN engine (Camunda or otherwise) using its REST API, if directly invoking workflows.  Hopefully the MSB project provides a message based interface between projects that avoids such bindings however.  That would mean that all bindings between the orchestrator and BPMN (either input or output) would be via the MSB, and Camunda would be a consumer and producer of messages for interested collaborators.

  8. jin xin

    Hi All:

    I suggest that we should also demonstrate Alternative 2 approach in the Release 1.

    The details is following:

    For Alternative 2 approach:

    • Demonstrate SO BPMN workflows to drive TOSCA-aware orchestration behavior for VoLTE use case, dependency to VF-C.
    • Support lifecycle operations such as Scale-In and Scale-Out. 

    I think it's better to achieve ONAP R1 goals on time.Thanks.



    1. DeWayne Filppi

      Unless you're suggesting a change to alternative 2, I see no TOSCA at all in there.  Alternative 1 is the blending of TOSCA and BPMN as you suggest.  I agree with scale in and scale out, although a look at the VF-C proposal and it seems to be automating the entire ETSI MANO stack, leaving little if anything for SO to do.  Need clarification there; perhaps the TSC will provide it.

      1. jin xin

        Thank you for your reply. 

        In my opinion,The alternative 2 should define some  tasks about the tosca model parser and workflow processes in the top-level bpmn workflow..

        For example?the first task parses the service template from SDC. The second task generates the directed graph base on the parser data.The third task executes the directed graph.


  9. maopeng zhang

    I suggest that we should also integrate with the VF-C in the VoLTE case, so the VF-C should be added  as dependancy.

    1. DeWayne Filppi

      Looking at VF-C, I'm not sure what that would look like.  It appears to be doing everything (or almost everything) the SO has proposed to do.  I think the many orchestrators are overlapping in ways that don't make sense.

      1. maopeng zhang

        Orchestrator in my mind is not one layer, and should be multi-layer. Higher orchestration is incharge of E2E service orchestration, cross-domain service, integrated with other domain orchestration, such as SDN orchestrator or NFV domain orchestrator. The domain orchestrators also maybe be mutiple in different clouds.  I think the higher orchestration and the domain orchestration are not overlapping and it is really the SP's requirement.

  10. DeWayne Filppi

    All, the proposal has been submitted as of the Friday deadline.

  11. Catherine Lefevre

    Hi Dewayne,

    I have added the seed repo as well as the major open source dependencies since the current SO source code has been integrated with several open sources.

    Do I need to add the complete list?

    I believe that Aria will be added to the existing SO framework to support TOSCA?


  12. Catherine Lefevre

    I have also added a comment regarding the Vendor Neutral section.

    Suggested tools or anything else equivalent: FOSSology, Blackduck suite, Sonar, CheckMarx

  13. Xiaojun XIE

    Hi, guys,

    I have a question about the definition of end-to-end service in SO. Does it refer to RFS (Resource-Facing Service defined in TMForum) or NS (Network Service defined in ETSI NFV)?

    If it refers to RFS, TOSCA may not be commonly used. Task-Driven Workflow may be more suitable for this, especially for the services supporting multiple customers.

    If it refers to NS, SO would overlay with VF-C, because VF-C also implements the TOSCA parsing for NS. At the same time, how do ONAP implement RFS orchestration? Or, is it not in the scope of ONAP?

  14. jin xin

    Hi All:

    I have updated the SO project proposal to demonstrate both alternative approaches.Please review and correct me.

    Thank you.

  15. Catherine Lefevre

    I also would like to consider the following JIRA items as part of this proposal if these are not solved earlier:

    https://jira.onap.org/browse/MSO-4

    https://jira.onap.org/browse/MSO-2

    MSO-6 - Getting issue details... STATUS

    MSO-7 - Getting issue details... STATUS

    MSO-10 - Getting issue details... STATUS if the epic is approved (COMMON-10)

    MSO-1 - Getting issue details... STATUS

    MSO-9 - Getting issue details... STATUS

    1. DeWayne Filppi

      Sorry for the long silence; have been on vacation.  I'm reluctant to endorse putting specific bug reports for the seed code into this document.  It seems to me that requirements level statements would be more appropriate (that perhaps refer to specific Jiras); and then those requirements be evaluated by the group.

  16. Catherine Lefevre

    Good morning DeWayne, these are indeed user stories to be evaluated as part of the new SO project or can be kept as part of SO backlog or simply to be closed if no interest.

    1. DeWayne Filppi

      OK, my vote would be:

      MSO-4: no

      MSO-2: no

      MSO-10: yes, but in some far flung release.

      MSO-1: not sure what these are, and provided link doesn't work for me.

      MSO-9: sure, but pretty low priority.


      1. Catherine Lefevre

        Hi Dewayne, I understand. Nevertheless the SO PTL will need to deal with the JIRA ticket.

        Here is the query to identify any SO tickets:

        Getting issues...


  17. Hui Deng

    In order to have unified modeling in the ONAP, it would better that SO projects depends on modeling project for example, service template, service component modeling,  workflow modeling, Parsers et al.

    1. DeWayne Filppi

      Hui, I had that in there at one time but it was removed early on.   I don't object to the explicit reference.  I think the argument against was that the actual software dependency was with the SDC.  How the SDC represents the orchestration template (and related workflows) will determine much.

  18. DeWayne Filppi

    JBoss was included as a requirement due to the seed code.  What are group opinions regarding JBoss (or Wildfly I guess), JEE in general (I'm not a fan), or at least eliminating JBoss as a project requirement?

    1. Catherine Lefevre

      If you decide to remove JBoss then what will be your alternative?

      1. DeWayne Filppi

          I think I was off base here.  I was viewing the JBOSS reference as a system requirement, not simply a fact of life in the seed code.  At this point, I would not remove it.  If starting from scratch I'd use something lighter like Guice or Spring.  I worked in JBoss (and other JEE platforms) for years, and grew to despise them.

  19. maopeng zhang

    what's the exact scope of the "off-the-shelf TOSCA Orchestrator" ? 

    1. DeWayne Filppi

      Orchestrating native TOSCA templates.  Essentially, the same function as the generic MSO workflows.

      1. maopeng zhang

        Could we called it "tosca workflow engine", not orchestrator? Orchestrator concept may cause some confusion.

        If it is an worklflow engine, what's the tosca grammars that the workflow engine depends on?  How does it integrate with the SO in the software layer? could you provide the workflow api specificaiton for community?  Thanks.

  20. Seshu Kumar Mudiganti

    DeWayne,

    As discussed in the F2F meeting, kindly find the points for clarification as they are critical for realization of the proposed design:

    1. ARIA TOSCA if could provide APIs for parser and engine separately.
    2. how to access them through REST.
    3. How will the BPMN interchange data between the TOSCA orchestration engine and  A&AI / Controllers.
    4. Response handling for both sunny and rainy day cases.
    5. ...

    1. DeWayne Filppi

      1. Aria is a library.  Parsing can be called separately from workflow execution.
      2. Aria is a library.  A REST API will be created (I've begun it already) to interface with Java.  In Python, such things are trivial.
      3. The highest level MSO workflow will somehow recognize the user intent, and forward requests to Aria via REST API.  Details TBD.
      4. Need precise definition of this before commenting...

      1. DeWayne Filppi

        Additional response for item 3.  At present, my intent is to model A&AI and Controllers as entities in the TOSCA model, access via their APIs via plugins (Python), because that's probably the fastest way to success.

  21. DeWayne Filppi

    Everyone, please note that the TSC has indicated that only 2 committers can be listed for each company.  Please edit the list as appropriate.

  22. Manoj Nair

    Couple of queries 

    1) If I understand correctly, now heat template is distributed from SDC to SO. Also VF-C expects a TOSCA based model. In this case, SDC is expected to distribute two types of artifacts? One for SO for instantiating VFs and other TOSCA templates for VF-C to consume? 

    2) In Open-O there used to be a mechanism to provide a workflow link as part of the Plan as described here - https://wiki.open-o.org/display/IN/Nanjing+Workshop+August+9-11,+2016#NanjingWorkshopAugust9-11,2016-CommonTOSCA. Will this capability be supported for enabling imperative workflows. Also in this case will SDC be capable of distributing the workflow file and SO be capable of handling such workflow artifacts?

    3) Is there validation done at SO for the heat template received from the SDC? 

    4) How is the NS state reconciled currently? Is there any polling mechanism in place to synchronize the state of NS? Or this is purely based on the DCAE Control loop capability? 

    1. DeWayne Filppi

      1) I'm working the TOSCA path and am hoping the SDC will provide CSARs.  My understanding that the SO orchestrates services, not VNFs.  As such, the SO will want TOSCA models that deal with VNFs at as high a level as possible (location, interconnection, etc..).  The VNFs themselves may consist of several compute instances and have their own independent lifecycle management (per CLAMP), and need their own TOSCA models.  I don't think we want a service designer role to be penetrating the VNF abstraction at all; they should be black boxes (i.e. a vIMS by company X version Y with certain capabilities).   That's just my understanding though.  Much of this can be derived from the desired end user experience, be it service designer or VNF vendor.

      2) No.  In TOSCA the model implies the workflow, at least ideally.  There will be a generic workflow that ultimately hands off to the TOSCA orchestrator, and will gather information needed by the orchestration (e.g. Homing/OF, possibly some A&AI stuff, and more (TBD)).  We've had discussions about having southbound BPMN flows tied to TOSCA operations, but that's not needed (or happening) in R1.

      3) Not sure about the BPMN side, but the TOSCA side doesn't care about (or want) HEAT, just TOSCA/CSAR, so no need to dumb down TOSCA.

      4) NS?

      1. Manoj Nair

        Thanks for the clarifications DeWayne. Couple of follow up questions 

        #1 : Can you please share the wiki link for this work. 

        #2 : So does it mean that SO will have a Cloudify engine capable of receiving the TOSCA models from Catalog and processing it ? In one of the SO calls there were discussions around having multiple TOSCA parsers in SO (which is not available currently) - one for imperative workflows and other for declarative TOSCA models. Is this plan holds good for R1 ? 

        #3 : I believe NS state (Active, InActive, Maintenance etc) is maintained in A&AI and my understanding is that SO in collaboration with DCAE is responsible for reconciling the state of NS based on the VNF events received. Is this understanding correct. 




  23. Sunil Kumar

    Hi,

      

    I am looking into CreateE2EServiceInstance workflow . I am able to locate Api ( SO -> VFC ) from it . But as per the workflow , Api url is different from the mentioned SO api list in
    wiki.

    Api url found from above workflow -> /vfc/rest/v1/vfcadapter/.. ( during Create NS )
    SO Api as per wiki /api/nslcm/v1/ns

    How both are being mapped ??

  24. Hao Kuang

    Hi,

    I am just wondering where can I put any bug or question about so?