Background

Multiple TOSCA Parsers have been used in ONAP across different projects.

List some projects as an example::

Project

Implementation language

 Parser used 

SO

Java

SDC Tosca Parser

VF-C

Python

NFV Tosca Parser

UUI

Java

SDC Tosca Parser

VNFSDK

Java/Python

NFV Tosca Parser

SDC Tosca Parser

Policy

Java

Internal Tosca Parser (Policy Type, Data Type, Policy only)

A&AIJavaSDC Tosca Parser
SDNCJavaSDC Tocsa Parser
VIDJavaSDC Tosca Parser
SDCJavaSDC Tosca Parser
CLAMPJavaSDC Tosca Parser
APPCJavaSDC Tosca Parser
CDSKotlinCDS/Custom Tosca Parser
.........

Here only list part of the projects, not all the parser consumer project.

Now, all projects consume the parser interfaces through the lib library and project chose to use different parser based on the implementation code language

In summary, currently , the same parser function has different implementations, but parser as a common service should be unified and provided as a service not the lib library. 


Advantages of Parser as a Microservice

  1. Microservice can be decoupled from applications 

    Effect: The application does not need to know the implementation details of the Passer, only need to pay attention to the results of the analysis.

    Current status: As a Library, an application can use parser only after understanding the internal implementation of Parser.

  2.  Multi-language support

    Effect: Parser provides services through restful, language-independent, and can support any language such as python and java.

    Current Status: There are different Parser tools for different languages, and cross-language support is difficult.


  3. Easy to upgrade 

    Effect: Parser upgrades independently, only keep the external restful interface unchanged. 

    Current Status: The current Parser as Library, the application sees the internal implementation details of Parser, if Lib upgrade, it will affect the application.

  4.  Easy to deploy, support horizontal expansion

     Current Status: With the application deployment, it is not easy to support horizontal expansion, and there is basically no flexibility.

     Effect: After Parser is as a service, it conforms to Cloud Native idea, can utilize platform advantages, flexible deployment, and horizontal expansion.

 Disadvantages of Parser as a microservice

  1.  Performance: Compared with the Lib usage, when Parser provides services through restful, the performance will be reduced 


  2. Delay: Compared with the Lib usage, when Parser provides services through restful, the delay will increase


    Note： Temporarily there is no data support for  the above performance and delay 

Implementation recommendations

  1.  For external interface

     After the tosca parser microservice is implemented, the Lib usage will continue to be provided, and the application can select to use  the microservice or the Library according to the specific requirements scenario


  2.  For Parser implementation


    1. Requirement Analysis ( which is tracked in SDC Template Type Analysis): collecting requirements, unifying model input and output, simultaneously implementing model output conversion from object to Json (this is the key point)

    2. API Incapsulation: Providing unified restful interface to parser client. 

      For parser API abstraction，there are two options considered

      OptionDescriptionAdvantageDisadvantage
      1Parser only provides one simple API with the csar file as input and return complex Object which contains all content in model API may looks like: ModelObject parseCsar( csarfile)The parser implementation is relatively simple， just focus on how to build the complex response

      Client need to extract the information that they need from complex response locally，this will bring more complexity on client side

      2

      Parser provides the extension APIs which are more granular and return the specific content in model to client according to different API invocation

      These APIs may look like:

      Metadata getServiceMetadata(InputParameter)

      Inputs getInputs(InputParameter)

      ......

      		Don’t need to maintain two library code and can solve cross-language issue.Parser should keep the context information and increase the difficulty of parser implementation




Supplement

In order to analysis the tosca parser requirements, we do some analysis on the tosca types which currently included and used in SDC and the analysis can be found here: SDC Template Type Analysis

Known TOSCA Parsers

See the list on this TOSCA page

Discussion Minutes (20190102)

  • Agreement: SDC Type fix ABC for Dublin
  • Target: Parser as a service
  • Problem: Rest API definitions for Parser us
  • Next Steps:
    • Parser API analysis for run time consumers;
    • Brain Storming in Paris for Parser API design.






  • No labels

4 Comments

  1. Anatoly Katzman

    Yan Yang Lingli Deng Hui Deng Stephen Terrill Chris Lauwers


    I totally agree that maintaining too many different parsers is expensive.
    I also understand the wish to have a parser as a service.

    The only question here is HOW to do this.

    Not any piece of software which is working as a local library can be externalized into a service.

    One of the most prominent conditions is that the response should have a plain structure. If the client side needs a local parser to understand the responses of the remote parser, then what's the point of the parsing service? Our local TOSCA parsers return VERY complicated object graphs, with hundreds of properties and references, and the depth of these graphs in not always predictable. It is like dumping a graph database.

    So we cannot just create a naive API based on a simple 'parseTheDocument' request.

    Another way would be to keep the resulting object graph on the server side, and then extend the service API with requests to navigate this graph.
    Not a good thing either:
    - stateful services are pain
    - parsing a document would take dozens of client-service roundtrips, hence HUGE performance impact
    - the client-side logic becomes complicated, and again, needs to be implemented in multiple programming languages.

    Maybe we can brainstorm this problem on the upcoming F2F in Paris.

  2. Kevin McDonnell

    Atul Purohit from Vodafone made a proposal last year on a Centralized TOSCA "Parsing Service". See Modeling sub-committee meetings

  3. Chris Lauwers

    I'm not at all in favor of adding a "TOSCA parsing service" to the architecture. It would only perpetuate what I believe is a sub-optimal ONAP architecture. In my opinion, the ONLY functional area where TOSCA parsing should be required is for onboarding externally-generated CSAR files into SDC. External CSARs should be parsed directly into a standard ONAP object model that is understood by all ONAP components, and all communication between components should be done in terms of these standard objects, not using TOSCA files. 

    Of course this approach requires "internal standardization" of an object model across all components, which I know is challenging. But the counter argument is that we would have this standardization problem anyway if we introduced a TOSCA Parsing Service. Such a service would have to parse TOSCA files into something, presumably into an object graph as Anatoly Katzmansuggested. For this graph to be useful, we would have to standardize everything that's in the graph, or we would have to standardize a graph navigation API. Either way, we have an "internal standardization" problem. If we're going to address and solve an internal standardization problem, then why not focus on the area with the biggest potential value, which is standardization of an internal object model across all components.

  4. Tal Liron

    Thank you for the analysis, but it is 1) based on overly limited assumptions, and 2) does not examine the best option, which is running a stateless TOSCA parser as a tool.

    First, the idea of wrapping a TOSCA parser in a "microservice" assumes that the TOSCA parser is a heavy piece of software that is stateful. We've inherited this assumption from the architecture of ARIA, which indeed was more than just a TOSCA parser. ARIA was actually a complete mini-orchestrator, which had to manage its internal state. Furthermore, such a stateful component is not really a "microservice", but a full-blown service. (A microservice architecture assumes stateless components.) But this assumption is unnecessary: a pure TOSCA parser does not have to have any state.

    If we assume that our TOSCA parser is stateless then there is no need to even consider the very problematic "microservice" scenario. Instead, I propose we consider "TOSCA parser as a tool". A small, self-contained TOSCA parser could be executed on demand as a child process from any component that needs it. It accepts input and emits output that can be captured and used by the component. It doesn't matter which programming language this tool is written in, because its input is TOSCA and its output could be a common, consumable syntax (YAML, JSON, etc.).

    One way to get there is to simply isolate any stateless TOSCA parser library as a (stateless) self-contained command-line tool. So, we can take the SDC parser and turn it into an independent tool.

    But I humbly suggest that you consider the Puccini TOSCA parser. It is designed from the ground-up exactly for such scenarios. It is a self-contained executable (installation = copying the file, that's it), very small and very fast, that accepts standard TOSCA (YAML or CSAR, versions 1.2 and 1.1) and compiles it into an easily digestible and straightforward YAML or JSON format. Beyond that, it is very easy to customize the output using embedded JavaScript scriptlets, which is how Puccini handles its orchestration integrations. (Although this shouldn't be necessary for ONAP components, because once they consume the output they can do their own manipulation.) Puccini is mainly developed by me, a member of the ONAP community, and I will be happy to adapt it further to suit ONAP requirements.

    Further Puccini features that are useful for us are TOSCA 1.2 support. This includes the substitution mapping extensions for properties and interfaces. I also humbly suggest that it has the most comprehensive coverage of TOSCA features (including TOSCA 1.1 workflows and policy triggers), as well as a robust and extensible requirements-and-capabilities algorithm implementation. It also features optional integrations to support BPMN, Kubernetes, and OpenStack orchestration via Ansible. The ONAP community can contribute and make it even better.

    Also, Puccini does suggest (via its "Clout" intermediary format) an internal structure for an object model that is independent from TOSCA, and may help to begin to address the comment above by Chris Lauwers. "Clout" is un-opinionated by design (and trivially stored in graph databases) but it's very much possible to create an opinionated derivative that can be standardized across ONAP components. The technology is there, it's just a matter of the community agreeing on structure.

    If there's interest, I would happy to work together to create a PoC to see how Puccini can work with for your favorite ONAP component (SDC?).