DCAE (5/11/17)

This is a potential draft of a project proposal template.  It is not final or to be used until the TSC approves it.

Project Name:

• Proposed name for the project: DCAE
• Proposed name for the top level repository name: dcaegen2
  • Note: For the 4Q17 release, one of major goals for DCAE is to evolve from the "old" controller that is currently in gerrit.onap.org with a new controller that follows the Common Controller Framework.  The switch will maintain external (to other ONAP component) compatibility, but NOT backwards compatible internally.   That is, subcomponents built for the old controller will not work with the new controller or vice versa.  We are proposing to us a new top level naming “dcaegen2” for repos of subcomponents that are compatible with the new controller because it appears to be the cleanest approach to avoid confusion.  The existing "dcae" top level still hosts repos of subcomponent projects that are compatible with the old controller.  Eventually we will phase out the "dcae" tree.
    
    

Project description:

DCAE is the umbrella name for a number of components collectively fulfilling the role of Data Collection, Analytics, and Events generation for ONAP.  The architecture of DCAE targets flexible, plug-able, micros-service oriented, model based component deployment and service composition.  DCAE also support multi-site collection and analytics operations which are essential for large ONAP deployments. 

DCAE components generally fall into two categories: DCAE Platform Components and DCAE Services Components.  DCAE Platform consists of components that are needed for any deployments.  They form the foundation for deploying and managing DCAE Service components, which are deployed on-demand based on specific collection and analytics needs.

DCAE Platform

DCAE Platform consists of a growing list of components.  For R1 release this includes the Cloudify Manager, Consul, Dispatcher, Policy Handler, Service Changing Handler, Service Infrastructure (Docker host and CDAP cluster), and CDAP Broker.  Their roles and functions are described below:

• Cloudify Manager: Cloudify, through its army of plugins, is capable of relationship topology base resource orchestration in many levels and cross different technologies.  It is the lifecycle management engine of DCAE.  Various resource deployment, change, allocation, configuration, etc, operations are all done through Cloudify.  [Note: This component is part of the Common Controller SDK, or ccsdk project.  How DCAE uses it will be described more below regarding deployment.]
• Consul:  Consul is a service discovery technology for distributed fault detection and KV store.  DCAE uses Consul for service and configuration registry.   [Note: This component is part of the Common Controller SDK, or ccsdk project. How DCAE uses it will be described more below regarding deployment. ]
• Service infrastructure: DCAE platform supports two kinds of infrastructures, the Docker container hosts and CDAP/Hadoop clusters.  The former is for running containerized applications and services.  And the latter is for running CDAP-Hadoop based big data analytics.
• Dispatcher: Dispatcher is a NB API provider for the DACE Services.  Service related triggers, such as deploying/undeploying services, changing configurations, etc all arrive at the Dispatcher, which then enriches the request, and invokes the right Blueprints and calling Cloudify Manager plugins to complete the necessary changes in virtual resources.
• Inventory: Inventory tracks DCAE related resource information such as various Blueprints and templates that are used by Cloudify Manager to deploy and configure components, as well as inventory information extracted from A&AI that is related to but not really part of DCAE, such as the relationships between virtual network resources and their physical infrastructures.  
• PGaaS: Inventory is backed by a PostgreSQL database for data storage.
• Policy Handler and Service Changing Handler:  They are the interfacing modules for specific external components such as Policy, SDC, etc.
• CDAP Broker: CDAP Broker interfaces between CDAP and Cloudify Manager, supporting carrying out various Cloudify CDAP operations onto the CDAP.

The DCAE platform is expected to be deployed through the ONAP Operations Manager (OOM).  There are two ways to deploy a DCAE system: 1. all DCAE components (both Platform and Service components) deployed by the OOM; or 2. the OOM deploys only the core DCAE platform functions, namely the Lifecycle Manager (Cloudify) and the Service & Config Registry (Consul).  Then this platform core subsequently deploys all other DCAE Platform components.  Note that the platform core is based on the same software tools that are part of the Common Controller SDK just like OOM.  Because of the multi-level responsibility structure, option 2 is more suitable for larger DCAE systems that span cross multiple sites and security and infrastructure technology boundaries.  We will use the option 2 as example in the DCAE Service component deployment description too. 

DCAE Services

DCAE provides data collection and analysis services to the ONAP.  Different analysis services will require different collection components and analytics components.  They are distinct from DCAE Platform components because they are dynamically deployed and managed based on service needs.  More over they are deployed and managed by the DCAE Platform.  These are the DCAE Service components.  

Under DCAE, the collectors and analytics, either as Docker containers or as CDAP applications, will be on-boarded as Blueprints to the DCAE Platform.  At deployment time, as triggered by API calls through Dispatcher and other interfacing Platform modules, the Cloudify Manager completes the deployment of the required DACE Service resources onto the Docker Host or CDAP Cluster that is determined by the DCAE Platform.  The illustration below depicts that the DCAE is deploying two services (yellow and red).  The required DCAE Service components are deployed onto the DCAE Platform.  In this example, the yellow service is completed by a VNF Event Streaming collector and a Threshold Crossing Analytics; while the red service consists of a SNMP Trap collector, followed by a data normalization step and a Correlation analytics.

In addition, the DCAE Platform will also configure the data paths between the Service components, i.e. by setting up DMaaP topics and configurations.

The figure below further illustrates how performance measurement and fault management data, i.e. VNF events, SNMP traps, and alert events, traverse through the DCAE Service components, and eventually depart from DCAE to reach downstream components such as ONAP Policy, or other external systems such as ticketing.

* Portal/GUI

(This section is for addressing related TSC comments, not as part of DCAE project.)

Dedicated DCAE Portal/GUI is not part of the DCAE project under the current scope.  In future if such portal is deemed necessary, it may be developed under DCAE, or under Portal or VID porject.  At present time, certain aspects of DCAE operation status can be displayed by utilizing a combination of native GUI/Portal of the open source software tools used by DCAE (e.g. CDAP GUI, Cloudify portal, etc), or CLAMP cockpit for a more service level end-to-end view in which DCAE is only a part, or CLI-style and RestAPI interaction for status probing. 

Project Scope:

Because of the large potential scope for DCAE, the components proposed for 4Q17 R1 are prioritized as follows:

1. DCAE Platform: This is top priority because everything DCAE depends on it.  All platform components listed above fall under this priority.
2. DCAE Services: collectors and analytics that are needed for supporting the 4Q17 use cases and Open-O harmonization. We have identified the VES and SNMP Trap collectors; Threshold Crossing and Data Normalization Analytics.  We expect the list to grow and be finalized as the ONAP R1 use cases and the control loops for the use cases are fully defined. 
3. Other collectors, analytics, and functions that have been identified as valuable by the community will be included based on resource availabilities in this and future releases of ONAP. 

• DCAE Portal and design studio
• Additional normalization of OpenECOMP/OpenO data collection needs and realization
• More TOSCA model based artifact (component, VNF, and data) on-boarding
• Catalog of artifact models
• VNF/PNF data collection
• MultiVIM interfacing
• Event triggered life cycle management
• Cloud computing Infrastructure event collection and analytics
• Predictive analytics
• AI/ML

Architecture Alignment:

• How does this project fit into the rest of the ONAP Architecture?

  
  DCAE performs a vital function within the ONAP architecture.  DCAE collects performance metrics and fault data from the VNFs, PNFs, and computing infrastructure, performing local and global analytics, and generating events that are provided for downstream ONAP components (e.g. Policy) for further operations.

  DCAE follows the TOSCA model based ONAP Operations Manager and Common Controller SDK architecture for component on-boarding, close-loop triggered or event (e.g. A&AI) triggered collector and analytics deployment, configuration, and scaling.

• 
  
  
• What other ONAP projects does this project depend on?
  • A&AI, Policy, Micro Services, Modeling, CLAMP, SDC, OOM, CCSDK, DMaaP, Common Services, MultiVIM, Integration, Holmes
• How does this align with external standards/specifications?
  • TOSCA
  • VES (OPNFV)
• Are there dependencies with other open source projects?
  • CDAP, Cloudify, Consul, Hadoop, Elastic Search, PostgreSQL, MariaDB

Resources:

Primary Contact Person

• John F. Murray (AT&T)
• Lusheng Ji (AT&T)

Technical Lead

• Lusheng Ji (AT&T)

Names, gerrit IDs, and company affiliations of the committers

• • • • Names and affiliations of any other contributors

        Company	Name	Email
        AT&T	Alok Gupta
        	Jerry Robinson
        	Gayathri Patrachari	gp2421@att.com
        	Alexei Nekrassov	nekrassov@att.com
        BOCO	Jingbo Liu	liujingbo@boco.com.cn
        	Zhangxiong Zhou
        China Mobile	Yuan Liu
        Deutsch Telcom	Mark Fiedler
        Futurewei	Xin Miao
        Huawei	Avinash S
        IBM	Yusuf Mirza
        	David Parfett
        	Janki Vora
        	Amandeep Singh
        Intel	Maryam Tahhan
        	Tim Verrall
        Orange	Vincent Colas
        	Olivier Augizeau
        	Pawel Pawlak
        Reliance Jio	Aayush Bhatnagar
        	Yog Vashishth
        	Adityakar Jha
        Tech Mahindra	Sandeep Singh
        	Abhinav Singh
        	Gokul Singraju
        VMware	Sumit Verdi

• Project Roles (include RACI chart, if applicable)

Other Information

• link to seed code (if applicable)
  • https://gerrit.onap.org/r/#/admin/projects/dcae/*
• Vendor Neutral
  • if the proposal is coming from an existing proprietary codebase, have you ensured that all proprietary trademarks, logos, product names, etc., have been removed?
    

    The current seed code has been already scanned and cleanup to remove all proprietary trademarks, logos, etc. except openecomp to be replaced by onap

    Subsequent modification to the existing seed code should continue to follow the same scanning and clean up principles.

• Meets Board policy (including IPR)

Use the above information to create a key project facts section on your project page

Key Project Facts

Project Name:

• JIRA project name:  Data Collection Analytics and Events
• JIRA project prefix: dcaegen2

Lifecycle State:

• Mature code that needs enhancement/integration: VES collector, TCA analytics, Dispatcher, Service Changing Handler, Inventory PGaaS, CDAP infrastructure, CDAP Broker
• Incubation: Policy Handler, SNMP Trap collector, additional use case specific collectors/analytics/Blueprints, ESaaS,

Primary Contact:  John F. Murray (AT&T),   Lusheng Ji (AT&T)

Project Lead:    Lusheng Ji (AT&T)

Mailing list tag: dcaegen2

Repo structure and names:

Link to approval of additional submitters: