This is a potential draft of a project proposal template.  It is not final or to be used until the TSC approves it.

Project Name:

• Proposed name for the project: DCAE
analytics/holmes
Miscellaneous:
DemoRepo: dcaegen2/demo
• Proposed name for the top level repository name: dcaegen2
  • Note: For the 4Q17 release, one of major goals for DCAE is to evolve from the "old" controller that is currently in gerrit.onap.org with a new controller that follows the Common Controller Framework.  The switch will maintain external (to other ONAP component) compatibility, but NOT backwards compatible internally.   That is, subcomponents built for the old controller will not work with the new controller or vice versa.  We are proposing to us a new top level naming “dcaegen2” for repos of subcomponents that are compatible with the new controller because it appears to be the cleanest approach to avoid confusion.  The existing "dcae" top level still hosts repos of subcomponent projects that are compatible with the old controller.  Eventually we will phase out the "dcae" tree but during the transition period both trees co-exist.
    Detailed list of proposed subprojects and their repos
    DCAE Platform components:
  • DCAE Gen2 Controller
    • Repo:
      • dcaegen2/controller
      • dcaegen2/controller/dispatcher
      • dcaegen2/controller/inventory
      • dcaegen2/controller/sch
  • DCAE Gen2 Analytics Platform
    • Repo:
      • dcaegen2/apod/cdap
  • DCAE Gen2 Persistent Storage
    • Repo:
      • dcaegen2/storage/pgaas
      • dcaegen2/storage/esaas
      • dcaegen2/storage/mdbaas
  DCAE Gen2 UtilitiesRepo:dcaegen2/utils
  DCAE applications and service components:
  
  • DCAE Gen2  VES Collector
    • Repo:
      • dcaegen2/collectors/ves
  • DCAE Gen2 SNMP Trap Collector
    • Repo:
      • dcaegen2/collectors/snmptrap
• DCAE Gen2 CDAP Analytics
  • Repo:
    • dcaegen2/apod/analytics-ms
Holmes correlation analyticsRepo:dcaegen2/
• • 
    
    

Project description:

DCAE is the umbrella name for a number of subprojects, components collectively fulfilling the role of Data Collection Events and Analytics for , Analytics, and Events generation for ONAP.  The architecture of DCAE targets flexible, plug-able, micros-service oriented, and model based component deployment and service composition.  DCAE also support multi-site collection and analytics operations which are essential for large ONAP deployments. 

Because of the large potential scope for DCAE, the subprojects proposed for 4Q17 R1 are prioritized as follows: 

1. DCAE Platform.  This is the foundation for DCAE, needed by any ONAP deployment that requires DCAE.  DCAE Platform includes the DCAE Controller Framework, dispatcher, inventory, service chaining, storage (Postgres), CDAP platform, etc.
2. DCAE collectors and analytics that are needed by the 4Q17 use cases. So far we have identified the VES and SNMP Trap collectors; Threshold Crossing and String matching analytics.  We expect the list to grow and be finalized as the ONAP R1 use cases and the control loops for the use cases are fully defined.  
3. Other collectors and analytics that have been identified as valuable by the community.

More specifically the DCAE team proposes the following sub-projects for 4Q17 R1 ONAP release.  

DCAE Platform components:

• Controller:
  • The new ONAP DCAE (gen2)  controller follows the paradigm of Common Controller Framework for command and control.  Hence the core function of the DCAE Controller code resides within the Common Controller Framework project.  This project is for adaptation of the Common Controller Framework for DCAE use.
• Analytics platform:
  • CDAP
• Storage: this group of projects provides data storage solutions for DCAE.  The common usages include supporting DCAE's own functions; storage for data analytics; persistent data storage; etc.  For this release we target three popular database technologies.
  • Postgres as a Service (PGaaS)
  • Elastic Search as a Service (ESaaS)
  • MariaDB as a Service (MDBaaS)
• Utilities: this project contains various utility for DCAE development, testing, and operations.
  

DCAE applications and service components:

• Collectors:
  • VES Collector:  VNF Event Stream collector
  • SNMP Trap collector
  • ... (Possibly more based on identified 4Q17 use case needs, community interests, and available resources).
    
• Analytics: this group of projects include various analytics related components and applications.  Three projects are proposed under this group:
  • Threshold Crossing Analytics: for performance metric threshold crossing analysis that is needed for supporting vFW/vDNS control flow.
  • String Match: for SNMP trap analysis that is needed for supporting the vVoLTE and vCPE CLAMP flows.
  • Holmes: integration of the OpenO Mercury release alert correlation analytics.

Miscellaneous:

...

DCAE components generally fall into two categories: DCAE Platform Components and DCAE Services Components.  DCAE Platform consists of components that are needed for any deployments.  They form the foundation for deploying and managing DCAE Service components, which are deployed on-demand based on specific collection and analytics needs.

DCAE Platform

DCAE Platform consists of a growing list of components.  For R1 release this includes the Cloudify Manager, Consul, Dispatcher, Policy Handler, Service Changing Handler, Service Infrastructure (Docker host and CDAP cluster), and CDAP Broker.  Their roles and functions are described below:

• Cloudify Manager: Cloudify, through its army of plugins, is capable of relationship topology base resource orchestration in many levels and cross different technologies.  It is the lifecycle management engine of DCAE.  Various resource deployment, change, allocation, configuration, etc, operations are all done through Cloudify.  [Note: This component is part of the Common Controller SDK, or ccsdk project.  How DCAE uses it will be described more below regarding deployment.]
• Consul:  Consul is a service discovery technology for distributed fault detection and KV store.  DCAE uses Consul for service and configuration registry.   [Note: This component is part of the Common Controller SDK, or ccsdk project. How DCAE uses it will be described more below regarding deployment. ]
• Service infrastructure: DCAE platform supports two kinds of infrastructures, the Docker container hosts and CDAP/Hadoop clusters.  The former is for running containerized applications and services.  And the latter is for running CDAP-Hadoop based big data analytics.
• Dispatcher: Dispatcher is a NB API provider for the DCAE Services.  Service related triggers, such as deploying/undeploying services, changing configurations, etc all arrive at the Dispatcher, which then enriches the request, and invokes the right Blueprints and calling Cloudify Manager plugins to complete the necessary changes in virtual resources.
• Inventory: Inventory tracks DCAE related resource information such as various Blueprints and templates that are used by Cloudify Manager to deploy and configure components, as well as inventory information extracted from A&AI that is related to but not really part of DCAE, such as the relationships between virtual network resources and their physical infrastructures.  
• PGaaS: Inventory is backed by a PostgreSQL database for data storage.
• Policy Handler and Service Changing Handler:  They are the interfacing modules for specific external components such as Policy, SDC, etc.
• CDAP Broker: CDAP Broker interfaces between CDAP and Cloudify Manager, supporting carrying out various Cloudify CDAP operations onto the CDAP.

Image Added

The DCAE platform is expected to be deployed through the ONAP Operations Manager (OOM).  There are two ways to deploy a DCAE system: 1. all DCAE components (both Platform and Service components) deployed by the OOM; or 2. the OOM deploys only the core DCAE platform functions, namely the Lifecycle Manager (Cloudify) and the Service & Config Registry (Consul).  Then this platform core subsequently deploys all other DCAE Platform components.  Note that the platform core is based on the same software tools that are part of the Common Controller SDK just like OOM.  Because of the multi-level responsibility structure, option 2 is more suitable for larger DCAE systems that span cross multiple sites and security and infrastructure technology boundaries.  We will use the option 2 as example in the DCAE Service component deployment description too. 

DCAE Services

DCAE provides data collection and analysis services to the ONAP.  Different analysis services will require different collection components and analytics components.  They are distinct from DCAE Platform components because they are dynamically deployed and managed based on service needs.  More over they are deployed and managed by the DCAE Platform.  These are the DCAE Service components.  

Under DCAE, the collectors and analytics, either as Docker containers or as CDAP applications, will be on-boarded as Blueprints to the DCAE Platform.  At deployment time, as triggered by API calls through Dispatcher and other interfacing Platform modules, the Cloudify Manager completes the deployment of the required DCAE Service resources onto the Docker Host or CDAP Cluster that is determined by the DCAE Platform.  The illustration below depicts that the DCAE is deploying two services (yellow and red).  The required DCAE Service components are deployed onto the DCAE Platform.  In this example, the yellow service is completed by a VNF Event Streaming collector and a Threshold Crossing Analytics; while the red service consists of a SNMP Trap collector, followed by a data normalization step and a Correlation analytics.

In addition, the DCAE Platform will also configure the data paths between the Service components, i.e. by setting up DMaaP topics and configurations.

Image Added

The figure below further illustrates how performance measurement and fault management data, i.e. VNF events, SNMP traps, and alert events, traverse through the DCAE Service components, and eventually depart from DCAE to reach downstream components such as ONAP Policy, or other external systems such as ticketing.

Image Added

* Portal/GUI

(This section is for addressing related TSC comments, not as part of DCAE project.)

Dedicated DCAE Portal/GUI is not part of the DCAE project under the current scope.  In future if such portal is deemed necessary, it may be developed under DCAE, or under Portal or VID project.  At present time, certain aspects of DCAE operation status can be displayed by utilizing a combination of native GUI/Portal of the open source software tools used by DCAE (e.g. CDAP GUI, Cloudify portal, etc), or CLAMP cockpit for a more service level end-to-end view in which DCAE is only a part, or CLI-style and RestAPI interaction for status probing. 

Project Scope:

Because of the large potential scope for DCAE, the components proposed for 4Q17 R1 are prioritized as follows:

1. DCAE Platform: This is top priority because everything DCAE depends on it.  All platform components listed above fall under this priority.
2. DCAE Services: collectors and analytics that are needed for supporting the 4Q17 use cases and Open-O harmonization. We have identified the VES and SNMP Trap collectors; Threshold Crossing and Data Normalization Analytics.  We expect the list to grow and be finalized as the ONAP R1 use cases and the control loops for the use cases are fully defined. 
3. Other collectors, analytics, and functions that have been identified as valuable by the community will be included based on resource availabilities in this and future releases of ONAP. 

• DCAE Portal and design studio
• Additional normalization

(Notes to DCAE team members:  previously we also had DMaaP topics and Call Flow proposals.  After consulting with other project teams, DMaaP topic related work is better suited under the DMaaP project; and call flow related work overlaps with the CLAMP team work.  It would seem that these two do not need separate project proposals under DCAE.  In addition, there were items related to code that is done outside of ONAP, e.g. collectd, VES agent.  ONAP will use them but do they need separate projects/repos under DCAE?)

Scope:

For ONAP 4Q17 release, the proposed DCAE projects are centered around three high level objectives: 1. normalization with Open-O DCAE functions; 2. supporting ONAP TOSCA modeling approach; 3. supporting ONAP use cases including CLAMP, vCPE, VoLTE, and vFW/vDNS. 

Under these high level objectives, the scopes of DCAE projects cover the following functional items:

• Normalization of OpenECOMP/OpenO data collection needs and realization
• More TOSCA model based artifact (component, VNF, and data) on-boarding
• Catalog of artifact models
• VNF/PNF data collection
• New DCAE Controller that follows the Common Controller Framework, and supports DCAE component placement (e.g. edge/central) and federation
• Supporting VES standardized VNF data modeling and collection
• MultiVIM interfacing
• Event triggered life cycle management
• Cloud computing Infrastructure event collection and analytics
• Predictive analytics
• AI/ML

...

• How does this project fit into the rest of the ONAP Architecture?

  
  DCAE performs a vital function within the ONAP architecture.  DCAE collects performance metrics and fault data from the VNFs, PNFs, and computing infrastructure, performing local and global analytics, and generating events that are provided for downstream ONAP components (e.g. Policy) for further operations.

  DCAE follows the TOSCA model based ONAP Operations Manager and Common Controller Framework SDK architecture for component on-boarding, close-loop triggered or event (e.g. A&AI) triggered collector and analytics deployment, configuration, and scaling.Image Removed

• Image Added
  
  
• What other ONAP projects does this project depend on?
  • A&AI, Policy, Micro Services, Modeling, CLAMP, SDC, ONAP Controller, Common Controller FrameworkOOM, CCSDK, DMaaP, Common Services, MultiVIM, Integration, Holmes
• How does this align with external standards/specifications?
  • TOSCA
  • VES (OPNFV)
• Are there dependencies with other open source projects?
  • CDAP, Cloudify, Consul, Hadoop, Elastic Search, PostgreSQL, MariaDB

Resources:

...

Project Technical Lead

• Current
  • Vijay Venkatesh Kumar

Primary Contact Person

John F. Murray
• • (AT&T)
• Past
  • Lusheng
Ji
• • Ji (AT&T)

Names, gerrit IDs, and company affiliations of the committers

...

Name	Gerrit ID	Company	Email	Time Zone	DCAE

...

Component Focus
Lusheng Ji	wrider	AT&T	lji@research.att.com	New Jersey, USA EST/EDT
Vijay Venkatesh Kumar	vv770d	AT&T	vv770d@att.com	New Jersey, USA EST/EDT	collectors controller
Tony Hansen	TonyLHansen	AT&T	tony@att.com	New Jersey, USA EST/EDT

...

database, storage, analytics
Mike Hwang	researchmike	AT&T	mhwang@research.att.com	New Jersey, USA EST/EDT	controller
Yan Yang	yangyan	China Mobile	yangyanyj@chinamobile.com

...

Beijing, China. UTC +8

collectors

...

Xinhui Li

...

xinhuili

...

VMware

...

lxinhui@vmware.

...

com

...

Beijing, China. UTC +8

...

collectors

...

Contributors

Names and affiliations of any other contributors

...

...

...

...

...

...

...

???

...

...

Names and affiliations of any other contributors

...

Company

...

Name

...

AT&T

...

Alok Gupta

...

Jerry Robinson

...

Lusheng Ji

...

Tech Mahindra

...

Sandeep Singh

...

Abhinav Singh

...

Gokul Singraju

...

Futurewei

...

 Xin Miao

...

Deutsch Telcom

...

Mark Fiedler

...

Intel 

...

...

Intel 

...

Maryam Tahhan

...

...

Avinash SIBMYusuf MirzaIBMDavid ParfettChina MobileYuan Liu
• Project Roles (include RACI chart, if applicable)

Other Information

• link to seed code (if applicable)
  • https://gerrit.onap.org/r/#/admin/projects/dcae/*https://gerrit.open-o.org/r/#/admin/projects/holmes-[actions,gui,rule-management,engine-management]
    
• Vendor Neutral
  • if the proposal is coming from an existing proprietary codebase, have you ensured that all proprietary trademarks, logos, product names, etc., have been removed?
    

    The current seed code has been already scanned and cleanup to remove all proprietary trademarks, logos, etc. except openecomp to be replaced by onap

    Subsequent modification to the existing seed code should continue to follow the same scanning and clean up principles.

• Meets Board policy (including IPR)

Use the above information to create a key project facts section on your project page

Key Project Facts

Project Name:

• JIRA project name:  Data  Data Collection , Analytics , and Events
• JIRA project prefix: DCAE-dcaegen2

Repo name: Lifecycle State:

• Mature code that needs enhancement/integration: VES collector, TCA analytics, Dispatcher, Service Changing Handler, Inventory PGaaS, CDAP infrastructure, CDAP Broker
• Incubation: Policy Handler, SNMP Trap collector, additional use case specific collectors/analytics/Blueprints, ESaaS,

Primary Contact:  John F. Murray (AT&T),   Lusheng Ji (AT&T)

Project Lead: mailing list tag [Should match Jira Project Prefix]    Lusheng Ji (AT&T)

Mailing list tag: dcaegen2

Repo structure and names:

Link to approval of additional submitters: 

...