DCAE service components are Docker containers that run data collection and data analysis software for monitoring the health and performance of xNFs deployed by ONAP. These components run in the ONAP Kubernetes environment. Currently (as of the Guilin release), these components are deployed using Cloudify. Each component has a Cloudify blueprint and a set of input parameters.  The DCAE Cloudify Kubernetes plugin (dcaegen2/platform/plugins/k8s, also known as k8splugin) processes a blueprint and the associated inputs and uses the Kubernetes API to create the Kubernetes resources (Deployments, Services, etc.) that make up a running instance of a service component.

For a variety of reasons, we would like to move away from using Cloudify in ONAP. The standard mechanism for deploying components in ONAP is Helm, and we would like to use Helm to deploy DCAE service components. At a high level, this is straightforward: instead of using a blueprint, a set of inputs, and Cloudify to create the needed Kubernetes resources, we use a Helm chart with a values.yaml file containing input parameters to create a similar collection of Kubernetes resources. There are a few complications, however:

  • DCAE service components get their configurations from a Consul key-value store. The DCAE Cloudify Kubernetes plugin knows how to extract configuration information from a Cloudify blueprint and store it into Consul. Over time, DCAE service components may move away from using Consul for configuration, but in the near and medium term, we will need to continue to support Consul-based configuration when we move to Helm deployment of service components. (Service components don't access Consul directly; instead they request their configurations through a DCAE platform component called the config binding service.)
  • Some DCAE service components use DMaaP data router feeds or authenticated DMaaP message router topics to exchange data with other components. Provisioning DMaaP feeds and authenticated topics requires interaction with the DMaaP bus controller. DMaaP provisioning information needs to added to the configuration of the service component. Currently we use a DCAE Cloudify DMaaP plugin to access the DMaaP bus controller provisioning API, and the DMaaP plugin knows how to place the resulting DMaaP configuration information into Consul so that the service component can access it.
  • Some DCAE service components use the ONAP policy subsystem to manage parts of their configurations. There is a DCAE Cloudify policy plugin as well as DCAE policy handler platform service that interact with the policy subsystem to retrieve policy-based configuration information, detect changes to configuration initiated by the policy subsystem, and notify service components of changes. Again, this works through the Consul-based configuration mechanism.
  • Some DCAE service components use a shared DCAE postgres database server. A DCAE Cloudify postgres-as-a-service plugin allows creating databases on the server and setting up access to it.

As we move to Helm-based deployment of DCAE service components, we will need to preserve the functionality we have today. In some cases, we will be able to implement the functionality using Helm and using Kubernetes features such as init containers, so that no changes will be needed to the service components. In other cases, we may need to change the service components themselves.

Basic capabilities

All service components, whether or not they use DMaaP, policy, or postgres,  need a Helm chart that creates a Kubernetes Deployment and (if the service exposes any ports) a Service, similar to what the k8splugin would create. This section describes how we will use Helm to create the necessary Kubernetes resources for a service component and how we will set up the component's configuration in Consul.

Creating a Kubernetes Deployment for a service component

We need to create a Kubernetes Deployment resource for the service component that looks like the Deployment currently created by the k8splugin.  The k8splugin deployment includes the following resources:

  • A container that runs the service component itself, using a Docker image pulled from the ONAP Nexus repository.
  • If the component is using the filebeat centralized logging mechanism, a sidecar container running filebeat, plus the volume and volume mounts needed for logging.
  • If the component is using TLS with certificates from AAF, an initContainer that retrieves the certificate information, plus the volume and volume mounts needed for certificates.
  • If the component is using TLS with certificates from an external CA, an initContainer that retrieves the certificate information.
  • If the component is using both AAF certificates and certificates from an external CA, an initContainer that merges the Java  truststores.

In addition to the resources that the k8splugin creates, we need:

  • An initContainer that waits for the other components that the service component needs, such as AAF and Consul  In the current Cloudify implementation, we can guarantee that a service component won't be deployed before these other components are ready.  We can't make that guarantee for Helm.
  • An initContainer that puts the component configuration into Consul (see next section for details).

The configuration parameters for the component (the image name, log directory,  TLS parameters) currently come from the Cloudify blueprint–either directly or through an inputs file.  With Helm deployment, these parameters will move into the values.yaml file for the component. 

Because there is a common structure to the Kubernetes Deployments for all service components, we should create a Helm template for a DCAE service component Deployment.  The Deployment will be configured for a specific service component through the parameters in the values.yaml file.  The OOM common Helm templates do not include a template for a Deployment, but they do include templates for various parts of a deployment (for instance, a logging sidecar).  The DCAE service component Deployment template should use OOM common templates wherever possible.

Setting up the component configuration in Consul

The k8splugin gets the application's initial configuration from the application_config property in the blueprint and uses the Consul API to push this into Consul, using the service component's name as the key. To accomplish the same thing with Helm, we add another init container to the Deployment. This init container runs a Docker image that can take a key name and a path to a file as an argument and push the content of the file to Consul.  A container that does already exists (org.onap.dcaegen2.deployments.consul-loader-container).  It's being used for some DCAE platform components that also get their configurations from Consul. (For an example, see the OOM chart for the DCAE deployment handler, in the OOM tree at oom/kubernetes/dcaegen2/components/dcae-deployment-handler.)

To support configuration via Consul for DCAE service components:

  • We put the application configuration into the component's values.yaml file as a property.
  • The Helm chart creates a Kubernetes configMap using the application configuration from values.yaml, converted to JSON, as the content of the configMap.
  • The Helm chart adds an initContainer to the component's Deployment, with the consul-loader-container as the image to run with the configMap containing the application configuration mounted onto the container's file system.

When the Helm chart for the service component is deployed, the initContainer will run and will store the application configuration into Consul. When the service component starts, the configuration information will be available
in Consul.

Kubernetes Service

The k8splugin currently creates a ClusterIP service if a component exposes IP ports within the cluster and a NodePort service plus a ClusterIP service if the component exposes a port outside the cluster.  In the latter case, the ClusterIP service is redundant–a NodePort service will also expose a component within the cluster.  The OOM common templates include a template for a Service.  We can use that template to create a Service for the component, with all of the details configured in the values.yaml file.    The k8splugin does not currently allow for exposing a component via an Ingress.  Supporting Ingresses should be investigated as part of the move to Helm.

Extended capabilities

Using DMaaP

Using policies

Using a Postgres database

ONAP already has a Helm-based mechanism for deploying instances of Postgres and setting up credentials for client access.  We use this mechanism to create the DCAE postgres database already.   The drawback to this approach is that it creates a separate instance of Postgres each time it is used.  There has been work on setting up shared instances of databases in ONAP, but so far Postgres has not been included in those efforts. 

As we move DCAE service component deployment to Helm, we should make use of the ONAP project-wide Helm mechanisms.  If DCAE has certain special requirements, we should work to enhance the project-wide mechanisms.

Implementation Phases

Service components deployed at DCAE installation time

As of the Guilin release, we are deploying 6 service components at DCAE installation time, via a script that runs on the DCAE k8s-bootstrap-container:

  1. tcagen2 (

  2. ves-tls (org.onap.dcaegen2.collectors.ves.vescollector)

  3. prh (

  4. hv-ves (org.onap.dcaegen2.collectors.hv-ves.hv-collector-main)

  5. holmes_rules (holmes/rule-management)

  6. holmes_engine (holmes/engine-management)

The first four components do not use any extended capabilities (DMaaP, policies, Postgres database), so they can be moved to a Helm-based deployment without implementing solutions for the extended capabilities.  The last two components use the DCAE postgres database.  Moving these components will require some alternative solution for the database.  The Holmes components are managed by the Holmes project, which is separate from the DCAE project.  The Holmes project will need to address the issue of handling the Postgres database differently.

For ONAP R8 ("Honolulu"), we propose creating the Helm templates and charts needed to deploy the first four components on the list above.

Service components deployed on demand

Helm Chart Management

Currently nearly all of the Helm charts for ONAP, including the charts for the DCAE platform components, are stored in a single source tree under the OOM repository.  Typically the charts are built and stored into a locally-running Helm repository server instance.   This works reasonably well for deploying the base ONAP platform in a single installation step.   Most DCAE service components are deployed and undeployed after the initial ONAP installation.  The charts for DCAE service components should be managed separately from the OOM tree, with charts for released versions of service components being pushed into a public ONAP Helm repository.

  • No labels


  1. Jack LucasVijay Venkatesh Kumar : interesting reading. Left some inline comments.

    I have few comments:

    • when I have firstly heard about moving to Helm charts I thought that we need to implement helm chart generator  (next to currently available blueprint generator ) that will generate helm charts out from component specs and that component specs would be still implementation-free high-level metadata that describes DCAE services - are we going to get rid of them?
    • DCAE microservices are also part of SDC distribution mechanism and can be instantiated using CLAMP UI - are we going to add new deployment artifact to SDC and remodel this part? 
    • Are we going to allow end users to directly deploy helm charts or there will be some kind of UI to onboard and deploy helm charts (e.g. part of DCAE MOD)?
    • for complicated parts like deploying configuration in Consul, DMaaP, policy or PGaaS we can also consider to use K8s operators. What do you think?
  2. HI Pawel Baniewski  - This page documents primarily the deployment aspect for supporting helm migration.

    The item #1 you noted (helm chart generator) will be required from MOD perspective.  Any utility to generate helm artifacts from spec-file (or other similar meta-file) could be added as pluggable module into MOD.

    #2 - DCAE no longer depends on SDC distribution since MOD was introduced; unless there is need for CLAMP, helm based MS onboarding can be managed through MOD. 

    #3 - Yes, DCAE MOD will be extended to support helm based MS onboarding. The deployment could be triggered also from MOD, however there is possibly mS/component helm charts are already available (e.g external repo or onap nexus) - deployment of these could be done manually or CI or via generic OOM service (which can listen on deployment request and invoke helm commands on-demand). The latter could be useful across ONAP for different scenarios besides MOD

    #4 - K8S operators is being explored for DMaap; could be an option also to watch for Policy updates and trigger action to distribute update to matching application/POD. I'm not sure though how k8s operator would fit for Consul or PGaaS?

  3. Vijay Venkatesh Kumar  and Jack Lucas :

    1. Ok
    2. I didn't know that. Does it mean that in Guilin you need to onboard twice DCAE M/S - once in SDC for CLAMP and once in MOD for DCAE? Or in other words: if I want to deploy DCAE M/S from CLAMP UI do I need to distribute it using SDC (as I did in Frankfurt) or I also need to onboard it using DCAE MOD?
    3. This (onboarding/deployment/runtime component to keep helm charts) requires sync with OOM.
    4. K8s operators can do anything. CRD can be created for CBS config/Postgres config and logic written in controller can do almost exactly what is currently done in Cloudify plugins - contact appropriate APIs and execute set of actions.
    5. (NEW) Could you address my inline comments?
    6. (NEW) This document and Vijay's presentation on DCAE weekly present two different approaches to configuration data - which one we should follow? Keep configuration close to apps (e.g. Config Maps) or relay on central service/component (e.g. usage of init container to add M/S configuration to CBS)? If you want to get rid of central configuration component in long term keeping config data close to app is the only future-proof option.
    7. (NEW) Have you considered to move DMaaP topics/feeds provisioning on DCAE MOD level? So such isn't a part of application helm chart but when you design your relationships in UI, MOD e.g. creates extra CRDs to provision config in DMaaP before application helm charts are deployed?
    1. Sorry, missed these comment/questions without email notification update

      • #2 -  yes, if the deployment needs to be triggered from CLAMP, then blueprint must be distributed from SDC. I beleive CLAMP team were working to simply the design - not sure if this is available in Guilin.  In this case - you probably dont have to onboard the MS through DCAE-MOD as the blueprint is already generated and can be loaded into inventory (through SCH when SDC does distribution). 
      • #3 - yes
      • #4 - Good to know; looking forward for your/team expertise in building this capability.
      • #5 - Both options were actually discussed in my presentation. Since we have close to ~20 apps in DCAE currently, cutting of CBS/consul will be big effort to transition. Hence the migration strategy for existing application would be to load initial configuration via the consul-init-container and retain CBS for first go.  Any new application being onboarded into DCAE, can directly move toward config map.  
      • #7 - This is certainly an option. However with open questions around how DMAAP topic/feed can be managed in Helm, design consideration through MOD has not been thought through extensively. This does warrant separate discussion and will depend on solution for topic/feed provisioning under helm.
  4. Vijay Venkatesh Kumar and Jack Lucas

    When you use MOD today its strictly for design time, when you give it a spec it generates blueprints and uploads to the inventory at which point you would configure and deploy the service via either the deployment handler or the DCAE dashboard.

    • If MOD is to generate a helm chart from component spec, what inventory (helm chart registry) would it store the charts in? 
      • Can it determine the registry that the ONAP charts themselves were deployed from and store here, or will a separate registry be required?
      • Will MOD be responsible for the Helm registry or is this the responsibility of the newly defined OOM service ?
    • Once the charts have found their way to the inventory, is the DCAE dashboard interface still going to enable deployment, or would you need to make a call to this OOM service directly (via rest) ? 

    In the presentation during the weekly, there was a suggestion to use either the DCAE SDK or a sidecar for policy (I'm assuming this is monitoring/dynamic config) interactions,
    If the SDK was to be used, I assume it would query the policy API periodically to see if there are changes, but if changes were discovered would it trigger something within the application and provide the config directly, or would it update the configMap ?

    Assuming the component spec remains the format in which you would design a DCAE service, will we see expansion of the spec to enable other k8s resources (such as horizontal pod autoscalers) that were previously not a concept when deploying via cloudify?

    1. Just summarizing our discussion from meeting today - MOD will evolve to support 

      1.  Onboarding MS with helm charts already available/loaded into external registry; this will be referenced in the spec file accordingly so MOD (or deployment engine) can source the chart correctly for deployment
      2. Generation of helm chart for components onboarded dynamically. In this case, the generated charts can be stored in MOD catalog and also pushed into registry created part of ONAP/OOM deployment (registry info could be configuration for MOD)

      Deployment will change for HELM.  Dashboard/Inventory are closely bound to tosca/blueprint workflow hence thinking separate OOM service (or helm client) can be used to trigger deployment of generated charts. 

      As for DCAE SDK for policy handling, in this case the library can provide config directly and trigger application refresh. The sidecar approach may involve ConfigMap to pass the retrieved configuration into application container.  

      And lastly the spec file will change to support new helm onboarding.  As none of functional block is build yet for helm support - we do have flexibility to fine tune and adapt them as we build it. (smile)

  5. I also left some inline comments on the page. Moving to Helm should make it much easier to integrate DCAE.