Note these instruction refers to current master branch. The release-1.0.0 branch have development issues that have only been fixed in master branch.

And this does NOT refer to DCAEGEN2 which have a completely new setup/process and is getting released in the R1/Amsterdam Release.

Refer to DCAE MOD User Guide for latest

DCAE Controller Overview

Make sure your settings.xml includes the onap public repo for some snapshot jar downloads during the build.

DCAE Controller LF GIT repositories

  1. dcae/controller
  2. dcae/operation/utils
  3. dcae/controller/analytics
  4. dcae/demo

SOMF - Sirius Operational Management Framework

SOMF is a model-driven framework for building controllers. SOMF Is using the Eclipse Modeling Framework (EMF) for data modeling.

LF SOMF repositories

  1. ncomp/maven - contains a few maven related artifacts
  2. ncomp/utils - contains Java utilities and libraries that are used across DCAE Controller projects
  3. ncomp/core - contains core models
  4. ncomp/sirius/manager - contains the core SOMF implementation
  5. ncomp/docker - contains SOMF adaptor used to communicate with Docker Engine API
  6. ncomp/openstack- contains SOMF adaptor used to communicate with OpenStack API
  7. ncomp/cdap- contains SOMF adaptor used to communicate with CDAP API+

Development Setup

GIT repositories and build order

  1. ncomp/maven
  2. ncomp/utils
  3. dcae/operation/utils
  4. ncomp/core
  5. ncomp/sirius/manager
  6. ncomp/cdap
  7. ncomp/docker
  8. ncomp/openstack
  9. dcae/controller
  10. dcae/controller/analytics
  11. dcae/demo

Eclipse Setup

Since DCAE Controller/SOMF is build on top of EMF development is tightly tied to using Eclipse IDE with specific plugin installed. This section describe the process to setup a development Eclipse.

  1. Install Neon.3 Java Developer Version http://www.eclipse.org/downloads/packages/eclipse-ide-java-developers/neon3
  2. Install software from Neon update site
    1. Eclipse Plug-in Development Environment           3.12.2.v20161124-1400
    2. EMF - Eclipse Modeling Framework Xcore SDK   1.4.0.v20160526-0606
  3. Install software from Eclipse Market Place
    1. YEdit Feature     1.0.20.201509041456-RELEASE
  4. Install Groovy from http://dist.springsource.org/snapshot/GRECLIPSE/e4.6/
    1. Groovy-Eclipse Feature             2.9.2.xx-201703131833-e46
    2. Groovy-Eclipse M2E integration            2.9.2.xx-201703131833-e46
    3. Groovy Compiler 2.4 Feature  2.9.2.xx-201703131833-e46

Import Eclipse Projects

The GIT repositories include the Eclipse Project Metadata (e.g., .project, .classpath, and etc files) and thus can be importing into a workspace using the "Import Projects" → "Import existing Eclipse projects". Only the projects that need to be worked on need to be in the workspace.

Working on DCAE Controller projects

There are 2 projects for most features.

  1. A model project which contains the data model  using XCORE file. These projects always have a name (and artifactId) that ends in -model.This file is located on the src/main/xcore directory. And the project is setup so Eclipse will update automatically the generated Java code in the src/main/xcore-gen directory. This project is a straight EMF type project and contain very little SOMF specifics.
  2. A SOMF project which contains SOMF generated code and implementation of the APIs that the model defines.
    1. SOMF generator class Generator.java under the src/main/java folder. It contains the defined customization for creating the SOMF implementation. Running this class will generate new contents in the two generated folders.
    2. src/main/sirius-gen folder contains generated Java code for implementing Client and Server side functions.
    3. src/main/server-gen folder contains generated Bash and Groovy scripts for implementing Client and Server side functions.
    4. SOMF provider *Provider.java Java classes under the src/main/java folder. These implements the operation/methods that are defined in the model.

DCAE Configuration Setup

The DCAE configuration setup is trying to carefully handle Environmental and Non-Environmental configuration.

The ONAP Demo setup is creating another level of complexity to the setup since the overall ONAP Heat templates will deliver environmental information that need to be include in the Environmental configuration. 

This is the various configuration entities.

  1. DCAE Non Environmental ONAP Demo configuration. https://gerrit.onap.org/r/gitweb?p=dcae/demo.git;a=tree;f=OPENECOMP-DEMO;h=38b2f51a1be5c12f357e228b7a655c0ee97606b8;hb=HEAD with the main files
    1. location-types.yaml define the various DCAE entities (e.g., docker-host VMs, cdap cluster VMs, VES collector etc) that are part of the demo deployment
    2. vm-templates/vm-docker-host.yaml, vm-templates/vm-cdap-cluster.yaml, vm-templates/vm-postgresql.yaml . Define the deployment configuration for various VM deployments.
    3. docker-templates/docker-XX.yaml. Similar for docker deployments.
    4. cdap-templates/cdap-YY.yaml. Similar for docker deployments.
    5. steams.yaml. Define DCAE DMaaP setup. 
    6. HEAT provided information. When the ONAP demo is getting deploy the vm1-dcae-controller VM which will come up with the DCAE controller Docker container. This container will use configuration attributes setup in the file /opt/app/dcae-controller/config.yaml to apply to DCAE controller environment file, which will have expressions like @{XXX} replaced with the value of XXX in the config.yaml file.

      Variable NameSample ValuesNotes
      BASERACKSPACE,2-NIC,1-NIC-FLOATING-IPSRACKSPACE will only work in Rackspace, 2-NIC can work in both Rackspace and non Rackspace environment, and 1-NIC-FLOATING-IPS does not work in Rackspace.
      DCAE-VERSION1.1.0Note only 1.1.0 version is working outside of Rackspace
      OPENSTACK-AUTH-METHODpassword, api-keyNeed to use password for non-Rackspace environments
      DOCKER-VERSION1.1-STAGING-latest
  2. DCAE Environmental ONAP Demo configuration. Currently 3 deployment scenarios are supported (See DCAE-7 - Getting issue details... STATUS for details about current status.)
    1. RACKSPACE. This scenario only work when the Cloud provider is Rackspace. In this each DCAE VM will be setup with a predefined fixed IP on a private network and it will get a random public IP on the public network. This scenario depends on the following values from the HEAT provided environment file: DCAE-VERSION, DOCKER-REGISTRY, DOCKER-VERSION, GIT-MR-REPO, HORIZON-URL, KEYSTONE-URL, NEXUS-PASSWORD, NEXUS-RAWURL, NEXUS-USER, OPENSTACK-KEYNAME, OPENSTACK-PASSWORD, OPENSTACK-PRIVATE-NETWORK, OPENSTACK-PUBKEY, OPENSTACK-REGION, OPENSTACK-TENANT-ID, OPENSTACK-TENANT-NAME, OPENSTACK-USER, POLICY-IP, STATE, ZONE.
    2. 2-NIC. This scenario works in most OpenStack environments and allows higher level of flexibility in assigning IPs etc. Similar to the RACKSPACE setup each DCAE VM will be setup with a predefined fixed IP on a private network and it will get a random public IP on the public network.This scenario depends on the following values from the HEAT provided environment file: BASE=2-NIC, DCAE-VERSION, DNS-IP-ADDR, DOCKER-REGISTRY, DOCKER-VERSION, FLAVOR-LARGE, GIT-MR-REPO, HORIZON-URL, KEYSTONE-URL, NEXUS-PASSWORD, NEXUS-RAWURL, NEXUS-USER, OPENSTACK-AUTH-METHOD, OPENSTACK-KEYNAME, OPENSTACK-PASSWORD, OPENSTACK-PRIVATE-NETWORK, OPENSTACK-AUTH-METHOD, OPENSTACK-PUBKEY, OPENSTACK-REGION, OPENSTACK-TENANT-ID, OPENSTACK-TENANT-NAME, OPENSTACK-USER, POLICY-IP, STATE, UBUNTU-1404-IMAGE, UBUNTU-1604-IMAGE, ZONE, dcae_cdap00_ip_addr, dcae_cdap01_ip_addr, dcae_cdap02_ip_addr, dcae_coll00_ip_addr, dcae_ip_addr, dcae_pstg00_ip_addr, public_net_id. The HEAT template demo/heat/OpenECOMP/onap_openstack_nofloat.yaml provides these variables.
    3. 1-NIC-FLOATING-IPS. This scenario works in most OpenStack environments and allows higher level of flexibility in assigning IPs etc. In this setup each DCAE VM will only have one NIC with an IP on the private network. In addition each VM will have a floating IP from the public network. In this case it is the floating VMs that can get predefined values and the IPs from the private network that are assigned to the NIC are randomly setup. This makes a few issues with the rest of the demo setup but should be working in the 1.1 release. This scenario depends on the following values from the HEAT provided environment file:BSAE=1-NIC-FLOATING-IPS, DCAE-VERSION, DNS-IP-ADDR, DOCKER-REGISTRY, DOCKER-VERSION, FLAVOR-LARGE, GIT-MR-REPO, HORIZON-URL, KEYSTONE-URL, NEXUS-PASSWORD, NEXUS-RAWURL, NEXUS-USER, OPENSTACK-AUTH-METHOD, OPENSTACK-KEYNAME, OPENSTACK-PASSWORD, OPENSTACK-PRIVATE-NETWORK, OPENSTACK-PUBKEY, OPENSTACK-REGION, OPENSTACK-TENANT-ID, OPENSTACK-TENANT-NAME, OPENSTACK-USER, POLICY-IP, STATE, UBUNTU-1404-IMAGE, UBUNTU-1604-IMAGE, ZONE, dcae_cdap00_float_ip_addr, dcae_cdap01_float_ip_addr, dcae_cdap02_float_ip_addr, dcae_coll00_float_ip_addr, dcae_float_ip_addr, dcae_pstg00_float_ip_addr. The HEAT template demo/heat/OpenECOMP/onap_openstack_float.yaml provides these variables. One requirement for this setup is that the floating IPs that is getting used are already associated with the Openstack Tenant used.

The overall deployment flow is the following.

  1. HEAT template
    1. Create the vm1-dcae-controller VM.
    2. Setup  /opt/app/dcae-controller/config.yaml
    3. Start the Docker Container for the DCAE Controller
  2. The DCAE controller will (running /opt/app/dcae-controller-platform-server/bin/controller-startup.sh)
    1. Determine the BASE based on the BASE attribute in config.yaml
    2. Determine the ZONE based on the ZONE attribute in config.yaml
    3. Substitute the HEAT delivered attributes from config.yaml into /opt/app/dcae-controller-platform-server/OPENECOMP-DEMO-$BASE and produce a /opt/app/dcae-controller-platform-server//opt/app/dcae-controller-platform-server/OPENECOMP-DEMO-$ZONE.
      1. bin/dcae-controller.sh rackspace-substitute --from OPENECOMP-DEMO-$BASE --to OPENECOMP-DEMO-$ZONE --file /opt/app/dcae-controller/config.yaml
    4. Merge the Non-environmental configuration OPENECOMP-DEMO with the environmental configuration OPENECOMP-DEMO-$ZONE to create the complete configuration setup for this specific environment which will be placed in GITLINK/OPENECOMP-DEMO-$ZONE
      1. java -cp 'lib/*' org.openecomp.dcae.controller.operation.utils.GenControllerConfiguration $ZONE . GITLINK OPENECOMP-DEMO
    5. The DCAE Controller will be started up and synced with the configuration
      1. bin/dcae-controller.sh start
      2. bin/dcae-controller.sh sync-configuration --environment OPENECOMP-DEMO-$ZONE

    6. The DCAE Controller can then Deploy the various DCAE components define in the demo environment

      1. bin/dcae-controller.sh deploy-service-instance -i $ZONE -s vm-docker-host-1
      2. bin/dcae-controller.sh deploy-service-instance -i $ZONE -s vm-postgresql
      3. bin/dcae-controller.sh deploy-service-instance -i $ZONE -s vm-cdap-cluster
      4. bin/dcae-controller.sh deploy-service-instance -i $ZONE -s docker-databus-controller
      5. bin/dcae-controller.sh deploy-service-instance -i $ZONE -s cdap-helloworld
      6. bin/dcae-controller.sh deploy-service-instance -i $ZONE -s cdap-tca-hi-lo
      7. bin/dcae-controller.sh deploy-service-instance -i $ZONE -s docker-common-event

Common Deployment Issues Seen

{"auth":{"RAX-KSKEY:apiKeyCredentials":{"username":"<some-user-here>","apiKey":"<some-key-here>"}}}

Using Rackspace authentication in non Rackspace. Make sure your are using 1.1 DCAE controller container and that OPENSTACK-AUTH-METHOD = 'password', DOCKER-VERSION =1.1-STAGING-latest and DCAE-VERSION = 1.1.0

java.lang.IllegalArgumentException: !Absolute URI: null/servers

Connectivity to OpenStack APIs is not working look in logs/*.err for the error returns by the keystone API call.

Reporting Issues

Due to the large number of possible issues in a deployment please run the following script and include the output in the Ticket.


#!/bin/bash
PW=$(grep OPENSTACK-PASSWORD /opt/app/dcae-controller/config.yaml | sed s/OPENSTACK-PASSWORD:.//)
set -e
echo ======= config.yaml
cat /opt/app/dcae-controller/config.yaml | sed "s/$PW/XXXXXX/"
echo ======= docker
docker images
docker ps -a
ID=$(docker ps | grep dcae-controller: | cut -c1-12)
echo ======= docker logs
docker logs $ID 2>&1
echo ======= dcae-controller.sh.log
docker exec $ID cat /opt/app/dcae-controller-platform-server/logs/dcae-controller.sh.log
echo ======= reports
docker exec -e GROOVY_HOME=/opt/app/groovy-2.4.6 $ID /opt/app/dcae-controller-platform-server/bin/dcae-controller.sh report -n /reports/dcae/vms
docker exec -e GROOVY_HOME=/opt/app/groovy-2.4.6 $ID /opt/app/dcae-controller-platform-server/bin/dcae-controller.sh report -n /reports/dcae/service-instances
echo ======= logs err
docker exec $ID cat /opt/app/dcae-controller-platform-server/logs/controller-platform-server-controller.err | head -10000 | sed "s/$PW/XXXXXX/"
echo ======= logs out
docker exec $ID cat /opt/app/dcae-controller-platform-server/logs/controller-platform-server-controller.out | head -10000



  • No labels

32 Comments

  1. Hello, when is the plan to publish this guide?

    1. Hope this helps out a little bit to get started.

  2. This should be available early next week. Thanks.

  3. I understand that the DCAE controller manages the rest of the DCAE components. Is there document describing how it deploys and instantiates the components? Is it possible to add more service/application components?

    Thanks.

    1. I will try to add some info about this later.

      Also note that based on the proposed projects that came out of the ONAP F2F meeting last week the current OpenECOMP DCAE controller may get replaced by Common Controller SDK Project Proposal depending on how the ONAP community is going forward.

  4. I am getting following error while compiling "ncomp-core-model".I see that jar file question does exist. 

     Could not resolve dependencies for project org.openecomp.ncomp.core:ncomp-core-model:jar:1.1.0-SNAPSHOT: The following artifacts could not be resolved: org.eclipse.core:org.eclipse.core.runtime:jar:3.10.0.v20140318-2214, org.eclipse.core:org.eclipse.core.contenttype:jar:3.4.200.v20140207-1251: Could not find artifact org.eclipse.core:org.eclipse.core.runtime:jar:3.10.0.v20140318-2214 in opendaylight-mirror (https://nexus.opendaylight.org/content/repositories/public/

    1. Are you sure that you are using the onap nexus in your .m2/settings.xml. The ONAP nexus is proxing all the Eclipse.org required artifacts.

      You may only be using the opendaylight nexus.

      1. Sam

        I am getting same error as posted above, i am using the settings.xml provided at Setting Up Your Development Environment

        1. Update: we can track resolution in our  DCAE-19 - Getting issue details... STATUS

          Guys, getting the same error specific to this project (settings.xml is ok, doing a force update) and it's use of Eclipse EMF in ncomp-maven-xcore.  I'll try the workaround in the following or likely raise a JIRA in ncomp if some excludes don't work

          There are open discussions and a bug on why this artifact is not pushed to maven central

          https://bugs.eclipse.org/bugs/show_bug.cgi?id=497982

          https://www.eclipse.org/forums/index.php/m/1738005/#msg_1738005

          the jar is there but in a "runtime" package - not "org.eclipse.core.runtime" which is what is used from 3.6 until 3.9

          http://repo.maven.apache.org/maven2/org/eclipse/core/runtime/3.10.0-v20140318-2214/

          http://repo.maven.apache.org/maven2/org/eclipse/core/org.eclipse.core.runtime/

          /michael

          Update: the following changes to revert these 2 libraries back to their previous versions resolves and compiles - however I don't know if runtime failures will occur on calls that require the new 3.10 and 3.4.200-v2014 from 3.9 and 3.4.200-v2013 changes.

          Note: IntelliJ will resolve these directly in the IDE as well.


          <dependency>
          <groupId>org.eclipse.core</groupId>
          <artifactId>runtime</artifactId> <!-- from org.eclipse.core.runtime -->
          <!--version>3.10.0.v20140318-2214</version--><!-- does not resolve -->
          <version>3.9.100-v20131218-1515</version><!-- resolves (same pom as 3.10 though -->
          </dependency>
          <dependency>
          <groupId>org.eclipse.core</groupId>
          <artifactId>contentype</artifactId> <!-- from org.eclipse.core.contentType -->
          <!--version>3.4.200.v20140207-1251</version--><!-- does not resolve -->
          <version>3.4.200-v20130326-1255</version> <!-- resolves -->
          </dependency>



          [INFO] ncomp-core-types ................................... SUCCESS [  1.748 s]

          [INFO] ncomp-core-model ................................... SUCCESS [  5.660 s]

          [INFO] ncomp-core-tools ................................... SUCCESS [  0.708 s]

          [INFO] ncomp-core ......................................... SUCCESS [  0.006 s]

          [INFO] BUILD SUCCESS

          wse_onap/onap/ncomp/core

          $ mvn clean install -U -DskipTests=true

          1. Sam

            I had to add one more profile in settings.xml , for "http://nexus.onap.org/content/repositories/public/". the specific version is there. dependencies resolved after this change. 

            1. Santosh - very nice - the eclipse libraries resolve with the public repo - thank you

              Also to resolve the 1.0.0 version (instead of 1.1.0) in org.openecomp.aai.logging-service:logging-api:jar:1.0.0 under aai/rest-client - the onap staging profile was also required

              I'll update the reference settings.xml

                    <activeProfile>openecomp-public</activeProfile>

                    <activeProfile>openecomp-staging</activeProfile>

              1. Thanks Mike and Santosh. Compilation ran without error with the new "settings.xml"

  5. If using eclipse, you will have to uncheck Preference.XCore.Compiler.Overwrite existing files and sth else, some java under gen directory was not purely generated

  6. for build dcae-controller-core in eclipse, you may have to add groovy runtime to build path as http://stackoverflow.com/questions/4495564/error-of-grails-project-from-svn-groovyobject-cannot-be-resolved

  7. Sam

    to resolve error  "Bundle 'org.eclipse.core.runtime' cannot be resolved" follow the steps

    http://stackoverflow.com/questions/24558273/bundle-org-eclipse-core-runtime-cannot-be-resolved

  8. DO NOT use newest Apach Maven 3.5.0, which failed on build most projects of dcae as error 

    [ERROR] 13508
    java.lang.ArrayIndexOutOfBoundsException: 13508
    at org.codehaus.plexus.util.xml.pull.MXParser.parsePI(MXParser.java:2502)
    at org.codehaus.plexus.util.xml.pull.MXParser.parseEpilog(MXParser.java:1604)
    at org.codehaus.plexus.util.xml.pull.MXParser.nextImpl(MXParser.java:1434)

    .......



    Apache Maven 3.3.9 do works.


  9. Comment to: "DCAE Configuration Setup" 2b. 2-NIC

    Hi,
    in the upper 2-NIC description the  "DCAE VM will be setup with a predefined fixed IP on a private network and it will get a random public IP on the public network", which seems not to be correct, as I described in my question
    The VM gets currently 2 interfaces in the private network. 

    Should I create a JIRA item ?

    Best regards
    Andreas 

  10. Hi all,
    I am trying to build dcae controller. For this I installed Eclipse IDE for java developer neon 3 version.
    In dcae controller developer guide, for eclipse setup its mentioned to install software from Neon update site
    Eclipse Plug-in Development Environment 3.12.2.v20161124-1400
    EMF - Eclipse Modeling Framework Xcore SDK 1.4.0.v20160526-060

    I couldn't find above mentioned versions.
    http://www.eclipse.org/modeling/emf/downloads/?


    Can anyone help me or this?


    Thanks & Regards,
    Jyothis S

    1. Hi Jyothi, I have the latest Eclipse Oxygen release. And in this IDE if you go to Help/Install New Software and choose 'Oxygen - http://download.eclipse.org/releases/oxygen/201706281000' repository, then type EMF in a Search field it will find a bunch of things. Open 'Modeling' group and you should be able to see and install 'Eclipse Modeling Framework Xcore SDK' (but the version is higher for Oxygen release). Should be the same flow for Neon. And I think if you use Neon the version would probably match...

      However to be fair I couldn't find 'Eclipse Plug-in Development Environment' anywhere. Perhaps it is only available for Neon release, so you could try searching for it the same way.

      1. The only time I tried Oxygen something was not working. I did not have time to debug so I am not sure if Oxygen works, please stay with the Neon.

    2. What versions to you see? As long as the major.minor is the same we should be fine.

  11. Am running into problem with message router with proxy


    Step 3/13 : RUN apk add --update tar wget curl docker coreutils
     ---> Running in 5df62cd19644
    fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
    ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/main: IO ERROR
    WARNING: Ignoring APKINDEX.167438ca.tar.gz: No such file or directory
    fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
    ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/community: IO ERROR
    WARNING: Ignoring APKINDEX.a2e6dac0.tar.gz: No such file or directory
    ERROR: unsatisfiable constraints:
      coreutils (missing):
        required by: world[coreutils]
      curl (missing):
        required by: world[curl]
      docker (missing):
        required by: world[docker]
      tar (missing):
        required by: world[tar]
      wget (missing):
        required by: world[wget]
    ERROR: Service 'kafka' failed to build: The command '/bin/sh -c apk add --update tar wget curl docker coreutils' returned a non-zero code: 5


  12. I am trying to start up the CDAP vms, I follow the commands in /tmp/dcae_install.sh in CDAP vm.

    when I try to fetch the artifact "dcae-controller-service-dmapp-drsub:1.0.0:zip:runtime", it reports can not find such file. I check the corresponding repositories https://nexus.onap.org/content/repositories/staging/org/openecomp/dcae/controller/dcae-controller-service-dmaap-drsub/1.0.0/, there is only pom file(so does 1.1.0 on both staging and snapshots). Is this correct?

    Can cdap run without this artifact?

    1. I also met this issue.Seems CDAP can run correctly without that.

      1. Yes that can be ignored. There JIRA ticket for this DCAEGEN2-9 - Getting issue details... STATUS but given that is this not part of R1 that is not likely going to be fixed.

        1. Hi Carsten, can you please explain your comment about 'this is not part of R1' ? What exactly is not in R1? As far as I understand DCAEGEN2 is in R1.

          1. The DCAE deployment process and DCAE development Process is changed in R1 Amsterdam Release. This page refer to GEN1.

            1. Right, dcaegen2 code is out in gerrit already. I'm aware of the changes. Do you have an idea when an updated installation code for dcaegen2 would be posted?

  13. I develop an analysis module consulting cdap-tca-hi-lo? how can I deploy it to DCAE? What files should I provide to DCAE Controller?

    1. Adding a new analysis module is not that straightforward and using GEN1 DCAE may not be the right direction at this time. I suggest you reach out to the DCAE project and target DCAEGEN2.

  14. Hi,

    I was thinking to update the ONAP on Vagrant script for DCAE[1] to make something that could be useful for setting your development environment. Can someone validate if all the steps and repositories  are covered in that script? The goal is to provision a VM with all the dependencies installed and sharing the source code with the host, so developers can test their functionally in the VM and keep ssh keys and IDE in the host.

    [1] https://git.onap.org/integration/tree/bootstrap/vagrant-onap/lib/dcae