Note these instruction refers to current master branch. The release-1.0.0 branch have development issues that have only been fixed in master branch.
And this does NOT refer to DCAEGEN2 which have a completely new setup/process and is getting released in the R1/Amsterdam Release.
Refer to DCAE MOD User Guide for latest
DCAE Controller Overview
Make sure your settings.xml includes the onap public repo for some snapshot jar downloads during the build.
DCAE Controller LF GIT repositories
- dcae/controller
- dcae/operation/utils
- dcae/controller/analytics
- dcae/demo
SOMF - Sirius Operational Management Framework
SOMF is a model-driven framework for building controllers. SOMF Is using the Eclipse Modeling Framework (EMF) for data modeling.
LF SOMF repositories
- ncomp/maven - contains a few maven related artifacts
- ncomp/utils - contains Java utilities and libraries that are used across DCAE Controller projects
- ncomp/core - contains core models
- ncomp/sirius/manager - contains the core SOMF implementation
- ncomp/docker - contains SOMF adaptor used to communicate with Docker Engine API
- ncomp/openstack- contains SOMF adaptor used to communicate with OpenStack API
- ncomp/cdap- contains SOMF adaptor used to communicate with CDAP API+
Development Setup
GIT repositories and build order
- ncomp/maven
- ncomp/utils
- dcae/operation/utils
- ncomp/core
- ncomp/sirius/manager
- ncomp/cdap
- ncomp/docker
- ncomp/openstack
- dcae/controller
- dcae/controller/analytics
- dcae/demo
Eclipse Setup
Since DCAE Controller/SOMF is build on top of EMF development is tightly tied to using Eclipse IDE with specific plugin installed. This section describe the process to setup a development Eclipse.
- Install Neon.3 Java Developer Version http://www.eclipse.org/downloads/packages/eclipse-ide-java-developers/neon3
- Install software from Neon update site
- Eclipse Plug-in Development Environment 3.12.2.v20161124-1400
- EMF - Eclipse Modeling Framework Xcore SDK 1.4.0.v20160526-0606
- Install software from Eclipse Market Place
- YEdit Feature 1.0.20.201509041456-RELEASE
- Install Groovy from http://dist.springsource.org/snapshot/GRECLIPSE/e4.6/
- Groovy-Eclipse Feature 2.9.2.xx-201703131833-e46
- Groovy-Eclipse M2E integration 2.9.2.xx-201703131833-e46
- Groovy Compiler 2.4 Feature 2.9.2.xx-201703131833-e46
Import Eclipse Projects
The GIT repositories include the Eclipse Project Metadata (e.g., .project, .classpath, and etc files) and thus can be importing into a workspace using the "Import Projects" → "Import existing Eclipse projects". Only the projects that need to be worked on need to be in the workspace.
Working on DCAE Controller projects
There are 2 projects for most features.
- A model project which contains the data model using XCORE file. These projects always have a name (and artifactId) that ends in
-model
.This file is located on thesrc/main/xcore
directory. And the project is setup so Eclipse will update automatically the generated Java code in thesrc/main/xcore-gen
directory. This project is a straight EMF type project and contain very little SOMF specifics. - A SOMF project which contains SOMF generated code and implementation of the APIs that the model defines.
- SOMF generator class
Generator.java
under thesrc/main/java
folder. It contains the defined customization for creating the SOMF implementation. Running this class will generate new contents in the two generated folders. src/main/sirius-gen
folder contains generated Java code for implementing Client and Server side functions.src/main/server-gen
folder contains generated Bash and Groovy scripts for implementing Client and Server side functions.- SOMF provider
*Provider.java
Java classes under thesrc/main/java
folder. These implements the operation/methods that are defined in the model.
- SOMF generator class
DCAE Configuration Setup
The DCAE configuration setup is trying to carefully handle Environmental and Non-Environmental configuration.
The ONAP Demo setup is creating another level of complexity to the setup since the overall ONAP Heat templates will deliver environmental information that need to be include in the Environmental configuration.
This is the various configuration entities.
- DCAE Non Environmental ONAP Demo configuration. https://gerrit.onap.org/r/gitweb?p=dcae/demo.git;a=tree;f=OPENECOMP-DEMO;h=38b2f51a1be5c12f357e228b7a655c0ee97606b8;hb=HEAD with the main files
- location-types.yaml define the various DCAE entities (e.g., docker-host VMs, cdap cluster VMs, VES collector etc) that are part of the demo deployment
- vm-templates/vm-docker-host.yaml, vm-templates/vm-cdap-cluster.yaml, vm-templates/vm-postgresql.yaml . Define the deployment configuration for various VM deployments.
- docker-templates/docker-XX.yaml. Similar for docker deployments.
- cdap-templates/cdap-YY.yaml. Similar for docker deployments.
- steams.yaml. Define DCAE DMaaP setup.
HEAT provided information. When the ONAP demo is getting deploy the vm1-dcae-controller VM which will come up with the DCAE controller Docker container. This container will use configuration attributes setup in the file /opt/app/dcae-controller/config.yaml to apply to DCAE controller environment file, which will have expressions like @{XXX} replaced with the value of XXX in the config.yaml file.
Variable Name Sample Values Notes BASE RACKSPACE,2-NIC,1-NIC-FLOATING-IPS RACKSPACE will only work in Rackspace, 2-NIC can work in both Rackspace and non Rackspace environment, and 1-NIC-FLOATING-IPS does not work in Rackspace. DCAE-VERSION 1.1.0 Note only 1.1.0 version is working outside of Rackspace OPENSTACK-AUTH-METHOD password, api-key Need to use password for non-Rackspace environments DOCKER-VERSION 1.1-STAGING-latest
- DCAE Environmental ONAP Demo configuration. Currently 3 deployment scenarios are supported (See
-
DCAE-7Getting issue details...
STATUS
for details about current status.)
- RACKSPACE. This scenario only work when the Cloud provider is Rackspace. In this each DCAE VM will be setup with a predefined fixed IP on a private network and it will get a random public IP on the public network. This scenario depends on the following values from the HEAT provided environment file: DCAE-VERSION, DOCKER-REGISTRY, DOCKER-VERSION, GIT-MR-REPO, HORIZON-URL, KEYSTONE-URL, NEXUS-PASSWORD, NEXUS-RAWURL, NEXUS-USER, OPENSTACK-KEYNAME, OPENSTACK-PASSWORD, OPENSTACK-PRIVATE-NETWORK, OPENSTACK-PUBKEY, OPENSTACK-REGION, OPENSTACK-TENANT-ID, OPENSTACK-TENANT-NAME, OPENSTACK-USER, POLICY-IP, STATE, ZONE.
- 2-NIC. This scenario works in most OpenStack environments and allows higher level of flexibility in assigning IPs etc. Similar to the RACKSPACE setup each DCAE VM will be setup with a predefined fixed IP on a private network and it will get a random public IP on the public network.This scenario depends on the following values from the HEAT provided environment file: BASE=2-NIC, DCAE-VERSION, DNS-IP-ADDR, DOCKER-REGISTRY, DOCKER-VERSION, FLAVOR-LARGE, GIT-MR-REPO, HORIZON-URL, KEYSTONE-URL, NEXUS-PASSWORD, NEXUS-RAWURL, NEXUS-USER, OPENSTACK-AUTH-METHOD, OPENSTACK-KEYNAME, OPENSTACK-PASSWORD, OPENSTACK-PRIVATE-NETWORK, OPENSTACK-AUTH-METHOD, OPENSTACK-PUBKEY, OPENSTACK-REGION, OPENSTACK-TENANT-ID, OPENSTACK-TENANT-NAME, OPENSTACK-USER, POLICY-IP, STATE, UBUNTU-1404-IMAGE, UBUNTU-1604-IMAGE, ZONE, dcae_cdap00_ip_addr, dcae_cdap01_ip_addr, dcae_cdap02_ip_addr, dcae_coll00_ip_addr, dcae_ip_addr, dcae_pstg00_ip_addr, public_net_id. The HEAT template demo/heat/OpenECOMP/onap_openstack_nofloat.yaml provides these variables.
- 1-NIC-FLOATING-IPS. This scenario works in most OpenStack environments and allows higher level of flexibility in assigning IPs etc. In this setup each DCAE VM will only have one NIC with an IP on the private network. In addition each VM will have a floating IP from the public network. In this case it is the floating VMs that can get predefined values and the IPs from the private network that are assigned to the NIC are randomly setup. This makes a few issues with the rest of the demo setup but should be working in the 1.1 release. This scenario depends on the following values from the HEAT provided environment file:BSAE=1-NIC-FLOATING-IPS, DCAE-VERSION, DNS-IP-ADDR, DOCKER-REGISTRY, DOCKER-VERSION, FLAVOR-LARGE, GIT-MR-REPO, HORIZON-URL, KEYSTONE-URL, NEXUS-PASSWORD, NEXUS-RAWURL, NEXUS-USER, OPENSTACK-AUTH-METHOD, OPENSTACK-KEYNAME, OPENSTACK-PASSWORD, OPENSTACK-PRIVATE-NETWORK, OPENSTACK-PUBKEY, OPENSTACK-REGION, OPENSTACK-TENANT-ID, OPENSTACK-TENANT-NAME, OPENSTACK-USER, POLICY-IP, STATE, UBUNTU-1404-IMAGE, UBUNTU-1604-IMAGE, ZONE, dcae_cdap00_float_ip_addr, dcae_cdap01_float_ip_addr, dcae_cdap02_float_ip_addr, dcae_coll00_float_ip_addr, dcae_float_ip_addr, dcae_pstg00_float_ip_addr. The HEAT template demo/heat/OpenECOMP/onap_openstack_float.yaml provides these variables. One requirement for this setup is that the floating IPs that is getting used are already associated with the Openstack Tenant used.
The overall deployment flow is the following.
- HEAT template
- Create the vm1-dcae-controller VM.
- Setup /opt/app/dcae-controller/config.yaml
- Start the Docker Container for the DCAE Controller
- The DCAE controller will (running /opt/app/dcae-controller-platform-server/bin/controller-startup.sh)
- Determine the BASE based on the BASE attribute in config.yaml
- Determine the ZONE based on the ZONE attribute in config.yaml
- Substitute the HEAT delivered attributes from config.yaml into /opt/app/dcae-controller-platform-server/OPENECOMP-DEMO-$BASE and produce a /opt/app/dcae-controller-platform-server//opt/app/dcae-controller-platform-server/OPENECOMP-DEMO-$ZONE.
bin/dcae-controller.sh rackspace-substitute --from OPENECOMP-DEMO-$BASE --to OPENECOMP-DEMO-$ZONE --file /opt/app/dcae-controller/config.yaml
- Merge the Non-environmental configuration OPENECOMP-DEMO with the environmental configuration OPENECOMP-DEMO-$ZONE to create the complete configuration setup for this specific environment which will be placed in GITLINK/OPENECOMP-DEMO-$ZONE
java -cp 'lib/*' org.openecomp.dcae.controller.operation.utils.GenControllerConfiguration $ZONE . GITLINK OPENECOMP-DEMO
- The DCAE Controller will be started up and synced with the configuration
bin/dcae-controller.sh start
bin/dcae-controller.sh sync-configuration --environment OPENECOMP-DEMO-$ZONE
The DCAE Controller can then Deploy the various DCAE components define in the demo environment
bin/dcae-controller.sh deploy-service-instance -i $ZONE -s vm-docker-host-1
bin/dcae-controller.sh deploy-service-instance -i $ZONE -s vm-postgresql
bin/dcae-controller.sh deploy-service-instance -i $ZONE -s vm-cdap-cluster
bin/dcae-controller.sh deploy-service-instance -i $ZONE -s docker-databus-controller
bin/dcae-controller.sh deploy-service-instance -i $ZONE -s cdap-helloworld
bin/dcae-controller.sh deploy-service-instance -i $ZONE -s cdap-tca-hi-lo
bin/dcae-controller.sh deploy-service-instance -i $ZONE -s docker-common-event
Common Deployment Issues Seen
{"auth":{"RAX-KSKEY:apiKeyCredentials":{"username":"<some-user-here>","apiKey":"<some-key-here>"}}}
Using Rackspace authentication in non Rackspace. Make sure your are using 1.1 DCAE controller container and that OPENSTACK-AUTH-METHOD = 'password', DOCKER-VERSION =1.1-STAGING-latest and DCAE-VERSION = 1.1.0
java.lang.IllegalArgumentException: !Absolute URI: null/servers
Connectivity to OpenStack APIs is not working look in logs/*.err for the error returns by the keystone API call.
Reporting Issues
Due to the large number of possible issues in a deployment please run the following script and include the output in the Ticket.
#!/bin/bash
PW=$(grep OPENSTACK-PASSWORD /opt/app/dcae-controller/config.yaml | sed s/OPENSTACK-PASSWORD:.//)
set -e
echo ======= config.yaml
cat /opt/app/dcae-controller/config.yaml | sed "s/$PW/XXXXXX/"
echo ======= docker
docker images
docker ps -a
ID=$(docker ps | grep dcae-controller: | cut -c1-12)
echo ======= docker logs
docker logs $ID 2>&1
echo ======= dcae-controller.sh.log
docker exec $ID cat /opt/app/dcae-controller-platform-server/logs/dcae-controller.sh.log
echo ======= reports
docker exec -e GROOVY_HOME=/opt/app/groovy-2.4.6 $ID /opt/app/dcae-controller-platform-server/bin/dcae-controller.sh report -n /reports/dcae/vms
docker exec -e GROOVY_HOME=/opt/app/groovy-2.4.6 $ID /opt/app/dcae-controller-platform-server/bin/dcae-controller.sh report -n /reports/dcae/service-instances
echo ======= logs err
docker exec $ID cat /opt/app/dcae-controller-platform-server/logs/controller-platform-server-controller.err | head -10000 | sed "s/$PW/XXXXXX/"
echo ======= logs out
docker exec $ID cat /opt/app/dcae-controller-platform-server/logs/controller-platform-server-controller.out | head -10000
32 Comments
Yusuf Mirza
Hello, when is the plan to publish this guide?
Carsten Lund
Hope this helps out a little bit to get started.
Yusuf Mirza
Yes, thanks!
Vijay Venkatesh Kumar
This should be available early next week. Thanks.
Yury Novitsky
I understand that the DCAE controller manages the rest of the DCAE components. Is there document describing how it deploys and instantiates the components? Is it possible to add more service/application components?
Thanks.
Carsten Lund
I will try to add some info about this later.
Also note that based on the proposed projects that came out of the ONAP F2F meeting last week the current OpenECOMP DCAE controller may get replaced by Common Controller SDK Project Proposal depending on how the ONAP community is going forward.
Sandeep Singh
I am getting following error while compiling "ncomp-core-model".I see that jar file question does exist.
Could not resolve dependencies for project org.openecomp.ncomp.core:ncomp-core-model:jar:1.1.0-SNAPSHOT: The following artifacts could not be resolved: org.eclipse.core:org.eclipse.core.runtime:jar:3.10.0.v20140318-2214, org.eclipse.core:org.eclipse.core.contenttype:jar:3.4.200.v20140207-1251: Could not find artifact org.eclipse.core:org.eclipse.core.runtime:jar:3.10.0.v20140318-2214 in opendaylight-mirror (https://nexus.opendaylight.org/content/repositories/public/
Carsten Lund
Are you sure that you are using the onap nexus in your .m2/settings.xml. The ONAP nexus is proxing all the Eclipse.org required artifacts.
You may only be using the opendaylight nexus.
Sam
I am getting same error as posted above, i am using the settings.xml provided at Setting Up Your Development Environment.
Michael O'Brien
Update: we can track resolution in our DCAE-19 - Getting issue details... STATUS
Guys, getting the same error specific to this project (settings.xml is ok, doing a force update) and it's use of Eclipse EMF in ncomp-maven-xcore. I'll try the workaround in the following or likely raise a JIRA in ncomp if some excludes don't work
There are open discussions and a bug on why this artifact is not pushed to maven central
https://bugs.eclipse.org/bugs/show_bug.cgi?id=497982
https://www.eclipse.org/forums/index.php/m/1738005/#msg_1738005
the jar is there but in a "runtime" package - not "org.eclipse.core.runtime" which is what is used from 3.6 until 3.9
http://repo.maven.apache.org/maven2/org/eclipse/core/runtime/3.10.0-v20140318-2214/
http://repo.maven.apache.org/maven2/org/eclipse/core/org.eclipse.core.runtime/
/michael
Update: the following changes to revert these 2 libraries back to their previous versions resolves and compiles - however I don't know if runtime failures will occur on calls that require the new 3.10 and 3.4.200-v2014 from 3.9 and 3.4.200-v2013 changes.
Note: IntelliJ will resolve these directly in the IDE as well.
[INFO] ncomp-core-types ................................... SUCCESS [ 1.748 s]
[INFO] ncomp-core-model ................................... SUCCESS [ 5.660 s]
[INFO] ncomp-core-tools ................................... SUCCESS [ 0.708 s]
[INFO] ncomp-core ......................................... SUCCESS [ 0.006 s]
[INFO] BUILD SUCCESS
wse_onap/onap/ncomp/core
$ mvn clean install -U -DskipTests=true
Sam
I had to add one more profile in settings.xml , for "http://nexus.onap.org/content/repositories/public/". the specific version is there. dependencies resolved after this change.
Michael O'Brien
Santosh - very nice - the eclipse libraries resolve with the public repo - thank you
Also to resolve the 1.0.0 version (instead of 1.1.0) in org.openecomp.aai.logging-service:logging-api:jar:1.0.0 under aai/rest-client - the onap staging profile was also required
I'll update the reference settings.xml
<activeProfile>openecomp-public</activeProfile>
<activeProfile>openecomp-staging</activeProfile>
Sandeep Singh
Thanks Mike and Santosh. Compilation ran without error with the new "settings.xml"
Cang Chen
If using eclipse, you will have to uncheck Preference.XCore.Compiler.Overwrite existing files and sth else, some java under gen directory was not purely generated
Cang Chen
for build dcae-controller-core in eclipse, you may have to add groovy runtime to build path as http://stackoverflow.com/questions/4495564/error-of-grails-project-from-svn-groovyobject-cannot-be-resolved
Sam
to resolve error "Bundle 'org.eclipse.core.runtime' cannot be resolved" follow the steps
http://stackoverflow.com/questions/24558273/bundle-org-eclipse-core-runtime-cannot-be-resolved
Cang Chen
DO NOT use newest Apach Maven 3.5.0, which failed on build most projects of dcae as error
[ERROR] 13508
java.lang.ArrayIndexOutOfBoundsException: 13508
at org.codehaus.plexus.util.xml.pull.MXParser.parsePI(MXParser.java:2502)
at org.codehaus.plexus.util.xml.pull.MXParser.parseEpilog(MXParser.java:1604)
at org.codehaus.plexus.util.xml.pull.MXParser.nextImpl(MXParser.java:1434)
.......
Apache Maven 3.3.9 do works.
Andreas Geissler
Comment to: "DCAE Configuration Setup" 2b. 2-NIC
Hi,
in the upper 2-NIC description the "DCAE VM will be setup with a predefined fixed IP on a private network and it will get a random public IP on the public network", which seems not to be correct, as I described in my question.
The VM gets currently 2 interfaces in the private network.
Should I create a JIRA item ?
Best regards
Andreas
Jyothis S
Hi all,
I am trying to build dcae controller. For this I installed Eclipse IDE for java developer neon 3 version.
In dcae controller developer guide, for eclipse setup its mentioned to install software from Neon update site
Eclipse Plug-in Development Environment 3.12.2.v20161124-1400
EMF - Eclipse Modeling Framework Xcore SDK 1.4.0.v20160526-060
I couldn't find above mentioned versions.
http://www.eclipse.org/modeling/emf/downloads/?
Can anyone help me or this?
Thanks & Regards,
Jyothis S
Evguenia (Jane) Freider
Hi Jyothi, I have the latest Eclipse Oxygen release. And in this IDE if you go to Help/Install New Software and choose 'Oxygen - http://download.eclipse.org/releases/oxygen/201706281000' repository, then type EMF in a Search field it will find a bunch of things. Open 'Modeling' group and you should be able to see and install 'Eclipse Modeling Framework Xcore SDK' (but the version is higher for Oxygen release). Should be the same flow for Neon. And I think if you use Neon the version would probably match...
However to be fair I couldn't find 'Eclipse Plug-in Development Environment' anywhere. Perhaps it is only available for Neon release, so you could try searching for it the same way.
Carsten Lund
The only time I tried Oxygen something was not working. I did not have time to debug so I am not sure if Oxygen works, please stay with the Neon.
Carsten Lund
What versions to you see? As long as the major.minor is the same we should be fine.
GOKUL SINGARAJU
Am running into problem with message router with proxy
Step 3/13 : RUN apk add --update tar wget curl docker coreutils
---> Running in 5df62cd19644
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/main: IO ERROR
WARNING: Ignoring APKINDEX.167438ca.tar.gz: No such file or directory
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/community: IO ERROR
WARNING: Ignoring APKINDEX.a2e6dac0.tar.gz: No such file or directory
ERROR: unsatisfiable constraints:
coreutils (missing):
required by: world[coreutils]
curl (missing):
required by: world[curl]
docker (missing):
required by: world[docker]
tar (missing):
required by: world[tar]
wget (missing):
required by: world[wget]
ERROR: Service 'kafka' failed to build: The command '/bin/sh -c apk add --update tar wget curl docker coreutils' returned a non-zero code: 5
You Li
I am trying to start up the CDAP vms, I follow the commands in /tmp/dcae_install.sh in CDAP vm.
when I try to fetch the artifact "dcae-controller-service-dmapp-drsub:1.0.0:zip:runtime", it reports can not find such file. I check the corresponding repositories https://nexus.onap.org/content/repositories/staging/org/openecomp/dcae/controller/dcae-controller-service-dmaap-drsub/1.0.0/, there is only pom file(so does 1.1.0 on both staging and snapshots). Is this correct?
Can cdap run without this artifact?
xing xiaocheng
I also met this issue.Seems CDAP can run correctly without that.
Carsten Lund
Yes that can be ignored. There JIRA ticket for this DCAEGEN2-9 - Getting issue details... STATUS but given that is this not part of R1 that is not likely going to be fixed.
Evguenia (Jane) Freider
Hi Carsten, can you please explain your comment about 'this is not part of R1' ? What exactly is not in R1? As far as I understand DCAEGEN2 is in R1.
Carsten Lund
The DCAE deployment process and DCAE development Process is changed in R1 Amsterdam Release. This page refer to GEN1.
Evguenia (Jane) Freider
Right, dcaegen2 code is out in gerrit already. I'm aware of the changes. Do you have an idea when an updated installation code for dcaegen2 would be posted?
You Li
I develop an analysis module consulting cdap-tca-hi-lo? how can I deploy it to DCAE? What files should I provide to DCAE Controller?
Carsten Lund
Adding a new analysis module is not that straightforward and using GEN1 DCAE may not be the right direction at this time. I suggest you reach out to the DCAE project and target DCAEGEN2.
Victor Morales
Hi,
I was thinking to update the ONAP on Vagrant script for DCAE[1] to make something that could be useful for setting your development environment. Can someone validate if all the steps and repositories are covered in that script? The goal is to provision a VM with all the dependencies installed and sharing the source code with the host, so developers can test their functionally in the VM and keep ssh keys and IDE in the host.
[1] https://git.onap.org/integration/tree/bootstrap/vagrant-onap/lib/dcae