The below steps covers manual setup of DCAE VM’s and DCAE service components.
VESCollector
DCAE VES Collector can be configured on VM with ubuntu-16.04 image (m1.small should suffice if this is only service) and 20Gb cinder storage
1) Install docker
sudo apt-get update
sudo apt install docker.io
2) Pull the latest container from onap nexus
sudo docker login -u docker -p docker nexus3.onap.org:10001
sudo docker pull nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:v1.1.0
3) Start the VESCollector with below command
sudo docker run -d --name vescollector -p 8080:8080/tcp -p 8443:8443/tcp -P -e DMAAPHOST='<dmaap IP>' nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:v1.1.0
Note: Change the dmaaphost to required DMAAP ip. To change the dmaap information for a running container, stop the active container and rerun above command changing the dmaap IP.
4) Verification
- Check logs under container /opt/app/VESCollector/logs/collector.log for errors
- If no active feed, you can simulate an event into collector via curl
curl -i -X POST -d @<sampleves> --header "Content-Type: application/json" http://localhost:8080/eventListener/v5 -k
Note: If DMAAPHOST provided is invalid, you will see exception around publish on the collector.logs (collector queues and attempts to resend the event hence exceptions reported will be periodic).
3. Below two topic configuration are pre-set into this container. When valid DMAAP instance ip was provided and VES events are received, the collector will post to below topics.
Fault - http://<dmaaphost>:3904/events/unauthenticated.SEC_FAULT_OUTPUT
Measurement -http://<dmaaphost>:3904/events/unauthenticated.SEC_MEASUREMENT_OUTPUT
VM Init
To address windriver server in-stability, the below init.sh script was used to start the container on VM restart.
#!/bin/sh
sudo docker ps | grep "vescollector"
if [ $? -ne 0 ]; then
sudo docker login -u docker -p docker nexus3.onap.org:10001
sudo docker pull nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:1.1
sudo docker rm -f vescollector
echo "Collector process not running - $(date)" >> /home/ubuntu/startuplog
sudo docker run -d --name vescollector -p 8080:8080/tcp -p 8443:8443/tcp -P -e DMAAPHOST='10.12.25.96' nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:v1.1.0
else
echo "Collector process running - $(date)" >> /home/ubuntu/startuplog
fi
This script was invoked via VM init script (rc.d).
ln -s /home/ubuntu/init.sh /etc/init.d/init.sh
sudo update-rc.d init.sh start 2
ThresholdCrossingAnalysis (TCA/CDAP)
The platform deploys CDAP as cluster and instantiates TCA. For the manual setup, we will leverage the CDAP SDK docker container to deploy TCA instances. To setup TCA, choose VM with ubuntu-16.04 image, m1.medium size and 50gb cinder volumes.
1) Install docker
sudo apt-get update
sudo apt install docker.io
2) Pull CDAP SDK container
sudo docker pull caskdata/cdap-standalone:4.1.2
3) Deploy and run the CDAP container
sudo docker run -d --name cdap-sdk-2 -p 11011:11011 -p 11015:11015 caskdata/cdap-standalone:4.1.2
4) Create Namespace on CDAP application
curl -X PUT http://localhost:11015/v3/namespaces/cdap_tca_hi_lo
5) Create TCA app config file - "tca_app_config.json" under ~ubuntu as below
{
"artifact": {
"name": "dcae-analytics-cdap-tca",
"version": "2.0.1",
"scope": "user"
},
"config": {
"appName": "dcae-tca",
"appDescription": "DCAE Analytics Threshold Crossing Alert Application",
"tcaVESMessageStatusTableName": "TCAVESMessageStatusTable",
"tcaVESMessageStatusTableTTLSeconds": 86400.0,
"tcaAlertsAbatementTableName": "TCAAlertsAbatementTable",
"tcaAlertsAbatementTableTTLSeconds": 1728000.0,
"tcaVESAlertsTableName": "TCAVESAlertsTable",
"tcaVESAlertsTableTTLSeconds": 1728000.0,
"thresholdCalculatorFlowletInstances": 2.0,
"tcaSubscriberOutputStreamName": "TCASubscriberOutputStream"
}
}
6) Create TCA app preference file - "tca_app_preferences.json" under ~ubuntu as below
{
"publisherContentType" : "application/json",
"publisherHostName" : "10.12.25.96",
"publisherHostPort" : "3904",
"publisherMaxBatchSize" : "1",
"publisherMaxRecoveryQueueSize" : "100000",
"publisherPollingInterval" : "20000",
"publisherProtocol" : "http",
"publisherTopicName" : "unauthenticated.DCAE_CL_OUTPUT",
"subscriberConsumerGroup" : "OpenDCAE-c1",
"subscriberConsumerId" : "c1",
"subscriberContentType" : "application/json",
"subscriberHostName" : "10.12.25.96",
"subscriberHostPort" : "3904",
"subscriberMessageLimit" : "-1",
"subscriberPollingInterval" : "20000",
"subscriberProtocol" : "http",
"subscriberTimeoutMS" : "-1",
"subscriberTopicName" : "unauthenticated.SEC_MEASUREMENT_OUTPUT",
"enableAAIEnrichment" : false,
"aaiEnrichmentHost" : "10.12.25.72",
"aaiEnrichmentPortNumber" : 8443,
"aaiEnrichmentProtocol" : "https",
"aaiEnrichmentUserName" : "DCAE",
"aaiEnrichmentUserPassword" : "DCAE",
"aaiEnrichmentIgnoreSSLCertificateErrors" : true,
"aaiVNFEnrichmentAPIPath" : "/aai/v11/network/generic-vnfs/generic-vnf",
"aaiVMEnrichmentAPIPath" : "/aai/v11/search/nodes-query",
"tca_policy" : "{
\"domain\": \"measurementsForVfScaling\",
\"metricsPerEventName\": [{
\"eventName\": \"vFirewallBroadcastPackets\",
\"controlLoopSchemaType\": \"VNF\",
\"policyScope\": \"DCAE\",
\"policyName\": \"DCAE.Config_tca-hi-lo\",
\"policyVersion\": \"v0.0.1\",
\"thresholds\": [{
\"closedLoopControlName\": \"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",
\"version\": \"1.0.2\",
\"fieldPath\": \"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\",
\"thresholdValue\": 300,
\"direction\": \"LESS_OR_EQUAL\",
\"severity\": \"MAJOR\",
\"closedLoopEventStatus\": \"ONSET\"
}, {
\"closedLoopControlName\": \"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",
\"version\": \"1.0.2\",
\"fieldPath\": \"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\",
\"thresholdValue\": 700,
\"direction\": \"GREATER_OR_EQUAL\",
\"severity\": \"CRITICAL\",
\"closedLoopEventStatus\": \"ONSET\"
}]
}, {
\"eventName\": \"vLoadBalancer\",
\"controlLoopSchemaType\": \"VM\",
\"policyScope\": \"DCAE\",
\"policyName\": \"DCAE.Config_tca-hi-lo\",
\"policyVersion\": \"v0.0.1\",
\"thresholds\": [{
\"closedLoopControlName\": \"ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3\",
\"version\": \"1.0.2\",
\"fieldPath\": \"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\",
\"thresholdValue\": 300,
\"direction\": \"GREATER_OR_EQUAL\",
\"severity\": \"CRITICAL\",
\"closedLoopEventStatus\": \"ONSET\"
}]
}, {
\"eventName\": \"Measurement_vGMUX\",
\"controlLoopSchemaType\": \"VNF\",
\"policyScope\": \"DCAE\",
\"policyName\": \"DCAE.Config_tca-hi-lo\",
\"policyVersion\": \"v0.0.1\",
\"thresholds\": [{
\"closedLoopControlName\": \"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",
\"version\": \"1.0.2\",
\"fieldPath\": \"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",
\"thresholdValue\": 0,
\"direction\": \"EQUAL\",
\"severity\": \"MAJOR\",
\"closedLoopEventStatus\": \"ABATED\"
}, {
\"closedLoopControlName\": \"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",
\"version\": \"1.0.2\",
\"fieldPath\": \"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",
\"thresholdValue\": 0,
\"direction\": \"GREATER\",
\"severity\": \"CRITICAL\",
\"closedLoopEventStatus\": \"ONSET\"
}]
}]
}"
}
Note: Dmaap configuration are specified on this file on publisherHostName and subscriberHostName. To be changed as required**
6 a) To enable AAI Enrichment in TCA for CL event, following updates should be done to TCA app preference file before proceeding
"enableAAIEnrichment" : true
"aaiEnrichmentHost" : <should be set to aai1_ip_addr>
"aaiEnrichmentIgnoreSSLCertificateErrors" : true
7) Copy below script to CDAP server (this gets latest image from nexus and deploys TCA application) and execute it
#!/bin/sh
TCA_JAR=dcae-analytics-cdap-tca-2.0.0.jar
rm -f /home/ubuntu/$TCA_JAR
cd /home/ubuntu/
wget https://nexus3.onap.org/service/local/repositories/releases/content/org/onap/dcaegen2/analytics/tca/dcae-analytics-cdap-tca/2.0.0/$TCA_JAR
if [ $? -eq 0 ]; then
if [ -f /home/ubuntu/$TCA_JAR ]; then
echo "Restarting TCA CDAP application using $TCA_JAR artifact"
else
echo "ERROR: $TCA_JAR missing"
exit 1
fi
else
echo "ERROR: $TCA_JAR not found in nexus"
exit 1
fi
# stop programs
curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/workers/TCADMaaPMRPublisherWorker/stop
curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/workers/TCADMaaPMRSubscriberWorker/stop
curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/flows/TCAVESCollectorFlow/stop
# delete application
curl -X DELETE http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca
# delete artifact
curl -X DELETE http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/artifacts/dcae-analytics-cdap-tca/versions/2.0.1
# load artifact
curl -X POST --data-binary @/home/ubuntu/$TCA_JAR http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/artifacts/dcae-analytics-cdap-tca
# create app
curl -X PUT -d @/home/ubuntu/tca_app_config.json http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca
# load preferences
curl -X PUT -d @/home/ubuntu/tca_app_preferences.json http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/preferences
# start programs
curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/workers/TCADMaaPMRPublisherWorker/start
curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/workers/TCADMaaPMRSubscriberWorker/start
curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/flows/TCAVESCollectorFlow/start
echo
# get status of programs
curl http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/workers/TCADMaaPMRPublisherWorker/status
curl http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/workers/TCADMaaPMRSubscriberWorker/status
curl http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/flows/TCAVESCollectorFlow/status
echo
Note: Attached the script also for reference : reload-tca.sh
8) Verify TCA application and logs via CDAP GUI processes
http://<hostname/ip>:11011/oldcdap/ns/cdap_tca_hi_lo/apps/dcae-tca/overview/programs
The overall flow can be checked here
http://<hostname/ip>:11011/oldcdap/ns/cdap_tca_hi_lo/apps/dcae-tca/programs/flows/TCAVESCollectorFlow/runs
TCA Configuration Change
Typical configuration changes include changing DMAAP host and/or Policy configuration. If necessary, modify the file on step #6 and run the script noted as step #7 to redeploy TCA with updated configuration.
VM Init
To address windriver server in-stability, the below init.sh script was used to restart the container on VM restart. This script was invoked via VM init script (rc.d).
#!/bin/sh
#docker run -d --name cdap-sdk -p 11011:11011 -p 11015:11015 caskdata/cdap-standalone:4.1.2
sudo docker restart cdap-sdk-2
sleep 30
# start program
curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/workers/TCADMaaPMRPublisherWorker/start
curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/workers/TCADMaaPMRSubscriberWorker/start
curl -X POST http://localhost:11015/v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/flows/TCAVESCollectorFlow/start
This script was invoked via VM init script (rc.d).
ln -s /home/ubuntu/init.sh /etc/init.d/init.sh
sudo update-rc.d init.sh start 2
22 Comments
Evguenia (Jane) Freider
Thanks a lot for these instructions, wonderful! I was able to install VES without issues. However step 6 fails while trying to install TCA application.
1) I had to rename tca_app_preference.json to tca_app_preferences.json as the script wanted. So you should update the wiki.
2) I had to update reload-tca.sh and fix every #comments after curl commands, they have to be on a new line, shell doesn't recognize the comments at the end of lines.
After these fixes it gets better however I get this error below:
Restarting TCA CDAP application using dcae-analytics-cdap-tca-2.0.0.jar artifact
'application:cdap_tca_hi_lo.dcae-tca.-SNAPSHOT' was not found.'application:cdap_tca_hi_lo.dcae-tca.-SNAPSHOT' was not found.'application:cdap_tca_hi_lo.dcae-tca.-SNAPSHOT' was not found.'application:cdap_tca_hi_lo.dcae-tca.-SNAPSHOT' was not found.'artifact:cdap_tca_hi_lo.dcae-analytics-cdap-tca.2.0.0' was not found.Artifact artifact:cdap_tca_hi_lo.dcae-analytics-cdap-tcaecho.2.0.0 already exists.Warning: Couldn't read data from file "/home/ubuntu/tca_app_config.json", this
Warning: makes an empty POST.
Application dcae-tca in Namespace cdap_tca_hi_lo not present'application:cdap_tca_hi_lo.dcae-tca.-SNAPSHOT' was not found.'application:cdap_tca_hi_lo.dcae-tca.-SNAPSHOT' was not found.startecho action was not found'application:cdap_tca_hi_lo.dcae-tca.-SNAPSHOT' was not found.'application:cdap_tca_hi_lo.dcae-tca.-SNAPSHOT' was not found./v3/namespaces/cdap
Could you please help?
Vijay Venkatesh Kumar
Thank you for validating the steps and providing feedback! There were few glitches in the documentation as you noted.
#1 – Yes, file name should be tca_app_preferences.json
#2 – This was copy paste error into confluence, missed to notice. Thanks for pointing it.
I’ve updated the TCA steps with the corrections - included information about tca app config file sample which is required as well. Please delete your container and repeat from steps #3 and let me know if you it helps.
Kang Xi
Hi Vijay Venkatesh Kumar,
Just tried the updated steps but the problem persists. Would you please take a look? Thanks.
2017-10-22 21:25:36 (5.67 MB/s) - ‘dcae-analytics-cdap-tca-2.0.0.jar’ saved [8645185/8645185]
Restarting TCA CDAP application using dcae-analytics-cdap-tca-2.0.0.jar artifact
'application:cdap_tca_hi_lo.dcae-tca.-SNAPSHOT' was not found.'application:cdap_tca_hi_lo.dcae-tca.-SNAPSHOT' was not found.'application:cdap_tca_hi_lo.dcae-tca.-SNAPSHOT' was not found.'application:cdap_tca_hi_lo.dcae-tca.-SNAPSHOT' was not found.curl: (6) Could not resolve host: delete
curl: (6) Could not resolve host: artifact
'artifact:cdap_tca_hi_lo.dcae-analytics-cdap-tca.2.0.0' was not found.Artifact added successfully'artifact:cdap_tca_hi_lo.dcae-analytics-cdap-tca.2.0.0' was not found.curl: (6) Could not resolve host: load
curl: (6) Could not resolve host: preferences
Application dcae-tca in Namespace cdap_tca_hi_lo not present'application:cdap_tca_hi_lo.dcae-tca.-SNAPSHOT' was not found.'application:cdap_tca_hi_lo.dcae-tca.-SNAPSHOT' was not found.startecho action was not found'application:cdap_tca_hi_lo.dcae-tca.-SNAPSHOT' was not found.'application:cdap_tca_hi_lo.dcae-tca.-SNAPSHOT' was not found./v3/namespaces/cdap_tca_hi_lo/apps/dcae-tca/flows/TCAVESCollectorFlow/statusecho%
Vijay Venkatesh Kumar
Hi Kang - The reload-tca.sh when pasted into confluence took out (again) few line break causing the error you noticed. I've fixed them. Pls remove the container and retry from step#3.
Also when the script runs first time, you will notice below error - which is okay (this is bcoz the script tries to stop/undeploy application which doesn't exist on first run).
The last status of the script should indicate something like below - which indicates TCA is operational
Kang Xi
I still see the same error and couldn't get the final RUNNING status. Do you want to upload all those script files instead of copy/paste the content? Those mistakes are hard to catch. After you upload all files I'll try again. Thanks.
Vijay Venkatesh Kumar
Kang - I've attached script as file under step#7; pls check if that helps.
Kang Xi
It's working, thanks.
Premnath K Narayanan
I think now the verion for dcae-analytics-cdap-tca 2.1.0 (as of 1st March) and nexus URL is in different directory. That requires right version in reload-tca.sh and also in configuration script. Once done it works awesome, as mentioned in this page. Hopefully DCAE also moves in to K8S environment soon. Looking forward.
Vijay Venkatesh Kumar
Yes, DCAE is moving toward K8S for R2; will have more details in coming weeks regarding the deployment.
Btw, version 2.0.1 TCA jar is R1 artifact (released version) hence the documentation refers to that version.
user-34041
Hi Vijay,
The jar file that works seems to be dcae-analytics-cdap-tca-2.0.0.jar. 2.0.1 seems be unavailable.
Jing Wang
To verify DMaaP installation before starting integration of DCAE, try the following to publish/subscribe events to DMaaP message router.
Bharath Thiruveedula
Hi Vijay Venkatesh Kumar
Does this standalone setup enough for running vFW closed loop? I think the other components like policy handler or so handler listed in MicroServices Onboarding in ONAP are missing here. Do those exists is already part of these two containers?
How does VES collector an CDAP machines we launched here communicate?
am I missing something here?
Vijay Venkatesh Kumar
Hi Bharath - The communication between VES and TCA application deployed here is through DMAAP (the topic name is preset, however the dmaap host has to be supplied as documented in the steps above). From DCAE standpoint, the standalone setup allows event data flow through DCAE to support closed-loop flow. But you will need to ensure rest of ONAP platform and service – vf5/vdns are deployed, to ensure closed loop action is completed.
The policy-handler and service change handler are platform component not involved in actual data-flow (they are necessary when services are dynamically instantiated and configured through CLAMP/Policy). Btw, there are some impending changes to address policy interface gaps in R2; for R1 – we went with static configuration.
Bharath Thiruveedula
Hi Vijay Venkatesh Kumar
Thanks for your reply.
I have already setup all ONAP components(without DCAE) and deployed vFW network service . Now I setup DCAE standalone using this guideline. So to create the closed loop between vFW → DCAE → SO first I have to pass VES collector VM to vFW?
And how can we achieve scaling operations in this closed loop setup. I mean how this standalone setup invokes SO to scale up?
Moreover, in the TCA VM in reload-tca.sh, "dcae-analytics-cdap-tca-2.0.0.jar" is not found. so I changed to "dcae-analytics-cdap-tca-2.0.1.jar" in both reload-tca.sh and tca_app_config.json. Probably we need to update this wiki accordingly?
Vijay Venkatesh Kumar
Bharath - Yes, you are right. The current tca jar version reflects pre-R1 version; I will update it. Thanks for catching it.
Btw, DCAE does not directly interface with SO. The output CL event is picked up by Policy which then communicates with APP-C/VF-C to complete the CL flow. Also for CL to work, AAI enrichment is neccessary for Policy. The APP preference property should be updated to enabled this for TCA in order for CL to work end-end. I will include that on steps.
Bharath Thiruveedula
Premnath K Narayanan, are you behind proxy? If yes then you have to set proxy in docker. And then try:
sudo docker login -u docker -p docker nexus.onap.org:10001
sudo docker pull nexus.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:v1.1.0
The URLs didn't modify.
user-34041
Hi Vijay Venkatesh Kumar,
i had modified the jar file version of the reload_tca script. Since that was mismatching as per the version in the attachment and the version pasted in the wiki.
Thamlur Raju
Hi Vijay Venkatesh Kumar,
The reload-tca.sh file having the 2.0.0 version JAR but it is not working, even i tried with 2.0.1, both the version are not working.
Can you please check it once.
Vijay Venkatesh Kumar
Hi Thamlur Raju, This deployment instruction for standalone was based off Amsterdam release; you can try with version 2.2.1 in nexus (https://nexus.onap.org/content/repositories/releases/org/onap/dcaegen2/analytics/tca/dcae-analytics-cdap-tca/2.2.1/) . Also the R2/R3 deployment procedure uses docker-compose (includes dockerized tca on cdap container) which should be easy to setup. You can find the steps documented here https://onap.readthedocs.io/en/latest/submodules/dcaegen2.git/docs/sections/installation.html
Rakesh Mhapsekar
Hi.. I am getting error " the current directory in invalid" while pulling one of the mS docker image to my local
-H:\>docker login -u docker -p docker nexus3.onap.org:10001
The current directory is invalid.
Rakesh Mhapsekar
i was able to login successfully.. i could figure out the issue..
Rakesh Mhapsekar
Hi, while pulling image for dcae data file getting error,but i was able to download VES collector.
docker pull nexus3.onap.org:10001/ onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server:1.0.5
"docker pull" requires exactly 1 argument.
See 'docker pull --help'.
Usage: docker pull [OPTIONS] NAME[:TAG|@DIGEST]
Pull an image or a repository from a registry