OOM_PlatformComponent_StartStopDelete Configbindingservice 9) docker stop ubuntu@vv-k8-2:~/scripts$ sudo docker ps | grep config sudo: unable to resolve host vv-k8-2 d8038ae2495a docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942 "filebeat -e" About an hour ago Up About an hour k8s_filebeat_dep-config-binding-service-759777c5dd-vtxnf_onap_e60b95f6-5614-11e8-b6e9-0290d8b003c6_0 1d535c4107f9 nexus3.onap.org:10001/onap/org.onap.dcaegen2.platform.configbinding@sha256:11f8aeecfbf6b0088b391701dbc4fef8048d2b8daa124549d9f3b639b8245f54 "/bin/sh -c run.py" About an hour ago Up About an hour k8s_config-binding-service_dep-config-binding-service-759777c5dd-vtxnf_onap_e60b95f6-5614-11e8-b6e9-0290d8b003c6_0 9f701b384262 gcr.io/google_containers/pause-amd64:3.0 "/pause" About an hour ago Up About an hour k8s_POD_dep-config-binding-service-759777c5dd-vtxnf_onap_e60b95f6-5614-11e8-b6e9-0290d8b003c6_0 ec64b86c063d gcr.io/google_containers/heapster-influxdb-amd64@sha256:f433e331c1865ad87bc5387589965528b78cd6b1b2f61697e589584d690c1edd "influxd --config /et" 2 weeks ago Up 2 weeks k8s_influxdb_monitoring-influxdb-56fdcd96b-gvx7b_kube-system_e94b7994-216e-11e8-b79b-0290d8b003c6_1 ubuntu@vv-k8-2:~/scripts$ sudo docker stop 1d535c4107f9 sudo: unable to resolve host vv-k8-2 1d535c4107f9 ubuntu@vv-k8-2:~/scripts$ sudo docker ps | grep config sudo: unable to resolve host vv-k8-2 78f9b6aaca83 nexus3.onap.org:10001/onap/org.onap.dcaegen2.platform.configbinding@sha256:11f8aeecfbf6b0088b391701dbc4fef8048d2b8daa124549d9f3b639b8245f54 "/bin/sh -c run.py" 9 seconds ago Up 9 seconds k8s_config-binding-service_dep-config-binding-service-759777c5dd-vtxnf_onap_e60b95f6-5614-11e8-b6e9-0290d8b003c6_1 d8038ae2495a docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942 "filebeat -e" About an hour ago Up About an hour k8s_filebeat_dep-config-binding-service-759777c5dd-vtxnf_onap_e60b95f6-5614-11e8-b6e9-0290d8b003c6_0 9f701b384262 gcr.io/google_containers/pause-amd64:3.0 "/pause" About an hour ago Up About an hour k8s_POD_dep-config-binding-service-759777c5dd-vtxnf_onap_e60b95f6-5614-11e8-b6e9-0290d8b003c6_0 ec64b86c063d gcr.io/google_containers/heapster-influxdb-amd64@sha256:f433e331c1865ad87bc5387589965528b78cd6b1b2f61697e589584d690c1edd "influxd --config /et" 2 weeks ago Up 2 weeks k8s_influxdb_monitoring-influxdb-56fdcd96b-gvx7b_kube-system_e94b7994-216e-11e8-b79b-0290d8b003c6_1 root@dcae-ves-collector:~# curl config-binding-service:10000/service_component_all/dcae-tca-analytics {"config": {"app_config": {"tcaVESAlertsTableTTLSeconds": "1728000", "thresholdCalculatorFlowletInstances": "2", "tcaAlertsAbatementTableTTLSeconds": "1728000", "tcaAlertsAbatementTableName": "TCAAlertsAbatementTable", "appName": "dcae-tca", "appDescription": "DCAE Analytics Threshold Crossing Alert Application", "tcaSubscriberOutputStreamName": "TCASubscriberOutputStream", "tcaVESMessageStatusTableTTLSeconds": "86400", "tcaVESAlertsTableName": "TCAVESAlertsTable", "tcaVESMessageStatusTableName": "TCAVESMessageStatusTable"}, "app_preferences": {"aaiEnrichmentIgnoreSSLCertificateErrors": "true", "publisherMaxRecoveryQueueSize": "100000", "subscriberConsumerId": "c12", "aaiVNFEnrichmentAPIPath": "/aai/v11/network/generic-vnfs/generic-vnf", "aaiEnrichmentUserPassword": "DCAE", "publisherHostName": "message-router.onap", "aaiEnrichmentPortNumber": "8443", "publisherPollingInterval": "20000", "subscriberProtocol": "http", "aaiEnrichmentHost": "aai.onap.svc.cluster.local", "enableRedisCaching": false, "subscriberContentType": "application/json", "subscriberHostName": "message-router.onap", "redisHosts": "dcae-redis.onap.svc.cluster.local:6379", "publisherContentType": "application/json", "subscriberTopicName": "unauthenticated.VES_MEASUREMENT_OUTPUT", "aaiVMEnrichmentAPIPath": "/aai/v11/search/nodes-query", "publisherMaxBatchSize": "1", "publisherProtocol": "http", "publisherHostPort": "3904", "subscriberMessageLimit": "-1", "tca_policy": "{\"domain\":\"measurementsForVfScaling\",\"metricsPerEventName\":[{\"eventName\":\"vFirewallBroadcastPackets\",\"controlLoopSchemaType\":\"VNF\",\"policyScope\":\"DCAE\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyVersion\":\"v0.0.1\",\"thresholds\":[{\"closedLoopControlName\":\"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\",\"thresholdValue\":300,\"direction\":\"LESS_OR_EQUAL\",\"severity\":\"MAJOR\",\"closedLoopEventStatus\":\"ONSET\"},{\"closedLoopControlName\":\"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\",\"thresholdValue\":700,\"direction\":\"GREATER_OR_EQUAL\",\"severity\":\"CRITICAL\",\"closedLoopEventStatus\":\"ONSET\"}]},{\"eventName\":\"vLoadBalancer\",\"controlLoopSchemaType\":\"VM\",\"policyScope\":\"DCAE\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyVersion\":\"v0.0.1\",\"thresholds\":[{\"closedLoopControlName\":\"ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\",\"thresholdValue\":300,\"direction\":\"GREATER_OR_EQUAL\",\"severity\":\"CRITICAL\",\"closedLoopEventStatus\":\"ONSET\"}]},{\"eventName\":\"Measurement_vGMUX\",\"controlLoopSchemaType\":\"VNF\",\"policyScope\":\"DCAE\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyVersion\":\"v0.0.1\",\"thresholds\":[{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"EQUAL\",\"severity\":\"MAJOR\",\"closedLoopEventStatus\":\"ABATED\"},{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"GREATER\",\"severity\":\"CRITICAL\",\"closedLoopEventStatus\":\"ONSET\"}]}]}", "enableAAIEnrichment": true, "publisherTopicName": "unauthenticated.DCAE_CL_OUTPUT", "subscriberPollingInterval": "30000", "aaiEnrichmentProtocol": "https", "enableAlertCEFFormat": "false", "subscriberConsumerGroup": "OpenDCAE-c12", "subscriberHostPort": "3904", "aaiEnrichmentUserName": "DCAE", "subscriberTimeoutMS": "-1"}}}"3904", "aaiEnrichmentUserName": "DCAE", "subscriberTimeoutMS": "-1"}}} root@dcae-ves-collector:~# curl dcae-healthcheck -analytics {"type":"summary","count":8,"ready":8,"items":[{"name":"dev-dcae-cloudify-manager","ready":1,"unavailable":0},{"name":"dep-config-binding-service","ready":1,"unavailable":0},{"name":"dep-deployment-handler","ready":1,"unavailable":0},{"name":"dep-inventory","ready":1,"unavailable":0},{"name":"dep-service-change-handler","ready":1,"unavailable":0},{"name":"dep-policy-handler","ready":1,"unavailable":0},{"name":"dep-dcae-ves-collector","ready":1,"unavailable":0},{"name":"dep-dcae-tca-analytics","ready":1,"unavailable":0}]} 10) POD delete ubuntu@vv-k8-2:~/scripts$ kubectl delete pod dep-config-binding-service-759777c5dd-vtxnf -n onap pod "dep-config-binding-service-759777c5dd-vtxnf" deleted ubuntu@vv-k8-2:~/scripts$ kubectl get pod -n onap | grep config dep-config-binding-service-759777c5dd-9xxks 2/2 Running 0 26s dep-config-binding-service-759777c5dd-vtxnf 2/2 Terminating 1 1h root@dcae-ves-collector:~# curl dcae-healthcheck {"type":"summary","count":8,"ready":8,"items":[{"name":"dev-dcae-cloudify-manager","ready":1,"unavailable":0},{"name":"dep-config-binding-service","ready":1,"unavailable":0},{"name":"dep-deployment-handler","ready":1,"unavailable":0},{"name":"dep-inventory","ready":1,"unavailable":0},{"name":"dep-service-change-handler","ready":1,"unavailable":0},{"name":"dep-policy-handler","ready":1,"unavailable":0},{"name":"dep-dcae-ves-collector","ready":1,"unavailable":0},{"name":"dep-dcae-tca-analytics","ready":1,"unavailable":0}]} Policy Handler 15) POD delete ubuntu@vv-k8-2:~/scripts$ kubectl delete pod dep-policy-handler-8688988b99-2bfgk -n onap pod "dep-policy-handler-8688988b99-2bfgk" deleted ubuntu@vv-k8-2:~/scripts$ kubectl get pod -n onap | grep policy-handler dep-policy-handler-8688988b99-f5nbr 0/2 ContainerCreating 0 3s ubuntu@vv-k8-2:~/scripts$ ubuntu@vv-k8-2:~/scripts$ kubectl get pod -n onap | grep policy-handler dep-policy-handler-8688988b99-f5nbr 2/2 Running 0 49s ubuntu@vv-k8-2:~/oom$ kubectl logs -f dep-policy-handler-8688988b99-2bfgk -n onap -c policy-handler APP_VER=2.4.3 /etc/hosts # Kubernetes-managed hosts file. 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet fe00::0 ip6-mcastprefix fe00::1 ip6-allnodes fe00::2 ip6-allrouters 10.42.41.29 policy-handler running policy_handler as 16 log logs/policy_handler.log killing policy_handler 16 2018_0512-20:57:00.235529490 killed policy_handler 16 2018_0512-20:57:00.240271459 ./run_policy.sh: line 42: 16 Killed python -m policyhandler/policy_handler >> ${LOG_FILE} 2>&1 rpc error: code = Unknown desc = Error: No such container: 97db8cb76607f56b497a1142655ff43f172269aecedb70326a08ba3a83188928ubuntu@vv-k8-2:~/oom$ ubuntu@vv-k8-2:~/oom$ ubuntu@vv-k8-2:~/oom$ kubectl logs -f dep-policy-handler-8688988b99-f5nbr -n onap -c policy-handler APP_VER=2.4.3 /etc/hosts # Kubernetes-managed hosts file. 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet fe00::0 ip6-mcastprefix fe00::1 ip6-allnodes fe00::2 ip6-allrouters 10.42.94.214 policy-handler running policy_handler as 16 log logs/policy_handler.log root@dcae-ves-collector:~# curl dcae-healthcheck {"type":"summary","count":8,"ready":8,"items":[{"name":"dev-dcae-cloudify-manager","ready":1,"unavailable":0},{"name":"dep-config-binding-service","ready":1,"unavailable":0},{"name":"dep-deployment-handler","ready":1,"unavailable":0},{"name":"dep-inventory","ready":1,"unavailable":0},{"name":"dep-service-change-handler","ready":1,"unavailable":0},{"name":"dep-policy-handler","ready":1,"unavailable":0},{"name":"dep-dcae-ves-collector","ready":1,"unavailable":0},{"name":"dep-dcae-tca-analytics","ready":1,"unavailable":0}]} Inventory API 18) Delete POD ubuntu@vv-k8-2:~/scripts$ kubectl delete pod dep-inventory-769bbfb69d-2rjnb -n onap pod "dep-inventory-769bbfb69d-2rjnb" deleted ubuntu@vv-k8-2:~/scripts$ kubectl get pod -n onap | grep inventory dep-inventory-769bbfb69d-2rjnb 0/1 Terminating 0 2h dep-inventory-769bbfb69d-zq7bs 1/1 Running 0 7s ubuntu@vv-k8-2:~/scripts$ kubectl get pod -n onap | grep inventory dep-inventory-769bbfb69d-zq7bs 1/1 Running 0 3m root@dcae-ves-collector:~# curl dcae-healthcheck {"type":"summary","count":8,"ready":8,"items":[{"name":"dev-dcae-cloudify-manager","ready":1,"unavailable":0},{"name":"dep-config-binding-service","ready":1,"unavailable":0},{"name":"dep-deployment-handler","ready":1,"unavailable":0},{"name":"dep-inventory","ready":1,"unavailable":0},{"name":"dep-service-change-handler","ready":1,"unavailable":0},{"name":"dep-policy-handler","ready":1,"unavailable":0},{"name":"dep-dcae-ves-collector","ready":1,"unavailable":0},{"name":"dep-dcae-tca-analytics","ready":1,"unavailable":0}] ---Deployment Handler 21) Delete pod ubuntu@vv-k8-2:~/scripts$ kubectl get pods -n onap | grep deployment dep-deployment-handler-6cc56bcc95-nxs4c 2/2 Running 0 2h ubuntu@vv-k8-2:~/scripts$ kubectl delete pod dep-deployment-handler-6cc56bcc95-nxs4c -n onap pod "dep-deployment-handler-6cc56bcc95-nxs4c" deleted ubuntu@vv-k8-2:~/scripts$ kubectl get pods -n onap | grep deployment dep-deployment-handler-6cc56bcc95-nf7ft 2/2 Running 0 27s ubuntu@vv-k8-2:~/oom$ kubectl logs -f dep-deployment-handler-6cc56bcc95-nxs4c -c deployment-handler -n onap rpc error: code = Unknown desc = Error: No such container: 97d3b734bd0ef373ced84067a616ca4ac81c6d3ad064ca31f787a3af101a2b0eubuntu@vv-k8-2:~/oom$ ubuntu@vv-k8-2:~/oom$ curl dcae-healthcheck {"type":"summary","count":8,"ready":7,"items":[{"name":"dev-dcae-cloudify-manager","ready":1,"unavailable":0},{"name":"dep-config-binding-service","ready":1,"unavailable":0},{"name":"dep-deployment-handler","ready":0,"unavailable":1},{"name":"dep-inventory","ready":1,"unavailable":0},{"name":"dep-service-change-handler","ready":1,"unavailable":0},{"name":"dep-policy-handler","ready":1,"unavailable":0},{"name":"dep-dcae-ves-collector","ready":1,"unavailable":0},{"name":"dep-dcae-tca-analytics","ready":1,"unavailable":0}]}root@dcae-ves-collector:~# root@dcae-ves-collector:~# curl dcae-healthcheck {"type":"summary","count":8,"ready":8,"items":[{"name":"dev-dcae-cloudify-manager","ready":1,"unavailable":0},{"name":"dep-config-binding-service","ready":1,"unavailable":0},{"name":"dep-deployment-handler","ready":1,"unavailable":0},{"name":"dep-inventory","ready":1,"unavailable":0},{"name":"dep-service-change-handler","ready":1,"unavailable":0},{"name":"dep-policy-handler","ready":1,"unavailable":0},{"name":"dep-dcae-ves-collector","ready":1,"unavailable":0},{"name":"dep-dcae-tca-analytics","ready":1,"unavailable":0}]} SCH 24) POD delete ubuntu@vv-k8-2:~/scripts$ kubectl delete pod dep-service-change-handler-777679588f-6w8bl -n onap pod "dep-service-change-handler-777679588f-6w8bl" deleted ubuntu@vv-k8-2:~/scripts$ kubectl get pods -n onap | grep change dep-service-change-handler-777679588f-sx679 1/1 Running 0 8s Logs okay root@dcae-ves-collector:~# curl dcae-healthcheck {"type":"summary","count":8,"ready":8,"items":[{"name":"dev-dcae-cloudify-manager","ready":1,"unavailable":0},{"name":"dep-config-binding-service","ready":1,"unavailable":0},{"name":"dep-deployment-handler","ready":1,"unavailable":0},{"name":"dep-inventory","ready":1,"unavailable":0},{"name":"dep-service-change-handler","ready":1,"unavailable":0},{"name":"dep-policy-handler","ready":1,"unavailable":0},{"name":"dep-dcae-ves-collector","ready":1,"unavailable":0},{"name":"dep-dcae-tca-analytics","ready":1,"unavailable":0}]} EVENT FLOW vv770d@pod-onap-01-vjhost:~$ curl -i -u 'sample1:sample1' -X POST -d @f5.txt --header "Content-Type: application/json" http://10.12.5.102:30235/eventListener/v5 -k HTTP/1.1 200 X-Rathravane: ~ software is craft ~ Content-Type: application/json;charset=ISO-8859-1 Content-Length: 17 Date: Sat, 12 May 2018 22:07:45 GMT Message Accepted VES LOGS [22:07:45,676][INFO ][pool-1-thread-1]pub.send invoked - no error [22:07:45,677][INFO ][pool-1-thread-1]StreamID:ves_measurement Event Published:{"event":{"commonEventHeader":{"startEpochMicrosec":1507750704126521,"sourceId":"4c32fa09-83e7-43cf-8bba-c72557afe408","eventId":"TrafficStats_1.2.3.4","nfcNamingCode":"vVNF","reportingEntityId":"No UUID available","internalHeaderFields":{"collectorTimeStamp":"Sat, 05 12 2018 10:07:45 GMT"},"eventType":"HTTP request rate","priority":"Normal","version":1.2,"reportingEntityName":"fwll","sequence":85,"domain":"measurementsForVfScaling","lastEpochMicrosec":1507750714317473,"eventName":"vFirewallBroadcastPackets","sourceName":"zdfw1fwl01fwl01","nfNamingCode":"vVNF"},"measurementsForVfScalingFields":{"cpuUsageArray":[{"percentUsage":0,"cpuIdentifier":"cpu1","cpuIdle":100,"cpuUsageSystem":0,"cpuUsageUser":0}],"measurementInterval":10,"requestRate":2256,"vNicUsageArray":[{"transmittedOctetsDelta":0,"receivedTotalPacketsDelta":1002,"vNicIdentifier":"eth0","valuesAreSuspect":"true","transmittedTotalPacketsDelta":0,"receivedOctetsDelta":61200}],"measurementsForVfScalingVersion":2.1}}} [22:07:45,777][INFO ][pool-4-thread-1]sending 1 msgs to /events/unauthenticated.VES_MEASUREMENT_OUTPUT. Oldest: 99 ms [22:07:45,782][INFO ][pool-4-thread-1]POST http://message-router:3904/events/unauthenticated.VES_MEASUREMENT_OUTPUT (anonymous) ... [22:07:46,267][INFO ][pool-4-thread-1] --> HTTP/1.1 200 OK [22:07:46,268][INFO ][pool-4-thread-1]cambria reply ok (489 ms):{"serverTimeMs":346,"count":1} TCA Logs /2018 18:16:23 DEBUG TCA_DMaaPMRPublisher_Scheduler_Worker-1:org.onap.dcae.apod.analytics.dmaap.service.publisher.DMaaPMRPublisherImpl#119 DMaaP MR Publisher Authentication is disabled as username or password is not present. 05/12/2018 18:16:23 DEBUG TCA_DMaaPMRPublisher_Scheduler_Worker-1:org.onap.dcae.apod.analytics.dmaap.service.publisher.DMaaPMRPublisherImpl#134 DMaaP MR Messages published successfully. DMaaP Response Code: 200. DMaaP Response Body: { "serverTimeMs": 4, "count": 1 }, Number of Messages published: 1 05/12/2018 18:16:23 DEBUG TCA_DMaaPMRPublisher_Scheduler_Worker-1:org.onap.dcae.apod.analytics.cdap.tca.worker.TCADMaaPMRPublisherJob#119 Publisher Response Code: 200, Publisher message: { "serverTimeMs": 4, "count": 1 }, Pending Messages Count: 0 05/12/2018 18:16:23 DEBUG TCA_DMaaPMRPublisher_Scheduler_Worker-1:org.onap.dcae.apod.analytics.cdap.tca.worker.TCADMaaPMRPublisherJob#123 Successfully Published alerts to DMaaP MR Topic. 05/12/2018 18:16:23 DEBUG TCA_DMaaPMRPublisher_Scheduler_Worker-1:org.onap.dcae.apod.analytics.cdap.tca.worker.TCADMaaPMRPublisherJob#182 Deleting Published Alerts from alerts table with rowKeys: 0000000000001526163382845 05/12/2018 18:16:23 DEBUG TCA_DMaaPMRPublisher_Scheduler_Worker-1:org.onap.dcae.apod.analytics.cdap.tca.worker.TCADMaaPMRPublisherJob#137 Finished DMaaP MR Topic Publisher fetch Job. root@dcae-tca-analytics:/# root@dcae-tca-analytics:/# curl http://message-router.onap:3904/events/unauthenticated.DCAE_CL_OUTPUT/vv/1 []root@dcae-tca-analytics:/# curl http://message-router.onap:3904/events/unauthenticated.DCAE_CL_OUTPUT/vv/1 ["{\"closedLoopEventClient\":\"DCAE_INSTANCE_ID.dcae-tca\",\"policyVersion\":\"v0.0.1\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyScope\":\"DCAE\",\"target_type\":\"VNF\",\"AAI\":{\"generic-vnf.vnf-name\":\"zdfw1fwl01fwl01\"},\"closedLoopAlarmStart\":1507750704126521,\"closedLoopEventStatus\":\"ONSET\",\"closedLoopControlName\":\"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",\"version\":\"1.0.2\",\"target\":\"generic-vnf.vnf-name\",\"requestID\":\"7341e37c-3e87-4ec9-90d7-96c3ecc2c3f2\",\"from\":\"DCAE\"}"]root@dcae-tca-analy tics:/#