VES-TCA Flow (Jakarta) 1) Send event to VES 2022-04-13 15:10:21.016 INFO 25 [nio-8443-exec-5] o.a.j.l.DirectJDKLog : Initializing Spring DispatcherServlet 'dispatcherServlet' 2022-04-13 15:10:21.018 INFO 25 [nio-8443-exec-5] o.s.w.s.FrameworkServlet : Initializing Servlet 'dispatcherServlet' 2022-04-13 15:10:21.020 INFO 25 [nio-8443-exec-5] o.s.w.s.FrameworkServlet : Completed initialization in 1 ms 2022-04-13 15:10:21.234 WARN 25 [nio-8443-exec-5] o.o.d.s.s.s.h.CustomHeaderUtils : X-MinorVersion is empty or null 2022-04-13 15:10:22.463 INFO 25 [nio-8443-exec-5] o.o.d.c.p.MessageRouterHttpStatusMapper : Successfully send event to MR 2) VES Collector o/p on DMAAP https://10.15.103.181:30226/events/unauthenticated.VES_MEASUREMENT_OUTPUT/vv/1 [ "{\"event\":{\"commonEventHeader\":{\"startEpochMicrosec\":1507750704126521,\"sourceId\":\"4c32fa09-83e7-43cf-8bba-c72557afe408\",\"eventId\":\"TrafficStats_1.2.3.4\",\"nfcNamingCode\":\"vVNF\",\"reportingEntityId\":\"No UUID available\",\"internalHeaderFields\":{\"collectorTimeStamp\":\"Wed, 04 13 2022 03:13:21 GMT\"},\"eventType\":\"HTTP request rate\",\"priority\":\"Normal\",\"version\":1.2,\"reportingEntityName\":\"fwll\",\"sequence\":85,\"domain\":\"measurementsForVfScaling\",\"lastEpochMicrosec\":1507750714317473,\"eventName\":\"vFirewallBroadcastPackets\",\"sourceName\":\"zdfw1fwl01fwl01\",\"nfNamingCode\":\"vVNF\"},\"measurementsForVfScalingFields\":{\"cpuUsageArray\":[{\"percentUsage\":0,\"cpuIdentifier\":\"cpu1\",\"cpuIdle\":100,\"cpuUsageSystem\":0,\"cpuUsageUser\":0}],\"measurementInterval\":10,\"requestRate\":2256,\"measurementsForVfScalingVersion\":2.1,\"vNicPerformanceArray\":[{\"transmittedOctetsDelta\":0,\"receivedTotalPacketsDelta\":100,\"vNicIdentifier\":\"eth0\",\"valuesAreSuspect\":\"true\",\"transmittedTotalPacketsDelta\":0,\"receivedOctetsDelta\":61200}]}}}" ] 3) TCA processing logs d1, Transaction Id: SGXJC, Elapsed Time: 258 ms, Error Message: java.net.UnknownHostException: aai: System error 2022-04-13 15:18:25.913 ERROR 1 --- [ask-scheduler-6] o.o.d.a.w.http.EelfAuditLogInterceptor : Request Id: c01eae77-f5d6-47eb-8277-403700e528d1, Transaction Id: SGXJC, Elapsed Time: 258 ms, Error Message: java.net.UnknownHostException: aai: System error 2022-04-13T15:18:25.930+0000|c01eae77-f5d6-47eb-8277-403700e528d1|task-scheduler-6|dcae-tcagen2|tca-gen2|DCAE-TCA|dcae-tcagen2|ERROR|400|Schema Error|Request id: c01eae77-f5d6-47eb-8277-403700e528d1. Unable to get A&AI enrichment details 2022-04-13 15:18:25.930 ERROR 1 --- [ask-scheduler-6] .o.d.a.t.w.a.TcaAaiEnrichmentServiceImpl : Request id: c01eae77-f5d6-47eb-8277-403700e528d1. Unable to get A&AI enrichment details 2022-04-13T15:18:22.222+0000|2022-04-13T15:18:25.931+0000|c01eae77-f5d6-47eb-8277-403700e528d1|UNKNOWN_INSTANCE_ID|task-scheduler-6||dcae-tcagen2|tca-gen2|COMPLETE|0|||INFO|0|10.42.0.74|3709|dcae-tcagen2||org.onap.dcae.analytics.tca.web.integration.TcaAlertTransformer|||||||Request Id: c01eae77-f5d6-47eb-8277-403700e528d1, Transaction Id: SGXJC, Message counts - Received: 1, Errors: 0, Terminated Early: 0, Abated: [TcaAlert(aai=Aai(genericVNFName=null, genericServerName=zdfw1fwl01fwl01), closedLoopAlarmEnd=null, closedLoopAlarmStart=1507750704126521, closedLoopControlName=ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a, closedLoopEventClient=dcae-tcagen2, closedLoopEventStatus=ONSET, from=DCAE, policyName=DCAE.Config_tca-hi-lo, policyScope=DCAE, policyVersion=v0.0.1, requestId=c01eae77-f5d6-47eb-8277-403700e528d1, target=vserver.vserver-name, targetType=VM, version=1.0.2)], Alerts: 0, Alerts size: 1 2022-04-13 15:18:25.947 INFO 1 --- [ask-scheduler-6] o.o.d.a.t.w.i.TcaAlertTransformer : Request Id: c01eae77-f5d6-47eb-8277-403700e528d1, Transaction Id: SGXJC, Message counts - Received: 1, Errors: 0, Terminated Early: 0, Abated: [TcaAlert(aai=Aai(genericVNFName=null, genericServerName=zdfw1fwl01fwl01), closedLoopAlarmEnd=null, closedLoopAlarmStart=1507750704126521, closedLoopControlName=ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a, closedLoopEventClient=dcae-tcagen2, closedLoopEventStatus=ONSET, from=DCAE, policyName=DCAE.Config_tca-hi-lo, policyScope=DCAE, policyVersion=v0.0.1, requestId=c01eae77-f5d6-47eb-8277-403700e528d1, target=vserver.vserver-name, targetType=VM, version=1.0.2)], Alerts: 0, Alerts size: 1 2022-04-13T15:18:25.954+0000|2022-04-13T15:18:25.960+0000|c01eae77-f5d6-47eb-8277-403700e528d1|UNKNOWN_INSTANCE_ID|task-scheduler-6||dcae-tcagen2||DMAAP_MR_PUBLISHER|POST-/events/unauthenticated.DCAE_CL_OUTPUT|COMPLETE|0|OK||INFO|0|10.42.0.74|6|dcae-tcagen2||org.onap.dcae.analytics.web.http.EelfAuditLogInterceptor|||message-router:3904|||||Request Id: c01eae77-f5d6-47eb-8277-403700e528d1, Transaction Id: SGXJC, Elapsed Time: 6 ms, REST Endpoint Call: OK-POST-/events/unauthenticated.DCAE_CL_OUTPUT 2022-04-13 15:18:25.960 INFO 1 --- [ask-scheduler-6] o.o.d.a.w.http.EelfAuditLogInterceptor : Request Id: c01eae77-f5d6-47eb-8277-403700e528d1, Transaction Id: SGXJC, Elapsed Time: 6 ms, REST Endpoint Call: OK-POST-/events/unauthenticated.DCAE_CL_OUTPUT 2022-04-13 15:18:25.961 INFO 1 --- [ask-scheduler-6] o.s.i.h.s.MessagingMethodInvokerHelper : Overriding default instance of MessageHandlerMethodFactory with provided one. 2022-04-13 15:18:25.962 INFO 1 --- [ask-scheduler-6] o.s.i.h.s.MessagingMethodInvokerHelper : Overriding default instance of MessageHandlerMethodFactory with provided one. 2022-04-13T15:18:22.222+0000|2022-04-13T15:18:25.962+0000|c01eae77-f5d6-47eb-8277-403700e528d1|UNKNOWN_INSTANCE_ID|task-scheduler-6||dcae-tcagen2|tca-gen2|COMPLETE|0|||INFO|0|10.42.0.74|3740|dcae-tcagen2||org.onap.dcae.analytics.tca.web.integration.TcaPublisherResponseHandler|||||||Request Id: c01eae77-f5d6-47eb-8277-403700e528d1, Transaction Id: SGXJC, Transaction completion Time: 3740 ms, DMaaP MR Publisher Response: { "serverTimeMs": 1, "count": 1 } 2022-04-13 15:18:25.964 INFO 1 --- [ask-scheduler-6] .o.d.a.t.w.i.TcaPublisherResponseHandler : Request Id: c01eae77-f5d6-47eb-8277-403700e528d1, Transaction Id: SGXJC, Transaction completion Time: 3740 ms, DMaaP MR Publisher Response: { "serverTimeMs": 1, "count": 1 } 4) TCA o/p in DMAAP https://10.15.103.181:30226/events/unauthenticated.DCAE_CL_OUTPUT/vv/1 [ "{\"closedLoopEventClient\":\"dcae-tcagen2\",\"policyVersion\":\"v0.0.1\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyScope\":\"DCAE\",\"target_type\":\"VM\",\"AAI\":{\"vserver.vserver-name\":\"zdfw1fwl01fwl01\"},\"closedLoopAlarmStart\":1507750704126521,\"closedLoopEventStatus\":\"ONSET\",\"closedLoopControlName\":\"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",\"version\":\"1.0.2\",\"target\":\"vserver.vserver-name\",\"requestID\":\"c01eae77-f5d6-47eb-8277-403700e528d1\",\"from\":\"DCAE\"}" ] 5) Change default config to ABATED (default config does not generate ABATED event for vFirewall) Can be done via either configmap edit OR helm upgrade root@onap-1-vv:~# kubectl edit configmap -n onap dev-dcae-tcagen2-application-config-configmap configmap/dev-dcae-tcagen2-application-config-configmap edited 6) TCA log with updated config 2022-04-13 16:43:25.496 INFO 1 --- [ parallel-1] o.o.d.s.s.r.s.c.c.i.CbsClientConfigMap : PolicySync file does not exist 2022-04-13 16:43:25.497 INFO 1 --- [ parallel-1] o.o.d.s.s.r.s.c.c.i.CbsClientConfigMap : Got successful output from ConfigMap file 2022-04-13 16:43:25.497 DEBUG 1 --- [ parallel-1] o.o.d.s.s.r.s.c.c.i.CbsClientConfigMap : ConfigMap output: {"config":{"service_calls":[],"spring.data.mongodb.uri":"mongodb://dcae-mongohost/dcae-tcagen2","streams_publishes":{"tca_handle_out":{"dmaap_info":{"topic_url":"http://message-router:3904/events/unauthenticated.DCAE_CL_OUTPUT"},"type":"message_router"}},"streams_subscribes":{"tca_handle_in":{"dmaap_info":{"topic_url":"http://message-router:3904/events/unauthenticated.VES_MEASUREMENT_OUTPUT"},"type":"message_router"}},"streams_subscribes.tca_handle_in.consumer_group":"cg1","streams_subscribes.tca_handle_in.consumer_ids[0]":"c0","streams_subscribes.tca_handle_in.consumer_ids[1]":"c1","streams_subscribes.tca_handle_in.message_limit":50000,"streams_subscribes.tca_handle_in.polling.auto_adjusting.max":60000,"streams_subscribes.tca_handle_in.polling.auto_adjusting.min":30000,"streams_subscribes.tca_handle_in.polling.auto_adjusting.step_down":30000,"streams_subscribes.tca_handle_in.polling.auto_adjusting.step_up":10000,"streams_subscribes.tca_handle_in.polling.fixed_rate":0,"streams_subscribes.tca_handle_in.timeout":-1,"tca.aai.enable_enrichment":true,"tca.aai.generic_vnf_path":"aai/v11/network/generic-vnfs/generic-vnf","tca.aai.node_query_path":"aai/v11/search/nodes-query","tca.aai.password":"DCAE","tca.aai.url":"https://aai:8443","tca.aai.username":"DCAE","tca.enable_abatement":true,"tca.enable_ecomp_logging":true,"tca.policy":"{\"domain\":\"measurementsForVfScaling\",\"metricsPerEventName\":[{\"eventName\":\"vFirewallBroadcastPackets\",\"controlLoopSchemaType\":\"VM\",\"policyScope\":\"DCAE\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyVersion\":\"v0.0.1\",\"thresholds\":[{\"closedLoopControlName\":\"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.vNicPerformanceArray[*].receivedTotalPacketsDelta\",\"thresholdValue\":300,\"direction\":\"LESS_OR_EQUAL\",\"severity\":\"MAJOR\",\"closedLoopEventStatus\":\"ABATED\"},{\"closedLoopControlName\":\"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.vNicPerformanceArray[*].receivedTotalPacketsDelta\",\"thresholdValue\":700,\"direction\":\"GREATER_OR_EQUAL\",\"severity\":\"CRITICAL\",\"closedLoopEventStatus\":\"ONSET\"}]},{\"eventName\":\"vLoadBalancer\",\"controlLoopSchemaType\":\"VM\",\"policyScope\":\"DCAE\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyVersion\":\"v0.0.1\",\"thresholds\":[{\"closedLoopControlName\":\"ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.vNicPerformanceArray[*].receivedTotalPacketsDelta\",\"thresholdValue\":300,\"direction\":\"GREATER_OR_EQUAL\",\"severity\":\"CRITICAL\",\"closedLoopEventStatus\":\"ONSET\"}]},{\"eventName\":\"Measurement_vGMUX\",\"controlLoopSchemaType\":\"VNF\",\"policyScope\":\"DCAE\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyVersion\":\"v0.0.1\",\"thresholds\":[{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"EQUAL\",\"severity\":\"MAJOR\",\"closedLoopEventStatus\":\"ABATED\"},{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"GREATER\",\"severity\":\"CRITICAL\",\"closedLoopEventStatus\":\"ONSET\"}]}]}","tca.processing_batch_size":10000}} 2022-04-13T16:43:35.391+0000|341ef151-4201-43a4-bf61-a6ea75feaa4b|Request Id: 341ef151-4201-43a4-bf61-a6ea75feaa4b. Transaction Id: PT44L. Begin TS: 2022-04-13T16:43:35.391+0000. Starting new DMaaP MR Subscriber poll. 7) VES input for abatement {"event":{"commonEventHeader":{"startEpochMicrosec":1507750704126521,"sourceId":"4c32fa09-83e7-43cf-8bba-c72557afe408","eventId":"TrafficStats_1.2.3.4","nfcNamingCode":"vVNF","reportingEntityId":"No UUID available","eventType":"HTTP request rate","priority":"Normal","version":1.2,"reportingEntityName":"fwll","sequence":85,"domain":"measurementsForVfScaling","lastEpochMicrosec":1507750714317473,"eventName":"vFirewallBroadcastPackets","sourceName":"zdfw1fwl01fwl01","nfNamingCode":"vVNF"},"measurementsForVfScalingFields":{"cpuUsageArray":[{"percentUsage":0,"cpuIdentifier":"cpu1","cpuIdle":100,"cpuUsageSystem":0,"cpuUsageUser":0}],"measurementInterval":10,"requestRate":2256,"vNicPerformanceArray":[{"transmittedOctetsDelta":0,"receivedTotalPacketsDelta":100,"vNicIdentifier":"eth0","valuesAreSuspect":"true","transmittedTotalPacketsDelta":0,"receivedOctetsDelta":61200}],"measurementsForVfScalingVersion":2.1}}} 8) TCA output for ABATED EVENT into DMAAP https://10.15.103.181:30226/events/unauthenticated.DCAE_CL_OUTPUT/vv/1 [ "{\"closedLoopEventClient\":\"dcae-tcagen2\",\"policyVersion\":\"v0.0.1\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyScope\":\"DCAE\",\"target_type\":\"VM\",\"AAI\":{\"vserver.vserver-name\":\"zdfw1fwl01fwl01\"},\"closedLoopAlarmStart\":1507750704126521,\"closedLoopEventStatus\":\"ABATED\",\"closedLoopControlName\":\"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",\"version\":\"1.0.2\",\"target\":\"vserver.vserver-name\",\"closedLoopAlarmEnd\":1507750714317473,\"requestID\":\"4c056ecb-f420-4770-999a-0d2b310aa796\",\"from\":\"DCAE\"}" ] 9) AAI lookup 2022-04-13 16:36:06.882 INFO 1 --- [ask-scheduler-4] .d.a.w.h.HttpClientPreferencesCustomizer : SSL Validation will be enforced for Http Client Id: aai-rest-client 2022-04-13T16:36:06.888+0000|2022-04-13T16:36:06.927+0000|4c056ecb-f420-4770-999a-0d2b310aa796|UNKNOWN_INSTANCE_ID|task-scheduler-4||dcae-tcagen2|DCAE|AAI_ENRICHMENT|GET-/aai/v11/search/nodes-query|ERROR|200|Timeout Error||ERROR|2|10.42.0.91|39|dcae-tcagen2||org.onap.dcae.analytics.web.http.EelfAuditLogInterceptor|||aai:8443|||||Request Id: 4c056ecb-f420-4770-999a-0d2b310aa796, Transaction Id: 3BN3V, Elapsed Time: 39 ms, Error Message: java.net.UnknownHostException: aai: System error 2022-04-13 16:36:06.928 ERROR 1 --- [ask-scheduler-4] o.o.d.a.w.http.EelfAuditLogInterceptor : Request Id: 4c056ecb-f420-4770-999a-0d2b310aa796, Transaction Id: 3BN3V, Elapsed Time: 39 ms, Error Message: java.net.UnknownHostException: aai: System error 2022-04-13T16:36:06.935+0000|4c056ecb-f420-4770-999a-0d2b310aa796|task-scheduler-4|dcae-tcagen2|tca-gen2|DCAE-TCA|dcae-tcagen2|ERROR|400|Schema Error|Request id: 4c056ecb-f420-4770-999a-0d2b310aa796. Unable to get A&AI enrichment details 2022-04-13 16:36:06.935 ERROR 1 --- [ask-scheduler-4] .o.d.a.t.w.a.TcaAaiEnrichmentServiceImpl : Request id: 4c056ecb-f420-4770-999a-0d2b310aa796. Unable to get A&AI enrichment details 2022-04-13T16:36:05.391+0000|2022-04-13T16:36:06.936+0000|4c056ecb-f420-4770-999a-0d2b310aa796|UNKNOWN_INSTANCE_ID|task-scheduler-4||dcae-tcagen2|tca-gen2|COMPLETE|0|||INFO|0|10.42.0.91|1545|dcae-tcagen2||org.onap.dcae.analytics.tca.web.integration.TcaAlertTransformer|||||||Request Id: 4c056ecb-f420-4770-999a-0d2b310aa796, Transaction Id: 3BN3V, Message counts - Received: 1, Errors: 0, Terminated Early: 0, Abated: [TcaAlert(aai=Aai(genericVNFName=null, genericServerName=zdfw1fwl01fwl01), closedLoopAlarmEnd=null, closedLoopAlarmStart=1507750704126521, closedLoopControlName=ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a, closedLoopEventClient=dcae-tcagen2, closedLoopEventStatus=ONSET, from=DCAE, policyName=DCAE.Config_tca-hi-lo, policyScope=DCAE, policyVersion=v0.0.1, requestId=4c056ecb-f420-4770-999a-0d2b310aa796, target=vserver.vserver-name, targetType=VM, version=1.0.2)], Alerts: 0, Alerts size: 1 2022-04-13 16:36:06.953 INFO 1 --- [ask-scheduler-4] o.o.d.a.t.w.i.TcaAlertTransformer : Request Id: 4c056ecb-f420-4770-999a-0d2b310aa796, Transaction Id: 3BN3V, Message counts - Received: 1, Errors: 0, Terminated Early: 0, Abated: [TcaAlert(aai=Aai(genericVNFName=null, genericServerName=zdfw1fwl01fwl01), closedLoopAlarmEnd=null, closedLoopAlarmStart=1507750704126521, closedLoopControlName=ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a, closedLoopEventClient=dcae-tcagen2, closedLoopEventStatus=ONSET, from=DCAE, policyName=DCAE.Config_tca-hi-lo, policyScope=DCAE, policyVersion=v0.0.1, requestId=4c056ecb-f420-4770-999a-0d2b310aa796, target=vserver.vserver-name, targetType=VM, version=1.0.2)], Alerts: 0, Alerts size: 1 2022-04-13T16:36:06.960+0000|2022-04-13T16:36:06.976+0000|4c056ecb-f420-4770-999a-0d2b310aa796|UNKNOWN_INSTANCE_ID|task-scheduler-4||dcae-tcagen2||DMAAP_MR_PUBLISHER|POST-/events/unauthenticated.DCAE_CL_OUTPUT|COMPLETE|0|OK||INFO|0|10.42.0.91|16|dcae-tcagen2||org.onap.dcae.analytics.web.http.EelfAuditLogInterceptor|||message-router:3904|||||Request Id: 4c056ecb-f420-4770-999a-0d2b310aa796, Transaction Id: 3BN3V, Elapsed Time: 16 ms, REST Endpoint Call: OK-POST-/events/unauthenticated.DCAE_CL_OUTPUT