...

In order to conduct client tests this will be conducted in following architecture:

Image RemovedImage Added

• HV-VES Client - produces high amount of events for processing.
• Processing Consumer - consumes events from Kafka topics and creates performance metrics.
• Offset Consumer - reads Kafka offsets.
• Prometheus - sends requests for performance metrics to HV-VES, Processing Consumer and Offset Consumer, provides data to Grafana.
• Grafana - delivers analytics and its visualization.

Link between HV-VES Client and HV-VES is TLS secured (provided scripts generate and place certificates on proper containers).

...

Preconditions

• Installed ONAP (Frankfurt)
• Plain TCP connection between HV-VES and clients (default configuration)
• Metric port exposed on HV-VES service

...

Before start tests, download docker image of producer which is available here. available here:

To extract image locally use command: 

docker load < hv-collector-go-client.tar.gz  

To execute performance tests we have to run functions from a shell script cloud-based-performance-test.sh in HV-VES project directory: ~/Modify tools/performance/cloud/

First we have to generate certificates in ~/tools/ssl folder by using gen_certs. This step only needs to be performed during the first test setup (or if the generated files have been deleted).

producer-pod.yaml file to use the above image and set imagePullPolicy to IfNotPresent:

...
spec:
  containers:
    - name: hv-collector-producer
      image: onap/org.onap.dcaegen2.collectors.hv-ves.hv-collector-go-client:latest
      imagePullPolicy: IfNotPresent
      volumeMounts:
...

...

...

To execute performance tests we have to run functions from a shell script 

cloud-based-performance-test.sh

...

...

in HV-VES project directory: ~/tools/performance/cloud/

1. First we have to generate certificates in ~/tools/ssl folder by using gen_certs. This step only needs to be performed during the first test setup (or if the generated files have been deleted).

   Code Block
   title Generating certificates
   ./cloud-based-performance-test.sh gen_certs

   
   

2. Then we call setup in order to send certificates to HV-VES, and deploy Consumers, order to send certificates to HV-VES, and deploy Consumers, Prometheus, Grafana and create their ConfigMaps.

   Code Block
   title Setting up the test environment
   ./cloud-based-performance-test.sh setup

   
   

3. After that we have to change HV-VES configuration in Consul KEY/VALUE tab (typically we can access Consul at port 30270 of any Controller node, i.e. http://slave1:30270/ui/#/dc1/kv/dcae-hv-ves-collector/edit).

   Code Block
   title HV-VES Consul configuration
   {"security.sslDisable": false, "logLevel": "INFO", "server.listenPort": 6061, "server.idleTimeoutSec": 300, "cbs.requestIntervalSec": 5, "streams_publishes": { "perf3gpp": { "type": "kafka", "aaf_credentials": { "username": "admin", "password": "admin_secret" }, "kafka_info": { "bootstrap_servers": "message-router-kafka:9092", "topic_name": "HV_VES_PERF3GPP" } } }, "security.keys.trustStoreFile": "/etc/ves-hv/ssl/custom/trust.p12", "security.keys.keyStoreFile": "/etc/ves-hv/ssl/custom/server.p12", "security.keys.trustStorePasswordFile":"/etc/ves-hv/ssl/custom/trust.pass", "security.keys.keyStorePasswordFile": "/etc/ves-hv/ssl/custom/server.pass"}

   After completing previous steps we can call the start function, which provides Producers and starts the test.

   Code Block
   title Performing the test
   ./cloud-based-performance-test.sh start

   For the start function we can use optional arguments:

   --loadshould the test keep defined number of running producers until script interruption (false)--containersnumber of producer containers to create (1)--properties-filepath to file with benchmark properties (./test.properties)--retention-time-minutesretention time of messages in kafka in minutes (60)

   Example invocations of test start:

   Code Block
   title Starting performance test with single producers creation
   ./cloud-based-performance-test.sh start --containers 10

   The command above starts the test that creates 10 producers which send the amount of messages defined in test.properties once.

   Code Block
   title Starting performance test with constant messages load
   ./cloud-based-performance-test.sh start --load true --containers 10 --retention-time-minutes 30

   This invocation starts load test, meaning the script will try to keep the amount of running containers at 10 with kafka message retention of 30 minutes.

   The test.properties file contains Producers and Consumers configurations and it allows setting following properties:

   ProducerhvVesAddressHV-VES address (dcae-hv-ves-collector.onap:6061)client.countNumber of clients per pod (1)message.sizeSize of a single message in bytes (16384)message.countAmount of messages to be send by each client (1000)message.intervalInterval between messages in miliseconds (1)Certificates pathsclient.cert.pathPath to cert file (/ssl/client.p12)client.cert.pass.pathPath to cert's pass file (/ssl/client.pass)Consumerkafka.bootstrapServersAdress of Kafka service to consume from (message-router-kafka:9092)kafka.topicsKafka topics to subscribe to (HV_VES_PERF3GPP)Results can be accessed under following links:
4. Prometheus: http://slave1:30000/graph?g0.range_input=1h&g0.expr=hv_kafka_consumer_travel_time_seconds_count&g0.tab=1
Grafana: http://slave1:30001/d/V94Kjlwmz/hv-ves-processing?orgId=1&refresh=5s


To remove created ConfigMaps, Consumers, Producers, Grafana and Prometheus from Kubernetes cluster we call clean function. Note: clean doesn't remove certificates from HV-VES.

Code Block
title Cleaning the environment
./cloud-based-performance-test.sh clean

In order to restart the test environment (perform steps in following order: 5, 2 and then 3), which means redeploying hv-ves pod, resetting kafka topic and performing setup, we use reboot-test-environment.sh.

...

1. completing previous steps we can call the start function, which provides Producers and starts the test.

   Code Block
   title Performing the test
   ./cloud-based-performance-test.sh start

   For the start function we can use optional arguments:

   --load	should the test keep defined number of running producers until script interruption (false)
   --containers	number of producer containers to create (1)
   --properties-file	path to file with benchmark properties (./test.properties)
   --retention-time-minutes	retention time of messages in kafka in minutes (60)

   Example invocations of test start:

   Code Block
   title Starting performance test with single producers creation
   ./cloud-based-performance-test.sh start --containers 10

   The command above starts the test that creates 10 producers which send the amount of messages defined in test.properties once.

   Code Block
   title Starting performance test with constant messages load
   ./cloud-based-performance-test.sh start --load true --containers 10 --retention-time-minutes 30

   This invocation starts load test, meaning the script will try to keep the amount of running containers at 10 with kafka message retention of 30 minutes.

   
   

   The test.properties file contains Producers and Consumers configurations and it allows setting following properties:

   Producer
   hvVesAddress	HV-VES address (dcae-hv-ves-collector.onap:6061)
   client.count	Number of clients per pod (1)
   message.size	Size of a single message in bytes (16384)
   message.count	Amount of messages to be send by each client (1000)
   message.interval	Interval between messages in miliseconds (1)
   Certificates paths
   client.cert.path	Path to cert file (/ssl/client.p12)
   client.cert.pass.path	Path to cert's pass file (/ssl/client.pass)
   Consumer
   kafka.bootstrapServers	Adress of Kafka service to consume from (message-router-kafka:9092)
   kafka.topics	Kafka topics to subscribe to (HV_VES_PERF3GPP)

...

1. 
   Results can be accessed under following links:

   • Prometheus: http://slave1:30000/graph?g0.range_input=1h&g0.expr=hv_kafka_consumer_travel_time_seconds_count&g0.tab=1
   • Grafana: http://slave1:30001/d/V94Kjlwmz/hv-ves-processing?orgId=1&refresh=5s

HV-VES Performance test results

...

Raw results data with screenshots can be found in following files:

• Series 1 - results_series_1_with_dmaap.zip
• Series 2 - results_series_2_with_dmaap.zip

Test Results - series 1

Expand
title Click here to see results...
Below tables show the test results across a wide range of containers' number.  NUMBER OF PRODUCERS TOTAL MESSAGES PROCESSED DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms] AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] PEAK INCOMING DATA RATE [MB/s] PEAK PROCESSING MESSAGE QUEUE SIZE PEAK CPU LOAD [%] PEAK MEMORY USAGE [GB] RESULTS PRESENTED IN GRAFANA 2 180000 0 0.03 54 1.6 175 2.9 0.38 Image RemovedImage RemovedImage RemovedImage AddedImage AddedImage Added 2 180000 0 0.021 8.5 1.6 6 3.2 0.37 Image RemovedImage RemovedImage RemovedImage AddedImage AddedImage Added 2 180000 0 0.022 14.8 1.6 66 2.4 0.38 Image RemovedImage RemovedImage RemovedImage AddedImage AddedImage Added 2 180000 0 0.023 19.4 1.6 59 2.2 0.38 Image RemovedImage RemovedImage RemovedImage AddedImage AddedImage Added 2 180000 0 0.021 7.6 1.6 58 2.6 0.36 Image AddedImage Added Image Added Image RemovedImage Removed Image Removed NUMBER OF PRODUCERS TOTAL MESSAGES PROCESSED DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms] AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] PEAK INCOMING DATA RATE [MB/s] PEAK PROCESSING MESSAGE QUEUE SIZE PEAK CPU LOAD [%] PEAK MEMORY USAGE [GB] RESULTS PRESENTED IN GRAFANA 4 315868 44824 82 260 3,2 2400 3 0,4 Image RemovedImage Added Image RemovedImage RemovedImage AddedImage Added 4 359193 807 3660 10408 3,2 3100 4,9 0,78 Image RemovedImage RemovedImage RemovedImage AddedImage AddedImage Added 4 359401 591 1350 3486 3.7 3000 4,9 0,56 Image RemovedImage RemovedImage RemovedImage AddedImage AddedImage Added 4 360000 0 1.5 130 3.3 1200 4.1 0.39 Image RemovedImage RemovedImage AddedImage Added Image RemovedImage Added 4 360000 0 0.02 57 3.3 180 3.5 0.38 Image RemovedImage RemovedImage RemovedImage AddedImage AddedImage Added NUMBER OF PRODUCERS TOTAL MESSAGES PROCESSED DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms] AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] PEAK INCOMING DATA RATE [MB/s] PEAK PROCESSING MESSAGE QUEUE SIZE PEAK CPU LOAD [%] PEAK MEMORY USAGE [GB] RESULTS PRESENTED IN GRAFANA 6 358475 1525 6600 17875 4.9 4900 5.8 0.97 Image RemovedImage RemovedImage RemovedImage AddedImage AddedImage Added 6 358494 1506 7080 20182 4.7 4500 4.5 0.97 Image RemovedImage RemovedImage RemovedImage AddedImage AddedImage Added 6 359244 756 4900 17231 4.9 1900 3.9 0.93 Image RemovedImage RemovedImage RemovedImage AddedImage AddedImage Added 6 358944 1056 6150 17200 4.9 4800 5 0.97 Image RemovedImage RemovedImage RemovedImage AddedImage AddedImage Added 6 358689 1311 5410 17202 4.9 4100 5 0.96 Image RemovedImage RemovedImage RemovedImage AddedImage AddedImage Added

Test Results - series 2

Expand
title Click here to see results...
NUMBER OF PRODUCERS TOTAL MESSAGES PROCESSED DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms] AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] PEAK INCOMING DATA RATE [MB/s] PEAK PROCESSING MESSAGE QUEUE SIZE PEAK CPU LOAD [%] PEAK MEMORY USAGE [GB] RESULTS PRESENTED IN GRAFANA 2 180000 0 0.026 8.9 1.6 11 3 0.35 Image RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage AddedImage AddedImage AddedImage AddedImage AddedImage Added 2 180000 0 0.026 11.3 1.6 23 3.2 0.35 Image Added Image AddedImage AddedImage AddedImage AddedImage Added Image Removed Image RemovedImage RemovedImage RemovedImage RemovedImage Removed NUMBER OF PRODUCERS TOTAL MESSAGES PROCESSED DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [s] AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] PEAK INCOMING DATA RATE [MB/s] PEAK PROCESSING MESSAGE QUEUE SIZE PEAK CPU LOAD [%] PEAK MEMORY USAGE [GB] RESULTS PRESENTED IN GRAFANA 4 360000 0 0.022 41.3 3.2 130 4 0.35 Image AddedImage AddedImage Added Image AddedImage AddedImage Added Image RemovedImage RemovedImage Removed Image RemovedImage RemovedImage Removed 4 360000 0 0.023 40.7 3.2 370 4.2 0.35 Image RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage AddedImage AddedImage AddedImage AddedImage AddedImage Added NUMBER OF PRODUCERS TOTAL MESSAGES PROCESSED DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms] AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] PEAK INCOMING DATA RATE [MB/s] PEAK PROCESSING MESSAGE QUEUE SIZE PEAK CPU LOAD [%] PEAK MEMORY USAGE [GB] RESULTS PRESENTED IN GRAFANA 6 359317 683 6240 15965 4.8 4500 4.3 1 Image RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage AddedImage AddedImage AddedImage AddedImage AddedImage Added 6 359217 783 6490 16834 4.8 4200 5.8 1 Image RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage AddedImage AddedImage AddedImage AddedImage AddedImage Added

No DMaaP Kafka SetUp

...

Raw results data with screenshots can be found in following files:

• Series 1 - results_series_1.zip
• Series 2 - results_series_2.zip

To see custom Kafka metrics you may want to change kafka-and-producers.json (located in HV-VES project directory: tools/performance/cloud/grafana/dashboards) to

...

Expand
title Click here to see results...
Below tables show the test results across a wide range of containers' number.  NUMBER OF PRODUCERS TOTAL MESSAGES PROCESSED DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms] AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] PEAK INCOMING DATA RATE [MB/s] PEAK PROCESSING MESSAGE QUEUE SIZE PEAK CPU LOAD [%] PEAK MEMORY USAGE [GB] RESULTS PRESENTED IN GRAFANA 2 180000 0 0.024 1.5 1.6 3 3.5 0.37 Image RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage AddedImage AddedImage AddedImage AddedImage AddedImage Added 4 360000 0 0.020 1.7 3.2 3 5.7 0.37 Image RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage AddedImage AddedImage AddedImage AddedImage AddedImage Added 6 540000 0 0.02 2.6 4.8 24 6.0 0.37 Image AddedImage Added Image AddedImage AddedImage AddedImage Added Image RemovedImage Removed Image RemovedImage RemovedImage RemovedImage Removed 8 720000 0 0.02 2.8 6.4 14 8.5 0.37 Image RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage AddedImage AddedImage AddedImage AddedImage AddedImage Added 10 900000 0 0.02 7.3 8.1 552 8.5 0.38 Image RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage AddedImage AddedImage AddedImage AddedImage AddedImage Added 12 10800000 0 0.02 78 9.7 1077 7.5 0.41 Image AddedImage AddedImage Added Image AddedImage Added Image Added Image RemovedImage RemovedImage Removed Image RemovedImage Removed Image Removed 14 12600000 0 6140 19000 13.0 10630 9.8 0.99 Image RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage RemovedImage AddedImage AddedImage AddedImage AddedImage AddedImage Added

Test results - series 2