...
In order to conduct client tests this will be conducted in following architecture:
- HV-VES Client - produces high amount of events for processing.
- Processing Consumer - consumes events from Kafka topics and creates performance metrics.
- Offset Consumer - reads Kafka offsets.
- Prometheus - sends requests for performance metrics to HV-VES, Processing Consumer and Offset Consumer, provides data to Grafana.
- Grafana - delivers analytics and its visualization.
Link between HV-VES Client and HV-VES is TLS secured (provided scripts generate and place certificates on proper containers).
Info |
---|
Note: In the Without DMaaP Kafka tests the DMaaP/Kafka service was substituted with wurstmeister kafka |
...
Preconditions
- Installed ONAP (Frankfurt)
- Plain TCP connection between HV-VES and clients (default configuration)
- Metric port exposed on HV-VES service
...
Before start tests, download docker image of producer which is available here. available here:
View file | ||||
---|---|---|---|---|
|
To extract image locally use command:Â
...
Modify tools/performance/cloud/producer-pod.yaml file to use the above image and set imagePullPolicy to IfNotPresent:
Code Block | ||
---|---|---|
| ||
... spec: containers: - name: hv-collector-producer image: onap/org.onap.dcaegen2.collectors.hv-ves.hv-collector-go-client:latest imagePullPolicy: IfNotPresent volumeMounts: ... |
To execute performance tests we have to run functions from a shell script cloud-based-performance-test.sh in HV-VES project directory: ~/tools/performance/cloud/
First we have to generate certificates in ~/tools/ssl folder by using gen_certs. This step only needs to be performed during the first test setup (or if the generated files have been deleted).
Code Block title Generating certificates ./cloud-based-performance-test.sh gen_certs
Then we call setup in order to send certificates to HV-VES, and deploy Consumers, Prometheus, Grafana and create their ConfigMaps.
Code Block title Setting up the test environment ./cloud-based-performance-test.sh setup
After that we have to change HV-VES configuration in Consul KEY/VALUE tab (typically we can access Consul at port 30270 of any Controller node, i.e. http://slave1:30270/ui/#/dc1/kv/dcae-hv-ves-collector/edit).
Code Block title HV-VES Consul configuration collapse true {"security.sslDisable": false, "logLevel": "INFO", "server.listenPort": 6061, "server.idleTimeoutSec": 300, "cbs.requestIntervalSec": 5, "streams_publishes": { "perf3gpp": { "type": "kafka", "aaf_credentials": { "username": "admin", "password": "admin_secret" }, "kafka_info": { "bootstrap_servers": "message-router-kafka:9092", "topic_name": "HV_VES_PERF3GPP" } } }, "security.keys.trustStoreFile": "/etc/ves-hv/ssl/custom/trust.p12", "security.keys.keyStoreFile": "/etc/ves-hv/ssl/custom/server.p12", "security.keys.trustStorePasswordFile":"/etc/ves-hv/ssl/custom/trust.pass", "security.keys.keyStorePasswordFile": "/etc/ves-hv/ssl/custom/server.pass"}
Info title Hint How to access Consul UI?
Consul's address: http://<worker external IP>:<Consul External Port>
To check Consul External Port, execute:
ubuntu@onap-5422-rke-node:~$ kubectl -n onap get svc | grep consul
consul-server-ui                  NodePort      10.43.132.178  <none>                                8500:31190/TCP                                          6d20h
----------------------------------------------------------------------------------------------------------------------------------------------
If service "consul-server-ui" is not exposed to external port (NodePort) and is configured as Cluster IP, please follow steps below.
ubuntu@onap-5422-rke-node:~$ kubectl -n onap get svc | grep consul
consul-server-ui                                      ClusterIP     10.43.132.178  <none>                                8500/TCP                                                25h
ubuntu@onap-5422-rke-node:~$Â kubectl -n onap edit svc consul-server-ui
----
...
apiVersion: v1
kind: Service
metadata:
...
spec:
...
 ports:
...
 type: ClusterIP --> NodePort   ### change value to NodePort
status:
 ...
---------------------------------------
service/consul-server-ui edited
ubuntu@onap-5422-rke-node:~$ kubectl -n onap get svc | grep consul
consul-server-ui                                      NodePort      10.43.132.178  <none>                                8500:31190/TCP                                          25h
completing previous steps we can call the start function, which provides Producers and starts the test.
Code Block title Performing the test ./cloud-
After completing previous steps we can call the start function, which provides Producers and starts the test.
Code Block title Performing the test ./cloud-based-performance-test.sh start
For the start function we can use optional arguments:
--load should the test keep defined number of running producers until script interruption (false) --containers number of producer containers to create (1) --properties-file path to file with benchmark properties (./test.properties) --retention-time-minutes retention time of messages in kafka in minutes (60) Example invocations of test start:
Code Block title Starting performance test with single producers creation ./cloud-based-performance-test.sh start --containers 10
The command above starts the test that creates 10 producers which send the amount of messages defined in test.properties once.
Code Block title Starting performance test with constant messages load ./cloud-based-performance-test.sh start --load true --containers 10 --retention-time-minutes 30
This invocation starts load test, meaning the script will try to keep the amount of running containers at 10 with kafka message retention of 30 minutes.
The test.properties file contains Producers and Consumers configurations and it allows setting following properties:
Producer hvVesAddress HV-VES address (dcae-hv-ves-collector.onap:6061) client.count Number of clients per pod (1) message.size Size of a single message in bytes (16384) message.count Amount of messages to be send by each client (1000) message.interval Interval between messages in miliseconds (1) Certificates paths client.cert.path Path to cert file (/ssl/client.p12) client.cert.pass.path Path to cert's pass file (/ssl/client.pass) Consumer kafka.bootstrapServers Adress of Kafka service to consume from (message-router-kafka:9092) kafka.topics Kafka topics to subscribe to (HV_VES_PERF3GPP)
Results can be accessed under following links:
...
Raw results data with screenshots can be found in following files:
- Series 1 -Â results_series_1_with_dmaap.zip
- Series 2 -Â results_series_2_with_dmaap.zip
Test Results - series 1
Expand | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Below tables show the test results across a wide range of containers' number.Â
|
Test Results - series 2
Expand | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
No DMaaP Kafka SetUp
...
Raw results data with screenshots can be found in following files:
- Series 1 -Â results_series_1.zip
- Series 2 -Â results_series_2.zip
To see custom Kafka metrics you may want to change kafka-and-producers.json (located in HV-VES project directory: tools/performance/cloud/grafana/dashboards) to
...
Expand | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Below tables show the test results across a wide range of containers' number.Â
|
Test results - series 2
Expand | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Below tables show the test results across a wide range of containers' number.Â
|