- Created by Vijay Venkatesh Kumar, last modified by Joanna Jeremicz on Jul 21, 2020
You are viewing an old version of this page. View the current version.
Compare with Current View Page History
« Previous Version 9 Next »
Environment and Resources
Kubernetes cluster with 4 worker nodes, sharing hardware configuration shown in a table below, is deployed in OpenStack cloud operating system. The test components in docker containers are further deployed on the Kubernetes cluster.
Configuration | ||
---|---|---|
CPU | Model | Intel(R) Xeon(R) CPU E5-2680 v4 |
No. of cores | 24 | |
CPU clock speed [GHz] | 2.40 | |
Total RAM [GB] | 62.9 |
Network Performance
Pod measurement method
In order to check cluster network performance tests with usage of Iperf3 have been applied. Iperf is a tool for measurement of the maximum bandwidth on IP networks, it runs on two mode: server and client. We used a docker image: networkstatic/iperf3.
Following deployment creates a pod with iperf (server mode) on one worker, and one pod with iperf client for each worker.
apiVersion: apps/v1 kind: Deployment metadata: name: iperf3-server namespace: onap labels: app: iperf3-server spec: replicas: 1 selector: matchLabels: app: iperf3-server template: metadata: labels: app: iperf3-server spec: containers: - name: iperf3-server image: networkstatic/iperf3 args: ['-s'] ports: - containerPort: 5201 name: server --- apiVersion: v1 kind: Service metadata: name: iperf3-server namespace: onap spec: selector: app: iperf3-server ports: - protocol: TCP port: 5201 targetPort: server --- apiVersion: apps/v1 kind: DaemonSet metadata: name: iperf3-clients namespace: onap labels: app: iperf3-client spec: selector: matchLabels: app: iperf3-client template: metadata: labels: app: iperf3-client spec: containers: - name: iperf3-client image: networkstatic/iperf3 command: ['/bin/sh', '-c', 'sleep infinity']
To create deployment, execute following command:
kubectl create -f deployment.yaml
To find all iperf pods, execute:
kubectl -n onap get pods -o wide | grep iperf
To measure connection between pods, run iperf on iperf-client pod, using following command:
kubectl -n onap exec -it <iperf-client-pod> -- iperf3 -c iperf3-server
To change output format from MBits/sec to MBytes/sec:
kubectl -n onap exec -it <iperf-client-pod> -- iperf3 -c iperf3-server -f MBytes
To change measure time:
kubectl -n onap exec -it <iperf-client-pod> -- iperf3 -c iperf3-server -t <time-in-second>
To gather results, the command was executed:
kubectl -n onap exec -it <iperf-client-pod> -- iperf3 -c iperf3-server -f MBytes
Results of performed tests
worker1 (136 MBytes/sec)
results Expand sourceConnecting to host iperf3-server, port 5201 [ 4] local 10.42.5.127 port 39752 connected to 10.43.25.161 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 141 MBytes 141 MBytes/sec 32 673 KBytes [ 4] 1.00-2.00 sec 139 MBytes 139 MBytes/sec 0 817 KBytes [ 4] 2.00-3.00 sec 139 MBytes 139 MBytes/sec 0 936 KBytes [ 4] 3.00-4.00 sec 138 MBytes 137 MBytes/sec 0 1.02 MBytes [ 4] 4.00-5.00 sec 138 MBytes 137 MBytes/sec 0 1.12 MBytes [ 4] 5.00-6.00 sec 129 MBytes 129 MBytes/sec 0 1.20 MBytes [ 4] 6.00-7.00 sec 129 MBytes 129 MBytes/sec 0 1.27 MBytes [ 4] 7.00-8.00 sec 134 MBytes 134 MBytes/sec 0 1.35 MBytes [ 4] 8.00-9.00 sec 135 MBytes 135 MBytes/sec 0 1.42 MBytes [ 4] 9.00-10.00 sec 135 MBytes 135 MBytes/sec 45 1.06 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 1.32 GBytes 136 MBytes/sec 77 sender [ 4] 0.00-10.00 sec 1.32 GBytes 135 MBytes/sec receiver
worker2 (87 MBytes/sec)
results Expand sourceConnecting to host iperf3-server, port 5201 [ 4] local 10.42.3.188 port 35472 connected to 10.43.25.161 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 88.3 MBytes 88.3 MBytes/sec 121 697 KBytes [ 4] 1.00-2.00 sec 96.2 MBytes 96.3 MBytes/sec 0 796 KBytes [ 4] 2.00-3.00 sec 92.5 MBytes 92.5 MBytes/sec 0 881 KBytes [ 4] 3.00-4.00 sec 90.0 MBytes 90.0 MBytes/sec 0 957 KBytes [ 4] 4.00-5.00 sec 87.5 MBytes 87.5 MBytes/sec 0 1.00 MBytes [ 4] 5.00-6.00 sec 88.8 MBytes 88.7 MBytes/sec 0 1.06 MBytes [ 4] 6.00-7.00 sec 80.0 MBytes 80.0 MBytes/sec 0 1.12 MBytes [ 4] 7.00-8.00 sec 81.2 MBytes 81.3 MBytes/sec 25 895 KBytes [ 4] 8.00-9.00 sec 85.0 MBytes 85.0 MBytes/sec 0 983 KBytes [ 4] 9.00-10.00 sec 83.8 MBytes 83.7 MBytes/sec 0 1.03 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 873 MBytes 87.3 MBytes/sec 146 sender [ 4] 0.00-10.00 sec 870 MBytes 87.0 MBytes/sec receiver
worker3 (135 MBytes/sec)
results Expand sourceConnecting to host iperf3-server, port 5201 [ 4] local 10.42.4.182 port 35288 connected to 10.43.25.161 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 129 MBytes 129 MBytes/sec 45 1.17 MBytes [ 4] 1.00-2.00 sec 134 MBytes 134 MBytes/sec 32 1.25 MBytes [ 4] 2.00-3.00 sec 135 MBytes 135 MBytes/sec 0 1.32 MBytes [ 4] 3.00-4.00 sec 139 MBytes 139 MBytes/sec 0 1.40 MBytes [ 4] 4.00-5.00 sec 144 MBytes 144 MBytes/sec 0 1.47 MBytes [ 4] 5.00-6.00 sec 131 MBytes 131 MBytes/sec 45 1.14 MBytes [ 4] 6.00-7.00 sec 129 MBytes 129 MBytes/sec 0 1.25 MBytes [ 4] 7.00-8.00 sec 134 MBytes 134 MBytes/sec 0 1.33 MBytes [ 4] 8.00-9.00 sec 138 MBytes 138 MBytes/sec 0 1.39 MBytes [ 4] 9.00-10.00 sec 135 MBytes 135 MBytes/sec 0 1.44 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 1.31 GBytes 135 MBytes/sec 122 sender [ 4] 0.00-10.00 sec 1.31 GBytes 134 MBytes/sec receiver
worker0 (2282 MBytes/sec) (iperf client and server exist on same worker )
results Expand sourceConnecting to host iperf3-server, port 5201 [ 4] local 10.42.6.132 port 51156 connected to 10.43.25.161 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 2.13 GBytes 2185 MBytes/sec 0 536 KBytes [ 4] 1.00-2.00 sec 1.66 GBytes 1702 MBytes/sec 0 621 KBytes [ 4] 2.00-3.00 sec 2.10 GBytes 2154 MBytes/sec 0 766 KBytes [ 4] 3.00-4.00 sec 1.89 GBytes 1937 MBytes/sec 0 1.01 MBytes [ 4] 4.00-5.00 sec 1.87 GBytes 1914 MBytes/sec 0 1.39 MBytes [ 4] 5.00-6.00 sec 2.76 GBytes 2826 MBytes/sec 0 1.39 MBytes [ 4] 6.00-7.00 sec 1.81 GBytes 1853 MBytes/sec 792 1.09 MBytes [ 4] 7.00-8.00 sec 2.54 GBytes 2600 MBytes/sec 0 1.21 MBytes [ 4] 8.00-9.00 sec 2.70 GBytes 2763 MBytes/sec 0 1.34 MBytes [ 4] 9.00-10.00 sec 2.82 GBytes 2889 MBytes/sec 0 1.34 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 22.3 GBytes 2282 MBytes/sec 792 sender [ 4] 0.00-10.00 sec 22.3 GBytes 2282 MBytes/sec receiver
Average speed (without worker 0 ) : 119 MBytes/sec
HV-VES Performance
Preconditions
Before start tests, download docker image of producer which is available here. To extract image locally use command:
docker load < hv-collector-go-client.tar.gz
With dmaap Kafka
Conditions
Tests were performed with 5 repetitions for each configuration shown in the table below.
Number of producers | Messages per producer | Payload size [B] | Interval [ms] |
---|---|---|---|
2 | 90000 | 8192 | 10 |
4 | 90000 | 8192 | 10 |
6 | 60000 | 8192 | 10 |
Raw results data
Raw results data with screenshots can be found in following files:
- Series 1 - results_series_1.zip
- Series 2 - results_series_2.zip
Test Results - series 1
Below tables show the test results across a wide range of containers' number.
NUMBER OF PRODUCERS | TOTAL MESSAGES PROCESSED | DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES | AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms] | AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] | PEAK INCOMING DATA RATE [MB/s] | PEAK PROCESSING MESSAGE QUEUE SIZE | PEAK CPU LOAD [%] | PEAK MEMORY USAGE [GB] | RESULTS PRESENTED IN GRAFANA |
---|---|---|---|---|---|---|---|---|---|
2 | 180000 | 0 | 0.03 | 54 | 1.6 | 175 | 2.9 | 0.38 | |
2 | 180000 | 0 | 0.021 | 8.5 | 1.6 | 6 | 3.2 | 0.37 | |
2 | 180000 | 0 | 0.022 | 14.8 | 1.6 | 66 | 2.4 | 0.38 | |
2 | 180000 | 0 | 0.023 | 19.4 | 1.6 | 59 | 2.2 | 0.38 | |
2 | 180000 | 0 | 0.021 | 7.6 | 1.6 | 58 | 2.6 | 0.36 | ![]() |
NUMBER OF PRODUCERS | TOTAL MESSAGES PROCESSED | DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES | AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms] | AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] | PEAK INCOMING DATA RATE [MB/s] | PEAK PROCESSING MESSAGE QUEUE SIZE | PEAK CPU LOAD [%] | PEAK MEMORY USAGE [GB] | RESULTS PRESENTED IN GRAFANA |
---|---|---|---|---|---|---|---|---|---|
4 | 315868 | 44824 | 82 | 260 | 3,2 | 2400 | 3 | 0,4 | ![]() ![]() |
4 | 359193 | 807 | 3660 | 10408 | 3,2 | 3100 | 4,9 | 0,78 | |
4 | 359401 | 591 | 1350 | 3486 | 3.7 | 3000 | 4,9 | 0,56 | |
4 | 360000 | 0 | 1.5 | 130 | 3.3 | 1200 | 4.1 | 0.39 | ![]() |
4 | 360000 | 0 | 0.02 | 57 | 3.3 | 180 | 3.5 | 0.38 |
NUMBER OF PRODUCERS | TOTAL MESSAGES PROCESSED | DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES | AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms] | AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] | PEAK INCOMING DATA RATE [MB/s] | PEAK PROCESSING MESSAGE QUEUE SIZE | PEAK CPU LOAD [%] | PEAK MEMORY USAGE [GB] | RESULTS PRESENTED IN GRAFANA |
---|---|---|---|---|---|---|---|---|---|
6 | 358475 | 1525 | 6600 | 17875 | 4.9 | 4900 | 5.8 | 0.97 | |
6 | 358494 | 1506 | 7080 | 20182 | 4.7 | 4500 | 4.5 | 0.97 | |
6 | 359244 | 756 | 4900 | 17231 | 4.9 | 1900 | 3.9 | 0.93 | |
6 | 358944 | 1056 | 6150 | 17200 | 4.9 | 4800 | 5 | 0.97 | |
6 | 358689 | 1311 | 5410 | 17202 | 4.9 | 4100 | 5 | 0.96 |
Test Results - series 2
NUMBER OF PRODUCERS | TOTAL MESSAGES PROCESSED | DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES | AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms] | AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] | PEAK INCOMING DATA RATE [MB/s] | PEAK PROCESSING MESSAGE QUEUE SIZE | PEAK CPU LOAD [%] | PEAK MEMORY USAGE [GB] | RESULTS PRESENTED IN GRAFANA |
---|---|---|---|---|---|---|---|---|---|
2 | 180000 | 0 | 0.026 | 8.9 | 1.6 | 11 | 3 | 0.35 | |
2 | 180000 | 0 | 0.026 | 11.3 | 1.6 | 23 | 3.2 | 0.35 | ![]() |
NUMBER OF PRODUCERS | TOTAL MESSAGES PROCESSED | DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES | AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [s] | AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] | PEAK INCOMING DATA RATE [MB/s] | PEAK PROCESSING MESSAGE QUEUE SIZE | PEAK CPU LOAD [%] | PEAK MEMORY USAGE [GB] | RESULTS PRESENTED IN GRAFANA |
---|---|---|---|---|---|---|---|---|---|
4 | 360000 | 0 | 0.022 | 41.3 | 3.2 | 130 | 4 | 0.35 | ![]() ![]() ![]() |
4 | 360000 | 0 | 0.023 | 40.7 | 3.2 | 370 | 4.2 | 0.35 |
NUMBER OF PRODUCERS | TOTAL MESSAGES PROCESSED | DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES | AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms] | AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] | PEAK INCOMING DATA RATE [MB/s] | PEAK PROCESSING MESSAGE QUEUE SIZE | PEAK CPU LOAD [%] | PEAK MEMORY USAGE [GB] | RESULTS PRESENTED IN GRAFANA |
---|---|---|---|---|---|---|---|---|---|
6 | 359317 | 683 | 6240 | 15965 | 4.8 | 4500 | 4.3 | 1 | |
6 | 359217 | 783 | 6490 | 16834 | 4.8 | 4200 | 5.8 | 1 |
Without DMaaP Kafka
Conditions
Environment was prepared using: Performance test without DMaaP - instruction.
To gather tcpdump data another container was added to hv-ves deployment in kubernetes and producer-pod.yaml (command to get tcpdump data file: kubectl cp -n onap <pod-name>:/tcpdump.pcap -c tcpdump ./<pod-name>.pcap):
... containers: ... <<default containers>> ... - name: tcpdump image: onap-dev-local.esisoj70.emea.nsn-net.net/rjanecze/my-tcpdump:1.0.7 ...
Tests were performed with following configuration:
Messages per producer | Payload size [B] | Interval [ms] |
---|---|---|
90000 | 8192 | 10 |
Raw results data
Raw results data with screenshots can be found in following files:
- Series 1 - results_series_1.zip
- Series 2 - results_series_2.zip
To see custom Kafka metrics you may want to change kafka-and-producers.json (located in HV-VES project directory: tools/performance/cloud/grafana/dashboards) to
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"gnetId": null,
"graphTooltip": 0,
"id": 3,
"links": [],
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 0,
"fillGradient": 0,
"gridPos": {
"h": 11,
"w": 8,
"x": 0,
"y": 0
},
"hiddenSeries": false,
"id": 12,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(irate(container_network_transmit_bytes_total{pod=~\".*producer.*\"}[1m]))",
"interval": "",
"legendFormat": "Total Transmit Producers",
"refId": "A"
},
{
"expr": "sum(irate(container_network_transmit_bytes_total{pod=~\".*kafka-broker.*\"}[1m]))",
"interval": "",
"legendFormat": "Total Transmit Kafka",
"refId": "B"
},
{
"expr": "irate(container_network_transmit_bytes_total{pod=~\".*producer.*\"}[1m])",
"interval": "",
"legendFormat": "{{pod}}",
"refId": "C"
},
{
"expr": "irate(container_network_transmit_bytes_total{pod=~\".*kafka-broker.*\"}[1m])",
"interval": "",
"legendFormat": "{{pod}}",
"refId": "D"
}
],
"thresholds": [],
"timeFrom": "20m",
"timeRegions": [],
"timeShift": null,
"title": "Network Usage",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "Bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 0,
"fillGradient": 0,
"gridPos": {
"h": 11,
"w": 8,
"x": 8,
"y": 0
},
"hiddenSeries": false,
"id": 22,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(rate(container_network_transmit_bytes_total{pod=~\".*kafka-broker.*|.*hv-ves-coll.*|.*producer.*\"}[1m]))",
"hide": false,
"interval": "",
"legendFormat": "Total transmit",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": "20m",
"timeRegions": [],
"timeShift": null,
"title": "Total network usage (producers + hv-ves + kafka)",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "Bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 0,
"fillGradient": 0,
"gridPos": {
"h": 11,
"w": 8,
"x": 16,
"y": 0
},
"hiddenSeries": false,
"id": 2,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "irate(container_network_receive_bytes_total{pod=~\"dep-dcae-hv-ves-collector.*\"}[1m])",
"hide": false,
"interval": "",
"legendFormat": "receive",
"refId": "A"
},
{
"expr": "irate(container_network_transmit_bytes_total{pod=~\"dep-dcae-hv-ves-collector.*\"}[1m])",
"interval": "",
"legendFormat": "transmit",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": "20m",
"timeRegions": [],
"timeShift": null,
"title": "HV-VES Network Usage",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"decimals": null,
"format": "Bps",
"label": "",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 0,
"fillGradient": 0,
"gridPos": {
"h": 12,
"w": 8,
"x": 0,
"y": 11
},
"hiddenSeries": false,
"id": 14,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum (rate (container_cpu_usage_seconds_total{pod=~\".*producer.*\"}[1m])) by (pod)",
"interval": "",
"legendFormat": "{{pod}}",
"refId": "A"
},
{
"expr": "sum (rate (container_cpu_usage_seconds_total{pod=~\".*kafka-broker.*\"}[1m])) by (pod)",
"interval": "",
"legendFormat": "{{pod}}",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": "20m",
"timeRegions": [],
"timeShift": null,
"title": "CPU usage [cores]",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "none",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 0,
"fillGradient": 0,
"gridPos": {
"h": 12,
"w": 8,
"x": 8,
"y": 11
},
"hiddenSeries": false,
"id": 18,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(container_memory_working_set_bytes{image!=\"\", pod=~\".*kafka-broker.*\"}) by (pod)",
"interval": "",
"legendFormat": "{{pod}}",
"refId": "A"
},
{
"expr": "sum(container_memory_working_set_bytes{image!=\"\", pod=~\".*producer.*\"}) by (pod)",
"hide": false,
"interval": "",
"legendFormat": "{{pod}}",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": "20m",
"timeRegions": [],
"timeShift": null,
"title": "Memory Usage",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"datasource": null,
"gridPos": {
"h": 6,
"w": 4,
"x": 16,
"y": 11
},
"id": 15,
"options": {
"colorMode": "value",
"fieldOptions": {
"calcs": [
"lastNotNull"
],
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": [],
"values": false
},
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto"
},
"pluginVersion": "6.7.3",
"targets": [
{
"expr": "sum(container_network_transmit_packets_dropped_total{pod=~\".*producer.*\"})",
"interval": "",
"legendFormat": "Packets dropped",
"refId": "A"
}
],
"timeFrom": null,
"timeShift": null,
"title": "Packets Dropped (producers)",
"type": "stat"
},
{
"datasource": null,
"gridPos": {
"h": 6,
"w": 4,
"x": 20,
"y": 11
},
"id": 4,
"options": {
"colorMode": "value",
"fieldOptions": {
"calcs": [
"lastNotNull"
],
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": [],
"values": false
},
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto"
},
"pluginVersion": "6.7.3",
"targets": [
{
"expr": "sum(container_network_transmit_packets_dropped_total{pod=~\".*hv-ves-collector.*\"})",
"interval": "",
"legendFormat": "Packets dropped",
"refId": "A"
}
],
"timeFrom": null,
"timeShift": null,
"title": "Packets Dropped (hv-ves)",
"type": "stat"
},
{
"datasource": null,
"gridPos": {
"h": 6,
"w": 4,
"x": 16,
"y": 17
},
"id": 17,
"options": {
"colorMode": "value",
"fieldOptions": {
"calcs": [
"max"
],
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "bytes"
},
"overrides": [],
"values": false
},
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto"
},
"pluginVersion": "6.7.3",
"targets": [
{
"expr": "sum(container_network_transmit_bytes_total{pod=~\".*producer.*\"})",
"interval": "",
"legendFormat": "",
"refId": "A"
}
],
"timeFrom": null,
"timeShift": null,
"title": "Sent (producers)",
"type": "stat"
},
{
"cacheTimeout": null,
"datasource": null,
"gridPos": {
"h": 6,
"w": 4,
"x": 20,
"y": 17
},
"id": 16,
"links": [],
"options": {
"colorMode": "value",
"fieldOptions": {
"calcs": [
"max"
],
"defaults": {
"mappings": [
{
"id": 0,
"op": "=",
"text": "N/A",
"type": 1,
"value": "null"
}
],
"nullValueMode": "connected",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "bytes"
},
"overrides": [],
"values": false
},
"graphMode": "none",
"justifyMode": "auto",
"orientation": "horizontal"
},
"pluginVersion": "6.7.3",
"targets": [
{
"expr": "container_network_receive_bytes_total{pod=~\".*hv-ves-collector.*\"}",
"interval": "",
"legendFormat": "Packets dropped",
"refId": "A"
}
],
"timeFrom": null,
"timeShift": null,
"title": "Received (hv-ves)",
"type": "stat"
}
],
"refresh": "5s",
"schemaVersion": 22,
"style": "dark",
"tags": [],
"templating": {
"list": []
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
]
},
"timezone": "",
"title": "Kafka and Producers Statistics",
"uid": "-28uHcjZk",
"variables": {
"list": []
},
"version": 1
}
Test results - series 1
Below tables show the test results across a wide range of containers' number.
NUMBER OF PRODUCERS | TOTAL MESSAGES PROCESSED | DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES | AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms] | AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] | PEAK INCOMING DATA RATE [MB/s] | PEAK PROCESSING MESSAGE QUEUE SIZE | PEAK CPU LOAD [%] | PEAK MEMORY USAGE [GB] | RESULTS PRESENTED IN GRAFANA |
---|---|---|---|---|---|---|---|---|---|
2 | 180000 | 0 | 0.024 | 1.5 | 1.6 | 3 | 3.5 | 0.37 | |
4 | 360000 | 0 | 0.020 | 1.7 | 3.2 | 3 | 5.7 | 0.37 | |
6 | 540000 | 0 | 0.02 | 2.6 | 4.8 | 24 | 6.0 | 0.37 | ![]() ![]() |
8 | 720000 | 0 | 0.02 | 2.8 | 6.4 | 14 | 8.5 | 0.37 | |
10 | 900000 | 0 | 0.02 | 7.3 | 8.1 | 552 | 8.5 | 0.38 | |
12 | 10800000 | 0 | 0.02 | 78 | 9.7 | 1077 | 7.5 | 0.41 | ![]() ![]() ![]() |
14 | 12600000 | 0 | 6140 | 19000 | 13.0 | 10630 | 9.8 | 0.99 |
Test results - series 2
Below tables show the test results across a wide range of containers' number.
NUMBER OF PRODUCERS | TOTAL MESSAGES PROCESSED | DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES | AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms] | AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms] | PEAK INCOMING DATA RATE [MB/s] | PEAK PROCESSING MESSAGE QUEUE SIZE | PEAK CPU LOAD [%] | PEAK MEMORY USAGE [GB] | RESULTS PRESENTED IN GRAFANA |
---|---|---|---|---|---|---|---|---|---|
2 | 180000 | 0 | 0.025 | 1.7 | 1.6 | 3 | 3.4 | 0.37 | |
4 | 360000 | 0 | 0.021 | 2.1 | 3.2 | 14 | 4.9 | 0.37 | |
6 | 540000 | 0 | 0.02 | 2.5 | 4.8 | 60 | 6.2 | 0.38 | |
8 | 720000 | 0 | 0.02 | 2.9 | 6.4 | 24 | 7.4 | 0.37 | |
10 | 900000 | 0 | 0.018 | 5.5 | 8.0 | 201 | 8.1 | 0.36 | |
12 | 10800000 | 0 | 0.019 | 141.7 | 9.7 | 1716 | 9.1 | 0.44 | |
14 | 12600000 | 0 | 31 | 7568 | 16.0 | 5778 | 8.6 | 0.50 |
- No labels