Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Expand
titleClick here to see results...


NUMBER OF PRODUCERSTOTAL MESSAGES PROCESSEDDIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VESAVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms]AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms]PEAK INCOMING DATA RATE [MB/s]PEAK PROCESSING MESSAGE QUEUE SIZEPEAK CPU LOAD [%]PEAK MEMORY USAGE [GB]RESULTS PRESENTED IN GRAFANA
218000000.0268.91.61130.35

218000000.02611.31.6233.20.35


NUMBER OF PRODUCERSTOTAL MESSAGES PROCESSEDDIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VESAVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [s]AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms]PEAK INCOMING DATA RATE [MB/s]PEAK PROCESSING MESSAGE QUEUE SIZEPEAK CPU LOAD [%]PEAK MEMORY USAGE [GB]RESULTS PRESENTED IN GRAFANA
436000000.02241.33.213040.35

436000000.02340.73.23704.20.35


NUMBER OF PRODUCERSTOTAL MESSAGES PROCESSEDDIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VESAVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms]AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms]PEAK INCOMING DATA RATE [MB/s]PEAK PROCESSING MESSAGE QUEUE SIZEPEAK CPU LOAD [%]PEAK MEMORY USAGE [GB]RESULTS PRESENTED IN GRAFANA
63593176836240159654.845004.31

63592177836490168344.842005.81



Without DMaaP Kafka

Conditions

Environment was prepared using: Performance test without DMaaP - instruction.

To gather tcpdump data another container was added to hv-ves deployment in kubernetes and producer-pod.yaml (command to get tcpdump data file: kubectl cp -n onap <pod-name>:/tcpdump.pcap -c tcpdump ./<pod-name>.pcap):

Code Block
titleTcpdump container
collapsetrue
...
containers:
  ...
<<default containers>>
  ...
- name: tcpdump
  image: onap-dev-local.esisoj70.emea.nsn-net.net/rjanecze/my-tcpdump:1.0.7
...

Tests were performed with following configuration:

Messages per producerPayload size [B]Interval [ms]
90000819210

Raw results data

Raw results data with screenshots can be found in following files:

To see custom Kafka metrics you may want to change kafka-and-producers.json (located in HV-VES project directory: tools/performance/cloud/grafana/dashboards) to

Expand
titlekafka-and-producers.json
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"gnetId": null,
"graphTooltip": 0,
"id": 3,
"links": [],
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 0,
"fillGradient": 0,
"gridPos": {
"h": 11,
"w": 8,
"x": 0,
"y": 0
},
"hiddenSeries": false,
"id": 12,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(irate(container_network_transmit_bytes_total{pod=~\".*producer.*\"}[1m]))",
"interval": "",
"legendFormat": "Total Transmit Producers",
"refId": "A"
},
{
"expr": "sum(irate(container_network_transmit_bytes_total{pod=~\".*kafka-broker.*\"}[1m]))",
"interval": "",
"legendFormat": "Total Transmit Kafka",
"refId": "B"
},
{
"expr": "irate(container_network_transmit_bytes_total{pod=~\".*producer.*\"}[1m])",
"interval": "",
"legendFormat": "{{pod}}",
"refId": "C"
},
{
"expr": "irate(container_network_transmit_bytes_total{pod=~\".*kafka-broker.*\"}[1m])",
"interval": "",
"legendFormat": "{{pod}}",
"refId": "D"
}
],
"thresholds": [],
"timeFrom": "20m",
"timeRegions": [],
"timeShift": null,
"title": "Network Usage",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "Bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 0,
"fillGradient": 0,
"gridPos": {
"h": 11,
"w": 8,
"x": 8,
"y": 0
},
"hiddenSeries": false,
"id": 22,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(rate(container_network_transmit_bytes_total{pod=~\".*kafka-broker.*|.*hv-ves-coll.*|.*producer.*\"}[1m]))",
"hide": false,
"interval": "",
"legendFormat": "Total transmit",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": "20m",
"timeRegions": [],
"timeShift": null,
"title": "Total network usage (producers + hv-ves + kafka)",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "Bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 0,
"fillGradient": 0,
"gridPos": {
"h": 11,
"w": 8,
"x": 16,
"y": 0
},
"hiddenSeries": false,
"id": 2,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "irate(container_network_receive_bytes_total{pod=~\"dep-dcae-hv-ves-collector.*\"}[1m])",
"hide": false,
"interval": "",
"legendFormat": "receive",
"refId": "A"
},
{
"expr": "irate(container_network_transmit_bytes_total{pod=~\"dep-dcae-hv-ves-collector.*\"}[1m])",
"interval": "",
"legendFormat": "transmit",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": "20m",
"timeRegions": [],
"timeShift": null,
"title": "HV-VES Network Usage",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"decimals": null,
"format": "Bps",
"label": "",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 0,
"fillGradient": 0,
"gridPos": {
"h": 12,
"w": 8,
"x": 0,
"y": 11
},
"hiddenSeries": false,
"id": 14,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum (rate (container_cpu_usage_seconds_total{pod=~\".*producer.*\"}[1m])) by (pod)",
"interval": "",
"legendFormat": "{{pod}}",
"refId": "A"
},
{
"expr": "sum (rate (container_cpu_usage_seconds_total{pod=~\".*kafka-broker.*\"}[1m])) by (pod)",
"interval": "",
"legendFormat": "{{pod}}",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": "20m",
"timeRegions": [],
"timeShift": null,
"title": "CPU usage [cores]",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "none",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 0,
"fillGradient": 0,
"gridPos": {
"h": 12,
"w": 8,
"x": 8,
"y": 11
},
"hiddenSeries": false,
"id": 18,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(container_memory_working_set_bytes{image!=\"\", pod=~\".*kafka-broker.*\"}) by (pod)",
"interval": "",
"legendFormat": "{{pod}}",
"refId": "A"
},
{
"expr": "sum(container_memory_working_set_bytes{image!=\"\", pod=~\".*producer.*\"}) by (pod)",
"hide": false,
"interval": "",
"legendFormat": "{{pod}}",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": "20m",
"timeRegions": [],
"timeShift": null,
"title": "Memory Usage",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"datasource": null,
"gridPos": {
"h": 6,
"w": 4,
"x": 16,
"y": 11
},
"id": 15,
"options": {
"colorMode": "value",
"fieldOptions": {
"calcs": [
"lastNotNull"
],
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": [],
"values": false
},
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto"
},
"pluginVersion": "6.7.3",
"targets": [
{
"expr": "sum(container_network_transmit_packets_dropped_total{pod=~\".*producer.*\"})",
"interval": "",
"legendFormat": "Packets dropped",
"refId": "A"
}
],
"timeFrom": null,
"timeShift": null,
"title": "Packets Dropped (producers)",
"type": "stat"
},
{
"datasource": null,
"gridPos": {
"h": 6,
"w": 4,
"x": 20,
"y": 11
},
"id": 4,
"options": {
"colorMode": "value",
"fieldOptions": {
"calcs": [
"lastNotNull"
],
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": [],
"values": false
},
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto"
},
"pluginVersion": "6.7.3",
"targets": [
{
"expr": "sum(container_network_transmit_packets_dropped_total{pod=~\".*hv-ves-collector.*\"})",
"interval": "",
"legendFormat": "Packets dropped",
"refId": "A"
}
],
"timeFrom": null,
"timeShift": null,
"title": "Packets Dropped (hv-ves)",
"type": "stat"
},
{
"datasource": null,
"gridPos": {
"h": 6,
"w": 4,
"x": 16,
"y": 17
},
"id": 17,
"options": {
"colorMode": "value",
"fieldOptions": {
"calcs": [
"max"
],
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "bytes"
},
"overrides": [],
"values": false
},
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto"
},
"pluginVersion": "6.7.3",
"targets": [
{
"expr": "sum(container_network_transmit_bytes_total{pod=~\".*producer.*\"})",
"interval": "",
"legendFormat": "",
"refId": "A"
}
],
"timeFrom": null,
"timeShift": null,
"title": "Sent (producers)",
"type": "stat"
},
{
"cacheTimeout": null,
"datasource": null,
"gridPos": {
"h": 6,
"w": 4,
"x": 20,
"y": 17
},
"id": 16,
"links": [],
"options": {
"colorMode": "value",
"fieldOptions": {
"calcs": [
"max"
],
"defaults": {
"mappings": [
{
"id": 0,
"op": "=",
"text": "N/A",
"type": 1,
"value": "null"
}
],
"nullValueMode": "connected",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "bytes"
},
"overrides": [],
"values": false
},
"graphMode": "none",
"justifyMode": "auto",
"orientation": "horizontal"
},
"pluginVersion": "6.7.3",
"targets": [
{
"expr": "container_network_receive_bytes_total{pod=~\".*hv-ves-collector.*\"}",
"interval": "",
"legendFormat": "Packets dropped",
"refId": "A"
}
],
"timeFrom": null,
"timeShift": null,
"title": "Received (hv-ves)",
"type": "stat"
}
],
"refresh": "5s",
"schemaVersion": 22,
"style": "dark",
"tags": [],
"templating": {
"list": []
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
]
},
"timezone": "",
"title": "Kafka and Producers Statistics",
"uid": "-28uHcjZk",
"variables": {
"list": []
},
"version": 1
}


Test results - series 1

Expand
titleClick here to see results...

Below tables show the test results across a wide range of containers' number. 

NUMBER OF PRODUCERSTOTAL MESSAGES PROCESSEDDIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VESAVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms]AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms]PEAK INCOMING DATA RATE [MB/s]PEAK PROCESSING MESSAGE QUEUE SIZEPEAK CPU LOAD [%]PEAK MEMORY USAGE [GB]RESULTS PRESENTED IN GRAFANA
218000000.0241.51.633.50.37

Image AddedImage AddedImage AddedImage AddedImage AddedImage Added

436000000.0201.73.235.70.37

Image AddedImage AddedImage AddedImage AddedImage AddedImage Added

654000000.022.64.8246.00.37
Image AddedImage Added

Image AddedImage AddedImage AddedImage Added

872000000.022.86.4148.50.37

Image AddedImage AddedImage AddedImage AddedImage AddedImage Added

1090000000.027.38.15528.50.38

Image AddedImage AddedImage AddedImage AddedImage AddedImage Added

121080000000.02789.710777.50.41

Image AddedImage AddedImage Added

Image AddedImage Added
Image Added
1412600000061401900013.0106309.80.99

Image AddedImage AddedImage AddedImage AddedImage AddedImage Added


Test results - series 2

Expand
titleClick here to see results...

Below tables show the test results across a wide range of containers' number. 

NUMBER OF PRODUCERSTOTAL MESSAGES PROCESSEDDIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VESAVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms]AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms]PEAK INCOMING DATA RATE [MB/s]PEAK PROCESSING MESSAGE QUEUE SIZEPEAK CPU LOAD [%]PEAK MEMORY USAGE [GB]RESULTS PRESENTED IN GRAFANA
218000000.0251.71.633.40.37

Image AddedImage AddedImage AddedImage AddedImage AddedImage Added

436000000.0212.13.2144.90.37

Image AddedImage AddedImage AddedImage AddedImage AddedImage Added

654000000.022.54.8606.20.38

Image AddedImage AddedImage AddedImage AddedImage AddedImage Added

872000000.022.96.4247.40.37

Image AddedImage AddedImage AddedImage AddedImage AddedImage Added

1090000000.0185.58.02018.10.36

Image AddedImage AddedImage AddedImage AddedImage AddedImage Added

121080000000.019141.79.717169.10.44

Image AddedImage AddedImage AddedImage AddedImage AddedImage Added

1412600000031756816.057788.60.50

Image AddedImage AddedImage AddedImage AddedImage AddedImage Added