...
| Gliffy Diagram | ||||||
|---|---|---|---|---|---|---|
|
7
DMaaP Simulator
DMaap Simulator is a simple Spring Boot application, which exposes two endpoints. First endpoint (@PostMapping("/events/unauthenticated.SEC_FAULT_OUTPUT")) which receives fault events topic and second (@GetMapping("/summary")) which display events count and average processing time in milliseconds(depending on "startEpochMicrosec").
...
| View file | ||||
|---|---|---|---|---|
|
Replacing Cambria with DMaaP Client
Presentation
| View file | ||||
|---|---|---|---|---|
|
Performance Tests with real DMaaP
Environment 1:
- CPU - 8 cores
- CPU clock speed - 2.4 GHz
- Max Heap - 512 MB
- Start Heap - 256 MB
Ves with Dmaap Client
Test scenario | Description | JMeter test results & metrics | Ves metrics | K8s metrics |
|---|---|---|---|---|
| 3_test_scenario_10_rps_time_300.jmx | 10RPS → 5min |
|
|
|
| 3_test_scenario_10_rps_time_300.jmx | 10RPS → 5min |
|
|
|
| 3_test_scenario_10_rps_time_300.jmx | 10RPS → 5min |
|
|
|
| 3_test_scenario_20_rps_time_300.jmx | 20RPS → 5min |
|
|
|
| 3_test_scenario_20_rps_time_300.jmx | 20RPS → 5min |
|
|
|
| 3_test_scenario_20_rps_time_300.jmx | 20RPS → 5min |
|
|
|
| 3_test_scenario_50_rps_time_300.jmx | 50RPS → 5min |
|
|
|
| 3_test_scenario_50_rps_time_300.jmx | 50RPS → 5min |
|
|
|
| 3_test_scenario_50_rps_time_300.jmx | 50RPS → 5min |
|
|
|
Ves with cambria
Test scenario | 10RPS → 5min | JMeter test results & metrics | Ves metrics | K8s metrics |
|---|---|---|---|---|
| 3_test_scenario_10_rps_time_300.jmx | 10RPS → 5min |
|
|
|
| 3_test_scenario_10_rps_time_300.jmx | 10RPS → 5min |
|
|
|
| 3_test_scenario_10_rps_time_300.jmx | 10RPS → 5min |
|
|
|
| 3_test_scenario_20_rps_time_300.jmx | 20RPS → 5min |
|
|
|
| 3_test_scenario_20_rps_time_300.jmx | 20RPS → 5min |
|
|
|
| 3_test_scenario_20_rps_time_300.jmx | 20RPS → 5min |
|
|
|
| 3_test_scenario_50_rps_time_300.jmx | 50RPS → 5min |
|
|
|
| 3_test_scenario_50_rps_time_300.jmx | 50RPS → 5min |
|
|
|
| 3_test_scenario_50_rps_time_300.jmx | 50RPS → 5min |
|
|
|
Environment 2
- CPU - 24 cores
- CPU clock speed - 2.4 GHz
- Max Heap - 512 MB
- Start Heap - 256 MB
Ves with Dmaap client
Test scenario | 10RPS → 5min | JMeter test results & metrics | Ves metrics | K8s metrics |
|---|---|---|---|---|
| 3_test_scenario_50_rps_time_600.jmx | 50RPS → 10min |
|
|
|
| 3_test_scenario_50_rps_time_600.jmx | 50RPS → 10min |
|
|
|
| 3_test_scenario_50_rps_time_600.jmx | 50RPS → 10min |
|
|
|
| 3_test_scenario_100_rps_time_600.jmx | 100RPS → 10min |
|
|
|
| 3_test_scenario_100_rps_time_600.jmx | 100RPS → 10min |
|
|
|
| 3_test_scenario_100_rps_time_600.jmx | 100RPS → 10min |
|
|
|
Ves with Cambria client
Test scenario | 10RPS → 5min | JMeter test results & metrics | Ves metrics | K8s metrics |
|---|---|---|---|---|
| 3_test_scenario_50_rps_time_600.jmx | 50RPS → 10min |
|
|
|
| 3_test_scenario_50_rps_time_600.jmx | 50RPS → 10min |
|
|
|
| 3_test_scenario_50_rps_time_600.jmx | 50RPS → 10min |
|
|
|
| 3_test_scenario_100_rps_time_600.jmx | 100RPS → 10min |
|
|
|
| 3_test_scenario_100_rps_time_600.jmx | 100RPS → 10min |
|
|
|
| 3_test_scenario_100_rps_time_600.jmx | 100RPS → 10min |
|
|
|
Summary test results:
Environment 1
- CPU - 8 cores
- CPU clock speed - 2.4 GHz
- Max Heap - 512 MB
- Start Heap - 256 MB
| Average and Max Sync Processing Time (Client → VES) | Average VES Processing time | Error Rate [%] | Max CPU Usage [%] | |||
|---|---|---|---|---|---|---|
| VES with Dmaap | 50th percentile | 95th percentile | 99th percentile | |||
| 10RPS → 5min | 127ms, 287ms | 160ms, 1.06s | 180ms, 1.07s | 98ms | 0 | 39 |
| 123ms, 172ms | 137ms, 528ms | 151ms, 544ms | 98ms | 0 | 37 | |
| 123ms, 170ms | 139ms, 513ms | 160ms, 531ms | 98ms | 0 | 37 | |
| 20RPS → 5min | 125ms, 412ms | 157ms, 1.23s | 178ms, 1.33s | 101ms | 0 | 48 |
| 123ms, 412ms | 157ms, 1.12s | 231ms, 1.53s | 101ms | 0 | 53 | |
| 123ms, 215ms | 158ms, 985ms | 221ms, 2.91s | 100ms | 0 | 47 | |
| 50RPS → 5min | 293ms, 3.33s | 484ms, 6.27s | 607ms,6.48s | 269ms | 4 | 95 |
| 281ms, 3.14s | 427ms, 5.99s | 531ms, 7.35s | 270ms | 0 | 94* | |
| 298ms, 3.36s | 463ms, 6.04s | 615ms, 7.48s | 271ms | 0 | 97 | |
| VES with Cambria | 50th percentile | 95th percentile | 99th percentile | |||
| 10RPS → 5min | 123ms, 272ms | 153ms, 1.04s | 172ms, 1.05s | 93ms | 0 | 40 |
| 119ms, 174ms | 135ms, 547ms | 147ms, 547ms | 95ms | 0 | 33 | |
| 119ms, 174ms | 135ms, 538ms | 149ms, 546ms | 93ms | 0 | 34 | |
| 20RPS → 5min | 124ms, 544ms | 152ms, 1.08s | 217ms, 3.86s | 96ms | 0 | 46 |
| 125ms, 595ms | 150ms, 1.08s | 202ms, 3.03s | 96ms | 0 | 49 | |
| 127ms, 682ms | 152ms, 1.18s | 213ms, 3,46s | 97ms | 0 | 46 | |
| 50RPS → 5min | 219ms, 2.6s | 335ms, 3.13s | 504ms, 7.s | 219ms | 144 | 97 |
| 240ms, 3.23s | 353ms, 4.84s | 458ms, 6.38 | 236ms | 144 | 93 | |
| 312ms, 3,82s | 569ms, 6.20s | 774ms, 9.47s | 276ms | 144 | 98 | |
Environment 2
- CPU - 24 cores
- CPU clock speed - 2.4 GHz
- Max Heap - 512 MB
- Start Heap - 256 MB
| Average and Max Sync Processing Time (Client → VES) | Average VES Processing time | Error Rate [%] | Max CPU Usage [%] | |||
|---|---|---|---|---|---|---|
| VES with Dmaap | 50th percentile | 95th percentile | 99th percentile | |||
| 50RPS → 10min | 122ms, 349ms | 181ms, 3.23s | 591ms, 4.43s | 91ms | 0 | 36 |
| 124ms, 515ms | 204ms, 4.27s | 609ms, 5.48s | 90ms | 0 | 42 | |
| 121ms, 399ms | 177ms, 2.17s | 561ms, 5.22 | 91ms | 0 | 30 | |
| 100RPS → 10min | 274ms, 7.06 | 570ms, 7.93s | 1.0s, 8.06s | 139ms | 0.05% | 72 |
| 574ms, 6,07s | 1,17s, 14.19s | 1.89s, 15.37s | 201ms | 0 | 91 | |
| 291ms, 5.88s | 415.5ms, 6.45s | 922ms, 11.3s | 143ms | 0 | 78 | |
| VES with Cambria | 50th percentile | 95th percentile | 99th percentile | |||
| 50RPS → 10min | 118ms, 520ms | 174ms, 2.10s | 571ms, 6.48s | 90ms | 0 | 48 |
| 122ms, 548ms | 230ms, 5.17s | 581ms, 5.54s | 88ms | 0 | 32 | |
| 123ms, 557ms | 194ms, 2.50s | 676ms, 5.12s | 88ms | 0 | 42 | |
| 100RPS → 10min | 301ms, 5.79s | 772ms, 16.97s | 1.16s, 17.08s | 153ms | 0 | 88 |
| 340ms, 7.13s | 636ms, 17.87s | 1.15s, 18.48s | 149ms | 0 | 76 | |
| 307ms, 8.29s | 506ms, 9.44s | 855ms, 9.78s | 155ms | 0 | 88 | |
Conclusion:
Results of performance tests for both VES collector client implementation (DMaap client and Cambria client) are very similar.
Max CPU usage, Error rate, Average VES processing time, average and max sync processing time(Client → Ves) are almost the same.
In Ves collector with DMaap client(100RPS for 10 min), appeared an error with the connection pool. In that specific case, we had a connection poll set to 16 and we got an error that the connection poll limits have been reached (stack trace in attachment).
We have to handle that kind of error in code.