Setting up Performance Tests in APEX

The apex-pdp has built in support for performance testing. A special performance testing REST server is available in the code base for performance testing. It is in the module performance-benchmark-test. To execute a benchmark test, you start the REST server, and then configure and run APEX against the server. There are example configurations for running tests in the resources of this module.

In order to run the test for 72 hours, set the batch count in the EventGeneratorConfig.json file to zero, which causes the REST server to generate batches forever.

Here is an example of how to do this:

1. Clone and build the apex-pdp git repo

2. Go into the performance-benchmark-test module and run the REST server

   cd testsuites/performance/performance-benchmark-test
   mvn exec:java -Dexec.mainClass="org.onap.policy.apex.testsuites.performance.benchmark.eventgenerator.EventGenerator" -Dexec.args="-c  src/main/resources/examples/benchmark/EventGeneratorConfig.json"

3. Separately, create a local directory and unzip the APEX tarball

   mkdir apex
   cd apex
   tar zxvf ~/git/onap/policy/apex-pdp/packages/apex-pdp-package-full/target/*gz

4. Run APEX with a configuration that runs against the benchmark REST server, select the configuration that is appropriate for the number of threads for the number of cores on the host on which APEX is running. For example on a 32 core machine, select the "32" configuration, on an 8 code machine, select the "8" configuration.

   bin/apexApps.sh engine -c ~/git/onap/policy/apex-pdp/testsuites/performance/performance-benchmark-test/src/main/resources/examples/benchmark/Javascript64.json

5. To get the test results, Issue the following command using CURL or from a browser

   curl http://localhost:32801/EventGenerator/Stats

6. The results are similar to those below:

   Example APEX performance metrics Expand source

    {
     "totalStats": {
       "batchNumber": -1,
       "batchSize": 1000,
       "apexClient": "TOTAL",
       "eventsNotSent": 0,
       "eventsSent": 1000,
       "eventsNotReceived": 0,
       "eventsReceived": 1000,
       "averageRoundTripNano": 159434725,
       "shortestRoundTripNano": 23059921,
       "longestRoundTripNano": 1106994552,
       "averageApexExecutionNano": 26780360,
       "shortestApexExecutionNano": 1066002,
       "longestApexExecutionNano": 323934929
     },
     "batchStatsList": [
       {
         "batchNumber": 0,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38428",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 995018520,
         "shortestRoundTripNano": 835971769,
         "longestRoundTripNano": 1106994552,
         "averageApexExecutionNano": 241706832,
         "shortestApexExecutionNano": 73723978,
         "longestApexExecutionNano": 323934929
       },
       {
         "batchNumber": 1,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38428",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 204429789,
         "shortestRoundTripNano": 56234319,
         "longestRoundTripNano": 305690768,
         "averageApexExecutionNano": 89544892,
         "shortestApexExecutionNano": 7785682,
         "longestApexExecutionNano": 177171848
       },
       {
         "batchNumber": 2,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38430",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 160524927,
         "shortestRoundTripNano": 41179914,
         "longestRoundTripNano": 266809149,
         "averageApexExecutionNano": 68178648,
         "shortestApexExecutionNano": 2784005,
         "longestApexExecutionNano": 160119899
       },
       {
         "batchNumber": 3,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38428",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 123442425,
         "shortestRoundTripNano": 32320688,
         "longestRoundTripNano": 204549694,
         "averageApexExecutionNano": 35810567,
         "shortestApexExecutionNano": 2325630,
         "longestApexExecutionNano": 102168076
       },
       {
         "batchNumber": 4,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38430",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 134903088,
         "shortestRoundTripNano": 61195662,
         "longestRoundTripNano": 205959446,
         "averageApexExecutionNano": 17987078,
         "shortestApexExecutionNano": 2243213,
         "longestApexExecutionNano": 39218327
       },
       {
         "batchNumber": 5,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38428",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 115759297,
         "shortestRoundTripNano": 46415800,
         "longestRoundTripNano": 180262438,
         "averageApexExecutionNano": 12719990,
         "shortestApexExecutionNano": 1868046,
         "longestApexExecutionNano": 32387313
       },
       {
         "batchNumber": 6,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38430",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 130974073,
         "shortestRoundTripNano": 60415786,
         "longestRoundTripNano": 191390626,
         "averageApexExecutionNano": 7577036,
         "shortestApexExecutionNano": 1888337,
         "longestApexExecutionNano": 21813501
       },
       {
         "batchNumber": 7,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38428",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 115724681,
         "shortestRoundTripNano": 23059921,
         "longestRoundTripNano": 170915878,
         "averageApexExecutionNano": 14406756,
         "shortestApexExecutionNano": 1402961,
         "longestApexExecutionNano": 57016237
       },
       {
         "batchNumber": 8,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38430",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 87268333,
         "shortestRoundTripNano": 29857934,
         "longestRoundTripNano": 142114113,
         "averageApexExecutionNano": 8295938,
         "shortestApexExecutionNano": 1292335,
         "longestApexExecutionNano": 22250460
       },
       {
         "batchNumber": 9,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38430",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 89515572,
         "shortestRoundTripNano": 27979472,
         "longestRoundTripNano": 151246340,
         "averageApexExecutionNano": 2988598,
         "shortestApexExecutionNano": 1251419,
         "longestApexExecutionNano": 10779438
       },
       {
         "batchNumber": 10,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38428",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 161559717,
         "shortestRoundTripNano": 54052523,
         "longestRoundTripNano": 220095891,
         "averageApexExecutionNano": 2439668,
         "shortestApexExecutionNano": 1413877,
         "longestApexExecutionNano": 7713140
       },
       {
         "batchNumber": 11,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38428",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 102126750,
         "shortestRoundTripNano": 43249960,
         "longestRoundTripNano": 147393457,
         "averageApexExecutionNano": 4201824,
         "shortestApexExecutionNano": 1343794,
         "longestApexExecutionNano": 20132831
       },
       {
         "batchNumber": 12,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38432",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 94621079,
         "shortestRoundTripNano": 44447504,
         "longestRoundTripNano": 136784978,
         "averageApexExecutionNano": 2053586,
         "shortestApexExecutionNano": 1314753,
         "longestApexExecutionNano": 4677176
       },
       {
         "batchNumber": 13,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38432",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 95569040,
         "shortestRoundTripNano": 45539798,
         "longestRoundTripNano": 131660842,
         "averageApexExecutionNano": 3325890,
         "shortestApexExecutionNano": 1711712,
         "longestApexExecutionNano": 14248153
       },
       {
         "batchNumber": 14,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38432",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 118841955,
         "shortestRoundTripNano": 41387040,
         "longestRoundTripNano": 164058281,
         "averageApexExecutionNano": 8456753,
         "shortestApexExecutionNano": 1066002,
         "longestApexExecutionNano": 40687580
       },
       {
         "batchNumber": 15,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38432",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 98047224,
         "shortestRoundTripNano": 53382064,
         "longestRoundTripNano": 137689355,
         "averageApexExecutionNano": 3103221,
         "shortestApexExecutionNano": 1642711,
         "longestApexExecutionNano": 13629027
       },
       {
         "batchNumber": 16,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38432",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 89551458,
         "shortestRoundTripNano": 43607586,
         "longestRoundTripNano": 125304371,
         "averageApexExecutionNano": 4844014,
         "shortestApexExecutionNano": 1753670,
         "longestApexExecutionNano": 18681287
       },
       {
         "batchNumber": 17,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38432",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 98811396,
         "shortestRoundTripNano": 41244373,
         "longestRoundTripNano": 141306528,
         "averageApexExecutionNano": 2908269,
         "shortestApexExecutionNano": 1398919,
         "longestApexExecutionNano": 10733647
       },
       {
         "batchNumber": 18,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38450",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 85947612,
         "shortestRoundTripNano": 40228871,
         "longestRoundTripNano": 124690954,
         "averageApexExecutionNano": 2324032,
         "shortestApexExecutionNano": 1478128,
         "longestApexExecutionNano": 5606719
       },
       {
         "batchNumber": 19,
         "batchSize": 50,
         "apexClient": "localhost(127.0.0.1):38432",
         "eventsNotSent": 0,
         "eventsSent": 50,
         "eventsNotReceived": 0,
         "eventsReceived": 50,
         "averageRoundTripNano": 86057583,
         "shortestRoundTripNano": 45376757,
         "longestRoundTripNano": 119651361,
         "averageApexExecutionNano": 2733608,
         "shortestApexExecutionNano": 1704628,
         "longestApexExecutionNano": 12086274
       }
     ]
   }