Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

CSIT and Testsuite Tests

Triggered by merges in policy/distribution

...

Test Case Id

...

Description

...

Pre-conditions

...

Test Steps

...

Expected Results

...

1

...

Perform healthcheck of the component

...

distribution

...

API – healthcheck

Method - GET

...

https://gerrit.onap.org/r/gitweb?p=integration/csit.git;a=blob;f=tests/policy/distribution/distribution-test.robot;h=2b1c6d64eb6eebf878e8795b9db0704e26bf5e5f;hb=refs/heads/master

<host>:6969/policy/distribution/v1/healthcheckQuery component for statistics - statisticsstatistics the current statistics of the component.

Test Case Id

Description

Pre-conditions

Test Steps

Expected Results

CSIT/External Lab

1

Perform healthcheck of the component

The component should return health status as “true”

CSIT - DONE

External Lab

2

Policy distribution docker image is available

API

– healthcheck

Method - GET

Endpoint: https://<host>:6969/policy/distribution/v1/

healthcheck

The component should return

health status as “true”

CSIT - DONE

Triggered by merges in policy/pap

External Lab

2Query component for statisticsPolicy distribution docker

Test Case Id

Description

Pre-conditions

Test Steps

Expected Results

CSIT/External Lab1Perform healthcheck of the componentPAP docker
image is available

API

– healthcheck

- statistics

Method - GET

Endpoint: https://<host>:6969/policy/

pap

distribution/v1/

healthcheck

statistics

The component should return
health status as “true”
the current statistics of the component.CSIT - DONE
3InvokeDistributionUsingFile And RunEventOnApexEngine



Triggered by merges in policy/pap

https://gerrit.onap.org/r/gitweb?p=integration/csit.git;a=blob;f=tests/policy/pap/pap-test.robot;h=67a578281ba8032a5926fbed5baf28890d1a427e;hb=refs/heads/master

External Lab - DONE

Test Case Id

Description

Pre-conditions

Test Steps

Expected Results

CSIT/External Lab
1Perform all component Healthcheck
2Query component for statisticsPAP docker image is available

API - statistics

Method - GET

Endpoint: https://<host>:6969/policy/pap/v1/statistics

The component should return the current statistics of the component.

CSIT - DONE

External Lab

3Query pdp group informationPAP docker image is available

API - pdpscomponent healthcheck

Method METHOD - GET

Endpoint: https:{url}:{port}//<host>:6969/policy/pap/v1/components/pdps

The component should return the list of PDP groups and subgroups together with the policies that are deployed on each PDP group and subgroup.CSIT - DONE

healthcheck


CSIT - 200 but only the pap should return "healthy" since not all the components are running.

TESTSUITE: 200 and all components should return "healthy"

CSIT - TBD

Testsuite - 

Jira
serverONAP JIRA
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyINT-1509

1Perform healthcheck 4Create PDP group & subgroupPAP docker image is available

API - pdps – healthcheck

Method - POSTGET

Endpoint: https:{url}:{port}//<host>:6969/policy/pap/v1/pdpshealthcheck

The component should create the relevant PDP group & subgroup in the database and return with operation success message/code.return health status as “true”

CSIT - DONE

External Lab - DONE

2Query component for statisticsPAP 5Create a PolicyAPI docker image is available

API policytypes/onap.policies.Monitoring.cdap.tca.hi.lo.app/versions/1.0.0/policies

Method - POST

Endpoint: http://<host>:6969/policy/api/v1/policytypes/onap.policies.Monitoring.cdap.tca.hi.lo.app/versions/1.0.0/policies

The Policy API should return code 201 and message "A new policy has been successfully created."CSIT

- statistics

Method - GET

Endpoint: https://<host>:6969/policy/pap/v1/statistics

The component should return the current statistics of the component.

CSIT - DONE

External Lab

3Query pdp group information6Deploy policy to PDP group & subgroupPAP docker image is available

API docker image is available

DMaaP simulator docker image is needed; a PDP can be simulated using curl

API - - pdps

Method - POSTGET

Endpoint: https:{url}:{port}/policy/pap/v1/pdps

The component should map the relevant policies with the corresponding PDP group & subgroup in database. Deploy the policy in relevant PDP and return with operation success message/code.CSITreturn the list of PDP groups and subgroups together with the policies that are deployed on each PDP group and subgroup.CSIT - DONE
4Create PDP 7UNDeploy policy to PDP group & subgroupPAP docker image is availableAPI docker image is available
NOTE: PDP's not needed for this test in Dublin.

API - pdps

Method - POST

Endpoint: https:{url}:{port}/policy/pap/v1/pdps

Based on test #5 - undeploy those policiesThe component should create the relevant PDP group & subgroup in the database and return with operation success message/code.CSIT CSIT - DONE8Query pdp group statistics
5Create a PolicyAPI

PAP docker image is available

PDP docker image is available

API - pdps

Method - GET

Endpoint: https:{url}:{port}/policy/pap/v1/pdps/statistics

The PAP component must return the statistics for PDP groups, subgroups and individual PDPs.

CSIT (Not done in this release)

The API will be implemented in next release.

9Delete PDP group & subgroup

PAP docker image is available

PDP docker image is available

API - pdps

Method - DELETE

Endpoint: https:{url}:{port}/policy/pap/v1/pdps?name=<group_name>&version=<version>

The component should delete the relevant PDP group & subgroup from database and kill the corresponding PDP instances (running as kubernetes pods). Finally, return with operation success message/code.

CSIT (Not done in this release)

The API will be implemented in next release.

Triggered by merges in policy/drools-pdp

policytypes/onap.policies.Monitoring.cdap.tca.hi.lo.app/versions/1.0.0/policies

Method - POST

Endpoint: http://<host>:6969/policy/api/v1/policytypes/onap.policies.Monitoring.cdap.tca.hi.lo.app/versions/1.0.0/policies

The Policy API should return code 201 and message "A new policy has been successfully created."CSIT
6Deploy policy to PDP group & subgroup

PAP docker image is available

API docker image is available

DMaaP simulator docker image is needed; a PDP can be simulated using curl


API - pdps

Method - POST

Endpoint: https:{url}:{port}/policy/pap/v1/pdps

The component should map the relevant policies with the corresponding PDP group & subgroup in database. Deploy the policy in relevant PDP and return with operation success message/code.CSIT
7UNDeploy policy to PDP group & subgroup

PAP docker image is available

API docker image is available

NOTE: PDP's not needed for this test in Dublin.

API - pdps

Method - POST

Endpoint: https:{url}:{port}/policy/pap/v1/pdps

Based on test #5 - undeploy those policiesCSIT - DONE
8Query pdp group statistics

PAP docker image is available

PDP docker image is available

API - pdps

Test Case Id

Description

Pre-conditions

Test Steps

Expected Results

CSIT/External Labs1AliveVerify that base barebone PDP-D comes alive

API – telemetry

Method - GET

Endpoint: https://<host>:9696/policy/pdpd/v1/engine

Verify alive fieldCSIT-DONE2 (stretch)Dynamic Controller CreationVerify dynamic addition of a PDP-D controller

API – telemetry

Method - GET

Endpoint: https://<host>:9696/policy/pdpd/v1/controllers/<controller>

Verify that the dynamically create controller is alive.CSIT3 (stretch)Dynamic Controller TrafficVerify that PDP-D can process traffic for the controller created in step-2.API – telemetry

Method - GET

Endpoint: https:

//<host>:9696

{url}:{port}/policy/

pdpd

pap/v1/

controllers

pdps/

<controller>/droolsVerify that the events structure contains input and expected output messages.CSIT

Triggered by merges in policy/drools-applications

...

Test Case Id

...

Description

...

Pre-conditions

...

Test Steps

...

Expected Results

...

API – telemetry

Method - GET

Endpoint: https://<host>:6969/policy/pdpd/v1/engine/healthcheck

...

Verify vFW use case.

Simulators should be installed.

...

  1. Add vFW Operational Policy
  2. Inject ONSET
  3. Trace the vFW use case across the complete sequence of messages

...

Verify Scale out use case.

Simulators should be installed.

...

  1. Add vDNS Operational Policy
  2. Inject ONSET
  3. Trace the vDNS use case across complete sequence of messages

...

Verify vCPE use case.

Simulators should be installed.

...

  1. Add vCPE Operational Policy
  2. Inject ONSET
  3. Trace the vCPE use case across the complete sequence of messages

...

statistics

The PAP component must return the statistics for PDP groups, subgroups and individual PDPs.

CSIT (Not done in this release)

The API will be implemented in next release.

9Delete PDP group & subgroup

PAP docker image is available

PDP docker image is available

API - pdps

Method - DELETE

Endpoint: https:{url}:{port}/policy/pap/v1/pdps?name=<group_name>&version=<version>

The component should delete the relevant PDP group & subgroup from database and kill the corresponding PDP instances (running as kubernetes pods). Finally, return with operation success message/code.

CSIT (Not done in this release)

The API will be implemented in next release.


Triggered by merges in policy/drools-pdp

https://gerrit.onap.org/r/gitweb?p=integration/csit.git;a=blob;f=tests/policy/drools-pdp/drools-pdp-test.robot;h=cf4d3ceac87dde285d997a3885ebb49bc4f16cb9;hb=refs/heads/master

Test Case Id

Description

Pre-conditions

Test Steps

Expected Results

CSIT/External Labs
1AliveVerify that a PDP-D with no applications comes alive

API – telemetry

Method - GET

Endpoint: https://<host>:9696/policy/pdpd/v1/engine

Verify alive fieldCSIT-DONE


Triggered by merges in policy/drools-applications

https://gerrit.onap.org/r/gitweb?p=integration/csit.git;a=blob;f=tests/policy/drools-applications/drools-applications-test.robot;h=c3586cfa9d54daf6829cba3dc27bc81b56e8d24d;hb=refs/heads/master

Test Case Id

Description

Pre-conditions

Test Steps

Expected Results

CSIT/External Labs
1AliveVerify that the PDP-D + Control Loop Apps come alive

API – telemetry

Method - GET

Endpoint: https://<host>:9696/policy/pdp/engine

Verify alive field is set to 'true'CSIT - DONE
2HealthcheckVerify that the PDP-D + Control Loop Apps passes healthchecks

API – telemetry

Method - GET

Endpoint: https://<host>:6969/healthcheck

Verify that the healthy field at the root object is set to 'true'CSIT - DONE
3FrankfurtVerify that the PDP-D + Control Loop Apps - Frankfurt controller has been successfully brought up

API – telemetry

Method - GET

Endpoint: https://<host>:6969/policy/pdp/engine/controllers/frankfurt/drools/facts

Verify that a 200 OK is received indicating that there is a drools session associated with this controller that has been successfully instantiated.CSIT - DONE
4vFW

Verify vFW use case.

Simulators should be installed.

  1. Add vFW Operational Policy
  2. Inject ONSET
  3. Trace the vFW use case across the complete sequence of messages
Verify vFirewall completes successfullyCSIT
5vDNS

Verify Scale out use case.

Simulators should be installed.

  1. Add vDNS Operational Policy
  2. Inject ONSET
  3. Trace the vDNS use case across complete sequence of messages
Verify scale out completes successfullyCSIT
6vCPE

Verify vCPE use case.

Simulators should be installed.

  1. Add vCPE Operational Policy
  2. Inject ONSET
  3. Trace the vCPE use case across the complete sequence of messages
Verify vCPE completes successfullyCSIT

Triggered by merges in policy/xacml-pdp

https://gerrit.onap.org/r/gitweb?p=integration/csit.git;a=blob;f=tests/policy/xacml-pdp/xacml-pdp-test.robot;h=0305af8e9fd2ba91d5711680f3be28efb829b37e;hb=refs/heads/master

Test Case Id

Description

Pre-conditions

Test Steps

Expected Results

CSIT/External Lab
1Health CheckXacml-PDP docker image available

API – healthcheck

Method - GET

Endpoint: http://<host>:6969/policy/pdpx/v1/healthcheck

The component should return health status as “true”

Status code 200

CSIT

External Lab

2StatisticsXacml-PDP docker image available

API – statistics

Method - GET

Endpoint: http://<host>:6969/policy/pdpx/v1/statistics

The Xacml PDP should return statistics report consisting of 0 policies loaded and 0 decisions

Status code 200


CSIT
3

ExecuteXacmlPolicy





3a

CreateNewMonitorPolicy

API docker image available

APIpolicytypes/onap.policies.Monitoring.cdap.tca.hi.lo.app/versions/1.0.0/policies

Method - POST

Endpoint: http://<host>:6969/policy/api/v1/policytypes/onap.policies.Monitoring.cdap.tca.hi.lo.app/versions/1.0.0/policies

The Policy API should return code 201 and message "A new policy has been successfully created."CSIT
3b

DeployMonitorPolicy

Xacml-PDP and PAP docker images available

Policy created on the PAP

API – deployPolicy

Method - POST

Endpoint: http://<host>:6969/policy/pdpx/v1/deployPolicy

Successfully deploy a policy to the XACML PDP

Statistics should increment policy count

Status code 201

CSIT
3c

GetAbbreviatedDecisionResult

Xacml-PDP docker image available

Test Case 4 success

API – decision

Method - POST

Triggered by merges in policy/xacml-pdp

API – decision

Method - POST

Endpoint: http

Test Case Id

Description

Pre-conditions

Test Steps

Expected Results

CSIT/External Lab
1Health CheckXacml-PDP docker image available

API – healthcheck

Method - GET

Endpoint: http://<host>:6969/policy/pdpx/v1/healthcheckdecision?abbrev=true

The component should return health status as “true”Xacml PDP should return a Decision object containing the decision and abbreviated policy results

Status code 200

CSIT

External Lab

3c

GetAbbreviatedDecisionResult

2Statistics

Xacml-PDP docker image available

Test Case 4 success

APIstatisticsdecision

Method - GETPOST

Endpoint: http://<host>:6969/policy/pdpx/v1/statisticsdecision?abbrev=true

The Xacml PDP should return statistics report consisting of 0 policies loaded and 0 decisionsa Decision object containing the decision and abbreviated policy results

Status code 200

CSIT/External Lab (stretch)
3d

GetDecision

Xacml-PDP

3

ExecuteXacmlPolicy

3a

CreateNewMonitorPolicy

API

docker image available

Test Case 5 success

API – decisionAPIpolicytypes/onap.policies.Monitoring.cdap.tca.hi.lo.app/versions/1.0.0/policies

Method - POST

Endpoint: http://<host>:6969/policy/apipdpx/v1/policytypes/onap.policies.Monitoring.cdap.tca.hi.lo.app/versions/1.0.0/policies

The Policy API should return code 201 and message "A new policy has been successfully created."CSIT
3b

DeployMonitorPolicy

Xacml-PDP and PAP docker images available

Policy created on the PAP

API – deployPolicy

Method - POST

Endpoint: http://<host>:6969/policy/pdpx/v1/deployPolicy

Successfully deploy a policy to the XACML PDP

Statistics should increment policy count

Status code 201

CSIT
3c

GetAbbreviatedDecisionResult

Xacml-PDP docker image available

Test Case 4 success

/decision

The Xacml PDP should return a Decision object containing the decision

Status code 200

CSIT/External Lab (stretch)


Triggered by merges in policy/apex-pdp

https://gerrit.onap.org/r/gitweb?p=integration/csit.git;a=blob;f=tests/policy/apex-pdp/apex-pdp-test.robot;h=9956731d585a7d0ef7157a8c73c41e9888e70cdb;hb=refs/heads/master

Test Case Id

Description

Pre-conditions

Test Steps

Expected Results

CSIT/External Labs

1

Perform healthcheck of the component


Apex PDP docker image is available

API – healthcheck

Method - GET

Endpoint: https://<host>:6969/policy/

pdpx

apex-pdp/v1/

decision?abbrev=true

healthcheck

The

Xacml PDP should return a Decision object containing the decision and abbreviated policy results

component should return health status as “true”

Status code 200

CSIT

3c

GetAbbreviatedDecisionResult

- DONE

External Lab

2

ExecuteApexPolicy





2a

CreateOperationalPolicyType

API
Xacml-PDP
docker image available
Test Case 4 success

API

decision

policytypes

Method - POST

Endpoint: http

://<host>:6969/policy/pdpx/v1/decision?abbrev=true

The Xacml PDP should return a Decision object containing the decision and abbreviated policy results

Status code 200

CSIT/External Lab (stretch)3d

GetDecision

Xacml-PDP docker image available

Test Case 5 success

API – decision

://<host>:6969/policy/api/v1/policytypes

200

CSIT - DONE

External Lab

2b

CreateNewOperationalPolicy

API docker image available

API – policytypes/onap.policies.controlloop.operational.Apex/versions/1.0.0/policies

Method - POST

Endpoint: http://<host>:6969/policy/

pdpx

api/v1/

decision

The Xacml PDP should return a Decision object containing the decision

Status code 200

CSIT/External Lab (stretch)

Triggered by merges in policy/apex-pdp

policytypes/onap.policies.controlloop.operational.Apex/versions/1.0.0/policies

200

CSIT - DONE

External Lab

2c

DeployOperationalPolicy

APEX-PDP and PAP docker images available

DMaaP Simulator available.

Policy created 

API – pdps/deployments/batch

Method - POST

Endpoint: http

Test Case Id

Description

Pre-conditions

Test Steps

Expected Results

CSIT/External Labs

1

Perform healthcheck of the component

Apex PDP docker image is available

API – healthcheck

Method - GET

Endpoint: https

://<host>:6969/policy/

apex-pdp

pap/v1/pdps/

healthcheckThe component should return health status as “true”

deployments/batch

200

CSIT - DONE

External Lab

2Query component for statisticsApex
2d

RunEventOnApexEngine

APEX-PDP docker image

is

available

API - statistics

2c succeeds

API – /apex/FirstConsumer/EventIn

Method -

GET

POST

Endpoint:

https

 http://

<host>:6969/policy

<APEX_IP>:23324/apex

-pdp

/

v1

FirstConsumer/

statisticsThe component should return the current statistics of the component.CSIT (Stretch)

3

Create Operational policies for the following use cases  

  • SampleDomain
  • BBS (if this is not a POC)

Policy components up and running

(Will be added once API page approved)

(Current integration tests for REST client moved to CSIT)

Operational Policy should be created in apex-pdp

CSIT (Stretch)

...

EventIn

200

CSIT - DONE

External Lab


3Query component for statisticsApex PDP docker image is available

API - statistics

Method - GET

Endpoint: https://<host>:6969/policy/apex-pdp/v1/statistics

The component should return the current statistics of the component.CSIT (Stretch)

Triggered by merges in policy/api

https://gerrit.onap.org/r/gitweb?p=integration/csit.git;a=blob;f=tests/policy/api/api-test.robot;h=76c1bdc6fd5096d2cb3d65300494dd5abd872c2b;hb=refs/heads/master


CSIT

Test Case Id

Description

Pre-conditions

Test Steps

Expected Results

CSIT External Labs

1

Healthcheck

Perform healthcheck for policy design API service


  • Policy API docker image available

API – healthcheck

Method - GET

Endpoint: http://<host>:6969/policy/api/v1/healthcheck

200

CSIT

External Lab

2

Statistics

Retrieve statistics of policy design API invocation

  • Policy API docker image available

API – statistics

Method - GET

Endpoint: http://<host>:6969/policy/api/v1/statistics

200

CSIT

External Lab

3

RetrievePolicyTypes

Retrieve pre-loaded generic policy types

  • Policy API docker image available

API – policytypes

Method - GET

Endpoint: http://<host>:6969/policy/api/v1/policytypes

200 - all preloaded policy typesCSIT
4

CreateTCAPolicyTypeV1

  • Policy API docker image available

API – policytypes

Method - GET

Endpoint: http://<host>:6969/policy/api/v1/policytypes

406 - policy type v1 is already preloadedCSIT
5

CreateTCAPolicyTypeV2

  • Policy API docker image available

API – policytypes

Method - GET

Endpoint: http://<host>:6969/policy/api/v1/policytypes

200 - v2 createdCSIT
6

RetrieveMonitoringPolicyTypes

  • Policy API docker image available

API – policytypes

Method - GET

Endpoint: http://<host>:6969/policy/api/v1/policytypes

200 - both v1 and v2 retrievedCSIT
76

CreateNewMonitoringPolicyV1

Create a new TCA policy type for DCAE TCA microservice

RetrieveMonitoringPolicyTypes

  • Policy API docker image available
  • DCAE TCA ms policy type is not created yet

API – policytypes

Method - GETPOST

Endpoint: http://<host>:6969/

/policy/api/v1/policytypes

/onap.policies.monitoring.cdap.tca.hi.lo.app/versions/1.0.0/policies

200200 - both v1 and v2 retrievedCSIT
78

SimpleCreateNewMonitoringPolicyV1

Trying to create an existing policy with any change and same version should cause error.

CreateNewMonitoringPolicyV1

Create a new TCA policy type for DCAE TCA microservice

  • Policy API docker image available
  • DCAE TCA ms policy type v1 is not created yet

API – policytypes

Method - POST

/policy/api/v1/policies

406 - must supply new version

CSIT
9

SimpleCreateNewMonitoringPolicyV2

Create a new Monitoring TCA policiy version 2 using simple endpoint

Endpoint: http://<host>:6969/


/policy/api/v1/policytypes/onap.policies.monitoring.cdap.tca.hi.lo.app/versions/1.0.0/policies

200 - v2 createdCSIT
810

RetrievePoliciesOfType

Retrieve Monitoring related policy types

SimpleCreateNewMonitoringPolicyV1

Trying to create an existing policy with any change and same version should cause error.

  • Policy API docker image available
  • DCAE TCA ms policy type v1 is and v2 are created

/policy/api/v1/policies

406 - must supply new version

CSIT9

API – policytypes/onap.policies.Monitoring

Method - GET

Endpoint: http://<host>:6969/

SimpleCreateNewMonitoringPolicyV2

Create a new Monitoring TCA policiy version 2 using simple endpoint

/policy/api/v1/policytypes/onap.policies.monitoring.cdap.tca.hi.lo.app/versions/1.0.0/policies

200 - contains both v1 and v2 createdCSIT
1011, 12

RetrievePoliciesOfTypeDeleteSpecificPolicyV1

Retrieve Monitoring related policy types

Delete the Monitoring Policy Version 1 of the TCA Policy Type

  • Policy API docker image available
  • DCAE TCA ms policy type v1 and v2 are created
API – policytypes

/

onap.policies.Monitoring

Method - GET

Endpoint: http://<host>:6969/

/policy/api/v1/policytypes/onap.policies.monitoring.cdap.tca.hi.lo.app/versions/1.0.0/policies/onap.restart.tca/versions/1.0.0

200 -

contains both v1 and v2

delete successful

404 - not found on 2nd delete attempt

CSIT
1113, 1214

DeleteSpecificPolicyV1DeleteSpecificPolicyV2

Delete the Monitoring Policy Version 1 2 of the TCA Policy Type

  • Policy API docker image available
  • DCAE TCA ms policy type v1 and v2 are is created

/policy/api/v1/policytypes/onap.policies.monitoring.cdap.tca.hi.lo.app/versions/1.0.0/policies/onap.restart.tca/versions/12.0.0

200 - delete successful

404 - not found on 2nd delete attempt

CSIT
1315, 1416

DeleteSpecificPolicyTypeV1

DeleteSpecificPolicyV2

Delete the Monitoring Policy Version 2 of the TCA Policy Type

  • Policy API docker image available
  • DCAE TCA ms policy type v2 v1 is created

/policy/api/v1/policytypes/onap.policies.monitoring.cdap.tca.hi.lo.app/versions/1.0.0/policies/onap.restart.tca/versions/2/onap.policies.monitoring.cdap.tca.hi.lo.app/versions/1.0.0

200 - delete successful

404 - not found on 2nd delete attempt

CSIT
1517, 1618

DeleteSpecificPolicyTypeV1DeleteSpecificPolicyTypeV2

  • Policy API docker image available
  • DCAE TCA ms policy type v1 v2 is created

/policy/api/v1/policytypes/onap.policies.monitoring.cdap.tca.hi.lo.app/versions/12.0.0

200 - delete successful

404 - not found on 2nd delete attempt

CSIT17, 18

DeleteSpecificPolicyTypeV2

  • Policy API docker image available
  • DCAE TCA ms policy type v2 is created

/policy/api/v1/policytypes/onap.policies.monitoring.cdap.tca.hi.lo.app/versions/2.0.0

200 - delete successful

404 - not found on 2nd delete attempt

404 - not found on 2nd delete attempt

CSIT


Pairwise Testing

POLICY Pair Wise Testing for Frankfurt Release

Stability and Performance Testing

Jira
serverONAP JIRA
serverId425b2b0a-557c-3c0c-b515-579789cceedb
keyPOLICY-2452

Results will be documented in the readthedocs: https://docs.onap.org/en/latest/submodules/policy/parent.git/docs/development/devtools/devtools.html#running-the-stability-performance-tests