The TSC has decided to launch a task to define how to improve Pair-Wise Testing for Dublin as defined in https://jira.onap.org/browse/TSC-42
Current situation
For Beijing Release after Santa-Clara event, it was decided to perform pair-wise tests to achieve S3P under Heat and OOM installations.
For Casablanca,the projects must perform pair-wise testing for RC0 milestone. Pair-wise testing have to be realized only OOM installation
Various projects defines their own tests and run them when possible with no formal recommendation.
PTL must declare the various tests in the Integration Board: Integration Weather Board for Casablanca Release
The problems
- Dependencies problem. To realize pair-wise testing, the two paired components must be ready. For Casablanca, some components were available late impacting the time to perform pair-wise tests.
- No formal explanation to explain what to test
- No formal way to describe the tests
- No test storage
- No consistent tests: Some projects perform tests in both ways, but there is not a global consistency (ie NBI tested SO - NBI, while SO did not test SO - NBI)
What to be test ?
Each ONAP component includes a large number of Docker containers. Each project must be responsible to perform unitary test to validate the interaction between these Docker containers and to validate the API offered by the components (eg SO is responsible to test the various offered API).
The main objective is to test that both components are interacting correctly (eg using the correct ports, protocol, API version...).
When to perform pair-wise tests ?
Performing tests manually just before RC1 is not optimal => automate tests as much as possible and replay tests to guarantee that there is no unexpected behavior when a component updates the code.
Identify a list of 'core' projects that must be ready before others to avoid last minute tests will put high pressure on core components => define pair-wise tests to be run on a daily basis after RC1
Draft Proposed recommendations:
- Define formally the component dependencies for each project (cf NBI proposal) by project
- Run CSIT on OOM installation by project
- Develop a tool to verify that pair-wise tests are defined based on component dependencies by integration
- Define pair-wise tests for RC0 by project
- Run pair-wise tests under Integration project on daily basis after M4 (eg deploy a full fresh ONAP and run the pair-wise tests) by integration
- Store tests in a DB by integration