History
Before Frankfurt
Until Frankfurt there were 2 tests
- stability test: vFw (then vFWCL) run continuously
- resilency test: test when we destroy some pods and retest that the use case vFw is still OK (only up to El Alto)
In frankfurt we also consider the stability of the installation through teh Dialy chains
Guilin
The stability tests considered for the release were:
- 1 week stability test based on basic_vm
- 1 day HC verification
- Daily CI Guilin installation chain
See https://docs.onap.org/projects/onap-integration/en/guilin/integration-s3p.html#integration-s3p
Evolution for Honolulu
In Honolulu we would like to revisit the stability/resiliency testing part by introducing automated tests on CI weekly chain.
It means we want to execute tests over a week to verify the resiliency and the stability of the solution during the development life cycle.
Definition of the KPIs
what do we want to test, which figures? Nb of onboardings / instantiations? test duration//
we estimate our needs to < to be discussed/commented/challenged/questioned/...>:
- 10 parallel service onboarding
- 50 parallel intsantiation
- ....
Parallel onboarding tests
Description
The goal of this test is to create in parallel several services in the SDC.
We estimate that this number is not very high in the reality of operations because it corresponds to the upload of a new service model, which does not occur frequently.
Environment
Tests executed on a Guilin lab. Reusing the basic_vm with different service names (it means that we recreate all the SDC objects VSP, VF Services).
2 series run several times:
- 5 simutalneous onboarding
- 10 simultaneous onboarding
The main component used for this test is the SDC (+AAI).
During the test we monitor the ONAP cluster resources through a prometheus/grafana
<graph grafana memory & CPU générale>
<graph grafana memory & CPU SDC>
Results
Data format is MM:SS
5 parallel onboarding (10 series)
| criteria \ Serie | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Average |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Success rate (%) | 100 | 100 | 100 | 100 | 100 | 80 | 100 | 100 | 100 | 100 | 98 |
| Min duration | 04:46 | 27:39 | 10:23 | 10:14 | 10:26 | 11:18,00 | 07:41 | 07:53 | 08:05 | 08:34 | 4:46 |
| Max duration | 04:53 | 27:43 | 10:36 | 10:17 | 10:26 | 11:20,00 | 07:53 | 07:59 | 08:18 | 08:42 | 27:43 |
| Average duration | 04:50 | 27:40 | 10:28 | 10:15 | 10:26 | 11:18,75 | 07:48 | 07:57 | 08:11 | 08:38 | |
| Median duration | 04:50 | 27:41 | 10:26 | 10:16 | 10:26 | 11:18,50 | 07:48 | 07:59 | 08:12 | 08:39 | |
| Comments/Errrors | / | / | / | / | / | ERROR : maximum recursion depth exceeded in a python object | / | / | / | / | / |
<graph min/max/mean/average = f(serie)
10 parallel onboarding (5 series)
| criteria \ Serie | 1 | 2 | 3 | 4 | 5 | Average |
|---|---|---|---|---|---|---|
| Success rate (%) | 100 | 100 | 100 | 100 | 90 | 98 |
| Min duration | 16:03 | 16:03 | 15:23 | 16:32 | 19:39,00 | 15:23 |
| Max duration | 16:22 | 16:22 | 17:10 | 17:36 | 20:00,00 | 20:00 |
| Average duration | 16:15 | 16:15 | 16:50 | 17:22 | 19:50,00 | |
| Median duration | 16:19 | 16:19 | 17:08 | 17:32 | 19:52,00 | |
| Comments/Errrors | / | / | / | / | ERROR : Resource Category with "Generic" name does not exist |
<graph min/max/mean/average = f(serie)
<pour toutes les séries durée =f(time)>
Conclusions
ONAP Guilin is able to support 10 parallel onboarding, which is what we do expect.
The creation of resources is linear. It means that on serie 10, 9 services have been already created. We could have expected a linear increase of the onboarding duration because the client used for test list several times the services.So the more services in SDC, the bigger the list is. So globally the SDC resources increases continuously because we cannot delete them but it has no direct impact on the onboarding duration. The duration evolution is not linear and the duration may depend on the cluster status.
The more // processing we have, the slower the onboarding this.