...
See El Alto Stability Run Notes for comparison to previous runs.
Summary of Results
The 72 hour stability run result was PASS.
The onboard and instantiate tests ran for over 115 hours before environment issues stopped the test. There were errors due to both tooling and environment errors as indicate in the log.
The overall memory utilization only grew about 2% on the work nodes despite the environment issues. Interestingly the kubernetes ochestration node memory grew more which could mean we are over driving the API's in some fashion.
We did not limit other tenant activities in Windriver during this test run and we saw the impact from things like the re-install of SB00 in the tenant and general network latency impacts that caused openstack to be slower to instantiate.
For future stability runs we should go back to the process of shutting down non-critical tenants in the test environment to free up host resources for the test run (or other ways to prevent other testing from affecting the stabiity run).
The control loop tests were 100% successful and the cycle time for the loop was fairly consistent despite the environment issues. Future control loop stability tests should consider doing more policy edit type activites and running more control loop if host resources are available. The 10 second VES telemetry event is quite aggressive so we are sending more load into the VES collector and TCA engine during onset events than would be typical so adding additional loops should factor that in.
The jenkins jobs ran fairly well although the instantiate Demo vFWCL took longer than usual and should be factored into future test planning.WORK IN PROGRESS
Setup
The integration-longevity tenant in Intel/Windriver environment was used for the 72 hour tests.
...
We will run final numbers at the end of the test but most of the problems appear to be environment and tooling issues.
Closed Loop Tests
...
Interim Status on closed loop testing ~30% through stability run
