The intent of the 72 hour stability test is not to exhaustively test all functions but to run a steady load against the system and look for issues like memory leaks that aren't found in the short duration install and functional testing during the development cycle.

This page will collect notes on the 72 hour stability test run for Frankfurt.

See El Alto Stability Run Notes for comparison to previous runs.

Summary of Results

WORK IN PROGRESS

Setup

The integration-longevity tenant in Intel/Windriver environment was used for the 72 hour tests.

The onap-ci job for  "Project windriver-longevity-release-manual" was used for the deployment with the OOM set to frankfurt and Integration branches set to master. Integraiton master was used so we could catch the latest updates to integration scripts and vnf heat templates.

The jenkins job needs a couple of updates for each release:

1. Set the integration branch to 'origin/master'
2. Modify the parameters to deploy.sh to specify "-i master" and "-o frankfurt" to get integration master an oom frankfurt clones onto the nfs server.

The path for robot logs on dockerdata-nfs  changed in Frankfurt so the /dev-robot/   becomes /dev/robot

The stability tests used robot container image  1.6.1-STAGING-20200519T201214Z

robot container updates:

API_TYPE was set to GRA_API since we have deprecated VNF_API.

Shakedown consists of creating some temporary tags for stability72hrvLB, stability72hrvVG,stability72hrVFWCL to make sure each sub test ran successfully (including cleanup) in the environment before the jenkins job started with the higher level testsuite tag stability72hr that covers all three test types.

Clean out the old buid jobs using a jenkins console script (manage jenkins)

def jobName = "windriver-longevity-stability72hr"

def job = Jenkins.instance.getItem(jobName)

job.getBuilds().each { it.delete() }

job.nextBuildNumber = 1

job.save()

appc.properties updated to apply the fix for DMaaP message processing to call http://localhost:8181 for the streams update.

VNF Orchestration Tests

This test uses the onap-ci job "Project windriver-longevity-stability72hr" to automatically onboard, distribute and instantiate the ONAP opensource test VNFs vLB, vVG and vFWCL.

The scripts run validation tests after the install.

The scripts then delete the VNFs and cleans up the environment for the next run.

The script tests AAF, DMaaP, SDC, VID, AAI, SO, SDNC, APPC with the open source VNFs.

There was a problem with the robot scripts for vLB where it was not finding the base_lb.yaml file in the artifacts due to a change in the structure. A two line change to the vnf orchestration script to look for the 'heat3' key was made to resolve the issue. A Jira was created to track the changes to the robot scrips.   INT-1598 - Getting issue details... STATUS

These tests started at jenkins job #1

Each test run generates over 500 MB of data on the test through robot framework.

Each test run also runs the kubectl top nodes command to see cpu and memory utilization across the k8 cluster.

We periodically will run the top pods command as well to check on the top memory and cpu using pods.

Closed Loop Tests

This test uses the onap-ci job "Project windriver-longevity-vfwclosedloop".

The test uses the robot test script "demo-k8s.sh vfwclosedloop ". The script sets the number of streams on the vPacket Generator to 10 , waits for the change from 10 set sreams to 5 streams by the control loop then sets the stream to 1 and again waits for the 5 streams.

Success tests the loop from VNF through  DCAE, DMaaP, Policy, AAI , AAF and APPC.

In the jenkins job:

Modify the NFS_IP and PKG_IP   in the jenkins job to point to the current nfs server and packet generator  in the tenant

NFS_IP=10.12.5.205

PKG_IP=10.12.5.247

The tests start with #1