Description

• See Project-specific CSIT structure

Motivation

The most obvious solution for taking full advantage of CSIT is to add the test cases under the same repository with the functionalities that they are testing (instead of having them in separate centralized CSIT repository as we currently have). This would have the following benefits:

• CSIT could be triggered by any commit to the project repo
• CSIT tests the code (or specifically, docker images that have been built) from the committed branch
• CSIT could have a vote on the commit based on the result of the test run
• If the implementation changes require changes in CSIT tests (to cover new functionality or to pass in the first place), that could be handled within the same commit
• ideally, local verification would become less complex (no need to work between CSIT repo and project repo) 
• No need of Integration for the integration team to merge any changes related to your project

...

• CSIT suites that test components from multiple project repositories at the same time 
  • such CSIT tests may have to be separated using additional simulators, or
  • project repository structures themselves may have to be reconsidered, or
  • the possibility of combining branches from multiple repositories under the same commit needs to be provided (if gerrit allows?)
  • In any case, the division between the images under test and images that are just necessary dependencies should be clearly made and documented
    • the images under test should be coming from the commit branch
    • in the case of necessary dependencies it must be decided whether they should be provided as simulators or real components
      • if provided as real components, they should be referred to with a fixed, unchanging released version number and should be stable and mature enough to develop on
        • ONAP's (unintentional?) practice of allowing the same versioned image to be changed might prove is problematic
    • project-specific simulators that are built on the fly are trivial case from dependency handling point of view, but if common simulators are in use, dependency considerations for them are the same as with real components
• Jenkins templates may have to be redesigned to support unified approach for triggering review branch-specific artifact and docker buildbuilds, CSIT execution and voting chain as part of review verification
  • What is the most elegant and effortless solution for ensuring that the docker images built from commit branch are taken in use in the CSIT (both in Jenkins and in local development environment)?
  • The redesign should also allow testing locally built docker images in local environment with CSIT as effortlessly as possible
• CSITs will CSITs would become blockers for merging code
  • local pre-commit verification should be supported better by common CSIT tools

  • are all projects and suites mature enough to deal with that?

Technical details to be decided

• Should we keep separate docker build and CSIT jobs and just chain them into review verification, or should we try to incorporate docker building and CSIT execution into existing review jobs?
  • Reusing existing jobs and chaining them would require some docker image tag tuning to make sure CSIT tests pick up the exact image that was produced by preceding docker build job 
  • Either way, JJB templates will probably have to be touched
• Should we still have common CSIT scripts (run-csit.sh etc) in CSIT repo and related procedures (setup, tests, teardown and related result collection) as the basis of project-specific test execution? 
• Execution of CSIT tests and incorporating locally built test images should be made as easy as possible following common guidelines
  • Setting up testing environment (specific project-specific dependencies should be handled by the setup scripts)
  • Specific environment variables expected by the test suite (like GERRIT_BRANCH)

Project status and readiness at the end of Guilin

...

aaf-frankfurt-csit-certservice

Frankfurt branch only? The tests triggered from here do not even exist in master (nor are there any jobs for still remaining aafapi and sms-test-plan?)

...

aaf-certservice-frankfurt-merge-java - this job neither builds a docker image nor triggers another job that does it, so why trigger CSIT?

...

nexus3.onap.org:10001/onap/ org.onap.aaf.certservice.aaf-certservice-api:frankfurt-latest

(hardcoded)

The source repository is aaf/certservice

...

aaf-certservice-maven-docker-stage-frankfurt

(note that aaf-certservice-release-verify and aaf-certservice-release-merge also build aaf-certservice-api image, but not with frankfurt-latest tag)

primekey/ejbca-ce:6.15.2.5 (hardcoded, real image used)

...

...

-

nexus3.onap.org:10001/onap/appc-image:1.7.2-SNAPSHOT-latest

nexus3.onap.org:10001/onap/appc-cdt-image:1.7.2-SNAPSHOT-latest

(hardcoded)

docker-compose.yml as well as appc-image and appc-cdt-image come from appc/deployment

No job is producing these images at the moment! 

appc-deployment-master-docker-java-daily produces 1.8.0-SNAPSHOT-latest and 

appc-deployment-frankfurt-docker-java-daily is producing 1.7.3-SNAPSHOT-latest

1.7.2-SNAPSHOT-latest images have not been created since May 27

...

mariadb:10.4.3

nexus3.onap.org:10001/onap/ccsdk-dgbuilder-image:0.6.0 (no longer produced, last update to Nexus on Aug 30 2019)

nexus3.onap.org:10001/onap/ccsdk-ansible-server-image:0.4.4 (no longer produced, last update to Nexus on Jun 06 2019)

(hardcoded, real images used)

In maintenance mode, CSIT disabled: 

...

...

...

ccsdk-oran-master-csit-polmansuite

ccsdk-oran-guilin-csit-polmansuite

...

...

• nexus3.onap.org:10003/onap/ccsdk-oran-a1policymanagementservice (1.1.0-SNAPSHOT for master, 1.0.1-SNAPSHOT for Guilin)

The source repository is ccsdk/oran

...

Same as triggers

...

nexus3.onap.org:10003/onap/sdnc-image (2.1.0-STAGING-latest for master, nexus3.onap.org:10003/onap/sdnc-image:2.0.1-STAGING-latest for Guilin)

nexus3.o-ran-sc.org:10004/o-ran-sc/nonrtric-controlpanel:2.0.0
mysql/mysql-server:5.6
nexus3.o-ran-sc.org:10004/o-ran-sc/a1-simulator:2.0.0
consul:1.7.2
nexus3.onap.org:10001/onap/org.onap.dcaegen2.platform.configbinding.app-app:2.3.0
mrstub:latest
callback-receiver:latest

...

These tests are currently cloned from ORAN repository and they are essentially plain shell scripts that are just wrapped into Robot

In Honolulu the test cases should be brought under ONAP repository and rewritten with Robot and ONAP CSIT scripts as much as possible

...

clamp-master-csit-APIs

clamp-master-csit-UIs

...

...

nexus3.onap.org:10001/onap/clamp-backend:4.1-STAGING-latest

(hardcoded)

The docker-compose.yml and source of the images is clamp

...

...

mariadb:10.5.4

python:2-slim

...

dcaegen2-services-son-handler-master-csit-testsuites

dcaegen2-services-pmsh-master-csit-testsuite

dcaegen2-services-bbs-event-processor-master-csit-bbs-testsuites

dcaegen2-pmmapper-master-csit-pmmapper

dcaegen2-master-csit-testsuites

dcaegen2-master-csit-prh-testsuites

dcaegen2-collectors-restconf-master-csit-testsuites

dcaegen2-collectors-hv-ves-master-csit-testsuites

dcaegen2-collectors-datafile-master-csit-ManagementInterface-suite

dcaegen2-collectors-datafile-master-csit-Functional-suite

dcaegen2-services-son-handler-master-merge-java

dcaegen2-services-pmsh-docker-merge-master

dcaegen2-services-master-bbs-event-processor-merge-java

(pmmapper has no trigger job that actually exists)

dcaegen2-collectors-ves-master-merge-java

dcaegen2-services-prh-master-merge-java

dcaegen2-collectors-restconf-master-merge-java

dcaegen2-collectors-hv-ves-master-merge-java

(datafile CSITs have no trigger jobs that actually exist)

...

nexus3.onap.org:10003/onap/org.onap.dcaegen2.services.son-handler:latest

nexus3.onap.org:10001/onap/org.onap.dcaegen2.services.pmsh:latest

nexus3.onap.org:10001/onap/org.onap.dcaegen2.services.components.bbs-event-processor:latest

nexus3.onap.org:10001/onap/org.onap.dcaegen2.services.pm-mapper:latest

nexus3.onap.org:10003/onap/org.onap.dcaegen2.collectors.ves.vescollector:latest

nexus3.onap.org:10001/onap/org.onap.dcaegen2.services.prh.prh-app-server:latest

nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.restconfcollector:latest

nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.hv-ves.hv-collector-main:latest

nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server:latest

(all hardcoded in various places in the tests)

TBD: most or all of these are in their own dedicated repositories, TBD: are the repo-image-CSIT relations straightforward 1:1 in all cases (or any case)?

...

dmaap-buscontroller-master-csit-ssl

dmaap-buscontroller-master-csit-with_dr

dmaap-buscontroller-master-csit-with_mr

dmaap-datarouter-master-csit-ssl-dr-suite

dmaap-master-csit-mrpubsub

...

nexus3.onap.org:10001/onap/dmaap/buscontroller

nexus3.onap.org:10001/onap/dmaap/datarouter-prov

nexus3.onap.org:10001/onap/dmaap/datarouter-node

nexus3.onap.org:10001/onap/dmaap/datarouter-subscriber

(all hardcoded, version not given so latest by default)

At least three different source repositories:

dmaap/buscontroller

dmaap/datarouter

dmaap/messagerouter/messageservice

There are no jobs producing onap/dmaap/buscontroller - the latest image in Nexus has been updated Thu Feb 21 2019 and belongs to Casablanca (?!) - ticket raised:

dmaap-datarouter-maven-docker-stage-master (same as job trigger)

...

multicloud-master-csit-functionality1

multicloud-starlingx-master-csit-functionality1

multicloud-vmware-master-csit-functionality1

...

nexus3.onap.org:10001/onap/multicloud/vio:1.4.1 

nexus3.onap.org:10001/onap/multicloud/framework:1.4.1

nexus3.onap.org:10001/onap/multicloud/openstack-starlingx

nexus3.onap.org:10001/onap/multicloud/vio

(all hardcoded, default latest for those that do not define version explicitly)

...

...

music-master-verify-csit-music-distributed-kv-store-test-plan

...

nexus3.onap.org:10001/onap/music/distributed-kv-store (hardcoded, latest by default)

...

...

oom-platform-cert-service-master-csit-certservice

oom-platform-cert-service-master-csit-postprocessor

...

...

nexus3.onap.org:10001/onap/org.onap.oom.platform.cert-service.oom-certservice-api:latest

nexus3.onap.org:10001/onap/org.onap.oom.platform.cert-service.oom-certservice-post-processor:latest (but started in Robot test and defined in Robot variable so it would be theoretically overridable)

oom-platform-cert-service-maven-docker-stage-master (for both)

primekey/ejbca-ce:6.15.2.5

...

optf-cmso-master-csit-cmso

optf-osdf-master-csit-osdf

optf-has-master-csit-has

...

nexus3.onap.org:10001/onap/optf-cmso-service:latest

nexus3.onap.org:10001/onap/optf-cmso-dbinit:latest

nexus3.onap.org:10001/onap/optf-cmso-topology:latest

nexus3.onap.org:10001/onap/optf-cmso-ticketmgt:latest

nexus3.onap.org:10001/onap/optf-cmso-optimizer:latest

nexus3.onap.org:10001/onap/optf-cmso-robot:latest (!! The actual Robot tests run here in setup phase instead of as separate Robot tests from CSIT repo)

nexus3.onap.org:10001/onap/optf-has:2.0.2-SNAPSHOT-latest

nexus3.onap.org:10001/onap/optf-osdf:2.0.3-SNAPSHOT-latest

Repositories:

optf/cmso

optf/osdf

optf/has

...

optf-cmso-maven-docker-stage-master

optf-has-maven-docker-stage-master

optf-osdf-maven-docker-stage-master

(word of warning: the frankfurt versions of the above jobs are also  pushing the same images with "latest"! Probably worth a ticket)

...

policy-master-csit-apex-pdp

policy-master-csit-api

policy-master-csit-distribution

policy-master-csit-drools-applications

policy-master-csit-drools-pdp

policy-master-csit-pap

policy-master-csit-xacml-pdp

(+frankfurt branches for each)

...

policy-apex-pdp-master-merge-java

policy-api-master-merge-java

policy-distribution-master-merge-java

policy-drools-applications-master-merge-java

policy-drools-pdp-master-merge-java

policy-pap-master-merge-java

policy-xacml-pdp-master-merge-java

(+frankfurt branches for each - note that these also produce the same images with latest tag; this is not critical though since CSITs do not refer to those)

...

Dynamic image version resolution based on GERRIT_BRANCH and pom.xmls in the referred branches of the related repositories amended with "SNAPSHOT-latest"

• The distinction between images under test and dependent images is blurred -any policy component change can cause any other's CSIT to fail

...

...

sdnc-master-csit-healthcheck

sdnc-master-csit-sdnc_netconf_tls_post_deploy

...

...

nexus3.onap.org:10001/onap/sdnc-image/1.8-STAGING-latest

nexus3.onap.org:10001/onap/sdnc-ansible-server-image/1.8-STAGING-latest

nexus3.onap.org:10001/onap/admportal-sdnc-image/1.8-STAGING-latest

nexus3.onap.org:10001/onap/sdnc-ueb-listener-image/1.8-STAGING-latest

nexus3.onap.org:10001/onap/sdnc-dmaap-listener-image/1.8-STAGING-latest

nexus3.onap.org:10001/onap/ccsdk-dgbuilder-image/0.7-STAGING-latest

---

nexus3.onap.org:10001/onap/sdnc-image/1.8.3-STAGING-latest

The source repo is sdnc/oam

SDNC images are going at version 2.1 in master )sdnc-oam-maven-docker-stage-master) and at 2.0 in Guilin (https://jenkins.onap.org/view/sdnc/job/sdnc-oam-maven-docker-stage-guilin)

Latest 1.8-STAGING-latest images are from Aug 31 2020 and 1.8.3-STAGING-latest from Jun 07 2020. These images are from Frankfurt and have no longer any jobs to generate them.

Ticket created: 

primekey/ejbca-ce:6.15.2.5

nexus3.onap.org:10001/onap/org.onap.aaf.certservice.aaf-certservice-api:latest

---

nexus3.onap.org:10001/mariadb:10.1.11

...

so-master-csit-integration-etsi-testing

so-master-csit-integration-testing

(and the same for frankfurt and elalto; see

...

nexus3.onap.org:10001/onap/so/catalog-db-adapter:1.3.1

nexus3.onap.org:10001/onap/so/request-db-adapter:1.3.1

nexus3.onap.org:10001/onap/so/sdnc-adapter:1.3.1

nexus3.onap.org:10001/onap/so/openstack-adapter:1.3.1

nexus3.onap.org:10001/onap/so/vfc-adapter::1.3.1

nexus3.onap.org:10001/onap/so/sdc-controller:1.3.1

nexus3.onap.org:10001/onap/so/bpmn-infra:1.3.1

nexus3.onap.org:10001/onap/so/api-handler-infra:1.3.1

---

1.6.0 tag for integration-etsi-testing (TBD: images to be added)

Source repo seems to be so for all of these (to be verified)

1.3.1 goes all the way back to early Casablanca - these have not been updated since Nov 16 2018! 

1.6.0 are early Frankfurt versions (no updates since April 2020)

so-maven-docker-stage-master is at 1.7.3

so-maven-docker-stage-frankfurt is at 1.6.4

so-maven-docker-stage-elalto is at 1.5.4

• Docker image production practices should be unified (see Docker Image Build Guidelines, Independent Versioning and Release Process and Release Versioning Strategy)

Technical decisions

• New templates have been introduced for all verification steps (see Project-specific CSIT structure ) while all the existing ones have been left untouched
  • review-verification template
    • triggered by review commit
    • has verify vote on the review
    • triggers artifact builds
      • Should maybe include also Sonar analysis in the future - those are currently still completely separate jobs executed only on merged code
    • triggers docker image build - different types of docker builds each require their own templates (identified and separated by "artifact-type"), and so far we have them for
      • maven docker build
      • golang docker build
      • images are not pushed to Nexus3
    • triggers CSIT execution that tests the produced docker images
  • merge-verification template
    • triggered by merge
    • triggers artifact build in the same way as review-verification
    • triggers docker image build in the same way as review-verification but builds the images from master
    • triggers CSIT execution that tests the produced docker images in the same way as review-verification
    • No images are pushed to Nexus3 as a result of merge-verification either; publication of new images still remains the responsibility of separate docker staging etc. jobs 
• Execution of CSIT tests and incorporating locally built test images should be made as easy as possible following common guidelines
  • Local docker build instructions should be maintained and easily accessible (for example, as part of CSIT README.md in the project repository)
  • Test environment setup should be as automated as possible (project-specific dependencies should be handled by the setup scripts)
  • Need for specific environment variables (like , for example, GERRIT_BRANCH) should be minimized and preferably avoided altogether  
• Artifact builds should take care of code coverage/Sonar analysis
  • apparently currently there are no common templates dealing with Sonar? (instead, all the project have their custom Sonar JJB definition)
  • all the Sonars run on daily schedule instead of being triggered
  • are projects mature enough to define the various Sonar violations as review blockers?
    • can we give each project the power to define their own quality gates? 

Project status and readiness at the end of Honolulu

• See  CSIT status and readiness per project at the end of Honolulu

Project status and readiness at the end of Guilin

• Moved under CSIT status and readiness per project at the end of Guilin for reference

...

vfc-gvnfm-vnflcm-master-csit-sanity-check

vfc-gvnfm-vnfres-master-csit-sanity-check

vfc-nfvo-driver-vnfm-gvnfm-master-csit-sanity-check

vfc-nfvo-lcm-master-csit-sanity-check

...

vfc-gvnfm-vnflcm-master-docker-java-version-shell-daily

vfc-gvnfm-vnfres-master-docker-java-version-shell-daily

vfc-nfvo-driver-vnfm-gvnfm-master-docker-java-version-shell-daily

vfc-nfvo-lcm-master-docker-java-version-shell-daily

nexus3.onap.org:10001/onap/vfc/vnflcm

nexus3.onap.org:10001/onap/vfc/vnfres

nexus3.onap.org:10001/onap/vfc/gvnfmdriver

nexus3.onap.org:10001/onap/vfc/nslcm

(all latest by default)

Source repositories:

vfc/gvnfm/vnflcm

vfc/gvnfm/vnfres

vfc/nfvo/driver/vnfm/gvnfm

vfc/nfvo/lcm

...

consul:0.9.3

nexus3.onap.org:10001/onap/msb/msb_discovery

nexus3.onap.org:10001/onap/msb/msb_a
pigateway

nexus3.onap.org:10001/library/mariadb

redis (!?)

It's not clear in all cases which VFC images are under test and which are just necessary dependencies

...

vid-master-csit-healthCheck

...

nexus3.onap.org:10001/onap/vid:6.0-STAGING-latest

Source repo is vid

...

nexus3.onap.org:10001/library/mariadb:10

so-simulator (specific to this suite)

aai-simulator (specific to this suite)

...

vnfsdk-marketplace-master-csit-testsuites

vnfsdk-refrepo-master-csit-testsuites

...

nexus3.onap.org:10001/onap/vnfsdk/refrepo/postgres:latest

nexus3.onap.org:10001/onap/vnfsdk/refrepo:latest

Source repo is vnfsdk/refrepo

vnfsdk-refrepo-maven-docker-stage-master

Where is onap/vnfsdk/refrepo/postgres coming from? It hasn't been produced since Jul 09 2019?

...