Overview of the verification flows

The following diagram illustrates the new review verification and merge verification flows

  1. Developer pushes a commit to Gerrit
  2. Gerrit triggers review verification job in Jenkins
    • The templates for review verification job are in ci-management/jjb/integration/integration-templates-csit.yaml, named as "{project-name}-review-verification-<artifact type>-{stream}"
      • The only currently supported artifact type in review verification templates is maven 
    • The trigger is "gerrit-trigger-patch-submitted" and it is activated by any file change in the project
    • the job starts with artifact build
      • Artifacts should be created from the patch under review
      • Currently the first step just compiles the artifacts locally (with unit tests), but additional operations - for example sonar analysis - should be considered here
  3. Docker build is started
  4. Docker build step builds docker image(s) to local temporary docker repository
    • The docker build scripts and the created artifacts are completely based on the submitted patch in Gerrit
    • The actual build script is specific to the project and has to be specified in the specific docker build job definition
    • The build should differentiate local image from any that could be found from Nexus
      • For example, no repository prefix in the image name would be an obvious hint 
  5. CSIT job is started
    • this step calls csit/run-project-csit.sh under the project's root directory via integration-run-project-test builder defined in ci-management/integration/integration-macros.yaml
    • run-project-csit.sh is project-specific, but can take advantage of common scripts in integration/csit if considered useful
    • run-project-csit.sh can and should run all the test plans of the project in one go and provide the results under csit/archives directory
  6. CSIT job pulls the docker images created in step 4. (by using project-specific setup scripts) and tests them
    • The test cases to be executed are also from the patch under review
    • the setup script responsible of pulling the required images should ensure that the images are local
      • To be considered: should CSIT ever be able to pull the images under test from Nexus?
  7. Review verification job gives +1 or -1 verify vote to the review based on the overall success of the verification
  8. Once the commit is approved (+2'd) and submitted, Gerrit triggers merge verification job in Jenkins
    • The template for merge verification job is in ci-management/jjb/integration/integration-templates-csit.yaml and currently named as "{project-name}-merge-verification-<artifact type>-{stream}
    • The only currently supported artifact type in merge verification templates is maven 
    • The trigger is "gerrit-trigger-patch-merged" and it is activated by any file change in the project
    • the job starts with artifact build
      • Artifacts should be created from master
      • Sonar analysis and other possible additional artifact-related steps should be considered here
  9. Docker build is started
  10. Docker build step builds docker image(s) to local temporary docker repository
    • The docker build scripts and the created artifacts are completely based on master
    • The actual build script is the same as in review verification and used in the same way
  11. CSIT is started in exactly the same way as review verification
  12. CSIT job pulls the docker images created in step 10. (by using project-specific setup scripts) and tests them

Requirements and recommendations

The following requirements must be met and recommendations should be followed by any project that moves their CSIT tests to project repository:

  • The project must define a review verification job based on one of "{project-name}-review-verification-<artifact type>-{stream}" templates (which is defined in ci-management/jjb/integration/integration-templates-csit.yaml)
    • the only currently supported artifact type is maven - new templates must be created to support different artifact types (such as golang)
      • The existing maven template assumes the following JJB variables for maven by default:
        •  mvn-goals: clean deploy
        • mvn-params: <empty> 
      • The purpose of these variables is to build both maven artifacts and then docker images containing the artifacts, but to not push either to Nexus
        • if your project uses different maven goals or parameters for achieving these, the above variables must be overridden appropriately in the project-specific review verification job definition
  • in addition, the following JJB variables must be defined in the project-specific review verification job definition:
    • project-name (project identifier that can be freely chosen and is used in generated job names)
    • project (should correspond project's name in Gerrit)
    • stream (list of supported branches)
    • mvn-settings
      • should be named as '<project-name>-settings'
      • note that you may have to ask to add support for your '<project-name>-settings' in Jenkins Sandbox separately from LF you intend to test the job there
  • The project must support CSIT as follows:
    • The project must have csit/run-project-csit.sh that is responsible of the entire CSIT execution
      • run-project-csit.sh must return exit code 0 on success and non-zero code on any failure - this is how the success/failure of the test suite(s) and +1/-1 verification vote is communicated to gerrit
      • currently run-project-csit.sh must provide robot test results or the entire review verification fails
        • A more generic basic template that allows other test frameworks should be added
    • The setup scripts must be able to use locally built docker images
      • preferably the images should be tagged and referred to unambiguously so there is no risk of them being pulled from Nexus3
  • Following the integration/csit subdirectory structure and using its run-csit.sh (by cloning the integration/csit and copying the common scripts to workspace) is possible but not mandatory
  • the csit subdirectory should also contain README.md that explains how to build the relevant docker images locally and run the CSIT with them


  • The project must define a merge verification job based on one of "{project-name}-merge-verification-<artifact type>-{stream}" templates (which are defined in ci-management/jjb/integration/integration-templates-csit.yaml)
    • Same requirements apply for merge verification job definition as for review verification job

Example projects

There are a handful of projects and repositories that are already applying this design:

  • ccsdk/integration
  • ccsdk/oran
  • cps
  • dmaap/buscontroller
  • optf/has
  • optf/odf

CPS

This is the first project to create their functional tests directly under their own repository instead of moving them from integration/csit

Open questions and remaining tasks

  • Use of Robot framework is currently mandatory - this should be improved
  • Should CSIT be able to test docker images from Nexus? If so, under which circumstances and for what purpose?
  • Currently there are verification templates available only for maven artifact type - others (for example golang) may have to be added
  • This proposal replaces plain Maven/Makefile/other artifact build jobs completely
    • Sonar builds seem to be completely separate scheduled builds anyway, and unsupported by any global templates?
    • Ideally all reviews should start with Sonar build and fail the review on violations
  • No labels

4 Comments

  1. Jim Hahn

    For verify jobs, I'd rather not slow down a job by running a CSIT - it just isn't typically worth the value, at least not for the Policy project - the faster we can turn builds around the better. On the other hand, it does make sense to trigger a CSIT as the last step of a merge job, but I wouldn't reject the merge just because the CSIT fails.  If that were done, then the CSITs could continue to use the SNAPSHOT docker images.

    1. Lasse Kaihlavirta

      Please note that the merge verification job is triggered by merge, i.e. any failure in that phase is unable to reject the merge itself as it has already happened by then (the only reason for running the CSIT in this phase is to decide whether the image built from master after merge is good enough for being tagged as STAGING). Only the CSIT executed in review verification phase is able to block the code merge.

      That having been said, I disagree with pretty much every point above (smile) 

      • Slowing down review verification by CSIT (and usually even more by the prerequisite docker image build, which tends to take longer than the actual tests) is definitely worth the value, as that value has to be pitted against the cost of the consequences of having faulty code in master (assuming of course that the CSIT is good enough to be able to detect that fault)
      • This added value should apply also to Policy (even though Policy repo structure vs. test structure has its known complications that may make getting the full benefits harder)
      • Basically, CSIT should never fail in review verification in the first place, because committers should make sure locally that the CSIT passes before pushing anything to review
      • A failed CSIT (in review verification) should absolutely reject the merge (committers can always override this verification downvote, but that power should not be used lightly)
      • If CSIT fails in merge verification, it should be treated as potential blocker bug, as it suggests that master is broken

  2. Chengkai Yan

    Step 5 mentioned "csit/run-project-csit.sh", I couldnt find this script, only "csit/run-csit.sh" is available.

    1. Lasse Kaihlavirta

      "csit/run-project-csit.sh under the project's root directory" - this is a script that the project itself must provide in its own repository