Self releases are now possible after the migration to latest global-jjbs staging jobs

gerrit-maven-stage and gerrit-maven-docker-stage


Before teams start using self releases ...

Please note that this process will be possible by committers with Merge/+2 Rights in the repos. 

Please make sure your INFO.yaml files are updated and that all committers have been added. 


More information of these procedure can be found here. (Please make sure to read this and ask any

questions to support.linuxfoundation.org:

https://github.com/lfit/releng-global-jjb/blob/master/docs/jjb/lf-release-jobs.rst


How does it work?


The process is very simple. It will require for tech teams to post a new "releases" folder with a yaml file for each release

they need. This file gets verified and, once merged, the release will be posted. 


Here are the steps:

  1. The tech team needs to make sure their committer list is up to date as these will be the people approved to make releases.
    1. The reason behind this, only committers can +2 and merge changes in their tech team repo.
    2. The releases merge job will be triggered by files merged in the repo and will execute a release.
  2. Tech team needs to add "{project-name}-gerrit-release-jobs" to their ci-management yaml files. 
    1. This group introduces "gerrit-release-verify" and "gerrit-release-merge"
  3. Once a release candidate is build using gerrit-maven-stage, a new file must be added to your project repo describing the release and referring to the gerrit-maven-stage log:
    1. This file needs to be located in the root repo under a "releases" folder.
    2. An example can be viewed here: https://github.com/lfit/releng-global-jjb/blob/master/docs/jjb/lf-release-jobs.rst 
      1. Name of the file should match the semantic version of the release being published. (For example: releases/1.0.0.yaml)
      2. distribution_type: 'maven'  (Future expansion will allow "container" to be provided"
      3. version: '#.#.#' (Release semantic version)
      4. tag_release: false (By default, tagging a repo with a release version is set to true. If you want to skip it, set it to false)
      5. project: 'project-name' (Project name, for example 'ccsdk-parent')
      6. log-dir: 'pointer_to_maven_stage_job/build_number/' (for example: 'ccsdk-parent-maven-stage-master/2/')
      7. maven_central_url: 'oss.sonatype.org' ( Optional, in case the team want's to publish to Maven Central)
  4. This file will trigger "{project-name}-release-verify-{stream}"
    1. This job can also be triggered using the comment "recheck|reverify"
    2. The verify job will make sure the release file contains the needed information and that the candidate exists.
  5. Tech team needs to +2 this new change and merge it. Please do not override any -1 Verify from Jenkins.
  6. The merge will trigger "{project-name}-release-merge-{stream}"
    1. This job can also be triggered using the comment "remerge"
    2. This job will push the release and tag the repo.

After your self release ...


Once your self release file was merged and processed by "gerrit-release-verify" and "gerrit-release-merge"...

  • DO NOT attempt to revert files in releases/  → Even if the release was not needed, we need to keep track on what happened in that repo also the tag needs to be kept in the repo
  • DO NOT attempt to re-tag the repo with the same version → this will fail as the gerrit-release-merge job already tagged it
  • DO NOT modify releases files → Once a releases file is merged, it is immediately processed. If the team needs to release a new build number, please bump your versions and generate another stage-release
  • DO NOT re-use the same stage-release build number for multiple releases files → Once an autorelease package is pushed, it is closed and it cannot be re-released. 
  • MAKE SURE your project is 100% free from using nexus-staging-maven-plugin. This was a must do in the global-jjb migration and teams still using it WILL BE facing issues deploying all their artifacts.
    • Using this plugin, pre-deploys image in a bad way before lftools has a chance to compile all artifacts needed for the release candidate
  • DO NOT cherry-pick release files across branches. Duplicated release files will make the verify and merge job fail as those release were already processed.
    • Releases from master should be created in master, releases from sub-branches like "el-alto" should be created in "el-alto"
  • No labels

22 Comments

  1. Dan Timoney

    These instructions are a great start, but I strongly encourage people moving to global-jjb to read the "release jobs" section of the global-jjb documentation that Jess posted above (https://github.com/lfit/releng-global-jjb/blob/master/docs/jjb/lf-release-jobs.rst).  There are excellent examples there of both the release YAML and of the yaml you need to add to your existing jjb to add the new release jobs.

    Some specific notes/tips:

    • As Jonathan mentioned above, by default, the 'staging' jobs will strip the string '-SNAPSHOT' from your pom.xml when it compiles.  If you truly need a snapshot build (e.g. because you need a fix version during development that has not yet been released), you will want to set "maven-versions-plugin" to true in your staging job.  This will cause the script not to strip -SNAPSHOT from the pom.xml but instead to use the maven versions plugin to change your artifacts' version from snapshot to release before compiling.
    • Before submitting your release yaml, look at the console log of the staging build you are about to use and search for the word 'autorelease'.  You should see that appear AFTER your maven build completes, and NOT within the maven build itself.  If you see an autorelease within the maven build, then you're using the nexus-maven-staging-plugin somewhere (e.g. some pom.xml(s) might still be using version 1.x.x of oparent).  Fix that problem and rebuild before you try to self-release.

    1. Jessica Wagantall

      Thanks Dan Timoney  this information you mentioned is 100% accurate. 

      Point #2 has been moved to my "After your self release ..." section. This is very important becuase if this teams are still using the deprecated plugin somewhere, they will face issues with their release artifacts not being complete like we have done in some cases. 

  2. Jonathan Gathman

    Moving my "Feed back" To Comment Section, like Dan.

    FEEDBACK:

    Jonathan (AAF): 

    • I found AAI's "Gizmo" example a bit more helpful than the example listed above, line item 3, because the above one isn't ONAP specific.  AAF's works now as well, if you want to look at that too.  Look in "ci-management/jjb/<project>"
    • The new process assumes that you build your Docker Containers with Maven.  If you have some other method, you will need to get it to work with Maven.  I was able to by using Exec Plugin
    • The normal "deploy" for Java Projects in Maven are Jars into Nexus.  Deploying into Docker, however, is a different artifact type altogether.  You can handle each one at the correct steps by using "Maven Profiles"... I created one just in my Docker Build Directory called "build-docker", and used the "exec" goal, which means I could do the following for Maven Params: " install -maven.test.skip exec:exec -Pdocker-build". (note, "Tests already performed twice before... why make this happen again?)
    • Do make sure your Docker Build in Maven includes "docker push".  It's not enough to simply build the Docker image.  LF's Template will cover the Docker Login (they had to add the credentials in my case, once they did, it worked)
    • Realize that LF's Template will convert all the Maven Jar Versions with "-SNAPSHOT" (i.e.2.1.15-SNAPSHOT) into it's release version.  Your Docker Build/Push needs to recognize this, even though the DOCKER VERSION generated has "-SNAPSHOT" in it.  What this means is that your Docker may have version 2.1.15-SNAPSHOT, while the jars copied in will be 2.1.15.  Your Docker Build command need to handle this.
    • don't miss the "release" file above.  Lack of it caused maven to not run at all in my case.
    • DO NOT BE DISCOURAGED!  If you keep cycling, reading the error outputs and pushing through, with LF's help, you'll make it!  (Thanks, Jess, Arin, Eric et al)

    RESOLUTION of my mistake:

      The thing to remember is that CREATING and CHECKING IN the "release" file above WILL cause the Self-Release mechanism of the FINAL Release of that version.  You do NOT want that for "Staging Release".... Use magic word "stage-release" for that.

      Try not to touch these files anymore, because if they are messed up, the "release-version" process will kick off.  With normal processes, you can't Delete it, you can't fix it, etc.  It has to be removed out of band, and while I solved, LF prefers that you enlist their help instead.  

    1. Jessica Wagantall

      Thanks Jonathan Gathman Sorry for all the scramble but, as we experienced, is always best to create new file.


      The process follows a lot what I was doing manually when releasing artifact, for example:

      • Releases cannot be released under the same version number.
      • Once a Jenkins build is used for a release, it cannot be re-used again since the autorelease package gets removed right after releasing. 


      However!, the teams can still ask for a TSC approval for LFRELENG to remove a release. This process will take a lot of manual intervention from my part though, which in an emergency can be done.  

      1. Jonathan Gathman

        We got through it and I truly appreciate all your help.  But your answers don't seem to reflect that my case was an unintentional change in file.   I had absolutely no intention of changing it... it was caught in a "Global search/replace" from 2.1.15 to 2.1.16.  

        IF that happens, then Apps will find that they are in a Race Condition:  The file WON'T work correctly, because version doesn't match other data and causes "Version validate" to fail.

        But neither can the End user fix or remove the offending file without the processing trying to do "Version Validate", and failing.  I saw evidence that the old file remained as well, though I can't prove it now.

        That was why I took the step that was severely reprimanded... it was the only way for an end-user to clear the file that wouldn't let me proceed because any touch of that file caused failure.  Catch 22.

  3. Jonathan Gathman

    Accidental or purposeful change of the "releases/X.yaml" is a problem.  Any change in this directory causes the attempt for a final release.  Removal of the file... attempts to validate the final release... Catch 22.  

    Best thing to do is invoke LF, and work on getting it out of that Directory without normal CI/CD processes.


    1. Jorge Hernandez

      I believe the releases/2.1.15.yaml should remain untouched moving forward.   I don't think a new releases/2.1.6.yaml should be added until the team is ready to do a release, in part because you are probably not ready to designate that "blessed" staging job that would cut the release (log_dir section).

      We have a 2 release files example here:  https://git.onap.org/policy/models/tree/releases, so multiple release files seems to be working ok at least for us.   AS background, we needed to publish 2 releases as the first release was partially published in error by the new tooling (which I believe the LF team has corrected already).

      1. Jonathan Gathman

        That makes sense.

        But did you have any issues with the "release-verify" job, that subsequently set "ONAP Jobbuilder" failures during simple checkin?

        Having that file remain there causes mine to fail...

        08:39:44 fatal: tag '2.1.15' already exists
08:39:44 Build step 'Execute shell' marked build as failure

        This MIGHT be because I made a mistake and touched the file with a global search/replace of "2.1.15" while doing POMs.

        1. Jorge Hernandez

          Umm, not sure, I didn't run into this particular problem.   In my case, for the second release file review https://gerrit.onap.org/r/#/c/policy/models/+/92734/, the release-verify job was sucessful (see in the comments section)>

        2. Jonathan Gathman

          Resolution:  get the file back were it was outside of normal Release... See LF for help.  

        3. Jessica Wagantall

          The tag error happened because there was already a previously released 2.1.15 version. Weather it was a good release or a bad release file, the merge jenkins job released whatever it could

          and it tagged the Gerrit repo with that tag. 

          On the next round, the verify job was trying to verify a the same file with a 2.1.15 version mentioned in it and it detected that the repo was already previously tagged. So it failed. 

  4. Jorge Hernandez

    This is a great improvement in the process, will be even better when the releasing docker images is added.

    With the exception of a glitch for one of the few repos we released where a partial release was done by the new tooling, the rest of the process seems smooth.   This is the process I followed and worked ok for me.

    1. Manually trigger the staging job once satisfied with current release contents by going to the gerrit review in the appropriate repo and type "stage-release".
    2. Once done, note the "*-maven-stage-*" job that will go into your "releases/<version>.yaml" file.
    3. Once done, note the "*-maven-docker-stage-*" job, this will be used to open a ticket in http://support.linuxfoundation.org to request the manual release process of the docker image (if there's any associated with the repo).
    4. Create the "releases/<version>.yaml" and merge in a gerrit review.
    5. If merge goes well, verify that your maven artifacts are released under https://nexus.onap.org/content/repositories/releases/.
    6. Open the ticket in http://support.linuxfoundation.org to request manual release of docker image noted in step 3.   This won't be necessary when the self-release process is supported for docker images.

    After releasing, bump the SNAPSHOT release +1, go on with your development process, and go to #1 when ready to release again.

    1. Jessica Wagantall

      Thanks Jorge Hernandez

      Actually, while the introduction of self releases was happening, my team an myself were working on Nexus3 releases too. 

      This process is still 1/2 way completed due to the fact that we see a lot of tag inconsistencies in the snapshots repository. 

      Trying to figure all the tagging and how to also benefit other projects is taking a little bit longer than expected. 

  5. Vijay Venkatesh Kumar

    Jessica Wagantall  - Could you clarify how (and where) the release files should be created for repo containing independent sub-project? An example here will be dcaegen2/services, which has two sub-projects currently.

    components

        ├── dl-handler

        ├── bbs-event-processor

    As the version can overlap, maintainling release/<major.minor.patch>.yaml under root will cause conflict.

    Can the release file path be specified as separate argument? If so, we can maintain under corresponding sub-tree. 

  6. Jessica Wagantall

    Vijay Venkatesh Kumar  

    After looking closer, dcaegen2/services is the only lucky project under this category. 

    Give me some time to investigate the workflow for this particular case and see if the current process works for it.

    Sorry for this, but in this case, please create an LF support ticket for help on your release for this project.

    Ill get back to you on this.  

    1. Vijay Venkatesh Kumar

      Jessica Wagantall  - Thanks for looking into this. 

       Besides dcaegen2/services, we actually have one more repo/job (dcaegen2-platform-cli-bp-gen) using similar construct.  I will log  LF support ticket to track these cases.

  7. Jessica Wagantall

    Vijay Venkatesh Kumar

    I spoke to Aric about this case. 

    Looks like this case is going to be covered in our next iteration for docker self releases. 

    https://gerrit.linuxfoundation.org/infra/#/c/releng/global-jjb/+/16470/


    Once you are ready to perform your releases, can you please continue as usual?

    Create your releases files in the root of dcaegen2.services.

    You can name your files:

    <root>/releases/1.0.0-component1.yaml

    <root>/releases/1.0.0-component2.yaml
     kind of thing... 


    Please let me know before you merge any changes since we might need some manual intervention as the patch that covers your case is still under

    development... 


  8. Brian Freeman

    Once a release candidate is build using gerrit-maven-stage, a new file will be posted describing the release.

    did you mean:

    Once a release candidate is build using gerrit-maven-stage, a new file must be added to your project repo describing the release and refering to the gerrit-maven-stage log.

    1. Jessica Wagantall

      Yes,, basically, the file releases/#.#.#.yaml

      Let me correct the info with what you worded.. 

  9. Pamela Dragosh

    This new process does not tag the repositories from what I see. That will now be on the committers/PTL's responsibility to do that? Or could something be put into the gerritt jobs to automatically do the tagging on the repository?

    1. Pamela Dragosh

      Nevermind - git pull --rebase doesn't seem to pull them. when I use 'git fetch --tags', that pulls the tags.

  10. hongyu zhao

    hello,please look this Self-Serve Release verify fail, i don't know how to deal with it.

    https://jenkins.onap.org/job/vfc-nfvo-driver-vnfm-svnfm-zte-release-verify/7/