Self releases are now possible after the migration to latest global-jjbs staging jobs
gerrit-maven-stage and gerrit-maven-docker-stage
Before teams start using self releases ...
Please note that this process will be possible by committers with Merge/+2 Rights in the repos.
Please make sure your INFO.yaml files are updated and that all committers have been added.
More information of these procedure can be found here. (Please make sure to read this and ask any
questions to support.linuxfoundation.org:
https://github.com/lfit/releng-global-jjb/blob/master/docs/jjb/lf-release-jobs.rst
How does it work?
The process is very simple. It will require for tech teams to post a new "releases" folder with a yaml file for each release
they need. This file gets verified and, once merged, the release will be posted.
Here are the steps:
- The tech team needs to make sure their committer list is up to date as these will be the people approved to make releases.
- The reason behind this, only committers can +2 and merge changes in their tech team repo.
- The releases merge job will be triggered by files merged in the repo and will execute a release.
- Tech team needs to add "{project-name}-gerrit-release-jobs" to their ci-management yaml files.
- This group introduces "gerrit-release-verify" and "gerrit-release-merge"
- Once a release candidate is build using gerrit-maven-stage, a new file must be added to your project repo describing the release and referring to the gerrit-maven-stage log:
- This file needs to be located in the root repo under a "releases" folder.
- An example can be viewed here: https://github.com/lfit/releng-global-jjb/blob/master/docs/jjb/lf-release-jobs.rst
- Name of the file should match the semantic version of the release being published. (For example: releases/1.0.0.yaml)
- distribution_type: 'maven' (Future expansion will allow "container" to be provided"
- version: '#.#.#' (Release semantic version)
- tag_release: false (By default, tagging a repo with a release version is set to true. If you want to skip it, set it to false)
- project: 'project-name' (Project name, for example 'ccsdk-parent')
- log-dir: 'pointer_to_maven_stage_job/build_number/' (for example: 'ccsdk-parent-maven-stage-master/2/')
- maven_central_url: 'oss.sonatype.org' ( Optional, in case the team want's to publish to Maven Central)
- This file will trigger "{project-name}-release-verify-{stream}"
- This job can also be triggered using the comment "recheck|reverify"
- The verify job will make sure the release file contains the needed information and that the candidate exists.
- Tech team needs to +2 this new change and merge it. Please do not override any -1 Verify from Jenkins.
- The merge will trigger "{project-name}-release-merge-{stream}"
- This job can also be triggered using the comment "remerge"
- This job will push the release and tag the repo.
After your self release ...
Once your self release file was merged and processed by "gerrit-release-verify" and "gerrit-release-merge"...
- DO NOT attempt to revert files in releases/ → Even if the release was not needed, we need to keep track on what happened in that repo also the tag needs to be kept in the repo
- DO NOT attempt to re-tag the repo with the same version → this will fail as the gerrit-release-merge job already tagged it
- DO NOT modify releases files → Once a releases file is merged, it is immediately processed. If the team needs to release a new build number, please bump your versions and generate another stage-release
- DO NOT re-use the same stage-release build number for multiple releases files → Once an autorelease package is pushed, it is closed and it cannot be re-released.
- MAKE SURE your project is 100% free from using nexus-staging-maven-plugin. This was a must do in the global-jjb migration and teams still using it WILL BE facing issues deploying all their artifacts.
- Using this plugin, pre-deploys image in a bad way before lftools has a chance to compile all artifacts needed for the release candidate
- DO NOT cherry-pick release files across branches. Duplicated release files will make the verify and merge job fail as those release were already processed.
- Releases from master should be created in master, releases from sub-branches like "el-alto" should be created in "el-alto"
22 Comments
Dan Timoney
These instructions are a great start, but I strongly encourage people moving to global-jjb to read the "release jobs" section of the global-jjb documentation that Jess posted above (https://github.com/lfit/releng-global-jjb/blob/master/docs/jjb/lf-release-jobs.rst). There are excellent examples there of both the release YAML and of the yaml you need to add to your existing jjb to add the new release jobs.
Some specific notes/tips:
Jessica Wagantall
Thanks Dan Timoney this information you mentioned is 100% accurate.
Point #2 has been moved to my "After your self release ..." section. This is very important becuase if this teams are still using the deprecated plugin somewhere, they will face issues with their release artifacts not being complete like we have done in some cases.
Jonathan Gathman
Moving my "Feed back" To Comment Section, like Dan.
FEEDBACK:
Jonathan (AAF):
RESOLUTION of my mistake:
The thing to remember is that CREATING and CHECKING IN the "release" file above WILL cause the Self-Release mechanism of the FINAL Release of that version. You do NOT want that for "Staging Release".... Use magic word "stage-release" for that.
Try not to touch these files anymore, because if they are messed up, the "release-version" process will kick off. With normal processes, you can't Delete it, you can't fix it, etc. It has to be removed out of band, and while I solved, LF prefers that you enlist their help instead.
Jessica Wagantall
Thanks Jonathan Gathman Sorry for all the scramble but, as we experienced, is always best to create new file.
The process follows a lot what I was doing manually when releasing artifact, for example:
However!, the teams can still ask for a TSC approval for LFRELENG to remove a release. This process will take a lot of manual intervention from my part though, which in an emergency can be done.
Jonathan Gathman
We got through it and I truly appreciate all your help. But your answers don't seem to reflect that my case was an unintentional change in file. I had absolutely no intention of changing it... it was caught in a "Global search/replace" from 2.1.15 to 2.1.16.
IF that happens, then Apps will find that they are in a Race Condition: The file WON'T work correctly, because version doesn't match other data and causes "Version validate" to fail.
But neither can the End user fix or remove the offending file without the processing trying to do "Version Validate", and failing. I saw evidence that the old file remained as well, though I can't prove it now.
That was why I took the step that was severely reprimanded... it was the only way for an end-user to clear the file that wouldn't let me proceed because any touch of that file caused failure. Catch 22.
Jonathan Gathman
Accidental or purposeful change of the "releases/X.yaml" is a problem. Any change in this directory causes the attempt for a final release. Removal of the file... attempts to validate the final release... Catch 22.
Best thing to do is invoke LF, and work on getting it out of that Directory without normal CI/CD processes.
Jorge Hernandez
I believe the releases/2.1.15.yaml should remain untouched moving forward. I don't think a new releases/2.1.6.yaml should be added until the team is ready to do a release, in part because you are probably not ready to designate that "blessed" staging job that would cut the release (log_dir section).
We have a 2 release files example here: https://git.onap.org/policy/models/tree/releases, so multiple release files seems to be working ok at least for us. AS background, we needed to publish 2 releases as the first release was partially published in error by the new tooling (which I believe the LF team has corrected already).
Jonathan Gathman
That makes sense.
But did you have any issues with the "release-verify" job, that subsequently set "ONAP Jobbuilder" failures during simple checkin?
Having that file remain there causes mine to fail...
Jorge Hernandez
Umm, not sure, I didn't run into this particular problem. In my case, for the second release file review https://gerrit.onap.org/r/#/c/policy/models/+/92734/, the release-verify job was sucessful (see in the comments section)>
Jonathan Gathman
Resolution: get the file back were it was outside of normal Release... See LF for help.
Jessica Wagantall
The tag error happened because there was already a previously released 2.1.15 version. Weather it was a good release or a bad release file, the merge jenkins job released whatever it could
and it tagged the Gerrit repo with that tag.
On the next round, the verify job was trying to verify a the same file with a 2.1.15 version mentioned in it and it detected that the repo was already previously tagged. So it failed.
Jorge Hernandez
This is a great improvement in the process, will be even better when the releasing docker images is added.
With the exception of a glitch for one of the few repos we released where a partial release was done by the new tooling, the rest of the process seems smooth. This is the process I followed and worked ok for me.
After releasing, bump the SNAPSHOT release +1, go on with your development process, and go to #1 when ready to release again.
Jessica Wagantall
Thanks Jorge Hernandez
Actually, while the introduction of self releases was happening, my team an myself were working on Nexus3 releases too.
This process is still 1/2 way completed due to the fact that we see a lot of tag inconsistencies in the snapshots repository.
Trying to figure all the tagging and how to also benefit other projects is taking a little bit longer than expected.
Vijay Venkatesh Kumar
Jessica Wagantall - Could you clarify how (and where) the release files should be created for repo containing independent sub-project? An example here will be dcaegen2/services, which has two sub-projects currently.
components
├── dl-handler
├── bbs-event-processor
As the version can overlap, maintainling release/<major.minor.patch>.yaml under root will cause conflict.
Can the release file path be specified as separate argument? If so, we can maintain under corresponding sub-tree.
Jessica Wagantall
Vijay Venkatesh Kumar
After looking closer, dcaegen2/services is the only lucky project under this category.
Give me some time to investigate the workflow for this particular case and see if the current process works for it.
Sorry for this, but in this case, please create an LF support ticket for help on your release for this project.
Ill get back to you on this.
Vijay Venkatesh Kumar
Jessica Wagantall - Thanks for looking into this.
Besides dcaegen2/services, we actually have one more repo/job (dcaegen2-platform-cli-bp-gen) using similar construct. I will log LF support ticket to track these cases.
Jessica Wagantall
Vijay Venkatesh Kumar
I spoke to Aric about this case.
Looks like this case is going to be covered in our next iteration for docker self releases.
https://gerrit.linuxfoundation.org/infra/#/c/releng/global-jjb/+/16470/
Once you are ready to perform your releases, can you please continue as usual?
Create your releases files in the root of dcaegen2.services.
You can name your files:
<root>/releases/1.0.0-component1.yaml
<root>/releases/1.0.0-component2.yaml
kind of thing...
Please let me know before you merge any changes since we might need some manual intervention as the patch that covers your case is still under
development...
Brian Freeman
Once a release candidate is build using gerrit-maven-stage, a new file will be posted describing the release.
did you mean:
Once a release candidate is build using gerrit-maven-stage, a new file must be added to your project repo describing the release and refering to the gerrit-maven-stage log.
Jessica Wagantall
Yes,, basically, the file releases/#.#.#.yaml
Let me correct the info with what you worded..
Pamela Dragosh
This new process does not tag the repositories from what I see. That will now be on the committers/PTL's responsibility to do that? Or could something be put into the gerritt jobs to automatically do the tagging on the repository?
Pamela Dragosh
Nevermind - git pull --rebase doesn't seem to pull them. when I use 'git fetch --tags', that pulls the tags.
hongyu zhao
hello,please look this Self-Serve Release verify fail, i don't know how to deal with it.
https://jenkins.onap.org/job/vfc-nfvo-driver-vnfm-svnfm-zte-release-verify/7/