Release Status
The release status of the policy framework is stored in the policy/parent repository in the pf_release_data.csv file. Look at that file under each branch to find details of the current release artifacts for each branch. The table below is a shortcut to that file for each release.
Release | Release Data |
---|---|
master | pf_release_data.csv |
kohn | not branched yet |
jakarta | pf_release_data.csv |
istanbul (security updates only) | pf_release_data.csv |
honolulu (security updates only) | pf_release_data.csv |
previous releases (unsupported) | Policy Framework Project: Component Versions |
Repository Dependencies and Docker Images
The diagram below shows the repositories and Docker images produced by the Policy Framework. The repositories (released as maven artifacts) are shown as rectangles, and the Docker images are shown as ovals. The dependencies between the repositories are shown as are the Docker images produced by each repository.
The Policy Framework Release Process
The Policy Framework release process is complex, time-consuming, and tedious. In the release process, the maven repositories and docker images must be released in the order shown in the diagram above: Blue-->Green-->Pink-->Yellow→Purple. In addition, the references for the repository layering and the parent Docker image file references must be set to release references before staging and back to snapshot references after staging.
A set of scripts have been developed to make the release process somewhat less tedious. The scripts are located in the policy/parent repo here.
The scripts guide one through the release process. The scripts:
- Run each step in the release process as a sequence of release phases
- Act on pom.xml, Docker, version.properties, and pf_release_data.csv files, changing the references in those files as is required for each phase in a release
- Generate the commits in Gerrit for each phase of the release
- Work on all branches from Honolulu on
- Execute a normal release, where there are no special cases like repositories or docker images being added or removed
The scripts do not:
- Provide a fully automatic release, manual checking and submitting of commits is still required
- Deal with all possible permutations and combinations of releases
- Guarantee that there are not bugs in the scripts, manual oversight is required
The scripts make the following assumptions:
- All 13 Policy Framework repositories are freshly cloned into a directory called policy
policy/parent
policy/common
......... - There are no updates or pending commits on any of the cloned repositories
- You have checked out the scripts and added their location to your PATH
- All references between Policy Framework Maven modules are in the root POM in each repository
- All references between Policy Framework Maven modules are controlled by the following properties in root pom.xml files
policy.common.version or version.policy.common: policy/common
policy.models.version or version.policy.models: policy/models
policy-drools-pdp.version or version.policy.drools-pdp: policy/drools-pdp
- policy-apex-pdp.version or version.policy.apex-pdp: policy/apex-pdp
- All files that match the regular expression '*Docker*' and have their base image matching the regular expression 'FROM onap\/policy-j[d|r][k|e]-alpine:)[0-9]*.[0-9]*.[0-9].*$' will have their base image changed to the appropriate policy-jre-alpine or policy-jdk-apline version for the phase in question
Getting Release Data and Performing a Release Phase
During a normal release, the most important scripts are the getReleaseData.sh script and the releasePhase.sh script.
getReleaseData.sh creates the file pf_release_data.csv for a specified branch. This file is used by the release process to keep track of where it is. It has the following 5 columns:
- Repo: The repository to which this row of the file refers
- Last Tag Version: The last hard release version of this repository on this branch
Snapshot Version: The current snapshot version of this repository on this branch
- Changed Files: The number of files that are changed between the Last Tag Version and the Snapshot Version
- Docker Images: A list of the Docker images generated on this repository, which can be none
getReleaseData.sh recreates the pf_release_data.sh each time it is executed so as the release proceeds through its phases, the content of the file changes. The script has a list of the 13 Policy Framework repositories, and for each repository it:
- Checks out the specified branch, pulls the latest changes, and rebases the repository
- Runs a "mvn clean" command on the repository to determine its current snapshot
- Reads the tags on the repository and determines what the latest tag on this branch of the repository is
- Runs a "git diff" command to detemine the number of changed files between the latest tag and the current snapshot
- Reads the releases/x.x.x-container.yaml file to find the docker images generated by the repository
releasePhase.sh is an orchestration script that runs the 15 phases in the release process. It calls the other scripts in the script set to perform whatever actions are required at each phase in the execution process. It has a function that codifies each phase of the release.
Each phase of the release is executed by calling getReleaseData.sh to update the status of the release phase followed by releasePhase.sh to actually perform the release. Note that it is safe to run getReleaseData.sh multiple times in a single release. This may be necessary if, for example, the merge job for a release commit is slow to run causing a phase to complete slowly. Always keep an eye on the output of getReleaseData.sh to verify that the release is proceeding correctly.
Running a Release
In each phase run getReleaseData.sh followed by releasePhase.sh.
% getReleaseData.sh -l onap -b jakarta % releasePhase.sh -l onap -i POLICY-112911 -p 1 % getReleaseData.sh -l onap -b jakarta % releasePhase.sh -l onap -i POLICY-112911 -p 2 ........................... % getReleaseData.sh -l onap -b jakarta % releasePhase.sh -l onap -i POLICY-112911 -p 15
Phase | Actions Performed |
---|---|
1 | Update internal references in policy/parent |
2 | Stage release on policy/parent in Gerrit |
3 | Update snapshots in policy/parent |
4 | Stage release on policy/common in Gerrit |
5 | Update snapshots in policy/common Update snapshots in policy/docker Update policy/parent and policy/common references in policy/models |
6 | Stage release on policy/models in Gerrit Release policy/models Maven artifacts |
7 | Release policy/models Docker images |
8 | Update snapshots in policy/models Update policy/parent, policy/common, policy/models and base Docker references in policy/api |
9 | Stage release on policy/api in Gerrit |
10 | Release policy/api Docker images |
11 | Update snapshots in policy/api Update snapshots in policy/pap Update snapshots in policy/distribution Update snapshots in policy/clamp Update snapshots in policy/apex-pdp Update snapshots in policy/drools-pdp Update snapshots in policy/xacml-pdp Update policy/parent, policy/common, policy/models, policy/drools-pdp and base Docker references in policy/drools-applications Update policy/parent, policy/common, policy/models, policy/apex-pdp and base Docker references in policy/gui |
12 | Stage release on policy/drools-applications in Gerrit |
13 | Stage release on policy/drools-applications in Gerrit (Note: a second stage-release is required on drools-applications following maven artifact release) |
14 | Update snapshots in policy/drools-applications |
15 | Store the updated pf_release_data.sh file in policy/parent |
Updating image revisions in OOM
When releasing in ONAP, the revisions of the Policy Framework Docker images must be updated in OOM.
The image revisions in the Policy Framework values.yaml files are updated using the updateOomImages.sh script. The script updates the image revisions in OOM and creates a commit in Gerrit for the change.
- Clone a copy of the OOM repository into a suitable location. The suggested location is onap/oom, where the Policy Framework repositories are in onap/policy/parent, onap/policy/common etc.
- Make sure that the correct branch of OOM is checked out, if you are, for example, working on a Jakarta release of the Policy Framework, check out the Jakarta branch on the OOM repository.
Run the updateOomImages.sh script:
% updateOomImages.sh -l onap -i POLICY-112911
A commit is raised that updates OOM for the new Policy Framework release.
Branching
Branching is where a new branch is created on each repository in the Policy Framework and the minor revision of each repository is stepped. Branching is usually performed when development is compelted for a release and development is moving onto the next release.
<TBC>
Performing a Release
A Release is where the all the Maven artifacts and Docker images are released on the current branch and and the snapshot version of the repositories is stepped.
The process of performing a release is best illustrated by example. Let's assume we are on the Jakarta release. We have checked out the Policy Framework repositories into the onap/policy subdirectory of the current directory and the release scripts are in our PATH.
% echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/liam/bin:/Users/meself/git/onap/policy/parent/integration/src/main/scripts/release % ls onap/policy apex-pdp clamp distribution drools-applications gui pap xacml-pdp api common docker drools-pdp models parent
We have also created a JIRA called POLICY-112911 to cover the task of performing the release.
Phase 1: Update references in policy/parent
Get the release data.
The -l parameter is pointing to the location of the checked out Policy Framework repositories (onap/policy/parent, onap/policy/common etc) and -b specifies the branch to sue for the release. If the -b option is not specified, the master branch is used.
% getReleaseData.sh -l onap -b jakarta updating the policy framework data from 'onap' to data file './pf_release_data.csv' . . . updating data from repo policy/parent branch jakarta to data file './pf_release_data.csv' . . . updating repository 'policy/parent' . . . Already on 'jakarta' Your branch is up to date with 'origin/jakarta'. Already up to date. Current branch jakarta is up to date. data from repo policy/parent updated to data file './pf_release_data.csv' . . . updating data from repo policy/docker branch jakarta to data file './pf_release_data.csv' . . . updating repository 'policy/docker' . . . Already on 'jakarta' Your branch is up to date with 'origin/jakarta'. Already up to date. Current branch jakarta is up to date. data from repo policy/docker updated to data file './pf_release_data.csv' . . . updating data from repo policy/common branch jakarta to data file './pf_release_data.csv' . . . updating repository 'policy/common' . . . Already on 'jakarta' Your branch is up to date with 'origin/jakarta'. Already up to date. Current branch jakarta is up to date. data from repo policy/common updated to data file './pf_release_data.csv' . . . updating data from repo policy/models branch jakarta to data file './pf_release_data.csv' . . . updating repository 'policy/models' . . . Already on 'jakarta' Your branch is up to date with 'origin/jakarta'. Already up to date. Current branch jakarta is up to date. data from repo policy/models updated to data file './pf_release_data.csv' . . . updating data from repo policy/api branch jakarta to data file './pf_release_data.csv' . . . updating repository 'policy/api' . . . Already on 'jakarta' Your branch is up to date with 'origin/jakarta'. Already up to date. Current branch jakarta is up to date. data from repo policy/api updated to data file './pf_release_data.csv' . . . updating data from repo policy/pap branch jakarta to data file './pf_release_data.csv' . . . updating repository 'policy/pap' . . . Already on 'jakarta' Your branch is up to date with 'origin/jakarta'. Already up to date. Current branch jakarta is up to date. data from repo policy/pap updated to data file './pf_release_data.csv' . . . updating data from repo policy/apex-pdp branch jakarta to data file './pf_release_data.csv' . . . updating repository 'policy/apex-pdp' . . . Already on 'jakarta' Your branch is up to date with 'origin/jakarta'. Already up to date. Current branch jakarta is up to date. data from repo policy/apex-pdp updated to data file './pf_release_data.csv' . . . updating data from repo policy/drools-pdp branch jakarta to data file './pf_release_data.csv' . . . updating repository 'policy/drools-pdp' . . . Already on 'jakarta' Your branch is up to date with 'origin/jakarta'. Already up to date. Current branch jakarta is up to date. data from repo policy/drools-pdp updated to data file './pf_release_data.csv' . . . updating data from repo policy/xacml-pdp branch jakarta to data file './pf_release_data.csv' . . . updating repository 'policy/xacml-pdp' . . . Already on 'jakarta' Your branch is up to date with 'origin/jakarta'. Already up to date. Current branch jakarta is up to date. data from repo policy/xacml-pdp updated to data file './pf_release_data.csv' . . . updating data from repo policy/distribution branch jakarta to data file './pf_release_data.csv' . . . updating repository 'policy/distribution' . . . Already on 'jakarta' Your branch is up to date with 'origin/jakarta'. Already up to date. Current branch jakarta is up to date. data from repo policy/distribution updated to data file './pf_release_data.csv' . . . updating data from repo policy/clamp branch jakarta to data file './pf_release_data.csv' . . . updating repository 'policy/clamp' . . . Already on 'jakarta' Your branch is up to date with 'origin/jakarta'. Already up to date. Current branch jakarta is up to date. data from repo policy/clamp updated to data file './pf_release_data.csv' . . . updating data from repo policy/gui branch jakarta to data file './pf_release_data.csv' . . . updating repository 'policy/gui' . . . Already on 'jakarta' Your branch is up to date with 'origin/jakarta'. Already up to date. Current branch jakarta is up to date. data from repo policy/gui updated to data file './pf_release_data.csv' . . . updating data from repo policy/drools-applications branch jakarta to data file './pf_release_data.csv' . . . updating repository 'policy/drools-applications' . . . Already on 'jakarta' Your branch is up to date with 'origin/jakarta'. Already up to date. Current branch jakarta is up to date. data from repo policy/drools-applications updated to data file './pf_release_data.csv' . . . policy framework data from 'onap' updated to data file './pf_release_data.csv' . . . Repo, Last Tag Version,Snapshot Version,Changed Files,Docker Images policy/parent,3.5.4,3.5.5-SNAPSHOT,1, policy/docker,2.4.4,2.4.5-SNAPSHOT,85,'policy-jre-alpine':'policy-jdk-alpine':'policy-db-migrator' policy/common,1.10.4,1.10.5-SNAPSHOT,0, policy/models,2.6.5,2.6.6-SNAPSHOT,2,'policy-models-simulator' policy/api,2.6.4,2.6.5-SNAPSHOT,1,'policy-api' policy/pap,2.6.4,2.6.5-SNAPSHOT,1,'policy-pap' policy/apex-pdp,2.7.4,2.7.5-SNAPSHOT,1,'policy-apex-pdp' policy/drools-pdp,1.10.4,1.10.5-SNAPSHOT,1,'policy-drools' policy/xacml-pdp,2.6.4,2.6.5-SNAPSHOT,1,'policy-xacml-pdp' policy/distribution,2.7.4,2.7.5-SNAPSHOT,1,'policy-distribution' policy/clamp,6.2.4,6.2.5-SNAPSHOT,8,'policy-clamp-backend':'policy-clamp-ac-pf-ppnt':'policy-clamp-ac-k8s-ppnt':'policy-clamp-ac-http-ppnt':'policy-clamp-runtime-acm' policy/gui,2.2.4,2.2.5-SNAPSHOT,2,'policy-gui' policy/drools-applications,1.10.4,1.10.5-SNAPSHOT,6,'policy-pdpd-cl'
Perform Phase 1, update of references in policy/parent
% releasePhase.sh -l onap -i POLICY-112911 -p 1 Updating parent references in the parent pom and generating commit . . . updating policy parent reference to 3.5.5 on onap/policy/parent . . . policy parent reference updated on onap/policy/parent generating commit 'update parent references in policy/parent pom' . . . [jakarta ae6cc59] update parent references in policy/parent pom 1 file changed, 1 insertion(+), 1 deletion(-) commit 'update parent references in policy/parent pom' generated sending commit 'update parent references in policy/parent pom' to gerrit . . . remote: remote: Processing changes: refs: 1, new: 1 remote: Processing changes: refs: 1, new: 1 remote: Processing changes: refs: 1, new: 1 remote: Processing changes: refs: 1, new: 1, done remote: remote: SUCCESS remote: remote: https://gerrit.onap.org/c/onap/policy/parent/+/15332 update parent references in policy/parent pom [NEW] remote: To ssh://gerrit.onap.org:29418/onap/policy/parent * [new reference] HEAD -> refs/for/jakarta commit 'update parent references in policy/parent pom' sent to gerrit . . . Updated parent references in the parent pom and generated commit
The references are updated in the local repositories and a commit is created in Gerrit for the change.
The commit is reviewed and merged as normal.
Phase 2: Release policy/parent
When the commit for Phase 1 has been merged and the merge job has completed, stage the release of the policy/parent Maven artifacts by giving the stage-release magic word as a comment on the release commit. When the stage release job has completed in Gerrit, you can proceed to run Phase 2.
Get the release data.
The output from git on the getReleaseData.sh script is omitted from Phase 2 for brevity.
% getReleaseData.sh -l onap -b jakarta Repo, Last Tag Version,Snapshot Version,Changed Files,Docker Images policy/parent,3.5.4,3.5.5-SNAPSHOT,1, policy/docker,2.4.4,2.4.5-SNAPSHOT,85,'policy-jre-alpine':'policy-jdk-alpine':'policy-db-migrator' policy/common,1.10.4,1.10.5-SNAPSHOT,0, policy/models,2.6.5,2.6.6-SNAPSHOT,2,'policy-models-simulator' policy/api,2.6.4,2.6.5-SNAPSHOT,1,'policy-api' policy/pap,2.6.4,2.6.5-SNAPSHOT,1,'policy-pap' policy/apex-pdp,2.7.4,2.7.5-SNAPSHOT,1,'policy-apex-pdp' policy/drools-pdp,1.10.4,1.10.5-SNAPSHOT,1,'policy-drools' policy/xacml-pdp,2.6.4,2.6.5-SNAPSHOT,1,'policy-xacml-pdp' policy/distribution,2.7.4,2.7.5-SNAPSHOT,1,'policy-distribution' policy/clamp,6.2.4,6.2.5-SNAPSHOT,8,'policy-clamp-backend':'policy-clamp-ac-pf-ppnt':'policy-clamp-ac-k8s-ppnt':'policy-clamp-ac-http-ppnt':'policy-clamp-runtime-acm' policy/gui,2.2.4,2.2.5-SNAPSHOT,2,'policy-gui' policy/drools-applications,1.10.4,1.10.5-SNAPSHOT,6,'policy-pdpd-cl'
Perform Phase 2, release policy/parent
% releasePhase.sh -l onap -i POLICY-112911 -p 2 Generating artifact release yaml file and commit for policy/parent . . . have you run 'stage-release' on the 'policy/parent' repo? y Branch: jakarta Project: policy/parent Version: 3.5.5 Stage ID: policy-parent-maven-stage-jakarta/145/ Creating /Users/liam/work/releases/onap/policy/parent/releases/3.5.5.yaml generating commit for policy/parent release: 3.5.4-->3.5.5 . . . generating commit 'Release policy/parent: 3.5.5' . . . [jakarta 2f0ceb1] Release policy/parent: 3.5.5 1 file changed, 4 insertions(+) create mode 100644 releases/3.5.5.yaml commit 'Release policy/parent: 3.5.5' generated sending commit 'Release policy/parent: 3.5.5' to gerrit . . . remote: remote: Processing changes: refs: 1, new: 1 remote: Processing changes: refs: 1, new: 1, done remote: remote: SUCCESS remote: remote: https://gerrit.onap.org/c/onap/policy/parent/+/15333 Release policy/parent: 3.5.5 [NEW] remote: To ssh://gerrit.onap.org:29418/onap/policy/parent * [new reference] HEAD -> refs/for/jakarta commit 'Release policy/parent: 3.5.5' sent to gerrit . . . commit for policy/parent release: 3.5.4-->3.5.5 generated Generated artifact release yaml file and commit for policy/parent
The onap/policy/parent/releases/3.5.5.yaml file is created and a commit is raised on gerrit.
The commit is reviewed and merged as normal.
Subsequent Phases
Subsequent phases proceed in a similar manner with executions of getReleaseData.sh followed by executions of releasePhase.sh.
Appendix I: Release Script Usage
Script | Description |
---|---|
bumpSnapshots.sh | bumpSnapshots.sh - generate commits to bump the snapshot version and update references to snapshot references |
generateCommit.sh | generateCommit.sh - generates a new commit or a patch on an existing commit for PF releases |
getReleaseData.sh | getReleaseData.sh - gets information from the checked out Policy Framework repos for the release process |
mkart.sh | mkart.sh - create the release yaml file to release the current snapshot on the current repo |
mkdock.sh | mkdock.sh - create the release container yaml file to release the docker images on the current repo |
releasePhase.sh | releasePhase.sh - execute a certain policy framework release phase |
releaseRepo.sh | releaseRepo.sh - release the specified repository by generating the release yaml file and the release commit |
releaseRepoImages.sh | releaseRepoImages.sh - release the docker images for the specified repository by generating the release yaml file and |
updateOomImages.sh | updateOomImages.sh - generate an OOM commit to update the versions of Policy Framework images in values.yaml files |
updateParentRef.sh | updateParentRef.sh - update the parent reference in a POM file usage: updateParentRef.sh [-options] options |
updateRefs.sh | updateRefs.sh - updates the inter-repo references in Policy Framework POM files |
The scripts:
- Operate on a cloned copy of all 13 Policy Framework repositories
- Break the release process into a set of phases, which are executed one after another to perform a release
- Can release on the Master branch or any other branch
- Act on the POM files, Docker files, the maven and docker release files, the version.properties files, and the pf_release_data.csv file in the repos, updating the references in those files as required for each phase
The scripts do NOT:
- Fully automate the release process, manual checking and submission of commits is still required
- Capture every possible permutation and corner case of releasing, manual intervention in the process is often required