releases/doc/source/reference/reviewer_guide.rst

Reviewer Guide

Review Inbox

General Review Considerations

As you're looking at any given review, you need to keep a few different things in mind:

• Where are we in the release cycle? Some rules change depending on the phase we're in.
• What "release model" does the deliverable being released follow? The release model sets some general rules for version numbers and schedule.
• Which branch is the release on?

Many of the rules tied to these questions are enforced by the validation job, so when you see an error understanding the rules helps you understand the error message.

Approval Policies

I prioritize the most current series, since that's where most development work is going to happen.

For the stable series we have an arrangement with the stable-maint-core team that if a deliverable has the stable:follows-policy tag we don't approve it until they have had a chance to review it (usually the Monday after the request is submitted). Releases for deliverables that do not have that governance tag can be approved at any time.

Releases from master can be approved with a single reviewer.

Code changes and doc changes and other things like that need 2 reviewers.

Releases from someone other than the PTL or release liaison must be acknowledged by one of them with a +1 vote in gerrit.

Review Checks

Does the commit message include the name of the deliverable and a version number? If the request includes multiple deliverables it is OK to include a team name and date.

The Validation Report

The validation job, openstack-tox-validate, applies the validation rules that can be automated. It produces a text report in tox/validate-request-results.log. The file contains the output of what you would see if you ran tox -e validate for the patch.

The output is organized based on the rule being enforced.

We've tried to separate the "debug" output so it is easier to skim for real content, with the important output left justified.

Warnings and errors are summarized at the bottom of the file.

The List Changes Report

The releases-tox-list-changes job produces a text report to support human reviewers. It writes the report to tox/list-changes-results.log. As with the validate job, it can be run as tox -e list-changes locally.

Reviewers should read this log file for every review. It includes all of the information needed to evaluate a release. The List Changes Report has multiple sections you will need to review.

Release model

At the top of the file we get the release model, which tells us things like when releases are allowed, what version numbers are allowed, etc.

Team details

The "team details" section tells us the PTL and Liaison, so we know who to make sure has acknowledged the request. If one of those people proposed the patch, we can go ahead without any delay. Otherwise we want to make sure one of them knows about the release and approves it so that teams know we aren't going to release things they know are broken, for example.

Tags

Next the report shows the governance tags for the repository. If the request is for a release on a stable branch and the project has that stable:follows-policy tag, there will be a large banner that says the release needs to be approved by the stable team. Releases from master will not include the banner, regardless of whether the deliverable has the tag.

Details for commit receiving new tag X.Y.Z

In the "Details for commit receiving new tag..." section (below the DEBUG lines) the report shows what git thinks the previous tag and number of added patches should be. That's a quick way to verify that we aren't tagging 1.8.0 after 1.9.0 or something like that.

Check existing tags

The next section shows any other tags already on the commit being tagged. Sometimes a team will have a 3-part deliverable but only 1 part changes in a release. If they have defined the 3 parts as 1 deliverable, they should tag all 3 anyway.

All branches with version numbers

The next section shows what versions are on all of the branches. This is somewhat important, since for the first release off of master after creating a stable branch we want to make sure we are moving ahead in version numbers. The validation job requires that least the Y value in a X.Y.Z version number is incremented.

Branches containing commit

The next step shows which branch(es) contain(s) the commit. That's useful for ensuring that someone has not merged 2 branches together and we are not releasing off of the wrong branch.

For the current cycle, releases should always come from the master branch. Stable releases should come from the appropriate stable branch.

Relationship to HEAD

The "Relationship to HEAD" section tells us if the release will skip any commits. Sometimes someone uses a commit hash locally that is older than the most recent commit on the branch. If this section does not say it is releasing HEAD (Request releases from HEAD), it is good ask the submitter to verify that they're doing what they mean to be doing. Sometimes they don't want to release the additional changes, and sometimes they don't know about them. It is not necessary to take this extra precaution for milestone tags, because those are date-based and it doesn't really matter if they don't include everything. We expect a lot of churn and progress around the milestone deadlines.

Open patches, Documentation patches and Patches with Release Notes

The next couple of sections show open patches matching various criteria. These are useful close to the release candidate phase of the cycle. When we are close to a freeze date the release team might encourage teams to approve outstanding changes for requirements updates, release notes, and translations before releasing.

Requirements Changes

The next two sections, "Requirements Changes..." and "setup.cfg Changes...", show the dependencies that have changed for the project since the last time it was tagged. We use those to ensure that the exception to the SemVer rules is applied:

• Projects tagging a regular release (not a "pre-release" like an alpha, beta, or rc) need to increment at least the Y part of their version number when the minimum version of a dependency changes or when a new dependency is added.

The report shows the changes to the test requirements as the second part of the "Requirements Changes" section. Those do not trigger Y version changes.

Release X.Y.Z will include

The "Release $version will include" section shows the actual changes being included in the new release -- the difference since the last version was tagged. This is where the subjective part of the review really comes in. If a patch release is being tagged and something in this list looks like a new feature, we want them to tag a minor update instead. If anything in the list appears to describe a backwards-incompatible change, we want them to tag a major version update.

The git log section gives a more detailed view of the log messages. Look for comments like "delete class X" or "add argument Y to method B" to indicate the release will not be backwards-compatible. It is not necessary to lower a version number, say, if the release does not have new features and has only fixed a bug. Sometimes if there is only one change and it is clearly a bug fix we may ask them to do that, but most of the time releases include a mix of fixes and features.

Another thing to look for is if there are only CI configuration changes. There is no reason to tag a release if the only change was to the zuul or tox configuration, because the end user won't see those changes. That happens sometimes with the projects that have a script to prepare the release proposal.

The next part of the output (below the Release Notes) shows the same text that will appear in the release announcement email. It is included so that if building that text fails for some reason this job will fail and the reno input files can be fixed instead of having the announce job fail.

Users of $PROJECT

The final part of the output is a list of projects that have the current deliverable being released in one of their dependency lists. That section is useful for evaluating the impact of a late release when we're in the freeze period.

Release Jobs

When a release request is submitted the check-release-approval job will be triggered to check that release requests were approved by PTL or release liaison.

After a release request merges, the tag-releases job will start up in the release-post pipeline.

tag-releases reads the file from the releases repository and adds the tag to the repository mentioned in the deliverable file.

Adding the tag triggers another job that actually builds the release and uploads it.

After a Python package release is uploaded, the job propose-update-constraints submits a change to openstack/requirements to update the upper-constraints.txt list. The constraints list is used along with the actual requirements list to tell the jobs which versions of which packages to install. Since we maintain that list, every time we release something that is constrained we want to make sure the value is updated. The job runs for all python packages, but not all of them are in the constraint list so sometimes it does not submit a patch.

Release Job Failures

When release jobs fail, messages are sent to release failure mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures

Release Approval Status

Depending on the kind of job failures experienced it could be mandatory to stop all our release approvals.

Indeed, sometimes job failures are systemic and should be fixed first to avoid repeated failures wich could lead inconsistent states in our coordinated releases.

To answer that case we defined three statuses to indicate if we should/shouldn't continue to validate patch:

• RED: no more approvals;
• ORANGE: a transient status where we think that the issue is solved but approvals must be carefully monitored first;
• GREEN: the issue have been fixed and everything works as expected. (approvals are reopen).

To inform all the release managers that something went wrong and ask them to hold approvals then follow the following process:

1. open a new thread on the ML with for topic [release] Status: RED - $subject to indicate the issue
2. notify directly the release managers on IRC (#openstack-release)

When you think that the problem is solved but that it still need some tests you just have to reply on the thread by moving the topic from RED to ORANGE.

When everything seems under control then you can reply on the thread by moving the topic from ORANGE to GREEN.