The stable branches are intended to be a safe source of fixes for high impact bugs and security issues which have been fixed on master since a given release.
Stable branches are cut from the last release of a given deliverable, at the end of the common 6-month development cycle.
Project stable branches will be in one of the following states:
|Maintained||Approximately 18 months||All bugfixes (that meet the criteria described below) are appropriate. Releases produced.|
|Extended Maintenance||While there are community members maintaining it.||All bugfixes (that meet the criteria described below) are appropriate. No Releases produced, reduced CI commitment.|
|Unmaintained||0 - 6 months||The branch is under Extended Maintenance rules, but there are no maintainers.|
|End of Life (EOL)||N/A||Branch no longer accepting changes.|
It is not required that all projects for a given branch transition between phases at the same time. For example it's quite reasonable for the
stable/$series branch of
openstack/long-life-project to still be in the Maintained phase while all other projects have transitioned to either Extended Maintenance or even End of Life.
At this time the exact mechanism for describing and updating this state is undefined but it's probable it will involved updating a meta-data in a projects deliverable file in the
For any project/branch combination that is considered Maintained, OpenStack Infrastructure, OpenStack Vulnerability Management and QE tools are expected to work and be active. Project teams will produce consumable releases and upgrades are tested.
Per Project Stable teams and the Stable Maintainers are responsible for all tagged projects during the this phase.
Once a branch reaches Extended Maintenance project teams will cease producing releases and OpenStack Vulnerability Management will be reasonable efforts only. There is no statement about the level of testing and upgrades from Extended Maintenance are not supported within the Community.
last release of the appropriate branch will be tagged as
$series-em, for example: https://review.opendev.org/608296/. For all projects that follow the stable policy a patch with a
$series-em tag will be automatically generated after the final release from the latest development cycle happened. This is because this is a less busy period in development perspective compared to feature freeze and release periods.
Members of the community interested in a given project/branch are encouraged to engage with the appropriate stable team early in its life-cycle to ensure this process runs well. In the absence of identified maintainers the project will immediately enter the 6 month notification period as described under End of Life below.
Some project teams may choose to NOT enter extended maintenance and go directly to End of Life. At this point should a group wish to maintain that branch of a project they can do so within license and trademark constraints. Some OpenStack CI testing may be available via Zuul drivers
At this stage of the project/branch the Extended Maintenance policy applies but CI may not be working and/or there aren't any active maintainers. Projects that remain in this state for 6 months will be transitioned to End of Life. Should maintainers be found a project can be placed back into Extended Maintenance.
After a project/branch exceeds the time allocation as Unmaintained, or a team decides to explicitly end support for a branch, it will become End of Life. The HEAD of the appropriate branch will be tagged as
$series-eol and the branch deleted.
To initiate this transition, either the PTL of the given project or other stable maintainer should:
other repositoriesand not needed anymore.
Only a limited class of changes are appropriate for inclusion on the stable branch. A number of factors must be weighed when considering a change:
It's nevertheless allowed to backport fixes for other bugs if their safety can be easily proved. For example, documentation fixes, debug log message typo corrections, test only changes, patches that enhance test coverage, configuration file content fixes can apply to all supported branches. For those types of backports, stable maintainers will decide on case by case basis.
Some patches may get exception from rule 4 above. These are patches that do not touch production code, like test-only patches, or tox.ini changes that fix major gate breakage, etc.; or security patches that should not take much time to merge once the patches are published. In those cases, stable patches may be pushed into gate without waiting for all consequent branches to be fixed.
Anyone can propose stable branch backports. See Proposing Fixes for more information on how to do that.
Each project team should designate a stable branch cross-project liaison as the main point of contact for all stable branch support issues in the team. If nobody is specifically designated, the PTL will be assumed to cover that duty.
Each project with a stable branch will have a project-specific stable maintenance Gerrit team called PROJECTNAME-stable-maint. This team will have CodeReview+2 and Workflow+1 rights over the stable branches, and be in charge of reviewing backports for a given project, following the rules of the stable branch policy. Originally that group should be the project Stable Branch Cross-Project Liaison + the stable maintenance core team. Those groups are managed by the stable maintenance core team, names are added after the suggestion of the Stable Branch cross-project liaison.
The stable maintenance core team is responsible for the definition and enforcement of the Stable Branch policy. It will be granting exceptions for all questionable backports raised by project-specific stable maintenance groups, providing backports reviews help everywhere, maintaining the stable branch policy (and make sure its rules are respected), educating proposed project-specific team members on those rules and adding them to those project-specific teams.
Project-specific teams are expected to be actively maintaining their stable branches which generally includes:
Monitoring and resolving issues in the continuous integration 'gate' system. This basically means making sure there aren't things blocking proposed backports from passing tests. These could be project-specific or global in nature and are usually tracked in the stable tracker etherpad. From time to time the Stable Maintenance Core team may also ask for help from individual projects in IRC or the openstack-discuss mailing list and expect a reasonably prompt response.
Each project stable review team need to balance the risk of any given patch with the value that it will provide to users of the stable branch. A large, risky patch for a major data corruption issue might make sense. As might a trivial fix for a fairly obscure error handling case.
Some types of changes are completely forbidden:
Proposed backports breaking any of the above guidelines can be discussed as exception requests on the openstack-discuss list (prefix with [stable]) where the stable maintenance core team will have the final say.
Each backported commit proposed to Gerrit should be reviewed and +2ed by two project-specific stable maintenance team members before it is approved. Where a team member has backported a fix, a single other +2 is sufficient for approval.
If unsure about the technical details of a given fix, project-specific stable maintenance team members should consult with the appropriate project core reviewers for a more detailed technical review.
If unsure if a fix is appropriate for the stable branch, project-specific stable maintenance team members should seek stable maintenance core team members opinion.
Existing core reviewers are greatly encouraged to join the stable maintenance teams in order to help with reviewing backports, judging their appropriateness for the stable branch and approving them.
Fixes for embargoed security issues receive special treatment. See the chapter on vulnerability management for more information.
OpenStack development typically has 3 branches active at any point of time, master (the current development release), stable (the most recent release) and oldstable (previous release). There can from time to time exist older branches but a discussion around that is beyond the scope of this guide.
In order to accept a change into
$release it must first be accepted into all releases back to master.
For the sake of discussion assume a hypothetical development milestones:
master) will be the Uniform release.
Anyone can propose a cherry-pick to the stable-maint team.
One way is that if a bug in launchpad looks like a good candidate for backporting - e.g. if it's a significant bug with the previous release - then just nominating the bug for a stable series (either stable or oldstable) will bring it to the attention of the maintainers e.g. Nova Kilo nominations
If you don't have the appropriate permissions to nominate the bug, then tagging it with e.g. $release-backport-potential is also sufficient e.g. Nova Liberty potential
The best way to get the patch merged in a timely manner is to send it backported by yourself. To do so, you may try to use the "Cherry Pick To" button in the Gerrit UI for the original patch in master. Gerrit will take care of creating a new review, modifying the commit message to include 'cherry-picked from ...' line etc.
The backport must match the master commit, unless there is a serious need to differ e.g gate failure, test framework changed in master, code refactoring or some other reason. If you get a suggestion to enhance your backport in some way that would be contrary to this intent, the reviewer should be referred to the warning above <stable-modifications>.
For code that touches code from oslo-incubator, special backporting rules apply. More details in Oslo policies
You can use git-review to propose a change to the hypothetical stable branch with:
cherry-pick -x option includes 'cherry-picked from ...' line in the commit message which is required to avoid Gerrit bug
Failing all that, just ping one of the team and mention that you think the bug/commit is a good candidate.
If the patch you're proposing will not cherry-pick cleanly, you can help by resolving the conflicts yourself and proposing the resulting patch. Please keep "Conflicts" lines in the commit message to help reviewers, for example: https://review.opendev.org/686292/
If a cherry-picked patch's commit message contains "Conflicts" lines that are not valid anymore in the target branch, then remove those lines.
When cherry-picking a commit, keep the original
Change-Id and gerrit will show a separate review for the stable branch while still allowing you to use the Change-Id to see all the reviews associated with it. See this change as an example.
Change-Id line must be in the last paragraph. Conflicts in the backport add a new paragraph, creating a new
Change-Id but you can avoid that by moving conflicts above the paragraph with
Change-Id line or removing empty lines to make a single paragraph.
If you want to be notified of new stable patches you can create a watch on the gerrit watched projects screen with the following settings.
Project Name: All-Projects Only If: branch:stable/liberty
Then check the "Email Notifications - New Changes" checkbox. That will cause gerrit to send an email whenever a matching change is proposed, and better yet, the change shows up in your 'watched changes' list in gerrit.
Bugs tagged with $release-backport-potential are bugs which apply to a stable release and may be suitable for backporting once fixed. Once the backport has been proposed, the tag should be removed.
Gerrit tags bugs with in-stable-$release when they are merged into the stable branch. The release manager later removes the tag when the bug is targeted to the appropriate series.
Keeping the stable branches in good health in an ongoing effort. To see what bugs are currently causing gate failures and preventing code from merging into stable branches, please see the stable tracker etherpad, where we will track current bugs and in-flight fixes.
Scheduled test runs occur daily for each project's stable branch. If failures crop up, the bot will email the openstack-stable-maint mailing list. It is best to react quickly to these and get them resolved ASAP to prevent them from piling up. Please subscribe if you're interested in helping out.