openstack/project-team-guide
Code Issues Proposed changes

stable: document the 'proactive' backporting workflow

Browse Source 
Neutron team follows the process for a while (since Liberty release). I
hear more projects are interested in adopting the process. Let's
document it in project team guide so that different teams stay
consistent in how they approach it, both tools as well as process wise.

Change-Id: I7d427d56e64d57af675faeff221fe4148867e6e9
This commit is contained in:
Ihar Hrachyshka 2016-05-27 19:00:42 +02:00
parent e27bfc3ce5
commit c5da030c60
1 changed files with 179 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

179
doc/source/stable-branches.rst
View File

@ -324,6 +324,183 @@ crop up, the bot will email the `openstack-stable-maint mailing list`_. It is
best to react quickly to these and get them resolved ASAP to prevent them from
piling up. Please subscribe if you're interested in helping out.
Proactive backports
===================
To make sure suitable bug fixes that land in *master* branches are delivered to
stable branch consumers in timely manner, and to avoid situations when a high
impact bug fix falls through the cracks and does not get quickly provisioned to
users, projects may adopt a 'proactive' approach towards tracking patches that
are candidates for backports, as described below.
.. note::
The first project that adopted the described approach is Neutron. Other
projects are welcome to experiment with similar practices and provide
feedback and improvements.
.. note::
Tools mentioned in the guidelines below are currently maintained in
`openstack-infra/release-tools repository`_. Most of them are implemented as
Unix filters that can be interconnected into a pipeline to accommodate for
specific project needs and practices.
.. note::
Guidelines below assume that there is a group of people behind the effort
that are willing to help. Tips on how to build the subteam are out of scope
for the document.
From high level perspective, proactive backporting process consists of the following steps:
#. identify bugs fixed since the previous triage event;
#. of those, pick only those bugs that does not break stable policy policies;
#. distribute identified backport candidates among subteam members;
#. subteam members consider each candidate bug for inclusion into *stable* and
*oldstable* branches; if applicable, backports are proposed for review and
tracked until inclusion into appropriate branches;
#. new stable releases are created in due time.
.. note::
Most of those steps require human intervension (with the prominent
exception of the first step) because triaging requires specific judgement.
New release proposals can be automated, but at the moment, this is left out
of scope for this document.
This is ongoing process, and it's usually executed on weekly basis, or with
other frequency that fits better the subteam and the project in question.
Now, let's cover each step with more details.
Identify new bug fixes
----------------------
The process assumes that the subteam keeps track of the last git hash that was
validated somewhere. For the initial candidate list generation, it's advised to
start on a branch boundary (the latest common git commit between *stable* and
*master* branches).
For every new git commit found in *master* branch, commit message is checked
for bug tags (Closes-Bug, Partial-Bug, Related-Bug, ...) All bugs mentioned are
considered for initial filtering.
For this exact need, use the following release tool:
.. code-block:: bash
$ ./bugs-fixed-since.py --repo ../neutron --start 1ce8ce9546479c0ce6055c0c205a8885699e3051
1514424
1560464
1546110
...
Filter out features and enhancements
------------------------------------
Due to stable policy described above, new features and enhancements are
generally not allowed in stable branches. For example, to filter out bugs that
have importance set to Wishlist in Launchpad, you can use the following tool:
.. code-block:: bash
$ ./bugs-fixed-since.py [...] | lp-filter-bugs-by-importance.py neutron --importance Wishlist
1514424
1560464
1546110
...
The resulting list is expected to contain only actual bug fixes.
In case you also want to filter out bugs of Low importance, append another call to the tool:
.. code-block:: bash
$ [...] | lp-filter-bugs-by-importance.py neutron --importance Low
1514424
1560464
1546110
...
Once you are satisfied with the query result, you should remember the latest
commit checked, and also store the bug list somewhere.
To achieve the latter, multiple directions can be taken.
#. One way is to store it in some external tool like Etherpad. If this
direction is chosen, the following tool may become handy to make the list
more consumable:
.. code-block:: bash
$ [...] | ./annotate-lp-bugs.py neutron
https://bugs.launchpad.net/bugs/1514424 "neutron metadata ns proxy does not support ssl" (Medium,Fix Released) [in-stable-mitaka,l3-ipam-dhcp] [vuntz]
https://bugs.launchpad.net/bugs/1560464 "ovsdb monitor doesn't return ofport" (High,Fix Released) [in-stable-liberty,in-stable-mitaka] [kevinbenton]
https://bugs.launchpad.net/bugs/1546110 "DB error causes router rescheduling loop to fail" (Medium,Fix Released) [in-stable-kilo,in-stable-liberty,in-stable-mitaka,l3-ipam-dhcp,liberty-backport-potential] [brian-haley]
...
#. Another alternative is to tag backport candidates in Launchpad. For that,
it's advised to avoid using *$release-backport-potential* tags, and instead
introduce a new tag per project team (f.e. *neutron-proactive-backport-potential*
for Neutron). This is to avoid conflicts in the tag usage by multiple teams
running independent backporting processes when bug fixes spanning multiple
projects are considered.
.. code-block:: bash
$ [...] | ./lp-tag.py neutron-proactive-backport-potential
This command will tag all identified backport candidates with the project
specific tag. For example, check the `neutron proactive-backport-potential
dashboard`_.
With that, you get access to all filtering features available in Launchpad.
Distribute the work
-------------------
Once you have a list of candidate bug fixes to consider for backporting, it's
time to distribute it among subteam members. Depending on which method is
chosen above to track candidate bug fixes, you may utilize Launchpad search
queries, or other filtering technique to identify bugs of specific topics of
interest, to distribute the work to folks who are experts in those topics.
.. note::
Exact search queries and filters are project specific and largely depend on
existing bug tracking practices adopted by projects. Hence they are out of
scope for the document.
Candidate triage
----------------
Each candidate bug should be assessed on its applicability to *stable* and
*oldstable* branches, as per corresponding support phase definitions. For
example, for *stable* branch, all bug fixes of user value can be considered to
backport; while for *oldstable* branch, only critical bugs are allowed to be
backported.
For every applicable stable branch, a backport is proposed in Gerrit. The
backporter is expected to follow the progress of the backport to make sure it's
not lost in reviews.
Once all applicable backports are proposed to Gerrit and are on their path
towards stable inclusion, *<project>-proactive-backport-potential* tag can be
removed from the bug.
.. note::
If possible, consider keeping the order of backports in a way that would
reduce the number of git conflicts.
Release often
-------------
Proactive backporting process is expected to trigger higher volume of changes
in stable branches. To make releases more granular, it's advised participating
projects create new stable releases often. It may be done on a bi-weekly basis,
or any other schedule that fits better the project and its actual backports
volume.
.. _Nova Kilo nominations: https://bugs.launchpad.net/nova/kilo/+nominations
.. _Nova Liberty potential: https://bugs.launchpad.net/nova/+bugs?field.tag=liberty-backport-potential
.. _Oslo policies: http://specs.openstack.org/openstack/oslo-specs/specs/policy/incubator.html#stable-branches
@ -337,3 +514,5 @@ piling up. Please subscribe if you're interested in helping out.
.. _stable maintenance core team: https://review.openstack.org/#/admin/groups/530,members
.. _openstack-infra/project-config repo: http://git.openstack.org/cgit/openstack-infra/project-config/
.. _on Designate: https://review.openstack.org/#/c/292617/
.. _openstack-infra/release-tools repository: http://git.openstack.org/cgit/openstack-infra/release-tools/
.. _neutron proactive-backport-potential dashboard: https://bugs.launchpad.net/neutron/+bugs?field.searchtext=&orderby=-importance&search=Search&field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=FIXRELEASED&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=neutron-proactive-backport-potential&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on