![Alexis Lee](/assets/img/avatar_default.png)
Basic conversion done with pandoc then I had to fix some links and the title underlines. The Mentoring page has been merged into the (mostly empty) "How to get (more) involved with Nova" page. Change-Id: Ib4a11b21c0fd6aaf86e643897193f59cefb626c4
1009 lines
40 KiB
ReStructuredText
1009 lines
40 KiB
ReStructuredText
..
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may
|
|
not use this file except in compliance with the License. You may obtain
|
|
a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
|
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
|
License for the specific language governing permissions and limitations
|
|
under the License.
|
|
|
|
.. _process:
|
|
|
|
=================
|
|
Nova team process
|
|
=================
|
|
|
|
Nova is always evolving its processes, but it's important to explain why we
|
|
have them: so that we can all work to ensure the interactions we need to
|
|
happen do happen. The process we have should always be there to make good
|
|
communication between all members of our community easier.
|
|
|
|
OpenStack Wide Patterns
|
|
=======================
|
|
|
|
Nova follows most of the generally adopted norms for OpenStack projects.
|
|
You can get more details here:
|
|
|
|
* http://docs.openstack.org/infra/manual/developers.html
|
|
* http://git.openstack.org/cgit/openstack/project-team-guide
|
|
|
|
If you are new to Nova, please read this first: :ref:`getting_involved`.
|
|
|
|
Dates overview
|
|
==============
|
|
|
|
For Mitaka, please see:
|
|
https://wiki.openstack.org/wiki/Nova/Mitaka_Release_Schedule
|
|
|
|
For Liberty, please see:
|
|
https://wiki.openstack.org/wiki/Nova/Liberty_Release_Schedule
|
|
|
|
Feature Freeze
|
|
~~~~~~~~~~~~~~
|
|
|
|
This effort is primarily to help the horizontal teams help prepare their
|
|
items for release, while at the same time giving developers time to
|
|
focus on stabilising what is currently in master, and encouraging users
|
|
and packages to perform tests (automated, and manual) on the release, to
|
|
spot any major bugs.
|
|
|
|
As such we have the following processes:
|
|
|
|
- https://wiki.openstack.org/wiki/FeatureProposalFreeze
|
|
|
|
- make sure all code is up for review
|
|
- so we can optimise for completed features, not lots of half
|
|
completed features
|
|
|
|
- https://wiki.openstack.org/wiki/FeatureFreeze
|
|
|
|
- make sure all feature code is merged
|
|
|
|
- https://wiki.openstack.org/wiki/StringFreeze
|
|
|
|
- give translators time to translate all our strings
|
|
- Note: debug logs are no longer translated
|
|
|
|
- https://wiki.openstack.org/wiki/DepFreeze
|
|
|
|
- time to co-ordinate the final list of deps, and give packagers
|
|
time to package them
|
|
- generally it is also quite destabilising to take upgrades (beyond
|
|
bug fixes) this late
|
|
|
|
We align with this in Nova and the dates for this release are stated
|
|
above.
|
|
|
|
As with all processes here, there are exceptions. But the exceptions at
|
|
this stage need to be discussed with the horizontal teams that might be
|
|
affected by changes beyond this point, and as such are discussed with
|
|
one of the OpenStack release managers.
|
|
|
|
Spec and Blueprint Approval Freeze
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
This is a (mostly) Nova specific process.
|
|
|
|
Why do we have a Spec Freeze:
|
|
|
|
- specs take a long time to review, keeping it open distracts from code
|
|
reviews
|
|
- keeping them "open" and being slow at reviewing the specs (or just
|
|
ignoring them) really annoys the spec submitters
|
|
- we generally have more code submitted that we can review, this time
|
|
bounding is a way to limit the number of submissions
|
|
|
|
By the freeze date, we expect this to also be the complete list of
|
|
approved blueprints for liberty:
|
|
https://blueprints.launchpad.net/nova/liberty
|
|
|
|
The date listed above is when we expect all specifications for Liberty
|
|
to be merged and displayed here:
|
|
http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/
|
|
|
|
New in Liberty, we will keep the backlog open for submission at all
|
|
times. Note: the focus is on accepting and agreeing problem statements
|
|
as being in scope, rather than queueing up work items for the next
|
|
release. We are still working on a new lightweight process to get our of
|
|
the backlog and approved for a particular release. For more details on
|
|
backlog specs, please see:
|
|
http://specs.openstack.org/openstack/nova-specs/specs/backlog/index.html
|
|
|
|
Also new in Liberty, we will allow people to submit Mitaka specs from
|
|
liberty-2 (rather than liberty-3 as normal).
|
|
|
|
There can be exceptions, usually it's an urgent feature request that
|
|
comes up after the initial deadline. These will generally be discussed
|
|
at the weekly Nova meeting, by adding the spec or blueprint to discuss
|
|
in the appropriate place in the meeting agenda here (ideally make
|
|
yourself available to discuss the blueprint, or alternatively make your
|
|
case on the ML before the meeting):
|
|
https://wiki.openstack.org/wiki/Meetings/Nova
|
|
|
|
Non-priority Feature Freeze
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
This is a Nova specific process.
|
|
|
|
This only applies to low priority blueprints in this list:
|
|
https://blueprints.launchpad.net/nova/liberty
|
|
|
|
We currently have a very finite amount of review bandwidth. In order to
|
|
make code review time for the agreed community wide priorities, we have
|
|
to not do some other things. To this end, we are reserving liberty-3 for
|
|
priority features and bug fixes. As such, we intend not to merge any
|
|
non-priority things during liberty-3, so around liberty-2 is the
|
|
"Feature Freeze" for blueprints that are not a priority for liberty.
|
|
|
|
For liberty, we are not aligning the Non-priority Feature Freeze with
|
|
the tagging of liberty-2. That means the liberty-2 tag will not include
|
|
some features that merge later in the week. This means, we only require
|
|
the code to be approved before the end of July 30th, we don't require it
|
|
to be merged by that date. This should help stop any gate issues
|
|
disrupting our ability to merge all the code that we have managed to get
|
|
reviewed in time. Ideally all code should be merged by the end of July
|
|
31st, but the state of the gate will determine how possible that is.
|
|
|
|
You can see the list of priorities for each release:
|
|
http://specs.openstack.org/openstack/nova-specs/#priorities
|
|
|
|
For things that are very close to merging, it's possible it might get an
|
|
exception for one week after the freeze date, given the patches get
|
|
enough +2s from the core team to get the code merged. But we expect this
|
|
list to be zero, if everything goes to plan (no massive gate failures,
|
|
etc). For details, process see:
|
|
http://lists.openstack.org/pipermail/openstack-dev/2015-July/070920.html
|
|
|
|
Exception process:
|
|
|
|
- Please add request in here:
|
|
https://etherpad.openstack.org/p/liberty-nova-non-priority-feature-freeze
|
|
(ideally with core reviewers to sponsor your patch, normally the
|
|
folks who have already viewed those patches)
|
|
- make sure you make your request before the end of Wednesday 5th
|
|
August
|
|
- nova-drivers will meet to decide what gets an exception (just like
|
|
they did last release:
|
|
http://lists.openstack.org/pipermail/openstack-dev/2015-February/056208.html)
|
|
- an initial list of exceptions (probably just a PTL compiled list at
|
|
that point) will be available for discussion during the Nova meeting
|
|
on Thursday 6th August
|
|
- the aim is to merge the code for all exceptions by the end of Monday
|
|
10th August
|
|
|
|
Alternatives:
|
|
|
|
- It was hoped to make this a continuous process using "slots" to
|
|
control what gets reviewed, but this was rejected by the community
|
|
when it was last discussed. There is hope this can be resurrected to
|
|
avoid the "lumpy" nature of this process.
|
|
- Currently the runways/kanban ideas are blocked on us adopting
|
|
something like phabricator that could support such workflows
|
|
|
|
String Freeze
|
|
~~~~~~~~~~~~~
|
|
|
|
NOTE: this is still a provisional idea
|
|
|
|
There are general guidelines here:
|
|
https://wiki.openstack.org/wiki/StringFreeze
|
|
|
|
But below is an experiment for Nova during liberty, to trial a new
|
|
process. There are four views onto this process.
|
|
|
|
First, the user point of view:
|
|
|
|
- Would like to see untranslated strings, rather than hiding
|
|
error/info/warn log messages as debug
|
|
|
|
Second, the translators:
|
|
|
|
- Translators will start translation without string freeze, just after
|
|
feature freeze.
|
|
- Then we have a strict string freeze date (around RC1 date)
|
|
- After at least 10 days to finish up the translations before the final
|
|
release
|
|
|
|
Third, the docs team:
|
|
|
|
- Config string updates often mean there is a DocImpact and docs need
|
|
updating
|
|
- best to avoid those during feature freeze, where possible
|
|
|
|
Fourth, the developer point of view:
|
|
|
|
- Add any translated strings before Feature Freeze
|
|
- Post Feature Freeze, allow string changes where an untranslated
|
|
string is better than no string
|
|
|
|
- i.e. allow new log message strings, until the hard freeze
|
|
|
|
- Post Feature Freeze, have a soft string freeze, try not to change
|
|
existing strings, where possible
|
|
|
|
- Note: moving a string and re-using a existing string is fine, as
|
|
the tooling deals with that automatically
|
|
|
|
- Post Hard String Freeze, there should be no extra strings to
|
|
translate
|
|
|
|
- Assume any added strings will not be translated
|
|
- Send email about the string freeze exception in this case only,
|
|
but there should be zero of these
|
|
|
|
So, what has changed from https://wiki.openstack.org/wiki/StringFreeze,
|
|
well:
|
|
|
|
- no need to block new strings until much later in the cycle
|
|
|
|
- should stop the need to rework bug fixes to remove useful log
|
|
messages
|
|
|
|
- instead, just accept the idea of untranslated strings being better
|
|
than no strings in those cases
|
|
|
|
So for Liberty, 21st September, so we will call 21st September the hard
|
|
freeze date, as we expect RC1 to be cut sometime after 21st September.
|
|
Note the date is fixed, it's not aligned with the cutting of RC1.
|
|
|
|
This means we must cut another tarball (RC2 or higher) at some point
|
|
after 5th October to include new translations, even if there are no more
|
|
bug fixes, to give time before the final release on 13th-16th October.
|
|
|
|
How do I get my code merged?
|
|
============================
|
|
|
|
OK, so you are new to Nova, and you have been given a feature to
|
|
implement. How do I make that happen?
|
|
|
|
You can get most of your questions answered here:
|
|
|
|
- http://docs.openstack.org/infra/manual/developers.html
|
|
|
|
But let's put a Nova specific twist on things...
|
|
|
|
Overview
|
|
~~~~~~~~
|
|
|
|
.. image:: ./images/Nova_spec_process.svg
|
|
:alt: Flow chart showing the Nova bug/feature process
|
|
|
|
Where do you track bugs?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
We track bugs here:
|
|
|
|
- http://bugs.launchpad.net/nova
|
|
|
|
If you fix an issue, please raise a bug so others who spot that issue
|
|
can find the fix you kindly created for them.
|
|
|
|
Also before submitting your patch it's worth checking to see if someone
|
|
has already fixed it for you (Launchpad helps you with that, at little,
|
|
when you create the bug report).
|
|
|
|
When do I need a blueprint vs a spec?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
For more details see:
|
|
|
|
- http://docs.openstack.org/developer/nova/devref/kilo.blueprints.html#when-is-a-blueprint-needed
|
|
|
|
To understand this question, we need to understand why blueprints and
|
|
specs are useful.
|
|
|
|
But here is the rough idea:
|
|
|
|
- if it needs a spec, it will need a blueprint.
|
|
- if it's an API change, it needs a spec.
|
|
- if it's a single small patch that touches a small amount of code,
|
|
with limited deployer and doc impact, it probably doesn't need a
|
|
spec.
|
|
|
|
If you are unsure, please ask johnthetubaguy on IRC, or one of the other
|
|
nova-drivers.
|
|
|
|
How do I get my blueprint approved?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
So you need your blueprint approved? Here is how:
|
|
|
|
- if you don't need a spec, please add a link to your blueprint to the
|
|
agenda for the next nova meeting:
|
|
https://wiki.openstack.org/wiki/Meetings/Nova
|
|
|
|
- be sure your blueprint description has enough context for the
|
|
review in that meeting.
|
|
- As of Mitaka, this list is stored in an etherpad:
|
|
https://etherpad.openstack.org/p/mitaka-nova-spec-review-tracking
|
|
|
|
- if you need a spec, then please submit a nova-spec for review, see:
|
|
http://docs.openstack.org/infra/manual/developers.html
|
|
|
|
Got any more questions? Contact johnthetubaguy or one of the other
|
|
nova-specs-core who are awake at the same time as you. IRC is best as
|
|
you will often get an immediate response, if they are too busy send
|
|
him/her an email.
|
|
|
|
How do I get a procedural -2 removed from my patch?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
When feature freeze hits, any patches for blueprints that are still in review
|
|
get a procedural -2 to stop them merging. In Nova a blueprint is only approved
|
|
for a single release. To have the -2 removed, you need to get the blueprint
|
|
approved for the current release (see `How do I get my blueprint approved?`_).
|
|
|
|
Why are the reviewers being mean to me?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Code reviews take intense concentration and a lot of time. This tends to
|
|
lead to terse responses with very little preamble or nicety. That said,
|
|
there's no excuse for being actively rude or mean. OpenStack has a Code
|
|
of Conduct (https://www.openstack.org/legal/community-code-of-conduct/)
|
|
and if you feel this has been breached please raise the matter
|
|
privately. Either with the relevant parties, the PTL or failing those,
|
|
the OpenStack Foundation.
|
|
|
|
That said, there are many objective reasons for applying a -1 or -2 to a
|
|
patch:
|
|
|
|
- Firstly and simply, patches must address their intended purpose
|
|
successfully.
|
|
- Patches must not have negative side-effects like wiping the database
|
|
or causing a functional regression. Usually removing anything,
|
|
however tiny, requires a deprecation warning be issued for a cycle.
|
|
- Code must be maintainable, that is it must adhere to coding standards
|
|
and be as readable as possible for an average OpenStack developer
|
|
(acknowledging this person is ill-defined).
|
|
- Patches must respect the direction of the project, for example they
|
|
should not make approved specs substantially more difficult to
|
|
implement.
|
|
- Release coordinators need the correct process to be followed so scope
|
|
can be tracked accurately. Bug fixes require bugs, features require
|
|
blueprints and all but the simplest features require specs. If there
|
|
is a blueprint, it must be approved for the release/milestone the
|
|
patch is attempting to merge into.
|
|
|
|
Please particularly bear in mind that a -2 does not mean "never ever"
|
|
nor does it mean "your idea is bad and you are dumb". It simply means
|
|
"do not merge today". You may need to wait some time, rethink your
|
|
approach or even revisit the problem definition but there is almost
|
|
always some way forward. The core who applied the -2 should tell you
|
|
what you need to do.
|
|
|
|
My code review seems stuck, what can I do?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
First and foremost - address any -1s and -2s! The review load on Nova is
|
|
high enough that patches with negative reviews often get filtered out
|
|
entirely. A few tips:
|
|
|
|
- Be precise. Ensure you're not talking at cross purposes.
|
|
- Try to understand where the reviewer is coming from. They may have a
|
|
very different perspective and/or use-case to you.
|
|
- If you don't understand the problem, ask them to explain - this is
|
|
common and helpful behaviour.
|
|
- Be positive. Everyone's patches have issues, including core
|
|
reviewers. No-one cares once the issues are fixed.
|
|
- Try not to flip-flop. When two reviewers are pulling you in different
|
|
directions, stop pushing code and negotiate the best way forward.
|
|
- If the reviewer does not respond to replies left on the patchset,
|
|
reach out to them on IRC or email. If they still don't respond, you
|
|
can try to ask their colleagues if they're on holiday (or simply
|
|
wait). Finally, you can ask for mediation in the Nova meeting by
|
|
adding it to the agenda
|
|
(https://wiki.openstack.org/wiki/Meetings/Nova). This is also what
|
|
you should do if you are unable to negotiate a resolution to an
|
|
issue.
|
|
|
|
Secondly, Nova is a big project, be aware of the average wait times:
|
|
http://russellbryant.net/openstack-stats/nova-openreviews.html
|
|
|
|
Eventually you should get some +1s from people working through the
|
|
review queue. Expect to get -1s as well. You can ask for reviews within
|
|
your company, 1-2 are useful (not more), especially if those reviewers
|
|
are known to give good reviews. You can spend some time while you wait
|
|
reviewing other people's code - they may reciprocate and you may learn
|
|
something (:ref:`Why do code reviews when I'm not core? <why_plus1>`).
|
|
|
|
If you've waited an appropriate amount of time and you haven't had any
|
|
+1s, you can ask on IRC for reviews. Please don't ask for core review
|
|
straight away, especially not directly (IRC or email). Core reviewer
|
|
time is very valuable and gaining some +1s is a good way to show your
|
|
patch meets basic quality standards.
|
|
|
|
Once you have a few +1s, be patient. Remember the average wait times.
|
|
You can ask for reviews each week in IRC, it helps to ask when cores are
|
|
awake.
|
|
|
|
Bugs
|
|
^^^^
|
|
|
|
It helps to apply correct tracking information.
|
|
|
|
- Put "Closes-Bug", "Partial-Bug" or "Related-Bug" in the commit
|
|
message tags as necessary.
|
|
- If you have to raise a bug in Launchpad first, do it - this helps
|
|
someone else find your fix.
|
|
- Make sure the bug has the correct priority and tag set:
|
|
https://wiki.openstack.org/wiki/Nova/BugTriage#Step_2:_Triage_Tagged_Bugs
|
|
- If it's a trivial fix (<100 lines as a rule of thumb), add it to:
|
|
https://etherpad.openstack.org/p/liberty-nova-priorities-tracking
|
|
|
|
Features
|
|
^^^^^^^^
|
|
|
|
Again, it helps to apply correct tracking information. For
|
|
blueprint-only features:
|
|
|
|
- Put your blueprint in the commit message, EG "blueprint
|
|
simple-feature".
|
|
- Mark the blueprint as NeedsCodeReview if you are finished.
|
|
- Maintain the whiteboard on the blueprint so it's easy to understand
|
|
which patches need reviews.
|
|
- Use a single topic for all related patches. All patches for one
|
|
blueprint should share a topic.
|
|
|
|
For blueprint and spec features, do everything for blueprint-only
|
|
features and also:
|
|
|
|
- If it's a project or subteam priority, add it to:
|
|
https://etherpad.openstack.org/p/liberty-nova-priorities-tracking
|
|
- Ensure your spec is approved for the current release cycle.
|
|
|
|
If your code is a project or subteam priority, the cores interested in
|
|
that priority might not mind a ping after it has sat with +1s for a
|
|
week. If you abuse this privilege, you'll lose respect.
|
|
|
|
If it's not a priority, your blueprint/spec has been approved for the
|
|
cycle and you have been patient, you can raise it during the Nova
|
|
meeting. The outcome may be that your spec gets unapproved for the
|
|
cycle, so that priority items can take focus. If this happens to you,
|
|
sorry - it should not have been approved in the first place, Nova team
|
|
bit off more than they could chew, it is their mistake not yours. You
|
|
can re-propose it for the next cycle.
|
|
|
|
If it's not a priority and your spec has not been approved, your code
|
|
will not merge this cycle. Please re-propose your spec for the next
|
|
cycle.
|
|
|
|
Nova Process Mission
|
|
====================
|
|
|
|
This section takes a high level look at the guiding principles behind
|
|
the Nova process.
|
|
|
|
Open
|
|
~~~~
|
|
|
|
Our mission is to have:
|
|
|
|
- Open Source
|
|
- Open Design
|
|
- Open Development
|
|
- Open Community
|
|
|
|
We have to work out how to keep communication open in all areas. We need
|
|
to be welcoming and mentor new people, and make it easy for them to
|
|
pickup the knowledge they need to get involved with OpenStack. For more
|
|
info on Open, please see: https://wiki.openstack.org/wiki/Open
|
|
|
|
Interoperable API, supporting a vibrant ecosystem
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
An interoperable API that gives users on-demand access to compute
|
|
resources is at the heart of Nova's mission:
|
|
http://docs.openstack.org/developer/nova/project_scope.html#mission
|
|
|
|
Nova has a vibrant ecosystem of tools built on top of the current Nova
|
|
API. All features should be designed to work with all technology
|
|
combinations, so the feature can be adopted by our ecosystem. If a new
|
|
feature is not adopted by the ecosystem, it will make it hard for your
|
|
users to make use of those features, defeating most of the reason to add
|
|
the feature in the first place. The microversion system allows users to
|
|
isolate themselves
|
|
|
|
This is a very different aim to being "pluggable" or wanting to expose
|
|
all capabilities to end users. At the same time, it is not just a
|
|
"lowest common denominator" set of APIs. It should be discoverable which
|
|
features are available, and while no implementation details should leak
|
|
to the end users, purely admin concepts may need to understand
|
|
technology specific details that back the interoperable and more
|
|
abstract concepts that are exposed to the end user. This is a hard goal,
|
|
and one area we currently don't do well is isolating image creators from
|
|
these technology specific details.
|
|
|
|
Smooth Upgrades
|
|
~~~~~~~~~~~~~~~
|
|
|
|
As part of our mission for a vibrant ecosystem around our APIs, we want
|
|
to make it easy for those deploying Nova to upgrade with minimal impact
|
|
to their users. Here is the scope of Nova's upgrade support:
|
|
|
|
- upgrade from any commit, to any future commit, within the same major
|
|
release
|
|
- only support upgrades between N and N+1 major versions, to reduce
|
|
technical debt relating to upgrades
|
|
|
|
Here are some of the things we require developers to do, to help with
|
|
upgrades:
|
|
|
|
- when replacing an existing feature or configuration option, make it
|
|
clear how to transition to any replacement
|
|
- deprecate configuration options and features before removing them
|
|
|
|
- i.e. continue to support and test features for at least one
|
|
release before they are removed
|
|
- this gives time for operator feedback on any removals
|
|
|
|
- End User API will always be kept backwards compatible
|
|
|
|
Interaction goals
|
|
~~~~~~~~~~~~~~~~~
|
|
|
|
When thinking about the importance of process, we should take a look at:
|
|
http://agilemanifesto.org
|
|
|
|
With that in mind, let's look at how we want different members of the
|
|
community to interact. Let's start with looking at issues we have tried
|
|
to resolve in the past (currently in no particular order). We must:
|
|
|
|
- have a way for everyone to review blueprints and designs, including
|
|
allowing for input from operators and all types of users (keep it
|
|
open)
|
|
- take care to not expand Nova's scope any more than absolutely
|
|
necessary
|
|
- ensure we get sufficient focus on the core of Nova so that we can
|
|
maintain or improve the stability and flexibility of the overall
|
|
codebase
|
|
- support any API we release approximately for ever. We currently
|
|
release every commit, so we're motivate to get the API right first
|
|
time
|
|
- avoid low priority blueprints slowing work on high priority work,
|
|
without blocking those forever
|
|
- focus on a consistent experience for our users, rather than ease of
|
|
development
|
|
- optimise for completed blueprints, rather than more half completed
|
|
blueprints, so we get maximum value for our users out of our review
|
|
bandwidth
|
|
- focus efforts on a subset of patches to allow our core reviewers to
|
|
be more productive
|
|
- set realistic expectations on what can be reviewed in a particular
|
|
cycle, to avoid sitting in an expensive rebase loop
|
|
- be aware of users that do not work on the project full time
|
|
- be aware of users that are only able to work on the project at
|
|
certain times that may not align with the overall community cadence
|
|
- discuss designs for non-trivial work before implementing it, to avoid
|
|
the expense of late-breaking design issues
|
|
|
|
FAQs
|
|
====
|
|
|
|
Why bother with all this process?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
We are a large community, spread across multiple timezones, working with
|
|
several horizontal teams. Good communication is a challenge and the
|
|
processes we have are mostly there to try and help fix some
|
|
communication challenges.
|
|
|
|
If you have a problem with a process, please engage with the community,
|
|
discover the reasons behind our current process, and help fix the issues
|
|
you are experiencing.
|
|
|
|
Why don't you remove old process?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
We do! For example, in Liberty we stopped trying to predict the
|
|
milestones when a feature will land.
|
|
|
|
As we evolve, it is important to unlearn new habits and explore if
|
|
things get better if we choose to optimise for a different set of
|
|
issues.
|
|
|
|
Why are specs useful?
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Spec reviews allow anyone to step up and contribute to reviews, just
|
|
like with code. Before we used gerrit, it was a very messy review
|
|
process, that felt very "closed" to most people involved in that
|
|
process.
|
|
|
|
As Nova has grown in size, it can be hard to work out how to modify Nova
|
|
to meet your needs. Specs are a great way of having that discussion with
|
|
the wider Nova community.
|
|
|
|
For Nova to be a success, we need to ensure we don't break our existing
|
|
users. The spec template helps focus the mind on the impact your change
|
|
might have on existing users and gives an opportunity to discuss the
|
|
best way to deal with those issues.
|
|
|
|
However, there are some pitfalls with the process. Here are some top
|
|
tips to avoid them:
|
|
|
|
- keep it simple. Shorter, simpler, more decomposed specs are quicker
|
|
to review and merge much quicker (just like code patches).
|
|
- specs can help with documentation but they are only intended to
|
|
document the design discussion rather than document the final code.
|
|
- don't add details that are best reviewed in code, it's better to
|
|
leave those things for the code review.
|
|
|
|
If we have specs, why still have blueprints?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
We use specs to record the design agreement, we use blueprints to track
|
|
progress on the implementation of the spec.
|
|
|
|
Currently, in Nova, specs are only approved for one release, and must be
|
|
re-submitted for each release you want to merge the spec, although that
|
|
is currently under review.
|
|
|
|
Why do we have priorities?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
To be clear, there is no "nova dev team manager", we are an open team of
|
|
professional software developers, that all work for a variety of (mostly
|
|
competing) companies that collaborate to ensure the Nova project is a
|
|
success.
|
|
|
|
Over time, a lot of technical debt has accumulated, because there was a
|
|
lack of collective ownership to solve those cross-cutting concerns.
|
|
Before the Kilo release, it was noted that progress felt much slower,
|
|
because we were unable to get appropriate attention on the architectural
|
|
evolution of Nova. This was important, partly for major concerns like
|
|
upgrades and stability. We agreed it's something we all care about and
|
|
it needs to be given priority to ensure that these things get fixed.
|
|
|
|
Since Kilo, priorities have been discussed at the summit. This turns in
|
|
to a spec review which eventually means we get a list of priorities
|
|
here: http://specs.openstack.org/openstack/nova-specs/#priorities
|
|
|
|
Allocating our finite review bandwidth to these efforts means we have to
|
|
limit the reviews we do on non-priority items. This is mostly why we now
|
|
have the non-priority Feature Freeze. For more on this, see below.
|
|
|
|
Blocking a priority effort is one of the few widely acceptable reasons
|
|
to block someone adding a feature. One of the great advantages of being
|
|
more explicit about that relationship is that people can step up to help
|
|
review and/or implement the work that is needed to unblock the feature
|
|
they want to get landed. This is a key part of being an Open community.
|
|
|
|
Why is there a Feature Freeze (and String Freeze) in Nova?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The main reason Nova has a feature freeze is that it gives people
|
|
working on docs and translations to sync up with the latest code.
|
|
Traditionally this happens at the same time across multiple projects, so
|
|
the docs are synced between what used to be called the "integrated
|
|
release".
|
|
|
|
We also use this time period as an excuse to focus our development
|
|
efforts on bug fixes, ideally lower risk bug fixes, and improving test
|
|
coverage.
|
|
|
|
In theory, with a waterfall hat on, this would be a time for testing and
|
|
stabilisation of the product. In Nova we have a much stronger focus on
|
|
keeping every commit stable, by making use of extensive continuous
|
|
testing. In reality, we frequently see the biggest influx of fixes in
|
|
the few weeks after the release, as distributions do final testing of
|
|
the released code.
|
|
|
|
It is hoped that the work on Feature Classification will lead us to
|
|
better understand the levels of testing of different Nova features, so
|
|
we will be able to reduce and dependency between Feature Freeze and
|
|
regression testing. It is also likely that the move away from
|
|
"integrated" releases will help find a more developer friendly approach
|
|
to keep the docs and translations in sync.
|
|
|
|
Why is there a non-priority Feature Freeze in Nova?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
We have already discussed why we have priority features.
|
|
|
|
The rate at which code can be merged to Nova is primarily constrained by
|
|
the amount of time able to be spent reviewing code. Given this,
|
|
earmarking review time for priority items means depriving it from
|
|
non-priority items.
|
|
|
|
The simplest way to make space for the priority features is to stop
|
|
reviewing and merging non-priority features for a whole milestone. The
|
|
idea being developers should focus on bug fixes and priority features
|
|
during that milestone, rather than working on non-priority features.
|
|
|
|
A known limitation of this approach is developer frustration. Many
|
|
developers are not being given permission to review code, work on bug
|
|
fixes or work on priority features, and so feel very unproductive
|
|
upstream. An alternative approach of "slots" or "runways" has been
|
|
considered, that uses a kanban style approach to regulate the influx of
|
|
work onto the review queue. We are yet to get agreement on a more
|
|
balanced approach, so the existing system is being continued to ensure
|
|
priority items are more likely to get the attention they require.
|
|
|
|
Why do you still use Launchpad?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
We are actively looking for an alternative to Launchpad's bugs and
|
|
blueprints.
|
|
|
|
Originally the idea was to create Storyboard. However the development
|
|
has stalled. A more likely front runner is this:
|
|
http://phabricator.org/applications/projects/
|
|
|
|
When should I submit my spec?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Ideally we want to get all specs for a release merged before the summit.
|
|
For things that we can't get agreement on, we can then discuss those at
|
|
the summit. There will always be ideas that come up at the summit and
|
|
need to be finalised after the summit. This causes a rush which is best
|
|
avoided.
|
|
|
|
How can I get my code merged faster?
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
So no-one is coming to review your code, how do you speed up that
|
|
process?
|
|
|
|
Firstly, make sure you are following the above process. If it's a
|
|
feature, make sure you have an approved blueprint. If it's a bug, make
|
|
sure it is triaged, has its priority set correctly, it has the correct
|
|
bug tag and is marked as in progress. If the blueprint has all the code
|
|
up for review, change it from Started into NeedsCodeReview so people
|
|
know only reviews are blocking you, make sure it hasn't accidentally got
|
|
marked as implemented.
|
|
|
|
Secondly, if you have a negative review (-1 or -2) and you responded to
|
|
that in a comment or uploading a new change with some updates, but that
|
|
reviewer hasn't come back for over a week, it's probably a good time to
|
|
reach out to the reviewer on IRC (or via email) to see if they could
|
|
look again now you have addressed their comments. If you can't get
|
|
agreement, and your review gets stuck (i.e. requires mediation), you can
|
|
raise your patch during the Nova meeting and we will try to resolve any
|
|
disagreement.
|
|
|
|
Thirdly, is it in merge conflict with master or are any of the CI tests
|
|
failing? Particularly any third-party CI tests that are relevant to the
|
|
code you are changing. If you're fixing something that only occasionally
|
|
failed before, maybe recheck a few times to prove the tests stay
|
|
passing. Without green tests, reviews tend to move on and look at the
|
|
other patches that have the tests passing.
|
|
|
|
OK, so you have followed all the process (i.e. your patches are getting
|
|
advertised via the project's tracking mechanisms), and your patches
|
|
either have no reviews, or only positive reviews. Now what?
|
|
|
|
Have you considered reviewing other people's patches? Firstly,
|
|
participating in the review process is the best way for you to
|
|
understand what reviewers are wanting to see in the code you are
|
|
submitting. As you get more practiced at reviewing it will help you to
|
|
write "merge-ready" code. Secondly, if you help review other peoples
|
|
code and help get their patches ready for the core reviewers to add a
|
|
+2, it will free up a lot of non-core and core reviewer time, so they
|
|
are more likely to get time to review your code. For more details,
|
|
please see: :ref:`Why do code reviews when I'm not core? <why_plus1>`
|
|
|
|
Please note, I am not recommending you go to ask people on IRC or via
|
|
email for reviews. Please try to get your code reviewed using the above
|
|
process first. In many cases multiple direct pings generate frustration
|
|
on both sides and that tends to be counter productive.
|
|
|
|
Now you have got your code merged, lets make sure you don't need to fix
|
|
this bug again. The fact the bug exists means there is a gap in our
|
|
testing. Your patch should have included some good unit tests to stop
|
|
the bug coming back. But don't stop there, maybe its time to add tempest
|
|
tests, to make sure your use case keeps working? Maybe you need to set
|
|
up a third party CI so your combination of drivers will keep working?
|
|
Getting that extra testing in place should stop a whole heap of bugs,
|
|
again giving reviewers more time to get to the issues or features you
|
|
want to add in the future.
|
|
|
|
Process Evolution Ideas
|
|
=======================
|
|
|
|
We are always evolving our process as we try to improve and adapt to the
|
|
changing shape of the community. Here we discuss some of the ideas,
|
|
along with their pros and cons.
|
|
|
|
Splitting out the virt drivers (or other bits of code)
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Currently, Nova doesn't have strong enough interfaces to split out the
|
|
virt drivers, scheduler or REST API. This is seen as the key blocker.
|
|
Let's look at both sides of the debate here.
|
|
|
|
Reasons for the split:
|
|
|
|
- can have separate core teams for each repo
|
|
|
|
- this leads to quicker turn around times, largely due to focused
|
|
teams
|
|
|
|
- splitting out things from core means less knowledge required to
|
|
become core in a specific area
|
|
|
|
Reasons against the split:
|
|
|
|
- loss of interoperability between drivers
|
|
|
|
- this is a core part of Nova's mission, to have a single API across
|
|
all deployments, and a strong ecosystem of tools and apps built on
|
|
that
|
|
- we can overcome some of this with stronger interfaces and
|
|
functional tests
|
|
|
|
- new features often need changes in the API and virt driver anyway
|
|
|
|
- the new "depends-on" can make these cross-repo dependencies easier
|
|
|
|
- loss of code style consistency across the code base
|
|
- fear of fragmenting the nova community, leaving few to work on the
|
|
core of the project
|
|
- could work in subteams within the main tree
|
|
|
|
TODO - need to complete analysis
|
|
|
|
Subteam recommendation as a +2
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
There are groups of people with great knowledge of particular bits of
|
|
the code base. It may be a good idea to give their recommendation of a
|
|
merge. In addition, having the subteam focus review efforts on a subset
|
|
of patches should help concentrate the nova-core reviews they get, and
|
|
increase the velocity of getting code merged.
|
|
|
|
The first part is for subgroups to show they can do a great job of
|
|
recommending patches. This is starting in here:
|
|
https://etherpad.openstack.org/p/liberty-nova-priorities-tracking
|
|
|
|
Ideally this would be done with gerrit user "tags" rather than an
|
|
etherpad. There are some investigations by sdague in how feasible it
|
|
would be to add tags to gerrit.
|
|
|
|
Stop having to submit a spec for each release
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
As mentioned above, we use blueprints for tracking, and specs to record
|
|
design decisions. Targeting specs to a specific release is a heavyweight
|
|
solution and blurs the lines between specs and blueprints. At the same
|
|
time, we don't want to lose the opportunity to revise existing
|
|
blueprints. Maybe there is a better balance?
|
|
|
|
What about this kind of process:
|
|
|
|
- backlog has these folders:
|
|
|
|
- backlog/incomplete - merge a partial spec
|
|
- backlog/complete - merge complete specs (remove tracking details,
|
|
such as assignee part of the template)
|
|
- ?? backlog/expired - specs are moved here from incomplete or
|
|
complete when no longer seem to be given attention (after 1 year,
|
|
by default)
|
|
- /implemented - when a spec is complete it gets moved into the
|
|
release directory and possibly updated to reflect what actually
|
|
happened
|
|
- there will no longer be a per-release approved spec list
|
|
|
|
To get your blueprint approved:
|
|
|
|
- add it to the next nova meeting
|
|
|
|
- if a spec is required, update the URL to point to the spec merged
|
|
in a spec to the blueprint
|
|
- ensure there is an assignee in the blueprint
|
|
|
|
- a day before the meeting, a note is sent to the ML to review the list
|
|
before the meeting
|
|
- discuss any final objections in the nova-meeting
|
|
|
|
- this may result in a request to refine the spec, if things have
|
|
changed since it was merged
|
|
|
|
- trivial cases can be approved in advance by a nova-driver, so not all
|
|
folks need to go through the meeting
|
|
|
|
This still needs more thought, but should decouple the spec review from
|
|
the release process. It is also more compatible with a runway style
|
|
system, that might be less focused on milestones.
|
|
|
|
Runways
|
|
~~~~~~~
|
|
|
|
Runways are a form of Kanban, where we look at optimising the flow
|
|
through the system, by ensure we focus our efforts on reviewing a
|
|
specific subset of patches.
|
|
|
|
The idea goes something like this:
|
|
|
|
- define some states, such as: design backlog, design review, code
|
|
backlog, code review, test+doc backlog, complete
|
|
- blueprints must be in one of the above state
|
|
|
|
- large or high priority bugs may also occupy a code review slot
|
|
|
|
- core reviewer member moves item between the slots
|
|
|
|
- must not violate the rules on the number of items in each state
|
|
- states have a limited number of slots, to ensure focus
|
|
- certain percentage of slots are dedicated to priorities, depending
|
|
on point in the cycle, and the type of the cycle, etc
|
|
|
|
Reasons for:
|
|
|
|
- more focused review effort, get more things merged more quickly
|
|
- more upfront about when your code is likely to get reviewed
|
|
- smooth out current "lumpy" non-priority feature freeze system
|
|
|
|
Reasons against:
|
|
|
|
- feels like more process overhead
|
|
- control is too centralised
|
|
|
|
Replacing Milestones with SemVer Releases
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
You can deploy any commit of Nova and upgrade to a later commit in that
|
|
same release. Making our milestones versioned more like an official
|
|
release would help signal to our users that people can use the
|
|
milestones in production, and get a level of upgrade support.
|
|
|
|
It could go something like this:
|
|
|
|
- 14.0.0 is milestone 1
|
|
- 14.0.1 is milestone 2 (maybe, because we add features, it should be
|
|
14.1.0?)
|
|
- 14.0.2 is milestone 3
|
|
- we might do other releases (once a critical bug is fixed?), as it
|
|
makes sense, but we will always be the time bound ones
|
|
- 14.0.3 two weeks after milestone 3, adds only bug fixes (and updates
|
|
to RPC versions?)
|
|
|
|
- maybe a stable branch is created at this point?
|
|
|
|
- 14.1.0 adds updated translations and co-ordinated docs
|
|
|
|
- this is released from the stable branch?
|
|
|
|
- 15.0.0 is the next milestone, in the following cycle
|
|
|
|
- not the bump of the major version to signal an upgrade
|
|
incompatibility with 13.x
|
|
|
|
We are currently watching Ironic to see how their use of semver goes,
|
|
and see what lessons need to be learnt before we look to maybe apply
|
|
this technique during M.
|
|
|
|
Feature Classification
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
This is a look at moving forward this effort:
|
|
|
|
- http://docs.openstack.org/developer/nova/support-matrix.html
|
|
|
|
The things we need to cover:
|
|
|
|
- note what is tested, and how often that test passes (via 3rd party
|
|
CI, or otherwise)
|
|
|
|
- link to current test results for stable and master (time since
|
|
last pass, recent pass rate, etc)
|
|
- TODO - sync with jogo on his third party CI audit and getting
|
|
trends, ask infra
|
|
|
|
- include experimental features (untested feature)
|
|
- get better at the impact of volume drivers and network drivers on
|
|
available features (not just hypervisor drivers)
|
|
|
|
Main benefits:
|
|
|
|
- users get a clear picture of what is known to work
|
|
- be clear about when experimental features are removed, if no tests
|
|
are added
|
|
- allows a way to add experimental things into Nova, and track either
|
|
their removal or maturation
|
|
|
|
* https://wiki.openstack.org/wiki/Nova/Mitaka_Release_Schedule
|