openstack/governance
Code Issues Proposed changes

Project structure reform specification

Browse Source 
Add a resolution to describe the rationale and proposed implementation
for the OpenStack project structure reform.

Change-Id: I11ab187b9cb32d26da5378236e451d1f46721543
This commit is contained in:
Thierry Carrez 2014-12-02 20:48:39 +01:00
parent f79bb7c7b7
commit 185f85f7c8
1 changed files with 298 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

298
resolutions/20141202-project-structure-reform-spec.rst Normal file
View File

@ -0,0 +1,298 @@
=============================================================
2014-12-02 OpenStack project structure reform specification
=============================================================
Problem description
===================
Our project structure is currently organized as a ladder. Developers form
teams, work on a project, then apply for incubation and ultimately graduate
to be part of the OpenStack integrated release. Only integrated projects
(and the few horizontal efforts necessary to build them) are recognized
officially as "OpenStack" efforts. This creates a number of issues, which
were particularly visible at the Technical Committee level over the Juno
development cycle.
First, the integrated release as it stands today is not a useful product for
our users. The current collection of services in the integrated release spans
from cloud native APIs (swift, zaqar in incubation), base-level IaaS blocks
(nova, glance, cinder), high-level aaS (savana, trove), and lots of things
that span domains. Some projects (swift, ironic...) can be used quite well
outside of the rest of the OpenStack stack, while others (glance, nova)
really don't function in a different context. Skilled operators aren't
deploying "the integrated release": they are picking and choosing between
components they feel are useful. New users, however, are presented with a
complex and scary "integrated release" as the thing they have to deploy and
manage: this inhibits adoption, and this inhibits the adoption of a slice of
OpenStack that could serve their need.
Second, the integrated release being the only and ultimate goal for projects,
there is no lack of candidates, and the list is always-growing. Why reject
Sahara if you accepted Trove? However, processes and services are applied
equally to all members of the integrated release: we gate everything in the
integrated release against everything else, we do a common, time-based
release every 6 months, we produce documentation for all the integrated
release components, etc. The resources working on those integrated horizontal
tasks are very finite, and complexity grows non-linearly as we add more
projects. So there is outside pressure to add more projects, and internal
pressure to resist further additions.
Third, the binary nature of the integrated release results in projects
outside the integrated release failing to get the recognition they deserve.
"Non-official" projects are second- or third-class citizens which can't get
development resources. Alternative solutions can't emerge in the shadow of
the blessed approach. Becoming part of the integrated release, which was
originally designed to be a technical decision, quickly became a
life-or-death question for new projects, and a political/community minefield.
In summary, the "integrated release" is paradoxically too large to be
effectively integrated, installed or upgraded in one piece, and too small to
express the diversity of our rich ecosystem. Its slow-moving, binary nature
is too granular to represent the complexity of what our community produces,
and therefore we need to reform it.
The challenge is to find a solution which allows to address all those three
issues. Embrace the diversity of our ecosystem while making sure that what
we produce is easily understandable and consumable by our downstream users
(distributions, deployers, end users), all that without putting more stress
on the already overworked horizontal teams providing services to all
OpenStack projects, and without limiting the current teams access to common
finite resources.
Proposed change
===============
Provide a precise taxonomy to help navigating the ecosystem
-----------------------------------------------------------
We can't add any more "OpenStack" projects without dramatically revisiting
the information we provide. It is the duty of the Technical Committee to
help downstream consumers of OpenStack understand what each project means
to them, and provide them with accurate statuses for those projects.
Currently the landscape is very simple: you're in the integrated release, or
you're not. But since there was only one category (or badge of honor), it
ended up meaning different things to different people. From a release
management perspective, it meant what we released on the same day. From a
CI perspective, it meant what was co-gated. From an OpenStack distribution
perspective, it meant what you should be packaging. From some operator
perspective, it meant the base set of projects you should be deploying. From
some other operator perspective, it meant the set of mature, stable projects.
Those are all different things, and yet we used a single category to describe
it.
The first part of the change is to create a framework of tags to describe
more accurately and more objectively what each project produced in the
OpenStack community means. The Technical Committee will define tags and the
objective rules to apply them. This framework will allow us to progressively
replace the "integrated release" single badge with a richer and more nuanced
description of all "OpenStack" projects. It will allow the Technical
Committee to provide more precise answers to the Foundation Board of
Directors questions about which set of projects may make sense for a given
trademark license program. It will allow our downstream users to know which
projects are mature, which are security-supported, which are used in more
than one public cloud, or which are really massively scalable.
Recognize all our community is a part of OpenStack
--------------------------------------------------
The second part of the change is recognizing that there is more to
"OpenStack" than a finite set of projects blessed by the Technical
Committee. We already have plenty of projects that are developed on
OpenStack infrastructure, follow the OpenStack way of doing things, have
development discussions on the openstack-dev mailing-list and use
#openstack-meeting channels for their team meetings. Those are part of
the OpenStack community as well, and we propose that those should considered
"OpenStack projects" (and be hosted under openstack git namespaces), as
long as they meet an objective criteria for inclusion (to be developed as one
of the work items below). This might include items such as:
* They align with the OpenStack Mission: the project should help further the
OpenStack mission, by providing a cloud infrastructure service, or
directly building on an existing OpenStack infrastructure service
* They follow the OpenStack way: open source (licensing), open community
(leadership chosen by the contributors to the project), open development
(public reviews on Gerrit, core reviewers, gate, assigned liaisons), and
open design (direction discussed at Design Summit and/or on public forums)
* They ensure basic interoperability (API services should support at least
Keystone)
* They submit to the OpenStack Technical Committee oversight
These criteria are objective, and therefore the Technical Committee may
delegate processing applications to another team. However, the TC would
still vote to approve or reject applications itself, based on the
recommendations and input of any delegates, but without being bound to
that advice. The TC may also decide to encourage collaboration between
similar projects (to reduce unnecessary duplication of effort), or to
remove dead projects.
This proposed structure will replace the current program-driven structure.
We'll still track which team owns which git repository, but this will let
multiple different "OpenStack" teams potentially address the same problem
space. Contributors to projects in the OpenStack git namespace will all be
considered ATCs and participate in electing the Technical Committee.
Transition
----------
As for all significant governance changes, we need to ensure a seamless
transition and reduce the effect of the reform on the current development
cycle. To ensure this seamless transition, the OpenStack taxonomy will
initially define one tag, "integrated-release", which will contain the
integrated projects for the Kilo cycle. To minimize disruption, this tag
will be used throughout the Kilo development cycle and for the Kilo end
release. This tag may be split, replaced or redefined in the future, but
that will be discussed as separate changes.
Future evolution
----------------
These proposed changes are just the base foundational step, enabling the
future evolution of our project structure and governance. It puts in place
the framework that will be used to discuss future changes (new tags and the
rules to apply them).
Implementation
==============
Assignee(s)
-----------
The work on this transition is assigned to the Technical Committee members,
under the coordination of the Chair of the Technical Committee.
Work Items
----------
* Communication about the changes and their impact to the wider OpenStack
community (end of December, start of January, ttx)
* Create project taxonomy base structure and templates in governance
repository (mid-January, ttx)
* Replace incubation-integration-requirements.rst by rules definition for
the "integrated-release" transitional tag (end of January, assignee tbd)
* Create base taxonomy navigation website, to make the taxonomy easily
discoverable, searchable and browseable (kilo-2 milestone, jaypipes)
* Define new objective OpenStack project requirements (to replace old
new-programs-requirements.rst) (kilo-2 milestone, assignee tbd)
* Update Technical Committee charter to get rid of the "Programs" concept
(and redefine ATC as contributors to any OpenStack project) (kilo-2
milestone, ttx)
Most of those work items will result in governance changes that will be
discussed, reviewed and approved by the Technical Committee separately.
Impact
======
Impact for horizontal teams
---------------------------
Horizontal teams (documentation, infrastructure, QA, release management,
stable maintenance, vulnerability management, translators...) have set a
number of expectations toward the projects in the "integrated release".
This is what created tension as the Technical Committee added more projects
which those horizontal teams had to support. Those expectations have to be
revisited as we replace the "integrated release" with a richer landscape.
With this proposed change, the work of horizontal teams shall gradually move
away from centrally handling work for all projects, to a more decentralized
model where they provide processes and tools to empower projects to do the
work themselves. The horizontal teams become responsible for passing along
the knowledge, tools and processes to the projects in order to produce
quality artifacts, rather than being the direct producers of those artifacts.
Each horizontal team will have to redefine how they organize their work
under the new project structure, and which (if any) projects they still
directly handle. They will be able to define tags to communicate that new
organization. Note that most teams already started that transition as more
projects were being added to the integrated release, so this will help them
to more explicitly describe the service they provide.
Impact for existing integrated projects and the Kilo cycle
----------------------------------------------------------
This change in itself doesn't adversely impact existing integrated projects:
they will continue to exist and be defined under the transitional
"integrated-release" tag. However, one end goal of the reform is to
deconstruct the "integrated release" binary concept and replace it with
more precise and objective groupings, so there should come a time in the
future where the "integrated-release" concept won't mean anything anymore,
and the transitional tag will be discontinued. This change puts in place the
framework that will allow us to do that, but doesn't actually do anything
yet. In particular, the "integrated release" as a concept will still very
much exist at least until the end of the Kilo development cycle.
Impact for currently-incubated projects
---------------------------------------
Currently-incubated projects would directly become "OpenStack projects"
under the new structure, without needing another formal application. Future
tags will be defined and applied to them to further describe their nature
and maturity status.
Trademark checks
----------------
The OpenStack Foundation legal staff currently performs trademark checks as
a project is incubated, before its inclusion in the integrated release. It
will continue to apply the same preventive analysis to any project that will
be used as part of OpenStack Foundation trademark license programs. However
projects under the openstack git namespaces are considered projects from the
OpenStack Community, and won't all be preventively checked for potential
trademark conflicts. To communicate that, a note will be posted on the
relevant git organization pages stating that the OpenStack Foundation is not
responsible for the project names or content below, which are posted by
independent developers.
The Technical Committee however expects that if a reasonable challenge is
presented to a given project under an openstack git namespace, a rename of
the project has to be considered.
References
==========
* Original mailing-list discussion:
http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html
* Blogposts:
* "OpenStack as Layers"
https://dague.net/2014/08/26/openstack-as-layers/ (Sean Dague)
* "OpenStack as Layers but also a Big Tents but also a bunch of Cats"
http://inaugust.com/post/108 (Monty Taylor)
* "The problem space in the big tent"
http://ttx.re/problem-space-in-the-big-tent.html (Thierry Carrez)
* "So, What is the Core of OpenStack?"
http://www.joinfu.com/2014/09/so-what-is-the-core-of-openstack/ (Jay Pipes)
* "On Layers"
http://www.stillhq.com/openstack/kilo/000002.html (Mikal Still)
* Strawman governance change proposals:
* Doug's strawman v1:
https://review.openstack.org/#/q/status:open+topic:big-tent,n,z
* Doug's strawman v2:
https://review.openstack.org/#/c/131422/
* Jay's strawman:
https://review.openstack.org/#/c/126582/
* Public notes from discussions between TC members:
https://etherpad.openstack.org/p/project-restructure-hangouts