diff --git a/resolutions/20141202-project-structure-reform-spec.rst b/resolutions/20141202-project-structure-reform-spec.rst new file mode 100644 index 000000000..5af077f51 --- /dev/null +++ b/resolutions/20141202-project-structure-reform-spec.rst @@ -0,0 +1,298 @@ +============================================================= + 2014-12-02 OpenStack project structure reform specification +============================================================= + +Problem description +=================== + +Our project structure is currently organized as a ladder. Developers form +teams, work on a project, then apply for incubation and ultimately graduate +to be part of the OpenStack integrated release. Only integrated projects +(and the few horizontal efforts necessary to build them) are recognized +officially as "OpenStack" efforts. This creates a number of issues, which +were particularly visible at the Technical Committee level over the Juno +development cycle. + +First, the integrated release as it stands today is not a useful product for +our users. The current collection of services in the integrated release spans +from cloud native APIs (swift, zaqar in incubation), base-level IaaS blocks +(nova, glance, cinder), high-level aaS (savana, trove), and lots of things +that span domains. Some projects (swift, ironic...) can be used quite well +outside of the rest of the OpenStack stack, while others (glance, nova) +really don't function in a different context. Skilled operators aren't +deploying "the integrated release": they are picking and choosing between +components they feel are useful. New users, however, are presented with a +complex and scary "integrated release" as the thing they have to deploy and +manage: this inhibits adoption, and this inhibits the adoption of a slice of +OpenStack that could serve their need. + +Second, the integrated release being the only and ultimate goal for projects, +there is no lack of candidates, and the list is always-growing. Why reject +Sahara if you accepted Trove? However, processes and services are applied +equally to all members of the integrated release: we gate everything in the +integrated release against everything else, we do a common, time-based +release every 6 months, we produce documentation for all the integrated +release components, etc. The resources working on those integrated horizontal +tasks are very finite, and complexity grows non-linearly as we add more +projects. So there is outside pressure to add more projects, and internal +pressure to resist further additions. + +Third, the binary nature of the integrated release results in projects +outside the integrated release failing to get the recognition they deserve. +"Non-official" projects are second- or third-class citizens which can't get +development resources. Alternative solutions can't emerge in the shadow of +the blessed approach. Becoming part of the integrated release, which was +originally designed to be a technical decision, quickly became a +life-or-death question for new projects, and a political/community minefield. + +In summary, the "integrated release" is paradoxically too large to be +effectively integrated, installed or upgraded in one piece, and too small to +express the diversity of our rich ecosystem. Its slow-moving, binary nature +is too granular to represent the complexity of what our community produces, +and therefore we need to reform it. + +The challenge is to find a solution which allows to address all those three +issues. Embrace the diversity of our ecosystem while making sure that what +we produce is easily understandable and consumable by our downstream users +(distributions, deployers, end users), all that without putting more stress +on the already overworked horizontal teams providing services to all +OpenStack projects, and without limiting the current teams access to common +finite resources. + + +Proposed change +=============== + +Provide a precise taxonomy to help navigating the ecosystem +----------------------------------------------------------- + +We can't add any more "OpenStack" projects without dramatically revisiting +the information we provide. It is the duty of the Technical Committee to +help downstream consumers of OpenStack understand what each project means +to them, and provide them with accurate statuses for those projects. + +Currently the landscape is very simple: you're in the integrated release, or +you're not. But since there was only one category (or badge of honor), it +ended up meaning different things to different people. From a release +management perspective, it meant what we released on the same day. From a +CI perspective, it meant what was co-gated. From an OpenStack distribution +perspective, it meant what you should be packaging. From some operator +perspective, it meant the base set of projects you should be deploying. From +some other operator perspective, it meant the set of mature, stable projects. +Those are all different things, and yet we used a single category to describe +it. + +The first part of the change is to create a framework of tags to describe +more accurately and more objectively what each project produced in the +OpenStack community means. The Technical Committee will define tags and the +objective rules to apply them. This framework will allow us to progressively +replace the "integrated release" single badge with a richer and more nuanced +description of all "OpenStack" projects. It will allow the Technical +Committee to provide more precise answers to the Foundation Board of +Directors questions about which set of projects may make sense for a given +trademark license program. It will allow our downstream users to know which +projects are mature, which are security-supported, which are used in more +than one public cloud, or which are really massively scalable. + +Recognize all our community is a part of OpenStack +-------------------------------------------------- + +The second part of the change is recognizing that there is more to +"OpenStack" than a finite set of projects blessed by the Technical +Committee. We already have plenty of projects that are developed on +OpenStack infrastructure, follow the OpenStack way of doing things, have +development discussions on the openstack-dev mailing-list and use +#openstack-meeting channels for their team meetings. Those are part of +the OpenStack community as well, and we propose that those should considered +"OpenStack projects" (and be hosted under openstack git namespaces), as +long as they meet an objective criteria for inclusion (to be developed as one +of the work items below). This might include items such as: + +* They align with the OpenStack Mission: the project should help further the + OpenStack mission, by providing a cloud infrastructure service, or + directly building on an existing OpenStack infrastructure service + +* They follow the OpenStack way: open source (licensing), open community + (leadership chosen by the contributors to the project), open development + (public reviews on Gerrit, core reviewers, gate, assigned liaisons), and + open design (direction discussed at Design Summit and/or on public forums) + +* They ensure basic interoperability (API services should support at least + Keystone) + +* They submit to the OpenStack Technical Committee oversight + +These criteria are objective, and therefore the Technical Committee may +delegate processing applications to another team. However, the TC would +still vote to approve or reject applications itself, based on the +recommendations and input of any delegates, but without being bound to +that advice. The TC may also decide to encourage collaboration between +similar projects (to reduce unnecessary duplication of effort), or to +remove dead projects. + +This proposed structure will replace the current program-driven structure. +We'll still track which team owns which git repository, but this will let +multiple different "OpenStack" teams potentially address the same problem +space. Contributors to projects in the OpenStack git namespace will all be +considered ATCs and participate in electing the Technical Committee. + +Transition +---------- + +As for all significant governance changes, we need to ensure a seamless +transition and reduce the effect of the reform on the current development +cycle. To ensure this seamless transition, the OpenStack taxonomy will +initially define one tag, "integrated-release", which will contain the +integrated projects for the Kilo cycle. To minimize disruption, this tag +will be used throughout the Kilo development cycle and for the Kilo end +release. This tag may be split, replaced or redefined in the future, but +that will be discussed as separate changes. + +Future evolution +---------------- + +These proposed changes are just the base foundational step, enabling the +future evolution of our project structure and governance. It puts in place +the framework that will be used to discuss future changes (new tags and the +rules to apply them). + + +Implementation +============== + +Assignee(s) +----------- + +The work on this transition is assigned to the Technical Committee members, +under the coordination of the Chair of the Technical Committee. + +Work Items +---------- + +* Communication about the changes and their impact to the wider OpenStack + community (end of December, start of January, ttx) + +* Create project taxonomy base structure and templates in governance + repository (mid-January, ttx) + +* Replace incubation-integration-requirements.rst by rules definition for + the "integrated-release" transitional tag (end of January, assignee tbd) + +* Create base taxonomy navigation website, to make the taxonomy easily + discoverable, searchable and browseable (kilo-2 milestone, jaypipes) + +* Define new objective OpenStack project requirements (to replace old + new-programs-requirements.rst) (kilo-2 milestone, assignee tbd) + +* Update Technical Committee charter to get rid of the "Programs" concept + (and redefine ATC as contributors to any OpenStack project) (kilo-2 + milestone, ttx) + +Most of those work items will result in governance changes that will be +discussed, reviewed and approved by the Technical Committee separately. + + +Impact +====== + +Impact for horizontal teams +--------------------------- + +Horizontal teams (documentation, infrastructure, QA, release management, +stable maintenance, vulnerability management, translators...) have set a +number of expectations toward the projects in the "integrated release". +This is what created tension as the Technical Committee added more projects +which those horizontal teams had to support. Those expectations have to be +revisited as we replace the "integrated release" with a richer landscape. + +With this proposed change, the work of horizontal teams shall gradually move +away from centrally handling work for all projects, to a more decentralized +model where they provide processes and tools to empower projects to do the +work themselves. The horizontal teams become responsible for passing along +the knowledge, tools and processes to the projects in order to produce +quality artifacts, rather than being the direct producers of those artifacts. + +Each horizontal team will have to redefine how they organize their work +under the new project structure, and which (if any) projects they still +directly handle. They will be able to define tags to communicate that new +organization. Note that most teams already started that transition as more +projects were being added to the integrated release, so this will help them +to more explicitly describe the service they provide. + +Impact for existing integrated projects and the Kilo cycle +---------------------------------------------------------- + +This change in itself doesn't adversely impact existing integrated projects: +they will continue to exist and be defined under the transitional +"integrated-release" tag. However, one end goal of the reform is to +deconstruct the "integrated release" binary concept and replace it with +more precise and objective groupings, so there should come a time in the +future where the "integrated-release" concept won't mean anything anymore, +and the transitional tag will be discontinued. This change puts in place the +framework that will allow us to do that, but doesn't actually do anything +yet. In particular, the "integrated release" as a concept will still very +much exist at least until the end of the Kilo development cycle. + +Impact for currently-incubated projects +--------------------------------------- + +Currently-incubated projects would directly become "OpenStack projects" +under the new structure, without needing another formal application. Future +tags will be defined and applied to them to further describe their nature +and maturity status. + +Trademark checks +---------------- + +The OpenStack Foundation legal staff currently performs trademark checks as +a project is incubated, before its inclusion in the integrated release. It +will continue to apply the same preventive analysis to any project that will +be used as part of OpenStack Foundation trademark license programs. However +projects under the openstack git namespaces are considered projects from the +OpenStack Community, and won't all be preventively checked for potential +trademark conflicts. To communicate that, a note will be posted on the +relevant git organization pages stating that the OpenStack Foundation is not +responsible for the project names or content below, which are posted by +independent developers. + +The Technical Committee however expects that if a reasonable challenge is +presented to a given project under an openstack git namespace, a rename of +the project has to be considered. + + +References +========== + +* Original mailing-list discussion: + http://lists.openstack.org/pipermail/openstack-dev/2014-August/041929.html + +* Blogposts: + + * "OpenStack as Layers" + https://dague.net/2014/08/26/openstack-as-layers/ (Sean Dague) + + * "OpenStack as Layers but also a Big Tents but also a bunch of Cats" + http://inaugust.com/post/108 (Monty Taylor) + + * "The problem space in the big tent" + http://ttx.re/problem-space-in-the-big-tent.html (Thierry Carrez) + + * "So, What is the Core of OpenStack?" + http://www.joinfu.com/2014/09/so-what-is-the-core-of-openstack/ (Jay Pipes) + + * "On Layers" + http://www.stillhq.com/openstack/kilo/000002.html (Mikal Still) + +* Strawman governance change proposals: + + * Doug's strawman v1: + https://review.openstack.org/#/q/status:open+topic:big-tent,n,z + + * Doug's strawman v2: + https://review.openstack.org/#/c/131422/ + + * Jay's strawman: + https://review.openstack.org/#/c/126582/ + +* Public notes from discussions between TC members: + https://etherpad.openstack.org/p/project-restructure-hangouts