openstack
/
neutron
Code Issues Proposed changes
neutron/doc/source/devref/contribute.rst

551 lines
28 KiB
ReStructuredText
Raw Blame History

Contributing new extensions to Neutron
======================================
Neutron has a pluggable architecture, with a number of extension points.
This documentation covers aspects relevant to contributing new Neutron
v2 core (aka monolithic) plugins, ML2 mechanism drivers, and L3 service
plugins. This document will initially cover a number of process-oriented
aspects of the contribution process, and proceed to provide a how-to guide
that shows how to go from 0 LOC's to successfully contributing new
extensions to Neutron. In the remainder of this guide, we will try to
use practical examples as much as we can so that people have working
solutions they can start from.
This guide is for a developer who wants to have a degree of visibility
within the OpenStack Networking project. If you are a developer who
wants to provide a Neutron-based solution without interacting with the
Neutron community, you are free to do so, but you can stop reading now,
as this guide is not for you.
In fact, from the Kilo release onwards, the Neutron core team propose that
additions to the codebase adopt a structure where the *monolithic plugins*,
*ML2 MechanismDrivers*, and *L3 service plugins* are integration-only
(called "vendor integration" hereinafter) to code that lives outside the
tree (called "vendor library" hereinafter); the same applies for any
vendor-specific agents. The only part that is to stay in the tree is the
agent 'main' (a small python file that imports agent code from the vendor
library and starts it). 'Outside the tree' can be anything that is publicly
available: it may be a stackforge repo for instance, a tarball, a pypi package,
etc. A plugin/drivers maintainer team self-governs in order to promote sharing,
reuse, innovation, and release of the 'out-of-tree' deliverable. It should not
be required for any member of the core team to be involved with this process,
although core members of the Neutron team can participate in whichever capacity
is deemed necessary to facilitate out-of-tree development.
Below, the following strategies will be documented:
* Design and Development;
* Testing and Continuous Integration;
* Defect Management;
* Backport Management for plugin specific code;
* DevStack Integration;
* Documentation;
This document will then provide a working example on how to contribute
new additions to Neutron.
Blueprint Spec Submission Strategy
----------------------------------
Provided contributors adhere to the abovementioned development footprint
they should not be required to follow the spec process for changes that
only affect their vendor integration and library. New contributions can
simply be submitted for code review, with the proviso that adequate
documentation and 3rd CI party is supplied at the time of the code
submission. For tracking purposes, the review itself can be tagged
with a Launchpad bug report. The bug should be marked as wishlist to
avoid complicating tracking of Neutron's primary deliverables. Design
documents can still be supplied in form of RST documents, within the same
vendor library repo. If substantial change to the common Neutron code are
required, a spec that targets common Neutron code will be required, however
every case is different and a contributor is invited to seek guidance from
the Neutron core team as to what steps to follow, and whether a spec or
a bug report is more suited for what a contributor needs to deliver.
Once again, for submitting the integration module to the Neutron codebase,
no spec is required.
Development Strategy
--------------------
* The following elements are suggested to be contributed in the tree
for plugins and drivers (called vendor integration hereinafter):
* Data models
* Extension definitions
* Configuration files
* Requirements file targeting vendor code
* Things that do not remain in the tree (called vendor library hereinafter):
* Vendor specific logic
* Associated unit tests
The idea here would be to provide in-tree the plugin/driver code that
implements an API, but have it delegate to out-of-tree code for
backend-specific interactions. The vendor integration will then typically
involve minor passthrough/parsing of parameters, minor handling of DB objects
as well as handling of responses, whereas the vendor library will do the
heavylifting and implement the vendor-specific logic. The boundary between
the in-tree layer and the out-of-tree one should be defined by the contributor
while asking these types of questions:
* If something changes in my backend, do I need to alter the integration
layer drastically? Clearly, the less impact there is, the better the
separation being achieved.
* If I expose vendor details (e.g. protocols, auth, etc.), can I easily swap
and replace the targeted backend (e.g. hardware with a newer version
being supplied) without affecting the integration too much? Clearly, the
more reusable the integration the better the separation.
As mentioned above, the vendor code *must* be available publicly, and a git
repository makes the most sense. By doing so, the module itself can be made
accessible using a pip requirements file. This file should not be confused
with the Neutron requirements file that lists all common dependencies. Instead
it should be a file 'requirements.txt' that is located in neutron/plugins/pluginXXX/,
whose content is something along the lines of 'my_plugin_xxx_library>=X.Y.Z'.
Vendors are responsible for ensuring that their library does not depend on
libraries conflicting with global requirements, but it could depend on
libraries not included in the global requirements. Just as in Neutron's
main requirements.txt, it will be possible to pin the version of the vendor
library.
For instance, a vendor integration module can become as simple as one that
performs only the following:
* Registering config options
* Registering the plugin class
* Registering the models
* Registering the extensions
Testing Strategy
----------------
The testing process will be as follow:
* No unit tests for the vendor integration of plugins and drivers are deemed
necessary. The expectation is that contributors would run unit test in their
own external library (e.g. in stackforge where Jenkins setup is for free).
For unit tests that validate the vendor library, it is the responsibility of
the vendor to choose what CI system they see fit to run them. There is no
need or requirement to use OpenStack CI resources if they do not want to.
Having said that, it may be useful to provide coverage for the shim layer in
the form of basic validation as done in `ODL <https://github.com/openstack/neutron/blob/master/neutron/tests/unit/ml2/test_mechanism_odl.py>`_ and `LBaaS A10 driver <https://github.com/openstack/neutron-lbaas/blob/master/neutron_lbaas/tests/unit/services/loadbalancer/drivers/a10networks/test_driver_v1.py>`_.
* 3rd Party CI will continue to validate vendor integration with Neutron via
functional testing. 3rd Party CI is a communication mechanism. This objective
of this mechanism is as follows:
* it communicates to plugin/driver contributors when someone has contributed
a change that is potentially breaking. It is then up to a given
contributor maintaining the affected plugin to determine whether the
failure is transient or real, and resolve the problem if it is.
* it communicates to a patch author that they may be breaking a plugin/driver.
If they have the time/energy/relationship with the maintainer of the
plugin/driver in question, then they can (at their discretion) work to
resolve the breakage.
* it communicates to the community at large whether a given plugin/driver
is being actively maintained.
* A maintainer that is perceived to be responsive to failures in their
3rd party CI jobs is likely to generate community goodwill.
It is worth noting that if the vendor library is hosted on StackForge, due to
current openstack-infra limitations, it is not possible to have 3rd party CI systems
participating in the gate pipeline for the StackForge repo. This means that the only
validation provided during the merge process to the StackForge repo is through unit
tests. Post-merge hooks can still be exploited to provide 3rd party CI feedback, and
alert the contributor/reviewer of potential issues. As mentioned above, 3rd party CI
systems will continue to validate Neutron core commits. This will allow them to
detect when incompatible changes occur, whether they are in Neutron or in the vendor
library repo.
Review and Defect Management Strategies
---------------------------------------
The usual process applies to the code that is part of OpenStack Neutron. More
precisely:
* Bugs that affect vendor code can be filed against the Neutron integration,
if the integration code is at fault. Otherwise, the code maintainer may
decide to fix a bug without oversight, and update their requirements file
to target a new version of their vendor library. It makes sense to
require 3rd party CI for a given plugin/driver to pass when changing their
dependency before merging to any branch (i.e. both master and stable branches).
* Vendor specific code should follow the same review guidelines as any other
code in the tree. However, the maintainer has flexibility to choose who
can approve/merge changes in this repo.
Backport Management Strategies
------------------------------
As outlined in the `Spec proposal <http://specs.openstack.org/openstack/neutron-specs/specs/kilo/core-vendor-decomposition.html>`_
all new plugins and drivers will have to follow the contribution model
described here. As for existing plugins and drivers, no in-tree features can
be merged until some progress has been done to make the solution adhere to
this model. That said, there is the question of critical fixes and/or backports
to `stable branches <https://wiki.openstack.org/wiki/StableBranch>`_. The possible
scenarios are:
* The decomposition just completed, we are in the cycle (X) where the decomposition
initiated: in this case, the Neutron master branch no longer have the vendor
library code, but the stable branch still does. Backports via straight
cherry-picks may not be possible, or as easy, therefore a custom backport to
stable could be deemed acceptable to Neutron's stable branches (e.g. stable/X-1
and/or stable/X-2), as required.
* The decomposition is complete, we are in the next cycle where the
decomposition work completed (X+1): backports will be done to the stable branch
available of the vendor library (stable/X), and Neutron's stable branch
(stable/X-1), as outlined in the previous step.
* The decomposition is complete, we are in two or more cycles after the
decomposition work completed (X+2, or later). Backports will be done to the
stable branch(s) available of the vendor library (stable/X, stable/X+1).
* The decomposition is in progress: as long as the vendor code is still in
master, patches will need to go to master before a backport to stable.
Acceptance will be determined on the scope of changes (based on both the
amount of work and severity of the issue). In this case, the plugin or
driver maintainer will need to ensure that the fix gets applied to the
external repo, if necessary (to avoid missing it during the migration process).
* The decomposition has not started: in this case, depending on the issue,
review attention from core members is best effort, and although there is no
explicit rule to prevent them from merging to master, it is in the best interest
of the maintainer to avoid introducing or modifying existing code that will
ultimately be deprecated.
DevStack Integration Strategies
-------------------------------
When developing and testing a new or existing plugin or driver, the aid provided
by DevStack is incredibly valuable: DevStack can help get all the software bits
installed, and configured correctly, and more importantly in a predictable way.
For DevStack integration there are a few options available, and they may or may not
make sense depending on whether you are contributing a new or existing plugin or
driver.
If you are contributing a new plugin, the approach to choose should be based on
`Extras.d Hooks' externally hosted plugins <http://docs.openstack.org/developer/devstack/plugins.html#extras-d-hooks>`_.
With the extra.d hooks, the DevStack integration is colocated with the vendor integration
library, and it leads to the greatest level of flexibility when dealing with DevStack based
dev/test deployments.
Having said that, most Neutron plugins developed in the past likely already have
integration with DevStack in the form of `neutron_plugins <https://github.com/openstack-dev/devstack/tree/master/lib/neutron_plugins>`_.
If the plugin is being decomposed in vendor integration plus vendor library, it would
be necessary to adjust the instructions provided in the neutron_plugin file to pull the
vendor library code as a new dependency. For instance, the instructions below:
::
INSTALL_FROM_REQUIREMENTS=$(trueorfalse True INSTALL_FROM_REQUIREMENTS)
if [[ "$INSTALL_FROM_REQUIREMENTS" == "False" ]]; then
git_clone $NEUTRON_LIB_REPO $NEUTRON_LIB_DIR $NEUTRON_LIB_BRANCH
setup_package $NEUTRON_LIB_DIR
else
# Retrieve the package from the vendor library's requirements.txt
plugin_package=$(cat $NEUTRON_LIB_REQUIREMENTS_FILE)
pip_install "$plugin_package"
fi
could be placed in 'neutron_plugin_configure_service', ahead of the service
configuration. An alternative could be under the `third_party section
<https://github.com/openstack-dev/devstack/tree/master/lib/neutron_thirdparty>`_,
if available. This solution can be similarly exploited for both monolithic
plugins or ML2 mechanism drivers. The configuration of the plugin or driver itself can be
done by leveraging the extensibility mechanisms provided by `local.conf <http://docs.openstack.org/developer/devstack/configuration.html>`_. In fact, since the .ini file for the vendor plugin or driver lives
in the Neutron tree, it is possible to do add the section below to local.conf:
::
[[post-config|$THE_FILE_YOU_NEED_TO_CUSTOMIZE]]
# Override your section config as you see fit
[DEFAULT]
verbose=True
Which in turn it is going to edit the file with the options outlined in the post-config
section.
The above mentioned approach, albeit valid, has the shortcoming of depending on DevStack's
explicit support for the plugin installation and configuration, and the plugin maintainer
is strongly encouraged to revise the existing DevStack integration, in order to evolve it
in an extras.d hooks based approach.
One final consideration is worth making for 3rd party CI setups: if `Devstack Gate
<https://github.com/openstack-infra/devstack-gate>`_ is used, it does provide hook
functions that can be executed at specific times of the devstack-gate-wrap script run.
For example, the `Neutron Functional job <https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/neutron-functional.yaml>`_ uses them. For more details see `devstack-vm-gate-wrap.sh <https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh>`_.
Documentation Strategies
------------------------
It is the duty of the new contributor to provide working links that can be
referenced from the OpenStack upstream documentation.
#TODO(armax): provide more info, when available.
How-to
------
The how-to below assumes that the vendor library will be hosted on StackForge.
Stackforge lets you tap in the entire OpenStack CI infrastructure and can be
a great place to start from to contribute your new or existing driver/plugin.
The list of steps below are somewhat the tl;dr; version of what you can find
on http://docs.openstack.org/infra/manual/creators.html. They are meant to
be the bare minimum you have to complete in order to get you off the ground.
* Create a public repository: this can be a personal github.com repo or any
publicly available git repo, e.g. https://github.com/john-doe/foo.git. This
would be a temporary buffer to be used to feed the StackForge one.
* Initialize the repository: if you are starting afresh, you may *optionally*
want to use cookiecutter to get a skeleton project. You can learn how to use
cookiecutter on https://github.com/openstack-dev/cookiecutter.
If you want to build the repository from an existing Neutron module, you may
want to skip this step now, build the history first (next step), and come back
here to initialize the remainder of the repository with other files being
generated by the cookiecutter (like tox.ini, setup.cfg, setup.py, etc.).
* Building the history: if you are contributing an existing driver/plugin,
you may want to preserve the existing history. If not, you can go to the
next step. To import the history from an existing project this is what
you need to do:
* Clone a copy of the neutron repository to be manipulated.
* Go into the Neutron repo to be changed.
* Execute file split.sh, available in ./tools, and follow instructions.
::
git clone https://github.com/openstack/neutron.git
cd neutron
./tools/split.sh
# Sit and wait for a while, or grab a cup of your favorite drink
At this point you will have the project pruned of everything else but
the files you want to export, with their history. The next steps are:
* Check out stable branches for the project: even though stable branches
are not strictly necessary during the creation of the StackForge repository
(as outlined in the next step below), they do not hurt, and it is
recommended to keep them during the import process.
* Add a remote that points to the repository created before.
* (Optional) If the repository has already being initialized with
cookiecutter, you need to pull first; if not, you can either push
the existing commits/tags or apply and commit further changes to fix
up the structure of repo the way you see fit.
* Finally, push commits and tags to the public repository. If you followed
theses instructions step-by-step, you will have a source repository
that contains both a master and stable branches, as well as tags. Some
of these steps are outlined below:
::
git remote add <foo> https://github.com/john-doe/foo.git
git pull foo master # OPTIONAL, if foo is non-empty
git push --all foo && git push --tags foo
* Create a StackForge repository: for this you need the help of the OpenStack
infra team. It is worth noting that you only get one shot at creating the
StackForge repository. This is the time you get to choose whether you want
to start from a clean slate, or you want to import the repo created during
the previous step. In the latter case, you can do so by specifying the
upstream section for your project in project-config/gerrit/project.yaml.
Steps are documented on the
`Project Creators Manual <http://docs.openstack.org/infra/manual/creators.html>`_.
* Ask for a Launchpad user to be assigned to the core team created. Steps are
documented in
`this section <http://docs.openstack.org/infra/manual/creators.html#update-the-gerrit-group-members>`_.
* Fix, fix, fix: at this point you have an external base to work on. You
can develop against the new stackforge project, the same way you work
with any other OpenStack project: you have pep8, docs, and python27 CI
jobs that validate your patches when posted to Gerrit. For instance, one
thing you would need to do is to define an entry point for your plugin
or driver in your own setup.cfg similarly as to how it is done
`here <https://github.com/stackforge/networking-odl/blob/master/setup.cfg#L31>`_.
* Define an entry point for your plugin or driver in setup.cfg
* Create 3rd Party CI account: if you do not already have one, follow
instructions for
`3rd Party CI <http://ci.openstack.org/third_party.html>`_ to get one.
* TODO(armax): ...
Decomposition progress chart
============================
The following chart captures the following aspects:
* Name: the name of the project that implements a Neutron plugin or driver. The
name is an internal target for links that point to source code, etc.
* Plugins/Drivers: whether the source code contains a core (aka monolithic)
plugin, a set of ML2 drivers, and/or (service) plugins (or extensions) for
firewall, vpn, and load balancers.
* Launchpad: whether the project is managed through Launchpad.
* PyPI: whether the project deliverables are available through PyPI.
* State: a code to represent the current state of the decomposition. Possible
values are:
* [A] External repo available, no code decomposition
* [B] External repo available, partial code decomposition
* [C] External repo available, code decomposition is complete
* [D] Not deemed required. Driver is already bare-bone and decomposition
effort is not considered justified. Assessment may change in the
future
Absense of an entry for an existing plugin or driver means no active effort
has been observed or potentially not required.
* Completed in: the release in which the effort is considered completed. Code
completion can be deemed as such, if there is no overlap/duplication between
what exists in the Neutron tree, and what it exists in the vendor repo.
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| Name | Plugins/Drivers | Launchpad | PyPI | State | Completed in |
+===============================+=======================+===========+==================+=========+==============+
| freescale-nscs_ | ml2,fw | no | no | [D] | |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| networking-arista_ | ml2,l3 | yes | yes | [C] | Kilo |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| networking-bigswitch_ | ml2,core,l3 | no | yes | [C] | Kilo |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| networking-brocade_ | ml2,l3 | yes | yes | [C] | Kilo |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| networking-cisco_ | core,ml2,l3,fw,vpn | yes | yes | [B] | |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| networking-hyperv_ | | | | | |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| networking-metaplugin_ | core | no | no | [C] | Kilo |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| networking-midonet_ | core,lb | yes | yes | [C] | Kilo |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| networking-mlnx_ | ml2 | yes | no | [C] | Kilo |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| networking-nec_ | core | yes | no | [C] | Kilo |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| nuage-openstack-neutron_ | | | | | |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| networking-odl_ | ml2,l3,lb,fw | yes | no | [C] | Kilo |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| networking-ofagent_ | ml2 | yes | yes | [C] | Kilo |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| networking-ovs-dpdk_ | ml2 | yes | no | [C] | Kilo |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| networking-plumgrid_ | core | yes | yes | [C] | Kilo |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| networking-vsphere_ | | | | | |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| pluribus_ | | | | | |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| vmware-nsx_ | core | yes | yes | [C] | Kilo |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
.. _networking-arista:
Arista
------
* Git: https://github.com/stackforge/networking-arista
* Launchpad: https://launchpad.net/networking-arista
* Pypi: https://pypi.python.org/pypi/networking-arista
.. _networking-bigswitch:
Big Switch Networks
-------------------
* Git: https://git.openstack.org/stackforge/networking-bigswitch
* Pypi: https://pypi.python.org/pypi/bsnstacklib
.. _networking-brocade:
* Git: https://github.com/stackforge/networking-brocade
* Launchpad: https://launchpad.net/networking-brocade
* PyPI: https://pypi.python.org/pypi/networking-brocade
.. _networking-cisco:
Cisco
-----
* Git: https://git.openstack.org/stackforge/networking-cisco
* Launchpad: https://launchpad.net/networking-cisco
* PyPI: https://pypi.python.org/pypi/networking-cisco
.. _networking-hyperv:
.. _networking-metaplugin:
Metaplugin
----------
* Git: https://github.com/ntt-sic/networking-metaplugin
.. _networking-midonet:
MidoNet
-------
* Git: https://github.com/stackforge/networking-midonet
* Launchpad: https://launchpad.net/networking-midonet
* PyPI: https://pypi.python.org/pypi/networking-midonet
.. _networking-mlnx:
Mellanox
--------
* Git: https://github.com/stackforge/networking-mlnx
* Launchpad: https://launchpad.net/networking-mlnx
.. _networking-nec:
NEC
---
* Git: https://github.com/stackforge/networking-nec
* Launchpad: https://launchpad.net/networking-nec
.. _nuage-openstack-neutron:
.. _networking-odl:
OpenDayLight
------------
* Git: https://github.com/stackforge/networking-odl
* Launchpad: https://launchpad.net/networking-odl
.. _networking-ofagent:
OpenFlow Agent (ofagent)
------------------------
* Git: https://github.com/stackforge/networking-ofagent
* Launchpad: https://launchpad.net/networking-ofagent
* PyPI: https://pypi.python.org/pypi/networking-ofagent
.. _networking-ovs-dpdk:
Networking OVS-DPDK
-------------------
* Git: https://github.com/stackforge/networking-ovs-dpdk
* Launchpad: https://launchpad.net/networking-ovs-dpdk
.. _networking-plumgrid:
PLUMgrid
---------
* Git: https://github.com/stackforge/networking-plumgrid
* Launchpad: https://launchpad.net/networking-plumgrid
* PyPI: https://pypi.python.org/pypi/networking-plumgrid
.. _networking-vsphere:
.. _pluribus:
.. _vmware-nsx:
VMware
------
* Git: https://github.com/stackforge/vmware-nsx
* Launchpad: https://launchpad.net/vmware-nsx
* PyPI: https://pypi.python.org/pypi/vmware-nsx
View Git Blame Copy Permalink