openstack/neutron
Code Issues Proposed changes

Devref for out-of-tree plugin/driver contribution

Browse Source 
Change-Id: I6198acce97409e0e87520a31f2749b62d607e9c1
This commit is contained in:
Henry Gessau 2015-06-01 13:52:18 -04:00
parent 211c035577
commit b510dd5c2e
1 changed files with 467 additions and 317 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

784
doc/source/devref/contribute.rst
View File

@ -1,6 +1,19 @@
Contributing new extensions to Neutron
======================================
**NOTE!**
---------
**Third-party plugins/drivers which do not start decomposition in Liberty will
be marked as deprecated, and they will be removed before the Mxxx-3
milestone.**
Read on for details ...
Introduction
------------
Neutron has a pluggable architecture, with a number of extension points.
This documentation covers aspects relevant to contributing new Neutron
v2 core (aka monolithic) plugins, ML2 mechanism drivers, and L3 service
@ -16,22 +29,44 @@ within the OpenStack Networking project. If you are a developer who
wants to provide a Neutron-based solution without interacting with the
Neutron community, you are free to do so, but you can stop reading now,
as this guide is not for you.
In fact, from the Kilo release onwards, the Neutron core team propose that
additions to the codebase adopt a structure where the *monolithic plugins*,
*ML2 MechanismDrivers*, and *L3 service plugins* are integration-only
(called "vendor integration" hereinafter) to code that lives outside the
tree (called "vendor library" hereinafter); the same applies for any
vendor-specific agents. The only part that is to stay in the tree is the
agent 'main' (a small python file that imports agent code from the vendor
library and starts it). 'Outside the tree' can be anything that is publicly
available: it may be a stackforge repo for instance, a tarball, a pypi package,
etc. A plugin/drivers maintainer team self-governs in order to promote sharing,
reuse, innovation, and release of the 'out-of-tree' deliverable. It should not
be required for any member of the core team to be involved with this process,
Plugins and drivers for non-reference implementations are known as
"third-party" code. This includes code for supporting vendor products, as well
as code for supporting open-source networking implementations.
Before the Kilo release these plugins and drivers were included in the Neutron
tree. During the Kilo cycle the third-party plugins and drivers underwent the
first phase of a process called decomposition. During this phase, each plugin
and driver moved the bulk of its logic to a separate git repository, while
leaving a thin "shim" in the neutron tree together with the DB models and
migrations (and perhaps some config examples).
During the Liberty cycle the decomposition concept was taken to its conclusion
by allowing third-party code to exist entirely out of tree. Further extension
mechanisms have been provided to better support external plugins and drivers
that alter the API and/or the data model.
In the Mxxx cycle we will **require** all third-party code to be moved out of
the neutron tree completely.
'Outside the tree' can be anything that is publicly available: it may be a repo
on git.openstack.org for instance, a tarball, a pypi package, etc. A
plugin/drivers maintainer team self-governs in order to promote sharing, reuse,
innovation, and release of the 'out-of-tree' deliverable. It should not be
required for any member of the core team to be involved with this process,
although core members of the Neutron team can participate in whichever capacity
is deemed necessary to facilitate out-of-tree development.
Below, the following strategies will be documented:
This guide is aimed at you as the maintainer of code that integrates with
Neutron but resides in a separate repository.
Contribution Process
--------------------
If you want to extend OpenStack Networking with your technology, and you want
to do it within the visibility of the OpenStack project, follow the guidelines
and examples below. We'll describe best practices for:
* Design and Development;
* Testing and Continuous Integration;
@ -40,105 +75,61 @@ Below, the following strategies will be documented:
* DevStack Integration;
* Documentation;
This document will then provide a working example on how to contribute
new additions to Neutron.
Once you have everything in place you may want to add your project to the list
of Neutron sub-projects. Submit a patch via a gerrit review to neutron to add
your project to ``doc/source/devref/sub_projects.rst``.
Blueprint Spec Submission Strategy
Design and Development
----------------------
Assuming you have a working repository, any development to your own repo does
not need any blueprint, specification or bugs against Neutron. However, if your
project is a part of the Neutron Stadium effort, you are expected to
participate in the principles of the Four Opens, meaning your design should be
done in the open. Thus, it is encouraged to file documentation for changes in
your own repository.
If your code is hosted on git.openstack.org then the gerrit review system is
automatically provided. Contributors should follow the review guidelines
similar to those of Neutron. However, you as the maintainer have the
flexibility to choose who can approve/merge changes in your own repo.
It is recommended (but not required, see `policies
<http://docs.openstack.org/developer/neutron/policies/thirdparty-ci.html>`_)
that you set up a third-party CI system. This will provide a vehicle for
checking the third-party code against Neutron changes. See `Testing and
Continuous Integration`_ below for more detailed recommendations.
Design documents can still be supplied in form of Restructured Text (RST)
documents, within the same third-party library repo. If changes to the common
Neutron code are required, an `RFE
<http://docs.openstack.org/developer/neutron/policies/blueprints.html#neutron-request-for-feature-enhancements>`_
may need to be filed. However every case is different and you are invited to
seek guidance from Neutron core reviewers about what steps to follow.
Testing and Continuous Integration
----------------------------------
Provided contributors adhere to the abovementioned development footprint
they should not be required to follow the spec process for changes that
only affect their vendor integration and library. New contributions can
simply be submitted for code review, with the proviso that adequate
documentation and 3rd CI party is supplied at the time of the code
submission. For tracking purposes, the review itself can be tagged
with a Launchpad bug report. The bug should be marked as wishlist to
avoid complicating tracking of Neutron's primary deliverables. Design
documents can still be supplied in form of RST documents, within the same
vendor library repo. If substantial change to the common Neutron code are
required, a spec that targets common Neutron code will be required, however
every case is different and a contributor is invited to seek guidance from
the Neutron core team as to what steps to follow, and whether a spec or
a bug report is more suited for what a contributor needs to deliver.
The following strategies are recommendations only, since third-party CI testing
is not a enforced requirement. However, these strategies are employed by the
majority of the plugin/driver contributors that actively participate in the
Neutron development community, since they have learned from experience how
quickly their code can fall out of sync with the rapidly changing Neutron core
code base.
Once again, for submitting the integration module to the Neutron codebase,
no spec is required.
* You should run unit tests in your own external library (e.g. on
git.openstack.org where Jenkins setup is for free).
Development Strategy
--------------------
* Your third-party CI should validate third-party integration with Neutron via
functional testing. The third-party CI is a communication mechanism. The
objective of this mechanism is as follows:
* The following elements are suggested to be contributed in the tree
for plugins and drivers (called vendor integration hereinafter):
* Data models
* Extension definitions
* Configuration files
* Requirements file targeting vendor code
* Things that do not remain in the tree (called vendor library hereinafter):
* Vendor specific logic
* Associated unit tests
The idea here would be to provide in-tree the plugin/driver code that
implements an API, but have it delegate to out-of-tree code for
backend-specific interactions. The vendor integration will then typically
involve minor passthrough/parsing of parameters, minor handling of DB objects
as well as handling of responses, whereas the vendor library will do the
heavylifting and implement the vendor-specific logic. The boundary between
the in-tree layer and the out-of-tree one should be defined by the contributor
while asking these types of questions:
* If something changes in my backend, do I need to alter the integration
layer drastically? Clearly, the less impact there is, the better the
separation being achieved.
* If I expose vendor details (e.g. protocols, auth, etc.), can I easily swap
and replace the targeted backend (e.g. hardware with a newer version
being supplied) without affecting the integration too much? Clearly, the
more reusable the integration the better the separation.
As mentioned above, the vendor code *must* be available publicly, and a git
repository makes the most sense. By doing so, the module itself can be made
accessible using a pip requirements file. This file should not be confused
with the Neutron requirements file that lists all common dependencies. Instead
it should be a file 'requirements.txt' that is located in neutron/plugins/pluginXXX/,
whose content is something along the lines of 'my_plugin_xxx_library>=X.Y.Z'.
Vendors are responsible for ensuring that their library does not depend on
libraries conflicting with global requirements, but it could depend on
libraries not included in the global requirements. Just as in Neutron's
main requirements.txt, it will be possible to pin the version of the vendor
library.
For instance, a vendor integration module can become as simple as one that
performs only the following:
* Registering config options
* Registering the plugin class
* Registering the models
* Registering the extensions
Testing Strategy
----------------
The testing process will be as follow:
* No unit tests for the vendor integration of plugins and drivers are deemed
necessary. The expectation is that contributors would run unit test in their
own external library (e.g. in stackforge where Jenkins setup is for free).
For unit tests that validate the vendor library, it is the responsibility of
the vendor to choose what CI system they see fit to run them. There is no
need or requirement to use OpenStack CI resources if they do not want to.
Having said that, it may be useful to provide coverage for the shim layer in
the form of basic validation as done in `ODL <https://git.openstack.org/cgit/openstack/networking-odl/tree/networking_odl/tests/unit/ml2/test_mechanism_odl.py>`_ and `LBaaS A10 driver <https://git.openstack.org/cgit/openstack/neutron-lbaas/tree/neutron_lbaas/tests/unit/services/loadbalancer/drivers/a10networks/test_driver_v1.py>`_.
* 3rd Party CI will continue to validate vendor integration with Neutron via
functional testing. 3rd Party CI is a communication mechanism. This objective
of this mechanism is as follows:
* it communicates to plugin/driver contributors when someone has contributed
a change that is potentially breaking. It is then up to a given
contributor maintaining the affected plugin to determine whether the
failure is transient or real, and resolve the problem if it is.
* it communicates to you when someone has contributed a change that
potentially breaks your code. It is then up to you maintaining the affected
plugin/driver to determine whether the failure is transient or real, and
resolve the problem if it is.
* it communicates to a patch author that they may be breaking a plugin/driver.
If they have the time/energy/relationship with the maintainer of the
plugin/driver in question, then they can (at their discretion) work to
@ -146,69 +137,104 @@ The testing process will be as follow:
* it communicates to the community at large whether a given plugin/driver
is being actively maintained.
* A maintainer that is perceived to be responsive to failures in their
3rd party CI jobs is likely to generate community goodwill.
third-party CI jobs is likely to generate community goodwill.
It is worth noting that if the vendor library is hosted on StackForge, due to
current openstack-infra limitations, it is not possible to have 3rd party CI systems
participating in the gate pipeline for the StackForge repo. This means that the only
validation provided during the merge process to the StackForge repo is through unit
tests. Post-merge hooks can still be exploited to provide 3rd party CI feedback, and
alert the contributor/reviewer of potential issues. As mentioned above, 3rd party CI
systems will continue to validate Neutron core commits. This will allow them to
detect when incompatible changes occur, whether they are in Neutron or in the vendor
library repo.
It is worth noting that if the plugin/driver repository is hosted on
git.openstack.org, due to current openstack-infra limitations, it is not
possible to have third-party CI systems participating in the gate pipeline
for the repo. This means that the only validation provided during the merge
process to the repo is through unit tests. Post-merge hooks can still be
exploited to provide third-party CI feedback, and alert you of potential
issues. As mentioned above, third-party CI systems will continue to validate
Neutron core commits. This will allow them to detect when incompatible
changes occur, whether they are in Neutron or in the third-party repo.
Review and Defect Management Strategies
---------------------------------------
The usual process applies to the code that is part of OpenStack Neutron. More
precisely:
Defect Management
-----------------
Bugs affecting third-party code should *not* be filed in the Neutron project on
launchpad. Bug tracking can be done in any system you choose, but by creating a
third-party project in launchpad, bugs that affect both Neutron and your code
can be more easily tracked using launchpad's "also affects project" feature.
Security Issues
~~~~~~~~~~~~~~~
Here are some answers to how to handle security issues in your repo, taken
from `this openstack-dev mailing list message
<http://lists.openstack.org/pipermail/openstack-dev/2015-July/068617.html>`_:
- How should security your issues be managed?
The OpenStack Vulnerability Management Team (VMT) follows a `documented process
<https://security.openstack.org/vmt-process.html>`_ which can basically be
reused by any project-team when needed.
- Should the OpenStack security team be involved?
The OpenStack VMT directly oversees vulnerability reporting and disclosure for
a `subset of OpenStack source code repositories
<https://wiki.openstack.org/wiki/Security_supported_projects>`_. However they
are still quite happy to answer any questions you might have about
vulnerability management for your own projects even if they're not part of that
set. Feel free to reach out to the VMT in public or in private.
Also, the VMT is an autonomous subgroup of the much larger `OpenStack Security
project-team
<http://governance.openstack.org/reference/projects/security.html>`_. They're a
knowledgeable bunch and quite responsive if you want to get their opinions or
help with security-related issues (vulnerabilities or otherwise).
- Does a CVE need to be filed?
It can vary widely. If a commercial distribution such as Red Hat is
redistributing a vulnerable version of your software then they may assign one
anyway even if you don't request one yourself. Or the reporter may request one;
the reporter may even be affiliated with an organization who has already
assigned/obtained a CVE before they initiate contact with you.
- Do the maintainers need to publish OSSN or equivalent documents?
OpenStack Security Advisories (OSSA) are official publications of the OpenStack
VMT and only cover VMT-supported software. OpenStack Security Notes (OSSN) are
published by editors within the OpenStack Security project-team on more general
security topics and may even cover issues in non-OpenStack software commonly
used in conjunction with OpenStack, so it's at their discretion as to whether
they would be able to accommodate a particular issue with an OSSN.
However, these are all fairly arbitrary labels, and what really matters in the
grand scheme of things is that vulnerabilities are handled seriously, fixed
with due urgency and care, and announced widely -- not just on relevant
OpenStack mailing lists but also preferably somewhere with broader distribution
like the `Open Source Security mailing list
<http://oss-security.openwall.org/wiki/mailing-lists/oss-security>`_. The goal
is to get information on your vulnerabilities, mitigating measures and fixes
into the hands of the people using your software in a timely manner.
- Anything else to consider here?
The OpenStack VMT is in the process of trying to reinvent itself so that it can
better scale within the context of the "Big Tent." This includes making sure
the policy/process documentation is more consumable and reusable even by
project-teams working on software outside the scope of our charter. It's a work
in progress, and any input is welcome on how we can make this function well for
everyone.
* Bugs that affect vendor code can be filed against the Neutron integration,
if the integration code is at fault. Otherwise, the code maintainer may
decide to fix a bug without oversight, and update their requirements file
to target a new version of their vendor library. It makes sense to
require 3rd party CI for a given plugin/driver to pass when changing their
dependency before merging to any branch (i.e. both master and stable branches).
* Vendor specific code should follow the same review guidelines as any other
code in the tree. However, the maintainer has flexibility to choose who
can approve/merge changes in this repo.
Backport Management Strategies
------------------------------
As outlined in the `Spec proposal <http://specs.openstack.org/openstack/neutron-specs/specs/kilo/core-vendor-decomposition.html>`_
all new plugins and drivers will have to follow the contribution model
described here. As for existing plugins and drivers, no in-tree features can
be merged until some progress has been done to make the solution adhere to
this model. That said, there is the question of critical fixes and/or backports
to `stable branches <https://wiki.openstack.org/wiki/StableBranch>`_. The possible
scenarios are:
This section applies only to third-party maintainers who had code in the
Neutron tree during the Kilo and earlier releases. It will be obsolete once the
Kilo release is no longer supported.
If a change made to out-of-tree third-party code needs to be back-ported to
in-tree code in a stable branch, you may submit a review without a
corresponding master branch change. The change will be evaluated by core
reviewers for stable branches to ensure that the backport is justified and that
it does not affect Neutron core code stability.
* The decomposition just completed, we are in the cycle (X) where the decomposition
initiated: in this case, the Neutron master branch no longer have the vendor
library code, but the stable branch still does. Backports via straight
cherry-picks may not be possible, or as easy, therefore a custom backport to
stable could be deemed acceptable to Neutron's stable branches (e.g. stable/X-1
and/or stable/X-2), as required.
* The decomposition is complete, we are in the next cycle where the
decomposition work completed (X+1): backports will be done to the stable branch
available of the vendor library (stable/X), and Neutron's stable branch
(stable/X-1), as outlined in the previous step.
* The decomposition is complete, we are in two or more cycles after the
decomposition work completed (X+2, or later). Backports will be done to the
stable branch(s) available of the vendor library (stable/X, stable/X+1).
* The decomposition is in progress: as long as the vendor code is still in
master, patches will need to go to master before a backport to stable.
Acceptance will be determined on the scope of changes (based on both the
amount of work and severity of the issue). In this case, the plugin or
driver maintainer will need to ensure that the fix gets applied to the
external repo, if necessary (to avoid missing it during the migration process).
* The decomposition has not started: in this case, depending on the issue,
review attention from core members is best effort, and although there is no
explicit rule to prevent them from merging to master, it is in the best interest
of the maintainer to avoid introducing or modifying existing code that will
ultimately be deprecated.
DevStack Integration Strategies
-------------------------------
@ -221,79 +247,34 @@ make sense depending on whether you are contributing a new or existing plugin or
driver.
If you are contributing a new plugin, the approach to choose should be based on
`Extras.d Hooks' externally hosted plugins <http://docs.openstack.org/developer/devstack/plugins.html#extras-d-hooks>`_.
With the extra.d hooks, the DevStack integration is colocated with the vendor integration
library, and it leads to the greatest level of flexibility when dealing with DevStack based
dev/test deployments.
`Extras.d Hooks' externally hosted plugins
<http://docs.openstack.org/developer/devstack/plugins.html#extras-d-hooks>`_.
With the extra.d hooks, the DevStack integration is co-located with the
third-party integration library, and it leads to the greatest level of
flexibility when dealing with DevStack based dev/test deployments.
Having said that, most Neutron plugins developed in the past likely already have
integration with DevStack in the form of `neutron_plugins <https://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/neutron_plugins>`_.
If the plugin is being decomposed in vendor integration plus vendor library, it would
be necessary to adjust the instructions provided in the neutron_plugin file to pull the
vendor library code as a new dependency. For instance, the instructions below:
One final consideration is worth making for third-party CI setups: if `Devstack
Gate <https://git.openstack.org/cgit/openstack-infra/devstack-gate>`_ is used,
it does provide hook functions that can be executed at specific times of the
devstack-gate-wrap script run. For example, the `Neutron Functional job
<https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/neutron.yaml>`_
uses them. For more details see `devstack-vm-gate-wrap.sh
<https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh>`_.
::
INSTALL_FROM_REQUIREMENTS=$(trueorfalse True INSTALL_FROM_REQUIREMENTS)
Project Initial Setup
---------------------
if [[ "$INSTALL_FROM_REQUIREMENTS" == "False" ]]; then
git_clone $NEUTRON_LIB_REPO $NEUTRON_LIB_DIR $NEUTRON_LIB_BRANCH
setup_package $NEUTRON_LIB_DIR
else
# Retrieve the package from the vendor library's requirements.txt
plugin_package=$(cat $NEUTRON_LIB_REQUIREMENTS_FILE)
pip_install "$plugin_package"
fi
could be placed in 'neutron_plugin_configure_service', ahead of the service
configuration. An alternative could be under the `third_party section
<https://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/neutron_thirdparty>`_,
if available. This solution can be similarly exploited for both monolithic
plugins or ML2 mechanism drivers. The configuration of the plugin or driver itself can be
done by leveraging the extensibility mechanisms provided by `local.conf <http://docs.openstack.org/developer/devstack/configuration.html>`_. In fact, since the .ini file for the vendor plugin or driver lives
in the Neutron tree, it is possible to do add the section below to local.conf:
::
[[post-config|$THE_FILE_YOU_NEED_TO_CUSTOMIZE]]
# Override your section config as you see fit
[DEFAULT]
verbose=True
Which in turn it is going to edit the file with the options outlined in the post-config
section.
The above mentioned approach, albeit valid, has the shortcoming of depending on DevStack's
explicit support for the plugin installation and configuration, and the plugin maintainer
is strongly encouraged to revise the existing DevStack integration, in order to evolve it
in an extras.d hooks based approach.
One final consideration is worth making for 3rd party CI setups: if `Devstack Gate
<https://git.openstack.org/cgit/openstack-infra/devstack-gate>`_ is used, it does provide hook
functions that can be executed at specific times of the devstack-gate-wrap script run.
For example, the `Neutron Functional job <https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/neutron.yaml>`_ uses them. For more details see `devstack-vm-gate-wrap.sh <https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh>`_.
Documentation Strategies
------------------------
It is the duty of the new contributor to provide working links that can be
referenced from the OpenStack upstream documentation.
#TODO(armax): provide more info, when available.
How-to
------
The how-to below assumes that the vendor library will be hosted on StackForge.
Stackforge lets you tap in the entire OpenStack CI infrastructure and can be
a great place to start from to contribute your new or existing driver/plugin.
The list of steps below are somewhat the tl;dr; version of what you can find
on http://docs.openstack.org/infra/manual/creators.html. They are meant to
The how-to below assumes that the third-party library will be hosted on
git.openstack.org. This lets you tap in the entire OpenStack CI infrastructure
and can be a great place to start from to contribute your new or existing
driver/plugin. The list of steps below are summarized version of what you can
find on http://docs.openstack.org/infra/manual/creators.html. They are meant to
be the bare minimum you have to complete in order to get you off the ground.
* Create a public repository: this can be a personal git.openstack.org repo or any
publicly available git repo, e.g. ``https://github.com/john-doe/foo.git``. This
would be a temporary buffer to be used to feed the StackForge one.
would be a temporary buffer to be used to feed the one on git.openstack.org.
* Initialize the repository: if you are starting afresh, you may *optionally*
want to use cookiecutter to get a skeleton project. You can learn how to use
cookiecutter on https://git.openstack.org/cgit/openstack-dev/cookiecutter.
@ -301,104 +282,273 @@ be the bare minimum you have to complete in order to get you off the ground.
want to skip this step now, build the history first (next step), and come back
here to initialize the remainder of the repository with other files being
generated by the cookiecutter (like tox.ini, setup.cfg, setup.py, etc.).
* Building the history: if you are contributing an existing driver/plugin,
you may want to preserve the existing history. If not, you can go to the
next step. To import the history from an existing project this is what
you need to do:
* Clone a copy of the neutron repository to be manipulated.
* Go into the Neutron repo to be changed.
* Execute file split.sh, available in ./tools, and follow instructions.
::
git clone https://git.openstack.org/openstack/neutron.git
cd neutron
./tools/split.sh
# Sit and wait for a while, or grab a cup of your favorite drink
At this point you will have the project pruned of everything else but
the files you want to export, with their history. The next steps are:
* Check out stable branches for the project: even though stable branches
are not strictly necessary during the creation of the StackForge repository
(as outlined in the next step below), they do not hurt, and it is
recommended to keep them during the import process.
* Add a remote that points to the repository created before.
* (Optional) If the repository has already being initialized with
cookiecutter, you need to pull first; if not, you can either push
the existing commits/tags or apply and commit further changes to fix
up the structure of repo the way you see fit.
* Finally, push commits and tags to the public repository. If you followed
theses instructions step-by-step, you will have a source repository
that contains both a master and stable branches, as well as tags. Some
of these steps are outlined below:
::
git remote add <foo> https://github.com/john-doe/foo.git
git pull foo master # OPTIONAL, if foo is non-empty
git push --all foo && git push --tags foo
* Create a StackForge repository: for this you need the help of the OpenStack
infra team. It is worth noting that you only get one shot at creating the
StackForge repository. This is the time you get to choose whether you want
to start from a clean slate, or you want to import the repo created during
the previous step. In the latter case, you can do so by specifying the
upstream section for your project in project-config/gerrit/project.yaml.
Steps are documented on the
`Repository Creator's Guide <http://docs.openstack.org/infra/manual/creators.html>`_.
* Create a repository on git.openstack.org (see `Official Sub-Projects
<http://docs.openstack.org/developer/neutron/devref/sub_projects.html>`_). For
this you need the help of the OpenStack infra team. It is worth noting that
you only get one shot at creating the repository on git.openstack.org. This
is the time you get to choose whether you want to start from a clean slate,
or you want to import the repo created during the previous step. In the
latter case, you can do so by specifying the upstream section for your
project in project-config/gerrit/project.yaml. Steps are documented on the
`Repository Creator's Guide
<http://docs.openstack.org/infra/manual/creators.html>`_.
* Ask for a Launchpad user to be assigned to the core team created. Steps are
documented in
`this section <http://docs.openstack.org/infra/manual/creators.html#update-the-gerrit-group-members>`_.
* Fix, fix, fix: at this point you have an external base to work on. You
can develop against the new stackforge project, the same way you work
with any other OpenStack project: you have pep8, docs, and python27 CI
jobs that validate your patches when posted to Gerrit. For instance, one
thing you would need to do is to define an entry point for your plugin
or driver in your own setup.cfg similarly as to how it is done
`here <https://git.openstack.org/cgit/openstack/networking-odl/tree/setup.cfg#n31>`_.
documented in `this section
<http://docs.openstack.org/infra/manual/creators.html#update-the-gerrit-group-members>`_.
* Fix, fix, fix: at this point you have an external base to work on. You can
develop against the new git.openstack.org project, the same way you work with
any other OpenStack project: you have pep8, docs, and python27 CI jobs that
validate your patches when posted to Gerrit. For instance, one thing you
would need to do is to define an entry point for your plugin or driver in
your own setup.cfg similarly as to how it is done in the `setup.cfg for ODL
<https://git.openstack.org/cgit/openstack/networking-odl/tree/setup.cfg#n31>`_.
* Define an entry point for your plugin or driver in setup.cfg
* Create 3rd Party CI account: if you do not already have one, follow
instructions for
`3rd Party CI <http://docs.openstack.org/infra/system-config/third_party.html>`_ to get one.
* TODO(armax): ...
* Create third-party CI account: if you do not already have one, follow
instructions for `third-party CI
<http://docs.openstack.org/infra/system-config/third_party.html>`_ to get
one.
Decomposition progress chart
============================
Integrating with the Neutron system
-----------------------------------
The chart below captures the progress of the core-vendor-decomposition effort
for existing plugins and drivers at the time the decomp effort started. New
drivers and plugins are not required to be listed here. This chart is short
lived: once the effort is complete, this chart no longer needs to exist and
will be removed. The following aspects are captured:
(This section currently describes the goals and progress of the completion of
the decomposition work during the Liberty development cycle. The content here
will be updated as the work progresses. In its final form this section will be
merged with the previous section. When all existing plugins/drivers are fully
decomposed, this document will be a recipe for how to add a new Neutron plugin
or driver completely out-of-tree.)
* Name: the name of the project that implements a Neutron plugin or driver. The
name is an internal target for links that point to source code, etc.
* Plugins/Drivers: whether the source code contains a core (aka monolithic)
plugin, a set of ML2 drivers, and/or (service) plugins (or extensions) for
firewall, vpn, and load balancers.
* Launchpad: whether the project is managed through Launchpad.
* PyPI: whether the project deliverables are available through PyPI.
* State: a code to represent the current state of the decomposition. Possible
values are:
For the Liberty cycle we aim to move all the existing third-party code out of
the Neutron tree. Each category of code and its removal plan is described
below.
* [A] External repo available, no code decomposition
* [B] External repo available, partial code decomposition
* [C] External repo available, code decomposition is complete
* [D] Not deemed required. Driver is already bare-bone and decomposition
effort is not considered justified. Assessment may change in the
future.
Absence of an entry for an existing plugin or driver means no active effort
has been observed or potentially not required.
* Completed in: the release in which the effort is considered completed. Code
completion can be deemed as such, if there is no overlap/duplication between
what exists in the Neutron tree, and what it exists in the vendor repo.
Existing Shims
~~~~~~~~~~~~~~
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
| Name | Plugins/Drivers | Launchpad | PyPI | State | Completed in |
+===============================+=======================+===========+==================+=========+==============+
| freescale-nscs | ml2,fw | no | no | [D] | |
+-------------------------------+-----------------------+-----------+------------------+---------+--------------+
Liberty Steps
+++++++++++++
The existing shims shall now be moved out of tree, together with any test
code. The entry points shall be moved as described below in `Entry Points`_.
Configuration Files
~~~~~~~~~~~~~~~~~~~
The ``data_files`` in the ``[files]`` section of ``setup.cfg`` of Neutron shall
not contain any third-party references. These shall be located in the same
section of the third-party repo's own ``setup.cfg`` file.
* Note: Care should be taken when naming sections in configuration files. When
the Neutron service or an agent starts, oslo.config loads sections from all
specified config files. This means that if a section [foo] exists in multiple
config files, duplicate settings will collide. It is therefore recommended to
prefix section names with a third-party string, e.g. [vendor_foo].
Liberty Steps
+++++++++++++
Third-party configuration files still in the neutron tree have no dependencies
and can simply be moved. The maintainers should add their configuration file(s)
to their repo and then remove them from neutron.
**ToDo: Inclusion in OpenStack documentation?**
Is there a recommended way to have third-party config options listed in the
configuration guide in docs.openstack.org?
Database Models and Migrations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A third-party repo may contain database models for its own tables. Although
these tables are in the Neutron database, they are independently managed
entirely within the third-party code. Third-party code shall **never** modify
neutron core tables in any way.
Each repo has its own alembic migration branch that adds, removes and modifies
its own tables in the neutron database schema.
* Note: Care should be taken when adding new tables. To prevent collision of
table names it is recommended to prefix them with a vendor/plugin string.
* Note: A third-party maintainer may opt to use a separate database for their
tables. This may complicate cases where there are foreign key constraints
across schemas for DBMS that do not support this well. Third-party maintainer
discretion advised.
The database tables owned by a third-party repo can have references to fields
in neutron core tables. However, the alembic branch for a plugin/driver repo
shall never update any part of a table that it does not own.
**Note: What happens when a referenced item changes?**
* **Q:** If a driver's table has a reference (for example a foreign key) to a
neutron core table, and the referenced item is changed in neutron, what
should you do?
* **A:** Fortunately, this should be an extremely rare occurrence. Neutron core
reviewers will not allow such a change unless there is a very carefully
thought-out design decision behind it. That design will include how to
address any third-party code affected. (This is another good reason why you
should stay actively involved with the Neutron developer community.)
The ``neutron-db-manage`` alembic wrapper script for neutron detects alembic
branches for installed third-party repos, and the upgrade command automatically
applies to all of them. A third-party repo must register its alembic migrations
at installation time. This is done by providing an entrypoint in setup.cfg as
follows:
For a third-party repo named ``networking-foo``, add the alembic_migrations
directory as an entrypoint in the ``neutron.db.alembic_migrations`` group::
[entry_points]
neutron.db.alembic_migrations =
networking-foo = networking_foo.db.migration:alembic_migrations
Liberty Steps
+++++++++++++
Each decomposed plugin/driver that has its own tables in the neutron database
should take these steps to move the models for the tables out of tree.
#. Add the models to the external repo.
#. Create a start migration for the repo's alembic branch. Note: it is
recommended to keep the migration file(s) in the same location in the
third-party repo as is done in the neutron repo,
i.e. ``networking_foo/db/migration/alembic_migrations/versions/*.py``
#. Remove the models from the neutron repo.
#. Add the names of the removed tables to ``DRIVER_TABLES`` in
``neutron/db/migration/alembic_migrations/external.py`` (this is used for
testing, see below).
**ToDo: neutron-db-manage autogenerate**
The alembic autogenerate command needs to support branches in external
repos. Bug #1471333 has been filed for this.
DB Model/Migration Testing
~~~~~~~~~~~~~~~~~~~~~~~~~~
Here is a `template functional test
<https://bugs.launchpad.net/neutron/+bug/1470678>`_ (TODO:Ann) third-party
maintainers can use to develop tests for model-vs-migration sync in their
repos. It is recommended that each third-party CI sets up such a test, and runs
it regularly against Neutron master.
Liberty Steps
+++++++++++++
The model_sync test will be updated to ignore the models that have been moved
out of tree. A ``DRIVER_TABLES`` list will be maintained in
``neutron/db/migration/alembic_migrations/external.py``.
Entry Points
~~~~~~~~~~~~
The `Python setuptools <https://pythonhosted.org/setuptools>`_ installs all
entry points for packages in one global namespace for an environment. Thus each
third-party repo can define its package's own ``[entry_points]`` in its own
``setup.cfg`` file.
For example, for the ``networking-foo`` repo::
[entry_points]
console_scripts =
neutron-foo-agent = networking_foo.cmd.eventlet.agents.foo:main
neutron.core_plugins =
foo_monolithic = networking_foo.plugins.monolithic.plugin:FooPluginV2
neutron.service_plugins =
foo_l3 = networking_foo.services.l3_router.l3_foo:FooL3ServicePlugin
neutron.ml2.type_drivers =
foo_type = networking_foo.plugins.ml2.drivers.foo:FooType
neutron.ml2.mechanism_drivers =
foo_ml2 = networking_foo.plugins.ml2.drivers.foo:FooDriver
neutron.ml2.extension_drivers =
foo_ext = networking_foo.plugins.ml2.drivers.foo:FooExtensionDriver
* Note: It is advisable to include ``foo`` in the names of these entry points to
avoid conflicts with other third-party packages that may get installed in the
same environment.
API Extensions
~~~~~~~~~~~~~~
Extensions can be loaded in two ways:
#. Use the ``append_api_extensions_path()`` library API. This method is defined
in ``neutron/api/extensions.py`` in the neutron tree.
#. Leverage the ``api_extensions_path`` config variable when deploying. See the
example config file ``etc/neutron.conf`` in the neutron tree where this
variable is commented.
Interface Drivers
~~~~~~~~~~~~~~~~~
Interface (VIF) drivers for the reference implementations are defined in
``neutron/agent/linux/interface.py``. Third-party interface drivers shall be
defined in a similar location within their own repo.
The entry point for the interface driver is a Neutron config option. It is up to
the installer to configure this item in the ``[default]`` section. For example::
[default]
interface_driver = networking_foo.agent.linux.interface.FooInterfaceDriver
**ToDo: Interface Driver port bindings.**
These are currently defined by the ``VIF_TYPES`` in
``neutron/extensions/portbindings.py``. We could make this config-driven
for agents. For Nova, selecting the VIF driver can be done outside of
Neutron (using the new `os-vif python library
<https://review.openstack.org/193668>`_?). Armando and Akihiro to discuss.
Rootwrap Filters
~~~~~~~~~~~~~~~~
If a third-party repo needs a rootwrap filter for a command that is not used by
Neutron core, then the filter shall be defined in the third-party repo.
For example, to add a rootwrap filters for commands in repo ``networking-foo``:
* In the repo, create the file:
``etc/neutron/rootwrap.d/foo.filters``
* In the repo's ``setup.cfg`` add the filters to data_files::
[files]
data_files =
etc/neutron/rootwrap.d =
etc/neutron/rootwrap.d/foo.filters
Extending python-neutronclient
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The maintainer of a third-party component may wish to add extensions to the
Neutron CLI client. Thanks to https://review.openstack.org/148318 this can now
be accomplished. See `Client Command Extensions
<client_command_extensions.html>`_.
Other repo-split items
~~~~~~~~~~~~~~~~~~~~~~
(These are still TBD.)
* Splitting policy.json? **ToDo** Armando will investigate.
* Generic instructions (or a template) for installing an out-of-tree plugin or
driver for Neutron. Possibly something for the networking guide, and/or a
template that plugin/driver maintainers can modify and include with their
package.
Decomposition Phase II Progress Chart
=====================================
TBD.