diff --git a/doc/source/devref/contribute.rst b/doc/source/devref/contribute.rst new file mode 100644 index 00000000000..cc167083a7b --- /dev/null +++ b/doc/source/devref/contribute.rst @@ -0,0 +1,256 @@ +Contributing new extensions to Neutron +====================================== + +Neutron has a pluggable architecture, with a number of extension points. +This documentation covers aspects relevant to contributing new Neutron +v2 core (aka monolithic) plugins, ML2 mechanism drivers, and L3 service +plugins. This document will initially cover a number of process-oriented +aspects of the contribution process, and proceed to provide a how-to guide +that shows how to go from 0 LOC's to successfully contributing new +extensions to Neutron. In the remainder of this guide, we will try to +use practical examples as much as we can so that people have working +solutions they can start from. + +This guide is for a developer who wants to have a degree of visibility +within the OpenStack Networking project. If you are a developer who +wants to provide a Neutron-based solution without interacting with the +Neutron community, you are free to do so, but you can stop reading now, +as this guide is not for you. +In fact, from the Kilo release onwards, the Neutron core team propose that +additions to the codebase adopt a structure where the *monolithic plugins*, +*ML2 MechanismDrivers*, and *L3 service plugins* are integration-only +(called "vendor integration" hereinafter) to code that lives outside the +tree (called "vendor library" hereinafter); the same applies for any +vendor-specific agents. The only part that is to stay in the tree is the +agent 'main' (a small python file that imports agent code from the vendor +library and starts it). 'Outside the tree' can be anything that is publicly +available: it may be a stackforge repo for instance, a tarball, a pypi package, +etc. A plugin/drivers maintainer team self-governs in order to promote sharing, +reuse, innovation, and release of the 'out-of-tree' deliverable. It should not +be required for any member of the core team to be involved with this process, +although core members of the Neutron team can participate in whichever capacity +is deemed necessary to facilitate out-of-tree development. + +Below, the following strategies will be documented: + +* Design and Development; +* Testing and Continuous Integration; +* Defect Management; +* Documentation; + +This document will then provide a working example on how to contribute +new additions to Neutron. + +Blueprint Spec Submission Strategy +---------------------------------- + +Provided contributors adhere to the abovementioned development footprint +they should not be required to follow the spec process for changes that +only affect their vendor integration and library. New contributions can +simply be submitted for code review, with the proviso that adequate +documentation and 3rd CI party is supplied at the time of the code +submission. For tracking purposes, the review itself can be tagged +with a Launchpad bug report. The bug should be marked as wishlist to +avoid complicating tracking of Neutron's primary deliverables. Design +documents can still be supplied in form of RST documents, within the same +vendor library repo. If substantial change to the common Neutron code are +required, a spec that targets common Neutron code will be required, however +every case is different and a contributor is invited to seek guidance from +the Neutron core team as to what steps to follow, and whether a spec or +a bug report is more suited for what a contributor needs to deliver. + +Once again, for submitting the integration module to the Neutron codebase, +no spec is required. + +Development Strategy +-------------------- + +* The following elements are suggested to be contributed in the tree + for plugins and drivers (called vendor integration hereinafter): + + * Data models + * Extension definitions + * Configuration files + * Requirements file targeting vendor code + +* Things that do not remain in the tree (called vendor library hereinafter): + + * Vendor specific logic + * Associated unit tests + +The idea here would be to provide in-tree the plugin/driver code that +implements an API, but have it delegate to out-of-tree code for +backend-specific interactions. The vendor integration will then typically +involve minor passthrough/parsing of parameters, minor handling of DB objects +as well as handling of responses, whereas the vendor library will do the +heavylifting and implement the vendor-specific logic. The boundary between +the in-tree layer and the out-of-tree one should be defined by the contributor +while asking these types of questions: + + * If something changes in my backend, do I need to alter the integration + layer drastically? Clearly, the less impact there is, the better the + separation being achieved. + * If I expose vendor details (e.g. protocols, auth, etc.), can I easily swap + and replace the targeted backend (e.g. hardware with a newer version + being supplied) without affecting the integration too much? Clearly, the + more reusable the integration the better the separation. + +As mentioned above, the vendor code *must* be available publicly, and a git +repository makes the most sense. By doing so, the module itself can be made +accessible using a pip requirements file. This file should not be confused +with the Neutron requirements file that lists all common dependencies. Instead +it should be a file 'requirements.txt' that is located in neutron/plugins/pluginXXX/, +whose content is something along the lines of 'my_plugin_xxx_library>=X.Y.Z'. +Vendors are responsible for ensuring that their library does not depend on +libraries conflicting with global requirements, but it could depend on +libraries not included in the global requirements. Just as in Neutron's +main requirements.txt, it will be possible to pin the version of the vendor +library. + +For instance, a vendor integration module can become as simple as one that +performs only the following: + +* Registering config options +* Registering the plugin class +* Registering the models +* Registering the extensions + +Testing Strategy +---------------- + +The testing process will be as follow: + +* There will be no unit tests for plugins and drivers in the tree; The + expectation is that contributors would run unit test in their own external + library (e.g. in stackforge where Jenkins setup is for free). For unit tests + that validate the vendor library, it is the responsibility of the vendor to + choose what CI system they see fit to run them. There is no need or + requirement to use OpenStack CI resources if they do not want to. +* 3rd Party CI will continue to validate vendor integration with Neutron via + functional testing. 3rd Party CI is a communication mechanism. This objective + of this mechanism is as follows: + + * it communicates to plugin/driver contributors when someone has contributed + a change that is potentially breaking. It is then up to a given + contributor maintaining the affected plugin to determine whether the + failure is transient or real, and resolve the problem if it is. + * it communicates to a patch author that they may be breaking a plugin/driver. + If they have the time/energy/relationship with the maintainer of the + plugin/driver in question, then they can (at their discretion) work to + resolve the breakage. + * it communicates to the community at large whether a given plugin/driver + is being actively maintained. + * A maintainer that is perceived to be responsive to failures in their + 3rd party CI jobs is likely to generate community goodwill. + +Review and Defect Management Strategies +--------------------------------------- + +The usual process applies to the code that is part of OpenStack Neutron. More +precisely: + +* Bugs that affect vendor code can be filed against the Neutron integration, + if the integration code is at fault. Otherwise, the code maintainer may + decide to fix a bug without oversight, and update their requirements file + to target a new version of their vendor library. It makes sense to + require 3rd party CI for a given plugin/driver to pass when changing their + dependency before merging to any branch (i.e. both master and stable branches). +* Vendor specific code should follow the same review guidelines as any other + code in the tree. However, the maintainer has flexibility to choose who + can approve/merge changes in this repo. + +Documentation Strategies +------------------------ + +It is the duty of the new contributor to provide working links that can be +referenced from the OpenStack upstream documentation. +#TODO(armax): provide more info, when available. + +How-to +------ + +The how-to below assumes that the vendor library will be hosted on StackForge. +Stackforge lets you tap in the entire OpenStack CI infrastructure and can be +a great place to start from to contribute your new or existing driver/plugin. +The list of steps below are somewhat the tl;dr; version of what you can find +on http://docs.openstack.org/infra/manual/creators.html. They are meant to +be the bare minimum you have to complete in order to get you off the ground. + +* Create a public repository: this can be a personal github.com repo or any + publicly available git repo, e.g. https://github.com/john-doe/foo.git. This + would be a temporary buffer to be used to feed the StackForge one. +* Initialize the repository: if you are starting afresh, you may *optionally* + want to use cookiecutter to get a skeleton project. You can learn how to use + cookiecutter on https://github.com/openstack-dev/cookiecutter. + If you want to build the repository from an existing Neutron module, you may + want to skip this step now, build the history first (next step), and come back + here to initialize the remainder of the repository with other files being + generated by the cookiecutter (like tox.ini, setup.cfg, setup.py, etc.). +* Building the history: if you are contributing an existing driver/plugin, + you may want to preserve the existing history. If not, you can go to the + next step. To import the history from an existing project this is what + you need to do: + + * Clone a copy of the neutron repository to be manipulated. + * Go into the Neutron repo to be changed. + * Execute file split.sh, available in ./tools, and follow instructions. + + :: + + git clone https://github.com/openstack/neutron.git + cd neutron + ./tools/split.sh + # Sit and wait for a while, or grab a cup of your favorite drink + + At this point you will have the project pruned of everything else but + the files you want to export, with their history. The next steps are: + + * Add a remote that points to the repository created before. + * (Optional) If the repository has already being initialized with + cookiecutter, you need to pull first; if not, you can either push + the existing commits/tags or apply and commit further changes to fix + up the structure of repo the way you see fit. + * Finally, push commits and tags to the public repository. + + :: + + git remote add https://github.com/john-doe/foo.git + git pull foo master # OPTIONAL, if foo is non-empty + git push --all foo && git push --tags foo + +* Create a StackForge repository: for this you need the help of the OpenStack + infra team. It is worth noting that you only get one shot at creating the + StackForge repository. This is the time you get to choose whether you want + to start from a clean slate, or you want to import the repo created during + the previous step. In the latter case, you can do so by specifying the + upstream section for your project in project-config/gerrit/project.yaml. + Steps are documented on the + `Project Creators Manual http://docs.openstack.org/infra/manual/creators.html`. +* Ask for a Launchpad user to be assigned to the core team created. Steps are + documented in + `this section http://docs.openstack.org/infra/manual/creators.html#update-the-gerrit-group-members`. +* Fix, fix, fix: at this point you have an external base to work on. You + can develop against the new stackforge project, the same way you work + with any other OpenStack project: you have pep8, docs, and python27 CI + jobs that validate your patches when posted to Gerrit. For instance, one + thing you would need to do is to define an entry point for your plugin + or driver in your own setup.cfg similarly as to how it is done + `here https://github.com/stackforge/networking-odl/blob/master/setup.cfg#L31`. +* Define an entry point for your plugin or driver in setup.cfg +* Create 3rd Party CI account: if you do not already have one, follow + instructions for + `3rd Party CI http://ci.openstack.org/third_party.html` to get one. +* TODO(armax): ... + +The 'ODL ML2 Mechanism Driver' - example 1 +------------------------------------------ + +* Create the StackForge repo: https://review.openstack.org/#/c/136854/ +* TODO(armax): continue with adding meat on the bone here + +The 'OVSvAPP Mechanism Driver' - example 2 +------------------------------------------ + +* Create the StackForge repo: https://review.openstack.org/#/c/136091/ +* Cookiecutter initial commit: https://review.openstack.org/#/c/141268/ +* TODO(armax): continue with adding meat on the bone here diff --git a/doc/source/devref/development.environment.rst b/doc/source/devref/development.environment.rst index 21b90b4ebf2..d7de7786dc0 100644 --- a/doc/source/devref/development.environment.rst +++ b/doc/source/devref/development.environment.rst @@ -47,3 +47,5 @@ Grab the code:: .. include:: ../../../TESTING.rst + +.. include:: ./contribute.rst diff --git a/tools/split.sh b/tools/split.sh new file mode 100755 index 00000000000..0de018210cc --- /dev/null +++ b/tools/split.sh @@ -0,0 +1,102 @@ +#!/bin/sh +# +# This script has been shamelessly copied and tweaked from original copy: +# +# https://github.com/openstack/oslo-incubator/blob/master/tools/graduate.sh +# +# Use this script to export a Neutron module to a separate git repo. +# +# You can call this script Call script like so: +# +# ./split.sh +# +# The file should be a text file like the one below: +# +# /path/to/file/file1 +# /path/to/file/file2 +# ... +# /path/to/file/fileN +# +# Such a list can be generated with a command like this: +# +# find $path -type f # path is the base dir you want to list files for + +set -ex + +file_list_path="$1" +project_name="$2" +files_to_keep=$(cat $file_list_path) + + +# Build the grep pattern for ignoring files that we want to keep +keep_pattern="\($(echo $files_to_keep | sed -e 's/^/\^/' -e 's/ /\\|\^/g')\)" +# Prune all other files in every commit +pruner="git ls-files | grep -v \"$keep_pattern\" | git update-index --force-remove --stdin; git ls-files > /dev/stderr" + +# Find all first commits with listed files and find a subset of them that +# predates all others + +roots="" +for file in $files_to_keep; do + file_root="$(git rev-list --reverse HEAD -- $file | head -n1)" + fail=0 + for root in $roots; do + if git merge-base --is-ancestor $root $file_root; then + fail=1 + break + elif !git merge-base --is-ancestor $file_root $root; then + new_roots="$new_roots $root" + fi + done + if [ $fail -ne 1 ]; then + roots="$new_roots $file_root" + fi +done + +# Purge all parents for those commits + +set_roots=" +if [ 1 -eq 0 $(for root in $roots; do echo " -o \"\$GIT_COMMIT\" = '$root' "; done) ]; then + echo ''; +else + cat; +fi" + +# Enhance git_commit_non_empty_tree to skip merges with: +# a) either two equal parents (commit that was about to land got purged as well +# as all commits on mainline); +# b) or with second parent being an ancestor to the first one (just as with a) +# but when there are some commits on mainline). +# In both cases drop second parent and let git_commit_non_empty_tree to decide +# if commit worth doing (most likely not). + +skip_empty=$(cat << \EOF +if [ $# = 5 ] && git merge-base --is-ancestor $5 $3; then + git_commit_non_empty_tree $1 -p $3 +else + git_commit_non_empty_tree "$@" +fi +EOF +) + +# Filter out commits for unrelated files +echo "Pruning commits for unrelated files..." +git filter-branch \ + --index-filter "$pruner" \ + --parent-filter "$set_roots" \ + --commit-filter "$skip_empty" \ + --tag-name-filter cat \ + -- --all + +# Generate the new .gitreview file +echo "Generating new .gitreview file..." +cat > .gitreview <