16 KiB
Split Tempest Plugins into Separate Repos/Projects
Tempest plugins rely on setuptools entrypoints and therefore can be included in any python project as long as it properly sets up the entrypoint. A common pattern has emerged of bundling these plugins inside a project repo. This is likely because it makes contributing tests at the same time as an API feature must simpler. However, this is really an antipattern and makes consuming the test plugins much more difficult, especially in all-in-one deployments. In order to make using plugins more consistent and easier for end users, packagers, and deployers this goal is to make sure we always use a separate python project for each plugin.
There are several issues with the current situation including:
1. When you deploy an all in one system with multiple services that have bundled tempest plugins and then run tempest outside of a venv tempest will see the tests for all the plugins whether you intended to use them or not. It is also a fairly common occurrence for a project to ship a broken plugin that breaks at import time. This prevents unittest discovery from working which will block any tests from running.
2. A packager tries to use a stable version of your project with a newer tempest. This results in conflicting requirements because tempest is branchless and follows master requirements and makes using stable release plugins with master tempest impossible.
3. Most projects don't actually use the same tests across release boundaries (or just don't use plugins on stable branches), which means breaking api changes can land. There are 2 exceptions to this trove and ironic which we hacked together a mechanism to install the project repo from master in the tempest venv.
4. Plugin requirements aren't exposed at install time. Some projects use test-requirements to set tempest plugin requirements (since it is for tests) which do not get installed when you pip install a project. However the entrypoint always gets exposed and tempest will pick it up regardless of whether all the requirements are installed.
Having tempest plugins in a separate repo/project solves all of these
issues and makes using a plugin opt-in in all cases. (ie you have to
pip install plugin-foo
to use it and won't show up as an
unexpected side effect of installing a service) For community
development and testing it means we are actually ensuring API
compatibility across releases. It also means for packagers that you
don't have to install all of a project repo to ship a plugin
package.
Champion
Goals need a main driver to project-manage them to completion. Project teams need assistance, reminders and sometimes direct help in order for them to complete the goals.
Chandan Kumar (chandankumar) has volunteered to drive this goal.
Gerrit Topic
To facilitate tracking, commits related to this goal should use the gerrit topic:
goal-split-tempest-plugins
Completion Criteria
For all projects with a bundled tempest plugin:
- Create a new separate repo for the tempest plugin
- Migrate all the functionality from the bundled plugin to the new repo
- Switch gating jobs to use the new plugin project instead of the bundled one
- Delete the bundled tempest plugin from the project repo
Note, there is no change in governance or ownership of a repository involved with this change. The new plugin repo/project will still be owned by the project it was spun out from.
More detailed recommended steps for accomplishing the split can be found here:
https://etherpad.openstack.org/p/tempest-separate-plugin
References
The tempest documentation elaborates on why separate plugins are a better pattern:
https://docs.openstack.org/tempest/latest/plugin.html#standalone-plugin-vs-in-repo-plugin
The following links may be useful to developers attempting to create their tempest plugin project:
- https://www.openstack.org/videos/boston-2017/tempest-plugin-crash-course
- https://etherpad.openstack.org/p/tempest-separate-plugin
- https://etherpad.openstack.org/p/tempest-plugin-testing
- https://etherpad.openstack.org/p/pike-tempest-plugin-dev
Current State / Anticipated Impact
The following script was used to find all the openstack projects with tempest plugins.
import json
import re
import sys
import requests
import yaml
= 'https://review.openstack.org/projects/'
url with open(sys.argv[1], 'r') as fd:
= yaml.load(fd.read())
projects_list
= []
repo_list for i in projects_list:
= projects_list[i]['deliverables']
deliverables = [x['repos'] for x in deliverables.values()]
repo_paths for repos in repo_paths:
for name in repos:
repo_list.append(name)
# This is what a project looks like
'''
"openstack-attic/akanda": {
"id": "openstack-attic%2Fakanda",
"state": "READ_ONLY"
},
'''
def is_openstack_proj(proj):
= False
res if proj in repo_list:
= True
res return res
# Rather than returning a 404 for a nonexistent file, cgit delivers a
# 0-byte response to a GET request. It also does not provide a
# Content-Length in a HEAD response, so the way we tell if a file exists
# is to check the length of the entire GET response body.
def has_tempest_plugin(proj):
= requests.get(
r "https://git.openstack.org/cgit/%s/plain/setup.cfg" % proj)
= re.compile('^tempest\.test_plugins', re.M)
p if p.findall(r.text):
return True
else:
False
= requests.get(url)
r # Gerrit prepends 4 garbage octets to the JSON, in order to counter
# cross-site scripting attacks. Therefore we must discard it so the
# json library won't choke.
= sorted(filter(is_openstack_proj, json.loads(r.text[4:])))
projects
= filter(has_tempest_plugin, projects)
found_plugins
# Every element of the found_plugins list begins with "openstack/".
# We drop those initial 10 octets when printing the list.
for project in found_plugins:
print(project)
On 03 Jan 2017 this was run and found all the projects with bundled tempest plugins. 4 matches were found that are standalone plugins: barbican-tempest-plugin, tempest-horizon, designate-tempest-plugin, and sahara-tests (this is a multitest repo but it's not bundled with the project so it's not applicable here either) Also all the deb packaging repos were removed as they are duplicates. This list is:
openstack/aodh
openstack/ceilometer
openstack/cinder
openstack/congress
openstack/ec2-api
openstack/freezer
openstack/freezer-api
openstack/gnocchi
openstack/heat
openstack/ironic
openstack/ironic-inspector
openstack/keystone
openstack/magnum
openstack/manila
openstack/mistral
openstack/monasca-api
openstack/monasca-log-api
openstack/murano
openstack/networking-bgpvpn
openstack/networking-midonet
openstack/networking-sfc
openstack/neutron
openstack/neutron-dynamic-routing
openstack/neutron-fwaas
openstack/neutron-lbaas
openstack/octavia
openstack/senlin
openstack/tricircle
openstack/trove
openstack/vitrage
openstack/watcher
openstack/zaqar
openstack/zun
Project Teams
barbican
Planning Artifacts:
The barbican project does not have in-tree tempest plugin.
Completion Artifacts:
The barbican team is maintaining it's tempest plugin in a seperate repo:
http://git.openstack.org/cgit/openstack/barbican-tempest-plugin
Chef OpenStack
Planning Artifacts:
- The Chef OpenStack team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
cinder
Planning Artifacts:
Completion Artifacts:
cloudkitty
Planning Artifacts:
- The cloudkitty team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
Community App Catalog
Planning Artifacts:
Completion Artifacts:
congress
Planning Artifacts:
Completion Artifacts:
designate
Planning Artifacts:
Completion Artifacts:
The Designate team was already compliant before that the goal was defined. Here is the tempest plugin repo:
http://git.openstack.org/cgit/openstack/designate-tempest-plugin
Documentation
Planning Artifacts:
- The Documentation team does not maintain tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
dragonflow
Planning Artifacts:
- The dragnonflow team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
ec2-api
Planning Artifacts:
Completion Artifacts:
freezer
Planning Artifacts:
Completion Artifacts:
fuel
Planning Artifacts:
Completion Artifacts:
glance
Planning Artifacts:
- The glance team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
heat
Planning Artifacts:
Completion Artifacts:
horizon
Planning Artifacts:
Completion Artifacts:
The Horizon team was already compliant before that the goal was defined. Here is the tempest plugin repo:
http://git.openstack.org/cgit/openstack/tempest-horizon
I18n
Planning Artifacts:
- The I18n team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
Infrastructure
Planning Artifacts:
- The Infrastructure team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable
ironic
Planning Artifacts:
Completion Artifacts:
karbor
Planning Artifacts:
- The karbor team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
keystone
Planning Artifacts:
The keystone team followed the documented steps outlined in this goal as the planning document.
Completion Artifacts:
The following review removed the tempest plugin that keystone was maintaining in tree and links to the new repository the plugin was moved to:
http://git.openstack.org/cgit/openstack/keystone/commit/?id=6f4e37e9e6810e24f45d034261f4a6ec4aa85fb1
kolla
Planning Artifacts:
- The kolla team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
kuryr
Planning Artifacts:
The kuryr project does not have in-tree tempest plugin.
Completion Artifacts:
The kuryr team is maintaining it's tempest plugin in a seperate repo:
http://git.openstack.org/cgit/openstack/kuryr-tempest-plugin
magnum
Planning Artifacts:
Completion Artifacts:
manila
Planning Artifacts:
Completion Artifacts:
mistral
Planning Artifacts:
Completion Artifacts:
monasca
Planning Artifacts:
Completion Artifacts:
murano
Planning Artifacts:
Completion Artifacts:
neutron
Planning Artifacts:
Completion Artifacts:
nova
Planning Artifacts:
- The nova team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
octavia
Planning Artifacts:
Completion Artifacts:
OpenStack Charms
Planning Artifacts:
- The OpenStack Charms team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
OpenStack UX
Planning Artifacts:
Completion Artifacts:
OpenStackAnsible
Planning Artifacts:
- The OpenStackAnsible team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- The OpenStack Ansible Deployment tool is ready to handle the installation and configuration of tempest plugins.
OpenStackClient
Planning Artifacts:
- The OpenStackClient team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
oslo
Planning Artifacts:
- The oslo team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable
Packaging-deb
Planning Artifacts:
- The Packaging-deb team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
Packaging-rpm
Planning Artifacts:
Completion Artifacts:
Puppet OpenStack
Planning Artifacts:
- The Puppet OpenStack team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable
Quality Assurance
Planning Artifacts:
Completion Artifacts:
rally
Planning Artifacts:
- The rally team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable
RefStack
Planning Artifacts:
- The RefStack team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
Release Management
Planning Artifacts:
- The Release Management team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
requirements
Planning Artifacts:
Completion Artifacts:
sahara
Planning Artifacts:
Completion Artifacts:
The Sahara team was already compliant before that the goal was defined. Here is the commit link for the same:
http://git.openstack.org/cgit/openstack/sahara/commit/?id=83a6a2868377dd61530a9de80c6ca49061c5f248
searchlight
Planning Artifacts:
Completion Artifacts:
Security
Planning Artifacts:
- The Security team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
senlin
Planning Artifacts:
Completion Artifacts:
shade
Planning Artifacts:
- The shade team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- None
solum
Planning Artifacts:
The solum project does not have in-tree tempest plugin.
Completion Artifacts:
The solum team is maintaining it's tempest plugin in a seperate repo:
http://git.openstack.org/cgit/openstack/solum-tempest-plugin
Stable branch maintenance
Planning Artifacts:
- The Stable branch maintenance team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
storlets
Planning Artifacts:
- The storlets team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
swift
Planning Artifacts:
Completion Artifacts:
tacker
Planning Artifacts:
Completion Artifacts:
Telemetry
Planning Artifacts:
Completion Artifacts:
tricircle
Planning Artifacts:
- The tricircle team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable.
tripleo
Planning Artifacts:
- The tripleo team does not have tempest plugins and therefore has nothing to do.
Completion Artifacts:
- Not applicable
trove
Planning Artifacts:
Completion Artifacts:
vitrage
Planning Artifacts:
Completion Artifacts:
watcher
Planning Artifacts:
The watcher team followed the documented steps outlined in this goal as the planning document.
Completion Artifacts:
The following review removed the tempest plugin that watcher was maintaining in tree and links to the new repository the plugin was moved to:
http://git.openstack.org/cgit/openstack/watcher/commit/?id=0c4b439c5ea1206263f39c118daf6d2ff1422480
winstackers
Planning Artifacts:
The os-win project does not have in-tree tempest plugin.
Completion Artifacts:
The Winstackers team is maintaining its tempest plugin in a seperate repo:
http://git.openstack.org/cgit/openstack/oswin-tempest-plugin
zaqar
Planning Artifacts:
Completion Artifacts:
zun
Planning Artifacts:
Completion Artifacts: