Add spec for bp:post-run-cleanup
Made changes according to comments provided by community Removed database access Added reporting detail Added pluggability suggestion Added example of how deletes work when --preserve-state is true Added --dry-run argument definition Added --init-saved-state argument definition Change-Id: Ice4bdc3d6059c0549e7b648dc187830abbf89a61
This commit is contained in:
David Paterson
137
specs/post-run-cleanup.rst
Normal file
137
specs/post-run-cleanup.rst
Normal file
@@ -0,0 +1,137 @@
|
|||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
========================
|
||||||
|
Tempest Post-run Cleanup
|
||||||
|
========================
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/tempest/+spec/post-run-cleanup
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
====================
|
||||||
|
The existing script, /tempest/stress/cleanup.py, can be used to do some
|
||||||
|
basic cleanup after a Tempest run, but a more robust tool is needed
|
||||||
|
to report as much information as possible about dangling objects (leaks)
|
||||||
|
left behind after a tempest run in an attempt to help find out why the
|
||||||
|
object(s) was left behind and report a bug against the root cause. Also
|
||||||
|
the tool should completely reset the environment to the pre-run state
|
||||||
|
should tempest leave behind any dangling objects.
|
||||||
|
|
||||||
|
The idea is that the user should be able look at the report generated
|
||||||
|
and find the root cause as to why an object was not deleted. Also the
|
||||||
|
tool will return the system back into a state where tempest can be
|
||||||
|
re-run with the expectation that the same test results will be returned.
|
||||||
|
|
||||||
|
Currently there can be a good deal of manual work needed, depending
|
||||||
|
on what tests fail, to return to this pre-run state. This blueprint
|
||||||
|
is designed to alleviate this issue.
|
||||||
|
|
||||||
|
Proposed changes
|
||||||
|
================
|
||||||
|
- Keep /tempest/stress/cleanup.py as a starting point and extend it.
|
||||||
|
It should be moved to from /tempest/stress to tempest/cmd/ and
|
||||||
|
an entry point should be added for it as well. This way it is installed
|
||||||
|
as a binary when setup.py is run, which also allows it to be unit
|
||||||
|
tested. Currently the tools uses the tempest OpenStack clients and
|
||||||
|
this will remain unchanged.
|
||||||
|
- Fix cleanup.py to delete objects by looping through each tenant/user,
|
||||||
|
over the way it currently works, which is to use the admin user and
|
||||||
|
"all_tenants" argument, as some object types don't support this
|
||||||
|
argument, Floating IPs for example.
|
||||||
|
- Currently cleanup.py deletes all objects across all users/tenants.
|
||||||
|
Add two runtime arguments: --init-saved-state, that creates a JSON
|
||||||
|
file containg the pre-tempest run state and --preserve-state, that
|
||||||
|
will preserve the deployment's pre-tempest run state, including tenants
|
||||||
|
and users defined in tempest.conf. This will enforce that the
|
||||||
|
deployment is in the same state it was prior to running tempest and
|
||||||
|
allow tempest to be run again without having to reconfigure tempest
|
||||||
|
and recreate the tempest test users etc. For example, if
|
||||||
|
--preserve-state is true cleanup will load the JSON file (created by
|
||||||
|
running cleanup with --init-saved-state flag prior to tempest run)
|
||||||
|
containing the preserved state of the environment and marshal the data
|
||||||
|
to some defined instance variables. Then, when cleanup is looping
|
||||||
|
through floating ips we would have something like:
|
||||||
|
|
||||||
|
for f in floating_ips:
|
||||||
|
if not preserve or (preserve and f['id'] not in self.floating_ips):
|
||||||
|
try:
|
||||||
|
admin_manager.floating_ips_client.delete_floating_ip(f['id'])
|
||||||
|
except Exception:
|
||||||
|
...
|
||||||
|
|
||||||
|
- cleanup.py currently deletes servers (instances), keypairs,
|
||||||
|
security groups, floating ips, users, tenants, snapshots and volumes.
|
||||||
|
It should also delete any stacks, availability zones and any other objects
|
||||||
|
created by Tempest, full list TBD.
|
||||||
|
- As mentioned in the overview section above some test failures leave the
|
||||||
|
system in a strange state. For example, an instance cannot be deleted
|
||||||
|
because it is in Error state. Even after using CLI to reset the instance
|
||||||
|
to Active state, future delete calls just result in Error state once
|
||||||
|
again. Such a case indicates a bug in OpenStack. This tool should
|
||||||
|
should provide as much detail as possible as to what went wrong so
|
||||||
|
a defect can be opened against the problem(s).
|
||||||
|
- Add argument, --dry-run, that runs cleanup in reporting mode only, showing
|
||||||
|
what would be deleted without doing the actual deletes
|
||||||
|
|
||||||
|
Scenario 1: run cleanup.py
|
||||||
|
--------------------------
|
||||||
|
This is the current behavior, which deletes all objects in the system,
|
||||||
|
with the exception of the missing ones, stacks and availability zones
|
||||||
|
for example.
|
||||||
|
|
||||||
|
Scenario 2: run cleanup.py --preserve-state
|
||||||
|
-------------------------------------------
|
||||||
|
Same as Scenario 1 except that objects defined in tempest.conf, that
|
||||||
|
are used in a Tempest run are preserved.
|
||||||
|
|
||||||
|
For example (exceptions are variables defined in tempest.conf):
|
||||||
|
|
||||||
|
- delete all users except: username, alt_username, admin_username
|
||||||
|
- delete all tenants except: tenant_name, alt_tenant_name, admin_tenant_name
|
||||||
|
- delete all images except: image_ref, image_ref_alt
|
||||||
|
|
||||||
|
Additional Implications
|
||||||
|
-----------------------
|
||||||
|
There are cases where cruft will be left in the database do to openstack defects
|
||||||
|
that don't allow objects to be removed during the cleanup process.
|
||||||
|
In such cases resetting the system to the pre-existing state requires direct
|
||||||
|
interaction with the database. It may be useful to design the cleanup script
|
||||||
|
so that it has a pluggable interface, where downstream functionality can be
|
||||||
|
added to automate required database interactions for example. Although
|
||||||
|
the API delete failure indicates an upstream bug that needs to be fixed, until
|
||||||
|
that bug is fixed testing the environment further is blocked until the records
|
||||||
|
are deleted.
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
Primary assignee:
|
||||||
|
David Paterson <davpat2112@yahoo.com>
|
||||||
|
|
||||||
|
Can optionally can list additional ids if they intend on doing
|
||||||
|
substantial implementation work on this blueprint.
|
||||||
|
|
||||||
|
Milestones
|
||||||
|
----------
|
||||||
|
Target Milestone for completion:
|
||||||
|
|
||||||
|
- Juno release cycle, approximately the week of July 24th, 2014.
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
- refactor location of cleanup.py
|
||||||
|
- register new runtime arguments in cleanup.py
|
||||||
|
- enable filtering deletions based on --preserve-state argument
|
||||||
|
and values defined in tempest.conf
|
||||||
|
- write code for detailed reporting on dangling resources and
|
||||||
|
possible root cause for cleanup failure.
|
||||||
|
- implement code for --dry-run argument, report only mode.
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
Only those listed above
|
||||||
|
|
||||||
Reference in New Issue
Block a user