openstack
/
qa-specs
Code Issues Proposed changes

Add a user-isolated cleanup option to tempest cleanup 

Add an option that allows for concurrent testing and a more functional
option for tempest cleanup

Change-Id: Id4d04040e53b9ca56f04485c996917b4c3e452c2
This commit is contained in:
nh202b 2018-08-22 10:58:59 -07:00 committed by Nicolas Helgeson
parent 09de20ea67
commit 26bd2d00bb
1 changed files with 191 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

191
specs/tempest/smart-cleanup.rst Normal file
View File

@ -0,0 +1,191 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
..
=============================
Tempest User-Isolated Cleanup
=============================
https://blueprints.launchpad.net/tempest/+spec/tempest-smart-cleanup
The `tempest cleanup`_ command allows a user to delete resources based on a
saved state json file that contains a list of `global resource ids`_ (Domain,
Flavor, Image, Project, Role, and User) and names. Additional
`project service`_ resources are cleaned up by association with project ids.
Running the cleanup command will delete all global resources and associated
project service resources not listed in the saved state json file.
.. note:: Tempest cleanup runs using a `configuration file`_ or CONF. This file
contains a selection of admin resources that are excluded from the basic
cleanup function. Adding the "--delete-tempest-conf-objects" argument will
include these admin resources
.. _tempest cleanup: https://docs.openstack.org/tempest/16.1.0/cleanup.html
.. _global resource ids: https://opendev.org/openstack/tempest/src/commit/ecddd38ecbdb5df05543f106e10e14a9e7d0d3da/tempest/cmd/cleanup_service.py#L954
.. _configuration file: https://opendev.org/openstack/tempest/src/commit/ecddd38ecbdb5df05543f106e10e14a9e7d0d3da/tempest/config.py
.. _project service: https://opendev.org/openstack/tempest/src/commit/ecddd38ecbdb5df05543f106e10e14a9e7d0d3da/tempest/cmd/cleanup_service.py#L925
Problem description
===================
Tempest cleanup is a one-size-fits-all approach which deletes all resources
that are not listed in the save_state.json. If multiple tests are being run,
multiple saved_states are required and, in turn, creates issues:
* Creates potential to clean up more than expected
* Example 1: User A saves state, user B adds a resource, user A runs tests, user A cleans up
after tests, user B's resource gets cleaned up because it does not appear in
the saved_state.json.
* Initializing a saved state while other tests are running will incorporate outside resources.
* Example 2: User A saves state, user A runs tests, user B saves state,
user B runs tests, user A cleans up - Some test A elements are in the saved state from user
B, and cleanup removes resources created in test B.
* Multiple CONF files can result in admin resources being deleted
* Example 3: Two users are running two different CONF files and their admin resources do not
overlap. There is potential for one cleanup command to remove the other users admin resources.
New proposed change / Solution
==============================
Create an alternate cleanup function that uses a list of resources to be deleted.
Essentially the inverse of the current function. By including the ``--target <'file'>``
parameter a user will be able to supply a list of resources that need to be
deleted and the function will double check against any existing saved_state file
before specifically deleting only the resources listed in the target file. If a
resource is present in both saved_state and target files, and error will be thrown,
and no resources will be deleted. Additionally, any test run will automatically
record created resources into a file labeled with the time and date of the
test. This feature will be on by default, but can be turned off using the
``--test-record-off`` parameter when running tests. Include support for ``--dry-run``
to provide functionality that can help track down resource leaks.
The list will automatically be generated by adding a logging function to the client
create function and a log cleanup function to the client delete function. The scope
of this is pretty large, but we can reduce it by focusing on the same resources
that the save-state function tracks: domains, flavors, images, projects, roles and users.
Example:
.. code-block:: bash
$ tempest cleanup --target <'file'>
$ tempest run --test-record-off
Additional consideration: We could also add code to the innate cleanup function
of the test so that if a resource is properly deleted as part of the test, it
will be removed from the created file. If the file is found to be empty, it will
be deleted. This will help track resource leaks by the presence or absence of the
file. The file will be labeled with time and date, so it can be
tracked back to the test run.
Previously proposed change / Solution
=====================================
Restore the `resources_prefix`_ using `rand_name`_ in order to track individual
test runs. Add code that allows for targeted deletion of these named resources
so that individual test runs can be specifically deleted without interacting
with other test runs. Include support for ``--dry-run`` to provide
functionality that can help track down resource leaks. Once implemented, a user
will be able to run:
.. code-block:: bash
$ tempest cleanup --prefix-delete <'prefix'>
and remove specific test resources without interfering with any other tests.
As a positive side effect, any resource leaks will be traceable back to the
origin test run.
.. _resources_prefix: https://docs.openstack.org/releasenotes/tempest/unreleased.html#relnotes-16-0-0-deprecation-notes
.. _rand_name: https://github.com/openstack/tempest/blob/9594233ffd93066677aa601c594d55f3a4f188d8/tempest/lib/common/utils/data_utils.py#L49
Alternatives
------------
One possible alternative that has been suggested is that we create separate
domains for each user. This will enable each user to cleanup their own
resources based on their specific domains. In this way, no other user's
resources will be affected.
* Requires each test to be modified to use the test domain
* Cleanup function will have to be modified to clean up based on domains
* May have an issue with the pre-provisioned credential provider. The users will need to be updated
in each test case to be in the test domain and a safeguard must be put in place so that there is
not an attempt to delete.
Projects
========
Tempest
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Nicolas Helgeson <nhelgeson>
Milestones
----------
Target Milestone for completion:
Train-3
Work Items
----------
* Add functionality that deletes specific resources based on file contents
* Delete only specified resources
* Do not delete any resource in saved_state
* Include support for ``--dry-run``
* Add functionality that records into a time stamped file all created resources
* All resource create functions will add to file
* Option: All delete functions will remove from file
* Update tempest cleanup CLI with new parameters
* Add option to cleanup based on file contents
* tempest cleanup ``--target <'file'>``
* Include support for ``--dry-run`` to help identify resources leaks
* Update tempest run CLI with new parameters
* Add option to not record tests
* tempest run ``--test-record-off``
* Add documentation
* For test-record-off command
* For target cleanup command
Dependencies
============
This has an internal dependency on ensuring that all resource names are
created using data_utils.rand_name().
References
===========
* https://etherpad.openstack.org/p/tempest-cleanup-feature