64d34ae594
Current cleanup mechanism is awful. It's hardcoded, without repeat on failures, contains a lot of mistakes, in some places it is too fast (in case of deletion of VMs) in other to slow (like deletion of users). As well there is mix of cleanup mechanism and resource cleanup mechanism. To resolve these all issues this patch introcude a cleanup engine that resolves all issues above. It's resource based, so to add new resouce you should just a make subclass of base.Resource and probably override some of methods like (list, delete, is_deleted) and that's all. All complexity of managing: 0) waiting until async deletion is finished 1) repeat on failure logic 2) greaceful failure handling 3) parallelization 4) plugin support Is hidden deep inside cleanup engine + bonus we are able to specify now single resource (without clenaping whole service) PART 3: ------- *) Refactor all contexts cleanups method to use new generic cleanup engine insted of cleanup.utils *) Remove obsolate cleanup.utils *) Fix all tests bp benchmark-context-cleanup-refactor Change-Id: I70557e6ebb56bbe565792d9ee854d3e78428a881
1007 B
Production read cleanups
Use Case
Rally should delete in any case all resources that it created during benchmark.
Problem Description
(implemented) Deletion rate limit
You can kill cloud by deleting too many objects simultaneously, so deletion rate limit is required
(implemented) Retry on failures
There should be few attempts to delete resource in case of failures
(implemented) Log resources that failed to be deleted
We should log warnings about all non deleted resources. This information should include UUID of resource, it's type and project.
(implemented) Pluggable
It should be simple to add new cleanups adding just plugins somewhere.
Disaster recovery
Rally should use special name patterns, to be able to delete resources in such case if something went wrong with server that is running rally. And you have just new instance (without old rally db) of rally on new server.