*) Change task config format
. Split "context" & "runner" stuff
*) Refactor Verification
. Move validation to context.base, runner.base and scenario.base
. Validate whole config fully before starting any of tasks
. Optimize scenario args validation (create only one time clients)
. Optimize order of validation:
1) Validate names of benchmarks
2) Validate all static parameters, e.g. configuration of runner
and context
3) If everything is ok in all benchmarks, then start validation
of scenario args.
. Store validation result (exception) in task["verification_log"]
. Remove verification logic from BenchmarkEngine.__exit__
. Remove scenario args verification results from task["results"]
*) Fix & Swtich to new format doc/samples/tasks
. Switch to new fromat
. Add missing task configratuion
. Better formatting
. json & yaml samples
*) Refactored unit tests
. tests.rally.benchmark.test_engine
. tests.rally.benchmark.context.base
. tests.orcestrator.test_api.start_task
cover validation step as well and new change format
*) Refactor orchestrator api start task
. Remove benchmark engine context
. Call verify explicity
. Do not raise any excpetion in case of validation error
. Catch in start task any unexcepted Exceptions a set deployment
in incosistance state
*) Refactor CLI
. Properly handle new behaviour of verification
. Replace table on task start to just message
. Add HINTs to task detailed command
*) Add unit test for checking doc samples
*) Improve benchmark engine LOGing
blueprint benchmark-new-task-config
Change-Id: I23d3f6b3439fdb44946a7c2491d5a9b3559dc671
d7635776d7