.. This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode .. This template should be in ReSTructured text. The filename in the git repository should match the launchpad URL, for example a URL of https://blueprints.launchpad.net/heat/+spec/awesome-thing should be named awesome-thing.rst . Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None For help with syntax, see http://sphinx-doc.org/rest.html To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html ==================================== Make the new Rally input task format ==================================== Current Rally format is not flexible enough to cover all use cases that are required. Let's change it! Problem description =================== Why do we need such fundamental change? - Multi scenarios load generation support. This is very important, because it will allow to use Rally for more real life load generation. Like making load on different components and HA testing (where one scenario tries for example to authenticate another is disabling controller) - Ability to add require meta information like (title and descriptions) That are required to generate clear reports - Fixing UX issues. Previous format is very hard for understanding and end users have issues with understanding how it works exactly. Proposed change =============== Make a new format that address all issues. Old format JSON schema: .. code-block:: python { "type": "object", "$schema": "http://json-schema.org/draft-04/schema", "patternProperties": { ".*": { "type": "array", "items": { "type": "object", "properties": { "args": { "type": "object" }, "runner": { "type": "object", "properties": { "type": {"type": "string"} }, "required": ["type"] }, "context": { "type": "object" }, "sla": { "type": "object", }, }, "additionalProperties": False } } } } Old format sample: .. code-block:: yaml --- : - args: runner: context: : ... sla: : - -//- - -//- : -//- Every element of list corresponding to is separated task, that generates environment according to context, generates load using specified runner that runs multiple times with it's args. New format JSON schema: .. code-block:: python { "type": "object", "$schema": "http://json-schema.org/draft-04/schema", "properties": { "version": {"type": "number"}, "title": {"type": "string"}, "description": {"type": "string"}, "tags": { "type": "array", "items": {"type": "string"} }, "subtasks": { "type": "array", "items": { "type": "object", "properties": { "title": {"type": "string"}, "description": {"type": "string"}, "tags": { "type": "array", "items": {"type": "string"} }, "run_in_parallel": {"type": "boolean"}, "workloads": { "type": "array", "items": { "type": "object", "properties": { "scenario": {"type": "object"}, "runner": {"type": "object"} "slas": {"type": "object"}, "contexts": {"type": "object"} }, "required": ["scenario", "runner"] } }, "context": {"type": "object"} }, "required": ["title", "workloads"] } } }, "required": ["title", "tasks"] } New format sample: .. code-block:: yaml --- # Having Dictionary on top level allows us in future to add any new keys. # Keeping the schema of format more or less same for end users. # Version of format version: 1 # Allows to set title of report. Which allows end users to understand # what they can find in task report. title: "New Input Task format" # Description allows us to put all required information to explain end # users what kind of results they can find in reports. description: "This task allows you to certify that your cloud works" # Explicit usage "rally task start --tag" --tag attribute tags: ["periodic", "nova", "cinder", "ha"] subtasks: # Note every task is executed serially (one by one) # # Using list for describing what subtasks to run is much better idea then # using dictionary. It resolves at least 3 big issues: # # 1) Bad user experience # 1.1) Users do not realize that Rally can run N subtask # 1.2) Keys of Dictionary were Scenario names (reasonable question why?!) # 1.3) Users tried to put N times same k-v (to run one subtask N times) # 2) No way to specify order of scenarios execution, especially in case # where we need to do chain like: ScenarioA -> SecnearioB -> ScenarioA # 3) No way to support multi scenario load, because we used scenario name # as a identifier of single task - # title field is required because in case of multi scenario load # we can't use scenario name for it's value. title: "First task to execute" description: "We will stress Nova" # optional # Tags are going to be used in various rally task reports for filtering # and grouping. tags: ["nova", "my_favorite_task", "do it"] # The way to execute scenarios (one by one or all in parallel) run_in_parallel: False # Single scenario load can be generated by specifying only one element # in "workloads" section. workloads: - scenario: NovaServers.boot_and_delete: image: name: "^cirros$" flavors: name: "m1.small" runner: constant: times: 100 concurrency: 10 # Subtask success of criteria based on results slas: # Every key means SLA plugin name, values are config of plugin # Only if all criteria pass task is marked as passed failure_rate: max: 0 # Specification of context that creates env for scenarios # E.g. it creates users, tenants, sets quotas, uploads images... contexts: # Each key is the name of context plugin # This context creates temporary users and tenants users: # These k-v will be passed as arguments to this `users` plugin tenants: 2 users_per_tenant: 10 # This context set's quotas for created by `users` context tenants quotas: nova: cpu: -1 - title: "Second task to execute" description: "Multi Scenario load generation with common context" run_in_parallel: True # If we put 2 or more scenarios to `scenarios` section we will run # all of them simultaneously which allows us to generate more real life # load workloads: - scenario: CinderVolumes.create_and_delete: size: 10 runner: constant: times: 100 concurrency: 10 sla: failure_rate: max: 0 - scenario: KeystoneBasic.create_and_delete_users: name_length: 20 runner: rps: rps: 1 times: 1000 slas: max_seconds_per_iteration: 10 - scenario: PhysicalNode.restart: ip: "..." user: "..." password: "..." runner: rps: rps: 10 times: 10 slas: max_seconds_per_iteration: 100 # This scenario is called in own independent and isolated context contexts: {} # Global context that is used if scenario doesn't specify own contexts: users: tenants: 2 users_per_tenant: 10 Alternatives ------------ No way Implementation ============== Assignee(s) ----------- Primary assignee: boris-42 aka Boris Pavlovic Work Items ---------- - Implement OLD -> NEW format converter - Switch task engine to use new format. This should affect only task engine - Implement new DB schema format, that will allow to store multi-scenario output data - Add support for multi scenario results processing in rally task detailed|sla_check|report - Add timestamps to task, scenarios and atomics - Add support for usage multi-runner instance in single task with common context - Add support for scenario's own context - Add ability to use new format in rally task start. - Deprecate OLD format Dependencies ============ None