diff --git a/doc/release_notes/archive/v0.10.0.rst b/doc/release_notes/archive/v0.10.0.rst new file mode 100644 index 00000000..96dc29d3 --- /dev/null +++ b/doc/release_notes/archive/v0.10.0.rst @@ -0,0 +1,520 @@ +============= +Rally v0.10.0 +============= + +Overview +-------- + ++------------------+-----------------------+ +| Release date | **10/20/2017** | ++------------------+-----------------------+ + +* Ability to use OpenStack deployments without admin credentials +* Better validation process of tasks +* New task format (with an ability to set description for workloads) +* New JSON report +* ElasticSearch exporter +* `OSProfiler support + `_ +* Restructure of configuration options. + +Details +------- + +Command Line Interface +~~~~~~~~~~~~~~~~~~~~~~ + +* Introduce `rally task import + `_ + command for importing task results into database. + +* Extend tags support for tasks. Now you can specify several tags for a single + task using `--tag argument + `_. + Also filtering tasks by tags is now available. + +* Move DB category from ``rally-manage db`` to `rally db + `_ and + introduce `rally db show + `_ + command for printing the used connection string. + +Deployments +~~~~~~~~~~~ + +This release we started a huge work related to simplification of deployment +component of Rally. There is a good progress which includes several nice +features: + +* The format. + "ExistingCloud" deployment type covers 99.99% cases and is used as a base for + all new things. Also, it will be extended to support different platforms + soon. The new format looks like (for OpenStack case): + + .. code-block:: json + + { + "openstack": { + "admin": { + "username": "admin", + "password": "changeme", + "tenant_name": "foo", + }, + "auth_url": "https://example.com", + } + } + + +* admin user is optional in case of setting existing users. + From the beginning, setting admin credentials was a required section of Rally + deployment configuration. Even with introducing existing users feature, this + behaviour left. + Finally, we finished a big code refactoring and admin credential become + optional section. If a set of plugins for particular workload doesn't require + admin user, you can launch this task at deployment without setting it. + + The information about the requirements of plugins you can find at + `Plugins Reference page + `_ (see + ``Requires platform(s):`` section at the bottom of each plugin). + +* Originally, Rally project was designed to check performance of OpenStack and + we succeeded in building awesome tool. We do not plan to stop and just want + to inform about our future plans to expand a number of supported platforms. + Subscribe to our `GitHub organization + `_ to not miss new plugins. + +Task component +~~~~~~~~~~~~~~ + +* The new task format is introduced. It includes a bunch of improvements, + unification, etc. All the docs and samples will be updated soon. + + As for now, you can check `a spec + `_ + for this big change. + +* SLA failure_rate max=0 become a default if nothing else is specified. + +* Totally reworked atomic actions. The atomic actions now supports nested + actions which allows to measure durations inside the scenario even more + precise. You can find them in HTML report or in our new json report + (see ``rally task report --json``). + +* Generation of names for new resources takes care about particular workload + id, so it helps to provide a better cleanup and prepare for new feature - + disaster cleanup. + +Plugins +~~~~~~~ + +We started supporting discovering plugins by entry-points, so you can easily +deliver your custom plugins as a simple python package. + +To make you package after-discoverable, you need to specify the proper +entry-point at your setup.cfg file: + + .. code-block:: + + rally_plugins = + path=package_name + +**Deployment Engines**: + +Remove serverproviders & rarely used deployers + +Unfortunately, seems like nobody is using deployers for deploying +their clouds and mostly people would like just to execute their code. + +1) Remove server provides +2) Remove engines that uses server providers + +**OpenStack clients**: + +* Deprecate EC2 client. It wasn't used in any of plugins and doesn't support + keystone v3 + +* Move ``rally.osclients`` module to ``rally.plugins.openstack.oscliens`` + +**Scenarios**: + +The old way to describe scenario plugin via method is finally removed. +Initially Rally scenario plugins were methods of special class, like below: + + .. code-block:: python + + from rally.task import scenario + + class SomeBasicClass(scenario.Scenario): + + @scenario.configure() + def some_scenario(self, arg1): + """An implementation of SomeBasicClass.foo scenario.""" + + @scenario.configure() + def another_scenario(self): + """Implementation of another scenario, SomeBasicClass.bar.""" + +However to unify scenarios with other plugins we moved to model where +plugin is class. It was done long time ago. + + .. code-block:: python + + from rally.task import scenario + + @scenario.configure(name="CustomName") + class Some(scenario.Scenario): + + def run(self, arg1): + """An implementation of the scenario.""" + +We had a bunch of code that was used for transition and backward compatibility +that we finally removed. + + +* *NEW!!* + + - `CinderQos.create_and_get_qos + `_ + + - `CinderQos.create_and_list_qos + `_ + + - `CinderQos.create_and_set_qos + `_ + + - `CinderQos.create_qos_associate_and_disassociate_type + `_ + + - `CinderVolumeTypes.create_and_get_volume_type + `_ + + - `CinderVolumeTypes.create_and_list_volume_types + `_ + + - `CinderVolumeTypes.create_and_update_encryption_type + `_ + + - `CinderVolumeTypes.create_and_update_volume_type + `_ + + - `CinderVolumeTypes.create_get_and_delete_encryption_type + `_ + + - `CinderVolumeTypes.create_volume_type_add_and_list_type_access + `_ + + - `Dummy.openstack + `_ + + - `GlanceImages.create_and_deactivate_image + `_ + + - `GlanceImages.create_and_download_image + `_ + + - `GlanceImages.create_and_get_image + `_ + + - `GlanceImages.create_and_update_image + `_ + + - `K8sPods.create_pods + `_ + + - `K8sPods.create_rcs + `_ + + - `K8sPods.list_pods + `_ + + - `ManilaShares.create_and_extend_share + `_ + + - `ManilaShares.create_and_shrink_share + `_ + + - `ManilaShares.create_share_then_allow_and_deny_access + `_ + + - `NeutronBGPVPN.create_and_delete_bgpvpns + `_ + + - `NeutronBGPVPN.create_and_list_bgpvpns + `_ + + - `NeutronBGPVPN.create_and_list_networks_associations + `_ + + - `NeutronBGPVPN.create_and_list_routers_associations + `_ + + - `NeutronBGPVPN.create_and_update_bgpvpns + `_ + + - `NeutronBGPVPN.create_bgpvpn_assoc_disassoc_networks + `_ + + - `NeutronBGPVPN.create_bgpvpn_assoc_disassoc_routers + `_ + + - `NeutronNetworks.create_and_show_ports + `_ + + - `NeutronNetworks.create_and_show_routers + `_ + + - `NeutronNetworks.create_and_show_subnets + `_ + + - `NeutronNetworks.set_and_clear_router_gateway + `_ + + - `NeutronSecurityGroup.create_and_delete_security_group_rule + `_ + + - `NeutronSecurityGroup.create_and_list_security_group_rules + `_ + + - `NeutronSecurityGroup.create_and_show_security_group + `_ + + - `NeutronSecurityGroup.create_and_show_security_group_rule + `_ + + - `NovaServerGroups.create_and_delete_server_group + `_ + + - `NovaServerGroups.create_and_get_server_group + `_ + + - `NovaServers.boot_and_get_console_url + `_ + + - `NovaServers.boot_server_and_attach_interface + `_ + + - `NovaServers.boot_server_and_list_interfaces + `_ + + - `NovaServers.boot_server_attach_volume_and_list_attachments + `_ + +* *UPDATED!!* + + - The new argument ``properties`` is added to scenario + `IronicNodes.create_and_list_node + `_ + + +* *DELETED* + +Fuel and Nova-Network are not alive any more. So we removed those scenarios. +If any of those scenarios a critical for you, please contact us. + + - `FuelEnvironments.create_and_delete_environment + `_ + + - `FuelEnvironments.create_and_list_environments + `_ + + - `FuelNodes.add_and_remove_node + `_ + + - `NovaFloatingIpsBulk.create_and_delete_floating_ips_bulk + `_ + + - `NovaFloatingIpsBulk.create_and_list_floating_ips_bulk + `_ + + - `NovaNetworks.create_and_delete_network + `_ + + - `NovaNetworks.create_and_list_networks + `_ + + - `NovaSecGroup.boot_and_delete_server_with_secgroups + `_ + + - `NovaSecGroup.boot_server_and_add_secgroups + `_ + + - `NovaSecGroup.create_and_delete_secgroups + `_ + + - `NovaSecGroup.create_and_list_secgroups + `_ + + - `NovaSecGroup.create_and_update_secgroups + `_ + +**Validators**: + +The validators refactoring was a long-term task which blocked us to abandon +alignment to only OpenStack platform and requirements of setting admin +credentials. In this release, we made a great progress and fixed a lot of +issues and blockers which made possible to refactor validators. +Now validation step is equal for all types of plugins (Scenario, SLA, Context, +Hooks, Runners, etc). + +The old way to add validator for scenario is deprecated. The new unified way +looks like: + + .. code-block:: python + + import time + + from rally.common import validation + from rally.task import scenario + + @validation.add("number", param_name="timeout", minval=0) + @scenario.configure(name="Foo.bar") + class FooScenario(scenario.Scenario): + def run(self, timeout): + time.sleep() + +The old validators from ``rally.task.validators`` module is deprecated too, see +equivalents which can be used with ``add`` decorator: + + - required_openstack --> `required_platform + `_ + with setting platform argument to "openstack" + + - external_network_exists ->`external_network_exists + `_ + + - file_exists ->`file_exists + `_ + + - flavor_exists ->`flavor_exists + `_ + + - image_exists ->`image_exists + `_ + + - image_valid_on_flavor ->`image_valid_on_flavor + `_ + + - number ->`number + `_ + + - required_api_versions ->`required_api_versions + `_ + + - required_cinder_services ->`required_cinder_services + `_ + + - required_clients ->`required_clients + `_ + + - required_contexts ->`required_contexts + `_ + + - required_neutron_extensions ->`required_neutron_extensions + `_ + + - required_param_or_context ->`required_param_or_context + `_ + + - required_services ->`required_services + `_ + + - restricted_parameters ->`restricted_parameters + `_ + + - validate_heat_template ->`validate_heat_template + `_ + + - volume_type_exists ->`volume_type_exists + `_ + + - workbook_contains_workflow -> `workbook_contains_workflow + `_ + + - network_exists is removed, since we do not find any customers for it. + Please contact us if it was useful for you. + + - validate_share_proto is removed in favor of enum validator + +Fixed bugs +~~~~~~~~~~ + +* [plugins] JSON schema of `servers + `_ + context allows to transmit a list of nics in two formats. First one is a + format that novaclient expects to see (each network should be represented + like ``{"nic-id": "the id of the network"}``). The second one is more + user-friendly - just list of strings (each network should be represented + just by id of the network). Unfortunately, the second case was not covered + by our code base. + + Also, the first described format works in limited cases due to bad + serialization. + + `Launchpad bug-report #1695245 + `_ + +* [deployment] ~/rally/.openrc not working for keystone v3 + + `Launchpad bug-report #1683820 + `_ + +* [plugins] Failed to list volumes in case of missed name in the object. + +* [backported into 0.9.1][deployment] Credentials is not updated as soon as + deployment is recreated. Need to call recreate request twice. + + `Launchpad bug-report #1675271 + `_ + +* [backported into 0.9.1][plugins] Scenario `IronicNodes.create_and_list_node + `_ + had a wrong check that list of all nodes contains newly created one. + +* [backported into 0.9.1][task][cleanup] Do not remove quotas in case of + existing users + +* [backported into 0.9.1][task][cleanup] Various traces of neutron resources + +* [backported into 0.9.1][core] Keystone v3, authentication error for Rally + users if the value of project_domain_name of admin user isn't equal "default" + + `Launchpad bug-report #1680837 + `_ + +* [backported into 0.9.1][task] Scenario `NovaHosts.list_and_get_hosts + `_ + obtains hostname for all hosts. But it fails in some environments if host is + not compute. + + `Launchpad bug-report #1675254 + `_ + +* [backported into 0.9.1][verification] Rally fails to run on systems on which + python-virtualenv is not installed + + `Launchpad bug-report #1678047 + `_ + +* [backported into 0.9.1][verification] CLI `rally verify rerun + `_ + fails with TypeError due to wrong integration with Rally API. + +* [plugins] Rally fails while creating neutron router on the clouds where + ext-gw-mode extension is not installed. + +* [plugins] Scenario `CinderVolumes.create_nested_snapshots_and_attach_volume + `_ + fails on a big load due to the fact that one server is used for several + iterations. In such case we are facing 2 issues: the maximum number of + volumes per VM is 26 (which is a number of available names for volumes); + detaching volume of one iteration can block attaching of other iterations. + + `Launchpad bug-report #1708160 + `_ + + +Thanks +~~~~~~ + + 2 Everybody! diff --git a/doc/release_notes/latest.rst b/doc/release_notes/latest.rst index 7ff2b9be..763dd522 120000 --- a/doc/release_notes/latest.rst +++ b/doc/release_notes/latest.rst @@ -1 +1 @@ -./archive/v0.9.2.rst \ No newline at end of file +./archive/v0.10.0.rst \ No newline at end of file diff --git a/doc/specs/implemented/improve_atomic_actions_format.rst b/doc/specs/implemented/improve_atomic_actions_format.rst new file mode 100644 index 00000000..0993f686 --- /dev/null +++ b/doc/specs/implemented/improve_atomic_actions_format.rst @@ -0,0 +1,157 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +.. + This template should be in ReSTructured text. The filename in the git + repository should match the launchpad URL, for example a URL of + https://blueprints.launchpad.net/heat/+spec/awesome-thing should be named + awesome-thing.rst . Please do not delete any of the sections in this + template. If you have nothing to say for a whole section, just write: None + For help with syntax, see http://sphinx-doc.org/rest.html + To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html + + +============================================= +New Atomic actions format in workload results +============================================= + +Currently atomic actions data in workload results is insufficient, +therefore some new features can not be implemented. + +Problem description +=================== + +The main problem is that current format does not support nested +atomic actions. + +Also, atomic actions data does not include timestamps for each action +start and end time. Having this data will allow us to inspect atomic +actions runtime better and generate detailed reports. + +Since word "atomic" means something that can not be split into parts +and we introduce nested atomic actions, we should use different term +instead of "atomic actions". + +Proposed change +=============== + +Term "atomic actions" should be renamed to just "actions". + +Change actions results schema from type "object" to "array" +and extend it with timestamps and nested actions. + +Nested actions will be represented by "children" key and have +unlimited nesting. + +With timestamps, there is no need to save durations anymore, +so get rid of this value. + +Since this change is not backward compatible, we need to create +a database migration script. The migration will use iteration start +timestamp as start timestamp for first action and then calculate +further timestamps based on actions order and their durations. + +Benefits of new format +---------------------- + +Nested actions will make actions measurement more detailed and flexible +since we could have data what sub-actions were run during specific action +runtime, without complicated changes at code. + +Start and end timestamps will provide us with accurate information +about action runtime within the whole iteration and ability to create +`Gantt charts `_. + +Schema modification +------------------- + +Schema location is *rally.common.objects.task.TASK_RESULT_SCHEMA +["properties"]["result"]["properties"]["atomic_actions"]* + +should be moved to *rally.common.objects.task.TASK_RESULT_SCHEMA +["properties"]["result"]["properties"]["actions"]* + +and changed: + +AS IS: + +.. code-block:: python + + { + "type": "object" + } + +Here keys are actions names, and values are their durations. +Actions data is actually represented by collections.OrderedDict, +so we have real order saved. + +Example: + +.. code-block:: python + + OrderedDict([("keystone.create_tenant", 0.1234), + ("keystone.create_users", 1.234)]) + +TO BE: + +.. code-block:: python + + { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": {"type": "string"}, # name of action + "started_at": {"type": "number"}, # float UNIX timestamp + "finished_at": {"type": "number"}, # float UNIX timestamp + "children": {"$ref": "#/"}, + }, + "required": ["name", "started_at", "finished_at", "children"], + "additionalProperties": False + }, + "minItems": 0 + } + +Example how this data can be represented: + +.. code-block:: python + + [{"name": "keystone.create_tenant", + "started_at": 1455281370.288397, + "finished_at": 1455281372.672342, + "children": []}, + {"name": "keystone.create_users", + "started_at": 1455281372.931324, + "finished_at": 1455281373.375184, + "children": []}] + +Alternatives +------------ + +None + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + Alexander Maretskiy + + +Work Items +---------- + + - Rename atomic actions into actions + - Improve actions results format + - Create a DB migartion that transforms results to new format + +Dependencies +============ + +None diff --git a/doc/specs/implemented/new_rally_input_task_format.rst b/doc/specs/implemented/new_rally_input_task_format.rst new file mode 100644 index 00000000..65c8eaa8 --- /dev/null +++ b/doc/specs/implemented/new_rally_input_task_format.rst @@ -0,0 +1,348 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +.. + This template should be in ReSTructured text. The filename in the git + repository should match the launchpad URL, for example a URL of + https://blueprints.launchpad.net/heat/+spec/awesome-thing should be named + awesome-thing.rst . Please do not delete any of the sections in this + template. If you have nothing to say for a whole section, just write: None + For help with syntax, see http://sphinx-doc.org/rest.html + To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html + + +==================================== +Make the new Rally input task format +==================================== + +Current Rally format is not flexible enough to cover all use cases that are +required. Let's change it! + + +Problem description +=================== + + Why do we need such fundamental change? + +- Multi scenarios load generation support. + This is very important, because it will allow to use Rally for more + real life load generation. Like making load on different components + and HA testing (where one scenario tries for example to authenticate + another is disabling controller) + +- Ability to add require meta information like (title and descriptions) + That are required to generate clear reports + +- Fixing UX issues. Previous format is very hard for understanding and + end users have issues with understanding how it works exactly. + + +Proposed change +=============== + +Make a new format that address all issues. + + +Old format JSON schema: + +.. code-block:: python + + { + "type": "object", + "$schema": "http://json-schema.org/draft-04/schema", + "patternProperties": { + ".*": { + "type": "array", + "items": { + "type": "object", + "properties": { + "args": { + "type": "object" + }, + "runner": { + "type": "object", + "properties": { + "type": {"type": "string"} + }, + "required": ["type"] + }, + "context": { + "type": "object" + }, + "sla": { + "type": "object", + }, + }, + "additionalProperties": False + } + } + } + } + + +Old format sample: + +.. code-block:: yaml + + --- + : + - + args: + runner: + context: + : + ... + sla: + : + - + -//- + - + -//- + : + -//- + + Every element of list corresponding to is separated task, + that generates environment according to context, generates load using + specified runner that runs multiple times with it's args. + + +New format JSON schema: + +.. code-block:: python + + { + "type": "object", + "$schema": "http://json-schema.org/draft-04/schema", + "properties": { + "version": {"type": "number"}, + "title": {"type": "string"}, + "description": {"type": "string"}, + "tags": { + "type": "array", + "items": {"type": "string"} + }, + + "subtasks": { + "type": "array", + "items": { + "type": "object", + "properties": { + "title": {"type": "string"}, + "description": {"type": "string"}, + "tags": { + "type": "array", + "items": {"type": "string"} + }, + + "run_in_parallel": {"type": "boolean"}, + "workloads": { + "type": "array", + "items": { + "type": "object", + "properties": { + "scenario": {"type": "object"}, + "runner": {"type": "object"} + "sla": {"type": "object"}, + "contexts": {"type": "object"} + }, + "required": ["scenario", "runner"] + } + }, + "contexts": {"type": "object"} + }, + "required": ["title", "workloads"] + } + } + }, + "required": ["title", "tasks"] + } + + +New format sample: + +.. code-block:: yaml + + --- + + # Having Dictionary on top level allows us in future to add any new keys. + # Keeping the schema of format more or less same for end users. + + # Version of format + version: 2 + + # Allows to set title of report. Which allows end users to understand + # what they can find in task report. + title: "New Input Task format" + + # Description allows us to put all required information to explain end + # users what kind of results they can find in reports. + description: "This task allows you to certify that your cloud works" + + # Explicit usage "rally task start --tag" --tag attribute + tags: ["periodic", "nova", "cinder", "ha"] + + subtasks: + # Note every task is executed serially (one by one) + # + # Using list for describing what subtasks to run is much better idea then + # using dictionary. It resolves at least 3 big issues: + # + # 1) Bad user experience + # 1.1) Users do not realize that Rally can run N subtask + # 1.2) Keys of Dictionary were Scenario names (reasonable question why?!) + # 1.3) Users tried to put N times same k-v (to run one subtask N times) + # 2) No way to specify order of scenarios execution, especially in case + # where we need to do chain like: ScenarioA -> SecnearioB -> ScenarioA + # 3) No way to support multi scenario load, because we used scenario name + # as a identifier of single task + - + # title field is required because in case of multi scenario load + # we can't use scenario name for it's value. + title: "First task to execute" + description: "We will stress Nova" # optional + + # Tags are going to be used in various rally task reports for filtering + # and grouping. + tags: ["nova", "my_favorite_task", "do it"] + + # The way to execute scenarios (one by one or all in parallel) + run_in_parallel: False + + # Single scenario load can be generated by specifying only one element + # in "workloads" section. + workloads: + - + scenario: + NovaServers.boot_and_delete: + image: + name: "^cirros$" + flavors: + name: "m1.small" + runner: + constant: + times: 100 + concurrency: 10 + # Subtask success of criteria based on results + sla: + # Every key means SLA plugin name, values are config of plugin + # Only if all criteria pass task is marked as passed + failure_rate: + max: 0 + + # Specification of context that creates env for scenarios + # E.g. it creates users, tenants, sets quotas, uploads images... + contexts: + # Each key is the name of context plugin + + # This context creates temporary users and tenants + users: + # These k-v will be passed as arguments to this `users` plugin + tenants: 2 + users_per_tenant: 10 + + # This context set's quotas for created by `users` context tenants + quotas: + nova: + cpu: -1 + + - + title: "Second task to execute" + description: "Multi Scenario load generation with common context" + + run_in_parallel: True + + # If we put 2 or more scenarios to `scenarios` section we will run + # all of them simultaneously which allows us to generate more real life + # load + workloads: + - + scenario: + CinderVolumes.create_and_delete: + size: 10 + runner: + constant: + times: 100 + concurrency: 10 + sla: + failure_rate: + max: 0 + - + scenario: + KeystoneBasic.create_and_delete_users: + name_length: 20 + runner: + rps: + rps: 1 + times: 1000 + sla: + max_seconds_per_iteration: 10 + - + scenario: + PhysicalNode.restart: + ip: "..." + user: "..." + password: "..." + runner: + rps: + rps: 10 + times: 10 + sla: + max_seconds_per_iteration: 100 + # This scenario is called in own independent and isolated context + contexts: {} + + # Global context that is used if scenario doesn't specify own + contexts: + users: + tenants: 2 + users_per_tenant: 10 + + +Alternatives +------------ + +No way + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + boris-42 aka Boris Pavlovic + + +Work Items +---------- + +- Implement OLD -> NEW format converter + +- Switch task engine to use new format. This should affect only task engine + +- Implement new DB schema format, that will allow to store multi-scenario + output data + +- Add support for multi scenario results processing in rally task + detailed|sla_check|report + +- Add timestamps to task, scenarios and atomics + +- Add support for usage multi-runner instance in single task with + common context + +- Add support for scenario's own context + +- Add ability to use new format in rally task start. + +- Deprecate OLD format + + +Dependencies +============ + +None diff --git a/doc/specs/implemented/pluggable_validators.rst b/doc/specs/implemented/pluggable_validators.rst new file mode 100644 index 00000000..c4329df6 --- /dev/null +++ b/doc/specs/implemented/pluggable_validators.rst @@ -0,0 +1,206 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +.. + This template should be in ReSTructured text. The filename in the git + repository should match the launchpad URL, for example a URL of + https://blueprints.launchpad.net/heat/+spec/awesome-thing should be named + awesome-thing.rst . Please do not delete any of the sections in this + template. If you have nothing to say for a whole section, just write: None + For help with syntax, see http://sphinx-doc.org/rest.html + To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html + + +================================= +Rally Task Validation Refactoring +================================= + +Problem description +=================== + +* Current validator system is pluggable - but it doesn't use our plugin + mechanism which creates problems (e.g. validators are imported directly and + used in code, instead of using their names, which doesn't allow to rename + them or move without breaking backward compatibility). + +* Current mechanism of validation leads to a lot of OpenStack related code in + the Rally task engine. + +* It's hard to use the same validators for different types of plugins, current + approach is used only for scenarios. + +Proposed change +=============== + +To create unified validation mechanism that can be used for all types of +future deployments and type of plugins in the same way. So we will be able +to remove `OpenStack related code `_ from the task engine, and create a bunch of +common validators (e.g. jsonschema) that can be used by any +plugin. +As a bonus of refactoring, it allows us to switch to common mechanism of +plugins. + +Alternatives +------------ + +No way + + +Implementation +============== + +Here is an example of base class for all pluggable validators. + +.. code-block:: python + + import abc + import six + + from rally.common.plugin import plugin + from rally.task import validation + + + def configure(name, platform="default"): + return plugin.configure(name=name, platform=platform) + + @six.add_metaclass(abc.ABCMeta) + @configure(name="base_validator") + class Validator(plugin.Plugin): + + def validate(self, cache, deployment, cfg, plugin_cfg): + """ + Method that validates something. + + :param cache: this is cross validator cache where different + validators could store information about + environment like initialized OpenStack clients, + images, etc and share it through validators. + E.g. if your custom validators need to perform 200 + OpenStack checks and each validator plugin need to + initialize client, Rally will take extra 2 minutes + for validation step. As well, its not efficient to + fetch all image each time if we have image related + validators. + :param deployment: Deployment object, deployment which would be + used for validation + :param cfg: dict, configuration of subtask + :param plugin_cfg: dict, with exact configuration of the plugin + """ + pass + + def add(name, **kwargs): + """ + Add validator instance to the validator plugin class meta. + + Get validator class by name. Initialize an instance. Add validator + instance to validators list stored in the Validator meta by + 'validator_v2' key. This would be used to iterate and execute through + all validators used during execution of subtask. + + :param kwargs: dict, arguments used to initialize validator class + instance + :param name: str, name of the validator plugin + """ + validator = Validator.get(name)(**kwargs) + + def wrapper(p): + p._meta_setdefault("validators_v2", []) + p._meta_get("validators_v2").append(validator) + return p + + return wrapper + + + @abc.abstractmethod + def validate(plugin, deployment, cfg, plugin_cfg): + """ + Execute all validate() method of all validators stored in meta of + Validator. + + Iterate during all validators stored in the meta of Validator and + execute proper validate() method and add validation result to the + list. + + :param plugin: is plugin class instance that has validators and should + be validated + :param deployment: Deployment object, deployment which would be + used for validation + :param cfg: dict, configuration of subtask + :param plugin_cfg: dict, with exact configuration of the plugin + """ + results = [] + cache = {} + + for v in plugin._meta_get("validators_v2"): + try: + v.validate(cache, deployment, cfg, plugin_cfg) + except Exception as e: + results.append(validation.ValidationResult(is_valid=False, + msg=e)) + return results + + +New design allows us to use the same validator and same validation mechanism +for different types of plugins (context, sla, runner, scenarios) which was not +possible before. For example, we could implement jsonschema validation as a +plugin. + +.. code-block:: python + + import jsonschema + + @configure(name="jsonschema") + class JsonSchemaValidator(Validator): + + def __init__(self, schema=None): + super(JsonSchemaValidator, self).__init__() + self.schema = schema or {} + + def validate(self, cache, deployment, cfg, plugin_cfg): + jsonschema.validate(plugin_cfg, self.schema) + + + + @validator.add("jsonschema", schema="") + class SomeContext(base.Context): + pass + + + class SomeScenario(base.Scenario): + + @validator.add("jsonschema", schema="") + def some_function(self): + pass + + +Assignee(s) +----------- + +Primary assignee: + +- boris-42 +- rvasilets + +Work Items +---------- + +- Create validation module with base plugin and method of adding validators + +- Add support to task engine of new validation mechanism + +- Port all old validators to new mechanism + +- Deprecate old validation mechanism + +- Remove deprecated in new release + + +Dependencies +============ + +None diff --git a/doc/specs/implemented/task_and_verification_export.rst b/doc/specs/implemented/task_and_verification_export.rst new file mode 100644 index 00000000..9cc7a724 --- /dev/null +++ b/doc/specs/implemented/task_and_verification_export.rst @@ -0,0 +1,109 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +.. + This template should be in ReSTructured text. The filename in the git + repository should match the launchpad URL, for example a URL of + https://blueprints.launchpad.net/heat/+spec/awesome-thing should be named + awesome-thing.rst . Please do not delete any of the sections in this + template. If you have nothing to say for a whole section, just write: None + For help with syntax, see http://sphinx-doc.org/rest.html + To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html + + +==================================================== +Export task and verifications into external services +==================================================== + +Currently Rally stores all information about executed tasks and verifications +in its database and it is also able to provide this data in JSON format or +in the form of HTML reports. There is a request for Rally to export this data +into external services (like test management system or Google Docs) +via its API. + +Problem description +=================== + +There are many, including a lot of proprietary, test management systems +in the market available as SaaS and/or On-Premises, like TestRail, TestLink, +TestLodge etc, which objective is to manage, organize and track all testing +efforts. + +Most of the systems provide an API for importing test data. The systems also +possess data model somewhat similar to Rally's one. +It usually includes (among others) models for project, test suite test case, +test plan and test execution results. + +It is suggested to provide Rally users an ability to export information about +testing their environments into such test management systems in order +to integrate testing via Rally into rest of their testing activities. + +Since different test management systems have alike yet different API +for the purpose it is reasonable to implement this export functionality via +plugins. + +Proposed change +=============== + +1. Implement a base class Exporter for an export plugin at +*rally/task/exporter.py*. + +..code-block:: python + + class Exporter(plugin.Plugin): + def export(self, task, connection_string): + ... + +2. Implement a CLI command of the form + +..code-block:: shell + + rally task export + +3. Implement a base class VerifyExporter for an export plugin at +*rally/verify/exporter.py*. + +..code-block:: python + + class VerifyExporter(plugin.Plugin): + def export(self, verification, connection_string): + ... + +4. Implement a CLI command of the form + +..code-block:: shell + + rally verify export + +Alternatives +------------ + +No way + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + +rvasilets + +Work Items +---------- + +- Implement plugin base class + +- Implement CLI command + +- Implement plugin for TestRail + +Dependencies +============ + +None