diff --git a/doc/source/images/Hook-Aggregated-Report.png b/doc/source/images/Hook-Aggregated-Report.png new file mode 100644 index 00000000..f7564364 Binary files /dev/null and b/doc/source/images/Hook-Aggregated-Report.png differ diff --git a/doc/source/images/Hook-Per-Hook-Report.png b/doc/source/images/Hook-Per-Hook-Report.png new file mode 100644 index 00000000..7bfc523d Binary files /dev/null and b/doc/source/images/Hook-Per-Hook-Report.png differ diff --git a/doc/source/images/Hook-Results.png b/doc/source/images/Hook-Results.png new file mode 100644 index 00000000..cc9306bd Binary files /dev/null and b/doc/source/images/Hook-Results.png differ diff --git a/doc/source/plugins/implementation/hook_and_trigger_plugins.rst b/doc/source/plugins/implementation/hook_and_trigger_plugins.rst new file mode 100644 index 00000000..59c2e30f --- /dev/null +++ b/doc/source/plugins/implementation/hook_and_trigger_plugins.rst @@ -0,0 +1,402 @@ +.. + Copyright 2016 Mirantis Inc. All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +.. _hook_and_trigger_plugins: + + +Hooks. Hook trigger plugins +=========================== + + +Why Hooks? +---------- + +All Rally workloads repeat their actions as many times as it is configured by +runner. Once run, there is no way to interupt the runner to evaluate any change +or restart event on the stability of the cloud under test. +For example we would like to test how configuration change or cloud component +restart would affect performance and stability. + +Task hooks were added to fill this gap and allow to use Rally for reliability +and high availability testing. Generally, hooks allow to perform any +actions on specified iteration or specified time since the workload has been +started. + +Also, task html-report provides results of hook execution. They can contain +graphical or textual information with timing and statistics. + + +Hooks & Triggers Overview +------------------------- + + +Architecture +^^^^^^^^^^^^ + +Rally uses runners to specify how many times the workload should be executed. +Hooks do not use runners, instead they rely on trigger plugins to specify when +and how many times hook should be called. Therefore hooks are isolated from +workload runners and do not affect them because each hook is executed in +separate thread. + + +Sample of usage +^^^^^^^^^^^^^^^ + +Hooks can be added to the task configuration. Lets take a look at hook +configuration: + +.. code-block:: json + + { + "name": "sys_call", + "args": "/bin/echo 123", + "trigger": { + "name": "event", + "args": { + "unit": "iteration", + "at": [5, 50, 200, 1000] + } + } + } + + +It specifies hook plugin with name "sys_call". "args" field contains string +that will be used by sys_call plugin, but in case of any other hook plugin it +can contain any other Python object, that is assumed to be passed to the hook. +"trigger" field specifies which trigger plugin should be used to run this hook. +"trigger" contains similar fields "name" and "args" which represent trigger +plugin name and arguments for trigger plugin. In this example "event" trigger +is specified and configured to run the hook at 5th, 50th, 200th and 1000th +iterations. + +Here is a full task config that contains previous hook configuraiton: + +.. code-block:: json + + { + "Dummy.dummy": [ + { + "args": { + "sleep": 0.01 + }, + "runner": { + "type": "constant", + "times": 1500, + "concurrency": 1 + }, + "hooks": [ + { + "name": "sys_call", + "args": "/bin/echo 123", + "trigger": { + "name": "event", + "args": { + "unit": "iteration", + "at": [5, 50, 200, 1000] + } + } + } + ] + } + ] + } + + +.. note:: + In this example, runner is configured to run workload 1500 times. So there + is a limit for iterations and hook will be triggered only if certain + iteration is started by runner. In other words, if trigger specifies + iteration out of runner iterations scope then such trigger will not be + called. + +Task report for this example will contain minimal information about hook +execution: duration of each hook call and its status(success of failure). + +Let's take a look at more complicated config that can produce graphical +and textual information. + +.. code-block:: yaml + + --- + Dummy.dummy: + - + args: + sleep: 0.75 + runner: + type: "constant" + times: 20 + concurrency: 2 + hooks: + - name: sys_call + description: Run script + args: sh rally/rally-jobs/extra/hook_example_script.sh + trigger: + name: event + args: + unit: iteration + at: [2, 5, 8, 13, 17] + - name: sys_call + description: Show time + args: date +%Y-%m-%dT%H:%M:%S + trigger: + name: event + args: + unit: time + at: [0, 2, 5, 6, 9] + - name: sys_call + description: Show system name + args: uname -a + trigger: + name: event + args: + unit: iteration + at: [2, 3, 4, 5, 6, 8, 10, 12, 13, 15, 17, 18] + sla: + failure_rate: + max: 0 + + +hook_example_script.sh generates dummy output in JSON format. Grafical +information format is the same as for workloads and the same types of +charts are supported for the hooks. + +Here is a report that shows aggregated table and chart with hook results: + +.. image:: ../../images/Hook-Aggregated-Report.png + +Here is report that shows lines chart and pie chart for first hook on +the second iteration: + +.. image:: ../../images/Hook-Per-Hook-Report.png + +Browse existing Hooks_ and Triggers_. + + +Writing your own Hook plugin +---------------------------- + +Problem description +^^^^^^^^^^^^^^^^^^^ + +Hook plugin should implement custom action that can be done one or multiple +times during the workload. Examples of such actions might be the following: + +- Destructive action inside cloud (`Fault Injection`_) +- Getting information about current state of cloud (load/health) +- Upgrading/downgrading a component of cloud +- Changing configuration of cloud +- etc. + + +Plugin code +^^^^^^^^^^^ + +The following example shows simple hook code that performs system call. +It is inherited from the base *Hook* class and contains implemented ``run()`` +method: + +.. code-block:: python + + import shlex + import subprocess + + from rally import consts + from rally.task import hook + + + @hook.configure(name="simple_sys_call") + class SimpleSysCallHook(hook.Hook): + """Performs system call.""" + + CONFIG_SCHEMA = { + "$schema": consts.JSON_SCHEMA, + "type": "string", + } + + def run(self): + proc = subprocess.Popen(shlex.split(self.config), + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT) + proc.wait() + if proc.returncode: + self.set_error( + exception_name="n/a", # no exception class + description="Subprocess returned {}".format(proc.returncode), + details=proc.stdout.read(), + ) + +Any exceptions risen during execution of ``run`` method will be caught by Hook +base class and saved as a result. Although hook should manually call +``Hook.set_error()`` to indicate logical error in case if there is no exception +raised. + +Also there is a method for saving charts data: ``Hook.add_output()``. + + +Plugin Placement +^^^^^^^^^^^^^^^^ + +There are two folders for hook plugins: + +- `OpenStack Hooks`_ +- `Common Hooks`_ + + +Sample of task that uses Hook +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: json + + { + "Dummy.dummy": [ + { + "args": { + "sleep": 0.01 + }, + "runner": { + "type": "constant", + "times": 10, + "concurrency": 1 + }, + "hooks": [ + { + "name": "simple_sys_call", + "args": "/bin/echo 123", + "trigger": { + "name": "event", + "args": { + "unit": "iteration", + "at": [3, 6] + } + } + } + ] + } + ] + } + + +Results of task execution +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Result of previous task example: + +.. image:: ../../images/Hook-Results.png + + +Writing your own Trigger plugin +------------------------------- + + +Problem description +^^^^^^^^^^^^^^^^^^^ + +Trigger plugin should implement an event processor that decides whether to +start hook or not. Rally has two basic triggers that should cover most cases: + +- `Event Trigger`_ +- `Periodic Trigger`_ + + +Plugin code +^^^^^^^^^^^ + +This example shows the code of the existing Event trigger: + +.. code-block:: python + + from rally import consts + from rally.task import trigger + + + @trigger.configure(name="event") + class EventTrigger(trigger.Trigger): + """Triggers hook on specified event and list of values.""" + + CONFIG_SCHEMA = { + "type": "object", + "$schema": consts.JSON_SCHEMA, + "oneOf": [ + { + "properties": { + "unit": {"enum": ["time"]}, + "at": { + "type": "array", + "minItems": 1, + "uniqueItems": True, + "items": { + "type": "integer", + "minimum": 0, + } + }, + }, + "required": ["unit", "at"], + "additionalProperties": False, + }, + { + "properties": { + "unit": {"enum": ["iteration"]}, + "at": { + "type": "array", + "minItems": 1, + "uniqueItems": True, + "items": { + "type": "integer", + "minimum": 1, + } + }, + }, + "required": ["unit", "at"], + "additionalProperties": False, + }, + ] + } + + def get_listening_event(self): + return self.config["unit"] + + def on_event(self, event_type, value=None): + if not (event_type == self.get_listening_event() + and value in self.config["at"]): + # do nothing + return + super(EventTrigger, self).on_event(event_type, value) + + +Trigger plugins must override two methods: + +- ``get_listening_event`` - this method should return currently configured + event name. (So far Rally supports only "time" and "iteration") +- ``on_event`` - this method is called each time certain events occur. + It calls base method when the hook is triggered on specified event. + + +Plugin Placement +^^^^^^^^^^^^^^^^ + +All trigger plugins should be placed in `Trigger folder`_. + + +.. references: + +.. _Hooks: ../plugin_reference.html#task-hooks +.. _Triggers: ../plugin_reference.html#task-hook-triggers +.. _Fault Injection: ../plugin_reference.html#fault-injection-hook +.. _Event Trigger: ../plugin_reference.html#event-hook-trigger +.. _Periodic Trigger: ../plugin_reference.html#periodic-hook-trigger +.. _Common Hooks: https://github.com/openstack/rally/tree/master/rally/plugins/common/hook +.. _OpenStack Hooks: https://github.com/openstack/rally/tree/master/rally/plugins/openstack/hook +.. _Trigger folder: https://github.com/openstack/rally/tree/master/rally/plugins/common/trigger diff --git a/doc/specs/implemented/hook_plugins.rst b/doc/specs/implemented/hook_plugins.rst index c80d6c88..f2c6b4fa 100644 --- a/doc/specs/implemented/hook_plugins.rst +++ b/doc/specs/implemented/hook_plugins.rst @@ -41,7 +41,7 @@ of hook plugins that should be executed on this iteration. "args": { "cmd": "bash enable_factor_1" }, - "trigger: { + "trigger": { "name": "event", "args": { "unit": "time", @@ -54,7 +54,7 @@ of hook plugins that should be executed on this iteration. "args": { "cmd": "bash enable_factor_2" }, - "trigger: { + "trigger": { "name": "event", "args": { "unit": "iteration", @@ -67,7 +67,7 @@ of hook plugins that should be executed on this iteration. "args": { "cmd": "bash enable_factor_3" }, - "trigger: { + "trigger": { "name": "periodic", "args": { "unit": "iteration", @@ -82,7 +82,7 @@ of hook plugins that should be executed on this iteration. "args": { "cmd": "bash enable_factor_4" }, - "trigger: { + "trigger": { "name": "periodic", "args": { "unit": "time",