[doc] Add documentation for hook and trigger plugins

update and fix json errors in hook_plugins.rst

Change-Id: I8b9d7b40048fdcae2d997a0a757f226d3487d297

doc/source/plugins/implementation/hook_and_trigger_plugins.rst

@@ -0,0 +1,402 @@
+..
+      Copyright 2016 Mirantis Inc. All Rights Reserved.
+
+      Licensed under the Apache License, Version 2.0 (the "License"); you may
+      not use this file except in compliance with the License. You may obtain
+      a copy of the License at
+
+          http://www.apache.org/licenses/LICENSE-2.0
+
+      Unless required by applicable law or agreed to in writing, software
+      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+      License for the specific language governing permissions and limitations
+      under the License.
+
+.. _hook_and_trigger_plugins:
+
+
+Hooks. Hook trigger plugins
+===========================
+
+
+Why Hooks?
+----------
+
+All Rally workloads repeat their actions as many times as it is configured by
+runner. Once run, there is no way to interupt the runner to evaluate any change
+or restart event on the stability of the cloud under test.
+For example we would like to test how configuration change or cloud component
+restart would affect performance and stability.
+
+Task hooks were added to fill this gap and allow to use Rally for reliability
+and high availability testing. Generally, hooks allow to perform any
+actions on specified iteration or specified time since the workload has been
+started.
+
+Also, task html-report provides results of hook execution. They can contain
+graphical or textual information with timing and statistics.
+
+
+Hooks & Triggers Overview
+-------------------------
+
+
+Architecture
+^^^^^^^^^^^^
+
+Rally uses runners to specify how many times the workload should be executed.
+Hooks do not use runners, instead they rely on trigger plugins to specify when
+and how many times hook should be called. Therefore hooks are isolated from
+workload runners and do not affect them because each hook is executed in
+separate thread.
+
+
+Sample of usage
+^^^^^^^^^^^^^^^
+
+Hooks can be added to the task configuration. Lets take a look at hook
+configuration:
+
+.. code-block:: json
+
+    {
+        "name": "sys_call",
+        "args": "/bin/echo 123",
+        "trigger": {
+            "name": "event",
+            "args": {
+                "unit": "iteration",
+                "at": [5, 50, 200, 1000]
+            }
+        }
+    }
+
+
+It specifies hook plugin with name "sys_call". "args" field contains string
+that will be used by sys_call plugin, but in case of any other hook plugin it
+can contain any other Python object, that is assumed to be passed to the hook.
+"trigger" field specifies which trigger plugin should be used to run this hook.
+"trigger" contains similar fields "name" and "args" which represent trigger
+plugin name and arguments for trigger plugin. In this example "event" trigger
+is specified and configured to run the hook at 5th, 50th, 200th and 1000th
+iterations.
+
+Here is a full task config that contains previous hook configuraiton:
+
+.. code-block:: json
+
+  {
+      "Dummy.dummy": [
+          {
+              "args": {
+                  "sleep": 0.01
+              },
+              "runner": {
+                  "type": "constant",
+                  "times": 1500,
+                  "concurrency": 1
+              },
+              "hooks": [
+                  {
+                      "name": "sys_call",
+                      "args": "/bin/echo 123",
+                      "trigger": {
+                          "name": "event",
+                          "args": {
+                              "unit": "iteration",
+                              "at": [5, 50, 200, 1000]
+                          }
+                      }
+                  }
+              ]
+          }
+      ]
+  }
+
+
+.. note::
+    In this example, runner is configured to run workload 1500 times. So there
+    is a limit for iterations and hook will be triggered only if certain
+    iteration is started by runner. In other words, if trigger specifies
+    iteration out of runner iterations scope then such trigger will not be
+    called.
+
+Task report for this example will contain minimal information about hook
+execution: duration of each hook call and its status(success of failure).
+
+Let's take a look at more complicated config that can produce graphical
+and textual information.
+
+.. code-block:: yaml
+
+  ---
+    Dummy.dummy:
+      -
+        args:
+          sleep: 0.75
+        runner:
+          type: "constant"
+          times: 20
+          concurrency: 2
+        hooks:
+          - name: sys_call
+            description: Run script
+            args: sh rally/rally-jobs/extra/hook_example_script.sh
+            trigger:
+              name: event
+              args:
+                unit: iteration
+                at: [2, 5, 8, 13, 17]
+          - name: sys_call
+            description: Show time
+            args: date +%Y-%m-%dT%H:%M:%S
+            trigger:
+              name: event
+              args:
+                unit: time
+                at: [0, 2, 5, 6, 9]
+          - name: sys_call
+            description: Show system name
+            args: uname -a
+            trigger:
+              name: event
+              args:
+                unit: iteration
+                at: [2, 3, 4, 5, 6, 8, 10, 12, 13, 15, 17, 18]
+        sla:
+          failure_rate:
+            max: 0
+
+
+hook_example_script.sh generates dummy output in JSON format. Grafical
+information format is the same as for workloads and the same types of
+charts are supported for the hooks.
+
+Here is a report that shows aggregated table and chart with hook results:
+
+.. image:: ../../images/Hook-Aggregated-Report.png
+
+Here is report that shows lines chart and pie chart for first hook on
+the second iteration:
+
+.. image:: ../../images/Hook-Per-Hook-Report.png
+
+Browse existing Hooks_ and Triggers_.
+
+
+Writing your own Hook plugin
+----------------------------
+
+Problem description
+^^^^^^^^^^^^^^^^^^^
+
+Hook plugin should implement custom action that can be done one or multiple
+times during the workload. Examples of such actions might be the following:
+
+- Destructive action inside cloud (`Fault Injection`_)
+- Getting information about current state of cloud (load/health)
+- Upgrading/downgrading a component of cloud
+- Changing configuration of cloud
+- etc.
+
+
+Plugin code
+^^^^^^^^^^^
+
+The following example shows simple hook code that performs system call.
+It is inherited from the base *Hook* class and contains implemented ``run()``
+method:
+
+.. code-block:: python
+
+  import shlex
+  import subprocess
+
+  from rally import consts
+  from rally.task import hook
+
+
+  @hook.configure(name="simple_sys_call")
+  class SimpleSysCallHook(hook.Hook):
+      """Performs system call."""
+
+      CONFIG_SCHEMA = {
+          "$schema": consts.JSON_SCHEMA,
+          "type": "string",
+      }
+
+      def run(self):
+          proc = subprocess.Popen(shlex.split(self.config),
+                                  stdout=subprocess.PIPE,
+                                  stderr=subprocess.STDOUT)
+          proc.wait()
+          if proc.returncode:
+              self.set_error(
+                  exception_name="n/a",  # no exception class
+                  description="Subprocess returned {}".format(proc.returncode),
+                  details=proc.stdout.read(),
+              )
+
+Any exceptions risen during execution of ``run`` method will be caught by Hook
+base class and saved as a result. Although hook should manually call
+``Hook.set_error()`` to indicate logical error in case if there is no exception
+raised.
+
+Also there is a method for saving charts data: ``Hook.add_output()``.
+
+
+Plugin Placement
+^^^^^^^^^^^^^^^^
+
+There are two folders for hook plugins:
+
+- `OpenStack Hooks`_
+- `Common Hooks`_
+
+
+Sample of task that uses Hook
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: json
+
+  {
+      "Dummy.dummy": [
+          {
+              "args": {
+                  "sleep": 0.01
+              },
+              "runner": {
+                  "type": "constant",
+                  "times": 10,
+                  "concurrency": 1
+              },
+              "hooks": [
+                  {
+                      "name": "simple_sys_call",
+                      "args": "/bin/echo 123",
+                      "trigger": {
+                          "name": "event",
+                          "args": {
+                              "unit": "iteration",
+                              "at": [3, 6]
+                          }
+                      }
+                  }
+              ]
+          }
+      ]
+  }
+
+
+Results of task execution
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Result of previous task example:
+
+.. image:: ../../images/Hook-Results.png
+
+
+Writing your own Trigger plugin
+-------------------------------
+
+
+Problem description
+^^^^^^^^^^^^^^^^^^^
+
+Trigger plugin should implement an event processor that decides whether to
+start hook or not. Rally has two basic triggers that should cover most cases:
+
+- `Event Trigger`_
+- `Periodic Trigger`_
+
+
+Plugin code
+^^^^^^^^^^^
+
+This example shows the code of the existing Event trigger:
+
+.. code-block:: python
+
+  from rally import consts
+  from rally.task import trigger
+
+
+  @trigger.configure(name="event")
+  class EventTrigger(trigger.Trigger):
+      """Triggers hook on specified event and list of values."""
+
+      CONFIG_SCHEMA = {
+          "type": "object",
+          "$schema": consts.JSON_SCHEMA,
+          "oneOf": [
+              {
+                  "properties": {
+                      "unit": {"enum": ["time"]},
+                      "at": {
+                          "type": "array",
+                          "minItems": 1,
+                          "uniqueItems": True,
+                          "items": {
+                              "type": "integer",
+                              "minimum": 0,
+                          }
+                      },
+                  },
+                  "required": ["unit", "at"],
+                  "additionalProperties": False,
+              },
+              {
+                  "properties": {
+                      "unit": {"enum": ["iteration"]},
+                      "at": {
+                          "type": "array",
+                          "minItems": 1,
+                          "uniqueItems": True,
+                          "items": {
+                              "type": "integer",
+                              "minimum": 1,
+                          }
+                      },
+                  },
+                  "required": ["unit", "at"],
+                  "additionalProperties": False,
+              },
+          ]
+      }
+
+      def get_listening_event(self):
+          return self.config["unit"]
+
+      def on_event(self, event_type, value=None):
+          if not (event_type == self.get_listening_event()
+                  and value in self.config["at"]):
+              # do nothing
+              return
+          super(EventTrigger, self).on_event(event_type, value)
+
+
+Trigger plugins must override two methods:
+
+- ``get_listening_event`` - this method should return currently configured
+  event name. (So far Rally supports only "time" and "iteration")
+- ``on_event`` - this method is called each time certain events occur.
+  It calls base method when the hook is triggered on specified event.
+
+
+Plugin Placement
+^^^^^^^^^^^^^^^^
+
+All trigger plugins should be placed in `Trigger folder`_.
+
+
+.. references:
+
+.. _Hooks: ../plugin_reference.html#task-hooks
+.. _Triggers: ../plugin_reference.html#task-hook-triggers
+.. _Fault Injection: ../plugin_reference.html#fault-injection-hook
+.. _Event Trigger: ../plugin_reference.html#event-hook-trigger
+.. _Periodic Trigger: ../plugin_reference.html#periodic-hook-trigger
+.. _Common Hooks: https://github.com/openstack/rally/tree/master/rally/plugins/common/hook
+.. _OpenStack Hooks: https://github.com/openstack/rally/tree/master/rally/plugins/openstack/hook
+.. _Trigger folder: https://github.com/openstack/rally/tree/master/rally/plugins/common/trigger

doc/specs/implemented/hook_plugins.rst

@@ -41,7 +41,7 @@ of hook plugins that should be executed on this iteration.
                         "args": {
                             "cmd": "bash enable_factor_1"
                         },
-                        "trigger: {
+                        "trigger": {
                             "name": "event",
                             "args": {
                                 "unit": "time",
@@ -54,7 +54,7 @@ of hook plugins that should be executed on this iteration.
                         "args": {
                             "cmd": "bash enable_factor_2"
                         },
-                        "trigger: {
+                        "trigger": {
                             "name": "event",
                             "args": {
                                 "unit": "iteration",
@@ -67,7 +67,7 @@ of hook plugins that should be executed on this iteration.
                         "args": {
                             "cmd": "bash enable_factor_3"
                         },
-                        "trigger: {
+                        "trigger": {
                             "name": "periodic",
                             "args": {
                                 "unit": "iteration",
@@ -82,7 +82,7 @@ of hook plugins that should be executed on this iteration.
                         "args": {
                             "cmd": "bash enable_factor_4"
                         },
-                        "trigger: {
+                        "trigger": {
                             "name": "periodic",
                             "args": {
                                 "unit": "time",