Updated DE architecture doc + 'period' param

In this changeset, I updated the architecture documentation
about the Decision Engine by adding a new sequence diagram
outlining the CDM synchronization workflow.
I also explained the default 'period' parameter used in the
CDMC plugin.

Change-Id: I09790281ba9117e302ab8e66a887667929c6c261
Partially-Implements: blueprint cluster-model-objects-wrapper

doc/source/architecture.rst

@@ -176,30 +176,42 @@ associated :ref:`Audit Template <audit_template_definition>` and knows the
 :ref:`Goal <goal_definition>` to achieve.
 
 It then selects the most appropriate :ref:`Strategy <strategy_definition>`
-depending on how Watcher was configured for this :ref:`Goal <goal_definition>`.
+from the list of available strategies achieving this goal.
 
 The :ref:`Strategy <strategy_definition>` is then dynamically loaded (via
-`stevedore <https://github.com/openstack/stevedore/>`_). The
+`stevedore <http://docs.openstack.org/developer/stevedore/>`_). The
-:ref:`Watcher Decision Engine <watcher_decision_engine_definition>` calls the
+:ref:`Watcher Decision Engine <watcher_decision_engine_definition>` executes
-**execute()** method of the :ref:`Strategy <strategy_definition>` class which
+the strategy.
-generates a solution composed of a set of :ref:`Actions <action_definition>`.
+
+In order to compute the potential :ref:`Solution <solution_definition>` for the
+Audit, the :ref:`Strategy <strategy_definition>` relies on different sets of
+data:
+
+- :ref:`Cluster data models <cluster_data_model_definition>` that are
+  periodically synchronized through pluggable cluster data model collectors.
+  These models contain the current state of various
+  :ref:`Managed resources <managed_resource_definition>` (e.g., the data stored
+  in the Nova database). These models gives a strategy the ability to reason on
+  the current state of a given :ref:`cluster <cluster_definition>`.
+- The data stored in the :ref:`Cluster History Database
+  <cluster_history_db_definition>` which provides information about the past of
+  the :ref:`Cluster <cluster_definition>`.
+
+Here below is a sequence diagram showing how the Decision Engine builds and
+maintains the :ref:`cluster data models <cluster_data_model_definition>` that
+are used by the strategies.
+
+.. image:: ./images/sequence_architecture_cdmc_sync.png
+   :width: 100%
+
+The execution of a strategy then yields a solution composed of a set of
+:ref:`Actions <action_definition>` as well as a set of :ref:`efficacy
+indicators <efficacy_indicator_definition>`.
 
 These :ref:`Actions <action_definition>` are scheduled in time by the
 :ref:`Watcher Planner <watcher_planner_definition>` (i.e., it generates an
 :ref:`Action Plan <action_plan_definition>`).
 
-In order to compute the potential :ref:`Solution <solution_definition>` for the
-Audit, the :ref:`Strategy <strategy_definition>` relies on two sets of data:
-
--   the current state of the
-    :ref:`Managed resources <managed_resource_definition>`
-    (e.g., the data stored in the Nova database)
--   the data stored in the
-    :ref:`Cluster History Database <cluster_history_db_definition>`
-    which provides information about the past of the
-    :ref:`Cluster <cluster_definition>`
-
-
 .. _data_model:
 
 Data model
@@ -216,8 +228,6 @@ Here below is a diagram representing the main objects in Watcher from a
 database perspective:
 
 .. image:: ./images/watcher_db_schema_diagram.png
-   :width: 100%
-
 
 
 .. _sequence_diagrams:

doc/source/dev/plugin/action-plugin.rst

@@ -114,12 +114,15 @@ tune the action to its needs. To do so, you can implement the
         def execute(self):
             assert self.config.test_opt == 0
 
-        def get_config_opts(self):
+        @classmethod
-            return [
+        def get_config_opts(cls):
+            return super(
+                DummyAction, cls).get_config_opts() + [
                 cfg.StrOpt('test_opt', help="Demo Option.", default=0),
                 # Some more options ...
             ]
 
+
 The configuration options defined within this class method will be included
 within the global ``watcher.conf`` configuration file under a section named by
 convention: ``{namespace}.{plugin_name}``. In our case, the ``watcher.conf``

doc/source/dev/plugin/cdmc-plugin.rst

@@ -54,6 +54,10 @@ Define configuration parameters
 ===============================
 
 At this point, you have a fully functional cluster data model collector.
+By default, cluster data model collectors define an ``period`` option (see
+:py:meth:`~.BaseClusterDataModelCollector.get_config_opts`) that corresponds
+to the interval of time between each synchronization of the in-memory model.
+
 However, in more complex implementation, you may want to define some
 configuration options so one can tune the cluster data model collector to its
 needs. To do so, you can implement the :py:meth:`~.Loadable.get_config_opts`
@@ -73,8 +77,10 @@ class method as followed:
             # Do something here...
             return model
 
-        def get_config_opts(self):
+        @classmethod
-            return [
+        def get_config_opts(cls):
+            return super(
+                DummyClusterDataModelCollector, cls).get_config_opts() + [
                 cfg.StrOpt('test_opt', help="Demo Option.", default=0),
                 # Some more options ...
             ]

doc/source/dev/plugin/planner-plugin.rst

@@ -91,8 +91,10 @@ tune the planner to its needs. To do so, you can implement the
             assert self.config.test_opt == 0
             # [...]
 
-        def get_config_opts(self):
+        @classmethod
-            return [
+        def get_config_opts(cls):
+            return super(
+                DummyPlanner, cls).get_config_opts() + [
                 cfg.StrOpt('test_opt', help="Demo Option.", default=0),
                 # Some more options ...
             ]

doc/source/image_src/plantuml/sequence_architecture_cdmc_sync.txt

@@ -0,0 +1,41 @@
+@startuml
+skinparam maxMessageSize 100
+
+actor "Administrator"
+
+== Initialization ==
+
+"Administrator" -> "Decision Engine" : Start all services
+"Decision Engine" -> "Background Task Scheduler" : Start
+
+activate "Background Task Scheduler"
+"Background Task Scheduler" -> "Cluster Model Collector Loader"\
+: List available cluster data models
+"Cluster Model Collector Loader" --> "Background Task Scheduler"\
+: list of BaseClusterModelCollector instances
+
+loop for every available cluster data model collector
+    "Background Task Scheduler" -> "Background Task Scheduler"\
+    : add periodic synchronization job
+    create "Jobs Pool"
+    "Background Task Scheduler" -> "Jobs Pool" : Create sync job
+end
+deactivate "Background Task Scheduler"
+
+hnote over "Background Task Scheduler" : Idle
+
+== Job workflow ==
+
+"Background Task Scheduler" -> "Jobs Pool" : Trigger synchronization job
+"Jobs Pool" -> "Nova Cluster Data Model Collector" : synchronize
+
+activate "Nova Cluster Data Model Collector"
+    "Nova Cluster Data Model Collector" -> "Nova API"\
+    : Fetch needed data to build the cluster data model
+    "Nova API" --> "Nova Cluster Data Model Collector" : Needed data
+    "Nova Cluster Data Model Collector" -> "Nova Cluster Data Model Collector"\
+    : Build an in-memory cluster data model
+    ]o<-- "Nova Cluster Data Model Collector" : Done
+deactivate "Nova Cluster Data Model Collector"
+
+@enduml

doc/source/image_src/plantuml/sequence_trigger_audit_in_decision_engine.txt

@@ -10,45 +10,41 @@ activate "Decision Engine"
 "AMQP Bus" <[#blue]- "Decision Engine" : notify new audit state = ONGOING
 "Decision Engine" -> "Database" : get audit parameters (goal, strategy, ...)
 "Decision Engine" <-- "Database" : audit parameters (goal, strategy, ...)
-"Decision Engine" --> "Decision Engine": select appropriate \
+"Decision Engine" --> "Decision Engine"\
-optimization strategy (via the Strategy Selector)
+: select appropriate optimization strategy (via the Strategy Selector)
 create Strategy
-"Decision Engine" -> "Strategy" : execute()
+"Decision Engine" -> "Strategy" : execute strategy
 activate "Strategy"
-create "Cluster Data Model Collector"
+    "Strategy" -> "Cluster Data Model Collector" : get cluster data model
-"Strategy" -> "Cluster Data Model Collector" : get cluster data model
+    "Cluster Data Model Collector" --> "Strategy"\
-
+    : copy of the in-memory cluster data model
-activate "Cluster Data Model Collector"
+    loop while enough history data for the strategy
-    loop while enough data to build cluster data model
+        "Strategy" -> "Ceilometer API" : get necessary metrics
-      "Cluster Data Model Collector" -> "Nova API" : get resource state (\
+        "Strategy" <-- "Ceilometer API" : aggregated metrics
-host, instance, ...)
-      "Cluster Data Model Collector" <-- "Nova API" : resource state
     end
-"Cluster Data Model Collector" -> "Strategy" : cluster data model
+    "Strategy" -> "Strategy"\
-deactivate "Cluster Data Model Collector"
+    : compute/set needed actions for the solution so it achieves its goal
-
+    "Strategy" -> "Strategy" : compute/set efficacy indicators for the solution
-loop while enough history data for the strategy
+    "Strategy" -> "Strategy" : compute/set the solution global efficacy
-    "Strategy" -> "Ceilometer API": get necessary metrics
+    "Decision Engine" <-- "Strategy"\
-    "Strategy" <-- "Ceilometer API": aggregated metrics
+    : solution (unordered actions, efficacy indicators and global efficacy)
-end
-"Strategy" -> "Strategy" : compute/set needed actions for the solution \
-so it achieves its goal
-"Strategy" -> "Strategy" : compute/set efficacy indicators for the solution
-"Strategy" -> "Strategy" : compute/set the solution global efficacy
-"Decision Engine" <-- "Strategy" : solution (contains a list of unordered \
-actions alongside its efficacy indicators as well as its global efficacy)
 deactivate "Strategy"
 
-"Decision Engine" --> "Planner": load actions scheduler (i.e. Planner plugin)
 create "Planner"
-"Decision Engine" -> "Planner": schedule()
+"Decision Engine" -> "Planner" : load actions scheduler
-"Planner" -> "Planner": schedule actions according to \
+"Planner" --> "Decision Engine" : planner plugin
-scheduling rules/policies
+"Decision Engine" -> "Planner" : schedule actions
-"Decision Engine" <-- "Planner": new action plan
+activate "Planner"
+    "Planner" -> "Planner"\
+    : schedule actions according to scheduling rules/policies
+    "Decision Engine" <-- "Planner" : new action plan
+deactivate "Planner"
 "Decision Engine" -> "Database" : save new action plan in database
 "Decision Engine" -> "Database" : update audit.state = SUCCEEDED
 "AMQP Bus" <[#blue]- "Decision Engine" : notify new audit state = SUCCEEDED
 
 deactivate "Decision Engine"
 
+hnote over "Decision Engine" : Idle
+
 @enduml