Add the workflow section

Change-Id: I2fe228c6ca4b32cd38bff8522bab5bcac8d845c2

_templates/openstackdocs/theme.conf

@@ -4,6 +4,6 @@ stylesheet = css/basic.css
 pygments_style = native
 
 [options]
-globaltoc_depth = 5
+globaltoc_depth = 6
 globaltoc_includehidden = true
 analytics_tracking_code = UA-17511903-1

userdocs/fuel-user-guide/configure-environment.rst

@@ -40,3 +40,4 @@ This section includes the following topics:
    configure-environment/dns-ntp-support.rst
    configure-environment/configure-multipath.rst
    configure-environment/nfv.rst
+   configure-environment/workflows.rst

userdocs/fuel-user-guide/configure-environment/workflows.rst

@@ -0,0 +1,34 @@
+.. _workflow-intro:
+
+Modify the deployment workflow
+==============================
+
+A deployment workflow, or deployment graph, is an hierarchy of
+deployment tasks with dependencies that Fuel executes to deploy
+an OpenStack environment.
+A deployment graph enables you to execute complex orchestrated workflows,
+such as bugfixes application, reference architecture altering, or upgrades
+in a particular order. For example, you can enable specific
+network verification tasks for a Fuel plugin or change the default image
+delivering protocol (HTTP) for OpenStack nodes provisioning, and so on.
+
+.. warning::
+   This section describes advanced usage and requires the user to deeply
+   understand Fuel internals. Do not modify deployment workflows if you are
+   deploying an OpenStack environment for the first time.
+
+This section includes the following topics:
+
+.. toctree::
+   :maxdepth: 1
+
+   workflows/workflows-overview.rst
+   workflows/workflows-create.rst
+   workflows/workflows-data-driven.rst
+   workflows/workflows-run.rst
+
+.. seealso::
+
+   - :ref:`cli-graphs`
+   - :ref:`manage_workflows`
+   - :ref:`data-driven`

userdocs/fuel-user-guide/configure-environment/workflows/workflows-create.rst

@@ -0,0 +1,15 @@
+.. _workflow-create:
+
+Create a custom deployment workflow
+===================================
+
+You can modify the release workflow configuration by creating a
+cluster or plugin workflow with the required changes.
+
+This section includes the following topics:
+
+.. toctree::
+   :maxdepth: 1
+
+   workflows-create/structure.rst
+   workflows-create/examples.rst

userdocs/fuel-user-guide/configure-environment/workflows/workflows-create/examples.rst

@@ -0,0 +1,14 @@
+Examples
+--------
+
+Workflow customization is a powerful mechanism that enables you to
+create complex configurations. This section provides examples
+of how you can modify a deployment workflow.
+
+.. toctree::
+   :maxdepth: 1
+
+   examples/workflows-add-task.rst
+   examples/workflows-skip-task.rst
+   examples/workflows-create-role.rst
+   examples/workflows-swap-task.rst

userdocs/fuel-user-guide/configure-environment/workflows/workflows-create/examples/workflows-add-task.rst

@@ -0,0 +1,39 @@
+.. _workflows-add-task:
+
+Add a task
+----------
+
+You can add a task to an existing role.
+
+**To add a task:**
+
+#. Create a ``.yaml`` file.
+
+   For example, ``my_tasks.yaml``.
+
+#. Add the task description to the ``my_tasks.yaml`` file.
+
+   **Example:**
+
+   .. code-block:: console
+
+      - id: my_task
+        type: puppet
+        groups: [compute]
+        required_for: [deploy_end]
+        requires: [netconfig]
+        parameters:
+          puppet_manifest: /etc/puppet/modules/my_task.pp
+          puppet_modules: /etc/puppet/modules
+          timeout: 3600
+
+#. Log in to the Fuel CLI.
+#. Run the following command:
+
+   .. code-block:: console
+
+      fuel rel --sync-deployment-tasks --dir <path-to-dir-with-the-task.pp>
+
+   Fuel syncs the with the internal database.
+
+#. Deploy the OpenStack environment.

userdocs/fuel-user-guide/configure-environment/workflows/workflows-create/examples/workflows-create-role.rst

@@ -0,0 +1,121 @@
+.. _workflows-create-task:
+
+Creating a separate role with a task
+------------------------------------
+
+You can create a separate role and attach a custom task to that
+role. Examples in this section describe installation of a Redis
+database server. You can apply similar principles for other
+tasks.
+
+.. warning::
+
+   We recommend that you add new roles through creating plugins.
+
+**To create a separate role with a task:**
+
+#. Create a ``.yaml`` file with a user-friendly name.
+
+   For example, ``redis.yaml``.
+
+#. Open the file for editing and add meta information:
+
+   **Example:**
+
+   .. code-block:: console
+
+      meta:
+        description: Simple Redis server
+        name: Controller
+      name: redis
+      volumes_roles_mapping:
+        - allocate_size: min
+          id: os
+
+#. Create a role using Fuel CLI:
+
+   .. code-block:: console
+
+      fuel role --rel 1 --create --file <my.yaml>
+
+   **Example:**
+
+   .. code-block:: console
+
+      fuel role --rel 1 --create --file redis.yaml
+
+#. Log in to the Fuel Web UI.
+#. In the :guilabel:`Nodes` tab, verify that the required role is created.
+#. Attach a task to the newly created role:
+
+   #. Log in to Fuel CLI.
+   #. Install the required Puppet modules.
+
+      **Example:**
+
+      .. code-block:: console
+
+         puppet module install thomasvandoren-redis
+
+   #. Write a simple manifest to the required location. For example:
+      ``/etc/puppet/modules/redis/example/simple_redis.pp`` and include redis.
+
+   #. Create a deployment task in you ``tasks.yaml``. For example, 
+      ``in /etc/puppet/modules/redis/example/redis_tasks.yaml``.
+
+      **Example:**
+
+      .. code-block:: console
+
+         # redis group
+         - id: redis
+           type: group
+           role: [redis]
+           required_for: [deploy_end]
+           tasks: [globals, hiera, netconfig, install_redis]
+           parameters:
+           strategy:
+           type: parallel
+
+        # Install simple redis server
+        - id: install_redis
+          type: puppet
+          requires: [netconfig]
+          required_for: [deploy_end]
+          parameters:
+            puppet_manifest: /etc/puppet/modules/redis/example/simple_redis.pp
+            puppet_modules: /etc/puppet/modules
+            timeout: 180
+
+   #. Synchronize deployment tasks:
+
+      .. code-block:: console
+
+         fuel rel --sync-deployment-tasks --dir <path-to-puppet-manifest>
+
+      **Example:**
+
+      .. code-block:: console
+
+         fuel rel --sync-deployment-tasks --dir /etc/puppet/mitaka-9.0/
+
+   #. Configure and create an OpenStack environment with all required
+      network, storage, and other settings.
+   #. Provision a node with the created role:
+
+      **Example:**
+
+      .. code-block:: console
+
+         fuel node --node <node_ID> --env <env_ID> --provision
+
+   #. Deploy a node with the created role:
+
+      .. code-block:: console
+
+         fuel node --node <node_ID> --env <env_ID> --deploy
+
+.. seealso::
+
+   - `Plugin node roles
+     <http://docs.openstack.org/developer/fuel-docs/plugindocs/fuel-plugin-sdk-guide/create-plugin/plugin-node-roles.html>`_

userdocs/fuel-user-guide/configure-environment/workflows/workflows-create/examples/workflows-skip-task.rst

@@ -0,0 +1,55 @@
+.. _workflows-skip-task:
+
+Skip a deployment task
+----------------------
+
+You can skip a deployment task using the type or condition parameter
+of the task, or through an API request.
+
+When using API requests, you can specify the list of tasks to skip or
+indicate the first and the last task to skip.
+
+**To skip a deployment task:**
+
+#. Select from the following options:
+
+   * Specify the ``type`` parameter:
+
+     **Example:**
+
+     .. code-block:: console
+
+        - id: horizon
+          type: skipped
+          role: [primary-controller]
+          requires: [post_deployment_start]
+          required_for: [post_deployment_end]
+
+   * Specify a ``false`` condition:
+
+     **Example:**
+
+     .. code-block:: console
+
+        - id: horizon
+          type: puppet
+          role: [primary-controller]
+          requires: [post_deployment_start]
+          required_for: [post_deployment_end]
+          condition: 'true != false'
+
+#. Synchronize deployment tasks:
+
+     .. code-block:: console
+
+        fuel rel --sync-deployment-tasks --dir <path-to-puppet-manifest>
+
+     **Example:**
+
+     .. code-block:: console
+
+        fuel rel --sync-deployment-tasks --dir /etc/puppet/mitaka-9.0/
+
+.. seealso::
+
+   - :ref:`data-driven`

userdocs/fuel-user-guide/configure-environment/workflows/workflows-create/examples/workflows-swap-task.rst

@@ -0,0 +1,43 @@
+.. _workloads-replace-task:
+
+Replace a task
+--------------
+
+You can replace a task with a custom task in the
+``task.yaml`` file by replacing the path to the executable
+file.
+
+**To replace a task:**
+
+#. Log in to Fuel CLI:
+#. Open the ``task.yaml`` file for editing.
+#. Replace the path to the path to the executable file:
+
+   .. code-block:: console
+
+      - id: netconfig
+        type: puppet
+        groups: [primary-controller, controller, cinder, compute, ceph-osd,
+        zabbix-server, primary-mongo, mongo]
+        required_for: [deploy_end]
+        requires: [logging]
+        parameters:
+            # old puppet manifest
+            # puppet_manifest: /etc/puppet/modules/osnailyfacter/netconfig.pp
+
+            puppet manifest:
+            /etc/puppet/modules/osnailyfacter/custom_network_configuration.pp
+            puppet_modules: /etc/puppet/modules
+            timeout: 3600
+
+#. Synchronize deployment tasks:
+
+   .. code-block:: console
+
+      fuel rel --sync-deployment-tasks --dir <path-to-puppet-manifest>
+
+   **Example:**
+
+   .. code-block:: console
+
+      fuel rel --sync-deployment-tasks --dir /etc/puppet/mitaka-9.0/

userdocs/fuel-user-guide/configure-environment/workflows/workflows-create/structure.rst

@@ -0,0 +1,218 @@
+.. _workflow-create-structure:
+
+Workflow task structure
+-----------------------
+
+You can group the tasks defined in the deployment workflow
+file using the types described in this section.
+
+The following table describes the workflow task structure:
+
+.. list-table:: **Workflow task structure**
+   :widths: 10 10 10
+   :header-rows: 1
+
+   * - Parameter
+     - Description
+     - Example
+   * - ``id``
+     - Name of the deployment task.
+     -
+   * - ``version``
+     - Version of the workflow execution engine.
+     -
+   * - ``type``
+     - A type of task that Fuel executes. You can specify the
+       following types of tasks:
+     -
+   * - ``stage``
+     - Divides a deployment workflow on phases. In the default
+       release workflow Fuel has the following stages:
+
+       - ``pre_deployment_start``
+       - ``pre_deployment_end``
+       - ``deploy_start``
+       - ``deploy_end``
+       - ``post_deployment_start``
+       - ``post_deployment_end``
+
+       However, you can create any stages as required or do
+       not use them at all.
+
+     - ::
+
+         id: deploy_end
+         type: stage
+         requires: [deploy_start]
+   * - ``groups``
+     - Describes node roles. By default, Fuel executes all tasks
+       simultaneously. Hovewer, the order of tasks execution can be defined
+       in the cross-depends and cross-depended-by fields of tasks.
+
+       * ``one_by_one`` - deploy all nodes in this group one after another.
+
+     - ::
+
+        - id: controller
+          type: group
+          role: [controller]
+          requires: [primary-controller]
+          required_for: [deploy_end]
+
+   * - ``skipped``
+     - Fuel does not execute this task, but preserves all dependencies specified
+       in the tasks. For example, if you use a plugin that requires some to be
+       skipped, you can create a ``skipped`` task in the plugin workflow file.
+     - ::
+
+        - id: netconfig
+          type: skipped
+          groups: [primary-controller, controller, cinder, compute, ceph-osd,
+                  zabbix-server, primary-mongo, mongo]
+          required_for: [deploy_end]
+          requires: [logging]
+          parameters:
+            puppet_manifest: /etc/puppet/modules/osnailyfacter/netconfig.pp
+            puppet_modules: /etc/puppet/modules
+            timeout: 3600
+
+   * - ``puppet``
+     - Deploys Puppet manifests.
+     - ::
+
+        - id: netconfig
+          type: puppet
+          groups: [primary-controller, controller, cinder, compute, ceph-osd,
+                  zabbix-server, primary-mongo, mongo]
+          required_for: [deploy_end]
+          requires: [logging]
+          parameters:
+            puppet_manifest:
+            /etc/puppet/modules/osnailyfacter/other_path/netconfig.pp
+            puppet_modules: /etc/puppet/modules
+            timeout: 3600
+   * - ``shell``
+     - Executes shell scripts.
+     - ::
+
+        - id: enable_quorum
+          type: shell
+          role: [primary-controller]
+          requires: [post_deployment_start]
+          required_for: [post_deployment_end]
+          parameters:
+            cmd: ruby
+            /etc/puppet/modules/osnailyfacter/modular/astute/enable_quorum.rb
+            timeout: 180
+   * - ``upload_file``
+     - Uploads values specified in ``data`` in the ``parameters`` section.
+     - ::
+
+        - id: upload_data_to_file
+          type: upload_file
+          role: '*'
+          requires: [pre_deployment_start]
+          parameters:
+            path: /etc/file_name
+            data: 'arbitrary info'
+   * - ``sync``
+     - Distributes files from the ``src`` direcory on the Fuel Master node
+       to the ``dst`` directory on the Fuel Slave nodes that match the
+       specified roles.
+     - ::
+
+        - id: rsync_core_puppet
+          type: sync
+          role: '*'
+          required_for: [pre_deployment_end]
+          requires: [upload_core_repos]
+          parameters:
+            src: rsync://<FUEL_MASTER_IP>:/puppet/
+            dst: /etc/puppet
+            timeout:
+
+   * - ``copy_files``
+     - Reads data from ``src`` and saves it in the file specified in the
+       ``dst`` argument. Permissions can be specified for a group of files.
+     - ::
+
+        - id: copy_keys
+          type: copy_files
+          role: '*'
+          required_for: [pre_deployment_end]
+          requires: [generate_keys]
+          parameters:
+          files:
+            src: /var/lib/fuel/keys/{CLUSTER_ID}/neutron/neutron.pub
+            dst: /var/lib/astute/neutron/neutron.pub
+          permissions: '0600'
+          dir_permissions: '0700'
+
+   * - ``role``
+     - Node roles on which the task is executed. To select all roles assigned
+       to the node, you can use a wildcard '*'.
+     - ::
+
+         role: [primary-controller]
+
+   * - ``requires``
+     - Requirements for a specific task or stage on the same node.
+     - ::
+
+         requires: [generate_keys]
+
+   * - ``required_for``
+     - Specifies which tasks and stages depend on this task.
+     - ::
+
+        required_for: [pre_deployment_end]
+
+   * - ``reexecute_on``
+     - Re-run the task after completion.
+     - ::
+
+        reexecute_on: [deploy_changes]
+
+   * - ``cross-depended-by``
+     - Specifies which tasks and stages on other nodes are dependent by this
+       task. You can specify a value in a form of a regular expression.
+       For example, use ``name:`` entries. Do not use lists
+       not use lists.
+
+       .. warning:: The ``cross-depends`` and ``requires`` fields have
+          different syntax!
+
+     - ::
+
+          cross-depended-by:
+            - name: neutron-keystone
+
+   * - ``cross-depends``
+     - Reverse to ``cross-depended-by``. You can specify the value in a form
+       of a regular expression. Do not use lists.
+     - ::
+
+         cross-depends:
+           - name: neutron-keystone
+             role: primary-controller
+           - name: openstack-haproxy
+
+   * - ``condition``
+     - Describes a condition required to execute this task.
+       For more information, see: :ref:`data-driven`.
+     - ::
+
+        condition: yaql_exp: {yaql expression}
+        parameters:
+          data: yaql_exp: {yaql expression}
+
+   * - ``parameters``
+     - Task execution parameters. Differ for each task.
+     - ::
+
+         parameters:
+           files:
+             - src: /var/lib/fuel/keys/{CLUSTER_ID}/neutron/neutron.pub
+               dst: /var/lib/astute/neutron/neutron.pub
+           permissions: '0600'
+           dir_permissions: '0700'

userdocs/fuel-user-guide/configure-environment/workflows/workflows-overview.rst

@@ -0,0 +1,37 @@
+.. _workflow-overview:
+
+Overview of the deployment workflows
+------------------------------------
+
+You can modify the following OpenStack environment deployment stages
+using custom deployment workflows:
+
+* **Deletion** - if you deploy a brand new environment, Fuel
+  skips this step. If you make changes to the existing environment, Fuel
+  deletes nodes that were marked for deletion.
+
+* **Network verification** - Fuel verifies network configuration
+  of the OpenStack environment.
+
+* **Provisioning** - Fuel provisions the OpenStack nodes.
+
+* **Deployment** - Fuel deploys the OpenStack environment.
+
+An OpenStack environment has the following level of deployment workflows:
+
+* ``release`` - describes the default OpenStack environment workflows.
+  We do not recommend to make any changes in the release workflow. Apply
+  all changes to the release workflow by creating and modifying the plugin
+  and cluster workflows.
+
+* ``plugin`` - describes plugin deployment workflow. During the deployment,
+  Fuel merges the configuration described in the plugin workflows with
+  the release workflow and applies a unified deployment graph. Typically,
+  plugin's source code includes the descriptions of deployment workflow in
+  ``deployment_tasks.yaml``. By default, Fuel includes an empty plugin
+  workflow. If you want to make changes in an OpenStack environment,
+  creating a plugin is the preferred way.
+
+* ``cluster`` - describes the changes that will be applied to this OpenStack
+  environment. Use this option only if creating a plugin is not possible.
+  In all other cases, create a plugin.

userdocs/fuel-user-guide/configure-environment/workflows/workflows-run.rst

@@ -0,0 +1,112 @@
+
+.. _custom-graph:
+
+==================
+Run a custom graph
+==================
+
+You can execute a custom deployment graph and merge it with existing deployment
+graphs of the upstream master release.
+
+This allows you to implement complex orchestrated workflows, such as
+bugfixes application, reference architecture altering, or even upgrades.
+
+Each cluster has three classes of deployment graphs with the following
+hierarchy listed in a descending order of importance:
+
+* Cluster-specific graph (most important).
+* Graphs introduced by plugins -- These are the graphs of the enabled
+  plugins merged by the task ID.
+* Release-default graph (least important).
+
+The custom graph may be of any particular type and is stored in the database
+with this type. The default deployment graphs have the type ``default``.
+
+Each deployment run executes the deployment of a particular type
+(``default`` by default). Fuel fetches the graphs of each of the three classes
+for the corresponding type of deployment and merges the graphs by merging all
+tasks by task IDs where high-level tasks override the lower ones.
+
+An example in the order of the graphs overriding:
+
+**Default deployment:**
+
+#. Release the default graph as derived from the tasks.yaml file of
+   fuel-library.
+#. Plugins default graphs, which are ``deployment_tasks.yaml`` from plugins
+   manifests.
+#. Cluster default grap, which is empty by default, with cluster-specific
+   tasks specified by the user
+
+The plugins graph merge order is not deterministic and it is supposed that
+plugins graphs have no tasks intersections by task ID.
+
+**Custom-type graph:**
+
+#. Release custom-type graph, which is empty at this stage but can be derived
+   from the tasks.yaml file of fuel-library or be delivered by a maintenance
+   update.
+#. Plugins custom-type graphs can be specified by plugin developers.
+#. Cluster default grap, which is empty by default, with cluster-specific
+   tasks specified by the user
+
+To list a cluster-specific table with graphs, their relations, names, and
+types:
+
+The graph ``name`` is an arbitrary parameter that defines additional
+information about the graph. It has no impact on the business logic.
+
+#. Log in to the Fuel master node.
+#. Run the following command:
+
+   .. code-block:: console
+
+      fuel2 graph list --env env_id
+
+   where ``env_id`` is the ID of the environment.
+
+**To upload a graph:**
+
+#. Log in to the Fuel master node.
+#. Run the following command:
+
+   .. code-block:: console
+
+      fuel2 graph upload --env env_id [--type graph_type] --file tasks.yaml
+      fuel2 graph upload --release release_id [--type graph_type] --file tasks.yaml
+      fuel2 graph upload --plugin plugin_id [--type graph_type] --file tasks.yaml
+
+   where ``--type`` is an optional parameter. The ``default`` graph type with
+   confirmation should be used if no type is defined.
+
+The graphs downloaded with the keys ``--all`` and ``--plugins`` are the
+result of other graphs merge performed by Nailgun and are not supposed to be
+edited and uploaded back.
+In most cases because they will completely override further changes made in
+source graphs.
+
+**To download a graph:**
+
+#. Log in to the Fuel master node.
+#. Run the following command:
+
+   .. code-block:: console
+
+      fuel2 graph download --env env_id --all [--type graph_type] [--file cluster_graph.yaml]
+      fuel2 graph download --env env_id --cluster [--type graph_type] [--file cluster_graph.yaml]
+      fuel2 graph download --env env_id --plugins [--type graph_type] [--file plugins_graph.yaml]
+      fuel2 graph download --env env_id --release [--type graph_type] [--file release_graph.yaml]
+
+     where ``--type`` is an optional parameter. The command downloads the
+     ``default``if no type is defined.
+
+**To execute a graph:**
+
+#. Log in to the Fuel master node.
+#. Run the following command:
+
+   .. code-block:: console
+
+      fuel2 graph execute --env env_id [--type graph_type] [--node node_ids]
+
+Graph execution is available only for the environment.

userdocs/fuel-user-guide/maintain-environment.rst

@@ -23,6 +23,5 @@ This section includes the following topics:
    maintain-environment/reinstall-virtual-role.rst
    maintain-environment/create-snapshot.rst
    maintain-environment/manage-workflows.rst
-   maintain-environment/data-driven.rst
    maintain-environment/shutdown-env.rst
-   maintain-environment/start-env.rst
+   maintain-environment/start-env.rst