Add documentation for task based deployment

Change-Id: I6ef8ad96706dd25fb9970f43958f5200223d1380

contents/contents-refarch.rst

@@ -14,5 +14,5 @@
 .. include:: /pages/reference-architecture/8100-zabbix.rst
 .. include:: /pages/reference-architecture/8000-fuel-upgrade.rst
 .. include:: /pages/reference-architecture/8800-base-os.rst
+.. include:: /pages/reference-architecture/9900-task-deployment.rst
 .. include:: /pages/reference-architecture/containers-master-node.rst
-

contents/contents-task-deployment.rst

@@ -0,0 +1 @@
+.. include :: /pages/reference-architecture/9900-task-deployment.rst

pages/reference-architecture/9900-task-deployment.rst

@@ -0,0 +1,12 @@
+
+
+Task-based deployment
+=====================
+
+.. include :: /pages/reference-architecture/task-deployment/0010-task-schema.rst
+.. include :: /pages/reference-architecture/task-deployment/0020-api.rst
+.. include :: /pages/reference-architecture/task-deployment/0030-faq.rst
+.. include :: /pages/reference-architecture/task-deployment/0040-add-task.rst
+.. include :: /pages/reference-architecture/task-deployment/0050-skip-task.rst
+.. include :: /pages/reference-architecture/task-deployment/0060-add-new-role.rst
+.. include :: /pages/reference-architecture/task-deployment/0070-change-task.rst

pages/reference-architecture/task-deployment/0010-task-schema.rst

@@ -0,0 +1,206 @@
+.. _0010-tasks-schema:
+
+Task schema
+-----------
+
+Tasks that are used to build a deployment graph can be grouped
+according to the common types:
+
+::
+
+     - id: graph_node_id
+       type: one of [stage, group, skipped, puppet, shell etc]
+       role: [match where this tasks should be executed]
+       requires: [requirements for a specific node]
+       required_for: [specify which nodes depend on this task]
+
+
+Stages
+------
+
+Stages are used to build a graph skeleton.
+The skeleton is then extended with additional functionality like provisioning, etc.
+
+The deployment graph of Fuel 6.1 has the following stages:
+
+::
+
+       - pre_deployment_start
+       - pre_deployment_end
+       - deploy_start
+       - deploy_end
+       - post_deployment_start
+       - post_deployment_end
+
+
+
+Here is the stage example:
+
+.. code-block:: yaml
+
+  - id: deploy_end
+    type: stage
+    requires: [deploy_start]
+
+Groups
+------
+
+In Fuel 6.1, groups are a representation of roles in the main deployment graph:
+
+::
+
+  - id: controller
+    type: group
+    role: [controller]
+    requires: [primary-controller]
+    required_for: [deploy_end]
+    parameters:
+      strategy:
+        type: parallel
+          amount: 6
+
+.. note:: Primary-controller should be installed when Controller starts its own execution.
+          The execution of this group should be finished to consider ``deploy_end`` done.
+
+Here is the full graph of groups, available in 6.1:
+
+.. image:: /_images/groups.png
+
+Strategy
+~~~~~~~~
+
+You can also specify a strategy for groups in the ``parameters`` section.
+Fuel 6.1 supports the following strategies:
+
+* parallel - all nodes in this group will be executed in parallel. If there are
+  other groups that do not depend on each other, they will be executed in parallel
+  as well. For example, Cinder and Compute groups.
+
+* parallel by amount - run in parallel by a specified number. For example, ``amount: 6``.
+
+* one_by_one - deploy all nodes in this group in a strict one-by-one succession.
+
+
+Skipped
+-------
+
+Making a task ``skipped`` will guarantee that this task will not be executed,
+but all the task's depdendencies will be preserved:
+
+.. code-block:: yaml
+
+    - id: netconfig
+      type: skipped
+      groups: [primary-controller, controller, cinder, compute, ceph-osd,
+               zabbix-server, primary-mongo, mongo]
+      required_for: [deploy_end]
+      requires: [logging]
+      parameters:
+        puppet_manifest: /etc/puppet/modules/osnailyfacter/other_path/netconfig.pp
+        puppet_modules: /etc/puppet/modules
+        timeout: 3600
+
+Puppet
+------
+
+Task of ``type: puppet`` is the preferable way to execute the deployment code on nodes.
+Only mcollective agent is capable of executing code in background.
+
+In Fuel 6.1, this is the only task that can be used in the main deployment stages,
+between ``deploy_start`` and ``deploy_end``.
+
+Example:
+
+.. code-block:: yaml
+
+  - id: netconfig
+      type: puppet
+      groups: [primary-controller, controller, cinder, compute, ceph-osd,
+               zabbix-server, primary-mongo, mongo]
+      required_for: [deploy_end]
+      requires: [logging]
+      parameters:
+        puppet_manifest: /etc/puppet/modules/osnailyfacter/other_path/netconfig.pp
+        puppet_modules: /etc/puppet/modules
+        timeout: 3600
+
+Shell
+-----
+
+Shell tasks should be used outside of the main deployment procedure.
+Basically, shell tasks will just execute the blocking command on specified roles.
+
+Example:
+
+.. code-block:: yaml
+
+  - id: enable_quorum
+    type: shell
+    role: [primary-controller]
+    requires: [post_deployment_start]
+    required_for: [post_deployment_end]
+    parameters:
+      cmd: ruby /etc/puppet/modules/osnailyfacter/modular/astute/enable_quorum.rb
+      timeout: 180
+
+
+Upload file
+-----------
+
+This task will upload data specified in ``data`` parameters to the
+``path`` destination:
+
+.. code-block:: yaml
+
+  - id: upload_data_to_file
+    type: upload_file
+    role: '*'
+    requires: [pre_deployment_start]
+    parameters:
+      path: /etc/file_name
+      data: 'arbitrary info'
+
+Sync
+----
+
+Sync task will distribute files from ``src`` direcory
+on the Fuel Master node
+to ``dst`` directory on target hosts
+that will be matched by role:
+
+
+.. code-block:: yaml
+
+  - id: rsync_core_puppet
+    type: sync
+    role: '*'
+    required_for: [pre_deployment_end]
+    requires: [upload_core_repos]
+    parameters:
+      src: rsync://10.20.0.2:/puppet/
+      dst: /etc/puppet
+      timeout:
+
+Copy files
+----------
+
+Task with ``copy_files`` type
+will read data from ``src`` and save it in the file
+specified in ``dst`` argument.
+Permissions can be specified for a group
+of files, as provided in example:
+
+.. code-block:: yaml
+
+  - id: copy_keys
+    type: copy_files
+    role: '*'
+    required_for: [pre_deployment_end]
+    requires: [generate_keys]
+    parameters:
+      files:
+        - src: /var/lib/fuel/keys/{CLUSTER_ID}/neutron/neutron.pub
+          dst: /var/lib/astute/neutron/neutron.pub
+      permissions: '0600'
+      dir_permissions: '0700'
+

pages/reference-architecture/task-deployment/0020-api.rst

@@ -0,0 +1,94 @@
+.. _0020-api:
+
+API
+---
+
+If you want to change or add some tasks right on
+the Fuel Master node, just add the ``tasks.yaml`` file
+and respective manifests in the folder for the release that you are interested in.
+Then run the following command:
+
+::
+
+       fuel rel --sync-deployment-tasks --dir /etc/puppet
+
+If you want to overwrite the deployment tasks for any specific
+release/cluster, use the following commands:
+
+::
+
+       fuel rel --rel <id> --deployment-tasks --download
+       fuel rel --rel <id> --deployment-tasks --upload
+
+       fuel env --env <id> --deployment-tasks --download
+       fuel env --env <id> --deployment-tasks --upload
+
+
+After this is done, you will be able to run a customized graph of tasks.
+To do that, use a basic command:
+
+::
+
+      fuel node --node <1>,<2>,<3> --tasks upload_repos netconfig
+
+The developer will need to specify nodes that should be used in deployment and
+task IDs. The order in which these are provided does not matter.
+It will be computed from the dependencies specified in the database.
+
+.. note:: The node will not be executed, if a task is mapped to Controller role,
+          but the node where you want to apply the task does not have this role.
+
+Skipping tasks
+--------------
+
+Use the ``skip`` parameter to skip tasks:
+
+::
+
+       fuel node --node <1>,<2>,<3> --skip netconfig hiera
+
+The list of tasks specified with the ``skip`` parameter will
+be skipped during
+graph traversal in Nailgun.
+
+If there are task dependencies, you may want to make use of a "smarter" traversal
+- you will need to
+specify the start and end nodes in the graph:
+
+::
+
+       fuel node --node <1>,<2>,<3> --end netconfig
+
+
+This will deploy everything up to the netconfig task, including it.
+This means, that this commands will deploy all tasks
+that are a part of ``pre_deployment``: keys generation, rsync
+manifests, sync time, upload repos, including such tasks as hiera setup,
+globals computation and maybe some other
+basic preparatory tasks:
+
+::
+
+      fuel node --node <1>,<2>,<3> --start netconfig
+
+
+Start from ``netconfig`` task (including it), deploy all the tasks that are
+a part of ``post_deployment``.
+
+For example, if you want to execute only the netconfig successors,
+use:
+
+::
+
+       fuel node --node <1>,<2>,<3> --start netconfig --skip netconfig
+
+
+You will also be able to use ``start`` and ``end`` at the same time:
+
+::
+
+       fuel node --node <1>,<2>,<3> --start netconfig --end upload_cirros
+
+
+Nailgun will build a path that includes only necessary tasks to join these two
+points.

pages/reference-architecture/task-deployment/0030-faq.rst

@@ -0,0 +1,40 @@
+.. _0030-faq:
+
+FAQ
+---
+
+What can I use for deployment with groups?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In Fuel 6.1, it is possible to use only Puppet for the main deployment.
+
+All agents, except for Puppet, work in a blocking way. The current deployment
+model cannot execute some tasks that are blocking and non-blocking.
+
+In the ``pre_deployment`` and ``post_deployment`` stages,
+any of the supported task drivers can be used.
+
+Is it possible to specify cross-dependencies between groups?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In Fuel 6.0 or earlier, there is no model that will allow to run tasks
+on a primary Controller, then run on a controlle with getting back to the
+primary Controller.
+
+In Fuel 6.1, cross-dependencies are resolved by the ``post_deployment`` stage.
+
+How I can end at the provision state?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Provision is not a part of task-based deployment in Fuel 6.1.
+
+How to stop deployment at the network configuration state?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+:ref:`Fuel CLI <cli_usage>` call can be used:
+it will execute the deployment up to the network configuration
+state:
+
+::
+
+  fuel node --node <1>,<2>,<3> --end netconfig

pages/reference-architecture/task-deployment/0040-add-task.rst

@@ -0,0 +1,31 @@
+.. _0040-add-task:
+
+Additional task for an existing role
+------------------------------------
+
+If you would like to add extra task for
+an existing role, follow these steps:
+
+#. Add the task description to
+   ``/etc/puppet/2014.2-6.1/modules/my_tasks.yaml`` file.
+
+   .. code-block:: yaml
+
+      - id: my_task
+      type: puppet
+      groups: [compute]
+      required_for: [deploy_end]
+      requires: [netconfig]
+      parameters:
+         puppet_manifest: /etc/puppet/modules/my_task.pp
+         puppet_modules: /etc/puppet/modules
+         timeout: 3600
+
+#. Run the following command:
+
+   ::
+
+      fuel rel --sync-deployment-tasks --dir /etc/puppet/2014.2-6.1
+
+After syncing the task to nailgun database, you will be able to deploy it on
+the selected groups.

pages/reference-architecture/task-deployment/0050-skip-task.rst

@@ -0,0 +1,35 @@
+.. _0050-add-task:
+
+Skipping task by API or by configuration
+----------------------------------------
+
+There are several mechanisms to skip a certain task.
+
+To skip a task, you can use one of the following:
+
+* Change the task's type to ``skipped``:
+
+  ::
+
+      - id: horizon
+      type: skipped
+      role: [primary-controller]
+      requires: [post_deployment_start]
+      required_for: [post_deployment_end]
+
+* Add a condition that is always false:
+
+  ::
+
+       - id: horizon
+      type: puppet
+      role: [primary-controller]
+      requires: [post_deployment_start]
+      required_for: [post_deployment_end]
+      condition: 'true != false'
+
+* Do an API request:
+
+  ::
+
+      fuel node --node <1>,<2>,<3> --skip horizon

pages/reference-architecture/task-deployment/0060-add-new-role.rst

@@ -0,0 +1,93 @@
+.. _0060-add-new-role:
+
+Creating a separate role and attaching a task to it
+---------------------------------------------------
+
+
+To create a separate role and attach a task to it,
+follow these steps:
+
+#. Create a file with ``redis.yaml`` with the following
+   content:
+
+   .. code-block:: yaml
+
+     meta:
+       description: Simple redis server
+       name: Controller
+     name: redis
+     volumes_roles_mapping:
+       - allocate_size: min
+         id: os
+
+#. Create a role:
+
+   ::
+
+
+         fuel role --rel 1 --create --file redis.yaml
+
+#. After this is done, you can go to the Fuel web UI and check if a role
+   *redis* is created.
+
+#. You can now attach tasks to the role. First, install redis puppet module:
+
+   ::
+
+        puppet module install thomasvandoren-redis
+
+
+#. Write a simple manifest to ``/etc/puppet/modules/redis/example/simple_redis.pp``
+   and include *redis*.
+
+#. Create a configuration for Fuel in ``/etc/puppet/modules/redis/example/redis_tasks.yaml``:
+
+   ::
+
+      # redis group
+        - id: redis
+          type: group
+          role: [redis]
+          required_for: [deploy_end]
+          tasks: [globals, hiera, netconfig, install_redis]
+          parameters:
+            strategy:
+              type: parallel
+
+      # Install simple redis server
+        - id: install_redis
+          type: puppet
+          requires: [netconfig]
+          required_for: [deploy_end]
+          parameters:
+            puppet_manifest: /etc/puppet/modules/redis/example/simple_redis.pp
+            puppet_modules: /etc/puppet/modules
+            timeout: 180
+
+#. Run the following command:
+
+   ::
+
+      fuel rel --sync-deployment-tasks --dir /etc/puppet/2014.2-6.1/
+
+#. :ref:`Create an enviroment <create-env-ug>`. Note the following:
+
+   * :ref:`configure public network <network-settings-ug>`
+     properly since *redis* packages are fetched from the upstream.
+
+   * enable *Assign public network to all nodes* option on the
+     :ref:`Settings <settings-ug>` tab
+     of the Fuel web UI.
+
+#. Provision the *redis* node:
+
+   ::
+
+       fuel node --node <1> --env <1> --provision
+
+#. Finish the installation on ``install_redis``
+   (there is no need to execute all tasks from ``post_deployment`` stage):
+
+   ::
+
+        fuel node --node <1> --end install_redis

pages/reference-architecture/task-deployment/0070-change-task.rst

@@ -0,0 +1,22 @@
+.. _0070-change-task:
+
+Swapping a task with a custom task
+----------------------------------
+
+To swap a task with a custom one,
+you should change the path to the executable file:
+
+.. code-block:: yaml
+
+     - id: netconfig
+       type: puppet
+       groups: [primary-controller, controller, cinder, compute, ceph-osd, zabbix-server, primary-mongo, mongo]
+       required_for: [deploy_end]
+       requires: [logging]
+       parameters:
+           # old puppet manifest
+           # puppet_manifest: /etc/puppet/modules/osnailyfacter/netconfig.pp
+
+           puppet manifest: /etc/puppet/modules/osnailyfacter/custom_netwrok_configuration.pp
+           puppet_modules: /etc/puppet/modules
+           timeout: 3600