tripleo-docs/deploy-guide/source/deployment/ansible_config_download.rst

.. _config_download:

TripleO config-download User's Guide: Deploying with Ansible
=============================================================

Introduction
------------
This documentation details using ``config-download``.

``config-download`` is the feature that enables deploying the Overcloud software
configuration with Ansible in TripleO.

Summary
-------
Starting with the Queens release, it is possible to use Ansible to apply the
overcloud configuration. In the Rocky release, this method is the new default
behavior.

Ansible is used to replace the communication and transport of the software
configuration deployment data between Heat and the Heat agent
(os-collect-config) on the overcloud nodes.

Instead of os-collect-config running on each overcloud node and polling for
deployment data from Heat, the Ansible control node applies the configuration
by running ``ansible-playbook`` with an Ansible inventory file and a set of
playbooks and tasks.

The Ansible control node (the node running ``ansible-playbook``) is the
undercloud by default.

``config-download`` is the feature name that enables using Ansible in this
manner, and will often be used to refer to the method detailed in this
documentation.

Heat is still used to create the stack and all of the OpenStack resources. The
same parameter values and environment files are passed to Heat as they were
previously. During the stack creation, Heat creates any OpenStack service
resources such as Nova servers and Neutron networks and ports, and then creates
the software configuration data necessary to configure the overcloud via
SoftwareDeployment resources.

The difference with ``config-download`` is that although Heat creates all the
deployment data necessary via SoftwareDeployment resources to perform the
overcloud installation and configuration, it does not apply any of the software
deployments. The data is only made available via the Heat API. Once the stack
is created, deployment data is downloaded from Heat and ansible playbooks are
generated.

Using the downloaded deployment data and ansible playbooks configuration of
the overcloud using ``ansible-playbook`` are completed.

This diagram details the overall sequence of how using config-download
completes an overcloud deployment:

.. image:: ../_images/tripleo_ansible_arch.png
    :scale: 40%


Deployment with config-download
-------------------------------
Ansible and ``config-download`` are used by default when ``openstack
overcloud deploy`` (tripleoclient) is run. The command is backwards compatible
in terms of functionality, meaning that running ``openstack overcloud deploy``
will still result in a full overcloud deployment.

The deployment is done through a series of steps in tripleoclient. All of the
workflow steps are automated by tripleoclient. The workflow steps are summarized
as:

#. Create deployment plan
#. Create Heat stack along with any OpenStack resources (Neutron networks,
   Nova/Ironic instances, etc)
#. Create software configuration within the Heat stack
#. Create tripleo-admin ssh user
#. Download the software configuration from Heat
#. Applying the downloaded software configuration to the overcloud nodes with
   ``ansible-playbook``.

.. _`authorized on the overcloud nodes`:

Creating the ``tripleo-admin`` user on each overcloud node is necessary since
ansible uses ssh to connect to each node to perform configuration.

The following steps are done to create the ``tripleo-admin`` user:

#. Runs a playbook to create ``tripleo-admin`` on each node. Also, gives sudo
   permissions to the user, as well as creates and stores a new ssh keypair
   for ``tripleo-admin``.


The values for these cli arguments must be the same for all nodes in the
overcloud deployment. ``overcloud-ssh-key`` should be the private key that
corresponds with the public key specified by the Heat parameter ``KeyName``
when using Ironic deployed nodes.

config-download related CLI arguments
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
There are some new CLI arguments for ``openstack overcloud deploy`` that can be
used to influence the behavior of the overcloud deployment as it relates to
``config-download``::

    --overcloud-ssh-user   # Initial ssh user used for creating tripleo-admin.
                           # Defaults to heat-admin

    --overcloud-ssh-key    # Initial ssh private key (file path) to be used for
                           # creating tripleo-admin.
                           # Defaults to ~/.ssh/id_rsa

    --override-ansible-cfg # path to an ansible config file, to inject any
                           # arbitrary ansible config to be used when running
                           # ansible-playbook

    --stack-only           # Only update the plan and stack. Skips applying the
                           # software configuration with ansible-playbook.

    --config-download-only # Only apply the software configuration with
                           # ansible-playbook. Skips the stack and plan update.

See ``openstack overcloud deploy --help`` for further help text.

.. include:: deployment_output.rst

.. _deployment_status:

.. include:: deployment_status.rst

.. include:: deployment_log.rst

Ansible configuration
^^^^^^^^^^^^^^^^^^^^^
When ``ansible-playbook`` runs, it will use a configuration file with the
following default values::

    [defaults]
    retry_files_enabled = False
    log_path = <working directory>/ansible.log
    forks = 25

    [ssh_connection]
    ssh_args = -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ControlMaster=auto -o ControlPersist=60s
    control_path_dir = <working directory>/ansible-ssh

Any of the above configuration options can be overridden, or any additional
ansible configuration used by passing the path to an ansible configuration file
with ``--override-ansible-cfg`` on the deployment command.

For example the following command will use the configuration options from
``/home/stack/ansible.cfg``. Any options specified in the override file will
take precendence over the defaults::

    openstack overcloud deploy \
      ...
      --override-ansible-cfg /home/stack/ansible.cfg


config-download with deployed-server
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When using ``config-download`` with :doc:`deployed-server <../features/deployed_server>`
(pre-provisioned nodes), a ``HostnameMap`` parameter **must** be provided.
Create an environment file to define the parameter, and assign the node
hostnames in the parameter value. The following example shows a sample value::

  parameter_defaults:
    HostnameMap:
      overcloud-controller-0: controller-00-rack01
      overcloud-controller-1: controller-01-rack02
      overcloud-controller-2: controller-02-rack03
      overcloud-novacompute-0: compute-00-rack01
      overcloud-novacompute-1: compute-01-rack01
      overcloud-novacompute-2: compute-02-rack01

Write the contents to an environment file such as ``hostnamemap.yaml``, and
pass the environment as part of the deployment command with ``-e``.

Ansible project directory
^^^^^^^^^^^^^^^^^^^^^^^^^
The workflow will create an Ansible project directory with the plan name under
``$HOME/config-download``. For the default plan name of ``overcloud`` the working
directory will be::

    $HOME/config-download/overcloud

The project directory is where the downloaded software configuration from
Heat will be saved. It also includes other ansible-related files necessary to
run ``ansible-playbook`` to configure the overcloud.

The contents of the project directory include the following files:

tripleo-ansible-inventory.yaml
  Ansible inventory file containing hosts and vars for all the overcloud nodes.
ansible.log
  Log file from the last run of ``ansible-playbook``.
ansible.cfg
  Config file used when running ``ansible-playbook``.
ansible-playbook-command.sh
  Executable script that can be used to rerun ``ansible-playbook``.
ssh_private_key
  Private ssh key used to ssh to the overcloud nodes.

Reproducing ansible-playbook
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Once in the project directory created, simply run ``ansible-playbook-command.sh``
to reproduce the deployment::

    ./ansible-playbook-command.sh

Any additional arguments passed to this script will be passed unchanged to the
``ansible-playbook`` command::

    ./ansible-playbook-command.sh --check

Using this method it is possible to take advantage of various Ansible features,
such as check mode (``--check``), limiting hosts (``--limit``), or overriding
variables (``-e``).

Git repository
^^^^^^^^^^^^^^
The ansible project directory is a git repository. Each time config-download
downloads the software configuration data from Heat, the project directory will
be checked for differences. A new commit will be created if there are any
changes from the previous revision.

From within the ansible project directory, standard git commands can be used to
explore each revision. Commands such as ``git log``, ``git show``, and ``git
diff`` are useful ways to describe how each commit to the software
configuration differs from previous commits.

Applying earlier versions of configuration
__________________________________________
Using commands such as ``git revert`` or ``git checkout``, it is possible to
update the ansible project directory to an earlier version of the software
configuration.

It is possible to then apply this earlier version with ``ansible-playbook``.
However, caution should be exercised as this could lead to a broken overcloud
deployment. Only well understood earlier versions should be attempted to be
applied.

.. note::

    Data migration changes will never be undone by applying an earlier version
    of the software configuration with config-download. For example, database
    schema migrations that had already been applied would never be undone by
    only applying an earlier version of the configuration.

    Software changes that were related to hardware changes in the overcloud
    (such as scaling up or down) would also not be completely undone by
    applying earlier versions of the software configuration.

.. note::

    Reverting to earlier revisions of the project directory has no effect on
    the configuration stored in the Heat stack. A corresponding change should
    be made to the deployment templates, and the stack updated to make the
    changes permanent.

.. _manual-config-download:

Manual config-download
----------------------
The config-download steps can be skipped when running ``openstack overcloud deploy``
by passing ``--stack-only``. This will cause tripleoclient to only deploy the Heat
stack.

When using ``--stack-only``, the deployment data needs to be pulled from Heat
with a separate command and ``ansible-playbook`` run manually. This enables
more manual interaction and debugging.

This method is described in the following sections.

Enable tripleo-admin via SSH
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The tripleo-admin user must be `authorized on the overcloud nodes`_ for use by
``ansible-playbook``, if using the default user set by
``tripleo-ansible-inventory``. Authorizing the tripleo-admin user is done by
running the ``openstack overcloud admin authorize`` command::

    openstack overcloud admin authorize \
      --overcloud-ssh-user heat-admin \
      --overcloud-ssh-key ~/.ssh/id_rsa

Alternatively, a user and key that are already authorized on the overcloud
nodes can be used if that user and key are specified when running
``tripleo-ansible-inventory``. See `Generate an inventory`_.

Run config-download
^^^^^^^^^^^^^^^^^^^
When using the ``--stack-only`` argument, the deployment data needs to be first
downloaded from Heat. To manually download the software configuration data, use
the ``openstack overcloud config download`` command::

    openstack overcloud config download \
      --name overcloud \
      --config-dir config-download

The ansible data will be generated under a directory called ``config-download``
as specified by the ``--config-dir`` CLI argument.

Generate an inventory
^^^^^^^^^^^^^^^^^^^^^
To generate an inventory file to use with ``ansible-playbook`` use the
``tripleo-ansible-inventory`` command::

    tripleo-ansible-inventory \
      --ansible_ssh_user centos \
      --static-yaml-inventory inventory.yaml

The above example shows setting the ansible ssh user as ``centos``. This can be
changed depending on the environment. See ``tripleo-ansible-inventory --help``
for a full list of CLI arguments and options.

Run ansible-playbook
^^^^^^^^^^^^^^^^^^^^
Once the configuration has been downloaded and the inventory generated, run
``ansible-playbook`` to configure the overcloud nodes::

    ansible-playbook \
      -i inventory.yaml \
      --private-key /path/private/ssh/key \
      --become \
      config-download/deploy_steps_playbook.yaml

.. note::

    ``--become`` is required when running ansible-playbook.

All default ansible configuration values will be used when manually running
``ansible-playbook`` in this manner. These values can be customized through
`ansible configuration
<https://docs.ansible.com/ansible/latest/installation_guide/intro_configuration.html>`_.

The following minimum configuration is recommended::

    [defaults]
    log_path = ansible.log
    forks = 25
    timeout = 30

    [ssh_connection]
    ssh_args = -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ControlMaster=auto -o ControlPersist=30m
    retries = 8
    pipelining = True

.. admonition:: Ceph
   :class: ceph

   When config-download configures Ceph, Ansible executes ceph-ansible
   from within the config-download external_deploy_steps_tasks
   playbook. When config-download is run manually the `ssh_args`
   argument above will not be inherited by the second Ansible
   execution. To pass Ansible environment variables to this execution
   use a Heat environment file like the following::

     parameter_defaults:
       CephAnsibleEnvironmentVariables:
         ANSIBLE_HOST_KEY_CHECKING: 'False'
         ANSIBLE_PRIVATE_KEY_FILE: '/home/stack/.ssh/id_rsa'

.. note::

   When running ``ansible-playbook`` manually, the overcloud status as returned
   by ``openstack overcloud status`` won't be automatically updated due to the
   configuration being applied outside of the API.

   See :ref:`deployment_status` for setting the status manually.

Ansible project directory contents
----------------------------------
This section details the structure of the ``config-download`` generated
Ansible project directory.

Playbooks
^^^^^^^^^
deploy_steps_playbook.yaml
  Initial deployment or template update (not minor update)

  Further detailed in :ref:`deploy_steps_playbook.yaml`
fast_forward_upgrade_playbook.yaml
  Fast forward upgrades
post_upgrade_steps_playbook.yaml
  Post upgrade steps for major upgrade
pre_upgrade_rolling_steps_playbook.yaml
  Pre upgrade steps for major upgrade
update_steps_playbook.yaml
  Minor update steps
upgrade_steps_playbook.yaml
  Major upgrade steps

.. _deploy_steps_playbook.yaml:

deploy_steps_playbook.yaml
__________________________
``deploy_steps_playbook.yaml`` is the playbook used for deployment and template
update. It applies all the software configuration necessary to deploy a full
overcloud based on the templates provided as input to the deployment command.

This section will summarize at high level the different ansible plays used
within this playbook. The play names shown here are the same names used within
the playbook and are what will be shown in the output when ``ansible-playbook`` is
run.

The ansible tags set on each play are also shown below.

Gather facts from undercloud
  Fact gathering for the undercloud node

  tags: facts
Gather facts from overcloud
  Fact gathering for the overcloud nodes

  tags: facts
Load global variables
  Loads all variables from `l`global_vars.yaml``

  tags: always
Common roles for TripleO servers
  Applies common ansible roles to all overcloud nodes. Includes
  ``tripleo_bootstrap`` for installing bootstrap packages and
  ``tripleo_ssh_known_hosts`` for configuring ssh known hosts.

  tags: common_roles
Overcloud deploy step tasks for step 0
  Applies tasks from the ``deploy_steps_tasks`` template interface

  tags: overcloud, deploy_steps
Server deployments
  Applies server specific Heat deployments for configuration such as networking
  and hieradata. Includes ``NetworkDeployment``, ``<Role>Deployment``,
  ``<Role>AllNodesDeployment``, etc.

  tags: overcloud, pre_deploy_steps
Host prep steps
  Applies tasks from the ``host_prep_steps`` template interface

  tags: overcloud, host_prep_steps
External deployment step [1,2,3,4,5]
  Applies tasks from the ``external_deploy_steps_tasks`` template interface.
  These tasks are run against the undercloud node only.

  tags: external, external_deploy_steps
Overcloud deploy step tasks for [1,2,3,4,5]
  Applies tasks from the ``deploy_steps_tasks`` template interface

  tags: overcloud, deploy_steps
Overcloud common deploy step tasks [1,2,3,4,5]
  Applies the common tasks done at each step to include puppet host
  configuration, ``container-puppet.py``, and ``paunch`` or
  ``tripleo_container_manage`` Ansible role (container configuration).

  tags: overcloud, deploy_steps
Server Post Deployments
  Applies server specific Heat deployments for configuration done after the 5
  step deployment process.

  tags: overcloud, post_deploy_steps
External deployment Post Deploy tasks
  Applies tasks from the ``external_post_deploy_steps_tasks`` template interface.
  These tasks are run against the undercloud node only.

  tags: external, external_deploy_steps


Task files
^^^^^^^^^^
These task files include tasks specific to their intended function. The task
files are automatically used by specific playbooks from the previous section.

**boot_param_tasks.yaml**

**common_deploy_steps_tasks.yaml**

**docker_puppet_script.yaml**

**external_deploy_steps_tasks.yaml**

**external_post_deploy_steps_tasks.yaml**

**fast_forward_upgrade_bootstrap_role_tasks.yaml**

**fast_forward_upgrade_bootstrap_tasks.yaml**

**fast_forward_upgrade_post_role_tasks.yaml**

**fast_forward_upgrade_prep_role_tasks.yaml**

**fast_forward_upgrade_prep_tasks.yaml**

**fast_forward_upgrade_release_tasks.yaml**

**upgrade_steps_tasks.yaml**

**update_steps_tasks.yaml**

**pre_upgrade_rolling_steps_tasks.yaml**

**post_upgrade_steps_tasks.yaml**

**post_update_steps_tasks.yaml**

Heat Role directories
^^^^^^^^^^^^^^^^^^^^^
Each Heat role from the roles data file used in the deployment (specified with
``-r`` from the ``openstack overcloud deploy`` command), will have a
correspondingly named directory.

When using the default roles, these directories would be:

**Controller**

**Compute**

**ObjectStorage**

**BlockStorage**

**CephStorage**

A given role directory contains role specific task files and a subdirectory for
each host for that role. For example, when using the default hostnames, the
**Controller** role directory would contain the following host subdirectories:

**overcloud-controller-0**

**overcloud-controller-1**

**overcloud-controller-2**

Variable and template related files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
group_vars
  Directory which contains variables specific to different ansible inventory
  groups.
global_vars.yaml
  Global ansible variables applied to all overcloud nodes
templates
  Directory containing any templates used during the deployment

Other files
^^^^^^^^^^^
Other files in the project directory are:

ansible-playbook-command.sh
  Script to reproduce ansible-playbook command
tripleo-ansible-inventory.yaml
  Ansible inventory file
overcloud-config.tar.gz
  Tarball of Ansible project directory

Running specific tasks
----------------------
Running only specific tasks (or skipping certain tasks) can be done from within
the ansible project directory.

.. note::

    Running specific tasks is an advanced use case and only recommended for
    specific scenarios where the deployer is aware of the impact of skipping or
    only running certain tasks.

    This can be useful during troubleshooting and debugging scenarios, but
    should be used with caution as it can result in an overcloud that is not
    fully configured.

.. warning::

   All tasks that are part of the deployment need to be run, and in the order
   specified.  When skipping tasks with ``--tags``, ``-skip-tags``,
   ``--start-at-task``, the deployment could be left in an inoperable state.

   The functionality to skip tasks or only run certain tasks is meant to aid in
   troubleshooting and iterating more quickly on failing deployments and
   updates.

   All changes to the deployed cloud must still be applied through the Heat
   templates and environment files passed to the ``openstack overcloud deploy``
   command. Doing so ensures that the deployed cloud is kept in sync with the
   state of the templates and the state of the Heat stack.

.. warning::

   When skipping tasks, the overcloud must be in the state expected by the task
   starting task. Meaning, the state of the overcloud should be the same as if
   all the skipped tasks had been applied. Otherwise, the result of the tasks
   that get executed will be undefined and could leave the cloud in an
   inoperable state.

   Likewise, the deployed cloud may not be left in its fully configured state
   if tasks are skipped at the end of the deployment.

Complete the :ref:`manual-config-download` steps to create the ansible project
directory, or use the existing project directory at
``$HOME/config-download/<plan>``.

.. note::

    The project directory under ``$HOME/config-download/<plan>`` is only updated
    by ``openstack overcloud deploy`` if ``--stack-only`` is **not** used.

Tags
^^^^
The playbooks use tagged tasks for finer-grained control of what to apply if
desired. Tags can be used with the ``ansible-playbook`` CLI arguments ``--tags`` or
``--skip-tags`` to control what tasks are executed. The enabled tags are:

facts
  fact gathering
common_roles
  ansible roles common to all nodes
overcloud
  all plays for overcloud deployment
pre_deploy_steps
  deployments that happen pre deploy_steps
host_prep_steps
  Host preparation steps
deploy_steps
  deployment steps
post_deploy_steps
  deployments that happen post deploy_steps
external
  all external deployments
external_deploy_steps
  external deployments that run on the undercloud

See :ref:`deploy_steps_playbook.yaml` for a description of which tags apply to
specific plays in the deployment playbook.

Server specific pre and post deployments
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The list of server specific pre and post deployments run during the `Server
deployments` and `Server Post Deployments` plays (see
:ref:`deploy_steps_playbook.yaml`) are dependent upon what custom roles and
templates are used with the deployment.

The list of these tasks are defined in an ansible group variable that applies
to each server in the inventory group named after the Heat role. From the
ansible project directory, the value can be seen within the group variable file
named after the Heat role::

    $ cat group_vars/Compute
    Compute_pre_deployments:
      - UpgradeInitDeployment
      - HostsEntryDeployment
      - DeployedServerBootstrapDeployment
      - InstanceIdDeployment
      - NetworkDeployment
      - ComputeUpgradeInitDeployment
      - ComputeDeployment
      - ComputeHostsDeployment
      - ComputeAllNodesDeployment
      - ComputeAllNodesValidationDeployment
      - ComputeHostPrepDeployment
      - ComputeArtifactsDeploy

    Compute_post_deployments:  []

``<Role>_pre_deployments`` is the list of pre deployments, and
``<Role>_post_deployments`` is the list of post deployments.

To specify the specific task to run for each deployment, the value of the
variable can be defined on the command line when running ``ansible-playbook``,
which will overwrite the value from the group variable file for that role.

For example::

    ansible-playbook \
      -e Compute_pre_deployments=NetworkDeployment \
      --tags pre_deploy_steps
      # other CLI arguments

Using the above example, only the task for the ``NetworkDeployment`` resource
would get applied since it would be the only value defined in
``Compute_pre_deployments``, and ``--tags pre_deploy_steps`` is also specified,
causing all other plays to get skipped.

Starting at a specific task
^^^^^^^^^^^^^^^^^^^^^^^^^^^
To start the deployment at a specific task, use the ``ansible-playbook`` CLI
argument ``--start-at-task``. To see a list of task names for a given playbook,
``--list-tasks`` can be used to list the task names.

.. note::

   Some tasks that include the ``step`` variable or other ansible variables in
   the task name do not work with ``--start-at-task`` due to a limitation in
   ansible. For example the task with the name::

         Start containers for step 1

   won't work with ``--start-at-task`` since the step number is in the name
   (1).

When using ``--start-at-task``, the tasks that gather facts and load global
variables for the playbook execution are skipped by default. Skipping those
tasks can cause unexpected errors in later tasks. To avoid errors, those tasks
can be forced to execute when using ``--start-at-task`` by including the
following options to the ``ansible-playbook`` command::

      ansible-playbook \
        <other options > \
        -e gather_facts=true \
        -e @global_vars.yaml

The ``global_vars.yaml`` variable file exists in the config-download directory
that was either generated manually or under ``$HOME/config-download``.

Previewing changes
------------------
Changes can be previewed to see what will be changed before any changes are
applied to the overcloud. To preview changes, the stack update must be run with
the ``--stack-only`` cli argument::

    openstack overcloud deploy \
      --stack-only
      # other CLI arguments

Once the update is complete, complete the :ref:`manual-config-download` steps
to create the ansible project directory.

When ansible-playbook is run, use the ``--check`` CLI argument with
ansible-playbook to preview any changes. The extent to which changes can be
previewed is dependent on many factors such as the underlying tools in use
(puppet, docker, etc) and the support for ansible check mode in the given
ansible module.

The ``--diff`` option can also be used with ``--check`` to show the
differences that would result from changes.

See `Ansible Check Mode ("Dry Run")
<https://docs.ansible.com/ansible/2.5/user_guide/playbooks_checkmode.html>`_
for more details.

Generating overcloudrc
----------------------
In some cases, it may be required to manually generate the ``overcloudrc``
file if ``ansible-playbook`` was used manually outside of the workflow.

The following command can be used to generate the ``overcloudrc`` file::

    openstack overcloud credential overcloud

It will generate the ``overcloudrc`` file in the current directory. The
``--directory`` option can be used to generate it in a different location.
If needed, substitute the name of the deployment for overcloud.

config-download with Heat SoftwareDeployment outputs
----------------------------------------------------
``config-download`` does not support outputs on Heat
SoftwareDeployment/SoftwareConfig resources. Often, ``deploy_steps_tasks`` can
be used to reproduce the same behavior that would be handled by an output, by
using Ansible tasks and the ``register`` keyword.