Document using tripleo_ceph_client role with standalone Ansible

Change-Id: I222c4a3a6d6ed647dd2f7971dcf6e4df337be26f
Depends-On: I06bfb7347715b2ae5a1b37d633ec66f1b069225a

deploy-guide/source/features/ceph_external.rst

@@ -260,5 +260,135 @@ For Train use::
 
   openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible-external.yaml -e ~/my-additional-ceph-settings.yaml
 
+Standalone Ansible Roles for External Ceph
+------------------------------------------
+
+To configure an overcloud to use an external Ceph cluster, a directory
+(e.g. /etc/ceph) in the overcloud containers should be populated with
+Ceph configuration files and overcloud services (e.g. Nova) should be
+configured to use those files. Tripleo provides Ansible roles to do
+this standalone without tripleo-heat-templates or config-download.
+
+Single Ceph Cluster
+^^^^^^^^^^^^^^^^^^^
+
+The `tripleo_ceph_client_files` Ansible role copies files from a
+source directory (`tripleo_ceph_client_files_source`) on the host
+where Ansible is run to a destination directory
+(`tripleo_ceph_client_config_home`) on the overcloud nodes.
+The user must create and populate the
+`tripleo_ceph_client_files_source` directory with actual Ceph
+configuration and cephx key files before running the role. For
+example::
+
+  $ ls -l /home/stack/ceph_files/
+  total 16
+  -rw-r--r--. 1 stack stack 245 Nov 14 13:40 ceph.client.openstack.keyring
+  -rw-r--r--. 1 stack stack 173 Nov 14 13:40 ceph.conf
+
+If the above directory exists on the host where the `ansible-playbook`
+command is run, then the `tripleo_ceph_client_files_source` parameter
+should be set to `/home/stack/ceph_files/`. The optional parameter
+`tripleo_ceph_client_config_home` defaults to
+`/var/lib/tripleo-config/ceph` since OpenStack containers will bind
+mount this directory to `/etc/ceph`. The `tripleo_nova_libvirt`
+Ansible role will add a secret key to libvirt so that it uses the
+cephx key put in place by the `tripleo_ceph_client_files` role; it
+does this if either `tripleo_nova_libvirt_enable_rbd_backend` or
+`tripleo_cinder_enable_rbd_backend` are true. When these roles
+are used to configure a compute node the following `group_vars` should
+be set::
+
+  tripleo_ceph_client_files_source: /home/stack/ceph_files
+  tripleo_ceph_client_config_home: /var/lib/tripleo-config/ceph
+  tripleo_nova_libvirt_enable_rbd_backend: true
+  tripleo_cinder_enable_rbd_backend: true
+
+The `tripleo_ceph_client_files` role may then be included in a
+playbook as follows in order to configure a standalone compute node to
+use a single Ceph cluster::
+
+    - name: configure ceph client
+      import_role:
+        name: tripleo_ceph_client_files
+
+In order for Nova to use the Ceph cluster, the `libvirt` section of
+the `nova.conf` file should be configured. The `tripleo_nova_compute`
+role `tripleo_nova_compute_config_overrides` variable may be set as
+follows in the inventory to set the `libvirt` values along with
+others::
+
+  Compute:
+    vars:
+      tripleo_nova_compute_config_overrides:
+        libvirt:
+          images_rbd_ceph_conf: /etc/ceph/ceph.conf
+          images_rbd_glance_copy_poll_interval: '15'
+          images_rbd_glance_copy_timeout: '600'
+          images_rbd_glance_store_name: default_backend
+          images_rbd_pool: vms
+          images_type: rbd
+          rbd_secret_uuid: 604c9994-1d82-11ed-8ae5-5254003d6107
+          rbd_user: openstack
+
+TripleO's convention is to set the `rbd_secret_uuid` to the FSID of
+the Ceph cluster. The FSID should be in the ceph.conf file. The
+`tripleo_nova_libvirt` role will use `virsh secret-*` commands so that
+libvirt can retrieve the cephx secret using the FSID as a key. This
+can be confirmed after running Ansible with `podman exec
+nova_virtsecretd virsh secret-get-value $FSID`.
+
+The `tripleo_ceph_client_files` role only supports the _configure_
+aspect of the standalone tripleo-ansible roles because it just
+configures one or more pairs of files on its target nodes. Thus, the
+`import_role` example above could be placed in a playbook file like
+`deploy-tripleo-openstack-configure.yml`, before the roles for
+`tripleo_nova_libvirt` and `tripleo_nova_compute` are imported.
+
+Multiple Ceph Clusters
+^^^^^^^^^^^^^^^^^^^^^^
+
+To configure more than one Ceph backend include the
+`tripleo_ceph_client_files` role from the single cluster example
+above. Populate the `tripleo_ceph_client_files_source` directory with
+all of the ceph configuration and cephx key files For example::
+
+  $ ls -l /home/stack/ceph_files/
+  total 16
+  -rw-r--r--. 1 stack stack 213 Nov 14 13:41 ceph2.client.openstack.keyring
+  -rw-r--r--. 1 stack stack 228 Nov 14 13:41 ceph2.conf
+  -rw-r--r--. 1 stack stack 245 Nov 14 13:40 ceph.client.openstack.keyring
+  -rw-r--r--. 1 stack stack 173 Nov 14 13:40 ceph.conf
+
+For multiple Ceph clusters, the `tripleo_nova_libvirt` role expects a
+`tripleo_cinder_rbd_multi_config` Ansible variable like this::
+
+  tripleo_cinder_rbd_multi_config:
+    ceph2:
+      CephClusterName: ceph2
+      CephClientUserName: openstack
+
+It is not necessary to put the default Ceph cluster (named "ceph" from
+the single node example) in `tripleo_cinder_rbd_multi_config`. Only
+the additional clusters (e.g. ceph2) and name their keys so they
+match the `CephClusterName`. In the above example, the
+`CephClusterName` value "ceph2" matches the "ceph2.conf" and
+"ceph2.client.openstack.keyring". Also, the `CephClientUserName` value
+"openstack" matches "ceph2.client.openstack.keyring". The
+`tripleo_nova_libvirt` Ansible role uses the
+`tripleo_cinder_rbd_multi_config` map as a guide to know which libvirt
+secrets to create and which cephx keys to make available within the
+Nova containers.
+
+If the combined examples above from the single cluster section for
+the primary cluster "ceph" and this section for the seconary Ceph
+cluster "ceph2" are used, then the directory defined by
+`tripleo_ceph_client_config_home` will be populated with four files:
+`ceph.conf`, `ceph2.conf`, `ceph.client.openstack.keyring` and
+`ceph2.client.openstack.keyring`, which will be mounted into the Nova
+containers and two libvirt secrets will be created for each cephx
+key. To add more Ceph clusters, extend the list
+`tripleo_cinder_rbd_multi_config` and populate
+`tripleo_ceph_client_files_source` with additional files.
 
 .. _`ceph-ansible/group_vars`: https://github.com/ceph/ceph-ansible/tree/master/group_vars