Distributed Multibackend Storage
================================

In Ussuri and newer, |project| is able to extend
:doc:`distributed_compute_node` to include distributed image
management and persistent storage with the benefits of using
OpenStack and Ceph.

Features
--------

This Distributed Multibackend Storage design extends the architecture
described in :doc:`distributed_compute_node` to support the following
worklow.

- Upload an image to the Central site, and any additional DCN sites
  with storage, concurrently using one command like `glance
  image-create-via-import --stores central,dcn1,dcn3`.
- Move a copy of the same image to additional DCN sites when needed
  using a command like `glance image-import <IMAGE-ID> --stores
  dcn2,dcn4 --import-method copy-image`.
- The image's unique ID will be shared consistently across sites
- The image may be copy-on-write booted on any DCN site as the RBD
  pools for Glance and Nova will use the same local Ceph cluster.
- If the Glance server at each DCN site was configured with write
  access to the Central Ceph cluster as an additional store, then an
  image generated from making a snapshot of an instance running at a
  DCN site may be copied back to the central site and then copied to
  additional DCN sites.
- The same Ceph cluster per site may also be used by Cinder as an RBD
  store to offer local volumes in active/active mode.

In the above workflow the only time RBD traffic crosses the WAN is
when an image is imported or copied between sites. Otherwise all RBD
traffic is local to each site for fast COW boots, and performant IO
to the local Cinder and Nova Ceph pools.

Architecture
------------

The architecture to support the above features has the following
properties.

- A separate Ceph cluster at each availability zone or geographic
  location
- Glance servers at each availability zone or geographic location
- The containers implementing the Ceph clusters may be collocated on
  the same hardware providing compute services, i.e. the compute nodes
  may be hyper-converged, though it is not necessary that they be
  hyper-converged
- It is not necessary to deploy Glance and Ceph at each DCN site, if
  storage services are not needed at that DCN site

In this scenario the Glance service at the central site is configured
with multiple stores such that.

- The central Glance server's default store is the central Ceph
  cluster using the RBD driver
- The central Glance server has additional RBD stores; one per DCN
  site running Ceph

Similarly the Glance server at each DCN site is configured with
multiple stores such that.

- Each DCN Glance server's default store is the DCN Ceph
  cluster that is in the same geographic location.
- Each DCN Glance server is configured with one additional store which
  is the Central RBD Ceph cluster.

Though there are Glance services distributed to multiple sites, the
Glance client for overcloud users should use the public Glance
endpoints at the central site. These endpoints may be determined by
querying the Keystone service, which only runs at the central site,
with `openstack endpoint list`. Ideally all images should reside in
the central Glance and be copied to DCN sites before instances of
those images are booted on DCN sites. If an image is not copied to a
DCN site before it is booted, then the image will be streamed to the
DCN site and then the image will boot as an instance. This happens
because Glance at the DCN site has access to the images store at the
Central ceph cluster. Though the booting of the image will take time
because it has not been copied in advance, this is still preferable
to failing to boot the image.

Stacks
------

In the example deployment three stacks are deployed:

control-plane
   All control plane services including Glance. Includes a Ceph
   cluster named central which is hypercoverged with compute nodes and
   runs Cinder in active/passive mode managed by pacemaker.
dcn0
   Runs Compute, Glance and Ceph services. The Cinder volume service
   is configured in active/active mode and not managed by pacemaker.
   The Compute and Cinder services are deployed in a separate
   availability zone and may also be in a separate geographic
   location.
dcn1
   Deploys the same services as dcn0 but in a different availability
   zone and also in a separate geographic location.

Note how the above differs from the :doc:`distributed_compute_node`
example which splits services at the primary location into two stacks
called `control-plane` and `central`. This example combines the two
into one stack.

During the deployment steps all templates used to deploy the
control-plane stack will be kept on the undercloud in
`/home/stack/control-plane`, all templates used to deploy the dcn0
stack will be kept on the undercloud in `/home/stack/dcn0` and dcn1
will follow the same pattern as dcn0. The sites dcn2, dcn3 and so on
may be created, based on need, by following the same pattern.

Ceph Deployment Types
---------------------

|project| supports two types of Ceph deployments. An "internal" Ceph
deployment is one where a Ceph cluster is deployed as part of the
overcloud as described in :doc:`ceph_config`. An "external" Ceph
deployment is one where a Ceph cluster already exists and an overcloud
is configured to be a client of that Ceph cluster as described in
:doc:`ceph_external`. Ceph external deployments have special meaning
to |project| in the following ways:

- The Ceph cluster was not deployed by |project|
- The OpenStack Ceph client is configured by |project|

The deployment example in this document uses the "external" term to
focus on the second of the above because the client configuration is
important. This example differs from the first of the above because
Ceph was deployed by |project|, however relative to other stacks, it
is an external Ceph cluster because, for the stacks which configure
the Ceph clients, it doesn't matter that the Ceph server came from a
different stack. In this sense, the example in this document uses both
types of deployments as described in the following sequence:

- The central site deploys an internal Ceph cluster called central
  with an additional cephx keyring which may be used to access the
  central ceph pools.
- The dcn0 site deploys an internal Ceph cluster called dcn0 with an
  additional cephx keyring which may be used to access the dcn0 Ceph
  pools. During the same deployment the dcn0 site is also configured
  with the cephx keyring from the previous step so that it is also a
  client of the external Ceph cluster, relative to dcn0, called
  central from the previous step. The `GlanceMultistoreConfig`
  parameter is also used during this step so that Glance will use the
  dcn0 Ceph cluster as an RBD store by default but it will also be
  configured to use the central Ceph cluster as an additional RBD
  backend.
- The dcn1 site is deployed the same way as the dcn0 site and the
  pattern may be continued for as many DCN sites as necessary.
- The central site is then updated so that in addition to having an
  internal Ceph deployment for the cluster called central, it is also
  configured with multiple external ceph clusters, relative to the
  central site, for each DCN site. This is accomplished by passing
  the cephx keys which were created during each DCN site deployment
  as input to the stack update. During the stack update the
  `GlanceMultistoreConfig` parameter is added so that Glance will
  continue to use the central Ceph cluster as an RBD store by
  default but it will also be configured to use each DCN Ceph cluster
  as an additional RBD backend.

The above sequence is possible by using the `CephExtraKeys` parameter
as described in :doc:`ceph_config` and the `CephExternalMultiConfig`
parameter described in :doc:`ceph_external`.

Deployment Steps
----------------

This section shows the deployment commands and associated environment
files of an example DCN deployment with distributed image
management. It is based on the :doc:`distributed_compute_node`
example and does not cover redundant aspects of it such as networking.

Create extra Ceph key
^^^^^^^^^^^^^^^^^^^^^

Create ``/home/stack/control-plane/ceph_keys.yaml`` with contents like
the following::

  parameter_defaults:
    CephExtraKeys:
        - name: "client.external"
          caps:
            mgr: "allow *"
            mon: "profile rbd"
            osd: "profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=images"
          key: "AQD29WteAAAAABAAphgOjFD7nyjdYe8Lz0mQ5Q=="
          mode: "0600"

The key should be considered sensitive and may be randomly generated
with the following command::

  python3 -c 'import os,struct,time,base64; key = os.urandom(16); header = struct.pack("<hiih", 1, int(time.time()), 0, len(key)) ; print(base64.b64encode(header + key).decode())'

Passing `CephExtraKeys`, as above, during deployment will result in a
Ceph cluster with pools which may be accessed by the cephx user
"client.external". The same parameters will be used later when the
DCN overclouds are configured as external Ceph clusters. For more
information on the `CephExtraKeys` parameter see the document
:doc:`ceph_config` section called `Configuring CephX Keys`.

Create control-plane roles
^^^^^^^^^^^^^^^^^^^^^^^^^^

Generate the roles used for the deployment::

  openstack overcloud roles generate Controller ComputeHCI -o ~/control-plane/control_plane_roles.yaml

To determine the number of nodes per role create
``~/control-plane/roles-counts.yaml`` with the following::

  parameter_defaults:
    ControllerCount: 3
    ComputeHCICount: 3

If you do not wish to hyper-converge the compute nodes with Ceph OSD
services, then substitute `CephStorage` for `ComputeHCI` and increment
the number of `Compute` nodes. There should at least three
`Controller` nodes and at least three `CephStorage` or `ComputeHCI`
nodes in order to have a redundant Ceph cluster.

Deploy the control-plane stack
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Deploy the control-plane stack::

  openstack overcloud deploy \
         --stack control-plane \
         --templates /usr/share/openstack-tripleo-heat-templates/ \
         -r ~/control-plane/control_plane_roles.yaml \
         -n ~/network-data.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/net-multiple-nics.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/disable-swift.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
         -e ~/control-plane/role-counts.yaml \
         -e ~/control-plane/ceph.yaml \
         -e ~/control-plane/ceph_keys.yaml

The network related files are included to make the example complete
but are not displayed in this document. For more information on
configuring networks with distributed compute nodes see
:doc:`distributed_compute_node`.

The ``environments/ceph-ansible/ceph-ansible.yaml`` results in
ceph-ansible deploying Ceph as part of the ``control-plane`` stack.
This file also contains both `NovaEnableRbdBackend: true` and
`GlanceBackend: rbd`. When both of these settings are used, the Glance
`image_import_plugins` setting will contain `image_conversion`. With
this setting enabled commands like `glance image-create-via-import`
with `--disk-format qcow2` will result in the image being converted
into a raw format, which is optimal for the Ceph RBD driver. If
you need to disable image conversion you may override the
`GlanceImageImportPlugin` parameter. For example::

   parameter_defaults:
     GlanceImageImportPlugin: []

The ``ceph.yaml`` file contains the following which sets the name of
the Ceph cluster to "central"::

  parameter_defaults:
    CephClusterName: central

The ``ceph.yaml`` file should also contain additional parameters like
`CephAnsibleDisksConfig`, `CephPoolDefaultSize`,
`CephPoolDefaultPgNum` to configure the Ceph cluster relative to the
available hardware as described in :doc:`ceph_config`.

The ``environments/disable-swift.yaml`` file was passed to disable
Swift simply because an object storage system is not needed for this
example. However, if an object storage system is desired at the
Central site, substitute ``environments/ceph-ansible/ceph-rgw.yaml``
in its place to configure Ceph RGW.

The ``environments/cinder-backup.yaml`` file is not used in this
deployment. It's possible to enable the Cinder-backup service by using
this file but it will only write to the backups pool of the central
Ceph cluster.

The ``~/control-plane/ceph_keys.yaml`` and
``~/control-plane/role-counts.yaml`` files were created in the
previous sections.

Extract overcloud control-plane and Ceph configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Use the `openstack overcloud export` command to create
``~/control-plane-export.yaml`` as described in
:doc:`distributed_compute_node`::

  openstack overcloud export \
          --config-download-dir /var/lib/mistral/control-plane/ \
          --stack control-plane \
          --output-file ~/control-plane-export.yaml

In the above example `--config-download-dir` may be at a different
location if you deployed with a manual config-download as described in
:doc:`../deployment/ansible_config_download`.

Create ``~/central_ceph_external.yaml`` with content like the following::

  parameter_defaults:
    CephExternalMultiConfig:
      - cluster: "central"
        fsid: "3161a3b4-e5ff-42a0-9f53-860403b29a33"
        external_cluster_mon_ips: "172.16.11.84, 172.16.11.87, 172.16.11.92"
        keys:
          - name: "client.external"
            caps:
              mgr: "allow *"
              mon: "profile rbd"
              osd: "profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=images"
            key: "AQD29WteAAAAABAAphgOjFD7nyjdYe8Lz0mQ5Q=="
            mode: "0600"
        dashboard_enabled: false
        ceph_conf_overrides:
          client:
            keyring: /etc/ceph/central.client.external.keyring

The `CephExternalMultiConfig` section of the above is used to
configure any DCN node as a Ceph client of the central Ceph
cluster. All of the values, except `external_cluster_mon_ips`, for
this section may be obtained from the directory specified by
`--config-download-dir` when the `openstack overcloud export` command
was run. Based on the example provided in this document, the relevant
file with the desired values is
``/var/lib/mistral/control-plane/ceph-ansible/group_vars/all.yml``.

For example, the `fsid` and `cluster` (name) may be found like this::

  cd /var/lib/mistral/control-plane/ceph-ansible/group_vars/
  grep fsid: all.yml
  grep name: all.yml

The `keys` section should contain the same list item that was passed
to the `CephExtraKeys` parameter in the first step of this procedure.
The key value may also be obtained by looking at the key used by the
`client.external` name in the `openstack_keys` list found in
``all.yml``. For example, ``all.yml`` should contain something which
looks like the following and the second item in the list should be
used to get the value because it has `name` set to `client.external`::

  openstack_keys:
  -   caps:
        mgr: allow *
        mon: profile rbd
        osd: profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=images
      key: AQDIl2teAAAAABAAtFuRHdcS8v3+kk9Y6RzehA==
      mode: '0600'
      name: client.openstack
  -   caps:
        mgr: allow *
        mon: profile rbd
        osd: profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=images
      key: AQD29WteAAAAABAAphgOjFD7nyjdYe8Lz0mQ5Q==
      mode: '0600'
      name: client.external

To determine the value for `external_cluster_mon_ips` to provide
within the `CephExternalMultiConfig` parameter, use the inventory
generated by config-download and select the storage IP or storage
hostname of every node which runs the `CephMons` service. Based on the
example provided in this document, the relevant inventory file is
``/var/lib/mistral/control-plane/inventory.yaml``. For example the
nodes in the `Controller` role run the `CephMon` service by default
on IPs in the storage network so the IPs will be in a section which
looks like this::

  Controller:
    hosts:
      control-plane-controller-0:
        ansible_host: 192.168.24.16
        ...
        storage_hostname: control-plane-controller-0.storage.localdomain
        storage_ip: 172.16.11.84
        ...
      control-plane-controller-1:
        ansible_host: 192.168.24.22
        ...
        storage_hostname: control-plane-controller-1.storage.localdomain
        storage_ip: 172.16.11.87
        ...

The `storage_ip` from each host should be combined in a comma delimited
list. In this example the parameter is set to
`external_cluster_mon_ips: "172.16.11.84, 172.16.11.87,
172.16.11.92"`. If necessary, the inventory may be regenerated by
running the `tripleo-ansible-inventory` command as described in
:doc:`../deployment/ansible_config_download`.

The `ceph_conf_overrides` section should look like the following::

        ceph_conf_overrides:
          client:
            keyring: /etc/ceph/central.client.external.keyring

The above will result in the following lines in
``/etc/ceph/central.conf`` on all DCN nodes which interact with
the central Ceph cluster::

  [client]
  keyring = /etc/ceph/central.client.external.keyring

The name of the external Ceph cluster, relative to the DCN nodes,
is `central` so the relevant Ceph configuration file is called
``/etc/ceph/central.conf``. Optionally, the path to the key may be
confirmed by looking directly on any node running the `CephMon`
service on the control-plane stack if desired. If the conventions in
this document are followed, then it should remain consistent. This
directive is necessary so that the Glance service on all DCN nodes,
which will be deployed in the next section, knows which keyring to use
when connecting to the central Ceph cluster.

It is necessary to always pass `dashboard_enabled: false` when using
`CephExternalMultiConfig` as the Ceph dashboard cannot be deployed
when configuring an overcloud as a client of an external Ceph cluster.

For more information on the `CephExternalMultiConfig` parameter see
:doc:`ceph_external`.

Create extra Ceph key for dcn0
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Create ``~/dcn0/ceph_keys.yaml`` with content like the following::

  parameter_defaults:
    CephExtraKeys:
      - name: "client.external"
        caps:
          mgr: "allow *"
          mon: "profile rbd"
          osd: "profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=images"
        key: "AQBO/mteAAAAABAAc4mVMTpq7OFtrPlRFqN+FQ=="
        mode: "0600"

The `CephExtraKeys` section of the above should follow the same
pattern as the first step of this procedure. It should use a
new key, which should be considered sensitive and can be randomly
generated with the same Python command from the first step. This same
key will be used later when Glance on the central site needs to
connect to dcn0 "images".

Override Glance defaults for dcn0
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Create ``~/dcn0/glance.yaml`` with content like the following::

  parameter_defaults:
    GlanceShowMultipleLocations: true
    GlanceEnabledImportMethods: web-download,copy-image
    GlanceBackend: rbd
    GlanceStoreDescription: 'dcn0 rbd glance store'
    GlanceMultistoreConfig:
      central:
        GlanceBackend: rbd
        GlanceStoreDescription: 'central rbd glance store'
        CephClientUserName: 'external'
        CephClusterName: central

The `GlanceEnabledImportMethods` parameter is used to override the
default of 'web-download' to also include 'copy-image', which is
necessary to support the workflow described earlier.

By default Glance on the dcn0 node will use the RBD store of the
dcn0 Ceph cluster. The `GlanceMultistoreConfig` parameter is then used
to add an additional store of type RBD called `central` which uses
the Ceph cluster deployed by the control-plane stack so the
`CephClusterName` is set to "central". The `CephClientUserName` is set
to "external" because the additional key which was passed with
`CephExtraKeys` to the control-plane stack had a name of
"client.external".

Create DCN roles for dcn0
^^^^^^^^^^^^^^^^^^^^^^^^^

Generate the roles used for the deployment::

  openstack overcloud roles generate DistributedComputeHCI DistributedComputeHCIScaleOut -o ~/dcn0/dcn_roles.yaml

The `DistributedComputeHCI` role includes the default compute
services, the cinder volume service, and also includes the Ceph Mon,
Mgr, and OSD services for deploying a Ceph cluster at the distributed
site. Using this role, both the compute services and Ceph services are
deployed on the same nodes, enabling a hyper-converged infrastructure
for persistent storage at the distributed site. When Ceph is used,
there must be a minimum of three `DistributedComputeHCI` nodes. This
role also includes a Glance server, provided by the `GlanceApiEdge`
service with in the `DistributedComputeHCI` role. The Nova compute
service of each node in the `DistributedComputeHCI` role is configured
by default to use its local Glance server.

`DistributedComputeHCIScaleOut` role is like the `DistributedComputeHCI`
role but does not run the Ceph Mon and Mgr service. It offers the Ceph
OSD service however, so it may be used to scale up storage and compute
services at each DCN site after the minimum of three
`DistributedComputeHCI` nodes have been deployed. There is no
`GlanceApiEdge` service in the `DistributedComputeHCIScaleOut` role but
in its place the Nova compute service of the role is configured by
default to connect to a local `HaProxyEdge` service which in turn
proxies image requests to the Glance servers running on the
`DistributedComputeHCI` roles.

To determine the number of each nodes create ``~/dcn0/roles-counts.yaml``
with the following::

  parameter_defaults:
    ControllerCount: 0
    ComputeCount: 0
    DistributedComputeHCICount: 3
    DistributedComputeHCIScaleOutCount: 1

If you do not wish to hyper-converge the compute nodes with Ceph OSD
services, then substitute `DistributedCompute` for
`DistributedComputeHCI`, `DistributedComputeScaleOut` for
`DistributedComputeHCIScaleOut`, and add `CephStorage` nodes. The
`DistributedCompute` role contains the `GlanceApiEdge` service so that
the Compute service uses its the local Glance and local Ceph server at
the dcn0 site. The `DistributedComputeScaleOut` contains the
`HAproxyEdge` service so that any compute instances booting on the
`DistributedComputeScaleOut` node proxy their request for images to the
Glance services running on the `DistributedCompute` nodes. It is only
necessary to deploy the `ScaleOut` roles if more than three
`DistributedComputeHCI` or `DistributedCompute` nodes are necessary.
Unlike the `DistributedComputeHCI` role, there is no minimum number of
`DistributedCompute` required.

Deploy the dcn0 stack
^^^^^^^^^^^^^^^^^^^^^

Deploy the dcn0 stack::

    openstack overcloud deploy \
         --stack dcn0 \
         --templates /usr/share/openstack-tripleo-heat-templates/ \
         -r ~/dcn0/dcn_roles.yaml \
         -n ~/network-data.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/net-multiple-nics.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
         -e /usr/share/openstack-tripleo-heat-templates/environments/dcn-hci.yaml \
         -e ~/control-plane-export.yaml \
         -e ~/central_ceph_external.yaml \
         -e ~/dcn0/dcn_ceph_keys.yaml \
         -e ~/dcn0/role-counts.yaml \
         -e ~/dcn0/ceph.yaml \
         -e ~/dcn0/az.yaml \
         -e ~/dcn0/glance.yaml

The network related files are included to make the example complete
but are not displayed in this document. For more information on
configuring networks with distributed compute nodes see
:doc:`distributed_compute_node`.

The ``environments/cinder-volume-active-active.yaml`` file is NOT used
to configure Cinder active/active on the DCN site because
``environments/dcn-hci.yaml`` contains the same parameters. The
``environments/dcn-hci.yaml`` file is also used to configure the
`GlanceApiEdge` and `HAproxyEdge` edge services. If you are not using
hyper-converged Ceph, then use ``environments/dcn.yaml`` instead.
Both ``environments/dcn-hci.yaml`` and ``environments/dcn.yaml`` use
`NovaCrossAZAttach: False` to override the Nova configuration `[cinder]`
`cross_az_attach` setting from its default of `true`. This setting
should be `false` for all nodes in the dcn0 stack so that volumes
attached to an instance must be in the same availability zone in
Cinder as the instance availability zone in Nova. This is useful when
booting an instance from a volume on DCN nodes because Nova will
attempt to create a volume using the same availability zone as what is
assigned to the instance.

The ``~/dcn0/ceph.yaml`` file contains the following which sets the
name of the ceph cluster to "dcn0"::

  parameter_defaults:
    CephClusterName: dcn0

The ``~/dcn0/ceph.yaml`` file should also contain additional
parameters like `CephAnsibleDisksConfig`, `CephPoolDefaultSize`,
`CephPoolDefaultPgNum` to configure the Ceph cluster relative to
the available hardware as described in :doc:`ceph_config`.

The ``~/dcn0/az.yaml`` file contains the following::

  parameter_defaults:
    ManageNetworks: false
    NovaComputeAvailabilityZone: dcn0
    CinderStorageAvailabilityZone: dcn0
    CinderVolumeCluster: dcn0

`CinderVolumeCluster` is the name of the Cinder active/active cluster
which is deployed per DCN site. The above setting overrides the
default of "dcn" to "dcn0" found in `environments/dcn-hci.yaml`. See
:doc:`distributed_compute_node` for details on the other parameters
above.

The ``~/control-plane-export.yaml``, ``~/dcn0/dcn_ceph_keys.yaml``,
``~/dcn0/glance.yaml``, and ``role-counts.yaml`` files were created in
the previous steps. The ``~/central_ceph_external.yaml`` file should
also have been created in a previous step. Deployment with this file
is only necessary if images on DCN sites will be pushed back to the
central site so that they may then be shared with other DCN sites.
This may be useful for sharing snapshots between sites.

Deploy additional DCN sites
^^^^^^^^^^^^^^^^^^^^^^^^^^^

All of the previous sections which were done for dcn0 may be repeated
verbatim except with "dcn1" substituted for "dcn0" and a new cephx key
should be generated for each DCN site as described under `Create extra
Ceph key`. Other than that, the same process may be continued to
deploy as many DCN sites as needed. Once all of the desired DCN sites
have been deployed proceed to the next section. The
``~/control-plane-export.yaml`` and ``~/central_ceph_external.yaml``
which were created earlier may be reused for each DCN deployment and
do not need to be recreated. The roles in the previous section were
created specifically for dcn0 to allow for variations between DCN
sites.

Update central site to use additional Ceph clusters as Glance stores
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Once all of the desired DCN sites are deployed the central site needs
to be updated so that the central Glance service may push images to
the DCN sites.

In this example only one additional DCN site, dcn1, has been deployed
as indicated by the list of undercloud Heat stacks::

  $ openstack stack list -c "Stack Name" -c "Stack Status"
  +---------------+-----------------+
  | Stack Name    | Stack Status    |
  +---------------+-----------------+
  | dcn1          | CREATE_COMPLETE |
  | dcn0          | CREATE_COMPLETE |
  | control-plane | CREATE_COMPLETE |
  +---------------+-----------------+
  $

Create ``~/control-plane/glance_update.yaml`` with content like the
following::

  parameter_defaults:
    GlanceShowMultipleLocations: true
    GlanceEnabledImportMethods: web-download,copy-image
    GlanceBackend: rbd
    GlanceStoreDescription: 'central rbd glance store'
    CephClusterName: central
    GlanceMultistoreConfig:
      dcn0:
        GlanceBackend: rbd
        GlanceStoreDescription: 'dcn0 rbd glance store'
        CephClientUserName: 'external'
        CephClusterName: dcn0
      dcn1:
        GlanceBackend: rbd
        GlanceStoreDescription: 'dcn1 rbd glance store'
        CephClientUserName: 'external'
        CephClusterName: dcn1

The above will configure the Glance service running on the Controllers
to use two additional stores called "dcn0" and "dcn1".

Create ``~/control-plane/dcn_ceph_external.yaml`` with content like the
following::

  parameter_defaults:
    CephExternalMultiConfig:
      - cluster: "dcn0"
        fsid: "539e2b96-316e-4c23-b7df-035a3037ddd1"
        external_cluster_mon_ips: "172.16.11.61, 172.16.11.64, 172.16.11.66"
        keys:
          - name: "client.external"
            caps:
              mgr: "allow *"
              mon: "profile rbd"
              osd: "profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=images"
            key: "AQBO/mteAAAAABAAc4mVMTpq7OFtrPlRFqN+FQ=="
            mode: "0600"
        dashboard_enabled: false
        ceph_conf_overrides:
          client:
            keyring: /etc/ceph/dcn0.client.external.keyring
      - cluster: "dcn1"
        fsid: "7504a91e-5a0f-4408-bb55-33c3ee2c67e9"
        external_cluster_mon_ips: "172.16.11.182, 172.16.11.185, 172.16.11.187"
        keys:
          - name: "client.external"
            caps:
              mgr: "allow *"
              mon: "profile rbd"
              osd: "profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=images"
            key: "AQACCGxeAAAAABAAHocX/cnygrVnLBrKiZHJfw=="
            mode: "0600"
        dashboard_enabled: false
        ceph_conf_overrides:
          client:
            keyring: /etc/ceph/dcn1.client.external.keyring

The `CephExternalMultiConfig` section of the above is used to
configure the Glance service at the central site as a Ceph client of
all of the Ceph clusters of the DCN sites; that is "dcn0" and "dcn1"
in this example. This will be possible because the central nodes will
have the following files created:

- /etc/ceph/dcn0.conf
- /etc/ceph/dcn0.client.external.keyring
- /etc/ceph/dcn1.conf
- /etc/ceph/dcn1.client.external.keyring

All of the values under `CephExternalMultiConfig`, except
`external_cluster_mon_ips`, for this section may be obtained from the
config-download directory as described in
:doc:`../deployment/ansible_config_download`. Based on the examples in
this document the relevant files with the desired values are
``/var/lib/mistral/dcn0/ceph-ansible/group_vars/all.yml`` and
``/var/lib/mistral/dcn1/ceph-ansible/group_vars/all.yml``.

For example, the `fsid` and `cluster` (name) for dcn0 may be found
like this::

  cd /var/lib/mistral/dcn0/ceph-ansible/group_vars/
  grep fsid: all.yml
  grep name: all.yml

The `keys` section for dcn0 should contain the same list item that was
passed to the `CephExtraKeys` parameter in an earlier step of this
procedure. The key value may also be obtained by looking at the key
used by the `client.external` name in the `openstack_keys` list found in
``all.yml``. For example, ``all.yml`` should contain something which
looks like the following and the second item in the list should be
used to get the value because it has `name` set to `client.external`::

  openstack_keys:
  -   caps:
          mgr: allow *
          mon: profile rbd
          osd: profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=images
      key: AQB7/mteAAAAABAAZzufVwFpSN4Hg2TCsR5AfA==
      mode: '0600'
      name: client.openstack
  -   caps:
          mgr: allow *
          mon: profile rbd
          osd: profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=images
      key: AQBO/mteAAAAABAAc4mVMTpq7OFtrPlRFqN+FQ==
      mode: '0600'
      name: client.external

To determine the value for `external_cluster_mon_ips` to provide
within the `CephExternalMultiConfig` parameter for dcn0, use the
inventory generated by config-download and select the storage IP or
storage hostname of every node which runs the `CephMons` service.
Based on the example provided in this document, the relevant inventory
file is ``/var/lib/mistral/dcn0/inventory.yaml``. For example
the nodes in the `DistributedComputeHCI` role run the `CephMon`
service by default on IPs in the storage network so the IPs will be in
a section which looks like this::

  DistributedComputeHCI:
    hosts:
      dcn0-distributedcomputehci-0:
        ansible_host: 192.168.24.20
        ...
        storage_hostname: dcn0-distributedcomputehci-0.storage.localdomain
        storage_ip: 172.16.11.61
        ...
      dcn0-distributedcomputehci-1:
        ansible_host: 192.168.24.25
        ...
        storage_hostname: dcn0-distributedcomputehci-1.storage.localdomain
        storage_ip: 172.16.11.64
        ...

The `storage_ip` from each host should be combined in a comma delimited
list. In this example the parameter is set to
`external_cluster_mon_ips: "172.16.11.61, 172.16.11.64, 172.16.11.66"`.
If necessary, the inventory may be regenerated by running the
`tripleo-ansible-inventory` command as described in
:doc:`../deployment/ansible_config_download`.

The `ceph_conf_overrides` section should look like the following::

        ceph_conf_overrides:
          client:
            keyring: /etc/ceph/dcn0.client.external.keyring

The above will result in the following lines in
``/etc/ceph/dcn0.conf`` on the central nodes which interact with
the dcn0 Ceph cluster::

  [client]
  keyring = /etc/ceph/dcn0.client.external.keyring

The name of the external Ceph cluster, relative to the central node,
is `dcn0` so the relevant Ceph configuration file is called
``/etc/ceph/dcn0.conf``. Optionally, the path to the key may be
confirmed by looking directly on any node running the `CephMon`
service on the dcn0 stack if desired. If the conventions in this
document are followed, then it should remain consistent. This
directive is necessary so that the Glance service on central site
knows which keyring to use when connecting to the central Ceph
cluster.

It is necessary to always pass `dashboard_enabled: false` when using
`CephExternalMultiConfig` as the Ceph dashboard cannot be deployed
when configuring an overcloud as a client of an external Ceph cluster.

The second item in the `CephExternalMultiConfig` list which starts
with `cluster: "dcn1"` may have its values determined exactly as they
were determined for `cluster: "dcn0"`, except the relevant data should
come from ``/var/lib/mistral/dcn1/ceph-ansible/group_vars/all.yml``
and ``/var/lib/mistral/dcn1/inventory.yaml``. The same pattern may be
continued for additional DCN sites which the central site wishes to
use as an additional Glance store.

For more information on the `CephExternalMultiConfig` parameter see
:doc:`ceph_external`.

The number of lines in the ``~/control-plane/glance_update.yaml`` and
``~/control-plane/glance_update.yaml`` files will be proportional to
the number of DCN sites deployed.

Run the same `openstack overcloud deploy --stack control-plane ...`
command which was run in the previous section but also include the
the ``~/control-plane/glance_update.yaml`` and
``~/control-plane/dcn_ceph_external.yaml`` files with a `-e`. When the
stack update is complete, proceed to the next section.

Confirm images may be copied between sites
------------------------------------------

Ensure you have Glance 3.0.0 or newer as provided by the
`python3-glanceclient` RPM:

.. code-block:: bash

  $ glance --version
  3.0.0

Authenticate to the control-plane using the RC file generated
by the stack from the first deployment which contains Keystone.
In this example the stack was called "control-plane" so the file
to source beofre running Glance commands will be called
"control-planerc".

Confirm the expected stores are available:

.. code-block:: bash

  $ glance stores-info
  +----------+----------------------------------------------------------------------------------+
  | Property | Value                                                                            |
  +----------+----------------------------------------------------------------------------------+
  | stores   | [{"default": "true", "id": "default_backend", "description": "central rbd glance |
  |          | store"}, {"id": "http", "read-only": "true"}, {"id": "dcn0", "description":      |
  |          | "dcn0 rbd glance store"}, {"id": "dcn1", "description": "dcn1 rbd glance         |
  |          | store"}]                                                                         |
  +----------+----------------------------------------------------------------------------------+

Create an image and import it into the default backend at central as
well as the dcn0 backend, by listing both stores with the `--stores`
option, as seen in the following example:

.. code-block:: bash

  glance --verbose image-create-via-import --disk-format qcow2 --container-format bare --name cirros --uri http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img --import-method web-download --stores default_backend,dcn0

.. note:: The example above assumes that Glance image format
          conversion is enabled. Thus, even though `--disk-format` is
          set to `qcow2`, which is the format of the image file, Glance
          will convert and store the image in raw format after it's
          uploaded because the raw format is the optimal setting for
          Ceph RBD. The conversion may be confirmed by running
          `glance image-show <ID> | grep disk_format` after the image
          is uploaded.

Set an environment variable to the ID of the newly created image:

.. code-block:: bash

  ID=$(openstack image show cirros -c id -f value)

Copy the image from the default store to the dcn1 store:

.. code-block:: bash

  glance image-import $ID --stores dcn1 --import-method copy-image

Confirm a copy of the image is in each store by looking at the image properties:

.. code-block:: bash

  $ openstack image show $ID | grep properties
  | properties       | direct_url='rbd://d25504ce-459f-432d-b6fa-79854d786f2b/images/8083c7e7-32d8-4f7a-b1da-0ed7884f1076/snap', locations='[{u'url': u'rbd://d25504ce-459f-432d-b6fa-79854d786f2b/images/8083c7e7-32d8-4f7a-b1da-0ed7884f1076/snap', u'metadata': {u'store': u'default_backend'}}, {u'url': u'rbd://0c10d6b5-a455-4c4d-bd53-8f2b9357c3c7/images/8083c7e7-32d8-4f7a-b1da-0ed7884f1076/snap', u'metadata': {u'store': u'dcn0'}}, {u'url': u'rbd://8649d6c3-dcb3-4aae-8c19-8c2fe5a853ac/images/8083c7e7-32d8-4f7a-b1da-0ed7884f1076/snap', u'metadata': {u'store': u'dcn1'}}]', os_glance_failed_import='', os_glance_importing_to_stores='', os_hash_algo='sha512', os_hash_value='b795f047a1b10ba0b7c95b43b2a481a59289dc4cf2e49845e60b194a911819d3ada03767bbba4143b44c93fd7f66c96c5a621e28dff51d1196dae64974ce240e', os_hidden='False', stores='default_backend,dcn0,dcn1' |

The `stores` key, which is the last item in the properties map is set
to 'default_backend,dcn0,dcn1'.

On further inspection the `direct_url` key is set to::

  rbd://d25504ce-459f-432d-b6fa-79854d786f2b/images/8083c7e7-32d8-4f7a-b1da-0ed7884f1076/snap

Which contains 'd25504ce-459f-432d-b6fa-79854d786f2b', the FSID of the
central Ceph cluster, the name of the pool, 'images', followed by
'8083c7e7-32d8-4f7a-b1da-0ed7884f1076', the Glance image ID and name
of the Ceph object.

The properties map also contains `locations` which is set to similar
RBD paths for the dcn0 and dcn1 cluster with their respective FSIDs
and pool names. Note that the Glance image ID is consistent in all RBD
paths.

If the image were deleted with `glance image-delete`, then the image
would be removed from all three RBD stores to ensure consistency.
However, if the glanceclient is >3.1.0, then an image may be deleted
from a specific store only by using a syntax like `glance
stores-delete --store <store_id> <image_id>`.

Optionally, run the following on any Controller node from the
control-plane stack:

.. code-block:: bash

  sudo podman exec ceph-mon-$(hostname) rbd --cluster central -p images ls -l

Run the following on any DistributedComputeHCI node from the dcn0 stack:

.. code-block:: bash

  sudo podman exec ceph-mon-$(hostname) rbd --id external --keyring /etc/ceph/dcn0.client.external.keyring --conf /etc/ceph/dcn0.conf -p images ls -l

Run the following on any DistributedComputeHCI node from the dcn1 stack:

.. code-block:: bash

  sudo podman exec ceph-mon-$(hostname) rbd --id external --keyring /etc/ceph/dcn1.client.external.keyring --conf /etc/ceph/dcn1.conf -p images ls -l

The results in all cases should produce output like the following::

  NAME                                      SIZE   PARENT FMT PROT LOCK
  8083c7e7-32d8-4f7a-b1da-0ed7884f1076      44 MiB          2
  8083c7e7-32d8-4f7a-b1da-0ed7884f1076@snap 44 MiB          2 yes

When an ephemeral instance is COW booted from the image a similar
command in the vms pool should show the same parent image:

.. code-block:: bash

  $ sudo podman exec ceph-mon-$(hostname) rbd --id external --keyring /etc/ceph/dcn1.client.external.keyring --conf /etc/ceph/dcn1.conf -p vms ls -l
  NAME                                      SIZE  PARENT                                           FMT PROT LOCK
  2b431c77-93b8-4edf-88d9-1fd518d987c2_disk 1 GiB images/8083c7e7-32d8-4f7a-b1da-0ed7884f1076@snap   2      excl
  $


Confirm image-based volumes may be booted as DCN instances
----------------------------------------------------------

An instance with a persistent root volume may be created on a DCN
site by using the active/active Cinder service at the DCN site.
Assuming the Glance image created in the previous step is available,
identify the image ID and pass it to `openstack volume create` with
the `--image` option to create a volume based on that image.

.. code-block:: bash

  IMG_ID=$(openstack image show cirros -c id -f value)
  openstack volume create --size 8 --availability-zone dcn0 pet-volume-dcn0 --image $IMG_ID

Once the volume is created identify its volume ID and pass it to
`openstack server create` with the `--volume` option. This example
assumes a flavor, key, security group and network have already been
created.

.. code-block:: bash

  VOL_ID=$(openstack volume show -f value -c id pet-volume-dcn0)
  openstack server create --flavor tiny --key-name dcn0-key --network dcn0-network --security-group basic --availability-zone dcn0 --volume $VOL_ID pet-server-dcn0

It is also possible to issue one command to have Nova ask Cinder
to create the volume before it boots the instance by passing the
`--image` and `--boot-from-volume` options as in the shown in the
example below:

.. code-block:: bash

  openstack server create --flavor tiny --image $IMG_ID --key-name dcn0-key --network dcn0-network --security-group basic --availability-zone dcn0 --boot-from-volume 4 pet-server-dcn0

The above will only work if the Nova `cross_az_attach` setting
of the relevant compute node is set to `false`. This is automatically
configured by deploying with `environments/dcn-hci.yaml`. If the
`cross_az_attach` setting is `true` (the default), then the volume
will be created from the image not in the dcn0 site, but on the
default central site (as verified with the `rbd` command on the
central Ceph cluster) and then the instance will fail to boot on the
dcn0 site. Even if `cross_az_attach` is `true`, it's still possible to
create an instance from a volume by using `openstack volume create`
and then `openstack server create` as shown earlier.

Optionally, after creating the volume from the image at the dcn0
site and then creating an instance from the existing volume, verify
that the volume is based on the image by running the `rbd` command
within a ceph-mon container on the dcn0 site to list the volumes pool.

.. code-block:: bash

  $ sudo podman exec ceph-mon-$HOSTNAME rbd --cluster dcn0 -p volumes ls -l
  NAME                                      SIZE  PARENT                                           FMT PROT LOCK
  volume-28c6fc32-047b-4306-ad2d-de2be02716b7 8 GiB images/8083c7e7-32d8-4f7a-b1da-0ed7884f1076@snap   2      excl
  $

The following commands may be used to create a Cinder snapshot of the
root volume of the instance.

.. code-block:: bash

  openstack server stop pet-server-dcn0
  openstack volume snapshot create pet-volume-dcn0-snap --volume $VOL_ID --force
  openstack server start pet-server-dcn0

In the above example the server is stopped to quiesce data for clean
a snapshot. The `--force` option is necessary when creating the
snapshot because the volume status will remain "in-use" even when the
server is shut down. When the snapshot is completed start the
server. Listing the contents of the volumes pool on the dcn0 Ceph
cluster should show the snapshot which was created and how it is
connected to the original volume and original image.

.. code-block:: bash

  $ sudo podman exec ceph-mon-$HOSTNAME rbd --cluster dcn0 -p volumes ls -l
  NAME                                                                                      SIZE  PARENT                                           FMT PROT LOCK
  volume-28c6fc32-047b-4306-ad2d-de2be02716b7                                               8 GiB images/8083c7e7-32d8-4f7a-b1da-0ed7884f1076@snap   2      excl
  volume-28c6fc32-047b-4306-ad2d-de2be02716b7@snapshot-a1ca8602-6819-45b4-a228-b4cd3e5adf60 8 GiB images/8083c7e7-32d8-4f7a-b1da-0ed7884f1076@snap   2 yes
  $

Confirm image snapshots may be created and copied between sites
---------------------------------------------------------------

A new image called "cirros-snapshot" may be created at the dcn0 site
from the instance created in the previous section by running the
following commands.

.. code-block:: bash

  NOVA_ID=$(openstack server show pet-server-dcn0 -f value -c id)
  openstack server stop $NOVA_ID
  openstack server image create --name cirros-snapshot $NOVA_ID
  openstack server start $NOVA_ID

In the above example the instance is stopped to quiesce data for clean
a snapshot image and is then restarted after the image has been
created. The output of `openstack image show $IMAGE_ID -f value -c
properties` should contain a JSON data structure whose key called
`stores` should only contain "dcn0" as that is the only store
which has a copy of the new cirros-snapshot image.

The new image may then by copied from the dcn0 site to the central
site, which is the default backend for Glance.

.. code-block:: bash

  IMAGE_ID=$(openstack image show cirros-snapshot -f value -c id)
  glance image-import $IMAGE_ID --stores default_backend --import-method copy-image

After the above is run the output of `openstack image show
$IMAGE_ID -f value -c properties` should contain a JSON data structure
whose key called `stores` should looke like "dcn0,default_backend" as
the image will also exist in the "default_backend" which stores its
data on the central Ceph cluster. The same image at the Central site
may then be copied to other DCN sites, booted in the vms or volumes
pool, and snapshotted so that the same process may repeat.