Document Ceph configuration overrides for cephadm

Also cover using environments/cephadm/cephadm-rbd-only.yaml

Change-Id: Iad18241963eddfb6400df01aebd99bacfeca0139

deploy-guide/source/features/cephadm.rst

@@ -77,10 +77,17 @@ appropriate environment file as in the example below::
   openstack overcloud deploy --templates \
     -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml
 
-Do not directly edit the `environments/cephadm/cephadm.yaml` file.
+If you only wish to deploy Ceph RBD without RGW then use the following
-If you wish to override the defaults, as described below in the
+variation of the above.
-sections starting with "Overriding", then place those overrides
+
-in a separate `cephadm-overrides.yaml` file and deploy like this::
+  openstack overcloud deploy --templates \
+    -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm-rbd-only.yaml
+
+Do not directly edit the `environments/cephadm/cephadm.yaml`
+or `cephadm-rbd-only.yaml` file. If you wish to override the defaults,
+as described below in the sections starting with "Overriding", then
+place those overrides in a separate `cephadm-overrides.yaml` file and
+deploy like this::
 
   openstack overcloud deploy --templates \
     -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \
@@ -116,7 +123,7 @@ this type of access in order to scale from more than one Ceph node.
 
 The deployment definition as described TripleO Heat Templates,
 e.g. which servers run which services according to composable
-roles, will be converted by the tripleo-ansible `ceph_spec_bootstrap`
+roles, will be converted by the tripleo-ansible `ceph_spec_bootstrap`_
 module into a `Ceph Service Specification`_ file. The module has the
 ability to do this based on the Ansible inventory generated by the
 `tripleo-ansible-inventory` but it can also generate the Ceph Service
@@ -151,11 +158,12 @@ as described in the :doc:`ceph_external` documentation.
 When `openstack overcloud deploy` is re-run in order to update
 the stack, the cephadm bootstrap process is not repeated because
 that process is only run if `cephadm list` returns an empty
-list. Thus, configuration changes to the running Ceph cluster
+list. Thus, configuration changes to the running Ceph cluster, outside
-should be made directly with `Ceph Orchestrator`_.
+of scale up as described below, should be made directly with `Ceph
+Orchestrator`_.
 
-Overriding Ceph Configuration Options
+Overriding Ceph Configuration Options during deployment
--------------------------------------
+-------------------------------------------------------
 
 To override the keys and values of the Ceph configuration
 database, which has been traditionally stored in the Ceph
@@ -185,6 +193,48 @@ monitor. The supported configuration groups are 'global', 'mon',
 'mgr', 'osd', 'mds', and 'client'. If no group is provided, then the
 default configuration group is 'global'.
 
+Overriding Server Configuration after deployment
+------------------------------------------------
+
+To make a Ceph *server* configuration change, after the cluster has
+been deployed, use the `ceph config command`_. Updating the parameters
+in `CephConfigOverrides` and running a stack update will not change
+the running Ceph server. The parameters in `CephConfigOverrides`
+are only applied to a new Ceph server at bootstrap. A
+'/etc/ceph/ceph.conf' file is not distributed to all Ceph servers and
+instead `Ceph's centralized configuration management`_ is used.
+
+A single '/etc/ceph/ceph.conf' file may be found on the bootstrap node.
+The directives under `CephConfigOverrides` are used to create a config
+file, e.g. ceph_bootstrap.conf, which is passed to `cephadm bootstrap`
+with `--config ceph_bootstrap.conf` so that those directives are
+applied to the new cluster at bootstrap. The option `--output-config
+/etc/ceph/ceph.conf` is also passed to the `cephadm bootstrap` command
+and that's what creates the `ceph.conf` on the bootstrap node. The
+name of the file is `ceph.conf` because the `CephClusterName`
+parameter defaults to "ceph". If `CephClusterName` was set to "foo",
+then the file would be called `/etc/ceph/foo.conf`.
+
+
+Overriding Client Configuration after deployment
+------------------------------------------------
+
+To make a Ceph *client* configuration change, update the parameters in
+`CephConfigOverrides` and run a stack update. This will not change the
+configuration for the Ceph servers (as described in the section
+above); only for the Ceph clients. Examples of Ceph clients include
+Nova compute containers, Cinder volume containers, Glance image
+containers, etc.
+
+The `CephConfigOverrides` directive updates all Ceph client
+configuration files on the overcloud in the `CephConfigPath` (which
+defaults to /var/lib/tripleo-config/ceph). The `CephConfigPath` is
+mounted on the client containers as `/etc/ceph`. The name of the
+configuration file is `ceph.conf` because the `CephClusterName`
+parameter defaults to "ceph". If `CephClusterName` was set to "foo",
+then the file would be called `/etc/ceph/foo.conf`.
+
+
 Overriding the Ceph Service Specification
 -----------------------------------------
 
@@ -251,8 +301,8 @@ available but with the new syntax. When using this option consider
 setting `CephDynamicSpec` to false and defining a custom specification
 which is passed to TripleO by setting the `CephSpecPath`.
 
-Overriding Ceph placement group values
+Overriding Ceph placement group values during deployment
---------------------------------------
+--------------------------------------------------------
 
 The default cephadm deployment as triggered by TripleO has
 `Autoscaling Placement Groups`_ enabled. Thus, it is not necessary to
@@ -561,7 +611,10 @@ the "Scaling Down" section of :doc:`../provisioning/baremetal_provision`.
 .. _`cephadm`: https://docs.ceph.com/en/latest/cephadm/index.html
 .. _`cleaning instructions in the Ironic documentation`: https://docs.openstack.org/ironic/latest/admin/cleaning.html
 .. _`Ceph Orchestrator`: https://docs.ceph.com/en/latest/mgr/orchestrator/
+.. _`ceph config command`: https://docs.ceph.com/en/latest/man/8/ceph/#config
+.. _`Ceph's centralized configuration management`: https://ceph.io/community/new-mimic-centralized-configuration-management/
 .. _`Ceph Service Specification`: https://docs.ceph.com/en/latest/cephadm/service-management/#orchestrator-cli-service-spec
+.. _`ceph_spec_bootstrap`: https://docs.openstack.org/tripleo-ansible/latest/modules/modules-ceph_spec_bootstrap.html
 .. _`Advanced OSD Service Specifications`: https://docs.ceph.com/en/latest/cephadm/osd/#drivegroups
 .. _`Autoscaling Placement Groups`: https://docs.ceph.com/en/latest/rados/operations/placement-groups/
 .. _`pgcalc`: http://ceph.com/pgcalc