openstack/tripleo-docs
Code Issues Proposed changes

Remove docs on deploying Ceph during overcloud deployment

Browse Source 
We now only support deploying Ceph before overcloud deployment
as described in deployed_ceph.rst.

Remove cephadm.rst and ceph_config.rst which describe deploying
Ceph during overcloud deployment. ceph_config.rst covered use
of ceph-ansible or puppet-ceph which was deprecated so it needs
to be removed. Relevant content on configuring Ceph during the
overcloud deploy from cephadm.rst was moved to deployed_ceph.rst.

Remove or replace references to ceph_config as appropriate.
The deprecated ceph_config documentation for puppet-ceph and
ceph-ansible is still available in wallaby branch [1].

[1] https://docs.openstack.org/project-deploy-guide/tripleo-docs/wallaby/features/ceph_config.html

Change-Id: I521b0855bc1476e715731f4ebe0d09bb4b39963d
This commit is contained in:
John Fulton 2022-06-03 17:42:16 -04:00
parent 16e104ee4c
commit 7a92221c4d
9 changed files with 850 additions and 2083 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
deploy-guide/source/features/backends.rst
View File

@ -10,9 +10,7 @@ OpenStack projects.
deploy_manila
cinder_custom_backend
cinder_netapp
cephadm
deployed_ceph
ceph_config
ceph_external
domain_specific_ldap_backends
swift_external

1020
deploy-guide/source/features/ceph_config.rst
View File

File diff suppressed because it is too large Load Diff

6
deploy-guide/source/features/ceph_external.rst
View File

@ -94,7 +94,7 @@ Do not use `CephExternalMultiConfig` when configuring an overcloud to
use only one external Ceph cluster. Instead follow the example in the
previous section. The example in the previous section and the method
of deploying an internal Ceph cluster documented in
:doc:`ceph_config` are mutually exclusive per Heat stack. The
:doc:`deployed_ceph` are mutually exclusive per Heat stack. The
following scenarios are the only supported ones in which
`CephExternalMultiConfig` may be used per Heat stack:
@ -102,7 +102,7 @@ following scenarios are the only supported ones in which
section, in addition to multiple external Ceph clusters configured
via `CephExternalMultiConfig`.
* One internal Ceph cluster, as described in :doc:`ceph_config` in
* One internal Ceph cluster, as described in :doc:`deployed_ceph` in
addition to multiple external ceph clusters configured via
`CephExternalMultiConfig`.
@ -211,7 +211,7 @@ overcloud to connect to an external ceph cluster:
ceph-ansible, then the deployer of that cluster could share that map
with the TripleO deployer so that it could be used as a list item of
`CephExternalMultiConfig`. Similarly, the `CephExtraKeys` parameter,
described in the :doc:`ceph_config` documentation, has the same
described in the :doc:`deployed_ceph` documentation, has the same
syntax.
Deploying Manila with an External CephFS Service

917
deploy-guide/source/features/cephadm.rst
View File

@ -1,917 +0,0 @@
Deploying Ceph with cephadm
===========================
TripleO can deploy and configure Ceph as if it was a composable
OpenStack service and configure OpenStack services like Nova, Glance,
Cinder, and Cinder Backup to use its RBD interface as a storage
backend as well as configure Ceph's RGW service as the backend for
OpenStack object storage. Both Ceph and OpenStack containers can also
run "hyperconverged" on the same container host.
This guide assumes that the undercloud is already installed and ready
to deploy an overcloud as described in :doc:`../deployment/index`.
Limitations
-----------
TripleO deployments of Ceph with cephadm_ are only supported in Wallaby
or newer. The default version of Ceph deployed by TripleO in Wallaby
is Pacific, regardless of if cephadm or ceph-ansible is used to deploy
it.
TripleO can only deploy one Ceph cluster in the overcloud per Heat
stack. However, within that Heat stack it's possible to configure
an overcloud to communicate with multiple Ceph clusters which are
external to the overcloud. To do this, follow this document to
configure the "internal" Ceph cluster which is part of the overcloud
and also use the `CephExternalMultiConfig` parameter described in the
:doc:`ceph_external` documentation.
Prerequisite: Ensure the Ceph container is available
----------------------------------------------------
Before deploying Ceph follow the
:ref:`prepare-environment-containers` documentation so
the appropriate Ceph container image is used.
The output of the `openstack tripleo container image prepare`
command should contain a line like the following::
ContainerCephDaemonImage: undercloud.ctlplane.mydomain.tld:8787/ceph-ci/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64
Prerequisite: Ensure the cephadm package is installed
-----------------------------------------------------
The `cephadm` package needs to be installed on at least one node in
the overcloud in order to bootstrap the first node of the Ceph
cluster.
The `cephadm` package is pre-built into the overcloud-full image.
The `tripleo_cephadm` role will also use Ansible's package module
to ensure it is present. If `tripleo-repos` is passed the `ceph`
argument for Wallaby or newer, then the CentOS SIG Ceph repository
will be enabled with the appropriate version containing the `cephadm`
package, e.g. for Wallaby the ceph-pacific repository is enabled.
Prerequisite: Ensure Disks are Clean
------------------------------------
cephadm does not reformat the OSD disks and expect them to be clean to
complete successfully. Consequently, when reusing the same nodes (or
disks) for new deployments, it is necessary to clean the disks before
every new attempt. One option is to enable the automated cleanup
functionality in Ironic, which will zap the disks every time that a
node is released. The same process can be executed manually or only
for some target nodes, see `cleaning instructions in the Ironic documentation`_.
Deploying Ceph During Overcloud Deployment
------------------------------------------
To deploy an overcloud with a Ceph include the appropriate environment
file as in the example below::
openstack overcloud deploy --templates \
-e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml
If you only wish to deploy Ceph RBD without RGW then use the following
variation of the above::
openstack overcloud deploy --templates \
-e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm-rbd-only.yaml
Do not directly edit the `environments/cephadm/cephadm.yaml`
or `cephadm-rbd-only.yaml` file. If you wish to override the defaults,
as described below in the sections starting with "Overriding", then
place those overrides in a separate `cephadm-overrides.yaml` file and
deploy like this::
openstack overcloud deploy --templates \
-e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \
-e cephadm-overrides.yaml
Deploying with the commands above will result in the processes described
in the rest of this document.
Deploying Ceph Before Overcloud Deployment
------------------------------------------
In Wallaby and newer it is possible to provision hardware and deploy
Ceph before deploying the overcloud on the same hardware. This feature
is called "deployed ceph" and it uses the command `openstack overcloud
ceph deploy` which executes the same Ansible roles described
below. For more details see :doc:`deployed_ceph`.
Overview of Ceph Deployment with TripleO and cephadm
----------------------------------------------------
When Ceph is deployed during overcloud configuration or when Ceph is
deployed before overcloud configuration with :doc:`deployed_ceph`,
TripleO will use Ansible automate the process described in the
`cephadm`_ documentation to bootstrap a new cluster. It will
bootstrap a single Ceph monitor and manager on one server
(by default on the first Controller node) and then add the remaining
servers to the cluster by using Ceph orchestrator to apply a `Ceph
Service Specification`_. The Ceph Service Specification is generated
automatically based on TripleO composable roles and most of the
existing THT parameters remain backwards compatible. During stack
updates the same bootstrap process is not executed.
Details of Ceph Deployment with TripleO and cephadm
---------------------------------------------------
After the hardware is provisioned, the user `ceph-admin` is created
on the overcloud nodes. The `ceph-admin` user has one set of public
and private SSH keys created on the undercloud (in
/home/stack/.ssh/ceph-admin-id_rsa.pub and .ssh/ceph-admin-id_rsa)
which is distributed to all overcloud nodes which host the Ceph
Mgr and Mon service; only the public key is distributed to nodes
in the Ceph cluster which do not run the Mgr or Mon service. Unlike
the `tripleo-admin` user, this allows the `ceph-admin` user to SSH
from any overcloud node hosting the Mon or Mgr service to any other
overcloud node hosting the Mon or Mgr service. By default these
services run on the controller nodes so this means by default that
Controllers can SSH to each other but other nodes, e.g. CephStorage
nodes, cannot SSH to Controller nodes. `cephadm`_ requires this type
of access in order to scale from more than one Ceph node.
The deployment definition as described TripleO Heat Templates,
e.g. which servers run which services according to composable
roles, will be converted by the tripleo-ansible `ceph_spec_bootstrap`_
module into a `Ceph Service Specification`_ file. The module has the
ability to do this based on the Ansible inventory generated by the
`tripleo-ansible-inventory`. When Ceph is deployed *during* overcloud
configuration by including the cephadm.yaml environment file, the
module uses the Ansible inventory to create the `Ceph Service
Specification`_. In this scenario the default location of the
generated Ceph Service Specification file is
`config-download/<STACK>/cephadm/ceph_spec.yaml`.
The same `ceph_spec_bootstrap`_ module can also generate the Ceph
Service Specification file from a combination of a TripleO roles data
file
(e.g. /usr/share/openstack-tripleo-heat-templates/roles_data.yaml)
and the output of the command
`openstack overcloud node provision --output deployed_metal.yaml`.
When Ceph is deployed *before* overcloud configuration as described in
:doc:`deployed_ceph`, the module uses the deployed_metal.yaml and
roles_data.yaml to create the `Ceph Service Specification`_.
After the `ceph-admin` user is created, `ceph_spec.yaml` is copied
to the bootstrap host. The bootstrap host will be the first host
in the `ceph_mons` group of the inventory generated by the
`tripleo-ansible-inventory` command. By default this is the first
controller node.
Ansible will then interact only with the bootstrap host. It will run
the `cephadm` commands necessary to bootstrap a small Ceph cluster on
the bootstrap node and then run `ceph orch apply -i ceph_spec.yaml`
and `cephadm` will use the `ceph-admin` account and SSH keys to add
the other nodes.
After the full Ceph cluster is running, either as a result of
:doc:`deployed_ceph` or by cephadm being triggered during the
overcloud deployment via the `cephadm.yaml` environment file, the
Ceph pools and the cephx keys to access the pools will be created as
defined or overridden as described in the Heat environment examples
below. The information necessary to configure Ceph clients will then
be extracted to `/home/stack/ceph_client.yml` on the undercloud and
passed to the as input to the tripleo-ansible role tripleo_ceph_client
which will then configure the rest of the overcloud to use the new
Ceph cluster as described in the :doc:`ceph_external` documentation.
When `openstack overcloud deploy` is re-run in order to update
the stack, the cephadm bootstrap process is not repeated because
that process is only run if `cephadm list` returns an empty
list. Thus, configuration changes to the running Ceph cluster, outside
of scale up as described below, should be made directly with `Ceph
Orchestrator`_.
Overriding Ceph Configuration Options during deployment
-------------------------------------------------------
To override the keys and values of the Ceph configuration
database, which has been traditionally stored in the Ceph
configuration file, e.g. `/etc/ceph/ceph.conf`, use the
`CephConfigOverrides` parameter. For example, if the
`cephadm-overrides.yaml` file referenced in the example `openstack
overcloud deploy` command in the previous section looked like the
following::
parameter_defaults:
CephConfigOverrides:
mon:
mon_warn_on_pool_no_redundancy: false
Then the Ceph monitors would be configured with the above parameter
and a command like the following could confirm it::
[stack@standalone ~]$ sudo cephadm shell -- ceph config dump | grep warn
Inferring fsid 65e8d744-eaec-4ff1-97be-2551d452426d
Inferring config /var/lib/ceph/65e8d744-eaec-4ff1-97be-2551d452426d/mon.standalone.localdomain/config
Using recent ceph image quay.ceph.io/ceph-ci/daemon@sha256:6b3c720e58ae84b502bd929d808ba63a1e9b91f710418be9df3ee566227546c0
mon advanced mon_warn_on_pool_no_redundancy false
[stack@standalone ~]$
In the above example the configuration group is 'mon' for the Ceph
monitor. The supported configuration groups are 'global', 'mon',
'mgr', 'osd', 'mds', and 'client'. If no group is provided, then the
default configuration group is 'global'.
The above does not apply to :doc:`deployed_ceph`.
Overriding Server Configuration after deployment
------------------------------------------------
To make a Ceph *server* configuration change, after the cluster has
been deployed, use the `ceph config command`_. A '/etc/ceph/ceph.conf'
file is not distributed to all Ceph servers and instead `Ceph's
centralized configuration management`_ is used.
A single '/etc/ceph/ceph.conf' file may be found on the bootstrap node.
The directives under `CephConfigOverrides` are used to create a config
file, e.g. assimilate_ceph.conf, which is passed to `cephadm bootstrap`
with `--config assimilate_ceph.conf` so that those directives are
applied to the new cluster at bootstrap. The option `--output-config
/etc/ceph/ceph.conf` is also passed to the `cephadm bootstrap` command
and that's what creates the `ceph.conf` on the bootstrap node. The
name of the file is `ceph.conf` because the `CephClusterName`
parameter defaults to "ceph". If `CephClusterName` was set to "foo",
then the file would be called `/etc/ceph/foo.conf`.
By default the parameters in `CephConfigOverrides` are only applied to
a new Ceph server at bootstrap. They are ignored during stack updates
because `ApplyCephConfigOverridesOnUpdate` defaults to false. When
`ApplyCephConfigOverridesOnUpdate` is set to true, parameters in
`CephConfigOverrides` are put into a file, e.g. assimilate_ceph.conf,
and a command like `ceph config assimilate-conf -i
assimilate_ceph.conf` is run.
When using :doc:`deployed_ceph` the `openstack overcloud ceph deploy`
command outputs an environment file with
`ApplyCephConfigOverridesOnUpdate` set to true so that services not
covered by deployed ceph, e.g. RGW, can have the configuration changes
that they need applied during overcloud deployment. After the deployed
ceph process has run and then after the overcloud is deployed, it is
recommended to set `ApplyCephConfigOverridesOnUpdate` to false.
Overriding Client Configuration after deployment
------------------------------------------------
To make a Ceph *client* configuration change, update the parameters in
`CephConfigOverrides` and run a stack update. This will not change the
configuration for the Ceph servers unless
`ApplyCephConfigOverridesOnUpdate` is set to true (as described in the
section above). By default it should only change configurations for
the Ceph clients. Examples of Ceph clients include Nova compute
containers, Cinder volume containers, Glance image containers, etc.
The `CephConfigOverrides` directive updates all Ceph client
configuration files on the overcloud in the `CephConfigPath` (which
defaults to /var/lib/tripleo-config/ceph). The `CephConfigPath` is
mounted on the client containers as `/etc/ceph`. The name of the
configuration file is `ceph.conf` because the `CephClusterName`
parameter defaults to "ceph". If `CephClusterName` was set to "foo",
then the file would be called `/etc/ceph/foo.conf`.
Overriding the Ceph Service Specification
-----------------------------------------
All TripleO cephadm deployments rely on a valid `Ceph Service
Specification`_. It is not necessary to provide a service
specification directly as TripleO will generate one dynamically.
However, one may provide their own service specification by disabling
the dynamic spec generation and providing a path to their service
specification as shown in the following::
parameter_defaults:
CephDynamicSpec: false
CephSpecPath: /home/stack/cephadm_spec.yaml
The `CephDynamicSpec` parameter defaults to true. The `CephSpecPath`
defaults to "{{ playbook_dir }}/cephadm/ceph_spec.yaml", where the
value of "{{ playbook_dir }}" is controlled by config-download.
If `CephDynamicSpec` is true and `CephSpecPath` is set to a valid
path, then the spec will be created at that path before it is used to
deploy Ceph.
The `CephDynamicSpec` and `CephSpecPath` parameters are not available
when using "deployed ceph", but the functionality is available via
the `--ceph-spec` command line option as described in
:doc:`deployed_ceph`.
Overriding which disks should be OSDs
-------------------------------------
The `Advanced OSD Service Specifications`_ should be used to define
how disks are used as OSDs.
By default all available disks (excluding the disk where the operating
system is installed) are used as OSDs. This is because the
`CephOsdSpec` parameter defaults to the following::
data_devices:
all: true
In the above example, the `data_devices` key is valid for any `Ceph
Service Specification`_ whose `service_type` is "osd". Other OSD
service types, as found in the `Advanced OSD Service
Specifications`_, may be set by overriding the `CephOsdSpec`
parameter. In the example below all rotating devices will be data
devices and all non-rotating devices will be used as shared devices
(wal, db) following::
parameter_defaults:
CephOsdSpec:
data_devices:
rotational: 1
db_devices:
rotational: 0
When the dynamic Ceph service specification is built (whenever
`CephDynamicSpec` is true) whatever is in the `CephOsdSpec` will
be appended to that section of the specification if the `service_type`
is "osd".
If `CephDynamicSpec` is false, then the OSD definition can also be
placed directly in the `Ceph Service Specification`_ located at the
path defined by `CephSpecPath` as described in the previous section.
The :doc:`node_specific_hieradata` feature is not supported by the
cephadm integration but the `Advanced OSD Service Specifications`_ has
a `host_pattern` parameter which specifies which host to target for
certain `data_devices` definitions, so the equivalent functionality is
available but with the new syntax. When using this option consider
setting `CephDynamicSpec` to false and defining a custom specification
which is passed to TripleO by setting the `CephSpecPath`.
The `CephOsdSpec` parameter is not available when using "deployed
ceph", but the same functionality is available via `--osd-spec`
command line option as described in :doc:`deployed_ceph`.
Overriding Ceph Pools and Placement Group values during deployment
------------------------------------------------------------------
The default cephadm deployment as triggered by TripleO has
`Autoscaling Placement Groups`_ enabled. Thus, it is not necessary to
use `pgcalc`_ and hard code a PG number per pool.
However, the interfaces described in the :doc:`ceph_config`
for configuring the placement groups per pool remain backwards
compatible. For example, to set the default pool size and default PG
number per pool use an example like the following::
parameter_defaults:
CephPoolDefaultSize: 3
CephPoolDefaultPgNum: 128
In addition to setting the default PG number for each pool created,
each Ceph pool created for OpenStack can have its own PG number.
TripleO supports customization of these values by using a syntax like
the following::
parameter_defaults:
CephPools:
- {"name": backups, "pg_num": 512, "pgp_num": 512, "application": rbd}
- {"name": volumes, "pg_num": 1024, "pgp_num": 1024, "application": rbd}
- {"name": vms, "pg_num": 512, "pgp_num": 512, "application": rbd}
- {"name": images, "pg_num": 128, "pgp_num": 128, "application": rbd}
Regardless of if the :doc:`deployed_ceph` feature is used, pools will
always be created during overcloud deployment as documented above.
Additional pools may also be created directly via the Ceph command
line tools.
Overriding CRUSH rules
----------------------
To deploy Ceph pools with custom `CRUSH Map Rules`_ use the
`CephCrushRules` parameter to define a list of named rules and
then associate the `rule_name` per pool with the `CephPools`
parameter::
parameter_defaults:
CephCrushRules:
- name: HDD
root: default
type: host
class: hdd
default: true
- name: SSD
root: default
type: host
class: ssd
default: false
CephPools:
- {'name': 'slow_pool', 'rule_name': 'HDD', 'application': 'rbd'}
- {'name': 'fast_pool', 'rule_name': 'SSD', 'application': 'rbd'}
Regardless of if the :doc:`deployed_ceph` feature is used, custom
CRUSH rules may be created during overcloud deployment as documented
above. CRUSH rules may also be created directly via the Ceph command
line tools.
Overriding CephX Keys
---------------------
TripleO will create a Ceph cluster with a CephX key file for OpenStack
RBD client connections that is shared by the Nova, Cinder, and Glance
services to read and write to their pools. Not only will the keyfile
be created but the Ceph cluster will be configured to accept
connections when the key file is used. The file will be named
`ceph.client.openstack.keyring` and it will be stored in `/etc/ceph`
within the containers, but on the container host it will be stored in
a location defined by a TripleO exposed parameter which defaults to
`/var/lib/tripleo-config/ceph`.
The keyring file is created using the following defaults:
* CephClusterName: 'ceph'
* CephClientUserName: 'openstack'
* CephClientKey: This value is randomly generated per Heat stack. If
it is overridden the recommendation is to set it to the output of
`ceph-authtool --gen-print-key`.
If the above values are overridden, the keyring file will have a
different name and different content. E.g. if `CephClusterName` was
set to 'foo' and `CephClientUserName` was set to 'bar', then the
keyring file would be called `foo.client.bar.keyring` and it would
contain the line `[client.bar]`.
The `CephExtraKeys` parameter may be used to generate additional key
files containing other key values and should contain a list of maps
where each map describes an additional key. The syntax of each
map must conform to what the `ceph-ansible/library/ceph_key.py`
Ansible module accepts. The `CephExtraKeys` parameter should be used
like this::
CephExtraKeys:
- name: "client.glance"
caps:
mgr: "allow *"
mon: "profile rbd"
osd: "profile rbd pool=images"
key: "AQBRgQ9eAAAAABAAv84zEilJYZPNuJ0Iwn9Ndg=="
mode: "0600"
If the above is used, in addition to the
`ceph.client.openstack.keyring` file, an additional file called
`ceph.client.glance.keyring` will be created which contains::
[client.glance]
key = AQBRgQ9eAAAAABAAv84zEilJYZPNuJ0Iwn9Ndg==
caps mgr = "allow *"
caps mon = "profile rbd"
caps osd = "profile rbd pool=images"
The Ceph cluster will also allow the above key file to be used to
connect to the images pool. Ceph RBD clients which are external to the
overcloud could then use this CephX key to connect to the images
pool used by Glance. The default Glance deployment defined in the Heat
stack will continue to use the `ceph.client.openstack.keyring` file
unless that Glance configuration itself is overridden.
Regardless of if the :doc:`deployed_ceph` feature is used, CephX keys
may be created during overcloud deployment as documented above.
Additional CephX keys may also be created directly via the Ceph
command line tools.
Enabling cephadm debug mode
---------------------------
TripleO can deploy the Ceph cluster enabling the cephadm backend in debug
mode; this is useful for troubleshooting purposes, and can be activated
by using a syntax like the following::
parameter_defaults:
CephAdmDebug: true
After step 2, when the Ceph cluster is up and running, after SSH'ing into
one of your controller nodes run::
sudo cephadm shell ceph -W cephadm --watch-debug
The command above shows a more verbose cephadm execution, and it's useful
to identify potential issues with the deployment of the Ceph cluster.
Accessing the Ceph Command Line
-------------------------------
After step 2 of the overcloud deployment is completed you can login to
check the status of your Ceph cluster. By default the Ceph Monitor
containers will be running on the Controller nodes. After SSH'ing into
one of your controller nodes run `sudo cephadm shell`. An example of
what you might see is below::
[stack@standalone ~]$ sudo cephadm shell
Inferring fsid 65e8d744-eaec-4ff1-97be-2551d452426d
Inferring config /var/lib/ceph/65e8d744-eaec-4ff1-97be-2551d452426d/mon.standalone.localdomain/config
Using recent ceph image quay.ceph.io/ceph-ci/daemon@sha256:6b3c720e58ae84b502bd929d808ba63a1e9b91f710418be9df3ee566227546c0
[ceph: root@standalone /]# ceph -s
cluster:
id: 65e8d744-eaec-4ff1-97be-2551d452426d
health: HEALTH_OK
services:
mon: 1 daemons, quorum standalone.localdomain (age 61m)
mgr: standalone.localdomain.saojan(active, since 61m)
osd: 1 osds: 1 up (since 61m), 1 in (since 61m)
rgw: 1 daemon active (1 hosts, 1 zones)
data:
pools: 8 pools, 201 pgs
objects: 315 objects, 24 KiB
usage: 19 MiB used, 4.6 GiB / 4.7 GiB avail
pgs: 201 active+clean
[ceph: root@standalone /]#
If you need to make updates to your Ceph deployment use the `Ceph
Orchestrator`_.
Scenario: Deploy Ceph with TripleO and Metalsmith
-------------------------------------------------
Deploy the hardware as described in :doc:`../provisioning/baremetal_provision`
and include nodes with in the `CephStorage` role. For example, the
following could be the content of ~/overcloud_baremetal_deploy.yaml::
- name: Controller
count: 3
instances:
- hostname: controller-0
name: controller-0
- hostname: controller-1
name: controller-1
- hostname: controller-2
name: controller-2
- name: CephStorage
count: 3
instances:
- hostname: ceph-0
name: ceph-0
- hostname: ceph-1
name: ceph-2
- hostname: ceph-2
name: ceph-2
- name: Compute
count: 1
instances:
- hostname: compute-0
name: compute-0
which is passed to the following command::
openstack overcloud node provision \
--stack overcloud \
--output ~/overcloud-baremetal-deployed.yaml \
~/overcloud_baremetal_deploy.yaml
If desired at this stage, then Ceph may be deployed early as described
in :doc:`deployed_ceph`. Otherwise Ceph may be deployed during the
overcloud deployment. Either way, as described in
:doc:`../provisioning/baremetal_provision`, pass
~/overcloud_baremetal_deploy.yaml as input, along with
/usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml
and cephadm-overrides.yaml described above, to the `openstack overcloud
deploy` command.
Scenario: Scale Up Ceph with TripleO and Metalsmith
---------------------------------------------------
Modify the ~/overcloud_baremetal_deploy.yaml file described above to
add more CephStorage nodes. In the example below the number of storage
nodes is doubled::
- name: CephStorage
count: 6
instances:
- hostname: ceph-0
name: ceph-0
- hostname: ceph-1
name: ceph-2
- hostname: ceph-2
name: ceph-2
- hostname: ceph-3
name: ceph-3
- hostname: ceph-4
name: ceph-4
- hostname: ceph-5
name: ceph-5
As described in :doc:`../provisioning/baremetal_provision`, re-run the
same `openstack overcloud node provision` command with the updated
~/overcloud_baremetal_deploy.yaml file. This will result in the three
new storage nodes being provisioned and output an updated copy of
~/overcloud-baremetal-deployed.yaml. The updated copy will have the
`CephStorageCount` changed from 3 to 6 and the `DeployedServerPortMap`
and `HostnameMap` will contain the new storage nodes.
After the three new storage nodes are deployed run the same
`openstack overcloud deploy` command as described in the previous
section with updated copy of ~/overcloud-baremetal-deployed.yaml.
The additional Ceph Storage nodes will be added to the Ceph and
the increased capacity will available.
In particular, the following will happen as a result of running
`openstack overcloud deploy`:
- The storage networks and firewall rules will be appropriately
configured on the new CephStorage nodes
- The ceph-admin user will be created on the new CephStorage nodes
- The ceph-admin user's public SSH key will be distributed to the new
CephStorage nodes so that cephadm can use SSH to add extra nodes
- If a new host with the Ceph Mon or Ceph Mgr service is being added,
then the private SSH key will also be added to that node.
- An updated Ceph spec will be generated and installed on the
bootstrap node, i.e. /home/ceph-admin/specs/ceph_spec.yaml on the
bootstrap node will contain new entries for the new CephStorage
nodes.
- The cephadm bootstrap process will be skipped because `cephadm ls`
will indicate that Ceph containers are already running.
- The updated spec will be applied and cephadm will schedule the new
nodes to join the cluster.
Scenario: Scale Down Ceph with TripleO and Metalsmith
-----------------------------------------------------
.. warning:: This procedure is only possible if the Ceph cluster has
the capacity to lose OSDs.
Before using TripleO to remove hardware which is part of a Ceph
cluster, use Ceph orchestrator to deprovision the hardware gracefully.
This example uses commands from the `OSD Service Documentation for
cephadm`_ to remove the OSDs, and their host, before using TripleO
to scale down the Ceph storage nodes.
Start a Ceph shell as described in "Accessing the Ceph Command Line"
above and identify the OSDs to be removed by server. In the following
example we will identify the OSDs of the host ceph-2::
[ceph: root@oc0-controller-0 /]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.58557 root default
... <redacted>
-7 0.19519 host ceph-2
5 hdd 0.04880 osd.5 up 1.00000 1.00000
7 hdd 0.04880 osd.7 up 1.00000 1.00000
9 hdd 0.04880 osd.9 up 1.00000 1.00000
11 hdd 0.04880 osd.11 up 1.00000 1.00000
... <redacted>
[ceph: root@oc0-controller-0 /]#
As per the example above the ceph-2 host has OSDs 5,7,9,11 which can
be removed by running `ceph orch osd rm 5 7 9 11`. For example::
[ceph: root@oc0-controller-0 /]# ceph orch osd rm 5 7 9 11
Scheduled OSD(s) for removal
[ceph: root@oc0-controller-0 /]# ceph orch osd rm status
OSD_ID HOST STATE PG_COUNT REPLACE FORCE DRAIN_STARTED_AT
7 ceph-2 draining 27 False False 2021-04-23 21:35:51.215361
9 ceph-2 draining 8 False False 2021-04-23 21:35:49.111500
11 ceph-2 draining 14 False False 2021-04-23 21:35:50.243762
[ceph: root@oc0-controller-0 /]#
Use `ceph orch osd rm status` to check the status::
[ceph: root@oc0-controller-0 /]# ceph orch osd rm status
OSD_ID HOST STATE PG_COUNT REPLACE FORCE DRAIN_STARTED_AT
7 ceph-2 draining 34 False False 2021-04-23 21:35:51.215361
11 ceph-2 done, waiting for purge 0 False False 2021-04-23 21:35:50.243762
[ceph: root@oc0-controller-0 /]#
Only proceed if `ceph orch osd rm status` returns no output.
Remove the host with `ceph orch host rm <HOST>`. For example::
[ceph: root@oc0-controller-0 /]# ceph orch host rm ceph-2
Removed host 'ceph-2'
[ceph: root@oc0-controller-0 /]#
Now that the host and OSDs have been logically removed from the Ceph
cluster proceed to remove the host from the overcloud as described in
the "Scaling Down" section of :doc:`../provisioning/baremetal_provision`.
Scenario: Deploy Hyperconverged Ceph
------------------------------------
Use a command like the following to create a `roles.yaml` file
containing a standard Controller role and a ComputeHCI role::
openstack overcloud roles generate Controller ComputeHCI -o ~/roles.yaml
The ComputeHCI role is a Compute node which also runs co-located Ceph
OSD daemons. This kind of service co-location is referred to as HCI,
or hyperconverged infrastructure. See the :doc:`composable_services`
documentation for details on roles and services.
When collocating Nova Compute and Ceph OSD services boundaries can be
set to reduce contention for CPU and Memory between the two services.
This is possible by adding parameters to `cephadm-overrides.yaml` like
the following::
parameter_defaults:
CephHciOsdType: hdd
CephHciOsdCount: 4
CephConfigOverrides:
osd:
osd_memory_target_autotune: true
osd_numa_auto_affinity: true
mgr:
mgr/cephadm/autotune_memory_target_ratio: 0.2
The `CephHciOsdType` and `CephHciOsdCount` parameters are used by the
Derived Parameters workflow to tune the Nova scheduler to not allocate
a certain amount of memory and CPU from the hypervisor to virtual
machines so that Ceph can use them instead. See the
:doc:`derived_parameters` documentation for details. If you do not use
Derived Parameters workflow, then at least set the
`NovaReservedHostMemory` to the number of OSDs multipled by 5 GB per
OSD per host.
The `CephConfigOverrides` map passes Ceph OSD parameters to limit the
CPU and memory used by the OSDs.
The `osd_memory_target_autotune`_ is set to true so that the OSD
daemons will adjust their memory consumption based on the
`osd_memory_target` config option. The `autotune_memory_target_ratio`
defaults to 0.7. So 70% of the total RAM in the system is the starting
point, from which any memory consumed by non-autotuned Ceph daemons
are subtracted, and then the remaining memory is divided by the OSDs
(assuming all OSDs have `osd_memory_target_autotune` true). For HCI
deployments the `mgr/cephadm/autotune_memory_target_ratio` can be set
to 0.2 so that more memory is available for the Nova Compute
service. This has the same effect as setting the ceph-ansible `is_hci`
parameter to true.
A two NUMA node system can host a latency sensitive Nova workload on
one NUMA node and a Ceph OSD workload on the other NUMA node. To
configure Ceph OSDs to use a specific NUMA node (and not the one being
used by the Nova Compute workload) use either of the following Ceph
OSD configurations:
- `osd_numa_node` sets affinity to a numa node (-1 for none)
- `osd_numa_auto_affinity` automatically sets affinity to the NUMA
node where storage and network match
If there are network interfaces on both NUMA nodes and the disk
controllers are NUMA node 0, then use a network interface on NUMA node
0 for the storage network and host the Ceph OSD workload on NUMA
node 0. Then host the Nova workload on NUMA node 1 and have it use the
network interfaces on NUMA node 1. Setting `osd_numa_auto_affinity`,
to true, as in the example `cephadm-overrides.yaml` file above, should
result in this configuration. Alternatively, the `osd_numa_node` could
be set directly to 0 and `osd_numa_auto_affinity` could be unset so
that it will default to false.
When a hyperconverged cluster backfills as a result of an OSD going
offline, the backfill process can be slowed down. In exchange for a
slower recovery, the backfill activity has less of an impact on
the collocated Compute workload. Ceph Pacific has the following
defaults to control the rate of backfill activity::
parameter_defaults:
CephConfigOverrides:
osd:
osd_recovery_op_priority: 3
osd_max_backfills: 1
osd_recovery_max_active_hdd: 3
osd_recovery_max_active_ssd: 10
It is not necessary to pass the above as they are the default values,
but if these values need to be deployed with different values modify
an example like the above before deployment. If the values need to be
adjusted after the deployment use `ceph config set osd <key> <value>`.
Deploy the overcloud as described in "Scenario: Deploy Ceph with
TripleO and Metalsmith" but use the `-r` option to include generated
`roles.yaml` file and the `-e` option with the
`cephadm-overrides.yaml` file containing the HCI tunings described
above.
The examples above may be used to tune a hyperconverged system during
deployment. If the values need to be changed after deployment, then
use the `ceph orchestrator` command to set them directly.
After deployment start a Ceph shell as described in "Accessing the
Ceph Command Line" and confirm the above values were applied. For
example, to check that the NUMA and memory target auto tuning run
commands lke this::
[ceph: root@oc0-controller-0 /]# ceph config dump | grep numa
osd advanced osd_numa_auto_affinity true
[ceph: root@oc0-controller-0 /]# ceph config dump | grep autotune
osd advanced osd_memory_target_autotune true
[ceph: root@oc0-controller-0 /]# ceph config get mgr mgr/cephadm/autotune_memory_target_ratio
0.200000
[ceph: root@oc0-controller-0 /]#
We can then confirm that a specific OSD, e.g. osd.11, inherited those
values with commands like this::
[ceph: root@oc0-controller-0 /]# ceph config get osd.11 osd_memory_target
4294967296
[ceph: root@oc0-controller-0 /]# ceph config get osd.11 osd_memory_target_autotune
true
[ceph: root@oc0-controller-0 /]# ceph config get osd.11 osd_numa_auto_affinity
true
[ceph: root@oc0-controller-0 /]#
To confirm that the default backfill values are set for the same
example OSD, use commands like this::
[ceph: root@oc0-controller-0 /]# ceph config get osd.11 osd_recovery_op_priority
3
[ceph: root@oc0-controller-0 /]# ceph config get osd.11 osd_max_backfills
1
[ceph: root@oc0-controller-0 /]# ceph config get osd.11 osd_recovery_max_active_hdd
3
[ceph: root@oc0-controller-0 /]# ceph config get osd.11 osd_recovery_max_active_ssd
10
[ceph: root@oc0-controller-0 /]#
The above example assumes that :doc:`deployed_ceph` is not used.
Add the Ceph Dashboard to a Overcloud deployment
------------------------------------------------
During the overcloud deployment most of the Ceph daemons can be added and
configured.
To deploy the ceph dashboard include the ceph-dashboard.yaml environment
file as in the following example::
openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/ceph-dashboard.yaml
The command above will include the ceph dashboard related services and
generates all the `cephadm` required variables to render the monitoring
stack related spec that can be applied against the deployed Ceph cluster.
When the deployment has been completed the Ceph dashboard containers,
including prometheus and grafana, will be running on the controller nodes
and will be accessible using the port 3100 for grafana and 9092 for prometheus;
since this service is only internal and doesn’t listen on the public vip, users
can reach both grafana and the exposed ceph dashboard using the controller
provisioning network vip on the specified port (8444 is the default for a generic
overcloud deployment).
The resulting deployment will be composed by an external stack made by grafana,
prometheus, alertmanager, node-exporter containers and the ceph dashboard mgr
module that acts as the backend for this external stack, embedding the grafana
layouts and showing the ceph cluster specific metrics coming from prometheus.
The Ceph Dashboard backend services run on the specified `CephDashboardNetwork`
and `CephGrafanaNetwork`, while the high availability is realized by haproxy and
Pacemaker.
The Ceph Dashboard frontend is fully integrated with the tls-everywhere framework,
hence providing the tls environments files will trigger the certificate request for
both grafana and the ceph dashboard: the generated crt and key files are then
configured by cephadm, resulting in a key-value pair within the Ceph orchestrator,
which is able to mount the required files to the dashboard related containers.
The Ceph Dashboard admin user role is set to `read-only` mode by default for safe
monitoring of the Ceph cluster. To permit an admin user to have elevated privileges
to alter elements of the Ceph cluster with the Dashboard, the operator can change the
default.
For this purpose, TripleO exposes a parameter that can be used to change the Ceph
Dashboard admin default mode.
Log in to the undercloud as `stack` user and create the `ceph_dashboard_admin.yaml`
environment file with the following content::
parameter_defaults:
CephDashboardAdminRO: false
Run the overcloud deploy command to update the existing stack and include the environment
file created with all other environment files that are already part of the existing
deployment::
openstack overcloud deploy --templates -e <existing_overcloud_environment_files> -e ceph_dashboard_admin.yml
The ceph dashboard will also work with composable networks.
In order to isolate the monitoring access for security purposes, operators can
take advantage of composable networks and access the dashboard through a separate
network vip. By doing this, it's not necessary to access the provisioning network
and separate authorization profiles may be implemented.
To deploy the overcloud with the ceph dashboard composable network we need first
to generate the controller specific role created for this scenario::
openstack overcloud roles generate -o /home/stack/roles_data.yaml ControllerStorageDashboard Compute BlockStorage ObjectStorage CephStorage
Finally, run the overcloud deploy command including the new generated `roles_data.yaml`
and the `network_data_dashboard.yaml` file that will trigger the generation of this
new network.
The final overcloud command must look like the following::
openstack overcloud deploy --templates -r /home/stack/roles_data.yaml -n /usr/share/openstack-tripleo-heat-templates/network_data_dashboard.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml -e ~/my-ceph-settings.yaml
.. _`cephadm`: https://docs.ceph.com/en/latest/cephadm/index.html
.. _`cleaning instructions in the Ironic documentation`: https://docs.openstack.org/ironic/latest/admin/cleaning.html
.. _`Ceph Orchestrator`: https://docs.ceph.com/en/latest/mgr/orchestrator/
.. _`ceph config command`: https://docs.ceph.com/en/latest/man/8/ceph/#config
.. _`Ceph's centralized configuration management`: https://ceph.io/community/new-mimic-centralized-configuration-management/
.. _`Ceph Service Specification`: https://docs.ceph.com/en/octopus/mgr/orchestrator/#orchestrator-cli-service-spec
.. _`ceph_spec_bootstrap`: https://docs.openstack.org/tripleo-ansible/latest/modules/modules-ceph_spec_bootstrap.html
.. _`Advanced OSD Service Specifications`: https://docs.ceph.com/en/octopus/cephadm/drivegroups/
.. _`Autoscaling Placement Groups`: https://docs.ceph.com/en/latest/rados/operations/placement-groups/
.. _`pgcalc`: http://ceph.com/pgcalc
.. _`CRUSH Map Rules`: https://docs.ceph.com/en/latest/rados/operations/crush-map-edits/?highlight=ceph%20crush%20rules#crush-map-rules
.. _`OSD Service Documentation for cephadm`: https://docs.ceph.com/en/latest/cephadm/services/osd/
.. _`osd_memory_target_autotune`: https://docs.ceph.com/en/latest/cephadm/services/osd/#automatically-tuning-osd-memory

973
deploy-guide/source/features/deployed_ceph.rst
View File

File diff suppressed because it is too large Load Diff

2
deploy-guide/source/features/derived_parameters.rst
View File

@ -196,7 +196,7 @@ devices like ``CephAnsibleDisksConfig``, setting the count directly is
necessary in order to know how much CPU/RAM to reserve. Similarly,
because a device path is not hard coded, we cannot look up that device
in Ironic to determine its type. For information on the
``CephOsdSpec`` parameter see the :doc:`cephadm` documentation.
``CephOsdSpec`` parameter see the :doc:`deployed_ceph` documentation.
``CephHciOsdType`` is the type of data_device (not db_device) used for
each OSD and must be one of hdd, ssd, or nvme. These are used by

6
deploy-guide/source/features/distributed_multibackend_storage.rst
View File

@ -115,7 +115,7 @@ Ceph Deployment Types
|project| supports two types of Ceph deployments. An "internal" Ceph
deployment is one where a Ceph cluster is deployed as part of the
overcloud as described in :doc:`ceph_config`. An "external" Ceph
overcloud as described in :doc:`deployed_ceph`. An "external" Ceph
deployment is one where a Ceph cluster already exists and an overcloud
is configured to be a client of that Ceph cluster as described in
:doc:`ceph_external`. Ceph external deployments have special meaning
@ -160,7 +160,7 @@ types of deployments as described in the following sequence:
as an additional RBD backend.
The above sequence is possible by using the `CephExtraKeys` parameter
as described in :doc:`ceph_config` and the `CephExternalMultiConfig`
as described in :doc:`deployed_ceph` and the `CephExternalMultiConfig`
parameter described in :doc:`ceph_external`.
Decide which cephx key will be used to access remote Ceph clusters
@ -239,7 +239,7 @@ Ceph cluster with pools which may be accessed by the cephx user
"client.external". The same parameters will be used later when the
DCN overclouds are configured as external Ceph clusters. For more
information on the `CephExtraKeys` parameter see the document
:doc:`ceph_config` section called `Configuring CephX Keys`.
:doc:`deployed_ceph` section called `Overriding CephX Keys`.
Create control-plane roles
^^^^^^^^^^^^^^^^^^^^^^^^^^

3
deploy-guide/source/post_deployment/upgrade/fast_forward_upgrade.rst
View File

@ -641,8 +641,7 @@ Following there is a list of all the changes needed:
ensure the devices list has been migrated to the format expected by
ceph-ansible. It is possible to use the ``CephAnsibleExtraConfig`` and
`CephAnsibleDisksConfig`` parameters to pass arbitrary variables to
ceph-ansible, like ``devices`` and ``dedicated_devices``. See the
:doc:`TripleO Ceph config guide <../../../features/ceph_config>`
ceph-ansible, like ``devices`` and ``dedicated_devices``.
The other parameters (for example ``CinderRbdPoolName``,
``CephClientUserName``, ...) will behave as they used to with puppet-ceph

4
deploy-guide/source/post_deployment/upgrade/major_upgrade.rst
View File

@ -559,9 +559,7 @@ major-upgrade-composable-steps that come first, as described above.
ensure the devices list has been migrated to the format expected by
ceph-ansible. It is possible to use the ``CephAnsibleExtraConfig`` and
``CephAnsibleDisksConfig`` parameters to pass arbitrary variables to
ceph-ansible, like ``devices`` and ``dedicated_devices``. See the
`ceph-ansible scenarios`_ or the :doc:`TripleO Ceph config guide
<../../features/ceph_config>`
ceph-ansible, like ``devices`` and ``dedicated_devices``.
The other parameters (for example ``CinderRbdPoolName``,
``CephClientUserName``, ...) will behave as they used to with puppet-ceph