openstack/tripleo-docs
Code Issues Proposed changes

Merge "Document "deployed ceph""

Browse Source
This commit is contained in:
Zuul 2021-08-23 15:01:39 +00:00 committed by Gerrit Code Review
commit 2d1a4a1c43
4 changed files with 437 additions and 26 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
deploy-guide/source/deployment/network_v2.rst
View File

@ -95,6 +95,12 @@ that it can run all the steps:
As an external deploy step the neutron ports for Service Virtual IPs are
created, and the properties of the Virtual IPs are included in hieradata.
.. admonition:: Ceph
:class: ceph
Optionally Ceph may be deployed after the baremetal instances
are provisioned but before the ephemeral Heat stack is created
as described in :doc:`../features/deployed_ceph`.
Using
-----

1
deploy-guide/source/features/backends.rst
View File

@ -11,6 +11,7 @@ OpenStack projects.
cinder_custom_backend
cinder_netapp
cephadm
deployed_ceph
ceph_config
ceph_external
domain_specific_ldap_backends

100
deploy-guide/source/features/cephadm.rst
View File

@ -15,13 +15,9 @@ Limitations
-----------
TripleO deployments of Ceph with cephadm_ are only supported in Wallaby
or newer and are currently limited to only Ceph RBD and RGW
services. Support for deployment of CephFS with Manila and Ceph
Dashboard via TripleO's cephadm integration are not yet available in
Wallaby though these services may be deployed with ceph-ansible as
described in the :doc:`ceph_config` documentation. The default version
of Ceph deployed by TripleO in Wallaby is Pacific, regardless of if
cephadm or ceph-ansible is used to deploy it.
or newer. The default version of Ceph deployed by TripleO in Wallaby
is Pacific, regardless of if cephadm or ceph-ansible is used to deploy
it.
TripleO can only deploy one Ceph cluster in the overcloud per Heat
stack. However, within that Heat stack it's possible to configure
@ -71,8 +67,8 @@ for some target nodes, see `cleaning instructions in the Ironic documentation`_.
Deploying Ceph During Overcloud Deployment
------------------------------------------
To deploy an overcloud with a Ceph RBD and RGW server include the
appropriate environment file as in the example below::
To deploy an overcloud with a Ceph include the appropriate environment
file as in the example below::
openstack overcloud deploy --templates \
-e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml
@ -93,12 +89,23 @@ deploy like this::
-e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \
-e cephadm-overrides.yaml
Deploying with the commands above will result in the process described
in the next section.
Deploying with the commands above will result in the processes described
in the rest of this document.
Deploying Ceph Before Overcloud Deployment
------------------------------------------
In Wallaby and newer it is possible to provision hardware and deploy
Ceph before deploying the overcloud on the same hardware. This feature
is called "deployed ceph" and it uses the command `openstack overcloud
ceph deploy` which executes the same Ansible roles described
below. For more details see :doc:`deployed_ceph`.
Overview of Ceph Deployment with TripleO and cephadm
----------------------------------------------------
When Ceph is deployed during overcloud configuration or when Ceph is
deployed before overcloud configuration with :doc:`deployed_ceph`,
TripleO will use Ansible automate the process described in the
`cephadm`_ documentation to bootstrap a new cluster. It will
bootstrap a single Ceph monitor and manager on one server
@ -132,13 +139,22 @@ e.g. which servers run which services according to composable
roles, will be converted by the tripleo-ansible `ceph_spec_bootstrap`_
module into a `Ceph Service Specification`_ file. The module has the
ability to do this based on the Ansible inventory generated by the
`tripleo-ansible-inventory` but it can also generate the Ceph Service
Specification file from a combination of a TripleO roles data file
`tripleo-ansible-inventory`. When Ceph is deployed *during* overcloud
configuration by including the cephadm.yaml environment file, the
module uses the Ansible inventory to create the `Ceph Service
Specification`_. In this scenario the default location of the
generated Ceph Service Specification file is
`config-download/<STACK>/cephadm/ceph_spec.yaml`.
The same `ceph_spec_bootstrap`_ module can also generate the Ceph
Service Specification file from a combination of a TripleO roles data
file
(e.g. /usr/share/openstack-tripleo-heat-templates/roles_data.yaml)
and the output of the command
`openstack overcloud node provision --output deployed_metal.yaml`.
The default location of the generated Ceph Service Specification
file is in `config-download/<STACK>/cephadm/ceph_spec.yaml`.
When Ceph is deployed *before* overcloud configuration as described in
:doc:`deployed_ceph`, the module uses the deployed_metal.yaml and
roles_data.yaml to create the `Ceph Service Specification`_.
After the `ceph-admin` user is created, `ceph_spec.yaml` is copied
to the bootstrap host. The bootstrap host will be the first host
@ -152,14 +168,16 @@ the bootstrap node and then run `ceph orch apply -i ceph_spec.yaml`
and `cephadm` will use the `ceph-admin` account and SSH keys to add
the other nodes.
After the full Ceph cluster is running the Ceph pools and the cephx
keys to access the pools will be created as defined or overridden as
described in the Heat environment examples below. The information
necessary to configure Ceph clients will then be extracted to
`/home/stack/ceph_client.yml` on the undercloud and passed to the
as input to the tripleo-ansible role tripleo_ceph_client which will
then configure the rest of the overcloud to use the new Ceph cluster
as described in the :doc:`ceph_external` documentation.
After the full Ceph cluster is running, either as a result of
:doc:`deployed_ceph` or by cephadm being triggered during the
overcloud deployment via the `cephadm.yaml` environment file, the
Ceph pools and the cephx keys to access the pools will be created as
defined or overridden as described in the Heat environment examples
below. The information necessary to configure Ceph clients will then
be extracted to `/home/stack/ceph_client.yml` on the undercloud and
passed to the as input to the tripleo-ansible role tripleo_ceph_client
which will then configure the rest of the overcloud to use the new
Ceph cluster as described in the :doc:`ceph_external` documentation.
When `openstack overcloud deploy` is re-run in order to update
the stack, the cephadm bootstrap process is not repeated because
@ -199,6 +217,8 @@ monitor. The supported configuration groups are 'global', 'mon',
'mgr', 'osd', 'mds', and 'client'. If no group is provided, then the
default configuration group is 'global'.
The above does not apply to :doc:`deployed_ceph`.
Overriding Server Configuration after deployment
------------------------------------------------
@ -262,6 +282,11 @@ If `CephDynamicSpec` is true and `CephSpecPath` is set to a valid
path, then the spec will be created at that path before it is used to
deploy Ceph.
The `CephDynamicSpec` and `CephSpecPath` parameters are not available
when using "deployed ceph", but the functionality is available via
the `--ceph-spec` command line option as described in
:doc:`deployed_ceph`.
Overriding which disks should be OSDs
-------------------------------------
@ -307,8 +332,12 @@ available but with the new syntax. When using this option consider
setting `CephDynamicSpec` to false and defining a custom specification
which is passed to TripleO by setting the `CephSpecPath`.
Overriding Ceph placement group values during deployment
--------------------------------------------------------
The `CephOsdSpec` parameter is not available when using "deployed
ceph", but the same functionality is available via `--osd-spec`
command line option as described in :doc:`deployed_ceph`.
Overriding Ceph Pools and Placement Group values during deployment
------------------------------------------------------------------
The default cephadm deployment as triggered by TripleO has
`Autoscaling Placement Groups`_ enabled. Thus, it is not necessary to
@ -335,6 +364,12 @@ the following::
- {"name": vms, "pg_num": 512, "pgp_num": 512, "application": rbd}
- {"name": images, "pg_num": 128, "pgp_num": 128, "application": rbd}
Regardless of if the :doc:`deployed_ceph` feature is used, pools will
always be created during overcloud deployment as documented above.
Additional pools may also be created directly via the Ceph command
line tools.
Overriding CRUSH rules
----------------------
@ -359,6 +394,10 @@ parameter::
- {'name': 'slow_pool', 'rule_name': 'HDD', 'application': 'rbd'}
- {'name': 'fast_pool', 'rule_name': 'SSD', 'application': 'rbd'}
Regardless of if the :doc:`deployed_ceph` feature is used, custom
CRUSH rules may be created during overcloud deployment as documented
above. CRUSH rules may also be created directly via the Ceph command
line tools.
Overriding CephX Keys
---------------------
@ -420,6 +459,11 @@ pool used by Glance. The default Glance deployment defined in the Heat
stack will continue to use the `ceph.client.openstack.keyring` file
unless that Glance configuration itself is overridden.
Regardless of if the :doc:`deployed_ceph` feature is used, CephX keys
may be created during overcloud deployment as documented above.
Additional CephX keys may also be created directly via the Ceph
command line tools.
Enabling cephadm debug mode
---------------------------
@ -512,7 +556,10 @@ which is passed to the following command::
--output ~/overcloud-baremetal-deployed.yaml \
~/overcloud_baremetal_deploy.yaml
As described in :doc:`../provisioning/baremetal_provision`, pass
If desired at this stage, then Ceph may be deployed early as described
in :doc:`deployed_ceph`. Otherwise Ceph may be deployed during the
overcloud deployment. Either way, as described in
:doc:`../provisioning/baremetal_provision`, pass
~/overcloud_baremetal_deploy.yaml as input, along with
/usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml
and cephadm-overrides.yaml described above, to the `openstack overcloud
@ -772,6 +819,7 @@ example OSD, use commands like this::
10
[ceph: root@oc0-controller-0 /]#
The above example assumes that :doc:`deployed_ceph` is not used.
Add the Ceph Dashboard to a Overcloud deployment
------------------------------------------------

356
deploy-guide/source/features/deployed_ceph.rst Normal file
View File

@ -0,0 +1,356 @@
Deployed Ceph
=============
In Wallaby and newer it is possible to provision hardware and deploy
Ceph before deploying the overcloud on the same hardware.
Deployed Ceph Workflow
----------------------
As described in the :doc:`../deployment/network_v2` the ``overcloud
deploy`` command was extended so that it can run all of the following
as separate steps:
#. Create Networks
#. Create Virtual IPs
#. Provision Baremetal Instances
#. Deploy Ceph
#. Create the overcloud Ephemeral Heat stack
#. Run Config-Download and the deploy-steps playbook
This document covers the "Deploy Ceph" step above. For details on the
other steps see :doc:`../deployment/network_v2`.
The "Provision Baremetal Instances" step outputs a YAML file
describing the deployed baremetal, for example::
openstack overcloud node provision \
-o ~/deployed_metal.yaml \
...
The deployed_metal.yaml file can be passed as input to the ``openstack
overcloud ceph deploy`` command, which in turn outputs a YAML file
describing the deployed Ceph cluster, for example::
openstack overcloud ceph deploy \
~/deployed_metal.yaml \
-o ~/deployed_ceph.yaml \
...
Both the deployed_metal.yaml and deployed_ceph.yaml files may then be
passed as input to the step to "Create the overcloud Ephemeral Heat
stack", for example::
openstack overcloud deploy --templates \
-e ~/deployed_metal.yaml \
-e ~/deployed_ceph.yaml \
...
While the overcloud is being deployed the data in the
deployed_ceph.yaml file will be used to configure the OpenStack
clients to connect to the Ceph cluster as well as configure the Ceph
cluster to host OpenStack.
The above workflow is called "Deployed Ceph" because Ceph is already
deployed when the overcloud is configured.
Deployed Ceph Scope
-------------------
The "Deployed Ceph" feature deploys a Ceph cluster ready to serve RBD
by calling the same TripleO Ansible roles described in :doc:`cephadm`.
When the "Deployed Ceph" process is over you should expect to find the
following:
- The CephMon, CephMgr, and CephOSD services are running on all nodes
which should have those services
- It's possible to SSH into a node with the CephMon service and run
`sudo cepham shell`
- All OSDs should be running unless there were environmental issues
(e.g. disks were not cleaned)
- A ceph configuration file and client admin keyring file in /etc/ceph
of overcloud nodes with the CephMon service
- The Ceph cluster is ready to serve RBD
You should not expect the following after "Deployed Ceph" has run:
- No pools or cephx keys for OpenStack will be created yet
- No CephDashboard, CephRGW or CephMds services will be running yet
The above will be configured during overcloud deployment by the
`openstack overcloud deploy` command as they were prior to the
"Deployed Ceph" feature. The reasons for this are the following:
- The Dashboard and RGW services need to integrate with haproxy which
is deployed with the overcloud
- The list of pools to create and their respective cephx keys are a
function of which OpenStack clients (e.g. Nova, Cinder, etc) will be
used so they must be in the overcloud definition. Thus, they are
created during overcloud deployment
During the overcloud deployment the above resources will be
created in Ceph by the TripleO Ansible roles described in
:doc:`cephadm` using the client admin keyring file and the
``~/deployed_ceph.yaml`` file output by `openstack overcloud ceph
deploy`. Because these resources are created directly on the Ceph
cluster with admin level access, "Deployed Ceph" is different from
the "External Ceph" feature described in :doc:`ceph_external`.
The main benefits of using "Deployed Ceph" are the following:
- Use cephadm to deploy Ceph on the hardware managed by TripleO
without having to write your own cephadm spec file (though you may
provide your own if you wish)
- Focus on debugging the basic Ceph deployment without debugging the
overcloud deployment at the same time
- Fix any Ceph deployment problems directly using either Ansible or
the Ceph orchestrator tools before starting the overcloud deployment
- Have the benefits above while maintaining hyperconverged support by
using a tested workflow
In summary, `openstack overcloud ceph deploy` deploys the Ceph cluster
while `openstack overcloud deploy` (and the commands that follow)
deploy OpenStack and configure that Ceph cluster to be used by
OpenStack.
Deployed Ceph Command Line Interface
------------------------------------
The command line interface supports the following options::
$ openstack overcloud ceph deploy --help
usage: openstack overcloud ceph deploy [-h] -o <deployed_ceph.yaml> [-y]
[--stack STACK]
[--working-dir WORKING_DIR]
[--roles-data ROLES_DATA]
[--ceph-spec CEPH_SPEC | --osd-spec OSD_SPEC]
[--container-image-prepare CONTAINER_IMAGE_PREPARE]
[--container-namespace CONTAINER_NAMESPACE]
[--container-image CONTAINER_IMAGE]
[--container-tag CONTAINER_TAG]
[--registry-url REGISTRY_URL]
[--registry-username REGISTRY_USERNAME]
[--registry-password REGISTRY_PASSWORD]
<deployed_baremetal.yaml>
positional arguments:
<deployed_baremetal.yaml>
Path to the environment file output from "openstack
overcloud node provision".
optional arguments:
-h, --help show this help message and exit
-o <deployed_ceph.yaml>, --output <deployed_ceph.yaml>
The path to the output environment file describing the
Ceph deployment to pass to the overcloud deployment.
-y, --yes Skip yes/no prompt before overwriting an existing
<deployed_ceph.yaml> output file (assume yes).
--stack STACK Name or ID of heat stack (default=Env:
OVERCLOUD_STACK_NAME)
--working-dir WORKING_DIR
The working directory for the deployment where all
input, output, and generated files will be stored.
Defaults to "$HOME/overcloud-deploy/<stack>"
--roles-data ROLES_DATA
Path to an alternative roles_data.yaml. Used to decide
which node gets which Ceph mon, mgr, or osd service
based on the node's role in <deployed_baremetal.yaml>.
--ceph-spec CEPH_SPEC
Path to an existing Ceph spec file. If not provided a
spec will be generated automatically based on --roles-
data and <deployed_baremetal.yaml>
--osd-spec OSD_SPEC Path to an existing OSD spec file. Mutually exclusive
with --ceph-spec. If the Ceph spec file is generated
automatically, then the OSD spec in the Ceph spec file
defaults to {data_devices: {all: true}} for all
service_type osd. Use --osd-spec to override the
data_devices value inside the Ceph spec file.
--container-image-prepare CONTAINER_IMAGE_PREPARE
Path to an alternative
container_image_prepare_defaults.yaml. Used to control
which Ceph container is pulled by cephadm via the
ceph_namespace, ceph_image, and ceph_tag variables in
addition to registry authentication via
ContainerImageRegistryCredentials.
container-image-prepare overrides:
The following options may be used to override individual values set via
--container-image-prepare. If the example variables below were set the
image would be concatenated into quay.io/ceph/ceph:latest and a custom
registry login would be used.
--container-namespace CONTAINER_NAMESPACE
e.g. quay.io/ceph
--container-image CONTAINER_IMAGE
e.g. ceph
--container-tag CONTAINER_TAG
e.g. latest
--registry-url REGISTRY_URL
--registry-username REGISTRY_USERNAME
--registry-password REGISTRY_PASSWORD
This command is provided by the python-tripleoclient plugin.
$
Run `openstack overcloud ceph deploy --help` in your own environment
to see the latest options which you have available.
Ceph Spec Options
-----------------
The roles file, described in the next section, and the output of
`openstack overcloud node provision` are passed to the
`ceph_spec_bootstrap`_ Ansible module to create a `Ceph Service
Specification`_. The `openstack overcloud ceph deploy` command does
this automatically so it is not necessary to use the options described
in this section unless desired.
It's possible to generate a Ceph Spec on the undercloud before
deployment by using the `ceph_spec_bootstrap`_ Ansible module
directly, for example::
ansible localhost -m ceph_spec_bootstrap \
-a deployed_metalsmith=deployed_metal.yaml
By default the above creates the file ``~/ceph_spec.yaml``. For more
information on the ``ceph_spec_bootstrap`` module run `ansible-doc
ceph_spec_bootstrap`. The spec file may then be edited if desired and
passed directly like this::
openstack overcloud ceph deploy \
deployed_metal.yaml \
-o deployed_ceph.yaml \
--ceph-spec ~/ceph_spec.yaml
All available disks (excluding the disk where the operating system is
installed) are used as OSDs as per the following default inside the
ceph spec::
data_devices:
all: true
In the above example, the `data_devices` key is valid for any `Ceph
Service Specification`_ whose `service_type` is "osd". Other OSD
service types, as found in the `Advanced OSD Service
Specifications`_, may be set by using the ``--osd-spec`` option.
If the file ``osd_spec.yaml`` contains the following::
data_devices:
rotational: 1
db_devices:
rotational: 0
and the following command is run::
openstack overcloud ceph deploy \
deployed_metal.yaml \
-o deployed_ceph.yaml \
--osd-spec osd_spec.yaml
Then all rotating devices will be data devices and all non-rotating
devices will be used as shared devices (wal, db). This is because when
the dynamic Ceph service specification is built whatever is in the
file referenced by ``--osd-spec`` will be appended to the section of
the specification if the `service_type` is "osd".
Service Placement Options
-------------------------
The Ceph services defined in the roles_data.yaml file as described in
:doc:`composable_services` determine which baremetal node runs which
service. By default the Controller role has the CephMon and CephMgr
service while the CephStorage role has the CephOSD service. Most
composable services require Heat output in order to determine how
services are configured, but not the Ceph services. Thus, the
roles_data.yaml file remains authoritative for Ceph service placement
even though the "Deployed Ceph" process happens before Heat is run.
It is only necessary to use the `--roles-file` option if the default
roles_data.yaml file is not being used. For example if you intend to
deploy hyperconverged nodes, then you want the predeployed compute
nodes to be in the ceph spec with the "osd" label and for the
`service_type` "osd" to have a placement list containing a list of the
compute nodes. To do this generate a custom roles file as described in
:doc:`composable_services` like this::
openstack overcloud roles generate Controller ComputeHCI > custom_roles.yaml
and then pass that roles file like this::
openstack overcloud ceph deploy \
deployed_metal.yaml \
-o deployed_ceph.yaml \
--roles-data custom_roles.yaml
After running the above the compute nodes should have running OSD
containers and when the overcloud is deployed Nova compute services
will then be set up on the same hosts.
If you wish to generate the ceph spec with the modified placement
described above before the ceph deployment, then the same file may be
passed to a direct call of the ceph_spec_bootstrap ansible module::
ansible localhost -m ceph_spec_bootstrap \
-a "deployed_metalsmith=deployed_metal.yaml tripleo_roles=custom_roles.yaml"
Container Options
-----------------
As described in :doc:`../deployment/container_image_prepare` the
undercloud may be used as a container registry for ceph containers
and there is a supported syntax to download containers from
authenticated registries. By default `openstack overcloud ceph deploy`
will pull the Ceph container in the default
``container_image_prepare_defaults.yaml`` file. The version of the
Ceph used in each OpenStack release changes per release and can be
seen by running a command like this::
egrep "ceph_namespace|ceph_image|ceph_tag" \
/usr/share/tripleo-common/container-images/container_image_prepare_defaults.yaml
The `--container-image-prepare` option can be used to override which
``container_image_prepare_defaults.yaml`` file is used. If a version
of this file called ``custom_container_image_prepare.yaml`` is
modified to contain syntax like the following::
ContainerImageRegistryCredentials:
quay.io/ceph-ci:
quay_username: quay_password
Then when a command like the following is run::
openstack overcloud ceph deploy \
deployed_metal.yaml \
-o deployed_ceph.yaml \
--container-image-prepare custom_container_image_prepare.yaml
The credentials will be extracted from the file and the tripleo
ansible role to bootstrap Ceph will be executed like this::
cephadm bootstrap
--registry-url quay.io/ceph-ci
--registry-username quay_username
--registry-password quay_password
...
The syntax of the container image prepare file can also be ignored and
instead the following command line options may be used instead::
--container-namespace CONTAINER_NAMESPACE
e.g. quay.io/ceph
--container-image CONTAINER_IMAGE
e.g. ceph
--container-tag CONTAINER_TAG
e.g. latest
--registry-url REGISTRY_URL
--registry-username REGISTRY_USERNAME
--registry-password REGISTRY_PASSWORD
If a variable above is unused, then it defaults to the ones found in
the default ``container_image_prepare_defaults.yaml`` file. In other
words, the above options are overrides.
.. _`ceph_spec_bootstrap`: https://docs.openstack.org/tripleo-ansible/latest/modules/modules-ceph_spec_bootstrap.html
.. _`Ceph Service Specification`: https://docs.ceph.com/en/latest/cephadm/service-management/#orchestrator-cli-service-spec
.. _`Advanced OSD Service Specifications`: https://docs.ceph.com/en/latest/cephadm/osd/#drivegroups