diff --git a/deploy-guide/source/deployment/network_v2.rst b/deploy-guide/source/deployment/network_v2.rst index 4751cf3a..bf335874 100644 --- a/deploy-guide/source/deployment/network_v2.rst +++ b/deploy-guide/source/deployment/network_v2.rst @@ -95,6 +95,12 @@ that it can run all the steps: As an external deploy step the neutron ports for Service Virtual IPs are created, and the properties of the Virtual IPs are included in hieradata. + .. admonition:: Ceph + :class: ceph + + Optionally Ceph may be deployed after the baremetal instances + are provisioned but before the ephemeral Heat stack is created + as described in :doc:`../features/deployed_ceph`. Using ----- diff --git a/deploy-guide/source/features/backends.rst b/deploy-guide/source/features/backends.rst index 72294eaa..47a25515 100644 --- a/deploy-guide/source/features/backends.rst +++ b/deploy-guide/source/features/backends.rst @@ -11,6 +11,7 @@ OpenStack projects. cinder_custom_backend cinder_netapp cephadm + deployed_ceph ceph_config ceph_external domain_specific_ldap_backends diff --git a/deploy-guide/source/features/cephadm.rst b/deploy-guide/source/features/cephadm.rst index 16b7ba38..c4da7432 100644 --- a/deploy-guide/source/features/cephadm.rst +++ b/deploy-guide/source/features/cephadm.rst @@ -15,13 +15,9 @@ Limitations ----------- TripleO deployments of Ceph with cephadm_ are only supported in Wallaby -or newer and are currently limited to only Ceph RBD and RGW -services. Support for deployment of CephFS with Manila and Ceph -Dashboard via TripleO's cephadm integration are not yet available in -Wallaby though these services may be deployed with ceph-ansible as -described in the :doc:`ceph_config` documentation. The default version -of Ceph deployed by TripleO in Wallaby is Pacific, regardless of if -cephadm or ceph-ansible is used to deploy it. +or newer. The default version of Ceph deployed by TripleO in Wallaby +is Pacific, regardless of if cephadm or ceph-ansible is used to deploy +it. TripleO can only deploy one Ceph cluster in the overcloud per Heat stack. However, within that Heat stack it's possible to configure @@ -71,8 +67,8 @@ for some target nodes, see `cleaning instructions in the Ironic documentation`_. Deploying Ceph During Overcloud Deployment ------------------------------------------ -To deploy an overcloud with a Ceph RBD and RGW server include the -appropriate environment file as in the example below:: +To deploy an overcloud with a Ceph include the appropriate environment +file as in the example below:: openstack overcloud deploy --templates \ -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml @@ -93,12 +89,23 @@ deploy like this:: -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \ -e cephadm-overrides.yaml -Deploying with the commands above will result in the process described -in the next section. +Deploying with the commands above will result in the processes described +in the rest of this document. + +Deploying Ceph Before Overcloud Deployment +------------------------------------------ + +In Wallaby and newer it is possible to provision hardware and deploy +Ceph before deploying the overcloud on the same hardware. This feature +is called "deployed ceph" and it uses the command `openstack overcloud +ceph deploy` which executes the same Ansible roles described +below. For more details see :doc:`deployed_ceph`. Overview of Ceph Deployment with TripleO and cephadm ---------------------------------------------------- +When Ceph is deployed during overcloud configuration or when Ceph is +deployed before overcloud configuration with :doc:`deployed_ceph`, TripleO will use Ansible automate the process described in the `cephadm`_ documentation to bootstrap a new cluster. It will bootstrap a single Ceph monitor and manager on one server @@ -132,13 +139,22 @@ e.g. which servers run which services according to composable roles, will be converted by the tripleo-ansible `ceph_spec_bootstrap`_ module into a `Ceph Service Specification`_ file. The module has the ability to do this based on the Ansible inventory generated by the -`tripleo-ansible-inventory` but it can also generate the Ceph Service -Specification file from a combination of a TripleO roles data file +`tripleo-ansible-inventory`. When Ceph is deployed *during* overcloud +configuration by including the cephadm.yaml environment file, the +module uses the Ansible inventory to create the `Ceph Service +Specification`_. In this scenario the default location of the +generated Ceph Service Specification file is +`config-download//cephadm/ceph_spec.yaml`. + +The same `ceph_spec_bootstrap`_ module can also generate the Ceph +Service Specification file from a combination of a TripleO roles data +file (e.g. /usr/share/openstack-tripleo-heat-templates/roles_data.yaml) and the output of the command `openstack overcloud node provision --output deployed_metal.yaml`. -The default location of the generated Ceph Service Specification -file is in `config-download//cephadm/ceph_spec.yaml`. +When Ceph is deployed *before* overcloud configuration as described in +:doc:`deployed_ceph`, the module uses the deployed_metal.yaml and +roles_data.yaml to create the `Ceph Service Specification`_. After the `ceph-admin` user is created, `ceph_spec.yaml` is copied to the bootstrap host. The bootstrap host will be the first host @@ -152,14 +168,16 @@ the bootstrap node and then run `ceph orch apply -i ceph_spec.yaml` and `cephadm` will use the `ceph-admin` account and SSH keys to add the other nodes. -After the full Ceph cluster is running the Ceph pools and the cephx -keys to access the pools will be created as defined or overridden as -described in the Heat environment examples below. The information -necessary to configure Ceph clients will then be extracted to -`/home/stack/ceph_client.yml` on the undercloud and passed to the -as input to the tripleo-ansible role tripleo_ceph_client which will -then configure the rest of the overcloud to use the new Ceph cluster -as described in the :doc:`ceph_external` documentation. +After the full Ceph cluster is running, either as a result of +:doc:`deployed_ceph` or by cephadm being triggered during the +overcloud deployment via the `cephadm.yaml` environment file, the +Ceph pools and the cephx keys to access the pools will be created as +defined or overridden as described in the Heat environment examples +below. The information necessary to configure Ceph clients will then +be extracted to `/home/stack/ceph_client.yml` on the undercloud and +passed to the as input to the tripleo-ansible role tripleo_ceph_client +which will then configure the rest of the overcloud to use the new +Ceph cluster as described in the :doc:`ceph_external` documentation. When `openstack overcloud deploy` is re-run in order to update the stack, the cephadm bootstrap process is not repeated because @@ -199,6 +217,8 @@ monitor. The supported configuration groups are 'global', 'mon', 'mgr', 'osd', 'mds', and 'client'. If no group is provided, then the default configuration group is 'global'. +The above does not apply to :doc:`deployed_ceph`. + Overriding Server Configuration after deployment ------------------------------------------------ @@ -262,6 +282,11 @@ If `CephDynamicSpec` is true and `CephSpecPath` is set to a valid path, then the spec will be created at that path before it is used to deploy Ceph. +The `CephDynamicSpec` and `CephSpecPath` parameters are not available +when using "deployed ceph", but the functionality is available via +the `--ceph-spec` command line option as described in +:doc:`deployed_ceph`. + Overriding which disks should be OSDs ------------------------------------- @@ -307,8 +332,12 @@ available but with the new syntax. When using this option consider setting `CephDynamicSpec` to false and defining a custom specification which is passed to TripleO by setting the `CephSpecPath`. -Overriding Ceph placement group values during deployment --------------------------------------------------------- +The `CephOsdSpec` parameter is not available when using "deployed +ceph", but the same functionality is available via `--osd-spec` +command line option as described in :doc:`deployed_ceph`. + +Overriding Ceph Pools and Placement Group values during deployment +------------------------------------------------------------------ The default cephadm deployment as triggered by TripleO has `Autoscaling Placement Groups`_ enabled. Thus, it is not necessary to @@ -335,6 +364,12 @@ the following:: - {"name": vms, "pg_num": 512, "pgp_num": 512, "application": rbd} - {"name": images, "pg_num": 128, "pgp_num": 128, "application": rbd} + +Regardless of if the :doc:`deployed_ceph` feature is used, pools will +always be created during overcloud deployment as documented above. +Additional pools may also be created directly via the Ceph command +line tools. + Overriding CRUSH rules ---------------------- @@ -359,6 +394,10 @@ parameter:: - {'name': 'slow_pool', 'rule_name': 'HDD', 'application': 'rbd'} - {'name': 'fast_pool', 'rule_name': 'SSD', 'application': 'rbd'} +Regardless of if the :doc:`deployed_ceph` feature is used, custom +CRUSH rules may be created during overcloud deployment as documented +above. CRUSH rules may also be created directly via the Ceph command +line tools. Overriding CephX Keys --------------------- @@ -420,6 +459,11 @@ pool used by Glance. The default Glance deployment defined in the Heat stack will continue to use the `ceph.client.openstack.keyring` file unless that Glance configuration itself is overridden. +Regardless of if the :doc:`deployed_ceph` feature is used, CephX keys +may be created during overcloud deployment as documented above. +Additional CephX keys may also be created directly via the Ceph +command line tools. + Enabling cephadm debug mode --------------------------- @@ -512,7 +556,10 @@ which is passed to the following command:: --output ~/overcloud-baremetal-deployed.yaml \ ~/overcloud_baremetal_deploy.yaml -As described in :doc:`../provisioning/baremetal_provision`, pass +If desired at this stage, then Ceph may be deployed early as described +in :doc:`deployed_ceph`. Otherwise Ceph may be deployed during the +overcloud deployment. Either way, as described in +:doc:`../provisioning/baremetal_provision`, pass ~/overcloud_baremetal_deploy.yaml as input, along with /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml and cephadm-overrides.yaml described above, to the `openstack overcloud @@ -772,6 +819,7 @@ example OSD, use commands like this:: 10 [ceph: root@oc0-controller-0 /]# +The above example assumes that :doc:`deployed_ceph` is not used. Add the Ceph Dashboard to a Overcloud deployment ------------------------------------------------ diff --git a/deploy-guide/source/features/deployed_ceph.rst b/deploy-guide/source/features/deployed_ceph.rst new file mode 100644 index 00000000..54526f18 --- /dev/null +++ b/deploy-guide/source/features/deployed_ceph.rst @@ -0,0 +1,356 @@ +Deployed Ceph +============= + +In Wallaby and newer it is possible to provision hardware and deploy +Ceph before deploying the overcloud on the same hardware. + +Deployed Ceph Workflow +---------------------- + +As described in the :doc:`../deployment/network_v2` the ``overcloud +deploy`` command was extended so that it can run all of the following +as separate steps: + +#. Create Networks +#. Create Virtual IPs +#. Provision Baremetal Instances +#. Deploy Ceph +#. Create the overcloud Ephemeral Heat stack +#. Run Config-Download and the deploy-steps playbook + +This document covers the "Deploy Ceph" step above. For details on the +other steps see :doc:`../deployment/network_v2`. + +The "Provision Baremetal Instances" step outputs a YAML file +describing the deployed baremetal, for example:: + + openstack overcloud node provision \ + -o ~/deployed_metal.yaml \ + ... + +The deployed_metal.yaml file can be passed as input to the ``openstack +overcloud ceph deploy`` command, which in turn outputs a YAML file +describing the deployed Ceph cluster, for example:: + + openstack overcloud ceph deploy \ + ~/deployed_metal.yaml \ + -o ~/deployed_ceph.yaml \ + ... + +Both the deployed_metal.yaml and deployed_ceph.yaml files may then be +passed as input to the step to "Create the overcloud Ephemeral Heat +stack", for example:: + + openstack overcloud deploy --templates \ + -e ~/deployed_metal.yaml \ + -e ~/deployed_ceph.yaml \ + ... + +While the overcloud is being deployed the data in the +deployed_ceph.yaml file will be used to configure the OpenStack +clients to connect to the Ceph cluster as well as configure the Ceph +cluster to host OpenStack. + +The above workflow is called "Deployed Ceph" because Ceph is already +deployed when the overcloud is configured. + +Deployed Ceph Scope +------------------- + +The "Deployed Ceph" feature deploys a Ceph cluster ready to serve RBD +by calling the same TripleO Ansible roles described in :doc:`cephadm`. +When the "Deployed Ceph" process is over you should expect to find the +following: + +- The CephMon, CephMgr, and CephOSD services are running on all nodes + which should have those services +- It's possible to SSH into a node with the CephMon service and run + `sudo cepham shell` +- All OSDs should be running unless there were environmental issues + (e.g. disks were not cleaned) +- A ceph configuration file and client admin keyring file in /etc/ceph + of overcloud nodes with the CephMon service +- The Ceph cluster is ready to serve RBD + +You should not expect the following after "Deployed Ceph" has run: + +- No pools or cephx keys for OpenStack will be created yet +- No CephDashboard, CephRGW or CephMds services will be running yet + +The above will be configured during overcloud deployment by the +`openstack overcloud deploy` command as they were prior to the +"Deployed Ceph" feature. The reasons for this are the following: + +- The Dashboard and RGW services need to integrate with haproxy which + is deployed with the overcloud +- The list of pools to create and their respective cephx keys are a + function of which OpenStack clients (e.g. Nova, Cinder, etc) will be + used so they must be in the overcloud definition. Thus, they are + created during overcloud deployment + +During the overcloud deployment the above resources will be +created in Ceph by the TripleO Ansible roles described in +:doc:`cephadm` using the client admin keyring file and the +``~/deployed_ceph.yaml`` file output by `openstack overcloud ceph +deploy`. Because these resources are created directly on the Ceph +cluster with admin level access, "Deployed Ceph" is different from +the "External Ceph" feature described in :doc:`ceph_external`. + +The main benefits of using "Deployed Ceph" are the following: + +- Use cephadm to deploy Ceph on the hardware managed by TripleO + without having to write your own cephadm spec file (though you may + provide your own if you wish) +- Focus on debugging the basic Ceph deployment without debugging the + overcloud deployment at the same time +- Fix any Ceph deployment problems directly using either Ansible or + the Ceph orchestrator tools before starting the overcloud deployment +- Have the benefits above while maintaining hyperconverged support by + using a tested workflow + +In summary, `openstack overcloud ceph deploy` deploys the Ceph cluster +while `openstack overcloud deploy` (and the commands that follow) +deploy OpenStack and configure that Ceph cluster to be used by +OpenStack. + +Deployed Ceph Command Line Interface +------------------------------------ + +The command line interface supports the following options:: + + $ openstack overcloud ceph deploy --help + usage: openstack overcloud ceph deploy [-h] -o [-y] + [--stack STACK] + [--working-dir WORKING_DIR] + [--roles-data ROLES_DATA] + [--ceph-spec CEPH_SPEC | --osd-spec OSD_SPEC] + [--container-image-prepare CONTAINER_IMAGE_PREPARE] + [--container-namespace CONTAINER_NAMESPACE] + [--container-image CONTAINER_IMAGE] + [--container-tag CONTAINER_TAG] + [--registry-url REGISTRY_URL] + [--registry-username REGISTRY_USERNAME] + [--registry-password REGISTRY_PASSWORD] + + + positional arguments: + + Path to the environment file output from "openstack + overcloud node provision". + + optional arguments: + -h, --help show this help message and exit + -o , --output + The path to the output environment file describing the + Ceph deployment to pass to the overcloud deployment. + -y, --yes Skip yes/no prompt before overwriting an existing + output file (assume yes). + --stack STACK Name or ID of heat stack (default=Env: + OVERCLOUD_STACK_NAME) + --working-dir WORKING_DIR + The working directory for the deployment where all + input, output, and generated files will be stored. + Defaults to "$HOME/overcloud-deploy/" + --roles-data ROLES_DATA + Path to an alternative roles_data.yaml. Used to decide + which node gets which Ceph mon, mgr, or osd service + based on the node's role in . + --ceph-spec CEPH_SPEC + Path to an existing Ceph spec file. If not provided a + spec will be generated automatically based on --roles- + data and + --osd-spec OSD_SPEC Path to an existing OSD spec file. Mutually exclusive + with --ceph-spec. If the Ceph spec file is generated + automatically, then the OSD spec in the Ceph spec file + defaults to {data_devices: {all: true}} for all + service_type osd. Use --osd-spec to override the + data_devices value inside the Ceph spec file. + --container-image-prepare CONTAINER_IMAGE_PREPARE + Path to an alternative + container_image_prepare_defaults.yaml. Used to control + which Ceph container is pulled by cephadm via the + ceph_namespace, ceph_image, and ceph_tag variables in + addition to registry authentication via + ContainerImageRegistryCredentials. + + container-image-prepare overrides: + The following options may be used to override individual values set via + --container-image-prepare. If the example variables below were set the + image would be concatenated into quay.io/ceph/ceph:latest and a custom + registry login would be used. + + --container-namespace CONTAINER_NAMESPACE + e.g. quay.io/ceph + --container-image CONTAINER_IMAGE + e.g. ceph + --container-tag CONTAINER_TAG + e.g. latest + --registry-url REGISTRY_URL + --registry-username REGISTRY_USERNAME + --registry-password REGISTRY_PASSWORD + + This command is provided by the python-tripleoclient plugin. + $ + +Run `openstack overcloud ceph deploy --help` in your own environment +to see the latest options which you have available. + +Ceph Spec Options +----------------- + +The roles file, described in the next section, and the output of +`openstack overcloud node provision` are passed to the +`ceph_spec_bootstrap`_ Ansible module to create a `Ceph Service +Specification`_. The `openstack overcloud ceph deploy` command does +this automatically so it is not necessary to use the options described +in this section unless desired. + +It's possible to generate a Ceph Spec on the undercloud before +deployment by using the `ceph_spec_bootstrap`_ Ansible module +directly, for example:: + + ansible localhost -m ceph_spec_bootstrap \ + -a deployed_metalsmith=deployed_metal.yaml + +By default the above creates the file ``~/ceph_spec.yaml``. For more +information on the ``ceph_spec_bootstrap`` module run `ansible-doc +ceph_spec_bootstrap`. The spec file may then be edited if desired and +passed directly like this:: + + openstack overcloud ceph deploy \ + deployed_metal.yaml \ + -o deployed_ceph.yaml \ + --ceph-spec ~/ceph_spec.yaml + +All available disks (excluding the disk where the operating system is +installed) are used as OSDs as per the following default inside the +ceph spec:: + + data_devices: + all: true + +In the above example, the `data_devices` key is valid for any `Ceph +Service Specification`_ whose `service_type` is "osd". Other OSD +service types, as found in the `Advanced OSD Service +Specifications`_, may be set by using the ``--osd-spec`` option. + +If the file ``osd_spec.yaml`` contains the following:: + + data_devices: + rotational: 1 + db_devices: + rotational: 0 + +and the following command is run:: + + openstack overcloud ceph deploy \ + deployed_metal.yaml \ + -o deployed_ceph.yaml \ + --osd-spec osd_spec.yaml + +Then all rotating devices will be data devices and all non-rotating +devices will be used as shared devices (wal, db). This is because when +the dynamic Ceph service specification is built whatever is in the +file referenced by ``--osd-spec`` will be appended to the section of +the specification if the `service_type` is "osd". + +Service Placement Options +------------------------- + +The Ceph services defined in the roles_data.yaml file as described in +:doc:`composable_services` determine which baremetal node runs which +service. By default the Controller role has the CephMon and CephMgr +service while the CephStorage role has the CephOSD service. Most +composable services require Heat output in order to determine how +services are configured, but not the Ceph services. Thus, the +roles_data.yaml file remains authoritative for Ceph service placement +even though the "Deployed Ceph" process happens before Heat is run. + +It is only necessary to use the `--roles-file` option if the default +roles_data.yaml file is not being used. For example if you intend to +deploy hyperconverged nodes, then you want the predeployed compute +nodes to be in the ceph spec with the "osd" label and for the +`service_type` "osd" to have a placement list containing a list of the +compute nodes. To do this generate a custom roles file as described in +:doc:`composable_services` like this:: + + openstack overcloud roles generate Controller ComputeHCI > custom_roles.yaml + +and then pass that roles file like this:: + + openstack overcloud ceph deploy \ + deployed_metal.yaml \ + -o deployed_ceph.yaml \ + --roles-data custom_roles.yaml + +After running the above the compute nodes should have running OSD +containers and when the overcloud is deployed Nova compute services +will then be set up on the same hosts. + +If you wish to generate the ceph spec with the modified placement +described above before the ceph deployment, then the same file may be +passed to a direct call of the ceph_spec_bootstrap ansible module:: + + ansible localhost -m ceph_spec_bootstrap \ + -a "deployed_metalsmith=deployed_metal.yaml tripleo_roles=custom_roles.yaml" + +Container Options +----------------- + +As described in :doc:`../deployment/container_image_prepare` the +undercloud may be used as a container registry for ceph containers +and there is a supported syntax to download containers from +authenticated registries. By default `openstack overcloud ceph deploy` +will pull the Ceph container in the default +``container_image_prepare_defaults.yaml`` file. The version of the +Ceph used in each OpenStack release changes per release and can be +seen by running a command like this:: + + egrep "ceph_namespace|ceph_image|ceph_tag" \ + /usr/share/tripleo-common/container-images/container_image_prepare_defaults.yaml + +The `--container-image-prepare` option can be used to override which +``container_image_prepare_defaults.yaml`` file is used. If a version +of this file called ``custom_container_image_prepare.yaml`` is +modified to contain syntax like the following:: + + ContainerImageRegistryCredentials: + quay.io/ceph-ci: + quay_username: quay_password + +Then when a command like the following is run:: + + openstack overcloud ceph deploy \ + deployed_metal.yaml \ + -o deployed_ceph.yaml \ + --container-image-prepare custom_container_image_prepare.yaml + +The credentials will be extracted from the file and the tripleo +ansible role to bootstrap Ceph will be executed like this:: + + cephadm bootstrap + --registry-url quay.io/ceph-ci + --registry-username quay_username + --registry-password quay_password + ... + +The syntax of the container image prepare file can also be ignored and +instead the following command line options may be used instead:: + + --container-namespace CONTAINER_NAMESPACE + e.g. quay.io/ceph + --container-image CONTAINER_IMAGE + e.g. ceph + --container-tag CONTAINER_TAG + e.g. latest + --registry-url REGISTRY_URL + --registry-username REGISTRY_USERNAME + --registry-password REGISTRY_PASSWORD + +If a variable above is unused, then it defaults to the ones found in +the default ``container_image_prepare_defaults.yaml`` file. In other +words, the above options are overrides. + +.. _`ceph_spec_bootstrap`: https://docs.openstack.org/tripleo-ansible/latest/modules/modules-ceph_spec_bootstrap.html +.. _`Ceph Service Specification`: https://docs.ceph.com/en/latest/cephadm/service-management/#orchestrator-cli-service-spec +.. _`Advanced OSD Service Specifications`: https://docs.ceph.com/en/latest/cephadm/osd/#drivegroups