From 69b6c94dbae28ca1caa7f17b65330f97c012581f Mon Sep 17 00:00:00 2001 From: Wes Hayutin Date: Tue, 25 Jul 2017 13:49:29 -0400 Subject: [PATCH] Update Pike storage documentation for ceph-ansible The user is required to install ceph-ansible in pike. The package can not be made a dependency of tripleoclient [1] Updated install, update/upgrade, and backup and restore The installation section was updated to cover deployment of Ceph via TripleO with either puppet-ceph (Ceph not in containers) or ceph-ansible (Ceph in containers). [1] http://eavesdrop.openstack.org/meetings/rdo_meeting___2017_06_21/2017/rdo_meeting___2017_06_21.2017-06-21-15.01.log.html#l-29 Co-Authored-By: John Fulton Change-Id: I8e9def9577dc84b63895e10924a01a4949d02aef --- .../advanced_deployment/ceph_config.rst | 148 +++++++++++++++++- .../install/installation/installing.rst | 7 + doc/source/install/installation/updating.rst | 8 + .../backup_restore_undercloud.rst | 6 + .../install/post_deployment/upgrade.rst | 24 ++- 5 files changed, 184 insertions(+), 9 deletions(-) diff --git a/doc/source/install/advanced_deployment/ceph_config.rst b/doc/source/install/advanced_deployment/ceph_config.rst index 2ee65249..a2547f5a 100644 --- a/doc/source/install/advanced_deployment/ceph_config.rst +++ b/doc/source/install/advanced_deployment/ceph_config.rst @@ -1,11 +1,62 @@ Configuring Ceph with Custom Config Settings ============================================ -This guide assumes that your undercloud is already installed and ready to -deploy an overcloud. +This guide assumes that the undercloud is already installed and ready +to deploy an overcloud and that the appropriate repositories +containing Ceph packages, including ceph-ansible if applicable, have +been enabled and installed as described in +:doc:`../installation/installing`. -Customizing ceph.conf ---------------------- +Deploying an Overcloud with Ceph +-------------------------------- + +TripleO can deploy and configure Ceph as if it was a composable +OpenStack service and configure OpenStack services like Nova, Glance, +Cinder, Cinder Backup, and Gnocchi to use it as a storage backend. + +Prior to Pike, TripleO deployed Ceph with `puppet-ceph`_. With the +Pike release it is possible to use TripleO to deploy Ceph with +either `ceph-ansible`_ or puppet-ceph, though puppet-ceph is +deprecated. To deploy Ceph in containers use `ceph-ansible`, for which +only a containerized Ceph deployment is possible. It is not possible +to deploy a containerized Ceph with `puppet-ceph`. + +To deploy with Ceph include either of the appropriate environment +files. For puppet-ceph use "environments/puppet-ceph.yaml" +like the following:: + + openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-ceph.yaml + +For ceph-ansible use "environments/ceph-ansible/ceph-ansible.yaml" +like the following:: + + openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml + +In both the puppet-ceph and ceph-ansible examples above, at least one +Ceph storage node is required. The following example will configure +one Ceph storage nodes on servers matching the `ceph-storage` +profile. It will also set the default pool size, the number of times +that an object should be written for data protection, to one. These +`parameter_defaults` may be saved in an environment file +"~/my-ceph-settings.yaml" and added to the deploy commandline:: + + parameter_defaults: + OvercloudCephStorageFlavor: ceph-storage + CephStorageCount: 1 + CephDefaultPoolSize: 1 + +The values above are only appropriate for a development or POC +deployment. The default pool size is three but if there are less +than three Ceph OSDs, then the cluster will never reach status +`HEALTH_OK` because it has no place to make additional copies. +Thus, a POC deployment with less than three OSDs should override the +default default pool size. However, a production deployment should +replace both of the ones above with threes, or greater, in order to +have at least three storage nodes and at least three back up copies of +each object at minimum. + +Customizing ceph.conf with puppet-ceph +-------------------------------------- Ceph demands for more careful configuration when deployed at scale. @@ -19,10 +70,6 @@ It is possible to override any of the configuration parameters supported by will customize the default `osd_journal_size` overriding any default provided in the `ceph.yaml static hieradata`_. -Remember to add such an environment file to the deploy commandline:: - - openstack overcloud deploy --templates --ceph-storage-scale -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e ~/my-ceph-settings.yaml - It is also possible to provide arbitrary stanza/key/value lines for `ceph.conf` using the special `ceph::conf` configuration class. For example by using:: @@ -40,10 +87,95 @@ the resulting `ceph.conf` file should be populated with the following:: max_open_files: 131072 my_setting: my_value +To specify a set of dedicated block devices to use as Ceph OSDs use +the following:: + + parameter_defaults: + ExtraConfig: + ceph::profile::params::osds: + '/dev/sdb': + journal: '/dev/sde' + '/dev/sdc': + journal: '/dev/sde' + '/dev/sdd': + journal: '/dev/sde' + +The above will produce three OSDs which run on `/dev/sdb`, `/dev/sdc`, +and `/dev/sdd` which all journal to `/dev/sde`. This same setup will +be duplicated per Ceph storage node and assumes uniform hardware. + +The `parameter_defaults` like the above may be saved in an environment +file "~/my-ceph-settings.yaml" and added to the deploy commandline:: + + openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-ceph.yaml -e ~/my-ceph-settings.yaml + +Customizing ceph.conf with ceph-ansible +--------------------------------------- + +The playbooks provided by `ceph-ansible` are triggered by Mistral +workflow. A new `CephAnsibleExtraConfig` parameter has been added to +the templates and can be used to provide arbitrary config variables +consumed by `ceph-ansible`. The pre-existing template params consumed +by the TripleO Pike release to drive `puppet-ceph` continue to work +and are translated, when possible, into their equivalent +`ceph-ansible` variable. + +Global settings in the `ceph.conf` may be set using +`ceph_conf_overrides` like the following:: + + CephAnsibleExtraConfig: + ceph_conf_overrides: + global: + journal_size: 2048 + max_open_files: 131072 + osd_pool_default_size: 3 + osd_pool_default_pg_num: 256 + +`CephAnsibleExtraConfig` isn't just for `ceph.conf` overrides. For +example, to encrypt the data stored on OSDs use the following:: + + CephAnsibleExtraConfig: + dmcrypt: true + +The above overrides the defaults found in the +`ceph-ansible/group_vars`_. + +To specify a set of dedicated block devices to use as Ceph OSDs use +a variation of the following:: + + parameter_defaults: + CephAnsibleDisksConfig: + devices: + - /dev/sdb + - /dev/sdc + - /dev/sdd + raw_journal_devices: + - /dev/sde + - /dev/sde + - /dev/sde + journal_collocation: false + raw_multi_journal: true + +The above will produce three OSDs which run on `/dev/sdb`, `/dev/sdc`, +and `/dev/sdd` which all journal to `/dev/sde`. This same setup will +be duplicated per Ceph storage node and assumes uniform hardware. + +The `parameter_defaults` like the above may be saved in an environment +file "~/my-ceph-settings.yaml" and added to the deploy commandline:: + + openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml -e ~/my-ceph-settings.yaml + .. note:: The :doc:`extra_config` doc has a more details on the usage of the different ExtraConfig interfaces. +.. note:: + + Deployment with `ceph-ansible` requires that OSDs run on dedicated + block devices. + .. _`puppet-ceph`: https://github.com/openstack/puppet-ceph +.. _`ceph-ansible`: https://github.com/ceph/ceph-ansible .. _`ceph.yaml static hieradata`: https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/hieradata/ceph.yaml +.. _`ceph-ansible/group_vars`: https://github.com/ceph/ceph-ansible/tree/master/group_vars diff --git a/doc/source/install/installation/installing.rst b/doc/source/install/installation/installing.rst index 7aab0b29..73d4db05 100644 --- a/doc/source/install/installation/installing.rst +++ b/doc/source/install/installation/installing.rst @@ -69,6 +69,13 @@ Installing the Undercloud sudo yum install -y python-tripleoclient + .. admonition:: Ceph + :class: ceph + + If you intend to deploy Ceph in the overcloud and are running Pike or newer, then install ceph-ansible on the undercloud:: + + sudo yum install -y ceph-ansible + #. Copy in the sample configuration file and edit it to reflect your environment:: cp /usr/share/instack-undercloud/undercloud.conf.sample ~/undercloud.conf diff --git a/doc/source/install/installation/updating.rst b/doc/source/install/installation/updating.rst index 9399e147..fe13de2b 100644 --- a/doc/source/install/installation/updating.rst +++ b/doc/source/install/installation/updating.rst @@ -31,6 +31,14 @@ You can upgrade any packages that are installed on the undercloud machine. sudo yum -y update python-tripleoclient + .. admonition:: Ceph + :class: ceph + + If you are using Pike or newer and Ceph was deployed in the + overcloud, update ceph-ansible on the undercloud:: + + sudo yum -y update ceph-ansible + #. Run the undercloud upgrade command. This command will upgrade all packages and use puppet to apply new configuration and restart all OpenStack services.:: diff --git a/doc/source/install/post_deployment/backup_restore_undercloud.rst b/doc/source/install/post_deployment/backup_restore_undercloud.rst index 6f5e5bbb..82685059 100644 --- a/doc/source/install/post_deployment/backup_restore_undercloud.rst +++ b/doc/source/install/post_deployment/backup_restore_undercloud.rst @@ -66,6 +66,12 @@ Finally we rerun the undercloud installation from the stack user, making sure to # Double check hostname is correctly set in /etc/hosts openstack install undercloud +If you are using Pike and Ceph will be used in the overcloud, install +ceph-ansible on the undercloud:: + + sudo yum install -y ceph-ansible + + Reconnect the restored undercloud to the overcloud -------------------------------------------------- Having completed the steps above, the undercloud can be expected to automatically diff --git a/doc/source/install/post_deployment/upgrade.rst b/doc/source/install/post_deployment/upgrade.rst index 501f713c..013318ff 100644 --- a/doc/source/install/post_deployment/upgrade.rst +++ b/doc/source/install/post_deployment/upgrade.rst @@ -86,12 +86,29 @@ Upgrading the Undercloud you are using a later newton release you don't need to explicitly enable this option. + .. admonition:: Ocata to Pike + :class: mton + + Prior to Pike, TripleO deployed Ceph with puppet-ceph. With the + Pike release it is possible to use TripleO to deploy Ceph with + either ceph-ansible or puppet-ceph, though puppet-ceph is + deprecated. To use ceph-ansible, the CentOS Storage SIG Ceph + repository must be enabled on the undercloud and the + ceph-ansible package must then be installed:: + + sudo yum -y install --enablerepo=extras centos-release-ceph-jewel + sudo yum -y install ceph-ansible + + It is not yet possible to migrate an existing puppet-ceph + deployment to a ceph-ansible deployment. Only new deployments + are currently possible with ceph-ansible. + The following command will upgrade the undercloud:: sudo systemctl stop openstack-* sudo systemctl stop neutron-* sudo systemctl stop httpd - sudo yum -y update instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient + sudo yum -y update instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient ceph-ansible openstack undercloud upgrade Once the undercloud upgrade is fully completed you may @@ -111,6 +128,11 @@ Upgrading the Undercloud .. _validations: ../validations/validations.html#running-a-group-of-validations .. _tripleo-validations: https://github.com/openstack/tripleo-validations/tree/master/validations +.. note:: + + It is not necessary to update ceph-ansible if Ceph is not used in + the overcloud. + Upgrading the Overcloud to Ocata and beyond -------------------------------------------