Add config and documentation for ceph perisistent write log cache

Change-Id: I7024e52c4750d4fd7b205cc2376d73bde7b4c11b

defaults/main.yml

@@ -137,3 +137,9 @@ ceph_immutable_object_cache_group: "libvirt-qemu"
 ceph_immutable_object_cache_mode: "0755"
 ceph_immutable_object_cache_key_owner: "{{ ceph_immutable_object_cache_owner }}"
 ceph_immutable_object_cache_key_group: "{{ ceph_immutable_object_cache_group }}"
+
+ceph_persistent_write_log_cache_enabled: False
+ceph_persistent_write_log_cache_dir: "/rbd-write-log-cache"
+ceph_persistent_write_log_cache_owner: "libvirt-qemu"
+ceph_persistent_write_log_cache_group: "ceph"
+ceph_persistent_write_log_cache_mode: "0755"

doc/source/config-persistent-write-log-cache.rst

@@ -0,0 +1,79 @@
+=========================================
+Configuring the Peristent Write Log Cache
+=========================================
+
+The Persistent Write Log Cache is simpler than the Immutable Object Cache
+as it is implemented entirely within the RBD client libraries and needs
+no extra packages, daemon running or client keyring to be installed.
+
+As part of compute node preparation:
+
+Assuming the spare disk available for a write cache is nvme2n1 for example:
+
+..code-block:: console
+
+  mkfs.ext4 /dev/nvme2n1     # create ext4 filesystem on disk
+  mkdir /rbd-write-log-cache
+  mount /dev/nvme2n1 /rbd-write-log-cache
+
+
+The ceph_client ansible role will ensure that the directory permissions
+are set correctly during deployment if
+`ceph_persistent_write_log_cache_enabled: True` is set in
+`/etc/openstack_deploy/group_vars/nova_compute.yml`. The variable
+can be defined globally, or on a per group or per host basis.
+
+To enable the Persistent Write Log Cache the following config must be
+applied to the compute node also through group_vars, so that it is only
+enabled on the nova_compute ceph clients.
+
+Adjust the cache size based on the expected number of volumes mounted on
+the compute node and the size of the cache device. The cache size is
+allocated on the disk seperately for each pool/volume combination that is
+active on the host.
+
+.. code-block:: yaml
+
+    ceph_client_ceph_conf_overrides:
+      global:
+        rbd_plugins = pwl_cache
+        rbd_persistent_cache_mode = ssd
+        rbd_persistent_cache_path = /rbd-write-log-cache
+        rbd_persistent_cache_size = 10G       # size of cache used for each active rbd device
+
+To see the activity within a write-log cache, use the following command
+on the compute host `rbd status -n client.cinder <pool_name>/volume-<volume_uuid>`
+
+Example:
+
+.. code-block:: console
+
+    # rbd status -n client.cinder cinder-volumes-nvme/volume-93f5a8fa-2e73-40c8-a9f1-bbeff3a3e6bc
+
+    Watchers:
+        watcher=10.51.1.134:0/2452041141 client.192434419 cookie=281466789599248
+    Persistent cache state:
+        host: compute1a01
+        path: /rbd-write-log-cache/rbd-pwl.cinder-volumes-nvme.9058c8720de65b.pool
+        size: 10 GiB
+        mode: ssd
+        stats_timestamp: Mon Apr  3 12:13:38 2023
+        present: true    empty: false    clean: true
+        allocated: 48 KiB
+        cached: 24 KiB
+        dirty: 0 B
+        free: 1024 MiB
+        hits_full: 6 / 0%
+        hits_partial: 0 / 0%
+        misses: 160340
+        hit_bytes: 10 KiB / 0%
+        miss_bytes: 20 GiB
+
+
+When a new VM is created, a single 10GB file (whose name includes the ceph pool
+name and a volume id) will be created in the `/rbd-write-log-cache` directory.
+Note this is only used with new VMs created after caching was enabled.
+
+If both Immutible Object Cache and Persistent Write Log are required to be
+enabled on the same node then it is important to define the settings for
+both in a single definiton of `ceph_client_ceph_conf_overrides`.

doc/source/index.rst

@@ -8,6 +8,7 @@ OpenStack-Ansible Ceph client
    configure-ceph.rst
    config-from-file.rst
    config-immutable-object-cache.rst
+   config-persistent-write-log-cache.rst
 
 This Ansible role installs the Ceph operating system
 packages used to interact with a Ceph cluster.

tasks/main.yml

@@ -77,3 +77,13 @@
     - ceph_immutable_object_cache_enabled | bool
   tags:
     - ceph-config
+
+- name: Set permissions on persistent write log cache directory
+  file:
+    path: "{{ ceph_persistent_write_log_cache_dir }}"
+    owner: "{{ ceph_persistent_write_log_cache_owner }}"
+    group: "{{ ceph_persistent_write_log_cache_group }}"
+    mode: "{{ ceph_persistent_write_log_cache_mode }}"
+  when: ceph_persistent_write_log_cache_enabled | bool
+  tags:
+    - ceph-config