diff --git a/specs/2024.2/approved/ephemeral-encryption-libvirt.rst b/specs/2024.2/approved/ephemeral-encryption-libvirt.rst new file mode 100644 index 000000000..08612e682 --- /dev/null +++ b/specs/2024.2/approved/ephemeral-encryption-libvirt.rst @@ -0,0 +1,306 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +======================================================================== +libvirt driver support for flavor and image defined ephemeral encryption +======================================================================== + +https://blueprints.launchpad.net/nova/+spec/ephemeral-encryption-libvirt + +This spec outlines the specific libvirt virt driver implementation to support +the Flavor and Image defined ephemeral storage encryption [1]_ spec. + +Problem description +=================== + +The libvirt virt driver currently provides very limited support for ephemeral +disk encryption through the ``LVM`` imagebackend and the use of the ``PLAIN`` +encryption format provided by ``dm-crypt``. + +As outlined in the Flavor and Image defined ephemeral storage encryption [1]_ +spec this current implementation is controlled through compute host +configurables and is transparent to end users, unlike block storage volume +encryption via Cinder. + +With the introduction of the Flavor and Image defined ephemeral storage +encryption [1]_ spec we can now implement support for encrypting ephemeral +disks via images and flavors, allowing support for newer encryption formats +such as `LUKSv1`. This also has the benefit of being natively supported by +`QEMU`, as already seen in the libvirt driver when attaching `LUKSv1` +encrypted volumes provided by Cinder. + +Use Cases +--------- + +* As a user of a cloud with libvirt based computes I want to request that all + of my ephemeral storage be encrypted at rest through the selection of a + specific flavor or image. + +* As a user of a cloud with libvirt based computes I want to be able to pick + how my ephemeral storage be encrypted at rest through the selection of a + specific flavor or image. + +* As a user I want each encrypted ephemeral disk attached to my instance to + have a separate unique secret associated with it. + +* As an operator I want to allow users to request that the ephemeral storage of + their instances is encrypted using the flexible ``LUKSv1`` encryption format. + +Proposed change +=============== + +Deprecate the legacy implementation within the libvirt driver +------------------------------------------------------------- + +The legacy implementation using ``dm-crypt`` within the libvirt virt driver +needs to be deprecated ahead of removal in a future release, this includes the +following options: + +* ``[ephemeral_storage_encryption]/enabled`` +* ``[ephemeral_storage_encryption]/cipher`` +* ``[ephemeral_storage_encryption]/key_size`` + +Limited support for ``dm-crypt`` will be introduced using the new framework +before this original implementation is removed. + +Populate disk_info with encryption properties +--------------------------------------------- + +The libvirt driver has an additional ``disk_info`` dict built from the contents +of the previously mentioned ``block_device_info`` and image metadata associated +with an instance. With the introduction of the ``DriverImageBlockDevice`` +within the Flavor and Image defined ephemeral storage encryption [1]_ spec we +can now avoid the need to look again at image metadata while also adding some +ephemeral encryption related metadata to the dict. + +This dict currently contains the following: + +``disk_bus`` + The default bus used by disks + +``cdrom_bus`` + The default bus used by cd-rom drives + +``mapping`` + A nested dict keyed by disk name including information about each disk. + +Each item within the ``mapping`` dict containing following keys: + +``bus`` + The bus for this disk + +``dev`` + The device name for this disk as known to libvirt + +``type`` + A type from the BlockDeviceType enum ('disk', 'cdrom','floppy', + 'fs', or 'lun') + +It can also contain the following optional keys: + +``format`` + Used to format swap/ephemeral disks before passing to instance (e.g. + 'swap', 'ext4') + +``boot_index`` + The 1-based boot index of the disk. + +In addition to the above this spec will also optionally add the following keys +for encrypted disks: + +``encryption_format`` + The encryption format used by the disk + +``encryption_options`` + A dict of encryption options + +``encryption_secret_uuid`` + The UUID of the encryption secret associated with the disk + +``backing_encryption_secret_uuid`` + The UUID of the encryption secret for the backing file associated with the + disk in the case of qcow2. + +Handle ephemeral disk encryption within imagebackend +---------------------------------------------------- + +With the above in place we can now add encryption support within each image +backend. As highlighted at the start of this spec this initial support will +only be for the ``LUKSv1`` encryption format. + +Generic key management code will be introduced into the base +``nova.virt.libvirt.imagebackend.Image`` class and used to create and store the +encryption secret within the configured key manager. The initial ``LUKSv1`` +support will store a passphrase for each disk within the key manager. This is +unlike the current ephemeral storage encryption or encrypted volume +implementations that currently store a symmetric key in the key manager. This +remains a long running piece of technical debt in the encrypted volume +implementation as ``LUKSv1`` does not directly encrypt data with the provided +key. + +The base ``nova.virt.libvirt.imagebackend.Image`` class will also be extended +to accept and store the optional encryption details provided by ``disk_info`` +above including the format, options and secret UUID(s). + +Each backend will then be modified to encrypt disks during +``nova.virt.libvirt.imagebackend.Image.create_image`` using the provided +format, options and secret. + +Enable the ``COMPUTE_EPHEMERAL_ENCRYPTION_LUKS`` trait +------------------------------------------------------ + +Finally, with the above support in place the ``COMPUTE_EPHEMERAL_ENCRYPTION`` +and ``COMPUTE_EPHEMERAL_ENCRYPTION_LUKS`` traits can be enabled when using a +backend that supports ``LUKSv1``. This will in turn enable scheduling to the +compute of any user requests asking for ephemeral storage encryption using the +format. + +Alternatives +------------ + +Continue to use the transparent host configurables and expand support to other +encryption formats such as ``LUKS``. + +Data model impact +----------------- + +As discussed above the ephemeral encryption keys will be added to the disk_info +for individual disks within the libvirt driver. + +REST API impact +--------------- + +N/A + +Security impact +--------------- + +This should hopefully be positive given the unique secret per disk and user +visible choice regarding how their ephemeral storage is encrypted at rest. + +Notifications impact +-------------------- + +N/A + +Other end user impact +--------------------- + +Users will now need to opt-in to ephemeral storage encryption being used by +their instances through their choice of image or flavors. + +Performance Impact +------------------ + +QEMU will natively decrypt these ``LUKSv1`` ephemeral disks for us using the +``libgcrypt`` library. While there have been performance issues with this in +the past workarounds [2]_ can be implemented that use ``dm-crypt`` instead. + +Other deployer impact +--------------------- + +N/A + +Developer impact +---------------- + +This spec will aim to implement ``LUKSv1`` support for all imagebackends but in +the future any additional encryption formats supported by these backends will +need to ensure matching traits are also enabled. + +Upgrade impact +-------------- + +The legacy implementation is deprecated but will continue to work for the time +being. As the new implementation is separate there is no further upgrade +impact. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + melwitt + +Other contributors: + lyarwood + +Feature Liaison +--------------- + +Feature liaison: + melwitt + +Work Items +---------- + +* Populate the individual disk dicts within ``disk_info`` with any + ephemeral encryption properties. + +* Provide these properties to the imagebackends when creating each disk. + +* Introduce support for ``LUKSv1`` based encryption within the imagebackends. + +* Enable the ``COMPUTE_EPHEMERAL_ENCRYPTION_LUKS`` trait when the selected + imagebackend supports ``LUKSv1``. + +Dependencies +============ + +* Flavor and Image defined ephemeral storage encryption [1]_ + +Testing +======= + +Unlike the parent spec once imagebackends support ``LUKSv1`` and enable the +required trait we can introduce Tempest based testing of this implementation in +addition to extensive functional and unit based tests. + +Documentation Impact +==================== + +* New user documentation around the specific ``LUKSv1`` support for ephemeral + encryption within the libvirt driver. + +* Reference documentation around the changes to the virt block device layer. + +* Document that for the ``raw`` imagebackend, both ``[libvirt]images_type = + raw`` and ``[DEFAULT]use_cow_images = False`` must be configured in order for + resize to work. This is also true without encryption but it may still be + helpful to users. + +* Document that a user must have policy permission to create secrets in + Barbican in order for encryption to work for that user. Secrets are created + in Barbican using the user's auth token. Admins have permission to create + secrets in Barbican by default. + +References +========== + +.. [1] https://specs.openstack.org/openstack/nova-specs/specs/2023.2/approved/ephemeral-encryption.html +.. [2] https://docs.openstack.org/nova/victoria/configuration/config.html#workarounds.disable_native_luksv1 + +.. list-table:: Revisions + :header-rows: 1 + + * - Release Name + - Description + * - Wallaby + - Introduced + * - Yoga + - Reproposed + * - Zed + - Reproposed + * - 2023.1 Antelope + - Reproposed + * - 2023.2 Bobcat + - Reproposed + * - 2024.1 Caracal + - Reproposed + * - 2024.2 Dalmation + - Reproposed diff --git a/specs/2024.2/approved/ephemeral-storage-encryption.rst b/specs/2024.2/approved/ephemeral-storage-encryption.rst new file mode 100644 index 000000000..abc364fec --- /dev/null +++ b/specs/2024.2/approved/ephemeral-storage-encryption.rst @@ -0,0 +1,918 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +====================================================== +Flavour and Image defined ephemeral storage encryption +====================================================== + +https://blueprints.launchpad.net/nova/+spec/ephemeral-storage-encryption + +This spec outlines a new approach to ephemeral storage encryption in Nova +allowing users to select how their ephemeral storage is encrypted at rest +through the use of flavors with specific extra specs or images with specific +properties. The aim being to bring the ephemeral storage encryption experience +within Nova in line with the block storage encryption implementation provided +by Cinder where user selectable `encrypted volume types`_ are available. + +.. note:: + + This spec will only cover the high level changes to the API and compute + layers, implementation within specific virt drivers is left for separate + specs. + +------------------- +Problem description +------------------- + +At present the only in-tree ephemeral storage encryption support is provided by +the libvirt virt driver when using the lvm imagebackend. The current +implementation provides basic operator controlled and configured host specific +support for ephemeral disk encryption at rest where all instances on a given +compute are forced to use encrypted ephemeral storage using the dm-crypt +``PLAIN`` encryption format. + +This is not ideal and makes ephemeral storage encryption completely opaque +to the end user as opposed to the block storage encryption support provided by +Cinder where users are able to opt-in to using admin defined encrypted volume +types to ensure their storage is encrypted at rest. + +Additionally the current implementation uses a single symmetric key to encrypt +all ephemeral storage associated with the instance. As the ``PLAIN`` encryption +format is used there is no way to rotate this key in-place. + +Use Cases +========= + +* As a user I want to request that all of my ephemeral storage is encrypted + at rest through the selection of a specific flavor or image. + +* As a user I want to be able to pick how my ephemeral storage is encrypted + at rest through the selection of a specific flavor or image. + +* As an admin/operator I want to either enforce ephemeral encryption per flavor + or per image. + +* As an admin/operator I want to provide sane choices to my end users regarding + how their ephemeral storage is encrypted at rest. + +* As a virt driver maintainer/developer I want to indicate that my driver + supports ephemeral storage encryption using a specific encryption format. + +* As a virt driver maintainer/developer I want to provide sane default + encryption format and options for users looking to encrypt their ephemeral + storage at rest. I want these associated with the encrypted storage until it + is deleted. + +--------------- +Proposed change +--------------- + +To enable this new flavor extra specs, image properties and host configurables +will be introduced. These will control when and how ephemeral storage +encryption at rest is enabled for an instance. + +.. note:: + + The following ``hw_ephemeral_encryption`` image properties do not relate to + if an image is encrypted at rest within the Glance service. They only relate + to how ephemeral storage will be encrypted at rest when used by a + provisioned instance within Nova. + + Separate image properties have been documented in the + `Glance image encryption`_ and `Cinder image encryption`_ specs to cover + how images can be encrypted at rest within Glance. + +Allow ephemeral encryption to be configured by flavor, image, or config +======================================================================= + +To enable ephemeral encryption per instance the following boolean based flavor +extra spec and image property will be introduced: + +* ``hw:ephemeral_encryption`` +* ``hw_ephemeral_encryption`` + +The above will enable ephemeral storage encryption for an instance but does not +control the encryption format used or the associated options. For this the +following flavor extra specs, image properties and configurables will be +introduced. + +The encryption format used will be controlled by the following flavor extra +specs and image properties: + +* ``hw:ephemeral_encryption_format`` +* ``hw_ephemeral_encryption_format`` + +When neither of the above are provided but ephemeral encryption is still +requested an additional host configurable will be used to provide a default +format per compute, this will initially default to ``luks``: + +* ``[ephemeral_storage_encryption]/default_format`` + +This could lead to requests against different clouds resulting in a different +ephemeral encryption format being used but as this is transparent to the end +user from within the instance it shouldn't have any real impact. + +The format will be provided as a string that maps to a +``BlockDeviceEncryptionFormatTypeField`` oslo.versionedobjects field value: + +* ``plain`` for the plain dm-crypt format +* ``luks`` for the LUKSv1 format + +To enable snapshot and shelve of instances using ephemeral encryption, the UUID +of the encryption secret is stored in the key manager for the resultant image +will be kept with the image as an image property: + +* ``hw_ephemeral_encryption_secret_uuid`` + +The secret UUID is needed when creating an instance from an ephemeral encrypted +snapshot or when unshelving an ephemeral encrypted instance. + +Create a new key manager secret for each block device mapping +============================================================= + +The approach for disk image secrets is to never share secrets between different +disk images and that each disk image has a unique secret. This is done to +address both 1) the security implications and 2) the logistics of cleaning up +secrets that are no longer in use. + +For example: + +Let's say ``Instance A`` has 3 disks: one root disk, one ephemeral disk, and +one swap disk. Each disk will have its own secret. + +This table is intended to illustrate the way secrets are handled in various +scenarios. + +.. table:: + :align: left + + +--------------------+-------------+--------------------------------------+------------------------------------------------------+ + | Instance or Image | Disk | Secret | Notes | + | | | (passphrase) | | + +====================+=============+======================================+======================================================+ + | Instance A | disk (root) | Secret 1 | Secret 1, 2, and 3 will be automatically deleted | + | +-------------+--------------------------------------+ by Nova when Instance A is deleted and its disks are | + | | disk.eph0 | Secret 2 | destroyed | + | +-------------+--------------------------------------+ | + | | disk.swap | Secret 3 | | + +--------------------+-------------+--------------------------------------+------------------------------------------------------+ + | Image Z (snapshot) | disk (root) | Secret 4 (copy of Secret 1 **or** | Secret 4 will **not** be automatically deleted and | + | created from | | new passphrase) | manual deletion will be needed if/when Image Z is | + | Instance A | | | deleted from Glance | + +--------------------+-------------+--------------------------------------+------------------------------------------------------+ + | Instance B | disk (root) | Secret 5 | Secret 5, 6, 7, and 8 will be automatically deleted | + | created from | +--------------------------------------+ by Nova when Instance B is deleted and its disks are | + | Image Z (snapshot) | | Secret 6 :sup:`*` (copy of Secret 4) | destroyed | + | +-------------+--------------------------------------+ | + | | disk.eph0 | Secret 7 | | + | +-------------+--------------------------------------+ | + | | disk.swap | Secret 8 | | + +--------------------+-------------+--------------------------------------+------------------------------------------------------+ + | Instance C | disk (root) | Secret 9 | Secret 9, 10, and 11 will be automatically deleted | + | +-------------+--------------------------------------+ by Nova when Instance C is deleted and its disks are | + | | disk.eph0 | Secret 10 | destroyed | + | +-------------+--------------------------------------+ | + | | disk.swap | Secret 11 | | + +--------------------+-------------+--------------------------------------+------------------------------------------------------+ + | Image Y (snapshot) | disk (root) | Secret 9 | Secret 9 is **reused** when Instance C is shelved | + | created by shelve | | | in part to prevent the possibility of a change in | + | of Instance C | | | ownership of the root disk secret if, for example, | + | | | | an admin user shelves a non-admin user's instance. | + | | | | This approach could be avoided if there is some way | + | | | | we could create a new secret using the instance's | + | | | | user/project rather than the shelver's user/project | + +--------------------+-------------+--------------------------------------+------------------------------------------------------+ + | Rescue disk | disk (root) | Secret 12 | Secret 12 is stashed in the instance's system | + | created by rescue | | | metadata with key | + | of Instance A | | | ``rescue_disk_ephemeral_encryption_secret_uuid``. | + | | | | This is done because a BDM record for the rescue | + | | | | disk is not going to be persisted to the database. | + | | | | Secret 12 will be automatically deleted by Nova when | + | | | | Instance A is unrescued | + +--------------------+-------------+--------------------------------------+------------------------------------------------------+ + | Rescue disk | disk (root) | Secret 13 | Secret 13 is stashed in the instance's system | + | created by rescue | | | metadata with key | + | of Instance A | | | ``rescue_disk_ephemeral_encryption_secret_uuid``. | + | using encrypted | +--------------------------------------+ This is done because a BDM record for the rescue | + | rescue image | | Secret 14 :sup:`*` | disk is not going to be persisted to the database. | + | | | (copy of rescue image secret) | Secret 13 and 14 will be automatically deleted by | + | | | | Nova when Instance A is unrescued | + +--------------------+-------------+--------------------------------------+------------------------------------------------------+ + +:sup:`*` backing file secret for qcow2 only + +Encrypted backing files (qcow2) +=============================== + +The approach regarding backing files is that they will be encrypted if the +source image for the given disk is encrypted. If the source image from which +the disk is created is not encrypted, the backing file stored internally in +Nova will also not be encrypted. If the source image is encrypted, the backing +file will be encrypted. + +An encrypted backing file uses the same passphrase as the source image from +which it was created. This allows an encrypted backing file to be shared among +multiple instances in the same project. + +Encrypted source images will have the ``hw_ephemeral_encryption_secret_uuid`` +image property in their image metadata. The project that owns the key manager +secret will be allowed to create instances from the encrypted source image. + +Backing files for ephemeral disks and swap disks are never encrypted as they +are always created from blank disks. + +Snapshots of instances with ephemeral encryption +================================================ + +When an instance with ephemeral encryption is snapshotted, the behavior for +encrypting the image snapshot is determined by request parameters which will +be added to the snapshot API. + +The API request parameters are intended to support workflows that involve +sharing of encrypted image snapshots with other projects or users. + +Examples: + +* An instance owner wants to back up their disk + +* An instance owner wants to make a copy of their disk that is encrypted with + a new key + +* An instance owner wants to make a copy of their disk using an existing key + that belongs to a different project or user (provided that project or user + has created the necessary `access control list`_ for the secret) + +* An instance owner wants to create an unencrypted public copy of their disk + +New API microversion for Create Image (createImage Action) +---------------------------------------------------------- + +A new microversion will be added to the `create image API`_ to support +ephemeral encryption options. Users will be able to choose how they want +encryption of the new image snapshot to be handled. They can use the same key +as the image being snapshotted (the default), have Nova generate a new key +and use it to encrypt the image snapshot, provide their own key secret UUID +to use to encrypt the image snapshot, or not encrypt the image snapshot at +all. + +Request for ``POST /servers/{server_id}/action`` with ``createImage`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. list-table:: Request + :align: left + :widths: 20, 5, 5, 70 + :header-rows: 1 + + * - Name + - In + - Type + - Description + * - server_id + - path + - string + - The UUID of the server. + * - createImage + - body + - object + - The action to create a snapshot of the image or the volume(s) of the + server. + * - name + - body + - string + - The display name of an Image. + * - metadata (Optional) + - body + - object + - Metadata key and value pairs for the image. The maximum size for each + metadata key and value pair is 255 bytes. + * - encryption (Optional) + - body + - object + - Encryption options for the image to create. These options apply only to + encrypted local disks. + * - encryption.key + - body + - string + - The key to use to encrypt the image snapshot. Valid values are: + + * ``same``: Use the same key to encrypt the image snapshot. + This is the default. + + * ``new``: Generate a new key and use it to encrypt the image snapshot. + + * ````: The UUID of an existing secret in the + key manager service to use to encrypt the image snapshot. + + * ``none``: Do not encrypt the image snapshot. + +.. code-block:: json + :emphasize-lines: 7-9 + + { + "createImage" : { + "name" : "foo-image", + "metadata": { + "meta_var": "meta_val" + }, + "encryption": { + "key": "same|new||none" + } + } + } + +Request choices for encryption.key: + +``same`` + Use the same key to encrypt the new disk image. This is the default. + +``new`` + Generate a new key to encrypt the new disk image. + +```` + Use the provided ```` to encrypt the new disk image. + +``none`` + Do not encrypt the new disk image. + + +.. note:: + + Ceph release Quincy (v17) does not support creating a cloned image with an + encryption key different from its parent. For this reason, the + ``encryption.key`` request parameter with a value of ``new`` will not be + supported with the ``rbd`` image backend. + + See https://github.com/ceph/ceph/commit/1d3de19 for reference. + +Response for ``POST /servers/{server_id}/action`` with ``createImage`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. list-table:: Request + :align: left + :widths: 20, 5, 5, 70 + :header-rows: 1 + + * - Name + - In + - Type + - Description + * - image_id + - body + - string + - The UUID for the resulting image snapshot. + * - encryption (Optional) + - body + - object + - Encryption information about the resulting image snapshot. + * - encryption.key + - body + - string + - The UUID in the key manager service of the key that was used to encrypt + the resulting image snapshot, if a new key was generated by Nova. + +.. code-block:: json + :emphasize-lines: 3-5 + + { + "image_id": "0e7761dd-ee98-41f0-ba35-05994e446431", + "encryption": { + "key": "" + } + } + +.. _`create image API`: https://docs.openstack.org/api-ref/compute/#create-image-createimage-action +.. _`access control list`: https://docs.openstack.org/barbican/latest/admin/access_control.html + +Create Server Back Up (createBackup Action) API +----------------------------------------------- + +The ``POST /servers/{server_id}/action`` with ``createBackup`` API will not be +changed. Image snapshots created by this API will be encrypted using the same +key. + + +Image metadata for image snaphots of encrypted disks +---------------------------------------------------- + +When an encrypted image snapshot is created, the UUID of the key manager secret +that was used to encrypt it is recorded as an image property +``hw_ephemeral_encryption_secret_uuid`` on the created image that Nova uploads +to Glance. + +When a new instance is created from an encrypted image, the image property +``hw_ephemeral_encryption_secret_uuid`` is passed down to the lower layers by +storing it in the instance's system metadata with key +``image_hw_ephemeral_encryption_secret_uuid``. This is done because at the +lower layers (where ``qemu-img convert`` is called, for example) we no longer +have access to the image metadata and nontrivial refactoring to pass image +metadata to several lower layer methods, or similar, would be required +otherwise. + +Snapshots created by shelving instances with ephemeral encryption +================================================================= + +When an instance with ephemeral encryption is shelved, the existing root disk +encryption secret is **reused** and will be used to unshelve the instance +later. This is done to prevent a potential change in ownership of the root disk +encryption secret in a scenario where an admin user shelves a non-admin user's +instance, for example. If a new secret were created owned by the admin user, +the non-admin user who owns the instance would be unable to unshelve the +instance. + +This behavior could be avoided however if there is some way we could create a +new encryption secret using the instance's user and project rather than the +shelver's user and project. If that were possible, we would not need to reuse +the encryption secret. + +Rescue disk images created by rescuing instances with ephemeral encryption +========================================================================== + +When rescuing an instance and an encrypted rescue image is specified, the +rescue image secret UUID from the image property will be stashed in the +instance's system metadata with key +``rescue_image_hw_ephemeral_encryption_secret_uuid`` to pass it down to the +lower layers. The rescue image encryption format is similiarly stashed in the +instance's system metadata with key +``rescue_image_hw_ephemeral_encryption_format``. This is considered separate +from ``image_hw_ephemeral_encryption_secret_uuid`` which means the encrypted +image from which the instance was created. Another reason to keep it separate +is to avoid confusion for those reading or working on the code. + +A new encryption secret is created when the rescue disk is created and its UUID +is stashed in the instance's system metadata with key +``rescue_disk_ephemeral_encryption_secret_uuid``. This is done because a block +device mapping record for the rescue disk is not going to be persisted to the +database. Similarly, the backing file secret UUID will be stashed in the +instance system metadata with key +``rescue_disk_ephemeral_backing_encryption_secret_uuid`` in the case of qcow2. + +The virt driver secrets for the rescue disk are stashed in the system metadata +as well using keys ``rescue_disk_ephemeral_encryption_secret_usage`` and +``rescue_disk_ephemeral_backing_encryption_secret_usage`` respectively. + +The new encryption secret(s) for the rescue disk is deleted from the key +manager and the virt driver secret are also deleted when the instance is +unrescued. + +Cleanup of ephemeral encryption secrets +======================================= + +Ephemeral encryption secrets are deleted from the key manager and the virt +driver when the corresponding instance is deleted and its disks are destroyed. + +Virt driver secrets may be created on destination hosts and deleted from source +hosts as needed during instance migrations. + +Key manager secrets are however **only** deleted when the disks associated with +them are destroyed. + +Encryption secrets that are created when a snapshot is created are **never** +deleted by Nova. It would only be acceptable to delete the secret if and when +the image snapshot is deleted from Glance. Cleanup of secrets whose images have +been deleted from Glance must be deleted manually by the user or an admin. + +BlockDeviceMapping changes +========================== + +The ``BlockDeviceMapping`` object will be extended to include the following +fields encapsulating some of the above information per ephemeral disk within +the instance: + +``encrypted`` + A simple boolean to indicate if the block device is encrypted. This will + initially only be populated when ephemeral encryption is used but could + easily be used for encrypted volumes as well in the future. + +``encryption_secret_uuid`` + As the name suggests this will contain the UUID of the associated + encryption secret for the disk. The type of secret used here will be + specific to the encryption format and virt driver used, it should not be + assumed that this will always been an symmetric key as is currently the + case with all encrypted volumes provided by Cinder. For example, for + ``luks`` based ephemeral storage this secret will be a ``passphrase``. + +``backing_encryption_secret_uuid`` + This will contain the UUID of the associated encryption secret for the + backing file for the disk in the case of qcow2. + +``encryption_format`` + A new ``BlockDeviceEncryptionFormatType`` enum and associated + ``BlockDeviceEncryptionFormatTypeField`` field listing the encryption + format. The available options being kept in line with the constants + currently provided by os-brick and potentially merged in the future if both + can share these types and fields somehow. + +``encryption_options`` + A simple unversioned dict of strings containing encryption options specific + to the virt driver implementation, underlying hypervisor and format being + used. + +.. note:: + + The ``encryption_options`` field will be unused and not exposed to end + users initially because of the security and upgrade implications around it. + For the first pass, sensible defaults for the cipher algorithm, cipher + mode, and initialization vector generator algorithm will be hard-coded + instead. + + Encryption options could be exposed to end users in the future when a + proper design which addresses security and handles all upgrade scenarios is + developed. + +Populate ephemeral encryption BlockDeviceMapping attributes during build +======================================================================== + +When launching an instance with ephemeral encryption requested via either the +image or flavor the ``BlockDeviceMapping.encrypted`` attribute will be set to +``True`` for each ``BlockDeviceMapping`` record with a ``destination_type`` +value of ``local``. This will happen after the original API BDM dicts have been +transformed into objects within the Compute API but before scheduling the +instance(s). + +The ``encryption_format`` attribute will also take its value from the image or +flavor if provided. Any differences or conflicts between the image and flavor +for this will raise a ``409 Conflict`` error being raised by the API. + +Use ``COMPUTE_EPHEMERAL_ENCRYPTION`` compatibility traits +========================================================= + +A ``COMPUTE_EPHEMERAL_ENCRYPTION`` compute compatibility trait was introduced +during `Wallaby`__ and will be reported by virt drivers to indicate overall +support for ephemeral storage encryption using this new approach. This trait +will always be used by pre-filter outlined in the following section when +ephemeral encryption has been requested, regardless of any format being +specified in the request, allowing the compute that eventually handles the +request to select a format it supports using the +``[ephemeral_storage_encryption]/default_format`` configurable. + +.. __: https://review.opendev.org/c/openstack/os-traits/+/759878 + +``COMPUTE_EPHEMERAL_ENCRYPTION_$FORMAT`` compute compatibility traits were also +added to os-traits during Wallaby and will be reported by virt drivers to +indicate support for specific ephemeral storage encryption formats. For +example: + + +* ``COMPUTE_EPHEMERAL_ENCRYPTION_LUKS`` +* ``COMPUTE_EPHEMERAL_ENCRYPTION_LUKSV2`` +* ``COMPUTE_EPHEMERAL_ENCRYPTION_PLAIN`` + +These traits will only be used alongside the ``COMPUTE_EPHEMERAL_ENCRYPTION`` +trait when the ``hw_ephemeral_encryption_format`` image property or +``hw:ephemeral_encryption_format`` extra spec have been provided in the initial +request. + +Introduce an ephemeral encryption request pre-filter +==================================================== + +A new pre-filter will be introduced that adds the above traits as required to +the request spec when the aforementioned image properties or flavor extra specs +are provided. As outlined above this will always include the +``COMPUTE_EPHEMERAL_ENCRYPTION`` trait when ephemeral encryption has been +requested and may optionally include one of the format specific traits if a +format is included in the request. + +Expose ephemeral encryption attributes via block_device_info +============================================================ + +Once the ``BlockDeviceMapping`` objects have been updated and the instance +scheduled to a compute the objects are transformed once again into a +``block_device_info`` dict understood by the virt layer that at present +contains the following: + +``root_device_name`` + The root device path used by the instance. + +``ephemerals`` + A list of ``DriverEphemeralBlockDevice`` dict objects detailing the + ephemeral disks attached to the instance. Note this does not include the + initial image based disk used by the instance that is classified as an + ephemeral disk in terms of the ephemeral encryption feature. + +``block_device_mapping`` + A list of ``DriverVol*BlockDevice`` dict objects detailing the volume based + disks attached to the instance. + +``swap`` + An optional ``DriverSwapBlockDevice`` dict object detailing the swap + device. + + +For example: + +.. code-block:: json + + { + "root_device_name": "/dev/vda", + "ephemerals": [ + { + "guest_format": null, + "device_name": "/dev/vdb", + "device_type": "disk", + "size": 1, + "disk_bus": "virtio" + } + ], + "block_device_mapping": [], + "swap": { + "swap_size": 1, + "device_name": "/dev/vdc", + "disk_bus": "virtio" + } + } + +As noted above ``block_device_info`` does not provide a complete overview of +the storage associated with an instance. In order for it to be useful in the +context of ephemeral storage encryption we would need to extend the dict to +always include information relating to local image based disks. + +As such a new ``DriverImageBlockDevice`` dict class will be introduced covering +image based block devices and provided to the virt layer via an additional +``image`` key within the ``block_device_info`` dict when the instance uses such +a disk. As with the other ``Driver*BlockDevice`` dict classes this will proxy +access to the underlying ``BlockDeviceMapping`` object allowing the virt layer +to lookup the previously listed ``encrypted`` and ``encryption_*`` attributes. + +While outside the scope of this spec the above highlights a huge amount of +complexity and technical debt still residing in the codebase around how storage +configurations are handled between the different layers. In the long term we +should plan to remove ``block_device_info`` and replace it with direct access +to ``BlockDeviceMapping`` based objects ensuring the entire configuration is +always exposed to the virt layer. + +Report that a disk is encrypted at rest through the metadata API +================================================================ + +Extend the metadata API so that users can confirm that their ephemeral storage +is encrypted at rest through the metadata API, accessible from within their +instance. + +.. code-block:: json + + { + "devices": [ + { + "type": "nic", + "bus": "pci", + "address": "0000:00:02.0", + "mac": "00:11:22:33:44:55", + "tags": ["trusted"] + }, + { + "type": "disk", + "bus": "virtio", + "address": "0:0", + "serial": "12352423", + "path": "/dev/vda", + "encrypted": "True" + }, + { + "type": "disk", + "bus": "ide", + "address": "0:0", + "serial": "disk-vol-2352423", + "path": "/dev/sda", + "tags": ["baz"] + } + ] + } + +This should also be extended to cover disks provided by encrypted volumes but +this is obviously out of scope for this implementation. + +Block resize between flavors with different hw:ephemeral_encryption settings +============================================================================ + +Ephemeral data is expected to persist through a resize and as such any resize +between flavors that differed in their configuration of ephemeral encryption +(one enabled, another disabled or formats etc) would cause us to convert this +data in place. This isn't trivial and so for this initial implementation +resizing between flavors that differ will be blocked. + +Provide a migration path from the legacy implementation +======================================================= + +New ``nova-manage`` and ``nova-status`` commands will be introduced to migrate +any instances using the legacy libvirt virt driver implementation ahead of the +removal of this in a future release. + +The ``nova-manage`` command will ensure that any existing instances with +``ephemeral_key_uuid`` set will have their associated ``BlockDeviceMapping`` +records updated to reference said secret key, the ``plain`` encryption format +and configured options on the host before clearing ``ephemeral_key_uuid``. + +Additionally the libvirt virt driver will also attempt to migrate instances +with ``ephemeral_key_uuid`` set during spawn. This should allow at least some +of the instances to be moved during the W release ahead of X. + +The ``nova-status`` command will simply report on the existence of any +instances with ``ephemeral_key_uuid`` set that do not have the corresponding +``BlockDeviceMapping`` attributes enabled etc. + +Deprecate the now legacy implementation +======================================= + +The legacy implementation within the libvirt virt driver will be deprecated for +removal in a future release once the ability to migrate is in place. + +Alternatives +============ + +Continue to use the transparent host configurables and expand support to other +encryption formats such as ``LUKS``. + +Data model impact +================= + +See above for the various flavor extra spec, image property, +``BlockDeviceMapping`` and ``DriverBlockDevice`` object changes. + +REST API impact +=============== + +* A new API microversion will be created to add encryption options to the + ``createImage`` server action API. + +* Flavor extra specs and image property validation will be introduced for the + any ephemeral encryption provided options. + +* Attempts to resize between flavors that differ in their ephemeral encryption + options will be rejected. + +* Attempts to rebuild between images that differ in their ephemeral encryption + options will be allowed by the user who owns the instance. Requests to + rebuild between images that differ in their ephemeral encryption options will + be rejected. This is to prevent a change in the ownership of secrets for the + instance disks. + +* The metadata API will be changed to allow users to determine if their + ephemeral storage is encrypted as discussed above. + +Security impact +=============== + +This should hopefully be positive given the unique secret per disk and user +visible choice regarding how their ephemeral storage is encrypted at rest. + +Additionally this should allow additional virt drivers to support ephemeral +storage encryption while also allowing the libvirt virt driver to increase +coverage of the feature across more imagebackends such as qcow2 and rbd. + +Notifications impact +==================== + +N/A + +Other end user impact +===================== + +Users will now need to opt-in to ephemeral storage encryption being used by +their instances through their choice of image or flavor. + +Performance Impact +================== + +The additional pre-filter will add a small amount of overhead when scheduling +instances but this should fail fast if ephemeral encryption is not requested +through the image or flavor. + +The performance impact of increased use of ephemeral storage encryption by +instances is left to be discussed in the virt driver specific specs as this +will vary between hypervisors. + +Other deployer impact +===================== + +N/A + +Developer impact +================ + +Virt driver developers will be able to indicate support for specific ephemeral +storage encryption formats using the newly introduced compute compatibility +traits. + +Upgrade impact +============== + +The compute traits should ensure that requests to schedule instances using +ephemeral storage encryption with mixed computes (N-1 and N) will work during a +rolling upgrade. + +As discussed earlier in the spec future upgrades will need to provide a path +for existing ephemeral storage encryption users to migrate from the legacy +implementation. This should be trivial but may require an additional grenade +based job in CI during the W cycle to prove out the migration path. + +-------------- +Implementation +-------------- + +Assignee(s) +=========== + +Primary assignee: + melwitt + +Other contributors: + lyarwood + +Feature Liaison +=============== + +Feature liaison: + melwitt + +Work Items +========== + +* Introduce ``hw_ephemeral_encryption*`` image properties and + ``hw:ephemeral_encryption`` flavor extra specs. + +* Introduce a new ``encrypted``. ``encryption_secret_uuid``, + ``backing_encryption_secret_uuid``, ``encryption_format`` and + ``encryption_options`` attributes to the BlockDeviceMapping Object. + +* Wire up the new ``BlockDeviceMapping`` object attributes through the + ``Driver*BlockDevice`` layer and ``block_device_info`` dict. + +* Report ephemeral storage encryption through the metadata API. + +* Introduce new ``nova-manage`` and ``nova-status`` commands to allow existing + users to migrate to this new implementation. This should however be blocked + outside of testing until a virt driver implementation is landed. + +* Validate all of the above in functional tests ahead of any virt driver + implementation landing. + +------------ +Dependencies +------------ + +None + +------- +Testing +------- + +At present without a virt driver implementation this will be tested entirely +within our unit and functional test suites. + +Once a virt driver implementation is available additional integration tests in +Tempest and whitebox tests can be written. + +Testing of the migration path from the legacy implementation will require an +additional grenade job but this will require the libvirt virt driver +implementation to be completed first. + +-------------------- +Documentation Impact +-------------------- + +* The new host configurables, flavor extra specs and image properties should be + documented. + +* New user documentation should be written covering the overall use of the + feature from a Nova point of view. + +* Reference documentation around `BlockDeviceMapping` objects etc should be + updated to make note of the new encryption attributes. + +---------- +References +---------- + +.. _`Glance image encryption`: https://specs.openstack.org/openstack/glance-specs/specs/victoria/approved/glance/image-encryption.html +.. _`Cinder image encryption`: https://specs.openstack.org/openstack/cinder-specs/specs/wallaby/image-encryption.html +.. _`encrypted volume types`: https://docs.openstack.org/cinder/latest/configuration/block-storage/volume-encryption.html#create-an-encrypted-volume-type +.. _`libvirt virt driver`: https://libvirt.org/formatstorageencryption.html#StorageEncryptionLuks + +------- +History +------- + +.. list-table:: Revisions + :align: left + :header-rows: 1 + + * - Release Name + - Description + * - Wallaby + - Introduced + * - Xena + - Reproposed + * - Yoga + - Reproposed + * - Zed + - Reproposed + * - 2023.1 Antelope + - Reproposed + * - 2023.2 Bobcat + - Reproposed + * - 2024.1 Caracal + - Reproposed + * - 2024.2 Dalmation + - Reproposed