openstack
/
nova-specs
Code Issues Proposed changes

Re-propose specs for ephemeral encryption 

The driver agnostic spec is updated to include proposed REST API
changes and default behavior for snapshots of encrypted disks.

The libvirt driver spec is updated to add
``backing_encryption_secret_uuid`` to the ``disk_info`` dict.

Related to blueprint ephemeral-storage-encryption
Related to blueprint ephemeral-encryption-libvirt

Change-Id: I9be6902af17d957a0b5187087e76461cdf0dec94
This commit is contained in:
melanie witt 2024-03-05 01:26:25 +00:00
parent 80295ed3c4
commit 7fc7e2c573
2 changed files with 1224 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

306
specs/2024.2/approved/ephemeral-encryption-libvirt.rst Normal file
View File

@ -0,0 +1,306 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
========================================================================
libvirt driver support for flavor and image defined ephemeral encryption
========================================================================
https://blueprints.launchpad.net/nova/+spec/ephemeral-encryption-libvirt
This spec outlines the specific libvirt virt driver implementation to support
the Flavor and Image defined ephemeral storage encryption [1]_ spec.
Problem description
===================
The libvirt virt driver currently provides very limited support for ephemeral
disk encryption through the ``LVM`` imagebackend and the use of the ``PLAIN``
encryption format provided by ``dm-crypt``.
As outlined in the Flavor and Image defined ephemeral storage encryption [1]_
spec this current implementation is controlled through compute host
configurables and is transparent to end users, unlike block storage volume
encryption via Cinder.
With the introduction of the Flavor and Image defined ephemeral storage
encryption [1]_ spec we can now implement support for encrypting ephemeral
disks via images and flavors, allowing support for newer encryption formats
such as `LUKSv1`. This also has the benefit of being natively supported by
`QEMU`, as already seen in the libvirt driver when attaching `LUKSv1`
encrypted volumes provided by Cinder.
Use Cases
---------
* As a user of a cloud with libvirt based computes I want to request that all
of my ephemeral storage be encrypted at rest through the selection of a
specific flavor or image.
* As a user of a cloud with libvirt based computes I want to be able to pick
how my ephemeral storage be encrypted at rest through the selection of a
specific flavor or image.
* As a user I want each encrypted ephemeral disk attached to my instance to
have a separate unique secret associated with it.
* As an operator I want to allow users to request that the ephemeral storage of
their instances is encrypted using the flexible ``LUKSv1`` encryption format.
Proposed change
===============
Deprecate the legacy implementation within the libvirt driver
-------------------------------------------------------------
The legacy implementation using ``dm-crypt`` within the libvirt virt driver
needs to be deprecated ahead of removal in a future release, this includes the
following options:
* ``[ephemeral_storage_encryption]/enabled``
* ``[ephemeral_storage_encryption]/cipher``
* ``[ephemeral_storage_encryption]/key_size``
Limited support for ``dm-crypt`` will be introduced using the new framework
before this original implementation is removed.
Populate disk_info with encryption properties
---------------------------------------------
The libvirt driver has an additional ``disk_info`` dict built from the contents
of the previously mentioned ``block_device_info`` and image metadata associated
with an instance. With the introduction of the ``DriverImageBlockDevice``
within the Flavor and Image defined ephemeral storage encryption [1]_ spec we
can now avoid the need to look again at image metadata while also adding some
ephemeral encryption related metadata to the dict.
This dict currently contains the following:
``disk_bus``
The default bus used by disks
``cdrom_bus``
The default bus used by cd-rom drives
``mapping``
A nested dict keyed by disk name including information about each disk.
Each item within the ``mapping`` dict containing following keys:
``bus``
The bus for this disk
``dev``
The device name for this disk as known to libvirt
``type``
A type from the BlockDeviceType enum ('disk', 'cdrom','floppy',
'fs', or 'lun')
It can also contain the following optional keys:
``format``
Used to format swap/ephemeral disks before passing to instance (e.g.
'swap', 'ext4')
``boot_index``
The 1-based boot index of the disk.
In addition to the above this spec will also optionally add the following keys
for encrypted disks:
``encryption_format``
The encryption format used by the disk
``encryption_options``
A dict of encryption options
``encryption_secret_uuid``
The UUID of the encryption secret associated with the disk
``backing_encryption_secret_uuid``
The UUID of the encryption secret for the backing file associated with the
disk in the case of qcow2.
Handle ephemeral disk encryption within imagebackend
----------------------------------------------------
With the above in place we can now add encryption support within each image
backend. As highlighted at the start of this spec this initial support will
only be for the ``LUKSv1`` encryption format.
Generic key management code will be introduced into the base
``nova.virt.libvirt.imagebackend.Image`` class and used to create and store the
encryption secret within the configured key manager. The initial ``LUKSv1``
support will store a passphrase for each disk within the key manager. This is
unlike the current ephemeral storage encryption or encrypted volume
implementations that currently store a symmetric key in the key manager. This
remains a long running piece of technical debt in the encrypted volume
implementation as ``LUKSv1`` does not directly encrypt data with the provided
key.
The base ``nova.virt.libvirt.imagebackend.Image`` class will also be extended
to accept and store the optional encryption details provided by ``disk_info``
above including the format, options and secret UUID(s).
Each backend will then be modified to encrypt disks during
``nova.virt.libvirt.imagebackend.Image.create_image`` using the provided
format, options and secret.
Enable the ``COMPUTE_EPHEMERAL_ENCRYPTION_LUKS`` trait
------------------------------------------------------
Finally, with the above support in place the ``COMPUTE_EPHEMERAL_ENCRYPTION``
and ``COMPUTE_EPHEMERAL_ENCRYPTION_LUKS`` traits can be enabled when using a
backend that supports ``LUKSv1``. This will in turn enable scheduling to the
compute of any user requests asking for ephemeral storage encryption using the
format.
Alternatives
------------
Continue to use the transparent host configurables and expand support to other
encryption formats such as ``LUKS``.
Data model impact
-----------------
As discussed above the ephemeral encryption keys will be added to the disk_info
for individual disks within the libvirt driver.
REST API impact
---------------
N/A
Security impact
---------------
This should hopefully be positive given the unique secret per disk and user
visible choice regarding how their ephemeral storage is encrypted at rest.
Notifications impact
--------------------
N/A
Other end user impact
---------------------
Users will now need to opt-in to ephemeral storage encryption being used by
their instances through their choice of image or flavors.
Performance Impact
------------------
QEMU will natively decrypt these ``LUKSv1`` ephemeral disks for us using the
``libgcrypt`` library. While there have been performance issues with this in
the past workarounds [2]_ can be implemented that use ``dm-crypt`` instead.
Other deployer impact
---------------------
N/A
Developer impact
----------------
This spec will aim to implement ``LUKSv1`` support for all imagebackends but in
the future any additional encryption formats supported by these backends will
need to ensure matching traits are also enabled.
Upgrade impact
--------------
The legacy implementation is deprecated but will continue to work for the time
being. As the new implementation is separate there is no further upgrade
impact.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
melwitt
Other contributors:
lyarwood
Feature Liaison
---------------
Feature liaison:
melwitt
Work Items
----------
* Populate the individual disk dicts within ``disk_info`` with any
ephemeral encryption properties.
* Provide these properties to the imagebackends when creating each disk.
* Introduce support for ``LUKSv1`` based encryption within the imagebackends.
* Enable the ``COMPUTE_EPHEMERAL_ENCRYPTION_LUKS`` trait when the selected
imagebackend supports ``LUKSv1``.
Dependencies
============
* Flavor and Image defined ephemeral storage encryption [1]_
Testing
=======
Unlike the parent spec once imagebackends support ``LUKSv1`` and enable the
required trait we can introduce Tempest based testing of this implementation in
addition to extensive functional and unit based tests.
Documentation Impact
====================
* New user documentation around the specific ``LUKSv1`` support for ephemeral
encryption within the libvirt driver.
* Reference documentation around the changes to the virt block device layer.
* Document that for the ``raw`` imagebackend, both ``[libvirt]images_type =
raw`` and ``[DEFAULT]use_cow_images = False`` must be configured in order for
resize to work. This is also true without encryption but it may still be
helpful to users.
* Document that a user must have policy permission to create secrets in
Barbican in order for encryption to work for that user. Secrets are created
in Barbican using the user's auth token. Admins have permission to create
secrets in Barbican by default.
References
==========
.. [1] https://specs.openstack.org/openstack/nova-specs/specs/2023.2/approved/ephemeral-encryption.html
.. [2] https://docs.openstack.org/nova/victoria/configuration/config.html#workarounds.disable_native_luksv1
.. list-table:: Revisions
:header-rows: 1
* - Release Name
- Description
* - Wallaby
- Introduced
* - Yoga
- Reproposed
* - Zed
- Reproposed
* - 2023.1 Antelope
- Reproposed
* - 2023.2 Bobcat
- Reproposed
* - 2024.1 Caracal
- Reproposed
* - 2024.2 Dalmation
- Reproposed

918
specs/2024.2/approved/ephemeral-storage-encryption.rst Normal file
View File

@ -0,0 +1,918 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
======================================================
Flavour and Image defined ephemeral storage encryption
======================================================
https://blueprints.launchpad.net/nova/+spec/ephemeral-storage-encryption
This spec outlines a new approach to ephemeral storage encryption in Nova
allowing users to select how their ephemeral storage is encrypted at rest
through the use of flavors with specific extra specs or images with specific
properties. The aim being to bring the ephemeral storage encryption experience
within Nova in line with the block storage encryption implementation provided
by Cinder where user selectable `encrypted volume types`_ are available.
.. note::
This spec will only cover the high level changes to the API and compute
layers, implementation within specific virt drivers is left for separate
specs.
-------------------
Problem description
-------------------
At present the only in-tree ephemeral storage encryption support is provided by
the libvirt virt driver when using the lvm imagebackend. The current
implementation provides basic operator controlled and configured host specific
support for ephemeral disk encryption at rest where all instances on a given
compute are forced to use encrypted ephemeral storage using the dm-crypt
``PLAIN`` encryption format.
This is not ideal and makes ephemeral storage encryption completely opaque
to the end user as opposed to the block storage encryption support provided by
Cinder where users are able to opt-in to using admin defined encrypted volume
types to ensure their storage is encrypted at rest.
Additionally the current implementation uses a single symmetric key to encrypt
all ephemeral storage associated with the instance. As the ``PLAIN`` encryption
format is used there is no way to rotate this key in-place.
Use Cases
=========
* As a user I want to request that all of my ephemeral storage is encrypted
at rest through the selection of a specific flavor or image.
* As a user I want to be able to pick how my ephemeral storage is encrypted
at rest through the selection of a specific flavor or image.
* As an admin/operator I want to either enforce ephemeral encryption per flavor
or per image.
* As an admin/operator I want to provide sane choices to my end users regarding
how their ephemeral storage is encrypted at rest.
* As a virt driver maintainer/developer I want to indicate that my driver
supports ephemeral storage encryption using a specific encryption format.
* As a virt driver maintainer/developer I want to provide sane default
encryption format and options for users looking to encrypt their ephemeral
storage at rest. I want these associated with the encrypted storage until it
is deleted.
---------------
Proposed change
---------------
To enable this new flavor extra specs, image properties and host configurables
will be introduced. These will control when and how ephemeral storage
encryption at rest is enabled for an instance.
.. note::
The following ``hw_ephemeral_encryption`` image properties do not relate to
if an image is encrypted at rest within the Glance service. They only relate
to how ephemeral storage will be encrypted at rest when used by a
provisioned instance within Nova.
Separate image properties have been documented in the
`Glance image encryption`_ and `Cinder image encryption`_ specs to cover
how images can be encrypted at rest within Glance.
Allow ephemeral encryption to be configured by flavor, image, or config
=======================================================================
To enable ephemeral encryption per instance the following boolean based flavor
extra spec and image property will be introduced:
* ``hw:ephemeral_encryption``
* ``hw_ephemeral_encryption``
The above will enable ephemeral storage encryption for an instance but does not
control the encryption format used or the associated options. For this the
following flavor extra specs, image properties and configurables will be
introduced.
The encryption format used will be controlled by the following flavor extra
specs and image properties:
* ``hw:ephemeral_encryption_format``
* ``hw_ephemeral_encryption_format``
When neither of the above are provided but ephemeral encryption is still
requested an additional host configurable will be used to provide a default
format per compute, this will initially default to ``luks``:
* ``[ephemeral_storage_encryption]/default_format``
This could lead to requests against different clouds resulting in a different
ephemeral encryption format being used but as this is transparent to the end
user from within the instance it shouldn't have any real impact.
The format will be provided as a string that maps to a
``BlockDeviceEncryptionFormatTypeField`` oslo.versionedobjects field value:
* ``plain`` for the plain dm-crypt format
* ``luks`` for the LUKSv1 format
To enable snapshot and shelve of instances using ephemeral encryption, the UUID
of the encryption secret is stored in the key manager for the resultant image
will be kept with the image as an image property:
* ``hw_ephemeral_encryption_secret_uuid``
The secret UUID is needed when creating an instance from an ephemeral encrypted
snapshot or when unshelving an ephemeral encrypted instance.
Create a new key manager secret for each block device mapping
=============================================================
The approach for disk image secrets is to never share secrets between different
disk images and that each disk image has a unique secret. This is done to
address both 1) the security implications and 2) the logistics of cleaning up
secrets that are no longer in use.
For example:
Let's say ``Instance A`` has 3 disks: one root disk, one ephemeral disk, and
one swap disk. Each disk will have its own secret.
This table is intended to illustrate the way secrets are handled in various
scenarios.
.. table::
:align: left
+--------------------+-------------+--------------------------------------+------------------------------------------------------+
| Instance or Image | Disk | Secret | Notes |
| | | (passphrase) | |
+====================+=============+======================================+======================================================+
| Instance A | disk (root) | Secret 1 | Secret 1, 2, and 3 will be automatically deleted |
| +-------------+--------------------------------------+ by Nova when Instance A is deleted and its disks are |
| | disk.eph0 | Secret 2 | destroyed |
| +-------------+--------------------------------------+ |
| | disk.swap | Secret 3 | |
+--------------------+-------------+--------------------------------------+------------------------------------------------------+
| Image Z (snapshot) | disk (root) | Secret 4 (copy of Secret 1 **or** | Secret 4 will **not** be automatically deleted and |
| created from | | new passphrase) | manual deletion will be needed if/when Image Z is |
| Instance A | | | deleted from Glance |
+--------------------+-------------+--------------------------------------+------------------------------------------------------+
| Instance B | disk (root) | Secret 5 | Secret 5, 6, 7, and 8 will be automatically deleted |
| created from | +--------------------------------------+ by Nova when Instance B is deleted and its disks are |
| Image Z (snapshot) | | Secret 6 :sup:`*` (copy of Secret 4) | destroyed |
| +-------------+--------------------------------------+ |
| | disk.eph0 | Secret 7 | |
| +-------------+--------------------------------------+ |
| | disk.swap | Secret 8 | |
+--------------------+-------------+--------------------------------------+------------------------------------------------------+
| Instance C | disk (root) | Secret 9 | Secret 9, 10, and 11 will be automatically deleted |
| +-------------+--------------------------------------+ by Nova when Instance C is deleted and its disks are |
| | disk.eph0 | Secret 10 | destroyed |
| +-------------+--------------------------------------+ |
| | disk.swap | Secret 11 | |
+--------------------+-------------+--------------------------------------+------------------------------------------------------+
| Image Y (snapshot) | disk (root) | Secret 9 | Secret 9 is **reused** when Instance C is shelved |
| created by shelve | | | in part to prevent the possibility of a change in |
| of Instance C | | | ownership of the root disk secret if, for example, |
| | | | an admin user shelves a non-admin user's instance. |
| | | | This approach could be avoided if there is some way |
| | | | we could create a new secret using the instance's |
| | | | user/project rather than the shelver's user/project |
+--------------------+-------------+--------------------------------------+------------------------------------------------------+
| Rescue disk | disk (root) | Secret 12 | Secret 12 is stashed in the instance's system |
| created by rescue | | | metadata with key |
| of Instance A | | | ``rescue_disk_ephemeral_encryption_secret_uuid``. |
| | | | This is done because a BDM record for the rescue |
| | | | disk is not going to be persisted to the database. |
| | | | Secret 12 will be automatically deleted by Nova when |
| | | | Instance A is unrescued |
+--------------------+-------------+--------------------------------------+------------------------------------------------------+
| Rescue disk | disk (root) | Secret 13 | Secret 13 is stashed in the instance's system |
| created by rescue | | | metadata with key |
| of Instance A | | | ``rescue_disk_ephemeral_encryption_secret_uuid``. |
| using encrypted | +--------------------------------------+ This is done because a BDM record for the rescue |
| rescue image | | Secret 14 :sup:`*` | disk is not going to be persisted to the database. |
| | | (copy of rescue image secret) | Secret 13 and 14 will be automatically deleted by |
| | | | Nova when Instance A is unrescued |
+--------------------+-------------+--------------------------------------+------------------------------------------------------+
:sup:`*` backing file secret for qcow2 only
Encrypted backing files (qcow2)
===============================
The approach regarding backing files is that they will be encrypted if the
source image for the given disk is encrypted. If the source image from which
the disk is created is not encrypted, the backing file stored internally in
Nova will also not be encrypted. If the source image is encrypted, the backing
file will be encrypted.
An encrypted backing file uses the same passphrase as the source image from
which it was created. This allows an encrypted backing file to be shared among
multiple instances in the same project.
Encrypted source images will have the ``hw_ephemeral_encryption_secret_uuid``
image property in their image metadata. The project that owns the key manager
secret will be allowed to create instances from the encrypted source image.
Backing files for ephemeral disks and swap disks are never encrypted as they
are always created from blank disks.
Snapshots of instances with ephemeral encryption
================================================
When an instance with ephemeral encryption is snapshotted, the behavior for
encrypting the image snapshot is determined by request parameters which will
be added to the snapshot API.
The API request parameters are intended to support workflows that involve
sharing of encrypted image snapshots with other projects or users.
Examples:
* An instance owner wants to back up their disk
* An instance owner wants to make a copy of their disk that is encrypted with
a new key
* An instance owner wants to make a copy of their disk using an existing key
that belongs to a different project or user (provided that project or user
has created the necessary `access control list`_ for the secret)
* An instance owner wants to create an unencrypted public copy of their disk
New API microversion for Create Image (createImage Action)
----------------------------------------------------------
A new microversion will be added to the `create image API`_ to support
ephemeral encryption options. Users will be able to choose how they want
encryption of the new image snapshot to be handled. They can use the same key
as the image being snapshotted (the default), have Nova generate a new key
and use it to encrypt the image snapshot, provide their own key secret UUID
to use to encrypt the image snapshot, or not encrypt the image snapshot at
all.
Request for ``POST /servers/{server_id}/action`` with ``createImage``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. list-table:: Request
:align: left
:widths: 20, 5, 5, 70
:header-rows: 1
* - Name
- In
- Type
- Description
* - server_id
- path
- string
- The UUID of the server.
* - createImage
- body
- object
- The action to create a snapshot of the image or the volume(s) of the
server.
* - name
- body
- string
- The display name of an Image.
* - metadata (Optional)
- body
- object
- Metadata key and value pairs for the image. The maximum size for each
metadata key and value pair is 255 bytes.
* - encryption (Optional)
- body
- object
- Encryption options for the image to create. These options apply only to
encrypted local disks.
* - encryption.key
- body
- string
- The key to use to encrypt the image snapshot. Valid values are:
* ``same``: Use the same key to encrypt the image snapshot.
This is the default.
* ``new``: Generate a new key and use it to encrypt the image snapshot.
* ``<encryption secret uuid>``: The UUID of an existing secret in the
key manager service to use to encrypt the image snapshot.
* ``none``: Do not encrypt the image snapshot.
.. code-block:: json
:emphasize-lines: 7-9
{
"createImage" : {
"name" : "foo-image",
"metadata": {
"meta_var": "meta_val"
},
"encryption": {
"key": "same|new|<secret uuid>|none"
}
}
}
Request choices for encryption.key:
``same``
Use the same key to encrypt the new disk image. This is the default.
``new``
Generate a new key to encrypt the new disk image.
``<secret uuid>``
Use the provided ``<secret uuid>`` to encrypt the new disk image.
``none``
Do not encrypt the new disk image.
.. note::
Ceph release Quincy (v17) does not support creating a cloned image with an
encryption key different from its parent. For this reason, the
``encryption.key`` request parameter with a value of ``new`` will not be
supported with the ``rbd`` image backend.
See https://github.com/ceph/ceph/commit/1d3de19 for reference.
Response for ``POST /servers/{server_id}/action`` with ``createImage``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. list-table:: Request
:align: left
:widths: 20, 5, 5, 70
:header-rows: 1
* - Name
- In
- Type
- Description
* - image_id
- body
- string
- The UUID for the resulting image snapshot.
* - encryption (Optional)
- body
- object
- Encryption information about the resulting image snapshot.
* - encryption.key
- body
- string
- The UUID in the key manager service of the key that was used to encrypt
the resulting image snapshot, if a new key was generated by Nova.
.. code-block:: json
:emphasize-lines: 3-5
{
"image_id": "0e7761dd-ee98-41f0-ba35-05994e446431",
"encryption": {
"key": "<secret uuid used to encrypt the image snapshot>"
}
}
.. _`create image API`: https://docs.openstack.org/api-ref/compute/#create-image-createimage-action
.. _`access control list`: https://docs.openstack.org/barbican/latest/admin/access_control.html
Create Server Back Up (createBackup Action) API
-----------------------------------------------
The ``POST /servers/{server_id}/action`` with ``createBackup`` API will not be
changed. Image snapshots created by this API will be encrypted using the same
key.
Image metadata for image snaphots of encrypted disks
----------------------------------------------------
When an encrypted image snapshot is created, the UUID of the key manager secret
that was used to encrypt it is recorded as an image property
``hw_ephemeral_encryption_secret_uuid`` on the created image that Nova uploads
to Glance.
When a new instance is created from an encrypted image, the image property
``hw_ephemeral_encryption_secret_uuid`` is passed down to the lower layers by
storing it in the instance's system metadata with key
``image_hw_ephemeral_encryption_secret_uuid``. This is done because at the
lower layers (where ``qemu-img convert`` is called, for example) we no longer
have access to the image metadata and nontrivial refactoring to pass image
metadata to several lower layer methods, or similar, would be required
otherwise.
Snapshots created by shelving instances with ephemeral encryption
=================================================================
When an instance with ephemeral encryption is shelved, the existing root disk
encryption secret is **reused** and will be used to unshelve the instance
later. This is done to prevent a potential change in ownership of the root disk
encryption secret in a scenario where an admin user shelves a non-admin user's
instance, for example. If a new secret were created owned by the admin user,
the non-admin user who owns the instance would be unable to unshelve the
instance.
This behavior could be avoided however if there is some way we could create a
new encryption secret using the instance's user and project rather than the
shelver's user and project. If that were possible, we would not need to reuse
the encryption secret.
Rescue disk images created by rescuing instances with ephemeral encryption
==========================================================================
When rescuing an instance and an encrypted rescue image is specified, the
rescue image secret UUID from the image property will be stashed in the
instance's system metadata with key
``rescue_image_hw_ephemeral_encryption_secret_uuid`` to pass it down to the
lower layers. The rescue image encryption format is similiarly stashed in the
instance's system metadata with key
``rescue_image_hw_ephemeral_encryption_format``. This is considered separate
from ``image_hw_ephemeral_encryption_secret_uuid`` which means the encrypted
image from which the instance was created. Another reason to keep it separate
is to avoid confusion for those reading or working on the code.
A new encryption secret is created when the rescue disk is created and its UUID
is stashed in the instance's system metadata with key
``rescue_disk_ephemeral_encryption_secret_uuid``. This is done because a block
device mapping record for the rescue disk is not going to be persisted to the
database. Similarly, the backing file secret UUID will be stashed in the
instance system metadata with key
``rescue_disk_ephemeral_backing_encryption_secret_uuid`` in the case of qcow2.
The virt driver secrets for the rescue disk are stashed in the system metadata
as well using keys ``rescue_disk_ephemeral_encryption_secret_usage`` and
``rescue_disk_ephemeral_backing_encryption_secret_usage`` respectively.
The new encryption secret(s) for the rescue disk is deleted from the key
manager and the virt driver secret are also deleted when the instance is
unrescued.
Cleanup of ephemeral encryption secrets
=======================================
Ephemeral encryption secrets are deleted from the key manager and the virt
driver when the corresponding instance is deleted and its disks are destroyed.
Virt driver secrets may be created on destination hosts and deleted from source
hosts as needed during instance migrations.
Key manager secrets are however **only** deleted when the disks associated with
them are destroyed.
Encryption secrets that are created when a snapshot is created are **never**
deleted by Nova. It would only be acceptable to delete the secret if and when
the image snapshot is deleted from Glance. Cleanup of secrets whose images have
been deleted from Glance must be deleted manually by the user or an admin.
BlockDeviceMapping changes
==========================
The ``BlockDeviceMapping`` object will be extended to include the following
fields encapsulating some of the above information per ephemeral disk within
the instance:
``encrypted``
A simple boolean to indicate if the block device is encrypted. This will
initially only be populated when ephemeral encryption is used but could
easily be used for encrypted volumes as well in the future.
``encryption_secret_uuid``
As the name suggests this will contain the UUID of the associated
encryption secret for the disk. The type of secret used here will be
specific to the encryption format and virt driver used, it should not be
assumed that this will always been an symmetric key as is currently the
case with all encrypted volumes provided by Cinder. For example, for
``luks`` based ephemeral storage this secret will be a ``passphrase``.
``backing_encryption_secret_uuid``
This will contain the UUID of the associated encryption secret for the
backing file for the disk in the case of qcow2.
``encryption_format``
A new ``BlockDeviceEncryptionFormatType`` enum and associated
``BlockDeviceEncryptionFormatTypeField`` field listing the encryption
format. The available options being kept in line with the constants
currently provided by os-brick and potentially merged in the future if both
can share these types and fields somehow.
``encryption_options``
A simple unversioned dict of strings containing encryption options specific
to the virt driver implementation, underlying hypervisor and format being
used.
.. note::
The ``encryption_options`` field will be unused and not exposed to end
users initially because of the security and upgrade implications around it.
For the first pass, sensible defaults for the cipher algorithm, cipher
mode, and initialization vector generator algorithm will be hard-coded
instead.
Encryption options could be exposed to end users in the future when a
proper design which addresses security and handles all upgrade scenarios is
developed.
Populate ephemeral encryption BlockDeviceMapping attributes during build
========================================================================
When launching an instance with ephemeral encryption requested via either the
image or flavor the ``BlockDeviceMapping.encrypted`` attribute will be set to
``True`` for each ``BlockDeviceMapping`` record with a ``destination_type``
value of ``local``. This will happen after the original API BDM dicts have been
transformed into objects within the Compute API but before scheduling the
instance(s).
The ``encryption_format`` attribute will also take its value from the image or
flavor if provided. Any differences or conflicts between the image and flavor
for this will raise a ``409 Conflict`` error being raised by the API.
Use ``COMPUTE_EPHEMERAL_ENCRYPTION`` compatibility traits
=========================================================
A ``COMPUTE_EPHEMERAL_ENCRYPTION`` compute compatibility trait was introduced
during `Wallaby`__ and will be reported by virt drivers to indicate overall
support for ephemeral storage encryption using this new approach. This trait
will always be used by pre-filter outlined in the following section when
ephemeral encryption has been requested, regardless of any format being
specified in the request, allowing the compute that eventually handles the
request to select a format it supports using the
``[ephemeral_storage_encryption]/default_format`` configurable.
.. __: https://review.opendev.org/c/openstack/os-traits/+/759878
``COMPUTE_EPHEMERAL_ENCRYPTION_$FORMAT`` compute compatibility traits were also
added to os-traits during Wallaby and will be reported by virt drivers to
indicate support for specific ephemeral storage encryption formats. For
example:
* ``COMPUTE_EPHEMERAL_ENCRYPTION_LUKS``
* ``COMPUTE_EPHEMERAL_ENCRYPTION_LUKSV2``
* ``COMPUTE_EPHEMERAL_ENCRYPTION_PLAIN``
These traits will only be used alongside the ``COMPUTE_EPHEMERAL_ENCRYPTION``
trait when the ``hw_ephemeral_encryption_format`` image property or
``hw:ephemeral_encryption_format`` extra spec have been provided in the initial
request.
Introduce an ephemeral encryption request pre-filter
====================================================
A new pre-filter will be introduced that adds the above traits as required to
the request spec when the aforementioned image properties or flavor extra specs
are provided. As outlined above this will always include the
``COMPUTE_EPHEMERAL_ENCRYPTION`` trait when ephemeral encryption has been
requested and may optionally include one of the format specific traits if a
format is included in the request.
Expose ephemeral encryption attributes via block_device_info
============================================================
Once the ``BlockDeviceMapping`` objects have been updated and the instance
scheduled to a compute the objects are transformed once again into a
``block_device_info`` dict understood by the virt layer that at present
contains the following:
``root_device_name``
The root device path used by the instance.
``ephemerals``
A list of ``DriverEphemeralBlockDevice`` dict objects detailing the
ephemeral disks attached to the instance. Note this does not include the
initial image based disk used by the instance that is classified as an
ephemeral disk in terms of the ephemeral encryption feature.
``block_device_mapping``
A list of ``DriverVol*BlockDevice`` dict objects detailing the volume based
disks attached to the instance.
``swap``
An optional ``DriverSwapBlockDevice`` dict object detailing the swap
device.
For example:
.. code-block:: json
{
"root_device_name": "/dev/vda",
"ephemerals": [
{
"guest_format": null,
"device_name": "/dev/vdb",
"device_type": "disk",
"size": 1,
"disk_bus": "virtio"
}
],
"block_device_mapping": [],
"swap": {
"swap_size": 1,
"device_name": "/dev/vdc",
"disk_bus": "virtio"
}
}
As noted above ``block_device_info`` does not provide a complete overview of
the storage associated with an instance. In order for it to be useful in the
context of ephemeral storage encryption we would need to extend the dict to
always include information relating to local image based disks.
As such a new ``DriverImageBlockDevice`` dict class will be introduced covering
image based block devices and provided to the virt layer via an additional
``image`` key within the ``block_device_info`` dict when the instance uses such
a disk. As with the other ``Driver*BlockDevice`` dict classes this will proxy
access to the underlying ``BlockDeviceMapping`` object allowing the virt layer
to lookup the previously listed ``encrypted`` and ``encryption_*`` attributes.
While outside the scope of this spec the above highlights a huge amount of
complexity and technical debt still residing in the codebase around how storage
configurations are handled between the different layers. In the long term we
should plan to remove ``block_device_info`` and replace it with direct access
to ``BlockDeviceMapping`` based objects ensuring the entire configuration is
always exposed to the virt layer.
Report that a disk is encrypted at rest through the metadata API
================================================================
Extend the metadata API so that users can confirm that their ephemeral storage
is encrypted at rest through the metadata API, accessible from within their
instance.
.. code-block:: json
{
"devices": [
{
"type": "nic",
"bus": "pci",
"address": "0000:00:02.0",
"mac": "00:11:22:33:44:55",
"tags": ["trusted"]
},
{
"type": "disk",
"bus": "virtio",
"address": "0:0",
"serial": "12352423",
"path": "/dev/vda",
"encrypted": "True"
},
{
"type": "disk",
"bus": "ide",
"address": "0:0",
"serial": "disk-vol-2352423",
"path": "/dev/sda",
"tags": ["baz"]
}
]
}
This should also be extended to cover disks provided by encrypted volumes but
this is obviously out of scope for this implementation.
Block resize between flavors with different hw:ephemeral_encryption settings
============================================================================
Ephemeral data is expected to persist through a resize and as such any resize
between flavors that differed in their configuration of ephemeral encryption
(one enabled, another disabled or formats etc) would cause us to convert this
data in place. This isn't trivial and so for this initial implementation
resizing between flavors that differ will be blocked.
Provide a migration path from the legacy implementation
=======================================================
New ``nova-manage`` and ``nova-status`` commands will be introduced to migrate
any instances using the legacy libvirt virt driver implementation ahead of the
removal of this in a future release.
The ``nova-manage`` command will ensure that any existing instances with
``ephemeral_key_uuid`` set will have their associated ``BlockDeviceMapping``
records updated to reference said secret key, the ``plain`` encryption format
and configured options on the host before clearing ``ephemeral_key_uuid``.
Additionally the libvirt virt driver will also attempt to migrate instances
with ``ephemeral_key_uuid`` set during spawn. This should allow at least some
of the instances to be moved during the W release ahead of X.
The ``nova-status`` command will simply report on the existence of any
instances with ``ephemeral_key_uuid`` set that do not have the corresponding
``BlockDeviceMapping`` attributes enabled etc.
Deprecate the now legacy implementation
=======================================
The legacy implementation within the libvirt virt driver will be deprecated for
removal in a future release once the ability to migrate is in place.
Alternatives
============
Continue to use the transparent host configurables and expand support to other
encryption formats such as ``LUKS``.
Data model impact
=================
See above for the various flavor extra spec, image property,
``BlockDeviceMapping`` and ``DriverBlockDevice`` object changes.
REST API impact
===============
* A new API microversion will be created to add encryption options to the
``createImage`` server action API.
* Flavor extra specs and image property validation will be introduced for the
any ephemeral encryption provided options.
* Attempts to resize between flavors that differ in their ephemeral encryption
options will be rejected.
* Attempts to rebuild between images that differ in their ephemeral encryption
options will be allowed by the user who owns the instance. Requests to
rebuild between images that differ in their ephemeral encryption options will
be rejected. This is to prevent a change in the ownership of secrets for the
instance disks.
* The metadata API will be changed to allow users to determine if their
ephemeral storage is encrypted as discussed above.
Security impact
===============
This should hopefully be positive given the unique secret per disk and user
visible choice regarding how their ephemeral storage is encrypted at rest.
Additionally this should allow additional virt drivers to support ephemeral
storage encryption while also allowing the libvirt virt driver to increase
coverage of the feature across more imagebackends such as qcow2 and rbd.
Notifications impact
====================
N/A
Other end user impact
=====================
Users will now need to opt-in to ephemeral storage encryption being used by
their instances through their choice of image or flavor.
Performance Impact
==================
The additional pre-filter will add a small amount of overhead when scheduling
instances but this should fail fast if ephemeral encryption is not requested
through the image or flavor.
The performance impact of increased use of ephemeral storage encryption by
instances is left to be discussed in the virt driver specific specs as this
will vary between hypervisors.
Other deployer impact
=====================
N/A
Developer impact
================
Virt driver developers will be able to indicate support for specific ephemeral
storage encryption formats using the newly introduced compute compatibility
traits.
Upgrade impact
==============
The compute traits should ensure that requests to schedule instances using
ephemeral storage encryption with mixed computes (N-1 and N) will work during a
rolling upgrade.
As discussed earlier in the spec future upgrades will need to provide a path
for existing ephemeral storage encryption users to migrate from the legacy
implementation. This should be trivial but may require an additional grenade
based job in CI during the W cycle to prove out the migration path.
--------------
Implementation
--------------
Assignee(s)
===========
Primary assignee:
melwitt
Other contributors:
lyarwood
Feature Liaison
===============
Feature liaison:
melwitt
Work Items
==========
* Introduce ``hw_ephemeral_encryption*`` image properties and
``hw:ephemeral_encryption`` flavor extra specs.
* Introduce a new ``encrypted``. ``encryption_secret_uuid``,
``backing_encryption_secret_uuid``, ``encryption_format`` and
``encryption_options`` attributes to the BlockDeviceMapping Object.
* Wire up the new ``BlockDeviceMapping`` object attributes through the
``Driver*BlockDevice`` layer and ``block_device_info`` dict.
* Report ephemeral storage encryption through the metadata API.
* Introduce new ``nova-manage`` and ``nova-status`` commands to allow existing
users to migrate to this new implementation. This should however be blocked
outside of testing until a virt driver implementation is landed.
* Validate all of the above in functional tests ahead of any virt driver
implementation landing.
------------
Dependencies
------------
None
-------
Testing
-------
At present without a virt driver implementation this will be tested entirely
within our unit and functional test suites.
Once a virt driver implementation is available additional integration tests in
Tempest and whitebox tests can be written.
Testing of the migration path from the legacy implementation will require an
additional grenade job but this will require the libvirt virt driver
implementation to be completed first.
--------------------
Documentation Impact
--------------------
* The new host configurables, flavor extra specs and image properties should be
documented.
* New user documentation should be written covering the overall use of the
feature from a Nova point of view.
* Reference documentation around `BlockDeviceMapping` objects etc should be
updated to make note of the new encryption attributes.
----------
References
----------
.. _`Glance image encryption`: https://specs.openstack.org/openstack/glance-specs/specs/victoria/approved/glance/image-encryption.html
.. _`Cinder image encryption`: https://specs.openstack.org/openstack/cinder-specs/specs/wallaby/image-encryption.html
.. _`encrypted volume types`: https://docs.openstack.org/cinder/latest/configuration/block-storage/volume-encryption.html#create-an-encrypted-volume-type
.. _`libvirt virt driver`: https://libvirt.org/formatstorageencryption.html#StorageEncryptionLuks
-------
History
-------
.. list-table:: Revisions
:align: left
:header-rows: 1
* - Release Name
- Description
* - Wallaby
- Introduced
* - Xena
- Reproposed
* - Yoga
- Reproposed
* - Zed
- Reproposed
* - 2023.1 Antelope
- Reproposed
* - 2023.2 Bobcat
- Reproposed
* - 2024.1 Caracal
- Reproposed
* - 2024.2 Dalmation
- Reproposed