From 49767beaa92fabf17d546e7345a0a1776f51bd6b Mon Sep 17 00:00:00 2001 From: Stephen Finucane Date: Mon, 22 Feb 2021 18:50:47 +0000 Subject: [PATCH] docs: Add SEV guide This was previously hidden in the hypervisor configuration guide. Make it a top-level document. Change-Id: If402522c859c1413f0d90912e357496a0a67c5cf Signed-off-by: Stephen Finucane --- .../admin/configuration/hypervisor-kvm.rst | 265 ----------------- doc/source/admin/index.rst | 1 + doc/source/admin/sev.rst | 272 ++++++++++++++++++ doc/source/user/flavors.rst | 12 - 4 files changed, 273 insertions(+), 277 deletions(-) create mode 100644 doc/source/admin/sev.rst diff --git a/doc/source/admin/configuration/hypervisor-kvm.rst b/doc/source/admin/configuration/hypervisor-kvm.rst index 2ee73f653251..3e2f94a128a8 100644 --- a/doc/source/admin/configuration/hypervisor-kvm.rst +++ b/doc/source/admin/configuration/hypervisor-kvm.rst @@ -525,271 +525,6 @@ See `the KVM documentation `_ for more information on these limitations. -.. _amd-sev: - -AMD SEV (Secure Encrypted Virtualization) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -`Secure Encrypted Virtualization (SEV)`__ is a technology from AMD which -enables the memory for a VM to be encrypted with a key unique to the VM. -SEV is particularly applicable to cloud computing since it can reduce the -amount of trust VMs need to place in the hypervisor and administrator of -their host system. - -__ https://developer.amd.com/sev/ - -Nova supports SEV from the Train release onwards. - -Requirements for SEV --------------------- - -First the operator will need to ensure the following prerequisites are met: - -- At least one of the Nova compute hosts must be AMD hardware capable - of supporting SEV. It is entirely possible for the compute plane to - be a mix of hardware which can and cannot support SEV, although as - per the section on `Permanent limitations`_ below, the maximum - number of simultaneously running guests with SEV will be limited by - the quantity and quality of SEV-capable hardware available. - -- An appropriately configured software stack on those compute hosts, - so that the various layers are all SEV ready: - - - kernel >= 4.16 - - QEMU >= 2.12 - - libvirt >= 4.5 - - ovmf >= commit 75b7aa9528bd 2018-07-06 - -.. _deploying-sev-capable-infrastructure: - -Deploying SEV-capable infrastructure ------------------------------------- - -In order for users to be able to use SEV, the operator will need to -perform the following steps: - -- Ensure that sufficient memory is reserved on the SEV compute hosts - for host-level services to function correctly at all times. This is - particularly important when hosting SEV-enabled guests, since they - pin pages in RAM, preventing any memory overcommit which may be in - normal operation on other compute hosts. - - It is `recommended`__ to achieve this by configuring an ``rlimit`` at - the ``/machine.slice`` top-level ``cgroup`` on the host, with all VMs - placed inside that. (For extreme detail, see `this discussion on the - spec`__.) - - __ http://specs.openstack.org/openstack/nova-specs/specs/train/approved/amd-sev-libvirt-support.html#memory-reservation-solutions - __ https://review.opendev.org/#/c/641994/2/specs/train/approved/amd-sev-libvirt-support.rst@167 - - An alternative approach is to configure the - :oslo.config:option:`reserved_host_memory_mb` option in the - ``[DEFAULT]`` section of :file:`nova.conf`, based on the expected - maximum number of SEV guests simultaneously running on the host, and - the details provided in `an earlier version of the AMD SEV spec`__ - regarding memory region sizes, which cover how to calculate it - correctly. - - __ https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/amd-sev-libvirt-support.html#proposed-change - - See `the Memory Locking and Accounting section of the AMD SEV spec`__ - and `previous discussion for further details`__. - - __ http://specs.openstack.org/openstack/nova-specs/specs/train/approved/amd-sev-libvirt-support.html#memory-locking-and-accounting - __ https://review.opendev.org/#/c/641994/2/specs/train/approved/amd-sev-libvirt-support.rst@167 - -- A cloud administrator will need to define one or more SEV-enabled - flavors :ref:`as described in the user guide - `, unless it is sufficient for users - to define SEV-enabled images. - -Additionally the cloud operator should consider the following optional -steps: - -.. _num_memory_encrypted_guests: - -- Configure the :oslo.config:option:`libvirt.num_memory_encrypted_guests` - option in :file:`nova.conf` to represent the number of guests an SEV - compute node can host concurrently with memory encrypted at the - hardware level. For example: - - .. code-block:: ini - - [libvirt] - num_memory_encrypted_guests = 15 - - This option exists because on AMD SEV-capable hardware, the memory - controller has a fixed number of slots for holding encryption keys, - one per guest. For example, at the time of writing, earlier - generations of hardware only have 15 slots, thereby limiting the - number of SEV guests which can be run concurrently to 15. Nova - needs to track how many slots are available and used in order to - avoid attempting to exceed that limit in the hardware. - - At the time of writing (September 2019), work is in progress to - allow QEMU and libvirt to expose the number of slots available on - SEV hardware; however until this is finished and released, it will - not be possible for Nova to programmatically detect the correct - value. - - So this configuration option serves as a stop-gap, allowing the - cloud operator the option of providing this value manually. It may - later be demoted to a fallback value for cases where the limit - cannot be detected programmatically, or even removed altogether when - Nova's minimum QEMU version guarantees that it can always be - detected. - - .. note:: - - When deciding whether to use the default of ``None`` or manually - impose a limit, operators should carefully weigh the benefits - vs. the risk. The benefits of using the default are a) immediate - convenience since nothing needs to be done now, and b) convenience - later when upgrading compute hosts to future versions of Nova, - since again nothing will need to be done for the correct limit to - be automatically imposed. However the risk is that until - auto-detection is implemented, users may be able to attempt to - launch guests with encrypted memory on hosts which have already - reached the maximum number of guests simultaneously running with - encrypted memory. This risk may be mitigated by other limitations - which operators can impose, for example if the smallest RAM - footprint of any flavor imposes a maximum number of simultaneously - running guests which is less than or equal to the SEV limit. - -- Configure :oslo.config:option:`libvirt.hw_machine_type` on all - SEV-capable compute hosts to include ``x86_64=q35``, so that all - x86_64 images use the ``q35`` machine type by default. (Currently - Nova defaults to the ``pc`` machine type for the ``x86_64`` - architecture, although `it is expected that this will change in the - future`__.) - - Changing the default from ``pc`` to ``q35`` makes the creation and - configuration of images by users more convenient by removing the - need for the ``hw_machine_type`` property to be set to ``q35`` on - every image for which SEV booting is desired. - - .. caution:: - - Consider carefully whether to set this option. It is - particularly important since a limitation of the implementation - prevents the user from receiving an error message with a helpful - explanation if they try to boot an SEV guest when neither this - configuration option nor the image property are set to select - a ``q35`` machine type. - - On the other hand, setting it to ``q35`` may have other - undesirable side-effects on other images which were expecting to - be booted with ``pc``, so it is suggested to set it on a single - compute node or aggregate, and perform careful testing of typical - images before rolling out the setting to all SEV-capable compute - hosts. - - __ https://bugs.launchpad.net/nova/+bug/1780138 - -Launching SEV instances ------------------------ - -Once an operator has covered the above steps, users can launch SEV -instances either by requesting a flavor for which the operator set the -``hw:mem_encryption`` extra spec to ``True``, or by using an image -with the ``hw_mem_encryption`` property set to ``True``. - -These do not inherently cause a preference for SEV-capable hardware, -but for now SEV is the only way of fulfilling the requirement for -memory encryption. However in the future, support for other -hardware-level guest memory encryption technology such as Intel MKTME -may be added. If a guest specifically needs to be booted using SEV -rather than any other memory encryption technology, it is possible to -ensure this by adding ``trait:HW_CPU_X86_AMD_SEV=required`` to the -flavor extra specs or image properties. - -In all cases, SEV instances can only be booted from images which have -the ``hw_firmware_type`` property set to ``uefi``, and only when the -machine type is set to ``q35``. This can be set per image by setting -the image property ``hw_machine_type=q35``, or per compute node by -the operator via :oslo.config:option:`libvirt.hw_machine_type` as -explained above. - -Impermanent limitations ------------------------ - -The following limitations may be removed in the future as the -hardware, firmware, and various layers of software receive new -features: - -- SEV-encrypted VMs cannot yet be live-migrated or suspended, - therefore they will need to be fully shut down before migrating off - an SEV host, e.g. if maintenance is required on the host. - -- SEV-encrypted VMs cannot contain directly accessible host devices - (PCI passthrough). So for example mdev vGPU support will not - currently work. However technologies based on `vhost-user`__ should - work fine. - - __ https://wiki.qemu.org/Features/VirtioVhostUser - -- The boot disk of SEV-encrypted VMs can only be ``virtio``. - (``virtio-blk`` is typically the default for libvirt disks on x86, - but can also be explicitly set e.g. via the image property - ``hw_disk_bus=virtio``). Valid alternatives for the disk - include using ``hw_disk_bus=scsi`` with - ``hw_scsi_model=virtio-scsi`` , or ``hw_disk_bus=sata``. - -- QEMU and libvirt cannot yet expose the number of slots available for - encrypted guests in the memory controller on SEV hardware. Until - this is implemented, it is not possible for Nova to programmatically - detect the correct value. As a short-term workaround, operators can - optionally manually specify the upper limit of SEV guests for each - compute host, via the new - :oslo.config:option:`libvirt.num_memory_encrypted_guests` - configuration option :ref:`described above - `. - -Permanent limitations ---------------------- - -The following limitations are expected long-term: - -- The number of SEV guests allowed to run concurrently will always be - limited. `On the first generation of EPYC machines it will be - limited to 15 guests`__; however this limit becomes much higher with - the second generation (Rome). - - __ https://www.redhat.com/archives/libvir-list/2019-January/msg00652.html - -- The operating system running in an encrypted virtual machine must - contain SEV support. - -Non-limitations ---------------- - -For the sake of eliminating any doubt, the following actions are *not* -expected to be limited when SEV encryption is used: - -- Cold migration or shelve, since they power off the VM before the - operation at which point there is no encrypted memory (although this - could change since there is work underway to add support for `PMEM - `_) - -- Snapshot, since it only snapshots the disk - -- ``nova evacuate`` (despite the name, more akin to resurrection than - evacuation), since this is only initiated when the VM is no longer - running - -- Attaching any volumes, as long as they do not require attaching via - an IDE bus - -- Use of spice / VNC / serial / RDP consoles - -- `VM guest virtual NUMA (a.k.a. vNUMA) - `_ - -For further technical details, see `the nova spec for SEV support`__. - -__ http://specs.openstack.org/openstack/nova-specs/specs/train/approved/amd-sev-libvirt-support.html - - Guest agent support ~~~~~~~~~~~~~~~~~~~ diff --git a/doc/source/admin/index.rst b/doc/source/admin/index.rst index c5c3c296f1db..39f5f0e87838 100644 --- a/doc/source/admin/index.rst +++ b/doc/source/admin/index.rst @@ -116,6 +116,7 @@ instance for these kind of workloads. emulated-tpm uefi secure-boot + sev managing-resource-providers resource-limits diff --git a/doc/source/admin/sev.rst b/doc/source/admin/sev.rst new file mode 100644 index 000000000000..1737b66830c3 --- /dev/null +++ b/doc/source/admin/sev.rst @@ -0,0 +1,272 @@ +.. _amd-sev: + +AMD SEV (Secure Encrypted Virtualization) +========================================= + +.. versionadded:: 20.0.0 (Train) + +`Secure Encrypted Virtualization (SEV)`__ is a technology from AMD which +enables the memory for a VM to be encrypted with a key unique to the VM. +SEV is particularly applicable to cloud computing since it can reduce the +amount of trust VMs need to place in the hypervisor and administrator of +their host system. + +.. __: https://developer.amd.com/sev/ + + +.. _deploying-sev-capable-infrastructure: + +Enabling SEV +------------ + +First the operator will need to ensure the following prerequisites are met: + +- Currently SEV is only supported when using the libvirt compute driver with a + :oslo.config:option:`libvirt.virt_type` of ``kvm`` or ``qemu``. + +- At least one of the Nova compute hosts must be AMD hardware capable + of supporting SEV. It is entirely possible for the compute plane to + be a mix of hardware which can and cannot support SEV, although as + per the section on `Permanent limitations`_ below, the maximum + number of simultaneously running guests with SEV will be limited by + the quantity and quality of SEV-capable hardware available. + +In order for users to be able to use SEV, the operator will need to +perform the following steps: + +- Ensure that sufficient memory is reserved on the SEV compute hosts + for host-level services to function correctly at all times. This is + particularly important when hosting SEV-enabled guests, since they + pin pages in RAM, preventing any memory overcommit which may be in + normal operation on other compute hosts. + + It is `recommended`__ to achieve this by configuring an ``rlimit`` at + the ``/machine.slice`` top-level ``cgroup`` on the host, with all VMs + placed inside that. (For extreme detail, see `this discussion on the + spec`__.) + + __ http://specs.openstack.org/openstack/nova-specs/specs/train/approved/amd-sev-libvirt-support.html#memory-reservation-solutions + __ https://review.opendev.org/#/c/641994/2/specs/train/approved/amd-sev-libvirt-support.rst@167 + + An alternative approach is to configure the + :oslo.config:option:`reserved_host_memory_mb` option in the + ``[DEFAULT]`` section of :file:`nova.conf`, based on the expected + maximum number of SEV guests simultaneously running on the host, and + the details provided in `an earlier version of the AMD SEV spec`__ + regarding memory region sizes, which cover how to calculate it + correctly. + + __ https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/amd-sev-libvirt-support.html#proposed-change + + See `the Memory Locking and Accounting section of the AMD SEV spec`__ + and `previous discussion for further details`__. + + __ http://specs.openstack.org/openstack/nova-specs/specs/train/approved/amd-sev-libvirt-support.html#memory-locking-and-accounting + __ https://review.opendev.org/#/c/641994/2/specs/train/approved/amd-sev-libvirt-support.rst@167 + +- A cloud administrator will need to define one or more SEV-enabled + flavors :ref:`as described below `, unless it + is sufficient for users to define SEV-enabled images. + +Additionally the cloud operator should consider the following optional +steps: + +.. _num_memory_encrypted_guests: + +- Configure the :oslo.config:option:`libvirt.num_memory_encrypted_guests` + option in :file:`nova.conf` to represent the number of guests an SEV + compute node can host concurrently with memory encrypted at the + hardware level. For example: + + .. code-block:: ini + + [libvirt] + num_memory_encrypted_guests = 15 + + This option exists because on AMD SEV-capable hardware, the memory + controller has a fixed number of slots for holding encryption keys, + one per guest. For example, at the time of writing, earlier + generations of hardware only have 15 slots, thereby limiting the + number of SEV guests which can be run concurrently to 15. Nova + needs to track how many slots are available and used in order to + avoid attempting to exceed that limit in the hardware. + + At the time of writing (September 2019), work is in progress to + allow QEMU and libvirt to expose the number of slots available on + SEV hardware; however until this is finished and released, it will + not be possible for Nova to programmatically detect the correct + value. + + So this configuration option serves as a stop-gap, allowing the + cloud operator the option of providing this value manually. It may + later be demoted to a fallback value for cases where the limit + cannot be detected programmatically, or even removed altogether when + Nova's minimum QEMU version guarantees that it can always be + detected. + + .. note:: + + When deciding whether to use the default of ``None`` or manually + impose a limit, operators should carefully weigh the benefits + vs. the risk. The benefits of using the default are a) immediate + convenience since nothing needs to be done now, and b) convenience + later when upgrading compute hosts to future versions of Nova, + since again nothing will need to be done for the correct limit to + be automatically imposed. However the risk is that until + auto-detection is implemented, users may be able to attempt to + launch guests with encrypted memory on hosts which have already + reached the maximum number of guests simultaneously running with + encrypted memory. This risk may be mitigated by other limitations + which operators can impose, for example if the smallest RAM + footprint of any flavor imposes a maximum number of simultaneously + running guests which is less than or equal to the SEV limit. + +- Configure :oslo.config:option:`libvirt.hw_machine_type` on all + SEV-capable compute hosts to include ``x86_64=q35``, so that all + x86_64 images use the ``q35`` machine type by default. (Currently + Nova defaults to the ``pc`` machine type for the ``x86_64`` + architecture, although `it is expected that this will change in the + future`__.) + + Changing the default from ``pc`` to ``q35`` makes the creation and + configuration of images by users more convenient by removing the + need for the ``hw_machine_type`` property to be set to ``q35`` on + every image for which SEV booting is desired. + + .. caution:: + + Consider carefully whether to set this option. It is + particularly important since a limitation of the implementation + prevents the user from receiving an error message with a helpful + explanation if they try to boot an SEV guest when neither this + configuration option nor the image property are set to select + a ``q35`` machine type. + + On the other hand, setting it to ``q35`` may have other + undesirable side-effects on other images which were expecting to + be booted with ``pc``, so it is suggested to set it on a single + compute node or aggregate, and perform careful testing of typical + images before rolling out the setting to all SEV-capable compute + hosts. + + __ https://bugs.launchpad.net/nova/+bug/1780138 + + +.. _extra-specs-memory-encryption: + +Configuring a flavor or image +----------------------------- + +Once an operator has covered the above steps, users can launch SEV +instances either by requesting a flavor for which the operator set the +:nova:extra-spec:`hw:mem_encryption` extra spec to ``True``, or by using an +image with the ``hw_mem_encryption`` property set to ``True``. For example, to +enable SEV for a flavor: + +.. code-block:: console + + $ openstack flavor set FLAVOR-NAME \ + --property hw:mem_encryption=true + +These do not inherently cause a preference for SEV-capable hardware, +but for now SEV is the only way of fulfilling the requirement for +memory encryption. However in the future, support for other +hardware-level guest memory encryption technology such as Intel MKTME +may be added. If a guest specifically needs to be booted using SEV +rather than any other memory encryption technology, it is possible to +ensure this by setting the :nova:extra-spec:`trait{group}:HW_CPU_X86_AMD_SEV` +extra spec or equivalent image metadata property to ``required``. + +In all cases, SEV instances can only be booted from images which have +the ``hw_firmware_type`` property set to ``uefi``, and only when the +machine type is set to ``q35``. This can be set per image by setting +the image property ``hw_machine_type=q35``, or per compute node by +the operator via :oslo.config:option:`libvirt.hw_machine_type` as +explained above. + + +Limitations +----------- + +Impermanent limitations +~~~~~~~~~~~~~~~~~~~~~~~ + +The following limitations may be removed in the future as the +hardware, firmware, and various layers of software receive new +features: + +- SEV-encrypted VMs cannot yet be live-migrated or suspended, + therefore they will need to be fully shut down before migrating off + an SEV host, e.g. if maintenance is required on the host. + +- SEV-encrypted VMs cannot contain directly accessible host devices + (PCI passthrough). So for example mdev vGPU support will not + currently work. However technologies based on `vhost-user`__ should + work fine. + + __ https://wiki.qemu.org/Features/VirtioVhostUser + +- The boot disk of SEV-encrypted VMs can only be ``virtio``. + (``virtio-blk`` is typically the default for libvirt disks on x86, + but can also be explicitly set e.g. via the image property + ``hw_disk_bus=virtio``). Valid alternatives for the disk + include using ``hw_disk_bus=scsi`` with + ``hw_scsi_model=virtio-scsi`` , or ``hw_disk_bus=sata``. + +- QEMU and libvirt cannot yet expose the number of slots available for + encrypted guests in the memory controller on SEV hardware. Until + this is implemented, it is not possible for Nova to programmatically + detect the correct value. As a short-term workaround, operators can + optionally manually specify the upper limit of SEV guests for each + compute host, via the new + :oslo.config:option:`libvirt.num_memory_encrypted_guests` + configuration option :ref:`described above + `. + +Permanent limitations +~~~~~~~~~~~~~~~~~~~~~ + +The following limitations are expected long-term: + +- The number of SEV guests allowed to run concurrently will always be + limited. `On the first generation of EPYC machines it will be + limited to 15 guests`__; however this limit becomes much higher with + the second generation (Rome). + + __ https://www.redhat.com/archives/libvir-list/2019-January/msg00652.html + +- The operating system running in an encrypted virtual machine must + contain SEV support. + +Non-limitations +~~~~~~~~~~~~~~~ + +For the sake of eliminating any doubt, the following actions are *not* +expected to be limited when SEV encryption is used: + +- Cold migration or shelve, since they power off the VM before the + operation at which point there is no encrypted memory (although this + could change since there is work underway to add support for `PMEM + `_) + +- Snapshot, since it only snapshots the disk + +- ``nova evacuate`` (despite the name, more akin to resurrection than + evacuation), since this is only initiated when the VM is no longer + running + +- Attaching any volumes, as long as they do not require attaching via + an IDE bus + +- Use of spice / VNC / serial / RDP consoles + +- `VM guest virtual NUMA (a.k.a. vNUMA) + `_ + + +References +---------- + +- `libvirt driver launching AMD SEV-encrypted instances (spec)`__ + +.. __: http://specs.openstack.org/openstack/nova-specs/specs/train/approved/amd-sev-libvirt-support.html diff --git a/doc/source/user/flavors.rst b/doc/source/user/flavors.rst index 5d0117aaa143..83e322861f50 100644 --- a/doc/source/user/flavors.rst +++ b/doc/source/user/flavors.rst @@ -184,18 +184,6 @@ Performance Monitoring Unit (vPMU) required, such workloads should set ``hw:pmu=False``. For most workloads the default of unset or enabling the vPMU ``hw:pmu=True`` will be correct. -.. _extra-specs-memory-encryption: - -Hardware encryption of guest memory - If there are compute hosts which support encryption of guest memory - at the hardware level, this functionality can be requested via the - ``hw:mem_encryption`` extra spec parameter: - - .. code-block:: console - - $ openstack flavor set FLAVOR-NAME \ - --property hw:mem_encryption=True - .. _extra-specs-hiding-hypervisor-signature: Hiding hypervisor signature