diff --git a/specs/rocky/approved/libvirt-file-backed-memory.rst b/specs/rocky/approved/libvirt-file-backed-memory.rst new file mode 100644 index 000000000..9ec93d777 --- /dev/null +++ b/specs/rocky/approved/libvirt-file-backed-memory.rst @@ -0,0 +1,220 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +========================================== +Libvirt file backed memory +========================================== + +Spec for blueprint libvirt-file-backed-memory +https://blueprints.launchpad.net/nova/+spec/libvirt-file-backed-memory + +With the advent of large capacity memory devices, it's now reasonable to run +virtual machines with file backed memory. This enables a much larger total +memory area per compute node + +Problem description +=================== + +New memory technology and modern NVMe SSDs have progressed to the point where +it is reasonable to utilize these devices as a backing store for VM memory to +expand the memory capacity of a compute host. + +Use Cases +--------- + +As an operator, I want to be able to leverage libvirt's support for +file-backed memory technologies to expand my compute node memory capacity. + +Proposed change +=============== + +The proposed change is to add a new option in ``nova.conf`` under the +``libvirt`` section to configure file backed memory: + +* ``file_backed_memory`` + +``file_backed_memory`` will default to ``0``, indicating file backed memory is +disabled. When set to non-zero, the libvirt driver will include elements +to enable file backed memory within instances. + +When ``file_backed_memory`` is set to non-zero, the libvirt driver will +include a ``MemoryBacking`` element for all instances on the compute node, +with a ``source`` subelement with type ``file``, and an ``access`` subelement +with mode ``shared``. + +The value configured in ``file_backed_memory`` will be reported by the libvirt +driver as the total memory capacity of the compute node in MiB. As the memory +capacity is dependent on the backing store(s) in use, libvirt must report a +value other than the real system memory capacity. Available capacity will then +be calculated from the value in ``file_backed_memory`` minus the currently +used memory for instances. System memory will be used as a cache for the file +backed memory through the kernel pagecache. + +As overcommit is not expected to work with file backed memory, enabling this +option requires the value of ``ram_allocation_ratio`` in ``nova.conf``, to be +set to the value ``1.0``, and will block Nova startup if this is not +configured properly. + +.. note:: ``shared`` access mode is required, as ``private`` access will not + utilize an underlying backing store for pages in-use by the + instance, but will keep those pages within main system memory. + +.. note:: During migration, the destination compute node must be checked for + the ``file_backed_memory`` option, and add or remove the + MemoryBacking element and subelements as appropriate, to + ensure memory is appropriately allocated during the migration + process. This can be in nova/virt/libvirt/migration.py, within the + get_updated_guest_xml method, similar to how graphics, serial, + volume, and perf events are handled today. + +.. note:: Migration from compute nodes running versions of Nova without this + feature will not include the appropriate libvirt XML for + file backed memory. Nova will block these migrations, to ensure + all instances migrated to a node with ``file_backed_memory`` enabled + are actually using file backed memory. Migrations will be blocked + within ``check_can_live_migrate_destination`` + +Alternatives +------------ + +In Nova, the available memory capacity could be detected dynamically. Due to +the variety of memory and SSD technologies and ways the memory backing +directory could be configured within libvirt and on the system, this would +require implementing an external call point to determine the available +capacity, or require vendor-specific code included in Nova. Either option +would significantly increase the scope of this spec. + +In Nova, file backed memory could be enabled via a flavor extra-spec, allowing +for control per individual instance. This results in an inconsistent use of RAM +and backing devices, leading to a confusing / conflicting memory capacity +calculation on the compute node. + +Not implement this spec at all. In this case, a compute node is limited to +the capacity of the standard DRAM in the compute node. + +Data model impact +----------------- + +None + +REST API impact +--------------- + +None + +Security impact +--------------- + +As the instance memory will now be contained within a file on a filesystem, +instance memory will be accessible to any process owned by a user with +permissions necessary to access the files. Currently, the root user on a +compute node already has capability to read system memory and VM memory. + +Notifications impact +-------------------- + +None + +Other end user impact +--------------------- + +None + +Performance Impact +------------------ + +When the ``file_backed_memory`` option is enabled, instance memory performance +will be dependent on the backing store, as configured by the operator. + +Other deployer impact +--------------------- + +New config options would need to be explicitly enabled to take effect. + +Prior to enabling new config options, an operator should configure the libvirt +``memory_backing_dir`` configuration setting to point to their selected backing +store, such as a filesystem on an NVMe device. + +Enabling ``file_backed_memory`` will reject migrations from compute nodes +running versions of Nova that do not support file backed memory. + +Developer impact +---------------- + +None + +Upgrade impact +-------------- + +Enabling ``file_backed_memory`` will reject migrations from compute nodes +running versions of Nova that do not support file backed memory. + +It's recommended to only enable ``file_backed_memory`` after all compute nodes +are upgraded. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + Zack Cornelius + +Other contributors: + None + +Work Items +---------- + +* Add new configuration options to nova.conf +* Check for configuration options within libvirt driver and report the + file-backed capacity value instead of system memory +* Generate additional libvirt domain XML as needed +* Validate / correct libvirt domain XML during migration process + +Dependencies +============ + +- Qemu >= 2.6.0 +- Libvirt >= 4.0.0 + +Testing +======= + +Unit tests will be added to validate the instances booted on host have files +in the libvirt ``memory_backing_dir`` on the host. + +A test will be needed for the edge cases of migrating between a host with the +new options enabled, and a host without the new options enabled, to ensure the +memory is allocated from the correct source. + +We will investigate adding an integration test for this by creating a large +ramdisk, enabling these settings, and validating that an instance is utilizing +memory within files on that ramdisk. We believe this test layout should be +possible, but if not, will fall back to relying on unit tests. + +Documentation Impact +==================== + +The documentation for ``nova.conf`` should be updated with the new +configuration options. + +References +========== + +* https://libvirt.org/formatdomain.html#elementsMemoryBacking + +History +======= + +.. list-table:: Revisions + :header-rows: 1 + + * - Release Name + - Description + * - Rocky + - Introduced