diff --git a/specs/2024.1/approved/assisted-volume-extend.rst b/specs/2024.1/approved/assisted-volume-extend.rst new file mode 100644 index 000000000..a5099198c --- /dev/null +++ b/specs/2024.1/approved/assisted-volume-extend.rst @@ -0,0 +1,297 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +=================================== +Use extend volume completion action +=================================== + +https://blueprints.launchpad.net/nova/+spec/assisted-volume-extend + +This blueprint proposes to use the ``os-extend_volume_completion`` volume +action that has been proposed for Cinder in [3]_, to provide feedback on +success or failure when handling ``volume-extended`` external server events. + +Problem description +=================== + +Many remotefs-based volume drivers in Cinder use the ``qemu-img resize`` +command to extend volume files. +However, when the volume is attached to a guest, QEMU will lock the file and +``qemu-img`` will be unable to resize it. + +In this case, only the QEMU process holding the lock can resize the volume, +which can be triggered through the QEMU monitor command ``block-resize``. + +There is currently no adequate way for Cinder to use this feature, so the NFS, +NetApp NFS, Powerstore NFS, and Quobyte volume drivers all disable extending +attached volumes. + +Use Cases +--------- + +As a user, I want to extend a NFS/NetApp NFS/Powerstore NFS/Quobyte volume +while it is attached to an instance and I want the volume size and status to +reflect the success or failure of the operation. + +Proposed change +=============== + +Nova's libvirt driver uses the ``block-resize`` command when handling the +``volume-extended`` external server event, to inform QEMU that the size of an +attached volume has changed. +It is in principle also capable of extending a volume file, but is currently +unable to provide feedback to Cinder on the success of the operation. + +Currently, Cinder will send the ``volume-extended`` external server event to +Nova only after it has finalized the extend operation and reset the volume +status from ``extending`` back to ``in-use``. + +With [3]_, Cinder will allow volume drivers to hold off finalizing the extend +operation and leave the volume status as ``extending``, until after it has +send the ``volume-extended`` event and received feedback from Nova in form of +the ``os-extend_volume_completion`` volume action, with an ``error`` argument +indicating whether to finalize or to roll back the operation. + +This will currently affect only the volume drivers mentioned above, all of +which did not previously support online extend. +All other drivers will continue to send the ``volume-extended`` event after +finalizing the operation and resetting to ``in-use`` status, and will not +expect a ``os-extend_volume_completion`` volume action. + +Compute Agent +------------- + +Nova's compute agent will use the volume status to differentiate between the +two behaviors when handling ``volume-extended`` events: + +* If the volume status is ``extending``, then it will attempt to read + ``extend_new_size`` from the volume's metadata and use this value as the + new size of the volume, instead of the volume size field. + + After successfully extending the volume, it will call the extend volume + completion action of the volume, with ``"error": false``. + + If anything goes wrong, including ``extend_new_size`` being missing from the + metadata, or being smaller than the current size of the volume, it will + log the error and call the ``os-extend_volume_completion`` action with + ``"error": true``, so Cinder can roll back the operation. + +* For any other volume status, including ``in-use``, the event will be handled + as before. + +API +--- + +Nova's API will introduce a new microversion, so that Cinder can make sure the +new behavior is available, before leaving an extend operation unfinished. + +To handle older compute agents during a rolling upgrade, the API will also +check the compute service version of the target agent when receiving a +``volume-extended`` event with the new microversion. +If a target compute agent is too old to support the feature, the API will +discard the event and call the ``os-extend_volume_completion`` action with +``"error": true``. + +Alternatives +------------ + +* A previous change tried to use the ``volume-extended`` external server event + to support online extend for the NFS driver [1]_, but did not rely on + feedback from Nova to Cinder at all. + Instead, it would just set the new size of the volume, change the status + back to ``in-use``, notify Nova, and hope for the best. + + If anything went wrong on Nova's side, this would still result in a volume + state indicating that the operation was successful, which is not acceptable. + +* A previous version of this spec proposed a new synchronous API in Nova [2]_, + that would directly call ``CompVirtAPI.extend_image`` of the nova-compute + instance managing the guest that a volume was attached to. + This API would provide a single mechanism to trigger the resize operation, + communicate the new size to Nova, and get feedback on the success of the + operation. + + The problem with a synchronous API is, that RPC and API timeouts limit the + maximum time an extend operation can take. + For QEMU, this seemed to be acceptable, because storage preallocation is + hard disabled for the ``block-resize`` command, and because all currently + plausible file systems support sparse file operations. + + However, this may not be true for other volume or virt drivers that might + require this API in the future. + It would also break with the established pattern of asynchronous + coordination between Nova and Cinder, which includes the assisted snapshot + and volume migration features. + +* Following this pattern, we could make the proposed API asynchronous and use + a new callback in Cinder, similar to Nova's ``os-assisted-volume-snapshots`` + API, which uses the ``os-update_snapshot_status`` snapshot action to provide + feedback to Cinder. + + The function of the new Nova API would then just be to trigger the operation + and to communicate the new size. + The question is then, whether that warrants adding a new API to Nova, since + there are existing mechanisms that could be used for either. + +* The existing mechanism for triggering the extend operation in Nova is of + course the ``volume-extended`` external server event. + Using it for this purpose, as this spec proposes, requires the target size + to be transferred separately, because external server events only have a + single text field that is freely usable, which for ``volume-extended`` + is already used for the volume ID. + + Besides storing it in the admin metadata, as [3]_ and this spec propose, + there is also the option of updating the size field of the volume, as [1]_ + was essentially doing. + + This would require the volume size field to be reset on a failure. + If an error response from Nova was lost, the volume would just keep the new + size. + We would need to extend ``os-reset_status`` to allow a size reset, or + something similar to clean up volumes like this. + This would be possible, but updating the size field only after the volume + was successfully extended seems like a cleaner solution. + +* We could also extend the external server event API to accept additional data + for events, and use this to communicate the new size to Nova. + + This option was judged favorably by reviewers on the previous version of + this spec, [2]_, but it would be a more complex change to the Nova API. + + However, if additional data fields become available in a future version of + the external server event API, it would be a relatively minor change to use + this instead of volume metadata. + +Data model impact +----------------- + +None + +REST API impact +--------------- + +The behavior of the external server event API will change. + +* If Nova receives a ``volume-extended`` event, and the referenced volume has + status of ``extending``, Nova will look for the ``extend_new_size`` key in + the volume metadata, and use this instead of the volume size field as the + target size to update the block device mapping and to pass to the virt + driver's ``extend_volume`` method. + + Nova will also attempt to call Cinder's new ``os-extend_volume_completion`` + volume action proposed in [3]_ to let Cinder know if the operation was + successful or not. + +* Otherwise, the API will behave as before. + +Security impact +--------------- + +None + +Notifications impact +-------------------- + +None + +Other end user impact +--------------------- + +None + +Performance Impact +------------------ + +None + +Other deployer impact +--------------------- + +None + +Developer impact +---------------- + +None + +Upgrade impact +-------------- + +Checking the target compute service version allows the API to handle rolling +upgrades gracefully. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + kgube + +Other contributors: + None + +Feature Liaison +--------------- + +Feature liaison: + None yet + +Work Items +---------- + +* Update the external server event API to check the target compute service + version for ``volume-extended`` events. +* Update the ``ComputeVirtAPI.extend_volume`` method to follow the behavior + outlined in `Compute Agent`_. +* Add unit tests. +* Adapt NFS job in the Nova gate to validate online extend. + +Dependencies +============ + +* The extend volume completion action [3]_ + +Testing +======= + +We should test that the ``os-extend_volume_completion`` gets called correctly +in all possible error or success condition if a volume has ``extending`` +status. + +We should test the case that the call to ``os-extend_volume_completion`` fails. + +We also need to test that ``volume-extended`` continues to be handled correctly +for volumes not in ``extending`` status. + +Documentation Impact +==================== + +The new behavior of the ``volume-extended`` event should be added to the +documentation of the external server event API. + +References +========== + +.. [1] https://review.opendev.org/c/openstack/cinder/+/739079 +.. [2] https://review.opendev.org/c/openstack/nova-specs/+/855490/6 +.. [3] https://review.opendev.org/c/openstack/cinder-specs/+/877230 + +History +======= + +.. list-table:: Revisions + :header-rows: 1 + + * - Release Name + - Description + * - 2023.1 Antelope + - Accepted + * - 2023.2 Bobcat + - Accepted + * - 2024.1 Caracal + - Reproposed