From d9116eaf42d1108f2ff54d8b08251afc0b5d8683 Mon Sep 17 00:00:00 2001 From: Matt Riedemann Date: Mon, 12 Feb 2018 13:39:57 -0500 Subject: [PATCH] Re-propose use Neutron's new port binding API This was approved for Queens in change: Ie776601d5966d14fea55d958c7407dd6b710df38 Updates here are assignee (I plan on taking over this work), updating a s/queens/rocky/ part in the description, and adding the upgrade impact section. Previously-approved: Queens Spec for blueprint neutron-new-port-binding-api Change-Id: Ibd1728b5cb094f312c241c5854fd95aa6a218031 --- .../approved/neutron-new-port-binding-api.rst | 393 ++++++++++++++++++ 1 file changed, 393 insertions(+) create mode 100644 specs/rocky/approved/neutron-new-port-binding-api.rst diff --git a/specs/rocky/approved/neutron-new-port-binding-api.rst b/specs/rocky/approved/neutron-new-port-binding-api.rst new file mode 100644 index 000000000..90d0b01fd --- /dev/null +++ b/specs/rocky/approved/neutron-new-port-binding-api.rst @@ -0,0 +1,393 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +=================================== +Use Neutron's new port binding API +=================================== + +Make use of Neutron's new port binding API in all cases where port binding +occurs. In the special case of move operations, the new API will allow us to +model both source and destination hosts having a port binding +which is not accounted for during live migration today. + +https://blueprints.launchpad.net/nova/+spec/neutron-new-port-binding-api + +Problem description +=================== + +The main motivation for the change is a selection of problems around +live-migration, in particular: + +* If port binding would fail at destination, we would know that before + starting the live-migration. +* In many cases the two hosts need a very different binding on source host + vs destination host, but that's not possible today. In particular, macvtap + live migration often needs different libvirt XML on source vs destination. +* In Mitaka, Neutron introduced a connection tracker based security group + driver for ovs based backends. Live migrating between a host with a + different security group driver is another motivating special case of + the more general act of live migrating between different ml2 drivers which + both require different libvirt XML definitions on the source and + destination hosts. +* To make the switch between source and destination host as quickly as + possible, it's good to get most things ready before the migration is started. + We have added a short-term hack https://review.openstack.org/#/c/275073/ + for DVR, but let's do it properly. + +More details can be found in the neutron spec: +https://specs.openstack.org/openstack/neutron-specs/specs/pike/portbinding_information_for_nova.html + +When thinking about this spec, we should be clear on the difference between: + +* port binding, the DB record in Neutron of what host and instance a port is + associated with, and its current state. +* plug/unplug VIFs, where information from Neutron is passed to OS-VIF (or + a legacy driver) to get the port ready on the host +* attaching the instances to the above preparations (via a tap device or + PCI passthrough, etc) +* an active binding, the host where the traffic for associated port should + go to, because that's where the VM is actually running + +To address this problem statement we need to consider all the places where +we deal with port bindings, to use the new API flow. +We generally update the port bindings when a VM is moved. +The main API actions to consider are: + +* Attach a port to an instance, including during spawning an instance +* Detach a port from an instance +* Live-migrate an instance, involves setting up the VIF on the destination + host, before kicking off the live-migrate, then removing VIFs on source + host once the live-migrate has completed. +* Migrate and resize are very similar to live-migrate +* Evacuate, we know the old host is dead, we want to kill any record of that + old connection, and attach the port on the new host. +* Shelve, we want the port to stay logically attached to the instance, but + we need to unbind the port for the host when the instance is offloaded. + +In the move operations above neutron should be notified when the traffic needs +to be switched to the destination by activating the new port binding. + +Use Cases +--------- + +As an admin, I want live migration to fail early during pre_live_migration +if it can be detected that network setup on the destination host +will not work. + +As a user, I want minimal network downtime while my instance is +being live migrated. + +As an operator, I want to start using new ML2 backends but need +to live migrate existing instances off of a different backend +with minimal network downtime. + +As an operator, I want to leverage new security group implementations +and need to be able to live migrate existing instances from my +legacy hosts. + +Proposed change +=============== + +There is no additional configuration for deployers. +The use of multiple bindings will be enabled automatically. +We decide whether to use the new or old API flow, if both compute nodes +support this feature and based on the available Neutron API extensions. +We cache extensions support in the usual way utilizing the existing +neutron_extensions_cache. + +Note: The new neutron API extension will be implemented in the ml2 plugin +layer, above the ml2 driver layer so if the extension is exposed it will be +supported for all ml2 drivers. Monolithic plugins will have to implement +the extension separately and will continue to use the old workflow until +their maintainers support the new neutron extension. + +The old interaction model should be removed when sufficient time has elapsed +for all neutron backends to support this model with an aim of completing this +in two cycles. We should keep this in mind in how the code is structured. +Given we are also looking at removing nova-network, we should see if this can +be added as a new set of network API calls, that are only for neutron, making +the existing calls needed for nova-network be no-ops for Neutron. + +Migration across mixed software versions +---------------------------------------- + +If an old neutron is present then the existing workflow will be followed +regardless of the compute node version. Where a new neutron is deployed +that supports this feature the following behaviour will be implemented. + +* If both compute nodes are rocky or newer. In this case the new workflow + will be used as described below. + +* In all other cases the old workflow is used + +Let's consider live-migration and what the calls to neutron will look like. + +Today the workflow is documented here: +https://docs.openstack.org/nova/latest/reference/live-migration.html + +Going forward the workflow will be altered as follows: + +* Conductor does its pre-checks on the dest and source which + creates the migrate_data object. + +* Conductor checks the source and dest version to see if + they support the new flow. + + * If new enough, conductor calls a new method on the dest + compute to bind the port on the destination host. + The new method will POST to /v2.0/ports/{port_id}/bindings passing + just the destination host_id:: + + { + "binding": { + "host_id": "target-host_id" + } + } + + * If this fails, the live migrate task can ignore that host and + reschedule / retry another host because it's before we've cast + to the source to start the live migration on the hypervisor. + + * If this succeeds, the port binding results are put into the + migrate_data object which is sent to the source live_migration + method, and after that all code that cares checks the new + attribute in the migrate_data object. + + * The new attribute will consist of the minimal subset of the port + binding response and will be encoded in a new nova object:: + + fields = { + 'port_id': fields.StringField(), + 'host_id': fields.StringField(), + 'vnic_type': fields.StringField(), # could be enum + 'vif_type': fields.StringField(), + 'vif_details': fields.DictOfStringsField(), + } + + During implementation we will try to restrict the ``vif_details`` + field to the subset of vif_details required by nova to generate + the updated domain xml and plug the vif. This is to avoid random + ML2 backend-specific data from changing behavior in our versioned + object. In the future this object will be replaced by one defined + by os-vif. + +* In pre_live_migration on destination: + + * Prior to the RPC call from live_migration on the source host to + pre_live_migration on the dest host, start a wait thread for the + vif-plugged event from Neutron, similar to during initial spawn. + + .. note:: This vif-plugged wait change can be made irrespective of this + blueprint - it could be done as a bug fix or hardening opportunity. + + * Check if migrate_data contains new VIFs attribute, if so, + plug vif on destination host using the new port bindings, + else fall back to old workflow and plug vif with old vif bindings. + +* At this point it is safe to start live migrating the instance. + + * This involves calling the virt driver to live migrate + the instance and then activating the port binding. If migrate_data + contains the new dest host port binding VIFs attribute, it will + be used to configure the dest guest prior to starting the actual + live migration in the hypervisor. This is in case the VIF type on + the dest host is different from the source host. + + * In the example of the libvirt virt driver, we will wait for a qemu event + on the source host called VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY, + so we know the VM has just been paused by libvirt and mark the new + port binding as active. This is described in more detail here: + https://review.openstack.org/#/c/434870/ + + * For other virt drivers the decision of when to activate the port + binding is left to them. They may serialise the calls by activating + the port binding immediately before or after migrating the instance + or they may concurrently wait for an event if the hypervisor allows + them to reduce the network downtime, or just activate the dest host + port binding in post_live_migration. + +* We should only hit an error here if the migration times out. + If we hit any other error, there is no rollback and we just + put the instance into the ERROR state. If we timeout we abort + as described below. + +* During post_live_migration: + + After cleaning up VIFs on the source host, we remove the old port binding + associated with the source host. Should the operation get interrupted, + there is enough information in the binding to ensure manual + cleanup is feasible. + +Aborts +------ + +* If the admin aborts an in-progress live migration, the rollback actions vary + depending on what phase of the migration we are currently in. + +* If we are in the pre_live_migration phase and have not started the migration + we simply delete the destination port binding. + +* If we have started the VM on the remote node and plugged the interface but + not unpaused the instance, we unplug the instance, activate the source + binding if required and delete the destination binding. + +Other +----- + +We can follow this pattern wherever there are VIFs present on two hosts, such +as during resize and migrate. + +Evacuate is a special case, where we delete the port binding on the old host, +without knowing if it has had VIFs deleted, as we assume the host is dead and +will never be coming back to life. + +With this change, live migration between hosts with different +neutron backends and/or security group drivers should be possible. +While not explicitly described in this spec the implementation of this +feature should not block that effort or the efforts to adopt oslo versioned +objects for nova / neutron portbinding negotiation, however, it is also not +dependent on either activity to be completed. + +Alternatives +------------ + +We could leave live-migration broken for some Neutron drivers. + +Note: there are additional plans to allow live-migrate to be used to switch +between different Neutron plugins, and allowing live-migrate for macvtap +attached SR-IOV, but this is not in scope for this change. + +We could support live migration between mixed compute nodes. +In this case assuming neutron supported the new flow, the +following behaviour would be introduced. + +* old source compute node and a new destination. Taking libvirt as an example, + as the migration XML generation is done by the source node if the new + destination compute node detects that an XML change would be required it + should fail the migration. This changes existing behaviour where + live migration may complete successfully but result in no network + connectivity. + +* new source compute node and an old destination. + In this case, the source node can create the port binding and update + the xml. There are 2 options with regard to activating the binding for + the destination host. The source node can activate the binding before + starting the live migration or after it succeeds. Pre-activating the + binding will lead to more work should the migration fail, whereas + activating the binding after migration success could increase network + downtime. The option chosen is left to the review of the + implementation to define and would be documented as a update to the + existing live migration devref. + +This has not been supported due to complexity of code and testing required. + + +Data model impact +----------------- + +None + +REST API impact +--------------- + +None + +Security impact +--------------- + +None + +Notifications impact +-------------------- + +None + +Other end user impact +--------------------- + +None + +Performance Impact +------------------ + +There are extra API calls, but it should have little impact on performance. + +Other deployer impact +--------------------- + +None + +Developer impact +---------------- + +None + +Upgrade impact +-------------- + +As mentioned in `Migration across mixed software versions`_, this change +will account for rolling upgrades by having conductor check the source and +destination compute service versions to see if they are compatible for +using the new port binding flow, otherwise the legacy workflow will be used. + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + Matt Riedemann (mriedem) + +Other contributors: + Sean Mooney (sean-k-mooney) + +Work Items +---------- + +* Add the source/dest host version checks in conductor and the new + compute RPC API method for creating the port binding on the destination + host prior to initiating the live migration on the source host. +* Check for the new migrate_data attribute in the various compute methods + related to live migration to determine if we are old or new flow. + +Dependencies +============ + +* Neutron API changes, see spec: https://specs.openstack.org/openstack/neutron-specs/specs/pike/portbinding_information_for_nova.html + +Testing +======= + +Need functional tests for the new path. + +Documentation Impact +==================== + +Need to update the developer docs to include details on +how Nova now interacts with Neutron during live migration. + +References +========== + +* Neutron spec: https://specs.openstack.org/openstack/neutron-specs/specs/pike/portbinding_information_for_nova.html + +History +======= + +.. list-table:: Revisions + :header-rows: 1 + + * - Release Name + - Description + * - Queens + - Introduced; approved + * - Rocky + - Re-proposed + + +