Re-propose use Neutron's new port binding API
This was approved for Queens in change: Ie776601d5966d14fea55d958c7407dd6b710df38 Updates here are assignee (I plan on taking over this work), updating a s/queens/rocky/ part in the description, and adding the upgrade impact section. Previously-approved: Queens Spec for blueprint neutron-new-port-binding-api Change-Id: Ibd1728b5cb094f312c241c5854fd95aa6a218031
This commit is contained in:
parent
32b00cceb7
commit
d9116eaf42
393
specs/rocky/approved/neutron-new-port-binding-api.rst
Normal file
393
specs/rocky/approved/neutron-new-port-binding-api.rst
Normal file
@ -0,0 +1,393 @@
|
|||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
===================================
|
||||||
|
Use Neutron's new port binding API
|
||||||
|
===================================
|
||||||
|
|
||||||
|
Make use of Neutron's new port binding API in all cases where port binding
|
||||||
|
occurs. In the special case of move operations, the new API will allow us to
|
||||||
|
model both source and destination hosts having a port binding
|
||||||
|
which is not accounted for during live migration today.
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/nova/+spec/neutron-new-port-binding-api
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
The main motivation for the change is a selection of problems around
|
||||||
|
live-migration, in particular:
|
||||||
|
|
||||||
|
* If port binding would fail at destination, we would know that before
|
||||||
|
starting the live-migration.
|
||||||
|
* In many cases the two hosts need a very different binding on source host
|
||||||
|
vs destination host, but that's not possible today. In particular, macvtap
|
||||||
|
live migration often needs different libvirt XML on source vs destination.
|
||||||
|
* In Mitaka, Neutron introduced a connection tracker based security group
|
||||||
|
driver for ovs based backends. Live migrating between a host with a
|
||||||
|
different security group driver is another motivating special case of
|
||||||
|
the more general act of live migrating between different ml2 drivers which
|
||||||
|
both require different libvirt XML definitions on the source and
|
||||||
|
destination hosts.
|
||||||
|
* To make the switch between source and destination host as quickly as
|
||||||
|
possible, it's good to get most things ready before the migration is started.
|
||||||
|
We have added a short-term hack https://review.openstack.org/#/c/275073/
|
||||||
|
for DVR, but let's do it properly.
|
||||||
|
|
||||||
|
More details can be found in the neutron spec:
|
||||||
|
https://specs.openstack.org/openstack/neutron-specs/specs/pike/portbinding_information_for_nova.html
|
||||||
|
|
||||||
|
When thinking about this spec, we should be clear on the difference between:
|
||||||
|
|
||||||
|
* port binding, the DB record in Neutron of what host and instance a port is
|
||||||
|
associated with, and its current state.
|
||||||
|
* plug/unplug VIFs, where information from Neutron is passed to OS-VIF (or
|
||||||
|
a legacy driver) to get the port ready on the host
|
||||||
|
* attaching the instances to the above preparations (via a tap device or
|
||||||
|
PCI passthrough, etc)
|
||||||
|
* an active binding, the host where the traffic for associated port should
|
||||||
|
go to, because that's where the VM is actually running
|
||||||
|
|
||||||
|
To address this problem statement we need to consider all the places where
|
||||||
|
we deal with port bindings, to use the new API flow.
|
||||||
|
We generally update the port bindings when a VM is moved.
|
||||||
|
The main API actions to consider are:
|
||||||
|
|
||||||
|
* Attach a port to an instance, including during spawning an instance
|
||||||
|
* Detach a port from an instance
|
||||||
|
* Live-migrate an instance, involves setting up the VIF on the destination
|
||||||
|
host, before kicking off the live-migrate, then removing VIFs on source
|
||||||
|
host once the live-migrate has completed.
|
||||||
|
* Migrate and resize are very similar to live-migrate
|
||||||
|
* Evacuate, we know the old host is dead, we want to kill any record of that
|
||||||
|
old connection, and attach the port on the new host.
|
||||||
|
* Shelve, we want the port to stay logically attached to the instance, but
|
||||||
|
we need to unbind the port for the host when the instance is offloaded.
|
||||||
|
|
||||||
|
In the move operations above neutron should be notified when the traffic needs
|
||||||
|
to be switched to the destination by activating the new port binding.
|
||||||
|
|
||||||
|
Use Cases
|
||||||
|
---------
|
||||||
|
|
||||||
|
As an admin, I want live migration to fail early during pre_live_migration
|
||||||
|
if it can be detected that network setup on the destination host
|
||||||
|
will not work.
|
||||||
|
|
||||||
|
As a user, I want minimal network downtime while my instance is
|
||||||
|
being live migrated.
|
||||||
|
|
||||||
|
As an operator, I want to start using new ML2 backends but need
|
||||||
|
to live migrate existing instances off of a different backend
|
||||||
|
with minimal network downtime.
|
||||||
|
|
||||||
|
As an operator, I want to leverage new security group implementations
|
||||||
|
and need to be able to live migrate existing instances from my
|
||||||
|
legacy hosts.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
There is no additional configuration for deployers.
|
||||||
|
The use of multiple bindings will be enabled automatically.
|
||||||
|
We decide whether to use the new or old API flow, if both compute nodes
|
||||||
|
support this feature and based on the available Neutron API extensions.
|
||||||
|
We cache extensions support in the usual way utilizing the existing
|
||||||
|
neutron_extensions_cache.
|
||||||
|
|
||||||
|
Note: The new neutron API extension will be implemented in the ml2 plugin
|
||||||
|
layer, above the ml2 driver layer so if the extension is exposed it will be
|
||||||
|
supported for all ml2 drivers. Monolithic plugins will have to implement
|
||||||
|
the extension separately and will continue to use the old workflow until
|
||||||
|
their maintainers support the new neutron extension.
|
||||||
|
|
||||||
|
The old interaction model should be removed when sufficient time has elapsed
|
||||||
|
for all neutron backends to support this model with an aim of completing this
|
||||||
|
in two cycles. We should keep this in mind in how the code is structured.
|
||||||
|
Given we are also looking at removing nova-network, we should see if this can
|
||||||
|
be added as a new set of network API calls, that are only for neutron, making
|
||||||
|
the existing calls needed for nova-network be no-ops for Neutron.
|
||||||
|
|
||||||
|
Migration across mixed software versions
|
||||||
|
----------------------------------------
|
||||||
|
|
||||||
|
If an old neutron is present then the existing workflow will be followed
|
||||||
|
regardless of the compute node version. Where a new neutron is deployed
|
||||||
|
that supports this feature the following behaviour will be implemented.
|
||||||
|
|
||||||
|
* If both compute nodes are rocky or newer. In this case the new workflow
|
||||||
|
will be used as described below.
|
||||||
|
|
||||||
|
* In all other cases the old workflow is used
|
||||||
|
|
||||||
|
Let's consider live-migration and what the calls to neutron will look like.
|
||||||
|
|
||||||
|
Today the workflow is documented here:
|
||||||
|
https://docs.openstack.org/nova/latest/reference/live-migration.html
|
||||||
|
|
||||||
|
Going forward the workflow will be altered as follows:
|
||||||
|
|
||||||
|
* Conductor does its pre-checks on the dest and source which
|
||||||
|
creates the migrate_data object.
|
||||||
|
|
||||||
|
* Conductor checks the source and dest version to see if
|
||||||
|
they support the new flow.
|
||||||
|
|
||||||
|
* If new enough, conductor calls a new method on the dest
|
||||||
|
compute to bind the port on the destination host.
|
||||||
|
The new method will POST to /v2.0/ports/{port_id}/bindings passing
|
||||||
|
just the destination host_id::
|
||||||
|
|
||||||
|
{
|
||||||
|
"binding": {
|
||||||
|
"host_id": "target-host_id"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
* If this fails, the live migrate task can ignore that host and
|
||||||
|
reschedule / retry another host because it's before we've cast
|
||||||
|
to the source to start the live migration on the hypervisor.
|
||||||
|
|
||||||
|
* If this succeeds, the port binding results are put into the
|
||||||
|
migrate_data object which is sent to the source live_migration
|
||||||
|
method, and after that all code that cares checks the new
|
||||||
|
attribute in the migrate_data object.
|
||||||
|
|
||||||
|
* The new attribute will consist of the minimal subset of the port
|
||||||
|
binding response and will be encoded in a new nova object::
|
||||||
|
|
||||||
|
fields = {
|
||||||
|
'port_id': fields.StringField(),
|
||||||
|
'host_id': fields.StringField(),
|
||||||
|
'vnic_type': fields.StringField(), # could be enum
|
||||||
|
'vif_type': fields.StringField(),
|
||||||
|
'vif_details': fields.DictOfStringsField(),
|
||||||
|
}
|
||||||
|
|
||||||
|
During implementation we will try to restrict the ``vif_details``
|
||||||
|
field to the subset of vif_details required by nova to generate
|
||||||
|
the updated domain xml and plug the vif. This is to avoid random
|
||||||
|
ML2 backend-specific data from changing behavior in our versioned
|
||||||
|
object. In the future this object will be replaced by one defined
|
||||||
|
by os-vif.
|
||||||
|
|
||||||
|
* In pre_live_migration on destination:
|
||||||
|
|
||||||
|
* Prior to the RPC call from live_migration on the source host to
|
||||||
|
pre_live_migration on the dest host, start a wait thread for the
|
||||||
|
vif-plugged event from Neutron, similar to during initial spawn.
|
||||||
|
|
||||||
|
.. note:: This vif-plugged wait change can be made irrespective of this
|
||||||
|
blueprint - it could be done as a bug fix or hardening opportunity.
|
||||||
|
|
||||||
|
* Check if migrate_data contains new VIFs attribute, if so,
|
||||||
|
plug vif on destination host using the new port bindings,
|
||||||
|
else fall back to old workflow and plug vif with old vif bindings.
|
||||||
|
|
||||||
|
* At this point it is safe to start live migrating the instance.
|
||||||
|
|
||||||
|
* This involves calling the virt driver to live migrate
|
||||||
|
the instance and then activating the port binding. If migrate_data
|
||||||
|
contains the new dest host port binding VIFs attribute, it will
|
||||||
|
be used to configure the dest guest prior to starting the actual
|
||||||
|
live migration in the hypervisor. This is in case the VIF type on
|
||||||
|
the dest host is different from the source host.
|
||||||
|
|
||||||
|
* In the example of the libvirt virt driver, we will wait for a qemu event
|
||||||
|
on the source host called VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY,
|
||||||
|
so we know the VM has just been paused by libvirt and mark the new
|
||||||
|
port binding as active. This is described in more detail here:
|
||||||
|
https://review.openstack.org/#/c/434870/
|
||||||
|
|
||||||
|
* For other virt drivers the decision of when to activate the port
|
||||||
|
binding is left to them. They may serialise the calls by activating
|
||||||
|
the port binding immediately before or after migrating the instance
|
||||||
|
or they may concurrently wait for an event if the hypervisor allows
|
||||||
|
them to reduce the network downtime, or just activate the dest host
|
||||||
|
port binding in post_live_migration.
|
||||||
|
|
||||||
|
* We should only hit an error here if the migration times out.
|
||||||
|
If we hit any other error, there is no rollback and we just
|
||||||
|
put the instance into the ERROR state. If we timeout we abort
|
||||||
|
as described below.
|
||||||
|
|
||||||
|
* During post_live_migration:
|
||||||
|
|
||||||
|
After cleaning up VIFs on the source host, we remove the old port binding
|
||||||
|
associated with the source host. Should the operation get interrupted,
|
||||||
|
there is enough information in the binding to ensure manual
|
||||||
|
cleanup is feasible.
|
||||||
|
|
||||||
|
Aborts
|
||||||
|
------
|
||||||
|
|
||||||
|
* If the admin aborts an in-progress live migration, the rollback actions vary
|
||||||
|
depending on what phase of the migration we are currently in.
|
||||||
|
|
||||||
|
* If we are in the pre_live_migration phase and have not started the migration
|
||||||
|
we simply delete the destination port binding.
|
||||||
|
|
||||||
|
* If we have started the VM on the remote node and plugged the interface but
|
||||||
|
not unpaused the instance, we unplug the instance, activate the source
|
||||||
|
binding if required and delete the destination binding.
|
||||||
|
|
||||||
|
Other
|
||||||
|
-----
|
||||||
|
|
||||||
|
We can follow this pattern wherever there are VIFs present on two hosts, such
|
||||||
|
as during resize and migrate.
|
||||||
|
|
||||||
|
Evacuate is a special case, where we delete the port binding on the old host,
|
||||||
|
without knowing if it has had VIFs deleted, as we assume the host is dead and
|
||||||
|
will never be coming back to life.
|
||||||
|
|
||||||
|
With this change, live migration between hosts with different
|
||||||
|
neutron backends and/or security group drivers should be possible.
|
||||||
|
While not explicitly described in this spec the implementation of this
|
||||||
|
feature should not block that effort or the efforts to adopt oslo versioned
|
||||||
|
objects for nova / neutron portbinding negotiation, however, it is also not
|
||||||
|
dependent on either activity to be completed.
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
We could leave live-migration broken for some Neutron drivers.
|
||||||
|
|
||||||
|
Note: there are additional plans to allow live-migrate to be used to switch
|
||||||
|
between different Neutron plugins, and allowing live-migrate for macvtap
|
||||||
|
attached SR-IOV, but this is not in scope for this change.
|
||||||
|
|
||||||
|
We could support live migration between mixed compute nodes.
|
||||||
|
In this case assuming neutron supported the new flow, the
|
||||||
|
following behaviour would be introduced.
|
||||||
|
|
||||||
|
* old source compute node and a new destination. Taking libvirt as an example,
|
||||||
|
as the migration XML generation is done by the source node if the new
|
||||||
|
destination compute node detects that an XML change would be required it
|
||||||
|
should fail the migration. This changes existing behaviour where
|
||||||
|
live migration may complete successfully but result in no network
|
||||||
|
connectivity.
|
||||||
|
|
||||||
|
* new source compute node and an old destination.
|
||||||
|
In this case, the source node can create the port binding and update
|
||||||
|
the xml. There are 2 options with regard to activating the binding for
|
||||||
|
the destination host. The source node can activate the binding before
|
||||||
|
starting the live migration or after it succeeds. Pre-activating the
|
||||||
|
binding will lead to more work should the migration fail, whereas
|
||||||
|
activating the binding after migration success could increase network
|
||||||
|
downtime. The option chosen is left to the review of the
|
||||||
|
implementation to define and would be documented as a update to the
|
||||||
|
existing live migration devref.
|
||||||
|
|
||||||
|
This has not been supported due to complexity of code and testing required.
|
||||||
|
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Notifications impact
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
There are extra API calls, but it should have little impact on performance.
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Upgrade impact
|
||||||
|
--------------
|
||||||
|
|
||||||
|
As mentioned in `Migration across mixed software versions`_, this change
|
||||||
|
will account for rolling upgrades by having conductor check the source and
|
||||||
|
destination compute service versions to see if they are compatible for
|
||||||
|
using the new port binding flow, otherwise the legacy workflow will be used.
|
||||||
|
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
Matt Riedemann (mriedem)
|
||||||
|
|
||||||
|
Other contributors:
|
||||||
|
Sean Mooney (sean-k-mooney)
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
* Add the source/dest host version checks in conductor and the new
|
||||||
|
compute RPC API method for creating the port binding on the destination
|
||||||
|
host prior to initiating the live migration on the source host.
|
||||||
|
* Check for the new migrate_data attribute in the various compute methods
|
||||||
|
related to live migration to determine if we are old or new flow.
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
* Neutron API changes, see spec: https://specs.openstack.org/openstack/neutron-specs/specs/pike/portbinding_information_for_nova.html
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
Need functional tests for the new path.
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
Need to update the developer docs to include details on
|
||||||
|
how Nova now interacts with Neutron during live migration.
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
* Neutron spec: https://specs.openstack.org/openstack/neutron-specs/specs/pike/portbinding_information_for_nova.html
|
||||||
|
|
||||||
|
History
|
||||||
|
=======
|
||||||
|
|
||||||
|
.. list-table:: Revisions
|
||||||
|
:header-rows: 1
|
||||||
|
|
||||||
|
* - Release Name
|
||||||
|
- Description
|
||||||
|
* - Queens
|
||||||
|
- Introduced; approved
|
||||||
|
* - Rocky
|
||||||
|
- Re-proposed
|
||||||
|
|
||||||
|
|
||||||
|
|
Loading…
Reference in New Issue
Block a user