Blueprint for nova-powervm SR-IOV VIFs
This is the blueprint to add support for SR-IOV VIFs into nova-powervm. Refer to networking-powervm blueprint for corresponding changes https://blueprints.launchpad.net/networking-powervm/+spec/ powervm-sriov Change-Id: I44600be808da3be9a485e0fc210116339bcfcb77 Implements: https://blueprints.launchpad.net/nova-powervm/spec/powervm-sriov-nova
This commit is contained in:
parent
bd9e2b5bea
commit
d1c023b0bf
|
@ -0,0 +1,350 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
=================================
|
||||
Nova support for SR-IOV VIF Types
|
||||
=================================
|
||||
|
||||
https://blueprints.launchpad.net/nova-powervm/+spec/powervm-sriov-nova
|
||||
|
||||
This blueprint will address support of SR-IOV in conjunction with SR-IOV
|
||||
VF attached to VM via PowerVM vNIC into nova-powervm. SR-IOV support
|
||||
was added to Juno release of OpenStack, this blueprint will fit
|
||||
this scenario implementation into it.
|
||||
|
||||
A separate `blueprint for networking-powervm`_ has been made available for
|
||||
design elements regarding networking-powervm.
|
||||
|
||||
These blueprints will be implemented during Newton cycle of OpenStack
|
||||
development. Referring to Newton schedule, development should be completed
|
||||
during newton-3.
|
||||
|
||||
Refer to glossary section for explanation of terms.
|
||||
|
||||
.. _`blueprint for networking-powervm`: https://review.openstack.org/#/c/322210/
|
||||
|
||||
Problem Description
|
||||
===================
|
||||
OpenStack PowerVM drivers currently support networking aspect of PowerVM
|
||||
virtualization using Shared Ethernet Adapter, Open vSwitch and Linux Bridge.
|
||||
There is a need for supporting SR-IOV ports with redundancy/failover and
|
||||
migration. It is possible to associate SR-IOV VF to a VM directly, but this path
|
||||
will not be supported by this design. Such a setup will not provide migration
|
||||
support anyway. Support for this configuration will be added in future. This
|
||||
path also does not utilize advantages of hardware level virtualization offered
|
||||
by SR-IOV architecture.
|
||||
|
||||
Users should be able to manage a VM with SR-IOV vNIC as a network interface.
|
||||
This management should include migration of VM with SR-IOV vNIC attached to it.
|
||||
|
||||
PowerVM has a feature called vNIC which can is tied in with SR-IOV. By using
|
||||
vNIC the following use cases are supported:
|
||||
- Fail over I/O to a different I/O Server and physical function
|
||||
- Live Migration with SR-IOV, without significant intervention
|
||||
The vNIC is exposed to the VM, and the mac address of the client vNIC will
|
||||
match the neutron port.
|
||||
|
||||
In summary, this blueprint will solve support of SR-IOV in nova-powervm for
|
||||
these scenarios:
|
||||
|
||||
1. Ability to attach/detach a SR-IOV VF to a VM as a network interface using
|
||||
vNIC intermediary during and after deployment, including migration.
|
||||
2. Ability to provide redundancy/failover support across VFs from Physical Ports
|
||||
within or across SR-IOV cards using vNIC intermediary.
|
||||
3. Ability to associate a VLAN with vNIC backed by SR-IOV VF.
|
||||
|
||||
Ability to associate a SR-IOV VF directly to a VM will be done in future.
|
||||
|
||||
Refer to separate `blueprint for networking-powervm`_ for changes in
|
||||
networking-powervm component. This blueprint will focus on changes to
|
||||
nova-powervm only.
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
1. Attach vNIC backed by SR-IOV VF(s) to a VM during boot time
|
||||
2. Attach vNIC backed by SR-IOV VF(s) to a VM after it is deployed
|
||||
3. Detach vNIC backed by SR-IOV VF(s) from a VM
|
||||
4. When a VM with vNIC backed by SR-IOV is deleted, perform detach and cleanup
|
||||
5. Live migrate a VM if using vNIC backed SR-IOV support
|
||||
6. Provide redundancy/failover support of vNIC backed by SR-IOV VF attached to
|
||||
a VM during both deploy and post deploy scenarios.
|
||||
|
||||
Proposed changes
|
||||
================
|
||||
The changes will be made in two areas:
|
||||
|
||||
1. **Compute virt driver.**
|
||||
PowerVM compute driver is in nova.virt.powervm.driver.PowerVMDriver and it will
|
||||
be enhanced for SR-IOV vNIC support. A dictionary is maintained in virt driver
|
||||
vif code to map between vif type and vif driver class. Based on vif type of vif
|
||||
object that needs to be plugged, appropriate vif driver will be invoked. This
|
||||
dictionary will be modified to include a new vif driver class and its vif type
|
||||
(pvm_sriov).
|
||||
|
||||
The PCI Claims process expects to be able to "claim" a VF from the
|
||||
``pci_passthrough_devices`` list each time a vNIC is plugged, and return it to
|
||||
the pool on unplug. Thus the ``get_available_resource`` API will be enhanced to
|
||||
populate this device list with a suitable number of VFs.
|
||||
|
||||
2. **VIF driver.**
|
||||
PowerVM VIF driver is in nova_powervm.virt.powervm.vif.PvmVifDriver. A VIF
|
||||
driver to attach network interface via vNIC (PvmVnicSriovVifDriver) and plug/
|
||||
unplug methods will be implemented. Plug and unplug methods will use pypowervm
|
||||
code to create VF/vNIC server/vNIC clients and attach/detach them. Neutron port
|
||||
carries binding:vif_type and binding:vnic_type attributes. The vif type for this
|
||||
implementation will be pvm_sriov. The vnic_type will be 'direct'.
|
||||
|
||||
A VIF driver (PvmVFSriovVifDriver) for directly attached to VM will get
|
||||
implemented in future.
|
||||
|
||||
Deployment of VM with SR-IOV vNIC will involve picking Physical Port(s),
|
||||
VIOS(es) and a VM and invoking pypowervm library. Similarly, attachment of the
|
||||
same to an existing VM will be implemented. RMC will be required. Evacuate and
|
||||
migration of VM will be supported with changes to compute virt driver and VIF
|
||||
driver via pypowervm library.
|
||||
|
||||
Physical Port information will be derived from port label attribute of physical
|
||||
ports on SR-IOV adapters. Port label attribute of physical ports will have to be
|
||||
updated with 'physical network' names during configuration of the environment.
|
||||
During attachment of SR-IOV backed vNIC to a VM, physical network attribute of
|
||||
neutron network will be matched with port labels of physical ports to gather a
|
||||
list of physical ports.
|
||||
|
||||
**Failover/redundancy:** VIF plug during deploy (or attach of network interface
|
||||
to a VM) will pass more than one Physical port and VIOS(es) (as stated above in
|
||||
deploy scenario) to pypowervm library to create vNIC on VIOS with redundancy. It
|
||||
should be noted that failover is handled automatically by the platform when a
|
||||
vNIC is backed by multiple VFs. The redundancy level will be controlled by an
|
||||
``AGENT`` option ``vnic_required_vfs`` in the ML2 configuration file (see the
|
||||
`blueprint for networking-powervm`_). It will have a default of 2.
|
||||
|
||||
**Quality of Service:** Each VF backing a vNIC can be configured with a capacity
|
||||
value, dictating the minimum percentage of the physical port's total bandwidth
|
||||
that will be available to that VF. The ML2 configuration file allows a
|
||||
``vnic_vf_capacity`` option in the ``AGENT`` section to set the capacity for all
|
||||
vNIC-backing VFs. If omitted, the platform defaults to the capacity granularity
|
||||
for each physical port. See the `blueprint for networking-powervm`_ for
|
||||
details of the configuration option; and see section 1.3.3 of the `IBM Power
|
||||
Systems SR-IOV Technical Overview and Introduction
|
||||
<https://www.redbooks.ibm.com/redpapers/pdfs/redp5065.pdf>`_ for details on VF
|
||||
capacity.
|
||||
|
||||
For future implementation of VF - VM direct attach of SR-IOV to a VM, the
|
||||
request will include physical network name. PvmVFSriovVifDriver can lookup
|
||||
devname(s) associated with it from port label, get physical port information
|
||||
and create a SR-IOV logical port on the corresponding VM.
|
||||
Or may include a configuration option to allow the user to dictate how many
|
||||
ports to attach. Using NIB technique, users can setup redundancy.
|
||||
|
||||
For VF - vNIC - VM attach of SR-IOV port to a VM, the corresponding neutron
|
||||
network object will include physical network name, PvmVnicSriovVifDriver can
|
||||
lookup devname(s) associated with it from port label, get physical port
|
||||
information. Along with adapter ID and physical port ID, VIOS information will
|
||||
be added and a VNIC dedicated port on the corresponding VM will be created.
|
||||
|
||||
For migration scenario, physical network names should match on source and
|
||||
destination compute nodes, and accordingly in the physical port labels. On the
|
||||
destination, vNICs will be rebuilt based on the SR-IOV port configuration. The
|
||||
platform decides how to reconstruct the vNIC on the destination in terms of
|
||||
number and distribution of backing VFs, etc.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
None
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
None
|
||||
|
||||
Performance impact
|
||||
------------------
|
||||
Since the number of VMs deployed on the host will depend on number of VFs
|
||||
offered by SR-IOV cards in the environment, scale tests will be limited in
|
||||
density of VMs.
|
||||
|
||||
Deployer impact
|
||||
---------------
|
||||
1. SR-IOV cards must be configured in ``Sriov`` mode. This can be done via the
|
||||
``pvmctl`` command, e.g.:
|
||||
|
||||
``pvmctl sriov update -i phys_loc=U78C7.001.RCH0004-P1-C1 -s mode=Sriov``
|
||||
|
||||
2. SR-IOV physical ports must be labeled with the name of the neutron physical
|
||||
network to which they are cabled. This can be done via the ``pvmctl``
|
||||
command, e.g.:
|
||||
|
||||
``pvmctl sriov update --port-loc U78C7.001.RCH0004-P1-C1-T1 -s label=prod_net``
|
||||
|
||||
3. The ``pci_passthrough_whitelist`` option in the nova configuration file must
|
||||
include entries for each neutron physical network to be enabled for vNIC.
|
||||
Only the ``physical_network`` key is required. For example:
|
||||
|
||||
``pci_passthrough_whitelist = [{"physical_network": "default"}, {"physical_network": "prod_net"}]``
|
||||
|
||||
Configuration is also required on the networking side - see the `blueprint for
|
||||
networking-powervm`_ for details.
|
||||
|
||||
**To deploy a vNIC to a VM,** the neutron port(s) must be pre-created with vnic
|
||||
type ``direct``, e.g.:
|
||||
|
||||
``neutron port-create --vnic-type direct``
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
None
|
||||
|
||||
Dependencies
|
||||
------------
|
||||
|
||||
#. SR-IOV cards and SR-IOV-capable hardware
|
||||
#. Updated levels of system firmware and the Virtual I/O Server operating system
|
||||
#. An updated version of Novalink PowerVM feature
|
||||
#. pypowervm library - https://github.com/powervm/pypowervm
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
- Eric Fried (efried)
|
||||
- Sridhar Venkat (svenkat)
|
||||
- Eric Larese (erlarese)
|
||||
- Esha Seth (eshaseth)
|
||||
- Drew Thorstensen (thorst)
|
||||
|
||||
Work Items
|
||||
----------
|
||||
nova-powervm changes:
|
||||
|
||||
- Updates to PowerVM compute driver to support attachment of SR-IOV VF via vNIC.
|
||||
- VIF driver for SR-IOV VF connected to VM via vNIC.
|
||||
- Migration of VM with SR-IOV VF connected to VM via vNIC. This involves live
|
||||
migration, cold migration and evacuation.
|
||||
- Failover/redundancy support for SR-IOV VF(s) connected to VM via vNIC(s).
|
||||
|
||||
VIF driver for SR-IOV VF connected to VM directly will be a future work item.
|
||||
|
||||
Testing
|
||||
=======
|
||||
1. Unit test
|
||||
All developed code will accompany structured unit test around them. These
|
||||
tests validate granular function logic.
|
||||
|
||||
2. Function test
|
||||
Function test will be performed along with CI infrastructure. Changes
|
||||
implemented for this blueprint will be tested via CI framework that exists
|
||||
and used by IBM team. CI framework needs to be enhanced with SR-IOV hardware.
|
||||
The tests can be executed in batch mode, probably as nightly jobs.
|
||||
|
||||
Documentation impact
|
||||
====================
|
||||
All use-cases need to be documented in developer docs that accompany
|
||||
nova-powervm.
|
||||
|
||||
References
|
||||
==========
|
||||
1. This blog describes how to work with SR-IOV and vNIC (without redundancy/
|
||||
failover) using HMC interface: http://chmod666.org/index.php/a-first-look-at-sriov-vnic-adapters/
|
||||
|
||||
2. These describe vNIC and its usage with SR-IOV.
|
||||
|
||||
- https://www.ibm.com/developerworks/community/wikis/home?lang=en_us#!/wiki/Power%20Systems/page/vNIC%20-%20Introducing%20a%20New%20PowerVM%20Virtual%20Networking%20Technology
|
||||
- https://www.ibm.com/developerworks/community/wikis/home?lang=en_us#!/wiki/Power%20Systems/page/Introduction%20to%20SR-IOV%20FAQs
|
||||
- https://www.ibm.com/developerworks/community/wikis/home?lang=en_us#!/wiki/Power%20Systems/page/Introduction%20to%20vNIC%20FAQs
|
||||
- https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power%20Systems/page/vNIC%20Frequently%20Asked%20Questions
|
||||
|
||||
3. These describe SR-IOV in OpenStack.
|
||||
|
||||
- https://wiki.openstack.org/wiki/Nova-neutron-sriov
|
||||
- http://docs.openstack.org/mitaka/networking-guide/adv-config-sriov.html
|
||||
|
||||
4. This blueprint addresses SR-IOV attach/detach function in nova: https://review.openstack.org/#/c/139910/
|
||||
|
||||
5. networking-powervm blueprint for same work: https://review.openstack.org/#/c/322210/
|
||||
|
||||
6. This is a detailed description of SR-IOV implementation in PowerVM: https://www.redbooks.ibm.com/redpapers/pdfs/redp5065.pdf
|
||||
|
||||
7. This provides a overall view of SR-IOV support in nova: https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov
|
||||
|
||||
8. Attach/detach of SR-IOV ports to VM with respect to libvirt. Provided here
|
||||
for comparison purposes: https://review.openstack.org/#/c/139910/
|
||||
|
||||
9. SR-IOV PCI passthrough reference: https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking
|
||||
|
||||
10. pypowervm: https://github.com/powervm/pypowervm
|
||||
|
||||
Glossary
|
||||
========
|
||||
:SR-IOV: Single Root I/O Virtualization, used for virtual environments where VMs
|
||||
need direct access to network interface without any hypervisor overheads.
|
||||
|
||||
:Physical Port: Represents Physical port in SR-IOV adapter. This is not same
|
||||
as Physical Function. A Physical Port can have many physical functions
|
||||
associated with it. To clarify further, if a Physical Port supports RCoE, then
|
||||
it will have two Physical Functions. In other words, one Physical Function per
|
||||
protocol that port supports.
|
||||
|
||||
:Virtual Function (VF): Represents Virtual port belonging to a Physical Port
|
||||
(PF). Either directly or indirectly (using vNIC) a Virtual Function (VF) is
|
||||
connected to a VM. This is otherwise called SR-IOV logical port.
|
||||
|
||||
:Dedicated SR-IOV: This is equivalent to any regular ethernet card and it
|
||||
can be used with SEA. A logical port of a physical port can be assigned as a
|
||||
backing device for SEA.
|
||||
|
||||
:Shared SR-IOV: A VF to VM is not supported in Newton release. But an SR-IOV
|
||||
card in sriov mode is what we will be used for vNIC as described in this
|
||||
blueprint. Also, a SR-IOV in Sriov mode can have a promiscous VF assigned to
|
||||
the VIOS and configured for SEA(said configuration to be done outside of the
|
||||
auspices of OpenStack), which can then be used just like any other SEA
|
||||
configuration, and is supported (as described in next item below).
|
||||
|
||||
:Shared Ethernet Adapter: Alternate technique to provide network interface to a
|
||||
VM.
|
||||
|
||||
This involves attachment to a physical interface on PowerVM host and one or
|
||||
many virtual interfaces that are connected to VMs. A VF of PF in SR-IOV based
|
||||
environment can be a physical interface to Shared Ethernet Adapter. Existing
|
||||
support for this configuration in nova-powervm and networking-powervm will
|
||||
continue.
|
||||
|
||||
:vNIC: A vNIC is an intermediary between VF of PF and VM. This resides on VIOS
|
||||
and connects to a VF one one end and vNIC client adapter inside a VM. This is
|
||||
mainly to support migration of VMs across hosts.
|
||||
|
||||
:vNIC failover/redundancy: Multiple vNIC servers (connected to as many VFs that
|
||||
belong to as many PFs either on same SR-IOV card or across) connected to same
|
||||
VM as one network interface. Failure of one vNIC/VF/PF path will result in
|
||||
activation of other such path.
|
||||
|
||||
:VIOS: A partition in PowerVM systems dedicated for i/o operations. In the
|
||||
context of this blueprint, vNIC server will be created on VIOS. For redundancy
|
||||
management purposes, a specific PowerVM system may employ more than one VIOS
|
||||
partitions.
|
||||
|
||||
:VM migration types:
|
||||
|
||||
- **Live Migration:** migration of VM while both host and VM are alive.
|
||||
- **Cold Migration:** migration of VM while host is alive and VM is down.
|
||||
- **Evacuation:** migration of VM while hots is down (VM is down as well).
|
||||
- **Rebuild:** recreation of a VM.
|
||||
|
||||
:pypowervm: A python library that runs on the PowerVM management VM and allows
|
||||
virtualization control of the system. This is similar to the python library
|
||||
for libvirt.
|
||||
|
||||
History
|
||||
=======
|
||||
|
||||
============ ===========
|
||||
Release Name Description
|
||||
============ ===========
|
||||
Newton Introduced
|
||||
============ ===========
|
Loading…
Reference in New Issue