diff --git a/specs/newton/powervm-sriov-nova.rst b/specs/newton/powervm-sriov-nova.rst new file mode 100644 index 00000000..8eeadd75 --- /dev/null +++ b/specs/newton/powervm-sriov-nova.rst @@ -0,0 +1,350 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +================================= +Nova support for SR-IOV VIF Types +================================= + +https://blueprints.launchpad.net/nova-powervm/+spec/powervm-sriov-nova + +This blueprint will address support of SR-IOV in conjunction with SR-IOV +VF attached to VM via PowerVM vNIC into nova-powervm. SR-IOV support +was added to Juno release of OpenStack, this blueprint will fit +this scenario implementation into it. + +A separate `blueprint for networking-powervm`_ has been made available for +design elements regarding networking-powervm. + +These blueprints will be implemented during Newton cycle of OpenStack +development. Referring to Newton schedule, development should be completed +during newton-3. + +Refer to glossary section for explanation of terms. + +.. _`blueprint for networking-powervm`: https://review.openstack.org/#/c/322210/ + +Problem Description +=================== +OpenStack PowerVM drivers currently support networking aspect of PowerVM +virtualization using Shared Ethernet Adapter, Open vSwitch and Linux Bridge. +There is a need for supporting SR-IOV ports with redundancy/failover and +migration. It is possible to associate SR-IOV VF to a VM directly, but this path +will not be supported by this design. Such a setup will not provide migration +support anyway. Support for this configuration will be added in future. This +path also does not utilize advantages of hardware level virtualization offered +by SR-IOV architecture. + +Users should be able to manage a VM with SR-IOV vNIC as a network interface. +This management should include migration of VM with SR-IOV vNIC attached to it. + +PowerVM has a feature called vNIC which can is tied in with SR-IOV. By using +vNIC the following use cases are supported: +- Fail over I/O to a different I/O Server and physical function +- Live Migration with SR-IOV, without significant intervention +The vNIC is exposed to the VM, and the mac address of the client vNIC will +match the neutron port. + +In summary, this blueprint will solve support of SR-IOV in nova-powervm for +these scenarios: + +1. Ability to attach/detach a SR-IOV VF to a VM as a network interface using + vNIC intermediary during and after deployment, including migration. +2. Ability to provide redundancy/failover support across VFs from Physical Ports + within or across SR-IOV cards using vNIC intermediary. +3. Ability to associate a VLAN with vNIC backed by SR-IOV VF. + +Ability to associate a SR-IOV VF directly to a VM will be done in future. + +Refer to separate `blueprint for networking-powervm`_ for changes in +networking-powervm component. This blueprint will focus on changes to +nova-powervm only. + +Use Cases +--------- +1. Attach vNIC backed by SR-IOV VF(s) to a VM during boot time +2. Attach vNIC backed by SR-IOV VF(s) to a VM after it is deployed +3. Detach vNIC backed by SR-IOV VF(s) from a VM +4. When a VM with vNIC backed by SR-IOV is deleted, perform detach and cleanup +5. Live migrate a VM if using vNIC backed SR-IOV support +6. Provide redundancy/failover support of vNIC backed by SR-IOV VF attached to + a VM during both deploy and post deploy scenarios. + +Proposed changes +================ +The changes will be made in two areas: + +1. **Compute virt driver.** +PowerVM compute driver is in nova.virt.powervm.driver.PowerVMDriver and it will +be enhanced for SR-IOV vNIC support. A dictionary is maintained in virt driver +vif code to map between vif type and vif driver class. Based on vif type of vif +object that needs to be plugged, appropriate vif driver will be invoked. This +dictionary will be modified to include a new vif driver class and its vif type +(pvm_sriov). + +The PCI Claims process expects to be able to "claim" a VF from the +``pci_passthrough_devices`` list each time a vNIC is plugged, and return it to +the pool on unplug. Thus the ``get_available_resource`` API will be enhanced to +populate this device list with a suitable number of VFs. + +2. **VIF driver.** +PowerVM VIF driver is in nova_powervm.virt.powervm.vif.PvmVifDriver. A VIF +driver to attach network interface via vNIC (PvmVnicSriovVifDriver) and plug/ +unplug methods will be implemented. Plug and unplug methods will use pypowervm +code to create VF/vNIC server/vNIC clients and attach/detach them. Neutron port +carries binding:vif_type and binding:vnic_type attributes. The vif type for this +implementation will be pvm_sriov. The vnic_type will be 'direct'. + +A VIF driver (PvmVFSriovVifDriver) for directly attached to VM will get +implemented in future. + +Deployment of VM with SR-IOV vNIC will involve picking Physical Port(s), +VIOS(es) and a VM and invoking pypowervm library. Similarly, attachment of the +same to an existing VM will be implemented. RMC will be required. Evacuate and +migration of VM will be supported with changes to compute virt driver and VIF +driver via pypowervm library. + +Physical Port information will be derived from port label attribute of physical +ports on SR-IOV adapters. Port label attribute of physical ports will have to be +updated with 'physical network' names during configuration of the environment. +During attachment of SR-IOV backed vNIC to a VM, physical network attribute of +neutron network will be matched with port labels of physical ports to gather a +list of physical ports. + +**Failover/redundancy:** VIF plug during deploy (or attach of network interface +to a VM) will pass more than one Physical port and VIOS(es) (as stated above in +deploy scenario) to pypowervm library to create vNIC on VIOS with redundancy. It +should be noted that failover is handled automatically by the platform when a +vNIC is backed by multiple VFs. The redundancy level will be controlled by an +``AGENT`` option ``vnic_required_vfs`` in the ML2 configuration file (see the +`blueprint for networking-powervm`_). It will have a default of 2. + +**Quality of Service:** Each VF backing a vNIC can be configured with a capacity +value, dictating the minimum percentage of the physical port's total bandwidth +that will be available to that VF. The ML2 configuration file allows a +``vnic_vf_capacity`` option in the ``AGENT`` section to set the capacity for all +vNIC-backing VFs. If omitted, the platform defaults to the capacity granularity +for each physical port. See the `blueprint for networking-powervm`_ for +details of the configuration option; and see section 1.3.3 of the `IBM Power +Systems SR-IOV Technical Overview and Introduction +`_ for details on VF +capacity. + +For future implementation of VF - VM direct attach of SR-IOV to a VM, the +request will include physical network name. PvmVFSriovVifDriver can lookup +devname(s) associated with it from port label, get physical port information +and create a SR-IOV logical port on the corresponding VM. +Or may include a configuration option to allow the user to dictate how many +ports to attach. Using NIB technique, users can setup redundancy. + +For VF - vNIC - VM attach of SR-IOV port to a VM, the corresponding neutron +network object will include physical network name, PvmVnicSriovVifDriver can +lookup devname(s) associated with it from port label, get physical port +information. Along with adapter ID and physical port ID, VIOS information will +be added and a VNIC dedicated port on the corresponding VM will be created. + +For migration scenario, physical network names should match on source and +destination compute nodes, and accordingly in the physical port labels. On the +destination, vNICs will be rebuilt based on the SR-IOV port configuration. The +platform decides how to reconstruct the vNIC on the destination in terms of +number and distribution of backing VFs, etc. + +Alternatives +------------ +None + +Security impact +--------------- +None + +Other end user impact +--------------------- +None + +Performance impact +------------------ +Since the number of VMs deployed on the host will depend on number of VFs +offered by SR-IOV cards in the environment, scale tests will be limited in +density of VMs. + +Deployer impact +--------------- +1. SR-IOV cards must be configured in ``Sriov`` mode. This can be done via the + ``pvmctl`` command, e.g.: + + ``pvmctl sriov update -i phys_loc=U78C7.001.RCH0004-P1-C1 -s mode=Sriov`` + +2. SR-IOV physical ports must be labeled with the name of the neutron physical + network to which they are cabled. This can be done via the ``pvmctl`` + command, e.g.: + + ``pvmctl sriov update --port-loc U78C7.001.RCH0004-P1-C1-T1 -s label=prod_net`` + +3. The ``pci_passthrough_whitelist`` option in the nova configuration file must + include entries for each neutron physical network to be enabled for vNIC. + Only the ``physical_network`` key is required. For example: + + ``pci_passthrough_whitelist = [{"physical_network": "default"}, {"physical_network": "prod_net"}]`` + +Configuration is also required on the networking side - see the `blueprint for +networking-powervm`_ for details. + +**To deploy a vNIC to a VM,** the neutron port(s) must be pre-created with vnic +type ``direct``, e.g.: + + ``neutron port-create --vnic-type direct`` + +Developer impact +---------------- +None + +Dependencies +------------ + +#. SR-IOV cards and SR-IOV-capable hardware +#. Updated levels of system firmware and the Virtual I/O Server operating system +#. An updated version of Novalink PowerVM feature +#. pypowervm library - https://github.com/powervm/pypowervm + +Implementation +============== + +Assignee(s) +----------- +- Eric Fried (efried) +- Sridhar Venkat (svenkat) +- Eric Larese (erlarese) +- Esha Seth (eshaseth) +- Drew Thorstensen (thorst) + +Work Items +---------- +nova-powervm changes: + +- Updates to PowerVM compute driver to support attachment of SR-IOV VF via vNIC. +- VIF driver for SR-IOV VF connected to VM via vNIC. +- Migration of VM with SR-IOV VF connected to VM via vNIC. This involves live + migration, cold migration and evacuation. +- Failover/redundancy support for SR-IOV VF(s) connected to VM via vNIC(s). + +VIF driver for SR-IOV VF connected to VM directly will be a future work item. + +Testing +======= +1. Unit test +All developed code will accompany structured unit test around them. These +tests validate granular function logic. + +2. Function test +Function test will be performed along with CI infrastructure. Changes +implemented for this blueprint will be tested via CI framework that exists +and used by IBM team. CI framework needs to be enhanced with SR-IOV hardware. +The tests can be executed in batch mode, probably as nightly jobs. + +Documentation impact +==================== +All use-cases need to be documented in developer docs that accompany +nova-powervm. + +References +========== +1. This blog describes how to work with SR-IOV and vNIC (without redundancy/ + failover) using HMC interface: http://chmod666.org/index.php/a-first-look-at-sriov-vnic-adapters/ + +2. These describe vNIC and its usage with SR-IOV. + + - https://www.ibm.com/developerworks/community/wikis/home?lang=en_us#!/wiki/Power%20Systems/page/vNIC%20-%20Introducing%20a%20New%20PowerVM%20Virtual%20Networking%20Technology + - https://www.ibm.com/developerworks/community/wikis/home?lang=en_us#!/wiki/Power%20Systems/page/Introduction%20to%20SR-IOV%20FAQs + - https://www.ibm.com/developerworks/community/wikis/home?lang=en_us#!/wiki/Power%20Systems/page/Introduction%20to%20vNIC%20FAQs + - https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power%20Systems/page/vNIC%20Frequently%20Asked%20Questions + +3. These describe SR-IOV in OpenStack. + + - https://wiki.openstack.org/wiki/Nova-neutron-sriov + - http://docs.openstack.org/mitaka/networking-guide/adv-config-sriov.html + +4. This blueprint addresses SR-IOV attach/detach function in nova: https://review.openstack.org/#/c/139910/ + +5. networking-powervm blueprint for same work: https://review.openstack.org/#/c/322210/ + +6. This is a detailed description of SR-IOV implementation in PowerVM: https://www.redbooks.ibm.com/redpapers/pdfs/redp5065.pdf + +7. This provides a overall view of SR-IOV support in nova: https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov + +8. Attach/detach of SR-IOV ports to VM with respect to libvirt. Provided here + for comparison purposes: https://review.openstack.org/#/c/139910/ + +9. SR-IOV PCI passthrough reference: https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking + +10. pypowervm: https://github.com/powervm/pypowervm + +Glossary +======== +:SR-IOV: Single Root I/O Virtualization, used for virtual environments where VMs + need direct access to network interface without any hypervisor overheads. + +:Physical Port: Represents Physical port in SR-IOV adapter. This is not same + as Physical Function. A Physical Port can have many physical functions + associated with it. To clarify further, if a Physical Port supports RCoE, then + it will have two Physical Functions. In other words, one Physical Function per + protocol that port supports. + +:Virtual Function (VF): Represents Virtual port belonging to a Physical Port + (PF). Either directly or indirectly (using vNIC) a Virtual Function (VF) is + connected to a VM. This is otherwise called SR-IOV logical port. + +:Dedicated SR-IOV: This is equivalent to any regular ethernet card and it + can be used with SEA. A logical port of a physical port can be assigned as a + backing device for SEA. + +:Shared SR-IOV: A VF to VM is not supported in Newton release. But an SR-IOV + card in sriov mode is what we will be used for vNIC as described in this + blueprint. Also, a SR-IOV in Sriov mode can have a promiscous VF assigned to + the VIOS and configured for SEA(said configuration to be done outside of the + auspices of OpenStack), which can then be used just like any other SEA + configuration, and is supported (as described in next item below). + +:Shared Ethernet Adapter: Alternate technique to provide network interface to a + VM. + + This involves attachment to a physical interface on PowerVM host and one or + many virtual interfaces that are connected to VMs. A VF of PF in SR-IOV based + environment can be a physical interface to Shared Ethernet Adapter. Existing + support for this configuration in nova-powervm and networking-powervm will + continue. + +:vNIC: A vNIC is an intermediary between VF of PF and VM. This resides on VIOS + and connects to a VF one one end and vNIC client adapter inside a VM. This is + mainly to support migration of VMs across hosts. + +:vNIC failover/redundancy: Multiple vNIC servers (connected to as many VFs that + belong to as many PFs either on same SR-IOV card or across) connected to same + VM as one network interface. Failure of one vNIC/VF/PF path will result in + activation of other such path. + +:VIOS: A partition in PowerVM systems dedicated for i/o operations. In the + context of this blueprint, vNIC server will be created on VIOS. For redundancy + management purposes, a specific PowerVM system may employ more than one VIOS + partitions. + +:VM migration types: + + - **Live Migration:** migration of VM while both host and VM are alive. + - **Cold Migration:** migration of VM while host is alive and VM is down. + - **Evacuation:** migration of VM while hots is down (VM is down as well). + - **Rebuild:** recreation of a VM. + +:pypowervm: A python library that runs on the PowerVM management VM and allows + virtualization control of the system. This is similar to the python library + for libvirt. + +History +======= + +============ =========== +Release Name Description +============ =========== +Newton Introduced +============ ===========