Add spec for SR-IOV in Octavia Amphora
This patch proposes a specification for adding SR-IOV VF port support to Octavia Amphora load balancers. Change-Id: I98961b162ae6811366bb036e6286f972908d785b
This commit is contained in:
Michael Johnson
parent
12d8e0de5d
commit
8b503d1eaf
@ -90,6 +90,13 @@ Version 1.1 (queens)
|
||||
|
||||
specs/version1.1/*
|
||||
|
||||
Version 14.0 (caracal)
|
||||
``````````````````````
|
||||
.. toctree::
|
||||
:glob:
|
||||
:maxdepth: 1
|
||||
|
||||
specs/version14.0/*
|
||||
|
||||
.. only:: latex
|
||||
|
||||
|
||||
220
specs/version14.0/SRIOV.rst
Normal file
220
specs/version14.0/SRIOV.rst
Normal file
@ -0,0 +1,220 @@
|
||||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
==========================================
|
||||
Support SR-IOV network ports in Octavia
|
||||
==========================================
|
||||
|
||||
The maximum performance of Octavia Amphora based load balancers is often
|
||||
limited by the Software Defined Networking (SDN) used in the OpenStack
|
||||
deployment. There are users that want very high connection rates and high
|
||||
bandwidth through their load balancers.
|
||||
|
||||
This specification describes how we can add Single Root I/O Virtualization
|
||||
(SR-IOV) support to Octavia Amphora load balancers.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
* Users would like to use SR-IOV VFs for the VIP and member ports on their
|
||||
Amphora based load balancers for improved maximum performance and reduced
|
||||
latency. Initial testing showed a 9% increase in bandwidth and a 70% drop
|
||||
in latency through the load balancer when using SR-IOV.
|
||||
|
||||
* Users are overflowing tap interfaces with bursty "thundering herd" traffic
|
||||
such that packets are unable to make it into the Amphora instance.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
Since Octavia hot plugs the network interfaces into the Amphora instances, the
|
||||
first work will be documenting how to configure nova to properly place the
|
||||
Amphorae on hosts with the required hardware and networks. There is some
|
||||
existing documentation for this in the nova guide, but we should summarize it
|
||||
with a focus on Amphora.
|
||||
|
||||
This documentation will include how to configure host aggregates, the compute
|
||||
flavor, and the Octavia flavor to properly schedule the Amphora instances.
|
||||
|
||||
In general, the SR-IOV ports will be handled the same as ports are with the
|
||||
AAP driver, including registering the VIP as an AAP address even though this
|
||||
is technically not required for SR-IOV ports, it will make sure the address is
|
||||
allocated in neutron. Only the base VRRP ports will allocate an SR-IOV VF as
|
||||
the AAP port will be "unbound" with a vnic_type of "normal".
|
||||
|
||||
The create load balancer flow creation will be enhanced to create the base VRRP
|
||||
port using an SR-IOV VF if the Octavia flavor has SRIOV_VIP set to true. If
|
||||
placement/nova scheduling fail to find an appropriate host or the SR-IOV VF
|
||||
port fails to plug into the Amphora, additional logging may be required, but
|
||||
the normal revert flows should continue to handle the error situation and mark
|
||||
the load balancer in provisioning status ERROR.
|
||||
|
||||
The building of the listener create and update flows will need to be updated to
|
||||
include extra tasks to configure nftables inside the Amphora to
|
||||
replace the functionality of the neutron security groups lost when using
|
||||
SR-IOV ports.
|
||||
|
||||
The Amphora agent will need to be enhanced for a new "security group" endpoint
|
||||
and to configure the Amphora nftables. The nftables rules will be added as
|
||||
stateless rules, meaning conntrack will not be enabled. The load balancing
|
||||
engines are already managing state for the flows, so there is no reason to also
|
||||
have state management in the firewall.
|
||||
|
||||
I am proposing we only support nftables inside the Amphora as most
|
||||
distributions are moving away from iptables towards nftables.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
There are two obvious alternatives:
|
||||
|
||||
* Do nothing and continue to rely on SDN performance.
|
||||
|
||||
* Use provider networks to remove some of the overhead of the SDN.
|
||||
|
||||
It is not clear that SDN performance can improve to a level that would meet the
|
||||
needs of Octavia Amphora load balancers and provider networks still have some
|
||||
overhead and limitations depending on how they are implemented (tap interfaces,
|
||||
etc.)
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
The load balancer and member objects will be expanded to include the vnic type
|
||||
for the ports.
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
The Octavia API will be expanded to include the vnic type used for the VIP and
|
||||
member ports. The field with either be "normal" for OVS/OVN ports or "direct"
|
||||
for SR-IOV ports. This field with use the same terminology as neutron uses.
|
||||
|
||||
The Amphora API will need to be expanded to have a security group endpoint.
|
||||
This endpoint will accept POST calls that contain the: allowed_cidrs, protocol,
|
||||
and port information required to configure the appropriate nftable rules.
|
||||
|
||||
When this endpoint is called, the amphora agent will flush the current tables
|
||||
and build up a fresh table. There will be chains for the VIP, VRRP, and member
|
||||
ports. This will be implemented using the python nftables bindings.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
Neutron security groups do not work on SR-IOV ports, so the amphora agent will
|
||||
need to manage nftables for the SR-IOV ports.
|
||||
|
||||
There is no current use case where Octavia would need TRUST mode VFs, so this
|
||||
specification does not include any discussion of enabling TRUST on VFs used by
|
||||
the Octavia amphora driver. The amphora will treat TRUST VFs as if they were
|
||||
not TRUST enabled.
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
End users will need to select the appropriate Octavia flavor at load balancer
|
||||
creation time. They will also need to specify the proper network that matches
|
||||
the network(s) defined in the compute and Octavia flavors.
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
This proposal is specifically about improving data plane performance.
|
||||
|
||||
I would expect little change to the provisioning time, or possibly a faster
|
||||
provisioning time, when using SR-IOV ports as it should require fewer API
|
||||
calls to Neutron.
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
If deployers want SR-IOV interface support at deployment time, they will need
|
||||
to configure the required compute host aggregates, compute flavors, and
|
||||
octavia flavor supporting the SR-IOV enabled hosts and networks.
|
||||
|
||||
We also recommend that the FDB L2 agent be enabled, when needed, so that
|
||||
virtual ports on the same compute host can communicate with the SR-IOV ports.
|
||||
|
||||
The Amphora images will now require the nftables and python3-nftables packages.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
There should be minimal developer impact as it is enhancing existing flows.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
johnsom
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
1. Document the required host aggregates, compute flavor, and Octavia flavor.
|
||||
2. Update the load balancer "create" flow creation to use the SR-IOV tasks
|
||||
when creating the VRRP base ports.
|
||||
3. Update the load balancer data model to store the port vnic type.
|
||||
4. Expand the load balancer API to include the vnic type used for the VIP.
|
||||
5. Update the listener create/update flows to add the extra tasks to configure
|
||||
the nftables inside the Amphora.
|
||||
6. Add a security group endpoint to the Amphora agent to allow configuring and
|
||||
updating the nftables inside the Amphora.
|
||||
7. Add any necessary logging and error handling should nova fail to attach
|
||||
SR-IOV ports.
|
||||
8. Add the required unit and functional tests for the new code.
|
||||
9. Add the required tempest tests to cover the usage scenarios (pending igb
|
||||
driver support in the PTI platforms)
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
None
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Currently this feature cannot fully be tested in the OpenDev gates as it will
|
||||
require an SR-IOV capable nic in the test system.
|
||||
|
||||
There will be unit and function test coverage.
|
||||
|
||||
Recently qemu has added a virtual device, the "igb" device, that is capable of
|
||||
emulating an SR-IOV device. Versions of qemu and the associated libraries that
|
||||
include this new device are not yet shipping in any distribution supported by
|
||||
OpenStack.
|
||||
|
||||
When the "igb" device becomes available, we should be able to run scenario
|
||||
tests with SR-IOV VIP and member ports.
|
||||
|
||||
Performance testing will be out of scope because the OpenDev testing
|
||||
environment does not contain SR-IOV capable NICs and is not setup for data
|
||||
plane performance testing.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
An administrative document will need to be created that describes the process
|
||||
required to setup a compute and octavia flavor for SR-IOV devices.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
* https://docs.openstack.org/neutron/latest/admin/config-sriov.html
|
||||
|
||||
* https://docs.openstack.org/nova/latest/reference/scheduler-hints-vs-flavor-extra-specs.html
|
||||
|
||||
* https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/granular-resource-requests.html
|
||||
|
||||
* https://www.qemu.org/docs/master/system/devices/igb.html
|
||||
Loading…
Reference in New Issue
Block a user