OVN Neutron Agent and hardware offloaded QoS extension.
Change-Id: I3d1c17303c6aa4491fa156ebed76babe23593663 Related-Bug: #1998608
This commit is contained in:
parent
e5e141f134
commit
f116fcd004
|
@ -0,0 +1,317 @@
|
||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
======================================================
|
||||||
|
OVN Neutron Agent and hardware offloaded QoS extension
|
||||||
|
======================================================
|
||||||
|
|
||||||
|
Launchpad bug: https://bugs.launchpad.net/neutron/+bug/1998608
|
||||||
|
|
||||||
|
The aim of the RFE is to define the architecture of a new generic OVN agent
|
||||||
|
that will execute any needed functionality, being extensible as any other ML2
|
||||||
|
agent. The first functionality to be implemented will be the hardware
|
||||||
|
offloaded QoS extension.
|
||||||
|
|
||||||
|
|
||||||
|
Problem Description
|
||||||
|
===================
|
||||||
|
|
||||||
|
ML2/OVN is a mechanism driver that doesn't have backend agents. This mechanism
|
||||||
|
driver exposes two agent types: the OVN controller agent and the metadata
|
||||||
|
agent. The OVN controller agent is a representation of the status of the
|
||||||
|
ovn-controller service on the node; the metadata agent is the service that
|
||||||
|
provides the metadata to the virtual machines.
|
||||||
|
|
||||||
|
However the ML2/OVN mechanism driver does not have an agent to perform the
|
||||||
|
local Open vSwitch service configuration as in ML2/OVS, with extension drivers
|
||||||
|
(like QoS or Trunk). The ovn-controller reads the OVN Southbound database to
|
||||||
|
configure the local OVS database.
|
||||||
|
|
||||||
|
The lack of an agent running on the local node, owned by Neutron, prevents us
|
||||||
|
from implementing some functionalities currently not supported by OVN nor the
|
||||||
|
drivers. For example, and this will be the first feature that the OVN agent
|
||||||
|
will implement, hardware offloaded ports cannot apply the OVS QoS rules due to
|
||||||
|
limitations in the drivers, in particular the ones for the NVIDIA Mellanox
|
||||||
|
ConnectX-5 network cards [0]_.
|
||||||
|
|
||||||
|
|
||||||
|
Proposed Change
|
||||||
|
===============
|
||||||
|
|
||||||
|
This RFE proposes to create a generic agent running on the compute node. This
|
||||||
|
agent will be called "OVN Neutron Agent". The execution of this agent is
|
||||||
|
discretionary and will be needed only if the specific features implemented
|
||||||
|
on it are requested in a particular compute node. In other words, initially
|
||||||
|
this service will be needed if a compute node has hardware offloaded ports
|
||||||
|
and QoS is needed (new features could be implemented in the future).
|
||||||
|
|
||||||
|
Unlike other mechanism driver agents (like in ML2/OVS or ML2/SRIOV), this
|
||||||
|
agent does not implement an RPC client. The information required to work
|
||||||
|
will be retrieved from the local OVS database, the OVN Northbound database
|
||||||
|
and the OVN Southbound database. It will be discussed in a future RFE to
|
||||||
|
include RPC capabilities to have a direct connection to the Neutron
|
||||||
|
server and the Neutron database.
|
||||||
|
|
||||||
|
Extensible functionalities
|
||||||
|
--------------------------
|
||||||
|
|
||||||
|
The new agent functionalities will be defined via extensions. It will use
|
||||||
|
the ``neutron.agent.agent_extensions_manager.AgentExtensionsManager`` class
|
||||||
|
to load the configured OVN Neutron Agent extensions. It will use the same
|
||||||
|
configuration variable "agent.extensions" used by other ML2 agents. The OVN
|
||||||
|
Neutron Agent extensions will inherit from
|
||||||
|
``neutron_lib.agent.extensions.AgentExtension`` and will be loaded during the
|
||||||
|
agent startup by the agent extension manager.
|
||||||
|
|
||||||
|
As commented before, the OVN agent will have active OVN Northbound, OVN
|
||||||
|
Southbound and local OVS database connections, using the corresponding IDL
|
||||||
|
classes. These classes will monitor a certain set of tables and will
|
||||||
|
register a set of events. Each extension must report what tables and events
|
||||||
|
are needed during the startup process. The extensions will inherit from a
|
||||||
|
specific OVN Neutron Agent extension class (derived from the mentioned
|
||||||
|
``AgentExtension``) that will provide API methods to retrieve this
|
||||||
|
information.
|
||||||
|
|
||||||
|
|
||||||
|
OVN agent status report
|
||||||
|
-----------------------
|
||||||
|
|
||||||
|
This new OVN Neutron Agent needs to be tracked in the Neutron server.
|
||||||
|
There are currently two types of agents: the "OVN Controller agent"/"OVN
|
||||||
|
Controller Gateway agent" and the "OVN metadata agent". This RFE will
|
||||||
|
introduce the "OVN Neutron Agent" type. This is a generic name matching
|
||||||
|
the goal of this spec that is to create a generic OVN agent to implement
|
||||||
|
all needed functionalities that the OVN controller cannot provide
|
||||||
|
(metadata will be the next feature to be implemented there, migrated
|
||||||
|
from the current "OVN metadata agent").
|
||||||
|
|
||||||
|
The reporting mechanism will be the same as the one implemented in the
|
||||||
|
OVN metadata agent. When the Southbound table "SB_Global" is updated,
|
||||||
|
the agent will update a key in the "external_ids" dictionary. Please check
|
||||||
|
``neutron.agent.ovn.metadata.agent.SbGlobalUpdateEvent`` for an
|
||||||
|
implementation example. This variable will be read by the server that will
|
||||||
|
check the timestamp and determine the agent status.
|
||||||
|
|
||||||
|
|
||||||
|
QoS for hardware offloaded ports
|
||||||
|
--------------------------------
|
||||||
|
|
||||||
|
The first feature (extension) that will be implemented within this agent
|
||||||
|
is the QoS extension for hardware offloaded ports. Because of the current
|
||||||
|
drivers limitations, these ports cannot be properly configured with the
|
||||||
|
current QoS rules.
|
||||||
|
|
||||||
|
This RFE proposes to use the corresponding "ip-link" [1]_ commands (or
|
||||||
|
their "pyroute2" equivalents) to configure the port representor virtual
|
||||||
|
function rates (maximum and minimum). This is similar to what is done in
|
||||||
|
ML2/SRIOV for a virtual function (VF).
|
||||||
|
|
||||||
|
A port representor (or a devlink port [2]_) is an API to expose the device
|
||||||
|
information; it is not a physical port but a representative of a hardware
|
||||||
|
port. The devlink port is connected to the local OVS switch; using the
|
||||||
|
"devlink" tool (or the "pyroute2" equivalent), it is possible to retrieve
|
||||||
|
the port information, including the physical function (PF) PCI address.
|
||||||
|
E.g.:
|
||||||
|
|
||||||
|
.. code::
|
||||||
|
|
||||||
|
$ devlink port show enp4s0f1_5 -jp
|
||||||
|
"port": {
|
||||||
|
"pci/0000:04:00.1/65542": {
|
||||||
|
"type": "eth",
|
||||||
|
"netdev": "enp4s0f1_5",
|
||||||
|
"flavour": "pcivf",
|
||||||
|
"pfnum": 1,
|
||||||
|
"vfnum": 5,
|
||||||
|
"splittable": false,
|
||||||
|
"function": {
|
||||||
|
"hw_addr": "fa:16:3e:f8:7b:10"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
With the PF PCI address, it is possible to retrieve the PF name, that will
|
||||||
|
be stored in:
|
||||||
|
|
||||||
|
.. code::
|
||||||
|
|
||||||
|
$ cat /sys/bus/pci/devices/$pf_pci_address/net
|
||||||
|
|
||||||
|
|
||||||
|
Using the ML2/SRIOV class ``PciDeviceIPWrapper``, it is possible to set the
|
||||||
|
maximum and minimum rates of a VF, knowing the VF index and the PF name.
|
||||||
|
|
||||||
|
|
||||||
|
OVN QoS information location
|
||||||
|
----------------------------
|
||||||
|
|
||||||
|
The OVN QoS information is stored in two different places (due to how QoS
|
||||||
|
is currently implemented in ML2/OVN and core OVN):
|
||||||
|
|
||||||
|
* The maximum bandwidth ("rate") and maximum burst ("burst") limits are stored
|
||||||
|
in the "Qos:bandwidth" dictionary:
|
||||||
|
|
||||||
|
.. code::
|
||||||
|
|
||||||
|
$ ovn-nbctl list Qos
|
||||||
|
_uuid : 376303c4-5290-4c4e-a489-d35894315199
|
||||||
|
action : {}
|
||||||
|
bandwidth : {rate=1000, burst=800}
|
||||||
|
direction : to-lport
|
||||||
|
external_ids : {"neutron:port_id"="89a81cc0-7b3e-473d-8c01-2539cf2a9a6a"}
|
||||||
|
match : "outport == \"89a81cc0-7b3e-473d-8c01-2539cf2a9a6a\""
|
||||||
|
priority : 2002
|
||||||
|
|
||||||
|
|
||||||
|
* The minimum bandwidth rate is stored in the "Logical_Switch_Port:options"
|
||||||
|
dictionary:
|
||||||
|
|
||||||
|
.. code::
|
||||||
|
|
||||||
|
$ ovn-nbctl list Logical_Switch_Port
|
||||||
|
_uuid : 502b3852-4a46-4fcb-9b49-03063cfc0b34
|
||||||
|
addresses : ["fa:16:3e:c4:b8:29 10.0.0.8"]
|
||||||
|
dhcpv4_options : a7e4cfb1-ef22-490a-9ffe-fea04de3fe1c
|
||||||
|
dhcpv6_options : []
|
||||||
|
dynamic_addresses : []
|
||||||
|
enabled : true
|
||||||
|
external_ids : {...(skipped)}
|
||||||
|
ha_chassis_group : []
|
||||||
|
name : "e6808371-c9ac-4015-94a3-7f16ac3fbb2d"
|
||||||
|
options : {mcast_flood_reports="true", qos_min_rate="1000",
|
||||||
|
requested-chassis=u20ovn}
|
||||||
|
parent_name : []
|
||||||
|
port_security : ["fa:16:3e:c4:b8:29 10.0.0.8"]
|
||||||
|
tag : []
|
||||||
|
tag_request : []
|
||||||
|
type : ""
|
||||||
|
up : true
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
OVN QoS events
|
||||||
|
--------------
|
||||||
|
|
||||||
|
The hardware offloaded QoS extension will configure the QoS setting on a
|
||||||
|
port reacting to the following events (please check the working POC [3]_
|
||||||
|
for more information):
|
||||||
|
|
||||||
|
* The local OVS interface creation: with this event, the OVN monitor will
|
||||||
|
store what ports are bound to the local instance. It will store the
|
||||||
|
Neutron port ID (stored in the "Interface.external_ids:iface-id" string)
|
||||||
|
and the OVS port name.
|
||||||
|
* The local OVS interface deletion: this event will trigger the QoS reset
|
||||||
|
and the interface local cache deletion.
|
||||||
|
* The OVN Southbound "Port_Binding" creation event: this event is received
|
||||||
|
after a port is created in a local OVS. If this event is received, that
|
||||||
|
will trigger the QoS update of the local port. It's worth mentioning that
|
||||||
|
this event happens always after the local OVS interface creation; that
|
||||||
|
means the OVN monitor has already registered that this port is bound
|
||||||
|
locally.
|
||||||
|
* The OVN Northbound "Logical_Switch_Port" register change: if minimum
|
||||||
|
bandwidth of a locally bound LSP changes, this event triggers the QoS
|
||||||
|
update.
|
||||||
|
* The OVN Northbound "QoS" register change: similar to the previous one
|
||||||
|
but affecting the maximum bandwidth limit.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
REST API Impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
This RFE does not introduce any API change.
|
||||||
|
|
||||||
|
|
||||||
|
Data Model Impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
This RFE does not introduce any model change.
|
||||||
|
|
||||||
|
|
||||||
|
Security Impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
Each monitor will have a connection to the local OVS database and the remotes
|
||||||
|
OVN Northbound and Southbound databases. The connection to the remote OVN
|
||||||
|
databases can have a severe impact on the load of the OVN database node (that
|
||||||
|
are usually the OpenStack controllers). This initial implementation will
|
||||||
|
subscribe to the following tables:
|
||||||
|
|
||||||
|
* Northbound: QoS, Logical_Switch_Port and Logical_Switch
|
||||||
|
* Southbound: Chassis, Encap, Port_Binding, Datapath_Binding and SB_Global
|
||||||
|
|
||||||
|
|
||||||
|
This performance impact should be reduced by:
|
||||||
|
|
||||||
|
* Reducing the number of agents on the node. The next step to be implemented
|
||||||
|
(out of scope in this RFE) is to move the OVN metadata agent functionality
|
||||||
|
to this new OVN Neutron agent. That will reduce the Southbound database
|
||||||
|
connections to one single agent.
|
||||||
|
* Find a way to store the minimum bandwidth information outside the Logical
|
||||||
|
Switch Port. By not subscribing to this table, the Northbound connection
|
||||||
|
will reduce the load notably. The Logical Switch Port is one of the most
|
||||||
|
populated and active ones; not locally caching nor receiving its updates
|
||||||
|
will reduce the impact on the OVN database.
|
||||||
|
|
||||||
|
|
||||||
|
Other Impact
|
||||||
|
------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignees:
|
||||||
|
Rodolfo Alonso Hernandez <ralonsoh@redhat.com> (IRC: ralonsoh)
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
* OVN monitor implementation and the hardware offloaded QoS extension.
|
||||||
|
* Tests and CI related changes.
|
||||||
|
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
* Unit/functional tests.
|
||||||
|
|
||||||
|
.. NOTE::
|
||||||
|
|
||||||
|
The hardware offloaded QoS extension requires specific hardware to
|
||||||
|
test this feature. Currently is not possible to implement any
|
||||||
|
tempest test on the CI.
|
||||||
|
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
User Documentation
|
||||||
|
------------------
|
||||||
|
|
||||||
|
* Information about how to configure and spawn the OVN monitor.
|
||||||
|
* Information about the hardware offloaded QoS extension.
|
||||||
|
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
.. [0] https://bugzilla.redhat.com/show_bug.cgi?id=2002406
|
||||||
|
.. [1] https://man7.org/linux/man-pages/man8/ip-link.8.html
|
||||||
|
.. [2] https://www.kernel.org/doc/html/latest/networking/devlink/devlink-port.html
|
||||||
|
.. [3] https://review.opendev.org/c/openstack/neutron/+/866480
|
Loading…
Reference in New Issue