Add Support for Smart NIC
Story: #2003346 Change-Id: I68c7334e1e377a85694a9791295683ff4fcbee35
This commit is contained in:
Moshe Levi
parent
a6daa1cd33
commit
f358fbdde9
412
specs/approved/support-smart-nic.rst
Executable file
412
specs/approved/support-smart-nic.rst
Executable file
@ -0,0 +1,412 @@
|
|||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
====================
|
||||||
|
Smart NIC Networking
|
||||||
|
====================
|
||||||
|
|
||||||
|
https://storyboard.openstack.org/#!/story/2003346
|
||||||
|
|
||||||
|
This spec describes proposed changes to Ironic to enable a generic,
|
||||||
|
vendor-agnostic, baremetal networking service running on smart NICs,
|
||||||
|
enabling baremetal networking with feature parity to the virtualization
|
||||||
|
use-case.
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
While Ironic today supports Neutron provisioned network connectivity for
|
||||||
|
baremetal servers through an ML2 mechanism driver, the existing support
|
||||||
|
is based largely on configuration of TORs through vendor-specific mechanism
|
||||||
|
drivers, with limited capabilities.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
There is a wide range of smart/intelligent NICs emerging on the market.
|
||||||
|
These NICs generally incorporate one or more general purpose CPU cores along
|
||||||
|
with data-plane packet processing acceleration, and can efficiently run
|
||||||
|
virtual switches such as OVS, while maintaining the existing interfaces to the
|
||||||
|
SDN layer.
|
||||||
|
|
||||||
|
The proposal is to extend Ironic to enable use of smart NICs to implement
|
||||||
|
generic networking services for Bare Metal servers. The goal is to enable
|
||||||
|
running the standard Neutron Open vSwitch L2 agent, providing a generic,
|
||||||
|
vendor-agnostic bare metal networking service with feature parity compared
|
||||||
|
to the virtualization use-case. The Neutron Open vSwitch L2 agent manages the
|
||||||
|
OVS bridges on the smart NIC.
|
||||||
|
|
||||||
|
In this proposal, we address two use-cases:
|
||||||
|
|
||||||
|
#. Neutron OVS L2 agent runs locally on the smart NIC.
|
||||||
|
|
||||||
|
This use case requires a smart NIC capable or running openstack control
|
||||||
|
services such as the Neutron OVS L2 agent. This use case strives to view
|
||||||
|
the smart NIC as an isolated hypervisor for the baremetal node, with the
|
||||||
|
smart NIC providing the services to the bare metal image running on the host
|
||||||
|
(as a hypervisor would provide services to a VM). While this spec initially
|
||||||
|
targets Neutron OVS L2 agent, the same implementation would naturally and
|
||||||
|
easily be extended to any other ML2 plugin as well as to additional
|
||||||
|
agents/services (for example exposing emulated NVMe storage devices
|
||||||
|
back-ended by a storage initiator on the smart NIC).
|
||||||
|
|
||||||
|
#. Neutron OVS L2 agent(s) run remotely and manages
|
||||||
|
the OVS bridges for all the baremetal smart NICs.
|
||||||
|
|
||||||
|
|
||||||
|
The enhancements for Neutron OVS L2 agent captured in [1]_, [2]_ and [3]_.
|
||||||
|
|
||||||
|
* Set the smart NIC configuration
|
||||||
|
|
||||||
|
smart NIC configuration includes the following:
|
||||||
|
|
||||||
|
#. extend the ironic port with is_smartnic field. (default to False)
|
||||||
|
#. smart NIC hostname - the hostname of server/smart NIC where the Neutron
|
||||||
|
OVS agent is running. (required)
|
||||||
|
#. smart NIC port id - the port name that needs to be plugged to the
|
||||||
|
integration bridge. B in the diagram below (required)
|
||||||
|
#. smart NIC SSH public key - ssh public key of the smart NIC
|
||||||
|
(required only for remote)
|
||||||
|
#. smart NIC OVSDB SSL certificate - OVSDB SSL of the OVS in smart NIC
|
||||||
|
(required only for remote)
|
||||||
|
|
||||||
|
The OVS ML2 mechanism driver will determine if the Neutron OVS Agent runs
|
||||||
|
locally or remotely based on smart NIC configuration passed from ironic.
|
||||||
|
The config attribute will be stored in the local_link_information of the
|
||||||
|
baremetal port.
|
||||||
|
|
||||||
|
In the scope of this spec the smart NIC config will be set manually by
|
||||||
|
the admin.
|
||||||
|
|
||||||
|
* Deployment Interfaces
|
||||||
|
|
||||||
|
Extending the ramdisk, direct, iscsi and ansible to support the smart nic
|
||||||
|
use-cases.
|
||||||
|
|
||||||
|
The Deployment Interfaces call network interface methods such as:
|
||||||
|
add_provisioning_network, remove_provisioning_network,
|
||||||
|
configure_tenant_networks, unconfigure_tenant_networks, add_cleaning_network
|
||||||
|
and remove_cleaning_network.
|
||||||
|
|
||||||
|
These network methods are currently ordinarily called when the baremetal is
|
||||||
|
powered down, ensuring proper network configuration on the TOR before booting
|
||||||
|
the bare metal.
|
||||||
|
|
||||||
|
smart NICs share the power state with the baremetal, requiring the baremetal
|
||||||
|
to be powered up before configuring the network. This leads to a potential
|
||||||
|
race where the baremetal boots and access the network prior to the network
|
||||||
|
being properly configured on the OVS within the smart NIC.
|
||||||
|
|
||||||
|
To ensure proper network configuration prior to baremetal boot, the
|
||||||
|
deployment interfaces will intermittently boot the baremetal into the BIOS
|
||||||
|
shell, providing a state where the ovs on the smart NIC may be configured
|
||||||
|
properly before rebooting the bare metal into the actual guest image or
|
||||||
|
ramdisk.
|
||||||
|
|
||||||
|
|
||||||
|
The following code for configure/unconfigure network:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
if task.driver.network.need_power_on(task):
|
||||||
|
old_power_state = task.driver.power.get_power_state(task)
|
||||||
|
if old_power_state == states.POWER_OFF:
|
||||||
|
# set next boot to BIOS to halt the baremetal boot
|
||||||
|
manager_utils.node_set_boot_device(task, boot_devices.BIOS,
|
||||||
|
persistent=False)
|
||||||
|
manager_utils.node_power_action(task, states.POWER_ON)
|
||||||
|
|
||||||
|
# ...
|
||||||
|
# call task.driver.network method(s)
|
||||||
|
# ...
|
||||||
|
|
||||||
|
if task.driver.network.need_power_on(task):
|
||||||
|
manager_utils.node_power_action(task, old_power_state)
|
||||||
|
|
||||||
|
The following methods in the deployment interface are calling to one or
|
||||||
|
more configure/unconfigure networks and should be updated with the logic
|
||||||
|
above.
|
||||||
|
|
||||||
|
* iscsi Deploy Interface
|
||||||
|
|
||||||
|
- iscsi_deploy::prepare
|
||||||
|
- iscsi_deploy::deploy
|
||||||
|
- iscsi_deploy::tear_down
|
||||||
|
|
||||||
|
* ansible Deploy Interface
|
||||||
|
|
||||||
|
- ansible/deploy::reboot_and_finish_deploy
|
||||||
|
- ansible/deploy::prepare
|
||||||
|
- ansible/deploy::tear_down
|
||||||
|
- ansible/deploy::prepare_cleaning
|
||||||
|
- ansible/deploy::tear_down_cleaning
|
||||||
|
|
||||||
|
* direct Interface
|
||||||
|
|
||||||
|
- agent::prepare
|
||||||
|
- agent::tear_down
|
||||||
|
- agent::deploy
|
||||||
|
- agent::rescue
|
||||||
|
- agent::unrescue
|
||||||
|
- agent_base_vendor::reboot_and_finish_deploy
|
||||||
|
- agent_base_vendor::_finalize_rescue
|
||||||
|
|
||||||
|
* RAM Disk Interface
|
||||||
|
|
||||||
|
- pxe::deploy
|
||||||
|
|
||||||
|
* Common cleaning methods
|
||||||
|
|
||||||
|
- deploy_utils::prepare_inband_cleaning
|
||||||
|
- deploy_utils::tear_down_inband_clean
|
||||||
|
|
||||||
|
* Network Interface
|
||||||
|
|
||||||
|
Extend the base `network_interface` with need_power_on -
|
||||||
|
return true if any ironic port attached to the node is a smart nic
|
||||||
|
|
||||||
|
Extend the ironic.common.neutron add_ports_to_network/
|
||||||
|
remove_ports_from_network methods for the smart NIC case:
|
||||||
|
|
||||||
|
* on add_ports_to_network and has smartNIC do the following:
|
||||||
|
|
||||||
|
- check neutron agent alive - verify that neutron agent is alive
|
||||||
|
- create neutron port
|
||||||
|
- check neutron port active - verify that neutron port is in active state
|
||||||
|
|
||||||
|
* on remove_ports_from_network and has smartNIC do the following:
|
||||||
|
|
||||||
|
- check neutron agent alive - verify that neutron agent is alive
|
||||||
|
- delete neutron port
|
||||||
|
- check neutron port is removed
|
||||||
|
|
||||||
|
|
||||||
|
* Neutron ml2 OVS changes:
|
||||||
|
|
||||||
|
- Introduce a new vnic_type for ``smart-nic``.
|
||||||
|
- Update the Neutron ml2 OVS to bind smart-nic vnic_type with
|
||||||
|
`binding:profile` smart NIC config.
|
||||||
|
|
||||||
|
* Neutron OVS agent changes:
|
||||||
|
|
||||||
|
Example of smart NIC model::
|
||||||
|
|
||||||
|
+---------------------+
|
||||||
|
| baremetal |
|
||||||
|
| +-----------------+ |
|
||||||
|
| | OS Server | | |
|
||||||
|
| | | | |
|
||||||
|
| | +A | | |
|
||||||
|
| +------|--------+ | |
|
||||||
|
| | | |
|
||||||
|
| +------|--------+ | |
|
||||||
|
| | OS SmartNIC | | |
|
||||||
|
| | +-+B-+ | | |
|
||||||
|
| | |OVS | | | |
|
||||||
|
| | +-+C-+ | | |
|
||||||
|
| +------|--------+ | |
|
||||||
|
+--------|------------+
|
||||||
|
|
|
||||||
|
|
||||||
|
A - port on the baremetal host.
|
||||||
|
B - port that represents the baremetal port in the smart NIC.
|
||||||
|
C - port that represents to the physical port in the smart NIC.
|
||||||
|
|
||||||
|
Add/Remove Port B to the OVS br-int with external-ids
|
||||||
|
|
||||||
|
In our case we will use the neutron OVS agent to plug the port on update
|
||||||
|
port event with the following external-ids: iface-id,iface-status, attached-mac
|
||||||
|
and node-uuid
|
||||||
|
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
* Delay the Neutron port binding (port binding means setting all the
|
||||||
|
OVSDB/Openflows config on the SmartNIC) to be performed by Neutron
|
||||||
|
later (once the bare metal is powered up). The problem with this
|
||||||
|
approach is that we have no guarantee of if/when the rules will be
|
||||||
|
programmed, and thus may inadvertently boot the baremetal while
|
||||||
|
the smart NIC is still programmed on the old network.
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
A new ``is_smartnic`` boolean field will be added to Port object.
|
||||||
|
|
||||||
|
|
||||||
|
State Machine Impact
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
The port REST API will be modified to support the new ``is_smartnic``
|
||||||
|
field. The field will be readable by users with the baremetal observer role
|
||||||
|
and writable by users with the baremetal admin role.
|
||||||
|
|
||||||
|
Updates to the is_smartnic field of ports will be restricted in the
|
||||||
|
same way as for other connectivity related fields (link local connection, etc.)
|
||||||
|
- they will be restricted to nodes in the ``enroll``, ``inspecting`` and
|
||||||
|
``manageable`` states.
|
||||||
|
|
||||||
|
Client (CLI) impact
|
||||||
|
-------------------
|
||||||
|
|
||||||
|
|
||||||
|
"ironic" CLI
|
||||||
|
~~~~~~~~~~~~
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
"openstack baremetal" CLI
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The openstack baremetal CLI will be updated to support getting and setting the
|
||||||
|
``is_smartnic`` field on ports.
|
||||||
|
|
||||||
|
RPC API impact
|
||||||
|
--------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Driver API impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Nova driver impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Ramdisk impact
|
||||||
|
--------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
* Smart NIC Isolation
|
||||||
|
|
||||||
|
Both use cases run infrastructure functionality on the smart NIC, with
|
||||||
|
the first use case also running control plane functionality.
|
||||||
|
|
||||||
|
This requires proper isolation between the untrusted bare metal host and the
|
||||||
|
smart NIC, preventing any/all direct or indirect access, both through the
|
||||||
|
network interface exposed to the host and through side channels such as the
|
||||||
|
platform BMC.
|
||||||
|
|
||||||
|
Such isolation is implemented by the smart NIC device and/or the hardware
|
||||||
|
platform vendor. There are multiple approaches for such isolation,
|
||||||
|
ranging from completely physical disconnection of the smart NIC from the
|
||||||
|
platform BMC to a platform with a trusted BMC wherein the BMC considers
|
||||||
|
the baremetal host an untrusted entity and restricts its capabilities/access
|
||||||
|
to the platform.
|
||||||
|
|
||||||
|
In the absence of such isolation, the untrusted baremetal tenant
|
||||||
|
may be able to gain access to the provisioning network, and in the second
|
||||||
|
may be able to compromise the control plane.
|
||||||
|
|
||||||
|
Proper isolation is dependent on the platform hardware/firmware, and cannot
|
||||||
|
be directly enforced/guaranteed by ironic. Users of smart NIC use case should
|
||||||
|
be made well aware of this via explicit documentation, and should be guided
|
||||||
|
to verify the proper isolation exists on their platform when enabling such
|
||||||
|
use cases.
|
||||||
|
|
||||||
|
* Security Groups
|
||||||
|
|
||||||
|
This will allow to use Neutron OVS agent pipeline. One of the features in the
|
||||||
|
pipeline is security groups which will enhance the security model when using
|
||||||
|
baremetal in a cloud.
|
||||||
|
|
||||||
|
* Security credentials
|
||||||
|
|
||||||
|
The node running the Neutron OVS agent (smart NIC or remote, according to use
|
||||||
|
case) should be configured with the message bus credentials for the Neutron
|
||||||
|
server.
|
||||||
|
|
||||||
|
In addition, for the second use case, the SSH public key and OVSDB SSL
|
||||||
|
certificate should be configured for the smart NIC port.
|
||||||
|
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
* Baremetal admin needs to update the SmartNIC config manually.
|
||||||
|
|
||||||
|
Scalability impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
hamdyk - hamdy@mellanox.com
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
* Update the Neutron network interface to populate the Smart NIC config from
|
||||||
|
the ironic port to the Neutron port `binding:profile` attribute.
|
||||||
|
* Update the network_interface and common.neutron as described above
|
||||||
|
* Update deployment interfaces as described above
|
||||||
|
* Documentation updates.
|
||||||
|
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
None, but the Neutron specs [1]_, [2]_ and [3]_ depend on this spec.
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
* Mellanox CI Jobs testing with Bluefield SmartNIC
|
||||||
|
|
||||||
|
Upgrades and Backwards Compatibility
|
||||||
|
====================================
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
* Update the multitenancy.rst with setting the SmartNIC config
|
||||||
|
* Document the security implications/guidelines under admin/security.rst
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
.. [1] https://review.openstack.org/#/c/619920/
|
||||||
|
|
||||||
|
.. [2] https://review.openstack.org/#/c/595402/
|
||||||
|
|
||||||
|
.. [3] https://review.openstack.org/#/c/595512/
|
||||||
1
specs/not-implemented/support-smart-nic.rst
Symbolic link
1
specs/not-implemented/support-smart-nic.rst
Symbolic link
@ -0,0 +1 @@
|
|||||||
|
../approved/support-smart-nic.rst
|
||||||
Loading…
Reference in New Issue
Block a user