Blueprint: support-infiniband-network
Change-Id: Idb464cc6a9b17a7c10e0594dfff9e02997abc8a3
This commit is contained in:
Aviram Bar-Haim
parent
32f5fd8f43
commit
085b54b750
325
specs/6.1/support-infiniband-network.rst
Normal file
325
specs/6.1/support-infiniband-network.rst
Normal file
@ -0,0 +1,325 @@
|
||||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
==============================================================================
|
||||
Support discovery over Mellanox Infiniband network using interface driver data
|
||||
==============================================================================
|
||||
|
||||
https://blueprints.launchpad.net/fuel/+spec/support-infiniband-network
|
||||
|
||||
In Fuel 5.1, support for Mellanox high performance Ethernet virtualization
|
||||
and storage features has been added.
|
||||
|
||||
Starting with version 6.0, Fuel supports a Pluggable architecture.
|
||||
|
||||
This blueprint suggests to publish interface driver and networks
|
||||
allocation data, for new network / interface oriented features.
|
||||
This will also ease the process of exporting core functionality
|
||||
of partners into plugins and allowing them to access the required
|
||||
nodes model information.
|
||||
|
||||
Moreover, this blueprint describes Mellanox Infiniband drivers installation
|
||||
for the bootstrap discovery stage.
|
||||
|
||||
InfiniBand (abbreviated IB) is a computer network communications link
|
||||
used in high-performance computing featuring very high throughput and
|
||||
very low latency.
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
There are two main problems that this blueprint suggests to solve:
|
||||
|
||||
1. Currently in Fuel, Installation over Infiniband mode is not supported, and
|
||||
in particular bootstrap discovery and network verification over
|
||||
infiniband based network.
|
||||
|
||||
Some of the high performance computing users, prefer to use Infinband due
|
||||
to its highest throughput and lowest latency preferences.
|
||||
|
||||
A POC ISO already exists, and shows significant performance improvement
|
||||
from the 56Gb Ethernet solution.
|
||||
|
||||
Adding Infiniband network support will allow users use IPOIB Virtual
|
||||
Functions with higher performance, and will bring them closer to the wire
|
||||
speed.
|
||||
|
||||
#. Network interfaces information, that was accessible via the nodes model
|
||||
in Fuel web serializers, is now missing, for implementing network oriented
|
||||
plugins.
|
||||
|
||||
Moreover, in this blueprint, extensive network information publishment will
|
||||
be added, such as discovering port bus ID and vendor drivers information.
|
||||
This will allow to add more vendor and network based features.
|
||||
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
Support nodes discovery over a prepared Infiniband network via Fuel
|
||||
(with a running and pre-configured switch IB and Subnet Manager).
|
||||
|
||||
The change should be transparent to users that already knows how to install
|
||||
Mellanox components via Fuel installer, and be as standard as possible, without
|
||||
harming the Infiniband performance.
|
||||
|
||||
The Fuel master will discover and use EIPOIB (Ethernet IP Over InfiniBand)
|
||||
interfaces for the network roles.
|
||||
|
||||
In the complementary Fuel plugin for Infiniband, OVS will run over EIPOIB
|
||||
interfaces that was discovered in the bootstrap stage.
|
||||
|
||||
Infiniband interfaces (IPOIB) will be used only for RDMA (Remote Direct
|
||||
Memory Access), and for the guests Virtual Functions (Virtual Interfaces)
|
||||
in case of SR-IOV.
|
||||
|
||||
The network mode will be calculated in the backend using the new interfaces
|
||||
driver data - "eth_ipoib" for Infiniband or "mlx4_en" for Ethernet (RoCE).
|
||||
|
||||
**The main changes:**
|
||||
|
||||
*Packages:*
|
||||
|
||||
1. New Rethtool with bus and driver info (add a new cstruct for ethtool -i
|
||||
<interface>).
|
||||
#. Adding kernel-ib driver to Fuel repository.
|
||||
|
||||
*Fuel-main (Bootstrap):*
|
||||
|
||||
1. Adding bus and drivers info to nailgun agent, in order to
|
||||
discover ports vendor, protocol and parents / bus ID.
|
||||
#. Extracting the IB drivers.
|
||||
#. Changing Mellanox drivers loading to auto link mode (Eth or IB).
|
||||
#. Start eth_ipoib and ib_ipoib kernel modules if mlx4_core is loaded.
|
||||
#. Adding daemon to convert SM PKEYs to VLANs for verify network,
|
||||
and execute it if Mellanox eth_ipoib kernel module started.
|
||||
|
||||
*Fuel-web:*
|
||||
|
||||
1. Adding bus and drivers info to nailgun NIC db and model.
|
||||
#. Publish interfaces discovered driver and bus info data (under
|
||||
interfaces).
|
||||
#. Publish the physical interface and VLAN tag for each network role (private,
|
||||
storage, management, admin) to astute.yaml (under roles).
|
||||
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
* Parsing ethtool output for drivers data in nailgun agent - very inelegant.
|
||||
* Not using EIPOIB for Eth virtualization over IB - EIPOIB is the only method
|
||||
that currently supports OVS/Linuxbridge VLAN rules with PKEYs conversion.
|
||||
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
Adding bus_info and driver columns to the NIC model, as described at the
|
||||
proposed change section under Fuel-web.
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
The parameters 'bus_info' and 'driver' are added into REST responses for Node
|
||||
network interface parameters.
|
||||
|
||||
Upgrade impact
|
||||
--------------
|
||||
|
||||
Upgrade impact in this case is more of Plugin upgrade impact, so it is out
|
||||
of scope in this spec.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
None
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
Will add driver and bus_info parameter to the nailgun interfaces report
|
||||
(in the Fuel web UI).
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
No Such, but it will enable to support the Infiniband plugin that increases
|
||||
default OVS performance dramatically, and increase 56Gb link performance
|
||||
significantly from Eth mode, in VMs connection and Cinder volumes.
|
||||
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
Adding driver and bus_info may enable Fuel developers to add new features
|
||||
based on the port vendor identification.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
aviramb <aviramb@mellanox.com>
|
||||
|
||||
Other contributors:
|
||||
gilmeir-d <gilmeir@mellanox.com>
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
*Packages:*
|
||||
|
||||
1. New Rethtool with bus and driver info (add a new cstruct for ethtool -i
|
||||
<interface>):
|
||||
|
||||
* Change /usr/lib/ruby/gems/1.8/gems/rethtool-0.0.3/lib/rethtool/
|
||||
interface_settings.rb to have bus and driver info getters.
|
||||
* Change /usr/lib/ruby/gems/1.8/gems/rethtool-0.0.3/lib/rethtool.rb
|
||||
to include drivers CMD ID.
|
||||
* Add /usr/lib/ruby/gems/1.8/gems/rethtool-0.0.3/lib/rethtool/ethtool_cmd.rb
|
||||
with the driver CStruct.
|
||||
* Example code in
|
||||
`this example <https://github.com/avirambh/rethtool/commit/dce5d747c1ea654ff6c4430a2fa4c6337f7e9527>`_
|
||||
.
|
||||
|
||||
2. Add kernel-ib rpm to the bootstrap image:
|
||||
|
||||
* Compile OFED on the bootstrap kernel and add the OFED kernel-ib rpm to
|
||||
the Nailgun repo.
|
||||
|
||||
*Fuel-main (Bootstrap):*
|
||||
|
||||
1. Adding bus and driver information to nailgun agent and rethtool, in order to
|
||||
discover ports vendor, protocol and parents / bus ID:
|
||||
|
||||
* Replace rethtool package in repo with the new package that includes
|
||||
the drivers functions (already built it for the POC ISO using Fuel packaging
|
||||
spec in
|
||||
`Fuel repository <http://fuel-repository.mirantis.com/repos/centos-fuel-6.0-stable-916/centos/noarch/>`_
|
||||
).
|
||||
|
||||
* Add to fuel-web/bin/agent interfaces report the bus_info and driver rethtool
|
||||
functions call.
|
||||
|
||||
2. Adding IB drivers:
|
||||
|
||||
* Change /bootstrap/module.mk to extract kernel-ib package and its
|
||||
dependencies.
|
||||
|
||||
3. Changing mellanox drivers loading to auto link mode:
|
||||
|
||||
* Remove /bootstrap/sync/etc/modprobe.d/mlnx4_core.conf
|
||||
|
||||
4. Start EIPOIB and IPOIB kernel modules with mlx4_core:
|
||||
|
||||
* Create /bootstrap/sync/etc/modprobe.d/eth_ipoib.conf
|
||||
|
||||
5. Adding a daemon to create SM PKEYs conversion for verify network:
|
||||
|
||||
* Adding /bootstrap/sync/usr/bin/init_eipoib.sh
|
||||
* Implement A daemon that reads once in 30 seconds the PKEYs configured at the
|
||||
SM machine (published in /sys/class/infiniband/<HCA>/ports/<port>/pkeys/),
|
||||
and verifies all pkeys are converted to vlans on all ports.
|
||||
* Execute this daemon in bootstrap/sync/etc/modprobe.d/eth_ipoib.conf after
|
||||
ib_ipoib and eth_ipoib, if mlx4_ib is executed (mlx4_ib and mlx4_en are
|
||||
executed if mlx4_core kernel module is executed, which occurs only if
|
||||
Mellanox card has been found - as in Fuel 5.1).
|
||||
|
||||
6. Adding script to convert SM pkeys to VLANs for verify network, and append
|
||||
it to rc.local if Mellanox kmods started:
|
||||
|
||||
* Change /bootstrap/sync/etc/modprobe.d/eth_ipoib.conf
|
||||
* Increase buffers in /bootstrap/sync/etc/modprobe.d/ipoib.conf
|
||||
(for large amount of PKEYs)
|
||||
|
||||
7. Example code can be found in `Mellanox fuel-main fork <https://github.com/Mellanox/fuel-main/commit/6788f44acbcdae06e5f77a1fa4350808b5bbe5fa>`_.
|
||||
|
||||
*Fuel-web:*
|
||||
|
||||
1. Adding bus and driver info to nailgun db and model:
|
||||
|
||||
* Change bin/agent to call drivers functions (int.driver and int.bus_info),
|
||||
as described in the Bootstrap section.
|
||||
* Change the relevant upgrade/downgrade modules in
|
||||
nailgun/nailgun/db/migration/alembic_migrations/versions/
|
||||
to have the driver and bus info columns.
|
||||
* Change nailgun/nailgun/db/sqlalchemy/models/node.py NodeNICInterface to
|
||||
include driver and bus_info columns.
|
||||
* Change nailgun/nailgun/objects/serializers/node.py
|
||||
NodeInterfacesSerializer nic_fields dict to include driver and bus_info.
|
||||
|
||||
2. Publish interfaces discovered driver and bus info data:
|
||||
|
||||
* Add to the networking dict of deployment serializers the driver and bus ID
|
||||
of the nodes interfaces, under the interfaces dict.
|
||||
|
||||
3. Publish the physical interface and VLAN tag for each network role (private,
|
||||
storage, management, admin) to astute.yaml:
|
||||
|
||||
* Add to the networking dict of deployment serializers the physical interfaces
|
||||
for each role and its VLAN tag (as selected in the Fuel UI).
|
||||
This data is required for SR-IOV vNIC alocation and for establishing RDMA
|
||||
connection on the storage interface parent / probbed interface
|
||||
(OVS bridge/LB does not support RDMA).
|
||||
|
||||
4. Example code can be found in `Mellanox fuel-web fork <https://github.com/Mellanox/fuel-web/commit/3386c6cc787d2d0ae48a386023b8b5c1998c0eeb>`_ (serializers and UI code are not relevant in this link).
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
None
|
||||
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
1. Integration tests for Fuel-Web:
|
||||
|
||||
* Integration tests for testing creation of a node with driver and bus_info
|
||||
parameters.
|
||||
* Integration tests for testing access to the node driver and bus_info
|
||||
parameters after creation.
|
||||
|
||||
2. CI and Verificaiton an Mellanox Lab:
|
||||
|
||||
* Nodes discovery over Infiniband network.
|
||||
* Network verification over Infiniband network.
|
||||
* Host and switch based SM.
|
||||
* Large number of PKEYs.
|
||||
* Verifying that bootstrap is loaded without Mellanox drivers if now Mellanox
|
||||
HW has been discovered.
|
||||
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
1. Instructions for "How to configure SM" will be added to the Planning guide.
|
||||
#. Instructions for "Network drivers identification" will be added to the
|
||||
User guide.
|
||||
#. Instructions for "How to install Mirantis Openstack with Infiniband Network"
|
||||
will be added to the Mellanox community, similarly to `this post <https://community.mellanox.com/docs/DOC-2036>`_
|
||||
that has been made to the 5.1 based Fuel IB POC.
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
* Infiniband network - http://en.wikipedia.org/wiki/InfiniBand
|
||||
* Configuring EIPOIB interfaces - https://community.mellanox.com/docs/DOC-1316
|
||||
|
||||
Loading…
Reference in New Issue
Block a user