Since the documentation PTI job config was implemented in zuul, the local docs builds no longer matched the builds performed by zuul in terms of treating warnings as errors. Adds flag to treat warnings as errors, and addresses the numerous formatting issues that resulted in errors. In some cases, the footnote no longer made sense to retain and was as such removed. Change-Id: Ie80d9c4480507c52fd0f3eee41fd6478e4c5c758
15 KiB
Physical Network Awareness
https://bugs.launchpad.net/ironic/+bug/1666009
Ironic ports and portgroups correspond to the physical NICs that are available on baremetal nodes. Neutron ports are logical ports attached to tenant and provider networks. When booting baremetal nodes in a system with multiple physical networks, the mapping between logical and physical ports must account for the physical network connectivity of the system.
This feature aims to make ironic aware of the
physical_network
attribute of neutron networks and network
segments.
Problem description
Physical Networks
A neutron port on a provider network or network segment is associated with a physical network. The physical network concept is used to specify the physical connectivity of networks and network segments in the system.
There are many reasons why an operator might use multiple physical networks, including:
- routed provider networks1
- security, through physical segregation of traffic
- redundancy, through independence of multiple networks
- traffic isolation (e.g. storage vs. application)
- different characteristics (bandwidth, latency, reliability)
- different technologies (Ethernet, Infiniband, Omnipath, etc.)
It is possible that some or all nodes may be connected to more than one physical network. OpenStack adoption is growing rapidly in the scientific computing community, an area in which the use of nodes connected to multiple physical networks is prevalent.
For context, we provide some examples of how the physical network
attribute is used in other OpenStack components. The neutron ML2 Open
vSwitch agent is made physical network-aware via the
[OVS] bridge_mappings
configuration option. This option
maps physical network names to Open vSwitch bridges that have a physical
network interface as a port. This option is used for flat and VLAN
network types, and is taken into consideration when binding ports to
network segments in the neutron server ML2 driver. This acts both to
ensure that physical connectivity allows for the requested logical
topology, and also supports hosts being connected to multiple physical
networks.
A similar mapping exists in nova for pass-through of PCI physical
devices or SR-IOV virtual functions via the
[DEFAULT] pci_passthrough_whitelist
configuration
option.
Physical Networks in Ironic
An ironic port is connected to a physical network. As portgroups are a layer-2 construct, all ports in a portgroup should be in the same physical network. If a neutron port is mapped to a port or portgroup and is attached to a neutron network or network segment on a different physical network, there will be no connectivity between the bare metal node's NIC and other neutron ports on the network. This is perhaps most obvious when it results in a failure to acquire a DHCP lease for the interface.
The mapping between logical ports in neutron and physical ports and portgroups in ironic has always been somewhat unpredictable2. The ironic-neutron integration work added support for local link information for ports3. In the interface attach/detach API work4 ironic moved the responsibility for attaching virtual interfaces from nova to ironic. In both of these features physical network-awareness was seen as out of scope.
Currently when a virtual interface is attached to a node, the procedure used to map it to an ironic port or portgroup is roughly as follows:
- if there is a free portgroup, select one
- else if there is a free port with PXE enabled, select one
- else if there is a free port with PXE disabled, select one
- else fail
This algorithm takes no account of the physical network to which the ports and portgroups are connected, and consequently can result in invalid mappings. Further, there is currently no standard way in ironic to specify the physical network to which a port or portgroup is connected.
Provisioning and Cleaning Networks
The ironic pluggable network provider5 work added support for attaching nodes to dedicated provisioning and cleaning networks during provisioning and cleaning operations respectively. Currently, all ironic ports and portgroups with PXE enabled are attached to the provisioning or cleaning network. While this ensures that the node has a port on the provisioning or cleaning physical network, it may result in unnecessary neutron ports being created if some of the ironic ports or portgroups are connected to a different physical network. In practice this can be avoided by disabling PXE on ports where it is not required.
Scheduling
In a system with multiple physical networks where not all nodes are connected to every physical network, it becomes possible for a user to request a logical network topology that cannot be realised. Without awareness of the physical networks that each ironic node is connected to, nova cannot reliably schedule instances to ironic nodes. This problem is considered beyond the scope of this spec.
Proposed change
There are four parts to the proposed solution to this problem.
- It must be possible to specify the physical network to which an ironic port is connected.
- The ironic network interfaces must account for the physical network of an ironic node's ports and portgroups when attaching tenant virtual interfaces.
- When attaching a node to a provisioning or cleaning network, neutron ports should be created only for ironic ports and portgroups on the same physical network as the provisioning or cleaning network.
- The binding profiles of attached neutron ports should be updated with the physical network of the ironic port or portgroup to which the port was mapped.
Tagging Ports
There are a few options for how ports might be tagged with a physical network:
- a new attribute of the
local_link_connection
field - a new attribute of the
extra
field - a new first class field
Reusing an existing field would certainly be easier to implement than
adding a new one, but OpenStack history has frequently shown that buried
fields often end up needing to become first class citizens. A relevant
example here is the provider:physical_network
extension
field on neutron networks, which later became a first class field on
network segments6. Further, if ironic intends to
support physical network-aware scheduling in future, the ability to
efficiently filter ports by their physical network may be advantageous.
This spec therefore proposes to add a new first class
physical_network
field to ironic's Port
object. For backwards compatibility, this field will be optional.
The process of mapping physical networks to ironic ports is out of scope for ironic. This could be done either through a manual procedure or through an automated process using information gathered during a hardware introspection process. For example, if using ironic inspector to perform introspection it would be possible to create an introspection plugin7 that maps switch IDs discovered via LLDP to physical networks.
Portgroups
The physical network of a portgroup will be determined through the physical network of its constituent ports. All ports in the portgroup must have the same physical network, and this will be enforced in the ironic API when creating and updating ports.
This has the unfortunate consequence of making it rather unwieldy to update the physical network of the ports in a portgroup, since the ports must be removed from the portgroup while their physical network is updated. This may be improved upon in future through the use of a virtual physical network field in the portgroups API that allows simultaneous update of the physical network field of all the ports in the group.
Mapping Logical Ports to Physical Ports
In order to account for physical network connectivity, the virtual
interface attachment algorithm must determine the physical networks that
the neutron port being attached can be bound to. This information is
available via the neutron API as the physical_network
field
on network segments in the port's network or as
provider:physical_network
on the port's network.
The virtual interface attachment mapping algorithm will be modified to the use the following set of criteria listed in order of descending priority:
- reject ports and portgroups with a non-null physical network that is different than all of the network's physical networks
- prefer ports and portgroups with a non-null physical network to ports with a null physical network
- prefer portgroups to ports
- prefer ports with PXE enabled to ports with PXE disabled
This algorithm provides backwards compatibility for environments in
which the port(s) and/or portgroup(s) associated with the ironic node do
not have a physical_network
property configured.
Provisioning and Cleaning Networks
In ironic network drivers that support network flipping for provisioning and cleaning operations, we will create neutron ports only for those ironic ports and portgroups that have PXE enabled and are on the same physical network as the provisioning or cleaning network in question, or do not have a physical network specified.
Neutron Port Binding Profiles
When attaching virtual interfaces to physical or virtual functions of
PCI network devices, nova sets a physical_network
attribute
in the binding:profile
field of the neutron port. Further
research is required to determine what effect it would have if ironic
were to do the same.
Alternatives
We could continue to use an unpredictable mapping between logical ports and physical ports. This limits the use of ironic to environments in which there is only one physical network.
We could continue with the existing mapping algorithm in ironic but
provide neutron with the information required to determine whether a
mapping is valid from the local_link_connection
binding
information. Ironic would then be modified to retry interface attachment
with a different neutron port if neutron determined the mapping to be
invalid. This method would be inefficient due to the retries
necessary.
We could avoid the need to tag ironic ports with a physical network
by providing a mechanism to map from the information in their
local_link_connection
fields to a physical network. This
would require either an addition to ironic's data model to support
Switch objects or a new neutron API providing a lookup from switch ID to
physical network.
Data model impact
A new physical_network
field will be added to Port
object. In neutron the Segment
object's
physical_network
field is defined as
sqlalchemy.String(64)
, so the same will be used in
ironic.
State Machine Impact
None
REST API impact
The port REST API will be modified to support the new
physical_network
field. The field will be readable by users
with the baremetal observer role and writable by users with the
baremetal admin role. If the port is a member of a portgroup, the API
will enforce that all ports in the portgroup have the same value in
their physical network field.
Updates to the physical network field of ports will be restricted in
the same way as for other connectivity related fields (link local
connection, etc.) - they will be restricted to nodes in the
enroll
, inspecting
and manageable
states.
The API microversion will be bumped.
Client (CLI) impact
"ironic" CLI
The ironic CLI will not be updated.
"openstack baremetal" CLI
The openstack baremetal CLI will be updated to support getting and
setting the physical_network
field on ports.
RPC API impact
None
Driver API impact
None
Nova driver impact
None
Ramdisk impact
None
Security impact
This change should increase the potential security level of an ironic bare metal cloud by supporting multiple segregated physical networks and honoring the physical network restrictions assigned by the operator.
Other end user impact
None
Scalability impact
None
Performance Impact
None
Other deployer impact
In order to make use of this feature, deployers must tag ironic ports and portgroups with the physical network to which they are attached. This implies that they must have a mechanism to correctly determine this information.
Developer impact
None
Implementation
Assignee(s)
mgoddard
Work Items
- Modify the ironic port model to include a
physical_network
field. - Modify the ironic ports REST API to support the
physical_network
field. - Modify the openstack baremetal CLI to support the
physical_network
field. - Modify the ironic
VIFPortIDMixin
plugin with the new port mapping algorithm. - Modify the ironic
NeutronNetwork
network driver to be physical network-aware when creating neutron ports for cleaning and provisioning. - Modify the ironic network drivers to add the physical network to neutron ports' binding profiles.
- Add support for multiple (virtual) physical networks to DevStack.
- Update the ironic developer documentation to cover the use of physical networks.
Dependencies
None
Testing
Support will be added to DevStack for ironic environments with multiple (virtual) physical networks.
Upgrades and Backwards Compatibility
The proposed data model and algorithm changes are backwards
compatible. A database migration will be provided to add the
physical_network
field to existing ports with a null
value.
Documentation Impact
The ironic developer documentation will be updated to cover the use of this feature.