diff --git a/doc/source/app-openvswitch-dpdk.rst b/doc/source/app-openvswitch-dpdk.rst index a2ce009e..fdcfc5d3 100644 --- a/doc/source/app-openvswitch-dpdk.rst +++ b/doc/source/app-openvswitch-dpdk.rst @@ -27,7 +27,7 @@ We recommend that you read the following documents before proceeding: * Open vSwitch with DPDK datapath: ``_ * Getting the best performance from DPDK: - ``_ + ``_ * OpenStack documentation on hugepages: ``_ @@ -156,11 +156,13 @@ The remaining cores can be reserved for virtual machine instances. In this example, the breakdown would resemble the following: +``` | Reserved Cores | Purpose | node0 | node1 | | ---------------------- | --------------------- | --------- | ----- | | 0,8,16,24 | Host Operating System | 0,16 | 8,24 | | 1,9,17,25 | DPDK PMDs | 1,17 | 9,25 | | 2-7,18-23 | Virtual Machines | 2-7,18-23 | N/A | +``` The variables are overrides used to define this configuration are discussed in the following sections. @@ -280,9 +282,11 @@ to a hexidecimal mask and defined using the ``ovs_dpdk_lcore_mask`` override. To convert to a hex mask you must first establish the binary mask of chosen cores using the following table: +``` | 31 | 30 | . | 24 | 23 | . | 17 | 16 | 15 | . | 9 | 8 | 7 | . | 1 | 0 | | -- | -- | - | -- | -- | - | -- | -- | -- | - | -- | -- | -- | - | -- | -- | | 0 | 0 | . | 1 | 0 | . | 0 | 1 | 0 | . | 0 | 1 | 0 | . | 0 | 1 | +``` The ellipses represent cores not shown. The binary mask for cores 0,8,16,24 can be determined in the following way: @@ -302,9 +306,11 @@ file or ``openstack_user_config.yml``: The mask for cores 1,9,17,25 reserved for DPDK PMDs can be determined in a similar fashion. The table would resemble the following: +``` | 31 | 30 | . | 25 | 24 | . | 17 | 16 | 15 | . | 9 | 8 | 7 | . | 1 | 0 | | -- | -- | - | -- | -- | - | -- | -- | -- | - | -- | -- | -- | - | -- | -- | | 0 | 0 | . | 1 | 0 | . | 1 | 0 | 0 | . | 1 | 0 | 0 | . | 1 | 0 | +``` The ellipses represent cores not shown. The binary mask for cores 1,9,17,254 can be determined in the following way: @@ -401,6 +407,12 @@ Specify provider network definitions in your ``/etc/openstack_deploy/openstack_user_config.yml`` that define one or more Neutron provider bridges and related configuration: +.. note:: + + Bridges specified here will be created automatically. If *network_interface* + is defined, the interface will be placed into the bridge automatically + as a *DPDK-accelerated* interface. + .. code-block:: yaml - network: @@ -409,13 +421,55 @@ Neutron provider bridges and related configuration: type: "vlan" range: "101:200,301:400" net_name: "physnet1" + network_interface: "eno49" group_binds: - neutron_openvswitch_agent -.. note:: +A *DPDK-accelerated* **bond** interface can be created by specifying a list +of member interfaces using `network_bond_interfaces`. The bond port will +be created automatically and added to the respective bridge in OVS: - A single DPDK interface can be connected to an OVS provider bridge, and - must be done using the ``ovs-vsctl`` command as a post-installation step. +.. code-block:: yaml + + - network: + container_bridge: "br-provider" + container_type: "veth" + type: "vlan" + range: "101:200,301:400" + net_name: "physnet1" + network_bond_interfaces: + - "0000:04:00.0" + - "0000:04:00.1" + group_binds: + - neutron_openvswitch_agent + +Additional OVS bond parameters can be specified using the following keys: + +* bond_mode (Default: active-backup) +* lacp (Default: off) +* bond_downdelay (Default: 100) +* bond_updelay (Default: 100) + +.. code-block:: yaml + + - network: + container_bridge: "br-provider" + container_type: "veth" + type: "vlan" + range: "101:200,301:400" + net_name: "physnet1" + network_bond_interfaces: + - "0000:04:00.0" + - "0000:04:00.1" + bond_mode: balance-tcp + lacp: active + bond_downdelay: 200 + bond_updelay: 200 + group_binds: + - neutron_openvswitch_agent + +For more information on possible values, visit: +``_ Set the following user variables in your ``/etc/openstack_deploy/user_variables.yml`` to enable the Open vSwitch driver @@ -430,7 +484,9 @@ and DPDK support: ovs_dpdk_support: True # Add these overrides or set on per-host basis in openstack_user_config.yml - ovs_dpdk_pci_addresses: "0000:03:00.0" + ovs_dpdk_pci_addresses: + - "0000:04:00.0" + - "0000:04:00.1" ovs_dpdk_lcore_mask: 1010101 ovs_dpdk_pmd_cpu_mask: 2020202 ovs_dpdk_socket_mem: "1024,1024" @@ -442,20 +498,29 @@ and DPDK support: Post-installation ~~~~~~~~~~~~~~~~~ -Once the playbooks have been run and OVS/DPDK has been configured, it will be +Once the playbooks have been run and OVS/DPDK has been configured, it may be necessary to add a physical interface to the provider bridge before networking -can be fully established. +can be fully established *if* `network_interface` or `network_bond_interfaces` +have not been defined. On compute nodes, the following command can be used to attach a NIC port -``0000:03:00.0`` to the provider bridge ``br-provider``: +``0000:04:00.0`` to the provider bridge ``br-provider``: .. code-block:: console - ovs-vsctl add-port br-provider 0000:03:00.0 -- set Interface 0000:03:00.0 type=dpdk options:dpdk-devargs=0000:03:00.0 + ovs-vsctl add-port br-provider 0000:04:00.0 -- set interface 0000:04:00.0 type=dpdk options:dpdk-devargs=0000:04:00.0 -The command can be adjusted according to your configuration. +Additionally, it may be necessary to make post-installation adjustments to +interface queues or other parameters to avoid errors within Open vSwitch: + +.. code-block:: console + + ovs-vsctl set interface 0000:04:00.0 options:n_txq=5 + ovs-vsctl set interface 0000:04:00.0 options:n_rxq=5 + +The command(s) can be adjusted according to your configuration. .. warning:: - Adding multiple ports to the bridge may result in bridging loops unless - bonding is configured. DPDK bonding is outside the scope of this guide. + Adding multiple ports to a bridge may result in bridging loops unless + bonding is configured. diff --git a/library/provider_networks b/library/provider_networks index cfe15cdf..5099762f 100644 --- a/library/provider_networks +++ b/library/provider_networks @@ -23,7 +23,7 @@ from ansible.module_utils.basic import AnsibleModule DOCUMENTATION = """ --- module: provider_networks -version_added: "1.8.6" +version_added: "1.8.7" short_description: - Parse a list of networks and return data that Ansible can use description: @@ -116,6 +116,22 @@ EXAMPLES = """ # group_binds: # - neutron_openvswitch_agent # - network: +# container_bridge: "br-provider" +# container_type: "veth" +# container_interface: "eth11" +# network_bond_interfaces: +# - "0000:02:00.0" +# - "0000:02:00.1" +# bond_mode: balance-tcp +# bond_updelay: 100 +# bond_downdelay: 100 +# lacp: active +# type: "vlan" +# range: "1:1, 101:101" +# net_name: "physnet1" +# group_binds: +# - neutron_openvswitch_agent +# - network: # container_bridge: "br-storage" # container_type: "veth" # container_interface: "eth2" @@ -157,6 +173,14 @@ EXAMPLES = """ # "flat:brx-eth12", # "vlan:brx-eth11" # ], +# "network_bond_interfaces_mappings": [{ +# 'bridge': 'br-provider', +# 'interfaces': ['0000:02:00.0', '0000:02:00.1'], +# 'bond_mode': 'balance-tcp', +# 'lacp': 'active', +# 'bond_updelay': 100, +# 'bond_downdelay': 100 +# }], # "network_sriov_mappings": "physnet1:p1p1,physnet1:p1p2", # "network_sriov_mappings_list": [ # "physnet1:p1p1" @@ -202,6 +226,7 @@ class ProviderNetworksParsing(object): self.network_types = list() self.network_sriov_mappings = list() self.network_interface_mappings = list() + self.network_bond_interfaces_mappings = list() def load_networks(self, provider_networks, is_metal=False, bind_prefix=None, group_names=None): @@ -299,6 +324,29 @@ class ProviderNetworksParsing(object): ) ) + # Builds a list of provider bridge to physical + # interface (bond member) mappings and is used + # when constructing OVS bonds + if 'network_bond_interfaces' in net['network']: + self.network_bond_interfaces_mappings.append({ + 'bridge': net['network'][ + 'container_bridge' + ], + 'interfaces': net['network'][ + 'network_bond_interfaces' + ], + 'bond_mode': net['network'].get( + 'bond_mode', 'active-backup' + ), + 'lacp': net['network'].get('lacp', 'off'), + 'bond_updelay': net['network'].get( + 'bond_updelay', 100 + ), + 'bond_downdelay': net['network'].get( + 'bond_downdelay', 100 + ) + }) + # SR-IOV interface mappings if 'sriov_host_interfaces' in net['network']: host_interfaces = \ @@ -367,7 +415,12 @@ def main(): 'network_interface_mappings': ','.join( pnp.network_interface_mappings ), - 'network_interface_mappings_list': pnp.network_interface_mappings + 'network_interface_mappings_list': pnp.network_interface_mappings, + 'network_bond_interfaces_mappings': ','.join( + map(str, pnp.network_bond_interfaces_mappings) + ), + 'network_bond_interfaces_mappings_list': + pnp.network_bond_interfaces_mappings } module.exit_json(changed=True, **resp) diff --git a/releasenotes/notes/neutron-dpdk-bonds-4dd98fc0b341ebfb.yaml b/releasenotes/notes/neutron-dpdk-bonds-4dd98fc0b341ebfb.yaml new file mode 100644 index 00000000..c2630104 --- /dev/null +++ b/releasenotes/notes/neutron-dpdk-bonds-4dd98fc0b341ebfb.yaml @@ -0,0 +1,10 @@ +--- +features: + - | + The ``provider_networks`` library has been updated to support the + definition of bond member interfaces that can automatically be added as + bond ports to OVS provider bridges setup during a deployment. This + feature is currently limited to DPDK-based deployments. To activate this + feature, add the ``network_bond_interfaces`` key to the respective provider + network definition in ``openstack_user_config.yml``. For more information, + refer to the latest Open vSwitch w/ DPDK deployment guide. diff --git a/tasks/providers/setup_ovs_dpdk.yml b/tasks/providers/setup_ovs_dpdk.yml index 672d4693..93b67997 100644 --- a/tasks/providers/setup_ovs_dpdk.yml +++ b/tasks/providers/setup_ovs_dpdk.yml @@ -28,6 +28,7 @@ dest: "/etc/dpdk/interfaces" owner: "root" group: "root" + mode: "0640" when: - neutron_services['neutron-openvswitch-agent']['group'] in group_names - '"nova_compute" in group_names' @@ -38,6 +39,7 @@ dest: "/etc/dpdk/dpdk.conf" owner: "root" group: "root" + mode: "0640" when: - neutron_services['neutron-openvswitch-agent']['group'] in group_names - '"nova_compute" in group_names' @@ -111,7 +113,7 @@ - neutron_plugin_type in ['ml2.ovs', 'ml2.ovs.dvr'] - neutron_provider_networks.network_mappings is defined -# (todo) Loop thru ints or build a bond with ints. TBD. +# Adds a single host interface to an OVS bridge - name: Add ports to Network Provider Bridges openvswitch_port: bridge: "{{ interface_mapping.split(':')[0] }}" @@ -125,3 +127,23 @@ - neutron_services['neutron-openvswitch-agent']['group'] in group_names - neutron_plugin_type in ['ml2.ovs', 'ml2.ovs.dvr'] - neutron_provider_networks.network_interface_mappings is defined and (neutron_provider_networks.network_interface_mappings|length > 0) + +# Adds a DPDK-accelerated bond interface to an OVS bridge +- name: Add Bonds to Network Provider Bridges + openvswitch.openvswitch.openvswitch_bond: + bridge: "{{ bond_interfaces_mapping.bridge }}" + port: "{{ bond_interfaces_mapping.bridge }}-dpdkbond" + interfaces: "{{ bond_interfaces_mapping.interfaces }}" + bond_mode: "{{ bond_interfaces_mapping.bond_mode | default('active-backup') }}" + lacp: "{{ bond_interfaces_mapping.lacp | default('off') }}" + bond_updelay: "{{ bond_interfaces_mapping.bond_updelay | default(100) }}" + bond_downdelay: "{{ bond_interfaces_mapping.bond_downdelay | default(100) }}" + set: "{% for interface in bond_interfaces_mapping.interfaces %}interface {{ interface }} type=dpdk options:dpdk-devargs='{{ interface }}'{% if not loop.last %},{% endif %}{% endfor %}" + state: present + with_items: "{{ neutron_provider_networks.network_bond_interfaces_mappings }}" + loop_control: + loop_var: bond_interfaces_mapping + when: + - neutron_services['neutron-openvswitch-agent']['group'] in group_names + - neutron_plugin_type in ['ml2.ovs', 'ml2.ovs.dvr'] + - neutron_provider_networks.network_bond_interfaces_mappings is defined and (neutron_provider_networks.network_bond_interfaces_mappings|length > 0)