L23network ========== Puppet module for configuring network interfaces on 2nd and 3rd level (802.1q vlans, access ports, NIC-bondind, assign IP addresses, dhcp, and interfaces without IP addresses). Can work together with Open vSwitch or standart linux way. At this moment support Centos 6.3+ (RHEL6) and Ubuntu 12.04 or above. L23network module have a same behavior for both operation systems. **WARNING!!!** This is a L23network v1.1, it contains some incompatibles with earlier versions. *Be carefully*. ## Usage ### Initializing Place this module at /etc/puppet/modules/l23network or another directory with your puppet modules. Include L23network module and initialize it. It is recommended to do it on the early stage: #Network configuration stage {'netconfig': before => Stage['main'], } class { 'l23network': use_ovs => true, use_lnx => true, stage => 'netconfig' } Initialization class 'l23network' has following incoming parameters and its default values: class { 'l23network': use_ovs => false, use_lnx => true, install_ovs => $use_ovs, install_brtool => $use_lnx, install_ethtool => $use_lnx, install_bondtool => $use_lnx, install_vlantool => $use_lnx, ovs_module_name => undef, use_ovs_dkms_datapath_module => undef, ovs_datapath_package_name => undef, ovs_common_package_name => undef, } For highly customized configurations you can redefine each of ones. For example, if you plan to use open vSwitch you should enable it: class {'l23network': use_ovs=>true } ### L2 network features configuation Current layout is: * *bridges* -- A "Bridge" is a virtual ethernet L2 switch. You can plug ports into it. * *ports* -- A Port is an interface you plug into the bridge. It may be virtual or native interface. Then in your manifest you can either use it as a parameterized classes: l23network::l2::bridge{"br-mgmt": } l23network::l2::port{"eth0": bridge => "br-mgmt"} l23network::l2::bridge{"br-ex": provider => ovs } l23network::l2::port{"eth1": bridge => "br-ex" } l23network::l2::port{"ve0": bridge => "br-ex" } l23network::l2::port{"ve1": bridge => "br-ex" } #### L2::Bridge This resource implemented for configire of bridge. l23network::l2::bridge { 'br1': ensure => present, stp => true, # or false vendor_specific => { ..... }, provider => lnx, } Non-obligatory fields: * *stp* -- enable/disable STP for bridge * *bpdu_forward* -- enable/disable BPDU forward on bridge * *bridge_id* -- bridge_id for STP protocol. * *vendor_specific* -- vendor_specific hash (see below) * *delay_while_up* -- delay, in seconds, which will happens each time while node will boot after interface up. #### L2::Port Resource for configuring port L2 options. Only L2 options. For configuring L3 options -- use *L23network::l3::ifconfig* resource l23network::l2::port { 'eth1': mtu => 9000, # MTU value, unchanged if absent. onboot => true, # whether port has UP state after setup or node boot ethtool => { ..... }, vendor_specific => { ..... }, provider => lnx } l23network::l2::port { 'eth1.101': ensure => present, bridge => 'br1', # port can be a member of bridge. # If no value given this property was unchanged, # if given 'absent' port will be excluded from any # bridges. onboot => true, delay_while_up => 10 provider => lnx } Alternative VLAN definition (not recommended for 'lnx' provider) l23network::l2::port { 'vlan77': vlan_id => 77, vlan_dev => eth1, provider => lnx } #### L2::Bond It's a special type of port. Designed for bonding two or more interfaces. Detail description of bonding feature you can read here: https://www.kernel.org/doc/Documentation/networking/bonding.txt If you plan use LACP -- we highly recommend do not use OVS. Also we don't recommend insert native linux bonds to OVS bridges. This case works, but leads many heavy diagnostic surprises. l23network::l2::bond { 'bond0': interfaces => ['eth1', 'eth2'], bridge => 'br0', # obligatory only for OVS provider mtu => 9000, onboot => true, bond_properties => { # bond configuration properties (see bonding.txt) mode => '803.1ad', lacp_rate => 'slow', xmit_hash_policy => 'encap3+4' }, interface_properties => { # config properties for included ifaces ethtool => { ..... }, }, vendor_specific => { ..... }, delay_while_up => 45 provider => lnx, } Bond **mode** and **xmit_hash_policy** configuration has some differences for *lnx* and *ovs* providers: For *lnx* provider **mode** can be: * balance-rr *(default)* * active-backup * balance-xor * broadcast * 802.3ad * balance-tlb * balance-alb For 802.3ad (LACP), balance-xor, balance-tlb and balance-alb cases should be defined **xmit_hash_policy** as one of: * layer2 *(default)* * layer2+3 * layer3+4 * encap2+3 * encap3+4 For *ovs* provider **mode** can be: * active-backup * balance-slb *(default)* * balance-tcp Field **xmit_hash_policy** shouldn't use for any mode. For *balance-tcp* mode **lacp** bond-property should be set to 'active' or 'passive' value. While bond will created also will created ports, included to the bond. This ports will be created as slave ports for this bond with properties, listed in **interface_properties** field. If you want more flexibility, you can create this ports by *l23network::l2::port* resource and shouldn't define **interface_properties** field. **MTU** field, setted for bond interface will be passed to interfaces, included to the bond automatically. I recommend use **delay_while_up** property, while configure LACP bonds, because such bonds may take some time for settle. For some providers (ex: ovs) **bridge** field is obligatory. #### L2::Patch It's a patchcord for connecting two bridges. Architecture limitation: two bridges may be connected only by one patchcord. Name for patchcord interfaces calculated automatically and can't changed in configuration. OVS provider can connect OVS-to-OVS and OVS-to-LNX bridges. If you connect OVS-to-LNX bridges, you SHOULD put OVS bridge first in order. l23network::l2::patch { 'patch__br0--br1': bridges => ['br0','br1'], vendor_specific => { ..... }, } **Naming conviency** Each low-level puppet patchcord resource *l2_patch* has his name in 'bridge__%bridge1%--%bridge2%' format, and bridges provided in alphabetical order for all providers. This resource also contain 'bridges' property. It's a array of two bridge names. Order of names depends of provider implementation. For example, 'ovs' provider bridge names listed in alphabetical order for OVS-to-OVS connectivity, and ovs-bridge always first for OVS-to-LNX bridges connectivity. Each *L2_patch* instance contains read-only 'jacks' property. It's a array of two names of jacks, 'inserted' to each bridge. This property has the same ordering style, that a 'bridges' property for this provider. If patchcord connect two bridges different nature, the 'cross' flag will be setting to 'true'. #### Ethtool hash and offloading settings You can manage offloading and another options, controlled by ethtool utility, for any resources, that has *ethtool* hash as one of incoming properties. *Ethtool* field look like hash of hashes. Keys of the external hash -- are a section names from ethtool manual. Ones maps to an internal hashes. Internal hashes -- is a option to value mappings. Option names corresponds to ethtool output option naming. For example, you can see list of offloading options by executing 'ethtool -k eth0'. Ethtool options are pre-defined and stateful. All implemented sections and options you can see bellow: ethtool => { offload => { rx-checksumming => true or false, tx-checksumming => true or false, scatter-gather => true or false, tcp-segmentation-offload => true or false, udp-fragmentation-offload => true or false, generic-segmentation-offload => true or false, generic-receive-offload => true or false, large-receive-offload => true or false, rx-vlan-offload => true or false, tx-vlan-offload => true or false, ntuple-filters => true or false, receive-hashing => true or false, rx-fcs => true or false, rx-all => true or false, highdma => true or false, rx-vlan-filter => true or false, fcoe-mtu => true or false, l2-fwd-offload => true or false, loopback => true or false, tx-nocache-copy => true or false, tx-gso-robust => true or false, tx-fcoe-segmentation => true or false, tx-gre-segmentation => true or false, tx-ipip-segmentation => true or false, tx-sit-segmentation => true or false, tx-udp_tnl-segmentation => true or false, tx-mpls-segmentation => true or false, tx-vlan-stag-hw-insert => true or false, rx-vlan-stag-hw-parse => true or false, rx-vlan-stag-filter => true or false, }, #settings => { # duplex => 'half', # mdix => off #} } ### L3 network configuration #### L3::Ifconfig Resource for configuring IP addresses on interface. Only L3 options. For configuring L2 options -- use corresponded L2 resource. l23network::l3::ifconfig { 'eth1.101': ensure => present, ipaddr => ['192.168.10.3/24', '10.20.30.40/25'], gateway => '192.168.10.1', gateway_metric => 10, # different Ifconfig resources should not has # gateways with same metrics vendor_specific => { ..... }, } **DHCP or address-less interfaces** l23network::l3::ifconfig {"eth2": ipaddr=>'dhcp'} l23network::l3::ifconfig {"eth3": ipaddr=>'none'} Option *ipaddr* can contains array of IP addresses (even setup one ipaddr), 'dhcp', or 'none' string. CIDR-notated form of IP address is required. **Default gateway** l23network::l3::ifconfig {"eth1": ipaddr => ['192.168.2.5/24'], gateway => '192.168.2.1', gateway_metric => 10, } if *gateway_metric* omited, gateway will be setup without metric definition. ## Network Scheme Network scheme is a hierarchical-based manner for define network topology for host. In following examples I use yaml format for represent it. Main idea: * when we got undeployed server we have some number of NICs. NICs, managed by puppet should be listed in *interfaces* section. It is giveg. * The result of our network configuration process is a some network topology on the host and some interfaces with assignet IP addresses (or without IPs). It's a *endpoints*. * Interfaces become endpoints by successive *transformations*. I try explain how it works in the following document: [Transformations. How they work.](https://docs.google.com/document/d/12RvBjOYO83_yqeiAgxttrRaa90-8un80aEO8OzDlQ9Y) Example of typical network scheme: --- network_scheme: version: 1.1 provider: lnx interfaces: eth1: mtu: 7777 eth2: mtu: 9000 transformations: - action: add-br name: br1 - action: add-port name: eth1 bridge: br1 - action: add-br name: br-mgmt - action: add-port name: eth1.101 bridge: br-mgmt - action: add-br name: br-ex - action: add-port name: eth1.102 bridge: br-ex - action: add-br name: br-storage - action: add-port name: eth1.103 bridge: br-storage - action: add-br name: br-prv provider: ovs - action: add-port name: eth2 bridge: br-prv provider: ovs endpoints: br-mgmt: IP: - 192.168.101.3/24 gateway: 192.168.101.1 gateway-metric: 100 routes: - net: 192.168.210.0/24 via: 192.168.101.1 - net: 192.168.211.0/24 via: 192.168.101.1 - net: 192.168.212.0/24 via: 192.168.101.1 br-ex: gateway: 192.168.102.1 IP: - 192.168.102.3/24 br-storage: IP: - 192.168.103.3/24 br-prv: IP: none roles: management: br-mgmt ceph: br-mgmt private: br-prv fw-admin: br1 ex: br-ex floating: br-ex storage: br-storage Example of typical network scheme with bonds and disabling offloads: --- network_scheme: version: "1.1" provider: lnx interfaces: eth1: mtu: 9000 eth2: eth3: transformations: - action: add-br name: br1 - action: add-port bridge: br1 name: eth1 ethtool: offload: tcp-segmentation-offload: off udp-fragmentation-offload: off generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off - action: add-br name: br2 - action: add-bond name: bond23 bridge: br2 interfaces: - eth2 - eth3 mtu: 9000 interface_properties: ethtool: offload: tcp-segmentation-offload: off udp-fragmentation-offload: off bond_properties: mode: balance-rr xmit_hash_policy: encap3+4 updelay: 10 downdelay: 40 use_carrier: 0 - action: add-br name: br-mgmt - action: add-port name: bond23.101 bridge: br-mgmt - action: add-br name: br-ex - action: add-port name: bond23.102 bridge: br-ex - action: add-br name: br-storage - action: add-port name: bond23.103 bridge: br-storage endpoints: br-mgmt: IP: - 192.168.101.3/24 gateway: 192.168.101.1 gateway-metric: 100 br-ex: gateway: 192.168.102.1 IP: - 192.168.102.3/24 br-storage: IP: - 192.168.103.3/24 roles: fw-admin: br1 ex: br-ex management: br-mgmt storage: br-storage ## Vendor_specific hash **Vendor_specific** field - is a hash, empty by default, required only for plug-ins. It allows plugin developers not to change custom type code for adding non-standart parameters. Due to inheriting and extending puppet type (not the provider one), is a non-trivial task. Plugin developers may pass any data structures by this hash and its subhashes. All data from this hash pass to the provider transparently. ## Debugging For debug purpose you can use following puppet calls for get prefetchable properties for existing resources. Please note, that bridges and bonds in linux are a port too, and present in l2_port output with corresponded flags (if_type). # puppet resource -vd --trace l23_stored_config # puppet resource -vd --trace l2_port # puppet resource -vd --trace l2_bridge # puppet resource -vd --trace l2_bond # puppet resource -vd --trace l3_ifconfig # puppet resource -vd --trace l3_route This commands may be fail before 1st configuration networking by L23network because some kernel modules may wasn't loaded or some command-line tools wasn't installed. ## Internals Each L23network resource has interface trought puppet 'define' resource. This define may conains some non difficult logic, define provider for low-level resources and call two low level resources: * *l23_stored_config* -- for modifying OS config files * low level resource for configuring it in runtime (e.x: *l2_bridge*) ### L23_stored_config custom type This resource is implemented to manage interface config files. Each possible parameter should be described in resource type. This resource allows to forget about ERB templates, because in some cases (i.e. bridge + port with same name + ip address for this port) we should modify config file content three times. l23_stored_config { 'br1': onboot => true, method => manual, mtu => 1500, ethtool => { ..... }, provider => lnx_ubuntu } Place of config files location defined inside provider for corresponded operation system and provider. Provider name for l23_stored_config depends from operation system (may be with version) and network provider (native linux, ovs, etc...) ## References * [Transformations. How they work.](https://docs.google.com/document/d/12RvBjOYO83_yqeiAgxttrRaa90-8un80aEO8OzDlQ9Y) --- When I started working on this module I was inspired by https://github.com/ekarlso/puppet-vswitch. Endre, big thanks...