From f548f0acff3b4ebe31c5bfa84de17641a09e0ecd Mon Sep 17 00:00:00 2001 From: Dan Sneddon Date: Wed, 14 Oct 2020 13:39:20 -0700 Subject: [PATCH] Spec for installing and configuring FRRouter for BGP This spec covers installing and configuring FRRouter packages on overcloud nodes to provide dynamic routing using BGP and optionally bi-directional forwarding detection (BFD). Change-Id: I37a4a906c177030ec7a38e2ba1d805a9704baffd --- specs/wallaby/triplo-bgp-frrouter.rst | 245 ++++++++++++++++++++++++++ 1 file changed, 245 insertions(+) create mode 100644 specs/wallaby/triplo-bgp-frrouter.rst diff --git a/specs/wallaby/triplo-bgp-frrouter.rst b/specs/wallaby/triplo-bgp-frrouter.rst new file mode 100644 index 00000000..50ed079d --- /dev/null +++ b/specs/wallaby/triplo-bgp-frrouter.rst @@ -0,0 +1,245 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +============================== +Install and Configure FRRouter +============================== + +The goal of this spec is to design and plan requirements for adding support to +TripleO to install and provide a basic configuration of Free Range Router (FRR) +on overcloud nodes in order to support BGP dynamic routing. There are multiple +reasons why an administrator might want to run FRR, including to obtain +multiple routes on multiple uplinks to northbound switches, or to advertise +routes to networks or IP addresses via dynamic routing protocols. + +Problem description +=================== + +There are several use cases for using BGP, and in fact there are separate +efforts underway to utilize BGP for the control plane and data plane. + +BGP may be used for equal-cost multipath (ECMP) load balancing of outbound +links, and bi-directional forwarding detection (BFD) for resiliency to ensure +that a path provides connectivity. For outbound connectivity BGP will learn +routes from BGP peers. + +BGP may be used for advertising routes to API endpoints. In this model HAProxy +will listen on an IP address and FRR will advertise routes to that IP to BGP +peers. High availability for HAProxy is provided via other means such as +Pacemaker, and FRR will simply advertise the virtual IP address when it is +active on an API controller. + +BGP may also be used for routing inbound traffic to provider network IPs or +floating IPs for instance connectivity. The Compute nodes will run FRR to +advertise routes to the local VM IPs or floating IPs hosted on the node. FRR +has a daemon named Zebra that is responsible for exchanging routes between +routing daemons such as BGP and the kernel. The *redistribute connected* +statement in the FRR configuration will cause local IP addresses on the host +to be advertised via BGP. Floating IP addresses are attached to a loopback +interface in a namespace, so they will be redistributed using this method. +Changes to OVN will be required to ensure provider network IPs assigned to VMs +will be assigned to a loopback interface in a namespace in a similar fashion. + +Proposed Change +=============== + +Overview +-------- + +Create a container with FRR. The container will run the BGP daemon, BFD +daemon, and Zebra daemon (which copies routes to/from the kernel). Provide a +basic configuration that would allow BGP peering with multiple peers. In the +control plane use case the FRR container needs to be started along with the HA +components, but in the data plane use case the container will be a sidecar +container supporting Neutron. The container is defined in a change proposed +here: [1]_ + +The container will be deployed using a TripleO Deployment Service. The service +will use Ansible to template the FRR configuration file, and a simple +implementation exists in a proposed change here: [2]_ + +The current FRR Ansible module is insufficient to configure BGP parameters and +would need to be extended. At this time the Ansible Networking development +team is not interested in extending the FRR module, so the configuration will +be provided using TripleO templates for the FRR main configuration file and +daemon configuration file. Those templates are defined in a change proposed +here: [3]_ + +A user-modifiable environment file will need to be provided so the installer +can provide the configuration data needed for FRR (see User Experience below). + +OVN will need to be modified to enable the Compute node to assign VM provider +network IPs to a loopback interface inside a namespace. These IP address will +not be used for sending or receiving traffic, only for redistributing routes +to the IPs to BGP peers. Traffic which is sent to those IP addresses will be +forwarded to the VM using OVS flows on the hypervisor. An example agent for +OVN has been written to demonstrate how to monitor the southbound OVN DB and +create loopback IP addresses when a VM is started on a Compute node. The OVN +changes will be detailed in a separate OVN spec. Demonstration code is +available on Github: [4]_ + +User Experience +^^^^^^^^^^^^^^^ + +The installer will need to provide some basic information for the FRR +configuration, including whether to enable BFD, BGP IPv4, BGP IPv6, +and other settings. See the Example Configuration Data section below. + +Additional user-provided data may include inbound or outbound filter prefixes. +The default filter prefixes will accept only default routes via BGP, and will +export only loopback IPs, which have a /32 subnet mask for IPv4 or /128 subnet +mask for IPv6. + +Example Configuration Data +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: yaml + + tripleo_frr_bfd: false + tripleo_frr_bgp: false + tripleo_frr_bgp_ipv4: true + tripleo_frr_bgp_ipv4_allowas_in: false + tripleo_frr_bgp_ipv6: true + tripleo_frr_bgp_ipv6_allowas_in: false + tripleo_frr_config_basedir: "/var/lib/config-data/ansible-generated/frr" + tripleo_frr_hostname: "{{ ansible_hostname }}" + tripleo_frr_log_level: informational + tripleo_frr_watchfrr: true + tripleo_frr_zebra: false + +Alternatives +============ + +1. Routing outbound traffic via multiple uplinks + + Fault-tolerance and load-balancing for outbound traffic is typically + provided by bonding Ethernet interfaces. This works for most cases, but + is susceptible to unidirectional interface failure, a situation where + traffic works in only one direction. The LACP protocol for bonding does + provide some protection against unidirectional traffic failures, but is not + as robust as bi-directional forwarding detection (BFD) provided by FRR. + +2. Routing inbound traffic to highly-available API endpoints + + The most common method currently used to provide HA for API endpoints is + to use a virtual IP that fails over from active to standby nodes using a + shared Ethernet MAC address. The drawback to this method is that all + standby API controllers must reside on the same layer 2 segment (VLAN) as + the active controller. This presents a challenge if the operator wishes + to place API controllers in different failure domains for power and/or + networking. A BGP daemon avoids this limitation by advertising a route + to the shared IP address directly to the BGP peering router over a routed + layer 3 link. + + +3. Routing to Neutron IP addresses + + Data plane traffic is usually delivered to provider network or floating + IP addresses via the Ethernet MAC address associated with the IP and + determined via ARP requests on a shared VLAN. This requires that every + Compute node which may host a provider network IP or floating IP has + the appropriate VLAN trunked to a provider bridge attached to an interface + or bond. This makes it impossible to migrate VMs or floating IPs across + layer 3 boundaries in edge computing topologies or in a fully layer 3 + routed datacenter. + + +Security Impact +=============== + +There have been no direct security impacts identified with this approach. The +installer should ensure that security policy on the network as whole prevents +IP spoofing which could divert legitimate traffic to an unintended host. This +is a concern whether or not the OpenStack nodes are using BGP themselves, and +may be an issue in environments using traditional routing architecture or +static routes. + + +Upgrade Impact +============== + +When (if) we remove the capability to manage network resources in the +overcloud heat stack, we will need to evaluate whether we want to continue +to provide BGP configuration as a part of the overcloud configuration. + +If an operator wishes to begin using BGP routing at the same time as +upgrading the version of OpenStack used they will need to provide the +required configuration parameters if they differ from the defaults provided +in the TripleO deployment service. + + +Performance Impact +================== + +No performance impacts are expected, either positive or negative by using +this approach. Attempts have been made to minimize memory and CPU usage by +using conservative defaults in the configuration. + + +Documentation Impact +==================== + +This is a new TripleO deployment service and should be properly documented +to instruct installers in the configuration of FRR for their environment. + +The TripleO docs will need updates in many sections, including: + +* `TripleO OpenStack Deployment + `_ +* `Provisioning Baremetal Before Overcloud Deploy + `_ +* `Deploying with Custom Networks + `_ +* `Configuring Network Isolation + `_ +* `Deploying Overcloud with L3 routed networking + `_ + +The FRR daemons are documented elsewhere, and we should not need to document +usage of BGP in general, as this is a standard protocol. The configuration of +top-of-rack switches is different depending on the make and model of routing +switch used, and we should not expect to provide configuration examples for +network hardware. + +Implementation +============== + +The implementation will require a new TripleO deployment service, container +definition, and modifications to the existing role definitions. Those changes +are proposed upstream, see the References section for URL links. + + +Assignee(s) +=========== + +Primary assignee: + * Dan Sneddon + +Secondary assignees: + * Michele Baldessari + * Carlos Gonclaves + * Daniel Alvarez Sanchez + * Luis Tomas Bolivar + + +Work Items +========== + +* Develop the container definition +* Define the TripleO deployment service templates +* Define the TripleO Ansible role +* Modify the existing TripleO roles to support the above changes +* Merge the changes to the container, deployment service, and Ansible role +* Ensure FRR packages are available for supported OS versions + + +References +========== + +.. [1] `Review: DNR/DNM Frr support `_. +.. [2] `Review: Add tripleo_frr role `_. +.. [3] `Review: WIP/DNR/DNM FRR service `_. +.. [4] `OVN BGP Agent `_.