Browse Source

Merge "Spec for installing and configuring FRRouter for BGP"

Zuul 6 months ago
committed by Gerrit Code Review
  1. 245


@ -0,0 +1,245 @@
This work is licensed under a Creative Commons Attribution 3.0 Unported
Install and Configure FRRouter
The goal of this spec is to design and plan requirements for adding support to
TripleO to install and provide a basic configuration of Free Range Router (FRR)
on overcloud nodes in order to support BGP dynamic routing. There are multiple
reasons why an administrator might want to run FRR, including to obtain
multiple routes on multiple uplinks to northbound switches, or to advertise
routes to networks or IP addresses via dynamic routing protocols.
Problem description
There are several use cases for using BGP, and in fact there are separate
efforts underway to utilize BGP for the control plane and data plane.
BGP may be used for equal-cost multipath (ECMP) load balancing of outbound
links, and bi-directional forwarding detection (BFD) for resiliency to ensure
that a path provides connectivity. For outbound connectivity BGP will learn
routes from BGP peers.
BGP may be used for advertising routes to API endpoints. In this model HAProxy
will listen on an IP address and FRR will advertise routes to that IP to BGP
peers. High availability for HAProxy is provided via other means such as
Pacemaker, and FRR will simply advertise the virtual IP address when it is
active on an API controller.
BGP may also be used for routing inbound traffic to provider network IPs or
floating IPs for instance connectivity. The Compute nodes will run FRR to
advertise routes to the local VM IPs or floating IPs hosted on the node. FRR
has a daemon named Zebra that is responsible for exchanging routes between
routing daemons such as BGP and the kernel. The *redistribute connected*
statement in the FRR configuration will cause local IP addresses on the host
to be advertised via BGP. Floating IP addresses are attached to a loopback
interface in a namespace, so they will be redistributed using this method.
Changes to OVN will be required to ensure provider network IPs assigned to VMs
will be assigned to a loopback interface in a namespace in a similar fashion.
Proposed Change
Create a container with FRR. The container will run the BGP daemon, BFD
daemon, and Zebra daemon (which copies routes to/from the kernel). Provide a
basic configuration that would allow BGP peering with multiple peers. In the
control plane use case the FRR container needs to be started along with the HA
components, but in the data plane use case the container will be a sidecar
container supporting Neutron. The container is defined in a change proposed
here: [1]_
The container will be deployed using a TripleO Deployment Service. The service
will use Ansible to template the FRR configuration file, and a simple
implementation exists in a proposed change here: [2]_
The current FRR Ansible module is insufficient to configure BGP parameters and
would need to be extended. At this time the Ansible Networking development
team is not interested in extending the FRR module, so the configuration will
be provided using TripleO templates for the FRR main configuration file and
daemon configuration file. Those templates are defined in a change proposed
here: [3]_
A user-modifiable environment file will need to be provided so the installer
can provide the configuration data needed for FRR (see User Experience below).
OVN will need to be modified to enable the Compute node to assign VM provider
network IPs to a loopback interface inside a namespace. These IP address will
not be used for sending or receiving traffic, only for redistributing routes
to the IPs to BGP peers. Traffic which is sent to those IP addresses will be
forwarded to the VM using OVS flows on the hypervisor. An example agent for
OVN has been written to demonstrate how to monitor the southbound OVN DB and
create loopback IP addresses when a VM is started on a Compute node. The OVN
changes will be detailed in a separate OVN spec. Demonstration code is
available on Github: [4]_
User Experience
The installer will need to provide some basic information for the FRR
configuration, including whether to enable BFD, BGP IPv4, BGP IPv6,
and other settings. See the Example Configuration Data section below.
Additional user-provided data may include inbound or outbound filter prefixes.
The default filter prefixes will accept only default routes via BGP, and will
export only loopback IPs, which have a /32 subnet mask for IPv4 or /128 subnet
mask for IPv6.
Example Configuration Data
.. code-block:: yaml
tripleo_frr_bfd: false
tripleo_frr_bgp: false
tripleo_frr_bgp_ipv4: true
tripleo_frr_bgp_ipv4_allowas_in: false
tripleo_frr_bgp_ipv6: true
tripleo_frr_bgp_ipv6_allowas_in: false
tripleo_frr_config_basedir: "/var/lib/config-data/ansible-generated/frr"
tripleo_frr_hostname: "{{ ansible_hostname }}"
tripleo_frr_log_level: informational
tripleo_frr_watchfrr: true
tripleo_frr_zebra: false
1. Routing outbound traffic via multiple uplinks
Fault-tolerance and load-balancing for outbound traffic is typically
provided by bonding Ethernet interfaces. This works for most cases, but
is susceptible to unidirectional interface failure, a situation where
traffic works in only one direction. The LACP protocol for bonding does
provide some protection against unidirectional traffic failures, but is not
as robust as bi-directional forwarding detection (BFD) provided by FRR.
2. Routing inbound traffic to highly-available API endpoints
The most common method currently used to provide HA for API endpoints is
to use a virtual IP that fails over from active to standby nodes using a
shared Ethernet MAC address. The drawback to this method is that all
standby API controllers must reside on the same layer 2 segment (VLAN) as
the active controller. This presents a challenge if the operator wishes
to place API controllers in different failure domains for power and/or
networking. A BGP daemon avoids this limitation by advertising a route
to the shared IP address directly to the BGP peering router over a routed
layer 3 link.
3. Routing to Neutron IP addresses
Data plane traffic is usually delivered to provider network or floating
IP addresses via the Ethernet MAC address associated with the IP and
determined via ARP requests on a shared VLAN. This requires that every
Compute node which may host a provider network IP or floating IP has
the appropriate VLAN trunked to a provider bridge attached to an interface
or bond. This makes it impossible to migrate VMs or floating IPs across
layer 3 boundaries in edge computing topologies or in a fully layer 3
routed datacenter.
Security Impact
There have been no direct security impacts identified with this approach. The
installer should ensure that security policy on the network as whole prevents
IP spoofing which could divert legitimate traffic to an unintended host. This
is a concern whether or not the OpenStack nodes are using BGP themselves, and
may be an issue in environments using traditional routing architecture or
static routes.
Upgrade Impact
When (if) we remove the capability to manage network resources in the
overcloud heat stack, we will need to evaluate whether we want to continue
to provide BGP configuration as a part of the overcloud configuration.
If an operator wishes to begin using BGP routing at the same time as
upgrading the version of OpenStack used they will need to provide the
required configuration parameters if they differ from the defaults provided
in the TripleO deployment service.
Performance Impact
No performance impacts are expected, either positive or negative by using
this approach. Attempts have been made to minimize memory and CPU usage by
using conservative defaults in the configuration.
Documentation Impact
This is a new TripleO deployment service and should be properly documented
to instruct installers in the configuration of FRR for their environment.
The TripleO docs will need updates in many sections, including:
* `TripleO OpenStack Deployment
* `Provisioning Baremetal Before Overcloud Deploy
* `Deploying with Custom Networks
* `Configuring Network Isolation
* `Deploying Overcloud with L3 routed networking
The FRR daemons are documented elsewhere, and we should not need to document
usage of BGP in general, as this is a standard protocol. The configuration of
top-of-rack switches is different depending on the make and model of routing
switch used, and we should not expect to provide configuration examples for
network hardware.
The implementation will require a new TripleO deployment service, container
definition, and modifications to the existing role definitions. Those changes
are proposed upstream, see the References section for URL links.
Primary assignee:
* Dan Sneddon
Secondary assignees:
* Michele Baldessari
* Carlos Gonclaves
* Daniel Alvarez Sanchez
* Luis Tomas Bolivar
Work Items
* Develop the container definition
* Define the TripleO deployment service templates
* Define the TripleO Ansible role
* Modify the existing TripleO roles to support the above changes
* Merge the changes to the container, deployment service, and Ansible role
* Ensure FRR packages are available for supported OS versions
.. [1] `Review: DNR/DNM Frr support <>`_.
.. [2] `Review: Add tripleo_frr role <>`_.
.. [3] `Review: WIP/DNR/DNM FRR service <>`_.
.. [4] `OVN BGP Agent <>`_.