ovn-bgp-agent/doc/source/contributor/bgp_mode_stretched_l2_design.rst

OVN BGP Agent: Design of the stretched L2 BGP Driver

Purpose

The main reason for adding the L2 BGP driver is to announce networks via BGP that are not masqueraded (SNAT disabled) and can communicate directly via routing. The driver requires that all routers to be announced are in a L2 provider network and that the BGP Neighbor and Speaker are also part of this network. The whole tenant networks are announced via the external gateway IP of the router, instead of as /32 (or /128 for IPv6). This means that the tenant networks can be routed directly via the router IP and failover can run completely via BFD in OVN. No additional BGP announcements are needed incase of failover of routers, but only ARP/GARP in the respective L2 network.

The resulting routes are the same on all instances of the ovn-bgp-agent and are not bound to the machine the agent is running on. For redundancy reasons it is recommended to run multiple instances.

Overview

The OVN BGP Agent is a Python-based daemon that can run on any node. However, it is recommended to run the L2 BGP driver on the gateway node since all requirements are already met there, e.g. connectivity to the L2 provider network. The driver connects to the OVN SouthBound database (OVN SB DB) for all information and responds to events via it. It uses a VRF to create the routes locally and FRR to announce them to the BGP peer. The VRF is completely isolated and is not used for anything else than announcing routes via FRR. The tenant routers/networks cannot be reached from the VRF either.

Note

Note it is only intended for the N/S traffic, the E/W traffic will work exactly the same as before, i.e., VMs are connected through geneve tunnels.

The agent provides a multi-driver implementation that allows you to configure it for specific infrastructure running on top of OVN, for instance OpenStack or Kubernetes/OpenShift. This simple design allows the agent to implement different drivers, depending on what OVN SB DB events are being watched (watchers examples at ovn_bgp_agent/drivers/openstack/watchers/), and what actions are triggered in reaction to them (drivers examples at ovn_bgp_agent/drivers/openstack/XXXX_driver.py, implementing the ovn_bgp_agent/drivers/driver_api.py).

A common driver API is defined exposing the next methods:

• expose_ip and withdraw_ip: used to expose/withdraw IPs/Networks for OVN ports.
• expose_subnet, update_subnet and withdraw_subnet: used to expose/withdraw subnets through the external router gateway ip.

OVN SB DB Events

The watcher associated to this BGP driver detect the relevant events on the OVN SB DB to call the driver functions to configure BGP and linux kernel networking accordingly.

The BGP watcher detects OVN Southbound Database events at the Port_Binding and Load_Balancer tables. It creates new event classes named PortBindingChassisEvent and OVNLBMemberEvent, that all the events watched for BGP use as the base (inherit from).

The driver react specifically to the following events:

• PortBindingChassisCreatedEvent: Detects when a port of type "" (empty double-qoutes), virtual, or chassisredirect gets attached to the OVN chassis where the agent is running. This is the case for VM or amphora LB ports on the provider networks, VM or amphora LB ports on tenant networks with a FIP associated, and neutron gateway router ports (CR-LRPs). It calls expose_ip driver method to perform the needed actions to expose it.
• PortBindingChassisDeletedEvent: Detects when a port of type "" (empty double-quotes), virtual, or chassisredirect gets detached from the OVN chassis where the agent is running. This is the case for VM or amphora LB ports on the provider networks, VM or amphora LB ports on tenant networks with a FIP associated, and neutron gateway router ports (CR-LRPs). It calls withdraw_ip driver method to perform the needed actions to withdraw the exposed BGP route.
• SubnetRouterAttachedEvent: Detects when a patch port gets created. This means a subnet is attached to a router. If this port is associated to a cr-lrp port, the subnet will get announced.
• SubnetRouterDetachedEvent: Same as previous one, but for the deletion of the port. It calls withdraw_subnet.
• SubnetRouterUpdateEvent: Detects when a subnet/IP is added to an existing patch port. This can happen when multiple subnets are generated from an address pool and added to the same router. It calls update_subnet.

Driver Logic

The stretched L2 BGP driver is responsible for announcing all tenant networks that match the corresponding address scope (if used for filtering subnets). If the config option address_scopes is not set, all tenant networks will be announced via the corresponding provider network router IP.

BGP Advertisement

The OVN BGP Agent is in charge of triggering FRR (IP routing protocol suite for Linux which includes protocol daemons for BGP, OSPF, RIP, among others) to advertise/withdraw directly connected routes via BGP. To do that, when the agent starts, it ensures that:

• FRR local instance is reconfigured to leak routes for a new VRF. To do that it uses vtysh shell. It connects to the existing FRR socket ( --vty_socket option) and executes the next commands, passing them through a file (-c FILE_NAME option):

  LEAK_VRF_TEMPLATE = '''
  router bgp {{ bgp_as }}
    address-family ipv4 unicast
      import vrf {{ vrf_name }}
    exit-address-family
  
    address-family ipv6 unicast
      import vrf {{ vrf_name }}
    exit-address-family
  
  router bgp {{ bgp_as }} vrf {{ vrf_name }}
    bgp router-id {{ bgp_router_id }}
    address-family ipv4 unicast
      redistribute kernel
    exit-address-family
  
    address-family ipv6 unicast
      redistribute kernel
    exit-address-family
  
  '''

• There is a VRF created (the one leaked in the previous step) by default with name bgp_vrf.

• There is a dummy interface type (by default named bgp-nic), associated to the previously created VRF device.

Then, to expose the tenant networks as they are created (or upon initialization or re-sync), since the FRR configuration has the redistribute kernel option enabled, the only action needed to expose/withdraw the tenant networks is to add/remove the routes in the bgp_vrf_table_id table. Then it relies on Zebra to do the BGP advertisement, as Zebra detects the addition/deletion of the routes in the table and advertises/withdraw the route. In order to add these routes we have to make the Linux kernel believe that it can reach the respective router IPs. For this we use link-local routes pointing to the interface of the VRF. If we use the provider network 111.111.111.0/24, a router with the IP 111.111.111.17/24 on the gateway port and the tenant subnet 192.168.0.0/24, the route would be added like this (same logic applies to IPv6):

$ ip route add 111.111.111.0/24 dev bgp-nic table 10
$ ip route add 192.168.0.0/24 via 111.111.111.17 table 10

Note

The link-local route for the provider network is also announced and is only removed when no router to be announced has a gateway port on the network. Since all BGP peers should also be on this network, the BGP neighbor will prefer its connected route over the announced link-local route.

On the BGP neighbor side, the route should look like this:

$ ip route show 
192.168.0.0/24 via 111.111.111.17

Driver API

The BGP driver needs to implement the driver_api.py interface with the following functions:

• expose_ip: Creates the routes for all tenant networks and announces them via FRR. If no subnets are connected to this port, nothing is announced.
• withdraw_ip: Removes all routes for the tenant networks and withdraws them from FRR.
• expose_subnet: Announces the tenant network via the router IP if this router has an external gateway port.
• withdraw_subnet: Withdraws the tenant network if this router has an external gateway port.
• update_subnet: Does the same as expose_subnet / withdraw_subnet and is called when a subnet is added or removed from the port.

Agent deployment

The agent can be deployed anywhere as long as it is in the respective L2 network that is to be announced. In addition, OVS agent must be installed on the machine (from which it reads SB DB address) and it must be possible to connect to the Southbound Database. The L2 network can be filtered via the address scope, so it is not necessary that the agent has access to all L2 provider networks, but only the one in which it is to peer. Unlike the ovn_bgp_driver, it announces all routes regardless of which chassis they are on.

As an example of how to start the OVN BGP Agent on the nodes, see the commands below:

$ python setup.py install
$ cat bgp-agent.conf
# sample configuration that can be adapted based on needs
[DEFAULT]
debug=True
reconcile_interval=120
driver=ovn_stretched_l2_bgp_driver
address_scopes=2237917c7b12489a84de4ef384a2bcae

$ sudo bgp-agent --config-dir bgp-agent.conf
....

Note that the OVN BGP Agent operates under the next assumptions:

• A dynamic routing solution, in this case FRR, is deployed and advertises/withdraws routes added/deleted to/from the vrf routing table. A sample config for FRR is:

  frr version 7.0
  frr defaults traditional
  hostname cmp-1-0
  log file /var/log/frr/frr.log debugging
  log timestamp precision 3
  service integrated-vtysh-config
  line vty
  
  debug bgp neighbor-events
  debug bgp updates
  
  router bgp 64999
    bgp router-id 172.30.1.1
    neighbor pg peer-group
    neighbor 172.30.1.2 remote-as 64998
    address-family ipv6 unicast
      redistribute kernel
      neighbor pg activate
      neighbor pg route-map IMPORT in
      neighbor pg route-map EXPORT out
    exit-address-family
  
    address-family ipv4 unicast
      redistribute kernel
      neighbor pg activate
      neighbor pg route-map IMPORT in
      neighbor pg route-map EXPORT out
    exit-address-family
  
  route-map EXPORT deny 100
  
  route-map EXPORT permit 1
    match interface bgp-nic
  
  route-map IMPORT deny 1
  
  line vty

Limitations

• TBD