From 098af11b855de87d4944079dac7fe7241801971e Mon Sep 17 00:00:00 2001 From: joehuang Date: Tue, 21 Mar 2017 22:28:35 -0400 Subject: [PATCH] Layer-3 Networking multi-NS-with-EW-enabled 1. What is the problem? Multiple north-south external network topology is supported, but the east-west networking for local networks across multiple OpenStack clouds has not been supported yet. 2. What is the solution to the problem? In multi-region cloud deployment, a requirement is that each OpenStack cloud provides external network, north-south traffic is expected to be handled locally for shortest path, and/or use multiple external networks to help application north-south traffic redundancy, at the same time east-west networking of tenant's networks between OpenStack cloud is also needed. This path will introduce east-west gateway router to address the east-west networking for local networks in different OpenStack clouds. 3. What the features need to be implemented to the Tricircle to realize the solution? East-west gateway router. Change-Id: Ie92fc268249371fba86b19e9d070793e24199c91 Signed-off-by: joehuang --- ...l3-networking-multi-NS-with-EW-enabled.rst | 393 ++++++++++++++++++ 1 file changed, 393 insertions(+) create mode 100644 specs/pike/l3-networking-multi-NS-with-EW-enabled.rst diff --git a/specs/pike/l3-networking-multi-NS-with-EW-enabled.rst b/specs/pike/l3-networking-multi-NS-with-EW-enabled.rst new file mode 100644 index 00000000..d6bce4ae --- /dev/null +++ b/specs/pike/l3-networking-multi-NS-with-EW-enabled.rst @@ -0,0 +1,393 @@ +=========================================== +Layer-3 Networking multi-NS-with-EW-enabled +=========================================== + +Problems +======== + +There are already several scenarios fulfilled in Tricircle for north- +south networking. + +Scenario "North South Networking via Multiple External Networks"[1] meets +the demand for multiple external networks, but local network can not +reach other local networks which are not in the same OpenStack cloud. + +Scenario "North South Networking via Single External Network"[2] can meet +local networks east-west networking requirement, but the north-south traffic +needs to go to single gateway. + +In multi-region cloud deployment, a requirement is that each OpenStack cloud +provides external network, north-south traffic is expected to be handled +locally for shortest path, and/or use multiple external networks to ensure +application north-south traffic redundancy, at the same time east-west +networking of tenant's networks between OpenStack cloud is also needed. + +Proposal +======== + +To address the above problems, the key limitation is the pattern for router +gateway, one router in Neutron can only be attached to one external network. +As what's described in the spec of combined bridge network[3], only external +network is suitable for working as bridge network due to DVR challenge. + +North-south traffic via the external network in the same region is conflict +with external network as bridge network. + +The proposal is to introduce a new networking mode for this scenario:: + + + +-----------------------+ +----------------------+ + | ext-net1 | | ext-net2 | + | +---+---+ | | +--+---+ | + |RegionOne | | | RegionTwo | | + | +---+---+ | | +----+--+ | + | | R1 | | | | R2 | | + | +--+----+ | | +--+----+ | + | | net1 | | net2 | | + | +---+--+---+-+ | | ++-----+--+---+ | + | | | | | | | | + | +---------+-+ | | | | +--+--------+ | + | | Instance1 | | | | | | Instance2 | | + | +-----------+ | | | | +-----------+ | + | +----+--+ | bridge-net | +-+-----+ | + | | R3(1) +--------------------+ R3(2) | | + | +-------+ | | +-------+ | + +-----------------------+ +----------------------+ + Figure.1 Multiple external networks with east-west networking + +R1 is the router to connect the external network ext-net1 directly +in RegionOne. Net1's default gateway is R1, so all north-south traffic +will be forwarded by R1 by default. In short, north-south traffic of net2 +will be processed by R2 in RegionTwo. R1 and R2 are local routers which +is supposed to be presented in only one region. Region name should be +specified in availability-zone-hint during router creation in central +Neutron, for example:: + + openstack --os-region-name=CentralRegion router create --availability-zone-hint=RegionOne R1 + openstack --os-region-name=CentralRegion router create --availability-zone-hint=RegionTwo R2 + + openstack --os-region-name=CentralRegion router add subnet R1 + openstack --os-region-name=CentralRegion router add subnet R2 + +In order to process the east-west traffic from net1 to net2, R3(1) and R3(2) +will be introduced, R3(1) and R3(2) will be inter-connected by bridge-net. +Bridge-net could be VLAN or VxLAN cross Neutron L2 network, and it's the +"external network" for both R3(1) and R3(2), please note here the bridge-net +is not real external network, just the concept of Neutron network. R3(1) and +R3(2) will only forward the east-west traffic across OpenStack for local +networks, so it's not necessary to work as DVR, centralized router is good +enough. + +In central Neutron, we only need to create a virtual logical router R3, +and R3 router is called as east-west gateway, to handle the east-west +traffic for local networks in different region, and it's non-local router. +Tricircle central Neutron plugin will help to create R3(1) in RegionOne and +R3(2) in RegionTwo, and use the bridge network to inter-connect R3(1) and +R3(2). The logical topology in central Neutron looks like follows:: + + ext-net1 ext-net2 + +-------+ +--+---+ + | | + +---+---+ +----+--+ + | R1 | | R2 | + +--+----+ +--+----+ + | net1 net2 | + +---+--+---++ ++-----+--+---+ + | | | | + +---------+-+ | | +--+--------+ + | Instance1 | | | | Instance2 | + +-----------+ | | +-----------+ + +-+----+--+ + | R3 | + +---------+ + + Figure.2 Logical topology in central Neutron + +Tricircle central Neutron plugin will use logical router R3 to create R3(1) +in RegionOne, and R3(2) in RegionTwo. + +Please note that R3(1) is not the default gateway of net1, and R3(2) is not +the default gateway of net2 too. So the user has to create a port and use +this port as the router interface explicitly between router and local +network. + +In central Neutron, the topology could be created like this:: + + openstack --os-region-name=CentralRegion port create --network=net1 net1-R3-interface + openstack --os-region-name=CentralRegion router add port R3 + + openstack --os-region-name=CentralRegion port create --network=net2 net2-R3-interface + openstack --os-region-name=CentralRegion router add port R3 + +Tricircle central Neutron plugin will automatically configure R3(1), R3(2) and +bridge-network as follows: + +For net1, host route should be added:: + + destination=net2's cidr, nexthop= + +For net2, host route should be added:: + + destination=net1's cidr, nexthop= + +In R3(1), extra route will be configured:: + + destination=net2's cidr, nexthop=R3(2)'s interface in bridge-net + +In R3(2), extra route will be configured:: + + destination=net1's cidr, nexthop=R3(1)'s interface in bridge-net + +R3(1) and R3(2) will set the external gateway to bridge-net:: + + router-gateway-set R3(1) bridge-net + router-gateway-set R3(2) bridge-net + +Now, north-south traffic of Instance1 and Instance2 work like follows:: + + Instance1 -> net1 -> R1 -> ext-net1 + Instance2 -> net2 -> R2 -> ext-net2 + +Only one hop for north-south traffic. + +East-west traffic between Instance1 and Instance2 work like follows:: + + Instance1 <-> net1 <-> R3(1) <-> bridge-net <-> R3(2) <-> net2 <-> Instance2 + +Two hops for cross OpenStack east-west traffic. + +The topology will be more complex if there are cross OpenStack L2 networks +except local networks:: + + +-----------------------+ +----------------------+ + | ext-net1 | | ext-net2 | + | +-------+ | | +--+---+ | + |RegionOne | | | RegionTwo | | + | +---+----------+ | | +-------------+--+ | + | | R1 | | | | R2 | | + | +--+--+---+--+-+ | | ++-+----+---+----+ | + | net1 | | | | | | | | | | net2 | + | ++--++ | | | | | | | | +-+---+ | + | | net3| | | | | | | |net4| | + | | ++---+ | | | | | | ++---+ | | + | | | | | | net5 | | | | | | + | | | +++-------------------------+-++| | | + | | | | | | net6 | | | | | | + | | | |++-+--------------------+++ | | | | + | | | | | | | | | | | | + | | | | | | | | | | | | + | | | | | | | | | | | | + | | | | | | | | | | | | + | +----+---+----+-+-+ | bridge-net | ++--+-+-----+-----+ | + | | R3(1) +--------------------+ R3(2) | | + | +-----------------+ | | +-----------------+ | + +-----------------------+ +----------------------+ + + Figure.3 Multi-NS and cross OpenStack L2 networks + +The logical topology in central Neutron for Figure.3 looks like as follows:: + + ext-net1 ext-net2 + +-------+ +--+---+ + | | + +---+----------+ +-------------+--+ + | R1 | | R2 | + +--+--+---+--+-+ ++-+----+---+----+ + net1 | | | | | | | | net2 + ++--++ | | | | | | +-+---+ + | net3| | | | | |net4| + | ++---+ | | | | ++---+ | + | | | | net5 | | | | + | | +------+------------------+ | | + | | | | net6 | | | + | | +-------------+------+ | | + | | | | | | + | | | | | | + | | | | | | + | | | | | | + +-+---+------------+---------+------------+-----+-+ + | R3 | + +-------------------------------------------------+ + Figure.4 Logical topology in central Neutron with cross OpenStack L2 network + +East-west traffic inside one region will be processed locally through default +gateway. For example, in RegionOne, R1 has router interfaces in net1, net3, +net5, net6, the east-west traffic between these networks will work as follows:: + + net1 <-> R1 <-> net3 + net1 <-> R1 <-> net5 + net1 <-> R1 <-> net6 + net3 <-> R1 <-> net5 + net3 <-> R1 <-> net6 + net5 <-> R1 <-> net6 + +There is nothing special for east-west traffic between local networks +in different OpenStack regions. + +Net5 and net6 are cross OpenStack L2 networks, instances could be attached +to network from different regions, and instances are reachable in a remote +region via the cross OpenStack L2 network itself. There is no need to add host +route for cross OpenStack L2 network, for it's routable in the same region for +other local networks or cross OpenStack L2 networks, default route is enough +for east-west traffic. + +It's needed to address how one cross OpenStack L2 network will be +attached different local router: different gateway IP address will be used. +For example, in central Neutron, net5's default gateway IP is 192.168.0.1 +in R1, the user needs to create a gateway port explicitly for local router R2 +and net5, for example 192.168.0.2, then net5 will be attached to R2 using this +gateway port 192.168.0.2. Tricircle central Neutron plugin will make this +port's IP 192.168.0.2 as the default gateway IP for net5 in RegionTwo. + +Besides of gateway ports creation for local router R2, it's also needed to +create a gateway port for R3 and net5, which is used for east-west traffic. +Because R3 will be spread into RegionOne and RegionTwo, so net5 will have +different gateway ports in RegionOne and RegionTwo. Tricircle central Neutron +plugin needs to reserve the gateway ports in central Neutron, and create these +gateway ports in RegionOne and RegionTwo for net5 on R3. Because R3 is the +east-west gateway router for net5, so these gateway ports are not the default +gateway port. Then host route in net5 should be updated for local networks +which are not in the same region: + +For net5 in RegionOne, host route should be added:: + + destination=net2's cidr, nexthop= + destination=net4's cidr, nexthop= + +For net5 in RegionTwo, host route should be added:: + + destination=net1's cidr, nexthop= + destination=net3's cidr, nexthop= + +Similar operation for net6 in RegionOne and RegionTwo. + +If R1 and R2 are centralized routers, cross OpenStack L2 network will +work, but if R1 and R2 are DVRs, then DVR MAC issue mentioned in the +spec "l3-networking-combined-bridge-net" should be fixed[2]. + +In order to make the topology not too complex, this use case will not be +supported: a cross OpenStack L2 network is not able to be stretched into +the region where there are local networks. This use case is not useful +and will make the east-west traffic even more complex:: + + +-----------------------+ +----------+ +-----------------+ + | ext-net1 | | ext-net2 | | ext-net4 | + | +-------+ | | +------+ | | +--+---+ | + |RegionOne | | | RegionTwo| | Region4 | | + | +---+----------+ | | +------+ | | +-------+--+ | + | | R1 | | | | R2 | | | | R4 | | + | +--+--+---+--+-+ | | ++-+---+ | | +-+---+----+ | + | net1 | | | | | | | | | | | | net2 | + | ++--++ | | | | | | | | | | +-+---+ | + | | net3| | | | | | | | | |net4| | + | | ++---+ | | | | | | | | ++---+ | | + | | | | | | net5 | | | | | | | | + | | | +-+-------------------------+-+ | | | | | + | | | | | net6 | | | | | | | + | | | +-+--------------------+ | | | | | + | | | | | | | | | | + | | | | | | | | | | + | | | | | | | | | | + | | | | | | | | | | + | +----+---+--------+ | | +-----+ | | +-+-----+-----+ | + | | R3(1) | | | |R3(2)| | | | R3(3) | | + | +-----------+-----+ | | +-+---+ | | +-----+-------+ | + | | | | | | | | | + +-----------------------+ +----------+ +-----------------+ + | bridge-net | | + +----------------------------+-------------------+ + + Figure.5 Cross OpenStack L2 network not able to be stretched into some region + + +Implementation +-------------- + +Local router: It's a router which is created with region name specified in the +availability zone hint, this will be present only in the specific region. + +East-west gateway router: It's a router which will be spread into multiple +regions and this will handle the east-west traffic to attached local networks. + +The following description of implementation is not pseudo code, it's the +logical judgemenet for different conditions combination. + +Adding router interface to east-west gateway router:: + + if IP of the router interface is the subnet default gateway IP + # north-south traffic and east-west traffic will + # go through this router + # router is the default router gateway, it's the + # single north-south external network mode + if the network is cross OpenStack L2 network + reserve gateway port in different region + add router interface in each region using reserved gateway port IP + make sure the gateway port IP is the default route + else # local network + add router interface using the default gateway port or the port + specified in request + else # not the default gateway IP in this subnet + if the network is cross OpenStack L2 network + reserve gateway port in different region + add router interface in each region using reserved gateway port IP + update host route in each connected local network in each region, + next hop is the reserved gateway port IP + else # local network + create router in the region as needed + add router interface using the port specified in request + if there are more than one interfaces on this router + update host route in each connected local network in each + region, next hop is port IP on this router. + + Configure extra route to the router in each region for EW traffic + +Adding router interface to local router for cross OpenStack L2 network will +make the local router as the default gateway router in this region:: + + # default north-south traffic will go through this router + add router interface using the default gateway port or the port + specified in request + make sure this local router in the region is the default gateway + +If external network is attached to east-west gateway router, and network's +default gateway is the east-west gateway router, then the router will be +upgraded to north-south networking via single external network mode. + +Constraints: + Network can only be attached to one local router in one region. + + If a network has already been attached to a east-west gateway router, + and the east-west gateway router is the default gateway of this network, + then the network can't be attached to another local router. + +.. note:: Host route update in a subnet will function only in next + dhcp request. It may take dhcp_lease_duration for VMs in the subnet + to update the host route. It's better to compose the networking + topology before attached VMs to the netwrok. dhcp_lease_duration is + configured by the cloud operator. If tenant wants to make the host + route work immediately, can send dhcp request directly in VMs. + + +Data Model Impact +================= + +None + +Dependencies +============ + +None + +Documentation Impact +==================== + +1. Add new guide for North South Networking via Multiple External Networks + with east-west enabled. +2. Release notes. + +Reference +========= + +.. [1] North South Networking via Multiple External Networks: https://docs.openstack.org/developer/tricircle/networking-guide-multiple-external-networks.html +.. [2] l3-networking-combined-bridge-net: https://github.com/openstack/tricircle/blob/master/specs/ocata/l3-networking-combined-bridge-net.rst +.. [3] North South Networking via Single External Network: https://docs.openstack.org/developer/tricircle/networking-guide-single-external-network.html