11 KiB

Raw Blame History

Tricircle Local Neutron Plugin

Background

One of the key value we would like to achieve via the Tricircle project is to provide networking automation functionality across several Neutron servers. Each OpenStack instance runs its own Nova and Neutron services but shares the same Keystone service or uses federated Keystone, which is a multi-region deployment mode. With networking automation, virtual machines or bare metals booted in different OpenStack instances can inter-communicate via layer2 or layer3 network.

Considering the cross Neutron layer2 network case, if Neutron service in each OpenStack instance allocates ip address independently, the same ip address could be assigned to virtual machines in different OpenStack instances, thus ip address conflict could occur. One straightforward solution to this problem is to divide the ip allocation pool into several parts and each OpenStack instance has one. The drawback is that since virtual machines are not distributed evenly in each OpenStack instance, we may see some OpenStack instances uses up ip addresses while other OpenStack instances still have ip addresses not allocated. What's worse, dividing the ip allocation pool makes it impossible for us to process virtual machine migration from one OpenStack instance to another.

Thanks to Neutron's flexible plugin framework, by writing a new plugin and configuring Neutron server to use it, developers can define what Neutron server should do after receiving a network resources operation request. So for the ip address conflict issue discussed above, we decide to run one central Neutron server with the Tricircle central Neutron plugin(abbr: "central plugin") to manage ip allocation pool centrally.

Besides central plugin, we need a bridge to connect central and local Neutron servers since each OpenStack instance has its own local Nova and Neutron server but these two services are not aware of the central Neutron server. This bridge should validate requested network data via the central Neutron server, then create necessary network resources in the target OpenStack instance with the data retrieved from the central Neutron server.

Local Plugin

For connecting central and local Neutron servers, Neutron plugin is again a good place for us to build the bridge. We can write our own plugin, the Tricircle local Neutron plugin(abbr: "local plugin") to trigger the cross Neutron networking automation in local Neutron server. During virtual machine booting, local Nova server will interact with local Neutron server to query network or create port, which will trigger local plugin to retrieve data from central Neutron server and create necessary network resources according to the data. To support different core plugins, we will introduce a new option "real_core_plugin" in the "tricircle" configuration group. During initialization, local plugin will load the plugin specified by "real_core_plugin". Local plugin only adds logic to interact with central Neutron server, but invokes the real core plugin to finish the CRUD operations of local network resources. The following graph shows the relation between user and Nova and Neutron servers: :

+------+
| user |
+-+--+-+
  |  |
+-----------+  +----------------------+
|  boot vm         create and query   |
|                  network resource   |
v                                     |
+----+-------+                             |
| local Nova |     xxxxxxxxxxxxxxx         |
+----+-------+   xxx             xxx       |
|          xx                 xx      |
+---+    xxx      +--------+   xxx    |
|    x        |        |     x    |
|    x        |        |     x    |
v    V        |        v     x    v
+--------+---------+   |   +----+----------+----+
| local Neutron    |   |   | central Neutron    |
| +--------------+ |   |   | +----------------+ |
| | local plugin | |   |   | | central plugin | |
| +--------------+ |   |   | +----------------+ |
+------------------+   |   +--------------------+
|             |
+-------------+

Next using virtual machine booting procedure to elaborate how local plugin works. To begin with, user creates network and subnet via central Neutron server. Then this user passes the network id as the requested network information to local Nova server to boot a virtual machine. During parameter validation, local Nova server queries local Neutron server to ensure the passed-in network id is valid, which is a "network-get" request. In the "network-get" handle function, local plugin first checks if local Neutron already has a network with that id. If not, local plugin retrieves network and also subnet information from central Neutron server then creates network and subnet based on this information. User may pass an invalid network id by mistake, in this case, local plugin will receive a 404 response from central Neutron server, it just returns a 404 response to local Nova server.

After the network id validation passes, local Nova server continues to schedule a host so compute manager running in that host will do the left works. Compute manager creates a port in the requested network via local Neutron server, which is a "port-create" request. In the "port-create" handle function, local plugin sends the same request to central Neutron server to create a port, and uses the returned port information to create a local port. With local plugin, we ensure all ip addresses are allocated by central Neutron server.

At the end of the network setup of the virtual machine, compute manager issues a "port-update" request to local Neutron server to associate the host with the port. In the "port-update" handle function, local plugin recognizes that this request is sent from local Nova server by the request body that the request body contains host information, so it sends a "port-update" request to central Neutron server with region name in the request body. In Keystone, we register services inside one OpenStack instance as one unique region, so we can use region name to identify one OpenStack instance. After receiving the request, central Neutron server is informed that one virtual machine port is correctly setup in one OpenStack instance, so it starts the cross Neutron networking automation process, like security group rule population, tunnel setup for layer2 communication and route setup for layer3 communication, which are done by making Neutron API call to each local Neutron server.

Implementation

Implementation details of the local plugin is discussed in this section.

Resource Id

Local plugin always retrieves data of networks resources from central Neutron server and use these data to create network resources in local Neutron server. During the creation of these network resources, we need to guarantee resource ids in central and local server the same. Consider the scenario that user creates a port via central Neutron server then use this port to boot a virtual machine. After local Nova server receives the request, it will use the port id to create a tap device for the virtual machine. If port ids in central and local Neutron servers are different, OVS agent can't correctly recognize the tap device and configure it. As a result, virtual machine fails to connect to the network. Fortunately, database access module in Neutron allow us to specify id before creating the resource record, so in local plugin, we just specify id the same as central resource's to create local resource.

Network Type Adaption

Two network types are supported currently in central plugin, which are local and vlan type. Before creating network based on information retrieved from central Neutron server, local plugin needs to adapt network type. For local type, local plugin creates the network without specifying the network type, so the default tenant network type is used. For vlan type, local plugin keeps the network type, segmentation id and physical network parameter.

We plan to support another two network types later. They are shared_vxlan and mixed network type. For shared_vxlan type, local plugin changes the network type parameter from "shared_vxlan" to "vxlan", but keeps the segmentation id parameter(vxlan type doesn't need physical network parameter). For mixed type, like local type, local plugin uses the default tenant network type to create the network, but it needs to do one more thing, that is to save the segment information in central Neutron server. Neutron has a extension which allows one network to carry multiple segments information[1], so segment information of each local network can all be saved in the central network.

Dhcp Port Handle

After local subnet creation, local Neutron server will schedule one dhcp agent for that subnet, and dhcp agent will automatically create a dhcp port. The ip address of this dhcp port is not allocated by central Neutron server, so we may encounter ip address conflict. We need to address this problem to ensure all ip addresses are allocated by central Neutron server.

Here is the approach. After central Neutron server receives subnet creation subnet, central plugin not only creates the requested subnet, but also create a port to pre-allocate an ip address for the dhcp port. So during creation of local subnet, local plugin will query central Neutron server to retrieve the data of the pre-created port and use its ip address to create a local dhcp port. The "device_id" of the dhcp port is set to "reserved_dhcp_port" so after one dhcp agent is scheduled, it will use this port other than create a new one.

Gateway Port Handle

If cross Neutron layer2 networking is enabled in one network, we need to allocate one gateway ip for that network in each OpenStack instance. The reason is that we want layer3 routing to be finished locally in each OpenStack instance. If all the OpenStack instances have the same gateway ip, packets sent to the gateway may reach the remote one, so the path is not the best and not predictable.

How we address this problem in local plugin is that before creating local subnet, local plugin sends request to central Neutron server to create an "gateway port", then uses the ip of this port as the gateway ip of the local subnet. Name of the gateway port includes the region name of the OpenStack instance and the id of the subnet so each OpenStack instance can have its own gateway port and gateway ip for one specific subnet.

Data Model Impact

None

Dependencies