openstack/neutron-specs
Code Issues Proposed changes

Add spec for Distributed DHCP

Browse Source 
Closes-Bug: #1900934
Change-Id: I0427339489140457a2b56911cbabe74082c751c8
This commit is contained in:
LIU Yulong 2020-12-28 10:10:25 +08:00
parent 288f422f74
commit 3f645640d9
1 changed files with 352 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

352
specs/wallaby/distributed_dhcp.rst Normal file
View File

@ -0,0 +1,352 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
======================================
Distributed DHCP for Openvswitch Agent
======================================
RFE: https://bugs.launchpad.net/neutron/+bug/1900934
Neutron DHCP agents and the scheduled network instances are relatively
simple in function. But the configuration is complex, and it depends
on external process (dnsmasq) and namespace. When the user's demand is
merely unique, for example, they only need the DHCP response during the
virtual machine booting process, then the existing DHCP agent and its
configuration procedure for network and port make things complicated.
This spec describes how to implement a DHCP extension for Neutron
openvswitch agent to achive a simple and efficient solution for virtual
machine DHCP function by leveraging the openflow with openvswitch.
Problem Description
===================
Response the DHCP request is the main function for the scheduled network
instance of DHCP agent. Except this, it has other functions, like
isolated metadata and DNS lookup. But there are alternatives for these
extended functions, such as config drive for metadata and Designate
for DNS.
Then, the use frequency of the DHCP agent and its scheduled instance are
relatively low. It will be used only during the VM booting if no DNS lookup.
If you use config drive, aslo the scheduled DHCP instance is not useful.
And we have more problems for large scale clusters:
* The scheduled network instances of DHCP port increase the consuming of
L2 agent's capacity and performance.
* The DHCP provisioning block sometimes causes virtual machine booting
failure.
* Full sync in unknown reason causes the message queue, neutron-server and
DB in high load.
* DVR local router creation for the scheduled DHCP port is default behavior,
which increased resource load for L2 and L3 agents implicitly [1]_.
* Down a DHCP node causes long time recovery due to it has tons of scheduled
network instance.
And it is hard to find the balance between ``how many DHCP agents should the
deployment have`` and ``how many resources could one agent handle``. Too few
DHCP agent will finally make each one agent has a huge mount of resources.
Too many DHCP agents will increase maintenance pressure for the operators,
and make the centralized components overloaded.
There is a way to schedule one network to all DHCP agents on all compute
nodes. Firstly, this may work for tiny deployment with extremely few resources.
For large-scale deployment, it is basically impossible. Because there will be
tens of thousands of Networks in Neutron, this will directly lead to a surge of
resource pressure on each node, consume too many IP in user network,
the results basically are unable to operate, virtual machine startup failure
and unable to obtain IP and so on.
Proposed Change
===============
A new extension of Neutron openvswitch agent will be added to achieve the
``Distributed DHCP``.
.. note:: This extension is only for openvswitch agent, other mechanism drivers
will not be considered, because this new extension will rely on the
openflow protocol and principle. For OVN, it has supported similar
DHCP local response mechanism.
Solution Proposed
-----------------
As we know Neutron openvswitch agent has the entire information of the ports
which are pluged to the ovs bridge (If the port information was not
synchronized by the ovs-agent, there is a simple cache pull mechanism which
will fill the information). Neutron has the all conditions to support
distributed DHCP natively:
* ovs-agent is based on the python SDN controller ryu/os-ken
* ovs-agent is fully distributed
* ovs-agent has the entire resource information
So we can assume the Neutron openvswitch agent is a local SDN controller which
will try to response the VM's DHCP request. The basical data pipeline can be
described as this:
::
+---------+ +---------------------+
+-------+ DHCP Request | | packet-in | +----------------+ |
| VM +---------------> Flows +----------------> | os-ken app | |
| <---------------+ <----------------+ | | |
+-------+ DHCP Response | | packet-out | | DHCP Responder | |
| | | +----------------+ |
| br-int | | OVS-agent |
+---------+ +---------------------+
After this we will have:
* Higher level availability than DHCP agent with it's schedlued ``dnsmasq``,
DHCP requests are directly processed in the
computing nodes, it is completely distributed.
* No DHCP agent and its scheduling mechanism anymore
* No extra external process for DHCP anymore
* The (Neutron openvswith) agent downtime will no longer affect the
address acquisition and virtual machine startup in other nodes.
* Virtual machine startup will no longer be affected by port's DHCP
configuration, which reduces the probability of VM spawning failure.
* DHCP request and reponse for VM will achieve a high success rate.
DHCP(v4/v6) protocol options
++++++++++++++++++++++++++++
We will not repeat the DHCPv4 [2]_ and DHCPv6 [3]_ in details. But for
this spec, we will ensure the following features of the related protocol:
* DHCPv4 all message types
* DHCPv4 host Configurations
* DHCPv6 types Solicit, Advertise, Confirm, Renew, Rebind, Release and Reply
* DHCPv6 Options
For Neutron, we have some Port attributes like dns_domain, dns_name and
extra_dhcp_opts, and Subnet dns_nameservers, host_routes and gateway_ip
and so on, all these options will be added to the final DHCP
response like dnsmasq do.
Server side changes
+++++++++++++++++++
Based on the new added config options, some DHCP related DB options,
APIs, notifications and RPCs will be changed to no operation or be skipped.
The config options only controls Neutron itself, the DHCP protocol will have
no effect on. The final goal is to support the full features of DHCP protocol
defined. The changes are:
* disable DHCP scheduling mechanism and its failover forever
* disable DHCP provisioning block
* disable DHCP related RPCs and notificatons
OpenvSwitch Agent side changes
++++++++++++++++++++++++++++++
For Neutron openvswitch agent, we will add a new agent extension which will
process the basical flow installation for each port's further DHCP request.
There will be two basic flows which will direct DHCPv4 and DHCPv6 to independent tables.
``table 77`` is for DHCPv4, ``table 78`` is for DHCPv6. The flows are:
::
table=60, priority=101,udp,nw_dst=255.255.255.255,tp_src=68,tp_dst=67 actions=resubmit(,77)
table=60, priority=101,udp6,ipv6_dst=ff02::1:2,tp_src=546,tp_dst=547 actions=resubmit(,78)
For table 77, each DHCP request will be checked to verify the source mac and in_port in order
to avoid the DHCP spoofing. If the DHCP request is matched, then submit it to the controller,
aka the Neutron openvswitch agent. Any unmatched packets will be dropped. One example for a
VM's port is:
::
table=77, priority=100,udp,in_port="tapcc4f2da4-c5",dl_src=fa:16:3e:46:58:fe,tp_src=68,tp_dst=67 actions=CONTROLLER:0
table=77, priority=0 actions=drop
For table 78, DHCPv6 match and drop flows structure are basically same to DHCPv4:
::
table=78, priority=100,udp6,in_port="tapcc4f2da4-c5",dl_src=fa:16:3e:46:58:fe,tp_src=546,tp_dst=547 actions=CONTROLLER:0
table=78, priority=0 actions=drop
For the new extension of openvswitch agent, it will add a local
``packet_in_handler`` which will do the following works:
* Listen on the EventOFPPacketIn event
* Verify each packet to be DHCPv4 or DHCPv6
* According to the openflow inport number to retrieve the port's information
* Assemble the DHCP(v4/v6) response and ``packet_out`` to ``in_port``.
The response DHCP packet structure will be:
::
+------------------------------------------+
| *Source Mac Address |
|The gateway Port's MAC or A fake fixed MAC|
+------------------------------------------+
| *Destination Mac Address |
| Neutron Port Mac Address |
+------------------------------------------+
| *Source IP Address |
| Gateway IP address from Subnet |
+------------------------------------------+
| *Destination IP address |
| Neutron Port IP |
+------------------------------------------+
* ``Source Mac Address`` will be the internal subnet gateway port's Mac address.
But actually this is not necessary for the DHCP protocol, we can use a fake
fixed mac address to avoid some DB/RPC query.
* ``Destination Mac Address`` will be the port's MAC.
* ``Source IP Address`` will be the internal subnet's gateway IP.
* ``Destination IP address`` will be the port's first IP(v4/v6) address,
the secondary IPs will be ignored.
Potential configurations
++++++++++++++++++++++++
Config option ``disable_traditional_dhcp`` for neutron server side will be
added which is aiming to control:
* to disable DHCP scheduling for networks
* to disable DHCP provisioning block
* to disable DHCP RPC/notification
* to disable all DHCP related API/attibutes network, subnet and port.
A new extension alias name ``dhcp`` will be added for neutron
openvswitch agent:
::
[agent]
extensions = ...,dhcp
Config section ``[dhcp]`` will be added for neutron openvswitch agent
and register some common options to determine DHCP protocol related
parameters, the final ``[dhcp]`` section for openvswitch agent will be:
::
dhcp_opts = [
cfg.BoolOpt('enable_dhcp_ipv6', default=False,
help=_("Whether enable DHCP for IPv6")),
cfg.IntOpt('dhcp_renewal_time', default=0,
help=_("DHCP renewal time T1 (in seconds). If set to 0, it "
"will default to half of the lease time.")),
cfg.IntOpt('dhcp_rebinding_time', default=0,
help=_("DHCP rebinding time T2 (in seconds). If set to 0, it "
"will default to 7/8 of the lease time.")),
]
The Neutron basic workflow
--------------------------
1. User creates a VM in a network
2. Nova plug the VM's NIC port to ovs-bridge
3. Ovs-agent process the port and install the DHCP related flows
4. L2 provisioning block released (No DHCP provisioning block)
5. VM booting and send DHCP request out
6. Match the flows and ``packet_in`` to ovs-agent
7. Ovs-agent directly send DHCP(v4/v6) response to VM's port
8. VM booting success
Data Model Impact
-----------------
None
REST API Impact
---------------
With the new config options, the following APIs will be disabled [4]_:
* add_network_to_dhcp_agent
* remove_network_from_dhcp_agent
* list_networks_on_dhcp_agent
* list_dhcp_agents_hosting_network
For the option ``enable_dhcp`` of ``Subnet``, this agent extension will set
the flows based on that. If it is False, ports under this subnet will have
no flows installed in table 77 and 78. The DHCP request will hit the final
DROP action.
Upgrading
---------
For native ml2/ovs deployment, this feature will be easily to upgrade to
enforce. A simple way is to run all agents as they are. But disable the DHCP
provisioning block. After enable the ``dhcp`` extension for ovs-agent, the
DHCP request will be handled by it earlier than ``dnsmasq``.
If you need a pure deployment without DHCP agents, the following is an overview
about how to migrate to use this new feature:
* Upgrading the Neutron code and restart neutron-server processes.
* Setup the ovs-agent with ``dhcp`` extension.
* Disable all DHCP agents to make sure no more scheduled network are created.
::
openstack network agent set --disable <dhcp_agent_id>
.. note:: This action will remove all scheduled network instances from
the admin state DOWN DHCP agent.
* Set the ``disable_traditional_dhcp = True`` option for neutron-server to
disable the scheduling related API/RPCs.
* (optional, just in case) Remove all scheduled network from all DHCP agents,
this step is to pure all DHCP namespace and DHCP woker process ``dnsmasq``.
* After no more scheduled network, stop all DHCP agents and remove it from DB.
.. note:: This feature does not support DNS lookup. If your running deployments
are using the DNS lookup function from ``dnsmasq``, consider use
``designate`` as an alternative.
Implementation
==============
Assignee(s)
-----------
* LIU Yulong <i@liuyulong.me>
Work Items
----------
* Config options for neutron server to control DHCP related codes.
* Create agent extension.
* Testing.
* Documentation.
Dependencies
============
None
Testing
=======
Functionality
-------------
We will add fullstack test case to verify this new agent extension:
* Create two fake fullstack VMs in two test namespaces
* Use DHCP(v4 and v6) to config the fake VM ports
* Ping (-4/6) from one fake VM to another
References
==========
.. [1] https://review.opendev.org/c/openstack/neutron/+/364793
.. [2] https://tools.ietf.org/html/rfc2131
.. [3] https://tools.ietf.org/html/rfc8415
.. [4] https://review.opendev.org/c/openstack/neutron/+/772255/8/neutron/extensions/dhcpagentscheduler.py#106-126