openstack
/
neutron-specs
Code Issues Proposed changes

Allow more flexible connection to external network 

This feature will add an interface and driver mechanism to make
plugging in to an external network more flexible.  This will enable
solving problems with scalability and public IP address waste.

bp pluggable-ext-net

Change-Id: I5074548d4d18f5086bb2114597278a456f9dc3ea
This commit is contained in:
Carl Baldwin 2014-04-04 21:14:44 -06:00
parent d76bec4015
commit 1430c6b008
1 changed files with 650 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

650
specs/juno/pluggable-ext-net.rst Normal file
View File

@ -0,0 +1,650 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
======================
Pluggable External Net
======================
https://blueprints.launchpad.net/neutron/+spec/pluggable-ext-net
:Author: Carl Baldwin <carl.baldwin@hp.com>
:Copyright: 2014 Hewlett-Packard Development Company, L.P.
Plugging a router external gateway to an external network in the L3 agent can
only be done in one way. It was designed to operate on a single L2 network
with little configuration on the upstream router. But, it has some weaknesses.
This blueprint will add a driver interface so that other implementations can be
written to overcome its weaknesses.
Problem description
===================
In Neutron, an external network is hosted on a single L2 network. Routers
connect directly and use arping to announce their ips. This could limit the
scalability or flexibility of the external network. For example, floating ips
are not free to float between L2 networks.
Another weakness is that each Neutron router consumes a public IP address that
is dedicated to it in addition to any floating ips it hosts.
The DVR [#]_ blueprint adds an additional requirement of an extra namespace on
each compute host in the datapath. This namespace consumes an extra public IP
address for each compute host in the cloud. No traffic should ever be sourced
using this IP and so these IP addresses are essentially wasted.
.. [#] https://blueprints.launchpad.net/neutron/+spec/neutron-ovs-dvr
Proposed change
===============
Overview
--------
This blueprint is to abstract the concept of plugging in to an external network
to allow for more flexibility.
#. Reduce or eliminate unnecessary public IP address consumption for
infrastructure.
#. Allow public IP pools to span L2 networks in some scenarios.
#. Allow other forms of address advertisement.
.. TODO These are likely to be done before this blueprint
#. Use multiple arbitrary IP subnets on the external network [#]_.
#. Allow the use of a provider network as an external network [#]_.
.. [#] https://bugs.launchpad.net/neutron/+bug/1212947
.. [#] https://bugs.launchpad.net/neutron/+bug/1056437
.. TODO https://blueprints.launchpad.net/fuel/+spec/separate-public-floating
What is an External Network?
____________________________
To abstract the concept of plugging in to an external network it is important
to understand what it is used for in an OpenStack cloud. Plugging in to an
external network is all about getting connectivity outside of the cloud at L3
and above. That an external network's subnet sits on top of a single L2
network is really just an implementation detail.
An external network ...
#. Provides a gateway for traffic to escape the cloud.
#. Hosts one or more "public" IP address pools by which compute instances can
be reached.
#. Announces addresses externally (e.g. arping)
.. This was work that I thought would need a blueprint to do. However, I
played around with it and discovered that the current L3 can *almost* do
it. I think I'd like to do this as a bug instead of in the blueprint
Multiple Subnets
----------------
One important capability that this blueprint will add is the ability for a
single external network to host more than one subnet and more than one
floating ip pool on each subnet. This enables the full flexibility of the
Neutron model for networks, subnets, and allocation pools for external
networks.
.. TODO neutron subnet-create ext-net XX.YY.ZZ.0/24 --disable-dhcp
Works but I'm not sure where things break down.
I was able to create a floating ip on this second subnet and even
associate the floating ip with an instance.
I watched the floating ip address get added to the qg interface. I did
not expect this to work.
The floating IP is useable from inside the router namespace. The
floating IP appeared to work all the way through once I added a route to
the default namespace in devstack:
sudo ip addr add 10.224.24.1/24 dev br-ex
I did some tcpdumps. Saw this in both router and default namespaces.
19:06:13.375330 ARP, Request who-has 10.224.24.2 tell 172.24.4.1, length 28
19:06:13.375340 ARP, Reply 10.224.24.2 is-at fa:16:3e:f9:c3:f9 (oui Unknown), length 28
The question is what happens with traffic to another FIP on the subnet
where the router would send it to the upstream router because it doesn't
have an on-link route. I tried this out and found that
Eliminating the Need for a Dedicated Router IP
______________________________________________
Currently in Neutron, each router gets a primary address on the public network.
This address is currently needed for a number of purposes. The router's
dedicated primary public IP ...
#. provides a Default SNAT address for instances without a floating IP.
- For DVR, this is only needed on the central component. Not the IRs.
- It can be turned off router-by-router.
- There may be alternatives to default SNAT that have not yet been
considered.
#. serves as the `Router Primary Address`_ for the router gateway port on the
subnet.
#. allows DNS queries passed through from the dnsmasq instance running in the
DHCP namespace to access an Internet accessible upstream DNS server..
- DNS passthrough could be configured to use an address on a private subnet.
A DNS service needs to be configured that will listen on the
`Private Subnet`_.
#. allows for troubleshooting (e.g. sending pings to and from the router
namespaces.)
- This could be done using a `Private Subnet`_.
The goal is to avoid allocating a dedicated IP to the router if it isn't
absolutely necessary. This will save IP space where there is a crunch.
Eliminating the public IP address entirely may require more work that is out of
the scope of this blueprint.
Router Primary Address
~~~~~~~~~~~~~~~~~~~~~~
This section describes a technique to eliminate the waste of public addresses
as primary interface addresses.
In Linux, when an interface has multiple addresses on the same subnet, one
becomes the primary address and the others secondary. An interesting thing to
note is that when the primary address is removed, all of the secondary
addresses are removed with it [#]_.
.. [#] The sysctl setting net.ipv4.conf.*.promote_secondaries overrides this
behavior.
A common practice is to allocate a primary address to the interface using a /NN
subnet prefix (e.g. /21) and then add floating or virtual IPs as secondary IP
addresses using the same prefix. The primary address is treated as permanent.
Another similar practice is to add extra addresses using /32 after adding the
primary. Neutron does this. In this scenario, the primary address is needed
for two reasons.
.. There is some code in the L3 agent that uses this fact to distinguish
floating IPs by the /32 prefix.
#. It generates a corresponding on-link route for the subnet.
#. It is used as the source address for traffic originating from the host.
An on-link route can be added without an address::
ip route add 10.1.1.0/24 dev eth0 scope link
To provide a default source address, only one subnet (**primary**) on the
network needs an address. If the router itself does not source traffic to the
Internet, this primary subnet can be a private non-routable subnet. All other
subnets (**secondary**) can be added using on-link routes. Floating IPs can be
added and used as they are today.
This technique brings the flexibility to operate a router without a dedicated
public primary address.
Private Subnet
~~~~~~~~~~~~~~
To serve some of these needs, a private non-routable subnet can be used as the
primary subnet. Care should be taken to avoid creating a security problem by
introducing this new subnet.
#. It must not open up access to any under-cloud resources.
#. It must not inadvertently allow routing or NAT to the Internet.
#. It must not open up access to other routers owned by other tenants.
This subnet could be used to intentionally allow compute instances access to
some infrastructure resources even where it does not have an associated public
IP address or default public SNAT is not enabled. For example, it could
provide ...
#. A DNS service accessible by compute instances.
#. A HTTP proxy to the Internet as an alternative to default SNAT.
#. Repository mirrors for guest operating systems.
#. License Servers that should be accessible by compute instances.
Default SNAT
~~~~~~~~~~~~
Default SNAT is currently dependent on the assumption that a `Router Primary
Address`_ is allocated from the public IP space. It piggy-backs off of that
and uses it as its source address.
This blueprint changes that assumption. It particular, it will allow the
flexibility to avoid allocating a public IP address for a router if not needed.
The code around default SNAT will need to change to operate given the new
rules. If it is enabled on a router, then a check will need to be performed to
see if a (not-floating) public IP is already available for use. If not, it
will take action to allocate one.
Driver Interface
________________
A new driver interface needs to be added to the L3 agent and integrate with the
code. This new interface will support the following operations. A reference
implementation will be developed along with the interface to preserve the
current behavior of the code.
======================= ==========================================
Call Inputs
======================= ==========================================
Host Setup network, subnets, driver_config
Host Teardown network, subnets, driver_config
Plug External Gateway network, subnets, gw_port, driver_config
Unplug External Gateway network, subnets, gw_port, driver_config
Connect Floating IP network, subnets, gw_port, driver_config
Disconnect Floating IP network, subnets, gw_port, driver_config
======================= ==========================================
Implementations should take care to only make changes when necessary. For
example, if the agent gets restarted, the implementation should efficiently
detect whether things are already in the right state and do nothing in that
case. It would be unacceptable to tear down the connection and rebuild it on
an agent restart or to remove a floating ip and add it back if it was already
operational. These operations cause disruption in service.
Host Setup/Teardown
~~~~~~~~~~~~~~~~~~~
Each driver will implement a call to setup a connection to an external network
on the host when the network is associated with an agent. Also, a
corresponding call to tear it down.
The following are possible actions performed as part of this step.
#. Create or ensure the existence of a bridge or other network plumbing.
#. Create a namespace for the external network if one is required.
#. Make calls to RPC to allocate addresses if needed.
Plug/Unplug External Gateway
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is called to plug or unplug a router in to an external network when the
router's external gateway is set or cleared.
There needs to be an API server component to this. The driver may determine
how to allocate L2 and L3 addresses for the gateway on the API side.
Connect/Disconnect Floating IP
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Called to associate a floating IP with a given gateway port.
This does not include setting up the DNAT and SNAT rules to associate the
floating IP with an internal port. That will remain outside the driver for
now.
Implementations
_______________
=================== =========== ================= ============ =================
_ `Direct`_ `Direct Private`_ `Routed`_ `Routed Private`_
=================== =========== ================= ============ =================
Extra Address Per Router/Host None Compute Host None
Announcement arp arp proxyarp proxyarp
Private Subnet None Yes None Yes
Extra Router Config None Private Subnet None Private Subnet
...
=================== =========== ================= ============ =================
The following implementations may not all be completed as part of this
blueprint. They will be implemented as needed. There are other possible
implementations which are not discussed here.
Direct
~~~~~~
This will be the initial reference implementation. It works exactly the same
way as external networks work today in Neutron. It will be the default for the
basic (non-DVR) router and will require no difference in configuration from
today. Upgrading a working external network to this model will be seamlessly
done with a code update.
Given this implementation, an easy derivative implementation could be easily
created to experiment with using a macvtap to connect to the network rather
than a veth pair.
Direct Private
~~~~~~~~~~~~~~
This implementation will build from the `Direct`_ implementation. It uses a
private subnet on link that will not be visible to tenants through the API.
The notable difference is that this implementation does not require a public IP
address dedicated to the router.
================= =================================
Driver specific configuration
---------------------------------------------------
Field Description
================= =================================
cidr The private subnet address space.
gateway The router's gateway address.
allocations_pools
================= =================================
In order to use Neutron's IPAM capabilities for the allocation of IPs on the
private subnet, it will be necessary to create a subnet linked to the network
that is only available to the driver. This subnet should be created in a way
that it will be unavailable when allocating ports or floating ips on the
network. How to do this is an open question.
Requires the configuration of an extra private subnet with gateway on the
upstream router.
Replaces `Direct`_ and `Routed`_ for all DVR and non-DVR router components.
If a Neutron router is connected directly to both the tenant's internal
networks and a private non-routable external subnet then there is a potential
address space conflict. The solution to this problem is to make use of the
"Shared Address Space" allocated by IANA in 2012 for applications like carrier
grade NAT [#]_. Using this space will guarantee no conflict with tenant
networks.
.. [#] http://tools.ietf.org/html/rfc6598
..
Shared Address Space is distinct from RFC 1918 private address space
because it is intended for use on Service Provider networks.
Routed
~~~~~~
The `Direct`_ implementation presented a problem for the DVR blueprint because
each IR would need to have its own address. This would have meant consuming
one address for every router/host combination. That would be unacceptable.
To solve this, a slightly different method for connecting each IR to the
external net was devised. This method adds a new namespace on each compute
host to route floating IP traffic to Neutron routers. The namespace is
configured with proxyarp for the IPs it hosts. It consumes a single extra IP
address from the public IP pool for each compute host to use as the primary
address on the public interface.
This will be the default connection for each DVR IR component. Upgrading a DVR
IR to this model will be seamless and not require any extra configuration. If
DVR has not been released and is not in wide use by the time this blueprint is
implemented then it may not be necessary to implement this at all.
Routed Private
~~~~~~~~~~~~~~
This method is to `Routed`_ as `Direct Private`_ is to `Direct`_. It uses a
private primary subnet to avoid the need to allocate public IP addresses for
compute hosts. It adds an extra namespace per host for routing and proxyarp in
the same manner.
The driver specific configuration is the same as for `Direct Private`_.
IPs will be allocated from the private subnet in the same way as in `Direct
Private`_
Requires the configuration of an extra private subnet with gateway on the
upstream router.
Replaces `Routed`_ for all DVR IR components.
Dynamic Routed
~~~~~~~~~~~~~~
This blueprint allows the flexibility to implement a driver that uses dynamic
routing on the external network to announce public IP addresses. The
implementation is out of this scope.
Alternatives
------------
I haven't thought of any yet.
Data model impact
-----------------
The database will need to store a driver association for each network/host.
This association will need to carry some configuration specific to the driver.
=========== ========================
Table: **external_network_driver**
------------------------------------
Column Description
=========== ========================
network_id External Network Id
host_id Host ID
driver The driver class to use
config Driver specific config
=========== ========================
There are a few reasons to include the host_id instead of just the network. A
row with null for host_id can be used as the default driver configuration.
#. With the initial DVR implementation, IRs are connected using the `Routed`_
model but the central component of a DVR router may be connected using a
`Direct`_ connection.
#. Different hosts can be connected to different L2 networks.
#. This will allow for controlled migrations where a new plugin can be tested
on a small subset of hosts before being rolled out to the rest.
Rolling Upgrades
________________
During a rolling transition to a new implementation, there will be some Neutron
Network and Compute hosts that still use the old implementation and some that
have the new. Even inside of a single host it will take the L3 agent some time
to iterate through the routers and rewire them.
Each implementation should work out a method to transition seamlessly from at
least the `Direct`_ implementation so that routers can connect to the same
external network using either implementation during the transation and still
maintain connectivity.A brief interruption in network connectivity through the
router may be tolerable for each router. It should not be long enough that TCP
connections through the router will be interrupted.
If DVR is in wide use before this is implemented, implementations should do the
same from `Routed`_.
REST API impact
---------------
The API will be extended to allow configuring the driver for "router:external"
networks for given hosts. Needs driver specific parameters. If the extension
is not used, a default driver will be used to mimic today's behavior.
This section is a work-in-progress.
.. TODO What extensions are there that I can look at. router:external
Notifications impact
--------------------
The name of the driver and a configuration dictionary will be returned with the
router data when the L3 agent requests them. I'm not certain if this
constitutes an RPC version change or not. This will be investigated.
::
************************************************************************
************************************************************************
************************************************************************
************************************************************************
************************************************************************
************************************************************************
************************************************************************
************************************************************************
Security impact
---------------
Describe any potential security impact on the system. Some of the items to
consider include:
* Does this change touch sensitive data such as tokens, keys, or user data?
* Does this change alter the API in a way that may impact security, such as
a new way to access sensitive information or a new way to login?
* Does this change involve cryptography or hashing?
* Does this change require the use of sudo or any elevated privileges?
* Does this change involve using or parsing user-provided data? This could
be directly at the API level or indirectly such as changes to a cache layer.
* Can this change enable a resource exhaustion attack, such as allowing a
single API interaction to consume significant server resources? Some examples
of this include launching subprocesses for each connection, or entity
expansion attacks in XML.
For more detailed guidance, please see the OpenStack Security Guidelines as
a reference (https://wiki.openstack.org/wiki/Security/Guidelines). These
guidelines are a work in progress and are designed to help you identify
security best practices. For further information, feel free to reach out
to the OpenStack Security Group at openstack-security@lists.openstack.org.
Other end user impact
---------------------
Aside from the API, are there other ways a user will interact with this feature?
* Does this change have an impact on python-neutronclient? What does the user
interface there look like?
Performance Impact
------------------
Describe any potential performance impact on the system, for example
how often will new code be called, and is there a major change to the calling
pattern of existing code.
Examples of things to consider here include:
* A periodic task might look like a small addition but if it calls conductor or
another service the load is multiplied by the number of nodes in the system.
* A small change in a utility function or a commonly used decorator can have a
large impacts on performance.
* Calls which result in a database queries (whether direct or via conductor) can
have a profound impact on performance when called in critical sections of the
code.
* Will the change include any locking, and if so what considerations are there on
holding the lock?
Other deployer impact
---------------------
Discuss things that will affect how you deploy and configure OpenStack
that have not already been mentioned, such as:
* What config options are being added? Should they be more generic than
proposed (for example a flag that other hypervisor drivers might want to
implement as well)? Are the default values ones which will work well in
real deployments?
* Is this a change that takes immediate effect after its merged, or is it
something that has to be explicitly enabled?
* If this change is a new binary, how would it be deployed?
* Please state anything that those doing continuous deployment, or those
upgrading from the previous release, need to be aware of. Also describe
any plans to deprecate configuration values or features. For example, if we
change the directory name that instances are stored in, how do we handle
instance directories created before the change landed? Do we move them? Do
we have a special case in the code? Do we assume that the operator will
recreate all the instances in their cloud?
Developer impact
----------------
Discuss things that will affect other developers working on OpenStack,
such as:
* If the blueprint proposes a change to the API, discussion of how other
plugins would implement the feature is required.
Implementation
==============
Assignee(s)
-----------
Who is leading the writing of the code? Or is this a blueprint where you're
throwing it out there to see who picks it up?
If more than one person is working on the implementation, please designate the
primary author and contact.
Primary assignee:
<launchpad-id or None>
Other contributors:
<launchpad-id or None>
Work Items
----------
Work items or tasks -- break the feature up into the things that need to be
done to implement it. Those parts might end up being done by different people,
but we're mostly trying to understand the timeline for implementation.
Dependencies
============
* Include specific references to specs and/or blueprints in neutron, or in other
projects, that this one either depends on or is related to.
* If this requires functionality of another project that is not currently used
by Neutron (such as the glance v2 API when we previously only required v1),
document that fact.
* Does this feature require any new library dependencies or code otherwise not
included in OpenStack? Or does it depend on a specific version of library?
Testing
=======
Please discuss how the change will be tested. We especially want to know what
tempest tests will be added. It is assumed that unit test coverage will be
added so that doesn't need to be mentioned explicitly, but discussion of why
you think unit tests are sufficient and we don't need to add more tempest
tests would need to be included.
Is this untestable in gate given current limitations (specific hardware /
software configurations available)? If so, are there mitigation plans (3rd
party testing, gate enhancements, etc).
Documentation Impact
====================
What is the impact on the docs team of this change? Some changes might require
donating resources to the docs team to have the documentation updated. Don't
repeat details discussed above, but please reference them here.
References
==========
Please add any useful references here. You are not required to have any
reference. Moreover, this specification should still make sense when your
references are unavailable. Examples of what you could include are:
* Links to mailing list or IRC discussions
* Links to notes from a summit session
* Links to relevant research, if appropriate
* Related specifications as appropriate (e.g. link any vendor documentation)
* Anything else you feel it is worthwhile to refer to