Support security group rules with remote_address_group_id in openvswitch
firewall. This change reuses most of the firewall functions handling remote
security groups to also process remote address groups. The conjunctive flows
for a rule with remote_adress_group_id are similar to others with
remote_group_id but have different conj_ids.
Implements: blueprint address-groups-in-sg-rules
This change adds code to retrieve for the agent the security group ids
affected by an update or deletion of an address group.
Also adds event notificatoins to add and remove addresses from address
Co-authored-by: Hang Yang <firstname.lastname@example.org>
A partial upgrading of neutron cluster, neutron-server
has a newer version while neutron-agents not, does not
run well after a RPC data structure upgrading. This
patch upgrades the security group related RPC version
between neutron-server and agents. A partial upgrading
neutron cluster will explicitly raise error. The RPC
version should be aligned.
As spotted in Focal testing patch , pep8 test fails with many
C0321 false-positives, reported in pylint as current version does not
support python 3.8 
Use a newer version of pylint and astroid, fixing or disabling some of
the new checks: no-else-*, unnecessary-comprehension, import-outside-toplevel
When add allowed-address-pair 0.0.0.0/0 to one port, it will
unexpectedly open all others' protocol under same security
group. IPv6 has the same problem.
The root cause is the openflow rules calculation of the
security group, it will unexpectedly allow all IP(4&6)
traffic to get through.
For openvswitch openflow firewall, this patch adds a source
mac address match for the allowed-address-pair which has
prefix lenght 0, that means all ethernet packets from this
mac will be accepted. It exactly will meet the request of
accepting any IP address from the configured VM.
Test result shows that the remote security group and
allowed address pair works:
1. Port has 0.0.0.0/0 allowed-address-pair clould send any
IP (src) packet out.
2. Port has x.x.x.x/y allowed-address-pair could be accepted
for those VMs under same security group.
3. Ports under same network can reach each other (remote
4. Protocol port number could be accessed only when there
has related rule.
All of the externally consumed variables from neutron.common.constants
now live in neutron-lib. This patch removes neutron.common.constants
and switches all uses over to lib.
The neutron.common.rpc module has been in neutron-lib for awhile now and
neutron is shimmed to use neutron-lib already.
This patch removes neutron.common.rpc and switches the code over to use
neutron-lib's implementation where needed.
The neutron.common.topics module was rehomed into neutron-lib with
This patch consumes it by removing the rehomed module from neutron
and using the module from neutron-lib instead.
Changing rpc_api.rst file path from doc/source/devref/rpc_api.rst
to /doc/source/contributor/internals/rpc_api.rst. Because rpc_api.rst
file is located at this path
In Pike, the agent side of security_groups_provider_updated()
RPC code was changed to a NOOP when the provider rules were
changed to be static, https://review.openstack.org/#/c/432506
Now that we're in Queens we can deprecate it.
SecurityGroupServerRpcApi and SecurityGroupServerRpcCallback are a pair,
and SecurityGroupAgentRpcApiMixin and SecurityGroupAgentRpcCallbackMixin
should be a pair.
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Firewall drivers check if port security is enabled. After ovo is sent
over the wire, the port_security_enabled is part of 'security' field.
The patch translates the RPC call from agent to server so the payload
containing port_security_enabled is at the same place.
We may consider implementing change of OVO field to contain boolean
neutron-lib 1.6.0 is out and among other things contains the rehomed
is_port_trusted function. This patch switches usage of that function
from neutron to neutron-lib.
Setting up rules to allow DHCPv6, DHCP, and RAs from specific
IP addresses based on Neutron resources has a few issues:
1. It violates separation of concerns. We are implementing logic to
calculate where an IPv6 RA advertisement or DHCP advertisement
should be coming from in the security group code. This code should
not be trying to guess IPv6 LLAs, know about subnet modes, DHCP server
implementations, or the type of L3 plugin being used. Currently all
of these assumptions are baked into code that should only be
filtering, which makes it very rigid and brittle when it comes to
other implementations for DHCP and/or RAs.
2. It has scaling issues on large networks. Every time one of these
provider rules is updated, it triggers every L2 agent to refresh
all of the security group rules for ports in that network, which puts
significant load on the server.
3. It's main purpose: preventing spoofing of RA[1,2] and DHCP packets,
has long been superceded by preventing VMs from acting as DHCP/RA
This patch completely removes all of this logic and just returns
static provider rules to the agents that allow all DHCP server
and RA traffic ingress to the client. This addresses the issues
highlighted above since the code is significantly simplified and
the provider rules don't require refreshes on the agents.
Now that the provider rules never change, the RPC notification
listener on the agent-side for 'notify_provider_updated' is now
just a NOOP that doesn't trigger any refreshes. The notification
was left in place on the server side for older version agents
that have stale IP-specific provider rules. The entire notification
can be removed in the future.
The one open concern with this approach is that VMs will now be
able to receive DHCP offers from other DHCP servers on the same
network that aren't being filtered (e.g. a VM with port security
disabled or another device on a provider network). In order to
address this for DHCP, this patch adds two rules that only allow
DHCP offers targeted to either the broadcast or the correct client
IP. This prevents incorrect offers from ever reaching the client.
For RAs, this patch just allows all RAs so we may pick up
advertisements from other v6 routers attached to a network;
however, the instance won't actually be allowed to use bad addresses.
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.
Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.
This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.
Usage pattern [1,2] can be translated to [3,4] respectively.
The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.
Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.
Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.
Partially-implements: blueprint neutron-lib
Initially, we bumped the required version for the agent endpoint from
1.1 (the initial version that implemented security groups) to 1.3
without considering that the code should work with old agents that do
not yet know about the new devices_to_update argument.
Actually, there was no need to bump the version: old agent side code
already captures all unknown arguments that could be passed from the
server, ignoring them:
(^ the link shows the latest Kilo code as of writing)
Note: some people may argue that the approach that is taken in Neutron
to support backwards compatibility for server notifications is wrong,
and we instead should adopt some stricter mechanism like nova version
pinning. While that is a noble thing to do, it's out of scope for the
patch that is designed to be easily backportable to stable/liberty.
Note: some people may also argue that the patch should go straight into
stable/liberty because we don't claim support for rolling upgrade
scenarios that span multiple releases. That's indeed true, though my
take on it is that if we have a way to handle more unofficial scenarios
without more coding effort, it's worth doing it.
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
Ports that have a device_owner that starts with 'network:'
are trusted in several places throughout the codebase. Each
of these did a startswith check on each field and it's not
immediately obvious why it's done.
This patch adds a utility function called 'is_port_trusted'
that performs the same check and makes it obvious what is
This patch changes the get_port_from_device_mac() and
get_sg_ids_grouped_by_port() methods in ML2 db.py module so that
they do not create a new database session (via get_session()), but
instead reuse the session associated with the caller's context.
In order to make the session that is associated with the caller's
context available to these ML2 DB methods, the
get_ports_from_devices plugin API in securitygroups_rps_base.py
needs to be modified so that the context can be passed down to the
ML2 plugin. (A similar change is made to the get_port_from_device
plugin API for consistency.)
When a security_groups_provider_updated is received then a global
refresh of the firewall is performed. This can be avoided if the
plugins pass as parameter of the call the devices that belongs to
the network updated.
Partially-Implements: blueprint restructure-l2-agent
Use set operations instead of using list.
Currently complexity of the method is O(n^2) where n is
amount of ips (amount of VMs in the network).
When amount of VM is big (large L2 domain), this method
can significantly load the controller.
Reduce method complexity to O(n) on average.
This patch scopes the agent to plugin security group rpc interface
using a messaging namespace. Right now some plugins expose several
interfaces via the default namespace. This effectively means they are
a single API and should be managed with a single version stream. It's
much more managable to just treat these as separate interfaces and
this change makes that explicit and functionally true. Now when a
method is invoked, the only classes considered for handling that
request will be ones marked with the right namespace.
Part of blueprint rpc-docs-and-namespaces.
Create SecurityGroupServerRpcApi, which is intended to replace
SecurityGroupServerRpcApiMixin over time. The new class is the same
code, but used as a standalone class. This is needed to be able to
eventually move this API into a messaging namespace.
In passing, add docstrings that show where to find the corresponding
other side of this interface in the code base.
Part of blueprint rpc-docs-and-namespaces.
This patch drops several uses of the RpcCallback compatibility class.
All of these were trivial and straight forward conversions so I
batched them up.
There are still several other uses of RpcCallback, but the conversions
are were not necessarily trivial, and may be broken in one way or
another, so I wanted to address them separately. In particular, the
use of mixin classes means that there could be cases where the version
declaration is being stepped on, so they need to be investigated more
Part of blueprint drop-rpc-compat.
The security groups RPC handler calls get_port_from_device
individually for each device in a list it receives. Each
one of these results in a separate SQL query for the security
groups and port details. This becomes very inefficient as the
number of devices on a single node increases.
This patch adds logic to the RPC handler to see if the core
plugin has a method to lookup all of the device IDs at once.
If so, it uses that method, otherwise it continues as normal.
The ML2 plugin is modified to include the batch function, which
uses one SQL query regardless of the number of devices.
Refactor rpc call of 'security_group_rules_for_devices' into
'security_group_info_for_devices' to reduce the response
message size and server side processing time. Includes
agent-side auto detection of rpc version, and fall back
to the old one.
Implements: blueprint security-group-rules-for-devices-rpc-call-refactor
Co-Authored-By: Miguel Angel Ajo <email@example.com>
RPC has a version of itself. In Neutron a plugin implements
several RPC interface, so a single RPC version doesn't work.
In Mixin callback class approach, RPC versioning depends on
each plugin implementation and it makes harder to maintain
RPC version appropriately. This patch series replaces mixin
RPC callback of server side with a separate class.
This commit handles server-side callback of security group
* The server-side callback of Security group RPC is moved to
api/rpc/handler and db/securitygroups_rpc_base now only
contains a mixin class to add agent-based security group
implementation with db operations.
* get_port_from_device method in server-side callback class
is moved to a mixin class of plugin implementation
(SecurityGroupServerRpcMixin) because it involves DB lookup
and is tightly coupled with plugin implementation rather
than RPC interface definition.
Most unit tests for SGServerRpcCallBackTestCase were skipped
in the base class before, but now they are no longer skipped.
The following items will be planned in later patches
to avoid drastic changes in a single patch.
* Merge security group RPC API and agent callback classes in
agent/securitygroups_rpc into api/rpc/handlers/securitygroup_rpc
* Remove completely duplicated db access code in get_port_from_device