Commit Graph

124 Commits (master)

Author SHA1 Message Date
Mike Bayer e98a268de4 Propose replacement of ORM from_self()
The from_self() method in SQLAlchemy is currently
being considered for removal from the library,
with a deprecation phase throughout 1.4 and then
removal by SQLAlchemy 2.0.

The from_self() method takes an ORM query object,
turns it into a subquery, then returns a new query
object that will SELECT from that subquery, while transparently
altering subsequent criteria added to the query to
be stated in terms of the subquery.   The current
design direction of SQLAlchemy hopes to
de-emphasize the "transparently altering criteria"
part of the above use case, and to move users towards a
more explicit and model of usage where a subquery should
be created and used explicitly using the aliased()
construct, which is now very mature and can be used in ways
that were not available when from_self() was first introduced.

On the SQLAlchemy side, from_self() has proven to be one
of the most difficult features to maintain and test
as it can easily lead to extremely complicated scenarios, and
while I am also experimenting with some alternatives that
may still retain some of the "automatic translation" features,
those features are still proving to add similar internal
complexity which is having me lean towards the original
plan of removing open-ended "entity translation" features
like that of from_self() at least through the start
of the 2.0 series.

A code search for all of Openstack shows that the
two files modified here are the only usages of the
from_self() method throughout all of searchable Openstack
code.  This speaks to the general obscurity of this method,
although neutron's Subnet code is actually using this
method as intended.    The new approach necessarily changes
some of the method signatures here so that the explicit
"subquery" entity can be passed; code searches again
show that these methods are not being called anywhere
outside, so the query_filter_service_subnets method
becomes the private _query_entity_service_subnets method.


Closes-Bug: #2004263

Change-Id: Icec998873221ac8e6a1566a157b2044c1f6cd7f3
4 months ago
Sebastian Lohff 9a483f02be Make MechanismDriverContext plugin_context public
MechanismDriverContext has an attribute _plugin_context, which carries
the current context with it. This is used by many ml2 drivers, as it is
the only way for them to get the current context. We now make this a
public API by adding a property to MechanismDriverContext that returns
_plugin_context as a read-only attribute.

Change-Id: If9b05655286f42081cf26c90c563429ca2e63244
7 months ago
Brian Haley b1714a2b9d Fix some pylint indentation warnings
Running with a stricter .pylintrc generates a lot of
C0330 warnings (hanging/continued indentation). Fix
the ones in neutron/plugins.


Change-Id: Id9138652f5f07ef12fa682e182fe210019e8f975
7 months ago
Sahid Orentino Ferdjaoui 6ec0bc70a7 ovs: make vlanmanager to handle more vlan mapping per network
This change is updating the vlanmanager data structure to handle for a
given network more than one vlan mapping. This is a prerequisite work
needed to progress on accepting several segments per network per

The work done here is trying to avoid changing logic in the
current implementation. Unit test should not have value updated,
but probably signatures changed.

Partial-Bug: #1956435
Partial-Bug: #1764738
Signed-off-by: Sahid Orentino Ferdjaoui <>
Change-Id: Ic3c147136549b17aea0fe78e930a41a5b33ab9d8
9 months ago
Weronika Sikora a45cebbfcd Avoid unnecessary db calls with l2pop
When processing ports that are incompatible with l2_population,
certain methods do unnecessary db calls to retrieve agent port
info from the db. This info is used only if the processed
port is of the correct network type. The check is done after
the db call, but before the info is used.

This commit moves the check earlier in the method. The db will
be called only when the port is supposed to be set up with the

Closes-Bug: #1983558
Change-Id: I5a83bfce60a933af781b1fd96037c7de6b2b1f38
10 months ago
Rodolfo Alonso Hernandez b48595e419 [sqlalchemy-20] Add DB context in L2pop module
Some database operations in L2pop module didn't have the
needed database context to perform a query.

Closes-Bug: #1975797

Change-Id: I32a49efd9d56f7c06d3bb1de1352be1311686c42
1 year ago
Brian Haley 08a60f7483 Remove usage of six.add_metaclass
With python 3.x, classes can use the metaclass= logic
to not require usage of the six library.

One step in removing all of six usage from neutron.

Change-Id: I2f815e412d9a96eb5faf2b3bb3a1e393a9db9309
3 years ago
Darragh O'Reilly 93e9dc5426 ovs agent: signal to plugin if tunnel refresh needed
Currently the ovs agent calls update_device_list with the
agent_restarted flag set only on the first loop iteration. Then the
server knows to send the l2pop flooding entries for the network to
the agent. But when a compute node with many instances on many
networks reboots, it takes time to readd all the active devices and
some may be readded after the first loop iteration. Then the server
can fail to send the flooding entries which means there will be no
flood_to_tuns flow and broadcasts like dhcp will fail.

This patch fixes that by renaming the agent_restarted flag to
refresh_tunnels and setting it if the agent has not received the
flooding entries for the network.

Change-Id: I607aa8fa399e72b037fd068ad4f02b6210e57e91
Closes-Bug: #1853613
3 years ago
Brian Haley b79842f289 Start enforcing E125 flake8 directive
Removed E125 (continuation line does not distinguish itself
from next logical line) from the ignore list and fixed all
the indentation issues.  Didn't think it was going to be
close to 100 files when I started.

Change-Id: I0a6f5efec4b7d8d3632dd9dbb43e0ab58af9dff3
4 years ago
LIU Yulong 76c0280635 Remove the l2pop agent_boot_time config
It was marked as deprecated, so let's do a quick

Related-Bug: #1813714
Change-Id: Ibc039b34b826641811a7e08b0d1bff0fd21b9193
4 years ago
Zuul 07969cfe15 Merge "Don't add arp responder for non tunnel network port" 4 years ago
Yang Li 5301ecf41b Don't add arp responder for non tunnel network port
When the vlan and vxlan both exist in env, and l2population
and arp_responder are enabled, if we update a port's ip address
from vlan network, there will be arp responder related flows
added into br-tun, this will cause too many arp reply for
one arp request, and vm connections will be unnormal.

Closes-Bug: #1824504
Change-Id: I1b6154b9433a9442d3e0118dedfa01c4a9b4740b
4 years ago
LIU Yulong a5244d6d44 More accurate agent restart state transfer
Ovs-agent can be very time-consuming in handling a large number
of ports. At this point, the ovs-agent status report may have
exceeded the set timeout value. Some flows updating operations
will not be triggerred. This results in flows loss during agent
restart, especially for hosts to hosts of vxlan tunnel flow.

This fix will let the ovs-agent explicitly, in the first rpc loop,
indicate that the status is restarted. Then l2pop will be required
to update fdb entries.

Closes-Bug: #1813703
Closes-Bug: #1813714
Closes-Bug: #1813715
Closes-Bug: #1794991
Closes-Bug: #1799178

Change-Id: I8edc2deb509216add1fb21e1893f1c17dda80961
4 years ago
Brian Haley eaf990b2bc Fix pep8 E128 warnings in non-test code
Reduces E128 warnings by ~260 to just ~900,
no way we're getting rid of all of them at once (or ever).
Files under neutron/tests still have a ton of E128 warnings.

Change-Id: I9137150ccf129bf443e33428267cd4bc9c323b54
Co-Authored-By: Akihiro Motoki <>
4 years ago
Boden R 024802aafd remove neutron.common.rpc
The neutron.common.rpc module has been in neutron-lib for awhile now and
neutron is shimmed to use neutron-lib already.
This patch removes neutron.common.rpc and switches the code over to use
neutron-lib's implementation where needed.


Change-Id: I733f07a8c4a2af071b3467bd710290eee11a4f4c
4 years ago
Oleg Bondarev b32db30874 l2 pop: check for more than 1 first active port on a node
With high concurrency more than 1 port may be activated on an
OVS agent at the same time (like VM port + a DVR port),
so the patch mitigates the condition by checking for 1 or 2
first active ports.

Given that the condition also contains "or self.agent_restarted(context)"
which makes it True first 180 sec (by default) after agent restart,
I believe the downside of changing 1 to 2 should be negligible.

Please see bug for more details on the issue.

Closes-Bug: #1789846
Change-Id: Ieab0186cbe05185d47bbf5a31141563cf923f66f
5 years ago
Slawek Kaplonski 6c300b1a9b Remove fdb entries for ha router interfaces when going DOWN
When HA router's interface on host is going DOWN but router
is still available on this host, L2 population
mechanism driver will now send to other hosts info to remove
fdb unicast entries to this port on host.

It will not send FLOODING_ENTRY because this port is still on
host but in standby mode and might be transformed to master
in future.

This solves issue with migration router from Legacy to HA.
In such case, port which was originally attached to legacy
router is transformed to be HA backup port before changing
its status to DOWN.
Now in such case unicast entries to this port and backup
node will be removed properly so packets to HA router will
be really send to host which is master node for router.

Closes-Bug: #1785582

Change-Id: Icc14e5f5d40fc6fbb49e0f7b18cc3b15ebec8508
5 years ago
Yang JianFeng d0fa2c9ac5 Don't skip DVR port while neutron-openvswitch-agent is restared.
neutron-openvswitch-agent will refresh flows when it's restarted.
But the port's binding status is not changed, update_port_postcommit
will be skipped at function '_update_individual_port_db_status' in
'neutron/plugins/ml2/', l2pop don't handle DVR ports, the
fdb entries about DVR port will not be added.

So, we can't skip DVR port at notify_l2pop_port_wiring when agent
is restared.

Closes-Bug: #1773286
Change-Id: I54e3db4822830a0c83daf7b5150575f8d6e2497b
5 years ago
Brian Haley 7cfdf4aa81 Fix all pep8 E129 errors
Fixed all pep8 E129 errors and changed tox.ini to no longer
ignore them.

Change-Id: I0b06d99ce1d473b79a4cfdd173baa4f02e653847
5 years ago
Brian Haley 90cd939047 Fix W503 pep8 warnings
Fix W503 (line break before binary operator) pep8 warnings
and no longer ignore new failures.


Change-Id: I7539f3b7187f2ad40681781f74b6e05a01bac474
5 years ago
Boden R ef93f7e7f0 use common agent topics from neutron-lib
The neutron.common.topics module was rehomed into neutron-lib with
commit Ie88b84949cbd55a4e7ad06341aab77b286cdc485
This patch consumes it by removing the rehomed module from neutron
and using the module from neutron-lib instead.


Change-Id: Ia4a4604c259ce862597de80c6deeb3d408bf0e95
5 years ago
Vu Cong Tuan 943f384093 Integration of L3HARouterAgentPortBinding in ml2/drivers/l2pop/
This patch integrates L3HARouterAgentPortBinding in

Co-Authored-By: Nguyen Phuong An <>
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Change-Id: Idf72866d36f27ad0aa9b7a0b53034d2db45cd56d
6 years ago
Inessa Vasilevskaya 7322bd6efb Make code follow log translation guideline
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.

Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
6 years ago
Ihar Hrachyshka 5c331ecbd2 Fixed AttributeError in l2pop.delete_port_postcommit
The error sneaked in with Ib6e59ab3405857d3ed4d82df1a80800089c3f06e
where is_ha_router_port expects a NeutronContext object but we still
pass PortContext instead.

Change-Id: I593af5d050de00ddea7d758007d9856c4b97695f
Closes-Bug: #1703938
6 years ago
Jenkins e0c08bba30 Merge "Use subqueryload in l2pop DB for binding ports" 6 years ago
Kevin Benton 5be1ace546 Use subqueryload in l2pop DB for binding ports
This adjusts the l2pop DB queries that leverage the port relationship
on the PortBinding and DistributedPortBinding models to ensure that the
port gets loaded from the DB as part of the initial query.

Without this the number of DB queries will increase with the number
of ports since the port wasn't being loaded until the 'port' attribute
was referenced on its corresponding binding object.

Closes-Bug: #1701288
Change-Id: I7c3b08d525b2c90100c9fe4efaee973cf3a076f3
6 years ago
Boden R 0a1405794f use service type constants from neutron_lib plugins
The well known service type constants are in
neutron_lib.plugins.constants, but for legacy reasons a few still exist
and are referenced from neutron_lib.constants that we'd like to remove.
This patch switches references over to neutron_lib's plugin constants.

Change-Id: I1861448cec303725b30cef8f42029f467f9e03a3
6 years ago
Boden R fde6710515 use MechanismDriver from neutron-lib + shim
The ml2 MechanismDriver is now in neutron-lib along with its associated
constants. This patch switches over to the lib versions of those, but
leaves a shim of the MechanismDriver that just ref's the driver from
lib. This shim allows our broad consumer base of the driver to switch
over at their leisure.


Change-Id: I99e3de6d933a1bb341394f85415fb07306a82a01
6 years ago
Shashank Kumar Shankar 79cf488c4c Integration of Router Extra Attributes OVO
This patch integrates Router Extra Attributes OVO and uses proper
context in calling methods for object operations.

The other integration parts of this OVO in
and are being done in patch [1] and [2] respectively.

[1] - I0af665a97087ad72431d58f04089a804088ef005
[2] - Id5ed0a541d09cd1105f9cb067401e2afa8cd9b83

Change-Id: Ib6e59ab3405857d3ed4d82df1a80800089c3f06e
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
6 years ago
Jenkins 1414ea9538 Merge "Change the way to distinguish the port type" 6 years ago
Kevin Benton c7fb24b3cb Check for None in _get_agent_fdb for agent
get_agent_by_host can return None in the l2pop
driver so we need to check for that case before
we blindly try to decode configuration values on
the result.

There are a couple of cases that can lead to this.
* The deployment can be misconfigured and is missing
  either a tunneling_ip option for the agent on a
  host or is missing an L2 agent with that host_id
* Multiple mech drivers are in use and a port is being
  deleted from an agentless host.

Related-Bug: #1533013
Closes-Bug: #1672564
Change-Id: I1e79f600172edad1e31e8231a0a6a2c55f46804c
6 years ago
Yaohua Yan 3593620faa Change the way to distinguish the port type
In delete_port_postcommit, a DVR port (port['device_owner'] =
but can not get any fdb entries by _get_ha_port_agents_fdb. Then,
the fdb_entries[network_id]['ports'] is been overwritten by {}. So
the associated flow entries will not be deleted.

Closes-Bug: #1668277
Change-Id: I7b621157fe85945acd99e4f08b6370d2f9c3d44d
6 years ago
Armando Migliaccio ca751a1486 Spin off context module

Partially-implements: blueprint neutron-lib

Change-Id: I48cf45dc1b07035d952152eac2548a3bd9fc2832
6 years ago
Ann Kamyshnikova 4f17f70089 Stop using legacy facade
Introduce get_reader_session() and get_writer_session()
and replace get_session() with them.
Mark get_session as depricated.

Stop using get_engine from legacy facade. Use writer engine for
places where it is required.

Partially-Implements blueprint: enginefacade-switch

Change-Id: I28b741bfa27bf04cbe273586e6e3e00e14fbe683
6 years ago
Jenkins ff9f88f8b8 Merge "Removed deprecated class LocalVLANMapping" 7 years ago
Jenkins b2399d847a Merge "Refactoring config options for plugin l2pop opts" 7 years ago
Armando Migliaccio 17563a802e Adopt neutron-lib plugin directory
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.

Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.

This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.

Usage pattern [1,2] can be translated to [3,4] respectively.

[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()

The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.

Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.

Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.


Partially-implements: blueprint neutron-lib

Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
7 years ago
Anindita Das 1e91b6392e Refactoring config options for plugin l2pop opts
Refactoring neutron ml2 plugin l2pop driver config opts to be in
neutron/conf/plugins/ml2/drivers/l2pop so that all the
configuration options for l2pop drivers reside in a centralized
location. This simplifies the process of looking up the l2pop driver
config opts and provides an easy way to import.

Change-Id: Ia06c0c31ef40c691d67cfc5d514fdff907f5dca4
Partial-Bug: #1563069
7 years ago
Shashank Kumar Shankar f5a721e0c2 Introduce context in methods for Router Extra Attributes OVO usage.
This patch introduces context in methods where it is needed for
object operations done during the integration of Router Extra
Attributes OVO.

Used in integration patch:

Change-Id: I4dea67207220a19abf5d3cc754f02150b3621550
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
7 years ago
Dariusz Smigiel 4ec456d315 Removed deprecated class LocalVLANMapping
VLAN mapping was separated from agent in change
Please use new manager located in

Change-Id: Id6cc9c332c241602526d4c325ca4d1c80380b203
7 years ago
Gary Kotton 29016406f0 Remove deprecation warnings for l3_hamode_db
Commit f9b1f0906c relocated these

Change-Id: Ia0477d478583c5ace06858d4aa0492b18273f3f8
7 years ago
tonytan4ever e669e7b96b Relocate Agent DB model
This patch set is for breaking the circular dependency between
See: for details.

Change-Id: I7be4ce2513e49e6da46a7bdffb8538613f0be7c7
Partial-Bug: #1597913
Co-Authored-By: Victor Morales <>
Co-Authored-By: Sindhu Devale <>
7 years ago
Kevin Benton 6e0b8c176c Ensure there are fdb_entries before iterating
_get_agent_fdb may return None so we need to check for
that before we try to iterate over a key inside of it
in delete_port_postcommit.

Closes-Bug: #1622996
Change-Id: I2256df0e08380e550f32248fb9589ee43b0923ff
7 years ago
venkata anil 26d8702b9d l2pop fdb flows for HA router ports
This patch makes L3 HA failover not depended on neutron components
(during failover).

All HA agents(active and backup) call update_device_up/down after wiring
the ports. But l2pop driver is called for only active agent as port
binding in DB reflects active agent. Then l2pop creates unicast and
multicast flows for active agent.
On failover, flows to new active agent is created. For this to happen -
all of database, messaging server, neutron-server and destination L3
agent should be active during failover. This creates two issues -
1) When any of the above resources(i.e neutron-server, .. ) are dead,
   flows between new master and other agents won't be created and
   L3 Ha failover is not working. In same scenario, L3 Ha failover will
   work if l2pop is disabled.
2) Packet loss during failover is higher as above neutron resources
   interact multiple times, so will take time to create l2 flows.

In this change, we allow plugin to notify l2pop when update_device_up/down
is called by backup agents also. Then l2pop will create flood flows to
all HA agents(both active and slave). L2pop won't create unicast flow for
this port, instead unicast flow is created by learning action of table 10
when keepalived sends GARP after assigning ip address to master router's
qr-xx port. As flood flows are already created and unicast flow is
dynamically added, L3 HA failover is not depended on l2pop.

This solves two isses
1) with L3 HA + l2pop, failover will work even if any of above agents
   or processes dead.
2) Reduce failover time as we are not depending on neutron to create
   flows during failover.
We use L3HARouterAgentPortBinding table for getting all HA agents of a
router port. HA router port on slave agent is also considered for l2pop
distributed_active_network_ports and agent_network_active_port_count

Closes-bug: #1522980
Closes-bug: #1602614
Change-Id: Ie1f5289390b3ff3f7f3ed7ffc8f6a8258ee8662e
7 years ago
Gary Kotton 9f09f27c5d Fix deprecation warnings
Remove deprecation warnings for various constants
and exceptions that have moved to neutron_lib.

Fix miscellaneous other deprecations.

Uses constants instead of l3_constants when importing
neutron-lib constants.

Co-Authored By: Henry Gessau <>
Co-Authored By: Gary Kotton <>

Change-Id: Ib0e8ff5c3e23677c1009241a1818cbc8a3430c38
7 years ago
Jenkins fbe83b88af Merge "Add mechanism driver error details to MechanismDriverError" 7 years ago
Jenkins 99eab4b8da Merge "Implement check_vlan_transparency to return True in L2population mech driver" 7 years ago
Erik Colnick 676770ba75 Implement check_vlan_transparency to return True in L2population mech driver
As the L2population mechanism driver does not impact VLAN transparency
support, this patch overrides the check_vlan_transparency method from
the MechanismDriver base class and returns 'True', In this way,
inclusion of the L2population mechanism driver in the list of configured
mechanism_drivers does not prevent the creation of VLAN transparent

Change-Id: I8b26e899180544e2ad49ef470b9624015fd4d5de
Closes-Bug: #1611920
7 years ago
Hong Hui Xiao 8398ec0d77 Add mechanism driver error details to MechanismDriverError
Now the ML2 core plugin maps driver errors to MechanismDriverError
and hides the error details from the caller.

This patch change MechanismDriverError from an instance of
NeutronException to an instance of MultipleExceptions. Add add
exceptions from mechanism driver as inner_exceptions of
MultipleExceptions. As a result, the api layer will unwrap the
MechanismDriverError and return the real error to client.

Change-Id: I3a46932848d59f7f027640bfb598650f064b0a12
Closes-bug: #1273730
7 years ago
Jenkins 739bd92bb0 Merge "ovs-agent: Seperate VLAN mapping outside of the agent" 7 years ago