The from_self() method in SQLAlchemy is currently
being considered for removal from the library,
with a deprecation phase throughout 1.4 and then
removal by SQLAlchemy 2.0.
The from_self() method takes an ORM query object,
turns it into a subquery, then returns a new query
object that will SELECT from that subquery, while transparently
altering subsequent criteria added to the query to
be stated in terms of the subquery. The current
design direction of SQLAlchemy hopes to
de-emphasize the "transparently altering criteria"
part of the above use case, and to move users towards a
more explicit and model of usage where a subquery should
be created and used explicitly using the aliased()
construct, which is now very mature and can be used in ways
that were not available when from_self() was first introduced.
On the SQLAlchemy side, from_self() has proven to be one
of the most difficult features to maintain and test
as it can easily lead to extremely complicated scenarios, and
while I am also experimenting with some alternatives that
may still retain some of the "automatic translation" features,
those features are still proving to add similar internal
complexity which is having me lean towards the original
plan of removing open-ended "entity translation" features
like that of from_self() at least through the start
of the 2.0 series.
A code search for all of Openstack shows that the
two files modified here are the only usages of the
from_self() method throughout all of searchable Openstack
code. This speaks to the general obscurity of this method,
although neutron's Subnet code is actually using this
method as intended. The new approach necessarily changes
some of the method signatures here so that the explicit
"subquery" entity can be passed; code searches again
show that these methods are not being called anywhere
outside, so the query_filter_service_subnets method
becomes the private _query_entity_service_subnets method.
References: https://github.com/sqlalchemy/sqlalchemy/issues/5368
Closes-Bug: #2004263
Change-Id: Icec998873221ac8e6a1566a157b2044c1f6cd7f3
MechanismDriverContext has an attribute _plugin_context, which carries
the current context with it. This is used by many ml2 drivers, as it is
the only way for them to get the current context. We now make this a
public API by adding a property to MechanismDriverContext that returns
_plugin_context as a read-only attribute.
Change-Id: If9b05655286f42081cf26c90c563429ca2e63244
Running with a stricter .pylintrc generates a lot of
C0330 warnings (hanging/continued indentation). Fix
the ones in neutron/plugins.
Trivialfix
Change-Id: Id9138652f5f07ef12fa682e182fe210019e8f975
This change is updating the vlanmanager data structure to handle for a
given network more than one vlan mapping. This is a prerequisite work
needed to progress on accepting several segments per network per
host.
The work done here is trying to avoid changing logic in the
current implementation. Unit test should not have value updated,
but probably signatures changed.
Partial-Bug: #1956435
Partial-Bug: #1764738
Signed-off-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@industrialdiscipline.com>
Change-Id: Ic3c147136549b17aea0fe78e930a41a5b33ab9d8
When processing ports that are incompatible with l2_population,
certain methods do unnecessary db calls to retrieve agent port
info from the db. This info is used only if the processed
port is of the correct network type. The check is done after
the db call, but before the info is used.
This commit moves the check earlier in the method. The db will
be called only when the port is supposed to be set up with the
l2pop.
Closes-Bug: #1983558
Change-Id: I5a83bfce60a933af781b1fd96037c7de6b2b1f38
Some database operations in L2pop module didn't have the
needed database context to perform a query.
Closes-Bug: #1975797
Change-Id: I32a49efd9d56f7c06d3bb1de1352be1311686c42
With python 3.x, classes can use the metaclass= logic
to not require usage of the six library.
One step in removing all of six usage from neutron.
Change-Id: I2f815e412d9a96eb5faf2b3bb3a1e393a9db9309
Currently the ovs agent calls update_device_list with the
agent_restarted flag set only on the first loop iteration. Then the
server knows to send the l2pop flooding entries for the network to
the agent. But when a compute node with many instances on many
networks reboots, it takes time to readd all the active devices and
some may be readded after the first loop iteration. Then the server
can fail to send the flooding entries which means there will be no
flood_to_tuns flow and broadcasts like dhcp will fail.
This patch fixes that by renaming the agent_restarted flag to
refresh_tunnels and setting it if the agent has not received the
flooding entries for the network.
Change-Id: I607aa8fa399e72b037fd068ad4f02b6210e57e91
Closes-Bug: #1853613
Removed E125 (continuation line does not distinguish itself
from next logical line) from the ignore list and fixed all
the indentation issues. Didn't think it was going to be
close to 100 files when I started.
Change-Id: I0a6f5efec4b7d8d3632dd9dbb43e0ab58af9dff3
When the vlan and vxlan both exist in env, and l2population
and arp_responder are enabled, if we update a port's ip address
from vlan network, there will be arp responder related flows
added into br-tun, this will cause too many arp reply for
one arp request, and vm connections will be unnormal.
Closes-Bug: #1824504
Change-Id: I1b6154b9433a9442d3e0118dedfa01c4a9b4740b
Ovs-agent can be very time-consuming in handling a large number
of ports. At this point, the ovs-agent status report may have
exceeded the set timeout value. Some flows updating operations
will not be triggerred. This results in flows loss during agent
restart, especially for hosts to hosts of vxlan tunnel flow.
This fix will let the ovs-agent explicitly, in the first rpc loop,
indicate that the status is restarted. Then l2pop will be required
to update fdb entries.
Closes-Bug: #1813703
Closes-Bug: #1813714
Closes-Bug: #1813715
Closes-Bug: #1794991
Closes-Bug: #1799178
Change-Id: I8edc2deb509216add1fb21e1893f1c17dda80961
Reduces E128 warnings by ~260 to just ~900,
no way we're getting rid of all of them at once (or ever).
Files under neutron/tests still have a ton of E128 warnings.
Change-Id: I9137150ccf129bf443e33428267cd4bc9c323b54
Co-Authored-By: Akihiro Motoki <amotoki@gmail.com>
The neutron.common.rpc module has been in neutron-lib for awhile now and
neutron is shimmed to use neutron-lib already.
This patch removes neutron.common.rpc and switches the code over to use
neutron-lib's implementation where needed.
NeutronLibImpact
Change-Id: I733f07a8c4a2af071b3467bd710290eee11a4f4c
With high concurrency more than 1 port may be activated on an
OVS agent at the same time (like VM port + a DVR port),
so the patch mitigates the condition by checking for 1 or 2
first active ports.
Given that the condition also contains "or self.agent_restarted(context)"
which makes it True first 180 sec (by default) after agent restart,
I believe the downside of changing 1 to 2 should be negligible.
Please see bug for more details on the issue.
Closes-Bug: #1789846
Change-Id: Ieab0186cbe05185d47bbf5a31141563cf923f66f
When HA router's interface on host is going DOWN but router
is still available on this host, L2 population
mechanism driver will now send to other hosts info to remove
fdb unicast entries to this port on host.
It will not send FLOODING_ENTRY because this port is still on
host but in standby mode and might be transformed to master
in future.
This solves issue with migration router from Legacy to HA.
In such case, port which was originally attached to legacy
router is transformed to be HA backup port before changing
its status to DOWN.
Now in such case unicast entries to this port and backup
node will be removed properly so packets to HA router will
be really send to host which is master node for router.
Closes-Bug: #1785582
Change-Id: Icc14e5f5d40fc6fbb49e0f7b18cc3b15ebec8508
neutron-openvswitch-agent will refresh flows when it's restarted.
But the port's binding status is not changed, update_port_postcommit
will be skipped at function '_update_individual_port_db_status' in
'neutron/plugins/ml2/plugin.py', l2pop don't handle DVR ports, the
fdb entries about DVR port will not be added.
So, we can't skip DVR port at notify_l2pop_port_wiring when agent
is restared.
Closes-Bug: #1773286
Change-Id: I54e3db4822830a0c83daf7b5150575f8d6e2497b
Fix W503 (line break before binary operator) pep8 warnings
and no longer ignore new failures.
Trivialfix
Change-Id: I7539f3b7187f2ad40681781f74b6e05a01bac474
The neutron.common.topics module was rehomed into neutron-lib with
commit Ie88b84949cbd55a4e7ad06341aab77b286cdc485
This patch consumes it by removing the rehomed module from neutron
and using the module from neutron-lib instead.
NeutronLibImpact
Change-Id: Ia4a4604c259ce862597de80c6deeb3d408bf0e95
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
The error sneaked in with Ib6e59ab3405857d3ed4d82df1a80800089c3f06e
where is_ha_router_port expects a NeutronContext object but we still
pass PortContext instead.
Change-Id: I593af5d050de00ddea7d758007d9856c4b97695f
Closes-Bug: #1703938
This adjusts the l2pop DB queries that leverage the port relationship
on the PortBinding and DistributedPortBinding models to ensure that the
port gets loaded from the DB as part of the initial query.
Without this the number of DB queries will increase with the number
of ports since the port wasn't being loaded until the 'port' attribute
was referenced on its corresponding binding object.
Closes-Bug: #1701288
Change-Id: I7c3b08d525b2c90100c9fe4efaee973cf3a076f3
The well known service type constants are in
neutron_lib.plugins.constants, but for legacy reasons a few still exist
and are referenced from neutron_lib.constants that we'd like to remove.
This patch switches references over to neutron_lib's plugin constants.
Change-Id: I1861448cec303725b30cef8f42029f467f9e03a3
The ml2 MechanismDriver is now in neutron-lib along with its associated
constants. This patch switches over to the lib versions of those, but
leaves a shim of the MechanismDriver that just ref's the driver from
lib. This shim allows our broad consumer base of the driver to switch
over at their leisure.
NeutronLibImpact
Change-Id: I99e3de6d933a1bb341394f85415fb07306a82a01
This patch integrates Router Extra Attributes OVO and uses proper
context in calling methods for object operations.
The other integration parts of this OVO in l3_agentschedulers_db.py
and l3_attrs_db.py are being done in patch [1] and [2] respectively.
[1] - I0af665a97087ad72431d58f04089a804088ef005
[2] - Id5ed0a541d09cd1105f9cb067401e2afa8cd9b83
Change-Id: Ib6e59ab3405857d3ed4d82df1a80800089c3f06e
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
get_agent_by_host can return None in the l2pop
driver so we need to check for that case before
we blindly try to decode configuration values on
the result.
There are a couple of cases that can lead to this.
* The deployment can be misconfigured and is missing
either a tunneling_ip option for the agent on a
host or is missing an L2 agent with that host_id
entirely.
* Multiple mech drivers are in use and a port is being
deleted from an agentless host.
Related-Bug: #1533013
Closes-Bug: #1672564
Change-Id: I1e79f600172edad1e31e8231a0a6a2c55f46804c
In delete_port_postcommit, a DVR port (port['device_owner'] =
DEVICE_OWNER_ROUTER_SNAT) can match on l2pop_db.HA_ROUTER_PORTS[1],
but can not get any fdb entries by _get_ha_port_agents_fdb. Then,
the fdb_entries[network_id]['ports'] is been overwritten by {}. So
the associated flow entries will not be deleted.
Closes-Bug: #1668277
Change-Id: I7b621157fe85945acd99e4f08b6370d2f9c3d44d
Introduce get_reader_session() and get_writer_session()
and replace get_session() with them.
Mark get_session as depricated.
Stop using get_engine from legacy facade. Use writer engine for
places where it is required.
Partially-Implements blueprint: enginefacade-switch
Change-Id: I28b741bfa27bf04cbe273586e6e3e00e14fbe683
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.
Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.
This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.
Usage pattern [1,2] can be translated to [3,4] respectively.
[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()
The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.
Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.
Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.
NeutronLibImpact
Partially-implements: blueprint neutron-lib
Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
Refactoring neutron ml2 plugin l2pop driver config opts to be in
neutron/conf/plugins/ml2/drivers/l2pop so that all the
configuration options for l2pop drivers reside in a centralized
location. This simplifies the process of looking up the l2pop driver
config opts and provides an easy way to import.
Change-Id: Ia06c0c31ef40c691d67cfc5d514fdff907f5dca4
Partial-Bug: #1563069
This patch introduces context in methods where it is needed for
object operations done during the integration of Router Extra
Attributes OVO.
Used in integration patch: https://review.openstack.org/#/c/381209/
Change-Id: I4dea67207220a19abf5d3cc754f02150b3621550
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
VLAN mapping was separated from agent in change
I514c7632c1c26d6cfeb706fc5d829a46dcce3782
Please use new manager located in
neutron.plugins.ml2.drivers.openvswitch.agent.vlanmanager
Change-Id: Id6cc9c332c241602526d4c325ca4d1c80380b203
This patch set is for breaking the circular dependency between
Agent/AgentVersionedObject.
See:https://review.openstack.org/#/c/297887/ for details.
Change-Id: I7be4ce2513e49e6da46a7bdffb8538613f0be7c7
Partial-Bug: #1597913
Co-Authored-By: Victor Morales <victor.morales@intel.com>
Co-Authored-By: Sindhu Devale <sindhu.devale@intel.com>
_get_agent_fdb may return None so we need to check for
that before we try to iterate over a key inside of it
in delete_port_postcommit.
Closes-Bug: #1622996
Change-Id: I2256df0e08380e550f32248fb9589ee43b0923ff
This patch makes L3 HA failover not depended on neutron components
(during failover).
All HA agents(active and backup) call update_device_up/down after wiring
the ports. But l2pop driver is called for only active agent as port
binding in DB reflects active agent. Then l2pop creates unicast and
multicast flows for active agent.
On failover, flows to new active agent is created. For this to happen -
all of database, messaging server, neutron-server and destination L3
agent should be active during failover. This creates two issues -
1) When any of the above resources(i.e neutron-server, .. ) are dead,
flows between new master and other agents won't be created and
L3 Ha failover is not working. In same scenario, L3 Ha failover will
work if l2pop is disabled.
2) Packet loss during failover is higher as above neutron resources
interact multiple times, so will take time to create l2 flows.
In this change, we allow plugin to notify l2pop when update_device_up/down
is called by backup agents also. Then l2pop will create flood flows to
all HA agents(both active and slave). L2pop won't create unicast flow for
this port, instead unicast flow is created by learning action of table 10
when keepalived sends GARP after assigning ip address to master router's
qr-xx port. As flood flows are already created and unicast flow is
dynamically added, L3 HA failover is not depended on l2pop.
This solves two isses
1) with L3 HA + l2pop, failover will work even if any of above agents
or processes dead.
2) Reduce failover time as we are not depending on neutron to create
flows during failover.
We use L3HARouterAgentPortBinding table for getting all HA agents of a
router port. HA router port on slave agent is also considered for l2pop
distributed_active_network_ports and agent_network_active_port_count
Closes-bug: #1522980
Closes-bug: #1602614
Change-Id: Ie1f5289390b3ff3f7f3ed7ffc8f6a8258ee8662e
Remove deprecation warnings for various constants
and exceptions that have moved to neutron_lib.
Fix miscellaneous other deprecations.
Uses constants instead of l3_constants when importing
neutron-lib constants.
Co-Authored By: Henry Gessau <gessau@gmail.com>
Co-Authored By: Gary Kotton <gkotton@vmware.com>
Change-Id: Ib0e8ff5c3e23677c1009241a1818cbc8a3430c38
As the L2population mechanism driver does not impact VLAN transparency
support, this patch overrides the check_vlan_transparency method from
the MechanismDriver base class and returns 'True', In this way,
inclusion of the L2population mechanism driver in the list of configured
mechanism_drivers does not prevent the creation of VLAN transparent
networks.
Change-Id: I8b26e899180544e2ad49ef470b9624015fd4d5de
Closes-Bug: #1611920
Now the ML2 core plugin maps driver errors to MechanismDriverError
and hides the error details from the caller.
This patch change MechanismDriverError from an instance of
NeutronException to an instance of MultipleExceptions. Add add
exceptions from mechanism driver as inner_exceptions of
MultipleExceptions. As a result, the api layer will unwrap the
MechanismDriverError and return the real error to client.
Change-Id: I3a46932848d59f7f027640bfb598650f064b0a12
Closes-bug: #1273730
Mike Bayer