Commit Graph

706 Commits (b79842f289f0c9b9fdce9d6cc689538ec61216e7)

Author SHA1 Message Date
Brian Haley b79842f289 Start enforcing E125 flake8 directive
Removed E125 (continuation line does not distinguish itself
from next logical line) from the ignore list and fixed all
the indentation issues.  Didn't think it was going to be
close to 100 files when I started.

Change-Id: I0a6f5efec4b7d8d3632dd9dbb43e0ab58af9dff3
4 years ago
Nate Johnston 8b50a36341 Add base support for update_port_bulk
As a complement to the create_port_bulk optimizations that took place in
the Stein cycle, create the ability to have an ML2 plugin optionally
expose an 'update_port_bulk' operation as well.  This will permit ML2
plugins to optimize this code path for bulk operations as well.

Change-Id: Id2c88a82715bccd1c23213828ef904322ab4386f
4 years ago
Zuul b7a37b3192 Merge "Optimize the code that fixes the race condition of DHCP agent." 4 years ago
Zuul 8c774d3f1d Merge "Fix bug: AttributeError arises while sorting with standard attributes" 4 years ago
pandatt 335ac4e2d9 Fix bug: AttributeError arises while sorting with standard attributes
Common neutron resource(e.g, Port) consists of:
1. Resource Attributes, e.g: Port.mac_address, etc.
2. Standard Attributes, e.g: created_at, and are shared among all
   neutron resources.
The `sort` opt only supports limited attributes. We need to filter
attributes that are defined with `is_sort_key=True` and it's preferred
to explicitly warn CLI & API users of illegal sort keys rather than
just accept without check, pass forward and then hit a internal error
which's quite confusing.

Change-Id: I8d206f909b09f1279dfcdc25c39989a67bff93d5
Closes-Bug: #1659175
4 years ago
Rodolfo Alonso Hernandez 5d35e7d360 Switch to new engine facade for L3RpcCallback
Partially-Implements blueprint: enginefacade-switch

Change-Id: Ida1bcb896c02f1ad04090a99296bc641bd21896d
4 years ago
Yang JianFeng 494b65d951 Optimize the code that fixes the race condition of DHCP agent.
The above patchs that resolve the race condition of DHCP agent will
result in neutron-server raise DhcpPortInUse ERROR log. And, the
second patch may result in old dhcp agent create a redundant port.

Closes-Bug: #1829332
Change-Id: If7a7ac2f88ce5b0e799c1104c936735a6cc860aa
4 years ago
Harald Jensås 3ab7878384 Conntrack Helper - OVO and db script
Implements the conntrack helper OVO and db layer code.
- New object 'ConntrackHelper'
- New db model
- migration db script

Related-Bug: #1823633
Change-Id: I0d9c039b260845b6544eccf63f5a2ffaa929120b
4 years ago
LIU Yulong 3d99147e73 Ensure dvr ha router gateway port binding host
There are some extreme conditions which will result the unbound
router gateway port. Then all the centralized floating IPs will
not be reachable since the gateway port was set to 4095 tag.

This patch adds the HA status to the router related port
processing code path. If it is HA router, the gateway port
will go to the right HA router processing code branch.

Closes-Bug: #1827754
Change-Id: Ida1c9f3a38171ea82adc2f11cb17945d6e2434be
4 years ago
Boden R 9bbe9911c4 remove neutron.common.constants
All of the externally consumed variables from neutron.common.constants
now live in neutron-lib. This patch removes neutron.common.constants
and switches all uses over to lib.


Change-Id: I3c2f28ecd18996a1cee1ae3af399166defe9da87
4 years ago
Boden R 97376032b4 use EXT_PARENT_RESOURCE_MAPPING from neutron-lib
The neutron.common.constants.EXT_PARENT_RESOURCE_MAPPING constant lives
in neutron-lib. This patch consumes it by removing it from neutron
and switching all uses over to libs constant.


Change-Id: Ib7ea6dea58bf211a2da2b103cb526233df04ba49
4 years ago
Brian Haley eaf990b2bc Fix pep8 E128 warnings in non-test code
Reduces E128 warnings by ~260 to just ~900,
no way we're getting rid of all of them at once (or ever).
Files under neutron/tests still have a ton of E128 warnings.

Change-Id: I9137150ccf129bf443e33428267cd4bc9c323b54
Co-Authored-By: Akihiro Motoki <>
4 years ago
LIU Yulong 5b7d444b31 Not set the HA port down at regular l3-agent restart
If l3-agent was restarted by a regular action, such as config change,
package upgrade, manually service restart etc. We should not set the
HA port down during such scenarios. Unless the physical host was
rebooted, aka the VRRP processes were all terminated.

This patch adds a new RPC call during l3 agent init, it will try to
retrieve the HA router count first. And then compare the VRRP process
(keepalived) count and 'neutron-keepalived-state-change' count
with the hosting router count. If the count matches, then that
set HA port to 'DOWN' state action will not be triggered anymore.

Closes-Bug: #1798475
Change-Id: I5e2bb64df0aaab11a640a798963372c8d91a06a8
4 years ago
Boden R 024802aafd remove neutron.common.rpc
The neutron.common.rpc module has been in neutron-lib for awhile now and
neutron is shimmed to use neutron-lib already.
This patch removes neutron.common.rpc and switches the code over to use
neutron-lib's implementation where needed.


Change-Id: I733f07a8c4a2af071b3467bd710290eee11a4f4c
4 years ago
Boden R 68fd13af40 remove neutron.common.exceptions
Today the neutron common exceptions already live in neutron-lib and are
shimmed from neutron. This patch removes the neutron.common.exceptions
module and changes neutron's imports over to use their respective
neutron-lib exception module instead.


Change-Id: I9704f20eb21da85d2cf024d83338b3d94593671e
4 years ago
yangjianfeng 99f4495c94 Improve port dhcp Provisioning
Currently, the dhcp Provisioning of ports is the crucial bottleneck
of that concurrently boot multiple VM.

The root cause is that these ports will be processed one by one by dhcp
agent when they belong to the same network, And the 'Provisioning complete'
port is still blocked other port's processing in other dhcp agents. The
patch aim to optimize the dispatch strategy of the port cast to agent to
improve the Provisioning process.

In server side, I classify messages to multi levels. Especially, I classify
the port_update_end or port_create_end message to two levels, the high-level
message only cast to one agent, the low-level message cast to all agent. In
agent side I put these messages to `resource_processing_queue`, with the queue,
We can delete `_net_lock` and process these messages in order of priority.

Additonally, I modified the `resource_processing_queue` for my demand. I update
`_queue` from LIST to PriorityQueue in `ExclusiveResourceProcessor`, by this
way, we can sort all message which cached in `ExclusiveResourceProcessor` by

Related-Bug: #1760047
Change-Id: I255caa0571c42fb012fe882259ef181070beccef
4 years ago
Kailun Qin b70ee4df88 Block port update from unbound DHCP agent
Current DHCP port management in Neutron makes the server to clear the
device_id while the agent is responsible for setting it.

This may cause a potential race condition, for example during network
rescheduling. The server aims to clear the device_id on a DHCP port and
assign the network to another agent while the old agent might just be
taking possession of the port. If the DHCP agent takes possession of the
port (i.e., update port...set the device_id) before the server clears
it, then there is no issue. However, if this happens after the clear
operation by server then the DHCP port would be updated/marked to be
owned by the old agent.

When the new agent takes over the network scheduled to it, it won't be
able to find a port to reuse so that an extra port might need to be
created. This leads to two issues:
1) an extra port is created and never deleted;
2) the extra port creation may fail if there are no available IP

This patch proposes a validation check to prevent an agent from updating
a DHCP port unless the network is bound to that agent.

Co-authored-by: Allain Legacy <>

Closes-Bug: #1795126
Story: 2003919
Change-Id: Ie619516c07fb3dc9d025f64c0e1e59d5d808cb6f
5 years ago
Boden R e4aa5902f7 use context manager from neutron-lib
The neutron.db.api.context_manager already references neutron-lib's
context manager; so consumers of it are already using neutron-lib. This
patch switches neutron's references to the context_manager over to
use neutron-lib's directly rather than that in neutron.db.api.


Change-Id: I97120faeec73690592ed21a5ec3c6202f61e1429
5 years ago
Doug Hellmann 2530ccff8f Upgrade pylint to a version that works with python3
Also fixed some six.moves errors as well.

Change-Id: I9783ee588de1ec4b91a665b9dc89fdba3e2d90df
Signed-off-by: Doug Hellmann <>
5 years ago
Zuul 88741f1547 Merge "Generate correct url in api pagination" 5 years ago
Swaminathan Vasudevan fd72643a61 Revert "DVR: Inter Tenant Traffic between networks not possible with shared net"
This reverts commit d019790fe4.

Closes-Bug: #1783654
Change-Id: I4fd2610e185fb60cae62693cd4032ab700209b5f
5 years ago
ZhaoBo 35d945e92f Add ext_parent policy check
Add common parent owner check for the resources which introduced by
service plugin.

Then port forwarding resource will share the same tenant_id with
floatingip. That means only the fip owner can create/update/get/delete
the associated port forwarding resource.

Partially-Implements: blueprint port-forwarding
Partial-Bug: #1491317
Change-Id: I450c674e55ca15e1d9a6a6224138f3305427da68
5 years ago
ZhaoBo 21ae99d5b3 [server side] Floating IP port forwarding plugin
This patch implements the plugin.
This patch introduces an new service plugin for port forwarding resources,
named 'pf_plugin', and supports create/update/delete port forwarding
operation towards a free Floating IP.

This patch including some works below:
* Introduces portforwarding extension and the base class of plugin
* Introduces portforwarding plugin, support CRUD port forwarding
* Add the policy of portforwarding

The race issue fix in:

Fip extend port forwarding field addition in:

Partially-Implements: blueprint port-forwarding
Change-Id: Ibc446f8234bff80d5b16c988f900d3940245ba89
Partial-Bug: #1491317
5 years ago
Zuul ca13e651c9 Merge "Implement filter validation" 5 years ago
Hongbin Lu 2b1d8ea4a2 Implement filter validation
Enforce validation on filter parameters on list requests.
If an API request contains an unknown or unsupported parameter,
the server will return a 400 response instead of silently ignoring
the invalid input.

In resource attributes map, all filter parameters are annotated by
the ``is_filter`` keyword. Attributes with is_filter set to True
are candidates for validation.

Enabling filter validation requires support from core plugin and
all service plugins so each plugin need to indicate if it supports
the validation by setting ``__filter_validation_support`` to True.
If this field is not set, the default is False and validation is
turned off. Right now, the ML2 plugin and all the in-tree service
plugin support filter validation. Out-of-tree plugins will have
filter validation disabled by default.

An API extension is introduced to allow API users to discover this
new API behavior. This feature can be disabled by cloud operators
if they choose to do that. If it is disabled, the extension won't
be presented.

Depends-On: Ic3ab5b3ffdc378d570678b9c967cb42b0c7a8a9b
Depends-On: I4397df1c35463a8b532afdc9c5d28b37224a37b4
Depends-On: I3f2e6e861adaeef81a1a5819a57b28f5c6281d80
Depends-On: I1189bc9a50308df5c7e18c329f3a1262c90b9e12
Depends-On: I057cd917628c77dd20c0ff7747936c3fec7b4844
Depends-On: I0b24a304cc3466a2c05426cdbb6f9d99f1797edd

Change-Id: I21bf8a752813802822fd9966dda6ab3b6c4abfdc
Partial-Bug: #1749820
5 years ago
Zuul 587bbe171e Merge "[server side] Floating IP port forwarding OVO and db script" 5 years ago
Zuul 414c0d40c9 Merge "use retry_db_errors from neutron-lib" 5 years ago
Zuul 5d8a0d9ca3 Merge "Ensure request's object type is dict" 5 years ago
Boden R e4348eb1e1 use retry_db_errors from neutron-lib
The externally consumed APIs from neutron.db.api were rehomed into
neutron-lib with

This patch consumes the retry_db_errors function from lib by:
- Removing retry_db_errors from neutron.db.api
- Updating the imports for retry_db_errors to use it from lib
- Using the DB API retry fixture from lib in the UTs where applicable
- Removing the UTs for neutron.db.api as they are now covered in lib


Change-Id: I1feb842d3e0e92c945efb01ece29856335a398fe
5 years ago
ZhaoBo 5bd6281f9c [server side] Floating IP port forwarding OVO and db script
This patch implements the port forwarding OVO and db layer code.
Such as:
* Introduces a new OVO named 'PortForwarding'.
* Introduces a new db model for OVO.
* A migration db script for port forwarding function.

Partially-Implements: blueprint port-forwarding
This patch partially implements the following spec:

The race issue fix in:

Fip extend port forwarding field addition in:

Partial-Bug: #1491317
Change-Id: If24e1b3161e2a86ccc5cc21acf05d0a17f6856e7
5 years ago
Zuul 14a22799f5 Merge "use plugin utils from neutron-lib" 5 years ago
Boden R 839e575fa6 use plugin utils from neutron-lib
The remainder of the neutron.plugins.common.utils were rehomed into
neutron-lib with [1][2]. This patch consumes them by using the functions
from neutron-lib, and removing the neutron.plugins.common.utils module
all together as it's fully rehomed now.



Change-Id: Ic0f7b37861f078ce8c5ee92d97e977b8d2b468ad
5 years ago
Hongbin Lu d950949b90 Ensure request's object type is dict
Handle an edge case that API users send a POST/PUT request with
invalid data in request body.

Closes-Bug: #1780327
Change-Id: I3877c18a18ac506dc8f4a9ded2a18b53b9f6cfae
5 years ago
IWAMOTO Toshihiro 0f180d8e2e Adopt hacking 1.1.0
This incorporates flake8 2.6.x and pycodestyle will be used
instead of older pep8.  This ensures future python3 compatibility
and a bit better code styling.

Change-Id: Ia7c7c5a44727f615a151e1e68dd94c7ed42f974f
5 years ago
Mykola Yakovliev d62d82b0fe Generate correct url in api pagination
Use protocol from X-Forwarded-Proto, and not from current endpoint.

Change-Id: I3fafc4ef1cf56cb8f1cddc1a003a755e3f93c75c
Closes-Bug: 1774460
5 years ago
Zuul 5cf8f699e2 Merge "Fix lack of routes for neighbour IPv4 subnets" 5 years ago
Brian Haley 2e34279ec3 Fix lack of routes for neighbour IPv4 subnets
According to [1], when a network contains more that one IPv4
subnet, they are returned in the 'classless-static-routes'
DHCP option, regardless of whether DHCP is enabled for them
or not.

However, the get_active_networks_info() method used for
synchronizing networks after the dhcp agent restarts filters
subnets with "enable_dhcp=True", which differs from the
get_network_info() method.  This will block VM access to
other VMs in the dhcp disabled subnets, even though they are
in the same network.  This is visible by looking at the "opts"
file before and after a restart.

Change the dhcp agent to ask for all subnets in its
get_active_networks_info() RPC call by adding an
enable_dhcp_filter argument to toggle the behavior, with the
default being True to not break backwards compatibility.

Based on by Quan Tian.


Change-Id: I11ca1d1a603d02587f3b8d4a5a52a96b0587d61f
Closes-Bug: #1652654
5 years ago
Zuul ea40415d22 Merge "Populate project info before using it" 5 years ago
Hongbin Lu f213ba487b Populate project info before using it
Sorting and filtering will rely on the attributes information. It is
necessary to populate project info before using it to sort/filter.

Closes-Bug: #1765452
Change-Id: Ife90268530b6e86a0b0d213e4742a2ef81cb2395
5 years ago
Allain Legacy fa78b58010 dhcp: serializing port delete and network rpc calls
The port delete events are not synchronized with network rpc events.  This
creates a condition which makes it possible for a port delete event to be
processed just before a previously started network query completes.

The problematic order of operations is as follows:

  1) a network is scheduled to an agent; a network rpc is sent to the

  2) the agent queries the network data from the server

  3) while that query is in progress a port on that network is deleted; a
     port rpc is sent to the agent

  4) that port delete rpc is received before the network query rpc

  5) the port delete results in no action because the port was not present
     on the agent

  6) the network query finishes and adds the port to the cache (even
     though the port has already been deleted)

  7) some time passes and a new port is configured with the same IP
     address as the port that was deleted in (3)

  8) the dhcp host file is corrupted with 2 entries for the same IP

  9) dhcp queries for the newest port is rejected because of the duplicate
     entry in the dhcp host file.

The solution is to add the network_id to the port_delete_end rpc event
so that the _net_lock(network_id) synchronization point can be acquired
so that it is processed serially with other network related events.

To ensure backwards compatibility with newer agents running against older
servers the determination of which network_id value to use in the lock is
handled using a utility that will fallback to the previous mode of operation
whenever the network_id attribute is not present in the *_delete_end RPC
events.  That utility can be removed in the future when it is guaranteed
that the network_id attribute will be present in RPC messages from the

Closes-Bug: #1732456

Change-Id: I735f8b1c9248b12e5feb6cbe970cf67f321e6ebc
Signed-off-by: Allain Legacy <>
5 years ago
Zuul d1a580541a Merge "Support filter attribute with empty string" 5 years ago
Brian Haley 7cfdf4aa81 Fix all pep8 E129 errors
Fixed all pep8 E129 errors and changed tox.ini to no longer
ignore them.

Change-Id: I0b06d99ce1d473b79a4cfdd173baa4f02e653847
5 years ago
Hongbin Lu a732bbf19e Support filter attribute with empty string
This will enable users to filter list of results with attributes
with empty value. For example, the request below will list
all unbound ports (unbound ports have blank device_id).

  GET "/ports?device_id="

Change-Id: I9001214de126eb888c2425b6a6275f59ec8478e7
Closes-Bug: #1749304
5 years ago
Brian Haley c3b83a9ca6 Fix all pep8 E265 errors
Fixed all pep8 E265 errors and changed tox.ini to no longer
ignore them.  Also removed an N536 comment missed from a
previous change.

Change-Id: Ie6db8406c3b884c95b2a54a7598ea83476b8dba1
5 years ago
Brian Haley 90cd939047 Fix W503 pep8 warnings
Fix W503 (line break before binary operator) pep8 warnings
and no longer ignore new failures.


Change-Id: I7539f3b7187f2ad40681781f74b6e05a01bac474
5 years ago
Dmitrii Shcherbakov ff5e8d7d6c Refresh router objects after port binding
Post-binding information about router ports is missing in results of RPC
calls made by l3 agents. sync_routers code ensures that bindings are
present, however, it does not refresh router objects before returning
them - for RPC clients ports remain unbound before the next sync and
there is no necessary address scope information present to create routes
from fip namespaces to qrouter namespaces.

Change-Id: Ia135f0ed7ca99887d5208fa78fe4df1ff6412c26
Closes-Bug: #1759971
5 years ago
Vu Cong Tuan 7988e32202 Removing deprecated module neutron.api.versions
'Versions.factory()' is deprecated in version 'Queens'
and was marked to be removed 'Rocky'.
This patch removes it and switches to use instead.

In addition, neutron.api.versions is removed
because this deprecated module isn't used anywhere.

Change-Id: I08d1725a973c734b55771918915428377fb7d3b4
5 years ago
Zuul 522da182b4 Merge "DVR: Inter Tenant Traffic between networks not possible with shared net" 5 years ago
Boden R ef93f7e7f0 use common agent topics from neutron-lib
The neutron.common.topics module was rehomed into neutron-lib with
commit Ie88b84949cbd55a4e7ad06341aab77b286cdc485
This patch consumes it by removing the rehomed module from neutron
and using the module from neutron-lib instead.


Change-Id: Ia4a4604c259ce862597de80c6deeb3d408bf0e95
5 years ago
Boden R 062ef79381 use is_extension_supported from neutron-lib
The is_extension_supported function now lives in neutron-lib. This patch
removes the function from neutron and uses lib's version instead.


Change-Id: Iccb72e00f85043b3dff0299df7eb1279655e313e
5 years ago