Removed E125 (continuation line does not distinguish itself
from next logical line) from the ignore list and fixed all
the indentation issues. Didn't think it was going to be
close to 100 files when I started.
Change-Id: I0a6f5efec4b7d8d3632dd9dbb43e0ab58af9dff3
https://review.opendev.org/#/c/236983/https://review.opendev.org/#/c/606383/
The above patchs that resolve the race condition of DHCP agent will
result in neutron-server raise DhcpPortInUse ERROR log. And, the
second patch may result in old dhcp agent create a redundant port.
Closes-Bug: #1829332
Change-Id: If7a7ac2f88ce5b0e799c1104c936735a6cc860aa
All of the externally consumed variables from neutron.common.constants
now live in neutron-lib. This patch removes neutron.common.constants
and switches all uses over to lib.
NeutronLibImpact
Depends-On: https://review.openstack.org/#/c/647836/
Change-Id: I3c2f28ecd18996a1cee1ae3af399166defe9da87
Reduces E128 warnings by ~260 to just ~900,
no way we're getting rid of all of them at once (or ever).
Files under neutron/tests still have a ton of E128 warnings.
Change-Id: I9137150ccf129bf443e33428267cd4bc9c323b54
Co-Authored-By: Akihiro Motoki <amotoki@gmail.com>
Today the neutron common exceptions already live in neutron-lib and are
shimmed from neutron. This patch removes the neutron.common.exceptions
module and changes neutron's imports over to use their respective
neutron-lib exception module instead.
NeutronLibImpact
Change-Id: I9704f20eb21da85d2cf024d83338b3d94593671e
Current DHCP port management in Neutron makes the server to clear the
device_id while the agent is responsible for setting it.
This may cause a potential race condition, for example during network
rescheduling. The server aims to clear the device_id on a DHCP port and
assign the network to another agent while the old agent might just be
taking possession of the port. If the DHCP agent takes possession of the
port (i.e., update port...set the device_id) before the server clears
it, then there is no issue. However, if this happens after the clear
operation by server then the DHCP port would be updated/marked to be
owned by the old agent.
When the new agent takes over the network scheduled to it, it won't be
able to find a port to reuse so that an extra port might need to be
created. This leads to two issues:
1) an extra port is created and never deleted;
2) the extra port creation may fail if there are no available IP
addresses.
This patch proposes a validation check to prevent an agent from updating
a DHCP port unless the network is bound to that agent.
Co-authored-by: Allain Legacy <Allain.legacy@windriver.com>
Closes-Bug: #1795126
Story: 2003919
Change-Id: Ie619516c07fb3dc9d025f64c0e1e59d5d808cb6f
The externally consumed APIs from neutron.db.api were rehomed into
neutron-lib with https://review.openstack.org/#/c/557040/
This patch consumes the retry_db_errors function from lib by:
- Removing retry_db_errors from neutron.db.api
- Updating the imports for retry_db_errors to use it from lib
- Using the DB API retry fixture from lib in the UTs where applicable
- Removing the UTs for neutron.db.api as they are now covered in lib
NeutronLibImpact
Change-Id: I1feb842d3e0e92c945efb01ece29856335a398fe
The remainder of the neutron.plugins.common.utils were rehomed into
neutron-lib with [1][2]. This patch consumes them by using the functions
from neutron-lib, and removing the neutron.plugins.common.utils module
all together as it's fully rehomed now.
NeutronLibImpact
[1] https://review.openstack.org/#/c/560950/
[2] https://review.openstack.org/#/c/554546/
Change-Id: Ic0f7b37861f078ce8c5ee92d97e977b8d2b468ad
According to [1], when a network contains more that one IPv4
subnet, they are returned in the 'classless-static-routes'
DHCP option, regardless of whether DHCP is enabled for them
or not.
However, the get_active_networks_info() method used for
synchronizing networks after the dhcp agent restarts filters
subnets with "enable_dhcp=True", which differs from the
get_network_info() method. This will block VM access to
other VMs in the dhcp disabled subnets, even though they are
in the same network. This is visible by looking at the "opts"
file before and after a restart.
Change the dhcp agent to ask for all subnets in its
get_active_networks_info() RPC call by adding an
enable_dhcp_filter argument to toggle the behavior, with the
default being True to not break backwards compatibility.
Based on https://review.openstack.org/#/c/352530/ by Quan Tian.
[1] https://review.openstack.org/#/c/125043/
Change-Id: I11ca1d1a603d02587f3b8d4a5a52a96b0587d61f
Closes-Bug: #1652654
Fix W503 (line break before binary operator) pep8 warnings
and no longer ignore new failures.
Trivialfix
Change-Id: I7539f3b7187f2ad40681781f74b6e05a01bac474
The is_extension_supported function now lives in neutron-lib. This patch
removes the function from neutron and uses lib's version instead.
NeutronLibImpact
Change-Id: Iccb72e00f85043b3dff0299df7eb1279655e313e
The neutron-lib commit I360545b6ee4291547e0c5c8e668ad03d3efa4725 moved
the externally consumed globals from neutron.common.constants into lib.
With the exception of PROVISIONAL_IPV6_PD_PREFIX all other constants
in neutron.common.constants should only be used in neutron, and will
hopefully remain that way. External consumers needing access to other
common constants should move them into lib first.
NeutronLibImpact
Change-Id: Ie4bcffccf626a6e1de84af01f3487feb825f8b65
Changing rpc_api.rst file path from doc/source/devref/rpc_api.rst
to /doc/source/contributor/internals/rpc_api.rst. Because rpc_api.rst
file is located at this path
doc/source/contributor/internals/rpc_api.rst.
Closes-Bug #1722072
Change-Id: Ic243aab9e3428bfec69db61a94b4129cd768e233
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
In order to allow the DHCP agent to service other subnets
on the network in other segments via DHCP relay, we need to
return all subnets regardless of segment association.
However, this behavior will break older DHCP agents that
then try to get IPs on the subnets belonging to other segments.
This patch adds a new subnet attribute, 'non_local_subnets'
that will be returned in the DHCP RPC calls, so that agents
that can deal with off-link subnets can handle them
accordingly.
Change-Id: I9cce7b8a19c1201435df0c6baac7be57c57847e6
Partial-Bug: #1692486
The callback modules have been available in neutron-lib since commit [1]
and are ready for consumption.
As the callback registry is implemented with a singleton manager
instance, sync complications can arise ensuring all consumers switch to
lib's implementation at the same time. Therefore this consumption has
been broken down:
1) Shim neutron's callbacks using lib's callback system and remove
existing neutron internals related to callbacks (devref, UTs, etc.).
2) Switch all neutron's callback imports over to neutron-lib's.
3) Have all sub-projects using callbacks move their imports over to use
neutron-lib's callbacks implementation.
4) Remove the callback shims in neutron-lib once sub-projects are moved
over to lib's callbacks.
5) Follow-on patches moving our existing uses of callbacks to the new
event payload model provided by neutron-lib.callback.events
This patch implements #2 from above, moving all neutron's callback
imports to use neutron-lib's callbacks.
There are also a few places in the UT code that still patch callbacks,
we can address those in step #4 which may need [2].
NeutronLibImpact
[1] fea8bb64ba7ff52632c2bd3e3298eaedf623ee4f
[2] I9966c90e3f90552b41ed84a68b19f3e540426432
Change-Id: I8dae56f0f5c009bdf3e8ebfa1b360756216ab886
Neutron-lib 1.1.0 is now out and contains the portbindings
API definition (as per commit [1]). This patch moves neutron
references over to the neutron-lib version.
NeutronLibImpact
- Consumers using the public constants within neutron's
portbindings API extension must now use the values
from neutron-lib.
[1] 87e42f993c07ae320159d5123662ee9f3bd4d903
Change-Id: I669af9b4c712877772d91a03857ab108714001d4
This is a runtime state users can easily create by creating tiny subnets
with no room for DHCP agent addresses. It's not something we need to
dump tracebacks on the server for because it's not a bug or unexpected
outcome.
These traces can be seen all over the API test logs where small subnets
are created without the need to actuall put things on them.
TrivialFix
Change-Id: I663b0df07608d8511a40c82e589875c50c7a6066
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.
Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.
This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.
Usage pattern [1,2] can be translated to [3,4] respectively.
[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()
The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.
Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.
Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.
NeutronLibImpact
Partially-implements: blueprint neutron-lib
Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
With multiple concurrent DHCP operations, the server log may be
filled with warning traces about the inability to complete the
DHCP action correctly. These should be emitted only when reraising
the underlying exception, namely when the underlying error is
not an expected nor a handled one.
Related-bug: 1621345
Related-bug: 1620864
Change-Id: I1f00609f8181e3765238550f45e6e114d2df7e6e
Capture port not found exceptions from port updates of DHCP ports
that no longer exist. The DHCP agent already checks the return
value for None in case any of the other things went missing
(e.g. Subnet, Network), so checking for ports disappearing makes
sense. The corresponding agent-side log message for this has also
been downgraded to debug since this is a normal occurrence.
This also cleans up log noise from calling reload_allocations on
networks that have already been torn down due to all of the subnets
being removed.
Closes-Bug: #1621650
Change-Id: I495401d225c664b8f1cf7b3d51747f3b47c24fc0
Commit 85ed7017ff removed the DBError
handling to let the retry decorator do its magic, however the
full implications of this change were not evaluated. As a result,
DBReferenceError (which derives from DBError) is not processed
correctly and that caused a regression of the existing logic.
Rather than bloat the retry's responsibility even further, this
patch partially reverts commit 85ed7017f by narrowing down the
exception handling to DBReferenceError only.
Related-bug: #1618216
Change-Id: Icf4e5e4145dcdcdc710b8e42044467913ed01ec1
The DHCP port action handler has been catching DBErrors
since f1b9ac5a54, which is
well before we had the retry decorator to deal with these.
With the port action handler catching these, it means there
will not be retries on deadlocks or connection errors so
transient situations can result in a permanently broken
DHCP service for a network.
This removes the catch for DBError so the decorator can retry
the operation.
Closes-Bug: #1618216
Change-Id: I42031b481958bbfdb8f52902c294022717af7adf
The exception handler checking for concurrently deleted subnets
was missing InvalidInput, which comes from IPAM when a port creation
request references a subnet that doesn't exist.
This patch just catches that exception.
Change-Id: I14f9e4bddde845e3fef2b0d8649d3954ba5c93bd
Closes-Bug: #1618187
The dhcp agent will request network information from neutron-server.
neutron-server will give the dhcp agent the entire network tree back
to the agent including subnets. If that agent is on a host that
is mapped to a segment, subnets that do not belong to that
segment are also returned and the dhcp agent will attempt to create
or update the dhcp port based on those subnets. These requests
will fail because the ipam logic forbids it for multiple segments.
This patch strips out those subnets that do not belong to the host's
segment. If the segments plugin is not enabled or the network is not
a routed network then the neutron-server will return back to the agent
what it normally would without routed networks.
Change-Id: I379b56f510853c60bb52c2d710117fecfb9aca2e
Partially-Implements: blueprint routed-networks
This method was originally left for backwards compatibility during
an upgrade of neutron. It stopped being used in havana so I think it
should be safe to remove now.
Closes-bug: #1583835
Change-Id: I503d10ee59ba2fa046b5030e8281fd41a49ee74e
When a new subnet is added to a network, the network cache
is updated with the list of subnets regardless of which ones
have DHCP enabled. This changes the index order of the subnet
list which means that the tags used for each subnet change.
This means we must restart the process because the opts file
will be using different tags than the process args. This patch
implements that change. It also sorts the subnets on the RPC
side so the agent indexes don't change if subnets aren't
added/deleted.
The previous logic was only restarting the process when DHCP
enabled subnets changed, which meant that adding a DHCP disabled
subnet would break the association between the opts file tags and
the process arg tags, which led to the reported bug.
Closes-Bug: #1581918
Change-Id: If1452c0e8fe95eb94cd78c7a05b57aead75662b5
Sometimes an object requires multiple disjoint actors to complete
a set of tasks before the status of the object should be transitioned
to ACTIVE. The main example of this is when a port is being created.
The L2 agent has to do its business to wire up the VIF, but at the same
time the DHCP agent has to setup the DHCP reservation. This led to
Nova booting the VM when the L2 agent was done even though the DHCP
agent may have been nowhere near ready.
This patch introduces a provisioning blocks mechansim that allows the
entities to be tracked that need to be involved to make a transition
to ACTIVE happen. See the devref in the dependent patch for a high-level
view of how this works.
The ML2 code is updated to use this new mechanism to prevent updating
the port status to ACTIVE without both the DHCP agent and L2 agent
reporting that the port is ready.
The DHCP RPC API required a version bump to allow the port ready
notification.
This also adds a devref doc for the provisioning_blocks
module with a high-level overview of how it works in addition
to a detailed description of how it is used specifically with
ML2, the L2 agents, and the DHCP agents.
Closes-Bug: #1453350
Change-Id: Id85ff6de1a14a550ab50baf4f79d3130af3680c8
Python 3 deprecated the logger.warn method, see:
https://docs.python.org/3/library/logging.html#logging.warning
so we prefer to use warning to avoid DeprecationWarning.
Closes-Bugs: #1529913
Change-Id: Icc01ce5fbd10880440cf75a2e0833394783464a0
Co-Authored-By: Gary Kotton <gkotton@vmware.com>
This is a normal occurence as networks are created and
deleted. It's not something that an operator should be
warned about.
Change-Id: I3bb498a29d93a88b059a669d510d21b4c65ea014
Closes-Bug: #1554695
Wrong usage of "a" in the messages:
"'%s' is not a integer"
"Create a address scope"
"Return True if port has as a allowed address pair"
"But if a agent does not report its status"
"a ipv4 link-local address"
"Retrieve and return a extended information about a network"
"It could be a eui-64 address, a random IPv6 address"
"push a elastic-recheck query for it (see below)"
"is not a enforced requirement"
"a ovs_lib.VifPort object"
Should be:
"'%s' is not an integer"
"Create an address scope"
"Return True if port has as an allowed address pair"
"But if an agent does not report its status"
"an ipv4 link-local address"
"Retrieve and return extended information about a network"
"It could be an eui-64 address, a random IPv6 address"
"push an elastic-recheck query for it (see below)"
"is not an enforced requirement"
"an ovs_lib.VifPort object"
Totally 9 occurrences in Neutron base code.
Change-Id: I0f980fc8c98524db3d194ecb779f76abb37eb31c
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
This patch introduces mechanism similar to compare-and-swap
for updating reserved DHCP port.
This addresses a case when two DHCP agents that start nearly at
the same time are assigned to one network and there is a reserved
DHCP port in the network. Then each of agents will try to use it
because agents don't check if reserved port is still available.
Reserved DHCP port can be acquired by different agent between calls to
get_active_networks and update_port, so this patch adds a check for
this case.
Change-Id: I0277ab537ff9d3a664c03ea291b9ec2b0e784dbb
Closes-Bug: #1425402
Added deepcopy of port dict in dhcp rpc handler to prevent operating on
changed dict in case of raising retry request.
Change-Id: Ie1816fe819ba77061e71bd61de2fd9ebe4574cef
Related-Bug: #1494351
When API controller calls method create_{network, subnet, port),
it made sure that the necessary default values for attrs are filled properly
according to attr mapping.
However, internal calls to these methods do not follow the convention,
when extension codes miss these values, exceptions will be thrown.
This patch introduces helper functions to fix up arguments and replaces
the direct callers of those methods.
Co-Authored-By: gong yong sheng <gong.yongsheng@99cloud.net>
Co-Authored-By: yalei wang <yalei.wang@intel.com>
Change-Id: Ibc6ff897a1a00665a403981a218100a698eb1c33
Closes-Bug: #1383546
The decorator was previously added at the API layer
(commit 4e77442d52,
commit d04335c448).
However some RPC handlers are also dealing with port
create/update/delete operations, like dhcp ports for example.
We need to cover these cases too.
Also remove db retry from ml2 plugin delete_port()
as it's not needed once we retry at the API and RPC layers.
(there is already a unit test on this)
The patch also adds a unit test for checking deadlock
handling during port creation at API layer.
Though it's not directly related to the current fix,
I decided to leave it for regression preventing purposes.
Closes-Bug: #1479738
Change-Id: I7793a8f7c37ca542b8bc12372168aaaa0826ac4c
Brian Haley