The resource_extend module was rehomed into neutron-lib with commit
https://review.openstack.org/#/c/613431/
This patch shims neutron.db._resource_extend to reference neutron-lib's
implementation allowing consumers to switch over with worrying about
dependency chains. Once all consumers switch over a follow-up patch
will be submitted to remove neutron's _resource_extend module.
NeutronLibImpact
Change-Id: I9ffe91b6dedaf46d679c6a3c4f465f2bfd66e32d
In Neutron we hit quite often same issue as Manila, see [1] for
details.
It looks that solution for this problem may be increase timeout
for test_walk_version functional tests.
Higher timeout will be applied for tests for both pgsql and mysql
backends but it is mostly needed for mysql because 'pymysql' works
much slower on slow nodes than 'psycopg2'
This patch adds also new decorator to set individual timeout for
tests.
[1] https://bugs.launchpad.net/manila/+bug/1501272
Change-Id: I5f344af6dc3e5a6ee5f52c250b6c719e1b43e02d
Closes-Bug: #1687027
The deprecated rpc_backend parameter was removed from oslo.messaging in
I193cc0e613459a6dbbfd54ed0901a54ded78d712
Use transport_url parameter instead
Change-Id: Ia73fe052986617dd6ce5e68cced0c4c88516a33e
Closes-Bug: #1795878
The common rpc and exceptions were rehomed into
neutron-lib with [1]. This patch shims those rehomed
modules in neutron to switch over to neutron-lib's
versions under the covers.
To do so:
- The rpc and common exceptions are changed to
reference their counterpart in neutron-lib effectively
swapping the impl over to neutron-lib.
- The fake_notifier is removed from neutron and lib's
version is used instead.
- The rpc tests are removed; they live in lib now.
- A few unit test related changes are required
including changing mock.patch to mock.patch.object,
changing the mock checks for a few UTs as they don't
quite work the same with the shim in place.
- Using the RPC fixture from neutron-lib rather than
that setup in neutron's base test class.
With this shim in place, consumers are effectively using
neutron-lib's RPC plumbing and thus we can move consumers
over to neutron-lib's version at will. Once all
consumers are moved over we can come back and remove
the RPC logic from neutron and follow-up with a consumption
patch.
NeutronLibImpact
[1] https://review.openstack.org/#/c/319328/
Change-Id: I87685be8764a152ac24366f13e190de9d4f6f8d8
This patch shims the model query hook functionality to use neutron-lib.
A shim is used as the query hooks are cached in a shared dict and by
shimming them to use neutron-lib we can let consumers move over to
neutron-lib's version at will. Once all current consumers are moved
over to lib, we can remove the shim and consume from lib directly.
The DBQueryHooksFixture is also used in this patch to replace the
existing cleanup function in the unit tests.
Change-Id: I18a4e836ce89b607b5452527e5f8dd172cad6b4f
The public db apis are available in neutron-lib. This patch consumes
the sqla_listen, sqla_remove and sqla_remove_all functions from lib
by removing them from neutron and using lib's version.
There's also a TODO added here so we can eventually rehome the
check_no_sqlalchemy_event_import hacking check to neutron-lib.
NeutronLibImpact
Change-Id: I3b3862016125303fb227b4ff41420a89bb33c917
oslo_messaging's rpc_backend setting, which is set by
ConfFixture.transport_driver has been deprecated since Newton. To allow
oslo_messaging to remove it, switch to setting transport_url instead.
Partial-Bug: #1712399
Change-Id: I2fbf5159febe9d4970935d77bd75e84204710c8e
Check if network device exists is now done by checking
interface index with pyroute2 interface instead of checking
if MAC address for device is set.
Change-Id: I2d5b95ec109fb19fc2a46c1017959f74011b9a22
Related-Bug: #1492714
Today neutron.common.rpc has a handful of public constants and functions
that are not used outside of neutron. This patch removes those not
externally used and enhances the rpc.init function to accept extra
RPC exception modules (though it's not used today) to still support
the notion of "extra mods" for RPC.
This patch also paves the way for easier rehoming the module to
neutron-lib.
Change-Id: I51607216a39980c17807dcccace3ab66b549607b
The core resource API definitions are in neutron-lib and are already
setup in a RESOURCES map, similar to neutron's global
RESOURCE_ATTRIBUTE_MAP. However, a number of consumers directly use
RESOURCE_ATTRIBUTE_MAP and moreover can perform some interesting
operations on it while mockin" for tests. For that reason this
patch proposes we phase in the use of neutron's RESOURCES map rather
than a rip and replace of RESOURCE_ATTRIBUTE_MAP.
This patch removes the API definitions for the core neutron resources
by updating neutron's global RESOURCE_ATTRIBUTE_MAP to reference lib's
RESOURCE map. In addition the AttributeMapMemento class is removed; it
no longer servers a purpose and neutron-lib's fixture should be used
in its place.
This patch should be safe to merge without worry of impacting consumers.
No one is using AttributeMapMemento [1] and the global map only changed
its reference value.
[1] http://codesearch.openstack.org/?q=AttributeMapMemento
Change-Id: Ib04fddcbd4465074452b71b16befa4e33b27259e
As it was agreed on Neutron CI meeting, we're going to mark unstable
tests in fullstack suite with this decorator while working in paralel on
stabilization of such tests.
One test in fullstack suite was marked with this decorator to prove its
functionality.
Change-Id: I3beb6e7a4d96da778378e9d979cb8c6261f6036b
Today our unit test code uses various ways to "patch" the global
RESOURCE_ATTRIBUTE_MAP as well as extension specific maps in some cases.
This patch consolidates such patching whereby tests should use neutron's
AttributeMapMemento in their setup() chain (only once) if they update
the global map and they should individually handle backup/restore of per
extension map updates. This change will simplify the code and make it
easier to phase-in API definition usage with neutron-lib where we have
some as API definitions and others not. Longer term the
AttributeMapMemento will be replace with neutron-lib's fixture as we
move all extension maps to API definitions in neutron-lib.
Change-Id: I2586f0b11b107d7f57214a0d65bcf7c38a5f0ebb
We plan to switch to devstack-gate for fullstack job, and it revokes
direct sudo calls before executing tests, so we can't rely on sudo
working anymore.
This also moves functional-testing.filters to a more generic filename
(testing.filters) because the filters are now deployed and used by
fullstack target too.
Related-Bug: #1557168
Related-Bug: #1693689
Change-Id: I1718ea51836adbb8ef8dea79822a722dcf111127
Eventlet now deprecate TimeoutError exception [1].
Updating eventlet to latest 0.21.0 can cause UT fail because
of misunderstood between deprecated exception and the new exception class
Timeout. [2]
This patch remove deprecated exception usage, fix the UT failure with eventlet
0.21.0 in order to upgrade eventlet upper-constraint to latest version.
[1]. https://github.com/eventlet/eventlet/blob/master/eventlet/__init__.py#L55
[2]. https://review.openstack.org/#/c/460424/
Change-Id: I48179953ef19ec6b2f3f6706337a1dae1b3830bd
The callback modules have been available in neutron-lib since commit [1]
and are ready for consumption.
As the callback registry is implemented with a singleton manager
instance, sync complications can arise ensuring all consumers switch to
lib's implementation at the same time. Therefore this consumption has
been broken down:
1) Shim neutron's callbacks using lib's callback system and remove
existing neutron internals related to callbacks (devref, UTs, etc.).
2) Switch all neutron's callback imports over to neutron-lib's.
3) Have all sub-projects using callbacks move their imports over to use
neutron-lib's callbacks implementation.
4) Remove the callback shims in neutron-lib once sub-projects are moved
over to lib's callbacks.
5) Follow-on patches moving our existing uses of callbacks to the new
event payload model provided by neutron-lib.callback.events
This patch implements #2 from above, moving all neutron's callback
imports to use neutron-lib's callbacks.
There are also a few places in the UT code that still patch callbacks,
we can address those in step #4 which may need [2].
NeutronLibImpact
[1] fea8bb64ba7ff52632c2bd3e3298eaedf623ee4f
[2] I9966c90e3f90552b41ed84a68b19f3e540426432
Change-Id: I8dae56f0f5c009bdf3e8ebfa1b360756216ab886
The callback modules have been available in neutron-lib since commit [1]
and are ready for consumption.
As the callback registry is implemented with a singleton manager
instance, sync complications can arise ensuring all consumers switch to
lib's implementation at the same time. Therefore this consumption has
been broken down:
1) Shim neutron's callbacks using lib's callback system and remove
existing neutron internals related to callbacks (devref, UTs, etc.).
2) Switch all neutron's callback imports over to neutron-lib's.
3) Have all sub-projects using callbacks move their imports over to use
neutron-lib's callbacks implementation.
4) Remove the callback shims in neutron-lib once sub-projects are moved
over to lib's callbacks.
5) Follow-on patches moving our existing uses of callbacks to the new
event payload model provided by neutron-lib.callback.events
This patch implements #1 from above, shimming neutron's callbacks and
removing devref + UTs. Rather than shimming using debtcollector, this
patch leaves callback constants as-is, and simply references the lib
class/function in its respective neutron callback module. This allows
consumers to test callback types without changing code. For example,
an except block block like that below continues to work even though
the raised exception now lives in lib::
try:
neutron_cb_registry.notify(...)
except neutron_cb_exceptions.CallbackFailure:
handle_exception()
In addition this patch contains minor UT updates to support the shim
approach.
NeutronLibImpact
[1] fea8bb64ba7ff52632c2bd3e3298eaedf623ee4f
Change-Id: Ib6baee2aaeb044aaba42a97b35900d75dd43021f
By registering functions directly we cut off the dependency of the
"resource extend" functions on the plugin. This is a step towards
the goal of removing the CommonDbMixin mixin class.
Also, we register all "resource extend" functions at plugin create
(in __new__) instead of in the class definition (which caused the
hooks to be registered on import). This ensures the "resource
extend" functions are only registered for the plugins/mixins that
are actually used.
Note that decorators are used to register "resource extend" methods,
similar to the callback receiver decorators.
Related-Blueprint: neutron-lib
Change-Id: I128cfda773d5f9597df9cd61261fdc05f2a174aa
according to https://wiki.openstack.org/wiki/Python3, now we should avoid
using six.iteritems and replace it with dict.items.
Change-Id: I58a399baa2275f280acc0e6d649f81838648ce5c
Closes-Bug: #1680761
By registering functions directly we cut off the dependency of the
query hooks on the plugin. This is a step towards the goal of
removing the CommonDbMixin mixin class.
Also, we register all query hooks at plugin create (in __new__)
instead of in the class definition (which caused the hooks to be
registered on import). This ensures the query hooks are only
registered for the plugins/mixins that are actually used.
Since the query hooks are decoupled from the plugin, we remove them
from the extension mixins (make them global in their module). This is
a step towards refactoring all extension mixins for removal.
Extra: In this patch we also remove the CommonDbMixinHooksFixture test
fixture and instead just clear out the hooks after each test.
Related-Blueprint: neutron-lib
Change-Id: Ib6c2134d29e1764de627c3355f6cdee789d6301e
Since the option is False by default in the oslo.log library, this is
effectively a no-op. Also it may confuse readers that try to understand
how logging is set up for tests.
TrivialFix
Change-Id: I091566519912251547c703139b38daea2979a93d
This change will make neutron-server use default values from oslo.db if
any of them are not overridden in config files, instead of applying its
own unique values that may be unexpected to users and operators.
It also makes our config files correct since now we don't claim wrong
values being default. (Remember the config generator was never aware of
our snowflake behaviour.)
If you wonder why we were overriding it in the first place, it was
needed back in the times when we were using db module from
oslo-incubator that has not provided proper defaults. Those times are
long gone.
Of course the change is not impact free, and operators who for some
reason want to keep the old values may need to adjust their
configuration files. Still, I believe using default values from oslo.db
is the correct thing to do long term.
We still need to set connection string for unit tests to sqlite:// since
oslo.db doesn't provide any default value for the option. It should be
fine to effectively unset default value for the option for
neutron-server since no one in production can use sqlite anyway.
This change also drops set_db_defaults function from
neutron.common.config. I have checked that there are no consumers for
the function outside the tree.
NeutronLibImpact
UpgradeImpact
Co-Authored-By: Ihar Hrachyshka <ihrachys@redhat.com>
Change-Id: Ia97d6cf24e554f7b3878dd301ae4e7e10a1c9998
Closes-Bug: #1682307
AFAIU this is now bloatware not really used by anyone. It's better to
clean it up to keep the code cleaner.
TrivialFix
Change-Id: I8ec0a086301e15820d97afde5354e4ef40356989
Probably the most common format for documenting arguments is reST field
lists [1]. This change updates some docstrings to comply with the field
lists syntax.
[1] http://sphinx-doc.org/domains.html#info-field-lists
Change-Id: I8fb91bcb8ae4bfbfa68ddf7c38a4f1f5cf548b36
This will deprecate the notification_driver config setting,
and no config setting will be needed.
Also it lays down the foundation for a more decoupled interaction
with mechanism drivers.
Closes-Bug: #1657379
Related-Bug: #1627749
DocImpact
Change-Id: I2f166a43f0b980ad22617f8a3f7b4cc7f4786c48
Register all sqlalchemy events through a new function in
neutron.db.api so we can keep track of active events and
ensure all are removed at the end of each test run.
Without this, an instance of a plugin may be left around
with the only reference to it existing in SQLAlchemy, where
it will receive events for tests unrelated to it and potentially
interfere.
Change-Id: I8e93eb4e8ef5a13f015db9cd20e44941cdcb72ef
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.
Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.
This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.
Usage pattern [1,2] can be translated to [3,4] respectively.
[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()
The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.
Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.
Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.
NeutronLibImpact
Partially-implements: blueprint neutron-lib
Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
Adopt the timeout override fixture from nova.
Borrowed from I591f3c270804dfb01eab7c8c5d43b87a7abcdeda
Co-Authored-By: Sean Dague <sean@dague.net>
Closes-Bug: #1626277
Change-Id: I0b40ce97ba42d55f048e9728d9f032ada5c85100
This exception kills the running test worker with all test cases
scheduled to it, probably because unittest module is not capable of
surviving it.
This patch makes all test cases to catch the exception and convert it to
unittest' friendly failure mode (triggered with self.fail).
Change-Id: I5b0d1efa458ca57dfce637dc75d419fe127751ed
Closes-Bug: #1625221
neutron-sanity-check tool was importing neutron.tests.base module, which
may be not present on some systems (f.e. RDO splits neutron/tests/
subtree in a separate python-neutron-tests package). It made the tool
not usable in some setups.
https://bugzilla.redhat.com/show_bug.cgi?id=1374282
This is not the first time when we by mistake import from
neutron.tests.* and break distributions. It's time to stop it by
proactively forbidding that pattern via a new hacking check.
Some functions were moved from neutron.tests.base to
neutron.common.utils to fulfill the need requirement. They were moved
using debtcollector, no current consumers should be affected.
Closes-Bug: #1621782
Change-Id: I790777ddcbd1b02218b3db54ae3d5c931d72d4fa
If a resource was registered in the quota engine before
being set as a tracked resource, it would have been registered
as a CountableResource instead of a TrackedResource.
This would cause a failure later when the resources were
being unregistered because the quota engine thought it
should have been a TrackedResource and tried to call
unregister_events on it.
This patch just prevents resources from being registered
as both types. It also callse unregister_all_resources as
part of the standard test case cleanup since the vast majority
of tests are using a plugin that may register resources.
Closes-Bug: #1612222
Change-Id: I8a40f38d7c0e5aeca257ba62115fa9b02ad5aa93
Refactoring neutron common config opts to be in neutron/conf/common so
that all the configuration options reside in a centralized location.
This simplifies the process of looking up the config opts and provides
an easy way to import.
Moved conf/common/config.py to conf/common.py as per review comments.
Partial-Bug: #1563069
Change-Id: Ib5fa294906549237630f87b9c848eebe0644088c
This reverts commit b1cdba1696
Original patch was reverted because it broke neutron plugin's
backward compatibility and needed more work.
This patch fixes that problems:
1) original behaviour of add_agent_status_check,
start_periodic_l3_agent_status_check and
start_periodic_dhcp_agent_status_check methods is deprecated but kept
for using in third part plugins for backward compatibility
2) new add_agent_status_check_worker, add_periodic_l3_agent_status_check
and add_periodic_dhcp_agent_status_check method are implemented
instead and are used for implementing plugins in neutron codebase
Closes-Bug: #1569404
Change-Id: I3a32a95489831f0d862930384309eefdc881d8f6
I think this needs a little more thought. This broke OVN at least and
I don't think that's good. We need to figure out how to be compatible
with existing plugins, even out of tree ones.
This reverts commit 1cafff0871.
Change-Id: Ie087fb11213cc85911483c2d32c463fa9c973e54
Forking a process when multiple threads are running is an unsafe
operation and could cause a lot of problems because only current
thread will continue working in child thread. Any locked by other
thread resource will remain locked forever.
We faced with this problem during oslo.messaging development and
added workaround to hide this problem:
https://review.openstack.org/#/c/274255/
I tried to fix this problem in oslo.service:
https://review.openstack.org/#/c/270832/
but oslo folks said that this fix is ugly and it is wrong way to add
workarounds to common libraries because projects use them incorrectly.
I think that is fair.
So this patch fixes incorrect usage of oslo libraries. In this patch
I extended functionality of NeutronWorker and add there
`worker_process_count` parameter which determines how many processes
should be spawned for this worker. If `worker_process_count` = 0 - don't
create process and spawn thread in scope of current process for worker
Then I moved all background tasks to workers and return them by
`get_workers` method. start_plugin_workers collects plugin's workers
using `get_workers` method and starts in ProcessLauncher first workers
with `worker_process_count` > 0 and only after this starts threaded
workers by simple Launcher
Closes-bug: #1569404
Change-Id: I0544f1d47ae53d572adda872847a56fa0b202d2e
Boden R