Commit Graph

23 Commits (99f4495c940011293e3cabbb590770dc1e7b6900)

Author SHA1 Message Date
Slawek Kaplonski 4ad302af87 All functional tests logs results now
Some functional tests clases didn't inherit from
neutron.tests.functional.base.BaseLoggingTestCase class
and because of that there was no logs from such tests

This patch changes it and logs should be available for all
functional tests.


Change-Id: Ia71390afe9c8191179c626375ed683f242d35b3e
2018-11-21 19:58:18 +00:00
Brian Haley 90cd939047 Fix W503 pep8 warnings
Fix W503 (line break before binary operator) pep8 warnings
and no longer ignore new failures.


Change-Id: I7539f3b7187f2ad40681781f74b6e05a01bac474
2018-04-17 14:22:58 +00:00
Dirk Mueller 3a15d1a710 Remove compat checks for psutil 1.x
We're depending on much newer psutil (3.x at least), so the 1.x
code path doesn't get executed anymore.

Change-Id: I335ebc0e8e49f20be7d05c968f0ddc79e5307820
2017-07-30 09:22:48 +02:00
Jenkins 9d9ae5af4a Merge "functional: Don't write strings to pipe" 2017-06-21 01:47:13 +00:00
Jakub Libosvar 0c8dd7447a functional: Don't write strings to pipe
Python 3 doesn't support writing strings to descriptors. Write bytes

Change-Id: Ie732d00acc67c429d9f714bcfc116554fb523d36
2017-06-20 18:02:45 +00:00
Bernard Cafarelli 033ab6cc17 python3: use binary mode to open file in test
With text mode, trying to write a binary string generates a TypeError in
python 3

Change-Id: I1b7d1a46d98814aacb4bf9a7f80b168c2b786e30
2017-06-20 17:20:56 +00:00
Boden R b65d881792 use worker from neutron-lib
neutron-lib 1.6 is out and among other things contains the rehomed
NeutronWorker class; now named BaseWorker in neutron-lib. This patch
updates references to use the lib version.

Since there's only 1 other consumer of these changes [1], this
patch depends on it so that it's free to merge after the single
consumer is updated.



Change-Id: Id6bd414ae90119bf527bcaca6b88cc33ff79f685
Depends-On: Ifc0ecf6d5d3d4f8c406caad233d3648646724b12
2017-05-16 07:44:08 -06:00
Dirk Mueller f8a224a63f Adjust psutil usage for psutil > 2
psutil 2.x and above has a lot of API changes as described in:

Change-Id: I1380f488390c83a44d91b84218f1b1ba826d03a0
Closes-Bug: 1659146
2017-01-25 00:54:26 +01:00
Armando Migliaccio 17563a802e Adopt neutron-lib plugin directory
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.

Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.

This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.

Usage pattern [1,2] can be translated to [3,4] respectively.

[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()

The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.

Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.

Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.


Partially-implements: blueprint neutron-lib

Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
2016-11-23 04:45:33 -07:00
Ihar Hrachyshka 2a23527a9a tests: stop using ml2 plugin full import paths in tests
Stevedore documentation suggest that full import paths are not supposed
to be user visible. Since unit tests emulate users when configuring
oslo.config, we better off relying on well known plugin aliases than
internal details.

For in-tree that may be not a big deal, but with it we set a bad example
for third parties that may later find their tests broken eg. when we
decide to move code around.


Change-Id: I7bd036ac3df7e7f4c678356d0a793e7d38599dda
2016-07-27 10:40:22 +00:00
Jakub Libosvar a626172706 Move wait_until_true to neutron.common.utils
We need to be able to re-use wait_until_true in tempest scenario tests.
There is tempest bug
that prevents us to do so.

Also wait_until_true is not linux specific so it makes more sense to
have it in common package.

Change-Id: Ib8b0e51dbd9edaa58391774d428a737836dfdf77
2016-06-27 11:40:11 +00:00
Dmitriy Ukhlov 483c5982c0 Revert "Revert "Remove threading before process forking""
This reverts commit b1cdba1696

Original patch was reverted because it broke neutron plugin's
backward compatibility and needed more work.

This patch fixes that problems:
1) original behaviour of add_agent_status_check,
   start_periodic_l3_agent_status_check and
   start_periodic_dhcp_agent_status_check methods is deprecated but kept
   for using in third part plugins for backward compatibility
2) new add_agent_status_check_worker, add_periodic_l3_agent_status_check
   and add_periodic_dhcp_agent_status_check method are implemented
   instead and are used for implementing plugins in neutron codebase

Closes-Bug: #1569404

Change-Id: I3a32a95489831f0d862930384309eefdc881d8f6
2016-06-02 21:26:08 +00:00
Carl Baldwin b1cdba1696 Revert "Remove threading before process forking"
I think this needs a little more thought.  This broke OVN at least and
I don't think that's good.  We need to figure out how to be compatible 
with existing plugins, even out of tree ones.

This reverts commit 1cafff0871.

Change-Id: Ie087fb11213cc85911483c2d32c463fa9c973e54
2016-05-05 21:07:21 +00:00
dukhlov 1cafff0871 Remove threading before process forking
Forking a process when multiple threads are running is an unsafe
operation and could cause a lot of problems because only current
thread will continue working in child thread. Any locked by other
thread resource will remain locked forever.

We faced with this problem during oslo.messaging development and
added workaround to hide this problem:

I tried to fix this problem in oslo.service:

but oslo folks said that this fix is ugly and it is wrong way to add
workarounds to common libraries because projects use them incorrectly.
I think that is fair.

So this patch fixes incorrect usage of oslo libraries. In this patch
I extended functionality of NeutronWorker and add there
`worker_process_count` parameter which determines how many processes
should be spawned for this worker. If `worker_process_count` = 0 - don't
create process and spawn thread in scope of current process for worker

Then I moved all background tasks to workers and return them by
`get_workers` method. start_plugin_workers collects plugin's workers
using `get_workers` method and starts in ProcessLauncher first workers
with `worker_process_count` > 0 and only after this starts threaded
workers by simple Launcher

Closes-bug: #1569404

Change-Id: I0544f1d47ae53d572adda872847a56fa0b202d2e
2016-05-03 21:54:23 +03:00
Victor Stinner 6355173e4c Reuse to_utf8() and jsonutils.dump_as_bytes()
Reactor code:

* Reuse oslo_utils.encodeutils.to_utf8() instead of existing
  isinstance(text, six.text_type) test
* Replace jsonutils.dumps(obj).encode('utf-8') with
* Other minor bytes/Unicode changes

Change-Id: I03b8eff0fd70ab65ac66d6f3221e8ced0a56db17
2016-04-11 12:25:53 +02:00
Bhagyashri Shewale 88e899f7a0 Fix module's import order
Made corrections in import order for built-in, third party and
project specific modules as per OpenStack import standards [1].


Change-Id: I899deefd6ee4732d6c0afd17a5afbe42b0fa37ba
2016-01-22 06:38:42 -08:00
Elena Ezhova dab4acd15d Refactor test_server functional tests
* Launching services with 0 workers makes no sense and would soon
  be prohibited in oslo.service. Passing workers=1 means that a
  service will be run in a current process.
* Extract the dummy _check_active method to the base class.

Change-Id: Ib687bf1b321845098e0564ada46e68a7850b01c6
2015-11-13 14:44:33 +03:00
Eugene Nikanorov 678b431ba4 Spawn dedicated rpc workers for state reports queue
By default spawn one additional rpc worker to process
state report queue.
State report queue will also be processed by regular
rpc workers, but in case these workers are busy with
processing heavy requests, state reports queue will
automatically be consumed by dedicated rpc workers.

This change applies to ML2 plugin only.
Other plugins should implement start_rpc_state_reports_listener
to enable additional rpc workers.

Change-Id: I5f8df6a478f7c82382049274b34b07109eeafbdb
Closes-Bug: #1505217
2015-10-21 17:55:25 +04:00
Ihar Hrachyshka 7bb4092166 Make test_server work with older versions of oslo.service
Change I18a11283925369bc918002477774f196010a1bc3 fixed the test for
oslo.service >= 0.10.0, but it also broke it for older versions of
oslo.service. Since the library has minimal version of >= 0.7.0 in
requirements.txt, test should pass for those versions too.

Now, instead of validating that either reset() or restart() of workers
are triggered on SIGHUP, just validate that .start() is triggered the
expected number of times (either way, no matter how oslo.service decide
to clean up the children, they exit and then are respawned).

Change-Id: I41f9d3af780b3178b075bc1e7084f417a2bd1378
Closes-Bug: #1505645
2015-10-13 14:46:02 +02:00
armando-migliaccio 090fe71359 Fix functional test_server tests
Now oslo.service 0.10.0 no longer sends SIGHUP to parent and
children services.

This was a chance introduced by 286a6ea, and since it invalidated
the very logic under test, this must be revised.

Change-Id: I18a11283925369bc918002477774f196010a1bc3
Closes-bug: #1505438
2015-10-09 20:19:33 -07:00
Neil Jerram 42f80682d3 Handle process disappearing before we ask for its PPID
Change-Id: I573aba8e11dca16f8a6565f7e9704be18e938566
Closes-Bug: #1478190
2015-09-06 01:09:16 +01:00
Terry Wilson 9f6bd17703 Add support for PluginWorker and Process creation notification
There are several cases where plugin initialization should be
handled after neutron-server forks API/RPC workers. For example,
starting a client connection to an SDN controller before forking
copies the fd of the socket to the child process, but then you have
multiple processes trying to read/write the same socket connection.

It is also useful for a plugin to be able to do something in only
one process, regardless of how many workers are forked. One example
would be handling syncing from an external system to the neutron

This patch does 3 things:
1) Treats rpc_workers=0 as = 1. This simplifies the code for
   handling notification that forking has completed. In the
   existing code, calling the notification in the Worker object's
   start() method would happen twice in the case where both api
   and rpc workers were 0, despite there being only one process.
   An earlier patch already changed the default api_workers to be
   the number of processors.
2) Adds notification of forking via the callbacks mechanism.
   Plugins can subscribe to resources.PROCESS, event.AFTER_CREATE
   and do any post-fork initialization that needs to be done for
   every spawned process.
3) Adds core/service plugin calls to get_workers() which defaults
   to returning (). Plugins that need additional processes to spawn
   should just return an iterable of NeutronWorkers that will be
   spawned in their own process.


Closes-Bug: #1463129
Change-Id: Ib99954678c2b4f32f486b537979d446aafbea07b
2015-09-03 06:40:50 +00:00
Elena Ezhova 6d0d729731 Handle SIGHUP: neutron-server (multiprocess) and metadata agent
All launchers implemented in common.service require each service to
implement reset method because it is called in case a process
receives a SIGHUP.

This change adds the reset method to neutron.service.RpcWorker and
neutron.wsgi.WorkerService which are used to wrap rpc and api
workers correspondingly.

Now neutron-server running in multiprocess mode (api_workers > 0 and
rpc_workers > 0) and metadata agent don't die on receiving SIGHUP and support
reloading policy_path and logging options in config.

Note that reset is called only in case a service is running in daemon mode.

Other changes made in the scope of this patch that need to be mentioned:

* Don't empty self._servers list in RpcWorker's stop method

  When a service is restarted all services are gracefully shutdowned,
  resetted and started again (see openstack.common.service code).
  As graceful shutdown implies calling service.stop() and then
  service.wait() we don't want to clean self._servers list because
  it would be impossible to wait for them to stop processing
  requests and cleaning up their resources.
  Otherwise, this would lead to problems with rpc after starting
  the rpc server again.

* Create a duplicate socket each time WorkerService starts

  When api worker is stopped it kills the eventlet wsgi server
  which internally closes the wsgi server socket object. This server
  socket object becomes not usable which leads to "Bad file
  descriptor" errors on service restart.

Added functional and unit tests.

Partial-Bug: #1276694
Change-Id: I75b00946b7cae891c6eb192e853118e7d49e4a24
2015-06-09 16:15:37 +03:00