Now that we are python3 only, we should move to using the built
in version of mock that supports all of our testing needs and
remove the dependency on the "mock" package.
This patch moves all references to "import mock" to
"from unittest import mock". It also cleans up some new line
Fixed an inconsistency in the OVSBridge.deferred() definition
as it needs to also have an *args argument.
Fixed an issue where an l3-agent test was mocking
functools.partial, causing a python3.8 failure.
Unit tests only, removing from tests/base.py affects
functional tests which need additional work.
The help for the rpc_workers config option is:
Number of RPC worker processes for service. If not specified, the
default is equal to half the number of API workers.
However, this does not accurately describe the current behaviour,
which is to default to half the _default_ number of API workers. This
can make a big difference; for example on a 256-CPU machine with 256GB
of RAM which has api_workers configured to 8 but rpc_workers not
configured to anything, this will result in 64 RPC workers, which is 8
for every API worker!
Therefore tweak the default to rely on the actual value of
api_workers, which may be different than the default value.
We spawn a lot of neutron-servers, on all but the smallest systems.
It's often hard to tell which are busy/overloaded or spinning.
Add an option to set the process names to their role.
This has a small chance of breaking existing scripting, depending
how they're parsing ps output.
$ ps xw | grep neutron-server
1126 pts/2 S+ 0:00 grep --color=auto neutron-server
25355 ? Ss 0:26 /usr/bin/python /usr/local/bin/neutron-server \
--config-file /etc/neutron/neutron.conf \
25368 ? S 0:00 neutron-server: api worker
25369 ? S 0:00 neutron-server: api worker
25370 ? S 0:00 neutron-server: api worker
25371 ? S 0:00 neutron-server: api worker
25372 ? S 0:02 neutron-server: rpc worker
25373 ? S 0:02 neutron-server: rpc worker
25374 ? S 0:02 neutron-server: services worker
The "normal" looking ps output is the main parent.
- Limit number of api workers to roughly using half of system
RAM. Spawning a bunch, just to have the OOM killer nuke them
regularly is not useful.
- Bump the rpc_workers default to half of the api_workers.
A default of 1 falls behind on any reasonably sized node.
This patch switches callbacks over to the payload object style events
 for BEFORE_SPAWN and AFTER_SPAWN based notifications. As no event
data is passed for SPAWN notifications, an actual payload object is
not needed and thus this change uses publish() rather than notify()
and updates and callback receivers to accept the payload parameter.
Add callback (PRCESS, AFTER_SPAWN) to do housekeeping task.
ML2 driver(in fact, networking-odl) wants to run house keeping task
which needs to run only in api server.
The following is very netwokring-odl specific as FYI:
The task is to run a timer which is extended when rest requests are
handled. The timer is fired to kick journal thread only when rest
request is idle in order to check if there is journal entries that was
created by other neutron servers that crash before processing them.
Such timer isn't needed for each workers. Only single timer among api
worker and rpc workers is enough.
The current approach is to run timer unconditionally for all processes
and stop it if it's worker process. With this patch, the timer can be
simply run for main process(api worker).
TestRpcWorker and TestWorkerService have a duplicate test_reset.
This patch introduces a base class from which tests for service
workers can inherit.
All launchers implemented in common.service require each service to
implement reset method because it is called in case a process
receives a SIGHUP.
This change adds the reset method to neutron.service.RpcWorker and
neutron.wsgi.WorkerService which are used to wrap rpc and api
Now neutron-server running in multiprocess mode (api_workers > 0 and
rpc_workers > 0) and metadata agent don't die on receiving SIGHUP and support
reloading policy_path and logging options in config.
Note that reset is called only in case a service is running in daemon mode.
Other changes made in the scope of this patch that need to be mentioned:
* Don't empty self._servers list in RpcWorker's stop method
When a service is restarted all services are gracefully shutdowned,
resetted and started again (see openstack.common.service code).
As graceful shutdown implies calling service.stop() and then
service.wait() we don't want to clean self._servers list because
it would be impossible to wait for them to stop processing
requests and cleaning up their resources.
Otherwise, this would lead to problems with rpc after starting
the rpc server again.
* Create a duplicate socket each time WorkerService starts
When api worker is stopped it kills the eventlet wsgi server
which internally closes the wsgi server socket object. This server
socket object becomes not usable which leads to "Bad file
descriptor" errors on service restart.
Added functional and unit tests.