Adopt neutron-lib plugin directory

Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.

Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.

This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.

Usage pattern [1,2] can be translated to [3,4] respectively.

[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()

The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.

Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.

Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.

NeutronLibImpact

Partially-implements: blueprint neutron-lib

Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf

neutron/service.py

@@ -17,6 +17,7 @@ import inspect
 import os
 import random
 
+from neutron_lib.plugins import directory
 from oslo_concurrency import processutils
 from oslo_config import cfg
 from oslo_log import log as logging
@@ -36,7 +37,6 @@ from neutron.common import rpc as n_rpc
 from neutron.conf import service
 from neutron import context
 from neutron.db import api as session
-from neutron import manager
 from neutron import worker as neutron_worker
 from neutron import wsgi
 
@@ -150,9 +150,8 @@ class RpcReportsWorker(RpcWorker):
 
 
 def _get_rpc_workers():
-    plugin = manager.NeutronManager.get_plugin()
-    service_plugins = (
-        manager.NeutronManager.get_service_plugins().values())
+    plugin = directory.get_plugin()
+    service_plugins = directory.get_plugins().values()
 
     if cfg.CONF.rpc_workers < 1:
         cfg.CONF.set_override('rpc_workers', 1)
@@ -185,8 +184,8 @@ def _get_rpc_workers():
 
 
 def _get_plugins_workers():
-    # NOTE(twilson) get_service_plugins also returns the core plugin
-    plugins = manager.NeutronManager.get_unique_service_plugins()
+    # NOTE(twilson) get_plugins also returns the core plugin
+    plugins = directory.get_unique_plugins()
 
     # TODO(twilson) Instead of defaulting here, come up with a good way to
     # share a common get_workers default between NeutronPluginBaseV2 and