openstack
/
searchlight
Code Issues Proposed changes

Use Notification Pools by Default 

Using notifications pools is now the recommended way
to deploy searchlight, so this patch updates Searchlight
to use notification pools by default and to use
the same notification topic as ceilometer.

Change-Id: I2556128ae7ba2b05a0f1a997676083a394f8c388
See: Ie40876e88059d0326efa0b4abaa76b8b64f0b86b
Partial-Bug: #1583215
This commit is contained in:
Travis Tripp 2016-05-23 09:40:25 -06:00
parent 6798c5df60
commit 787752a248
13 changed files with 157 additions and 75 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
devstack/README.rst
View File

@ -60,8 +60,8 @@ Existing local.conf customization
The search service is driven using a plugin mechanism for integrating to other
services. Each integrated service may need to be specifically enabled
in devstack and may require additional configuration settings to work with
searchlight. For example, typically, you will need to add the
``searchlight_indexer`` notification topic to each service's configuration.
searchlight. For example, you typically will need to set the notifications
driver in each service's configuration.
Please review the plugin documentation and add configuration appropriately:

37
devstack/local.conf
View File

@ -144,16 +144,10 @@ enable_plugin searchlight-ui https://git.openstack.org/openstack/searchlight-ui
[[post-config|$NOVA_CONF]]
[DEFAULT]
#novncproxy_base_url = http://172.18.200.23:6127/vnc_auto.html
notification_topics = notifications, searchlight_indexer
notification_driver = messaging
rpc_backend = 'rabbit'
notify_on_state_change=vm_and_task_state
[[post-config|$NEUTRON_CONF]]
[DEFAULT]
notification_topics = notifications, searchlight_indexer
notification_driver = messaging
rpc_backend = 'rabbit'
[oslo_messaging_notifications]
driver = messaging
# decrease number of Heat engine workers, when too much for devstack
[[post-config|$HEAT_CONF]]
@ -161,25 +155,26 @@ rpc_backend = 'rabbit'
num_engine_workers = 2
[[post-config|$GLANCE_API_CONF]]
[DEFAULT]
default_store=file
notification_topics = notifications, searchlight_indexer
[[post-config|$DESIGNATE_CONF]]
[DEFAULT]
notification_driver = messaging
notification_topics = notifications, searchlight_indexer
rpc_backend = 'rabbit'
[oslo_messaging_notifications]
driver = messaging
[[post-config|$CINDER_CONF]]
[DEFAULT]
notification_driver = messaging
notification_topics = notifications, searchlight_indexer
rpc_backend = 'rabbit'
[oslo_messaging_notifications]
driver = messaging
[[post-config|$DESIGNATE_CONF]]
[oslo_messaging_notifications]
driver = messaging
### Searchlight Configuration ###
[[post-config|$SEARCHLIGHT_CONF]]
[listener]
#notifications_pool = searchlight-listener
[resource_plugin]
#notifications_topic = notifications
#resource_group_name = searchlight
[resource_plugin:os_nova_server]

33
doc/source/dev-environment.rst
View File

@ -194,19 +194,15 @@ Plugin Configuration
The search service is driven using a plugin mechanism for integrating to other
services. Each integrated service may require additional configuration
settings. For example, typically, you will need to add the
``searchlight_indexer`` notification topic to each service's configuration.
settings. For example, you typically will need to enable the
``oslo_messaging_notifications`` messaging driver and may need to add
the ``notifications`` topic to each service's configuration.
.. note:: In Newton, notification messaging pools will become the default
recommended configuration, which does not require changing any
service configurations beyond enabling notifications. To enable
the use of a messaging pool instead of separate topics, add the
``notifications_pool`` option in the ``listener`` section of
``searchlight.conf``. In this case, ensure that services are using
the same notification topic as searchlight.
See :ref:`searchlight-plugins` for plugin installation and general
configuration information.
Searchlight uses notification messaging pools. This usually does not
require changing any service configurations beyond enabling the notifications
driver. Searchlight uses sensible defaults for most deployments, but if you
want to customize the settings see :ref:`searchlight-plugins` for plugin
installation and general configuration information.
See each plugin below for detailed information about specific plugins:
@ -389,8 +385,11 @@ Use rabbitmqctl to examine unconsumed notifications::
sudo rabbitmqctl list_queues | grep info
Currently notifications from other services may be consumed by other
listeners and searchlight will not receive the notifications. This is
being investigated. This does mean that you can not currently use listener
based updates in an environment that has ceilometer deployed. You will need
to periodically perform a few full re-index operation.
There are also a number of management tools available to help with
troubleshooting.
Please see: https://www.rabbitmq.com/management.html
If you have searchlight setup to share a notification topic, but do
not have a notification pool configured, then notifications may be consumed
by other listeners and searchlight will not receive the notifications.

45
doc/source/plugins.rst
View File

@ -69,6 +69,9 @@ The following shows a sampling of various configuration options in
or default configuration values. They are intended for exemplary purposes only.
Please read the rest of the guide for detailed information.::
[listener]
notifications_pool = searchlight
[resource_plugin]
resource_group_name = searchlight
@ -127,10 +130,10 @@ configuration group of ``[resource_plugin]`` in ``searchlight.conf`` and
optionally overridden in a specific plugin's configuration. For example::
[resource_plugin]
notifications_topic = searchlight_indexer
resource_group_name = searchlight
[resource_plugin:os_nova_server]
notifications_topic = searchlight_indexer_nova
resource_group_name = searchlight-nova-servers
**Non-Inheritable** common configuration options are honored by all plugins,
but must be specified directly in that plugin's configuration group. They
@ -140,6 +143,28 @@ example::
[resource_plugin:os_glance_image]
enabled = false
.. _plugin_notifications:
Notifcations
............
There are two ways to configure services to send notifications that
Searchlight can receive. The recommended method is to configure
Searchlight to use the notification topic that each service is already
configured to use and then to allow Searchlight to consume messages from
that topic using a pool, touched on in the `messaging documentation`_.
Searchlight uses this configuration by default.
.. _`messaging documentation`: http://docs.openstack.org/developer/oslo.messaging/notification_listener.html
**Topics**
Searchlight defaults to using the oslo notification topic of
``notifications``. This is the oslo default topic which most services also
use to broadcast their notifications. You will need to change the topic in both
``searchlight.conf`` and the various service configuration files if you want
to modify the topic used by Searchlight. Each plugin can use a different topic.
Notification topics are a special case. It is possible to override
the notification ``topic`` as a shared setting; it is also possible to
override ``<topic>,<exchange>`` pairs per-plugin in the case where some
@ -156,6 +181,18 @@ example) neutron is using a separate notification topic::
If you override one service topic, you must provide topic,exchange pairs
for all service notifications a plugin supports.
**Pools**
In addition, Searchlight uses a notification pool. This allows Searchlight
to listen on the same topic to which other services are listening while
ensuring that Searchlight still gets its own copy of each notification. The
default notification pool is set to ``searchlight``. This is set using the
``notifications_pool`` setting in the ``[listener]`` configuration group.
Example::
[listener]
notification_pools = searchlight
See :ref:`individual-plugin-configuration` for more information and examples
on individual plugin configuration.
@ -188,8 +225,8 @@ Inheritable Common Configuration Options
| | | has some advantages, particularly | |
| | | around memory usage. | |
+---------------------+---------------+-------------------------------------+---------------------------+
| notifications_topic | searchlight\_ | The oslo.messaging topic on which | | Restart listener |
| | indexer | services send notifications. Each | |
| notifications_topic | notifications | The oslo.messaging topic on which | Restart listener |
| | | services send notifications. Each | |
| | | plugin defines a list of exchanges | |
| | | to which it will subscribe. | |
+---------------------+---------------+-------------------------------------+---------------------------+

14
doc/source/plugins/cinder.rst
View File

@ -67,14 +67,16 @@ cinder.conf
-----------
Notifications must be configured properly for searchlight to process
incremental updates. Use the following::
incremental updates. Enable notifications using the following::
notification_driver = messaging
notification_topics = searchlight_indexer
[oslo_messaging_notifications]
driver = messaging
.. note::
Restart the Cinder api service (c-api) after making changes.
See :ref:`plugin_notifications` for more information on
notification topics.
local.conf (devstack)
---------------------
@ -93,6 +95,12 @@ Release Notes
0.2.0.0 (Mitaka)
----------------
Notifications must be configured properly for searchlight to process
incremental updates. Searchlight must use its own topic. Use the following::
notification_driver = messaging
notification_topics = searchlight_indexer
The following fields are exposed to adminstrators only for cinder volumes:
* os-vol-mig-status-attr:*
* os-vol-host-attr:*

21
doc/source/plugins/designate.rst
View File

@ -71,14 +71,17 @@ designate.conf
--------------
Notifications must be configured properly for searchlight to process
incremental updates. Use the following::
incremental updates. Enable notifications using the following::
notification_driver = messaging
notification_topics = notifications,searchlight_indexer
rpc_backend = 'rabbit'
[oslo_messaging_notifications]
driver = messaging
Restart designate-central, designate-pool-manager, designate-zone-manager and
you should be good to go!
.. note::
Restart ``designate-central``, ``designate-pool-manager``, and
``designate-zone-manager`` after making changes.
See :ref:`plugin_notifications` for more information on
notification topics.
local.conf (devstack)
---------------------
@ -114,6 +117,12 @@ Release Notes
0.2.0.0 (Mitaka)
----------------
Notifications must be configured properly for searchlight to process
incremental updates. Searchlight must use its own topic. Use the following::
notification_driver = messaging
notification_topics = searchlight_indexer
The Designate notification limitations mentioned in Liberty still apply.
You no longer need to use the --no-delete option mentioned below. Zero

20
doc/source/plugins/glance.rst
View File

@ -81,15 +81,16 @@ glance-api.conf
---------------
Notifications must be configured properly for searchlight to process
incremental updates. Use the following::
incremental updates. Enable notifications using the following::
notification_driver = messaging
notification_topics = notifications, searchlight_indexer
rpc_backend = 'rabbit'
[oslo_messaging_notifications]
driver = messaging
.. note::
Restart glance API (g-api) after making changes.
Restart the Glance api service (g-api) after making changes.
See :ref:`plugin_notifications` for more information on
notification topics.
local.conf (devstack)
---------------------
@ -105,6 +106,15 @@ underneath it.::
Release Notes
=============
0.2.0.0 (Mitaka)
----------------
Notifications must be configured properly for searchlight to process
incremental updates. Searchlight must use its own topic. Use the following::
notification_driver = messaging
notification_topics = searchlight_indexer
0.1.0.0 (Liberty)
-----------------

16
doc/source/plugins/neutron.rst
View File

@ -82,14 +82,16 @@ neutron.conf
------------
Notifications must be configured properly for searchlight to process
incremental updates. Use the following::
incremental updates. Enable notifications using the following::
notification_driver = messaging
notification_topics = searchlight_indexer
[oslo_messaging_notifications]
driver = messaging
.. note::
Restart the Neutron api service (q-svc) after making changes.
See :ref:`plugin_notifications` for more information on
notification topics.
local.conf (devstack)
---------------------
@ -118,7 +120,13 @@ Release Notes
=============
0.2.0.0 (Mitaka)
-----------------
----------------
Notifications must be configured properly for searchlight to process
incremental updates. Searchlight must use its own topic. Use the following::
notification_driver = messaging
notification_topics = searchlight_indexer
DHCP ports are *not* indexed. Neutron doesn't provide a reliable way for
Searchlight to index these ports since they are created and modified

26
doc/source/plugins/nova.rst
View File

@ -69,16 +69,17 @@ nova.conf
---------
Notifications must be configured properly for searchlight to process
incremental updates. Use the following::
incremental updates. Enable notifications using the following::
notification_driver = messaging
notification_topics = notifications, searchlight_indexer
rpc_backend = 'rabbit'
[oslo_messaging_notifications]
driver = messaging
notify_on_state_change=vm_and_task_state
.. note::
Restart Nova API and Nova scheduler (n-api, n-sch) after making changes.
See :ref:`plugin_notification_topics` for more information on
notification topics.
local.conf (devstack)
---------------------
@ -102,15 +103,16 @@ neutron.conf
------------
Notifications must be configured properly for searchlight to process
incremental updates. Use the following::
incremental updates. Enable notifications using the following::
notification_driver = messaging
notification_topics = searchlight_indexer
rpc_backend = 'rabbit'
[oslo_messaging_notifications]
driver = messaging
.. note::
Restart the Neutron service (q-svc) after making changes.
Restart the Neutron api service (q-svc) after making changes.
See :ref:`plugin_notifications` for more information on
notification topics.
local.conf (devstack)
---------------------
@ -129,6 +131,12 @@ Release Notes
0.2.0.0 (Mitaka)
----------------
Notifications must be configured properly for searchlight to process
incremental updates. Searchlight must use its own topic. Use the following::
notification_driver = messaging
notification_topics = searchlight_indexer
The following fields are exposed to adminstrators only for nova instances:
* OS-EXT-SRV-ATTR:*

11
doc/source/plugins/swift.rst
View File

@ -73,17 +73,18 @@ via middleware is provided in the following patch:
#Replace <user>,<password>,<rabbitip> and <rabbitport> for your environment values
transport_url = rabbit://<user>:<password>@<rabbitip>:<rabbitport>/
notification_driver = messaging
notification_topics = searchlight_indexer
notification_topics = notifications
# Add oslomiddleware to pipeline:main
# see example below.
[pipeline:main]
pipeline = catch_errors gatekeeper ...<other>... oslomiddleware proxy-logging proxy-server
.. note::
Restart the swift proxy API service (s-proxy) after making changes.
Starting in Newton, Searchlight can share the same notification topic as
other services, because it uses a messaging pool.
Searchlight Configuration
=========================
@ -161,6 +162,12 @@ Release Notes
0.2.0.0 (Mitaka)
----------------
Notifications must be configured properly for searchlight to process
incremental updates. Searchlight must use its own topic. Use the following::
notification_driver = messaging
notification_topics = searchlight_indexer
Large scale swift cluster support is targeted at a future release, but
we encourage trial deployments to help us address issues as soon as possible.

2
searchlight/elasticsearch/plugins/base.py
View File

@ -39,7 +39,7 @@ _ = i18n._
indexer_opts = [
cfg.StrOpt('resource_group_name', default="searchlight",
help="The default base name for accessing Elasticsearch"),
cfg.StrOpt('notifications_topic', default="searchlight_indexer",
cfg.StrOpt('notifications_topic', default="notifications",
help="The default messaging notifications topic"),
cfg.BoolOpt('mapping_use_doc_values', default=True,
help='Use doc_values for mapped fields where applicable.'

1
searchlight/listener.py
View File

@ -37,6 +37,7 @@ listener_opts = [
help='Number of workers for notification service. A single '
'notification agent is enabled by default.'),
cfg.StrOpt('notifications_pool',
default='searchlight-listener',
help='Use an oslo.messaging pool, which can be an alternative '
'to multiple topics. ')
]

2
test-scripts/listener.py
View File

@ -30,7 +30,7 @@ except ImportError:
import urllib.parse as urlparse
topic = 'searchlight_indexer'
topic = 'notifications'
password = os.environ.get('RABBIT_PASSWORD', os.environ.get('OS_PASSWORD'))
host = urlparse.urlparse(os.environ.get('OS_AUTH_URL')).hostname
username = os.environ.get('RABBIT_USER', 'stackrabbit')