openstack/searchlight
Code Issues Proposed changes

Display ES indices

Browse Source 
To allow admins to better control their
Elasticsearch usage, we want to display all
aliases that are currently being used by
Searchlight. This will allow the observant admin
to detect any orphaned indices within
Elasticsearch and deal with them appropriately.

Change-Id: I990f442ea079985615306917bc8a1d59900bbd20
Implements: blueprint Orphaned ES Aliases
This commit is contained in:
Rick Aulino 2016-07-07 10:46:05 -06:00
parent 260dc3c56a
commit 558a6c5674
4 changed files with 112 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

47
doc/source/indexingservice.rst
View File

@ -201,3 +201,50 @@ setting can be added to the Searchlight configuration file::
logging_default_format_string = %(asctime)s.%(msecs)03d %(process)d %(thread)d %(threadName)s %(levelname)s %(name)s [-] %(instance)s%(message)s
Elasticsearch Index Cleanup
===========================
In some cases, there may be orphaned Searchlight indices in Elasticsearch.
An orphaned index is one that is no longer used by Searchlight, either
directly or through an alias.
To help detect which Searchlight-related indices may be orphaned in
Elasticsearch, the ``searchlight-manage`` command will display all indices
that are currently being used by Searchlight. This is the ``aliases``
option to the ``index`` command::
$ searchlight-manage index aliases
This command outputs a listing of all indices that are used by
Searchlight (based on the current configuration file). The aliases
associated with each index is also shown. A sample output will look
like this::
$ searchlight-manage index aliases
List of Elasticsearch indices (and their associated aliases) used by Searchlight.
Note:
The indices are based on the current config file.
To view indices used by other Searchlight config files, use the --config-file option.
Indices are denoted with a '*'
Aliases are denoted with a '+'
* searchlight-2016_07_13_17_09_27
+ searchlight-listener
+ searchlight-search
* sl-swift-2016_07_13_17_09_26
+ sl-swift-listener
+ sl-swift-search
The example shows that Searchlight is using two indices in Elasticsearch:
``searchlight-2016_07_13_17_09_27`` and ``sl-swift-2016_07_13_17_09_26``.
The index ``searchlight-2016_07_13_17_09_27`` has two aliases: ``searchlight-listener``
and ``searchlight-search``. The index ``sl-swift-2016_07_13_17_09_26`` has
two aliases: ``sl-swift-listener`` and ``sl-swift-search``.
Any other indices or aliases in Elasticsearch are not used by this specific
Searchlight configuration. NOTE: If there are other Searchlight
instances running with a different configuration, their indices and aliases
will not by displayed by this command. The user will need to rerun the
``index aliases`` command using these other configuration files.

10
releasenotes/notes/bp-orphaned-es-aliases-24c5b5dcd2eb3a41.yaml Normal file
View File

@ -0,0 +1,10 @@
---
prelude: >
Add support to searchlight-manage for displaying
with Elasticsearch indices/aliases are currently
being used by Searchlight.
features:
- Add support to searchlight-manage for displaying
with Elasticsearch indices/aliases are currently
being used by Searchlight. The indices/aliases
are based on the specified configuration file.

49
searchlight/cmd/manage.py
View File

@ -24,13 +24,13 @@ from oslo_config import cfg
from oslo_log import log as logging
from oslo_utils import encodeutils
from elasticsearch import exceptions as es_exc
from keystoneclient import exceptions
from searchlight.common import config
from searchlight.common import utils
from searchlight.elasticsearch.plugins import utils as es_utils
from searchlight.i18n import _, _LE, _LI, _LW
CONF = cfg.CONF
LOG = logging.getLogger(__name__)
@ -416,6 +416,51 @@ class IndexCommands(object):
except Exception as e:
LOG.error(encodeutils.exception_to_unicode(e))
def aliases(self):
# Grab a list of aliases used by Searchlight.
aliases = []
for res_type, ext in six.iteritems(utils.get_search_plugins()):
aliases.append(ext.obj.alias_name_listener)
aliases.append(ext.obj.alias_name_search)
# Grab the indices associated with the aliases. The end result is
# a dictionary where the key is the index and the value is a list
# of aliases associated with that index.
indices = {}
for alias in set(aliases):
try:
response = es_utils.get_indices(alias)
except es_exc.NotFoundError:
# Ignore and continue.
response = {}
except Exception as e:
# Probably an ES connection issue. Alert the user.
LOG.error(_LE("Failed retrieving indices from Elasticsearch "
"%(a)s %(e)s") % {'a': alias, 'e': e})
sys.exit(3)
for index in response.keys():
if index not in indices:
indices[index] = [alias]
else:
indices[index].append(alias)
if not indices:
print("\nNo Elasticsearch indices for Searchlight exist.")
else:
print("\nList of Elasticsearch indices (and their associated"
" aliases) used by Searchlight.\n")
print("The indices are based on the config file.")
print("To view indices used by other Searchlight config "
"files, use the --config-file option.\n")
print("Indices are denoted with a '*'")
print("Aliases are denoted with a '+'\n")
for index in indices:
print(" * " + index)
for alias in indices[index]:
print(" + " + alias)
print("\n")
def add_command_parsers(subparsers):
"""Adds any commands and subparsers for their actions. This code's
@ -456,7 +501,7 @@ command_opt = cfg.SubCommandOpt('command',
COMMANDS = {
'index': IndexCommands
'index': IndexCommands,
}

8
searchlight/elasticsearch/plugins/utils.py
View File

@ -392,6 +392,14 @@ def alias_error_cleanup(indexes):
LOG.error(encodeutils.exception_to_unicode(e))
def get_indices(alias):
"""Return a list of indices associated with the specified alias.
"""
es_engine = searchlight.elasticsearch.get_api()
return es_engine.indices.get_alias(name=alias)
def normalize_date_fields(document,
created_at='created',
updated_at='updated'):