glance-specs/specs/kilo/catalog-index-service.rst

Catalog Index Service

https://blueprints.launchpad.net/glance/+spec/catalog-index-service

This is intended to improve performance of Glance API services while dramatically improving search capabilities.

It will improve performance by offloading user search queries from existing API servers. In addition, we are working on numerous improvements in Horizon which will include improvements to image, snapshot, artifact details and searching. The desired user experience is greatly dependent upon a rich, dynamic, near real time faceted and aggregated search capability with a strong query language.

This will initially be considered "Experimental API". The API is intended to be as close to final as possible, but we reserve the right to completely abandon this or change it making it backwards in-compatible.

Problem description

Glance has metadata for all images and provides listing of them. If you want to search for images based on criteria, the API is limited and somewhat inflexible. Search currently is limited to union ("AND" operations) searching of certain hard coded attributes. There is no support for intersection ( "OR" operations).

Searching does not include searching descriptions and has limited support for property based searches. Full text searching of descriptions is typically slow with a traditional relational database. Adding full text indexing to the database can degrade overall performance of the database (affects inserts).

With the addition of metadata definitions in Juno release, the new search API should allow the users to specify the search criteria using the tags and property type definition in addition to adhoc string search. They should ideally get auto-completion of possible properties and property values with near real time response.

Artifact definitions will support storing dynamic properties based on the artifact type. However, proper indexing and search is not possible with a traditional RDBMS. You end up with full table scans across tables that are not applicable for a given query. In addition, writing and understanding the query engine is difficult to understand and maintain, especially after the original authors move on.

Adding new properties with constraints for highly performant searching is difficult to achieve in a relational database.

A search engine needs to be easily customizable so that the data being collected can be changed dynamically and the way it is indexed and searched can be easily modified without requiring source data migration. (e.g. a new type of object to group namespaces together called "category")

Typical search interfaces should also provide facilities like auto-completion and search suggestions with near real time performance. This is not possible with a traditional database.

Adding more attributes to a search query over time should be easy to extend and maintain without exponentially increasing the complexity of the search logic.

User should be able to combine the search query across resources (images, metadata, artifacts) easily.

A search engine should not put additional load on the normal functions of the primary service and should easily accommodate more users by distributing the load on separate server instances. For example, a search query from Horizon should not impact Nova.

Proposed change

We are proposing a new catalog search service for Glance that will improve performance of Glance API services while dramatically improving search capabilities. The following subsections detail the concepts.

Block Diagram:

https://wiki.openstack.org/w/images/7/74/Index-service-block-diagram.png

The service will be based on Elasticsearch. Elasticsearch is a search server based on Lucene. It provides a distributed, scalable, near real-time, faceted, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. Elasticsearch is developed and released as open source under the terms of the Apache License. Notable users of Elasticsearch include Wikimedia, StumbleUpon, Mozilla, Quora, Foursquare, Etsy, SoundCloud, GitHub, FDA, CERN, and Stack Exchange. (Source: http://en.wikipedia.org/wiki/Elasticsearch)

The elastic-recheck project also uses Elasticsearch (and kibana) to classify and track OpenStack gate failures. (Source: http://status.openstack.org/elastic-recheck)

Indexing

Index

This will serve as the cache for all search requests. It will be backed by Elasticsearch.

Index Loaders

Index loaders define the data mappings for indexing and load the data from the source. They are called during initialization of service and on-demand later when required to index everything. They ensure that all appropriate RBAC information is included in the index to facilitate appropriate authorization on search responses.

The index loaders will attempt to maintain native API format as best as possible with as much direct pass through as possible so that data manipulation and maintenance is kept to a minimum.

An example Glance Image data mapping would be:

{
  'dynamic': True,
  'properties': {
      'id': {'type': 'string'},
      'name': {'type': 'string'},
      'description': {'type': 'string'},
      'tags': {'type': 'string'},
      'disk_format': {'type': 'string'},
      'container_format': {'type': 'string'},
      'size': {'type': 'long'},
      'virtual_size': {'type': 'long'},
      'status': {'type': 'string'},
      'visibility': {'type': 'string'},
      'checksum': {'type': 'string'},
      'min_disk': {'type': 'long'},
      'min_ram': {'type': 'long'},
      'owner': {'type': 'string'},
      'protected': {'type': 'boolean'},
  },
}

Index Updates

Once the index is initialized it needs to be constantly updated to keep it in sync with the data source. Update clients will listen for notifications from data sources to re-index the data for specific resources (e.g. an image or artifact).

For glance, it would listen on message Topic for notifications like (image.create, image.update, etc) and reindex for the effected image metadata. (More info at http://docs.openstack.org/developer/glance/notifications.html)

Index Management API

Allows for CRUD management of loading, updating and deleting data in the index. Indexing is allowed only for admin users.

Default policy.json will be:

{
  "catalog_index": "role:admin",
  "catalog_search": ""
}

Searching

The search API allows users to execute a search query and get back search hits that match the query. The query can either be provided using a simple query string as a parameter, or using a request body.

Note

Search query is not parsed and passed "as-is" to elastic search engine except for adding filters. Response from search engine could be filtered based on the plugin implementation of document type.

All search APIs can be applied across multiple types within an index, and across multiple indices with support for the multi index syntax.

This will allow for search phrase completion as well as search suggestions( such as handling misspellings)

The search will have two levels of RBAC.

1. API level policy checks using policy.json files. This will allow coarse grained RBAC support for simple deny / allow on API usage.

2. RBAC query filters. These will be defined in conjunction with index loaders. When a request comes in, the type(s) of resource(s) being requested will map to an RBAC query filter.

The RBAC query filter will add any appropriate filters to the request being sent into the elastic search service, such that only specific results that the user is allowed to view will be returned.

For example, the image index loader will include indexing owner information and visibility information. The RBAC filter will examine the incoming request and adds filters to the request so that the results don't include non-shared / non-public images from a different project than the user making the request.

Property protected fields will be read from the config file and will be added as "source filtering" field(s) in elasticsearch query which will keep/remove the protected fields from the search output based on the authorization of the user.

Alternatives

Searching data could also be achieved by writing SQL queries on the Glance database but there are several factors which do not make it an ideal solution:

• Joins across multiple tables in real time will make the response time very slow
• Full text searching of descriptions is typically slow with a traditional relational database. Adding full text indexing to the database can degrade overall performance of the database
• Property types can be added dynamically using metadefs and proper indexing in relational databases is not possible
• Search queries will be running against the same database used by Glance core functions and inadvertently effecting their response time.
• Adding more attributes to search query over time should be easy to extend and maintain without exponentially increasing the complexity of the search logic

User should be able to combine the search query across resources (images, artifacts) etc. and the search engine should not be tightly integrated with any specific module.

Another alternative would be for clients to load the entire data set and search within the client. This means every user gets all the data every time the user loads the page and has to keep it in sync with server side data. This is increases the load and burden on the core OpenStack service providing the data and is slower since the client has to load the entire dataset across the network. In addition, the client has to recreate the logic for things like search suggestions and complex queries with AND / OR logic.

It should be noted that NONE of these options also include an ability to do things like get search request scoring of results returned with a configurable threshold for results (something elastic search provides).

Data model impact

The data being indexed will be stored outside the Glance SQL database and therefore we don't expect any data model changes in Glance.

REST API impact

Common Response Codes

• Create Success: 201 Created
• Modify Success: 200 OK
• Delete Success: 204 No Content
• Failure: 400 Bad Request with details.
• Forbidden: 403 Forbidden
• Not found: 404 Not found e.g. if specific entity not found
• Method Not Allowed: 405 Not allowed e.g. if trying to delete on a list resource
• Not Implemented: 501 Not Implemented e.g. HEAD not implemented

This is an experimental API

API Version

Search images supports both GET and POST. Elasticsearch supports GET with query params but its a limited subset of query DSL. GET is implemented here with a request body to make use of all the available query options

Please refer to the following URI for the Query DSL http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html

Search images(GET):

GET /v2/search

Example Request Body:

{
  "index": ["glance"],
  "type": ["image"],
  "query": {
      "query_string": {
          "query": "cirros"
      }
  }
}

Example Response Body:

{
  "took": 5,
  "timed_out": false,
  "_shards": {
      "total": 10,
      "successful": 10,
      "failed": 0
  },
  "hits": {
      "total": 3,
      "max_score": 0.40409642,
      "hits": [
          {
              "_index": "search",
              "_type": "image",
              "_id": "75fbdd4c-3e5b-4552-8950-9bb5262babcd",
              "_score": 0.40409642,
              "_source": {
                  "status": "active",
                  "virtual_size": null,
                  "name": "cirros-0.3.2-x86_64-uec-ramdisk",
                  "property": [],
                  "container_format": "ari",
                  "min_ram": 0,
                  "disk_format": "ari",
                  "properties": [],
                  "owner": "f72690e85b2a4ff095f50b7fad99429a",
                  "protected": false,
                  "checksum": "68085af2609d03e51c7662395b5b6e4b",
                  "min_disk": 0,
                  "is_public": true,
                  "size": 3723817,
                  "id": "75fbdd4c-3e5b-4552-8950-9bb5262babcd",
                  "description": ""
              }
          },
          {
              "_index": "search",
              "_type": "image",
              "_id": "95467ea8-dd34-4bdd-8a6a-f52e47ee9bce",
              "_score": 0.23091224,
              "_source": {
                  "status": "active",
                  "virtual_size": null,
                  "name": "cirros-0.3.2-x86_64-uec",
                  "property": [
                      "kernel_id_d00ea383-a1fa-48d3-b56c-880093730b53",
                      "ramdisk_id_75fbdd4c-3e5b-4552-8950-9bb5262babcd",
                      "hypervisor_type_uml",
                      "hw_watchdog_action_poweroff"
                  ],
                  "container_format": "ami",
                  "min_ram": 0,
                  "disk_format": "ami",
                  "properties": [
                     {
                          "name": "kernel_id",
                          "value": "d00ea383-a1fa-48d3-b56c-880093730b53"
                      },
                      {
                          "name": "ramdisk_id",
                          "value": "75fbdd4c-3e5b-4552-8950-9bb5262babcd"
                      },
                      {
                          "name": "hypervisor_type",
                          "value": "uml"
                      },
                      {
                          "name": "hw_watchdog_action",
                          "value": "poweroff"
                      }
                  ],
                  "owner": "f72690e85b2a4ff095f50b7fad99429a",
                  "protected": false,
                  "checksum": "4eada48c2843d2a262c814ddc92ecf2c",
                  "min_disk": 0,
                  "is_public": true,
                  "size": 25165824,
                  "id": "95467ea8-dd34-4bdd-8a6a-f52e47ee9bce",
                  "description": ""
              }
          },
          {
              "_index": "search",
              "_type": "image",
              "_id": "d00ea383-a1fa-48d3-b56c-880093730b53",
              "_score": 0.067124054,
              "_source": {
                  "status": "active",
                  "virtual_size": null,
                  "name": "cirros-0.3.2-x86_64-uec-kernel",
                  "property": [],
                  "container_format": "aki",
                  "min_ram": 0,
                  "disk_format": "aki",
                  "properties": [],
                  "owner": "f72690e85b2a4ff095f50b7fad99429a",
                  "protected": false,
                  "checksum": "836c69cbcd1dc4f225daedbab6edc7c7",
                  "min_disk": 0,
                  "is_public": true,
                  "size": 4969360,
                  "id": "d00ea383-a1fa-48d3-b56c-880093730b53",
                  "description": ""
              }
          }
      ]
  }
}

Search images(POST):

POST /v2/search

Please refer to the following URI for the Query DSL http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html

Example Request Body:

{
  "index": ["glance"],
  "type": ["image"],
  "query": {
      "query_string": {
          "query": "cirros"
      }
  }
}

Example Response Body:

{
  "took": 5,
  "timed_out": false,
  "_shards": {
      "total": 10,
      "successful": 10,
      "failed": 0
  },
  "hits": {
      "total": 3,
      "max_score": 0.40409642,
      "hits": [
          {
              "_index": "search",
              "_type": "image",
              "_id": "75fbdd4c-3e5b-4552-8950-9bb5262babcd",
              "_score": 0.40409642,
              "_source": {
                  "status": "active",
                  "virtual_size": null,
                  "name": "cirros-0.3.2-x86_64-uec-ramdisk",
                  "property": [],
                  "container_format": "ari",
                  "min_ram": 0,
                  "disk_format": "ari",
                  "properties": [],
                  "owner": "f72690e85b2a4ff095f50b7fad99429a",
                  "protected": false,
                  "checksum": "68085af2609d03e51c7662395b5b6e4b",
                  "min_disk": 0,
                  "is_public": true,
                  "size": 3723817,
                  "id": "75fbdd4c-3e5b-4552-8950-9bb5262babcd",
                  "description": ""
              }
          },
          {
              "_index": "search",
              "_type": "image",
              "_id": "95467ea8-dd34-4bdd-8a6a-f52e47ee9bce",
              "_score": 0.23091224,
              "_source": {
                  "status": "active",
                  "virtual_size": null,
                  "name": "cirros-0.3.2-x86_64-uec",
                  "property": [
                      "kernel_id_d00ea383-a1fa-48d3-b56c-880093730b53",
                      "ramdisk_id_75fbdd4c-3e5b-4552-8950-9bb5262babcd",
                      "hypervisor_type_uml",
                      "hw_watchdog_action_poweroff"
                  ],
                  "container_format": "ami",
                  "min_ram": 0,
                  "disk_format": "ami",
                  "properties": [
                     {
                          "name": "kernel_id",
                          "value": "d00ea383-a1fa-48d3-b56c-880093730b53"
                      },
                      {
                          "name": "ramdisk_id",
                          "value": "75fbdd4c-3e5b-4552-8950-9bb5262babcd"
                      },
                      {
                          "name": "hypervisor_type",
                          "value": "uml"
                      },
                      {
                          "name": "hw_watchdog_action",
                          "value": "poweroff"
                      }
                  ],
                  "owner": "f72690e85b2a4ff095f50b7fad99429a",
                  "protected": false,
                  "checksum": "4eada48c2843d2a262c814ddc92ecf2c",
                  "min_disk": 0,
                  "is_public": true,
                  "size": 25165824,
                  "id": "95467ea8-dd34-4bdd-8a6a-f52e47ee9bce",
                  "description": ""
              }
          },
          {
              "_index": "search",
              "_type": "image",
              "_id": "d00ea383-a1fa-48d3-b56c-880093730b53",
              "_score": 0.067124054,
              "_source": {
                  "status": "active",
                  "virtual_size": null,
                  "name": "cirros-0.3.2-x86_64-uec-kernel",
                  "property": [],
                  "container_format": "aki",
                  "min_ram": 0,
                  "disk_format": "aki",
                  "properties": [],
                  "owner": "f72690e85b2a4ff095f50b7fad99429a",
                  "protected": false,
                  "checksum": "836c69cbcd1dc4f225daedbab6edc7c7",
                  "min_disk": 0,
                  "is_public": true,
                  "size": 4969360,
                  "id": "d00ea383-a1fa-48d3-b56c-880093730b53",
                  "description": ""
              }
          }
      ]
  }
}

Index images: index, create, update and delete data:

POST /v2/index

Indexing is allowed only for admin users. Supported actions are index, create, update and delete

Example Request Body:

{
  "default_index": "search",
  "default_type": "image",
  "actions": [
      {
          "action": "create",
          "index": "search",
          "type": "image",
          "id": "d00ea383-a1fa-48d3-b56c-880093730b54",
          "data": {
              "status": "active",
              "virtual_size": null,
              "name": "cirros-0.3.3-x86_64-uec-kernel",
              "property": [],
              "container_format": "aki",
              "min_ram": 0,
              "disk_format": "aki",
              "properties": [],
              "owner": "f72690e85b2a4ff095f50b7fad99429a",
              "protected": false,
              "checksum": "836c69cbcd1dc4f225daedbab6edc7c7",
              "min_disk": 0,
              "is_public": false,
              "size": 4969360,
              "id": "d00ea383-a1fa-48d3-b56c-880093730b54",
              "description": ""
          }
      },
      {
          "action": "update",
          "index": "search",
          "type": "image",
          "id": "75fbdd4c-3e5b-4552-8950-9bb5262babcd",
          "data": {
              "name": "cirros x86",
              "status": "inactive"
          }
      },
      {
          "action": "delete",
          "index": "search",
          "type": "image",
          "id": "95467ea8-dd34-4bdd-8a6a-f52e47ee9bce"
      }
  ]
}

Security impact

None to existing Glance API. Search queries will apply filters to return data that the user is authorized to see. See description.

Notifications impact

None to existing notifications. Will only consume notifications Need to add metadef notifications to Glance service.

Other end user impact

Update python-glanceclient as needed

Performance Impact

No changes to existing API or code Data from Glance DB will read once during initialization to index it inside search engine.

This is intended to improve performance of Glance API services while dramatically improving search capabilities. It will improve performance by offloading user search queries from existing API servers.

Other deployer impact

Glance Catalog Index service will be installed as a separate service with its own port and endpoint.

This will initially be considered "Experimental API". The API is intended to be as close to final as possible, but we reserve the right to completely abandon this or change it making it backwards in-compatible.

glance-manage will have new commands for indexing image, metadef, and artifact data

The deployment will be targeted as a single region service. In future if required an "Aggregate search" of all regions which can search across all the regions could be provided.

Developer impact

These are new API's and will not impact any existing API's.

Implementation

Assignee(s)

Primary assignee:

lakshmi-sampath, kamil-rykowski,

Other contributors:

wayne-okuma, travis-tripp

Reviewers

Core reviewer(s):

nikhil-komawar zhiyan

Other reviewer(s):

icordasc

Work Items

• Installation of Elastic Search in Glance environment (single node)

• Index Dictionary data in ElasticSearch

  • Write a tool to index all metadata objects (namespaces objects, properties) from database into elasticsearch
  • Write a tool to index all images from database into elasticsearch
  • Merge used properties from Glance(optional)
  • Listen to notifications/events form Glance on Image CRUD(optional) for continuous indexing of new/old data

• Create Glance Search API - Interface to backend ElasticSearch

  • Make Policy checks on requests
  • Filter request based on RBAC with user token

• Search Images

  • List all the results by given search query string

• Create Glance Index API

  • Policy checks

• Discuss with Openstack/Infra

  • Test environment for elasticsearch

• Devstack integration of single node elastic search.

• Metadef notifications

  • Generate and Listen to metadef notifications

• Calls the tools (loaders)
• Documentation update
• Update glance client
• Update glance manage

Dependencies

• Depends on elasticsearch for search engine

Testing

Unit tests will be added for all possible code with a goal of being able to isolate functionality as much as possible.

Tempest tests will be added wherever possible.

Documentation Impact

Docs needed for new service and usage.

All document changes will indicate this as "Experimental API"

References

• Elasticsearch Query DSL http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html