openstack/nova
Code Issues Proposed changes

Add image caching API for aggregates

Browse Source 
This adds a new microversion and support for requesting image pre-caching
on an aggregate.

Related to blueprint image-precache-support

Change-Id: I4ab96095106b38737ed355fcad07e758f8b5a9b0
This commit is contained in:
Dan Smith 2019-10-14 08:11:27 -07:00 committed by Matt Riedemann
parent 11d909c2cb
commit 3391298706
40 changed files with 570 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

33
api-ref/source/os-aggregates.inc
View File

@ -354,3 +354,36 @@ Response
.. literalinclude:: ../../doc/api_samples/os-aggregates/v2.41/aggregates-metadata-post-resp.json
:language: javascript
Request Image Pre-caching for Aggregate
=======================================
.. rest_method:: POST /os-aggregates/{aggregate_id}/images
Requests that a set of images be pre-cached on compute nodes within the referenced aggregate.
This API is available starting with microversion 2.81.
Normal response codes: 202
Error response codes: badRequest(400), unauthorized(401), forbidden(403),
itemNotFound(404)
Request
-------
.. rest_parameters:: parameters.yaml
- aggregate_id: aggregate_id
- cache: cache
- cache.id: image_id_body
**Example Request Image pre-caching for Aggregate (v2.81): JSON request**
.. literalinclude:: ../../doc/api_samples/os-aggregates/v2.81/aggregate-images-post-req.json
:language: javascript
Response
--------
The response body is always empty.

5
api-ref/source/parameters.yaml
View File

@ -1997,6 +1997,11 @@ boot_index:
in: body
required: true
type: integer
cache:
description: A list of image objects to cache.
in: body
required: true
type: array
certificate:
description: |
The certificate object.

5
doc/api_samples/os-aggregates/v2.81/aggregate-add-host-post-req.json Normal file
View File

@ -0,0 +1,5 @@
{
"add_host": {
"host": "compute"
}
}

6
doc/api_samples/os-aggregates/v2.81/aggregate-images-post-req.json Normal file
View File

@ -0,0 +1,6 @@
{
"cache":
[
{"id": "70a599e0-31e7-49b7-b260-868f441e862b"}
]
}

9
doc/api_samples/os-aggregates/v2.81/aggregate-metadata-post-req.json Normal file
View File

@ -0,0 +1,9 @@
{
"set_metadata":
{
"metadata":
{
"key": "value"
}
}
}

7
doc/api_samples/os-aggregates/v2.81/aggregate-post-req.json Normal file
View File

@ -0,0 +1,7 @@
{
"aggregate":
{
"name": "name",
"availability_zone": "london"
}
}

12
doc/api_samples/os-aggregates/v2.81/aggregate-post-resp.json Normal file
View File

@ -0,0 +1,12 @@
{
"aggregate": {
"availability_zone": "london",
"created_at": "2019-10-08T15:15:27.988513",
"deleted": false,
"deleted_at": null,
"id": 1,
"name": "name",
"updated_at": null,
"uuid": "a25e34a2-4fc1-4876-82d0-cf930fa04b82"
}
}

5
doc/api_samples/os-aggregates/v2.81/aggregate-remove-host-post-req.json Normal file
View File

@ -0,0 +1,5 @@
{
"remove_host": {
"host": "compute"
}
}

7
doc/api_samples/os-aggregates/v2.81/aggregate-update-post-req.json Normal file
View File

@ -0,0 +1,7 @@
{
"aggregate":
{
"name": "newname",
"availability_zone": "nova2"
}
}

16
doc/api_samples/os-aggregates/v2.81/aggregate-update-post-resp.json Normal file
View File

@ -0,0 +1,16 @@
{
"aggregate": {
"availability_zone": "nova2",
"created_at": "2019-10-11T14:19:00.718841",
"deleted": false,
"deleted_at": null,
"hosts": [],
"id": 1,
"metadata": {
"availability_zone": "nova2"
},
"name": "newname",
"updated_at": "2019-10-11T14:19:00.785838",
"uuid": "4e7fa22f-f6cf-4e81-a5c7-6dc485815f81"
}
}

18
doc/api_samples/os-aggregates/v2.81/aggregates-add-host-post-resp.json Normal file
View File

@ -0,0 +1,18 @@
{
"aggregate": {
"availability_zone": "london",
"created_at": "2019-10-11T14:19:05.250053",
"deleted": false,
"deleted_at": null,
"hosts": [
"compute"
],
"id": 1,
"metadata": {
"availability_zone": "london"
},
"name": "name",
"updated_at": null,
"uuid": "47832b50-a192-4900-affe-8f7fdf2d7f22"
}
}

16
doc/api_samples/os-aggregates/v2.81/aggregates-get-resp.json Normal file
View File

@ -0,0 +1,16 @@
{
"aggregate": {
"availability_zone": "london",
"created_at": "2019-10-11T14:19:07.366577",
"deleted": false,
"deleted_at": null,
"hosts": [],
"id": 1,
"metadata": {
"availability_zone": "london"
},
"name": "name",
"updated_at": null,
"uuid": "7c5ff84a-c901-4733-adf8-06875e265080"
}
}

20
doc/api_samples/os-aggregates/v2.81/aggregates-list-get-resp.json Normal file
View File

@ -0,0 +1,20 @@
{
"aggregates": [
{
"availability_zone": "london",
"created_at": "2019-10-11T14:19:07.386637",
"deleted": false,
"deleted_at": null,
"hosts": [
"compute"
],
"id": 1,
"metadata": {
"availability_zone": "london"
},
"name": "name",
"updated_at": null,
"uuid": "070cb72c-f463-4f72-9c61-2c0556eb8c07"
}
]
}

17
doc/api_samples/os-aggregates/v2.81/aggregates-metadata-post-resp.json Normal file
View File

@ -0,0 +1,17 @@
{
"aggregate": {
"availability_zone": "london",
"created_at": "2019-10-11T14:19:03.103465",
"deleted": false,
"deleted_at": null,
"hosts": [],
"id": 1,
"metadata": {
"availability_zone": "london",
"key": "value"
},
"name": "name",
"updated_at": "2019-10-11T14:19:03.169058",
"uuid": "0843db7c-f161-446d-84c8-d936320da2e8"
}
}

16
doc/api_samples/os-aggregates/v2.81/aggregates-remove-host-post-resp.json Normal file
View File

@ -0,0 +1,16 @@
{
"aggregate": {
"availability_zone": "london",
"created_at": "2019-10-11T14:19:05.250053",
"deleted": false,
"deleted_at": null,
"hosts": [],
"id": 1,
"metadata": {
"availability_zone": "london"
},
"name": "name",
"updated_at": null,
"uuid": "47832b50-a192-4900-affe-8f7fdf2d7f22"
}
}

2
doc/api_samples/versions/v21-version-get-resp.json
View File

@ -19,7 +19,7 @@
}
],
"status": "CURRENT",
"version": "2.80",
"version": "2.81",
"min_version": "2.1",
"updated": "2013-07-23T11:33:21Z"
}

2
doc/api_samples/versions/versions-get-resp.json
View File

@ -22,7 +22,7 @@
}
],
"status": "CURRENT",
"version": "2.80",
"version": "2.81",
"min_version": "2.1",
"updated": "2013-07-23T11:33:21Z"
}

4
nova/api/openstack/api_version_request.py
View File

@ -217,6 +217,8 @@ REST_API_VERSION_HISTORY = """REST API Version History:
``GET /os-migrations``,
``GET /servers/{server_id}/migrations``, and
``GET /servers/{server_id}/migrations/{migration_id}``.
* 2.81 - Adds support for image cache management by aggregate by adding
``POST /os-aggregates/{aggregate_id}/images``.
"""
# The minimum and maximum versions of the API supported
@ -225,7 +227,7 @@ REST_API_VERSION_HISTORY = """REST API Version History:
# Note(cyeoh): This only applies for the v2.1 API once microversions
# support is fully merged. It does not affect the V2 API.
_MIN_API_VERSION = "2.1"
_MAX_API_VERSION = "2.80"
_MAX_API_VERSION = "2.81"
DEFAULT_API_VERSION = _MIN_API_VERSION
# Almost all proxy APIs which are related to network, images and baremetal

30
nova/api/openstack/compute/aggregates.py
View File

@ -21,10 +21,12 @@ from webob import exc
from nova.api.openstack import api_version_request
from nova.api.openstack import common
from nova.api.openstack.compute.schemas import aggregate_images
from nova.api.openstack.compute.schemas import aggregates
from nova.api.openstack import wsgi
from nova.api import validation
from nova.compute import api as compute
from nova.conductor import api as conductor
from nova import exception
from nova.i18n import _
from nova.policies import aggregates as aggr_policies
@ -39,6 +41,7 @@ class AggregateController(wsgi.Controller):
def __init__(self):
super(AggregateController, self).__init__()
self.api = compute.AggregateAPI()
self.conductor_tasks = conductor.ComputeTaskAPI()
@wsgi.expected_errors(())
def index(self, req):
@ -222,3 +225,30 @@ class AggregateController(wsgi.Controller):
key in aggregate.obj_extra_fields) and
(show_uuid or key != 'uuid')):
yield key, getattr(aggregate, key)
@wsgi.Controller.api_version('2.81')
@wsgi.response(202)
@wsgi.expected_errors((400, 404))
@validation.schema(aggregate_images.aggregate_images_v2_81)
def images(self, req, id, body):
"""Allows image cache management requests."""
context = _get_context(req)
context.can(aggr_policies.NEW_POLICY_ROOT % 'images')
image_ids = []
for image_req in body.get('cache'):
image_ids.append(image_req['id'])
if image_ids != list(set(image_ids)):
raise exc.HTTPBadRequest(
explanation=_('Duplicate images in request'))
try:
aggregate = self.api.get_aggregate(context, id)
except exception.AggregateNotFound as e:
raise exc.HTTPNotFound(explanation=e.format_message())
try:
self.conductor_tasks.cache_images(context, aggregate, image_ids)
except exception.NovaException as e:
raise exc.HTTPBadRequest(explanation=e.format_message())

6
nova/api/openstack/compute/rest_api_version_history.rst
View File

@ -1048,3 +1048,9 @@ project, for example:
* ``GET /os-migrations?user_id=ef9d34b4-45d0-4530-871b-3fb535988394``
* ``GET /os-migrations?project_id=011ee9f4-8f16-4c38-8633-a254d420fd54``
* ``GET /os-migrations?user_id=ef9d34b4-45d0-4530-871b-3fb535988394&project_id=011ee9f4-8f16-4c38-8633-a254d420fd54``
2.81
----
Adds support for image cache management by aggregate by adding
``POST /os-aggregates/{aggregate_id}/images``.

3
nova/api/openstack/compute/routes.py
View File

@ -454,6 +454,9 @@ ROUTE_LIST = (
('/os-aggregates/{id}/action', {
'POST': [aggregates_controller, 'action'],
}),
('/os-aggregates/{id}/images', {
'POST': [aggregates_controller, 'images'],
}),
('/os-assisted-volume-snapshots', {
'POST': [assisted_volume_snapshots_controller, 'create']
}),

34
nova/api/openstack/compute/schemas/aggregate_images.py Normal file
View File

@ -0,0 +1,34 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from nova.api.validation import parameter_types
aggregate_images_v2_81 = {
'type': 'object',
'properties': {
'cache': {
'type': ['array'],
'minItems': 1,
'items': {
'type': 'object',
'properties': {
'id': parameter_types.image_id,
},
'additionalProperties': False,
'required': ['id'],
},
},
},
'required': ['cache'],
'additionalProperties': False,
}

11
nova/policies/aggregates.py
View File

@ -19,6 +19,7 @@ from nova.policies import base
POLICY_ROOT = 'os_compute_api:os-aggregates:%s'
NEW_POLICY_ROOT = 'compute:aggregates:%s'
aggregates_policies = [
@ -102,6 +103,16 @@ aggregates_policies = [
'method': 'GET'
}
]),
policy.DocumentedRuleDefault(
NEW_POLICY_ROOT % 'images',
base.RULE_ADMIN_API,
"Request image caching for an aggregate",
[
{
'path': '/os-aggregates/{aggregate_id}/images',
'method': 'POST'
}
]),
]

5
nova/tests/functional/api_sample_tests/api_samples/os-aggregates/v2.81/aggregate-add-host-post-req.json.tpl Normal file
View File

@ -0,0 +1,5 @@
{
"add_host": {
"host": "%(host_name)s"
}
}

6
nova/tests/functional/api_sample_tests/api_samples/os-aggregates/v2.81/aggregate-images-post-req.json.tpl Normal file
View File

@ -0,0 +1,6 @@
{
"cache":
[
{"id": "%(image_id)s"}
]
}

9
nova/tests/functional/api_sample_tests/api_samples/os-aggregates/v2.81/aggregate-metadata-post-req.json.tpl Normal file
View File

@ -0,0 +1,9 @@
{
"set_metadata":
{
"metadata":
{
"key": "value"
}
}
}

7
nova/tests/functional/api_sample_tests/api_samples/os-aggregates/v2.81/aggregate-post-req.json.tpl Normal file
View File

@ -0,0 +1,7 @@
{
"aggregate":
{
"name": "name",
"availability_zone": "london"
}
}

12
nova/tests/functional/api_sample_tests/api_samples/os-aggregates/v2.81/aggregate-post-resp.json.tpl Normal file
View File

@ -0,0 +1,12 @@
{
"aggregate": {
"availability_zone": "london",
"created_at": "%(strtime)s",
"deleted": false,
"deleted_at": null,
"id": %(aggregate_id)s,
"name": "name",
"updated_at": null,
"uuid": "%(uuid)s"
}
}

5
nova/tests/functional/api_sample_tests/api_samples/os-aggregates/v2.81/aggregate-remove-host-post-req.json.tpl Normal file
View File

@ -0,0 +1,5 @@
{
"remove_host": {
"host": "%(host_name)s"
}
}

7
nova/tests/functional/api_sample_tests/api_samples/os-aggregates/v2.81/aggregate-update-post-req.json.tpl Normal file
View File

@ -0,0 +1,7 @@
{
"aggregate":
{
"name": "newname",
"availability_zone": "nova2"
}
}

16
nova/tests/functional/api_sample_tests/api_samples/os-aggregates/v2.81/aggregate-update-post-resp.json.tpl Normal file
View File

@ -0,0 +1,16 @@
{
"aggregate": {
"availability_zone": "nova2",
"created_at": "%(strtime)s",
"deleted": false,
"deleted_at": null,
"hosts": [],
"id": 1,
"metadata": {
"availability_zone": "nova2"
},
"name": "newname",
"updated_at": "%(strtime)s",
"uuid": "%(uuid)s"
}
}

18
nova/tests/functional/api_sample_tests/api_samples/os-aggregates/v2.81/aggregates-add-host-post-resp.json.tpl Normal file
View File

@ -0,0 +1,18 @@
{
"aggregate": {
"availability_zone": "london",
"created_at": "%(strtime)s",
"deleted": false,
"deleted_at": null,
"hosts": [
"%(compute_host)s"
],
"id": 1,
"metadata": {
"availability_zone": "london"
},
"name": "name",
"updated_at": null,
"uuid": "%(uuid)s"
}
}

16
nova/tests/functional/api_sample_tests/api_samples/os-aggregates/v2.81/aggregates-get-resp.json.tpl Normal file
View File

@ -0,0 +1,16 @@
{
"aggregate": {
"availability_zone": "london",
"created_at": "%(strtime)s",
"deleted": false,
"deleted_at": null,
"hosts": [],
"id": 1,
"metadata": {
"availability_zone": "london"
},
"name": "name",
"updated_at": null,
"uuid": "%(uuid)s"
}
}

20
nova/tests/functional/api_sample_tests/api_samples/os-aggregates/v2.81/aggregates-list-get-resp.json.tpl Normal file
View File

@ -0,0 +1,20 @@
{
"aggregates": [
{
"availability_zone": "london",
"created_at": "%(strtime)s",
"deleted": false,
"deleted_at": null,
"hosts": [
"%(compute_host)s"
],
"id": 1,
"metadata": {
"availability_zone": "london"
},
"name": "name",
"updated_at": null,
"uuid": "%(uuid)s"
}
]
}

17
nova/tests/functional/api_sample_tests/api_samples/os-aggregates/v2.81/aggregates-metadata-post-resp.json.tpl Normal file
View File

@ -0,0 +1,17 @@
{
"aggregate": {
"availability_zone": "london",
"created_at": "%(strtime)s",
"deleted": false,
"deleted_at": null,
"hosts": [],
"id": 1,
"metadata": {
"availability_zone": "london",
"key": "value"
},
"name": "name",
"updated_at": %(strtime)s,
"uuid": "%(uuid)s"
}
}

16
nova/tests/functional/api_sample_tests/api_samples/os-aggregates/v2.81/aggregates-remove-host-post-resp.json.tpl Normal file
View File

@ -0,0 +1,16 @@
{
"aggregate": {
"availability_zone": "london",
"created_at": "%(strtime)s",
"deleted": false,
"deleted_at": null,
"hosts": [],
"id": 1,
"metadata": {
"availability_zone": "london"
},
"name": "name",
"updated_at": null,
"uuid": "%(uuid)s"
}
}

21
nova/tests/functional/api_sample_tests/test_aggregates.py
View File

@ -16,6 +16,7 @@
from oslo_serialization import jsonutils
from nova.tests.functional.api_sample_tests import api_sample_base
from nova.tests.unit.image import fake as fake_image
class AggregatesSampleJsonTest(api_sample_base.ApiSampleTestBaseV21):
@ -117,3 +118,23 @@ class AggregatesV2_41_SampleJsonTest(AggregatesSampleJsonTest):
self.extra_subs['uuid'] = subs['uuid']
return self._verify_response('aggregate-post-resp',
subs, response, 200)
class AggregatesV2_81_SampleJsonTest(AggregatesV2_41_SampleJsonTest):
microversion = '2.81'
scenarios = [
(
"v2_81", {
'api_major_version': 'v2.1',
},
)
]
def test_images(self):
agg_id = self._test_aggregate_create()
image = fake_image.get_valid_image_id()
response = self._do_post('os-aggregates/%s/images' % agg_id,
'aggregate-images-post-req',
{'image_id': image})
# No response body, so just check the status
self.assertEqual(202, response.status_code)

98
nova/tests/functional/test_aggregates.py
View File

@ -41,6 +41,7 @@ class AggregatesTest(integrated_helpers._IntegratedTestBase):
agg = self.api.post_aggregate(agg)
for service in compute_services:
self.api.add_host_to_aggregate(agg['id'], service['host'])
self._test_aggregate = agg
return len(compute_services)
def test_add_hosts(self):
@ -56,6 +57,103 @@ class AggregatesTest(integrated_helpers._IntegratedTestBase):
self.assertEqual(2, self._add_hosts_to_aggregate())
class AggregatesV281Test(AggregatesTest):
api_major_version = 'v2.1'
microversion = '2.81'
def setUp(self):
self.flags(compute_driver='fake.FakeDriverWithCaching')
super(AggregatesV281Test, self).setUp()
def test_cache_images_on_aggregate(self):
self._add_hosts_to_aggregate()
agg = self._test_aggregate
img = '155d900f-4e14-4e4c-a73d-069cbf4541e6'
self.assertEqual(set(), self.compute.driver.cached_images)
body = {'cache': [
{'id': img},
]}
self.api.api_post('/os-aggregates/%s/images' % agg['id'], body,
check_response_status=[202])
self.assertEqual(set([img]), self.compute.driver.cached_images)
def test_cache_images_on_aggregate_missing_image(self):
agg = {'aggregate': {'name': 'test-aggregate'}}
agg = self.api.post_aggregate(agg)
# NOTE(danms): This image-id does not exist
img = '155d900f-4e14-4e4c-a73d-069cbf4541e0'
body = {'cache': [
{'id': img},
]}
self.api.api_post('/os-aggregates/%s/images' % agg['id'], body,
check_response_status=[400])
def test_cache_images_on_missing_aggregate(self):
img = '155d900f-4e14-4e4c-a73d-069cbf4541e6'
body = {'cache': [
{'id': img},
]}
self.api.api_post('/os-aggregates/123/images', body,
check_response_status=[404])
def test_cache_images_with_duplicates(self):
agg = {'aggregate': {'name': 'test-aggregate'}}
agg = self.api.post_aggregate(agg)
img = '155d900f-4e14-4e4c-a73d-069cbf4541e6'
body = {'cache': [
{'id': img},
{'id': img},
]}
self.api.api_post('/os-aggregates/%i/images' % agg['id'], body,
check_response_status=[400])
def test_cache_images_with_no_images(self):
agg = {'aggregate': {'name': 'test-aggregate'}}
agg = self.api.post_aggregate(agg)
body = {'cache': []}
self.api.api_post('/os-aggregates/%i/images' % agg['id'], body,
check_response_status=[400])
def test_cache_images_with_additional_in_image(self):
agg = {'aggregate': {'name': 'test-aggregate'}}
agg = self.api.post_aggregate(agg)
img = '155d900f-4e14-4e4c-a73d-069cbf4541e6'
body = {'cache': [
{'id': img, 'power': '1.21 gigawatts'},
]}
self.api.api_post('/os-aggregates/%i/images' % agg['id'], body,
check_response_status=[400])
def test_cache_images_with_missing_image_id(self):
agg = {'aggregate': {'name': 'test-aggregate'}}
agg = self.api.post_aggregate(agg)
body = {'cache': [
{'power': '1.21 gigawatts'},
]}
self.api.api_post('/os-aggregates/%i/images' % agg['id'], body,
check_response_status=[400])
def test_cache_images_with_missing_cache(self):
agg = {'aggregate': {'name': 'test-aggregate'}}
agg = self.api.post_aggregate(agg)
body = {}
self.api.api_post('/os-aggregates/%i/images' % agg['id'], body,
check_response_status=[400])
def test_cache_images_with_additional_in_cache(self):
agg = {'aggregate': {'name': 'test-aggregate'}}
agg = self.api.post_aggregate(agg)
img = '155d900f-4e14-4e4c-a73d-069cbf4541e6'
body = {'cache': [{'id': img}],
'power': '1.21 gigawatts',
}
self.api.api_post('/os-aggregates/%i/images' % agg['id'], body,
check_response_status=[400])
class AggregateRequestFiltersTest(
integrated_helpers.ProviderUsageBaseTestCase):
microversion = 'latest'

1
nova/tests/unit/test_policy.py
View File

@ -288,6 +288,7 @@ class RealRolePolicyTestCase(test.NoDBTestCase):
self.fake_policy = jsonutils.loads(fake_policy.policy_data)
self.admin_only_rules = (
"compute:aggregates:images",
"compute:server:topology:host:index",
"network:attach_external_network",
"os_compute_api:servers:create:forced_host",

15
releasenotes/notes/image-precaching-d46506568fefa1ea.yaml Normal file
View File

@ -0,0 +1,15 @@
---
features:
- |
Image pre-caching on hosts by aggregate is now supported (where
supported by the underlying virt driver) as of microversion
2.81. A group of hosts within an aggregate can be compelled to
fetch and cache a list of images to reduce time-to-boot
latency. Adds the new API:
* ``POST /os-aggregates/{aggregate_id}/images``
which is controlled by the policy ``compute:aggregates:images`` rule.
See the `[image_cache]/precache_concurrency` config option
for more information about throttling this operation.