Extend rest api v2 with multiple sort keys support.
Example:
/v2/images/detail?sort_key=name&sort_key=size
Changed database api which now takes sort_key param as a list instead of string
python-glanceclient support will be added in separate commit
Change-Id: Ib7a6aeb2df3bc5d23fe8e070290b5bfcab00c0f5
DocImpact
Partial-Bug: 1221274
Since v2 api has no sort key validation and the default
pagination check is used it's possible to request something
like /images?sort_key=_sa_class_manager, which causes an
inner SQL exception with 500 response code from the server.
This code validates input sort key and raises an exception
if the parameter is out of the supported keys list.
Closes-bug: 1400366
Change-Id: I0cf58ad198375a2f6f58bd7820cbb9d86003247a
Container, disk format and checksum - among other fields - are not
required parameters, therfore this patch allows them to be None.
This will allow the client to have attributes with value `None` and
it'll make it consistent across responses and with regards to the API.
NOTE: A follow-up patch will enable None values to be returned by the API.
Closes-bug: #1398313
Change-Id: I75da1e9309e0f7ef8839dea3ec9c99c58edc5d63
It fixes some eyesores that have been found in glance domain:
1. Removes duplicate declaration of a varariable 'name'
2. Changes type of several image transitions from 'str' to 'tuple'
3. Sets the correct default values to tags and extra_properties
4. Changes strange constructions like
self.tags = kwargs.pop('tags', None) or [] to
self.tags = kwargs.pop('tags', [])
Change-Id: I9309c7dc70b9910c6c09111f26ee489fa7ab9260
Image create or update with params containing 4 byte unicode characters
results in 400 server error code (instead of former 500).
Image name/location/tags/properties are validated this way
Change-Id: Ib0fcf2456f0996e8235983b361d1ee5f66136375
Closes-bug: #1370954
The builtin _ should not longer be used. Instead _
should be explicitly assigned from glance.i18n.
This patch does the following:
(1) adds these explict assignments to any
glance file that was missing them,
(2) removes the defintion of _ as a builtin from
tox.ini so that a pep8 failure will occur
if _ is not defined,
(3) removes calls to install().
This removes the last direct use of gettextutils by glance.
The indirect uses, via openstack/common, will be removed
in the next sync with oslo.
Change-Id: Ie54e8a67e747e022cc01022dfad35d89686bdfc4
Changing all Glance files to use i18n instead of gettextutils for log
translations.
Finishes bp refactoring-glance-logging
Change-Id: I90d6ab4f7e09f4e7591921f9526de6370ebdb789
This change complements glance change #106776 by raising some logging
levels and fixing some of the translations of logging.
Related to bp refactoring-glance-logging
Closes-Bug: #1336955
Change-Id: Ib547f4603025ebe4f6c055779032ab4064784490
Currently, in image_create v2 api source we can see
ReservedProperty, ReadonlyProperty and TypeError caused in
glance/domain/__init__.py are not handled, which will cause
500 when these exceptions happens.The internal error should
be interpreted to the appropriate exception to the user.
This patch catches them and changes them to http exception.
Change-Id: I0c82a142b3357c8b8faab7dc0905967983cb8f7c
Currently it is impossible to determine if given image schema property
is base or custom one and knowledge of that can be handy in some
situations. Proposed change appends to every custom property special
key which determiness that it is not a base property.
Change-Id: I49255255df311036d516768afc55475c1f9aad47
Partial-Bug: #1371559
(cherry picked from commit 94c05cbdbb)
Currently it is impossible to determine if given image schema property
is base or custom one and knowledge of that can be handy in some
situations. Proposed change appends to every custom property special
key which determiness that it is not a base property.
Change-Id: I49255255df311036d516768afc55475c1f9aad47
Partial-Bug: #1371559
According to the RFC for JSON Patch
http://tools.ietf.org/html/rfc6902#section-4.1
the 'add' operation, in the event that the property being
added already exists, should replace the existing value. For backward
compatibility We've fix it only for the latest content type.
Change-Id: Ia4be8b384f8b7f193f139f2206ceea25786b5cc3
Closes-bug: 1250158
DocImpact
Partial-bug: #1277104
Based on current code, in API v2 duplicated image id will
return HTTP 500 error instead of 409. This patch fix this
issue and add an unit test case for that.
Closes-Bug: 1366503
Change-Id: Ia6c27f9edffd63619d63013e456072d811daf127
Based on current implement, API V2 is using **id** in image schema,
but using **image_id** for most of the internal functions as key
word arguments. So user will got a key error if using:
glance --os-image-api-version 2 image-create --id <id>
This patch will map 'id' to 'image_id' in RequestDeserializer and
add an unit test case fo the change.
Closes-Bug: 1366515
Change-Id: I4c72881a7ada728307e806776426b42cf4cf0993
This commits removes the old `store` package from glance and adopts the
usage of the new glance.store library. The library was designed to
preserve backwards compatibility as much as possible. In fact, most of
the changes in this patch are related to function args ordering and not
function renames or workflow changes.
Some changes that are worth mentioning:
1. Glance store doesn't rely on a global config object. All config
options must be explicitly registered.
2. All store operations now accepted an optional context. This is a
fallout from the context not being required in the `Store` constructor
anymore.
3. Store drivers are behind a private package called `_drivers` and
they're not suppose to be accessed directly. Instead, functions like
`get_store_from_scheme` should be used.
4. Stores are disabled by default
5. All the store specific options are under the `glance_store` group.
DocImpact:
The old store related configuration options have been moved under the
`glance_store` section. However, the old options will go through a
deprecation path. That is, they'll still be read from the `DEFAULT`
section to give deployers enough time to update their config files.
In k-2, the deprecated options will be completely obsolete.
Closes-bug: #1291848
Implements-blueprint: create-store-package
Change-Id: Iaacc70993ad5da292b93de42bbecda73d53b19fd
A sample import script to successfully import image from http or https
location is introduced. This should work on a devstack installtion.
Also, the following changes are introduced:-
1. An interface for implementing any other type of executor.
2. Provides namespace for keeping Tasks scripts.
3. A config for choosing deployer specific executor.
4. An interface for writing Tasks scripts.
5. A module for common script related methods.
6. Logic for limiting number of simultaneous tasks execution on the
Glance API server.
partially implements bp async-glance-workers
DocImpact
Change-Id: I382472fffd0fdad43573e72b2e78a9a6ed1e1f1a
A randomized PYTHONHASHSEED caused the glance API to return certain data
in a nondeterministic order, breaking some tests. This patch sorts the
returned data before testing against a pre-sorted expected values.
This patch also fixes some unit test failures with the new version of
testtools (0.9.36) and directly calling setUp(). This was added to this
patch to avoid blocking more Glance tests.
Co-authored-by: Fei Long Wang <flwang@catalyst.net.nz>
Co-authored-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Partial-bug: #1348818
Change-Id: Ie92c6bb5663826713ca99e8f2f7d70f3faef6675
If user create an image with Ceph backend driver, and then create
an instance based on the image. As a result, user will run into
500 error when deleting the image. Because rbd driver is raising
exception.InUseByStore()at [1] when the image is in use, but no
code to catch the exception at REST API layer.
[1] https://github.com/openstack/glance/blob/master/glance/store/rbd.py#L290
Closes-Bug: 1345942
Change-Id: I350c235944aab226dc96de169be2087e2b599ab4
This change fixes translations to be using correct translation functions.
Contains few log level drops for too high logging levels.
This PS does not address logging changes for modules under glance/store/
due to the new repo request. After stores are moved to their own repo
new PS will be submitted for those.
This PS does not touch the test code.
Separate PS will be submitted for log lever raises withing codebase
as requested during the spec review.
Related to bp refactoring-glance-logging
Change-Id: I683c35face8d5b7056ef7fedcd75deb63dbbf8e2
In Glance API V2, currently, the user gets a 500 error when tries
to use the "changes-since" filter.This filter, which was available
on API v1 is no longer available on V2.
This patch returns to the user a 400 Bad Request with a proper message,
instead of a generic 500 Internal Server Error.
Change-Id: Icece859a7f12f6f4aa1f7bb945134a935048b315
Closes-bug: 1273975
Adding a status field to image's each location property, each location
status can be 'active', 'pending_delete' and 'deleted'.
Under location's status information Scrubber service can make cleanup
based on DB records also but not a dedicated queue-file for each image.
This is second part of this change which covered DB API, domain and REST
API.
Partially-Implements BP: image-location-status
Change-Id: I744679e2dadbaec099aef33d8c5a3fe4ecf96865
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Use utils.exception_to_str to consistently cast the
exception to avoid the unicodeError stack trace.
Closes-Bug: #1340002
Related-Id: I54fce91c73adca846ec99ed50a57b95d16428ec1
Change-Id: I77478a8eeeefd85bf98138156429df53d0a4c079
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
It's good practice to use with statement to work with files. This way files get
closed automatically at the end of the block. Fixed some cases, including some
where file wasn't closed after usage
Change-Id: I9399f2a2b7edb1cd5aa64782cd9f980f9233b827
To support Python3, unicode() calls has been replaced by
six.text_type.
Added utils.exception_to_str(): is the proper way to convert
an exception to string, since it manages logic related to
encoding.
Change-Id: I27101390e4f95e5c7690b1b445b7e75b8bcb9a08
Closes-Bug: #1284677
Add support for doing language resolution for a request,
based on the Accept-Language HTTP header. For example,
an HTTP client can receive API messages in Chinese even
if the locale language of the server is English.
The underscore (_) method is initialized so that it returns
openstack.common.gettextutils.Message objects. The locale
of these Message objects is set when exceptions with which
they are associated are raised in the context of an HTTP
request.
docImpact
Partially implements bp i18n-messages
Signed-off-by: Fei Long Wang <flwang@cn.ibm.com>
Signed-off-by: John Warren <jswarren@us.ibm.com>
Change-Id: I352cda57fe119022c59c6c813b5c8053765b2d3c
If no default value is specified, the get method of dictionaries
defaults to 'None' already.
So removing the 'None' argument to make the code more clean and
readable.
Change-Id: I2f29315488d515aa2837d06ea59ab41e8ca05fc8
Glance server currently uses 1-based indices for indexing location
entries. This violates the JSON-pointer RFC (RFC 6901) which states that
indicies should be 0-based. Furthermore, since this deviation is not
mentioned in the docs any application directly calling the REST APIs
would trip over it.
This commit makes Glance use zero-based indexing for location entries.
Note that the "show_multiple_locations" config option should be set to
True to expose location related REST operations.
DocImpact: Since this change modifies user-visible Glance behaviour it
should be explicitly documented because the previous behaviour (1-based
indexing), though not explicitly documented, was exposed through the
REST APIs.
Change-Id: I6326455874c381fcb0d8babf9cc4fa8311e219d2
Closes-bug: #1282437
This patch adds the knowledge of a virtual_size field to Glance's API
v2. The virtual_size field should respect the same rules applied to the
size field in terms of readability, access control and propagation.
Glance's API v1 has been left unmodified.
docImpact
Implements blueprint: split-image-size
Change-Id: Ie4f58ee2e4da3a6c1229840295c7f62023a95b70
Apply image location selection strategy into Glance server side. Image
download handling and "direct URL" exporting will be effected by this
mechanism.
Implements bp: image-location-selection-strategy
Related-Id: I86f192aeae8e5f21a72f946552f6507654c25a6c
Change-Id: I7bd093a16db3af2b604cad22a6b6971345af82a2
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
The 'owner' property on an image is exposed in v1 but does
not appear in the v2 image response. It would be useful to
have this available.
docImpact
Implement blueprint expose-owner-in-v2
Change-Id: I90f68d57e185f4a5ee63cd97bd5597948dde33ea
When user updates an image with a nonexisted location, it's
expected to run into location not found exception. However,
because the error message is rewritten in v2 image update,
so user will get below wrong message:
Failed to find image <image-id> to update
This fix will remove the rewritten message, given the NotFound
message has been set at: glance/db/sqlalchemy/api.py#L159
Closes-Bug: #1265448
Change-Id: I4e5fbbe40ce0fd13e828a40a424d4fcdc66a236b
Move the configurations which core domain objects depend on to
common/config.py, and add a argument "task_time_to_live" of Task's
__init__ method and TaskFactory's new_task method for delivering
the CONF.task.task_time_to_live value.
For convenience,the argument "task_time_to_live" is set a default value 48.
Change-Id: Iffda1ecd25470824c812d66a27fa64ebbaabcf07
Closes-Bug: #1250633
At the moment, in both api versions, min_disk and min_ram can be set to
negative value - which doesn't make sense. This commit fixes this behavior
in the two versions of the glance api.
Change-Id: Ib7855c1d928ab8262dd2a109619a4e3d2299ff9e
Closes-bug: #1265711
There are some unused import in glance, such as 'import domain' in v2.image_data.py.
Change-Id: I9c3f06bfde4dbf130dd8525fc00b3f73c30ab4d7
Closes-Bug:#1264245
Every call to json.loads/json.dumps have been changed to
jsonutils.loads/jsonutils.dumps respectively. Import json has been removed
also replaced by import glance.openstack.common.jsonutils
654d80b416dc5f413cb791aa838ec8688bf7da44 Create openstack.common.jsonutils
Change-Id: I8ef580e5eb91526dfaef37050ce1f5c6d88d56b5
Closes-bug: #1257922
Use common oslo database code for work with session, connection, engines
instead of glance implementations.
Work with config options modified due to new common code config.
Removed unused glance code and tests for it.
Details:
glance/cmd/manage.py
- removed load load the glance-registry config. We load sql connection
in common code now.
glance/db/sqlalchemy/api.py
- removed functions _wrap_db_error(), _is_db_connection_error(),
_get_maker(), get_engine(), _ping_listener() and config
`sql_connection_opt`. We have all these thigs in module
`glance.openstack.common.db.sqlalchemy`.
- modified functions setup_db_env(), add_cli_options(), clear_db_env().
- functions _get_session() and get_engine() now call same functions
from module `glance.openstack.common.db.sqlalchemy`.
- replaced sqlalchemy exception `sqlalchemy.exc.IntegrityError` to
database duplicate exception - `db_exc.DBDuplicateEntry`.
Removed redundant tests - we use common code , so a lot of features was
already tested in Oslo.
Global variable _ENGINE was replaced by common function get_engine().
DocImpact
New database config options came from Oslo. See Table below
-------------------------------------------------------------------
Database config options ([group] option)
-----------------------------|-------------------------------------
Glance | Oslo
-----------------------------|-------------------------------------
[DEFAULT] sql_connection | [database] connection
[DEFAULT] sql_idle_timeout | [database] idle_timeout
[DEFAULT] sql_max_retries | [database] max_retries
[DEFAULT] sql_retry_interval | [database] retry_interval
[DEFAULT] db_auto_create | NONE
[DEFAULT] sqlalchemy_debug | [database] connection_debug
[DEFAULT] use_tpool | [database] use_tpool
NONE | [database] slave_connection
NONE | [database] min_pool_size
NONE | [database] max_pool_size
NONE | [database] slave_connection
NONE | [database] min_pool_size
NONE | [database] max_pool_size
NONE | [database] max_overflow
NONE | [database] connection_trace
NONE | [database] pool_timeout
-------------------------------------------------------------------
blueprint db-use-oslo-common-code
Change-Id: I3ff976545b1a82ff8df780e34128fcaf6f892b8c
This patch adds the image_tag_quota config option. This allows a deployer
to limit the number of image tags allowed on an image for v2. The default value
is 128. If an image is somehow over the limit, tags can still be removed but no
more may be added.
Fixes bug 1252337
docImpact
Change-Id: I2036e2a88601d7c5aa85fad32d90fe0ed30b84c8
Clean up imports due to F401 and F403 checks.
F401 'module' imported but unused
F403 unable to detect undefined names with wildcard import
Change-Id: I487edb157de1a6babc7ad8a3fb65f195e476c490
Glance allow users to have duplicated locations for the same image. This
could led to some confusion and bad UX. This patch introduces such check
in the insert method and makes both extend and append use it to add new
images.
The patch also adds a new exception - DuplicateLocation - and handles it
in the v2 of the API. When `DuplicateLocation` is raised, a BadRequest
will be returned back to the client.
A new test for this functionality was added as well.
Change-Id: I32400d467408e3d56bbbdbb3379357f2ee812e56
Closes-bug: 1251244
This patch adds the image_property_quota config option. This allows a deployer
to limit the number of image properties allowed on an image. The default value
is 128, as is currently the limit enforced by nova. Users will only be able to
update an image if the result of the transaction would be under this limit.
This behavior is intended to be similar to 'quota_metadata_items' in nova.
This is for both Glance v1 and v2.
Fixes bug 1251518
docImpact
Change-Id: I4aa9504deae836404f11c9ada71a91f85caeba4c
Per the hacking guidelines locals() should not be used when creating
messages, but additionally this is preparation for upcoming message
internationalization work.
Change-Id: I9ce616b94faf2d7df895c6bffb94baac8c82c7ce
Instead of globally ignoring Pyflakes and Hacking
warnings, blacklist explitly only those that trigger
numerously. Fix the rest alongway and start to gate
on those that are now passing.
Change-Id: Ia19dc708cf0ccab2da5b46d1652f16e901499c24
This patch will fix below two issues of V2 ResponseSerializer for
images-list, image-show, image-update and image-download.
1. A user should be able to list/show/update/download image without
needing permission on get_image_location.
2. A policy failure should result in a 403 return code. We're
getting a 500.
Fixes bug 1231255
Change-Id: Ie0ec2d574eea4433c4f610ec66a22cb16cae6dc6
Mike Fedosin
Flavio Percoco
Inessa Vasilevskaya
James Carey
Erno Kuvaja
Haiwei Xu
Kamil Rykowski
Julien Danjou
Roman Vasilets
Fei Long Wang
Nikhil Komawar
Louis Taylor
henriquetruta
Zhi Yan Liu
pran1990
Leandro I. Costantino
Fei Long Wang
Vyacheslav Vakhlyuev
Sergey Nikitin
David Koo
Fengqian Gao
Jia Dong
Yanis Guenane
llg8212
Boris Pavlovic
Alex Meade
Victor Sergeyev
Luis A. Garcia
Dirk Mueller