This brings in a new version of flake8 as well as additional checks of
its own. Address all of them in advance of the bump.
Change-Id: I870bd8110106d4714db7ead94429fad6a74beb88
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Provides override arguments so that subresources that require different envelope/resource keys for requests/responses don't need to override base resource methods in a hacky manner. It also makes it so subresources don't need to "reinvent the wheel" as often by making custom request/response processing. These two patterns are relatively common and are often caused by the need to override these resource/envelope keys. For the sake of consistency across the SDK, this patch moves the logic needed in these cases into the base resource.
Change-Id: I9d928a55539c17ab54d983165afc7912cd0926f8
We were using logging.warning() to warn the user about fields that had
been removed in recent API versions or behavior that was now considered
deprecated in SDK. This was the wrong API to use. We shouldn't have been
logging, we have been using 'warnings'. From the Python docs [1]:
Task you want to perform: Issue a warning regarding a particular
runtime event
warnings.warn() in library code if the issue is avoidable and the
client application should be modified to eliminate the warning
logging.warning() if there is nothing the client application can do
about the situation, but the event should still be noted
Based on this, introduce a new module, 'openstack.warnings', containing
a number of custom 'DeprecationWarning' subclasses. 'DeprecationWarning'
isn't show by default in most cases [2] but users can opt-in to showing
them and do so selectively. For example, they may wish to ignore
warnings about fields that have been removed in recent API versions
while raising errors if they are relying on deprecated SDK behavior.
[1] https://docs.python.org/3/howto/logging.html#when-to-use-logging
[2] https://docs.python.org/3/library/exceptions.html#DeprecationWarning
Change-Id: I3846e8fcffdb5de2afe64365952d90b5ecb0f74a
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Importing munch inside of SDK is taking around 0.3 second. Itself it is
not a big problem, but it hurts on the openstackclient front. In
addition to that munch project does not seem to be actively maintained
and had no releases since 2 years.
Dropping this dependency at once is requiring quite a big rework so
instead copy a heavily stripped version of what we really require from
it. This helps us to gain performance improvement while giving time to
rework our code to come up with a decicion on how to deal with it.
Change-Id: I6612278ae798d48b296239e3359026584efb8a70
A number of compute and block storage APIs support this parameter. Plumb
them through. This turns out to be a bigger change than expected as it
highlights a number of gaps in our documentation.
Change-Id: Id4049193ab2e6c173208692ed46fc5f9491da3dc
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Next bunch of functional tests adapted to be included in the acceptance
tests.
Inclusion of tests uncovered issue in getting quota related to the fact
that not every user is capable to find/list projects. This is fixed by
introduction of ForbiddenException, skipping it in the _find call and
using current_project_id.
Change-Id: I53e718de239de7fb6f0347ff995282da079c68f3
if the status attribute has a None value, such as task_state,
wait_for_status will not work as it only compares the string form,
so if expected status is None, no need to convert to lower case
Change-Id: I9010ef6c0eb67c971b71881bd370c77a04346ab6
This makes things a little easier to grasp as we attempt to decompose
the Resource object.
Change-Id: Icd73b2ac18d783254d882a3ecd538c086e86f179
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
The 'warning_if_attribute_deprecated' helper was added in change
Iba2b0d09fdd631f8bd2c3c951fd69b243deed652 but doesn't seem to have been
used since. Similarly, the '_delete_response_class' attribute was added
in change Ifa44aacc4b4719b73e59d27ed0fcd35130358608 but had not been
used since. Remove both.
Change-Id: I9f0fb886f1c66842d97f57c2962de1627841aeb6
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This is a simple off-by-one error, but there are lots of things feeding
into this that I figured were worth explaining. Apologies in advance for
the length of this screed. Hopefully it's useful for someone.
There's a small bug in how we paginate when listing resources, or rather
when listing Glance objects. Glance doesn't return a 'Links' response
header or 'links' response body field like most other OpenStack
services. Instead, it returns a simple 'next' body field that is
relative. For example, consider the paginated response to the '/images'
API.
{
"images": [ ... ],
"first":"/v2/images",
"schema":"/v2/schemas/images",
"next":"/v2/images?marker=823762fb-3128-43cf-8136-cee6b749f708"
}
As a result, we're forced to treat Glance as yet another "special"
service when it comes to pagination. The first step of this is to look
for the 'next' field and grab the value. As you can see, this value
also includes the version prefix, which we don't want this since
keystoneauth1 already appends the version prefix when generating
absolute URLs from information in the service catalog. This means our
next step is to strip this version filter, which we have done since
change Iad8e69d20783cfe59c3e86bd6980da7dee802869. Unfortunately, the
original author of that change was a little too liberal in their
stripping (heh) and also removed the leading new slash. This causes
issues with the final part of our generation of the next link, namely
the deduplication of query parameters. Because most servers enforce an
upper limit on the size of URLs, it's important that we don't constantly
append our query parameters to the request URL for each page. Instead,
we parse the query parameters present in the 'next' URL and munge them
with our our parameters. The way we do this is using
'urllib.parse.urljoin' method with this pattern:
urljoin(url, urlparse(url).path)
For example:
>>> url = 'http://example.com/foo/bar?pet=dog&name=baxter'
>>> urljoin(url, urlparse(url).path)
'http://example.com/foo/bar'
>>> url = '/foo/bar?pet=dog&name=baxter'
>>> urljoin(url, urlparse(url).path)
'/foo/bar'
Because we were stripping the leading '/', the first part of the path
was being identified as the server and was stripped.
>>> url = 'foo/bar?pet=dog&name=baxter'
>>> urljoin(url, urlparse(url).path)
'foo/foo/bar'
After all that, the solution is simple. Stop stripping the leading
slash and generate a proper relative URL, as expected.
Change-Id: Ic9e1891ae7171a0cc45dd9faf1b2a6e37535b777
Co-authored-by: Stephen Finucane <stephenfin@redhat.com>
Story: 2010273
Task: 46200
Extend resource list method to accept all possible filtering parameters.
What is supported by the API are sent to the server. Remaining
parameters are applied to the fetched results. With this
`allow_unknown_params` parameter of the list call is dropped.
Change-Id: Ie9cfb81330d6b98b97b7abad9cf5ae6334ba12e7
The proxy layer has been simplified to stay closer to the wire.
It no longer runs automated cleaning. The initial state of available
is implemented via using an old API version. The default state is
enroll (like in Ironic).
Change-Id: I2894c82d847f8a3dc6bdae1913cb72a1ca4b764b
Sometimes users may want a specific behavior of a certain microversion
rather than just the most recent supported one. For example, Ironic only
supports creating nodes directly in the "available" state before 1.11.
Change-Id: I2458650a9ce30440b5e29b940eaf1df60239ff32
We had '_get_microversion_for_list' and '_get_microversion_for', the
latter of which simply called the former. There's no reason for this
distinction and I suspect it's simply there because of legacy reasons.
Merge them.
Change-Id: I5124fa0005e809891a1fc970c6ccdc5f4a766d33
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Neutron clients are still refering to tenant_id in very many places and
the warning become very intruisive. Disable them untill we ourselves get
rid of those usages in OSC and other similar places.
Change-Id: Iaf5b467cf42a53b60e14085b0c0cb86d9ede3310
Resources allow storing arbitrary values in the body with
_allow_unknown_attrs_in_body, but these unknown attributes aren't
returned when converting to_dict.
This patch enhances to_dict with an unknown_attrs parameter, that allows
including the unknown attributes stored by the resource in the converted
dictionary.
Change-Id: I4c7685a4bb4aafb4cceadb6a507980e99c1e1a11
Replace caching solution:
- cache on the proxy layer (API communication)
- support caching all GET calls
- add possibility to bypass cache (important for _wait_for_ operations)
Cheery-Picked-From: https://review.opendev.org/c/openstack/openstacksdk/+/805851
Change-Id: I2c8ae2c59d15c750ea8ebd3031ffdd2ced2421ed
There were multiple calls to subclasses of 'SDKException' that were
providing a 'msg' kwarg. This doesn't exist. Instead, the parameter is
called 'message'. Correct this by using positional args instead.
Change-Id: Id425b9a57033c89b8a8466a32ab202cdf7556cb2
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
We're going to be doing some work on these functions to improve how we
handle pagination and nested resources. Before we start this surgery,
rewrap the function signatures.
Change-Id: I660599355fa6d1501acb5e8584245985dd05440e
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
The goal is to avoid the error
```
Attribute [] not found in [<openstack.resource._ComponentManager object at 0x7fc649c52520>]: ''.
```
in debug messages when the field `_alternate_id` is empty
Change-Id: I70dc26ff161227ea1d65d935a743dc6f6d3485ec
In order to keep backward compatibility while we are switching more and
more things to the proxy layer add possibility to get attribute (in dict
style only) by server-side attribute name. While we do this inform user
not to use this anymore.
Change-Id: I981892aaba8d3f476a5d1ad71d22ab82b8798975
It is optional for resources to send in request's body attributes
which aren't included in mapping.
This is needed for Neutron to allow sending extra arguments in
POST/PUT requests to Neutron.
If that will be possible in SDK and OSC, Neutron team should be finally
able to deprecate (and drop) support for neutronclient CLI.
Change-Id: Ia3fd1052f438aedc6bc89bcca71834bcbafec9b9
Update version of tox to 3.9.0 to support inline comments in tox.ini
Import pep8 test requirements directly in tox.ini and do not import all
the test-requirements
Update version of hacking
Fix import orders in various modules
Leave filter for imports in tests/ for the time being
Change-Id: Ia625036d1f50ae97880ef70335804228320a9c6d
When resource listing is invoked the URI based properties are not saved
on the resource, as opposed to all other methods. Fix this by injecting
them into each fetched resource.
Change-Id: I95cb50a691d095764df30f46c204ea5ee80dacca
There is potential (i.e. bug in older nova
https://bugs.launchpad.net/nova/+bug/1721791) for resource listing to
end up in an endless loop. Try preventing this if next marker is same as
marker we have just used.
Change-Id: I5fb9ff1675cef7c4cb5e1f50a9efbbd8efc23900
In case of some resources we might want to override base_path for list
function invoked by find (find resource using /details path).
Change-Id: Ie2387c2f262b9a6696358e2874399a095da1ba5e
In order to switch next part of OSC towards SDK we need to add some
missing bits to the aggregates.
Since this is another example when API returns 400 if we try accessing
resource with name and it doesn't support - modify general behavior for
also skipping 400 in the find method
Change-Id: Ia6711b1c27514d0698fec1efedaefeeb93722b9d
Using SDK uncovered 2 code deprecation warnings:
openstack/resource.py:210: DeprecationWarning: Using or importing the
ABCs from 'collections' instead of from 'collections.abc' is deprecated
since Python 3.3, and in 3.9 it will stop working class
_ComponentManager(collections.MutableMapping):
openstack/resource.py:351: DeprecationWarning: inspect.getargspec() is
deprecated since Python 3.0, use inspect.signature() or
inspect.getfullargspec() len(inspect.getargspec(type_).args) > 1) #
Follow recommendations and replace those usages
Change-Id: I0a38dceb1739ba0601d751d5a9837524353475b5
We do this in list, but we should really also do it in
translate response - because seriously this can never work.
Change-Id: Ic97499b9fe1b563049da59899c34dbd810fe1509
