Commit Graph

345 Commits (6ee21bd722b3e1dbec3e5a211e32f10fb2a20603)

Author SHA1 Message Date
Grzegorz Grasza 5098d45cca Allow passing of version header
Add keyword option to get_version_data() to allow passing
of the version header so that we can get the microversions.
Specifically, this is so that we can re-use this function
in barbican, which recently implemented microversions, but
doesn't return them by default, for backward compatibility
with old clients.

Change-Id: I909750381a559f9dc61650c9f98c88d4481012b7
2022-12-20 15:58:04 +01:00
Yi Feng aa9c5d230f OAuth2.0 Client Credentials Grant Flow Support
Added a new OAuth2ClientCredential plugin, accessible via the
'v3oauth2clientcredential' entry point, making possible to authenticate
using an application credentials as an OAuth2.0 client credentials.

Change-Id: I77d6faef4cbc75abb8e7d86f386fb6d16e40cabf
2022-08-30 06:29:20 +00:00
Zuul f194e6a820 Merge "Allow logging of Content-Type text/plain" 2022-05-13 17:11:37 +00:00
melanie witt bc491817e1 Allow logging of Content-Type text/plain
Noticed this while doing some local testing, if a WSGI app replies with
a text/plain content type to communicate a server error, we aren't able
to see the error response message when passing --debug to the
openstackclient, example:

  RESP: [500] Date: Thu, 01 Oct 2020 23:54:15 GMT Server: Apache/2.4.18
  (Ubuntu) Content-Type: text/plain; charset=UTF-8 Connection: close
  Transfer-Encoding: chunked
  RESP BODY: Omitted, Content-Type is set to text/plain; charset=UTF-8.
  Only application/json responses have their bodies logged.

Change-Id: Ibfd46c7725bd0aa26f1f80b0e8fc6eda2ac2e090
2022-04-29 15:57:52 +00:00
Dylan McCulloch 8e27ff5d13 Fix version discovery check of url for integer project id
Check if the last url segment matches the project id.
Previously the check only confirmed whether the last url segment
endswith the project id which could cause problems with spurious
matches of some legacy integer project ids.

Closes-Bug: 1968793
Change-Id: I7c6c22e41bde2a73508635b7e964c58a02c12146
2022-04-13 09:43:29 +10:00
Goutham Pacha Ravi 112bcae1fb Specify manila microversion header
Manila API honors a "X-OpenStack-Manila-API-Version"
header to specify microversions.

It may support the OpenStack-API-Version header
in a future release, however, we'll need to maintain
backwards compatibility with the existing API.

Change-Id: Ia2e62d3a11a08adeb6d488b7c9b365f7ff2be3c8
2021-02-18 08:38:29 -08:00
Dmitry Tantsur 981a19bba1 Correct major version discovery for non-keystone plugins
When a non-keystone plugin is used together with an unversioned endpoint,
we give up on discovery before figuring out both major version and
the correct endpoint. This is because get_endpoint_data is called with
discover_versions=False, so discovery assumes we have all information
already. It may be an issue in discovery itself, but I'm afraid to
touch that code. Instead, if get_endpoint_data returns no API version
with discover_versions=False, try with discover_versions=True, which
matches what the identity plugins do.

Also increase the unit test coverage.

Change-Id: Ie623931b150748d7759cf276e0023a2f06a8d4db
2020-07-31 11:32:26 +02:00
Monty Taylor b95a89e3ff Fix get_endpoint_data for non-keystone plugins
We expect endpoint_override, but these plugins won't necessary
have it, they have endpoint instead.

Co-Authored-By: Dmitry Tantsur <>
Change-Id: Iead4b95c1f5b8d84cec705da32f41049e2eea641
2020-07-27 17:20:58 +02:00
Steve Baker ff68663217 Implement HTTP Basic client support in keystoneauth1
A new basic auth plugin is added which enables HTTP Basic
authentication for standalone services. Like the noauth plugin, the
endpoint needs to be specified explicitly, along with the
username and password.

An example of a standalone server implementing HTTP Basic can be seen
in Ironic change

Change-Id: Ib3f0a9c518d031a67f9605cf64a8a9cc81131ed3
Story: 2007656
Task: 39741
2020-06-15 10:26:35 +12:00
Zuul 94314329e9 Merge "Replace assertItemsEqual with assertCountEqual" 2020-06-05 04:45:24 +00:00
Zuul e96c2102ad Merge "Use unittest.mock instead of third party mock" 2020-06-05 04:21:48 +00:00
Zuul 1f0412a042 Merge "Make header Case Insensitive" 2020-06-03 17:57:33 +00:00
Joel Capitao 1fe8df2bc8 Replace assertItemsEqual with assertCountEqual
assertItemsEqual was removed from Python's unittest.TestCase in
Python 3.3 [1][2]. We have been able to use them since then, because
testtools required unittest2, which still included it. With testtools
removing Python 2.7 support [3][4], we will lose support for
assertItemsEqual, so we should switch to use assertCountEqual.
Credits to [5].

[1] -
[2] -
[3] - testing-cabal/testtools#286
[4] - testing-cabal/testtools#277
[5] -

Change-Id: Ib1db7694a8f0f59d8762b02acbb4ef16e5176098
2020-06-03 15:14:41 +02:00
Ghanshyam Mann c00fca4a09 Make header Case Insensitive
In case of global-request-id request, Adapter
send two global request id header
 - "X-OpenStack-Request-ID"
 - "X-Openstack-Request-Id".


This is becasue of the header not being Case Insensitive
and end up with two different name of same header with difference
of cap 'D'.

Unit test for whether request global-request-id has precedence
over adapter fail many times because of how different python version
treat the dict. py3.6 and above are all good as dict maintain the
insertion ordered but py3.5 can fail it any time.

We can see consistent failure in py35 jobs:

Let's make the headers always Case Insensitive which is
what RFC says.

Change-Id: Iba707dd0506d22e144aca4fdfc9b140c8e37ae02
Closes-Bug: #1881351
2020-05-29 17:17:11 -05:00
Sean McGinnis edc2ae4249
Use unittest.mock instead of third party mock
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.

Change-Id: I07d61e1a8f18d65acdf86cdd61f7d9e28157f1d7
Signed-off-by: Sean McGinnis <>
2020-05-29 10:37:58 -05:00
Lance Bragstad ad46262148 Inject /v3 in token path for v3 plugins
Without this, it's possible to get HTTP 404 errors from keystone if
OS_AUTH_URL isn't versioned (e.g., instead
of, even if OS_IDENTITY_API is set to

This commit works around this issue by checking the AUTH_URL before
building the token_url and appending '/v3' to the URL before sending the

Closes-Bug: 1876317

Change-Id: Ic75f0c9b36022b884105b87bfe05f4f8292d53b2
2020-05-22 09:38:27 -05:00
Monty Taylor 4743b7f8e4 Fix E741 pep8 failure
Change-Id: I5ab94b07a5fc64a6ab662cccb63ef25486982ca6
2020-05-22 09:37:34 -05:00
Andreas Jaeger c096099416 Update hacking for Python3
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.

Fix problems found.

Update local hacking checks for new flake8.

Remove hacking and friends from lower-constraints, those are not
needed for co-installing.

Change-Id: I59f0854c089a6ed4f0c4dad7755f946dc95ada3a
2020-03-31 20:11:31 +02:00
Zuul de53f90bf9 Merge "Fetch discovery documents with auth when needed" 2019-10-21 19:25:50 +00:00
Zuul 5e5185f80f Merge "Allow initializing session with connection retries" 2019-10-09 20:23:21 +00:00
Zuul 46835f19b8 Merge "Make tests pass in 2020" 2019-10-05 09:51:47 +00:00
Zuul c102a7e9f5 Merge "Cleanup session on delete" 2019-10-05 09:38:48 +00:00
Monty Taylor 26ad02db0f Fetch discovery documents with auth when needed
Some services, like Nova, default to requiring auth for their
versioned discovery documents. This means strict discovery
does not work on them, because discovery as it is now defaults
to not sending auth. Just changing the default would be a behavior
change resulting in sending unneeded data with *every* request.
Instead, respond to Unauthorized exceptions by retrying the request
with auth token. This way discovery will work for services that
are otherwise blocking unauthenticated access, and will get more
efficient over time as those services improve.

Change-Id: I8a33e8a05bed0f18e4e42431f6d16b8a6a5270ef
2019-10-04 18:36:29 +02:00
Bernhard M. Wiedemann 4461358098 Make tests pass in 2020
This issue was found while working on reproducible builds for openSUSE.

This solves it similar to change I73bde68be53afff4e8dff12d756b8381f34b2adb

Changed month to February to avoid races around new year.

Change-Id: I2a28f3f4eaabaa772df395f3f5d55b6fd78f8968
2019-09-27 13:58:34 +02:00
Colleen Murphy 8b06c57292 Simplify session logger object tests
With the requests-mock logger now configured to log the request[1],
checking that the logger output does *not* contain the request is
invalid. Simplify these two tests by omitting the assertion.


Closes-bug: #1842978

Change-Id: If3c0447502917bce831d3e9f7ae4c31374dd4380
2019-09-05 13:53:53 -07:00
Rabi Mishra 373cbdbda8 Allow initializing session with connection retries
Though we can now set ``connect_retires`` while creating an adapter object,
that  would allow retries in case of connection timeout (ex. with session
clients derived from Adapater/LegacyJsonAdapater), it can't be used in
certain scenarios like endpoint discovery with auth plugin get_discovery()
or getting AccessInfo with get_access()/get_auth_ref().

Having ``connect_retries`` in Session constructor would allow users
with option of setting it when creating session objects (if they want)
and can be overridden per service with the adapter interface.

This commit also changes the default value of ``connect_retries`` from
0 to None to allow for adapter's to override retries on the session


Change-Id: Iffb671fefae23926b1f09017d9db438341eae238
Partial-Bug: #1840235
2019-09-05 19:31:22 +00:00
Alex Schultz b2b5ad3cb1 Cleanup session on delete
If an external session object was not passed to the Session class, we
create a requests.Session() on our own. Once this is used, it may still
have an open connection when the auth Session is closed. We need to
handle the closing of the requests.Session() ourselves if we created
one. If you do not close it, a ResourceWarning may be reported about the
socket that is left open. If a session object is provided, we do not
attempt to close it as it will be up to the code consuming keystoneauth
to properly handle cleaning up the provided session.

Change-Id: I590755d665b371c76ba8e02836d81d41a95ac601
Closes-Bug: #1838704
2019-08-26 08:12:27 -06:00
Zuul 38cd5fc6c3 Merge "add support for auth_receipts and multi-method auth" 2019-08-15 01:44:00 +00:00
Adrian Turjak 6a69e4dfbd add support for auth_receipts and multi-method auth
- new exception when an auth receipt is returned.
- a new method for auth receipt.
- support to existing v3 Auth plugins to add additional methods.
- Added a new MultiFactor plugin with loading support which
  takes method names as strings.

Change-Id: Ie6601a50011118e3a07be9752f747c2298ff5230
Closes-Bug: #1839748
2019-08-14 11:51:28 +12:00
Monty Taylor 8e59fb20b3 Add remove_service to token fixtures
SDK is going to start using these Token fixtures to programmatically
create fake service catalogs in the test suite containing entries
for everything in service-types-authority.

In order to be able to test code paths where some service does not
exist, it would be good to be able to just remove a service from
the catalog, instead of needing to construct a full new one from

Change-Id: I4b5469aefbe9b91c125da482509cdc627faa5525
2019-08-07 21:42:57 +00:00
Dmitry Tantsur bca9ee7d3c Allow requesting fixed retry delay instead of exponential
Clients like ironicclient and swiftclient use fixed delay for their
build-in retry functionality. To replace it without changing behavior
we need a similar feature.

Change-Id: I1f9de98dae5719842f03d45e5a9d724199d5718b
2019-07-29 13:07:38 +02:00
Eric Fried df57e0ec3b Add a per-request global_request_id
Adapter.__init__ takes a global_request_id which causes the
X-Openstack-Request-Id header to be set on each request. This is fine if
the Adapter is used for only one "request" (in the sense of e.g. "a
server create" -- see [1]), but is too broad if the Adapter is reused
for multiple requests. For example, Nova's SchedulerReportClient (used
to communicate with Placement) creates a single instance of Adapter for
the life of the process [2][3][4]. Openstack SDK's Proxy objects [5]
endure for the life of a Connection.

So what is needed is a way to manage the X-Openstack-Request-Id header
on a per-request basis.

This commit adds a global_request_id kwarg to
keystoneauth1.session.Session.request, which is the funnel point for all
requests coming through Adapter as well as Session itself. (All the
methods feeding into that one already accept and pass through arbitrary
**kwargs.) If present, the value in the X-Openstack-Request-Id header is
set accordingly. Note that this will *override*
Adapter.global_request_id, which is exactly what we want, as described

[2] bea9058f02/nova/scheduler/client/ (L200)
[3] bea9058f02/nova/scheduler/client/ (L243)
[4] bea9058f02/nova/ (L1219-L1221)
[5] bf6651f149/openstack/ (L114)

Change-Id: Ied73320fcd813ae796e40cbdb30717900486b92c
2019-07-11 10:35:44 -05:00
Zuul aee0d8a130 Merge "Limit interval between retries to 1 minute" 2019-06-21 19:55:04 +00:00
Dmitry Tantsur 34c005ae5f Limit interval between retries to 1 minute
Currently it grows exponentially, exceeding 1 hour after 15 retries.
While we don't expect people to have so many retries, we should not
let them shoot their legs.

Change-Id: I01dfaa1c379340a0d41fcfdb07298fdef6110941
2019-06-19 15:28:35 +02:00
Dmitry Tantsur 92921c6016 Allow setting retry counts for Adapter via configuration options
Change-Id: I67ba69bfff69676ceb28b8a7515f10f5eff21c4c
2019-06-19 15:24:11 +02:00
Michael McCune 96559d6009 add a handler for unknown HTTP errors
This change adds logic to handle a situation where an error response has
been received by HTTP but its body schema is an unknown format.

This issue came up during a review of related changes:

Change-Id: I21a33052e951f515988fdfd8ab1f42440ca9d4f8
2019-06-04 16:03:33 -04:00
Michael McCune 01d2da9e47 add handling for multiple error returns
This change adds logic to the `exceptions.from_response` to handle
errors formatted in accordance with the API-SIG guidelines. When there
are multiple errors returned, only the first error will be included in
the exception with a note informing that there were more errors.

API SIG guideline:

email thread for content:

related neutron bug:

Change-Id: I1f06c2cd5c4e93e04582d4ffbb434db92010d712
2019-06-03 11:09:34 -04:00
Zuul 0828f7048e Merge "Add support for client-side rate limiting" 2019-03-01 01:58:36 +00:00
Monty Taylor 09934718f7 Add support for client-side rate limiting
shade/openstacksdk has implemented client-side rate limiting on top of
keystoneauth for ages and uses it extensively in nodepool. As part of an
effort to refactor that code a new approach was devised which was much
simpler and therfore suitable for inclusion in keystoneauth directly.

The underlying goal is two-fold, but fundamentally is about allowing a
user to add some settings so that they can avoid slamming their cloud.
First, allow a user to express that they never want to exceed a given
rate. Second, allow a user to limit the number of concurrent requests
allowed to be in flight.

The settings and logic are added to Adapter and not Session so that the
settings can easily be per-service. There is no need to block requests
to nova on a neutron rate limit, after all.

Co-Authored-By: Ian Wienand <>
Change-Id: Ic831e03a37d804f45b7ee58c87f92fa0f4411ad8
2019-02-28 22:14:24 +00:00
Monty Taylor 4960c48aec
Fix version discovery for clouds with int project_ids
On a cloud that has inaccessible version discovery documents AND uses
integer project ids, the discovery fallback logic can fail because the
project id parses as a (very large) version.

Check to see that the url segment in the fallback code begins with a v,
so that we're only attempting to parse versions from actual candidate

Closes-Bug: #1806109
Change-Id: Id90b3b9e4852494a4678b0a9bb67362babdc971c
2018-11-30 13:09:32 -06:00
Dmitry Tantsur 72288d3b18 Make new-style single endpoint version discovery actually work for ironic
For (unclear) historical reasons the root single version endpoint also
contains "id" and "links" fields. This makes the current workaround
for old-style endpoints take priority over the correct algorithm.
This change reorders the code, so that if "version" is present, it
always take priority over the workaround.

The unit tests are updated to be closer to real output from ironic.

Change-Id: I743b954c6c5b2f986c213acb6ec6af7e08c9f5f8
2018-10-23 14:26:34 +02:00
Monty Taylor 106d91fb41
Cache root urls with and without trailing slashes
The trailng slash on a pathless url is not meaningful, but we were
treating the url given to the discovery cache as if it were. In some
circumstances, such as an endpoint_override that didn't match the
found discovery document perfectly, a double-request could be made.
Normalize root urls in the caching code so that and would be the same.

Change-Id: I70a5911cf0f213a7816fe8d58c6cca4702ff71bb
2018-09-23 10:23:03 -05:00
Monty Taylor 2585047ffc
Protect against endpoint_data not existing
It's possible in get_api_major_version that the endpoint in question is
not found at all. In that case, we are documented to return None, but
what we do instead is throw an exception trying to get data off of the
None object.

Change-Id: I06ad497854f4e95a1a2a4a93241b244fc476b139
2018-09-11 16:18:40 -06:00
Monty Taylor c40eb2951d
Add support for ironic single-version responses
The ironic payload looks like:

  {'id': 'v1',
   'links': [{"href": "",
              "rel": "self"}]}

This does not have version info in it, nor min/max ranges for
microversion discovery. We can't really get any useful information from
this document, but we can at least not fail when trying to deal with it.
This should then be upwards-compatible with ironic adding version discovery
information to the document that is returned.

Change-Id: I47e0f9b295c24ef168f4a033faf573b953025d4c
2018-09-06 15:49:38 -05:00
Gage Hugo ccf6cb7903 Change log hashing to SHA256
With the recent Bandit update[0], the usage of SHA1 is now being
tagged as an issue. This changes the hashing of logs to SHA256
instead of SHA1.

Change-Id: Icde62b8d5ff78b4155e9df8231d63be3ecc53520
2018-08-17 12:35:47 -05:00
Zuul 171f6bd2e7 Merge "Add ability to filter version data by service-type" 2018-07-24 18:22:43 +00:00
Monty Taylor 83be7453fa
Add ability to filter version data by service-type
The get_all_version_data method is useful for getting a full listing of
what's going on with version discovery on a cloud. Sometimes though
people just want to see the versions for a specific service. Add a
filter to allow skipping making the version discovery call in the first
place, instead of needing to do that as a post-filtering step.

Change-Id: Ia3ca4be2976d1a5e7914fa8f2adbf7297e8cb1e1
2018-07-23 18:41:18 -05:00
wangxiyuan 323f4e4bc4 Add netloc and version check for version discovery
If the url netloc in the catalog and service's response
are not the same, we should choose the catalog's and
add the version info to it if needed.

Change-Id: If78d368bd505156a5416bb9cbfaf988204925c79
Closes-bug: #1733052
2018-07-19 10:18:44 +08:00
Eric Fried 51bfa030b1 raise_exc default in Adapter
It can be annoying to have to say raise_exc=False (or use try/except) on
every call when talking to an API where 4xx response codes are
useful/normal/informative or where the preferred coding style is to use
conditionals rather than try/except.

With this change, the Adapter constructor takes a new kwarg, raise_exc.
It defaults to None, and the existing behavior is unchanged.  If set to
a boolean value, that is used as the default for requests.  Specifying
raise_exc to the primitives (get, head, put, post, patch, delete,
request) at any point along the chain will still take precedence.

Change-Id: Ie291c3cb891467728d8ca33cf62afdab37c82f34
Closes-Bug: #1776501
2018-06-12 10:48:39 -05:00
Dmitry Tantsur 3c2cf44e1c Add optional support for retrying certain HTTP codes
Ironic commonly returns HTTP 409 when a node is locked by another routine
and HTTP 503 when the conductor has no free threads to process the request.
Currently it is managed by custom code in ironicclient and openstacksdk,
this change will allow to move it to Session itself.

Change-Id: I04e356e7856b020cd20aa598e291ef31e02730d2
2018-05-29 14:54:56 +02:00