Noticed this while doing some local testing, if a WSGI app replies with
a text/plain content type to communicate a server error, we aren't able
to see the error response message when passing --debug to the
RESP:  Date: Thu, 01 Oct 2020 23:54:15 GMT Server: Apache/2.4.18
(Ubuntu) Content-Type: text/plain; charset=UTF-8 Connection: close
RESP BODY: Omitted, Content-Type is set to text/plain; charset=UTF-8.
Only application/json responses have their bodies logged.
Manila API honors a "X-OpenStack-Manila-API-Version"
header to specify microversions.
It may support the OpenStack-API-Version header
in a future release, however, we'll need to maintain
backwards compatibility with the existing API.
In case of global-request-id request, Adapter
send two global request id header
This is becasue of the header not being Case Insensitive
and end up with two different name of same header with difference
of cap 'D'.
Unit test for whether request global-request-id has precedence
over adapter fail many times because of how different python version
treat the dict. py3.6 and above are all good as dict maintain the
insertion ordered but py3.5 can fail it any time.
We can see consistent failure in py35 jobs:
Let's make the headers always Case Insensitive which is
what RFC says.
Though we can now set ``connect_retires`` while creating an adapter object,
that would allow retries in case of connection timeout (ex. with session
clients derived from Adapater/LegacyJsonAdapater), it can't be used in
certain scenarios like endpoint discovery with auth plugin get_discovery()
or getting AccessInfo with get_access()/get_auth_ref().
Having ``connect_retries`` in Session constructor would allow users
with option of setting it when creating session objects (if they want)
and can be overridden per service with the adapter interface.
This commit also changes the default value of ``connect_retries`` from
0 to None to allow for adapter's to override retries on the session
If an external session object was not passed to the Session class, we
create a requests.Session() on our own. Once this is used, it may still
have an open connection when the auth Session is closed. We need to
handle the closing of the requests.Session() ourselves if we created
one. If you do not close it, a ResourceWarning may be reported about the
socket that is left open. If a session object is provided, we do not
attempt to close it as it will be up to the code consuming keystoneauth
to properly handle cleaning up the provided session.
Clients like ironicclient and swiftclient use fixed delay for their
build-in retry functionality. To replace it without changing behavior
we need a similar feature.
Adapter.__init__ takes a global_request_id which causes the
X-Openstack-Request-Id header to be set on each request. This is fine if
the Adapter is used for only one "request" (in the sense of e.g. "a
server create" -- see ), but is too broad if the Adapter is reused
for multiple requests. For example, Nova's SchedulerReportClient (used
to communicate with Placement) creates a single instance of Adapter for
the life of the process . Openstack SDK's Proxy objects 
endure for the life of a Connection.
So what is needed is a way to manage the X-Openstack-Request-Id header
on a per-request basis.
This commit adds a global_request_id kwarg to
keystoneauth1.session.Session.request, which is the funnel point for all
requests coming through Adapter as well as Session itself. (All the
methods feeding into that one already accept and pass through arbitrary
**kwargs.) If present, the value in the X-Openstack-Request-Id header is
set accordingly. Note that this will *override*
Adapter.global_request_id, which is exactly what we want, as described
 bea9058f02/nova/scheduler/client/report.py (L200)
 bea9058f02/nova/scheduler/client/report.py (L243)
 bea9058f02/nova/utils.py (L1219-L1221)
 bf6651f149/openstack/proxy.py (L114)
Currently it grows exponentially, exceeding 1 hour after 15 retries.
While we don't expect people to have so many retries, we should not
let them shoot their legs.
shade/openstacksdk has implemented client-side rate limiting on top of
keystoneauth for ages and uses it extensively in nodepool. As part of an
effort to refactor that code a new approach was devised which was much
simpler and therfore suitable for inclusion in keystoneauth directly.
The underlying goal is two-fold, but fundamentally is about allowing a
user to add some settings so that they can avoid slamming their cloud.
First, allow a user to express that they never want to exceed a given
rate. Second, allow a user to limit the number of concurrent requests
allowed to be in flight.
The settings and logic are added to Adapter and not Session so that the
settings can easily be per-service. There is no need to block requests
to nova on a neutron rate limit, after all.
Co-Authored-By: Ian Wienand <firstname.lastname@example.org>
The get_all_version_data method is useful for getting a full listing of
what's going on with version discovery on a cloud. Sometimes though
people just want to see the versions for a specific service. Add a
filter to allow skipping making the version discovery call in the first
place, instead of needing to do that as a post-filtering step.
Ironic commonly returns HTTP 409 when a node is locked by another routine
and HTTP 503 when the conductor has no free threads to process the request.
Currently it is managed by custom code in ironicclient and openstacksdk,
this change will allow to move it to Session itself.
python-openstackclient does this in a wrapper class around Session,
and openstacksdk does something similar that could be removed if support
were directly in keystoneauth.
Add this so that we can remove the custom wrapper/manipulation in
openstackclient and openstacksdk.
A change introduced in 3.5.0 sorts headers, but runs into a problem
when the headers are bytes, such as the headers provided by the
requests expects headers to be str type in both python2 and python3.
This means in python2 we need to encode unicode objects as ASCII (the
encoding that should be used for HTTP headers) and in python3 we need to
decode bytes as ASCII into str.
The new get_all_version_data call lists 'public' as the default value
for interface, but had None in the arguments. 'public' was the intent,
and is what the similar call on the base auth plugin does.
We're repeating ourselves a bunch with a plain dict that contains the
version data. Make a class to encapsulate it. Make the class a subclass
of dict so that json translation works.
We've got great discovery support, but if someone is wanting to find out
what is available and doesn't otherwise know what they're looking for,
they're out of luck.
Add a method to EndpointData which will return all of the version data
for a given service, and then add a method to the base auth plugin that
will use that method to collect all of the version discovery documents
for every service in the cloud.
This commit adds os-service-types so that the resulting datastructure
can return only official service type keys. A followup patch will also
use os-service-types to allow catalog lookups by service-type alias.
There is a change to the test_identity_common.V2.get_auth_data method
to remove the public and internal urls for keystone from the catalog.
The V3 catalog only has keystone on admin, so this makes them have
Python logging is pretty amazingly flexible, and allows us to emit
to arbitrary logging domains so that a consumer can direct log output
Turning on HTTP debug logging currently produces an avalanche of output,
when sometimes just seeing that the requests were made and responded to
is perfectly fine.
Split the loggers used into four - one for request ids, one for request
commands, one for response headers and one for response body content.
Make them subloggers of keystoneauth.session so that if a user does nothing,
their existing logging config will be unchanged.
If someone passes in a logger, behave as before logging all things to
the provided logger.
While we're at it, document this in the using-sessions document, so that
people know that the loggers exist and what they do.
The tox (>=1.7.0) by default sets a random python hash seed which
causes ordering of dicts and sets to be different between tests runs.
Disabled the random python hash seed by setting PYTHONHASHSEED=0 to
fix the random failure of below test:
The PYTHONHASHSEED=0 is removed in the followup patch so that we can
separate the tracking down of ordering issues in tests from this patch.
The additional_user_agent is intended for things, such as
os-client-config, that sit somewhere in the stack between keystoneauth
and the actual client. So according to jamielennox in irc, it should be:
app client *additional_user_agent keystoneauth requests
The current logic puts additional_user_agent between app and client,
leading to things that look like this:
"User-Agent: v.py/1 os-client-config/1.26.1 shade/1.19.1
keystoneauth1/2.18.0 python-requests/2.13.0 CPython/2.7.12"
Which is a bit off.
Similar to get_endpoint, which knows it doesn't need full endpoint_data,
if a user just wants to know what major version the discovery process
wound up with, there are cases in which we do not need to fetch
discovery documents. Provide an API call that a user can use when this
is the information they need to avoid them having to play games with
The positional decorator results in poorly maintainable code in
a misguided effort to emulate python3's key-word-arg only notation
and functionality. This patch removes keysteonauth's dependance
on the positional decorator.
We're discouraging the use of the ambiguous and difficult-to-understand
'version' parameter in new discovery methods, instead encouraging the
use of min_version and max_version.
In order to make it possible to get the same functionality, though, we
need a way to say the same thing as version="M.m", which actually means,
"min version is M.m, and max version is the latest within major version
Introducing 'latest' syntax, which can be used in various ways,
...which is equivalent to the old school version='2.3'
The user now has the ability to know what microversions are available,
but needs to be able to send a microversion header with their request.
Add a microversion parameter to Session that will construct and send the
header. The microversion header requires a service_type. One should be
available but it's possible for it to be missing if someone is using an
endpoint_override. Provide a parameter to let the user specify a
service_type for the microversion call in such cases.
There are a two interrelated pieces in this patch which are around
fixing up places where discovery was being re-run inappropriately.
They fall out from adding tests for the functionality and couldn't
be sanely shifted back further in the stack without a big dance.
Switch the default for "discover_versions" on all of the calls that
return an EndpointData to "True". It's a new feature and is a thing that
doesn't make a ton of sense to call if you don't want discovery run.
However, get_endpoint uses it, so needs to be able to pass in
discover_version=False, so the option is still useful. Make sure that
get_endpoint and other places where ksa calls get_endpoint_data on
behalf of the user work as before without unneeded discovery.
Add tests to show that we use actually use the discovery cache properly when
we've previously done discovery that can satisfy the new request. This
works from the microversion optimization patch, but we had to clean up
a couple of things to show it fully in a test.
If a user has provided an endpoint_override, they may still be
interested in version discovery data for the endpoint. Doing that is
always an opt-in behavior, so we set the strictness flag to prevent any
URL manipulations. We'll either return data or None.
All of this "get endpoint data" stuff is great, but it's no good if the
user can't ask "hey - what are we working with here". It's safe enough
to not cache this data on the session or adapter objects, because the
source data is all cached anyway. That way calling the method with
different filters will always return the correct data.
Allow the user to pass in a cache dict that will be used
in addition to the session and auth level caches. Make Session
always have a discovery_cache attribute and allow the user to
provide the cache at Session creation time. Finally, rename
the private variable to _discovery_cache from _endpoint_cache
since it's caching discovery objects, not endpoints.
Co-Authored-By: Samuel de Medeiros Queiroz <email@example.com>
A response with header Content-Type set to "application/json; charset=UTF-8"
would be omitted but not correctly logged. This patch set correctly omits
and logs a response with the mentioned header.
Co-Authored-By: Tin Lam <firstname.lastname@example.org>
When whitelisting content types to debug print from session we chose
application/json and application/text. application/text is not a real
mime type, text is typically text/plain.
Rather than guess at mime types only print application/json to start
with, but make it easy for additional types to be added later.
Currently, logs display the hash values of X-Auth-Token,
Authorization, and X-Subject-Token, but not the value of
the X-Service-Token. This patch set adds the X-Service-Token
to the list of header fields to be hashed for logging purposes.
Added support to log 'X-Openstack-Request-Id' or
'X-Compute-Request-Id' in case of Nova for each api call.
If any python-*client which is using session is used from
command line then following log will be logged on console
if --debug flag is used.
DEBUG (session:350) GET call to compute for
request id req-c54b8f3e-a7e4-4085-a5e3-fd5244ef3ce5
If any python-*client which is using session is used in
applications (e.g. Nova) then following log message will be
logged in service logs.
[req-a6929d46-765c-44a9-8370-49ff0f1958ca admin admin] GET call
to network for
http://10.232.48.201:9696/v2.0/security-groups.json?id=040cc729-9086-4f41-8977-acb4ef71c7de used request id req-de6bfe07-22ac-4940-b65e-367cb0e8102d
To use this feature user need to set 'default_log_levels' in third
party application. For example in nova.conf 'default_log_levels'
should be set as below:
default_log_levels = keystoneauth1=DEBUG
Response bodies are loaded into memory prior to
Loading huge response bodies may result in a
This patch proposes that only JSON and TEXT
responses be logged, i.e when the Content-Type
header is application/json or application/text.
Responses that do not include or have a different
Content-Type header will have their body omitted.