An earlier change added support for versionless authurls, but the
huristic to detect them didn't work for some configurations I've
encountered.
Now we use a little bit tighter pattern matching and support auth_url
values with more than one path component.
Change-Id: I5a99c7b4e957ee7c8a5b5470477db49ab2ddba4b
Related-Change-Id: If7ecb67776cb77828f93ad8278cc5040015216b7
Add the --prompt option for the CLI which will cause the user to be
prompted to enter a password. Any password otherwise specified by
--key, --os-password or an environment variable will be ignored.
The swift client will exit with a warning if the password cannot be
entered without its value being echoed.
Closes-Bug: #1357562
Change-Id: I513647eed460007617f129691069c6fb1bfe62d7
openstackdocstheme is a theme and extension support for
Sphinx documentation that is published to docs.openstack.org
and developer.openstack.org.
Change-Id: I37d1d50fb88b35e72b017d5dfbf148c35ac7e323
The valid set of values for auth_version does not include
values starting with the 'v'.
In this particular function, the auth_version variable is
only used for comparisons with v3. So, the code worked
correctly. However, let's clean this up in order to reduce
review confusion and defuse possible future landmine in case
of code changes.
Change-Id: I671016d7992a1922b786b4eb8876b3fbb2532e15
Create a tox environment for running the unit tests against the lower
bounds of the dependencies.
Create a lower-constraints.txt to be used to enforce the lower bounds
in those tests.
Add openstack-tox-lower-constraints job to the zuul configuration.
Update the dependencies needed to make the unit tests pass while
constrained to the lower bounds.
See http://lists.openstack.org/pipermail/openstack-dev/2018-March/128352.html
for more details.
Co-Authored-By: Nguyen Hai <nguyentrihai93@gmail.com>
Change-Id: I2a8f465c8b08370517cbec857933b08fca94ca38
Depends-On: https://review.openstack.org/555034
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
ading multiple options on the same line makes
it easy to miss when quickly scanning the options.
Change-Id: I8e324fca48cd05d9e381d5106135542274c2ff7f
Signed-off-by: Thiago da Silva <thiago@redhat.com>
This patch attemps to add an option to force get_auth call while retrying
an operation even if it gets errors other than 401 Unauthorized.
Why we need this:
The main reason why we need this is current python-swiftclient requests could
never get succeeded under certion situation using third party proxies/load balancers
between the client and swift-proxy server. I think, it would be general situation
of the use case.
Specifically describing nginx case, the nginx can close the socket from the client
when the response code from swift is not 2xx series. In default, nginx can wait the
buffers from the client for a while (default 30s)[1] but after the time past, nginx
will close the socket immediately. Unfortunately, if python-swiftclient has still been
sending the data into the socket, python-swiftclient will get socket error (EPIPE,
BrokenPipe). From the swiftclient perspective, this is absolutely not an auth error,
so current python-swiftclient will continue to retry without re-auth.
However, if the root cause is sort of 401 (i.e. nginx got 401 unauthorized from the
swift-proxy because of token expiration), swiftclient will loop 401 -> EPIPE -> 401...
until it consume the max retry times.
In particlar, less time to live of the token and multipart object upload with large
segments could not get succeeded as below:
Connection Model:
python-swiftclient -> nginx -> swift-proxy -> swift-backend
Case: Try to create slo with large segments and the auth token expired with 1 hour
1. client create a connection to nginx with successful response from swift-proxy and its auth
2. client continue to put large segment objects
(e.g. 1~5GB for each and the total would 20~30GB, i.e. 20~30 segments)
3. after some of segments uploaded, 1 hour past but client is still trying to
send remaining segment objects.
4. nginx got 401 from swift-proxy for a request and wait that the connection is closed
from the client but timeout past because the python-swiftclient is still sending much data
into the socket before reading the 401 response.
5. client got socket error because nginx closed the connection during sending the buffer.
6. client retries a new connection to nginx without re-auth...
<loop 4-6>
7. finally python-swiftclient failed with socket error (Broken Pipe)
In operational perspective, setting longer timeout for lingering close would be an option but
it's not complete solution because any other proxy/LB may not support the options.
If we actually do THE RIGHT THING in python-swiftclient, we should send expects: 100-continue
header and handle the first response to re-auth correctly.
HOWEVER, the current python's httplib and requests module used by python-swiftclient doesn't
support expects: 100-continue header [2] and the thread proposed a fix [3] is not super active.
And we know the reason we depends on the library is to fix a security issue that existed
in older python-swiftclient [4] so that we should touch around it super carefully.
In the reality, as the hot fix, this patch try to mitigate the unfortunate situation
described above WITHOUT 100-continue fix, just users can force to re-auth when any errors
occurred during the retries that can be accepted in the upstream.
1: http://nginx.org/en/docs/http/ngx_http_core_module.html#lingering_close
2: https://github.com/requests/requests/issues/713
3: https://bugs.python.org/issue1346874
4: https://review.openstack.org/#/c/69187/
Change-Id: I3470b56e3f9cf9cdb8c2fc2a94b2c551927a3440
Submitting a path parameter with a HEAD request on an object can be
useful if one is trying to find out information about an SLO/DLO without
retrieving the manifest.
Change-Id: I39efd098e72bd31de271ac51d4d75381929c9638
When uploading from standard input, swiftclient should turn the upload
into an SLO in the case of large objects. This patch picks the
threshold as 10MB (and uses that as the default segment size). The
consumers can also supply the --segment-size option to alter that
threshold and the SLO segment size. The patch does buffer one segment
in memory (which is why 10MB default was chosen).
(test is updated)
Change-Id: Ib13e0b687bc85930c29fe9f151cf96bc53b2e594
I noticed a disturbing lack of quote-wrapping in change
I7cb4b44952713752435e1faf0f63bf0d37e7dda6 but as I poked at it, I
realized that trouble runs rampant.
This seems to clean it all up, though I haven't tested *every*
environment we define.
Change-Id: I1454eb113e5bd9125d39f2e57e2ed96f6ddc42fc
The updates to the sphinx docs jobs in support of the updates to
the PTI wound up exposing an unintended interface. There are two flavors
of the tox_install.sh file out there, and we basically need to collapse
them into one flavor.
Update the tox_install.sh script to match the
constraints-as-first-argument form.
Change-Id: I7cb4b44952713752435e1faf0f63bf0d37e7dda6
Release notes are version independent, so remove version/release
values. We've found that projects now require the service package
to be installed in order to build release notes, and this is entirely
due to the current convention of pulling in the version information.
Release notes should not need installation in order to build, so this
unnecessary version setting needs to be removed.
This is needed for new release notes publishing, see
I56909152975f731a9d2c21b2825b972195e48ee8 and the discussion starting
at
http://lists.openstack.org/pipermail/openstack-dev/2017-November/124480.html
.
Change-Id: I623fe918c1e4ddafa93efc91ed550a365cec1cf0
Also, don't try to do any pypy stuff on Fedora -- apparently that's only
available from the Everything repo instead of the stripped-down Server
one the gate images use? I only wanted Fedora for py36 testing, anyway.
Change-Id: Iba8142e4e1093cf7f7a9dcf782288364d43cb64d
After this, we need to
* add release notes jobs for python-swiftclient in openstack-infra/project-config
* add release notes links for python-swiftclient in openstack/releases
For the corresponding change in the swift repo, see
I4e5f1ce1fcfbb2943036c821a24a0b4a3a2d9fc8
Change-Id: Iea6ed2ee26873edb3ef10146cdc906cf1a236255
If "-" is passed in for the source, python-swiftclient will upload
the object by reading the contents of the standard input. The object
name option must be set, as well, and this cannot be used in
conjunction with other files.
This approach stores the entire contents as one object. A follow on
patch will change this behavior to upload from standard input as SLO,
unless the segment size is larger than the content size.
Change-Id: I1a8be6377de06f702e0f336a5a593408ed49be02
Currently, the swiftclient upload command passes a custom metadata
header for each object (called object-meta-mtime), whose value is
the current UNIX timestamp. When downloading such an object with the
swiftclient, the mtime header is parsed and passed as the atime and
mtime for the newly created file.
There are use-cases where this is not desired, for example when using
tmp or scratch directories in which files older than a specific date
are deleted. This commit provides a boolean option for ignoring the
mtime header.
Change-Id: If60b389aa910c6f1969b999b5d3b6d0940375686
