This patch adds a new object versioning mode. This new mode provides
a new set of APIs for users to interact with older versions of an
object. It also changes the naming scheme of older versions and adds
a version-id to each object.
This new mode is not backwards compatible or interchangeable with the
other two modes (i.e., stack and history), especially due to the changes
in the namimg scheme of older versions. This new mode will also serve
as a foundation for adding S3 versioning compatibility in the s3api
middleware.
Note that this does not (yet) support using a versioned container as
a source in container-sync. Container sync should be enhanced to sync
previous versions of objects.
Change-Id: Ic7d39ba425ca324eeb4543a2ce8d03428e2225a1
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Thiago da Silva <thiagodasilva@gmail.com>
... just like they always have and server per port strategy still does.
Related-Change-Id: I3e5229d2fb04be67e53533ff65b0870038accbb7
Change-Id: I14e3ed201ceaceef0f8dbc44685395f350a0e7fc
Swift servers can now be seamlessly reloaded by sending them a SIGUSR1
(instead of a SIGHUP). The server forks off a synchronized child to
wait to close the old listen socket(s) until the new server has started
up and bound its listen socket(s). The new server is exec'ed from the
old one so its PID doesn't change. This makes Systemd happier, so a
ReloadExec= stanza can now be used.
The seamless part means that incoming connections will alwyas get
accepted either by the old server or the new one. This eliminates
client-perceived "downtime" during server reloads, while allowing the
server to fully reload, re-reading configuration, becoming a fresh
Python interpreter instance, etc. The SO_REUSEPORT socket option has
already been getting used, so nothing had to change there.
This patch also includes a non-invasive fix for a current eventlet bug;
see https://github.com/eventlet/eventlet/pull/590
That bug prevents a SIGHUP "reload" from properly servicing existing
requests before old worker processes close sockets and exit. The
existing probtests missed this, but the new ones, in this patch, caught
it.
New probe tests cover both old SIGHUP "reload" behavior as well as the
new SIGUSR1 seamless reload behavior.
Change-Id: I3e5229d2fb04be67e53533ff65b0870038accbb7
Previously, configs like
fallocate_reserve = 1%
would cause a py3 backend server to fail to start, complaining like
configparser.InterpolationSyntaxError: Error in file
/etc/swift/object-server/1.conf.d: '%' must be followed
by '%' or '(', found: '%'
This could also come up in proxy-server configs, with things like
percent signs in tempauth password.
In general, we haven't really thought much about interpolation in
configs. Python's default ConfigParser has always supported it, though,
so we got it "for free". On py2, we didn't really have to think about
it, since values like "1%" would pass through just fine. (It would blow
up a SafeConfigParser, but a normal ConfigParser only does replacements
when there's something like a "%(opt)s" in the value.)
On py3, SafeConfigParser became ConfigParser, and the old interpolation
mode (AFAICT) doesn't exist.
Unfortunatley, since we "supported" interpolation, we have to assume
there are deployments in the wild that use it, and try not to break
them. So, do what we can to mimic the py2 behavior.
Change-Id: I0f9cecd11f00b522a8486972551cb30af151ce32
Closes-Bug: #1844368
Otherwise, paste complains about not being able to interpolate values
into the replication_semaphore. As long as it gets dropped in before we
fork(), I think it's OK?
Closes-Bug: 1691075
Change-Id: Ib7e065c47871876786bcc9ff39737f5d1bb3c12c
There's a bug in CPython [1] that causes servers to mis-parse request
lines that include the bytes \x85 or \xa0. Naturally, we have
functional tests that (with high probability) will send such request
lines. There's a fix proposed, but the earliest it's likely to land
would be for 3.8, and we need to be able to target an earlier Python.
So, intercept the request line immediately before parsing and re-write
it to be RFC-compliant. Note that this is done for Python 2 as well,
though there should be no change in the request environment that
eventlet eventually hands to us.
[1] https://bugs.python.org/issue33973
Change-Id: Ie648f5c04d4415f3b620fb196fa567ce7575d522
You could *try* doing something similar to what we were doing
there over in email.message for py3, but you would end up
breaking pkg_resources (and therefor entrypoints) in the
process.
Drive-by: have mem_diskfile implement more of the diskfile API.
Change-Id: I1ece4b4500ce37408799ee634ed6d7832fb7b721
Previously, we'd sometimes shove strings into HASH_PATH_PREFIX or
HASH_PATH_SUFFIX, which would blow up on py3. Now, always use bytes.
Change-Id: Icab9981e8920da505c2395eb040f8261f2da6d2e
User can cofigure KEEPIDLE time for sockets in TCP connection.
The default value is the old value which is 600.
Change-Id: Ib7fb166deb8a87ae4e97ba0671048b1ec079a2ef
Closes-Bug:1759606
This patch adds an additional optional parameter to tempurl
which restricts the ip's from which a temp url can be used from.
Change-Id: I23fe998a980960d4a32df042b3f6a21f096c36af
eventlet<=0.17.4 LoggerFileWrapper does not have an error() method
so don't try to call it. Instead, use info() which even in newer
eventlet versions does exactly the same as error().
This bug only manifests in unit tests once the Related-Bug has been fixed
(see Related-Change).
Closes-Bug: #1777836
Related-Bug: #1777701
Related-Change: I68fd9d0263a6720aaf0d90b8ea6da1158105ac01
Change-Id: I46190567549826c811ffa51e9a71a38a20d9ce97
Older versions of eventlet expect getsocketname() to return a tuple,
and newer versions are happy to receive a tuple, so fix the mock to
return a tuple.
Change-Id: I68fd9d0263a6720aaf0d90b8ea6da1158105ac01
Closes-Bug: #1777701
The mock call got pulled out in the related change.
Change-Id: Ife1e452fa34026ac934e88a05708cdc5978215e1
Related-Change-Id: Id089e29e7ecfc8cec79c520f604aa01bdae0dcf0
For a while, this was conditional because we supported old Eventlet
versions that didn't have this keyword arg. Now, we require new-enough
Eventlet that it's always available, so let's get rid of the
conditional crud.
The flag was introduced in Eventlet 0.15, and we require >= 0.17.4.
Change-Id: Id089e29e7ecfc8cec79c520f604aa01bdae0dcf0
...to the proxy-server.
The point is to allow the Swift proxy server to log accurate
client IP addresses when there is a proxy or SSL-terminator between the
client and the Swift proxy server. Example servers supporting this
PROXY protocol:
stud (v1 only)
stunnel
haproxy
hitch (v2 only)
varnish
See http://www.haproxy.org/download/1.7/doc/proxy-protocol.txt
The feature is enabled by adding this to your proxy config file:
[app:proxy-server]
use = egg:swift#proxy
...
require_proxy_protocol = true
The protocol specification states:
The receiver MUST be configured to only receive the protocol
described in this specification and MUST not try to guess
whether the protocol header is present or not.
so valid deployments are:
1) require_proxy_protocol = false (or missing; default is false)
and NOT behind a proxy that adds or proxies existing PROXY lines.
2) require_proxy_protocol = true
and IS behind a proxy that adds or proxies existing PROXY lines.
Specifically, in the default configuration, one cannot send the swift
proxy PROXY lines (no change from before this patch). When this
feature is enabled, one _must_ send PROXY lines.
Change-Id: Icb88902f0a89b8d980c860be032d5e822845d03a
...in preparation for the container sharding feature.
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I4455677abb114a645cff93cd41b394d227e805de
This includes every pipeline from etc/proxy-server.conf-sample since we
switched from swauth to tempauth in 1.4.1. As much as anything, I view
this as a canary for auto-insertion changes, so we can (somewhat easily)
see where new middlewares will be placed when upgrading from old swifts.
Change-Id: I117937ab6ce28f3bc219e809f07b563c45fc486f
Related-Change: I88678fddc7a25b0f065b33eb26047321d0db4d38
There were two middlewares using a common pattern to load
the proxy-server app config section. The existing pattern
fails to recognise option overrides that are declared using
paste-deploy's 'set' notation, as illustrated by the change
to test_dlo.py in this patch.
This patch replaces the existing code with a helper function
that loads the proxy-server config using the paste-deploy loader.
The resulting config dict is therefore exactly the same as that
used to initialise the proxy-server app.
Change-Id: Ib58ce03e2010f41e7eb11f1a6dc78b0b7f55d466
Since Python 2.7, unittest in the standard library has included mulitple
facilities for skipping tests by decorators as well as an exception.
Switch to that directly, rather than importing nose.
Change-Id: I4009033473ea24f0d0faed3670db844f40051f30
Make some json -> (text, xml) stuff in a common module, reference that in
account/container servers so we don't break existing clients (including
out-of-date proxies), but have the proxy controllers always force a json
listing.
This simplifies operations on listings (such as the ones already happening in
decrypter, or the ones planned for symlink and sharding) by only needing to
consider a single response type.
There is a downside of larger backend requests for text/plain listings, but
it seems like a net win?
Change-Id: Id3ce37aa0402e2d8dd5784ce329d7cb4fbaf700d
The "method" `bind_ports` only gets called once on startup - and *during
it's execution* it gets mutated to a list inside `_reload_bind_ports` -
afterwards all references are only to the attribute `bind_ports`.
It 100% works, but at first glance seems dubious. Renaming the method
do_bind_ports seems more obvious.
Change-Id: Ic01621e478f1140644e26ba41715d93963a00945
Use assertIsNone() instead of assertEqual(), because assertEqual()
still fails on false values when compared to None
Change-Id: Ic52c319e3e55135df834fdf857982e1721bc44bb
Following OpenStack Style Guidelines:
[1] http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises
[H203] Unit test assertions tend to give better messages for more specific
assertions. As a result, assertIsNone(...) is preferred over
assertEqual(None, ...) and assertIs(..., None)
Change-Id: If4db8872c4f5705c1fff017c4891626e9ce4d1e4
With the new default behavior of eventlet 0.20 to set SO_REUSEPORT on listening
sockets, it becomes easy to have a zero-downtime reload.
By binding the listening socket as late as we can, we can reduce the
time an incoming connection will stay in queue, waiting for the process
to be ready to handle the connection.
This commit moves the socket binding after the load of the application.
On some benchmarks, it reduced the wait-time for incoming connections from
about 1.1s to about 350ms.
Change-Id: I045f4c9aa2c07cdc5ed95afd028540f1b3874637
This is an alternative approach to that proposed in [1]
Adds support for optional per-policy config sections
to be added in proxy-server.conf. This is highly desirable
to allow per-policy affinity options to be set for use with
duplicated EC policies [2] and composite rings [3].
Certain options found in per-policy conf sections will
override their equivalents that may be set in the
[app:proxy-server] section. Currently the options
handled that way are:
sorting_method
read_affinity
write_affinity
write_affinity_node_count
For example:
[proxy-server:policy:0]
sorting_method = affinity
read_affinity = r1=100
write_affinity = r1
write_affinity_node_count = 1 * replicas
The corresponding attributes of the proxy-server Application
are now available from instances of an OverrideConf object
that is obtained from Application.get_policy_options(policy).
[1] Related-Change: I9104fc789ba85ab3ab5ccd34096125b482821389
[2] Related-Change: Idd155401982a2c48110c30b480966a863f6bd305
[3] Related-Change: I0d8928b55020592f8e75321d1f7678688301d797
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Change-Id: I3f718f425f525baa80045ba067950c752bcaaefc
Recently out gate started blowing up intermittently with a strange
case of ports mixed up. Sometimes a functional tests tries to
authorize on a port that's clearly an object server port, and
the like. As it turns out, eventlet developers added an unavoidable
SO_REUSEPORT into listen(), which makes listen(("localhost",0)
to reuse ports.
There's an issue about it:
https://github.com/eventlet/eventlet/issues/411
This patch is working around the problem while eventlet people
consider the issue.
Change-Id: I67522909f96495a6a30e1acdb79835dce2189549
This patch moves some code from the crypto files
to a more common modules that will be used by symlinks
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I1758693c5dd428f9f2157966aac49d97c2c7ab12
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Previously, we would set the TZ environment variable to the result of
time.strftime("%z", time.gmtime())
This has a few problems.
1. The "%z" format does not appear in the table of formatting
directives for strftime [1]. While it *does* appear in a
footnote [2] for that section, it is described as "not supported by
all ANSI C libraries." This may explain the next point.
2. On the handful of Linux platforms I've tested, the above produces
"+0000" regardless of the system's timezone. This seems to run
counter to the intent of the patches that introduced the TZ
mangling. (See the first two related changes.)
3. The above does not produce a valid Posix TZ format, which expects
(at minimum) a name consisting of three or more alphabetic
characters followed by the offset to be added to the local time to
get Coordinated Universal Time (UTC).
Further, while we would change os.environ['TZ'], we would *not* call
time.tzset like it says in the docs [3], which seems like a Bad Thing.
Some combination of the above has the net effect of changing some of the
functions in the time module to use UTC. (Maybe all of them? At the very
least, time.localtime and time.mktime.) However, it does *not* change
the offset stored in time.timezone, which causes bad behavior when
dealing with local timestamps [4].
Now, set TZ to "UTC+0" and call tzset. Apparently we don't have a good
way of getting local timezone info, we were (unintentionally?) using UTC
before, and you should probably be running your servers in UTC anyway.
[1] https://docs.python.org/2/library/time.html#time.strftime
[2] https://docs.python.org/2/library/time.html#id2
[3] https://docs.python.org/2/library/time.html#time.tzset
[4] Like in email.utils.mktime_tz, prior to being fixed in
https://hg.python.org/cpython/rev/a283563c8cc4
Change-Id: I007425301914144e228b9cfece5533443e851b6e
Related-Change: Ifc78236a99ed193a42389e383d062b38f57a5a31
Related-Change: I8ec80202789707f723abfe93ccc9cf1e677e4dc6
Related-Change: Iee7488d03ab404072d3d0c1a262f004bb0f2da26
We're functioning as a WSGI server here, so this bit from PEP-3333 seems
to apply:
> The start_response callable must not actually transmit the response
> headers. Instead, it must store them for the server or gateway to
> transmit only after the first iteration of the application return
> value that yields a non-empty bytestrin ... . In other words, response
> headers must not be sent until there is actual body data available, or
> until the application's returned iterable is exhausted.
Plus, it mirrors what swob.Request.call_application does.
Change-Id: I1e8501f8ce91ea912780db64fee1c56bef809a98
The mimetools module has been removed from Python 3: modify
monkey_patch_mimetools() to do nothing on Python 3.
Skip test_monkey_patch_mimetools() on Python 3.
Change-Id: I50f01ec159efedbb4df759ddd1e13928ac28fba6
Previously, if you called get_account_info, get_container_info, or
get_object_info, then the results of that call would be cached in the
WSGI environment as top-level keys. This is okay, except that if you,
in middleware, copy the WSGI environment and then make a subrequest
using the copy, information retrieved in the subrequest is cached
only in the copy and not in the original. This can mean lots of extra
trips to memcache for, say, SLO validation where the segments are in
another container; the object HEAD ends up getting container info for
the segment container, but then the next object HEAD gets it again.
This commit moves the cache for get_*_info into a dictionary at
environ['swift.infocache']; this way, you can shallow-copy the request
environment and still get the benefits from the cache.
Change-Id: I3481b38b41c33cd1e39e19baab56193c5f9bf6ac
Rewrite server side copy and 'object post as copy' feature as middleware to
simplify the PUT method in the object controller code. COPY is no longer
a verb implemented as public method in Proxy application.
The server side copy middleware is inserted to the left of dlo, slo and
versioned_writes middlewares in the proxy server pipeline. As a result,
dlo and slo copy_hooks are no longer required. SLO manifests are now
validated when copied so when copying a manifest to another account the
referenced segments must be readable in that account for the manifest
copy to succeed (previously this validation was not made, meaning the
manifest was copied but could be unusable if the segments were not
readable).
With this change, there should be no change in functionality or existing
behavior. This is asserted with (almost) no changes required to existing
functional tests.
Some notes (for operators):
* Middleware required to be auto-inserted before slo and dlo and
versioned_writes
* Turning off server side copy is not configurable.
* object_post_as_copy is no longer a configurable option of proxy server
but of this middleware. However, for smooth upgrade, config option set
in proxy server app is also read.
DocImpact: Introducing server side copy as middleware
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Change-Id: Ic96a92e938589a2f6add35a40741fd062f1c29eb
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Oslo.messaging pika driver requires patching of select module if thread
is patched.
Pika driver uses select call and if it is not patched onsuming messages
blocks whole eventlet loop
Closes-Bug: #1570242
Change-Id: I9756737309f401ebddb7475eb84725f65bca01bf
As swift no longer supports Python 2.6, replace assertEqual(None, *)
with assertIsNone in tests to have more clear messages in case of
failure.
Change-Id: I94af3e8156ef40465d4f7a2cb79fb99fc7bbda56
Closes-Bug: #1280522
If not explicitly configured the versioned_writes middleware
should be auto-inserted in the pipeline after slo and dlo, which
is where the versioned_writes filter section's comments say it
should be in proxy-server.conf-sample. At the moment it can end up
being placed ahead of slo and dlo if they have been explicitly
configured, which results in the linked bug manifesting.
Closes-Bug: #1537042
Change-Id: I6ac95a331f4ef0d4887311940acc6f8bc00fb4eb
Currently a HTTP_REFERER (Referer) header isn't passed down to
subrequests. This means *LO subrequests to segment containers
return a 403 on a *LO GET when accessed by requests using referer
ACLs.
Currently the only way around referer access to *LO's is to make the
segments container world readable.
This change makes sure the referer header is passed into subrequests
allowing a segments container to only need to be locked down with
the same referer as the *LO container.
This is a 1 line change to code, but also adds a unit and 2 functional
functional tests (one for DLO and one for SLO).
Change-Id: I1fa5328979302d9c8133aa739787c8dae6084f54
Closes-Bug: #1526575
keystoneclient uses threading.Lock(), but swift doesn't
monkeypatch threading, this result in lockup when two
greenthreads try to acquire a non green lock.
This change fixes that.
Change-Id: I9b44284a5eb598a6978364819f253e031f4eaeef
Closes-bug: #1508424
