setgid provides the primary group, setgroups sets the secondary
groups. Prior to this patch, we would do a setgroups with an empty
list, effectively wiping secondary groups. We now verify which
secondary groups the user is member of and escalate the privileges
accordingly.
Change-Id: I33a10edd448b3ac5aa758a8d1d70e582cf421c7d
Closes-Bug: 1269473
The changes from using os.path.ismount to using
swift.common.utils.ismount has caused problems since the new one
raises exceptions in cases where the old one did not. Daemons have
been encountering this and exiting; servers have been 500ing instead
of 507ing in this case, changing handoff behaviors, etc.
Since the new one was specifically written and tested for this new
behavior, I left that original function as ismount_raw and made
ismount do what it did before.
If there really isn't some reason for this new behavior, I'll be glad
to get rid of ismount_raw and just keep ismount. I couldn't see any
reason for the new behavior myself.
Change-Id: I2b5b17f9ed9656cd8804a5ed568170697d0b183d
This way, with zero additional effort, SLO will support enhancements
to object storage and retrieval, such as:
* automatic resume of GETs on broken connection (today)
* storage policies (in the near future)
* erasure-coded object segments (in the far future)
This also lets SLOs work with other sorts of hypothetical third-party
middleware, for example object compression or encryption.
Getting COPY to work here is sort of a hack; the proxy's object
controller now checks for "swift.copy_response_hook" in the request's
environment and feeds the GET response (the source of the new object's
data) through it. This lets a COPY of a SLO manifest actually combine
the segments instead of merely copying the manifest document.
Updated ObjectController to expect a response's app_iter to be an
iterable, not just an iterator. (PEP 333 says "When called by the
server, the application object must return an iterable yielding zero
or more strings." ObjectController was just being too strict.) This
way, SLO can re-use the same response-generation logic for GET and
COPY requests.
Added a (sort of hokey) mechanism to allow middlewares to close
incompletely-consumed app iterators without triggering a warning. SLO
does this when it realizes it's performed a ranged GET on a manifest;
it closes the iterable, removes the range, and retries the
request. Without this change, the proxy logs would get 'Client
disconnected on read' in them.
DocImpact
blueprint multi-ring-large-objects
Change-Id: Ic11662eb5c7176fbf422a6fc87a569928d6f85a1
Summary of the new configuration option:
The cluster operators add the container_sync middleware to their
proxy pipeline and create a container-sync-realms.conf for their
cluster and copy this out to all their proxy and container servers.
This file specifies the available container sync "realms".
A container sync realm is a group of clusters with a shared key that
have agreed to provide container syncing to one another.
The end user can then set the X-Container-Sync-To value on a
container to //realm/cluster/account/container instead of the
previously required URL.
The allowed hosts list is not used with this configuration and
instead every container sync request sent is signed using the realm
key and user key.
This offers better security as source hosts can be faked much more
easily than faking per request signatures. Replaying signed requests,
assuming it could easily be done, shouldn't be an issue as the
X-Timestamp is part of the signature and so would just short-circuit
as already current or as superceded.
This also makes configuration easier for the end user, especially
with difficult networking situations where a different host might
need to be used for the container sync daemon since it's connecting
from within a cluster. With this new configuration option, the end
user just specifies the realm and cluster names and that is resolved
to the proper endpoint configured by the operator. If the operator
changes their configuration (key or endpoint), the end user does not
need to change theirs.
DocImpact
Change-Id: Ie1704990b66d0434e4991e26ed1da8b08cb05a37
Calling get_logger({}) instantiates a logging.handlers.SyslogHandler,
which opens and keeps a socket around (either /dev/log or UDP or
whatever; not important).
Under Python 2.6, all logging handlers instantiated anywhere at all
will live for the entire lifetime of the program; they get stored in
logging._handlerList and logging._handlers. Python 2.7 is very
similar, but uses weakrefs instead of strong references in those
module-level variables, so logging handlers can actually get cleaned
up prior to program exit.
The net effect is that any program that calls get_logger() more than a
fixed number of times will leak file descriptors under Python 2.6.
This commit throws encapsulation out the window and, under 2.6 only,
replaces strong references with weakrefs in logging._handlerList and
logging._handlers, thus avoiding the leak.
Change-Id: I5dc0d1619c5a4500f892b898afd9e3668ec0ee7c
Now the traceback goes all the way down to where the exception came
from, not just down to run_in_thread. Better for debugging.
Change-Id: Iac6acb843a6ecf51ea2672a563d80fa43d731f23
The early quorum change has maybe added a little bit too much
eventual to the consistency of requests in Swift, and users can
sometimes get unexpected
results.
This change gives us a knob to turn in finding the right balance,
by adding a timeout where pending requests can finish after quorum
is achieved.
Change-Id: Ife91aaa8653e75b01313bbcf19072181739e932c
Swift can now optionally be configured to allow requests to '/info',
providing information about the swift cluster. Additionally a HMAC
signed requests to
'/info?swiftinfo_sig=<sign>&swiftinfo_expires=<expires>' can be
configured allowing privileged access to more sensitive information
not meant to be public.
DocImpact
Change-Id: I2379360fbfe3d9e9e8b25f1dc34517d199574495
Implements: blueprint capabilities
Closes-Bug: #1245694
New replication_one_per_device (True by default)
that restricts incoming REPLICATION requests to
one per device, replication_currency allowing.
Also has replication_lock_timeout (15 by default)
to control how long a request will wait to obtain
a replication device lock before giving up.
This should be very useful in that you can be
assured any concurrent REPLICATION requests are
each writing to distinct devices. If you have 100
devices on a server, you can set
replication_concurrency to 100 and be confident
that, even if 100 replication requests were
executing concurrently, they'd each be writing to
separate devices. Before, all 100 could end up
writing to the same device, bringing it to a
horrible crawl.
NOTE: This is only for ssync replication. The
current default rsync replication still has the
potentially horrible behavior.
Change-Id: I36e99a3d7e100699c76db6d3a4846514537ff685
Allow the proxy to respond to many types of requests as soon as it has a
quorum. This can help speed up responses (without changing the results),
especially when one node is acting up.
I had to fix a few unit tests that no longer match the backend http requests
made by our proxy.
Change-Id: Ieb070dc3019e217e717b96154a7a809409bf40a5
This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32
Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62
Signed-off-by: Peter Portante <peter.portante@redhat.com>
assertTrue accepts a parameter msg which will be printed when
assertion fails, usually msg is a str. This patch fixes unsuitable
usage of assertTrue which set msg to bool type True.
Change-Id: I731f8ea553c935eba0e112ffded16f41a5ea86c0
Fixes-Bug: #1226374
Place all the methods related to on-disk layout and / or configuration
into a new common module that can be shared by the various modules
using the same on-disk layout.
Change-Id: I27ffd4665d5115ffdde649c48a4d18e12017e6a9
Signed-off-by: Peter Portante <peter.portante@redhat.com>
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.
Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
Each `isdir` or `exists` call performs a stat, but that info is already
available from the exception if it doesn't exist in `listdir`.
Change-Id: I831494e3dbc8fda62ab29431471566bdb8dc6d27
Address all the "hacking" lines that are flagged, and all the modules
that just have one item flagged.
Change-Id: I372a4bdf9c7748f73e38c4fd55e5954f1afade5b
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Use tempfile.mkstemp to generate temporary
file to avoid collision with an existing 'test'
file in /tmp.
Change-Id: Ic2f7f64ee9826afa6f04debd763e8c7a0eb25988
Fixes: bug #1213845
In unit tests, we should use: self.assertTrue or self.assertFalse
instead of 'assert xxx is True/False', and use self.assertRaises instead
of 'try ... except ...'
fixes bug #1212602
Change-Id: I25318eb59cd26a98afdce5495273e3b3266655a4
- Since netifaces is appending a %interface (i.e: %eth0) at the end and
does not make a correct ipv6 address we are removing it.
- Improve the tests of whataremyips along the way.
Closes-Bug: 1209443
Change-Id: I585e795083783009961b429607ca3f66b8d7ec30
Have json and xml use common record cleanup code.
Do a somewhat better job of parsing extensions from content-types.
Use a real XML serializer.
Change-Id: I10e14dffd1da590b4fd180b4d33ab5de862e2b55
The os.path.exists call performs an lstat, but os.path.ismount already
performs the same check. However, it performs a separate lstat() call
to check for a symlink, which we remove as well, cutting the number
performed in half.
Sample program to be straced for comparison:
from swift.common.constraints import check_mount
import os
os.write(1, "Starting\n")
if check_mount("/", "tmp"):
os.write(1, "Mounted\n")
Here is the output of a check on a mounted file system (common case)
using the new method:
---- strace new ----
write(1, "Starting\n", 9) = 9
lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0
lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
write(1, "Mounted\n", 8) = 8
---- strace old ----
write(1, "Starting\n", 9) = 9
stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0
lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0
lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0
lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
write(1, "Mounted\n", 8) = 8
Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Change the default value of wsgi workers from 1 to auto. The new default
value for workers in the proxy, container, account & object wsgi servers will
spawn as many workers per process as you have cpu cores.
This will not be ideal for some configurations, but it's much more likely to
produce a successful out of the box deployment.
Inspect the number of cpu_cores using python's multiprocessing when available.
Multiprocessing was added in python 2.6, but I know I've compiled python
without it before on accident. The cpu_count method seems to be pretty system
agnostic, but it says it can raise NotImplementedError or sometimes return 0.
Add a new utility method 'config_auto_int_value' to pull an integer out of the
config which has a dynamic default.
* drive by s/container/proxy/ in proxy-server.conf.5
* fix misplaced max_clients in *-server.conf-sample
* update doc/development_saio to force workers = 1
DocImpact
Change-Id: Ifa563d22952c902ab8cbe1d339ba385413c54e95
The proxy can now be configured to prefer local object servers for PUT
requests, where "local" is governed by the "write_affinity". The
"write_affinity_node_count" setting controls how many local object
servers to try before giving up and going on to remote ones.
I chose to simply re-order the object servers instead of filtering out
nonlocal ones so that, if all of the local ones are down, clients can
still get successful responses (just slower).
The goal is to trade availability for throughput. By writing to local
object servers across fast LAN links, clients get better throughput
than if the object servers were far away over slow WAN links. The
downside, of course, is that data availability (not durability) may
suffer when drives fail.
The default configuration has no write affinity in it, so the default
behavior is unchanged.
Added some words about these settings to the admin guide.
DocImpact
Change-Id: I09a0bd00524544ff627a3bccdcdc48f40720a86e
A really simple version of this was in container sync already, and I
needed a more complete version for work I'm doing, and I noticed
https://review.openstack.org/#/c/33405/ was also making use of it.
So, here's a more full version.
If https://review.openstack.org/#/c/33405/ lands before this, I'll
update it accordingly.
Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
