openstack/swift
Code Issues Proposed changes
88b2159961
swift/test/unit/common/test_utils.py

5609 lines
226 KiB
Python
Raw Normal View History

Change OpenStack LLC to Foundation Change-Id: I7c3df47c31759dbeb3105f8883e2688ada848d58 Closes-bug: #1214176
2013-09-20 01:00:54 +08:00
# Copyright (c) 2010-2012 OpenStack Foundation
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
Pep8 unit test modules for hacking and one liners (4 of 12) Address all the "hacking" lines that are flagged, and all the modules that just have one item flagged. Change-Id: I372a4bdf9c7748f73e38c4fd55e5954f1afade5b Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-08-31 22:36:58 -04:00
"""Tests for swift.common.utils"""
test/unit: Replace python print operator with print function (pep H233, py33) 'print' function is compatible with 2.x and 3.x python versions Link : https://www.python.org/dev/peps/pep-3105/ Python 2.6 has a __future__ import that removes print as language syntax, letting you use the functional form instead Change-Id: I94e1bc6bd83ad6b05695c7ebdf7cbfd8f6d9f9af
2015-07-28 21:03:05 +05:30
from __future__ import print_function
Prepare for trunk merge. Refactored some of the swift_init classes into a new module in swift.common, changed some names. Removed the bin test stuff. Fixed some bugs, added some features.
2011-02-11 13:18:19 -06:00
from test.unit import temptree
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
import ctypes
Auditor will clean up stale rsync tempfiles DiskFile already fills in the _ondisk_info attribute when it tries to open a diskfile - even if the DiskFile's fileset is not valid or deleted. During this process the rsync tempfiles would be discovered and logged, but no-one would attempt to clean them up - even if they were really old. Instead of logging and ignoring unexpected files when validate a DiskFile fileset we'll add unexpected files to the unexpected key in the _ondisk_info attribute. With a little bit of re-organization in the auditor's object_audit method to get things into a single return path we can add an unconditional check for unexpected files and remove those that are "old enough". Since the replicator will kill any rsync processes that are running longer than the configured rsync_timeout we know that any rsync tempfiles older than this can be deleted. Split unlink_older_than in common.utils into two functions to allow an explicit list of previously discovered paths to be passed in to avoid an extra listdir. Since the getmtime handling already ignores OSError there's less concern of race condition where a previous discovered unexpected file is reaped by rsync while we're attempting to clean it up. Update some doc on the new config option. Closes-Bug: #1554005 Change-Id: Id67681cb77f605e3491b8afcb9c69d769e154283
2016-03-15 17:09:21 -07:00
import contextlib
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
import errno
Fix ValueError from handoff logging. If you have StatsD logging turned on, then the iterator returned by Ring.get_part_nodes() will emit StatsD packets when it yields a handoff node. That network IO may make eventlet trampoline to another greenthread before returning from next(). Now, if that other greenthread tries to call next() on that same iterator, it blows up with a ValueError. Any socket IO inside a generator's next() method can cause this. It's easiest to reproduce with StatsD logging turned on, but logging to syslog can trigger it too. You can see this happen sometimes in the proxy's make_requests method if two of the primary nodes are down. Greenthread A goes into next() to get a handoff node, then sends a StatsD packet, and so eventlet trampolines to Greenthread B. Now, Greenthread B also calls next() to get a handoff node, and dies with a ValueError. This commit wraps up concurrently-accessed iter_nodes() iterators in a new thing called a GreenthreadSafeIterator that serializes access. Bug 1180110 Change-Id: I8fe13d7295c056a2cab9e084f5966078a49bdc13
2013-05-14 13:04:01 -07:00
import eventlet
early quorum responses Allow the proxy to respond to many types of requests as soon as it has a quorum. This can help speed up responses (without changing the results), especially when one node is acting up. I had to fix a few unit tests that no longer match the backend http requests made by our proxy. Change-Id: Ieb070dc3019e217e717b96154a7a809409bf40a5
2013-10-30 21:43:35 +00:00
import eventlet.event
Fix logger test coupling Two tests fail if test/unit/common/test_daemon.py is excluded from the test set: test.unit.common.test_utils.TestUtils.test_swift_log_formatter test.unit.common.test_utils.TestStatsdLoggingDelegation.test_thread_locals They fail because they assert that logger.txn_id is None, when previous tests cause txn_id to be set. This coupling is masked when test_daemon.py is included because it reloads the common/utils module, and is executed just before test.unit.common.test_utils. The coupling can be observed by running, for example: nosetests ./test/unit/common/middleware/test_except.py ./test/unit/common/test_utils.py or nosetests ./test/unit/proxy ./test/unit/common/test_utils.py The failing tests should reset logger thread_locals before making assertions. test_utils.reset_loggers() attempts this but is broken because it sets thread_local on the wrong object. Changes in this patch: - fix reset_loggers() to reset the LogAdapter thread local state - add a reset_logger_state decorator to call reset_loggers before and after a decorated method - use the decorator for increased isolation of tests that previously called reset_loggers only on exit Change-Id: If9aa781a2dd1929a47ef69322ec8c53263d47660
2015-02-11 17:49:41 +00:00
import functools
Add secondary groups to user during privilege escalation setgid provides the primary group, setgroups sets the secondary groups. Prior to this patch, we would do a setgroups with an empty list, effectively wiping secondary groups. We now verify which secondary groups the user is member of and escalate the privileges accordingly. Change-Id: I33a10edd448b3ac5aa758a8d1d70e582cf421c7d Closes-Bug: 1269473
2014-01-20 13:30:58 -05:00
import grp
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
import logging
import os
Unify backend logging Make account, object, and container servers construct log lines using the same utility function so they will produce identically formatted lines. This change reorders the fields logged for the account server. This change also adds the "additional info" field to the two servers that didn't log that field. This makes the log lines identical across all 3 servers. If people don't like that, I can take that out. I think it makes the documentation, parsing of the log lines, and the code a tad cleaner. DocImpact Change-Id: I268dc0df9dd07afa5382592a28ea37b96c6c2f44 Closes-Bug: 1280955
2014-03-26 22:55:55 +00:00
import mock
Honor sample_rate in StatsD logging. It's there to let administrators turn down the barrage of stats data that StatsD must cope with, but it wasn't actually honored. Worse, if the sample rate was set to e.g. 0.2, the stats would all be multiplied by its inverse, e.g. 2. This patch actually drops packets when sample_rate < 1, so you get correct measurements. Fortunately, the default sample rate is 1 (i.e. drop nothing), and multiplying by 1/1 doesn't change anything, so stats with the default sample rate of 1.0 are, and have been, just fine. Fixes bug 1065643. Also, make the two touched files compliant with pep8 v1.3.3. Change-Id: I66663144009ae4c9ee96f6a111745d8f5d2f5ca3
2012-10-11 14:04:02 -07:00
import random
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
import re
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
import socket
Make ThreadPools deallocatable. Currently, a ThreadPool acquires resources that last until process exit. You can let the ThreadPool go out of scope, but that doesn't terminate the worker threads or close file descriptors or anything. This commit makes it so you can .terminate() a ThreadPool object and get its resources back. Also, after you call .terminate(), trying to use the ThreadPool raises an exception so you know you've goofed. I have some internal code that could really use this, plus it makes the unit test run not leak resources, which is nice. Change-Id: Ibf7c6dc14c14f379421a79afb6c90a5e64b235fa
2015-01-07 16:50:58 -08:00
import stat
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
import sys
Allow specification of object devices for audit In object audit "once" mode we are allowing the user to specify a sub-set of devices to audit using the "--devices" command-line option. The sub-set is specified as a comma-separated list. This patch is taken from a larger patch to enable parallel processing in the object auditor. We've had to modify recon so that it will work properly with this change to "once" mode. We've modified dump_recon_cache() so that it will store nested dictionaries, in other words it will store a recon cache entry such as {'key1': {'key2': {...}}}. When the object auditor is run in "once" mode with "--devices" set the object_auditor_stats_ALL and ZBF entries look like: {'object_auditor_stats_ALL': {'disk1disk2..diskn': {...}}}. When swift-recon is run, it hunts through the nested dicts to find the appropriate entries. The object auditor recon cache entries are set to {} at the beginning of each audit cycle, and individual disk entries are cleared from cache at the end of each disk's audit cycle. DocImpact Change-Id: Icc53dac0a8136f1b2f61d5e08baf7b4fd87c8123
2014-02-24 11:24:56 +00:00
import json
Add LRUCache to common.utils This decorator will memonize a function using a fixed size cache that evicts the oldest entries. It also supports a maxtime paramter to configure a "time-to-live" for entries in the cache. The reconciler code uses this to cache computations of the correct storage policy index for a container for 30 seconds. DocImpact Implements: blueprint storage-policies Change-Id: I0f220869e33c461a4100b21c6324ad725da864fa
2014-04-28 17:05:43 -07:00
import math
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
Port swift.common.utils.StatsdClient to Python 3 * StatsdClient._send(): on Python 3, encode parts to UTF-8 and replace '|' with b'|' to join parts. * timing_stats(): replace func.func_name with func.__name__. The func_name attribute of functions was removed on Python 3, whereas the __name__ attribute is available on Python 2 and Python 3. * Fix unit tests to use bytes Change-Id: Ic279c9b54e91aabcc52587eed7758e268ffb155e
2015-10-19 16:19:28 +02:00
import six
replace use of deprecated rfc822.Message with a helper utility The rfc822 module has been deprecated since Python 2.3, and in particular is absent from the Python 3 standard library. However, Swift uses instances of rfc822.Message in a number of places, relying on its behavior of immediately parsing the headers of a file-like object without consuming the body, leaving the position of the file at the start of the body. Python 3's http.client has an undocumented parse_headers function with the same behavior, which inspired the new parse_mime_headers utility introduced here. (The HeaderKeyDict returned by parse_mime_headers doesn't have a `.getheader(key)` method like rfc822.Message did; the dictionary-like `[key]` or `.get(key)` interface should be used exclusively.) The implementation in this commit won't actually work with Python 3, the email.parser.Parser().parsestr of which expects a Unicode string, but it is believed that this can be addressed in followup work. Change-Id: Ia5ee2ead67e36e8c6416183667f64ae255887736
2015-09-08 19:21:39 -07:00
from six import BytesIO, StringIO
Use six to fix imports on Python 3 Get configparser, queue, http_client modules from six.moves. Patch generated by the six_moves operation of the sixer tool: https://pypi.python.org/pypi/sixer Change-Id: I666241ab50101b8cc6f992dd80134ce27327bd7d
2015-05-25 18:26:38 +02:00
from six.moves.queue import Queue, Empty
from six.moves import range
conf.d support Allow Swift daemons and servers to optionally accept a directory as the configuration parameter. Directory based configuration leverages ConfigParser's native multi-file support. Files ending in '.conf' in the given directory are parsed in lexicographical order. Filenames starting with '.' are ignored. A mixture of file and directory configuration paths is not supported - if the configuration path is a file behavior is unchanged. * update swift-init to search for conf.d paths when building servers (e.g. /etc/swift/proxy-server.conf.d/) * new script swift-config can be used to inspect the cumulative configuration * pull a little bit of code out of run_wsgi and test separately * fix example config bug for the proxy servers client_disconnect option * added section on directory based configuration to deployment guide DocImpact Implements: blueprint confd Change-Id: I89b0f48e538117f28590cf6698401f74ef58003b
2013-03-25 16:34:43 -07:00
from textwrap import dedent
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
Use randomly named file for temporay file Use tempfile.mkstemp to generate temporary file to avoid collision with an existing 'test' file in /tmp. Change-Id: Ic2f7f64ee9826afa6f04debd763e8c7a0eb25988 Fixes: bug #1213845
2013-08-19 11:41:38 +02:00
import tempfile
adding in rate limiting and unit tests
2010-12-16 16:20:57 -08:00
import time
Preserve tracebacks from run_in_thread Now the traceback goes all the way down to where the exception came from, not just down to run_in_thread. Better for debugging. Change-Id: Iac6acb843a6ecf51ea2672a563d80fa43d731f23
2013-12-17 16:11:26 -08:00
import traceback
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
import unittest
Add test for swift.common.utils.lock_file Change-Id: Id03f130ad7016b11b740050b7aadf879773d5a5f
2013-06-21 11:48:05 -04:00
import fcntl
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
import shutil
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
from getpass import getuser
from shutil import rmtree
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
from functools import partial
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
from tempfile import TemporaryFile, NamedTemporaryFile, mkdtemp
Don't append %interface with ipv6 ips. - Since netifaces is appending a %interface (i.e: %eth0) at the end and does not make a correct ipv6 address we are removing it. - Improve the tests of whataremyips along the way. Closes-Bug: 1209443 Change-Id: I585e795083783009961b429607ca3f66b8d7ec30
2013-08-08 11:11:39 +02:00
from netifaces import AF_INET6
Moved some code out of swift.obj.replicator This will be needed in future replication work to avoid circular imports. I used swift.obj.base as the module name just because we seemed to avoid putting code in __init__.py files so far and I didn't want to buck the trend. I would love to see other obj things like *_metadata and DiskFile move into swift.obj.base as well and swift.obj.server just be the WSGI server logic, but I'll leave that for the future. I have changed the tests as little as possible (just the references to where they get the code to test) to show the refactor has not broken anything. I did add a test for tpool_reraise since there was none before. There will be a follow on patch for moving the tests to their new location(s). I figured I'd wait to put the bikes in the shed until everyone's done painting it. Change-Id: I32b4ac88be21eb76c877d3f4cc1e6ac33304835b
2013-05-11 19:51:03 +00:00
from mock import MagicMock, patch
Add unit tests for utils.validate_hash_path This patch add some unit tests to prevent regression and to describe validate_hash_path behavior, I found in review with https://review.openstack.org/#/c/231864/ *bonus* - Fix test_hash_path to use "with" syntax instead of "try/finally" for assigning a testing value into a global variable. Change-Id: I948999a8fb8addb9a378dbf8bee853b205aeafad
2015-10-07 02:14:49 -07:00
from six.moves.configparser import NoSectionError, NoOptionError
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
Fix proxy-server's support for chunked transferring in GET object Proxy-server now requires Content-Length in the response header when getting object and does not support chunked transferring with "Transfer-Encoding: chunked" This doesn't matter in normal swift, but prohibits us from putting any middelwares to execute something like streaming processing of objects, which can't calculate the length of their response body before they start to send their response. Change-Id: I60fc6c86338d734e39b7e5f1e48a2647995045ef
2015-11-17 16:15:59 +09:00
from swift.common.exceptions import Timeout, MessageTimeout, \
ConnectionTimeout, LockTimeout, ReplicationLockTimeout, \
MimeInvalid, ThreadPoolDead
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
from swift.common import utils
New container sync configuration option Summary of the new configuration option: The cluster operators add the container_sync middleware to their proxy pipeline and create a container-sync-realms.conf for their cluster and copy this out to all their proxy and container servers. This file specifies the available container sync "realms". A container sync realm is a group of clusters with a shared key that have agreed to provide container syncing to one another. The end user can then set the X-Container-Sync-To value on a container to //realm/cluster/account/container instead of the previously required URL. The allowed hosts list is not used with this configuration and instead every container sync request sent is signed using the realm key and user key. This offers better security as source hosts can be faked much more easily than faking per request signatures. Replaying signed requests, assuming it could easily be done, shouldn't be an issue as the X-Timestamp is part of the signature and so would just short-circuit as already current or as superceded. This also makes configuration easier for the end user, especially with difficult networking situations where a different host might need to be used for the container sync daemon since it's connecting from within a cluster. With this new configuration option, the end user just specifies the realm and cluster names and that is resolved to the proper endpoint configured by the operator. If the operator changes their configuration (key or endpoint), the end user does not need to change theirs. DocImpact Change-Id: Ie1704990b66d0434e4991e26ed1da8b08cb05a37
2013-12-08 09:13:59 +00:00
from swift.common.container_sync_realms import ContainerSyncRealms
Move HeaderKeyDict to avoid an inline import There was a function in swift.common.utils that was importing swob.HeaderKeyDict at call time. It couldn't import it at compilation time since utils can't import from swob or else it blows up with a circular import error. This commit just moves HeaderKeyDict into swift.common.header_key_dict so that we can remove the inline import. Change-Id: I656fde8cc2e125327c26c589cf1045cb81ffc7e5
2016-03-02 10:28:51 +00:00
from swift.common.header_key_dict import HeaderKeyDict
from swift.common.swob import Request, Response
SLOs broken for range requests Change-Id: I21175a4be0cda9a8a98c425bff11c80895cd6d3e
2013-09-10 09:01:32 -07:00
from test.unit import FakeLogger
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
monkeypatch thread for keystoneclient keystoneclient uses threading.Lock(), but swift doesn't monkeypatch threading, this result in lockup when two greenthreads try to acquire a non green lock. This change fixes that. Change-Id: I9b44284a5eb598a6978364819f253e031f4eaeef Closes-bug: #1508424
2015-10-22 17:33:09 +02:00
threading = eventlet.patcher.original('threading')
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
Convert all old-style classes to new-style This cleanup has been slowly happening for a while; let's finish it. Change-Id: I1561e3540d524834e0cc5bc725ab80936eae1f0e
2014-03-03 17:28:48 -08:00
class MockOs(object):
adding defaults, docs, and unit tests
2010-12-28 14:54:00 -08:00
Uses None instead of mutables for function param defaults As seen on #1174809, changes use of mutable types as default arguments and defaults them within the method. Otherwise, those defaults can be unexpectedly persisted with the function between invocations and erupt into mass hysteria on the streets. There was indeed a test (TestSimpleClient.test_get_with_retries) that was erroneously relying on this behavior. Since previous tests had populated their own instantiations with a token, this test only passed because the modified headers dict from previous tests was being overridden. As expected, with the mutable defaults fix in SimpleClient, this test begain to fail since it never specified any token, yet it has always passed anyway. This change also now provides the expected token. Change-Id: If95f11d259008517dab511e88acfe9731e5a99b5 Related-Bug: #1174809
2014-05-10 05:15:12 -05:00
def __init__(self, pass_funcs=None, called_funcs=None, raise_funcs=None):
if pass_funcs is None:
pass_funcs = []
if called_funcs is None:
called_funcs = []
if raise_funcs is None:
raise_funcs = []
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
self.closed_fds = []
for func in pass_funcs:
setattr(self, func, self.pass_func)
self.called_funcs = {}
for func in called_funcs:
removed unneeded daemonize function from utils, pulled get_socket out of run_wsgi, reworked test_utils and test_wsgi
2010-11-19 12:15:41 -06:00
c_func = partial(self.called_func, func)
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
setattr(self, func, c_func)
for func in raise_funcs:
removed unneeded daemonize function from utils, pulled get_socket out of run_wsgi, reworked test_utils and test_wsgi
2010-11-19 12:15:41 -06:00
r_func = partial(self.raise_func, func)
setattr(self, func, r_func)
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
def pass_func(self, *args, **kwargs):
pass
Call setgroups() before setuid() and setgid(). Fixes bug 989569. This patch ensures that the list of groups is completely reset when dropping privileges. Change-Id: I049f75e66e08a4a6361504b013bc68c4c38ef093
2012-02-07 14:19:22 -05:00
setgroups = chdir = setsid = setgid = setuid = umask = pass_func
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
def called_func(self, name, *args, **kwargs):
self.called_funcs[name] = True
removed unneeded daemonize function from utils, pulled get_socket out of run_wsgi, reworked test_utils and test_wsgi
2010-11-19 12:15:41 -06:00
def raise_func(self, name, *args, **kwargs):
self.called_funcs[name] = True
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
raise OSError()
def dup2(self, source, target):
self.closed_fds.append(target)
Call setgroups() before setuid() and setgid(). Fixes bug 989569. This patch ensures that the list of groups is completely reset when dropping privileges. Change-Id: I049f75e66e08a4a6361504b013bc68c4c38ef093
2012-02-07 14:19:22 -05:00
def geteuid(self):
'''Pretend we are running as root.'''
return 0
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
def __getattr__(self, name):
# I only over-ride portions of the os module
try:
return object.__getattr__(self, name)
except AttributeError:
return getattr(os, name)
Convert all old-style classes to new-style This cleanup has been slowly happening for a while; let's finish it. Change-Id: I1561e3540d524834e0cc5bc725ab80936eae1f0e
2014-03-03 17:28:48 -08:00
class MockUdpSocket(object):
Catch permissions errors when writing StatsD packets Closes-Bug: #1183152 Change-Id: I4b2c6e947241c987779a385fdff270d037470a57
2014-08-10 23:50:17 -07:00
def __init__(self, sendto_errno=None):
Honor sample_rate in StatsD logging. It's there to let administrators turn down the barrage of stats data that StatsD must cope with, but it wasn't actually honored. Worse, if the sample rate was set to e.g. 0.2, the stats would all be multiplied by its inverse, e.g. 2. This patch actually drops packets when sample_rate < 1, so you get correct measurements. Fortunately, the default sample rate is 1 (i.e. drop nothing), and multiplying by 1/1 doesn't change anything, so stats with the default sample rate of 1.0 are, and have been, just fine. Fixes bug 1065643. Also, make the two touched files compliant with pep8 v1.3.3. Change-Id: I66663144009ae4c9ee96f6a111745d8f5d2f5ca3
2012-10-11 14:04:02 -07:00
self.sent = []
Catch permissions errors when writing StatsD packets Closes-Bug: #1183152 Change-Id: I4b2c6e947241c987779a385fdff270d037470a57
2014-08-10 23:50:17 -07:00
self.sendto_errno = sendto_errno
Honor sample_rate in StatsD logging. It's there to let administrators turn down the barrage of stats data that StatsD must cope with, but it wasn't actually honored. Worse, if the sample rate was set to e.g. 0.2, the stats would all be multiplied by its inverse, e.g. 2. This patch actually drops packets when sample_rate < 1, so you get correct measurements. Fortunately, the default sample rate is 1 (i.e. drop nothing), and multiplying by 1/1 doesn't change anything, so stats with the default sample rate of 1.0 are, and have been, just fine. Fixes bug 1065643. Also, make the two touched files compliant with pep8 v1.3.3. Change-Id: I66663144009ae4c9ee96f6a111745d8f5d2f5ca3
2012-10-11 14:04:02 -07:00
def sendto(self, data, target):
Catch permissions errors when writing StatsD packets Closes-Bug: #1183152 Change-Id: I4b2c6e947241c987779a385fdff270d037470a57
2014-08-10 23:50:17 -07:00
if self.sendto_errno:
raise socket.error(self.sendto_errno,
'test errno %s' % self.sendto_errno)
Honor sample_rate in StatsD logging. It's there to let administrators turn down the barrage of stats data that StatsD must cope with, but it wasn't actually honored. Worse, if the sample rate was set to e.g. 0.2, the stats would all be multiplied by its inverse, e.g. 2. This patch actually drops packets when sample_rate < 1, so you get correct measurements. Fortunately, the default sample rate is 1 (i.e. drop nothing), and multiplying by 1/1 doesn't change anything, so stats with the default sample rate of 1.0 are, and have been, just fine. Fixes bug 1065643. Also, make the two touched files compliant with pep8 v1.3.3. Change-Id: I66663144009ae4c9ee96f6a111745d8f5d2f5ca3
2012-10-11 14:04:02 -07:00
self.sent.append((data, target))
def close(self):
pass
Convert all old-style classes to new-style This cleanup has been slowly happening for a while; let's finish it. Change-Id: I1561e3540d524834e0cc5bc725ab80936eae1f0e
2014-03-03 17:28:48 -08:00
class MockSys(object):
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
fixed lockups and lost messages in swift-init and capture_stdio
2011-03-30 15:04:15 -05:00
def __init__(self):
self.stdin = TemporaryFile('w')
self.stdout = TemporaryFile('r')
self.stderr = TemporaryFile('r')
self.__stderr__ = self.stderr
self.stdio_fds = [self.stdin.fileno(), self.stdout.fileno(),
self.stderr.fileno()]
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
def reset_loggers():
if hasattr(utils.get_logger, 'handler4logger'):
for logger, handler in utils.get_logger.handler4logger.items():
logger.removeHandler(handler)
delattr(utils.get_logger, 'handler4logger')
if hasattr(utils.get_logger, 'console_handler4logger'):
for logger, h in utils.get_logger.console_handler4logger.items():
logger.removeHandler(h)
delattr(utils.get_logger, 'console_handler4logger')
Fix logger test coupling Two tests fail if test/unit/common/test_daemon.py is excluded from the test set: test.unit.common.test_utils.TestUtils.test_swift_log_formatter test.unit.common.test_utils.TestStatsdLoggingDelegation.test_thread_locals They fail because they assert that logger.txn_id is None, when previous tests cause txn_id to be set. This coupling is masked when test_daemon.py is included because it reloads the common/utils module, and is executed just before test.unit.common.test_utils. The coupling can be observed by running, for example: nosetests ./test/unit/common/middleware/test_except.py ./test/unit/common/test_utils.py or nosetests ./test/unit/proxy ./test/unit/common/test_utils.py The failing tests should reset logger thread_locals before making assertions. test_utils.reset_loggers() attempts this but is broken because it sets thread_local on the wrong object. Changes in this patch: - fix reset_loggers() to reset the LogAdapter thread local state - add a reset_logger_state decorator to call reset_loggers before and after a decorated method - use the decorator for increased isolation of tests that previously called reset_loggers only on exit Change-Id: If9aa781a2dd1929a47ef69322ec8c53263d47660
2015-02-11 17:49:41 +00:00
# Reset the LogAdapter class thread local state. Use get_logger() here
# to fetch a LogAdapter instance because the items from
# get_logger.handler4logger above are the underlying logger instances,
# not the LogAdapter.
utils.get_logger(None).thread_locals = (None, None)
def reset_logger_state(f):
@functools.wraps(f)
def wrapper(self, *args, **kwargs):
reset_loggers()
try:
return f(self, *args, **kwargs)
finally:
reset_loggers()
return wrapper
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
Add two vector timestamps The normalized form of the X-Timestamp header looks like a float with a fixed width to ensure stable string sorting - normalized timestamps look like "1402464677.04188" To support overwrites of existing data without modifying the original timestamp but still maintain consistency a second internal offset vector is append to the normalized timestamp form which compares and sorts greater than the fixed width float format but less than a newer timestamp. The internalized format of timestamps looks like "1402464677.04188_0000000000000000" - the portion after the underscore is the offset and is a formatted hexadecimal integer. The internalized form is not exposed to clients in responses from Swift. Normal client operations will not create a timestamp with an offset. The Timestamp class in common.utils supports internalized and normalized formatting of timestamps and also comparison of timestamp values. When the offset value of a Timestamp is 0 - it's considered insignificant and need not be represented in the string format; to support backwards compatibility during a Swift upgrade the internalized and normalized form of a Timestamp with an insignificant offset are identical. When a timestamp includes an offset it will always be represented in the internalized form, but is still excluded from the normalized form. Timestamps with an equivalent timestamp portion (the float part) will compare and order by their offset. Timestamps with a greater timestamp portion will always compare and order greater than a Timestamp with a lesser timestamp regardless of it's offset. String comparison and ordering is guaranteed for the internalized string format, and is backwards compatible for normalized timestamps which do not include an offset. The reconciler currently uses a offset bump to ensure that objects can move to the wrong storage policy and be moved back. This use-case is valid because the content represented by the user-facing timestamp is not modified in way. Future consumers of the offset vector of timestamps should be mindful of HTTP semantics of If-Modified and take care to avoid deviation in the response from the object server without an accompanying change to the user facing timestamp. DocImpact Implements: blueprint storage-policies Change-Id: Id85c960b126ec919a481dc62469bf172b7fb8549
2014-06-10 22:17:47 -07:00
class TestTimestamp(unittest.TestCase):
"""Tests for swift.common.utils.Timestamp"""
def test_invalid_input(self):
self.assertRaises(ValueError, utils.Timestamp, time.time(), offset=-1)
def test_invalid_string_conversion(self):
t = utils.Timestamp(time.time())
self.assertRaises(TypeError, str, t)
Raise ValueError for offset on Timestamp over limit We can't order a Timestamp with an offset larger than 16 hex digits correctly, so we raise a ValueError if you try to create one. Change-Id: I8c8d4cf13785a1a8eb7416392263eae5242aa407
2014-12-01 11:44:10 -08:00
def test_offset_limit(self):
t = 1417462430.78693
# can't have a offset above MAX_OFFSET
self.assertRaises(ValueError, utils.Timestamp, t,
offset=utils.MAX_OFFSET + 1)
# exactly max offset is fine
ts = utils.Timestamp(t, offset=utils.MAX_OFFSET)
self.assertEqual(ts.internal, '1417462430.78693_ffffffffffffffff')
# but you can't offset it further
self.assertRaises(ValueError, utils.Timestamp, ts.internal, offset=1)
# unless you start below it
ts = utils.Timestamp(t, offset=utils.MAX_OFFSET - 1)
self.assertEqual(utils.Timestamp(ts.internal, offset=1),
'1417462430.78693_ffffffffffffffff')
Add two vector timestamps The normalized form of the X-Timestamp header looks like a float with a fixed width to ensure stable string sorting - normalized timestamps look like "1402464677.04188" To support overwrites of existing data without modifying the original timestamp but still maintain consistency a second internal offset vector is append to the normalized timestamp form which compares and sorts greater than the fixed width float format but less than a newer timestamp. The internalized format of timestamps looks like "1402464677.04188_0000000000000000" - the portion after the underscore is the offset and is a formatted hexadecimal integer. The internalized form is not exposed to clients in responses from Swift. Normal client operations will not create a timestamp with an offset. The Timestamp class in common.utils supports internalized and normalized formatting of timestamps and also comparison of timestamp values. When the offset value of a Timestamp is 0 - it's considered insignificant and need not be represented in the string format; to support backwards compatibility during a Swift upgrade the internalized and normalized form of a Timestamp with an insignificant offset are identical. When a timestamp includes an offset it will always be represented in the internalized form, but is still excluded from the normalized form. Timestamps with an equivalent timestamp portion (the float part) will compare and order by their offset. Timestamps with a greater timestamp portion will always compare and order greater than a Timestamp with a lesser timestamp regardless of it's offset. String comparison and ordering is guaranteed for the internalized string format, and is backwards compatible for normalized timestamps which do not include an offset. The reconciler currently uses a offset bump to ensure that objects can move to the wrong storage policy and be moved back. This use-case is valid because the content represented by the user-facing timestamp is not modified in way. Future consumers of the offset vector of timestamps should be mindful of HTTP semantics of If-Modified and take care to avoid deviation in the response from the object server without an accompanying change to the user facing timestamp. DocImpact Implements: blueprint storage-policies Change-Id: Id85c960b126ec919a481dc62469bf172b7fb8549
2014-06-10 22:17:47 -07:00
def test_normal_format_no_offset(self):
expected = '1402436408.91203'
test_values = (
'1402436408.91203',
'1402436408.91203_00000000',
'1402436408.912030000',
'1402436408.912030000_0000000000000',
'000001402436408.912030000',
'000001402436408.912030000_0000000000',
1402436408.91203,
1402436408.912029,
1402436408.9120300000000000,
1402436408.91202999999999999,
utils.Timestamp(1402436408.91203),
utils.Timestamp(1402436408.91203, offset=0),
utils.Timestamp(1402436408.912029),
utils.Timestamp(1402436408.912029, offset=0),
utils.Timestamp('1402436408.91203'),
utils.Timestamp('1402436408.91203', offset=0),
utils.Timestamp('1402436408.91203_00000000'),
utils.Timestamp('1402436408.91203_00000000', offset=0),
)
for value in test_values:
timestamp = utils.Timestamp(value)
self.assertEqual(timestamp.normal, expected)
# timestamp instance can also compare to string or float
self.assertEqual(timestamp, expected)
self.assertEqual(timestamp, float(expected))
self.assertEqual(timestamp, utils.normalize_timestamp(expected))
def test_isoformat(self):
expected = '2014-06-10T22:47:32.054580'
test_values = (
'1402440452.05458',
'1402440452.054579',
'1402440452.05458_00000000',
'1402440452.054579_00000000',
'1402440452.054580000',
'1402440452.054579999',
'1402440452.054580000_0000000000000',
'1402440452.054579999_0000ff00',
'000001402440452.054580000',
'000001402440452.0545799',
'000001402440452.054580000_0000000000',
'000001402440452.054579999999_00000fffff',
1402440452.05458,
1402440452.054579,
1402440452.0545800000000000,
1402440452.054579999,
utils.Timestamp(1402440452.05458),
utils.Timestamp(1402440452.0545799),
utils.Timestamp(1402440452.05458, offset=0),
utils.Timestamp(1402440452.05457999999, offset=0),
utils.Timestamp(1402440452.05458, offset=100),
utils.Timestamp(1402440452.054579, offset=100),
utils.Timestamp('1402440452.05458'),
utils.Timestamp('1402440452.054579999'),
utils.Timestamp('1402440452.05458', offset=0),
utils.Timestamp('1402440452.054579', offset=0),
utils.Timestamp('1402440452.05458', offset=300),
utils.Timestamp('1402440452.05457999', offset=300),
utils.Timestamp('1402440452.05458_00000000'),
utils.Timestamp('1402440452.05457999_00000000'),
utils.Timestamp('1402440452.05458_00000000', offset=0),
utils.Timestamp('1402440452.05457999_00000aaa', offset=0),
utils.Timestamp('1402440452.05458_00000000', offset=400),
utils.Timestamp('1402440452.054579_0a', offset=400),
)
for value in test_values:
self.assertEqual(utils.Timestamp(value).isoformat, expected)
expected = '1970-01-01T00:00:00.000000'
test_values = (
'0',
'0000000000.00000',
'0000000000.00000_ffffffffffff',
0,
0.0,
)
for value in test_values:
self.assertEqual(utils.Timestamp(value).isoformat, expected)
def test_not_equal(self):
ts = '1402436408.91203_0000000000000001'
test_values = (
utils.Timestamp('1402436408.91203_0000000000000002'),
utils.Timestamp('1402436408.91203'),
utils.Timestamp(1402436408.91203),
utils.Timestamp(1402436408.91204),
utils.Timestamp(1402436408.91203, offset=0),
utils.Timestamp(1402436408.91203, offset=2),
)
for value in test_values:
self.assertTrue(value != ts)
Let equal Timestamps not be unequal Make the result of Timestamp(x) != Timestamp(x) be False. In python 2.7 this requires the __ne__ method to be defined [1]. "The truth of x==y does not imply that x!=y is false." The functools.total_ordering decorator does not autocreate a __ne__ method. In python 3 the __ne__ method is not required [2]. "By default, __ne__() delegates to __eq__() and inverts the result". This patch puts back the __ne__ method removed in [3]. Whilst no tests fail on master with python2.7, they do on this patch [4] and it seems dangerous to have this absurd behaviour lurking. [1] https://docs.python.org/2/reference/datamodel.html#object.__ne__ [2] https://docs.python.org/3.4/reference/datamodel.html#object.__ne__ [3] Change-Id: Id26777ac2c780316ff10ef7d954c48cc1fd480b5 [4] Change-Id: Ia597cd460bb5fd40aa92e886e3e18a7542603d01 Change-Id: I01fbfa310df3c74390f8e8c2e9ffff81bbf05e47
2016-01-28 16:47:37 +00:00
self.assertIs(True, utils.Timestamp(ts) == ts) # sanity
self.assertIs(False, utils.Timestamp(ts) != utils.Timestamp(ts))
self.assertIs(False, utils.Timestamp(ts) != ts)
Update container on fast-POST This patch makes a number of changes to enable content-type metadata to be updated when using the fast-POST mode of operation, as proposed in the associated spec [1]. * the object server and diskfile are modified to allow content-type to be updated by a POST and the updated value to be stored in .meta files. * the object server accepts PUTs and DELETEs with older timestamps than existing .meta files. This is to be consistent with replication that will leave a later .meta file in place when replicating a .data file. * the diskfile interface is modified to provide accessor methods for the content-type and its timestamp. * the naming of .meta files is modified to encode two timestamps when the .meta file contains a content-type value that was set prior to the latest metadata update; this enables consistency to be achieved when rsync is used for replication. * ssync is modified to sync meta files when content-type differs between local and remote copies of objects. * the object server issues container updates when handling POST requests, notifying the container server of the current immutable metadata (etag, size, hash, swift_bytes), content-type with their respective timestamps, and the mutable metadata timestamp. * the container server maintains the most recently reported values for immutable metadata, content-type and mutable metadata, each with their respective timestamps, in a single db row. * new probe tests verify that replication achieves eventual consistency of containers and objects after discrete updates to content-type and mutable metadata, and that container-sync sync's objects after fast-post updates. [1] spec change-id: I60688efc3df692d3a39557114dca8c5490f7837e Change-Id: Ia597cd460bb5fd40aa92e886e3e18a7542603d01
2015-08-10 10:30:10 -05:00
self.assertIs(False, utils.Timestamp(ts) is None)
self.assertIs(True, utils.Timestamp(ts) is not None)
Let equal Timestamps not be unequal Make the result of Timestamp(x) != Timestamp(x) be False. In python 2.7 this requires the __ne__ method to be defined [1]. "The truth of x==y does not imply that x!=y is false." The functools.total_ordering decorator does not autocreate a __ne__ method. In python 3 the __ne__ method is not required [2]. "By default, __ne__() delegates to __eq__() and inverts the result". This patch puts back the __ne__ method removed in [3]. Whilst no tests fail on master with python2.7, they do on this patch [4] and it seems dangerous to have this absurd behaviour lurking. [1] https://docs.python.org/2/reference/datamodel.html#object.__ne__ [2] https://docs.python.org/3.4/reference/datamodel.html#object.__ne__ [3] Change-Id: Id26777ac2c780316ff10ef7d954c48cc1fd480b5 [4] Change-Id: Ia597cd460bb5fd40aa92e886e3e18a7542603d01 Change-Id: I01fbfa310df3c74390f8e8c2e9ffff81bbf05e47
2016-01-28 16:47:37 +00:00
Add two vector timestamps The normalized form of the X-Timestamp header looks like a float with a fixed width to ensure stable string sorting - normalized timestamps look like "1402464677.04188" To support overwrites of existing data without modifying the original timestamp but still maintain consistency a second internal offset vector is append to the normalized timestamp form which compares and sorts greater than the fixed width float format but less than a newer timestamp. The internalized format of timestamps looks like "1402464677.04188_0000000000000000" - the portion after the underscore is the offset and is a formatted hexadecimal integer. The internalized form is not exposed to clients in responses from Swift. Normal client operations will not create a timestamp with an offset. The Timestamp class in common.utils supports internalized and normalized formatting of timestamps and also comparison of timestamp values. When the offset value of a Timestamp is 0 - it's considered insignificant and need not be represented in the string format; to support backwards compatibility during a Swift upgrade the internalized and normalized form of a Timestamp with an insignificant offset are identical. When a timestamp includes an offset it will always be represented in the internalized form, but is still excluded from the normalized form. Timestamps with an equivalent timestamp portion (the float part) will compare and order by their offset. Timestamps with a greater timestamp portion will always compare and order greater than a Timestamp with a lesser timestamp regardless of it's offset. String comparison and ordering is guaranteed for the internalized string format, and is backwards compatible for normalized timestamps which do not include an offset. The reconciler currently uses a offset bump to ensure that objects can move to the wrong storage policy and be moved back. This use-case is valid because the content represented by the user-facing timestamp is not modified in way. Future consumers of the offset vector of timestamps should be mindful of HTTP semantics of If-Modified and take care to avoid deviation in the response from the object server without an accompanying change to the user facing timestamp. DocImpact Implements: blueprint storage-policies Change-Id: Id85c960b126ec919a481dc62469bf172b7fb8549
2014-06-10 22:17:47 -07:00
def test_no_force_internal_no_offset(self):
"""Test that internal is the same as normal with no offset"""
with mock.patch('swift.common.utils.FORCE_INTERNAL', new=False):
self.assertEqual(utils.Timestamp(0).internal, '0000000000.00000')
self.assertEqual(utils.Timestamp(1402437380.58186).internal,
'1402437380.58186')
self.assertEqual(utils.Timestamp(1402437380.581859).internal,
'1402437380.58186')
self.assertEqual(utils.Timestamp(0).internal,
utils.normalize_timestamp(0))
def test_no_force_internal_with_offset(self):
"""Test that internal always includes the offset if significant"""
with mock.patch('swift.common.utils.FORCE_INTERNAL', new=False):
self.assertEqual(utils.Timestamp(0, offset=1).internal,
'0000000000.00000_0000000000000001')
self.assertEqual(
utils.Timestamp(1402437380.58186, offset=16).internal,
'1402437380.58186_0000000000000010')
self.assertEqual(
utils.Timestamp(1402437380.581859, offset=240).internal,
'1402437380.58186_00000000000000f0')
self.assertEqual(
utils.Timestamp('1402437380.581859_00000001',
offset=240).internal,
'1402437380.58186_00000000000000f1')
def test_force_internal(self):
"""Test that internal always includes the offset if forced"""
with mock.patch('swift.common.utils.FORCE_INTERNAL', new=True):
self.assertEqual(utils.Timestamp(0).internal,
'0000000000.00000_0000000000000000')
self.assertEqual(utils.Timestamp(1402437380.58186).internal,
'1402437380.58186_0000000000000000')
self.assertEqual(utils.Timestamp(1402437380.581859).internal,
'1402437380.58186_0000000000000000')
self.assertEqual(utils.Timestamp(0, offset=1).internal,
'0000000000.00000_0000000000000001')
self.assertEqual(
utils.Timestamp(1402437380.58186, offset=16).internal,
'1402437380.58186_0000000000000010')
self.assertEqual(
utils.Timestamp(1402437380.581859, offset=16).internal,
'1402437380.58186_0000000000000010')
def test_internal_format_no_offset(self):
expected = '1402436408.91203_0000000000000000'
test_values = (
'1402436408.91203',
'1402436408.91203_00000000',
'1402436408.912030000',
'1402436408.912030000_0000000000000',
'000001402436408.912030000',
'000001402436408.912030000_0000000000',
1402436408.91203,
1402436408.9120300000000000,
1402436408.912029,
1402436408.912029999999999999,
utils.Timestamp(1402436408.91203),
utils.Timestamp(1402436408.91203, offset=0),
utils.Timestamp(1402436408.912029),
utils.Timestamp(1402436408.91202999999999999, offset=0),
utils.Timestamp('1402436408.91203'),
utils.Timestamp('1402436408.91203', offset=0),
utils.Timestamp('1402436408.912029'),
utils.Timestamp('1402436408.912029', offset=0),
utils.Timestamp('1402436408.912029999999999'),
utils.Timestamp('1402436408.912029999999999', offset=0),
)
for value in test_values:
# timestamp instance is always equivalent
self.assertEqual(utils.Timestamp(value), expected)
if utils.FORCE_INTERNAL:
# the FORCE_INTERNAL flag makes the internal format always
# include the offset portion of the timestamp even when it's
# not significant and would be bad during upgrades
self.assertEqual(utils.Timestamp(value).internal, expected)
else:
# unless we FORCE_INTERNAL, when there's no offset the
# internal format is equivalent to the normalized format
self.assertEqual(utils.Timestamp(value).internal,
'1402436408.91203')
def test_internal_format_with_offset(self):
expected = '1402436408.91203_00000000000000f0'
test_values = (
'1402436408.91203_000000f0',
'1402436408.912030000_0000000000f0',
'1402436408.912029_000000f0',
'1402436408.91202999999_0000000000f0',
'000001402436408.912030000_000000000f0',
'000001402436408.9120299999_000000000f0',
utils.Timestamp(1402436408.91203, offset=240),
utils.Timestamp(1402436408.912029, offset=240),
utils.Timestamp('1402436408.91203', offset=240),
utils.Timestamp('1402436408.91203_00000000', offset=240),
utils.Timestamp('1402436408.91203_0000000f', offset=225),
utils.Timestamp('1402436408.9120299999', offset=240),
utils.Timestamp('1402436408.9120299999_00000000', offset=240),
utils.Timestamp('1402436408.9120299999_00000010', offset=224),
)
for value in test_values:
timestamp = utils.Timestamp(value)
self.assertEqual(timestamp.internal, expected)
# can compare with offset if the string is internalized
self.assertEqual(timestamp, expected)
# if comparison value only includes the normalized portion and the
# timestamp includes an offset, it is considered greater
normal = utils.Timestamp(expected).normal
self.assertTrue(timestamp > normal,
'%r is not bigger than %r given %r' % (
timestamp, normal, value))
self.assertTrue(timestamp > float(normal),
'%r is not bigger than %f given %r' % (
timestamp, float(normal), value))
Update container on fast-POST This patch makes a number of changes to enable content-type metadata to be updated when using the fast-POST mode of operation, as proposed in the associated spec [1]. * the object server and diskfile are modified to allow content-type to be updated by a POST and the updated value to be stored in .meta files. * the object server accepts PUTs and DELETEs with older timestamps than existing .meta files. This is to be consistent with replication that will leave a later .meta file in place when replicating a .data file. * the diskfile interface is modified to provide accessor methods for the content-type and its timestamp. * the naming of .meta files is modified to encode two timestamps when the .meta file contains a content-type value that was set prior to the latest metadata update; this enables consistency to be achieved when rsync is used for replication. * ssync is modified to sync meta files when content-type differs between local and remote copies of objects. * the object server issues container updates when handling POST requests, notifying the container server of the current immutable metadata (etag, size, hash, swift_bytes), content-type with their respective timestamps, and the mutable metadata timestamp. * the container server maintains the most recently reported values for immutable metadata, content-type and mutable metadata, each with their respective timestamps, in a single db row. * new probe tests verify that replication achieves eventual consistency of containers and objects after discrete updates to content-type and mutable metadata, and that container-sync sync's objects after fast-post updates. [1] spec change-id: I60688efc3df692d3a39557114dca8c5490f7837e Change-Id: Ia597cd460bb5fd40aa92e886e3e18a7542603d01
2015-08-10 10:30:10 -05:00
def test_short_format_with_offset(self):
expected = '1402436408.91203_f0'
timestamp = utils.Timestamp(1402436408.91203, 0xf0)
self.assertEqual(expected, timestamp.short)
expected = '1402436408.91203'
timestamp = utils.Timestamp(1402436408.91203)
self.assertEqual(expected, timestamp.short)
Add POST capability to ssync for .meta files ssync currently does the wrong thing when replicating object dirs containing both a .data and a .meta file. The ssync sender uses a single PUT to send both object content and metadata to the receiver, using the metadata (.meta file) timestamp. This results in the object content timestamp being advanced to the metadata timestamp, potentially overwriting newer object data on the receiver and causing an inconsistency with the container server record for the object. For example, replicating an object dir with {t0.data(etag=x), t2.meta} to a receiver with t1.data(etag=y) will result in the creation of t2.data(etag=x) on the receiver. However, the container server will continue to list the object as t1(etag=y). This patch modifies ssync to replicate the content of .data and .meta separately using a PUT request for the data (no change) and a POST request for the metadata. In effect, ssync replication replicates the client operations that generated the .data and .meta files so that the result of replication is the same as if the original client requests had persisted on all object servers. Apart from maintaining correct timestamps across sync'd nodes, this has the added benefit of not needing to PUT objects when only the metadata has changed and a POST will suffice. Taking the same example, ssync sender will no longer PUT t0.data but will POST t2.meta resulting in the receiver having t1.data and t2.meta. The changes are backwards compatible: an upgraded sender will only sync data files to a legacy receiver and will not sync meta files (fixing the erroneous behavior described above); a legacy sender will operate as before when sync'ing to an upgraded receiver. Changes: - diskfile API provides methods to get the data file timestamp as distinct from the diskfile timestamp. - diskfile yield_hashes return tuple now passes a dict mapping data and meta (if any) timestamps to their respective values in the timestamp field. - ssync_sender will encode data and meta timestamps in the (hash_path, timestamp) tuple sent to the receiver during missing_checks. - ssync_receiver compares sender's data and meta timestamps to any local diskfile and may specify that only data or meta parts are sent during updates phase by appending a qualifier to the hash returned in its 'wanted' list. - ssync_sender now sends POST subrequests when a meta file exists and its content needs to be replicated. - ssync_sender may send *only* a POST if the receiver indicates that is the only part required to be sync'd. - object server will allow PUT and DELETE with earlier timestamp than a POST - Fixed TODO related to replicated objects with fast-POST and ssync Related spec change-id: I60688efc3df692d3a39557114dca8c5490f7837e Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Closes-Bug: 1501528 Change-Id: I97552d194e5cc342b0a3f4b9800de8aa6b9cb85b
2015-04-22 12:56:50 +01:00
def test_raw(self):
expected = 140243640891203
timestamp = utils.Timestamp(1402436408.91203)
self.assertEqual(expected, timestamp.raw)
# 'raw' does not include offset
timestamp = utils.Timestamp(1402436408.91203, 0xf0)
self.assertEqual(expected, timestamp.raw)
def test_delta(self):
def _assertWithinBounds(expected, timestamp):
tolerance = 0.00001
minimum = expected - tolerance
maximum = expected + tolerance
self.assertTrue(float(timestamp) > minimum)
self.assertTrue(float(timestamp) < maximum)
timestamp = utils.Timestamp(1402436408.91203, delta=100)
_assertWithinBounds(1402436408.91303, timestamp)
self.assertEqual(140243640891303, timestamp.raw)
timestamp = utils.Timestamp(1402436408.91203, delta=-100)
_assertWithinBounds(1402436408.91103, timestamp)
self.assertEqual(140243640891103, timestamp.raw)
timestamp = utils.Timestamp(1402436408.91203, delta=0)
_assertWithinBounds(1402436408.91203, timestamp)
self.assertEqual(140243640891203, timestamp.raw)
# delta is independent of offset
timestamp = utils.Timestamp(1402436408.91203, offset=42, delta=100)
self.assertEqual(140243640891303, timestamp.raw)
self.assertEqual(42, timestamp.offset)
# cannot go negative
self.assertRaises(ValueError, utils.Timestamp, 1402436408.91203,
delta=-140243640891203)
Add two vector timestamps The normalized form of the X-Timestamp header looks like a float with a fixed width to ensure stable string sorting - normalized timestamps look like "1402464677.04188" To support overwrites of existing data without modifying the original timestamp but still maintain consistency a second internal offset vector is append to the normalized timestamp form which compares and sorts greater than the fixed width float format but less than a newer timestamp. The internalized format of timestamps looks like "1402464677.04188_0000000000000000" - the portion after the underscore is the offset and is a formatted hexadecimal integer. The internalized form is not exposed to clients in responses from Swift. Normal client operations will not create a timestamp with an offset. The Timestamp class in common.utils supports internalized and normalized formatting of timestamps and also comparison of timestamp values. When the offset value of a Timestamp is 0 - it's considered insignificant and need not be represented in the string format; to support backwards compatibility during a Swift upgrade the internalized and normalized form of a Timestamp with an insignificant offset are identical. When a timestamp includes an offset it will always be represented in the internalized form, but is still excluded from the normalized form. Timestamps with an equivalent timestamp portion (the float part) will compare and order by their offset. Timestamps with a greater timestamp portion will always compare and order greater than a Timestamp with a lesser timestamp regardless of it's offset. String comparison and ordering is guaranteed for the internalized string format, and is backwards compatible for normalized timestamps which do not include an offset. The reconciler currently uses a offset bump to ensure that objects can move to the wrong storage policy and be moved back. This use-case is valid because the content represented by the user-facing timestamp is not modified in way. Future consumers of the offset vector of timestamps should be mindful of HTTP semantics of If-Modified and take care to avoid deviation in the response from the object server without an accompanying change to the user facing timestamp. DocImpact Implements: blueprint storage-policies Change-Id: Id85c960b126ec919a481dc62469bf172b7fb8549
2014-06-10 22:17:47 -07:00
def test_int(self):
expected = 1402437965
test_values = (
'1402437965.91203',
'1402437965.91203_00000000',
'1402437965.912030000',
'1402437965.912030000_0000000000000',
'000001402437965.912030000',
'000001402437965.912030000_0000000000',
1402437965.91203,
1402437965.9120300000000000,
1402437965.912029,
1402437965.912029999999999999,
utils.Timestamp(1402437965.91203),
utils.Timestamp(1402437965.91203, offset=0),
utils.Timestamp(1402437965.91203, offset=500),
utils.Timestamp(1402437965.912029),
utils.Timestamp(1402437965.91202999999999999, offset=0),
utils.Timestamp(1402437965.91202999999999999, offset=300),
utils.Timestamp('1402437965.91203'),
utils.Timestamp('1402437965.91203', offset=0),
utils.Timestamp('1402437965.91203', offset=400),
utils.Timestamp('1402437965.912029'),
utils.Timestamp('1402437965.912029', offset=0),
utils.Timestamp('1402437965.912029', offset=200),
utils.Timestamp('1402437965.912029999999999'),
utils.Timestamp('1402437965.912029999999999', offset=0),
utils.Timestamp('1402437965.912029999999999', offset=100),
)
for value in test_values:
timestamp = utils.Timestamp(value)
self.assertEqual(int(timestamp), expected)
self.assertTrue(timestamp > expected)
def test_float(self):
expected = 1402438115.91203
test_values = (
'1402438115.91203',
'1402438115.91203_00000000',
'1402438115.912030000',
'1402438115.912030000_0000000000000',
'000001402438115.912030000',
'000001402438115.912030000_0000000000',
1402438115.91203,
1402438115.9120300000000000,
1402438115.912029,
1402438115.912029999999999999,
utils.Timestamp(1402438115.91203),
utils.Timestamp(1402438115.91203, offset=0),
utils.Timestamp(1402438115.91203, offset=500),
utils.Timestamp(1402438115.912029),
utils.Timestamp(1402438115.91202999999999999, offset=0),
utils.Timestamp(1402438115.91202999999999999, offset=300),
utils.Timestamp('1402438115.91203'),
utils.Timestamp('1402438115.91203', offset=0),
utils.Timestamp('1402438115.91203', offset=400),
utils.Timestamp('1402438115.912029'),
utils.Timestamp('1402438115.912029', offset=0),
utils.Timestamp('1402438115.912029', offset=200),
utils.Timestamp('1402438115.912029999999999'),
utils.Timestamp('1402438115.912029999999999', offset=0),
utils.Timestamp('1402438115.912029999999999', offset=100),
)
tolerance = 0.00001
minimum = expected - tolerance
maximum = expected + tolerance
for value in test_values:
timestamp = utils.Timestamp(value)
self.assertTrue(float(timestamp) > minimum,
'%f is not bigger than %f given %r' % (
timestamp, minimum, value))
self.assertTrue(float(timestamp) < maximum,
'%f is not smaller than %f given %r' % (
timestamp, maximum, value))
Spelling mistakes corrected in comments. Change-Id: Ibbd7511c3a2b08519feb4db18eca6e000603ea32
2014-09-03 10:40:30 -07:00
# direct comparison of timestamp works too
Add two vector timestamps The normalized form of the X-Timestamp header looks like a float with a fixed width to ensure stable string sorting - normalized timestamps look like "1402464677.04188" To support overwrites of existing data without modifying the original timestamp but still maintain consistency a second internal offset vector is append to the normalized timestamp form which compares and sorts greater than the fixed width float format but less than a newer timestamp. The internalized format of timestamps looks like "1402464677.04188_0000000000000000" - the portion after the underscore is the offset and is a formatted hexadecimal integer. The internalized form is not exposed to clients in responses from Swift. Normal client operations will not create a timestamp with an offset. The Timestamp class in common.utils supports internalized and normalized formatting of timestamps and also comparison of timestamp values. When the offset value of a Timestamp is 0 - it's considered insignificant and need not be represented in the string format; to support backwards compatibility during a Swift upgrade the internalized and normalized form of a Timestamp with an insignificant offset are identical. When a timestamp includes an offset it will always be represented in the internalized form, but is still excluded from the normalized form. Timestamps with an equivalent timestamp portion (the float part) will compare and order by their offset. Timestamps with a greater timestamp portion will always compare and order greater than a Timestamp with a lesser timestamp regardless of it's offset. String comparison and ordering is guaranteed for the internalized string format, and is backwards compatible for normalized timestamps which do not include an offset. The reconciler currently uses a offset bump to ensure that objects can move to the wrong storage policy and be moved back. This use-case is valid because the content represented by the user-facing timestamp is not modified in way. Future consumers of the offset vector of timestamps should be mindful of HTTP semantics of If-Modified and take care to avoid deviation in the response from the object server without an accompanying change to the user facing timestamp. DocImpact Implements: blueprint storage-policies Change-Id: Id85c960b126ec919a481dc62469bf172b7fb8549
2014-06-10 22:17:47 -07:00
self.assertTrue(timestamp > minimum,
'%s is not bigger than %f given %r' % (
timestamp.normal, minimum, value))
self.assertTrue(timestamp < maximum,
'%s is not smaller than %f given %r' % (
timestamp.normal, maximum, value))
# ... even against strings
self.assertTrue(timestamp > '%f' % minimum,
'%s is not bigger than %s given %r' % (
timestamp.normal, minimum, value))
self.assertTrue(timestamp < '%f' % maximum,
'%s is not smaller than %s given %r' % (
timestamp.normal, maximum, value))
def test_false(self):
self.assertFalse(utils.Timestamp(0))
self.assertFalse(utils.Timestamp(0, offset=0))
self.assertFalse(utils.Timestamp('0'))
self.assertFalse(utils.Timestamp('0', offset=0))
self.assertFalse(utils.Timestamp(0.0))
self.assertFalse(utils.Timestamp(0.0, offset=0))
self.assertFalse(utils.Timestamp('0.0'))
self.assertFalse(utils.Timestamp('0.0', offset=0))
self.assertFalse(utils.Timestamp(00000000.00000000))
self.assertFalse(utils.Timestamp(00000000.00000000, offset=0))
self.assertFalse(utils.Timestamp('00000000.00000000'))
self.assertFalse(utils.Timestamp('00000000.00000000', offset=0))
def test_true(self):
self.assertTrue(utils.Timestamp(1))
self.assertTrue(utils.Timestamp(1, offset=1))
self.assertTrue(utils.Timestamp(0, offset=1))
self.assertTrue(utils.Timestamp('1'))
self.assertTrue(utils.Timestamp('1', offset=1))
self.assertTrue(utils.Timestamp('0', offset=1))
self.assertTrue(utils.Timestamp(1.1))
self.assertTrue(utils.Timestamp(1.1, offset=1))
self.assertTrue(utils.Timestamp(0.0, offset=1))
self.assertTrue(utils.Timestamp('1.1'))
self.assertTrue(utils.Timestamp('1.1', offset=1))
self.assertTrue(utils.Timestamp('0.0', offset=1))
self.assertTrue(utils.Timestamp(11111111.11111111))
self.assertTrue(utils.Timestamp(11111111.11111111, offset=1))
self.assertTrue(utils.Timestamp(00000000.00000000, offset=1))
self.assertTrue(utils.Timestamp('11111111.11111111'))
self.assertTrue(utils.Timestamp('11111111.11111111', offset=1))
self.assertTrue(utils.Timestamp('00000000.00000000', offset=1))
def test_greater_no_offset(self):
now = time.time()
older = now - 1
timestamp = utils.Timestamp(now)
test_values = (
0, '0', 0.0, '0.0', '0000.0000', '000.000_000',
1, '1', 1.1, '1.1', '1111.1111', '111.111_111',
1402443112.213252, '1402443112.213252', '1402443112.213252_ffff',
older, '%f' % older, '%f_0000ffff' % older,
)
for value in test_values:
other = utils.Timestamp(value)
self.assertNotEqual(timestamp, other) # sanity
self.assertTrue(timestamp > value,
'%r is not greater than %r given %r' % (
timestamp, value, value))
self.assertTrue(timestamp > other,
'%r is not greater than %r given %r' % (
timestamp, other, value))
self.assertTrue(timestamp > other.normal,
'%r is not greater than %r given %r' % (
timestamp, other.normal, value))
self.assertTrue(timestamp > other.internal,
'%r is not greater than %r given %r' % (
timestamp, other.internal, value))
self.assertTrue(timestamp > float(other),
'%r is not greater than %r given %r' % (
timestamp, float(other), value))
self.assertTrue(timestamp > int(other),
'%r is not greater than %r given %r' % (
timestamp, int(other), value))
def test_greater_with_offset(self):
now = time.time()
older = now - 1
test_values = (
0, '0', 0.0, '0.0', '0000.0000', '000.000_000',
1, '1', 1.1, '1.1', '1111.1111', '111.111_111',
1402443346.935174, '1402443346.93517', '1402443346.935169_ffff',
older, '%f' % older, '%f_0000ffff' % older,
now, '%f' % now, '%f_00000000' % now,
)
for offset in range(1, 1000, 100):
timestamp = utils.Timestamp(now, offset=offset)
for value in test_values:
other = utils.Timestamp(value)
self.assertNotEqual(timestamp, other) # sanity
self.assertTrue(timestamp > value,
'%r is not greater than %r given %r' % (
timestamp, value, value))
self.assertTrue(timestamp > other,
'%r is not greater than %r given %r' % (
timestamp, other, value))
self.assertTrue(timestamp > other.normal,
'%r is not greater than %r given %r' % (
timestamp, other.normal, value))
self.assertTrue(timestamp > other.internal,
'%r is not greater than %r given %r' % (
timestamp, other.internal, value))
self.assertTrue(timestamp > float(other),
'%r is not greater than %r given %r' % (
timestamp, float(other), value))
self.assertTrue(timestamp > int(other),
'%r is not greater than %r given %r' % (
timestamp, int(other), value))
def test_smaller_no_offset(self):
now = time.time()
newer = now + 1
timestamp = utils.Timestamp(now)
test_values = (
9999999999.99999, '9999999999.99999', '9999999999.99999_ffff',
newer, '%f' % newer, '%f_0000ffff' % newer,
)
for value in test_values:
other = utils.Timestamp(value)
self.assertNotEqual(timestamp, other) # sanity
self.assertTrue(timestamp < value,
'%r is not smaller than %r given %r' % (
timestamp, value, value))
self.assertTrue(timestamp < other,
'%r is not smaller than %r given %r' % (
timestamp, other, value))
self.assertTrue(timestamp < other.normal,
'%r is not smaller than %r given %r' % (
timestamp, other.normal, value))
self.assertTrue(timestamp < other.internal,
'%r is not smaller than %r given %r' % (
timestamp, other.internal, value))
self.assertTrue(timestamp < float(other),
'%r is not smaller than %r given %r' % (
timestamp, float(other), value))
self.assertTrue(timestamp < int(other),
'%r is not smaller than %r given %r' % (
timestamp, int(other), value))
def test_smaller_with_offset(self):
now = time.time()
newer = now + 1
test_values = (
9999999999.99999, '9999999999.99999', '9999999999.99999_ffff',
newer, '%f' % newer, '%f_0000ffff' % newer,
)
for offset in range(1, 1000, 100):
timestamp = utils.Timestamp(now, offset=offset)
for value in test_values:
other = utils.Timestamp(value)
self.assertNotEqual(timestamp, other) # sanity
self.assertTrue(timestamp < value,
'%r is not smaller than %r given %r' % (
timestamp, value, value))
self.assertTrue(timestamp < other,
'%r is not smaller than %r given %r' % (
timestamp, other, value))
self.assertTrue(timestamp < other.normal,
'%r is not smaller than %r given %r' % (
timestamp, other.normal, value))
self.assertTrue(timestamp < other.internal,
'%r is not smaller than %r given %r' % (
timestamp, other.internal, value))
self.assertTrue(timestamp < float(other),
'%r is not smaller than %r given %r' % (
timestamp, float(other), value))
self.assertTrue(timestamp < int(other),
'%r is not smaller than %r given %r' % (
timestamp, int(other), value))
Update container on fast-POST This patch makes a number of changes to enable content-type metadata to be updated when using the fast-POST mode of operation, as proposed in the associated spec [1]. * the object server and diskfile are modified to allow content-type to be updated by a POST and the updated value to be stored in .meta files. * the object server accepts PUTs and DELETEs with older timestamps than existing .meta files. This is to be consistent with replication that will leave a later .meta file in place when replicating a .data file. * the diskfile interface is modified to provide accessor methods for the content-type and its timestamp. * the naming of .meta files is modified to encode two timestamps when the .meta file contains a content-type value that was set prior to the latest metadata update; this enables consistency to be achieved when rsync is used for replication. * ssync is modified to sync meta files when content-type differs between local and remote copies of objects. * the object server issues container updates when handling POST requests, notifying the container server of the current immutable metadata (etag, size, hash, swift_bytes), content-type with their respective timestamps, and the mutable metadata timestamp. * the container server maintains the most recently reported values for immutable metadata, content-type and mutable metadata, each with their respective timestamps, in a single db row. * new probe tests verify that replication achieves eventual consistency of containers and objects after discrete updates to content-type and mutable metadata, and that container-sync sync's objects after fast-post updates. [1] spec change-id: I60688efc3df692d3a39557114dca8c5490f7837e Change-Id: Ia597cd460bb5fd40aa92e886e3e18a7542603d01
2015-08-10 10:30:10 -05:00
def test_cmp_with_none(self):
self.assertGreater(utils.Timestamp(0), None)
self.assertGreater(utils.Timestamp(1.0), None)
self.assertGreater(utils.Timestamp(1.0, 42), None)
Add two vector timestamps The normalized form of the X-Timestamp header looks like a float with a fixed width to ensure stable string sorting - normalized timestamps look like "1402464677.04188" To support overwrites of existing data without modifying the original timestamp but still maintain consistency a second internal offset vector is append to the normalized timestamp form which compares and sorts greater than the fixed width float format but less than a newer timestamp. The internalized format of timestamps looks like "1402464677.04188_0000000000000000" - the portion after the underscore is the offset and is a formatted hexadecimal integer. The internalized form is not exposed to clients in responses from Swift. Normal client operations will not create a timestamp with an offset. The Timestamp class in common.utils supports internalized and normalized formatting of timestamps and also comparison of timestamp values. When the offset value of a Timestamp is 0 - it's considered insignificant and need not be represented in the string format; to support backwards compatibility during a Swift upgrade the internalized and normalized form of a Timestamp with an insignificant offset are identical. When a timestamp includes an offset it will always be represented in the internalized form, but is still excluded from the normalized form. Timestamps with an equivalent timestamp portion (the float part) will compare and order by their offset. Timestamps with a greater timestamp portion will always compare and order greater than a Timestamp with a lesser timestamp regardless of it's offset. String comparison and ordering is guaranteed for the internalized string format, and is backwards compatible for normalized timestamps which do not include an offset. The reconciler currently uses a offset bump to ensure that objects can move to the wrong storage policy and be moved back. This use-case is valid because the content represented by the user-facing timestamp is not modified in way. Future consumers of the offset vector of timestamps should be mindful of HTTP semantics of If-Modified and take care to avoid deviation in the response from the object server without an accompanying change to the user facing timestamp. DocImpact Implements: blueprint storage-policies Change-Id: Id85c960b126ec919a481dc62469bf172b7fb8549
2014-06-10 22:17:47 -07:00
def test_ordering(self):
given = [
'1402444820.62590_000000000000000a',
'1402444820.62589_0000000000000001',
'1402444821.52589_0000000000000004',
'1402444920.62589_0000000000000004',
'1402444821.62589_000000000000000a',
'1402444821.72589_000000000000000a',
'1402444920.62589_0000000000000002',
'1402444820.62589_0000000000000002',
'1402444820.62589_000000000000000a',
'1402444820.62590_0000000000000004',
'1402444920.62589_000000000000000a',
'1402444820.62590_0000000000000002',
'1402444821.52589_0000000000000002',
'1402444821.52589_0000000000000000',
'1402444920.62589',
'1402444821.62589_0000000000000004',
'1402444821.72589_0000000000000001',
'1402444820.62590',
'1402444820.62590_0000000000000001',
'1402444820.62589_0000000000000004',
'1402444821.72589_0000000000000000',
'1402444821.52589_000000000000000a',
'1402444821.72589_0000000000000004',
'1402444821.62589',
'1402444821.52589_0000000000000001',
'1402444821.62589_0000000000000001',
'1402444821.62589_0000000000000002',
'1402444821.72589_0000000000000002',
'1402444820.62589',
'1402444920.62589_0000000000000001']
expected = [
'1402444820.62589',
'1402444820.62589_0000000000000001',
'1402444820.62589_0000000000000002',
'1402444820.62589_0000000000000004',
'1402444820.62589_000000000000000a',
'1402444820.62590',
'1402444820.62590_0000000000000001',
'1402444820.62590_0000000000000002',
'1402444820.62590_0000000000000004',
'1402444820.62590_000000000000000a',
'1402444821.52589',
'1402444821.52589_0000000000000001',
'1402444821.52589_0000000000000002',
'1402444821.52589_0000000000000004',
'1402444821.52589_000000000000000a',
'1402444821.62589',
'1402444821.62589_0000000000000001',
'1402444821.62589_0000000000000002',
'1402444821.62589_0000000000000004',
'1402444821.62589_000000000000000a',
'1402444821.72589',
'1402444821.72589_0000000000000001',
'1402444821.72589_0000000000000002',
'1402444821.72589_0000000000000004',
'1402444821.72589_000000000000000a',
'1402444920.62589',
'1402444920.62589_0000000000000001',
'1402444920.62589_0000000000000002',
'1402444920.62589_0000000000000004',
'1402444920.62589_000000000000000a',
]
# less visual version
"""
now = time.time()
given = [
utils.Timestamp(now + i, offset=offset).internal
for i in (0, 0.00001, 0.9, 1.0, 1.1, 100.0)
for offset in (0, 1, 2, 4, 10)
]
expected = [t for t in given]
random.shuffle(given)
"""
self.assertEqual(len(given), len(expected)) # sanity
timestamps = [utils.Timestamp(t) for t in given]
# our expected values don't include insignificant offsets
with mock.patch('swift.common.utils.FORCE_INTERNAL', new=False):
self.assertEqual(
[t.internal for t in sorted(timestamps)], expected)
# string sorting works as well
self.assertEqual(
sorted([t.internal for t in timestamps]), expected)
Make ECDiskFile report all fragments found on disk Refactor the disk file get_ondisk_files logic to enable ECDiskfile to gather *all* fragments found on disk (not just those with a matching .durable file) and make the fragments available via the DiskFile interface as a dict mapping: Timestamp --> list of fragment indexes Also, if a durable fragment has been found then the timestamp of the durable file is exposed via the diskfile interface. Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Change-Id: I55e20a999685b94023d47b231d51007045ac920e
2015-10-05 16:15:29 +01:00
def test_hashable(self):
ts_0 = utils.Timestamp('1402444821.72589')
ts_0_also = utils.Timestamp('1402444821.72589')
self.assertEqual(ts_0, ts_0_also) # sanity
self.assertEqual(hash(ts_0), hash(ts_0_also))
d = {ts_0: 'whatever'}
self.assertIn(ts_0, d) # sanity
self.assertIn(ts_0_also, d)
Add two vector timestamps The normalized form of the X-Timestamp header looks like a float with a fixed width to ensure stable string sorting - normalized timestamps look like "1402464677.04188" To support overwrites of existing data without modifying the original timestamp but still maintain consistency a second internal offset vector is append to the normalized timestamp form which compares and sorts greater than the fixed width float format but less than a newer timestamp. The internalized format of timestamps looks like "1402464677.04188_0000000000000000" - the portion after the underscore is the offset and is a formatted hexadecimal integer. The internalized form is not exposed to clients in responses from Swift. Normal client operations will not create a timestamp with an offset. The Timestamp class in common.utils supports internalized and normalized formatting of timestamps and also comparison of timestamp values. When the offset value of a Timestamp is 0 - it's considered insignificant and need not be represented in the string format; to support backwards compatibility during a Swift upgrade the internalized and normalized form of a Timestamp with an insignificant offset are identical. When a timestamp includes an offset it will always be represented in the internalized form, but is still excluded from the normalized form. Timestamps with an equivalent timestamp portion (the float part) will compare and order by their offset. Timestamps with a greater timestamp portion will always compare and order greater than a Timestamp with a lesser timestamp regardless of it's offset. String comparison and ordering is guaranteed for the internalized string format, and is backwards compatible for normalized timestamps which do not include an offset. The reconciler currently uses a offset bump to ensure that objects can move to the wrong storage policy and be moved back. This use-case is valid because the content represented by the user-facing timestamp is not modified in way. Future consumers of the offset vector of timestamps should be mindful of HTTP semantics of If-Modified and take care to avoid deviation in the response from the object server without an accompanying change to the user facing timestamp. DocImpact Implements: blueprint storage-policies Change-Id: Id85c960b126ec919a481dc62469bf172b7fb8549
2014-06-10 22:17:47 -07:00
Update container on fast-POST This patch makes a number of changes to enable content-type metadata to be updated when using the fast-POST mode of operation, as proposed in the associated spec [1]. * the object server and diskfile are modified to allow content-type to be updated by a POST and the updated value to be stored in .meta files. * the object server accepts PUTs and DELETEs with older timestamps than existing .meta files. This is to be consistent with replication that will leave a later .meta file in place when replicating a .data file. * the diskfile interface is modified to provide accessor methods for the content-type and its timestamp. * the naming of .meta files is modified to encode two timestamps when the .meta file contains a content-type value that was set prior to the latest metadata update; this enables consistency to be achieved when rsync is used for replication. * ssync is modified to sync meta files when content-type differs between local and remote copies of objects. * the object server issues container updates when handling POST requests, notifying the container server of the current immutable metadata (etag, size, hash, swift_bytes), content-type with their respective timestamps, and the mutable metadata timestamp. * the container server maintains the most recently reported values for immutable metadata, content-type and mutable metadata, each with their respective timestamps, in a single db row. * new probe tests verify that replication achieves eventual consistency of containers and objects after discrete updates to content-type and mutable metadata, and that container-sync sync's objects after fast-post updates. [1] spec change-id: I60688efc3df692d3a39557114dca8c5490f7837e Change-Id: Ia597cd460bb5fd40aa92e886e3e18a7542603d01
2015-08-10 10:30:10 -05:00
class TestTimestampEncoding(unittest.TestCase):
def setUp(self):
t0 = utils.Timestamp(0.0)
t1 = utils.Timestamp(997.9996)
t2 = utils.Timestamp(999)
t3 = utils.Timestamp(1000, 24)
t4 = utils.Timestamp(1001)
t5 = utils.Timestamp(1002.00040)
# encodings that are expected when explicit = False
self.non_explicit_encodings = (
('0000001000.00000_18', (t3, t3, t3)),
('0000001000.00000_18', (t3, t3, None)),
)
# mappings that are expected when explicit = True
self.explicit_encodings = (
('0000001000.00000_18+0+0', (t3, t3, t3)),
('0000001000.00000_18+0', (t3, t3, None)),
)
# mappings that are expected when explicit = True or False
self.encodings = (
('0000001000.00000_18+0+186a0', (t3, t3, t4)),
('0000001000.00000_18+186a0+186c8', (t3, t4, t5)),
('0000001000.00000_18-186a0+0', (t3, t2, t2)),
('0000001000.00000_18+0-186a0', (t3, t3, t2)),
('0000001000.00000_18-186a0-186c8', (t3, t2, t1)),
('0000001000.00000_18', (t3, None, None)),
('0000001000.00000_18+186a0', (t3, t4, None)),
('0000001000.00000_18-186a0', (t3, t2, None)),
('0000001000.00000_18', (t3, None, t1)),
('0000001000.00000_18-5f5e100', (t3, t0, None)),
('0000001000.00000_18+0-5f5e100', (t3, t3, t0)),
('0000001000.00000_18-5f5e100+5f45a60', (t3, t0, t2)),
)
# decodings that are expected when explicit = False
self.non_explicit_decodings = (
('0000001000.00000_18', (t3, t3, t3)),
('0000001000.00000_18+186a0', (t3, t4, t4)),
('0000001000.00000_18-186a0', (t3, t2, t2)),
('0000001000.00000_18+186a0', (t3, t4, t4)),
('0000001000.00000_18-186a0', (t3, t2, t2)),
('0000001000.00000_18-5f5e100', (t3, t0, t0)),
)
# decodings that are expected when explicit = True
self.explicit_decodings = (
('0000001000.00000_18+0+0', (t3, t3, t3)),
('0000001000.00000_18+0', (t3, t3, None)),
('0000001000.00000_18', (t3, None, None)),
('0000001000.00000_18+186a0', (t3, t4, None)),
('0000001000.00000_18-186a0', (t3, t2, None)),
('0000001000.00000_18-5f5e100', (t3, t0, None)),
)
# decodings that are expected when explicit = True or False
self.decodings = (
('0000001000.00000_18+0+186a0', (t3, t3, t4)),
('0000001000.00000_18+186a0+186c8', (t3, t4, t5)),
('0000001000.00000_18-186a0+0', (t3, t2, t2)),
('0000001000.00000_18+0-186a0', (t3, t3, t2)),
('0000001000.00000_18-186a0-186c8', (t3, t2, t1)),
('0000001000.00000_18-5f5e100+5f45a60', (t3, t0, t2)),
)
def _assertEqual(self, expected, actual, test):
self.assertEqual(expected, actual,
'Got %s but expected %s for parameters %s'
% (actual, expected, test))
def test_encoding(self):
for test in self.explicit_encodings:
actual = utils.encode_timestamps(test[1][0], test[1][1],
test[1][2], True)
self._assertEqual(test[0], actual, test[1])
for test in self.non_explicit_encodings:
actual = utils.encode_timestamps(test[1][0], test[1][1],
test[1][2], False)
self._assertEqual(test[0], actual, test[1])
for explicit in (True, False):
for test in self.encodings:
actual = utils.encode_timestamps(test[1][0], test[1][1],
test[1][2], explicit)
self._assertEqual(test[0], actual, test[1])
def test_decoding(self):
for test in self.explicit_decodings:
actual = utils.decode_timestamps(test[0], True)
self._assertEqual(test[1], actual, test[0])
for test in self.non_explicit_decodings:
actual = utils.decode_timestamps(test[0], False)
self._assertEqual(test[1], actual, test[0])
for explicit in (True, False):
for test in self.decodings:
actual = utils.decode_timestamps(test[0], explicit)
self._assertEqual(test[1], actual, test[0])
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
class TestUtils(unittest.TestCase):
Pep8 unit test modules for hacking and one liners (4 of 12) Address all the "hacking" lines that are flagged, and all the modules that just have one item flagged. Change-Id: I372a4bdf9c7748f73e38c4fd55e5954f1afade5b Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-08-31 22:36:58 -04:00
"""Tests for swift.common.utils """
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
Revert "Refactor common/utils methods to common/ondisk" This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32 Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 12:10:31 +00:00
def setUp(self):
utils.HASH_PATH_SUFFIX = 'endcap'
utils.HASH_PATH_PREFIX = 'startcap'
Per device replication_lock New replication_one_per_device (True by default) that restricts incoming REPLICATION requests to one per device, replication_currency allowing. Also has replication_lock_timeout (15 by default) to control how long a request will wait to obtain a replication device lock before giving up. This should be very useful in that you can be assured any concurrent REPLICATION requests are each writing to distinct devices. If you have 100 devices on a server, you can set replication_concurrency to 100 and be confident that, even if 100 replication requests were executing concurrently, they'd each be writing to separate devices. Before, all 100 could end up writing to the same device, bringing it to a horrible crawl. NOTE: This is only for ssync replication. The current default rsync replication still has the potentially horrible behavior. Change-Id: I36e99a3d7e100699c76db6d3a4846514537ff685
2013-11-09 03:18:11 +00:00
def test_lock_path(self):
tmpdir = mkdtemp()
try:
with utils.lock_path(tmpdir, 0.1):
exc = None
success = False
try:
with utils.lock_path(tmpdir, 0.1):
success = True
except LockTimeout as err:
exc = err
self.assertTrue(exc is not None)
self.assertTrue(not success)
finally:
shutil.rmtree(tmpdir)
Sleep for longer at a time in lock_path. When lock_path is called and the lock goes for the whole 10 seconds, the flock is called 1000 times. With this patch, the short 0.01 sleep is used for the first 1% of the total lock time and then 1% of the total lock time is used. Change-Id: Ibed6bdb49bddcdb868742c41f86d2482a7edfd29
2014-08-11 09:43:13 -07:00
def test_lock_path_num_sleeps(self):
tmpdir = mkdtemp()
num_short_calls = [0]
exception_raised = [False]
def my_sleep(to_sleep):
if to_sleep == 0.01:
num_short_calls[0] += 1
else:
raise Exception('sleep time changed: %s' % to_sleep)
try:
with mock.patch('swift.common.utils.sleep', my_sleep):
with utils.lock_path(tmpdir):
with utils.lock_path(tmpdir):
pass
except Exception as e:
exception_raised[0] = True
self.assertTrue('sleep time changed' in str(e))
finally:
shutil.rmtree(tmpdir)
self.assertEqual(num_short_calls[0], 11)
self.assertTrue(exception_raised[0])
Per device replication_lock New replication_one_per_device (True by default) that restricts incoming REPLICATION requests to one per device, replication_currency allowing. Also has replication_lock_timeout (15 by default) to control how long a request will wait to obtain a replication device lock before giving up. This should be very useful in that you can be assured any concurrent REPLICATION requests are each writing to distinct devices. If you have 100 devices on a server, you can set replication_concurrency to 100 and be confident that, even if 100 replication requests were executing concurrently, they'd each be writing to separate devices. Before, all 100 could end up writing to the same device, bringing it to a horrible crawl. NOTE: This is only for ssync replication. The current default rsync replication still has the potentially horrible behavior. Change-Id: I36e99a3d7e100699c76db6d3a4846514537ff685
2013-11-09 03:18:11 +00:00
def test_lock_path_class(self):
tmpdir = mkdtemp()
try:
with utils.lock_path(tmpdir, 0.1, ReplicationLockTimeout):
exc = None
exc2 = None
success = False
try:
with utils.lock_path(tmpdir, 0.1, ReplicationLockTimeout):
success = True
except ReplicationLockTimeout as err:
exc = err
except LockTimeout as err:
exc2 = err
self.assertTrue(exc is not None)
self.assertTrue(exc2 is None)
self.assertTrue(not success)
exc = None
exc2 = None
success = False
try:
with utils.lock_path(tmpdir, 0.1):
success = True
except ReplicationLockTimeout as err:
exc = err
except LockTimeout as err:
exc2 = err
self.assertTrue(exc is None)
self.assertTrue(exc2 is not None)
self.assertTrue(not success)
finally:
shutil.rmtree(tmpdir)
Revert "Refactor common/utils methods to common/ondisk" This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32 Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 12:10:31 +00:00
def test_normalize_timestamp(self):
# Test swift.common.utils.normalize_timestamp
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils.normalize_timestamp('1253327593.48174'),
"1253327593.48174")
self.assertEqual(utils.normalize_timestamp(1253327593.48174),
"1253327593.48174")
self.assertEqual(utils.normalize_timestamp('1253327593.48'),
"1253327593.48000")
self.assertEqual(utils.normalize_timestamp(1253327593.48),
"1253327593.48000")
self.assertEqual(utils.normalize_timestamp('253327593.48'),
"0253327593.48000")
self.assertEqual(utils.normalize_timestamp(253327593.48),
"0253327593.48000")
self.assertEqual(utils.normalize_timestamp('1253327593'),
"1253327593.00000")
self.assertEqual(utils.normalize_timestamp(1253327593),
"1253327593.00000")
Revert "Refactor common/utils methods to common/ondisk" This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32 Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 12:10:31 +00:00
self.assertRaises(ValueError, utils.normalize_timestamp, '')
self.assertRaises(ValueError, utils.normalize_timestamp, 'abc')
Consolidating and standardizing x-delete-at format Change-Id: Idc916da1c7fe1cc43a2c26f7f7ee1d4fcdd52c89
2014-01-13 17:45:56 +00:00
def test_normalize_delete_at_timestamp(self):
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Consolidating and standardizing x-delete-at format Change-Id: Idc916da1c7fe1cc43a2c26f7f7ee1d4fcdd52c89
2014-01-13 17:45:56 +00:00
utils.normalize_delete_at_timestamp(1253327593),
'1253327593')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Consolidating and standardizing x-delete-at format Change-Id: Idc916da1c7fe1cc43a2c26f7f7ee1d4fcdd52c89
2014-01-13 17:45:56 +00:00
utils.normalize_delete_at_timestamp(1253327593.67890),
'1253327593')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Consolidating and standardizing x-delete-at format Change-Id: Idc916da1c7fe1cc43a2c26f7f7ee1d4fcdd52c89
2014-01-13 17:45:56 +00:00
utils.normalize_delete_at_timestamp('1253327593'),
'1253327593')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Consolidating and standardizing x-delete-at format Change-Id: Idc916da1c7fe1cc43a2c26f7f7ee1d4fcdd52c89
2014-01-13 17:45:56 +00:00
utils.normalize_delete_at_timestamp('1253327593.67890'),
'1253327593')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Consolidating and standardizing x-delete-at format Change-Id: Idc916da1c7fe1cc43a2c26f7f7ee1d4fcdd52c89
2014-01-13 17:45:56 +00:00
utils.normalize_delete_at_timestamp(-1253327593),
'0000000000')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Consolidating and standardizing x-delete-at format Change-Id: Idc916da1c7fe1cc43a2c26f7f7ee1d4fcdd52c89
2014-01-13 17:45:56 +00:00
utils.normalize_delete_at_timestamp(-1253327593.67890),
'0000000000')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Consolidating and standardizing x-delete-at format Change-Id: Idc916da1c7fe1cc43a2c26f7f7ee1d4fcdd52c89
2014-01-13 17:45:56 +00:00
utils.normalize_delete_at_timestamp('-1253327593'),
'0000000000')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Consolidating and standardizing x-delete-at format Change-Id: Idc916da1c7fe1cc43a2c26f7f7ee1d4fcdd52c89
2014-01-13 17:45:56 +00:00
utils.normalize_delete_at_timestamp('-1253327593.67890'),
'0000000000')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Consolidating and standardizing x-delete-at format Change-Id: Idc916da1c7fe1cc43a2c26f7f7ee1d4fcdd52c89
2014-01-13 17:45:56 +00:00
utils.normalize_delete_at_timestamp(71253327593),
'9999999999')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Consolidating and standardizing x-delete-at format Change-Id: Idc916da1c7fe1cc43a2c26f7f7ee1d4fcdd52c89
2014-01-13 17:45:56 +00:00
utils.normalize_delete_at_timestamp(71253327593.67890),
'9999999999')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Consolidating and standardizing x-delete-at format Change-Id: Idc916da1c7fe1cc43a2c26f7f7ee1d4fcdd52c89
2014-01-13 17:45:56 +00:00
utils.normalize_delete_at_timestamp('71253327593'),
'9999999999')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Consolidating and standardizing x-delete-at format Change-Id: Idc916da1c7fe1cc43a2c26f7f7ee1d4fcdd52c89
2014-01-13 17:45:56 +00:00
utils.normalize_delete_at_timestamp('71253327593.67890'),
'9999999999')
self.assertRaises(ValueError, utils.normalize_timestamp, '')
self.assertRaises(ValueError, utils.normalize_timestamp, 'abc')
Revert "Refactor common/utils methods to common/ondisk" This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32 Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 12:10:31 +00:00
Add container-reconciler daemon This daemon will take objects that are in the wrong storage policy and move them to the right ones, or delete requests that went to the wrong storage policy and apply them to the right ones. It operates on a queue similar to the object-expirer's queue. Discovering that the object is in the wrong policy will be done in subsequent commits by the container replicator; this is the daemon that handles them once they happen. Like the object expirer, you only need to run one of these per cluster see etc/container-reconciler.conf. DocImpact Implements: blueprint storage-policies Change-Id: I5ea62eb77ddcbc7cfebf903429f2ee4c098771c9
2014-04-28 20:03:48 -07:00
def test_last_modified_date_to_timestamp(self):
expectations = {
'1970-01-01T00:00:00.000000': 0.0,
'2014-02-28T23:22:36.698390': 1393629756.698390,
'2011-03-19T04:03:00.604554': 1300507380.604554,
}
for last_modified, ts in expectations.items():
real = utils.last_modified_date_to_timestamp(last_modified)
self.assertEqual(real, ts, "failed for %s" % last_modified)
Fix last_modified_date_to_timestamp on non-UTC systems Before, we were calling datetime.datetime.strftime('%s.%f') to convert a datetime to epoch seconds + microseconds. However, the '%s' format isn't actually part of Python's library. Rather, Python passes that on to the system C library, which is typically glibc. Now, glibc takes the '%s' format and helpfully* applies the current timezone as an offset. This gives bogus results on machines where UTC is not the system timezone. (Yes, some people really do that.) For example: >>> import os >>> from swift.common import utils >>> os.environ['TZ'] = 'PST8PDT,M3.2.0,M11.1.0' >>> float(utils.last_modified_date_to_timestamp('1970-01-01T00:00:00.000000')) 28800.0 >>> That timestamp should obviously be 0. This patch replaces the strftime() call with datetime arithmetic, which is entirely in Python so the system timezone doesn't mess it up. * unhelpfully Change-Id: I56855acd79a5d8f2c98a771fa9fd2729e4f490b1
2014-08-29 15:48:38 -07:00
def test_last_modified_date_to_timestamp_when_system_not_UTC(self):
try:
old_tz = os.environ.get('TZ')
# Western Argentina Summer Time. Found in glibc manual; this
# timezone always has a non-zero offset from UTC, so this test is
# always meaningful.
os.environ['TZ'] = 'WART4WARST,J1/0,J365/25'
self.assertEqual(utils.last_modified_date_to_timestamp(
'1970-01-01T00:00:00.000000'),
0.0)
finally:
if old_tz is not None:
os.environ['TZ'] = old_tz
else:
os.environ.pop('TZ')
Change in swift-drive-audit handling log rotation. Change supports kern.log rotation in order to avoid loss of significant information. There is a year change functionality added as kern.log does not keep record of year. There is also backwards function added which allows reading logs from the back to the front, speeding up the execution along with the unit test for it Fixes Bug 1080682 Change-Id: I93436c405aff5625396514000cab774b66022dd0
2012-11-19 13:02:34 +00:00
def test_backwards(self):
Pep8 unit test modules for hacking and one liners (4 of 12) Address all the "hacking" lines that are flagged, and all the modules that just have one item flagged. Change-Id: I372a4bdf9c7748f73e38c4fd55e5954f1afade5b Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-08-31 22:36:58 -04:00
# Test swift.common.utils.backward
Change in swift-drive-audit handling log rotation. Change supports kern.log rotation in order to avoid loss of significant information. There is a year change functionality added as kern.log does not keep record of year. There is also backwards function added which allows reading logs from the back to the front, speeding up the execution along with the unit test for it Fixes Bug 1080682 Change-Id: I93436c405aff5625396514000cab774b66022dd0
2012-11-19 13:02:34 +00:00
# The lines are designed so that the function would encounter
# all of the boundary conditions and typical conditions.
# Block boundaries are marked with '<>' characters
blocksize = 25
Port swift.common.utils.backward() to Python 3 backward() is written to handle binary files: * Replace literal native strings to literal byte strings: add b'...' prefix * Update unit tests: use byte strings * TemporaryFile(): use the default mode 'w+b', instead of using 'r+w' ('r+w' mode creates a Unicode file on Python 3). Change-Id: Ic91f7e6c605db0b888763080d49f0f501029837f
2015-10-19 16:59:14 +02:00
lines = [b'123456789x12345678><123456789\n', # block larger than rest
b'123456789x123>\n', # block ends just before \n character
b'123423456789\n',
b'123456789x\n', # block ends at the end of line
b'<123456789x123456789x123\n',
b'<6789x123\n', # block ends at the beginning of the line
b'6789x1234\n',
b'1234><234\n', # block ends typically in the middle of line
b'123456789x123456789\n']
with TemporaryFile() as f:
Change in swift-drive-audit handling log rotation. Change supports kern.log rotation in order to avoid loss of significant information. There is a year change functionality added as kern.log does not keep record of year. There is also backwards function added which allows reading logs from the back to the front, speeding up the execution along with the unit test for it Fixes Bug 1080682 Change-Id: I93436c405aff5625396514000cab774b66022dd0
2012-11-19 13:02:34 +00:00
for line in lines:
f.write(line)
count = len(lines) - 1
for line in utils.backward(f, blocksize):
Port swift.common.utils.backward() to Python 3 backward() is written to handle binary files: * Replace literal native strings to literal byte strings: add b'...' prefix * Update unit tests: use byte strings * TemporaryFile(): use the default mode 'w+b', instead of using 'r+w' ('r+w' mode creates a Unicode file on Python 3). Change-Id: Ic91f7e6c605db0b888763080d49f0f501029837f
2015-10-19 16:59:14 +02:00
self.assertEqual(line, lines[count].split(b'\n')[0])
Change in swift-drive-audit handling log rotation. Change supports kern.log rotation in order to avoid loss of significant information. There is a year change functionality added as kern.log does not keep record of year. There is also backwards function added which allows reading logs from the back to the front, speeding up the execution along with the unit test for it Fixes Bug 1080682 Change-Id: I93436c405aff5625396514000cab774b66022dd0
2012-11-19 13:02:34 +00:00
count -= 1
# Empty file case
with TemporaryFile('r') as f:
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual([], list(utils.backward(f)))
Change in swift-drive-audit handling log rotation. Change supports kern.log rotation in order to avoid loss of significant information. There is a year change functionality added as kern.log does not keep record of year. There is also backwards function added which allows reading logs from the back to the front, speeding up the execution along with the unit test for it Fixes Bug 1080682 Change-Id: I93436c405aff5625396514000cab774b66022dd0
2012-11-19 13:02:34 +00:00
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
def test_mkdirs(self):
Use a tempfile.mkdtemp() based temporary directory Change-Id: Ie0e8137615348b130b67323d0d2913dc5ebfd5fb
2014-01-16 00:49:28 -05:00
testdir_base = mkdtemp()
testroot = os.path.join(testdir_base, 'mkdirs')
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
try:
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(not os.path.exists(testroot))
Use a tempfile.mkdtemp() based temporary directory Change-Id: Ie0e8137615348b130b67323d0d2913dc5ebfd5fb
2014-01-16 00:49:28 -05:00
utils.mkdirs(testroot)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(os.path.exists(testroot))
Use a tempfile.mkdtemp() based temporary directory Change-Id: Ie0e8137615348b130b67323d0d2913dc5ebfd5fb
2014-01-16 00:49:28 -05:00
utils.mkdirs(testroot)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(os.path.exists(testroot))
Use a tempfile.mkdtemp() based temporary directory Change-Id: Ie0e8137615348b130b67323d0d2913dc5ebfd5fb
2014-01-16 00:49:28 -05:00
rmtree(testroot, ignore_errors=1)
testdir = os.path.join(testroot, 'one/two/three')
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(not os.path.exists(testdir))
Use a tempfile.mkdtemp() based temporary directory Change-Id: Ie0e8137615348b130b67323d0d2913dc5ebfd5fb
2014-01-16 00:49:28 -05:00
utils.mkdirs(testdir)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(os.path.exists(testdir))
Use a tempfile.mkdtemp() based temporary directory Change-Id: Ie0e8137615348b130b67323d0d2913dc5ebfd5fb
2014-01-16 00:49:28 -05:00
utils.mkdirs(testdir)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(os.path.exists(testdir))
Use a tempfile.mkdtemp() based temporary directory Change-Id: Ie0e8137615348b130b67323d0d2913dc5ebfd5fb
2014-01-16 00:49:28 -05:00
rmtree(testroot, ignore_errors=1)
open(testroot, 'wb').close()
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(not os.path.exists(testdir))
Use a tempfile.mkdtemp() based temporary directory Change-Id: Ie0e8137615348b130b67323d0d2913dc5ebfd5fb
2014-01-16 00:49:28 -05:00
self.assertRaises(OSError, utils.mkdirs, testdir)
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
os.unlink(testroot)
Use a tempfile.mkdtemp() based temporary directory Change-Id: Ie0e8137615348b130b67323d0d2913dc5ebfd5fb
2014-01-16 00:49:28 -05:00
finally:
rmtree(testdir_base)
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
def test_split_path(self):
Pep8 unit test modules for hacking and one liners (4 of 12) Address all the "hacking" lines that are flagged, and all the modules that just have one item flagged. Change-Id: I372a4bdf9c7748f73e38c4fd55e5954f1afade5b Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-08-31 22:36:58 -04:00
# Test swift.common.utils.split_account_path
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
self.assertRaises(ValueError, utils.split_path, '')
self.assertRaises(ValueError, utils.split_path, '/')
self.assertRaises(ValueError, utils.split_path, '//')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils.split_path('/a'), ['a'])
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
self.assertRaises(ValueError, utils.split_path, '//a')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils.split_path('/a/'), ['a'])
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
self.assertRaises(ValueError, utils.split_path, '/a/c')
self.assertRaises(ValueError, utils.split_path, '//c')
self.assertRaises(ValueError, utils.split_path, '/a/c/')
self.assertRaises(ValueError, utils.split_path, '/a//')
self.assertRaises(ValueError, utils.split_path, '/a', 2)
self.assertRaises(ValueError, utils.split_path, '/a', 2, 3)
self.assertRaises(ValueError, utils.split_path, '/a', 2, 3, True)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils.split_path('/a/c', 2), ['a', 'c'])
self.assertEqual(utils.split_path('/a/c/o', 3), ['a', 'c', 'o'])
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
self.assertRaises(ValueError, utils.split_path, '/a/c/o/r', 3, 3)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils.split_path('/a/c/o/r', 3, 3, True),
['a', 'c', 'o/r'])
self.assertEqual(utils.split_path('/a/c', 2, 3, True),
['a', 'c', None])
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
self.assertRaises(ValueError, utils.split_path, '/a', 5, 4)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils.split_path('/a/c/', 2), ['a', 'c'])
self.assertEqual(utils.split_path('/a/c/', 2, 3), ['a', 'c', ''])
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
try:
utils.split_path('o\nn e', 2)
Use Python 3.x compatible except construct except x,y: was deprected and is removed in Python 3.x. Use "except x as y:" instead which works in any Python version >= 2.6. Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
2013-08-28 21:16:08 +02:00
except ValueError as err:
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(str(err), 'Invalid path: o%0An%20e')
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
try:
utils.split_path('o\nn e', 2, 3, True)
Use Python 3.x compatible except construct except x,y: was deprected and is removed in Python 3.x. Use "except x as y:" instead which works in any Python version >= 2.6. Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
2013-08-28 21:16:08 +02:00
except ValueError as err:
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(str(err), 'Invalid path: o%0An%20e')
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
Revert "Refactor common/utils methods to common/ondisk" This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32 Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 12:10:31 +00:00
def test_validate_device_partition(self):
# Test swift.common.utils.validate_device_partition
utils.validate_device_partition('foo', 'bar')
self.assertRaises(ValueError,
utils.validate_device_partition, '', '')
self.assertRaises(ValueError,
utils.validate_device_partition, '', 'foo')
self.assertRaises(ValueError,
utils.validate_device_partition, 'foo', '')
self.assertRaises(ValueError,
utils.validate_device_partition, 'foo/bar', 'foo')
self.assertRaises(ValueError,
utils.validate_device_partition, 'foo', 'foo/bar')
self.assertRaises(ValueError,
utils.validate_device_partition, '.', 'foo')
self.assertRaises(ValueError,
utils.validate_device_partition, '..', 'foo')
self.assertRaises(ValueError,
utils.validate_device_partition, 'foo', '.')
self.assertRaises(ValueError,
utils.validate_device_partition, 'foo', '..')
try:
utils.validate_device_partition('o\nn e', 'foo')
except ValueError as err:
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(str(err), 'Invalid device: o%0An%20e')
Revert "Refactor common/utils methods to common/ondisk" This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32 Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 12:10:31 +00:00
try:
utils.validate_device_partition('foo', 'o\nn e')
except ValueError as err:
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(str(err), 'Invalid partition: o%0An%20e')
Revert "Refactor common/utils methods to common/ondisk" This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32 Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 12:10:31 +00:00
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
def test_NullLogger(self):
Pep8 unit test modules for hacking and one liners (4 of 12) Address all the "hacking" lines that are flagged, and all the modules that just have one item flagged. Change-Id: I372a4bdf9c7748f73e38c4fd55e5954f1afade5b Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-08-31 22:36:58 -04:00
# Test swift.common.utils.NullLogger
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
sio = StringIO()
nl = utils.NullLogger()
nl.write('test')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(sio.getvalue(), '')
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
def test_LoggerFileObject(self):
orig_stdout = sys.stdout
orig_stderr = sys.stderr
sio = StringIO()
handler = logging.StreamHandler(sio)
logger = logging.getLogger()
logger.addHandler(handler)
Fix the prefix of messages caputured from stderr Swift caputures both sys.stdout and sys.stderr messages and transfers them to syslog. However, whole of the messages uses the "STDOUT:" prefix, currently. It will be quite confusable for debugging purpose because we cannot make sure whether "Swift didn't caputure stderr" or "Swift captured stderr but didn't show it" when we want to log some stderr messages. This patch enables to allow a new prefix "STDERR:" and use it for sys.stderr. Change-Id: Idf4649e9860b77c3600a5cd0a3e0bd674b31fb4f
2015-03-13 04:39:16 -07:00
lfo_stdout = utils.LoggerFileObject(logger)
lfo_stderr = utils.LoggerFileObject(logger)
lfo_stderr = utils.LoggerFileObject(logger, 'STDERR')
test/unit: Replace python print operator with print function (pep H233, py33) 'print' function is compatible with 2.x and 3.x python versions Link : https://www.python.org/dev/peps/pep-3105/ Python 2.6 has a __future__ import that removes print as language syntax, letting you use the functional form instead Change-Id: I94e1bc6bd83ad6b05695c7ebdf7cbfd8f6d9f9af
2015-07-28 21:03:05 +05:30
print('test1')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(sio.getvalue(), '')
Fix the prefix of messages caputured from stderr Swift caputures both sys.stdout and sys.stderr messages and transfers them to syslog. However, whole of the messages uses the "STDOUT:" prefix, currently. It will be quite confusable for debugging purpose because we cannot make sure whether "Swift didn't caputure stderr" or "Swift captured stderr but didn't show it" when we want to log some stderr messages. This patch enables to allow a new prefix "STDERR:" and use it for sys.stderr. Change-Id: Idf4649e9860b77c3600a5cd0a3e0bd674b31fb4f
2015-03-13 04:39:16 -07:00
sys.stdout = lfo_stdout
test/unit: Replace python print operator with print function (pep H233, py33) 'print' function is compatible with 2.x and 3.x python versions Link : https://www.python.org/dev/peps/pep-3105/ Python 2.6 has a __future__ import that removes print as language syntax, letting you use the functional form instead Change-Id: I94e1bc6bd83ad6b05695c7ebdf7cbfd8f6d9f9af
2015-07-28 21:03:05 +05:30
print('test2')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(sio.getvalue(), 'STDOUT: test2\n')
Fix the prefix of messages caputured from stderr Swift caputures both sys.stdout and sys.stderr messages and transfers them to syslog. However, whole of the messages uses the "STDOUT:" prefix, currently. It will be quite confusable for debugging purpose because we cannot make sure whether "Swift didn't caputure stderr" or "Swift captured stderr but didn't show it" when we want to log some stderr messages. This patch enables to allow a new prefix "STDERR:" and use it for sys.stderr. Change-Id: Idf4649e9860b77c3600a5cd0a3e0bd674b31fb4f
2015-03-13 04:39:16 -07:00
sys.stderr = lfo_stderr
test/unit: Replace python print operator with print function (pep H233, py33) 'print' function is compatible with 2.x and 3.x python versions Link : https://www.python.org/dev/peps/pep-3105/ Python 2.6 has a __future__ import that removes print as language syntax, letting you use the functional form instead Change-Id: I94e1bc6bd83ad6b05695c7ebdf7cbfd8f6d9f9af
2015-07-28 21:03:05 +05:30
print('test4', file=sys.stderr)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(sio.getvalue(), 'STDOUT: test2\nSTDERR: test4\n')
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
sys.stdout = orig_stdout
test/unit: Replace python print operator with print function (pep H233, py33) 'print' function is compatible with 2.x and 3.x python versions Link : https://www.python.org/dev/peps/pep-3105/ Python 2.6 has a __future__ import that removes print as language syntax, letting you use the functional form instead Change-Id: I94e1bc6bd83ad6b05695c7ebdf7cbfd8f6d9f9af
2015-07-28 21:03:05 +05:30
print('test5')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(sio.getvalue(), 'STDOUT: test2\nSTDERR: test4\n')
test/unit: Replace python print operator with print function (pep H233, py33) 'print' function is compatible with 2.x and 3.x python versions Link : https://www.python.org/dev/peps/pep-3105/ Python 2.6 has a __future__ import that removes print as language syntax, letting you use the functional form instead Change-Id: I94e1bc6bd83ad6b05695c7ebdf7cbfd8f6d9f9af
2015-07-28 21:03:05 +05:30
print('test6', file=sys.stderr)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(sio.getvalue(), 'STDOUT: test2\nSTDERR: test4\n'
'STDERR: test6\n')
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
sys.stderr = orig_stderr
test/unit: Replace python print operator with print function (pep H233, py33) 'print' function is compatible with 2.x and 3.x python versions Link : https://www.python.org/dev/peps/pep-3105/ Python 2.6 has a __future__ import that removes print as language syntax, letting you use the functional form instead Change-Id: I94e1bc6bd83ad6b05695c7ebdf7cbfd8f6d9f9af
2015-07-28 21:03:05 +05:30
print('test8')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(sio.getvalue(), 'STDOUT: test2\nSTDERR: test4\n'
'STDERR: test6\n')
Fix the prefix of messages caputured from stderr Swift caputures both sys.stdout and sys.stderr messages and transfers them to syslog. However, whole of the messages uses the "STDOUT:" prefix, currently. It will be quite confusable for debugging purpose because we cannot make sure whether "Swift didn't caputure stderr" or "Swift captured stderr but didn't show it" when we want to log some stderr messages. This patch enables to allow a new prefix "STDERR:" and use it for sys.stderr. Change-Id: Idf4649e9860b77c3600a5cd0a3e0bd674b31fb4f
2015-03-13 04:39:16 -07:00
lfo_stdout.writelines(['a', 'b', 'c'])
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(sio.getvalue(), 'STDOUT: test2\nSTDERR: test4\n'
'STDERR: test6\nSTDOUT: a#012b#012c\n')
Fix the prefix of messages caputured from stderr Swift caputures both sys.stdout and sys.stderr messages and transfers them to syslog. However, whole of the messages uses the "STDOUT:" prefix, currently. It will be quite confusable for debugging purpose because we cannot make sure whether "Swift didn't caputure stderr" or "Swift captured stderr but didn't show it" when we want to log some stderr messages. This patch enables to allow a new prefix "STDERR:" and use it for sys.stderr. Change-Id: Idf4649e9860b77c3600a5cd0a3e0bd674b31fb4f
2015-03-13 04:39:16 -07:00
lfo_stdout.close()
lfo_stderr.close()
lfo_stdout.write('d')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(sio.getvalue(), 'STDOUT: test2\nSTDERR: test4\n'
'STDERR: test6\nSTDOUT: a#012b#012c\nSTDOUT: d\n')
Fix the prefix of messages caputured from stderr Swift caputures both sys.stdout and sys.stderr messages and transfers them to syslog. However, whole of the messages uses the "STDOUT:" prefix, currently. It will be quite confusable for debugging purpose because we cannot make sure whether "Swift didn't caputure stderr" or "Swift captured stderr but didn't show it" when we want to log some stderr messages. This patch enables to allow a new prefix "STDERR:" and use it for sys.stderr. Change-Id: Idf4649e9860b77c3600a5cd0a3e0bd674b31fb4f
2015-03-13 04:39:16 -07:00
lfo_stdout.flush()
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(sio.getvalue(), 'STDOUT: test2\nSTDERR: test4\n'
'STDERR: test6\nSTDOUT: a#012b#012c\nSTDOUT: d\n')
Fix the prefix of messages caputured from stderr Swift caputures both sys.stdout and sys.stderr messages and transfers them to syslog. However, whole of the messages uses the "STDOUT:" prefix, currently. It will be quite confusable for debugging purpose because we cannot make sure whether "Swift didn't caputure stderr" or "Swift captured stderr but didn't show it" when we want to log some stderr messages. This patch enables to allow a new prefix "STDERR:" and use it for sys.stderr. Change-Id: Idf4649e9860b77c3600a5cd0a3e0bd674b31fb4f
2015-03-13 04:39:16 -07:00
for lfo in (lfo_stdout, lfo_stderr):
got_exc = False
try:
for line in lfo:
pass
except Exception:
got_exc = True
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(got_exc)
Fix the prefix of messages caputured from stderr Swift caputures both sys.stdout and sys.stderr messages and transfers them to syslog. However, whole of the messages uses the "STDOUT:" prefix, currently. It will be quite confusable for debugging purpose because we cannot make sure whether "Swift didn't caputure stderr" or "Swift captured stderr but didn't show it" when we want to log some stderr messages. This patch enables to allow a new prefix "STDERR:" and use it for sys.stderr. Change-Id: Idf4649e9860b77c3600a5cd0a3e0bd674b31fb4f
2015-03-13 04:39:16 -07:00
got_exc = False
try:
Deprecated xreadlines() usage fixed Change-Id: Iadf42c2f86f78c11259e21c88b4aef51db3441ad
2015-07-18 14:12:58 +05:30
for line in lfo:
Fix the prefix of messages caputured from stderr Swift caputures both sys.stdout and sys.stderr messages and transfers them to syslog. However, whole of the messages uses the "STDOUT:" prefix, currently. It will be quite confusable for debugging purpose because we cannot make sure whether "Swift didn't caputure stderr" or "Swift captured stderr but didn't show it" when we want to log some stderr messages. This patch enables to allow a new prefix "STDERR:" and use it for sys.stderr. Change-Id: Idf4649e9860b77c3600a5cd0a3e0bd674b31fb4f
2015-03-13 04:39:16 -07:00
pass
except Exception:
got_exc = True
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(got_exc)
Fix the prefix of messages caputured from stderr Swift caputures both sys.stdout and sys.stderr messages and transfers them to syslog. However, whole of the messages uses the "STDOUT:" prefix, currently. It will be quite confusable for debugging purpose because we cannot make sure whether "Swift didn't caputure stderr" or "Swift captured stderr but didn't show it" when we want to log some stderr messages. This patch enables to allow a new prefix "STDERR:" and use it for sys.stderr. Change-Id: Idf4649e9860b77c3600a5cd0a3e0bd674b31fb4f
2015-03-13 04:39:16 -07:00
self.assertRaises(IOError, lfo.read)
self.assertRaises(IOError, lfo.read, 1024)
self.assertRaises(IOError, lfo.readline)
self.assertRaises(IOError, lfo.readline, 1024)
lfo.tell()
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
Fix infinite recursion during logging when syslog is down Change-Id: Ia9ecffc88ce43616977e141498e5ee404f2c29c4
2016-03-23 13:51:47 -07:00
def test_LoggerFileObject_recursion(self):
crashy_calls = [0]
class CrashyLogger(logging.Handler):
def emit(self, record):
crashy_calls[0] += 1
try:
# Pretend to be trying to send to syslog, but syslogd is
# dead. We need the raise here to set sys.exc_info.
raise socket.error(errno.ENOTCONN, "This is an ex-syslog")
except socket.error:
self.handleError(record)
logger = logging.getLogger()
logger.addHandler(CrashyLogger())
# Set up some real file descriptors for stdio. If you run
# nosetests with "-s", you already have real files there, but
# otherwise they're StringIO objects.
#
# In any case, since capture_stdio() closes sys.stdin and friends,
# we'd want to set up some sacrificial files so as to not goof up
# the testrunner.
new_stdin = open(os.devnull, 'r+b')
new_stdout = open(os.devnull, 'w+b')
new_stderr = open(os.devnull, 'w+b')
with contextlib.closing(new_stdin), contextlib.closing(new_stdout), \
contextlib.closing(new_stderr):
# logging.raiseExceptions is set to False in test/__init__.py, but
# is True in Swift daemons, and the error doesn't manifest without
# it.
with mock.patch('sys.stdin', new_stdin), \
mock.patch('sys.stdout', new_stdout), \
mock.patch('sys.stderr', new_stderr), \
mock.patch.object(logging, 'raiseExceptions', True):
# Note: since stdio is hooked up to /dev/null in here, using
# pdb is basically impossible. Sorry about that.
utils.capture_stdio(logger)
logger.info("I like ham")
self.assertTrue(crashy_calls[0], 1)
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
def test_parse_options(self):
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
# Get a file that is definitely on disk
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
with NamedTemporaryFile() as f:
conf_file = f.name
conf, options = utils.parse_options(test_args=[conf_file])
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(conf, conf_file)
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
# assert defaults
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(options['verbose'], False)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('once' not in options)
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
# assert verbose as option
conf, options = utils.parse_options(test_args=[conf_file, '-v'])
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(options['verbose'], True)
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
# check once option
conf, options = utils.parse_options(test_args=[conf_file],
once=True)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(options['once'], False)
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
test_args = [conf_file, '--once']
conf, options = utils.parse_options(test_args=test_args, once=True)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(options['once'], True)
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
# check options as arg parsing
test_args = [conf_file, 'once', 'plugin_name', 'verbose']
conf, options = utils.parse_options(test_args=test_args, once=True)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(options['verbose'], True)
self.assertEqual(options['once'], True)
self.assertEqual(options['extra_args'], ['plugin_name'])
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
def test_parse_options_errors(self):
orig_stdout = sys.stdout
orig_stderr = sys.stderr
stdo = StringIO()
stde = StringIO()
utils.sys.stdout = stdo
utils.sys.stderr = stde
audit zero byte files quickly without true value
2011-02-14 20:25:40 +00:00
self.assertRaises(SystemExit, utils.parse_options, once=True,
test_args=[])
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('missing config' in stdo.getvalue())
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
# verify conf file must exist, context manager will delete temp file
with NamedTemporaryFile() as f:
conf_file = f.name
audit zero byte files quickly without true value
2011-02-14 20:25:40 +00:00
self.assertRaises(SystemExit, utils.parse_options, once=True,
test_args=[conf_file])
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('unable to locate' in stdo.getvalue())
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
# reset stdio
utils.sys.stdout = orig_stdout
utils.sys.stderr = orig_stderr
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
Allow specification of object devices for audit In object audit "once" mode we are allowing the user to specify a sub-set of devices to audit using the "--devices" command-line option. The sub-set is specified as a comma-separated list. This patch is taken from a larger patch to enable parallel processing in the object auditor. We've had to modify recon so that it will work properly with this change to "once" mode. We've modified dump_recon_cache() so that it will store nested dictionaries, in other words it will store a recon cache entry such as {'key1': {'key2': {...}}}. When the object auditor is run in "once" mode with "--devices" set the object_auditor_stats_ALL and ZBF entries look like: {'object_auditor_stats_ALL': {'disk1disk2..diskn': {...}}}. When swift-recon is run, it hunts through the nested dicts to find the appropriate entries. The object auditor recon cache entries are set to {} at the beginning of each audit cycle, and individual disk entries are cleared from cache at the end of each disk's audit cycle. DocImpact Change-Id: Icc53dac0a8136f1b2f61d5e08baf7b4fd87c8123
2014-02-24 11:24:56 +00:00
def test_dump_recon_cache(self):
testdir_base = mkdtemp()
testcache_file = os.path.join(testdir_base, 'cache.recon')
logger = utils.get_logger(None, 'server', log_route='server')
try:
submit_dict = {'key1': {'value1': 1, 'value2': 2}}
utils.dump_recon_cache(submit_dict, testcache_file, logger)
fd = open(testcache_file)
file_dict = json.loads(fd.readline())
fd.close()
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(submit_dict, file_dict)
Allow specification of object devices for audit In object audit "once" mode we are allowing the user to specify a sub-set of devices to audit using the "--devices" command-line option. The sub-set is specified as a comma-separated list. This patch is taken from a larger patch to enable parallel processing in the object auditor. We've had to modify recon so that it will work properly with this change to "once" mode. We've modified dump_recon_cache() so that it will store nested dictionaries, in other words it will store a recon cache entry such as {'key1': {'key2': {...}}}. When the object auditor is run in "once" mode with "--devices" set the object_auditor_stats_ALL and ZBF entries look like: {'object_auditor_stats_ALL': {'disk1disk2..diskn': {...}}}. When swift-recon is run, it hunts through the nested dicts to find the appropriate entries. The object auditor recon cache entries are set to {} at the beginning of each audit cycle, and individual disk entries are cleared from cache at the end of each disk's audit cycle. DocImpact Change-Id: Icc53dac0a8136f1b2f61d5e08baf7b4fd87c8123
2014-02-24 11:24:56 +00:00
# Use a nested entry
submit_dict = {'key1': {'key2': {'value1': 1, 'value2': 2}}}
result_dict = {'key1': {'key2': {'value1': 1, 'value2': 2},
'value1': 1, 'value2': 2}}
utils.dump_recon_cache(submit_dict, testcache_file, logger)
fd = open(testcache_file)
file_dict = json.loads(fd.readline())
fd.close()
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(result_dict, file_dict)
Allow specification of object devices for audit In object audit "once" mode we are allowing the user to specify a sub-set of devices to audit using the "--devices" command-line option. The sub-set is specified as a comma-separated list. This patch is taken from a larger patch to enable parallel processing in the object auditor. We've had to modify recon so that it will work properly with this change to "once" mode. We've modified dump_recon_cache() so that it will store nested dictionaries, in other words it will store a recon cache entry such as {'key1': {'key2': {...}}}. When the object auditor is run in "once" mode with "--devices" set the object_auditor_stats_ALL and ZBF entries look like: {'object_auditor_stats_ALL': {'disk1disk2..diskn': {...}}}. When swift-recon is run, it hunts through the nested dicts to find the appropriate entries. The object auditor recon cache entries are set to {} at the beginning of each audit cycle, and individual disk entries are cleared from cache at the end of each disk's audit cycle. DocImpact Change-Id: Icc53dac0a8136f1b2f61d5e08baf7b4fd87c8123
2014-02-24 11:24:56 +00:00
finally:
rmtree(testdir_base)
Fix handling of "Permission Denied" error from NamedTemporaryFile function If "Permission Denied" has happen in NamedTemporaryFile function in dump_recon_cache method, swift will log a message of reference to a variable without assignment and not log a message of "Permission Denied". This patch fixes the handling and add an unit test. Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp> Change-Id: Iafdd94905e9e9c81f5966a923324b50c18fcf592
2016-01-21 14:18:31 +09:00
def test_dump_recon_cache_permission_denied(self):
testdir_base = mkdtemp()
testcache_file = os.path.join(testdir_base, 'cache.recon')
class MockLogger(object):
def __init__(self):
self._excs = []
def exception(self, message):
_junk, exc, _junk = sys.exc_info()
self._excs.append(exc)
logger = MockLogger()
try:
submit_dict = {'key1': {'value1': 1, 'value2': 2}}
with mock.patch(
'swift.common.utils.NamedTemporaryFile',
side_effect=IOError(13, 'Permission Denied')):
utils.dump_recon_cache(submit_dict, testcache_file, logger)
self.assertIsInstance(logger._excs[0], IOError)
finally:
rmtree(testdir_base)
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
def test_get_logger(self):
sio = StringIO()
Fix duplicate logging
2011-02-02 09:38:17 -08:00
logger = logging.getLogger('server')
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
logger.addHandler(logging.StreamHandler(sio))
logging: use routes to separate logging configurations
2011-02-02 13:39:08 -08:00
logger = utils.get_logger(None, 'server', log_route='server')
Python 3 deprecated the logger.warn method in favor of warning DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead Change-Id: I35df44374c4521b1f06be7a96c0b873e8c3674d8
2015-08-02 22:47:42 +05:30
logger.warning('test1')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(sio.getvalue(), 'test1\n')
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
logger.debug('test2')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(sio.getvalue(), 'test1\n')
logging: use routes to separate logging configurations
2011-02-02 13:39:08 -08:00
logger = utils.get_logger({'log_level': 'DEBUG'}, 'server',
log_route='server')
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
logger.debug('test3')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(sio.getvalue(), 'test1\ntest3\n')
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
# Doesn't really test that the log facility is truly being used all the
# way to syslog; but exercises the code.
logging: use routes to separate logging configurations
2011-02-02 13:39:08 -08:00
logger = utils.get_logger({'log_facility': 'LOG_LOCAL3'}, 'server',
log_route='server')
Python 3 deprecated the logger.warn method in favor of warning DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead Change-Id: I35df44374c4521b1f06be7a96c0b873e8c3674d8
2015-08-02 22:47:42 +05:30
logger.warning('test4')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(sio.getvalue(),
'test1\ntest3\ntest4\n')
Moved proxy server access log messages into their own log level Added new "access" log level available on swift loggers that will be routed to the LOG_NOTICE priority in syslog for easy redirection of access log messages via rsyslog and syslog-ng.
2011-02-03 15:23:07 -06:00
# make sure debug doesn't log by default
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
logger.debug('test5')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(sio.getvalue(),
'test1\ntest3\ntest4\n')
added new proxy-server configuration options for access_log_facility and access_log_name
2011-02-05 15:38:49 -06:00
# make sure notice lvl logs by default
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
logger.notice('test6')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(sio.getvalue(),
'test1\ntest3\ntest4\ntest6\n')
Moved proxy server access log messages into their own log level Added new "access" log level available on swift loggers that will be routed to the LOG_NOTICE priority in syslog for easy redirection of access log messages via rsyslog and syslog-ng.
2011-02-03 15:23:07 -06:00
Make statsd sample rate behave better. As Dieter pointed out in bug 1090495 (https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics can vary wildly between StatsD metrics. This patch implements a partial solution by reducing the sample_rate used for known high-volume metrics (operational experience will need to inform this over time) and introducing a new tunable, log_statsd_sample_rate_factor which is multiplied by the sample_rate for every statsd stat. This tunable can be used to reduce StatsD traffic proportionally for all metrics and is intended to replace log_statsd_default_sample_rate, which is left alone for backward-compatibility, should anyone be using it. This patch also includes a drive-by fix for log_udp_port which wasn't being converted to an int (I didn't verify that actually causes trouble in SysLogHandler(), but it's definitely an improvement regardles). Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
def test_get_logger_sysloghandler_plumbing(self):
orig_sysloghandler = utils.SysLogHandler
syslog_handler_args = []
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
Make statsd sample rate behave better. As Dieter pointed out in bug 1090495 (https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics can vary wildly between StatsD metrics. This patch implements a partial solution by reducing the sample_rate used for known high-volume metrics (operational experience will need to inform this over time) and introducing a new tunable, log_statsd_sample_rate_factor which is multiplied by the sample_rate for every statsd stat. This tunable can be used to reduce StatsD traffic proportionally for all metrics and is intended to replace log_statsd_default_sample_rate, which is left alone for backward-compatibility, should anyone be using it. This patch also includes a drive-by fix for log_udp_port which wasn't being converted to an int (I didn't verify that actually causes trouble in SysLogHandler(), but it's definitely an improvement regardles). Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
def syslog_handler_catcher(*args, **kwargs):
syslog_handler_args.append((args, kwargs))
return orig_sysloghandler(*args, **kwargs)
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
Make statsd sample rate behave better. As Dieter pointed out in bug 1090495 (https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics can vary wildly between StatsD metrics. This patch implements a partial solution by reducing the sample_rate used for known high-volume metrics (operational experience will need to inform this over time) and introducing a new tunable, log_statsd_sample_rate_factor which is multiplied by the sample_rate for every statsd stat. This tunable can be used to reduce StatsD traffic proportionally for all metrics and is intended to replace log_statsd_default_sample_rate, which is left alone for backward-compatibility, should anyone be using it. This patch also includes a drive-by fix for log_udp_port which wasn't being converted to an int (I didn't verify that actually causes trouble in SysLogHandler(), but it's definitely an improvement regardles). Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
syslog_handler_catcher.LOG_LOCAL0 = orig_sysloghandler.LOG_LOCAL0
syslog_handler_catcher.LOG_LOCAL3 = orig_sysloghandler.LOG_LOCAL3
try:
utils.SysLogHandler = syslog_handler_catcher
fixed some minor things in tests that pyflakes complained about Change-Id: Ifeab56a964630bcf941e932fcbe39e6572e62975
2013-03-26 20:42:26 +00:00
utils.get_logger({
Make statsd sample rate behave better. As Dieter pointed out in bug 1090495 (https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics can vary wildly between StatsD metrics. This patch implements a partial solution by reducing the sample_rate used for known high-volume metrics (operational experience will need to inform this over time) and introducing a new tunable, log_statsd_sample_rate_factor which is multiplied by the sample_rate for every statsd stat. This tunable can be used to reduce StatsD traffic proportionally for all metrics and is intended to replace log_statsd_default_sample_rate, which is left alone for backward-compatibility, should anyone be using it. This patch also includes a drive-by fix for log_udp_port which wasn't being converted to an int (I didn't verify that actually causes trouble in SysLogHandler(), but it's definitely an improvement regardles). Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
'log_facility': 'LOG_LOCAL3',
}, 'server', log_route='server')
Use /var/run/syslog on macosx for syslog tests. - This patch is based on zaictev review. - Fixes bug 1206560. Change-Id: I41f3d485df41fe443784f803b276712405fe72c6
2013-08-06 12:58:59 +02:00
expected_args = [((), {'address': '/dev/log',
'facility': orig_sysloghandler.LOG_LOCAL3})]
if not os.path.exists('/dev/log') or \
os.path.isfile('/dev/log') or \
os.path.isdir('/dev/log'):
# Since socket on OSX is in /var/run/syslog, there will be
# a fallback to UDP.
Pep8 unit test modules w/ <= 10 violations (5 of 12) Change-Id: I8e82c14ada52d44df5a31e08982ac79cd7e5c969 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-08-31 23:13:15 -04:00
expected_args.append(
((), {'facility': orig_sysloghandler.LOG_LOCAL3}))
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(expected_args, syslog_handler_args)
Make statsd sample rate behave better. As Dieter pointed out in bug 1090495 (https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics can vary wildly between StatsD metrics. This patch implements a partial solution by reducing the sample_rate used for known high-volume metrics (operational experience will need to inform this over time) and introducing a new tunable, log_statsd_sample_rate_factor which is multiplied by the sample_rate for every statsd stat. This tunable can be used to reduce StatsD traffic proportionally for all metrics and is intended to replace log_statsd_default_sample_rate, which is left alone for backward-compatibility, should anyone be using it. This patch also includes a drive-by fix for log_udp_port which wasn't being converted to an int (I didn't verify that actually causes trouble in SysLogHandler(), but it's definitely an improvement regardles). Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
syslog_handler_args = []
fixed some minor things in tests that pyflakes complained about Change-Id: Ifeab56a964630bcf941e932fcbe39e6572e62975
2013-03-26 20:42:26 +00:00
utils.get_logger({
Make statsd sample rate behave better. As Dieter pointed out in bug 1090495 (https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics can vary wildly between StatsD metrics. This patch implements a partial solution by reducing the sample_rate used for known high-volume metrics (operational experience will need to inform this over time) and introducing a new tunable, log_statsd_sample_rate_factor which is multiplied by the sample_rate for every statsd stat. This tunable can be used to reduce StatsD traffic proportionally for all metrics and is intended to replace log_statsd_default_sample_rate, which is left alone for backward-compatibility, should anyone be using it. This patch also includes a drive-by fix for log_udp_port which wasn't being converted to an int (I didn't verify that actually causes trouble in SysLogHandler(), but it's definitely an improvement regardles). Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
'log_facility': 'LOG_LOCAL3',
'log_address': '/foo/bar',
}, 'server', log_route='server')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual([
Make statsd sample rate behave better. As Dieter pointed out in bug 1090495 (https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics can vary wildly between StatsD metrics. This patch implements a partial solution by reducing the sample_rate used for known high-volume metrics (operational experience will need to inform this over time) and introducing a new tunable, log_statsd_sample_rate_factor which is multiplied by the sample_rate for every statsd stat. This tunable can be used to reduce StatsD traffic proportionally for all metrics and is intended to replace log_statsd_default_sample_rate, which is left alone for backward-compatibility, should anyone be using it. This patch also includes a drive-by fix for log_udp_port which wasn't being converted to an int (I didn't verify that actually causes trouble in SysLogHandler(), but it's definitely an improvement regardles). Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
((), {'address': '/foo/bar',
'facility': orig_sysloghandler.LOG_LOCAL3}),
# Second call is because /foo/bar didn't exist (and wasn't a
# UNIX domain socket).
((), {'facility': orig_sysloghandler.LOG_LOCAL3})],
syslog_handler_args)
# Using UDP with default port
syslog_handler_args = []
fixed some minor things in tests that pyflakes complained about Change-Id: Ifeab56a964630bcf941e932fcbe39e6572e62975
2013-03-26 20:42:26 +00:00
utils.get_logger({
Make statsd sample rate behave better. As Dieter pointed out in bug 1090495 (https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics can vary wildly between StatsD metrics. This patch implements a partial solution by reducing the sample_rate used for known high-volume metrics (operational experience will need to inform this over time) and introducing a new tunable, log_statsd_sample_rate_factor which is multiplied by the sample_rate for every statsd stat. This tunable can be used to reduce StatsD traffic proportionally for all metrics and is intended to replace log_statsd_default_sample_rate, which is left alone for backward-compatibility, should anyone be using it. This patch also includes a drive-by fix for log_udp_port which wasn't being converted to an int (I didn't verify that actually causes trouble in SysLogHandler(), but it's definitely an improvement regardles). Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
'log_udp_host': 'syslog.funtimes.com',
}, 'server', log_route='server')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual([
Make statsd sample rate behave better. As Dieter pointed out in bug 1090495 (https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics can vary wildly between StatsD metrics. This patch implements a partial solution by reducing the sample_rate used for known high-volume metrics (operational experience will need to inform this over time) and introducing a new tunable, log_statsd_sample_rate_factor which is multiplied by the sample_rate for every statsd stat. This tunable can be used to reduce StatsD traffic proportionally for all metrics and is intended to replace log_statsd_default_sample_rate, which is left alone for backward-compatibility, should anyone be using it. This patch also includes a drive-by fix for log_udp_port which wasn't being converted to an int (I didn't verify that actually causes trouble in SysLogHandler(), but it's definitely an improvement regardles). Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
((), {'address': ('syslog.funtimes.com',
logging.handlers.SYSLOG_UDP_PORT),
'facility': orig_sysloghandler.LOG_LOCAL0})],
syslog_handler_args)
# Using UDP with non-default port
syslog_handler_args = []
fixed some minor things in tests that pyflakes complained about Change-Id: Ifeab56a964630bcf941e932fcbe39e6572e62975
2013-03-26 20:42:26 +00:00
utils.get_logger({
Make statsd sample rate behave better. As Dieter pointed out in bug 1090495 (https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics can vary wildly between StatsD metrics. This patch implements a partial solution by reducing the sample_rate used for known high-volume metrics (operational experience will need to inform this over time) and introducing a new tunable, log_statsd_sample_rate_factor which is multiplied by the sample_rate for every statsd stat. This tunable can be used to reduce StatsD traffic proportionally for all metrics and is intended to replace log_statsd_default_sample_rate, which is left alone for backward-compatibility, should anyone be using it. This patch also includes a drive-by fix for log_udp_port which wasn't being converted to an int (I didn't verify that actually causes trouble in SysLogHandler(), but it's definitely an improvement regardles). Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
'log_udp_host': 'syslog.funtimes.com',
'log_udp_port': '2123',
}, 'server', log_route='server')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual([
Make statsd sample rate behave better. As Dieter pointed out in bug 1090495 (https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics can vary wildly between StatsD metrics. This patch implements a partial solution by reducing the sample_rate used for known high-volume metrics (operational experience will need to inform this over time) and introducing a new tunable, log_statsd_sample_rate_factor which is multiplied by the sample_rate for every statsd stat. This tunable can be used to reduce StatsD traffic proportionally for all metrics and is intended to replace log_statsd_default_sample_rate, which is left alone for backward-compatibility, should anyone be using it. This patch also includes a drive-by fix for log_udp_port which wasn't being converted to an int (I didn't verify that actually causes trouble in SysLogHandler(), but it's definitely an improvement regardles). Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
((), {'address': ('syslog.funtimes.com', 2123),
'facility': orig_sysloghandler.LOG_LOCAL0})],
syslog_handler_args)
finally:
utils.SysLogHandler = orig_sysloghandler
Fix logger test coupling Two tests fail if test/unit/common/test_daemon.py is excluded from the test set: test.unit.common.test_utils.TestUtils.test_swift_log_formatter test.unit.common.test_utils.TestStatsdLoggingDelegation.test_thread_locals They fail because they assert that logger.txn_id is None, when previous tests cause txn_id to be set. This coupling is masked when test_daemon.py is included because it reloads the common/utils module, and is executed just before test.unit.common.test_utils. The coupling can be observed by running, for example: nosetests ./test/unit/common/middleware/test_except.py ./test/unit/common/test_utils.py or nosetests ./test/unit/proxy ./test/unit/common/test_utils.py The failing tests should reset logger thread_locals before making assertions. test_utils.reset_loggers() attempts this but is broken because it sets thread_local on the wrong object. Changes in this patch: - fix reset_loggers() to reset the LogAdapter thread local state - add a reset_logger_state decorator to call reset_loggers before and after a decorated method - use the decorator for increased isolation of tests that previously called reset_loggers only on exit Change-Id: If9aa781a2dd1929a47ef69322ec8c53263d47660
2015-02-11 17:49:41 +00:00
@reset_logger_state
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
def test_clean_logger_exception(self):
# setup stream logging
sio = StringIO()
logger = utils.get_logger(None)
handler = logging.StreamHandler(sio)
logger.logger.addHandler(handler)
def strip_value(sio):
Fix Python 3 issues in utils * LogAdapter.exception(): on Python 3, socket.error is an alias to OSError, so group both exceptions to support Python 3. * ThreadPool: call GreenPipe() with indexed parameter, don't pass the third parameter by its name, since the parameter name changed between Python 2 (bufsize) and Python 3 (buffering). * strip_value() in test_utils: StringIO.getvalue() now requires seek(0), otherwise the buffer is filled with null characters. * test_lock_file(): on Python 3, seek(0) is now required to go the beginning of a file opened in append mode. In append mode, Python goes to the end of the file. Change-Id: I4e56a51690f016a0a2e1354380ce11cff1891f64
2015-10-19 17:16:21 +02:00
sio.seek(0)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
v = sio.getvalue()
sio.truncate(0)
return v
def log_exception(exc):
try:
raise exc
Updated TimeoutError and except Exception refs... Updated eventlet.TimeoutError (deprecated) references to Timeout and, more importantly, updated many except Exception clauses to except (Exception, Timeout). Change-Id: Ib089265551bd20b94c00ea84f11140ccd795d301
2011-11-07 16:24:08 +00:00
except (Exception, Timeout):
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
logger.exception('blah')
try:
# establish base case
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(strip_value(sio), '')
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
logger.info('test')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(strip_value(sio), 'test\n')
self.assertEqual(strip_value(sio), '')
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
logger.info('test')
logger.info('test')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(strip_value(sio), 'test\ntest\n')
self.assertEqual(strip_value(sio), '')
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
# test OSError
for en in (errno.EIO, errno.ENOSPC):
log_exception(OSError(en, 'my %s error message' % en))
log_msg = strip_value(sio)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('Traceback' not in log_msg)
self.assertTrue('my %s error message' % en in log_msg)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
# unfiltered
log_exception(OSError())
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('Traceback' in strip_value(sio))
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
# test socket.error
log_exception(socket.error(errno.ECONNREFUSED,
'my error message'))
log_msg = strip_value(sio)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('Traceback' not in log_msg)
self.assertTrue('errno.ECONNREFUSED message test' not in log_msg)
self.assertTrue('Connection refused' in log_msg)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
log_exception(socket.error(errno.EHOSTUNREACH,
'my error message'))
log_msg = strip_value(sio)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('Traceback' not in log_msg)
self.assertTrue('my error message' not in log_msg)
self.assertTrue('Host unreachable' in log_msg)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
log_exception(socket.error(errno.ETIMEDOUT, 'my error message'))
log_msg = strip_value(sio)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('Traceback' not in log_msg)
self.assertTrue('my error message' not in log_msg)
self.assertTrue('Connection timeout' in log_msg)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
# unfiltered
log_exception(socket.error(0, 'my error message'))
log_msg = strip_value(sio)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('Traceback' in log_msg)
self.assertTrue('my error message' in log_msg)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
# test eventlet.Timeout
Autocreate cleanups Autocreate was messy - now cleaned. Auto-create now occurs at account POST, and container PUT only. A method for autocreation was added Autocreation was removed from account_info and container_info. Fake-it as if the account exists on account HEAD and account GET. Return 404 on everything else when the account does not exist. Fix: Bug #1172223 Fix: Bug #1179140 Change-Id: Iac54c1438eb09883fbc29a1ad2ac2245b95efc92
2013-04-24 21:58:15 +03:00
connection_timeout = ConnectionTimeout(42, 'my error message')
log_exception(connection_timeout)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
log_msg = strip_value(sio)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('Traceback' not in log_msg)
self.assertTrue('ConnectionTimeout' in log_msg)
self.assertTrue('(42s)' in log_msg)
self.assertTrue('my error message' not in log_msg)
Autocreate cleanups Autocreate was messy - now cleaned. Auto-create now occurs at account POST, and container PUT only. A method for autocreation was added Autocreation was removed from account_info and container_info. Fake-it as if the account exists on account HEAD and account GET. Return 404 on everything else when the account does not exist. Fix: Bug #1172223 Fix: Bug #1179140 Change-Id: Iac54c1438eb09883fbc29a1ad2ac2245b95efc92
2013-04-24 21:58:15 +03:00
connection_timeout.cancel()
message_timeout = MessageTimeout(42, 'my error message')
log_exception(message_timeout)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
log_msg = strip_value(sio)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('Traceback' not in log_msg)
self.assertTrue('MessageTimeout' in log_msg)
self.assertTrue('(42s)' in log_msg)
self.assertTrue('my error message' in log_msg)
Autocreate cleanups Autocreate was messy - now cleaned. Auto-create now occurs at account POST, and container PUT only. A method for autocreation was added Autocreation was removed from account_info and container_info. Fake-it as if the account exists on account HEAD and account GET. Return 404 on everything else when the account does not exist. Fix: Bug #1172223 Fix: Bug #1179140 Change-Id: Iac54c1438eb09883fbc29a1ad2ac2245b95efc92
2013-04-24 21:58:15 +03:00
message_timeout.cancel()
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
# test unhandled
log_exception(Exception('my error message'))
log_msg = strip_value(sio)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('Traceback' in log_msg)
self.assertTrue('my error message' in log_msg)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
finally:
logger.logger.removeHandler(handler)
Fix logger test coupling Two tests fail if test/unit/common/test_daemon.py is excluded from the test set: test.unit.common.test_utils.TestUtils.test_swift_log_formatter test.unit.common.test_utils.TestStatsdLoggingDelegation.test_thread_locals They fail because they assert that logger.txn_id is None, when previous tests cause txn_id to be set. This coupling is masked when test_daemon.py is included because it reloads the common/utils module, and is executed just before test.unit.common.test_utils. The coupling can be observed by running, for example: nosetests ./test/unit/common/middleware/test_except.py ./test/unit/common/test_utils.py or nosetests ./test/unit/proxy ./test/unit/common/test_utils.py The failing tests should reset logger thread_locals before making assertions. test_utils.reset_loggers() attempts this but is broken because it sets thread_local on the wrong object. Changes in this patch: - fix reset_loggers() to reset the LogAdapter thread local state - add a reset_logger_state decorator to call reset_loggers before and after a decorated method - use the decorator for increased isolation of tests that previously called reset_loggers only on exit Change-Id: If9aa781a2dd1929a47ef69322ec8c53263d47660
2015-02-11 17:49:41 +00:00
@reset_logger_state
New log_max_line_length option. Log lines can get quite large, as we previously noticed with rsync error log lines. We added a setting to cap those, but it really looks like we should have just done this overall limit. We noticed the issue when we switched to UDP syslogging and it would occasionally blow past the 16436 lo MTU! This causes Python's logging code to get an error and hilarity ensues. Change-Id: I44bdbe68babd58da58c14360379e8fef8a6b75f7
2014-05-22 19:37:53 +00:00
def test_swift_log_formatter_max_line_length(self):
# setup stream logging
sio = StringIO()
logger = utils.get_logger(None)
handler = logging.StreamHandler(sio)
formatter = utils.SwiftLogFormatter(max_line_length=10)
handler.setFormatter(formatter)
logger.logger.addHandler(handler)
def strip_value(sio):
Fix Python 3 issues in utils * LogAdapter.exception(): on Python 3, socket.error is an alias to OSError, so group both exceptions to support Python 3. * ThreadPool: call GreenPipe() with indexed parameter, don't pass the third parameter by its name, since the parameter name changed between Python 2 (bufsize) and Python 3 (buffering). * strip_value() in test_utils: StringIO.getvalue() now requires seek(0), otherwise the buffer is filled with null characters. * test_lock_file(): on Python 3, seek(0) is now required to go the beginning of a file opened in append mode. In append mode, Python goes to the end of the file. Change-Id: I4e56a51690f016a0a2e1354380ce11cff1891f64
2015-10-19 17:16:21 +02:00
sio.seek(0)
New log_max_line_length option. Log lines can get quite large, as we previously noticed with rsync error log lines. We added a setting to cap those, but it really looks like we should have just done this overall limit. We noticed the issue when we switched to UDP syslogging and it would occasionally blow past the 16436 lo MTU! This causes Python's logging code to get an error and hilarity ensues. Change-Id: I44bdbe68babd58da58c14360379e8fef8a6b75f7
2014-05-22 19:37:53 +00:00
v = sio.getvalue()
sio.truncate(0)
return v
try:
logger.info('12345')
self.assertEqual(strip_value(sio), '12345\n')
logger.info('1234567890')
self.assertEqual(strip_value(sio), '1234567890\n')
logger.info('1234567890abcde')
self.assertEqual(strip_value(sio), '12 ... de\n')
formatter.max_line_length = 11
logger.info('1234567890abcde')
self.assertEqual(strip_value(sio), '123 ... cde\n')
formatter.max_line_length = 0
logger.info('1234567890abcde')
self.assertEqual(strip_value(sio), '1234567890abcde\n')
formatter.max_line_length = 1
logger.info('1234567890abcde')
self.assertEqual(strip_value(sio), '1\n')
formatter.max_line_length = 2
logger.info('1234567890abcde')
self.assertEqual(strip_value(sio), '12\n')
formatter.max_line_length = 3
logger.info('1234567890abcde')
self.assertEqual(strip_value(sio), '123\n')
formatter.max_line_length = 4
logger.info('1234567890abcde')
self.assertEqual(strip_value(sio), '1234\n')
formatter.max_line_length = 5
logger.info('1234567890abcde')
self.assertEqual(strip_value(sio), '12345\n')
formatter.max_line_length = 6
logger.info('1234567890abcde')
self.assertEqual(strip_value(sio), '123456\n')
formatter.max_line_length = 7
logger.info('1234567890abcde')
self.assertEqual(strip_value(sio), '1 ... e\n')
formatter.max_line_length = -10
logger.info('1234567890abcde')
self.assertEqual(strip_value(sio), '1234567890abcde\n')
finally:
logger.logger.removeHandler(handler)
Fix logger test coupling Two tests fail if test/unit/common/test_daemon.py is excluded from the test set: test.unit.common.test_utils.TestUtils.test_swift_log_formatter test.unit.common.test_utils.TestStatsdLoggingDelegation.test_thread_locals They fail because they assert that logger.txn_id is None, when previous tests cause txn_id to be set. This coupling is masked when test_daemon.py is included because it reloads the common/utils module, and is executed just before test.unit.common.test_utils. The coupling can be observed by running, for example: nosetests ./test/unit/common/middleware/test_except.py ./test/unit/common/test_utils.py or nosetests ./test/unit/proxy ./test/unit/common/test_utils.py The failing tests should reset logger thread_locals before making assertions. test_utils.reset_loggers() attempts this but is broken because it sets thread_local on the wrong object. Changes in this patch: - fix reset_loggers() to reset the LogAdapter thread local state - add a reset_logger_state decorator to call reset_loggers before and after a decorated method - use the decorator for increased isolation of tests that previously called reset_loggers only on exit Change-Id: If9aa781a2dd1929a47ef69322ec8c53263d47660
2015-02-11 17:49:41 +00:00
@reset_logger_state
added client_ip to all proxy log lines not otherwise containing it
2011-08-11 14:36:02 -05:00
def test_swift_log_formatter(self):
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
# setup stream logging
sio = StringIO()
logger = utils.get_logger(None)
handler = logging.StreamHandler(sio)
added client_ip to all proxy log lines not otherwise containing it
2011-08-11 14:36:02 -05:00
handler.setFormatter(utils.SwiftLogFormatter())
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
logger.logger.addHandler(handler)
def strip_value(sio):
Fix Python 3 issues in utils * LogAdapter.exception(): on Python 3, socket.error is an alias to OSError, so group both exceptions to support Python 3. * ThreadPool: call GreenPipe() with indexed parameter, don't pass the third parameter by its name, since the parameter name changed between Python 2 (bufsize) and Python 3 (buffering). * strip_value() in test_utils: StringIO.getvalue() now requires seek(0), otherwise the buffer is filled with null characters. * test_lock_file(): on Python 3, seek(0) is now required to go the beginning of a file opened in append mode. In append mode, Python goes to the end of the file. Change-Id: I4e56a51690f016a0a2e1354380ce11cff1891f64
2015-10-19 17:16:21 +02:00
sio.seek(0)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
v = sio.getvalue()
sio.truncate(0)
return v
try:
self.assertFalse(logger.txn_id)
logger.error('my error message')
log_msg = strip_value(sio)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('my error message' in log_msg)
self.assertTrue('txn' not in log_msg)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
logger.txn_id = '12345'
logger.error('test')
log_msg = strip_value(sio)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('txn' in log_msg)
self.assertTrue('12345' in log_msg)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
# test no txn on info message
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(logger.txn_id, '12345')
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
logger.info('test')
log_msg = strip_value(sio)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('txn' not in log_msg)
self.assertTrue('12345' not in log_msg)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
# test txn already in message
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(logger.txn_id, '12345')
Python 3 deprecated the logger.warn method in favor of warning DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead Change-Id: I35df44374c4521b1f06be7a96c0b873e8c3674d8
2015-08-02 22:47:42 +05:30
logger.warning('test 12345 test')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(strip_value(sio), 'test 12345 test\n')
Force log entries to be one line Different versions of syslog-ng and probably other syslog services handle multi line log messages differently and sometimes quite poorly. This patch collapses multi line log messages into single lines before sending them on to syslog. It's just a copy of what was already in Python's logging.Formatter but altered to replace the newlines with #012. I used #012 since that's a convention we've already used elsewhere in Swift. Change-Id: I8d0509b7cf48e45c2cf6480b51c67eec5bc94fe2
2013-02-26 18:38:09 +00:00
# Test multi line collapsing
logger.error('my\nerror\nmessage')
log_msg = strip_value(sio)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('my#012error#012message' in log_msg)
added client_ip to all proxy log lines not otherwise containing it
2011-08-11 14:36:02 -05:00
# test client_ip
self.assertFalse(logger.client_ip)
logger.error('my error message')
log_msg = strip_value(sio)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('my error message' in log_msg)
self.assertTrue('client_ip' not in log_msg)
added client_ip to all proxy log lines not otherwise containing it
2011-08-11 14:36:02 -05:00
logger.client_ip = '1.2.3.4'
logger.error('test')
log_msg = strip_value(sio)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('client_ip' in log_msg)
self.assertTrue('1.2.3.4' in log_msg)
added client_ip to all proxy log lines not otherwise containing it
2011-08-11 14:36:02 -05:00
# test no client_ip on info message
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(logger.client_ip, '1.2.3.4')
added client_ip to all proxy log lines not otherwise containing it
2011-08-11 14:36:02 -05:00
logger.info('test')
log_msg = strip_value(sio)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue('client_ip' not in log_msg)
self.assertTrue('1.2.3.4' not in log_msg)
added client_ip to all proxy log lines not otherwise containing it
2011-08-11 14:36:02 -05:00
# test client_ip (and txn) already in message
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(logger.client_ip, '1.2.3.4')
Python 3 deprecated the logger.warn method in favor of warning DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead Change-Id: I35df44374c4521b1f06be7a96c0b873e8c3674d8
2015-08-02 22:47:42 +05:30
logger.warning('test 1.2.3.4 test 12345')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(strip_value(sio), 'test 1.2.3.4 test 12345\n')
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
finally:
logger.logger.removeHandler(handler)
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
Revert "Refactor common/utils methods to common/ondisk" This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32 Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 12:10:31 +00:00
def test_storage_directory(self):
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils.storage_directory('objects', '1', 'ABCDEF'),
'objects/1/DEF/ABCDEF')
Revert "Refactor common/utils methods to common/ondisk" This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32 Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 12:10:31 +00:00
Allow hostnames for nodes in Rings This change modifies the swift-ring-builder and introduces new format of sub-commands (search, list_parts, set_weight, set_info and remove) in addition to add sub-command so that hostnames can be used in place of an ip-address for the sub-commands. The account reaper, container synchronizer, and replicators were also updated so that they still have a way to identify a particular device as being "local". Previously this was Change-Id: Ie471902413002872fc6755bacd36af3b9c613b74 Change-Id: Ieff583ffb932133e3820744a3f8f9f491686b08d Co-Authored-By: Alex Pecoraro <alex.pecoraro@emc.com> Implements: blueprint allow-hostnames-for-nodes-in-rings
2014-12-09 04:44:52 +09:00
def test_expand_ipv6(self):
expanded_ipv6 = "fe80::204:61ff:fe9d:f156"
upper_ipv6 = "fe80:0000:0000:0000:0204:61ff:fe9d:f156"
self.assertEqual(expanded_ipv6, utils.expand_ipv6(upper_ipv6))
omit_ipv6 = "fe80:0000:0000::0204:61ff:fe9d:f156"
self.assertEqual(expanded_ipv6, utils.expand_ipv6(omit_ipv6))
less_num_ipv6 = "fe80:0:00:000:0204:61ff:fe9d:f156"
self.assertEqual(expanded_ipv6, utils.expand_ipv6(less_num_ipv6))
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
def test_whataremyips(self):
myips = utils.whataremyips()
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(len(myips) > 1)
self.assertTrue('127.0.0.1' in myips)
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
Allow 1+ object-servers-per-disk deployment Enabled by a new > 0 integer config value, "servers_per_port" in the [DEFAULT] config section for object-server and/or replication server configs. The setting's integer value determines how many different object-server workers handle requests for any single unique local port in the ring. In this mode, the parent swift-object-server process continues to run as the original user (i.e. root if low-port binding is required), binds to all ports as defined in the ring, and forks off the specified number of workers per listen socket. The child, per-port servers drop privileges and behave pretty much how object-server workers always have, except that because the ring has unique ports per disk, the object-servers will only be handling requests for a single disk. The parent process detects dead servers and restarts them (with the correct listen socket), starts missing servers when an updated ring file is found with a device on the server with a new port, and kills extraneous servers when their port is found to no longer be in the ring. The ring files are stat'ed at most every "ring_check_interval" seconds, as configured in the object-server config (same default of 15s). Immediately stopping all swift-object-worker processes still works by sending the parent a SIGTERM. Likewise, a SIGHUP to the parent process still causes the parent process to close all listen sockets and exit, allowing existing children to finish serving their existing requests. The drop_privileges helper function now has an optional param to suppress the setsid() call, which otherwise screws up the child workers' process management. The class method RingData.load() can be told to only load the ring metadata (i.e. everything except replica2part2dev_id) with the optional kwarg, header_only=True. This is used to keep the parent and all forked off workers from unnecessarily having full copies of all storage policy rings in memory. A new helper class, swift.common.storage_policy.BindPortsCache, provides a method to return a set of all device ports in all rings for the server on which it is instantiated (identified by its set of IP addresses). The BindPortsCache instance will track mtimes of ring files, so they are not opened more frequently than necessary. This patch includes enhancements to the probe tests and object-replicator/object-reconstructor config plumbing to allow the probe tests to work correctly both in the "normal" config (same IP but unique ports for each SAIO "server") and a server-per-port setup where each SAIO "server" must have a unique IP address and unique port per disk within each "server". The main probe tests only work with 4 servers and 4 disks, but you can see the difference in the rings for the EC probe tests where there are 2 disks per server for a total of 8 disks. Specifically, swift.common.ring.utils.is_local_device() will ignore the ports when the "my_port" argument is None. Then, object-replicator and object-reconstructor both set self.bind_port to None if server_per_port is enabled. Bonus improvement for IPv6 addresses in is_local_device(). This PR for vagrant-swift-all-in-one will aid in testing this patch: https://github.com/swiftstack/vagrant-swift-all-in-one/pull/16/ Also allow SAIO to answer is_local_device() better; common SAIO setups have multiple "servers" all on the same host with different ports for the different "servers" (which happen to match the IPs specified in the rings for the devices on each of those "servers"). However, you can configure the SAIO to have different localhost IP addresses (e.g. 127.0.0.1, 127.0.0.2, etc.) in the ring and in the servers' config files' bind_ip setting. This new whataremyips() implementation combined with a little plumbing allows is_local_device() to accurately answer, even on an SAIO. In the default case (an unspecified bind_ip defaults to '0.0.0.0') as well as an explict "bind to everything" like '0.0.0.0' or '::', whataremyips() behaves as it always has, returning all IP addresses for the server. Also updated probe tests to handle each "server" in the SAIO having a unique IP address. For some (noisy) benchmarks that show servers_per_port=X is at least as good as the same number of "normal" workers: https://gist.github.com/dbishop/c214f89ca708a6b1624a#file-summary-md Benchmarks showing the benefits of I/O isolation with a small number of slow disks: https://gist.github.com/dbishop/fd0ab067babdecfb07ca#file-results-md If you were wondering what the overhead of threads_per_disk looks like: https://gist.github.com/dbishop/1d14755fedc86a161718#file-tabular_results-md DocImpact Change-Id: I2239a4000b41a7e7cc53465ce794af49d44796c6
2015-05-14 22:14:15 -07:00
def test_whataremyips_bind_to_all(self):
for any_addr in ('0.0.0.0', '0000:0000:0000:0000:0000:0000:0000:0000',
'::0', '::0000', '::',
# Wacky parse-error input produces all IPs
'I am a bear'):
myips = utils.whataremyips(any_addr)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(len(myips) > 1)
self.assertTrue('127.0.0.1' in myips)
Allow 1+ object-servers-per-disk deployment Enabled by a new > 0 integer config value, "servers_per_port" in the [DEFAULT] config section for object-server and/or replication server configs. The setting's integer value determines how many different object-server workers handle requests for any single unique local port in the ring. In this mode, the parent swift-object-server process continues to run as the original user (i.e. root if low-port binding is required), binds to all ports as defined in the ring, and forks off the specified number of workers per listen socket. The child, per-port servers drop privileges and behave pretty much how object-server workers always have, except that because the ring has unique ports per disk, the object-servers will only be handling requests for a single disk. The parent process detects dead servers and restarts them (with the correct listen socket), starts missing servers when an updated ring file is found with a device on the server with a new port, and kills extraneous servers when their port is found to no longer be in the ring. The ring files are stat'ed at most every "ring_check_interval" seconds, as configured in the object-server config (same default of 15s). Immediately stopping all swift-object-worker processes still works by sending the parent a SIGTERM. Likewise, a SIGHUP to the parent process still causes the parent process to close all listen sockets and exit, allowing existing children to finish serving their existing requests. The drop_privileges helper function now has an optional param to suppress the setsid() call, which otherwise screws up the child workers' process management. The class method RingData.load() can be told to only load the ring metadata (i.e. everything except replica2part2dev_id) with the optional kwarg, header_only=True. This is used to keep the parent and all forked off workers from unnecessarily having full copies of all storage policy rings in memory. A new helper class, swift.common.storage_policy.BindPortsCache, provides a method to return a set of all device ports in all rings for the server on which it is instantiated (identified by its set of IP addresses). The BindPortsCache instance will track mtimes of ring files, so they are not opened more frequently than necessary. This patch includes enhancements to the probe tests and object-replicator/object-reconstructor config plumbing to allow the probe tests to work correctly both in the "normal" config (same IP but unique ports for each SAIO "server") and a server-per-port setup where each SAIO "server" must have a unique IP address and unique port per disk within each "server". The main probe tests only work with 4 servers and 4 disks, but you can see the difference in the rings for the EC probe tests where there are 2 disks per server for a total of 8 disks. Specifically, swift.common.ring.utils.is_local_device() will ignore the ports when the "my_port" argument is None. Then, object-replicator and object-reconstructor both set self.bind_port to None if server_per_port is enabled. Bonus improvement for IPv6 addresses in is_local_device(). This PR for vagrant-swift-all-in-one will aid in testing this patch: https://github.com/swiftstack/vagrant-swift-all-in-one/pull/16/ Also allow SAIO to answer is_local_device() better; common SAIO setups have multiple "servers" all on the same host with different ports for the different "servers" (which happen to match the IPs specified in the rings for the devices on each of those "servers"). However, you can configure the SAIO to have different localhost IP addresses (e.g. 127.0.0.1, 127.0.0.2, etc.) in the ring and in the servers' config files' bind_ip setting. This new whataremyips() implementation combined with a little plumbing allows is_local_device() to accurately answer, even on an SAIO. In the default case (an unspecified bind_ip defaults to '0.0.0.0') as well as an explict "bind to everything" like '0.0.0.0' or '::', whataremyips() behaves as it always has, returning all IP addresses for the server. Also updated probe tests to handle each "server" in the SAIO having a unique IP address. For some (noisy) benchmarks that show servers_per_port=X is at least as good as the same number of "normal" workers: https://gist.github.com/dbishop/c214f89ca708a6b1624a#file-summary-md Benchmarks showing the benefits of I/O isolation with a small number of slow disks: https://gist.github.com/dbishop/fd0ab067babdecfb07ca#file-results-md If you were wondering what the overhead of threads_per_disk looks like: https://gist.github.com/dbishop/1d14755fedc86a161718#file-tabular_results-md DocImpact Change-Id: I2239a4000b41a7e7cc53465ce794af49d44796c6
2015-05-14 22:14:15 -07:00
def test_whataremyips_bind_ip_specific(self):
self.assertEqual(['1.2.3.4'], utils.whataremyips('1.2.3.4'))
Don't append %interface with ipv6 ips. - Since netifaces is appending a %interface (i.e: %eth0) at the end and does not make a correct ipv6 address we are removing it. - Improve the tests of whataremyips along the way. Closes-Bug: 1209443 Change-Id: I585e795083783009961b429607ca3f66b8d7ec30
2013-08-08 11:11:39 +02:00
def test_whataremyips_error(self):
def my_interfaces():
return ['eth0']
def my_ifaddress_error(interface):
raise ValueError
Get rid of contextlib.nested() for py3 contextlib.nested() is missing completely in Python 3. Since 2.7, we can use multiple context managers in a 'with' statement, like so: with thing1() as t1, thing2() as t2: do_stuff() Now, if we had some code that needed to nest an arbitrary number of context managers, there's stuff we could do with contextlib.ExitStack and such... but we don't. We only use contextlib.nested() in tests to set up bunches of mocks without crazy-deep indentation, and all that stuff fits perfectly into multiple-context-manager 'with' statements. Change-Id: Id472958b007948f05dbd4c7fb8cf3ffab58e2681
2015-10-21 13:05:37 -07:00
with patch('netifaces.interfaces', my_interfaces), \
patch('netifaces.ifaddresses', my_ifaddress_error):
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils.whataremyips(), [])
Don't append %interface with ipv6 ips. - Since netifaces is appending a %interface (i.e: %eth0) at the end and does not make a correct ipv6 address we are removing it. - Improve the tests of whataremyips along the way. Closes-Bug: 1209443 Change-Id: I585e795083783009961b429607ca3f66b8d7ec30
2013-08-08 11:11:39 +02:00
def test_whataremyips_ipv6(self):
test_ipv6_address = '2001:6b0:dead:beef:2::32'
test_interface = 'eth0'
def my_ipv6_interfaces():
return ['eth0']
def my_ipv6_ifaddresses(interface):
return {AF_INET6:
[{'netmask': 'ffff:ffff:ffff:ffff::',
'addr': '%s%%%s' % (test_ipv6_address, test_interface)}]}
Get rid of contextlib.nested() for py3 contextlib.nested() is missing completely in Python 3. Since 2.7, we can use multiple context managers in a 'with' statement, like so: with thing1() as t1, thing2() as t2: do_stuff() Now, if we had some code that needed to nest an arbitrary number of context managers, there's stuff we could do with contextlib.ExitStack and such... but we don't. We only use contextlib.nested() in tests to set up bunches of mocks without crazy-deep indentation, and all that stuff fits perfectly into multiple-context-manager 'with' statements. Change-Id: Id472958b007948f05dbd4c7fb8cf3ffab58e2681
2015-10-21 13:05:37 -07:00
with patch('netifaces.interfaces', my_ipv6_interfaces), \
patch('netifaces.ifaddresses', my_ipv6_ifaddresses):
Don't append %interface with ipv6 ips. - Since netifaces is appending a %interface (i.e: %eth0) at the end and does not make a correct ipv6 address we are removing it. - Improve the tests of whataremyips along the way. Closes-Bug: 1209443 Change-Id: I585e795083783009961b429607ca3f66b8d7ec30
2013-08-08 11:11:39 +02:00
myips = utils.whataremyips()
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(myips), 1)
self.assertEqual(myips[0], test_ipv6_address)
Don't append %interface with ipv6 ips. - Since netifaces is appending a %interface (i.e: %eth0) at the end and does not make a correct ipv6 address we are removing it. - Improve the tests of whataremyips along the way. Closes-Bug: 1209443 Change-Id: I585e795083783009961b429607ca3f66b8d7ec30
2013-08-08 11:11:39 +02:00
Revert "Refactor common/utils methods to common/ondisk" This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32 Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 12:10:31 +00:00
def test_hash_path(self):
# Yes, these tests are deliberately very fragile. We want to make sure
# that if someones changes the results hash_path produces, they know it
Add unit tests for utils.validate_hash_path This patch add some unit tests to prevent regression and to describe validate_hash_path behavior, I found in review with https://review.openstack.org/#/c/231864/ *bonus* - Fix test_hash_path to use "with" syntax instead of "try/finally" for assigning a testing value into a global variable. Change-Id: I948999a8fb8addb9a378dbf8bee853b205aeafad
2015-10-07 02:14:49 -07:00
with mock.patch('swift.common.utils.HASH_PATH_PREFIX', ''):
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils.hash_path('a'),
'1c84525acb02107ea475dcd3d09c2c58')
self.assertEqual(utils.hash_path('a', 'c'),
'33379ecb053aa5c9e356c68997cbb59e')
self.assertEqual(utils.hash_path('a', 'c', 'o'),
'06fbf0b514e5199dfc4e00f42eb5ea83')
self.assertEqual(utils.hash_path('a', 'c', 'o', raw_digest=False),
'06fbf0b514e5199dfc4e00f42eb5ea83')
self.assertEqual(utils.hash_path('a', 'c', 'o', raw_digest=True),
'\x06\xfb\xf0\xb5\x14\xe5\x19\x9d\xfcN'
'\x00\xf4.\xb5\xea\x83')
Revert "Refactor common/utils methods to common/ondisk" This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32 Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 12:10:31 +00:00
self.assertRaises(ValueError, utils.hash_path, 'a', object='o')
utils.HASH_PATH_PREFIX = 'abcdef'
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils.hash_path('a', 'c', 'o', raw_digest=False),
'363f9b535bfb7d17a43a46a358afca0e')
Add unit tests for utils.validate_hash_path This patch add some unit tests to prevent regression and to describe validate_hash_path behavior, I found in review with https://review.openstack.org/#/c/231864/ *bonus* - Fix test_hash_path to use "with" syntax instead of "try/finally" for assigning a testing value into a global variable. Change-Id: I948999a8fb8addb9a378dbf8bee853b205aeafad
2015-10-07 02:14:49 -07:00
def test_validate_hash_conf(self):
# no section causes InvalidHashPathConfigError
self._test_validate_hash_conf([], [], True)
# 'swift-hash' section is there but no options causes
# InvalidHashPathConfigError
self._test_validate_hash_conf(['swift-hash'], [], True)
# if we have the section and either of prefix or suffix,
# InvalidHashPathConfigError doesn't occur
self._test_validate_hash_conf(
['swift-hash'], ['swift_hash_path_prefix'], False)
self._test_validate_hash_conf(
['swift-hash'], ['swift_hash_path_suffix'], False)
# definitely, we have the section and both of them,
# InvalidHashPathConfigError doesn't occur
self._test_validate_hash_conf(
['swift-hash'],
['swift_hash_path_suffix', 'swift_hash_path_prefix'], False)
# But invalid section name should make an error even if valid
# options are there
self._test_validate_hash_conf(
['swift-hash-xxx'],
['swift_hash_path_suffix', 'swift_hash_path_prefix'], True)
def _test_validate_hash_conf(self, sections, options, should_raise_error):
class FakeConfigParser(object):
def read(self, conf_path):
return True
def get(self, section, option):
if section not in sections:
raise NoSectionError('section error')
elif option not in options:
raise NoOptionError('option error', 'this option')
else:
return 'some_option_value'
with mock.patch('swift.common.utils.HASH_PATH_PREFIX', ''), \
mock.patch('swift.common.utils.HASH_PATH_SUFFIX', ''), \
mock.patch('swift.common.utils.ConfigParser',
FakeConfigParser):
try:
utils.validate_hash_conf()
except utils.InvalidHashPathConfigError:
if not should_raise_error:
self.fail('validate_hash_conf should not raise an error')
else:
if should_raise_error:
self.fail('validate_hash_conf should raise an error')
Revert "Refactor common/utils methods to common/ondisk" This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32 Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 12:10:31 +00:00
Improved fallocate and posix_fadvise no-ops when unavailable.
2010-07-29 00:44:21 +00:00
def test_load_libc_function(self):
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(callable(
Improved fallocate and posix_fadvise no-ops when unavailable.
2010-07-29 00:44:21 +00:00
utils.load_libc_function('printf')))
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(callable(
Improved fallocate and posix_fadvise no-ops when unavailable.
2010-07-29 00:44:21 +00:00
utils.load_libc_function('some_not_real_function')))
Zero-copy object-server GET responses with splice() This commit lets the object server use splice() and tee() to move data from disk to the network without ever copying it into user space. Requires Linux. Sorry, FreeBSD folks. You still have the old mechanism, as does anyone who doesn't want to use splice. This requires a relatively recent kernel (2.6.38+) to work, which includes the two most recent Ubuntu LTS releases (Precise and Trusty) as well as RHEL 7. However, it excludes Lucid and RHEL 6. On those systems, setting "splice = on" will result in warnings in the logs but no actual use of splice. Note that this only applies to GET responses without Range headers. It can easily be extended to single-range GET requests, but this commit leaves that for future work. Same goes for PUT requests, or at least non-chunked ones. On some real hardware I had laying around (not a VM), this produced a 37% reduction in CPU usage for GETs made directly to the object server. Measurements were done by looking at /proc/<pid>/stat, specifically the utime and stime fields (user and kernel CPU jiffies, respectively). Note: There is a Python module called "splicetee" available on PyPi, but it's licensed under the GPL, so it cannot easily be added to OpenStack's requirements. That's why this patch uses ctypes instead. Also fixed a long-standing annoyance in FakeLogger: >>> fake_logger.warn('stuff') >>> fake_logger.get_lines_for_level('warn') [] >>> This, of course, is because the correct log level is 'warning'. Now you get a KeyError if you call get_lines_for_level with a bogus log level. Change-Id: Ic6d6b833a5b04ca2019be94b1b90d941929d21c8
2014-06-10 14:15:27 -07:00
self.assertRaises(AttributeError,
utils.load_libc_function, 'some_not_real_function',
fail_if_missing=True)
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
added readconf unit test
2010-10-08 14:14:08 -05:00
def test_readconf(self):
updated readconf test
2010-10-08 14:29:24 -05:00
conf = '''[section1]
foo = bar
[section2]
updated readconf test
2010-10-08 14:47:40 -05:00
log_name = yarr'''
went ahead and spoofed a [func_test] section
2011-02-24 12:28:17 -06:00
# setup a real file
Use randomly named file for temporay file Use tempfile.mkstemp to generate temporary file to avoid collision with an existing 'test' file in /tmp. Change-Id: Ic2f7f64ee9826afa6f04debd763e8c7a0eb25988 Fixes: bug #1213845
2013-08-19 11:41:38 +02:00
fd, temppath = tempfile.mkstemp(dir='/tmp')
with os.fdopen(fd, 'wb') as f:
went ahead and spoofed a [func_test] section
2011-02-24 12:28:17 -06:00
f.write(conf)
Use randomly named file for temporay file Use tempfile.mkstemp to generate temporary file to avoid collision with an existing 'test' file in /tmp. Change-Id: Ic2f7f64ee9826afa6f04debd763e8c7a0eb25988 Fixes: bug #1213845
2013-08-19 11:41:38 +02:00
make_filename = lambda: temppath
went ahead and spoofed a [func_test] section
2011-02-24 12:28:17 -06:00
# setup a file stream
make_fp = lambda: StringIO(conf)
for conf_object_maker in (make_filename, make_fp):
Expiring Objects Support Please see the doc/source/overview_expiring_objects.rst for more detail. Change-Id: I4ab49e731248cf62ce10001016e0c819cc531738
2011-10-26 21:42:24 +00:00
conffile = conf_object_maker()
result = utils.readconf(conffile)
expected = {'__file__': conffile,
'log_name': None,
went ahead and spoofed a [func_test] section
2011-02-24 12:28:17 -06:00
'section1': {'foo': 'bar'},
'section2': {'log_name': 'yarr'}}
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(result, expected)
Expiring Objects Support Please see the doc/source/overview_expiring_objects.rst for more detail. Change-Id: I4ab49e731248cf62ce10001016e0c819cc531738
2011-10-26 21:42:24 +00:00
conffile = conf_object_maker()
result = utils.readconf(conffile, 'section1')
expected = {'__file__': conffile, 'log_name': 'section1',
'foo': 'bar'}
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(result, expected)
Expiring Objects Support Please see the doc/source/overview_expiring_objects.rst for more detail. Change-Id: I4ab49e731248cf62ce10001016e0c819cc531738
2011-10-26 21:42:24 +00:00
conffile = conf_object_maker()
result = utils.readconf(conffile,
went ahead and spoofed a [func_test] section
2011-02-24 12:28:17 -06:00
'section2').get('log_name')
expected = 'yarr'
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(result, expected)
Expiring Objects Support Please see the doc/source/overview_expiring_objects.rst for more detail. Change-Id: I4ab49e731248cf62ce10001016e0c819cc531738
2011-10-26 21:42:24 +00:00
conffile = conf_object_maker()
result = utils.readconf(conffile, 'section1',
went ahead and spoofed a [func_test] section
2011-02-24 12:28:17 -06:00
log_name='foo').get('log_name')
expected = 'foo'
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(result, expected)
Expiring Objects Support Please see the doc/source/overview_expiring_objects.rst for more detail. Change-Id: I4ab49e731248cf62ce10001016e0c819cc531738
2011-10-26 21:42:24 +00:00
conffile = conf_object_maker()
result = utils.readconf(conffile, 'section1',
went ahead and spoofed a [func_test] section
2011-02-24 12:28:17 -06:00
defaults={'bar': 'baz'})
Expiring Objects Support Please see the doc/source/overview_expiring_objects.rst for more detail. Change-Id: I4ab49e731248cf62ce10001016e0c819cc531738
2011-10-26 21:42:24 +00:00
expected = {'__file__': conffile, 'log_name': 'section1',
'foo': 'bar', 'bar': 'baz'}
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(result, expected)
Use randomly named file for temporay file Use tempfile.mkstemp to generate temporary file to avoid collision with an existing 'test' file in /tmp. Change-Id: Ic2f7f64ee9826afa6f04debd763e8c7a0eb25988 Fixes: bug #1213845
2013-08-19 11:41:38 +02:00
self.assertRaises(SystemExit, utils.readconf, temppath, 'section3')
os.unlink(temppath)
self.assertRaises(SystemExit, utils.readconf, temppath)
added readconf unit test
2010-10-08 14:14:08 -05:00
adding unit tests
2011-08-30 12:07:32 -07:00
def test_readconf_raw(self):
conf = '''[section1]
foo = bar
[section2]
log_name = %(yarr)s'''
# setup a real file
Use randomly named file for temporay file Use tempfile.mkstemp to generate temporary file to avoid collision with an existing 'test' file in /tmp. Change-Id: Ic2f7f64ee9826afa6f04debd763e8c7a0eb25988 Fixes: bug #1213845
2013-08-19 11:41:38 +02:00
fd, temppath = tempfile.mkstemp(dir='/tmp')
with os.fdopen(fd, 'wb') as f:
adding unit tests
2011-08-30 12:07:32 -07:00
f.write(conf)
Use randomly named file for temporay file Use tempfile.mkstemp to generate temporary file to avoid collision with an existing 'test' file in /tmp. Change-Id: Ic2f7f64ee9826afa6f04debd763e8c7a0eb25988 Fixes: bug #1213845
2013-08-19 11:41:38 +02:00
make_filename = lambda: temppath
adding unit tests
2011-08-30 12:07:32 -07:00
# setup a file stream
make_fp = lambda: StringIO(conf)
for conf_object_maker in (make_filename, make_fp):
Expiring Objects Support Please see the doc/source/overview_expiring_objects.rst for more detail. Change-Id: I4ab49e731248cf62ce10001016e0c819cc531738
2011-10-26 21:42:24 +00:00
conffile = conf_object_maker()
result = utils.readconf(conffile, raw=True)
expected = {'__file__': conffile,
'log_name': None,
adding unit tests
2011-08-30 12:07:32 -07:00
'section1': {'foo': 'bar'},
'section2': {'log_name': '%(yarr)s'}}
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(result, expected)
Use randomly named file for temporay file Use tempfile.mkstemp to generate temporary file to avoid collision with an existing 'test' file in /tmp. Change-Id: Ic2f7f64ee9826afa6f04debd763e8c7a0eb25988 Fixes: bug #1213845
2013-08-19 11:41:38 +02:00
os.unlink(temppath)
self.assertRaises(SystemExit, utils.readconf, temppath)
adding unit tests
2011-08-30 12:07:32 -07:00
conf.d support Allow Swift daemons and servers to optionally accept a directory as the configuration parameter. Directory based configuration leverages ConfigParser's native multi-file support. Files ending in '.conf' in the given directory are parsed in lexicographical order. Filenames starting with '.' are ignored. A mixture of file and directory configuration paths is not supported - if the configuration path is a file behavior is unchanged. * update swift-init to search for conf.d paths when building servers (e.g. /etc/swift/proxy-server.conf.d/) * new script swift-config can be used to inspect the cumulative configuration * pull a little bit of code out of run_wsgi and test separately * fix example config bug for the proxy servers client_disconnect option * added section on directory based configuration to deployment guide DocImpact Implements: blueprint confd Change-Id: I89b0f48e538117f28590cf6698401f74ef58003b
2013-03-25 16:34:43 -07:00
def test_readconf_dir(self):
config_dir = {
'server.conf.d/01.conf': """
[DEFAULT]
port = 8080
foo = bar
[section1]
name=section1
""",
'server.conf.d/section2.conf': """
[DEFAULT]
port = 8081
bar = baz
[section2]
name=section2
""",
'other-server.conf.d/01.conf': """
[DEFAULT]
port = 8082
[section3]
name=section3
"""
}
# strip indent from test config contents
config_dir = dict((f, dedent(c)) for (f, c) in config_dir.items())
with temptree(*zip(*config_dir.items())) as path:
conf_dir = os.path.join(path, 'server.conf.d')
conf = utils.readconf(conf_dir)
expected = {
'__file__': os.path.join(path, 'server.conf.d'),
'log_name': None,
'section1': {
'port': '8081',
'foo': 'bar',
'bar': 'baz',
'name': 'section1',
},
'section2': {
'port': '8081',
'foo': 'bar',
'bar': 'baz',
'name': 'section2',
},
}
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(conf, expected)
conf.d support Allow Swift daemons and servers to optionally accept a directory as the configuration parameter. Directory based configuration leverages ConfigParser's native multi-file support. Files ending in '.conf' in the given directory are parsed in lexicographical order. Filenames starting with '.' are ignored. A mixture of file and directory configuration paths is not supported - if the configuration path is a file behavior is unchanged. * update swift-init to search for conf.d paths when building servers (e.g. /etc/swift/proxy-server.conf.d/) * new script swift-config can be used to inspect the cumulative configuration * pull a little bit of code out of run_wsgi and test separately * fix example config bug for the proxy servers client_disconnect option * added section on directory based configuration to deployment guide DocImpact Implements: blueprint confd Change-Id: I89b0f48e538117f28590cf6698401f74ef58003b
2013-03-25 16:34:43 -07:00
def test_readconf_dir_ignores_hidden_and_nondotconf_files(self):
config_dir = {
'server.conf.d/01.conf': """
[section1]
port = 8080
""",
'server.conf.d/.01.conf.swp': """
[section]
port = 8081
""",
'server.conf.d/01.conf-bak': """
[section]
port = 8082
""",
}
# strip indent from test config contents
config_dir = dict((f, dedent(c)) for (f, c) in config_dir.items())
with temptree(*zip(*config_dir.items())) as path:
conf_dir = os.path.join(path, 'server.conf.d')
conf = utils.readconf(conf_dir)
expected = {
'__file__': os.path.join(path, 'server.conf.d'),
'log_name': None,
'section1': {
'port': '8080',
},
}
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(conf, expected)
conf.d support Allow Swift daemons and servers to optionally accept a directory as the configuration parameter. Directory based configuration leverages ConfigParser's native multi-file support. Files ending in '.conf' in the given directory are parsed in lexicographical order. Filenames starting with '.' are ignored. A mixture of file and directory configuration paths is not supported - if the configuration path is a file behavior is unchanged. * update swift-init to search for conf.d paths when building servers (e.g. /etc/swift/proxy-server.conf.d/) * new script swift-config can be used to inspect the cumulative configuration * pull a little bit of code out of run_wsgi and test separately * fix example config bug for the proxy servers client_disconnect option * added section on directory based configuration to deployment guide DocImpact Implements: blueprint confd Change-Id: I89b0f48e538117f28590cf6698401f74ef58003b
2013-03-25 16:34:43 -07:00
removed unneeded daemonize function from utils, pulled get_socket out of run_wsgi, reworked test_utils and test_wsgi
2010-11-19 12:15:41 -06:00
def test_drop_privileges(self):
user = getuser()
# over-ride os with mock
Call setgroups() before setuid() and setgid(). Fixes bug 989569. This patch ensures that the list of groups is completely reset when dropping privileges. Change-Id: I049f75e66e08a4a6361504b013bc68c4c38ef093
2012-02-07 14:19:22 -05:00
required_func_calls = ('setgroups', 'setgid', 'setuid', 'setsid',
'chdir', 'umask')
removed unneeded daemonize function from utils, pulled get_socket out of run_wsgi, reworked test_utils and test_wsgi
2010-11-19 12:15:41 -06:00
utils.os = MockOs(called_funcs=required_func_calls)
# exercise the code
utils.drop_privileges(user)
for func in required_func_calls:
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(utils.os.called_funcs[func])
drop_privileges should also set HOME E.g. if HOME is not set, swift-proxy will create the keystone_signing file not in HOME but in /root. This is because the swift user doesn't have a shell in /etc/passwd and so it doesn't set environment variables when impersonating. Change-Id: I3013007e0dadf6ddccc176e142b7c78c5d63a351
2012-11-30 16:08:17 +00:00
import pwd
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(pwd.getpwnam(user)[5], utils.os.environ['HOME'])
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
Add secondary groups to user during privilege escalation setgid provides the primary group, setgroups sets the secondary groups. Prior to this patch, we would do a setgroups with an empty list, effectively wiping secondary groups. We now verify which secondary groups the user is member of and escalate the privileges accordingly. Change-Id: I33a10edd448b3ac5aa758a8d1d70e582cf421c7d Closes-Bug: 1269473
2014-01-20 13:30:58 -05:00
groups = [g.gr_gid for g in grp.getgrall() if user in g.gr_mem]
groups.append(pwd.getpwnam(user).pw_gid)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(set(groups), set(os.getgroups()))
Add secondary groups to user during privilege escalation setgid provides the primary group, setgroups sets the secondary groups. Prior to this patch, we would do a setgroups with an empty list, effectively wiping secondary groups. We now verify which secondary groups the user is member of and escalate the privileges accordingly. Change-Id: I33a10edd448b3ac5aa758a8d1d70e582cf421c7d Closes-Bug: 1269473
2014-01-20 13:30:58 -05:00
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
# reset; test same args, OSError trying to get session leader
removed unneeded daemonize function from utils, pulled get_socket out of run_wsgi, reworked test_utils and test_wsgi
2010-11-19 12:15:41 -06:00
utils.os = MockOs(called_funcs=required_func_calls,
raise_funcs=('setsid',))
for func in required_func_calls:
self.assertFalse(utils.os.called_funcs.get(func, False))
utils.drop_privileges(user)
for func in required_func_calls:
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(utils.os.called_funcs[func])
removed unneeded daemonize function from utils, pulled get_socket out of run_wsgi, reworked test_utils and test_wsgi
2010-11-19 12:15:41 -06:00
Allow 1+ object-servers-per-disk deployment Enabled by a new > 0 integer config value, "servers_per_port" in the [DEFAULT] config section for object-server and/or replication server configs. The setting's integer value determines how many different object-server workers handle requests for any single unique local port in the ring. In this mode, the parent swift-object-server process continues to run as the original user (i.e. root if low-port binding is required), binds to all ports as defined in the ring, and forks off the specified number of workers per listen socket. The child, per-port servers drop privileges and behave pretty much how object-server workers always have, except that because the ring has unique ports per disk, the object-servers will only be handling requests for a single disk. The parent process detects dead servers and restarts them (with the correct listen socket), starts missing servers when an updated ring file is found with a device on the server with a new port, and kills extraneous servers when their port is found to no longer be in the ring. The ring files are stat'ed at most every "ring_check_interval" seconds, as configured in the object-server config (same default of 15s). Immediately stopping all swift-object-worker processes still works by sending the parent a SIGTERM. Likewise, a SIGHUP to the parent process still causes the parent process to close all listen sockets and exit, allowing existing children to finish serving their existing requests. The drop_privileges helper function now has an optional param to suppress the setsid() call, which otherwise screws up the child workers' process management. The class method RingData.load() can be told to only load the ring metadata (i.e. everything except replica2part2dev_id) with the optional kwarg, header_only=True. This is used to keep the parent and all forked off workers from unnecessarily having full copies of all storage policy rings in memory. A new helper class, swift.common.storage_policy.BindPortsCache, provides a method to return a set of all device ports in all rings for the server on which it is instantiated (identified by its set of IP addresses). The BindPortsCache instance will track mtimes of ring files, so they are not opened more frequently than necessary. This patch includes enhancements to the probe tests and object-replicator/object-reconstructor config plumbing to allow the probe tests to work correctly both in the "normal" config (same IP but unique ports for each SAIO "server") and a server-per-port setup where each SAIO "server" must have a unique IP address and unique port per disk within each "server". The main probe tests only work with 4 servers and 4 disks, but you can see the difference in the rings for the EC probe tests where there are 2 disks per server for a total of 8 disks. Specifically, swift.common.ring.utils.is_local_device() will ignore the ports when the "my_port" argument is None. Then, object-replicator and object-reconstructor both set self.bind_port to None if server_per_port is enabled. Bonus improvement for IPv6 addresses in is_local_device(). This PR for vagrant-swift-all-in-one will aid in testing this patch: https://github.com/swiftstack/vagrant-swift-all-in-one/pull/16/ Also allow SAIO to answer is_local_device() better; common SAIO setups have multiple "servers" all on the same host with different ports for the different "servers" (which happen to match the IPs specified in the rings for the devices on each of those "servers"). However, you can configure the SAIO to have different localhost IP addresses (e.g. 127.0.0.1, 127.0.0.2, etc.) in the ring and in the servers' config files' bind_ip setting. This new whataremyips() implementation combined with a little plumbing allows is_local_device() to accurately answer, even on an SAIO. In the default case (an unspecified bind_ip defaults to '0.0.0.0') as well as an explict "bind to everything" like '0.0.0.0' or '::', whataremyips() behaves as it always has, returning all IP addresses for the server. Also updated probe tests to handle each "server" in the SAIO having a unique IP address. For some (noisy) benchmarks that show servers_per_port=X is at least as good as the same number of "normal" workers: https://gist.github.com/dbishop/c214f89ca708a6b1624a#file-summary-md Benchmarks showing the benefits of I/O isolation with a small number of slow disks: https://gist.github.com/dbishop/fd0ab067babdecfb07ca#file-results-md If you were wondering what the overhead of threads_per_disk looks like: https://gist.github.com/dbishop/1d14755fedc86a161718#file-tabular_results-md DocImpact Change-Id: I2239a4000b41a7e7cc53465ce794af49d44796c6
2015-05-14 22:14:15 -07:00
def test_drop_privileges_no_call_setsid(self):
user = getuser()
# over-ride os with mock
required_func_calls = ('setgroups', 'setgid', 'setuid', 'chdir',
'umask')
bad_func_calls = ('setsid',)
utils.os = MockOs(called_funcs=required_func_calls,
raise_funcs=bad_func_calls)
# exercise the code
utils.drop_privileges(user, call_setsid=False)
for func in required_func_calls:
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(utils.os.called_funcs[func])
Allow 1+ object-servers-per-disk deployment Enabled by a new > 0 integer config value, "servers_per_port" in the [DEFAULT] config section for object-server and/or replication server configs. The setting's integer value determines how many different object-server workers handle requests for any single unique local port in the ring. In this mode, the parent swift-object-server process continues to run as the original user (i.e. root if low-port binding is required), binds to all ports as defined in the ring, and forks off the specified number of workers per listen socket. The child, per-port servers drop privileges and behave pretty much how object-server workers always have, except that because the ring has unique ports per disk, the object-servers will only be handling requests for a single disk. The parent process detects dead servers and restarts them (with the correct listen socket), starts missing servers when an updated ring file is found with a device on the server with a new port, and kills extraneous servers when their port is found to no longer be in the ring. The ring files are stat'ed at most every "ring_check_interval" seconds, as configured in the object-server config (same default of 15s). Immediately stopping all swift-object-worker processes still works by sending the parent a SIGTERM. Likewise, a SIGHUP to the parent process still causes the parent process to close all listen sockets and exit, allowing existing children to finish serving their existing requests. The drop_privileges helper function now has an optional param to suppress the setsid() call, which otherwise screws up the child workers' process management. The class method RingData.load() can be told to only load the ring metadata (i.e. everything except replica2part2dev_id) with the optional kwarg, header_only=True. This is used to keep the parent and all forked off workers from unnecessarily having full copies of all storage policy rings in memory. A new helper class, swift.common.storage_policy.BindPortsCache, provides a method to return a set of all device ports in all rings for the server on which it is instantiated (identified by its set of IP addresses). The BindPortsCache instance will track mtimes of ring files, so they are not opened more frequently than necessary. This patch includes enhancements to the probe tests and object-replicator/object-reconstructor config plumbing to allow the probe tests to work correctly both in the "normal" config (same IP but unique ports for each SAIO "server") and a server-per-port setup where each SAIO "server" must have a unique IP address and unique port per disk within each "server". The main probe tests only work with 4 servers and 4 disks, but you can see the difference in the rings for the EC probe tests where there are 2 disks per server for a total of 8 disks. Specifically, swift.common.ring.utils.is_local_device() will ignore the ports when the "my_port" argument is None. Then, object-replicator and object-reconstructor both set self.bind_port to None if server_per_port is enabled. Bonus improvement for IPv6 addresses in is_local_device(). This PR for vagrant-swift-all-in-one will aid in testing this patch: https://github.com/swiftstack/vagrant-swift-all-in-one/pull/16/ Also allow SAIO to answer is_local_device() better; common SAIO setups have multiple "servers" all on the same host with different ports for the different "servers" (which happen to match the IPs specified in the rings for the devices on each of those "servers"). However, you can configure the SAIO to have different localhost IP addresses (e.g. 127.0.0.1, 127.0.0.2, etc.) in the ring and in the servers' config files' bind_ip setting. This new whataremyips() implementation combined with a little plumbing allows is_local_device() to accurately answer, even on an SAIO. In the default case (an unspecified bind_ip defaults to '0.0.0.0') as well as an explict "bind to everything" like '0.0.0.0' or '::', whataremyips() behaves as it always has, returning all IP addresses for the server. Also updated probe tests to handle each "server" in the SAIO having a unique IP address. For some (noisy) benchmarks that show servers_per_port=X is at least as good as the same number of "normal" workers: https://gist.github.com/dbishop/c214f89ca708a6b1624a#file-summary-md Benchmarks showing the benefits of I/O isolation with a small number of slow disks: https://gist.github.com/dbishop/fd0ab067babdecfb07ca#file-results-md If you were wondering what the overhead of threads_per_disk looks like: https://gist.github.com/dbishop/1d14755fedc86a161718#file-tabular_results-md DocImpact Change-Id: I2239a4000b41a7e7cc53465ce794af49d44796c6
2015-05-14 22:14:15 -07:00
for func in bad_func_calls:
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(func not in utils.os.called_funcs)
Allow 1+ object-servers-per-disk deployment Enabled by a new > 0 integer config value, "servers_per_port" in the [DEFAULT] config section for object-server and/or replication server configs. The setting's integer value determines how many different object-server workers handle requests for any single unique local port in the ring. In this mode, the parent swift-object-server process continues to run as the original user (i.e. root if low-port binding is required), binds to all ports as defined in the ring, and forks off the specified number of workers per listen socket. The child, per-port servers drop privileges and behave pretty much how object-server workers always have, except that because the ring has unique ports per disk, the object-servers will only be handling requests for a single disk. The parent process detects dead servers and restarts them (with the correct listen socket), starts missing servers when an updated ring file is found with a device on the server with a new port, and kills extraneous servers when their port is found to no longer be in the ring. The ring files are stat'ed at most every "ring_check_interval" seconds, as configured in the object-server config (same default of 15s). Immediately stopping all swift-object-worker processes still works by sending the parent a SIGTERM. Likewise, a SIGHUP to the parent process still causes the parent process to close all listen sockets and exit, allowing existing children to finish serving their existing requests. The drop_privileges helper function now has an optional param to suppress the setsid() call, which otherwise screws up the child workers' process management. The class method RingData.load() can be told to only load the ring metadata (i.e. everything except replica2part2dev_id) with the optional kwarg, header_only=True. This is used to keep the parent and all forked off workers from unnecessarily having full copies of all storage policy rings in memory. A new helper class, swift.common.storage_policy.BindPortsCache, provides a method to return a set of all device ports in all rings for the server on which it is instantiated (identified by its set of IP addresses). The BindPortsCache instance will track mtimes of ring files, so they are not opened more frequently than necessary. This patch includes enhancements to the probe tests and object-replicator/object-reconstructor config plumbing to allow the probe tests to work correctly both in the "normal" config (same IP but unique ports for each SAIO "server") and a server-per-port setup where each SAIO "server" must have a unique IP address and unique port per disk within each "server". The main probe tests only work with 4 servers and 4 disks, but you can see the difference in the rings for the EC probe tests where there are 2 disks per server for a total of 8 disks. Specifically, swift.common.ring.utils.is_local_device() will ignore the ports when the "my_port" argument is None. Then, object-replicator and object-reconstructor both set self.bind_port to None if server_per_port is enabled. Bonus improvement for IPv6 addresses in is_local_device(). This PR for vagrant-swift-all-in-one will aid in testing this patch: https://github.com/swiftstack/vagrant-swift-all-in-one/pull/16/ Also allow SAIO to answer is_local_device() better; common SAIO setups have multiple "servers" all on the same host with different ports for the different "servers" (which happen to match the IPs specified in the rings for the devices on each of those "servers"). However, you can configure the SAIO to have different localhost IP addresses (e.g. 127.0.0.1, 127.0.0.2, etc.) in the ring and in the servers' config files' bind_ip setting. This new whataremyips() implementation combined with a little plumbing allows is_local_device() to accurately answer, even on an SAIO. In the default case (an unspecified bind_ip defaults to '0.0.0.0') as well as an explict "bind to everything" like '0.0.0.0' or '::', whataremyips() behaves as it always has, returning all IP addresses for the server. Also updated probe tests to handle each "server" in the SAIO having a unique IP address. For some (noisy) benchmarks that show servers_per_port=X is at least as good as the same number of "normal" workers: https://gist.github.com/dbishop/c214f89ca708a6b1624a#file-summary-md Benchmarks showing the benefits of I/O isolation with a small number of slow disks: https://gist.github.com/dbishop/fd0ab067babdecfb07ca#file-results-md If you were wondering what the overhead of threads_per_disk looks like: https://gist.github.com/dbishop/1d14755fedc86a161718#file-tabular_results-md DocImpact Change-Id: I2239a4000b41a7e7cc53465ce794af49d44796c6
2015-05-14 22:14:15 -07:00
Fix logger test coupling Two tests fail if test/unit/common/test_daemon.py is excluded from the test set: test.unit.common.test_utils.TestUtils.test_swift_log_formatter test.unit.common.test_utils.TestStatsdLoggingDelegation.test_thread_locals They fail because they assert that logger.txn_id is None, when previous tests cause txn_id to be set. This coupling is masked when test_daemon.py is included because it reloads the common/utils module, and is executed just before test.unit.common.test_utils. The coupling can be observed by running, for example: nosetests ./test/unit/common/middleware/test_except.py ./test/unit/common/test_utils.py or nosetests ./test/unit/proxy ./test/unit/common/test_utils.py The failing tests should reset logger thread_locals before making assertions. test_utils.reset_loggers() attempts this but is broken because it sets thread_local on the wrong object. Changes in this patch: - fix reset_loggers() to reset the LogAdapter thread local state - add a reset_logger_state decorator to call reset_loggers before and after a decorated method - use the decorator for increased isolation of tests that previously called reset_loggers only on exit Change-Id: If9aa781a2dd1929a47ef69322ec8c53263d47660
2015-02-11 17:49:41 +00:00
@reset_logger_state
removed unneeded daemonize function from utils, pulled get_socket out of run_wsgi, reworked test_utils and test_wsgi
2010-11-19 12:15:41 -06:00
def test_capture_stdio(self):
# stubs
logger = utils.get_logger(None, 'dummy')
# mock utils system modules
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
_orig_sys = utils.sys
_orig_os = utils.os
try:
utils.sys = MockSys()
utils.os = MockOs()
# basic test
utils.capture_stdio(logger)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(utils.sys.excepthook is not None)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils.os.closed_fds, utils.sys.stdio_fds)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(
isinstance(utils.sys.stdout, utils.LoggerFileObject))
self.assertTrue(
isinstance(utils.sys.stderr, utils.LoggerFileObject))
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
# reset; test same args, but exc when trying to close stdio
utils.os = MockOs(raise_funcs=('dup2',))
utils.sys = MockSys()
# test unable to close stdio
utils.capture_stdio(logger)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(utils.sys.excepthook is not None)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils.os.closed_fds, [])
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(
isinstance(utils.sys.stdout, utils.LoggerFileObject))
self.assertTrue(
isinstance(utils.sys.stderr, utils.LoggerFileObject))
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
# reset; test some other args
utils.os = MockOs()
utils.sys = MockSys()
fixed lockups and lost messages in swift-init and capture_stdio
2011-03-30 15:04:15 -05:00
logger = utils.get_logger(None, log_to_console=True)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
# test console log
utils.capture_stdio(logger, capture_stdout=False,
capture_stderr=False)
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(utils.sys.excepthook is not None)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
# when logging to console, stderr remains open
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils.os.closed_fds, utils.sys.stdio_fds[:2])
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
reset_loggers()
# stdio not captured
fixed lockups and lost messages in swift-init and capture_stdio
2011-03-30 15:04:15 -05:00
self.assertFalse(isinstance(utils.sys.stdout,
utils.LoggerFileObject))
self.assertFalse(isinstance(utils.sys.stderr,
utils.LoggerFileObject))
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
finally:
utils.sys = _orig_sys
utils.os = _orig_os
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
Fix logger test coupling Two tests fail if test/unit/common/test_daemon.py is excluded from the test set: test.unit.common.test_utils.TestUtils.test_swift_log_formatter test.unit.common.test_utils.TestStatsdLoggingDelegation.test_thread_locals They fail because they assert that logger.txn_id is None, when previous tests cause txn_id to be set. This coupling is masked when test_daemon.py is included because it reloads the common/utils module, and is executed just before test.unit.common.test_utils. The coupling can be observed by running, for example: nosetests ./test/unit/common/middleware/test_except.py ./test/unit/common/test_utils.py or nosetests ./test/unit/proxy ./test/unit/common/test_utils.py The failing tests should reset logger thread_locals before making assertions. test_utils.reset_loggers() attempts this but is broken because it sets thread_local on the wrong object. Changes in this patch: - fix reset_loggers() to reset the LogAdapter thread local state - add a reset_logger_state decorator to call reset_loggers before and after a decorated method - use the decorator for increased isolation of tests that previously called reset_loggers only on exit Change-Id: If9aa781a2dd1929a47ef69322ec8c53263d47660
2015-02-11 17:49:41 +00:00
@reset_logger_state
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
def test_get_logger_console(self):
logger = utils.get_logger(None)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
console_handlers = [h for h in logger.logger.handlers if
isinstance(h, logging.StreamHandler)]
self.assertFalse(console_handlers)
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
logger = utils.get_logger(None, log_to_console=True)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
console_handlers = [h for h in logger.logger.handlers if
isinstance(h, logging.StreamHandler)]
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(console_handlers)
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
# make sure you can't have two console handlers
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(console_handlers), 1)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
old_handler = console_handlers[0]
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
logger = utils.get_logger(None, log_to_console=True)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
console_handlers = [h for h in logger.logger.handlers if
isinstance(h, logging.StreamHandler)]
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(console_handlers), 1)
logging refactor to support proxy access logs New log level "notice" set to python log level 25 maps to syslog priority LOG_NOTICE. Used for some messages in the proxy server, but will be available to all apps using the LogAdapter returned from get_logger. Cleaned up some code in get_logger so that console logging works with log_routes and removed some unneeded bits. NamedFormatter functionality was split between LogAdapter (which now inherits from logging.LoggerAdapter) and TxnFormatter (which now is only responsible for adding the log records txn_id). The proxy server app now configures a separate logger for access line logging. By default it will use the same settings as the regular proxy logger.
2011-02-10 14:59:52 -06:00
new_handler = console_handlers[0]
pep8 fix: assertNotEquals -> assertNotEqual assertNotEquals is deprecated in py3 Change-Id: Ib611351987bed1199fb8f73a750955a61d022d0a
2015-08-31 21:49:49 +05:30
self.assertNotEqual(new_handler, old_handler)
added helper/util to parse command line args; removed some duplicated code in server/daemon bin scripts; more standized python/linux daemonization procedures; fixed lp:666957 "devauth server creates auth.db with the wrong privileges"; new run_daemon helper based on run_wsgi simplifies daemon launching/testing; new - all servers/daemons support verbose option when started interactivlty which will log to the console; fixed lp:667839 "can't start servers with relative paths to configs"; added tests
2010-11-11 16:41:07 -06:00
Mock out time and sleep to avoid races We mock out time.time(), time.sleep() and eventlet.sleep() so that we avoid test problems caused by exceedingly long delays during the execution of the test. We also make sure to convert the units used in the tests to milliseconds for a bit more clarity. Closes bug: 1298154 Change-Id: I803d06cbf205a02a4f7bb1e0c467d276632cd6a3
2014-03-26 22:38:03 -04:00
def verify_under_pseudo_time(
self, func, target_runtime_ms=1, *args, **kwargs):
curr_time = [42.0]
def my_time():
curr_time[0] += 0.001
return curr_time[0]
def my_sleep(duration):
curr_time[0] += 0.001
curr_time[0] += duration
Get rid of contextlib.nested() for py3 contextlib.nested() is missing completely in Python 3. Since 2.7, we can use multiple context managers in a 'with' statement, like so: with thing1() as t1, thing2() as t2: do_stuff() Now, if we had some code that needed to nest an arbitrary number of context managers, there's stuff we could do with contextlib.ExitStack and such... but we don't. We only use contextlib.nested() in tests to set up bunches of mocks without crazy-deep indentation, and all that stuff fits perfectly into multiple-context-manager 'with' statements. Change-Id: Id472958b007948f05dbd4c7fb8cf3ffab58e2681
2015-10-21 13:05:37 -07:00
with patch('time.time', my_time), \
patch('time.sleep', my_sleep), \
patch('eventlet.sleep', my_sleep):
Mock out time and sleep to avoid races We mock out time.time(), time.sleep() and eventlet.sleep() so that we avoid test problems caused by exceedingly long delays during the execution of the test. We also make sure to convert the units used in the tests to milliseconds for a bit more clarity. Closes bug: 1298154 Change-Id: I803d06cbf205a02a4f7bb1e0c467d276632cd6a3
2014-03-26 22:38:03 -04:00
start = time.time()
func(*args, **kwargs)
# make sure it's accurate to 10th of a second, converting the time
# difference to milliseconds, 100 milliseconds is 1/10 of a second
diff_from_target_ms = abs(
target_runtime_ms - ((time.time() - start) * 1000))
self.assertTrue(diff_from_target_ms < 100,
"Expected %d < 100" % diff_from_target_ms)
adding in rate limiting and unit tests
2010-12-16 16:20:57 -08:00
def test_ratelimit_sleep(self):
Mock out time and sleep to avoid races We mock out time.time(), time.sleep() and eventlet.sleep() so that we avoid test problems caused by exceedingly long delays during the execution of the test. We also make sure to convert the units used in the tests to milliseconds for a bit more clarity. Closes bug: 1298154 Change-Id: I803d06cbf205a02a4f7bb1e0c467d276632cd6a3
2014-03-26 22:38:03 -04:00
def testfunc():
running_time = 0
for i in range(100):
running_time = utils.ratelimit_sleep(running_time, -5)
self.verify_under_pseudo_time(testfunc, target_runtime_ms=1)
def testfunc():
running_time = 0
for i in range(100):
running_time = utils.ratelimit_sleep(running_time, 0)
self.verify_under_pseudo_time(testfunc, target_runtime_ms=1)
def testfunc():
running_time = 0
for i in range(50):
running_time = utils.ratelimit_sleep(running_time, 200)
self.verify_under_pseudo_time(testfunc, target_runtime_ms=250)
adding in rate limiting and unit tests
2010-12-16 16:20:57 -08:00
def test_ratelimit_sleep_with_incr(self):
Mock out time and sleep to avoid races We mock out time.time(), time.sleep() and eventlet.sleep() so that we avoid test problems caused by exceedingly long delays during the execution of the test. We also make sure to convert the units used in the tests to milliseconds for a bit more clarity. Closes bug: 1298154 Change-Id: I803d06cbf205a02a4f7bb1e0c467d276632cd6a3
2014-03-26 22:38:03 -04:00
def testfunc():
running_time = 0
vals = [5, 17, 0, 3, 11, 30,
40, 4, 13, 2, -1] * 2 # adds up to 248
total = 0
for i in vals:
running_time = utils.ratelimit_sleep(running_time,
500, incr_by=i)
total += i
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(248, total)
Mock out time and sleep to avoid races We mock out time.time(), time.sleep() and eventlet.sleep() so that we avoid test problems caused by exceedingly long delays during the execution of the test. We also make sure to convert the units used in the tests to milliseconds for a bit more clarity. Closes bug: 1298154 Change-Id: I803d06cbf205a02a4f7bb1e0c467d276632cd6a3
2014-03-26 22:38:03 -04:00
self.verify_under_pseudo_time(testfunc, target_runtime_ms=500)
adding in rate limiting and unit tests
2010-12-16 16:20:57 -08:00
Move like unit tests together; add comments Some simple code movement to move the utils.ratelimit_sleep() unit tests together so that they can be viewed all at once. We also add some comments to document the behavior of utils.ratelimit_sleep(); small modification to max_rate parameter checking to match intended use. Change-Id: I3b11acfb6634d16a4b3594dba8dbc7a2d3ee8d1a
2014-03-26 22:10:29 -04:00
def test_ratelimit_sleep_with_sleep(self):
Mock out time and sleep to avoid races We mock out time.time(), time.sleep() and eventlet.sleep() so that we avoid test problems caused by exceedingly long delays during the execution of the test. We also make sure to convert the units used in the tests to milliseconds for a bit more clarity. Closes bug: 1298154 Change-Id: I803d06cbf205a02a4f7bb1e0c467d276632cd6a3
2014-03-26 22:38:03 -04:00
def testfunc():
running_time = 0
sleeps = [0] * 7 + [.2] * 3 + [0] * 30
for i in sleeps:
running_time = utils.ratelimit_sleep(running_time, 40,
rate_buffer=1)
time.sleep(i)
self.verify_under_pseudo_time(testfunc, target_runtime_ms=900)
Move like unit tests together; add comments Some simple code movement to move the utils.ratelimit_sleep() unit tests together so that they can be viewed all at once. We also add some comments to document the behavior of utils.ratelimit_sleep(); small modification to max_rate parameter checking to match intended use. Change-Id: I3b11acfb6634d16a4b3594dba8dbc7a2d3ee8d1a
2014-03-26 22:10:29 -04:00
ipv6 support
2011-01-16 09:52:08 +00:00
def test_urlparse(self):
parsed = utils.urlparse('http://127.0.0.1/')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(parsed.scheme, 'http')
self.assertEqual(parsed.hostname, '127.0.0.1')
self.assertEqual(parsed.path, '/')
ipv6 support
2011-01-16 09:52:08 +00:00
parsed = utils.urlparse('http://127.0.0.1:8080/')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(parsed.port, 8080)
ipv6 support
2011-01-16 09:52:08 +00:00
parsed = utils.urlparse('https://127.0.0.1/')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(parsed.scheme, 'https')
ipv6 support
2011-01-16 09:52:08 +00:00
parsed = utils.urlparse('http://[::1]/')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(parsed.hostname, '::1')
ipv6 support
2011-01-16 09:52:08 +00:00
parsed = utils.urlparse('http://[::1]:8080/')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(parsed.hostname, '::1')
self.assertEqual(parsed.port, 8080)
ipv6 support
2011-01-16 09:52:08 +00:00
parsed = utils.urlparse('www.example.com')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(parsed.hostname, '')
adding in rate limiting and unit tests
2010-12-16 16:20:57 -08:00
Prepare for trunk merge. Refactored some of the swift_init classes into a new module in swift.common, changed some names. Removed the bin test stuff. Fixed some bugs, added some features.
2011-02-11 13:18:19 -06:00
def test_search_tree(self):
# file match & ext miss
with temptree(['asdf.conf', 'blarg.conf', 'asdf.cfg']) as t:
asdf = utils.search_tree(t, 'a*', '.conf')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(asdf), 1)
self.assertEqual(asdf[0],
os.path.join(t, 'asdf.conf'))
Prepare for trunk merge. Refactored some of the swift_init classes into a new module in swift.common, changed some names. Removed the bin test stuff. Fixed some bugs, added some features.
2011-02-11 13:18:19 -06:00
# multi-file match & glob miss & sort
with temptree(['application.bin', 'apple.bin', 'apropos.bin']) as t:
app_bins = utils.search_tree(t, 'app*', 'bin')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(app_bins), 2)
self.assertEqual(app_bins[0],
os.path.join(t, 'apple.bin'))
self.assertEqual(app_bins[1],
os.path.join(t, 'application.bin'))
Prepare for trunk merge. Refactored some of the swift_init classes into a new module in swift.common, changed some names. Removed the bin test stuff. Fixed some bugs, added some features.
2011-02-11 13:18:19 -06:00
# test file in folder & ext miss & glob miss
files = (
'sub/file1.ini',
'sub/file2.conf',
'sub.bin',
'bus.ini',
'bus/file3.ini',
)
with temptree(files) as t:
sub_ini = utils.search_tree(t, 'sub*', '.ini')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(sub_ini), 1)
self.assertEqual(sub_ini[0],
os.path.join(t, 'sub/file1.ini'))
Prepare for trunk merge. Refactored some of the swift_init classes into a new module in swift.common, changed some names. Removed the bin test stuff. Fixed some bugs, added some features.
2011-02-11 13:18:19 -06:00
# test multi-file in folder & sub-folder & ext miss & glob miss
files = (
'folder_file.txt',
'folder/1.txt',
'folder/sub/2.txt',
'folder2/3.txt',
'Folder3/4.txt'
'folder.rc',
)
with temptree(files) as t:
folder_texts = utils.search_tree(t, 'folder*', '.txt')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(folder_texts), 4)
Prepare for trunk merge. Refactored some of the swift_init classes into a new module in swift.common, changed some names. Removed the bin test stuff. Fixed some bugs, added some features.
2011-02-11 13:18:19 -06:00
f1 = os.path.join(t, 'folder_file.txt')
f2 = os.path.join(t, 'folder/1.txt')
f3 = os.path.join(t, 'folder/sub/2.txt')
f4 = os.path.join(t, 'folder2/3.txt')
for f in [f1, f2, f3, f4]:
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(f in folder_texts)
Prepare for trunk merge. Refactored some of the swift_init classes into a new module in swift.common, changed some names. Removed the bin test stuff. Fixed some bugs, added some features.
2011-02-11 13:18:19 -06:00
conf.d support Allow Swift daemons and servers to optionally accept a directory as the configuration parameter. Directory based configuration leverages ConfigParser's native multi-file support. Files ending in '.conf' in the given directory are parsed in lexicographical order. Filenames starting with '.' are ignored. A mixture of file and directory configuration paths is not supported - if the configuration path is a file behavior is unchanged. * update swift-init to search for conf.d paths when building servers (e.g. /etc/swift/proxy-server.conf.d/) * new script swift-config can be used to inspect the cumulative configuration * pull a little bit of code out of run_wsgi and test separately * fix example config bug for the proxy servers client_disconnect option * added section on directory based configuration to deployment guide DocImpact Implements: blueprint confd Change-Id: I89b0f48e538117f28590cf6698401f74ef58003b
2013-03-25 16:34:43 -07:00
def test_search_tree_with_directory_ext_match(self):
files = (
'object-server/object-server.conf-base',
'object-server/1.conf.d/base.conf',
'object-server/1.conf.d/1.conf',
'object-server/2.conf.d/base.conf',
'object-server/2.conf.d/2.conf',
'object-server/3.conf.d/base.conf',
'object-server/3.conf.d/3.conf',
'object-server/4.conf.d/base.conf',
'object-server/4.conf.d/4.conf',
)
with temptree(files) as t:
conf_dirs = utils.search_tree(t, 'object-server', '.conf',
dir_ext='conf.d')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(conf_dirs), 4)
conf.d support Allow Swift daemons and servers to optionally accept a directory as the configuration parameter. Directory based configuration leverages ConfigParser's native multi-file support. Files ending in '.conf' in the given directory are parsed in lexicographical order. Filenames starting with '.' are ignored. A mixture of file and directory configuration paths is not supported - if the configuration path is a file behavior is unchanged. * update swift-init to search for conf.d paths when building servers (e.g. /etc/swift/proxy-server.conf.d/) * new script swift-config can be used to inspect the cumulative configuration * pull a little bit of code out of run_wsgi and test separately * fix example config bug for the proxy servers client_disconnect option * added section on directory based configuration to deployment guide DocImpact Implements: blueprint confd Change-Id: I89b0f48e538117f28590cf6698401f74ef58003b
2013-03-25 16:34:43 -07:00
for i in range(4):
conf_dir = os.path.join(t, 'object-server/%d.conf.d' % (i + 1))
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(conf_dir in conf_dirs)
conf.d support Allow Swift daemons and servers to optionally accept a directory as the configuration parameter. Directory based configuration leverages ConfigParser's native multi-file support. Files ending in '.conf' in the given directory are parsed in lexicographical order. Filenames starting with '.' are ignored. A mixture of file and directory configuration paths is not supported - if the configuration path is a file behavior is unchanged. * update swift-init to search for conf.d paths when building servers (e.g. /etc/swift/proxy-server.conf.d/) * new script swift-config can be used to inspect the cumulative configuration * pull a little bit of code out of run_wsgi and test separately * fix example config bug for the proxy servers client_disconnect option * added section on directory based configuration to deployment guide DocImpact Implements: blueprint confd Change-Id: I89b0f48e538117f28590cf6698401f74ef58003b
2013-03-25 16:34:43 -07:00
Add targeted config loading to swift-init This allows an easier and more explicit way to tell swift-init to run on specific servers. For example with an SAIO, this allows you to do something like: swift-init object-server.1 reload to reload just the 1st object server. A more real world example is when you are running separate servers for replication. In this example you might have an object-server/public.conf and object-server/replication.conf. With this change you can do something like: swift-init object-server.replication reload to just reload the replication server. DocImpact Change-Id: I5c6046b5ee28e17dadfc5fc53d1d872d9bb8fe48
2014-05-08 22:52:15 +00:00
def test_search_tree_conf_dir_with_named_conf_match(self):
files = (
'proxy-server/proxy-server.conf.d/base.conf',
'proxy-server/proxy-server.conf.d/pipeline.conf',
'proxy-server/proxy-noauth.conf.d/base.conf',
'proxy-server/proxy-noauth.conf.d/pipeline.conf',
)
with temptree(files) as t:
conf_dirs = utils.search_tree(t, 'proxy-server', 'noauth.conf',
dir_ext='noauth.conf.d')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(conf_dirs), 1)
Add targeted config loading to swift-init This allows an easier and more explicit way to tell swift-init to run on specific servers. For example with an SAIO, this allows you to do something like: swift-init object-server.1 reload to reload just the 1st object server. A more real world example is when you are running separate servers for replication. In this example you might have an object-server/public.conf and object-server/replication.conf. With this change you can do something like: swift-init object-server.replication reload to just reload the replication server. DocImpact Change-Id: I5c6046b5ee28e17dadfc5fc53d1d872d9bb8fe48
2014-05-08 22:52:15 +00:00
conf_dir = conf_dirs[0]
expected = os.path.join(t, 'proxy-server/proxy-noauth.conf.d')
self.assertEqual(conf_dir, expected)
def test_search_tree_conf_dir_pid_with_named_conf_match(self):
files = (
'proxy-server/proxy-server.pid.d',
'proxy-server/proxy-noauth.pid.d',
)
with temptree(files) as t:
pid_files = utils.search_tree(t, 'proxy-server',
exts=['noauth.pid', 'noauth.pid.d'])
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(pid_files), 1)
Add targeted config loading to swift-init This allows an easier and more explicit way to tell swift-init to run on specific servers. For example with an SAIO, this allows you to do something like: swift-init object-server.1 reload to reload just the 1st object server. A more real world example is when you are running separate servers for replication. In this example you might have an object-server/public.conf and object-server/replication.conf. With this change you can do something like: swift-init object-server.replication reload to just reload the replication server. DocImpact Change-Id: I5c6046b5ee28e17dadfc5fc53d1d872d9bb8fe48
2014-05-08 22:52:15 +00:00
pid_file = pid_files[0]
expected = os.path.join(t, 'proxy-server/proxy-noauth.pid.d')
self.assertEqual(pid_file, expected)
Prepare for trunk merge. Refactored some of the swift_init classes into a new module in swift.common, changed some names. Removed the bin test stuff. Fixed some bugs, added some features.
2011-02-11 13:18:19 -06:00
def test_write_file(self):
with temptree([]) as t:
file_name = os.path.join(t, 'test')
utils.write_file(file_name, 'test')
with open(file_name, 'r') as f:
contents = f.read()
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(contents, 'test')
Prepare for trunk merge. Refactored some of the swift_init classes into a new module in swift.common, changed some names. Removed the bin test stuff. Fixed some bugs, added some features.
2011-02-11 13:18:19 -06:00
# and also subdirs
file_name = os.path.join(t, 'subdir/test2')
utils.write_file(file_name, 'test2')
with open(file_name, 'r') as f:
contents = f.read()
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(contents, 'test2')
Prepare for trunk merge. Refactored some of the swift_init classes into a new module in swift.common, changed some names. Removed the bin test stuff. Fixed some bugs, added some features.
2011-02-11 13:18:19 -06:00
# but can't over-write files
file_name = os.path.join(t, 'subdir/test2/test3')
self.assertRaises(IOError, utils.write_file, file_name,
'test3')
def test_remove_file(self):
with temptree([]) as t:
file_name = os.path.join(t, 'blah.pid')
# assert no raise
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(os.path.exists(file_name), False)
self.assertEqual(utils.remove_file(file_name), None)
Prepare for trunk merge. Refactored some of the swift_init classes into a new module in swift.common, changed some names. Removed the bin test stuff. Fixed some bugs, added some features.
2011-02-11 13:18:19 -06:00
with open(file_name, 'w') as f:
f.write('1')
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(os.path.exists(file_name))
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils.remove_file(file_name), None)
Prepare for trunk merge. Refactored some of the swift_init classes into a new module in swift.common, changed some names. Removed the bin test stuff. Fixed some bugs, added some features.
2011-02-11 13:18:19 -06:00
self.assertFalse(os.path.exists(file_name))
Tests; bug fixes
2011-03-24 07:46:02 +00:00
def test_human_readable(self):
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils.human_readable(0), '0')
self.assertEqual(utils.human_readable(1), '1')
self.assertEqual(utils.human_readable(10), '10')
self.assertEqual(utils.human_readable(100), '100')
self.assertEqual(utils.human_readable(999), '999')
self.assertEqual(utils.human_readable(1024), '1Ki')
self.assertEqual(utils.human_readable(1535), '1Ki')
self.assertEqual(utils.human_readable(1536), '2Ki')
self.assertEqual(utils.human_readable(1047552), '1023Ki')
self.assertEqual(utils.human_readable(1048063), '1023Ki')
self.assertEqual(utils.human_readable(1048064), '1Mi')
self.assertEqual(utils.human_readable(1048576), '1Mi')
self.assertEqual(utils.human_readable(1073741824), '1Gi')
self.assertEqual(utils.human_readable(1099511627776), '1Ti')
self.assertEqual(utils.human_readable(1125899906842624), '1Pi')
self.assertEqual(utils.human_readable(1152921504606846976), '1Ei')
self.assertEqual(utils.human_readable(1180591620717411303424), '1Zi')
self.assertEqual(utils.human_readable(1208925819614629174706176),
'1Yi')
self.assertEqual(utils.human_readable(1237940039285380274899124224),
'1024Yi')
Tests; bug fixes
2011-03-24 07:46:02 +00:00
Merged from trunk
2011-05-11 02:51:31 +00:00
def test_validate_sync_to(self):
New container sync configuration option Summary of the new configuration option: The cluster operators add the container_sync middleware to their proxy pipeline and create a container-sync-realms.conf for their cluster and copy this out to all their proxy and container servers. This file specifies the available container sync "realms". A container sync realm is a group of clusters with a shared key that have agreed to provide container syncing to one another. The end user can then set the X-Container-Sync-To value on a container to //realm/cluster/account/container instead of the previously required URL. The allowed hosts list is not used with this configuration and instead every container sync request sent is signed using the realm key and user key. This offers better security as source hosts can be faked much more easily than faking per request signatures. Replaying signed requests, assuming it could easily be done, shouldn't be an issue as the X-Timestamp is part of the signature and so would just short-circuit as already current or as superceded. This also makes configuration easier for the end user, especially with difficult networking situations where a different host might need to be used for the container sync daemon since it's connecting from within a cluster. With this new configuration option, the end user just specifies the realm and cluster names and that is resolved to the proper endpoint configured by the operator. If the operator changes their configuration (key or endpoint), the end user does not need to change theirs. DocImpact Change-Id: Ie1704990b66d0434e4991e26ed1da8b08cb05a37
2013-12-08 09:13:59 +00:00
fname = 'container-sync-realms.conf'
fcontents = '''
[US]
key = 9ff3b71c849749dbaec4ccdd3cbab62b
cluster_dfw1 = http://dfw1.host/v1/
'''
with temptree([fname], [fcontents]) as tempdir:
logger = FakeLogger()
fpath = os.path.join(tempdir, fname)
csr = ContainerSyncRealms(fpath, logger)
for realms_conf in (None, csr):
for goodurl, result in (
('http://1.1.1.1/v1/a/c',
(None, 'http://1.1.1.1/v1/a/c', None, None)),
('http://1.1.1.1:8080/a/c',
(None, 'http://1.1.1.1:8080/a/c', None, None)),
('http://2.2.2.2/a/c',
(None, 'http://2.2.2.2/a/c', None, None)),
('https://1.1.1.1/v1/a/c',
(None, 'https://1.1.1.1/v1/a/c', None, None)),
('//US/DFW1/a/c',
(None, 'http://dfw1.host/v1/a/c', 'US',
'9ff3b71c849749dbaec4ccdd3cbab62b')),
('//us/DFW1/a/c',
(None, 'http://dfw1.host/v1/a/c', 'US',
'9ff3b71c849749dbaec4ccdd3cbab62b')),
('//us/dfw1/a/c',
(None, 'http://dfw1.host/v1/a/c', 'US',
'9ff3b71c849749dbaec4ccdd3cbab62b')),
('//',
(None, None, None, None)),
('',
(None, None, None, None))):
if goodurl.startswith('//') and not realms_conf:
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
New container sync configuration option Summary of the new configuration option: The cluster operators add the container_sync middleware to their proxy pipeline and create a container-sync-realms.conf for their cluster and copy this out to all their proxy and container servers. This file specifies the available container sync "realms". A container sync realm is a group of clusters with a shared key that have agreed to provide container syncing to one another. The end user can then set the X-Container-Sync-To value on a container to //realm/cluster/account/container instead of the previously required URL. The allowed hosts list is not used with this configuration and instead every container sync request sent is signed using the realm key and user key. This offers better security as source hosts can be faked much more easily than faking per request signatures. Replaying signed requests, assuming it could easily be done, shouldn't be an issue as the X-Timestamp is part of the signature and so would just short-circuit as already current or as superceded. This also makes configuration easier for the end user, especially with difficult networking situations where a different host might need to be used for the container sync daemon since it's connecting from within a cluster. With this new configuration option, the end user just specifies the realm and cluster names and that is resolved to the proper endpoint configured by the operator. If the operator changes their configuration (key or endpoint), the end user does not need to change theirs. DocImpact Change-Id: Ie1704990b66d0434e4991e26ed1da8b08cb05a37
2013-12-08 09:13:59 +00:00
utils.validate_sync_to(
goodurl, ['1.1.1.1', '2.2.2.2'], realms_conf),
(None, None, None, None))
else:
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
New container sync configuration option Summary of the new configuration option: The cluster operators add the container_sync middleware to their proxy pipeline and create a container-sync-realms.conf for their cluster and copy this out to all their proxy and container servers. This file specifies the available container sync "realms". A container sync realm is a group of clusters with a shared key that have agreed to provide container syncing to one another. The end user can then set the X-Container-Sync-To value on a container to //realm/cluster/account/container instead of the previously required URL. The allowed hosts list is not used with this configuration and instead every container sync request sent is signed using the realm key and user key. This offers better security as source hosts can be faked much more easily than faking per request signatures. Replaying signed requests, assuming it could easily be done, shouldn't be an issue as the X-Timestamp is part of the signature and so would just short-circuit as already current or as superceded. This also makes configuration easier for the end user, especially with difficult networking situations where a different host might need to be used for the container sync daemon since it's connecting from within a cluster. With this new configuration option, the end user just specifies the realm and cluster names and that is resolved to the proper endpoint configured by the operator. If the operator changes their configuration (key or endpoint), the end user does not need to change theirs. DocImpact Change-Id: Ie1704990b66d0434e4991e26ed1da8b08cb05a37
2013-12-08 09:13:59 +00:00
utils.validate_sync_to(
goodurl, ['1.1.1.1', '2.2.2.2'], realms_conf),
result)
for badurl, result in (
('http://1.1.1.1',
('Path required in X-Container-Sync-To', None, None,
None)),
('httpq://1.1.1.1/v1/a/c',
('Invalid scheme \'httpq\' in X-Container-Sync-To, '
'must be "//", "http", or "https".', None, None,
None)),
('http://1.1.1.1/v1/a/c?query',
('Params, queries, and fragments not allowed in '
'X-Container-Sync-To', None, None, None)),
('http://1.1.1.1/v1/a/c#frag',
('Params, queries, and fragments not allowed in '
'X-Container-Sync-To', None, None, None)),
('http://1.1.1.1/v1/a/c?query#frag',
('Params, queries, and fragments not allowed in '
'X-Container-Sync-To', None, None, None)),
('http://1.1.1.1/v1/a/c?query=param',
('Params, queries, and fragments not allowed in '
'X-Container-Sync-To', None, None, None)),
('http://1.1.1.1/v1/a/c?query=param#frag',
('Params, queries, and fragments not allowed in '
'X-Container-Sync-To', None, None, None)),
('http://1.1.1.2/v1/a/c',
("Invalid host '1.1.1.2' in X-Container-Sync-To",
None, None, None)),
('//us/invalid/a/c',
("No cluster endpoint for 'us' 'invalid'", None,
None, None)),
('//invalid/dfw1/a/c',
("No realm key for 'invalid'", None, None, None)),
('//us/invalid1/a/',
("Invalid X-Container-Sync-To format "
"'//us/invalid1/a/'", None, None, None)),
('//us/invalid1/a',
("Invalid X-Container-Sync-To format "
"'//us/invalid1/a'", None, None, None)),
('//us/invalid1/',
("Invalid X-Container-Sync-To format "
"'//us/invalid1/'", None, None, None)),
('//us/invalid1',
("Invalid X-Container-Sync-To format "
"'//us/invalid1'", None, None, None)),
('//us/',
("Invalid X-Container-Sync-To format "
"'//us/'", None, None, None)),
('//us',
("Invalid X-Container-Sync-To format "
"'//us'", None, None, None))):
if badurl.startswith('//') and not realms_conf:
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
New container sync configuration option Summary of the new configuration option: The cluster operators add the container_sync middleware to their proxy pipeline and create a container-sync-realms.conf for their cluster and copy this out to all their proxy and container servers. This file specifies the available container sync "realms". A container sync realm is a group of clusters with a shared key that have agreed to provide container syncing to one another. The end user can then set the X-Container-Sync-To value on a container to //realm/cluster/account/container instead of the previously required URL. The allowed hosts list is not used with this configuration and instead every container sync request sent is signed using the realm key and user key. This offers better security as source hosts can be faked much more easily than faking per request signatures. Replaying signed requests, assuming it could easily be done, shouldn't be an issue as the X-Timestamp is part of the signature and so would just short-circuit as already current or as superceded. This also makes configuration easier for the end user, especially with difficult networking situations where a different host might need to be used for the container sync daemon since it's connecting from within a cluster. With this new configuration option, the end user just specifies the realm and cluster names and that is resolved to the proper endpoint configured by the operator. If the operator changes their configuration (key or endpoint), the end user does not need to change theirs. DocImpact Change-Id: Ie1704990b66d0434e4991e26ed1da8b08cb05a37
2013-12-08 09:13:59 +00:00
utils.validate_sync_to(
badurl, ['1.1.1.1', '2.2.2.2'], realms_conf),
(None, None, None, None))
else:
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
New container sync configuration option Summary of the new configuration option: The cluster operators add the container_sync middleware to their proxy pipeline and create a container-sync-realms.conf for their cluster and copy this out to all their proxy and container servers. This file specifies the available container sync "realms". A container sync realm is a group of clusters with a shared key that have agreed to provide container syncing to one another. The end user can then set the X-Container-Sync-To value on a container to //realm/cluster/account/container instead of the previously required URL. The allowed hosts list is not used with this configuration and instead every container sync request sent is signed using the realm key and user key. This offers better security as source hosts can be faked much more easily than faking per request signatures. Replaying signed requests, assuming it could easily be done, shouldn't be an issue as the X-Timestamp is part of the signature and so would just short-circuit as already current or as superceded. This also makes configuration easier for the end user, especially with difficult networking situations where a different host might need to be used for the container sync daemon since it's connecting from within a cluster. With this new configuration option, the end user just specifies the realm and cluster names and that is resolved to the proper endpoint configured by the operator. If the operator changes their configuration (key or endpoint), the end user does not need to change theirs. DocImpact Change-Id: Ie1704990b66d0434e4991e26ed1da8b08cb05a37
2013-12-08 09:13:59 +00:00
utils.validate_sync_to(
badurl, ['1.1.1.1', '2.2.2.2'], realms_conf),
result)
Merged from trunk
2011-05-11 02:51:31 +00:00
Adding some tests
2011-06-10 15:55:25 +00:00
def test_TRUE_VALUES(self):
for v in utils.TRUE_VALUES:
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(v, v.lower())
Adding some tests
2011-06-10 15:55:25 +00:00
changed TRUE_VALUES references to utils.config_true_value() call cleaned up pep8 (v1.3.3) in all files this patch touches Change-Id: I30e8314dfdc23fb70ab83741a548db9905dfccff
2012-10-19 13:50:57 -07:00
def test_config_true_value(self):
orig_trues = utils.TRUE_VALUES
try:
utils.TRUE_VALUES = 'hello world'.split()
for val in 'hello world HELLO WORLD'.split():
self.assertTrue(utils.config_true_value(val) is True)
self.assertTrue(utils.config_true_value(True) is True)
self.assertTrue(utils.config_true_value('foo') is False)
self.assertTrue(utils.config_true_value(False) is False)
finally:
utils.TRUE_VALUES = orig_trues
Set default wsgi workers to cpu_count Change the default value of wsgi workers from 1 to auto. The new default value for workers in the proxy, container, account & object wsgi servers will spawn as many workers per process as you have cpu cores. This will not be ideal for some configurations, but it's much more likely to produce a successful out of the box deployment. Inspect the number of cpu_cores using python's multiprocessing when available. Multiprocessing was added in python 2.6, but I know I've compiled python without it before on accident. The cpu_count method seems to be pretty system agnostic, but it says it can raise NotImplementedError or sometimes return 0. Add a new utility method 'config_auto_int_value' to pull an integer out of the config which has a dynamic default. * drive by s/container/proxy/ in proxy-server.conf.5 * fix misplaced max_clients in *-server.conf-sample * update doc/development_saio to force workers = 1 DocImpact Change-Id: Ifa563d22952c902ab8cbe1d339ba385413c54e95
2013-07-11 17:00:57 -07:00
def test_config_auto_int_value(self):
expectations = {
# (value, default) : expected,
('1', 0): 1,
(1, 0): 1,
('asdf', 0): ValueError,
('auto', 1): 1,
('AutO', 1): 1,
('Aut0', 1): ValueError,
(None, 1): 1,
}
for (value, default), expected in expectations.items():
try:
rv = utils.config_auto_int_value(value, default)
Use Python 3.x compatible except construct except x,y: was deprected and is removed in Python 3.x. Use "except x as y:" instead which works in any Python version >= 2.6. Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
2013-08-28 21:16:08 +02:00
except Exception as e:
Set default wsgi workers to cpu_count Change the default value of wsgi workers from 1 to auto. The new default value for workers in the proxy, container, account & object wsgi servers will spawn as many workers per process as you have cpu cores. This will not be ideal for some configurations, but it's much more likely to produce a successful out of the box deployment. Inspect the number of cpu_cores using python's multiprocessing when available. Multiprocessing was added in python 2.6, but I know I've compiled python without it before on accident. The cpu_count method seems to be pretty system agnostic, but it says it can raise NotImplementedError or sometimes return 0. Add a new utility method 'config_auto_int_value' to pull an integer out of the config which has a dynamic default. * drive by s/container/proxy/ in proxy-server.conf.5 * fix misplaced max_clients in *-server.conf-sample * update doc/development_saio to force workers = 1 DocImpact Change-Id: Ifa563d22952c902ab8cbe1d339ba385413c54e95
2013-07-11 17:00:57 -07:00
if e.__class__ is not expected:
raise
else:
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(expected, rv)
Set default wsgi workers to cpu_count Change the default value of wsgi workers from 1 to auto. The new default value for workers in the proxy, container, account & object wsgi servers will spawn as many workers per process as you have cpu cores. This will not be ideal for some configurations, but it's much more likely to produce a successful out of the box deployment. Inspect the number of cpu_cores using python's multiprocessing when available. Multiprocessing was added in python 2.6, but I know I've compiled python without it before on accident. The cpu_count method seems to be pretty system agnostic, but it says it can raise NotImplementedError or sometimes return 0. Add a new utility method 'config_auto_int_value' to pull an integer out of the config which has a dynamic default. * drive by s/container/proxy/ in proxy-server.conf.5 * fix misplaced max_clients in *-server.conf-sample * update doc/development_saio to force workers = 1 DocImpact Change-Id: Ifa563d22952c902ab8cbe1d339ba385413c54e95
2013-07-11 17:00:57 -07:00
fixed pep8 issue and renamed strcmp_const_time to streq_const_time Change-Id: Ifc21f0ec08ac2ba120a67795e82082b18e08584e
2012-02-28 21:38:30 -06:00
def test_streq_const_time(self):
self.assertTrue(utils.streq_const_time('abc123', 'abc123'))
self.assertFalse(utils.streq_const_time('a', 'aaaaa'))
self.assertFalse(utils.streq_const_time('ABC123', 'abc123'))
Use constant time string comparisons for auth. Fix bug 942644. Use constant time string comparisons when doing authentication to help guard against timing attacks. Change-Id: I88c4c5cd9edd9e5d60db07b6ae2638b74a2a2e17
2012-02-28 11:17:21 -05:00
Foundational support for PUT and GET of erasure-coded objects This commit makes it possible to PUT an object into Swift and have it stored using erasure coding instead of replication, and also to GET the object back from Swift at a later time. This works by splitting the incoming object into a number of segments, erasure-coding each segment in turn to get fragments, then concatenating the fragments into fragment archives. Segments are 1 MiB in size, except the last, which is between 1 B and 1 MiB. +====================================================================+ | object data | +====================================================================+ | +------------------------+----------------------+ | | | v v v +===================+ +===================+ +==============+ | segment 1 | | segment 2 | ... | segment N | +===================+ +===================+ +==============+ | | | | v v /=========\ /=========\ | pyeclib | | pyeclib | ... \=========/ \=========/ | | | | +--> fragment A-1 +--> fragment A-2 | | | | | | | | | | +--> fragment B-1 +--> fragment B-2 | | | | ... ... Then, object server A gets the concatenation of fragment A-1, A-2, ..., A-N, so its .data file looks like this (called a "fragment archive"): +=====================================================================+ | fragment A-1 | fragment A-2 | ... | fragment A-N | +=====================================================================+ Since this means that the object server never sees the object data as the client sent it, we have to do a few things to ensure data integrity. First, the proxy has to check the Etag if the client provided it; the object server can't do it since the object server doesn't see the raw data. Second, if the client does not provide an Etag, the proxy computes it and uses the MIME-PUT mechanism to provide it to the object servers after the object body. Otherwise, the object would not have an Etag at all. Third, the proxy computes the MD5 of each fragment archive and sends it to the object server using the MIME-PUT mechanism. With replicated objects, the proxy checks that the Etags from all the object servers match, and if they don't, returns a 500 to the client. This mitigates the risk of data corruption in one of the proxy --> object connections, and signals to the client when it happens. With EC objects, we can't use that same mechanism, so we must send the checksum with each fragment archive to get comparable protection. On the GET path, the inverse happens: the proxy connects to a bunch of object servers (M of them, for an M+K scheme), reads one fragment at a time from each fragment archive, decodes those fragments into a segment, and serves the segment to the client. When an object server dies partway through a GET response, any partially-fetched fragment is discarded, the resumption point is wound back to the nearest fragment boundary, and the GET is retried with the next object server. GET requests for a single byterange work; GET requests for multiple byteranges do not. There are a number of things _not_ included in this commit. Some of them are listed here: * multi-range GET * deferred cleanup of old .data files * durability (daemon to reconstruct missing archives) Co-Authored-By: Alistair Coles <alistair.coles@hp.com> Co-Authored-By: Thiago da Silva <thiago@redhat.com> Co-Authored-By: John Dickinson <me@not.mn> Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com> Co-Authored-By: Paul Luse <paul.e.luse@intel.com> Co-Authored-By: Christian Schwede <christian.schwede@enovance.com> Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com> Change-Id: I9c13c03616489f8eab7dcd7c5f21237ed4cb6fd2
2014-10-22 13:18:34 -07:00
def test_replication_quorum_size(self):
Extract quorum-size calculation to utility method. The behavior's the same as before; this just makes the code (IMHO) a little easier to read. Change-Id: Ie54d836d81af7410057219f60b72b840c9ce51b9
2013-06-28 11:54:35 -07:00
expected_sizes = {1: 1,
2: 2,
3: 2,
4: 3,
5: 3}
Foundational support for PUT and GET of erasure-coded objects This commit makes it possible to PUT an object into Swift and have it stored using erasure coding instead of replication, and also to GET the object back from Swift at a later time. This works by splitting the incoming object into a number of segments, erasure-coding each segment in turn to get fragments, then concatenating the fragments into fragment archives. Segments are 1 MiB in size, except the last, which is between 1 B and 1 MiB. +====================================================================+ | object data | +====================================================================+ | +------------------------+----------------------+ | | | v v v +===================+ +===================+ +==============+ | segment 1 | | segment 2 | ... | segment N | +===================+ +===================+ +==============+ | | | | v v /=========\ /=========\ | pyeclib | | pyeclib | ... \=========/ \=========/ | | | | +--> fragment A-1 +--> fragment A-2 | | | | | | | | | | +--> fragment B-1 +--> fragment B-2 | | | | ... ... Then, object server A gets the concatenation of fragment A-1, A-2, ..., A-N, so its .data file looks like this (called a "fragment archive"): +=====================================================================+ | fragment A-1 | fragment A-2 | ... | fragment A-N | +=====================================================================+ Since this means that the object server never sees the object data as the client sent it, we have to do a few things to ensure data integrity. First, the proxy has to check the Etag if the client provided it; the object server can't do it since the object server doesn't see the raw data. Second, if the client does not provide an Etag, the proxy computes it and uses the MIME-PUT mechanism to provide it to the object servers after the object body. Otherwise, the object would not have an Etag at all. Third, the proxy computes the MD5 of each fragment archive and sends it to the object server using the MIME-PUT mechanism. With replicated objects, the proxy checks that the Etags from all the object servers match, and if they don't, returns a 500 to the client. This mitigates the risk of data corruption in one of the proxy --> object connections, and signals to the client when it happens. With EC objects, we can't use that same mechanism, so we must send the checksum with each fragment archive to get comparable protection. On the GET path, the inverse happens: the proxy connects to a bunch of object servers (M of them, for an M+K scheme), reads one fragment at a time from each fragment archive, decodes those fragments into a segment, and serves the segment to the client. When an object server dies partway through a GET response, any partially-fetched fragment is discarded, the resumption point is wound back to the nearest fragment boundary, and the GET is retried with the next object server. GET requests for a single byterange work; GET requests for multiple byteranges do not. There are a number of things _not_ included in this commit. Some of them are listed here: * multi-range GET * deferred cleanup of old .data files * durability (daemon to reconstruct missing archives) Co-Authored-By: Alistair Coles <alistair.coles@hp.com> Co-Authored-By: Thiago da Silva <thiago@redhat.com> Co-Authored-By: John Dickinson <me@not.mn> Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com> Co-Authored-By: Paul Luse <paul.e.luse@intel.com> Co-Authored-By: Christian Schwede <christian.schwede@enovance.com> Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com> Change-Id: I9c13c03616489f8eab7dcd7c5f21237ed4cb6fd2
2014-10-22 13:18:34 -07:00
got_sizes = dict([(n, utils.quorum_size(n))
for n in expected_sizes])
Extract quorum-size calculation to utility method. The behavior's the same as before; this just makes the code (IMHO) a little easier to read. Change-Id: Ie54d836d81af7410057219f60b72b840c9ce51b9
2013-06-28 11:54:35 -07:00
self.assertEqual(expected_sizes, got_sizes)
fix object replication on older rsync versions when using ipv4 Fixes bug 987388 Change-Id: I6eb5c45fe1f5844ad853a4ff9bc8fd23cc9abd5d
2012-06-04 13:27:39 +02:00
def test_rsync_ip_ipv4_localhost(self):
self.assertEqual(utils.rsync_ip('127.0.0.1'), '127.0.0.1')
def test_rsync_ip_ipv6_random_ip(self):
self.assertEqual(
utils.rsync_ip('fe80:0000:0000:0000:0202:b3ff:fe1e:8329'),
'[fe80:0000:0000:0000:0202:b3ff:fe1e:8329]')
def test_rsync_ip_ipv6_ipv4_compatible(self):
self.assertEqual(
utils.rsync_ip('::ffff:192.0.2.128'), '[::ffff:192.0.2.128]')
Allows to configure the rsync modules where the replicators will send data Currently, the rsync module where the replicators send data is static. It forbids administrators to set rsync configuration based on their current deployment or needs. As an example, the rsyncd configuration example encourages to set a connections limit for the modules account, container and object. It permits to protect devices from excessives parallels connections, because it would impact performances. On a server with many devices, it is tempting to increase this number proportionally, but nothing guarantees that the distribution of the connections will be balanced. In the worst scenario, a single device can receive all the connections, which is a severe impact on performances. This commit adds a new option named 'rsync_module' to the *-replicator sections of the *-server configuration file. This configuration variable can be extrapolated with device attributes like ip, port, device, zone, ... by using the format {NAME}. eg: rsync_module = {replication_ip}::object_{device} With this configuration, an administrators can solve the problem of connections distribution by creating one module per device in rsyncd configuration. The default values are backward compatible: {replication_ip}::account {replication_ip}::container {replication_ip}::object Option vm_test_mode is deprecated by this commit, but backward compatibility is maintained. The option is only effective when rsync_module is not set. In that case, {replication_port} is appended to the default value of rsync_module. Change-Id: Iad91df50dadbe96c921181797799b4444323ce2e
2015-06-16 12:47:26 +02:00
def test_rsync_module_interpolation(self):
fake_device = {'ip': '127.0.0.1', 'port': 11,
'replication_ip': '127.0.0.2', 'replication_port': 12,
'region': '1', 'zone': '2', 'device': 'sda1',
'meta': 'just_a_string'}
self.assertEqual(
utils.rsync_module_interpolation('{ip}', fake_device),
'127.0.0.1')
self.assertEqual(
utils.rsync_module_interpolation('{port}', fake_device),
'11')
self.assertEqual(
utils.rsync_module_interpolation('{replication_ip}', fake_device),
'127.0.0.2')
self.assertEqual(
utils.rsync_module_interpolation('{replication_port}',
fake_device),
'12')
self.assertEqual(
utils.rsync_module_interpolation('{region}', fake_device),
'1')
self.assertEqual(
utils.rsync_module_interpolation('{zone}', fake_device),
'2')
self.assertEqual(
utils.rsync_module_interpolation('{device}', fake_device),
'sda1')
self.assertEqual(
utils.rsync_module_interpolation('{meta}', fake_device),
'just_a_string')
self.assertEqual(
utils.rsync_module_interpolation('{replication_ip}::object',
fake_device),
'127.0.0.2::object')
self.assertEqual(
utils.rsync_module_interpolation('{ip}::container{port}',
fake_device),
'127.0.0.1::container11')
self.assertEqual(
utils.rsync_module_interpolation(
'{replication_ip}::object_{device}', fake_device),
'127.0.0.2::object_sda1')
self.assertEqual(
utils.rsync_module_interpolation(
'127.0.0.3::object_{replication_port}', fake_device),
'127.0.0.3::object_12')
self.assertRaises(ValueError, utils.rsync_module_interpolation,
'{replication_ip}::object_{deivce}', fake_device)
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
def test_fallocate_reserve(self):
class StatVFS(object):
f_frsize = 1024
f_bavail = 1
def fstatvfs(fd):
return StatVFS()
orig_FALLOCATE_RESERVE = utils.FALLOCATE_RESERVE
orig_fstatvfs = utils.os.fstatvfs
try:
fallocate = utils.FallocateWrapper(noop=True)
utils.os.fstatvfs = fstatvfs
# Want 1023 reserved, have 1024 * 1 free, so succeeds
utils.FALLOCATE_RESERVE = 1023
StatVFS.f_frsize = 1024
StatVFS.f_bavail = 1
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fallocate(0, 1, 0, ctypes.c_uint64(0)), 0)
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
# Want 1023 reserved, have 512 * 2 free, so succeeds
utils.FALLOCATE_RESERVE = 1023
StatVFS.f_frsize = 512
StatVFS.f_bavail = 2
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fallocate(0, 1, 0, ctypes.c_uint64(0)), 0)
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
# Want 1024 reserved, have 1024 * 1 free, so fails
utils.FALLOCATE_RESERVE = 1024
StatVFS.f_frsize = 1024
StatVFS.f_bavail = 1
exc = None
try:
fallocate(0, 1, 0, ctypes.c_uint64(0))
Use Python 3.x compatible except construct except x,y: was deprected and is removed in Python 3.x. Use "except x as y:" instead which works in any Python version >= 2.6. Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
2013-08-28 21:16:08 +02:00
except OSError as err:
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
exc = err
fix fallocate_reserve traceback Previously, fallocate_reserve could result in a traceback. The OSError being raised didn't have the proper errno set. This patch sets the errno to ENOSPC. Change-Id: I017b0584972ca8832f3b160bbcdff335ae9a1aa6
2016-04-13 11:57:45 -05:00
self.assertEqual(str(exc),
'[Errno 28] FALLOCATE_RESERVE fail 1024 <= 1024')
self.assertEqual(err.errno, errno.ENOSPC)
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
# Want 1024 reserved, have 512 * 2 free, so fails
utils.FALLOCATE_RESERVE = 1024
StatVFS.f_frsize = 512
StatVFS.f_bavail = 2
exc = None
try:
fallocate(0, 1, 0, ctypes.c_uint64(0))
Use Python 3.x compatible except construct except x,y: was deprected and is removed in Python 3.x. Use "except x as y:" instead which works in any Python version >= 2.6. Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
2013-08-28 21:16:08 +02:00
except OSError as err:
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
exc = err
fix fallocate_reserve traceback Previously, fallocate_reserve could result in a traceback. The OSError being raised didn't have the proper errno set. This patch sets the errno to ENOSPC. Change-Id: I017b0584972ca8832f3b160bbcdff335ae9a1aa6
2016-04-13 11:57:45 -05:00
self.assertEqual(str(exc),
'[Errno 28] FALLOCATE_RESERVE fail 1024 <= 1024')
self.assertEqual(err.errno, errno.ENOSPC)
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
# Want 2048 reserved, have 1024 * 1 free, so fails
utils.FALLOCATE_RESERVE = 2048
StatVFS.f_frsize = 1024
StatVFS.f_bavail = 1
exc = None
try:
fallocate(0, 1, 0, ctypes.c_uint64(0))
Use Python 3.x compatible except construct except x,y: was deprected and is removed in Python 3.x. Use "except x as y:" instead which works in any Python version >= 2.6. Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
2013-08-28 21:16:08 +02:00
except OSError as err:
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
exc = err
fix fallocate_reserve traceback Previously, fallocate_reserve could result in a traceback. The OSError being raised didn't have the proper errno set. This patch sets the errno to ENOSPC. Change-Id: I017b0584972ca8832f3b160bbcdff335ae9a1aa6
2016-04-13 11:57:45 -05:00
self.assertEqual(str(exc),
'[Errno 28] FALLOCATE_RESERVE fail 1024 <= 2048')
self.assertEqual(err.errno, errno.ENOSPC)
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
# Want 2048 reserved, have 512 * 2 free, so fails
utils.FALLOCATE_RESERVE = 2048
StatVFS.f_frsize = 512
StatVFS.f_bavail = 2
exc = None
try:
fallocate(0, 1, 0, ctypes.c_uint64(0))
Use Python 3.x compatible except construct except x,y: was deprected and is removed in Python 3.x. Use "except x as y:" instead which works in any Python version >= 2.6. Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
2013-08-28 21:16:08 +02:00
except OSError as err:
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
exc = err
fix fallocate_reserve traceback Previously, fallocate_reserve could result in a traceback. The OSError being raised didn't have the proper errno set. This patch sets the errno to ENOSPC. Change-Id: I017b0584972ca8832f3b160bbcdff335ae9a1aa6
2016-04-13 11:57:45 -05:00
self.assertEqual(str(exc),
'[Errno 28] FALLOCATE_RESERVE fail 1024 <= 2048')
self.assertEqual(err.errno, errno.ENOSPC)
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
# Want 1023 reserved, have 1024 * 1 free, but file size is 1, so
# fails
utils.FALLOCATE_RESERVE = 1023
StatVFS.f_frsize = 1024
StatVFS.f_bavail = 1
exc = None
try:
fallocate(0, 1, 0, ctypes.c_uint64(1))
Use Python 3.x compatible except construct except x,y: was deprected and is removed in Python 3.x. Use "except x as y:" instead which works in any Python version >= 2.6. Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
2013-08-28 21:16:08 +02:00
except OSError as err:
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
exc = err
fix fallocate_reserve traceback Previously, fallocate_reserve could result in a traceback. The OSError being raised didn't have the proper errno set. This patch sets the errno to ENOSPC. Change-Id: I017b0584972ca8832f3b160bbcdff335ae9a1aa6
2016-04-13 11:57:45 -05:00
self.assertEqual(str(exc),
'[Errno 28] FALLOCATE_RESERVE fail 1023 <= 1023')
self.assertEqual(err.errno, errno.ENOSPC)
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
# Want 1022 reserved, have 1024 * 1 free, and file size is 1, so
# succeeds
utils.FALLOCATE_RESERVE = 1022
StatVFS.f_frsize = 1024
StatVFS.f_bavail = 1
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fallocate(0, 1, 0, ctypes.c_uint64(1)), 0)
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
# Want 1023 reserved, have 1024 * 1 free, and file size is 0, so
# succeeds
utils.FALLOCATE_RESERVE = 1023
StatVFS.f_frsize = 1024
StatVFS.f_bavail = 1
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fallocate(0, 1, 0, ctypes.c_uint64(0)), 0)
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
# Want 1024 reserved, have 1024 * 1 free, and even though
# file size is 0, since we're under the reserve, fails
utils.FALLOCATE_RESERVE = 1024
StatVFS.f_frsize = 1024
StatVFS.f_bavail = 1
exc = None
try:
fallocate(0, 1, 0, ctypes.c_uint64(0))
Use Python 3.x compatible except construct except x,y: was deprected and is removed in Python 3.x. Use "except x as y:" instead which works in any Python version >= 2.6. Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
2013-08-28 21:16:08 +02:00
except OSError as err:
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
exc = err
fix fallocate_reserve traceback Previously, fallocate_reserve could result in a traceback. The OSError being raised didn't have the proper errno set. This patch sets the errno to ENOSPC. Change-Id: I017b0584972ca8832f3b160bbcdff335ae9a1aa6
2016-04-13 11:57:45 -05:00
self.assertEqual(str(exc),
'[Errno 28] FALLOCATE_RESERVE fail 1024 <= 1024')
self.assertEqual(err.errno, errno.ENOSPC)
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
finally:
utils.FALLOCATE_RESERVE = orig_FALLOCATE_RESERVE
utils.os.fstatvfs = orig_fstatvfs
def test_fallocate_func(self):
class FallocateWrapper(object):
def __init__(self):
self.last_call = None
def __call__(self, *args):
self.last_call = list(args)
self.last_call[-1] = self.last_call[-1].value
return 0
orig__sys_fallocate = utils._sys_fallocate
try:
utils._sys_fallocate = FallocateWrapper()
# Ensure fallocate calls _sys_fallocate even with 0 bytes
utils._sys_fallocate.last_call = None
utils.fallocate(1234, 0)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils._sys_fallocate.last_call,
[1234, 1, 0, 0])
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
# Ensure fallocate calls _sys_fallocate even with negative bytes
utils._sys_fallocate.last_call = None
utils.fallocate(1234, -5678)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils._sys_fallocate.last_call,
[1234, 1, 0, 0])
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
# Ensure fallocate calls _sys_fallocate properly with positive
# bytes
utils._sys_fallocate.last_call = None
utils.fallocate(1234, 1)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils._sys_fallocate.last_call,
[1234, 1, 0, 1])
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
utils._sys_fallocate.last_call = None
utils.fallocate(1234, 10 * 1024 * 1024 * 1024)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils._sys_fallocate.last_call,
[1234, 1, 0, 10 * 1024 * 1024 * 1024])
Added fallocate_reserve option Some systems behave badly when they completely run out of space. To alleviate this problem, you can set the fallocate_reserve conf value to a number of bytes to "reserve" on each disk. When the disk free space falls at or below this amount, fallocate calls will fail, even if the underlying OS fallocate call would succeed. For example, a fallocate_reserve of 5368709120 (5G) would make all fallocate calls fail, even for zero-byte files, when the disk free space falls under 5G. The default fallocate_reserve is 0, meaning "no reserve", and so the software behaves exactly as it always has unless you set this conf value to something non-zero. Also fixed ring builder's search_devs doc bugs. Related: To get rsync to do the same, see https://github.com/rackspace/cloudfiles-rsync Specifically, see this patch: https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff DocImpact Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-25 02:11:19 +00:00
finally:
utils._sys_fallocate = orig__sys_fallocate
Included time inside trans_id Including the time inside the trans_id can be very useful for knowing which logs to scan. I made this so the trans_id will still be the same length (the randomness of the remaining uuid4 should be enough for this use). I also added a convenience function for retreiving the time information from a trans_id. If you're wondering why I just didn't use uuid1 that embeds the time, it's because it also embeds uuid.getnode() which "The first time this runs, it may launch a separate program, which could be quite slow." We could supply our own getnode value, but then we have to guarantee its uniqueness, yada yada yada. Change-Id: Ie33caf1e839fd1a21b01a928a8b301126bef7396
2013-04-20 03:34:48 +00:00
def test_generate_trans_id(self):
fake_time = 1366428370.5163341
with patch.object(utils.time, 'time', return_value=fake_time):
trans_id = utils.generate_trans_id('')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(trans_id), 34)
self.assertEqual(trans_id[:2], 'tx')
self.assertEqual(trans_id[23], '-')
self.assertEqual(int(trans_id[24:], 16), int(fake_time))
Included time inside trans_id Including the time inside the trans_id can be very useful for knowing which logs to scan. I made this so the trans_id will still be the same length (the randomness of the remaining uuid4 should be enough for this use). I also added a convenience function for retreiving the time information from a trans_id. If you're wondering why I just didn't use uuid1 that embeds the time, it's because it also embeds uuid.getnode() which "The first time this runs, it may launch a separate program, which could be quite slow." We could supply our own getnode value, but then we have to guarantee its uniqueness, yada yada yada. Change-Id: Ie33caf1e839fd1a21b01a928a8b301126bef7396
2013-04-20 03:34:48 +00:00
with patch.object(utils.time, 'time', return_value=fake_time):
trans_id = utils.generate_trans_id('-suffix')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(trans_id), 41)
self.assertEqual(trans_id[:2], 'tx')
self.assertEqual(trans_id[34:], '-suffix')
self.assertEqual(trans_id[23], '-')
self.assertEqual(int(trans_id[24:34], 16), int(fake_time))
Included time inside trans_id Including the time inside the trans_id can be very useful for knowing which logs to scan. I made this so the trans_id will still be the same length (the randomness of the remaining uuid4 should be enough for this use). I also added a convenience function for retreiving the time information from a trans_id. If you're wondering why I just didn't use uuid1 that embeds the time, it's because it also embeds uuid.getnode() which "The first time this runs, it may launch a separate program, which could be quite slow." We could supply our own getnode value, but then we have to guarantee its uniqueness, yada yada yada. Change-Id: Ie33caf1e839fd1a21b01a928a8b301126bef7396
2013-04-20 03:34:48 +00:00
def test_get_trans_id_time(self):
ts = utils.get_trans_id_time('tx8c8bc884cdaf499bb29429aa9c46946e')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(ts, None)
Included time inside trans_id Including the time inside the trans_id can be very useful for knowing which logs to scan. I made this so the trans_id will still be the same length (the randomness of the remaining uuid4 should be enough for this use). I also added a convenience function for retreiving the time information from a trans_id. If you're wondering why I just didn't use uuid1 that embeds the time, it's because it also embeds uuid.getnode() which "The first time this runs, it may launch a separate program, which could be quite slow." We could supply our own getnode value, but then we have to guarantee its uniqueness, yada yada yada. Change-Id: Ie33caf1e839fd1a21b01a928a8b301126bef7396
2013-04-20 03:34:48 +00:00
ts = utils.get_trans_id_time('tx1df4ff4f55ea45f7b2ec2-0051720c06')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(ts, 1366428678)
self.assertEqual(
Included time inside trans_id Including the time inside the trans_id can be very useful for knowing which logs to scan. I made this so the trans_id will still be the same length (the randomness of the remaining uuid4 should be enough for this use). I also added a convenience function for retreiving the time information from a trans_id. If you're wondering why I just didn't use uuid1 that embeds the time, it's because it also embeds uuid.getnode() which "The first time this runs, it may launch a separate program, which could be quite slow." We could supply our own getnode value, but then we have to guarantee its uniqueness, yada yada yada. Change-Id: Ie33caf1e839fd1a21b01a928a8b301126bef7396
2013-04-20 03:34:48 +00:00
time.asctime(time.gmtime(ts)) + ' UTC',
'Sat Apr 20 03:31:18 2013 UTC')
ts = utils.get_trans_id_time(
'tx1df4ff4f55ea45f7b2ec2-0051720c06-suffix')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(ts, 1366428678)
self.assertEqual(
Included time inside trans_id Including the time inside the trans_id can be very useful for knowing which logs to scan. I made this so the trans_id will still be the same length (the randomness of the remaining uuid4 should be enough for this use). I also added a convenience function for retreiving the time information from a trans_id. If you're wondering why I just didn't use uuid1 that embeds the time, it's because it also embeds uuid.getnode() which "The first time this runs, it may launch a separate program, which could be quite slow." We could supply our own getnode value, but then we have to guarantee its uniqueness, yada yada yada. Change-Id: Ie33caf1e839fd1a21b01a928a8b301126bef7396
2013-04-20 03:34:48 +00:00
time.asctime(time.gmtime(ts)) + ' UTC',
'Sat Apr 20 03:31:18 2013 UTC')
ts = utils.get_trans_id_time('')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(ts, None)
Included time inside trans_id Including the time inside the trans_id can be very useful for knowing which logs to scan. I made this so the trans_id will still be the same length (the randomness of the remaining uuid4 should be enough for this use). I also added a convenience function for retreiving the time information from a trans_id. If you're wondering why I just didn't use uuid1 that embeds the time, it's because it also embeds uuid.getnode() which "The first time this runs, it may launch a separate program, which could be quite slow." We could supply our own getnode value, but then we have to guarantee its uniqueness, yada yada yada. Change-Id: Ie33caf1e839fd1a21b01a928a8b301126bef7396
2013-04-20 03:34:48 +00:00
ts = utils.get_trans_id_time('garbage')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(ts, None)
Included time inside trans_id Including the time inside the trans_id can be very useful for knowing which logs to scan. I made this so the trans_id will still be the same length (the randomness of the remaining uuid4 should be enough for this use). I also added a convenience function for retreiving the time information from a trans_id. If you're wondering why I just didn't use uuid1 that embeds the time, it's because it also embeds uuid.getnode() which "The first time this runs, it may launch a separate program, which could be quite slow." We could supply our own getnode value, but then we have to guarantee its uniqueness, yada yada yada. Change-Id: Ie33caf1e839fd1a21b01a928a8b301126bef7396
2013-04-20 03:34:48 +00:00
ts = utils.get_trans_id_time('tx1df4ff4f55ea45f7b2ec2-almostright')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(ts, None)
Included time inside trans_id Including the time inside the trans_id can be very useful for knowing which logs to scan. I made this so the trans_id will still be the same length (the randomness of the remaining uuid4 should be enough for this use). I also added a convenience function for retreiving the time information from a trans_id. If you're wondering why I just didn't use uuid1 that embeds the time, it's because it also embeds uuid.getnode() which "The first time this runs, it may launch a separate program, which could be quite slow." We could supply our own getnode value, but then we have to guarantee its uniqueness, yada yada yada. Change-Id: Ie33caf1e839fd1a21b01a928a8b301126bef7396
2013-04-20 03:34:48 +00:00
Moved some code out of swift.obj.replicator This will be needed in future replication work to avoid circular imports. I used swift.obj.base as the module name just because we seemed to avoid putting code in __init__.py files so far and I didn't want to buck the trend. I would love to see other obj things like *_metadata and DiskFile move into swift.obj.base as well and swift.obj.server just be the WSGI server logic, but I'll leave that for the future. I have changed the tests as little as possible (just the references to where they get the code to test) to show the refactor has not broken anything. I did add a test for tpool_reraise since there was none before. There will be a follow on patch for moving the tests to their new location(s). I figured I'd wait to put the bikes in the shed until everyone's done painting it. Change-Id: I32b4ac88be21eb76c877d3f4cc1e6ac33304835b
2013-05-11 19:51:03 +00:00
def test_tpool_reraise(self):
with patch.object(utils.tpool, 'execute', lambda f: f()):
self.assertTrue(
utils.tpool_reraise(MagicMock(return_value='test1')), 'test1')
Pep8 unit test modules w/ <= 10 violations (5 of 12) Change-Id: I8e82c14ada52d44df5a31e08982ac79cd7e5c969 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-08-31 23:13:15 -04:00
self.assertRaises(
Exception,
Moved some code out of swift.obj.replicator This will be needed in future replication work to avoid circular imports. I used swift.obj.base as the module name just because we seemed to avoid putting code in __init__.py files so far and I didn't want to buck the trend. I would love to see other obj things like *_metadata and DiskFile move into swift.obj.base as well and swift.obj.server just be the WSGI server logic, but I'll leave that for the future. I have changed the tests as little as possible (just the references to where they get the code to test) to show the refactor has not broken anything. I did add a test for tpool_reraise since there was none before. There will be a follow on patch for moving the tests to their new location(s). I figured I'd wait to put the bikes in the shed until everyone's done painting it. Change-Id: I32b4ac88be21eb76c877d3f4cc1e6ac33304835b
2013-05-11 19:51:03 +00:00
utils.tpool_reraise, MagicMock(side_effect=Exception('test2')))
Pep8 unit test modules w/ <= 10 violations (5 of 12) Change-Id: I8e82c14ada52d44df5a31e08982ac79cd7e5c969 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-08-31 23:13:15 -04:00
self.assertRaises(
BaseException,
Moved some code out of swift.obj.replicator This will be needed in future replication work to avoid circular imports. I used swift.obj.base as the module name just because we seemed to avoid putting code in __init__.py files so far and I didn't want to buck the trend. I would love to see other obj things like *_metadata and DiskFile move into swift.obj.base as well and swift.obj.server just be the WSGI server logic, but I'll leave that for the future. I have changed the tests as little as possible (just the references to where they get the code to test) to show the refactor has not broken anything. I did add a test for tpool_reraise since there was none before. There will be a follow on patch for moving the tests to their new location(s). I figured I'd wait to put the bikes in the shed until everyone's done painting it. Change-Id: I32b4ac88be21eb76c877d3f4cc1e6ac33304835b
2013-05-11 19:51:03 +00:00
utils.tpool_reraise,
MagicMock(side_effect=BaseException('test3')))
Add test for swift.common.utils.lock_file Change-Id: Id03f130ad7016b11b740050b7aadf879773d5a5f
2013-06-21 11:48:05 -04:00
def test_lock_file(self):
flags = os.O_CREAT | os.O_RDWR
with NamedTemporaryFile(delete=False) as nt:
nt.write("test string")
nt.flush()
nt.close()
with utils.lock_file(nt.name, unlink=False) as f:
self.assertEqual(f.read(), "test string")
# we have a lock, now let's try to get a newer one
fd = os.open(nt.name, flags)
self.assertRaises(IOError, fcntl.flock, fd,
fcntl.LOCK_EX | fcntl.LOCK_NB)
with utils.lock_file(nt.name, unlink=False, append=True) as f:
Fix Python 3 issues in utils * LogAdapter.exception(): on Python 3, socket.error is an alias to OSError, so group both exceptions to support Python 3. * ThreadPool: call GreenPipe() with indexed parameter, don't pass the third parameter by its name, since the parameter name changed between Python 2 (bufsize) and Python 3 (buffering). * strip_value() in test_utils: StringIO.getvalue() now requires seek(0), otherwise the buffer is filled with null characters. * test_lock_file(): on Python 3, seek(0) is now required to go the beginning of a file opened in append mode. In append mode, Python goes to the end of the file. Change-Id: I4e56a51690f016a0a2e1354380ce11cff1891f64
2015-10-19 17:16:21 +02:00
f.seek(0)
Add test for swift.common.utils.lock_file Change-Id: Id03f130ad7016b11b740050b7aadf879773d5a5f
2013-06-21 11:48:05 -04:00
self.assertEqual(f.read(), "test string")
f.seek(0)
f.write("\nanother string")
f.flush()
f.seek(0)
self.assertEqual(f.read(), "test string\nanother string")
# we have a lock, now let's try to get a newer one
fd = os.open(nt.name, flags)
self.assertRaises(IOError, fcntl.flock, fd,
fcntl.LOCK_EX | fcntl.LOCK_NB)
with utils.lock_file(nt.name, timeout=3, unlink=False) as f:
try:
Pep8 unit test modules w/ <= 10 violations (5 of 12) Change-Id: I8e82c14ada52d44df5a31e08982ac79cd7e5c969 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-08-31 23:13:15 -04:00
with utils.lock_file(
nt.name, timeout=1, unlink=False) as f:
self.assertTrue(
False, "Expected LockTimeout exception")
Add test for swift.common.utils.lock_file Change-Id: Id03f130ad7016b11b740050b7aadf879773d5a5f
2013-06-21 11:48:05 -04:00
except LockTimeout:
pass
with utils.lock_file(nt.name, unlink=True) as f:
self.assertEqual(f.read(), "test string\nanother string")
# we have a lock, now let's try to get a newer one
fd = os.open(nt.name, flags)
Pep8 unit test modules w/ <= 10 violations (5 of 12) Change-Id: I8e82c14ada52d44df5a31e08982ac79cd7e5c969 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-08-31 23:13:15 -04:00
self.assertRaises(
IOError, fcntl.flock, fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
Add test for swift.common.utils.lock_file Change-Id: Id03f130ad7016b11b740050b7aadf879773d5a5f
2013-06-21 11:48:05 -04:00
self.assertRaises(OSError, os.remove, nt.name)
lock_file race fixes I attempted to use this function and found a few problems. We shouldn’t unlink the file after closing it, because someone else could lock it in between. Switch to unlink before close. If someone else locked the file between our open and flock, they are likely to unlink it out from underneath us. Then we have a lock on a file that no longer exists. So stat the filename after locking to make sure the inode hasn't changed or gone away. We probably shouldn’t unlink the file if we time out waiting for a lock. So move that to before the finally block. Change-Id: Id1858c97805d3ab81c584eaee8ce0d43d34a8089
2014-07-21 07:45:56 +00:00
def test_lock_file_unlinked_after_open(self):
os_open = os.open
first_pass = [True]
def deleting_open(filename, flags):
# unlink the file after it's opened. once.
fd = os_open(filename, flags)
if first_pass[0]:
os.unlink(filename)
first_pass[0] = False
return fd
with NamedTemporaryFile(delete=False) as nt:
with mock.patch('os.open', deleting_open):
with utils.lock_file(nt.name, unlink=True) as f:
self.assertNotEqual(os.fstat(nt.fileno()).st_ino,
os.fstat(f.fileno()).st_ino)
first_pass = [True]
def recreating_open(filename, flags):
# unlink and recreate the file after it's opened
fd = os_open(filename, flags)
if first_pass[0]:
os.unlink(filename)
os.close(os_open(filename, os.O_CREAT | os.O_RDWR))
first_pass[0] = False
return fd
with NamedTemporaryFile(delete=False) as nt:
with mock.patch('os.open', recreating_open):
with utils.lock_file(nt.name, unlink=True) as f:
self.assertNotEqual(os.fstat(nt.fileno()).st_ino,
os.fstat(f.fileno()).st_ino)
def test_lock_file_held_on_unlink(self):
os_unlink = os.unlink
def flocking_unlink(filename):
# make sure the lock is held when we unlink
fd = os.open(filename, os.O_RDWR)
self.assertRaises(
IOError, fcntl.flock, fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
os.close(fd)
os_unlink(filename)
with NamedTemporaryFile(delete=False) as nt:
with mock.patch('os.unlink', flocking_unlink):
with utils.lock_file(nt.name, unlink=True):
pass
def test_lock_file_no_unlink_if_fail(self):
os_open = os.open
with NamedTemporaryFile(delete=True) as nt:
def lock_on_open(filename, flags):
# lock the file on another fd after it's opened.
fd = os_open(filename, flags)
fd2 = os_open(filename, flags)
fcntl.flock(fd2, fcntl.LOCK_EX | fcntl.LOCK_NB)
return fd
try:
timedout = False
with mock.patch('os.open', lock_on_open):
with utils.lock_file(nt.name, unlink=False, timeout=0.01):
pass
except LockTimeout:
timedout = True
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(timedout)
self.assertTrue(os.path.exists(nt.name))
lock_file race fixes I attempted to use this function and found a few problems. We shouldn’t unlink the file after closing it, because someone else could lock it in between. Switch to unlink before close. If someone else locked the file between our open and flock, they are likely to unlink it out from underneath us. Then we have a lock on a file that no longer exists. So stat the filename after locking to make sure the inode hasn't changed or gone away. We probably shouldn’t unlink the file if we time out waiting for a lock. So move that to before the finally block. Change-Id: Id1858c97805d3ab81c584eaee8ce0d43d34a8089
2014-07-21 07:45:56 +00:00
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
def test_ismount_path_does_not_exist(self):
tmpdir = mkdtemp()
try:
use testCase.assert instead of assert and try...except In unit tests, we should use: self.assertTrue or self.assertFalse instead of 'assert xxx is True/False', and use self.assertRaises instead of 'try ... except ...' fixes bug #1212602 Change-Id: I25318eb59cd26a98afdce5495273e3b3266655a4
2013-08-15 17:05:07 +08:00
self.assertFalse(utils.ismount(os.path.join(tmpdir, 'bar')))
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
finally:
shutil.rmtree(tmpdir)
def test_ismount_path_not_mount(self):
tmpdir = mkdtemp()
try:
use testCase.assert instead of assert and try...except In unit tests, we should use: self.assertTrue or self.assertFalse instead of 'assert xxx is True/False', and use self.assertRaises instead of 'try ... except ...' fixes bug #1212602 Change-Id: I25318eb59cd26a98afdce5495273e3b3266655a4
2013-08-15 17:05:07 +08:00
self.assertFalse(utils.ismount(tmpdir))
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
finally:
shutil.rmtree(tmpdir)
def test_ismount_path_error(self):
def _mock_os_lstat(path):
raise OSError(13, "foo")
tmpdir = mkdtemp()
try:
with patch("os.lstat", _mock_os_lstat):
ismount should not raise exceptions The changes from using os.path.ismount to using swift.common.utils.ismount has caused problems since the new one raises exceptions in cases where the old one did not. Daemons have been encountering this and exiting; servers have been 500ing instead of 507ing in this case, changing handoff behaviors, etc. Since the new one was specifically written and tested for this new behavior, I left that original function as ismount_raw and made ismount do what it did before. If there really isn't some reason for this new behavior, I'll be glad to get rid of ismount_raw and just keep ismount. I couldn't see any reason for the new behavior myself. Change-Id: I2b5b17f9ed9656cd8804a5ed568170697d0b183d
2014-01-04 06:22:46 +00:00
# Raises exception with _raw -- see next test.
utils.ismount(tmpdir)
finally:
shutil.rmtree(tmpdir)
def test_ismount_raw_path_error(self):
def _mock_os_lstat(path):
raise OSError(13, "foo")
tmpdir = mkdtemp()
try:
with patch("os.lstat", _mock_os_lstat):
self.assertRaises(OSError, utils.ismount_raw, tmpdir)
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
finally:
shutil.rmtree(tmpdir)
def test_ismount_path_is_symlink(self):
tmpdir = mkdtemp()
try:
link = os.path.join(tmpdir, "tmp")
os.symlink("/tmp", link)
use testCase.assert instead of assert and try...except In unit tests, we should use: self.assertTrue or self.assertFalse instead of 'assert xxx is True/False', and use self.assertRaises instead of 'try ... except ...' fixes bug #1212602 Change-Id: I25318eb59cd26a98afdce5495273e3b3266655a4
2013-08-15 17:05:07 +08:00
self.assertFalse(utils.ismount(link))
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
finally:
shutil.rmtree(tmpdir)
def test_ismount_path_is_root(self):
use testCase.assert instead of assert and try...except In unit tests, we should use: self.assertTrue or self.assertFalse instead of 'assert xxx is True/False', and use self.assertRaises instead of 'try ... except ...' fixes bug #1212602 Change-Id: I25318eb59cd26a98afdce5495273e3b3266655a4
2013-08-15 17:05:07 +08:00
self.assertTrue(utils.ismount('/'))
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
def test_ismount_parent_path_error(self):
_os_lstat = os.lstat
def _mock_os_lstat(path):
if path.endswith(".."):
raise OSError(13, "foo")
else:
return _os_lstat(path)
tmpdir = mkdtemp()
try:
with patch("os.lstat", _mock_os_lstat):
ismount should not raise exceptions The changes from using os.path.ismount to using swift.common.utils.ismount has caused problems since the new one raises exceptions in cases where the old one did not. Daemons have been encountering this and exiting; servers have been 500ing instead of 507ing in this case, changing handoff behaviors, etc. Since the new one was specifically written and tested for this new behavior, I left that original function as ismount_raw and made ismount do what it did before. If there really isn't some reason for this new behavior, I'll be glad to get rid of ismount_raw and just keep ismount. I couldn't see any reason for the new behavior myself. Change-Id: I2b5b17f9ed9656cd8804a5ed568170697d0b183d
2014-01-04 06:22:46 +00:00
# Raises exception with _raw -- see next test.
utils.ismount(tmpdir)
finally:
shutil.rmtree(tmpdir)
def test_ismount_raw_parent_path_error(self):
_os_lstat = os.lstat
def _mock_os_lstat(path):
if path.endswith(".."):
raise OSError(13, "foo")
else:
return _os_lstat(path)
tmpdir = mkdtemp()
try:
with patch("os.lstat", _mock_os_lstat):
self.assertRaises(OSError, utils.ismount_raw, tmpdir)
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
finally:
shutil.rmtree(tmpdir)
def test_ismount_successes_dev(self):
_os_lstat = os.lstat
class MockStat(object):
def __init__(self, mode, dev, ino):
self.st_mode = mode
self.st_dev = dev
self.st_ino = ino
def _mock_os_lstat(path):
if path.endswith(".."):
parent = _os_lstat(path)
return MockStat(parent.st_mode, parent.st_dev + 1,
parent.st_ino)
else:
return _os_lstat(path)
tmpdir = mkdtemp()
try:
with patch("os.lstat", _mock_os_lstat):
use testCase.assert instead of assert and try...except In unit tests, we should use: self.assertTrue or self.assertFalse instead of 'assert xxx is True/False', and use self.assertRaises instead of 'try ... except ...' fixes bug #1212602 Change-Id: I25318eb59cd26a98afdce5495273e3b3266655a4
2013-08-15 17:05:07 +08:00
self.assertTrue(utils.ismount(tmpdir))
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
finally:
shutil.rmtree(tmpdir)
def test_ismount_successes_ino(self):
_os_lstat = os.lstat
class MockStat(object):
def __init__(self, mode, dev, ino):
self.st_mode = mode
self.st_dev = dev
self.st_ino = ino
def _mock_os_lstat(path):
if path.endswith(".."):
return _os_lstat(path)
else:
parent_path = os.path.join(path, "..")
child = _os_lstat(path)
parent = _os_lstat(parent_path)
return MockStat(child.st_mode, parent.st_ino,
child.st_dev)
tmpdir = mkdtemp()
try:
with patch("os.lstat", _mock_os_lstat):
use testCase.assert instead of assert and try...except In unit tests, we should use: self.assertTrue or self.assertFalse instead of 'assert xxx is True/False', and use self.assertRaises instead of 'try ... except ...' fixes bug #1212602 Change-Id: I25318eb59cd26a98afdce5495273e3b3266655a4
2013-08-15 17:05:07 +08:00
self.assertTrue(utils.ismount(tmpdir))
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
finally:
shutil.rmtree(tmpdir)
some container serialization cleanup Have json and xml use common record cleanup code. Do a somewhat better job of parsing extensions from content-types. Use a real XML serializer. Change-Id: I10e14dffd1da590b4fd180b4d33ab5de862e2b55
2013-07-23 14:54:51 -07:00
def test_parse_content_type(self):
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(utils.parse_content_type('text/plain'),
('text/plain', []))
self.assertEqual(utils.parse_content_type('text/plain;charset=utf-8'),
('text/plain', [('charset', 'utf-8')]))
self.assertEqual(
some container serialization cleanup Have json and xml use common record cleanup code. Do a somewhat better job of parsing extensions from content-types. Use a real XML serializer. Change-Id: I10e14dffd1da590b4fd180b4d33ab5de862e2b55
2013-07-23 14:54:51 -07:00
utils.parse_content_type('text/plain;hello="world";charset=utf-8'),
('text/plain', [('hello', '"world"'), ('charset', 'utf-8')]))
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
some container serialization cleanup Have json and xml use common record cleanup code. Do a somewhat better job of parsing extensions from content-types. Use a real XML serializer. Change-Id: I10e14dffd1da590b4fd180b4d33ab5de862e2b55
2013-07-23 14:54:51 -07:00
utils.parse_content_type('text/plain; hello="world"; a=b'),
('text/plain', [('hello', '"world"'), ('a', 'b')]))
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
some container serialization cleanup Have json and xml use common record cleanup code. Do a somewhat better job of parsing extensions from content-types. Use a real XML serializer. Change-Id: I10e14dffd1da590b4fd180b4d33ab5de862e2b55
2013-07-23 14:54:51 -07:00
utils.parse_content_type(r'text/plain; x="\""; a=b'),
('text/plain', [('x', r'"\""'), ('a', 'b')]))
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
some container serialization cleanup Have json and xml use common record cleanup code. Do a somewhat better job of parsing extensions from content-types. Use a real XML serializer. Change-Id: I10e14dffd1da590b4fd180b4d33ab5de862e2b55
2013-07-23 14:54:51 -07:00
utils.parse_content_type(r'text/plain; x; a=b'),
('text/plain', [('x', ''), ('a', 'b')]))
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
some container serialization cleanup Have json and xml use common record cleanup code. Do a somewhat better job of parsing extensions from content-types. Use a real XML serializer. Change-Id: I10e14dffd1da590b4fd180b4d33ab5de862e2b55
2013-07-23 14:54:51 -07:00
utils.parse_content_type(r'text/plain; x="\""; a'),
('text/plain', [('x', r'"\""'), ('a', '')]))
SLOs broken for range requests Change-Id: I21175a4be0cda9a8a98c425bff11c80895cd6d3e
2013-09-10 09:01:32 -07:00
def test_override_bytes_from_content_type(self):
listing_dict = {
'bytes': 1234, 'hash': 'asdf', 'name': 'zxcv',
'content_type': 'text/plain; hello="world"; swift_bytes=15'}
utils.override_bytes_from_content_type(listing_dict,
logger=FakeLogger())
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(listing_dict['bytes'], 15)
self.assertEqual(listing_dict['content_type'],
'text/plain;hello="world"')
SLOs broken for range requests Change-Id: I21175a4be0cda9a8a98c425bff11c80895cd6d3e
2013-09-10 09:01:32 -07:00
listing_dict = {
'bytes': 1234, 'hash': 'asdf', 'name': 'zxcv',
'content_type': 'text/plain; hello="world"; swift_bytes=hey'}
utils.override_bytes_from_content_type(listing_dict,
logger=FakeLogger())
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(listing_dict['bytes'], 1234)
self.assertEqual(listing_dict['content_type'],
'text/plain;hello="world"')
SLOs broken for range requests Change-Id: I21175a4be0cda9a8a98c425bff11c80895cd6d3e
2013-09-10 09:01:32 -07:00
Container sync no longer sending swift_bytes value Container sync had a bug where it'd send out the trailing "; swift_bytes=xxx" part of the content-type header. That trailing part is just for internal cluster usage by SLO. Since that needed to be stripped in two places now, I separated it out to a function that both spots call. Change-Id: Ibd6035d7a6b78205344bcc9d98bc1b7a9d463427
2014-05-20 20:19:47 +00:00
def test_clean_content_type(self):
subtests = {
'': '', 'text/plain': 'text/plain',
'text/plain; someother=thing': 'text/plain; someother=thing',
'text/plain; swift_bytes=123': 'text/plain',
'text/plain; someother=thing; swift_bytes=123':
'text/plain; someother=thing',
# Since Swift always tacks on the swift_bytes, clean_content_type()
# only strips swift_bytes if it's last. The next item simply shows
# that if for some other odd reason it's not last,
# clean_content_type() will not remove it from the header.
'text/plain; swift_bytes=123; someother=thing':
'text/plain; swift_bytes=123; someother=thing'}
for before, after in subtests.items():
self.assertEqual(utils.clean_content_type(before), after)
Clean up utf8ness quote Move common codes on utf8ness and quote to common/utils.py Change-Id: I91d98a06fa94ac608119a0d70adedc6d73337c64
2013-08-16 16:55:39 +08:00
def test_quote(self):
res = utils.quote('/v1/a/c3/subdirx/')
assert res == '/v1/a/c3/subdirx/'
res = utils.quote('/v1/a&b/c3/subdirx/')
assert res == '/v1/a%26b/c3/subdirx/'
res = utils.quote('/v1/a&b/c3/subdirx/', safe='&')
assert res == '%2Fv1%2Fa&b%2Fc3%2Fsubdirx%2F'
unicode_sample = u'\uc77c\uc601'
account = 'abc_' + unicode_sample
valid_utf8_str = utils.get_valid_utf8_str(account)
account = 'abc_' + unicode_sample.encode('utf-8')[::-1]
invalid_utf8_str = utils.get_valid_utf8_str(account)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual('abc_%EC%9D%BC%EC%98%81',
utils.quote(valid_utf8_str))
self.assertEqual('abc_%EF%BF%BD%EF%BF%BD%EC%BC%9D%EF%BF%BD',
utils.quote(invalid_utf8_str))
Clean up utf8ness quote Move common codes on utf8ness and quote to common/utils.py Change-Id: I91d98a06fa94ac608119a0d70adedc6d73337c64
2013-08-16 16:55:39 +08:00
Added discoverable capabilities. Swift can now optionally be configured to allow requests to '/info', providing information about the swift cluster. Additionally a HMAC signed requests to '/info?swiftinfo_sig=<sign>&swiftinfo_expires=<expires>' can be configured allowing privileged access to more sensitive information not meant to be public. DocImpact Change-Id: I2379360fbfe3d9e9e8b25f1dc34517d199574495 Implements: blueprint capabilities Closes-Bug: #1245694
2013-10-16 19:28:37 -05:00
def test_get_hmac(self):
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Added discoverable capabilities. Swift can now optionally be configured to allow requests to '/info', providing information about the swift cluster. Additionally a HMAC signed requests to '/info?swiftinfo_sig=<sign>&swiftinfo_expires=<expires>' can be configured allowing privileged access to more sensitive information not meant to be public. DocImpact Change-Id: I2379360fbfe3d9e9e8b25f1dc34517d199574495 Implements: blueprint capabilities Closes-Bug: #1245694
2013-10-16 19:28:37 -05:00
utils.get_hmac('GET', '/path', 1, 'abc'),
'b17f6ff8da0e251737aa9e3ee69a881e3e092e2f')
Output logs of policy index To make it easier for Swift operators to specify problematic devices, a policy index will be recorded in log files of proxy and storage servers for each user request which is related to storage policy. This patch simply adds 'storage_policy_index' field in a log format. If there is no specified policy index, '-' is output in this field. Extra fix: Doc about the log line of storage nodes now properly reflects 'server_pid' field. DocImpact Change-Id: I7286ae85bcbcec73b5377dc115cbdb0f57d1b025 Implements: blueprint logging-policy-number
2014-11-27 18:42:49 +09:00
def test_get_policy_index(self):
# Account has no information about a policy
req = Request.blank(
'/sda1/p/a',
environ={'REQUEST_METHOD': 'GET'})
res = Response()
Replace assertEqual(None, *) with assertIsNone in tests As swift no longer supports Python 2.6, replace assertEqual(None, *) with assertIsNone in tests to have more clear messages in case of failure. Change-Id: I94af3e8156ef40465d4f7a2cb79fb99fc7bbda56 Closes-Bug: #1280522
2016-02-16 23:44:27 +08:00
self.assertIsNone(utils.get_policy_index(req.headers,
res.headers))
Output logs of policy index To make it easier for Swift operators to specify problematic devices, a policy index will be recorded in log files of proxy and storage servers for each user request which is related to storage policy. This patch simply adds 'storage_policy_index' field in a log format. If there is no specified policy index, '-' is output in this field. Extra fix: Doc about the log line of storage nodes now properly reflects 'server_pid' field. DocImpact Change-Id: I7286ae85bcbcec73b5377dc115cbdb0f57d1b025 Implements: blueprint logging-policy-number
2014-11-27 18:42:49 +09:00
# The policy of a container can be specified by the response header
req = Request.blank(
'/sda1/p/a/c',
environ={'REQUEST_METHOD': 'GET'})
res = Response(headers={'X-Backend-Storage-Policy-Index': '1'})
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual('1', utils.get_policy_index(req.headers,
res.headers))
Output logs of policy index To make it easier for Swift operators to specify problematic devices, a policy index will be recorded in log files of proxy and storage servers for each user request which is related to storage policy. This patch simply adds 'storage_policy_index' field in a log format. If there is no specified policy index, '-' is output in this field. Extra fix: Doc about the log line of storage nodes now properly reflects 'server_pid' field. DocImpact Change-Id: I7286ae85bcbcec73b5377dc115cbdb0f57d1b025 Implements: blueprint logging-policy-number
2014-11-27 18:42:49 +09:00
# The policy of an object to be created can be specified by the request
# header
req = Request.blank(
'/sda1/p/a/c/o',
environ={'REQUEST_METHOD': 'PUT'},
headers={'X-Backend-Storage-Policy-Index': '2'})
res = Response()
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual('2', utils.get_policy_index(req.headers,
res.headers))
Output logs of policy index To make it easier for Swift operators to specify problematic devices, a policy index will be recorded in log files of proxy and storage servers for each user request which is related to storage policy. This patch simply adds 'storage_policy_index' field in a log format. If there is no specified policy index, '-' is output in this field. Extra fix: Doc about the log line of storage nodes now properly reflects 'server_pid' field. DocImpact Change-Id: I7286ae85bcbcec73b5377dc115cbdb0f57d1b025 Implements: blueprint logging-policy-number
2014-11-27 18:42:49 +09:00
Unify backend logging Make account, object, and container servers construct log lines using the same utility function so they will produce identically formatted lines. This change reorders the fields logged for the account server. This change also adds the "additional info" field to the two servers that didn't log that field. This makes the log lines identical across all 3 servers. If people don't like that, I can take that out. I think it makes the documentation, parsing of the log lines, and the code a tad cleaner. DocImpact Change-Id: I268dc0df9dd07afa5382592a28ea37b96c6c2f44 Closes-Bug: 1280955
2014-03-26 22:55:55 +00:00
def test_get_log_line(self):
req = Request.blank(
'/sda1/p/a/c/o',
environ={'REQUEST_METHOD': 'HEAD', 'REMOTE_ADDR': '1.2.3.4'})
res = Response()
trans_time = 1.2
additional_info = 'some information'
added process pid to the end of storage node log lines Change-Id: I1c2709d85575fc7d4868fafd9ee757fd01868436
2014-07-07 17:11:38 -07:00
server_pid = 1234
Unify backend logging Make account, object, and container servers construct log lines using the same utility function so they will produce identically formatted lines. This change reorders the fields logged for the account server. This change also adds the "additional info" field to the two servers that didn't log that field. This makes the log lines identical across all 3 servers. If people don't like that, I can take that out. I think it makes the documentation, parsing of the log lines, and the code a tad cleaner. DocImpact Change-Id: I268dc0df9dd07afa5382592a28ea37b96c6c2f44 Closes-Bug: 1280955
2014-03-26 22:55:55 +00:00
exp_line = '1.2.3.4 - - [01/Jan/1970:02:46:41 +0000] "HEAD ' \
Output logs of policy index To make it easier for Swift operators to specify problematic devices, a policy index will be recorded in log files of proxy and storage servers for each user request which is related to storage policy. This patch simply adds 'storage_policy_index' field in a log format. If there is no specified policy index, '-' is output in this field. Extra fix: Doc about the log line of storage nodes now properly reflects 'server_pid' field. DocImpact Change-Id: I7286ae85bcbcec73b5377dc115cbdb0f57d1b025 Implements: blueprint logging-policy-number
2014-11-27 18:42:49 +09:00
'/sda1/p/a/c/o" 200 - "-" "-" "-" 1.2000 "some information" 1234 -'
Unify backend logging Make account, object, and container servers construct log lines using the same utility function so they will produce identically formatted lines. This change reorders the fields logged for the account server. This change also adds the "additional info" field to the two servers that didn't log that field. This makes the log lines identical across all 3 servers. If people don't like that, I can take that out. I think it makes the documentation, parsing of the log lines, and the code a tad cleaner. DocImpact Change-Id: I268dc0df9dd07afa5382592a28ea37b96c6c2f44 Closes-Bug: 1280955
2014-03-26 22:55:55 +00:00
with mock.patch(
'time.gmtime',
mock.MagicMock(side_effect=[time.gmtime(10001.0)])):
added process pid to the end of storage node log lines Change-Id: I1c2709d85575fc7d4868fafd9ee757fd01868436
2014-07-07 17:11:38 -07:00
with mock.patch(
'os.getpid', mock.MagicMock(return_value=server_pid)):
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
added process pid to the end of storage node log lines Change-Id: I1c2709d85575fc7d4868fafd9ee757fd01868436
2014-07-07 17:11:38 -07:00
exp_line,
utils.get_log_line(req, res, trans_time, additional_info))
Unify backend logging Make account, object, and container servers construct log lines using the same utility function so they will produce identically formatted lines. This change reorders the fields logged for the account server. This change also adds the "additional info" field to the two servers that didn't log that field. This makes the log lines identical across all 3 servers. If people don't like that, I can take that out. I think it makes the documentation, parsing of the log lines, and the code a tad cleaner. DocImpact Change-Id: I268dc0df9dd07afa5382592a28ea37b96c6c2f44 Closes-Bug: 1280955
2014-03-26 22:55:55 +00:00
Unit test for common.utils.cache_from_env The behavior of common.utils.cache_from_env was changed by https://review.openstack.org/#/c/89488/. This patch adds unit test for that function. Change-Id: If757e12990c971325f7705731ef529a7e2a9eee7
2014-04-25 13:42:30 +01:00
def test_cache_from_env(self):
# should never get logging when swift.cache is found
env = {'swift.cache': 42}
logger = FakeLogger()
with mock.patch('swift.common.utils.logging', logger):
self.assertEqual(42, utils.cache_from_env(env))
self.assertEqual(0, len(logger.get_lines_for_level('error')))
logger = FakeLogger()
with mock.patch('swift.common.utils.logging', logger):
self.assertEqual(42, utils.cache_from_env(env, False))
self.assertEqual(0, len(logger.get_lines_for_level('error')))
logger = FakeLogger()
with mock.patch('swift.common.utils.logging', logger):
self.assertEqual(42, utils.cache_from_env(env, True))
self.assertEqual(0, len(logger.get_lines_for_level('error')))
# check allow_none controls logging when swift.cache is not found
err_msg = 'ERROR: swift.cache could not be found in env!'
env = {}
logger = FakeLogger()
with mock.patch('swift.common.utils.logging', logger):
Replace assertEqual(None, *) with assertIsNone in tests As swift no longer supports Python 2.6, replace assertEqual(None, *) with assertIsNone in tests to have more clear messages in case of failure. Change-Id: I94af3e8156ef40465d4f7a2cb79fb99fc7bbda56 Closes-Bug: #1280522
2016-02-16 23:44:27 +08:00
self.assertIsNone(utils.cache_from_env(env))
Unit test for common.utils.cache_from_env The behavior of common.utils.cache_from_env was changed by https://review.openstack.org/#/c/89488/. This patch adds unit test for that function. Change-Id: If757e12990c971325f7705731ef529a7e2a9eee7
2014-04-25 13:42:30 +01:00
self.assertTrue(err_msg in logger.get_lines_for_level('error'))
logger = FakeLogger()
with mock.patch('swift.common.utils.logging', logger):
Replace assertEqual(None, *) with assertIsNone in tests As swift no longer supports Python 2.6, replace assertEqual(None, *) with assertIsNone in tests to have more clear messages in case of failure. Change-Id: I94af3e8156ef40465d4f7a2cb79fb99fc7bbda56 Closes-Bug: #1280522
2016-02-16 23:44:27 +08:00
self.assertIsNone(utils.cache_from_env(env, False))
Unit test for common.utils.cache_from_env The behavior of common.utils.cache_from_env was changed by https://review.openstack.org/#/c/89488/. This patch adds unit test for that function. Change-Id: If757e12990c971325f7705731ef529a7e2a9eee7
2014-04-25 13:42:30 +01:00
self.assertTrue(err_msg in logger.get_lines_for_level('error'))
logger = FakeLogger()
with mock.patch('swift.common.utils.logging', logger):
Replace assertEqual(None, *) with assertIsNone in tests As swift no longer supports Python 2.6, replace assertEqual(None, *) with assertIsNone in tests to have more clear messages in case of failure. Change-Id: I94af3e8156ef40465d4f7a2cb79fb99fc7bbda56 Closes-Bug: #1280522
2016-02-16 23:44:27 +08:00
self.assertIsNone(utils.cache_from_env(env, True))
Unit test for common.utils.cache_from_env The behavior of common.utils.cache_from_env was changed by https://review.openstack.org/#/c/89488/. This patch adds unit test for that function. Change-Id: If757e12990c971325f7705731ef529a7e2a9eee7
2014-04-25 13:42:30 +01:00
self.assertEqual(0, len(logger.get_lines_for_level('error')))
fsync() on directories renamer() method now does a fsync on containing directory of target path and also on parent dirs of newly created directories, by default. This can be explicitly turned off in cases where it is not necessary (For example- quarantines). The following article explains why this is necessary: http://lwn.net/Articles/457667/ Although, it may seem like the right thing to do, this change does come at a performance penalty. However, no configurable option is provided to turn it off. Also, lock_path() inside invalidate_hash() was always creating part of object path in filesystem. Those are never fsync'd. This has been fixed. Change-Id: Id8e02f84f48370edda7fb0c46e030db3b53a71e3 Signed-off-by: Prashanth Pai <ppai@redhat.com>
2014-10-08 19:49:39 +05:30
def test_fsync_dir(self):
tempdir = None
fd = None
try:
tempdir = mkdtemp(dir='/tmp')
fd, temppath = tempfile.mkstemp(dir=tempdir)
_mock_fsync = mock.Mock()
_mock_close = mock.Mock()
with patch('swift.common.utils.fsync', _mock_fsync):
with patch('os.close', _mock_close):
utils.fsync_dir(tempdir)
self.assertTrue(_mock_fsync.called)
self.assertTrue(_mock_close.called)
self.assertTrue(isinstance(_mock_fsync.call_args[0][0], int))
self.assertEqual(_mock_fsync.call_args[0][0],
_mock_close.call_args[0][0])
# Not a directory - arg is file path
self.assertRaises(OSError, utils.fsync_dir, temppath)
logger = FakeLogger()
def _mock_fsync(fd):
raise OSError(errno.EBADF, os.strerror(errno.EBADF))
with patch('swift.common.utils.fsync', _mock_fsync):
with mock.patch('swift.common.utils.logging', logger):
utils.fsync_dir(tempdir)
self.assertEqual(1, len(logger.get_lines_for_level('warning')))
finally:
if fd is not None:
os.close(fd)
os.unlink(temppath)
if tempdir:
os.rmdir(tempdir)
def test_renamer_with_fsync_dir(self):
tempdir = None
try:
tempdir = mkdtemp(dir='/tmp')
# Simulate part of object path already existing
part_dir = os.path.join(tempdir, 'objects/1234/')
os.makedirs(part_dir)
obj_dir = os.path.join(part_dir, 'aaa', 'a' * 32)
obj_path = os.path.join(obj_dir, '1425276031.12345.data')
# Object dir had to be created
_m_os_rename = mock.Mock()
_m_fsync_dir = mock.Mock()
with patch('os.rename', _m_os_rename):
with patch('swift.common.utils.fsync_dir', _m_fsync_dir):
utils.renamer("fake_path", obj_path)
_m_os_rename.assert_called_once_with('fake_path', obj_path)
# fsync_dir on parents of all newly create dirs
self.assertEqual(_m_fsync_dir.call_count, 3)
# Object dir existed
_m_os_rename.reset_mock()
_m_fsync_dir.reset_mock()
with patch('os.rename', _m_os_rename):
with patch('swift.common.utils.fsync_dir', _m_fsync_dir):
utils.renamer("fake_path", obj_path)
_m_os_rename.assert_called_once_with('fake_path', obj_path)
# fsync_dir only on the leaf dir
self.assertEqual(_m_fsync_dir.call_count, 1)
finally:
if tempdir:
shutil.rmtree(tempdir)
def test_renamer_when_fsync_is_false(self):
_m_os_rename = mock.Mock()
_m_fsync_dir = mock.Mock()
_m_makedirs_count = mock.Mock(return_value=2)
with patch('os.rename', _m_os_rename):
with patch('swift.common.utils.fsync_dir', _m_fsync_dir):
with patch('swift.common.utils.makedirs_count',
_m_makedirs_count):
utils.renamer("fake_path", "/a/b/c.data", fsync=False)
_m_makedirs_count.assert_called_once_with("/a/b")
_m_os_rename.assert_called_once_with('fake_path', "/a/b/c.data")
self.assertFalse(_m_fsync_dir.called)
def test_makedirs_count(self):
tempdir = None
fd = None
try:
tempdir = mkdtemp(dir='/tmp')
os.makedirs(os.path.join(tempdir, 'a/b'))
# 4 new dirs created
dirpath = os.path.join(tempdir, 'a/b/1/2/3/4')
ret = utils.makedirs_count(dirpath)
self.assertEqual(ret, 4)
# no new dirs created - dir already exists
ret = utils.makedirs_count(dirpath)
self.assertEqual(ret, 0)
# path exists and is a file
fd, temppath = tempfile.mkstemp(dir=dirpath)
os.close(fd)
self.assertRaises(OSError, utils.makedirs_count, temppath)
finally:
if tempdir:
shutil.rmtree(tempdir)
Added discoverable capabilities. Swift can now optionally be configured to allow requests to '/info', providing information about the swift cluster. Additionally a HMAC signed requests to '/info?swiftinfo_sig=<sign>&swiftinfo_expires=<expires>' can be configured allowing privileged access to more sensitive information not meant to be public. DocImpact Change-Id: I2379360fbfe3d9e9e8b25f1dc34517d199574495 Implements: blueprint capabilities Closes-Bug: #1245694
2013-10-16 19:28:37 -05:00
Add multiple reseller prefixes and composite tokens This change is in support of Composite Tokens and Service Accounts (see http://specs.openstack.org/openstack/swift-specs/specs/in_progress/ service_token.html) During coding, minor changes were made compared to the original specification. See https://review.openstack.org/138771 for these changes. DocImpact Change-Id: I6072b4efb3a479a8e0cc2d9c11ffda5764b55e30
2014-11-25 14:42:42 +00:00
class ResellerConfReader(unittest.TestCase):
def setUp(self):
self.default_rules = {'operator_roles': ['admin', 'swiftoperator'],
'service_roles': [],
'require_group': ''}
def test_defaults(self):
conf = {}
prefixes, options = utils.config_read_reseller_options(
conf, self.default_rules)
self.assertEqual(prefixes, ['AUTH_'])
self.assertEqual(options['AUTH_'], self.default_rules)
def test_same_as_default(self):
conf = {'reseller_prefix': 'AUTH',
'operator_roles': 'admin, swiftoperator'}
prefixes, options = utils.config_read_reseller_options(
conf, self.default_rules)
self.assertEqual(prefixes, ['AUTH_'])
self.assertEqual(options['AUTH_'], self.default_rules)
def test_single_blank_reseller(self):
conf = {'reseller_prefix': ''}
prefixes, options = utils.config_read_reseller_options(
conf, self.default_rules)
self.assertEqual(prefixes, [''])
self.assertEqual(options[''], self.default_rules)
def test_single_blank_reseller_with_conf(self):
conf = {'reseller_prefix': '',
"''operator_roles": 'role1, role2'}
prefixes, options = utils.config_read_reseller_options(
conf, self.default_rules)
self.assertEqual(prefixes, [''])
self.assertEqual(options[''].get('operator_roles'),
['role1', 'role2'])
self.assertEqual(options[''].get('service_roles'),
self.default_rules.get('service_roles'))
self.assertEqual(options[''].get('require_group'),
self.default_rules.get('require_group'))
def test_multiple_same_resellers(self):
conf = {'reseller_prefix': " '' , '' "}
prefixes, options = utils.config_read_reseller_options(
conf, self.default_rules)
self.assertEqual(prefixes, [''])
conf = {'reseller_prefix': '_, _'}
prefixes, options = utils.config_read_reseller_options(
conf, self.default_rules)
self.assertEqual(prefixes, ['_'])
conf = {'reseller_prefix': 'AUTH, PRE2, AUTH, PRE2'}
prefixes, options = utils.config_read_reseller_options(
conf, self.default_rules)
self.assertEqual(prefixes, ['AUTH_', 'PRE2_'])
def test_several_resellers_with_conf(self):
conf = {'reseller_prefix': 'PRE1, PRE2',
'PRE1_operator_roles': 'role1, role2',
'PRE1_service_roles': 'role3, role4',
'PRE2_operator_roles': 'role5',
'PRE2_service_roles': 'role6',
'PRE2_require_group': 'pre2_group'}
prefixes, options = utils.config_read_reseller_options(
conf, self.default_rules)
self.assertEqual(prefixes, ['PRE1_', 'PRE2_'])
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(set(['role1', 'role2']),
set(options['PRE1_'].get('operator_roles')))
self.assertEqual(['role5'],
options['PRE2_'].get('operator_roles'))
self.assertEqual(set(['role3', 'role4']),
set(options['PRE1_'].get('service_roles')))
self.assertEqual(['role6'], options['PRE2_'].get('service_roles'))
self.assertEqual('', options['PRE1_'].get('require_group'))
self.assertEqual('pre2_group', options['PRE2_'].get('require_group'))
Add multiple reseller prefixes and composite tokens This change is in support of Composite Tokens and Service Accounts (see http://specs.openstack.org/openstack/swift-specs/specs/in_progress/ service_token.html) During coding, minor changes were made compared to the original specification. See https://review.openstack.org/138771 for these changes. DocImpact Change-Id: I6072b4efb3a479a8e0cc2d9c11ffda5764b55e30
2014-11-25 14:42:42 +00:00
def test_several_resellers_first_blank(self):
conf = {'reseller_prefix': " '' , PRE2",
"''operator_roles": 'role1, role2',
"''service_roles": 'role3, role4',
'PRE2_operator_roles': 'role5',
'PRE2_service_roles': 'role6',
'PRE2_require_group': 'pre2_group'}
prefixes, options = utils.config_read_reseller_options(
conf, self.default_rules)
self.assertEqual(prefixes, ['', 'PRE2_'])
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(set(['role1', 'role2']),
set(options[''].get('operator_roles')))
self.assertEqual(['role5'],
options['PRE2_'].get('operator_roles'))
self.assertEqual(set(['role3', 'role4']),
set(options[''].get('service_roles')))
self.assertEqual(['role6'], options['PRE2_'].get('service_roles'))
self.assertEqual('', options[''].get('require_group'))
self.assertEqual('pre2_group', options['PRE2_'].get('require_group'))
Add multiple reseller prefixes and composite tokens This change is in support of Composite Tokens and Service Accounts (see http://specs.openstack.org/openstack/swift-specs/specs/in_progress/ service_token.html) During coding, minor changes were made compared to the original specification. See https://review.openstack.org/138771 for these changes. DocImpact Change-Id: I6072b4efb3a479a8e0cc2d9c11ffda5764b55e30
2014-11-25 14:42:42 +00:00
def test_several_resellers_with_blank_comma(self):
conf = {'reseller_prefix': "AUTH , '', PRE2",
"''operator_roles": 'role1, role2',
"''service_roles": 'role3, role4',
'PRE2_operator_roles': 'role5',
'PRE2_service_roles': 'role6',
'PRE2_require_group': 'pre2_group'}
prefixes, options = utils.config_read_reseller_options(
conf, self.default_rules)
self.assertEqual(prefixes, ['AUTH_', '', 'PRE2_'])
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(set(['admin', 'swiftoperator']),
set(options['AUTH_'].get('operator_roles')))
self.assertEqual(set(['role1', 'role2']),
set(options[''].get('operator_roles')))
self.assertEqual(['role5'],
options['PRE2_'].get('operator_roles'))
self.assertEqual([],
options['AUTH_'].get('service_roles'))
self.assertEqual(set(['role3', 'role4']),
set(options[''].get('service_roles')))
self.assertEqual(['role6'], options['PRE2_'].get('service_roles'))
self.assertEqual('', options['AUTH_'].get('require_group'))
self.assertEqual('', options[''].get('require_group'))
self.assertEqual('pre2_group', options['PRE2_'].get('require_group'))
Add multiple reseller prefixes and composite tokens This change is in support of Composite Tokens and Service Accounts (see http://specs.openstack.org/openstack/swift-specs/specs/in_progress/ service_token.html) During coding, minor changes were made compared to the original specification. See https://review.openstack.org/138771 for these changes. DocImpact Change-Id: I6072b4efb3a479a8e0cc2d9c11ffda5764b55e30
2014-11-25 14:42:42 +00:00
def test_stray_comma(self):
conf = {'reseller_prefix': "AUTH ,, PRE2",
"''operator_roles": 'role1, role2',
"''service_roles": 'role3, role4',
'PRE2_operator_roles': 'role5',
'PRE2_service_roles': 'role6',
'PRE2_require_group': 'pre2_group'}
prefixes, options = utils.config_read_reseller_options(
conf, self.default_rules)
self.assertEqual(prefixes, ['AUTH_', 'PRE2_'])
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(set(['admin', 'swiftoperator']),
set(options['AUTH_'].get('operator_roles')))
self.assertEqual(['role5'],
options['PRE2_'].get('operator_roles'))
self.assertEqual([],
options['AUTH_'].get('service_roles'))
self.assertEqual(['role6'], options['PRE2_'].get('service_roles'))
self.assertEqual('', options['AUTH_'].get('require_group'))
self.assertEqual('pre2_group', options['PRE2_'].get('require_group'))
Add multiple reseller prefixes and composite tokens This change is in support of Composite Tokens and Service Accounts (see http://specs.openstack.org/openstack/swift-specs/specs/in_progress/ service_token.html) During coding, minor changes were made compared to the original specification. See https://review.openstack.org/138771 for these changes. DocImpact Change-Id: I6072b4efb3a479a8e0cc2d9c11ffda5764b55e30
2014-11-25 14:42:42 +00:00
def test_multiple_stray_commas_resellers(self):
conf = {'reseller_prefix': ' , , ,'}
prefixes, options = utils.config_read_reseller_options(
conf, self.default_rules)
self.assertEqual(prefixes, [''])
self.assertEqual(options[''], self.default_rules)
def test_unprefixed_options(self):
conf = {'reseller_prefix': "AUTH , '', PRE2",
"operator_roles": 'role1, role2',
"service_roles": 'role3, role4',
'require_group': 'auth_blank_group',
'PRE2_operator_roles': 'role5',
'PRE2_service_roles': 'role6',
'PRE2_require_group': 'pre2_group'}
prefixes, options = utils.config_read_reseller_options(
conf, self.default_rules)
self.assertEqual(prefixes, ['AUTH_', '', 'PRE2_'])
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(set(['role1', 'role2']),
set(options['AUTH_'].get('operator_roles')))
self.assertEqual(set(['role1', 'role2']),
set(options[''].get('operator_roles')))
self.assertEqual(['role5'],
options['PRE2_'].get('operator_roles'))
self.assertEqual(set(['role3', 'role4']),
set(options['AUTH_'].get('service_roles')))
self.assertEqual(set(['role3', 'role4']),
set(options[''].get('service_roles')))
self.assertEqual(['role6'], options['PRE2_'].get('service_roles'))
self.assertEqual('auth_blank_group',
options['AUTH_'].get('require_group'))
self.assertEqual('auth_blank_group', options[''].get('require_group'))
self.assertEqual('pre2_group', options['PRE2_'].get('require_group'))
Add multiple reseller prefixes and composite tokens This change is in support of Composite Tokens and Service Accounts (see http://specs.openstack.org/openstack/swift-specs/specs/in_progress/ service_token.html) During coding, minor changes were made compared to the original specification. See https://review.openstack.org/138771 for these changes. DocImpact Change-Id: I6072b4efb3a479a8e0cc2d9c11ffda5764b55e30
2014-11-25 14:42:42 +00:00
Auditor will clean up stale rsync tempfiles DiskFile already fills in the _ondisk_info attribute when it tries to open a diskfile - even if the DiskFile's fileset is not valid or deleted. During this process the rsync tempfiles would be discovered and logged, but no-one would attempt to clean them up - even if they were really old. Instead of logging and ignoring unexpected files when validate a DiskFile fileset we'll add unexpected files to the unexpected key in the _ondisk_info attribute. With a little bit of re-organization in the auditor's object_audit method to get things into a single return path we can add an unconditional check for unexpected files and remove those that are "old enough". Since the replicator will kill any rsync processes that are running longer than the configured rsync_timeout we know that any rsync tempfiles older than this can be deleted. Split unlink_older_than in common.utils into two functions to allow an explicit list of previously discovered paths to be passed in to avoid an extra listdir. Since the getmtime handling already ignores OSError there's less concern of race condition where a previous discovered unexpected file is reaped by rsync while we're attempting to clean it up. Update some doc on the new config option. Closes-Bug: #1554005 Change-Id: Id67681cb77f605e3491b8afcb9c69d769e154283
2016-03-15 17:09:21 -07:00
class TestUnlinkOlder(unittest.TestCase):
def setUp(self):
self.tempdir = mkdtemp()
self.mtime = {}
def tearDown(self):
rmtree(self.tempdir, ignore_errors=True)
def touch(self, fpath, mtime=None):
self.mtime[fpath] = mtime or time.time()
open(fpath, 'w')
@contextlib.contextmanager
def high_resolution_getmtime(self):
orig_getmtime = os.path.getmtime
def mock_getmtime(fpath):
mtime = self.mtime.get(fpath)
if mtime is None:
mtime = orig_getmtime(fpath)
return mtime
with mock.patch('os.path.getmtime', mock_getmtime):
yield
def test_unlink_older_than_path_not_exists(self):
path = os.path.join(self.tempdir, 'does-not-exist')
# just make sure it doesn't blow up
utils.unlink_older_than(path, time.time())
def test_unlink_older_than_file(self):
path = os.path.join(self.tempdir, 'some-file')
self.touch(path)
with self.assertRaises(OSError) as ctx:
utils.unlink_older_than(path, time.time())
self.assertEqual(ctx.exception.errno, errno.ENOTDIR)
def test_unlink_older_than_now(self):
self.touch(os.path.join(self.tempdir, 'test'))
with self.high_resolution_getmtime():
utils.unlink_older_than(self.tempdir, time.time())
self.assertEqual([], os.listdir(self.tempdir))
def test_unlink_not_old_enough(self):
start = time.time()
self.touch(os.path.join(self.tempdir, 'test'))
with self.high_resolution_getmtime():
utils.unlink_older_than(self.tempdir, start)
self.assertEqual(['test'], os.listdir(self.tempdir))
def test_unlink_mixed(self):
self.touch(os.path.join(self.tempdir, 'first'))
cutoff = time.time()
self.touch(os.path.join(self.tempdir, 'second'))
with self.high_resolution_getmtime():
utils.unlink_older_than(self.tempdir, cutoff)
self.assertEqual(['second'], os.listdir(self.tempdir))
def test_unlink_paths(self):
paths = []
for item in ('first', 'second', 'third'):
path = os.path.join(self.tempdir, item)
self.touch(path)
paths.append(path)
# don't unlink everyone
with self.high_resolution_getmtime():
utils.unlink_paths_older_than(paths[:2], time.time())
self.assertEqual(['third'], os.listdir(self.tempdir))
def test_unlink_empty_paths(self):
# just make sure it doesn't blow up
utils.unlink_paths_older_than([], time.time())
def test_unlink_not_exists_paths(self):
path = os.path.join(self.tempdir, 'does-not-exist')
# just make sure it doesn't blow up
utils.unlink_paths_older_than([path], time.time())
Added discoverable capabilities. Swift can now optionally be configured to allow requests to '/info', providing information about the swift cluster. Additionally a HMAC signed requests to '/info?swiftinfo_sig=<sign>&swiftinfo_expires=<expires>' can be configured allowing privileged access to more sensitive information not meant to be public. DocImpact Change-Id: I2379360fbfe3d9e9e8b25f1dc34517d199574495 Implements: blueprint capabilities Closes-Bug: #1245694
2013-10-16 19:28:37 -05:00
class TestSwiftInfo(unittest.TestCase):
def tearDown(self):
utils._swift_info = {}
utils._swift_admin_info = {}
def test_register_swift_info(self):
utils.register_swift_info(foo='bar')
utils.register_swift_info(lorem='ipsum')
utils.register_swift_info('cap1', cap1_foo='cap1_bar')
utils.register_swift_info('cap1', cap1_lorem='cap1_ipsum')
self.assertTrue('swift' in utils._swift_info)
self.assertTrue('foo' in utils._swift_info['swift'])
self.assertEqual(utils._swift_info['swift']['foo'], 'bar')
self.assertTrue('lorem' in utils._swift_info['swift'])
self.assertEqual(utils._swift_info['swift']['lorem'], 'ipsum')
self.assertTrue('cap1' in utils._swift_info)
self.assertTrue('cap1_foo' in utils._swift_info['cap1'])
self.assertEqual(utils._swift_info['cap1']['cap1_foo'], 'cap1_bar')
self.assertTrue('cap1_lorem' in utils._swift_info['cap1'])
self.assertEqual(utils._swift_info['cap1']['cap1_lorem'], 'cap1_ipsum')
self.assertRaises(ValueError,
utils.register_swift_info, 'admin', foo='bar')
self.assertRaises(ValueError,
utils.register_swift_info, 'disallowed_sections',
disallowed_sections=None)
Add ability to remove subsections from /info Change-Id: Ic881065962cf5f69f7a5b64f6e38d9e6e1f8fd18
2014-05-20 12:33:59 -07:00
utils.register_swift_info('goodkey', foo='5.6')
self.assertRaises(ValueError,
utils.register_swift_info, 'bad.key', foo='5.6')
data = {'bad.key': '5.6'}
self.assertRaises(ValueError,
utils.register_swift_info, 'goodkey', **data)
Added discoverable capabilities. Swift can now optionally be configured to allow requests to '/info', providing information about the swift cluster. Additionally a HMAC signed requests to '/info?swiftinfo_sig=<sign>&swiftinfo_expires=<expires>' can be configured allowing privileged access to more sensitive information not meant to be public. DocImpact Change-Id: I2379360fbfe3d9e9e8b25f1dc34517d199574495 Implements: blueprint capabilities Closes-Bug: #1245694
2013-10-16 19:28:37 -05:00
def test_get_swift_info(self):
utils._swift_info = {'swift': {'foo': 'bar'},
'cap1': {'cap1_foo': 'cap1_bar'}}
utils._swift_admin_info = {'admin_cap1': {'ac1_foo': 'ac1_bar'}}
info = utils.get_swift_info()
self.assertTrue('admin' not in info)
self.assertTrue('swift' in info)
self.assertTrue('foo' in info['swift'])
self.assertEqual(utils._swift_info['swift']['foo'], 'bar')
self.assertTrue('cap1' in info)
self.assertTrue('cap1_foo' in info['cap1'])
self.assertEqual(utils._swift_info['cap1']['cap1_foo'], 'cap1_bar')
def test_get_swift_info_with_disallowed_sections(self):
utils._swift_info = {'swift': {'foo': 'bar'},
'cap1': {'cap1_foo': 'cap1_bar'},
'cap2': {'cap2_foo': 'cap2_bar'},
'cap3': {'cap3_foo': 'cap3_bar'}}
utils._swift_admin_info = {'admin_cap1': {'ac1_foo': 'ac1_bar'}}
info = utils.get_swift_info(disallowed_sections=['cap1', 'cap3'])
self.assertTrue('admin' not in info)
self.assertTrue('swift' in info)
self.assertTrue('foo' in info['swift'])
self.assertEqual(info['swift']['foo'], 'bar')
self.assertTrue('cap1' not in info)
self.assertTrue('cap2' in info)
self.assertTrue('cap2_foo' in info['cap2'])
self.assertEqual(info['cap2']['cap2_foo'], 'cap2_bar')
self.assertTrue('cap3' not in info)
def test_register_swift_admin_info(self):
utils.register_swift_info(admin=True, admin_foo='admin_bar')
utils.register_swift_info(admin=True, admin_lorem='admin_ipsum')
utils.register_swift_info('cap1', admin=True, ac1_foo='ac1_bar')
utils.register_swift_info('cap1', admin=True, ac1_lorem='ac1_ipsum')
self.assertTrue('swift' in utils._swift_admin_info)
self.assertTrue('admin_foo' in utils._swift_admin_info['swift'])
self.assertEqual(
utils._swift_admin_info['swift']['admin_foo'], 'admin_bar')
self.assertTrue('admin_lorem' in utils._swift_admin_info['swift'])
self.assertEqual(
utils._swift_admin_info['swift']['admin_lorem'], 'admin_ipsum')
self.assertTrue('cap1' in utils._swift_admin_info)
self.assertTrue('ac1_foo' in utils._swift_admin_info['cap1'])
self.assertEqual(
utils._swift_admin_info['cap1']['ac1_foo'], 'ac1_bar')
self.assertTrue('ac1_lorem' in utils._swift_admin_info['cap1'])
self.assertEqual(
utils._swift_admin_info['cap1']['ac1_lorem'], 'ac1_ipsum')
self.assertTrue('swift' not in utils._swift_info)
self.assertTrue('cap1' not in utils._swift_info)
def test_get_swift_admin_info(self):
utils._swift_info = {'swift': {'foo': 'bar'},
'cap1': {'cap1_foo': 'cap1_bar'}}
utils._swift_admin_info = {'admin_cap1': {'ac1_foo': 'ac1_bar'}}
info = utils.get_swift_info(admin=True)
self.assertTrue('admin' in info)
self.assertTrue('admin_cap1' in info['admin'])
self.assertTrue('ac1_foo' in info['admin']['admin_cap1'])
self.assertEqual(info['admin']['admin_cap1']['ac1_foo'], 'ac1_bar')
self.assertTrue('swift' in info)
self.assertTrue('foo' in info['swift'])
self.assertEqual(utils._swift_info['swift']['foo'], 'bar')
self.assertTrue('cap1' in info)
self.assertTrue('cap1_foo' in info['cap1'])
self.assertEqual(utils._swift_info['cap1']['cap1_foo'], 'cap1_bar')
def test_get_swift_admin_info_with_disallowed_sections(self):
utils._swift_info = {'swift': {'foo': 'bar'},
'cap1': {'cap1_foo': 'cap1_bar'},
'cap2': {'cap2_foo': 'cap2_bar'},
'cap3': {'cap3_foo': 'cap3_bar'}}
utils._swift_admin_info = {'admin_cap1': {'ac1_foo': 'ac1_bar'}}
info = utils.get_swift_info(
admin=True, disallowed_sections=['cap1', 'cap3'])
self.assertTrue('admin' in info)
self.assertTrue('admin_cap1' in info['admin'])
self.assertTrue('ac1_foo' in info['admin']['admin_cap1'])
self.assertEqual(info['admin']['admin_cap1']['ac1_foo'], 'ac1_bar')
self.assertTrue('disallowed_sections' in info['admin'])
self.assertTrue('cap1' in info['admin']['disallowed_sections'])
self.assertTrue('cap2' not in info['admin']['disallowed_sections'])
self.assertTrue('cap3' in info['admin']['disallowed_sections'])
self.assertTrue('swift' in info)
self.assertTrue('foo' in info['swift'])
self.assertEqual(info['swift']['foo'], 'bar')
self.assertTrue('cap1' not in info)
self.assertTrue('cap2' in info)
self.assertTrue('cap2_foo' in info['cap2'])
self.assertEqual(info['cap2']['cap2_foo'], 'cap2_bar')
self.assertTrue('cap3' not in info)
Add ability to remove subsections from /info Change-Id: Ic881065962cf5f69f7a5b64f6e38d9e6e1f8fd18
2014-05-20 12:33:59 -07:00
def test_get_swift_admin_info_with_disallowed_sub_sections(self):
utils._swift_info = {'swift': {'foo': 'bar'},
'cap1': {'cap1_foo': 'cap1_bar',
'cap1_moo': 'cap1_baa'},
'cap2': {'cap2_foo': 'cap2_bar'},
'cap3': {'cap2_foo': 'cap2_bar'},
'cap4': {'a': {'b': {'c': 'c'},
'b.c': 'b.c'}}}
utils._swift_admin_info = {'admin_cap1': {'ac1_foo': 'ac1_bar'}}
info = utils.get_swift_info(
admin=True, disallowed_sections=['cap1.cap1_foo', 'cap3',
'cap4.a.b.c'])
self.assertTrue('cap3' not in info)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(info['cap1']['cap1_moo'], 'cap1_baa')
Add ability to remove subsections from /info Change-Id: Ic881065962cf5f69f7a5b64f6e38d9e6e1f8fd18
2014-05-20 12:33:59 -07:00
self.assertTrue('cap1_foo' not in info['cap1'])
self.assertTrue('c' not in info['cap4']['a']['b'])
self.assertEqual(info['cap4']['a']['b.c'], 'b.c')
def test_get_swift_info_with_unmatched_disallowed_sections(self):
cap1 = {'cap1_foo': 'cap1_bar',
'cap1_moo': 'cap1_baa'}
utils._swift_info = {'swift': {'foo': 'bar'},
'cap1': cap1}
# expect no exceptions
Allow hostnames for nodes in Rings This change modifies the swift-ring-builder and introduces new format of sub-commands (search, list_parts, set_weight, set_info and remove) in addition to add sub-command so that hostnames can be used in place of an ip-address for the sub-commands. The account reaper, container synchronizer, and replicators were also updated so that they still have a way to identify a particular device as being "local". Previously this was Change-Id: Ie471902413002872fc6755bacd36af3b9c613b74 Change-Id: Ieff583ffb932133e3820744a3f8f9f491686b08d Co-Authored-By: Alex Pecoraro <alex.pecoraro@emc.com> Implements: blueprint allow-hostnames-for-nodes-in-rings
2014-12-09 04:44:52 +09:00
info = utils.get_swift_info(
disallowed_sections=['cap2.cap1_foo', 'cap1.no_match',
'cap1.cap1_foo.no_match.no_match'])
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(info['cap1'], cap1)
Add ability to remove subsections from /info Change-Id: Ic881065962cf5f69f7a5b64f6e38d9e6e1f8fd18
2014-05-20 12:33:59 -07:00
Prepare for trunk merge. Refactored some of the swift_init classes into a new module in swift.common, changed some names. Removed the bin test stuff. Fixed some bugs, added some features.
2011-02-11 13:18:19 -06:00
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
class TestFileLikeIter(unittest.TestCase):
def test_iter_file_iter(self):
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
in_iter = [b'abc', b'de', b'fghijk', b'l']
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
chunks = []
for chunk in utils.FileLikeIter(in_iter):
chunks.append(chunk)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(chunks, in_iter)
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
def test_next(self):
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
in_iter = [b'abc', b'de', b'fghijk', b'l']
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
chunks = []
iter_file = utils.FileLikeIter(in_iter)
while True:
try:
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
chunk = next(iter_file)
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
except StopIteration:
break
chunks.append(chunk)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(chunks, in_iter)
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
def test_read(self):
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
in_iter = [b'abc', b'de', b'fghijk', b'l']
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
iter_file = utils.FileLikeIter(in_iter)
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
self.assertEqual(iter_file.read(), b''.join(in_iter))
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
def test_read_with_size(self):
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
in_iter = [b'abc', b'de', b'fghijk', b'l']
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
chunks = []
iter_file = utils.FileLikeIter(in_iter)
while True:
chunk = iter_file.read(2)
if not chunk:
break
self.assertTrue(len(chunk) <= 2)
chunks.append(chunk)
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
self.assertEqual(b''.join(chunks), b''.join(in_iter))
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
def test_read_with_size_zero(self):
# makes little sense, but file supports it, so...
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
self.assertEqual(utils.FileLikeIter(b'abc').read(0), b'')
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
def test_readline(self):
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
in_iter = [b'abc\n', b'd', b'\nef', b'g\nh', b'\nij\n\nk\n',
b'trailing.']
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
lines = []
iter_file = utils.FileLikeIter(in_iter)
while True:
line = iter_file.readline()
if not line:
break
lines.append(line)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
lines,
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
[v if v == b'trailing.' else v + b'\n'
for v in b''.join(in_iter).split(b'\n')])
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
def test_readline2(self):
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
utils.FileLikeIter([b'abc', b'def\n']).readline(4),
b'abcd')
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
def test_readline3(self):
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
utils.FileLikeIter([b'a' * 1111, b'bc\ndef']).readline(),
(b'a' * 1111) + b'bc\n')
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
def test_readline_with_size(self):
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
in_iter = [b'abc\n', b'd', b'\nef', b'g\nh', b'\nij\n\nk\n',
b'trailing.']
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
lines = []
iter_file = utils.FileLikeIter(in_iter)
while True:
line = iter_file.readline(2)
if not line:
break
lines.append(line)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
lines,
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
[b'ab', b'c\n', b'd\n', b'ef', b'g\n', b'h\n', b'ij', b'\n', b'\n',
b'k\n', b'tr', b'ai', b'li', b'ng', b'.'])
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
def test_readlines(self):
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
in_iter = [b'abc\n', b'd', b'\nef', b'g\nh', b'\nij\n\nk\n',
b'trailing.']
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
lines = utils.FileLikeIter(in_iter).readlines()
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
lines,
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
[v if v == b'trailing.' else v + b'\n'
for v in b''.join(in_iter).split(b'\n')])
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
def test_readlines_with_size(self):
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
in_iter = [b'abc\n', b'd', b'\nef', b'g\nh', b'\nij\n\nk\n',
b'trailing.']
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
iter_file = utils.FileLikeIter(in_iter)
lists_of_lines = []
while True:
lines = iter_file.readlines(2)
if not lines:
break
lists_of_lines.append(lines)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
lists_of_lines,
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
[[b'ab'], [b'c\n'], [b'd\n'], [b'ef'], [b'g\n'], [b'h\n'], [b'ij'],
[b'\n', b'\n'], [b'k\n'], [b'tr'], [b'ai'], [b'li'], [b'ng'],
[b'.']])
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
def test_close(self):
Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8
2015-10-19 16:38:24 +02:00
iter_file = utils.FileLikeIter([b'a', b'b', b'c'])
self.assertEqual(next(iter_file), b'a')
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
iter_file.close()
Fix unsuitable assertTrue assertTrue accepts a parameter msg which will be printed when assertion fails, usually msg is a str. This patch fixes unsuitable usage of assertTrue which set msg to bool type True. Change-Id: I731f8ea553c935eba0e112ffded16f41a5ea86c0 Fixes-Bug: #1226374
2013-09-20 23:34:06 +08:00
self.assertTrue(iter_file.closed)
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
self.assertRaises(ValueError, iter_file.next)
self.assertRaises(ValueError, iter_file.read)
self.assertRaises(ValueError, iter_file.readline)
self.assertRaises(ValueError, iter_file.readlines)
# Just make sure repeated close calls don't raise an Exception
iter_file.close()
Fix unsuitable assertTrue assertTrue accepts a parameter msg which will be printed when assertion fails, usually msg is a str. This patch fixes unsuitable usage of assertTrue which set msg to bool type True. Change-Id: I731f8ea553c935eba0e112ffded16f41a5ea86c0 Fixes-Bug: #1226374
2013-09-20 23:34:06 +08:00
self.assertTrue(iter_file.closed)
Added utils.FileLikeIter A really simple version of this was in container sync already, and I needed a more complete version for work I'm doing, and I noticed https://review.openstack.org/#/c/33405/ was also making use of it. So, here's a more full version. If https://review.openstack.org/#/c/33405/ lands before this, I'll update it accordingly. Change-Id: Iba66b6a97f65e312e04fdba273e8f4ad1d3e1594
2013-06-19 15:14:13 +00:00
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
class TestStatsdLogging(unittest.TestCase):
Fix StatsD tests to not use real DNS In common/test_utils.py, TestStatsdLogging had the majority of its test cases calling the real socket.getaddrinfo(), which uses real DNS. This is very slightly slower than using a mock getaddrinfo() when the machine running the tests has functioning DNS, but on a machine with no network connection at all, the tests are excruciatingly slow due to timeouts. This commit mocks things out as appropriate. There's still one user of the real getaddrinfo(), but it's for ::1, so that's just local resolution based on /etc/hosts. Timing numbers for "./.unittests test.unit.common.test_utils:TestStatsdLogging": * network, without this patch: 1.8s * no network, without this patch: 221.2s (ouch) * network, with this patch: 1.1s * no network, with this patch: 1.1s Change-Id: I1a2d6f24fc9bb928894fb1fd8383516250e29e0c
2016-02-23 14:00:34 -08:00
def setUp(self):
def fake_getaddrinfo(host, port, *args):
# this is what a real getaddrinfo('localhost', port,
# socket.AF_INET) returned once
return [(socket.AF_INET, # address family
socket.SOCK_STREAM, # socket type
socket.IPPROTO_TCP, # socket protocol
'', # canonical name,
('127.0.0.1', port)), # socket address
(socket.AF_INET,
socket.SOCK_DGRAM,
socket.IPPROTO_UDP,
'',
('127.0.0.1', port))]
self.real_getaddrinfo = utils.socket.getaddrinfo
self.getaddrinfo_patcher = mock.patch.object(
utils.socket, 'getaddrinfo', fake_getaddrinfo)
self.mock_getaddrinfo = self.getaddrinfo_patcher.start()
self.addCleanup(self.getaddrinfo_patcher.stop)
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
def test_get_logger_statsd_client_not_specified(self):
logger = utils.get_logger({}, 'some-name', log_route='some-route')
# white-box construction validation
Replace assertEqual(None, *) with assertIsNone in tests As swift no longer supports Python 2.6, replace assertEqual(None, *) with assertIsNone in tests to have more clear messages in case of failure. Change-Id: I94af3e8156ef40465d4f7a2cb79fb99fc7bbda56 Closes-Bug: #1280522
2016-02-16 23:44:27 +08:00
self.assertIsNone(logger.logger.statsd_client)
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
def test_get_logger_statsd_client_defaults(self):
logger = utils.get_logger({'log_statsd_host': 'some.host.com'},
'some-name', log_route='some-route')
# white-box construction validation
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(isinstance(logger.logger.statsd_client,
utils.StatsdClient))
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
self.assertEqual(logger.logger.statsd_client._host, 'some.host.com')
self.assertEqual(logger.logger.statsd_client._port, 8125)
self.assertEqual(logger.logger.statsd_client._prefix, 'some-name.')
self.assertEqual(logger.logger.statsd_client._default_sample_rate, 1)
logger.set_statsd_prefix('some-name.more-specific')
self.assertEqual(logger.logger.statsd_client._prefix,
'some-name.more-specific.')
logger.set_statsd_prefix('')
self.assertEqual(logger.logger.statsd_client._prefix, '')
def test_get_logger_statsd_client_non_defaults(self):
logger = utils.get_logger({
'log_statsd_host': 'another.host.com',
Make statsd sample rate behave better. As Dieter pointed out in bug 1090495 (https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics can vary wildly between StatsD metrics. This patch implements a partial solution by reducing the sample_rate used for known high-volume metrics (operational experience will need to inform this over time) and introducing a new tunable, log_statsd_sample_rate_factor which is multiplied by the sample_rate for every statsd stat. This tunable can be used to reduce StatsD traffic proportionally for all metrics and is intended to replace log_statsd_default_sample_rate, which is left alone for backward-compatibility, should anyone be using it. This patch also includes a drive-by fix for log_udp_port which wasn't being converted to an int (I didn't verify that actually causes trouble in SysLogHandler(), but it's definitely an improvement regardles). Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
'log_statsd_port': '9876',
'log_statsd_default_sample_rate': '0.75',
'log_statsd_sample_rate_factor': '0.81',
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
'log_statsd_metric_prefix': 'tomato.sauce',
}, 'some-name', log_route='some-route')
self.assertEqual(logger.logger.statsd_client._prefix,
'tomato.sauce.some-name.')
logger.set_statsd_prefix('some-name.more-specific')
self.assertEqual(logger.logger.statsd_client._prefix,
'tomato.sauce.some-name.more-specific.')
logger.set_statsd_prefix('')
self.assertEqual(logger.logger.statsd_client._prefix, 'tomato.sauce.')
self.assertEqual(logger.logger.statsd_client._host, 'another.host.com')
self.assertEqual(logger.logger.statsd_client._port, 9876)
Honor sample_rate in StatsD logging. It's there to let administrators turn down the barrage of stats data that StatsD must cope with, but it wasn't actually honored. Worse, if the sample rate was set to e.g. 0.2, the stats would all be multiplied by its inverse, e.g. 2. This patch actually drops packets when sample_rate < 1, so you get correct measurements. Fortunately, the default sample rate is 1 (i.e. drop nothing), and multiplying by 1/1 doesn't change anything, so stats with the default sample rate of 1.0 are, and have been, just fine. Fixes bug 1065643. Also, make the two touched files compliant with pep8 v1.3.3. Change-Id: I66663144009ae4c9ee96f6a111745d8f5d2f5ca3
2012-10-11 14:04:02 -07:00
self.assertEqual(logger.logger.statsd_client._default_sample_rate,
0.75)
Make statsd sample rate behave better. As Dieter pointed out in bug 1090495 (https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics can vary wildly between StatsD metrics. This patch implements a partial solution by reducing the sample_rate used for known high-volume metrics (operational experience will need to inform this over time) and introducing a new tunable, log_statsd_sample_rate_factor which is multiplied by the sample_rate for every statsd stat. This tunable can be used to reduce StatsD traffic proportionally for all metrics and is intended to replace log_statsd_default_sample_rate, which is left alone for backward-compatibility, should anyone be using it. This patch also includes a drive-by fix for log_udp_port which wasn't being converted to an int (I didn't verify that actually causes trouble in SysLogHandler(), but it's definitely an improvement regardles). Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
self.assertEqual(logger.logger.statsd_client._sample_rate_factor,
0.81)
Honor sample_rate in StatsD logging. It's there to let administrators turn down the barrage of stats data that StatsD must cope with, but it wasn't actually honored. Worse, if the sample rate was set to e.g. 0.2, the stats would all be multiplied by its inverse, e.g. 2. This patch actually drops packets when sample_rate < 1, so you get correct measurements. Fortunately, the default sample rate is 1 (i.e. drop nothing), and multiplying by 1/1 doesn't change anything, so stats with the default sample rate of 1.0 are, and have been, just fine. Fixes bug 1065643. Also, make the two touched files compliant with pep8 v1.3.3. Change-Id: I66663144009ae4c9ee96f6a111745d8f5d2f5ca3
2012-10-11 14:04:02 -07:00
Allow IPv6 addresses/hostnames in StatsD target The log_statsd_host value can now be an IPv6 address or a hostname which only resolves to an IPv6 address. In both cases, the new behavior is to use an AF_INET6 socket on which .sendto() is called with the originally-configured hostname (or IP). This means the Swift process is not caching a DNS resolution for the lifetime of the process (a good thing). If a hostname resolves to both an IPv6 or IPv4 address, an AF_INET socket is used (i.e. only the IPv4 address will receive the UDP packet). The old behavior is preserved: any invalid IP address literals and failures in DNS resolution or actual StatsD packet sending do not halt the process or bubble up; they are caught, logged, and otherwise ignored. Change-Id: Ibddddcf140e2e69b08edf3feed3e9a5fa17307cf
2016-01-21 11:18:18 -08:00
def test_ipv4_or_ipv6_hostname_defaults_to_ipv4(self):
def stub_getaddrinfo_both_ipv4_and_ipv6(host, port, family, *rest):
if family == socket.AF_INET:
return [(socket.AF_INET, 'blah', 'blah', 'blah',
('127.0.0.1', int(port)))]
elif family == socket.AF_INET6:
# Implemented so an incorrectly ordered implementation (IPv6
# then IPv4) would realistically fail.
return [(socket.AF_INET6, 'blah', 'blah', 'blah',
('::1', int(port), 0, 0))]
with mock.patch.object(utils.socket, 'getaddrinfo',
new=stub_getaddrinfo_both_ipv4_and_ipv6):
logger = utils.get_logger({
'log_statsd_host': 'localhost',
'log_statsd_port': '9876',
}, 'some-name', log_route='some-route')
statsd_client = logger.logger.statsd_client
self.assertEqual(statsd_client._sock_family, socket.AF_INET)
self.assertEqual(statsd_client._target, ('localhost', 9876))
got_sock = statsd_client._open_socket()
self.assertEqual(got_sock.family, socket.AF_INET)
def test_ipv4_instantiation_and_socket_creation(self):
logger = utils.get_logger({
'log_statsd_host': '127.0.0.1',
'log_statsd_port': '9876',
}, 'some-name', log_route='some-route')
statsd_client = logger.logger.statsd_client
self.assertEqual(statsd_client._sock_family, socket.AF_INET)
self.assertEqual(statsd_client._target, ('127.0.0.1', 9876))
got_sock = statsd_client._open_socket()
self.assertEqual(got_sock.family, socket.AF_INET)
def test_ipv6_instantiation_and_socket_creation(self):
# We have to check the given hostname or IP for IPv4/IPv6 on logger
# instantiation so we don't call getaddrinfo() too often and don't have
# to call bind() on our socket to detect IPv4/IPv6 on every send.
Fix StatsD tests to not use real DNS In common/test_utils.py, TestStatsdLogging had the majority of its test cases calling the real socket.getaddrinfo(), which uses real DNS. This is very slightly slower than using a mock getaddrinfo() when the machine running the tests has functioning DNS, but on a machine with no network connection at all, the tests are excruciatingly slow due to timeouts. This commit mocks things out as appropriate. There's still one user of the real getaddrinfo(), but it's for ::1, so that's just local resolution based on /etc/hosts. Timing numbers for "./.unittests test.unit.common.test_utils:TestStatsdLogging": * network, without this patch: 1.8s * no network, without this patch: 221.2s (ouch) * network, with this patch: 1.1s * no network, with this patch: 1.1s Change-Id: I1a2d6f24fc9bb928894fb1fd8383516250e29e0c
2016-02-23 14:00:34 -08:00
#
# This test uses the real getaddrinfo, so we patch over the mock to
# put the real one back. If we just stop the mock, then
# unittest.exit() blows up, but stacking real-fake-real works okay.
with mock.patch.object(utils.socket, 'getaddrinfo',
self.real_getaddrinfo):
logger = utils.get_logger({
'log_statsd_host': '::1',
'log_statsd_port': '9876',
}, 'some-name', log_route='some-route')
Allow IPv6 addresses/hostnames in StatsD target The log_statsd_host value can now be an IPv6 address or a hostname which only resolves to an IPv6 address. In both cases, the new behavior is to use an AF_INET6 socket on which .sendto() is called with the originally-configured hostname (or IP). This means the Swift process is not caching a DNS resolution for the lifetime of the process (a good thing). If a hostname resolves to both an IPv6 or IPv4 address, an AF_INET socket is used (i.e. only the IPv4 address will receive the UDP packet). The old behavior is preserved: any invalid IP address literals and failures in DNS resolution or actual StatsD packet sending do not halt the process or bubble up; they are caught, logged, and otherwise ignored. Change-Id: Ibddddcf140e2e69b08edf3feed3e9a5fa17307cf
2016-01-21 11:18:18 -08:00
statsd_client = logger.logger.statsd_client
self.assertEqual(statsd_client._sock_family, socket.AF_INET6)
self.assertEqual(statsd_client._target, ('::1', 9876, 0, 0))
got_sock = statsd_client._open_socket()
self.assertEqual(got_sock.family, socket.AF_INET6)
def test_bad_hostname_instantiation(self):
Fix StatsD tests to not use real DNS In common/test_utils.py, TestStatsdLogging had the majority of its test cases calling the real socket.getaddrinfo(), which uses real DNS. This is very slightly slower than using a mock getaddrinfo() when the machine running the tests has functioning DNS, but on a machine with no network connection at all, the tests are excruciatingly slow due to timeouts. This commit mocks things out as appropriate. There's still one user of the real getaddrinfo(), but it's for ::1, so that's just local resolution based on /etc/hosts. Timing numbers for "./.unittests test.unit.common.test_utils:TestStatsdLogging": * network, without this patch: 1.8s * no network, without this patch: 221.2s (ouch) * network, with this patch: 1.1s * no network, with this patch: 1.1s Change-Id: I1a2d6f24fc9bb928894fb1fd8383516250e29e0c
2016-02-23 14:00:34 -08:00
with mock.patch.object(utils.socket, 'getaddrinfo',
side_effect=utils.socket.gaierror("whoops")):
logger = utils.get_logger({
'log_statsd_host': 'i-am-not-a-hostname-or-ip',
'log_statsd_port': '9876',
}, 'some-name', log_route='some-route')
Allow IPv6 addresses/hostnames in StatsD target The log_statsd_host value can now be an IPv6 address or a hostname which only resolves to an IPv6 address. In both cases, the new behavior is to use an AF_INET6 socket on which .sendto() is called with the originally-configured hostname (or IP). This means the Swift process is not caching a DNS resolution for the lifetime of the process (a good thing). If a hostname resolves to both an IPv6 or IPv4 address, an AF_INET socket is used (i.e. only the IPv4 address will receive the UDP packet). The old behavior is preserved: any invalid IP address literals and failures in DNS resolution or actual StatsD packet sending do not halt the process or bubble up; they are caught, logged, and otherwise ignored. Change-Id: Ibddddcf140e2e69b08edf3feed3e9a5fa17307cf
2016-01-21 11:18:18 -08:00
statsd_client = logger.logger.statsd_client
self.assertEqual(statsd_client._sock_family, socket.AF_INET)
self.assertEqual(statsd_client._target,
('i-am-not-a-hostname-or-ip', 9876))
got_sock = statsd_client._open_socket()
self.assertEqual(got_sock.family, socket.AF_INET)
# Maybe the DNS server gets fixed in a bit and it starts working... or
# maybe the DNS record hadn't propagated yet. In any case, failed
# statsd sends will warn in the logs until the DNS failure or invalid
# IP address in the configuration is fixed.
def test_sending_ipv6(self):
Fix StatsD tests to not use real DNS In common/test_utils.py, TestStatsdLogging had the majority of its test cases calling the real socket.getaddrinfo(), which uses real DNS. This is very slightly slower than using a mock getaddrinfo() when the machine running the tests has functioning DNS, but on a machine with no network connection at all, the tests are excruciatingly slow due to timeouts. This commit mocks things out as appropriate. There's still one user of the real getaddrinfo(), but it's for ::1, so that's just local resolution based on /etc/hosts. Timing numbers for "./.unittests test.unit.common.test_utils:TestStatsdLogging": * network, without this patch: 1.8s * no network, without this patch: 221.2s (ouch) * network, with this patch: 1.1s * no network, with this patch: 1.1s Change-Id: I1a2d6f24fc9bb928894fb1fd8383516250e29e0c
2016-02-23 14:00:34 -08:00
def fake_getaddrinfo(host, port, *args):
# this is what a real getaddrinfo('::1', port,
# socket.AF_INET6) returned once
return [(socket.AF_INET6,
socket.SOCK_STREAM,
socket.IPPROTO_TCP,
'', ('::1', port, 0, 0)),
(socket.AF_INET6,
socket.SOCK_DGRAM,
socket.IPPROTO_UDP,
'',
('::1', port, 0, 0))]
with mock.patch.object(utils.socket, 'getaddrinfo', fake_getaddrinfo):
logger = utils.get_logger({
'log_statsd_host': '::1',
'log_statsd_port': '9876',
}, 'some-name', log_route='some-route')
Allow IPv6 addresses/hostnames in StatsD target The log_statsd_host value can now be an IPv6 address or a hostname which only resolves to an IPv6 address. In both cases, the new behavior is to use an AF_INET6 socket on which .sendto() is called with the originally-configured hostname (or IP). This means the Swift process is not caching a DNS resolution for the lifetime of the process (a good thing). If a hostname resolves to both an IPv6 or IPv4 address, an AF_INET socket is used (i.e. only the IPv4 address will receive the UDP packet). The old behavior is preserved: any invalid IP address literals and failures in DNS resolution or actual StatsD packet sending do not halt the process or bubble up; they are caught, logged, and otherwise ignored. Change-Id: Ibddddcf140e2e69b08edf3feed3e9a5fa17307cf
2016-01-21 11:18:18 -08:00
statsd_client = logger.logger.statsd_client
fl = FakeLogger()
statsd_client.logger = fl
mock_socket = MockUdpSocket()
statsd_client._open_socket = lambda *_: mock_socket
logger.increment('tunafish')
self.assertEqual(fl.get_lines_for_level('warning'), [])
self.assertEqual(mock_socket.sent,
[(b'some-name.tunafish:1|c', ('::1', 9876, 0, 0))])
Catch permissions errors when writing StatsD packets Closes-Bug: #1183152 Change-Id: I4b2c6e947241c987779a385fdff270d037470a57
2014-08-10 23:50:17 -07:00
def test_no_exception_when_cant_send_udp_packet(self):
logger = utils.get_logger({'log_statsd_host': 'some.host.com'})
statsd_client = logger.logger.statsd_client
fl = FakeLogger()
statsd_client.logger = fl
mock_socket = MockUdpSocket(sendto_errno=errno.EPERM)
statsd_client._open_socket = lambda *_: mock_socket
logger.increment('tunafish')
expected = ["Error sending UDP message to ('some.host.com', 8125): "
"[Errno 1] test errno 1"]
self.assertEqual(fl.get_lines_for_level('warning'), expected)
Honor sample_rate in StatsD logging. It's there to let administrators turn down the barrage of stats data that StatsD must cope with, but it wasn't actually honored. Worse, if the sample rate was set to e.g. 0.2, the stats would all be multiplied by its inverse, e.g. 2. This patch actually drops packets when sample_rate < 1, so you get correct measurements. Fortunately, the default sample rate is 1 (i.e. drop nothing), and multiplying by 1/1 doesn't change anything, so stats with the default sample rate of 1.0 are, and have been, just fine. Fixes bug 1065643. Also, make the two touched files compliant with pep8 v1.3.3. Change-Id: I66663144009ae4c9ee96f6a111745d8f5d2f5ca3
2012-10-11 14:04:02 -07:00
def test_sample_rates(self):
logger = utils.get_logger({'log_statsd_host': 'some.host.com'})
mock_socket = MockUdpSocket()
# encapsulation? what's that?
statsd_client = logger.logger.statsd_client
self.assertTrue(statsd_client.random is random.random)
statsd_client._open_socket = lambda *_: mock_socket
statsd_client.random = lambda: 0.50001
logger.increment('tribbles', sample_rate=0.5)
self.assertEqual(len(mock_socket.sent), 0)
statsd_client.random = lambda: 0.49999
logger.increment('tribbles', sample_rate=0.5)
self.assertEqual(len(mock_socket.sent), 1)
payload = mock_socket.sent[0][0]
Port swift.common.utils.StatsdClient to Python 3 * StatsdClient._send(): on Python 3, encode parts to UTF-8 and replace '|' with b'|' to join parts. * timing_stats(): replace func.func_name with func.__name__. The func_name attribute of functions was removed on Python 3, whereas the __name__ attribute is available on Python 2 and Python 3. * Fix unit tests to use bytes Change-Id: Ic279c9b54e91aabcc52587eed7758e268ffb155e
2015-10-19 16:19:28 +02:00
self.assertTrue(payload.endswith(b"|@0.5"))
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
Make statsd sample rate behave better. As Dieter pointed out in bug 1090495 (https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics can vary wildly between StatsD metrics. This patch implements a partial solution by reducing the sample_rate used for known high-volume metrics (operational experience will need to inform this over time) and introducing a new tunable, log_statsd_sample_rate_factor which is multiplied by the sample_rate for every statsd stat. This tunable can be used to reduce StatsD traffic proportionally for all metrics and is intended to replace log_statsd_default_sample_rate, which is left alone for backward-compatibility, should anyone be using it. This patch also includes a drive-by fix for log_udp_port which wasn't being converted to an int (I didn't verify that actually causes trouble in SysLogHandler(), but it's definitely an improvement regardles). Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
def test_sample_rates_with_sample_rate_factor(self):
logger = utils.get_logger({
'log_statsd_host': 'some.host.com',
'log_statsd_default_sample_rate': '0.82',
'log_statsd_sample_rate_factor': '0.91',
})
effective_sample_rate = 0.82 * 0.91
mock_socket = MockUdpSocket()
# encapsulation? what's that?
statsd_client = logger.logger.statsd_client
self.assertTrue(statsd_client.random is random.random)
statsd_client._open_socket = lambda *_: mock_socket
statsd_client.random = lambda: effective_sample_rate + 0.001
logger.increment('tribbles')
self.assertEqual(len(mock_socket.sent), 0)
statsd_client.random = lambda: effective_sample_rate - 0.001
logger.increment('tribbles')
self.assertEqual(len(mock_socket.sent), 1)
payload = mock_socket.sent[0][0]
Port swift.common.utils.StatsdClient to Python 3 * StatsdClient._send(): on Python 3, encode parts to UTF-8 and replace '|' with b'|' to join parts. * timing_stats(): replace func.func_name with func.__name__. The func_name attribute of functions was removed on Python 3, whereas the __name__ attribute is available on Python 2 and Python 3. * Fix unit tests to use bytes Change-Id: Ic279c9b54e91aabcc52587eed7758e268ffb155e
2015-10-19 16:19:28 +02:00
suffix = "|@%s" % effective_sample_rate
if six.PY3:
suffix = suffix.encode('utf-8')
self.assertTrue(payload.endswith(suffix), payload)
Make statsd sample rate behave better. As Dieter pointed out in bug 1090495 (https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics can vary wildly between StatsD metrics. This patch implements a partial solution by reducing the sample_rate used for known high-volume metrics (operational experience will need to inform this over time) and introducing a new tunable, log_statsd_sample_rate_factor which is multiplied by the sample_rate for every statsd stat. This tunable can be used to reduce StatsD traffic proportionally for all metrics and is intended to replace log_statsd_default_sample_rate, which is left alone for backward-compatibility, should anyone be using it. This patch also includes a drive-by fix for log_udp_port which wasn't being converted to an int (I didn't verify that actually causes trouble in SysLogHandler(), but it's definitely an improvement regardles). Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
effective_sample_rate = 0.587 * 0.91
statsd_client.random = lambda: effective_sample_rate - 0.001
logger.increment('tribbles', sample_rate=0.587)
self.assertEqual(len(mock_socket.sent), 2)
payload = mock_socket.sent[1][0]
Port swift.common.utils.StatsdClient to Python 3 * StatsdClient._send(): on Python 3, encode parts to UTF-8 and replace '|' with b'|' to join parts. * timing_stats(): replace func.func_name with func.__name__. The func_name attribute of functions was removed on Python 3, whereas the __name__ attribute is available on Python 2 and Python 3. * Fix unit tests to use bytes Change-Id: Ic279c9b54e91aabcc52587eed7758e268ffb155e
2015-10-19 16:19:28 +02:00
suffix = "|@%s" % effective_sample_rate
if six.PY3:
suffix = suffix.encode('utf-8')
self.assertTrue(payload.endswith(suffix), payload)
Make statsd sample rate behave better. As Dieter pointed out in bug 1090495 (https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics can vary wildly between StatsD metrics. This patch implements a partial solution by reducing the sample_rate used for known high-volume metrics (operational experience will need to inform this over time) and introducing a new tunable, log_statsd_sample_rate_factor which is multiplied by the sample_rate for every statsd stat. This tunable can be used to reduce StatsD traffic proportionally for all metrics and is intended to replace log_statsd_default_sample_rate, which is left alone for backward-compatibility, should anyone be using it. This patch also includes a drive-by fix for log_udp_port which wasn't being converted to an int (I didn't verify that actually causes trouble in SysLogHandler(), but it's definitely an improvement regardles). Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
statsd timing refactor Change-Id: I99d9ddfbcad0f88e75c49235c8317ea97237d4e4
2012-11-06 02:44:43 -08:00
def test_timing_stats(self):
class MockController(object):
def __init__(self, status):
self.status = status
self.logger = self
self.args = ()
self.called = 'UNKNOWN'
def timing_since(self, *args):
self.called = 'timing'
self.args = args
Make statsd sample rate behave better. As Dieter pointed out in bug 1090495 (https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics can vary wildly between StatsD metrics. This patch implements a partial solution by reducing the sample_rate used for known high-volume metrics (operational experience will need to inform this over time) and introducing a new tunable, log_statsd_sample_rate_factor which is multiplied by the sample_rate for every statsd stat. This tunable can be used to reduce StatsD traffic proportionally for all metrics and is intended to replace log_statsd_default_sample_rate, which is left alone for backward-compatibility, should anyone be using it. This patch also includes a drive-by fix for log_udp_port which wasn't being converted to an int (I didn't verify that actually causes trouble in SysLogHandler(), but it's definitely an improvement regardles). Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
@utils.timing_stats()
statsd timing refactor Change-Id: I99d9ddfbcad0f88e75c49235c8317ea97237d4e4
2012-11-06 02:44:43 -08:00
def METHOD(controller):
return Response(status=controller.status)
mock_controller = MockController(200)
METHOD(mock_controller)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(mock_controller.called, 'timing')
self.assertEqual(len(mock_controller.args), 2)
self.assertEqual(mock_controller.args[0], 'METHOD.timing')
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(mock_controller.args[1] > 0)
statsd timing refactor Change-Id: I99d9ddfbcad0f88e75c49235c8317ea97237d4e4
2012-11-06 02:44:43 -08:00
mock_controller = MockController(404)
METHOD(mock_controller)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(mock_controller.args), 2)
self.assertEqual(mock_controller.called, 'timing')
self.assertEqual(mock_controller.args[0], 'METHOD.timing')
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(mock_controller.args[1] > 0)
statsd timing refactor Change-Id: I99d9ddfbcad0f88e75c49235c8317ea97237d4e4
2012-11-06 02:44:43 -08:00
Don't count 412 or 416 as errors in stats The backend HTTP servers emit StatsD metrics of the form <server>.<method>.timing and <server>.<method>.errors.timing (e.g. object-server.GET.timing). Whether something counts as an error or not is based on the response's HTTP status code. Prior to this commit, "success" was 2xx, 3xx, or 404, while everything else was considered "error". This adds 412 and 416 to the "success" set. Like 404, these status codes indicate that we got the request and processed it without error, but the response was "no". They shouldn't be in the same category as statuses like 507 that indicate something stopped the request from being processed. Change-Id: I5582a51cf6f64aa22c890da01aaaa077f3a54202
2014-06-06 14:29:44 -07:00
mock_controller = MockController(412)
METHOD(mock_controller)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(mock_controller.args), 2)
self.assertEqual(mock_controller.called, 'timing')
self.assertEqual(mock_controller.args[0], 'METHOD.timing')
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(mock_controller.args[1] > 0)
Don't count 412 or 416 as errors in stats The backend HTTP servers emit StatsD metrics of the form <server>.<method>.timing and <server>.<method>.errors.timing (e.g. object-server.GET.timing). Whether something counts as an error or not is based on the response's HTTP status code. Prior to this commit, "success" was 2xx, 3xx, or 404, while everything else was considered "error". This adds 412 and 416 to the "success" set. Like 404, these status codes indicate that we got the request and processed it without error, but the response was "no". They shouldn't be in the same category as statuses like 507 that indicate something stopped the request from being processed. Change-Id: I5582a51cf6f64aa22c890da01aaaa077f3a54202
2014-06-06 14:29:44 -07:00
mock_controller = MockController(416)
METHOD(mock_controller)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(mock_controller.args), 2)
self.assertEqual(mock_controller.called, 'timing')
self.assertEqual(mock_controller.args[0], 'METHOD.timing')
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(mock_controller.args[1] > 0)
Don't count 412 or 416 as errors in stats The backend HTTP servers emit StatsD metrics of the form <server>.<method>.timing and <server>.<method>.errors.timing (e.g. object-server.GET.timing). Whether something counts as an error or not is based on the response's HTTP status code. Prior to this commit, "success" was 2xx, 3xx, or 404, while everything else was considered "error". This adds 412 and 416 to the "success" set. Like 404, these status codes indicate that we got the request and processed it without error, but the response was "no". They shouldn't be in the same category as statuses like 507 that indicate something stopped the request from being processed. Change-Id: I5582a51cf6f64aa22c890da01aaaa077f3a54202
2014-06-06 14:29:44 -07:00
statsd timing refactor Change-Id: I99d9ddfbcad0f88e75c49235c8317ea97237d4e4
2012-11-06 02:44:43 -08:00
mock_controller = MockController(401)
METHOD(mock_controller)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(mock_controller.args), 2)
self.assertEqual(mock_controller.called, 'timing')
self.assertEqual(mock_controller.args[0], 'METHOD.errors.timing')
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(mock_controller.args[1] > 0)
statsd timing refactor Change-Id: I99d9ddfbcad0f88e75c49235c8317ea97237d4e4
2012-11-06 02:44:43 -08:00
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
Fix ValueError from handoff logging. If you have StatsD logging turned on, then the iterator returned by Ring.get_part_nodes() will emit StatsD packets when it yields a handoff node. That network IO may make eventlet trampoline to another greenthread before returning from next(). Now, if that other greenthread tries to call next() on that same iterator, it blows up with a ValueError. Any socket IO inside a generator's next() method can cause this. It's easiest to reproduce with StatsD logging turned on, but logging to syslog can trigger it too. You can see this happen sometimes in the proxy's make_requests method if two of the primary nodes are down. Greenthread A goes into next() to get a handoff node, then sends a StatsD packet, and so eventlet trampolines to Greenthread B. Now, Greenthread B also calls next() to get a handoff node, and dies with a ValueError. This commit wraps up concurrently-accessed iter_nodes() iterators in a new thing called a GreenthreadSafeIterator that serializes access. Bug 1180110 Change-Id: I8fe13d7295c056a2cab9e084f5966078a49bdc13
2013-05-14 13:04:01 -07:00
class UnsafeXrange(object):
"""
Like xrange(limit), but with extra context switching to screw things up.
"""
def __init__(self, upper_bound):
self.current = 0
self.concurrent_calls = 0
self.upper_bound = upper_bound
Fix TestGreenThreadSafeIterator test Change-Id: Ic121ae5c99b4e5c5e3d1efc432b55b762d50f3fb Closes-Bug: 1190244
2014-03-18 21:17:52 +00:00
self.concurrent_call = False
Fix ValueError from handoff logging. If you have StatsD logging turned on, then the iterator returned by Ring.get_part_nodes() will emit StatsD packets when it yields a handoff node. That network IO may make eventlet trampoline to another greenthread before returning from next(). Now, if that other greenthread tries to call next() on that same iterator, it blows up with a ValueError. Any socket IO inside a generator's next() method can cause this. It's easiest to reproduce with StatsD logging turned on, but logging to syslog can trigger it too. You can see this happen sometimes in the proxy's make_requests method if two of the primary nodes are down. Greenthread A goes into next() to get a handoff node, then sends a StatsD packet, and so eventlet trampolines to Greenthread B. Now, Greenthread B also calls next() to get a handoff node, and dies with a ValueError. This commit wraps up concurrently-accessed iter_nodes() iterators in a new thing called a GreenthreadSafeIterator that serializes access. Bug 1180110 Change-Id: I8fe13d7295c056a2cab9e084f5966078a49bdc13
2013-05-14 13:04:01 -07:00
def __iter__(self):
return self
def next(self):
if self.concurrent_calls > 0:
Fix TestGreenThreadSafeIterator test Change-Id: Ic121ae5c99b4e5c5e3d1efc432b55b762d50f3fb Closes-Bug: 1190244
2014-03-18 21:17:52 +00:00
self.concurrent_call = True
Fix ValueError from handoff logging. If you have StatsD logging turned on, then the iterator returned by Ring.get_part_nodes() will emit StatsD packets when it yields a handoff node. That network IO may make eventlet trampoline to another greenthread before returning from next(). Now, if that other greenthread tries to call next() on that same iterator, it blows up with a ValueError. Any socket IO inside a generator's next() method can cause this. It's easiest to reproduce with StatsD logging turned on, but logging to syslog can trigger it too. You can see this happen sometimes in the proxy's make_requests method if two of the primary nodes are down. Greenthread A goes into next() to get a handoff node, then sends a StatsD packet, and so eventlet trampolines to Greenthread B. Now, Greenthread B also calls next() to get a handoff node, and dies with a ValueError. This commit wraps up concurrently-accessed iter_nodes() iterators in a new thing called a GreenthreadSafeIterator that serializes access. Bug 1180110 Change-Id: I8fe13d7295c056a2cab9e084f5966078a49bdc13
2013-05-14 13:04:01 -07:00
self.concurrent_calls += 1
try:
if self.current >= self.upper_bound:
raise StopIteration
else:
val = self.current
self.current += 1
eventlet.sleep() # yield control
return val
finally:
self.concurrent_calls -= 1
Add __next__() methods to utils iterators for py3 On Python 3, next(obj) calls obj.__next__(), not obj.next(). Add an alias from __next__() to next() to be compatible with Python 2 and Python 3. Change-Id: Ida104d3bd7cdba557e523f18df43d56847060054
2015-10-19 17:22:24 +02:00
__next__ = next
Fix ValueError from handoff logging. If you have StatsD logging turned on, then the iterator returned by Ring.get_part_nodes() will emit StatsD packets when it yields a handoff node. That network IO may make eventlet trampoline to another greenthread before returning from next(). Now, if that other greenthread tries to call next() on that same iterator, it blows up with a ValueError. Any socket IO inside a generator's next() method can cause this. It's easiest to reproduce with StatsD logging turned on, but logging to syslog can trigger it too. You can see this happen sometimes in the proxy's make_requests method if two of the primary nodes are down. Greenthread A goes into next() to get a handoff node, then sends a StatsD packet, and so eventlet trampolines to Greenthread B. Now, Greenthread B also calls next() to get a handoff node, and dies with a ValueError. This commit wraps up concurrently-accessed iter_nodes() iterators in a new thing called a GreenthreadSafeIterator that serializes access. Bug 1180110 Change-Id: I8fe13d7295c056a2cab9e084f5966078a49bdc13
2013-05-14 13:04:01 -07:00
Local read affinity for GET/HEAD requests. Now you can configure the proxy server to read from "local" primary nodes first, where "local" is governed by the newly-introduced "read_affinity" setting in the proxy config. This is desirable when the network links between regions/zones are of varying capacities; in such a case, it's a good idea to prefer fetching data from closer backends. The new setting looks like rN[zM]=P, where N is the region number, M is the optional zone number, and P is the priority. Multiple values can be specified by separating them with commas. The priority for nodes that don't match anything is a very large number, so they'll sort last. This only affects the ordering of the primary nodes; it doesn't affect handoffs at all. Further, while the primary nodes are reordered for all requests, it only matters for GET/HEAD requests since handling the other verbs ends up making concurrent requests to *all* the primary nodes, so ordering is irrelevant. Note that the default proxy config does not have this setting turned on, so the default configuration's behavior is unaffected. blueprint multi-region Change-Id: Iea4cd367ed37fe5ee69b63234541d358d29963a4
2013-06-06 18:01:35 -07:00
class TestAffinityKeyFunction(unittest.TestCase):
def setUp(self):
self.nodes = [dict(id=0, region=1, zone=1),
dict(id=1, region=1, zone=2),
dict(id=2, region=2, zone=1),
dict(id=3, region=2, zone=2),
dict(id=4, region=3, zone=1),
dict(id=5, region=3, zone=2),
dict(id=6, region=4, zone=0),
dict(id=7, region=4, zone=1)]
def test_single_region(self):
keyfn = utils.affinity_key_function("r3=1")
ids = [n['id'] for n in sorted(self.nodes, key=keyfn)]
self.assertEqual([4, 5, 0, 1, 2, 3, 6, 7], ids)
def test_bogus_value(self):
self.assertRaises(ValueError,
utils.affinity_key_function, "r3")
self.assertRaises(ValueError,
utils.affinity_key_function, "r3=elephant")
def test_empty_value(self):
# Empty's okay, it just means no preference
keyfn = utils.affinity_key_function("")
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(callable(keyfn))
Local read affinity for GET/HEAD requests. Now you can configure the proxy server to read from "local" primary nodes first, where "local" is governed by the newly-introduced "read_affinity" setting in the proxy config. This is desirable when the network links between regions/zones are of varying capacities; in such a case, it's a good idea to prefer fetching data from closer backends. The new setting looks like rN[zM]=P, where N is the region number, M is the optional zone number, and P is the priority. Multiple values can be specified by separating them with commas. The priority for nodes that don't match anything is a very large number, so they'll sort last. This only affects the ordering of the primary nodes; it doesn't affect handoffs at all. Further, while the primary nodes are reordered for all requests, it only matters for GET/HEAD requests since handling the other verbs ends up making concurrent requests to *all* the primary nodes, so ordering is irrelevant. Note that the default proxy config does not have this setting turned on, so the default configuration's behavior is unaffected. blueprint multi-region Change-Id: Iea4cd367ed37fe5ee69b63234541d358d29963a4
2013-06-06 18:01:35 -07:00
ids = [n['id'] for n in sorted(self.nodes, key=keyfn)]
self.assertEqual([0, 1, 2, 3, 4, 5, 6, 7], ids)
def test_all_whitespace_value(self):
# Empty's okay, it just means no preference
keyfn = utils.affinity_key_function(" \n")
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(callable(keyfn))
Local read affinity for GET/HEAD requests. Now you can configure the proxy server to read from "local" primary nodes first, where "local" is governed by the newly-introduced "read_affinity" setting in the proxy config. This is desirable when the network links between regions/zones are of varying capacities; in such a case, it's a good idea to prefer fetching data from closer backends. The new setting looks like rN[zM]=P, where N is the region number, M is the optional zone number, and P is the priority. Multiple values can be specified by separating them with commas. The priority for nodes that don't match anything is a very large number, so they'll sort last. This only affects the ordering of the primary nodes; it doesn't affect handoffs at all. Further, while the primary nodes are reordered for all requests, it only matters for GET/HEAD requests since handling the other verbs ends up making concurrent requests to *all* the primary nodes, so ordering is irrelevant. Note that the default proxy config does not have this setting turned on, so the default configuration's behavior is unaffected. blueprint multi-region Change-Id: Iea4cd367ed37fe5ee69b63234541d358d29963a4
2013-06-06 18:01:35 -07:00
ids = [n['id'] for n in sorted(self.nodes, key=keyfn)]
self.assertEqual([0, 1, 2, 3, 4, 5, 6, 7], ids)
def test_with_zone_zero(self):
keyfn = utils.affinity_key_function("r4z0=1")
ids = [n['id'] for n in sorted(self.nodes, key=keyfn)]
self.assertEqual([6, 0, 1, 2, 3, 4, 5, 7], ids)
def test_multiple(self):
keyfn = utils.affinity_key_function("r1=100, r4=200, r3z1=1")
ids = [n['id'] for n in sorted(self.nodes, key=keyfn)]
self.assertEqual([4, 0, 1, 6, 7, 2, 3, 5], ids)
def test_more_specific_after_less_specific(self):
keyfn = utils.affinity_key_function("r2=100, r2z2=50")
ids = [n['id'] for n in sorted(self.nodes, key=keyfn)]
self.assertEqual([3, 2, 0, 1, 4, 5, 6, 7], ids)
Local write affinity for object PUT requests. The proxy can now be configured to prefer local object servers for PUT requests, where "local" is governed by the "write_affinity". The "write_affinity_node_count" setting controls how many local object servers to try before giving up and going on to remote ones. I chose to simply re-order the object servers instead of filtering out nonlocal ones so that, if all of the local ones are down, clients can still get successful responses (just slower). The goal is to trade availability for throughput. By writing to local object servers across fast LAN links, clients get better throughput than if the object servers were far away over slow WAN links. The downside, of course, is that data availability (not durability) may suffer when drives fail. The default configuration has no write affinity in it, so the default behavior is unchanged. Added some words about these settings to the admin guide. DocImpact Change-Id: I09a0bd00524544ff627a3bccdcdc48f40720a86e
2013-06-13 11:24:29 -07:00
class TestAffinityLocalityPredicate(unittest.TestCase):
def setUp(self):
self.nodes = [dict(id=0, region=1, zone=1),
dict(id=1, region=1, zone=2),
dict(id=2, region=2, zone=1),
dict(id=3, region=2, zone=2),
dict(id=4, region=3, zone=1),
dict(id=5, region=3, zone=2),
dict(id=6, region=4, zone=0),
dict(id=7, region=4, zone=1)]
def test_empty(self):
pred = utils.affinity_locality_predicate('')
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(pred is None)
Local write affinity for object PUT requests. The proxy can now be configured to prefer local object servers for PUT requests, where "local" is governed by the "write_affinity". The "write_affinity_node_count" setting controls how many local object servers to try before giving up and going on to remote ones. I chose to simply re-order the object servers instead of filtering out nonlocal ones so that, if all of the local ones are down, clients can still get successful responses (just slower). The goal is to trade availability for throughput. By writing to local object servers across fast LAN links, clients get better throughput than if the object servers were far away over slow WAN links. The downside, of course, is that data availability (not durability) may suffer when drives fail. The default configuration has no write affinity in it, so the default behavior is unchanged. Added some words about these settings to the admin guide. DocImpact Change-Id: I09a0bd00524544ff627a3bccdcdc48f40720a86e
2013-06-13 11:24:29 -07:00
def test_region(self):
pred = utils.affinity_locality_predicate('r1')
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(callable(pred))
Local write affinity for object PUT requests. The proxy can now be configured to prefer local object servers for PUT requests, where "local" is governed by the "write_affinity". The "write_affinity_node_count" setting controls how many local object servers to try before giving up and going on to remote ones. I chose to simply re-order the object servers instead of filtering out nonlocal ones so that, if all of the local ones are down, clients can still get successful responses (just slower). The goal is to trade availability for throughput. By writing to local object servers across fast LAN links, clients get better throughput than if the object servers were far away over slow WAN links. The downside, of course, is that data availability (not durability) may suffer when drives fail. The default configuration has no write affinity in it, so the default behavior is unchanged. Added some words about these settings to the admin guide. DocImpact Change-Id: I09a0bd00524544ff627a3bccdcdc48f40720a86e
2013-06-13 11:24:29 -07:00
ids = [n['id'] for n in self.nodes if pred(n)]
self.assertEqual([0, 1], ids)
def test_zone(self):
pred = utils.affinity_locality_predicate('r1z1')
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(callable(pred))
Local write affinity for object PUT requests. The proxy can now be configured to prefer local object servers for PUT requests, where "local" is governed by the "write_affinity". The "write_affinity_node_count" setting controls how many local object servers to try before giving up and going on to remote ones. I chose to simply re-order the object servers instead of filtering out nonlocal ones so that, if all of the local ones are down, clients can still get successful responses (just slower). The goal is to trade availability for throughput. By writing to local object servers across fast LAN links, clients get better throughput than if the object servers were far away over slow WAN links. The downside, of course, is that data availability (not durability) may suffer when drives fail. The default configuration has no write affinity in it, so the default behavior is unchanged. Added some words about these settings to the admin guide. DocImpact Change-Id: I09a0bd00524544ff627a3bccdcdc48f40720a86e
2013-06-13 11:24:29 -07:00
ids = [n['id'] for n in self.nodes if pred(n)]
self.assertEqual([0], ids)
def test_multiple(self):
pred = utils.affinity_locality_predicate('r1, r3, r4z0')
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
self.assertTrue(callable(pred))
Local write affinity for object PUT requests. The proxy can now be configured to prefer local object servers for PUT requests, where "local" is governed by the "write_affinity". The "write_affinity_node_count" setting controls how many local object servers to try before giving up and going on to remote ones. I chose to simply re-order the object servers instead of filtering out nonlocal ones so that, if all of the local ones are down, clients can still get successful responses (just slower). The goal is to trade availability for throughput. By writing to local object servers across fast LAN links, clients get better throughput than if the object servers were far away over slow WAN links. The downside, of course, is that data availability (not durability) may suffer when drives fail. The default configuration has no write affinity in it, so the default behavior is unchanged. Added some words about these settings to the admin guide. DocImpact Change-Id: I09a0bd00524544ff627a3bccdcdc48f40720a86e
2013-06-13 11:24:29 -07:00
ids = [n['id'] for n in self.nodes if pred(n)]
self.assertEqual([0, 1, 4, 5, 6], ids)
def test_invalid(self):
self.assertRaises(ValueError,
utils.affinity_locality_predicate, 'falafel')
self.assertRaises(ValueError,
utils.affinity_locality_predicate, 'r8zQ')
self.assertRaises(ValueError,
utils.affinity_locality_predicate, 'r2d2')
self.assertRaises(ValueError,
utils.affinity_locality_predicate, 'r1z1=1')
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
Move all SLO functionality to middleware This way, with zero additional effort, SLO will support enhancements to object storage and retrieval, such as: * automatic resume of GETs on broken connection (today) * storage policies (in the near future) * erasure-coded object segments (in the far future) This also lets SLOs work with other sorts of hypothetical third-party middleware, for example object compression or encryption. Getting COPY to work here is sort of a hack; the proxy's object controller now checks for "swift.copy_response_hook" in the request's environment and feeds the GET response (the source of the new object's data) through it. This lets a COPY of a SLO manifest actually combine the segments instead of merely copying the manifest document. Updated ObjectController to expect a response's app_iter to be an iterable, not just an iterator. (PEP 333 says "When called by the server, the application object must return an iterable yielding zero or more strings." ObjectController was just being too strict.) This way, SLO can re-use the same response-generation logic for GET and COPY requests. Added a (sort of hokey) mechanism to allow middlewares to close incompletely-consumed app iterators without triggering a warning. SLO does this when it realizes it's performed a ranged GET on a manifest; it closes the iterable, removes the range, and retries the request. Without this change, the proxy logs would get 'Client disconnected on read' in them. DocImpact blueprint multi-ring-large-objects Change-Id: Ic11662eb5c7176fbf422a6fc87a569928d6f85a1
2013-11-13 12:06:55 -08:00
class TestRateLimitedIterator(unittest.TestCase):
Mock out time and sleep to avoid races We mock out time.time(), time.sleep() and eventlet.sleep() so that we avoid test problems caused by exceedingly long delays during the execution of the test. We also make sure to convert the units used in the tests to milliseconds for a bit more clarity. Closes bug: 1298154 Change-Id: I803d06cbf205a02a4f7bb1e0c467d276632cd6a3
2014-03-26 22:38:03 -04:00
def run_under_pseudo_time(
self, func, *args, **kwargs):
curr_time = [42.0]
def my_time():
curr_time[0] += 0.001
return curr_time[0]
def my_sleep(duration):
curr_time[0] += 0.001
curr_time[0] += duration
Get rid of contextlib.nested() for py3 contextlib.nested() is missing completely in Python 3. Since 2.7, we can use multiple context managers in a 'with' statement, like so: with thing1() as t1, thing2() as t2: do_stuff() Now, if we had some code that needed to nest an arbitrary number of context managers, there's stuff we could do with contextlib.ExitStack and such... but we don't. We only use contextlib.nested() in tests to set up bunches of mocks without crazy-deep indentation, and all that stuff fits perfectly into multiple-context-manager 'with' statements. Change-Id: Id472958b007948f05dbd4c7fb8cf3ffab58e2681
2015-10-21 13:05:37 -07:00
with patch('time.time', my_time), \
patch('eventlet.sleep', my_sleep):
Mock out time and sleep to avoid races We mock out time.time(), time.sleep() and eventlet.sleep() so that we avoid test problems caused by exceedingly long delays during the execution of the test. We also make sure to convert the units used in the tests to milliseconds for a bit more clarity. Closes bug: 1298154 Change-Id: I803d06cbf205a02a4f7bb1e0c467d276632cd6a3
2014-03-26 22:38:03 -04:00
return func(*args, **kwargs)
Move all SLO functionality to middleware This way, with zero additional effort, SLO will support enhancements to object storage and retrieval, such as: * automatic resume of GETs on broken connection (today) * storage policies (in the near future) * erasure-coded object segments (in the far future) This also lets SLOs work with other sorts of hypothetical third-party middleware, for example object compression or encryption. Getting COPY to work here is sort of a hack; the proxy's object controller now checks for "swift.copy_response_hook" in the request's environment and feeds the GET response (the source of the new object's data) through it. This lets a COPY of a SLO manifest actually combine the segments instead of merely copying the manifest document. Updated ObjectController to expect a response's app_iter to be an iterable, not just an iterator. (PEP 333 says "When called by the server, the application object must return an iterable yielding zero or more strings." ObjectController was just being too strict.) This way, SLO can re-use the same response-generation logic for GET and COPY requests. Added a (sort of hokey) mechanism to allow middlewares to close incompletely-consumed app iterators without triggering a warning. SLO does this when it realizes it's performed a ranged GET on a manifest; it closes the iterable, removes the range, and retries the request. Without this change, the proxy logs would get 'Client disconnected on read' in them. DocImpact blueprint multi-ring-large-objects Change-Id: Ic11662eb5c7176fbf422a6fc87a569928d6f85a1
2013-11-13 12:06:55 -08:00
def test_rate_limiting(self):
Mock out time and sleep to avoid races We mock out time.time(), time.sleep() and eventlet.sleep() so that we avoid test problems caused by exceedingly long delays during the execution of the test. We also make sure to convert the units used in the tests to milliseconds for a bit more clarity. Closes bug: 1298154 Change-Id: I803d06cbf205a02a4f7bb1e0c467d276632cd6a3
2014-03-26 22:38:03 -04:00
def testfunc():
Replace xrange() with six.moves.range() Patch generated by the xrange operation of the sixer tool: https://pypi.python.org/pypi/sixer Manual changes: * Fix indentation for pep8 checks * Fix TestGreenthreadSafeIterator.test_access_is_serialized of test.unit.common.test_utils: replace range(1, 11) with list(range(1, 11)) * Fix UnsafeXrange docstring, revert change Change-Id: Icb7e26135c5e57b5302b8bfe066b33cafe69fe4d
2015-05-25 18:28:02 +02:00
limited_iterator = utils.RateLimitedIterator(range(9999), 100)
Mock out time and sleep to avoid races We mock out time.time(), time.sleep() and eventlet.sleep() so that we avoid test problems caused by exceedingly long delays during the execution of the test. We also make sure to convert the units used in the tests to milliseconds for a bit more clarity. Closes bug: 1298154 Change-Id: I803d06cbf205a02a4f7bb1e0c467d276632cd6a3
2014-03-26 22:38:03 -04:00
got = []
started_at = time.time()
try:
while time.time() - started_at < 0.1:
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
got.append(next(limited_iterator))
Mock out time and sleep to avoid races We mock out time.time(), time.sleep() and eventlet.sleep() so that we avoid test problems caused by exceedingly long delays during the execution of the test. We also make sure to convert the units used in the tests to milliseconds for a bit more clarity. Closes bug: 1298154 Change-Id: I803d06cbf205a02a4f7bb1e0c467d276632cd6a3
2014-03-26 22:38:03 -04:00
except StopIteration:
pass
return got
got = self.run_under_pseudo_time(testfunc)
Move all SLO functionality to middleware This way, with zero additional effort, SLO will support enhancements to object storage and retrieval, such as: * automatic resume of GETs on broken connection (today) * storage policies (in the near future) * erasure-coded object segments (in the far future) This also lets SLOs work with other sorts of hypothetical third-party middleware, for example object compression or encryption. Getting COPY to work here is sort of a hack; the proxy's object controller now checks for "swift.copy_response_hook" in the request's environment and feeds the GET response (the source of the new object's data) through it. This lets a COPY of a SLO manifest actually combine the segments instead of merely copying the manifest document. Updated ObjectController to expect a response's app_iter to be an iterable, not just an iterator. (PEP 333 says "When called by the server, the application object must return an iterable yielding zero or more strings." ObjectController was just being too strict.) This way, SLO can re-use the same response-generation logic for GET and COPY requests. Added a (sort of hokey) mechanism to allow middlewares to close incompletely-consumed app iterators without triggering a warning. SLO does this when it realizes it's performed a ranged GET on a manifest; it closes the iterable, removes the range, and retries the request. Without this change, the proxy logs would get 'Client disconnected on read' in them. DocImpact blueprint multi-ring-large-objects Change-Id: Ic11662eb5c7176fbf422a6fc87a569928d6f85a1
2013-11-13 12:06:55 -08:00
# it's 11, not 10, because ratelimiting doesn't apply to the very
# first element.
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(got), 11)
Move all SLO functionality to middleware This way, with zero additional effort, SLO will support enhancements to object storage and retrieval, such as: * automatic resume of GETs on broken connection (today) * storage policies (in the near future) * erasure-coded object segments (in the far future) This also lets SLOs work with other sorts of hypothetical third-party middleware, for example object compression or encryption. Getting COPY to work here is sort of a hack; the proxy's object controller now checks for "swift.copy_response_hook" in the request's environment and feeds the GET response (the source of the new object's data) through it. This lets a COPY of a SLO manifest actually combine the segments instead of merely copying the manifest document. Updated ObjectController to expect a response's app_iter to be an iterable, not just an iterator. (PEP 333 says "When called by the server, the application object must return an iterable yielding zero or more strings." ObjectController was just being too strict.) This way, SLO can re-use the same response-generation logic for GET and COPY requests. Added a (sort of hokey) mechanism to allow middlewares to close incompletely-consumed app iterators without triggering a warning. SLO does this when it realizes it's performed a ranged GET on a manifest; it closes the iterable, removes the range, and retries the request. Without this change, the proxy logs would get 'Client disconnected on read' in them. DocImpact blueprint multi-ring-large-objects Change-Id: Ic11662eb5c7176fbf422a6fc87a569928d6f85a1
2013-11-13 12:06:55 -08:00
Allow smaller segments in static large objects The addition of range support for SLO segments (commit 25d5e68) required the range size to be at least the SLO minimum segment size (default 1 MiB). However, if you're doing something like assembling a video of short clips out of a larger one, then you might not need a full 1 MiB. The reason for the 1 MiB restriction was to protect Swift from resource overconsumption. It takes CPU, RAM, and internal bandwidth to connect to an object server, so it's much cheaper to serve a 10 GiB SLO if it has 10 MiB segments than if it has 10 B segments. Instead of a strict limit, now we apply ratelimiting to small segments. The threshold for "small" is configurable and defaults to 1 MiB. SLO segments may now be as small as 1 byte. If a client makes SLOs as before, it'll still be able to download the objects as fast as Swift can serve them. However, a SLO with a lot of small ranges or segments will be slowed down to avoid resource overconsumption. This is similar to how DLOs work, except that DLOs ratelimit *every* segment, not just small ones. UpgradeImpact For operators: if your cluster has enabled ratelimiting for SLO, you will want to set rate_limit_under_size to a large number prior to upgrade. This will preserve your existing behavior of ratelimiting all SLO segments. 5368709123 is a good value, as that's 1 greater than the default max object size. Alternately, hold down the 9 key until you get bored. If your cluster has not enabled ratelimiting for SLO (the default), no action is needed. Change-Id: Id1ff7742308ed816038a5c44ec548afa26612b95
2015-11-30 18:06:09 -08:00
def test_rate_limiting_sometimes(self):
def testfunc():
limited_iterator = utils.RateLimitedIterator(
range(9999), 100,
ratelimit_if=lambda item: item % 23 != 0)
got = []
started_at = time.time()
try:
while time.time() - started_at < 0.5:
got.append(next(limited_iterator))
except StopIteration:
pass
return got
got = self.run_under_pseudo_time(testfunc)
# we'd get 51 without the ratelimit_if, but because 0, 23 and 46
# weren't subject to ratelimiting, we get 54 instead
self.assertEqual(len(got), 54)
Move all SLO functionality to middleware This way, with zero additional effort, SLO will support enhancements to object storage and retrieval, such as: * automatic resume of GETs on broken connection (today) * storage policies (in the near future) * erasure-coded object segments (in the far future) This also lets SLOs work with other sorts of hypothetical third-party middleware, for example object compression or encryption. Getting COPY to work here is sort of a hack; the proxy's object controller now checks for "swift.copy_response_hook" in the request's environment and feeds the GET response (the source of the new object's data) through it. This lets a COPY of a SLO manifest actually combine the segments instead of merely copying the manifest document. Updated ObjectController to expect a response's app_iter to be an iterable, not just an iterator. (PEP 333 says "When called by the server, the application object must return an iterable yielding zero or more strings." ObjectController was just being too strict.) This way, SLO can re-use the same response-generation logic for GET and COPY requests. Added a (sort of hokey) mechanism to allow middlewares to close incompletely-consumed app iterators without triggering a warning. SLO does this when it realizes it's performed a ranged GET on a manifest; it closes the iterable, removes the range, and retries the request. Without this change, the proxy logs would get 'Client disconnected on read' in them. DocImpact blueprint multi-ring-large-objects Change-Id: Ic11662eb5c7176fbf422a6fc87a569928d6f85a1
2013-11-13 12:06:55 -08:00
def test_limit_after(self):
Mock out time and sleep to avoid races We mock out time.time(), time.sleep() and eventlet.sleep() so that we avoid test problems caused by exceedingly long delays during the execution of the test. We also make sure to convert the units used in the tests to milliseconds for a bit more clarity. Closes bug: 1298154 Change-Id: I803d06cbf205a02a4f7bb1e0c467d276632cd6a3
2014-03-26 22:38:03 -04:00
def testfunc():
limited_iterator = utils.RateLimitedIterator(
Replace xrange() with six.moves.range() Patch generated by the xrange operation of the sixer tool: https://pypi.python.org/pypi/sixer Manual changes: * Fix indentation for pep8 checks * Fix TestGreenthreadSafeIterator.test_access_is_serialized of test.unit.common.test_utils: replace range(1, 11) with list(range(1, 11)) * Fix UnsafeXrange docstring, revert change Change-Id: Icb7e26135c5e57b5302b8bfe066b33cafe69fe4d
2015-05-25 18:28:02 +02:00
range(9999), 100, limit_after=5)
Mock out time and sleep to avoid races We mock out time.time(), time.sleep() and eventlet.sleep() so that we avoid test problems caused by exceedingly long delays during the execution of the test. We also make sure to convert the units used in the tests to milliseconds for a bit more clarity. Closes bug: 1298154 Change-Id: I803d06cbf205a02a4f7bb1e0c467d276632cd6a3
2014-03-26 22:38:03 -04:00
got = []
started_at = time.time()
try:
while time.time() - started_at < 0.1:
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
got.append(next(limited_iterator))
Mock out time and sleep to avoid races We mock out time.time(), time.sleep() and eventlet.sleep() so that we avoid test problems caused by exceedingly long delays during the execution of the test. We also make sure to convert the units used in the tests to milliseconds for a bit more clarity. Closes bug: 1298154 Change-Id: I803d06cbf205a02a4f7bb1e0c467d276632cd6a3
2014-03-26 22:38:03 -04:00
except StopIteration:
pass
return got
got = self.run_under_pseudo_time(testfunc)
# it's 16, not 15, because ratelimiting doesn't apply to the very
# first element.
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(len(got), 16)
Move all SLO functionality to middleware This way, with zero additional effort, SLO will support enhancements to object storage and retrieval, such as: * automatic resume of GETs on broken connection (today) * storage policies (in the near future) * erasure-coded object segments (in the far future) This also lets SLOs work with other sorts of hypothetical third-party middleware, for example object compression or encryption. Getting COPY to work here is sort of a hack; the proxy's object controller now checks for "swift.copy_response_hook" in the request's environment and feeds the GET response (the source of the new object's data) through it. This lets a COPY of a SLO manifest actually combine the segments instead of merely copying the manifest document. Updated ObjectController to expect a response's app_iter to be an iterable, not just an iterator. (PEP 333 says "When called by the server, the application object must return an iterable yielding zero or more strings." ObjectController was just being too strict.) This way, SLO can re-use the same response-generation logic for GET and COPY requests. Added a (sort of hokey) mechanism to allow middlewares to close incompletely-consumed app iterators without triggering a warning. SLO does this when it realizes it's performed a ranged GET on a manifest; it closes the iterable, removes the range, and retries the request. Without this change, the proxy logs would get 'Client disconnected on read' in them. DocImpact blueprint multi-ring-large-objects Change-Id: Ic11662eb5c7176fbf422a6fc87a569928d6f85a1
2013-11-13 12:06:55 -08:00
Fix ValueError from handoff logging. If you have StatsD logging turned on, then the iterator returned by Ring.get_part_nodes() will emit StatsD packets when it yields a handoff node. That network IO may make eventlet trampoline to another greenthread before returning from next(). Now, if that other greenthread tries to call next() on that same iterator, it blows up with a ValueError. Any socket IO inside a generator's next() method can cause this. It's easiest to reproduce with StatsD logging turned on, but logging to syslog can trigger it too. You can see this happen sometimes in the proxy's make_requests method if two of the primary nodes are down. Greenthread A goes into next() to get a handoff node, then sends a StatsD packet, and so eventlet trampolines to Greenthread B. Now, Greenthread B also calls next() to get a handoff node, and dies with a ValueError. This commit wraps up concurrently-accessed iter_nodes() iterators in a new thing called a GreenthreadSafeIterator that serializes access. Bug 1180110 Change-Id: I8fe13d7295c056a2cab9e084f5966078a49bdc13
2013-05-14 13:04:01 -07:00
class TestGreenthreadSafeIterator(unittest.TestCase):
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
Fix ValueError from handoff logging. If you have StatsD logging turned on, then the iterator returned by Ring.get_part_nodes() will emit StatsD packets when it yields a handoff node. That network IO may make eventlet trampoline to another greenthread before returning from next(). Now, if that other greenthread tries to call next() on that same iterator, it blows up with a ValueError. Any socket IO inside a generator's next() method can cause this. It's easiest to reproduce with StatsD logging turned on, but logging to syslog can trigger it too. You can see this happen sometimes in the proxy's make_requests method if two of the primary nodes are down. Greenthread A goes into next() to get a handoff node, then sends a StatsD packet, and so eventlet trampolines to Greenthread B. Now, Greenthread B also calls next() to get a handoff node, and dies with a ValueError. This commit wraps up concurrently-accessed iter_nodes() iterators in a new thing called a GreenthreadSafeIterator that serializes access. Bug 1180110 Change-Id: I8fe13d7295c056a2cab9e084f5966078a49bdc13
2013-05-14 13:04:01 -07:00
def increment(self, iterable):
plus_ones = []
for n in iterable:
plus_ones.append(n + 1)
return plus_ones
def test_setup_works(self):
# it should work without concurrent access
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual([0, 1, 2, 3], list(UnsafeXrange(4)))
Fix ValueError from handoff logging. If you have StatsD logging turned on, then the iterator returned by Ring.get_part_nodes() will emit StatsD packets when it yields a handoff node. That network IO may make eventlet trampoline to another greenthread before returning from next(). Now, if that other greenthread tries to call next() on that same iterator, it blows up with a ValueError. Any socket IO inside a generator's next() method can cause this. It's easiest to reproduce with StatsD logging turned on, but logging to syslog can trigger it too. You can see this happen sometimes in the proxy's make_requests method if two of the primary nodes are down. Greenthread A goes into next() to get a handoff node, then sends a StatsD packet, and so eventlet trampolines to Greenthread B. Now, Greenthread B also calls next() to get a handoff node, and dies with a ValueError. This commit wraps up concurrently-accessed iter_nodes() iterators in a new thing called a GreenthreadSafeIterator that serializes access. Bug 1180110 Change-Id: I8fe13d7295c056a2cab9e084f5966078a49bdc13
2013-05-14 13:04:01 -07:00
iterable = UnsafeXrange(10)
pile = eventlet.GreenPile(2)
Replace xrange() with six.moves.range() Patch generated by the xrange operation of the sixer tool: https://pypi.python.org/pypi/sixer Manual changes: * Fix indentation for pep8 checks * Fix TestGreenthreadSafeIterator.test_access_is_serialized of test.unit.common.test_utils: replace range(1, 11) with list(range(1, 11)) * Fix UnsafeXrange docstring, revert change Change-Id: Icb7e26135c5e57b5302b8bfe066b33cafe69fe4d
2015-05-25 18:28:02 +02:00
for _ in range(2):
Fix ValueError from handoff logging. If you have StatsD logging turned on, then the iterator returned by Ring.get_part_nodes() will emit StatsD packets when it yields a handoff node. That network IO may make eventlet trampoline to another greenthread before returning from next(). Now, if that other greenthread tries to call next() on that same iterator, it blows up with a ValueError. Any socket IO inside a generator's next() method can cause this. It's easiest to reproduce with StatsD logging turned on, but logging to syslog can trigger it too. You can see this happen sometimes in the proxy's make_requests method if two of the primary nodes are down. Greenthread A goes into next() to get a handoff node, then sends a StatsD packet, and so eventlet trampolines to Greenthread B. Now, Greenthread B also calls next() to get a handoff node, and dies with a ValueError. This commit wraps up concurrently-accessed iter_nodes() iterators in a new thing called a GreenthreadSafeIterator that serializes access. Bug 1180110 Change-Id: I8fe13d7295c056a2cab9e084f5966078a49bdc13
2013-05-14 13:04:01 -07:00
pile.spawn(self.increment, iterable)
Fix TestGreenThreadSafeIterator test Change-Id: Ic121ae5c99b4e5c5e3d1efc432b55b762d50f3fb Closes-Bug: 1190244
2014-03-18 21:17:52 +00:00
sorted([resp for resp in pile])
self.assertTrue(
iterable.concurrent_call, 'test setup is insufficiently crazy')
Fix ValueError from handoff logging. If you have StatsD logging turned on, then the iterator returned by Ring.get_part_nodes() will emit StatsD packets when it yields a handoff node. That network IO may make eventlet trampoline to another greenthread before returning from next(). Now, if that other greenthread tries to call next() on that same iterator, it blows up with a ValueError. Any socket IO inside a generator's next() method can cause this. It's easiest to reproduce with StatsD logging turned on, but logging to syslog can trigger it too. You can see this happen sometimes in the proxy's make_requests method if two of the primary nodes are down. Greenthread A goes into next() to get a handoff node, then sends a StatsD packet, and so eventlet trampolines to Greenthread B. Now, Greenthread B also calls next() to get a handoff node, and dies with a ValueError. This commit wraps up concurrently-accessed iter_nodes() iterators in a new thing called a GreenthreadSafeIterator that serializes access. Bug 1180110 Change-Id: I8fe13d7295c056a2cab9e084f5966078a49bdc13
2013-05-14 13:04:01 -07:00
def test_access_is_serialized(self):
pile = eventlet.GreenPile(2)
Fix TestGreenThreadSafeIterator test Change-Id: Ic121ae5c99b4e5c5e3d1efc432b55b762d50f3fb Closes-Bug: 1190244
2014-03-18 21:17:52 +00:00
unsafe_iterable = UnsafeXrange(10)
iterable = utils.GreenthreadSafeIterator(unsafe_iterable)
Replace xrange() with six.moves.range() Patch generated by the xrange operation of the sixer tool: https://pypi.python.org/pypi/sixer Manual changes: * Fix indentation for pep8 checks * Fix TestGreenthreadSafeIterator.test_access_is_serialized of test.unit.common.test_utils: replace range(1, 11) with list(range(1, 11)) * Fix UnsafeXrange docstring, revert change Change-Id: Icb7e26135c5e57b5302b8bfe066b33cafe69fe4d
2015-05-25 18:28:02 +02:00
for _ in range(2):
Fix ValueError from handoff logging. If you have StatsD logging turned on, then the iterator returned by Ring.get_part_nodes() will emit StatsD packets when it yields a handoff node. That network IO may make eventlet trampoline to another greenthread before returning from next(). Now, if that other greenthread tries to call next() on that same iterator, it blows up with a ValueError. Any socket IO inside a generator's next() method can cause this. It's easiest to reproduce with StatsD logging turned on, but logging to syslog can trigger it too. You can see this happen sometimes in the proxy's make_requests method if two of the primary nodes are down. Greenthread A goes into next() to get a handoff node, then sends a StatsD packet, and so eventlet trampolines to Greenthread B. Now, Greenthread B also calls next() to get a handoff node, and dies with a ValueError. This commit wraps up concurrently-accessed iter_nodes() iterators in a new thing called a GreenthreadSafeIterator that serializes access. Bug 1180110 Change-Id: I8fe13d7295c056a2cab9e084f5966078a49bdc13
2013-05-14 13:04:01 -07:00
pile.spawn(self.increment, iterable)
response = sorted(sum([resp for resp in pile], []))
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(list(range(1, 11)), response)
Fix TestGreenThreadSafeIterator test Change-Id: Ic121ae5c99b4e5c5e3d1efc432b55b762d50f3fb Closes-Bug: 1190244
2014-03-18 21:17:52 +00:00
self.assertTrue(
not unsafe_iterable.concurrent_call, 'concurrent call occurred')
Fix ValueError from handoff logging. If you have StatsD logging turned on, then the iterator returned by Ring.get_part_nodes() will emit StatsD packets when it yields a handoff node. That network IO may make eventlet trampoline to another greenthread before returning from next(). Now, if that other greenthread tries to call next() on that same iterator, it blows up with a ValueError. Any socket IO inside a generator's next() method can cause this. It's easiest to reproduce with StatsD logging turned on, but logging to syslog can trigger it too. You can see this happen sometimes in the proxy's make_requests method if two of the primary nodes are down. Greenthread A goes into next() to get a handoff node, then sends a StatsD packet, and so eventlet trampolines to Greenthread B. Now, Greenthread B also calls next() to get a handoff node, and dies with a ValueError. This commit wraps up concurrently-accessed iter_nodes() iterators in a new thing called a GreenthreadSafeIterator that serializes access. Bug 1180110 Change-Id: I8fe13d7295c056a2cab9e084f5966078a49bdc13
2013-05-14 13:04:01 -07:00
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
class TestStatsdLoggingDelegation(unittest.TestCase):
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
def setUp(self):
self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
Make StatsD tests less flaky. There have been a bunch of Jenkins failures lately where the StatsD tests fail because they can't bind to their desired port. There's nothing special about the particular port they're using, so now we let the kernel pick an available one for us. This also lets us get rid of a sleep() in the test that looked like an attempt to alleviate EADDRINUSE errors, so now in the happy case, the tests are a few fractions of a second faster. Change-Id: Idee11349254107a59643539b1566f3588eee7ef4
2012-10-15 11:32:26 -07:00
self.sock.bind(('localhost', 0))
self.port = self.sock.getsockname()[1]
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
self.queue = Queue()
Use threadpools in the object server for performance. Without a (per-disk) threadpool, requests to a slow disk would affect all clients by blocking the entire eventlet reactor on read/write/etc. The slower the disk, the worse the performance. On an object server, you frequently have at least one slow disk due to auditing and replication activity sucking up all the available IO. By kicking those blocking calls out to a separate OS thread, we let the eventlet reactor make progress in other greenthreads, and by having a per-disk pool, we ensure that one slow disk can't suck up all the resources of an entire object server. There were a few blocking calls that were done with eventlet.tpool, but that's a fixed-size global threadpool, so I moved them to the per-disk threadpools. If the object server is configured not to use per-disk threadpools, (i.e. threads_per_disk = 0, which is the default), those call sites will still ultimately end up using eventlet.tpool.execute. You won't end up blocking a whole object server while waiting for a huge fsync. If you decide not to use threadpools, the only extra overhead should be a few extra Python function calls here and there. This is accomplished by setting threads_per_disk = 0 in the config. blueprint concurrent-disk-io Change-Id: I490f8753d926fdcee3a0c65c5aaf715bc2b7c290
2013-03-19 22:27:21 -07:00
self.reader_thread = threading.Thread(target=self.statsd_reader)
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
self.reader_thread.setDaemon(1)
self.reader_thread.start()
def tearDown(self):
# The "no-op when disabled" test doesn't set up a real logger, so
# create one here so we can tell the reader thread to stop.
if not getattr(self, 'logger', None):
self.logger = utils.get_logger({
'log_statsd_host': 'localhost',
'log_statsd_port': str(self.port),
}, 'some-name')
self.logger.increment('STOP')
self.reader_thread.join(timeout=4)
self.sock.close()
del self.logger
def statsd_reader(self):
while True:
try:
payload = self.sock.recv(4096)
Port swift.common.utils.StatsdClient to Python 3 * StatsdClient._send(): on Python 3, encode parts to UTF-8 and replace '|' with b'|' to join parts. * timing_stats(): replace func.func_name with func.__name__. The func_name attribute of functions was removed on Python 3, whereas the __name__ attribute is available on Python 2 and Python 3. * Fix unit tests to use bytes Change-Id: Ic279c9b54e91aabcc52587eed7758e268ffb155e
2015-10-19 16:19:28 +02:00
if payload and b'STOP' in payload:
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
return 42
self.queue.put(payload)
Use Python 3.x compatible except construct except x,y: was deprected and is removed in Python 3.x. Use "except x as y:" instead which works in any Python version >= 2.6. Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
2013-08-28 21:16:08 +02:00
except Exception as e:
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
sys.stderr.write('statsd_reader thread: %r' % (e,))
break
def _send_and_get(self, sender_fn, *args, **kwargs):
"""
Because the client library may not actually send a packet with
sample_rate < 1, we keep trying until we get one through.
"""
got = None
while not got:
sender_fn(*args, **kwargs)
try:
got = self.queue.get(timeout=0.5)
except Empty:
pass
return got
def assertStat(self, expected, sender_fn, *args, **kwargs):
got = self._send_and_get(sender_fn, *args, **kwargs)
Port swift.common.utils.StatsdClient to Python 3 * StatsdClient._send(): on Python 3, encode parts to UTF-8 and replace '|' with b'|' to join parts. * timing_stats(): replace func.func_name with func.__name__. The func_name attribute of functions was removed on Python 3, whereas the __name__ attribute is available on Python 2 and Python 3. * Fix unit tests to use bytes Change-Id: Ic279c9b54e91aabcc52587eed7758e268ffb155e
2015-10-19 16:19:28 +02:00
if six.PY3:
got = got.decode('utf-8')
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
return self.assertEqual(expected, got)
def assertStatMatches(self, expected_regexp, sender_fn, *args, **kwargs):
got = self._send_and_get(sender_fn, *args, **kwargs)
Port swift.common.utils.StatsdClient to Python 3 * StatsdClient._send(): on Python 3, encode parts to UTF-8 and replace '|' with b'|' to join parts. * timing_stats(): replace func.func_name with func.__name__. The func_name attribute of functions was removed on Python 3, whereas the __name__ attribute is available on Python 2 and Python 3. * Fix unit tests to use bytes Change-Id: Ic279c9b54e91aabcc52587eed7758e268ffb155e
2015-10-19 16:19:28 +02:00
if six.PY3:
got = got.decode('utf-8')
unit tests: Replace "self.assert_" by "self.assertTrue" The assert_() method is deprecated and can be safely replaced by assertTrue(). This patch makes sure that running the tests does not create undesired warnings. Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
2015-07-21 19:23:00 +05:30
return self.assertTrue(re.search(expected_regexp, got),
[got, expected_regexp])
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
def test_methods_are_no_ops_when_not_enabled(self):
logger = utils.get_logger({
# No "log_statsd_host" means "disabled"
'log_statsd_port': str(self.port),
}, 'some-name')
# Delegate methods are no-ops
Replace assertEqual(None, *) with assertIsNone in tests As swift no longer supports Python 2.6, replace assertEqual(None, *) with assertIsNone in tests to have more clear messages in case of failure. Change-Id: I94af3e8156ef40465d4f7a2cb79fb99fc7bbda56 Closes-Bug: #1280522
2016-02-16 23:44:27 +08:00
self.assertIsNone(logger.update_stats('foo', 88))
self.assertIsNone(logger.update_stats('foo', 88, 0.57))
self.assertIsNone(logger.update_stats('foo', 88,
sample_rate=0.61))
self.assertIsNone(logger.increment('foo'))
self.assertIsNone(logger.increment('foo', 0.57))
self.assertIsNone(logger.increment('foo', sample_rate=0.61))
self.assertIsNone(logger.decrement('foo'))
self.assertIsNone(logger.decrement('foo', 0.57))
self.assertIsNone(logger.decrement('foo', sample_rate=0.61))
self.assertIsNone(logger.timing('foo', 88.048))
self.assertIsNone(logger.timing('foo', 88.57, 0.34))
self.assertIsNone(logger.timing('foo', 88.998, sample_rate=0.82))
self.assertIsNone(logger.timing_since('foo', 8938))
self.assertIsNone(logger.timing_since('foo', 8948, 0.57))
self.assertIsNone(logger.timing_since('foo', 849398,
sample_rate=0.61))
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
# Now, the queue should be empty (no UDP packets sent)
self.assertRaises(Empty, self.queue.get_nowait)
def test_delegate_methods_with_no_default_sample_rate(self):
self.logger = utils.get_logger({
'log_statsd_host': 'localhost',
'log_statsd_port': str(self.port),
}, 'some-name')
self.assertStat('some-name.some.counter:1|c', self.logger.increment,
'some.counter')
self.assertStat('some-name.some.counter:-1|c', self.logger.decrement,
'some.counter')
self.assertStat('some-name.some.operation:4900.0|ms',
self.logger.timing, 'some.operation', 4.9 * 1000)
self.assertStatMatches('some-name\.another\.operation:\d+\.\d+\|ms',
self.logger.timing_since, 'another.operation',
time.time())
self.assertStat('some-name.another.counter:42|c',
self.logger.update_stats, 'another.counter', 42)
# Each call can override the sample_rate (also, bonus prefix test)
self.logger.set_statsd_prefix('pfx')
self.assertStat('pfx.some.counter:1|c|@0.972', self.logger.increment,
'some.counter', sample_rate=0.972)
self.assertStat('pfx.some.counter:-1|c|@0.972', self.logger.decrement,
'some.counter', sample_rate=0.972)
self.assertStat('pfx.some.operation:4900.0|ms|@0.972',
self.logger.timing, 'some.operation', 4.9 * 1000,
sample_rate=0.972)
self.assertStatMatches('pfx\.another\.op:\d+\.\d+\|ms|@0.972',
self.logger.timing_since, 'another.op',
time.time(), sample_rate=0.972)
self.assertStat('pfx.another.counter:3|c|@0.972',
self.logger.update_stats, 'another.counter', 3,
sample_rate=0.972)
# Can override sample_rate with non-keyword arg
self.logger.set_statsd_prefix('')
self.assertStat('some.counter:1|c|@0.939', self.logger.increment,
'some.counter', 0.939)
self.assertStat('some.counter:-1|c|@0.939', self.logger.decrement,
'some.counter', 0.939)
self.assertStat('some.operation:4900.0|ms|@0.939',
self.logger.timing, 'some.operation',
4.9 * 1000, 0.939)
self.assertStatMatches('another\.op:\d+\.\d+\|ms|@0.939',
self.logger.timing_since, 'another.op',
time.time(), 0.939)
self.assertStat('another.counter:3|c|@0.939',
self.logger.update_stats, 'another.counter', 3, 0.939)
def test_delegate_methods_with_default_sample_rate(self):
self.logger = utils.get_logger({
'log_statsd_host': 'localhost',
'log_statsd_port': str(self.port),
'log_statsd_default_sample_rate': '0.93',
}, 'pfx')
self.assertStat('pfx.some.counter:1|c|@0.93', self.logger.increment,
'some.counter')
self.assertStat('pfx.some.counter:-1|c|@0.93', self.logger.decrement,
'some.counter')
self.assertStat('pfx.some.operation:4760.0|ms|@0.93',
self.logger.timing, 'some.operation', 4.76 * 1000)
self.assertStatMatches('pfx\.another\.op:\d+\.\d+\|ms|@0.93',
self.logger.timing_since, 'another.op',
time.time())
self.assertStat('pfx.another.counter:3|c|@0.93',
self.logger.update_stats, 'another.counter', 3)
# Each call can override the sample_rate
self.assertStat('pfx.some.counter:1|c|@0.9912', self.logger.increment,
'some.counter', sample_rate=0.9912)
self.assertStat('pfx.some.counter:-1|c|@0.9912', self.logger.decrement,
'some.counter', sample_rate=0.9912)
self.assertStat('pfx.some.operation:4900.0|ms|@0.9912',
self.logger.timing, 'some.operation', 4.9 * 1000,
sample_rate=0.9912)
self.assertStatMatches('pfx\.another\.op:\d+\.\d+\|ms|@0.9912',
self.logger.timing_since, 'another.op',
time.time(), sample_rate=0.9912)
self.assertStat('pfx.another.counter:3|c|@0.9912',
self.logger.update_stats, 'another.counter', 3,
sample_rate=0.9912)
# Can override sample_rate with non-keyword arg
self.logger.set_statsd_prefix('')
self.assertStat('some.counter:1|c|@0.987654', self.logger.increment,
'some.counter', 0.987654)
self.assertStat('some.counter:-1|c|@0.987654', self.logger.decrement,
'some.counter', 0.987654)
self.assertStat('some.operation:4900.0|ms|@0.987654',
self.logger.timing, 'some.operation',
4.9 * 1000, 0.987654)
self.assertStatMatches('another\.op:\d+\.\d+\|ms|@0.987654',
self.logger.timing_since, 'another.op',
time.time(), 0.987654)
self.assertStat('another.counter:3|c|@0.987654',
self.logger.update_stats, 'another.counter',
3, 0.987654)
def test_delegate_methods_with_metric_prefix(self):
self.logger = utils.get_logger({
'log_statsd_host': 'localhost',
'log_statsd_port': str(self.port),
'log_statsd_metric_prefix': 'alpha.beta',
}, 'pfx')
self.assertStat('alpha.beta.pfx.some.counter:1|c',
self.logger.increment, 'some.counter')
self.assertStat('alpha.beta.pfx.some.counter:-1|c',
self.logger.decrement, 'some.counter')
self.assertStat('alpha.beta.pfx.some.operation:4760.0|ms',
self.logger.timing, 'some.operation', 4.76 * 1000)
self.assertStatMatches(
'alpha\.beta\.pfx\.another\.op:\d+\.\d+\|ms',
self.logger.timing_since, 'another.op', time.time())
self.assertStat('alpha.beta.pfx.another.counter:3|c',
self.logger.update_stats, 'another.counter', 3)
self.logger.set_statsd_prefix('')
self.assertStat('alpha.beta.some.counter:1|c|@0.9912',
self.logger.increment, 'some.counter',
sample_rate=0.9912)
self.assertStat('alpha.beta.some.counter:-1|c|@0.9912',
self.logger.decrement, 'some.counter', 0.9912)
self.assertStat('alpha.beta.some.operation:4900.0|ms|@0.9912',
self.logger.timing, 'some.operation', 4.9 * 1000,
sample_rate=0.9912)
self.assertStatMatches('alpha\.beta\.another\.op:\d+\.\d+\|ms|@0.9912',
self.logger.timing_since, 'another.op',
time.time(), sample_rate=0.9912)
self.assertStat('alpha.beta.another.counter:3|c|@0.9912',
self.logger.update_stats, 'another.counter', 3,
sample_rate=0.9912)
Adapt Swift for WebOb 1.2 Based on PatchSet 3 of https://review.openstack.org/#/c/7569/ , make them to pass all funcional tests with both webob 1.x and 1.2. The additional following compatibility issues were addressed: - Until patch for range header issue is merged into official webob release, testRangedGetsWithLWSinHeader() should skip test against webob 1.2 (https://github.com/Pylons/webob/commit/49c175aec2b5c134511dfea6564bfb1d37ab23a9) - common.constraints.check_utf8() can accept both utf8 str and unicode. - To convert unicode to utf-8 str if necessary. - Making proxy_logging can handle invalid utf-8 str bug 888371 bug 959881 blueprint webob-support Change-Id: I00e5fd04cd1653259606a4ffdd4926db3c84c496
2012-06-06 03:39:53 +09:00
def test_get_valid_utf8_str(self):
unicode_sample = u'\uc77c\uc601'
valid_utf8_str = unicode_sample.encode('utf-8')
invalid_utf8_str = unicode_sample.encode('utf-8')[::-1]
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(valid_utf8_str,
utils.get_valid_utf8_str(valid_utf8_str))
self.assertEqual(valid_utf8_str,
utils.get_valid_utf8_str(unicode_sample))
Port swift.common.utils.StatsdClient to Python 3 * StatsdClient._send(): on Python 3, encode parts to UTF-8 and replace '|' with b'|' to join parts. * timing_stats(): replace func.func_name with func.__name__. The func_name attribute of functions was removed on Python 3, whereas the __name__ attribute is available on Python 2 and Python 3. * Fix unit tests to use bytes Change-Id: Ic279c9b54e91aabcc52587eed7758e268ffb155e
2015-10-19 16:19:28 +02:00
self.assertEqual(b'\xef\xbf\xbd\xef\xbf\xbd\xec\xbc\x9d\xef\xbf\xbd',
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
utils.get_valid_utf8_str(invalid_utf8_str))
Logging improvements: handoffs and thread locals A warning log line is emitted whenever the proxy has to use a handoff node. Monitoring these warnings can indicate a problem within your cluster; however, you can disable these log lines by setting the proxy conf's log_handoffs to false. While working on this, I also noticed why many proxy log lines did not have txn_id and client_ip -- subcoroutines. Now the logger thread locals are copied to the subcoroutines. Change-Id: Ibac086e1b985f566c068d083620287509de35da8
2012-07-29 17:22:18 +00:00
Fix logger test coupling Two tests fail if test/unit/common/test_daemon.py is excluded from the test set: test.unit.common.test_utils.TestUtils.test_swift_log_formatter test.unit.common.test_utils.TestStatsdLoggingDelegation.test_thread_locals They fail because they assert that logger.txn_id is None, when previous tests cause txn_id to be set. This coupling is masked when test_daemon.py is included because it reloads the common/utils module, and is executed just before test.unit.common.test_utils. The coupling can be observed by running, for example: nosetests ./test/unit/common/middleware/test_except.py ./test/unit/common/test_utils.py or nosetests ./test/unit/proxy ./test/unit/common/test_utils.py The failing tests should reset logger thread_locals before making assertions. test_utils.reset_loggers() attempts this but is broken because it sets thread_local on the wrong object. Changes in this patch: - fix reset_loggers() to reset the LogAdapter thread local state - add a reset_logger_state decorator to call reset_loggers before and after a decorated method - use the decorator for increased isolation of tests that previously called reset_loggers only on exit Change-Id: If9aa781a2dd1929a47ef69322ec8c53263d47660
2015-02-11 17:49:41 +00:00
@reset_logger_state
Logging improvements: handoffs and thread locals A warning log line is emitted whenever the proxy has to use a handoff node. Monitoring these warnings can indicate a problem within your cluster; however, you can disable these log lines by setting the proxy conf's log_handoffs to false. While working on this, I also noticed why many proxy log lines did not have txn_id and client_ip -- subcoroutines. Now the logger thread locals are copied to the subcoroutines. Change-Id: Ibac086e1b985f566c068d083620287509de35da8
2012-07-29 17:22:18 +00:00
def test_thread_locals(self):
logger = utils.get_logger(None)
Fix logger test coupling Two tests fail if test/unit/common/test_daemon.py is excluded from the test set: test.unit.common.test_utils.TestUtils.test_swift_log_formatter test.unit.common.test_utils.TestStatsdLoggingDelegation.test_thread_locals They fail because they assert that logger.txn_id is None, when previous tests cause txn_id to be set. This coupling is masked when test_daemon.py is included because it reloads the common/utils module, and is executed just before test.unit.common.test_utils. The coupling can be observed by running, for example: nosetests ./test/unit/common/middleware/test_except.py ./test/unit/common/test_utils.py or nosetests ./test/unit/proxy ./test/unit/common/test_utils.py The failing tests should reset logger thread_locals before making assertions. test_utils.reset_loggers() attempts this but is broken because it sets thread_local on the wrong object. Changes in this patch: - fix reset_loggers() to reset the LogAdapter thread local state - add a reset_logger_state decorator to call reset_loggers before and after a decorated method - use the decorator for increased isolation of tests that previously called reset_loggers only on exit Change-Id: If9aa781a2dd1929a47ef69322ec8c53263d47660
2015-02-11 17:49:41 +00:00
# test the setter
logger.thread_locals = ('id', 'ip')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(logger.thread_locals, ('id', 'ip'))
Fix logger test coupling Two tests fail if test/unit/common/test_daemon.py is excluded from the test set: test.unit.common.test_utils.TestUtils.test_swift_log_formatter test.unit.common.test_utils.TestStatsdLoggingDelegation.test_thread_locals They fail because they assert that logger.txn_id is None, when previous tests cause txn_id to be set. This coupling is masked when test_daemon.py is included because it reloads the common/utils module, and is executed just before test.unit.common.test_utils. The coupling can be observed by running, for example: nosetests ./test/unit/common/middleware/test_except.py ./test/unit/common/test_utils.py or nosetests ./test/unit/proxy ./test/unit/common/test_utils.py The failing tests should reset logger thread_locals before making assertions. test_utils.reset_loggers() attempts this but is broken because it sets thread_local on the wrong object. Changes in this patch: - fix reset_loggers() to reset the LogAdapter thread local state - add a reset_logger_state decorator to call reset_loggers before and after a decorated method - use the decorator for increased isolation of tests that previously called reset_loggers only on exit Change-Id: If9aa781a2dd1929a47ef69322ec8c53263d47660
2015-02-11 17:49:41 +00:00
# reset
logger.thread_locals = (None, None)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(logger.thread_locals, (None, None))
Fix logger test coupling Two tests fail if test/unit/common/test_daemon.py is excluded from the test set: test.unit.common.test_utils.TestUtils.test_swift_log_formatter test.unit.common.test_utils.TestStatsdLoggingDelegation.test_thread_locals They fail because they assert that logger.txn_id is None, when previous tests cause txn_id to be set. This coupling is masked when test_daemon.py is included because it reloads the common/utils module, and is executed just before test.unit.common.test_utils. The coupling can be observed by running, for example: nosetests ./test/unit/common/middleware/test_except.py ./test/unit/common/test_utils.py or nosetests ./test/unit/proxy ./test/unit/common/test_utils.py The failing tests should reset logger thread_locals before making assertions. test_utils.reset_loggers() attempts this but is broken because it sets thread_local on the wrong object. Changes in this patch: - fix reset_loggers() to reset the LogAdapter thread local state - add a reset_logger_state decorator to call reset_loggers before and after a decorated method - use the decorator for increased isolation of tests that previously called reset_loggers only on exit Change-Id: If9aa781a2dd1929a47ef69322ec8c53263d47660
2015-02-11 17:49:41 +00:00
logger.txn_id = '1234'
logger.client_ip = '1.2.3.4'
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(logger.thread_locals, ('1234', '1.2.3.4'))
Fix logger test coupling Two tests fail if test/unit/common/test_daemon.py is excluded from the test set: test.unit.common.test_utils.TestUtils.test_swift_log_formatter test.unit.common.test_utils.TestStatsdLoggingDelegation.test_thread_locals They fail because they assert that logger.txn_id is None, when previous tests cause txn_id to be set. This coupling is masked when test_daemon.py is included because it reloads the common/utils module, and is executed just before test.unit.common.test_utils. The coupling can be observed by running, for example: nosetests ./test/unit/common/middleware/test_except.py ./test/unit/common/test_utils.py or nosetests ./test/unit/proxy ./test/unit/common/test_utils.py The failing tests should reset logger thread_locals before making assertions. test_utils.reset_loggers() attempts this but is broken because it sets thread_local on the wrong object. Changes in this patch: - fix reset_loggers() to reset the LogAdapter thread local state - add a reset_logger_state decorator to call reset_loggers before and after a decorated method - use the decorator for increased isolation of tests that previously called reset_loggers only on exit Change-Id: If9aa781a2dd1929a47ef69322ec8c53263d47660
2015-02-11 17:49:41 +00:00
logger.txn_id = '5678'
logger.client_ip = '5.6.7.8'
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(logger.thread_locals, ('5678', '5.6.7.8'))
Honor sample_rate in StatsD logging. It's there to let administrators turn down the barrage of stats data that StatsD must cope with, but it wasn't actually honored. Worse, if the sample rate was set to e.g. 0.2, the stats would all be multiplied by its inverse, e.g. 2. This patch actually drops packets when sample_rate < 1, so you get correct measurements. Fortunately, the default sample rate is 1 (i.e. drop nothing), and multiplying by 1/1 doesn't change anything, so stats with the default sample rate of 1.0 are, and have been, just fine. Fixes bug 1065643. Also, make the two touched files compliant with pep8 v1.3.3. Change-Id: I66663144009ae4c9ee96f6a111745d8f5d2f5ca3
2012-10-11 14:04:02 -07:00
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
def test_no_fdatasync(self):
called = []
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
Convert all old-style classes to new-style This cleanup has been slowly happening for a while; let's finish it. Change-Id: I1561e3540d524834e0cc5bc725ab80936eae1f0e
2014-03-03 17:28:48 -08:00
class NoFdatasync(object):
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
pass
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
def fsync(fd):
called.append(fd)
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
with patch('swift.common.utils.os', NoFdatasync()):
with patch('swift.common.utils.fsync', fsync):
utils.fdatasync(12345)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(called, [12345])
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
def test_yes_fdatasync(self):
called = []
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
Convert all old-style classes to new-style This cleanup has been slowly happening for a while; let's finish it. Change-Id: I1561e3540d524834e0cc5bc725ab80936eae1f0e
2014-03-03 17:28:48 -08:00
class YesFdatasync(object):
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
def fdatasync(self, fd):
called.append(fd)
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
with patch('swift.common.utils.os', YesFdatasync()):
utils.fdatasync(12345)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(called, [12345])
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
def test_fsync_bad_fullsync(self):
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
Convert all old-style classes to new-style This cleanup has been slowly happening for a while; let's finish it. Change-Id: I1561e3540d524834e0cc5bc725ab80936eae1f0e
2014-03-03 17:28:48 -08:00
class FCNTL(object):
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
F_FULLSYNC = 123
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
def fcntl(self, fd, op):
raise IOError(18)
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
with patch('swift.common.utils.fcntl', FCNTL()):
self.assertRaises(OSError, lambda: utils.fsync(12345))
def test_fsync_f_fullsync(self):
called = []
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
Convert all old-style classes to new-style This cleanup has been slowly happening for a while; let's finish it. Change-Id: I1561e3540d524834e0cc5bc725ab80936eae1f0e
2014-03-03 17:28:48 -08:00
class FCNTL(object):
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
F_FULLSYNC = 123
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
def fcntl(self, fd, op):
called[:] = [fd, op]
return 0
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
with patch('swift.common.utils.fcntl', FCNTL()):
utils.fsync(12345)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(called, [12345, 123])
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
def test_fsync_no_fullsync(self):
called = []
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
Convert all old-style classes to new-style This cleanup has been slowly happening for a while; let's finish it. Change-Id: I1561e3540d524834e0cc5bc725ab80936eae1f0e
2014-03-03 17:28:48 -08:00
class FCNTL(object):
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
pass
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
def fsync(fd):
called.append(fd)
Remove extra lstat() calls from check_mount The os.path.exists call performs an lstat, but os.path.ismount already performs the same check. However, it performs a separate lstat() call to check for a symlink, which we remove as well, cutting the number performed in half. Sample program to be straced for comparison: from swift.common.constraints import check_mount import os os.write(1, "Starting\n") if check_mount("/", "tmp"): os.write(1, "Mounted\n") Here is the output of a check on a mounted file system (common case) using the new method: ---- strace new ---- write(1, "Starting\n", 9) = 9 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 ---- strace old ---- write(1, "Starting\n", 9) = 9 stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=8460, ...}) = 0 lstat("/tmp/..", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 write(1, "Mounted\n", 8) = 8 Change-Id: I027c862a2b7d9ff99d7f61bd43ccc0825dba525c Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-07-19 11:34:12 -04:00
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
with patch('swift.common.utils.fcntl', FCNTL()):
with patch('os.fsync', fsync):
utils.fsync(12345)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(called, [12345])
make swift fsync Swift never fsyncs, it only fdatasyncs. That is dumb, we have important metadata we need to save. Also, the code was weird and had no tests. Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-26 02:37:11 -08:00
Adding StatsD logging to Swift. Documentation, including a list of metrics reported and their semantics, is in the Admin Guide in a new section, "Reporting Metrics to StatsD". An optional "metric prefix" may be configured which will be prepended to every metric name sent to StatsD. Here is the rationale for doing a deep integration like this versus only sending metrics to StatsD in middleware. It's the only way to report some internal activities of Swift in a real-time manner. So to have one way of reporting to StatsD and one place/style of configuration, even some things (like, say, timing of PUT requests into the proxy-server) which could be logged via middleware are consistently logged the same way (deep integration via the logger delegate methods). When log_statsd_host is configured, get_logger() injects a swift.common.utils.StatsdClient object into the logger as logger.statsd_client. Then a set of delegate methods on LogAdapter either pass through to the StatsdClient object or become no-ops. This allows StatsD logging to look like: self.logger.increment('some.metric.here') and do the right thing in all cases and with no messy conditional logic. I wanted to use the pystatsd module for the StatsD client, but the version on PyPi is lagging the git repo (and is missing both the prefix functionality and timing_since() method). So I wrote my swift.common.utils.StatsdClient. The interface is the same as pystatsd.Client, but the code was written from scratch. It's pretty simple, and the tests I added cover it. This also frees Swift from an optional dependency on the pystatsd module, making this feature easier to enable. There's test coverage for the new code and all existing tests continue to pass. Refactored out _one_audit_pass() method in swift/account/auditor.py and swift/container/auditor.py. Fixed some misc. PEP8 violations. Misc test cleanups and refactorings (particularly the way "fake logging" is handled). Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-04-01 16:47:08 -07:00
Make ThreadPools deallocatable. Currently, a ThreadPool acquires resources that last until process exit. You can let the ThreadPool go out of scope, but that doesn't terminate the worker threads or close file descriptors or anything. This commit makes it so you can .terminate() a ThreadPool object and get its resources back. Also, after you call .terminate(), trying to use the ThreadPool raises an exception so you know you've goofed. I have some internal code that could really use this, plus it makes the unit test run not leak resources, which is nice. Change-Id: Ibf7c6dc14c14f379421a79afb6c90a5e64b235fa
2015-01-07 16:50:58 -08:00
class TestThreadPool(unittest.TestCase):
def setUp(self):
self.tp = None
def tearDown(self):
if self.tp:
self.tp.terminate()
def _pipe_count(self):
# Counts the number of pipes that this process owns.
fd_dir = "/proc/%d/fd" % os.getpid()
def is_pipe(path):
try:
stat_result = os.stat(path)
return stat.S_ISFIFO(stat_result.st_mode)
except OSError:
return False
return len([fd for fd in os.listdir(fd_dir)
if is_pipe(os.path.join(fd_dir, fd))])
Use threadpools in the object server for performance. Without a (per-disk) threadpool, requests to a slow disk would affect all clients by blocking the entire eventlet reactor on read/write/etc. The slower the disk, the worse the performance. On an object server, you frequently have at least one slow disk due to auditing and replication activity sucking up all the available IO. By kicking those blocking calls out to a separate OS thread, we let the eventlet reactor make progress in other greenthreads, and by having a per-disk pool, we ensure that one slow disk can't suck up all the resources of an entire object server. There were a few blocking calls that were done with eventlet.tpool, but that's a fixed-size global threadpool, so I moved them to the per-disk threadpools. If the object server is configured not to use per-disk threadpools, (i.e. threads_per_disk = 0, which is the default), those call sites will still ultimately end up using eventlet.tpool.execute. You won't end up blocking a whole object server while waiting for a huge fsync. If you decide not to use threadpools, the only extra overhead should be a few extra Python function calls here and there. This is accomplished by setting threads_per_disk = 0 in the config. blueprint concurrent-disk-io Change-Id: I490f8753d926fdcee3a0c65c5aaf715bc2b7c290
2013-03-19 22:27:21 -07:00
def _thread_id(self):
return threading.current_thread().ident
def _capture_args(self, *args, **kwargs):
return {'args': args, 'kwargs': kwargs}
def _raise_valueerror(self):
return int('fishcakes')
def test_run_in_thread_with_threads(self):
Make ThreadPools deallocatable. Currently, a ThreadPool acquires resources that last until process exit. You can let the ThreadPool go out of scope, but that doesn't terminate the worker threads or close file descriptors or anything. This commit makes it so you can .terminate() a ThreadPool object and get its resources back. Also, after you call .terminate(), trying to use the ThreadPool raises an exception so you know you've goofed. I have some internal code that could really use this, plus it makes the unit test run not leak resources, which is nice. Change-Id: Ibf7c6dc14c14f379421a79afb6c90a5e64b235fa
2015-01-07 16:50:58 -08:00
tp = self.tp = utils.ThreadPool(1)
Use threadpools in the object server for performance. Without a (per-disk) threadpool, requests to a slow disk would affect all clients by blocking the entire eventlet reactor on read/write/etc. The slower the disk, the worse the performance. On an object server, you frequently have at least one slow disk due to auditing and replication activity sucking up all the available IO. By kicking those blocking calls out to a separate OS thread, we let the eventlet reactor make progress in other greenthreads, and by having a per-disk pool, we ensure that one slow disk can't suck up all the resources of an entire object server. There were a few blocking calls that were done with eventlet.tpool, but that's a fixed-size global threadpool, so I moved them to the per-disk threadpools. If the object server is configured not to use per-disk threadpools, (i.e. threads_per_disk = 0, which is the default), those call sites will still ultimately end up using eventlet.tpool.execute. You won't end up blocking a whole object server while waiting for a huge fsync. If you decide not to use threadpools, the only extra overhead should be a few extra Python function calls here and there. This is accomplished by setting threads_per_disk = 0 in the config. blueprint concurrent-disk-io Change-Id: I490f8753d926fdcee3a0c65c5aaf715bc2b7c290
2013-03-19 22:27:21 -07:00
my_id = self._thread_id()
other_id = tp.run_in_thread(self._thread_id)
pep8 fix: assertNotEquals -> assertNotEqual assertNotEquals is deprecated in py3 Change-Id: Ib611351987bed1199fb8f73a750955a61d022d0a
2015-08-31 21:49:49 +05:30
self.assertNotEqual(my_id, other_id)
Use threadpools in the object server for performance. Without a (per-disk) threadpool, requests to a slow disk would affect all clients by blocking the entire eventlet reactor on read/write/etc. The slower the disk, the worse the performance. On an object server, you frequently have at least one slow disk due to auditing and replication activity sucking up all the available IO. By kicking those blocking calls out to a separate OS thread, we let the eventlet reactor make progress in other greenthreads, and by having a per-disk pool, we ensure that one slow disk can't suck up all the resources of an entire object server. There were a few blocking calls that were done with eventlet.tpool, but that's a fixed-size global threadpool, so I moved them to the per-disk threadpools. If the object server is configured not to use per-disk threadpools, (i.e. threads_per_disk = 0, which is the default), those call sites will still ultimately end up using eventlet.tpool.execute. You won't end up blocking a whole object server while waiting for a huge fsync. If you decide not to use threadpools, the only extra overhead should be a few extra Python function calls here and there. This is accomplished by setting threads_per_disk = 0 in the config. blueprint concurrent-disk-io Change-Id: I490f8753d926fdcee3a0c65c5aaf715bc2b7c290
2013-03-19 22:27:21 -07:00
result = tp.run_in_thread(self._capture_args, 1, 2, bert='ernie')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(result, {'args': (1, 2),
'kwargs': {'bert': 'ernie'}})
Use threadpools in the object server for performance. Without a (per-disk) threadpool, requests to a slow disk would affect all clients by blocking the entire eventlet reactor on read/write/etc. The slower the disk, the worse the performance. On an object server, you frequently have at least one slow disk due to auditing and replication activity sucking up all the available IO. By kicking those blocking calls out to a separate OS thread, we let the eventlet reactor make progress in other greenthreads, and by having a per-disk pool, we ensure that one slow disk can't suck up all the resources of an entire object server. There were a few blocking calls that were done with eventlet.tpool, but that's a fixed-size global threadpool, so I moved them to the per-disk threadpools. If the object server is configured not to use per-disk threadpools, (i.e. threads_per_disk = 0, which is the default), those call sites will still ultimately end up using eventlet.tpool.execute. You won't end up blocking a whole object server while waiting for a huge fsync. If you decide not to use threadpools, the only extra overhead should be a few extra Python function calls here and there. This is accomplished by setting threads_per_disk = 0 in the config. blueprint concurrent-disk-io Change-Id: I490f8753d926fdcee3a0c65c5aaf715bc2b7c290
2013-03-19 22:27:21 -07:00
caught = False
try:
tp.run_in_thread(self._raise_valueerror)
except ValueError:
caught = True
self.assertTrue(caught)
def test_force_run_in_thread_with_threads(self):
# with nthreads > 0, force_run_in_thread looks just like run_in_thread
Make ThreadPools deallocatable. Currently, a ThreadPool acquires resources that last until process exit. You can let the ThreadPool go out of scope, but that doesn't terminate the worker threads or close file descriptors or anything. This commit makes it so you can .terminate() a ThreadPool object and get its resources back. Also, after you call .terminate(), trying to use the ThreadPool raises an exception so you know you've goofed. I have some internal code that could really use this, plus it makes the unit test run not leak resources, which is nice. Change-Id: Ibf7c6dc14c14f379421a79afb6c90a5e64b235fa
2015-01-07 16:50:58 -08:00
tp = self.tp = utils.ThreadPool(1)
Use threadpools in the object server for performance. Without a (per-disk) threadpool, requests to a slow disk would affect all clients by blocking the entire eventlet reactor on read/write/etc. The slower the disk, the worse the performance. On an object server, you frequently have at least one slow disk due to auditing and replication activity sucking up all the available IO. By kicking those blocking calls out to a separate OS thread, we let the eventlet reactor make progress in other greenthreads, and by having a per-disk pool, we ensure that one slow disk can't suck up all the resources of an entire object server. There were a few blocking calls that were done with eventlet.tpool, but that's a fixed-size global threadpool, so I moved them to the per-disk threadpools. If the object server is configured not to use per-disk threadpools, (i.e. threads_per_disk = 0, which is the default), those call sites will still ultimately end up using eventlet.tpool.execute. You won't end up blocking a whole object server while waiting for a huge fsync. If you decide not to use threadpools, the only extra overhead should be a few extra Python function calls here and there. This is accomplished by setting threads_per_disk = 0 in the config. blueprint concurrent-disk-io Change-Id: I490f8753d926fdcee3a0c65c5aaf715bc2b7c290
2013-03-19 22:27:21 -07:00
my_id = self._thread_id()
other_id = tp.force_run_in_thread(self._thread_id)
pep8 fix: assertNotEquals -> assertNotEqual assertNotEquals is deprecated in py3 Change-Id: Ib611351987bed1199fb8f73a750955a61d022d0a
2015-08-31 21:49:49 +05:30
self.assertNotEqual(my_id, other_id)
Use threadpools in the object server for performance. Without a (per-disk) threadpool, requests to a slow disk would affect all clients by blocking the entire eventlet reactor on read/write/etc. The slower the disk, the worse the performance. On an object server, you frequently have at least one slow disk due to auditing and replication activity sucking up all the available IO. By kicking those blocking calls out to a separate OS thread, we let the eventlet reactor make progress in other greenthreads, and by having a per-disk pool, we ensure that one slow disk can't suck up all the resources of an entire object server. There were a few blocking calls that were done with eventlet.tpool, but that's a fixed-size global threadpool, so I moved them to the per-disk threadpools. If the object server is configured not to use per-disk threadpools, (i.e. threads_per_disk = 0, which is the default), those call sites will still ultimately end up using eventlet.tpool.execute. You won't end up blocking a whole object server while waiting for a huge fsync. If you decide not to use threadpools, the only extra overhead should be a few extra Python function calls here and there. This is accomplished by setting threads_per_disk = 0 in the config. blueprint concurrent-disk-io Change-Id: I490f8753d926fdcee3a0c65c5aaf715bc2b7c290
2013-03-19 22:27:21 -07:00
result = tp.force_run_in_thread(self._capture_args, 1, 2, bert='ernie')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(result, {'args': (1, 2),
'kwargs': {'bert': 'ernie'}})
Preserve tracebacks from run_in_thread Now the traceback goes all the way down to where the exception came from, not just down to run_in_thread. Better for debugging. Change-Id: Iac6acb843a6ecf51ea2672a563d80fa43d731f23
2013-12-17 16:11:26 -08:00
self.assertRaises(ValueError, tp.force_run_in_thread,
self._raise_valueerror)
Use threadpools in the object server for performance. Without a (per-disk) threadpool, requests to a slow disk would affect all clients by blocking the entire eventlet reactor on read/write/etc. The slower the disk, the worse the performance. On an object server, you frequently have at least one slow disk due to auditing and replication activity sucking up all the available IO. By kicking those blocking calls out to a separate OS thread, we let the eventlet reactor make progress in other greenthreads, and by having a per-disk pool, we ensure that one slow disk can't suck up all the resources of an entire object server. There were a few blocking calls that were done with eventlet.tpool, but that's a fixed-size global threadpool, so I moved them to the per-disk threadpools. If the object server is configured not to use per-disk threadpools, (i.e. threads_per_disk = 0, which is the default), those call sites will still ultimately end up using eventlet.tpool.execute. You won't end up blocking a whole object server while waiting for a huge fsync. If you decide not to use threadpools, the only extra overhead should be a few extra Python function calls here and there. This is accomplished by setting threads_per_disk = 0 in the config. blueprint concurrent-disk-io Change-Id: I490f8753d926fdcee3a0c65c5aaf715bc2b7c290
2013-03-19 22:27:21 -07:00
def test_run_in_thread_without_threads(self):
# with zero threads, run_in_thread doesn't actually do so
tp = utils.ThreadPool(0)
my_id = self._thread_id()
other_id = tp.run_in_thread(self._thread_id)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(my_id, other_id)
Use threadpools in the object server for performance. Without a (per-disk) threadpool, requests to a slow disk would affect all clients by blocking the entire eventlet reactor on read/write/etc. The slower the disk, the worse the performance. On an object server, you frequently have at least one slow disk due to auditing and replication activity sucking up all the available IO. By kicking those blocking calls out to a separate OS thread, we let the eventlet reactor make progress in other greenthreads, and by having a per-disk pool, we ensure that one slow disk can't suck up all the resources of an entire object server. There were a few blocking calls that were done with eventlet.tpool, but that's a fixed-size global threadpool, so I moved them to the per-disk threadpools. If the object server is configured not to use per-disk threadpools, (i.e. threads_per_disk = 0, which is the default), those call sites will still ultimately end up using eventlet.tpool.execute. You won't end up blocking a whole object server while waiting for a huge fsync. If you decide not to use threadpools, the only extra overhead should be a few extra Python function calls here and there. This is accomplished by setting threads_per_disk = 0 in the config. blueprint concurrent-disk-io Change-Id: I490f8753d926fdcee3a0c65c5aaf715bc2b7c290
2013-03-19 22:27:21 -07:00
result = tp.run_in_thread(self._capture_args, 1, 2, bert='ernie')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(result, {'args': (1, 2),
'kwargs': {'bert': 'ernie'}})
Preserve tracebacks from run_in_thread Now the traceback goes all the way down to where the exception came from, not just down to run_in_thread. Better for debugging. Change-Id: Iac6acb843a6ecf51ea2672a563d80fa43d731f23
2013-12-17 16:11:26 -08:00
self.assertRaises(ValueError, tp.run_in_thread,
self._raise_valueerror)
Use threadpools in the object server for performance. Without a (per-disk) threadpool, requests to a slow disk would affect all clients by blocking the entire eventlet reactor on read/write/etc. The slower the disk, the worse the performance. On an object server, you frequently have at least one slow disk due to auditing and replication activity sucking up all the available IO. By kicking those blocking calls out to a separate OS thread, we let the eventlet reactor make progress in other greenthreads, and by having a per-disk pool, we ensure that one slow disk can't suck up all the resources of an entire object server. There were a few blocking calls that were done with eventlet.tpool, but that's a fixed-size global threadpool, so I moved them to the per-disk threadpools. If the object server is configured not to use per-disk threadpools, (i.e. threads_per_disk = 0, which is the default), those call sites will still ultimately end up using eventlet.tpool.execute. You won't end up blocking a whole object server while waiting for a huge fsync. If you decide not to use threadpools, the only extra overhead should be a few extra Python function calls here and there. This is accomplished by setting threads_per_disk = 0 in the config. blueprint concurrent-disk-io Change-Id: I490f8753d926fdcee3a0c65c5aaf715bc2b7c290
2013-03-19 22:27:21 -07:00
def test_force_run_in_thread_without_threads(self):
# with zero threads, force_run_in_thread uses eventlet.tpool
tp = utils.ThreadPool(0)
my_id = self._thread_id()
other_id = tp.force_run_in_thread(self._thread_id)
pep8 fix: assertNotEquals -> assertNotEqual assertNotEquals is deprecated in py3 Change-Id: Ib611351987bed1199fb8f73a750955a61d022d0a
2015-08-31 21:49:49 +05:30
self.assertNotEqual(my_id, other_id)
Use threadpools in the object server for performance. Without a (per-disk) threadpool, requests to a slow disk would affect all clients by blocking the entire eventlet reactor on read/write/etc. The slower the disk, the worse the performance. On an object server, you frequently have at least one slow disk due to auditing and replication activity sucking up all the available IO. By kicking those blocking calls out to a separate OS thread, we let the eventlet reactor make progress in other greenthreads, and by having a per-disk pool, we ensure that one slow disk can't suck up all the resources of an entire object server. There were a few blocking calls that were done with eventlet.tpool, but that's a fixed-size global threadpool, so I moved them to the per-disk threadpools. If the object server is configured not to use per-disk threadpools, (i.e. threads_per_disk = 0, which is the default), those call sites will still ultimately end up using eventlet.tpool.execute. You won't end up blocking a whole object server while waiting for a huge fsync. If you decide not to use threadpools, the only extra overhead should be a few extra Python function calls here and there. This is accomplished by setting threads_per_disk = 0 in the config. blueprint concurrent-disk-io Change-Id: I490f8753d926fdcee3a0c65c5aaf715bc2b7c290
2013-03-19 22:27:21 -07:00
result = tp.force_run_in_thread(self._capture_args, 1, 2, bert='ernie')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(result, {'args': (1, 2),
'kwargs': {'bert': 'ernie'}})
Preserve tracebacks from run_in_thread Now the traceback goes all the way down to where the exception came from, not just down to run_in_thread. Better for debugging. Change-Id: Iac6acb843a6ecf51ea2672a563d80fa43d731f23
2013-12-17 16:11:26 -08:00
self.assertRaises(ValueError, tp.force_run_in_thread,
self._raise_valueerror)
def test_preserving_stack_trace_from_thread(self):
def gamma():
return 1 / 0 # ZeroDivisionError
def beta():
return gamma()
def alpha():
return beta()
Make ThreadPools deallocatable. Currently, a ThreadPool acquires resources that last until process exit. You can let the ThreadPool go out of scope, but that doesn't terminate the worker threads or close file descriptors or anything. This commit makes it so you can .terminate() a ThreadPool object and get its resources back. Also, after you call .terminate(), trying to use the ThreadPool raises an exception so you know you've goofed. I have some internal code that could really use this, plus it makes the unit test run not leak resources, which is nice. Change-Id: Ibf7c6dc14c14f379421a79afb6c90a5e64b235fa
2015-01-07 16:50:58 -08:00
tp = self.tp = utils.ThreadPool(1)
Use threadpools in the object server for performance. Without a (per-disk) threadpool, requests to a slow disk would affect all clients by blocking the entire eventlet reactor on read/write/etc. The slower the disk, the worse the performance. On an object server, you frequently have at least one slow disk due to auditing and replication activity sucking up all the available IO. By kicking those blocking calls out to a separate OS thread, we let the eventlet reactor make progress in other greenthreads, and by having a per-disk pool, we ensure that one slow disk can't suck up all the resources of an entire object server. There were a few blocking calls that were done with eventlet.tpool, but that's a fixed-size global threadpool, so I moved them to the per-disk threadpools. If the object server is configured not to use per-disk threadpools, (i.e. threads_per_disk = 0, which is the default), those call sites will still ultimately end up using eventlet.tpool.execute. You won't end up blocking a whole object server while waiting for a huge fsync. If you decide not to use threadpools, the only extra overhead should be a few extra Python function calls here and there. This is accomplished by setting threads_per_disk = 0 in the config. blueprint concurrent-disk-io Change-Id: I490f8753d926fdcee3a0c65c5aaf715bc2b7c290
2013-03-19 22:27:21 -07:00
try:
Preserve tracebacks from run_in_thread Now the traceback goes all the way down to where the exception came from, not just down to run_in_thread. Better for debugging. Change-Id: Iac6acb843a6ecf51ea2672a563d80fa43d731f23
2013-12-17 16:11:26 -08:00
tp.run_in_thread(alpha)
except ZeroDivisionError:
# NB: format is (filename, line number, function name, text)
tb_func = [elem[2] for elem
sys.exc_type/exc_value/exc_traceback are Deprecated sys.exc_info() contains a tuple of these three. Change-Id: I530cbeb37c43da98b4924db41f6604871077bd47
2015-07-12 14:48:35 +05:30
in traceback.extract_tb(sys.exc_info()[2])]
Preserve tracebacks from run_in_thread Now the traceback goes all the way down to where the exception came from, not just down to run_in_thread. Better for debugging. Change-Id: Iac6acb843a6ecf51ea2672a563d80fa43d731f23
2013-12-17 16:11:26 -08:00
else:
self.fail("Expected ZeroDivisionError")
self.assertEqual(tb_func[-1], "gamma")
self.assertEqual(tb_func[-2], "beta")
self.assertEqual(tb_func[-3], "alpha")
# omit the middle; what's important is that the start and end are
# included, not the exact names of helper methods
self.assertEqual(tb_func[1], "run_in_thread")
self.assertEqual(tb_func[0], "test_preserving_stack_trace_from_thread")
Use threadpools in the object server for performance. Without a (per-disk) threadpool, requests to a slow disk would affect all clients by blocking the entire eventlet reactor on read/write/etc. The slower the disk, the worse the performance. On an object server, you frequently have at least one slow disk due to auditing and replication activity sucking up all the available IO. By kicking those blocking calls out to a separate OS thread, we let the eventlet reactor make progress in other greenthreads, and by having a per-disk pool, we ensure that one slow disk can't suck up all the resources of an entire object server. There were a few blocking calls that were done with eventlet.tpool, but that's a fixed-size global threadpool, so I moved them to the per-disk threadpools. If the object server is configured not to use per-disk threadpools, (i.e. threads_per_disk = 0, which is the default), those call sites will still ultimately end up using eventlet.tpool.execute. You won't end up blocking a whole object server while waiting for a huge fsync. If you decide not to use threadpools, the only extra overhead should be a few extra Python function calls here and there. This is accomplished by setting threads_per_disk = 0 in the config. blueprint concurrent-disk-io Change-Id: I490f8753d926fdcee3a0c65c5aaf715bc2b7c290
2013-03-19 22:27:21 -07:00
Make ThreadPools deallocatable. Currently, a ThreadPool acquires resources that last until process exit. You can let the ThreadPool go out of scope, but that doesn't terminate the worker threads or close file descriptors or anything. This commit makes it so you can .terminate() a ThreadPool object and get its resources back. Also, after you call .terminate(), trying to use the ThreadPool raises an exception so you know you've goofed. I have some internal code that could really use this, plus it makes the unit test run not leak resources, which is nice. Change-Id: Ibf7c6dc14c14f379421a79afb6c90a5e64b235fa
2015-01-07 16:50:58 -08:00
def test_terminate(self):
initial_thread_count = threading.activeCount()
initial_pipe_count = self._pipe_count()
tp = utils.ThreadPool(4)
# do some work to ensure any lazy initialization happens
tp.run_in_thread(os.path.join, 'foo', 'bar')
tp.run_in_thread(os.path.join, 'baz', 'quux')
# 4 threads in the ThreadPool, plus one pipe for IPC; this also
# serves as a sanity check that we're actually allocating some
# resources to free later
self.assertEqual(initial_thread_count, threading.activeCount() - 4)
self.assertEqual(initial_pipe_count, self._pipe_count() - 2)
tp.terminate()
self.assertEqual(initial_thread_count, threading.activeCount())
self.assertEqual(initial_pipe_count, self._pipe_count())
def test_cant_run_after_terminate(self):
tp = utils.ThreadPool(0)
tp.terminate()
self.assertRaises(ThreadPoolDead, tp.run_in_thread, lambda: 1)
self.assertRaises(ThreadPoolDead, tp.force_run_in_thread, lambda: 1)
def test_double_terminate_doesnt_crash(self):
tp = utils.ThreadPool(0)
tp.terminate()
tp.terminate()
tp = utils.ThreadPool(1)
tp.terminate()
tp.terminate()
def test_terminate_no_threads_doesnt_crash(self):
tp = utils.ThreadPool(0)
tp.terminate()
Use threadpools in the object server for performance. Without a (per-disk) threadpool, requests to a slow disk would affect all clients by blocking the entire eventlet reactor on read/write/etc. The slower the disk, the worse the performance. On an object server, you frequently have at least one slow disk due to auditing and replication activity sucking up all the available IO. By kicking those blocking calls out to a separate OS thread, we let the eventlet reactor make progress in other greenthreads, and by having a per-disk pool, we ensure that one slow disk can't suck up all the resources of an entire object server. There were a few blocking calls that were done with eventlet.tpool, but that's a fixed-size global threadpool, so I moved them to the per-disk threadpools. If the object server is configured not to use per-disk threadpools, (i.e. threads_per_disk = 0, which is the default), those call sites will still ultimately end up using eventlet.tpool.execute. You won't end up blocking a whole object server while waiting for a huge fsync. If you decide not to use threadpools, the only extra overhead should be a few extra Python function calls here and there. This is accomplished by setting threads_per_disk = 0 in the config. blueprint concurrent-disk-io Change-Id: I490f8753d926fdcee3a0c65c5aaf715bc2b7c290
2013-03-19 22:27:21 -07:00
Revert "Refactor common/utils methods to common/ondisk" This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32 Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 12:10:31 +00:00
class TestAuditLocationGenerator(unittest.TestCase):
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
def test_drive_tree_access(self):
orig_listdir = utils.listdir
def _mock_utils_listdir(path):
if 'bad_part' in path:
raise OSError(errno.EACCES)
elif 'bad_suffix' in path:
raise OSError(errno.EACCES)
elif 'bad_hash' in path:
raise OSError(errno.EACCES)
else:
return orig_listdir(path)
Fix pep8 E265 warning of hacking 0.10 Fix the warning E265 "block comment should start with '# '" added in pep 1.5. Change-Id: Ib57282e958be9c7cddffc7bca34fbbf1d4c460fd
2015-07-30 00:16:06 +02:00
# Check Raise on Bad partition
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
tmpdir = mkdtemp()
data = os.path.join(tmpdir, "drive", "data")
os.makedirs(data)
obj_path = os.path.join(data, "bad_part")
with open(obj_path, "w"):
pass
part1 = os.path.join(data, "partition1")
os.makedirs(part1)
part2 = os.path.join(data, "partition2")
os.makedirs(part2)
with patch('swift.common.utils.listdir', _mock_utils_listdir):
audit = lambda: list(utils.audit_location_generator(
tmpdir, "data", mount_check=False))
self.assertRaises(OSError, audit)
Fixes unit tests to clean up temporary directories This patch fixes the unit tests to remove the temporary directories created during run of unit tests. Some of unit tests did not tear down correctly, whatever it had set it up for running. This would over period of time bloat up the tmp directory. As on date, there were around 49 tmp directories left uncleared per round of unit tests. This patch fixes it. Change-Id: If591375ca9cc87d52c7c9c6dc16c9fb4b49e99fc
2014-09-25 00:48:49 +05:30
rmtree(tmpdir)
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
Fix pep8 E265 warning of hacking 0.10 Fix the warning E265 "block comment should start with '# '" added in pep 1.5. Change-Id: Ib57282e958be9c7cddffc7bca34fbbf1d4c460fd
2015-07-30 00:16:06 +02:00
# Check Raise on Bad Suffix
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
tmpdir = mkdtemp()
data = os.path.join(tmpdir, "drive", "data")
os.makedirs(data)
part1 = os.path.join(data, "partition1")
os.makedirs(part1)
part2 = os.path.join(data, "partition2")
os.makedirs(part2)
obj_path = os.path.join(part1, "bad_suffix")
with open(obj_path, 'w'):
pass
suffix = os.path.join(part2, "suffix")
os.makedirs(suffix)
with patch('swift.common.utils.listdir', _mock_utils_listdir):
audit = lambda: list(utils.audit_location_generator(
tmpdir, "data", mount_check=False))
self.assertRaises(OSError, audit)
Fixes unit tests to clean up temporary directories This patch fixes the unit tests to remove the temporary directories created during run of unit tests. Some of unit tests did not tear down correctly, whatever it had set it up for running. This would over period of time bloat up the tmp directory. As on date, there were around 49 tmp directories left uncleared per round of unit tests. This patch fixes it. Change-Id: If591375ca9cc87d52c7c9c6dc16c9fb4b49e99fc
2014-09-25 00:48:49 +05:30
rmtree(tmpdir)
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
Fix pep8 E265 warning of hacking 0.10 Fix the warning E265 "block comment should start with '# '" added in pep 1.5. Change-Id: Ib57282e958be9c7cddffc7bca34fbbf1d4c460fd
2015-07-30 00:16:06 +02:00
# Check Raise on Bad Hash
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
tmpdir = mkdtemp()
data = os.path.join(tmpdir, "drive", "data")
os.makedirs(data)
part1 = os.path.join(data, "partition1")
os.makedirs(part1)
suffix = os.path.join(part1, "suffix")
os.makedirs(suffix)
hash1 = os.path.join(suffix, "hash1")
os.makedirs(hash1)
obj_path = os.path.join(suffix, "bad_hash")
with open(obj_path, 'w'):
pass
with patch('swift.common.utils.listdir', _mock_utils_listdir):
audit = lambda: list(utils.audit_location_generator(
tmpdir, "data", mount_check=False))
self.assertRaises(OSError, audit)
Fixes unit tests to clean up temporary directories This patch fixes the unit tests to remove the temporary directories created during run of unit tests. Some of unit tests did not tear down correctly, whatever it had set it up for running. This would over period of time bloat up the tmp directory. As on date, there were around 49 tmp directories left uncleared per round of unit tests. This patch fixes it. Change-Id: If591375ca9cc87d52c7c9c6dc16c9fb4b49e99fc
2014-09-25 00:48:49 +05:30
rmtree(tmpdir)
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
def test_non_dir_drive(self):
with temptree([]) as tmpdir:
logger = FakeLogger()
data = os.path.join(tmpdir, "drive", "data")
os.makedirs(data)
Fix pep8 E265 warning of hacking 0.10 Fix the warning E265 "block comment should start with '# '" added in pep 1.5. Change-Id: Ib57282e958be9c7cddffc7bca34fbbf1d4c460fd
2015-07-30 00:16:06 +02:00
# Create a file, that represents a non-dir drive
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
open(os.path.join(tmpdir, 'asdf'), 'w')
locations = utils.audit_location_generator(
tmpdir, "data", mount_check=False, logger=logger
)
self.assertEqual(list(locations), [])
self.assertEqual(1, len(logger.get_lines_for_level('warning')))
Fix pep8 E265 warning of hacking 0.10 Fix the warning E265 "block comment should start with '# '" added in pep 1.5. Change-Id: Ib57282e958be9c7cddffc7bca34fbbf1d4c460fd
2015-07-30 00:16:06 +02:00
# Test without the logger
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
locations = utils.audit_location_generator(
tmpdir, "data", mount_check=False
)
self.assertEqual(list(locations), [])
def test_mount_check_drive(self):
with temptree([]) as tmpdir:
logger = FakeLogger()
data = os.path.join(tmpdir, "drive", "data")
os.makedirs(data)
Fix pep8 E265 warning of hacking 0.10 Fix the warning E265 "block comment should start with '# '" added in pep 1.5. Change-Id: Ib57282e958be9c7cddffc7bca34fbbf1d4c460fd
2015-07-30 00:16:06 +02:00
# Create a file, that represents a non-dir drive
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
open(os.path.join(tmpdir, 'asdf'), 'w')
locations = utils.audit_location_generator(
tmpdir, "data", mount_check=True, logger=logger
)
self.assertEqual(list(locations), [])
self.assertEqual(2, len(logger.get_lines_for_level('warning')))
Fix pep8 E265 warning of hacking 0.10 Fix the warning E265 "block comment should start with '# '" added in pep 1.5. Change-Id: Ib57282e958be9c7cddffc7bca34fbbf1d4c460fd
2015-07-30 00:16:06 +02:00
# Test without the logger
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
locations = utils.audit_location_generator(
tmpdir, "data", mount_check=True
)
self.assertEqual(list(locations), [])
Revert "Refactor common/utils methods to common/ondisk" This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32 Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 12:10:31 +00:00
def test_non_dir_contents(self):
with temptree([]) as tmpdir:
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
logger = FakeLogger()
Revert "Refactor common/utils methods to common/ondisk" This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32 Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 12:10:31 +00:00
data = os.path.join(tmpdir, "drive", "data")
os.makedirs(data)
with open(os.path.join(data, "partition1"), "w"):
pass
partition = os.path.join(data, "partition2")
os.makedirs(partition)
with open(os.path.join(partition, "suffix1"), "w"):
pass
suffix = os.path.join(partition, "suffix2")
os.makedirs(suffix)
with open(os.path.join(suffix, "hash1"), "w"):
pass
locations = utils.audit_location_generator(
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
tmpdir, "data", mount_check=False, logger=logger
Revert "Refactor common/utils methods to common/ondisk" This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32 Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 12:10:31 +00:00
)
self.assertEqual(list(locations), [])
Adding more unit tests for audit_location_generator Change-Id: I40410fbbb79cea8647074f703e4675364c69d930
2013-10-19 11:40:35 +08:00
def test_find_objects(self):
with temptree([]) as tmpdir:
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
expected_objs = list()
logger = FakeLogger()
Adding more unit tests for audit_location_generator Change-Id: I40410fbbb79cea8647074f703e4675364c69d930
2013-10-19 11:40:35 +08:00
data = os.path.join(tmpdir, "drive", "data")
os.makedirs(data)
Fix pep8 E265 warning of hacking 0.10 Fix the warning E265 "block comment should start with '# '" added in pep 1.5. Change-Id: Ib57282e958be9c7cddffc7bca34fbbf1d4c460fd
2015-07-30 00:16:06 +02:00
# Create a file, that represents a non-dir drive
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
open(os.path.join(tmpdir, 'asdf'), 'w')
partition = os.path.join(data, "partition1")
os.makedirs(partition)
suffix = os.path.join(partition, "suffix")
os.makedirs(suffix)
hash_path = os.path.join(suffix, "hash")
os.makedirs(hash_path)
obj_path = os.path.join(hash_path, "obj1.db")
with open(obj_path, "w"):
pass
expected_objs.append((obj_path, 'drive', 'partition1'))
Adding more unit tests for audit_location_generator Change-Id: I40410fbbb79cea8647074f703e4675364c69d930
2013-10-19 11:40:35 +08:00
partition = os.path.join(data, "partition2")
os.makedirs(partition)
suffix = os.path.join(partition, "suffix2")
os.makedirs(suffix)
hash_path = os.path.join(suffix, "hash2")
os.makedirs(hash_path)
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
obj_path = os.path.join(hash_path, "obj2.db")
Adding more unit tests for audit_location_generator Change-Id: I40410fbbb79cea8647074f703e4675364c69d930
2013-10-19 11:40:35 +08:00
with open(obj_path, "w"):
pass
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
expected_objs.append((obj_path, 'drive', 'partition2'))
Adding more unit tests for audit_location_generator Change-Id: I40410fbbb79cea8647074f703e4675364c69d930
2013-10-19 11:40:35 +08:00
locations = utils.audit_location_generator(
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
tmpdir, "data", mount_check=False, logger=logger
Adding more unit tests for audit_location_generator Change-Id: I40410fbbb79cea8647074f703e4675364c69d930
2013-10-19 11:40:35 +08:00
)
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
got_objs = list(locations)
self.assertEqual(len(got_objs), len(expected_objs))
self.assertEqual(sorted(got_objs), sorted(expected_objs))
self.assertEqual(1, len(logger.get_lines_for_level('warning')))
Adding more unit tests for audit_location_generator Change-Id: I40410fbbb79cea8647074f703e4675364c69d930
2013-10-19 11:40:35 +08:00
def test_ignore_metadata(self):
with temptree([]) as tmpdir:
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
logger = FakeLogger()
Adding more unit tests for audit_location_generator Change-Id: I40410fbbb79cea8647074f703e4675364c69d930
2013-10-19 11:40:35 +08:00
data = os.path.join(tmpdir, "drive", "data")
os.makedirs(data)
partition = os.path.join(data, "partition2")
os.makedirs(partition)
suffix = os.path.join(partition, "suffix2")
os.makedirs(suffix)
hash_path = os.path.join(suffix, "hash2")
os.makedirs(hash_path)
obj_path = os.path.join(hash_path, "obj1.dat")
with open(obj_path, "w"):
pass
meta_path = os.path.join(hash_path, "obj1.meta")
with open(meta_path, "w"):
pass
locations = utils.audit_location_generator(
Container Auditor should log a warning if the devices path contains a non-directory. If the devices path configured in container-server.conf contains a file then an uncaught exception is seen in the logs. For example if file foo exists as such /srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is not a directory is seen in the logs This patch was essentially clayg and can be found in the bug I tested it and wanted to get a feel of the openstack workflow so going through the commit process I have added a unit test as well as cleaned up and improved the unit test coverage for this module. - unit test for above fix is added - unit test to verify exceptions that are raised in the module - unit test to verify the logger's behavior - unit test to verify mount_check behavior Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844 Closes-Bug: #1317257
2014-06-02 05:32:12 +00:00
tmpdir, "data", ".dat", mount_check=False, logger=logger
Adding more unit tests for audit_location_generator Change-Id: I40410fbbb79cea8647074f703e4675364c69d930
2013-10-19 11:40:35 +08:00
)
self.assertEqual(list(locations),
[(obj_path, "drive", "partition2")])
Revert "Refactor common/utils methods to common/ondisk" This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32 Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62 Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 12:10:31 +00:00
early quorum responses Allow the proxy to respond to many types of requests as soon as it has a quorum. This can help speed up responses (without changing the results), especially when one node is acting up. I had to fix a few unit tests that no longer match the backend http requests made by our proxy. Change-Id: Ieb070dc3019e217e717b96154a7a809409bf40a5
2013-10-30 21:43:35 +00:00
class TestGreenAsyncPile(unittest.TestCase):
def test_runs_everything(self):
def run_test():
tests_ran[0] += 1
return tests_ran[0]
tests_ran = [0]
pile = utils.GreenAsyncPile(3)
Replace xrange() with six.moves.range() Patch generated by the xrange operation of the sixer tool: https://pypi.python.org/pypi/sixer Manual changes: * Fix indentation for pep8 checks * Fix TestGreenthreadSafeIterator.test_access_is_serialized of test.unit.common.test_utils: replace range(1, 11) with list(range(1, 11)) * Fix UnsafeXrange docstring, revert change Change-Id: Icb7e26135c5e57b5302b8bfe066b33cafe69fe4d
2015-05-25 18:28:02 +02:00
for x in range(3):
early quorum responses Allow the proxy to respond to many types of requests as soon as it has a quorum. This can help speed up responses (without changing the results), especially when one node is acting up. I had to fix a few unit tests that no longer match the backend http requests made by our proxy. Change-Id: Ieb070dc3019e217e717b96154a7a809409bf40a5
2013-10-30 21:43:35 +00:00
pile.spawn(run_test)
self.assertEqual(sorted(x for x in pile), [1, 2, 3])
def test_is_asynchronous(self):
def run_test(index):
events[index].wait()
return index
pile = utils.GreenAsyncPile(3)
for order in ((1, 2, 0), (0, 1, 2), (2, 1, 0), (0, 2, 1)):
events = [eventlet.event.Event(), eventlet.event.Event(),
eventlet.event.Event()]
Replace xrange() with six.moves.range() Patch generated by the xrange operation of the sixer tool: https://pypi.python.org/pypi/sixer Manual changes: * Fix indentation for pep8 checks * Fix TestGreenthreadSafeIterator.test_access_is_serialized of test.unit.common.test_utils: replace range(1, 11) with list(range(1, 11)) * Fix UnsafeXrange docstring, revert change Change-Id: Icb7e26135c5e57b5302b8bfe066b33cafe69fe4d
2015-05-25 18:28:02 +02:00
for x in range(3):
early quorum responses Allow the proxy to respond to many types of requests as soon as it has a quorum. This can help speed up responses (without changing the results), especially when one node is acting up. I had to fix a few unit tests that no longer match the backend http requests made by our proxy. Change-Id: Ieb070dc3019e217e717b96154a7a809409bf40a5
2013-10-30 21:43:35 +00:00
pile.spawn(run_test, x)
for x in order:
events[x].send()
self.assertEqual(next(pile), x)
def test_next_when_empty(self):
def run_test():
pass
pile = utils.GreenAsyncPile(3)
pile.spawn(run_test)
self.assertEqual(next(pile), None)
self.assertRaises(StopIteration, lambda: next(pile))
slightly less early quorum The early quorum change has maybe added a little bit too much eventual to the consistency of requests in Swift, and users can sometimes get unexpected results. This change gives us a knob to turn in finding the right balance, by adding a timeout where pending requests can finish after quorum is achieved. Change-Id: Ife91aaa8653e75b01313bbcf19072181739e932c
2013-11-25 18:58:34 +00:00
def test_waitall_timeout_timesout(self):
def run_test(sleep_duration):
eventlet.sleep(sleep_duration)
completed[0] += 1
return sleep_duration
completed = [0]
pile = utils.GreenAsyncPile(3)
pile.spawn(run_test, 0.1)
pile.spawn(run_test, 1.0)
Bump up a timeout in a test Got a slow crappy VM like I do? You might see this fail occasionally. Bump up the timeout a little to help it out. Change-Id: I8c0e5b99012830ea3525fa55b0811268db3da2a2
2015-05-04 15:08:51 -07:00
self.assertEqual(pile.waitall(0.5), [0.1])
slightly less early quorum The early quorum change has maybe added a little bit too much eventual to the consistency of requests in Swift, and users can sometimes get unexpected results. This change gives us a knob to turn in finding the right balance, by adding a timeout where pending requests can finish after quorum is achieved. Change-Id: Ife91aaa8653e75b01313bbcf19072181739e932c
2013-11-25 18:58:34 +00:00
self.assertEqual(completed[0], 1)
def test_waitall_timeout_completes(self):
def run_test(sleep_duration):
eventlet.sleep(sleep_duration)
completed[0] += 1
return sleep_duration
completed = [0]
pile = utils.GreenAsyncPile(3)
pile.spawn(run_test, 0.1)
pile.spawn(run_test, 0.1)
self.assertEqual(pile.waitall(0.5), [0.1, 0.1])
self.assertEqual(completed[0], 2)
Add concurrent reads option to proxy This change adds 2 new parameters to enable and control concurrent GETs in swift, these are 'concurrent_gets' and 'concurrency_timeout'. 'concurrent_gets' allows you to turn on or off concurrent GETs, when on it will set the GET/HEAD concurrency to replica count. And in the case of EC HEADs it will set it to ndata. The proxy will then serve only the first valid source to respond. This applies to all account, container and object GETs except for EC. For EC only HEAD requests are effected. It achieves this by changing the request sending mechanism to using GreenAsyncPile and green threads with a time out between each request. 'concurrency_timeout' is related to concurrent_gets. And is the amount of time to wait before firing the next thread. A value of 0 will fire at the same time (fully concurrent), setting another value will stagger the firing allowing you the ability to give a node a shorter chance to respond before firing the next. This value is a float and should be somewhere between 0 and node_timeout. The default is conn_timeout. Meaning by default it will stagger the firing. DocImpact Implements: blueprint concurrent-reads Change-Id: I789d39472ec48b22415ff9d9821b1eefab7da867
2014-08-29 17:14:46 +10:00
def test_waitfirst_only_returns_first(self):
def run_test(name):
eventlet.sleep(0)
completed.append(name)
return name
completed = []
pile = utils.GreenAsyncPile(3)
pile.spawn(run_test, 'first')
pile.spawn(run_test, 'second')
pile.spawn(run_test, 'third')
self.assertEqual(pile.waitfirst(0.5), completed[0])
# 3 still completed, but only the first was returned.
self.assertEqual(3, len(completed))
def test_wait_with_firstn(self):
def run_test(name):
eventlet.sleep(0)
completed.append(name)
return name
for first_n in [None] + list(range(6)):
completed = []
pile = utils.GreenAsyncPile(10)
for i in range(10):
pile.spawn(run_test, i)
actual = pile._wait(1, first_n)
expected_n = first_n if first_n else 10
self.assertEqual(completed[:expected_n], actual)
self.assertEqual(10, len(completed))
EC GET path: require fragments to be of same set And if they are not, exhaust the node iter to go get more. The problem without this implementation is a simple overwrite where a GET follows before the handoff has put the newer obj back on the 'alive again' node such that the proxy gets n-1 fragments of the newest set and 1 of the older. This patch bucketizes the fragments by etag and if it doesn't have enough continues to exhaust the node iterator until it has a large enough matching set. Change-Id: Ib710a133ce1be278365067fd0d6610d80f1f7372 Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Alistair Coles <alistair.coles@hp.com> Closes-Bug: 1457691
2015-08-12 13:32:50 -07:00
def test_pending(self):
pile = utils.GreenAsyncPile(3)
self.assertEqual(0, pile._pending)
for repeats in range(2):
# repeat to verify that pending will go again up after going down
for i in range(4):
pile.spawn(lambda: i)
self.assertEqual(4, pile._pending)
for i in range(3, -1, -1):
py3: Replace gen.next() with next(gen) The next() method of Python 2 generators was renamed to __next__(). Call the builtin next() function instead which works on Python 2 and Python 3. The patch was generated by the next operation of the sixer tool. Change-Id: Id12bc16cba7d9b8a283af0d392188a185abe439d
2015-10-08 15:38:36 +02:00
next(pile)
EC GET path: require fragments to be of same set And if they are not, exhaust the node iter to go get more. The problem without this implementation is a simple overwrite where a GET follows before the handoff has put the newer obj back on the 'alive again' node such that the proxy gets n-1 fragments of the newest set and 1 of the older. This patch bucketizes the fragments by etag and if it doesn't have enough continues to exhaust the node iterator until it has a large enough matching set. Change-Id: Ib710a133ce1be278365067fd0d6610d80f1f7372 Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Alistair Coles <alistair.coles@hp.com> Closes-Bug: 1457691
2015-08-12 13:32:50 -07:00
self.assertEqual(i, pile._pending)
# sanity check - the pile is empty
self.assertRaises(StopIteration, pile.next)
# pending remains 0
self.assertEqual(0, pile._pending)
early quorum responses Allow the proxy to respond to many types of requests as soon as it has a quorum. This can help speed up responses (without changing the results), especially when one node is acting up. I had to fix a few unit tests that no longer match the backend http requests made by our proxy. Change-Id: Ieb070dc3019e217e717b96154a7a809409bf40a5
2013-10-30 21:43:35 +00:00
Add LRUCache to common.utils This decorator will memonize a function using a fixed size cache that evicts the oldest entries. It also supports a maxtime paramter to configure a "time-to-live" for entries in the cache. The reconciler code uses this to cache computations of the correct storage policy index for a container for 30 seconds. DocImpact Implements: blueprint storage-policies Change-Id: I0f220869e33c461a4100b21c6324ad725da864fa
2014-04-28 17:05:43 -07:00
class TestLRUCache(unittest.TestCase):
def test_maxsize(self):
@utils.LRUCache(maxsize=10)
def f(*args):
return math.sqrt(*args)
_orig_math_sqrt = math.sqrt
# setup cache [0-10)
for i in range(10):
self.assertEqual(math.sqrt(i), f(i))
self.assertEqual(f.size(), 10)
# validate cache [0-10)
with patch('math.sqrt'):
for i in range(10):
self.assertEqual(_orig_math_sqrt(i), f(i))
self.assertEqual(f.size(), 10)
# update cache [10-20)
for i in range(10, 20):
self.assertEqual(math.sqrt(i), f(i))
# cache size is fixed
self.assertEqual(f.size(), 10)
# validate cache [10-20)
with patch('math.sqrt'):
for i in range(10, 20):
self.assertEqual(_orig_math_sqrt(i), f(i))
# validate un-cached [0-10)
with patch('math.sqrt', new=None):
for i in range(10):
self.assertRaises(TypeError, f, i)
# cache unchanged
self.assertEqual(f.size(), 10)
with patch('math.sqrt'):
for i in range(10, 20):
self.assertEqual(_orig_math_sqrt(i), f(i))
self.assertEqual(f.size(), 10)
def test_maxtime(self):
@utils.LRUCache(maxtime=30)
def f(*args):
return math.sqrt(*args)
self.assertEqual(30, f.maxtime)
_orig_math_sqrt = math.sqrt
now = time.time()
the_future = now + 31
# setup cache [0-10)
with patch('time.time', lambda: now):
for i in range(10):
self.assertEqual(math.sqrt(i), f(i))
self.assertEqual(f.size(), 10)
# validate cache [0-10)
with patch('math.sqrt'):
for i in range(10):
self.assertEqual(_orig_math_sqrt(i), f(i))
self.assertEqual(f.size(), 10)
# validate expired [0-10)
with patch('math.sqrt', new=None):
with patch('time.time', lambda: the_future):
for i in range(10):
self.assertRaises(TypeError, f, i)
# validate repopulates [0-10)
with patch('time.time', lambda: the_future):
for i in range(10):
self.assertEqual(math.sqrt(i), f(i))
# reuses cache space
self.assertEqual(f.size(), 10)
def test_set_maxtime(self):
@utils.LRUCache(maxtime=30)
def f(*args):
return math.sqrt(*args)
self.assertEqual(30, f.maxtime)
self.assertEqual(2, f(4))
self.assertEqual(1, f.size())
# expire everything
f.maxtime = -1
# validate un-cached [0-10)
with patch('math.sqrt', new=None):
self.assertRaises(TypeError, f, 4)
def test_set_maxsize(self):
@utils.LRUCache(maxsize=10)
def f(*args):
return math.sqrt(*args)
for i in range(12):
f(i)
self.assertEqual(f.size(), 10)
f.maxsize = 4
for i in range(12):
f(i)
self.assertEqual(f.size(), 4)
Foundational support for PUT and GET of erasure-coded objects This commit makes it possible to PUT an object into Swift and have it stored using erasure coding instead of replication, and also to GET the object back from Swift at a later time. This works by splitting the incoming object into a number of segments, erasure-coding each segment in turn to get fragments, then concatenating the fragments into fragment archives. Segments are 1 MiB in size, except the last, which is between 1 B and 1 MiB. +====================================================================+ | object data | +====================================================================+ | +------------------------+----------------------+ | | | v v v +===================+ +===================+ +==============+ | segment 1 | | segment 2 | ... | segment N | +===================+ +===================+ +==============+ | | | | v v /=========\ /=========\ | pyeclib | | pyeclib | ... \=========/ \=========/ | | | | +--> fragment A-1 +--> fragment A-2 | | | | | | | | | | +--> fragment B-1 +--> fragment B-2 | | | | ... ... Then, object server A gets the concatenation of fragment A-1, A-2, ..., A-N, so its .data file looks like this (called a "fragment archive"): +=====================================================================+ | fragment A-1 | fragment A-2 | ... | fragment A-N | +=====================================================================+ Since this means that the object server never sees the object data as the client sent it, we have to do a few things to ensure data integrity. First, the proxy has to check the Etag if the client provided it; the object server can't do it since the object server doesn't see the raw data. Second, if the client does not provide an Etag, the proxy computes it and uses the MIME-PUT mechanism to provide it to the object servers after the object body. Otherwise, the object would not have an Etag at all. Third, the proxy computes the MD5 of each fragment archive and sends it to the object server using the MIME-PUT mechanism. With replicated objects, the proxy checks that the Etags from all the object servers match, and if they don't, returns a 500 to the client. This mitigates the risk of data corruption in one of the proxy --> object connections, and signals to the client when it happens. With EC objects, we can't use that same mechanism, so we must send the checksum with each fragment archive to get comparable protection. On the GET path, the inverse happens: the proxy connects to a bunch of object servers (M of them, for an M+K scheme), reads one fragment at a time from each fragment archive, decodes those fragments into a segment, and serves the segment to the client. When an object server dies partway through a GET response, any partially-fetched fragment is discarded, the resumption point is wound back to the nearest fragment boundary, and the GET is retried with the next object server. GET requests for a single byterange work; GET requests for multiple byteranges do not. There are a number of things _not_ included in this commit. Some of them are listed here: * multi-range GET * deferred cleanup of old .data files * durability (daemon to reconstruct missing archives) Co-Authored-By: Alistair Coles <alistair.coles@hp.com> Co-Authored-By: Thiago da Silva <thiago@redhat.com> Co-Authored-By: John Dickinson <me@not.mn> Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com> Co-Authored-By: Paul Luse <paul.e.luse@intel.com> Co-Authored-By: Christian Schwede <christian.schwede@enovance.com> Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com> Change-Id: I9c13c03616489f8eab7dcd7c5f21237ed4cb6fd2
2014-10-22 13:18:34 -07:00
class TestParseContentRange(unittest.TestCase):
def test_good(self):
start, end, total = utils.parse_content_range("bytes 100-200/300")
self.assertEqual(start, 100)
self.assertEqual(end, 200)
self.assertEqual(total, 300)
def test_bad(self):
self.assertRaises(ValueError, utils.parse_content_range,
"100-300/500")
self.assertRaises(ValueError, utils.parse_content_range,
"bytes 100-200/aardvark")
self.assertRaises(ValueError, utils.parse_content_range,
"bytes bulbous-bouffant/4994801")
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
class TestParseContentDisposition(unittest.TestCase):
def test_basic_content_type(self):
name, attrs = utils.parse_content_disposition('text/plain')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(name, 'text/plain')
self.assertEqual(attrs, {})
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
def test_content_type_with_charset(self):
name, attrs = utils.parse_content_disposition(
'text/plain; charset=UTF8')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(name, 'text/plain')
self.assertEqual(attrs, {'charset': 'UTF8'})
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
def test_content_disposition(self):
name, attrs = utils.parse_content_disposition(
'form-data; name="somefile"; filename="test.html"')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(name, 'form-data')
self.assertEqual(attrs, {'name': 'somefile', 'filename': 'test.html'})
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
fixup!Patch of "parse_content_disposition" method to meet RFC2183 The spec of Content-Disposition does not require a space character after comma: http://www.ietf.org/rfc/rfc2183.txt Change-Id: Iff438dc36ce78c6a79bb66ab3d889a8dae7c0e1f Closes-Bug: #1458497
2015-05-25 15:13:01 +02:00
def test_content_disposition_without_white_space(self):
name, attrs = utils.parse_content_disposition(
'form-data;name="somefile";filename="test.html"')
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(name, 'form-data')
self.assertEqual(attrs, {'name': 'somefile', 'filename': 'test.html'})
fixup!Patch of "parse_content_disposition" method to meet RFC2183 The spec of Content-Disposition does not require a space character after comma: http://www.ietf.org/rfc/rfc2183.txt Change-Id: Iff438dc36ce78c6a79bb66ab3d889a8dae7c0e1f Closes-Bug: #1458497
2015-05-25 15:13:01 +02:00
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
class TestIterMultipartMimeDocuments(unittest.TestCase):
def test_bad_start(self):
it = utils.iter_multipart_mime_documents(StringIO('blah'), 'unique')
exc = None
try:
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
next(it)
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
except MimeInvalid as err:
exc = err
Foundational support for PUT and GET of erasure-coded objects This commit makes it possible to PUT an object into Swift and have it stored using erasure coding instead of replication, and also to GET the object back from Swift at a later time. This works by splitting the incoming object into a number of segments, erasure-coding each segment in turn to get fragments, then concatenating the fragments into fragment archives. Segments are 1 MiB in size, except the last, which is between 1 B and 1 MiB. +====================================================================+ | object data | +====================================================================+ | +------------------------+----------------------+ | | | v v v +===================+ +===================+ +==============+ | segment 1 | | segment 2 | ... | segment N | +===================+ +===================+ +==============+ | | | | v v /=========\ /=========\ | pyeclib | | pyeclib | ... \=========/ \=========/ | | | | +--> fragment A-1 +--> fragment A-2 | | | | | | | | | | +--> fragment B-1 +--> fragment B-2 | | | | ... ... Then, object server A gets the concatenation of fragment A-1, A-2, ..., A-N, so its .data file looks like this (called a "fragment archive"): +=====================================================================+ | fragment A-1 | fragment A-2 | ... | fragment A-N | +=====================================================================+ Since this means that the object server never sees the object data as the client sent it, we have to do a few things to ensure data integrity. First, the proxy has to check the Etag if the client provided it; the object server can't do it since the object server doesn't see the raw data. Second, if the client does not provide an Etag, the proxy computes it and uses the MIME-PUT mechanism to provide it to the object servers after the object body. Otherwise, the object would not have an Etag at all. Third, the proxy computes the MD5 of each fragment archive and sends it to the object server using the MIME-PUT mechanism. With replicated objects, the proxy checks that the Etags from all the object servers match, and if they don't, returns a 500 to the client. This mitigates the risk of data corruption in one of the proxy --> object connections, and signals to the client when it happens. With EC objects, we can't use that same mechanism, so we must send the checksum with each fragment archive to get comparable protection. On the GET path, the inverse happens: the proxy connects to a bunch of object servers (M of them, for an M+K scheme), reads one fragment at a time from each fragment archive, decodes those fragments into a segment, and serves the segment to the client. When an object server dies partway through a GET response, any partially-fetched fragment is discarded, the resumption point is wound back to the nearest fragment boundary, and the GET is retried with the next object server. GET requests for a single byterange work; GET requests for multiple byteranges do not. There are a number of things _not_ included in this commit. Some of them are listed here: * multi-range GET * deferred cleanup of old .data files * durability (daemon to reconstruct missing archives) Co-Authored-By: Alistair Coles <alistair.coles@hp.com> Co-Authored-By: Thiago da Silva <thiago@redhat.com> Co-Authored-By: John Dickinson <me@not.mn> Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com> Co-Authored-By: Paul Luse <paul.e.luse@intel.com> Co-Authored-By: Christian Schwede <christian.schwede@enovance.com> Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com> Change-Id: I9c13c03616489f8eab7dcd7c5f21237ed4cb6fd2
2014-10-22 13:18:34 -07:00
self.assertTrue('invalid starting boundary' in str(exc))
self.assertTrue('--unique' in str(exc))
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
def test_empty(self):
it = utils.iter_multipart_mime_documents(StringIO('--unique'),
'unique')
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
fp = next(it)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fp.read(), '')
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
exc = None
try:
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
next(it)
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
except StopIteration as err:
exc = err
self.assertTrue(exc is not None)
def test_basic(self):
it = utils.iter_multipart_mime_documents(
StringIO('--unique\r\nabcdefg\r\n--unique--'), 'unique')
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
fp = next(it)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fp.read(), 'abcdefg')
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
exc = None
try:
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
next(it)
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
except StopIteration as err:
exc = err
self.assertTrue(exc is not None)
def test_basic2(self):
it = utils.iter_multipart_mime_documents(
StringIO('--unique\r\nabcdefg\r\n--unique\r\nhijkl\r\n--unique--'),
'unique')
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
fp = next(it)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fp.read(), 'abcdefg')
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
fp = next(it)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fp.read(), 'hijkl')
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
exc = None
try:
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
next(it)
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
except StopIteration as err:
exc = err
self.assertTrue(exc is not None)
def test_tiny_reads(self):
it = utils.iter_multipart_mime_documents(
StringIO('--unique\r\nabcdefg\r\n--unique\r\nhijkl\r\n--unique--'),
'unique')
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
fp = next(it)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fp.read(2), 'ab')
self.assertEqual(fp.read(2), 'cd')
self.assertEqual(fp.read(2), 'ef')
self.assertEqual(fp.read(2), 'g')
self.assertEqual(fp.read(2), '')
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
fp = next(it)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fp.read(), 'hijkl')
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
exc = None
try:
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
next(it)
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
except StopIteration as err:
exc = err
self.assertTrue(exc is not None)
def test_big_reads(self):
it = utils.iter_multipart_mime_documents(
StringIO('--unique\r\nabcdefg\r\n--unique\r\nhijkl\r\n--unique--'),
'unique')
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
fp = next(it)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fp.read(65536), 'abcdefg')
self.assertEqual(fp.read(), '')
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
fp = next(it)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fp.read(), 'hijkl')
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
exc = None
try:
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
next(it)
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
except StopIteration as err:
exc = err
self.assertTrue(exc is not None)
EC: support multiple ranges for GET requests This commit lets clients receive multipart/byteranges responses (see RFC 7233, Appendix A) for erasure-coded objects. Clients can already do this for replicated objects, so this brings EC closer to feature parity (ha!). GetOrHeadHandler got a base class extracted from it that treats an HTTP response as a sequence of byte-range responses. This way, it can continue to yield whole fragments, not just N-byte pieces of the raw HTTP response, since an N-byte piece of a multipart/byteranges response is pretty much useless. There are a couple of bonus fixes in here, too. For starters, download resuming now works on multipart/byteranges responses. Before, it only worked on 200 responses or 206 responses for a single byte range. Also, BufferedHTTPResponse grew a readline() method. Also, the MIME response for replicated objects got tightened up a little. Before, it had some leading and trailing CRLFs which, while allowed by RFC 7233, provide no benefit. Now, both replicated and EC multipart/byteranges avoid extraneous bytes. This let me re-use the Content-Length calculation in swob instead of having to either hack around it or add extraneous whitespace to match. Change-Id: I16fc65e0ec4e356706d327bdb02a3741e36330a0
2015-03-20 09:56:30 -07:00
def test_leading_crlfs(self):
it = utils.iter_multipart_mime_documents(
StringIO('\r\n\r\n\r\n--unique\r\nabcdefg\r\n'
'--unique\r\nhijkl\r\n--unique--'),
'unique')
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
fp = next(it)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fp.read(65536), 'abcdefg')
self.assertEqual(fp.read(), '')
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
fp = next(it)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fp.read(), 'hijkl')
EC: support multiple ranges for GET requests This commit lets clients receive multipart/byteranges responses (see RFC 7233, Appendix A) for erasure-coded objects. Clients can already do this for replicated objects, so this brings EC closer to feature parity (ha!). GetOrHeadHandler got a base class extracted from it that treats an HTTP response as a sequence of byte-range responses. This way, it can continue to yield whole fragments, not just N-byte pieces of the raw HTTP response, since an N-byte piece of a multipart/byteranges response is pretty much useless. There are a couple of bonus fixes in here, too. For starters, download resuming now works on multipart/byteranges responses. Before, it only worked on 200 responses or 206 responses for a single byte range. Also, BufferedHTTPResponse grew a readline() method. Also, the MIME response for replicated objects got tightened up a little. Before, it had some leading and trailing CRLFs which, while allowed by RFC 7233, provide no benefit. Now, both replicated and EC multipart/byteranges avoid extraneous bytes. This let me re-use the Content-Length calculation in swob instead of having to either hack around it or add extraneous whitespace to match. Change-Id: I16fc65e0ec4e356706d327bdb02a3741e36330a0
2015-03-20 09:56:30 -07:00
self.assertRaises(StopIteration, it.next)
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
def test_broken_mid_stream(self):
# We go ahead and accept whatever is sent instead of rejecting the
# whole request, in case the partial form is still useful.
it = utils.iter_multipart_mime_documents(
StringIO('--unique\r\nabc'), 'unique')
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
fp = next(it)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fp.read(), 'abc')
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
exc = None
try:
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
next(it)
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
except StopIteration as err:
exc = err
self.assertTrue(exc is not None)
def test_readline(self):
it = utils.iter_multipart_mime_documents(
StringIO('--unique\r\nab\r\ncd\ref\ng\r\n--unique\r\nhi\r\n\r\n'
'jkl\r\n\r\n--unique--'), 'unique')
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
fp = next(it)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fp.readline(), 'ab\r\n')
self.assertEqual(fp.readline(), 'cd\ref\ng')
self.assertEqual(fp.readline(), '')
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
fp = next(it)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fp.readline(), 'hi\r\n')
self.assertEqual(fp.readline(), '\r\n')
self.assertEqual(fp.readline(), 'jkl\r\n')
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
exc = None
try:
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
next(it)
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
except StopIteration as err:
exc = err
self.assertTrue(exc is not None)
def test_readline_with_tiny_chunks(self):
it = utils.iter_multipart_mime_documents(
StringIO('--unique\r\nab\r\ncd\ref\ng\r\n--unique\r\nhi\r\n'
'\r\njkl\r\n\r\n--unique--'),
'unique',
read_chunk_size=2)
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
fp = next(it)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fp.readline(), 'ab\r\n')
self.assertEqual(fp.readline(), 'cd\ref\ng')
self.assertEqual(fp.readline(), '')
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
fp = next(it)
pep8 fix: assertEquals -> assertEqual assertEquals is deprecated in py3, replacing it. Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8 Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
2015-08-05 23:58:14 +05:30
self.assertEqual(fp.readline(), 'hi\r\n')
self.assertEqual(fp.readline(), '\r\n')
self.assertEqual(fp.readline(), 'jkl\r\n')
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
exc = None
try:
Replace it.next() with next(it) for py3 compat The Python 2 next() method of iterators was renamed to __next__() on Python 3. Use the builtin next() function instead which works on Python 2 and Python 3. Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
2015-06-15 22:10:45 +05:30
next(it)
Move multipart MIME parser into utils Over on the EC branch, we need to be able to parse multipart MIME documents in the object server. The formpost middleware has a perfectly good MIME parser, but it seems sort of awful to import things from formpost in swift/obj/server.py, so I pulled it out into common.utils. Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60
2014-09-10 17:29:05 -07:00
except StopIteration as err:
exc = err
self.assertTrue(exc is not None)
replace use of deprecated rfc822.Message with a helper utility The rfc822 module has been deprecated since Python 2.3, and in particular is absent from the Python 3 standard library. However, Swift uses instances of rfc822.Message in a number of places, relying on its behavior of immediately parsing the headers of a file-like object without consuming the body, leaving the position of the file at the start of the body. Python 3's http.client has an undocumented parse_headers function with the same behavior, which inspired the new parse_mime_headers utility introduced here. (The HeaderKeyDict returned by parse_mime_headers doesn't have a `.getheader(key)` method like rfc822.Message did; the dictionary-like `[key]` or `.get(key)` interface should be used exclusively.) The implementation in this commit won't actually work with Python 3, the email.parser.Parser().parsestr of which expects a Unicode string, but it is believed that this can be addressed in followup work. Change-Id: Ia5ee2ead67e36e8c6416183667f64ae255887736
2015-09-08 19:21:39 -07:00
class TestParseMimeHeaders(unittest.TestCase):
def test_parse_mime_headers(self):
doc_file = BytesIO(b"""Content-Disposition: form-data; name="file_size"
Foo: Bar
NOT-title-cAsED: quux
Connexion: =?iso8859-1?q?r=E9initialis=E9e_par_l=27homologue?=
Status: =?utf-8?b?5byA5aeL6YCa6L+H5a+56LGh5aSN5Yi2?=
Latin-1: Resincronizaci\xf3n realizada con \xe9xito
Utf-8: \xd0\xba\xd0\xbe\xd0\xbd\xd1\x82\xd0\xb5\xd0\xb9\xd0\xbd\xd0\xb5\xd1\x80
This is the body
""")
headers = utils.parse_mime_headers(doc_file)
Port parse_mime_headers() to Python 3 Port swift.common.utils.parse_mime_headers() to Python 3: * On Python 3, tries to decode headers from UTF-8. If an header was was not encoded to UTF-8, decode the header from Latin1. * Update the parse_mime_headers() tests: on Python 3, HTTP header values are Unicode strings. This change is a follow-up of the change Ia5ee2ead67e36e8c6416183667f64ae255887736. Change-Id: I042dd13e9eb0e9844ccd832d538cdac84359ed42
2015-10-19 16:45:33 +02:00
utf8 = u'\u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440'
if six.PY2:
utf8 = utf8.encode('utf-8')
replace use of deprecated rfc822.Message with a helper utility The rfc822 module has been deprecated since Python 2.3, and in particular is absent from the Python 3 standard library. However, Swift uses instances of rfc822.Message in a number of places, relying on its behavior of immediately parsing the headers of a file-like object without consuming the body, leaving the position of the file at the start of the body. Python 3's http.client has an undocumented parse_headers function with the same behavior, which inspired the new parse_mime_headers utility introduced here. (The HeaderKeyDict returned by parse_mime_headers doesn't have a `.getheader(key)` method like rfc822.Message did; the dictionary-like `[key]` or `.get(key)` interface should be used exclusively.) The implementation in this commit won't actually work with Python 3, the email.parser.Parser().parsestr of which expects a Unicode string, but it is believed that this can be addressed in followup work. Change-Id: Ia5ee2ead67e36e8c6416183667f64ae255887736
2015-09-08 19:21:39 -07:00
expected_headers = {
'Content-Disposition': 'form-data; name="file_size"',
'Foo': "Bar",
'Not-Title-Cased': "quux",
# Encoded-word or non-ASCII values are treated just like any other
# bytestring (at least for now)
'Connexion': "=?iso8859-1?q?r=E9initialis=E9e_par_l=27homologue?=",
'Status': "=?utf-8?b?5byA5aeL6YCa6L+H5a+56LGh5aSN5Yi2?=",
'Latin-1': "Resincronizaci\xf3n realizada con \xe9xito",
Port parse_mime_headers() to Python 3 Port swift.common.utils.parse_mime_headers() to Python 3: * On Python 3, tries to decode headers from UTF-8. If an header was was not encoded to UTF-8, decode the header from Latin1. * Update the parse_mime_headers() tests: on Python 3, HTTP header values are Unicode strings. This change is a follow-up of the change Ia5ee2ead67e36e8c6416183667f64ae255887736. Change-Id: I042dd13e9eb0e9844ccd832d538cdac84359ed42
2015-10-19 16:45:33 +02:00
'Utf-8': utf8,
replace use of deprecated rfc822.Message with a helper utility The rfc822 module has been deprecated since Python 2.3, and in particular is absent from the Python 3 standard library. However, Swift uses instances of rfc822.Message in a number of places, relying on its behavior of immediately parsing the headers of a file-like object without consuming the body, leaving the position of the file at the start of the body. Python 3's http.client has an undocumented parse_headers function with the same behavior, which inspired the new parse_mime_headers utility introduced here. (The HeaderKeyDict returned by parse_mime_headers doesn't have a `.getheader(key)` method like rfc822.Message did; the dictionary-like `[key]` or `.get(key)` interface should be used exclusively.) The implementation in this commit won't actually work with Python 3, the email.parser.Parser().parsestr of which expects a Unicode string, but it is believed that this can be addressed in followup work. Change-Id: Ia5ee2ead67e36e8c6416183667f64ae255887736
2015-09-08 19:21:39 -07:00
}
self.assertEqual(expected_headers, headers)
self.assertEqual(b"This is the body\n", doc_file.read())
EC: support multiple ranges for GET requests This commit lets clients receive multipart/byteranges responses (see RFC 7233, Appendix A) for erasure-coded objects. Clients can already do this for replicated objects, so this brings EC closer to feature parity (ha!). GetOrHeadHandler got a base class extracted from it that treats an HTTP response as a sequence of byte-range responses. This way, it can continue to yield whole fragments, not just N-byte pieces of the raw HTTP response, since an N-byte piece of a multipart/byteranges response is pretty much useless. There are a couple of bonus fixes in here, too. For starters, download resuming now works on multipart/byteranges responses. Before, it only worked on 200 responses or 206 responses for a single byte range. Also, BufferedHTTPResponse grew a readline() method. Also, the MIME response for replicated objects got tightened up a little. Before, it had some leading and trailing CRLFs which, while allowed by RFC 7233, provide no benefit. Now, both replicated and EC multipart/byteranges avoid extraneous bytes. This let me re-use the Content-Length calculation in swob instead of having to either hack around it or add extraneous whitespace to match. Change-Id: I16fc65e0ec4e356706d327bdb02a3741e36330a0
2015-03-20 09:56:30 -07:00
class FakeResponse(object):
def __init__(self, status, headers, body):
self.status = status
self.headers = HeaderKeyDict(headers)
self.body = StringIO(body)
def getheader(self, header_name):
return str(self.headers.get(header_name, ''))
def getheaders(self):
return self.headers.items()
def read(self, length=None):
return self.body.read(length)
def readline(self, length=None):
return self.body.readline(length)
class TestDocumentItersToHTTPResponseBody(unittest.TestCase):
def test_no_parts(self):
body = utils.document_iters_to_http_response_body(
iter([]), 'dontcare',
multipart=False, logger=FakeLogger())
self.assertEqual(body, '')
def test_single_part(self):
body = "time flies like an arrow; fruit flies like a banana"
doc_iters = [{'part_iter': iter(StringIO(body).read, '')}]
resp_body = ''.join(
utils.document_iters_to_http_response_body(
iter(doc_iters), 'dontcare',
multipart=False, logger=FakeLogger()))
self.assertEqual(resp_body, body)
def test_multiple_parts(self):
part1 = "two peanuts were walking down a railroad track"
part2 = "and one was a salted. ... peanut."
doc_iters = [{
'start_byte': 88,
'end_byte': 133,
'content_type': 'application/peanut',
'entity_length': 1024,
'part_iter': iter(StringIO(part1).read, ''),
}, {
'start_byte': 500,
'end_byte': 532,
'content_type': 'application/salted',
'entity_length': 1024,
'part_iter': iter(StringIO(part2).read, ''),
}]
resp_body = ''.join(
utils.document_iters_to_http_response_body(
iter(doc_iters), 'boundaryboundary',
multipart=True, logger=FakeLogger()))
self.assertEqual(resp_body, (
"--boundaryboundary\r\n" +
# This is a little too strict; we don't actually care that the
# headers are in this order, but the test is much more legible
# this way.
"Content-Type: application/peanut\r\n" +
"Content-Range: bytes 88-133/1024\r\n" +
"\r\n" +
part1 + "\r\n" +
"--boundaryboundary\r\n"
"Content-Type: application/salted\r\n" +
"Content-Range: bytes 500-532/1024\r\n" +
"\r\n" +
part2 + "\r\n" +
"--boundaryboundary--"))
Reject overly-taxing ranged-GET requests RFC 7233 says that servers MAY reject egregious range-GET requests such as requests with hundreds of ranges, requests with non-ascending ranges, and so on. Such requests are fairly hard for Swift to process. Consider a Range header that asks for the first byte of every 10th MiB in a 4 GiB object, but in some random order. That'll cause a lot of seeks on the object server, but the corresponding response body is quite small in comparison to the workload. This commit makes Swift reject, with a 416 response, any ranged GET request with more than fifty ranges, more than three overlapping ranges, or more than eight non-increasing ranges. This is a necessary prerequisite for supporting multi-range GETs on large objects. Otherwise, a malicious user could construct a Range header with hundreds of byte ranges where each individual byterange requires the proxy to contact a different object server. If seeking all over a disk is bad, connecting all over the cluster is way worse. DocImpact Change-Id: I4dcedcaae6c3deada06a0223479e611094d57234
2014-08-28 09:39:38 -08:00
class TestPairs(unittest.TestCase):
def test_pairs(self):
items = [10, 20, 30, 40, 50, 60]
got_pairs = set(utils.pairs(items))
self.assertEqual(got_pairs,
set([(10, 20), (10, 30), (10, 40), (10, 50), (10, 60),
(20, 30), (20, 40), (20, 50), (20, 60),
(30, 40), (30, 50), (30, 60),
(40, 50), (40, 60),
(50, 60)]))
Make _get_addr() method a function in utils. The patch moves the MemcacheConnPool._get_addr() method a function in swift.common.utils. The function is renamed to parse_socket_string() and the documentation is updated accordingly. The test for it has also been moved. Change-Id: Ida65b2fded28d0a059e668646f5b89714298f348
2016-01-08 18:42:33 -08:00
class TestSocketStringParser(unittest.TestCase):
def test_socket_string_parser(self):
default = 1337
addrs = [('1.2.3.4', '1.2.3.4', default),
('1.2.3.4:5000', '1.2.3.4', 5000),
('[dead:beef::1]', 'dead:beef::1', default),
('[dead:beef::1]:5000', 'dead:beef::1', 5000),
('example.com', 'example.com', default),
('example.com:5000', 'example.com', 5000),
('foo.1-2-3.bar.com:5000', 'foo.1-2-3.bar.com', 5000),
('1.2.3.4:10:20', None, None),
('dead:beef::1:5000', None, None)]
for addr, expected_host, expected_port in addrs:
if expected_host:
host, port = utils.parse_socket_string(addr, default)
self.assertEqual(expected_host, host)
self.assertEqual(expected_port, int(port))
else:
with self.assertRaises(ValueError):
utils.parse_socket_string(addr, default)
Initial commit of Swift code
2010-07-12 17:03:45 -05:00
if __name__ == '__main__':
unittest.main()
Copy Permalink