381 Commits

Author SHA1 Message Date
gholt
a80c720af5 Object replication ssync (an rsync alternative)
For this commit, ssync is just a direct replacement for how
we use rsync. Assuming we switch over to ssync completely
someday and drop rsync, we will then be able to improve the
algorithms even further (removing local objects as we
successfully transfer each one rather than waiting for whole
partitions, using an index.db with hash-trees, etc., etc.)

For easier review, this commit can be thought of in distinct
parts:

1)  New global_conf_callback functionality for allowing
    services to perform setup code before workers, etc. are
    launched. (This is then used by ssync in the object
    server to create a cross-worker semaphore to restrict
    concurrent incoming replication.)

2)  A bit of shifting of items up from object server and
    replicator to diskfile or DEFAULT conf sections for
    better sharing of the same settings. conn_timeout,
    node_timeout, client_timeout, network_chunk_size,
    disk_chunk_size.

3)  Modifications to the object server and replicator to
    optionally use ssync in place of rsync. This is done in
    a generic enough way that switching to FutureSync should
    be easy someday.

4)  The biggest part, and (at least for now) completely
    optional part, are the new ssync_sender and
    ssync_receiver files. Nice and isolated for easier
    testing and visibility into test coverage, etc.

All the usual logging, statsd, recon, etc. instrumentation
is still there when using ssync, just as it is when using
rsync.

Beyond the essential error and exceptional condition
logging, I have not added any additional instrumentation at
this time. Unless there is something someone finds super
pressing to have added to the logging, I think such
additions would be better as separate change reviews.

FOR NOW, IT IS NOT RECOMMENDED TO USE SSYNC ON PRODUCTION
CLUSTERS. Some of us will be in a limited fashion to look
for any subtle issues, tuning, etc. but generally ssync is
an experimental feature. In its current implementation it is
probably going to be a bit slower than rsync, but if all
goes according to plan it will end up much faster.

There are no comparisions yet between ssync and rsync other
than some raw virtual machine testing I've done to show it
should compete well enough once we can put it in use in the
real world.

If you Tweet, Google+, or whatever, be sure to indicate it's
experimental. It'd be best to keep it out of deployment
guides, howtos, etc. until we all figure out if we like it,
find it to be stable, etc.

Change-Id: If003dcc6f4109e2d2a42f4873a0779110fff16d6
2013-11-07 16:52:01 +00:00
Jenkins
920680ffdd Merge "Faster swift-dispersion-populate" 2013-10-22 17:55:59 +00:00
Peter Portante
9411a24ba7 Revert "Refactor common/utils methods to common/ondisk"
This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32

Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 17:18:09 -04:00
Jenkins
111ebb9a4b Merge "Fix bad hash_path reference" 2013-10-07 21:08:13 +00:00
Jenkins
0b594bc3af Merge "Change OpenStack LLC to Foundation" 2013-10-07 16:09:37 +00:00
Peter Portante
61e135e211 Fix bad hash_path reference
Mea culpa: these two scripts were missed in commit:

  https://review.openstack.org/46956

Fixes bug 1235441

Change-Id: I4303bc808448a79bddbb991526b0cca26150b392
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 11:18:28 -04:00
ZhiQiang Fan
f72704fc82 Change OpenStack LLC to Foundation
Change-Id: I7c3df47c31759dbeb3105f8883e2688ada848d58
Closes-bug: #1214176
2013-09-20 01:02:31 +08:00
Clay Gerrard
b0aeed1ec7 Fix default replication options for ring-builder add
Change-Id: I957deeb0e711bfe7cd9d852726c77179a4613ee0
2013-09-16 20:02:20 -07:00
Dirk Mueller
3d36a76156 Use Python 3.x compatible except construct
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.

Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
2013-09-07 10:50:54 +02:00
Florian Hines
42f4b150e3 Faster swift-dispersion-populate
- Makes swift-dispersion-populate a bit faster when using a larger
  dispersion_coverage with a larger part_power.
- Adds option to only run population for container OR objects
- Adds option to let you resume population at given point (useful if you
  need to resume population after a previous run error'd out or the
  like) by specifying which suffix to start at.

The original populate just randomly used uuid4().hex as a suffix on the
container/object names until all the partition's required where covered.
This isn't a big deal if you're only doing 1% coverage on a ring with a
small part power but takes ages if you're doing 100% on a larger ring.

Change-Id: I52f890a774412c1d6179f12db9081aedc58b6bc2
2013-09-05 18:12:15 -05:00
Jenkins
3498ddbbb9 Merge "Add delay in swift-bench" 2013-08-29 22:54:08 +00:00
Monty Taylor
5a785aa5ee Add missing copyright license headers
We're working on adding enforcement that things have appropriate
copyright license headers. In anticipation of that, fix the files that don't
have them.

Change-Id: Ie0a9fd5eece5b6671ff4389b07b69ca29be7d017
2013-08-07 17:50:32 -03:00
Vincent Untz
7f1aa9d1e8 Allow dispersion tools to use keystone server with insecure certificate
The swift-dispersion-populate and swift-dispersion-report tools now
accept a --insecure option.

Also, dispersion.conf now has a keystone_api_insecure option.

Default is obviously to use the secure path.

DocImpact

Change-Id: I4000352e547d9ce5b08ade54e0c886281caff891
2013-08-05 22:44:12 +02:00
Ksenia Demina
600e9c8672 Add delay in swift-bench
With enable write affinity, it's necessary to wait until
replication has moved things to their proper homes before
running delete request. With write affinity turned on, only
nodes in local region will get the object right after PUT request.

Fix bug #1198926

Change-Id: I3aa8933d45c47a010ae05561e12176479e7c9bcc
2013-08-02 11:47:33 +00:00
Jenkins
6a9f55d876 Merge "Removed unnecessary monkeypatching of __builtin__" 2013-07-30 04:02:10 +00:00
Jenkins
82aeacf5bf Merge "Allow floating point value for dispersion_coverage" 2013-07-29 21:53:19 +00:00
Jenkins
3349013dff Merge "Configuration options for error regex and log file in the config now" 2013-07-29 18:23:39 +00:00
Alex Gaynor
0f3b0410e3 Removed unnecessary monkeypatching of __builtin__
Replaced it with explicitly importing the gettext function, which is
significantly more readable.

Change-Id: Ia0a7edcf685fb6e4052a8290367b233169529ab8
2013-07-27 21:34:35 -07:00
Marcelo Martins
d2dd3e5488 Configuration options for error regex and log file in the config now
Making it possible for one to overwrite the default set of regexes
used to search for device block errors in the log file. Also making
the log file naming pattern configurable by setting them in the
drive-audit.conf file.

Updating "Detecting Failed Drives" section on the admin guide as well.

Change-Id: I7bd3acffed196da3e09db4c9dcbb48a20bdd1cf0
2013-07-23 07:24:29 -05:00
Kun Huang
63061e37ed Add notes for /srv/node in swift-object-info
'devices' is set in object-server.conf on each node, not in ring data,
and the things printed here is just for watching not for running, so
just leave a note here. (this https://review.openstack.org/#/c/23951/
is used for running, so just a note is not enough)

mark this commit as bug fixing is because this script is the last place
using /srv/node but not from conf as Chmouel said.

fixes import change on read_metadata
fixes bug #885006
Change-Id: I727ec2d01c093af61fd3895e5701d87ef67cd9ff
2013-07-23 13:46:51 +08:00
Dirk Mueller
8aba2d602e Start using Hacking
Instead of blacklisting Hacking globally,
only blacklist those that currently occur frequently
(for a later followup patch), and fix the rest. In
detail:

H101  Use TODO(NAME)
H201  no 'except:' at least use 'except Exception:'
H231  octal number 022 should be written as 0o22
H401  docstring should not start with a space
H701  Empty localization string

Change-Id: Ib3b3d56b68d1cf15d3b67ac9749fcbdb876dc52a
2013-07-15 11:41:58 +02:00
Thomas Leaman
5449155fb0 Allow floating point value for dispersion_coverage
For systems with very large numbers of partitions, 1% dispersion
coverage may simply be too much/take too long. This fix allows <1
values to be used for dispersion_coverage.

DocImpact

Change-Id: I5ed35b69754d55a410e66e658b3854de57c7666b
2013-07-09 09:29:07 +00:00
Samuel Merritt
57d4e6e718 Fix swift-dispersion-report.
Fixes bug 1180680.

Change-Id: I1c0eaa6f934f73853e832a8e468e1b8369a21117
2013-06-24 10:58:22 -07:00
Jenkins
2412f6c737 Merge "Add "normal", optparse-style options to swift-ring-builder add." 2013-06-21 15:22:51 +00:00
Jenkins
5bfd2d798d Merge "Add parallelism to object expirer daemon." 2013-06-11 22:48:24 +00:00
Samuel Merritt
d69f929be9 Add "normal", optparse-style options to swift-ring-builder add.
The old format is still present and works just like it did before, so
your existing scripts won't break.

New format pros:
 * it's readable even for Swift newcomers
 * it's easy to extend
 * it's familiar to anyone who's used a Unix command line
 * we don't have to maintain the parser

New format cons:
 * you can't add multiple devices in one go

Old format pros:
 * you can add many devices with one command
 * it's compact

Old format cons:
 * it confuses newcomers
 * "wait, is that zone dash IP colon port slash device, or zone slash
   IP dash port colon meta underscore device?" Just try walking
   someone through adding a device over voice chat.
 * it's annoying to add new fields

Note that this only affects the command "swift-ring-builder
<builderfile> add". Other swift-ring-builder commands are unchanged.

DocImpact

Change-Id: I034b7f79eb6f4d81a5c4da193e1358741441c5b5
2013-06-10 14:09:27 -07:00
Greg Lange
209c5ec418 Add parallelism to object expirer daemon.
Two types of parallelism are added:

- concurrency to speed up what a single process does
- a way to run multiple daemons to work on different parts of the work

DocImpact

Change-Id: I48997f68eb2fd8de19a5ee8b9fcdf76dde2ba0ab
2013-06-07 20:49:47 +00:00
Ilya Kharin
3957dbc5d4 RingBuilder.add_dev returns device id
When added a new devices into builder the add_dev function returns it
unique id.

blueprint argparse-in-swift-ring-builder

Change-Id: I57080bb625e812f6cea71199df907a44b332b552
2013-05-24 17:34:24 +04:00
Ilya Kharin
43bf568f48 Move parse search logic outside from builder
Dramatic part of RingBuilder.search_devs which parse a complex format
of a search device string moved to the swift-ring-builder script.
Instead, the search_devs has a simple interface to search devices.

blueprint argparse-in-swift-ring-builder

Change-Id: If3dd77b297b474fb9a058e4693fef2dfb11fca3d
2013-05-24 17:12:34 +04:00
Jenkins
4a1f3d3e1a Merge "Add ability to save builder data to a disk file" 2013-05-21 06:13:46 +00:00
Jenkins
d1f1e9fb67 Merge "Don't delete containers when they weren't created." 2013-05-21 01:04:38 +00:00
Ilya Kharin
cc040a9c29 Add ability to save builder data to a disk file
Instances of the RingBuilder class can store its data to a disk file by
the save method and load it by the load method.

blueprint argparse-in-swift-ring-builder

Change-Id: I69fdf0693ca9f520d235a795ecdd2da310dcd5d3
2013-05-16 19:49:00 +04:00
Darrell Bishop
5f295d0329 Don't delete containers when they weren't created.
When swift-bench is run in direct mode, don't try to delete the
containers which weren't created.

Fixes bug 1177960.

Change-Id: Ice07e8729bb776e2b215894cf95fb80b64167a8d
2013-05-11 13:37:16 -07:00
Jenkins
4074ba93f5 Merge "Fix rebalance for zero weighted devices." 2013-05-04 17:03:31 +00:00
Jenkins
f7b4a6d0ce Merge "conf.d support" 2013-05-03 02:27:00 +00:00
Monty Taylor
de2e5aa462 Update to flake8 instead of pep8.
Change-Id: I3d4a31111c3044da06611405ce80f208ef8a0ce3
2013-05-01 14:19:38 -04:00
Clay Gerrard
34f5085c3e conf.d support
Allow Swift daemons and servers to optionally accept a directory as the
configuration parameter.  Directory based configuration leverages
ConfigParser's native multi-file support.  Files ending in '.conf' in the
given directory are parsed in lexicographical order.  Filenames starting with
'.' are ignored.  A mixture of file and directory configuration paths is not
supported - if the configuration path is a file behavior is unchanged.

 * update swift-init to search for conf.d paths when building servers
    (e.g. /etc/swift/proxy-server.conf.d/)
 * new script swift-config can be used to inspect the cumulative configuration
 * pull a little bit of code out of run_wsgi and test separately
 * fix example config bug for the proxy servers client_disconnect option
 * added section on directory based configuration to deployment guide

DocImpact

Implements: blueprint confd
Change-Id: I89b0f48e538117f28590cf6698401f74ef58003b
2013-04-30 00:17:46 -07:00
Eohyung Lee
98acf42f92 Fix rebalance for zero weighted devices.
If we set device's weight to zero, currently balance will be set
special value(999.99) until zero weighted device return all
its partitions. So we cannot check balance has changed.
Thus we need to check balance or last_balance is special value.

Change-Id: I5b7db8b8e48db0c4771c51a764bda689869817d5
Fixes: bug #1171731
2013-04-26 16:36:08 +09:00
Sergey Kraynev
ea7858176b Implementation of replication servers
Support separate replication ip address:
- Added new function in utils. This function provides ability
  to select separate IP address for replication service.
- Db_replicator and object replicators were changed.
  Replication process uses new function now.

Replication network parameters:
- Replication network fields (replication_ip, replication_port)
  support was added to device dictionary in swift-ring-builder script.
- Changes were made to support new fields in search, show and set_info
  functions.

Implementation of replication servers:
- Separate replication servers use the same code as normal replication
  servers, but with replication_server parameter = True.  When using a
  separate replication network, the non-replication servers set
  replication_server = False.  When there is no separate replication
  network (the default case), replication_server is not included in the config.

DocImpact
Change-Id: Ie9af5bdcdf9241c355e36053ca4adfe49dc35bd0
Implements: blueprint dedicated-replication-network
2013-04-21 18:14:42 -04:00
Victor Rodionov
7f534fac38 fix dict duplicated key
remove duplicated key from swift-bench bin; and also add .idea to gitignore

Change-Id: I31360489716d03324b8d2bb1c51e282393069c13
2013-04-22 01:31:53 +04:00
Jenkins
e3ccb3f52d Merge "Correcting output of swift-ring-builder" 2013-04-15 22:49:48 +00:00
Sergey Kraynev
5afd0c3bb6 Refactoring format_device function
* Algorithm format_device was changed for simplicity extension new
  ip addresses parameters.
* Some prints outputs was replacement by function format_device.

Change-Id: I8565d42fcdb62eeb398c4432bb6f499c27c05cf6
2013-04-12 12:40:27 -04:00
Sergey Kraynev
d7873b5ce3 Correcting output of swift-ring-builder
Indent swift-ring-builder output was changed according
with old style (before implementation of region tier).

Change-Id: I0d1cc7acdc5baf86f343745aea6fc2120838fd36
2013-04-11 09:32:56 -04:00
gholt
1cb952a958 Allow a configurable set of TempURL methods
Folks have actually been asking for this. I think they're sending a
DELETE TempURL to someone way ahead of time and the someone issues it
when they're ready. Honestly, I'm not entirely sure of the use case,
but having the set of methods configurable wouldn't hurt.

Change-Id: Ibdb48f8a72077b045eeedddfae4c0a1f56098d7a
2013-04-04 20:37:23 +00:00
Jenkins
ba5a0b0fe1 Merge "Change logger name to module itself instead of root" 2013-04-02 14:45:12 +00:00
Jenkins
220db2e5b5 Merge "Give friendly notification in swift-get-nodes" 2013-03-18 23:29:41 +00:00
tong li
b530de2ad5 swift-ring-builder blows up when not supported command requested.
swift-ring-builder uses outdated unbounded class method to handle
not supported command. It worked for python 2.6 or older but not
for python 2.7 or newer. This patch fixes the problem.

Change-Id: I7dbc681ef6be44f6d79ff93189ccca13c51eab74
Fixes: bug #1154882
2013-03-18 09:04:50 -04:00
Kun Huang
45dda5e356 Change logger name to module itself instead of root
Using root logger will make public module log failed by unknow keyword.
Just change the logger name to module itself could dismiss this impact.
And disable logger.propagate could prevent double outputing.

Change-Id: I18696d124ebac9ca970d502558972e51de759097
Fixes: bug #1105133
2013-03-17 01:59:40 +08:00
Ilya Kharin
e56832c138 Fix format device
Change-Id: Icef35175682c11a406ef85accc6c6e1acbf89f4d
2013-03-16 12:41:14 +04:00
Jenkins
fd3a8a6650 Merge "Fix swift-ring-builder's list_parts command." 2013-03-13 22:53:34 +00:00