368 Commits

Author SHA1 Message Date
Vincent Untz
7f1aa9d1e8 Allow dispersion tools to use keystone server with insecure certificate
The swift-dispersion-populate and swift-dispersion-report tools now
accept a --insecure option.

Also, dispersion.conf now has a keystone_api_insecure option.

Default is obviously to use the secure path.

DocImpact

Change-Id: I4000352e547d9ce5b08ade54e0c886281caff891
2013-08-05 22:44:12 +02:00
Jenkins
6a9f55d876 Merge "Removed unnecessary monkeypatching of __builtin__" 2013-07-30 04:02:10 +00:00
Jenkins
82aeacf5bf Merge "Allow floating point value for dispersion_coverage" 2013-07-29 21:53:19 +00:00
Jenkins
3349013dff Merge "Configuration options for error regex and log file in the config now" 2013-07-29 18:23:39 +00:00
Alex Gaynor
0f3b0410e3 Removed unnecessary monkeypatching of __builtin__
Replaced it with explicitly importing the gettext function, which is
significantly more readable.

Change-Id: Ia0a7edcf685fb6e4052a8290367b233169529ab8
2013-07-27 21:34:35 -07:00
Marcelo Martins
d2dd3e5488 Configuration options for error regex and log file in the config now
Making it possible for one to overwrite the default set of regexes
used to search for device block errors in the log file. Also making
the log file naming pattern configurable by setting them in the
drive-audit.conf file.

Updating "Detecting Failed Drives" section on the admin guide as well.

Change-Id: I7bd3acffed196da3e09db4c9dcbb48a20bdd1cf0
2013-07-23 07:24:29 -05:00
Kun Huang
63061e37ed Add notes for /srv/node in swift-object-info
'devices' is set in object-server.conf on each node, not in ring data,
and the things printed here is just for watching not for running, so
just leave a note here. (this https://review.openstack.org/#/c/23951/
is used for running, so just a note is not enough)

mark this commit as bug fixing is because this script is the last place
using /srv/node but not from conf as Chmouel said.

fixes import change on read_metadata
fixes bug #885006
Change-Id: I727ec2d01c093af61fd3895e5701d87ef67cd9ff
2013-07-23 13:46:51 +08:00
Dirk Mueller
8aba2d602e Start using Hacking
Instead of blacklisting Hacking globally,
only blacklist those that currently occur frequently
(for a later followup patch), and fix the rest. In
detail:

H101  Use TODO(NAME)
H201  no 'except:' at least use 'except Exception:'
H231  octal number 022 should be written as 0o22
H401  docstring should not start with a space
H701  Empty localization string

Change-Id: Ib3b3d56b68d1cf15d3b67ac9749fcbdb876dc52a
2013-07-15 11:41:58 +02:00
Thomas Leaman
5449155fb0 Allow floating point value for dispersion_coverage
For systems with very large numbers of partitions, 1% dispersion
coverage may simply be too much/take too long. This fix allows <1
values to be used for dispersion_coverage.

DocImpact

Change-Id: I5ed35b69754d55a410e66e658b3854de57c7666b
2013-07-09 09:29:07 +00:00
Samuel Merritt
57d4e6e718 Fix swift-dispersion-report.
Fixes bug 1180680.

Change-Id: I1c0eaa6f934f73853e832a8e468e1b8369a21117
2013-06-24 10:58:22 -07:00
Jenkins
2412f6c737 Merge "Add "normal", optparse-style options to swift-ring-builder add." 2013-06-21 15:22:51 +00:00
Jenkins
5bfd2d798d Merge "Add parallelism to object expirer daemon." 2013-06-11 22:48:24 +00:00
Samuel Merritt
d69f929be9 Add "normal", optparse-style options to swift-ring-builder add.
The old format is still present and works just like it did before, so
your existing scripts won't break.

New format pros:
 * it's readable even for Swift newcomers
 * it's easy to extend
 * it's familiar to anyone who's used a Unix command line
 * we don't have to maintain the parser

New format cons:
 * you can't add multiple devices in one go

Old format pros:
 * you can add many devices with one command
 * it's compact

Old format cons:
 * it confuses newcomers
 * "wait, is that zone dash IP colon port slash device, or zone slash
   IP dash port colon meta underscore device?" Just try walking
   someone through adding a device over voice chat.
 * it's annoying to add new fields

Note that this only affects the command "swift-ring-builder
<builderfile> add". Other swift-ring-builder commands are unchanged.

DocImpact

Change-Id: I034b7f79eb6f4d81a5c4da193e1358741441c5b5
2013-06-10 14:09:27 -07:00
Greg Lange
209c5ec418 Add parallelism to object expirer daemon.
Two types of parallelism are added:

- concurrency to speed up what a single process does
- a way to run multiple daemons to work on different parts of the work

DocImpact

Change-Id: I48997f68eb2fd8de19a5ee8b9fcdf76dde2ba0ab
2013-06-07 20:49:47 +00:00
Ilya Kharin
3957dbc5d4 RingBuilder.add_dev returns device id
When added a new devices into builder the add_dev function returns it
unique id.

blueprint argparse-in-swift-ring-builder

Change-Id: I57080bb625e812f6cea71199df907a44b332b552
2013-05-24 17:34:24 +04:00
Ilya Kharin
43bf568f48 Move parse search logic outside from builder
Dramatic part of RingBuilder.search_devs which parse a complex format
of a search device string moved to the swift-ring-builder script.
Instead, the search_devs has a simple interface to search devices.

blueprint argparse-in-swift-ring-builder

Change-Id: If3dd77b297b474fb9a058e4693fef2dfb11fca3d
2013-05-24 17:12:34 +04:00
Jenkins
4a1f3d3e1a Merge "Add ability to save builder data to a disk file" 2013-05-21 06:13:46 +00:00
Jenkins
d1f1e9fb67 Merge "Don't delete containers when they weren't created." 2013-05-21 01:04:38 +00:00
Ilya Kharin
cc040a9c29 Add ability to save builder data to a disk file
Instances of the RingBuilder class can store its data to a disk file by
the save method and load it by the load method.

blueprint argparse-in-swift-ring-builder

Change-Id: I69fdf0693ca9f520d235a795ecdd2da310dcd5d3
2013-05-16 19:49:00 +04:00
Darrell Bishop
5f295d0329 Don't delete containers when they weren't created.
When swift-bench is run in direct mode, don't try to delete the
containers which weren't created.

Fixes bug 1177960.

Change-Id: Ice07e8729bb776e2b215894cf95fb80b64167a8d
2013-05-11 13:37:16 -07:00
Jenkins
4074ba93f5 Merge "Fix rebalance for zero weighted devices." 2013-05-04 17:03:31 +00:00
Jenkins
f7b4a6d0ce Merge "conf.d support" 2013-05-03 02:27:00 +00:00
Monty Taylor
de2e5aa462 Update to flake8 instead of pep8.
Change-Id: I3d4a31111c3044da06611405ce80f208ef8a0ce3
2013-05-01 14:19:38 -04:00
Clay Gerrard
34f5085c3e conf.d support
Allow Swift daemons and servers to optionally accept a directory as the
configuration parameter.  Directory based configuration leverages
ConfigParser's native multi-file support.  Files ending in '.conf' in the
given directory are parsed in lexicographical order.  Filenames starting with
'.' are ignored.  A mixture of file and directory configuration paths is not
supported - if the configuration path is a file behavior is unchanged.

 * update swift-init to search for conf.d paths when building servers
    (e.g. /etc/swift/proxy-server.conf.d/)
 * new script swift-config can be used to inspect the cumulative configuration
 * pull a little bit of code out of run_wsgi and test separately
 * fix example config bug for the proxy servers client_disconnect option
 * added section on directory based configuration to deployment guide

DocImpact

Implements: blueprint confd
Change-Id: I89b0f48e538117f28590cf6698401f74ef58003b
2013-04-30 00:17:46 -07:00
Eohyung Lee
98acf42f92 Fix rebalance for zero weighted devices.
If we set device's weight to zero, currently balance will be set
special value(999.99) until zero weighted device return all
its partitions. So we cannot check balance has changed.
Thus we need to check balance or last_balance is special value.

Change-Id: I5b7db8b8e48db0c4771c51a764bda689869817d5
Fixes: bug #1171731
2013-04-26 16:36:08 +09:00
Sergey Kraynev
ea7858176b Implementation of replication servers
Support separate replication ip address:
- Added new function in utils. This function provides ability
  to select separate IP address for replication service.
- Db_replicator and object replicators were changed.
  Replication process uses new function now.

Replication network parameters:
- Replication network fields (replication_ip, replication_port)
  support was added to device dictionary in swift-ring-builder script.
- Changes were made to support new fields in search, show and set_info
  functions.

Implementation of replication servers:
- Separate replication servers use the same code as normal replication
  servers, but with replication_server parameter = True.  When using a
  separate replication network, the non-replication servers set
  replication_server = False.  When there is no separate replication
  network (the default case), replication_server is not included in the config.

DocImpact
Change-Id: Ie9af5bdcdf9241c355e36053ca4adfe49dc35bd0
Implements: blueprint dedicated-replication-network
2013-04-21 18:14:42 -04:00
Victor Rodionov
7f534fac38 fix dict duplicated key
remove duplicated key from swift-bench bin; and also add .idea to gitignore

Change-Id: I31360489716d03324b8d2bb1c51e282393069c13
2013-04-22 01:31:53 +04:00
Jenkins
e3ccb3f52d Merge "Correcting output of swift-ring-builder" 2013-04-15 22:49:48 +00:00
Sergey Kraynev
5afd0c3bb6 Refactoring format_device function
* Algorithm format_device was changed for simplicity extension new
  ip addresses parameters.
* Some prints outputs was replacement by function format_device.

Change-Id: I8565d42fcdb62eeb398c4432bb6f499c27c05cf6
2013-04-12 12:40:27 -04:00
Sergey Kraynev
d7873b5ce3 Correcting output of swift-ring-builder
Indent swift-ring-builder output was changed according
with old style (before implementation of region tier).

Change-Id: I0d1cc7acdc5baf86f343745aea6fc2120838fd36
2013-04-11 09:32:56 -04:00
gholt
1cb952a958 Allow a configurable set of TempURL methods
Folks have actually been asking for this. I think they're sending a
DELETE TempURL to someone way ahead of time and the someone issues it
when they're ready. Honestly, I'm not entirely sure of the use case,
but having the set of methods configurable wouldn't hurt.

Change-Id: Ibdb48f8a72077b045eeedddfae4c0a1f56098d7a
2013-04-04 20:37:23 +00:00
Jenkins
ba5a0b0fe1 Merge "Change logger name to module itself instead of root" 2013-04-02 14:45:12 +00:00
Jenkins
220db2e5b5 Merge "Give friendly notification in swift-get-nodes" 2013-03-18 23:29:41 +00:00
tong li
b530de2ad5 swift-ring-builder blows up when not supported command requested.
swift-ring-builder uses outdated unbounded class method to handle
not supported command. It worked for python 2.6 or older but not
for python 2.7 or newer. This patch fixes the problem.

Change-Id: I7dbc681ef6be44f6d79ff93189ccca13c51eab74
Fixes: bug #1154882
2013-03-18 09:04:50 -04:00
Kun Huang
45dda5e356 Change logger name to module itself instead of root
Using root logger will make public module log failed by unknow keyword.
Just change the logger name to module itself could dismiss this impact.
And disable logger.propagate could prevent double outputing.

Change-Id: I18696d124ebac9ca970d502558972e51de759097
Fixes: bug #1105133
2013-03-17 01:59:40 +08:00
Ilya Kharin
e56832c138 Fix format device
Change-Id: Icef35175682c11a406ef85accc6c6e1acbf89f4d
2013-03-16 12:41:14 +04:00
Jenkins
fd3a8a6650 Merge "Fix swift-ring-builder's list_parts command." 2013-03-13 22:53:34 +00:00
Samuel Merritt
35acd5912f Fix swift-ring-builder's list_parts command.
I broke it in 7548cb9 when a ring's replica count changed from an int
to a float.

Change-Id: I49fa7e2961a09daad0c96e19cf1e39fec248d998
2013-03-13 13:34:19 -07:00
Samuel Merritt
ebcd60f7d9 Add a region tier to Swift's ring.
The region is one level above the zone; it is intended to represent a
chunk of machines that is distant from others with respect to
bandwidth and latency.

Old rings will default to having all their devices in region 1. Since
everything is in the same region by default, the ring builder will
simply distribute across zones as it did before, so your partition
assignment won't move because of this change. If you start adding
devices in other regions, of course, the assignment will change to
take that into account.

swift-ring-builder still accepts the same syntax as before, but will
default added devices to region 1 if no region is specified.

Examples:

$ swift-ring-builder foo.builder add r2z1-1.2.3.4:555/sda

$ swift-ring-builder foo.builder add r1z3-1.2.3.4:555/sda

$ swift-ring-builder foo.builder add z3-1.2.3.4:555/sda

Also, some updates to ring-overview doc.

Change-Id: Ifefbb839cdcf033e6c9201fadca95224c7303a29
2013-03-13 10:00:58 -07:00
Kun Huang
cb69dc881a Give friendly notification in swift-get-nodes
Hard coding of '/srv/node' maybe drive to some errors. But it seems
unfixable (please see talks in bug report). To avoid erros as we can,
showing a friendly notification to let user set their own device
location works better than hard coding directly.

Change-Id: I45d749dd52af8d3b051dfe98dbb9d71a5a374f97
Fixes: bug #907908
2013-03-09 13:43:54 +08:00
jola-mirecka
5e427e5e3b Add exception to swift-drive-audit if no date in line.
Sometimes there is no date at the beginning of a line in kern.log.
Although it does not happen often, there should be a check ensuring
the program doesn't crash in case it happens.

Added try-except block surrounding parsing string to date format.

Change-Id: I44a101266582eea2199189a006afa1037a9bd4ea
Fixes: bug #1152658
2013-03-08 16:38:11 +00:00
Jenkins
457ff9672d Merge "Make rings' replica counts adjustable." 2013-03-04 06:36:48 +00:00
Kun Huang
13eabdc744 Improve bin/swift-account-audit
1.Correct the usage of initializing a Ring object.
2.Fix typo.
3.Align lines of print_status part. (keep origin author mind)

note: Adam Kijak has commited it at early time, but didn't put it
      in Gerrit. I import his commit and send it in right Workflow.

Fixes: bug #1131901

Change-Id: If56316cc424045f7df662d60088382f7fd40e8b1
2013-02-24 20:04:30 +08:00
Samuel Merritt
7548cb9c47 Make rings' replica counts adjustable.
Example:

$ swift-ring-builder account.builder set_replicas 4
$ swift-ring-builder rebalance

This is a prerequisite for supporting globally-distributed clusters,
as operators of such clusters will probably want at least as many
replicas as they have regions. Therefore, adding a region requires
adding a replica. Similarly, removing a region lets an operator remove
a replica and save some money on disks.

In order to not hose clusters with lots of data, swift-ring-builder
now allows for setting of fractional replicas. Thus, one can gradually
increase the replica count at a rate that does not adversely affect
cluster performance.

Example:

$ swift-ring-builder object.builder set_replicas 3.01
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>

$ swift-ring-builder object.builder set_replicas 3.02
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>...

Obviously, fractional replicas are nonsensical for a single
partition. A fractional replica count is for the whole ring, not for
any individual partition, and indicates the average number of replicas
of each partition. For example, a replica count of 3.2 means that 20%
of partitions have 4 replicas and 80% have 3 replicas.

Changes do not take effect until after the ring is rebalanced. Thus,
if you mean to go from 3 replicas to 3.01 but you accidentally type
2.01, no data is lost.

Additionally, 'swift-ring-builder X.builder create' can now take a
decimal argument for the number of replicas.

DocImpact

Change-Id: I12b34dacf60350a297a46be493d5d171580243ff
2013-02-22 15:03:10 -08:00
Jenkins
44e3915564 Merge "Change in swift-drive-audit handling log rotation." 2013-02-21 02:41:37 +00:00
Joe Gordon
45f0502b52 Fix spelling mistakes
git ls-files | misspellings -f -
Source: https://github.com/lyda/misspell-check

Change-Id: I4132e6a276e44e2a8985238358533d315ee8d9c4
2013-02-12 16:39:40 -08:00
jola-mirecka
902b66d3ae Change in swift-drive-audit handling log rotation.
Change supports kern.log rotation in order to avoid loss
of significant information.

There is a year change functionality added as kern.log
does not keep record of year.

There is also backwards function added which allows
reading logs from the back to the front, speeding up the
execution along with the unit test for it

Fixes Bug 1080682

Change-Id: I93436c405aff5625396514000cab774b66022dd0
2013-02-12 13:34:49 +00:00
Jenkins
2dca894aae Merge "Fixed some swift-recon typos" 2013-02-09 22:32:35 +00:00
Jenkins
c0e8ad609b Merge "Allow change the endpoint_type when use swift-dispersion tools" 2013-02-08 23:55:03 +00:00
gholt
0640a61776 Fixed some swift-recon typos
Change-Id: Ibe4b59a90b82742b70b00353d22aedd972bbeb59
2013-02-08 20:19:48 +00:00