Chuck Thier
12 years ago
commit
001407b969
164 changed files with 30061 additions and 0 deletions
@ -0,0 +1,4 @@
|
||||
#!/bin/bash |
||||
|
||||
nosetests test/unit --exe --with-coverage --cover-package swift --cover-erase |
||||
rm -f .coverage |
||||
@ -0,0 +1,20 @@
|
||||
Maintainer |
||||
---------- |
||||
OpenStack, LLC. |
||||
IRC: #openstack |
||||
|
||||
Original Authors |
||||
---------------- |
||||
Michael Barton |
||||
John Dickinson |
||||
Greg Holt |
||||
Greg Lange |
||||
Jay Payne |
||||
Will Reese |
||||
Chuck Thier |
||||
|
||||
Contributors |
||||
------------ |
||||
Chmouel Boudjnah |
||||
Ed Leafe |
||||
Conrad Weidenkeller |
||||
@ -0,0 +1,202 @@
|
||||
|
||||
Apache License |
||||
Version 2.0, January 2004 |
||||
http://www.apache.org/licenses/ |
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION |
||||
|
||||
1. Definitions. |
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction, |
||||
and distribution as defined by Sections 1 through 9 of this document. |
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by |
||||
the copyright owner that is granting the License. |
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all |
||||
other entities that control, are controlled by, or are under common |
||||
control with that entity. For the purposes of this definition, |
||||
"control" means (i) the power, direct or indirect, to cause the |
||||
direction or management of such entity, whether by contract or |
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the |
||||
outstanding shares, or (iii) beneficial ownership of such entity. |
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity |
||||
exercising permissions granted by this License. |
||||
|
||||
"Source" form shall mean the preferred form for making modifications, |
||||
including but not limited to software source code, documentation |
||||
source, and configuration files. |
||||
|
||||
"Object" form shall mean any form resulting from mechanical |
||||
transformation or translation of a Source form, including but |
||||
not limited to compiled object code, generated documentation, |
||||
and conversions to other media types. |
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or |
||||
Object form, made available under the License, as indicated by a |
||||
copyright notice that is included in or attached to the work |
||||
(an example is provided in the Appendix below). |
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object |
||||
form, that is based on (or derived from) the Work and for which the |
||||
editorial revisions, annotations, elaborations, or other modifications |
||||
represent, as a whole, an original work of authorship. For the purposes |
||||
of this License, Derivative Works shall not include works that remain |
||||
separable from, or merely link (or bind by name) to the interfaces of, |
||||
the Work and Derivative Works thereof. |
||||
|
||||
"Contribution" shall mean any work of authorship, including |
||||
the original version of the Work and any modifications or additions |
||||
to that Work or Derivative Works thereof, that is intentionally |
||||
submitted to Licensor for inclusion in the Work by the copyright owner |
||||
or by an individual or Legal Entity authorized to submit on behalf of |
||||
the copyright owner. For the purposes of this definition, "submitted" |
||||
means any form of electronic, verbal, or written communication sent |
||||
to the Licensor or its representatives, including but not limited to |
||||
communication on electronic mailing lists, source code control systems, |
||||
and issue tracking systems that are managed by, or on behalf of, the |
||||
Licensor for the purpose of discussing and improving the Work, but |
||||
excluding communication that is conspicuously marked or otherwise |
||||
designated in writing by the copyright owner as "Not a Contribution." |
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity |
||||
on behalf of whom a Contribution has been received by Licensor and |
||||
subsequently incorporated within the Work. |
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of |
||||
this License, each Contributor hereby grants to You a perpetual, |
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable |
||||
copyright license to reproduce, prepare Derivative Works of, |
||||
publicly display, publicly perform, sublicense, and distribute the |
||||
Work and such Derivative Works in Source or Object form. |
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of |
||||
this License, each Contributor hereby grants to You a perpetual, |
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable |
||||
(except as stated in this section) patent license to make, have made, |
||||
use, offer to sell, sell, import, and otherwise transfer the Work, |
||||
where such license applies only to those patent claims licensable |
||||
by such Contributor that are necessarily infringed by their |
||||
Contribution(s) alone or by combination of their Contribution(s) |
||||
with the Work to which such Contribution(s) was submitted. If You |
||||
institute patent litigation against any entity (including a |
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work |
||||
or a Contribution incorporated within the Work constitutes direct |
||||
or contributory patent infringement, then any patent licenses |
||||
granted to You under this License for that Work shall terminate |
||||
as of the date such litigation is filed. |
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the |
||||
Work or Derivative Works thereof in any medium, with or without |
||||
modifications, and in Source or Object form, provided that You |
||||
meet the following conditions: |
||||
|
||||
(a) You must give any other recipients of the Work or |
||||
Derivative Works a copy of this License; and |
||||
|
||||
(b) You must cause any modified files to carry prominent notices |
||||
stating that You changed the files; and |
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works |
||||
that You distribute, all copyright, patent, trademark, and |
||||
attribution notices from the Source form of the Work, |
||||
excluding those notices that do not pertain to any part of |
||||
the Derivative Works; and |
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its |
||||
distribution, then any Derivative Works that You distribute must |
||||
include a readable copy of the attribution notices contained |
||||
within such NOTICE file, excluding those notices that do not |
||||
pertain to any part of the Derivative Works, in at least one |
||||
of the following places: within a NOTICE text file distributed |
||||
as part of the Derivative Works; within the Source form or |
||||
documentation, if provided along with the Derivative Works; or, |
||||
within a display generated by the Derivative Works, if and |
||||
wherever such third-party notices normally appear. The contents |
||||
of the NOTICE file are for informational purposes only and |
||||
do not modify the License. You may add Your own attribution |
||||
notices within Derivative Works that You distribute, alongside |
||||
or as an addendum to the NOTICE text from the Work, provided |
||||
that such additional attribution notices cannot be construed |
||||
as modifying the License. |
||||
|
||||
You may add Your own copyright statement to Your modifications and |
||||
may provide additional or different license terms and conditions |
||||
for use, reproduction, or distribution of Your modifications, or |
||||
for any such Derivative Works as a whole, provided Your use, |
||||
reproduction, and distribution of the Work otherwise complies with |
||||
the conditions stated in this License. |
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise, |
||||
any Contribution intentionally submitted for inclusion in the Work |
||||
by You to the Licensor shall be under the terms and conditions of |
||||
this License, without any additional terms or conditions. |
||||
Notwithstanding the above, nothing herein shall supersede or modify |
||||
the terms of any separate license agreement you may have executed |
||||
with Licensor regarding such Contributions. |
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade |
||||
names, trademarks, service marks, or product names of the Licensor, |
||||
except as required for reasonable and customary use in describing the |
||||
origin of the Work and reproducing the content of the NOTICE file. |
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or |
||||
agreed to in writing, Licensor provides the Work (and each |
||||
Contributor provides its Contributions) on an "AS IS" BASIS, |
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
||||
implied, including, without limitation, any warranties or conditions |
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A |
||||
PARTICULAR PURPOSE. You are solely responsible for determining the |
||||
appropriateness of using or redistributing the Work and assume any |
||||
risks associated with Your exercise of permissions under this License. |
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory, |
||||
whether in tort (including negligence), contract, or otherwise, |
||||
unless required by applicable law (such as deliberate and grossly |
||||
negligent acts) or agreed to in writing, shall any Contributor be |
||||
liable to You for damages, including any direct, indirect, special, |
||||
incidental, or consequential damages of any character arising as a |
||||
result of this License or out of the use or inability to use the |
||||
Work (including but not limited to damages for loss of goodwill, |
||||
work stoppage, computer failure or malfunction, or any and all |
||||
other commercial damages or losses), even if such Contributor |
||||
has been advised of the possibility of such damages. |
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing |
||||
the Work or Derivative Works thereof, You may choose to offer, |
||||
and charge a fee for, acceptance of support, warranty, indemnity, |
||||
or other liability obligations and/or rights consistent with this |
||||
License. However, in accepting such obligations, You may act only |
||||
on Your own behalf and on Your sole responsibility, not on behalf |
||||
of any other Contributor, and only if You agree to indemnify, |
||||
defend, and hold each Contributor harmless for any liability |
||||
incurred by, or claims asserted against, such Contributor by reason |
||||
of your accepting any such warranty or additional liability. |
||||
|
||||
END OF TERMS AND CONDITIONS |
||||
|
||||
APPENDIX: How to apply the Apache License to your work. |
||||
|
||||
To apply the Apache License to your work, attach the following |
||||
boilerplate notice, with the fields enclosed by brackets "[]" |
||||
replaced with your own identifying information. (Don't include |
||||
the brackets!) The text should be enclosed in the appropriate |
||||
comment syntax for the file format. We also recommend that a |
||||
file or class name and description of purpose be included on the |
||||
same "printed page" as the copyright notice for easier |
||||
identification within third-party archives. |
||||
|
||||
Copyright [yyyy] [name of copyright owner] |
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); |
||||
you may not use this file except in compliance with the License. |
||||
You may obtain a copy of the License at |
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
Unless required by applicable law or agreed to in writing, software |
||||
distributed under the License is distributed on an "AS IS" BASIS, |
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
||||
See the License for the specific language governing permissions and |
||||
limitations under the License. |
||||
@ -0,0 +1,17 @@
|
||||
Swift |
||||
----- |
||||
|
||||
A distributed object store that was originally developed as the basis for |
||||
Rackspace's Cloud Files. |
||||
|
||||
To build documentation run `make html` in the /doc folder, and then browse to |
||||
/doc/build/html/index.html. |
||||
|
||||
The best place to get started is the "SAIO - Swift All In One", which will walk |
||||
you through setting up a development cluster of Swift in a VM. |
||||
|
||||
For more information, vist us at http://launchpad.net/swift, or come hang out |
||||
on our IRC channel, #openstack on freenode. |
||||
|
||||
-- |
||||
Swift Development Team |
||||
@ -0,0 +1,351 @@
|
||||
#!/usr/bin/python |
||||
# Copyright (c) 2010 OpenStack, LLC. |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
||||
# implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
import os |
||||
import sys |
||||
from urllib import quote |
||||
from hashlib import md5 |
||||
import getopt |
||||
from itertools import chain |
||||
|
||||
import simplejson |
||||
from eventlet.greenpool import GreenPool |
||||
from eventlet.event import Event |
||||
|
||||
from swift.common.ring import Ring |
||||
from swift.common.utils import split_path |
||||
from swift.common.bufferedhttp import http_connect |
||||
|
||||
|
||||
usage = """ |
||||
Usage! |
||||
|
||||
%(cmd)s [options] [url 1] [url 2] ... |
||||
-c [concurrency] Set the concurrency, default 50 |
||||
-r [ring dir] Ring locations, default /etc/swift |
||||
-e [filename] File for writing a list of inconsistent urls |
||||
-d Also download files and verify md5 |
||||
|
||||
You can also feed a list of urls to the script through stdin. |
||||
|
||||
Examples! |
||||
|
||||
%(cmd)s SOSO_88ad0b83-b2c5-4fa1-b2d6-60c597202076 |
||||
%(cmd)s SOSO_88ad0b83-b2c5-4fa1-b2d6-60c597202076/container/object |
||||
%(cmd)s -e errors.txt SOSO_88ad0b83-b2c5-4fa1-b2d6-60c597202076/container |
||||
%(cmd)s < errors.txt |
||||
%(cmd)s -c 25 -d < errors.txt |
||||
""" % {'cmd': sys.argv[0]} |
||||
|
||||
|
||||
class Auditor(object): |
||||
def __init__(self, swift_dir='/etc/swift', concurrency=50, deep=False, |
||||
error_file=None): |
||||
self.pool = GreenPool(concurrency) |
||||
self.object_ring = Ring(os.path.join(swift_dir, 'object.ring.gz')) |
||||
self.container_ring = Ring(os.path.join(swift_dir, 'container.ring.gz')) |
||||
self.account_ring = Ring(os.path.join(swift_dir, 'account.ring.gz')) |
||||
self.deep = deep |
||||
self.error_file = error_file |
||||
# zero out stats |
||||
self.accounts_checked = self.account_exceptions = \ |
||||
self.account_not_found = self.account_container_mismatch = \ |
||||
self.account_object_mismatch = self.objects_checked = \ |
||||
self.object_exceptions = self.object_not_found = \ |
||||
self.object_checksum_mismatch = self.containers_checked = \ |
||||
self.container_exceptions = self.container_count_mismatch = \ |
||||
self.container_not_found = self.container_obj_mismatch = 0 |
||||
self.list_cache = {} |
||||
self.in_progress = {} |
||||
|
||||
def audit_object(self, account, container, name): |
||||
path = '/%s/%s/%s' % (quote(account), quote(container), quote(name)) |
||||
part, nodes = self.object_ring.get_nodes(account, container, name) |
||||
container_listing = self.audit_container(account, container) |
||||
consistent = True |
||||
if name not in container_listing: |
||||
print " Object %s missing in container listing!" % path |
||||
consistent = False |
||||
hash = None |
||||
else: |
||||
hash = container_listing[name]['hash'] |
||||
etags = [] |
||||
for node in nodes: |
||||
try: |
||||
if self.deep: |
||||
conn = http_connect(node['ip'], node['port'], |
||||
node['device'], part, 'GET', path, {}) |
||||
resp = conn.getresponse() |
||||
calc_hash = md5() |
||||
chunk = True |
||||
while chunk: |
||||
chunk = resp.read(8192) |
||||
calc_hash.update(chunk) |
||||
calc_hash = calc_hash.hexdigest() |
||||
if resp.status // 100 != 2: |
||||
self.object_not_found += 1 |
||||
consistent = False |
||||
print ' Bad status GETting object "%s" on %s/%s' \ |
||||
% (path, node['ip'], node['device']) |
||||
continue |
||||
if resp.getheader('ETag').strip('"') != calc_hash: |
||||
self.object_checksum_mismatch += 1 |
||||
consistent = False |
||||
print ' MD5 doesnt match etag for "%s" on %s/%s' \ |
||||
% (path, node['ip'], node['device']) |
||||
etags.append(resp.getheader('ETag')) |
||||
else: |
||||
conn = http_connect(node['ip'], node['port'], |
||||
node['device'], part, 'HEAD', path, {}) |
||||
resp = conn.getresponse() |
||||
if resp.status // 100 != 2: |
||||
self.object_not_found += 1 |
||||
consistent = False |
||||
print ' Bad status HEADing object "%s" on %s/%s' \ |
||||
% (path, node['ip'], node['device']) |
||||
continue |
||||
etags.append(resp.getheader('ETag')) |
||||
except Exception: |
||||
self.object_exceptions += 1 |
||||
consistent = False |
||||
print ' Exception fetching object "%s" on %s/%s' \ |
||||
% (path, node['ip'], node['device']) |
||||
continue |
||||
if not etags: |
||||
consistent = False |
||||
print " Failed fo fetch object %s at all!" % path |
||||
elif hash: |
||||
for etag in etags: |
||||
if resp.getheader('ETag').strip('"') != hash: |
||||
consistent = False |
||||
self.object_checksum_mismatch += 1 |
||||
print ' ETag mismatch for "%s" on %s/%s' \ |
||||
% (path, node['ip'], node['device']) |
||||
if not consistent and self.error_file: |
||||
print >>open(self.error_file, 'a'), path |
||||
self.objects_checked += 1 |
||||
|
||||
def audit_container(self, account, name, recurse=False): |
||||
if (account, name) in self.in_progress: |
||||
self.in_progress[(account, name)].wait() |
||||
if (account, name) in self.list_cache: |
||||
return self.list_cache[(account, name)] |
||||
self.in_progress[(account, name)] = Event() |
||||
print 'Auditing container "%s"...' % name |
||||
path = '/%s/%s' % (quote(account), quote(name)) |
||||
account_listing = self.audit_account(account) |
||||
consistent = True |
||||
if name not in account_listing: |
||||
consistent = False |
||||
print " Container %s not in account listing!" % path |
||||
part, nodes = self.container_ring.get_nodes(account, name) |
||||
rec_d = {} |
||||
responses = {} |
||||
for node in nodes: |
||||
marker = '' |
||||
results = True |
||||
while results: |
||||
node_id = node['id'] |
||||
try: |
||||
conn = http_connect(node['ip'], node['port'], node['device'], |
||||
part, 'GET', path, {}, |
||||
'format=json&marker=%s' % quote(marker)) |
||||
resp = conn.getresponse() |
||||
if resp.status // 100 != 2: |
||||
self.container_not_found += 1 |
||||
consistent = False |
||||
print ' Bad status GETting container "%s" on %s/%s' % \ |
||||
(path, node['ip'], node['device']) |
||||
break |
||||
if node['id'] not in responses: |
||||
responses[node['id']] = dict(resp.getheaders()) |
||||
results = simplejson.loads(resp.read()) |
||||
except Exception: |
||||
self.container_exceptions += 1 |
||||
consistent = False |
||||
print ' Exception GETting container "%s" on %s/%s' % \ |
||||
(path, node['ip'], node['device']) |
||||
break |
||||
if results: |
||||
marker = results[-1]['name'] |
||||
for obj in results: |
||||
obj_name = obj['name'] |
||||
if obj_name not in rec_d: |
||||
rec_d[obj_name] = obj |
||||
if obj['last_modified'] != rec_d[obj_name]['last_modified']: |
||||
self.container_obj_mismatch += 1 |
||||
consistent = False |
||||
print " Different versions of %s/%s in container dbs." % \ |
||||
(quote(name), quote(obj['name'])) |
||||
if obj['last_modified'] > rec_d[obj_name]['last_modified']: |
||||
rec_d[obj_name] = obj |
||||
obj_counts = [int(header['x-container-object-count']) |
||||
for header in responses.values()] |
||||
if not obj_counts: |
||||
consistent = False |
||||
print " Failed to fetch container %s at all!" % path |
||||
else: |
||||
if len(set(obj_counts)) != 1: |
||||
self.container_count_mismatch += 1 |
||||
consistent = False |
||||
print " Container databases don't agree on number of objects." |
||||
print " Max: %s, Min: %s" % (max(obj_counts), min(obj_counts)) |
||||
self.containers_checked += 1 |
||||
self.list_cache[(account, name)] = rec_d |
||||
self.in_progress[(account, name)].send(True) |
||||
del self.in_progress[(account, name)] |
||||
if recurse: |
||||
for obj in rec_d.keys(): |
||||
self.pool.spawn_n(self.audit_object, account, name, obj) |
||||
if not consistent and self.error_file: |
||||
print >>open(self.error_file, 'a'), path |
||||
return rec_d |
||||
|
||||
def audit_account(self, account, recurse=False): |
||||
if account in self.in_progress: |
||||
self.in_progress[account].wait() |
||||
if account in self.list_cache: |
||||
return self.list_cache[account] |
||||
self.in_progress[account] = Event() |
||||
print "Auditing account %s..." % account |
||||
consistent = True |
||||
path = '/%s' % account |
||||
part, nodes = self.account_ring.get_nodes(account) |
||||
responses = {} |
||||
for node in nodes: |
||||
marker = '' |
||||
results = True |
||||
while results: |
||||
node_id = node['id'] |
||||
try: |
||||
conn = http_connect(node['ip'], node['port'], |
||||
node['device'], part, 'GET', path, {}, |
||||
'format=json&marker=%s' % quote(marker)) |
||||
resp = conn.getresponse() |
||||
if resp.status // 100 != 2: |
||||
self.account_not_found += 1 |
||||
consistent = False |
||||
print " Bad status GETting account %(ip)s:%(device)s" \ |
||||
% node |
||||
break |
||||
results = simplejson.loads(resp.read()) |
||||
except Exception: |
||||
self.account_exceptions += 1 |
||||
consistent = False |
||||
print " Exception GETting account %(ip)s:%(device)s" % node |
||||
break |
||||
if node_id not in responses: |
||||
responses[node_id] = [dict(resp.getheaders()), []] |
||||
responses[node_id][1].extend(results) |
||||
if results: |
||||
marker = results[-1]['name'] |
||||
headers = [resp[0] for resp in responses.values()] |
||||
cont_counts = [int(header['x-account-container-count']) |
||||
for header in headers] |
||||
if len(set(cont_counts)) != 1: |
||||
self.account_container_mismatch += 1 |
||||
consistent = False |
||||
print " Account databases don't agree on number of containers." |
||||
print " Max: %s, Min: %s" % (max(cont_counts), min(cont_counts)) |
||||
obj_counts = [int(header['x-account-object-count']) |
||||
for header in headers] |
||||
if len(set(obj_counts)) != 1: |
||||
self.account_object_mismatch += 1 |
||||
consistent = False |
||||
print " Account databases don't agree on number of objects." |
||||
print " Max: %s, Min: %s" % (max(obj_counts), min(obj_counts)) |
||||
containers = set() |
||||
for resp in responses.values(): |
||||
containers.update(container['name'] for container in resp[1]) |
||||
self.list_cache[account] = containers |
||||
self.in_progress[account].send(True) |
||||
del self.in_progress[account] |
||||
self.accounts_checked += 1 |
||||
if recurse: |
||||
for container in containers: |
||||
self.pool.spawn_n(self.audit_container, account, container, True) |
||||
if not consistent and self.error_file: |
||||
print >>open(self.error_file, 'a'), path |
||||
return containers |
||||
|
||||
def audit(self, account, container=None, obj=None): |
||||
if obj and container: |
||||
self.pool.spawn_n(self.audit_object, account, container, obj) |
||||
elif container: |
||||
self.pool.spawn_n(self.audit_container, account, container, True) |
||||
else: |
||||
self.pool.spawn_n(self.audit_account, account, True) |
||||
|
||||
def wait(self): |
||||
self.pool.waitall() |
||||
|
||||
def print_stats(self): |
||||
print |
||||
print " Accounts checked: %d" % self.accounts_checked |
||||
if self.account_not_found: |
||||
print " Missing Replicas: %d" % self.account_not_found |
||||
if self.account_exceptions: |
||||
print " Exceptions: %d" % self.account_exceptions |
||||
if self.account_container_mismatch: |
||||
print " Cntainer mismatch: %d" % self.account_container_mismatch |
||||
if self.account_object_mismatch: |
||||
print " Object mismatch: %d" % self.account_object_mismatch |
||||
print |
||||
print "Containers checked: %d" % self.containers_checked |
||||
if self.container_not_found: |
||||
print " Missing Replicas: %d" % self.container_not_found |
||||
if self.container_exceptions: |
||||
print " Exceptions: %d" % self.container_exceptions |
||||
if self.container_count_mismatch: |
||||
print " Count mismatch: %d" % self.container_count_mismatch |
||||
if self.container_obj_mismatch: |
||||
print " Obj mismatch: %d" % self.container_obj_mismatch |
||||
print |
||||
print " Objects checked: %d" % self.objects_checked |
||||
if self.object_not_found: |
||||
print " Missing Replicas: %d" % self.object_not_found |
||||
if self.object_exceptions: |
||||
print " Exceptions: %d" % self.object_exceptions |
||||
if self.object_checksum_mismatch: |
||||
print " MD5 Mismatch: %d" % self.object_checksum_mismatch |
||||
|
||||
|
||||
if __name__ == '__main__': |
||||
try: |
||||
optlist, args = getopt.getopt(sys.argv[1:], 'c:r:e:d') |
||||
except getopt.GetoptError, err: |
||||
print str(err) |
||||
print usage |
||||
sys.exit(2) |
||||
if not args and os.isatty(sys.stdin.fileno()): |
||||
print usage |
||||
sys.exit() |
||||
opts = dict(optlist) |
||||
options = { |
||||
'concurrency': int(opts.get('-c', 50)), |
||||
'error_file': opts.get('-e', None), |
||||
'swift_dir': opts.get('-r', '/etc/swift'), |
||||
'deep': '-d' in opts, |
||||
} |
||||
auditor = Auditor(**options) |
||||
if not os.isatty(sys.stdin.fileno()): |
||||
args = chain(args, sys.stdin) |
||||
for path in args: |
||||
path = '/' + path.rstrip('\r\n').lstrip('/') |
||||
auditor.audit(*split_path(path, 1, 3, True)) |
||||
auditor.wait() |
||||
auditor.print_stats() |
||||
|
||||
@ -0,0 +1,69 @@
|
||||
#!/usr/bin/python |
||||
# Copyright (c) 2010 OpenStack, LLC. |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
||||
# implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
import os |
||||
import signal |
||||
import sys |
||||
from ConfigParser import ConfigParser |
||||
|
||||
from swift.account.auditor import AccountAuditor |
||||
from swift.common import utils |
||||
|
||||
if __name__ == '__main__': |
||||
if len(sys.argv) < 2: |
||||
print "Usage: account-auditor CONFIG_FILE [once]" |
||||
sys.exit() |
||||
|
||||
once = len(sys.argv) > 2 and sys.argv[2] == 'once' |
||||
|
||||
c = ConfigParser() |
||||
if not c.read(sys.argv[1]): |
||||
print "Unable to read config file." |
||||
sys.exit(1) |
||||
|
||||
server_conf = dict(c.items('account-server')) |
||||
if c.has_section('account-auditor'): |
||||
auditor_conf = dict(c.items('account-auditor')) |
||||
else: |
||||
print "Unable to find account-auditor config section in %s." % \ |
||||
sys.argv[1] |
||||
sys.exit(1) |
||||
|
||||
logger = utils.get_logger(auditor_conf, 'account-auditor') |
||||
# log uncaught exceptions |
||||
sys.excepthook = lambda *exc_info: \ |
||||
logger.critical('UNCAUGHT EXCEPTION', exc_info=exc_info) |
||||
sys.stdout = sys.stderr = utils.LoggerFileObject(logger) |
||||
|
||||
utils.drop_privileges(server_conf.get('user', 'swift')) |
||||
|
||||
try: |
||||
os.setsid() |
||||
except OSError: |
||||
pass |
||||
|
||||
def kill_children(*args): |
||||
signal.signal(signal.SIGTERM, signal.SIG_IGN) |
||||
os.killpg(0, signal.SIGTERM) |
||||
sys.exit() |
||||
|
||||
signal.signal(signal.SIGTERM, kill_children) |
||||
|
||||
auditor = AccountAuditor(server_conf, auditor_conf) |
||||
if once: |
||||
auditor.audit_once() |
||||
else: |
||||
auditor.audit_forever() |
||||
@ -0,0 +1,69 @@
|
||||
#!/usr/bin/python |
||||
# Copyright (c) 2010 OpenStack, LLC. |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
||||
# implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
import os |
||||
import signal |
||||
import sys |
||||
from ConfigParser import ConfigParser |
||||
|
||||
from swift.account.reaper import AccountReaper |
||||
from swift.common import utils |
||||
|
||||
if __name__ == '__main__': |
||||
if len(sys.argv) < 2: |
||||
print "Usage: account-reaper CONFIG_FILE [once]" |
||||
sys.exit() |
||||
|
||||
once = len(sys.argv) > 2 and sys.argv[2] == 'once' |
||||
|
||||
c = ConfigParser() |
||||
if not c.read(sys.argv[1]): |
||||
print "Unable to read config file." |
||||
sys.exit(1) |
||||
|
||||
server_conf = dict(c.items('account-server')) |
||||
if c.has_section('account-reaper'): |
||||
reaper_conf = dict(c.items('account-reaper')) |
||||
else: |
||||
print "Unable to find account-reaper config section in %s." % \ |
||||
sys.argv[1] |
||||
sys.exit(1) |
||||
|
||||
logger = utils.get_logger(reaper_conf, 'account-reaper') |
||||
# log uncaught exceptions |
||||
sys.excepthook = lambda *exc_info: \ |
||||
logger.critical('UNCAUGHT EXCEPTION', exc_info=exc_info) |
||||
sys.stdout = sys.stderr = utils.LoggerFileObject(logger) |
||||
|
||||
utils.drop_privileges(server_conf.get('user', 'swift')) |
||||
|
||||
try: |
||||
os.setsid() |
||||
except OSError: |
||||
pass |
||||
|
||||
def kill_children(*args): |
||||
signal.signal(signal.SIGTERM, signal.SIG_IGN) |
||||
os.killpg(0, signal.SIGTERM) |
||||
sys.exit() |
||||
|
||||
signal.signal(signal.SIGTERM, kill_children) |
||||
|
||||
reaper = AccountReaper(server_conf, reaper_conf) |
||||
if once: |
||||
reaper.reap_once() |
||||
else: |
||||
reaper.reap_forever() |
||||
@ -0,0 +1,57 @@
|
||||
#!/usr/bin/python |
||||
# Copyright (c) 2010 OpenStack, LLC. |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
||||
# implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
import sys |
||||
from ConfigParser import ConfigParser |
||||
import getopt |
||||
|
||||
from swift.account import server as account_server |
||||
from swift.common import db, db_replicator, utils |
||||
|
||||
class AccountReplicator(db_replicator.Replicator): |
||||
server_type = 'account' |
||||
ring_file = 'account.ring.gz' |
||||
brokerclass = db.AccountBroker |
||||
datadir = account_server.DATADIR |
||||
default_port = 6002 |
||||
|
||||
if __name__ == '__main__': |
||||
optlist, args = getopt.getopt(sys.argv[1:], '', ['once']) |
||||
|
||||
if not args: |
||||
print "Usage: account-replicator <--once> CONFIG_FILE [once]" |
||||
sys.exit() |
||||
|
||||
c = ConfigParser() |
||||
if not c.read(args[0]): |
||||
print "Unable to read config file." |
||||
sys.exit(1) |
||||
once = len(args) > 1 and args[1] == 'once' |
||||
|
||||
server_conf = dict(c.items('account-server')) |
||||
if c.has_section('account-replicator'): |
||||
replicator_conf = dict(c.items('account-replicator')) |
||||
else: |
||||
print "Unable to find account-replicator config section in %s." % \ |
||||
args[0] |
||||
sys.exit(1) |
||||
|
||||
utils.drop_privileges(server_conf.get('user', 'swift')) |
||||
if once or '--once' in [opt[0] for opt in optlist]: |
||||
AccountReplicator(server_conf, replicator_conf).replicate_once() |
||||
else: |
||||
AccountReplicator(server_conf, replicator_conf).replicate_forever() |
||||
|
||||
@ -0,0 +1,30 @@
|
||||
#!/usr/bin/python |
||||
# Copyright (c) 2010 OpenStack, LLC. |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
||||
# implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
from ConfigParser import ConfigParser |
||||
import sys |
||||
|
||||
from swift.common.wsgi import run_wsgi |
||||
from swift.account.server import AccountController |
||||
|
||||
if __name__ == '__main__': |
||||
c = ConfigParser() |
||||
if not c.read(sys.argv[1]): |
||||
print "Unable to read config file." |
||||
sys.exit(1) |
||||
conf = dict(c.items('account-server')) |
||||
run_wsgi(AccountController, conf, default_port=6002) |
||||
|
||||
@ -0,0 +1,45 @@
|
||||
#!/usr/bin/python |
||||
# Copyright (c) 2010 OpenStack, LLC. |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
||||
# implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
from ConfigParser import ConfigParser |
||||
from sys import argv, exit |
||||
|
||||
from swift.common.bufferedhttp import http_connect_raw as http_connect |
||||
|
||||
|
||||
if __name__ == '__main__': |
||||
f = '/etc/swift/auth-server.conf' |
||||
if len(argv) == 5: |
||||
f = argv[4] |
||||
elif len(argv) != 4: |
||||
exit('Syntax: %s <new_account> <new_user> <new_password> [conf_file]' % |
||||
argv[0]) |
||||
new_account = argv[1] |
||||
new_user = argv[2] |
||||
new_password = argv[3] |
||||
c = ConfigParser() |
||||
if not c.read(f): |
||||
exit('Unable to read conf file: %s' % f) |
||||
conf = dict(c.items('auth-server')) |
||||
host = conf.get('bind_ip', '127.0.0.1') |
||||
port = int(conf.get('bind_port', 11000)) |
||||
path = '/account/%s/%s' % (new_account, new_user) |
||||
conn = http_connect(host, port, 'PUT', path, {'x-auth-key':new_password}) |
||||
resp = conn.getresponse() |
||||
if resp.status == 204: |
||||
print resp.getheader('x-storage-url') |
||||
else: |
||||
print 'Account creation failed. (%d)' % resp.status |
||||
@ -0,0 +1,40 @@
|
||||
#!/usr/bin/python |
||||
# Copyright (c) 2010 OpenStack, LLC. |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
||||
# implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
from ConfigParser import ConfigParser |
||||
from sys import argv, exit |
||||
|
||||
from swift.common.bufferedhttp import http_connect_raw as http_connect |
||||
|
||||
if __name__ == '__main__': |
||||
f = '/etc/swift/auth-server.conf' |
||||
if len(argv) == 2: |
||||
f = argv[1] |
||||
elif len(argv) != 1: |
||||
exit('Syntax: %s [conf_file]' % argv[0]) |
||||
c = ConfigParser() |
||||
if not c.read(f): |
||||
exit('Unable to read conf file: %s' % f) |
||||
conf = dict(c.items('auth-server')) |
||||
host = conf.get('bind_ip', '127.0.0.1') |
||||
port = int(conf.get('bind_port', 11000)) |
||||
path = '/recreate_accounts' |
||||
conn = http_connect(host, port, 'POST', path) |
||||
resp = conn.getresponse() |
||||
if resp.status == 200: |
||||
print resp.read() |
||||
else: |
||||
print 'Recreating accounts failed. (%d)' % resp.status |
||||
@ -0,0 +1,30 @@
|
||||
#!/usr/bin/python |
||||
# Copyright (c) 2010 OpenStack, LLC. |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
||||
# implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
from ConfigParser import ConfigParser |
||||
import sys |
||||
|
||||
from swift.common.wsgi import run_wsgi |
||||
from swift.auth.server import AuthController |
||||
|
||||
if __name__ == '__main__': |
||||
c = ConfigParser() |
||||
if not c.read(sys.argv[1]): |
||||
print "Unable to read config file." |
||||
sys.exit(1) |
||||
conf = dict(c.items('auth-server')) |
||||
run_wsgi(AuthController, conf, default_port=11000) |
||||
|
||||
@ -0,0 +1,69 @@
|
||||
#!/usr/bin/python |
||||
# Copyright (c) 2010 OpenStack, LLC. |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
||||
# implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
import os |
||||
import signal |
||||
import sys |
||||
from ConfigParser import ConfigParser |
||||
|
||||
from swift.container.auditor import ContainerAuditor |
||||
from swift.common import utils |
||||
|
||||
if __name__ == '__main__': |
||||
if len(sys.argv) < 2: |
||||
print "Usage: container-auditor CONFIG_FILE [once]" |
||||
sys.exit() |
||||
|
||||
once = len(sys.argv) > 2 and sys.argv[2] == 'once' |
||||
|
||||
c = ConfigParser() |
||||
if not c.read(sys.argv[1]): |
||||
print "Unable to read config file." |
||||
sys.exit(1) |
||||
|
||||
server_conf = dict(c.items('container-server')) |
||||
if c.has_section('container-auditor'): |
||||
auditor_conf = dict(c.items('container-auditor')) |
||||
else: |
||||
print "Unable to find container-auditor config section in %s." % \ |
||||
sys.argv[1] |
||||
sys.exit(1) |
||||
|
||||
logger = utils.get_logger(auditor_conf, 'container-auditor') |
||||
# log uncaught exceptions |
||||
sys.excepthook = lambda *exc_info: \ |
||||
logger.critical('UNCAUGHT EXCEPTION', exc_info=exc_info) |
||||
sys.stdout = sys.stderr = utils.LoggerFileObject(logger) |
||||
|
||||
utils.drop_privileges(server_conf.get('user', 'swift')) |
||||
|
||||
try: |
||||
os.setsid() |
||||
except OSError: |
||||
pass |
||||
|
||||
def kill_children(*args): |
||||
signal.signal(signal.SIGTERM, signal.SIG_IGN) |
||||
os.killpg(0, signal.SIGTERM) |
||||
sys.exit() |
||||
|
||||
signal.signal(signal.SIGTERM, kill_children) |
||||
|
||||
auditor = ContainerAuditor(server_conf, auditor_conf) |
||||
if once: |
||||
auditor.audit_once() |
||||
else: |
||||
auditor.audit_forever() |
||||
@ -0,0 +1,57 @@
|
||||
#!/usr/bin/python |
||||
# Copyright (c) 2010 OpenStack, LLC. |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
||||
# implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
import sys |
||||
from ConfigParser import ConfigParser |
||||
import getopt |
||||
|
||||
from swift.container import server as container_server |
||||
from swift.common import db, db_replicator, utils |
||||
|
||||
class ContainerReplicator(db_replicator.Replicator): |
||||
server_type = 'container' |
||||
ring_file = 'container.ring.gz' |
||||
brokerclass = db.ContainerBroker |
||||
datadir = container_server.DATADIR |
||||
default_port = 6001 |
||||
|
||||
if __name__ == '__main__': |
||||
optlist, args = getopt.getopt(sys.argv[1:], '', ['once']) |
||||
|
||||
if not args: |
||||
print "Usage: container-replicator <--once> CONFIG_FILE [once]" |
||||
sys.exit() |
||||
|
||||
c = ConfigParser() |
||||
if not c.read(args[0]): |
||||
print "Unable to read config file." |
||||
sys.exit(1) |
||||
once = len(args) > 1 and args[1] == 'once' |
||||
|
||||
server_conf = dict(c.items('container-server')) |
||||
if c.has_section('container-replicator'): |
||||
replicator_conf = dict(c.items('container-replicator')) |
||||
else: |
||||
print "Unable to find container-replicator config section in %s." % \ |
||||
args[0] |
||||
sys.exit(1) |
||||
|
||||
utils.drop_privileges(server_conf.get('user', 'swift')) |
||||
if once or '--once' in [opt[0] for opt in optlist]: |
||||
ContainerReplicator(server_conf, replicator_conf).replicate_once() |
||||
else: |
||||
ContainerReplicator(server_conf, replicator_conf).replicate_forever() |
||||
|
||||
@ -0,0 +1,30 @@
|
||||
#!/usr/bin/python |
||||
# Copyright (c) 2010 OpenStack, LLC. |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
||||
# implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
from ConfigParser import ConfigParser |
||||
import sys |
||||
|
||||
from swift.common.wsgi import run_wsgi |
||||
from swift.container.server import ContainerController |
||||
|
||||
if __name__ == '__main__': |
||||
c = ConfigParser() |
||||
if not c.read(sys.argv[1]): |
||||
print "Unable to read config file." |
||||
sys.exit(1) |
||||
conf = dict(c.items('container-server')) |
||||
run_wsgi(ContainerController, conf, default_port=6001) |
||||
|
||||
@ -0,0 +1,63 @@
|
||||
#!/usr/bin/python |
||||
# Copyright (c) 2010 OpenStack, LLC. |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
||||
# implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
import os |
||||
import signal |
||||
import sys |
||||
from ConfigParser import ConfigParser |
||||
|
||||
from swift.container.updater import ContainerUpdater |
||||
from swift.common import utils |
||||
|
||||
if __name__ == '__main__': |
||||
if len(sys.argv) < 2: |
||||
print "Usage: container-updater CONFIG_FILE [once]" |
||||
sys.exit() |
||||
|
||||
once = len(sys.argv) > 2 and sys.argv[2] == 'once' |
||||
|
||||
c = ConfigParser() |
||||
if not c.read(sys.argv[1]): |
||||
print "Unable to read config file." |
||||
sys.exit(1) |
||||
|
||||
server_conf = dict(c.items('container-server')) |
||||
if c.has_section('container-updater'): |
||||
updater_conf = dict(c.items('container-updater')) |
||||
else: |
||||
print "Unable to find container-updater config section in %s." % \ |
||||
sys.argv[1] |
||||
sys.exit(1) |
||||
|
||||
utils.drop_privileges(server_conf.get('user', 'swift')) |
||||
|
||||
try: |
||||
os.setsid() |
||||
except OSError: |
||||
pass |
||||
|
||||
def kill_children(*args): |
||||
signal.signal(signal.SIGTERM, signal.SIG_IGN) |
||||
os.killpg(0, signal.SIGTERM) |
||||
sys.exit() |
||||
|
||||
signal.signal(signal.SIGTERM, kill_children) |
||||
|
||||
updater = ContainerUpdater(server_conf, updater_conf) |
||||
if once: |
||||
updater.update_once_single_threaded() |
||||
else: |
||||
updater.update_forever() |
||||
@ -0,0 +1,125 @@
|
||||
#!/usr/bin/python |
||||
# Copyright (c) 2010 OpenStack, LLC. |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
||||
# implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
import datetime |
||||
import os |
||||
import re |
||||
import subprocess |
||||
import sys |
||||
from ConfigParser import ConfigParser |
||||
|
||||
from swift.common.utils import get_logger |
||||
|
||||
# To search for more types of errors, add the regex to the list below |
||||
error_re = [ |
||||
'error.*(sd[a-z])', |
||||
'(sd[a-z]).*error', |
||||
] |
||||
|
||||
def get_devices(device_dir, logger): |
||||
devices = [] |
||||
for line in open('/proc/mounts').readlines(): |
||||
data = line.strip().split() |
||||
block_device = data[0] |
||||
mount_point = data[1] |
||||
if mount_point.startswith(device_dir): |
||||
device = {} |
||||
device['mount_point'] = mount_point |
||||
device['block_device'] = block_device |
||||
try: |
||||
device_num = os.stat(block_device).st_rdev |
||||
except OSError, e: |
||||
# If we can't stat the device, then something weird is going on |
||||
logger.error("Error: Could not stat %s!" % |
||||
block_device) |
||||
continue |
||||
device['major'] = str(os.major(device_num)) |
||||
device['minor'] = str(os.minor(device_num)) |
||||
devices.append(device) |
||||
for line in open('/proc/partitions').readlines()[2:]: |
||||
major,minor,blocks,kernel_device = line.strip().split() |
||||
device = [d for d in devices |
||||
if d['major'] == major and d['minor'] == minor] |
||||
if device: |
||||
device[0]['kernel_device'] = kernel_device |
||||
return devices |
||||
|
||||
def get_errors(minutes): |
||||
errors = {} |
||||
start_time = datetime.datetime.now() - datetime.timedelta(minutes=minutes) |
||||
for line in open('/var/log/kern.log'): |
||||
if '[ 0.000000]' in line: |
||||
# Ignore anything before the last boot |
||||
errors = {} |
||||
continue |
||||
log_time_string = '%s %s' % (start_time.year,' '.join(line.split()[:3])) |
||||
log_time = datetime.datetime.strptime( |
||||
log_time_string,'%Y %b %d %H:%M:%S') |
||||
if log_time > start_time: |
||||
for err in error_re: |
||||
for device in re.findall(err,line): |
||||
errors[device] = errors.get(device,0) + 1 |
||||
return errors |
||||
|
||||
def comment_fstab(mount_point): |
||||
with open('/etc/fstab', 'r') as fstab: |
||||
with open('/etc/fstab.new', 'w') as new_fstab: |
||||
for line in fstab: |
||||
parts = line.split() |
||||
if len(parts) > 2 and line.split()[1] == mount_point: |
||||
new_fstab.write('#' + line) |
||||
else: |
||||
new_fstab.write(line) |
||||
os.rename('/etc/fstab.new', '/etc/fstab') |
||||
|
||||
if __name__ == '__main__': |
||||
c = ConfigParser() |
||||
try: |
||||
conf_path = sys.argv[1] |
||||
except: |
||||
print "Usage: %s CONF_FILE" % sys.argv[0].split('/')[-1] |
||||
sys.exit(1) |
||||
if not c.read(conf_path): |
||||
print "Unable to read config file %s" % conf_path |
||||
sys.exit(1) |
||||
conf = dict(c.items('drive-audit')) |
||||
device_dir = conf.get('device_dir', '/srv/node') |
||||
minutes = int(conf.get('minutes', 60)) |
||||
error_limit = int(conf.get('error_limit', 1)) |
||||
logger = get_logger(conf, 'drive-audit') |
||||
devices = get_devices(device_dir, logger) |
||||
logger.debug("Devices found: %s" % str(devices)) |
||||
if not devices: |
||||
logger.error("Error: No devices found!") |
||||
errors = get_errors(minutes) |
||||
logger.debug("Errors found: %s" % str(errors)) |
||||
unmounts = 0 |
||||
for kernel_device,count in errors.items(): |
||||
if count >= error_limit: |
||||
device = [d for d in devices |
||||
if d['kernel_device'].startswith(kernel_device)] |
||||
if device: |
||||
mount_point = device[0]['mount_point'] |
||||
if mount_point.startswith('/srv/node'): |
||||
logger.info("Unmounting %s with %d errors" % |
||||
(mount_point, count)) |
||||
subprocess.call(['umount','-fl',mount_point]) |
||||
logger.info("Commenting out %s from /etc/fstab" % |
||||
(mount_point)) |
||||
comment_fstab(mount_point) |
||||
unmounts += 1 |
||||
if unmounts == 0: |
||||
logger.info("No drives were unmounted") |
||||
@ -0,0 +1,87 @@
|
||||
#!/usr/bin/python |
||||
# Copyright (c) 2010 OpenStack, LLC. |
||||
# |
||||
# Licensed under the Apache License, Version 2.0 (the "License"); |
||||
# you may not use this file except in compliance with the License. |
||||
# You may obtain a copy of the License at |
||||
# |
||||
# http://www.apache.org/licenses/LICENSE-2.0 |
||||
# |
||||
# Unless required by applicable law or agreed to in writing, software |
||||
# distributed under the License is distributed on an "AS IS" BASIS, |
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or |
||||
# implied. |
||||
# See the License for the specific language governing permissions and |
||||
# limitations under the License. |
||||
|
||||
import sys |
||||
import urllib |
||||
|
||||
from swift.common.ring import Ring |
||||
from swift.common.utils import hash_path |
||||
|
||||
|
||||
if len(sys.argv) < 3 or len(sys.argv) > 5: |
||||
print 'Usage: %s <ring.gz> <account> [<container>] [<object>]' % sys.argv[0] |
||||
print 'Shows the nodes responsible for the item specified.' |
||||
print 'Example:' |
||||
print ' $ %s /etc/swift/account.ring.gz MyAccount' % sys.argv[0] |
||||
print ' Partition 5743883' |
||||