Browse Source

Merge branch 'master' into feature/hummingbird

Change-Id: Ib4a2d12a47f023235c1dc1d67d3c458cc967a7b4
changes/10/221410/1
John Dickinson 6 years ago
parent
commit
eb8f1f83f1
193 changed files with 13584 additions and 8004 deletions
  1. +3
    -0
      .mailmap
  2. +19
    -0
      AUTHORS
  3. +154
    -11
      CHANGELOG
  4. +149
    -0
      bandit.yaml
  5. +1
    -1
      bin/swift-account-info
  6. +1
    -1
      bin/swift-container-info
  7. +1
    -1
      bin/swift-container-sync
  8. +13
    -3
      bin/swift-dispersion-populate
  9. +16
    -5
      bin/swift-dispersion-report
  10. +1
    -1
      bin/swift-recon
  11. +1
    -1
      bin/swift-ring-builder
  12. +1
    -1
      bin/swift-ring-builder-analyzer
  13. +0
    -6
      doc/manpages/account-server.conf.5
  14. +10
    -1
      doc/manpages/dispersion.conf.5
  15. +2
    -0
      doc/manpages/object-server.conf.5
  16. +3
    -0
      doc/manpages/swift-dispersion-populate.1
  17. +3
    -0
      doc/manpages/swift-dispersion-report.1
  18. +4
    -2
      doc/manpages/swift-recon.1
  19. +0
    -1
      doc/saio/swift/container-server/1.conf
  20. +0
    -1
      doc/saio/swift/container-server/2.conf
  21. +0
    -1
      doc/saio/swift/container-server/3.conf
  22. +0
    -1
      doc/saio/swift/container-server/4.conf
  23. +5
    -0
      doc/saio/swift/container-sync-realms.conf
  24. +1
    -1
      doc/saio/swift/object-expirer.conf
  25. +9
    -1
      doc/saio/swift/proxy-server.conf
  26. +14
    -4
      doc/source/admin_guide.rst
  27. +1
    -1
      doc/source/api/object_api_v1_overview.rst
  28. +1
    -1
      doc/source/cors.rst
  29. +92
    -71
      doc/source/deployment_guide.rst
  30. +21
    -8
      doc/source/development_guidelines.rst
  31. +1
    -1
      doc/source/development_ondisk_backends.rst
  32. +24
    -1
      doc/source/development_saio.rst
  33. +204
    -0
      doc/source/first_contribution_swift.rst
  34. +3
    -20
      doc/source/getting_started.rst
  35. +7
    -0
      doc/source/howto_installmultinode.rst
  36. +1
    -0
      doc/source/index.rst
  37. +2
    -1
      doc/source/logs.rst
  38. +9
    -0
      doc/source/middleware.rst
  39. +1
    -1
      doc/source/overview_architecture.rst
  40. +1
    -1
      doc/source/overview_erasure_code.rst
  41. +3
    -86
      doc/source/overview_object_versioning.rst
  42. +15
    -0
      doc/source/overview_ring.rst
  43. +0
    -7
      etc/account-server.conf-sample
  44. +10
    -0
      etc/dispersion.conf-sample
  45. +1
    -1
      etc/object-expirer.conf-sample
  46. +2
    -0
      etc/object-server.conf-sample
  47. +12
    -1
      etc/proxy-server.conf-sample
  48. +1
    -1
      etc/swift.conf-sample
  49. +1
    -1
      requirements.txt
  50. +1
    -0
      setup.cfg
  51. +4
    -5
      swift/account/auditor.py
  52. +11
    -3
      swift/account/backend.py
  53. +54
    -15
      swift/account/reaper.py
  54. +41
    -40
      swift/cli/form_signature.py
  55. +111
    -108
      swift/cli/info.py
  56. +89
    -82
      swift/cli/recon.py
  57. +62
    -38
      swift/cli/ring_builder_analyzer.py
  58. +170
    -159
      swift/cli/ringbuilder.py
  59. +20
    -13
      swift/common/constraints.py
  60. +6
    -5
      swift/common/container_sync_realms.py
  61. +1
    -1
      swift/common/db.py
  62. +33
    -4
      swift/common/db_replicator.py
  63. +4
    -4
      swift/common/direct_client.py
  64. +5
    -5
      swift/common/http.py
  65. +36
    -34
      swift/common/manager.py
  66. +2
    -2
      swift/common/memcached.py
  67. +3
    -1
      swift/common/middleware/dlo.py
  68. +2
    -2
      swift/common/middleware/keystoneauth.py
  69. +2
    -1
      swift/common/middleware/memcache.py
  70. +17
    -9
      swift/common/middleware/recon.py
  71. +5
    -2
      swift/common/middleware/slo.py
  72. +16
    -16
      swift/common/middleware/tempauth.py
  73. +110
    -26
      swift/common/middleware/tempurl.py
  74. +496
    -0
      swift/common/middleware/versioned_writes.py
  75. +1
    -1
      swift/common/middleware/x_profile/html_viewer.py
  76. +1
    -1
      swift/common/ring/builder.py
  77. +1
    -1
      swift/common/ring/ring.py
  78. +3
    -1
      swift/common/storage_policy.py
  79. +32
    -15
      swift/common/swob.py
  80. +15
    -11
      swift/common/utils.py
  81. +12
    -7
      swift/common/wsgi.py
  82. +1
    -1
      swift/container/backend.py
  83. +5
    -1
      swift/container/server.py
  84. +3
    -2
      swift/container/updater.py
  85. +5
    -5
      swift/locale/swift-log-critical.pot
  86. +5
    -5
      swift/locale/swift-log-error.pot
  87. +5
    -5
      swift/locale/swift-log-info.pot
  88. +5
    -5
      swift/locale/swift-log-warning.pot
  89. +210
    -230
      swift/locale/swift.pot
  90. +1118
    -0
      swift/locale/tr_TR/LC_MESSAGES/swift.po
  91. +5
    -5
      swift/locale/zh_CN/LC_MESSAGES/swift.po
  92. +3
    -3
      swift/obj/auditor.py
  93. +517
    -510
      swift/obj/diskfile.py
  94. +2
    -2
      swift/obj/mem_diskfile.py
  95. +49
    -26
      swift/obj/reconstructor.py
  96. +145
    -23
      swift/obj/replicator.py
  97. +39
    -9
      swift/obj/server.py
  98. +6
    -2
      swift/obj/ssync_receiver.py
  99. +8
    -3
      swift/obj/ssync_sender.py
  100. +2
    -2
      swift/obj/updater.py

+ 3
- 0
.mailmap View File

@ -78,3 +78,6 @@ Jaivish Kothari <jaivish.kothari@nectechnologies.in> <janonymous.codevulture@gma
Michael Matur <michael.matur@gmail.com>
Kazuhiro Miyahara <miyahara.kazuhiro@lab.ntt.co.jp>
Alexandra Settle <alexandra.settle@rackspace.com>
Kenichiro Matsuda <matsuda_kenichi@jp.fujitsu.com>
Atsushi Sakai <sakaia@jp.fujitsu.com>
Takashi Natsume <natsume.takashi@lab.ntt.co.jp>

+ 19
- 0
AUTHORS View File

@ -26,6 +26,7 @@ Chuck Thier (cthier@gmail.com)
Contributors
------------
Mehdi Abaakouk (mehdi.abaakouk@enovance.com)
Timur Alperovich (timur.alperovich@gmail.com)
Jesse Andrews (anotherjesse@gmail.com)
Joe Arnold (joe@swiftstack.com)
Ionuț Arțăriși (iartarisi@suse.cz)
@ -47,6 +48,7 @@ Tim Burke (tim.burke@gmail.com)
Brian D. Burns (iosctr@gmail.com)
Devin Carlen (devin.carlen@gmail.com)
Thierry Carrez (thierry@openstack.org)
Carlos Cavanna (ccavanna@ca.ibm.com)
Emmanuel Cazenave (contact@emcaz.fr)
Mahati Chamarthy (mahati.chamarthy@gmail.com)
Zap Chang (zapchang@gmail.com)
@ -55,6 +57,7 @@ Ray Chen (oldsharp@163.com)
Harshit Chitalia (harshit@acelio.com)
Brian Cline (bcline@softlayer.com)
Alistair Coles (alistair.coles@hp.com)
Clément Contini (ccontini@cloudops.com)
Brian Curtin (brian.curtin@rackspace.com)
Thiago da Silva (thiago@redhat.com)
Julien Danjou (julien@danjou.info)
@ -64,6 +67,7 @@ Cedric Dos Santos (cedric.dos.sant@gmail.com)
Gerry Drudy (gerry.drudy@hp.com)
Morgan Fainberg (morgan.fainberg@gmail.com)
ZhiQiang Fan (aji.zqfan@gmail.com)
Oshrit Feder (oshritf@il.ibm.com)
Mike Fedosin (mfedosin@mirantis.com)
Ricardo Ferreira (ricardo.sff@gmail.com)
Flaper Fesp (flaper87@gmail.com)
@ -91,8 +95,10 @@ Dan Hersam (dan.hersam@hp.com)
Derek Higgins (derekh@redhat.com)
Alex Holden (alex@alexjonasholden.com)
Edward Hope-Morley (opentastic@gmail.com)
Charles Hsu (charles0126@gmail.com)
Joanna H. Huang (joanna.huitzu.huang@gmail.com)
Kun Huang (gareth@unitedstack.com)
Bill Huber (wbhuber@us.ibm.com)
Matthieu Huin (mhu@enovance.com)
Hodong Hwang (hodong.hwang@kt.com)
Motonobu Ichimura (motonobu@gmail.com)
@ -126,6 +132,7 @@ John Leach (john@johnleach.co.uk)
Ed Leafe (ed.leafe@rackspace.com)
Thomas Leaman (thomas.leaman@hp.com)
Eohyung Lee (liquidnuker@gmail.com)
Zhao Lei (zhaolei@cn.fujitsu.com)
Jamie Lennox (jlennox@redhat.com)
Tong Li (litong01@us.ibm.com)
Changbin Liu (changbin.liu@gmail.com)
@ -136,10 +143,12 @@ Zhongyue Luo (zhongyue.nah@intel.com)
Paul Luse (paul.e.luse@intel.com)
Christopher MacGown (chris@pistoncloud.com)
Dragos Manolescu (dragosm@hp.com)
Ben Martin (blmartin@us.ibm.com)
Steve Martinelli (stevemar@ca.ibm.com)
Juan J. Martinez (juan@memset.com)
Marcelo Martins (btorch@gmail.com)
Dolph Mathews (dolph.mathews@gmail.com)
Kenichiro Matsuda (matsuda_kenichi@jp.fujitsu.com)
Michael Matur (michael.matur@gmail.com)
Donagh McCabe (donagh.mccabe@hp.com)
Andy McCrae (andy.mccrae@gmail.com)
@ -151,11 +160,13 @@ Jola Mirecka (jola.mirecka@hp.com)
Kazuhiro Miyahara (miyahara.kazuhiro@lab.ntt.co.jp)
Daisuke Morita (morita.daisuke@lab.ntt.co.jp)
Dirk Mueller (dirk@dmllr.de)
Takashi Natsume (natsume.takashi@lab.ntt.co.jp)
Russ Nelson (russ@crynwr.com)
Maru Newby (mnewby@internap.com)
Newptone (xingchao@unitedstack.com)
Colin Nicholson (colin.nicholson@iomart.com)
Zhenguo Niu (zhenguo@unitedstack.com)
Ondrej Novy (ondrej.novy@firma.seznam.cz)
Timothy Okwii (tokwii@cisco.com)
Matthew Oliver (matt@oliver.net.au)
Hisashi Osanai (osanai.hisashi@jp.fujitsu.com)
@ -169,18 +180,24 @@ Constantine Peresypkin (constantine.peresypk@rackspace.com)
Dieter Plaetinck (dieter@vimeo.com)
Dan Prince (dprince@redhat.com)
Sarvesh Ranjan (saranjan@cisco.com)
Falk Reimann (falk.reimann@sap.com)
Brian Reitz (brian.reitz@oracle.com)
Felipe Reyes (freyes@tty.cl)
Janie Richling (jrichli@us.ibm.com)
Matt Riedemann (mriedem@us.ibm.com)
Li Riqiang (lrqrun@gmail.com)
Rafael Rivero (rafael@cloudscaling.com)
Victor Rodionov (victor.rodionov@nexenta.com)
Eran Rom (eranr@il.ibm.com)
Aaron Rosen (arosen@nicira.com)
Brent Roskos (broskos@internap.com)
Hamdi Roumani (roumani@ca.ibm.com)
Shilla Saebi (shilla.saebi@gmail.com)
Atsushi Sakai (sakaia@jp.fujitsu.com)
Cristian A Sanchez (cristian.a.sanchez@intel.com)
Christian Schwede (cschwede@redhat.com)
Mark Seger (Mark.Seger@hp.com)
Azhagu Selvan SP (tamizhgeek@gmail.com)
Alexandra Settle (alexandra.settle@rackspace.com)
Andrew Clay Shafer (acs@parvuscaptus.com)
Mitsuhiro SHIGEMATSU (shigematsu.mitsuhiro@lab.ntt.co.jp)
@ -198,6 +215,7 @@ Jeremy Stanley (fungi@yuggoth.org)
Mauro Stettler (mauro.stettler@gmail.com)
Tobias Stevenson (tstevenson@vbridges.com)
Victor Stinner (vstinner@redhat.com)
Akihito Takai (takaiak@nttdata.co.jp)
Pearl Yajing Tan (pearl.y.tan@seagate.com)
Yuriy Taraday (yorik.sar@gmail.com)
Monty Taylor (mordred@inaugust.com)
@ -231,5 +249,6 @@ Guang Yee (guang.yee@hp.com)
Pete Zaitcev (zaitcev@kotori.zaitcev.us)
Hua Zhang (zhuadl@cn.ibm.com)
Jian Zhang (jian.zhang@intel.com)
Kai Zhang (zakir.exe@gmail.com)
Ning Zhang (ning@zmanda.com)
Yuan Zhou (yuan.zhou@intel.com)

+ 154
- 11
CHANGELOG View File

@ -1,4 +1,133 @@
swift (2.3.0)
swift (2.4.0)
* Dependency changes
- Added six requirement. This is part of an ongoing effort to add
support for Python 3.
- Dropped support for Python 2.6.
* Config changes
- Recent versions of Python restrict the number of headers allowed in a
request to 100. This number may be too low for custom middleware. The
new "extra_header_count" config value in swift.conf can be used to
increase the number of headers allowed.
- Renamed "run_pause" setting to "interval" (current configs with
run_pause still work). Future versions of Swift may remove the
run_pause setting.
* Versioned writes middleware
The versioned writes feature has been refactored and reimplemented as
middleware. You should explicitly add the versioned_writes middleware to
your proxy pipeline, but do not remove or disable the existing container
server config setting ("allow_versions"), if it is currently enabled.
The existing container server config setting enables existing
containers to continue being versioned. Please see
http://swift.openstack.org/middleware.html#how-to-enable-object-versioning-in-a-swift-cluster
for further upgrade notes.
* Allow 1+ object-servers-per-disk deployment
Enabled by a new > 0 integer config value, "servers_per_port" in the
[DEFAULT] config section for object-server and/or replication server
configs. The setting's integer value determines how many different
object-server workers handle requests for any single unique local port
in the ring. In this mode, the parent swift-object-server process
continues to run as the original user (i.e. root if low-port binding
is required), binds to all ports as defined in the ring, and forks off
the specified number of workers per listen socket. The child, per-port
servers drop privileges and behave pretty much how object-server workers
always have, except that because the ring has unique ports per disk, the
object-servers will only be handling requests for a single disk. The
parent process detects dead servers and restarts them (with the correct
listen socket), starts missing servers when an updated ring file is
found with a device on the server with a new port, and kills extraneous
servers when their port is found to no longer be in the ring. The ring
files are stat'ed at most every "ring_check_interval" seconds, as
configured in the object-server config (same default of 15s).
In testing, this deployment configuration (with a value of 3) lowers
request latency, improves requests per second, and isolates slow disk
IO as compared to the existing "workers" setting. To use this, each
device must be added to the ring using a different port.
* Do container listing updates in another (green)thread
The object server has learned the "container_update_timeout" setting
(with a default of 1 second). This value is the number of seconds that
the object server will wait for the container server to update the
listing before returning the status of the object PUT operation.
Previously, the object server would wait up to 3 seconds for the
container server response. The new behavior dramatically lowers object
PUT latency when container servers in the cluster are busy (e.g. when
the container is very large). Setting the value too low may result in a
client PUT'ing an object and not being able to immediately find it in
listings. Setting it too high will increase latency for clients when
container servers are busy.
* TempURL fixes (closes CVE-2015-5223)
Do not allow PUT tempurls to create pointers to other data.
Specifically, disallow the creation of DLO object manifests via a PUT
tempurl. This prevents discoverability attacks which can use any PUT
tempurl to probe for private data by creating a DLO object manifest and
then using the PUT tempurl to head the object.
* Ring changes
- Partition placement no longer uses the port number to place
partitions. This improves dispersion in small clusters running one
object server per drive, and it does not affect dispersion in
clusters running one object server per server.
- Added ring-builder-analyzer tool to more easily test and analyze a
series of ring management operations.
- Stop moving partitions unnecessarily when overload is on.
* Significant improvements and bug fixes have been made to erasure code
support. This feature is suitable for beta testing, but it is not yet
ready for broad production usage.
* Bulk upload now treats user xattrs on files in the given archive as
object metadata on the resulting created objects.
* Emit warning log in object replicator if "handoffs_first" or
"handoff_delete" is set.
* Enable object replicator's failure count in swift-recon.
* Added storage policy support to dispersion tools.
* Support keystone v3 domains in swift-dispersion.
* Added domain_remap information to the /info endpoint.
* Added support for a "default_reseller_prefix" in domain_remap
middleware config.
* Allow SLO PUTs to forgo per-segment integrity checks. Previously, each
segment referenced in the manifest also needed the correct etag and
bytes setting. These fields now allow the "null" value to skip those
particular checks on the given segment.
* Allow rsync to use compression via a "rsync_compress" config. If set to
true, compression is only enabled for an rsync to a device in a
different region. In some cases, this can speed up cross-region
replication data transfer.
* Added time synchronization check in swift-recon (the --time option).
* The account reaper now runs faster on large accounts.
* Various other minor bug fixes and improvements.
swift (2.3.0, OpenStack Kilo)
* Erasure Code support (beta)
@ -58,6 +187,7 @@ swift (2.3.0)
* Various other minor bug fixes and improvements.
swift (2.2.2)
* Data placement changes
@ -117,6 +247,7 @@ swift (2.2.2)
* Various other minor bug fixes and improvements.
swift (2.2.1)
* Swift now rejects object names with Unicode surrogates.
@ -164,7 +295,7 @@ swift (2.2.1)
* Various other minor bug fixes and improvements.
swift (2.2.0)
swift (2.2.0, OpenStack Juno)
* Added support for Keystone v3 auth.
@ -338,7 +469,7 @@ swift (2.0.0)
* Various other minor bug fixes and improvements
swift (1.13.1)
swift (1.13.1, OpenStack Icehouse)
* Change the behavior of CORS responses to better match the spec
@ -605,7 +736,7 @@ swift (1.11.0)
* Various other bug fixes and improvements
swift (1.10.0)
swift (1.10.0, OpenStack Havana)
* Added support for pooling memcache connections
@ -776,7 +907,7 @@ swift (1.9.0)
* Various other minor bug fixes and improvements
swift (1.8.0)
swift (1.8.0, OpenStack Grizzly)
* Make rings' replica count adjustable
@ -947,7 +1078,7 @@ swift (1.7.5)
* Various other minor bug fixes and improvements
swift (1.7.4)
swift (1.7.4, OpenStack Folsom)
* Fix issue where early client disconnects may have caused a memory leak
@ -962,14 +1093,14 @@ swift (1.7.0)
Serialize RingData in a versioned, custom format which is a combination
of a JSON-encoded header and .tostring() dumps of the
replica2part2dev_id arrays. This format deserializes hundreds of times
replica2part2dev_id arrays. This format deserializes hundreds of times
faster than rings serialized with Python 2.7's pickle (a significant
performance regression for ring loading between Python 2.6 and Python
2.7). Fixes bug 1031954.
2.7). Fixes bug 1031954.
The new implementation is backward-compatible; if a ring
does not begin with a new-style magic string, it is assumed to be an
old-style pickle-dumped ring and is handled as before. So new Swift
old-style pickle-dumped ring and is handled as before. So new Swift
code can read old rings, but old Swift code will not be able to read
newly-serialized rings.
@ -1153,7 +1284,7 @@ swift (1.5.0)
* Various other minor bug fixes and improvements
swift (1.4.8)
swift (1.4.8, OpenStack Essex)
* Added optional max_containers_per_account restriction
@ -1296,7 +1427,7 @@ swift (1.4.4)
* Query only specific zone via swift-recon.
swift (1.4.3)
swift (1.4.3, OpenStack Diablo)
* Additional quarantine catching code.
@ -1421,3 +1552,15 @@ swift (1.4.0)
* Stats uploaders now allow overrides for source_filename_pattern and
new_log_cutoff values.
---
Changelog entries for previous versions are incomplete
swift (1.3.0, OpenStack Cactus)
swift (1.2.0, OpenStack Bexar)
swift (1.1.0, OpenStack Austin)
swift (1.0.0, Initial Release)

+ 149
- 0
bandit.yaml View File

@ -0,0 +1,149 @@
# optional: after how many files to update progress
#show_progress_every: 100
# optional: plugins directory name
#plugins_dir: 'plugins'
# optional: plugins discovery name pattern
plugin_name_pattern: '*.py'
# optional: terminal escape sequences to display colors
#output_colors:
# DEFAULT: '\033[0m'
# HEADER: '\033[95m'
# LOW: '\033[94m'
# MEDIUM: '\033[93m'
# HIGH: '\033[91m'
# optional: log format string
#log_format: "[%(module)s]\t%(levelname)s\t%(message)s"
# globs of files which should be analyzed
include:
- '*.py'
# a list of strings, which if found in the path will cause files to be
# excluded
# for example /tests/ - to remove all all files in tests directory
#exclude_dirs:
# - '/tests/'
#configured for swift
profiles:
gate:
include:
- blacklist_calls
- blacklist_imports
- exec_used
- linux_commands_wildcard_injection
- request_with_no_cert_validation
- set_bad_file_permissions
- subprocess_popen_with_shell_equals_true
- ssl_with_bad_version
- password_config_option_not_marked_secret
# - any_other_function_with_shell_equals_true
# - ssl_with_bad_defaults
# - jinja2_autoescape_false
# - use_of_mako_templates
# - subprocess_without_shell_equals_true
# - any_other_function_with_shell_equals_true
# - start_process_with_a_shell
# - start_process_with_no_shell
# - hardcoded_sql_expressions
# - hardcoded_tmp_director
# - linux_commands_wildcard_injection
#For now some items are commented which could be included as per use later.
blacklist_calls:
bad_name_sets:
# - pickle:
# qualnames: [pickle.loads, pickle.load, pickle.Unpickler,
# cPickle.loads, cPickle.load, cPickle.Unpickler]
# level: LOW
# message: "Pickle library appears to be in use, possible security
#issue."
- marshal:
qualnames: [marshal.load, marshal.loads]
message: "Deserialization with the marshal module is possibly
dangerous."
# - md5:
# qualnames: [hashlib.md5]
# level: LOW
# message: "Use of insecure MD5 hash function."
- mktemp_q:
qualnames: [tempfile.mktemp]
message: "Use of insecure and deprecated function (mktemp)."
# - eval:
# qualnames: [eval]
# level: LOW
# message: "Use of possibly insecure function - consider using safer
#ast.literal_eval."
- mark_safe:
names: [mark_safe]
message: "Use of mark_safe() may expose cross-site scripting
vulnerabilities and should be reviewed."
- httpsconnection:
qualnames: [httplib.HTTPSConnection]
message: "Use of HTTPSConnection does not provide security, see
https://wiki.openstack.org/wiki/OSSN/OSSN-0033"
- yaml_load:
qualnames: [yaml.load]
message: "Use of unsafe yaml load. Allows instantiation of
arbitrary objects. Consider yaml.safe_load()."
- urllib_urlopen:
qualnames: [urllib.urlopen, urllib.urlretrieve, urllib.URLopener,
urllib.FancyURLopener, urllib2.urlopen, urllib2.Request]
message: "Audit url open for permitted schemes. Allowing use of
file:/ or custom schemes is often unexpected."
- paramiko_injection:
qualnames: [paramiko.exec_command, paramiko.invoke_shell]
message: "Paramiko exec_command() and invoke_shell() usage may
expose command injection vulnerabilities and should be reviewed."
shell_injection:
# Start a process using the subprocess module, or one of its wrappers.
subprocess: [subprocess.Popen, subprocess.call, subprocess.check_call,
subprocess.check_output, utils.execute,
utils.execute_with_timeout]
# Start a process with a function vulnerable to shell injection.
shell: [os.system, os.popen, os.popen2, os.popen3, os.popen4,
popen2.popen2, popen2.popen3, popen2.popen4, popen2.Popen3,
popen2.Popen4, commands.getoutput, commands.getstatusoutput]
# Start a process with a function that is not vulnerable to shell
# injection.
no_shell: [os.execl, os.execle, os.execlp, os.execlpe, os.execv,os.execve,
os.execvp, os.execvpe, os.spawnl, os.spawnle, os.spawnlp,
os.spawnlpe, os.spawnv, os.spawnve, os.spawnvp, os.spawnvpe,
os.startfile]
blacklist_imports:
bad_import_sets:
- telnet:
imports: [telnetlib]
level: HIGH
message: "Telnet is considered insecure. Use SSH or some other
encrypted protocol."
- info_libs:
imports: [Crypto]
level: LOW
message: "Consider possible security implications associated with
#{module} module."
hardcoded_password:
word_list: "wordlist/default-passwords"
ssl_with_bad_version:
bad_protocol_versions:
- 'PROTOCOL_SSLv2'
- 'SSLv2_METHOD'
- 'SSLv23_METHOD'
- 'PROTOCOL_SSLv3' # strict option
- 'PROTOCOL_TLSv1' # strict option
- 'SSLv3_METHOD' # strict option
- 'TLSv1_METHOD' # strict option
password_config_option_not_marked_secret:
function_names:
- oslo.config.cfg.StrOpt
- oslo_config.cfg.StrOpt

+ 1
- 1
bin/swift-account-info View File

@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/env python
# Licensed under the Apache License, Version 2.0 (the "License"); you may not
# use this file except in compliance with the License. You may obtain a copy
# of the License at


+ 1
- 1
bin/swift-container-info View File

@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/env python
# Licensed under the Apache License, Version 2.0 (the "License"); you may not
# use this file except in compliance with the License. You may obtain a copy
# of the License at


+ 1
- 1
bin/swift-container-sync View File

@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/env python
# Copyright (c) 2010-2012 OpenStack Foundation
#
# Licensed under the Apache License, Version 2.0 (the "License");


+ 13
- 3
bin/swift-dispersion-populate View File

@ -16,13 +16,13 @@
import traceback
from ConfigParser import ConfigParser
from cStringIO import StringIO
from optparse import OptionParser
from sys import exit, stdout
from time import time
from six.moves import range
from eventlet import GreenPool, patcher, sleep
from eventlet.pools import Pool
from six.moves import cStringIO as StringIO
try:
from swiftclient import get_auth
@ -76,8 +76,9 @@ def report(success):
return
next_report = time() + 5
eta, eta_unit = compute_eta(begun, created, need_to_create)
print '\r\x1B[KCreating %s: %d of %d, %d%s left, %d retries' % (item_type,
created, need_to_create, round(eta), eta_unit, retries_done),
print ('\r\x1B[KCreating %s: %d of %d, %d%s left, %d retries'
% (item_type, created, need_to_create, round(eta), eta_unit,
retries_done)),
stdout.flush()
@ -132,6 +133,9 @@ Usage: %%prog [options] [conf_file]
retries = int(conf.get('retries', 5))
concurrency = int(conf.get('concurrency', 25))
endpoint_type = str(conf.get('endpoint_type', 'publicURL'))
user_domain_name = str(conf.get('user_domain_name', ''))
project_domain_name = str(conf.get('project_domain_name', ''))
project_name = str(conf.get('project_name', ''))
insecure = options.insecure \
or config_true_value(conf.get('keystone_api_insecure', 'no'))
container_populate = config_true_value(
@ -146,6 +150,12 @@ Usage: %%prog [options] [conf_file]
retries_done = 0
os_options = {'endpoint_type': endpoint_type}
if user_domain_name:
os_options['user_domain_name'] = user_domain_name
if project_domain_name:
os_options['project_domain_name'] = project_domain_name
if project_name:
os_options['project_name'] = project_name
url, token = get_auth(conf['auth_url'], conf['auth_user'],
conf['auth_key'],


+ 16
- 5
bin/swift-dispersion-report View File

@ -26,6 +26,7 @@ except ImportError:
from eventlet import GreenPool, hubs, patcher, Timeout
from eventlet.pools import Pool
from eventlet.green import urllib2
from swift.common import direct_client
try:
@ -126,7 +127,7 @@ def container_dispersion_report(coropool, connpool, account, container_ring,
if not json_output:
print '\r\x1B[KQuerying containers: %d of %d, %d%s left, %d ' \
'retries' % (containers_queried[0], containers_listed,
round(eta), eta_unit, retries_done[0]),
round(eta), eta_unit, retries_done[0]),
stdout.flush()
container_parts = {}
for container in containers:
@ -145,7 +146,7 @@ def container_dispersion_report(coropool, connpool, account, container_ring,
if not json_output:
print '\r\x1B[KQueried %d containers for dispersion reporting, ' \
'%d%s, %d retries' % (containers_listed, round(elapsed),
elapsed_unit, retries_done[0])
elapsed_unit, retries_done[0])
if containers_listed - distinct_partitions:
print 'There were %d overlapping partitions' % (
containers_listed - distinct_partitions)
@ -176,9 +177,10 @@ def object_dispersion_report(coropool, connpool, account, object_ring,
try:
objects = [o['name'] for o in conn.get_container(
container, prefix='dispersion_', full_listing=True)[1]]
except ClientException as err:
if err.http_status != 404:
except urllib2.HTTPError as err:
if err.getcode() != 404:
raise
print >>stderr, 'No objects to query. Has ' \
'swift-dispersion-populate been run?'
stderr.flush()
@ -255,7 +257,7 @@ def object_dispersion_report(coropool, connpool, account, object_ring,
if not json_output:
print '\r\x1B[KQueried %d objects for dispersion reporting, ' \
'%d%s, %d retries' % (objects_listed, round(elapsed),
elapsed_unit, retries_done[0])
elapsed_unit, retries_done[0])
if objects_listed - distinct_partitions:
print 'There were %d overlapping partitions' % (
objects_listed - distinct_partitions)
@ -363,6 +365,9 @@ Usage: %%prog [options] [conf_file]
and not options.container_only
if not (object_report or container_report):
exit("Neither container or object report is set to run")
user_domain_name = str(conf.get('user_domain_name', ''))
project_domain_name = str(conf.get('project_domain_name', ''))
project_name = str(conf.get('project_name', ''))
insecure = options.insecure \
or config_true_value(conf.get('keystone_api_insecure', 'no'))
if options.debug:
@ -371,6 +376,12 @@ Usage: %%prog [options] [conf_file]
coropool = GreenPool(size=concurrency)
os_options = {'endpoint_type': endpoint_type}
if user_domain_name:
os_options['user_domain_name'] = user_domain_name
if project_domain_name:
os_options['project_domain_name'] = project_domain_name
if project_name:
os_options['project_name'] = project_name
url, token = get_auth(conf['auth_url'], conf['auth_user'],
conf['auth_key'],


+ 1
- 1
bin/swift-recon View File

@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/env python
# Copyright (c) 2014 Christian Schwede <christian.schwede@enovance.com>
#
# Licensed under the Apache License, Version 2.0 (the "License");


+ 1
- 1
bin/swift-ring-builder View File

@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/env python
# Copyright (c) 2014 Christian Schwede <christian.schwede@enovance.com>
#
# Licensed under the Apache License, Version 2.0 (the "License");


+ 1
- 1
bin/swift-ring-builder-analyzer View File

@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/env python
# Copyright (c) 2015 Samuel Merritt <sam@swiftstack.com>
#
# Licensed under the Apache License, Version 2.0 (the "License");


+ 0
- 6
doc/manpages/account-server.conf.5 View File

@ -188,12 +188,6 @@ Number of replication workers to spawn. The default is 8.
Time in seconds to wait between replication passes. The default is 30.
.IP \fBinterval\fR
Replaces run_pause with the more standard "interval", which means the replicator won't pause unless it takes less than the interval set. The default is 30.
.IP \fBerror_suppression_interval\fR
How long without an error before a node's error count is reset. This will also be how long before a node is re-enabled after suppression is triggered.
The default is 60 seconds.
.IP \fBerror_suppression_limit\fR
How many errors can accumulate before a node is temporarily ignored. The default
is 10 seconds.
.IP \fBnode_timeout\fR
Request timeout to external services. The default is 10 seconds.
.IP \fBconn_timeout\fR


+ 10
- 1
doc/manpages/dispersion.conf.5 View File

@ -43,7 +43,13 @@ Authentication system URL
.IP "\fBauth_user\fR"
Authentication system account/user name
.IP "\fBauth_key\fR"
Authentication system account/user password
Authentication system account/user password
.IP "\fBproject_name\fR"
Project name in case of keystone auth version 3
.IP "\fBproject_domain_name\fR"
Project domain name in case of keystone auth version 3
.IP "\fBuser_domain_name\fR"
User domain name in case of keystone auth version 3
.IP "\fBswift_dir\fR"
Location of openstack-swift configuration and ring files
.IP "\fBdispersion_coverage\fR"
@ -70,6 +76,9 @@ Whether to run the object report. The default is yes.
.IP "auth_key = dpstats"
.IP "swift_dir = /etc/swift"
.IP "# keystone_api_insecure = no"
.IP "# project_name = dpstats"
.IP "# project_domain_name = default"
.IP "# user_domain_name = default"
.IP "# dispersion_coverage = 1.0"
.IP "# retries = 5"
.IP "# concurrency = 25"


+ 2
- 0
doc/manpages/object-server.conf.5 View File

@ -129,6 +129,8 @@ Logging address. The default is /dev/log.
Request timeout to external services. The default is 3 seconds.
.IP \fBconn_timeout\fR
Connection timeout to external services. The default is 0.5 seconds.
.IP \fBcontainer_update_timeout\fR
Time to wait while sending a container update on object update. The default is 1 second.
.RE
.PD


+ 3
- 0
doc/manpages/swift-dispersion-populate.1 View File

@ -85,6 +85,9 @@ Example \fI/etc/swift/dispersion.conf\fR:
.IP "auth_user = dpstats:dpstats"
.IP "auth_key = dpstats"
.IP "swift_dir = /etc/swift"
.IP "# project_name = dpstats"
.IP "# project_domain_name = default"
.IP "# user_domain_name = default"
.IP "# dispersion_coverage = 1.0"
.IP "# retries = 5"
.IP "# concurrency = 25"


+ 3
- 0
doc/manpages/swift-dispersion-report.1 View File

@ -101,6 +101,9 @@ Example \fI/etc/swift/dispersion.conf\fR:
.IP "auth_user = dpstats:dpstats"
.IP "auth_key = dpstats"
.IP "swift_dir = /etc/swift"
.IP "# project_name = dpstats"
.IP "# project_domain_name = default"
.IP "# user_domain_name = default"
.IP "# dispersion_coverage = 1.0"
.IP "# retries = 5"
.IP "# concurrency = 25"


+ 4
- 2
doc/manpages/swift-recon.1 View File

@ -25,7 +25,7 @@
.SH SYNOPSIS
.LP
.B swift-recon
\ <server_type> [-v] [--suppress] [-a] [-r] [-u] [-d] [-l] [--md5] [--auditor] [--updater] [--expirer] [--sockstat]
\ <server_type> [-v] [--suppress] [-a] [-r] [-u] [-d] [-l] [-T] [--md5] [--auditor] [--updater] [--expirer] [--sockstat]
.SH DESCRIPTION
.PP
@ -80,8 +80,10 @@ Get md5sum of servers ring and compare to local copy
Get cluster socket usage stats
.IP "\fB--driveaudit\fR"
Get drive audit error stats
.IP "\fB-T, --time\fR"
Check time synchronization
.IP "\fB--all\fR"
Perform all checks. Equivalent to \-arudlq \-\-md5
Perform all checks. Equivalent to \-arudlqT \-\-md5
.IP "\fB--region=REGION\fR"
Only query servers in specified region
.IP "\fB-z ZONE, --zone=ZONE\fR"


+ 0
- 1
doc/saio/swift/container-server/1.conf View File

@ -9,7 +9,6 @@ user = <your-user-name>
log_facility = LOG_LOCAL2
recon_cache_path = /var/cache/swift
eventlet_debug = true
allow_versions = true
[pipeline:main]
pipeline = recon container-server


+ 0
- 1
doc/saio/swift/container-server/2.conf View File

@ -9,7 +9,6 @@ user = <your-user-name>
log_facility = LOG_LOCAL3
recon_cache_path = /var/cache/swift2
eventlet_debug = true
allow_versions = true
[pipeline:main]
pipeline = recon container-server


+ 0
- 1
doc/saio/swift/container-server/3.conf View File

@ -9,7 +9,6 @@ user = <your-user-name>
log_facility = LOG_LOCAL4
recon_cache_path = /var/cache/swift3
eventlet_debug = true
allow_versions = true
[pipeline:main]
pipeline = recon container-server


+ 0
- 1
doc/saio/swift/container-server/4.conf View File

@ -9,7 +9,6 @@ user = <your-user-name>
log_facility = LOG_LOCAL5
recon_cache_path = /var/cache/swift4
eventlet_debug = true
allow_versions = true
[pipeline:main]
pipeline = recon container-server


+ 5
- 0
doc/saio/swift/container-sync-realms.conf View File

@ -0,0 +1,5 @@
[saio]
key = changeme
key2 = changeme
cluster_saio_endpoint = http://127.0.0.1:8080/v1/

+ 1
- 1
doc/saio/swift/object-expirer.conf View File

@ -37,7 +37,7 @@ interval = 300
# config value
# processes = 0
# process is which of the parts a particular process will work on
# process can also be specified on the command line and will overide the config
# process can also be specified on the command line and will override the config
# value
# process is "zero based", if you want to use 3 processes, you should run
# processes with process set to 0, 1, and 2


+ 9
- 1
doc/saio/swift/proxy-server.conf View File

@ -9,7 +9,7 @@ eventlet_debug = true
[pipeline:main]
# Yes, proxy-logging appears twice. This is so that
# middleware-originated requests get logged too.
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache bulk tempurl ratelimit crossdomain tempauth staticweb container-quotas account-quotas slo dlo proxy-logging proxy-server
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache bulk tempurl ratelimit crossdomain container_sync tempauth staticweb container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server
[filter:catch_errors]
use = egg:swift#catch_errors
@ -35,6 +35,10 @@ use = egg:swift#dlo
[filter:slo]
use = egg:swift#slo
[filter:container_sync]
use = egg:swift#container_sync
current = //saio/saio_endpoint
[filter:tempurl]
use = egg:swift#tempurl
@ -60,6 +64,10 @@ use = egg:swift#memcache
[filter:gatekeeper]
use = egg:swift#gatekeeper
[filter:versioned_writes]
use = egg:swift#versioned_writes
allow_versioned_writes = true
[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true


+ 14
- 4
doc/source/admin_guide.rst View File

@ -154,6 +154,10 @@ until it has been resolved. If the drive is going to be replaced immediately,
then it is just best to replace the drive, format it, remount it, and let
replication fill it up.
After the drive is unmounted, make sure the mount point is owned by root
(root:root 755). This ensures that rsync will not try to replicate into the
root drive once the failed drive is unmounted.
If the drive can't be replaced immediately, then it is best to leave it
unmounted, and set the device weight to 0. This will allow all the
replicas that were on that drive to be replicated elsewhere until the drive
@ -270,7 +274,8 @@ configuration file, /etc/swift/dispersion.conf. Example conf file::
There are also options for the conf file for specifying the dispersion coverage
(defaults to 1%), retries, concurrency, etc. though usually the defaults are
fine.
fine. If you want to use keystone v3 for authentication there are options like
auth_version, user_domain_name, project_domain_name and project_name.
Once the configuration is in place, run `swift-dispersion-populate` to populate
the containers and objects throughout the cluster.
@ -544,18 +549,22 @@ Request URI Description
/recon/sockstat returns consumable info from /proc/net/sockstat|6
/recon/devices returns list of devices and devices dir i.e. /srv/node
/recon/async returns count of async pending
/recon/replication returns object replication times (for backward compatibility)
/recon/replication returns object replication info (for backward compatibility)
/recon/replication/<type> returns replication info for given type (account, container, object)
/recon/auditor/<type> returns auditor stats on last reported scan for given type (account, container, object)
/recon/updater/<type> returns last updater sweep times for given type (container, object)
========================= ========================================================================================
Note that 'object_replication_last' and 'object_replication_time' in object
replication info are considered to be transitional and will be removed in
the subsequent releases. Use 'replication_last' and 'replication_time' instead.
This information can also be queried via the swift-recon command line utility::
fhines@ubuntu:~$ swift-recon -h
Usage:
usage: swift-recon <server_type> [-v] [--suppress] [-a] [-r] [-u] [-d]
[-l] [--md5] [--auditor] [--updater] [--expirer] [--sockstat]
[-l] [-T] [--md5] [--auditor] [--updater] [--expirer] [--sockstat]
<server_type> account|container|object
Defaults to object server.
@ -578,7 +587,8 @@ This information can also be queried via the swift-recon command line utility::
-q, --quarantined Get cluster quarantine stats
--md5 Get md5sum of servers ring and compare to local copy
--sockstat Get cluster socket usage stats
--all Perform all checks. Equal to -arudlq --md5 --sockstat
-T, --time Check time synchronization
--all Perform all checks. Equal to -arudlqT --md5 --sockstat
-z ZONE, --zone=ZONE Only query servers in specified zone
-t SECONDS, --timeout=SECONDS
Time to wait for a response from a server


+ 1
- 1
doc/source/api/object_api_v1_overview.rst View File

@ -7,7 +7,7 @@ metadata by using the Object Storage API, which is implemented as a set
of Representational State Transfer (REST) web services.
For an introduction to OpenStack Object Storage, see `Object
Storage <http://docs.openstack.org/admin-guide-cloud/content/ch_admin-openstack-object-storage.html>`__
Storage <http://docs.openstack.org/admin-guide-cloud/objectstorage.html>`
in the *OpenStack Cloud Administrator Guide*.
You use the HTTPS (SSL) protocol to interact with Object Storage, and


+ 1
- 1
doc/source/cors.rst View File

@ -2,7 +2,7 @@
CORS
====
CORS_ is a mechanisim to allow code running in a browser (Javascript for
CORS_ is a mechanism to allow code running in a browser (Javascript for
example) make requests to a domain other then the one from where it originated.
Swift supports CORS requests to containers and objects.


+ 92
- 71
doc/source/deployment_guide.rst View File

@ -340,7 +340,7 @@ paste.deploy works (at least at the time of this writing.)
`name3` got the local value from the `app:myapp` subsection because it is using
the special paste.deploy syntax of ``set option_name = value``. So, if you want
a default value for most app/filters but want to overridde it in one
a default value for most app/filters but want to override it in one
subsection, this is how you do it.
`name4` got the global value from `DEFAULT` since it's only in that section
@ -390,6 +390,13 @@ max_header_size 8192 max_header_size is the max number of bytes in
See also include_service_catalog in
proxy-server.conf-sample (documented in
overview_auth.rst).
extra_header_count 0 By default the maximum number of allowed
headers depends on the number of max
allowed metadata settings plus a default
value of 32 for regular http headers.
If for some reason this is not enough (custom
middleware for example) it can be increased
with the extra_header_count constraint.
=================== ========== =============================================
---------------------------
@ -405,76 +412,86 @@ The following configuration options are available:
[DEFAULT]
=================== ========== =============================================
Option Default Description
------------------- ---------- ---------------------------------------------
swift_dir /etc/swift Swift configuration directory
devices /srv/node Parent directory of where devices are mounted
mount_check true Whether or not check if the devices are
mounted to prevent accidentally writing
to the root device
bind_ip 0.0.0.0 IP Address for server to bind to
bind_port 6000 Port for server to bind to
bind_timeout 30 Seconds to attempt bind before giving up
workers auto Override the number of pre-forked workers
that will accept connections. If set it
should be an integer, zero means no fork. If
unset, it will try to default to the number
of effective cpu cores and fallback to one.
Increasing the number of workers helps slow
filesystem operations in one request from
negatively impacting other requests, but only
the :ref:`servers_per_port
<server-per-port-configuration>`
option provides complete I/O isolation with
no measurable overhead.
servers_per_port 0 If each disk in each storage policy ring has
unique port numbers for its "ip" value, you
can use this setting to have each
object-server worker only service requests
for the single disk matching the port in the
ring. The value of this setting determines
how many worker processes run for each port
(disk) in the ring. If you have 24 disks
per server, and this setting is 4, then
each storage node will have 1 + (24 * 4) =
97 total object-server processes running.
This gives complete I/O isolation, drastically
reducing the impact of slow disks on storage
node performance. The object-replicator and
object-reconstructor need to see this setting
too, so it must be in the [DEFAULT] section.
See :ref:`server-per-port-configuration`.
max_clients 1024 Maximum number of clients one worker can
process simultaneously (it will actually
accept(2) N + 1). Setting this to one (1)
will only handle one request at a time,
without accepting another request
concurrently.
disable_fallocate false Disable "fast fail" fallocate checks if the
underlying filesystem does not support it.
log_max_line_length 0 Caps the length of log lines to the
value given; no limit if set to 0, the
default.
log_custom_handlers None Comma-separated list of functions to call
to setup custom log handlers.
eventlet_debug false If true, turn on debug logging for eventlet
fallocate_reserve 0 You can set fallocate_reserve to the number of
bytes you'd like fallocate to reserve, whether
there is space for the given file size or not.
This is useful for systems that behave badly
when they completely run out of space; you can
make the services pretend they're out of space
early.
conn_timeout 0.5 Time to wait while attempting to connect to
another backend node.
node_timeout 3 Time to wait while sending each chunk of data
to another backend node.
client_timeout 60 Time to wait while receiving each chunk of
data from a client or another backend node.
network_chunk_size 65536 Size of chunks to read/write over the network
disk_chunk_size 65536 Size of chunks to read/write to disk
=================== ========== =============================================
======================== ========== ==========================================
Option Default Description
------------------------ ---------- ------------------------------------------
swift_dir /etc/swift Swift configuration directory
devices /srv/node Parent directory of where devices are
mounted
mount_check true Whether or not check if the devices are
mounted to prevent accidentally writing
to the root device
bind_ip 0.0.0.0 IP Address for server to bind to
bind_port 6000 Port for server to bind to
bind_timeout 30 Seconds to attempt bind before giving up
workers auto Override the number of pre-forked workers
that will accept connections. If set it
should be an integer, zero means no fork.
If unset, it will try to default to the
number of effective cpu cores and fallback
to one. Increasing the number of workers
helps slow filesystem operations in one
request from negatively impacting other
requests, but only the
:ref:`servers_per_port
<server-per-port-configuration>` option
provides complete I/O isolation with no
measurable overhead.
servers_per_port 0 If each disk in each storage policy ring
has unique port numbers for its "ip"
value, you can use this setting to have
each object-server worker only service
requests for the single disk matching the
port in the ring. The value of this
setting determines how many worker
processes run for each port (disk) in the
ring. If you have 24 disks per server, and
this setting is 4, then each storage node
will have 1 + (24 * 4) = 97 total
object-server processes running. This
gives complete I/O isolation, drastically
reducing the impact of slow disks on
storage node performance. The
object-replicator and object-reconstructor
need to see this setting too, so it must
be in the [DEFAULT] section.
See :ref:`server-per-port-configuration`.
max_clients 1024 Maximum number of clients one worker can
process simultaneously (it will actually
accept(2) N + 1). Setting this to one (1)
will only handle one request at a time,
without accepting another request
concurrently.
disable_fallocate false Disable "fast fail" fallocate checks if
the underlying filesystem does not support
it.
log_max_line_length 0 Caps the length of log lines to the
value given; no limit if set to 0, the
default.
log_custom_handlers None Comma-separated list of functions to call
to setup custom log handlers.
eventlet_debug false If true, turn on debug logging for
eventlet
fallocate_reserve 0 You can set fallocate_reserve to the
number of bytes you'd like fallocate to
reserve, whether there is space for the
given file size or not. This is useful for
systems that behave badly when they
completely run out of space; you can
make the services pretend they're out of
space early.
conn_timeout 0.5 Time to wait while attempting to connect
to another backend node.
node_timeout 3 Time to wait while sending each chunk of
data to another backend node.
client_timeout 60 Time to wait while receiving each chunk of
data from a client or another backend node
network_chunk_size 65536 Size of chunks to read/write over the
network
disk_chunk_size 65536 Size of chunks to read/write to disk
container_update_timeout 1 Time to wait while sending a container
update on object update.
======================== ========== ==========================================
.. _object-server-options:
@ -1229,6 +1246,10 @@ For a standard swift install, all data drives are mounted directly under
be sure to set the `devices` config option in all of the server configs to
point to the correct directory.
The mount points for each drive in /srv/node/ should be owned by the root user
almost exclusively (root:root 755). This is required to prevent rsync from
syncing files into the root drive in the event a drive is unmounted.
Swift uses system calls to reserve space for new objects being written into
the system. If your filesystem does not support `fallocate()` or
`posix_fallocate()`, be sure to set the `disable_fallocate = true` config


+ 21
- 8
doc/source/development_guidelines.rst View File

@ -42,7 +42,7 @@ To execute the unit tests:
Remarks:
If you installed using: `cd ~/swift; sudo python setup.py develop`,
you may need to do: `cd ~/swift; sudo chown -R swift:swift swift.egg-info`
you may need to do: `cd ~/swift; sudo chown -R ${USER}:${USER} swift.egg-info`
prior to running tox.
* Optionally, run only specific tox builds:
@ -71,6 +71,18 @@ The endpoint and authorization credentials to be used by functional tests
should be configured in the ``test.conf`` file as described in the section
:ref:`setup_scripts`.
The environment variable ``SWIFT_TEST_POLICY`` may be set to specify a
particular storage policy *name* that will be used for testing. When set, tests
that would otherwise not specify a policy or choose a random policy from
those available will instead use the policy specified. Tests that use more than
one policy will include the specified policy in the set of policies used. The
specified policy must be available on the cluster under test.
For example, this command would run the functional tests using policy
'silver'::
SWIFT_TEST_POLICY=silver tox -e func
If the ``test.conf`` file is not found then the functional test framework will
instantiate a set of Swift servers in the same process that executes the
functional tests. This 'in-process test' mode may also be enabled (or disabled)
@ -95,13 +107,14 @@ found in ``<custom_conf_source_dir>``, the search will then look in the
the corresponding sample config file from ``etc/`` is used (e.g.
``proxy-server.conf-sample`` or ``swift.conf-sample``).
The environment variable ``SWIFT_TEST_POLICY`` may be set to specify
a particular storage policy *name* that will be used for testing. When set,
this policy must exist in the ``swift.conf`` file and its corresponding ring
file must exist in ``<custom_conf_source_dir>`` (if specified) or ``etc/``. The
test setup will set the specified policy to be the default and use its ring
file properties for constructing the test object ring. This allows in-process
testing to be run against various policy types and ring files.
When using the 'in-process test' mode ``SWIFT_TEST_POLICY`` may be set to
specify a particular storage policy *name* that will be used for testing as
described above. When set, this policy must exist in the ``swift.conf`` file
and its corresponding ring file must exist in ``<custom_conf_source_dir>`` (if
specified) or ``etc/``. The test setup will set the specified policy to be the
default and use its ring file properties for constructing the test object ring.
This allows in-process testing to be run against various policy types and ring
files.
For example, this command would run the in-process mode functional tests
using config files found in ``$HOME/my_tests`` and policy 'silver'::


+ 1
- 1
doc/source/development_ondisk_backends.rst View File

@ -4,7 +4,7 @@ Pluggable On-Disk Back-end APIs
The internal REST API used between the proxy server and the account, container
and object server is almost identical to public Swift REST API, but with a few
internal extentsions (for example, update an account with a new container).
internal extensions (for example, update an account with a new container).
The pluggable back-end APIs for the three REST API servers (account,
container, object) abstracts the needs for servicing the various REST APIs


+ 24
- 1
doc/source/development_saio.rst View File

@ -95,6 +95,16 @@ another device when creating the VM, and follow these instructions:
# **Make sure to include the trailing slash after /srv/$x/**
for x in {1..4}; do sudo chown -R ${USER}:${USER} /srv/$x/; done
Note: We create the mount points and mount the storage disk under
/mnt/sdb1. This disk will contain one directory per simulated swift node,
each owned by the current swift user.
We then create symlinks to these directories under /srv.
If the disk sdb is unmounted, files will not be written under
/srv/\*, because the symbolic link destination /mnt/sdb1/* will not
exist. This prevents disk sync operations from writing to the root
partition in the event a drive is unmounted.
#. Next, skip to :ref:`common-dev-section`.
@ -135,6 +145,15 @@ these instructions:
# **Make sure to include the trailing slash after /srv/$x/**
for x in {1..4}; do sudo chown -R ${USER}:${USER} /srv/$x/; done
Note: We create the mount points and mount the loopback file under
/mnt/sdb1. This file will contain one directory per simulated swift node,
each owned by the current swift user.
We then create symlinks to these directories under /srv.
If the loopback file is unmounted, files will not be written under
/srv/\*, because the symbolic link destination /mnt/sdb1/* will not
exist. This prevents disk sync operations from writing to the root
partition in the event a drive is unmounted.
.. _common-dev-section:
@ -184,7 +203,7 @@ Getting the code
#. Install swift's test dependencies::
sudo pip install -r swift/test-requirements.txt
cd $HOME/swift; sudo pip install -r test-requirements.txt
----------------
Setting up rsync
@ -352,6 +371,10 @@ commands are as follows:
.. literalinclude:: /../saio/swift/container-reconciler.conf
#. ``/etc/swift/container-sync-realms.conf``
.. literalinclude:: /../saio/swift/container-sync-realms.conf
#. ``/etc/swift/account-server/1.conf``
.. literalinclude:: /../saio/swift/account-server/1.conf


+ 204
- 0
doc/source/first_contribution_swift.rst View File

@ -0,0 +1,204 @@
===========================
First Contribution to Swift
===========================
-------------
Getting Swift
-------------
Swift's source code is hosted on github and managed with git. The current
trunk can be checked out like this:
``git clone https://github.com/openstack/swift.git``
This will clone the Swift repository under your account.
A source tarball for the latest release of Swift is available on the
`launchpad project page <https://launchpad.net/swift>`_.
Prebuilt packages for Ubuntu and RHEL variants are available.
* `Swift Ubuntu Packages <https://launchpad.net/ubuntu/+source/swift>`_
* `Swift RDO Packages <https://www.rdoproject.org/Repositories>`_
--------------------
Source Control Setup
--------------------
Swift uses `git` for source control. The OpenStack
`Developer's Guide <http://docs.openstack.org/infra/manual/developers.html>`_
describes the steps for setting up Git and all the necessary accounts for
contributing code to Swift.
----------------
Changes to Swift
----------------
Once you have the source code and source control set up, you can make your
changes to Swift.
-------
Testing
-------
The `Development Guidelines <development_guidelines>`_ describes the testing
requirements before submitting Swift code.
In summary, you can execute tox from the swift home directory (where you
checked out the source code):
``tox``
Tox will present tests results. Notice that in the beginning, it is very common
to break many coding style guidelines.
--------------------------
Proposing changes to Swift
--------------------------
The OpenStack
`Developer's Guide <http://docs.openstack.org/infra/manual/developers.html>`_
describes the most common `git` commands that you will need.
Following is a list of the commands that you need to know for your first
contribution to Swift:
To clone a copy of Swift:
``git clone https://github.com/openstack/swift.git``
Under the swift directory, set up the Gerrit repository. The following command
configures the repository to know about Gerrit and makes the Change-Id commit
hook get installed. You only need to do this once:
``git review -s``
To create your development branch (substitute branch_name for a name of your
choice:
``git checkout -b <branch_name>``
To check the files that have been updated in your branch:
``git status``
To check the differences between your branch and the repository:
``git diff``
Assuming you have not added new files, you commit all your changes using:
``git commit -a``
Read the `Summary of Git commit message structure <https://wiki.openstack.org/wiki/GitCommitMessages?%22Summary%20of%20Git%20commit%20message%20structure%22#Summary_of_Git_commit_message_structure>`_
for best practices on writing the commit message. When you are ready to send
your changes for review use:
``git review``
If successful, Git response message will contain a URL you can use to track your
changes.
If you need to make further changes to the same review, you can commit them
using:
``git commit -a --amend``
This will commit the changes under the same set of changes you issued earlier.
Notice that in order to send your latest version for review, you will still
need to call:
``git review``
---------------------
Tracking your changes
---------------------
After you proposed your changes to Swift, you can track the review in:
* `<https://review.openstack.org>`_
.. _post-rebase-instructions:
------------------------
Post rebase instructions
------------------------
After rebasing, the following steps should be performed to rebuild the swift
installation. Note that these commands should be performed from the root of the
swift repo directory (e.g. $HOME/swift/):
``sudo python setup.py develop``
``sudo pip install -r test-requirements.txt``
If using TOX, depending on the changes made during the rebase, you may need to
rebuild the TOX environment (generally this will be the case if
test-requirements.txt was updated such that a new version of a package is
required), this can be accomplished using the '-r' argument to the TOX cli:
``tox -r``
You can include any of the other TOX arguments as well, for example, to run the
pep8 suite and rebuild the TOX environment the following can be used:
``tox -r -e pep8``
The rebuild option only needs to be specified once for a particular build (e.g.
pep8), that is further invocations of the same build will not require this
until the next rebase.
---------------
Troubleshooting
---------------
You may run into the following errors when starting Swift if you rebase
your commit using:
``git rebase``
.. code-block:: python
Traceback (most recent call last):
File "/usr/local/bin/swift-init", line 5, in <module>
from pkg_resources import require
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2749, in <module>
working_set = WorkingSet._build_master()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 446, in _build_master
return cls._build_from_requirements(__requires__)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 459, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 628, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: swift==2.3.1.devXXX
(where XXX represents a dev version of Swift).
.. code-block:: python
Traceback (most recent call last):
File "/usr/local/bin/swift-proxy-server", line 10, in <module>
execfile(__file__)
File "/home/swift/swift/bin/swift-proxy-server", line 23, in <module>
sys.exit(run_wsgi(conf_file, 'proxy-server', **options))
File "/home/swift/swift/swift/common/wsgi.py", line 888, in run_wsgi
loadapp(conf_path, global_conf=global_conf)
File "/home/swift/swift/swift/common/wsgi.py", line 390, in loadapp
func(PipelineWrapper(ctx))
File "/home/swift/swift/swift/proxy/server.py", line 602, in modify_wsgi_pipeline
ctx = pipe.create_filter(filter_name)
File "/home/swift/swift/swift/common/wsgi.py", line 329, in create_filter
global_conf=self.context.global_conf)
File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
global_conf=global_conf)
File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 328, in _loadegg
return loader.get_context(object_type, name, global_conf)
File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 620, in get_context
object_type, name=name)
File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 659, in find_egg_entry_point
for prot in protocol_options] or '(no entry points)'))))
LookupError: Entry point 'versioned_writes' not found in egg 'swift' (dir: /home/swift/swift; protocols: paste.filter_factory, paste.filter_app_factory; entry_points: )
This happens because `git rebase` will retrieve code for a different version of
Swift in the development stream, but the start scripts under `/usr/local/bin` have
not been updated. The solution is to follow the steps described in the
:ref:`post-rebase-instructions` section.

+ 3
- 20
doc/source/getting_started.rst View File

@ -18,23 +18,6 @@ Swift is written in Python and has these dependencies:
There is no current support for Python 3.
-------------
Getting Swift
-------------
Swift's source code is hosted on github and managed with git. The current
trunk can be checked out like this:
``git clone https://github.com/openstack/swift.git``
A source tarball for the latest release of Swift is available on the
`launchpad project page <https://launchpad.net/swift>`_.
Prebuilt packages for Ubuntu and RHEL variants are available.
* `Swift Ubuntu Packages <https://launchpad.net/ubuntu/+source/swift>`_
* `Swift RDO Packages <https://openstack.redhat.com/Repositories>`_
-----------
Development
-----------
@ -42,10 +25,10 @@ Development
To get started with development with Swift, or to just play around, the
following docs will be useful:
* :doc:`Swift All in One <development_saio>` - Set up a VM with Swift
installed
* :doc:`Swift All in One <development_saio>` - Set up a VM with Swift installed
* :doc:`Development Guidelines <development_guidelines>`
* `Associated Projects <http://docs.openstack.org/developer/swift/associated_projects.html>`
* :doc:`First Contribution to Swift <first_contribution_swift>`
* :doc:`Associated Projects <associated_projects>`
--------------------------
CLI client and SDK library


+ 7
- 0
doc/source/howto_installmultinode.rst View File

@ -6,6 +6,13 @@ Please refer to the latest official
`Openstack Installation Guides <http://docs.openstack.org/#install-guides>`_
for the most up-to-date documentation.
Object Storage installation guide for Openstack Kilo
----------------------------------------------------
* `openSUSE 13.2 and SUSE Linux Enterprise Server 12 <http://docs.openstack.org/kilo/install-guide/install/zypper/content/ch_swift.html>`_
* `RHEL 7, CentOS 7, and Fedora 21 <http://docs.openstack.org/kilo/install-guide/install/yum/content/ch_swift.html>`_
* `Ubuntu 14.04 <http://docs.openstack.org/kilo/install-guide/install/apt/content/ch_swift.html>`_
Object Storage installation guide for Openstack Juno
----------------------------------------------------


+ 1
- 0
doc/source/index.rst View File

@ -68,6 +68,7 @@ Developer Documentation
development_guidelines
development_saio
first_contribution_swift
policies_saio
development_auth
development_middleware


+ 2
- 1
doc/source/logs.rst View File

@ -59,7 +59,7 @@ client_etag The etag header value given by the client.
transaction_id The transaction id of the request.
headers The headers given in the request.
request_time The duration of the request.
source The "source" of the reuqest. This may be set for requests
source The "source" of the request. This may be set for requests
that are generated in order to fulfill client requests,
e.g. bulk uploads.
log_info Various info that may be useful for diagnostics, e.g. the
@ -102,6 +102,7 @@ DLO :ref:`dynamic-large-objects`
LE :ref:`list_endpoints`
KS :ref:`keystoneauth`
RL :ref:`ratelimit`
VW :ref:`versioned_writes`
======================= =============================


+ 9
- 0
doc/source/middleware.rst View File

@ -155,6 +155,15 @@ Name Check (Forbidden Character Filter)
:members:
:show-inheritance:
.. _versioned_writes: