Browse Source

Merge master into feature/hummingbird

Change-Id: I449596b4f167a88aa7ed999a6a657c762e9a4597
changes/48/290148/1
Michael Barton 6 years ago
parent
commit
0f7f1de233
  1. 6
      .mailmap
  2. 18
      .manpages
  3. 33
      AUTHORS
  4. 89
      CHANGELOG
  5. 4
      CONTRIBUTING.md
  6. 5
      bin/swift-dispersion-report
  7. 4
      bin/swift-drive-audit
  8. 5
      bin/swift-init
  9. 2
      bin/swift-oldies
  10. 2
      bin/swift-orphans
  11. 6
      doc/manpages/account-server.conf.5
  12. 6
      doc/manpages/container-server.conf.5
  13. 6
      doc/manpages/object-expirer.conf.5
  14. 10
      doc/manpages/object-server.conf.5
  15. 25
      doc/manpages/proxy-server.conf.5
  16. 1
      doc/manpages/swift-init.1
  17. 2
      doc/saio/swift/container-reconciler.conf
  18. 2
      doc/saio/swift/object-expirer.conf
  19. 3
      doc/saio/swift/swift.conf
  20. 16
      doc/source/admin_guide.rst
  21. 12
      doc/source/api/form_post_middleware.rst
  22. 127
      doc/source/api/large_objects.rst
  23. 16
      doc/source/api/object_api_v1_overview.rst
  24. 2
      doc/source/api/use_content-encoding_metadata.rst
  25. 5
      doc/source/associated_projects.rst
  26. 106
      doc/source/conf.py
  27. 38
      doc/source/deployment_guide.rst
  28. 36
      doc/source/development_guidelines.rst
  29. 10
      doc/source/development_saio.rst
  30. 12
      doc/source/howto_installmultinode.rst
  31. 0
      doc/source/images/ec_overview.png
  32. 13
      doc/source/index.rst
  33. 1031
      doc/source/ops_runbook/diagnose.rst
  34. 36
      doc/source/ops_runbook/general.rst
  35. 79
      doc/source/ops_runbook/index.rst
  36. 322
      doc/source/ops_runbook/maintenance.rst
  37. 367
      doc/source/ops_runbook/procedures.rst
  38. 177
      doc/source/ops_runbook/sec-furtherdiagnose.rst
  39. 264
      doc/source/ops_runbook/troubleshooting.rst
  40. 10
      doc/source/overview_auth.rst
  41. 10
      doc/source/overview_backing_store.rst
  42. 73
      doc/source/overview_container_sync.rst
  43. 12
      doc/source/overview_erasure_code.rst
  44. 4
      doc/source/overview_large_objects.rst
  45. 4
      doc/source/overview_policies.rst
  46. 271
      doc/source/overview_ring.rst
  47. 0
      doc/source/policies_saio.rst
  48. 2
      etc/account-server.conf-sample
  49. 2
      etc/container-reconciler.conf-sample
  50. 2
      etc/container-server.conf-sample
  51. 2
      etc/internal-client.conf-sample
  52. 1
      etc/memcache.conf-sample
  53. 4
      etc/object-expirer.conf-sample
  54. 10
      etc/object-server.conf-sample
  55. 37
      etc/proxy-server.conf-sample
  56. 9
      etc/swift.conf-sample
  57. 4
      requirements.txt
  58. 70
      swift/cli/recon.py
  59. 1
      swift/cli/ring_builder_analyzer.py
  60. 70
      swift/cli/ringbuilder.py
  61. 142
      swift/common/direct_client.py
  62. 14
      swift/common/internal_client.py
  63. 30
      swift/common/manager.py
  64. 25
      swift/common/memcached.py
  65. 341
      swift/common/middleware/bulk.py
  66. 65
      swift/common/middleware/slo.py
  67. 25
      swift/common/middleware/staticweb.py
  68. 11
      swift/common/middleware/tempauth.py
  69. 9
      swift/common/middleware/versioned_writes.py
  70. 33
      swift/common/middleware/xprofile.py
  71. 56
      swift/common/request_helpers.py
  72. 32
      swift/common/ring/builder.py
  73. 23
      swift/common/ring/ring.py
  74. 4
      swift/common/storage_policy.py
  75. 42
      swift/common/swob.py
  76. 370
      swift/common/utils.py
  77. 157
      swift/container/backend.py
  78. 18
      swift/container/reconciler.py
  79. 18
      swift/container/replicator.py
  80. 19
      swift/container/server.py
  81. 72
      swift/container/sync.py
  82. 177
      swift/container/sync_store.py
  83. 4
      swift/container/updater.py
  84. 63
      swift/locale/de/LC_MESSAGES/swift.po
  85. 29
      swift/locale/es/LC_MESSAGES/swift.po
  86. 29
      swift/locale/fr/LC_MESSAGES/swift.po
  87. 29
      swift/locale/it/LC_MESSAGES/swift.po
  88. 29
      swift/locale/ja/LC_MESSAGES/swift.po
  89. 36
      swift/locale/ko_KR/LC_MESSAGES/swift.po
  90. 31
      swift/locale/pt_BR/LC_MESSAGES/swift.po
  91. 140
      swift/locale/ru/LC_MESSAGES/swift.po
  92. 455
      swift/locale/swift.pot
  93. 31
      swift/locale/tr_TR/LC_MESSAGES/swift.po
  94. 30
      swift/locale/zh_CN/LC_MESSAGES/swift.po
  95. 31
      swift/locale/zh_TW/LC_MESSAGES/swift.po
  96. 30
      swift/obj/auditor.py
  97. 528
      swift/obj/diskfile.py
  98. 77
      swift/obj/mem_diskfile.py
  99. 18
      swift/obj/replicator.py
  100. 119
      swift/obj/server.py

6
.mailmap

@ -87,3 +87,9 @@ Donagh McCabe <donagh.mccabe@hpe.com> <donagh.mccabe@hp.com>
Eamonn O'Toole <eamonn.otoole@hpe.com> <eamonn.otoole@hp.com>
Gerry Drudy <gerry.drudy@hpe.com> <gerry.drudy@hp.com>
Mark Seger <mark.seger@hpe.com> <mark.seger@hp.com>
Timur Alperovich <timur.alperovich@gmail.com> <timuralp@swiftstack.com>
Mehdi Abaakouk <sileht@redhat.com> <mehdi.abaakouk@enovance.com>
Richard Hawkins <richard.hawkins@rackspace.com> <hurricanerix@gmail.com>
Ondrej Novy <ondrej.novy@firma.seznam.cz>
Peter Lisak <peter.lisak@firma.seznam.cz>
Ke Liang <ke.liang@easystack.cn>

18
.manpages

@ -0,0 +1,18 @@
#!/bin/sh
RET=0
for MAN in doc/manpages/* ; do
OUTPUT=$(LC_ALL=en_US.UTF-8 MANROFFSEQ='' MANWIDTH=80 man --warnings -E UTF-8 -l \
-Tutf8 -Z "$MAN" 2>&1 >/dev/null)
if [ -n "$OUTPUT" ] ; then
RET=1
echo "$MAN:"
echo "$OUTPUT"
fi
done
if [ "$RET" -eq "0" ] ; then
echo "All manpages are fine"
fi
exit "$RET"

33
AUTHORS

@ -18,6 +18,7 @@ CORE Emeritus
Chmouel Boudjnah (chmouel@enovance.com)
Florian Hines (syn@ronin.io)
Greg Holt (gholt@rackspace.com)
Paul Luse (paul.e.luse@intel.com)
Jay Payne (letterj@gmail.com)
Peter Portante (peter.portante@redhat.com)
Will Reese (wreese@gmail.com)
@ -25,7 +26,7 @@ Chuck Thier (cthier@gmail.com)
Contributors
------------
Mehdi Abaakouk (mehdi.abaakouk@enovance.com)
Mehdi Abaakouk (sileht@redhat.com)
Timur Alperovich (timur.alperovich@gmail.com)
Jesse Andrews (anotherjesse@gmail.com)
Joe Arnold (joe@swiftstack.com)
@ -41,7 +42,7 @@ James E. Blair (jeblair@openstack.org)
Fabien Boucher (fabien.boucher@enovance.com)
Clark Boylan (clark.boylan@gmail.com)
Pádraig Brady (pbrady@redhat.com)
Lorcan Browne (lorcan.browne@hp.com)
Lorcan Browne (lorcan.browne@hpe.com)
Russell Bryant (rbryant@redhat.com)
Jay S. Bryant (jsbryant@us.ibm.com)
Tim Burke (tim.burke@gmail.com)
@ -56,15 +57,17 @@ François Charlier (francois.charlier@enovance.com)
Ray Chen (oldsharp@163.com)
Harshit Chitalia (harshit@acelio.com)
Brian Cline (bcline@softlayer.com)
Alistair Coles (alistair.coles@hp.com)
Alistair Coles (alistair.coles@hpe.com)
Clément Contini (ccontini@cloudops.com)
Brian Curtin (brian.curtin@rackspace.com)
Thiago da Silva (thiago@redhat.com)
Julien Danjou (julien@danjou.info)
Paul Dardeau (paul.dardeau@intel.com)
Zack M. Davis (zdavis@swiftstack.com)
Ksenia Demina (kdemina@mirantis.com)
Dan Dillinger (dan.dillinger@sonian.net)
Cedric Dos Santos (cedric.dos.sant@gmail.com)
Gerry Drudy (gerry.drudy@hp.com)
Gerry Drudy (gerry.drudy@hpe.com)
Morgan Fainberg (morgan.fainberg@gmail.com)
ZhiQiang Fan (aji.zqfan@gmail.com)
Oshrit Feder (oshritf@il.ibm.com)
@ -85,6 +88,7 @@ David Goetz (david.goetz@rackspace.com)
Tushar Gohad (tushar.gohad@intel.com)
Jonathan Gonzalez V (jonathan.abdiel@gmail.com)
Joe Gordon (jogo@cloudscaling.com)
ChangBo Guo(gcb) (eric.guo@easystack.cn)
David Hadas (davidh@il.ibm.com)
Andrew Hale (andy@wwwdata.eu)
Soren Hansen (soren@linux2go.dk)
@ -92,9 +96,12 @@ Richard Hawkins (richard.hawkins@rackspace.com)
Gregory Haynes (greg@greghaynes.net)
Doug Hellmann (doug.hellmann@dreamhost.com)
Dan Hersam (dan.hersam@hp.com)
hgangwx (hgangwx@cn.ibm.com)
Derek Higgins (derekh@redhat.com)
Jonathan Hinson (jlhinson@us.ibm.com)
Alex Holden (alex@alexjonasholden.com)
Edward Hope-Morley (opentastic@gmail.com)
Ferenc Horváth (hferenc@inf.u-szeged.hu)
Charles Hsu (charles0126@gmail.com)
Joanna H. Huang (joanna.huitzu.huang@gmail.com)
Kun Huang (gareth@unitedstack.com)
@ -111,6 +118,7 @@ Jason Johnson (jajohnson@softlayer.com)
Brian K. Jones (bkjones@gmail.com)
Arnaud JOST (arnaud.jost@ovh.net)
Kiyoung Jung (kiyoung.jung@kt.com)
Harshada Mangesh Kakad (harshadak@metsi.co.uk)
Takashi Kajinami (kajinamit@nttdata.co.jp)
Matt Kassawara (mkassawara@gmail.com)
Morita Kazutaka (morita.kazutaka@gmail.com)
@ -136,6 +144,8 @@ Eohyung Lee (liquidnuker@gmail.com)
Zhao Lei (zhaolei@cn.fujitsu.com)
Jamie Lennox (jlennox@redhat.com)
Tong Li (litong01@us.ibm.com)
Ke Liang (ke.liang@easystack.cn)
Peter Lisak (peter.lisak@firma.seznam.cz)
Changbin Liu (changbin.liu@gmail.com)
Jing Liuqing (jing.liuqing@99cloud.net)
Victor Lowther (victor.lowther@gmail.com)
@ -143,6 +153,7 @@ Sergey Lukjanov (slukjanov@mirantis.com)
Zhongyue Luo (zhongyue.nah@intel.com)
Paul Luse (paul.e.luse@intel.com)
Christopher MacGown (chris@pistoncloud.com)
Ganesh Maharaj Mahalingam (ganesh.mahalingam@intel.com)
Dragos Manolescu (dragosm@hp.com)
Ben Martin (blmartin@us.ibm.com)
Steve Martinelli (stevemar@ca.ibm.com)
@ -152,7 +163,7 @@ Nakagawa Masaaki (nakagawamsa@nttdata.co.jp)
Dolph Mathews (dolph.mathews@gmail.com)
Kenichiro Matsuda (matsuda_kenichi@jp.fujitsu.com)
Michael Matur (michael.matur@gmail.com)
Donagh McCabe (donagh.mccabe@hp.com)
Donagh McCabe (donagh.mccabe@hpe.com)
Andy McCrae (andy.mccrae@gmail.com)
Paul McMillan (paul.mcmillan@nebula.com)
Ewan Mellor (ewan.mellor@citrix.com)
@ -168,19 +179,22 @@ Maru Newby (mnewby@internap.com)
Newptone (xingchao@unitedstack.com)
Colin Nicholson (colin.nicholson@iomart.com)
Zhenguo Niu (zhenguo@unitedstack.com)
Catherine Northcott (catherine@northcott.nz)
Ondrej Novy (ondrej.novy@firma.seznam.cz)
Timothy Okwii (tokwii@cisco.com)
Matthew Oliver (matt@oliver.net.au)
Hisashi Osanai (osanai.hisashi@jp.fujitsu.com)
Eamonn O'Toole (eamonn.otoole@hp.com)
Eamonn O'Toole (eamonn.otoole@hpe.com)
James Page (james.page@ubuntu.com)
Prashanth Pai (ppai@redhat.com)
Venkateswarlu Pallamala (p.venkatesh551@gmail.com)
Pawel Palucki (pawel.palucki@gmail.com)
Alex Pecoraro (alex.pecoraro@emc.com)
Sascha Peilicke (saschpe@gmx.de)
Constantine Peresypkin (constantine.peresypk@rackspace.com)
Dieter Plaetinck (dieter@vimeo.com)
Dan Prince (dprince@redhat.com)
Sivasathurappan Radhakrishnan (siva.radhakrishnan@intel.com)
Sarvesh Ranjan (saranjan@cisco.com)
Falk Reimann (falk.reimann@sap.com)
Brian Reitz (brian.reitz@oracle.com)
@ -198,7 +212,7 @@ Shilla Saebi (shilla.saebi@gmail.com)
Atsushi Sakai (sakaia@jp.fujitsu.com)
Cristian A Sanchez (cristian.a.sanchez@intel.com)
Christian Schwede (cschwede@redhat.com)
Mark Seger (Mark.Seger@hp.com)
Mark Seger (mark.seger@hpe.com)
Azhagu Selvan SP (tamizhgeek@gmail.com)
Alexandra Settle (alexandra.settle@rackspace.com)
Andrew Clay Shafer (acs@parvuscaptus.com)
@ -212,6 +226,7 @@ Pradeep Kumar Singh (pradeep.singh@nectechnologies.in)
Liu Siqi (meizu647@gmail.com)
Adrian Smith (adrian_f_smith@dell.com)
Jon Snitow (otherjon@swiftstack.com)
Emile Snyder (emile.snyder@gmail.com)
Emett Speer (speer.emett@gmail.com)
TheSriram (sriram@klusterkloud.com)
Jeremy Stanley (fungi@yuggoth.org)
@ -234,7 +249,9 @@ Dmitry Ukov (dukov@mirantis.com)
Vincent Untz (vuntz@suse.com)
Daniele Valeriani (daniele@dvaleriani.net)
Koert van der Veer (koert@cloudvps.com)
Béla Vancsics (vancsics@inf.u-szeged.hu)
Vladimir Vechkanov (vvechkanov@mirantis.com)
venkatamahesh (venkatamaheshkotha@gmail.com)
Gil Vernik (gilv@il.ibm.com)
Hou Ming Wang (houming.wang@easystack.cn)
Shane Wang (shane.wang@intel.com)
@ -248,7 +265,7 @@ Ye Jia Xu (xyj.asmy@gmail.com)
Alex Yang (alex890714@gmail.com)
Lin Yang (lin.a.yang@intel.com)
Yee (mail.zhang.yee@gmail.com)
Guang Yee (guang.yee@hp.com)
Guang Yee (guang.yee@hpe.com)
Pete Zaitcev (zaitcev@kotori.zaitcev.us)
Hua Zhang (zhuadl@cn.ibm.com)
Jian Zhang (jian.zhang@intel.com)

89
CHANGELOG

@ -1,3 +1,92 @@
swift (2.6.0)
* Dependency changes
- Updated minimum version of eventlet to 0.17.4 to support IPv6.
- Updated the minimum version of PyECLib to 1.0.7.
* The ring rebalancing algorithm was updated to better handle edge cases
and to give better (more balanced) rings in the general case. New rings
will have better initial placement, capacity adjustments will move less
data for better balance, and existing rings that were imbalanced should
start to become better balanced as they go through rebalance cycles.
* Added container and account reverse listings.
A GET request to an account or container resource with a "reverse=true"
query parameter will return the listing in reverse order. When
iterating over pages of reverse listings, the relative order of marker
and end_marker are swapped.
* Storage policies now support having more than one name.
This allows operators to fix a typo without breaking existing clients,
or, alternatively, have "short names" for policies. This is implemented
with the "aliases" config key in the storage policy config in
swift.conf. The aliases value is a list of names that the storage
policy may also be identified by. The storage policy "name" is used to
report the policy to users (eg in container headers). The aliases have
the same naming restrictions as the policy's primary name.
* The object auditor learned the "interval" config value to control the
time between each audit pass.
* `swift-recon --all` now includes the config checksum check.
* `swift-init` learned the --kill-after-timeout option to force a service
to quit (SIGKILL) after a designated time.
* `swift-recon` now correctly shows timestamps in UTC instead of local
time.
* Fixed bug where `swift-ring-builder` couldn't select device id 0.
* Documented the previously undocumented
`swift-ring-builder pretend_min_part_hours_passed` command.
* The "node_timeout" config value now accepts decimal values.
* `swift-ring-builder` now properly removes devices with zero weight.
* `swift-init` return codes are updated via "--strict" and "--non-strict"
options. Please see the usage string for more information.
* `swift-ring-builder` now reports the min_part_hours lockout time
remaining
* Container sync has been improved to more quickly find and iterate over
the containers to be synced. This reduced server load and lowers the
time required to see data propagate between two clusters. Please see
http://swift.openstack.org/overview_container_sync.html for more details
about the new on-disk structure for tracking synchronized containers.
* A container POST will now update that container's put-timestamp value.
* TempURL header restrictions are now exposed in /info.
* Error messages on static large object manifest responses have been
greatly improved.
* Closed a bug where an unfinished read of a large object would leak a
socket file descriptor and a small amount of memory. (CVE-2016-0738)
* Fixed an issue where a zero-byte object PUT with an incorrect Etag
would return a 503.
* Fixed an error when a static large object manifest references the same
object more than once.
* Improved performance of finding handoff nodes if a zone is empty.
* Fixed duplication of headers in Access-Control-Expose-Headers on CORS
requests.
* Fixed handling of IPv6 connections to memcache pools.
* Continued work towards python 3 compatibility.
* Various other minor bug fixes and improvements.
swift (2.5.0, OpenStack Liberty)
* Added the ability to specify ranges for Static Large Object (SLO)

4
CONTRIBUTING.md

@ -89,8 +89,8 @@ Specs
The [``swift-specs``](https://github.com/openstack/swift-specs) repo
can be used for collaborative design work before a feature is implemented.
Openstack's gerrit system is used to collaborate on the design spec. Once
approved Openstack provides a doc site to easily read these [specs](http://specs.openstack.org/openstack/swift-specs/)
OpenStack's gerrit system is used to collaborate on the design spec. Once
approved OpenStack provides a doc site to easily read these [specs](http://specs.openstack.org/openstack/swift-specs/)
A spec is needed for more impactful features. Coordinating a feature between
many devs (especially across companies) is a great example of when a spec is

5
bin/swift-dispersion-report

@ -23,7 +23,6 @@ from time import time
from eventlet import GreenPool, hubs, patcher, Timeout
from eventlet.pools import Pool
from eventlet.green import urllib2
from swift.common import direct_client
try:
@ -174,8 +173,8 @@ def object_dispersion_report(coropool, connpool, account, object_ring,
try:
objects = [o['name'] for o in conn.get_container(
container, prefix='dispersion_', full_listing=True)[1]]
except urllib2.HTTPError as err:
if err.getcode() != 404:
except ClientException as err:
if err.http_status != 404:
raise
print >>stderr, 'No objects to query. Has ' \

4
bin/swift-drive-audit

@ -200,6 +200,10 @@ if __name__ == '__main__':
(mount_point))
comment_fstab(mount_point)
unmounts += 1
else:
logger.info("Detected %s with %d errors "
"(Device not unmounted)" %
(mount_point, count))
recon_errors[mount_point] = count
total_errors += count
recon_file = recon_cache_path + "/drive.recon"

5
bin/swift-init

@ -74,6 +74,11 @@ def main():
help="Return zero status code even if some config is "
"missing. Default mode if any server is a glob or "
"one of aliases `all`, `main` or `rest`.")
# SIGKILL daemon after kill_wait period
parser.add_option('--kill-after-timeout', dest='kill_after_timeout',
action='store_true',
help="Kill daemon and all childs after kill-wait "
"period.")
options, args = parser.parse_args()

2
bin/swift-oldies

@ -59,7 +59,7 @@ Lists old Swift processes.
listing.append((str(hours), pid, args))
if not listing:
exit()
sys.exit()
hours_len = len('Hours')
pid_len = len('PID')

2
bin/swift-orphans

@ -93,7 +93,7 @@ Example (sends SIGTERM to all orphaned Swift processes older than two hours):
listing.append((str(hours), pid, args))
if not listing:
exit()
sys.exit()
hours_len = len('Hours')
pid_len = len('PID')

6
doc/manpages/account-server.conf.5

@ -102,8 +102,10 @@ adapted_logger. The default is empty.
If set, log_udp_host will override log_address.
.IP "\fBlog_udp_port\fR
UDP log port, the default is 514.
.IP \fBlog_statsd_host\fR = localhost
log_statsd_* enable StatsD logging.
.IP \fBlog_statsd_host\fR
StatsD server. IPv4/IPv6 addresses and hostnames are
supported. If a hostname resolves to an IPv4 and IPv6 address, the IPv4
address will be used.
.IP \fBlog_statsd_port\fR
The default is 8125.
.IP \fBlog_statsd_default_sample_rate\fR

6
doc/manpages/container-server.conf.5

@ -108,8 +108,10 @@ adapted_logger. The default is empty.
If set, log_udp_host will override log_address.
.IP "\fBlog_udp_port\fR
UDP log port, the default is 514.
.IP \fBlog_statsd_host\fR = localhost
log_statsd_* enable StatsD logging.
.IP \fBlog_statsd_host\fR
StatsD server. IPv4/IPv6 addresses and hostnames are
supported. If a hostname resolves to an IPv4 and IPv6 address, the IPv4
address will be used.
.IP \fBlog_statsd_port\fR
The default is 8125.
.IP \fBlog_statsd_default_sample_rate\fR

6
doc/manpages/object-expirer.conf.5

@ -76,8 +76,10 @@ adapted_logger. The default is empty.
If set, log_udp_host will override log_address.
.IP "\fBlog_udp_port\fR
UDP log port, the default is 514.
.IP \fBlog_statsd_host\fR = localhost
log_statsd_* enable StatsD logging.
.IP \fBlog_statsd_host\fR
StatsD server. IPv4/IPv6 addresses and hostnames are
supported. If a hostname resolves to an IPv4 and IPv6 address, the IPv4
address will be used.
.IP \fBlog_statsd_port\fR
The default is 8125.
.IP \fBlog_statsd_default_sample_rate\fR

10
doc/manpages/object-server.conf.5

@ -111,8 +111,10 @@ adapted_logger. The default is empty.
If set, log_udp_host will override log_address.
.IP "\fBlog_udp_port\fR
UDP log port, the default is 514.
.IP \fBlog_statsd_host\fR = localhost
log_statsd_* enable StatsD logging.
.IP \fBlog_statsd_host\fR
StatsD server. IPv4/IPv6 addresses and hostnames are
supported. If a hostname resolves to an IPv4 and IPv6 address, the IPv4
address will be used.
.IP \fBlog_statsd_port\fR
The default is 8125.
.IP \fBlog_statsd_default_sample_rate\fR
@ -365,7 +367,7 @@ Depending on the method of deployment you may need to create this directory manu
and ensure that swift has read/write.The default is /var/cache/swift.
.IP "\fBhandoffs_first\fR"
The flag to replicate handoffs prior to canonical partitions.
It allows to force syncing and deleting handoffs quickly.
It allows one to force syncing and deleting handoffs quickly.
If set to a True value(e.g. "True" or "1"), partitions
that are not supposed to be on the node will be replicated first.
The default is false.
@ -425,7 +427,7 @@ Depending on the method of deployment you may need to create this directory manu
and ensure that swift has read/write.The default is /var/cache/swift.
.IP "\fBhandoffs_first\fR"
The flag to replicate handoffs prior to canonical partitions.
It allows to force syncing and deleting handoffs quickly.
It allows one to force syncing and deleting handoffs quickly.
If set to a True value(e.g. "True" or "1"), partitions
that are not supposed to be on the node will be replicated first.
The default is false.

25
doc/manpages/proxy-server.conf.5

@ -118,8 +118,10 @@ adapted_logger. The default is empty.
If set, log_udp_host will override log_address.
.IP "\fBlog_udp_port\fR
UDP log port, the default is 514.
.IP \fBlog_statsd_host\fR = localhost
log_statsd_* enable StatsD logging.
.IP \fBlog_statsd_host\fR
StatsD server. IPv4/IPv6 addresses and hostnames are
supported. If a hostname resolves to an IPv4 and IPv6 address, the IPv4
address will be used.
.IP \fBlog_statsd_port\fR
The default is 8125.
.IP \fBlog_statsd_default_sample_rate\fR
@ -328,8 +330,8 @@ This allows middleware higher in the WSGI pipeline to override auth
processing, useful for middleware such as tempurl and formpost. If you know
you're not going to use such middleware and you want a bit of extra security,
you can set this to false.
.IP \fBis_admin [DEPRECATED]\fR
If is_admin is true, a user whose username is the same as the project name
.IP \fBis_admin\fR
[DEPRECATED] If is_admin is true, a user whose username is the same as the project name
and who has any role on the project will have access rights elevated to be
the same as if the user had an operator role. Note that the condition
compares names rather than UUIDs. This option is deprecated.
@ -384,7 +386,8 @@ Sets the maximum number of connections to each memcached server per worker.
If not set in the configuration file, the value for memcache_servers will be
read from /etc/swift/memcache.conf (see memcache.conf-sample) or lacking that
file, it will default to 127.0.0.1:11211. You can specify multiple servers
separated with commas, as in: 10.1.2.3:11211,10.1.2.4:11211.
separated with commas, as in: 10.1.2.3:11211,10.1.2.4:11211. (IPv6
addresses must follow rfc3986 section-3.2.2, i.e. [::1]:11211)
.IP \fBmemcache_serialization_support\fR
This sets how memcache values are serialized and deserialized:
.RE
@ -665,7 +668,9 @@ unset.
Default is 514.
.IP \fBaccess_log_statsd_host\fR
You can use log_statsd_* from [DEFAULT], or override them here.
Default is localhost.
StatsD server. IPv4/IPv6 addresses and hostnames are
supported. If a hostname resolves to an IPv4 and IPv6 address, the IPv4
address will be used.
.IP \fBaccess_log_statsd_port\fR
Default is 8125.
.IP \fBaccess_log_statsd_default_sample_rate\fR
@ -949,7 +954,7 @@ chunk of data from the object servers while serving GET / HEAD requests.
Timeouts from these requests can be recovered from so setting this to
something lower than node_timeout would provide quicker error recovery
while allowing for a longer timeout for non-recoverable requests (PUTs).
Defaults to node_timeout, should be overriden if node_timeout is set to a
Defaults to node_timeout, should be overridden if node_timeout is set to a
high number to prevent client timeouts from firing before the proxy server
has a chance to retry.
.IP \fBconn_timeout\fR
@ -997,11 +1002,9 @@ The valid values for sorting_method are "affinity", "shuffle", and "timing".
.IP \fBtiming_expiry\fR
If the "timing" sorting_method is used, the timings will only be valid for
the number of seconds configured by timing_expiry. The default is 300.
.IP \fBmax_large_object_get_time\fR
The maximum time (seconds) that a large object connection is allowed to last. The default is 86400.
.IP \fBrequest_node_count\fR
Set to the number of nodes to contact for a normal request. You can use
'* replicas' at the end to have it use the number given times the number of
Set to the number of nodes to contact for a normal request. You can use '* replicas'
at the end to have it use the number given times the number of
replicas for the ring being used for the request. The default is '2 * replicas'.
.IP \fBread_affinity\fR
Which backend servers to prefer on reads. Format is r<N> for region

1
doc/manpages/swift-init.1

@ -111,6 +111,7 @@ allows one to use the keywords such as "all", "main" and "rest" for the <server>
.IP "-r RUN_DIR, --run-dir=RUN_DIR directory where the pids will be stored (default /var/run/swift)
.IP "--strict return non-zero status code if some config is missing. Default mode if server is explicitly named."
.IP "--non-strict return zero status code even if some config is missing. Default mode if server is one of aliases `all`, `main` or `rest`."
.IP "--kill-after-timeout kill daemon and all children after kill-wait period."
.PD
.RE

2
doc/saio/swift/container-reconciler.conf

@ -17,7 +17,7 @@ user = <your-user-name>
# log_udp_port = 514
#
# You can enable StatsD logging here:
# log_statsd_host = localhost
# log_statsd_host =
# log_statsd_port = 8125
# log_statsd_default_sample_rate = 1.0
# log_statsd_sample_rate_factor = 1.0

2
doc/saio/swift/object-expirer.conf

@ -17,7 +17,7 @@ log_level = INFO
# log_udp_port = 514
#
# You can enable StatsD logging here:
# log_statsd_host = localhost
# log_statsd_host =
# log_statsd_port = 8125
# log_statsd_default_sample_rate = 1.0
# log_statsd_sample_rate_factor = 1.0

3
doc/saio/swift/swift.conf

@ -1,5 +1,6 @@
[swift-hash]
# random unique strings that can never change (DO NOT LOSE)
# Use only printable chars (python -c "import string; print(string.printable)")
swift_hash_path_prefix = changeme
swift_hash_path_suffix = changeme
@ -15,6 +16,6 @@ policy_type = replication
[storage-policy:2]
name = ec42
policy_type = erasure_coding
ec_type = jerasure_rs_vand
ec_type = liberasurecode_rs_vand
ec_num_data_fragments = 4
ec_num_parity_fragments = 2

16
doc/source/admin_guide.rst

@ -463,7 +463,12 @@ Example::
Assuming 3 replicas, this configuration will make object PUTs try
storing the object's replicas on up to 6 disks ("2 * replicas") in
region 1 ("r1").
region 1 ("r1"). Proxy server tries to find 3 devices for storing the
object. While a device is unavailable, it queries the ring for the 4th
device and so on until 6th device. If the 6th disk is still unavailable,
the last replica will be sent to other region. It doesn't mean there'll
have 6 replicas in region 1.
You should be aware that, if you have data coming into SF faster than
your link to NY can transfer it, then your cluster's data distribution
@ -624,7 +629,11 @@ configuration entries (see the sample configuration files)::
log_statsd_metric_prefix = [empty-string]
If `log_statsd_host` is not set, this feature is disabled. The default values
for the other settings are given above.
for the other settings are given above. The `log_statsd_host` can be a
hostname, an IPv4 address, or an IPv6 address (not surrounded with brackets, as
this is unnecessary since the port is specified separately). If a hostname
resolves to an IPv4 address, an IPv4 socket will be used to send StatsD UDP
packets, even if the hostname would also resolve to an IPv6 address.
.. _StatsD: http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/
.. _Graphite: http://graphite.wikidot.com/
@ -675,8 +684,7 @@ of async_pendings in real-time, but will not tell you the current number of
async_pending container updates on disk at any point in time.
Note also that the set of metrics collected, their names, and their semantics
are not locked down and will change over time. StatsD logging is currently in
a "beta" stage and will continue to evolve.
are not locked down and will change over time.
Metrics for `account-auditor`:

12
doc/source/api/form_post_middleware.rst

@ -27,7 +27,7 @@ request.
The format of the form **POST** request is:
**Example 1.14. Form POST format**
**Example 1.14. Form POST format**
.. code::
@ -140,7 +140,7 @@ Form **POST** middleware uses an HMAC-SHA1 cryptographic signature. This
signature includes these elements from the form:
- The path. Starting with ``/v1/`` onwards and including a container
name and, optionally, an object prefix. In `Example 1.15`, “HMAC-SHA1
name and, optionally, an object prefix. In `Example 1.15`, “HMAC-SHA1
signature for form
POST” the path is
``/v1/my_account/container/object_prefix``. Do not URL-encode the
@ -148,15 +148,15 @@ signature includes these elements from the form:
- A redirect URL. If there is no redirect URL, use the empty string.
- Maximum file size. In `Example 1.15`, “HMAC-SHA1 signature for form
- Maximum file size. In `Example 1.15`, “HMAC-SHA1 signature for form
POST” the
``max_file_size`` is ``104857600`` bytes.
- The maximum number of objects to upload. In `Example 1.15`, “HMAC-SHA1
- The maximum number of objects to upload. In `Example 1.15`, “HMAC-SHA1
signature for form
POST” ``max_file_count`` is ``10``.
- Expiry time. In `Example 1.15, “HMAC-SHA1 signature for form
- Expiry time. In `Example 1.15, “HMAC-SHA1 signature for form
POST” the expiry time
is set to ``600`` seconds into the future.
@ -167,7 +167,7 @@ signature includes these elements from the form:
The following example code generates a signature for use with form
**POST**:
**Example 1.15. HMAC-SHA1 signature for form POST**
**Example 1.15. HMAC-SHA1 signature for form POST**
.. code::

127
doc/source/api/large_objects.rst

@ -2,7 +2,7 @@
Large objects
=============
By default, the content of an object cannot be greater than 5 GB.
By default, the content of an object cannot be greater than 5 GB.
However, you can use a number of smaller objects to construct a large
object. The large object is comprised of two types of objects:
@ -40,9 +40,9 @@ Note
If you make a **COPY** request by using a manifest object as the source,
the new object is a normal, and not a segment, object. If the total size
of the source segment objects exceeds 5 GB, the **COPY** request fails.
of the source segment objects exceeds 5 GB, the **COPY** request fails.
However, you can make a duplicate of the manifest object and this new
object can be larger than 5 GB.
object can be larger than 5 GB.
Static large objects
~~~~~~~~~~~~~~~~~~~~
@ -58,7 +58,7 @@ header. This ensures that the upload cannot corrupt your data.
List the name of each segment object along with its size and MD5
checksum in order.
Create a manifest object. Include the *``?multipart-manifest=put``*
Create a manifest object. Include the ``multipart-manifest=put``
query string at the end of the manifest object name to indicate that
this is a manifest object.
@ -74,7 +74,7 @@ list, where each element contains the following attributes:
- ``size_bytes``. The size of the segment object. This value must match
the ``Content-Length`` of that object.
**Example Static large object manifest list**
**Example Static large object manifest list**
This example shows three segment objects. You can use several containers
and the object names do not have to conform to a specific pattern, in
@ -112,8 +112,8 @@ set to be the MD5 checksum of the concatenated ``ETag`` values of the
object segments. You can also set the ``Content-Type`` request header
and custom object metadata.
When the **PUT** operation sees the *``?multipart-manifest=put``* query
parameter, it reads the request body and verifies that each segment
When the **PUT** operation sees the ``multipart-manifest=put`` query
string, it reads the request body and verifies that each segment
object exists and that the sizes and ETags match. If there is a
mismatch, the **PUT**\ operation fails.
@ -124,25 +124,25 @@ this is a static object manifest.
Normally when you perform a **GET** operation on the manifest object,
the response body contains the concatenated content of the segment
objects. To download the manifest list, use the
*``?multipart-manifest=get``* query parameter. The resulting list is not
``multipart-manifest=get`` query string. The resulting list is not
formatted the same as the manifest you originally used in the **PUT**
operation.
If you use the **DELETE** operation on a manifest object, the manifest
object is deleted. The segment objects are not affected. However, if you
add the *``?multipart-manifest=delete``* query parameter, the segment
add the ``multipart-manifest=delete`` query string, the segment
objects are deleted and if all are successfully deleted, the manifest
object is also deleted.
To change the manifest, use a **PUT** operation with the
*``?multipart-manifest=put``* query parameter. This request creates a
``multipart-manifest=put`` query string. This request creates a
manifest object. You can also update the object metadata in the usual
way.
Dynamic large objects
~~~~~~~~~~~~~~~~~~~~~
You must segment objects that are larger than 5 GB before you can upload
You must segment objects that are larger than 5 GB before you can upload
them. You then upload the segment objects like you would any other
object and create a dynamic large manifest object. The manifest object
tells Object Storage how to find the segment objects that comprise the
@ -168,7 +168,7 @@ of segments to a second location and update the manifest to point to
this new location. During the upload of the new segments, the original
manifest is still available to download the first set of segments.
**Example Upload segment of large object request: HTTP**
**Example Upload segment of large object request: HTTP**
.. code::
@ -190,7 +190,7 @@ Unprocessable Entity response is returned.
You can continue uploading segments like this example shows, prior to
uploading the manifest.
**Example Upload next segment of large object request: HTTP**
**Example Upload next segment of large object request: HTTP**
.. code::
@ -220,7 +220,7 @@ subsequent additional segments.
X-Object-Manifest: {container}/{prefix}
**Example Upload manifest response: HTTP**
**Example Upload manifest response: HTTP**
.. code::
@ -238,67 +238,88 @@ Comparison of static and dynamic large objects
While static and dynamic objects have similar behavior, here are
their differences:
**Comparing static and dynamic large objects**
End-to-end integrity
--------------------
Static large object: Assured end-to-end integrity. The list of segments
includes the MD5 checksum (``ETag``) of each segment. You cannot upload the
manifest object if the ``ETag`` in the list differs from the uploaded segment
object. If a segment is somehow lost, an attempt to download the manifest
object results in an error. You must upload the segment objects before you
upload the manifest object. You cannot add or remove segment objects from the
manifest. However, you can create a completely new manifest object of the same
name with a different manifest list.
With static large objects, integrity can be assured.
The list of segments may include the MD5 checksum (``ETag``) of each segment.
You cannot upload the manifest object if the ``ETag`` in the list differs
from the uploaded segment object. If a segment is somehow lost, an attempt
to download the manifest object results in an error.
With static large objects, you can upload new segment objects or remove
existing segments. The names must simply match the ``{prefix}`` supplied
in ``X-Object-Manifest``. The segment objects must be at least 1 MB in size
(by default). The final segment object can be any size. At most, 1000 segments
are supported (by default). The manifest list includes the container name of
each object. Segment objects can be in different containers.
Dynamic large object: End-to-end integrity is not guaranteed. The eventual
With dynamic large objects, integrity is not guaranteed. The eventual
consistency model means that although you have uploaded a segment object, it
might not appear in the container listing until later. If you download the
manifest before it appears in the container, it does not form part of the
content returned in response to a **GET** request.
Upload Order
------------
With static large objects, you must upload the
segment objects before you upload the manifest object.
With dynamic large objects, you can upload manifest and segment objects
in any order. In case a premature download of the manifest occurs, we
recommend users upload the manifest object after the segments. However,
the system does not enforce the order. Segment objects can be any size. All
segment objects must be in the same container.
the system does not enforce the order.
Removal or addition of segment objects
--------------------------------------
With static large objects, you cannot add or
remove segment objects from the manifest. However, you can create a
completely new manifest object of the same name with a different manifest
list.
With dynamic large objects, you can upload new segment objects or remove
existing segments. The names must simply match the ``{prefix}`` supplied
in ``X-Object-Manifest``.
Segment object size and number
------------------------------
With static large objects, the segment objects must be at least 1 byte in size.
However, if the segment objects are less than 1MB (by default),
the SLO download is (by default) rate limited. At most,
1000 segments are supported (by default) and the manifest has a limit
(by default) of 2MB in size.
With dynamic large objects, segment objects can be any size.
Segment object container name
-----------------------------
With static large objects, the manifest list includes the container name of each object.
Segment objects can be in different containers.
With dynamic large objects, all segment objects must be in the same container.
Manifest object metadata
------------------------
For static large objects, the object has ``X-Static-Large-Object`` set to
``true``. You do not set this metadata directly. Instead the system sets
it when you **PUT** a static manifest object.
With static large objects, the manifest object has ``X-Static-Large-Object``
set to ``true``. You do not set this
metadata directly. Instead the system sets it when you **PUT** a static
manifest object.
For dynamic object,s the ``X-Object-Manifest`` value is the
``{container}/{prefix}``, which indicates where the segment objects are
located. You supply this request header in the **PUT** operation.
With dynamic large objects, the ``X-Object-Manifest`` value is the
``{container}/{prefix}``, which indicates
where the segment objects are located. You supply this request header in the
**PUT** operation.
Copying the manifest object
---------------------------
With static large objects, you include the *``?multipart-manifest=get``*
The semantics are the same for both static and dynamic large objects.
When copying large objects, the **COPY** operation does not create
a manifest object but a normal object with content same as what you would
get on a **GET** request to the original manifest object.
To copy the manifest object, you include the ``multipart-manifest=get``
query string in the **COPY** request. The new object contains the same
manifest as the original. The segment objects are not copied. Instead,
both the original and new manifest objects share the same set of segment
objects.
When creating dynamic large objects, the **COPY** operation does not create
a manifest object but a normal object with content same as what you would
get on a **GET** request to original manifest object.
To duplicate a manifest object:
* Use the **GET** operation to read the value of ``X-Object-Manifest`` and
use this value in the ``X-Object-Manifest`` request header in a **PUT**
operation.
* Alternatively, you can include *``?multipart-manifest=get``* query
string in the **COPY** request.
This creates a new manifest object that shares the same set of segment
objects as the original manifest object.

16
doc/source/api/object_api_v1_overview.rst

@ -58,7 +58,7 @@ The Object Storage system organizes data in a hierarchy, as follows:
object versioning, at the container level.
You can bulk-delete up to 10,000 containers in a single request.
You can set a storage policy on a container with predefined names
and definitions from your cloud provider.
@ -68,7 +68,7 @@ The Object Storage system organizes data in a hierarchy, as follows:
With the Object Storage API, you can:
- Store an unlimited number of objects. Each object can be as large
as 5 GB, which is the default. You can configure the maximum
as 5 GB, which is the default. You can configure the maximum
object size.
- Upload and store objects of any size with large object creation.
@ -78,7 +78,7 @@ The Object Storage system organizes data in a hierarchy, as follows:
- Compress files using content-encoding metadata.
- Override browser behavior for an object using content-disposition metadata.
- Schedule objects for deletion.
- Bulk-delete up to 10,000 objects in a single request.
@ -154,11 +154,11 @@ Your service provider might use different default values.
Item Maximum value Notes
============================ ============= =====
Number of HTTP headers 90
Length of HTTP headers 4096 bytes
Length per HTTP request line 8192 bytes
Length of HTTP request 5 GB
Length of container names 256 bytes Cannot contain the ``/`` character.
Length of object names 1024 bytes By default, there are no character restrictions.
Length of HTTP headers 4096 bytes
Length per HTTP request line 8192 bytes
Length of HTTP request 5 GB
Length of container names 256 bytes Cannot contain the ``/`` character.
Length of object names 1024 bytes By default, there are no character restrictions.
============================ ============= =====
You must UTF-8-encode and then URL-encode container and object names

2
doc/source/api/use_content-encoding_metadata.rst

@ -7,7 +7,7 @@ the ``Content-Encoding`` metadata. This metadata enables you to indicate
that the object content is compressed without losing the identity of the
underlying media type (``Content-Type``) of the file, such as a video.
**Example Content-Encoding header request: HTTP**
**Example Content-Encoding header request: HTTP**
This example assigns an attachment type to the ``Content-Encoding``
header that indicates how the file is downloaded:

5
doc/source/associated_projects.rst

@ -25,7 +25,7 @@ Application Bindings
* `java-openstack-swift <https://github.com/dkocher/java-openstack-swift>`_ - Java bindings for OpenStack Swift
* `swift_client <https://github.com/mrkamel/swift_client>`_ - Small but powerful Ruby client to interact with OpenStack Swift
* `nightcrawler_swift <https://github.com/tulios/nightcrawler_swift>`_ - This Ruby gem teleports your assets to a OpenStack Swift bucket/container
* `swift storage <https://rubygems.org/gems/swift-storage>`_ - Simple Openstack Swift storage client.
* `swift storage <https://rubygems.org/gems/swift-storage>`_ - Simple OpenStack Swift storage client.
Authentication
--------------
@ -65,6 +65,7 @@ Alternative API
* `Swift3 <https://github.com/openstack/swift3>`_ - Amazon S3 API emulation.
* `CDMI <https://github.com/osaddon/cdmi>`_ - CDMI support
* `SwiftHLM <https://github.com/ibm-research/SwiftHLM>`_ - a middleware for using OpenStack Swift with tape and other high latency media storage backends
Benchmarking/Load Generators
@ -106,7 +107,7 @@ Other
* `Glance <https://github.com/openstack/glance>`_ - Provides services for discovering, registering, and retrieving virtual machine images (for OpenStack Compute [Nova], for example).
* `Better Staticweb <https://github.com/CloudVPS/better-staticweb>`_ - Makes swift containers accessible by default.
* `Swiftsync <https://github.com/stackforge/swiftsync>`_ - A massive syncer between two swift clusters.
* `Django Swiftbrowser <https://github.com/cschwede/django-swiftbrowser>`_ - Simple Django web app to access Openstack Swift.
* `Django Swiftbrowser <https://github.com/cschwede/django-swiftbrowser>`_ - Simple Django web app to access OpenStack Swift.
* `Swift-account-stats <https://github.com/enovance/swift-account-stats>`_ - Swift-account-stats is a tool to report statistics on Swift usage at tenant and global levels.
* `PyECLib <https://bitbucket.org/kmgreen2/pyeclib>`_ - High Level Erasure Code library used by Swift
* `liberasurecode <http://www.bytebucket.org/tsg-/liberasurecode>`_ - Low Level Erasure Code library used by PyECLib

106
doc/source/conf.py

@ -1,4 +1,17 @@
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Copyright (c) 2010-2012 OpenStack Foundation.
#
# Swift documentation build configuration file, created by
@ -13,9 +26,11 @@
# All configuration values have a default; values that are commented out
# serve to show the default.
import sys
import os
import datetime
import os
from swift import __version__
import subprocess
import sys
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
@ -28,7 +43,7 @@ sys.path.extend([os.path.abspath('../swift'), os.path.abspath('..'),
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = ['sphinx.ext.autodoc',
'sphinx.ext.todo', 'sphinx.ext.coverage', 'sphinx.ext.pngmath',
'sphinx.ext.todo', 'sphinx.ext.coverage',
'sphinx.ext.ifconfig', 'oslosphinx']
todo_include_todos = True
@ -36,17 +51,17 @@ todo_include_todos = True
# Changing the path so that the Hudson build output contains GA code and the
# source docs do not contain the code so local, offline sphinx builds are
# "clean."
#templates_path = []
#if os.getenv('HUDSON_PUBLISH_DOCS'):
# templates_path = ['_ga', '_templates']
#else:
# templates_path = ['_templates']
# templates_path = []
# if os.getenv('HUDSON_PUBLISH_DOCS'):
# templates_path = ['_ga', '_templates']
# else:
# templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8'
# source_encoding = 'utf-8'
# The master toctree document.
master_doc = 'index'
@ -60,23 +75,22 @@ copyright = u'%d, OpenStack Foundation' % datetime.datetime.now().year
# built documents.
#
# The short X.Y version.
from swift import __version__
version = __version__.rsplit('.', 1)[0]
# The full version, including alpha/beta/rc tags.
release = __version__
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#language = None
# language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# today_fmt = '%B %d, %Y'
# List of documents that shouldn't be included in the build.
#unused_docs = []
# unused_docs = []
# List of directories, relative to source directory, that shouldn't be searched
# for source files.
@ -84,14 +98,14 @@ exclude_trees = []
# The reST default role (used for this markup: `text`) to use for all
# documents.
#default_role = None
# default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True
# add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
@ -109,74 +123,76 @@ modindex_common_prefix = ['swift.']
# The theme to use for HTML and HTML Help pages. Major themes that come with
# Sphinx are currently 'default' and 'sphinxdoc'.
# html_theme = 'default'
#html_theme_path = ["."]
#html_theme = '_theme'
# html_theme_path = ["."]
# html_theme = '_theme'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}
# html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
#html_title = None
# html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None
# html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
# html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
#html_static_path = ['_static']
# html_static_path = ['_static']
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'
git_cmd = "git log --pretty=format:'%ad, commit %h' --date=local -n1"
html_last_updated_fmt = os.popen(git_cmd).read()
# html_last_updated_fmt = '%b %d, %Y'
git_cmd = ["git", "log", "--pretty=format:'%ad, commit %h'", "--date=local",
"-n1"]
html_last_updated_fmt = subprocess.Popen(
git_cmd, stdout=subprocess.PIPE).communicate()[0]
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True
# html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# html_additional_pages = {}
# If false, no module index is generated.
#html_use_modindex = True
# html_use_modindex = True
# If false, no index is generated.
#html_use_index = True
# html_use_index = True
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# html_split_index = False
# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True
# html_show_sourcelink = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# html_use_opensearch = ''
# If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = ''
# html_file_suffix = ''
# Output file base name for HTML help builder.
htmlhelp_basename = 'swiftdoc'
@ -185,10 +201,10 @@ htmlhelp_basename = 'swiftdoc'
# -- Options for LaTeX output -------------------------------------------------
# The paper size ('letter' or 'a4').
#latex_paper_size = 'letter'
# latex_paper_size = 'letter'
# The font size ('10pt', '11pt' or '12pt').
#latex_font_size = '10pt'
# latex_font_size = '10pt'
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass
@ -200,17 +216,17 @@ latex_documents = [
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#latex_logo = None
# latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False
# latex_use_parts = False
# Additional stuff for the LaTeX preamble.
#latex_preamble = ''
# latex_preamble = ''
# Documents to append as an appendix to all manuals.
#latex_appendices = []
# latex_appendices = []
# If false, no module index is generated.
#latex_use_modindex = True
# latex_use_modindex = True

38
doc/source/deployment_guide.rst

@ -478,7 +478,11 @@ log_custom_handlers None Comma-separated list of functions t
to setup custom log handlers.
log_udp_host Override log_address
log_udp_port 514 UDP log port
log_statsd_host localhost StatsD logging
log_statsd_host None Enables StatsD logging; IPv4/IPv6
address or a hostname. If a
hostname resolves to an IPv4 and IPv6
address, the IPv4 address will be
used.
log_statsd_port 8125
log_statsd_default_sample_rate 1.0
log_statsd_sample_rate_factor 1.0
@ -526,9 +530,10 @@ set log_address /dev/log Logging directory
user swift User to run as
max_upload_time 86400 Maximum time allowed to upload an
object
slow 0 If > 0, Minimum time in seconds
for a PUT or DELETE request to
complete
slow 0 If > 0, Minimum time in seconds for a PUT or
DELETE request to complete. This is only
useful to simulate slow devices during testing
and development.
mb_per_sync 512 On PUT requests, sync file every
n MB
keep_cache_size 5242880 Largest object size to keep in
@ -719,6 +724,8 @@ log_facility LOG_LOCAL0 Syslog log facility
log_level INFO Logging level
log_address /dev/log Logging directory
log_time 3600 Frequency of status logs in seconds.
interval 30 Time in seconds to wait between
auditor passes
disk_chunk_size 65536 Size of chunks read during auditing
files_per_second 20 Maximum files audited per second per
auditor process. Should be tuned according
@ -787,7 +794,11 @@ log_custom_handlers None Comma-separated list of functions t
to setup custom log handlers.
log_udp_host Override log_address
log_udp_port 514 UDP log port
log_statsd_host localhost StatsD logging
log_statsd_host None Enables StatsD logging; IPv4/IPv6
address or a hostname. If a
hostname resolves to an IPv4 and IPv6
address, the IPv4 address will be
used.
log_statsd_port 8125
log_statsd_default_sample_rate 1.0
log_statsd_sample_rate_factor 1.0
@ -998,7 +1009,11 @@ log_custom_handlers None Comma-separated list of functions t
to setup custom log handlers.
log_udp_host Override log_address
log_udp_port 514 UDP log port
log_statsd_host localhost StatsD logging
log_statsd_host None Enables StatsD logging; IPv4/IPv6
address or a hostname. If a
hostname resolves to an IPv4 and IPv6
address, the IPv4 address will be
used.
log_statsd_port 8125
log_statsd_default_sample_rate 1.0
log_statsd_sample_rate_factor 1.0
@ -1226,7 +1241,11 @@ log_custom_handlers None Comma separated
handlers.
log_udp_host Override log_address
log_udp_port 514 UDP log port
log_statsd_host localhost StatsD logging
log_statsd_host None Enables StatsD logging; IPv4/IPv6
address or a hostname. If a
hostname resolves to an IPv4 and IPv6
address, the IPv4 address will be
used.
log_statsd_port 8125
log_statsd_default_sample_rate 1.0
log_statsd_sample_rate_factor 1.0
@ -1278,7 +1297,8 @@ object_chunk_size 65536 Chunk size to read from
client_chunk_size 65536 Chunk size to read from
clients
memcache_servers 127.0.0.1:11211 Comma separated list of
memcached servers ip:port
memcached servers
ip:port or [ipv6addr]:port
memcache_max_connections 2 Max number of connections to
each memcached server per
worker
@ -1469,7 +1489,7 @@ At Rackspace, our Proxy servers have dual quad core processors, giving us 8
cores. Our testing has shown 16 workers to be a pretty good balance when
saturating a 10g network and gives good CPU utilization.
Our Storage servers all run together on the same servers. These servers have
Our Storage server processes all run together on the same servers. These servers have
dual quad core processors, for 8 cores total. We run the Account, Container,
and Object servers with 8 workers each. Most of the background jobs are run at
a concurrency of 1, with the exception of the replicators which are run at a

36
doc/source/development_guidelines.rst

@ -83,15 +83,35 @@ For example, this command would run the functional tests using policy
SWIFT_TEST_POLICY=silver tox -e func
In-process functional testing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If the ``test.conf`` file is not found then the functional test framework will
instantiate a set of Swift servers in the same process that executes the
functional tests. This 'in-process test' mode may also be enabled (or disabled)
by setting the environment variable ``SWIFT_TEST_IN_PROCESS`` to a true (or
false) value prior to executing `tox -e func`.
When using the 'in-process test' mode, the optional in-memory
object server may be selected by setting the environment variable
``SWIFT_TEST_IN_MEMORY_OBJ`` to a true value.
When using the 'in-process test' mode some server configuration options may be
set using environment variables:
- the optional in-memory object server may be selected by setting the
environment variable ``SWIFT_TEST_IN_MEMORY_OBJ`` to a true value.
- the proxy-server ``object_post_as_copy`` option may be set using the
environment variable ``SWIFT_TEST_IN_PROCESS_OBJECT_POST_AS_COPY``.
For example, this command would run the in-process mode functional tests with
the proxy-server using object_post_as_copy=False (the 'fast-POST' mode)::
SWIFT_TEST_IN_PROCESS=1 SWIFT_TEST_IN_PROCESS_OBJECT_POST_AS_COPY=False \
tox -e func
This particular example may also be run using the ``func-in-process-fast-post``
tox environment::
tox -e func-in-process-fast-post
The 'in-process test' mode searches for ``proxy-server.conf`` and
``swift.conf`` config files from which it copies config options and overrides
@ -127,7 +147,7 @@ using config files found in ``$HOME/my_tests`` and policy 'silver'::
Coding Style
------------
Swift use flake8 with the OpenStack `hacking`_ module to enforce
Swift uses flake8 with the OpenStack `hacking`_ module to enforce
coding style.
Install flake8 and hacking with pip or by the packages of your
@ -164,6 +184,14 @@ Installing Sphinx:
#. Install sphinx (On Ubuntu: `sudo apt-get install python-sphinx`)
#. `python setup.py build_sphinx`
--------
Manpages
--------
For sanity check of your change in manpage, use this command in the root
of your Swift repo::
./.manpages
---------------------
License and Copyright

10
doc/source/development_saio.rst

@ -37,7 +37,8 @@ Installing dependencies
sudo apt-get update
sudo apt-get install curl gcc memcached rsync sqlite3 xfsprogs \
git-core libffi-dev python-setuptools
git-core libffi-dev python-setuptools \
liberasurecode-dev
sudo apt-get install python-coverage python-dev python-nose \
python-xattr python-eventlet \
python-greenlet python-pastedeploy \
@ -48,7 +49,8 @@ Installing dependencies
sudo yum update
sudo yum install curl gcc memcached rsync sqlite xfsprogs git-core \
libffi-devel xinetd python-setuptools \
libffi-devel xinetd liberasurecode-devel \
python-setuptools \
python-coverage python-devel python-nose \
pyxattr python-eventlet \
python-greenlet python-paste-deploy \
@ -585,3 +587,7 @@ doesn't work, here are some good starting places to look for issues:
cannot rate limit (unit tests generate a lot of logs very quickly).
Open the file ``SWIFT_TEST_CONFIG_FILE`` points to, and change the
value of ``fake_syslog`` to ``True``.
#. If you encounter a ``401 Unauthorized`` when following Step 12 where
you check that you can ``GET`` account, use ``sudo service memcached status``
and check if memcache is running. If memcache is not running, start it using
``sudo service memcached start``. Once memcache is running, rerun ``GET`` account.

12
doc/source/howto_installmultinode.rst

@ -3,31 +3,31 @@ Instructions for a Multiple Server Swift Installation
=====================================================
Please refer to the latest official
`Openstack Installation Guides <http://docs.openstack.org/#install-guides>`_
`OpenStack Installation Guides <http://docs.openstack.org/#install-guides>`_
for the most up-to-date documentation.
Object Storage installation guide for Openstack Liberty
----------------------------------------------------
Object Storage installation guide for OpenStack Liberty
-------------------------------------------------------
* `openSUSE 13.2 and SUSE Linux Enterprise Server 12 <http://docs.openstack.org/liberty/install-guide-obs/swift.html>`_
* `RHEL 7, CentOS 7 <http://docs.openstack.org/liberty/install-guide-rdo/swift.html>`_
* `Ubuntu 14.04 <http://docs.openstack.org/liberty/install-guide-ubuntu/swift.html>`_
Object Storage installation guide for Openstack Kilo
Object Storage installation guide for OpenStack Kilo
----------------------------------------------------
* `openSUSE 13.2 and SUSE Linux Enterprise Server 12 <http://docs.openstack.org/kilo/install-guide/install/zypper/content/ch_swift.html>`_
* `RHEL 7, CentOS 7, and Fedora 21 <http://docs.openstack.org/kilo/install-guide/install/yum/content/ch_swift.html>`_
* `Ubuntu 14.04 <http://docs.openstack.org/kilo/install-guide/install/apt/content/ch_swift.html>`_
Object Storage installation guide for Openstack Juno
Object Storage installation guide for OpenStack Juno
----------------------------------------------------
* `openSUSE 13.1 and SUSE Linux Enterprise Server 11 <http://docs.openstack.org/juno/install-guide/install/zypper/content/ch_swift.html>`_
* `RHEL 7, CentOS 7, and Fedora 20 <http://docs.openstack.org/juno/install-guide/install/yum/content/ch_swift.html>`_
* `Ubuntu 14.04 <http://docs.openstack.org/juno/install-guide/install/apt/content/ch_swift.html>`_
Object Storage installation guide for Openstack Icehouse
Object Storage installation guide for OpenStack Icehouse
--------------------------------------------------------
* `openSUSE and SUSE Linux Enterprise Server <http://docs.openstack.org/icehouse/install-guide/install/zypper/content/ch_swift.html>`_

0
doc/source/images/ec_overview.png

Before

Width:  |  Height:  |  Size: 145 KiB

After

Width:  |  Height:  |  Size: 145 KiB

13
doc/source/index.rst

@ -86,10 +86,15 @@ Administrator Documentation