450 Commits

Author SHA1 Message Date
Donagh McCabe
34e2ab3f31 account-reaper warns if not making progress
DocImpact
If account reaper has not managed to clean out an account after a long
period, it prints a message to the log (you can search your system looking
for such messages). Introduce reap_warn_after config variable to determine
when to emit the message (defaults to 30 days).

Also fix bug 1181995 (edge case where object name is an empty string)

Change-Id: Ic0dfee04742d06b6a51b59f302d7a272d7c1de92
2013-05-22 15:07:17 +01:00
Jenkins
53f3c61adf Merge "Fix two documentation warnings." 2013-05-21 01:04:41 +00:00
Darrell Bishop
4f2f2435ac Fix two documentation warnings.
The crossdomain doc was named *.xml instead of *.rst causing it to not
get built or included in the toctree where it was supposed to.

The apache deployment guide wasn't linked to from anywhere, so I added
it under the normal deployment guide.

Change-Id: I817a1f2ca1ed7913e8ea5155cc1fac07caf0b637
2013-05-11 14:31:42 -07:00
Clay Gerrard
34f5085c3e conf.d support
Allow Swift daemons and servers to optionally accept a directory as the
configuration parameter.  Directory based configuration leverages
ConfigParser's native multi-file support.  Files ending in '.conf' in the
given directory are parsed in lexicographical order.  Filenames starting with
'.' are ignored.  A mixture of file and directory configuration paths is not
supported - if the configuration path is a file behavior is unchanged.

 * update swift-init to search for conf.d paths when building servers
    (e.g. /etc/swift/proxy-server.conf.d/)
 * new script swift-config can be used to inspect the cumulative configuration
 * pull a little bit of code out of run_wsgi and test separately
 * fix example config bug for the proxy servers client_disconnect option
 * added section on directory based configuration to deployment guide

DocImpact

Implements: blueprint confd
Change-Id: I89b0f48e538117f28590cf6698401f74ef58003b
2013-04-30 00:17:46 -07:00
Jenkins
52a6595033 Merge "Add the max_clients parameter to bound clients" 2013-04-30 05:57:10 +00:00
Jenkins
d5e96e7747 Merge "Add crossdomain.xml middleware" 2013-04-27 08:56:19 +00:00
Peter Portante
2d42b37303 Add the max_clients parameter to bound clients
The new max_clients parameter allows one full control over the maximum
number of client requests that will be handled by a given worker for
any of the proxy, account, container or object servers.

Lowering the number of clients handled per worker, and raising the
number of workers can lessen the impact that a CPU intensive, or
blocking, request can have on other requests served by the same
worker.

If the maximum number of clients is set to one, then a given worker
will not perform another accept(2) call while processing, allowing
other workers a chance to process it.

DocImpact
Signed-off-by: Peter Portante <peter.portante@redhat.com>

Change-Id: Ic01430f7a6c5ff48d7aa349dc86a5f8ac463a420
2013-04-26 10:29:57 -04:00
Jenkins
2b7b69321d Merge "Add auth_version to dispersion.conf.5" 2013-04-26 02:40:29 +00:00
Donagh McCabe
eb4b29d243 Add crossdomain.xml middleware
Allows client-side technologies such as Flash, Java and Silverlight running
on web pages served elsewhere to interact with the Swift API.

Bug #1159960

Change-Id: I7d0533a0aaf189ac452abbd983469acb064fdca4
2013-04-23 13:52:34 +01:00
Pete Zaitcev
93a737b406 Add auth_version to dispersion.conf.5
The lack of any documentation for the auth_version was observed
by Red Hat QA, see rhbz#955247.

Change-Id: I0afc04d848082a80480724bdbf99031e4b99e38e
2013-04-22 11:31:21 -06:00
Sergey Kraynev
ea7858176b Implementation of replication servers
Support separate replication ip address:
- Added new function in utils. This function provides ability
  to select separate IP address for replication service.
- Db_replicator and object replicators were changed.
  Replication process uses new function now.

Replication network parameters:
- Replication network fields (replication_ip, replication_port)
  support was added to device dictionary in swift-ring-builder script.
- Changes were made to support new fields in search, show and set_info
  functions.

Implementation of replication servers:
- Separate replication servers use the same code as normal replication
  servers, but with replication_server parameter = True.  When using a
  separate replication network, the non-replication servers set
  replication_server = False.  When there is no separate replication
  network (the default case), replication_server is not included in the config.

DocImpact
Change-Id: Ie9af5bdcdf9241c355e36053ca4adfe49dc35bd0
Implements: blueprint dedicated-replication-network
2013-04-21 18:14:42 -04:00
David Hadas
39fc4c37bf Doc changes for new pip-requirements
dnspython was added recently to the list of requirments.

This patch completes the change:
1. The new requirement is documented
2. The need for tox --recreate is documented
(assuming there is no way to automate 2 above)

Change-Id: I49f0cb4b01b79dd7cf4329984a2f8416378ebe19
Fixes: Bug #1176627
2013-05-06 19:21:53 +03:00
Jenkins
a186c53b24 Merge "fix slo docs" 2013-04-17 19:39:43 +00:00
Jenkins
dfcfb767d3 Merge "Add x-remove-versions-location for feature: disable versioning" 2013-04-17 17:57:53 +00:00
David Goetz
d2a4f75a95 fix slo docs
Change-Id: I49f8c272457547d8aae853986573549828d2503a
2013-04-17 07:03:46 -07:00
Jenkins
8ddfa5e533 Merge "Allow all headers requested for CORS." 2013-04-16 15:56:58 +00:00
Jenkins
c87576bb94 Merge "Refactored lists of nodes to contact for requests" 2013-04-15 23:05:47 +00:00
Jenkins
5140c0d5da Merge "Adding a new optional variable called trans_id_suffix" 2013-04-10 20:05:03 +00:00
Marcelo Martins
1126e59c12 Adding a new optional variable called trans_id_suffix
The trans_id_suffix (default is empty) would be appended to the swift transaction
id allowing one to easily figure out from which cluster that X-Trans-Id
belongs to. This is very useful when one is managing more than one swift
cluster. Also updated sample and manpage to reflect the changes.

Change-Id: Icdf63643e9c1bde36a9ef5e3f41ee9fb20e55f5d
2013-04-10 06:37:32 -05:00
gholt
d79a67ebf6 Refactored lists of nodes to contact for requests
Extensive refactor here to consolidate what nodes are contacted for
any request. This consolidation means reads will contact the same set
of nodes that writes would, giving a very good chance that
read-your-write behavior will succeed. This also means that writes
will not necessarily try all nodes in the cluster as it would
previously, which really wasn't desirable anyway. (If you really want
that, you can set request_node_count to a really big number, but
understand that also means reads will contact every node looking for
something that might not exist.)

* Added a request_node_count proxy-server conf value that allows
  control of how many nodes are contacted for a normal request.

In proxy.controllers.base.Controller:

* Got rid of error_increment since it was only used in one spot by
  another method and just served to confuse.

* Made error_occurred also log the device name.

* Made error_limit require an error message and also documented a bit
  better.

* Changed iter_nodes to just take a ring and a partition and yield
  all the nodes itself so it could control the number of nodes used
  in a given request. Also happens to consolidate where sort_nodes is
  called.

* Updated account_info and container_info to use all nodes from
  iter_nodes and to call error_occurred appropriately.

* Updated GETorHEAD_base to not track attempts on its own and just
  stop when iter_nodes tells it to stop. Also, it doesn't take the
  nodes to contact anymore; instead it takes the ring and gets the
  nodes from iter_nodes itself.

Elsewhere:

* Ring now has a get_part method.

* Made changes to reflect all of the above.

Change-Id: I37f76c99286b6456311abf25167cd0485bfcafac
2013-04-08 20:48:32 +00:00
Chmouel Boudjnah
c687f6956c Allow all headers requested for CORS.
- We allow all headers requested in preflight request. The CORS
  specification does leave the door open for this, as mentioned in
  http://www.w3.org/TR/cors/#resource-preflight-requests
  Note: Since the list of headers can be unbounded
  simply returning headers can be enough.
- This is a followup to review:
  https://review.openstack.org/#/c/24415/.
- Fixes bug 1155034.

Change-Id: If7b8f2f3a581c5209892d1ccc9f06ddb8fac92dd
2013-04-05 21:12:15 +01:00
tong li
f1aefcbc13 directory /var/cache/swift /srv/node should be created.
For multiple node swift cluster setup, these directories should be
created in the setup steps. Otherwise, when start up storage servers
there will be errors. The error messages won't tell if it is because
of missing the directory.

Change-Id: I67d74abb7743b76739b5e747d6dcbd3214b00774
Fixes: bug #1158519
2013-04-04 20:39:01 +00:00
Christian Schwede
e5cd78d902 Update documentation for account_quota middleware
Change-Id: I3dc7930ee2a1eb26b1f60e83fce2bc1bea0a8b0f
2013-04-03 15:10:42 +02:00
David Hadas
a979c8007b Add support for Hash Prefix
A new configuration parameter is added to /etc/swift/swift.conf
[swift-hash]
swift_hash_path_prefix = 'random unique string'

New installations are advised to set this parameter to a random secret,
which would not be disclosed ouside the organization.
The same secret needs to be used by all swift servers of the same cluster.

Existing installations should set this parameter to an empty string
(the default)

DocImpact

Fixes: Bug #1157454

Change-Id: I63b10d0b7d6dd3f74e0f10bb41b5f240fa03578a
2013-03-22 19:41:55 +02:00
tong li
c1dfe891e7 instruction command was not documented in right format.
Instructions for a Multiple Server Swift Installation (Ubuntu) doc
for creating /var/run/swift and changing owner command were not
using the right format. Missing two colons.

Change-Id: Ie23007a0da498373fbfb137c7edb3d80813c6ba5
Fixes: bug #1158310
2013-03-21 09:45:13 -04:00
Jenkins
85b7346808 Merge "Improved container-sync resiliency" 2013-03-19 07:20:57 +00:00
Jenkins
591a3a599d Merge "Small clarification to swift-recon section of admin guide." 2013-03-18 23:29:15 +00:00
tong li
68355d7ea5 Instruction for multi node setup misses /var/run/swift creation
Multiple Server Swift Installation (Ubuntu) instruction does not
indicate that the directory /var/run/swift needs to be created.
That directory actually needs to be created and the ownship needs
to be changed to the user/group which swift service runs under.

This patch will fix the document and gives the steps how to create
the directory and set the ownership right. It also gives instruction
on how the script can be added so that swift services can be resarted
after system reboots.

Change-Id: Id61aa67cc0d6f66d749701e6ea824b1ff3b2c478
Fixes: bug #1156631
2013-03-18 13:57:40 -04:00
Samuel Merritt
d88d12b120 Small clarification to swift-recon section of admin guide.
Apparently the use of port 6030 was causing some confusion.

Fixes bug 1095474.

Change-Id: I0cc71f4733ad91694e015a9b75c3eda080aca6fb
2013-03-17 15:58:06 -07:00
Kun Huang
5c3f75ba6a Add x-remove-versions-location for feature: disable versioning
We can set x-versions-location empty to remove this header in API, but
to keep consistent, we should allow x-remove-versions-location too. The
usage is post "x-remove-versions-locaion: x", just like ACL remove
headers.

Fixed: bug #1107592

Change-Id: I1271eec6401d4a0e8c1a7c2d63aeb8dfef00bf6d
2013-03-17 07:30:00 +08:00
Jenkins
94defcbc53 Merge "Add tests and doc entry for request.environ[reseller_request]" 2013-03-16 19:29:47 +00:00
Samuel Merritt
ebcd60f7d9 Add a region tier to Swift's ring.
The region is one level above the zone; it is intended to represent a
chunk of machines that is distant from others with respect to
bandwidth and latency.

Old rings will default to having all their devices in region 1. Since
everything is in the same region by default, the ring builder will
simply distribute across zones as it did before, so your partition
assignment won't move because of this change. If you start adding
devices in other regions, of course, the assignment will change to
take that into account.

swift-ring-builder still accepts the same syntax as before, but will
default added devices to region 1 if no region is specified.

Examples:

$ swift-ring-builder foo.builder add r2z1-1.2.3.4:555/sda

$ swift-ring-builder foo.builder add r1z3-1.2.3.4:555/sda

$ swift-ring-builder foo.builder add z3-1.2.3.4:555/sda

Also, some updates to ring-overview doc.

Change-Id: Ifefbb839cdcf033e6c9201fadca95224c7303a29
2013-03-13 10:00:58 -07:00
Christian Schwede
157c3c91ee Add tests and doc entry for request.environ[reseller_request]
The recent account_quotas (https://review.openstack.org/23434)
patch added a new setting request.environ[reseller_request].
This patch adds tests for tempauth and keystoneauth as well as
an updated overview_auth.rst.

Change-Id: Icdb7ec9948ae7424b0721fc51a143782b2fdc5a6
2013-03-10 19:21:13 +01:00
David Hadas
8b140033f0 Improved container-sync resiliency
container-sync now skips faulty objects in the first and second rounds.
All replicas try in the second round.
No server will give up until the faulty object suceeds

Fixes: bug #1068423

Change-Id: I0defc174b2ce3796a6acf410a2d2eae138e8193d
2013-03-08 22:28:06 +02:00
Christian Schwede
28c75db0e7 Account quotas
Add a new middleware implementing account quotas.

This middleware blocks write requests (PUT, POST) if a given quota (in bytes)
is exceeded while DELETE requests are still allowed.

Quotas are stored in the x-account-meta-quota-bytes metadata entry.
Write requests to this metadata setting are only allowed for resellers.

Change-Id: I57fd7c6209f34cc79d4bab72d500d43ba2a62083
2013-03-08 14:31:35 +01:00
Jenkins
09f1387af5 Merge "Support tests for Apache" 2013-03-07 13:33:48 +00:00
David Hadas
40782ed20c Support tests for Apache
Add support for functional tests that work with Apache web front end

Change-Id: I72358a12016eeccc842d834461dbebaa188aa117
Implements: blueprint wsgi-application-interface
2013-03-07 01:32:16 +02:00
Sergey Lukjanov
7d5095c122 Support listing endpoints for an object.
Implements blueprint list-endpoints.

DocImpact: new middleware list_endpoints.

Change-Id: I0c4911ff726abd4cb8ce2b6245c99786ad46b410
2013-03-07 01:38:21 +04:00
Jenkins
28ac46ded3 Merge "Added per disk PUT timing monitoring support." 2013-03-06 01:59:23 +00:00
David Goetz
5d73da158b Static Large Object Support
DocImpact

Change-Id: I7edaa5e44208ab451f7f7566b64bb571b8eea1f9
2013-03-01 16:46:10 -08:00
Jian Zhang
1d8a02f25c Added per disk PUT timing monitoring support.
Fixes bug 1104708

There could be severe performance drop for swift is one disk of one
storage node is problematic due to the tragic state of async disk I/O.

This patch provided PUT timing per kB transfered (ms/kB) monitoring
support for each non-zero-byte request of each disk and report to
statsD for alert.
-adding "object-server.PUT.<device>.timing" metrics for object-server.

DocImpact.

Change-Id: Ie94bddad28e8be52e71683bf6c9db988664abe47
2013-02-28 02:52:06 -08:00
Jenkins
2a119b650e Merge "Move the swift.1 manpage to python-swiftclient" 2013-02-27 16:50:17 +00:00
Jenkins
a06c71c624 Merge "Add cache=swift.cache for authtoken example." 2013-02-27 16:10:57 +00:00
Pete Zaitcev
9a3e0f41bd Move the swift.1 manpage to python-swiftclient
When someone split away python-swiftclient, he left behind the
client manpage.

Apparently all we need now is to delete it from swift and
add it to python-swiftclient. Except that of course the releases
have to be synchronized somehow or else files conflict.

Change-Id: I8b5f7b5557c28818048d8941df36473dacfb1d57
2013-02-26 12:41:27 -07:00
Jenkins
249a65461e Merge "Adding speed limit options for DB auditor" 2013-02-26 06:22:25 +00:00
Jenkins
d1381ba86b Merge "Adds first-byte latency timings for GET requests." 2013-02-23 01:10:58 +00:00
Chmouel Boudjnah
e69f3bef8f Add cache=swift.cache for authtoken example.
- Things swill go badly with swift if we leave the default to authtoken
  to use its own memcache cache connection based python-memcache c based
  binding.

Change-Id: I293b875acdcb06e5a7a0cfa9a9bb5d7678675da0
2013-02-21 22:58:27 +01:00
yuan-zhou
09370862ca Adding speed limit options for DB auditor
Fix bug 1129760

Without speed limit, DB auditor will likely consume high CPU% on
storage node. That will highly impact the cluster's performance.

This patch adds two options for account/container auditor:
 - containers_per_second: Maximum containers audited per second
 - accounts_per_second: Maximum accounts audited per second

DocImpact

Change-Id: I9faa506438185a83ca77db4906969328624d015f
2013-02-20 13:54:59 +08:00
Pete Zaitcev
93ea7c63b1 Documentation fixups
These are mostly cosmetic fixes for irritating imperfections:
- "separated with commas" was duplicated, leave just one
- extra whitespace here and there, man pages are not PEP8, drop
- weird extra commas, drop
- Fedora logs to /var/log/messages
- "drive is has failed", drop "is"

Change-Id: I5ceba2e61b16db4855d76c92cbc83663b9b2a0da
2013-02-18 10:54:27 -07:00
Chuck Thier
e88ff34685 Cleanup of file permissions
Mostly removed execute on non-executable files

Change-Id: Ibfbe7e0cf0fbeabef602d70b20f75e1dd3bdf9c9
2013-02-15 11:06:11 -06:00