This change adds the ability to tell the container or account server to
reverse their listings. This is done by sending a reverse=TRUE_VALUE,
Where TRUE_VALUE is one of the values true can be in common/utils:
TRUE_VALUES = set(('true', '1', 'yes', 'on', 't', 'y'))
For example:
curl -i -X GET -H "X-Auth-Token: $TOKEN" $STORAGE_URL/c/?reverse=on
I borrowed the swapping of the markers code from Kevin's old change,
thanks Kevin. And Tim Burke added some real nuggets of awesomeness.
DocImpact
Co-Authored-By: Kevin McDonald <kmcdonald@softlayer.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Implements: blueprint reverse-object-listing
Change-Id: I5eb655360ac95042877da26d18707aebc11c02f6
a1c32702, 736cf54a, and 38787d0f remove uses of `simplejson` from
various parts of Swift in favor of the standard libary `json`
module (introduced in Python 2.6). This commit performs the remaining
`simplejson` to `json` replacements, removes two comments highlighting
quirks of simplejson with respect to Unicode, and removes the references
to it in setup documentation and requirements.txt.
There were a lot of places where we were importing json from
swift.common.utils, which is less intuitive than a direct `import json`,
so that replacement is made as well.
(And in two more tiny drive-bys, we add some pretty-indenting to an XML
fragment and use `super` rather than naming a base class explicitly.)
Change-Id: I769e88dda7f76ce15cf7ce930dc1874d24f9498a
Added region prefix to example commands for adding devices to ring.
Also updates description to include region prefix.
Change-Id: Ie6d6485b497cea973e37909b5b19b44946c8aa89
The doc for these sections was missing because of an rst error - the
source is there in rst file but didn't make it into the html output.
Add doc for per_diff and max_diffs in account and container doc sections.
Also, fix a bunch of other sphinx build errors and most of the warnings.
Change-Id: If9ed2619b2f92c6c65a94f41d8819db8726d3893
This change moves the RST DLO documentation from
statically inside overview_large_objects.rst and moves it
to middleware/dlo.py.
This is where all middleware RST documentation is defined.
The overview_large_objects.rst is still the main page
for information on large objects, so now dynamically
points to both the DLO and SLO middleware RST
documentation and the relevant middleware.rst page
simply points to it.
Change-Id: I40d918c8b7bc608ab945805d69fe359521df038a
Closes-bug: #1276852
Update the doc to reflect the change [1] to ndata + 1
.durable files being committed before a success response
is returned for a PUT.
[1] Ifd36790faa0a5d00ec79c23d1f96a332a0ca0f0b
Change-Id: I1744d457bda8a52eb2451029c4031962e92c2bb7
Update the EC overview docs 'under the hood' section to reflect the
change in durable file parity from 2 to ec_nparity + 1 [1].
Also fix some typos and cleanup the text.
[1] change id I80d666f61273e589d0990baa78fd657b3470785d
Change-Id: I23f6299da59ba8357da2bb5976d879d9a4bb173e
This doc is being updated to specify the replicas count parameter
to build an EC ring that enforces both data and parity placements
for each partition.
Change-Id: I770ad268e4017e610be3357e89b89f0b7d3c18af
Closes-Bug: 1487203
Currently, the rsync module where the replicators send data is static. It
forbids administrators to set rsync configuration based on their current
deployment or needs.
As an example, the rsyncd configuration example encourages to set a connections
limit for the modules account, container and object. It permits to protect
devices from excessives parallels connections, because it would impact
performances.
On a server with many devices, it is tempting to increase this number
proportionally, but nothing guarantees that the distribution of the connections
will be balanced. In the worst scenario, a single device can receive all the
connections, which is a severe impact on performances.
This commit adds a new option named 'rsync_module' to the *-replicator sections
of the *-server configuration file. This configuration variable can be
extrapolated with device attributes like ip, port, device, zone, ... by using
the format {NAME}. eg:
rsync_module = {replication_ip}::object_{device}
With this configuration, an administrators can solve the problem of connections
distribution by creating one module per device in rsyncd configuration.
The default values are backward compatible:
{replication_ip}::account
{replication_ip}::container
{replication_ip}::object
Option vm_test_mode is deprecated by this commit, but backward compatibility is
maintained. The option is only effective when rsync_module is not set. In that
case, {replication_port} is appended to the default value of rsync_module.
Change-Id: Iad91df50dadbe96c921181797799b4444323ce2e
SAIO Configuration and documentation changes enabling to run the
container sync probe test by default
Change-Id: Iccf59533d0d4fe72549d318339ab125d04dde006
Related-Bug: #1476623
Currently, statistics are organized by command. However, it would also be
useful to display statistics organized by policy. Different policies may be
based on different storage properties (ie, faster disks).
With this change, all the statistics for object timers will be sent per policy
as well.
Policy statistics reporting will use policy index and the name in Graphite will
show as proxy-server.object.policy.<policy-index>.<verb>, etc.
Updated unit tests for per-policy stat reporting and added new unit tests for
invalid cases.
Updated documentation in the Administrator's Guide to reflect this new
aggregation.
Change-Id: Id70491e4833791a3fb8ff385953d69018514cd9c
Improve the 'first_contribution_swift' by adding instructions for how to
rebuild swift following a rebase.
Change-Id: If5c91dc4e1e8d1712bbd8b326c675967fb4b8c15
This patch makes the count of object replication failure in recon.
And "failure_nodes" is added to Account Replicator and
Container Replicator.
Recon shows the count of object repliction failure as follows:
$ curl http://<ip>:<port>/recon/replication/object
{
"replication_last": 1416334368.60865,
"replication_stats": {
"attempted": 13346,
"failure": 870,
"failure_nodes": {
"192.168.0.1": {"sdb1": 3},
"192.168.0.2": {"sdb1": 851,
"sdc1": 1,
"sdd1": 8},
"192.168.0.3": {"sdb1": 3,
"sdc1": 4}
},
"hashmatch": 0,
"remove": 0,
"rsync": 0,
"start": 1416354240.9761429,
"success": 1908
},
"replication_time": 2316.5563162644703,
"object_replication_last": 1416334368.60865,
"object_replication_time": 2316.5563162644703
}
Note that 'object_replication_last' and 'object_replication_time' are
considered to be transitional and will be removed in the subsequent
releases. Use 'replication_last' and 'replication_time' instead.
Additionaly this patch adds the count in swift-recon and it will be
showed as follows:
$ swift-recon object -r
========================================================================
=======
--> Starting reconnaissance on 4 hosts
========================================================================
=======
[2014-11-27 16:14:09] Checking on replication
[replication_failure] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%,
no_result: 0, reported: 4
[replication_success] low: 3, high: 3, avg: 3.0, total: 12,
Failed: 0.0%, no_result: 0, reported: 4
[replication_time] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%,
no_result: 0, reported: 4
[replication_attempted] low: 1, high: 1, avg: 1.0, total: 4,
Failed: 0.0%, no_result: 0, reported: 4
Oldest completion was 2014-11-27 16:09:45 (4 minutes ago) by
192.168.0.4:6002.
Most recent completion was 2014-11-27 16:14:19 (-10 seconds ago) by
192.168.0.1:6002.
========================================================================
=======
In case there is a cluster which has servers, a server runs with this
patch and the other servers run without this patch. If swift-recon
executes on the server which runs with this patch, there are unnecessary
information on the output such as [failure], [success] and [attempted].
Because other servers which run without this patch are not able to
send a response with information that this patch needs.
Therefore once you apply this patch, you also apply this patch to other
servers before you execute swift-recon.
DocImpact
Change-Id: Iecd33655ae2568482833131f422679996c374d78
Co-Authored-By: Kenichiro Matsuda <matsuda_kenichi@jp.fujitsu.com>
Co-Authored-By: Brian Cline <bcline@softlayer.com>
Implements: blueprint enable-object-replication-failure-in-recon
Rewrite object versioning as middleware to simplify the PUT method
in the object controller.
The functionality remains basically the
same with the only major difference being the ability to now
version slo manifest files. dlo manifests are still not
supported as part of this patch.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
DocImpact
Change-Id: Ie899290b3312e201979eafefb253d1a60b65b837
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Signed-off-by: Prashanth Pai <ppai@redhat.com>
When rsync pushes to a remote node with an unmounted drive and if
certain steps are not taken, rsync may attempt to write files to
the local drive at the location where the drive was mounted.
There are two suggested solutions for this issue:
1) Set the permissions for all mount points in /srv/node/
to root:root 755
2) Mount the drives elsewhere and symlink the drives to /srv/.../
The first method ensures that only root and not the swift user
can write in the /srv/.../ directories.
The second method will prompt a broken link issue if rsync
attempts to write to an unmounted drive.
Change-Id: I60ce4ed9ef8401768d5f78b6806cbb2e2a65303e
Closes-Bug: #1470576
In this change, there are proposed improvement to Swift documentation aimed at
first-time contributors. They include a simplification of the Getting
Started page and a new page with some basic instructions/commands that the
first-time contributor should know. In addition, it shows some common errors
that the first-time contributor may find when executing git rebase and
information on how to track your changes.
Change-Id: I704202955093736b2f3b4102a649690a0392c6b0
