Commit Graph

84 Commits

Author SHA1 Message Date
Cedric Dos Santos
a8751ae557 Correct misspelled words
In some files I found misspelling words.

bin/swift-reconciler-enqueue#l26
   primarly => primarily
swift/account/backend.py#l309
   ommited => omitted
swift/container/replicator.py#l158
   successfull => successful
test/unit/account/test_backend.py#1450
   non_existant_policy_index => non_existent_policy_index
test/unit/account/test_backend.py#1451
   'test-non-existant-policy'=> 'test-non-existent-policy'
test/unit/account/test_backend.py#1453
   non_existant_policy_index => non_existent_policy_index

Change-Id: I976236e3200a6fbdc20be464acff182b6cface81
2014-11-25 15:44:30 +00:00
Jenkins
813cc0fe0d Merge "Add container_count to policy_stat table" 2014-09-25 05:12:33 +00:00
paul luse
15fbf9fe7c Add container_count to policy_stat table
Start tracking the container count per policy including reporting
it in account HEAD and supporting installations where the DB
existed before the updated schema.

Migration is triggered by the account audtior; if the database is
un-migrated it will continue to report policy_stats without the per
policy container_count keys.

Closes-Bug: #1367514
Change-Id: I07331cea177e19b3df303609a4ac510765a19162
2014-09-24 08:39:01 -07:00
Samuel Merritt
1678083d0e Test for unicode names in AccountBroker.merge_items()
ContainerBroker.merge_items() had a bug in it where non-ASCII Unicode
names would possibly result in duplicate entries in container
databases. AccountBroker.merge_items() doesn't do the same
bulk-operations tricks that ContainerBroker does, so it doesn't
currently have the bug. This commit just adds a test to ensure the bug
doesn't creep in should someone decide to make AccountBroker look more
like ContainerBroker someday.

Change-Id: Id2ac129828dbdf55b609d839ce4d9d42437ee0a3
2014-09-18 17:04:30 -07:00
saranjan
2a8b43e5e7 Spelling mistakes corrected in comments.
Change-Id: Ibbd7511c3a2b08519feb4db18eca6e000603ea32
2014-09-03 10:40:30 -07:00
Jenkins
75d9533466 Merge "Remove fake _get_part and use the real thing" 2014-08-22 01:50:33 +00:00
Jenkins
9de6b3e33a Merge "Add timestamp checking in AccountBroker.is_status_deleted" 2014-08-15 17:31:41 +00:00
Jenkins
cd0e2c23ef Merge "Increase Account Auditor Test Code Coverage" 2014-08-15 03:34:59 +00:00
Takashi Kajinami
41d851387c Add timestamp checking in AccountBroker.is_status_deleted
Account-reaper works only at account-server with the first replica, and reaps
account with "deleted" status.

On the other hand, account-replicator doesn't replicate the status, only
replicates *_timestamp.
When swift fails to delete the first account replica, account-reaper never
reaps the account, because the first replica never gets marked as "deleted".

This patch adds a timestamp checking into is_status_deleted method, and
account-reaper will start to reap the account after account-replicator
replicates *_timestamp.

Change-Id: I75e3f15ad217a71b4fd39552cf6db2957597efca
Closes-Bug: #1304755
2014-08-12 12:31:05 +09:00
Jenkins
3a723e2c22 Merge "added process pid to the end of storage node log lines" 2014-07-11 19:45:57 +00:00
John Dickinson
7ab2afe5bd added process pid to the end of storage node log lines
Change-Id: I1c2709d85575fc7d4868fafd9ee757fd01868436
2014-07-09 12:12:33 -07:00
Clay Gerrard
e114a628dc Remove fake _get_part and use the real thing
The get_part method is fast and stable given a consistent hash path
suffix/prefix, so there's no absolute requirement for the fake
implementation other than convenience.  OTOH, removing the fake
implementation and fixing the tests that were relying on it should make
it easier to write better tests going forward and harder to hide bugs
that don't show up when using the fakes.

There may be some overhead when writing new tests that use the ring if
you're making assertions on partitions or paths, but with a part power
of zero it's normally trivially obvious when a 1 needs to be a 0 or vice
versa.  Or you can just drop the assertions around the parts you were
faking anyway.

Change-Id: I8bfc388a04eff6491038991cdfd7686c9d961545
2014-07-08 15:57:43 +00:00
Christian Berendt
f6ff06b678 Use except x as y instead of except x, y
According to https://docs.python.org/3/howto/pyporting.html the
syntax changed in Python 3.x. The new syntax is usable with
Python >= 2.6 and should be preferred to be compatible with Python3.

Enabled hacking check H231.

Change-Id: I2c41dc3ec83e79181e8fd50e76771a74c393269c
2014-07-07 15:42:13 -07:00
Paul Luse
51fa32f53d Increase Account Auditor Test Code Coverage
Some simple additional test code to bring coverage from 82% to 99%

Change-Id: I561bc0a043f3451bb0a84cad274fedc907e61ee8
2014-07-04 09:48:25 -07:00
Paul Luse
873c52e608 Replace POLICY and POLICY_INDEX with string literals
Replaced throughout code base &  tox'd. Functional as well
as probe tests pass with and without policies defined.

POLICY --> 'X-Storage-Policy'
POLICY_INDEX --> 'X-Backend-Storage-Policy-Index'

Change-Id: Iea3d06de80210e9e504e296d4572583d7ffabeac
2014-06-23 12:52:50 -07:00
Clay Gerrard
c1dc2fa624 Add two vector timestamps
The normalized form of the X-Timestamp header looks like a float with a fixed
width to ensure stable string sorting - normalized timestamps look like
"1402464677.04188"

To support overwrites of existing data without modifying the original
timestamp but still maintain consistency a second internal offset
vector is append to the normalized timestamp form which compares and
sorts greater than the fixed width float format but less than a newer
timestamp.  The internalized format of timestamps looks like
"1402464677.04188_0000000000000000" - the portion after the underscore
is the offset and is a formatted hexadecimal integer.

The internalized form is not exposed to clients in responses from Swift.
Normal client operations will not create a timestamp with an offset.

The Timestamp class in common.utils supports internalized and normalized
formatting of timestamps and also comparison of timestamp values.  When the
offset value of a Timestamp is 0 - it's considered insignificant and need not
be represented in the string format; to support backwards compatibility during
a Swift upgrade the internalized and normalized form of a Timestamp with an
insignificant offset are identical.  When a timestamp includes an offset it
will always be represented in the internalized form, but is still excluded
from the normalized form.  Timestamps with an equivalent timestamp portion
(the float part) will compare and order by their offset.  Timestamps with a
greater timestamp portion will always compare and order greater than a
Timestamp with a lesser timestamp regardless of it's offset.  String
comparison and ordering is guaranteed for the internalized string format, and
is backwards compatible for normalized timestamps which do not include an
offset.

The reconciler currently uses a offset bump to ensure that objects can move to
the wrong storage policy and be moved back.  This use-case is valid because
the content represented by the user-facing timestamp is not modified in way.
Future consumers of the offset vector of timestamps should be mindful of HTTP
semantics of If-Modified and take care to avoid deviation in the response from
the object server without an accompanying change to the user facing timestamp.

DocImpact
Implements: blueprint storage-policies
Change-Id: Id85c960b126ec919a481dc62469bf172b7fb8549
2014-06-19 10:18:06 -07:00
Paul Luse
2e1ea825aa Add Storage Policy support to the Account Reaper
Extract X-Storage-Policy-Index header from container listing request
and use it when making direct object DELETE requests.

DocImpact
Implements: blueprint storage-policies
Change-Id: Icd4b2611b4169e46f216ff9a9839af732971a2bf
2014-06-18 21:09:54 -07:00
Pete Zaitcev
b02f0db126 Refactoring storage policies merge_timestamps
* base implementation of is_deleted phrased to use _is_deleted
 * wrap pre-conn coded _is_deleted inside a transation for merge_timestamps

Implements: blueprint storage-policies
Change-Id: I6a948908c3e45b70707981d87171cb2cb910fe1e
2014-06-18 20:57:09 -07:00
Paul Luse
00a162c4d4 Add Storage Policy Support to Accounts
This change updates the account HEAD handler to report out per
policy object and byte usage for the account.  Cumulative values
are still reported and policy names are used in the report
out (unless request is sent to an account server directly in
which case policy indexes are used for easier accounting).

Below is an example of the relevant HEAD response for a cluster
with 3 policies and just a few small objects:

   X-Account-Container-Count: 3
   X-Account-Object-Count: 3
   X-Account-Bytes-Used: 21
   X-Storage-Policy-Bronze-Object-Count: 1
   X-Storage-Policy-Bronze-Bytes-Used: 7
   X-Storage-Policy-Silver-Object-Count: 1
   X-Storage-Policy-Silver-Bytes-Used: 7
   X-Storage-Policy-Gold-Object-Count: 1
   X-Storage-Policy-Gold-Bytes-Used: 7

Set a DEFAULT storage_policy_index for existing container rows during
migration.

Copy existing object_count and bytes_used in policy_stat table during
migration.

DocImpact
Implements: blueprint storage-policies
Change-Id: I5ec251f9a8014dd89764340de927d09466c72221
2014-06-18 20:57:09 -07:00
Brian Cline
b4c5a13664 Uses None instead of mutables for function param defaults
As seen on #1174809, changes use of mutable types as default
arguments and defaults them within the method. Otherwise, those
defaults can be unexpectedly persisted with the function between
invocations and erupt into mass hysteria on the streets.

There was indeed a test (TestSimpleClient.test_get_with_retries)
that was erroneously relying on this behavior. Since previous tests
had populated their own instantiations with a token, this test only
passed because the modified headers dict from previous tests was
being overridden. As expected, with the mutable defaults fix in
SimpleClient, this test begain to fail since it never specified any
token, yet it has always passed anyway. This change also now provides
the expected token.

Change-Id: If95f11d259008517dab511e88acfe9731e5a99b5
Related-Bug: #1174809
2014-05-10 11:15:56 +00:00
Greg Lange
d32dc8d49c Unify backend logging
Make account, object, and container servers construct log lines using the
same utility function so they will produce identically formatted lines.

This change reorders the fields logged for the account server.

This change also adds the "additional info" field to the two servers that
didn't log that field.  This makes the log lines identical across all 3
servers.  If people don't like that, I can take that out.  I think it makes
the documentation, parsing of the log lines, and the code a tad cleaner.

DocImpact

Change-Id: I268dc0df9dd07afa5382592a28ea37b96c6c2f44
Closes-Bug: 1280955
2014-04-07 18:38:04 +00:00
Peter Portante
07fcf50c3a Rework use of constraints to ease testing
Prior to this patch both mainline code and testing modules imported
and used constraints directly into their own namespace, or relied on
the namespace of other modules that were not the constraints
module. This meant that if a unit test wanted to change a constraint
for its operation, it had to know how that module was using the
constraint, instead of referencing the constraint module itself.

This patch unifies the use of constraints so that all constraints are
referenced via the constraints module. In turn, this allows a test to
leverage the re-loadable nature of the constraints in the constraints
module.

It addition, a number of functional tests where using the default
values for constraints, instead of the configured value discovered in
a test.conf or in an existing swift.conf. This patch removes those
direct references in favor of the load_constraint() method from the
test/functional/tests.py module.

Change-Id: Ia5313d653c667dd9ca800786de59b59334c34eaa
2014-04-02 23:48:01 -04:00
Pete Zaitcev
a7cfcc3d7a Relocate DATADIR to backends
It simply makes sense that the definition of DATADIR belongs to
backends. After all, some of them may not even have any.

Coincidentially, a few unnecessary imports are dropped.

By the way, on the object server side, diskfile.py provides DATADIR
in the same way already.

Change-Id: I60bfd522c77c4a0ee13697a2e31141777c7e2398
2014-04-01 23:22:22 -06:00
Peter Portante
6145c57f4a Make initialization test failure more explicit
A few changes to make the db initialization test failure mode more
explicit.

Change-Id: I533cd33476a57ecb854a1af1a5fb170c65daa825
2014-03-25 17:00:44 -04:00
Samuel Merritt
09ef06fd99 Convert all old-style classes to new-style
This cleanup has been slowly happening for a while; let's finish it.

Change-Id: I1561e3540d524834e0cc5bc725ab80936eae1f0e
2014-03-03 17:28:48 -08:00
Christian Schwede
b24a7dd4cb Allow log disable in account/container server
Logging can be disabled in the object server, but not in the
account and container server though both sample configs already
include that setting. This patch allows to disable logging for
account and container server as well.

Functionality is ensured by some additional tests.

Closes-Bug: 1280954
Change-Id: Ia4e87911863089c3bea093d9f181ff29e9e963eb
2014-02-17 19:38:26 +00:00
Peter Portante
ca87827db9 Use a tempfile.mkdtemp() based temporary directory
Change-Id: Ie0e8137615348b130b67323d0d2913dc5ebfd5fb
2014-01-19 16:29:45 -05:00
Jenkins
f310006fae Merge "Generic means for persisting system metadata." 2014-01-10 14:44:59 +00:00
anc
6164fa246d Generic means for persisting system metadata.
Middleware or core features may need to store metadata
against accounts or containers. This patch adds a
generic mechanism for system metadata to be persisted
in backend databases, without polluting the user
metadata namespace, by using the reserved header
namespace x-<server_type>-sysmeta-*.

Modifications are firstly that backend servers persist
system metadata headers alongside user metadata and
other system state.

For accounts and containers, system metadata in PUT
and POST requests is treated in a similar way to user
metadata. System metadata is not yet supported for
object requests.

Secondly, changes in the proxy controllers ensure that
headers in the system metadata namespace will pass through
in requests to backend servers.

Thirdly, system metadata returned from backend servers
in GET or HEAD responses is added to the cached info
dict, which middleware can access.

Finally, a gatekeeper middleware module is provided
which filters all system metadata headers from requests
and responses by removing headers with names starting
x-account-sysmeta-, x-container-sysmeta-. The gatekeeper
also removes headers starting x-object-sysmeta- in
anticipation of future support for system metadata being
set for objects. This prevents clients from writing or
reading system metadata.

The required_filters list in swift/proxy/server.py is
modified to include the gatekeeper middleware so that
if the gatekeeper has not been configured in the
pipeline then it will be automatically inserted close
to the start of the pipeline.

blueprint cluster-federation

Change-Id: I80b8b14243cc59505f8c584920f8f527646b5f45
2014-01-06 22:29:37 +00:00
Chmouel Boudjnah
150f338fc2 Remove swiftclient dep on direct_client
Partial Implements: blueprint remove-swiftclient-dependency
Change-Id: I9af7150e5d21d50e5f880e57796314b8f05822d2
2013-12-24 03:11:43 -08:00
Samuel Merritt
b5b0b78fc7 Remove obsolete future imports
The with statement has been standard since Python 2.5, so we can get
rid of these imports.

Change-Id: I280971c3d8c01e94cc2c17cacaedcbe9d9c8a3c3
2013-11-22 12:23:58 -08:00
Peter Portante
b1f51d00cd Use utils.ismount in place of os.path.ismount
See comments from: https://review.openstack.org/55991

Change-Id: Ibb4153702b3dc4c60f66abb11cd3fa1953449827
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-19 11:27:40 -05:00
Pete Zaitcev
fc9f9d7bae Strengthen account tests
This fell out of Peter Portante's "acctcont-api" branch and seems
obviously good. Apparently one of these would've triggered while
doing the Pluggable Backend work.

Now, why commit this separately while simultaneously working on
a big unified Backend Patch? Because posting them separately
proves that the test changes worked on the old code.

Change-Id: I9ca6ad45fb255f5c0a177a93b93c1acc68da5bbe
2013-10-29 16:10:47 -06:00
Peter Portante
b48435cd25 Fix UnboundLocalError on account PUT
Fixes bug-1243973

Change-Id: I67143535c0f7a0c6b53f67329a0bb128a355a4de
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-24 13:04:42 -04:00
Peter Portante
5202b0e586 DiskFile API, with reference implementation
Refactor on-disk knowledge out of the object server by pushing the
async update pickle creation to the new DiskFileManager class (name is
not the best, so suggestions welcome), along with the REPLICATOR
method logic. We also move the mount checking and thread pool storage
to the new ondisk.Devices object, which then also becomes the new home
of the audit_location_generator method.

For the object server, a new setup() method is now called at the end
of the controller's construction, and the _diskfile() method has been
renamed to get_diskfile(), to allow implementation specific behavior.

We then hide the need for the REST API layer to know how and where
quarantining needs to be performed. There are now two places it is
checked internally, on open() where we verify the content-length,
name, and x-timestamp metadata, and in the reader on close where the
etag metadata is checked if the entire file was read.

We add a reader class to allow implementations to isolate the WSGI
handling code for that specific environment (it is used no-where else
in the REST APIs). This simplifies the caller's code to just use a
"with" statement once open to avoid multiple points where close needs
to be called.

For a full historical comparison, including the usage patterns see:
https://gist.github.com/portante/5488238

(as of master, 2b639f5, Merge
 "Fix 500 from account-quota     This Commit
 middleware")
--------------------------------+------------------------------------
                                 DiskFileManager(conf)

                                   Methods:
                                     .pickle_async_update()
                                     .get_diskfile()
                                     .get_hashes()

                                   Attributes:
                                     .devices
                                     .logger
                                     .disk_chunk_size
                                     .keep_cache_size
                                     .bytes_per_sync

DiskFile(a,c,o,keep_data_fp=)    DiskFile(a,c,o)

  Methods:                         Methods:
   *.__iter__()
    .close(verify_file=)
    .is_deleted()
    .is_expired()
    .quarantine()
    .get_data_file_size()
                                     .open()
                                     .read_metadata()
    .create()                        .create()
                                     .write_metadata()
    .delete()                        .delete()

  Attributes:                      Attributes:
    .quarantined_dir
    .keep_cache
    .metadata
                                *DiskFileReader()

                                   Methods:
                                     .__iter__()
                                     .close()

                                   Attributes:
                                    +.was_quarantined

DiskWriter()                     DiskFileWriter()

  Methods:                         Methods:
    .write()                         .write()
    .put()                           .put()

* Note that the DiskFile class   * Note that the DiskReader() object
  implements all the methods       returned by the
  necessary for a WSGI app         DiskFileOpened.reader() method
  iterator                         implements all the methods
                                   necessary for a WSGI app iterator

                                 + Note that if the auditor is
                                   refactored to not use the DiskFile
                                   class, see
                                   https://review.openstack.org/44787
                                   then we don't need the
                                   was_quarantined attribute

A reference "in-memory" object server implementation of a backend
DiskFile class in swift/obj/mem_server.py and
swift/obj/mem_diskfile.py.

One can also reference
https://github.com/portante/gluster-swift/commits/diskfile for the
proposed integration with the gluster-swift code based on these
changes.

Change-Id: I44e153fdb405a5743e9c05349008f94136764916
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-17 15:03:31 -04:00
Peter Portante
9411a24ba7 Revert "Refactor common/utils methods to common/ondisk"
This reverts commit 7760f41c3c

Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 17:18:09 -04:00
Jenkins
0b594bc3af Merge "Change OpenStack LLC to Foundation" 2013-10-07 16:09:37 +00:00
Dirk Mueller
6b67952442 assertEquals is deprecated, use assertEqual
The very same functionality for fewer keystrokes.

Change-Id: I8c9513412f398132db5636fdfb7bf8301e8443cf
2013-10-02 21:27:40 +02:00
ZhiQiang Fan
f72704fc82 Change OpenStack LLC to Foundation
Change-Id: I7c3df47c31759dbeb3105f8883e2688ada848d58
Closes-bug: #1214176
2013-09-20 01:02:31 +08:00
Peter Portante
7760f41c3c Refactor common/utils methods to common/ondisk
Place all the methods related to on-disk layout and / or configuration
into a new common module that can be shared by the various modules
using the same on-disk layout.

Change-Id: I27ffd4665d5115ffdde649c48a4d18e12017e6a9
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-09-17 17:32:04 -04:00
Fabien Boucher
84296aae92 Improve unittest coverage of account reaper
Add more unittests for account_reaper in order to
improve code coverage.

Fixes: bug #1210045
Change-Id: I73490f80ac70b814a0abc490dc932f2a86fca703
2013-09-11 10:02:29 +02:00
Pete Zaitcev
d4b024ad7d Split backends off swift/common/db.py
The main purpose of this patch is to lay the groundwork for allowing
the container and account servers to optionally use pluggable backend
implementations. The backend.py files will eventually be the module
where the backend APIs are defined via docstrings of this reference
implementation. The swift/common/db.py module will remain an internal
module used by the reference implementation.

We have a raft of changes to docstrings staged for later, but this
patch takes care to relocate ContainerBroker and AccountBroker into
their new home intact.

Change-Id: Ibab5c7605860ab768c8aa5a3161a705705689b04
2013-09-10 13:30:28 -06:00
Dirk Mueller
3d36a76156 Use Python 3.x compatible except construct
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.

Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
2013-09-07 10:50:54 +02:00
Peter Portante
6ae4e17af1 Pep8 account and proxy server unit tests (10 of 12)
Change-Id: Ib83d164997b0d98be921c8b4857caa2429344aa4
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-09-01 16:12:51 -04:00
Peter Portante
c067abd21e Pep8 unit test modules for hacking and one liners (4 of 12)
Address all the "hacking" lines that are flagged, and all the modules
that just have one item flagged.

Change-Id: I372a4bdf9c7748f73e38c4fd55e5954f1afade5b
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-09-01 15:12:39 -04:00
Samuel Merritt
61a8a9af18 Refactor path splitting and validation.
The account, container, and object servers all do the same thing at
the top of GET/PUT/etc.: they split the path, validate that the first
two components are a good (device, partition) pair, and return a 400
if they're not. The object server already had a module-local helper
function for this, but the account and container servers just had
duplicate boilerplate code. Now it's all in one common helper.

Change-Id: I9a20d37fc9e1a68b10149a7aa78cb9691fc04ea9
2013-08-27 09:39:02 -07:00
Samuel Merritt
a4f371439b Refactor how we pick listings' content type.
There were a few different places where we had some repeated code to
figure out what format an account or container listing response should
be in (text, JSON, or XML). Now that's been pulled into a single
function.

As part of this, you can now raise HTTPException subclasses in proxy
controllers instead of laboriously plumbing error responses all the
way back up to swift.proxy.server.Application.handle_request(). This
lets us avoid certain ugly patterns, like the one where a method
returns a tuple of (x, y, z, error) and the caller has to see if it
got (value, value, value, None) or (None, None, None, errorvalue). Now
we can just raise the error.

Change-Id: I316873df289160d526487ad116f6fbb9a575e3de
2013-08-16 15:45:45 -07:00
Kun Huang
6fb327bc97 Add unit test for account/auditor.py
* Here lack tests for a long time; let's add tests for it.
* use mock module instead of monkeypatch in both account/test_auditor.py
  and container/test_auditor.py

Change-Id: I9ea23dc5034f7147629db59b1ef6cb491de9049f
2013-08-15 12:32:47 +08:00
Jenkins
3e82858a21 Merge "Catch swob responses that are raised." 2013-07-25 17:51:43 +00:00
Samuel Merritt
df7fc9658b Catch swob responses that are raised.
This lets us get rid of some really repetitive exception conversion
code, like everybody that called common.utils.get_param() had to catch
a UnicodeDecodeError and turn that into returning HTTPBadRequest. Now
get_param() just raises HTTPBadRequest directly, and the __call__
methods in the account/container/object servers catch and return
it. All that "except UnicodeDecodeError" stuff goes away.

Refactored the path splitting and device validation in the object
server too.

There are other things that can benefit from this as well, but this
patch is big enough.

Change-Id: I2be96ef757d04bfd6af180cd9c92393c841db21f
2013-07-24 16:59:45 -07:00