swift/swift/obj
Samuel Merritt d5c532a94e object-updater: add concurrent updates
The object updater now supports two configuration settings:
"concurrency" and "updater_workers". The latter controls how many
worker processes are spawned, while the former controls how many
concurrent container updates are performed by each worker
process. This should speed the processing of async_pendings.

There is a change to the semantics of the configuration
options. Previously, "concurrency" controlled the number of worker
processes spawned, and "updater_workers" did not exist. I switched the
meanings for consistency with other configuration options. In the
object reconstructor, object replicator, object server, object
expirer, container replicator, container server, account replicator,
account server, and account reaper, "concurrency" refers to the number
of concurrent tasks performed within one process (for reference, the
container updater and object auditor use "concurrency" to mean number
of processes).

On upgrade, a node configured with concurrency=N will still handle
async updates N-at-a-time, but will do so using only one process
instead of N.

UpgradeImpact:

If you have a config file like this:

    [object-updater]
    concurrency = <N>

and you want to take advantage of faster updates, then do this:

    [object-updater]
    concurrency = 8  # the default; you can omit this line
    updater_workers = <N>

If you want updates to be processed exactly as before, do this:

    [object-updater]
    concurrency = 1
    updater_workers = <N>

Change-Id: I17e18088e61f664e1b9942d66423666d0cae1689
2018-06-13 17:39:34 -07:00
..
__init__.py Initial commit of Swift code 2010-07-12 17:03:45 -05:00
auditor.py Change object_audit_location_generator() to yield for a single policy. 2018-03-22 03:02:35 +00:00
diskfile.py Use X-Timestamp when checking object expiration 2018-05-22 16:42:53 -07:00
expirer.py Fix object-server to not 400 all expirer DELETEs 2018-03-02 15:25:38 +00:00
mem_diskfile.py Use X-Timestamp when checking object expiration 2018-05-22 16:42:53 -07:00
mem_server.py Per-policy DiskFile classes 2015-04-14 00:52:16 -07:00
reconstructor.py Make final stats dump after reconstructor runs once 2018-06-04 15:24:45 -07:00
replicator.py Merge "Truncate error logs to prevent log handler runs out of the buffer." 2018-05-22 18:06:08 +00:00
server.py Merge "Use X-Timestamp when checking object expiration" 2018-05-31 23:55:59 +00:00
ssync_receiver.py Merge "Fix SSYNC failing to replicate unexpired object" 2017-05-31 22:49:55 +00:00
ssync_sender.py Allow to rebuild a fragment of an expired object 2017-08-04 23:05:08 +02:00
updater.py object-updater: add concurrent updates 2018-06-13 17:39:34 -07:00