openstack/swift
Code Issues Proposed changes

Use custom encoding for RingData, not pickle.

Browse Source 
Serialize RingData in a versioned, custom format which is a combination
of a JSON-encoded header and .tostring() dumps of the
replica2part2dev_id arrays.  This format deserializes hundreds of times
faster than rings serialized with Python 2.7's pickle (a significant
performance regression for ring loading between Python 2.6 and Python
2.7).  Fixes bug 1031954.

swift.common.ring.ring.RingData is now responsible for serialization and
deserialization of its data via a new load() class method and save()
object method.  The new implementation is backward-compatible; if a ring
does not begin with a new-style magic string, it is assumed to be an
old-style pickle-dumped ring and is handled as before.  So new Swift
code can read old rings, but old Swift code will not be able to read
newly-serialized rings. THIS SHOULD BE MENTIONED PROMINENTLY IN THE
RELEASE NOTES.

I didn't want to bite of more than necessary, so I didn't mess with
builder file serialization.

Change-Id: I799b9a4c894d54fb16592443904ac055b2638e2d
This commit is contained in:
Darrell Bishop
2012-08-05 00:51:49 -07:00
parent ef3f8bb335
commit f8ce43a218
4 changed files with 129 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
bin/swift-ring-builder
View File

@@ -610,13 +610,11 @@ swift-ring-builder <builder_file> rebalance
print '-' * 79
status = EXIT_WARNING
ts = time()
pickle.dump(builder.get_ring().to_dict(),
GzipFile(pathjoin(backup_dir, '%d.' % ts +
basename(ring_file)), 'wb'), protocol=2)
builder.get_ring().save(
pathjoin(backup_dir, '%d.' % ts + basename(ring_file)))
pickle.dump(builder.to_dict(), open(pathjoin(backup_dir,
'%d.' % ts + basename(argv[1])), 'wb'), protocol=2)
pickle.dump(builder.get_ring().to_dict(), GzipFile(ring_file, 'wb'),
protocol=2)
builder.get_ring().save(ring_file)
pickle.dump(builder.to_dict(), open(argv[1], 'wb'), protocol=2)
exit(status)
@@ -644,10 +642,9 @@ swift-ring-builder <builder_file> write_ring
'"rebalance"?'
else:
print 'Warning: Writing an empty ring'
pickle.dump(ring_data.to_dict(),
GzipFile(pathjoin(backup_dir, '%d.' % time() +
basename(ring_file)), 'wb'), protocol=2)
pickle.dump(ring_data.to_dict(), GzipFile(ring_file, 'wb'), protocol=2)
ring_data.save(
pathjoin(backup_dir, '%d.' % time() + basename(ring_file)))
ring_data.save(ring_file)
exit(EXIT_SUCCESS)
def pretend_min_part_hours_passed():