442cc1d16d
The assumption that we don't need to write an entry in the invalidations
file when the hashes.pkl does not exist turned out to be a premature
optimization and also wrong.
Primarily we should recognize the creation of hashes.pkl is the first
thing that happens in a part when it lands on a new primary. The code
should be optimized toward the assumption of the most common disk state.
Also, in this case the extra stat calls to check if the hashes.pkl exists
were not only un-optimized - but introducing a race.
Consider the common case:
proc 1 | proc 2
-------------------------------|---------------------------
a) read then truncate journal |
b) do work | c) append to journal
d) apply "a" to index |
The index written at "d" may not (yet) reflect the entry writen by proc
2 at "c"; however, it's clearly in the journal so it's easy to see we're
safe.
Adding in the extra stat call for the index existence check increases
the state which can effect correctness.
proc 1 | proc 2
------------------------------|---------------------------
a) no index, truncate journal |
b) do work | b) iff index exists
| c) append to journal
d) apply (or create) index |
If step "c" doesn't happen because the index does not yet exist - the
update is clearly lost.
In our case we'd skip marking a suffix as invalid when the hashes.pkl
does not exist because we know "the next time we rehash" we'll have to
os.listdir suffixes anyway. But if another process is *currently*
rehashing (and has already done it's os.listdir) instead we've just
dropped an invalidation on the floor.
Don't do that.
Instead - write down the invalidation. The running rehash is welcome to
proceed on outdated information - as long as the next pass will grab and
hash the new suffix.
Known-Issue(s):
If the suffix already exists there's an even chance the running rehash
will hash in the very update for which we want to invalidate the suffix,
but that's ok it's idempotent.
Co-Author: Pavel Kvasnička <pavel.kvasnicka@firma.seznam.cz>
Co-Author: Alistair Coles <alistair.coles@hpe.com>
Co-Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Related-Change-Id: I64cadb1a3feb4d819d545137eecfc295389794f0
Change-Id: I2b48238d9d684e831d9777a7b18f91a3cef57cd1
Closes-Bug: #1651530
|
||
|---|---|---|
| .. | ||
| functional | ||
| probe | ||
| unit | ||
| __init__.py | ||
| sample.conf | ||