Browse Source

Require that known-bad EC schemes be deprecated

We said we were going to do it, we've had two releases saying we'd do
it, we've even backported our saying it to Newton -- let's actually do

Upgrade Consideration

Erasure-coded storage policies using isa_l_rs_vand and nparity >= 5 must
be configured as deprecated, preventing any new containers from being
created with such a policy. This configuration is known to harm data
durability. Any data in such policies should be migrated to a new
policy. See for more

Related-Change: I50159c9d19f2385d5f60112e9aaefa1a68098313
Change-Id: I8f9de0bec01032d9d9b58848e2a76ac92e65ab09
Closes-Bug: 1639691
Tim Burke 4 years ago
4 changed files with 25 additions and 16 deletions
  1. +9
  2. +4
  3. +4
  4. +8

+ 9
- 0
doc/source/overview_erasure_code.rst View File

@ -252,6 +252,15 @@ with the specified policy name and interacting as usual.
case a change in the configuration is desired, you must create a new policy
and migrate the data to a new container.
.. warning::
Using ``isa_l_rs_vand`` with more than 4 parity fragments creates fragments
which may in some circumstances fail to reconstruct properly or (with
liberasurecode < 1.3.1) reconstruct corrupted data. New policies that need
large numbers of parity fragments should consider using ``isa_l_rs_cauchy``.
Any existing affected policies must be marked deprecated, and data in
containers with that policy should be migrated to a new policy.
Migrating Between Policies

+ 4
- 3
etc/swift.conf-sample View File

@ -62,13 +62,14 @@ aliases = yellow, orange
# EC configuration parameters 'ec_type', 'ec_num_data_fragments' and
# 'ec_num_parity_fragments' must be specified. 'ec_type' is chosen from the
# list of EC backends supported by PyECLib. The ring configured for the
# storage policy must have it's "replica" count configured to
# storage policy must have its "replica" count configured to
# 'ec_num_data_fragments' + 'ec_num_parity_fragments' - this requirement is
# validated when services start. 'ec_object_segment_size' is the amount of
# data that will be buffered up before feeding a segment into the
# encoder/decoder. More information about these configuration options and
# supported `ec_type` schemes is available in the Swift documentation. Please
# refer to Swift documentation for details on how to configure EC policies.
# supported 'ec_type' schemes is available in the Swift documentation. See
# for more information on how to configure EC policies.
# The example 'deepfreeze10-4' policy defined below is a _sample_
# configuration with an alias of 'df10-4' as well as 10 'data' and 4 'parity'

+ 4
- 8
swift/common/ View File

@ -468,14 +468,10 @@ class ECStoragePolicy(BaseStoragePolicy):
'See for '
'more information.' %
if not is_deprecated:
# TODO: To fully close bug 1639691, uncomment the raise and
# removing the warning below. This will be in the Pike release
# at the earliest.
'In a future release, this will prevent services from '
'starting unless the policy is marked as deprecated.')
# raise PolicyError('Storage policy %s MUST be deprecated' %
raise PolicyError(
'Storage policy %s uses an EC configuration known to harm '
'data durability. This policy MUST be deprecated.'
# Initialize PyECLib EC backend

+ 8
- 5
test/unit/common/ View File

@ -676,18 +676,21 @@ class TestStoragePolicies(unittest.TestCase):
ec_num_parity_fragments = 5
with capture_logging('swift.common.storage_policy') as records:
with capture_logging('swift.common.storage_policy') as records, \
self.assertRaises(PolicyError) as exc_mgr:
'Storage policy bad-policy uses an EC '
'configuration known to harm data durability. This '
'policy MUST be deprecated.')
self.assertEqual([r.levelname for r in records],
for msg in ('known to harm data durability',
'Any data in this policy should be migrated',
self.assertIn(msg, records[0].msg)
self.assertIn('In a future release, this will prevent services from '
'starting', records[1].msg)
slightly_less_bad_conf = self._conf("""