This passes currently, but only because of the GIL. Testing on py313t
with PYTHON_GIL=0 and liberasurecode 1.6.2, it segfaults.
Related-Change: https://review.opendev.org/c/openstack/liberasurecode/+/929325
Change-Id: I46ec7d3ee28700c5ebb7b563dafc6a0fb3c9a10f
We can use the default value by fixing the only one error.
test/test_pyeclib_api.py:111:80: E501 line too long (83 > 79 characters)
Change-Id: Idcdd6f643330b76328b5cd28c32900eeceacafb8
Six years seems like more than enough warning, and the data corruption
bug fixed in 1.3.1 is pretty bad.
This also gives us an opportunity to clean up a DeprecationWarning:
The distutils package is deprecated and slated for removal in
Python 3.12. Use setuptools or check PEP 632 for potential
alternatives
Change-Id: Ic34b671dfb2b5bfa7e3a21813cd49d7b720cc94a
Closes-Bug: #1639691
...and clean up the unused variables, unused imports, and inconsistent
spacing that flake8 revealed.
Change-Id: I3ae1e834f01f477be88dfc48de954eeaf19e43cf
This came up when running the py3 unit tests after adding a test that
(erroneously) tried to encode native strings. At some point, we should
probably enforce tighter type-checking, but I'm not sure what all types
we need (or want) to support.
Change-Id: Ibb1f4f52207be83f842740d9f3c39c2a03fb1396
- Add a docstring, since I can never remember the **kwargs.
- Drop the (unused) *args.
- Raise an error earlier if neither ec_type nor library_import_str
is provided.
Change-Id: I38b71ede59b6b0e249223792207b0f48438f7b97
Currently, there are several implementations of erasure codes that are
available within OpenStack Swift. Most, if not all, of which are based
on the Reed Solomon coding algorithm.
Phazr.IO’s Erasure Coding technology uses a patented algorithm which are
significantly more efficient and improves the speed of coding, decoding
and reconstruction. In addition, Phazr.IO Erasure Code use a non-systematic
algorithm which provides data protection at rest and in transport without
the need to use encryption.
Please contact support@phazr.io for more info on our technology.
Change-Id: I9377fa32426a190efd0a7f0675ecf13d7e90367d
These will either pass if the backend is available, or skip if it isn't.
What we definitely *don't* want is a situation where someone runs tests,
sees all passes with no skips, and assumes they must therefore have all
backends available.
Change-Id: I1d2c691d01f38d5e8f7bc9cf4ec4b9c50ace32b9
Since we allocate the result buffer in hex_encode_string, we're
responsible for freeing it later.
Why didn't we catch it before? Turns out, our tests didn't
loop enough. I wasn't reliably seeing a test failure at 200k
iterations, but 400k seems to be enough?
Closes-Bug: #1634006
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Related-Change: I678e10008c3c5bc04640f7f19498334d94cb0cd6
Change-Id: I0f6e922ba25ad56142f3d095896b9856a436a81c
This is for supporting ISA-L cauchy based matrix. The difference
from isa_l_rs_vand is only the matrix to use the encode/decode
calculation.
As a known issue, isa_l_rs_vand backend has constraint for the
combinations of the available fragment to be able to
decode/reconstuct. (See related change in detail)
To avoid the constraint, this patch adds another isa-l backend to use
cauchy matrix. The reason I try to add this intead of changing at
sa_l_rs_vand, the isa_l_rs_vand backend may be already used and if we
change the matrix, the current users won't be able to decode the data.
To avoid the problem and keep the backward compatibility, this is in
another isa_l_rs_cauchy namespace.
NOTE: this depends on tag may be meaningless because the depending target
is a c lang project so probably no scripts to install it we have and then,
almost of the tests for isa-l will be skipped due to a lack of isa-l
backend installation.
Related-Change: Icee788a0931fe692fe0de31fabc4ba450e338a87
Depends-On: I6eb150d9d0c3febf233570fa7729f9f72df2e9be
Change-Id: I3a5516545d17ab7ac67a9a3627c85243e2110764
* Test flat_xor_hd decoding/reconstruction
* Use random input
* Make py3-friendly
Additionally:
* Accept iterables as fragment payloads, rather than just lists
* Add a __repr__ for ECDrivers
Change-Id: Ic5b5e5ef2420afdc318b403fcbea1ff106e16a33
Once using erasure code scheme in storage system, we have to trust the
backend's result consistency from decode/reconstruct. Otherwise, the
storage system can be broken easily. To trust the result, this patch
adds greedy combination testing for decode/reconstruct for read-solomn
based backends.
Testing:
- decode result from any k fragments must be consistent with original
data
- The Nth fragment reconstructed from other k fragments must be
consistent with the encoded Nth fragment
With this patch, we can confirm the consistency, at least, the testing
spec defined at get_pyeclib_testspec() method, and if you want to test
either other parameters or backends, you can add the spec you want to
test.
Change-Id: Ieb46f4ab5c8f6cb60f35c632728cbcf4a1727c53
PyDict_SetItems doesn't steal the reference count so that the reference
count of each item in the dict should be decremented via Py_DECREF
*before* returning the dict to Python VM. Otherwise, those items can be
leaked which can never be collected as garbage.
The evicence for memory leak is avaialble for using debug mode python VM like:
https://gist.github.com/bloodeagle40234/f4c0cd267e085cc6224ffdc1b1822631
Plus, this patch add one unit test to check the memory increasement for
many call of get_segment_info.
Closes-Bug: #1604335
Change-Id: I6780efb9718017d296606f3c7e60684910584a1a
Right now ECDriver doesn't check if k, m exist in the kwargs for init.
That can tigger unfortunately success to init and that will fail at
encode/decode (or result in odd with get_segment_info)
Once, we have jerasure_rs_vand in default, that worked becuase jerasure
did the assertion for the k, m parameters but we can reproduce the
failures which this patch want fix with like:
driver = ECDriver(ec_type='liberasurecode_rs_vand') # <- this is
default and it succeeded!
driver.encode(' ') # <- And then, this will result in ECDriverError:
Out of memory, whooa!
This patch fixes this to check the k, m existence in the init process.
Change-Id: I0757c0a4e510ba42f357db0cac22861919d0ca26
Plus, we need liberasurecode version handling for a few place because
some tests/engine itself is broken with a lack of backword
compatibility.
Closes-Bug: #1586220
Change-Id: I72adaefa10875a73e3e5304eb40fe5d9f6d2598a
* Use relative imports
* Fix bytes/unicode issues in tests
* Replace long() with int()
* Replace a/b with a//b to get integer division
* Use a key function to sort list
* Replace zip() with list(zip()) in a test to get a list on Python 3
Fedora packaging mandates support for py3 nowadays, so I have to
fix stuff like this. Interestingly enough, we don't seem to need
from future import __print_statement__ here.
If ECDriverError is initialized with a non-string argument then its
str method returns non-string, it will return a confusable message
for users.
This patch fixes ECDriverError to set a string message when
the argument doesn't have str method which return a string.
