Apparently the author of our old crc32 assumed that shifting an int
to the right sign-extends, which is not always the case. Result is,
building and running make test on s390x fails. The fix is to force
a sign-extension using the "xor 0x80; sub 0x80" trick.
N.B. This does not cause a compatibility problem, because by a
miracle the "broken" crc32_alt was actually computing a stock
crc32, same that zlib has. Therefore, if someone, somewhere,
ran a Swift cluster on s390x with erasure coding policy,
the data in it is already stored with zlib checksums, as we
do it now anyway. This fix only permits the tests pass, which
used the bad data sample from x86.
Change-Id: Ibd5e4e6c02be00540a9648cc7e0f8efda275bf3f
Related-Change: Ib5ea2a830c7c23d66bf2ca404a3eb84ad00c5bc5
Related-Bug: 1666320
This popped up because Fedora mandates warning-free builds,
and get_chksum triggers a warning because it returns an
unaligned pointer (it is so analigned, static analysis in
the compiler can detect it).
The easiest fix is to remove it altogether. We think it should
be safe, because:
- The function is not listed in any headers
- Its counterpart is called "set_checksum", not "set_chksum"
- PyECLib does not use it
We also hush some doxygen warnings about wrong comments.
Change-Id: Ie5bc736f912706e0ffd507765abd24e7f4761233
If LD_LIBRARY_PATH is set to any value, build will fail, example:
Making check in doc
...
/bin/bash: line 1: /usr/lib/libeatmydata: No such file or directory
...
See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=906480
Same is true for DYLD_LIBRARY_PATH and DYLD_FALLBACK_LIBRARY_PATH.
This bug was introduced with commit 19442df2cd9a5bc8ec4deded7046ea7aca1d50a2
where Kota removed evals, but forgot to remove prepending of same env
variables.
Change-Id: Ib7a1d6b839d4a207ee0471b55233e1ce5d958705
... and vice-versa. We'll fix up frag header values for our output
parameter from liberasurecode_get_fragment_metadata but otherwise
avoid manipulating the in-memory fragment much.
Change-Id: Idd6833bdea60e27c9a0148ee28b4a2c1070be148
Add the unittests in-tree and convert them as Zuul v3 native tests so
that they can be modified locally.
This adds a new job running on Ubuntu and an experimental CentOS-7 one.
Change-Id: Ib43e40280411623fda84f2068958b64a17cdc3dc
This simplifies compiling by allowing users to include
CFLAGS += `pkg-config --cflags erasurecode-1`
LDFLAGS += `pkg-config --libs erasurecode-1`
in makefiles. Otherwise, trying to use pkg-config results in errors like
/usr/local/lib/liberasurecode.so: undefined reference to `dlopen'
/usr/local/lib/liberasurecode.so: undefined reference to `dlclose'
/usr/local/lib/liberasurecode.so: undefined reference to `dlerror'
/usr/local/lib/liberasurecode.so: undefined reference to `dlsym'
collect2: error: ld returned 1 exit status
Change-Id: I39fb137b1a3b6b2beda1d0b28faef3132229ec3b
Otherwise, we're left with error-prone copy & pasting, which means we'll
constantly be cleaning it up.
Change-Id: I57e2cbef2c9221cffccf2e73b6af9bd003c04968
Related-Change: Ibd72ba4ae609ad77e8808aa1594b0adb62e34ef0
Related-Change: I9ee9ec3d8f86a10c4c7b5d6425a530b9c44d1156
* When we specify a particular backend, we don't need to include it in
the display string; we'll already include it in the output.
* test_destroy_backend_invalid_args shouldn't be reported as
test_create_backend_invalid_args.
* Consistently include the leading "test_" in the display string.
Change-Id: Ibd72ba4ae609ad77e8808aa1594b0adb62e34ef0
Each was only really used in one place, they had some strange return types,
and recent versions of clang on OS X would refuse to compile with
erasurecode_helpers.c:531:26: error: taking address of packed member 'metadata_chksum' of
class or structure 'fragment_header_s' may result in an unaligned pointer value
[-Werror,-Waddress-of-packed-member]
return (uint32_t *) &header->metadata_chksum;
^~~~~~~~~~~~~~~~~~~~~~~
We don't really *care* about the pointer; we just want the value!
Change-Id: I8a5e42312948a75f5dd8b23b6f5ccfa7bd22eb1d
Previously, we had our own CRC that was almost but not quite like
zlib's implementation. However,
* it hasn't been subjected to the same rigor with regard to error-detection
properties and
* it may not even get used, depending upon whether zlib happens to get
loaded before or after liberasurecode.
Now, we'll use zlib's CRC-32 when writing new frags, while still
tolerating frags that were created with the old implementation.
Change-Id: Ib5ea2a830c7c23d66bf2ca404a3eb84ad00c5bc5
Closes-Bug: 1666320
It isn't the CRC we want, and we never used it anyway. While we may
define INTEL_SSE41 or INTEL_SSE42 if CPU seems to support it, we've
never defined INTEL_SSE4.
Change-Id: I04e1dd6458ccde58a0a2f3f4d6947569a31e9697
Partial-Bug: 1666320
Broke down the README a little bit to make it more readable.
Moved the API and dir organization to their own md files.
Change-Id: I7161daaa9515bd091e82a78dd8496ba23d5b7549
Signed-off-by: Thiago da Silva <thiago@redhat.com>
The well-known idiom to compute a required number of data blocks
of size B to contain data of length d is:
(d + (B-1))/B
The code we use, with ceill(), computes the same value, but does
it in an unorthodox way. This makes a reviewer to doubt himself
and even run tests to make sure we're really computing the
obvious thing.
Apropos the reviewer confusion, the code in Phazr.IO looks weird.
It uses (word_size - hamming_distance) to compute the necessary
number of blocks... but then returns the amount of memory needed
to store blocks of a different size (word_size). We left all of it
alone and return exactly the same values that the old computation
returned.
All these computations were the only thing in the code that used
-lm, so drop that too.
Coincidentially, this patch solves the crash of distro-built
packages of liberasurecode (see Red Hat bug #1454543). But it's
a side effect. Expect a proper patch soon.
Change-Id: Ib297f6df304abf5ca8c27d3392b1107a525e0be0
We're having trouble on Fedora when the build system runs
on Intel CPUs. The ./configure detects AVX instructions and
builds liberasurecode with them. The resulting library crashes
with SIGILL when users run it on ADM CPUs without AVX.
See the details here:
https://bugzilla.redhat.com/show_bug.cgi?id=1454543
The patch allows to disable the optimizations, so that
distro packaging then can invoke this options and build
portable libraries.
For the record, "make test" runs about 16% slower on an Intel
CPU if optimizations are disabled. So, there's a measurable
performance impact. However, operators intersted in outright
performance might want to consider Erasure Coding implementations
other than the one built-in into liberasurecode. The library
supports other back-ends that are significantly faster than
even optimized built-in code.
Change-Id: I09603b97ceeb833ba582cf3217e0be51c019d645
Currently, there are several implementations of erasure codes that are
available within OpenStack Swift. Most, if not all, of which are based
on the Reed Solomon coding algorithm.
Phazr.IO’s Erasure Coding technology uses a patented algorithm which are
significantly more efficient and improves the speed of coding, decoding
and reconstruction. In addition, Phazr.IO Erasure Code use a non-systematic
algorithm which provides data protection at rest and in transport without
the need to use encryption.
Please contact support@phazr.io for more info on our technology.
Change-Id: I4e40d02a8951e38409ad3c604c5dd6f050fa7ea0
It uses hard-coded backend and arguments, so it should use EC_BACKEND_MAX
and have a name that reflect how it's only targeted to Jerasure.
This is similar to how we handled the targeted test case for ISA-L.
Change-Id: I0a397930b2f49c0775290970c820bc70310a475e
Jerasure inits some global variables on init.
We need to free them, or we'll leak memory.
Partial-Bug: #1666674
Change-Id: Ie7073738428a71910016e910a66dbd92ca98eb92
Signed-off-by: Daniel Axtens <dja@axtens.net>
This patch fixes automake tools warnings as follows (and a my fault
in previous patch [1]):
- Change INCLUDES variable to APP_CPP_FLAGS and APP_C_FLAGS according to
the warning message
- Remove non-POSIX variable to suppress that warnings
Note that the latter change may be affected to the testing so please
be careful on your review what's going on your testing environment.
In the past change [2], they ware designed to use the shared
object binaries in the build path. However, Daniel had pointed out the good
thing at [1] if we runs the test/liberasruecode_test (it's shell with linker)
in the dir, we can touch the expected binaries.
My fault in the previous patch is that I didn't replace
./test/.libs/liberasurecode_test to ./test/liberasurecode_test even
we use --trace-children=yes option for the valgrind.
Now, all tests run via ./test/<test name> syntax and IMO no matters
exist even if we remove the non-POSIX variable for test settings.
1: https://review.openstack.org/#/c/434162
2: 93446db9414f311bd9fc7dc047eb4dbbeb3e6feb
Change-Id: I0e79bed7755a1f286b746a70fcf56fdc972bfd5d
Can you believe that we ware testing the memory leak with valgrind
to just *bash scripts* instead of actual binaries on liberasurecode_test
and libec_slap?
That is why we cannot find such an easy memory leak[1] at the gate.
Now this patch enable to run the valgrind against to the binaries.
With this fix, we found various memory leak at liberasurecode_test as
follows and this patch also fixes them:
- If we create fake fragments, we're responsible for freeing all of the
frags as well as the array holding the pointers to the frags.
- If we allocate any space, we're responsible for freeing it.
- If we create an EC descriptor, we're responsible for destroying it.
- If we create a fragment or skip array, we're responsible for freeing it.
- If that happens inside a loop, we're responsible for doing it *inside
that same loop*.
In addition to the test fix, this patch fixes following memory leaks at
the code which is affected to other users (pyeclib, OpenStack Swift)
* Refuse to decode fragments that aren't even long enough to include
fragment headers.
* Fix a small memory leak in the builtin rs_vand implementation.
Closes-Bug: #1665242
Co-Authored-By: Tim Burke <tim@swiftstack.com>
1: https://review.openstack.org/#/c/431812
Change-Id: I96f124e4e536bbd7544208acc084de1cda5c19b2
Currently, the Galois Field multiplication tables are recalcuated
every time an encode is done. This is wasteful, as they are fixed
by k and m, which is set on init.
Calculate the tables only once, on init.
This trades off a little bit of per-context memory and creation
time for measurably faster encodes when using the same context.
On powerpc64le, when repeatedly encoding a 4kB file with pyeclib,
this increases the measured speed by over 10%.
Change-Id: I2f025aaee2d13cb1717a331e443e179ad5a13302
Signed-off-by: Daniel Axtens <dja@axtens.net>
