7e97b2f808
Apparently the author of our old crc32 assumed that shifting an int to the right sign-extends, which is not always the case. Result is, building and running make test on s390x fails. The fix is to force a sign-extension using the "xor 0x80; sub 0x80" trick. N.B. This does not cause a compatibility problem, because by a miracle the "broken" crc32_alt was actually computing a stock crc32, same that zlib has. Therefore, if someone, somewhere, ran a Swift cluster on s390x with erasure coding policy, the data in it is already stored with zlib checksums, as we do it now anyway. This fix only permits the tests pass, which used the bad data sample from x86. Change-Id: Ibd5e4e6c02be00540a9648cc7e0f8efda275bf3f Related-Change: Ib5ea2a830c7c23d66bf2ca404a3eb84ad00c5bc5 Related-Bug: 1666320 |
||
|---|---|---|
| doc | ||
| include | ||
| playbooks/unittests | ||
| roles | ||
| src | ||
| test | ||
| .gitignore | ||
| .gitreview | ||
| .mailmap | ||
| .zuul.yaml | ||
| AUTHORS | ||
| COPYING | ||
| ChangeLog | ||
| Makefile.am | ||
| NEWS | ||
| README | ||
| README.md | ||
| Xorcode.pc.in | ||
| autogen.sh | ||
| bindep.txt | ||
| configure.ac | ||
| erasurecode.pc.in | ||
| erasurecode_rs_vand.pc.in | ||
| get_flags_from_cpuid.c | ||
README.md
liberasurecode
liberasurecode is an Erasure Code API library written in C with pluggable Erasure Code backends.
Highlights
-
Unified Erasure Coding interface for common storage workloads.
-
Pluggable Erasure Code backends - liberasurecode supports the following backends:
- 'liberasurecode_rs_vand' - Native, software-only Erasure Coding implementation that supports a Reed-Solomon backend
- 'Jerasure' - Erasure Coding library that supports Reed-Solomon, Cauchy backends [1]
- 'ISA-L' - Intel Storage Acceleration Library - SIMD accelerated Erasure Coding backends [2]
- 'SHSS' - NTT Lab Japan's hybrid Erasure Coding backend [4]
- 'Flat XOR HD' - built-in to liberasurecode, based on [3]
- 'libphazr' - Phazr.IO's erasure code backend with built-in privacy [5]
- 'NULL' template backend implemented to help future backend writers
-
True 'plugin' architecture - liberasurecode uses Dynamically Loaded (DL) libraries to realize a true 'plugin' architecture. This also allows one to build liberasurecode indepdendent of the Erasure Code backend libraries.
-
Cross-platform - liberasurecode is known to work on Linux (Fedora/Debian flavors), Solaris, BSD and Darwin/Mac OS X.
-
Community support - Developed alongside Erasure Code authority Kevin Greenan, liberasurecode is an actively maintained open-source project with growing community involvement (Openstack Swift, Ceph, PyECLib, NTT Labs).
Active Users
- PyECLib - Python EC library: https://github.com/openstack/pyeclib
- Openstack Swift Object Store - https://wiki.openstack.org/wiki/Swift
Build and Install
Install dependencies -
Debian/Ubuntu hosts:
$ sudo apt-get install build-essential autoconf automake libtool
Fedora/RedHat/CentOS hosts:
$ sudo yum install -y gcc make autoconf automake libtool
To build the liberasurecode repository, perform the following from the top-level directory:
$ ./autogen.sh
$ ./configure
$ make
$ make test
$ sudo make install
References
[1] Jerasure, C library that supports erasure coding in storage applications, http://jerasure.org
[2] Intel(R) Storage Acceleration Library (Open Source Version), https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version
[3] Greenan, Kevin M et al, "Flat XOR-based erasure codes in storage systems", http://www.kaymgee.com/Kevin_Greenan/Publications_files/greenan-msst10.pdf
[4] Kota Tsuyuzaki tsuyuzaki.kota@lab.ntt.co.jp, "NTT SHSS Erasure Coding backend"
[5] Jim Cheung support@phazr.io, "Phazr.IO libphazr erasure code backend with built-in privacy"