This library provides a simple Python interface for implementing erasure codes and is known to work with Python v2.6, 2.7 and 3.x. To obtain the best possible performance, the library utilizes liberasurecode, which is a C based erasure code library.
PyECLib supports a variety of Erasure Coding backends including the standard Reed-Solomon implementations provided by Jerasure , liberasurecode , Intel's ISA-L  and Phazr.IO's libphazr. It also provides support for a flat XOR-based encoder and decoder (part of liberasurecode) - a class of HD Combination Codes based on "Flat XOR-based erasure codes in storage systems: Constructions, efficient recovery, and tradeoffs" in IEEE MSST 2010). These codes are well-suited to archival use-cases, have a simple construction and require a minimum number of participating disks during single-disk reconstruction (think XOR-based LRC code).
- Python 2.6, 2.7 or 3.x (including development packages), argparse, setuptools
- liberasurecode v1.3.1 or greater 
- Erasure code backend libraries, gf-complete and Jerasure ,, ISA-L , etc
$ sudo apt-get install build-essential python-dev python-pip liberasurecode-dev $ sudo pip install -U bindep -r test-requirements.txt
$ sudo yum install -y redhat-lsb python2-pip python-devel liberasurecode-devel $ sudo pip install -U bindep -r test-requirements.txt $ tools/test-setup.sh
If you want to confirm all dependency packages installed successfully, try:
$ sudo bindep -f bindep.txt
Note: Currently, for Ubuntu, liberasurecode-dev in package repo is older than v1.2.0. For CentOS, make sure to install the latest Openstack Cloud SIG repo to be able to install the latest available version of liberasurecode-devel.
$ sudo python setup.py install
Run test suite included:
If the test suite fails because it cannot find any of the shared libraries, then you probably need to add /usr/local/lib to the path searched when loading libraries. The best way to do this (on Linux) is to add '/usr/local/lib' to:
and then make sure to run:
$ sudo ldconfig
Examples of using PyECLib are provided in the "tools" directory:
Utility to determine what is needed to reconstruct missing fragments:
A configuration utility to help compare available EC schemes in terms of performance and redundancy:
ec_driver = ECDriver(k=<num_encoded_data_fragments>, m=<num_encoded_parity_fragments>, ec_type=<ec_scheme>))
liberasurecode_rs_vand=> Vandermonde Reed-Solomon encoding, software-only backend implemented by liberasurecode 
jerasure_rs_vand=> Vandermonde Reed-Solomon encoding, based on Jerasure 
jerasure_rs_cauchy=> Cauchy Reed-Solomon encoding (Jerasure variant), based on Jerasure 
flat_xor_hd_4=> Flat-XOR based HD combination codes, liberasurecode 
isa_l_rs_vand=> Intel Storage Acceleration Library (ISA-L) - SIMD accelerated Erasure Coding backends 
isa_l_rs_cauchy=> Cauchy Reed-Solomon encoding (ISA-L variant) 
shss=> NTT Lab Japan's Erasure Coding Library 
libphazr=> Phazr.IO's erasure code library with built-in privacy 
This library is currently mainly maintained by the Openstack Swift community. For questions or any other help, come ask in #openstack-swift on OFTC.
 Jerasure, C library that supports erasure coding in storage applications, http://jerasure.org
 Greenan, Kevin M et al, "Flat XOR-based erasure codes in storage systems", http://www.kaymgee.com/Kevin_Greenan/Publications_files/greenan-msst10.pdf
 liberasurecode, C API abstraction layer for erasure coding backends, https://opendev.org/openstack/liberasurecode
 Intel(R) Storage Acceleration Library (Open Source Version), https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version
 Kota Tsuyuzaki <email@example.com>, "NTT SHSS Erasure Coding backend"
 Jim Cheung <firstname.lastname@example.org>, "Phazr.IO libphazr erasure code backend with built-in privacy"