A simple Python interface for implementing erasure codes
Go to file
Tim Burke f8a0983959 Test under py312
Also add trove classifiers for py311 and py312.

Depends-On: https://review.opendev.org/c/openstack/pyeclib/+/839643
Change-Id: I2b0a44ca1e4af480c3da185a37d21e4e26397337
2024-04-29 18:14:23 +00:00
doc remove unicode from code 2022-08-25 10:56:22 +08:00
pyeclib Drop support for liberasurecode<1.4.0 2023-12-18 22:49:29 +00:00
src/c/pyeclib_c Merge "Drop support for liberasurecode<1.4.0" 2024-04-20 00:36:39 +00:00
test Merge "Drop support for liberasurecode<1.4.0" 2024-04-20 00:36:39 +00:00
tools Merge "Test on CentOS 9 Stream" 2023-08-02 00:03:45 +00:00
.gitignore Update .gitignore for python output 2014-07-05 17:46:55 -07:00
.gitreview Update .gitreview and playbooks following project rename 2019-06-12 11:09:11 -07:00
.mailmap Release 1.5.0 2017-06-06 21:11:13 -04:00
.travis.yml tox related fixes for travis-ci 2015-11-23 03:40:47 +00:00
.unittests Add .unittests script to standardize nosetests invocation 2015-02-24 23:08:45 -07:00
.zuul.yaml Test under py312 2024-04-29 18:14:23 +00:00
AUTHORS Prep for 1.6.1 release 2022-04-13 22:10:27 -07:00
ChangeLog Prep for 1.6.1 release 2022-04-13 22:10:27 -07:00
LICENSE Move license file 2021-11-27 16:47:19 -08:00
MANIFEST.in removed c_eclib from manifest 2014-06-16 13:51:24 -07:00
Makefile Clean py34 shared libraries created during build 2015-08-05 17:45:15 +00:00
README.rst Drop support for liberasurecode<1.4.0 2023-12-18 22:49:29 +00:00
bindep.txt Switch from yasm to nasm 2023-07-11 13:55:29 -07:00
setup.py Test under py312 2024-04-29 18:14:23 +00:00
test-requirements.txt Fix gate 2023-05-01 08:49:55 -07:00
tox.ini pep8: Use default line width 2024-01-26 01:49:00 +09:00



This library provides a simple Python interface for implementing erasure codes and is known to work with Python v2.6, 2.7 and 3.x. To obtain the best possible performance, the library utilizes liberasurecode, which is a C based erasure code library.

PyECLib supports a variety of Erasure Coding backends including the standard Reed-Solomon implementations provided by Jerasure [1], liberasurecode [3], Intel's ISA-L [4] and Phazr.IO's libphazr. It also provides support for a flat XOR-based encoder and decoder (part of liberasurecode) - a class of HD Combination Codes based on "Flat XOR-based erasure codes in storage systems: Constructions, efficient recovery, and tradeoffs" in IEEE MSST 2010[2]). These codes are well-suited to archival use-cases, have a simple construction and require a minimum number of participating disks during single-disk reconstruction (think XOR-based LRC code).


Install pre-requisites:

  • Python 2.6, 2.7 or 3.x (including development packages), argparse, setuptools
  • liberasurecode v1.4.0 or greater [3]
  • Erasure code backend libraries, gf-complete and Jerasure [1],[2], ISA-L [4], etc

Install dependencies:

Debian/Ubuntu hosts:

$ sudo apt-get install build-essential python-dev python-pip liberasurecode-dev
$ sudo pip install -U bindep -r test-requirements.txt

RHEL/CentOS hosts:

$ sudo yum install -y redhat-lsb python2-pip python-devel liberasurecode-devel
$ sudo pip install -U bindep -r test-requirements.txt
$ tools/test-setup.sh

If you want to confirm all dependency packages installed successfully, try:

$ sudo bindep -f bindep.txt

For CentOS, make sure to install the latest Openstack Cloud SIG repo to be able to install the latest available version of liberasurecode-devel.

Install PyECLib:

$ sudo python setup.py install

Run test suite included:

$ ./.unittests

If the test suite fails because it cannot find any of the shared libraries, then you probably need to add /usr/local/lib to the path searched when loading libraries. The best way to do this (on Linux) is to add '/usr/local/lib' to:


and then make sure to run:

$ sudo ldconfig

Getting started

Examples of using PyECLib are provided in the "tools" directory:

Command-line encoder:


Command-line decoder:


Utility to determine what is needed to reconstruct missing fragments:


A configuration utility to help compare available EC schemes in terms of performance and redundancy:


PyEClib initialization:

ec_driver = ECDriver(k=<num_encoded_data_fragments>,

Supported ec_type values:

  • liberasurecode_rs_vand => Vandermonde Reed-Solomon encoding, software-only backend implemented by liberasurecode [3]
  • jerasure_rs_vand => Vandermonde Reed-Solomon encoding, based on Jerasure [1]
  • jerasure_rs_cauchy => Cauchy Reed-Solomon encoding (Jerasure variant), based on Jerasure [1]
  • flat_xor_hd_3, flat_xor_hd_4 => Flat-XOR based HD combination codes, liberasurecode [3]
  • isa_l_rs_vand => Intel Storage Acceleration Library (ISA-L) - SIMD accelerated Erasure Coding backends [4]
  • isa_l_rs_cauchy => Cauchy Reed-Solomon encoding (ISA-L variant) [4]
  • shss => NTT Lab Japan's Erasure Coding Library [5]
  • libphazr => Phazr.IO's erasure code library with built-in privacy [6]

Code Maintenance

This library is currently mainly maintained by the Openstack Swift community. For questions or any other help, come ask in #openstack-swift on OFTC.


[1] Jerasure, C library that supports erasure coding in storage applications, http://jerasure.org

[2] Greenan, Kevin M et al, "Flat XOR-based erasure codes in storage systems", http://www.kaymgee.com/Kevin_Greenan/Publications_files/greenan-msst10.pdf

[3] liberasurecode, C API abstraction layer for erasure coding backends, https://opendev.org/openstack/liberasurecode

[4] Intel(R) Storage Acceleration Library (Open Source Version), https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version

[5] Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>, "NTT SHSS Erasure Coding backend"

[6] Jim Cheung <support@phazr.io>, "Phazr.IO libphazr erasure code backend with built-in privacy"