release 1.4.0
Change-Id: I1e7f96cf96906450ef1323e1f9c48fcfd30f14ff Signed-off-by: Thiago da Silva <thiago@redhat.com>
This commit is contained in:
1
AUTHORS
1
AUTHORS
@@ -6,6 +6,7 @@ Kevin Greenan (kmgreen2@gmail.com)
|
|||||||
Contributors
|
Contributors
|
||||||
------------
|
------------
|
||||||
Timur Alperovich (timuralp@swiftstack.com)
|
Timur Alperovich (timuralp@swiftstack.com)
|
||||||
|
Tim Burke (tim.burke@gmail.com)
|
||||||
Thiago da Silva (thiago@redhat.com)
|
Thiago da Silva (thiago@redhat.com)
|
||||||
Eric Lambert (eric_lambert@xyratex.com)
|
Eric Lambert (eric_lambert@xyratex.com)
|
||||||
Ondřej Nový (ondrej.novy@firma.seznam.cz)
|
Ondřej Nový (ondrej.novy@firma.seznam.cz)
|
||||||
|
@@ -1,3 +1,10 @@
|
|||||||
|
New in 1.4.0
|
||||||
|
------------
|
||||||
|
|
||||||
|
* Add support for ISA-L Cauchy
|
||||||
|
* Fixed memory leak in get_metadata
|
||||||
|
* Added soft warning log line when using liberasurecode <1.3.1
|
||||||
|
|
||||||
New in 1.3.1
|
New in 1.3.1
|
||||||
------------
|
------------
|
||||||
|
|
||||||
|
@@ -1,3 +1,6 @@
|
|||||||
|
PyEClib
|
||||||
|
-------
|
||||||
|
|
||||||
This library provides a simple Python interface for implementing erasure codes
|
This library provides a simple Python interface for implementing erasure codes
|
||||||
and is known to work with Python v2.6, 2.7 and 3.x.
|
and is known to work with Python v2.6, 2.7 and 3.x.
|
||||||
|
|
||||||
@@ -6,28 +9,32 @@ which is a C based erasure code library. Please let us know if you have any
|
|||||||
issues building or installing (email: kmgreen2@gmail.com or tusharsg@gmail.com).
|
issues building or installing (email: kmgreen2@gmail.com or tusharsg@gmail.com).
|
||||||
|
|
||||||
PyECLib supports a variety of Erasure Coding backends including the standard Reed
|
PyECLib supports a variety of Erasure Coding backends including the standard Reed
|
||||||
Soloman implementations provided by Jerasure [2], liberasurecode [3] and Intel
|
Soloman implementations provided by Jerasure [1], liberasurecode [3] and Intel
|
||||||
ISA-L [4]. It also provides support for a flat XOR-based encoder and decoder
|
ISA-L [4]. It also provides support for a flat XOR-based encoder and decoder
|
||||||
(part of liberasurecode) - a class of HD Combination Codes based on "Flat
|
(part of liberasurecode) - a class of HD Combination Codes based on "Flat
|
||||||
XOR-based erasure codes in storage systems: Constructions, efficient recovery,
|
XOR-based erasure codes in storage systems: Constructions, efficient recovery,
|
||||||
and tradeoffs" in IEEE MSST 2010). These codes are well-suited to archival
|
and tradeoffs" in IEEE MSST 2010[2]). These codes are well-suited to archival
|
||||||
use-cases, have a simple construction and require a minimum number of
|
use-cases, have a simple construction and require a minimum number of
|
||||||
participating disks during single-disk reconstruction (think XOR-based LRC code).
|
participating disks during single-disk reconstruction (think XOR-based LRC code).
|
||||||
|
|
||||||
Examples of using PyECLib are provided in the "tools" directory:
|
Examples of using PyECLib are provided in the "tools" directory:
|
||||||
|
|
||||||
Command-line encoder::
|
Command-line encoder::
|
||||||
|
|
||||||
tools/pyeclib_encode.py
|
tools/pyeclib_encode.py
|
||||||
|
|
||||||
Command-line decoder::
|
Command-line decoder::
|
||||||
|
|
||||||
tools/pyeclib_decode.py
|
tools/pyeclib_decode.py
|
||||||
|
|
||||||
Utility to determine what is needed to reconstruct missing fragments::
|
Utility to determine what is needed to reconstruct missing fragments::
|
||||||
|
|
||||||
tools/pyeclib_fragments_needed.py
|
tools/pyeclib_fragments_needed.py
|
||||||
|
|
||||||
|
A configuration utility to help compare available EC schemes in terms of
|
||||||
|
performance and redundancy::
|
||||||
|
|
||||||
|
tools/pyeclib_conf_tool.py
|
||||||
|
|
||||||
PyEClib initialization::
|
PyEClib initialization::
|
||||||
|
|
||||||
@@ -39,13 +46,11 @@ Supported ``ec_type`` values:
|
|||||||
|
|
||||||
* ``liberasurecode_rs_vand`` => Vandermonde Reed-Solomon encoding, software-only backend implemented by liberasurecode [3]
|
* ``liberasurecode_rs_vand`` => Vandermonde Reed-Solomon encoding, software-only backend implemented by liberasurecode [3]
|
||||||
* ``jerasure_rs_vand`` => Vandermonde Reed-Solomon encoding, based on Jerasure [1]
|
* ``jerasure_rs_vand`` => Vandermonde Reed-Solomon encoding, based on Jerasure [1]
|
||||||
* ``jerasure_rs_cauchy`` => Cauchy Reed-Solomon encoding (Jerasure variant), based on Jerasure [2]
|
* ``jerasure_rs_cauchy`` => Cauchy Reed-Solomon encoding (Jerasure variant), based on Jerasure [1]
|
||||||
* ``flat_xor_hd_3``, ``flat_xor_hd_4`` => Flat-XOR based HD combination codes, liberasurecode [3]
|
* ``flat_xor_hd_3``, ``flat_xor_hd_4`` => Flat-XOR based HD combination codes, liberasurecode [3]
|
||||||
* ``isa_l_rs_vand`` => Intel Storage Acceleration Library (ISA-L) - SIMD accelerated Erasure Coding backends [4]
|
* ``isa_l_rs_vand`` => Intel Storage Acceleration Library (ISA-L) - SIMD accelerated Erasure Coding backends [4]
|
||||||
* ``shss`` => NTT Lab Japan's Erasure Coding Library
|
* ``isa_l_rs_cauchy`` => Cauchy Reed-Solomon encoding (ISA-L variant) [4]
|
||||||
|
* ``shss`` => NTT Lab Japan's Erasure Coding Library [5]
|
||||||
A configuration utility is provided to help compare available EC schemes in
|
|
||||||
terms of performance and redundancy:: `tools/pyeclib_conf_tool.py`
|
|
||||||
|
|
||||||
|
|
||||||
The Python API supports the following functions:
|
The Python API supports the following functions:
|
||||||
@@ -106,16 +111,16 @@ needs to be taken when handling input to and output from the ``encode()`` and
|
|||||||
ECInsufficientFragments - if an insufficient set of fragments has been provided (e.g. not enough)
|
ECInsufficientFragments - if an insufficient set of fragments has been provided (e.g. not enough)
|
||||||
ECInvalidFragmentMetadata - if the fragment headers appear to be corrupted
|
ECInvalidFragmentMetadata - if the fragment headers appear to be corrupted
|
||||||
ECDriverError - if an unknown error occurs
|
ECDriverError - if an unknown error occurs
|
||||||
|
|
||||||
|
|
||||||
- Minimum parity fragments needed for durability gurantees::
|
- Minimum parity fragments needed for durability gurantees::
|
||||||
|
|
||||||
def min_parity_fragments_needed(self)
|
def min_parity_fragments_needed(self)
|
||||||
|
|
||||||
NOTE: Currently hard-coded to 1, so this can only be trusted for MDS codes, such as
|
NOTE: Currently hard-coded to 1, so this can only be trusted for MDS codes, such as
|
||||||
Reed-Solomon.
|
Reed-Solomon.
|
||||||
|
|
||||||
output: minimum number of additional fragments needed to be synchronously written to tolerate
|
output: minimum number of additional fragments needed to be synchronously written to tolerate
|
||||||
the loss of any one fragment (similar guarantees to 2 out of 3 with 3x replication)
|
the loss of any one fragment (similar guarantees to 2 out of 3 with 3x replication)
|
||||||
throws:
|
throws:
|
||||||
ECBackendInstanceNotAvailable - if the backend library cannot be found
|
ECBackendInstanceNotAvailable - if the backend library cannot be found
|
||||||
@@ -124,13 +129,13 @@ needs to be taken when handling input to and output from the ``encode()`` and
|
|||||||
ECOutOfMemory - if the process has run out of memory
|
ECOutOfMemory - if the process has run out of memory
|
||||||
ECDriverError - if an unknown error occurs
|
ECDriverError - if an unknown error occurs
|
||||||
|
|
||||||
|
|
||||||
- Fragments needed for EC Reconstruct
|
- Fragments needed for EC Reconstruct
|
||||||
|
|
||||||
Return the indexes of fragments needed to reconstruct "missing_fragment_indexes"::
|
Return the indexes of fragments needed to reconstruct "missing_fragment_indexes"::
|
||||||
|
|
||||||
def fragments_needed(self, missing_fragment_indexes)
|
def fragments_needed(self, missing_fragment_indexes)
|
||||||
|
|
||||||
input: list of missing_fragment_indexes
|
input: list of missing_fragment_indexes
|
||||||
output: list of fragments needed to reconstruct fragments listed in missing_fragment_indexes
|
output: list of fragments needed to reconstruct fragments listed in missing_fragment_indexes
|
||||||
throws:
|
throws:
|
||||||
@@ -207,13 +212,13 @@ needs to be taken when handling input to and output from the ``encode()`` and
|
|||||||
|
|
||||||
Assume a range request is given for an object with segment size 3K and
|
Assume a range request is given for an object with segment size 3K and
|
||||||
a 1 MB file::
|
a 1 MB file::
|
||||||
|
|
||||||
Ranges = (0, 1), (1, 12), (10, 1000), (0, segment_size-1),
|
Ranges = (0, 1), (1, 12), (10, 1000), (0, segment_size-1),
|
||||||
(1, segment_size+1), (segment_size-1, 2*segment_size)
|
(1, segment_size+1), (segment_size-1, 2*segment_size)
|
||||||
|
|
||||||
This will return a map keyed on the ranges, where there is a recipe
|
This will return a map keyed on the ranges, where there is a recipe
|
||||||
given for each range::
|
given for each range::
|
||||||
|
|
||||||
{
|
{
|
||||||
(0, 1): {0: (0, 1)},
|
(0, 1): {0: (0, 1)},
|
||||||
(10, 1000): {0: (10, 1000)},
|
(10, 1000): {0: (10, 1000)},
|
||||||
@@ -226,30 +231,30 @@ needs to be taken when handling input to and output from the ``encode()`` and
|
|||||||
|
|
||||||
Quick Start
|
Quick Start
|
||||||
|
|
||||||
Install pre-requisites:
|
Install pre-requisites::
|
||||||
|
|
||||||
* Python 2.6, 2.7 or 3.x (including development packages), argparse, setuptools
|
* Python 2.6, 2.7 or 3.x (including development packages), argparse, setuptools
|
||||||
* liberasurecode v1.2.0 or greater [3]
|
* liberasurecode v1.2.0 or greater [3]
|
||||||
* Erasure code backend libraries, gf-complete and Jerasure [1],[2], ISA-L [4] etc
|
* Erasure code backend libraries, gf-complete and Jerasure [1],[2], ISA-L [4] etc
|
||||||
|
|
||||||
An example for ubuntu to install dependency packages:
|
An example for ubuntu to install dependency packages::
|
||||||
|
|
||||||
$ sudo apt-get install build-essential python-dev python-pip liberasurecode-dev
|
$ sudo apt-get install build-essential python-dev python-pip liberasurecode-dev
|
||||||
$ sudo pip install -U bindep -r test-requirement.txt
|
$ sudo pip install -U bindep -r test-requirement.txt
|
||||||
|
|
||||||
If you want to confirm all dependency packages installed succuessfully, try:
|
If you want to confirm all dependency packages installed successfully, try::
|
||||||
|
|
||||||
$ sudo bindep -f bindep.txt
|
$ sudo bindep -f bindep.txt
|
||||||
|
|
||||||
That shows missing dependency packages for you, http://docs.openstack.org/infra/bindep/
|
*Note*: currently liberasurecode-dev/liberasurecode-devel in package repo is older than v1.2.0
|
||||||
|
|
||||||
*Note*: currently liberasurecode-dev/liberasurecode-devel in package repo is older
|
|
||||||
than v1.2.0
|
|
||||||
|
|
||||||
Install PyECLib::
|
Install PyECLib::
|
||||||
$ sudo python setup.py install
|
|
||||||
|
$ sudo python setup.py install
|
||||||
|
|
||||||
Run test suite included::
|
Run test suite included::
|
||||||
|
|
||||||
$ ./.unittests
|
$ ./.unittests
|
||||||
|
|
||||||
If all of this works, then you should be good to go. If not, send us an email!
|
If all of this works, then you should be good to go. If not, send us an email!
|
||||||
|
|
||||||
@@ -257,11 +262,11 @@ Quick Start
|
|||||||
then you probably need to add /usr/local/lib to the path searched when loading
|
then you probably need to add /usr/local/lib to the path searched when loading
|
||||||
libraries. The best way to do this (on Linux) is to add '/usr/local/lib' to::
|
libraries. The best way to do this (on Linux) is to add '/usr/local/lib' to::
|
||||||
|
|
||||||
/etc/ld.so.conf
|
/etc/ld.so.conf
|
||||||
|
|
||||||
and then make sure to run::
|
and then make sure to run::
|
||||||
|
|
||||||
$ sudo ldconfig
|
$ sudo ldconfig
|
||||||
|
|
||||||
|
|
||||||
References
|
References
|
@@ -48,9 +48,9 @@ copyright = u'2016, Kevin Greenan, Tushar Gohad, Kota Tsuyuzaki'
|
|||||||
# built documents.
|
# built documents.
|
||||||
#
|
#
|
||||||
# The short X.Y version.
|
# The short X.Y version.
|
||||||
version = '1.3.1'
|
version = '1.4.0'
|
||||||
# The full version, including alpha/beta/rc tags.
|
# The full version, including alpha/beta/rc tags.
|
||||||
release = '1.3.1'
|
release = '1.4.0'
|
||||||
|
|
||||||
# The language for content autogenerated by Sphinx. Refer to documentation
|
# The language for content autogenerated by Sphinx. Refer to documentation
|
||||||
# for a list of supported languages.
|
# for a list of supported languages.
|
||||||
|
5
setup.py
5
setup.py
@@ -159,7 +159,7 @@ class install(_install):
|
|||||||
|
|
||||||
module = Extension('pyeclib_c',
|
module = Extension('pyeclib_c',
|
||||||
define_macros=[('MAJOR VERSION', '1'),
|
define_macros=[('MAJOR VERSION', '1'),
|
||||||
('MINOR VERSION', '3')],
|
('MINOR VERSION', '4')],
|
||||||
include_dirs=[default_python_incdir,
|
include_dirs=[default_python_incdir,
|
||||||
'src/c/pyeclib_c',
|
'src/c/pyeclib_c',
|
||||||
'/usr/include',
|
'/usr/include',
|
||||||
@@ -172,12 +172,13 @@ module = Extension('pyeclib_c',
|
|||||||
sources=['src/c/pyeclib_c/pyeclib_c.c'])
|
sources=['src/c/pyeclib_c/pyeclib_c.c'])
|
||||||
|
|
||||||
setup(name='pyeclib',
|
setup(name='pyeclib',
|
||||||
version='1.3.1',
|
version='1.4.0',
|
||||||
author='Kevin Greenan',
|
author='Kevin Greenan',
|
||||||
author_email='kmgreen2@gmail.com',
|
author_email='kmgreen2@gmail.com',
|
||||||
maintainer='Kevin Greenan and Tushar Gohad',
|
maintainer='Kevin Greenan and Tushar Gohad',
|
||||||
maintainer_email='kmgreen2@gmail.com, tusharsg@gmail.com',
|
maintainer_email='kmgreen2@gmail.com, tusharsg@gmail.com',
|
||||||
url='http://git.openstack.org/cgit/openstack/pyeclib/',
|
url='http://git.openstack.org/cgit/openstack/pyeclib/',
|
||||||
|
bugtrack_url='https://bugs.launchpad.net/pyeclib',
|
||||||
description='This library provides a simple Python interface for \
|
description='This library provides a simple Python interface for \
|
||||||
implementing erasure codes. To obtain the best possible \
|
implementing erasure codes. To obtain the best possible \
|
||||||
performance, the underlying erasure code algorithms are \
|
performance, the underlying erasure code algorithms are \
|
||||||
|
Reference in New Issue
Block a user