release 1.4.0

Change-Id: I1e7f96cf96906450ef1323e1f9c48fcfd30f14ff
Signed-off-by: Thiago da Silva <thiago@redhat.com>
This commit is contained in:
Thiago da Silva
2016-12-07 15:02:17 -05:00
parent ca4d2dd7c3
commit baf8be7f92
5 changed files with 49 additions and 35 deletions

View File

@@ -6,6 +6,7 @@ Kevin Greenan (kmgreen2@gmail.com)
Contributors Contributors
------------ ------------
Timur Alperovich (timuralp@swiftstack.com) Timur Alperovich (timuralp@swiftstack.com)
Tim Burke (tim.burke@gmail.com)
Thiago da Silva (thiago@redhat.com) Thiago da Silva (thiago@redhat.com)
Eric Lambert (eric_lambert@xyratex.com) Eric Lambert (eric_lambert@xyratex.com)
Ondřej Nový (ondrej.novy@firma.seznam.cz) Ondřej Nový (ondrej.novy@firma.seznam.cz)

View File

@@ -1,3 +1,10 @@
New in 1.4.0
------------
* Add support for ISA-L Cauchy
* Fixed memory leak in get_metadata
* Added soft warning log line when using liberasurecode <1.3.1
New in 1.3.1 New in 1.3.1
------------ ------------

View File

@@ -1,3 +1,6 @@
PyEClib
-------
This library provides a simple Python interface for implementing erasure codes This library provides a simple Python interface for implementing erasure codes
and is known to work with Python v2.6, 2.7 and 3.x. and is known to work with Python v2.6, 2.7 and 3.x.
@@ -6,28 +9,32 @@ which is a C based erasure code library. Please let us know if you have any
issues building or installing (email: kmgreen2@gmail.com or tusharsg@gmail.com). issues building or installing (email: kmgreen2@gmail.com or tusharsg@gmail.com).
PyECLib supports a variety of Erasure Coding backends including the standard Reed PyECLib supports a variety of Erasure Coding backends including the standard Reed
Soloman implementations provided by Jerasure [2], liberasurecode [3] and Intel Soloman implementations provided by Jerasure [1], liberasurecode [3] and Intel
ISA-L [4]. It also provides support for a flat XOR-based encoder and decoder ISA-L [4]. It also provides support for a flat XOR-based encoder and decoder
(part of liberasurecode) - a class of HD Combination Codes based on "Flat (part of liberasurecode) - a class of HD Combination Codes based on "Flat
XOR-based erasure codes in storage systems: Constructions, efficient recovery, XOR-based erasure codes in storage systems: Constructions, efficient recovery,
and tradeoffs" in IEEE MSST 2010). These codes are well-suited to archival and tradeoffs" in IEEE MSST 2010[2]). These codes are well-suited to archival
use-cases, have a simple construction and require a minimum number of use-cases, have a simple construction and require a minimum number of
participating disks during single-disk reconstruction (think XOR-based LRC code). participating disks during single-disk reconstruction (think XOR-based LRC code).
Examples of using PyECLib are provided in the "tools" directory: Examples of using PyECLib are provided in the "tools" directory:
Command-line encoder:: Command-line encoder::
tools/pyeclib_encode.py tools/pyeclib_encode.py
Command-line decoder:: Command-line decoder::
tools/pyeclib_decode.py tools/pyeclib_decode.py
Utility to determine what is needed to reconstruct missing fragments:: Utility to determine what is needed to reconstruct missing fragments::
tools/pyeclib_fragments_needed.py tools/pyeclib_fragments_needed.py
A configuration utility to help compare available EC schemes in terms of
performance and redundancy::
tools/pyeclib_conf_tool.py
PyEClib initialization:: PyEClib initialization::
@@ -39,13 +46,11 @@ Supported ``ec_type`` values:
* ``liberasurecode_rs_vand`` => Vandermonde Reed-Solomon encoding, software-only backend implemented by liberasurecode [3] * ``liberasurecode_rs_vand`` => Vandermonde Reed-Solomon encoding, software-only backend implemented by liberasurecode [3]
* ``jerasure_rs_vand`` => Vandermonde Reed-Solomon encoding, based on Jerasure [1] * ``jerasure_rs_vand`` => Vandermonde Reed-Solomon encoding, based on Jerasure [1]
* ``jerasure_rs_cauchy`` => Cauchy Reed-Solomon encoding (Jerasure variant), based on Jerasure [2] * ``jerasure_rs_cauchy`` => Cauchy Reed-Solomon encoding (Jerasure variant), based on Jerasure [1]
* ``flat_xor_hd_3``, ``flat_xor_hd_4`` => Flat-XOR based HD combination codes, liberasurecode [3] * ``flat_xor_hd_3``, ``flat_xor_hd_4`` => Flat-XOR based HD combination codes, liberasurecode [3]
* ``isa_l_rs_vand`` => Intel Storage Acceleration Library (ISA-L) - SIMD accelerated Erasure Coding backends [4] * ``isa_l_rs_vand`` => Intel Storage Acceleration Library (ISA-L) - SIMD accelerated Erasure Coding backends [4]
* ``shss`` => NTT Lab Japan's Erasure Coding Library * ``isa_l_rs_cauchy`` => Cauchy Reed-Solomon encoding (ISA-L variant) [4]
* ``shss`` => NTT Lab Japan's Erasure Coding Library [5]
A configuration utility is provided to help compare available EC schemes in
terms of performance and redundancy:: `tools/pyeclib_conf_tool.py`
The Python API supports the following functions: The Python API supports the following functions:
@@ -106,16 +111,16 @@ needs to be taken when handling input to and output from the ``encode()`` and
ECInsufficientFragments - if an insufficient set of fragments has been provided (e.g. not enough) ECInsufficientFragments - if an insufficient set of fragments has been provided (e.g. not enough)
ECInvalidFragmentMetadata - if the fragment headers appear to be corrupted ECInvalidFragmentMetadata - if the fragment headers appear to be corrupted
ECDriverError - if an unknown error occurs ECDriverError - if an unknown error occurs
- Minimum parity fragments needed for durability gurantees:: - Minimum parity fragments needed for durability gurantees::
def min_parity_fragments_needed(self) def min_parity_fragments_needed(self)
NOTE: Currently hard-coded to 1, so this can only be trusted for MDS codes, such as NOTE: Currently hard-coded to 1, so this can only be trusted for MDS codes, such as
Reed-Solomon. Reed-Solomon.
output: minimum number of additional fragments needed to be synchronously written to tolerate output: minimum number of additional fragments needed to be synchronously written to tolerate
the loss of any one fragment (similar guarantees to 2 out of 3 with 3x replication) the loss of any one fragment (similar guarantees to 2 out of 3 with 3x replication)
throws: throws:
ECBackendInstanceNotAvailable - if the backend library cannot be found ECBackendInstanceNotAvailable - if the backend library cannot be found
@@ -124,13 +129,13 @@ needs to be taken when handling input to and output from the ``encode()`` and
ECOutOfMemory - if the process has run out of memory ECOutOfMemory - if the process has run out of memory
ECDriverError - if an unknown error occurs ECDriverError - if an unknown error occurs
- Fragments needed for EC Reconstruct - Fragments needed for EC Reconstruct
Return the indexes of fragments needed to reconstruct "missing_fragment_indexes":: Return the indexes of fragments needed to reconstruct "missing_fragment_indexes"::
def fragments_needed(self, missing_fragment_indexes) def fragments_needed(self, missing_fragment_indexes)
input: list of missing_fragment_indexes input: list of missing_fragment_indexes
output: list of fragments needed to reconstruct fragments listed in missing_fragment_indexes output: list of fragments needed to reconstruct fragments listed in missing_fragment_indexes
throws: throws:
@@ -207,13 +212,13 @@ needs to be taken when handling input to and output from the ``encode()`` and
Assume a range request is given for an object with segment size 3K and Assume a range request is given for an object with segment size 3K and
a 1 MB file:: a 1 MB file::
Ranges = (0, 1), (1, 12), (10, 1000), (0, segment_size-1), Ranges = (0, 1), (1, 12), (10, 1000), (0, segment_size-1),
(1, segment_size+1), (segment_size-1, 2*segment_size) (1, segment_size+1), (segment_size-1, 2*segment_size)
This will return a map keyed on the ranges, where there is a recipe This will return a map keyed on the ranges, where there is a recipe
given for each range:: given for each range::
{ {
(0, 1): {0: (0, 1)}, (0, 1): {0: (0, 1)},
(10, 1000): {0: (10, 1000)}, (10, 1000): {0: (10, 1000)},
@@ -226,30 +231,30 @@ needs to be taken when handling input to and output from the ``encode()`` and
Quick Start Quick Start
Install pre-requisites: Install pre-requisites::
* Python 2.6, 2.7 or 3.x (including development packages), argparse, setuptools * Python 2.6, 2.7 or 3.x (including development packages), argparse, setuptools
* liberasurecode v1.2.0 or greater [3] * liberasurecode v1.2.0 or greater [3]
* Erasure code backend libraries, gf-complete and Jerasure [1],[2], ISA-L [4] etc * Erasure code backend libraries, gf-complete and Jerasure [1],[2], ISA-L [4] etc
An example for ubuntu to install dependency packages: An example for ubuntu to install dependency packages::
$ sudo apt-get install build-essential python-dev python-pip liberasurecode-dev $ sudo apt-get install build-essential python-dev python-pip liberasurecode-dev
$ sudo pip install -U bindep -r test-requirement.txt $ sudo pip install -U bindep -r test-requirement.txt
If you want to confirm all dependency packages installed succuessfully, try: If you want to confirm all dependency packages installed successfully, try::
$ sudo bindep -f bindep.txt $ sudo bindep -f bindep.txt
That shows missing dependency packages for you, http://docs.openstack.org/infra/bindep/ *Note*: currently liberasurecode-dev/liberasurecode-devel in package repo is older than v1.2.0
*Note*: currently liberasurecode-dev/liberasurecode-devel in package repo is older
than v1.2.0
Install PyECLib:: Install PyECLib::
$ sudo python setup.py install
$ sudo python setup.py install
Run test suite included:: Run test suite included::
$ ./.unittests $ ./.unittests
If all of this works, then you should be good to go. If not, send us an email! If all of this works, then you should be good to go. If not, send us an email!
@@ -257,11 +262,11 @@ Quick Start
then you probably need to add /usr/local/lib to the path searched when loading then you probably need to add /usr/local/lib to the path searched when loading
libraries. The best way to do this (on Linux) is to add '/usr/local/lib' to:: libraries. The best way to do this (on Linux) is to add '/usr/local/lib' to::
/etc/ld.so.conf /etc/ld.so.conf
and then make sure to run:: and then make sure to run::
$ sudo ldconfig $ sudo ldconfig
References References

View File

@@ -48,9 +48,9 @@ copyright = u'2016, Kevin Greenan, Tushar Gohad, Kota Tsuyuzaki'
# built documents. # built documents.
# #
# The short X.Y version. # The short X.Y version.
version = '1.3.1' version = '1.4.0'
# The full version, including alpha/beta/rc tags. # The full version, including alpha/beta/rc tags.
release = '1.3.1' release = '1.4.0'
# The language for content autogenerated by Sphinx. Refer to documentation # The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages. # for a list of supported languages.

View File

@@ -159,7 +159,7 @@ class install(_install):
module = Extension('pyeclib_c', module = Extension('pyeclib_c',
define_macros=[('MAJOR VERSION', '1'), define_macros=[('MAJOR VERSION', '1'),
('MINOR VERSION', '3')], ('MINOR VERSION', '4')],
include_dirs=[default_python_incdir, include_dirs=[default_python_incdir,
'src/c/pyeclib_c', 'src/c/pyeclib_c',
'/usr/include', '/usr/include',
@@ -172,12 +172,13 @@ module = Extension('pyeclib_c',
sources=['src/c/pyeclib_c/pyeclib_c.c']) sources=['src/c/pyeclib_c/pyeclib_c.c'])
setup(name='pyeclib', setup(name='pyeclib',
version='1.3.1', version='1.4.0',
author='Kevin Greenan', author='Kevin Greenan',
author_email='kmgreen2@gmail.com', author_email='kmgreen2@gmail.com',
maintainer='Kevin Greenan and Tushar Gohad', maintainer='Kevin Greenan and Tushar Gohad',
maintainer_email='kmgreen2@gmail.com, tusharsg@gmail.com', maintainer_email='kmgreen2@gmail.com, tusharsg@gmail.com',
url='http://git.openstack.org/cgit/openstack/pyeclib/', url='http://git.openstack.org/cgit/openstack/pyeclib/',
bugtrack_url='https://bugs.launchpad.net/pyeclib',
description='This library provides a simple Python interface for \ description='This library provides a simple Python interface for \
implementing erasure codes. To obtain the best possible \ implementing erasure codes. To obtain the best possible \
performance, the underlying erasure code algorithms are \ performance, the underlying erasure code algorithms are \