diff --git a/doc/source/apis.rst b/doc/source/apis.rst
new file mode 100644
index 00000000..1a8e8f7d
--- /dev/null
+++ b/doc/source/apis.rst
@@ -0,0 +1,719 @@
+============
+Introduction
+============
+
+The python-swiftclient includes two levels of API; a low level client API that
+provides simple python wrappers around the various authentication mechanisms
+and the individual HTTP requests, and a high level service API that provides
+methods for performing common operations in parallel on a thread pool.
+
+This document aims to provide guidance for choosing between these APIs and
+examples of usage for the service API.
+
+------------------------
+Important Considerations
+------------------------
+
+This section covers some important considerations, helpful hints, and things
+to avoid when integrating an object store into your workflow.
+
+An Object Store is not a filesystem
+-----------------------------------
+
+It cannot be stressed enough that your usage of the object store should reflect
+the proper use case, and not treat the storage like a filesystem. There are 2
+main restrictions to bear in mind here when designing your use of the object
+store:
+
+ * Objects cannot be renamed due to the way in which objects are stored and
+ references by the object store. This usually requires multiple copies of
+ the data to be moved between physical storage devices.
+ As a result, a move operation is not provided. If the user wants to move an
+ object they must re-upload to the new location and delete the
+ original.
+ * Objects cannot be modified. Objects are stored in multiple locations and are
+ checked for integrity based on the ``MD5 sum`` calculated during upload.
+ Object creation is a 1-shot event, and in order to modify the contents of an
+ object the entire new contents must be re-uploaded. In certain special cases
+ it is possible to work around this restriction using large objects, but no
+ general file-like access is available to modify a stored object.
+
+------------------------------
+The swiftclient.Connection API
+------------------------------
+
+A low level API that provides methods for authentication and methods that
+correspond to the individual REST API calls described in the swift
+documentation.
+
+For usage details see the client docs: :mod:`swiftclient.client`.
+
+--------------------------------
+The swiftclient.SwiftService API
+--------------------------------
+
+A higher level API aimed at allowing developers an easy way to perform multiple
+operations asynchronously using a configurable thread pool. Docs for each
+service method call can be found here: :mod:`swiftclient.service`.
+
+Configuration
+-------------
+
+When you create an instance of a ``SwiftService``, you can override a collection
+of default options to suit your use case. Typically, the defaults are sensible to
+get us started, but depending on your needs you might want to tweak them to
+improve performance (options affecting large objects and thread counts can
+significantly alter performance in the right situation).
+
+Service level defaults and some extra options can also be overridden on a
+per-operation (or even in some cases per-object) basis, and you will call out
+which options affect which operations later in the document.
+
+The configuration of the service API is performed using an options dictionary
+passed to the ``SwiftService`` during initialisation. The options available
+in this dictionary are described below, along with their defaults:
+
+Options
+~~~~~~~
+
+ ``retries``: ``5``
+ The number of times that the library should attempt to retry HTTP
+ actions before giving up and reporting a failure.
+
+ ``container_threads``: ``10``
+
+ ``object_dd_threads``: ``10``
+
+ ``object_uu_threads``: ``10``
+
+ ``segment_threads``: ``10``
+ The above options determine the size of the available thread pools for
+ performing swift operations. Container operations (such as listing a
+ container) operate in the container threads, and a similar pattern
+ applies to object and segment threads.
+
+ .. note::
+
+ Object threads are separated into two separate thread pools:
+ ``uu`` and ``dd``. This stands for "upload/update" and "download/delete",
+ and the corresponding actions will be run on separate threads pools.
+
+ ``segment_size``: ``None``
+ If specified, this option enables uploading of large objects. Should the
+ object being uploaded be larger than 5G in size, this option is
+ mandatory otherwise the upload will fail. This option should be
+ specified as a size in bytes.
+
+ ``use_slo``: ``False``
+ Used in combination with the above option, ``use_slo`` will upload large
+ objects as static rather than dynamic. Only static large objects provide
+ error checking for the downloaded object, so we recommend this option.
+
+ ``segment_container``: ``None``
+ Allows the user to select the container into which large object segments
+ will be uploaded. We do not recommend changing this value as it could make
+ locating orphaned segments more difficult in the case of errors.
+
+ ``leave_segments``: ``False``
+ Setting this option to true means that when deleting or overwriting a large
+ object, its segments will be left in the object store and must be cleaned
+ up manually. This option can be useful when sharing large object segments
+ between multiple objects in more advanced scenarios, but must be treated
+ with care, as it could lead to ever increasing storage usage.
+
+ ``changed``: ``None``
+ This option affects uploads and simply means that those objects which
+ already exist in the object store will not be overwritten if the ``mtime``
+ and size of the source is the same as the existing object.
+
+ ``skip_identical``: ``False``
+ A slightly more thorough case of the above, but rather than ``mtime`` and size
+ uses an object's ``MD5 sum``.
+
+ ``yes_all``: ``False``
+ This options affects only download and delete, and in each case must be
+ specified in order to download/delete the entire contents of an account.
+ This option has no effect on any other calls.
+
+ ``no_download``: ``False``
+ This option only affects download and means that all operations proceed as
+ normal with the exception that no data is written to disk.
+
+ ``header``: ``[]``
+ Used with upload and post operations to set headers on objects. Headers
+ are specified as colon separated strings, e.g. "content-type:text/plain".
+
+ ``meta``: ``[]``
+ Used to set metadata on an object similarly to headers.
+
+ .. note::
+ Setting metadata is a destructive operation, so when updating one
+ of many metadata values all desired metadata for an object must be re-applied.
+
+ ``long``: ``False``
+ Affects only list operations, and results in more metrics being made
+ available in the results at the expense of lower performance.
+
+ ``fail_fast``: ``False``
+ Applies to delete and upload operations, and attempts to abort queued
+ tasks in the event of errors.
+
+ ``prefix``: ``None``
+ Affects list operations; only objects with the given prefix will be
+ returned/affected. It is not advisable to set at the service level, as
+ those operations that call list to discover objects on which they should
+ operate will also be affected.
+
+ ``delimiter``: ``None``
+ Affects list operations, and means that listings only contain results up
+ to the first instance of the delimiter in the object name. This is useful
+ for working with objects containing '/' in their names to simulate folder
+ structures.
+
+ ``dir_marker``: ``False``
+ Affects uploads, and allows empty 'pseudofolder' objects to be created
+ when the source of an upload is ``None``.
+
+ ``shuffle``: ``False``
+ When downloading objects, the default behaviour of the CLI is to shuffle
+ lists of objects in order to spread the load on storage drives when multiple
+ clients are downloading the same files to multiple locations (e.g. in the
+ event of distributing an update). When using the ``SwiftService`` directly,
+ object downloads are scheduled in the same order as they appear in the container
+ listing. When combined with a single download thread this means that objects
+ are downloaded in lexically-sorted order. Setting this option to ``True``
+ gives the same shuffling behaviour as the CLI.
+
+Other available options can be found in ``swiftclient/service.py`` in the
+source code for ``python-swiftclient``. Each ``SwiftService`` method also allows
+for an optional dictionary to override those specified at init time, and the
+appropriate docstrings show which options modify each method's behaviour.
+
+Authentication
+--------------
+
+This section covers the various options for authenticating with a swift
+object store. The combinations of options required for each authentication
+version are detailed below.
+
+Version 1.0 Auth
+~~~~~~~~~~~~~~~~
+
+ ``auth_version``: ``environ.get('ST_AUTH_VERSION')``
+
+ ``auth``: ``environ.get('ST_AUTH')``
+
+ ``user``: ``environ.get('ST_USER')``
+
+ ``key``: ``environ.get('ST_KEY')``
+
+
+Version 2.0 & 3.0 Auth
+~~~~~~~~~~~~~~~~~~~~~~
+
+ ``auth_version``: ``environ.get('ST_AUTH_VERSION')``
+
+ ``os_username``: ``environ.get('OS_USERNAME')``
+
+ ``os_password``: ``environ.get('OS_PASSWORD')``
+
+ ``os_tenant_name``: ``environ.get('OS_TENANT_NAME')``
+
+ ``os_auth_url``: ``environ.get('OS_AUTH_URL')``
+
+As is evident from the default values, if these options are not set explicitly
+in the options dictionary, then they will default to the values of the given
+environment variables. The ``SwiftService`` authentication automatically selects
+the auth version based on the combination of options specified, but
+having options from different auth versions can cause unexpected behaviour.
+
+ .. note::
+
+ Leftover environment variables are a common source of confusion when
+ authorization fails.
+
+Operation Return Values
+-----------------------
+
+Each operation provided by the service API may raise a ``SwiftError`` or
+``ClientException`` for any call that fails completely (or a call which
+performs only one operation at an account or container level). In the case of a
+successful call an operation returns one of the following:
+
+* A dictionary detailing the results of a single operation.
+* An iterator that produces result dictionaries (for calls that perform
+ multiple sub-operations).
+
+A result dictionary can indicate either the success or failure of an individual
+operation (detailed in the ``success`` key), and will either contain the
+successful result, or an ``error`` key detailing the error encountered
+(usually an instance of Exception).
+
+An example result dictionary is given below:
+
+.. code-block:: python
+
+ result = {
+ 'action': 'download_object',
+ 'success': True,
+ 'container': container,
+ 'object': obj,
+ 'path': path,
+ 'start_time': start_time,
+ 'finish_time': finish_time,
+ 'headers_receipt': headers_receipt,
+ 'auth_end_time': conn.auth_end_time,
+ 'read_length': bytes_read,
+ 'attempts': conn.attempts
+ }
+
+All the possible ``action`` values are detailed below:
+
+.. code-block:: python
+
+ [
+ 'stat_account',
+ 'stat_container',
+ 'stat_object',
+ 'post_account',
+ 'post_container',
+ 'post_object',
+ 'list_part', # list yields zero or more 'list_part' results
+ 'download_object',
+ 'create_container', # from upload
+ 'create_dir_marker', # from upload
+ 'upload_object',
+ 'upload_segment',
+ 'delete_container',
+ 'delete_object',
+ 'delete_segment', # from delete_object operations
+ 'capabilities',
+ ]
+
+Stat
+----
+
+Stat can be called against an account, a container, or a list of objects to
+get account stats, container stats or information about the given objects. In
+the first two cases a dictionary is returned containing the results of the
+operation, and in the case of a list of object names being supplied, an
+iterator over the results generated for each object is returned.
+
+Information returned includes the amount of data used by the given
+object/container/account and any headers or metadata set (this includes
+user set data as well as content-type and modification times).
+
+See :mod:`swiftclient.service.SwiftService.stat` for docs generated from the
+method docstring.
+
+Valid calls for this method are as follows:
+
+ * ``stat([options])``: Returns stats for the configured account.
+ * ``stat(<container>, [options])``: Returns stats for the given container.
+ * ``stat(<container>, <object_list>, [options])``: Returns stats for each
+ of the given objects in the the given container (through the returned
+ iterator).
+
+Results from stat are dictionaries indicating the success or failure of each
+operation. In the case of a successful stat against an account or container,
+the method returns immediately with one of the following results:
+
+.. code-block:: python
+
+ {
+ 'action': 'stat_account',
+ 'success': True,
+ 'items': items,
+ 'headers': headers
+ }
+
+.. code-block:: python
+
+ {
+ 'action': 'stat_container',
+ 'container': <container>,
+ 'success': True,
+ 'items': items,
+ 'headers': headers
+ }
+
+In the case of stat called against a list of objects, the method returns a
+generator that returns the results of individual object stat operations as they
+are performed on the thread pool:
+
+.. code-block:: python
+
+ {
+ 'action': 'stat_object',
+ 'object': <object_name>,
+ 'container': <container>,
+ 'success': True,
+ 'items': items,
+ 'headers': headers
+ }
+
+In the case of a failure the dictionary returned will indicate that the
+operation was not successful, and will include the keys below:
+
+.. code-block:: python
+
+ {
+ 'action': <'stat_object'|'stat_container'|'stat_account'>,
+ 'object': <'object_name'>, # Only for stat with objects list
+ 'container': <container>, # Only for stat with objects list or container
+ 'success': False,
+ 'error': <error>,
+ 'traceback': <trace>,
+ 'error_timestamp': <timestamp>
+ }
+
+Example
+~~~~~~~
+
+The code below demonstrates the use of ``stat`` to retrieve the headers for a
+given list of objects in a container using 20 threads. The code creates a
+mapping from object name to headers.
+
+.. code-block:: python
+
+ import logging
+
+ from swiftclient.service import SwiftService
+
+ logger = logging.getLogger()
+ _opts = {'object_dd_threads': 20}
+ with SwiftService(options=_opts) as swift:
+ container = 'container1'
+ objects = [ 'object_%s' % n for n in range(0,100) ]
+ header_data = {}
+ stats_it = swift.stat(container=container, objects=objects)
+ for stat_res in stats_it:
+ if stat_res['success']:
+ header_data[stat_res['object']] = stat_res['headers']
+ else:
+ logger.error(
+ 'Failed to retrieve stats for %s' % stat_res['object']
+ )
+
+List
+----
+
+List can be called against an account or a container to retrieve the containers
+or objects contained within them. Each call returns an iterator that returns
+pages of results (by default, up to 10000 results in each page).
+
+See :mod:`swiftclient.service.SwiftService.list` for docs generated from the
+method docstring.
+
+If the given container or account does not exist, the list method will raise
+a ``SwiftError``, but for all other success/failures a dictionary is returned.
+Each successfully listed page returns a dictionary as described below:
+
+.. code-block:: python
+
+ {
+ 'action': <'list_account_part'|'list_container_part'>,
+ 'container': <container>, # Only for listing a container
+ 'prefix': <prefix>, # The prefix of returned objects/containers
+ 'success': True,
+ 'listing': [Item], # A list of results
+ # (only in the event of success)
+ 'marker': <marker> # The last item name in the list
+ # (only in the event of success)
+ }
+
+Where an item contains the following keys:
+
+.. code-block:: python
+
+ {
+ 'name': <name>,
+ 'bytes': 10485760,
+ 'last_modified': '2014-12-11T12:02:38.774540',
+ 'hash': 'fb938269cbeabe4c234e1127bbd3b74a',
+ 'content_type': 'application/octet-stream',
+ 'meta': <metadata> # Full metadata listing from stat'ing each object
+ # this key only exists if 'long' is specified in options
+ }
+
+Any failure listing an account or container that exists will return a failure
+dictionary as described below:
+
+.. code-block:: python
+
+ {
+ 'action': <'list_account_part'|'list_container_part'>,,
+ 'container': container, # Only for listing a container
+ 'prefix': options['prefix'],
+ 'success': success,
+ 'marker': marker,
+ 'error': error,
+ 'traceback': <trace>,
+ 'error_timestamp': <timestamp>
+ }
+
+Example
+~~~~~~~
+
+The code below demonstrates the use of ``list`` to list all items in a
+container that are over 10MiB in size:
+
+.. code-block:: python
+
+ container = 'example_container'
+ minimum_size = 10*1024**2
+ with SwiftService() as swift:
+ try:
+ stats_parts_gen = swift.list(container=container)
+ for stats in stats_parts_gen:
+ if stats["success"]:
+ for item in stats["listing"]:
+ i_size = int(item["bytes"])
+ if i_size > minimum_size:
+ i_name = item["name"]
+ i_etag = item["hash"]
+ print(
+ "%s [size: %s] [etag: %s]" %
+ (i_name, i_size, i_etag)
+ )
+ else:
+ raise stats["error"]
+ except SwiftError as e:
+ output_manager.error(e.value)
+
+Post
+----
+
+Post can be called against an account, container or list of objects in order to
+update the metadata attached to the given items. Each element of the object list
+may be a plain string of the object name, or a ``SwiftPostObject`` that
+allows finer control over the options applied to each of the individual post
+operations. In the first two cases a single dictionary is returned containing the
+results of the operation, and in the case of a list of objects being supplied,
+an iterator over the results generated for each object post is returned. If the
+given container or account does not exist, the ``post`` method will raise a
+``SwiftError``.
+
+When a string is given for the object name, the options
+
+Successful metadata update results are dictionaries as described below:
+
+.. code-block:: python
+
+ {
+ 'action': <'post_account'|<'post_container'>|'post_object'>,
+ 'success': True,
+ 'container': <container>,
+ 'object': <object>,
+ 'headers': {},
+ 'response_dict': <HTTP response details>
+ }
+
+.. note::
+ Updating user metadata keys will not only add any specified keys, but
+ will also remove user metadata that has previously been set. This means
+ that each time user metadata is updated, the complete set of desired
+ key-value pairs must be specified.
+
+Example
+~~~~~~~
+
+.. Do we want to hide this section until it is complete?
+
+TBD
+
+Download
+--------
+
+.. Do we want to hide this section until it is complete?
+
+TBD
+
+Example
+~~~~~~~
+
+.. Do we want to hide this section until it is complete?
+
+TBD
+
+Upload
+------
+
+Upload is always called against an account and container and with a list of
+objects to upload. Each element of the object list may be a plain string
+detailing the path of the object to upload, or a ``SwiftUploadObject`` that
+allows finer control over some aspects of the individual operations.
+
+When a simple string is supplied to specify a file to upload, the name of the
+object uploaded is the full path of the specified file and the options used for
+the upload are those supplied to the call to ``upload``.
+
+Constructing a ``SwiftUploadObject`` allows the user to supply an object name
+for the uploaded file, and modify the options used by ``upload`` at the
+granularity of invidivual files.
+
+If the given container or account does not exist, the ``upload`` method will
+raise a ``SwiftError``, otherwise an iterator over the results generated for
+each object upload is returned.
+
+See :mod:`swiftclient.service.SwiftService.upload` for docs generated from the
+method docstring.
+
+For each successfully uploaded object (or object segment), the results returned
+by the iterator will be a dictionary as described below:
+
+.. code-block:: python
+
+ {
+ 'action': 'upload_object',
+ 'container': <container>,
+ 'object': <object name>,
+ 'success': True,
+ 'status': <'uploaded'|'skipped-identical'|'skipped-changed'>,
+ 'attempts': <attempt count>,
+ 'response_dict': <HTTP response details>
+ }
+
+ {
+ 'action': 'upload_segment',
+ 'for_container': <container>,
+ 'for_object': <object name>,
+ 'segment_index': <segment_index>,
+ 'segment_size': <segment_size>,
+ 'segment_location': <segment_path>
+ 'segment_etag': <etag>,
+ 'log_line': <object segment n>
+ 'success': True,
+ 'response_dict': <HTTP response details>,
+ 'attempts': <attempt count>
+ }
+
+Any failure uploading an object will return a failure dictionary as described
+below:
+
+.. code-block:: python
+
+ {
+ 'action': 'upload_object',
+ 'container': <container>,
+ 'object': <object name>,
+ 'success': False,
+ 'attempts': <attempt count>,
+ 'error': <error>,
+ 'traceback': <trace>,
+ 'error_timestamp': <timestamp>,
+ 'response_dict': <HTTP response details>
+ }
+
+ {
+ 'action': 'upload_segment',
+ 'for_container': <container>,
+ 'for_object': <object name>,
+ 'segment_index': <segment_index>,
+ 'segment_size': <segment_size>,
+ 'segment_location': <segment_path>,
+ 'log_line': <object segment n>,
+ 'success': False,
+ 'error': <error>,
+ 'traceback': <trace>,
+ 'error_timestamp': <timestamp>,
+ 'response_dict': <HTTP response details>,
+ 'attempts': <attempt count>
+ }
+
+Example
+~~~~~~~
+
+The code below demonstrates the use of ``upload`` to upload all files and
+folders in ``/tmp``, and renaming each object by replacing ``/tmp`` in the
+object or directory marker names with ``temporary-objects``:
+
+.. code-block:: python
+
+ _opts['object_uu_threads'] = 20
+ with SwiftService(options=_opts) as swift, OutputManager() as out_manager:
+ try:
+ # Collect all the files and folders in '/tmp'
+ objs = []
+ dir_markers = []
+ dir = '/tmp':
+ for (_dir, _ds, _fs) in walk(f):
+ if not (_ds + _fs):
+ dir_markers.append(_dir)
+ else:
+ objs.extend([join(_dir, _f) for _f in _fs])
+
+ # Now that we've collected all the required files and dir markers
+ # build the ``SwiftUploadObject``s for the call to upload
+ objs = [
+ SwiftUploadObject(
+ o, object_name=o.replace(
+ '/tmp', 'temporary-objects', 1
+ )
+ ) for o in objs
+ ]
+ dir_markers = [
+ SwiftUploadObject(
+ None, object_name=d.replace(
+ '/tmp', 'temporary-objects', 1
+ ), options={'dir_marker': True}
+ ) for d in dir_markers
+ ]
+
+ # Schedule uploads on the SwiftService thread pool and iterate
+ # over the results
+ for r in swift.upload(container, objs + dir_markers):
+ if r['success']:
+ if 'object' in r:
+ out_manager.print_msg(r['object'])
+ elif 'for_object' in r:
+ out_manager.print_msg(
+ '%s segment %s' % (r['for_object'],
+ r['segment_index'])
+ )
+ else:
+ error = r['error']
+ if r['action'] == "create_container":
+ out_manager.warning(
+ 'Warning: failed to create container '
+ "'%s'%s", container, msg
+ )
+ elif r['action'] == "upload_object":
+ out_manager.error(
+ "Failed to upload object %s to container %s: %s" %
+ (container, r['object'], error)
+ )
+ else:
+ out_manager.error("%s" % error)
+
+ except SwiftError as e:
+ out_manager.error(e.value)
+
+Delete
+------
+
+.. Do we want to hide this section until it is complete?
+
+TBD
+
+Example
+~~~~~~~
+
+.. Do we want to hide this section until it is complete?
+
+TBD
+
+Capabilities
+------------
+
+.. Do we want to hide this section until it is complete?
+
+TBD
+
+Example
+~~~~~~~
+
+.. Do we want to hide this section until it is complete?
+
+TBD
+
diff --git a/doc/source/index.rst b/doc/source/index.rst
index 55ec112a..3b8535af 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -1,5 +1,31 @@
-SwiftClient Web
-***************
+Welcome to the python-swiftclient Docs
+**************************************
+
+Developer Documentation
+=======================
+
+.. toctree::
+ :maxdepth: 2
+
+ apis
+
+Code-Generated Documentation
+============================
+
+.. toctree::
+ :maxdepth: 2
+
+ swiftclient
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
+
+License
+=======
Copyright 2013 OpenStack, LLC.
@@ -15,18 +41,3 @@ SwiftClient Web
See the License for the specific language governing permissions and
limitations under the License.
-Code-Generated Documentation
-============================
-
-.. toctree::
- :maxdepth: 2
-
- swiftclient
-
-
-Indices and tables
-==================
-
-* :ref:`genindex`
-* :ref:`modindex`
-* :ref:`search`
diff --git a/swiftclient/client.py b/swiftclient/client.py
index 2e0cf72a..bd41ec1b 100644
--- a/swiftclient/client.py
+++ b/swiftclient/client.py
@@ -452,7 +452,6 @@ def get_auth(auth_url, user, key, **kwargs):
auth_version = kwargs.get('auth_version', '1')
os_options = kwargs.get('os_options', {})
- storage_url, token = None, None
cacert = kwargs.get('cacert', None)
insecure = kwargs.get('insecure', False)
timeout = kwargs.get('timeout', None)
diff --git a/swiftclient/service.py b/swiftclient/service.py
index 3f1f7e59..8df13897 100644
--- a/swiftclient/service.py
+++ b/swiftclient/service.py
@@ -92,7 +92,7 @@ def process_options(options):
# Use new-style args if old ones not present
if not options['auth'] and options['os_auth_url']:
options['auth'] = options['os_auth_url']
- if not options['user']and options['os_username']:
+ if not options['user'] and options['os_username']:
options['user'] = options['os_username']
if not options['key'] and options['os_password']:
options['key'] = options['os_password']
@@ -1628,6 +1628,7 @@ class SwiftService(object):
res = {
'action': 'upload_segment',
+ 'for_container': container,
'for_object': obj_name,
'segment_index': segment_index,
'segment_size': segment_size,
diff --git a/tests/unit/test_service.py b/tests/unit/test_service.py
index 8eea4c3f..6304b82b 100644
--- a/tests/unit/test_service.py
+++ b/tests/unit/test_service.py
@@ -868,6 +868,7 @@ class TestServiceUpload(_TestServiceBase):
type(mock_conn).attempts = mock.PropertyMock(return_value=2)
expected_r = {
'action': 'upload_segment',
+ 'for_container': 'test_c',
'for_object': 'test_o',
'segment_index': 2,
'segment_size': 10,