openstack
/
python-swiftclient
Code Issues Proposed changes

Add new doc structure and contents for swiftclient 

As a result of the Hackathon we have produced a new
documentation structure for the python-swiftclient.
This patch introduces the new structure and adds the
required content.

The intention is to document the CLI, the SwiftService
and Connection API. Importantly, we also provide
guidance on important considerations when using a swift
object store, such as which aspect of the python-swiftclient
to use for various use cases, common authentication patterns
and some useful examples.

Co-Authored-By: Alexandra Settle <alexandra.settle@rackspace.com>
Co-Authored-By: Mohit Motiani <mohit.motiani@intel.com>
Co-Authored-By: Hisashi Osanai <osanai.hisashi@jp.fujitsu.com>

Change-Id: I9eb41f8e9137efa66cead67dc264a76a3c03fbda
This commit is contained in:
Joel Wright 2016-03-03 17:22:33 +00:00
parent f86b2d8138
commit 3a5a25fe98
14 changed files with 1222 additions and 326 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

353
doc/source/cli.rst
View File

@ -1,29 +1,334 @@
===
====
CLI
===
====
Top-level commands
~~~~~~~~~~~~~~~~~~
The ``swift`` tool is a command line utility for communicating with an OpenStack
Object Storage (swift) environment. It allows one to perform several types of
operations.
.. TODO
delete
download
list
post
stat
upload
info/capabilities
tempurl
auth
Authentication
~~~~~~~~~~~~~~
Prescriptive examples
~~~~~~~~~~~~~~~~~~~~~
This section covers the options for authenticating with a swift
object store. The combinations of options required for each authentication
version are detailed below, but are just a subset of those that can be used
to successfully authenticate. These are the most common and recommended
combinations.
.. TODO
A "Hello World" example
uploading an object
creating a tempurl
listing the contents of a container
downloading an object
You should obtain the details of your authentication version and credentials
from your storage provider. These details should make it clearer which of the
authentication sections below are most likely to allow you to connect to your
storage account.
Keystone v3
-----------
.. code-block:: bash
swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \
--os-project-name project1 --os-project-domain-name domain1 \
--os-username user --os-user-domain-name domain1 \
--os-password password list
swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \
--os-project-id 0123456789abcdef0123456789abcdef \
--os-user-id abcdef0123456789abcdef0123456789 \
--os-password password list
Manually specifying the options above on the command line can be avoided by
setting the following combinations of environment variables:
.. code-block:: bash
ST_AUTH_VERSION=3
OS_USERNAME=user
OS_USER_DOMAIN_NAME=domain1
OS_PASSWORD=password
OS_PROJECT_NAME=project1
OS_PROJECT_DOMAIN_NAME=domain1
OS_AUTH_URL=https://api.example.com:5000/v3
ST_AUTH_VERSION=3
OS_USER_ID=abcdef0123456789abcdef0123456789
OS_PASSWORD=password
OS_PROJECT_ID=0123456789abcdef0123456789abcdef
OS_AUTH_URL=https://api.example.com:5000/v3
Keystone v2
-----------
.. code-block:: bash
swift --os-auth-url https://api.example.com:5000/v2.0 \
--os-tenant-name tenant \
--os-username user --os-password password list
Manually specifying the options above on the command line can be avoided by
setting the following environment variables:
.. code-block:: bash
ST_AUTH_VERSION=2.0
OS_USERNAME=user
OS_PASSWORD=password
OS_TENANT_NAME=tenant
OS_AUTH_URL=https://api.example.com:5000/v2.0
Legacy auth systems
-------------------
You can configure swift to work with any number of other authentication systems
that we will not cover in this document. If your storage provider is not using
Keystone to provide access tokens, please contact them for instructions on the
required options. It is likely that the options will need to be specified as
below:
.. code-block:: bash
swift -A https://auth.api.rackspacecloud.com/v1.0 -U user -K api_key list
Specifying the options above manually on the command line can be avoided by
setting the following environment variables:
.. code-block:: bash
ST_AUTH_VERSION=1.0
ST_AUTH=https://auth.api.rackspacecloud.com/v1.0
ST_USER=user
ST_KEY=key
It is also possible that you need to use a completely separate auth system, in which
case ``swiftclient`` cannot request a token for you. In this case you should make the
authentication request separately and access your storage using the token and
storage URL options shown below:
.. code-block:: bash
swift --os-auth-token 6ee5eb33efad4e45ab46806eac010566 \
--os-storage-url https://10.1.5.2:8080/v1/AUTH_ced809b6a4baea7aeab61a \
list
.. We need the backslash below in order to indent the note
\
.. note::
Leftover environment variables are a common source of confusion when
authorization fails.
CLI commands
~~~~~~~~~~~~
Stat
----
``stat [container [object]]``
Displays information for the account, container, or object depending on
the arguments given (if any). In verbose mode, the storage URL and the
authentication token are displayed as well.
List
----
``list [command-options] [container]``
Lists the containers for the account or the objects for a container.
The ``-p <prefix>`` or ``--prefix <prefix>`` is an option that will only
list items beginning with that prefix. The ``-d <delimiter>`` or
``--delimiter <delimiter>`` is an option (for container listings only)
that will roll up items with the given delimiter (see `OpenStack Swift
general documentation <http://docs.openstack.org/developer/swift/>` for
what this means).
The ``-l`` and ``--lh`` options provide more detail, similar to ``ls -l``
and ``ls -lh``, the latter providing sizes in human readable format
(For example: ``3K``, ``12M``, etc). The latter two switches use more
overhead to retrieve the displayed details, which is directly proportional
to the number of container or objects listed.
Upload
------
``upload [command-options] container file_or_directory [file_or_directory] [...]``
Uploads the files and directories specified by the remaining arguments to the
given container. The ``-c`` or ``--changed`` is an option that will only
upload files that have changed since the last upload. The
``--object-name <object-name>`` is an option that will upload a file and
name object to ``<object-name>`` or upload a directory and use ``<object-name>``
as object prefix. The ``-S <size>`` or ``--segment-size <size>`` and
``--leave-segments`` are options as well (see ``--help`` for more).
Post
----
``post [command-options] [container] [object]``
Updates meta information for the account, container, or object depending
on the arguments given. If the container is not found, the ``swiftclient``
will create it automatically, but this is not true for accounts and
objects. Containers also allow the ``-r <read-acl>`` (or ``--read-acl
<read-acl>``) and ``-w <write-acl>`` (or ``--write-acl <write-acl>``) options.
The ``-m`` or ``--meta`` option is allowed on accounts, containers and objects,
and is used to define the user metadata items to set in the form ``Name:Value``.
You can repeat this option. For example: ``post -m Color:Blue -m Size:Large``
For more information about ACL formats see the documentation:
`ACLs <http://docs.openstack.org/developer/swift/misc.html#acls/>`_.
Download
--------
``download [command-options] [container] [object] [object] [...]``
Downloads everything in the account (with ``--all``), or everything in a
container, or a list of objects depending on the arguments given. For a
single object download, you may use the ``-o <filename>`` or ``--output <filename>``
option to redirect the output to a specific file or ``-`` to
redirect to stdout. You can specify optional headers with the repeatable
cURL-like option ``-H [--header <name:value>]``.
Delete
------
``delete [command-options] [container] [object] [object] [...]``
Deletes everything in the account (with ``--all``), or everything in a
container, or a list of objects depending on the arguments given. Segments
of manifest objects will be deleted as well, unless you specify the
``--leave-segments`` option.
Capabilities
------------
``capabilities [proxy-url]``
Displays cluster capabilities. The output includes the list of the
activated Swift middlewares as well as relevant options for each ones.
Additionally the command displays relevant options for the Swift core. If
the ``proxy-url`` option is not provided, the storage URL retrieved after
authentication is used as ``proxy-url``.
Examples
~~~~~~~~
In this section we present some example usage of the ``swift`` CLI. To keep the
examples as short as possible, these examples assume that the relevant authentication
options have been set using environment variables. You can obtain the full list of
commands and options available in the ``swift`` CLI by executing the following:
.. code-block:: bash
> swift --help
> swift <command> --help
Simple examples
---------------
List the existing swift containers:
.. code-block:: bash
> swift list
container_1
Create a new container:
.. code-block:: bash
> swift post TestContainer
Upload an object into a container:
.. code-block:: bash
> swift upload TestContainer testSwift.txt
testSwift.txt
List the contents of a container:
.. code-block:: bash
> swift list TestContainer
testSwift.txt
Download an object from a container:
.. code-block:: bash
> swift download TestContainer testSwift.txt
testSwift.txt [auth 0.028s, headers 0.045s, total 0.045s, 0.002 MB/s]
.. We need the backslash below in order to indent the note
\
.. note::
To upload an object to a container, your current working directory must be
where the file is located or you must provide the complete path to the file.
In the case that you provide the complete path of the file, that complete
path will be the name of the uploaded object.
For example:
.. code-block:: bash
> swift upload TestContainer /home/swift/testSwift/testSwift.txt
home/swift/testSwift/testSwift.txt
> swift list TestContainer
home/swift/testSwift/testSwift.txt
More complex examples
---------------------
Swift has a single object size limit of 5GiB. In order to upload files larger
than this, we must create a large object that consists of smaller segments.
The example below shows how to upload a large video file as a static large
object in 1GiB segments:
.. code-block:: bash
> swift upload videos --use-slo --segment-size 1G myvideo.mp4
myvideo.mp4 segment 8
myvideo.mp4 segment 4
myvideo.mp4 segment 2
myvideo.mp4 segment 7
myvideo.mp4 segment 0
myvideo.mp4 segment 1
myvideo.mp4 segment 3
myvideo.mp4 segment 6
myvideo.mp4 segment 5
myvideo.mp4
This command will upload segments to a container named ``videos_segments``, and
create a manifest file describing the entire object in the ``videos`` container.
For more information on large objects, see the documentation `here
<http://docs.openstack.org/developer/swift/overview_large_objects.html>`_.
.. code-block:: bash
> swift list videos
myvideo.mp4
> swift list videos_segments
myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000000
myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000001
myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000002
myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000003
myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000004
myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000005
myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000006
myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000007
myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000008

177
doc/source/client-api.rst Normal file
View File

@ -0,0 +1,177 @@
==============================
The swiftclient.Connection API
==============================
A low level API that provides methods for authentication and methods that
correspond to the individual REST API calls described in the swift
documentation.
For usage details see the client docs: :mod:`swiftclient.client`.
Authentication
--------------
This section covers the various combinations of kwargs required when creating
and instance of the ``Connection`` object for communicating with a swift
object store. The combinations of options required for each authentication
version are detailed below, but are
just a subset of those that can be used to successfully authenticate. These
are the most common and recommended combinations.
Keystone v3
~~~~~~~~~~~
.. code-block:: python
_authurl = 'http://127.0.0.1:5000/v3/'
_auth_version = '3'
_user = 'tester'
_key = 'testing'
_os_options = {
'user_domain_name': 'Default',
'project_domain_name': 'Default',
'project_name': 'Default'
}
conn = Connection(
authurl=_authurl,
user=_user,
key=_key,
os_options=_os_options,
auth_version=_auth_version
)
.. code-block:: python
_authurl = 'http://127.0.0.1:5000/v3/'
_auth_version = '3'
_user = 'tester'
_key = 'testing'
_os_options = {
'user_domain_id': 'Default',
'project_domain_id': 'Default',
'project_id': 'Default'
}
conn = Connection(
authurl=_authurl,
user=_user,
key=_key,
os_options=_os_options,
auth_version=_auth_version
)
Keystone v2
~~~~~~~~~~~
.. code-block:: python
_authurl = 'http://127.0.0.1:5000/v2.0/'
_auth_version = '2'
_user = 'tester'
_key = 'testing'
_tenant_name = 'test'
conn = Connection(
authurl=_authurl,
user=_user,
key=_key,
tenant_name=_tenant_name,
auth_version=_auth_version
)
Legacy Auth
~~~~~~~~~~~
.. code-block:: python
_authurl = 'http://127.0.0.1:8080/'
_auth_version = '1'
_user = 'tester'
_key = 'testing'
_tenant_name = 'test'
conn = Connection(
authurl=_authurl,
user=_user,
key=_key,
tenant_name=_tenant_name,
auth_version=_auth_version
)
Examples
--------
In this section we present some simple code examples that demonstrate the usage
of the ``Connection`` API. You can find full details of the options and methods
available to the ``Connection`` API in the docstring generated documentation:
:mod:`swiftclient.client`.
List the available containers:
.. code-block:: python
resp_headers, containers = conn.get_account()
print("Response headers: %s" % resp_headers)
for container in containers:
print(container)
Create a new container:
.. code-block:: python
container = 'new-container'
conn.put_container(container)
resp_headers, containers = conn.get_account()
if container in containers:
print("The container was created")
Create a new object with the contents of a local text file:
.. code-block:: python
container = 'new-container'
with open('local.txt', 'r') as local:
conn.put_object(
container,
'local_object.txt',
contents=local,
content_type='text/plain'
)
Confirm presence of the object:
.. code-block:: python
obj = 'local_object.txt'
container = 'new-container'
try:
resp_headers = conn.head_object(container, obj)
print('The object was successfully created')
except ClientException as e:
if e.http_status = '404':
print('The object was not found')
else:
print('An error occurred checking for the existence of the object')
Download the created object:
.. code-block:: python
obj = 'local_object.txt'
container = 'new-container'
resp_headers, obj_contents = conn.get_object(container, obj)
with open('local_copy.txt', 'w') as local:
local.write(obj_contents)
Delete the created object:
.. code-block:: python
obj = 'local_object.txt'
container = 'new-container'
try:
conn.delete_object(container, obj)
print("Successfully deleted the object")
except ClientException as e:
print("Failed to delete the object with error: %s" % e)

26
doc/source/index.rst
View File

@ -1,18 +1,28 @@
======================================
Welcome to the python-swiftclient Docs
**************************************
======================================
Developer Documentation
=======================
Introduction
~~~~~~~~~~~~
.. toctree::
:maxdepth: 2
introduction
Developer Documentation
~~~~~~~~~~~~~~~~~~~~~~~
.. toctree::
:maxdepth: 2
apis
cli
sdk
service-api
client-api
Code-Generated Documentation
============================
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. toctree::
:maxdepth: 2
@ -20,14 +30,14 @@ Code-Generated Documentation
swiftclient
Indices and tables
==================
~~~~~~~~~~~~~~~~~~
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
License
=======
~~~~~~~
Copyright 2013 OpenStack, LLC.

94
doc/source/introduction.rst Normal file
View File

@ -0,0 +1,94 @@
============
Introduction
============
Where to Start?
~~~~~~~~~~~~~~~
The ``python-swiftclient`` project comprises a command line tool and two
separate APIs for accessing swift programmatically. Choosing the most
appropriate method for a given use case is the first problem a user needs to
solve.
Use Cases
---------
Alongside the command line tool, the ``python-swiftclient`` includes two
levels of API:
* A low level client API that provides simple Python wrappers around the
various authentication mechanisms and the individual HTTP requests.
* A high level service API that provides methods for performing common
operations in parallel on a thread pool.
Example use cases:
* Uploading and retrieving data
Use the command line tool if you are simply uploading and downloading
files and directories to and from your filesystem. The command line tool
can be integrated into a shell script to automate tasks.
* Integrating into an automated Python workflow
Use the ``SwiftService`` API to perform operations offered by the CLI
if your use case requires integration with a Python-based workflow.
This method offers greater control and flexibility over individual object
operations, such as the metadata set on each object. The ``SwiftService``
class provides methods to perform multiple sets of operations against a
swift object store using a configurable shared thread pool. A single
instance of the ``SwiftService`` class can be shared between multiple
threads in your own code.
* Developing an application in Python to access a swift object store
Use the ``SwiftService`` API to develop Python applications that use
swift to store and retrieve objects. A ``SwiftService`` instance provides
a configurable thread pool for performing all operations supported by the
CLI.
* Fine-grained control over threading or the requests being performed
Use the ``Connection`` API if your use case requires fine grained control
over advanced features or you wish to use your own existing threading
model. Examples of advanced features requiring the use of the
``Connection`` API include creating an SLO manifest that references
already existing objects, or fine grained control over the query strings
supplied with each HTTP request.
Important considerations
~~~~~~~~~~~~~~~~~~~~~~~~
This section covers some important considerations, helpful hints, and things to
avoid when integrating an object store into your workflow.
An object store is not a filesystem
-----------------------------------
It cannot be stressed enough that your usage of the object store should reflect
the proper use case, and not treat the storage like a traditional filesystem.
There are two main restrictions to bear in mind when designing an application
that uses an object store:
* You cannot rename objects. Due to fact that the name of an object is one
of the factors that determines where the object and its replicas are stored,
renaming would require multiple copies of the data to be moved between
physical storage devices. If you want to rename an object you must upload
to the new location, or make a server side copy request to the new location,
and then delete the original.
* You cannot modify objects. Objects are stored in multiple locations and
are checked for integrity based on the MD5 sum calculated during
upload. In order to modify the contents of an object, the entire desired
contents must be re-uploaded. In certain special cases it is possible to
work around this restriction using large objects, but no general
file-like access is available to modify a stored object.
Objects cannot be locked
------------------------
There is no mechanism to perform a combination of reading the
data/metadata from an object and writing an update to that data/metadata in an
atomic way. Any user with access to a container could update the contents or
metadata associated with an object at any time.
Workflows that assume that no updates have been made since the last read of an
object should be discouraged. Enabling a workflow of this type requires an
external object locking mechanism and/or cooperation between all clients
accessing the data.

48
doc/source/sdk.rst
View File

@ -1,48 +0,0 @@
===
SDK
===
Where to start?
~~~~~~~~~~~~~~~
.. TODO
when to use SwiftService
when to use client.py
SwiftService classes and methods
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. TODO
docs for each method (autogen from docstrings?)
Client classes and methods
~~~~~~~~~~~~~~~~~~~~~~~~~~
.. TODO
docs for each method (autogen from docstrings?)
Guidelines for writing an app
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. TODO
auth
how to use various features
when to use various features
pooling connections
concurrency
retries
Prescriptive examples
~~~~~~~~~~~~~~~~~~~~~
.. TODO
A "Hello World" example
connecting
uploading an object
uploading a directory

599
doc/source/apis.rst → doc/source/service-api.rst
View File

@ -1,63 +1,93 @@
======================
python-swiftclient API
======================
The python-swiftclient includes two levels of API. A low level client API that
provides simple python wrappers around the various authentication mechanisms,
the individual HTTP requests, and a high level service API that provides
methods for performing common operations in parallel on a thread pool.
This document aims to provide guidance for choosing between these APIs and
examples of usage for the service API.
Important Considerations
~~~~~~~~~~~~~~~~~~~~~~~~
This section covers some important considerations, helpful hints, and things
to avoid when integrating an object store into your workflow.
An Object Store is not a filesystem
-----------------------------------
.. important::
It cannot be stressed enough that your usage of the object store should reflect
the use case, and not treat the storage like a filesystem.
There are 2 main restrictions to bear in mind here when designing your use of the object
store:
#. Objects cannot be renamed due to the way in which objects are stored and
references by the object store. This usually requires multiple copies of
the data to be moved between physical storage devices.
As a result, a move operation is not provided. If the user wants to move an
object they must re-upload to the new location and delete the
original.
#. Objects cannot be modified. Objects are stored in multiple locations and are
checked for integrity based on the ``MD5 sum`` calculated during upload.
Object creation is a 1-shot event, and in order to modify the contents of an
object the entire new contents must be re-uploaded. In certain special cases
it is possible to work around this restriction using large objects, but no
general file-like access is available to modify a stored object.
The swiftclient.Connection API
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A low level API that provides methods for authentication and methods that
correspond to the individual REST API calls described in the swift
documentation.
For usage details see the client docs: :mod:`swiftclient.client`.
================================
The swiftclient.SwiftService API
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
================================
A higher level API aimed at allowing developers an easy way to perform multiple
operations asynchronously using a configurable thread pool. Documentation for each
service method call can be found here: :mod:`swiftclient.service`.
A higher-level API aimed at allowing developers an easy way to perform multiple
operations asynchronously using a configurable thread pool. Documentation for
each service method call can be found here: :mod:`swiftclient.service`.
Authentication
--------------
This section covers the various options for authenticating with a swift
object store. The combinations of options required for each authentication
version are detailed below. Once again, these are just a subset of those that
can be used to successfully authenticate, but they are the most common and
recommended.
The relevant authentication options are presented as python dictionaries that
should be added to any other options you are supplying to your ``SwiftService``
instance. As indicated in the python code, you can also set these options as
environment variables that will be loaded automatically if the relevant option
is not specified.
The ``SwiftService`` authentication attempts to automatically select
the auth version based on the combination of options specified, but
supplying options from multiple different auth versions can cause unexpected
behaviour.
.. note::
Leftover environment variables are a common source of confusion when
authorization fails.
Keystone V3
~~~~~~~~~~~
.. code-block:: python
{
...
"auth_version": environ.get('ST_AUTH_VERSION'), # Should be '3'
"os_username": environ.get('OS_USERNAME'),
"os_password": environ.get('OS_PASSWORD'),
"os_project_name": environ.get('OS_PROJECT_NAME'),
"os_project_domain_name": environ.get('OS_PROJECT_DOMAIN_NAME'),
"os_auth_url": environ.get('OS_AUTH_URL'),
...
}
.. code-block:: python
{
...
"auth_version": environ.get('ST_AUTH_VERSION'), # Should be '3'
"os_username": environ.get('OS_USERNAME'),
"os_password": environ.get('OS_PASSWORD'),
"os_project_id": environ.get('OS_PROJECT_ID'),
"os_project_domain_id": environ.get('OS_PROJECT_DOMAIN_ID'),
"os_auth_url": environ.get('OS_AUTH_URL'),
...
}
Keystone V2
~~~~~~~~~~~
.. code-block:: python
{
...
"auth_version": environ.get('ST_AUTH_VERSION'), # Should be '2.0'
"os_username": environ.get('OS_USERNAME'),
"os_password": environ.get('OS_PASSWORD'),
"os_tenant_name": environ.get('OS_TENANT_NAME'),
"os_auth_url": environ.get('OS_AUTH_URL'),
...
}
Legacy Auth
~~~~~~~~~~~
.. code-block:: python
{
...
"auth_version": environ.get('ST_AUTH_VERSION'), # Should be '1.0'
"auth": environ.get('ST_AUTH'),
"user": environ.get('ST_USER'),
"key": environ.get('ST_KEY'),
...
}
Configuration
-------------
@ -77,7 +107,7 @@ passed to the ``SwiftService`` during initialisation. The options available
in this dictionary are described below, along with their defaults:
Options
^^^^^^^
~~~~~~~
``retries``: ``5``
The number of times that the library should attempt to retry HTTP
@ -192,51 +222,8 @@ source code for ``python-swiftclient``. Each ``SwiftService`` method also allows
for an optional dictionary to override those specified at init time, and the
appropriate docstrings show which options modify each method's behaviour.
Authentication
~~~~~~~~~~~~~~
This section covers the various options for authenticating with a swift
object store. The combinations of options required for each authentication
version are detailed below.
Version 1.0 Auth
----------------
``auth_version``: ``environ.get('ST_AUTH_VERSION')``
``auth``: ``environ.get('ST_AUTH')``
``user``: ``environ.get('ST_USER')``
``key``: ``environ.get('ST_KEY')``
Version 2.0 and 3.0 Auth
------------------------
``auth_version``: ``environ.get('ST_AUTH_VERSION')``
``os_username``: ``environ.get('OS_USERNAME')``
``os_password``: ``environ.get('OS_PASSWORD')``
``os_tenant_name``: ``environ.get('OS_TENANT_NAME')``
``os_auth_url``: ``environ.get('OS_AUTH_URL')``
As is evident from the default values, if these options are not set explicitly
in the options dictionary, then they will default to the values of the given
environment variables. The ``SwiftService`` authentication automatically selects
the auth version based on the combination of options specified, but
having options from different auth versions can cause unexpected behaviour.
.. note::
Leftover environment variables are a common source of confusion when
authorization fails.
Operation Return Values
~~~~~~~~~~~~~~~~~~~~~~~
Available Operations
--------------------
Each operation provided by the service API may raise a ``SwiftError`` or
``ClientException`` for any call that fails completely (or a call which
@ -371,32 +358,14 @@ operation was not successful, and will include the keys below:
}
Example
-------
^^^^^^^
The code below demonstrates the use of ``stat`` to retrieve the headers for a
given list of objects in a container using 20 threads. The code creates a
mapping from object name to headers.
mapping from object name to headers which is then pretty printed to the log.
.. code-block:: python
import logging
from swiftclient.service import SwiftService
logger = logging.getLogger()
_opts = {'object_dd_threads': 20}
with SwiftService(options=_opts) as swift:
container = 'container1'
objects = [ 'object_%s' % n for n in range(0,100) ]
header_data = {}
stats_it = swift.stat(container=container, objects=objects)
for stat_res in stats_it:
if stat_res['success']:
header_data[stat_res['object']] = stat_res['headers']
else:
logger.error(
'Failed to retrieve stats for %s' % stat_res['object']
)
.. literalinclude:: ../../examples/stat.py
:language: python
List
~~~~
@ -456,55 +425,38 @@ dictionary as described below:
}
Example
-------
^^^^^^^
The code below demonstrates the use of ``list`` to list all items in a
container that are over 10MiB in size:
.. code-block:: python
container = 'example_container'
minimum_size = 10*1024**2
with SwiftService() as swift:
try:
stats_parts_gen = swift.list(container=container)
for stats in stats_parts_gen:
if stats["success"]:
for item in stats["listing"]:
i_size = int(item["bytes"])
if i_size > minimum_size:
i_name = item["name"]
i_etag = item["hash"]
print(
"%s [size: %s] [etag: %s]" %
(i_name, i_size, i_etag)
)
else:
raise stats["error"]
except SwiftError as e:
output_manager.error(e.value)
.. literalinclude:: ../../examples/list.py
:language: python
Post
~~~~
Post can be called against an account, container or list of objects in order to
update the metadata attached to the given items. Each element of the object list
may be a plain string of the object name, or a ``SwiftPostObject`` that
allows finer control over the options applied to each of the individual post
operations. In the first two cases a single dictionary is returned containing the
results of the operation, and in the case of a list of objects being supplied,
an iterator over the results generated for each object post is returned. If the
given container or account does not exist, the ``post`` method will raise a
``SwiftError``.
update the metadata attached to the given items. In the first two cases a single
dictionary is returned containing the results of the operation, and in the case
of a list of objects being supplied, an iterator over the results generated for
each object post is returned.
.. When a string is given for the object name, the options
Each element of the object list may be a plain string of the object name, or a
``SwiftPostObject`` that allows finer control over the options and metadata
applied to each of the individual post operations. When a string is given for
the object name, the options and metadata applied are a combination of those
supplied to the call to ``post()`` and the defaults of the ``SwiftService``
object.
Successful metadata update results are dictionaries as described below:
If the given container or account does not exist, the ``post`` method will
raise a ``SwiftError``. Successful metadata update results are dictionaries as
described below:
.. code-block:: python
{
'action': <'post_account'|<'post_container'>|'post_object'>,
'action': <'post_account'|'post_container'|'post_object'>,
'success': True,
'container': <container>,
'object': <object>,
@ -513,28 +465,87 @@ Successful metadata update results are dictionaries as described below:
}
.. note::
Updating user metadata keys will not only add any specified keys, but
will also remove user metadata that has previously been set. This means
that each time user metadata is updated, the complete set of desired
key-value pairs must be specified.
Example
^^^^^^^
The code below demonstrates the use of ``post`` to set an archive folder in a
given container to expire after a 24 hour delay:
.. Example
.. -------
.. literalinclude:: ../../examples/post.py
:language: python
.. TBD
Download
~~~~~~~~
.. Download
.. ~~~~~~~~
Download can be called against an entire account, a single container, or a list
of objects in a given container. Each element of the object list is a string
detailing the full name of an object to download.
.. TBD
In order to download the full contents of an entire account, you must set the
value of ``yes_all`` to ``True`` in the ``options`` dictionary supplied to
either the ``SwiftService`` instance or the call to ``download``.
.. Example
.. -------
If the given container or account does not exist, the ``download`` method will
raise a ``SwiftError``, otherwise an iterator over the results generated for
each object download is returned.
.. TBD
See :mod:`swiftclient.service.SwiftService.download` for docs generated from the
method docstring.
For each successfully downloaded object, the results returned by the iterator
will be a dictionary as described below (results are not returned for completed
container or object segment downloads):
.. code-block:: python
{
'action': 'download_object',
'container': <container>,
'object': <object name>,
'success': True,
'path': <local path to downloaded object>,
'pseudodir': <if true, the download created an empty directory>,
'start_time': <time download started>,
'end_time': <time download completed>,
'headers_receipt': <time the headers from the object were retrieved>,
'auth_end_time': <time authentication completed>,
'read_length': <bytes_read>,
'attempts': <attempt count>,
'response_dict': <HTTP response details>
}
Any failure uploading an object will return a failure dictionary as described
below:
.. code-block:: python
{
'action': 'download_object',
'container': <container>,
'object': <object name>,
'success': False,
'path': <local path of the failed download>,
'pseudodir': <if true, the failed download was an empty directory>,
'attempts': <attempt count>,
'error': <error>,
'traceback': <trace>,
'error_timestamp': <timestamp>,
'response_dict': <HTTP response details>
}
Example
^^^^^^^
The code below demonstrates the use of ``download`` to download all PNG images
from a dated archive folder in a given container:
.. literalinclude:: ../../examples/download.py
:language: python
Upload
~~~~~~
@ -550,7 +561,7 @@ the upload are those supplied to the call to ``upload``.
Constructing a ``SwiftUploadObject`` allows the user to supply an object name
for the uploaded file, and modify the options used by ``upload`` at the
granularity of invidivual files.
granularity of individual files.
If the given container or account does not exist, the ``upload`` method will
raise a ``SwiftError``, otherwise an iterator over the results generated for
@ -622,97 +633,195 @@ below:
}
Example
-------
^^^^^^^
The code below demonstrates the use of ``upload`` to upload all files and
folders in ``/tmp``, and renaming each object by replacing ``/tmp`` in the
object or directory marker names with ``temporary-objects``:
folders in a given directory, and rename each object by replacing the root
directory name with 'my-<d>-objects', where <d> is the name of the uploaded
directory:
.. literalinclude:: ../../examples/upload.py
:language: python
Delete
~~~~~~
Delete can be called against an account or a container to remove the containers
or objects contained within them. Each call to ``delete`` returns an iterator
over results of each resulting sub-request.
If the number of requested delete operations is large and the target swift
cluster is running the bulk middleware, the call to ``SwiftService.delete`` will
make use of bulk operations and the returned result iterator will return
``bulk_delete`` results rather than individual ``delete_object``,
``delete_container`` or ``delete_segment`` results.
See :mod:`swiftclient.service.SwiftService.delete` for docs generated from the
method docstring.
For each successfully deleted container, object or segment, the results returned
by the iterator will be a dictionary as described below:
.. code-block:: python
_opts['object_uu_threads'] = 20
with SwiftService(options=_opts) as swift, OutputManager() as out_manager:
try:
# Collect all the files and folders in '/tmp'
objs = []
dir_markers = []
dir = '/tmp':
for (_dir, _ds, _fs) in walk(f):
if not (_ds + _fs):
dir_markers.append(_dir)
else:
objs.extend([join(_dir, _f) for _f in _fs])
{
'action': <'delete_object'|'delete_segment'>,
'container': <container>,
'object': <object name>,
'success': True,
'attempts': <attempt count>,
'response_dict': <HTTP response details>
}
# Now that we've collected all the required files and dir markers
# build the ``SwiftUploadObject``s for the call to upload
objs = [
SwiftUploadObject(
o, object_name=o.replace(
'/tmp', 'temporary-objects', 1
)
) for o in objs
]
dir_markers = [
SwiftUploadObject(
None, object_name=d.replace(
'/tmp', 'temporary-objects', 1
), options={'dir_marker': True}
) for d in dir_markers
]
{
'action': 'delete_container',
'container': <container>,
'success': True,
'response_dict': <HTTP response details>,
'attempts': <attempt count>
}
# Schedule uploads on the SwiftService thread pool and iterate
# over the results
for r in swift.upload(container, objs + dir_markers):
if r['success']:
if 'object' in r:
out_manager.print_msg(r['object'])
elif 'for_object' in r:
out_manager.print_msg(
'%s segment %s' % (r['for_object'],
r['segment_index'])
)
else:
error = r['error']
if r['action'] == "create_container":
out_manager.warning(
'Warning: failed to create container '
"'%s'%s", container, msg
)
elif r['action'] == "upload_object":
out_manager.error(
"Failed to upload object %s to container %s: %s" %
(container, r['object'], error)
)
else:
out_manager.error("%s" % error)
{
'action': 'bulk_delete',
'container': <container>,
'objects': <[objects]>,
'success': True,
'attempts': <attempt count>,
'response_dict': <HTTP response details>
}
except SwiftError as e:
out_manager.error(e.value)
Any failure in a delete operation will return a failure dictionary as described
below:
.. Delete
.. ~~~~~~
.. Do we want to hide this section until it is complete?
.. code-block:: python
.. TBD
{
'action': ('delete_object'|'delete_segment'),
'container': <container>,
'object': <object name>,
'success': False,
'attempts': <attempt count>,
'error': <error>,
'traceback': <trace>,
'error_timestamp': <timestamp>,
'response_dict': <HTTP response details>
}
.. Example
.. -------
{
'action': 'delete_container',
'container': <container>,
'success': False,
'error': <error>,
'traceback': <trace>,
'error_timestamp': <timestamp>,
'response_dict': <HTTP response details>,
'attempts': <attempt count>
}
.. Do we want to hide this section until it is complete?
{
'action': 'bulk_delete',
'container': <container>,
'objects': <[objects]>,
'success': False,
'attempts': <attempt count>,
'error': <error>,
'traceback': <trace>,
'error_timestamp': <timestamp>,
'response_dict': <HTTP response details>
}
.. TBD
Example
^^^^^^^
.. Capabilities
.. ~~~~~~~~~~~~
The code below demonstrates the use of ``delete`` to remove a given list of
objects from a specified container. As the objects are deleted the transaction
id of the relevant request is printed along with the object name and number
of attempts required. By printing the transaction id, the printed operations
can be easily linked to events in the swift server logs:
.. Do we want to hide this section until it is complete?
.. literalinclude:: ../../examples/delete.py
:language: python
.. TBD
Capabilities
~~~~~~~~~~~~
.. Example
.. -------
Capabilities can be called against an account or a particular proxy URL in
order to determine the capabilities of the swift cluster. These capabilities
include details about configuration options and the middlewares that are
installed in the proxy pipeline.
.. Do we want to hide this section until it is complete?
See :mod:`swiftclient.service.SwiftService.capabilities` for docs generated from
the method docstring.
.. TBD
For each successful call to list capabilities, a result dictionary will be
returned with the contents described below:
{
'action': 'capabilities',
'timestamp': <time of the call>,
'success': True,
'capabilities': <dictionary containing capability details>
}
The contents of the capabilities dictionary contain the core swift capabilities
under the key ``swift``, all other keys show the configuration options for
additional middlewares deployed in the proxy pipeline. An example capabilities
dictionary is given below:
.. code-block:: python
{
'account_quotas': {},
'bulk_delete': {
'max_deletes_per_request': 10000,
'max_failed_deletes': 1000
},
'bulk_upload': {
'max_containers_per_extraction': 10000,
'max_failed_extractions': 1000
},
'container_quotas': {},
'container_sync': {'realms': {}},
'formpost': {},
'keystoneauth': {},
'slo': {
'max_manifest_segments': 1000,
'max_manifest_size': 2097152,
'min_segment_size': 1048576
},
'swift': {
'account_autocreate': True,
'account_listing_limit': 10000,
'allow_account_management': True,
'container_listing_limit': 10000,
'extra_header_count': 0,
'max_account_name_length': 256,
'max_container_name_length': 256,
'max_file_size': 5368709122,
'max_header_size': 8192,
'max_meta_count': 90,
'max_meta_name_length': 128,
'max_meta_overall_size': 4096,
'max_meta_value_length': 256,
'max_object_name_length': 1024,
'policies': [
{'default': True, 'name': 'Policy-0'}
],
'strict_cors_mode': False,
'version': '2.2.2'
},
'tempurl': {
'methods': ['GET', 'HEAD', 'PUT']
}
}
Example
^^^^^^^
The code below demonstrates the us of ``capabilities`` to determine if the
Swift cluster supports static large objects, and if so, the maximum number of
segments that can be described in a single manifest file, along with the
size restrictions on those objects:
.. literalinclude:: ../../examples/capabilities.py
:language: python

20
examples/capabilities.py Normal file
View File

@ -0,0 +1,20 @@
import logging
from swiftclient.exceptions import ClientException
from swiftclient.service import SwiftService
logging.basicConfig(level=logging.ERROR)
logging.getLogger("requests").setLevel(logging.CRITICAL)
logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)
with SwiftService() as swift:
try:
capabilities_result = swift.capabilities()
capabilities = capabilities_result['capabilities']
if 'slo' in capabilities:
print('SLO is supported')
else:
print('SLO is not supported')
except ClientException as e:
logger.error(e.value)

34
examples/delete.py Normal file
View File

@ -0,0 +1,34 @@
import logging
from swiftclient.service import SwiftService
from sys import argv
logging.basicConfig(level=logging.ERROR)
logging.getLogger("requests").setLevel(logging.CRITICAL)
logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)
_opts = {'object_dd_threads': 20}
container = argv[1]
objects = argv[2:]
with SwiftService(options=_opts) as swift:
del_iter = swift.delete(container=container, objects=objects)
for del_res in del_iter:
c = del_res.get('container', '')
o = del_res.get('object', '')
a = del_res.get('attempts')
if del_res['success'] and not del_res['action'] == 'bulk_delete':
rd = del_res.get('response_dict')
if rd is not None:
t = dict(rd.get('headers', {}))
if t:
print(
'Successfully deleted {0}/{1} in {2} attempts '
'(transaction id: {3})'.format(c, o, a, t)
)
else:
print(
'Successfully deleted {0}/{1} in {2} '
'attempts'.format(c, o, a)
)

37
examples/download.py Normal file
View File

@ -0,0 +1,37 @@
import logging
from swiftclient.service import SwiftService, SwiftError
from sys import argv
logging.basicConfig(level=logging.ERROR)
logging.getLogger("requests").setLevel(logging.CRITICAL)
logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)
def is_png(obj):
return (
obj["name"].lower().endswith('.png') or
obj["content_type"] == 'image/png'
)
container = argv[1]
with SwiftService() as swift:
try:
list_options = {"prefix": "archive_2016-01-01/"}
list_parts_gen = swift.list(container=container)
for page in list_parts_gen:
if page["success"]:
objects = [
obj["name"] for obj in page["listing"] if is_png(obj)
]
for down_res in swift.download(
container=container,
objects=objects):
if down_res['success']:
print("'%s' downloaded" % down_res['object'])
else:
print("'%s' download failed" % down_res['object'])
else:
raise page["error"]
except SwiftError as e:
logger.error(e.value)

32
examples/list.py Normal file
View File

@ -0,0 +1,32 @@
import logging
from swiftclient.service import SwiftService, SwiftError
from sys import argv
logging.basicConfig(level=logging.ERROR)
logging.getLogger("requests").setLevel(logging.CRITICAL)
logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)
container = argv[1]
minimum_size = 10*1024**2
with SwiftService() as swift:
try:
list_parts_gen = swift.list(container=container)
for page in list_parts_gen:
if page["success"]:
for item in page["listing"]:
i_size = int(item["bytes"])
if i_size > minimum_size:
i_name = item["name"]
i_etag = item["hash"]
print(
"%s [size: %s] [etag: %s]" %
(i_name, i_size, i_etag)
)
else:
raise page["error"]
except SwiftError as e:
logger.error(e.value)

31
examples/post.py Normal file
View File

@ -0,0 +1,31 @@
import logging
from swiftclient.service import SwiftService, SwiftError
from sys import argv
logging.basicConfig(level=logging.ERROR)
logging.getLogger("requests").setLevel(logging.CRITICAL)
logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)
container = argv[1]
with SwiftService() as swift:
try:
list_options = {"prefix": "archive_2016-01-01/"}
list_parts_gen = swift.list(container=container)
for page in list_parts_gen:
if page["success"]:
objects = [obj["name"] for obj in page["listing"]]
post_options = {"header": "X-Delete-After:86400"}
for post_res in swift.post(
container=container,
objects=objects,
options=post_options):
if post_res['success']:
print("Object '%s' POST success" % post_res['object'])
else:
print("Object '%s' POST failed" % post_res['object'])
else:
raise page["error"]
except SwiftError as e:
logger.error(e.value)

25
examples/stat.py Normal file
View File

@ -0,0 +1,25 @@
import logging
import pprint
from swiftclient.service import SwiftService
from sys import argv
logging.basicConfig(level=logging.ERROR)
logging.getLogger("requests").setLevel(logging.CRITICAL)
logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)
_opts = {'object_dd_threads': 20}
with SwiftService(options=_opts) as swift:
container = argv[1]
objects = argv[2:]
header_data = {}
stats_it = swift.stat(container=container, objects=objects)
for stat_res in stats_it:
if stat_res['success']:
header_data[stat_res['object']] = stat_res['headers']
else:
logger.error(
'Failed to retrieve stats for %s' % stat_res['object']
)
pprint.pprint(header_data)

71
examples/upload.py Normal file
View File

@ -0,0 +1,71 @@
import logging
from os.path import join, walk
from swiftclient.multithreading import OutputManager
from swiftclient.service import SwiftError, SwiftService, SwiftUploadObject
from sys import argv
logging.basicConfig(level=logging.ERROR)
logging.getLogger("requests").setLevel(logging.CRITICAL)
logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)
_opts = {'object_uu_threads': 20}
dir = argv[1]
container = argv[2]
with SwiftService(options=_opts) as swift, OutputManager() as out_manager:
try:
# Collect all the files and folders in the given directory
objs = []
dir_markers = []
for (_dir, _ds, _fs) in walk(dir):
if not (_ds + _fs):
dir_markers.append(_dir)
else:
objs.extend([join(_dir, _f) for _f in _fs])
# Now that we've collected all the required files and dir markers
# build the ``SwiftUploadObject``s for the call to upload
objs = [
SwiftUploadObject(
o, object_name=o.replace(
dir, 'my-%s-objects' % dir, 1
)
) for o in objs
]
dir_markers = [
SwiftUploadObject(
None, object_name=d.replace(
dir, 'my-%s-objects' % dir, 1
), options={'dir_marker': True}
) for d in dir_markers
]
# Schedule uploads on the SwiftService thread pool and iterate
# over the results
for r in swift.upload(container, objs + dir_markers):
if r['success']:
if 'object' in r:
print(r['object'])
elif 'for_object' in r:
print(
'%s segment %s' % (r['for_object'],
r['segment_index'])
)
else:
error = r['error']
if r['action'] == "create_container":
logger.warning(
'Warning: failed to create container '
"'%s'%s", container, error
)
elif r['action'] == "upload_object":
logger.error(
"Failed to upload object %s to container %s: %s" %
(container, r['object'], error)
)
else:
logger.error("%s" % error)
except SwiftError as e:
logger.error(e.value)

1
swiftclient/multithreading.py
View File

@ -126,7 +126,6 @@ class MultiThreadingManager(object):
def __init__(self, create_connection, segment_threads=10,
object_dd_threads=10, object_uu_threads=10,
container_threads=10):
"""
:param segment_threads: The number of threads allocated to segment
uploads