object-api/openstack-object-storage-dev/ch_object-api-troubleshooti...

994 lines
46 KiB
XML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter [
<!-- Some useful entities borrowed from HTML -->
<!ENTITY ndash "&#x2013;">
<!ENTITY mdash "&#x2014;">
<!ENTITY hellip "&#x2026;">
<!-- Useful for describing APIs -->
<!ENTITY GET '<command xmlns="http://docbook.org/ns/docbook">GET</command>'>
<!ENTITY HEAD '<command xmlns="http://docbook.org/ns/docbook">HEAD</command>'>
<!ENTITY PUT '<command xmlns="http://docbook.org/ns/docbook">PUT</command>'>
<!ENTITY POST '<command xmlns="http://docbook.org/ns/docbook">POST</command>'>
<!ENTITY DELETE '<command xmlns="http://docbook.org/ns/docbook">DELETE</command>'>
<!ENTITY CHECK '<inlinemediaobject xmlns="http://docbook.org/ns/docbook">
<imageobject>
<imagedata fileref="img/Check_mark_23x20_02.svg"
format="SVG" scale="60"/>
</imageobject>
</inlinemediaobject>'>
<!ENTITY ARROW '<inlinemediaobject xmlns="http://docbook.org/ns/docbook">
<imageobject>
<imagedata fileref="img/Arrow_east.svg"
format="SVG" scale="60"/>
</imageobject>
</inlinemediaobject>'>
]>
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xi="http://www.w3.org/2001/XInclude"
xmlns:xlink="http://www.w3.org/1999/xlink" version="5.0"
xml:id="ch_object-storage-dev-troubleshooting">
<title>Troubleshooting and Examples</title>
<para>This section introduces a command-line utility, cURL, and
demonstrates interacting with the ReST interfaces through that
utility.</para>
<section xml:id="using-curl-cli">
<title>Using cURL</title>
<para>cURL is a command-line tool which is available on most
UNIX®-like environments and Mac OS X® and can be
downloaded for Windows®. For more information on cURL,
visit <link xlink:href="http://curl.haxx.se/"
>http://curl.haxx.se/</link>.</para>
<para>cURL allows you to transmit and receive HTTP requests
and responses from the command-line or from within a shell
script. This makes it possible to work with the ReST API
directly without using one of the client APIs.</para>
<para>The following cURL command-line options will be
used</para>
<variablelist>
<title>cURL Command-Line Options</title>
<varlistentry>
<term><option>-X METHOD</option></term>
<listitem>
<para>Specify the HTTP method to request (&HEAD;,
&GET;, etc.)</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-i</option></term>
<listitem>
<para>Dump HTTP response headers to stdout.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-H HEADER</option></term>
<listitem>
<para>Specify an HTTP header in the
request.</para>
</listitem>
</varlistentry>
</variablelist>
</section>
<section xml:id="authentication-examples-curl">
<title>Authentication</title>
<para>To use the ReST API, you must obtain a authorization
token, which you pass to each request in the
<code>X-Auth-Token</code> header. The following
example demonstrates how to use cURL to obtain the
authorization token and the URL of the storage
system.</para>
<example>
<title>cURL Authenticate</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>-i</option> \
<option>-H "X-Auth-Key: jdoesecretpassword"</option> \
<option>-H "X-Auth-User: jdoe"</option> \
<uri>https://auth.api.yourcloud.com/v1.0</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 204 No Content
Date: Thu, 09 Jul 2009 15:31:39 GMT
Server: Apache/2.2.3
X-Storage-Url: https://storage.swiftdrive.com/v1/CF_xer7_343
X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae
Content-Length: 0
Connection: close
Content-Type: application/octet-stream</computeroutput></screen>
</example>
<para>The storage URL and authentication token are returned in
the headers of the response. After authentication, you can
use cURL to perform &HEAD;, &GET;, &DELETE;, &POST; and
&PUT; requests on the storage service.</para>
</section>
<section xml:id="determining-storage-useage">
<title>Determining Storage Usage</title>
<para>A &HEAD; request can be sent to the storage service to
determine how much data you have stored in the system and
the number of containers you are using. Use the
<code>-X</code> switch to specify the correct HTTP
method and the <code>-i</code> to dump the HTTP response
headers to terminal output (stdout).</para>
<example>
<title>cURL Get Storage Space</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X HEAD</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 204 No Content
Date: Thu, 09 Jul 2009 15:38:14 GMT
Server: Apache
X-Account-Container-Count: 22
X-Account-Bytes-Used: 9891628380
Content-Type: text/plain</computeroutput></screen>
</example>
<para>The HTTP request must include a header to specify the
authentication token. The HTTP headers in the response
indicate the number of containers in this storage account
and the total bytes stored for the entire account.</para>
</section>
<section xml:id="listing-and-creating-storage-containers">
<title>Listing and Creating Containers</title>
<para>The simplest operation for Object Storage is to simply
list the containers you have, which when you don't have
any containers yet isn't terribly exciting:</para>
<example>
<title>cURL List Storage Container</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 204 No Content
X-Account-Object-Count: 0
X-Account-Bytes-Used: 0
X-Account-Container-Count: 0
Accept-Ranges: bytes
X-Trans-Id: txe8ca5138ac8643ec84070543a0c9c91e
Content-Length: 0
Date: Mon, 07 Nov 2011 17:07:01 GMT</computeroutput></screen>
</example>
<para>So, you take the X-Auth-Token obtained from the
authentication operation, pass it as a header value,
execute the operation against the URL obtained from the
authentication operation, and force the GET verb with the
-X switch. What you get back tells you there aren't any
containers.</para>
<para>Next, let's create a container and then do the listing
again:</para>
<example>
<title>cURL Create Storage Container</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X PUT</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/george</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 201 Created
Content-Length: 18
Content-Type: text/html; charset=UTF-8
X-Trans-Id: txb25576385284476d9fa6c73835f21650
Date: Mon, 07 Nov 2011 17:44:20 GMT
201 Created</computeroutput></screen>
</example>
<para>Append the container name to the URL and force the PUT
verb. That creates a container, which we can now see when
we do a listing:</para>
<example>
<title>cURL List Storage Container After a
Creation</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 200 OK
X-Account-Object-Count: 0
X-Account-Bytes-Used: 0
X-Account-Container-Count: 1
Accept-Ranges: bytes
Content-Length: 7
Content-Type: text/plain; charset=utf-8
X-Trans-Id: txaedd6b080626453399c9f5febbddb73b
Date: Mon, 07 Nov 2011 17:44:23 GMT
george</computeroutput></screen>
</example>
<para>You may have noticed the account metadata that comes
back from the listing call. As you'd guess, it'll tell you
how many objects you have, how much space you are using,
and how many containers you are using.</para>
</section>
<section xml:id="paging-containers">
<title>Paging Lists of Containers</title>
<para>If you have a large number of containers, it is
sometimes more convenient to page through them than
getting some big long list of them. If I create more
containers and then do a regular listing, here's what it
looks like with five containers:</para>
<example>
<title>cURL List Storage Container (long list)</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 200 OK
X-Account-Object-Count: 0
X-Account-Bytes-Used: 0
X-Account-Container-Count: 5
Accept-Ranges: bytes
Content-Length: 31
Content-Type: text/plain; charset=utf-8
X-Trans-Id: txb28795cc25b04f0dbce408dfa5a3cfc9
Date: Mon, 07 Nov 2011 19:03:06 GMT
cosmo
dogs
elaine
george
jerry</computeroutput></screen>
</example>
<para>Suppose I want a page size of 2, all I do is append a
""?limit=2"" to my URL:</para>
<example>
<title>cURL List Storage Container with Paging (first
page)</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343?limit=2</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 200 OK
X-Account-Object-Count: 0
X-Account-Bytes-Used: 0
X-Account-Container-Count: 5
Accept-Ranges: bytes
Content-Length: 11
Content-Type: text/plain; charset=utf-8
X-Trans-Id: tx940ee02c1a65451e96a2a2532e3a7ce7
Date: Mon, 07 Nov 2011 19:05:30 GMT
cosmo
dogs</computeroutput></screen>
</example>
<para>Not surprisingly, I only get two containers. To get the
next page, you tell the system which item you last saw
with the "marker=" specifier:</para>
<example>
<title>cURL List Storage Container with Paging (later
pages)</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343?marker=dogs\&amp;limit=2</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 200 OK
X-Account-Object-Count: 0
X-Account-Bytes-Used: 0
X-Account-Container-Count: 5
Accept-Ranges: bytes
Content-Length: 14
Content-Type: text/plain; charset=utf-8
X-Trans-Id: tx2a69f7ec38c34078a185c5875a4c0e34
Date: Mon, 07 Nov 2011 19:15:00 GMT
elaine
george</computeroutput></screen>
</example>
<para>Notice that I had to use \&amp; so that my bash shell
didn't try to interpret the &amp; as wanting to run
something in its own thread. With that in place, you get
the next page of items that appear after the
marker.</para>
</section>
<section xml:id="serialized-output">
<title>Serialized Output</title>
<para>In other situations, like if you are working on a
language binding on top of the REST API, you might want
more structured data back from the method calls. By
appending a "format=" and then choosing either json or
xml, you can get that structured data back you've been
dreaming about.</para>
<example>
<title>cURL List Storage Container (JSON output)</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343?format=json</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 200 OK
X-Account-Object-Count: 0
X-Account-Bytes-Used: 0
X-Account-Container-Count: 5
Accept-Ranges: bytes
Content-Length: 187
Content-Type: application/json; charset=utf-8
X-Trans-Id: txd408573a51d2423c848cba191fbede9b
Date: Mon, 07 Nov 2011 19:17:33 GMT
[{"name":"cosmo", "count":0,"bytes":0},
{"name":"dogs","count":0,"bytes":0},
{"name":"elaine","count":0,"bytes":0},
{"name":"george","count":0,"bytes":0},
{"name":"jerry","count":0,"bytes":0}]</computeroutput></screen>
</example>
<example>
<title>cURL List Storage Container (XML output)</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343?format=xml</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 200 OK
X-Account-Object-Count: 0
X-Account-Bytes-Used: 0
X-Account-Container-Count: 5
Accept-Ranges: bytes
Content-Length: 479
Content-Type: application/xml; charset=utf-8
X-Trans-Id: tx5e5685a15d0b406799b6a425b1150e4c
Date: Mon, 07 Nov 2011 19:17:38 GMT
&lt;?xml version="1.0" encoding="UTF-8"?>
&lt;account name="AUTH_a23f73d2-abfb-4656-af94-32ddec35dab8">
&lt;container>&lt;name>cosmo&lt;/name>&lt;count>0&lt;/count>&lt;bytes>0&lt;/bytes>&lt;/container>
&lt;container>&lt;name>dogs&lt;/name>&lt;count>0&lt;/count>&lt;bytes>0&lt;/bytes>&lt;/container>
&lt;container>&lt;name>elaine&lt;/name>&lt;count>0&lt;/count>&lt;bytes>0&lt;/bytes>&lt;/container>
&lt;container>&lt;name>george&lt;/name>&lt;count>0&lt;/count>&lt;bytes>0&lt;/bytes>&lt;/container>
&lt;container>&lt;name>jerry&lt;/name>&lt;count>0&lt;/count>&lt;bytes>0&lt;/bytes>&lt;/container>
&lt;/account></computeroutput></screen>
</example>
<para>The remainder of the examples in this document will use
the standard, non-serialized output but all operations
accept the format argument. You might notice that when you
use one of the formats, you get more information about the
containers. That's the per-container metadata, which is
covered in the next section.</para>
</section>
<section xml:id="container-metadata-and-delete-containers">
<title>Container Metadata and Deleting Containers</title>
<para>You can get at container metadata directly simply by
appending the name of the container to a HEAD
request:</para>
<example>
<title>cURL List Container Metadata</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X HEAD</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/dogs</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 204 No Content
X-Container-Object-Count: 0
X-Container-Bytes-Used: 0
Accept-Ranges: bytes
X-Trans-Id: tx3dd984f9482341dd97546e9d49d65e90
Content-Length: 0
Date: Mon, 07 Nov 2011 20:39:41 GMT</computeroutput></screen>
</example>
<para>Not very exciting without any objects in the container,
but you get the idea. While you cannot update or delete
container metadata, you can delete a container:</para>
<example>
<title>cURL Delete Storage Container</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X DELETE</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/george</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 204 No Content
Content-Length: 0
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx3fa3857f266f44319d9b8f4bf7ce7fc8
Date: Mon, 07 Nov 2011 20:42:58 GMT</computeroutput></screen>
</example>
<para>Then let's confirm the delete by listing the containers
again:</para>
<example>
<title>cURL List Containers After a Delete</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 200 OK
X-Account-Object-Count: 0
X-Account-Bytes-Used: 0
X-Account-Container-Count: 4
Accept-Ranges: bytes
Content-Length: 24
Content-Type: text/plain; charset=utf-8
X-Trans-Id: tx2475741852b849ce9403e382fe3f8015
Date: Mon, 07 Nov 2011 20:43:08 GMT
cosmo
dogs
elaine
jerry</computeroutput></screen>
</example>
</section>
<section xml:id="special-metadata-acls">
<title>Special Metadata: Container ACLs</title>
<para>A particularly important metadata element for containers
is X-Container-Read, which establishes the ACL permissions
on who can read objects in the container. Prior to being
set, the ACL logic default to only be accessible to
someone with a valid X-Auth-Token for the account in
question. Doing a simple listing of a container shows us
the absence of X-Container-Read in this default
situation:</para>
<example>
<title>cURL List Container Showing Lack of ACL</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/jerry</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 204 No Content
X-Container-Object-Count: 0
X-Container-Bytes-Used: 0
Accept-Ranges: bytes
X-Trans-Id: tx3aa52e951fc64b63bc1fda27902b9bd3
Content-Length: 0
Date: Tue, 15 Nov 2011 03:29:22 GMT</computeroutput></screen>
</example>
<para>Now we'll set the X-Container-Read. For a full
explanation of valid values, see:
http://swift.openstack.org/misc.html#acls but for our
simple needs, we'll enable read access and listing access
to anybody:</para>
<example>
<title>cURL Setting an ACL on a Container</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X PUT</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<option>-H "X-Container-Read: .r:*,.rlistings"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/jerry</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 202 Accepted
Content-Length: 58
Content-Type: text/html; charset=UTF-8
X-Trans-Id: txf2befb56b1854a50995f710f2db48089
Date: Tue, 15 Nov 2011 03:33:16 GMT
202 Accepted
The request is accepted for processing.</computeroutput></screen>
</example>
<para>To see the metadata change, do a listing again:</para>
<example>
<title>cURL List Container Showing with an ACL</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/jerry</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 204 No Content
X-Container-Object-Count: 0
X-Container-Read: .r:*,.rlistings
X-Container-Bytes-Used: 0
Accept-Ranges: bytes
X-Trans-Id: txb40eb86d949345f7bc66b01e8b63c3a5
Content-Length: 0
Date: Tue, 15 Nov 2011 03:33:36 GMT</computeroutput></screen>
</example>
<para>The side effect of giving anybody read access is that
any object in the container is now accessible from a
browser simply by entering the X-Storage-URL used
throughout the session and append the object name. For
example:</para>
<para>https://storage.swiftdrive.com/v1/CF_xer7_343/jerry/cereal.jpg</para>
<para>would be the URL of an object named "cereal.jpg" in the
container "jerry" that has been made publicly accessible
using this method.</para>
</section>
<section xml:id="creating-objects">
<title>Creating Objects</title>
<para>Enough with containers already, let's start to upload
some objects. Suppose you had a local directory full of
dog pictures:</para>
<example>
<title>Sample File Listing</title>
<screen><prompt>$</prompt> <userinput>ls -l</userinput>
<computeroutput>
total 504
-rw-r--r--@ 1 petecj2 staff 44765 Nov 7 14:49 JingleRocky.jpg
-rw-r--r--@ 1 petecj2 staff 100864 Nov 7 14:47 RockyAndBuster.jpg
-rw-r--r--@ 1 petecj2 staff 107103 Nov 7 14:47 SittingBuster.jpg</computeroutput></screen>
</example>
<para>In order to put one of them in a container called "dogs"
with cURL, you'd do this:</para>
<example>
<title>Creating and Uploading an Object to a
Container</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X PUT</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<option>-T JingleRocky.jpg</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/JingleRocky.jpg</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 201 Created
Content-Length: 118
Content-Type: text/html; charset=UTF-8
Etag: f7d40eceffdd9c2ecab226105737b2a6
Last-Modified: Mon, 07 Nov 2011 22:51:29 GMT
X-Trans-Id: txd131cc897c78403daf5fad010d4d7152
Date: Mon, 07 Nov 2011 22:51:30 GMT
&lt;html>
&lt;head>
&lt;title>201 Created&lt;/title>
&lt;/head>
&lt;body>
&lt;h1>201 Created&lt;/h1>
&lt;br />&lt;br />
&lt;/body>
&lt;/html></computeroutput></screen>
</example>
<para>The object gets named from whatever we append to the URL
path beyond the container name and the -T switch lets us
name a file to push with the operation as the request
body. We can confirm the upload by checking the container
again:</para>
<example>
<title>cURL List Container Showing Newly Uploaded
Object</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/dogs</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 200 OK
X-Container-Object-Count: 1
X-Container-Read: .r:*,.rlistings
X-Container-Bytes-Used: 44765
Accept-Ranges: bytes
Content-Length: 16
Content-Type: text/plain; charset=utf-8
X-Trans-Id: tx83be89d4e1a34eacbfeebcdfc7a7f2e7
Date: Mon, 07 Nov 2011 22:56:25 GMT
JingleRocky.jpg</computeroutput></screen>
</example>
<para>Notice that the container metadata now reflects the
number of objects and the bytes match what we saw when we
did the directory listing. After uploading the other two
similarly, we get a full object listing:</para>
<example>
<title>cURL List Container Showing Multiple Newly Uploaded
Objects</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/dogs</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 200 OK
X-Container-Object-Count: 3
X-Container-Read: .r:*,.rlistings
X-Container-Bytes-Used: 252732
Accept-Ranges: bytes
Content-Length: 53
Content-Type: text/plain; charset=utf-8
X-Trans-Id: txae17dfa78da64117aaf07585a1b02115
Date: Mon, 07 Nov 2011 23:00:56 GMT
JingleRocky.jpg
RockyAndBuster.jpg
SittingBuster.jpg</computeroutput></screen>
</example>
</section>
<section xml:id="create_static_large_objects">
<title>Creating Static Large Objects</title>
<para>Creation of a static large object is done in several
steps. First we divide the content into pieces and upload
each piece into a segment object. Then we create a
manifest object. In this example, we will place the
segment objects into the "segments" container and the
manifest object into the "images" container. We are not
required to do this, but using a dedicated container for
segment objects is convenient.</para>
<para>Assuming we've already divided our image into three
files, let's upload them. We have removed non-essential
response headers so you can see the important
details.</para>
<example>
<title>Uploading first segment</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X PUT</option> <option>-i</option> <option>-H "X-Auth-Token: 12345"</option> <option>-T ./piece1</option>
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/segments/terrier-jpg-one</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 201 Created
Content-Length: 4000000
Etag: f7365c1419b4f349592c00bd0cfb9b9a</computeroutput></screen>
</example>
<example>
<title>Uploading second segment</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X PUT</option> <option>-i</option> <option>-H "X-Auth-Token: 12345"</option> <option>-T ./piece2</option>
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/segments/terrier-jpg-two</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 201 Created
Content-Length: 2000000
Etag: ad81e97b10e870613aecb5ced52adbaa</computeroutput></screen>
</example>
<example>
<title>Uploading final segment</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X PUT</option> <option>-i</option> <option>-H "X-Auth-Token: 12345"</option> <option>-T ./piece3</option>
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/segments/terrier-jpg-three</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 201 Created
Content-Length: 1000
Etag: 00b046c9d74c3e8f93b320c5e5fdc2c3</computeroutput></screen>
</example>
<para>At this stage we are ready to create the manifest
listing. Notice that the size and ETag are copied from
uploads above. Lets use an editor to create a file called
<code>manifest.json</code> with the following
content:</para>
<example>
<title>Manifest List Example</title>
<screen><computeroutput> [
{
"path": "segments/terrier-jpg-one",
"etag": "f7365c1419b4f349592c00bd0cfb9b9a",
"size_bytes": 4000000
},
{
"path": "segments/terrier-jpg-two",
"etag": "ad81e97b10e870613aecb5ced52adbaa",
"size_bytes": 2000000
},
"path": "segments/terrier-jpg-three",
"etag": "00b046c9d74c3e8f93b320c5e5fdc2c3",
"size_bytes": 1000
{
}
]</computeroutput></screen>
</example>
<para>The final operation is to upload this content into a
manifest object. To indicate that this is a manifest
object, you need to specify the
<code>?multipart-manifest=put</code> query
string.</para>
<example>
<title>Uploading manifest object</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X PUT</option> <option>-i</option> <option>-H "X-Auth-Token: 12345"</option> <option>-T ./manifest.json</option>
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/images/terrier-jpg?multipart-manifest=put</uri></userinput></screen>
</example>
<para>We can now examine our static large object. Notice that
the size is the total size of all the segments.</para>
<example>
<title>Examining a manifest object</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X HEAD</option> <option>-i</option> <option>-H "X-Auth-Token: 12345"</option>
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/images/terrier-jpg</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 200 OK
Content-Length: 6001000
Etag: "0c922c37f915efb1c9b97e6328b3e660"</computeroutput></screen>
</example>
</section>
<section xml:id="paging-objects">
<title>Paging Lists of Objects</title>
<para>Exactly like listing containers, objects can be listed
in pages at a time using markers to denote pages. From the
previous example with 3 objects in the container "dogs",
the list can be paged with the "limit" query string
variable:</para>
<example>
<title>cURL List Objects (first page)</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/dogs?limit=2</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 200 OK
X-Container-Object-Count: 3
X-Container-Read: .r:*,.rlistings
X-Container-Bytes-Used: 252732
Accept-Ranges: bytes
Content-Length: 35
Content-Type: text/plain; charset=utf-8
X-Trans-Id: tx5e00fa9fa895423198bc814cb0c6162d
Date: Tue, 15 Nov 2011 03:53:51 GMT
JingleRocky.jpg
RockyAndBuster.jpg</computeroutput></screen>
</example>
<para>And the second page fetched with:</para>
<example>
<title>cURL List Objects with Paging (later pages)</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/dogs?marker=RockyAndBuster.jpg\&amp;limit=2</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 200 OK
X-Container-Object-Count: 3
X-Container-Read: .r:*,.rlistings
X-Container-Bytes-Used: 252732
Accept-Ranges: bytes
Content-Length: 18
Content-Type: text/plain; charset=utf-8
X-Trans-Id: txe1287a7179dc4dfd98610850a0fff157
Date: Tue, 15 Nov 2011 03:54:21 GMT
SittingBuster.jpg</computeroutput></screen>
</example>
</section>
<section xml:id="retrieve-copy-retrieve-objects">
<title>Retrieve, Copy, and Delete Objects</title>
<para>Now we'll retrieve an object previously uploaded. First,
we'll remove the local copy:</para>
<example>
<title>Removing Local Copies</title>
<screen><prompt>$</prompt> <userinput>ls -l</userinput>
<computeroutput>
total 504
-rw-r--r--@ 1 petecj2 staff 44765 Nov 7 14:49 JingleRocky.jpg
-rw-r--r--@ 1 petecj2 staff 100864 Nov 7 14:47 RockyAndBuster.jpg
-rw-r--r--@ 1 petecj2 staff 107103 Nov 7 14:47 SittingBuster.jpg
$ rm JingleRocky.jpg
$ ls -l
total 416
-rw-r--r--@ 1 petecj2 staff 100864 Nov 7 14:47 RockyAndBuster.jpg
-rw-r--r--@ 1 petecj2 staff 107103 Nov 7 14:47 SittingBuster.jpg</computeroutput></screen>
</example>
<para>Be sure not to use -i switch here since what we want is
the raw data, which we'll then pipe to a file:</para>
<example>
<title>cURL Retrieve an Object</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/JingleRocky.jpg > JingleRocky.jpg</uri></userinput></screen>
<screen><prompt>$</prompt> <userinput>ls -l</userinput>
<computeroutput>
total 504
-rw-r--r-- 1 petecj2 staff 44765 Nov 7 15:11 JingleRocky.jpg
-rw-r--r--@ 1 petecj2 staff 100864 Nov 7 14:47 RockyAndBuster.jpg
-rw-r--r--@ 1 petecj2 staff 107103 Nov 7 14:47 SittingBuster.jpg</computeroutput></screen>
</example>
<para>Next, Object Storage provides a facility to copy objects
from one container to another entirely on the server side.
To do this, you do a PUT with the destination container
and new object name while passing a special X-Copy-From
header and a Content-Length of zero:</para>
<example>
<title>cURL Server-side Copy an Object</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X PUT</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<option>-H "X-Copy-From: /dogs/JingleRocky.jpg"</option> \
<option>-H "Content-Length: 0"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/elaine/JingleRocky.jpg</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 201 Created
Content-Length: 118
Content-Type: text/html; charset=UTF-8
Etag: f7d40eceffdd9c2ecab226105737b2a6
X-Copied-From: dogs/JingleRocky.jpg
Last-Modified: Mon, 07 Nov 2011 23:23:53 GMT
X-Trans-Id: tx244cd14df1b94d8c91ec5dcf8c5f9da4
Date: Mon, 07 Nov 2011 23:23:54 GMT
&lt;html>
&lt;head>
&lt;title>201 Created&lt;/title>
&lt;/head>
&lt;body>
&lt;h1>201 Created&lt;/h1>
&lt;br />&lt;br />
&lt;/body>
&lt;/html></computeroutput></screen>
</example>
<para>You can then confirm the new location of the object. To
do this, you do a GET with the destination container to
see the listing of the object:</para>
<example>
<title>cURL Confirming the Server-side Copy an
Object</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/elaine/</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 200 OK
X-Container-Object-Count: 1
X-Container-Bytes-Used: 44765
Accept-Ranges: bytes
Content-Length: 16
Content-Type: text/plain; charset=utf-8
X-Trans-Id: tx46986b4a09b34790924fd43842b2b0dd
Date: Mon, 07 Nov 2011 23:24:05 GMT
JingleRocky.jpg</computeroutput></screen>
</example>
<para>To delete an object from its container, simply use the
DELETE verb:</para>
<example>
<title>cURL Delete an Object</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X DELETE</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/elaine/JingleRocky.jpg</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 204 No Content
Content-Length: 0
Content-Type: text/html; charset=UTF-8
X-Trans-Id: txd45f04422b034e6f8447de400b78cbf3
Date: Mon, 07 Nov 2011 23:32:39 GMT</computeroutput></screen>
</example>
<para>Confirming the deletion by doing a container
listing:</para>
<example>
<title>cURL Confirming the Delete an Object</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/elaine/</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 204 No Content
X-Container-Object-Count: 0
X-Container-Bytes-Used: 0
Accept-Ranges: bytes
X-Trans-Id: txc9b43bf4d896405eb9a88ca468bf7b2d
Content-Length: 0
Date: Mon, 07 Nov 2011 23:32:41 GMT</computeroutput></screen>
</example>
</section>
<section xml:id="object-metadata">
<title>Object Metadata</title>
<para>Objects can have whatever metadata keys/values you
choose. Simply POST an HTTP Header to the object in the
form of X-Object-Meta-&lt;key>: &lt;value>. Like
this:</para>
<example>
<title>cURL Set Object Metadata</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X POST</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<option>-H "X-Object-Meta-Breed: Terrier pit bull mix""</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/JingleRocky.jpg</uri></userinput></screen>
<screen><computeroutput> &lt;html>
&lt;head>
&lt;title>202 Accepted&lt;/title>
&lt;/head>
&lt;body>
&lt;h1>202 Accepted&lt;/h1>
The request is accepted for processing.&lt;br />&lt;br />
&lt;/body>
&lt;/html></computeroutput></screen>
</example>
<para>And then read the object metadata with a HEAD on the
object path:</para>
<example>
<title>cURL Reading Object Metadata</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X HEAD</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/JingleRocky.jpg</uri></userinput></screen>
<screen><computeroutput> HTTP/1.1 200 OK
X-Object-Meta-Breed: Terrier pit bull mix
Last-Modified: Tue, 08 Nov 2011 01:26:49 GMT
Etag: f7d40eceffdd9c2ecab226105737b2a6
Accept-Ranges: bytes
Content-Length: 44765
Content-Type: image/jpeg
X-Trans-Id: txa8bff9ad7ef844829103c1f9b8c20781
Date: Tue, 08 Nov 2011 01:29:35 GMT</computeroutput></screen>
</example>
</section>
<section xml:id="folders-directories">
<title>Pseudo-Hierarchical Folders/Directories</title>
<para>For the last section, we come to the most confusing
concept in Object Storage. In most storage systems, you
have the ability to create custom hierarchies of files so
that you can better organize them. On its surface, Object
Storage only gives you one level of hierarchy in the form
of containers. However, it turns out that you can get
creative with naming your objects to give yourself the
same effect as having hierarchical containers.</para>
<para>Let's start with a fresh container without any objects
in it:</para>
<example>
<title>cURL Create New Container for Folders</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X PUT</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/photos</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 201 Created
Content-Length: 18
Content-Type: text/html; charset=UTF-8
X-Trans-Id: txc78254a41b374b6ea10590d90874f769
Date: Wed, 16 Nov 2011 00:06:22 GMT
201 Created</computeroutput></screen>
</example>
<para>Now list the new container:</para>
<example>
<title>cURL Listing the New Container</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/photos</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 204 No Content
X-Container-Object-Count: 0
X-Container-Bytes-Used: 0
Accept-Ranges: bytes
X-Trans-Id: tx49112200f7934c2bab1de3ae103c368e
Content-Length: 0
Date: Wed, 16 Nov 2011 00:06:26 GMT</computeroutput></screen>
</example>
<para>Next, add an object but prefix the name with the
hierarchy desired:</para>
<example>
<title>cURL Upload an Object with a Prefix</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X PUT</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<option>-T JingleRocky.jpg</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/photos/terriers/JingleRocky.jpg</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 201 Created
Content-Length: 118
Content-Type: text/html; charset=UTF-8
Etag: f7d40eceffdd9c2ecab226105737b2a6
Last-Modified: Wed, 16 Nov 2011 00:09:18 GMT
X-Trans-Id: txe34fdf2704f044e3a7102256386b1cb7
Date: Wed, 16 Nov 2011 00:09:19 GMT
&lt;html>
&lt;head>
&lt;title>201 Created&lt;/title>
&lt;/head>
&lt;body>
&lt;h1>201 Created&lt;/h1>
&lt;br />&lt;br />
&lt;/body>
&lt;/html></computeroutput></screen>
</example>
<para>Do it again with a different object and prefix:</para>
<example>
<title>cURL Upload a Different Object with a Different
Prefix</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X PUT</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<option>-T SittingBuster.jpg</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/photos/chihuahuas/SittingBuster.jpg</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 201 Created
Content-Length: 118
Content-Type: text/html; charset=UTF-8
Etag: e692e744c7180ee368166a24f1a2fa9b
Last-Modified: Wed, 16 Nov 2011 00:52:25 GMT
X-Trans-Id: txe229d03af5ea4d2ea1071def213c3f02
Date: Wed, 16 Nov 2011 00:52:25 GMT
&lt;html>
&lt;head>
&lt;title>201 Created&lt;/title>
&lt;/head>
&lt;body>
&lt;h1>201 Created&lt;/h1>
&lt;br />&lt;br />
&lt;/body>
&lt;/html></computeroutput></screen>
</example>
<para>Now list the container, revealing the prefixes:</para>
<example>
<title>cURL Listing a Container with Object Prefix</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/photos</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 200 OK
X-Container-Object-Count: 2
X-Container-Bytes-Used: 151868
Accept-Ranges: bytes
Content-Length: 54
Content-Type: text/plain; charset=utf-8
X-Trans-Id: tx8544a17e8b1e4da693145fb5f2e6db43
Date: Wed, 16 Nov 2011 00:53:43 GMT
chihuahuas/SittingBuster.jpg
terriers/JingleRocky.jpg</computeroutput></screen>
</example>
<para>If you want to perform hierarchical listings, with the
prefixes in place, you can use the "path" query string
variable:</para>
<example>
<title>cURL Listing a Container with a Path</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/photos?path=terriers</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 200 OK
X-Container-Object-Count: 2
X-Container-Bytes-Used: 151868
Accept-Ranges: bytes
Content-Length: 25
Content-Type: text/plain; charset=utf-8
X-Trans-Id: tx3f1b9575d4de4a7d97ba3f9ad81923cc
Date: Wed, 16 Nov 2011 00:55:12 GMT
terriers/JingleRocky.jpg</computeroutput></screen>
</example>
<para>If you wanted to see what prefixes were in place, you
can use the "delimiter" query string variable to
distinguish prefix paths from object names:</para>
<example>
<title>cURL Listing a Container with a Delimiter</title>
<screen><prompt>$</prompt> <userinput><command>curl</command> <option>X GET</option> <option>-i</option> \
<option>-H "X-Auth-Token: fc81aaa6-98a1-9ab0-94ba-aba9a89aa9ae"</option> \
<uri>https://storage.swiftdrive.com/v1/CF_xer7_343/photos?delimiter=/</uri></userinput></screen>
<screen><computeroutput>HTTP/1.1 200 OK
X-Container-Object-Count: 2
X-Container-Bytes-Used: 151868
Accept-Ranges: bytes
Content-Length: 22
Content-Type: text/plain; charset=utf-8
X-Trans-Id: tx7222a3dd73fe44b888db4e58cc647d1e
Date: Wed, 16 Nov 2011 00:57:40 GMT
chihuahuas/
terriers/</computeroutput></screen>
</example>
<para>Using these in combination allows you to discover
directories within a particular path and then further
drill down based on the results.</para>
</section>
</chapter>