673 lines
36 KiB
XML
673 lines
36 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<section xmlns="http://docbook.org/ns/docbook"
|
|
xmlns:xi="http://www.w3.org/2001/XInclude"
|
|
xmlns:xlink="http://www.w3.org/1999/xlink"
|
|
version="5.0"
|
|
xml:id="configuring-object-storage-features">
|
|
<title>Configure Object Storage features</title>
|
|
<section xml:id="swift-zones">
|
|
<title>Object Storage zones</title>
|
|
<para>In OpenStack Object Storage, data is placed across
|
|
different tiers of failure domains. First, data is spread
|
|
across regions, then zones, then servers, and finally
|
|
across drives. Data is placed to get the highest failure
|
|
domain isolation. If you deploy multiple regions, the
|
|
Object Storage service places the data across the regions.
|
|
Within a region, each replica of the data should be stored
|
|
in unique zones, if possible. If there is only one zone,
|
|
data should be placed on different servers. And if there
|
|
is only one server, data should be placed on different
|
|
drives.</para>
|
|
<para>Regions are widely separated installations with a
|
|
high-latency or otherwise constrained network link between
|
|
them. Zones are arbitrarily assigned, and it is up to the
|
|
administrator of the Object Storage cluster to choose an
|
|
isolation level and attempt to maintain the isolation
|
|
level through appropriate zone assignment. For example, a
|
|
zone may be defined as a rack with a single power source.
|
|
Or a zone may be a DC room with a common utility provider.
|
|
Servers are identified by a unique IP/port. Drives are
|
|
locally attached storage volumes identified by mount
|
|
point.</para>
|
|
<para>In small clusters (five nodes or fewer), everything is
|
|
normally in a single zone. Larger Object Storage
|
|
deployments may assign zone designations differently; for
|
|
example, an entire cabinet or rack of servers may be
|
|
designated as a single zone to maintain replica
|
|
availability if the cabinet becomes unavailable (for
|
|
example, due to failure of the top of rack switches or a
|
|
dedicated circuit). In very large deployments, such as
|
|
service provider level deployments, each zone might have
|
|
an entirely autonomous switching and power infrastructure,
|
|
so that even the loss of an electrical circuit or
|
|
switching aggregator would result in the loss of a single
|
|
replica at most.</para>
|
|
<section xml:id="swift-zones-rackspacerecs">
|
|
<title>Rackspace zone recommendations</title>
|
|
<para>For ease of maintenance on OpenStack Object Storage,
|
|
Rackspace recommends that you set up at least five
|
|
nodes. Each node is assigned its own zone (for a total
|
|
of five zones), which gives you host level redundancy.
|
|
This enables you to take down a single zone for
|
|
maintenance and still guarantee object availability in
|
|
the event that another zone fails during your
|
|
maintenance.</para>
|
|
<para>You could keep each server in its own cabinet to achieve cabinet level isolation,
|
|
but you may wish to wait until your Object Storage service is better established
|
|
before developing cabinet-level isolation. OpenStack Object Storage is flexible; if
|
|
you later decide to change the isolation level, you can take down one zone at a time
|
|
and move them to appropriate new homes.</para>
|
|
</section>
|
|
</section>
|
|
<section xml:id="swift-raid-controller">
|
|
<title>RAID controller configuration</title>
|
|
<para>OpenStack Object Storage does not require RAID. In fact,
|
|
most RAID configurations cause significant performance
|
|
degradation. The main reason for using a RAID controller
|
|
is the battery-backed cache. It is very important for data
|
|
integrity reasons that when the operating system confirms
|
|
a write has been committed that the write has actually
|
|
been committed to a persistent location. Most disks lie
|
|
about hardware commits by default, instead writing to a
|
|
faster write cache for performance reasons. In most cases,
|
|
that write cache exists only in non-persistent memory. In
|
|
the case of a loss of power, this data may never actually
|
|
get committed to disk, resulting in discrepancies that the
|
|
underlying file system must handle.</para>
|
|
<para>OpenStack Object Storage works best on the XFS file
|
|
system, and this document assumes that the hardware being
|
|
used is configured appropriately to be mounted with the
|
|
<command>nobarriers</command> option. For more
|
|
information, refer to the XFS FAQ: <link
|
|
xlink:href="http://xfs.org/index.php/XFS_FAQ"
|
|
>http://xfs.org/index.php/XFS_FAQ</link>
|
|
</para>
|
|
<para>To get the most out of your hardware, it is essential
|
|
that every disk used in OpenStack Object Storage is
|
|
configured as a standalone, individual RAID 0 disk; in the
|
|
case of 6 disks, you would have six RAID 0s or one JBOD.
|
|
Some RAID controllers do not support JBOD or do not
|
|
support battery backed cache with JBOD. To ensure the
|
|
integrity of your data, you must ensure that the
|
|
individual drive caches are disabled and the battery
|
|
backed cache in your RAID card is configured and used.
|
|
Failure to configure the controller properly in this case
|
|
puts data at risk in the case of sudden loss of
|
|
power.</para>
|
|
<para>You can also use hybrid drives or similar options for
|
|
battery backed up cache configurations without a RAID
|
|
controller.</para>
|
|
</section>
|
|
<section xml:id="object-storage-rate-limits">
|
|
<?dbhtml stop-chunking?>
|
|
<title>Throttle resources through rate limits</title>
|
|
<para>Rate limiting in OpenStack Object Storage is implemented
|
|
as a pluggable middleware that you configure on the proxy
|
|
server. Rate limiting is performed on requests that result
|
|
in database writes to the account and container SQLite
|
|
databases. It uses memcached and is dependent on the proxy
|
|
servers having highly synchronized time. The rate limits
|
|
are limited by the accuracy of the proxy server
|
|
clocks.</para>
|
|
<section xml:id="configuration-for-rate-limiting">
|
|
<title>Configure rate limiting</title>
|
|
<para>All configuration is optional. If no account or
|
|
container limits are provided, no rate limiting
|
|
occurs. Available configuration options
|
|
include:</para>
|
|
<xi:include
|
|
href="../../common/tables/swift-proxy-server-filter-ratelimit.xml"/>
|
|
<para>The container rate limits are linearly interpolated
|
|
from the values given. A sample container rate
|
|
limiting could be:</para>
|
|
<para>container_ratelimit_100 = 100</para>
|
|
<para>container_ratelimit_200 = 50</para>
|
|
<para>container_ratelimit_500 = 20</para>
|
|
<para>This would result in:</para>
|
|
<table rules="all">
|
|
<caption>Values for Rate Limiting with Sample
|
|
Configuration Settings</caption>
|
|
<tbody>
|
|
<tr>
|
|
<td>Container Size</td>
|
|
<td>Rate Limit</td>
|
|
</tr>
|
|
<tr>
|
|
<td>0-99</td>
|
|
<td>No limiting</td>
|
|
</tr>
|
|
<tr>
|
|
<td>100</td>
|
|
<td>100</td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td>150</td>
|
|
<td>75</td>
|
|
</tr>
|
|
<tr>
|
|
<td>500</td>
|
|
<td>20</td>
|
|
</tr>
|
|
<tr>
|
|
<td>1000</td>
|
|
<td>20</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</section>
|
|
</section>
|
|
<section xml:id="object-storage-healthcheck">
|
|
<title>Health check</title>
|
|
<para>Provides an easy way to monitor whether the Object Storage proxy server is alive. If
|
|
you access the proxy with the path <filename>/healthcheck</filename>, it responds with
|
|
<literal>OK</literal> in the response body, which monitoring tools can use.</para>
|
|
<xi:include
|
|
href="../../common/tables/swift-account-server-filter-healthcheck.xml"
|
|
/>
|
|
</section>
|
|
<section xml:id="object-storage-domain-remap">
|
|
<title>Domain remap</title>
|
|
<para>Middleware that translates container and account parts
|
|
of a domain to path parameters that the proxy server
|
|
understands.</para>
|
|
<xi:include
|
|
href="../../common/tables/swift-proxy-server-filter-domain_remap.xml"
|
|
/>
|
|
</section>
|
|
<section xml:id="object-storage-cname-lookup">
|
|
<title>CNAME lookup</title>
|
|
<para>Middleware that translates an unknown domain in the host
|
|
header to something that ends with the configured
|
|
<code>storage_domain</code> by looking up the given domain's CNAME
|
|
record in DNS.</para>
|
|
<xi:include
|
|
href="../../common/tables/swift-proxy-server-filter-cname_lookup.xml"
|
|
/>
|
|
</section>
|
|
<section xml:id="object-storage-tempurl">
|
|
<?dbhtml stop-chunking?>
|
|
<title>Temporary URL</title>
|
|
<para>Allows the creation of URLs to provide temporary access to objects. For example, a
|
|
website may wish to provide a link to download a large object in OpenStack Object
|
|
Storage, but the Object Storage account has no public access. The website can generate a
|
|
URL that provides GET access for a limited time to the resource. When the web browser
|
|
user clicks on the link, the browser downloads the object directly from Object Storage,
|
|
eliminating the need for the website to act as a proxy for the request. If the user
|
|
shares the link with all his friends, or accidentally posts it on a forum, the direct
|
|
access is limited to the expiration time set when the website created the link.</para>
|
|
<para>A temporary URL is the typical URL associated with an
|
|
object, with two additional query parameters:<variablelist>
|
|
<varlistentry>
|
|
<term><literal>temp_url_sig</literal></term>
|
|
<listitem>
|
|
<para>A cryptographic signature</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term><literal>temp_url_expires</literal></term>
|
|
<listitem>
|
|
<para>An expiration date, in Unix time</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist></para>
|
|
<para>An example of a temporary
|
|
URL:<programlisting>
|
|
https://swift-cluster.example.com/v1/AUTH_a422b2-91f3-2f46-74b7-d7c9e8958f5d30/container/object?
|
|
temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709&
|
|
temp_url_expires=1323479485
|
|
</programlisting></para>
|
|
<para>To create temporary URLs, first set the <literal>X-Account-Meta-Temp-URL-Key</literal>
|
|
header on your Object Storage account to an arbitrary string. This string serves as a
|
|
secret key. For example, to set a key of
|
|
<literal>b3968d0207b54ece87cccc06515a89d4</literal> using the
|
|
<command>swift</command> command-line tool:</para>
|
|
<screen><prompt>$</prompt> <userinput>swift post -m "Temp-URL-Key:<replaceable>b3968d0207b54ece87cccc06515a89d4</replaceable>"</userinput></screen>
|
|
<para>Next, generate an HMAC-SHA1 (RFC 2104) signature to
|
|
specify:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>Which HTTP method to allow (typically
|
|
<literal>GET</literal> or
|
|
<literal>PUT</literal>)</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>The expiry date as a Unix timestamp</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>The full path to the object</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>The secret key set as the
|
|
<literal>X-Account-Meta-Temp-URL-Key</literal></para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<para>Here is code generating the signature for a GET for 24
|
|
hours on
|
|
<code>/v1/AUTH_account/container/object</code>:</para>
|
|
<programlisting language="python">import hmac
|
|
from hashlib import sha1
|
|
from time import time
|
|
method = 'GET'
|
|
duration_in_seconds = 60*60*24
|
|
expires = int(time() + duration_in_seconds)
|
|
path = '/v1/AUTH_a422b2-91f3-2f46-74b7-d7c9e8958f5d30/container/object'
|
|
key = 'mykey'
|
|
hmac_body = '%s\n%s\n%s' % (method, expires, path)
|
|
sig = hmac.new(key, hmac_body, sha1).hexdigest()
|
|
s = 'https://{host}/{path}?temp_url_sig={sig}&temp_url_expires={expires}'
|
|
url = s.format(host='swift-cluster.example.com', path=path, sig=sig, expires=expires)</programlisting>
|
|
<para>Any alteration of the resource path or query arguments
|
|
results in a <errorcode>401</errorcode>
|
|
<errortext>Unauthorized</errortext> error. Similarly, a
|
|
PUT where GET was the allowed method returns a
|
|
<errorcode>401</errorcode>. HEAD is allowed if GET or
|
|
PUT is allowed. Using this in combination with browser
|
|
form post translation middleware could also allow
|
|
direct-from-browser uploads to specific locations in
|
|
Object Storage.</para>
|
|
<note>
|
|
<para>Changing the
|
|
<literal>X-Account-Meta-Temp-URL-Key</literal>
|
|
invalidates any previously generated temporary
|
|
URLs within 60 seconds (the memcache time for the
|
|
key). Object Storage supports up to two keys, specified by
|
|
<literal>X-Account-Meta-Temp-URL-Key</literal>
|
|
and
|
|
<literal>X-Account-Meta-Temp-URL-Key-2</literal>.
|
|
Signatures are checked against both keys, if
|
|
present. This is to allow for key rotation without
|
|
invalidating all existing temporary URLs.</para>
|
|
</note>
|
|
<para>Object Storage includes a script called
|
|
<command>swift-temp-url</command> that generates the
|
|
query parameters automatically:</para>
|
|
<screen><prompt>$</prompt> <userinput>bin/swift-temp-url GET 3600 /v1/AUTH_account/container/object mykey</userinput>
|
|
<computeroutput>/v1/AUTH_account/container/object?
|
|
temp_url_sig=5c4cc8886f36a9d0919d708ade98bf0cc71c9e91&
|
|
temp_url_expires=1374497657</computeroutput></screen>
|
|
<para>Because this command only returns the path, you must
|
|
prefix the Object Storage host name (for example,
|
|
<literal>https://swift-cluster.example.com</literal>).</para>
|
|
<para>With GET Temporary URLs, a
|
|
<literal>Content-Disposition</literal> header is set
|
|
on the response so that browsers interpret this as a file
|
|
attachment to be saved. The file name chosen is based on
|
|
the object name, but you can override this with a
|
|
<literal>filename</literal> query parameter. The
|
|
following example specifies a filename of <filename>My
|
|
Test File.pdf</filename>:</para>
|
|
<programlisting>https://swift-cluster.example.com/v1/AUTH_a422b2-91f3-2f46-74b7-d7c9e8958f5d30/container/object?
|
|
temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709&
|
|
temp_url_expires=1323479485&
|
|
filename=My+Test+File.pdf</programlisting>
|
|
|
|
<para>If you do not want the object to be downloaded, you can cause
|
|
<literal>Content-Disposition: inline</literal>
|
|
to be set on the response by adding
|
|
the <literal>inline</literal> parameter to the query string, as follows:</para>
|
|
<programlisting>https://swift-cluster.example.com/v1/AUTH_account/container/object?
|
|
temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709&
|
|
temp_url_expires=1323479485&inline</programlisting>
|
|
|
|
<para>To enable Temporary URL functionality, edit
|
|
<filename>/etc/swift/proxy-server.conf</filename> to
|
|
add <literal>tempurl</literal> to the
|
|
<literal>pipeline</literal> variable defined in the
|
|
<literal>[pipeline:main]</literal> section. The
|
|
<literal>tempurl</literal> entry should appear
|
|
immediately before the authentication filters in the
|
|
pipeline, such as <literal>authtoken</literal>,
|
|
<literal>tempauth</literal> or
|
|
<literal>keystoneauth</literal>. For
|
|
example:<programlisting>[pipeline:main]
|
|
pipeline = pipeline = healthcheck cache <emphasis role="bold">tempurl</emphasis> authtoken keystoneauth proxy-server</programlisting></para>
|
|
|
|
<xi:include
|
|
href="../../common/tables/swift-proxy-server-filter-tempurl.xml"
|
|
/>
|
|
</section>
|
|
<section xml:id="object-storage-name-check">
|
|
<title>Name check filter</title>
|
|
<para>Name Check is a filter that disallows any paths that
|
|
contain defined forbidden characters or that exceed a
|
|
defined length.</para>
|
|
<xi:include
|
|
href="../../common/tables/swift-proxy-server-filter-name_check.xml"
|
|
/>
|
|
</section>
|
|
<section xml:id="object-storage-constraints">
|
|
<title>Constraints</title>
|
|
<para>To change the OpenStack Object Storage internal limits,
|
|
update the values in the
|
|
<literal>swift-constraints</literal> section in the
|
|
<filename>swift.conf</filename> file. Use caution when
|
|
you update these values because they affect the
|
|
performance in the entire cluster.</para>
|
|
<xi:include
|
|
href="../../common/tables/swift-swift-swift-constraints.xml"
|
|
/>
|
|
</section>
|
|
<section xml:id="object-storage-dispersion">
|
|
<title>Cluster health</title>
|
|
<para>Use the <command>swift-dispersion-report</command> tool
|
|
to measure overall cluster health. This tool checks if a
|
|
set of deliberately distributed containers and objects are
|
|
currently in their proper places within the cluster. For
|
|
instance, a common deployment has three replicas of each
|
|
object. The health of that object can be measured by
|
|
checking if each replica is in its proper place. If only 2
|
|
of the 3 is in place the object's health can be said to be
|
|
at 66.66%, where 100% would be perfect. A single object's
|
|
health, especially an older object, usually reflects the
|
|
health of that entire partition the object is in. If you
|
|
make enough objects on a distinct percentage of the
|
|
partitions in the cluster,you get a good estimate of the
|
|
overall cluster health. In practice, about 1% partition
|
|
coverage seems to balance well between accuracy and the
|
|
amount of time it takes to gather results. The first thing
|
|
that needs to be done to provide this health value is
|
|
create a new account solely for this usage. Next, you need
|
|
to place the containers and objects throughout the system
|
|
so that they are on distinct partitions. The
|
|
<command>swift-dispersion-populate</command> tool does this
|
|
by making up
|
|
random container and object names until they fall on
|
|
distinct partitions. Last, and repeatedly for the life of
|
|
the cluster, you must run the
|
|
<command>swift-dispersion-report</command> tool to
|
|
check the health of each of these containers and objects.
|
|
These tools need direct access to the entire cluster and
|
|
to the ring files (installing them on a proxy server
|
|
suffices). The
|
|
<command>swift-dispersion-populate</command> and
|
|
<command>swift-dispersion-report</command> commands
|
|
both use the same configuration file,
|
|
<filename>/etc/swift/dispersion.conf</filename>.
|
|
Example <filename>dispersion.conf</filename> file:</para>
|
|
<programlisting language="ini">
|
|
[dispersion]
|
|
auth_url = http://localhost:8080/auth/v1.0
|
|
auth_user = test:tester
|
|
auth_key = testing
|
|
</programlisting>
|
|
<para>There are also configuration options for specifying the
|
|
dispersion coverage, which defaults to 1%, retries,
|
|
concurrency, and so on. However, the defaults are usually
|
|
fine. Once the configuration is in place, run
|
|
<command>swift-dispersion-populate</command> to
|
|
populate the containers and objects throughout the
|
|
cluster. Now that those containers and objects are in
|
|
place, you can run
|
|
<command>swift-dispersion-report</command> to get a
|
|
dispersion report, or the overall health of the cluster.
|
|
Here is an example of a cluster in perfect health:</para>
|
|
<screen><prompt>$</prompt> <userinput>swift-dispersion-report</userinput>
|
|
<computeroutput>Queried 2621 containers for dispersion reporting, 19s, 0 retries
|
|
100.00% of container copies found (7863 of 7863)
|
|
Sample represents 1.00% of the container partition space
|
|
|
|
Queried 2619 objects for dispersion reporting, 7s, 0 retries
|
|
100.00% of object copies found (7857 of 7857)
|
|
Sample represents 1.00% of the object partition space
|
|
</computeroutput></screen>
|
|
<para>Now, deliberately double the weight of a device in the
|
|
object ring (with replication turned off) and re-run the
|
|
dispersion report to show what impact that has:</para>
|
|
<screen><prompt>$</prompt> <userinput>swift-ring-builder object.builder set_weight d0 200</userinput>
|
|
<prompt>$</prompt> <userinput>swift-ring-builder object.builder rebalance</userinput>
|
|
...
|
|
<prompt>$</prompt> <userinput>swift-dispersion-report</userinput>
|
|
<computeroutput>Queried 2621 containers for dispersion reporting, 8s, 0 retries
|
|
100.00% of container copies found (7863 of 7863)
|
|
Sample represents 1.00% of the container partition space
|
|
|
|
Queried 2619 objects for dispersion reporting, 7s, 0 retries
|
|
There were 1763 partitions missing one copy.
|
|
77.56% of object copies found (6094 of 7857)
|
|
Sample represents 1.00% of the object partition space
|
|
</computeroutput></screen>
|
|
<para>You can see the health of the objects in the cluster has
|
|
gone down significantly. Of course, this test environment
|
|
has just four devices, in a production environment with
|
|
many devices the impact of one device change is much less.
|
|
Next, run the replicators to get everything put back into
|
|
place and then rerun the dispersion report:</para>
|
|
<programlisting>
|
|
... start object replicators and monitor logs until they're caught up ...
|
|
$ swift-dispersion-report
|
|
Queried 2621 containers for dispersion reporting, 17s, 0 retries
|
|
100.00% of container copies found (7863 of 7863)
|
|
Sample represents 1.00% of the container partition space
|
|
|
|
Queried 2619 objects for dispersion reporting, 7s, 0 retries
|
|
100.00% of object copies found (7857 of 7857)
|
|
Sample represents 1.00% of the object partition space
|
|
</programlisting>
|
|
<para>Alternatively, the dispersion report can also be output
|
|
in JSON format. This allows it to be more easily consumed
|
|
by third-party utilities:</para>
|
|
<screen><prompt>$</prompt> <userinput>swift-dispersion-report -j</userinput>
|
|
<computeroutput>{"object": {"retries:": 0, "missing_two": 0, "copies_found": 7863, "missing_one": 0,
|
|
"copies_expected": 7863, "pct_found": 100.0, "overlapping": 0, "missing_all": 0}, "container":
|
|
{"retries:": 0, "missing_two": 0, "copies_found": 12534, "missing_one": 0, "copies_expected":
|
|
12534, "pct_found": 100.0, "overlapping": 15, "missing_all": 0}}</computeroutput></screen>
|
|
<xi:include
|
|
href="../../common/tables/swift-dispersion-dispersion.xml"
|
|
/>
|
|
</section>
|
|
<section xml:id="object-storage-slo">
|
|
<!-- Usage documented in http://docs.openstack.org/developer/swift/overview_large_objects.html -->
|
|
<title>Static Large Object (SLO) support</title>
|
|
<para>This feature is very similar to Dynamic Large Object
|
|
(DLO) support in that it enables the user to upload many
|
|
objects concurrently and afterwards download them as a
|
|
single object. It is different in that it does not rely on
|
|
eventually consistent container listings to do so.
|
|
Instead, a user-defined manifest of the object segments is
|
|
used.</para>
|
|
<xi:include
|
|
href="../../common/tables/swift-proxy-server-filter-slo.xml"
|
|
/>
|
|
</section>
|
|
<section xml:id="object-storage-container-quotas">
|
|
<title>Container quotas</title>
|
|
<para>The <code>container_quotas</code> middleware implements simple quotas that can be
|
|
imposed on Object Storage containers by a user with the ability to set container
|
|
metadata, most likely the account administrator. This can be useful for limiting the
|
|
scope of containers that are delegated to non-admin users, exposed to formpost uploads,
|
|
or just as a self-imposed sanity check.</para>
|
|
<para>Any object PUT operations that exceed these quotas
|
|
return a 403 response (forbidden).</para>
|
|
<para>Quotas are subject to several limitations: eventual
|
|
consistency, the timeliness of the cached container_info
|
|
(60 second TTL by default), and it is unable to reject
|
|
chunked transfer uploads that exceed the quota (though
|
|
once the quota is exceeded, new chunked transfers are
|
|
refused).</para>
|
|
<para>Set quotas by adding meta values to the container. These
|
|
values are validated when you set them:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>X-Container-Meta-Quota-Bytes: Maximum size of
|
|
the container, in bytes.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>X-Container-Meta-Quota-Count: Maximum object
|
|
count of the container.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<xi:include
|
|
href="../../common/tables/swift-proxy-server-filter-container-quotas.xml"
|
|
/>
|
|
</section>
|
|
<section xml:id="object-storage-account-quotas">
|
|
<title>Account quotas</title>
|
|
<para>The <literal>x-account-meta-quota-bytes</literal>
|
|
metadata entry must be requests (PUT, POST) if a given
|
|
account quota (in bytes) is exceeded while DELETE requests
|
|
are still allowed.</para>
|
|
<para>The <literal>x-account-meta-quota-bytes</literal>
|
|
metadata entry must be
|
|
set to store and enable the quota. Write requests to this
|
|
metadata entry are only permitted for resellers. There is
|
|
no account quota limitation on a reseller account even if
|
|
<literal>x-account-meta-quota-bytes</literal> is set.
|
|
</para>
|
|
<para>Any object PUT operations that exceed the quota return a
|
|
413 response (request entity too large) with a descriptive
|
|
body.</para>
|
|
<para>The following command uses an admin account that own the
|
|
Reseller role to set a quota on the test account:</para>
|
|
<screen><prompt>$</prompt> <userinput>swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin \
|
|
--os-storage-url http://127.0.0.1:8080/v1/AUTH_test post -m quota-bytes:10000</userinput></screen>
|
|
<para>Here is the stat listing of an account where quota has
|
|
been set:</para>
|
|
<screen><prompt>$</prompt> <userinput>swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat</userinput>
|
|
<computeroutput>Account: AUTH_test
|
|
Containers: 0
|
|
Objects: 0
|
|
Bytes: 0
|
|
Meta Quota-Bytes: 10000
|
|
X-Timestamp: 1374075958.37454
|
|
X-Trans-Id: tx602634cf478546a39b1be-0051e6bc7a</computeroutput></screen>
|
|
<para>This command removes the account quota:</para>
|
|
<screen><prompt>$</prompt> <userinput>swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin --os-storage-url http://127.0.0.1:8080/v1/AUTH_test post -m quota-bytes:</userinput></screen>
|
|
</section>
|
|
<section xml:id="object-storage-bulk-delete">
|
|
<title>Bulk delete</title>
|
|
<para>Use <code>bulk-delete</code> to delete multiple files
|
|
from an account
|
|
with a single request. Responds to DELETE requests with a
|
|
header 'X-Bulk-Delete: true_value'. The body of the DELETE
|
|
request is a new line-separated list of files to delete.
|
|
The files listed must be URL encoded and in the
|
|
form:</para>
|
|
<programlisting>
|
|
/container_name/obj_name
|
|
</programlisting>
|
|
<para>If all files are successfully deleted (or did not
|
|
exist), the operation returns <code>HTTPOk</code>. If any
|
|
files failed to delete, the operation returns
|
|
<code>HTTPBadGateway</code>. In both cases, the response body
|
|
is a JSON dictionary that shows the number of files that were
|
|
successfully deleted or not found. The files that failed are
|
|
listed.</para>
|
|
<xi:include
|
|
href="../../common/tables/swift-proxy-server-filter-bulk.xml"
|
|
/>
|
|
</section>
|
|
<xi:include href="section_configure_s3.xml"/>
|
|
<section xml:id="object-storage-drive-audit">
|
|
<title>Drive audit</title>
|
|
<para>The <option>swift-drive-audit</option> configuration
|
|
items reference a script that can be run by using
|
|
<command>cron</command> to watch for bad drives. If
|
|
errors are detected, it unmounts the bad drive, so that
|
|
OpenStack Object Storage can work around it. It takes the
|
|
following options:</para>
|
|
<xi:include
|
|
href="../../common/tables/swift-drive-audit-drive-audit.xml"
|
|
/>
|
|
</section>
|
|
<section xml:id="object-storage-form-post">
|
|
<title>Form post</title>
|
|
<para>Middleware that provides the ability to upload objects
|
|
to a cluster using an HTML form POST. The format of the
|
|
form is:</para>
|
|
<programlisting><![CDATA[
|
|
<form action="<swift-url>" method="POST"
|
|
enctype="multipart/form-data">
|
|
<input type="hidden" name="redirect" value="<redirect-url>" />
|
|
<input type="hidden" name="max_file_size" value="<bytes>" />
|
|
<input type="hidden" name="max_file_count" value="<count>" />
|
|
<input type="hidden" name="expires" value="<unix-timestamp>" />
|
|
<input type="hidden" name="signature" value="<hmac>" />
|
|
<input type="file" name="file1" /><br />
|
|
<input type="submit" />
|
|
</form>]]>
|
|
</programlisting>
|
|
<para>The <literal>swift-url</literal> is the URL to the Object Storage destination, such
|
|
as: <uri>https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix</uri>
|
|
The name of each file uploaded is appended to the specified
|
|
<literal>swift-url</literal>. So, you can upload directly to the root of container with
|
|
a URL like: <uri>https://swift-cluster.example.com/v1/AUTH_account/container/</uri>
|
|
Optionally, you can include an object prefix to better separate different users'
|
|
uploads, such as:
|
|
<uri>https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix</uri>
|
|
</para>
|
|
<note>
|
|
<para>The form method must be POST and the enctype must be
|
|
set as <literal>multipart/form-data</literal>.</para>
|
|
</note>
|
|
<para>The redirect attribute is the URL to redirect the
|
|
browser to after the upload completes. The URL has status
|
|
and message query parameters added to it, indicating the
|
|
HTTP status code for the upload (2xx is success) and a
|
|
possible message for further information if there was an
|
|
error (such as <literal>"max_file_size
|
|
exceeded"</literal>).</para>
|
|
<para>The <literal>max_file_size</literal> attribute must be
|
|
included and indicates the largest single file upload that
|
|
can be done, in bytes.</para>
|
|
<para>The <literal>max_file_count</literal> attribute must be
|
|
included and indicates the maximum number of files that
|
|
can be uploaded with the form. Include additional
|
|
<code><![CDATA[<input type="file"
|
|
name="filexx"/>]]></code> attributes if
|
|
desired.</para>
|
|
<para>The expires attribute is the Unix timestamp before which
|
|
the form must be submitted before it is
|
|
invalidated.</para>
|
|
<para>The signature attribute is the HMAC-SHA1 signature of
|
|
the form. This sample Python code shows how to compute the
|
|
signature:</para>
|
|
<programlisting language="python">
|
|
import hmac
|
|
from hashlib import sha1
|
|
from time import time
|
|
path = '/v1/account/container/object_prefix'
|
|
redirect = 'https://myserver.com/some-page'
|
|
max_file_size = 104857600
|
|
max_file_count = 10
|
|
expires = int(time() + 600)
|
|
key = 'mykey'
|
|
hmac_body = '%s\n%s\n%s\n%s\n%s' % (path, redirect,
|
|
max_file_size, max_file_count, expires)
|
|
signature = hmac.new(key, hmac_body, sha1).hexdigest()
|
|
</programlisting>
|
|
<para>The key is the value of the
|
|
<literal>X-Account-Meta-Temp-URL-Key</literal> header
|
|
on the account.</para>
|
|
<para>Be certain to use the full path, from the
|
|
<literal>/v1/</literal> onward.</para>
|
|
<para>The command-line tool
|
|
<command>swift-form-signature</command> may be used
|
|
(mostly just when testing) to compute expires and
|
|
signature.</para>
|
|
<para>The file attributes must appear after the other
|
|
attributes to be processed correctly. If attributes come
|
|
after the file, they are not sent with the sub-request
|
|
because on the server side, all attributes in the file
|
|
cannot be parsed unless the whole file is read into memory
|
|
and the server does not have enough memory to service
|
|
these requests. So, attributes that follow the file are
|
|
ignored.</para>
|
|
<xi:include
|
|
href="../../common/tables/swift-proxy-server-filter-formpost.xml"
|
|
/>
|
|
</section>
|
|
<section xml:id="object-storage-static-web">
|
|
<title>Static web sites</title>
|
|
<para>When configured, this middleware serves container data
|
|
as a static web site with index file and error file
|
|
resolution and optional file listings. This mode is
|
|
normally only active for anonymous requests.</para>
|
|
<xi:include
|
|
href="../../common/tables/swift-proxy-server-filter-staticweb.xml"
|
|
/>
|
|
</section>
|
|
<xi:include href="section_object-storage-cors.xml"/>
|
|
<xi:include href="section_object-storage-listendpoints.xml"/>
|
|
</section>
|