diff --git a/doc/config-reference/object-storage/section_object-storage-features.xml b/doc/config-reference/object-storage/section_object-storage-features.xml index 0b83bd09e4..9b6e9593a0 100644 --- a/doc/config-reference/object-storage/section_object-storage-features.xml +++ b/doc/config-reference/object-storage/section_object-storage-features.xml @@ -1,4 +1,8 @@ + +%openstack; +]>
- OpenStack Object Storage works best on the XFS file - system, and this document assumes that the hardware being - used is configured appropriately to be mounted with the - nobarriers option. For more - information, refer to the XFS FAQ: http://xfs.org/index.php/XFS_FAQ + + OpenStack Object Storage works best on the XFS file system, and + this document assumes that the hardware being used is configured + appropriately to be mounted with the nobarriers + option. For more information, see the XFS FAQ. To get the most out of your hardware, it is essential that every disk used in OpenStack Object Storage is @@ -217,11 +220,14 @@ temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709& temp_url_expires=1323479485 - To create temporary URLs, first set the X-Account-Meta-Temp-URL-Key - header on your Object Storage account to an arbitrary string. This string serves as a - secret key. For example, to set a key of - b3968d0207b54ece87cccc06515a89d4 using the - swift command-line tool: + + To create temporary URLs, first set the + X-Account-Meta-Temp-URL-Key header on your + Object Storage account to an arbitrary string. This string serves + as a secret key. For example, to set a key of + b3968d0207b54ece87cccc06515a89d4 by using the + swift command-line tool: + $ swift post -m "Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4" Next, generate an HMAC-SHA1 (RFC 2104) signature to specify: @@ -257,31 +263,32 @@ hmac_body = '%s\n%s\n%s' % (method, expires, path) sig = hmac.new(key, hmac_body, sha1).hexdigest() s = 'https://{host}/{path}?temp_url_sig={sig}&temp_url_expires={expires}' url = s.format(host='swift-cluster.example.com', path=path, sig=sig, expires=expires) - Any alteration of the resource path or query arguments - results in a 401 - Unauthorized error. Similarly, a - PUT where GET was the allowed method returns a - 401. HEAD is allowed if GET or - PUT is allowed. Using this in combination with browser - form post translation middleware could also allow - direct-from-browser uploads to specific locations in - Object Storage. + + Any alteration of the resource path or query arguments results in + a 401 Unauthorized + error. Similarly, a PUT where GET was the allowed method returns a + 401 error. HEAD is allowed if GET or PUT is + allowed. Using this in combination with browser form post + translation middleware could also allow direct-from-browser + uploads to specific locations in Object Storage. + - Changing the - X-Account-Meta-Temp-URL-Key - invalidates any previously generated temporary - URLs within 60 seconds (the memcache time for the - key). Object Storage supports up to two keys, specified by - X-Account-Meta-Temp-URL-Key - and - X-Account-Meta-Temp-URL-Key-2. - Signatures are checked against both keys, if - present. This is to allow for key rotation without - invalidating all existing temporary URLs. + + Changing the X-Account-Meta-Temp-URL-Key + invalidates any previously generated + temporary URLs within 60 seconds, which is the memcache time for + the key. Object Storage supports up to two keys, + specified by X-Account-Meta-Temp-URL-Key + and X-Account-Meta-Temp-URL-Key-2. + Signatures are checked against both keys, + if present. This process enables key rotation without + invalidating all existing temporary URLs. + - Object Storage includes a script called - swift-temp-url that generates the - query parameters automatically: + + Object Storage includes the swift-temp-url + script that generates the query parameters automatically: + $ bin/swift-temp-url GET 3600 /v1/AUTH_account/container/object mykey /v1/AUTH_account/container/object? temp_url_sig=5c4cc8886f36a9d0919d708ade98bf0cc71c9e91& @@ -350,58 +357,62 @@ pipeline = pipeline = healthcheck cache tempurl
Cluster health - Use the swift-dispersion-report tool - to measure overall cluster health. This tool checks if a - set of deliberately distributed containers and objects are - currently in their proper places within the cluster. For - instance, a common deployment has three replicas of each - object. The health of that object can be measured by - checking if each replica is in its proper place. If only 2 - of the 3 is in place the object's health can be said to be - at 66.66%, where 100% would be perfect. A single object's - health, especially an older object, usually reflects the - health of that entire partition the object is in. If you - make enough objects on a distinct percentage of the - partitions in the cluster,you get a good estimate of the - overall cluster health. In practice, about 1% partition - coverage seems to balance well between accuracy and the - amount of time it takes to gather results. The first thing - that needs to be done to provide this health value is - create a new account solely for this usage. Next, you need - to place the containers and objects throughout the system - so that they are on distinct partitions. The - swift-dispersion-populate tool does this - by making up - random container and object names until they fall on - distinct partitions. Last, and repeatedly for the life of - the cluster, you must run the - swift-dispersion-report tool to - check the health of each of these containers and objects. - These tools need direct access to the entire cluster and - to the ring files (installing them on a proxy server - suffices). The - swift-dispersion-populate and - swift-dispersion-report commands - both use the same configuration file, - /etc/swift/dispersion.conf. - Example dispersion.conf file: + + Use the swift-dispersion-report tool to measure + overall cluster health. This tool checks if a set of deliberately + distributed containers and objects are currently in their proper + places within the cluster. For instance, a common deployment has + three replicas of each object. The health of that object can be + measured by checking if each replica is in its proper place. If + only 2 of the 3 is in place the object's health can be said to be + at 66.66%, where 100% would be perfect. A single object's health, + especially an older object, usually reflects the health of that + entire partition the object is in. If you make enough objects on a + distinct percentage of the partitions in the cluster,you get a + good estimate of the overall cluster health. + + + In practice, about 1% partition coverage seems to balance well + between accuracy and the amount of time it takes to gather + results. To provide this health value, you must create an account + solely for this usage. Next, you must place the containers and + objects throughout the system so that they are on distinct + partitions. Use the swift-dispersion-populate + tool to create random container and object names until they fall + on distinct partitions. + + + Last, and repeatedly for the life of the cluster, you must run the + swift-dispersion-report tool to check the + health of each container and object. + + + These tools must have direct access to the entire cluster and ring + files. Installing them on a proxy server suffices. + + + The swift-dispersion-populate and + swift-dispersion-report commands both use the + same /etc/swift/dispersion.conf configuration + file. Example dispersion.conf file: + [dispersion] auth_url = http://localhost:8080/auth/v1.0 auth_user = test:tester auth_key = testing - There are also configuration options for specifying the - dispersion coverage, which defaults to 1%, retries, - concurrency, and so on. However, the defaults are usually - fine. Once the configuration is in place, run - swift-dispersion-populate to - populate the containers and objects throughout the - cluster. Now that those containers and objects are in - place, you can run - swift-dispersion-report to get a - dispersion report, or the overall health of the cluster. - Here is an example of a cluster in perfect health: + + You can use configuration options to specify the dispersion + coverage, which defaults to 1%, retries, concurrency, and so on. + However, the defaults are usually fine. After the configuration is + in place, run the swift-dispersion-populate + tool to populate the containers and objects throughout the + cluster. Now that those containers and objects are in place, you + can run the swift-dispersion-report tool to get + a dispersion report or view the overall health of the cluster. + Here is an example of a cluster in perfect health: + $ swift-dispersion-report Queried 2621 containers for dispersion reporting, 19s, 0 retries 100.00% of container copies found (7863 of 7863) @@ -427,12 +438,14 @@ There were 1763 partitions missing one copy. 77.56% of object copies found (6094 of 7857) Sample represents 1.00% of the object partition space - You can see the health of the objects in the cluster has - gone down significantly. Of course, this test environment - has just four devices, in a production environment with - many devices the impact of one device change is much less. - Next, run the replicators to get everything put back into - place and then rerun the dispersion report: + + You can see the health of the objects in the cluster has gone down + significantly. Of course, this test environment has just four + devices, in a production environment with many devices the impact + of one device change is much less. Next, run the replicators to + get everything put back into place and then rerun the dispersion + report: + ... start object replicators and monitor logs until they're caught up ... $ swift-dispersion-report @@ -472,13 +485,16 @@ Sample represents 1.00% of the object partition space
Container quotas - The container_quotas middleware implements simple quotas that can be - imposed on Object Storage containers by a user with the ability to set container - metadata, most likely the account administrator. This can be useful for limiting the - scope of containers that are delegated to non-admin users, exposed to formpost uploads, - or just as a self-imposed sanity check. - Any object PUT operations that exceed these quotas - return a 403 response (forbidden). + + The container_quotas middleware implements simple + quotas that can be imposed on Object Storage containers by a user + with the ability to set container metadata, most likely the + account administrator. This can be useful for limiting the scope + of containers that are delegated to non-admin users, exposed to + form &POST; uploads, or just as a self-imposed sanity check. + + Any object &PUT; operations that exceed these quotas + return a Forbidden (403) status code. Quotas are subject to several limitations: eventual consistency, the timeliness of the cached container_info (60 second TTL by default), and it is unable to reject @@ -560,21 +576,26 @@ X-Trans-Id: tx602634cf478546a39b1be-0051e6bc7a
Drive audit - The configuration - items reference a script that can be run by using - cron to watch for bad drives. If - errors are detected, it unmounts the bad drive, so that - OpenStack Object Storage can work around it. It takes the - following options: + + The configuration items + reference a script that can be run by using + cron to watch for bad drives. If errors are + detected, it unmounts the bad drive so that OpenStack Object + Storage can work around it. It takes the following options: +
Form post - Middleware that provides the ability to upload objects - to a cluster using an HTML form POST. The format of the - form is: + + Middleware that enables you to upload objects to a cluster by + using an HTML form &POST;. + + + The format of the form is: + <![CDATA[ <form action="<swift-url>" method="POST" enctype="multipart/form-data"> @@ -583,47 +604,110 @@ X-Trans-Id: tx602634cf478546a39b1be-0051e6bc7a <input type="hidden" name="max_file_count" value="<count>" /> <input type="hidden" name="expires" value="<unix-timestamp>" /> <input type="hidden" name="signature" value="<hmac>" /> + <input type="hidden" name="x_delete_at" value="<unix-timestamp>"/> + <input type="hidden" name="x_delete_after" value="<seconds>"/> <input type="file" name="file1" /><br /> <input type="submit" /> </form>]]> - The swift-url is the URL to the Object Storage destination, such - as: https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix - The name of each file uploaded is appended to the specified - swift-url. So, you can upload directly to the root of container with - a URL like: https://swift-cluster.example.com/v1/AUTH_account/container/ - Optionally, you can include an object prefix to better separate different users' - uploads, such as: - https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix + + In the form: - - The form method must be POST and the enctype must be - set as multipart/form-data. - - The redirect attribute is the URL to redirect the - browser to after the upload completes. The URL has status - and message query parameters added to it, indicating the - HTTP status code for the upload (2xx is success) and a - possible message for further information if there was an - error (such as "max_file_size - exceeded"). - The max_file_size attribute must be - included and indicates the largest single file upload that - can be done, in bytes. - The max_file_count attribute must be - included and indicates the maximum number of files that - can be uploaded with the form. Include additional - <![CDATA[<input type="file" - name="filexx"/>]]> attributes if - desired. - The expires attribute is the Unix timestamp before which - the form must be submitted before it is - invalidated. - The signature attribute is the HMAC-SHA1 signature of - the form. This sample Python code shows how to compute the - signature: - -import hmac + + + + action="<swift-url>" + + + The URL to the Object Storage destination, such as + https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix. + + + The name of each uploaded file is appended to the specified + swift-url. So, you can upload directly to the root of + container with a URL like + https://swift-cluster.example.com/v1/AUTH_account/container/. + + + Optionally, you can include an object prefix to + separate different users' uploads, such as + https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix. + + + + + method="POST" + + The form method must be &POST;. + + + + + enctype="multipart/form-data + + The enctype must be set to multipart/form-data. + + + + + name="redirect" + + The URL to which to redirect the browser after the upload + completes. The URL has status and message query parameters + added to it that indicate the HTTP status code for the upload + and, optionally, additional error information. The 2nn status code indicates success. If + an error occurs, the URL might include error information, such + as "max_file_size exceeded". + + + + + name="max_file_size" + + Required. The maximum number of bytes that can be uploaded in + a single file upload. + + + + + name="max_file_count" + + Required. The maximum number of files that can be uploaded + with the form. + + + + + name="expires" + + + The expiration date and time for the form in UNIX + Epoch time stamp format. After this date and time, the + form is no longer valid. + + + For example, 1440619048 is equivalent to + Mon, Wed, 26 Aug 2015 19:57:28 GMT. + + + + + name="signature" + + + The HMAC-SHA1 signature of the form. This sample Python code + shows how to compute the signature: + + import hmac from hashlib import sha1 from time import time path = '/v1/account/container/object_prefix' @@ -634,28 +718,70 @@ expires = int(time() + 600) key = 'mykey' hmac_body = '%s\n%s\n%s\n%s\n%s' % (path, redirect, max_file_size, max_file_count, expires) -signature = hmac.new(key, hmac_body, sha1).hexdigest() - +signature = hmac.new(key, hmac_body, sha1).hexdigest() The key is the value of the X-Account-Meta-Temp-URL-Key header on the account. - Be certain to use the full path, from the - /v1/ onward. - The command-line tool - swift-form-signature may be used - (mostly just when testing) to compute expires and - signature. - The file attributes must appear after the other - attributes to be processed correctly. If attributes come - after the file, they are not sent with the sub-request - because on the server side, all attributes in the file - cannot be parsed unless the whole file is read into memory - and the server does not have enough memory to service - these requests. So, attributes that follow the file are - ignored. - + + Use the full path from the /v1/ value and + onward. + + + During testing, you can use the + swift-form-signature command-line tool to compute the + expires and signature + values. + + + + + name="x_delete_at" + + + The date and time in UNIX Epoch + time stamp format when the object will be removed. + + + For example, 1440619048 is equivalent to + Mon, Wed, 26 Aug 2015 19:57:28 GMT. + + + This attribute enables you to specify the X-Delete- + At header value in the form &POST;. + + + + + name="x_delete_after" + + + The number of seconds after which the object is removed. + Internally, the Object Storage system stores this value in the + X-Delete-At metadata item. This attribute + enables you to specify the X-Delete-After + header value in the form &POST;. + + + + + type="file" name="filexx" + + + Optional. One or more files to upload. Must appear after the other + attributes to be processed correctly. If attributes come after the + file attribute, they are not sent with the sub- + request because on the server side, all attributes in the file + cannot be parsed unless the whole file is read into memory and the + server does not have enough memory to service these requests. So, + attributes that follow the file attribute are + ignored. + + + +
Static web sites