openstack/openstack-manuals
Code Issues Proposed changes

Add X-Trans-Id-Extra header information to large objects section

Browse Source 
- Remove passive voice
- Use present tense
- Improve grammar

Change-Id: I425e8e12911f0d7d703ffd6659310896e4481650
Closes-Bug: #1320386
This commit is contained in:
Diane Fleming 2016-01-03 21:38:05 -06:00 committed by Olena Logvinova
parent 7ca56587e4
commit 995019ba8b
1 changed files with 104 additions and 79 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

183
doc/user-guide/source/cli_swift_large_object_creation.rst
View File

@ -4,30 +4,28 @@
Large objects Large objects
============= =============
To discover whether your Object Storage system supports this feature, To discover whether your Object Storage system supports this feature, see
see :ref:`discoverability`. Alternatively, check with your service :ref:`discoverability` or check with your service provider.
provider.
By default, the content of an object cannot be greater than 5 GB. By default, the content of an object cannot be greater than 5 GB.
However, you can use a number of smaller objects to construct a large However, you can use a number of smaller objects to construct a large
object. The large object is comprised of two types of objects: object. The large object is comprised of two types of objects:
* ``Segment objects`` store the object content. You can divide your * ``Segment objects`` store the object content. You can divide your content
content into segments, and upload each segment into its own segment into segments and upload each segment into its own segment object. Segment
object. Segment objects do not have any special features. You create, objects do not have any special features. You create, update, download, and
update, download, and delete segment objects just as you would normal delete segment objects just as you do with normal objects.
objects.
* A ``manifest object`` links the segment objects into one logical * A ``manifest object`` links the segment objects into one logical large
large object. When you download a manifest object, Object Storage object. When you download a manifest object, Object Storage concatenates and
concatenates and returns the contents of the segment objects in the returns the contents of the segment objects in the response body. This
response body of the request. This behavior extends to the response behavior extends to the response headers returned by ``GET`` and ``HEAD``
headers returned by ``GET`` and ``HEAD`` requests. The requests. The ``Content-Length`` response header contains the total size of
``Content-Length`` response header value is the total size of all all segment objects.
segment objects. Object Storage calculates the ``ETag`` response
header value by taking the ``ETag`` value of each segment, Object Storage takes the ``ETag`` value of each segment, concatenates them
concatenating them together, and returning the MD5 checksum of the together, and returns the MD5 checksum of the result to calculate the
result. The manifest object types are: ``ETag`` response header value. The manifest object types are:
Static large objects Static large objects
The manifest object content is an ordered list of the names of The manifest object content is an ordered list of the names of
@ -43,24 +41,24 @@ object. The large object is comprised of two types of objects:
.. note:: .. note::
If you make a ``COPY`` request by using a manifest object as the source, If you use a manifest object as the source of a ``COPY`` request, the
the new object is a normal, and not a segment, object. If the total size new object is a normal, and not a segment, object. If the total size of the
of the source segment objects exceeds 5 GB, the ``COPY`` request fails. source segment objects exceeds 5 GB, the ``COPY`` request fails. However,
However, you can make a duplicate of the manifest object and this new you can make a duplicate of the manifest object and this new object can be
object can be larger than 5 GB. larger than 5 GB.
.. _static_large_objects: .. _static_large_objects:
Static large objects Static large objects
~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
To create a static large object, divide your content into pieces and To create a static large object, divide your content into pieces and create
create (upload) a segment object to contain each piece. (upload) a segment object to contain each piece.
You must record the ``ETag`` response header that the ``PUT`` operation You must record the ``ETag`` response header value that the ``PUT`` operation
returns. Alternatively, you can calculate the MD5 checksum of the returns. Alternatively, you can calculate the MD5 checksum of the segment
segment prior to uploading and include this in the ``ETag`` request before you perform the upload and include this value in the ``ETag`` request
header. This ensures that the upload cannot corrupt your data. header. This action ensures that the upload cannot corrupt your data.
List the name of each segment object along with its size and MD5 List the name of each segment object along with its size and MD5
checksum in order. checksum in order.
@ -70,7 +68,7 @@ query string at the end of the manifest object name to indicate that
this is a manifest object. this is a manifest object.
The body of the ``PUT`` request on the manifest object comprises a JSON The body of the ``PUT`` request on the manifest object comprises a JSON
list, where each element contains the following attributes: list where each element contains these attributes:
path path
The container and object name in the format: The container and object name in the format:
@ -114,7 +112,7 @@ contrast to dynamic large objects.
| |
The ``Content-Length`` request header must contain the length of the The ``Content-Length`` request header must contain the length of the
JSON content—not the length of the segment objects. However, after the JSON content and not the length of the segment objects. However, after the
``PUT`` operation completes, the ``Content-Length`` metadata is set to ``PUT`` operation completes, the ``Content-Length`` metadata is set to
the total length of all the object segments. A similar situation applies the total length of all the object segments. A similar situation applies
to the ``ETag``. If used in the ``PUT`` operation, it must contain the to the ``ETag``. If used in the ``PUT`` operation, it must contain the
@ -128,16 +126,15 @@ parameter, it reads the request body and verifies that each segment
object exists and that the sizes and ETags match. If there is a object exists and that the sizes and ETags match. If there is a
mismatch, the ``PUT`` operation fails. mismatch, the ``PUT`` operation fails.
If everything matches, the manifest object is created. The If everything matches, the API creates the manifest object and sets the
``X-Static-Large-Object`` metadata is set to ``true`` indicating that ``X-Static-Large-Object`` metadata to ``true`` to indicate that the manifest is
this is a static object manifest. a static object manifest.
Normally when you perform a ``GET`` operation on the manifest object, Normally when you perform a ``GET`` operation on the manifest object, the
the response body contains the concatenated content of the segment response body contains the concatenated content of the segment objects. To
objects. To download the manifest list, use the download the manifest list, use the ``?multipart-manifest=get`` query
``?multipart-manifest=get`` query parameter. The resulting list is not parameter. The list in the response is not formatted the same as the manifest
formatted the same as the manifest you originally used in the ``PUT`` that you originally used in the ``PUT`` operation.
operation.
If you use the ``DELETE`` operation on a manifest object, the manifest If you use the ``DELETE`` operation on a manifest object, the manifest
object is deleted. The segment objects are not affected. However, if you object is deleted. The segment objects are not affected. However, if you
@ -155,24 +152,23 @@ way.
Dynamic large objects Dynamic large objects
~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~
You must segment objects that are larger than 5 GB before you can upload Before you can upload objects that are larger than 5 GB, you must segment
them. You then upload the segment objects like you would any other them. You upload the segment objects like you do with any other object and
object and create a dynamic large manifest object. The manifest object create a dynamic large manifest object. The manifest object tells Object
tells Object Storage how to find the segment objects that comprise the Storage how to find the segment objects that comprise the large object. You
large object. The segments remain individually addressable, but can still access each segment individually, but when you retrieve the manifest
retrieving the manifest object streams all the segments concatenated. object, the API concatenates the segments. You can include any number of
There is no limit to the number of segments that can be a part of a segments in a single large object.
single large object.
To ensure the download works correctly, you must upload all the object To ensure the download works correctly, you must upload all the object
segments to the same container and ensure that each object name is segments to the same container and prefix each object name so that the
prefixed in such a way that it sorts in the order in which it should be segments sort in correct concatenation order.
concatenated. You also create and upload a manifest file. The manifest
file is a zero-byte file with the extra ``X-Object-Manifest`` You also create and upload a manifest file. The manifest file is a zero-byte
``CONTAINER/PREFIX`` header, where ``CONTAINER`` is the container file with the extra ``X-Object-Manifest`` ``CONTAINER/PREFIX`` header. The
the object segments are in and ``PREFIX`` is the common prefix for all ``CONTAINER`` is the container the object segments are in and ``PREFIX`` is
the segments. You must UTF-8-encode and then URL-encode the container the common prefix for all the segments. You must UTF-8-encode and then
and common prefix in the ``X-Object-Manifest`` header. URL-encode the container and common prefix in the ``X-Object-Manifest`` header.
It is best to upload all the segments first and then create or update It is best to upload all the segments first and then create or update
the manifest. With this method, the full object is not available for the manifest. With this method, the full object is not available for
@ -193,15 +189,20 @@ Upload segment of large object request: HTTP
Content-Length: 1 Content-Length: 1
X-Object-Meta-PIN: 1234 X-Object-Meta-PIN: 1234
No response body is returned. A status code of 2``nn`` (between 200 No response body is returned.
and 299, inclusive) indicates a successful write; status 411 Length
Required denotes a missing ``Content-Length`` or ``Content-Type`` header
in the request. If the MD5 checksum of the data written to the storage
system does NOT match the (optionally) supplied ETag value, a 422
Unprocessable Entity response is returned.
You can continue uploading segments, like this example shows, prior to The 2``nn`` response code indicates a successful write. ``nn`` is a value from
uploading the manifest. 00 to 99.
The ``Length Required (411)`` response code indicates that the request does
not include a required ``Content-Length`` or ``Content-Type`` header.
The ``Unprocessable Entity (422)`` response code indicates that the MD5
checksum of the data written to the storage system does NOT match the optional
ETag value.
You can continue to upload segments, like this example shows, before you
upload the manifest.
Upload next segment of large object request: HTTP Upload next segment of large object request: HTTP
------------------------------------------------- -------------------------------------------------
@ -215,11 +216,10 @@ Upload next segment of large object request: HTTP
Content-Length: 1 Content-Length: 1
X-Object-Meta-PIN: 1234 X-Object-Meta-PIN: 1234
Next, upload the manifest you created that indicates the container where Next, upload the manifest. This manifest specifies the container where the
the object segments reside. Note that uploading additional segments object segments reside. Note that if you upload additional segments after you
after the manifest is created causes the concatenated object to be that create the manifest, the concatenated object becomes that much larger but you
much larger but you do not need to recreate the manifest file for do not need to recreate the manifest file for subsequent additional segments.
subsequent additional segments.
Upload manifest request: HTTP Upload manifest request: HTTP
----------------------------- -----------------------------
@ -239,20 +239,45 @@ Upload manifest response: HTTP
[...] [...]
The ``Content-Type`` in the response for a ``GET`` or ``HEAD`` on the A ``GET`` or ``HEAD`` request on the manifest returns a ``Content-Type``
manifest is the same as the ``Content-Type`` set during the ``PUT`` response header value that is the same as the ``Content-Type`` request header
request that created the manifest. You can change the ``Content-Type`` value in the ``PUT`` request that created the manifest. To change the
by reissuing the ``PUT`` request. ``Content- Type``, reissue the ``PUT`` request.
Extra transaction information
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can use the ``X-Trans-Id-Extra`` request header to include extra
information to help you debug any errors that might occur with large object
upload and other Object Storage transactions.
The Object Storage API appends the first 32 characters of the
``X-Trans-Id-Extra`` request header value to the transaction ID value in the
generated ``X-Trans-Id`` response header. You must UTF-8-encode and then
URL-encode the extra transaction information before you include it in
the ``X-Trans-Id-Extra`` request header.
For example, you can include extra transaction information when you upload
large objects such as images.
When you upload each segment and the manifest, include the same value in the
``X-Trans-Id-Extra`` request header. If an error occurs, you can find all
requests that are related to the large object upload in the Object Storage
logs.
You can also use ``X-Trans-Id-Extra`` strings to help operators debug requests
that fail to receive responses. The operator can search for the extra
information in the logs.
Comparison of static and dynamic large objects Comparison of static and dynamic large objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
While static and dynamic objects have similar behavior, this table While static and dynamic objects have similar behavior, this table describes
describes their differences: their differences:
.. list-table:: .. list-table::
:header-rows: 1 :header-rows: 1
:widths: 30 10 30 :widths: 20 25 25
:stub-columns: 1 :stub-columns: 1
* - Description * - Description
@ -285,7 +310,7 @@ describes their differences:
The names must simply match the ``PREFIX`` supplied in The names must simply match the ``PREFIX`` supplied in
``X-Object-Manifest``. ``X-Object-Manifest``.
* - Segment object size and number * - Segment object size and number
- Segment objects must be at least 1 MB in size (by default). The - Segment objects must be at least 1 MB in size (by default). The
final segment object can be any size. At most, 1000 segments final segment object can be any size. At most, 1000 segments
are supported (by default). are supported (by default).
- Segment objects can be any size. - Segment objects can be any size.
@ -293,7 +318,7 @@ describes their differences:
- The manifest list includes the container name of each object. - The manifest list includes the container name of each object.
Segment objects can be in different containers. Segment objects can be in different containers.
- All segment objects must be in the same container. - All segment objects must be in the same container.
* - Manfiest object metadata * - Manifest object metadata
- The object has ``X-Static-Large-Object`` set to ``true``. You - The object has ``X-Static-Large-Object`` set to ``true``. You
do not set this metadata directly. Instead the system sets it do not set this metadata directly. Instead the system sets it
when you ``PUT`` a static manifest object. when you ``PUT`` a static manifest object.