From 86d78f6667392b2ee0ca017da8f54ff153aca491 Mon Sep 17 00:00:00 2001 From: John Dickinson Date: Wed, 1 Aug 2018 15:17:42 -0700 Subject: [PATCH] imported some docs from the old user-guide These files are imported (and very lightly edited) from the old ocata user-guide. It has a few other swift-related docs that seemed more duplacative of what we already have, but these seem to fill existing gaps in our docs. Change-Id: Ib00bf6992327f15f271120dc5dbc86a4a235baec --- doc/source/api/bulk-delete.rst | 93 +++++++++++ doc/source/api/object-expiration.rst | 48 ++++++ doc/source/api/pagination.rst | 103 ++++++++++++ ...seudo-hierarchical-folders-directories.rst | 155 ++++++++++++++++++ .../api/serialized-response-formats.rst | 119 ++++++++++++++ doc/source/api/static-website.rst | 120 ++++++++++++++ doc/source/index.rst | 6 + 7 files changed, 644 insertions(+) create mode 100644 doc/source/api/bulk-delete.rst create mode 100644 doc/source/api/object-expiration.rst create mode 100644 doc/source/api/pagination.rst create mode 100644 doc/source/api/pseudo-hierarchical-folders-directories.rst create mode 100644 doc/source/api/serialized-response-formats.rst create mode 100644 doc/source/api/static-website.rst diff --git a/doc/source/api/bulk-delete.rst b/doc/source/api/bulk-delete.rst new file mode 100644 index 0000000000..367eed3aa8 --- /dev/null +++ b/doc/source/api/bulk-delete.rst @@ -0,0 +1,93 @@ +.. _bulk-delete: + +=========== +Bulk delete +=========== + +To discover whether your Object Storage system supports this feature, +see :ref:`discoverability`. Alternatively, check with your service provider. + +With bulk delete, you can delete up to 10,000 objects or containers +(configurable) in one request. + +Bulk delete request +~~~~~~~~~~~~~~~~~~~ + +To perform a bulk delete operation, add the ``bulk-delete`` query +parameter to the path of a ``POST`` or ``DELETE`` operation. + +.. note:: + + The ``DELETE`` operation is supported for backwards compatibility. + +The path is the account, such as ``/v1/12345678912345``, that contains +the objects and containers. + +In the request body of the ``POST`` or ``DELETE`` operation, list the +objects or containers to be deleted. Separate each name with a newline +character. You can include a maximum of 10,000 items (configurable) in +the list. + +In addition, you must: + +- UTF-8-encode and then URL-encode the names. + +- To indicate an object, specify the container and object name as: + ``CONTAINER_NAME``/``OBJECT_NAME``. + +- To indicate a container, specify the container name as: + ``CONTAINER_NAME``. Make sure that the container is empty. If it + contains objects, Object Storage cannot delete the container. + +- Set the ``Content-Type`` request header to ``text/plain``. + +Bulk delete response +~~~~~~~~~~~~~~~~~~~~ + +When Object Storage processes the request, it performs multiple +sub-operations. Even if all sub-operations fail, the operation returns a +200 status. The bulk operation returns a response body that contains +details that indicate which sub-operations have succeeded and failed. +Some sub-operations might succeed while others fail. Examine the +response body to determine the results of each delete sub-operation. + +You can set the ``Accept`` request header to one of the following values +to define the response format: + +``text/plain`` + Formats response as plain text. If you omit the + ``Accept`` header, ``text/plain`` is the default. + +``application/json`` + Formats response as JSON. + +``application/xml`` or ``text/xml`` + Formats response as XML. + +The response body contains the following information: + +- The number of files actually deleted. + +- The number of not found objects. + +- Errors. A list of object names and associated error statuses for the + objects that failed to delete. The format depends on the value that + you set in the ``Accept`` header. + +The following bulk delete response is in ``application/xml`` format. In +this example, the ``mycontainer`` container is not empty, so it cannot +be deleted. + +.. code-block:: xml + + + 2 + 4 + + + /v1/12345678912345/mycontainer + 409 Conflict + + + + diff --git a/doc/source/api/object-expiration.rst b/doc/source/api/object-expiration.rst new file mode 100644 index 0000000000..e101b4de5e --- /dev/null +++ b/doc/source/api/object-expiration.rst @@ -0,0 +1,48 @@ +================= +Object expiration +================= + +You can schedule Object Storage (swift) objects to expire by setting the +``X-Delete-At`` or ``X-Delete-After`` header. Once the object is deleted, +swift will no longer serve the object and it will be deleted from the cluster +shortly thereafter. + +* Set an object to expire at an absolute time (in Unix time). You + can get the current Unix time by running ``date +'%s'``. + + .. code-block:: console + + $ swift post CONTAINER OBJECT_FILENAME -H "X-Delete-At:UNIX_TIME" + + Verify the ``X-Delete-At`` header has posted to the object: + + .. code-block:: console + + $ swift stat CONTAINER OBJECT_FILENAME + +* Set an object to expire after a relative amount of time (in seconds): + + .. code-block:: console + + $ swift post CONTAINER OBJECT_FILENAME -H "X-Delete-After:SECONDS" + + The ``X-Delete-After`` header will be converted to ``X-Delete-At``. + Verify the ``X-Delete-At`` header has posted to the object: + + .. code-block:: console + + $ swift stat CONTAINER OBJECT_FILENAME + + If you no longer want to expire the object, you can remove the + ``X-Delete-At`` header: + + .. code-block:: console + + $ swift post CONTAINER OBJECT_FILENAME -H "X-Remove-Delete-At:" + +.. note:: + + In order for object expiration to work properly, the + ``swift-object-expirer`` daemon will need access to all backend + servers in the cluster. The daemon does not need access to the + proxy-server or public network. diff --git a/doc/source/api/pagination.rst b/doc/source/api/pagination.rst new file mode 100644 index 0000000000..61f92500fd --- /dev/null +++ b/doc/source/api/pagination.rst @@ -0,0 +1,103 @@ +================================================= +Page through large lists of containers or objects +================================================= + +If you have a large number of containers or objects, you can use the +``marker``, ``limit``, and ``end_marker`` parameters to control +how many items are returned in a list and where the list starts or ends. + +* marker + When you request a list of containers or objects, Object Storage + returns a maximum of 10,000 names for each request. To get + subsequent names, you must make another request with the + ``marker`` parameter. Set the ``marker`` parameter to the name of + the last item returned in the previous list. You must URL-encode the + ``marker`` value before you send the HTTP request. Object Storage + returns a maximum of 10,000 names starting after the last item + returned. + +* limit + To return fewer than 10,000 names, use the ``limit`` parameter. If + the number of names returned equals the specified ``limit`` (or + 10,000 if you omit the ``limit`` parameter), you can assume there + are more names to list. If the number of names in the list is + exactly divisible by the ``limit`` value, the last request has no + content. + +* end_marker + Limits the result set to names that are less than the + ``end_marker`` parameter value. You must URL-encode the + ``end_marker`` value before you send the HTTP request. + +To page through a large list of containers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Assume the following list of container names: + +.. code-block:: console + + apples + bananas + kiwis + oranges + pears + +#. Use a ``limit`` of two: + + .. code-block:: console + + # curl -i $publicURL/?limit=2 -X GET -H "X-Auth-Token: $token" + + .. code-block:: console + + apples + bananas + + Because two container names are returned, there are more names to + list. + +#. Make another request with a ``marker`` parameter set to the name of + the last item returned: + + .. code-block:: console + + # curl -i $publicURL/?limit=2&marker=bananas -X GET -H \ + “X-Auth-Token: $token" + + .. code-block:: console + + kiwis + oranges + + Again, two items are returned, and there might be more. + +#. Make another request with a ``marker`` of the last item returned: + + .. code-block:: console + + # curl -i $publicURL/?limit=2&marker=oranges -X GET -H \” + X-Auth-Token: $token" + + .. code-block:: console + + pears + + You receive a one-item response, which is fewer than the ``limit`` + number of names. This indicates that this is the end of the list. + +#. Use the ``end_marker`` parameter to limit the result set to object + names that are less than the ``end_marker`` parameter value: + + .. code-block:: console + + # curl -i $publicURL/?end_marker=oranges -X GET -H \” + X-Auth-Token: $token" + + .. code-block:: console + + apples + bananas + kiwis + + You receive a result set of all container names before the + ``end-marker`` value. diff --git a/doc/source/api/pseudo-hierarchical-folders-directories.rst b/doc/source/api/pseudo-hierarchical-folders-directories.rst new file mode 100644 index 0000000000..a46acd576d --- /dev/null +++ b/doc/source/api/pseudo-hierarchical-folders-directories.rst @@ -0,0 +1,155 @@ +=========================================== +Pseudo-hierarchical folders and directories +=========================================== + +Although you cannot nest directories in OpenStack Object Storage, you +can simulate a hierarchical structure within a single container by +adding forward slash characters (``/``) in the object name. To navigate +the pseudo-directory structure, you can use the ``delimiter`` query +parameter. This example shows you how to use pseudo-hierarchical folders +and directories. + +.. note:: + + In this example, the objects reside in a container called ``backups``. + Within that container, the objects are organized in a pseudo-directory + called ``photos``. The container name is not displayed in the example, + but it is a part of the object URLs. For instance, the URL of the + picture ``me.jpg`` is + ``https://swift.example.com/v1/CF_xer7_343/backups/photos/me.jpg``. + +List pseudo-hierarchical folders request: HTTP +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To display a list of all the objects in the storage container, use +``GET`` without a ``delimiter`` or ``prefix``. + +.. code-block:: console + + $ curl -X GET -i -H "X-Auth-Token: $token" \ + $publicurl/v1/AccountString/backups + +The system returns status code 2xx (between 200 and 299, inclusive) and +the requested list of the objects. + +.. code-block:: console + + photos/animals/cats/persian.jpg + photos/animals/cats/siamese.jpg + photos/animals/dogs/corgi.jpg + photos/animals/dogs/poodle.jpg + photos/animals/dogs/terrier.jpg + photos/me.jpg + photos/plants/fern.jpg + photos/plants/rose.jpg + +Use the delimiter parameter to limit the displayed results. To use +``delimiter`` with pseudo-directories, you must use the parameter slash +(``/``). + +.. code-block:: console + + $ curl -X GET -i -H "X-Auth-Token: $token" \ + $publicurl/v1/AccountString/backups?delimiter=/ + +The system returns status code 2xx (between 200 and 299, inclusive) and +the requested matching objects. Because you use the slash, only the +pseudo-directory ``photos/`` displays. The returned values from a slash +``delimiter`` query are not real objects. The value will refer to +a real object if it does not end with a slash. The pseudo-directories +have no content-type, rather, each pseudo-directory has +its own ``subdir`` entry in the response of JSON and XML results. +For example: + +.. code-block:: JSON + + [ + { + "subdir": "photos/" + } + ] + +.. code-block:: XML + + + + + photos/ + + + +Use the ``prefix`` and ``delimiter`` parameters to view the objects +inside a pseudo-directory, including further nested pseudo-directories. + +.. code-block:: console + + $ curl -X GET -i -H "X-Auth-Token: $token" \ + $publicurl/v1/AccountString/backups?prefix=photos/&delimiter=/ + +The system returns status code 2xx (between 200 and 299, inclusive) and +the objects and pseudo-directories within the top level +pseudo-directory. + +.. code-block:: console + + photos/animals/ + photos/me.jpg + photos/plants/ + +.. code-block:: JSON + + [ + { + "subdir": "photos/animals/" + }, + { + "hash": "b249a153f8f38b51e92916bbc6ea57ad", + "last_modified": "2015-12-03T17:31:28.187370", + "bytes": 2906, + "name": "photos/me.jpg", + "content_type": "image/jpeg" + }, + { + "subdir": "photos/plants/" + } + ] + +.. code-block:: XML + + + + + photos/animals/ + + + photos/me.jpg + b249a153f8f38b51e92916bbc6ea57ad + 2906 + image/jpeg + 2015-12-03T17:31:28.187370 + + + photos/plants/ + + + +You can create an unlimited number of nested pseudo-directories. To +navigate through them, use a longer ``prefix`` parameter coupled with +the ``delimiter`` parameter. In this sample output, there is a +pseudo-directory called ``dogs`` within the pseudo-directory +``animals``. To navigate directly to the files contained within +``dogs``, enter the following command: + +.. code-block:: console + + $ curl -X GET -i -H "X-Auth-Token: $token" \ + $publicurl/v1/AccountString/backups?prefix=photos/animals/dogs/&delimiter=/ + +The system returns status code 2xx (between 200 and 299, inclusive) and +the objects and pseudo-directories within the nested pseudo-directory. + +.. code-block:: console + + photos/animals/dogs/corgi.jpg + photos/animals/dogs/poodle.jpg + photos/animals/dogs/terrier.jpg diff --git a/doc/source/api/serialized-response-formats.rst b/doc/source/api/serialized-response-formats.rst new file mode 100644 index 0000000000..59951ed6af --- /dev/null +++ b/doc/source/api/serialized-response-formats.rst @@ -0,0 +1,119 @@ +=========================== +Serialized response formats +=========================== + +By default, the Object Storage API uses a ``text/plain`` response +format. In addition, both JSON and XML data serialization response +formats are supported. + +To define the response format, use one of these methods: + ++-------------------+-------------------------------------------------------+ +|Method |Description | ++===================+=======================================================+ +|format= ``format`` |Append this parameter to the URL for a ``GET`` request,| +|query parameter |where ``format`` is ``json`` or ``xml``. | ++-------------------+-------------------------------------------------------+ +|``Accept`` request |Include this header in the ``GET`` request. | +|header |The valid header values are: | +| | | +| |text/plain | +| | Plain text response format. The default. | +| |application/jsontext | +| | JSON data serialization response format. | +| |application/xml | +| | XML data serialization response format. | +| |text/xml | +| | XML data serialization response format. | ++-------------------+-------------------------------------------------------+ + +Example 1. JSON example with format query parameter +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For example, this request uses the ``format`` query parameter to ask +for a JSON response: + +.. code-block:: console + + $ curl -i $publicURL?format=json -X GET -H "X-Auth-Token: $token" + +.. code-block:: console + + HTTP/1.1 200 OK + Content-Length: 96 + X-Account-Object-Count: 1 + X-Timestamp: 1389453423.35964 + X-Account-Meta-Subject: Literature + X-Account-Bytes-Used: 14 + X-Account-Container-Count: 2 + Content-Type: application/json; charset=utf-8 + Accept-Ranges: bytes + X-Trans-Id: tx274a77a8975c4a66aeb24-0052d95365 + Date: Fri, 17 Jan 2014 15:59:33 GMT + +Object Storage lists container names with additional information in JSON +format: + +.. code-block:: json + + [ + { + "count":0, + "bytes":0, + "name":"janeausten" + }, + { + "count":1, + "bytes":14, + "name":"marktwain" + } + ] + + +Example 2. XML example with Accept header +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This request uses the ``Accept`` request header to ask for an XML +response: + +.. code-block:: console + + $ curl -i $publicURL -X GET -H "X-Auth-Token: $token" -H \ + ”Accept: application/xml; charset=utf-8" + +.. code-block:: console + + HTTP/1.1 200 OK + Content-Length: 263 + X-Account-Object-Count: 3 + X-Account-Meta-Book: MobyDick + X-Timestamp: 1389453423.35964 + X-Account-Bytes-Used: 47 + X-Account-Container-Count: 2 + Content-Type: application/xml; charset=utf-8 + Accept-Ranges: bytes + X-Trans-Id: txf0b4c9727c3e491694019-0052e03420 + Date: Wed, 22 Jan 2014 21:12:00 GMT + +Object Storage lists container names with additional information in XML +format: + +.. code-block:: xml + + + + + janeausten + 2 + 33 + + + marktwain + 1 + 14 + + + +The remainder of the examples in this guide use standard, non-serialized +responses. However, all ``GET`` requests that perform list operations +accept the ``format`` query parameter or ``Accept`` request header. diff --git a/doc/source/api/static-website.rst b/doc/source/api/static-website.rst new file mode 100644 index 0000000000..48dd34c9df --- /dev/null +++ b/doc/source/api/static-website.rst @@ -0,0 +1,120 @@ +.. _static-website: + +===================== +Create static website +===================== + +To discover whether your Object Storage system supports this feature, +see :ref:`discoverability`. Alternatively, check with your service +provider. + +You can use your Object Storage account to create a static website. This +static website is created with Static Web middleware and serves container +data with a specified index file, error file resolution, and optional +file listings. This mode is normally active only for anonymous requests, +which provide no authentication token. To use it with authenticated +requests, set the header ``X-Web-Mode`` to ``TRUE`` on the request. + +The Static Web filter must be added to the pipeline in your +``/etc/swift/proxy-server.conf`` file below any authentication +middleware. You must also add a Static Web middleware configuration +section. + +Your publicly readable containers are checked for two headers, +``X-Container-Meta-Web-Index`` and ``X-Container-Meta-Web-Error``. The +``X-Container-Meta-Web-Error`` header is discussed below, in the +section called :ref:`set_error_static_website`. + +Use ``X-Container-Meta-Web-Index`` to determine the index file (or +default page served, such as ``index.html``) for your website. When +someone initially enters your site, the ``index.html`` file displays +automatically. If you create sub-directories for your site by creating +pseudo-directories in your container, the index page for each +sub-directory is displayed by default. If your pseudo-directory does not +have a file with the same name as your index file, visits to the +sub-directory return a 404 error. + +You also have the option of displaying a list of files in your +pseudo-directory instead of a web page. To do this, set the +``X-Container-Meta-Web-Listings`` header to ``TRUE``. You may add styles +to your file listing by setting ``X-Container-Meta-Web-Listings-CSS`` +to a style sheet (for example, ``lists.css``). + +Static Web middleware through Object Storage +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The following sections show how to use Static Web middleware through +Object Storage. + +Make container publicly readable +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Make the container publicly readable. Once the container is publicly +readable, you can access your objects directly, but you must set the +index file to browse the main site URL and its sub-directories. + +.. code-block:: console + + $ swift post -r '.r:*,.rlistings' container + + +Set site index file +^^^^^^^^^^^^^^^^^^^ + +Set the index file. In this case, ``index.html`` is the default file +displayed when the site appears. + +.. code-block:: console + + $ swift post -m 'web-index:index.html' container + +Enable file listing +^^^^^^^^^^^^^^^^^^^ + +Turn on file listing. If you do not set the index file, the URL displays +a list of the objects in the container. Instructions on styling the list +with a CSS follow. + +.. code-block:: console + + $ swift post -m 'web-listings: true' container + +Enable CSS for file listing +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Style the file listing using a CSS. + +.. code-block:: console + + $ swift post -m 'web-listings-css:listings.css' container + +.. _set_error_static_website: + +Set error pages for static website +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +You can create and set custom error pages for visitors to your website; +currently, only 401 (Unauthorized) and 404 (Not Found) errors are +supported. To do this, set the metadata header, +``X-Container-Meta-Web-Error``. + +Error pages are served with the status code pre-pended to the name of +the error page you set. For instance, if you set +``X-Container-Meta-Web-Error`` to ``error.html``, 401 errors will +display the page ``401error.html``. Similarly, 404 errors will display +``404error.html``. You must have both of these pages created in your +container when you set the ``X-Container-Meta-Web-Error`` metadata, or +your site will display generic error pages. + +You only have to set the ``X-Container-Meta-Web-Error`` metadata once +for your entire static website. + +Set error pages for static website request +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: console + + $ swift post -m 'web-error:error.html' container + + +Any 2\ ``nn`` response indicates success. diff --git a/doc/source/index.rst b/doc/source/index.rst index 58449c2581..199be4e0d6 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -117,6 +117,12 @@ The following provides supporting information for the REST API: api/form_post_middleware.rst api/use_content-encoding_metadata.rst api/use_the_content-disposition_metadata.rst + api/pseudo-hierarchical-folders-directories.rst + api/pagination.rst + api/serialized-response-formats.rst + api/static-website.rst + api/object-expiration.rst + api/bulk-delete.rst S3 Compatibility Info =====================