FirstApp: Edits section4
Wrap overlong lines, follow markup conventions. Change-Id: I37c0eb543bae008175666a38cfa8805b78108ca1
This commit is contained in:
Andreas Jaeger
@@ -1,6 +1,6 @@
|
|||||||
===============================
|
=================
|
||||||
Section Four: Making it Durable
|
Making it durable
|
||||||
===============================
|
=================
|
||||||
|
|
||||||
.. todo:: https://github.com/apache/libcloud/pull/492
|
.. todo:: https://github.com/apache/libcloud/pull/492
|
||||||
|
|
||||||
@@ -8,67 +8,81 @@ Section Four: Making it Durable
|
|||||||
|
|
||||||
.. todo:: Explain how to get objects back out again.
|
.. todo:: Explain how to get objects back out again.
|
||||||
|
|
||||||
.. todo:: Large object support in Swift http://docs.openstack.org/developer/swift/overview_large_objects.html
|
.. todo:: Large object support in Swift
|
||||||
|
http://docs.openstack.org/developer/swift/overview_large_objects.html
|
||||||
|
|
||||||
This section introduces object storage.
|
This section introduces object storage. `OpenStack Object Storage
|
||||||
`OpenStack Object Storage <http://www.openstack.org/software/openstack-storage/>`_
|
<http://www.openstack.org/software/openstack-storage/>`_ (code-named
|
||||||
(code-named Swift) is open source software for creating redundant, scalable data storage
|
swift) is open source software for creating redundant, scalable data
|
||||||
using clusters of standardized servers to store petabytes of accessible data.
|
storage using clusters of standardized servers to store petabytes of
|
||||||
It is a long-term storage system for large amounts of static data that can be
|
accessible data. It is a long-term storage system for large amounts
|
||||||
retrieved, leveraged, and updated. Access is via an API, not through a file-system
|
of static data that can be retrieved, leveraged, and updated. Access
|
||||||
like more traditional storage.
|
is via an API, not through a file-system like more traditional
|
||||||
|
storage.
|
||||||
|
|
||||||
There are a two key concepts to understand in the Object Storage API. The Object
|
There are a two key concepts to understand in the Object Storage
|
||||||
Storage API is organized around two types of entities:
|
API. The Object Storage API is organized around two types of entities:
|
||||||
|
|
||||||
* Objects
|
* Objects
|
||||||
* Containers
|
* Containers
|
||||||
|
|
||||||
Similar to the Unix programming model, an Object is a "bag of bytes" that contains data,
|
Similar to the Unix programming model, an object is a "bag of bytes"
|
||||||
such as documents and images. Containers are used to group objects.
|
that contains data, such as documents and images. Containers are used
|
||||||
You can make many objects inside a container, and have many containers inside your account.
|
to group objects. You can make many objects inside a container, and
|
||||||
|
have many containers inside your account.
|
||||||
|
|
||||||
If you think about how you traditionally make what you store durable, very quickly you should come
|
If you think about how you traditionally make what you store durable,
|
||||||
to the conclusion that keeping multiple copies of your objects on separate systems is a good way
|
very quickly you should come to the conclusion that keeping multiple
|
||||||
to do that. However, keeping track of multiple copies of objects is a pain, and building that
|
copies of your objects on separate systems is a good way to do
|
||||||
into an app requires a lot of logic. OpenStack Object Storage does this automatically for you
|
that. However, keeping track of multiple copies of objects is a pain,
|
||||||
behind-the-scenes - replicating each object at least twice before returning 'write success' to your
|
and building that into an app requires a lot of logic. OpenStack
|
||||||
API call. It will always work to ensure that there are three copies of your objects (by default)
|
Object Storage does this automatically for you behind-the-scenes -
|
||||||
at all times - replicating them around the system in case of hardware failure, maintanance, network
|
replicating each object at least twice before returning 'write
|
||||||
outage or any other kind of breakage. This is very convenient for app creation - you can just dump
|
success' to your API call. It will always work to ensure that there
|
||||||
objects into object storage and not have to care about any of this additional work to keep them safe.
|
are three copies of your objects (by default) at all times -
|
||||||
|
replicating them around the system in case of hardware failure,
|
||||||
|
maintanance, network outage or any other kind of breakage. This is
|
||||||
|
very convenient for app creation - you can just dump objects into
|
||||||
|
object storage and not have to care about any of this additional work
|
||||||
|
to keep them safe.
|
||||||
|
|
||||||
|
|
||||||
Using Object Storage to store fractals
|
Using Object Storage to store fractals
|
||||||
--------------------------------------
|
--------------------------------------
|
||||||
|
|
||||||
The Fractals app currently uses the local filesystem on the instance to store the images it
|
The Fractals app currently uses the local filesystem on the instance
|
||||||
generates. This is not scalable or durable, for a number of reasons.
|
to store the images it generates. This is not scalable or durable, for
|
||||||
|
a number of reasons.
|
||||||
|
|
||||||
Because the local filesystem is ephemeral storage, if the instance is terminated, the fractal
|
Because the local filesystem is ephemeral storage, if the instance is
|
||||||
images will be lost along with the instance. Block based storage, which we'll discuss in :doc:`/section5`,
|
terminated, the fractal images will be lost along with the
|
||||||
avoids that problem, but like local filesystems, it
|
instance. Block based storage, which we'll discuss in
|
||||||
requires administration to ensure that it does not fill up, and immediate attention if disks fail.
|
:doc:`/section5`, avoids that problem, but like local filesystems, it
|
||||||
|
requires administration to ensure that it does not fill up, and
|
||||||
|
immediate attention if disks fail.
|
||||||
|
|
||||||
The Object Storage service manages many of these tasks that normally would require the application owner
|
The Object Storage service manages many of these tasks that normally
|
||||||
to manage them, and presents a scalable and durable API that you can use for the fractals app, without
|
would require the application owner to manage them, and presents a
|
||||||
having to be concerened with the low level details of how the objects are stored and replicated,
|
scalable and durable API that you can use for the fractals app,
|
||||||
and growing the storage pool. In fact, Object Storage handles replication intrinsicly, storing multiple
|
without having to be concerened with the low level details of how the
|
||||||
copies of each object and returning one of them on demand using the API.
|
objects are stored and replicated, and growing the storage pool. In
|
||||||
|
fact, Object Storage handles replication intrinsicly, storing multiple
|
||||||
|
copies of each object and returning one of them on demand using the
|
||||||
|
API.
|
||||||
|
|
||||||
First, let's learn how to connect to the Object Storage Endpoint:
|
First, let's learn how to connect to the Object Storage endpoint:
|
||||||
|
|
||||||
.. only:: dotnet
|
.. only:: dotnet
|
||||||
|
|
||||||
.. warning:: This section has not yet been completed for the .NET SDK
|
.. warning:: This section has not yet been completed for the .NET SDK.
|
||||||
|
|
||||||
.. only:: fog
|
.. only:: fog
|
||||||
|
|
||||||
.. warning:: This section has not yet been completed for the fog SDK
|
.. warning:: This section has not yet been completed for the fog SDK.
|
||||||
|
|
||||||
.. only:: jclouds
|
.. only:: jclouds
|
||||||
|
|
||||||
.. warning:: This section has not yet been completed for the jclouds SDK
|
.. warning:: This section has not yet been completed for the jclouds SDK.
|
||||||
|
|
||||||
.. only:: libcloud
|
.. only:: libcloud
|
||||||
|
|
||||||
@@ -79,25 +93,31 @@ First, let's learn how to connect to the Object Storage Endpoint:
|
|||||||
|
|
||||||
.. warning::
|
.. warning::
|
||||||
|
|
||||||
Libcloud 0.16 and 0.17 are afflicted with a bug that means authentication to
|
Libcloud 0.16 and 0.17 are afflicted with a bug that means
|
||||||
a swift endpoint can fail with `a Python exception <https://issues.apache.org/jira/browse/LIBCLOUD-635>`_.
|
authentication to a swift endpoint can fail with `a Python
|
||||||
If you encounter this, you can upgrade your libcloud version, or apply a simple
|
exception
|
||||||
`2-line patch <https://github.com/fifieldt/libcloud/commit/ec58868c3344a9bfe7a0166fc31c0548ed22ea87>`_.
|
<https://issues.apache.org/jira/browse/LIBCLOUD-635>`_. If
|
||||||
|
you encounter this, you can upgrade your libcloud version, or
|
||||||
|
apply a simple `2-line patch
|
||||||
|
<https://github.com/fifieldt/libcloud/commit/ec58868c3344a9bfe7a0166fc31c0548ed22ea87>`_.
|
||||||
|
|
||||||
.. note:: Libcloud uses a different connector for Object Storage to all other OpenStack services,
|
.. note:: Libcloud uses a different connector for Object Storage
|
||||||
so a conn object from previous sections won't work here and we have to create a new one named :code:`swift`.
|
to all other OpenStack services, so a conn object from
|
||||||
|
previous sections won't work here and we have to create
|
||||||
|
a new one named :code:`swift`.
|
||||||
|
|
||||||
.. only:: node
|
.. only:: node
|
||||||
|
|
||||||
.. warning:: This section has not yet been completed for the pkgcloud SDK
|
.. warning:: This section has not yet been completed for the pkgcloud SDK.
|
||||||
|
|
||||||
.. only:: openstacksdk
|
.. only:: openstacksdk
|
||||||
|
|
||||||
.. warning:: This section has not yet been completed for the OpenStack SDK
|
.. warning:: This section has not yet been completed for the OpenStack SDK.
|
||||||
|
|
||||||
.. only:: phpopencloud
|
.. only:: phpopencloud
|
||||||
|
|
||||||
.. warning:: This section has not yet been completed for the PHP-OpenCloud SDK
|
.. warning:: This section has not yet been completed for the
|
||||||
|
PHP-OpenCloud SDK.
|
||||||
|
|
||||||
|
|
||||||
To begin to store objects, we must first make a container.
|
To begin to store objects, we must first make a container.
|
||||||
@@ -131,7 +151,8 @@ all containers in your account:
|
|||||||
[<Container: name=fractals, provider=OpenStack Swift>]
|
[<Container: name=fractals, provider=OpenStack Swift>]
|
||||||
|
|
||||||
The next logical step is to upload an object. Find a photo of a goat
|
The next logical step is to upload an object. Find a photo of a goat
|
||||||
online, name it :code:`goat.jpg` and upload it to your container :code:`fractals`:
|
online, name it :code:`goat.jpg` and upload it to your container
|
||||||
|
:code:`fractals`:
|
||||||
|
|
||||||
.. only:: libcloud
|
.. only:: libcloud
|
||||||
|
|
||||||
@@ -139,8 +160,9 @@ online, name it :code:`goat.jpg` and upload it to your container :code:`fractals
|
|||||||
:start-after: step-4
|
:start-after: step-4
|
||||||
:end-before: step-5
|
:end-before: step-5
|
||||||
|
|
||||||
List objects in your container :code:`fractals` to see if the upload was successful, then download
|
List objects in your container :code:`fractals` to see if the upload
|
||||||
the file to verify the md5sum is the same:
|
was successful, then download the file to verify the md5sum is the
|
||||||
|
same:
|
||||||
|
|
||||||
.. only:: libcloud
|
.. only:: libcloud
|
||||||
|
|
||||||
@@ -194,7 +216,9 @@ Finally, let's clean up by deleting our test object:
|
|||||||
Backup the Fractals from the database on the Object Storage
|
Backup the Fractals from the database on the Object Storage
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
So let's now use the knowledge from above to backup the images of the Fractals app, stored inside the database right now, on the Object Storage.
|
So let's now use the knowledge from above to backup the images of the
|
||||||
|
Fractals app, stored inside the database right now, on the Object
|
||||||
|
Storage.
|
||||||
|
|
||||||
Use the :code:`fractals`' container from above to put the images in:
|
Use the :code:`fractals`' container from above to put the images in:
|
||||||
|
|
||||||
@@ -204,7 +228,8 @@ Use the :code:`fractals`' container from above to put the images in:
|
|||||||
:start-after: step-10
|
:start-after: step-10
|
||||||
:end-before: step-11
|
:end-before: step-11
|
||||||
|
|
||||||
Next, we backup all of our existing fractals from the database to our swift container. A simple for loop takes care of that:
|
Next, we backup all of our existing fractals from the database to our
|
||||||
|
swift container. A simple for loop takes care of that:
|
||||||
|
|
||||||
.. only:: libcloud
|
.. only:: libcloud
|
||||||
|
|
||||||
@@ -226,17 +251,19 @@ Next, we backup all of our existing fractals from the database to our swift cont
|
|||||||
Configure the Fractals app to use Object Storage
|
Configure the Fractals app to use Object Storage
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
.. warning:: Currenctly it is not possible to directly store generated images on the OpenStack Object Storage. Please revisit this section again in the future.
|
.. warning:: Currently it is not possible to directly store generated
|
||||||
|
images on the OpenStack Object Storage. Please revisit
|
||||||
|
this section again in the future.
|
||||||
|
|
||||||
Extra Features
|
Extra features
|
||||||
--------------
|
--------------
|
||||||
|
|
||||||
Delete containers
|
Delete containers
|
||||||
~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
One call we didn't cover above that you probably need to know is how to delete a container.
|
One call we didn't cover above that you probably need to know is how
|
||||||
Ensure that you have removed all objects from the container before running this, otherwise
|
to delete a container. Ensure that you have removed all objects from
|
||||||
it will fail:
|
the container before running this, otherwise it will fail:
|
||||||
|
|
||||||
.. only:: libcloud
|
.. only:: libcloud
|
||||||
|
|
||||||
@@ -268,23 +295,25 @@ This is more efficient, especially for larger files.
|
|||||||
Large objects
|
Large objects
|
||||||
~~~~~~~~~~~~~
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
For efficiency, most Object Storage installations treat large objects (say, :code:`> 5GB`)
|
For efficiency, most Object Storage installations treat large objects
|
||||||
differently than smaller objects.
|
(say, :code:`> 5GB`) differently than smaller objects.
|
||||||
|
|
||||||
.. only:: libcloud
|
.. only:: libcloud
|
||||||
|
|
||||||
If you are working with large objects, use the :code:`ex_multipart_upload_object`
|
If you are working with large objects, use the
|
||||||
call instead of the simpler :code:`upload_object` call. How the upload works behind-the-scenes
|
:code:`ex_multipart_upload_object` call instead of the simpler
|
||||||
is by splitting the large object into chunks, and creating a special manifest so
|
:code:`upload_object` call. How the upload works behind-the-scenes
|
||||||
they can be recombined on download. Alter the :code:`chunk_size` parameter (in bytes) according to
|
is by splitting the large object into chunks, and creating a
|
||||||
what your cloud can accept.
|
special manifest so they can be recombined on download. Alter the
|
||||||
|
:code:`chunk_size` parameter (in bytes) according to what your
|
||||||
|
cloud can accept.
|
||||||
|
|
||||||
.. literalinclude:: ../../samples/libcloud/section4.py
|
.. literalinclude:: ../../samples/libcloud/section4.py
|
||||||
:start-after: step-14
|
:start-after: step-14
|
||||||
:end-before: step-15
|
:end-before: step-15
|
||||||
|
|
||||||
|
|
||||||
Next Steps
|
Next steps
|
||||||
----------
|
----------
|
||||||
|
|
||||||
You should now be fairly confident working with Object Storage.
|
You should now be fairly confident working with Object Storage.
|
||||||
@@ -296,8 +325,8 @@ You can find more about the Object Storage SDK calls at:
|
|||||||
|
|
||||||
Or try a different step in the tutorial, including:
|
Or try a different step in the tutorial, including:
|
||||||
|
|
||||||
* :doc:`/section5` - to migrate the database to block storage, or use the database-as-as-service component
|
* :doc:`/section5`: to migrate the database to block storage, or use
|
||||||
* :doc:`/section6` - to automatically orchestrate the application
|
the database-as-as-service component
|
||||||
* :doc:`/section7` - to learn about more complex networking
|
* :doc:`/section6`: to automatically orchestrate the application
|
||||||
* :doc:`/section8` - for advice for developers new to operations
|
* :doc:`/section7`: to learn about more complex networking
|
||||||
|
* :doc:`/section8`: for advice for developers new to operations
|
||||||
|
|||||||
Reference in New Issue
Block a user