diff --git a/firstapp/samples/fog/durability.rb b/firstapp/samples/fog/durability.rb new file mode 100644 index 000000000..0cf94c6eb --- /dev/null +++ b/firstapp/samples/fog/durability.rb @@ -0,0 +1,102 @@ +# step-1 +require 'fog' + +auth_username = 'your_auth_username' +auth_password = 'your_auth_password' +auth_url = 'http://controller:5000' +project_name = 'your_project_name_or_id' +region_name = 'your_region_name' + +swift = Fog::Storage.new({ + :provider => 'openstack', + :openstack_auth_url => auth_url + '/v2.0/tokens', + :openstack_username => auth_username, + :openstack_tenant => project_name, + :openstack_api_key => auth_password, + :openstack_region => region_name +}) + +# step-2 +container_name = 'fractals' +container = swift.directories.create :key => container_name + +# step-3 +swift.directories + +# step-4 +file_path = 'goat.jpg' +object_name = 'an amazing goat' +container = swift.directories.get container_name +object = container.upload_object(file_path=file_path, object_name=object_name) +object = container.files.create :key => object_name, :body => File.open file_path + +# step-5 +objects = container.files +puts objects + +# step-6 +object = container.files.get(object_name) +puts object + +# step-7 +require 'digest/md5' +file = File.open('goat.jpg', "rb") +contents = file.read +file.close +digest = Digest::MD5.hexdigest(contents) +puts digest + + +# step-8 +object.destroy + +# step-9 +objects = container.files +puts objects + +# step-10 +container_name = 'fractals' +container = swift.directories.get container_name + +# step-11 +require "net/https" +require "uri" +require "json" + +endpoint = 'http://IP_API_1' +uri = URI.parse('%s/v1/fractal?results_per_page=-1' % endpoint) +http = Net::HTTP.new(uri.host, uri.port) +request = Net::HTTP::Get.new(uri.request_uri) +response = http.request(request) +data = json.parse(response.body) + +for fractal in data['objects'] + request = Net::HTTP::Get.new('%s/fractal/%s' % (endpoint, fractal['uuid']), stream=True) + response = http.request(request) + container.files.create :key => fractal['uuid'], :body => response.body +end + +for object in container.files: + puts object + +# step-12 +for object in container.files: + object.destroy +container.destroy + +# step-13 +file_path = 'goat.jpg' +object_name = 'backup_goat.jpg' +metadata = {'description' => 'a funny goat', 'created' => '2015-06-02'} +container.files.creare :key => object_name, :body File.open file_path, :metadata => metadata + +# step-14 +# XXX TODOZ TBC +chunk_size = 33554432 +File.open(file_path) do |chunk| + +end + +swift.put_object_manifest(container_name, object_name, 'X-Object-Manifest' => container_name + "/" + object_name "/") + +# step-15 diff --git a/firstapp/source/durability.rst b/firstapp/source/durability.rst index bec647a55..41e0c7967 100644 --- a/firstapp/source/durability.rst +++ b/firstapp/source/durability.rst @@ -1,6 +1,6 @@ -================= -Making it durable -================= +=============== +Make it durable +=============== .. todo:: https://github.com/apache/libcloud/pull/492 @@ -11,66 +11,59 @@ Making it durable .. todo:: Large object support in Swift http://docs.openstack.org/developer/swift/overview_large_objects.html -This section introduces object storage. `OpenStack Object Storage -`_ (code-named -swift) is open source software for creating redundant, scalable data -storage using clusters of standardized servers to store petabytes of -accessible data. It is a long-term storage system for large amounts -of static data that can be retrieved, leveraged, and updated. Access -is via an API, not through a file-system like more traditional -storage. +This section introduces object storage. -There are a two key concepts to understand in the Object Storage -API. The Object Storage API is organized around two types of entities: +`OpenStack Object Storage `_ +(code-named swift) is open-source software that enables you to create +redundant, scalable data storage by using clusters of standardized servers to +store petabytes of accessible data. It is a long-term storage system for large +amounts of static data that you can retrieve, leverage, and update. Unlike +more traditional storage systems that you access through a file system, you +access Object Storage through an API. -* Objects -* Containers +The Object Storage API is organized around objects and containers. -Similar to the Unix programming model, an object is a "bag of bytes" -that contains data, such as documents and images. Containers are used -to group objects. You can make many objects inside a container, and -have many containers inside your account. +Similar to the UNIX programming model, an object, such as a document or an +image, is a "bag of bytes" that contains data. You use containers to group +objects. You can place many objects inside a container, and your account can +have many containers. -If you think about how you traditionally make what you store durable, -very quickly you should come to the conclusion that keeping multiple -copies of your objects on separate systems is a good way to do -that. However, keeping track of multiple copies of objects is a pain, -and building that into an app requires a lot of logic. OpenStack -Object Storage does this automatically for you behind-the-scenes - -replicating each object at least twice before returning 'write -success' to your API call. It will always work to ensure that there -are three copies of your objects (by default) at all times - -replicating them around the system in case of hardware failure, -maintenance, network outage or any other kind of breakage. This is -very convenient for app creation - you can just dump objects into -object storage and not have to care about any of this additional work -to keep them safe. +If you think about how you traditionally make what you store durable, you +quickly conclude that keeping multiple copies of your objects on separate +systems is a good way strategy. However, keeping track of those multiple +copies is difficult, and building that into an app requires complicated logic. + +OpenStack Object Storage automatically replicates each object at least twice +before returning 'write success' to your API call. A good strategy is to keep +three copies of objects, by default, at all times, replicating them across the +system in case of hardware failure, maintenance, network outage, or another +kind of breakage. This strategy is very convenient for app creation. You can +just dump objects into object storage and not worry about the additional work +that it takes to keep them safe. -Using Object Storage to store fractals --------------------------------------- +Use Object Storage to store fractals +------------------------------------ -The Fractals app currently uses the local file system on the instance -to store the images it generates. This is not scalable or durable, for -a number of reasons. +The Fractals app currently uses the local file system on the instance to store +the images that it generates. For a number of reasons, this approach is not +scalable or durable. -Because the local file system is ephemeral storage, if the instance is -terminated, the fractal images will be lost along with the instance. -Block based storage, which we will discuss in :doc:`/block_storage`, -avoids that problem, but like local file systems, it requires -administration to ensure that it does not fill up, and immediate -attention if disks fail. +Because the local file system is ephemeral storage, the fractal images are +lost along with the instance when the instance is terminated. Block-based +storage, which the :doc:`/block_storage` section discusses, avoids that +problem, but like local file systems, it requires administration to ensure +that it does not fill up, and immediate attention if disks fail. -The Object Storage service manages many of these tasks that normally -would require the application owner to manage them, and presents a -scalable and durable API that you can use for the fractals app, -without having to be concerned with the low level details of how the -objects are stored and replicated, and growing the storage pool. In -fact, Object Storage handles replication intrinsically, storing multiple -copies of each object and returning one of them on demand using the -API. +The Object Storage service manages many of the tasks normally managed by the +application owner. The Object Storage service provides a scalable and durable +API that you can use for the fractals app, eliminating the need to be aware of +the low level details of how objects are stored and replicated, and how to +grow the storage pool. Object Storage handles replication for you. It stores +multiple copies of each object. You can use the Object Storage API to return +an object, on demand. -First, let's learn how to connect to the Object Storage endpoint: +First, learn how to connect to the Object Storage endpoint: .. only:: dotnet @@ -78,7 +71,10 @@ First, let's learn how to connect to the Object Storage endpoint: .. only:: fog - .. warning:: This section has not yet been completed for the fog SDK. + .. literalinclude:: ../samples/fog/durability.rb + :start-after: step-1 + :end-before: step-2 + .. only:: jclouds @@ -123,6 +119,18 @@ First, let's learn how to connect to the Object Storage endpoint: To begin to store objects, we must first make a container. Call yours :code:`fractals`: +.. only:: fog + + .. literalinclude:: ../samples/fog/durability.rb + :start-after: step-2 + :end-before: step-3 + + You should see output such as: + + .. code-block:: ruby + + TBC + .. only:: libcloud .. literalinclude:: ../samples/libcloud/durability.py @@ -138,6 +146,18 @@ Call yours :code:`fractals`: You should now be able to see this container appear in a listing of all containers in your account: +.. only:: fog + + .. literalinclude:: ../samples/fog/durability.rb + :start-after: step-3 + :end-before: step-4 + + You should see output such as: + + .. code-block:: ruby + + TBC + .. only:: libcloud .. literalinclude:: ../samples/libcloud/durability.py @@ -154,6 +174,12 @@ The next logical step is to upload an object. Find a photo of a goat on line, name it :code:`goat.jpg`, and upload it to your :code:`fractals` container: +.. only:: fog + + .. literalinclude:: ../samples/fog/durability.rb + :start-after: step-4 + :end-before: step-5 + .. only:: libcloud .. literalinclude:: ../samples/libcloud/durability.py @@ -164,6 +190,34 @@ List objects in your :code:`fractals` container to see if the upload was successful. Then, download the file to verify that the md5sum is the same: +.. only:: fog + + .. literalinclude:: ../samples/fog/durability.rb + :start-after: step-5 + :end-before: step-6 + + :: + + TBC + + + .. literalinclude:: ../samples/fog/durability.rb + :start-after: step-6 + :end-before: step-7 + + :: + + TBC + + .. literalinclude:: ../samples/fog/durability.rb + :start-after: step-7 + :end-before: step-8 + + :: + + 7513986d3aeb22659079d1bf3dc2468b + + .. only:: libcloud .. literalinclude:: ../samples/libcloud/durability.py @@ -192,8 +246,13 @@ the same: 7513986d3aeb22659079d1bf3dc2468b +Finally, clean up by deleting the test object: -Finally, let's clean up by deleting our test object: +.. only:: fog + + .. literalinclude:: ../samples/fog/durability.rb + :start-after: step-8 + :end-before: step-9 .. only:: libcloud @@ -201,9 +260,9 @@ Finally, let's clean up by deleting our test object: :start-after: step-8 :end-before: step-9 - .. note:: You need to pass in objects to the delete commands, not object names. + .. note:: You must pass in objects and not object names to the delete commands. - Now there should be no more objects be available in the container :code:`fractals`. + Now, no more objects are available in the :code:`fractals` container. .. literalinclude:: ../samples/libcloud/durability.py :start-after: step-9 @@ -213,14 +272,19 @@ Finally, let's clean up by deleting our test object: [] -Backup the Fractals from the database on the Object Storage -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Back up the Fractals from the database on the Object Storage +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -So let's now use the knowledge from above to backup the images of the -Fractals app, stored inside the database right now, on the Object -Storage. +Back up the Fractals app images, which are currently stored inside the +database, on Object Storage. -Use the :code:`fractals`' container from above to put the images in: +Place the images in the :code:`fractals`' container: + +.. only:: fog + + .. literalinclude:: ../samples/fog/durability.rb + :start-after: step-10 + :end-before: step-11 .. only:: libcloud @@ -228,8 +292,16 @@ Use the :code:`fractals`' container from above to put the images in: :start-after: step-10 :end-before: step-11 -Next, we backup all of our existing fractals from the database to our -swift container. A simple for loop takes care of that: +Next, back up all existing fractals from the database to the swift container. +A simple `for` loop takes care of that: + +.. note:: Replace :code:`IP_API_1` with the IP address of the API instance. + +.. only:: fog + + .. literalinclude:: ../samples/fog/durability.rb + :start-after: step-11 + :end-before: step-12 .. only:: libcloud @@ -264,9 +336,14 @@ Extra features Delete containers ~~~~~~~~~~~~~~~~~ -One call we did not cover and that you probably need to know is how -to delete a container. Ensure that you have removed all objects from -the container before running this script. Otherwise, the script fails: +To delete a container, make sure that you have removed all objects from the +container before running this script. Otherwise, the script fails: + +.. only:: fog + + .. literalinclude:: ../samples/fog/durability.rb + :start-after: step-12 + :end-before: step-13 .. only:: libcloud @@ -279,13 +356,18 @@ the container before running this script. Otherwise, the script fails: Add metadata to objects ~~~~~~~~~~~~~~~~~~~~~~~ -You can also do advanced things like uploading an object with -metadata, such as in following example. For more information, see the -documentation for your SDK. This option also uses a bit stream to -upload the file, iterating bit by bit over the file and passing those -bits to Object Storage as they come. Compared to loading the entire -file in memory and then sending it, this method is more efficient, -especially for larger files. +You can complete advanced tasks such as uploading an object with metadata, as +shown in following example. For more information, see the documentation for +your SDK. This option also uses a bit stream to upload the file, iterating bit +by bit over the file and passing those bits to Object Storage as they come. +Compared to loading the entire file in memory and then sending it, this method +is more efficient, especially for larger files. + +.. only:: fog + + .. literalinclude:: ../samples/fog/durability.rb + :start-after: step-13 + :end-before: step-14 .. only:: libcloud @@ -298,18 +380,22 @@ especially for larger files. Large objects ~~~~~~~~~~~~~ -For efficiency, most Object Storage installations treat large objects -(say, :code:`> 5GB`) differently than smaller objects. +For efficiency, most Object Storage installations treat large objects, +:code:`> 5GB`, differently than smaller objects. + +.. only:: fog + + .. literalinclude:: ../samples/fog/durability.rb + :start-after: step-14 + :end-before: step-15 .. only:: libcloud - If you are working with large objects, use the - :code:`ex_multipart_upload_object` call instead of the simpler - :code:`upload_object` call. Behind the scenes, the call splits the - large object into chunks and creates a special manifest so that - the chunks can be recombined on download. Alter the - :code:`chunk_size` parameter (in bytes) according to what your - cloud can accept. + If you work with large objects, use the :code:`ex_multipart_upload_object` + call instead of the simpler :code:`upload_object` call. The call splits + the large object into chunks and creates a manifest so that the chunks can + be recombined on download. Change the :code:`chunk_size` parameter, in + bytes, to a value that your cloud can accept. .. literalinclude:: ../samples/libcloud/durability.py :start-after: step-14 @@ -322,11 +408,15 @@ Next steps You should now be fairly confident working with Object Storage. You can find more information about the Object Storage SDK calls at: +.. only:: fog + + https://github.com/fog/fog/blob/master/lib/fog/openstack/docs/storage.md + .. only:: libcloud https://libcloud.readthedocs.org/en/latest/storage/api.html -Or, try one of these steps in the tutorial: +Or, try one of these tutorial steps: * :doc:`/block_storage`: Migrate the database to block storage, or use the database-as-a-service component. diff --git a/firstapp/source/networking.rst b/firstapp/source/networking.rst index 7becf0d0d..94c1c3349 100644 --- a/firstapp/source/networking.rst +++ b/firstapp/source/networking.rst @@ -202,7 +202,7 @@ instances. After also attaching this router to our internal networks, we will allocate floating IPs from the public network for instances which need to be accessed from the Internet. -Let's just confirm that we have a public network by listing the +Confirm that we have a public network by listing the networks our tenant has access to. The public network does not have to be named public - it could be 'external', 'net04_ext' or something else - the important thing is it exists and can be used to reach the @@ -529,7 +529,7 @@ nodes. The end result will be the following network topology: libcloud support added 0.14: https://developer.rackspace.com/blog/libcloud-0-dot-14-released/ -Let's start by looking at what's already in place. +Start by looking at what is already in place. :: @@ -686,7 +686,7 @@ You should be able to see them in the member list: | f3ba0605-4926-4498-b86d-51002892e93a | 203.0.113.22 | 80 | 1 | True | ACTIVE | +--------------------------------------+--------------+---------------+--------+----------------+--------+ -Now let's create a health monitor that will ensure that members of the +Now, create a health monitor that will ensure that members of the load balancer pool are active and able to respond to requests. If a member in the pool dies or is unresponsive, the member is removed from the pool so that client requests are routed to another active member. @@ -751,7 +751,7 @@ And confirm it is in place: | f0bcb66e-5eeb-447b-985e-faeb67540c2f | myvip | 203.0.113.63 | HTTP | True | ACTIVE | +--------------------------------------+-------+--------------+----------+----------------+--------+ -Now let's look at the big picture. +Now, look at the big picture. Final result ~~~~~~~~~~~~