openstack/storlets
Code Issues Proposed changes

Minor doc changes

Browse Source 
Cleans up some docs for clarity and fixes a broken link.
Removed whitespace and added some 'bash' code blocks.

Change-Id: I5cc6b7fb01cd6539d6aed2224788fb2acd917495
This commit is contained in:
Hamdi Roumani
2015-08-18 00:33:37 -04:00
parent a3c791eafa
commit 058fb7c222
3 changed files with 51 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
doc/source/building_and_deploying_docker_images.rst
View File

@@ -36,7 +36,7 @@ Below are the steps of this flow:
it to all Swift nodes. Again, <https://github.com/openstack/storlets/blob/master/doc/source/storlets_management.rst> describes the provided tool
the Storlet manager can use for the actual deployment.
The below sections describe in details the steps taken by the account manager
The sections below describe in detail the steps taken by the account manager.
Downloading the Docker Image
============================
@@ -53,19 +53,19 @@ facts:
The image will come in a .tar format.
Below is an example of downloading the image from the tenant's docker_images
container using the swift CLI. As with all examples using the Swicf CLI, we are
container using the swift CLI. As with all examples using the Swift CLI, we are
using environment variables defining the tenant, user credentials and auth URI.
All these are required for the operation of any Swift CLI. Please change them
accordingly.
::
.. code-block:: bash
export OS_USERNAME=swift
export OS_PASSWORD=passw0rd
export OS_TENANT_NAME=service
export OS_AUTH_URL=http://127.0.0.1:35357/v2.0
In the below be show:
In the below we show:
#. Listing the docker_images container.
#. Downloading the image object
@@ -147,7 +147,7 @@ The below steps illustrate the tuning process:
when building an image that copies stuff into the image, all that stuff must
reside in that building directory.
::
.. code-block:: bash
root@lnx-ccs8:/home/eranr/dockerfile_example# docker build -t service_tenant_image .
Sending build context to Docker daemon 2.56 kB
@@ -163,7 +163,7 @@ The below steps illustrate the tuning process:
4. At this point listing the images, shows the newly created image.
::
root@lnx-ccs8:/home/eranr/dockerfile_example# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
service_tenant_image latest 11975468ecf8 7 minutes ago 660.1 MB
@@ -177,7 +177,7 @@ We then invoke ffmpeg showing that the installation was indeed successful.
Note that the 'debug' parameter tells our entry point not to execute the storlet
engine but rather the /bin/bash from which we can run ffmpeg
::
.. code-block:: bash
root@lnx-ccs8:/home/eranr/dockerfile_example# docker run -i -t service_tenant_image debug /bin/bash
root@b129c3e6e76b:/# ffmpeg
@@ -197,19 +197,19 @@ engine but rather the /bin/bash from which we can run ffmpeg
Use -h to get full help or, even better, run 'man ffmpeg'
Uploading the Docker Image
==========================
1. Use docker save to save the image as a tar file:
::
.. code-block:: bash
root@lnx-ccs8:/home/eranr/dockerfile_example# docker save -o service_tenant_image.tar service_tenant_image
2. Again, we use the Swift CLI to upload the image. We assume the appropriate
environment variables are in place.
::
.. code-block:: bash
root@lnx-ccs8:/home/eranr/dockerfile_example# swift upload docker_images service_tenant_image.tar
service_tenant_image.tar

2
doc/source/dev_and_test_guide.rst
View File

@@ -4,7 +4,7 @@ Development and Testing
This guide explains how to build the various components, and how to deploy them once built.
Note that some of the components being built need to be inserted to a docker image before
they can be tested. Thus, once should have an installed environment (see <https://github.com/openstack/storlets/blob/master/doc/source/storlet_installation_guide.rst>)
they can be tested. Thus, one should have an installed environment (see <https://github.com/openstack/storlets/blob/master/doc/source/storlets_installation_guide.rst>)
before proceeding with the test and deploy steps (which are in fact a subset of the installation steps).
Building

80
doc/source/writing_and_deploying_storlets.rst
View File

@@ -6,16 +6,16 @@ implementing a single method interface and following some simple rules and best
practices described below.
Once the Storlet is written and tested it can be uploaded as an object to a
designated container (called 'storlet' by default). In addition in case the
Storlet is dependent on some Java library, that library can be uploaded as a
dependency of the Storlet. It is assumed that Storlet dependencies are small
(on the order of few MBs), heavier dependencies should be part of the Docker
designated container (called 'storlet' by default). In addition in case the
Storlet is dependent on some Java library, that library can be uploaded as a
dependency of the Storlet. It is assumed that Storlet dependencies are small
(on the order of few MBs), heavier dependencies should be part of the Docker
image. We describe below how to deploy a storlet and its dependencies.
Once uploaded, the Storlet can be invoked in several ways (depending on
how the Storlet is written):
#. As part of a GET, where the object appearing in the GET request is the
#. As part of a GET, where the object appearing in the GET request is the
storlet's input and the response to the GET request is the storlet's output.
The Storlet is also provided with the object user's metadata, and can output
user metadata.
@@ -26,14 +26,14 @@ how the Storlet is written):
and can output user metadata.
To write a storlet you will need the SCommon.jar which is being built as part of
the Storlets build process (see <https://github.com/openstack/storlets/blob/master/doc/source/dev_and_test_guide.rst>). Import the .jar to a Java
the Storlets build process (see <https://github.com/openstack/storlets/blob/master/doc/source/dev_and_test_guide.rst>). Import the .jar to a Java
project in Eclipse and implement the com.ibm.storlet.common.IStorlet interface.
The interface has a single method that looks like this:
::
public void invoke(ArrayList<StorletInputStream> inStreams,
ArrayList<StorletOutputStream> outStreams,
ArrayList<StorletOutputStream> outStreams,
Map<String,String> parameters, StorletLogger logger) throws StorletException;
Here is a class diagram illustrating the classes involved in the above API.
@@ -44,24 +44,24 @@ Here is a class diagram illustrating the classes involved in the above API.
:scale: 50 %
:alt: Programming Model Class Diagram
:align: center
#. The StorleInputStream is used to stream object's data into the storlet.
An instance of the class is provided whenever the Storlet gets an object as
an input. Practically, It is used in the GET and PUT scenarios to
stream in the object's data and metadata. To consume the data do a getStream()
to get a java.io.InputStream on which you can just read(). To consume the
an input. Practically, it is used in the GET and PUT scenarios to
stream in the object's data and metadata. To consume the data call getStream()
to get a java.io.InputStream on which you can just read(). To consume the
metadata call the getMetadata() method.
#. The StorleOutputStream is a base class for the StorletObjectOutputStream.
The actual instance ever received by the storlet will be StorletObjectOutputStream
The base class serves for a type of invocation that was removed for the interest
The actual instance received by the storlet will always be StorletObjectOutputStream,
the base class serves for a type of invocation that was removed for the interest
of code simplicity.
#. StorletObjectOutputStream. In the PUT and GET scenarios the storlet is
called with an instance of this class.
- Use the setMetadata method to set the Object's metadata.
- Use getStream to get a java.io.OutputStream on which you can just write()
the content of the object.
- Notice that setMetadata must be called. Also it must be called before
- Notice that setMetadata must be called. Also it must be called before
writing the data. Additional guidelines on using StorletObjectOutputStream
are given below.
#. StorletLogger. The StorletLogger class supports a single method called emitLog,
@@ -115,17 +115,17 @@ Musts
#. With the current implementation, a storlet must start to respond within 40
seconds of invocation. Otherwise, Swift would timeout. Moreover, the Storlet
must output something every 40 seconds so as not to timeout. This is a
mechanism to ensure that the Storlet code does not get stuck.Note that
mechanism to ensure that the Storlet code does not get stuck. Note that
outputting an empty string does not do the job in terms of resetting the 40
seconds timeout.
#. For StorletObjectOutputStream, the call to setMetadata must happen before the
storlet starts streaming out the output data. Note the applicability of the 40
seconds timeout here as well.
#. The total size of metadata given to setMetadata (when serialized as a string)
#. The total size of metadata given to setMetadata (when serialized as a string)
should not exceed 4096 Bytes
#. While Swift uses the prefix X-Object-Meta to specify that a certain header
#. While Swift uses the prefix X-Object-Meta to specify that a certain header
reflects a metadata key, the key itself should not begin with that prefix.
More specifically, metadata keys passed to setMetadata should not have that
More specifically, metadata keys passed to setMetadata should not have that
prefix (unless this is really part of the key)
Recommendations
@@ -136,10 +136,10 @@ Recommendations
of the content of an object is not a good example for a storlet as it requires
to read all the content into memory (random reads are not an option as the
input is provided as a stream). While we currently do not employ any restrictions
on the CPU usage or memory consumption of the storlet, reading large object
on the CPU usage or memory consumption of the storlet, reading large object
into memory or doing very intensive computations would have impact on the overall
system performance.
#. While this might be obvious it is advisable to test the storlet prior to its
deployment.
@@ -147,7 +147,7 @@ Tips
====
#. The storlets are executed in an open-jdk 8 environment. Thus, any dependencies
that the storlet code requires which are outside of open-jdk 8 should be
that the storlet code requires which are outside of open-jdk 8 should be
stated as storlet dependencies and uploaded with the storlet. Exact details
are found in the deployment section below.
@@ -155,9 +155,9 @@ Tips
as well as the dependencies are kept inside the Linux container. One reason
may be the need to invoke a binary dependency. To get that path use the
following code:
::
// Get the path of this class image
String strJarPath = StorletUtils.getClassFolder(this.getClass());
@@ -198,11 +198,11 @@ section below.
Storlet deployment is essentially uploading the storlet and its dependencies to
designated containers in the account we are working with. While a storlet and a
dependency are regular Swift objects, they must carry some metadata used by the
dependency are regular Swift objects, they must carry some metadata used by the
storlet engine. When a storlet is first executed, the engine fetches the necessary
objects from Swift and puts them is a directory accessible by the Docker container.
Note that the dependencies are meant to be small. Having a large list of dependencies
or a very large dependency may result in a timeout on the first attempt to execute a
or a very large dependency may result in a timeout on the first attempt to execute a
storlet. If this happens, just re-send the request again.
We consider two types of dependencies: libraries and executables. libraries would
@@ -211,43 +211,43 @@ have a binary dependency, that the storlet code can execute.
Following the Identity storlet example, we have 2 objects to upload:
#. The storlet packaged in a .jar. In our case the jar was named:
identitystorlet-1.0.jar The jar needs to be uploaded to a container named
#. The storlet packaged in a .jar. In our case the jar was named:
identitystorlet-1.0.jar The jar needs to be uploaded to a container named
storlet. The name of the uploaded storlet must be of the form <name>-<version>.
The metadata that must accompany a storlet is as follows:
::
X-Object-Meta-Storlet-Language - currently must be 'java'
X-Object-Meta-Storlet-Interface-Version - currenltly we have a single version '1.0'
X-Object-Meta-Storlet-Dependency - A comma separated list of dependencies. In our case: 'get42'
X-Object-Meta-Storlet-Object-Metadata - Currently, not in use, but must appear. Use the value 'no'
X-Object-Meta-Storlet-Main - The name of the class that implements the IStorlet API. In our case: 'com.ibm.storlet.identity.IdentityStorlet'
#. The binary file that the storlet code is dependent on. In our case it is a
binary called get42. The binary should be uploaded to a container named
dependency. The dependency metadata fields appear below. Note the permissions
header. This header is required so that the engine will chmod it accordingly
when placed in the container so that the storlet would be able to execute it.
::
X-Object-Meta-Storlet-Dependency-Version - While the engine currently does not parse this header, it must appear.
X-Object-Meta-Storlet-Dependency-Permissions - An optional metadata field, where the user can state the permissions
given to the dependency when it is copied to the Linux container. This is helpful for binary dependencies invoked by the
X-Object-Meta-Storlet-Dependency-Permissions - An optional metadata field, where the user can state the permissions
given to the dependency when it is copied to the Linux container. This is helpful for binary dependencies invoked by the
storlet. For a binary dependency once can specify: '0755'
If one wishes to update the storlet just upload again, the engine would recognize
the update and bring the updated code.
Important: Currently, dependency updates are not recognized, only the Storlet
Important: Currently, dependency updates are not recognized, only the Storlet
code itself can be updated.
Deploying a Storlet using Swift Client
======================================
When using the Swift client one needs to provide the credentials, as well as the
authentication URI. The credentials can be supplied either via environment
authentication URI. The credentials can be supplied either via environment
variables or via command line parameters. To make the commands more readable I
have used environment variables:
@@ -279,7 +279,7 @@ Here is the Swift client command for uploading the storlet. some notes:
-H "X-Object-Meta-Storlet-Main:com.ibm.storlet.identity.IdentityStorlet" \
-H "X-Object-Meta-Storlet-Dependency:get42"
Here is the Swift client command for uploading the get42 dependency. Again,
Here is the Swift client command for uploading the get42 dependency. Again,
some notes:
#. The container name used here is the first parameter for the upload command and is 'dependency'.
@@ -297,9 +297,9 @@ Deploying a Storlet with Python
Here is a code snippet that uploads both the storlet as well as the dependencies.
The code assumes v2 authentication, and was tested against a Swift cluster with:
#. Keystone configured with a 'service' account, having a user 'swift' whose
#. Keystone configured with a 'service' account, having a user 'swift' whose
password is 'passw0rd'
#. Under the service account there are already 'storlet', 'dependency', and
#. Under the service account there are already 'storlet', 'dependency', and
'storletlog' containers.
::