Adjusting the storlet writing and deploying guide to include python

The guide is broken into three parts as follows:
A language independent guide and two language specific
complimentary guides.

While at it the programming model UML was
simplified to include only what the storlet
writer should care about.

Bonus 1: Adjusted all docs to use the same heading
         format
Bonus 2: Moved to archive two outdated .rst to archive

Change-Id: I945f410aa96af1895c66856537ff392d5c18be25
This commit is contained in:
Eran Rom 2016-09-29 19:33:49 +03:00
parent 44148e2adc
commit cda9c650bf
22 changed files with 320 additions and 214 deletions

@ -1,4 +1,3 @@
===============
Storlets API v1
===============
The storlets API is an extension of the Swift API. In some cases we use
@ -234,7 +233,6 @@ Use the 'X-Storlet-Run-On-Proxy' header to enforce the engine to invoke the stor
'X-Storlet-Run-On-Proxy': ''
=============
Storlets ACLs
=============

@ -1,4 +1,3 @@
============
Introduction
============
The Swift account manager can supply a Docker image in which the account's storlets
@ -15,7 +14,6 @@ following:
The idea is that a user supplied docker image would contain dependencies
required by storlets in the form of libraries.
============
Logical Flow
============
The flow of deploying an account manager tailored Docker image to Swift involves
@ -39,7 +37,7 @@ Below are the steps of this flow:
The sections below describe in detail the steps taken by the account manager.
Downloading the Docker Image
============================
----------------------------
Downloading the Docker image involves a simple retrieval of a Swift object. To
get the exact name of the object just list the docker_images container in the
account. The name will carry the base OS system and engine language binding run
@ -93,7 +91,7 @@ In the below we show:
X-Trans-Id: tx794e21cd40b544e6a377b-00544bfed3
Tuning the Docker Image
=======================
-----------------------
To tune the Docker image, Docker must be used. To install please refer to
http://docs.docker.com/installation/
@ -199,7 +197,7 @@ engine but rather the /bin/bash from which we can run ffmpeg
Uploading the Docker Image
==========================
--------------------------
1. Use docker save to save the image as a tar file:
.. code-block:: bash

@ -1,4 +1,3 @@
============
Contributing
============
.. include:: ../../CONTRIBUTING.rst

@ -1,4 +1,3 @@
====================================
Installing a Development Environment
====================================
This guide gives a step by step installation instructions that are simpler
@ -28,9 +27,8 @@ This process has been tested on Ubuntu 14.04 using Swift 2.7.0.
pip install --upgrade pbr
pip install --upgrade setuptools
-----------------------
Make Swift use Keystone
-----------------------
=======================
SAIO uses tmpauth as an auth middleware. While storlets do not have a
direct dependency on the auth middleware used, Keystone seems to be
@ -139,9 +137,8 @@ Now do:
source .bashrc
swift stat
----------------
Install Storlets
----------------
================
Install Dependencies
--------------------
@ -513,7 +510,6 @@ We use the swift cli as follows:
--os-tenant-name=test \
storletlog
------------------------
Run the functional tests
------------------------
The functional tests upload various storlets and execute them.

@ -1,4 +1,3 @@
=============================
Development and Testing Guide
=============================
@ -10,7 +9,6 @@ the development environment installation instructions_ or in the getting started
.. _instructions: engine_dev_installation.html
.. _guide: getting_started.html
--------
Building
--------
The storlets repository consists of code written in Python, Java and C. We have chose ant to serve as a 'make' tool for all of the code.
@ -32,7 +30,6 @@ The main build task in build.xml is dependent on two other build tasks:
#. The Java SCommon code, which has the storlet interface declaration, as well as the accompanying classes appearing in the interface. This code
is compiled to a .jar that is required both in the Docker image as well as for building storlets.
---------
Deploying
---------
Two additional tasks of interest in our build.xml are the deploy_host_engine and deploy_container_engine.
@ -45,7 +42,6 @@ These tasks are based on the Ansible installation scripts and do the following:
in the cluster_config.json and distribute it to all nodes as defined in
the configuration.
-----------------
Running the Tests
-----------------

@ -1,4 +1,3 @@
===============
Getting Started
===============
@ -6,7 +5,6 @@ This is a TL-DR guide to get you started with Storlet experimentation
as fast as possible.
-------------------
System Requirements
-------------------
@ -14,7 +12,6 @@ Ubuntu Server 14.04 with total disk of 10Gb.
A disposal VM is always a good idea.
-------------------
Installation Guide
-------------------
@ -22,7 +19,6 @@ Installation Guide
For more information please refer to `"s2aio" <http://storlets.readthedocs.io/en/latest/s2aio.html>`__.
---------------------------------------
Writing Deploying and Running a Storlet
---------------------------------------

Binary file not shown.

Before

(image error) Size: 56 KiB

After

(image error) Size: 49 KiB

Binary file not shown.

After

(image error) Size: 29 KiB

@ -14,7 +14,6 @@
License for the specific language governing permissions and limitations
under the License.
===================================
Welcome to storlets' documentation!
===================================
@ -67,6 +66,8 @@ Storlets Developers
:maxdepth: 1
writing_and_deploying_storlets
writing_and_deploying_java_storlets
writing_and_deploying_python_storlets
Storlets Deployers
==================

@ -1,9 +1,8 @@
=====================
Storlets Installation
=====================
Background
==========
----------
Storlets installation (and build) assume an existing Swift cluster that works with Keystone.
The installation consists of the following components:
@ -36,7 +35,7 @@ At a high level the installation consists of the following steps:
#. Install a set of storlets management scripts. Done on a designated node having a storlet management role.
hosts file example
==================
------------------
::
@ -66,7 +65,7 @@ Few notes:
#. It is recommended that the memcache servers on the proxy servers would be accessible from the storage nodes.
common.yml
==========
----------
Below are the entries of interest of common.yml
Special attention should be given to the swift user and group IDs. Make sure they are identical on all hosts and match what is defined in the common.yml file.
@ -109,7 +108,7 @@ storlets with EC and encryption. Valid values are true / false
storlet_execute_on_proxy_only: false
Install
=======
-------
to perform the installation follow these steps:
#. Create a hosts file as described above

@ -1,4 +1,3 @@
=====
s2aio
=====

@ -1,15 +1,10 @@
=======================
Storlet Engine Overview
=======================
----------
Components
----------
At the high level the storlet engine is made of the components described below. See illustration.
The storlet middleware
======================
----------------------
The storlet middleware is a Swift WSGI middleware that intercepts storlet invocation
requests and routes the input data and the computation output into and out of the Docker
container where the storlet is executed. This middleware needs to be in both the
@ -24,7 +19,7 @@ Moreover, part of the storlet middleware configuration is what
class implementation of the API we refer to as "storlet docker gateway".
Swift accounts
==============
--------------
The storlet engine is tightly coupled with accounts in Swift in the following manners:
#. In order to invoke a storlet on a data object residing in some Swift account, that account
@ -40,7 +35,7 @@ The storlet engine is tightly coupled with accounts in Swift in the following ma
accounts. The Docker image name must be the account id to which it belongs.
The Docker image
================
----------------
As mentioned above there is a Docker image per account that is enabled for storlets.
At a high level this image containes:
@ -58,14 +53,13 @@ At a high level this image containes:
it has the definition of the invoke API the storlet must implement.
The storlet bus
===============
---------------
The storlet bus is a communication channel between the storlet middleware in the Swift side
and the factory daemon and storlet daemon in the Docker container.
For each Docker container (or Swift account) there is a communication channel with the storlet factory of that container.
For each storlet daemon in the container there is a communication channel on which is listens for invocations. These channels are
based on unix domain sockets.
------------------------------------------
The storlet engine components illustration
------------------------------------------
@ -75,14 +69,13 @@ The storlet engine components illustration
:scale: 50
:alt: alternate text
----
Flow
----
====
To tie everything together we illustrate an end-to-end scenario.
Writing and Deploying a storlet
===============================
-------------------------------
The flow begins with writing a storlet followed by deploying it.
writing and deploying a storlet is covered in the writing and
deploying storlets guide_.
@ -90,7 +83,7 @@ deploying storlets guide_.
.. _guide: writing_and_deploying_storlets.html
Invoking a Storlet
==================
------------------
A Storlet can be invoked on object download, upload or copy operations (GET, PUT, and COPY respectively).
For the flow description lets assume that we wish to invoke the
storlet on an object download. This involves doing a Swift GET request with the
@ -98,7 +91,7 @@ additional header "X-Run-Storlet" which specifies the storlet to invoke,
e.g. "X-Run-Storlet: compress-1.0.jar".
Handling the request at the proxy server
========================================
----------------------------------------
Seeing the "X-Run-Storlet" header the storlert_middleware at the proxy intercepts
the request and performs a HEAD on the storlet specified by the user.
This HEAD operation facilitates:
@ -115,7 +108,7 @@ with the request being routed to an object server that holds a replica of the ob
specified in the GET uri.
Handling the request at the object server
=========================================
-----------------------------------------
Seeing the "X-Run-Storlet" header the storlert_middleware at the object server intecepts
the request and perform the following two phased flow:

@ -1,11 +1,9 @@
====================
Storlets Terminology
====================
The overall storlets mechanism involves a lot of moving parts as well as
poeple or roles involved throughout the usage of the system. We give below
a definition of the various terms used throughout the documentation.
-------
Storlet
-------
A storlet is the binary code deployed as a Swift object. Invoking a storlet
@ -26,7 +24,6 @@ In a nutshell this interface consists of an input stream, an output stream and a
The storlet is assumed to read from the input stream, do its thing, and write the result
to the output stream.
------------------
Storlet Invocation
------------------
A storlet invocation is a user request wishing to perform a storlet computation
@ -43,7 +40,6 @@ different ways:
data object, where the storlet's output is kept in a newly crerated object. In a regular
Swift copy the newly created object is a identical to the source object.
------------------
The Storlet Engine
------------------
The storlet engine is the underlying mechanism that can take a storlet as a Swift object
@ -52,9 +48,8 @@ In a nutshell the engine intercepts invocation requests, route the input data st
the storlet and receives back the storlet output stream. The engine is implemented as a Swift
middleware.
-----
Roles
-----
=====
Storlet Developer
-----------------

@ -1,4 +1,3 @@
========================
The Chip Bakers Use Case
========================

@ -1,4 +1,3 @@
==================================
Queriable "Secondary Storage" Data
==================================
It is said that the primary use case for object stores is to serve as secondary

@ -1,4 +1,3 @@
============
Data Privacy
============

@ -0,0 +1,104 @@
Java Storlet Writing and Deploying Guide
========================================
This is the Java specific storlet writing and deploying guide. This guide complements
the more general_ guide for writing and deploying storlets which should be read first.
To write a Java storlet you will need the SCommon.jar which is being built as part of
the storlets build process as described in the development and testing guide_.
.. _guide: engine_dev_tests.html
.. _general: writing_and_deploying_storlets.html
Import the .jar to a Java project in Eclipse and implement the
org.openstack.storlet.common.IStorlet interface.
The interface has a single method that looks like this:
::
public void invoke(ArrayList<StorletInputStream> inStreams,
ArrayList<StorletOutputStream> outStreams,
Map<String,String> parameters, StorletLogger logger) throws StorletException;
Below is a class diagram illustrating the classes involved in the above API.
.. image:: images/java_prog_model.jpg
:height: 960px
:width: 1216 px
:scale: 50 %
:alt: Java Programming Model Class Diagram
:align: center
#. The StorleInputStream is used to stream object's data into the storlet.
An instance of the class is provided whenever the Storlet gets an object as
an input. Practically, it is used in all storlet invocation scenarios to
stream in the object's data and metadata. To consume the data call getStream()
to get a java.io.InputStream on which you can just read(). To consume the
metadata call the getMetadata() method.
#. StorletObjectOutputStream. In all invocation scenarios the storlet is
called with an instance of this class.
- Use the setMetadata method to set the Object's metadata.
- Use getStream to get a java.io.OutputStream on which you can just write()
the content of the object.
- Notice that setMetadata must be called. Also, it must be called before
writing the data. Additional guidelines on using StorletObjectOutputStream
are given below.
#. StorletLogger. The StorletLogger class supports a single method called emitLog,
and accepts a String. The storlet logs are written to the host machine.
When invoked via the Swift REST API the invoke method
will be called as follows:
#. The inStreams array would include one or more element(s) of type StorleInputStream
representing the object appearing in the request's URI (and possibly extra resources).
#. The outStreams would include a single element of type StorleObjectOutputStream
representing the response returned to the user.
#. The parameters map includes execution parameters sent. These parameters can be
specified in the storlet execution request.
#. A StorletLogger instance.
Tips
----
#. The storlets are executed in an open-jdk 8 environment. Thus, any dependencies
that the storlet code requires which are outside of open-jdk 8 should be
stated as storlet dependencies and uploaded with the storlet. Exact details
are found in the deployment section below.
#. In some cases the storlet may need to know the path where the storlet .jar
as well as the dependencies are kept inside the Linux container. One reason
may be the need to invoke a binary dependency. To get that path use the
following code:
::
// Get the path of this class image
String strJarPath = StorletUtils.getClassFolder(this.getClass());
Deploying a Java Storlet
------------------------
Below are specific guidelines for deploying a Java storlet:
#. The compiled class that implements the storlet needs to be wrapped in a .jar.
This jar must not include the SCommon.jar.
#. Any jars that the class implementation is dependent on should be uploaded as separate jars.
#. The name of the Java storlet being uploaded must be of the form <name>-<version>, e.g.
identitystorlet-1.0.jar
#. The 'X-Object-Meta-Storlet-Main' metadata key shold be the name of the class implementing
the IStorlet interface. E.g. "org.openstack.storlet.identity.IdentityStorlet"
Deploying a Java Dependency
---------------------------
#. A java dependency must be either a .jar or an executable

@ -0,0 +1,83 @@
Python Storlet Writing and Deployment Guideline
===============================================
This is the Python specific storlet writing and deploying guide. This guide complements
the more general_ guide for writing and deploying storlets which should be read first.
.. _general: writing_and_deploying_storlets.html
A python module implementing a storlet looks like this:
::
class <Class name>(object):
def __init__(self, logger):
self.logger = logger
def __call__(self, in_files, out_files, params):
"""
The function called for storlet invocation
:param in_files: a list of StorletInputFile
:param out_files: a list of StorletOutputFile
:param params: a dict of request parameters
"""
Below is a class diagram illustrating the classes behind the in_files, out_files, and logger.
The diagram lists only the methods that the storlet writer is expected to work with.
.. image:: images/python_prog_model.jpg
:height: 960px
:width: 1216 px
:scale: 50 %
:alt: Python Programming Model Class Diagram
:align: center
#. The StorletInputFile is used to stream object's data into the storlet.
StorletInputFile has the same read methods as python FileObject.
Trying to write to a StorletInputFile yields NotImplemented error.
Whenever a storlet is invoked, an instance of this class is provided.
To consume the metadata call the StorletInputFile.get_metadata method.
#. The StorleOutputFile is used for writing the storlet output.
StorletOutputFile has the same write methods as python FileObject.
Trying to read from a StorletOutputFile yields NotImplemented error.
Whenever a storlet is invoked, an instance of this class is provided.
Use the StorletInputFile.set_metadata method to set the Object's metadata.
Note that the storlet must call the StorletInputFile set_metadata method.
Moreowver, StorletInputFile.set_metadata must be called before writing
the data.
#. StorletLogger. The StorletLogger class implements the same log methods as the
Python logger.
When invoked via the Swift REST API the __call__ method
will be called as follows:
#. The in_files list would include one or more element(s) of type StorleInputFile
representing the object appearing in the request's URI (and possibly extra resources).
#. The out_files would include a single element of type StorleOutputFile
representing the response returned to the user.
#. The parameters is a dictionary with the execution parameters sent. These parameters can be
specified in the storlet execution request.
#. A StorletLogger instance.
Deploying a Python Storlet
--------------------------
Below are specific guidelines for deploying a Python storlet:
#. The object name of the python module containing the storlet class implemetation
must end with .py
#. Any python modules that the class implementation is dependent on should be uploaded as separate .py(s).
#. The 'X-Object-Meta-Storlet-Main' metadata key shold be of the form: <module_name>.<class_name>. For example,
if the storlet name is SimpleStorlet and it resides in simple_storlet.py, then the
'X-Object-Meta-Storlet-Main' metadata key shold be "simple_storlet.SimpleStorlet"
Deploying a Python Dependency
-----------------------------
#. Currently, there is no limitation as to what is being uploaded as a dependency.

@ -1,106 +1,50 @@
===================================
Storlet writing and deploying guide
===================================
Currently, storlets must be written in Java. Writing a storlet involves
implementing a single method interface and following some simple rules and best
practices described below.
Storlets can be written either in Java or in Python. This guide
is a language independent starting point for writing
and deploying storlets. The Java_ and Python_ specific guides
complement this guide and should be read next after this one.
.. _Java: writing_and_deploying_java_storlets.html
.. _Python: writing_and_deploying_python_storlets.html
Writing a storlet involves implementing a single function referred to
as invoke (the exact name varies between languages).
Typically, the invoke operation gets an input stream and an
output stream. The input stream will contain the data either being uploaded
(in case the storlet is invoked during upload) or being downloaded (when
the invocation is done during download). The input stream is accompanied with
metadata that reflects the Swift user defined metadata. The output stream
exposes a way to write back not only the storlet output data, but also the
metadata. The input and output streams are exposed to the storlet code
in a way that is native to the language being used.
Once the storlet is written and tested it can be uploaded as an object to a
designated container (called 'storlet' by default). In addition in case the
storlet is dependent on some Java library, that library can be uploaded as a
dependency of the storlet. It is assumed that storlet dependencies are small
(on the order of few MBs), heavier dependencies should be part of the Docker
image.
designated container (called 'storlet' by default). Storlet code external
dependencies such as Java libraries or Python modules can be uploaded as well.
It is assumed, however, that storlet dependencies are relatively small
(on the order of few MBs). Heavier dependencies should be part of the Docker
image. Deploying storlets and dependencies are covered below.
To write a storlet you will need the SCommon.jar which is being built as part of
the storlets build process as described in the development and testing guide_
There are various implementations of storlets in the StorletSamples/java
and StorletSamples/python directories. These storlets are used by the engine's
functional tests.
.. _guide: engine_dev_tests.html
The next two sections describe storlet writing and deploying guidelines that are
indpendent of the language used.
Import the .jar to a Java project in Eclipse and implement the
org.openstack.storlet.common.IStorlet interface.
The interface has a single method that looks like this:
::
public void invoke(ArrayList<StorletInputStream> inStreams,
ArrayList<StorletOutputStream> outStreams,
Map<String,String> parameters, StorletLogger logger) throws StorletException;
Below is a class diagram illustrating the classes involved in the above API.
.. image:: images/java_prog_model.jpg
:height: 960px
:width: 1216 px
:scale: 50 %
:alt: Programming Model Class Diagram
:align: center
#. The StorleInputStream is used to stream object's data into the storlet.
An instance of the class is provided whenever the Storlet gets an object as
an input. Practically, it is used in all storlet invocation scenarios to
stream in the object's data and metadata. To consume the data call getStream()
to get a java.io.InputStream on which you can just read(). To consume the
metadata call the getMetadata() method.
#. The StorleOutputStream is a base class for the StorletObjectOutputStream.
The actual instance received by the storlet will always be StorletObjectOutputStream.
#. StorletObjectOutputStream. In all invocation scenarios the storlet is
called with an instance of this class.
- Use the setMetadata method to set the Object's metadata.
- Use getStream to get a java.io.OutputStream on which you can just write()
the content of the object.
- Notice that setMetadata must be called. Also, it must be called before
writing the data. Additional guidelines on using StorletObjectOutputStream
are given below.
#. StorletLogger. The StorletLogger class supports a single method called emitLog,
and accepts a String. The storlet logs are written to the host machine.
When invoked via the Swift REST API the invoke method
will be called as follows:
#. The inStreams array would include a single element of type StorleInputStream
representing the object appearing in the request's URI.
#. The outStreams would include a single element of type StorleObjectOutputStream
representing the response returned to the user.
#. The parameters map includes execution parameters sent. These parameters can be
specified in the storlet execution request.
#. A StorletLogger instance.
==========================
Storlet Writing Guidelines
==========================
Below are some guidelines for writing a storlet. Some of them are musts, some are
recommendations, and some are tips.
Independent of the language used for writing a storlet, there are several guidelines
to follow. Some of them are musts and some are recommendations.
-----
Musts
-----
#. The storlet code must be thread safe and re-enterant. The invoke method will
be called many times and potentially in parallel.
#. Once the storlet has finished writing the response, it is important to close
the output stream. Failing to do so will result in a timeout. Specifically,
close the java.io.OutputStream obtained from the call to getStreasm()
#. With the current implementation, a storlet must start to respond within 40
seconds of invocation. Otherwise, Swift would timeout. Moreover, the storlet
must output something every 40 seconds so as not to timeout. This is a
mechanism to ensure that the storlet code does not get stuck. Note that
outputting an empty string does not do the job in terms of resetting the 40
seconds timeout.
#. For StorletObjectOutputStream, the call to setMetadata must happen before the
storlet starts streaming out the output data. Note the applicability of the 40
seconds timeout here as well.
#. The total size of metadata given to setMetadata (when serialized as a string)
should not exceed 4096 Bytes
#. While Swift uses the prefix X-Object-Meta to specify that a certain header
reflects a metadata key, the key itself should not begin with that prefix.
More specifically, metadata keys passed to setMetadata should not have that
prefix (unless this is really part of the key)
---------------
Recommendations
---------------
#. Storlets are tailored for stream processing, that is, process the input as it
#. Storlets are tailored for stream processing, that is, they process the input as it
is read and produce output while still reading. In other words a 'merge sort'
of the content of an object is not a good example for a storlet as it requires
to read all the content into memory (random reads are not an option as the
@ -109,103 +53,103 @@ Recommendations
into memory or doing very intensive computations would have impact on the overall
system performance.
#. While this might be obvious it is advisable to test the storlet prior to its
deployment.
#. While this might be obvious make sure to unit test your storlet prior to deploying it.
----
Tips
----
#. The storlets are executed in an open-jdk 8 environment. Thus, any dependencies
that the storlet code requires which are outside of open-jdk 8 should be
stated as storlet dependencies and uploaded with the storlet. Exact details
are found in the deployment section below.
Musts
-----
#. In some cases the storlet may need to know the path where the storlet .jar
as well as the dependencies are kept inside the Linux container. One reason
may be the need to invoke a binary dependency. To get that path use the
following code:
#. The storlet code must be thread safe and re-enterant. The invoke method will
be called many times and potentially in parallel.
::
#. Once the storlet has finished writing the response, it is important to close
the output stream. Failing to do so will result in a timeout.
// Get the path of this class image
String strJarPath = StorletUtils.getClassFolder(this.getClass());
#. A storlet must start to respond within 40 seconds of invocation. Otherwise,
Swift would timeout. Moreover, the storlet must output something every 40 seconds
so as not to timeout. This is a mechanism to ensure that the storlet code does not
get stuck. Note that outputting an empty string does not do the job in terms of
resetting the 40 seconds timeout.
----------------
Storlet Examples
----------------
There are various implementations of storlets in the StorletSamples
directory. These are used in the engine's functional tests
In the below example, we will be using the identity storlet
found under StorletSamples/IdentityStorlet
#. The storlet must write metadata to the output stream, and must do so before it
starts streaming out the data. A typical implementation would read the
input metadata and use it as a basis for the metadata being written.
Note the applicability of the 40 seconds timeout here as well.
=======================
How to Deploy a storlet
=======================
In this paragraph we cover:
#. The total size of metadata that can be set (when serialized as a string)
must not exceed 4096 Bytes
#. The principles behind storlet deployment, plus examples.
#. A Swift client example for uploading a storlet.
#. A python example for uploading a storlet.
#. While Swift uses the prefix X-Object-Meta to specify that a certain header
reflects a metadata key, the key itself should not begin with that prefix.
More specifically, metadata keys being set by the storlet should not have that
prefix (unless this is really part of the key)
-----------------------------
Storlet Deployment Principles
-----------------------------
The compiled class that implements the storlet needs to be wrapped in a .jar.
This jar must not include the SCommon.jar. Any jars that the class implementation
is dependent on should be uploaded as separate jars as shown in the deployment
section below.
Storlet Deployment Guidelines
=============================
Storlet deployment is essentially uploading the storlet and its dependencies to
designated containers in the account we are working with. While a storlet and a
designated containers in the account you are working with. While a storlet and a
dependency are regular Swift objects, they must carry some metadata used by the
storlet engine. When a storlet is first executed, the engine fetches the necessary
objects from Swift and puts them is a directory accessible by the Docker container.
objects from Swift and puts them in a directory accessible to the Docker container.
Note that the dependencies are meant to be small. Having a large list of dependencies
or a very large dependency may result in a timeout on the first attempt to execute a
storlet. If this happens, just re-send the request again.
We consider two types of dependencies: libraries and executables. libraries would
typically be .jar files the storlet code is dependent on. Alternatively, one can
have a binary dependency, that the storlet code can execute.
We support two types of dependencies:
Following the Identity storlet example, we have 2 objects to upload:
#. External libraries or modules that
are native to the storlet langauge
#. The storlet packaged in a .jar. In our case the jar was named:
identitystorlet-1.0.jar The jar needs to be uploaded to a container named
storlet. The name of the uploaded storlet must be of the form <name>-<version>.
The metadata that must accompany a storlet is as follows:
#. Executables dependency that the storlet code
can execute.
Storlet Object Metadata
-----------------------
Uploading a storlet must be done to a designated container, called by default "storlet". The uploaded object
must carry the following metadata. See the specific langauge guides for more information.
::
X-Object-Meta-Storlet-Language - currently must be 'java'
X-Object-Meta-Storlet-Language - must be 'python' or 'java'
X-Object-Meta-Storlet-Interface-Version - currenltly we have a single version '1.0'
X-Object-Meta-Storlet-Dependency - A comma separated list of dependencies. In our case: 'get42'
X-Object-Meta-Storlet-Object-Metadata - Currently, not in use, but must appear. Use the value 'no'
X-Object-Meta-Storlet-Main - The name of the class that implements the IStorlet API. In our case: 'org.openstack.storlet.identity.IdentityStorlet'
X-Object-Meta-Storlet-Main - The name of the class that implements the invoke operation
#. The binary file that the storlet code is dependent on. In our case it is a
binary called get42. The binary should be uploaded to a container named
dependency. The dependency metadata fields appear below. Note the permissions
header. This header is required so that the engine will chmod it accordingly
when placed in the container so that the storlet would be able to execute it.
Optional metadata item is:
::
::
X-Object-Meta-Storlet-Dependency-Version - While the engine currently does not parse this header, it must appear.
X-Object-Meta-Storlet-Dependency-Permissions - An optional metadata field, where the user can state the permissions
given to the dependency when it is copied to the Linux container. This is helpful for binary dependencies invoked by the
storlet. For a binary dependency once can specify: '0755'
X-Object-Meta-Storlet-Dependency - A comma separated list of dependencies.
If one wishes to update the storlet just upload again, the engine would recognize
the update and bring the updated code.
Important: Currently, dependency updates are not recognized, only the Storlet
code itself can be updated.
Dependency Object Metadata
--------------------------
Uploading a dependency must be done to a designated container, called by default "dependency". The uploaded object
must carry the following metadata.
--------------------------------------
Deploying a Storlet using Swift Client
--------------------------------------
::
X-Object-Meta-Storlet-Dependency-Version - While the engine currently does not parse this header, it must appear.
Optional metadata item is:
::
X-Object-Meta-Storlet-Dependency-Permissions - The permissions given to the dependency when it is copied to the
Docker container. This is helpful for binary dependencies invoked by the storlet.
For a binary dependency once can specify: '0755'
.. note:: Currently, dependency updates are not recognized.
Deploying a Storlet using Swift Client cli
------------------------------------------
We show below how to deploy a storlet using the Swift client cli.
The example uses a Java storlet. The differences from deploying a
Python storlet are minor and we highlight them where required.
When using the Swift client one needs to provide the credentials, as well as the
authentication URI. The credentials can be supplied either via environment
@ -243,10 +187,15 @@ Here is the Swift client command for uploading the storlet. some notes:
-H "X-Object-Meta-Storlet-Main:org.openstack.storlet.identity.IdentityStorlet" \
-H "X-Object-Meta-Storlet-Dependency:get42"
.. note:: When deploying a Python storlet the name of the object (identitystorlet-1.0.jar in the above) has a different format
Otherwise, "X-Object-Meta-Storlet-Language" is "Python", and "X-Object-Meta-Storlet-Main" has a different format.
Please refer to Python_ for the exact details.
Here is the Swift client command for uploading the get42 dependency. Again,
some notes:
#. The container name used here is the first parameter for the upload command and is 'dependency'.
#. We use the optional permissions header as this is a binary .
::
@ -255,9 +204,8 @@ some notes:
-H "X-Object-Meta-Storlet-Dependency-Version:1.0" \
-H "X-Object-Meta-Storlet-Dependency-Permissions:0755"
-------------------------------
Deploying a Storlet with Python
-------------------------------
Deploying a Storlet using the Python Swift Client
-------------------------------------------------
Here is a code snippet that uploads both the storlet as well as the dependencies.
The code assumes v2 authentication, and was tested against a Swift cluster with:
@ -267,6 +215,10 @@ The code assumes v2 authentication, and was tested against a Swift cluster with:
#. Under the service account there are already 'storlet' and 'dependency'
containers.
The example uses a Java storlet. The differences from deploying a
Python storlet are minor and are the same as the differences highlighted
in the deployment using Swift client section above.
::
from swiftclient import client