Adjusting the storlet writing and deploying guide to include python
The guide is broken into three parts as follows:
A language independent guide and two language specific
complimentary guides.
While at it the programming model UML was
simplified to include only what the storlet
writer should care about.
Bonus 1: Adjusted all docs to use the same heading
format
Bonus 2: Moved to archive two outdated .rst to archive
Change-Id: I945f410aa96af1895c66856537ff392d5c18be25
This commit is contained in:
Eran Rom
parent
44148e2adc
commit
cda9c650bf
doc/source
api
archive
contributing.rstengine_dev_installation.rstengine_dev_tests.rstgetting_started.rstimages
index.rstinstallation.rsts2aio.rststorlet_engine_overview.rststorlets_terminology.rstusecases
writing_and_deploying_java_storlets.rstwriting_and_deploying_python_storlets.rstwriting_and_deploying_storlets.rst@ -1,4 +1,3 @@
|
||||
===============
|
||||
Storlets API v1
|
||||
===============
|
||||
The storlets API is an extension of the Swift API. In some cases we use
|
||||
@ -234,7 +233,6 @@ Use the 'X-Storlet-Run-On-Proxy' header to enforce the engine to invoke the stor
|
||||
|
||||
'X-Storlet-Run-On-Proxy': ''
|
||||
|
||||
=============
|
||||
Storlets ACLs
|
||||
=============
|
||||
|
||||
|
||||
@ -1,4 +1,3 @@
|
||||
============
|
||||
Introduction
|
||||
============
|
||||
The Swift account manager can supply a Docker image in which the account's storlets
|
||||
@ -15,7 +14,6 @@ following:
|
||||
The idea is that a user supplied docker image would contain dependencies
|
||||
required by storlets in the form of libraries.
|
||||
|
||||
============
|
||||
Logical Flow
|
||||
============
|
||||
The flow of deploying an account manager tailored Docker image to Swift involves
|
||||
@ -39,7 +37,7 @@ Below are the steps of this flow:
|
||||
The sections below describe in detail the steps taken by the account manager.
|
||||
|
||||
Downloading the Docker Image
|
||||
============================
|
||||
----------------------------
|
||||
Downloading the Docker image involves a simple retrieval of a Swift object. To
|
||||
get the exact name of the object just list the docker_images container in the
|
||||
account. The name will carry the base OS system and engine language binding run
|
||||
@ -93,7 +91,7 @@ In the below we show:
|
||||
X-Trans-Id: tx794e21cd40b544e6a377b-00544bfed3
|
||||
|
||||
Tuning the Docker Image
|
||||
=======================
|
||||
-----------------------
|
||||
To tune the Docker image, Docker must be used. To install please refer to
|
||||
http://docs.docker.com/installation/
|
||||
|
||||
@ -199,7 +197,7 @@ engine but rather the /bin/bash from which we can run ffmpeg
|
||||
|
||||
|
||||
Uploading the Docker Image
|
||||
==========================
|
||||
--------------------------
|
||||
1. Use docker save to save the image as a tar file:
|
||||
|
||||
.. code-block:: bash
|
||||
@ -1,4 +1,3 @@
|
||||
============
|
||||
Contributing
|
||||
============
|
||||
.. include:: ../../CONTRIBUTING.rst
|
||||
|
||||
@ -1,4 +1,3 @@
|
||||
====================================
|
||||
Installing a Development Environment
|
||||
====================================
|
||||
This guide gives a step by step installation instructions that are simpler
|
||||
@ -28,9 +27,8 @@ This process has been tested on Ubuntu 14.04 using Swift 2.7.0.
|
||||
pip install --upgrade pbr
|
||||
pip install --upgrade setuptools
|
||||
|
||||
-----------------------
|
||||
Make Swift use Keystone
|
||||
-----------------------
|
||||
=======================
|
||||
|
||||
SAIO uses tmpauth as an auth middleware. While storlets do not have a
|
||||
direct dependency on the auth middleware used, Keystone seems to be
|
||||
@ -139,9 +137,8 @@ Now do:
|
||||
source .bashrc
|
||||
swift stat
|
||||
|
||||
----------------
|
||||
Install Storlets
|
||||
----------------
|
||||
================
|
||||
|
||||
Install Dependencies
|
||||
--------------------
|
||||
@ -513,7 +510,6 @@ We use the swift cli as follows:
|
||||
--os-tenant-name=test \
|
||||
storletlog
|
||||
|
||||
------------------------
|
||||
Run the functional tests
|
||||
------------------------
|
||||
The functional tests upload various storlets and execute them.
|
||||
|
||||
@ -1,4 +1,3 @@
|
||||
=============================
|
||||
Development and Testing Guide
|
||||
=============================
|
||||
|
||||
@ -10,7 +9,6 @@ the development environment installation instructions_ or in the getting started
|
||||
.. _instructions: engine_dev_installation.html
|
||||
.. _guide: getting_started.html
|
||||
|
||||
--------
|
||||
Building
|
||||
--------
|
||||
The storlets repository consists of code written in Python, Java and C. We have chose ant to serve as a 'make' tool for all of the code.
|
||||
@ -32,7 +30,6 @@ The main build task in build.xml is dependent on two other build tasks:
|
||||
#. The Java SCommon code, which has the storlet interface declaration, as well as the accompanying classes appearing in the interface. This code
|
||||
is compiled to a .jar that is required both in the Docker image as well as for building storlets.
|
||||
|
||||
---------
|
||||
Deploying
|
||||
---------
|
||||
Two additional tasks of interest in our build.xml are the deploy_host_engine and deploy_container_engine.
|
||||
@ -45,7 +42,6 @@ These tasks are based on the Ansible installation scripts and do the following:
|
||||
in the cluster_config.json and distribute it to all nodes as defined in
|
||||
the configuration.
|
||||
|
||||
-----------------
|
||||
Running the Tests
|
||||
-----------------
|
||||
|
||||
|
||||
@ -1,4 +1,3 @@
|
||||
===============
|
||||
Getting Started
|
||||
===============
|
||||
|
||||
@ -6,7 +5,6 @@ This is a TL-DR guide to get you started with Storlet experimentation
|
||||
as fast as possible.
|
||||
|
||||
|
||||
-------------------
|
||||
System Requirements
|
||||
-------------------
|
||||
|
||||
@ -14,7 +12,6 @@ Ubuntu Server 14.04 with total disk of 10Gb.
|
||||
A disposal VM is always a good idea.
|
||||
|
||||
|
||||
-------------------
|
||||
Installation Guide
|
||||
-------------------
|
||||
|
||||
@ -22,7 +19,6 @@ Installation Guide
|
||||
|
||||
For more information please refer to `"s2aio" <http://storlets.readthedocs.io/en/latest/s2aio.html>`__.
|
||||
|
||||
---------------------------------------
|
||||
Writing Deploying and Running a Storlet
|
||||
---------------------------------------
|
||||
|
||||
|
||||
Binary file not shown.
|
Before 
(image error) Size: 56 KiB After 
(image error) Size: 49 KiB
|
BIN
doc/source/images/python_prog_model.jpg
Normal file
BIN
doc/source/images/python_prog_model.jpg
Normal file
Binary file not shown.
|
After 
(image error) Size: 29 KiB |
Binary file not shown.
@ -14,7 +14,6 @@
|
||||
License for the specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
===================================
|
||||
Welcome to storlets' documentation!
|
||||
===================================
|
||||
|
||||
@ -67,6 +66,8 @@ Storlets Developers
|
||||
:maxdepth: 1
|
||||
|
||||
writing_and_deploying_storlets
|
||||
writing_and_deploying_java_storlets
|
||||
writing_and_deploying_python_storlets
|
||||
|
||||
Storlets Deployers
|
||||
==================
|
||||
|
||||
@ -1,9 +1,8 @@
|
||||
=====================
|
||||
Storlets Installation
|
||||
=====================
|
||||
|
||||
Background
|
||||
==========
|
||||
----------
|
||||
|
||||
Storlets installation (and build) assume an existing Swift cluster that works with Keystone.
|
||||
The installation consists of the following components:
|
||||
@ -36,7 +35,7 @@ At a high level the installation consists of the following steps:
|
||||
#. Install a set of storlets management scripts. Done on a designated node having a storlet management role.
|
||||
|
||||
hosts file example
|
||||
==================
|
||||
------------------
|
||||
|
||||
::
|
||||
|
||||
@ -66,7 +65,7 @@ Few notes:
|
||||
#. It is recommended that the memcache servers on the proxy servers would be accessible from the storage nodes.
|
||||
|
||||
common.yml
|
||||
==========
|
||||
----------
|
||||
Below are the entries of interest of common.yml
|
||||
|
||||
Special attention should be given to the swift user and group IDs. Make sure they are identical on all hosts and match what is defined in the common.yml file.
|
||||
@ -109,7 +108,7 @@ storlets with EC and encryption. Valid values are true / false
|
||||
storlet_execute_on_proxy_only: false
|
||||
|
||||
Install
|
||||
=======
|
||||
-------
|
||||
to perform the installation follow these steps:
|
||||
|
||||
#. Create a hosts file as described above
|
||||
|
||||
@ -1,4 +1,3 @@
|
||||
=====
|
||||
s2aio
|
||||
=====
|
||||
|
||||
|
||||
@ -1,15 +1,10 @@
|
||||
=======================
|
||||
Storlet Engine Overview
|
||||
=======================
|
||||
|
||||
----------
|
||||
Components
|
||||
----------
|
||||
At the high level the storlet engine is made of the components described below. See illustration.
|
||||
|
||||
|
||||
The storlet middleware
|
||||
======================
|
||||
----------------------
|
||||
The storlet middleware is a Swift WSGI middleware that intercepts storlet invocation
|
||||
requests and routes the input data and the computation output into and out of the Docker
|
||||
container where the storlet is executed. This middleware needs to be in both the
|
||||
@ -24,7 +19,7 @@ Moreover, part of the storlet middleware configuration is what
|
||||
class implementation of the API we refer to as "storlet docker gateway".
|
||||
|
||||
Swift accounts
|
||||
==============
|
||||
--------------
|
||||
The storlet engine is tightly coupled with accounts in Swift in the following manners:
|
||||
|
||||
#. In order to invoke a storlet on a data object residing in some Swift account, that account
|
||||
@ -40,7 +35,7 @@ The storlet engine is tightly coupled with accounts in Swift in the following ma
|
||||
accounts. The Docker image name must be the account id to which it belongs.
|
||||
|
||||
The Docker image
|
||||
================
|
||||
----------------
|
||||
As mentioned above there is a Docker image per account that is enabled for storlets.
|
||||
At a high level this image containes:
|
||||
|
||||
@ -58,14 +53,13 @@ At a high level this image containes:
|
||||
it has the definition of the invoke API the storlet must implement.
|
||||
|
||||
The storlet bus
|
||||
===============
|
||||
---------------
|
||||
The storlet bus is a communication channel between the storlet middleware in the Swift side
|
||||
and the factory daemon and storlet daemon in the Docker container.
|
||||
For each Docker container (or Swift account) there is a communication channel with the storlet factory of that container.
|
||||
For each storlet daemon in the container there is a communication channel on which is listens for invocations. These channels are
|
||||
based on unix domain sockets.
|
||||
|
||||
------------------------------------------
|
||||
The storlet engine components illustration
|
||||
------------------------------------------
|
||||
|
||||
@ -75,14 +69,13 @@ The storlet engine components illustration
|
||||
:scale: 50
|
||||
:alt: alternate text
|
||||
|
||||
----
|
||||
Flow
|
||||
----
|
||||
====
|
||||
|
||||
To tie everything together we illustrate an end-to-end scenario.
|
||||
|
||||
Writing and Deploying a storlet
|
||||
===============================
|
||||
-------------------------------
|
||||
The flow begins with writing a storlet followed by deploying it.
|
||||
writing and deploying a storlet is covered in the writing and
|
||||
deploying storlets guide_.
|
||||
@ -90,7 +83,7 @@ deploying storlets guide_.
|
||||
.. _guide: writing_and_deploying_storlets.html
|
||||
|
||||
Invoking a Storlet
|
||||
==================
|
||||
------------------
|
||||
A Storlet can be invoked on object download, upload or copy operations (GET, PUT, and COPY respectively).
|
||||
For the flow description lets assume that we wish to invoke the
|
||||
storlet on an object download. This involves doing a Swift GET request with the
|
||||
@ -98,7 +91,7 @@ additional header "X-Run-Storlet" which specifies the storlet to invoke,
|
||||
e.g. "X-Run-Storlet: compress-1.0.jar".
|
||||
|
||||
Handling the request at the proxy server
|
||||
========================================
|
||||
----------------------------------------
|
||||
Seeing the "X-Run-Storlet" header the storlert_middleware at the proxy intercepts
|
||||
the request and performs a HEAD on the storlet specified by the user.
|
||||
This HEAD operation facilitates:
|
||||
@ -115,7 +108,7 @@ with the request being routed to an object server that holds a replica of the ob
|
||||
specified in the GET uri.
|
||||
|
||||
Handling the request at the object server
|
||||
=========================================
|
||||
-----------------------------------------
|
||||
Seeing the "X-Run-Storlet" header the storlert_middleware at the object server intecepts
|
||||
the request and perform the following two phased flow:
|
||||
|
||||
|
||||
@ -1,11 +1,9 @@
|
||||
====================
|
||||
Storlets Terminology
|
||||
====================
|
||||
The overall storlets mechanism involves a lot of moving parts as well as
|
||||
poeple or roles involved throughout the usage of the system. We give below
|
||||
a definition of the various terms used throughout the documentation.
|
||||
|
||||
-------
|
||||
Storlet
|
||||
-------
|
||||
A storlet is the binary code deployed as a Swift object. Invoking a storlet
|
||||
@ -26,7 +24,6 @@ In a nutshell this interface consists of an input stream, an output stream and a
|
||||
The storlet is assumed to read from the input stream, do its thing, and write the result
|
||||
to the output stream.
|
||||
|
||||
------------------
|
||||
Storlet Invocation
|
||||
------------------
|
||||
A storlet invocation is a user request wishing to perform a storlet computation
|
||||
@ -43,7 +40,6 @@ different ways:
|
||||
data object, where the storlet's output is kept in a newly crerated object. In a regular
|
||||
Swift copy the newly created object is a identical to the source object.
|
||||
|
||||
------------------
|
||||
The Storlet Engine
|
||||
------------------
|
||||
The storlet engine is the underlying mechanism that can take a storlet as a Swift object
|
||||
@ -52,9 +48,8 @@ In a nutshell the engine intercepts invocation requests, route the input data st
|
||||
the storlet and receives back the storlet output stream. The engine is implemented as a Swift
|
||||
middleware.
|
||||
|
||||
-----
|
||||
Roles
|
||||
-----
|
||||
=====
|
||||
|
||||
Storlet Developer
|
||||
-----------------
|
||||
|
||||
@ -1,4 +1,3 @@
|
||||
========================
|
||||
The Chip Bakers Use Case
|
||||
========================
|
||||
|
||||
|
||||
@ -1,4 +1,3 @@
|
||||
==================================
|
||||
Queriable "Secondary Storage" Data
|
||||
==================================
|
||||
It is said that the primary use case for object stores is to serve as secondary
|
||||
|
||||
@ -1,4 +1,3 @@
|
||||
============
|
||||
Data Privacy
|
||||
============
|
||||
|
||||
|
||||
104
doc/source/writing_and_deploying_java_storlets.rst
Normal file
104
doc/source/writing_and_deploying_java_storlets.rst
Normal file
@ -0,0 +1,104 @@
|
||||
Java Storlet Writing and Deploying Guide
|
||||
========================================
|
||||
|
||||
This is the Java specific storlet writing and deploying guide. This guide complements
|
||||
the more general_ guide for writing and deploying storlets which should be read first.
|
||||
To write a Java storlet you will need the SCommon.jar which is being built as part of
|
||||
the storlets build process as described in the development and testing guide_.
|
||||
|
||||
.. _guide: engine_dev_tests.html
|
||||
.. _general: writing_and_deploying_storlets.html
|
||||
|
||||
Import the .jar to a Java project in Eclipse and implement the
|
||||
org.openstack.storlet.common.IStorlet interface.
|
||||
The interface has a single method that looks like this:
|
||||
|
||||
::
|
||||
|
||||
public void invoke(ArrayList<StorletInputStream> inStreams,
|
||||
ArrayList<StorletOutputStream> outStreams,
|
||||
Map<String,String> parameters, StorletLogger logger) throws StorletException;
|
||||
|
||||
Below is a class diagram illustrating the classes involved in the above API.
|
||||
|
||||
.. image:: images/java_prog_model.jpg
|
||||
:height: 960px
|
||||
:width: 1216 px
|
||||
:scale: 50 %
|
||||
:alt: Java Programming Model Class Diagram
|
||||
:align: center
|
||||
|
||||
#. The StorleInputStream is used to stream object's data into the storlet.
|
||||
An instance of the class is provided whenever the Storlet gets an object as
|
||||
an input. Practically, it is used in all storlet invocation scenarios to
|
||||
stream in the object's data and metadata. To consume the data call getStream()
|
||||
to get a java.io.InputStream on which you can just read(). To consume the
|
||||
metadata call the getMetadata() method.
|
||||
|
||||
#. StorletObjectOutputStream. In all invocation scenarios the storlet is
|
||||
called with an instance of this class.
|
||||
|
||||
- Use the setMetadata method to set the Object's metadata.
|
||||
|
||||
- Use getStream to get a java.io.OutputStream on which you can just write()
|
||||
the content of the object.
|
||||
|
||||
- Notice that setMetadata must be called. Also, it must be called before
|
||||
writing the data. Additional guidelines on using StorletObjectOutputStream
|
||||
are given below.
|
||||
|
||||
#. StorletLogger. The StorletLogger class supports a single method called emitLog,
|
||||
and accepts a String. The storlet logs are written to the host machine.
|
||||
|
||||
When invoked via the Swift REST API the invoke method
|
||||
will be called as follows:
|
||||
|
||||
#. The inStreams array would include one or more element(s) of type StorleInputStream
|
||||
representing the object appearing in the request's URI (and possibly extra resources).
|
||||
|
||||
#. The outStreams would include a single element of type StorleObjectOutputStream
|
||||
representing the response returned to the user.
|
||||
|
||||
#. The parameters map includes execution parameters sent. These parameters can be
|
||||
specified in the storlet execution request.
|
||||
|
||||
#. A StorletLogger instance.
|
||||
|
||||
Tips
|
||||
----
|
||||
|
||||
#. The storlets are executed in an open-jdk 8 environment. Thus, any dependencies
|
||||
that the storlet code requires which are outside of open-jdk 8 should be
|
||||
stated as storlet dependencies and uploaded with the storlet. Exact details
|
||||
are found in the deployment section below.
|
||||
|
||||
#. In some cases the storlet may need to know the path where the storlet .jar
|
||||
as well as the dependencies are kept inside the Linux container. One reason
|
||||
may be the need to invoke a binary dependency. To get that path use the
|
||||
following code:
|
||||
|
||||
::
|
||||
|
||||
// Get the path of this class image
|
||||
String strJarPath = StorletUtils.getClassFolder(this.getClass());
|
||||
|
||||
Deploying a Java Storlet
|
||||
------------------------
|
||||
|
||||
Below are specific guidelines for deploying a Java storlet:
|
||||
|
||||
#. The compiled class that implements the storlet needs to be wrapped in a .jar.
|
||||
This jar must not include the SCommon.jar.
|
||||
|
||||
#. Any jars that the class implementation is dependent on should be uploaded as separate jars.
|
||||
|
||||
#. The name of the Java storlet being uploaded must be of the form <name>-<version>, e.g.
|
||||
identitystorlet-1.0.jar
|
||||
|
||||
#. The 'X-Object-Meta-Storlet-Main' metadata key shold be the name of the class implementing
|
||||
the IStorlet interface. E.g. "org.openstack.storlet.identity.IdentityStorlet"
|
||||
|
||||
Deploying a Java Dependency
|
||||
---------------------------
|
||||
|
||||
#. A java dependency must be either a .jar or an executable
|
||||
83
doc/source/writing_and_deploying_python_storlets.rst
Normal file
83
doc/source/writing_and_deploying_python_storlets.rst
Normal file
@ -0,0 +1,83 @@
|
||||
Python Storlet Writing and Deployment Guideline
|
||||
===============================================
|
||||
|
||||
This is the Python specific storlet writing and deploying guide. This guide complements
|
||||
the more general_ guide for writing and deploying storlets which should be read first.
|
||||
|
||||
.. _general: writing_and_deploying_storlets.html
|
||||
|
||||
A python module implementing a storlet looks like this:
|
||||
|
||||
::
|
||||
|
||||
class <Class name>(object):
|
||||
def __init__(self, logger):
|
||||
self.logger = logger
|
||||
|
||||
def __call__(self, in_files, out_files, params):
|
||||
"""
|
||||
The function called for storlet invocation
|
||||
:param in_files: a list of StorletInputFile
|
||||
:param out_files: a list of StorletOutputFile
|
||||
:param params: a dict of request parameters
|
||||
"""
|
||||
|
||||
Below is a class diagram illustrating the classes behind the in_files, out_files, and logger.
|
||||
The diagram lists only the methods that the storlet writer is expected to work with.
|
||||
|
||||
.. image:: images/python_prog_model.jpg
|
||||
:height: 960px
|
||||
:width: 1216 px
|
||||
:scale: 50 %
|
||||
:alt: Python Programming Model Class Diagram
|
||||
:align: center
|
||||
|
||||
#. The StorletInputFile is used to stream object's data into the storlet.
|
||||
StorletInputFile has the same read methods as python FileObject.
|
||||
Trying to write to a StorletInputFile yields NotImplemented error.
|
||||
Whenever a storlet is invoked, an instance of this class is provided.
|
||||
To consume the metadata call the StorletInputFile.get_metadata method.
|
||||
|
||||
#. The StorleOutputFile is used for writing the storlet output.
|
||||
StorletOutputFile has the same write methods as python FileObject.
|
||||
Trying to read from a StorletOutputFile yields NotImplemented error.
|
||||
Whenever a storlet is invoked, an instance of this class is provided.
|
||||
Use the StorletInputFile.set_metadata method to set the Object's metadata.
|
||||
Note that the storlet must call the StorletInputFile set_metadata method.
|
||||
Moreowver, StorletInputFile.set_metadata must be called before writing
|
||||
the data.
|
||||
|
||||
#. StorletLogger. The StorletLogger class implements the same log methods as the
|
||||
Python logger.
|
||||
|
||||
When invoked via the Swift REST API the __call__ method
|
||||
will be called as follows:
|
||||
|
||||
#. The in_files list would include one or more element(s) of type StorleInputFile
|
||||
representing the object appearing in the request's URI (and possibly extra resources).
|
||||
|
||||
#. The out_files would include a single element of type StorleOutputFile
|
||||
representing the response returned to the user.
|
||||
|
||||
#. The parameters is a dictionary with the execution parameters sent. These parameters can be
|
||||
specified in the storlet execution request.
|
||||
|
||||
#. A StorletLogger instance.
|
||||
|
||||
Deploying a Python Storlet
|
||||
--------------------------
|
||||
Below are specific guidelines for deploying a Python storlet:
|
||||
|
||||
#. The object name of the python module containing the storlet class implemetation
|
||||
must end with .py
|
||||
|
||||
#. Any python modules that the class implementation is dependent on should be uploaded as separate .py(s).
|
||||
|
||||
#. The 'X-Object-Meta-Storlet-Main' metadata key shold be of the form: <module_name>.<class_name>. For example,
|
||||
if the storlet name is SimpleStorlet and it resides in simple_storlet.py, then the
|
||||
'X-Object-Meta-Storlet-Main' metadata key shold be "simple_storlet.SimpleStorlet"
|
||||
|
||||
Deploying a Python Dependency
|
||||
-----------------------------
|
||||
|
||||
#. Currently, there is no limitation as to what is being uploaded as a dependency.
|
||||
@ -1,106 +1,50 @@
|
||||
===================================
|
||||
Storlet writing and deploying guide
|
||||
===================================
|
||||
Currently, storlets must be written in Java. Writing a storlet involves
|
||||
implementing a single method interface and following some simple rules and best
|
||||
practices described below.
|
||||
|
||||
Storlets can be written either in Java or in Python. This guide
|
||||
is a language independent starting point for writing
|
||||
and deploying storlets. The Java_ and Python_ specific guides
|
||||
complement this guide and should be read next after this one.
|
||||
|
||||
.. _Java: writing_and_deploying_java_storlets.html
|
||||
.. _Python: writing_and_deploying_python_storlets.html
|
||||
|
||||
|
||||
Writing a storlet involves implementing a single function referred to
|
||||
as invoke (the exact name varies between languages).
|
||||
|
||||
Typically, the invoke operation gets an input stream and an
|
||||
output stream. The input stream will contain the data either being uploaded
|
||||
(in case the storlet is invoked during upload) or being downloaded (when
|
||||
the invocation is done during download). The input stream is accompanied with
|
||||
metadata that reflects the Swift user defined metadata. The output stream
|
||||
exposes a way to write back not only the storlet output data, but also the
|
||||
metadata. The input and output streams are exposed to the storlet code
|
||||
in a way that is native to the language being used.
|
||||
|
||||
Once the storlet is written and tested it can be uploaded as an object to a
|
||||
designated container (called 'storlet' by default). In addition in case the
|
||||
storlet is dependent on some Java library, that library can be uploaded as a
|
||||
dependency of the storlet. It is assumed that storlet dependencies are small
|
||||
(on the order of few MBs), heavier dependencies should be part of the Docker
|
||||
image.
|
||||
designated container (called 'storlet' by default). Storlet code external
|
||||
dependencies such as Java libraries or Python modules can be uploaded as well.
|
||||
It is assumed, however, that storlet dependencies are relatively small
|
||||
(on the order of few MBs). Heavier dependencies should be part of the Docker
|
||||
image. Deploying storlets and dependencies are covered below.
|
||||
|
||||
To write a storlet you will need the SCommon.jar which is being built as part of
|
||||
the storlets build process as described in the development and testing guide_
|
||||
There are various implementations of storlets in the StorletSamples/java
|
||||
and StorletSamples/python directories. These storlets are used by the engine's
|
||||
functional tests.
|
||||
|
||||
.. _guide: engine_dev_tests.html
|
||||
The next two sections describe storlet writing and deploying guidelines that are
|
||||
indpendent of the language used.
|
||||
|
||||
Import the .jar to a Java project in Eclipse and implement the
|
||||
org.openstack.storlet.common.IStorlet interface.
|
||||
The interface has a single method that looks like this:
|
||||
|
||||
::
|
||||
|
||||
public void invoke(ArrayList<StorletInputStream> inStreams,
|
||||
ArrayList<StorletOutputStream> outStreams,
|
||||
Map<String,String> parameters, StorletLogger logger) throws StorletException;
|
||||
|
||||
Below is a class diagram illustrating the classes involved in the above API.
|
||||
|
||||
.. image:: images/java_prog_model.jpg
|
||||
:height: 960px
|
||||
:width: 1216 px
|
||||
:scale: 50 %
|
||||
:alt: Programming Model Class Diagram
|
||||
:align: center
|
||||
|
||||
#. The StorleInputStream is used to stream object's data into the storlet.
|
||||
An instance of the class is provided whenever the Storlet gets an object as
|
||||
an input. Practically, it is used in all storlet invocation scenarios to
|
||||
stream in the object's data and metadata. To consume the data call getStream()
|
||||
to get a java.io.InputStream on which you can just read(). To consume the
|
||||
metadata call the getMetadata() method.
|
||||
#. The StorleOutputStream is a base class for the StorletObjectOutputStream.
|
||||
The actual instance received by the storlet will always be StorletObjectOutputStream.
|
||||
#. StorletObjectOutputStream. In all invocation scenarios the storlet is
|
||||
called with an instance of this class.
|
||||
|
||||
- Use the setMetadata method to set the Object's metadata.
|
||||
- Use getStream to get a java.io.OutputStream on which you can just write()
|
||||
the content of the object.
|
||||
- Notice that setMetadata must be called. Also, it must be called before
|
||||
writing the data. Additional guidelines on using StorletObjectOutputStream
|
||||
are given below.
|
||||
#. StorletLogger. The StorletLogger class supports a single method called emitLog,
|
||||
and accepts a String. The storlet logs are written to the host machine.
|
||||
|
||||
When invoked via the Swift REST API the invoke method
|
||||
will be called as follows:
|
||||
|
||||
#. The inStreams array would include a single element of type StorleInputStream
|
||||
representing the object appearing in the request's URI.
|
||||
#. The outStreams would include a single element of type StorleObjectOutputStream
|
||||
representing the response returned to the user.
|
||||
#. The parameters map includes execution parameters sent. These parameters can be
|
||||
specified in the storlet execution request.
|
||||
#. A StorletLogger instance.
|
||||
|
||||
==========================
|
||||
Storlet Writing Guidelines
|
||||
==========================
|
||||
Below are some guidelines for writing a storlet. Some of them are musts, some are
|
||||
recommendations, and some are tips.
|
||||
Independent of the language used for writing a storlet, there are several guidelines
|
||||
to follow. Some of them are musts and some are recommendations.
|
||||
|
||||
-----
|
||||
Musts
|
||||
-----
|
||||
#. The storlet code must be thread safe and re-enterant. The invoke method will
|
||||
be called many times and potentially in parallel.
|
||||
#. Once the storlet has finished writing the response, it is important to close
|
||||
the output stream. Failing to do so will result in a timeout. Specifically,
|
||||
close the java.io.OutputStream obtained from the call to getStreasm()
|
||||
#. With the current implementation, a storlet must start to respond within 40
|
||||
seconds of invocation. Otherwise, Swift would timeout. Moreover, the storlet
|
||||
must output something every 40 seconds so as not to timeout. This is a
|
||||
mechanism to ensure that the storlet code does not get stuck. Note that
|
||||
outputting an empty string does not do the job in terms of resetting the 40
|
||||
seconds timeout.
|
||||
#. For StorletObjectOutputStream, the call to setMetadata must happen before the
|
||||
storlet starts streaming out the output data. Note the applicability of the 40
|
||||
seconds timeout here as well.
|
||||
#. The total size of metadata given to setMetadata (when serialized as a string)
|
||||
should not exceed 4096 Bytes
|
||||
#. While Swift uses the prefix X-Object-Meta to specify that a certain header
|
||||
reflects a metadata key, the key itself should not begin with that prefix.
|
||||
More specifically, metadata keys passed to setMetadata should not have that
|
||||
prefix (unless this is really part of the key)
|
||||
|
||||
---------------
|
||||
Recommendations
|
||||
---------------
|
||||
#. Storlets are tailored for stream processing, that is, process the input as it
|
||||
|
||||
#. Storlets are tailored for stream processing, that is, they process the input as it
|
||||
is read and produce output while still reading. In other words a 'merge sort'
|
||||
of the content of an object is not a good example for a storlet as it requires
|
||||
to read all the content into memory (random reads are not an option as the
|
||||
@ -109,103 +53,103 @@ Recommendations
|
||||
into memory or doing very intensive computations would have impact on the overall
|
||||
system performance.
|
||||
|
||||
#. While this might be obvious it is advisable to test the storlet prior to its
|
||||
deployment.
|
||||
#. While this might be obvious make sure to unit test your storlet prior to deploying it.
|
||||
|
||||
----
|
||||
Tips
|
||||
----
|
||||
#. The storlets are executed in an open-jdk 8 environment. Thus, any dependencies
|
||||
that the storlet code requires which are outside of open-jdk 8 should be
|
||||
stated as storlet dependencies and uploaded with the storlet. Exact details
|
||||
are found in the deployment section below.
|
||||
Musts
|
||||
-----
|
||||
|
||||
#. In some cases the storlet may need to know the path where the storlet .jar
|
||||
as well as the dependencies are kept inside the Linux container. One reason
|
||||
may be the need to invoke a binary dependency. To get that path use the
|
||||
following code:
|
||||
#. The storlet code must be thread safe and re-enterant. The invoke method will
|
||||
be called many times and potentially in parallel.
|
||||
|
||||
::
|
||||
#. Once the storlet has finished writing the response, it is important to close
|
||||
the output stream. Failing to do so will result in a timeout.
|
||||
|
||||
// Get the path of this class image
|
||||
String strJarPath = StorletUtils.getClassFolder(this.getClass());
|
||||
#. A storlet must start to respond within 40 seconds of invocation. Otherwise,
|
||||
Swift would timeout. Moreover, the storlet must output something every 40 seconds
|
||||
so as not to timeout. This is a mechanism to ensure that the storlet code does not
|
||||
get stuck. Note that outputting an empty string does not do the job in terms of
|
||||
resetting the 40 seconds timeout.
|
||||
|
||||
----------------
|
||||
Storlet Examples
|
||||
----------------
|
||||
There are various implementations of storlets in the StorletSamples
|
||||
directory. These are used in the engine's functional tests
|
||||
In the below example, we will be using the identity storlet
|
||||
found under StorletSamples/IdentityStorlet
|
||||
#. The storlet must write metadata to the output stream, and must do so before it
|
||||
starts streaming out the data. A typical implementation would read the
|
||||
input metadata and use it as a basis for the metadata being written.
|
||||
Note the applicability of the 40 seconds timeout here as well.
|
||||
|
||||
=======================
|
||||
How to Deploy a storlet
|
||||
=======================
|
||||
In this paragraph we cover:
|
||||
#. The total size of metadata that can be set (when serialized as a string)
|
||||
must not exceed 4096 Bytes
|
||||
|
||||
#. The principles behind storlet deployment, plus examples.
|
||||
#. A Swift client example for uploading a storlet.
|
||||
#. A python example for uploading a storlet.
|
||||
#. While Swift uses the prefix X-Object-Meta to specify that a certain header
|
||||
reflects a metadata key, the key itself should not begin with that prefix.
|
||||
More specifically, metadata keys being set by the storlet should not have that
|
||||
prefix (unless this is really part of the key)
|
||||
|
||||
-----------------------------
|
||||
Storlet Deployment Principles
|
||||
-----------------------------
|
||||
|
||||
The compiled class that implements the storlet needs to be wrapped in a .jar.
|
||||
This jar must not include the SCommon.jar. Any jars that the class implementation
|
||||
is dependent on should be uploaded as separate jars as shown in the deployment
|
||||
section below.
|
||||
Storlet Deployment Guidelines
|
||||
=============================
|
||||
|
||||
Storlet deployment is essentially uploading the storlet and its dependencies to
|
||||
designated containers in the account we are working with. While a storlet and a
|
||||
designated containers in the account you are working with. While a storlet and a
|
||||
dependency are regular Swift objects, they must carry some metadata used by the
|
||||
storlet engine. When a storlet is first executed, the engine fetches the necessary
|
||||
objects from Swift and puts them is a directory accessible by the Docker container.
|
||||
objects from Swift and puts them in a directory accessible to the Docker container.
|
||||
Note that the dependencies are meant to be small. Having a large list of dependencies
|
||||
or a very large dependency may result in a timeout on the first attempt to execute a
|
||||
storlet. If this happens, just re-send the request again.
|
||||
|
||||
We consider two types of dependencies: libraries and executables. libraries would
|
||||
typically be .jar files the storlet code is dependent on. Alternatively, one can
|
||||
have a binary dependency, that the storlet code can execute.
|
||||
We support two types of dependencies:
|
||||
|
||||
Following the Identity storlet example, we have 2 objects to upload:
|
||||
#. External libraries or modules that
|
||||
are native to the storlet langauge
|
||||
|
||||
#. The storlet packaged in a .jar. In our case the jar was named:
|
||||
identitystorlet-1.0.jar The jar needs to be uploaded to a container named
|
||||
storlet. The name of the uploaded storlet must be of the form <name>-<version>.
|
||||
The metadata that must accompany a storlet is as follows:
|
||||
#. Executables dependency that the storlet code
|
||||
can execute.
|
||||
|
||||
Storlet Object Metadata
|
||||
-----------------------
|
||||
Uploading a storlet must be done to a designated container, called by default "storlet". The uploaded object
|
||||
must carry the following metadata. See the specific langauge guides for more information.
|
||||
|
||||
::
|
||||
|
||||
X-Object-Meta-Storlet-Language - currently must be 'java'
|
||||
X-Object-Meta-Storlet-Language - must be 'python' or 'java'
|
||||
X-Object-Meta-Storlet-Interface-Version - currenltly we have a single version '1.0'
|
||||
X-Object-Meta-Storlet-Dependency - A comma separated list of dependencies. In our case: 'get42'
|
||||
X-Object-Meta-Storlet-Object-Metadata - Currently, not in use, but must appear. Use the value 'no'
|
||||
X-Object-Meta-Storlet-Main - The name of the class that implements the IStorlet API. In our case: 'org.openstack.storlet.identity.IdentityStorlet'
|
||||
X-Object-Meta-Storlet-Main - The name of the class that implements the invoke operation
|
||||
|
||||
#. The binary file that the storlet code is dependent on. In our case it is a
|
||||
binary called get42. The binary should be uploaded to a container named
|
||||
dependency. The dependency metadata fields appear below. Note the permissions
|
||||
header. This header is required so that the engine will chmod it accordingly
|
||||
when placed in the container so that the storlet would be able to execute it.
|
||||
Optional metadata item is:
|
||||
|
||||
::
|
||||
::
|
||||
|
||||
X-Object-Meta-Storlet-Dependency-Version - While the engine currently does not parse this header, it must appear.
|
||||
X-Object-Meta-Storlet-Dependency-Permissions - An optional metadata field, where the user can state the permissions
|
||||
given to the dependency when it is copied to the Linux container. This is helpful for binary dependencies invoked by the
|
||||
storlet. For a binary dependency once can specify: '0755'
|
||||
X-Object-Meta-Storlet-Dependency - A comma separated list of dependencies.
|
||||
|
||||
If one wishes to update the storlet just upload again, the engine would recognize
|
||||
the update and bring the updated code.
|
||||
|
||||
Important: Currently, dependency updates are not recognized, only the Storlet
|
||||
code itself can be updated.
|
||||
Dependency Object Metadata
|
||||
--------------------------
|
||||
Uploading a dependency must be done to a designated container, called by default "dependency". The uploaded object
|
||||
must carry the following metadata.
|
||||
|
||||
--------------------------------------
|
||||
Deploying a Storlet using Swift Client
|
||||
--------------------------------------
|
||||
::
|
||||
|
||||
X-Object-Meta-Storlet-Dependency-Version - While the engine currently does not parse this header, it must appear.
|
||||
|
||||
Optional metadata item is:
|
||||
|
||||
::
|
||||
|
||||
X-Object-Meta-Storlet-Dependency-Permissions - The permissions given to the dependency when it is copied to the
|
||||
Docker container. This is helpful for binary dependencies invoked by the storlet.
|
||||
For a binary dependency once can specify: '0755'
|
||||
|
||||
|
||||
.. note:: Currently, dependency updates are not recognized.
|
||||
|
||||
|
||||
Deploying a Storlet using Swift Client cli
|
||||
------------------------------------------
|
||||
We show below how to deploy a storlet using the Swift client cli.
|
||||
The example uses a Java storlet. The differences from deploying a
|
||||
Python storlet are minor and we highlight them where required.
|
||||
|
||||
When using the Swift client one needs to provide the credentials, as well as the
|
||||
authentication URI. The credentials can be supplied either via environment
|
||||
@ -243,10 +187,15 @@ Here is the Swift client command for uploading the storlet. some notes:
|
||||
-H "X-Object-Meta-Storlet-Main:org.openstack.storlet.identity.IdentityStorlet" \
|
||||
-H "X-Object-Meta-Storlet-Dependency:get42"
|
||||
|
||||
.. note:: When deploying a Python storlet the name of the object (identitystorlet-1.0.jar in the above) has a different format
|
||||
Otherwise, "X-Object-Meta-Storlet-Language" is "Python", and "X-Object-Meta-Storlet-Main" has a different format.
|
||||
Please refer to Python_ for the exact details.
|
||||
|
||||
Here is the Swift client command for uploading the get42 dependency. Again,
|
||||
some notes:
|
||||
|
||||
#. The container name used here is the first parameter for the upload command and is 'dependency'.
|
||||
|
||||
#. We use the optional permissions header as this is a binary .
|
||||
|
||||
::
|
||||
@ -255,9 +204,8 @@ some notes:
|
||||
-H "X-Object-Meta-Storlet-Dependency-Version:1.0" \
|
||||
-H "X-Object-Meta-Storlet-Dependency-Permissions:0755"
|
||||
|
||||
-------------------------------
|
||||
Deploying a Storlet with Python
|
||||
-------------------------------
|
||||
Deploying a Storlet using the Python Swift Client
|
||||
-------------------------------------------------
|
||||
|
||||
Here is a code snippet that uploads both the storlet as well as the dependencies.
|
||||
The code assumes v2 authentication, and was tested against a Swift cluster with:
|
||||
@ -267,6 +215,10 @@ The code assumes v2 authentication, and was tested against a Swift cluster with:
|
||||
#. Under the service account there are already 'storlet' and 'dependency'
|
||||
containers.
|
||||
|
||||
The example uses a Java storlet. The differences from deploying a
|
||||
Python storlet are minor and are the same as the differences highlighted
|
||||
in the deployment using Swift client section above.
|
||||
|
||||
::
|
||||
|
||||
from swiftclient import client
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user