280 lines
12 KiB
ReStructuredText
280 lines
12 KiB
ReStructuredText
Storlet writing and deploying guide
|
|
===================================
|
|
|
|
Storlets can be written either in Java or in Python. This guide
|
|
is a language independent starting point for writing
|
|
and deploying storlets. The Java_ and Python_ specific guides
|
|
complement this guide and should be read next after this one.
|
|
|
|
.. _Java: writing_and_deploying_java_storlets.html
|
|
.. _Python: writing_and_deploying_python_storlets.html
|
|
|
|
|
|
Writing a storlet involves implementing a single function referred to
|
|
as invoke (the exact name varies between languages).
|
|
|
|
Typically, the invoke operation gets an input stream and an
|
|
output stream. The input stream will contain the data either being uploaded
|
|
(in case the storlet is invoked during upload) or being downloaded (when
|
|
the invocation is done during download). The input stream is accompanied with
|
|
metadata that reflects the Swift user defined metadata. The output stream
|
|
exposes a way to write back not only the storlet output data, but also the
|
|
metadata. The input and output streams are exposed to the storlet code
|
|
in a way that is native to the language being used.
|
|
|
|
Once the storlet is written and tested it can be uploaded as an object to a
|
|
designated container (called 'storlet' by default). Storlet code external
|
|
dependencies such as Java libraries or Python modules can be uploaded as well.
|
|
It is assumed, however, that storlet dependencies are relatively small
|
|
(on the order of few MBs). Heavier dependencies should be part of the Docker
|
|
image. Deploying storlets and dependencies are covered below.
|
|
|
|
There are various implementations of storlets in the StorletSamples/java
|
|
and StorletSamples/python directories. These storlets are used by the engine's
|
|
functional tests.
|
|
|
|
The next two sections describe storlet writing and deploying guidelines that are
|
|
independent of the language used.
|
|
|
|
Storlet Writing Guidelines
|
|
==========================
|
|
Independent of the language used for writing a storlet, there are several guidelines
|
|
to follow. Some of them are musts and some are recommendations.
|
|
|
|
Recommendations
|
|
---------------
|
|
|
|
#. Storlets are tailored for stream processing, that is, they process the input as it
|
|
is read and produce output while still reading. In other words a 'merge sort'
|
|
of the content of an object is not a good example for a storlet as it requires
|
|
to read all the content into memory (random reads are not an option as the
|
|
input is provided as a stream). While we currently do not employ any restrictions
|
|
on the CPU usage or memory consumption of the storlet, reading large object
|
|
into memory or doing very intensive computations would have impact on the overall
|
|
system performance.
|
|
|
|
#. While this might be obvious make sure to unit test your storlet prior to deploying it.
|
|
|
|
Musts
|
|
-----
|
|
|
|
#. The storlet code must be thread safe and re-enterant. The invoke method will
|
|
be called many times and potentially in parallel.
|
|
|
|
#. Once the storlet has finished writing the response, it is important to close
|
|
the output stream. Failing to do so will result in a timeout.
|
|
|
|
#. A storlet must start to respond within 40 seconds of invocation. Otherwise,
|
|
Swift would timeout. Moreover, the storlet must output something every 40 seconds
|
|
so as not to timeout. This is a mechanism to ensure that the storlet code does not
|
|
get stuck. Note that outputting an empty string does not do the job in terms of
|
|
resetting the 40 seconds timeout.
|
|
|
|
#. The storlet must write metadata to the output stream, and must do so before it
|
|
starts streaming out the data. A typical implementation would read the
|
|
input metadata and use it as a basis for the metadata being written.
|
|
Note the applicability of the 40 seconds timeout here as well.
|
|
|
|
#. The total size of metadata that can be set (when serialized as a string)
|
|
must not exceed 4096 Bytes
|
|
|
|
#. While Swift uses the prefix X-Object-Meta to specify that a certain header
|
|
reflects a metadata key, the key itself should not begin with that prefix.
|
|
More specifically, metadata keys being set by the storlet should not have that
|
|
prefix (unless this is really part of the key)
|
|
|
|
Storlet Deployment Guidelines
|
|
=============================
|
|
|
|
Storlet deployment is essentially uploading the storlet and its dependencies to
|
|
designated containers in the account you are working with. While a storlet and a
|
|
dependency are regular Swift objects, they must carry some metadata used by the
|
|
storlet engine. When a storlet is first executed, the engine fetches the necessary
|
|
objects from Swift and puts them in a directory accessible to the Docker container.
|
|
Note that the dependencies are meant to be small. Having a large list of dependencies
|
|
or a very large dependency may result in a timeout on the first attempt to execute a
|
|
storlet. If this happens, just re-send the request again.
|
|
|
|
We support two types of dependencies:
|
|
|
|
#. External libraries or modules that
|
|
are native to the storlet language
|
|
|
|
#. Executables dependency that the storlet code
|
|
can execute.
|
|
|
|
Storlet Object Metadata
|
|
-----------------------
|
|
Uploading a storlet must be done to a designated container, called by default "storlet". The uploaded object
|
|
must carry the following metadata. See the specific language guides for more information.
|
|
|
|
::
|
|
|
|
X-Object-Meta-Storlet-Language - must be 'python' or 'java'
|
|
X-Object-Meta-Storlet-Interface-Version - currenltly we have a single version '1.0'
|
|
X-Object-Meta-Storlet-Object-Metadata - Currently, not in use, but must appear. Use the value 'no'
|
|
X-Object-Meta-Storlet-Main - The name of the class that implements the invoke operation
|
|
|
|
Optional metadata item is:
|
|
|
|
::
|
|
|
|
X-Object-Meta-Storlet-Dependency - A comma separated list of dependencies.
|
|
|
|
If one wishes to update the storlet just upload again, the engine would recognize
|
|
the update and bring the updated code.
|
|
|
|
Dependency Object Metadata
|
|
--------------------------
|
|
Uploading a dependency must be done to a designated container, called by default "dependency". The uploaded object
|
|
must carry the following metadata.
|
|
|
|
::
|
|
|
|
X-Object-Meta-Storlet-Dependency-Version - While the engine currently does not parse this header, it must appear.
|
|
|
|
Optional metadata item is:
|
|
|
|
::
|
|
|
|
X-Object-Meta-Storlet-Dependency-Permissions - The permissions given to the dependency when it is copied to the
|
|
Docker container. This is helpful for binary dependencies invoked by the storlet.
|
|
For a binary dependency once can specify: '0755'
|
|
|
|
|
|
.. note:: Currently, dependency updates are not recognized.
|
|
|
|
|
|
Deploying a Storlet using Swift Client cli
|
|
------------------------------------------
|
|
We show below how to deploy a storlet using the Swift client cli.
|
|
The example uses a Java storlet. The differences from deploying a
|
|
Python storlet are minor and we highlight them where required.
|
|
|
|
When using the Swift client one needs to provide the credentials, as well as the
|
|
authentication URI. The credentials can be supplied either via environment
|
|
variables or via command line parameters. To make the commands more readable we
|
|
use environment variables as shown below. The actual values are aligned with the
|
|
development environment installation instructions_
|
|
|
|
.. _instructions: engine_dev_installation.html
|
|
|
|
::
|
|
|
|
export OS_USERNAME=tester
|
|
export OS_PASSWORD=testing
|
|
export OS_PROJECT_NAME=test
|
|
export OS_AUTH_URL=http://127.0.0.1/identity/v3
|
|
|
|
Here is the Swift client command for uploading the storlet. some notes:
|
|
|
|
#. We use the upload option of the swift cli.
|
|
#. The container name is the first parameter for the upload command and is
|
|
'storlet'
|
|
#. The name of the object and the local file to upload is 'identitystorlet-1.0-jar'
|
|
IMPORTANT: when uploading the file from another directory, that parameter would
|
|
be something of the form 'bin/identitystorlet-1.0-jar' in this case the name
|
|
of the object appearing in the storlet container would be 'bin/identitystorlet-1.0-jar'
|
|
which will not work for the engine.
|
|
#. The metadata that needs to accompany the storlet object is provided as headers.
|
|
|
|
::
|
|
|
|
eranr@lnx-ccs8:~/workspace/Storlets/StorletSamples/IdentityStorlet/bin$ swift upload storlet identitystorlet-1.0.jar \
|
|
-H "X-Object-Meta-Storlet-Language:Java" \
|
|
-H "X-Object-Meta-Storlet-Interface-Version:1.0" \
|
|
-H "X-Object-Meta-Storlet-Object-Metadata:no" \
|
|
-H "X-Object-Meta-Storlet-Main:org.openstack.storlet.identity.IdentityStorlet" \
|
|
-H "X-Object-Meta-Storlet-Dependency:get42"
|
|
|
|
.. note:: When deploying a Python storlet the name of the object (identitystorlet-1.0.jar in the above) has a different format
|
|
Otherwise, "X-Object-Meta-Storlet-Language" is "Python", and "X-Object-Meta-Storlet-Main" has a different format.
|
|
Please refer to Python_ for the exact details.
|
|
|
|
Here is the Swift client command for uploading the get42 dependency. Again,
|
|
some notes:
|
|
|
|
#. The container name used here is the first parameter for the upload command and is 'dependency'.
|
|
|
|
#. We use the optional permissions header as this is a binary .
|
|
|
|
::
|
|
|
|
eranr@lnx-ccs8:~/workspace/Storlets/StorletSamples/IdentityStorlet/bin$ swift upload dependency get42 \
|
|
-H "X-Object-Meta-Storlet-Dependency-Version:1.0" \
|
|
-H "X-Object-Meta-Storlet-Dependency-Permissions:0755"
|
|
|
|
Deploying a Storlet using the Python Swift Client
|
|
-------------------------------------------------
|
|
|
|
Here is a code snippet that uploads both the storlet as well as the dependencies.
|
|
The code assumes v2 authentication, and was tested against a Swift cluster with:
|
|
|
|
#. Keystone configured with a 'test' account, having a user 'tester' whose
|
|
password is 'testing'
|
|
#. Under the service account there are already 'storlet' and 'dependency'
|
|
containers.
|
|
|
|
The example uses a Java storlet. The differences from deploying a
|
|
Python storlet are minor and are the same as the differences highlighted
|
|
in the deployment using Swift client section above.
|
|
|
|
::
|
|
|
|
from swiftclient import client
|
|
|
|
def put_storlet_object(url, token, storlet_name, local_path_to_storlet, main_class_name, dependencies):
|
|
# Delete previous storlet
|
|
resp = dict()
|
|
|
|
metadata = {'X-Object-Meta-Storlet-Language':'Java',
|
|
'X-Object-Meta-Storlet-Interface-Version':'1.0',
|
|
'X-Object-Meta-Storlet-Dependency': dependencies,
|
|
'X-Object-Meta-Storlet-Object-Metadata':'no',
|
|
'X-Object-Meta-Storlet-Main': main_class_name}
|
|
f = open('%s/%s' % (local_path_to_storlet, storlet_name), 'rb')
|
|
content_length = None
|
|
response = dict()
|
|
client.put_object(url, token, 'storlet', storlet_name, f,
|
|
content_length, None, None,
|
|
"application/octet-stream",
|
|
metadata,
|
|
None, None, None,
|
|
response)
|
|
print response
|
|
f.close()
|
|
|
|
def put_storlet_dependency(url, token, dependency_name, local_path_to_dependency):
|
|
metadata = {'X-Object-Meta-Storlet-Dependency-Version': '1'}
|
|
# for an executable dependency
|
|
# metadata['X-Object-Meta-Storlet-Dependency-Permissions'] = '0755'
|
|
f = open('%s/%s'% (local_path_to_dependency, dependency_name), 'rb')
|
|
content_length = None
|
|
response = dict()
|
|
client.put_object(url, token, 'dependency', dependency_name, f,
|
|
content_length, None, None,
|
|
"application/octet-stream",
|
|
metadata,
|
|
None, None, None,
|
|
response)
|
|
print response
|
|
f.close()
|
|
status = response.get('status')
|
|
assert (status == 200 or status == 201)
|
|
|
|
AUTH_IP = '127.0.0.1'
|
|
ACCOUNT = 'test'
|
|
USER_NAME = 'tester'
|
|
PASSWORD = 'testing'
|
|
os_options = {'project_name': ACCOUNT}
|
|
|
|
url, token = client.get_auth("http://" + AUTH_IP + "/identity/v3", USER_NAME,
|
|
PASSWORD,
|
|
os_options = os_options,
|
|
auth_version="3")
|
|
put_storlet_object(url, token,'identitystorlet-1.0.jar','/tmp',
|
|
'org.openstack.storlet.identity.IdentityStorlet',
|
|
'get42')
|
|
put_storlet_dependency(url, token,'get42','/tmp')
|
|
|