openstack
/
storlets
Code Issues Proposed changes
Middleware and Compute Engine for an OpenStack Swift compute framework that runs compute within a Swift cluster
501 Commits 9 Branches 23 Tags 13 MiB
Python 70.3%
Java 22.5%
C 3.8%
Shell 2.4%
Jupyter Notebook 0.9%
Go to file
Eran Rom d64a586e9b Using devstack for swift and keystone installation 
This change gets rid of our proprietary scripts
for installing Swift and Keystone in favor of using
devstack.

So far we have used those scripts for:
1. The functional test gatejob
2. The s2aio.sh script

The merits are:
1. No need to maintain these ourselves.
   Specifically, this gives us Linux distros
   other than Ubuntu for free
2. Being more Openstack compliant. Specifically,
   avoid packages conflicts that arise from our
   script usage of apt and the storlets setup
   environemnt which uses pip.

The next step is to have a storlet devstack plugin
that would allow to setup a dev/test env for storlets
that is purely based on devstack.

It might be that once having a full devstack integration
we will clean the storlets repo from all ansible
scripts, and drop the goal for supplying a multi-cluster
installation support.

Change-Id: I4731fe7369bc86bc626f08833aba08e5ff1d7184
 		2016-11-23 09:54:06 +02:00
Engine Merge "Fix duplication of FD close in storlet_gateway" 2016-11-03 07:37:58 +00:00
StorletSamples Merge "Doc for multi input feature" 2016-09-23 03:11:35 +00:00
doc/source Using devstack for swift and keystone installation 2016-11-23 09:54:06 +02:00
install Using devstack for swift and keystone installation 2016-11-23 09:54:06 +02:00
tests Merge "Fix duplication of FD close in storlet_gateway" 2016-11-03 07:37:58 +00:00
tools Adding Storlets Get ACL feature 2016-09-25 14:44:55 +00:00
.coveragerc Change ignore-errors to ignore_errors 2015-09-21 14:55:43 +00:00
.functests Merge "Retrieve functest log for daemon-factory and storlets-daemon" 2016-09-07 16:00:44 +00:00
.gitignore Adding Storlets Get ACL feature 2016-09-25 14:44:55 +00:00
.gitreview Initial Cookiecutter Commit. 2015-08-09 18:08:41 +03:00
.mailmap Initial Cookiecutter Commit. 2015-08-09 18:08:41 +03:00
.testr.conf Use testr instead of nosetests 2016-09-21 15:50:28 +02:00
.unittests Prep works for Py3 gate 2016-09-08 11:54:48 +09:00
AUTHORS Add AUTHORS file 2016-03-08 08:00:29 +00:00
CHANGELOG Pre-cutting a stable-newton branch 2016-10-13 12:28:49 +03:00
CONTRIBUTING.rst Add meeting and irc info into CONTRIBUTING.rst 2016-01-25 17:04:19 -08:00
HACKING.rst Fix few typos 2015-09-20 08:11:57 +05:30
LICENSE Initial Cookiecutter Commit. 2015-08-09 18:08:41 +03:00
MANIFEST.in Initial Cookiecutter Commit. 2015-08-09 18:08:41 +03:00
README.rst Expand all tabs 2016-09-07 02:47:41 +09:00
babel.cfg Initial Cookiecutter Commit. 2015-08-09 18:08:41 +03:00
build.xml Towards python-centric repo 2016-09-14 10:38:17 +03:00
cluster_config.json-sample Introducing initial cluster config 2016-01-16 03:33:47 +02:00
openstack-common.conf Initial Cookiecutter Commit. 2015-08-09 18:08:41 +03:00
requirements.txt Add entry points for each gateway classes 2016-09-07 05:22:54 +09:00
s2aio.sh Using devstack for swift and keystone installation 2016-11-23 09:54:06 +02:00
sbus Use Pythonic name for SBusPythonFacade 2016-08-18 14:39:42 +09:00
sdeploy.sh Documentation face-lift 2016-06-22 15:45:00 +03:00
setup.cfg Add entry points for each gateway classes 2016-09-07 05:22:54 +09:00
setup.py Working with a JSON cluster configuration 2016-05-15 20:51:20 +03:00
storlet_daemon Implement Python SDaemon 2016-09-04 15:27:52 +09:00
storlet_daemon_factory Create an executable file for daemon_factory 2016-07-29 12:01:22 +09:00
storlet_gateway Create unittest runner env for storlet_middleware 2015-12-16 02:03:20 -08:00
storlet_middleware Create unittest runner env for storlet_middleware 2015-12-16 02:03:20 -08:00
test-requirements.txt Remove duplicated pexpect in test-requirements.txt 2016-09-23 01:50:11 +09:00
tox.ini Unpinning the Swift dependency version in master 2016-10-19 01:58:14 +00:00

README.rst

Storlets

Storlets extend Swift with the ability to run user defined computations - called storlets - near the data in a secure and isolated manner. A storlet is a compiled and packaged code (e.g. a .jar file) that can be uploaded to Swift as any other object. Once uploaded the storlet can be invoked over data objects in Swift. The Storlets API is documented at "Storlets API v1".

The project started off as an IBM research project, and was open sourced by IBM in 2014.

  • The research leading to the development of this code received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under the grant agreements for the CASPAR, ENSURE and VISION Cloud projects.
  • Various stages and different aspects of the development of this code received funding from the following European Community's Framework Programme:
    • Seventh Framework Programme (FP7/2007-2013) under the grant agreements for the ForgetIT project, where the code is used for offloading digital preservation functionalities to the storage.
    • Seventh Framework Programme (FP7/2007-2013) under the grant agreements for COSMOS project, where the code is used for analysis of IoT data.
    • Seventh Framework Programme (FP7/2007-2013) under the grant agreements for FI-CORE project where the code is integrated with a holistic cloud deployment solution, and from
    • Horizon 2020 (H2020/2014-2020) under the grant agreement for the IOStack project where the code is used as a backend implementing Storage policies and is used for analytics

Docs

The storlerts documentation is auto-generated after every commit and available online at http://storlets.readthedocs.io/en/latest/

Getting Started

The fastest way to get started is "S2AIO - Swift Storlets All In One".

For Engine Developers

Getting Started

The best way to get started is following this guide: "Installing a Development Environment".

Tests

There are two types of tests included in the Storlets repo.

  1. Unit tests
  2. Functional tests

Unit tests, are, well, unit tests... The functional tests are black box tests validating end-to-end scenarios using various storlets, including faulty ones. For more information please refer to the: "Development and Testing Guide".

Code Organization

  • Engine/: The code comprising the middleware and the compute engine
    • SBus/: A protocol layer between the middleware and the sandbox used to execute storlets
      • SBusJavaFacade/: A Java implementation of the protocol
      • SBusPythonFacade/: A Python implementation of the protocol
      • SBusTransportLayer/: A thin layer in "C" used for passing fds between the middleware and container
    • SCommon/: A Java library required for storlets development
    • SDaemon/: A generic Java daemon for loading storlets at runtime
    • SMSCripts/: Run time scripts for doing Docker management commands
    • agent/: Python code for Docker side storlets process management
    • swift/: Python swift side code
      • etc/: Sample config files
      • storlet_gateway/: Run time loadable code for managing storlets execution
      • storlet_middleware/: Swift middleware dealing with storlet invocation requests
  • StorletSamples/: Storlets examples, used for functional testing
  • doc/source/: Documentation
  • install/: Installation scripts
    • swift/: Scripts for invoking an extenral Swift ansible installation scrpts
    • storlets/: Scripts for installing storlets over a Swift cluster
  • tests/: Unit and functional tests
  • tools/: Various cluster config dependent tools for automatic and manual testing

For Storlets Developers

Currently, storlets can be developped in Java only. To get started, follow: "S2AIO - Swift Storlets All In One".

The write and deploy a storlet, follow: "Writing and deploying storelts".

For Deployers

Instructions for how to deploy Storlets over an existing Swift cluster (not SAIO): "Deploying storlets over an existing Swift cluster".