openstack
/
storlets
Code Issues Proposed changes
Middleware and Compute Engine for an OpenStack Swift compute framework that runs compute within a Swift cluster
908 Commits 9 Branches 23 Tags 13 MiB
Python 70.3%
Java 22.5%
C 3.8%
Shell 2.4%
Jupyter Notebook 0.9%
Go to file
OpenStack Release Bot 9256bd59bf Switch to 2023.1 Python3 unit tests and generic template name 
This is an automatically generated patch to ensure unit testing
is in place for all the of the tested runtimes for antelope. Also,
updating the template name to generic one.

See also the PTI in governance [1].

[1]: https://governance.openstack.org/tc/reference/project-testing-interface.html

Change-Id: I843494b1f7dfeffc038039b6784182dab503915a
 		2022-09-20 00:35:15 +00:00
StorletSamples Replace github by opendev 2022-05-04 21:54:14 +09:00
bin Do not bind python2 specific directory into container 2019-12-17 21:22:16 +09:00
devstack Remove remaining usage of six 2022-07-16 23:50:43 +09:00
doc remove unicode literal from code 2022-07-28 10:06:40 +00:00
etc Allow limiting number of containers on a node 2021-08-19 09:17:41 +00:00
playbooks/storlets-functional Drop Python 2 support in middleware/gateway 2022-05-04 21:50:15 +09:00
releasenotes Update master for stable/zed 2022-09-18 23:01:02 +09:00
src Bump version of java dependencies 2022-06-26 18:19:49 +00:00
storlets Merge "Fix outdated description of spawn_subprocess" 2022-06-26 19:13:01 +00:00
tests Merge "Use wildcard for CLASSPATH" 2022-06-26 18:43:53 +00:00
.coveragerc Change ignore-errors to ignore_errors 2015-09-21 14:55:43 +00:00
.functests Drop Python 2 support in middleware/gateway 2022-05-04 21:50:15 +09:00
.gitignore Get rid of SUID binary to restart docker containers 2021-08-19 09:17:07 +00:00
.gitreview OpenDev Migration Patch 2019-04-19 19:47:54 +00:00
.mailmap Initial Cookiecutter Commit. 2015-08-09 18:08:41 +03:00
.stestr.conf Switch to use stestr for unit test 2018-07-20 13:59:22 +07:00
.unittests Prep works for Py3 gate 2016-09-08 11:54:48 +09:00
.zuul.yaml Switch to 2023.1 Python3 unit tests and generic template name 2022-09-20 00:35:15 +00:00
AUTHORS Update IRC network to OFTC 2021-06-01 00:40:19 +09:00
CHANGELOG CHANGELOG and AUTHORS updates for storlets 1.0.0 2018-02-07 11:00:42 +09:00
CONTRIBUTING.rst Replace github by opendev 2022-05-04 21:54:14 +09:00
HACKING.rst Update the documention link for doc migration 2017-10-16 15:03:04 +08:00
LICENSE Initial Cookiecutter Commit. 2015-08-09 18:08:41 +03:00
README.rst typo: storlet, not storelt 2020-05-15 19:45:20 -07:00
bindep.txt Remove bindep entry for bionic 2022-06-27 00:37:13 +09:00
build.xml Create storlets package 2016-12-01 10:28:43 +09:00
cluster_config.json-sample Introducing initial cluster config 2016-01-16 03:33:47 +02:00
install_libs.sh Get rid of SUID binary to restart docker containers 2021-08-19 09:17:07 +00:00
requirements.txt Get rid of SUID binary to restart docker containers 2021-08-19 09:17:07 +00:00
s2aio.sh Replace github by opendev 2022-05-04 21:54:14 +09:00
setup.cfg Drop Python 2 support in middleware/gateway 2022-05-04 21:50:15 +09:00
setup.py Revert "Cleanup py27 support" 2020-04-16 11:48:13 +09:00
test-requirements.txt Merge "Replace third party mock" 2022-06-26 17:13:55 +00:00
tox.ini Use TOX_CONSTRAINTS_FILE 2022-07-26 03:06:35 +00:00

README.rst

Storlets

Storlets extend Swift with the ability to run user defined computations - called storlets - near the data in a secure and isolated manner. A storlet is a compiled and packaged code (e.g. a .jar file) that can be uploaded to Swift as any other object. Once uploaded the storlet can be invoked over data objects in Swift. The Storlets API is documented at "Storlets API v1".

The project started off as an IBM research project, and was open sourced by IBM in 2014.

  • The research leading to the development of this code received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under the grant agreements for the CASPAR, ENSURE and VISION Cloud projects.
  • Various stages and different aspects of the development of this code received funding from the following European Community's Framework Programme:
    • Seventh Framework Programme (FP7/2007-2013) under the grant agreements for the ForgetIT project, where the code is used for offloading digital preservation functionalities to the storage.
    • Seventh Framework Programme (FP7/2007-2013) under the grant agreements for COSMOS project, where the code is used for analysis of IoT data.
    • Seventh Framework Programme (FP7/2007-2013) under the grant agreements for FI-CORE project where the code is integrated with a holistic cloud deployment solution, and from
    • Horizon 2020 (H2020/2014-2020) under the grant agreement for the IOStack project where the code is used as a backend implementing Storage policies and is used for analytics

Docs

The storlerts documentation is auto-generated after every commit and available online at https://storlets.readthedocs.io/en/latest/

Getting Started for Users

The fastest way to get started is "S2AIO - Swift Storlets All In One".

For Engine Developers

Getting Started

The best way to get started is following this guide: "Installing a Development Environment".

Tests

There are two types of tests included in the Storlets repo.

  1. Unit tests
  2. Functional tests

Unit tests, are, well, unit tests... The functional tests are black box tests validating end-to-end scenarios using various storlets, including faulty ones. For more information please refer to the: "Development and Testing Guide".

Code Organization

  • Engine/: The code comprising the middleware and the compute engine
    • SBus/: A protocol layer between the middleware and the sandbox used to execute storlets
      • SBusJavaFacade/: A Java implementation of the protocol
      • SBusPythonFacade/: A Python implementation of the protocol
      • SBusTransportLayer/: A thin layer in "C" used for passing fds between the middleware and container
    • SCommon/: A Java library required for storlets development
    • SDaemon/: A generic Java daemon for loading storlets at runtime
    • SMSCripts/: Run time scripts for doing Docker management commands
    • agent/: Python code for Docker side storlets process management
    • swift/: Python swift side code
      • etc/: Sample config files
      • storlet_gateway/: Run time loadable code for managing storlets execution
      • storlet_middleware/: Swift middleware dealing with storlet invocation requests
  • StorletSamples/: Storlets examples, used for functional testing
  • doc/source/: Documentation
  • install/: Installation scripts
    • swift/: Scripts for invoking an extenral Swift ansible installation scrpts
    • storlets/: Scripts for installing storlets over a Swift cluster
  • tests/: Unit and functional tests
  • tools/: Various cluster config dependent tools for automatic and manual testing

For Storlets Developers

Currently, storlets can be developed in Java only. To get started, follow: "S2AIO - Swift Storlets All In One".

The write and deploy a storlet, follow: "Writing and deploying storlets".

For Deployers

Instructions for how to deploy Storlets over an existing Swift cluster (not SAIO): "Deploying storlets over an existing Swift cluster".