Middleware and Compute Engine for an OpenStack Swift compute framework that runs compute within a Swift cluster
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Zuul 583d126bb4 Merge "Avoid broken http header including white space" 1 week ago
StorletSamples Avoid broken http header including white space 1 week ago
bin Add basic client implementation for sbus protocol 3 months ago
devstack Upgrade base docker image to Bionic(18.04) 3 months ago
doc/source PY3: Fix file open to read/write the data as bytes 2 weeks ago
etc Remove unused parameters from sample conf files 6 months ago
playbooks/storlets-functional Port gathering logs routine from .functests to post ansible yaml 2 weeks ago
releasenotes Merge "Update master for stable/train" 1 month ago
scripts Fix build warning in c codes 3 months ago
src Use maven for simple_json jar 1 week ago
storlets Ensure task_id is a native string on py3 2 weeks ago
tests Merge "Avoid broken http header including white space" 1 week ago
.coveragerc Change ignore-errors to ignore_errors 4 years ago
.functests Port gathering logs routine from .functests to post ansible yaml 2 weeks ago
.gitignore Switch to use stestr for unit test 1 year ago
.gitreview OpenDev Migration Patch 7 months ago
.mailmap Initial Cookiecutter Commit. 4 years ago
.stestr.conf Switch to use stestr for unit test 1 year ago
.unittests Prep works for Py3 gate 3 years ago
.zuul.yaml Decrease fuctional test timeout 2 weeks ago
AUTHORS Update AUTHORS and add release note 8 months ago
CHANGELOG CHANGELOG and AUTHORS updates for storlets 1.0.0 1 year ago
CONTRIBUTING.rst Replace git.openstack.org URLs with opendev.org URLs 7 months ago
HACKING.rst Update the documention link for doc migration 2 years ago
LICENSE Initial Cookiecutter Commit. 4 years ago
README.rst Make the playbook's yaml file start with "---" 1 year ago
babel.cfg Initial Cookiecutter Commit. 4 years ago
bindep.txt correct the urls 1 year ago
build.xml Create storlets package 3 years ago
cluster_config.json-sample Introducing initial cluster config 3 years ago
install_libs.sh Create an independent directory to place jar libraries 4 months ago
openstack-common.conf Initial Cookiecutter Commit. 4 years ago
requirements.txt Drop oslosphinx package 4 months ago
s2aio.sh moving to devstack first phase 2 years ago
setup.cfg Merge "Add basic client implementation for sbus protocol" 3 months ago
setup.py Working with a JSON cluster configuration 3 years ago
test-requirements.txt Sync Sphinx requirement 1 month ago
tox.ini Be explicit about which python3 to use for py36 and py37 tox envs 2 weeks ago

README.rst

Storlets

Storlets extend Swift with the ability to run user defined computations - called storlets - near the data in a secure and isolated manner. A storlet is a compiled and packaged code (e.g. a .jar file) that can be uploaded to Swift as any other object. Once uploaded the storlet can be invoked over data objects in Swift. The Storlets API is documented at "Storlets API v1".

The project started off as an IBM research project, and was open sourced by IBM in 2014.

  • The research leading to the development of this code received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under the grant agreements for the CASPAR, ENSURE and VISION Cloud projects.
  • Various stages and different aspects of the development of this code received funding from the following European Community's Framework Programme:
    • Seventh Framework Programme (FP7/2007-2013) under the grant agreements for the ForgetIT project, where the code is used for offloading digital preservation functionalities to the storage.
    • Seventh Framework Programme (FP7/2007-2013) under the grant agreements for COSMOS project, where the code is used for analysis of IoT data.
    • Seventh Framework Programme (FP7/2007-2013) under the grant agreements for FI-CORE project where the code is integrated with a holistic cloud deployment solution, and from
    • Horizon 2020 (H2020/2014-2020) under the grant agreement for the IOStack project where the code is used as a backend implementing Storage policies and is used for analytics

Docs

The storlerts documentation is auto-generated after every commit and available online at https://storlets.readthedocs.io/en/latest/

Getting Started for Users

The fastest way to get started is "S2AIO - Swift Storlets All In One".

For Engine Developers

Getting Started

The best way to get started is following this guide: "Installing a Development Environment".

Tests

There are two types of tests included in the Storlets repo.

  1. Unit tests
  2. Functional tests

Unit tests, are, well, unit tests... The functional tests are black box tests validating end-to-end scenarios using various storlets, including faulty ones. For more information please refer to the: "Development and Testing Guide".

Code Organization

  • Engine/: The code comprising the middleware and the compute engine
    • SBus/: A protocol layer between the middleware and the sandbox used to execute storlets
      • SBusJavaFacade/: A Java implementation of the protocol
      • SBusPythonFacade/: A Python implementation of the protocol
      • SBusTransportLayer/: A thin layer in "C" used for passing fds between the middleware and container
    • SCommon/: A Java library required for storlets development
    • SDaemon/: A generic Java daemon for loading storlets at runtime
    • SMSCripts/: Run time scripts for doing Docker management commands
    • agent/: Python code for Docker side storlets process management
    • swift/: Python swift side code
      • etc/: Sample config files
      • storlet_gateway/: Run time loadable code for managing storlets execution
      • storlet_middleware/: Swift middleware dealing with storlet invocation requests
  • StorletSamples/: Storlets examples, used for functional testing
  • doc/source/: Documentation
  • install/: Installation scripts
    • swift/: Scripts for invoking an extenral Swift ansible installation scrpts
    • storlets/: Scripts for installing storlets over a Swift cluster
  • tests/: Unit and functional tests
  • tools/: Various cluster config dependent tools for automatic and manual testing

For Storlets Developers

Currently, storlets can be developed in Java only. To get started, follow: "S2AIO - Swift Storlets All In One".

The write and deploy a storlet, follow: "Writing and deploying storelts".

For Deployers

Instructions for how to deploy Storlets over an existing Swift cluster (not SAIO): "Deploying storlets over an existing Swift cluster".