05941f13eb
Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using these versions will allow especially: * Linking from HTML to PDF document * Allow parallel building of documents * Fix some rendering problems Update Sphinx version as well. Disable openstackdocs_auto_version to not auto-version the documents. Disable openstackdocs_auto_name to use 'project' variable as name. Change pygments_style to 'native' since old theme version always used 'native' and the theme now respects the setting and using 'sphinx' can lead to some strange rendering. Remove docs requirements from lower-constraints, they are not needed during install or test but only for docs building. openstackdocstheme renames some variables, so follow the renames before the next release removes them. A couple of variables are also not needed anymore, remove them. Use doc/requirements for doc requirements following PTI. See also http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html Change-Id: I1c0aeb590b1ce0f5ee43f9bcea51eeef1c98f057 |
||
|---|---|---|
| StorletSamples | ||
| bin | ||
| devstack | ||
| doc | ||
| etc | ||
| playbooks/storlets-functional | ||
| releasenotes | ||
| scripts | ||
| src | ||
| storlets | ||
| tests | ||
| .coveragerc | ||
| .functests | ||
| .gitignore | ||
| .gitreview | ||
| .mailmap | ||
| .stestr.conf | ||
| .unittests | ||
| .zuul.yaml | ||
| AUTHORS | ||
| CHANGELOG | ||
| CONTRIBUTING.rst | ||
| HACKING.rst | ||
| LICENSE | ||
| README.rst | ||
| babel.cfg | ||
| bindep.txt | ||
| build.xml | ||
| cluster_config.json-sample | ||
| install_libs.sh | ||
| openstack-common.conf | ||
| py2-constraints.txt | ||
| requirements.txt | ||
| s2aio.sh | ||
| setup.cfg | ||
| setup.py | ||
| test-requirements.txt | ||
| tox.ini | ||
README.rst
Storlets
Storlets extend Swift with the ability to run user defined computations - called storlets - near the data in a secure and isolated manner. A storlet is a compiled and packaged code (e.g. a .jar file) that can be uploaded to Swift as any other object. Once uploaded the storlet can be invoked over data objects in Swift. The Storlets API is documented at "Storlets API v1".
The project started off as an IBM research project, and was open sourced by IBM in 2014.
- The research leading to the development of this code received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under the grant agreements for the CASPAR, ENSURE and VISION Cloud projects.
- Various stages and different aspects of the development of this code
received funding from the following European Community's Framework
Programme:
- Seventh Framework Programme (FP7/2007-2013) under the grant agreements for the ForgetIT project, where the code is used for offloading digital preservation functionalities to the storage.
- Seventh Framework Programme (FP7/2007-2013) under the grant agreements for COSMOS project, where the code is used for analysis of IoT data.
- Seventh Framework Programme (FP7/2007-2013) under the grant agreements for FI-CORE project where the code is integrated with a holistic cloud deployment solution, and from
- Horizon 2020 (H2020/2014-2020) under the grant agreement for the IOStack project where the code is used as a backend implementing Storage policies and is used for analytics
Docs
The storlerts documentation is auto-generated after every commit and available online at https://storlets.readthedocs.io/en/latest/
Getting Started for Users
The fastest way to get started is "S2AIO - Swift Storlets All In One".
For Engine Developers
Getting Started
The best way to get started is following this guide: "Installing a Development Environment".
Tests
There are two types of tests included in the Storlets repo.
- Unit tests
- Functional tests
Unit tests, are, well, unit tests... The functional tests are black box tests validating end-to-end scenarios using various storlets, including faulty ones. For more information please refer to the: "Development and Testing Guide".
Code Organization
- Engine/: The code comprising the middleware and the compute engine
- SBus/: A protocol layer between the middleware and the sandbox used
to execute storlets
- SBusJavaFacade/: A Java implementation of the protocol
- SBusPythonFacade/: A Python implementation of the protocol
- SBusTransportLayer/: A thin layer in "C" used for passing fds between the middleware and container
- SCommon/: A Java library required for storlets development
- SDaemon/: A generic Java daemon for loading storlets at runtime
- SMSCripts/: Run time scripts for doing Docker management commands
- agent/: Python code for Docker side storlets process management
- swift/: Python swift side code
- etc/: Sample config files
- storlet_gateway/: Run time loadable code for managing storlets execution
- storlet_middleware/: Swift middleware dealing with storlet invocation requests
- SBus/: A protocol layer between the middleware and the sandbox used
to execute storlets
- StorletSamples/: Storlets examples, used for functional testing
- doc/source/: Documentation
- install/: Installation scripts
- swift/: Scripts for invoking an extenral Swift ansible installation scrpts
- storlets/: Scripts for installing storlets over a Swift cluster
- tests/: Unit and functional tests
- tools/: Various cluster config dependent tools for automatic and manual testing
For Storlets Developers
Currently, storlets can be developed in Java only. To get started, follow: "S2AIO - Swift Storlets All In One".
The write and deploy a storlet, follow: "Writing and deploying storlets".
For Deployers
Instructions for how to deploy Storlets over an existing Swift cluster (not SAIO): "Deploying storlets over an existing Swift cluster".