Create storlets package

Previously, we had multiple packages, like storlet_middleware,
storlet_gateway and so on, in our storlets repository.
This patch integrates these packages into one package, to follow
general manner of python packaging.

Also, this patch replaces previous install system using ant by
new install system using python setup.py.

(top directory)
 + bin          <- executable files
 + etc          <- configration samples
 + src          <- non-python codes
 |  + c         <- c codes
 |  | + sbus    <- Previously Engine/SBus/SBusTransportLayer
 |  + java      <- java codes
 |    + SBus    <- Previously Engine/SBus/SBusJavaFacade
 |    + SCommon <- Previously Engine/SBus/SCommon
 |    + SDaemon <- Previously Engine/SBus/SDaemon
 + storlets     <- python codes.
 |  + agent
 |  | + daemon         <- Previously Engine/agent/storlet_daemon
 |  | + daemon_factory <- Previously Engine/agent/storlet_daemon_factory
 |  + gateway          <- Previously Engine/swift/storlet_gateway
 |  + sbus             <- Previously Engine/SBus/SBusPythonFacade
 |  + swift_middleware <- Previously Engine/swift/storlet_middleware
 |  + tools            <- Previously tools
 + tests        <- test codes
 + scripts      <- Previously Engine/SMScripts
 + install_libs.sh     <- Script used to build/install c/java codes
 + setup.cfg
 + setup.py

NOTE1:
Currently we have an independent installation tool for c/java, as the
other existing OpenStack projects (ex. Monasca) do. To install c/java
modules, run './install_libs.sh' at the top directory. It installs
required files under /usr/local/lib/storlets, so you may need root
privilege.

NOTE2:
We install agent resources also on host, but this is currently kept
to be used for our future plan to let containers use agent resources
by mounting it from host node.

NOTE3:
This patch also removes outdated ansible playbooks.

Co-Authored-By: Eran Rom <eran@itsonlyme.name>
Change-Id: Icdcafdf6e6d06917d715ad9a8cba45305613a6fb
This commit is contained in:
Takashi Kajinami 2016-09-16 04:05:32 +09:00
parent d64a586e9b
commit 95cdd1663e
165 changed files with 568 additions and 1110 deletions

View File

@ -14,6 +14,8 @@ if [ "$FLAVOR" != "jenkins" ] && [ "$FLAVOR" != "dev" ]; then
fi
shift
ant build_storlets
SRC_DIR=$(python -c "import os; print os.path.dirname(os.path.realpath('$0'))")
cd ${SRC_DIR}/tests/functional
if [ "$FLAVOR" == "jenkins" ]; then

12
.gitignore vendored
View File

@ -1,6 +1,7 @@
*.py[cod]
# C extensions
*.o
*.so
# Packages
@ -35,11 +36,16 @@ install/swift/hosts
Engine/SMScripts/bin/
# auto created on build
Engine/dependencies/
Engine/SBus/SBusJavaFacade/org_openstack_storlet_sbus_SBusJNI.h
src/java/dependencies/
src/java/SBus/org_openstack_storlet_sbus_SBusJNI.h
# Storlets build
bin
*.class
*.jar
StorletSamples/java/*/bin
# scripts build
scripts/restart_docker_container
# Unit test / coverage reports
.coverage

View File

@ -1,3 +0,0 @@
/bin/*
/org_openstack_storlet_sbus_SBusJNI.h
/bin/

View File

@ -1,4 +0,0 @@
build
dist
*.egg-info
*.pyc

View File

@ -1,31 +0,0 @@
<!--
Copyright IBM Corp. 2015, 2015 All Rights Reserved
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
Limitations under the License.
-->
<project>
<target name="clean">
<delete dir="build" />
<delete dir="dist" />
<delete dir="SBusPythonFacade.egg-info" />
</target>
<target name="build">
<exec executable="python" dir="." failonerror="true">
<arg value="setup.py"/>
<arg value="bdist"/>
</exec>
</target>
</project>

View File

@ -1,20 +0,0 @@
#!/usr/bin/python
'''-------------------------------------------------------------------------
Copyright IBM Corp. 2015, 2015 All Rights Reserved
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
Limitations under the License.
-------------------------------------------------------------------------'''
from setuptools import setup, find_packages
setup(name='sbus',
version='1.0',
packages=find_packages())

View File

@ -1 +0,0 @@
/bin/*

View File

@ -1,30 +0,0 @@
<!--
Copyright IBM Corp. 2015, 2015 All Rights Reserved
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
Limitations under the License.
-->
<project>
<target name="so">
<mkdir dir="bin" />
<exec dir="." executable="gcc">
<arg line="-shared -o bin/sbus.so sbus.c -O2 -fPIC" />
</exec>
</target>
<target name="clean">
<delete dir="bin" />
</target>
<target name="build" depends="so">
</target>
</project>

View File

@ -1,2 +0,0 @@
bin

View File

@ -1,2 +0,0 @@
bin

View File

@ -1,30 +0,0 @@
<!--
Copyright IBM Corp. 2015, 2015 All Rights Reserved
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
Limitations under the License.
-->
<project>
<target name="container_restart">
<mkdir dir="bin" />
<exec dir="." executable="gcc">
<arg line="-o bin/restart_docker_container restart_docker_container.c" />
</exec>
</target>
<target name="clean">
<delete dir="bin" />
</target>
<target name="build" depends="container_restart">
</target>
</project>

View File

@ -1,4 +0,0 @@
*.pyc
build
dist
*.egg-info

View File

@ -1,31 +0,0 @@
<!--
Copyright IBM Corp. 2015, 2015 All Rights Reserved
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
Limitations under the License.
-->
<project>
<target name="clean">
<delete dir="build" />
<delete dir="dist" />
<delete dir="storlet_daemon_factory.egg-info" />
</target>
<target name="build">
<exec executable="python" dir="." failonerror="true">
<arg value="setup.py" />
<arg value="bdist" />
</exec>
</target>
</project>

View File

@ -1,22 +0,0 @@
#!/usr/bin/python
'''-------------------------------------------------------------------------
Copyright IBM Corp. 2015, 2015 All Rights Reserved
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
Limitations under the License.
-------------------------------------------------------------------------'''
from setuptools import setup, find_packages
setup(name='storlets_agent',
version='1.0',
packages=find_packages(),
scripts=['bin/storlets-daemon-factory', 'bin/storlets-daemon']
)

View File

@ -1,31 +0,0 @@
<!--
Copyright IBM Corp. 2015, 2015 All Rights Reserved
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
Limitations under the License.
-->
<project>
<target name="clean">
<delete dir="build" />
<delete dir="dist" />
<delete dir="storlets.egg-info" />
</target>
<target name="build">
<exec executable="python" dir="." failonerror="true">
<arg value="setup.py" />
<arg value="bdist" />
</exec>
</target>
</project>

View File

@ -1,30 +0,0 @@
"""-------------------------------------------------------------------------
Copyright IBM Corp. 2015, 2015 All Rights Reserved
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
Limitations under the License.
-------------------------------------------------------------------------"""
from setuptools import setup, find_packages
paste_factory = ['storlet_handler = '
'storlet_middleware.storlet_handler:filter_factory']
gateways = [
'stub = storlet_gateway.gateways.stub:StorletGatewayStub',
'docker = storlet_gateway.gateways.docker:StorletGatewayDocker']
setup(name='storlets',
version='1.0',
packages=find_packages(),
entry_points={'paste.filter_factory': paste_factory,
'storlets.gateways': gateways}
)

View File

@ -20,7 +20,7 @@
<target name="java">
<mkdir dir="bin" />
<javac srcdir="src" destdir="bin"
classpath="../../../Engine/SCommon/bin/SCommon.jar"
classpath="../../../src/java/SCommon/bin/SCommon.jar"
includeantruntime="false" />
</target>
<target name="text">

View File

@ -22,7 +22,7 @@
<target name="java">
<mkdir dir="bin" />
<javac srcdir="src" destdir="bin"
classpath="../../../Engine/SCommon/bin/SCommon.jar"
classpath="../../../src/java/SCommon/bin/SCommon.jar"
includeantruntime="false" />
</target>

View File

@ -20,7 +20,7 @@
<target name="java">
<mkdir dir="bin" />
<javac srcdir="src" destdir="bin"
classpath="../../../Engine/SCommon/bin/SCommon.jar"
classpath="../../../src/java/SCommon/bin/SCommon.jar"
includeantruntime="false" />
</target>
<target name="csrc" depends="jar">

View File

@ -20,7 +20,7 @@
<target name="java">
<mkdir dir="bin" />
<javac srcdir="src" destdir="bin"
classpath="../../../Engine/SCommon/bin/SCommon.jar"
classpath="../../../src/java/SCommon/bin/SCommon.jar"
includeantruntime="false" />
</target>
<target name="jar" depends="java">

View File

@ -21,7 +21,7 @@
<target name="java">
<mkdir dir="bin" />
<javac srcdir="src" destdir="bin"
classpath="../../../Engine/SCommon/bin/SCommon.jar"
classpath="../../../src/java/SCommon/bin/SCommon.jar"
includeantruntime="false" />
</target>

View File

@ -24,7 +24,7 @@
<target name="java">
<mkdir dir="bin" />
<javac srcdir="src" destdir="bin"
classpath="../../../Engine/SCommon/bin/SCommon.jar"
classpath="../../../src/java/SCommon/bin/SCommon.jar"
includeantruntime="false" />
</target>

View File

@ -23,7 +23,7 @@
<target name="java">
<mkdir dir="bin" />
<javac srcdir="src" destdir="bin"
classpath="../../../Engine/SCommon/bin/SCommon.jar"
classpath="../../../src/java/SCommon/bin/SCommon.jar"
includeantruntime="false" />
</target>

View File

@ -21,7 +21,7 @@
<javac srcdir="src/org/openstack/storlet/testmetadatastorlet" destdir="bin" includeantruntime="false">
<classpath>
<pathelement
path="../../../Engine/SCommon/bin/SCommon.jar"/>
path="../../../src/java/SCommon/bin/SCommon.jar"/>
</classpath>
</javac>
<jar destfile="bin/testmetadatastorlet-1.0.jar"

View File

@ -21,7 +21,7 @@
<javac srcdir="src/org/openstack/storlet/test" destdir="bin" includeantruntime="false">
<classpath>
<pathelement
path="../../../Engine/SCommon/bin/SCommon.jar"/>
path="../../../src/java/SCommon/bin/SCommon.jar"/>
</classpath>
</javac>
<jar destfile="bin/test-10.jar"

View File

@ -22,7 +22,7 @@
<mkdir dir="bin" />
<javac srcdir="src" destdir="bin" includeantruntime="false">
<classpath>
<pathelement path="../../../Engine/SCommon/bin/SCommon.jar"/>
<pathelement path="../../../src/java/SCommon/bin/SCommon.jar"/>
</classpath>
</javac>
</target>

View File

@ -1,6 +1,6 @@
#!/usr/bin/env python
import sys
from storlet_daemon.daemon import main
from storlets.agent.daemon.manager import main
if __name__ == "__main__":

View File

@ -1,6 +1,6 @@
#!/usr/bin/env python
import sys
from storlet_daemon_factory.daemon_factory import main
from storlets.agent.daemon_factory.manager import main
if __name__ == "__main__":

View File

@ -17,11 +17,20 @@
<project>
<!-- Storlets Engine build /-->
<target name="build_engine">
<ant dir="Engine" target="build" />
<ant dir="src/java" target="build" />
</target>
<target name="clean_engine">
<ant dir="Engine" target="clean" />
<ant dir="src/java" target="clean" />
</target>
<!-- Storlets Engine install /-->
<target name="install_engine">
<ant dir="src/java" target="install" />
</target>
<target name="uninstall_engine">
<ant dir="src/java" target="uninstall" />
</target>
<!-- Storlets Samples build -->
@ -35,9 +44,11 @@
<!-- Overall build -->
<target name="build" depends="build_engine, build_storlets" />
<target name="clean" depends="clean_engine, clean_storlets">
<delete dir="bin" />
</target>
<target name="clean" depends="clean_engine, clean_storlets" />
<!-- Overall install -->
<target name="install" depends="install_engine" />
<target name="uninstall" depends="uninstall_engine" />
<!-- Deploy -->
<!-- To execute the below tasks you must have:
@ -49,7 +60,7 @@
<arg value="-s"/>
<arg value="-i"/>
<arg value="storlets_dynamic_inventory.py"/>
<arg value="host_storlet_engine.yml"/>
<arg value="host_side_storlet_engine.yml"/>
</exec>
</target>
@ -63,5 +74,4 @@
</target>
<target name="deploy" depends="deploy_host_engine, deploy_container_engine"/>
</project>

View File

@ -130,17 +130,21 @@ We need the following for Docker
sed -i '$acomplete -F _docker docker' /etc/bash_completion.d/docker
update-rc.d docker defaults
Get and build the storlets code
-------------------------------
Get and install the storlets code
---------------------------------
::
cd $HOME
git clone https://github.com/openstack/storlets.git
cd storlets
ant build
sudo ./install_libs.sh
sudo python setup.py install
tar czf /tmp/storlets.tar.gz .
cd -
.. note:: You don't need sudo for 'python setup.py install' when installing the storlets package into your python virtualenv.
Build the Docker image to be used for running storlets
------------------------------------------------------
Step 1: Create a working space for building the docker images, e.g.
@ -156,11 +160,11 @@ Step 2: Create a Docker image with Java
mkdir -p $HOME/docker_repos/ubuntu_14.04_jre8
cd $HOME/docker_repos/ubuntu_14.04_jre8
cp $HOME/storlets/Engine/dependencies/logback-classic-1.1.2.jar .
cp $HOME/storlets/Engine/dependencies/logback-core-1.1.2.jar .
cp $HOME/storlets/Engine/dependencies/slf4j-api-1.7.7.jar .
cp $HOME/storlets/Engine/dependencies/json_simple-1.1.jar .
cp $HOME/storlets/Engine/dependencies/logback.xml .
cp $HOME/storlets/src/java/dependencies/logback-classic-1.1.2.jar .
cp $HOME/storlets/src/java/dependencies/logback-core-1.1.2.jar .
cp $HOME/storlets/src/java/dependencies/slf4j-api-1.7.7.jar .
cp $HOME/storlets/src/java/dependencies/json_simple-1.1.jar .
cp $HOME/storlets/src/java/dependencies/logback.xml .
cd -
Create the file: $HOME/docker_repos/ubuntu_14.04_jre8/Dockerfile
@ -211,14 +215,16 @@ Step 3: Augment the above created image with the storlets stuff
mkdir -p $HOME/docker_repos/ubuntu_14.04_jre8_storlets
cd $HOME/docker_repos/ubuntu_14.04_jre8_storlets
cp $HOME/storlets/Engine/SBus/SBusJavaFacade/bin/libjsbus.so .
cp $HOME/storlets/Engine/SBus/SBusJavaFacade/bin/SBusJavaFacade.jar .
cp $HOME/storlets/Engine/SBus/SBusPythonFacade/dist/SBusPythonFacade-1.0.linux-x86_64.tar.gz .
cp $HOME/storlets/Engine/SBus/SBusTransportLayer/bin/sbus.so .
cp $HOME/storlets/Engine/SDaemon/bin/SDaemon.jar .
cp $HOME/storlets/Engine/SCommon/bin/SCommon.jar .
cp $HOME/storlets/Engine/agent/dist/storlets_agent-1.0.linux-x86_64.tar.gz .
cp $HOME/storlets/src/java/SBus/bin/libjsbus.so .
cp $HOME/storlets/src/java/SBus/bin/SBusJavaFacade.jar .
cp $HOME/storlets/src/java/SDaemon/bin/SDaemon.jar .
cp $HOME/storlets/src/java/SCommon/bin/SCommon.jar .
cp $HOME/storlets/src/c/sbus/libsbus.so .
cp $HOME/storlets/install/storlets/roles/docker_storlet_engine_image/files/init_container.sh .
cp $HOME/storlets/install/storlets/roles/docker_storlet_engine_image/files/logback.xml .
wget https://bootstrap.pypa.io/get-pip.py
cp /tmp/storlets.tar.gz .
tar -xvf storlets.tar.gz
cd -
Create the file: $HOME/docker_repos/ubuntu_14.04_jre8_storlets/Dockerfile
@ -233,31 +239,31 @@ with the following content:
RUN [ "groupadd", "-g", "1003", "swift" ]
RUN [ "useradd", "-u" , "1003", "-g", "1003", "swift" ]
ADD SBusPythonFacade-1.0.linux-x86_64.tar.gz /
RUN chmod -R 0755 /usr/local/lib/python2.7/dist-packages/SBusPythonFacade*
# Copy files
COPY ["logback.xml", "init_container.sh", "/opt/storlets/"]
COPY sbus.so /usr/local/lib/python2.7/dist-packages/
RUN ["chmod", "0755", "/usr/local/lib/python2.7/dist-packages/sbus.so"]
COPY SBusJavaFacade.jar /opt/storlets/
RUN ["chmod", "0744", "/opt/storlets/SBusJavaFacade.jar"]
COPY libjsbus.so /opt/storlets/
RUN ["chmod", "0755", "/opt/storlets/libjsbus.so"]
COPY SDaemon.jar /opt/storlets/
RUN ["chmod", "0744", "/opt/storlets/SDaemon.jar"]
COPY SCommon.jar /opt/storlets/
RUN ["chmod", "0744", "/opt/storlets/SCommon.jar"]
ADD storlets_agent-1.0.linux-x86_64.tar.gz /
RUN ["chmod", "0755", "/usr/local/bin/storlets-daemon-factory"]
COPY init_container.sh /opt/storlets/
RUN ["chmod", "0744", "/opt/storlets/logback.xml"]
RUN ["chmod", "0755", "/opt/storlets/init_container.sh"]
CMD ["prod", "/mnt/channels/factory_pipe","DEBUG"]
# Install c java resources
COPY ["libsbus.so", "/usr/local/lib/storlets/"]
# Install storlets java resources
COPY ["SBusJavaFacade.jar", "libjsbus.so", "SDaemon.jar", "SCommon.jar", "/opt/storlets/"]
# Install pip
COPY ["get-pip.py", "/opt/storlets"]
RUN ["python", "/opt/storlets/get-pip.py"]
ENV PYTHONWARNINGS="ignore:a true SSLContext object"
# Install python codes
COPY ["storlets", "/opt/storlets/"]
RUN cd /opt/storlets/ && \
pip install -r requirements.txt && \
python setup.py install
CMD ["prod", "/mnt/channels/factory_pipe", "DEBUG"]
ENTRYPOINT ["/opt/storlets/init_container.sh"]
Build the image
@ -303,26 +309,6 @@ Build the image
sudo docker build -q -t <account id> .
cd -
Install the storlets middleware components
------------------------------------------
Install the SBus components used for comuunication between the host and container
::
cp $HOME/storlets/SBusPythonFacade/dist/SBusPythonFacade-1.0.linux-x86_64.tar.gz /tmp
cd /tmp
sudo tar -C / -xvf SBusPythonFacade-1.0.linux-x86_64.tar.gz
sudo cp $HOME/storlets/Engine/SBus/SBusTransportLayer/bin/sbus.so /usr/local/lib/python2.7/dist-packages/sbus.so
sudo chown $USER:$USER /usr/local/lib/python2.7/dist-packages/sbus.so
Install the swift middleware
::
cp $HOME/storlets/Engine/swift/dist/storlets-1.0.linux-x86_64.tar.gz /tmp
cd /tmp
sudo tar -C / -xvf storlets-1.0.linux-x86_64.tar.gz
Create the storlets run time environment
----------------------------------------
Create the run time directory
@ -341,10 +327,10 @@ require root privileges.
mkdir $STORLETS_HOME/scripts
cd STORLETS_HOME/scripts
cp $HOME/storlets/Engine/SMScripts/bin/restart_docker_container .
cp $HOME/scripts/restart_docker_container .
sudo chown root:root restart_docker_container
sudo chmod 04755 restart_docker_container
cp $HOME/storlets/Engine/SMScripts/send_halt_cmd_to_daemon_factory.py .
cp $HOME/scripts/send_halt_cmd_to_daemon_factory.py .
sudo chown root:root send_halt_cmd_to_daemon_factory.py
sudo chmod 04755 send_halt_cmd_to_daemon_factory.py

View File

@ -109,7 +109,7 @@ storlets with EC and encryption. Valid values are true / false
Install
-------
to perform the installation follow these steps:
To perform the installation follow these steps:
#. Create a hosts file as described above
#. Edit the file install/storlets/common.yml according to the above

View File

@ -1,18 +0,0 @@
#---------------------------------------------------------------------------
# Copyright IBM Corp. 2015, 2015 All Rights Reserved
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# Limitations under the License.
#---------------------------------------------------------------------------
- hosts: storlet-mgmt
roles:
- role: create_default_tenant

View File

@ -1,18 +0,0 @@
#---------------------------------------------------------------------------
# Copyright IBM Corp. 2015, 2015 All Rights Reserved
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# Limitations under the License.
#---------------------------------------------------------------------------
- hosts: docker
roles:
- role: docker_base_jre_image

View File

@ -1,19 +0,0 @@
#---------------------------------------------------------------------------
# Copyright IBM Corp. 2015, 2015 All Rights Reserved
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# Limitations under the License.
#---------------------------------------------------------------------------
- hosts: docker
roles:
- role: docker_client
- role: docker_registry

View File

@ -1,18 +0,0 @@
#---------------------------------------------------------------------------
# Copyright IBM Corp. 2015, 2015 All Rights Reserved
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# Limitations under the License.
#---------------------------------------------------------------------------
- hosts: docker
roles:
- role: docker_storlet_engine_image

View File

@ -1,21 +0,0 @@
#---------------------------------------------------------------------------
# Copyright IBM Corp. 2015, 2015 All Rights Reserved
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# Limitations under the License.
#---------------------------------------------------------------------------
- hosts: storlet-proxy[0]
tasks:
- fetch: src=/etc/swift/proxy-server.conf dest=/tmp/ flat=yes
- hosts: storlet-storage
tasks:
- copy: src=/tmp/proxy-server.conf dest=/etc/swift/proxy-server.conf

View File

@ -1,17 +0,0 @@
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# Limitations under the License.
#---------------------------------------------------------------------------
- hosts: storlet-proxy:storlet-storage
vars_files:
- [deploy/common.yml]
tasks:
- replace: dest=/etc/swift/proxy-server.conf regexp='^storlet_execute_on_proxy_only.*$' replace='storlet_execute_on_proxy_only = {{ run_on_proxy }}' owner="{{ swift_run_time_user }}" group="{{ swift_run_time_group }}" mode=644
- replace: dest=/etc/swift/object-server.conf regexp='^storlet_execute_on_proxy_only.*$' replace='storlet_execute_on_proxy_only = {{ run_on_proxy }}' owner="{{ swift_run_time_user }}" group="{{ swift_run_time_group }}" mode=644
- shell: swift-init all restart

View File

@ -1,23 +0,0 @@
#---------------------------------------------------------------------------
# Copyright IBM Corp. 2015, 2015 All Rights Reserved
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# Limitations under the License.
#---------------------------------------------------------------------------
- include: cluster_check.yml
- include: docker_repository.yml
- include: docker_base_storlet_images.yml
- include: docker_storlet_engine_image.yml
- include: storlet_mgmt.yml
- include: fetch_proxy_conf.yml
- include: host_storlet_engine.yml
- include: create_default_tenant.yml

View File

@ -1,39 +0,0 @@
#---------------------------------------------------------------------------
# Copyright IBM Corp. 2015, 2015 All Rights Reserved
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# Limitations under the License.
#---------------------------------------------------------------------------
- hosts: all
vars_files:
- [deploy/common.yml]
roles:
- role: remove_docker
- hosts: storlet-proxy:storlet-storage
vars_files:
- [deploy/common.yml]
roles:
- role: reconfigure_swift
engine_install_dir: /tmp/swift/
- hosts: storlet-proxy:storlet-storage
vars_files:
- [deploy/common.yml]
roles:
- role: remove_host_storlet_engine
- hosts: docker
vars_files:
- [deploy/common.yml]
roles:
- role: remove_docker_registry

View File

@ -13,12 +13,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
- hosts: docker:storlet-proxy:storlet-storage
roles:
- role: docker_client
- hosts: docker
roles:
- role: docker_client
- role: docker_registry
- role: docker_base_jre_image
- role: docker_storlet_engine_image

View File

@ -23,13 +23,7 @@
- hosts: storlet-proxy:storlet-storage
roles:
- role: docker_client
- role: host_storlet_engine_install
swift_middleware_dir: "{{ storlet_source_dir }}/Engine/swift"
swift_middleware_egg_dist: storlets-1.0.linux-x86_64.tar.gz
SBus_dir: "{{ storlet_source_dir }}/Engine/SBus"
SBusTransportLayer_dir: "{{ SBus_dir }}/SBusTransportLayer"
sbus_so: bin/sbus.so
SBusPythonFacade_dir: "{{ SBus_dir }}/SBusPythonFacade/"
SBusPythonFacade_egg_dist: sbus-1.0.linux-x86_64.tar.gz
- role: host_storlet_engine_configure
engine_install_dir: /tmp/swift/

View File

@ -1,27 +0,0 @@
#---------------------------------------------------------------------------
# Copyright IBM Corp. 2015, 2015 All Rights Reserved
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# Limitations under the License.
#---------------------------------------------------------------------------
- hosts: storlet-proxy:storlet-storage
roles:
- role: docker_client
- role: host_storlet_engine
swift_middleware_dir: "{{ storlet_source_dir }}/Engine/swift"
swift_middleware_egg_dist: storlets-1.0.linux-x86_64.tar.gz
engine_install_dir: /tmp/swift/
SBus_dir: "{{ storlet_source_dir }}/Engine/SBus"
SBusTransportLayer_dir: "{{ SBus_dir }}/SBusTransportLayer"
sbus_so: bin/sbus.so
SBusPythonFacade_dir: "{{ SBus_dir }}/SBusPythonFacade/"
SBusPythonFacade_egg_dist: sbus-1.0.linux-x86_64.tar.gz

View File

@ -1,7 +1,7 @@
#!/bin/bash
set -eu
# echo "Running hosts cluster_check playbook"
echo "Running hosts cluster_check playbook"
ansible-playbook -s -i storlets_dynamic_inventory.py cluster_check.yml
echo "Running hosts docker_cluster playbook"

View File

@ -40,8 +40,12 @@ sudo apt-get install -y ant
sudo apt-get install -y python
sudo apt-get install -y python-setuptools
# Build Storlets
ant build
sudo ./install_libs.sh
# Note(takashi): Currently we need to use tar ball to storelet repo
# files, to avoid slow perfomance of copy module
# when dealing with a directory
tar czf /tmp/storlets.tar.gz .
# The rest of the operations are all relative to
# install/storlets/

View File

@ -18,7 +18,8 @@
register: tenant_id
- name: Print the tenant id whose container we are processing
debug: msg="{{ tenant_id.stdout_lines[0] }}"
debug:
msg: "{{ tenant_id.stdout_lines[0] }}"
- name: Enable the default tenant for storlets
command: swift

View File

@ -21,6 +21,7 @@ MAINTAINER {{ base_image_maintainer }}
# to prevent docker images from including apt cache file.
RUN apt-get update && \
apt-get install python -y && \
apt-get install git -y && \
apt-get install software-properties-common -y && \
add-apt-repository ppa:openjdk-r/ppa -y && \
apt-get update && \

View File

@ -16,37 +16,60 @@
- name: Test for missing packages
shell: dpkg --get-selections | grep aufs-tools | wc -l
register: aufs
become: yes
- shell: dpkg --get-selections | grep linux-image-extra | wc -l
register: linux_image_extra
become: yes
- shell: dpkg --get-selections | grep docker-engine | awk {'print $2'} | grep -w install | wc -l
register: docker_engine
become: yes
- command: uname -r
register: uname
- apt: name=linux-image-extra-{{ uname.stdout_lines[0] }} update-cache=yes force=yes
- apt:
name: linux-image-extra-{{ uname.stdout_lines[0] }}
update-cache: yes
force: yes
when: "'0' in linux_image_extra.stdout_lines[0]"
- apt: name=aufs-tools=1:3.2+20130722-1.1 update-cache=yes force=yes
- apt:
name: aufs-tools=1:3.2+20130722-1.1
update-cache: yes
force: yes
when: "'0' in aufs.stdout_lines[0]"
become: yes
- name: Set Docker Ubuntu Repo Key
apt_key: keyserver=hkp://p80.pool.sks-keyservers.net:80 id=58118E89F3A912897C070ADBF76221572C52609D state=present
apt_key:
keyserver: hkp://p80.pool.sks-keyservers.net:80
id: 58118E89F3A912897C070ADBF76221572C52609D
state: present
ignore_errors: True
when: "'0' in docker_engine.stdout_lines[0]"
become: yes
- name: Install Apt Transport HTTPS
apt: name=apt-transport-https state=present update-cache=yes force=yes
when: "'0' in docker_engine.stdout_lines[0]"
become: yes
- name: Add Docker Apt repository
apt_repository: repo='deb https://apt.dockerproject.org/repo ubuntu-trusty main' state=present
apt_repository:
repo: 'deb https://apt.dockerproject.org/repo ubuntu-trusty main'
state: present
when: "'0' in docker_engine.stdout_lines[0]"
become: yes
- name: Install Docker Client
apt: name=docker-engine state=present update-cache=yes force=yes
apt:
name: docker-engine
state: present
update-cache: yes
force: yes
when: "'0' in docker_engine.stdout_lines[0]"
become: yes
# There seem to be a bug in the installation where config files are not installed.
# Might be a bug in the un-install where docket thinks it is upgrading.
@ -101,7 +124,9 @@
- name: Stop Docker Daemon
shell: bash -c "/usr/sbin/service docker stop"
become: yes
ignore_errors: yes
- name: Restart Docker Daemon
shell: bash -c "/usr/sbin/service docker start"
become: yes

View File

@ -15,46 +15,52 @@
- name: Create Repository for building the image
file:
path: /data/registry/repositories/{{ base_os_image }}_jre8_storlets
path: "/data/registry/repositories/{{ base_os_image }}_jre8_storlets"
state: directory
mode: 0755
owner: root
- name: Copy SBus resources
copy:
src: "{{ storlet_source_dir }}/{{ item }}"
dest: /data/registry/repositories/{{ base_os_image }}_jre8_storlets
with_items:
- "Engine/SBus/SBusJavaFacade/bin/libjsbus.so"
- "Engine/SBus/SBusJavaFacade/bin/SBusJavaFacade.jar"
- "Engine/SBus/SBusPythonFacade/dist/sbus-1.0.linux-x86_64.tar.gz"
- "Engine/SBus/SBusTransportLayer/bin/sbus.so"
- name: Get pip install script
get_url:
url: "https://bootstrap.pypa.io/get-pip.py"
dest: "/data/registry/repositories/{{ base_os_image }}_jre8_storlets"
- name: Copy SDaemon resources
- name: Copy c resources
copy:
src: "{{ storlet_source_dir }}/{{ item }}"
dest: /data/registry/repositories/{{ base_os_image }}_jre8_storlets
dest: "/data/registry/repositories/{{ base_os_image }}_jre8_storlets"
with_items:
- "Engine/SDaemon/bin/SDaemon.jar"
- "src/c/sbus/libsbus.so"
- name: Copy SCommon resources
- name: Copy java resources
copy:
src: "{{ storlet_source_dir }}/{{ item }}"
dest: /data/registry/repositories/{{ base_os_image }}_jre8_storlets
src: "{{ storlet_source_dir }}/{{ item }}"
dest: "/data/registry/repositories/{{ base_os_image }}_jre8_storlets"
with_items:
- "Engine/SCommon/bin/SCommon.jar"
- "src/java/SBus/bin/libjsbus.so"
- "src/java/SBus/bin/SBusJavaFacade.jar"
- "src/java/SCommon/bin/SCommon.jar"
- "src/java/SDaemon/bin/SDaemon.jar"
- name: Copy Daemon Factory resources
- name: Copy python code tar ball
copy:
src: "{{ storlet_source_dir }}/{{ item }}"
dest: /data/registry/repositories/{{ base_os_image }}_jre8_storlets
with_items:
- "Engine/agent/dist/storlets_agent-1.0.linux-x86_64.tar.gz"
src: "/tmp/storlets.tar.gz"
dest: "/tmp"
- name: Create a directory for python code
file:
path: "/data/registry/repositories/{{ base_os_image }}_jre8_storlets/storlets"
state: directory
- name: Unarchive python code tar ball
unarchive:
src: "/tmp/storlets.tar.gz"
dest: "/data/registry/repositories/{{ base_os_image }}_jre8_storlets/storlets"
- name: Copy Init Container Script
copy:
src: "{{item}}"
dest: /data/registry/repositories/{{ base_os_image }}_jre8_storlets
dest: "/data/registry/repositories/{{ base_os_image }}_jre8_storlets/"
with_items:
- init_container.sh
- logback.xml

View File

@ -17,20 +17,33 @@ FROM {{ inventory_hostname }}:{{ docker_registry_port }}/{{base_os_image}}_jre8
MAINTAINER {{ base_image_maintainer }}
# Create swift user/group
RUN [ "groupadd", "-g", "{{ swift_group_id }}", "swift" ]
RUN [ "useradd", "-u" , "{{ swift_user_id }}", "-g", "{{ swift_group_id }}", "swift" ]
ADD sbus-1.0.linux-x86_64.tar.gz /
RUN chmod -R 0755 /usr/local/lib/python2.7/dist-packages/sbus*
# Copy files
COPY ["logback.xml", "init_container.sh", "/opt/storlets/"]
COPY sbus.so /usr/local/lib/python2.7/dist-packages/
RUN ["chmod", "0744", "/opt/storlets/logback.xml"]
RUN ["chmod", "0755", "/opt/storlets/init_container.sh"]
COPY ["SBusJavaFacade.jar", "libjsbus.so", "SDaemon.jar", "SCommon.jar", "logback.xml", "init_container.sh", "/opt/storlets/"]
# Install c java resources
COPY ["libsbus.so", "/usr/local/lib/storlets/"]
ADD storlets_agent-1.0.linux-x86_64.tar.gz /
# Install storlets java resources
COPY ["SBusJavaFacade.jar", "libjsbus.so", "SDaemon.jar", "SCommon.jar", "/opt/storlets/"]
RUN ["chmod", "0744", "/opt/storlets/SBusJavaFacade.jar", "/opt/storlets/SDaemon.jar", "/opt/storlets/SCommon.jar", "/opt/storlets/logback.xml"]
RUN ["chmod", "0755", "/usr/local/bin/storlets-daemon-factory", "/opt/storlets/libjsbus.so", "/usr/local/lib/python2.7/dist-packages/sbus.so", "/opt/storlets/init_container.sh"]
# Install pip
COPY ["get-pip.py", "/opt/storlets"]
RUN ["python", "/opt/storlets/get-pip.py"]
ENV PYTHONWARNINGS="ignore:a true SSLContext object"
# Install python codes
COPY ["storlets", "/opt/storlets/"]
RUN cd /opt/storlets/ && \
pip install -r requirements.txt && \
python setup.py install
CMD ["prod", "/mnt/channels/factory_pipe", "DEBUG"]
CMD ["prod", "/mnt/channels/factory_pipe","DEBUG"]
ENTRYPOINT ["/opt/storlets/init_container.sh"]

View File

@ -1,20 +0,0 @@
#---------------------------------------------------------------------------
# Copyright IBM Corp. 2015, 2015 All Rights Reserved
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# Limitations under the License.
#---------------------------------------------------------------------------
- name: restart swift
shell: swift-init --run-dir="{{swift_run_time_dir}}" all {{ item }}
with_items:
- stop
- start

View File

@ -1,125 +0,0 @@
#---------------------------------------------------------------------------
# Copyright IBM Corp. 2015, 2015 All Rights Reserved
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# Limitations under the License.
#---------------------------------------------------------------------------
- name: copy python facade tar locally
copy: src={{ SBusPythonFacade_dir }}/dist/{{ SBusPythonFacade_egg_dist }} dest=/tmp
- name: test SBusPythonFacade state
stat: path="/{{python_dist_packages_dir}}/SBusPythonFacade"
register: st
- name: remove python facade if exists
file: path="/{{python_dist_packages_dir}}/SBusPythonFacade" state=absent
when: st.stat.exists
- name: test SBusPythonFacade egg state
stat: path="/{{python_dist_packages_dir}}/sbus-1.0-py2.7.egg-info"
register: st
- name: remove python facade egg if exists
file: path="/{{python_dist_packages_dir}}/sbus-1.0-py2.7.egg-info" state=absent
when: st.stat.exists
- name: unarchive python facade
unarchive: owner=root group=root mode=0755 dest=/ src=/tmp/{{ SBusPythonFacade_egg_dist }}
- name: delete python facade tar local copy
file: path=/tmp/{{ SBusPythonFacade_egg_dist }} state=absent
- name: install sbus transport layer on host
copy: src={{ SBusTransportLayer_dir }}/{{ sbus_so }} dest=/{{python_dist_packages_dir}} owner=root group=root mode=0755
- name: copy swift middleware tar locally
copy: src={{ swift_middleware_dir }}/dist/{{ swift_middleware_egg_dist }} dest=/tmp
- name: test swift_middleware state
stat: path="/{{python_dist_packages_dir}}/storlet_middleware"
register: st
- name: remove storlet_middleware if exists
file: path="/{{python_dist_packages_dir}}/storlet_middleware" state=absent
when: st.stat.exists
- name: test storlet_gateway state
stat: path="/{{python_dist_packages_dir}}/storlet_gateway"
register: st
- name: remove storlet_gateway if exists
file: path="/{{python_dist_packages_dir}}/storlet_gateway" state=absent
when: st.stat.exists
- name: test storlets egg state
stat: path="/{{python_dist_packages_dir}}/storlets-1.0-py2.7.egg-info"
register: st
- name: remove storlets egg if exists
file: path="/{{python_dist_packages_dir}}/storlets-1.0-py2.7.egg-info" state=absent
when: st.stat.exists
- name: unarchive swift middleware
unarchive: owner=root group=root mode=0755 dest=/ src=/tmp/{{ swift_middleware_egg_dist }}
- name: delete swift middleware tar local copy
file: path=/tmp/{{ swift_middleware_egg_dist }} state=absent
- stat: path={{ docker_device }}/scripts
register: st
- name: Create scripts directory
file: path="{{ docker_device }}/scripts/" state=directory owner="{{ swift_run_time_user }}" group="{{ swift_run_time_group }}" mode=0755
when: not st.stat.exists
- name: install runtime scripts used by swift middleware
copy:
src: "{{ storlet_source_dir }}/{{ item }}"
dest: "{{ docker_device }}/scripts/"
mode: "04755"
owner: root
group: root
with_items:
- "Engine/SMScripts/bin/restart_docker_container"
- "Engine/SMScripts/send_halt_cmd_to_daemon_factory.py"
- name: create installation directory
file: path={{ engine_install_dir }} state=directory recurse=yes
- name: Copy Swift middleware conf template
template:
src: ../../common_templates/swift_middleware_conf
dest: "{{ engine_install_dir }}"
owner: "{{ swift_run_time_user }}"
group: "{{ swift_run_time_group }}"
mode: "0644"
- name: Copy install script
copy:
src: "{{ item }}"
dest: "{{ engine_install_dir }}"
with_items:
- ../../common_files/swift_config.py
- name: Copy gateway conf template
template:
src: ../../common_templates/storlet-docker-gateway.conf-sample
dest: "{{ storlet_gateway_conf_file }}"
owner: "{{ swift_run_time_user }}"
group: "{{ swift_run_time_group }}"
mode: 0644
- name: configure swift
shell: chdir={{ engine_install_dir }}
python swift_config.py install swift_middleware_conf "{{ swift_run_time_user }}"
- name: restart swift
shell: swift-init --run-dir="{{swift_run_time_dir}}" all restart
ignore_errors: yes

View File

@ -25,7 +25,9 @@
mode: 0644
- name: copy install script
copy: src="{{ item }}" dest={{ engine_install_dir }}
copy:
src: "{{ item }}"
dest: "{{ engine_install_dir }}"
with_items:
- ../../common_files/swift_config.py

View File

@ -13,23 +13,52 @@
# See the License for the specific language governing permissions and
# limitations under the License.
- name: install python facade on host
unarchive: owner=root group=root mode=0755 dest=/ src={{ SBusPythonFacade_dir }}/dist/{{ SBusPythonFacade_egg_dist }}
- name: Copy python code tar ball
copy:
src: "/tmp/storlets.tar.gz"
dest: "/tmp"
- name: Create temp directory for python code
file:
path: "/tmp/storlets_src"
state: directory
- name: Unarchive python code tar ball
unarchive:
src: "/tmp/storlets.tar.gz"
dest: "/tmp/storlets_src/"
- name: Change permission of install scripts
file:
path: "/tmp/storlets_src/install_libs.sh"
mode: "0755"
- name: Install C/Java codes on remote host
shell: "/tmp/storlets_src/install_libs.sh"
args:
chdir: "/tmp/storlets_src"
become: true
- name: install sbus transport layer on host
copy: src={{ SBusTransportLayer_dir }}/{{ sbus_so }} dest=/{{python_dist_packages_dir}} owner=root group=root mode=0755
- name: Install python requirements
shell: "pip install -r /tmp/storlets_src/requirements.txt"
args:
chdir: "/tmp/storlets_src"
become: true
- name: install swift middleware on host
unarchive: owner=root group=root mode=0755 dest=/ src={{ swift_middleware_dir }}/dist/{{ swift_middleware_egg_dist }}
- name: Install Python codes on remote host
command: "python setup.py install"
args:
chdir: "/tmp/storlets_src"
become: true
- stat: path={{ docker_device }}/scripts
register: st
- name: Create scripts directory
file: path="{{ docker_device }}/scripts/" state=directory owner="{{ swift_run_time_user }}" group="{{ swift_run_time_group }}" mode=0755
when: not st.stat.exists
file:
path: "{{ docker_device }}/scripts/"
state: directory
owner: "{{ swift_run_time_user }}"
group: "{{ swift_run_time_group }}"
mode: 0755
become: true
- name: install runtime scripts used by swift middleware
copy:
@ -39,6 +68,12 @@
owner: root
group: root
with_items:
- "Engine/SMScripts/bin/restart_docker_container"
- "Engine/SMScripts/send_halt_cmd_to_daemon_factory.py"
- "scripts/restart_docker_container"
- "scripts/send_halt_cmd_to_daemon_factory.py"
become: true
- name: Remove code repository from remote host
file:
path: "/tmp/storlets_src"
state: absent
become: true

View File

@ -1,68 +0,0 @@
#---------------------------------------------------------------------------
# Copyright IBM Corp. 2015, 2015 All Rights Reserved
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# Limitations under the License.
#---------------------------------------------------------------------------
- stat: path=/usr/bin/docker
register: st
- name: Stop all running containers
shell: docker ps -a -q
register: docker
when: st.stat.exists
- shell: docker stop {{ docker.stdout_lines[0] }}
when: st.stat.exists and docker is defined and stdout_lines in docker
- name: Delete all running containers
shell: docker rm "{{ docker.stdout_lines[0] }}"
when: st.stat.exists and docker is defined and stdout_lines in docker
- name: stop the docker service
service: name=docker state=stopped
when: st.stat.exists
- name: uninstall docker
apt: name=lxc-docker state=absent purge=yes
when: st.stat.exists
- name: make sure all docker stuff is removed
shell: apt-get -y autoremove
when: st.stat.exists
- name: cleanup docker runtime
stat: path="{{ docker_device }}/docker"
register: p
- shell: rm -fr "{{ docker_device }}"/docker
when: p.stat.exists
ignore_errors: yes
- stat: path=/var/lib/docker
register: p
- shell: rm -fr /var/lib/docker
when: p.stat.exists
- name: cleanup conf files
stat: path=/etc/default/docker
register: p
- shell: rm -f /etc/default/docker
when: p.stat.exists
- stat: path=/etc/init/docker.conf
register: p
- shell: rm -f /etc/init/docker.conf
when: p.stat.exists
- stat: path=/etc/init.d/docker
register: p
- shell: rm -f /etc/init.d/docker
when: p.stat.exists

View File

@ -1,22 +0,0 @@
#---------------------------------------------------------------------------
# Copyright IBM Corp. 2015, 2015 All Rights Reserved
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# Limitations under the License.
#---------------------------------------------------------------------------
- name: Stop Docker repository container
shell: docker stop registry
ignore_errors: yes
- name: Remove Docker repository container
shell: docker rm registry
ignore_errors: yes

View File

@ -1,29 +0,0 @@
#---------------------------------------------------------------------------
# Copyright IBM Corp. 2015, 2015 All Rights Reserved
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# Limitations under the License.
#---------------------------------------------------------------------------
- name: remove storlet middleware
shell: rm -fr /usr/local/lib/python2.7/dist-packages/storlet*
- name: remove SBus transport layer
shell: rm -fr /usr/local/lib/python2.7/dist-packages/sbus.so
- name: remove run time
shell: rm -fr {{ docker_device }}/{{ item }}
with_items:
- cache
- storlets
- logs
- pipes
- scripts

20
install_libs.sh Executable file
View File

@ -0,0 +1,20 @@
#!/bin/bash
#
# Install required libraries written in c and java
#
# NOTE: The libraries are currently installed under /usr/lib/storlets
# so you may need root priviledge to execute this script
# build scripts
cd scripts
# TODO(takashi): also install them
make
cd -
# install c library
cd src/c/sbus
make && make install
cd -
# Install java library
ant install

1
sbus
View File

@ -1 +0,0 @@
Engine/SBus/SBusPythonFacade/sbus

16
scripts/Makefile Normal file
View File

@ -0,0 +1,16 @@
CC = gcc
CFLAGS =
LDFLAGS =
TARGET = restart_docker_container
SRCS = restart_docker_container.c
OBJS = $(SRCS:.c=.o)
.PHONY: all
all: ${TARGET}
$(TARGET): $(OBJS)
$(CC) ${LDFLAGS} -o $@ $^
clean:
rm ${TARGET} ${OBJS}

View File

@ -16,9 +16,9 @@
import os
import sys
from sbus import SBus
from sbus.datagram import SBusDatagram
from sbus.command import SBUS_CMD_HALT
from storlets.sbus import SBus
from storlets.sbus.datagram import SBusDatagram
from storlets.sbus.command import SBUS_CMD_HALT
def print_usage(argv):

View File

@ -16,21 +16,24 @@ classifier =
Programming Language :: Python :: 2
Programming Language :: Python :: 2.7
[pbr]
skip_authors = True
skip_changelog = True
[files]
packages =
storlet_middleware
storlet_gateway
storlet_daemon_factory
storlet_daemon
sbus
storlets
scripts =
bin/storlets-daemon
bin/storlets-daemon-factory
[entry_points]
paste.filter_factory =
storlet_handler = storlet_middleware.storlet_handler:filter_factory
storlet_handler = storlets.swift_middleware.storlet_handler:filter_factory
storlets.gateways =
stub = storlet_gateway.gateways.stub:StorletGatewayStub
docker = storlet_gateway.gateways.docker:StorletGatewayDocker
stub = storlets.gateway.gateways.stub:StorletGatewayStub
docker = storlets.gateway.gateways.docker:StorletGatewayDocker
[build_sphinx]
source-dir = doc/source

24
src/c/sbus/Makefile Normal file
View File

@ -0,0 +1,24 @@
CC = gcc
CFLAGS = -fPIC -Wall -O2
LDFLAGS = -shared
TARGET = libsbus.so
DESTDIR = /usr/local/lib/storlets
SRCS = sbus.c
OBJS = $(SRCS:.c=.o)
.PHONY: all
all: ${TARGET}
$(TARGET): $(OBJS)
$(CC) ${LDFLAGS} -o $@ $^
install: $(TARGET)
mkdir -p ${DESTDIR}
cp ${TARGET} ${DESTDIR}
uninstall:
rm -rf ${DESTDIR}
clean:
rm -f ${TARGET} ${OBJS}

View File

@ -13,6 +13,8 @@
Limitations under the License.
-->
<project>
<property name="jarname" value="SBusJavaFacade.jar"/>
<property name="libdir" value="/usr/local/lib/storlets"/>
<target name="clean">
<delete dir="bin" />
@ -22,9 +24,9 @@
<target name="java">
<mkdir dir="bin" />
<javac
srcdir="src"
srcdir="src/main"
destdir="bin"
classpath="../../dependencies/json_simple-1.1.jar"
classpath="../dependencies/json_simple-1.1.jar"
includeantruntime="false"/>
</target>
@ -39,8 +41,8 @@
<arg line="-shared -o bin/libjsbus.so -fPIC" />
<arg line="-I/usr/lib/jvm/java-8-openjdk-amd64/include/" />
<arg line="-I/usr/lib/jvm/java-8-openjdk-amd64/include/linux/" />
<arg line="-I../SBusTransportLayer" />
<arg line="SBusJNI.c ../SBusTransportLayer/sbus.c" />
<arg line="-I../../c/sbus/" />
<arg line="SBusJNI.c ../../c/sbus/sbus.c" />
</exec>
</target>
@ -56,4 +58,14 @@
<target name="build" depends="clean, jar">
</target>
<target name="install" depends="build">
<mkdir dir="${libdir}"/>
<copy file="bin/${jarname}" todir="${libdir}"/>
<copy file="bin/libjsbus.so" todir="${libdir}"/>
</target>
<target name="uninstall">
<delete file="${libdir}/${jarname}"/>
</target>
</project>

View File

@ -14,6 +14,9 @@
-->
<project>
<property name="jarname" value="SCommon.jar"/>
<property environment="env"/>
<property name="libdir" value="/usr/local/lib/storlets"/>
<target name="clean">
<delete dir="bin" />
@ -21,9 +24,9 @@
<target name="java">
<mkdir dir="bin" />
<javac srcdir="src" destdir="bin" includeantruntime="false">
<javac srcdir="src/main" destdir="bin" includeantruntime="false">
<classpath>
<pathelement path="../SBus/SBusJavaFacade/bin/SBusJavaFacade.jar"/>
<pathelement path="../SBus/bin/SBusJavaFacade.jar"/>
<pathelement path="../dependencies/json_simple-1.1.jar"/>
<pathelement path="../dependencies/slf4j-api-1.7.7.jar"/>
<pathelement path="../dependencies/logback-classic-1.1.2.jar"/>
@ -34,12 +37,12 @@
</target>
<target name="jar" depends="java">
<jar destfile="SCommon.jar" basedir="bin">
<jar destfile="bin/${jarname}" basedir="bin">
<manifest>
<attribute name="Main-Class" value="org.openstack.storlet.daemon.IStorlet" />
<attribute name="Main-Class"
value="org.openstack.storlet.daemon.IStorlet" />
</manifest>
</jar>
<move file="SCommon.jar" todir="bin/" />
</target>
<target name="test">
@ -53,4 +56,13 @@
<target name="build" depends="clean, jar, test">
</target>
<target name="install" depends="build">
<mkdir dir="${libdir}"/>
<copy file="bin/${jarname}" todir="${libdir}"/>
</target>
<target name="uninstall">
<delete file="${libdir}/${jarname}"/>
</target>
</project>

View File

@ -14,16 +14,19 @@
-->
<project>
<property name="jarname" value="SDaemon.jar"/>
<property name="libdir" value="/usr/local/lib/storlets"/>
<target name="clean">
<delete dir="bin" />
</target>
<target name="java">
<mkdir dir="bin" />
<javac srcdir="src" destdir="bin" includeantruntime="false">
<javac srcdir="src/main" destdir="bin" includeantruntime="false">
<classpath>
<pathelement
path="../SBus/SBusJavaFacade/bin/SBusJavaFacade.jar"/>
path="../SBus/bin/SBusJavaFacade.jar"/>
<pathelement
path="../SCommon/bin/SCommon.jar"/>
<pathelement
@ -40,7 +43,7 @@
</target>
<target name="jar" depends="java">
<jar destfile="SDaemon.jar"
<jar destfile="bin/${jarname}"
basedir="bin">
<manifest>
<attribute
@ -48,9 +51,17 @@
value="org.openstack.storlet.daemon.SDaemon" />
</manifest>
</jar>
<move file="SDaemon.jar" todir="bin/" />
</target>
<target name="build" depends="clean, jar">
</target>
<target name="install" depends="build">
<mkdir dir="${libdir}"/>
<copy file="bin/${jarname}" todir="${libdir}"/>
</target>
<target name="uninstall">
<delete file="${libdir}/${jarname}"/>
</target>
</project>

Some files were not shown because too many files have changed in this diff Show More