monasca-transform/devstack

Monasca-transform DevStack Plugin

The Monasca-transform DevStack plugin is tested only on Ubuntu 16.04 (Xenial).

A short cut to running monasca-transform in devstack is implemented with vagrant.

Variables

• DATABASE_PASSWORD(default: secretmysql) - password to upload monasca-transform schema
• MONASCA_TRANSFORM_DB_PASSWORD(default: password) - password for m-transform user

To run monasca-transform using the provided vagrant environment

Using any changes made locally to monasca-transform

cd tools/vagrant
vagrant up
vagrant ssh
cd devstack
./stack.sh

The devstack vagrant environment is set up to share the monasca-transform directory with the vm, copy it and commit any changes in the vm copy. This is because the devstack deploy process checks out the master branch to

/opt/stack

and deploys using that. Changes made by the user need to be committed in order to be used in the devstack instance. It is important therefore that changes should not be pushed from the vm as the unevaluated commit would be pushed.

N.B. If you are running with virtualbox you may find that the ./stack.sh fails with the filesystem becoming read only. There is a work around:

1. vagrant up --no-provision && vagrant halt
2. open virtualbox gui
3. open target vm settings and change storage controller from SCSI to SATA
4. vagrant up

Using the upstream committed state of monasca-transform

This should operate the same as for any other devstack plugin. However, to use the plugin from the upstream repo with the vagrant environment as described above it is sufficient to do:

cd tools/vagrant
vagrant up
vagrant ssh
cd devstack
vi local.conf

and change the line

enable_plugin monasca-transform /home/ubuntu/monasca-transform

to

enable_plugin monasca-transform https://git.openstack.org/openstack/monasca-transform

before running

./stack.sh

Connecting to devstack

The host key changes with each vagrant destroy/vagrant up cycle so it is necessary to manage host key verification for your workstation:

ssh-keygen -R 192.168.15.6

The devstack vm vagrant up process generates a private key which can be used for passwordless ssh to the host as follows:

cd tools/vagrant
ssh -i .vagrant/machines/default/virtualbox/private_key ubuntu@192.168.15.6

Running tox on devstack

Once the deploy is up use the following commands to set up tox.

sudo su monasca-transform
cd /opt/stack/monasca-transform
virtualenv .venv
. .venv/bin/activate
pip install tox
tox

Updating the code for dev

To regenerate the environment for development purposes a script is provided on the devstack instance at /opt/stack/monasca-transform/tools/vagrant/refresh_monasca_transform.sh To run the refresh_monasca_transform.sh script on devstack instance

cd /opt/stack/monasca-transform
tools/vagrant/refresh_monasca_transform.sh

(note: to use/run tox after running this script, the "Running tox on devstack" steps above have to be re-executed)

This mostly re-does the work of the devstack plugin, updating the code from the shared directory, regenerating the venv and the zip that is passed to spark during the spark-submit call. The configuration and the transform and pre transform specs in the database are updated with fresh copies, along with driver and service python code.

If refresh_monasca_transform.sh script completes successfully you should see a message like the following in the console.

    ***********************************************
    *                                             *
    * SUCCESS!! refresh monasca transform done.   *
    *                                             *
    ***********************************************

Development workflow

Here are the normal steps a developer can take to make any code changes. It is essential that the developer runs all tests in functional tests in a devstack environment before submitting any changes for review/merge.

Please follow steps mentioned in "To run monasca-transform using the provided vagrant environment" section above to create a devstack VM environment before following steps below:

1. Make code changes on the host machine (e.g. ~/monasca-transform)
2. vagrant ssh (to connect to the devstack VM)
3. cd /opt/stack/monasca-transform
4. tools/vagrant/refresh_monasca_transform.sh (See "Updating the code for dev"
                                               section above)
5. cd /opt/stack/monasca-transform (since monasca-transform folder
                                    gets recreated in Step 4. above)
6. tox -e pep8
7. tox -e py27
8. tox -e functional

Note: It is mandatory to run functional unit tests before submitting any changes for review/merge. These can be currently be run only in a devstack VM since tests need access to Apache Spark libraries. This is accomplished by setting SPARK_HOME environment variable which is being done in tox.ini.

export SPARK_HOME=/opt/spark/current

How to find and fix test failures ?

To find which tests failed after running functional tests (After you have run functional tests as per steps in Development workflow)

export OS_TEST_PATH=tests/functional
export SPARK_HOME=/opt/spark/current
source .tox/functional/bin/activate
testr run
testr failing (to get list of tests that failed)

You can add

import pdb
pdb.set_trace()

in test or in code where you want to start python debugger.

Run test using

python -m testtools.run <test>

For example:

python -m testtools.run  \
    tests.functional.usage.test_host_cpu_usage_component_second_agg.SparkTest

Reference: https://wiki.openstack.org/wiki/Testr

Access Spark Streaming and Spark Master/Worker User Interface

In a devstack environment ports on which Spark Streaming UI (4040), Spark Master(18080) and Spark Worker (18081) UI are available are forwarded to the host and are accessible from the host machine.

http://<host_machine_ip>:4040/ (Note: Spark Streaming UI,
                                is available only when
                                monasca-transform application
                                is running)
http://<host_machine_ip>:18080/ (Spark Master UI)
http://<host_machine_ip>:18081/ (Spark Worker UI)

To run monasca-transform using a different deployment technology

Monasca-transform requires supporting services, such as Kafka and Zookeeper, also are set up. So just adding "enable_plugin monasca-transform" to a default DevStack local.conf is not sufficient to configure a working DevStack deployment unless these services are also added.

Please reference the devstack/settings file for an example of a working list of plugins and services as used by the Vagrant deployment.

WIP

This is a work in progress. There are a number of improvements necessary to improve value as a development tool.

###TODO

1. Shorten initial deploy Currently the services deployed are the default set plus all of monasca. It's quite possible that not all of this is necessary to develop monasca-transform. So some services may be dropped in order to shorten the deploy.