Fix PEP8 tests for Python 3
* relace long with int * replace execfile with exec Change-Id: If98949fa5f49091fbf11c95a65302a4f844538c9 Story: 2003240 Task: 26827
|6 months ago|
|files||6 months ago|
|README.md||2 years ago|
|plugin.sh||1 year ago|
|post_test_hook.sh||1 year ago|
|pre_test_hook.sh||2 years ago|
|settings||1 year ago|
|test.sh||2 years ago|
The Monasca-transform DevStack plugin is tested only on Ubuntu 16.04 (Xenial).
A short cut to running monasca-transform in devstack is implemented with vagrant.
cd tools/vagrant vagrant up vagrant ssh cd devstack ./stack.sh
The devstack vagrant environment is set up to share the monasca-transform directory with the vm, copy it and commit any changes in the vm copy. This is because the devstack deploy process checks out the master branch to
and deploys using that. Changes made by the user need to be committed in order to be used in the devstack instance. It is important therefore that changes should not be pushed from the vm as the unevaluated commit would be pushed.
N.B. If you are running with virtualbox you may find that the
./stack.sh fails with the filesystem becoming read only. There is a work around:
This should operate the same as for any other devstack plugin. However, to use the plugin from the upstream repo with the vagrant environment as described above it is sufficient to do:
cd tools/vagrant vagrant up vagrant ssh cd devstack vi local.conf
and change the line
enable_plugin monasca-transform /home/ubuntu/monasca-transform
enable_plugin monasca-transform https://git.openstack.org/openstack/monasca-transform
The host key changes with each
vagrant up cycle so
it is necessary to manage host key verification for your workstation:
ssh-keygen -R 192.168.15.6
The devstack vm vagrant up process generates a private key which can be used for passwordless ssh to the host as follows:
cd tools/vagrant ssh -i .vagrant/machines/default/virtualbox/private_key firstname.lastname@example.org
Once the deploy is up use the following commands to set up tox.
sudo su monasca-transform cd /opt/stack/monasca-transform virtualenv .venv . .venv/bin/activate pip install tox tox
To regenerate the environment for development purposes a script is provided on the devstack instance at /opt/stack/monasca-transform/tools/vagrant/refresh_monasca_transform.sh To run the refresh_monasca_transform.sh script on devstack instance
cd /opt/stack/monasca-transform tools/vagrant/refresh_monasca_transform.sh
(note: to use/run tox after running this script, the “Running tox on devstack” steps above have to be re-executed)
This mostly re-does the work of the devstack plugin, updating the code from the shared directory, regenerating the venv and the zip that is passed to spark during the spark-submit call. The configuration and the transform and pre transform specs in the database are updated with fresh copies, along with driver and service python code.
If refresh_monasca_transform.sh script completes successfully you should see a message like the following in the console.
*********************************************** * * * SUCCESS!! refresh monasca transform done. * * * ***********************************************
Here are the normal steps a developer can take to make any code changes. It is essential that the developer runs all tests in functional tests in a devstack environment before submitting any changes for review/merge.
Please follow steps mentioned in “To run monasca-transform using the provided vagrant environment” section above to create a devstack VM environment before following steps below:
1. Make code changes on the host machine (e.g. ~/monasca-transform) 2. vagrant ssh (to connect to the devstack VM) 3. cd /opt/stack/monasca-transform 4. tools/vagrant/refresh_monasca_transform.sh (See "Updating the code for dev" section above) 5. cd /opt/stack/monasca-transform (since monasca-transform folder gets recreated in Step 4. above) 6. tox -e pep8 7. tox -e py27 8. tox -e functional
Note: It is mandatory to run functional unit tests before submitting any changes for review/merge. These can be currently be run only in a devstack VM since tests need access to Apache Spark libraries. This is accomplished by setting SPARK_HOME environment variable which is being done in tox.ini.
To find which tests failed after running functional tests (After you have run functional tests as per steps in Development workflow)
export OS_TEST_PATH=tests/functional export SPARK_HOME=/opt/spark/current source .tox/functional/bin/activate testr run testr failing (to get list of tests that failed)
You can add
import pdb pdb.set_trace()
in test or in code where you want to start python debugger.
Run test using
python -m testtools.run <test>
python -m testtools.run \ tests.functional.usage.test_host_cpu_usage_component_second_agg.SparkTest
In a devstack environment ports on which Spark Streaming UI (4040), Spark Master(18080) and Spark Worker (18081) UI are available are forwarded to the host and are accessible from the host machine.
http://<host_machine_ip>:4040/ (Note: Spark Streaming UI, is available only when monasca-transform application is running) http://<host_machine_ip>:18080/ (Spark Master UI) http://<host_machine_ip>:18081/ (Spark Worker UI)
Monasca-transform requires supporting services, such as Kafka and Zookeeper, also are set up. So just adding “enable_plugin monasca-transform” to a default DevStack local.conf is not sufficient to configure a working DevStack deployment unless these services are also added.
Please reference the devstack/settings file for an example of a working list of plugins and services as used by the Vagrant deployment.
This is a work in progress. There are a number of improvements necessary to improve value as a development tool.