From 2b8781379223c33d766912346209ad8f15bc5112 Mon Sep 17 00:00:00 2001
From: Ryan Beisner <ryan.beisner@canonical.com>
Date: Fri, 8 Jan 2016 21:45:38 +0000
Subject: [PATCH] Update tests/README
---
tests/README | 141 ++++++++++++++++++++++++++++++---------------------
1 file changed, 84 insertions(+), 57 deletions(-)
diff --git a/tests/README b/tests/README
index dbfc6a97..79c5b063 100644
--- a/tests/README
+++ b/tests/README
@@ -1,86 +1,113 @@
-This directory provides Amulet tests that focus on verification of
-neutron-api deployments.
+This directory provides Amulet tests to verify basic deployment functionality
+from the perspective of this charm, its requirements and its features, as
+exercised in a subset of the full OpenStack deployment test bundle topology.
-test_* methods are called in lexical sort order, although each individual test
-should be idempotent, and expected to pass regardless of run order.
+Reference: lp:openstack-charm-testing for full test bundles.
-Test name convention to ensure desired test order:
+A single topology and configuration is defined and deployed, once for each of
+the defined Ubuntu:OpenStack release combos. The ongoing goal is for this
+charm to always possess tests and combo definitions for all currently-supported
+release combinations of U:OS.
+
+test_* methods are called in lexical sort order, as with most runners. However,
+each individual test method should be idempotent and expected to pass regardless
+of run order or Ubuntu:OpenStack combo. When writing or modifying tests,
+ensure that every individual test is not dependent on another test_ method.
+
+Test naming convention, purely for code organization purposes:
1xx service and endpoint checks
2xx relation checks
3xx config checks
4xx functional checks
- 9xx restarts and other final checks
+ 9xx restarts, config changes, actions and other final checks
-Common relation definitions:
- - [ neutron-api, mysql ]
- - [ neutron-api, rabbitmq-server ]
- - [ neutron-api, nova-cloud-controller ]
- - [ neutron-api, neutron-openvswitch ]
- - [ neutron-api, keystone ]
- - [ neutron-api, neutron-gateway ]
-
-Resultant relations of neutron-api service:
- relations:
- amqp:
- - rabbitmq-server
- cluster:
- - neutron-api
- identity-service:
- - keystone
- neutron-api:
- - nova-cloud-controller
- neutron-plugin-api:
- - neutron-openvswitch
- shared-db:
- - mysql
-
-In order to run tests, you'll need charm-tools installed (in addition to
-juju, of course):
+In order to run tests, charm-tools and juju must be installed:
sudo add-apt-repository ppa:juju/stable
sudo apt-get update
- sudo apt-get install charm-tools
+ sudo apt-get install charm-tools juju juju-deployer amulet
-If you use a web proxy server to access the web, you'll need to set the
-AMULET_HTTP_PROXY environment variable to the http URL of the proxy server.
+Alternatively, tests may be exercised with proposed or development versions
+of juju and related tools:
+
+ # juju proposed version
+ sudo add-apt-repository ppa:juju/proposed
+ sudo apt-get update
+ sudo apt-get install charm-tools juju juju-deployer
+
+ # juju development version
+ sudo add-apt-repository ppa:juju/devel
+ sudo apt-get update
+ sudo apt-get install charm-tools juju juju-deployer
+
+Some tests may need to download files. If a web proxy server is required in
+the environment, the AMULET_HTTP_PROXY environment variable must be set and
+passed into the juju test command. This is unrelated to juju's http proxy
+settings or behavior.
The following examples demonstrate different ways that tests can be executed.
All examples are run from the charm's root directory.
- * To run all tests (starting with 00-setup):
+ * To run all +x tests in the tests directory:
- make test
+ bzr branch lp:charms/trusty/foo
+ cd foo
+ make functional_test
- * To run a specific test module (or modules):
+ * To run the tests against a specific release combo as defined in tests/:
- juju test -v -p AMULET_HTTP_PROXY 15-basic-trusty-icehouse
+ bzr branch lp:charms/trusty/foo
+ cd foo
+ juju test -v -p AMULET_HTTP_PROXY 015-basic-trusty-icehouse
- * To run a specific test module (or modules), and keep the environment
- deployed after a failure:
+ * To run tests and keep the juju environment deployed after a failure:
- juju test --set-e -v -p AMULET_HTTP_PROXY 15-basic-trusty-icehouse
+ bzr branch lp:charms/trusty/foo
+ cd foo
+ juju test --set-e -v -p AMULET_HTTP_PROXY 015-basic-trusty-icehouse
* To re-run a test module against an already deployed environment (one
that was deployed by a previous call to 'juju test --set-e'):
- ./tests/15-basic-trusty-icehouse
+ ./tests/015-basic-trusty-icehouse
-For debugging and test development purposes, all code should be idempotent.
-In other words, the code should have the ability to be re-run without changing
-the results beyond the initial run. This enables editing and re-running of a
-test module against an already deployed environment, as described above.
+ * Even with --set-e, `juju test` will tear down the deployment when all
+ tests pass. The following work flow may be more effective when
+ iterating on test writing.
-Manual debugging tips:
+ bzr branch lp:charms/trusty/foo
+ cd foo
+ ./tests/setup/00-setup
+ juju bootstrap
+ ./tests/015-basic-trusty-icehouse
+ # make some changes, run tests again
+ ./tests/015-basic-trusty-icehouse
+ # make some changes, run tests again
+ ./tests/015-basic-trusty-icehouse
- * Set the following env vars before using the OpenStack CLI as admin:
- export OS_AUTH_URL=http://`juju-deployer -f keystone 2>&1 | tail -n 1`:5000/v2.0
- export OS_TENANT_NAME=admin
+ * There may be test definitions in the tests/ dir which are not set +x
+ executable. This is generally true for deprecated releases, or for
+ upcoming releases which are not yet validated and enabled. To enable
+ and run these tests:
+ bzr branch lp:charms/trusty/foo
+ cd foo
+ ls tests
+ chmod +x tests/017-basic-trusty-kilo
+ ./tests/setup/00-setup
+ juju bootstrap
+ ./tests/017-basic-trusty-kilo
+
+
+Additional notes:
+
+ * Use DEBUG to turn on debug logging, use ERROR otherwise.
+ u = OpenStackAmuletUtils(ERROR)
+ u = OpenStackAmuletUtils(DEBUG)
+
+ * To interact with the deployed environment:
export OS_USERNAME=admin
export OS_PASSWORD=openstack
+ export OS_TENANT_NAME=admin
export OS_REGION_NAME=RegionOne
-
- * Set the following env vars before using the OpenStack CLI as demoUser:
- export OS_AUTH_URL=http://`juju-deployer -f keystone 2>&1 | tail -n 1`:5000/v2.0
- export OS_TENANT_NAME=demoTenant
- export OS_USERNAME=demoUser
- export OS_PASSWORD=password
- export OS_REGION_NAME=RegionOne
+ export OS_AUTH_URL=${OS_AUTH_PROTOCOL:-http}://`juju-deployer -e trusty -f keystone`:5000/v2.0
+ keystone user-list
+ glance image-list