7.8 KiB

Raw Blame History

Functional Testing Environments

The new direction for functional testing is to move it out of tempest and into the projects. This means that we need to provide direction and some sort of framework for how this is to be done in Keystone.

This information was originally captured in an etherpad while it was being discussed.

Problem Description

In another spec <http://specs.openstack.org/openstack/keystone-specs/specs/ kilo/functional-testing.html> we defined two types of functional tests. Shared tests that can be run against any Keystone instance to verify its behavior. Configuration specific tests that should only be run against a Keystone instance that has been configured a certain way, e.g. federation.

These tests can be run against any Keystone instance by specifying the URL. This means that a provider that has implemented federation with some domain specific IdP can run the federation tests to see if there are any major problems.

We also need a standard way to define expected configurations so that it is easy for developers to standup test environments and easy for CI.

Not all created configurations will be in CI. We will have to pick based on risk of bugs, popularity and CI resources. This spec doesn't talk too much more about that.

Proposed Change

Constraints

stick to the methods/mechanisms used for adding functional tests in other OpenStack projects; this may be slightly different so that we can experiment with better solutions, but I don't want to be too far outside of the box
it should be very easy to add new configurations that need to be tested
the tests should be runnable by the gate for enforcement
the tests should be runnable by developers using devstack
the tests, when possible, should be runnable with a builtin server so no devstack is required (likely only the shared tests)

Concepts

devstack vm (dsvm) configurations: the scripts and configuration necessary to setup Keystone for testing a type of deployment (eventlet, Apache, federation, etc)
scenario tests: the actual tests themselves; I am picturing this just using our current framework and tools used in unit testing (maybe with more focus on using the KeystoneClient API
shared tests: the scenario tests that show the base behavior of the system; these will run under every configuration

Basic idea

Configurations are actually implemented as devstack hooks. This allows them to be easily used for gate testing. Each configuration will provide a script that will restack a devstack instance. This restacking just automates the calls to the gate hooks. This will allow developers to switch between configurations as their testing needs change.

The tests themselves will be run as tox targets.

Possible configurations

These are things that we potentially want to gate against. There are many possible configurations to test against. Below are just a few. Generally speaking every run_tests.sh will run the standard tests as well as any tests specific to that configuration.

eventlet - deploy on eventlet
Apache - deploy behind mod_wsgi
federation - deploy on Apache using mod_shib

Directory structures

dsvm
└── {configuration name}
    ├── devstack
    │   ├── extras.d
    │   │   └── *.sh
    │   ├── files
    │   │   └── {...}
    │   ├── lib
    │   │   └── {...}
    │   └── local.conf
    ├── stack.sh
    ├── unstack.sh
    └── run_tests.sh

configuration name: This is the directory that will hold all of the files necessary for a configuraion.
extras.d: A directory of shell scripts that implement devstack plugins. This is not Keystone specific, but rather devstack specific. These shell scripts are really just calling to functions defined in 'lib' shell scripts instead of implementing significant logic.
files: A directory containing files used by devstack. This is there things like configuration templates would go.
lib: A directory containing supporting shell scripts that defined the logic used by plugin scripts.
local.conf: This is the devstack configuration. If defines the plugins necessary for testing the configuration.
stack.sh: Script used to initialize a devstack with a specific configuration. This will move any existing configuration (local.conf) out of the way before installing the one bundled with this dsvm configuration. We'll then call devstack's stack.sh.
unstack.sh: Removes any dsvm configuration specifics configs and moves back anything that was moved by stack.sh. We'll then call devstack's unstack.sh.
run_tests.sh: Script to call stack.sh, run the functional tests and call unstack.sh for a dsvm configuration.

Alternatives

I have not really investigated alternatives. This proposal represents what I learned from other projects and the changes I think are necessary to satisfy the constraints.