openstack/tempest
Code Issues Proposed changes
180fed1afa
tempest/stress
History
David Kranz 180fed1afa Fix and simplify reboot stress test. Add new env stuff. 
Added stress environment var for max_instances.
Added tempest var for log_level of rest_client.

Change-Id: Ia92536a547cdab4d2496bd9ba9067be3595b79cd

xxxxx

Change-Id: I8c2f499a41f74c2fe6fb08c80ab6fc31f6b93426
2012-04-02 09:43:39 -04:00
..
tests Fix and simplify reboot stress test. Add new env stuff. 2012-04-02 09:43:39 -04:00
tools Fix and simplify reboot stress test. Add new env stuff. 2012-04-02 09:43:39 -04:00
utils Generalize configuration for controller access. 2012-03-21 12:50:21 -04:00
__init__.py Initial checkin of Stress Test for nova. 2012-03-12 16:20:08 -04:00
basher.py Initial checkin of Stress Test for nova. 2012-03-12 16:20:08 -04:00
config.py Fix and simplify reboot stress test. Add new env stuff. 2012-04-02 09:43:39 -04:00
driver.py Fix and simplify reboot stress test. Add new env stuff. 2012-04-02 09:43:39 -04:00
pending_action.py Initial checkin of Stress Test for nova. 2012-03-12 16:20:08 -04:00
README.rst Fix and simplify reboot stress test. Add new env stuff. 2012-04-02 09:43:39 -04:00
state.py Initial checkin of Stress Test for nova. 2012-03-12 16:20:08 -04:00
test_case.py Initial checkin of Stress Test for nova. 2012-03-12 16:20:08 -04:00
test_server_actions.py Fix and simplify reboot stress test. Add new env stuff. 2012-04-02 09:43:39 -04:00
test_servers.py Fix and simplify reboot stress test. Add new env stuff. 2012-04-02 09:43:39 -04:00

README.rst

Quanta Research Cambridge OpenStack Stress Test System

Nova is a distributed, asynchronous system that is prone to race condition bugs. These bugs will not be easily found during functional testing but will be encountered by users in large deployments in a way that is hard to debug. The stress test tries to cause these bugs to happen in a more controlled environment.

The basic idea of the test is that there are a number of actions, roughly corresponding to the Compute API, that are fired pseudo-randomly at a nova cluster as fast as possible. These actions consist of what to do, how to verify success, and a state filter to make sure that the operation makes sense. For example, if the action is to reboot a server and none are active, nothing should be done. A test case is a set of actions to be performed and the probability that each action should be selected. There are also parameters controlling rate of fire and stuff like that.

This test framework is designed to stress test a Nova cluster. Hence, you must have a working Nova cluster.

Environment

This particular framework assumes your working Nova cluster understands Nova API 2.0. The stress tests can read the logs from the cluster. To enable this you have to provide the hostname to call 'nova-manage' and the private key and user name for ssh to the cluster in the [stress] section of tempest.conf. You also need to provide the value of --logdir in nova.conf:

host_private_key_path=<path to private ssh key> host_admin_user=<name of user for ssh command> nova_logdir=<value of --logdir in nova.conf> controller=<hostname for calling nova-manage> max_instances=<limit on instances that will be created>

The stress test needs the top-level tempest directory to be on PYTHONPATH if you are not using nosetests to run.

For real stress, you need to remove "ratelimit" from the pipeline in api-paste.ini.

Running the sample test

To test your installation, do the following (from the tempest directory):

PYTHONPATH=. python stress/tests/user_script_sample.py

This sample test tries to create a few VMs and kill a few VMs.

Additional Tools

Sometimes the tests don't finish, or there are failures. In these cases, you may want to clean out the nova cluster. We have provided some scripts to do this in the tools subdirectory. To use these tools, you will need to install python-novaclient. You can then use the following script to destroy any keypairs, floating ips, and servers:

stress/tools/nova_destroy_all.py