openinfra/refstack-client
Code Issues Proposed changes
Testing and result upload client for refstack
85 Commits 1 Branch 1 Tag 2.9 MiB
Python 91.7%
Shell 7.9%
Dockerfile 0.4%
1b7f413b60
Go to file
Dolph Mathews 1b7f413b60 Pass tempest return code back to caller 
Use case: I'm running refstack-client in a tox job, which always appears
to succeed, even when the test fails.

Currently, unless it throws a backtrace on it's own, the refstack-client
script always returns successfully, even when the tempest job it's
running fails.

This change simply passes non-zero return codes back through as the
refstack-client's return code.

Only the test() method was affected here, because it's the only
externally callable function that uses subprocess. Other functions can
now return non-zero return codes as well.

Change-Id: I0ce786babcbee47da1c038a3478bcd4de0dff16e
2016-02-04 15:22:28 +00:00
doc Initial documentation dump 2014-09-11 18:17:20 -04:00
refstack_client Pass tempest return code back to caller 2016-02-04 15:22:28 +00:00
.gitignore Added the actual post request to upload 2015-01-12 22:53:23 -08:00
.gitreview Update .gitreview for project rename 2015-09-12 00:08:35 +00:00
.testr.conf Post test results signed with RSA key 2015-04-13 22:01:03 +03:00
LICENSE First Commit 2014-07-10 11:48:59 -07:00
README.rst Add support for subunit file uploading 2016-01-15 11:35:09 -08:00
refstack-client Pass tempest return code back to caller 2016-02-04 15:22:28 +00:00
requirements.txt Remove RefStack dependency on Keystone client. 2016-01-23 14:43:01 +08:00
setup_env Cleanup comments and error message in setup_env 2016-02-02 14:54:16 +00:00
setup.cfg Rename refstack-client to refstack_client 2014-09-23 15:17:17 +03:00
setup.py First Commit 2014-07-10 11:48:59 -07:00
tempest-additional-requirements.txt Another way to use additional tests for tempest 2015-12-14 17:10:55 +03:00
test-requirements.txt Updating tests results format according to spec 2015-01-15 19:51:10 +02:00
tox.ini Deprecated tox -downloadcache option removed 2015-12-11 21:27:49 +01:00

README.rst

refstack-client

refstack-client is a command line utility that allows you to execute Tempest test runs based on configurations you specify. When finished running Tempest it can send the passed test data to a RefStack API server.

Environment setup

We've created an "easy button" for Ubuntu, Centos, RHEL and openSuSe.

  1. Make sure you have git installed

  2. Get the refstack client: git clone https://github.com/openstack/refstack-client

  3. Go into the refstack-client directory: cd refstack-client

  4. Run the "easy button" setup: ./setup_env

    Options:

    a. -c option allows to specify SHA of commit or branch in Tempest repository which will be installed.

    b. -t option allows to specify tag in Tempest repository which will be installed. For example: execute ./setup_env -t tags/3 to install Tempest tag-3.

    c. By default, Tempest will be installed from commit 551e1a9701e7e2b3edf6d49a2eaa62b7ab2435ad (11 September 2015).

Usage

  1. Prepare a tempest configuration file that is customized to your cloud environment.

  2. Go into the refstack-client directory.

    cd ~/refstack-client

  3. Source to use the correct Python environment.

    source .venv/bin/activate

  4. Validate your setup by running a short test.

    ./refstack-client test -c <Path of the tempest configuration file to use> -v -- tempest.api.identity.admin.v2.test_roles

    or

    ./refstack-client test -c <Path of the tempest configuration file to use> -v -- tempest.api.identity.v2.test_token

  5. Run tests.

    To run the entire API test set:

    ./refstack-client test -c <Path of the tempest configuration file to use> -v

    To run only those tests specified in a DefCore defined test file:

    ./refstack-client test -c <Path of the tempest configuration file to use> -v --test-list <Path or URL of test list>

    For example:

    ./refstack-client test -c ~/tempest.conf -v --test-list https://raw.githubusercontent.com/openstack/defcore/master/2015.05/2015.05.required.txt

    This will run only the test cases listed in 2015.05.required.txt.

    Note:

    1. Adding the -v option will show the Tempest test result output.
    2. Adding the --upload option will have your test results be uploaded to the default RefStack API server or the server specified by --url.
    3. Adding the --test-list option will allow you to specify the file path or URL of a test list text file. This test list should contain specific test cases that should be tested. Tests lists passed in using this argument will be normalized with the current Tempest evironment to eliminate any attribute mismatches.
    4. Adding the --url option will allow you to change where test results should be uploaded.
    5. Adding the -r option with a string will prefix the JSON result file with the given string (e.g. '-r my-test' will yield a result file like 'my-test-0.json').
    6. Adding -- enables you to pass arbitary arguments to the Tempest runner. After the first --, all other subsequent arguments will be passed to the Tempest runner as is. This is mainly used for quick verification of the target test cases. (e.g. -- tempest.api.identity.v2.test_token)

    Use ./refstack-client test --help for the full list of arguments.

  6. Upload your results.

    If you previously ran a test with refstack-client without the --upload option, you can upload your results to a RefStack API server by using the following command:

    ./refstack-client upload <Path of results file>

    The results file is a JSON file generated by refstack-client when a test has completed. This is saved in .tempest/.testrepository. When you use the upload command, you can also override the RefStack API server uploaded to with the --url option.

    Alternatively, you can use the 'upload-subunit' command to upload results using an existing subunit file. This requires that you pass in the Keystone endpoint URL for the cloud that was tested to generate the subunit data:

    ./refstack-client upload-subunit --keystone-endpoint http://some.url:5000/v3 <Path of subunit file>

    Note:

    1. Adding -i <path-to-private-key> option will upload test results with a digital signature. For signing, refstack-client uses private RSA keys. The OpenSSH format of RSA keys is supported, so you can just use your SSH key '~/.ssh/id-rsa' or generate a new one with ssh-keygen -b 4096. For now, signed test results can be considered private.

  7. List uploaded test set.

    You can list previously uploaded data from a RefStack API server by using the following command:

    ./refstack-client list --url <URL of the RefStack API server>

Tempest Hacking

By default, refstack-client installs Tempest into the .tempest directory. If you're interested in working with Tempest directly for debugging or configuration, you can activate a working Tempest environment by switching to that directory and using the installed dependencies.

  1. cd .tempest
  2. run tempest with ./run_tempest.sh -V or source ./.venv/bin/activate and run tests manually with testr.

This will make the entire Tempest environment available for you to run, including the run_tempest script and testr.