openstack/rally-openstack
Code Issues Proposed changes
4f099aed34
rally-openstack/tests/ci
History
Andrey Kurilin 0913743b43 [ci] Start fixing CLI job 
Patch I7fc109ee5785f426211fbff7eb9b4553a3671ba7 splitted functional and
samples check into 2 separate modules. It was done due to some conflicts
while launching tests in parallel.
Despite the fact it was a good patch, it had a bug - missed `set -e`
called in the launch script. Which results in ignoring results of
launching functional tests for more then 2 months. Now we have 1/4 tests
fail.

This patch starts refactoring CLI job. First of all, we need to stop
ignoring the results of functional tests. The broken tests are marked as
skipped, so we can enable a part of tests right away.

Also, this patch moves test_certification_task.py to the proper dir with
samples check.

The new tox environment is introduced for launching only functional
tests(which should not depend on OpenStack): tox -efunctional

Change-Id: I222c13f724e8e70d10d58ca546094c076d73d737
2017-10-12 12:13:18 +03:00
..
hooks Rename certification/ to tasks/ 2017-09-19 18:01:25 -07:00
__init__.py Reorganize test module structure 2014-10-07 13:50:40 +00:00
cover.sh [ci]merge 3 patchs to unblock our ci 2017-06-30 12:38:07 -07:00
osresources.py Simplify deployment config format 2017-08-22 16:26:17 +03:00
pytest_launcher.py [ci] Rework launching cli job 2017-08-03 01:17:58 +03:00
rally_app.py [Test] Add test for simple app based on rally api 2017-01-19 13:23:01 +03:00
rally_functional_job.sh [ci] Start fixing CLI job 2017-10-12 12:13:18 +03:00
rally_gate_functions.sh Simplify deployment config format 2017-08-22 16:26:17 +03:00
rally_self_job.sh [CLI] Add support for emtpy deployment 2017-09-09 09:15:43 +00:00
rally_verify.py Fix auth issue in verification 2017-09-26 17:01:56 +03:00
rally-gate.sh Fix identity v3 detection 2017-05-02 15:19:54 +03:00
README.rst Adjusting introduction of gate 2017-03-22 01:01:20 +08:00
render.py [Reports] Get rid of Mako in reports, finally 2016-10-19 15:29:03 +03:00
sync_requirements.py Improve Rally Logging (part 2) 2017-10-05 22:57:39 +00:00
test_install.sh Make sure we don't have a stale package cache 2016-04-05 09:28:54 -04:00
wip-rally-gate.py [CLI] group commands in help message 2016-11-28 22:58:14 +03:00

README.rst

Rally Gate Jobs

For each patch submitted for review on Gerrit, there is a set of tests called gate jobs to be run against it. These tests check whether the Rally code works correctly after applying the patch and provide additional guarantees that it won't break the software when it gets merged. Rally gate jobs contain tests checking the codestyle (via pep8), unit tests suites, functional tests and a set of Rally benchmark tasks that are executed against a real devstack deployment.

rally-gate.sh

This script runs a set of real Rally benchmark tasks and fetches their results in textual / visualized form (available via a special html page by clicking the corresponding job title in Gerrit). It checks that scenarios don't fail while being executed against a devstack deployment and also tests SLA criteria to ensure that benchmark tasks have completed successfully. Jenkins uses this script by runnint the 'gate-rally-dsvm-rally' job, 'gate-rally-dsvm-manila-multibackend' job and 'gate-rally-dsvm-neutron-rally' job.

rally_verify.py

This script runs various "rally verify" commands. This set of commands allow us to perform Tempest tests of OpenStack live cloud and display verification results. The verification results obtained by running various "rally verify <cmd>" commands including "start", "show", "list" are compared using the "rally verify results" command, which are then saved in csv, html and json formats in the "rally-verify" directory. Jenkins uses this script by running the 'gate-rally-dsvm-verify' job.

test_install.sh

This script tests the correct working of the install_rally.sh, used for the installation of Rally. Jenkins tests this script by running it against Centos7 and Ubuntu 16.04 in the corresponding jobs 'gate-rally-install-centos-7' and 'gate-rally-install-ubuntu-xenial'.

Jenkins

Jenkins is a Continuous Integration system which works as the scheduler. It receives events related to proposed changes, triggers tests based on those events, and reports back. For each patch that is uploaded for review on Gerrit, Jenkins runs it against the various rally gate jobs listed below along with their functions and local equivalents:

  • gate-rally-pep8 : code style check

    (equal to tox -epep8)

  • gate-rally-docs : documention generation

    (equal to tox -edocs)

  • gate-rally-python27 : unit tests against python27

    (equal to tox -epy27)

  • gate-rally-python35 : unit tests against python35

    (equal to tox -epy35)

  • rally-coverage : generates unit test

    coverage (equal to tox -cover)

  • gate-rally-install-centos-7 : testing of test_install.sh

    (described above) against Centos7

  • gate-rally-install-ubuntu-xenial : testing of test_install.sh

    (described above) against Ubuntu 16.04

  • gate-rally-dsvm-rally : runs rally-gate.sh

    (described above) against OpenStack deployed by devstack with nova-network (It is standard dsvm job)

  • gate-rally-dsvm-manila-multibackend(non-voting) : runs rally-gate.sh against

    manila

  • gate-rally-dsvm-neutron-rally : runs rally-gate.sh against

    OpenStack deployed by devastack with neutron

  • gate-rally-dsvm-cli : runs tests/functional/*

    (equal to tox -ecli)

  • gate-rally-dsvm-verify(non-voting) : runs rally_verify.py and

    tests Rally and Tempest integration in all possible ways

  • gate-rally-tox-self(non-voting) : not yet used

and a success in these tests(except non-voting) would mean that the patch is approved by Jenkins.