[ci] Rework launching cli job

There are 2 critical blocking issues related to cli job:

 * bug related to db conflict between 2 tests (samples and lib check).
   We already spent ~ a month reserching the root of an issue and trying
   to fix it. Unfortunately, we did not succeded. Since we are planning
   to rework samples mechanism, the applied solution in this patch
   should not look like a dirty workaround - split launching main cli
   tests and samples check. To do this, we need to modify out pytest
   launcher a bit

 * Rally-CI has two cli jobs for mysql and postgres backends. While
   porting them to use `tox -ecli` command, passing the proper env
   variable was missed and sqlite is used now in both cases.
   Passing the right environment variable is not only required step,
   setting serial mode of pytest is important too, since we do not want
   to have conflicts in tests connecting to the one db.

Also, this patch removes unused option (--timeout) of pytest launcher.

Change-Id: I7fc109ee5785f426211fbff7eb9b4553a3671ba7

tests/ci/pytest_launcher.py

@@ -25,7 +25,7 @@ PYTEST_ARGUMENTS = ("py.test"  # base command
                     " --html=%(html_report)s"  # html report
                     " --self-contained-html"  # embedded css
                     " --durations=10"  # get a list of the slowest 10 tests
-                    " -n auto"  # launch tests in parallel
+                    " -n %(concurrency)s"  # launch tests in parallel
                     " %(path)s"
                     )
 
@@ -42,9 +42,8 @@ def main(args):
     parser.add_argument("--posargs", metavar="<str>", type=str, default="",
                         help="TOX posargs. Currently supported only string to "
                              "partial test or tests group to launch.")
-    parser.add_argument("--timeout", metavar="<seconds>", type=int, default=60,
+    parser.add_argument("--concurrency", metavar="<N>", type=int,
-                        help="Timeout for individual test execution. "
+                        help="Number of parallel processes.")
-                             "Defaults to 60")
     args = parser.parse_args(args[1:])
 
     # We allow only one parameter - path to partial test or tests group
@@ -99,7 +98,7 @@ def main(args):
 
     args = PYTEST_ARGUMENTS % {"html_report": pytest_report,
                                "path": path,
-                               "timeout": args.timeout}
+                               "concurrency": args.concurrency or "auto"}
     try:
         subprocess.check_call(args.split(" "),
                               stderr=subprocess.STDOUT)

tests/ci/rally_cli_job.sh

@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+
+LOCAL_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+DB_CONNECTION="$(rally-manage db show)"
+
+if [[ $DB_CONNECTION == sqlite* ]]; then
+    CONCURRENCY=0
+else
+    # in case of not sqlite db backends we cannot launch tests in parallel due
+    # to possible conflicts
+    CONCURRENCY=1
+    # currently, RCI_KEEP_DB variable is used to not create new databases per
+    # each test
+    export RCI_KEEP_DB=1
+fi
+
+
+python $LOCAL_DIR/pytest_launcher.py "tests/functional" --concurrency $CONCURRENCY
+
+python $LOCAL_DIR/pytest_launcher.py "tests/check_samples"

tox.ini

@@ -57,7 +57,7 @@ commands = oslo_debug_helper -t tests {posargs}
 sitepackages = True
 commands =
   find . -type f -name "*.pyc" -delete
-  python {toxinidir}/tests/ci/pytest_launcher.py "tests/functional" --timeout -1 --posargs={posargs}
+  {toxinidir}/tests/ci/rally_cli_job.sh
 
 [testenv:cover]
 commands = {toxinidir}/tests/ci/cover.sh {posargs}