Update unit test container setup and instructions

Update the TESTING.rst file to reflect that just installing and
starting zookeeperd is no longer sufficient now that we require TLS
and auth for the connection, as well as running database servers.
Suggest the container-based setup script instead. Also improve that
script to allow it to be invoked as a normal user, with root command
escalation tool choice (e.g. "sudo") supplied through a ROOTCMD
environment variable, so that things created inside the git worktree
like the CA don't end up root-owned.

Related, the tox-docker plugin previously suggested in the document
is no longer a viable option for the same reasons, so clean up the
plumbing for it in tox.ini as well.

Change-Id: Iac32799425a5bd4b1bdbf56f34a2310241ac4499

TESTING.rst

@@ -1,34 +1,30 @@
-===========================
-Testing Your OpenStack Code
-===========================
+============
+Testing Zuul
+============
 ------------
 A Quickstart
 ------------
 
-This is designed to be enough information for you to run your first tests.
-Detailed information on testing can be found here: https://wiki.openstack.org/wiki/Testing
+This is designed to be enough information for you to run your first tests on
+an Ubuntu 20.04 (or later) host.
 
 *Install pip*::
 
-  [apt-get | yum] install python-pip
+  sudo apt-get install python3-pip
 
 More information on pip here: http://www.pip-installer.org/en/latest/
 
-*Use pip to install tox and tox-docker*::
+*Use pip to install tox*::
 
-  pip install tox tox-docker
+  pip install tox
 
-As of zuul v3, a running zookeeper is required to execute tests. Using the
-``-docker`` suffixed commands will ensure this is started automatically by tox,
-but if you do not wish to use this:
+A running zookeeper is required to execute tests, but it also needs to be
+configured for TLS and a certificate authority set up to handle socket
+authentication. Because of these complexities, it's recommended to use a
+helper script to set up these dependencies, as well as a database servers::
 
-*Install zookeeper*::
-
-  [apt-get | yum] install zookeeperd
-
-*Start zookeeper*::
-
-  service zookeeper start
+  sudo apt-get install docker-compose  # or podman-compose if preferred
+  ROOTCMD=sudo tools/test-setup-docker.sh
 
 .. note:: Installing and bulding javascript is not required, but tests that
           depend on the javascript assets having been built will be skipped

tools/test-setup-docker.sh

@@ -1,6 +1,14 @@
 #!/bin/bash
 
-set -eu
+# This runs ZooKeeper and databases in docker containers, which are
+# required for tests.
+
+# This setup needs to be run as a user that can run docker or podman, or by
+# setting $ROOTCMD to a user substitution tool like "sudo" in the calling
+# environment.
+
+set -xeu
+ROOTCMD=${ROOTCMD:-}
 
 cd $(dirname $0)
 SCRIPT_DIR="$(pwd)"
@@ -29,9 +37,9 @@ fi
 MYSQL="${DOCKER} exec zuul-test-mysql mysql  -u root -pinsecure_worker"
 
 if [ "${COMPOSE}" == "docker-compose" ]; then
-  docker-compose rm -sf
+  ${ROOTCMD} docker-compose rm -sf
 else
-  podman-compose down
+  ${ROOTCMD} podman-compose down
 fi
 
 CA_DIR=$SCRIPT_DIR/ca
@@ -39,15 +47,14 @@ CA_DIR=$SCRIPT_DIR/ca
 mkdir -p $CA_DIR
 $SCRIPT_DIR/zk-ca.sh $CA_DIR zuul-test-zookeeper
 
-export USER_ID=$(id -u)
-${COMPOSE} up -d
+${ROOTCMD} USER_ID=$(id -u) ${COMPOSE} up -d
 
 echo "Waiting for mysql"
-timeout 30 bash -c "until ${MYSQL} -e 'show databases'; do sleep 0.5; done"
+timeout 30 bash -c "until ${ROOTCMD} ${MYSQL} -e 'show databases'; do sleep 0.5; done"
 echo
 
 echo "Setting up permissions for zuul tests"
-${MYSQL} -e "GRANT ALL PRIVILEGES ON *.* TO 'openstack_citest'@'%' identified by 'openstack_citest' WITH GRANT OPTION;"
-${MYSQL} -u openstack_citest -popenstack_citest -e "SET default_storage_engine=MYISAM; DROP DATABASE IF EXISTS openstack_citest; CREATE DATABASE openstack_citest CHARACTER SET utf8;"
+${ROOTCMD} ${MYSQL} -e "GRANT ALL PRIVILEGES ON *.* TO 'openstack_citest'@'%' identified by 'openstack_citest' WITH GRANT OPTION;"
+${ROOTCMD} ${MYSQL} -u openstack_citest -popenstack_citest -e "SET default_storage_engine=MYISAM; DROP DATABASE IF EXISTS openstack_citest; CREATE DATABASE openstack_citest CHARACTER SET utf8;"
 
 echo "Finished"

tox.ini

@@ -1,11 +1,8 @@
 [tox]
 minversion = 3.2
 skipsdist = True
-envlist = linters,py3{-docker}
+envlist = linters,py3
 ignore_basepython_conflict = True
-# TODO(stephenfin): It would be good to set 'requires = tox-docker', but doing
-# so borks the tools/pip.sh wrapper we're using here and probably isn't what
-# we want in the gate :(
 
 [testenv]
 basepython = python3
@@ -19,7 +16,6 @@ setenv =
   SQLALCHEMY_WARN_20={env:SQLALCHEMY_WARN_20:1}
   PYTHONWARNINGS=always::DeprecationWarning:zuul.driver.sql.sqlconnection,always::DeprecationWarning:tests.base,always::DeprecationWarning:tests.unit.test_database,always::DeprecationWarning:zuul.driver.sql.alembic.env,always::DeprecationWarning:zuul.driver.sql.alembic.script
 passenv =
-  DOCKER_*
   NODEPOOL_ZK_HOST
   OS_LOG_CAPTURE
   OS_LOG_DEFAULTS
@@ -40,8 +36,6 @@ whitelist_externals = bash
 deps =
   -r{toxinidir}/requirements.txt
   -r{toxinidir}/test-requirements.txt
-docker =
-  docker: zookeeper:3.4
 commands =
   bash -c 'stestr run --slowest --concurrency=`python -c "import multiprocessing; print(max(int(multiprocessing.cpu_count()-1),1))"` {posargs}'
 
@@ -96,7 +90,6 @@ deps =
 
 [testenv:remote]
 passenv =
-  DOCKER_*
   NODEPOOL_ZK_HOST
   OS_LOG_CAPTURE
   OS_LOG_DEFAULTS