openstack/oslotest
Code Issues Proposed changes

Remove cross-testing functionality

Browse Source 
Some of this has a dependency on pkg_resources, which is being removed
imminently. Nothing in OpenStack appears to be using any of this
functionality anymore - likely thanks to zuul - so we remove it outright
rather than rewriting it.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I7b2cada76f718e0c20c0906d1d4d4e4ce853e2f5
This commit is contained in:
Stephen Finucane
2025-10-31 17:12:21 +00:00
parent 9c1cb8e6df
commit b6f74b6754
6 changed files with 4 additions and 449 deletions
Download Patch File Download Diff File Expand all files Collapse all files

102
doc/source/user/cross-testing.rst
View File

@@ -1,102 +0,0 @@
==========================
Cross-project Unit Testing
==========================
Libraries in OpenStack have an unusual ability to introduce breaking
changes. All of the projects are run together from source in one form
or another during the integration tests, but they are not combined
from source when unit tests are run. The server applications do not
generally import code from other projects, so their unit tests are
isolated. The libraries, however, are fundamentally intended to be
used during unit tests as well as integration tests. Testing the full
cross-product of libraries and consuming projects would consume all
available test servers, and so we cannot run all of the tests on all
patches to the libraries. As an alternative, we have a few scripts in
``oslotest`` for running the unit tests in serial. The result takes
far too long (usually overnight) to run in the OpenStack
infrastructure. Instead, they are usually run by hand on a dedicated
system. A cloud VM works well for this purpose, especially considering
how much of it is now automated.
Check Out OpenStack Source
==========================
The first step for all of the cross-project unit tests tools is to
ensure that you have a full copy of the OpenStack source checked
out. You can do this yourself through gerrit's ssh API, or you can use
the ``clone_openstack.sh`` command in the tools directory of the
``openstack/oslo-incubator`` repository.
For example::
$ mkdir -p ~/repos/openstack
$ cd ~/repos/openstack
$ git clone https://opendev.org/openstack/oslo-incubator
$ cd ~/repos
$ ./openstack/oslo-incubator/tools/clone_openstack.sh
The first time the script runs it will take quite some time, since it
has to download the entire history of every OpenStack project.
Testing One Project
===================
``oslo_run_cross_tests`` runs one set of unit tests for one library
against all of the consuming projects. It should be run from the
directory with the library to be tested, and passed arguments telling
it about another project whose tests should be run.
For example, to run the ``py27`` test suite from nova using the
currently checked out sources for ``oslo.config``, run::
$ cd ~/repos/openstack/oslo.config
$ tox -e venv -- oslo_run_cross_tests ~/repos/openstack/nova py27
Testing All Consumers
=====================
``oslo_run_pre_release_tests`` builds on ``oslo_run_cross_tests`` to
find all of the consuming projects and run their tests
automatically. The pre-release script needs to know where the source
has been checked out, so the first step is to create its configuration
file.
Edit ``~/.oslo.conf`` to contain::
[DEFAULT]
repo_root = /path/to/repos
Replace ``/path/to/repos`` with the full, expanded, absolute path to
the location where the source code was checked out. For example, if
you followed the instructions above using ``clone_openstack.sh`` in
``~/repos`` and your user name is ``theuser`` the path would be
``/home/theuser/repos``.
Returning to the earlier example, to test ``oslo.config`` with all of
the projects that use it, go to the ``oslo.config`` source directory
and run ``oslo_run_pre_release_tests``.
::
$ cd ~/repos/openstack/oslo.config
$ tox -e venv -- oslo_run_pre_release_tests
The output for each test set is logged to a separate file in the
current directory, to make them easy to examine.
Use the ``--update`` or ``-u`` option to force a ``git pull`` for each
consuming projects before running its tests (useful for maintaining a
long-running system to host these tests).
Use the ``--verbose`` or ``-v`` option to report more verbosely about
what is happening, including the number of projects being tested.
Use ``--env`` or ``-e`` to add a tox environment to test. By default
the ``py27`` and ``pep8`` environments are used because those have
been shown to provide a good balance between finding problems and
running the tests quickly.
Use the ``--ref`` option to test a specific commit reference of the
library under test. The default is to leave the source directory
untouched, but if this option is specified ``git checkout`` is used to
force the source tree to the specified reference.

3
doc/source/user/index.rst
View File

@@ -9,6 +9,5 @@ Using oslotest
debugging
testing
mock-autospec
cross-testing
resources
history
history

27
doc/source/user/testing.rst
View File

@@ -2,29 +2,8 @@
Testing
=======
Cross-testing With Other Projects
=================================
oslotest can be tested via ``tox``, like any other OpenStack project:
The oslotest package can be cross-tested against its consuming
projects to ensure that no changes to the library break the tests in
those other projects.
In the Gate
-----------
Refer to the instructions in
https://wiki.openstack.org/wiki/Oslo/UsingALibrary for setting up
cross-test jobs in the gate.
Locally
-------
To run the cross-tests locally, invoke the script directly, passing
the path to the other source repository and the tox environment name
to use:
::
$ cd oslo.test
$ ./tools/oslo_run_cross_tests ~/repos/openstack/oslo.config py27
.. code-block:: shell
tox -e py3

2
pyproject.toml
View File

@@ -37,6 +37,4 @@ packages = [
]
script-files = [
"tools/oslo_debug_helper",
"tools/oslo_run_cross_tests",
"tools/oslo_run_pre_release_tests",
]

109
tools/oslo_run_cross_tests
View File

@@ -1,109 +0,0 @@
#!/bin/bash
#
# Run cross-project tests
#
# Usage:
#
# oslo_run_cross_tests project_dir venv
# Fail the build if any command fails
set -e
PYTHON=${PYTHON:-python3}
function usage {
cat - <<EOF
ERROR: Missing argument(s)
Usage:
oslo_run_cross_tests PROJECT_DIR VIRTUAL_ENV [POSARGS]
Example, run the python 2.7 tests for python-neutronclient:
oslo_run_cross_tests /opt/stack/python-neutronclient py27
oslo_run_cross_tests /opt/stack/nova py27 xenapi
EOF
exit $1
}
if [[ $# -lt 2 ]]; then
usage 1
fi
project_dir="$1"
shift
venv="$1"
shift
posargs="$*"
if [ -z "$project_dir" -o -z "$venv" ]
then
usage 2
fi
# Set up the virtualenv without running the tests
(cd $project_dir && tox --notest -r -e $venv)
tox_envbin=$project_dir/.tox/$venv/bin
our_name=$(${PYTHON} setup.py --name)
# Build the egg-info, including the source file list,
# so we install all of the files, even if the package
# list or name has changed.
rm -rf $(${PYTHON} setup.py --name).egg-info
${PYTHON} setup.py egg_info
# Replace the pip-installed package with the version in our source
# tree. Look to see if we are already installed before trying to
# uninstall ourselves, to avoid failures from packages that do not use us
# yet.
if $tox_envbin/pip freeze | grep -q $our_name
then
$tox_envbin/pip uninstall -y $our_name || echo "Ignoring error"
fi
$tox_envbin/pip install -U .
# Run the tests
(cd $project_dir && tox -e $venv -- $posargs)
result=$?
# The below checks are modified from
# openstack-infra/config/modules/jenkins/files/slave_scripts/run-unittests.sh.
# They expect to be run in the project being tested.
cd $project_dir
echo "Begin pip freeze output from test virtualenv:"
echo "======================================================================"
.tox/$venv/bin/pip freeze
echo "======================================================================"
# We only want to run the next check if the tool is installed, so look
# for it before continuing.
if [ -f /usr/local/jenkins/slave_scripts/subunit2html.py -a -d ".testrepository" ] ; then
if [ -f ".testrepository/0.2" ] ; then
cp .testrepository/0.2 ./subunit_log.txt
elif [ -f ".testrepository/0" ] ; then
.tox/$venv/bin/subunit-1to2 < .testrepository/0 > ./subunit_log.txt
fi
.tox/$venv/bin/python /usr/local/jenkins/slave_scripts/subunit2html.py ./subunit_log.txt testr_results.html
gzip -9 ./subunit_log.txt
gzip -9 ./testr_results.html
rancount=$(.tox/$venv/bin/testr last | sed -ne 's/Ran \([0-9]\+\).*tests in.*/\1/p')
if [ "$rancount" -eq "0" ] ; then
echo
echo "Zero tests were run. At least one test should have been run."
echo "Failing this test as a result"
echo
exit 1
fi
fi
# If we make it this far, report status based on the tests that were
# run.
exit $result

210
tools/oslo_run_pre_release_tests
View File

@@ -1,210 +0,0 @@
#!/usr/bin/env python
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""Run unit tests for projects that use a library.
"""
import glob
import os
import subprocess
import sys
from oslo_config import cfg
from oslotest.tools import config as tconfig
from pbr import packaging
import pkg_resources
def find_all_projects(repo_root):
"""Scan the checked out repositories for all available projects."""
pattern = os.path.join(repo_root, 'openstack/*')
candidates = glob.glob(pattern)
prefix_len = len(repo_root)
return [
c[prefix_len:].lstrip('/')
for c in candidates
if os.path.isdir(c)
]
def find_consuming_projects(lib_name, repo_root, projects):
"""Filter the list of projects
Filter the list of projects to only include entries that use the library.
"""
for p in projects:
consumer = False
for base in packaging.get_requirements_files():
req_file = os.path.join(repo_root, p, base)
for req in packaging.parse_requirements([req_file]):
try:
parsed_req = pkg_resources.Requirement.parse(req)
req_name = parsed_req.project_name
except ValueError:
continue
if req_name == lib_name:
consumer = True
yield p
break
if consumer:
break
def main():
conf = tconfig.get_config_parser()
conf.register_cli_opt(
cfg.StrOpt(
'library-under-test',
short='l',
default='',
help=('the name of the library being tested; '
'defaults to current dir'),
)
)
conf.register_cli_opt(
cfg.BoolOpt(
'update',
short='u',
default=False,
help='update consumers before running tests',
)
)
conf.register_cli_opt(
cfg.BoolOpt(
'verbose',
short='v',
default=False,
help='print verbose output',
)
)
conf.register_cli_opt(
cfg.StrOpt(
'ref',
short='r',
default='HEAD',
help='the commit reference to test; defaults to HEAD',
)
)
conf.register_cli_opt(
cfg.MultiStrOpt(
'env',
short='e',
default=['py27', 'pep8'],
help=('the name of the tox environment to test; '
'defaults to py27 and pep8'),
)
)
conf.register_cli_opt(
cfg.MultiStrOpt(
'consumer',
positional=True,
default=[],
help='the name of a project to test with; may be repeated',
)
)
tconfig.parse_arguments(conf)
repo_root = os.path.expanduser(conf.repo_root)
# Figure out which library is being tested
lib_name = conf.library_under_test
if not lib_name:
if conf.verbose:
print('finding library name')
lib_name = subprocess.check_output(
['python', 'setup.py', '--name']
).strip()
lib_dir = os.getcwd()
else:
lib_dir = os.path.join(repo_root, 'openstack', lib_name)
print(f'testing {lib_name} in {lib_dir}')
projects = set(conf.consumer)
if not projects:
# TODO(dhellmann): Need to update this to look at gerrit, so
# we can check out the projects we want to test with.
if conf.verbose:
print('defaulting to all projects under %s/openstack' % repo_root)
projects = find_all_projects(repo_root)
# Filter out projects that do not require the library under test
before = len(projects)
projects = list(find_consuming_projects(lib_name, repo_root, projects))
after = len(projects)
if (after < before) and conf.verbose:
print('ignoring %s projects that do not use %s'
% (before - after, lib_name))
projects = list(sorted(projects))
if not projects:
print('ERROR: found no projects using %s' % lib_name)
return 1
if conf.verbose:
print('preparing to test %s projects' % after)
# Make sure the lib being tested is set to the reference intended.
if conf.ref != 'HEAD':
if conf.verbose:
print(f'ensuring {lib_name} is updated to {conf.ref}')
subprocess.check_call(
['git', 'checkout', conf.ref],
cwd=lib_dir,
)
git_quiet = ['-q'] if not conf.verbose else []
failures = []
for p in projects:
if conf.verbose:
print()
proj_dir = os.path.join(repo_root, p)
if conf.update:
if conf.verbose:
print('updating %s with "git pull"' % p)
subprocess.Popen(
['git', 'pull'] + git_quiet,
cwd=proj_dir,
).communicate()
p_log_name = p.split('/')[-1].replace('.', '-')
for e in conf.env:
log_name = f'cross-test-{p_log_name}-{e}.log'
with open(log_name, 'w') as log_file:
print(f'testing {e} in {p}, logging to {log_name}',
end=' ')
sys.stdout.flush()
command = ['oslo_run_cross_tests', proj_dir, e]
log_file.write('running: %s\n' % ' '.join(command))
log_file.flush() # since Popen is going to use the fd directly
cmd = subprocess.Popen(
command,
cwd=lib_dir,
stdout=log_file,
stderr=log_file
)
cmd.communicate()
log_file.write('\nexit code: %s\n' % cmd.returncode)
if cmd.returncode:
print('FAIL')
failures.append((p, e, cmd.returncode))
else:
print('PASS')
if failures:
print('\nFAILED %d jobs' % len(failures))
return 1
print('\nPASSED all jobs')
return 0
if __name__ == '__main__':
sys.exit(main())