db: Increase timeout for migration tests

We're seeing quite a few timeout failures on the following tests in
'nova.tests.unit.db.main.test_migrations':

- TestModelsLegacySyncMySQL.test_models_sync
- TestMigrationsWalkMySQL.test_walk_versions
- TestModelsSyncMySQL.test_innodb_tables
- TestModelsSyncMySQL.test_models_sync

Evidently MySQL is particularly affected here. Test run times are slow
even on a relatively powered machine like my localhost (Lenovo T460s w/
Intel Core i7-6600U CPU + 20G RAM) and the CI machines are only making
matters worse. Local experiments with alternative MySQL libraries, such
'mysqlclient', did not yield any improvements in performance so we must
simply live with this for now. Do so by setting 'TIMEOUT_SCALING_FACTOR'
for these tests to 4, meaning these tests will now get a whopping 640
seconds (or over 10 minutes) to execute (we set OS_TEST_TIMEOUT to 160
in 'tox.ini'). We set this for both main and API DB migrations, even
though only the former is currently exhibiting issues, to head off
future problems. An alternative to this would be to override the timeout
on a test-by-test basis, as Cinder has done [1], but that seems more
complicated for no good reason. Yet another alternative would be to
reintroduce the serialization of these tests first introduced in change
I6ce930fa86c82da1008089791942b1fff7d04c18, but that is left until later
in the hopes that simply increasing the timeout will resolve the issues.

[1] https://github.com/openstack/cinder/blob/19.0.0/cinder/tests/unit/db/test_migrations.py

Change-Id: I82b9a064d77251945ff1ae99d7049f367ddde92e
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This commit is contained in:
Stephen Finucane 2021-10-19 10:19:07 +01:00
parent e14eef0719
commit fb083138eb
2 changed files with 16 additions and 0 deletions

View File

@ -41,6 +41,10 @@ LOG = logging.getLogger(__name__)
class NovaModelsMigrationsSync(test_migrations.ModelsMigrationsSync):
"""Test that the models match the database after migrations are run."""
# Migrations can take a long time, particularly on underpowered CI nodes.
# Give them some breathing room.
TIMEOUT_SCALING_FACTOR = 4
def setUp(self):
super().setUp()
self.engine = enginefacade.writer.get_engine()
@ -185,6 +189,10 @@ class NovaMigrationsWalk(
test_fixtures.OpportunisticDBTestMixin, test.NoDBTestCase,
):
# Migrations can take a long time, particularly on underpowered CI nodes.
# Give them some breathing room.
TIMEOUT_SCALING_FACTOR = 4
def setUp(self):
super().setUp()
self.engine = enginefacade.writer.get_engine()

View File

@ -49,6 +49,10 @@ LOG = logging.getLogger(__name__)
class NovaModelsMigrationsSync(test_migrations.ModelsMigrationsSync):
"""Test sqlalchemy-migrate migrations."""
# Migrations can take a long time, particularly on underpowered CI nodes.
# Give them some breathing room.
TIMEOUT_SCALING_FACTOR = 4
def setUp(self):
# Ensure BaseTestCase's ConfigureLogging fixture is disabled since
# we're using our own (StandardLogging).
@ -197,6 +201,10 @@ class NovaMigrationsWalk(
test_fixtures.OpportunisticDBTestMixin, test.NoDBTestCase,
):
# Migrations can take a long time, particularly on underpowered CI nodes.
# Give them some breathing room.
TIMEOUT_SCALING_FACTOR = 4
def setUp(self):
super().setUp()
self.engine = enginefacade.writer.get_engine()