At the moment migration script does misbehave and will fail on attempt
to migrate database from old SQLAlchemy Migrate to Alembic due to
misconfigured alembic_table.
Right now alembic_version table is being used by taskflow. With that it
was existing before 2024.1, where Masakari has adopted alembic as well.
As a result, masakari-manage db sync returns early from
_migrate_legacy_database as it treats taskflow alembic DB as it's own.
And when it tries to apply migrations - it fails as table already exist.
masakari-manage db version also shows taskflow version instead of
masakari version which is misleading.
Passing correct version_table to MigrationContext fixes the issue.
Change-Id: I67e04640ec76cf000aadfc5834d89da7a772a8d6
concurrency.TpoolDbapiWrapper has been deprecated and removed with [1]
and is expected to be replaced with oslo_db.api.DBAPI
[1] https://review.opendev.org/c/openstack/oslo.db/+/908730
Change-Id: Idb74cd99f2779cee237d6cbc1f3c392647bb0573
Add file to the reno documentation build to show release notes for
stable/2024.1.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2024.1.
Sem-Ver: feature
Change-Id: I729ddc3c1f672c2c9840dce2982792154fd8f1ae
Under Python 3.12 this results in a syntax error rather than
M329 being detected on line 8 of the test code.
Change-Id: Ia1bb4dfb43b00115cde1158681d34d6ad7c7d744
As per the current release tested runtime, we test
till python 3.11 so updating the same in python
classifier in setup.cfg
Change-Id: I4bb6f61e833e1b32252db7dcf6773139746e5335
Resolve the following RemovedIn20Warning warning:
The current statement is being autocommitted using implicit
autocommit, which will be removed in SQLAlchemy 2.0. Use the .begin()
method of Engine or Connection in order to use an explicit transaction
for DML and DDL statements.
This is the beefiest one due to our extensive reliance on auto-commit.
Change-Id: Iebf9d022c312b8f5457ff34eb497cdb851aa4901
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Resolve the following RemovedIn20Warning warning:
The legacy calling style of select() is deprecated and will be removed
in SQLAlchemy 2.0. Please use the new calling style described at
select().
Change-Id: I00206489e97468d98a3a516c7c081c70348e3998
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Resolve the following RemovedIn20Warning warning:
Using strings to indicate column or relationship paths in loader
options is deprecated and will be removed in SQLAlchemy 2.0. Please
use the class-bound attribute directly.
Change-Id: I1b71331796c2730713797043489b6c11710b9851
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Resolve the following RemovedIn20Warning warning:
Passing a string to Connection.execute() is deprecated and will be
removed in version 2.0. Use the text() construct, or the
Connection.exec_driver_sql() method to invoke a driver-level SQL
string.
We do the latter.
Change-Id: I0ee518022772beeb0298e12b5b36279fd384fb4a
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
These are highlighted by the new WarningsFixture.
Change-Id: If9a27784e4f4fde61dff73c1f6fde99324e1aa07
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
These are no longer used. Curiously, sqlalchemy-migrate is not in the
dependencies and was instead being pulled in as a transitive dependency
of oslo.db. This will therefore disappear when oslo.db is bumped to 13.x
or later in upper-constraints, since that version removes
sqlalchemy-migrate support.
Change-Id: I4ea8ae36c9f8d3e5e5a9ac9548c5fe8f975cd666
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This is easier than Cinder, Nova etc. but more difficult than Heat.
Masakari hasn't had many database migrations in recent cycles but it did
have on in Antelope. This means we need to handle people doing
skip-level upgrades and validate which version of the legacy migrations
they are currently on. We support users coming from both Zed and
Antelope and anyone else will need to go through an older version of
Masakari to update their database migrations first. Other than this
difference, the logic is pretty similar: as with other projects, we
simply determine if we're upgrading a deployment that was previously
using sqlalchemy-migrate, upgrading a deployment that has already
migrated to alembic, or deploying a new deployment, and adjust
accordingly.
In addition, we also have to consider Taskflow's migrations. These were
previously being run once as part of the legacy
006_add_persistence_tables migrations. Since Taskflow uses Alembic under
the hood, it's safe to run every time. The presence of Taskflow does
force us to use a different table name in Masakari though.
Note that one curious side-effect of this is that the order than table
rows are purged change. This appears to be happening because the
notification table is now being created in the initial Alembic
migration, which alters the return value of 'MetaData.sorted_tables'.
In any case, the fix is simply a case of adjusting this order in the
tests.
Change-Id: I5285d7cc3c6da0059c0908cedc195b2262cb1fce
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Alembic's auto-generate functionality doesn't preserve index names or
column order. This causes comparisons between databases to fail. Update
the initial alembic migration to match the schema's generated by
sqlalchemy-migrate.
Change-Id: I16763a82cc2db9138882ace2dad6f0cae330b9e8
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This was generated using the same approach first used in heat [1], but
with a difference to account for the fact that, unlike heat, we have a
migration added in recent times. We generate our initial schema
automatically based on our models using alembic's auto-generate
functionality:
alembic --config masakari/db/sqlalchemy/alembic.ini \
revision -m 'Initial empty revision'
alembic --config masakari/db/sqlalchemy/alembic.ini \
upgrade head
alembic --config masakari/db/sqlalchemy/alembic.ini \
revision -m 'Auto-generated revision' --autogenerate
The two files were then combined, formatting and imports adjusted. We
then *remove* the parts of the migration related to the 'vm_moves' table
and run the autogeneration step again:
alembic --config masakari/db/sqlalchemy/alembic.ini \
revision -m 'Add vm moves table' --autogenerate
As noted above, this is necessary since we're going to need to support
alembic-based migrations for users jumping from e.g. Zed to Bobcat.
Once done, the schema of these migrations and the sqlalchemy-migrate
migrations are compared. This step was done by comparing the schemas of
a database created by the sqlalchemy-migrate tool to the one created by
alembic. First, we compared the initial alembic migration which
corresponds to the state of the database after sqlalchemy-migrate
migration 007:
alembic --config masakari/db/sqlalchemy/alembic.ini \
upgrade 8f848eb45d03
python masakari/db/sqlalchemy/migrate_repo/manage.py version_control \
sqlite:///masakari-old.db 000
python masakari/db/sqlalchemy/migrate_repo/manage.py upgrade \
sqlite:///masakari-old.db 007
With the two databases created, we can compare them using the
methodologies described in [2]. Once this is done, we do the same things
for the final migration to generate its own alembic variation of the
migration.
alembic --config masakari/db/sqlalchemy/alembic.ini \
upgrade head
python masakari/db/sqlalchemy/migrate_repo/manage.py upgrade \
sqlite:///masakari-old.db
These last steps highlight some small differences between the schemas.
These changes are kept to a separate change to make them more obvious.
[1] https://review.opendev.org/c/openstack/heat/+/878350
[2] https://that.guru/blog/comparing-nova-db-migrations/
Change-Id: I6d0f27ba1d81e75010e8b56c70172ccf32c1abb3
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
No migrations yet: this is simply the output of 'alembic init' with some
minor tweaks.
Change-Id: Ib9423c2f751d7ec0a0dec89bdc39f9b6ab043655
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
oslo.db has killed the idea of the migration wrappers. This knob is
therefore no longer useful.
Change-Id: Ia61b3291668b17b13d5f674c0dceff29a7b9b4cf
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
We also bump the hacking version and remove some unnecessary notes from
the top of the requirements files: these don't apply with the dependency
resolver introduced in pip 20.3.
Change-Id: Ifebaec916264bfd10eec13040295719fd47ae107
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
1.Add sqlalchemy-migrate dependence in test-requirements.txt.
It will remove the dependence after this project switchs to alembic.
Remove "check-requirements" temporarily.
2.Fix unit tests broken by olso.utils
Some of the object unit tests grub Mock object unintentionally, and
that results in failure during initializing an versioned object,
because the Mock object does not present its version correctly.
3.make functional jobs not voting
Fix functional jobs later.
Change-Id: Id3c952a54e77952cfd0d76d259f49a14edca1d8b
A recent change to oslo.messaging to redact contexts before sending them
to a notification bus broke masakari due to the invalid default for
read_deleted. This ensures that a "None" value for read_deleted ends up
being interpreted as a 'none'.
Related-bug: #2030976
Change-Id: Ic9b895601f301c1e9ba135766493d4234fa9b50b
Focal is no longer supported in devstack, and the PTI indicates this
should be testing against Jammy.
Change-Id: I4780d25e45b694527bf644cd228104f4429f6e6f
Python 3.11 job run on Debian Bookworm which does
not the mysql pkg for example, mariadb-server, libmysqlclient-dev
instead they have mariadb. Updating bindep.txt file
to adjust the pkgs as per Debian bookworm.
Change-Id: Ic1008c7368bd1f658c48ddaef260e33d14358a03
This tool provides a extensible way to configure pre-commit hooks
to do something from running linters to spell-checking and it proved
to be useful.
Change-Id: I77cf6c85559989bd051798e601640e0039342924
Add file to the reno documentation build to show release notes for
stable/2023.1.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.1.
Sem-Ver: feature
Change-Id: I9c0c1aebda728918029eb05f307c285581d8f0d7
