Move the public API out of oslo.db to oslo_db. Retain the ability to
import from the old namespace package for backwards compatibility for
this release cycle.
Blueprint: drop-namespace-packages
Change-Id: Ie96b482b9fbcb1d85203ad35bb65c1f43e912a44
The two regexp-based filters in exc_filters.py->_is_db_connection_error()
were both incorrectly formed. The tests for these filters would pass
because the fixture also set the is_disconnect flag to True, which
normally would be set by SQLAlchemy, and therefore the
_raise_operational_errors_directly_filter() or possibly
the _raise_for_remaining_DBAPIError() filters would catch this,
view the is_disconnect flag as True, and promote to a DBConnectionError.
However, the _is_db_connection_error() filters are intended to
work independently of whether this flag is set; so two new tests
are added which unset the flag, and ensure that the exception messages
as given are caught here.
Change-Id: I37ddd669b89669730ae1ff07c7bc7a6ba5705f67
This patch applies upgrades to the sqlalchemy/exc_filters.py and
sqlalchemy/compat/handle_error.py compatibility layers to accommodate
new changes in SQLAlchemy 1.0. SQLA 1.0 will now route errors that
occur upon connect through the handle_error() event, just like any other,
so that when 1.0 is present we no longer need to use
exc_filters.handle_connect_error; the method becomes a passthrough
as far as running the event handler. Additionally, SQLAlchemy 1.0
has added the "engine" parameter to ExceptionContext, specifically
to suit the case when the initial connect has failed and there is
no Connection object; the compatibility layer here now emulates
this behavior for SQLAlchemy versions prior to 1.0.
Change-Id: I61714f3c32625a621eaba501d20346519b8b12c7
Using nova-api in the wild shows that we've got an additional
exception, that must be treated like a deadlock, so that we
can handle this error properly and retry the transaction.
Closes-Bug: #1394298
Change-Id: If75f2b5984979fe55ad04ccec8713989307c56ae
We can get 'PRIMARY KEY must be unique' error on some
platforms and versions of sqlite library while trying to
insert the row with the same primary key that already exist.
In this case oslo.db should raise DBDuplicateEntry error.
Add corresponding filter to _sqlite_dupe_key_error
Closes-Bug: #1386145
Change-Id: Ifafd6a8e0b613a31e596043071aef4d410a976f2
is_backend_avail() helper function calls _ensure_backenv_available()
method,which creates a SQLAlchemy engine and opens a test connection,
but doesn't call engine.dispose(). Depending on Python interpreter
version used, the connection in the pool may remain open for some
time (even though, we don't store a reference to an Engine instance).
Closes-Bug: #1393633
Co-Authored-By: Victor Sergeyev <vsergeyev@mirantis.com>
Change-Id: I0297af709ed18fed52308500c13054cb49adf401
In python3.x the 'de' variable will be removed from the
scope after the except block exits (if it ever is entered)
so we need to use a different variable name to ensure that
it will not be deleted so we can use it later.
This avoids errors of the format:
UnboundLocalError: local variable 'de' referenced before
assignment
Change-Id: I9ea2bd6b3a8c392f8d27b0130dd6b7683fc44e7c
This change is the first part in a series of changes
that will allow for full flexibility in database usage
during tests. The first step is to add more facility
to the oslo.db.sqlalchemy.provision system, implementing
a dispatch system that allows flexibility in creation
and dropping of databases, as well as moving the awareness
of the "openstack_citest" convention into provisioning.
The OpportunisticFixture and OpportunisticTestCase now
fold into DbFixture and DbTestCase, which defers in a simple
way to provision.ProvisionedDatabase for all connectivity.
ProvisionedDatabase in turn decides based on the given
environment as to how an engine should be provisioned
for a given test.
Control of database connectivity remains via the
OS_TEST_DBAPI_ADMIN_CONNECTION environment variable. When not
set, connectivity defaults to sqlite://, plus those backends
found to be available using "opportunistic" naming conventions.
When the variable is present, it provides a semicolon-delimited
list of URLs, and only those URLs will be used for initial
connectivity.
Future changes will allow provisioning to hold onto a single
database engine per test run, as well as allow a single
test class to run in multiple backend scenarios (e.g. one test
against many different backends).
Change-Id: Ifc02505c4c8ebd4a1ca56e14f76c0989576875c3
Partial-Bug: #1339206
Currently, it's enforced in devstack through setting encoding in
connection URI. It would be beneficial if oslo.db handles that on its
own, so that we can
- remove those connection arguments from devstack;
- more importantly, not rely on users to set it in all affected
projects.
Oursql and MySQLdb also require use_unicode=0 to avoid performance drop.
Change-Id: I676c9c5e418905160e4ab647b8a0f2fb601e8962
Closes-Bug: 1340779
Add method that checks that foreign keys in models are synchonized
with foreign keys that are created in database.
Change-Id: I4a776da0f53a79218ed4b111cac33b37d264fa1b
This function is currently otherwise unused and does not
work correctly on SQLite, not only dropping the
target unique constraint but instead dropping all unique
constraints from the target table prior to SQLAlchemy
version 1.0 (unreleased). The tests miss that it isn't
working as well unless SQLAlchemy 1.0 is used, in which
case they fail.
Change-Id: Icaff621fba4df289247e96b87c439c62e91543d6
Closes-bug: #1377646
Use import_module instead of try_import so that any import exception
includes a full stack trace and the original error message.
Change-Id: I64449688f22a224b590807317472f26e64f377a3
Before return create_engine() tests the connectivity to a DB. At this
moment a DB might be unavailable for some reason, in which case
create_engine() retries a few times. When used with MySQL, we didn't
actually retry, but instead failed on the first connectivity error
when trying to get the value of sql_mode session variable.
Closes-Bug: #1376211
Co-Authored-By: Mike Bayer <mike_mp@zzzcomputing.com>
Change-Id: I14e25cfe1ed51b0d51a94e491b7267f26e42d34e
The "reconnect" listener that occurs on transaction begin
is not sufficient for libraries that use engines, connections
and sessions in "autocommit" mode, where no transaction is begun.
So we move our "ping" from the begin phase to the "connect"
phase instead. We use a newer SQLAlchemy event "engine_connect"
so that we can make use of the Connection and all the disconnect/
invalidation facilities that are included, rather than using
the low-level pool listener which is somewhat legacy.
Since the event was added in SQLAlchemy 0.9.0, a compatibility
layer is added which emulates the mechanics of the
handle_error compatibility layer.
Change-Id: I719fe45e657e747e5f8d64f20059e2b61c79b63d
Closes-bug: #1374497
The DialectFunctionDispatcher.dispatch_for() decorator method
necessarily returns the dispatcher itself and not the
decorated function, so that the object can continue to be
re-used even if the function name is the same as that of the
dispatcher. In order to support a single function being
wrapped by the dispatcher multiple times with different
criteria, dispatch_for() will now check for the last function
wrapped and use that.
Change-Id: I331670d9b76ae30e7a666648e7e2d4c72641c9ff
Closes-Bug: #1373568
The six module provides a helper wraps function that correctly
tracks the original wrapped function in python 2.x and 3.x so
use it where we can so that we maintain better compatability with
3.x
Change-Id: Ifa994bf63153f58f0aa1f6203a963ac7312ac2b1
Use "database" instead of "db" in help string as noticed during
review of https://review.openstack.org/#/c/120200/
Change-Id: I6e17c759ae101af23b7d57c13dfba26aa6feddb6
Due to pep-3155 functions and methods will now have more
uniform qualified names that we can and should use when
we can (mainly applicable to python 3.x).
Once https://review.openstack.org/#/c/122495/ goes in we
can just use that module instead to get similar functionality.
Change-Id: Iea9e74aca38aa79c9294fd6e786551e91143900c
With Fixture objects and test cases it is important to upcall first
rather than later to allow base classes to configure their internal
test tracking state. Failing to do so allows bugs like this one to
occur - when failure occurs early, the state required to gather
environmental details about the failure is not prepared, and the
test machinery will raise secondary exceptions as it tries to do that.
This may even obscure the original error.
I've taken a fairly minimal approach here - I think it would be
possible and likely even desirable to get rid of the test classes
altogether - this module appears to have very little, if any, actual
test code - its all glue for managing the test database connections.
However I know zzzeek is working on a fairly large reorg in this
area and I don't want to conceptually clash with that - we can do
further cleanups later.
Co-Authored-By: Robert Collins <rbtcollins@hp.com>
Change-Id: Ic9f24aa2352d097a10be8420b701fa9f588b21b0
Closes-Bug: #1330763
Add description of the diff list that is generated in
test_modes_sync function for readability reason.
Change-Id: I3ecaf2da5a22b1835e0c6604b59f9cd1874747de
Instead of having a customized next() and __next__() methods
just use the mix-in that six provides and used it to avoid
having to declare our own next() compatability function.
Change-Id: I1a5fc9bd43783577af879476ca46afcc668185f5
If Column which type is Enum has server_default it is compared
incorrectly for PostgreSQL.
For example, Column('test', sa.Enum('one', 'two', name='enum'),
server_default='one').
In model it has DefaultClause('one', for_update=False)) alembic
finds it different from server variant "'one'::enum".
Change-Id: Id30858f9c4c7a2bfb7f9313d47930845fe354a70
When we intercept SQLite transactions for BEGIN, check
a marker that we place there to indicate BEGIN has
already been called, which we then remove on commit or
rollback. This is to resolve the issue of the fact
that we currently use the StaticPool implementation
with SQLite, which shares a single SQLite connection
for all requests; any API method which makes use
of multiple sessions at once (not a good idea, but
this is prevalent throughout nova, neutron) will
therefore share the same SQLite connection with
multiple SQLAlchemy Connection wrappers that are not
aware of the existing state. While this practice
should be corrected, for now the marker, which is
local to the SQLite connection as stored by the
pool, will track when BEGIN is safe to call, or
has already been called.
Change-Id: I70d44104412bd99d8c7713eb6cd9ff0f80c5da34
Closes-Bug: #1367354
This instruction to set up a move in six for the mox module conflicts
with the one in oslotest. We are trying to use mox3 everywhere, so
remove this instruction and let oslotest handle it.
Imported from the incubator change with the same ID.
Change-Id: I59d5799283233f8411044ddb15c8abfc8850014c
It turns out the test was wrong as the error message used was not a good
one. Real world tests showed that, so I've updated the test and the code
to work.
Refactored tests to use real backends, fix error message for SQLite.
Co-Authored-By: Victor Sergeyev <vsergeyev@mirantis.com>
Change-Id: Ifa64478a9ac9ed0a3c317a8974615b770c85f313
Rename _walk_versions(), _migrate_up(), _migrate_down()
and mark them deprecated as they will be removed in few
releases. These functions will be the part of public API
as this class will be used for migrations in other
OpenStack projects, so there is no need to make them
'private'.
Change-Id: Ic9358445e60a0dd43a5900e8bda7b12f2bebf679
Switch to manually generated rst files for the API documentation so we
do not expose private parts of the library.
Fix formatting of usage.rst
Convert bullet list to section headers to fix rendering issues and make
the docs more readable.
Fix formatting of docstrings in classes exposed in the docs to eliminate
warnings/errors from Sphinx.
Add history.rst
Change-Id: I6f500775f801558f7c0c29f180b60f83a7150e02