163 Commits

Author SHA1 Message Date
Doug Hellmann
7063585c60 Move files out of the namespace package
Move the public API out of oslo.db to oslo_db. Retain the ability to
import from the old namespace package for backwards compatibility for
this release cycle.

Blueprint: drop-namespace-packages

Change-Id: Ie96b482b9fbcb1d85203ad35bb65c1f43e912a44
2014-12-24 13:28:48 +02:00
Mike Bayer
0265aa4e01 Repair string-based disconnect filters for MySQL, DB2
The two regexp-based filters in exc_filters.py->_is_db_connection_error()
were both incorrectly formed.  The tests for these filters would pass
because the fixture also set the is_disconnect flag to True, which
normally would be set by SQLAlchemy, and therefore the
_raise_operational_errors_directly_filter() or possibly
the _raise_for_remaining_DBAPIError() filters would catch this,
view the is_disconnect flag as True, and promote to a DBConnectionError.
However, the _is_db_connection_error() filters are intended to
work independently of whether this flag is set; so two new tests
are added which unset the flag, and ensure that the exception messages
as given are caught here.

Change-Id: I37ddd669b89669730ae1ff07c7bc7a6ba5705f67
2014-12-12 12:22:37 +02:00
Mike Bayer
32e5c6096a Upgrade exc_filters for 'engine' argument and connect behavior
This patch applies upgrades to the sqlalchemy/exc_filters.py and
sqlalchemy/compat/handle_error.py compatibility layers to accommodate
new changes in SQLAlchemy 1.0.  SQLA 1.0 will now route errors that
occur upon connect through the handle_error() event, just like any other,
so that when 1.0 is present we no longer need to use
exc_filters.handle_connect_error; the method becomes a passthrough
as far as running the event handler.  Additionally, SQLAlchemy 1.0
has added the "engine" parameter to ExceptionContext, specifically
to suit the case when the initial connect has failed and there is
no Connection object; the compatibility layer here now emulates
this behavior for SQLAlchemy versions prior to 1.0.

Change-Id: I61714f3c32625a621eaba501d20346519b8b12c7
2014-12-06 15:58:22 -05:00
Jenkins
8a37f67050 Merge "Add table name to foreign keys diff" 2014-11-21 11:19:56 +00:00
Jenkins
ff1b6ef15a Merge "Add exception filter for _sqlite_dupe_key_error" 2014-11-21 11:04:23 +00:00
Jenkins
3363683679 Merge "Handle Galera deadlock on SELECT FOR UPDATE" 2014-11-20 22:44:37 +00:00
Oleksii Chuprykov
10e8d15a26 Add table name to foreign keys diff
Add table to diff that is generated by check_foreign_keys
function.

Change-Id: Ib771ea49e873dd87cccacf6ec7463d6e005372a3
2014-11-20 19:26:22 +02:00
Jenkins
f7489fe52c Merge "Fix python3.x scoping issues with removed 'de' variable" 2014-11-19 18:53:25 +00:00
Roman Podoliaka
2269848495 Handle Galera deadlock on SELECT FOR UPDATE
Using nova-api in the wild shows that we've got an additional
exception, that must be treated like a deadlock, so that we
can handle this error properly and retry the transaction.

Closes-Bug: #1394298

Change-Id: If75f2b5984979fe55ad04ccec8713989307c56ae
2014-11-19 19:46:04 +02:00
Oleksii Chuprykov
4b2058b8c1 Add exception filter for _sqlite_dupe_key_error
We can get 'PRIMARY KEY must be unique' error on some
platforms and versions of sqlite library while trying to
insert the row with the same primary key that already exist.
In this case oslo.db should raise DBDuplicateEntry error.
Add corresponding filter to _sqlite_dupe_key_error

Closes-Bug: #1386145

Change-Id: Ifafd6a8e0b613a31e596043071aef4d410a976f2
2014-11-19 16:02:42 +02:00
Roman Podoliaka
7f755bf6fe Ensure is_backend_avail() doesn't leave open connections
is_backend_avail() helper function calls _ensure_backenv_available()
method,which creates a SQLAlchemy engine and opens a test connection,
but doesn't call engine.dispose(). Depending on Python interpreter
version used, the connection in the pool may remain open for some
time (even though, we don't store a reference to an Engine instance).

Closes-Bug: #1393633

Co-Authored-By: Victor Sergeyev <vsergeyev@mirantis.com>
Change-Id: I0297af709ed18fed52308500c13054cb49adf401
2014-11-18 16:05:01 +02:00
Joshua Harlow
135701b202 Fix python3.x scoping issues with removed 'de' variable
In python3.x the 'de' variable will be removed from the
scope after the except block exits (if it ever is entered)
so we need to use a different variable name to ensure that
it will not be deleted so we can use it later.

This avoids errors of the format:

UnboundLocalError: local variable 'de' referenced before
                   assignment

Change-Id: I9ea2bd6b3a8c392f8d27b0130dd6b7683fc44e7c
2014-10-27 21:21:10 -07:00
Mike Bayer
4e19870fd4 Reorganize DbTestCase to use provisioning completely
This change is the first part in a series of changes
that will allow for full flexibility in database usage
during tests.  The first step is to add more facility
to the oslo.db.sqlalchemy.provision system, implementing
a dispatch system that allows flexibility in creation
and dropping of databases, as well as moving the awareness
of the "openstack_citest" convention into provisioning.
The OpportunisticFixture and OpportunisticTestCase now
fold into DbFixture and DbTestCase, which defers in a simple
way to provision.ProvisionedDatabase for all connectivity.
ProvisionedDatabase in turn decides based on the given
environment as to how an engine should be provisioned
for a given test.

Control of database connectivity remains via the
OS_TEST_DBAPI_ADMIN_CONNECTION environment variable.  When not
set, connectivity defaults to sqlite://, plus those backends
found to be available using "opportunistic" naming conventions.
When the variable is present, it provides a semicolon-delimited
list of URLs, and only those URLs will be used for initial
connectivity.

Future changes will allow provisioning to hold onto a single
database engine per test run, as well as allow a single
test class to run in multiple backend scenarios (e.g. one test
against many different backends).

Change-Id: Ifc02505c4c8ebd4a1ca56e14f76c0989576875c3
Partial-Bug: #1339206
2014-10-22 13:02:20 -04:00
Jenkins
8629ddf424 Merge "ModelsMigrationsSync: Add check for foreign keys" 2014-10-21 23:18:05 +00:00
Ihar Hrachyshka
2a6dbcd09f Set utf8 encoding for mysql and postgresql
Currently, it's enforced in devstack through setting encoding in
connection URI. It would be beneficial if oslo.db handles that on its
own, so that we can

- remove those connection arguments from devstack;
- more importantly, not rely on users to set it in all affected
  projects.

Oursql and MySQLdb also require use_unicode=0 to avoid performance drop.

Change-Id: I676c9c5e418905160e4ab647b8a0f2fb601e8962
Closes-Bug: 1340779
2014-10-17 12:05:19 +02:00
Ann Kamyshnikova
1b410567bc ModelsMigrationsSync: Add check for foreign keys
Add method that checks that foreign keys in models are synchonized
with foreign keys that are created in database.

Change-Id: I4a776da0f53a79218ed4b111cac33b37d264fa1b
2014-10-17 14:03:28 +04:00
Jenkins
848a00bce8 Merge "ModelsMigrationsSync:add correct server_default check for Enum" 2014-10-17 09:25:04 +00:00
Jenkins
f95cd72dd4 Merge "Remove extraneous vim editor configuration comments" 2014-10-14 14:00:16 +00:00
Jenkins
b427eb4505 Merge "Improve error reporting for backend import failures" 2014-10-13 13:29:05 +00:00
Christian Berendt
ba4a881530 Remove extraneous vim editor configuration comments
Change-Id: Id8fb52db12695bff4488ea9e74b492a9dabbbccf
Partial-Bug: #1229324
2014-10-08 22:46:07 +02:00
Mike Bayer
33011a51b2 Remove utils.drop_unique_constraint()
This function is currently otherwise unused and does not
work correctly on SQLite, not only dropping the
target unique constraint but instead dropping all unique
constraints from the target table prior to SQLAlchemy
version 1.0 (unreleased).   The tests miss that it isn't
working as well unless SQLAlchemy 1.0 is used, in which
case they fail.

Change-Id: Icaff621fba4df289247e96b87c439c62e91543d6
Closes-bug: #1377646
2014-10-08 14:47:56 -04:00
Doug Hellmann
64f6062cb4 Improve error reporting for backend import failures
Use import_module instead of try_import so that any import exception
includes a full stack trace and the original error message.

Change-Id: I64449688f22a224b590807317472f26e64f377a3
2014-10-06 11:23:03 -04:00
Roman Podoliaka
01a54cc5cc Ensure create_engine() retries the initial connection test
Before return create_engine() tests the connectivity to a DB. At this
moment a DB might be unavailable for some reason, in which case
create_engine() retries a few times. When used with MySQL, we didn't
actually retry, but instead failed on the first connectivity error
when trying to get the value of sql_mode session variable.

Closes-Bug: #1376211

Co-Authored-By: Mike Bayer <mike_mp@zzzcomputing.com>
Change-Id: I14e25cfe1ed51b0d51a94e491b7267f26e42d34e
2014-10-03 18:59:56 +03:00
Jenkins
f8f53cf217 Merge "Use fixture from oslo.config instead of oslo-incubator" 2014-10-02 17:07:01 +00:00
Jenkins
abecb9ea1a Merge "Unwrap DialectFunctionDispatcher from itself." 2014-10-02 17:01:16 +00:00
Jenkins
39c63e7923 Merge "Use six.wraps instead of functools.wraps" 2014-10-02 16:11:09 +00:00
Jenkins
96f610bf72 Merge "Use the six provided iterator mix-in" 2014-10-02 16:11:01 +00:00
Davanum Srinivas
9129545a58 Use fixture from oslo.config instead of oslo-incubator
Change-Id: Ic80562c662960a946b3083f0509700e4809d628d
2014-09-29 07:32:21 -04:00
Mike Bayer
228531013b Move begin ping listener to a connect listener
The "reconnect" listener that occurs on transaction begin
is not sufficient for libraries that use engines, connections
and sessions in "autocommit" mode, where no transaction is begun.
So we move our "ping" from the begin phase to the "connect"
phase instead.   We use a newer SQLAlchemy event "engine_connect"
so that we can make use of the Connection and all the disconnect/
invalidation facilities that are included, rather than using
the low-level pool listener which is somewhat legacy.
Since the event was added in SQLAlchemy 0.9.0, a compatibility
layer is added which emulates the mechanics of the
handle_error compatibility layer.

Change-Id: I719fe45e657e747e5f8d64f20059e2b61c79b63d
Closes-bug: #1374497
2014-09-27 23:30:24 -04:00
Mike Bayer
7a3e091c1c Unwrap DialectFunctionDispatcher from itself.
The DialectFunctionDispatcher.dispatch_for() decorator method
necessarily returns the dispatcher itself and not the
decorated function, so that the object can continue to be
re-used even if the function name is the same as that of the
dispatcher.   In order to support a single function being
wrapped by the dispatcher multiple times with different
criteria, dispatch_for() will now check for the last function
wrapped and use that.

Change-Id: I331670d9b76ae30e7a666648e7e2d4c72641c9ff
Closes-Bug: #1373568
2014-09-24 15:04:53 -04:00
Joshua Harlow
696f3c1289 Use six.wraps instead of functools.wraps
The six module provides a helper wraps function that correctly
tracks the original wrapped function in python 2.x and 3.x so
use it where we can so that we maintain better compatability with
3.x

Change-Id: Ifa994bf63153f58f0aa1f6203a963ac7312ac2b1
2014-09-22 11:59:15 -07:00
Andreas Jaeger
8fac4c7c0e Update help string to use database
Use "database" instead of "db" in help string as noticed during
review of https://review.openstack.org/#/c/120200/

Change-Id: I6e17c759ae101af23b7d57c13dfba26aa6feddb6
2014-09-22 16:07:29 +02:00
Jenkins
aaf82a4637 Merge "Add description for test_models_sync function" 2014-09-22 05:56:51 +00:00
Jenkins
50e39e1907 Merge "Use __qualname__ if we can" 2014-09-21 08:59:13 +00:00
Jenkins
2849f1bdca Merge "Fixup Fixtures Use in db test classes." 2014-09-20 09:14:52 +00:00
Joshua Harlow
fc8eb627a7 Use __qualname__ if we can
Due to pep-3155 functions and methods will now have more
uniform qualified names that we can and should use when
we can (mainly applicable to python 3.x).

Once https://review.openstack.org/#/c/122495/ goes in we
can just use that module instead to get similar functionality.

Change-Id: Iea9e74aca38aa79c9294fd6e786551e91143900c
2014-09-19 13:03:26 -07:00
Sean Dague
9fcf1dbf75 Fixup Fixtures Use in db test classes.
With Fixture objects and test cases it is important to upcall first
rather than later to allow base classes to configure their internal
test tracking state. Failing to do so allows bugs like this one to
occur - when failure occurs early, the state required to gather
environmental details about the failure is not prepared, and the
test machinery will raise secondary exceptions as it tries to do that.
This may even obscure the original error.

I've taken a fairly minimal approach here - I think it would be
possible and likely even desirable to get rid of the test classes
altogether - this module appears to have very little, if any, actual
test code - its all glue for managing the test database connections.
However I know zzzeek is working on a fairly large reorg in this
area and I don't want to conceptually clash with that - we can do
further cleanups later.

Co-Authored-By: Robert Collins <rbtcollins@hp.com>

Change-Id: Ic9f24aa2352d097a10be8420b701fa9f588b21b0
Closes-Bug: #1330763
2014-09-19 09:10:27 -07:00
Oleksii Chuprykov
6a664b9e8a Add description for test_models_sync function
Add description of the diff list that is generated in
test_modes_sync function for readability reason.

Change-Id: I3ecaf2da5a22b1835e0c6604b59f9cd1874747de
2014-09-19 10:30:21 +00:00
Joshua Harlow
8bc1fb79ba Use the six provided iterator mix-in
Instead of having a customized next() and __next__() methods
just use the mix-in that six provides and used it to avoid
having to declare our own next() compatability function.

Change-Id: I1a5fc9bd43783577af879476ca46afcc668185f5
2014-09-16 12:50:30 -07:00
Ann Kamyshnikova
436dfdc737 ModelsMigrationsSync:add correct server_default check for Enum
If Column which type is Enum has server_default it is compared
incorrectly for PostgreSQL.

For example, Column('test', sa.Enum('one', 'two', name='enum'),
                     server_default='one').
In model it has DefaultClause('one', for_update=False)) alembic
finds it different from server variant "'one'::enum".

Change-Id: Id30858f9c4c7a2bfb7f9313d47930845fe354a70
2014-09-15 15:48:09 +04:00
Jenkins
cc7ae9fecc Merge "Add a check for SQLite transactional state" 2014-09-15 10:18:01 +00:00
Jenkins
2205cd7177 Merge "Clean up documentation" 2014-09-13 12:38:44 +00:00
Jenkins
9977e8a79f Merge "Fix DBReferenceError on MySQL and SQLite" 2014-09-11 23:51:48 +00:00
Jenkins
d71a6ccbb2 Merge "Use single quotes for db schema sanity check" 2014-09-11 05:26:22 +00:00
Mike Bayer
57f499eae5 Add a check for SQLite transactional state
When we intercept SQLite transactions for BEGIN, check
a marker that we place there to indicate BEGIN has
already been called, which we then remove on commit or
rollback.  This is to resolve the issue of the fact
that we currently use the StaticPool implementation
with SQLite, which shares a single SQLite connection
for all requests; any API method which makes use
of multiple sessions at once (not a good idea, but
this is prevalent throughout nova, neutron) will
therefore share the same SQLite connection with
multiple SQLAlchemy Connection wrappers that are not
aware of the existing state.  While this practice
should be corrected, for now the marker, which is
local to the SQLite connection as stored by the
pool, will track when BEGIN is safe to call, or
has already been called.

Change-Id: I70d44104412bd99d8c7713eb6cd9ff0f80c5da34
Closes-Bug: #1367354
2014-09-09 13:13:00 -04:00
Jenkins
8f7681946a Merge "Renaming in WalkVersionsMixin" 2014-09-09 05:02:20 +00:00
Doug Hellmann
bdcfb4a4df Let oslotest manage the six.move setting for mox
This instruction to set up a move in six for the mox module conflicts
with the one in oslotest. We are trying to use mox3 everywhere, so
remove this instruction and let oslotest handle it.

Imported from the incubator change with the same ID.

Change-Id: I59d5799283233f8411044ddb15c8abfc8850014c
2014-09-07 14:59:41 -04:00
Julien Danjou
4ed63fac85 Fix DBReferenceError on MySQL and SQLite
It turns out the test was wrong as the error message used was not a good
one. Real world tests showed that, so I've updated the test and the code
to work.

Refactored tests to use real backends, fix error message for SQLite.

Co-Authored-By: Victor Sergeyev <vsergeyev@mirantis.com>

Change-Id: Ifa64478a9ac9ed0a3c317a8974615b770c85f313
2014-09-05 19:01:59 +03:00
Oleksii Chuprykov
49ae6c0446 Renaming in WalkVersionsMixin
Rename _walk_versions(), _migrate_up(), _migrate_down()
and mark them deprecated as they will be removed in few
releases. These functions will be the part of public API
as this class will be used for migrations in other
OpenStack projects, so there is no need to make them
'private'.

Change-Id: Ic9358445e60a0dd43a5900e8bda7b12f2bebf679
2014-09-05 15:32:50 +03:00
Doug Hellmann
cae12025a9 Clean up documentation
Switch to manually generated rst files for the API documentation so we
do not expose private parts of the library.

Fix formatting of usage.rst

Convert bullet list to section headers to fix rendering issues and make
the docs more readable.

Fix formatting of docstrings in classes exposed in the docs to eliminate
warnings/errors from Sphinx.

Add history.rst

Change-Id: I6f500775f801558f7c0c29f180b60f83a7150e02
2014-09-04 16:27:00 -04:00