226 Commits

Author SHA1 Message Date
Mike Bayer
fce052b2af Use regex to compare SQL strings with IN
SQLAlchemy 1.4 will be changing the method by which it
renders an IN expression such that the bound parameters
are not fully rendered until the statement is executed,
rather than when the statement is compiled into its initial
string form [1].   Adjust tests which are asserting the presence
of IN expressions based on the compiled form to allow for an
arbitrary expression within the parenthesis.

[1] https://github.com/sqlalchemy/sqlalchemy/issues/4645

Change-Id: I8640ad0cbd96647ed74678631479190146bd908e
2019-12-11 11:04:26 -05:00
Mike Bayer
fce2a271c5 Use connect, not contextual_connect
Fixed one test that is using the deprecated, removed in
SQLAlchemy 1.4 contextual_connect method.

Change-Id: Ic9d23be6742bfc0aed4e032f3e1e01d5acf90131
2019-07-17 14:33:58 -04:00
Zuul
b9c51e295b Merge "Support context function argument as keyword" 2019-05-15 19:49:20 +00:00
Zuul
0a50e18965 Merge "Removing deprecated min_pool_size" 2019-04-24 14:35:10 +00:00
Sean McGinnis
3c50cb5cce Fix deprecation warnings under py36
This addresses two sources of deprecation warnings.

The collections package has moved ABC classes under collections.abc. Six
does not support this move yet, so this updates the code to try to
import from the newer locations, and if that fails, import from the old
location.

Py36 is also more strict about escape sequences in strings. This happens
move often with regex strings that are valid regex but not a valid
normal string escape sequence. This addresses those errors by switching
to raw strings.

Change-Id: I4c61df6b6432b135297f38c02b4538e4ba56be51
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2019-04-11 11:03:59 +00:00
Adrian Chiris
7c164cf938 Support context function argument as keyword
Today, when a user uses enginefacade reader/writer decorators
he/she is forced to pass the context argument as the
first positional argument. performing a decorated function
call with this argument passed as a keyword would lead to
an out of range exception.

This patch proposes to add support for the context argument
to be passed as a keyword argument in decorated functions.

e.g

@enginefacade.reader
def foo(context):
    ...

foo(context=ctxt)  <- will now be possible

Change-Id: Ief0b71bf9a7eb75935612431bdcc26d33bce852d
2019-04-07 19:16:59 +03:00
Erik Olof Gunnar Andersson
6cfdc11ea1 Removing deprecated min_pool_size
The database option min_pool_size is not actually
used in oslo.db.

This is a folllow up to
Ib07b7b4200b5cf3a34b8eed441e6754da7d446cf

Change-Id: I89754e9e4b42c45259729b0b0bb0a4bf464a981c
2019-04-06 12:54:28 -07:00
Pierre-Samuel Le Stang
ae227eb3b9 exc_filters: fix deadlock detection for MariaDB/Galera cluster
MariasDB/Galera cluster server produces an error like:

(1213, u'Deadlock: wsrep aborted transaction')

which is currently not detected properly and, thus, not retried.

Change-Id: Ifb4a81d50e490e734e12986e51efd6bd3c339c31
Signed-off-by: Pierre-Samuel Le Stang <pierre-samuel.le-stang@corp.ovh.com>
2019-02-08 16:46:21 +01:00
Zuul
fb12d340e8 Merge "Move warnings to their own module" 2018-10-30 03:35:54 +00:00
Mike Bayer
5b36e169ea Add "is_started" flag to enginefacade
Some module reloading scenarios such as that which occurs
within mod_wsgi mean that an existing module-level enginefacade
is already in the "started" state, however initialization
routines from the calling application may still attempt to
call the ``.configure`` method.  Add a new flag is_started
to both _TransactionContextManager and _TransactionFactory so
that calling code can check for this state ahead of time;
additionally, promote the TypeError raised to a specific subclass
enginefacade.AlreadyStartedError to allow for better optimistic
schemes.

Change-Id: I2f5a9e35c2fae0c28b78beef3dcd2c4794362766
References: I704196711d30c1124e713ac31111a8ea6fa2f1ba
2018-10-24 13:45:50 -04:00
Stephen Finucane
1f0f5654a7 Move warnings to their own module
Raising warnings from an exception module can result in a rather
confusing message:

  Running command: '/usr/bin/nova-scheduler '
  /.../sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported
    exception.NotSupportedWarning

For someone not used to Python's 'warning' infrastructure, this can look
worse than it is, as seen at [1].

While this particular warning was resolved in change I4a5f8958c3e, it's
possible that a similar warning could be raised for other options. Best
to move this out of the 'exception' module and into something more
sensible.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1639423

Change-Id: I8e0480fb11123067a111ed6aeda6b47614615645
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2018-10-17 14:04:42 +01:00
Zuul
dff11337f5 Merge "Rename enginefacade.async to enginefacade.async_" 2018-07-04 00:40:44 +00:00
Mike Bayer
df6bf34012 Rename enginefacade.async to enginefacade.async_
Python 3.7 now makes "async" a keyword, so deprecate the
"async" name and replace with "async_".  Additionally,
use attribute name access only in order to implement and
test "async" as it won't work under Python 3.7.

Closes-Bug: #1775866
Change-Id: Ifb2c59c2cfbd615b04570b31fcfe868f4ffea3c6
2018-07-02 18:02:24 -04:00
Mike Bayer
43af1cf083 Remove most server_default comparison logic
Added a new test case that covers comparison of a MySQL server default
defined as text("0") in the model as well as in the database.   The
MySQL default comaprison logic in oslo.db was not accommodating of the
model side here being a text() construct.   As modern Alembic
versions supports most of the use cases tested here,
remove all server defaut comparison logic except for the boolean
comparison workaround.

Alembic minimum is bumped to 0.9.6; the issue repaired between
0.8.10 and 0.9.6 is http://alembic.zzzcomputing.com/en/latest/
changelog.html#change-5459a7552a17d8b658f0f710bedf6da0

Depends-on: I1a8244edae47b04b7b1fed54e577d6534945bcd8
Closes-bug: #1776527
Change-Id: I48a103e28d86128ea84466da4d9d6faab1dd9a9d
2018-06-15 13:31:38 +10:00
Zuul
6a91f1e8b4 Merge "Improve exponential backoff for wrap_db_retry" 2018-04-17 19:14:18 +00:00
Yikun Jiang
4c20534179 Improve exponential backoff for wrap_db_retry
The @oslo_db.api.wrap_db_retry is used for db.api methods
retrying. If db_error raised, the decorator help us to call
the api methods again after a few seconds(accurately, is
2**retry_times seconds).

If the db_error is deadlock error, the old wrap_db_retry
is not so suitable anymore. As we know, some deadlocks cause
because we call some methods(transactions) concurrently. If
we only retry after stable 2**retry_times seconds, we will
recall the method concurrently again.

In order to minimize the chance of regenerating a deadlock
and reduce the average sleep time, we propose to add some
random jitter to the delay period by default when the
deadlock error is detected.

Change-Id: I206745708570f1f292529ff58eee9b83fc09a9f2
Closes-bug: #1737869
2018-04-13 09:26:12 +08:00
Mike Bayer
39c52cc7c9 Ensure all test fixtures in oslo_db.tests are private
Downstream projects have started using the fixtures inside
of oslo_db.tests in response to the deprecations added in
I0163e637ffef6d45d2573ebe29b5438911d01fce.   This was not
the intent of these deprecations, so add messsaging to
this effect and ensure fixtures used by the test suite
itself are noted as private.

Change-Id: I3afe0a440a3ab66904aaecf556948df9d4e16b8e
2018-03-02 16:39:56 -05:00
Zuul
9c6695969f Merge "Allow connection query string to be passed separately." 2018-02-20 22:02:00 +00:00
Mike Bayer
9993efdeef Allow connection query string to be passed separately.
The Nova project has made the decision to store the entire
contents of the "sql_connection" field in the database, with
alterations to the host value, such that various "cells" database
URLs can be located using Nova's database itself as a central
registry of database URLs.

This architecture has produced several problems.   The first
is that it is impossible to apply parameters to the URL that
are local to the client machine; the leading example of this
is the MySQL "bind_host" variable, which must match the hostname
of the connecting host.  Because cells puts these URLs into the
database and shares them with all controllers, we have to use a workaround
with the read_default_file parameter to specify a controller-local
file of options; this is not a standard thing
for other database drivers, and these parameters only apply to
the MySQL driver and not the SQLAlchemy engine.

The next issue is that it is inconvenient to add parameters
to the URL at all, once Nova has already been running, as one
must manually use Nova's command line tools
to alter all the URLs that have already been copied into the database
and alter the query parameters with each of those individually,
and then restart *all* services who will all receive the parameter
(no way to add params to just one controller).

Nova's "store the URL in the database" feature only needs to be able
to locate the host / database name of the alternate database,
and not change the URL tuning parameters.   This patch adds a new
oslo.db parameter connection_parameters which allows the params to
be applied separately from the sql_connection parameter, so that
Nova can continue persisting sql_connection but the parameters remain
local to the nova.conf file.   A URL parameter that truly had to remain
persisted in Nova's database (there aren't any) could still be applied
at the sql_connection level.

This feature is essential not just so that we can again place
simple parameters into controller-local files like "bind_host", but
also to allow for configuration of SQLAlchemy features such as
plugins that do connection pool monitoring.

Change-Id: Id4de4b09ec4719cbf8b372629fcf58cf368a33d4
2018-01-30 09:08:48 -05:00
Mike Bayer
5ca329884d Reverse role of synchronous_reader
Repaired the "synchronous_reader" modifier of enginefacade so that it
refers to the "writer" engine when set to True, thereby allowing
"synchronous" behavior with the writer.  When set to False, this is
"asynchronous", so this should be associated with the async engines.
The flag had the reverse behavior previously.

Change-Id: Id7fea7562ba90eb710176d497af103303f230531
Closes-bug: #1746116
2018-01-29 18:25:19 -05:00
wanghui
650779240e Fix a typo of "transaction" in comment
Change-Id: I99ca600d4211ef27791465a23100e484a10c323d
2018-01-18 20:09:33 +08:00
Zuul
f91554c2a7 Merge "Remove function optimize_db_test_loader in test_base.py" 2018-01-05 06:38:03 +00:00
Zuul
85cf42e841 Merge "Remove method get_connect_string and is_backend_avail" 2017-12-09 18:14:54 +00:00
Vu Cong Tuan
74638747ce Use assertRegex instead of assertRegexpMatches
In Python3, assertRegexpMatches & assertNotRegexpMatches
are deprecated in favor of assertRegex and assertNotRegex

Change-Id: I35fd27e9618dcc6e229267a9af215ca15a47db69
2017-11-01 16:49:10 +07:00
Zane Bitter
5be38d2c66 Use skipTest() method instead of deprecated skip()
The method for skipping a test in unittest is skipTest(). testtools also
provides skip(), but that is deprecated. This eliminates the deprecation
warnings by switching to the recommended skipTest().

Change-Id: I45da95958e6f65475beab0d7a3c6de6604fa82d7
2017-10-19 18:52:29 -04:00
Zuul
128e5438b1 Merge "Remove property message for DBInvalidUnicodeParameter and InvalidSortKey" 2017-10-19 06:59:19 +00:00
zhangyangyang
73e72f6cc2 Remove method get_connect_string and is_backend_avail
method get_connect_string  is deprecated and will be removed
from oslo_db in a few releases.Please use the provisioning
system for dealing with URLs and database provisioning.

method is_backend_avail  is deprecated and will be removed
from oslo_db in a few releases.Please use the provisioning
system to access databases based on backend availability.

Change-Id: Idc27d45e96aad7b28b4332b0d8ab8c10bca73e67
2017-10-13 13:02:12 +00:00
Mike Bayer
745110e674 Test illegal "boolean" values without Boolean datatype
Adjust a test that attempts to verify that a CHECK constraint
has been dropped to use raw SQL, as the Boolean datatype
in SQLAlchemy 1.1 is coercing the target value of "10" to a "1"
 in any case.   Starting with SQLAlchemy 1.2, the Python-side
Boolean datatype will also raise an error.

Change-Id: I99ffce63f9646323d9e2e3079e52b4829d996ea6
2017-10-12 11:23:58 -04:00
Jenkins
34c4b679fd Merge "Fix pagination when marker value is None" 2017-10-08 17:40:49 +00:00
oorgeron
9585ca8318 Add new foreign key utility function to utils
Adds two new functions:

- get_foreign_key_constraint_name() - common function
  used in subprojects which retrieves the name of a
  foreign key constraint given table name / column name
- suspend_fk_constraints_for_col_alter() - given table name and column
  name, locates matching foreign key constraints, drops them,
  yields context, then recreates them as they were.
  By default only runs for NDB thus avoiding expensive
  constraint regeneration on other backends.

Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Co-authored-by: Paul Bourke <paul.bourke@oracle.com>
Change-Id: I76c0d82fddf38add0ca05be270264b6e2d043f7f
2017-09-22 12:41:10 +00:00
Jenkins
4de33ebd50 Merge "Throw DBMigrationError instead of DbMigrationError." 2017-09-15 15:08:42 +00:00
Mehdi Abaakouk
3a40168056 Fix pagination when marker value is None
The change b3869d04cff7071c1226758eb8b58fde9eba5b8d was fixing #1615938
only for some cases, where the unicity of the row was done by one
column.

This change fixes it for other cases.

Change-Id: I4acb382c770b168f29fa35f02707fb22d402b13b
Closes-bug: #1615938
2017-09-12 14:42:53 +02:00
sunguangning
7a6ae8cefa Remove property message for DBInvalidUnicodeParameter and InvalidSortKey
Change-Id: Idace3cbe1ac3c454443d1c786054dc8ff0b0e8b5
Closes-Bug: #1714842
2017-09-11 17:38:31 +08:00
Yaguo Zhou
9a5a3b3395 Throw DBMigrationError instead of DbMigrationError.
Stop throwing the deprecated DbMigrationError exception,
Use the new DBMigrationError exception instead,
so that the consumers can catch the new exception.
DbMigrationError will not be removed until no consumers
still use it.

Change-Id: Iab0566cf9f4552e91fa417e64472fa106e8bc86d
2017-09-08 11:54:09 +08:00
zhangyangyang
f64264ebb9 Remove function optimize_db_test_loader in test_base.py
optimize_db_test_loader is replaced by optimize_package_test_loader
in test_fixtures.py.So this function is deprecated for removal.

Change-Id: I12cdf846f1e23d74ae8a87bea33fbc8477974bfc
Closes-Bug:#1715572
2017-09-07 19:57:15 +08:00
Jenkins
69910499a7 Merge "Remove AutoString* in ndb.py" 2017-09-05 14:27:32 +00:00
zhangyangyang
c81bb2ff56 Remove AutoString* in ndb.py
Remove AutoString* in ndb.py including
AutoStringTinyText,AutoStringText,AutoStringSize.

Change-Id: I707d69facb8f4542550243c1b5eff051264f3204
Closes-Bug:#1714913
2017-09-04 19:25:57 +08:00
zhangyangyang
3d8fe3e12b Remove class InsertFromSelect
This class in new edtion is useless.
So class InsertFromSelect is
deprecated for removal.

Change-Id: I200486a5a88e78b5223b9364fbea5901048d5cb9
Closes-Bug:#1714768
2017-09-04 11:05:44 +00:00
Jenkins
e9a9701d54 Merge "Replace six.iteritems() with .items()" 2017-08-21 08:02:19 +00:00
Mike Bayer
6634218415 Rename idle_timeout to connection_recycle_time
The "idle_timeout" name and documentation are misleading to users.
Rename the option to connection_recycle_time to more closely
match that of the underlying SQLAlchemy argument name
"pool_recycle", which refers to a time period for which
a connection present in the pool for longer than that number
of seconds will be closed and replaced with a new one *upon checkout*,
to prevent usage of stale connections that may have been closed by
server-side reapers.

Change-Id: I5e39d5ac81eaa8d85b1dfb18862fe27695ab8b96
2017-08-15 10:26:36 -04:00
Jenkins
92c5091d4c Merge "Replace ndb "auto" types with unified String" 2017-08-10 19:57:40 +00:00
Jenkins
8d3a7f05f2 Merge "Let others listen to SQLAlchemy errors" 2017-08-04 02:23:12 +00:00
Ilya Shakhat
d77af64d8e Let others listen to SQLAlchemy errors
SQLAlchemy has 2 options of implementing error event listener. It can
either raise exception directly or it can return exception into chain.
With the first option all other handlers are ignored (ref:
http://docs.sqlalchemy.org/en/latest/core/events.html?highlight=context#sqlalchemy.events.ConnectionEvents.handle_error)

OSProfiler is interested to subscribe to SQLAlchemy error events.
However with the current approach only Oslo.DB gets the events.

Change-Id: Ia6bccd5af5f2e38f4333f1cc435f6bb9c12bc09f
Related-Bug: 1706299
2017-08-03 17:59:32 +02:00
Mike Bayer
41174beecf Replace ndb "auto" types with unified String
This simplifies the ndb "AutoStringXYZ" series
of classes into a single oslo_db.sqlalchemy.types.String
type which features all necessary behaviors via
two new flags, mysql_ndb_length and mysql_ndb_type.

Change-Id: I7f9c830073bf9a30abce0aa4bb55b5c9cf661afe
2017-07-28 11:29:42 -06:00
Sam Betts
24f0402e27 Remove deprecation warning when loading tests/sqlalchemy
/home/sam/Work/ironic/.tox/py27/local/lib/python2.7/site-packages/oslo_db/tests/sqlalchemy/__init__.py:20:
DeprecationWarning: Function
'oslo_db.sqlalchemy.test_base.optimize_db_test_loader()' has moved to
'oslo_db.sqlalchemy.test_fixtures.optimize_package_test_loader()'

Change-Id: I7fb4e776cedb8adcf97c9a43210049c60f796873
2017-07-25 16:56:24 +01:00
liuyamin
3ef34da36c Replace six.iteritems() with .items()
1.As mentioned in [1], we should avoid using six.iteritems to achieve iterators.
We can use dict.items instead, as it will return iterators in PY3 as well.
And dict.items/keys will more readable.
2.In py2, the performance about list should be negligible, see the link [2].

[1] https://wiki.openstack.org/wiki/Python3
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html

Change-Id: If76115f706e9e100a1780471af0f9c5d92d3bc9f
2017-07-19 14:13:15 +08:00
Mike Bayer
7c5b7d3e03 Don't access connection.info if connection is invalidated
The connection.info dictionary comes from the underlying
pooled connection, which in a disconnection situation
is not there; trying to access it makes it try to reconnect.
Inside of rollback(), SQLAlchemy doesn't allow us to
reconnect yet because we haven't finished removing the
transctional state that corresponds to the now-discarded
connection.  Therefore we have to check connection.invalidated
inside of our rollback handlers before trying to add/remove tokens
from connection.info.

Change-Id: Icd10ada68d6d53410ac88eca3577f04c5e30087e
Resolves-bug: #1704474
2017-07-14 16:15:01 -04:00
Jenkins
23e431b0de Merge "Add 'save_and_reraise_exception' method when call 'session.rollback()'" 2017-07-05 11:32:54 +00:00
Javier Pena
17db7e39d1 Fix compatibility with SQLAlchemy < 1.1.0
https://review.openstack.org/427970 broke compatibility with
SQLAlchemy versions <1.1.0 by using sqlalchemy.dialects.mysql.types
in test_ndb.py.

Change-Id: I773e0931d4f9f345f8095559eff2eb35e0730027
2017-06-28 14:31:35 +00:00
Jenkins
8c69759ce9 Merge "Enable MySQL Storage Engine selection" 2017-06-28 08:19:46 +00:00