SQLAlchemy 1.4 will be changing the method by which it
renders an IN expression such that the bound parameters
are not fully rendered until the statement is executed,
rather than when the statement is compiled into its initial
string form [1]. Adjust tests which are asserting the presence
of IN expressions based on the compiled form to allow for an
arbitrary expression within the parenthesis.
[1] https://github.com/sqlalchemy/sqlalchemy/issues/4645
Change-Id: I8640ad0cbd96647ed74678631479190146bd908e
This addresses two sources of deprecation warnings.
The collections package has moved ABC classes under collections.abc. Six
does not support this move yet, so this updates the code to try to
import from the newer locations, and if that fails, import from the old
location.
Py36 is also more strict about escape sequences in strings. This happens
move often with regex strings that are valid regex but not a valid
normal string escape sequence. This addresses those errors by switching
to raw strings.
Change-Id: I4c61df6b6432b135297f38c02b4538e4ba56be51
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Today, when a user uses enginefacade reader/writer decorators
he/she is forced to pass the context argument as the
first positional argument. performing a decorated function
call with this argument passed as a keyword would lead to
an out of range exception.
This patch proposes to add support for the context argument
to be passed as a keyword argument in decorated functions.
e.g
@enginefacade.reader
def foo(context):
...
foo(context=ctxt) <- will now be possible
Change-Id: Ief0b71bf9a7eb75935612431bdcc26d33bce852d
The database option min_pool_size is not actually
used in oslo.db.
This is a folllow up to
Ib07b7b4200b5cf3a34b8eed441e6754da7d446cf
Change-Id: I89754e9e4b42c45259729b0b0bb0a4bf464a981c
MariasDB/Galera cluster server produces an error like:
(1213, u'Deadlock: wsrep aborted transaction')
which is currently not detected properly and, thus, not retried.
Change-Id: Ifb4a81d50e490e734e12986e51efd6bd3c339c31
Signed-off-by: Pierre-Samuel Le Stang <pierre-samuel.le-stang@corp.ovh.com>
Some module reloading scenarios such as that which occurs
within mod_wsgi mean that an existing module-level enginefacade
is already in the "started" state, however initialization
routines from the calling application may still attempt to
call the ``.configure`` method. Add a new flag is_started
to both _TransactionContextManager and _TransactionFactory so
that calling code can check for this state ahead of time;
additionally, promote the TypeError raised to a specific subclass
enginefacade.AlreadyStartedError to allow for better optimistic
schemes.
Change-Id: I2f5a9e35c2fae0c28b78beef3dcd2c4794362766
References: I704196711d30c1124e713ac31111a8ea6fa2f1ba
Raising warnings from an exception module can result in a rather
confusing message:
Running command: '/usr/bin/nova-scheduler '
/.../sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported
exception.NotSupportedWarning
For someone not used to Python's 'warning' infrastructure, this can look
worse than it is, as seen at [1].
While this particular warning was resolved in change I4a5f8958c3e, it's
possible that a similar warning could be raised for other options. Best
to move this out of the 'exception' module and into something more
sensible.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1639423
Change-Id: I8e0480fb11123067a111ed6aeda6b47614615645
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Python 3.7 now makes "async" a keyword, so deprecate the
"async" name and replace with "async_". Additionally,
use attribute name access only in order to implement and
test "async" as it won't work under Python 3.7.
Closes-Bug: #1775866
Change-Id: Ifb2c59c2cfbd615b04570b31fcfe868f4ffea3c6
Added a new test case that covers comparison of a MySQL server default
defined as text("0") in the model as well as in the database. The
MySQL default comaprison logic in oslo.db was not accommodating of the
model side here being a text() construct. As modern Alembic
versions supports most of the use cases tested here,
remove all server defaut comparison logic except for the boolean
comparison workaround.
Alembic minimum is bumped to 0.9.6; the issue repaired between
0.8.10 and 0.9.6 is http://alembic.zzzcomputing.com/en/latest/
changelog.html#change-5459a7552a17d8b658f0f710bedf6da0
Depends-on: I1a8244edae47b04b7b1fed54e577d6534945bcd8
Closes-bug: #1776527
Change-Id: I48a103e28d86128ea84466da4d9d6faab1dd9a9d
The @oslo_db.api.wrap_db_retry is used for db.api methods
retrying. If db_error raised, the decorator help us to call
the api methods again after a few seconds(accurately, is
2**retry_times seconds).
If the db_error is deadlock error, the old wrap_db_retry
is not so suitable anymore. As we know, some deadlocks cause
because we call some methods(transactions) concurrently. If
we only retry after stable 2**retry_times seconds, we will
recall the method concurrently again.
In order to minimize the chance of regenerating a deadlock
and reduce the average sleep time, we propose to add some
random jitter to the delay period by default when the
deadlock error is detected.
Change-Id: I206745708570f1f292529ff58eee9b83fc09a9f2
Closes-bug: #1737869
Downstream projects have started using the fixtures inside
of oslo_db.tests in response to the deprecations added in
I0163e637ffef6d45d2573ebe29b5438911d01fce. This was not
the intent of these deprecations, so add messsaging to
this effect and ensure fixtures used by the test suite
itself are noted as private.
Change-Id: I3afe0a440a3ab66904aaecf556948df9d4e16b8e
The Nova project has made the decision to store the entire
contents of the "sql_connection" field in the database, with
alterations to the host value, such that various "cells" database
URLs can be located using Nova's database itself as a central
registry of database URLs.
This architecture has produced several problems. The first
is that it is impossible to apply parameters to the URL that
are local to the client machine; the leading example of this
is the MySQL "bind_host" variable, which must match the hostname
of the connecting host. Because cells puts these URLs into the
database and shares them with all controllers, we have to use a workaround
with the read_default_file parameter to specify a controller-local
file of options; this is not a standard thing
for other database drivers, and these parameters only apply to
the MySQL driver and not the SQLAlchemy engine.
The next issue is that it is inconvenient to add parameters
to the URL at all, once Nova has already been running, as one
must manually use Nova's command line tools
to alter all the URLs that have already been copied into the database
and alter the query parameters with each of those individually,
and then restart *all* services who will all receive the parameter
(no way to add params to just one controller).
Nova's "store the URL in the database" feature only needs to be able
to locate the host / database name of the alternate database,
and not change the URL tuning parameters. This patch adds a new
oslo.db parameter connection_parameters which allows the params to
be applied separately from the sql_connection parameter, so that
Nova can continue persisting sql_connection but the parameters remain
local to the nova.conf file. A URL parameter that truly had to remain
persisted in Nova's database (there aren't any) could still be applied
at the sql_connection level.
This feature is essential not just so that we can again place
simple parameters into controller-local files like "bind_host", but
also to allow for configuration of SQLAlchemy features such as
plugins that do connection pool monitoring.
Change-Id: Id4de4b09ec4719cbf8b372629fcf58cf368a33d4
Repaired the "synchronous_reader" modifier of enginefacade so that it
refers to the "writer" engine when set to True, thereby allowing
"synchronous" behavior with the writer. When set to False, this is
"asynchronous", so this should be associated with the async engines.
The flag had the reverse behavior previously.
Change-Id: Id7fea7562ba90eb710176d497af103303f230531
Closes-bug: #1746116
In Python3, assertRegexpMatches & assertNotRegexpMatches
are deprecated in favor of assertRegex and assertNotRegex
Change-Id: I35fd27e9618dcc6e229267a9af215ca15a47db69
The method for skipping a test in unittest is skipTest(). testtools also
provides skip(), but that is deprecated. This eliminates the deprecation
warnings by switching to the recommended skipTest().
Change-Id: I45da95958e6f65475beab0d7a3c6de6604fa82d7
method get_connect_string is deprecated and will be removed
from oslo_db in a few releases.Please use the provisioning
system for dealing with URLs and database provisioning.
method is_backend_avail is deprecated and will be removed
from oslo_db in a few releases.Please use the provisioning
system to access databases based on backend availability.
Change-Id: Idc27d45e96aad7b28b4332b0d8ab8c10bca73e67
Adjust a test that attempts to verify that a CHECK constraint
has been dropped to use raw SQL, as the Boolean datatype
in SQLAlchemy 1.1 is coercing the target value of "10" to a "1"
in any case. Starting with SQLAlchemy 1.2, the Python-side
Boolean datatype will also raise an error.
Change-Id: I99ffce63f9646323d9e2e3079e52b4829d996ea6
Adds two new functions:
- get_foreign_key_constraint_name() - common function
used in subprojects which retrieves the name of a
foreign key constraint given table name / column name
- suspend_fk_constraints_for_col_alter() - given table name and column
name, locates matching foreign key constraints, drops them,
yields context, then recreates them as they were.
By default only runs for NDB thus avoiding expensive
constraint regeneration on other backends.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Co-authored-by: Paul Bourke <paul.bourke@oracle.com>
Change-Id: I76c0d82fddf38add0ca05be270264b6e2d043f7f
The change b3869d04cff7071c1226758eb8b58fde9eba5b8d was fixing #1615938
only for some cases, where the unicity of the row was done by one
column.
This change fixes it for other cases.
Change-Id: I4acb382c770b168f29fa35f02707fb22d402b13b
Closes-bug: #1615938
Stop throwing the deprecated DbMigrationError exception,
Use the new DBMigrationError exception instead,
so that the consumers can catch the new exception.
DbMigrationError will not be removed until no consumers
still use it.
Change-Id: Iab0566cf9f4552e91fa417e64472fa106e8bc86d
optimize_db_test_loader is replaced by optimize_package_test_loader
in test_fixtures.py.So this function is deprecated for removal.
Change-Id: I12cdf846f1e23d74ae8a87bea33fbc8477974bfc
Closes-Bug:#1715572
Remove AutoString* in ndb.py including
AutoStringTinyText,AutoStringText,AutoStringSize.
Change-Id: I707d69facb8f4542550243c1b5eff051264f3204
Closes-Bug:#1714913
This class in new edtion is useless.
So class InsertFromSelect is
deprecated for removal.
Change-Id: I200486a5a88e78b5223b9364fbea5901048d5cb9
Closes-Bug:#1714768
The "idle_timeout" name and documentation are misleading to users.
Rename the option to connection_recycle_time to more closely
match that of the underlying SQLAlchemy argument name
"pool_recycle", which refers to a time period for which
a connection present in the pool for longer than that number
of seconds will be closed and replaced with a new one *upon checkout*,
to prevent usage of stale connections that may have been closed by
server-side reapers.
Change-Id: I5e39d5ac81eaa8d85b1dfb18862fe27695ab8b96
SQLAlchemy has 2 options of implementing error event listener. It can
either raise exception directly or it can return exception into chain.
With the first option all other handlers are ignored (ref:
http://docs.sqlalchemy.org/en/latest/core/events.html?highlight=context#sqlalchemy.events.ConnectionEvents.handle_error)
OSProfiler is interested to subscribe to SQLAlchemy error events.
However with the current approach only Oslo.DB gets the events.
Change-Id: Ia6bccd5af5f2e38f4333f1cc435f6bb9c12bc09f
Related-Bug: 1706299
This simplifies the ndb "AutoStringXYZ" series
of classes into a single oslo_db.sqlalchemy.types.String
type which features all necessary behaviors via
two new flags, mysql_ndb_length and mysql_ndb_type.
Change-Id: I7f9c830073bf9a30abce0aa4bb55b5c9cf661afe
/home/sam/Work/ironic/.tox/py27/local/lib/python2.7/site-packages/oslo_db/tests/sqlalchemy/__init__.py:20:
DeprecationWarning: Function
'oslo_db.sqlalchemy.test_base.optimize_db_test_loader()' has moved to
'oslo_db.sqlalchemy.test_fixtures.optimize_package_test_loader()'
Change-Id: I7fb4e776cedb8adcf97c9a43210049c60f796873
1.As mentioned in [1], we should avoid using six.iteritems to achieve iterators.
We can use dict.items instead, as it will return iterators in PY3 as well.
And dict.items/keys will more readable.
2.In py2, the performance about list should be negligible, see the link [2].
[1] https://wiki.openstack.org/wiki/Python3
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
Change-Id: If76115f706e9e100a1780471af0f9c5d92d3bc9f
The connection.info dictionary comes from the underlying
pooled connection, which in a disconnection situation
is not there; trying to access it makes it try to reconnect.
Inside of rollback(), SQLAlchemy doesn't allow us to
reconnect yet because we haven't finished removing the
transctional state that corresponds to the now-discarded
connection. Therefore we have to check connection.invalidated
inside of our rollback handlers before trying to add/remove tokens
from connection.info.
Change-Id: Icd10ada68d6d53410ac88eca3577f04c5e30087e
Resolves-bug: #1704474
https://review.openstack.org/427970 broke compatibility with
SQLAlchemy versions <1.1.0 by using sqlalchemy.dialects.mysql.types
in test_ndb.py.
Change-Id: I773e0931d4f9f345f8095559eff2eb35e0730027