Python 3.7 now makes "async" a keyword, so deprecate the
"async" name and replace with "async_". Additionally,
use attribute name access only in order to implement and
test "async" as it won't work under Python 3.7.
Closes-Bug: #1775866
Change-Id: Ifb2c59c2cfbd615b04570b31fcfe868f4ffea3c6
The sheer act of registering the 'use_tpool' option seems to be enough
to cause the following warning:
NotSupportedWarning: Configuration option(s) ['use_tpool'] not
supported
Add it to the list of ignored opts, thus silencing this warning.
Change-Id: I4a5f8958c3e0fc68fa3a4ae979f284c4fbd9b82b
The Nova project has made the decision to store the entire
contents of the "sql_connection" field in the database, with
alterations to the host value, such that various "cells" database
URLs can be located using Nova's database itself as a central
registry of database URLs.
This architecture has produced several problems. The first
is that it is impossible to apply parameters to the URL that
are local to the client machine; the leading example of this
is the MySQL "bind_host" variable, which must match the hostname
of the connecting host. Because cells puts these URLs into the
database and shares them with all controllers, we have to use a workaround
with the read_default_file parameter to specify a controller-local
file of options; this is not a standard thing
for other database drivers, and these parameters only apply to
the MySQL driver and not the SQLAlchemy engine.
The next issue is that it is inconvenient to add parameters
to the URL at all, once Nova has already been running, as one
must manually use Nova's command line tools
to alter all the URLs that have already been copied into the database
and alter the query parameters with each of those individually,
and then restart *all* services who will all receive the parameter
(no way to add params to just one controller).
Nova's "store the URL in the database" feature only needs to be able
to locate the host / database name of the alternate database,
and not change the URL tuning parameters. This patch adds a new
oslo.db parameter connection_parameters which allows the params to
be applied separately from the sql_connection parameter, so that
Nova can continue persisting sql_connection but the parameters remain
local to the nova.conf file. A URL parameter that truly had to remain
persisted in Nova's database (there aren't any) could still be applied
at the sql_connection level.
This feature is essential not just so that we can again place
simple parameters into controller-local files like "bind_host", but
also to allow for configuration of SQLAlchemy features such as
plugins that do connection pool monitoring.
Change-Id: Id4de4b09ec4719cbf8b372629fcf58cf368a33d4
Repaired the "synchronous_reader" modifier of enginefacade so that it
refers to the "writer" engine when set to True, thereby allowing
"synchronous" behavior with the writer. When set to False, this is
"asynchronous", so this should be associated with the async engines.
The flag had the reverse behavior previously.
Change-Id: Id7fea7562ba90eb710176d497af103303f230531
Closes-bug: #1746116
getargspec has been deprecated in py3 with plans to remove it in py3.6.
The recommendation is to move to inspect.signature, but the results
of that call are different than the existing one.
There is also getfullargspec available under py3 that was originally
deprecated, but for the sake of handling 2/3 code, it has been
un-deprecated. This call uses signature internally, but returns a mostly
compatible result with what getargspec did. This handles getargspec
deprecation by just using getfullargspec instead if it is available.
Change-Id: I32dd29bde99f9d59c2b0e14dd6782162963b43ae
The "idle_timeout" name and documentation are misleading to users.
Rename the option to connection_recycle_time to more closely
match that of the underlying SQLAlchemy argument name
"pool_recycle", which refers to a time period for which
a connection present in the pool for longer than that number
of seconds will be closed and replaced with a new one *upon checkout*,
to prevent usage of stale connections that may have been closed by
server-side reapers.
Change-Id: I5e39d5ac81eaa8d85b1dfb18862fe27695ab8b96
Enables new functionality for selecting InnoDB or MySQL Cluster
as a DB storage backend in OpenStack services.
Closes-Bug: 1564110
Change-Id: I9f1fd2a87fdf75332de2339d3ff4f08ce9220dcf
Maybe we need to know the original exception when exception occurs on
method 'session.rollback()'.
Change-Id: Iccda37a3279d6f24de69cfccef3e29787f1ac694
Closes-Bug: #1684329
Attaching a reference to the context to the session/connection
is useful for SQLAlchemy event handlers that only get access to
the session and need access to the context for additional details.
The use case driving this is the related bug where we want to pass
conditional update constraints set in the context via the HTTP API
down into an SQLAlchemy event listener.
Related-Bug: #1493714
Change-Id: I5c08e672782344e8778187e257e03c1a1d1b019a
We deprecated config options sqlite_db in July, 2016 [1], and
it shows up in many projects' config files but these projects
don't use it either, so it's confusing to our users. It's time
to remove it, just use config option connection or slave_connection
to connect to the database.
[1] https://review.openstack.org/#/c/338047/
Closes-Bug: #1329086
Change-Id: Id269d921e40edf95eb977b011f1753f633b79d18
When converting existing session context manager code to
use enginefacade, normally a flush() is implicit on
a "subtransaction" when it closes. Enginefacade
has not maintained this behavior. Ideally, this should
have been on by default when enginefacade was first
released, but for now add this as a behavioral
option for projects that require it.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Closes-Bug: #1664643
Change-Id: I6f4f439928588cff954e749dfa938425892e0931
Fixed two issues in patch_engine() as used in test suites
where facade-level arguments would not be propagated to the
internal _TestTransactionFactory, and also if the parent
factory weren't "started" the patch operation would fail
as it assumes a started factory.
This removes the "synchronous_reader" argument from
_TestTransactionFactory as this is propagated from the parent
_TransactionFactory, where synchronous_reader defaults to True.
However, arbitrary keyword arguments are currently allowed to
allow for outside projects calling _TestTransactionFactory
(currently Neutron).
Change-Id: I15f85a0ebfd732c09d7cb92d885c1773c393aabd
The rollback_reader_sessions parameter on
_TransactionContext is unused; it is instead acquired from the
top level factory. Remove the parameter and also move the
documentation for it up to _TransactionFactory.configure_defaults.
As there was no documentation for parameters here, add docs
for all the other parameters too.
Change-Id: I30a2548ad038ae218933c5802c6a8b761705d834
The original idea of enginefacade was that applications
would call upon the global _TransactionContextManager
given in oslo_db.sqlalchemy.enginefacade. However, as it
turns out, virtually no Openstack projects seem to be
using that technique, and instead, everyone is creating
their own ad-hoc _TransactionContextManager objects
and usually establishing it as a module-level global.
Nova has two of them.
Additionally, projects add configuration to these
enginefacades (which IS part of the original idea), and
this configuration in some cases is necessary to be present
for tests that run as well, a key example being the
sqlite_fks flag. The original DbFixture integration
provided no way of reusing this configuration.
Finally, projects very much tend to use custom fixtures
in order to define their database communication.
Test classes themselves don't really make use of
oslo_db's DbTestCase anymore.
This patch introduces a modernized fixture system
which, in conjunction with the recent provisioning
patch, addresses these use cases. Applications will typically
create their own subclasses of these fixtures up front
to suit the various testing cases they have, including
SQLite fixed, SQLite ad-hoc, and opportunistic.
In order to accommodate the fixture-based flow
along with the use of testresources for opportunistic
database provisioning, a mixin class OpportunisticDbTestMixin
is still needed when a test needs to use "opportunistic"
testing in order to provide the .resources attribute.
The calculation of .resources is moved into the fixture
system, but because this attribute is consulted before
setUp(), the "opportunistic" fixture must be created
early and stored.
Closes-Bug: #1548960
Change-Id: I0163e637ffef6d45d2573ebe29b5438911d01fce
Adds new methods make_new_manager(), patch_factory(),
patch_engine(), to provide support to test fixtures that
wish to generate new managers from an existing one, and to
cross-patch managers, factories and engines into other managers.
Change-Id: I83c6296d992f08117fff1438bc079bbf71db787b
Propagates logging_name through enginefacade.configure()
to create_engine(), as will be used in the enginefacade-based
test suite.
Change-Id: Iabc03adbef9792e0be4d86c98851890846638efc
Add on_engine_create argument to transaction_context to enable users
to take some actions with newly created engines. Main intended usage
is providing oslo.db and osprofiler users with a way to hook them
together without making them depend on each other.
Closes-bug: #1600739
Change-Id: I78bef4979c2000d05658ce17d0348cd0a10c24d9
Openstack applications such as Neutron will refer to
the engine.pool.dispose() method in order to remove
any remaining connections, typically as part of
a test suite. Add a public-facing method to the context
and its underlying factory so this can be called without
underscore access.
Change-Id: I18b16e002a8507eb2eaa3a7f168e853b0ca17aa5
Allow to specify custom JSON serializer / deserializer functions to be
used by DB API dialects such as psycopg2 or pymysql for native JSON
types in DBs.
Change-Id: Iad98ca1e10169239f1f6b94b1b598f653b14f0c4
The previous behavior was to assume that a RequestContext was the first
argument of the wrapped method. This has meant creating staticmethods
within classes in order to be wrapped properly. It's not always
desirable to do so and should not be necessary.
Change-Id: I76b0d9c80dd7841c1333f8b8a2ccdd15e55dfcff
Closes-bug: 1520195
In Nova we need a decorator that is READER by default
but can be used as ASYNC if asynchronous session is started before.
Co-Authored-By: Sergey Nikitin <snikitin@mirantis.com>
Change-Id: I0c0f332d87bf6b1213c800763918340b1e226934
When LegacyEngineFacade is passed a _TransactionFactory
that is not started, or one that fails to start, the
_start() method should be called again each time the
factory is called upon; this ensures that the failed
start either can be made to succeed, or at least re-raises
the same error as earlier, instead of falling into attribute
access for non-existent attributes.
Change-Id: I7484f837d455c1f69c448e5ccadc277d19af47e7
Closes-Bug: #1515326
This public method was added to be able to create several
legacy facades from several transaction context managers.
It could be used in Nova where we have two DBs.
There we need some legal way to create two legacy facades
because right now in Nova we create facades by using
private field _factory. It is not quite right.
Change-Id: I96b3dbcbc5a05ac025c247ae6d8abfc988d0295e
There were some issues with the docstrings, and there are
probably still issues that I didn't get around to fixing.
Change-Id: I84d60621246776f77f9022f0f1d61792edccbfb0
When generating the docs some warnings were generated.
Also, configure doc build so that if there are warnings, the
build will fail.
Change-Id: I7e10499294cf1f06f766cbff392e9de32ea48e5a
In order to pickle a context object which refers
to an active _TransactionContextTLocal, we have to
use __reduce__ so that no database, engine, or session
state is pickled, and only a constructor remains. Upon
unpickling, the constructor will install a new, empty
_TransactionContextTLocal on the new context, as is
appropriate since this is a totally different context object
than the original one.
Change-Id: Ia50cf10ff91a013ee24773c095a3df69ae06847b
Check explicitly for the cases where no "sql_connection"
attribute was set when running _start(), so that the
lack of this parameter is documented by the
exception rather than failing into create_engine()
with an unclear failure mode.
If _start() fails as it will here, make sure _started
stays False so that repeated calls to _start() continue
to raise the same exception, rather than raising
attribute errors. When accessing the "session" or
"connection" attributes of the context when these
attributes were not requested by the decorator or
context manager, raise explicit exceptions
for each, rather than returning None which leads to
hard-to-debug NoneType errors.
Change-Id: Iadfbf4707daed4140285a3a472009f6863b18275
Closes-bug: 1477080
Some projects are calling EngineFacade._session_maker for
some use cases. To suit these cases more cleanly on the legacy
API, add a public get_sessionmaker() method.
Change-Id: Ica9574d70a6b8be80bd1a5e7cf250828455c5ade
This change removes RequestContext implicit decoration as some
applications have their own session management for the moment or don't
need this feature. Moreover this implicit decoration doesn't help
debugging.
This change doesn't disallow RequestContext decoration but applications
should require it explicitly:
from oslo_db.sqlalchemy import enginefacade
enginefacade.transaction_context_provider(oslo_context.RequestContext)
Closes-Bug: #1468707
Change-Id: I143f30c41e788c7aa9887c0e994f49ee55c94651
This module presents a replacement for the EngineFacade
system. At the center is the oslo.db.sqlalchemy.enginefacade
module, which when imported, provides decorators and context
managers which perform all database and ORM connectivity
functions transparently. The docstrings as well
as the blueprint provide an introduction.
The patch includes a refactoring of sqlalchemy/session.py
into three dependent modules engines.py, orm.py and
enginefacade.py. This is to maintain a non-cyclical import
structure as well as to maintain the import behavior of
oslo.db overall, as some projects such as glance currently
have dependencies on this structure.
There is also a slimming down and attempt at modernizing
some very old documentation in session.py. The enginefacade
system should be preferred moving forward.
Implements: blueprint make-enginefacade-a-facade
Change-Id: I9a3d0c26bb727eb2c0bd823b9a12fde57cc7c9c3
