x/deb-oslo.db
Code Issues Proposed changes

Retire repo

Browse Source 
This repo was created by accident, use deb-python-oslo.db
instead.

Needed-By: I1ac1a06931c8b6dd7c2e73620a0302c29e605f03
Change-Id: I81894aea69b9d09b0977039623c26781093a397a
This commit is contained in:
Andreas Jaeger 2017-04-17 19:36:32 +02:00
parent e9a0e9d568
commit e291fc297b
98 changed files with 13 additions and 17658 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
.coveragerc
View File

@ -1,8 +0,0 @@
[run]
branch = True
source = oslo_db
omit = oslo_db/tests/*
[report]
ignore_errors = True
precision = 2

21
.gitignore vendored
View File

@ -1,21 +0,0 @@
*~
*.swp
*.pyc
*.log
.coverage
.venv
.tox
cover/
.openstack-common-venv/
skeleton.egg-info/
build/
dist/
AUTHORS
.update-venv/
ChangeLog
*.egg
.testrepository/
.project
.pydevproject
oslo.db.egg-info/
doc/source/api

4
.gitreview
View File

@ -1,4 +0,0 @@
[gerrit]
host=review.openstack.org
port=29418
project=openstack/oslo.db.git

3
.mailmap
View File

@ -1,3 +0,0 @@
# Format is:
# <preferred e-mail> <other e-mail 1>
# <preferred e-mail> <other e-mail 2>

7
.testr.conf
View File

@ -1,7 +0,0 @@
[DEFAULT]
test_command=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-60} \
${PYTHON:-python} -m subunit.run discover -t ./ ./oslo_db/tests $LISTOPT $IDOPTION
test_id_option=--load-list $IDFILE
test_list_option=--list

89
CONTRIBUTING.rst
View File

@ -1,89 +0,0 @@
=================
How to contribute
=================
If you would like to contribute to the development of OpenStack,
you must follow the steps in this page:
http://docs.openstack.org/infra/manual/developers.html
Once those steps have been completed, changes to OpenStack
should be submitted for review via the Gerrit tool, following
the workflow documented at:
http://docs.openstack.org/infra/manual/developers.html#development-workflow
Pull requests submitted through GitHub will be ignored.
Bugs should be filed on Launchpad, not GitHub:
https://bugs.launchpad.net/oslo.db
How to run unit tests
=====================
oslo.db (as all OpenStack projects) uses tox to run unit tests. You can find
general information about OpenStack unit tests and testing with tox in wiki_.
oslo.db tests use PyMySQL as the default MySQL DB API driver (which is true for
OpenStack), and psycopg2 for PostgreSQL. pip will build these libs in your
venv, so you must ensure that you have the required system packages installed
for psycopg2 (PyMySQL is a pure-Python implementation and so needs no
additional system packages). For Ubuntu/Debian they are python-dev, and
libpq-dev. For Fedora/CentOS - gcc, python-devel and postgresql-devel.
There is also a separate env for testing with MySQL-python. If you are suppose
to run these tests as well, you need to install libmysqlclient-dev on
Ubuntu/Debian or mysql-devel for Fedora/CentOS.
The oslo.db unit tests system allows to run unittests on real databases. At the
moment it supports MySQL, PostgreSQL and SQLite.
For testing on a real database backend you need to set up a user
``openstack_citest`` with password ``openstack_citest`` on localhost (some
OpenStack projects require a database named 'openstack_citest' too).
Please note, that this user must have permissions to create and drop databases.
If the testing system is not able to connect to the backend, tests on it will
be skipped.
For PostgreSQL on Ubuntu you can create a user in the following way::
sudo -u postgres psql
postgres=# create user openstack_citest with createdb login password
'openstack_citest';
For MySQL you can use the following commands::
mysql -u root
mysql> CREATE USER 'openstack_citest'@'localhost' IDENTIFIED BY
'openstack_citest';
mysql> GRANT ALL PRIVILEGES ON * . * TO 'openstack_citest'@'localhost';
mysql> FLUSH PRIVILEGES;
Alternatively, you can use `pifpaf`_ to run the unit tests directly without
setting up the database yourself. You still need to have the database software
installed on your system. The following tox environments can be used::
tox -e py27-mysql
tox -e py27-postgresql
tox -e py34-mysql
tox -e py34-postgresql
tox -e py27-all
tox -e py34-all
The database will be set up for you locally and temporarily on each run.
Another way is to start `pifpaf` manually and use it to run the tests as you
wish::
$ eval `pifpaf -g OS_TEST_DBAPI_ADMIN_CONNECTION run postgresql`
$ echo $OS_TEST_DBAPI_ADMIN_CONNECTION
postgresql://localhost/postgres?host=/var/folders/7k/pwdhb_mj2cv4zyr0kyrlzjx40000gq/T/tmpMGqN8C&port=9824
$ tox -e py27
[…]
$ tox -e py34
[…]
# Kill pifpaf once you're done
$ kill $PIFPAF_PID
.. _wiki: https://wiki.openstack.org/wiki/Testing#Unit_Tests
.. _pifpaf: https://github.com/jd/pifpaf

4
HACKING.rst
View File

@ -1,4 +0,0 @@
Style Commandments
==================
Read the OpenStack Style Commandments http://docs.openstack.org/developer/hacking/

175
LICENSE
View File

@ -1,175 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.

20
README.rst
View File

@ -1,20 +0,0 @@
===============================================
oslo.db -- OpenStack Database Pattern Library
===============================================
.. image:: https://img.shields.io/pypi/v/oslo.db.svg
:target: https://pypi.python.org/pypi/oslo.db/
:alt: Latest Version
.. image:: https://img.shields.io/pypi/dm/oslo.db.svg
:target: https://pypi.python.org/pypi/oslo.db/
:alt: Downloads
The oslo db (database) handling library, provides database
connectivity to different database backends and various other helper
utils.
* Free software: Apache license
* Documentation: http://docs.openstack.org/developer/oslo.db
* Source: http://git.openstack.org/cgit/openstack/oslo.db
* Bugs: http://bugs.launchpad.net/oslo.db

13
README.txt Normal file
View File

@ -0,0 +1,13 @@
This project is no longer maintained.
The contents of this repository are still available in the Git
source code management system. To see the contents of this
repository before it reached its end of life, please check out the
previous commit with "git checkout HEAD^1".
Use instead the project deb-python-oslo.db at
http://git.openstack.org/cgit/openstack/deb-python-oslo.db .
For any further questions, please email
openstack-dev@lists.openstack.org or join #openstack-dev on
Freenode.

1
babel.cfg
View File

@ -1 +0,0 @@
[python: **.py]

86
doc/source/conf.py
View File

@ -1,86 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import sys
sys.path.insert(0, os.path.abspath('../..'))
# -- General configuration ----------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = [
'sphinx.ext.autodoc',
#'sphinx.ext.intersphinx',
'oslo_config.sphinxext',
'oslosphinx',
'stevedore.sphinxext'
]
# autodoc generation is a bit aggressive and a nuisance when doing heavy
# text edit cycles.
# execute "export SPHINX_DEBUG=1" in your terminal to disable
# A list of glob-style patterns that should be excluded when looking for source
# files.
exclude_patterns = [
'api/setup.rst', # workaround for https://launchpad.net/bugs/1260495
'api/tests.*', # avoid of docs generation from tests
]
# The suffix of source filenames.
source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'oslo.db'
copyright = u'2014, OpenStack Foundation'
# If true, '()' will be appended to :func: etc. cross-reference text.
add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
add_module_names = True
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
modindex_common_prefix = ['oslo_db.']
# -- Options for HTML output --------------------------------------------------
# The theme to use for HTML and HTML Help pages. Major themes that come with
# Sphinx are currently 'default' and 'sphinxdoc'.
# html_theme_path = ["."]
# html_theme = '_theme'
# html_static_path = ['static']
# Output file base name for HTML help builder.
htmlhelp_basename = '%sdoc' % project
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass
# [howto/manual]).
latex_documents = [
('index',
'%s.tex' % project,
u'%s Documentation' % project,
u'OpenStack Foundation', 'manual'),
]
# Example configuration for intersphinx: refer to the Python standard library.
#intersphinx_mapping = {'http://docs.python.org/': None}

1
doc/source/contributing.rst
View File

@ -1 +0,0 @@
.. include:: ../../CONTRIBUTING.rst

1
doc/source/history.rst
View File

@ -1 +0,0 @@
.. include:: ../../ChangeLog

31
doc/source/index.rst
View File

@ -1,31 +0,0 @@
===============================================
oslo.db -- OpenStack Database Pattern Library
===============================================
The oslo.db (database) handling library, provides database
connectivity to different database backends and various other helper
utils.
.. toctree::
:maxdepth: 2
installation
opts
usage
contributing
history
API Documentation
=================
.. toctree::
:maxdepth: 1
api/autoindex
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

49
doc/source/installation.rst
View File

@ -1,49 +0,0 @@
============
Installation
============
At the command line::
$ pip install oslo.db
You will also need to install at least one SQL backend::
$ pip install psycopg2
Or::
$ pip install PyMySQL
Or::
$ pip install pysqlite
Using with PostgreSQL
---------------------
If you are using PostgreSQL make sure to install the PostgreSQL client
development package for your distro. On Ubuntu this is done as follows::
$ sudo apt-get install libpq-dev
$ pip install psycopg2
The installation of psycopg2 will fail if libpq-dev is not installed first.
Note that even in a virtual environment the libpq-dev will be installed
system wide.
Using with MySQL-python
-----------------------
PyMySQL is a default MySQL DB API driver for oslo.db, as well as for the whole
OpenStack. But you still can use MySQL-python as an alternative DB API driver.
For MySQL-python you must install the MySQL client development package for
your distro. On Ubuntu this is done as follows::
$ sudo apt-get install libmysqlclient-dev
$ pip install MySQL-python
The installation of MySQL-python will fail if libmysqlclient-dev is not
installed first. Note that even in a virtual environment the MySQL package will
be installed system wide.

9
doc/source/opts.rst
View File

@ -1,9 +0,0 @@
=====================
Configuration Options
=====================
oslo.db uses oslo.config to define and manage configuration
options to allow the deployer to control how an application uses the
underlying database.
.. show-options:: oslo.db

169
doc/source/usage.rst
View File

@ -1,169 +0,0 @@
=======
Usage
=======
To use oslo.db in a project:
Session Handling
================
Session handling is achieved using the :mod:`oslo_db.sqlalchemy.enginefacade`
system. This module presents a function decorator as well as a
context manager approach to delivering :class:`.Session` as well as
:class:`.Connection` objects to a function or block.
Both calling styles require the use of a context object. This object may
be of any class, though when used with the decorator form, requires
special instrumentation.
The context manager form is as follows:
.. code:: python
from oslo_db.sqlalchemy import enginefacade
class MyContext(object):
"User-defined context class."
def some_reader_api_function(context):
with enginefacade.reader.using(context) as session:
return session.query(SomeClass).all()
def some_writer_api_function(context, x, y):
with enginefacade.writer.using(context) as session:
session.add(SomeClass(x, y))
def run_some_database_calls():
context = MyContext()
results = some_reader_api_function(context)
some_writer_api_function(context, 5, 10)
The decorator form accesses attributes off the user-defined context
directly; the context must be decorated with the
:func:`oslo_db.sqlalchemy.enginefacade.transaction_context_provider`
decorator. Each function must receive the context argument:
.. code:: python
from oslo_db.sqlalchemy import enginefacade
@enginefacade.transaction_context_provider
class MyContext(object):
"User-defined context class."
@enginefacade.reader
def some_reader_api_function(context):
return context.session.query(SomeClass).all()
@enginefacade.writer
def some_writer_api_function(context, x, y):
context.session.add(SomeClass(x, y))
def run_some_database_calls():
context = MyContext()
results = some_reader_api_function(context)
some_writer_api_function(context, 5, 10)
.. note:: The ``context.session`` and ``context.connection`` attributes
must be accessed within the scope of an appropriate writer/reader block
(either the decorator or contextmanager approach). An AttributeError is
raised otherwise.
The decorator form can also be used with class and instance methods which
implicitly receive the first positional argument:
.. code:: python
class DatabaseAccessLayer(object):
@classmethod
@enginefacade.reader
def some_reader_api_function(cls, context):
return context.session.query(SomeClass).all()
@enginefacade.writer
def some_writer_api_function(self, context, x, y):
context.session.add(SomeClass(x, y))
.. note:: Note that enginefacade decorators must be applied **before**
`classmethod`, otherwise you will get a ``TypeError`` at import time
(as enginefacade will try to use ``inspect.getargspec()`` on a descriptor,
not on a bound method, please refer to the `Data Model
<https://docs.python.org/3/reference/datamodel.html#data-model>`_ section
of the Python Language Reference for details).
The scope of transaction and connectivity for both approaches is managed
transparently. The configuration for the connection comes from the standard
:obj:`oslo_config.cfg.CONF` collection. Additional configurations can be
established for the enginefacade using the
:func:`oslo_db.sqlalchemy.enginefacade.configure` function, before any use of
the database begins:
.. code:: python
from oslo_db.sqlalchemy import enginefacade
enginefacade.configure(
sqlite_fk=True,
max_retries=5,
mysql_sql_mode='ANSI'
)
Base class for models usage
===========================
.. code:: python
from oslo_db.sqlalchemy import models
class ProjectSomething(models.TimestampMixin,
models.ModelBase):
id = Column(Integer, primary_key=True)
...
DB API backend support
======================
.. code:: python
from oslo_config import cfg
from oslo_db import api as db_api
_BACKEND_MAPPING = {'sqlalchemy': 'project.db.sqlalchemy.api'}
IMPL = db_api.DBAPI.from_config(cfg.CONF, backend_mapping=_BACKEND_MAPPING)
def get_engine():
return IMPL.get_engine()
def get_session():
return IMPL.get_session()
# DB-API method
def do_something(somethind_id):
return IMPL.do_something(somethind_id)
DB migration extensions
=======================
Available extensions for `oslo_db.migration`.
.. list-plugins:: oslo_db.sqlalchemy.migration
:detailed:

0
oslo_db/__init__.py
View File

35
oslo_db/_i18n.py
View File

@ -1,35 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""oslo.i18n integration module.
See http://docs.openstack.org/developer/oslo.i18n/usage.html .
"""
import oslo_i18n
_translators = oslo_i18n.TranslatorFactory(domain='oslo_db')
# The primary translation function using the well-known name "_"
_ = _translators.primary
# Translators for log levels.
#
# The abbreviated names are meant to reflect the usual use of a short
# name like '_'. The "L" is for "log" and the other letter comes from
# the level.
_LI = _translators.log_info
_LW = _translators.log_warning
_LE = _translators.log_error
_LC = _translators.log_critical

290
oslo_db/api.py
View File

@ -1,290 +0,0 @@
# Copyright (c) 2013 Rackspace Hosting
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""
=================================
Multiple DB API backend support.
=================================
A DB backend module should implement a method named 'get_backend' which
takes no arguments. The method can return any object that implements DB
API methods.
"""
import logging
import threading
import time
from debtcollector import removals
from oslo_utils import excutils
from oslo_utils import importutils
from oslo_utils import reflection
import six
from oslo_db._i18n import _LE
from oslo_db import exception
from oslo_db import options
LOG = logging.getLogger(__name__)
def safe_for_db_retry(f):
"""Indicate api method as safe for re-connection to database.
Database connection retries will be enabled for the decorated api method.
Database connection failure can have many causes, which can be temporary.
In such cases retry may increase the likelihood of connection.
Usage::
@safe_for_db_retry
def api_method(self):
self.engine.connect()
:param f: database api method.
:type f: function.
"""
f.__dict__['enable_retry_on_disconnect'] = True
return f
def retry_on_deadlock(f):
"""Retry a DB API call if Deadlock was received.
wrap_db_entry will be applied to all db.api functions marked with this
decorator.
"""
f.__dict__['enable_retry_on_deadlock'] = True
return f
def retry_on_request(f):
"""Retry a DB API call if RetryRequest exception was received.
wrap_db_entry will be applied to all db.api functions marked with this
decorator.
"""
f.__dict__['enable_retry_on_request'] = True
return f
class wrap_db_retry(object):
"""Retry db.api methods, if db_error raised
Retry decorated db.api methods. This decorator catches db_error and retries
function in a loop until it succeeds, or until maximum retries count
will be reached.
Keyword arguments:
:param retry_interval: seconds between transaction retries
:type retry_interval: int or float
:param max_retries: max number of retries before an error is raised
:type max_retries: int
:param inc_retry_interval: determine increase retry interval or not
:type inc_retry_interval: bool
:param max_retry_interval: max interval value between retries
:type max_retry_interval: int or float
:param exception_checker: checks if an exception should trigger a retry
:type exception_checker: callable
"""
@removals.removed_kwarg("retry_on_request",
"Retry on request is always enabled")
def __init__(self, retry_interval=1, max_retries=20,
inc_retry_interval=True,
max_retry_interval=10, retry_on_disconnect=False,
retry_on_deadlock=False, retry_on_request=False,
exception_checker=lambda exc: False):
super(wrap_db_retry, self).__init__()
self.db_error = (exception.RetryRequest, )
# default is that we re-raise anything unexpected
self.exception_checker = exception_checker
if retry_on_disconnect:
self.db_error += (exception.DBConnectionError, )
if retry_on_deadlock:
self.db_error += (exception.DBDeadlock, )
self.retry_interval = retry_interval
self.max_retries = max_retries
self.inc_retry_interval = inc_retry_interval
self.max_retry_interval = max_retry_interval
def __call__(self, f):
@six.wraps(f)
def wrapper(*args, **kwargs):
next_interval = self.retry_interval
remaining = self.max_retries
while True:
try:
return f(*args, **kwargs)
except Exception as e:
with excutils.save_and_reraise_exception() as ectxt:
expected = self._is_exception_expected(e)
if remaining > 0:
ectxt.reraise = not expected
else:
if expected:
LOG.exception(_LE('DB exceeded retry limit.'))
# if it's a RetryRequest, we need to unpack it
if isinstance(e, exception.RetryRequest):
ectxt.type_ = type(e.inner_exc)
ectxt.value = e.inner_exc
LOG.debug("Performing DB retry for function %s",
reflection.get_callable_name(f))
# NOTE(vsergeyev): We are using patched time module, so
# this effectively yields the execution
# context to another green thread.
time.sleep(next_interval)
if self.inc_retry_interval:
next_interval = min(
next_interval * 2,
self.max_retry_interval
)
remaining -= 1
return wrapper
def _is_exception_expected(self, exc):
if isinstance(exc, self.db_error):
# RetryRequest is application-initated exception
# and not an error condition in case retries are
# not exceeded
if not isinstance(exc, exception.RetryRequest):
LOG.debug('DB error: %s', exc)
return True
return self.exception_checker(exc)
class DBAPI(object):
"""Initialize the chosen DB API backend.
After initialization API methods is available as normal attributes of
``DBAPI`` subclass. Database API methods are supposed to be called as
DBAPI instance methods.
:param backend_name: name of the backend to load
:type backend_name: str
:param backend_mapping: backend name -> module/class to load mapping
:type backend_mapping: dict
:default backend_mapping: None
:param lazy: load the DB backend lazily on the first DB API method call
:type lazy: bool
:default lazy: False
:keyword use_db_reconnect: retry DB transactions on disconnect or not
:type use_db_reconnect: bool
:keyword retry_interval: seconds between transaction retries
:type retry_interval: int
:keyword inc_retry_interval: increase retry interval or not
:type inc_retry_interval: bool
:keyword max_retry_interval: max interval value between retries
:type max_retry_interval: int
:keyword max_retries: max number of retries before an error is raised
:type max_retries: int
"""
def __init__(self, backend_name, backend_mapping=None, lazy=False,
**kwargs):
self._backend = None
self._backend_name = backend_name
self._backend_mapping = backend_mapping or {}
self._lock = threading.Lock()
if not lazy:
self._load_backend()
self.use_db_reconnect = kwargs.get('use_db_reconnect', False)
self._wrap_db_kwargs = {k: v for k, v in kwargs.items()
if k in ('retry_interval',
'inc_retry_interval',
'max_retry_interval',
'max_retries')}
def _load_backend(self):
with self._lock:
if not self._backend:
# Import the untranslated name if we don't have a mapping
backend_path = self._backend_mapping.get(self._backend_name,
self._backend_name)
LOG.debug('Loading backend %(name)r from %(path)r',
{'name': self._backend_name,
'path': backend_path})
backend_mod = importutils.import_module(backend_path)
self._backend = backend_mod.get_backend()
def __getattr__(self, key):
if not self._backend:
self._load_backend()
attr = getattr(self._backend, key)
if not hasattr(attr, '__call__'):
return attr
# NOTE(vsergeyev): If `use_db_reconnect` option is set to True, retry
# DB API methods, decorated with @safe_for_db_retry
# on disconnect.
retry_on_disconnect = self.use_db_reconnect and attr.__dict__.get(
'enable_retry_on_disconnect', False)
retry_on_deadlock = attr.__dict__.get('enable_retry_on_deadlock',
False)
retry_on_request = attr.__dict__.get('enable_retry_on_request', False)
if retry_on_disconnect or retry_on_deadlock or retry_on_request:
attr = wrap_db_retry(
retry_on_disconnect=retry_on_disconnect,
retry_on_deadlock=retry_on_deadlock,
**self._wrap_db_kwargs)(attr)
return attr
@classmethod
def from_config(cls, conf, backend_mapping=None, lazy=False):
"""Initialize DBAPI instance given a config instance.
:param conf: oslo.config config instance
:type conf: oslo.config.cfg.ConfigOpts
:param backend_mapping: backend name -> module/class to load mapping
:type backend_mapping: dict
:param lazy: load the DB backend lazily on the first DB API method call
:type lazy: bool
"""
conf.register_opts(options.database_opts, 'database')
return cls(backend_name=conf.database.backend,
backend_mapping=backend_mapping,
lazy=lazy,
use_db_reconnect=conf.database.use_db_reconnect,
retry_interval=conf.database.db_retry_interval,
inc_retry_interval=conf.database.db_inc_retry_interval,
max_retry_interval=conf.database.db_max_retry_interval,
max_retries=conf.database.db_max_retries)

81
oslo_db/concurrency.py
View File

@ -1,81 +0,0 @@
# Copyright 2014 Mirantis.inc
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import copy
import logging
import threading
from oslo_config import cfg
from oslo_db._i18n import _LE
from oslo_db import api
LOG = logging.getLogger(__name__)
tpool_opts = [
cfg.BoolOpt('use_tpool',
default=False,
deprecated_name='dbapi_use_tpool',
deprecated_group='DEFAULT',
help='Enable the experimental use of thread pooling for '
'all DB API calls'),
]
class TpoolDbapiWrapper(object):
"""DB API wrapper class.
This wraps the oslo DB API with an option to be able to use eventlet's
thread pooling. Since the CONF variable may not be loaded at the time
this class is instantiated, we must look at it on the first DB API call.
"""
def __init__(self, conf, backend_mapping):
self._db_api = None
self._backend_mapping = backend_mapping
self._conf = conf
self._conf.register_opts(tpool_opts, 'database')
self._lock = threading.Lock()
@property
def _api(self):
if not self._db_api:
with self._lock:
if not self._db_api:
db_api = api.DBAPI.from_config(
conf=self._conf, backend_mapping=self._backend_mapping)
if self._conf.database.use_tpool:
try:
from eventlet import tpool
except ImportError:
LOG.exception(_LE("'eventlet' is required for "
"TpoolDbapiWrapper."))
raise
self._db_api = tpool.Proxy(db_api)
else:
self._db_api = db_api
return self._db_api
def __getattr__(self, key):
return getattr(self._api, key)
def list_opts():
"""Returns a list of oslo.config options available in this module.
:returns: a list of (group_name, opts) tuples
"""
return [('database', copy.deepcopy(tpool_opts))]

328
oslo_db/exception.py
View File

@ -1,328 +0,0 @@
# Copyright 2010 United States Government as represented by the
# Administrator of the National Aeronautics and Space Administration.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""DB related custom exceptions.
Custom exceptions intended to determine the causes of specific database
errors. This module provides more generic exceptions than the database-specific
driver libraries, and so users of oslo.db can catch these no matter which
database the application is using. Most of the exceptions are wrappers. Wrapper
exceptions take an original exception as positional argument and keep it for
purposes of deeper debug.
Example::
try:
statement(arg)
except sqlalchemy.exc.OperationalError as e:
raise DBDuplicateEntry(e)
This is useful to determine more specific error cases further at execution,
when you need to add some extra information to an error message. Wrapper
exceptions takes care about original error message displaying to not to loose
low level cause of an error. All the database api exceptions wrapped into
the specific exceptions provided belove.
Please use only database related custom exceptions with database manipulations
with `try/except` statement. This is required for consistent handling of
database errors.
"""
import debtcollector.removals
import six
from oslo_db._i18n import _
from oslo_utils.excutils import CausedByException
class DBError(CausedByException):
"""Base exception for all custom database exceptions.
:kwarg inner_exception: an original exception which was wrapped with
DBError or its subclasses.
"""
def __init__(self, inner_exception=None, cause=None):
self.inner_exception = inner_exception
super(DBError, self).__init__(six.text_type(inner_exception), cause)
class DBDuplicateEntry(DBError):
"""Duplicate entry at unique column error.
Raised when made an attempt to write to a unique column the same entry as
existing one. :attr: `columns` available on an instance of the exception
and could be used at error handling::
try:
instance_type_ref.save()
except DBDuplicateEntry as e:
if 'colname' in e.columns:
# Handle error.
:kwarg columns: a list of unique columns have been attempted to write a
duplicate entry.
:type columns: list
:kwarg value: a value which has been attempted to write. The value will
be None, if we can't extract it for a particular database backend. Only
MySQL and PostgreSQL 9.x are supported right now.
"""
def __init__(self, columns=None, inner_exception=None, value=None):
self.columns = columns or []
self.value = value
super(DBDuplicateEntry, self).__init__(inner_exception)
class DBConstraintError(DBError):
"""Check constraint fails for column error.
Raised when made an attempt to write to a column a value that does not
satisfy a CHECK constraint.
:kwarg table: the table name for which the check fails
:type table: str
:kwarg check_name: the table of the check that failed to be satisfied
:type check_name: str
"""
def __init__(self, table, check_name, inner_exception=None):
self.table = table
self.check_name = check_name
super(DBConstraintError, self).__init__(inner_exception)
class DBReferenceError(DBError):
"""Foreign key violation error.
:param table: a table name in which the reference is directed.
:type table: str
:param constraint: a problematic constraint name.
:type constraint: str
:param key: a broken reference key name.
:type key: str
:param key_table: a table name which contains the key.
:type key_table: str
"""
def __init__(self, table, constraint, key, key_table,
inner_exception=None):
self.table = table
self.constraint = constraint
self.key = key
self.key_table = key_table
super(DBReferenceError, self).__init__(inner_exception)
class DBNonExistentConstraint(DBError):
"""Constraint does not exist.
:param table: table name
:type table: str
:param constraint: constraint name
:type table: str
"""
def __init__(self, table, constraint, inner_exception=None):
self.table = table
self.constraint = constraint
super(DBNonExistentConstraint, self).__init__(inner_exception)
class DBNonExistentTable(DBError):
"""Table does not exist.
:param table: table name
:type table: str
"""
def __init__(self, table, inner_exception=None):
self.table = table
super(DBNonExistentTable, self).__init__(inner_exception)
class DBDeadlock(DBError):
"""Database dead lock error.
Deadlock is a situation that occurs when two or more different database
sessions have some data locked, and each database session requests a lock
on the data that another, different, session has already locked.
"""
def __init__(self, inner_exception=None):
super(DBDeadlock, self).__init__(inner_exception)
class DBInvalidUnicodeParameter(Exception):
"""Database unicode error.
Raised when unicode parameter is passed to a database
without encoding directive.
"""
@debtcollector.removals.removed_property
def message(self):
# NOTE(rpodolyaka): provided for compatibility with python 3k, where
# exceptions do not have .message attribute, while we used to have one
# in this particular exception class. See LP #1542961 for details.
return str(self)
def __init__(self):
super(DBInvalidUnicodeParameter, self).__init__(
_("Invalid Parameter: Encoding directive wasn't provided."))
class DbMigrationError(DBError):
"""Wrapped migration specific exception.
Raised when migrations couldn't be completed successfully.
"""
def __init__(self, message=None):
super(DbMigrationError, self).__init__(message)
class DBMigrationError(DbMigrationError):
"""Wrapped migration specific exception.
Raised when migrations couldn't be completed successfully.
"""
def __init__(self, message):
super(DBMigrationError, self).__init__(message)
debtcollector.removals.removed_class(DbMigrationError,
replacement=DBMigrationError)
class DBConnectionError(DBError):
"""Wrapped connection specific exception.
Raised when database connection is failed.
"""
pass
class DBDataError(DBError):
"""Raised for errors that are due to problems with the processed data.
E.g. division by zero, numeric value out of range, incorrect data type, etc
"""
class InvalidSortKey(Exception):
"""A sort key destined for database query usage is invalid."""
@debtcollector.removals.removed_property
def message(self):
# NOTE(rpodolyaka): provided for compatibility with python 3k, where
# exceptions do not have .message attribute, while we used to have one
# in this particular exception class. See LP #1542961 for details.
return str(self)
def __init__(self, key=None):
super(InvalidSortKey, self).__init__(
_("Sort key supplied is invalid: %s") % key)
self.key = key
class ColumnError(Exception):
"""Error raised when no column or an invalid column is found."""
class BackendNotAvailable(Exception):
"""Error raised when a particular database backend is not available
within a test suite.
"""
class RetryRequest(Exception):
"""Error raised when DB operation needs to be retried.
That could be intentionally raised by the code without any real DB errors.
"""
def __init__(self, inner_exc):
self.inner_exc = inner_exc
class NoEngineContextEstablished(AttributeError):
"""Error raised for enginefacade attribute access with no context.
This applies to the ``session`` and ``connection`` attributes
of a user-defined context and/or RequestContext object, when they
are accessed *outside* of the scope of an enginefacade decorator
or context manager.
The exception is a subclass of AttributeError so that
normal Python missing attribute behaviors are maintained, such
as support for ``getattr(context, 'session', None)``.
"""
class ContextNotRequestedError(AttributeError):
"""Error raised when requesting a not-setup enginefacade attribute.
This applies to the ``session`` and ``connection`` attributes
of a user-defined context and/or RequestContext object, when they
are accessed *within* the scope of an enginefacade decorator
or context manager, but the context has not requested that
attribute (e.g. like "with enginefacade.connection.using(context)"
and "context.session" is requested).
"""
class CantStartEngineError(Exception):
"""Error raised when the enginefacade cannot start up correctly."""
class NotSupportedWarning(Warning):
"""Warn that an argument or call that was passed is not supported.
This subclasses Warning so that it can be filtered as a distinct
category.
.. seealso::
https://docs.python.org/2/library/warnings.html
"""
class OsloDBDeprecationWarning(DeprecationWarning):
"""Issued per usage of a deprecated API.
This subclasses DeprecationWarning so that it can be filtered as a distinct
category.
.. seealso::
https://docs.python.org/2/library/warnings.html
"""

46
oslo_db/locale/en_GB/LC_MESSAGES/oslo_db-log-error.po
View File

@ -1,46 +0,0 @@
# Translations template for oslo.db.
# Copyright (C) 2015 ORGANIZATION
# This file is distributed under the same license as the oslo.db project.
#
# Translators:
# Andi Chandler <andi@gowling.com>, 2014-2015
# Andreas Jaeger <jaegerandi@gmail.com>, 2016. #zanata
msgid ""
msgstr ""
"Project-Id-Version: oslo.db 4.6.1.dev19\n"
"Report-Msgid-Bugs-To: https://bugs.launchpad.net/openstack-i18n/\n"
"POT-Creation-Date: 2016-04-19 04:28+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2015-03-19 07:04+0000\n"
"Last-Translator: Andi Chandler <andi@gowling.com>\n"
"Language: en-GB\n"
"Plural-Forms: nplurals=2; plural=(n != 1);\n"
"Generated-By: Babel 2.0\n"
"X-Generator: Zanata 3.7.3\n"
"Language-Team: English (United Kingdom)\n"
msgid "'eventlet' is required for TpoolDbapiWrapper."
msgstr "'eventlet' is required for TpoolDbapiWrapper."
msgid "DB exceeded retry limit."
msgstr "DB exceeded retry limit."
msgid "DB exception wrapped."
msgstr "DB exception wrapped."
#, python-format
msgid "DBAPIError exception wrapped from %s"
msgstr "DBAPIError exception wrapped from %s"
#, python-format
msgid "Failed to migrate to version %(ver)s on engine %(eng)s"
msgstr "Failed to migrate to version %(ver)s on engine %(eng)s"
msgid ""
"Migration number for migrate plugin must be valid integer or empty, if you "
"want to downgrade to initial state"
msgstr ""
"Migration number for migrate plugin must be valid integer or empty, if you "
"want to downgrade to initial state"

30
oslo_db/locale/en_GB/LC_MESSAGES/oslo_db-log-info.po
View File

@ -1,30 +0,0 @@
# Translations template for oslo.db.
# Copyright (C) 2015 ORGANIZATION
# This file is distributed under the same license as the oslo.db project.
#
# Translators:
# Andi Chandler <andi@gowling.com>, 2014
# Andreas Jaeger <jaegerandi@gmail.com>, 2016. #zanata
msgid ""
msgstr ""
"Project-Id-Version: oslo.db 4.6.1.dev19\n"
"Report-Msgid-Bugs-To: https://bugs.launchpad.net/openstack-i18n/\n"
"POT-Creation-Date: 2016-04-19 04:28+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2014-11-26 01:04+0000\n"
"Last-Translator: Andi Chandler <andi@gowling.com>\n"
"Language: en-GB\n"
"Plural-Forms: nplurals=2; plural=(n != 1);\n"
"Generated-By: Babel 2.0\n"
"X-Generator: Zanata 3.7.3\n"
"Language-Team: English (United Kingdom)\n"
#, python-format
msgid "Deleting duplicated row with id: %(id)s from table: %(table)s"
msgstr "Deleting duplicated row with id: %(id)s from table: %(table)s"
#, python-format
msgid "The %(dbapi)s backend is unavailable: %(err)s"
msgstr "The %(dbapi)s backend is unavailable: %(err)s"

46
oslo_db/locale/en_GB/LC_MESSAGES/oslo_db-log-warning.po
View File

@ -1,46 +0,0 @@
# Translations template for oslo.db.
# Copyright (C) 2015 ORGANIZATION
# This file is distributed under the same license as the oslo.db project.
#
# Translators:
# Andi Chandler <andi@gowling.com>, 2014-2015
# Andreas Jaeger <jaegerandi@gmail.com>, 2016. #zanata
msgid ""
msgstr ""
"Project-Id-Version: oslo.db 4.6.1.dev19\n"
"Report-Msgid-Bugs-To: https://bugs.launchpad.net/openstack-i18n/\n"
"POT-Creation-Date: 2016-04-19 04:28+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2015-03-19 07:04+0000\n"
"Last-Translator: Andi Chandler <andi@gowling.com>\n"
"Language: en-GB\n"
"Plural-Forms: nplurals=2; plural=(n != 1);\n"
"Generated-By: Babel 2.0\n"
"X-Generator: Zanata 3.7.3\n"
"Language-Team: English (United Kingdom)\n"
msgid "Id not in sort_keys; is sort_keys unique?"
msgstr "Id not in sort_keys; is sort_keys unique?"
#, python-format
msgid ""
"MySQL SQL mode is '%s', consider enabling TRADITIONAL or STRICT_ALL_TABLES"
msgstr ""
"MySQL SQL mode is '%s', consider enabling TRADITIONAL or STRICT_ALL_TABLES"
#, python-format
msgid ""
"Parent process %(orig)s forked (%(newproc)s) with an open database "
"connection, which is being discarded and recreated."
msgstr ""
"Parent process %(orig)s forked (%(newproc)s) with an open database "
"connection, which is being discarded and recreated."
#, python-format
msgid "SQL connection failed. %s attempts left."
msgstr "SQL connection failed. %s attempts left."
msgid "Unable to detect effective SQL mode"
msgstr "Unable to detect effective SQL mode"

85
oslo_db/locale/en_GB/LC_MESSAGES/oslo_db.po
View File

@ -1,85 +0,0 @@
# Translations template for oslo.db.
# Copyright (C) 2015 ORGANIZATION
# This file is distributed under the same license as the oslo.db project.
#
# Translators:
# Andi Chandler <andi@gowling.com>, 2014-2015
# Andreas Jaeger <jaegerandi@gmail.com>, 2016. #zanata
msgid ""
msgstr ""
"Project-Id-Version: oslo.db 4.6.1.dev46\n"
"Report-Msgid-Bugs-To: https://bugs.launchpad.net/openstack-i18n/\n"
"POT-Creation-Date: 2016-06-15 11:18+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2016-06-20 06:31+0000\n"
"Last-Translator: Andreas Jaeger <jaegerandi@gmail.com>\n"
"Language: en-GB\n"
"Plural-Forms: nplurals=2; plural=(n != 1);\n"
"Generated-By: Babel 2.0\n"
"X-Generator: Zanata 3.7.3\n"
"Language-Team: English (United Kingdom)\n"
msgid "Invalid Parameter: Encoding directive wasn't provided."
msgstr "Invalid Parameter: Encoding directive wasn't provided."
#, python-format
msgid ""
"Please specify column %s in col_name_col_instance param. It is required "
"because column has unsupported type by SQLite."
msgstr ""
"Please specify column %s in col_name_col_instance param. It is required "
"because column has unsupported type by SQLite."
#, python-format
msgid "Sort key supplied is invalid: %s"
msgstr "Sort key supplied is invalid: %s"
#, python-format
msgid ""
"Tables \"%s\" have non utf8 collation, please make sure all tables are "
"CHARSET=utf8"
msgstr ""
"Tables \"%s\" have non utf8 collation, please make sure all tables are "
"CHARSET=utf8"
msgid ""
"The database is not under version control, but has tables. Please stamp the "
"current version of the schema manually."
msgstr ""
"The database is not under version control, but has tables. Please stamp the "
"current version of the schema manually."
#, python-format
msgid ""
"There is no `deleted` column in `%s` table. Project doesn't use soft-deleted "
"feature."
msgstr ""
"There is no `deleted` column in `%s` table. Project doesn't use soft-deleted "
"feature."
#, python-format
msgid "There is no `project_id` column in `%s` table."
msgstr "There is no `project_id` column in `%s` table."
#, python-format
msgid "Unknown sort direction, must be one of: %s"
msgstr "Unknown sort direction, must be one of: %s"
msgid "Unsupported id columns type"
msgstr "Unsupported id columns type"
#, python-format
msgid ""
"col_name_col_instance param has wrong type of column instance for column %s "
"It should be instance of sqlalchemy.Column."
msgstr ""
"col_name_col_instance param has wrong type of column instance for column %s "
"It should be instance of sqlalchemy.Column."
msgid "model should be a subclass of ModelBase"
msgstr "model should be a subclass of ModelBase"
msgid "version should be an integer"
msgstr "version should be an integer"

46
oslo_db/locale/es/LC_MESSAGES/oslo_db-log-error.po
View File

@ -1,46 +0,0 @@
# Translations template for oslo.db.
# Copyright (C) 2015 ORGANIZATION
# This file is distributed under the same license as the oslo.db project.
#
# Translators:
# Adriana Chisco Landazábal <achisco94@gmail.com>, 2015
# Andreas Jaeger <jaegerandi@gmail.com>, 2016. #zanata
msgid ""
msgstr ""
"Project-Id-Version: oslo.db 4.6.1.dev19\n"
"Report-Msgid-Bugs-To: https://bugs.launchpad.net/openstack-i18n/\n"
"POT-Creation-Date: 2016-04-19 04:28+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2015-06-22 09:08+0000\n"
"Last-Translator: Adriana Chisco Landazábal <achisco94@gmail.com>\n"
"Language: es\n"
"Plural-Forms: nplurals=2; plural=(n != 1);\n"
"Generated-By: Babel 2.0\n"
"X-Generator: Zanata 3.7.3\n"
"Language-Team: Spanish\n"
msgid "'eventlet' is required for TpoolDbapiWrapper."
msgstr "Se necesita 'eventlet' para TpoolDbapiWrapper."
msgid "DB exceeded retry limit."
msgstr "La base de datos excedió el límite de intentos."
msgid "DB exception wrapped."
msgstr "excepción de base de datos empaquetada."
#, python-format
msgid "DBAPIError exception wrapped from %s"
msgstr "Excepción de error DBAPI empaquetada de %s"
#, python-format
msgid "Failed to migrate to version %(ver)s on engine %(eng)s"
msgstr "Error al migrar a versión %(ver)s en motor %(eng)s"
msgid ""
"Migration number for migrate plugin must be valid integer or empty, if you "
"want to downgrade to initial state"
msgstr ""
"Si desea volver al estado inicial de la versión anterior, el número de "
"migración para el complemento de migración debe ser un entero válido o vacío"

30
oslo_db/locale/es/LC_MESSAGES/oslo_db-log-info.po
View File

@ -1,30 +0,0 @@
# Translations template for oslo.db.
# Copyright (C) 2015 ORGANIZATION
# This file is distributed under the same license as the oslo.db project.
#
# Translators:
# Adriana Chisco Landazábal <achisco94@gmail.com>, 2015
# Andreas Jaeger <jaegerandi@gmail.com>, 2016. #zanata
msgid ""
msgstr ""
"Project-Id-Version: oslo.db 4.6.1.dev19\n"
"Report-Msgid-Bugs-To: https://bugs.launchpad.net/openstack-i18n/\n"
"POT-Creation-Date: 2016-04-19 04:28+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2015-06-22 09:10+0000\n"
"Last-Translator: Adriana Chisco Landazábal <achisco94@gmail.com>\n"
"Language: es\n"
"Plural-Forms: nplurals=2; plural=(n != 1);\n"
"Generated-By: Babel 2.0\n"
"X-Generator: Zanata 3.7.3\n"
"Language-Team: Spanish\n"
#, python-format
msgid "Deleting duplicated row with id: %(id)s from table: %(table)s"
msgstr "Eliminando fila duplicada con id: %(id)s de la tabla: %(table)s"
#, python-format
msgid "The %(dbapi)s backend is unavailable: %(err)s"
msgstr "El %(dbapi)s backend ino está disponible: %(err)s"

47
oslo_db/locale/es/LC_MESSAGES/oslo_db-log-warning.po
View File

@ -1,47 +0,0 @@
# Translations template for oslo.db.
# Copyright (C) 2015 ORGANIZATION
# This file is distributed under the same license as the oslo.db project.
#
# Translators:
# Adriana Chisco Landazábal <achisco94@gmail.com>, 2015
# Andreas Jaeger <jaegerandi@gmail.com>, 2016. #zanata
msgid ""
msgstr ""
"Project-Id-Version: oslo.db 4.6.1.dev19\n"
"Report-Msgid-Bugs-To: https://bugs.launchpad.net/openstack-i18n/\n"
"POT-Creation-Date: 2016-04-19 04:28+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2015-06-22 09:13+0000\n"
"Last-Translator: Adriana Chisco Landazábal <achisco94@gmail.com>\n"
"Language: es\n"
"Plural-Forms: nplurals=2; plural=(n != 1);\n"
"Generated-By: Babel 2.0\n"
"X-Generator: Zanata 3.7.3\n"
"Language-Team: Spanish\n"
msgid "Id not in sort_keys; is sort_keys unique?"
msgstr "Id no está en sort_keys; ¿sort_keys es exclusivo?"
#, python-format
msgid ""
"MySQL SQL mode is '%s', consider enabling TRADITIONAL or STRICT_ALL_TABLES"
msgstr ""
"El modo MySQL de SL es: '%s', considere habilitar TRADITIONAL o "
"STRICT_ALL_TABLES"
#, python-format
msgid ""
"Parent process %(orig)s forked (%(newproc)s) with an open database "
"connection, which is being discarded and recreated."
msgstr ""
"El proceso padre %(orig)s bifurcó (%(newproc)s) con una conexión abierta de "
"base de datos, la cual se está descartando y creando de nuevo."
#, python-format
msgid "SQL connection failed. %s attempts left."
msgstr "La conexión SQL ha fallado. Quedan %s intentos."
msgid "Unable to detect effective SQL mode"
msgstr "No es posible detectar modo SQL efectivo"

82
oslo_db/locale/es/LC_MESSAGES/oslo_db.po
View File

@ -1,82 +0,0 @@
# Translations template for oslo.db.
# Copyright (C) 2015 ORGANIZATION
# This file is distributed under the same license as the oslo.db project.
#
# Translators:
# Adriana Chisco Landazábal <achisco94@gmail.com>, 2015
# Miriam Godinez <miriamgc@hotmail.com>, 2015
# Andreas Jaeger <jaegerandi@gmail.com>, 2016. #zanata
msgid ""
msgstr ""
"Project-Id-Version: oslo.db 4.6.1.dev19\n"
"Report-Msgid-Bugs-To: https://bugs.launchpad.net/openstack-i18n/\n"
"POT-Creation-Date: 2016-04-19 04:28+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2015-09-07 10:45+0000\n"
"Last-Translator: Miriam Godinez <miriamgc@hotmail.com>\n"
"Language: es\n"
"Plural-Forms: nplurals=2; plural=(n != 1);\n"
"Generated-By: Babel 2.0\n"
"X-Generator: Zanata 3.7.3\n"
"Language-Team: Spanish\n"
msgid "Invalid Parameter: Encoding directive wasn't provided."
msgstr "Parámetro no válido: No se proporcionó directiva de codificación."
#, python-format
msgid ""
"Please specify column %s in col_name_col_instance param. It is required "
"because column has unsupported type by SQLite."
msgstr ""
"Por favor especifique la columna %s en el parámetro col_name_col_instance. "
"Es necesario porque la columna tiene un tipo no soportado por SQLite."
#, python-format
msgid ""
"Tables \"%s\" have non utf8 collation, please make sure all tables are "
"CHARSET=utf8"
msgstr ""
"Las tablas \"%s\" no tienen una colación utf8, por favor asegúrese de que "
"todas las tablas sean CHARSET=utf8"
msgid ""
"The database is not under version control, but has tables. Please stamp the "
"current version of the schema manually."
msgstr ""
"La base de datos no está bajo el control de la versión, pero tiene tablas. "
"Por favor indique manualmente la versión actual del esquema."
#, python-format
msgid ""
"There is no `deleted` column in `%s` table. Project doesn't use soft-deleted "
"feature."
msgstr ""
"No existe la columna `deleted` en la tabla `%s`. El projecto no utiliza la "
"característica de eliminación suave."
#, python-format
msgid "There is no `project_id` column in `%s` table."
msgstr "No existe la columna `project_id` en la tabla `%s`."
#, python-format
msgid "Unknown sort direction, must be one of: %s"
msgstr "Clase de dirección desconocida, debe ser una de: %s"
msgid "Unsupported id columns type"
msgstr "Tipo de identificador de columnas no soportado"
#, python-format
msgid ""
"col_name_col_instance param has wrong type of column instance for column %s "
"It should be instance of sqlalchemy.Column."
msgstr ""
"El parámetro col_name_col_instance contiene el tipo incorrecto de instancia "
"de columna para la columna %s. Debe ser una instancia de sqlalchemy.Column."
msgid "model should be a subclass of ModelBase"
msgstr "el modelo debe ser una subclase del ModelBase"
msgid "version should be an integer"
msgstr "la versión debe ser un entero"

47
oslo_db/locale/fr/LC_MESSAGES/oslo_db-log-error.po
View File

@ -1,47 +0,0 @@
# Translations template for oslo.db.
# Copyright (C) 2015 ORGANIZATION
# This file is distributed under the same license as the oslo.db project.
#
# Translators:
# Maxime COQUEREL <max.coquerel@gmail.com>, 2014-2015
# Andreas Jaeger <jaegerandi@gmail.com>, 2016. #zanata
msgid ""
msgstr ""
"Project-Id-Version: oslo.db 4.6.1.dev19\n"
"Report-Msgid-Bugs-To: https://bugs.launchpad.net/openstack-i18n/\n"
"POT-Creation-Date: 2016-04-19 04:28+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2015-02-18 02:50+0000\n"
"Last-Translator: Maxime COQUEREL <max.coquerel@gmail.com>\n"
"Language: fr\n"
"Plural-Forms: nplurals=2; plural=(n > 1);\n"
"Generated-By: Babel 2.0\n"
"X-Generator: Zanata 3.7.3\n"
"Language-Team: French\n"
msgid "'eventlet' is required for TpoolDbapiWrapper."
msgstr "'eventlet' est requis pour poolDbapiWrapper."
msgid "DB exceeded retry limit."
msgstr "DB limite de tentatives dépassé."
msgid "DB exception wrapped."
msgstr "DB exception enveloppé."
#, python-format
msgid "DBAPIError exception wrapped from %s"
msgstr "Exception DBAPIError enveloppé depuis %s"
#, python-format
msgid "Failed to migrate to version %(ver)s on engine %(eng)s"
msgstr "Échec de migration de la version %(ver)s sur le moteur %(eng)s"
msgid ""
"Migration number for migrate plugin must be valid integer or empty, if you "
"want to downgrade to initial state"
msgstr ""
"Le numéro de migration pour la migration de plugin doit être un entier "
"valide ou un champs nul, si vous souhaitez revenir vers l'état initial de la "
"précédente version"

32
oslo_db/locale/fr/LC_MESSAGES/oslo_db-log-info.po
View File

@ -1,32 +0,0 @@
# Translations template for oslo.db.
# Copyright (C) 2015 ORGANIZATION
# This file is distributed under the same license as the oslo.db project.
#
# Translators:
# Maxime COQUEREL <max.coquerel@gmail.com>, 2014
# Andreas Jaeger <jaegerandi@gmail.com>, 2016. #zanata
msgid ""
msgstr ""
"Project-Id-Version: oslo.db 4.6.1.dev19\n"
"Report-Msgid-Bugs-To: https://bugs.launchpad.net/openstack-i18n/\n"
"POT-Creation-Date: 2016-04-19 04:28+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2014-11-23 06:38+0000\n"
"Last-Translator: Maxime COQUEREL <max.coquerel@gmail.com>\n"
"Language: fr\n"
"Plural-Forms: nplurals=2; plural=(n > 1);\n"
"Generated-By: Babel 2.0\n"
"X-Generator: Zanata 3.7.3\n"
"Language-Team: French\n"
#, python-format
msgid "Deleting duplicated row with id: %(id)s from table: %(table)s"
msgstr ""
"Suppression de la ligne dupliquée avec l'id: %(id)s depuis la table: "
"%(table)s"
#, python-format
msgid "The %(dbapi)s backend is unavailable: %(err)s"
msgstr "%(dbapi)s backend n'est pas disponible: %(err)s"

46
oslo_db/locale/fr/LC_MESSAGES/oslo_db-log-warning.po
View File

@ -1,46 +0,0 @@
# Translations template for oslo.db.
# Copyright (C) 2015 ORGANIZATION
# This file is distributed under the same license as the oslo.db project.
#
# Translators:
# Lucas Mascaro <mascaro.lucas@yahoo.fr>, 2015
# Maxime COQUEREL <max.coquerel@gmail.com>, 2014
# Andreas Jaeger <jaegerandi@gmail.com>, 2016. #zanata
msgid ""
msgstr ""
"Project-Id-Version: oslo.db 4.6.1.dev19\n"
"Report-Msgid-Bugs-To: https://bugs.launchpad.net/openstack-i18n/\n"
"POT-Creation-Date: 2016-04-19 04:28+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2015-08-17 11:08+0000\n"
"Last-Translator: Lucas Mascaro <mascaro.lucas@yahoo.fr>\n"
"Language: fr\n"
"Plural-Forms: nplurals=2; plural=(n > 1);\n"
"Generated-By: Babel 2.0\n"
"X-Generator: Zanata 3.7.3\n"
"Language-Team: French\n"
msgid "Id not in sort_keys; is sort_keys unique?"
msgstr "Id n'est pas dans sort_keys; sort_keys est unique ?"
#, python-format
msgid ""
"MySQL SQL mode is '%s', consider enabling TRADITIONAL or STRICT_ALL_TABLES"
msgstr "MySQL SQL mode est '%s', TRADITIONAL ou STRICT_ALL_TABLES"
#, python-format
msgid ""
"Parent process %(orig)s forked (%(newproc)s) with an open database "
"connection, which is being discarded and recreated."
msgstr ""
"Le processus parent %(orig)s s'est séparé (%(newproc)s) avec une connection "
"a la base de donnée ouverte, elle sera rejetée et recréée."
#, python-format
msgid "SQL connection failed. %s attempts left."
msgstr "Echec de connexion SQL. %s tentatives échouées"
msgid "Unable to detect effective SQL mode"
msgstr "Impossible de détecter le mode SQL"

83
oslo_db/locale/fr/LC_MESSAGES/oslo_db.po
View File

@ -1,83 +0,0 @@
# Translations template for oslo.db.
# Copyright (C) 2015 ORGANIZATION
# This file is distributed under the same license as the oslo.db project.
#
# Translators:
# Lucas Mascaro <mascaro.lucas@yahoo.fr>, 2015
# Maxime COQUEREL <max.coquerel@gmail.com>, 2014-2015
# Andreas Jaeger <jaegerandi@gmail.com>, 2016. #zanata
msgid ""
msgstr ""
"Project-Id-Version: oslo.db 4.6.1.dev19\n"
"Report-Msgid-Bugs-To: https://bugs.launchpad.net/openstack-i18n/\n"
"POT-Creation-Date: 2016-04-19 04:28+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2015-08-07 04:24+0000\n"
"Last-Translator: Lucas Mascaro <mascaro.lucas@yahoo.fr>\n"
"Language: fr\n"
"Plural-Forms: nplurals=2; plural=(n > 1);\n"
"Generated-By: Babel 2.0\n"
"X-Generator: Zanata 3.7.3\n"
"Language-Team: French\n"
msgid "Invalid Parameter: Encoding directive wasn't provided."
msgstr "Paramètre non valide : La directive encodage n'a pas été fourni."
#, python-format
msgid ""
"Please specify column %s in col_name_col_instance param. It is required "
"because column has unsupported type by SQLite."
msgstr ""
"Spécifiez la colonne %s dans le paramètre col_name_col_instance. Ceci est "
"obligatoire car la colonne a un type non pris en charge dans SQLite."
#, python-format
msgid ""
"Tables \"%s\" have non utf8 collation, please make sure all tables are "
"CHARSET=utf8"
msgstr ""
"Les tables \"%s\" ont une collation non utf8, assurez-vous que pour toutes "
"les tables CHARSET=utf8."
msgid ""
"The database is not under version control, but has tables. Please stamp the "
"current version of the schema manually."
msgstr ""
"La base de données n'est pas versionnée, mais contient des tables. Veuillez "
"indiquer manuellement la version courante du schéma."
#, python-format
msgid ""
"There is no `deleted` column in `%s` table. Project doesn't use soft-deleted "
"feature."
msgstr ""
"Il n'y a aucune colonne `deleted` dans la table `%s`. Le projet ne peut pas "
"utiliser cette fonctionnalité."
#, python-format
msgid "There is no `project_id` column in `%s` table."
msgstr "Il n'y a pas de colonne `project_id` dans la table `%s`."
#, python-format
msgid "Unknown sort direction, must be one of: %s"
msgstr "Ordre de tris inconnu, il doit être un de: %s"
msgid "Unsupported id columns type"
msgstr "Type de colonnes id non pris en charge"
#, python-format
msgid ""
"col_name_col_instance param has wrong type of column instance for column %s "
"It should be instance of sqlalchemy.Column."
msgstr ""
"Le paramètre col_name_col_instance contient un type d'instance de colonne "
"incorrect pour la colonne %s. Il devrait être une instance de sqlalchemy."
"Column."
msgid "model should be a subclass of ModelBase"
msgstr "model doit etre une sous-classe de ModelBase"
msgid "version should be an integer"
msgstr "version doit être un entier"

229
oslo_db/options.py
View File

@ -1,229 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from debtcollector import removals
from oslo_config import cfg
database_opts = [
cfg.StrOpt('sqlite_db',
deprecated_for_removal=True,
deprecated_reason='Should use config option connection or '
'slave_connection to connect the database.',
deprecated_group='DEFAULT',
default='oslo.sqlite',
help='The file name to use with SQLite.'),
cfg.BoolOpt('sqlite_synchronous',
deprecated_group='DEFAULT',
default=True,
help='If True, SQLite uses synchronous mode.'),
cfg.StrOpt('backend',
default='sqlalchemy',
deprecated_name='db_backend',
deprecated_group='DEFAULT',
help='The back end to use for the database.'),
cfg.StrOpt('connection',
help='The SQLAlchemy connection string to use to connect to '
'the database.',
secret=True,
deprecated_opts=[cfg.DeprecatedOpt('sql_connection',
group='DEFAULT'),
cfg.DeprecatedOpt('sql_connection',
group='DATABASE'),
cfg.DeprecatedOpt('connection',
group='sql'), ]),
cfg.StrOpt('slave_connection',
secret=True,
help='The SQLAlchemy connection string to use to connect to the'
' slave database.'),
cfg.StrOpt('mysql_sql_mode',
default='TRADITIONAL',
help='The SQL mode to be used for MySQL sessions. '
'This option, including the default, overrides any '
'server-set SQL mode. To use whatever SQL mode '
'is set by the server configuration, '
'set this to no value. Example: mysql_sql_mode='),
cfg.IntOpt('idle_timeout',
default=3600,
deprecated_opts=[cfg.DeprecatedOpt('sql_idle_timeout',
group='DEFAULT'),
cfg.DeprecatedOpt('sql_idle_timeout',
group='DATABASE'),
cfg.DeprecatedOpt('idle_timeout',
group='sql')],
help='Timeout before idle SQL connections are reaped.'),
cfg.IntOpt('min_pool_size',
default=1,
deprecated_opts=[cfg.DeprecatedOpt('sql_min_pool_size',
group='DEFAULT'),
cfg.DeprecatedOpt('sql_min_pool_size',
group='DATABASE')],
help='Minimum number of SQL connections to keep open in a '
'pool.'),
cfg.IntOpt('max_pool_size',
default=5,
deprecated_opts=[cfg.DeprecatedOpt('sql_max_pool_size',
group='DEFAULT'),
cfg.DeprecatedOpt('sql_max_pool_size',
group='DATABASE')],
help='Maximum number of SQL connections to keep open in a '
'pool. Setting a value of 0 indicates no limit.'),
cfg.IntOpt('max_retries',
default=10,
deprecated_opts=[cfg.DeprecatedOpt('sql_max_retries',
group='DEFAULT'),
cfg.DeprecatedOpt('sql_max_retries',
group='DATABASE')],
help='Maximum number of database connection retries '
'during startup. Set to -1 to specify an infinite '
'retry count.'),
cfg.IntOpt('retry_interval',
default=10,
deprecated_opts=[cfg.DeprecatedOpt('sql_retry_interval',
group='DEFAULT'),
cfg.DeprecatedOpt('reconnect_interval',
group='DATABASE')],
help='Interval between retries of opening a SQL connection.'),
cfg.IntOpt('max_overflow',
default=50,
deprecated_opts=[cfg.DeprecatedOpt('sql_max_overflow',
group='DEFAULT'),
cfg.DeprecatedOpt('sqlalchemy_max_overflow',
group='DATABASE')],
help='If set, use this value for max_overflow with '
'SQLAlchemy.'),
cfg.IntOpt('connection_debug',
default=0,
min=0, max=100,
deprecated_opts=[cfg.DeprecatedOpt('sql_connection_debug',
group='DEFAULT')],
help='Verbosity of SQL debugging information: 0=None, '
'100=Everything.'),
cfg.BoolOpt('connection_trace',
default=False,
deprecated_opts=[cfg.DeprecatedOpt('sql_connection_trace',
group='DEFAULT')],
help='Add Python stack traces to SQL as comment strings.'),
cfg.IntOpt('pool_timeout',
deprecated_opts=[cfg.DeprecatedOpt('sqlalchemy_pool_timeout',
group='DATABASE')],
help='If set, use this value for pool_timeout with '
'SQLAlchemy.'),
cfg.BoolOpt('use_db_reconnect',
default=False,
help='Enable the experimental use of database reconnect '
'on connection lost.'),
cfg.IntOpt('db_retry_interval',
default=1,
help='Seconds between retries of a database transaction.'),
cfg.BoolOpt('db_inc_retry_interval',
default=True,
help='If True, increases the interval between retries '
'of a database operation up to db_max_retry_interval.'),
cfg.IntOpt('db_max_retry_interval',
default=10,
help='If db_inc_retry_interval is set, the '
'maximum seconds between retries of a '
'database operation.'),
cfg.IntOpt('db_max_retries',
default=20,
help='Maximum retries in case of connection error or deadlock '
'error before error is '
'raised. Set to -1 to specify an infinite retry '
'count.'),
]
@removals.removed_kwarg("sqlite_db",
"Config option sqlite_db is deprecated for removal,"
"please use option `connection`.")
def set_defaults(conf, connection=None, sqlite_db=None,
max_pool_size=None, max_overflow=None,
pool_timeout=None):
"""Set defaults for configuration variables.
Overrides default options values.
:param conf: Config instance specified to set default options in it. Using
of instances instead of a global config object prevents conflicts between
options declaration.
:type conf: oslo.config.cfg.ConfigOpts instance.
:keyword connection: SQL connection string.
Valid SQLite URL forms are:
* sqlite:///:memory: (or, sqlite://)
* sqlite:///relative/path/to/file.db
* sqlite:////absolute/path/to/file.db
:type connection: str
:keyword sqlite_db: path to SQLite database file.
:type sqlite_db: str
:keyword max_pool_size: maximum connections pool size. The size of the pool
to be maintained, defaults to 5. This is the largest number of connections
that will be kept persistently in the pool. Note that the pool begins with
no connections; once this number of connections is requested, that number
of connections will remain.
:type max_pool_size: int
:default max_pool_size: 5
:keyword max_overflow: The maximum overflow size of the pool. When the
number of checked-out connections reaches the size set in pool_size,
additional connections will be returned up to this limit. When those
additional connections are returned to the pool, they are disconnected and
discarded. It follows then that the total number of simultaneous
connections the pool will allow is pool_size + max_overflow, and the total
number of "sleeping" connections the pool will allow is pool_size.
max_overflow can be set to -1 to indicate no overflow limit; no limit will
be placed on the total number of concurrent connections. Defaults to 10,
will be used if value of the parameter in `None`.
:type max_overflow: int
:default max_overflow: None
:keyword pool_timeout: The number of seconds to wait before giving up on
returning a connection. Defaults to 30, will be used if value of the
parameter is `None`.
:type pool_timeout: int
:default pool_timeout: None
"""
conf.register_opts(database_opts, group='database')
if connection is not None:
conf.set_default('connection', connection, group='database')
if sqlite_db is not None:
conf.set_default('sqlite_db', sqlite_db, group='database')
if max_pool_size is not None:
conf.set_default('max_pool_size', max_pool_size, group='database')
if max_overflow is not None:
conf.set_default('max_overflow', max_overflow, group='database')
if pool_timeout is not None:
conf.set_default('pool_timeout', pool_timeout, group='database')
def list_opts():
"""Returns a list of oslo.config options available in the library.
The returned list includes all oslo.config options which may be registered
at runtime by the library.
Each element of the list is a tuple. The first element is the name of the
group under which the list of elements in the second element will be
registered. A group name of None corresponds to the [DEFAULT] group in
config files.
The purpose of this is to allow tools like the Oslo sample config file
generator to discover the options exposed to users by this library.
:returns: a list of (group_name, opts) tuples
"""
return [('database', database_opts)]

0
oslo_db/sqlalchemy/__init__.py
View File

0
oslo_db/sqlalchemy/compat/__init__.py
View File

44
oslo_db/sqlalchemy/compat/utils.py
View File

@ -1,44 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import re
import sqlalchemy
SQLA_VERSION = tuple(
int(num) if re.match(r'^\d+$', num) else num
for num in sqlalchemy.__version__.split(".")
)
sqla_100 = SQLA_VERSION >= (1, 0, 0)
sqla_097 = SQLA_VERSION >= (0, 9, 7)
sqla_094 = SQLA_VERSION >= (0, 9, 4)
sqla_090 = SQLA_VERSION >= (0, 9, 0)
sqla_08 = SQLA_VERSION >= (0, 8)
def get_postgresql_enums(conn):
"""Return a list of ENUM type names on a Postgresql backend.
For SQLAlchemy 0.9 and lower, makes use of the semi-private
_load_enums() method of the Postgresql dialect. In SQLAlchemy
1.0 this feature is supported using get_enums().
This function may only be called when the given connection
is against the Postgresql backend. It will fail for other
kinds of backends.
"""
if sqla_100:
return [e['name'] for e in sqlalchemy.inspect(conn).get_enums()]
else:
return conn.dialect._load_enums(conn).keys()

1171
oslo_db/sqlalchemy/enginefacade.py
View File

File diff suppressed because it is too large Load Diff

424
oslo_db/sqlalchemy/engines.py
View File

@ -1,424 +0,0 @@
# Copyright 2010 United States Government as represented by the
# Administrator of the National Aeronautics and Space Administration.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""Core SQLAlchemy connectivity routines.
"""
import itertools
import logging
import os
import re
import time
import six
import sqlalchemy
from sqlalchemy import event
from sqlalchemy import exc
from sqlalchemy import pool
from sqlalchemy.sql.expression import select
from oslo_db._i18n import _LW
from oslo_db import exception
from oslo_db.sqlalchemy import exc_filters
from oslo_db.sqlalchemy import utils
LOG = logging.getLogger(__name__)
def _thread_yield(dbapi_con, con_record):
"""Ensure other greenthreads get a chance to be executed.
If we use eventlet.monkey_patch(), eventlet.greenthread.sleep(0) will
execute instead of time.sleep(0).
Force a context switch. With common database backends (eg MySQLdb and
sqlite), there is no implicit yield caused by network I/O since they are
implemented by C libraries that eventlet cannot monkey patch.
"""
time.sleep(0)
def _connect_ping_listener(connection, branch):
"""Ping the server at connection startup.
Ping the server at transaction begin and transparently reconnect
if a disconnect exception occurs.
"""
if branch:
return
# turn off "close with result". This can also be accomplished
# by branching the connection, however just setting the flag is
# more performant and also doesn't get involved with some
# connection-invalidation awkardness that occurs (see
# https://bitbucket.org/zzzeek/sqlalchemy/issue/3215/)
save_should_close_with_result = connection.should_close_with_result
connection.should_close_with_result = False
try:
# run a SELECT 1. use a core select() so that
# any details like that needed by Oracle, DB2 etc. are handled.
connection.scalar(select([1]))
except exception.DBConnectionError:
# catch DBConnectionError, which is raised by the filter
# system.
# disconnect detected. The connection is now
# "invalid", but the pool should be ready to return
# new connections assuming they are good now.
# run the select again to re-validate the Connection.
connection.scalar(select([1]))
finally:
connection.should_close_with_result = save_should_close_with_result
def _setup_logging(connection_debug=0):
"""setup_logging function maps SQL debug level to Python log level.
Connection_debug is a verbosity of SQL debugging information.
0=None(default value),
1=Processed only messages with WARNING level or higher
50=Processed only messages with INFO level or higher
100=Processed only messages with DEBUG level
"""
if connection_debug >= 0:
logger = logging.getLogger('sqlalchemy.engine')
if connection_debug == 100:
logger.setLevel(logging.DEBUG)
elif connection_debug >= 50:
logger.setLevel(logging.INFO)
else:
logger.setLevel(logging.WARNING)
def create_engine(sql_connection, sqlite_fk=False, mysql_sql_mode=None,
idle_timeout=3600,
connection_debug=0, max_pool_size=None, max_overflow=None,
pool_timeout=None, sqlite_synchronous=True,
connection_trace=False, max_retries=10, retry_interval=10,
thread_checkin=True, logging_name=None,
json_serializer=None,
json_deserializer=None):
"""Return a new SQLAlchemy engine."""
url = sqlalchemy.engine.url.make_url(sql_connection)
engine_args = {
"pool_recycle": idle_timeout,
'convert_unicode': True,
'connect_args': {},
'logging_name': logging_name
}
_setup_logging(connection_debug)
_init_connection_args(
url, engine_args,
max_pool_size=max_pool_size,
max_overflow=max_overflow,
pool_timeout=pool_timeout,
json_serializer=json_serializer,
json_deserializer=json_deserializer,
)
engine = sqlalchemy.create_engine(url, **engine_args)
_init_events(
engine,
mysql_sql_mode=mysql_sql_mode,
sqlite_synchronous=sqlite_synchronous,
sqlite_fk=sqlite_fk,
thread_checkin=thread_checkin,
connection_trace=connection_trace
)
# register alternate exception handler
exc_filters.register_engine(engine)
# register engine connect handler
event.listen(engine, "engine_connect", _connect_ping_listener)
# initial connect + test
# NOTE(viktors): the current implementation of _test_connection()
# does nothing, if max_retries == 0, so we can skip it
if max_retries:
test_conn = _test_connection(engine, max_retries, retry_interval)
test_conn.close()
return engine
@utils.dispatch_for_dialect('*', multiple=True)
def _init_connection_args(
url, engine_args,
max_pool_size=None, max_overflow=None, pool_timeout=None, **kw):
pool_class = url.get_dialect().get_pool_class(url)
if issubclass(pool_class, pool.QueuePool):
if max_pool_size is not None:
engine_args['pool_size'] = max_pool_size
if max_overflow is not None:
engine_args['max_overflow'] = max_overflow
if pool_timeout is not None:
engine_args['pool_timeout'] = pool_timeout
@_init_connection_args.dispatch_for("sqlite")
def _init_connection_args(url, engine_args, **kw):
pool_class = url.get_dialect().get_pool_class(url)
# singletonthreadpool is used for :memory: connections;
# replace it with StaticPool.
if issubclass(pool_class, pool.SingletonThreadPool):
engine_args["poolclass"] = pool.StaticPool
engine_args['connect_args']['check_same_thread'] = False
@_init_connection_args.dispatch_for("postgresql")
def _init_connection_args(url, engine_args, **kw):
if 'client_encoding' not in url.query:
# Set encoding using engine_args instead of connect_args since
# it's supported for PostgreSQL 8.*. More details at:
# http://docs.sqlalchemy.org/en/rel_0_9/dialects/postgresql.html
engine_args['client_encoding'] = 'utf8'
engine_args['json_serializer'] = kw.get('json_serializer')
engine_args['json_deserializer'] = kw.get('json_deserializer')
@_init_connection_args.dispatch_for("mysql")
def _init_connection_args(url, engine_args, **kw):
if 'charset' not in url.query:
engine_args['connect_args']['charset'] = 'utf8'
@_init_connection_args.dispatch_for("mysql+mysqlconnector")
def _init_connection_args(url, engine_args, **kw):
# mysqlconnector engine (<1.0) incorrectly defaults to
# raise_on_warnings=True
# https://bitbucket.org/zzzeek/sqlalchemy/issue/2515
if 'raise_on_warnings' not in url.query:
engine_args['connect_args']['raise_on_warnings'] = False
@_init_connection_args.dispatch_for("mysql+mysqldb")
@_init_connection_args.dispatch_for("mysql+oursql")
def _init_connection_args(url, engine_args, **kw):
# Those drivers require use_unicode=0 to avoid performance drop due
# to internal usage of Python unicode objects in the driver
# http://docs.sqlalchemy.org/en/rel_0_9/dialects/mysql.html
if 'use_unicode' not in url.query:
if six.PY3:
engine_args['connect_args']['use_unicode'] = 1
else:
engine_args['connect_args']['use_unicode'] = 0
@utils.dispatch_for_dialect('*', multiple=True)
def _init_events(engine, thread_checkin=True, connection_trace=False, **kw):
"""Set up event listeners for all database backends."""
_add_process_guards(engine)
if connection_trace:
_add_trace_comments(engine)
if thread_checkin:
sqlalchemy.event.listen(engine, 'checkin', _thread_yield)
@_init_events.dispatch_for("mysql")
def _init_events(engine, mysql_sql_mode=None, **kw):
"""Set up event listeners for MySQL."""
if mysql_sql_mode is not None:
@sqlalchemy.event.listens_for(engine, "connect")
def _set_session_sql_mode(dbapi_con, connection_rec):
cursor = dbapi_con.cursor()
cursor.execute("SET SESSION sql_mode = %s", [mysql_sql_mode])
@sqlalchemy.event.listens_for(engine, "first_connect")
def _check_effective_sql_mode(dbapi_con, connection_rec):
if mysql_sql_mode is not None:
_set_session_sql_mode(dbapi_con, connection_rec)
cursor = dbapi_con.cursor()
cursor.execute("SHOW VARIABLES LIKE 'sql_mode'")
realmode = cursor.fetchone()
if realmode is None:
LOG.warning(_LW('Unable to detect effective SQL mode'))
else:
realmode = realmode[1]
LOG.debug('MySQL server mode set to %s', realmode)
if 'TRADITIONAL' not in realmode.upper() and \
'STRICT_ALL_TABLES' not in realmode.upper():
LOG.warning(
_LW(
"MySQL SQL mode is '%s', "
"consider enabling TRADITIONAL or STRICT_ALL_TABLES"),
realmode)
@_init_events.dispatch_for("sqlite")
def _init_events(engine, sqlite_synchronous=True, sqlite_fk=False, **kw):
"""Set up event listeners for SQLite.
This includes several settings made on connections as they are
created, as well as transactional control extensions.
"""
def regexp(expr, item):
reg = re.compile(expr)
return reg.search(six.text_type(item)) is not None
@sqlalchemy.event.listens_for(engine, "connect")
def _sqlite_connect_events(dbapi_con, con_record):
# Add REGEXP functionality on SQLite connections
dbapi_con.create_function('regexp', 2, regexp)
if not sqlite_synchronous:
# Switch sqlite connections to non-synchronous mode
dbapi_con.execute("PRAGMA synchronous = OFF")
# Disable pysqlite's emitting of the BEGIN statement entirely.
# Also stops it from emitting COMMIT before any DDL.
# below, we emit BEGIN ourselves.
# see http://docs.sqlalchemy.org/en/rel_0_9/dialects/\
# sqlite.html#serializable-isolation-savepoints-transactional-ddl
dbapi_con.isolation_level = None
if sqlite_fk:
# Ensures that the foreign key constraints are enforced in SQLite.
dbapi_con.execute('pragma foreign_keys=ON')
@sqlalchemy.event.listens_for(engine, "begin")
def _sqlite_emit_begin(conn):
# emit our own BEGIN, checking for existing
# transactional state
if 'in_transaction' not in conn.info:
conn.execute("BEGIN")
conn.info['in_transaction'] = True
@sqlalchemy.event.listens_for(engine, "rollback")
@sqlalchemy.event.listens_for(engine, "commit")
def _sqlite_end_transaction(conn):
# remove transactional marker
conn.info.pop('in_transaction', None)
def _test_connection(engine, max_retries, retry_interval):
if max_retries == -1:
attempts = itertools.count()
else:
attempts = six.moves.range(max_retries)
# See: http://legacy.python.org/dev/peps/pep-3110/#semantic-changes for
# why we are not using 'de' directly (it can be removed from the local
# scope).
de_ref = None
for attempt in attempts:
try:
return engine.connect()
except exception.DBConnectionError as de:
msg = _LW('SQL connection failed. %s attempts left.')
LOG.warning(msg, max_retries - attempt)
time.sleep(retry_interval)
de_ref = de
else:
if de_ref is not None:
six.reraise(type(de_ref), de_ref)
def _add_process_guards(engine):
"""Add multiprocessing guards.
Forces a connection to be reconnected if it is detected
as having been shared to a sub-process.
"""
@sqlalchemy.event.listens_for(engine, "connect")
def connect(dbapi_connection, connection_record):
connection_record.info['pid'] = os.getpid()
@sqlalchemy.event.listens_for(engine, "checkout")
def checkout(dbapi_connection, connection_record, connection_proxy):
pid = os.getpid()
if connection_record.info['pid'] != pid:
LOG.debug(_LW(
"Parent process %(orig)s forked (%(newproc)s) with an open "
"database connection, "
"which is being discarded and recreated."),
{"newproc": pid, "orig": connection_record.info['pid']})
connection_record.connection = connection_proxy.connection = None
raise exc.DisconnectionError(
"Connection record belongs to pid %s, "
"attempting to check out in pid %s" %
(connection_record.info['pid'], pid)
)
def _add_trace_comments(engine):
"""Add trace comments.
Augment statements with a trace of the immediate calling code
for a given statement.
"""
import os
import sys
import traceback
target_paths = set([
os.path.dirname(sys.modules['oslo_db'].__file__),
os.path.dirname(sys.modules['sqlalchemy'].__file__)
])
try:
skip_paths = set([
os.path.dirname(sys.modules['oslo_db.tests'].__file__),
])
except KeyError:
skip_paths = set()
@sqlalchemy.event.listens_for(engine, "before_cursor_execute", retval=True)
def before_cursor_execute(conn, cursor, statement, parameters, context,
executemany):
# NOTE(zzzeek) - if different steps per DB dialect are desirable
# here, switch out on engine.name for now.
stack = traceback.extract_stack()
our_line = None
for idx, (filename, line, method, function) in enumerate(stack):
for tgt in skip_paths:
if filename.startswith(tgt):
break
else:
for tgt in target_paths:
if filename.startswith(tgt):
our_line = idx
break
if our_line:
break
if our_line:
trace = "; ".join(
"File: %s (%s) %s" % (
line[0], line[1], line[2]
)
# include three lines of context.
for line in stack[our_line - 3:our_line]
)
statement = "%s -- %s" % (statement, trace)
return statement, parameters

477
oslo_db/sqlalchemy/exc_filters.py
View File

@ -1,477 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""Define exception redefinitions for SQLAlchemy DBAPI exceptions."""
import collections
import logging
import re
import sys
from sqlalchemy import event
from sqlalchemy import exc as sqla_exc
from oslo_db._i18n import _LE
from oslo_db import exception
LOG = logging.getLogger(__name__)
_registry = collections.defaultdict(
lambda: collections.defaultdict(
list
)
)
def filters(dbname, exception_type, regex):
"""Mark a function as receiving a filtered exception.
:param dbname: string database name, e.g. 'mysql'
:param exception_type: a SQLAlchemy database exception class, which
extends from :class:`sqlalchemy.exc.DBAPIError`.
:param regex: a string, or a tuple of strings, that will be processed
as matching regular expressions.
"""
def _receive(fn):
_registry[dbname][exception_type].extend(
(fn, re.compile(reg))
for reg in
((regex,) if not isinstance(regex, tuple) else regex)
)
return fn
return _receive
# NOTE(zzzeek) - for Postgresql, catch both OperationalError, as the
# actual error is
# psycopg2.extensions.TransactionRollbackError(OperationalError),
# as well as sqlalchemy.exc.DBAPIError, as SQLAlchemy will reraise it
# as this until issue #3075 is fixed.
@filters("mysql", sqla_exc.OperationalError, r"^.*\b1213\b.*Deadlock found.*")
@filters("mysql", sqla_exc.DatabaseError,
r"^.*\b1205\b.*Lock wait timeout exceeded.*")
@filters("mysql", sqla_exc.InternalError, r"^.*\b1213\b.*Deadlock found.*")
@filters("postgresql", sqla_exc.OperationalError, r"^.*deadlock detected.*")
@filters("postgresql", sqla_exc.DBAPIError, r"^.*deadlock detected.*")
@filters("ibm_db_sa", sqla_exc.DBAPIError, r"^.*SQL0911N.*")
def _deadlock_error(operational_error, match, engine_name, is_disconnect):
"""Filter for MySQL or Postgresql deadlock error.
NOTE(comstud): In current versions of DB backends, Deadlock violation
messages follow the structure:
mysql+mysqldb:
(OperationalError) (1213, 'Deadlock found when trying to get lock; try '
'restarting transaction') <query_str> <query_args>
mysql+mysqlconnector:
(InternalError) 1213 (40001): Deadlock found when trying to get lock; try
restarting transaction
postgresql:
(TransactionRollbackError) deadlock detected <deadlock_details>
ibm_db_sa:
SQL0911N The current transaction has been rolled back because of a
deadlock or timeout <deadlock details>
"""
raise exception.DBDeadlock(operational_error)
@filters("mysql", sqla_exc.IntegrityError,
r"^.*\b1062\b.*Duplicate entry '(?P<value>.*)'"
r" for key '(?P<columns>[^']+)'.*$")
# NOTE(jd) For binary types
@filters("mysql", sqla_exc.IntegrityError,
r"^.*\b1062\b.*Duplicate entry \\'(?P<value>.*)\\'"
r" for key \\'(?P<columns>.+)\\'.*$")
# NOTE(pkholkin): the first regex is suitable only for PostgreSQL 9.x versions
# the second regex is suitable for PostgreSQL 8.x versions
@filters("postgresql", sqla_exc.IntegrityError,
(r'^.*duplicate\s+key.*"(?P<columns>[^"]+)"\s*\n.*'
r'Key\s+\((?P<key>.*)\)=\((?P<value>.*)\)\s+already\s+exists.*$',
r"^.*duplicate\s+key.*\"(?P<columns>[^\"]+)\"\s*\n.*$"))
def _default_dupe_key_error(integrity_error, match, engine_name,
is_disconnect):
"""Filter for MySQL or Postgresql duplicate key error.
note(boris-42): In current versions of DB backends unique constraint
violation messages follow the structure:
postgres:
1 column - (IntegrityError) duplicate key value violates unique
constraint "users_c1_key"
N columns - (IntegrityError) duplicate key value violates unique
constraint "name_of_our_constraint"
mysql+mysqldb:
1 column - (IntegrityError) (1062, "Duplicate entry 'value_of_c1' for key
'c1'")
N columns - (IntegrityError) (1062, "Duplicate entry 'values joined
with -' for key 'name_of_our_constraint'")
mysql+mysqlconnector:
1 column - (IntegrityError) 1062 (23000): Duplicate entry 'value_of_c1' for
key 'c1'
N columns - (IntegrityError) 1062 (23000): Duplicate entry 'values
joined with -' for key 'name_of_our_constraint'
"""
columns = match.group('columns')
# note(vsergeyev): UniqueConstraint name convention: "uniq_t0c10c2"
# where `t` it is table name and columns `c1`, `c2`
# are in UniqueConstraint.
uniqbase = "uniq_"
if not columns.startswith(uniqbase):
if engine_name == "postgresql":
columns = [columns[columns.index("_") + 1:columns.rindex("_")]]
else:
columns = [columns]
else:
columns = columns[len(uniqbase):].split("0")[1:]
value = match.groupdict().get('value')
raise exception.DBDuplicateEntry(columns, integrity_error, value)
@filters("sqlite", sqla_exc.IntegrityError,
(r"^.*columns?(?P<columns>[^)]+)(is|are)\s+not\s+unique$",
r"^.*UNIQUE\s+constraint\s+failed:\s+(?P<columns>.+)$",
r"^.*PRIMARY\s+KEY\s+must\s+be\s+unique.*$"))
def _sqlite_dupe_key_error(integrity_error, match, engine_name, is_disconnect):
"""Filter for SQLite duplicate key error.
note(boris-42): In current versions of DB backends unique constraint
violation messages follow the structure:
sqlite:
1 column - (IntegrityError) column c1 is not unique
N columns - (IntegrityError) column c1, c2, ..., N are not unique
sqlite since 3.7.16:
1 column - (IntegrityError) UNIQUE constraint failed: tbl.k1
N columns - (IntegrityError) UNIQUE constraint failed: tbl.k1, tbl.k2
sqlite since 3.8.2:
(IntegrityError) PRIMARY KEY must be unique
"""
columns = []
# NOTE(ochuprykov): We can get here by last filter in which there are no
# groups. Trying to access the substring that matched by
# the group will lead to IndexError. In this case just
# pass empty list to exception.DBDuplicateEntry
try:
columns = match.group('columns')
columns = [c.split('.')[-1] for c in columns.strip().split(", ")]
except IndexError:
pass
raise exception.DBDuplicateEntry(columns, integrity_error)
@filters("sqlite", sqla_exc.IntegrityError,
r"(?i).*foreign key constraint failed")
@filters("postgresql", sqla_exc.IntegrityError,
r".*on table \"(?P<table>[^\"]+)\" violates "
"foreign key constraint \"(?P<constraint>[^\"]+)\".*\n"
"DETAIL: Key \((?P<key>.+)\)=\(.+\) "
"is (not present in|still referenced from) table "
"\"(?P<key_table>[^\"]+)\".")
@filters("mysql", sqla_exc.IntegrityError,
r".* u?'Cannot (add|delete) or update a (child|parent) row: "
'a foreign key constraint fails \([`"].+[`"]\.[`"](?P<table>.+)[`"], '
'CONSTRAINT [`"](?P<constraint>.+)[`"] FOREIGN KEY '
'\([`"](?P<key>.+)[`"]\) REFERENCES [`"](?P<key_table>.+)[`"] ')
def _foreign_key_error(integrity_error, match, engine_name, is_disconnect):
"""Filter for foreign key errors."""
try:
table = match.group("table")
except IndexError:
table = None
try:
constraint = match.group("constraint")
except IndexError:
constraint = None
try:
key = match.group("key")
except IndexError:
key = None
try:
key_table = match.group("key_table")
except IndexError:
key_table = None
raise exception.DBReferenceError(table, constraint, key, key_table,
integrity_error)
@filters("postgresql", sqla_exc.IntegrityError,
r".*new row for relation \"(?P<table>.+)\" "
"violates check constraint "
"\"(?P<check_name>.+)\"")
def _check_constraint_error(
integrity_error, match, engine_name, is_disconnect):
"""Filter for check constraint errors."""
try:
table = match.group("table")
except IndexError:
table = None
try:
check_name = match.group("check_name")
except IndexError:
check_name = None
raise exception.DBConstraintError(table, check_name, integrity_error)
@filters("postgresql", sqla_exc.ProgrammingError,
r".* constraint \"(?P<constraint>.+)\" "
"of relation "
"\"(?P<relation>.+)\" does not exist")
@filters("mysql", sqla_exc.InternalError,
r".*1091,.*Can't DROP '(?P<constraint>.+)'; "
"check that column/key exists")
@filters("mysql", sqla_exc.InternalError,
r".*1025,.*Error on rename of '.+/(?P<relation>.+)' to ")
def _check_constraint_non_existing(
programming_error, match, engine_name, is_disconnect):
"""Filter for constraint non existing errors."""
try:
relation = match.group("relation")
except IndexError:
relation = None
try:
constraint = match.group("constraint")
except IndexError:
constraint = None
raise exception.DBNonExistentConstraint(relation,
constraint,
programming_error)
@filters("sqlite", sqla_exc.OperationalError,
r".* no such table: (?P<table>.+)")
@filters("mysql", sqla_exc.InternalError,
r".*1051,.*\"Unknown table '(.+\.)?(?P<table>.+)'\"")
@filters("postgresql", sqla_exc.ProgrammingError,
r".* table \"(?P<table>.+)\" does not exist")
def _check_table_non_existing(
programming_error, match, engine_name, is_disconnect):
"""Filter for table non existing errors."""
raise exception.DBNonExistentTable(match.group("table"), programming_error)
@filters("ibm_db_sa", sqla_exc.IntegrityError, r"^.*SQL0803N.*$")
def _db2_dupe_key_error(integrity_error, match, engine_name, is_disconnect):
"""Filter for DB2 duplicate key errors.
N columns - (IntegrityError) SQL0803N One or more values in the INSERT
statement, UPDATE statement, or foreign key update caused by a
DELETE statement are not valid because the primary key, unique
constraint or unique index identified by "2" constrains table
"NOVA.KEY_PAIRS" from having duplicate values for the index
key.
"""
# NOTE(mriedem): The ibm_db_sa integrity error message doesn't provide the
# columns so we have to omit that from the DBDuplicateEntry error.
raise exception.DBDuplicateEntry([], integrity_error)
@filters("mysql", sqla_exc.DBAPIError, r".*\b1146\b")
def _raise_mysql_table_doesnt_exist_asis(
error, match, engine_name, is_disconnect):
"""Raise MySQL error 1146 as is.
Raise MySQL error 1146 as is, so that it does not conflict with
the MySQL dialect's checking a table not existing.
"""
raise error
@filters("mysql", sqla_exc.OperationalError,
r".*(1292|1366).*Incorrect \w+ value.*")
@filters("mysql", sqla_exc.DataError,
r".*1265.*Data truncated for column.*")
@filters("mysql", sqla_exc.DataError,
r".*1264.*Out of range value for column.*")
@filters("mysql", sqla_exc.InternalError,
r"^.*1366.*Incorrect string value:*")
@filters("sqlite", sqla_exc.ProgrammingError,
r"(?i).*You must not use 8-bit bytestrings*")
def _raise_data_error(error, match, engine_name, is_disconnect):
"""Raise DBDataError exception for different data errors."""
raise exception.DBDataError(error)
@filters("*", sqla_exc.OperationalError, r".*")
def _raise_operational_errors_directly_filter(operational_error,
match, engine_name,
is_disconnect):
"""Filter for all remaining OperationalError classes and apply.
Filter for all remaining OperationalError classes and apply
special rules.
"""
if is_disconnect:
# operational errors that represent disconnect
# should be wrapped
raise exception.DBConnectionError(operational_error)
else:
# NOTE(comstud): A lot of code is checking for OperationalError
# so let's not wrap it for now.
raise operational_error
@filters("mysql", sqla_exc.OperationalError, r".*\(.*(?:2002|2003|2006|2013|1047)") # noqa
@filters("postgresql", sqla_exc.OperationalError, r".*could not connect to server") # noqa
@filters("ibm_db_sa", sqla_exc.OperationalError, r".*(?:30081)")
def _is_db_connection_error(operational_error, match, engine_name,
is_disconnect):
"""Detect the exception as indicating a recoverable error on connect."""
raise exception.DBConnectionError(operational_error)
@filters("*", sqla_exc.DBAPIError, r".*")
def _raise_for_remaining_DBAPIError(error, match, engine_name, is_disconnect):
"""Filter for remaining DBAPIErrors.
Filter for remaining DBAPIErrors and wrap if they represent
a disconnect error.
"""
if is_disconnect:
raise exception.DBConnectionError(error)
else:
LOG.exception(
_LE('DBAPIError exception wrapped from %s') % error)
raise exception.DBError(error)
@filters('*', UnicodeEncodeError, r".*")
def _raise_for_unicode_encode(error, match, engine_name, is_disconnect):
raise exception.DBInvalidUnicodeParameter()
@filters("*", Exception, r".*")
def _raise_for_all_others(error, match, engine_name, is_disconnect):
LOG.exception(_LE('DB exception wrapped.'))
raise exception.DBError(error)
ROLLBACK_CAUSE_KEY = 'oslo.db.sp_rollback_cause'
def handler(context):
"""Iterate through available filters and invoke those which match.
The first one which raises wins. The order in which the filters
are attempted is sorted by specificity - dialect name or "*",
exception class per method resolution order (``__mro__``).
Method resolution order is used so that filter rules indicating a
more specific exception class are attempted first.
"""
def _dialect_registries(engine):
if engine.dialect.name in _registry:
yield _registry[engine.dialect.name]
if '*' in _registry:
yield _registry['*']
for per_dialect in _dialect_registries(context.engine):
for exc in (
context.sqlalchemy_exception,
context.original_exception):
for super_ in exc.__class__.__mro__:
if super_ in per_dialect:
regexp_reg = per_dialect[super_]
for fn, regexp in regexp_reg:
match = regexp.match(exc.args[0])
if match:
try:
fn(
exc,
match,
context.engine.dialect.name,
context.is_disconnect)
except exception.DBError as dbe:
if (
context.connection is not None and
not context.connection.closed and
not context.connection.invalidated and
ROLLBACK_CAUSE_KEY
in context.connection.info
):
dbe.cause = \
context.connection.info.pop(
ROLLBACK_CAUSE_KEY)
if isinstance(
dbe, exception.DBConnectionError):
context.is_disconnect = True
raise
def register_engine(engine):
event.listen(engine, "handle_error", handler)
@event.listens_for(engine, "rollback_savepoint")
def rollback_savepoint(conn, name, context):
exc_info = sys.exc_info()
if exc_info[1]:
conn.info[ROLLBACK_CAUSE_KEY] = exc_info[1]
# NOTE(zzzeek) this eliminates a reference cycle between tracebacks
# that would occur in Python 3 only, which has been shown to occur if
# this function were in fact part of the traceback. That's not the
# case here however this is left as a defensive measure.
del exc_info
# try to clear the "cause" ASAP outside of savepoints,
# by grabbing the end of transaction events...
@event.listens_for(engine, "rollback")
@event.listens_for(engine, "commit")
def pop_exc_tx(conn):
conn.info.pop(ROLLBACK_CAUSE_KEY, None)
# .. as well as connection pool checkin (just in case).
# the .info dictionary lasts as long as the DBAPI connection itself
# and is cleared out when the connection is recycled or closed
# due to invalidate etc.
@event.listens_for(engine, "checkin")
def pop_exc_checkin(dbapi_conn, connection_record):
connection_record.info.pop(ROLLBACK_CAUSE_KEY, None)
def handle_connect_error(engine):
"""Connect to the engine, including handle_error handlers.
The compat library now builds this into the engine.connect()
system as per SQLAlchemy 1.0's behavior.
"""
return engine.connect()

169
oslo_db/sqlalchemy/migration.py
View File

@ -1,169 +0,0 @@
# coding=utf-8
# Copyright (c) 2013 OpenStack Foundation
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
# Base on code in migrate/changeset/databases/sqlite.py which is under
# the following license:
#
# The MIT License
#
# Copyright (c) 2009 Evan Rosson, Jan Dittberner, Domen Kožar
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
import os
from migrate import exceptions as versioning_exceptions
from migrate.versioning import api as versioning_api
from migrate.versioning.repository import Repository
import sqlalchemy
from oslo_db._i18n import _
from oslo_db import exception
def db_sync(engine, abs_path, version=None, init_version=0, sanity_check=True):
"""Upgrade or downgrade a database.
Function runs the upgrade() or downgrade() functions in change scripts.
:param engine: SQLAlchemy engine instance for a given database
:param abs_path: Absolute path to migrate repository.
:param version: Database will upgrade/downgrade until this version.
If None - database will update to the latest
available version.
:param init_version: Initial database version
:param sanity_check: Require schema sanity checking for all tables
"""
if version is not None:
try:
version = int(version)
except ValueError:
raise exception.DBMigrationError(_("version should be an integer"))
current_version = db_version(engine, abs_path, init_version)
repository = _find_migrate_repo(abs_path)
if sanity_check:
_db_schema_sanity_check(engine)
if version is None or version > current_version:
migration = versioning_api.upgrade(engine, repository, version)
else:
migration = versioning_api.downgrade(engine, repository,
version)
if sanity_check:
_db_schema_sanity_check(engine)
return migration
def _db_schema_sanity_check(engine):
"""Ensure all database tables were created with required parameters.
:param engine: SQLAlchemy engine instance for a given database
"""
if engine.name == 'mysql':
onlyutf8_sql = ('SELECT TABLE_NAME,TABLE_COLLATION '
'from information_schema.TABLES '
'where TABLE_SCHEMA=%s and '
'TABLE_COLLATION NOT LIKE \'%%utf8%%\'')
# NOTE(morganfainberg): exclude the sqlalchemy-migrate and alembic
# versioning tables from the tables we need to verify utf8 status on.
# Non-standard table names are not supported.
EXCLUDED_TABLES = ['migrate_version', 'alembic_version']
table_names = [res[0] for res in
engine.execute(onlyutf8_sql, engine.url.database) if
res[0].lower() not in EXCLUDED_TABLES]
if len(table_names) > 0:
raise ValueError(_('Tables "%s" have non utf8 collation, '
'please make sure all tables are CHARSET=utf8'
) % ','.join(table_names))
def db_version(engine, abs_path, init_version):
"""Show the current version of the repository.
:param engine: SQLAlchemy engine instance for a given database
:param abs_path: Absolute path to migrate repository
:param init_version: Initial database version
"""
repository = _find_migrate_repo(abs_path)
try:
return versioning_api.db_version(engine, repository)
except versioning_exceptions.DatabaseNotControlledError:
meta = sqlalchemy.MetaData()
meta.reflect(bind=engine)
tables = meta.tables
if len(tables) == 0 or 'alembic_version' in tables:
db_version_control(engine, abs_path, version=init_version)
return versioning_api.db_version(engine, repository)
else:
raise exception.DBMigrationError(
_("The database is not under version control, but has "
"tables. Please stamp the current version of the schema "
"manually."))
def db_version_control(engine, abs_path, version=None):
"""Mark a database as under this repository's version control.
Once a database is under version control, schema changes should
only be done via change scripts in this repository.
:param engine: SQLAlchemy engine instance for a given database
:param abs_path: Absolute path to migrate repository
:param version: Initial database version
"""
repository = _find_migrate_repo(abs_path)
try:
versioning_api.version_control(engine, repository, version)
except versioning_exceptions.InvalidVersionError as ex:
raise exception.DBMigrationError("Invalid version : %s" % ex)
except versioning_exceptions.DatabaseAlreadyControlledError:
raise exception.DBMigrationError("Database is already controlled.")
return version
def _find_migrate_repo(abs_path):
"""Get the project's change script repository
:param abs_path: Absolute path to migrate repository
"""
if not os.path.exists(abs_path):
raise exception.DBMigrationError("Path %s not found" % abs_path)
return Repository(abs_path)

9
oslo_db/sqlalchemy/migration_cli/README.rst
View File

@ -1,9 +0,0 @@
This module could be used either for:
1. Smooth transition from migrate tool to alembic
2. As standalone alembic tool
Core points:
1. Upgrade/downgrade database with usage of alembic/migrate migrations
or both
2. Compatibility with oslo.config
3. The way to autogenerate new revisions or stamps

0
oslo_db/sqlalchemy/migration_cli/__init__.py
View File

112
oslo_db/sqlalchemy/migration_cli/ext_alembic.py
View File

@ -1,112 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import os
import alembic
from alembic import config as alembic_config
import alembic.migration as alembic_migration
from alembic import script as alembic_script
from oslo_db.sqlalchemy.migration_cli import ext_base
class AlembicExtension(ext_base.MigrationExtensionBase):
"""Extension to provide alembic features.
:param engine: SQLAlchemy engine instance for a given database
:type engine: sqlalchemy.engine.Engine
:param migration_config: Stores specific configuration for migrations
:type migration_config: dict
"""
order = 2
@property
def enabled(self):
return os.path.exists(self.alembic_ini_path)
def __init__(self, engine, migration_config):
self.alembic_ini_path = migration_config.get('alembic_ini_path', '')
self.config = alembic_config.Config(self.alembic_ini_path)
# TODO(viktors): Remove this, when we will use Alembic 0.7.5 or
# higher, because the ``attributes`` dictionary was
# added to Alembic in version 0.7.5.
if not hasattr(self.config, 'attributes'):
self.config.attributes = {}
# option should be used if script is not in default directory
repo_path = migration_config.get('alembic_repo_path')
if repo_path:
self.config.set_main_option('script_location', repo_path)
self.engine = engine
def upgrade(self, version):
with self.engine.begin() as connection:
self.config.attributes['connection'] = connection
return alembic.command.upgrade(self.config, version or 'head')
def downgrade(self, version):
if isinstance(version, int) or version is None or version.isdigit():
version = 'base'
with self.engine.begin() as connection:
self.config.attributes['connection'] = connection
return alembic.command.downgrade(self.config, version)
def version(self):
with self.engine.connect() as conn:
context = alembic_migration.MigrationContext.configure(conn)
return context.get_current_revision()
def revision(self, message='', autogenerate=False):
"""Creates template for migration.
:param message: Text that will be used for migration title
:type message: string
:param autogenerate: If True - generates diff based on current database
state
:type autogenerate: bool
"""
with self.engine.begin() as connection:
self.config.attributes['connection'] = connection
return alembic.command.revision(self.config, message=message,
autogenerate=autogenerate)
def stamp(self, revision):
"""Stamps database with provided revision.
:param revision: Should match one from repository or head - to stamp
database with most recent revision
:type revision: string
"""
with self.engine.begin() as connection:
self.config.attributes['connection'] = connection
return alembic.command.stamp(self.config, revision=revision)
def has_revision(self, rev_id):
if rev_id in ['base', 'head']:
return True
# Although alembic supports relative upgrades and downgrades,
# get_revision always returns False for relative revisions.
# Since only alembic supports relative revisions, assume the
# revision belongs to this plugin.
if rev_id: # rev_id can be None, so the check is required
if '-' in rev_id or '+' in rev_id:
return True
script = alembic_script.ScriptDirectory(
self.config.get_main_option('script_location'))
try:
script.get_revision(rev_id)
return True
except alembic.util.CommandError:
return False

88
oslo_db/sqlalchemy/migration_cli/ext_base.py
View File

@ -1,88 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import abc
import six
@six.add_metaclass(abc.ABCMeta)
class MigrationExtensionBase(object):
# used to sort migration in logical order
order = 0
@property
def enabled(self):
"""Used for availability verification of a plugin.
:rtype: bool
"""
return False
@abc.abstractmethod
def upgrade(self, version):
"""Used for upgrading database.
:param version: Desired database version
:type version: string
"""
@abc.abstractmethod
def downgrade(self, version):
"""Used for downgrading database.
:param version: Desired database version
:type version: string
"""
@abc.abstractmethod
def version(self):
"""Current database version.
:returns: Databse version
:rtype: string
"""
def revision(self, *args, **kwargs):
"""Used to generate migration script.
In migration engines that support this feature, it should generate
new migration script.
Accept arbitrary set of arguments.
"""
raise NotImplementedError()
def stamp(self, *args, **kwargs):
"""Stamps database based on plugin features.
Accept arbitrary set of arguments.
"""
raise NotImplementedError()
def has_revision(self, rev_id):
"""Checks whether the repo contains a revision
:param rev_id: Revision to check
:returns: Whether the revision is in the repo
:rtype: bool
"""
raise NotImplementedError()
def __cmp__(self, other):
"""Used for definition of plugin order.
:param other: MigrationExtensionBase instance
:rtype: bool
"""
return self.order > other.order

80
oslo_db/sqlalchemy/migration_cli/ext_migrate.py
View File

@ -1,80 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import logging
import os
from migrate.versioning import version as migrate_version
from oslo_db._i18n import _LE
from oslo_db.sqlalchemy import migration
from oslo_db.sqlalchemy.migration_cli import ext_base
LOG = logging.getLogger(__name__)
class MigrateExtension(ext_base.MigrationExtensionBase):
"""Extension to provide sqlalchemy-migrate features.
:param migration_config: Stores specific configuration for migrations
:type migration_config: dict
"""
order = 1
def __init__(self, engine, migration_config):
self.engine = engine
self.repository = migration_config.get('migration_repo_path', '')
self.init_version = migration_config.get('init_version', 0)
@property
def enabled(self):
return os.path.exists(self.repository)
def upgrade(self, version):
version = None if version == 'head' else version
return migration.db_sync(
self.engine, self.repository, version,
init_version=self.init_version)
def downgrade(self, version):
try:
# version for migrate should be valid int - else skip
if version in ('base', None):
version = self.init_version
version = int(version)
return migration.db_sync(
self.engine, self.repository, version,
init_version=self.init_version)
except ValueError:
LOG.error(
_LE('Migration number for migrate plugin must be valid '
'integer or empty, if you want to downgrade '
'to initial state')
)
raise
def version(self):
return migration.db_version(
self.engine, self.repository, init_version=self.init_version)
def has_revision(self, rev_id):
collection = migrate_version.Collection(self.repository)
try:
collection.version(rev_id)
return True
except (KeyError, ValueError):
# NOTE(breton): migrate raises KeyError if an int is passed but not
# found in the list of revisions and ValueError if non-int is
# passed. Both mean there is no requested revision.
return False

107
oslo_db/sqlalchemy/migration_cli/manager.py
View File

@ -1,107 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import sqlalchemy
from stevedore import enabled
from oslo_db import exception
MIGRATION_NAMESPACE = 'oslo.db.migration'
def check_plugin_enabled(ext):
"""Used for EnabledExtensionManager."""
return ext.obj.enabled
class MigrationManager(object):
def __init__(self, migration_config, engine=None):
if engine is None:
if migration_config.get('db_url'):
engine = sqlalchemy.create_engine(
migration_config['db_url'],
poolclass=sqlalchemy.pool.NullPool,
)
else:
raise ValueError('Either database url or engine'
' must be provided.')
self._manager = enabled.EnabledExtensionManager(
MIGRATION_NAMESPACE,
check_plugin_enabled,
invoke_args=(engine, migration_config),
invoke_on_load=True
)
if not self._plugins:
raise ValueError('There must be at least one plugin active.')
@property
def _plugins(self):
return sorted(ext.obj for ext in self._manager.extensions)
def upgrade(self, revision):
"""Upgrade database with all available backends."""
# a revision exists only in a single plugin. Until we reached it, we
# should upgrade to the plugins' heads.
# revision=None is a special case meaning latest revision.
rev_in_plugins = [p.has_revision(revision) for p in self._plugins]
if not any(rev_in_plugins) and revision is not None:
raise exception.DBMigrationError('Revision does not exist')
results = []
for plugin, has_revision in zip(self._plugins, rev_in_plugins):
if not has_revision or revision is None:
results.append(plugin.upgrade(None))
else:
results.append(plugin.upgrade(revision))
break
return results
def downgrade(self, revision):
"""Downgrade database with available backends."""
# a revision exists only in a single plugin. Until we reached it, we
# should upgrade to the plugins' first revision.
# revision=None is a special case meaning initial revision.
rev_in_plugins = [p.has_revision(revision) for p in self._plugins]
if not any(rev_in_plugins) and revision is not None:
raise exception.DBMigrationError('Revision does not exist')
# downgrading should be performed in reversed order
results = []
for plugin, has_revision in zip(reversed(self._plugins),
reversed(rev_in_plugins)):
if not has_revision or revision is None:
results.append(plugin.downgrade(None))
else:
results.append(plugin.downgrade(revision))
break
return results
def version(self):
"""Return last version of db."""
last = None
for plugin in self._plugins:
version = plugin.version()
if version is not None:
last = version
return last
def revision(self, message, autogenerate):
"""Generate template or autogenerated revision."""
# revision should be done only by last plugin
return self._plugins[-1].revision(message, autogenerate)
def stamp(self, revision):
"""Create stamp for a given revision."""
return self._plugins[-1].stamp(revision)

148
oslo_db/sqlalchemy/models.py
View File

@ -1,148 +0,0 @@
# Copyright (c) 2011 X.commerce, a business unit of eBay Inc.
# Copyright 2010 United States Government as represented by the
# Administrator of the National Aeronautics and Space Administration.
# Copyright 2011 Piston Cloud Computing, Inc.
# Copyright 2012 Cloudscaling Group, Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""
SQLAlchemy models.
"""
import six
from oslo_utils import timeutils
from sqlalchemy import Column, Integer
from sqlalchemy import DateTime
from sqlalchemy.orm import object_mapper
class ModelBase(six.Iterator):
"""Base class for models."""
__table_initialized__ = False
def save(self, session):
"""Save this object."""
# NOTE(boris-42): This part of code should be look like:
# session.add(self)
# session.flush()
# But there is a bug in sqlalchemy and eventlet that
# raises NoneType exception if there is no running
# transaction and rollback is called. As long as
# sqlalchemy has this bug we have to create transaction
# explicitly.
with session.begin(subtransactions=True):
session.add(self)
session.flush()
def __setitem__(self, key, value):
setattr(self, key, value)
def __getitem__(self, key):
return getattr(self, key)
def __contains__(self, key):
# Don't use hasattr() because hasattr() catches any exception, not only
# AttributeError. We want to passthrough SQLAlchemy exceptions
# (ex: sqlalchemy.orm.exc.DetachedInstanceError).
try:
getattr(self, key)
except AttributeError:
return False
else:
return True
def get(self, key, default=None):
return getattr(self, key, default)
@property
def _extra_keys(self):
"""Specifies custom fields
Subclasses can override this property to return a list
of custom fields that should be included in their dict
representation.
For reference check tests/db/sqlalchemy/test_models.py
"""
return []
def __iter__(self):
columns = list(dict(object_mapper(self).columns).keys())
# NOTE(russellb): Allow models to specify other keys that can be looked
# up, beyond the actual db columns. An example would be the 'name'
# property for an Instance.
columns.extend(self._extra_keys)
return ModelIterator(self, iter(columns))
def update(self, values):
"""Make the model object behave like a dict."""
for k, v in six.iteritems(values):
setattr(self, k, v)
def _as_dict(self):
"""Make the model object behave like a dict.
Includes attributes from joins.
"""
local = dict((key, value) for key, value in self)
joined = dict([(k, v) for k, v in six.iteritems(self.__dict__)
if not k[0] == '_'])
local.update(joined)
return local
def iteritems(self):
"""Make the model object behave like a dict."""
return six.iteritems(self._as_dict())
def items(self):
"""Make the model object behave like a dict."""
return self._as_dict().items()
def keys(self):
"""Make the model object behave like a dict."""
return [key for key, value in self.iteritems()]
class ModelIterator(six.Iterator):
def __init__(self, model, columns):
self.model = model
self.i = columns
def __iter__(self):
return self
# In Python 3, __next__() has replaced next().
def __next__(self):
n = six.advance_iterator(self.i)
return n, getattr(self.model, n)
class TimestampMixin(object):
created_at = Column(DateTime, default=lambda: timeutils.utcnow())
updated_at = Column(DateTime, onupdate=lambda: timeutils.utcnow())
class SoftDeleteMixin(object):
deleted_at = Column(DateTime)
deleted = Column(Integer, default=0)
def soft_delete(self, session):
"""Mark this object as deleted."""
self.deleted = self.id
self.deleted_at = timeutils.utcnow()
self.save(session=session)

66
oslo_db/sqlalchemy/orm.py
View File

@ -1,66 +0,0 @@
# Copyright 2010 United States Government as represented by the
# Administrator of the National Aeronautics and Space Administration.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""SQLAlchemy ORM connectivity and query structures.
"""
from oslo_utils import timeutils
import sqlalchemy.orm
from sqlalchemy.sql.expression import literal_column
from oslo_db.sqlalchemy import update_match
class Query(sqlalchemy.orm.query.Query):
"""Subclass of sqlalchemy.query with soft_delete() method."""
def soft_delete(self, synchronize_session='evaluate'):
return self.update({'deleted': literal_column('id'),
'updated_at': literal_column('updated_at'),
'deleted_at': timeutils.utcnow()},
synchronize_session=synchronize_session)
def update_returning_pk(self, values, surrogate_key):
"""Perform an UPDATE, returning the primary key of the matched row.
This is a method-version of
oslo_db.sqlalchemy.update_match.update_returning_pk(); see that
function for usage details.
"""
return update_match.update_returning_pk(self, values, surrogate_key)
def update_on_match(self, specimen, surrogate_key, values, **kw):
"""Emit an UPDATE statement matching the given specimen.
This is a method-version of
oslo_db.sqlalchemy.update_match.update_on_match(); see that function
for usage details.
"""
return update_match.update_on_match(
self, specimen, surrogate_key, values, **kw)
class Session(sqlalchemy.orm.session.Session):
"""oslo.db-specific Session subclass."""
def get_maker(engine, autocommit=True, expire_on_commit=False):
"""Return a SQLAlchemy sessionmaker using the given engine."""
return sqlalchemy.orm.sessionmaker(bind=engine,
class_=Session,
autocommit=autocommit,
expire_on_commit=expire_on_commit,
query_cls=Query)

582
oslo_db/sqlalchemy/provision.py
View File

@ -1,582 +0,0 @@
# Copyright 2013 Mirantis.inc
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""Provision test environment for specific DB backends"""
import abc
import logging
import os
import random
import re
import string
import six
from six import moves
import sqlalchemy
from sqlalchemy.engine import url as sa_url
from sqlalchemy import schema
import testresources
from oslo_db._i18n import _LI
from oslo_db import exception
from oslo_db.sqlalchemy import session
from oslo_db.sqlalchemy import utils
LOG = logging.getLogger(__name__)
class ProvisionedDatabase(object):
pass
class BackendResource(testresources.TestResourceManager):
def __init__(self, database_type):
super(BackendResource, self).__init__()
self.database_type = database_type
self.backend = Backend.backend_for_database_type(self.database_type)
def make(self, dependency_resources):
return self.backend
def isDirty(self):
return False
class DatabaseResource(testresources.TestResourceManager):
def __init__(self, database_type):
super(DatabaseResource, self).__init__()
self.database_type = database_type
self.resources = [
('backend', BackendResource(database_type))
]
def make(self, dependency_resources):
dependency_resources['db_token'] = db_token = _random_ident()
backend = dependency_resources['backend']
LOG.info(
"CREATE BACKEND %s TOKEN %s", backend.engine.url, db_token)
backend.create_named_database(db_token, conditional=True)
dependency_resources['engine'] = \
backend.provisioned_engine(db_token)
return ProvisionedDatabase()
def clean(self, resource):
resource.engine.dispose()
LOG.info(
"DROP BACKEND %s TOKEN %s",
resource.backend.engine, resource.db_token)
resource.backend.drop_named_database(resource.db_token)
def isDirty(self):
return False
class TransactionResource(testresources.TestResourceManager):
def __init__(self, database_resource, schema_resource):
super(TransactionResource, self).__init__()
self.resources = [
('database', database_resource),
('schema', schema_resource)
]
def clean(self, resource):
resource._dispose()
def make(self, dependency_resources):
conn = dependency_resources['database'].engine.connect()
return utils.NonCommittingEngine(conn)
def isDirty(self):
return True
class Schema(object):
pass
class SchemaResource(testresources.TestResourceManager):
def __init__(self, database_resource, generate_schema, teardown=False):
super(SchemaResource, self).__init__()
self.generate_schema = generate_schema
self.teardown = teardown
self.resources = [
('database', database_resource)
]
def clean(self, resource):
LOG.info(
"DROP ALL OBJECTS, BACKEND %s",
resource.database.engine.url)
resource.database.backend.drop_all_objects(
resource.database.engine)
def make(self, dependency_resources):
if self.generate_schema:
self.generate_schema(dependency_resources['database'].engine)
return Schema()
def isDirty(self):
if self.teardown:
return True
else:
return False
class Backend(object):
"""Represent a particular database backend that may be provisionable.
The ``Backend`` object maintains a database type (e.g. database without
specific driver type, such as "sqlite", "postgresql", etc.),
a target URL, a base ``Engine`` for that URL object that can be used
to provision databases and a ``BackendImpl`` which knows how to perform
operations against this type of ``Engine``.
"""
backends_by_database_type = {}
def __init__(self, database_type, url):
self.database_type = database_type
self.url = url
self.verified = False
self.engine = None
self.impl = BackendImpl.impl(database_type)
self.current_dbs = set()
@classmethod
def backend_for_database_type(cls, database_type):
"""Return the ``Backend`` for the given database type.
"""
try:
backend = cls.backends_by_database_type[database_type]
except KeyError:
raise exception.BackendNotAvailable(
"Backend '%s' is unavailable: No such backend" % database_type)
else:
return backend._verify()
@classmethod
def all_viable_backends(cls):
"""Return an iterator of all ``Backend`` objects that are present
and provisionable.
"""
for backend in cls.backends_by_database_type.values():
try:
yield backend._verify()
except exception.BackendNotAvailable:
pass
def _verify(self):
"""Verify that this ``Backend`` is available and provisionable.
:return: this ``Backend``
:raises: ``BackendNotAvailable`` if the backend is not available.
"""
if not self.verified:
try:
eng = self._ensure_backend_available(self.url)
except exception.BackendNotAvailable as bne:
self._no_engine_reason = str(bne)
raise
else:
self.engine = eng
finally:
self.verified = True
if self.engine is None:
raise exception.BackendNotAvailable(self._no_engine_reason)
return self
@classmethod
def _ensure_backend_available(cls, url):
url = sa_url.make_url(str(url))
try:
eng = sqlalchemy.create_engine(url)
except ImportError as i_e:
# SQLAlchemy performs an "import" of the DBAPI module
# within create_engine(). So if ibm_db_sa, cx_oracle etc.
# isn't installed, we get an ImportError here.
LOG.info(
_LI("The %(dbapi)s backend is unavailable: %(err)s"),
dict(dbapi=url.drivername, err=i_e))
raise exception.BackendNotAvailable(
"Backend '%s' is unavailable: No DBAPI installed" %
url.drivername)
else:
try:
conn = eng.connect()
except sqlalchemy.exc.DBAPIError as d_e:
# upon connect, SQLAlchemy calls dbapi.connect(). This
# usually raises OperationalError and should always at
# least raise a SQLAlchemy-wrapped DBAPI Error.
LOG.info(
_LI("The %(dbapi)s backend is unavailable: %(err)s"),
dict(dbapi=url.drivername, err=d_e)
)
raise exception.BackendNotAvailable(
"Backend '%s' is unavailable: Could not connect" %
url.drivername)
else:
conn.close()
return eng
def create_named_database(self, ident, conditional=False):
"""Create a database with the given name."""
if not conditional or ident not in self.current_dbs:
self.current_dbs.add(ident)
self.impl.create_named_database(
self.engine, ident, conditional=conditional)
def drop_named_database(self, ident, conditional=False):
"""Drop a database with the given name."""
self.impl.drop_named_database(
self.engine, ident,
conditional=conditional)
self.current_dbs.discard(ident)
def drop_all_objects(self, engine):
"""Drop all database objects.
Drops all database objects remaining on the default schema of the
given engine.
"""
self.impl.drop_all_objects(engine)
def database_exists(self, ident):
"""Return True if a database of the given name exists."""
return self.impl.database_exists(self.engine, ident)
def provisioned_engine(self, ident):
"""Given the URL of a particular database backend and the string
name of a particular 'database' within that backend, return
an Engine instance whose connections will refer directly to the
named database.
For hostname-based URLs, this typically involves switching just the
'database' portion of the URL with the given name and creating
an engine.
For URLs that instead deal with DSNs, the rules may be more custom;
for example, the engine may need to connect to the root URL and
then emit a command to switch to the named database.
"""
return self.impl.provisioned_engine(self.url, ident)
@classmethod
def _setup(cls):
"""Initial startup feature will scan the environment for configured
URLs and place them into the list of URLs we will use for provisioning.
This searches through OS_TEST_DBAPI_ADMIN_CONNECTION for URLs. If
not present, we set up URLs based on the "opportunstic" convention,
e.g. username+password = "openstack_citest".
The provisioning system will then use or discard these URLs as they
are requested, based on whether or not the target database is actually
found to be available.
"""
configured_urls = os.getenv('OS_TEST_DBAPI_ADMIN_CONNECTION', None)
if configured_urls:
configured_urls = configured_urls.split(";")
else:
configured_urls = [
impl.create_opportunistic_driver_url()
for impl in BackendImpl.all_impls()
]
for url_str in configured_urls:
url = sa_url.make_url(url_str)
m = re.match(r'([^+]+?)(?:\+(.+))?$', url.drivername)
database_type = m.group(1)
Backend.backends_by_database_type[database_type] = \
Backend(database_type, url)
@six.add_metaclass(abc.ABCMeta)
class BackendImpl(object):
"""Provide database-specific implementations of key provisioning
functions.
``BackendImpl`` is owned by a ``Backend`` instance which delegates
to it for all database-specific features.
"""
default_engine_kwargs = {}
supports_drop_fk = True
@classmethod
def all_impls(cls):
"""Return an iterator of all possible BackendImpl objects.
These are BackendImpls that are implemented, but not
necessarily provisionable.
"""
for database_type in cls.impl.reg:
if database_type == '*':
continue
yield BackendImpl.impl(database_type)
@utils.dispatch_for_dialect("*")
def impl(drivername):
"""Return a ``BackendImpl`` instance corresponding to the
given driver name.
This is a dispatched method which will refer to the constructor
of implementing subclasses.
"""
raise NotImplementedError(
"No provision impl available for driver: %s" % drivername)
def __init__(self, drivername):
self.drivername = drivername
@abc.abstractmethod
def create_opportunistic_driver_url(self):
"""Produce a string url known as the 'opportunistic' URL.
This URL is one that corresponds to an established Openstack
convention for a pre-established database login, which, when
detected as available in the local environment, is automatically
used as a test platform for a specific type of driver.
"""
@abc.abstractmethod
def create_named_database(self, engine, ident, conditional=False):
"""Create a database with the given name."""
@abc.abstractmethod
def drop_named_database(self, engine, ident, conditional=False):
"""Drop a database with the given name."""
def drop_all_objects(self, engine):
"""Drop all database objects.
Drops all database objects remaining on the default schema of the
given engine.
Per-db implementations will also need to drop items specific to those
systems, such as sequences, custom types (e.g. pg ENUM), etc.
"""
with engine.begin() as conn:
inspector = sqlalchemy.inspect(engine)
metadata = schema.MetaData()
tbs = []
all_fks = []
for table_name in inspector.get_table_names():
fks = []
for fk in inspector.get_foreign_keys(table_name):
# note that SQLite reflection does not have names
# for foreign keys until SQLAlchemy 1.0
if not fk['name']:
continue
fks.append(
schema.ForeignKeyConstraint((), (), name=fk['name'])
)
table = schema.Table(table_name, metadata, *fks)
tbs.append(table)
all_fks.extend(fks)
if self.supports_drop_fk:
for fkc in all_fks:
conn.execute(schema.DropConstraint(fkc))
for table in tbs:
conn.execute(schema.DropTable(table))
self.drop_additional_objects(conn)
def drop_additional_objects(self, conn):
pass
def provisioned_engine(self, base_url, ident):
"""Return a provisioned engine.
Given the URL of a particular database backend and the string
name of a particular 'database' within that backend, return
an Engine instance whose connections will refer directly to the
named database.
For hostname-based URLs, this typically involves switching just the
'database' portion of the URL with the given name and creating
an engine.
For URLs that instead deal with DSNs, the rules may be more custom;
for example, the engine may need to connect to the root URL and
then emit a command to switch to the named database.
"""
url = sa_url.make_url(str(base_url))
url.database = ident
return session.create_engine(
url,
logging_name="%s@%s" % (self.drivername, ident),
**self.default_engine_kwargs
)
@BackendImpl.impl.dispatch_for("mysql")
class MySQLBackendImpl(BackendImpl):
default_engine_kwargs = {'mysql_sql_mode': 'TRADITIONAL'}
def create_opportunistic_driver_url(self):
return "mysql+pymysql://openstack_citest:openstack_citest@localhost/"
def create_named_database(self, engine, ident, conditional=False):
with engine.connect() as conn:
if not conditional or not self.database_exists(conn, ident):
conn.execute("CREATE DATABASE %s" % ident)
def drop_named_database(self, engine, ident, conditional=False):
with engine.connect() as conn:
if not conditional or self.database_exists(conn, ident):
conn.execute("DROP DATABASE %s" % ident)
def database_exists(self, engine, ident):
return bool(engine.scalar("SHOW DATABASES LIKE '%s'" % ident))
@BackendImpl.impl.dispatch_for("sqlite")
class SQLiteBackendImpl(BackendImpl):
supports_drop_fk = False
def create_opportunistic_driver_url(self):
return "sqlite://"
def create_named_database(self, engine, ident, conditional=False):
url = self._provisioned_database_url(engine.url, ident)
filename = url.database
if filename and (not conditional or not os.access(filename, os.F_OK)):
eng = sqlalchemy.create_engine(url)
eng.connect().close()
def provisioned_engine(self, base_url, ident):
return session.create_engine(
self._provisioned_database_url(base_url, ident))
def drop_named_database(self, engine, ident, conditional=False):
url = self._provisioned_database_url(engine.url, ident)
filename = url.database
if filename and (not conditional or os.access(filename, os.F_OK)):
os.remove(filename)
def database_exists(self, engine, ident):
url = self._provisioned_database_url(engine.url, ident)
filename = url.database
return not filename or os.access(filename, os.F_OK)
def _provisioned_database_url(self, base_url, ident):
if base_url.database:
return sa_url.make_url("sqlite:////tmp/%s.db" % ident)
else:
return base_url
@BackendImpl.impl.dispatch_for("postgresql")
class PostgresqlBackendImpl(BackendImpl):
def create_opportunistic_driver_url(self):
return "postgresql://openstack_citest:openstack_citest"\
"@localhost/postgres"
def create_named_database(self, engine, ident, conditional=False):
with engine.connect().execution_options(
isolation_level="AUTOCOMMIT") as conn:
if not conditional or not self.database_exists(conn, ident):
conn.execute("CREATE DATABASE %s" % ident)
def drop_named_database(self, engine, ident, conditional=False):
with engine.connect().execution_options(
isolation_level="AUTOCOMMIT") as conn:
self._close_out_database_users(conn, ident)
if conditional:
conn.execute("DROP DATABASE IF EXISTS %s" % ident)
else:
conn.execute("DROP DATABASE %s" % ident)
def drop_additional_objects(self, conn):
enums = [e['name'] for e in sqlalchemy.inspect(conn).get_enums()]
for e in enums:
conn.execute("DROP TYPE %s" % e)
def database_exists(self, engine, ident):
return bool(
engine.scalar(
sqlalchemy.text(
"select datname from pg_database "
"where datname=:name"), name=ident)
)
def _close_out_database_users(self, conn, ident):
"""Attempt to guarantee a database can be dropped.
Optional feature which guarantees no connections with our
username are attached to the DB we're going to drop.
This method has caveats; for one, the 'pid' column was named
'procpid' prior to Postgresql 9.2. But more critically,
prior to 9.2 this operation required superuser permissions,
even if the connections we're closing are under the same username
as us. In more recent versions this restriction has been
lifted for same-user connections.
"""
if conn.dialect.server_version_info >= (9, 2):
conn.execute(
sqlalchemy.text(
"select pg_terminate_backend(pid) "
"from pg_stat_activity "
"where usename=current_user and "
"pid != pg_backend_pid() "
"and datname=:dname"
), dname=ident)
def _random_ident():
return ''.join(
random.choice(string.ascii_lowercase)
for i in moves.range(10))
Backend._setup()

196
oslo_db/sqlalchemy/session.py
View File

@ -1,196 +0,0 @@
# Copyright 2010 United States Government as represented by the
# Administrator of the National Aeronautics and Space Administration.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""Session Handling for SQLAlchemy backend.
Recommended ways to use sessions within this framework:
* Use the ``enginefacade`` system for connectivity, session and
transaction management:
.. code-block:: python
from oslo_db.sqlalchemy import enginefacade
@enginefacade.reader
def get_foo(context, foo):
return (model_query(models.Foo, context.session).
filter_by(foo=foo).
first())
@enginefacade.writer
def update_foo(context, id, newfoo):
(model_query(models.Foo, context.session).
filter_by(id=id).
update({'foo': newfoo}))
@enginefacade.writer
def create_foo(context, values):
foo_ref = models.Foo()
foo_ref.update(values)
foo_ref.save(context.session)
return foo_ref
In the above system, transactions are committed automatically, and
are shared among all dependent database methods. Ensure
that methods which "write" data are enclosed within @writer blocks.
.. note:: Statements in the session scope will not be automatically retried.
* If you create models within the session, they need to be added, but you
do not need to call `model.save()`:
.. code-block:: python
@enginefacade.writer
def create_many_foo(context, foos):
for foo in foos:
foo_ref = models.Foo()
foo_ref.update(foo)
context.session.add(foo_ref)
@enginefacade.writer
def update_bar(context, foo_id, newbar):
foo_ref = (model_query(models.Foo, context.session).
filter_by(id=foo_id).
first())
(model_query(models.Bar, context.session).
filter_by(id=foo_ref['bar_id']).
update({'bar': newbar}))
The two queries in `update_bar` can alternatively be expressed using
a single query, which may be more efficient depending on scenario:
.. code-block:: python
@enginefacade.writer
def update_bar(context, foo_id, newbar):
subq = (model_query(models.Foo.id, context.session).
filter_by(id=foo_id).
limit(1).
subquery())
(model_query(models.Bar, context.session).
filter_by(id=subq.as_scalar()).
update({'bar': newbar}))
For reference, this emits approximately the following SQL statement:
.. code-block:: sql
UPDATE bar SET bar = ${newbar}
WHERE id=(SELECT bar_id FROM foo WHERE id = ${foo_id} LIMIT 1);
.. note:: `create_duplicate_foo` is a trivially simple example of catching an
exception while using a savepoint. Here we create two duplicate
instances with same primary key, must catch the exception out of context
managed by a single session:
.. code-block:: python
@enginefacade.writer
def create_duplicate_foo(context):
foo1 = models.Foo()
foo2 = models.Foo()
foo1.id = foo2.id = 1
try:
with context.session.begin_nested():
session.add(foo1)
session.add(foo2)
except exception.DBDuplicateEntry as e:
handle_error(e)
* The enginefacade system eliminates the need to decide when sessions need
to be passed between methods. All methods should instead share a common
context object; the enginefacade system will maintain the transaction
across method calls.
.. code-block:: python
@enginefacade.writer
def myfunc(context, foo):
# do some database things
bar = _private_func(context, foo)
return bar
def _private_func(context, foo):
with enginefacade.using_writer(context) as session:
# do some other database things
session.add(SomeObject())
return bar
* Avoid ``with_lockmode('UPDATE')`` when possible.
FOR UPDATE is not compatible with MySQL/Galera. Instead, an "opportunistic"
approach should be used, such that if an UPDATE fails, the entire
transaction should be retried. The @wrap_db_retry decorator is one
such system that can be used to achieve this.
Enabling soft deletes:
* To use/enable soft-deletes, `SoftDeleteMixin` may be used. For example:
.. code-block:: python
class NovaBase(models.SoftDeleteMixin, models.ModelBase):
pass
Efficient use of soft deletes:
* While there is a ``model.soft_delete()`` method, prefer
``query.soft_delete()``. Some examples:
.. code-block:: python
@enginefacade.writer
def soft_delete_bar(context):
# synchronize_session=False will prevent the ORM from attempting
# to search the Session for instances matching the DELETE;
# this is typically not necessary for small operations.
count = model_query(BarModel, context.session).\\
find(some_condition).soft_delete(synchronize_session=False)
if count == 0:
raise Exception("0 entries were soft deleted")
@enginefacade.writer
def complex_soft_delete_with_synchronization_bar(context):
# use synchronize_session='evaluate' when you'd like to attempt
# to update the state of the Session to match that of the DELETE.
# This is potentially helpful if the operation is complex and
# continues to work with instances that were loaded, though
# not usually needed.
count = (model_query(BarModel, context.session).
find(some_condition).
soft_delete(synchronize_session='evaulate'))
if count == 0:
raise Exception("0 entries were soft deleted")
"""
from oslo_db.sqlalchemy import enginefacade
from oslo_db.sqlalchemy import engines
from oslo_db.sqlalchemy import orm
EngineFacade = enginefacade.LegacyEngineFacade
create_engine = engines.create_engine
get_maker = orm.get_maker
Query = orm.Query
Session = orm.Session
__all__ = ["EngineFacade", "create_engine", "get_maker", "Query", "Session"]

274
oslo_db/sqlalchemy/test_base.py
View File

@ -1,274 +0,0 @@
# Copyright (c) 2013 OpenStack Foundation
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import fixtures
import testresources
import testscenarios
try:
from oslotest import base as test_base
except ImportError:
raise NameError('Oslotest is not installed. Please add oslotest in your'
' test-requirements')
import os
from oslo_utils import reflection
import six
from oslo_db import exception
from oslo_db.sqlalchemy import enginefacade
from oslo_db.sqlalchemy import provision
from oslo_db.sqlalchemy import session
class DbFixture(fixtures.Fixture):
"""Basic database fixture.
Allows to run tests on various db backends, such as SQLite, MySQL and
PostgreSQL. By default use sqlite backend. To override default backend
uri set env variable OS_TEST_DBAPI_ADMIN_CONNECTION with database admin
credentials for specific backend.
"""
DRIVER = "sqlite"
# these names are deprecated, and are not used by DbFixture.
# they are here for backwards compatibility with test suites that
# are referring to them directly.
DBNAME = PASSWORD = USERNAME = 'openstack_citest'
def __init__(self, test, skip_on_unavailable_db=True):
super(DbFixture, self).__init__()
self.test = test
self.skip_on_unavailable_db = skip_on_unavailable_db
def setUp(self):
super(DbFixture, self).setUp()
testresources.setUpResources(
self.test, self.test.resources, testresources._get_result())
self.addCleanup(
testresources.tearDownResources,
self.test, self.test.resources, testresources._get_result()
)
if not self.test._has_db_resource():
msg = self.test._get_db_resource_not_available_reason()
if self.test.SKIP_ON_UNAVAILABLE_DB:
self.test.skip(msg)
else:
self.test.fail(msg)
if self.test.SCHEMA_SCOPE:
self.test.engine = self.test.transaction_engine
self.test.sessionmaker = session.get_maker(
self.test.transaction_engine)
else:
self.test.engine = self.test.db.engine
self.test.sessionmaker = session.get_maker(self.test.engine)
self.addCleanup(setattr, self.test, 'sessionmaker', None)
self.addCleanup(setattr, self.test, 'engine', None)
self.test.enginefacade = enginefacade._TestTransactionFactory(
self.test.engine, self.test.sessionmaker, apply_global=True,
synchronous_reader=True)
self.addCleanup(self.test.enginefacade.dispose_global)
class DbTestCase(test_base.BaseTestCase):
"""Base class for testing of DB code.
"""
FIXTURE = DbFixture
SCHEMA_SCOPE = None
SKIP_ON_UNAVAILABLE_DB = True
_db_not_available = {}
_schema_resources = {}
_database_resources = {}
def _get_db_resource_not_available_reason(self):
return self._db_not_available.get(self.FIXTURE.DRIVER, None)
def _has_db_resource(self):
return self._database_resources.get(
self.FIXTURE.DRIVER, None) is not None
def _resources_for_driver(self, driver, schema_scope, generate_schema):
# testresources relies on the identity and state of the
# TestResourceManager objects in play to correctly manage
# resources, and it also hardcodes to looking at the
# ".resources" attribute on the test object, even though the
# setUpResources() function passes the list of resources in,
# so we have to code the TestResourceManager logic into the
# .resources attribute and ensure that the same set of test
# variables always produces the same TestResourceManager objects.
if driver not in self._database_resources:
try:
self._database_resources[driver] = \
provision.DatabaseResource(driver)
except exception.BackendNotAvailable as bne:
self._database_resources[driver] = None
self._db_not_available[driver] = str(bne)
database_resource = self._database_resources[driver]
if database_resource is None:
return []
if schema_scope:
key = (driver, schema_scope)
if key not in self._schema_resources:
schema_resource = provision.SchemaResource(
database_resource, generate_schema)
transaction_resource = provision.TransactionResource(
database_resource, schema_resource)
self._schema_resources[key] = \
transaction_resource
transaction_resource = self._schema_resources[key]
return [
('transaction_engine', transaction_resource),
('db', database_resource),
]
else:
key = (driver, None)
if key not in self._schema_resources:
self._schema_resources[key] = provision.SchemaResource(
database_resource, generate_schema, teardown=True)
schema_resource = self._schema_resources[key]
return [
('schema', schema_resource),
('db', database_resource)
]
@property
def resources(self):
return self._resources_for_driver(
self.FIXTURE.DRIVER, self.SCHEMA_SCOPE, self.generate_schema)
def setUp(self):
super(DbTestCase, self).setUp()
self.useFixture(
self.FIXTURE(
self, skip_on_unavailable_db=self.SKIP_ON_UNAVAILABLE_DB))
def generate_schema(self, engine):
"""Generate schema objects to be used within a test.
The function is separate from the setUp() case as the scope
of this method is controlled by the provisioning system. A
test that specifies SCHEMA_SCOPE may not call this method
for each test, as the schema may be maintained from a previous run.
"""
if self.SCHEMA_SCOPE:
# if SCHEMA_SCOPE is set, then this method definitely
# has to be implemented. This is a guard against a test
# that inadvertently does schema setup within setUp().
raise NotImplementedError(
"This test requires schema-level setup to be "
"implemented within generate_schema().")
class OpportunisticTestCase(DbTestCase):
"""Placeholder for backwards compatibility."""
ALLOWED_DIALECTS = ['sqlite', 'mysql', 'postgresql']
def backend_specific(*dialects):
"""Decorator to skip backend specific tests on inappropriate engines.
::dialects: list of dialects names under which the test will be launched.
"""
def wrap(f):
@six.wraps(f)
def ins_wrap(self):
if not set(dialects).issubset(ALLOWED_DIALECTS):
raise ValueError(
"Please use allowed dialects: %s" % ALLOWED_DIALECTS)
if self.engine.name not in dialects:
msg = ('The test "%s" can be run '
'only on %s. Current engine is %s.')
args = (reflection.get_callable_name(f), ', '.join(dialects),
self.engine.name)
self.skip(msg % args)
else:
return f(self)
return ins_wrap
return wrap
class MySQLOpportunisticFixture(DbFixture):
DRIVER = 'mysql'
class PostgreSQLOpportunisticFixture(DbFixture):
DRIVER = 'postgresql'
class MySQLOpportunisticTestCase(OpportunisticTestCase):
FIXTURE = MySQLOpportunisticFixture
class PostgreSQLOpportunisticTestCase(OpportunisticTestCase):
FIXTURE = PostgreSQLOpportunisticFixture
def optimize_db_test_loader(file_):
"""Package level load_tests() function.
Will apply an optimizing test suite to all sub-tests, which groups DB
tests and other resources appropriately.
Place this in an __init__.py package file within the root of the test
suite, at the level where testresources loads it as a package::
from oslo_db.sqlalchemy import test_base
load_tests = test_base.optimize_db_test_loader(__file__)
Alternatively, the directive can be placed into a test module directly.
"""
this_dir = os.path.dirname(file_)
def load_tests(loader, found_tests, pattern):
# pattern is None if the directive is placed within
# a test module directly, as well as within certain test
# discovery patterns
if pattern is not None:
pkg_tests = loader.discover(start_dir=this_dir, pattern=pattern)
result = testresources.OptimisingTestSuite()
found_tests = testscenarios.load_tests_apply_scenarios(
loader, found_tests, pattern)
result.addTest(found_tests)
if pattern is not None:
result.addTest(pkg_tests)
return result
return load_tests

618
oslo_db/sqlalchemy/test_migrations.py
View File

@ -1,618 +0,0 @@
# Copyright 2010-2011 OpenStack Foundation
# Copyright 2012-2013 IBM Corp.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import abc
import collections
import functools
import logging
import pprint
import alembic
import alembic.autogenerate
import alembic.migration
import pkg_resources as pkg
import six
import sqlalchemy
import sqlalchemy.exc
import sqlalchemy.sql.expression as expr
import sqlalchemy.types as types
from oslo_db._i18n import _LE
from oslo_db import exception as exc
from oslo_db.sqlalchemy import utils
LOG = logging.getLogger(__name__)
@six.add_metaclass(abc.ABCMeta)
class WalkVersionsMixin(object):
"""Test mixin to check upgrade and downgrade ability of migration.
This is only suitable for testing of migrate_ migration scripts. An
abstract class mixin. `INIT_VERSION`, `REPOSITORY` and `migration_api`
attributes must be implemented in subclasses.
.. _auxiliary-dynamic-methods: Auxiliary Methods
Auxiliary Methods:
`migrate_up` and `migrate_down` instance methods of the class can be
used with auxiliary methods named `_pre_upgrade_<revision_id>`,
`_check_<revision_id>`, `_post_downgrade_<revision_id>`. The methods
intended to check applied changes for correctness of data operations.
This methods should be implemented for every particular revision
which you want to check with data. Implementation recommendations for
`_pre_upgrade_<revision_id>`, `_check_<revision_id>`,
`_post_downgrade_<revision_id>` implementation:
* `_pre_upgrade_<revision_id>`: provide a data appropriate to
a next revision. Should be used an id of revision which
going to be applied.
* `_check_<revision_id>`: Insert, select, delete operations
with newly applied changes. The data provided by
`_pre_upgrade_<revision_id>` will be used.
* `_post_downgrade_<revision_id>`: check for absence
(inability to use) changes provided by reverted revision.
Execution order of auxiliary methods when revision is upgrading:
`_pre_upgrade_###` => `upgrade` => `_check_###`
Execution order of auxiliary methods when revision is downgrading:
`downgrade` => `_post_downgrade_###`
.. _migrate: https://sqlalchemy-migrate.readthedocs.org/en/latest/
"""
@abc.abstractproperty
def INIT_VERSION(self):
"""Initial version of a migration repository.
Can be different from 0, if a migrations were squashed.
:rtype: int
"""
pass
@abc.abstractproperty
def REPOSITORY(self):
"""Allows basic manipulation with migration repository.
:returns: `migrate.versioning.repository.Repository` subclass.
"""
pass
@abc.abstractproperty
def migration_api(self):
"""Provides API for upgrading, downgrading and version manipulations.
:returns: `migrate.api` or overloaded analog.
"""
pass
@abc.abstractproperty
def migrate_engine(self):
"""Provides engine instance.
Should be the same instance as used when migrations are applied. In
most cases, the `engine` attribute provided by the test class in a
`setUp` method will work.
Example of implementation:
def migrate_engine(self):
return self.engine
:returns: sqlalchemy engine instance
"""
pass
def _walk_versions(self, snake_walk=False, downgrade=True):
"""Check if migration upgrades and downgrades successfully.
DEPRECATED: this function is deprecated and will be removed from
oslo.db in a few releases. Please use walk_versions() method instead.
"""
self.walk_versions(snake_walk, downgrade)
def _migrate_down(self, version, with_data=False):
"""Migrate down to a previous version of the db.
DEPRECATED: this function is deprecated and will be removed from
oslo.db in a few releases. Please use migrate_down() method instead.
"""
return self.migrate_down(version, with_data)
def _migrate_up(self, version, with_data=False):
"""Migrate up to a new version of the db.
DEPRECATED: this function is deprecated and will be removed from
oslo.db in a few releases. Please use migrate_up() method instead.
"""
self.migrate_up(version, with_data)
def walk_versions(self, snake_walk=False, downgrade=True):
"""Check if migration upgrades and downgrades successfully.
Determine the latest version script from the repo, then
upgrade from 1 through to the latest, with no data
in the databases. This just checks that the schema itself
upgrades successfully.
`walk_versions` calls `migrate_up` and `migrate_down` with
`with_data` argument to check changes with data, but these methods
can be called without any extra check outside of `walk_versions`
method.
:param snake_walk: enables checking that each individual migration can
be upgraded/downgraded by itself.
If we have ordered migrations 123abc, 456def, 789ghi and we run
upgrading with the `snake_walk` argument set to `True`, the
migrations will be applied in the following order::
`123abc => 456def => 123abc =>
456def => 789ghi => 456def => 789ghi`
:type snake_walk: bool
:param downgrade: Check downgrade behavior if True.
:type downgrade: bool
"""
# Place the database under version control
self.migration_api.version_control(self.migrate_engine,
self.REPOSITORY,
self.INIT_VERSION)
self.assertEqual(self.INIT_VERSION,
self.migration_api.db_version(self.migrate_engine,
self.REPOSITORY))
LOG.debug('latest version is %s', self.REPOSITORY.latest)
versions = range(int(self.INIT_VERSION) + 1,
int(self.REPOSITORY.latest) + 1)
for version in versions:
# upgrade -> downgrade -> upgrade
self.migrate_up(version, with_data=True)
if snake_walk:
downgraded = self.migrate_down(version - 1, with_data=True)
if downgraded:
self.migrate_up(version)
if downgrade:
# Now walk it back down to 0 from the latest, testing
# the downgrade paths.
for version in reversed(versions):
# downgrade -> upgrade -> downgrade
downgraded = self.migrate_down(version - 1)
if snake_walk and downgraded:
self.migrate_up(version)
self.migrate_down(version - 1)
def migrate_down(self, version, with_data=False):
"""Migrate down to a previous version of the db.
:param version: id of revision to downgrade.
:type version: str
:keyword with_data: Whether to verify the absence of changes from
migration(s) being downgraded, see
:ref:`auxiliary-dynamic-methods <Auxiliary Methods>`.
:type with_data: Bool
"""
try:
self.migration_api.downgrade(self.migrate_engine,
self.REPOSITORY, version)
except NotImplementedError:
# NOTE(sirp): some migrations, namely release-level
# migrations, don't support a downgrade.
return False
self.assertEqual(version, self.migration_api.db_version(
self.migrate_engine, self.REPOSITORY))
# NOTE(sirp): `version` is what we're downgrading to (i.e. the 'target'
# version). So if we have any downgrade checks, they need to be run for
# the previous (higher numbered) migration.
if with_data:
post_downgrade = getattr(
self, "_post_downgrade_%03d" % (version + 1), None)
if post_downgrade:
post_downgrade(self.migrate_engine)
return True
def migrate_up(self, version, with_data=False):
"""Migrate up to a new version of the db.
:param version: id of revision to upgrade.
:type version: str
:keyword with_data: Whether to verify the applied changes with data,
see :ref:`auxiliary-dynamic-methods <Auxiliary Methods>`.
:type with_data: Bool
"""
# NOTE(sdague): try block is here because it's impossible to debug
# where a failed data migration happens otherwise
try:
if with_data:
data = None
pre_upgrade = getattr(
self, "_pre_upgrade_%03d" % version, None)
if pre_upgrade:
data = pre_upgrade(self.migrate_engine)
self.migration_api.upgrade(self.migrate_engine,
self.REPOSITORY, version)
self.assertEqual(version,
self.migration_api.db_version(self.migrate_engine,
self.REPOSITORY))
if with_data:
check = getattr(self, "_check_%03d" % version, None)
if check:
check(self.migrate_engine, data)
except exc.DBMigrationError:
msg = _LE("Failed to migrate to version %(ver)s on engine %(eng)s")
LOG.error(msg, {"ver": version, "eng": self.migrate_engine})
raise
@six.add_metaclass(abc.ABCMeta)
class ModelsMigrationsSync(object):
"""A helper class for comparison of DB migration scripts and models.
It's intended to be inherited by test cases in target projects. They have
to provide implementations for methods used internally in the test (as
we have no way to implement them here).
test_model_sync() will run migration scripts for the engine provided and
then compare the given metadata to the one reflected from the database.
The difference between MODELS and MIGRATION scripts will be printed and
the test will fail, if the difference is not empty. The return value is
really a list of actions, that should be performed in order to make the
current database schema state (i.e. migration scripts) consistent with
models definitions. It's left up to developers to analyze the output and
decide whether the models definitions or the migration scripts should be
modified to make them consistent.
Output::
[(
'add_table',
description of the table from models
),
(
'remove_table',
description of the table from database
),
(
'add_column',
schema,
table name,
column description from models
),
(
'remove_column',
schema,
table name,
column description from database
),
(
'add_index',
description of the index from models
),
(
'remove_index',
description of the index from database
),
(
'add_constraint',
description of constraint from models
),
(
'remove_constraint,
description of constraint from database
),
(
'modify_nullable',
schema,
table name,
column name,
{
'existing_type': type of the column from database,
'existing_server_default': default value from database
},
nullable from database,
nullable from models
),
(
'modify_type',
schema,
table name,
column name,
{
'existing_nullable': database nullable,
'existing_server_default': default value from database
},
database column type,
type of the column from models
),
(
'modify_default',
schema,
table name,
column name,
{
'existing_nullable': database nullable,
'existing_type': type of the column from database
},
connection column default value,
default from models
)]
Method include_object() can be overridden to exclude some tables from
comparison (e.g. migrate_repo).
"""
@abc.abstractmethod
def db_sync(self, engine):
"""Run migration scripts with the given engine instance.
This method must be implemented in subclasses and run migration scripts
for a DB the given engine is connected to.
"""
@abc.abstractmethod
def get_engine(self):
"""Return the engine instance to be used when running tests.
This method must be implemented in subclasses and return an engine
instance to be used when running tests.
"""
@abc.abstractmethod
def get_metadata(self):
"""Return the metadata instance to be used for schema comparison.
This method must be implemented in subclasses and return the metadata
instance attached to the BASE model.
"""
def include_object(self, object_, name, type_, reflected, compare_to):
"""Return True for objects that should be compared.
:param object_: a SchemaItem object such as a Table or Column object
:param name: the name of the object
:param type_: a string describing the type of object (e.g. "table")
:param reflected: True if the given object was produced based on
table reflection, False if it's from a local
MetaData object
:param compare_to: the object being compared against, if available,
else None
"""
return True
def compare_type(self, ctxt, insp_col, meta_col, insp_type, meta_type):
"""Return True if types are different, False if not.
Return None to allow the default implementation to compare these types.
:param ctxt: alembic MigrationContext instance
:param insp_col: reflected column
:param meta_col: column from model
:param insp_type: reflected column type
:param meta_type: column type from model
"""
# some backends (e.g. mysql) don't provide native boolean type
BOOLEAN_METADATA = (types.BOOLEAN, types.Boolean)
BOOLEAN_SQL = BOOLEAN_METADATA + (types.INTEGER, types.Integer)
if issubclass(type(meta_type), BOOLEAN_METADATA):
return not issubclass(type(insp_type), BOOLEAN_SQL)
# Alembic <=0.8.4 do not contain logic of comparing Variant type with
# others.
if isinstance(meta_type, types.Variant):
orig_type = meta_col.type
impl_type = meta_type.load_dialect_impl(ctxt.dialect)
meta_col.type = impl_type
try:
return self.compare_type(ctxt, insp_col, meta_col, insp_type,
impl_type)
finally:
meta_col.type = orig_type
return ctxt.impl.compare_type(insp_col, meta_col)
def compare_server_default(self, ctxt, ins_col, meta_col,
insp_def, meta_def, rendered_meta_def):
"""Compare default values between model and db table.
Return True if the defaults are different, False if not, or None to
allow the default implementation to compare these defaults.
:param ctxt: alembic MigrationContext instance
:param insp_col: reflected column
:param meta_col: column from model
:param insp_def: reflected column default value
:param meta_def: column default value from model
:param rendered_meta_def: rendered column default value (from model)
"""
return self._compare_server_default(ctxt.bind, meta_col, insp_def,
meta_def)
@utils.DialectFunctionDispatcher.dispatch_for_dialect("*")
def _compare_server_default(bind, meta_col, insp_def, meta_def):
pass
@_compare_server_default.dispatch_for('mysql')
def _compare_server_default(bind, meta_col, insp_def, meta_def):
if isinstance(meta_col.type, sqlalchemy.Boolean):
if meta_def is None or insp_def is None:
return meta_def != insp_def
return not (
isinstance(meta_def.arg, expr.True_) and insp_def == "'1'" or
isinstance(meta_def.arg, expr.False_) and insp_def == "'0'"
)
impl_type = meta_col.type
if isinstance(impl_type, types.Variant):
impl_type = impl_type.load_dialect_impl(bind.dialect)
if isinstance(impl_type, (sqlalchemy.Integer, sqlalchemy.BigInteger)):
if meta_def is None or insp_def is None:
return meta_def != insp_def
return meta_def.arg != insp_def.split("'")[1]
@_compare_server_default.dispatch_for('postgresql')
def _compare_server_default(bind, meta_col, insp_def, meta_def):
if isinstance(meta_col.type, sqlalchemy.Enum):
if meta_def is None or insp_def is None:
return meta_def != insp_def
return insp_def != "'%s'::%s" % (meta_def.arg, meta_col.type.name)
elif isinstance(meta_col.type, sqlalchemy.String):
if meta_def is None or insp_def is None:
return meta_def != insp_def
return insp_def != "'%s'::character varying" % meta_def.arg
FKInfo = collections.namedtuple('fk_info', ['constrained_columns',
'referred_table',
'referred_columns'])
def check_foreign_keys(self, metadata, bind):
"""Compare foreign keys between model and db table.
:returns: a list that contains information about:
* should be a new key added or removed existing,
* name of that key,
* source table,
* referred table,
* constrained columns,
* referred columns
Output::
[('drop_key',
'testtbl_fk_check_fkey',
'testtbl',
fk_info(constrained_columns=(u'fk_check',),
referred_table=u'table',
referred_columns=(u'fk_check',)))]
DEPRECATED: this function is deprecated and will be removed from
oslo.db in a few releases. Alembic autogenerate.compare_metadata()
now includes foreign key comparison directly.
"""
diff = []
insp = sqlalchemy.engine.reflection.Inspector.from_engine(bind)
# Get all tables from db
db_tables = insp.get_table_names()
# Get all tables from models
model_tables = metadata.tables
for table in db_tables:
if table not in model_tables:
continue
# Get all necessary information about key of current table from db
fk_db = dict((self._get_fk_info_from_db(i), i['name'])
for i in insp.get_foreign_keys(table))
fk_db_set = set(fk_db.keys())
# Get all necessary information about key of current table from
# models
fk_models = dict((self._get_fk_info_from_model(fk), fk)
for fk in model_tables[table].foreign_keys)
fk_models_set = set(fk_models.keys())
for key in (fk_db_set - fk_models_set):
diff.append(('drop_key', fk_db[key], table, key))
LOG.info(("Detected removed foreign key %(fk)r on "
"table %(table)r"), {'fk': fk_db[key],
'table': table})
for key in (fk_models_set - fk_db_set):
diff.append(('add_key', fk_models[key], table, key))
LOG.info((
"Detected added foreign key for column %(fk)r on table "
"%(table)r"), {'fk': fk_models[key].column.name,
'table': table})
return diff
def _get_fk_info_from_db(self, fk):
return self.FKInfo(tuple(fk['constrained_columns']),
fk['referred_table'],
tuple(fk['referred_columns']))
def _get_fk_info_from_model(self, fk):
return self.FKInfo((fk.parent.name,), fk.column.table.name,
(fk.column.name,))
def filter_metadata_diff(self, diff):
"""Filter changes before assert in test_models_sync().
Allow subclasses to whitelist/blacklist changes. By default, no
filtering is performed, changes are returned as is.
:param diff: a list of differences (see `compare_metadata()` docs for
details on format)
:returns: a list of differences
"""
return diff
def test_models_sync(self):
# recent versions of sqlalchemy and alembic are needed for running of
# this test, but we already have them in requirements
try:
pkg.require('sqlalchemy>=0.8.4', 'alembic>=0.6.2')
except (pkg.VersionConflict, pkg.DistributionNotFound) as e:
self.skipTest('sqlalchemy>=0.8.4 and alembic>=0.6.3 are required'
' for running of this test: %s' % e)
# drop all tables after a test run
self.addCleanup(functools.partial(self.db.backend.drop_all_objects,
self.get_engine()))
# run migration scripts
self.db_sync(self.get_engine())
with self.get_engine().connect() as conn:
opts = {
'include_object': self.include_object,
'compare_type': self.compare_type,
'compare_server_default': self.compare_server_default,
}
mc = alembic.migration.MigrationContext.configure(conn, opts=opts)
# compare schemas and fail with diff, if it's not empty
diff = self.filter_metadata_diff(
alembic.autogenerate.compare_metadata(mc, self.get_metadata()))
if diff:
msg = pprint.pformat(diff, indent=2, width=20)
self.fail(
"Models and migration scripts aren't in sync:\n%s" % msg)

75
oslo_db/sqlalchemy/types.py
View File

@ -1,75 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import json
from sqlalchemy.types import TypeDecorator, Text
from sqlalchemy.dialects import mysql
class JsonEncodedType(TypeDecorator):
"""Base column type for data serialized as JSON-encoded string in db."""
type = None
impl = Text
def __init__(self, mysql_as_long=False, mysql_as_medium=False):
super(JsonEncodedType, self).__init__()
if mysql_as_long and mysql_as_medium:
raise TypeError("mysql_as_long and mysql_as_medium are mutually "
"exclusive")
if mysql_as_long:
self.impl = Text().with_variant(mysql.LONGTEXT(), 'mysql')
elif mysql_as_medium:
self.impl = Text().with_variant(mysql.MEDIUMTEXT(), 'mysql')
def process_bind_param(self, value, dialect):
if value is None:
if self.type is not None:
# Save default value according to current type to keep the
# interface consistent.
value = self.type()
elif self.type is not None and not isinstance(value, self.type):
raise TypeError("%s supposes to store %s objects, but %s given"
% (self.__class__.__name__,
self.type.__name__,
type(value).__name__))
serialized_value = json.dumps(value)
return serialized_value
def process_result_value(self, value, dialect):
if value is not None:
value = json.loads(value)
return value
class JsonEncodedDict(JsonEncodedType):
"""Represents dict serialized as json-encoded string in db.
Note that this type does NOT track mutations. If you want to update it, you
have to assign existing value to a temporary variable, update, then assign
back. See this page for more robust work around:
http://docs.sqlalchemy.org/en/rel_1_0/orm/extensions/mutable.html
"""
type = dict
class JsonEncodedList(JsonEncodedType):
"""Represents list serialized as json-encoded string in db.
Note that this type does NOT track mutations. If you want to update it, you
have to assign existing value to a temporary variable, update, then assign
back. See this page for more robust work around:
http://docs.sqlalchemy.org/en/rel_1_0/orm/extensions/mutable.html
"""
type = list

508
oslo_db/sqlalchemy/update_match.py
View File

@ -1,508 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import copy
from sqlalchemy import inspect
from sqlalchemy import orm
from sqlalchemy import sql
from sqlalchemy import types as sqltypes
from oslo_db.sqlalchemy import utils
def update_on_match(
query,
specimen,
surrogate_key,
values=None,
attempts=3,
include_only=None,
process_query=None,
handle_failure=None
):
"""Emit an UPDATE statement matching the given specimen.
E.g.::
with enginefacade.writer() as session:
specimen = MyInstance(
uuid='ccea54f',
interface_id='ad33fea',
vm_state='SOME_VM_STATE',
)
values = {
'vm_state': 'SOME_NEW_VM_STATE'
}
base_query = model_query(
context, models.Instance,
project_only=True, session=session)
hostname_query = model_query(
context, models.Instance, session=session,
read_deleted='no').
filter(func.lower(models.Instance.hostname) == 'SOMEHOSTNAME')
surrogate_key = ('uuid', )
def process_query(query):
return query.where(~exists(hostname_query))
def handle_failure(query):
try:
instance = base_query.one()
except NoResultFound:
raise exception.InstanceNotFound(instance_id=instance_uuid)
if session.query(hostname_query.exists()).scalar():
raise exception.InstanceExists(
name=values['hostname'].lower())
# try again
return False
persistent_instance = base_query.update_on_match(
specimen,
surrogate_key,
values=values,
process_query=process_query,
handle_failure=handle_failure
)
The UPDATE statement is constructed against the given specimen
using those values which are present to construct a WHERE clause.
If the specimen contains additional values to be ignored, the
``include_only`` parameter may be passed which indicates a sequence
of attributes to use when constructing the WHERE.
The UPDATE is performed against an ORM Query, which is created from
the given ``Session``, or alternatively by passing the ```query``
parameter referring to an existing query.
Before the query is invoked, it is also passed through the callable
sent as ``process_query``, if present. This hook allows additional
criteria to be added to the query after it is created but before
invocation.
The function will then invoke the UPDATE statement and check for
"success" one or more times, up to a maximum of that passed as
``attempts``.
The initial check for "success" from the UPDATE statement is that the
number of rows returned matches 1. If zero rows are matched, then
the UPDATE statement is assumed to have "failed", and the failure handling
phase begins.
The failure handling phase involves invoking the given ``handle_failure``
function, if any. This handler can perform additional queries to attempt
to figure out why the UPDATE didn't match any rows. The handler,
upon detection of the exact failure condition, should throw an exception
to exit; if it doesn't, it has the option of returning True or False,
where False means the error was not handled, and True means that there
was not in fact an error, and the function should return successfully.
If the failure handler is not present, or returns False after ``attempts``
number of attempts, then the function overall raises CantUpdateException.
If the handler returns True, then the function returns with no error.
The return value of the function is a persistent version of the given
specimen; this may be the specimen itself, if no matching object were
already present in the session; otherwise, the existing object is
returned, with the state of the specimen merged into it. The returned
persistent object will have the given values populated into the object.
The object is is returned as "persistent", meaning that it is
associated with the given
Session and has an identity key (that is, a real primary key
value).
In order to produce this identity key, a strategy must be used to
determine it as efficiently and safely as possible:
1. If the given specimen already contained its primary key attributes
fully populated, then these attributes were used as criteria in the
UPDATE, so we have the primary key value; it is populated directly.
2. If the target backend supports RETURNING, then when the update() query
is performed with a RETURNING clause so that the matching primary key
is returned atomically. This currently includes Postgresql, Oracle
and others (notably not MySQL or SQLite).
3. If the target backend is MySQL, and the given model uses a
single-column, AUTO_INCREMENT integer primary key value (as is
the case for Nova), MySQL's recommended approach of making use
of ``LAST_INSERT_ID(expr)`` is used to atomically acquire the
matching primary key value within the scope of the UPDATE
statement, then it fetched immediately following by using
``SELECT LAST_INSERT_ID()``.
http://dev.mysql.com/doc/refman/5.0/en/information-\
functions.html#function_last-insert-id
4. Otherwise, for composite keys on MySQL or other backends such
as SQLite, the row as UPDATED must be re-fetched in order to
acquire the primary key value. The ``surrogate_key``
parameter is used for this in order to re-fetch the row; this
is a column name with a known, unique value where
the object can be fetched.
"""
if values is None:
values = {}
entity = inspect(specimen)
mapper = entity.mapper
assert \
[desc['type'] for desc in query.column_descriptions] == \
[mapper.class_], "Query does not match given specimen"
criteria = manufacture_entity_criteria(
specimen, include_only=include_only, exclude=[surrogate_key])
query = query.filter(criteria)
if process_query:
query = process_query(query)
surrogate_key_arg = (
surrogate_key, entity.attrs[surrogate_key].loaded_value)
pk_value = None
for attempt in range(attempts):
try:
pk_value = query.update_returning_pk(values, surrogate_key_arg)
except MultiRowsMatched:
raise
except NoRowsMatched:
if handle_failure and handle_failure(query):
break
else:
break
else:
raise NoRowsMatched("Zero rows matched for %d attempts" % attempts)
if pk_value is None:
pk_value = entity.mapper.primary_key_from_instance(specimen)
# NOTE(mdbooth): Can't pass the original specimen object here as it might
# have lists of multiple potential values rather than actual values.
values = copy.copy(values)
values[surrogate_key] = surrogate_key_arg[1]
persistent_obj = manufacture_persistent_object(
query.session, specimen.__class__(), values, pk_value)
return persistent_obj
def manufacture_persistent_object(
session, specimen, values=None, primary_key=None):
"""Make an ORM-mapped object persistent in a Session without SQL.
The persistent object is returned.
If a matching object is already present in the given session, the specimen
is merged into it and the persistent object returned. Otherwise, the
specimen itself is made persistent and is returned.
The object must contain a full primary key, or provide it via the values or
primary_key parameters. The object is peristed to the Session in a "clean"
state with no pending changes.
:param session: A Session object.
:param specimen: a mapped object which is typically transient.
:param values: a dictionary of values to be applied to the specimen,
in addition to the state that's already on it. The attributes will be
set such that no history is created; the object remains clean.
:param primary_key: optional tuple-based primary key. This will also
be applied to the instance if present.
"""
state = inspect(specimen)
mapper = state.mapper
for k, v in values.items():
orm.attributes.set_committed_value(specimen, k, v)
pk_attrs = [
mapper.get_property_by_column(col).key
for col in mapper.primary_key
]
if primary_key is not None:
for key, value in zip(pk_attrs, primary_key):
orm.attributes.set_committed_value(
specimen,
key,
value
)
for key in pk_attrs:
if state.attrs[key].loaded_value is orm.attributes.NO_VALUE:
raise ValueError("full primary key must be present")
orm.make_transient_to_detached(specimen)
if state.key not in session.identity_map:
session.add(specimen)
return specimen
else:
return session.merge(specimen, load=False)
def manufacture_entity_criteria(entity, include_only=None, exclude=None):
"""Given a mapped instance, produce a WHERE clause.
The attributes set upon the instance will be combined to produce
a SQL expression using the mapped SQL expressions as the base
of comparison.
Values on the instance may be set as tuples in which case the
criteria will produce an IN clause. None is also acceptable as a
scalar or tuple entry, which will produce IS NULL that is properly
joined with an OR against an IN expression if appropriate.
:param entity: a mapped entity.
:param include_only: optional sequence of keys to limit which
keys are included.
:param exclude: sequence of keys to exclude
"""
state = inspect(entity)
exclude = set(exclude) if exclude is not None else set()
existing = dict(
(attr.key, attr.loaded_value)
for attr in state.attrs
if attr.loaded_value is not orm.attributes.NO_VALUE
and attr.key not in exclude
)
if include_only:
existing = dict(
(k, existing[k])
for k in set(existing).intersection(include_only)
)
return manufacture_criteria(state.mapper, existing)
def manufacture_criteria(mapped, values):
"""Given a mapper/class and a namespace of values, produce a WHERE clause.
The class should be a mapped class and the entries in the dictionary
correspond to mapped attribute names on the class.
A value may also be a tuple in which case that particular attribute
will be compared to a tuple using IN. The scalar value or
tuple can also contain None which translates to an IS NULL, that is
properly joined with OR against an IN expression if appropriate.
:param cls: a mapped class, or actual :class:`.Mapper` object.
:param values: dictionary of values.
"""
mapper = inspect(mapped)
# organize keys using mapped attribute ordering, which is deterministic
value_keys = set(values)
keys = [k for k in mapper.column_attrs.keys() if k in value_keys]
return sql.and_(*[
_sql_crit(mapper.column_attrs[key].expression, values[key])
for key in keys
])
def _sql_crit(expression, value):
"""Produce an equality expression against the given value.
This takes into account a value that is actually a collection
of values, as well as a value of None or collection that contains
None.
"""
values = utils.to_list(value, default=(None, ))
if len(values) == 1:
if values[0] is None:
return expression == sql.null()
else:
return expression == values[0]
elif _none_set.intersection(values):
return sql.or_(
expression == sql.null(),
_sql_crit(expression, set(values).difference(_none_set))
)
else:
return expression.in_(values)
def update_returning_pk(query, values, surrogate_key):
"""Perform an UPDATE, returning the primary key of the matched row.
The primary key is returned using a selection of strategies:
* if the database supports RETURNING, RETURNING is used to retrieve
the primary key values inline.
* If the database is MySQL and the entity is mapped to a single integer
primary key column, MySQL's last_insert_id() function is used
inline within the UPDATE and then upon a second SELECT to get the
value.
* Otherwise, a "refetch" strategy is used, where a given "surrogate"
key value (typically a UUID column on the entity) is used to run
a new SELECT against that UUID. This UUID is also placed into
the UPDATE query to ensure the row matches.
:param query: a Query object with existing criterion, against a single
entity.
:param values: a dictionary of values to be updated on the row.
:param surrogate_key: a tuple of (attrname, value), referring to a
UNIQUE attribute that will also match the row. This attribute is used
to retrieve the row via a SELECT when no optimized strategy exists.
:return: the primary key, returned as a tuple.
Is only returned if rows matched is one. Otherwise, CantUpdateException
is raised.
"""
entity = query.column_descriptions[0]['type']
mapper = inspect(entity).mapper
session = query.session
bind = session.connection(mapper=mapper)
if bind.dialect.implicit_returning:
pk_strategy = _pk_strategy_returning
elif bind.dialect.name == 'mysql' and \
len(mapper.primary_key) == 1 and \
isinstance(
mapper.primary_key[0].type, sqltypes.Integer):
pk_strategy = _pk_strategy_mysql_last_insert_id
else:
pk_strategy = _pk_strategy_refetch
return pk_strategy(query, mapper, values, surrogate_key)
def _assert_single_row(rows_updated):
if rows_updated == 1:
return rows_updated
elif rows_updated > 1:
raise MultiRowsMatched("%d rows matched; expected one" % rows_updated)
else:
raise NoRowsMatched("No rows matched the UPDATE")
def _pk_strategy_refetch(query, mapper, values, surrogate_key):
surrogate_key_name, surrogate_key_value = surrogate_key
surrogate_key_col = mapper.attrs[surrogate_key_name].expression
rowcount = query.\
filter(surrogate_key_col == surrogate_key_value).\
update(values, synchronize_session=False)
_assert_single_row(rowcount)
# SELECT my_table.id AS my_table_id FROM my_table
# WHERE my_table.y = ? AND my_table.z = ?
# LIMIT ? OFFSET ?
fetch_query = query.session.query(
*mapper.primary_key).filter(
surrogate_key_col == surrogate_key_value)
primary_key = fetch_query.one()
return primary_key
def _pk_strategy_returning(query, mapper, values, surrogate_key):
surrogate_key_name, surrogate_key_value = surrogate_key
surrogate_key_col = mapper.attrs[surrogate_key_name].expression
update_stmt = _update_stmt_from_query(mapper, query, values)
update_stmt = update_stmt.where(surrogate_key_col == surrogate_key_value)
update_stmt = update_stmt.returning(*mapper.primary_key)
# UPDATE my_table SET x=%(x)s, z=%(z)s WHERE my_table.y = %(y_1)s
# AND my_table.z = %(z_1)s RETURNING my_table.id
result = query.session.execute(update_stmt)
rowcount = result.rowcount
_assert_single_row(rowcount)
primary_key = tuple(result.first())
return primary_key
def _pk_strategy_mysql_last_insert_id(query, mapper, values, surrogate_key):
surrogate_key_name, surrogate_key_value = surrogate_key
surrogate_key_col = mapper.attrs[surrogate_key_name].expression
surrogate_pk_col = mapper.primary_key[0]
update_stmt = _update_stmt_from_query(mapper, query, values)
update_stmt = update_stmt.where(surrogate_key_col == surrogate_key_value)
update_stmt = update_stmt.values(
{surrogate_pk_col: sql.func.last_insert_id(surrogate_pk_col)})
# UPDATE my_table SET id=last_insert_id(my_table.id),
# x=%s, z=%s WHERE my_table.y = %s AND my_table.z = %s
result = query.session.execute(update_stmt)
rowcount = result.rowcount
_assert_single_row(rowcount)
# SELECT last_insert_id() AS last_insert_id_1
primary_key = query.session.scalar(sql.func.last_insert_id()),
return primary_key
def _update_stmt_from_query(mapper, query, values):
upd_values = dict(
(
mapper.column_attrs[key], value
) for key, value in values.items()
)
query = query.enable_eagerloads(False)
context = query._compile_context()
primary_table = context.statement.froms[0]
update_stmt = sql.update(primary_table,
context.whereclause,
upd_values)
return update_stmt
_none_set = frozenset([None])
class CantUpdateException(Exception):
pass
class NoRowsMatched(CantUpdateException):
pass
class MultiRowsMatched(CantUpdateException):
pass

1161
oslo_db/sqlalchemy/utils.py
View File

File diff suppressed because it is too large Load Diff

25
oslo_db/tests/__init__.py
View File

@ -1,25 +0,0 @@
# Copyright 2014 Rackspace
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import os
def should_run_eventlet_tests():
return bool(int(os.environ.get('TEST_EVENTLET') or '0'))
if should_run_eventlet_tests():
import eventlet
eventlet.monkey_patch()

53
oslo_db/tests/base.py
View File

@ -1,53 +0,0 @@
# Copyright 2010-2011 OpenStack Foundation
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import os
import fixtures
import testtools
_TRUE_VALUES = ('true', '1', 'yes')
# FIXME(dhellmann) Update this to use oslo.test library
class TestCase(testtools.TestCase):
"""Test case base class for all unit tests."""
def setUp(self):
"""Run before each test method to initialize test environment."""
super(TestCase, self).setUp()
test_timeout = os.environ.get('OS_TEST_TIMEOUT', 0)
try:
test_timeout = int(test_timeout)
except ValueError:
# If timeout value is invalid do not set a timeout.
test_timeout = 0
if test_timeout > 0:
self.useFixture(fixtures.Timeout(test_timeout, gentle=True))
self.useFixture(fixtures.NestedTempfile())
self.useFixture(fixtures.TempHomeDir())
if os.environ.get('OS_STDOUT_CAPTURE') in _TRUE_VALUES:
stdout = self.useFixture(fixtures.StringStream('stdout')).stream
self.useFixture(fixtures.MonkeyPatch('sys.stdout', stdout))
if os.environ.get('OS_STDERR_CAPTURE') in _TRUE_VALUES:
stderr = self.useFixture(fixtures.StringStream('stderr')).stream
self.useFixture(fixtures.MonkeyPatch('sys.stderr', stderr))
self.log_fixture = self.useFixture(fixtures.FakeLogger())

18
oslo_db/tests/sqlalchemy/__init__.py
View File

@ -1,18 +0,0 @@
# Copyright (c) 2014 OpenStack Foundation
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from oslo_db.sqlalchemy import test_base
load_tests = test_base.optimize_db_test_loader(__file__)

127
oslo_db/tests/sqlalchemy/test_async_eventlet.py
View File

@ -1,127 +0,0 @@
# Copyright (c) 2014 Rackspace Hosting
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""Unit tests for SQLAlchemy and eventlet interaction."""
import logging
import unittest2
from oslo_utils import importutils
import sqlalchemy as sa
from sqlalchemy.ext import declarative as sa_decl
from oslo_db import exception as db_exc
from oslo_db.sqlalchemy import models
from oslo_db.sqlalchemy import test_base
from oslo_db import tests
class EventletTestMixin(object):
def setUp(self):
super(EventletTestMixin, self).setUp()
BASE = sa_decl.declarative_base()
class TmpTable(BASE, models.ModelBase):
__tablename__ = 'test_async_eventlet'
id = sa.Column('id', sa.Integer, primary_key=True, nullable=False)
foo = sa.Column('foo', sa.Integer)
__table_args__ = (
sa.UniqueConstraint('foo', name='uniq_foo'),
)
self.test_table = TmpTable
TmpTable.__table__.create(self.engine)
self.addCleanup(lambda: TmpTable.__table__.drop(self.engine))
@unittest2.skipIf(not tests.should_run_eventlet_tests(),
'eventlet tests disabled unless TEST_EVENTLET=1')
def test_concurrent_transaction(self):
# Cause sqlalchemy to log executed SQL statements. Useful to
# determine exactly what and when was sent to DB.
sqla_logger = logging.getLogger('sqlalchemy.engine')
sqla_logger.setLevel(logging.INFO)
self.addCleanup(sqla_logger.setLevel, logging.NOTSET)
def operate_on_row(name, ready=None, proceed=None):
logging.debug('%s starting', name)
_session = self.sessionmaker()
with _session.begin():
logging.debug('%s ready', name)
# Modify the same row, inside transaction
tbl = self.test_table()
tbl.update({'foo': 10})
tbl.save(_session)
if ready is not None:
ready.send()
if proceed is not None:
logging.debug('%s waiting to proceed', name)
proceed.wait()
logging.debug('%s exiting transaction', name)
logging.debug('%s terminating', name)
return True
eventlet = importutils.try_import('eventlet')
if eventlet is None:
return self.skip('eventlet is required for this test')
a_ready = eventlet.event.Event()
a_proceed = eventlet.event.Event()
b_proceed = eventlet.event.Event()
# thread A opens transaction
logging.debug('spawning A')
a = eventlet.spawn(operate_on_row, 'A',
ready=a_ready, proceed=a_proceed)
logging.debug('waiting for A to enter transaction')
a_ready.wait()
# thread B opens transaction on same row
logging.debug('spawning B')
b = eventlet.spawn(operate_on_row, 'B',
proceed=b_proceed)
logging.debug('waiting for B to (attempt to) enter transaction')
eventlet.sleep(1) # should(?) advance B to blocking on transaction
# While B is still blocked, A should be able to proceed
a_proceed.send()
# Will block forever(*) if DB library isn't reentrant.
# (*) Until some form of timeout/deadlock detection kicks in.
# This is the key test that async is working. If this hangs
# (or raises a timeout/deadlock exception), then you have failed
# this test.
self.assertTrue(a.wait())
b_proceed.send()
# If everything proceeded without blocking, B will throw a
# "duplicate entry" exception when it tries to insert the same row
self.assertRaises(db_exc.DBDuplicateEntry, b.wait)
# Note that sqlite fails the above concurrency tests, and is not
# mentioned below.
# ie: This file performs no tests by default.
class MySQLEventletTestCase(EventletTestMixin,
test_base.MySQLOpportunisticTestCase):
pass
class PostgreSQLEventletTestCase(EventletTestMixin,
test_base.PostgreSQLOpportunisticTestCase):
pass

2051
oslo_db/tests/sqlalchemy/test_enginefacade.py
View File

File diff suppressed because it is too large Load Diff

1361
oslo_db/tests/sqlalchemy/test_exc_filters.py
View File

File diff suppressed because it is too large Load Diff

83
oslo_db/tests/sqlalchemy/test_fixtures.py
View File

@ -1,83 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import mock
from oslo_db.sqlalchemy import provision
from oslo_db.sqlalchemy import test_base
from oslotest import base as oslo_test_base
class BackendSkipTest(oslo_test_base.BaseTestCase):
def test_skip_no_dbapi(self):
class FakeDatabaseOpportunisticFixture(test_base.DbFixture):
DRIVER = 'postgresql'
class SomeTest(test_base.DbTestCase):
FIXTURE = FakeDatabaseOpportunisticFixture
def runTest(self):
pass
st = SomeTest()
# patch in replacement lookup dictionaries to avoid
# leaking from/to other tests
with mock.patch(
"oslo_db.sqlalchemy.provision."
"Backend.backends_by_database_type", {
"postgresql":
provision.Backend("postgresql", "postgresql://")}):
st._database_resources = {}
st._db_not_available = {}
st._schema_resources = {}
with mock.patch(
"sqlalchemy.create_engine",
mock.Mock(side_effect=ImportError())):
self.assertEqual([], st.resources)
ex = self.assertRaises(
self.skipException,
st.setUp
)
self.assertEqual(
"Backend 'postgresql' is unavailable: No DBAPI installed",
str(ex)
)
def test_skip_no_such_backend(self):
class FakeDatabaseOpportunisticFixture(test_base.DbFixture):
DRIVER = 'postgresql+nosuchdbapi'
class SomeTest(test_base.DbTestCase):
FIXTURE = FakeDatabaseOpportunisticFixture
def runTest(self):
pass
st = SomeTest()
ex = self.assertRaises(
self.skipException,
st.setUp
)
self.assertEqual(
"Backend 'postgresql+nosuchdbapi' is unavailable: No such backend",
str(ex)
)

348
oslo_db/tests/sqlalchemy/test_migrate_cli.py
View File

@ -1,348 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import alembic
import mock
from oslotest import base as test_base
import sqlalchemy
from oslo_db import exception
from oslo_db.sqlalchemy.migration_cli import ext_alembic
from oslo_db.sqlalchemy.migration_cli import ext_migrate
from oslo_db.sqlalchemy.migration_cli import manager
class MockWithCmp(mock.MagicMock):
order = 0
def __init__(self, *args, **kwargs):
super(MockWithCmp, self).__init__(*args, **kwargs)
self.__lt__ = lambda self, other: self.order < other.order
@mock.patch(('oslo_db.sqlalchemy.migration_cli.'
'ext_alembic.alembic.command'))
class TestAlembicExtension(test_base.BaseTestCase):
def setUp(self):
self.migration_config = {'alembic_ini_path': '.',
'db_url': 'sqlite://'}
self.engine = sqlalchemy.create_engine(self.migration_config['db_url'])
self.alembic = ext_alembic.AlembicExtension(
self.engine, self.migration_config)
super(TestAlembicExtension, self).setUp()
def test_check_enabled_true(self, command):
"""Check enabled returns True
Verifies that enabled returns True on non empty
alembic_ini_path conf variable
"""
self.assertTrue(self.alembic.enabled)
def test_check_enabled_false(self, command):
"""Check enabled returns False
Verifies enabled returns False on empty alembic_ini_path variable
"""
self.migration_config['alembic_ini_path'] = ''
alembic = ext_alembic.AlembicExtension(
self.engine, self.migration_config)
self.assertFalse(alembic.enabled)
def test_upgrade_none(self, command):
self.alembic.upgrade(None)
command.upgrade.assert_called_once_with(self.alembic.config, 'head')
def test_upgrade_normal(self, command):
self.alembic.upgrade('131daa')
command.upgrade.assert_called_once_with(self.alembic.config, '131daa')
def test_downgrade_none(self, command):
self.alembic.downgrade(None)
command.downgrade.assert_called_once_with(self.alembic.config, 'base')
def test_downgrade_int(self, command):
self.alembic.downgrade(111)
command.downgrade.assert_called_once_with(self.alembic.config, 'base')
def test_downgrade_normal(self, command):
self.alembic.downgrade('131daa')
command.downgrade.assert_called_once_with(
self.alembic.config, '131daa')
def test_revision(self, command):
self.alembic.revision(message='test', autogenerate=True)
command.revision.assert_called_once_with(
self.alembic.config, message='test', autogenerate=True)
def test_stamp(self, command):
self.alembic.stamp('stamp')
command.stamp.assert_called_once_with(
self.alembic.config, revision='stamp')
def test_version(self, command):
version = self.alembic.version()
self.assertIsNone(version)
def test_has_revision(self, command):
with mock.patch(('oslo_db.sqlalchemy.migration_cli.'
'ext_alembic.alembic_script')) as mocked:
self.alembic.config.get_main_option = mock.Mock()
# since alembic_script is mocked and no exception is raised, call
# will result in success
self.assertIs(True, self.alembic.has_revision('test'))
self.alembic.config.get_main_option.assert_called_once_with(
'script_location')
mocked.ScriptDirectory().get_revision.assert_called_once_with(
'test')
self.assertIs(True, self.alembic.has_revision(None))
self.assertIs(True, self.alembic.has_revision('head'))
# relative revision, should be True for alembic
self.assertIs(True, self.alembic.has_revision('+1'))
def test_has_revision_negative(self, command):
with mock.patch(('oslo_db.sqlalchemy.migration_cli.'
'ext_alembic.alembic_script')) as mocked:
mocked.ScriptDirectory().get_revision.side_effect = (
alembic.util.CommandError)
self.alembic.config.get_main_option = mock.Mock()
# exception is raised, the call should be false
self.assertIs(False, self.alembic.has_revision('test'))
self.alembic.config.get_main_option.assert_called_once_with(
'script_location')
mocked.ScriptDirectory().get_revision.assert_called_once_with(
'test')
@mock.patch(('oslo_db.sqlalchemy.migration_cli.'
'ext_migrate.migration'))
class TestMigrateExtension(test_base.BaseTestCase):
def setUp(self):
self.migration_config = {'migration_repo_path': '.',
'db_url': 'sqlite://'}
self.engine = sqlalchemy.create_engine(self.migration_config['db_url'])
self.migrate = ext_migrate.MigrateExtension(
self.engine, self.migration_config)
super(TestMigrateExtension, self).setUp()
def test_check_enabled_true(self, migration):
self.assertTrue(self.migrate.enabled)
def test_check_enabled_false(self, migration):
self.migration_config['migration_repo_path'] = ''
migrate = ext_migrate.MigrateExtension(
self.engine, self.migration_config)
self.assertFalse(migrate.enabled)
def test_upgrade_head(self, migration):
self.migrate.upgrade('head')
migration.db_sync.assert_called_once_with(
self.migrate.engine, self.migrate.repository, None, init_version=0)
def test_upgrade_normal(self, migration):
self.migrate.upgrade(111)
migration.db_sync.assert_called_once_with(
mock.ANY, self.migrate.repository, 111, init_version=0)
def test_downgrade_init_version_from_base(self, migration):
self.migrate.downgrade('base')
migration.db_sync.assert_called_once_with(
self.migrate.engine, self.migrate.repository, mock.ANY,
init_version=mock.ANY)
def test_downgrade_init_version_from_none(self, migration):
self.migrate.downgrade(None)
migration.db_sync.assert_called_once_with(
self.migrate.engine, self.migrate.repository, mock.ANY,
init_version=mock.ANY)
def test_downgrade_normal(self, migration):
self.migrate.downgrade(101)
migration.db_sync.assert_called_once_with(
self.migrate.engine, self.migrate.repository, 101, init_version=0)
def test_version(self, migration):
self.migrate.version()
migration.db_version.assert_called_once_with(
self.migrate.engine, self.migrate.repository, init_version=0)
def test_change_init_version(self, migration):
self.migration_config['init_version'] = 101
migrate = ext_migrate.MigrateExtension(
self.engine, self.migration_config)
migrate.downgrade(None)
migration.db_sync.assert_called_once_with(
migrate.engine,
self.migrate.repository,
self.migration_config['init_version'],
init_version=self.migration_config['init_version'])
def test_has_revision(self, command):
with mock.patch(('oslo_db.sqlalchemy.migration_cli.'
'ext_migrate.migrate_version')) as mocked:
self.migrate.has_revision('test')
mocked.Collection().version.assert_called_once_with('test')
# tip of the branch should always be True
self.assertIs(True, self.migrate.has_revision(None))
def test_has_revision_negative(self, command):
with mock.patch(('oslo_db.sqlalchemy.migration_cli.'
'ext_migrate.migrate_version')) as mocked:
mocked.Collection().version.side_effect = ValueError
self.assertIs(False, self.migrate.has_revision('test'))
mocked.Collection().version.assert_called_once_with('test')
# relative revision, should be False for migrate
self.assertIs(False, self.migrate.has_revision('+1'))
class TestMigrationManager(test_base.BaseTestCase):
def setUp(self):
self.migration_config = {'alembic_ini_path': '.',
'migrate_repo_path': '.',
'db_url': 'sqlite://'}
engine = sqlalchemy.create_engine(self.migration_config['db_url'])
self.migration_manager = manager.MigrationManager(
self.migration_config, engine)
self.ext = mock.Mock()
self.ext.obj.version = mock.Mock(return_value=0)
self.migration_manager._manager.extensions = [self.ext]
super(TestMigrationManager, self).setUp()
def test_manager_update(self):
self.migration_manager.upgrade('head')
self.ext.obj.upgrade.assert_called_once_with('head')
def test_manager_update_revision_none(self):
self.migration_manager.upgrade(None)
self.ext.obj.upgrade.assert_called_once_with(None)
def test_downgrade_normal_revision(self):
self.migration_manager.downgrade('111abcd')
self.ext.obj.downgrade.assert_called_once_with('111abcd')
def test_version(self):
self.migration_manager.version()
self.ext.obj.version.assert_called_once_with()
def test_version_return_value(self):
version = self.migration_manager.version()
self.assertEqual(0, version)
def test_revision_message_autogenerate(self):
self.migration_manager.revision('test', True)
self.ext.obj.revision.assert_called_once_with('test', True)
def test_revision_only_message(self):
self.migration_manager.revision('test', False)
self.ext.obj.revision.assert_called_once_with('test', False)
def test_stamp(self):
self.migration_manager.stamp('stamp')
self.ext.obj.stamp.assert_called_once_with('stamp')
def test_wrong_config(self):
err = self.assertRaises(ValueError,
manager.MigrationManager,
{'wrong_key': 'sqlite://'})
self.assertEqual('Either database url or engine must be provided.',
err.args[0])
class TestMigrationMultipleExtensions(test_base.BaseTestCase):
def setUp(self):
self.migration_config = {'alembic_ini_path': '.',
'migrate_repo_path': '.',
'db_url': 'sqlite://'}
engine = sqlalchemy.create_engine(self.migration_config['db_url'])
self.migration_manager = manager.MigrationManager(
self.migration_config, engine)
self.first_ext = MockWithCmp()
self.first_ext.obj.order = 1
self.first_ext.obj.upgrade.return_value = 100
self.first_ext.obj.downgrade.return_value = 0
self.second_ext = MockWithCmp()
self.second_ext.obj.order = 2
self.second_ext.obj.upgrade.return_value = 200
self.second_ext.obj.downgrade.return_value = 100
self.migration_manager._manager.extensions = [self.first_ext,
self.second_ext]
super(TestMigrationMultipleExtensions, self).setUp()
def test_upgrade_right_order(self):
results = self.migration_manager.upgrade(None)
self.assertEqual([100, 200], results)
def test_downgrade_right_order(self):
results = self.migration_manager.downgrade(None)
self.assertEqual([100, 0], results)
def test_upgrade_does_not_go_too_far(self):
self.first_ext.obj.has_revision.return_value = True
self.second_ext.obj.has_revision.return_value = False
self.second_ext.obj.upgrade.side_effect = AssertionError(
'this method should not have been called')
results = self.migration_manager.upgrade(100)
self.assertEqual([100], results)
def test_downgrade_does_not_go_too_far(self):
self.second_ext.obj.has_revision.return_value = True
self.first_ext.obj.has_revision.return_value = False
self.first_ext.obj.downgrade.side_effect = AssertionError(
'this method should not have been called')
results = self.migration_manager.downgrade(100)
self.assertEqual([100], results)
def test_upgrade_checks_rev_existence(self):
self.first_ext.obj.has_revision.return_value = False
self.second_ext.obj.has_revision.return_value = False
# upgrade to a specific non-existent revision should fail
self.assertRaises(exception.DBMigrationError,
self.migration_manager.upgrade, 100)
# upgrade to the "head" should succeed
self.assertEqual([100, 200], self.migration_manager.upgrade(None))
# let's assume the second ext has the revision, upgrade should succeed
self.second_ext.obj.has_revision.return_value = True
self.assertEqual([100, 200], self.migration_manager.upgrade(200))
# upgrade to the "head" should still succeed
self.assertEqual([100, 200], self.migration_manager.upgrade(None))
def test_downgrade_checks_rev_existence(self):
self.first_ext.obj.has_revision.return_value = False
self.second_ext.obj.has_revision.return_value = False
# upgrade to a specific non-existent revision should fail
self.assertRaises(exception.DBMigrationError,
self.migration_manager.downgrade, 100)
# downgrade to the "base" should succeed
self.assertEqual([100, 0], self.migration_manager.downgrade(None))
# let's assume the second ext has the revision, downgrade should
# succeed
self.first_ext.obj.has_revision.return_value = True
self.assertEqual([100, 0], self.migration_manager.downgrade(200))
# downgrade to the "base" should still succeed
self.assertEqual([100, 0], self.migration_manager.downgrade(None))
self.assertEqual([100, 0], self.migration_manager.downgrade('base'))

235
oslo_db/tests/sqlalchemy/test_migration_common.py
View File

@ -1,235 +0,0 @@
# Copyright 2013 Mirantis Inc.
# All Rights Reserved
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
import os
import tempfile
from migrate import exceptions as migrate_exception
from migrate.versioning import api as versioning_api
import mock
import sqlalchemy
from oslo_db import exception as db_exception
from oslo_db.sqlalchemy import migration
from oslo_db.sqlalchemy import test_base
from oslo_db.tests import utils as test_utils
class TestMigrationCommon(test_base.DbTestCase):
def setUp(self):
super(TestMigrationCommon, self).setUp()
migration._REPOSITORY = None
self.path = tempfile.mkdtemp('test_migration')
self.path1 = tempfile.mkdtemp('test_migration')
self.return_value = '/home/openstack/migrations'
self.return_value1 = '/home/extension/migrations'
self.init_version = 1
self.test_version = 123
self.patcher_repo = mock.patch.object(migration, 'Repository')
self.repository = self.patcher_repo.start()
self.repository.side_effect = [self.return_value, self.return_value1]
self.mock_api_db = mock.patch.object(versioning_api, 'db_version')
self.mock_api_db_version = self.mock_api_db.start()
self.mock_api_db_version.return_value = self.test_version
def tearDown(self):
os.rmdir(self.path)
self.mock_api_db.stop()
self.patcher_repo.stop()
super(TestMigrationCommon, self).tearDown()
def test_find_migrate_repo_path_not_found(self):
self.assertRaises(
db_exception.DBMigrationError,
migration._find_migrate_repo,
"/foo/bar/",
)
self.assertIsNone(migration._REPOSITORY)
def test_find_migrate_repo_called_once(self):
my_repository = migration._find_migrate_repo(self.path)
self.repository.assert_called_once_with(self.path)
self.assertEqual(self.return_value, my_repository)
def test_find_migrate_repo_called_few_times(self):
repo1 = migration._find_migrate_repo(self.path)
repo2 = migration._find_migrate_repo(self.path1)
self.assertNotEqual(repo1, repo2)
def test_db_version_control(self):
with test_utils.nested(
mock.patch.object(migration, '_find_migrate_repo'),
mock.patch.object(versioning_api, 'version_control'),
) as (mock_find_repo, mock_version_control):
mock_find_repo.return_value = self.return_value
version = migration.db_version_control(
self.engine, self.path, self.test_version)
self.assertEqual(self.test_version, version)
mock_version_control.assert_called_once_with(
self.engine, self.return_value, self.test_version)
@mock.patch.object(migration, '_find_migrate_repo')
@mock.patch.object(versioning_api, 'version_control')
def test_db_version_control_version_less_than_actual_version(
self, mock_version_control, mock_find_repo):
mock_find_repo.return_value = self.return_value
mock_version_control.side_effect = (migrate_exception.
DatabaseAlreadyControlledError)
self.assertRaises(db_exception.DBMigrationError,
migration.db_version_control, self.engine,
self.path, self.test_version - 1)
@mock.patch.object(migration, '_find_migrate_repo')
@mock.patch.object(versioning_api, 'version_control')
def test_db_version_control_version_greater_than_actual_version(
self, mock_version_control, mock_find_repo):
mock_find_repo.return_value = self.return_value
mock_version_control.side_effect = (migrate_exception.
InvalidVersionError)
self.assertRaises(db_exception.DBMigrationError,
migration.db_version_control, self.engine,
self.path, self.test_version + 1)
def test_db_version_return(self):
ret_val = migration.db_version(self.engine, self.path,
self.init_version)
self.assertEqual(self.test_version, ret_val)
def test_db_version_raise_not_controlled_error_first(self):
with mock.patch.object(migration, 'db_version_control') as mock_ver:
self.mock_api_db_version.side_effect = [
migrate_exception.DatabaseNotControlledError('oups'),
self.test_version]
ret_val = migration.db_version(self.engine, self.path,
self.init_version)
self.assertEqual(self.test_version, ret_val)
mock_ver.assert_called_once_with(self.engine, self.path,
version=self.init_version)
def test_db_version_raise_not_controlled_error_tables(self):
with mock.patch.object(sqlalchemy, 'MetaData') as mock_meta:
self.mock_api_db_version.side_effect = \
migrate_exception.DatabaseNotControlledError('oups')
my_meta = mock.MagicMock()
my_meta.tables = {'a': 1, 'b': 2}
mock_meta.return_value = my_meta
self.assertRaises(
db_exception.DBMigrationError, migration.db_version,
self.engine, self.path, self.init_version)
@mock.patch.object(versioning_api, 'version_control')
def test_db_version_raise_not_controlled_error_no_tables(self, mock_vc):
with mock.patch.object(sqlalchemy, 'MetaData') as mock_meta:
self.mock_api_db_version.side_effect = (
migrate_exception.DatabaseNotControlledError('oups'),
self.init_version)
my_meta = mock.MagicMock()
my_meta.tables = {}
mock_meta.return_value = my_meta
migration.db_version(self.engine, self.path, self.init_version)
mock_vc.assert_called_once_with(self.engine, self.return_value1,
self.init_version)
def test_db_sync_wrong_version(self):
self.assertRaises(db_exception.DBMigrationError,
migration.db_sync, self.engine, self.path, 'foo')
def test_db_sync_upgrade(self):
init_ver = 55
with test_utils.nested(
mock.patch.object(migration, '_find_migrate_repo'),
mock.patch.object(versioning_api, 'upgrade')
) as (mock_find_repo, mock_upgrade):
mock_find_repo.return_value = self.return_value
self.mock_api_db_version.return_value = self.test_version - 1
migration.db_sync(self.engine, self.path, self.test_version,
init_ver)
mock_upgrade.assert_called_once_with(
self.engine, self.return_value, self.test_version)
def test_db_sync_downgrade(self):
with test_utils.nested(
mock.patch.object(migration, '_find_migrate_repo'),
mock.patch.object(versioning_api, 'downgrade')
) as (mock_find_repo, mock_downgrade):
mock_find_repo.return_value = self.return_value
self.mock_api_db_version.return_value = self.test_version + 1
migration.db_sync(self.engine, self.path, self.test_version)
mock_downgrade.assert_called_once_with(
self.engine, self.return_value, self.test_version)
def test_db_sync_sanity_called(self):
with test_utils.nested(
mock.patch.object(migration, '_find_migrate_repo'),
mock.patch.object(migration, '_db_schema_sanity_check'),
mock.patch.object(versioning_api, 'downgrade')
) as (mock_find_repo, mock_sanity, mock_downgrade):
mock_find_repo.return_value = self.return_value
migration.db_sync(self.engine, self.path, self.test_version)
self.assertEqual([mock.call(self.engine), mock.call(self.engine)],
mock_sanity.call_args_list)
def test_db_sync_sanity_skipped(self):
with test_utils.nested(
mock.patch.object(migration, '_find_migrate_repo'),
mock.patch.object(migration, '_db_schema_sanity_check'),
mock.patch.object(versioning_api, 'downgrade')
) as (mock_find_repo, mock_sanity, mock_downgrade):
mock_find_repo.return_value = self.return_value
migration.db_sync(self.engine, self.path, self.test_version,
sanity_check=False)
self.assertFalse(mock_sanity.called)
def test_db_sanity_table_not_utf8(self):
with mock.patch.object(self, 'engine') as mock_eng:
type(mock_eng).name = mock.PropertyMock(return_value='mysql')
mock_eng.execute.return_value = [['table_A', 'latin1'],
['table_B', 'latin1']]
self.assertRaises(ValueError, migration._db_schema_sanity_check,
mock_eng)
def test_db_sanity_table_not_utf8_exclude_migrate_tables(self):
with mock.patch.object(self, 'engine') as mock_eng:
type(mock_eng).name = mock.PropertyMock(return_value='mysql')
# NOTE(morganfainberg): Check both lower and upper case versions
# of the migration table names (validate case insensitivity in
# the sanity check.
mock_eng.execute.return_value = [['migrate_version', 'latin1'],
['alembic_version', 'latin1'],
['MIGRATE_VERSION', 'latin1'],
['ALEMBIC_VERSION', 'latin1']]
migration._db_schema_sanity_check(mock_eng)

566
oslo_db/tests/sqlalchemy/test_migrations.py
View File

@ -1,566 +0,0 @@
# Copyright 2010-2011 OpenStack Foundation
# Copyright 2012-2013 IBM Corp.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import fixtures
from migrate.versioning import api as versioning_api
import mock
from oslotest import base as test
import six
import sqlalchemy as sa
import sqlalchemy.ext.declarative as sa_decl
from oslo_db import exception as exc
from oslo_db.sqlalchemy import test_base
from oslo_db.sqlalchemy import test_migrations as migrate
class TestWalkVersions(test.BaseTestCase, migrate.WalkVersionsMixin):
migration_api = mock.MagicMock()
REPOSITORY = mock.MagicMock()
engine = mock.MagicMock()
INIT_VERSION = versioning_api.VerNum(4)
@property
def migrate_engine(self):
return self.engine
def test_migrate_up(self):
self.migration_api.db_version.return_value = 141
self.migrate_up(141)
self.migration_api.upgrade.assert_called_with(
self.engine, self.REPOSITORY, 141)
self.migration_api.db_version.assert_called_with(
self.engine, self.REPOSITORY)
@staticmethod
def _fake_upgrade_boom(*args, **kwargs):
raise exc.DBMigrationError("boom")
def test_migrate_up_fail(self):
version = 141
self.migration_api.db_version.return_value = version
expected_output = (u"Failed to migrate to version %(version)s on "
"engine %(engine)s\n" %
{'version': version, 'engine': self.engine})
with mock.patch.object(self.migration_api,
'upgrade',
side_effect=self._fake_upgrade_boom):
log = self.useFixture(fixtures.FakeLogger())
self.assertRaises(exc.DBMigrationError, self.migrate_up, version)
self.assertEqual(expected_output, log.output)
def test_migrate_up_with_data(self):
test_value = {"a": 1, "b": 2}
self.migration_api.db_version.return_value = 141
self._pre_upgrade_141 = mock.MagicMock()
self._pre_upgrade_141.return_value = test_value
self._check_141 = mock.MagicMock()
self.migrate_up(141, True)
self._pre_upgrade_141.assert_called_with(self.engine)
self._check_141.assert_called_with(self.engine, test_value)
def test_migrate_down(self):
self.migration_api.db_version.return_value = 42
self.assertTrue(self.migrate_down(42))
self.migration_api.db_version.assert_called_with(
self.engine, self.REPOSITORY)
def test_migrate_down_not_implemented(self):
with mock.patch.object(self.migration_api,
'downgrade',
side_effect=NotImplementedError):
self.assertFalse(self.migrate_down(self.engine, 42))
def test_migrate_down_with_data(self):
self._post_downgrade_043 = mock.MagicMock()
self.migration_api.db_version.return_value = 42
self.migrate_down(42, True)
self._post_downgrade_043.assert_called_with(self.engine)
@mock.patch.object(migrate.WalkVersionsMixin, 'migrate_up')
@mock.patch.object(migrate.WalkVersionsMixin, 'migrate_down')
def test_walk_versions_all_default(self, migrate_up, migrate_down):
self.REPOSITORY.latest = versioning_api.VerNum(20)
self.migration_api.db_version.return_value = self.INIT_VERSION
self.walk_versions()
self.migration_api.version_control.assert_called_with(
self.engine, self.REPOSITORY, self.INIT_VERSION)
self.migration_api.db_version.assert_called_with(
self.engine, self.REPOSITORY)
versions = range(int(self.INIT_VERSION) + 1,
int(self.REPOSITORY.latest) + 1)
upgraded = [mock.call(v, with_data=True)
for v in versions]
self.assertEqual(upgraded, self.migrate_up.call_args_list)
downgraded = [mock.call(v - 1) for v in reversed(versions)]
self.assertEqual(downgraded, self.migrate_down.call_args_list)
@mock.patch.object(migrate.WalkVersionsMixin, 'migrate_up')
@mock.patch.object(migrate.WalkVersionsMixin, 'migrate_down')
def test_walk_versions_all_true(self, migrate_up, migrate_down):
self.REPOSITORY.latest = versioning_api.VerNum(20)
self.migration_api.db_version.return_value = self.INIT_VERSION
self.walk_versions(snake_walk=True, downgrade=True)
versions = range(int(self.INIT_VERSION) + 1,
int(self.REPOSITORY.latest) + 1)
upgraded = []
for v in versions:
upgraded.append(mock.call(v, with_data=True))
upgraded.append(mock.call(v))
upgraded.extend([mock.call(v) for v in reversed(versions)])
self.assertEqual(upgraded, self.migrate_up.call_args_list)
downgraded_1 = [mock.call(v - 1, with_data=True) for v in versions]
downgraded_2 = []
for v in reversed(versions):
downgraded_2.append(mock.call(v - 1))
downgraded_2.append(mock.call(v - 1))
downgraded = downgraded_1 + downgraded_2
self.assertEqual(downgraded, self.migrate_down.call_args_list)
@mock.patch.object(migrate.WalkVersionsMixin, 'migrate_up')
@mock.patch.object(migrate.WalkVersionsMixin, 'migrate_down')
def test_walk_versions_true_false(self, migrate_up, migrate_down):
self.REPOSITORY.latest = versioning_api.VerNum(20)
self.migration_api.db_version.return_value = self.INIT_VERSION
self.walk_versions(snake_walk=True, downgrade=False)
versions = range(int(self.INIT_VERSION) + 1,
int(self.REPOSITORY.latest) + 1)
upgraded = []
for v in versions:
upgraded.append(mock.call(v, with_data=True))
upgraded.append(mock.call(v))
self.assertEqual(upgraded, self.migrate_up.call_args_list)
downgraded = [mock.call(v - 1, with_data=True) for v in versions]
self.assertEqual(downgraded, self.migrate_down.call_args_list)
@mock.patch.object(migrate.WalkVersionsMixin, 'migrate_up')
@mock.patch.object(migrate.WalkVersionsMixin, 'migrate_down')
def test_walk_versions_all_false(self, migrate_up, migrate_down):
self.REPOSITORY.latest = versioning_api.VerNum(20)
self.migration_api.db_version.return_value = self.INIT_VERSION
self.walk_versions(snake_walk=False, downgrade=False)
versions = range(int(self.INIT_VERSION) + 1,
int(self.REPOSITORY.latest) + 1)
upgraded = [mock.call(v, with_data=True) for v in versions]
self.assertEqual(upgraded, self.migrate_up.call_args_list)
class ModelsMigrationSyncMixin(test_base.DbTestCase):
def setUp(self):
super(ModelsMigrationSyncMixin, self).setUp()
self.metadata = sa.MetaData()
self.metadata_migrations = sa.MetaData()
sa.Table(
'testtbl', self.metadata_migrations,
sa.Column('id', sa.Integer, primary_key=True),
sa.Column('spam', sa.String(10), nullable=False),
sa.Column('eggs', sa.DateTime),
sa.Column('foo', sa.Boolean,
server_default=sa.sql.expression.true()),
sa.Column('bool_wo_default', sa.Boolean),
sa.Column('bar', sa.Numeric(10, 5)),
sa.Column('defaulttest', sa.Integer, server_default='5'),
sa.Column('defaulttest2', sa.String(8), server_default=''),
sa.Column('defaulttest3', sa.String(5), server_default="test"),
sa.Column('defaulttest4', sa.Enum('first', 'second',
name='testenum'),
server_default="first"),
sa.Column('variant', sa.BigInteger()),
sa.Column('variant2', sa.BigInteger(), server_default='0'),
sa.Column('fk_check', sa.String(36), nullable=False),
sa.UniqueConstraint('spam', 'eggs', name='uniq_cons'),
)
BASE = sa_decl.declarative_base(metadata=self.metadata)
class TestModel(BASE):
__tablename__ = 'testtbl'
__table_args__ = (
sa.UniqueConstraint('spam', 'eggs', name='uniq_cons'),
)
id = sa.Column('id', sa.Integer, primary_key=True)
spam = sa.Column('spam', sa.String(10), nullable=False)
eggs = sa.Column('eggs', sa.DateTime)
foo = sa.Column('foo', sa.Boolean,
server_default=sa.sql.expression.true())
fk_check = sa.Column('fk_check', sa.String(36), nullable=False)
bool_wo_default = sa.Column('bool_wo_default', sa.Boolean)
defaulttest = sa.Column('defaulttest',
sa.Integer, server_default='5')
defaulttest2 = sa.Column('defaulttest2', sa.String(8),
server_default='')
defaulttest3 = sa.Column('defaulttest3', sa.String(5),
server_default="test")
defaulttest4 = sa.Column('defaulttest4', sa.Enum('first', 'second',
name='testenum'),
server_default="first")
variant = sa.Column(sa.BigInteger().with_variant(
sa.Integer(), 'sqlite'))
variant2 = sa.Column(sa.BigInteger().with_variant(
sa.Integer(), 'sqlite'), server_default='0')
bar = sa.Column('bar', sa.Numeric(10, 5))
class ModelThatShouldNotBeCompared(BASE):
__tablename__ = 'testtbl2'
id = sa.Column('id', sa.Integer, primary_key=True)
spam = sa.Column('spam', sa.String(10), nullable=False)
def get_metadata(self):
return self.metadata
def get_engine(self):
return self.engine
def db_sync(self, engine):
self.metadata_migrations.create_all(bind=engine)
def include_object(self, object_, name, type_, reflected, compare_to):
if type_ == 'table':
return name == 'testtbl'
else:
return True
def _test_models_not_sync_filtered(self):
self.metadata_migrations.clear()
sa.Table(
'table', self.metadata_migrations,
sa.Column('fk_check', sa.String(36), nullable=False),
sa.PrimaryKeyConstraint('fk_check'),
mysql_engine='InnoDB'
)
sa.Table(
'testtbl', self.metadata_migrations,
sa.Column('id', sa.Integer, primary_key=True),
sa.Column('spam', sa.String(8), nullable=True),
sa.Column('eggs', sa.DateTime),
sa.Column('foo', sa.Boolean,
server_default=sa.sql.expression.false()),
sa.Column('bool_wo_default', sa.Boolean, unique=True),
sa.Column('bar', sa.BigInteger),
sa.Column('defaulttest', sa.Integer, server_default='7'),
sa.Column('defaulttest2', sa.String(8), server_default=''),
sa.Column('defaulttest3', sa.String(5), server_default="fake"),
sa.Column('defaulttest4',
sa.Enum('first', 'second', name='testenum'),
server_default="first"),
sa.Column('fk_check', sa.String(36), nullable=False),
sa.UniqueConstraint('spam', 'foo', name='uniq_cons'),
sa.ForeignKeyConstraint(['fk_check'], ['table.fk_check']),
mysql_engine='InnoDB'
)
with mock.patch.object(self, 'filter_metadata_diff') as filter_mock:
def filter_diffs(diffs):
# test filter returning only constraint related diffs
return [
diff
for diff in diffs
if 'constraint' in diff[0]
]
filter_mock.side_effect = filter_diffs
msg = six.text_type(self.assertRaises(AssertionError,
self.test_models_sync))
self.assertNotIn('defaulttest', msg)
self.assertNotIn('defaulttest3', msg)
self.assertNotIn('remove_fk', msg)
self.assertIn('constraint', msg)
def _test_models_not_sync(self):
self.metadata_migrations.clear()
sa.Table(
'table', self.metadata_migrations,
sa.Column('fk_check', sa.String(36), nullable=False),
sa.PrimaryKeyConstraint('fk_check'),
mysql_engine='InnoDB'
)
sa.Table(
'testtbl', self.metadata_migrations,
sa.Column('id', sa.Integer, primary_key=True),
sa.Column('spam', sa.String(8), nullable=True),
sa.Column('eggs', sa.DateTime),
sa.Column('foo', sa.Boolean,
server_default=sa.sql.expression.false()),
sa.Column('bool_wo_default', sa.Boolean, unique=True),
sa.Column('bar', sa.BigInteger),
sa.Column('defaulttest', sa.Integer, server_default='7'),
sa.Column('defaulttest2', sa.String(8), server_default=''),
sa.Column('defaulttest3', sa.String(5), server_default="fake"),
sa.Column('defaulttest4',
sa.Enum('first', 'second', name='testenum'),
server_default="first"),
sa.Column('variant', sa.String(10)),
sa.Column('fk_check', sa.String(36), nullable=False),
sa.UniqueConstraint('spam', 'foo', name='uniq_cons'),
sa.ForeignKeyConstraint(['fk_check'], ['table.fk_check']),
mysql_engine='InnoDB'
)
msg = six.text_type(self.assertRaises(AssertionError,
self.test_models_sync))
# NOTE(I159): Check mentioning of the table and columns.
# The log is invalid json, so we can't parse it and check it for
# full compliance. We have no guarantee of the log items ordering,
# so we can't use regexp.
self.assertTrue(msg.startswith(
'Models and migration scripts aren\'t in sync:'))
self.assertIn('testtbl', msg)
self.assertIn('spam', msg)
self.assertIn('eggs', msg) # test that the unique constraint is added
self.assertIn('foo', msg)
self.assertIn('bar', msg)
self.assertIn('bool_wo_default', msg)
self.assertIn('defaulttest', msg)
self.assertIn('defaulttest3', msg)
self.assertIn('remove_fk', msg)
self.assertIn('variant', msg)
class ModelsMigrationsSyncMysql(ModelsMigrationSyncMixin,
migrate.ModelsMigrationsSync,
test_base.MySQLOpportunisticTestCase):
def test_models_not_sync(self):
self._test_models_not_sync()
def test_models_not_sync_filtered(self):
self._test_models_not_sync_filtered()
class ModelsMigrationsSyncPsql(ModelsMigrationSyncMixin,
migrate.ModelsMigrationsSync,
test_base.PostgreSQLOpportunisticTestCase):
def test_models_not_sync(self):
self._test_models_not_sync()
def test_models_not_sync_filtered(self):
self._test_models_not_sync_filtered()
class TestOldCheckForeignKeys(test_base.DbTestCase):
def setUp(self):
super(TestOldCheckForeignKeys, self).setUp()
test = self
class MigrateSync(migrate.ModelsMigrationsSync):
def get_engine(self):
return test.engine
def get_metadata(self):
return test.metadata
def db_sync(self):
raise NotImplementedError()
self.migration_sync = MigrateSync()
def _fk_added_fixture(self):
self.metadata = sa.MetaData()
self.metadata_migrations = sa.MetaData()
sa.Table(
'testtbl_one', self.metadata,
sa.Column('id', sa.Integer, primary_key=True),
mysql_engine='InnoDB'
)
sa.Table(
'testtbl_two', self.metadata,
sa.Column('id', sa.Integer, primary_key=True),
sa.Column('tone_id', sa.Integer),
mysql_engine='InnoDB'
)
sa.Table(
'testtbl_one', self.metadata_migrations,
sa.Column('id', sa.Integer, primary_key=True),
mysql_engine='InnoDB'
)
sa.Table(
'testtbl_two', self.metadata_migrations,
sa.Column('id', sa.Integer, primary_key=True),
sa.Column(
'tone_id', sa.Integer,
sa.ForeignKey('testtbl_one.id', name="tone_id_fk")),
mysql_engine='InnoDB'
)
def _fk_removed_fixture(self):
self.metadata = sa.MetaData()
self.metadata_migrations = sa.MetaData()
sa.Table(
'testtbl_one', self.metadata,
sa.Column('id', sa.Integer, primary_key=True),
mysql_engine='InnoDB'
)
sa.Table(
'testtbl_two', self.metadata,
sa.Column('id', sa.Integer, primary_key=True),
sa.Column(
'tone_id', sa.Integer,
sa.ForeignKey('testtbl_one.id', name="tone_id_fk")),
mysql_engine='InnoDB'
)
sa.Table(
'testtbl_one', self.metadata_migrations,
sa.Column('id', sa.Integer, primary_key=True),
mysql_engine='InnoDB'
)
sa.Table(
'testtbl_two', self.metadata_migrations,
sa.Column('id', sa.Integer, primary_key=True),
sa.Column('tone_id', sa.Integer),
mysql_engine='InnoDB'
)
def _fk_no_change_fixture(self):
self.metadata = sa.MetaData()
self.metadata_migrations = sa.MetaData()
sa.Table(
'testtbl_one', self.metadata,
sa.Column('id', sa.Integer, primary_key=True),
mysql_engine='InnoDB'
)
sa.Table(
'testtbl_two', self.metadata,
sa.Column('id', sa.Integer, primary_key=True),
sa.Column(
'tone_id', sa.Integer,
sa.ForeignKey('testtbl_one.id', name="tone_id_fk")),
mysql_engine='InnoDB'
)
sa.Table(
'testtbl_one', self.metadata_migrations,
sa.Column('id', sa.Integer, primary_key=True),
mysql_engine='InnoDB'
)
sa.Table(
'testtbl_two', self.metadata_migrations,
sa.Column('id', sa.Integer, primary_key=True),
sa.Column(
'tone_id', sa.Integer,
sa.ForeignKey('testtbl_one.id', name="tone_id_fk")),
mysql_engine='InnoDB'
)
def _run_test(self):
self.metadata.create_all(bind=self.engine)
return self.migration_sync.check_foreign_keys(
self.metadata_migrations, self.engine)
def _compare_diffs(self, diffs, compare_to):
diffs = [
(
cmd,
fk._get_colspec() if isinstance(fk, sa.ForeignKey)
else "tone_id_fk" if fk is None # sqlite workaround
else fk,
tname, fk_info
)
for cmd, fk, tname, fk_info in diffs
]
self.assertEqual(compare_to, diffs)
def test_fk_added(self):
self._fk_added_fixture()
diffs = self._run_test()
self._compare_diffs(
diffs,
[(
'add_key',
'testtbl_one.id',
'testtbl_two',
self.migration_sync.FKInfo(
constrained_columns=('tone_id',),
referred_table='testtbl_one',
referred_columns=('id',))
)]
)
def test_fk_removed(self):
self._fk_removed_fixture()
diffs = self._run_test()
self._compare_diffs(
diffs,
[(
'drop_key',
"tone_id_fk",
'testtbl_two',
self.migration_sync.FKInfo(
constrained_columns=('tone_id',),
referred_table='testtbl_one',
referred_columns=('id',))
)]
)
def test_fk_no_change(self):
self._fk_no_change_fixture()
diffs = self._run_test()
self._compare_diffs(
diffs,
[])
class PGTestOldCheckForeignKeys(
TestOldCheckForeignKeys, test_base.PostgreSQLOpportunisticTestCase):
pass
class MySQLTestOldCheckForeignKeys(
TestOldCheckForeignKeys, test_base.MySQLOpportunisticTestCase):
pass

181
oslo_db/tests/sqlalchemy/test_models.py
View File

@ -1,181 +0,0 @@
# Copyright 2012 Cloudscaling Group, Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import collections
from oslotest import base as oslo_test
from sqlalchemy import Column
from sqlalchemy import Integer, String
from sqlalchemy.ext.declarative import declarative_base
from oslo_db.sqlalchemy import models
from oslo_db.sqlalchemy import test_base
BASE = declarative_base()
class ModelBaseTest(test_base.DbTestCase):
def setUp(self):
super(ModelBaseTest, self).setUp()
self.mb = models.ModelBase()
self.ekm = ExtraKeysModel()
def test_modelbase_has_dict_methods(self):
dict_methods = ('__getitem__',
'__setitem__',
'__contains__',
'get',
'update',
'save',
'items',
'iteritems',
'keys')
for method in dict_methods:
self.assertTrue(hasattr(models.ModelBase, method),
"Method %s() is not found" % method)
def test_modelbase_is_iterable(self):
self.assertTrue(issubclass(models.ModelBase, collections.Iterable))
def test_modelbase_set(self):
self.mb['world'] = 'hello'
self.assertEqual('hello', self.mb['world'])
def test_modelbase_update(self):
h = {'a': '1', 'b': '2'}
self.mb.update(h)
for key in h.keys():
self.assertEqual(h[key], self.mb[key])
def test_modelbase_contains(self):
mb = models.ModelBase()
h = {'a': '1', 'b': '2'}
mb.update(h)
for key in h.keys():
# Test 'in' syntax (instead of using .assertIn)
self.assertTrue(key in mb)
self.assertFalse('non-existent-key' in mb)
def test_modelbase_contains_exc(self):
class ErrorModel(models.ModelBase):
@property
def bug(self):
raise ValueError
model = ErrorModel()
model.update({'attr': 5})
self.assertTrue('attr' in model)
self.assertRaises(ValueError, lambda: 'bug' in model)
def test_modelbase_items_iteritems(self):
h = {'a': '1', 'b': '2'}
expected = {
'id': None,
'smth': None,
'name': 'NAME',
'a': '1',
'b': '2',
}
self.ekm.update(h)
self.assertEqual(expected, dict(self.ekm.items()))
self.assertEqual(expected, dict(self.ekm.iteritems()))
def test_modelbase_dict(self):
h = {'a': '1', 'b': '2'}
expected = {
'id': None,
'smth': None,
'name': 'NAME',
'a': '1',
'b': '2',
}
self.ekm.update(h)
self.assertEqual(expected, dict(self.ekm))
def test_modelbase_iter(self):
expected = {
'id': None,
'smth': None,
'name': 'NAME',
}
i = iter(self.ekm)
found_items = 0
while True:
r = next(i, None)
if r is None:
break
self.assertEqual(expected[r[0]], r[1])
found_items += 1
self.assertEqual(len(expected), found_items)
def test_modelbase_keys(self):
self.assertEqual(set(('id', 'smth', 'name')), set(self.ekm.keys()))
self.ekm.update({'a': '1', 'b': '2'})
self.assertEqual(set(('a', 'b', 'id', 'smth', 'name')),
set(self.ekm.keys()))
def test_modelbase_several_iters(self):
mb = ExtraKeysModel()
it1 = iter(mb)
it2 = iter(mb)
self.assertFalse(it1 is it2)
self.assertEqual(dict(mb), dict(it1))
self.assertEqual(dict(mb), dict(it2))
def test_extra_keys_empty(self):
"""Test verifies that by default extra_keys return empty list."""
self.assertEqual([], self.mb._extra_keys)
def test_extra_keys_defined(self):
"""Property _extra_keys will return list with attributes names."""
self.assertEqual(['name'], self.ekm._extra_keys)
def test_model_with_extra_keys(self):
data = dict(self.ekm)
self.assertEqual({'smth': None,
'id': None,
'name': 'NAME'},
data)
class ExtraKeysModel(BASE, models.ModelBase):
__tablename__ = 'test_model'
id = Column(Integer, primary_key=True)
smth = Column(String(255))
@property
def name(self):
return 'NAME'
@property
def _extra_keys(self):
return ['name']
class TimestampMixinTest(oslo_test.BaseTestCase):
def test_timestampmixin_attr(self):
methods = ('created_at',
'updated_at')
for method in methods:
self.assertTrue(hasattr(models.TimestampMixin, method),
"Method %s() is not found" % method)

127
oslo_db/tests/sqlalchemy/test_options.py
View File

@ -1,127 +0,0 @@
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from oslo_config import cfg
from oslo_config import fixture as config
from oslo_db import options
from oslo_db.tests import utils as test_utils
class DbApiOptionsTestCase(test_utils.BaseTestCase):
def setUp(self):
super(DbApiOptionsTestCase, self).setUp()
config_fixture = self.useFixture(config.Config())
self.conf = config_fixture.conf
self.conf.register_opts(options.database_opts, group='database')
self.config = config_fixture.config
def test_deprecated_session_parameters(self):
path = self.create_tempfiles([["tmp", b"""[DEFAULT]
sql_connection=x://y.z
sql_min_pool_size=10
sql_max_pool_size=20
sql_max_retries=30
sql_retry_interval=40
sql_max_overflow=50
sql_connection_debug=60
sql_connection_trace=True
"""]])[0]
self.conf(['--config-file', path])
self.assertEqual('x://y.z', self.conf.database.connection)
self.assertEqual(10, self.conf.database.min_pool_size)
self.assertEqual(20, self.conf.database.max_pool_size)
self.assertEqual(30, self.conf.database.max_retries)
self.assertEqual(40, self.conf.database.retry_interval)
self.assertEqual(50, self.conf.database.max_overflow)
self.assertEqual(60, self.conf.database.connection_debug)
self.assertEqual(True, self.conf.database.connection_trace)
def test_session_parameters(self):
path = self.create_tempfiles([["tmp", b"""[database]
connection=x://y.z
min_pool_size=10
max_pool_size=20
max_retries=30
retry_interval=40
max_overflow=50
connection_debug=60
connection_trace=True
pool_timeout=7
"""]])[0]
self.conf(['--config-file', path])
self.assertEqual('x://y.z', self.conf.database.connection)
self.assertEqual(10, self.conf.database.min_pool_size)
self.assertEqual(20, self.conf.database.max_pool_size)
self.assertEqual(30, self.conf.database.max_retries)
self.assertEqual(40, self.conf.database.retry_interval)
self.assertEqual(50, self.conf.database.max_overflow)
self.assertEqual(60, self.conf.database.connection_debug)
self.assertEqual(True, self.conf.database.connection_trace)
self.assertEqual(7, self.conf.database.pool_timeout)
def test_dbapi_database_deprecated_parameters(self):
path = self.create_tempfiles([['tmp', b'[DATABASE]\n'
b'sql_connection=fake_connection\n'
b'sql_idle_timeout=100\n'
b'sql_min_pool_size=99\n'
b'sql_max_pool_size=199\n'
b'sql_max_retries=22\n'
b'reconnect_interval=17\n'
b'sqlalchemy_max_overflow=101\n'
b'sqlalchemy_pool_timeout=5\n'
]])[0]
self.conf(['--config-file', path])
self.assertEqual('fake_connection', self.conf.database.connection)
self.assertEqual(100, self.conf.database.idle_timeout)
self.assertEqual(99, self.conf.database.min_pool_size)
self.assertEqual(199, self.conf.database.max_pool_size)
self.assertEqual(22, self.conf.database.max_retries)
self.assertEqual(17, self.conf.database.retry_interval)
self.assertEqual(101, self.conf.database.max_overflow)
self.assertEqual(5, self.conf.database.pool_timeout)
def test_dbapi_database_deprecated_parameters_sql(self):
path = self.create_tempfiles([['tmp', b'[sql]\n'
b'connection=test_sql_connection\n'
b'idle_timeout=99\n'
]])[0]
self.conf(['--config-file', path])
self.assertEqual('test_sql_connection', self.conf.database.connection)
self.assertEqual(99, self.conf.database.idle_timeout)
def test_deprecated_dbapi_parameters(self):
path = self.create_tempfiles([['tmp', b'[DEFAULT]\n'
b'db_backend=test_123\n'
]])[0]
self.conf(['--config-file', path])
self.assertEqual('test_123', self.conf.database.backend)
def test_dbapi_parameters(self):
path = self.create_tempfiles([['tmp', b'[database]\n'
b'backend=test_123\n'
]])[0]
self.conf(['--config-file', path])
self.assertEqual('test_123', self.conf.database.backend)
def test_set_defaults(self):
conf = cfg.ConfigOpts()
options.set_defaults(conf,
connection='sqlite:///:memory:')
self.assertTrue(len(conf.database.items()) > 1)
self.assertEqual('sqlite:///:memory:', conf.database.connection)

213
oslo_db/tests/sqlalchemy/test_provision.py
View File

@ -1,213 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import mock
from oslotest import base as oslo_test_base
from sqlalchemy import exc as sa_exc
from sqlalchemy import inspect
from sqlalchemy import schema
from sqlalchemy import types
from oslo_db import exception
from oslo_db.sqlalchemy import provision
from oslo_db.sqlalchemy import test_base
class DropAllObjectsTest(test_base.DbTestCase):
def setUp(self):
super(DropAllObjectsTest, self).setUp()
self.metadata = metadata = schema.MetaData()
schema.Table(
'a', metadata,
schema.Column('id', types.Integer, primary_key=True),
mysql_engine='InnoDB'
)
schema.Table(
'b', metadata,
schema.Column('id', types.Integer, primary_key=True),
schema.Column('a_id', types.Integer, schema.ForeignKey('a.id')),
mysql_engine='InnoDB'
)
schema.Table(
'c', metadata,
schema.Column('id', types.Integer, primary_key=True),
schema.Column('b_id', types.Integer, schema.ForeignKey('b.id')),
schema.Column(
'd_id', types.Integer,
schema.ForeignKey('d.id', use_alter=True, name='c_d_fk')),
mysql_engine='InnoDB'
)
schema.Table(
'd', metadata,
schema.Column('id', types.Integer, primary_key=True),
schema.Column('c_id', types.Integer, schema.ForeignKey('c.id')),
mysql_engine='InnoDB'
)
metadata.create_all(self.engine, checkfirst=False)
# will drop nothing if the test worked
self.addCleanup(metadata.drop_all, self.engine, checkfirst=True)
def test_drop_all(self):
insp = inspect(self.engine)
self.assertEqual(
set(['a', 'b', 'c', 'd']),
set(insp.get_table_names())
)
self.db.backend.drop_all_objects(self.engine)
insp = inspect(self.engine)
self.assertEqual(
[],
insp.get_table_names()
)
class BackendNotAvailableTest(oslo_test_base.BaseTestCase):
def test_no_dbapi(self):
backend = provision.Backend(
"postgresql", "postgresql+nosuchdbapi://hostname/dsn")
with mock.patch(
"sqlalchemy.create_engine",
mock.Mock(side_effect=ImportError("nosuchdbapi"))):
# NOTE(zzzeek): Call and test the _verify function twice, as it
# exercises a different code path on subsequent runs vs.
# the first run
ex = self.assertRaises(
exception.BackendNotAvailable,
backend._verify)
self.assertEqual(
"Backend 'postgresql+nosuchdbapi' is unavailable: "
"No DBAPI installed", str(ex))
ex = self.assertRaises(
exception.BackendNotAvailable,
backend._verify)
self.assertEqual(
"Backend 'postgresql+nosuchdbapi' is unavailable: "
"No DBAPI installed", str(ex))
def test_cant_connect(self):
backend = provision.Backend(
"postgresql", "postgresql+nosuchdbapi://hostname/dsn")
with mock.patch(
"sqlalchemy.create_engine",
mock.Mock(return_value=mock.Mock(connect=mock.Mock(
side_effect=sa_exc.OperationalError(
"can't connect", None, None))
))
):
# NOTE(zzzeek): Call and test the _verify function twice, as it
# exercises a different code path on subsequent runs vs.
# the first run
ex = self.assertRaises(
exception.BackendNotAvailable,
backend._verify)
self.assertEqual(
"Backend 'postgresql+nosuchdbapi' is unavailable: "
"Could not connect", str(ex))
ex = self.assertRaises(
exception.BackendNotAvailable,
backend._verify)
self.assertEqual(
"Backend 'postgresql+nosuchdbapi' is unavailable: "
"Could not connect", str(ex))
class MySQLDropAllObjectsTest(
DropAllObjectsTest, test_base.MySQLOpportunisticTestCase):
pass
class PostgreSQLDropAllObjectsTest(
DropAllObjectsTest, test_base.PostgreSQLOpportunisticTestCase):
pass
class RetainSchemaTest(oslo_test_base.BaseTestCase):
DRIVER = "sqlite"
def setUp(self):
super(RetainSchemaTest, self).setUp()
metadata = schema.MetaData()
self.test_table = schema.Table(
'test_table', metadata,
schema.Column('x', types.Integer),
schema.Column('y', types.Integer),
mysql_engine='InnoDB'
)
def gen_schema(engine):
metadata.create_all(engine, checkfirst=False)
self._gen_schema = gen_schema
def test_once(self):
self._run_test()
def test_twice(self):
self._run_test()
def _run_test(self):
try:
database_resource = provision.DatabaseResource(self.DRIVER)
except exception.BackendNotAvailable:
self.skip("database not available")
schema_resource = provision.SchemaResource(
database_resource, self._gen_schema)
transaction_resource = provision.TransactionResource(
database_resource, schema_resource)
engine = transaction_resource.getResource()
with engine.connect() as conn:
rows = conn.execute(self.test_table.select())
self.assertEqual([], rows.fetchall())
trans = conn.begin()
conn.execute(
self.test_table.insert(),
{"x": 1, "y": 2}
)
trans.rollback()
rows = conn.execute(self.test_table.select())
self.assertEqual([], rows.fetchall())
trans = conn.begin()
conn.execute(
self.test_table.insert(),
{"x": 2, "y": 3}
)
trans.commit()
rows = conn.execute(self.test_table.select())
self.assertEqual([(2, 3)], rows.fetchall())
transaction_resource.finishedWith(engine)
class MySQLRetainSchemaTest(RetainSchemaTest):
DRIVER = "mysql"
class PostgresqlRetainSchemaTest(RetainSchemaTest):
DRIVER = "postgresql"

724
oslo_db/tests/sqlalchemy/test_sqlalchemy.py
View File

@ -1,724 +0,0 @@
# coding=utf-8
# Copyright (c) 2012 Rackspace Hosting
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""Unit tests for SQLAlchemy specific code."""
import logging
import os
import fixtures
import mock
from oslo_config import cfg
from oslotest import base as oslo_test
import six
import sqlalchemy
from sqlalchemy import Column, MetaData, Table
from sqlalchemy.engine import url
from sqlalchemy import Integer, String
from sqlalchemy.ext.declarative import declarative_base
from oslo_db import exception
from oslo_db import options as db_options
from oslo_db.sqlalchemy import engines
from oslo_db.sqlalchemy import models
from oslo_db.sqlalchemy import session
from oslo_db.sqlalchemy import test_base
BASE = declarative_base()
_TABLE_NAME = '__tmp__test__tmp__'
_REGEXP_TABLE_NAME = _TABLE_NAME + "regexp"
class RegexpTable(BASE, models.ModelBase):
__tablename__ = _REGEXP_TABLE_NAME
id = Column(Integer, primary_key=True)
bar = Column(String(255))
class RegexpFilterTestCase(test_base.DbTestCase):
def setUp(self):
super(RegexpFilterTestCase, self).setUp()
meta = MetaData()
meta.bind = self.engine
test_table = Table(_REGEXP_TABLE_NAME, meta,
Column('id', Integer, primary_key=True,
nullable=False),
Column('bar', String(255)))
test_table.create()
self.addCleanup(test_table.drop)
def _test_regexp_filter(self, regexp, expected):
_session = self.sessionmaker()
with _session.begin():
for i in ['10', '20', u'♥']:
tbl = RegexpTable()
tbl.update({'bar': i})
tbl.save(session=_session)
regexp_op = RegexpTable.bar.op('REGEXP')(regexp)
result = _session.query(RegexpTable).filter(regexp_op).all()
self.assertEqual(expected, [r.bar for r in result])
def test_regexp_filter(self):
self._test_regexp_filter('10', ['10'])
def test_regexp_filter_nomatch(self):
self._test_regexp_filter('11', [])
def test_regexp_filter_unicode(self):
self._test_regexp_filter(u'♥', [u'♥'])
def test_regexp_filter_unicode_nomatch(self):
self._test_regexp_filter(u'♦', [])
class SQLiteSavepointTest(test_base.DbTestCase):
def setUp(self):
super(SQLiteSavepointTest, self).setUp()
meta = MetaData()
self.test_table = Table(
"test_table", meta,
Column('id', Integer, primary_key=True),
Column('data', String(10)))
self.test_table.create(self.engine)
self.addCleanup(self.test_table.drop, self.engine)
def test_plain_transaction(self):
conn = self.engine.connect()
trans = conn.begin()
conn.execute(
self.test_table.insert(),
{'data': 'data 1'}
)
self.assertEqual(
[(1, 'data 1')],
self.engine.execute(
self.test_table.select().
order_by(self.test_table.c.id)
).fetchall()
)
trans.rollback()
self.assertEqual(
0,
self.engine.scalar(self.test_table.count())
)
def test_savepoint_middle(self):
with self.engine.begin() as conn:
conn.execute(
self.test_table.insert(),
{'data': 'data 1'}
)
savepoint = conn.begin_nested()
conn.execute(
self.test_table.insert(),
{'data': 'data 2'}
)
savepoint.rollback()
conn.execute(
self.test_table.insert(),
{'data': 'data 3'}
)
self.assertEqual(
[(1, 'data 1'), (2, 'data 3')],
self.engine.execute(
self.test_table.select().
order_by(self.test_table.c.id)
).fetchall()
)
def test_savepoint_beginning(self):
with self.engine.begin() as conn:
savepoint = conn.begin_nested()
conn.execute(
self.test_table.insert(),
{'data': 'data 1'}
)
savepoint.rollback()
conn.execute(
self.test_table.insert(),
{'data': 'data 2'}
)
self.assertEqual(
[(1, 'data 2')],
self.engine.execute(
self.test_table.select().
order_by(self.test_table.c.id)
).fetchall()
)
class FakeDBAPIConnection(object):
def cursor(self):
return FakeCursor()
class FakeCursor(object):
def execute(self, sql):
pass
class FakeConnectionProxy(object):
pass
class FakeConnectionRec(object):
pass
class OperationalError(Exception):
pass
class ProgrammingError(Exception):
pass
class FakeDB2Engine(object):
class Dialect(object):
def is_disconnect(self, e, *args):
expected_error = ('SQL30081N: DB2 Server connection is no longer '
'active')
return (str(e) == expected_error)
dialect = Dialect()
name = 'ibm_db_sa'
def dispose(self):
pass
class MySQLDefaultModeTestCase(test_base.MySQLOpportunisticTestCase):
def test_default_is_traditional(self):
with self.engine.connect() as conn:
sql_mode = conn.execute(
"SHOW VARIABLES LIKE 'sql_mode'"
).first()[1]
self.assertTrue("TRADITIONAL" in sql_mode)
class MySQLModeTestCase(test_base.MySQLOpportunisticTestCase):
def __init__(self, *args, **kwargs):
super(MySQLModeTestCase, self).__init__(*args, **kwargs)
# By default, run in empty SQL mode.
# Subclasses override this with specific modes.
self.mysql_mode = ''
def setUp(self):
super(MySQLModeTestCase, self).setUp()
mode_engine = session.create_engine(
self.engine.url,
mysql_sql_mode=self.mysql_mode)
self.connection = mode_engine.connect()
meta = MetaData()
self.test_table = Table(_TABLE_NAME + "mode", meta,
Column('id', Integer, primary_key=True),
Column('bar', String(255)))
self.test_table.create(self.connection)
def cleanup():
self.test_table.drop(self.connection)
self.connection.close()
mode_engine.dispose()
self.addCleanup(cleanup)
def _test_string_too_long(self, value):
with self.connection.begin():
self.connection.execute(self.test_table.insert(),
bar=value)
result = self.connection.execute(self.test_table.select())
return result.fetchone()['bar']
def test_string_too_long(self):
value = 'a' * 512
# String is too long.
# With no SQL mode set, this gets truncated.
self.assertNotEqual(value,
self._test_string_too_long(value))
class MySQLStrictAllTablesModeTestCase(MySQLModeTestCase):
"Test data integrity enforcement in MySQL STRICT_ALL_TABLES mode."
def __init__(self, *args, **kwargs):
super(MySQLStrictAllTablesModeTestCase, self).__init__(*args, **kwargs)
self.mysql_mode = 'STRICT_ALL_TABLES'
def test_string_too_long(self):
value = 'a' * 512
# String is too long.
# With STRICT_ALL_TABLES or TRADITIONAL mode set, this is an error.
self.assertRaises(exception.DBError,
self._test_string_too_long, value)
class MySQLTraditionalModeTestCase(MySQLStrictAllTablesModeTestCase):
"""Test data integrity enforcement in MySQL TRADITIONAL mode.
Since TRADITIONAL includes STRICT_ALL_TABLES, this inherits all
STRICT_ALL_TABLES mode tests.
"""
def __init__(self, *args, **kwargs):
super(MySQLTraditionalModeTestCase, self).__init__(*args, **kwargs)
self.mysql_mode = 'TRADITIONAL'
class EngineFacadeTestCase(oslo_test.BaseTestCase):
def setUp(self):
super(EngineFacadeTestCase, self).setUp()
self.facade = session.EngineFacade('sqlite://')
def test_get_engine(self):
eng1 = self.facade.get_engine()
eng2 = self.facade.get_engine()
self.assertIs(eng1, eng2)
def test_get_session(self):
ses1 = self.facade.get_session()
ses2 = self.facade.get_session()
self.assertIsNot(ses1, ses2)
def test_get_session_arguments_override_default_settings(self):
ses = self.facade.get_session(autocommit=False, expire_on_commit=True)
self.assertFalse(ses.autocommit)
self.assertTrue(ses.expire_on_commit)
@mock.patch('oslo_db.sqlalchemy.orm.get_maker')
@mock.patch('oslo_db.sqlalchemy.engines.create_engine')
def test_creation_from_config(self, create_engine, get_maker):
conf = cfg.ConfigOpts()
conf.register_opts(db_options.database_opts, group='database')
overrides = {
'connection': 'sqlite:///:memory:',
'slave_connection': None,
'connection_debug': 100,
'max_pool_size': 10,
'mysql_sql_mode': 'TRADITIONAL',
}
for optname, optvalue in overrides.items():
conf.set_override(optname, optvalue, group='database')
session.EngineFacade.from_config(conf,
autocommit=False,
expire_on_commit=True)
create_engine.assert_called_once_with(
sql_connection='sqlite:///:memory:',
connection_debug=100,
max_pool_size=10,
mysql_sql_mode='TRADITIONAL',
sqlite_fk=False,
idle_timeout=mock.ANY,
retry_interval=mock.ANY,
max_retries=mock.ANY,
max_overflow=mock.ANY,
connection_trace=mock.ANY,
sqlite_synchronous=mock.ANY,
pool_timeout=mock.ANY,
thread_checkin=mock.ANY,
json_serializer=None,
json_deserializer=None,
logging_name=mock.ANY,
)
get_maker.assert_called_once_with(engine=create_engine(),
autocommit=False,
expire_on_commit=True)
def test_slave_connection(self):
paths = self.create_tempfiles([('db.master', ''), ('db.slave', '')],
ext='')
master_path = 'sqlite:///' + paths[0]
slave_path = 'sqlite:///' + paths[1]
facade = session.EngineFacade(
sql_connection=master_path,
slave_connection=slave_path
)
master = facade.get_engine()
self.assertEqual(master_path, str(master.url))
slave = facade.get_engine(use_slave=True)
self.assertEqual(slave_path, str(slave.url))
master_session = facade.get_session()
self.assertEqual(master_path, str(master_session.bind.url))
slave_session = facade.get_session(use_slave=True)
self.assertEqual(slave_path, str(slave_session.bind.url))
def test_slave_connection_string_not_provided(self):
master_path = 'sqlite:///' + self.create_tempfiles(
[('db.master', '')], ext='')[0]
facade = session.EngineFacade(sql_connection=master_path)
master = facade.get_engine()
slave = facade.get_engine(use_slave=True)
self.assertIs(master, slave)
self.assertEqual(master_path, str(master.url))
master_session = facade.get_session()
self.assertEqual(master_path, str(master_session.bind.url))
slave_session = facade.get_session(use_slave=True)
self.assertEqual(master_path, str(slave_session.bind.url))
class SQLiteConnectTest(oslo_test.BaseTestCase):
def _fixture(self, **kw):
return session.create_engine("sqlite://", **kw)
def test_sqlite_fk_listener(self):
engine = self._fixture(sqlite_fk=True)
self.assertEqual(
1,
engine.scalar("pragma foreign_keys")
)
engine = self._fixture(sqlite_fk=False)
self.assertEqual(
0,
engine.scalar("pragma foreign_keys")
)
def test_sqlite_synchronous_listener(self):
engine = self._fixture()
# "The default setting is synchronous=FULL." (e.g. 2)
# http://www.sqlite.org/pragma.html#pragma_synchronous
self.assertEqual(
2,
engine.scalar("pragma synchronous")
)
engine = self._fixture(sqlite_synchronous=False)
self.assertEqual(
0,
engine.scalar("pragma synchronous")
)
class MysqlConnectTest(test_base.MySQLOpportunisticTestCase):
def _fixture(self, sql_mode):
return session.create_engine(self.engine.url, mysql_sql_mode=sql_mode)
def _assert_sql_mode(self, engine, sql_mode_present, sql_mode_non_present):
mode = engine.execute("SHOW VARIABLES LIKE 'sql_mode'").fetchone()[1]
self.assertTrue(
sql_mode_present in mode
)
if sql_mode_non_present:
self.assertTrue(
sql_mode_non_present not in mode
)
def test_set_mode_traditional(self):
engine = self._fixture(sql_mode='TRADITIONAL')
self._assert_sql_mode(engine, "TRADITIONAL", "ANSI")
def test_set_mode_ansi(self):
engine = self._fixture(sql_mode='ANSI')
self._assert_sql_mode(engine, "ANSI", "TRADITIONAL")
def test_set_mode_no_mode(self):
# If _mysql_set_mode_callback is called with sql_mode=None, then
# the SQL mode is NOT set on the connection.
# get the GLOBAL sql_mode, not the @@SESSION, so that
# we get what is configured for the MySQL database, as opposed
# to what our own session.create_engine() has set it to.
expected = self.engine.execute(
"SELECT @@GLOBAL.sql_mode").scalar()
engine = self._fixture(sql_mode=None)
self._assert_sql_mode(engine, expected, None)
def test_fail_detect_mode(self):
# If "SHOW VARIABLES LIKE 'sql_mode'" results in no row, then
# we get a log indicating can't detect the mode.
log = self.useFixture(fixtures.FakeLogger(level=logging.WARN))
mysql_conn = self.engine.raw_connection()
self.addCleanup(mysql_conn.close)
mysql_conn.detach()
mysql_cursor = mysql_conn.cursor()
def execute(statement, parameters=()):
if "SHOW VARIABLES LIKE 'sql_mode'" in statement:
statement = "SHOW VARIABLES LIKE 'i_dont_exist'"
return mysql_cursor.execute(statement, parameters)
test_engine = sqlalchemy.create_engine(self.engine.url,
_initialize=False)
with mock.patch.object(
test_engine.pool, '_creator',
mock.Mock(
return_value=mock.Mock(
cursor=mock.Mock(
return_value=mock.Mock(
execute=execute,
fetchone=mysql_cursor.fetchone,
fetchall=mysql_cursor.fetchall
)
)
)
)
):
engines._init_events.dispatch_on_drivername("mysql")(test_engine)
test_engine.raw_connection()
self.assertIn('Unable to detect effective SQL mode',
log.output)
def test_logs_real_mode(self):
# If "SHOW VARIABLES LIKE 'sql_mode'" results in a value, then
# we get a log with the value.
log = self.useFixture(fixtures.FakeLogger(level=logging.DEBUG))
engine = self._fixture(sql_mode='TRADITIONAL')
actual_mode = engine.execute(
"SHOW VARIABLES LIKE 'sql_mode'").fetchone()[1]
self.assertIn('MySQL server mode set to %s' % actual_mode,
log.output)
def test_warning_when_not_traditional(self):
# If "SHOW VARIABLES LIKE 'sql_mode'" results in a value that doesn't
# include 'TRADITIONAL', then a warning is logged.
log = self.useFixture(fixtures.FakeLogger(level=logging.WARN))
self._fixture(sql_mode='ANSI')
self.assertIn("consider enabling TRADITIONAL or STRICT_ALL_TABLES",
log.output)
def test_no_warning_when_traditional(self):
# If "SHOW VARIABLES LIKE 'sql_mode'" results in a value that includes
# 'TRADITIONAL', then no warning is logged.
log = self.useFixture(fixtures.FakeLogger(level=logging.WARN))
self._fixture(sql_mode='TRADITIONAL')
self.assertNotIn("consider enabling TRADITIONAL or STRICT_ALL_TABLES",
log.output)
def test_no_warning_when_strict_all_tables(self):
# If "SHOW VARIABLES LIKE 'sql_mode'" results in a value that includes
# 'STRICT_ALL_TABLES', then no warning is logged.
log = self.useFixture(fixtures.FakeLogger(level=logging.WARN))
self._fixture(sql_mode='TRADITIONAL')
self.assertNotIn("consider enabling TRADITIONAL or STRICT_ALL_TABLES",
log.output)
class CreateEngineTest(oslo_test.BaseTestCase):
"""Test that dialect-specific arguments/ listeners are set up correctly.
"""
def setUp(self):
super(CreateEngineTest, self).setUp()
self.args = {'connect_args': {}}
def test_queuepool_args(self):
engines._init_connection_args(
url.make_url("mysql+pymysql://u:p@host/test"), self.args,
max_pool_size=10, max_overflow=10)
self.assertEqual(10, self.args['pool_size'])
self.assertEqual(10, self.args['max_overflow'])
def test_sqlite_memory_pool_args(self):
for _url in ("sqlite://", "sqlite:///:memory:"):
engines._init_connection_args(
url.make_url(_url), self.args,
max_pool_size=10, max_overflow=10)
# queuepool arguments are not peresnet
self.assertTrue(
'pool_size' not in self.args)
self.assertTrue(
'max_overflow' not in self.args)
self.assertEqual(False,
self.args['connect_args']['check_same_thread'])
# due to memory connection
self.assertTrue('poolclass' in self.args)
def test_sqlite_file_pool_args(self):
engines._init_connection_args(
url.make_url("sqlite:///somefile.db"), self.args,
max_pool_size=10, max_overflow=10)
# queuepool arguments are not peresnet
self.assertTrue('pool_size' not in self.args)
self.assertTrue(
'max_overflow' not in self.args)
self.assertFalse(self.args['connect_args'])
# NullPool is the default for file based connections,
# no need to specify this
self.assertTrue('poolclass' not in self.args)
def _test_mysql_connect_args_default(self, connect_args):
if six.PY3:
self.assertEqual({'charset': 'utf8', 'use_unicode': 1},
connect_args)
else:
self.assertEqual({'charset': 'utf8', 'use_unicode': 0},
connect_args)
def test_mysql_connect_args_default(self):
engines._init_connection_args(
url.make_url("mysql://u:p@host/test"), self.args)
self._test_mysql_connect_args_default(self.args['connect_args'])
def test_mysql_oursql_connect_args_default(self):
engines._init_connection_args(
url.make_url("mysql+oursql://u:p@host/test"), self.args)
self._test_mysql_connect_args_default(self.args['connect_args'])
def test_mysql_pymysql_connect_args_default(self):
engines._init_connection_args(
url.make_url("mysql+pymysql://u:p@host/test"), self.args)
self.assertEqual({'charset': 'utf8'}, self.args['connect_args'])
def test_mysql_mysqldb_connect_args_default(self):
engines._init_connection_args(
url.make_url("mysql+mysqldb://u:p@host/test"), self.args)
self._test_mysql_connect_args_default(self.args['connect_args'])
def test_postgresql_connect_args_default(self):
engines._init_connection_args(
url.make_url("postgresql://u:p@host/test"), self.args)
self.assertEqual('utf8', self.args['client_encoding'])
self.assertFalse(self.args['connect_args'])
def test_mysqlconnector_raise_on_warnings_default(self):
engines._init_connection_args(
url.make_url("mysql+mysqlconnector://u:p@host/test"),
self.args)
self.assertEqual(False, self.args['connect_args']['raise_on_warnings'])
def test_mysqlconnector_raise_on_warnings_override(self):
engines._init_connection_args(
url.make_url(
"mysql+mysqlconnector://u:p@host/test"
"?raise_on_warnings=true"),
self.args
)
self.assertFalse('raise_on_warnings' in self.args['connect_args'])
def test_thread_checkin(self):
with mock.patch("sqlalchemy.event.listens_for"):
with mock.patch("sqlalchemy.event.listen") as listen_evt:
engines._init_events.dispatch_on_drivername(
"sqlite")(mock.Mock())
self.assertEqual(
listen_evt.mock_calls[0][1][-1],
engines._thread_yield
)
class ProcessGuardTest(test_base.DbTestCase):
def test_process_guard(self):
self.engine.dispose()
def get_parent_pid():
return 4
def get_child_pid():
return 5
with mock.patch("os.getpid", get_parent_pid):
with self.engine.connect() as conn:
dbapi_id = id(conn.connection.connection)
with mock.patch("os.getpid", get_child_pid):
with self.engine.connect() as conn:
new_dbapi_id = id(conn.connection.connection)
self.assertNotEqual(dbapi_id, new_dbapi_id)
# ensure it doesn't trip again
with mock.patch("os.getpid", get_child_pid):
with self.engine.connect() as conn:
newer_dbapi_id = id(conn.connection.connection)
self.assertEqual(new_dbapi_id, newer_dbapi_id)
class PatchStacktraceTest(test_base.DbTestCase):
def test_trace(self):
engine = self.engine
# NOTE(viktors): The code in oslo_db.sqlalchemy.session filters out
# lines from modules under oslo_db, so we should remove
# "oslo_db/" from file path in traceback.
import traceback
orig_extract_stack = traceback.extract_stack
def extract_stack():
return [(row[0].replace("oslo_db/", ""), row[1], row[2], row[3])
for row in orig_extract_stack()]
with mock.patch("traceback.extract_stack", side_effect=extract_stack):
engines._add_trace_comments(engine)
conn = engine.connect()
orig_do_exec = engine.dialect.do_execute
with mock.patch.object(engine.dialect, "do_execute") as mock_exec:
mock_exec.side_effect = orig_do_exec
conn.execute("select 1;")
call = mock_exec.mock_calls[0]
# we're the caller, see that we're in there
caller = os.path.join("tests", "sqlalchemy", "test_sqlalchemy.py")
self.assertIn(caller, call[1][1])

111
oslo_db/tests/sqlalchemy/test_types.py
View File

@ -1,111 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""Tests for JSON SQLAlchemy types."""
from sqlalchemy import Column, Integer
from sqlalchemy.dialects import mysql
from sqlalchemy.ext.declarative import declarative_base
from oslo_db import exception as db_exc
from oslo_db.sqlalchemy import models
from oslo_db.sqlalchemy import test_base
from oslo_db.sqlalchemy import types
BASE = declarative_base()
class JsonTable(BASE, models.ModelBase):
__tablename__ = 'test_json_types'
id = Column(Integer, primary_key=True)
jdict = Column(types.JsonEncodedDict)
jlist = Column(types.JsonEncodedList)
json = Column(types.JsonEncodedType)
class JsonTypesTestCase(test_base.DbTestCase):
def setUp(self):
super(JsonTypesTestCase, self).setUp()
JsonTable.__table__.create(self.engine)
self.addCleanup(JsonTable.__table__.drop, self.engine)
self.session = self.sessionmaker()
self.addCleanup(self.session.close)
def test_default_value(self):
with self.session.begin():
JsonTable(id=1).save(self.session)
obj = self.session.query(JsonTable).filter_by(id=1).one()
self.assertEqual([], obj.jlist)
self.assertEqual({}, obj.jdict)
self.assertIsNone(obj.json)
def test_dict(self):
test = {'a': 42, 'b': [1, 2, 3]}
with self.session.begin():
JsonTable(id=1, jdict=test).save(self.session)
obj = self.session.query(JsonTable).filter_by(id=1).one()
self.assertEqual(test, obj.jdict)
def test_list(self):
test = [1, True, "hello", {}]
with self.session.begin():
JsonTable(id=1, jlist=test).save(self.session)
obj = self.session.query(JsonTable).filter_by(id=1).one()
self.assertEqual(test, obj.jlist)
def test_dict_type_check(self):
self.assertRaises(db_exc.DBError,
JsonTable(id=1, jdict=[]).save, self.session)
def test_list_type_check(self):
self.assertRaises(db_exc.DBError,
JsonTable(id=1, jlist={}).save, self.session)
def test_generic(self):
tested = [
"string",
42,
True,
None,
[1, 2, 3],
{'a': 'b'}
]
for i, test in enumerate(tested):
with self.session.begin():
JsonTable(id=i, json=test).save(self.session)
obj = self.session.query(JsonTable).filter_by(id=i).one()
self.assertEqual(test, obj.json)
def test_mysql_variants(self):
self.assertEqual(
"LONGTEXT",
str(
types.JsonEncodedDict(mysql_as_long=True).compile(
dialect=mysql.dialect())
)
)
self.assertEqual(
"MEDIUMTEXT",
str(
types.JsonEncodedDict(mysql_as_medium=True).compile(
dialect=mysql.dialect())
)
)
self.assertRaises(
TypeError,
lambda: types.JsonEncodedDict(
mysql_as_long=True,
mysql_as_medium=True)
)

445
oslo_db/tests/sqlalchemy/test_update_match.py
View File

@ -1,445 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from oslotest import base as oslo_test_base
from sqlalchemy.ext import declarative
from sqlalchemy import schema
from sqlalchemy import sql
from sqlalchemy import types as sqltypes
from oslo_db.sqlalchemy import test_base
from oslo_db.sqlalchemy import update_match
Base = declarative.declarative_base()
class MyModel(Base):
__tablename__ = 'my_table'
id = schema.Column(sqltypes.Integer, primary_key=True)
uuid = schema.Column(sqltypes.String(36), nullable=False, unique=True)
x = schema.Column(sqltypes.Integer)
y = schema.Column(sqltypes.String(40))
z = schema.Column(sqltypes.String(40))
class ManufactureCriteriaTest(oslo_test_base.BaseTestCase):
def test_instance_criteria_basic(self):
specimen = MyModel(
y='y1', z='z3',
uuid='136254d5-3869-408f-9da7-190e0072641a'
)
self.assertEqual(
"my_table.uuid = :uuid_1 AND my_table.y = :y_1 "
"AND my_table.z = :z_1",
str(update_match.manufacture_entity_criteria(specimen).compile())
)
def test_instance_criteria_basic_wnone(self):
specimen = MyModel(
y='y1', z=None,
uuid='136254d5-3869-408f-9da7-190e0072641a'
)
self.assertEqual(
"my_table.uuid = :uuid_1 AND my_table.y = :y_1 "
"AND my_table.z IS NULL",
str(update_match.manufacture_entity_criteria(specimen).compile())
)
def test_instance_criteria_tuples(self):
specimen = MyModel(
y='y1', z=('z1', 'z2'),
)
self.assertEqual(
"my_table.y = :y_1 AND my_table.z IN (:z_1, :z_2)",
str(update_match.manufacture_entity_criteria(specimen).compile())
)
def test_instance_criteria_tuples_wnone(self):
specimen = MyModel(
y='y1', z=('z1', 'z2', None),
)
self.assertEqual(
"my_table.y = :y_1 AND (my_table.z IS NULL OR "
"my_table.z IN (:z_1, :z_2))",
str(update_match.manufacture_entity_criteria(specimen).compile())
)
def test_instance_criteria_none_list(self):
specimen = MyModel(
y='y1', z=[None],
)
self.assertEqual(
"my_table.y = :y_1 AND my_table.z IS NULL",
str(update_match.manufacture_entity_criteria(specimen).compile())
)
class UpdateMatchTest(test_base.DbTestCase):
def setUp(self):
super(UpdateMatchTest, self).setUp()
Base.metadata.create_all(self.engine)
self.addCleanup(Base.metadata.drop_all, self.engine)
# self.engine.echo = 'debug'
self.session = self.sessionmaker(autocommit=False)
self.addCleanup(self.session.close)
self.session.add_all([
MyModel(
id=1,
uuid='23cb9224-9f8e-40fe-bd3c-e7577b7af37d',
x=5, y='y1', z='z1'),
MyModel(
id=2,
uuid='136254d5-3869-408f-9da7-190e0072641a',
x=6, y='y1', z='z2'),
MyModel(
id=3,
uuid='094eb162-d5df-494b-a458-a91a1b2d2c65',
x=7, y='y1', z='z1'),
MyModel(
id=4,
uuid='94659b3f-ea1f-4ffd-998d-93b28f7f5b70',
x=8, y='y2', z='z2'),
MyModel(
id=5,
uuid='bdf3893c-ee3c-40a0-bc79-960adb6cd1d4',
x=8, y='y2', z=None),
])
self.session.commit()
def _assert_row(self, pk, values):
row = self.session.execute(
sql.select([MyModel.__table__]).where(MyModel.__table__.c.id == pk)
).first()
values['id'] = pk
self.assertEqual(values, dict(row))
def test_update_specimen_successful(self):
uuid = '136254d5-3869-408f-9da7-190e0072641a'
specimen = MyModel(
y='y1', z='z2', uuid=uuid
)
result = self.session.query(MyModel).update_on_match(
specimen,
'uuid',
values={'x': 9, 'z': 'z3'}
)
self.assertEqual(uuid, result.uuid)
self.assertEqual(2, result.id)
self.assertEqual('z3', result.z)
self.assertIn(result, self.session)
self._assert_row(
2,
{
'uuid': '136254d5-3869-408f-9da7-190e0072641a',
'x': 9, 'y': 'y1', 'z': 'z3'
}
)
def test_update_specimen_include_only(self):
uuid = '136254d5-3869-408f-9da7-190e0072641a'
specimen = MyModel(
y='y9', z='z5', x=6, uuid=uuid
)
# Query the object first to test that we merge when the object is
# already cached in the session.
self.session.query(MyModel).filter(MyModel.uuid == uuid).one()
result = self.session.query(MyModel).update_on_match(
specimen,
'uuid',
values={'x': 9, 'z': 'z3'},
include_only=('x', )
)
self.assertEqual(uuid, result.uuid)
self.assertEqual(2, result.id)
self.assertEqual('z3', result.z)
self.assertIn(result, self.session)
self.assertNotIn(result, self.session.dirty)
self._assert_row(
2,
{
'uuid': '136254d5-3869-408f-9da7-190e0072641a',
'x': 9, 'y': 'y1', 'z': 'z3'
}
)
def test_update_specimen_no_rows(self):
specimen = MyModel(
y='y1', z='z3',
uuid='136254d5-3869-408f-9da7-190e0072641a'
)
exc = self.assertRaises(
update_match.NoRowsMatched,
self.session.query(MyModel).update_on_match,
specimen, 'uuid', values={'x': 9, 'z': 'z3'}
)
self.assertEqual("Zero rows matched for 3 attempts", exc.args[0])
def test_update_specimen_process_query_no_rows(self):
specimen = MyModel(
y='y1', z='z2',
uuid='136254d5-3869-408f-9da7-190e0072641a'
)
def process_query(query):
return query.filter_by(x=10)
exc = self.assertRaises(
update_match.NoRowsMatched,
self.session.query(MyModel).update_on_match,
specimen, 'uuid', values={'x': 9, 'z': 'z3'},
process_query=process_query
)
self.assertEqual("Zero rows matched for 3 attempts", exc.args[0])
def test_update_specimen_given_query_no_rows(self):
specimen = MyModel(
y='y1', z='z2',
uuid='136254d5-3869-408f-9da7-190e0072641a'
)
query = self.session.query(MyModel).filter_by(x=10)
exc = self.assertRaises(
update_match.NoRowsMatched,
query.update_on_match,
specimen, 'uuid', values={'x': 9, 'z': 'z3'},
)
self.assertEqual("Zero rows matched for 3 attempts", exc.args[0])
def test_update_specimen_multi_rows(self):
specimen = MyModel(
y='y1', z='z1',
)
exc = self.assertRaises(
update_match.MultiRowsMatched,
self.session.query(MyModel).update_on_match,
specimen, 'y', values={'x': 9, 'z': 'z3'}
)
self.assertEqual("2 rows matched; expected one", exc.args[0])
def test_update_specimen_query_mismatch_error(self):
specimen = MyModel(
y='y1'
)
q = self.session.query(MyModel.x, MyModel.y)
exc = self.assertRaises(
AssertionError,
q.update_on_match,
specimen, 'y', values={'x': 9, 'z': 'z3'},
)
self.assertEqual("Query does not match given specimen", exc.args[0])
def test_custom_handle_failure_raise_new(self):
class MyException(Exception):
pass
def handle_failure(query):
# ensure the query is usable
result = query.count()
self.assertEqual(0, result)
raise MyException("test: %d" % result)
specimen = MyModel(
y='y1', z='z3',
uuid='136254d5-3869-408f-9da7-190e0072641a'
)
exc = self.assertRaises(
MyException,
self.session.query(MyModel).update_on_match,
specimen, 'uuid', values={'x': 9, 'z': 'z3'},
handle_failure=handle_failure
)
self.assertEqual("test: 0", exc.args[0])
def test_custom_handle_failure_cancel_raise(self):
uuid = '136254d5-3869-408f-9da7-190e0072641a'
class MyException(Exception):
pass
def handle_failure(query):
# ensure the query is usable
result = query.count()
self.assertEqual(0, result)
return True
specimen = MyModel(
id=2, y='y1', z='z3', uuid=uuid
)
result = self.session.query(MyModel).update_on_match(
specimen, 'uuid', values={'x': 9, 'z': 'z3'},
handle_failure=handle_failure
)
self.assertEqual(uuid, result.uuid)
self.assertEqual(2, result.id)
self.assertEqual('z3', result.z)
self.assertEqual(9, result.x)
self.assertIn(result, self.session)
def test_update_specimen_on_none_successful(self):
uuid = 'bdf3893c-ee3c-40a0-bc79-960adb6cd1d4'
specimen = MyModel(
y='y2', z=None, uuid=uuid
)
result = self.session.query(MyModel).update_on_match(
specimen,
'uuid',
values={'x': 9, 'z': 'z3'},
)
self.assertIn(result, self.session)
self.assertEqual(uuid, result.uuid)
self.assertEqual(5, result.id)
self.assertEqual('z3', result.z)
self._assert_row(
5,
{
'uuid': 'bdf3893c-ee3c-40a0-bc79-960adb6cd1d4',
'x': 9, 'y': 'y2', 'z': 'z3'
}
)
def test_update_specimen_on_multiple_nonnone_successful(self):
uuid = '094eb162-d5df-494b-a458-a91a1b2d2c65'
specimen = MyModel(
y=('y1', 'y2'), x=(5, 7), uuid=uuid
)
result = self.session.query(MyModel).update_on_match(
specimen,
'uuid',
values={'x': 9, 'z': 'z3'},
)
self.assertIn(result, self.session)
self.assertEqual(uuid, result.uuid)
self.assertEqual(3, result.id)
self.assertEqual('z3', result.z)
self._assert_row(
3,
{
'uuid': '094eb162-d5df-494b-a458-a91a1b2d2c65',
'x': 9, 'y': 'y1', 'z': 'z3'
}
)
def test_update_specimen_on_multiple_wnone_successful(self):
uuid = 'bdf3893c-ee3c-40a0-bc79-960adb6cd1d4'
specimen = MyModel(
y=('y1', 'y2'), x=(8, 7), z=('z1', 'z2', None), uuid=uuid
)
result = self.session.query(MyModel).update_on_match(
specimen,
'uuid',
values={'x': 9, 'z': 'z3'},
)
self.assertIn(result, self.session)
self.assertEqual(uuid, result.uuid)
self.assertEqual(5, result.id)
self.assertEqual('z3', result.z)
self._assert_row(
5,
{
'uuid': 'bdf3893c-ee3c-40a0-bc79-960adb6cd1d4',
'x': 9, 'y': 'y2', 'z': 'z3'
}
)
def test_update_returning_pk_matched(self):
pk = self.session.query(MyModel).\
filter_by(y='y1', z='z2').update_returning_pk(
{'x': 9, 'z': 'z3'},
('uuid', '136254d5-3869-408f-9da7-190e0072641a')
)
self.assertEqual((2,), pk)
self._assert_row(
2,
{
'uuid': '136254d5-3869-408f-9da7-190e0072641a',
'x': 9, 'y': 'y1', 'z': 'z3'
}
)
def test_update_returning_wrong_uuid(self):
exc = self.assertRaises(
update_match.NoRowsMatched,
self.session.query(MyModel).
filter_by(y='y1', z='z2').update_returning_pk,
{'x': 9, 'z': 'z3'},
('uuid', '23cb9224-9f8e-40fe-bd3c-e7577b7af37d')
)
self.assertEqual("No rows matched the UPDATE", exc.args[0])
def test_update_returning_no_rows(self):
exc = self.assertRaises(
update_match.NoRowsMatched,
self.session.query(MyModel).
filter_by(y='y1', z='z3').update_returning_pk,
{'x': 9, 'z': 'z3'},
('uuid', '136254d5-3869-408f-9da7-190e0072641a')
)
self.assertEqual("No rows matched the UPDATE", exc.args[0])
def test_update_multiple_rows(self):
exc = self.assertRaises(
update_match.MultiRowsMatched,
self.session.query(MyModel).
filter_by(y='y1', z='z1').update_returning_pk,
{'x': 9, 'z': 'z3'},
('y', 'y1')
)
self.assertEqual("2 rows matched; expected one", exc.args[0])
class PGUpdateMatchTest(
UpdateMatchTest,
test_base.PostgreSQLOpportunisticTestCase):
pass
class MySQLUpdateMatchTest(
UpdateMatchTest,
test_base.MySQLOpportunisticTestCase):
pass

1335
oslo_db/tests/sqlalchemy/test_utils.py
View File

File diff suppressed because it is too large Load Diff

255
oslo_db/tests/test_api.py
View File

@ -1,255 +0,0 @@
# Copyright (c) 2013 Rackspace Hosting
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""Unit tests for DB API."""
import mock
from oslo_config import cfg
from oslo_utils import importutils
from oslo_db import api
from oslo_db import exception
from oslo_db.tests import utils as test_utils
sqla = importutils.try_import('sqlalchemy')
if not sqla:
raise ImportError("Unable to import module 'sqlalchemy'.")
def get_backend():
return DBAPI()
class DBAPI(object):
def _api_raise(self, *args, **kwargs):
"""Simulate raising a database-has-gone-away error
This method creates a fake OperationalError with an ID matching
a valid MySQL "database has gone away" situation. It also decrements
the error_counter so that we can artificially keep track of
how many times this function is called by the wrapper. When
error_counter reaches zero, this function returns True, simulating
the database becoming available again and the query succeeding.
"""
if self.error_counter > 0:
self.error_counter -= 1
orig = sqla.exc.DBAPIError(False, False, False)
orig.args = [2006, 'Test raise operational error']
e = exception.DBConnectionError(orig)
raise e
else:
return True
def api_raise_default(self, *args, **kwargs):
return self._api_raise(*args, **kwargs)
@api.safe_for_db_retry
def api_raise_enable_retry(self, *args, **kwargs):
return self._api_raise(*args, **kwargs)
def api_class_call1(_self, *args, **kwargs):
return args, kwargs
class DBAPITestCase(test_utils.BaseTestCase):
def test_dbapi_full_path_module_method(self):
dbapi = api.DBAPI('oslo_db.tests.test_api')
result = dbapi.api_class_call1(1, 2, kwarg1='meow')
expected = ((1, 2), {'kwarg1': 'meow'})
self.assertEqual(expected, result)
def test_dbapi_unknown_invalid_backend(self):
self.assertRaises(ImportError, api.DBAPI, 'tests.unit.db.not_existent')
def test_dbapi_lazy_loading(self):
dbapi = api.DBAPI('oslo_db.tests.test_api', lazy=True)
self.assertIsNone(dbapi._backend)
dbapi.api_class_call1(1, 'abc')
self.assertIsNotNone(dbapi._backend)
def test_dbapi_from_config(self):
conf = cfg.ConfigOpts()
dbapi = api.DBAPI.from_config(conf,
backend_mapping={'sqlalchemy': __name__})
self.assertIsNotNone(dbapi._backend)
class DBReconnectTestCase(DBAPITestCase):
def setUp(self):
super(DBReconnectTestCase, self).setUp()
self.test_db_api = DBAPI()
patcher = mock.patch(__name__ + '.get_backend',
return_value=self.test_db_api)
patcher.start()
self.addCleanup(patcher.stop)
def test_raise_connection_error(self):
self.dbapi = api.DBAPI('sqlalchemy', {'sqlalchemy': __name__})
self.test_db_api.error_counter = 5
self.assertRaises(exception.DBConnectionError, self.dbapi._api_raise)
def test_raise_connection_error_decorated(self):
self.dbapi = api.DBAPI('sqlalchemy', {'sqlalchemy': __name__})
self.test_db_api.error_counter = 5
self.assertRaises(exception.DBConnectionError,
self.dbapi.api_raise_enable_retry)
self.assertEqual(4, self.test_db_api.error_counter, 'Unexpected retry')
def test_raise_connection_error_enabled(self):
self.dbapi = api.DBAPI('sqlalchemy',
{'sqlalchemy': __name__},
use_db_reconnect=True)
self.test_db_api.error_counter = 5
self.assertRaises(exception.DBConnectionError,
self.dbapi.api_raise_default)
self.assertEqual(4, self.test_db_api.error_counter, 'Unexpected retry')
@mock.patch('oslo_db.api.time.sleep', return_value=None)
def test_retry_one(self, p_time_sleep):
self.dbapi = api.DBAPI('sqlalchemy',
{'sqlalchemy': __name__},
use_db_reconnect=True,
retry_interval=1)
try:
func = self.dbapi.api_raise_enable_retry
self.test_db_api.error_counter = 1
self.assertTrue(func(), 'Single retry did not succeed.')
except Exception:
self.fail('Single retry raised an un-wrapped error.')
p_time_sleep.assert_called_with(1)
self.assertEqual(
0, self.test_db_api.error_counter,
'Counter not decremented, retry logic probably failed.')
@mock.patch('oslo_db.api.time.sleep', return_value=None)
def test_retry_two(self, p_time_sleep):
self.dbapi = api.DBAPI('sqlalchemy',
{'sqlalchemy': __name__},
use_db_reconnect=True,
retry_interval=1,
inc_retry_interval=False)
try:
func = self.dbapi.api_raise_enable_retry
self.test_db_api.error_counter = 2
self.assertTrue(func(), 'Multiple retry did not succeed.')
except Exception:
self.fail('Multiple retry raised an un-wrapped error.')
p_time_sleep.assert_called_with(1)
self.assertEqual(
0, self.test_db_api.error_counter,
'Counter not decremented, retry logic probably failed.')
@mock.patch('oslo_db.api.time.sleep', return_value=None)
def test_retry_float_interval(self, p_time_sleep):
self.dbapi = api.DBAPI('sqlalchemy',
{'sqlalchemy': __name__},
use_db_reconnect=True,
retry_interval=0.5)
try:
func = self.dbapi.api_raise_enable_retry
self.test_db_api.error_counter = 1
self.assertTrue(func(), 'Single retry did not succeed.')
except Exception:
self.fail('Single retry raised an un-wrapped error.')
p_time_sleep.assert_called_with(0.5)
self.assertEqual(
0, self.test_db_api.error_counter,
'Counter not decremented, retry logic probably failed.')
@mock.patch('oslo_db.api.time.sleep', return_value=None)
def test_retry_until_failure(self, p_time_sleep):
self.dbapi = api.DBAPI('sqlalchemy',
{'sqlalchemy': __name__},
use_db_reconnect=True,
retry_interval=1,
inc_retry_interval=False,
max_retries=3)
func = self.dbapi.api_raise_enable_retry
self.test_db_api.error_counter = 5
self.assertRaises(
exception.DBError, func,
'Retry of permanent failure did not throw DBError exception.')
p_time_sleep.assert_called_with(1)
self.assertNotEqual(
0, self.test_db_api.error_counter,
'Retry did not stop after sql_max_retries iterations.')
class DBRetryRequestCase(DBAPITestCase):
def test_retry_wrapper_succeeds(self):
@api.wrap_db_retry(max_retries=10)
def some_method():
pass
some_method()
def test_retry_wrapper_reaches_limit(self):
max_retries = 2
@api.wrap_db_retry(max_retries=max_retries)
def some_method(res):
res['result'] += 1
raise exception.RetryRequest(ValueError())
res = {'result': 0}
self.assertRaises(ValueError, some_method, res)
self.assertEqual(max_retries + 1, res['result'])
def test_retry_wrapper_exception_checker(self):
def exception_checker(exc):
return isinstance(exc, ValueError) and exc.args[0] < 5
@api.wrap_db_retry(max_retries=10,
exception_checker=exception_checker)
def some_method(res):
res['result'] += 1
raise ValueError(res['result'])
res = {'result': 0}
self.assertRaises(ValueError, some_method, res)
# our exception checker should have stopped returning True after 5
self.assertEqual(5, res['result'])
@mock.patch.object(DBAPI, 'api_class_call1')
@mock.patch.object(api, 'wrap_db_retry')
def test_mocked_methods_are_not_wrapped(self, mocked_wrap, mocked_method):
dbapi = api.DBAPI('oslo_db.tests.test_api')
dbapi.api_class_call1()
self.assertFalse(mocked_wrap.called)
@mock.patch('oslo_db.api.LOG')
def test_retry_wrapper_non_db_error_not_logged(self, mock_log):
# Tests that if the retry wrapper hits a non-db error (raised from the
# wrapped function), then that exception is reraised but not logged.
@api.wrap_db_retry(max_retries=5, retry_on_deadlock=True)
def some_method():
raise AttributeError('test')
self.assertRaises(AttributeError, some_method)
self.assertFalse(mock_log.called)

108
oslo_db/tests/test_concurrency.py
View File

@ -1,108 +0,0 @@
# Copyright 2014 Mirantis.inc
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import sys
import mock
from oslo_db import concurrency
from oslo_db.tests import utils as test_utils
FAKE_BACKEND_MAPPING = {'sqlalchemy': 'fake.db.sqlalchemy.api'}
class TpoolDbapiWrapperTestCase(test_utils.BaseTestCase):
def setUp(self):
super(TpoolDbapiWrapperTestCase, self).setUp()
self.db_api = concurrency.TpoolDbapiWrapper(
conf=self.conf, backend_mapping=FAKE_BACKEND_MAPPING)
# NOTE(akurilin): We are not going to add `eventlet` to `oslo_db` in
# requirements (`requirements.txt` and `test-requirements.txt`) due to
# the following reasons:
# - supporting of eventlet's thread pooling is totally optional;
# - we don't need to test `tpool.Proxy` functionality itself,
# because it's a tool from the third party library;
# - `eventlet` would prevent us from running unit tests on Python 3.x
# versions, because it doesn't support them yet.
#
# As we don't test `tpool.Proxy`, we can safely mock it in tests.
self.proxy = mock.MagicMock()
self.eventlet = mock.MagicMock()
self.eventlet.tpool.Proxy.return_value = self.proxy
sys.modules['eventlet'] = self.eventlet
self.addCleanup(sys.modules.pop, 'eventlet', None)
@mock.patch('oslo_db.api.DBAPI')
def test_db_api_common(self, mock_db_api):
# test context:
# CONF.database.use_tpool == False
# eventlet is installed
# expected result:
# TpoolDbapiWrapper should wrap DBAPI
fake_db_api = mock.MagicMock()
mock_db_api.from_config.return_value = fake_db_api
# get access to some db-api method
self.db_api.fake_call_1
mock_db_api.from_config.assert_called_once_with(
conf=self.conf, backend_mapping=FAKE_BACKEND_MAPPING)
self.assertEqual(fake_db_api, self.db_api._db_api)
self.assertFalse(self.eventlet.tpool.Proxy.called)
# get access to other db-api method to be sure that api didn't changed
self.db_api.fake_call_2
self.assertEqual(fake_db_api, self.db_api._db_api)
self.assertFalse(self.eventlet.tpool.Proxy.called)
self.assertEqual(1, mock_db_api.from_config.call_count)
@mock.patch('oslo_db.api.DBAPI')
def test_db_api_config_change(self, mock_db_api):
# test context:
# CONF.database.use_tpool == True
# eventlet is installed
# expected result:
# TpoolDbapiWrapper should wrap tpool proxy
fake_db_api = mock.MagicMock()
mock_db_api.from_config.return_value = fake_db_api
self.conf.set_override('use_tpool', True, group='database')
# get access to some db-api method
self.db_api.fake_call
# CONF.database.use_tpool is True, so we get tpool proxy in this case
mock_db_api.from_config.assert_called_once_with(
conf=self.conf, backend_mapping=FAKE_BACKEND_MAPPING)
self.eventlet.tpool.Proxy.assert_called_once_with(fake_db_api)
self.assertEqual(self.proxy, self.db_api._db_api)
@mock.patch('oslo_db.api.DBAPI')
def test_db_api_without_installed_eventlet(self, mock_db_api):
# test context:
# CONF.database.use_tpool == True
# eventlet is not installed
# expected result:
# raise ImportError
self.conf.set_override('use_tpool', True, group='database')
sys.modules['eventlet'] = None
self.assertRaises(ImportError, getattr, self.db_api, 'fake')

40
oslo_db/tests/utils.py
View File

@ -1,40 +0,0 @@
# Copyright 2010-2011 OpenStack Foundation
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import contextlib
from oslo_config import cfg
from oslotest import base as test_base
from oslotest import moxstubout
import six
if six.PY3:
@contextlib.contextmanager
def nested(*contexts):
with contextlib.ExitStack() as stack:
yield [stack.enter_context(c) for c in contexts]
else:
nested = contextlib.nested
class BaseTestCase(test_base.BaseTestCase):
def setUp(self, conf=cfg.CONF):
super(BaseTestCase, self).setUp()
moxfixture = self.useFixture(moxstubout.MoxStubout())
self.mox = moxfixture.mox
self.stubs = moxfixture.stubs
self.conf = conf
self.addCleanup(self.conf.reset)

3
releasenotes/notes/add-reno-e5c2f63e73c25959.yaml
View File

@ -1,3 +0,0 @@
---
other:
- Introduce reno for deployer release notes.

7
releasenotes/notes/connection_debug_min_max-bf6d53d49be7ca52.yaml
View File

@ -1,7 +0,0 @@
---
upgrade:
- The allowed values for the ``connection_debug`` option are now restricted to
the range between 0 and 100 (inclusive). Previously a number lower than 0
or higher than 100 could be given without error. But now, a
``ConfigFileValueError`` will be raised when the option value is outside this
range.

7
releasenotes/notes/deprecate_config_sqlite_db-bd41d49343049319.yaml
View File

@ -1,7 +0,0 @@
---
deprecations:
- |
The configuration option ``sqlite_db`` is now deprecated and
will be removed in the future. Please use configuration
option ``connection`` or ``slave_connection`` to connect to the database.

6
releasenotes/notes/enginefacade_decorators-4660862fe22d2669.yaml
View File

@ -1,6 +0,0 @@
---
features:
- enginefacade decorators can now be used for class and instance methods,
which implicitly receive the first positional argument. Previously, it
was required that all decorated functions receive a context value as the
first argument.

25
releasenotes/notes/increase-default-max-overflow-0af787268807f926.yaml
View File

@ -1,25 +0,0 @@
---
upgrade:
- |
The default value of ``max_overflow`` config option
has been increased from 10 to 50 in order to allow
OpenStack services heavily using DBs to better handle
spikes of concurrent requests and lower the probability
of getting a pool timeout issue.
This change potentially leads to increasing of the number
of open connections to an RDBMS server. Depending on the
configuration, you may see "too many connections" errors
in logs of OpenStack services / RDBMS server. The max limit of
connections can be set by the means of these config options:
http://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_max_connections
http://www.postgresql.org/docs/current/static/runtime-config-connection.html#GUC-MAX-CONNECTIONS
For details, please see the following LP:
https://bugs.launchpad.net/oslo.db/+bug/1535375
and the ML thread:
http://lists.openstack.org/pipermail/openstack-dev/2015-December/082717.html

6
releasenotes/notes/wrap_db_retry-34c7ff2d82afa3f5.yaml
View File

@ -1,6 +0,0 @@
---
fixes:
- Decorator ``oslo_db.api.wrap_db_retry`` now defaults to 10 retries.
Previously the number of attempts was 0, and users had to explicitly
pass ``max_retry_interval`` value greater than 0 to actually enable
retries on errors.

0
releasenotes/source/_static/.placeholder
View File

0
releasenotes/source/_templates/.placeholder
View File

277
releasenotes/source/conf.py
View File

@ -1,277 +0,0 @@
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# oslo.db Release Notes documentation build configuration file, created by
# sphinx-quickstart on Tue Nov 3 17:40:50 2015.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
# sys.path.insert(0, os.path.abspath('.'))
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'oslosphinx',
'reno.sphinxext',
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
# source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'oslo.db Release Notes'
copyright = u'2016, oslo.db Developers'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
# The full version, including alpha/beta/rc tags.
import pkg_resources
release = pkg_resources.get_distribution('oslo.db').version
# The short X.Y version.
version = release
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
# language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
# today = ''
# Else, today_fmt is used as the format for a strftime call.
# today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = []
# The reST default role (used for this markup: `text`) to use for all
# documents.
# default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
# add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
# add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
# show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
# modindex_common_prefix = []
# If true, keep warnings as "system message" paragraphs in the built documents.
# keep_warnings = False
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'default'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
# html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
# html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
# html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
# html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
# html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
# html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
# directly to the root of the documentation.
# html_extra_path = []
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
# html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
# html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
# html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
# html_additional_pages = {}
# If false, no module index is generated.
# html_domain_indices = True
# If false, no index is generated.
# html_use_index = True
# If true, the index is split into individual pages for each letter.
# html_split_index = False
# If true, links to the reST sources are added to the pages.
# html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
# html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
# html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
# html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
# html_file_suffix = None
# Output file base name for HTML help builder.
htmlhelp_basename = 'oslo.configReleaseNotesdoc'
# -- Options for LaTeX output ---------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
# 'preamble': '',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
('index', 'oslo.configReleaseNotes.tex',
u'oslo.db Release Notes Documentation',
u'oslo.db Developers', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
# latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
# latex_use_parts = False
# If true, show page references after internal links.
# latex_show_pagerefs = False
# If true, show URL addresses after external links.
# latex_show_urls = False
# Documents to append as an appendix to all manuals.
# latex_appendices = []
# If false, no module index is generated.
# latex_domain_indices = True
# -- Options for manual page output ---------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
('index', 'oslo.configreleasenotes',
u'oslo.db Release Notes Documentation',
[u'oslo.db Developers'], 1)
]
# If true, show URL addresses after external links.
# man_show_urls = False
# -- Options for Texinfo output -------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
('index', 'oslo.dbReleaseNotes',
u'oslo.db Release Notes Documentation',
u'oslo.db Developers', 'oslo.configReleaseNotes',
'An OpenStack library for parsing configuration options from the command'
' line and configuration files.',
'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
# texinfo_appendices = []
# If false, no module index is generated.
# texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
# texinfo_show_urls = 'footnote'
# If true, do not generate a @detailmenu in the "Top" node's menu.
# texinfo_no_detailmenu = False

10
releasenotes/source/index.rst
View File

@ -1,10 +0,0 @@
=======================
oslo.db Release Notes
=======================
.. toctree::
:maxdepth: 1
unreleased
liberty
mitaka

6
releasenotes/source/liberty.rst
View File

@ -1,6 +0,0 @@
==============================
Liberty Series Release Notes
==============================
.. release-notes::
:branch: origin/stable/liberty

89
releasenotes/source/locale/en_GB/LC_MESSAGES/releasenotes.po
View File

@ -1,89 +0,0 @@
# Andi Chandler <andi@gowling.com>, 2016. #zanata
msgid ""
msgstr ""
"Project-Id-Version: oslo.db Release Notes 4.6.1.dev51\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2016-06-27 15:51+0000\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"PO-Revision-Date: 2016-06-28 05:55+0000\n"
"Last-Translator: Andi Chandler <andi@gowling.com>\n"
"Language-Team: English (United Kingdom)\n"
"Language: en-GB\n"
"X-Generator: Zanata 3.7.3\n"
"Plural-Forms: nplurals=2; plural=(n != 1)\n"
msgid "2.6.0-7"
msgstr "2.6.0-7"
msgid "4.6.0"
msgstr "4.6.0"
msgid "For details, please see the following LP:"
msgstr "For details, please see the following LP:"
msgid "Introduce reno for deployer release notes."
msgstr "Introduce reno for deployer release notes."
msgid "Liberty Series Release Notes"
msgstr "Liberty Series Release Notes"
msgid "Mitaka Series Release Notes"
msgstr "Mitaka Series Release Notes"
msgid "Other Notes"
msgstr "Other Notes"
msgid ""
"The default value of ``max_overflow`` config option has been increased from "
"10 to 50 in order to allow OpenStack services heavily using DBs to better "
"handle spikes of concurrent requests and lower the probability of getting a "
"pool timeout issue."
msgstr ""
"The default value of ``max_overflow`` config option has been increased from "
"10 to 50 in order to allow OpenStack services heavily using DBs to better "
"handle spikes of concurrent requests and lower the probability of getting a "
"pool timeout issue."
msgid ""
"This change potentially leads to increasing of the number of open "
"connections to an RDBMS server. Depending on the configuration, you may see "
"\"too many connections\" errors in logs of OpenStack services / RDBMS "
"server. The max limit of connections can be set by the means of these config "
"options:"
msgstr ""
"This change potentially leads to increasing of the number of open "
"connections to an RDBMS server. Depending on the configuration, you may see "
"\"too many connections\" errors in logs of OpenStack services / RDBMS "
"server. The max limit of connections can be set by the means of these config "
"options:"
msgid "Unreleased Release Notes"
msgstr "Unreleased Release Notes"
msgid "Upgrade Notes"
msgstr "Upgrade Notes"
msgid "and the ML thread:"
msgstr "and the ML thread:"
msgid ""
"http://dev.mysql.com/doc/refman/5.7/en/server-system-variables."
"html#sysvar_max_connections http://www.postgresql.org/docs/current/static/"
"runtime-config-connection.html#GUC-MAX-CONNECTIONS"
msgstr ""
"http://dev.mysql.com/doc/refman/5.7/en/server-system-variables."
"html#sysvar_max_connections http://www.postgresql.org/docs/current/static/"
"runtime-config-connection.html#GUC-MAX-CONNECTIONS"
msgid ""
"http://lists.openstack.org/pipermail/openstack-dev/2015-December/082717.html"
msgstr ""
"http://lists.openstack.org/pipermail/openstack-dev/2015-December/082717.html"
msgid "https://bugs.launchpad.net/oslo.db/+bug/1535375"
msgstr "https://bugs.launchpad.net/oslo.db/+bug/1535375"
msgid "oslo.db Release Notes"
msgstr "oslo.db Release Notes"

6
releasenotes/source/mitaka.rst
View File

@ -1,6 +0,0 @@
===================================
Mitaka Series Release Notes
===================================
.. release-notes::
:branch: origin/stable/mitaka

5
releasenotes/source/unreleased.rst
View File

@ -1,5 +0,0 @@
==========================
Unreleased Release Notes
==========================
.. release-notes::

15
requirements.txt
View File

@ -1,15 +0,0 @@
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
pbr>=1.6 # Apache-2.0
alembic>=0.8.4 # MIT
debtcollector>=1.2.0 # Apache-2.0
oslo.i18n>=2.1.0 # Apache-2.0
oslo.config>=3.14.0 # Apache-2.0
oslo.context>=2.4.0 # Apache-2.0
oslo.utils>=3.16.0 # Apache-2.0
SQLAlchemy<1.1.0,>=1.0.10 # MIT
sqlalchemy-migrate>=0.9.6 # Apache-2.0
stevedore>=1.16.0 # Apache-2.0
six>=1.9.0 # MIT

96
setup.cfg
View File

@ -1,96 +0,0 @@
[metadata]
name = oslo.db
summary = Oslo Database library
description-file =
README.rst
author = OpenStack
author-email = openstack-dev@lists.openstack.org
home-page = http://wiki.openstack.org/wiki/Oslo#oslo.db
classifier =
Environment :: OpenStack
Intended Audience :: Information Technology
Intended Audience :: System Administrators
License :: OSI Approved :: Apache Software License
Operating System :: POSIX :: Linux
Programming Language :: Python
Programming Language :: Python :: 2
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
Programming Language :: Python :: 3.4
Programming Language :: Python :: 3.5
[extras]
# So e.g. nova can test-depend on oslo.db[mysql]
mysql =
PyMySQL>=0.6.2 # MIT License
# or oslo.db[mysql-c]
mysql-c =
MySQL-python:python_version=='2.7' # GPL with FOSS exception
# or oslo.db[postgresql]
postgresql =
psycopg2>=2.5 # LGPL/ZPL
# Dependencies for testing oslo.db itself.
test =
hacking<0.11,>=0.10.0
coverage>=3.6 # Apache-2.0
doc8 # Apache-2.0
eventlet!=0.18.3,>=0.18.2 # MIT
fixtures>=3.0.0 # Apache-2.0/BSD
mock>=2.0 # BSD
python-subunit>=0.0.18 # Apache-2.0/BSD
sphinx!=1.3b1,<1.3,>=1.2.1 # BSD
oslosphinx!=3.4.0,>=2.5.0 # Apache-2.0
oslotest>=1.10.0 # Apache-2.0
testrepository>=0.0.18 # Apache-2.0/BSD
testtools>=1.4.0 # MIT
os-testr>=0.7.0 # Apache-2.0
reno>=1.8.0 # Apache2
fixtures =
testresources>=0.2.4 # Apache-2.0/BSD
testscenarios>=0.4 # Apache-2.0/BSD
pifpaf =
pifpaf>=0.10.0 # Apache-2.0
[files]
packages =
oslo_db
[entry_points]
oslo.config.opts =
oslo.db = oslo_db.options:list_opts
oslo.db.concurrency = oslo_db.concurrency:list_opts
oslo.db.migration =
alembic = oslo_db.sqlalchemy.migration_cli.ext_alembic:AlembicExtension
migrate = oslo_db.sqlalchemy.migration_cli.ext_migrate:MigrateExtension
[wheel]
universal = 1
[build_sphinx]
source-dir = doc/source
build-dir = doc/build
all_files = 1
[upload_sphinx]
upload-dir = doc/build/html
[compile_catalog]
directory = oslo_db/locale
domain = oslo_db
[update_catalog]
domain = oslo_db
output_dir = oslo_db/locale
input_file = oslo_db/locale/oslo_db.pot
[extract_messages]
keywords = _ gettext ngettext l_ lazy_gettext
mapping_file = babel.cfg
output_file = oslo_db/locale/oslo_db.pot
[pbr]
warnerrors = True
autodoc_index_modules = True
autodoc_exclude_modules =
oslo_db.tests.*

29
setup.py
View File

@ -1,29 +0,0 @@
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT
import setuptools
# In python < 2.7.4, a lazy loading of package `pbr` will break
# setuptools if some other modules registered functions in `atexit`.
# solution from: http://bugs.python.org/issue15881#msg170215
try:
import multiprocessing # noqa
except ImportError:
pass
setuptools.setup(
setup_requires=['pbr>=1.8'],
pbr=True)

16
tools/pretty_tox.sh
View File

@ -1,16 +0,0 @@
#!/usr/bin/env bash
# return nonzero exit status of rightmost command, so that we
# get nonzero exit on test failure without halting subunit-trace
set -o pipefail
TESTRARGS=$1
python setup.py testr --testr-args="--subunit $TESTRARGS" | subunit-trace -f
retval=$?
# NOTE(mtreinish) The pipe above would eat the slowest display from pbr's testr
# wrapper so just manually print the slowest tests
echo -e "\nSlowest Tests:\n"
testr slowest
exit $retval

7
tools/run-pifpaf-tests.sh
View File

@ -1,7 +0,0 @@
#!/bin/bash
set -e
# Replace mysql:// by mysql+pymysql:// and add sqlite
export OS_TEST_DBAPI_ADMIN_CONNECTION="${OS_TEST_DBAPI_ADMIN_CONNECTION/#mysql:/mysql+pymysql:};sqlite://"
echo $OS_TEST_DBAPI_ADMIN_CONNECTION
tools/pretty_tox.sh $*
TEST_EVENTLET=1 tools/pretty_tox.sh $*

72
tox.ini
View File

@ -1,72 +0,0 @@
[tox]
minversion = 1.8
envlist = py35,py34,py27,pep8,pip-missing-reqs
[testenv]
whitelist_externals = bash
env
setenv =
VIRTUAL_ENV={envdir}
BASECOMMAND=bash tools/pretty_tox.sh
{postgresql,all}: PIFPAF_POSTGRESQL=pifpaf -g OS_TEST_DBAPI_ADMIN_CONNECTION run postgresql --
{mysql,all}: PIFPAF_MYSQL=pifpaf -g OS_TEST_DBAPI_ADMIN_CONNECTION run mysql --
{mysql,postgresql,all}: BASECOMMAND={toxinidir}/tools/run-pifpaf-tests.sh
deps = .[test,fixtures,mysql,postgresql]
{postgresql,mysql,all}: .[pifpaf]
commands =
{env:PIFPAF_MYSQL:} {env:PIFPAF_POSTGRESQL:} {env:BASECOMMAND:} '{posargs}'
passenv = OS_TEST_DBAPI_ADMIN_CONNECTION
[testenv:sqla_09]
commands = pip install SQLAlchemy>=0.9.0,!=0.9.5,<1.0.0
python setup.py testr --slowest --testr-args='{posargs}'
[testenv:py27]
commands =
env TEST_EVENTLET=0 bash tools/pretty_tox.sh '{posargs}'
env TEST_EVENTLET=1 bash tools/pretty_tox.sh '{posargs}'
[testenv:mysql-python]
deps = .[mysql-c,postgresql,test,fixtures]
setenv =
{[testenv]setenv}
OS_TEST_DBAPI_ADMIN_CONNECTION=mysql://openstack_citest:openstack_citest@localhost/;postgresql://openstack_citest:openstack_citest@localhost/postgres;sqlite://
[testenv:pep8]
commands = flake8
[testenv:venv]
commands = {posargs}
[testenv:cover]
commands = python setup.py test --coverage --coverage-package-name=oslo_db --testr-args='{posargs}'
[testenv:docs]
commands =
doc8 -e .rst CONTRIBUTING.rst HACKING.rst README.rst doc/source
python setup.py build_sphinx
[testenv:releasenotes]
commands = sphinx-build -a -E -W -d releasenotes/build/doctrees -b html releasenotes/source releasenotes/build/html
[flake8]
# E123, E125 skipped as they are invalid PEP-8.
ignore = E123,E125
show-source = True
exclude=.venv,.git,.tox,dist,doc,*lib/python*,*egg,build
[hacking]
import_exceptions =
oslo_db._i18n
[testenv:pip-missing-reqs]
# do not install test-requirements as that will pollute the virtualenv for
# determining missing packages
# this also means that pip-missing-reqs must be installed separately, outside
# of the requirements.txt files
deps = pip_missing_reqs
commands = pip-missing-reqs -d --ignore-module=oslo_db* --ignore-module=pkg_resources --ignore-file=oslo_db/tests/* oslo_db