489 Commits

Author SHA1 Message Date
Chuck Short
8082f31104 Convert unicode for python3 portability
From http://docs.python.org/3.1/whatsnew/3.0.html:

"Python 3.0 uses the concepts of text and (binary) data instead of
Unicode strings and 8-bit strings."

Use six.text_type to Type for representing (Unicode) textual data. This
is unicode() in Python 2 and str in Python 3.

Change-Id: I3da268a714a34a8e626a2590f01b86e414dc3411
Signed-off-by: Chuck Short <chuck.short@canonical.com>
2013-05-01 11:22:23 -05:00
Mark McLoughlin
c952a93db1 Add test coverage for sqlite regexp function
Add some tests which for the regexp function which we install for
sqlite.

Need test coverage to catch e.g. a regression of the fix for
bug #1031846.

Change-Id: I4bd77d0d93c4fc85588e6e6d5a559e68133dbf73
2013-05-02 22:09:26 +01:00
Chuck Short
0c877f584a Use range rather than xrange
Use range rather than xrange to improve python3
compatibiltiy.

Change-Id: I5a4434bbd2e5d509f84c3aa668eea8e46262816f
Signed-off-by: Chuck Short <chuck.short@canonical.com>
2013-05-01 10:25:40 -05:00
Gary Kotton
bf46aac8ca Add support to clear DB
Fixes bug 1171740

The method cleanup() can be called when a test is complete,
for example in the tearDown method.

Change-Id: Id30bcb0d7ef167f6cb3b1bdea06f6421e1234a86
2013-04-22 08:12:38 +00:00
Gary Kotton
bb0beaf1d2 Add enforcement for foreign key contraints with sqlite
Fixes bug 1171732

When creating an engine the called can indicate whether or
not to enforce the constraints for sqlite.

Change-Id: I1d5820376f60388c084afea500f807cd25541800
2013-04-22 07:50:31 +00:00
Dirk Mueller
67931e87ae Improve Python 3.x compatibility
Mechanical translation of deprecated constructs
to 3.x compatible variants.

Change-Id: I4988d0ac656903e0d0320aaa8361d4eeb774a0f9
2013-04-22 03:42:48 +02:00
Zhongyue Luo
1d6c260767 Removes metadata from ModelBase
Next step is adding metadata to nova.db.sqlalchemy.models.NovaBase

Fixes bug #1119699

Change-Id: I306a061e0b17660f612a111067ac618342f4ee99
2013-04-10 15:33:42 +08:00
Zhongyue Luo
ee71fe84c0 Removes created_at, updated_at from ModelBase
Created a 'TimestampMixin' class.
Next step is to update Nova which is the only project using oslo.db

Fixes bug #1119702

Change-Id: I1ffcc09a2971e6e6102da7ecb855a2837a159baf
2013-04-10 10:31:03 +08:00
Zhongyue Luo
8ad86dfb09 Fixes private functions private
Fixed functions below to private APIs.

raise_if_duplicate_entry_error()
raise_if_deadlock_error()
wrap_db_error()
synchronous_switch_listener()
add_regexp_listener()
greenthread_yield()
ping_listener()
is_db_connection_error()
patch_mysqldb_with_stacktrace_comments()

Fixes bug #1119709

Change-Id: I61f700cdbae0126a8b89f6895e4332d695191dd0
2013-04-10 17:02:17 +08:00
Michael J Fork
4428bb0720 Mark sql_connection with secret flag
sql_connection contains the password in some setups and marking as
secret prevents accidental logging

fixes bug 1160680

Change-Id: Ib2d00219ea40a010c62e6a29045309f030e6de28
2013-03-27 02:34:49 +00:00
Davanum Srinivas
9f21f6a508 Fix Copyright Headers - Rename LLC to Foundation
One code change, rest are in headers

Change-Id: I73f59681358629e1ad74e49d3d3ca13fcb5c2eb1
2013-03-11 14:49:31 -04:00
Zhongyue Luo
82c706bae1 Fixes import order nits
Change-Id: I6085bb4a0b990985c8f7a013c89b7d5acafdf312
2013-02-26 14:59:04 +08:00
Chris Behrens
338b7a11a9 Clean up sqlalchemy exception code
Moves DB exceptions that can be shared between DB implementations into
their own module.

Adds DBDeadlock() exception wrapping.  Nova has its own code for
determining Deadlock and it's better to consolidate it with
DBDuplicateKey checking.

Change-Id: I108bd0da2a14d62e460a997b1472f0b65bfc9b95
2013-02-22 16:13:27 +00:00
Chris Behrens
167251e28f Move DB thread pooling to DB API loader
Fixes bug 1128605

The dbpool code in sqlalchemy session is the wrong place to implement
thread pooling as it wraps each individual SQL call to run in its own
thread.  When combined with SQL server locking, all threads can be eaten
waiting on locks with none available to run a 'COMMIT'.

The correct place to do thread pooling is around each DB API call.

This patch removes dbpool from sqlalchemy and creates a common DB API
loader for all openstack projects which implements the following
configuration options:

db_backend: Full path to DB API backend module (or a known short name if
            a project chooses to implement a mapping)
dbapi_use_tpool: True or False whether to use thread pooling around all
                 DB API calls.

DB backend modules must implement a 'get_backend()' method.

Example usage for nova/db/api.py would be:

"""
from nova.openstack.common.db import api as db_api

_KNOWN_BACKENDS = {'sqlalchemy': 'nova.db.sqlalchemy.api'}

IMPL = db_api.DBAPI(backend_mapping=_KNOWN_BACKENDS)
"""

NOTE: Enabling thread pooling will be broken until this issue is
resolved in eventlet _OR_ until we modify our eventlet.monkey_patch()
calls to include 'thread=False':

https://bitbucket.org/eventlet/eventlet/issue/137/

Change-Id: Idf14563ea07cf8ccf2a77b3f53659d8528927fc7
2013-02-18 02:35:34 +00:00
Mark McLoughlin
44151a95d4 Use oslo-config-2013.1b3
The cfg API is now available via the oslo-config library, so switch to
it and remove the copied-and-pasted version.

Add the 2013.1b3 tarball to tools/pip-requires - this will be changed
to 'oslo-config>=2013.1' when oslo-config is published to pypi. This
will happen in time for grizzly final.

Remove the 'deps = pep8==1.3.3' and 'deps = pyflakes' from tox.ini as
it means all the other deps get installed with easy_install which can't
install oslo-config from the URL.

Change-Id: I4815aeb8a9341a31a250e920157f15ee15cfc5bc
2013-02-17 15:22:44 +00:00
Doug Hellmann
0d9f3f6a75 Add join_consumer_pool() to RPC connections
blueprint move-listener-framework-oslo
bug 1047015
bug 1111632

Ceilometer and Quantum use private methods of the RPC connection
object to configure themselves to listen to a queue shared among a
group of workers. This change adds a public method to the RPC
connection to support this use case, without resorting to using
private API calls.

Change-Id: I3a89f1dfdcf8accca70cf305f7a31315bea093d8
Signed-off-by: Doug Hellmann <doug.hellmann@dreamhost.com>
2013-02-13 18:02:22 -05:00
Mark McLoughlin
da135181cb Use importutils.try_import() for MySQLdb
This is a common pattern we've adopted elsewhere, so use it here too.

Change-Id: I2e0947b4857005b478e796e9c9a7c05ea1f0d926
2013-02-04 16:24:10 +00:00
Mark McLoughlin
0b4a7b72bf Minor tweak to make update.py happy
Fairly lame to do this instead of just fixing update.py, but we
seem to do it this way everywhere else, so ...

Change-Id: Ie1cebcfec57c6138b9b726d7d3bc6e0ea456f146
2013-02-04 17:47:27 +00:00
Mark McLoughlin
b69fa91b00 Remove pointless use of OpenStackException
The use of the base exception does nothing for the test but means
the "dead module walking" exception module.

Change-Id: Ibd7bb8aa526be366f005ab5e12fd4da89a976c57
2013-02-04 16:53:40 +00:00
Mark McLoughlin
aaffdc30e1 Remove unused context from test_sqlalchemy
Change-Id: I0b3c7341e91cfc57aed625754bc9bf009ef70b14
2013-02-04 16:51:30 +00:00
Mark McLoughlin
31af0d3011 Remove openstack.common.db.common
Since this only contains exceptions that are either used in session.py
or utils.py (not both), it seems sensible to move them into the modules
where they are used.

Also, remove the use of the Invalid base exception class since we don't
seem to be making use of the base class anywhere by catching it rather
than the more specialized exceptions.

Change-Id: Ib05bb2e0a9494e1dc60c60b8eee0e76b5d2ee555
2013-02-04 16:40:01 +00:00
Boris Pavlovic
b588d06635 Provide creating real unique constraints for columns
Main issue is that we can't create unique constraint for columns, because
we are using soft deletion of entries (set `deleted` column to True).

The main idea is to use `deleted` columns to create unique constraint for
columns. For example (`col1`, `deleted`).

To make (`col1`, `deleted`) unique after entry deletion, we should assign the
value of `id` to `deleted` column.

Change type of `deleted` column from Boolean to table.id.type for all tables.

Change models.soft_delete() method to assign table.id instead of True to
`deleted` column.

Change query.soft_delete() method to assign literal_column("id") instead of True

blueprint db-unique-keys

Change-Id: Ie1f67f49a5d085e6371efb63fc23a1c8b25d9464
2013-02-04 17:34:55 +00:00
Boris Pavlovic
3d823a5239 Fix missing wrap_db_error for Session.execute() method
We should add wrap_db_error for Session.execute() method, because
Query.update() method will call Session.execute() and it is not inside
Session.flush(), so exceptions from Query.update() wouldn't be wrapped.

Add test for Session.flush() wrapper
Add test for Session.execute() wrapper

Fixes bug 1107890

Change-Id: I96894e502f9f279999b61a1e66469e34a07a022d
2013-02-04 16:12:13 +00:00
Brian Elliott
d383518f98 Fix eventlet/mysql db pooling code
Eventlet versions >= 0.10 return a tuple from
db_pool.ConnectionPool().create(), not a scalar.

Also, set the FOUND_ROWS client flag, which sqlalchemy
appears to require to sanity check the number of rows
touch.

bug 1097992

Change-Id: I64e798660c506fcd13947a232cb7832db0318360
2013-02-04 15:30:32 +00:00
Mark McLoughlin
097720be8e Add missing DBDuplicateEntry
The session.py part of nova's 8835866 was imported, but not the newly
added exception.

Change-Id: I0d79a8536f0c8f14cb062dc61c5fce100b63fd64
2013-02-04 16:32:06 +00:00
Mark McLoughlin
1644e570d3 Be explicit about set_default() parameters
We probably only want to support people using set_defaults() for a
limited set of config options, so explicitly list those as parameters
to the method. This is what we do for rpc and log options too.

Change-Id: I3bfc5355e82eec17480b9547662e847dc0d74138
2013-02-04 16:26:39 +00:00
Mark McLoughlin
0fd36daea0 Remove duplicate DB options
sql_dbpool_enable and sql_connection_trace are declared twice.

Looks like this was a merge conflict when sql_dbpool_enable changed
from int to bool.

Change-Id: I553c5beea916e0ee296648342324fd53167c073a
2013-02-04 15:32:33 +00:00
Mark McLoughlin
4a7b60b396 Eliminate gratuitous DB difference vs Nova
Looks like this was an abritrary cleanup done as part of the import. It
conflicts with a more recent change in Nova.

We can move to using try_import() once we've killed the Nova copy
of this code.

Change-Id: I5c67d3fbd846a262feb724e26b14447a76561ae6
2013-02-04 15:23:58 +00:00
Eric Windisch
bed94c3b25 Import sqlalchemy session/models/utils
Bring in session, base model, utilities, and tests for sqlalchemy
from Nova.

Add sqlalchemy to pip-requires and and python-mysql to test-requires.

Partially implements blueprint common-db

Change-Id: I3e0065cdac87e10c4e0742d66c293c72bb3acbb2
2012-12-04 10:40:10 -05:00
Joe Heck
4ee08da470 updating sphinx documentation
* adding openstack theming (copied from keystone project theme)
* updating .gitignore to ignore generated API docs
* updated formatting in index.rst page
* updaed openstack/common/processutils.py to match hacking docstring
* updated docstrings to resolve sphinx warnings

Change-Id: Ie89afe20eeab1efd2daf440fc65ccdf90f723c51
2012-11-11 21:00:42 +00:00
Jeremy Stanley
745278192e Correcting openstack-common mv to oslo-incubator.
The earlier rename of openstack-common to oslo was incorrect; it should
have been oslo-incubator instead per the 2102-11-06 Project meeting log.
This change corrects its .gitreview file to match CI changes already
merged.

Change-Id: I9720387576a52654383bfc14f557ed1e06cc0a91
2012-11-11 23:44:12 +00:00
Jeremy Stanley
9c6394ed9a Update .gitreview for oslo.
At the Design Summit it was decided that the OpenStack common modules
would be named Oslo as part of an effort to convert them into a core
project housing a collection of library dependencies. This handles the
necessary update to .gitreview reflecting the new remote name.

Change-Id: I350ff95213efe85579bc6fdce125237cb1f7f614
2012-11-11 19:07:38 +00:00
Doug Hellmann
0fd6328e53 .gitignore updates for generated files
Ignore files created by packaging, builds, etc.

Change-Id: I9c1eed8c9791548b46e1532b345d80f3d84d4890
Signed-off-by: Doug Hellmann <doug.hellmann@dreamhost.com>
2012-09-28 15:57:15 -04:00
Monty Taylor
5b021a9b60 Updated tox config for multi-python testing.
Change-Id: Ib1f768c6783db0e63cb51dfc767ae5a31a295c76
2012-02-06 09:49:09 -05:00
Monty Taylor
11ad398568 Added .gitreview file. 2012-01-06 10:12:58 -08:00
Jason Kölker
6ca49631c3 ignore cover's html directory 2011-11-30 19:14:47 -06:00
Rajaram Mallya
388331fe2c Rajaram/Vinkesh|increased tests for Request and Response serializers 2011-09-14 15:52:30 +05:30
Rajaram Mallya
6a21ed8e7f Rajaram/Vinkesh|Added nova's serializaiton classes into common 2011-09-07 17:05:16 +05:30
Jay Pipes
18db4cca0a Initial skeleton project 2011-07-26 09:05:53 -04:00