Some introductory/overview docs added to db_pool, moved DatabaseConnector class towards bottom of file.
This commit is contained in:
rdw
@@ -21,6 +21,74 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
||||
THE SOFTWARE.
|
||||
|
||||
|
||||
The db_pool module is useful for managing database connections. It provides three primary benefits: cooperative yielding during database operations, concurrency limiting to a database host, and connection reuse. db_pool is intended to be db-agnostic, compatible with any DB-API 2.0 database module; however it has currently only been tested and used with MySQLdb.
|
||||
|
||||
== ConnectionPool ==
|
||||
|
||||
A ConnectionPool object represents a pool of connections open to a particular database. The arguments to the constructor include the database-software-specific module, the host name, and the credentials required for authentication. After construction, the ConnectionPool object decides when to create and sever connections with the target database.
|
||||
|
||||
>>> cp = ConnectionPool(MySQLdb, host='localhost', user='root', passwd='')
|
||||
|
||||
Once you have this pool object, you connect to the database by calling get() on it:
|
||||
|
||||
>>> conn = cp.get()
|
||||
|
||||
This call may either create a new connection, or reuse an existing open connection, depending on its internal state. You can then use the connection object as normal. When done, you return the connection to the pool in one of three ways: pool.put(), conn.close(), or conn.__del__().
|
||||
|
||||
>>> conn = cp.get()
|
||||
>>> try:
|
||||
>>> conn.cursor().execute('SELECT NOW()')
|
||||
>>> finally:
|
||||
>>> cp.put(conn)
|
||||
|
||||
or
|
||||
|
||||
>>> conn = cp.get()
|
||||
>>> conn.cursor().execute('SELECT NOW()')
|
||||
>>> conn.close()
|
||||
|
||||
or
|
||||
|
||||
>>> conn = cp.get()
|
||||
>>> conn.cursor().execute('SELECT NOW()')
|
||||
>>> del conn
|
||||
|
||||
Try/finally is the preferred method, because it has no reliance on __del__ being called by garbage collection.
|
||||
|
||||
After you've returned a connection object to the pool, it becomes useless and will raise exceptions if any of its methods are called.
|
||||
|
||||
=== Constructor Arguments ===
|
||||
|
||||
In addition to the database credentials, there are a bunch of keyword constructor arguments to the ConnectionPool that are useful.
|
||||
|
||||
* min_size, max_size : The normal Pool arguments. max_size is the most important constructor argument -- it determines the number of concurrent connections can be open to the destination database. min_size is not very useful.
|
||||
* max_idle : Connections are only allowed to remain unused in the pool for a limited amount of time. An asynchronous timer periodically wakes up and closes any connections in the pool that have been idle for longer than they are supposed to be. Without this parameter, the pool would tend to have a 'high-water mark', where the number of connections open at a given time corresponds to the peak historical demand. This number only has effect on the connections in the pool itself -- if you take a connection out of the pool, you can hold on to it for as long as you want. If this is set to 0, every connection is closed upon its return to the pool.
|
||||
* max_age : The lifespan of a connection. This works much like max_idle, but the timer is measured from the connection's creation time, and is tracked throughout the connection's life. This means that if you take a connection out of the pool and hold on to it for some lengthy operation that exceeds max_age, upon putting the connection back in to the pool, it will be closed. Like max_idle, max_age will not close connections that are taken out of the pool, and, if set to 0, will cause every connection to be closed when put back in the pool.
|
||||
* connect_timeout : How long to wait before raising an exception on connect(). If the database module's connect() method takes too long, it raises a ConnectTimeout exception from the get() method on the pool.
|
||||
|
||||
== DatabaseConnector ==
|
||||
|
||||
If you want to connect to multiple databases easily (and who doesn't), the DatabaseConnector is for you. It's a pool of pools, containing a ConnectionPool for every host you connect to.
|
||||
|
||||
The constructor arguments:
|
||||
* module : database module, e.g. MySQLdb. This is simply passed through to the ConnectionPool.
|
||||
* credentials : A dictionary, or dictionary-alike, mapping hostname to connection-argument-dictionary. This is used for the constructors of the ConnectionPool objects. Example:
|
||||
|
||||
>>> dc = DatabaseConnector(MySQLdb,
|
||||
{'db.internal.example.com':
|
||||
{'user':'internal', 'passwd':'s33kr1t'},
|
||||
'localhost':
|
||||
{'user':'root', 'passwd':''})
|
||||
|
||||
If the credentials contain a host named 'default', then the value for 'default' is used whenever trying to connect to a host that has no explicit entry in the database. This is useful if there is some pool of hosts that share arguments.
|
||||
|
||||
* conn_pool : The connection pool class to use. Defaults to db_pool.ConnectionPool.
|
||||
|
||||
The rest of the arguments to the DatabaseConnector constructor are passed on to the ConnectionPool.
|
||||
|
||||
NOTE: The DatabaseConnector is a bit unfinished, it only suits a subset of use cases.
|
||||
"""
|
||||
|
||||
from collections import deque
|
||||
@@ -32,48 +100,10 @@ from eventlet.pools import Pool
|
||||
from eventlet.processes import DeadProcess
|
||||
from eventlet import api
|
||||
|
||||
|
||||
class ConnectTimeout(Exception):
|
||||
pass
|
||||
|
||||
class DatabaseConnector(object):
|
||||
"""\
|
||||
@brief This is an object which will maintain a collection of database
|
||||
connection pools keyed on host,databasename"""
|
||||
def __init__(self, module, credentials,
|
||||
conn_pool=None, *args, **kwargs):
|
||||
"""\
|
||||
@brief constructor
|
||||
@param min_size the minimum size of a child pool.
|
||||
@param max_size the maximum size of a child pool."""
|
||||
assert(module)
|
||||
self._conn_pool_class = conn_pool
|
||||
if self._conn_pool_class is None:
|
||||
self._conn_pool_class = ConnectionPool
|
||||
self._module = module
|
||||
self._args = args
|
||||
self._kwargs = kwargs
|
||||
self._credentials = credentials # this is a map of hostname to username/password
|
||||
self._databases = {}
|
||||
|
||||
def credentials_for(self, host):
|
||||
if host in self._credentials:
|
||||
return self._credentials[host]
|
||||
else:
|
||||
return self._credentials.get('default', None)
|
||||
|
||||
def get(self, host, dbname):
|
||||
key = (host, dbname)
|
||||
if key not in self._databases:
|
||||
new_kwargs = self._kwargs.copy()
|
||||
new_kwargs['db'] = dbname
|
||||
new_kwargs['host'] = host
|
||||
new_kwargs.update(self.credentials_for(host))
|
||||
dbpool = self._conn_pool_class(self._module,
|
||||
*self._args, **new_kwargs)
|
||||
self._databases[key] = dbpool
|
||||
|
||||
return self._databases[key]
|
||||
|
||||
|
||||
class BaseConnectionPool(Pool):
|
||||
def __init__(self, db_module,
|
||||
@@ -316,6 +346,7 @@ class TpooledConnectionPool(BaseConnectionPool):
|
||||
|
||||
connect = classmethod(connect)
|
||||
|
||||
|
||||
class RawConnectionPool(BaseConnectionPool):
|
||||
"""A pool which gives out plain database connections.
|
||||
"""
|
||||
@@ -413,3 +444,44 @@ class PooledConnectionWrapper(GenericConnectionWrapper):
|
||||
|
||||
def __del__(self):
|
||||
self.close()
|
||||
|
||||
|
||||
class DatabaseConnector(object):
|
||||
"""\
|
||||
@brief This is an object which will maintain a collection of database
|
||||
connection pools on a per-host basis."""
|
||||
def __init__(self, module, credentials,
|
||||
conn_pool=None, *args, **kwargs):
|
||||
"""\
|
||||
@brief constructor
|
||||
@param module Database module to use.
|
||||
@param credentials Mapping of hostname to connect arguments (e.g. username and password)"""
|
||||
assert(module)
|
||||
self._conn_pool_class = conn_pool
|
||||
if self._conn_pool_class is None:
|
||||
self._conn_pool_class = ConnectionPool
|
||||
self._module = module
|
||||
self._args = args
|
||||
self._kwargs = kwargs
|
||||
self._credentials = credentials # this is a map of hostname to username/password
|
||||
self._databases = {}
|
||||
|
||||
def credentials_for(self, host):
|
||||
if host in self._credentials:
|
||||
return self._credentials[host]
|
||||
else:
|
||||
return self._credentials.get('default', None)
|
||||
|
||||
def get(self, host, dbname):
|
||||
""" Returns a ConnectionPool to the target host and schema. """
|
||||
key = (host, dbname)
|
||||
if key not in self._databases:
|
||||
new_kwargs = self._kwargs.copy()
|
||||
new_kwargs['db'] = dbname
|
||||
new_kwargs['host'] = host
|
||||
new_kwargs.update(self.credentials_for(host))
|
||||
dbpool = self._conn_pool_class(self._module,
|
||||
*self._args, **new_kwargs)
|
||||
self._databases[key] = dbpool
|
||||
|
||||
return self._databases[key]
|
||||
|
||||
Reference in New Issue
Block a user