Fixes bug #1213080 and implements blueprint condutor-workers.
Make it easy to launch a bunch of conductor processes on a host.
Deploying multiple conductor workers per host avoids serialization on
database accesses caused by libmysqlclient.so blocking eventlet's
single thread. In an experiment on a 24-core machine, when creating 20
VMs in parallel, maximum creation time was reduced by approx. 10s when
using 20 conductor processes vis-a-vis a single conductor process.
Profiling showed that all of the savings came from faster calls into
nova.db.sqlalchemy.api.
Note that there are alternative methods for preventing the eventlet
thread from blocking during database calls. However, none of these
alternatives performed as well as multiple nova-conductor processes.
* Instead of using the native database driver like _mysql.so, you
can use a pure-python driver, like pymysql by setting
sql_connection=mysql+pymysql://... in the [DEFAULT] section of
/etc/nova/nova.conf, which eventlet will monkeypatch to avoid
blocking. The problem with this approach is the vastly greater
CPU demand of the pure-python driver compared to the native
driver. Since the pure-python driver is so much more CPU
intensive, the eventlet thread spends most of its time talking to
the database, which effectively the problem we had before!
* Instead of making database calls from eventlet’s thread, you can
submit them to eventlet’s pool of worker threads and wait for the
results. Try this by setting dbapi_use_tpool=True in the
[DEFAULT] section of /etc/nova/nova.conf. The problem I found
with this approach was the overhead of synchronizing with the
worker threads. In particular, the time elapsed between the
worker thread finishing and the waiting coroutine being resumed
was typically several times greater than the duration of the
database call itself.
Change-Id: I8698997d211d7617ee14a1c6113056a694d70620
5450f09370