36 Commits

Author SHA1 Message Date
Mark McLoughlin
1a91aacb85 Port to oslo.messaging
The oslo.messaging library takes the existing RPC code from oslo and
wraps it in a sane API with well defined semantics around which we can
make a commitment to retain compatibility in future.

The patch is large, but the changes can be summarized as:

  * oslo.messaging>=1.3.0a4 is required; a proper 1.3.0 release will be
    pushed before the icehouse release candidates.

  * The new rpc module has init() and cleanup() methods which manage the
    global oslo.messaging transport state. The TRANSPORT and NOTIFIER
    globals are conceptually similar to the current RPCIMPL global,
    except we're free to create and use alternate Transport objects
    in e.g. the cells code.

  * The rpc.get_{client,server,notifier}() methods are just helpers
    which wrap the global messaging state, specifiy serializers and
    specify the use of the eventlet executor.

  * In oslo.messaging, a request context is expected to be a dict so
    we add a RequestContextSerializer which can serialize to and from
    dicts using RequestContext.{to,from}_dict()

  * The allowed_rpc_exception_modules configuration option is replaced
    by an allowed_remote_exmods get_transport() parameter. This is not
    something that users ever need to configure, but it is something
    each project using oslo.messaging needs to be able to customize.

  * The nova.rpcclient module is removed; it was only a helper class
    to allow us split a lot of the more tedious changes out of this
    patch.

  * Finalizing the port from RpcProxy to RPCClient is straightforward.
    We put the default topic, version and namespace into a Target and
    contstruct the client using that.

  * Porting endpoint classes (like ComputeManager) just involves setting
    a target attribute on the class.

  * The @client_exceptions() decorator has been renamed to
    @expected_exceptions since it's used on the server side to designate
    exceptions we expect the decorated method to raise.

  * We maintain a global NOTIFIER object and create specializations of
    it with specific publisher IDs in order to avoid notification driver
    loading overhead.

  * rpc.py contains transport aliases for backwards compatibility
    purposes. setup.cfg also contains notification driver aliases for
    backwards compat.

  * The messaging options are moved about in nova.conf.sample because
    the options are advertised via a oslo.config.opts entry point and
    picked up by the generator.

  * We use messaging.ConfFixture in tests to override oslo.messaging
    config options, rather than making assumptions about the options
    registered by the library.

The porting of cells code is particularly tricky:

  * messaging.TransportURL parse() and str() replaces the
    [un]parse_transport_url() methods. Note the complication that an
    oslo.messaging transport URL can actually have multiple hosts in
    order to support message broker clustering. Also the complication
    of transport aliases in rpc.get_transport_url().

  * proxy_rpc_to_manager() is fairly nasty. Right now, we're proxying
    the on-the-wire message format over this call, but you can't supply
    such messages to oslo.messaging's cast()/call() methods. Rather than
    change the inter-cell RPC API to suit oslo.messaging, we instead
    just unpack the topic, server, method and args from the message on
    the remote side.

    cells_api.RPCClientCellsProxy is a mock RPCClient implementation
    which allows us to wrap up a RPC in the message format currently
    used for inter-cell RPCs.

  * Similarly, proxy_rpc_to_manager uses the on-the-wire format for
    exception serialization, but this format is an implementation detail
    of oslo.messaging's transport drivers. So, we need to duplicate the
    exception serialization code in cells.messaging. We may find a way
    to reconcile this in future - for example a ExceptionSerializer
    class might work, but with the current format it might be difficult
    for the deserializer to generically detect a serialized exception.

  * CellsRPCDriver.start_servers() and InterCellRPCAPI._get_client()
    need close review, but they're pretty straightforward ports of code
    to listen on some specialized topics and connect to a remote cell
    using its transport URL.

blueprint: oslo-messaging
Change-Id: Ib613e6300f2c215be90f924afbd223a3da053a69
2014-02-01 09:53:24 +00:00
OpenStack Jenkins
d57f4690a1 Updated from global requirements
Change-Id: I27ec5dc38549666a4ee63f0d29f4ef35f81f465d
2014-01-25 05:46:54 +00:00
Thierry Carrez
e0e4b55930 Use oslo.rootwrap library instead of local copy
Remove rootwrap code copied from oslo-incubator, make the
nova-rootwrap console_script entrypoint point to oslo.rootwrap
code instead.

Change-Id: Iec7aaf2e1599b2faf403d11b779eeab199a89486
Implements: blueprint nova-oslo-rootwrap
2014-01-17 18:06:33 +01:00
Ivan A. Melnikov
79078e372e Add wsgiref to requirements.txt
Baremetal deploy helper (nova/cmd/baremetal_deploy_helper.py) uses
wsgiref module, so it should be in requirements.

Closes-bug: #1263480
Change-Id: Ibdd917d8065c085411a21a65e2d9ae016e265ee5
2013-12-22 15:18:57 +02:00
OpenStack Jenkins
2780838ad0 Merge "Updated from global requirements" 2013-12-11 09:10:32 +00:00
OpenStack Jenkins
d9089dae9b Updated from global requirements
(Also add the sphinx issue which wasn't in the automatic job and
blocked this from landing.)

Change-Id: I94bc14b0ce448ea7a103a6faab73c29b38fbdf3d
2013-12-10 18:02:18 -05:00
Brad Pokorny
8c0c9b1169 Sync middleware audit, base, and notifier from oslo
Pull in middleware modules required to support CADF auditing
reports from Nova.  audit.py requires notify.py, and
notify.py requires base.py.  This is why those 3 files are
being synced, and not others in the
openstack/common/middleware directory from oslo.

The following 3 change IDs included in this sync provide
the main parts of the CADF functions being synced:

Ie67cf4a7 Add audit middleware
Iead2431a Middleware: add notifier
I05ff58   Move wsgi module to deprecated package

The following change IDs additionally included in this sync
provide minor updates to the files that were made after the
main parts of the CADF functions were merged in oslo:

I032616   Changed header from LLC to Foundation based on
          trademark policies
I840c43bf add support to ignore certain request
I2099e1ee Enable hacking H404 test
I04b0cf   middleware: code cleanup

blueprint support-standard-audit-formats

Change-Id: Id15f95a0c354270ba2d5aecb12c4f910905e8c20
2013-12-08 18:45:11 +00:00
Dan Prince
e26fac5ed7 Bump to sqlalchemy-migrate 0.8.2
This allows us to drop some unsightly
0.7.3 patching and also avoids potential
issues in using 0.8.1 which had a bad
release version string (it was called '0.7.3dev' by mistake).

This will require a requirements change here:

 https://review.openstack.org/56662

Change-Id: I5f80dfc07f8256716d025002fc1a1f47e58b2142
Closes-Bug: 1251700
2013-11-25 15:44:40 -05:00
OpenStack Jenkins
831c7e6dec Updated from global requirements
Change-Id: Ibc58e9a061a56e9d66ee0b4e08fa771845fb0595
2013-10-19 23:59:59 +00:00
OpenStack Jenkins
390cfe9c93 Updated from global requirements
Change-Id: I98c312279426c2f95cefea67260afd1677a11184
2013-10-17 22:23:00 +00:00
OpenStack Jenkins
084eda92b3 Updated from global requirements
Change-Id: Ica7315effe308bd739e90f885849fac27a3c542b
2013-10-16 16:10:09 +00:00
OpenStack Jenkins
3f908c9c60 Updated from global requirements
Change-Id: Ie9e92b06eb28fa238c7cb923a65345c415dd3642
2013-10-01 16:13:56 +00:00
Mark McLoughlin
b85e1d8f04 Require oslo.config 1.2.0 final
Now that 1.2.0 has been released, require it rather than one of the beta
release tarballs.

Closes-Bug: #1182861
Change-Id: I75ffc4a572d14488d19491a38742d6920951c171
2013-09-24 21:54:57 +01:00
Yaguang Tang
bdac306b16 Update requirements not to boto 2.13.0
Latest boto 2.13.0 has a bug which lead to unit tests fails, before
it's fixed avoid use this version.

Fix bug #1224832

Change-Id: I8a81adfc0cb0ad37d27b11db3c881735a099a95c
2013-09-13 15:26:53 +08:00
Monty Taylor
446967ac0c Updated from global requirements
Change-Id: Ibf423f14a5c37aa298b2115bfd4936f660c6f530
2013-09-05 11:12:57 -04:00
Dirk Mueller
12b104751a Remove indirect dependency from requirements.txt
Only direct dependencies need to be listed in requirements.txt.
pyparsing was previously added to workaround installation ordering
issues, which is no longer necessary.

Change-Id: I45bdd1d0dabfcc7c80314c28072950cfc5a59615
2013-09-05 08:03:21 -04:00
Jenkins
28eca5936f Merge "update neutronclient to 2.3.0 minimum" 2013-09-05 01:45:24 +00:00
Mark McClain
a77a28d3f7 update neutronclient to 2.3.0 minimum
Change-Id: I21f9ae030a173621db3d21db018c3a416d050813
2013-09-03 16:36:07 -04:00
Sascha Peilicke
fa0d61084e Port Cheetah templates to Jinja2
Cheetah is unmaintained since 2010 and it's unlikely to get Python3
support soon. Also, the rest of OpenStack (mostly) standardized on
Jinja2.

Implements: blueprint jinja-templating-conversion
Change-Id: Ia15f00ee96d3c1d55d7c290f20ccc988e4c52e1a
2013-09-02 16:03:34 +02:00
Jenkins
e4b5274da8 Merge "Enable no_parent and file_only security" 2013-08-27 10:58:06 +00:00
He Yongli
f17dccfc45 Add jsonschema to Nova requirements.txt
jsonschema is requested for validate the whitelist and alias
definition.

Change-Id: Iec3c14bb407cce88fd5af54a3930d83c53033293
2013-08-21 09:22:05 +08:00
Rohit Karajgi
3eb67b811a Enable no_parent and file_only security
The websockify package was recently upgraded to version 0.5.1, which added two
parameters that provide additional security:
* no_parent - returns 403 response if contents out of the web root are
requested. Required to disable directory traversal.
* file_only - returns 404 response if non-file contents are requested. Required
to disable directory listing.
This version is available in PyPi.

Nova should upgrade it's required version for websockify, and set these
parameters while creating and starting the NovaWebSocketProxy to ensure
better security.

DocImpact
Fixes bug #1195700

Change-Id: I9c4b28eb1b384048f10d365ed204b916afd371bb
2013-08-07 08:13:15 -07:00
Jenkins
9c1bfcec50 Merge "Revert "Add requests requirement capped <1.2.1."" 2013-08-01 11:48:33 +00:00
Chris Yeoh
aad95316d4 Allow exceptions to propagate through stevedore map
Allow exceptions to propagate up through the stevedore map
function used by the V3 API servers core extension. This allows
for functionality such as extension specific parsing of client
supplied data formerly required to be handled by the core API to
be handled by the extension itself instead.

This functionality requires stevedore 0.10 or later

Partially implements blueprint nova-v3-api

Change-Id: Ib848147225707f1c7eda27b1ba796022161ba20f
2013-07-23 10:46:14 +09:30
Jeremy Stanley
661deb3be9 Revert "Add requests requirement capped <1.2.1."
Fixes bug 1182271.

This reverts commit c39687edb9138c6eac675b8781a5d4bf5a51c968. The
original workaround is no longer needed, as pip is now configured to
install with the --upgrade flag and will therefore pull a new enough
python-jinja2 into the tox virtualenv even though site packages is
enabled in nova and an older distribution package of the same
library is still present on CentOS 6.4.

Change-Id: I33335314c7c1b26d630137edfa0cdc2d1947432e
2013-07-22 19:19:49 +00:00
Jenkins
4110960378 Merge "Fix issue with pip installing oslo.config-1.2.0" 2013-07-16 15:44:30 +00:00
Sean Dague
b31be6e16f remove python-glanceclient cap
if any of the projects specify a capped client, it has the
potential for preventing that client from being tested in the
gate. To fix this we have to uncap maximum versions of all
openstack client code in all openstack projects.

(Note Neutronclient needs special dispensation for now, so
it is left capped.)

(Rebased after a neutron rename landed)

Fixes bug #1200214

Change-Id: I0168cb645279a819942a442c800afb1d0336203c
2013-07-15 10:43:17 -04:00
Mark McClain
566a44fbdf update Quantum usage to Neutron
This change updates to use the neutron client library and provides
backwards compatiblity with the network_api_class.

implements bug: 1197208

Change-Id: Ia6c74e40eff516a0fcbe53fdc48e16fb0f98e864
2013-07-12 15:02:59 -04:00
Mark McLoughlin
59b799d30e Fix issue with pip installing oslo.config-1.2.0
Fixes bug #1194807

Firstly, we update the oslo.config dep to 1.2.0a3 because of the issue
with namespace packages (bug #1194742).

But the main issue here is that when we previously depended on 1.2.0a3
we found that if you did:

  $> pip install -r nova/requirements.txt

then you end up with the oslo.config 1.1.1 code installed. This is
because oslo.config>=1.1.0 gets pulled in as a transitive dep and pip
gets confused.

See I977700d73342e81ee962019b76238d2cb2b1fff4

You can reproduce with e.g.

  $> pip install \
       http://.../oslo.config-1.2.0a3.tar.gz#egg=oslo.config-1.2.0a3 \
       python-keystoneclient
  $> pip freeze | grep oslo.config
  oslo.config-1.2.0a3
  $> python -c 'from oslo.config.cfg import DeprecatedOpt'
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  ImportError: cannot import name DeprecatedOpt

This is because of a bug with pip where it sees oslo.config-1.2.0a3 and
oslo.config as two unrelated things. It should strip the version part of
the egg= fragment before using it as a package name, but it doesn't.

However, we can simply use the -f/--find-links pip option in our
requirements.txt to add the tarball URL to the list of URLs considered
and also add the oslo.config>=1.2.0a3 dependency:

  $> pip install \
       -f http://.../oslo.config-1.2.0a3.tar.gz#egg=oslo.config-1.2.0a3 \
       'oslo.config>=1.2.0a3' \
       python-keystoneclient
  $> pip freeze | grep oslo.config
  oslo.config-1.2.0a3
  $> python -c 'from oslo.config.cfg import DeprecatedOpt'

This is actually exactly the semantics we want and we go to great
lengths in pbr to get these semantics while using a single tarball URL.
The only downside to this --find-links strategy is that we gain an extra
line in our requirements.txt ... but it does work around the pip bug.

Change-Id: I6f3eb5fd2c75615d9a1cae172aed859b36b27d4c
2013-07-10 22:29:04 +01:00
Xavier Queralt
ec010304db Update pyparsing to 1.5.7
The version of cliff we are using depends on pyparsing>=1.5.7 but
python-quantumclient pulls 1.5.6 because it appears first in the
requirements processing.

The requirements change is in
    https://review.openstack.org/#/c/33406/

Fixes bug #1191866

Change-Id: Ib1c11b53813ff96b750f5ffe4997ad01907d32c3
2013-06-25 10:03:00 +02:00
Jenkins
76750f3dcc Merge "Port missing bits from httplib2 to requests" 2013-06-22 01:04:49 +00:00
Sascha Peilicke
346d9419a6 Port missing bits from httplib2 to requests
This allows to drop the dependency on httplib2 altogether.

Change-Id: I18571e29d9f0d6527a93ed18becee9148e64e574
2013-06-21 14:54:47 +02:00
Dan Smith
d26581d939 Revert "Add oslo-config-1.2.0a2 and pbr>=0.5.16 to requirements."
This partially reverts commit:
   43b7f22b30244114f809e4c3b44dc337cade95f1.

The oslo.config change in that patch seems to break running unit
tests with tox. This reverts just that part of the change.

Change-Id: I977700d73342e81ee962019b76238d2cb2b1fff4
2013-06-20 15:13:58 -07:00
Michael Wilson
43b7f22b30 Add oslo-config-1.2.0a2 and pbr>=0.5.16 to requirements.
See bug #1189889.

Change-Id: Id0dae9cea96ee779bc94a095d1ec5d046f2b39b4
Fixes: bug #1189889
2013-06-19 11:54:40 -06:00
Doug Hellmann
b07994b1b0 Update to the latest stevedore
Use a version of stevedore that does not try to
install distribute.

The requirements change is in
https://review.openstack.org/#/c/32738/

Change-Id: I277a282ed86743a25138c431045af1f0ccb11b39
Signed-off-by: Doug Hellmann <doug.hellmann@dreamhost.com>
2013-06-12 11:15:29 -04:00
Ruby Loo
5740920bb3 Rename requires files to standard names.
Rename tools/pip-requires to requirements.txt and tools/test-requires
to test-requirements.txt. These are standard files, and tools in the
general world are growing intelligence about them.

Change-Id: I68ece8406fb1d4e082a42db8e76e17b1aaa7e775
Fixes: bug #1179008
2013-05-22 20:12:08 +00:00