Commit Graph

57 Commits

Author SHA1 Message Date
Hiroyasu.OHYAMA
beb23103b0 [document] The example which is written in the developer guide of
'Notification Listener' doesn't work.

I fixed the example which is described in developer guide to work fine.
These are the main points of this correction.

* specify namespace for NotificationFilter.
* set 2 blank lines between class definitions according to E302 of pep8.
* change argument of 'pool' to keyward arguemnt, because 4th argument of
  'get_notification_listener' requires the name of message executor.

Change-Id: I75800309f2294fe9f17ea2a2b1638de57ef2922e
Closes-Bug: #1618394
2016-10-07 18:07:00 +09:00
Kenneth Giusti
9bc9c0dc6a Fixups to the inline documentation
Rework the inline documentation for executors, RPC servers and
clients, notifiers and notification listeners for clarity and flow.

Change-Id: If4f1db853a7fc85340177fd2c9c43a479d72459d
2016-09-12 09:49:00 -04:00
tengqm
3cdfe15897 Fix notify filter when data item is None
When data[k] is None during match check, we are getting annoying
TypeError exceptions like: "TypeError: expected string or buffer".
This patch adds a check that data[k] is something the regex can do
matching.

Change-Id: I75c8f602a59a90aeb5a6973f2e9e25708194ad78
2016-09-03 14:45:44 +00:00
George Silvis, III
a011cb2448 Add docstring for get_notification_transport
This was previously undocumented.  Most of the information was taken from the
docstring for get_transport

Change-Id: I92652f4de9dd1e59ae5c3f9ce39e5ee808f24c7d
2016-08-15 11:51:47 -04:00
Gregory Haynes
317641c42f Fix syntax error on notification listener docs
We are missing a list delimiter in our docs example for a notification
listener.

Change-Id: Ib8b9c35e7bff241bf8473a98a2ee188da7e194e4
2016-07-28 01:57:35 +00:00
Julien Danjou
ff9b4bb938 notify: add a CLI tool to manually send notifications
Change-Id: I01ebd8402e322c5c27dc2fc4c61fb5beb2dddae8
2016-07-13 17:30:56 +02:00
liusheng
e57afac051 [Trival] fix a typo nit
Change-Id: I3d40726cf55aedb838ebe605f3d58618d594acca
2016-06-13 16:34:12 +08:00
Mehdi Abaakouk
5dd059aa3c Remove deprecated localcontext
Change-Id: Ib109ced5ee6a706853b0f54be6756cfaf63e2164
2016-05-18 09:28:15 +02:00
Dmitriy Ukhlov
4df633db74 Improves exception handling and logging
1) try to catch all possible exceptions during message acknowledging
    and dispatching to send ack/nack when we can to do it
2) improve logging in case of exceptions during message acknowledging
    and dispatching

Depends-On: I2d230d49e5aff6ab4d84ab8c3d2834f85e3405eb
Change-Id: I41a768c5624fa2212257ce20bf9a67d09de0c4ab
2016-04-26 12:34:46 +00:00
Jenkins
f22a1a4fec Merge "Refactor base interfaces" 2016-04-22 14:01:08 +00:00
ZhiQiang Fan
5b94b4b7a9 use thread safe fnmatch
fnmatch is not thread safe in python <2.7.10, let's use the safe
one in oslo.utils

Change-Id: If3d480436506ecc5f3c2fcbd51ceaac59f2b7dec
ref: https://hg.python.org/cpython/rev/fe12c34c39eb
2016-04-21 11:43:53 +08:00
Dmitriy Ukhlov
6db00c77b0 Refactor base interfaces
1) Add MessageHandler base interface for on_incoming_callback replacement
2) Move message_handler parameter form Listener's __init__() to start()
3) Remove wait method from listener

Change-Id: Id414446817e3d2ff67b815074d042a9ce637ec24
2016-04-20 20:46:28 +00:00
Jenkins
55c0b3b52f Merge "Make transport_url config option secret" 2016-04-07 15:05:23 +00:00
Juan Antonio Osorio Robles
cbaf71edfa Make transport_url config option secret
The transport_url may contain credentials to the message queue; For
this reason, this option should be secret for it to not leak into the
logs.

Closes-Bug: #1567233

Change-Id: I49c641a2662976d7220e4222e3c4a4d2586b1336
2016-04-07 05:43:18 +00:00
Dmitriy Ukhlov
5d7d7253d1 Refactor driver's listener interface
Current Listener interface has poll() method which return messages

To use it we need have poller thread which is located in MessageHandlerServer
But my investigations of existing driver's code shows that some implemetations have
its own thread inside for processing connection event loop. This event loop received
messages and store in queue object. And then our poller's thread reads this queue
This situation can be improved. we can remove poller's thread, remove queue object
and just call on_message server's callback from connection eventloop thread

This path provide posibility to do this for one of drivers and leave as is other drivers

Change-Id: I3e3d4369d8fdadcecf079d10af58b1e4f5616047
2016-04-05 18:08:08 +00:00
Jenkins
d4e8ac42b5 Merge "Move server related logic from dispatchers" 2016-03-30 15:36:03 +00:00
Dmitriy Ukhlov
990d894eaf Move server related logic from dispatchers
Dispatcher should be responsible for routing message to the
callback method of endpoint object and returning result back to the
server only. But now it is also responsible for sending reply,
ack/reque messages etc.

Also this patch makes small improvements:
   1) Notification dispatcher now requeue message if endpoint raises exception
   2) unstable behaviour of test_mask_passwords test is fixed

Change-Id: I5f23e23644e90919cb67f81fc306ee85c5e09974
2016-03-30 11:07:32 +00:00
Davanum Srinivas
f8969e97ce [py34] replace file() with open()
Running flake8 with python3 finds this problem:
./oslo_messaging/notify/_impl_routing.py:55:16: F821 undefined name 'file'
        return file(filename, 'r')

Change-Id: I5cd7475618f7a7532db770c2c5c61576a657fbb4
2016-03-24 11:32:25 +00:00
Jenkins
cc89fc3174 Merge "Revert "Ensure the json result type is bytes on Python 3"" 2016-03-11 14:25:05 +00:00
Victor Stinner
bb4121a465 Revert "Ensure the json result type is bytes on Python 3"
This reverts commit bd81d09c02.

I understand that the change was supposed to fix something, but instead it broke all tests on Python 3!?

It's wrong to replace blindly json.dumps() with jsonutils.dump_as_bytes(). In oslo messaging, the result is usually used as a value in a dictionary, and then the whole dictionary is passed to a second serializer which also serialize to JSON.

Sorry, I don't understand everything, but at least I see that tests passed on py3 before the change, and started to fail with the change.

Maybe json(utils).dumps() is misused in some places, but in this case, you should write a change which only fix these specific places, not replace all calls to dumps().

Change-Id: Icd54ee8e3f5c976dfd50b4b62c7f51288649e112
2016-03-11 09:00:08 +00:00
Jenkins
67c8987372 Merge "Ensure the json result type is bytes on Python 3" 2016-03-11 04:10:35 +00:00
Davanum Srinivas
2d53db6c51 Allow Notifier to have multiple topics
Looks like there is a disconnect between the __init__
parameter 'topic' in Notifier and what we need when
we look up a driver. We should allow multiple topics
to be specified as a new topics parameter and pass
that along directly.

Change-Id: Id89957411aa219cff92fafec2f448c81cb57b3ca
2016-02-25 06:43:00 +00:00
Javeme
bd81d09c02 Ensure the json result type is bytes on Python 3
Since the json.dumps() returns Unicode string On Python 3, to
ensure that the result type is bytes on Python 2 and Python 3
that if the result is used for the message body, let's replace
    json.dumps() /  oslo_serialization.jsonutils.dumps()
with
    oslo_serialization.jsoutils.dump_as_bytes()

Change-Id: I0e0f6b715ffc4a9ad82be52e55696d032b6d0976
2016-02-24 15:20:08 +08:00
Javeme
3288c4d7e2 Use more efficient mask_dict_password to mask password
The mask_dict_password method is added from oslo_utils 3.4.0, and it's
about five times faster than the mask_password method, so it's better
to use mask_dict_password instead of mask_password.

Change-Id: I6629ed1232f2e6f1c72b9f13b361e2f9e14b50a4
2016-02-24 13:19:53 +08:00
Victor Stinner
22fea728cc Remove executor callback
Revert the change I556b112371bec2ec29cea4dc254bb3f9c6d2c07a: the
executor callback API was only used by the aioeventlet executor which
was just removed.

Change-Id: I1223f32594d8c1be28cc43fdd9bf102c86d75325
2016-02-18 12:50:44 +01:00
ChangBo Guo(gcb)
6b20fa8597 Improvement of logging acorrding to oslo.i18n guideline
1. Use translation marker functions, their argument must just be a string
2. Any message with more than one variable should use named
   interpolation instead of positional to allow translators
   to move the variables around in the string to account for
   differences in grammar and writing direction.
3. String interpolation should be delayed to be handled by the logging
   code, rather than being done at the point of the logging call.
For more details, please refert to oslo.i18n guideline [1]

Note: this commit doesn't touch test code.

[1] http://docs.openstack.org/developer/oslo.i18n/guidelines.html

Change-Id: I5f013d65b20396bbe0e5a2cceaed2a33fad1af23
2016-01-07 13:26:55 +08:00
Jenkins
0a0e6d0d50 Merge "notif: Check the driver features in dispatcher" 2015-12-11 04:17:21 +00:00
Jenkins
213176657d Merge "batch notification listener" 2015-12-11 04:16:57 +00:00
Jenkins
4b6144a3db Merge "creates a dispatcher abstraction" 2015-12-10 11:15:48 +00:00
Mehdi Abaakouk
050024f798 Fix notifier options registration
Change-Id: I37082f6f349e89af6b74e6ec5e5c416902299263
2015-12-08 16:01:49 +01:00
Mehdi Abaakouk
185f94c013 notif: Check the driver features in dispatcher
The transport/driver features check is done into the get listener
methods.
So when these methods are not used the driver features checks is not
done.

This change moves it into the dispatcher layer to ensure the
requirements are always checked.

This changes a bit the behavior of when the check occurs. Before
it was during the listener object initialisation. Now this
when the listener server start.

Change-Id: I4d81a4e8496f04d62e48317829d5dd8b942d501c
2015-12-08 09:14:20 +01:00
Mehdi Abaakouk
4dd644ac20 batch notification listener
Gnocchi performs better if measurements are write in batch
When Ceilometer is used with Gnocchi, this is not possible.

This change introduce a new notification listener that allows that.

On the driver side, a default batch implementation is provided.
It's just call the legacy poll method many times.

Driver can override it to provide a better implementation.
For example, kafka handles batch natively and take benefit of this.

Change-Id: I16184da24b8661aff7f4fba6196ecf33165f1a77
2015-12-08 09:14:20 +01:00
Mehdi Abaakouk
bdf287e847 creates a dispatcher abstraction
This change creates a dispatcher abstraction
to document the interface of a dispatcher.

And also allows in the futur to have attributes with default values.

Change-Id: I9a7e5e03f89635a3790b3851f492a1a7aab58feb
2015-12-07 16:43:34 +01:00
Davanum Srinivas
33c1010c32 Option group for notifications
In change Ief6f95ea906bfd95b3218a930c9db5d8a764beb9, we 
decoupled RPC and Notifications a bit. We should take another
step and separate out the options for notifications into 
its own group.

Change-Id: Ib51e2839f9035d0cc0e3f459939d9f9003a8c810
2015-11-30 19:30:05 +00:00
Jenkins
9938f668f4 Merge "Add Warning when we cannot notify" 2015-11-25 19:28:42 +00:00
Davanum Srinivas
6dba2ed591 Add Warning when we cannot notify
Calling Notifier.audit() won't work with LogDriver as the
logger does not have a audit level anymore, So let's at
least not fail silently and let the operators know that
we are dropping stuff on the floor.

Closes-Bug: #1518170
Change-Id: I74002c72e6763ea8b5df7d97d722619bd4a1950b
2015-11-24 02:08:07 +00:00
Julien Danjou
47a906aff3 middleware: remove oslo.context usage
There's nothing interesting in the fake context we're passing. Let's
stop depending on oslo.context all together here.

Change-Id: I7784bc3b818e67118e03905857c39eac66765fad
2015-11-23 15:30:23 +00:00
Flavio Percoco
925eb734a9 Remove qpidd's driver from the tree
Back in liberty we marked this driver as deprecated. This patch removes
it from the tree. The patch also removes tests, options and other
references in the documentation. Note that one script is being kept
because it's required by the amqp driver.

Depends-On: If4b1773334e424d1f4a4e112bd1f10aca62682a9
Change-Id: I4a9cba314c4a2f24307504fa7b5427424268b114
2015-11-20 18:43:28 +00:00
Matt Riedemann
15cd99050c Provide alias to oslo_messaging.notify._impl_messaging
Ifb96c2ae9868426cac2700bf4917c27c02c90b15 moved the _impl_messaging
module to oslo_messaging.notify.messaging which breaks neutron.

Neutron is fixed on master for mitaka but neutron on stable/liberty
is broken, and changing neutron on stable/liberty to use the new
path would require a global-requirements minimum version bump for
oslo.messaging to 2.6.0, which we want to avoid for people that have
already shipped liberty.

So provide an alias to the moved module so neutron in stable/liberty
continues to work. We deprecate the module so consumers know they
need to upgrade and move off this. We may need to cap oslo.messaging
in global-requirements on stable/liberty at some point when we remove
the deprecated alias module.

Change-Id: I29453e0fbf30b0a571c2b1afc7cc81d1a11535f0
Closes-Bug: #1513630
2015-11-19 12:45:14 -08:00
Davanum Srinivas
6621b9010e Decouple transport for RPC and Notification
Add a new configuration option for setting up
an alternate notification_transport_url that
can be used for notifications. This allows
operators to separate the transport mechanisms
used for RPC and Notifications.

DocImpact

Closes-Bug: #1504622
Change-Id: Ief6f95ea906bfd95b3218a930c9db5d8a764beb9
2015-11-10 20:40:22 -05:00
Cyril Roelandt
c4a7ac0b65 Use yaml.safe_load instead of yaml.load
We currently use yaml.load to read a user-written config file. This can
lead to malicious code execution, so we should use yaml.safe_load
instead.

Found using bandit.

Change-Id: I27792f0435bc3cb9b9d31846d07a8d47a1e7679d
2015-10-21 17:22:20 +02:00
Daisuke Fujita
64fb61d37c Fix typo in rpc/server.py and notify/listener.py
1. "messaging configuration configuration" to "messaging configuration"

Change-Id: Icda5d2df67cc4d2d0e0cb3a453d0b04dd998e5f4
2015-10-09 22:20:51 +09:00
Davanum Srinivas
1893c495f6 Allow custom notification drivers
Our class hierarchy hides classes and modules that so its hard
for folks to write a custom Notification driver. We should
make these public and document them

Closes-Bug: #1426046
Change-Id: Ifb96c2ae9868426cac2700bf4917c27c02c90b15
2015-09-25 10:07:05 -04:00
Jenkins
b02dd25655 Merge "Mask passwords when logging messages" 2015-08-24 16:10:31 +00:00
Ryan Rossiter
c990ee02fa Mask passwords when logging messages
When logging a message, any secrets and passwords should be masked. This
uses oslo_utils.strutils to mask any passwords that are to be logged.

Change-Id: I263d44c0f2e900c5f6e210cbd7ec56e48d0d5bb2
Closes-Bug: #1487038
2015-08-24 13:13:44 +00:00
Davanum Srinivas
588d795b3c Use proper translating helper for logging
Add hacking rule borrowed from keystone to make sure
we don't regress and fix all the issues found by the
hacking check.

Change-Id: I41635fdd83c3e04d04f9849a72c49ccb5ac42875
2015-08-20 21:01:59 -04:00
Mehdi Abaakouk
c49594a62f Remove usage of contentmanager for executors
The context manager in the executor fit only for the blocking executor.
Even the dispatcher needs to run code before and after the application
callback, eventlet and future executors have to run the pre/post code
into the main thread and can run the callback into an other thread,
and that force them to run __enter__ and __exit__ manually and deal
the exception path.

This change adds a helper object instead of the context manager.
It is designed to be explicit on what must be executed
before and after the callback and what can be done in a thread or not.

All the executor code is now in the impl_pooledexecutor.py
and use the futures "PoolExecutor" API.

This use futurist to provide a eventlet and aioeventlet futures friendly
object.

Change-Id: I8cd7640f36beeda47560e3c82671bad3530e38d1
2015-07-08 13:42:10 +02:00
Brant Knudson
60624a62a4 Fix missing space in help text
The help text for the notification_driver option was missing a
space.

Change-Id: I7ccad49c308b04d8fd8e45cb2b0b937f2ca8b189
2015-04-29 09:47:02 -05:00
Davanum Srinivas
d214770b2b Document notification_driver possible values
Add list of drivers to help and FAQ.

Closes-Bug: #1440480
Change-Id: Ia0ea4693f88aa7550e55e7fb63111065ad9b6c3b
2015-04-06 10:12:08 -04:00
gtt116
fb8c431ad6 Make notify driver messaging play well with publish_errors
When publish_errors is True, and using `messaging` notify driver,
produces a infinite loop that report sending notification error.

The reason is we always passing None to the content argument in
logger handlers (LoggingNotificationHandler, LoggingErrorNotificationHandler),
when pack the context object which expected as a dict, raise TypeError
exception, so result in infinite retry looping.

To match what document said, passing a empty dict rather than None. Also
modify unittests to make sure context is a empty dict.

Change-Id: Ic2c4c43f5bbafa8107ea370ba959da16cfa4a24c
Closes-bug: #1346466
2015-03-28 11:32:24 +00:00