Commit Graph

62 Commits (5a7dcadc94ed4165cfc3c4f7e59e61c6ec5478bc)

Author SHA1 Message Date
Jenkins 4f7198597e Merge "Allow checking if notifier is enabled" 6 years ago
Matt Riedemann afd4ecfb42 Remove old messaging notify driver alias
15cd99050c added this alias
to workaround an issue with neutron using the internal
path in liberty. That was fixed in neutron in mitaka:


The alias was left for liberty, but liberty is end of life
now so we can remove this.

Change-Id: I145921b98d48e03df38ebcbaebc1a4375911f9ff
6 years ago
Balazs Gibizer ed149bae81 Allow checking if notifier is enabled
Generating the payload of a notificatiton might be expensive due
to database access. This expensive step is unnecessary if the
notification driver is set to noop because it means that the
generated payload will not be sent.

This patch makes it possible for the user of the notifier to check
if the notifier will emit any notification before the payload
is generated.

Cinder already implemented similar behavior in
I77f655d3ef90088ce71304da5d4ea7b543991e90 and nova also plans
to do it in I3e6741d59df49e1e58409314008c2ed609fdedc1

Change-Id: Ib992f5c20fef85224fb00823e1d8d9c6cff19bec
6 years ago
gord chung 94c818dc98 drop topic keyword from Notifier
we marked it for removal >=5.0.0

Change-Id: I4634c082c2383569c4aa8a6d9a8a560de7e3c12f
6 years ago
Li-zhigang ea8fad47a5 Replace six.iteritems() with .items()
1.As mentioned in [1], we should avoid using six.iteritems to achieve iterators.
We can use dict.items instead, as it will return iterators in PY3 as well.
And dict.items/keys will more readable. 2.In py2, the performance about
list should be negligible, see the link [2].

Change-Id: Ia235afc3532f62f265f91ca46d2306c72fc2a2a2
7 years ago
Hiroyasu.OHYAMA beb23103b0 [document] The example which is written in the developer guide of
'Notification Listener' doesn't work.

I fixed the example which is described in developer guide to work fine.
These are the main points of this correction.

* specify namespace for NotificationFilter.
* set 2 blank lines between class definitions according to E302 of pep8.
* change argument of 'pool' to keyward arguemnt, because 4th argument of
  'get_notification_listener' requires the name of message executor.

Change-Id: I75800309f2294fe9f17ea2a2b1638de57ef2922e
Closes-Bug: #1618394
7 years ago
Kenneth Giusti 9bc9c0dc6a Fixups to the inline documentation
Rework the inline documentation for executors, RPC servers and
clients, notifiers and notification listeners for clarity and flow.

Change-Id: If4f1db853a7fc85340177fd2c9c43a479d72459d
7 years ago
tengqm 3cdfe15897 Fix notify filter when data item is None
When data[k] is None during match check, we are getting annoying
TypeError exceptions like: "TypeError: expected string or buffer".
This patch adds a check that data[k] is something the regex can do

Change-Id: I75c8f602a59a90aeb5a6973f2e9e25708194ad78
7 years ago
George Silvis, III a011cb2448 Add docstring for get_notification_transport
This was previously undocumented.  Most of the information was taken from the
docstring for get_transport

Change-Id: I92652f4de9dd1e59ae5c3f9ce39e5ee808f24c7d
7 years ago
Gregory Haynes 317641c42f Fix syntax error on notification listener docs
We are missing a list delimiter in our docs example for a notification

Change-Id: Ib8b9c35e7bff241bf8473a98a2ee188da7e194e4
7 years ago
Julien Danjou ff9b4bb938 notify: add a CLI tool to manually send notifications
Change-Id: I01ebd8402e322c5c27dc2fc4c61fb5beb2dddae8
7 years ago
liusheng e57afac051 [Trival] fix a typo nit
Change-Id: I3d40726cf55aedb838ebe605f3d58618d594acca
7 years ago
Mehdi Abaakouk 5dd059aa3c Remove deprecated localcontext
Change-Id: Ib109ced5ee6a706853b0f54be6756cfaf63e2164
7 years ago
Dmitriy Ukhlov 4df633db74 Improves exception handling and logging
1) try to catch all possible exceptions during message acknowledging
    and dispatching to send ack/nack when we can to do it
2) improve logging in case of exceptions during message acknowledging
    and dispatching

Depends-On: I2d230d49e5aff6ab4d84ab8c3d2834f85e3405eb
Change-Id: I41a768c5624fa2212257ce20bf9a67d09de0c4ab
7 years ago
Jenkins f22a1a4fec Merge "Refactor base interfaces" 7 years ago
ZhiQiang Fan 5b94b4b7a9 use thread safe fnmatch
fnmatch is not thread safe in python <2.7.10, let's use the safe
one in oslo.utils

Change-Id: If3d480436506ecc5f3c2fcbd51ceaac59f2b7dec
7 years ago
Dmitriy Ukhlov 6db00c77b0 Refactor base interfaces
1) Add MessageHandler base interface for on_incoming_callback replacement
2) Move message_handler parameter form Listener's __init__() to start()
3) Remove wait method from listener

Change-Id: Id414446817e3d2ff67b815074d042a9ce637ec24
7 years ago
Jenkins 55c0b3b52f Merge "Make transport_url config option secret" 7 years ago
Juan Antonio Osorio Robles cbaf71edfa Make transport_url config option secret
The transport_url may contain credentials to the message queue; For
this reason, this option should be secret for it to not leak into the

Closes-Bug: #1567233

Change-Id: I49c641a2662976d7220e4222e3c4a4d2586b1336
7 years ago
Dmitriy Ukhlov 5d7d7253d1 Refactor driver's listener interface
Current Listener interface has poll() method which return messages

To use it we need have poller thread which is located in MessageHandlerServer
But my investigations of existing driver's code shows that some implemetations have
its own thread inside for processing connection event loop. This event loop received
messages and store in queue object. And then our poller's thread reads this queue
This situation can be improved. we can remove poller's thread, remove queue object
and just call on_message server's callback from connection eventloop thread

This path provide posibility to do this for one of drivers and leave as is other drivers

Change-Id: I3e3d4369d8fdadcecf079d10af58b1e4f5616047
7 years ago
Jenkins d4e8ac42b5 Merge "Move server related logic from dispatchers" 7 years ago
Dmitriy Ukhlov 990d894eaf Move server related logic from dispatchers
Dispatcher should be responsible for routing message to the
callback method of endpoint object and returning result back to the
server only. But now it is also responsible for sending reply,
ack/reque messages etc.

Also this patch makes small improvements:
   1) Notification dispatcher now requeue message if endpoint raises exception
   2) unstable behaviour of test_mask_passwords test is fixed

Change-Id: I5f23e23644e90919cb67f81fc306ee85c5e09974
7 years ago
Davanum Srinivas f8969e97ce [py34] replace file() with open()
Running flake8 with python3 finds this problem:
./oslo_messaging/notify/ F821 undefined name 'file'
        return file(filename, 'r')

Change-Id: I5cd7475618f7a7532db770c2c5c61576a657fbb4
7 years ago
Jenkins cc89fc3174 Merge "Revert "Ensure the json result type is bytes on Python 3"" 7 years ago
Victor Stinner bb4121a465 Revert "Ensure the json result type is bytes on Python 3"
This reverts commit bd81d09c02.

I understand that the change was supposed to fix something, but instead it broke all tests on Python 3!?

It's wrong to replace blindly json.dumps() with jsonutils.dump_as_bytes(). In oslo messaging, the result is usually used as a value in a dictionary, and then the whole dictionary is passed to a second serializer which also serialize to JSON.

Sorry, I don't understand everything, but at least I see that tests passed on py3 before the change, and started to fail with the change.

Maybe json(utils).dumps() is misused in some places, but in this case, you should write a change which only fix these specific places, not replace all calls to dumps().

Change-Id: Icd54ee8e3f5c976dfd50b4b62c7f51288649e112
7 years ago
Jenkins 67c8987372 Merge "Ensure the json result type is bytes on Python 3" 7 years ago
Davanum Srinivas 2d53db6c51 Allow Notifier to have multiple topics
Looks like there is a disconnect between the __init__
parameter 'topic' in Notifier and what we need when
we look up a driver. We should allow multiple topics
to be specified as a new topics parameter and pass
that along directly.

Change-Id: Id89957411aa219cff92fafec2f448c81cb57b3ca
7 years ago
Javeme bd81d09c02 Ensure the json result type is bytes on Python 3
Since the json.dumps() returns Unicode string On Python 3, to
ensure that the result type is bytes on Python 2 and Python 3
that if the result is used for the message body, let's replace
    json.dumps() /  oslo_serialization.jsonutils.dumps()

Change-Id: I0e0f6b715ffc4a9ad82be52e55696d032b6d0976
7 years ago
Javeme 3288c4d7e2 Use more efficient mask_dict_password to mask password
The mask_dict_password method is added from oslo_utils 3.4.0, and it's
about five times faster than the mask_password method, so it's better
to use mask_dict_password instead of mask_password.

Change-Id: I6629ed1232f2e6f1c72b9f13b361e2f9e14b50a4
7 years ago
Victor Stinner 22fea728cc Remove executor callback
Revert the change I556b112371bec2ec29cea4dc254bb3f9c6d2c07a: the
executor callback API was only used by the aioeventlet executor which
was just removed.

Change-Id: I1223f32594d8c1be28cc43fdd9bf102c86d75325
7 years ago
ChangBo Guo(gcb) 6b20fa8597 Improvement of logging acorrding to oslo.i18n guideline
1. Use translation marker functions, their argument must just be a string
2. Any message with more than one variable should use named
   interpolation instead of positional to allow translators
   to move the variables around in the string to account for
   differences in grammar and writing direction.
3. String interpolation should be delayed to be handled by the logging
   code, rather than being done at the point of the logging call.
For more details, please refert to oslo.i18n guideline [1]

Note: this commit doesn't touch test code.


Change-Id: I5f013d65b20396bbe0e5a2cceaed2a33fad1af23
8 years ago
Jenkins 0a0e6d0d50 Merge "notif: Check the driver features in dispatcher" 8 years ago
Jenkins 213176657d Merge "batch notification listener" 8 years ago
Jenkins 4b6144a3db Merge "creates a dispatcher abstraction" 8 years ago
Mehdi Abaakouk 050024f798 Fix notifier options registration
Change-Id: I37082f6f349e89af6b74e6ec5e5c416902299263
8 years ago
Mehdi Abaakouk 185f94c013 notif: Check the driver features in dispatcher
The transport/driver features check is done into the get listener
So when these methods are not used the driver features checks is not

This change moves it into the dispatcher layer to ensure the
requirements are always checked.

This changes a bit the behavior of when the check occurs. Before
it was during the listener object initialisation. Now this
when the listener server start.

Change-Id: I4d81a4e8496f04d62e48317829d5dd8b942d501c
8 years ago
Mehdi Abaakouk 4dd644ac20 batch notification listener
Gnocchi performs better if measurements are write in batch
When Ceilometer is used with Gnocchi, this is not possible.

This change introduce a new notification listener that allows that.

On the driver side, a default batch implementation is provided.
It's just call the legacy poll method many times.

Driver can override it to provide a better implementation.
For example, kafka handles batch natively and take benefit of this.

Change-Id: I16184da24b8661aff7f4fba6196ecf33165f1a77
8 years ago
Mehdi Abaakouk bdf287e847 creates a dispatcher abstraction
This change creates a dispatcher abstraction
to document the interface of a dispatcher.

And also allows in the futur to have attributes with default values.

Change-Id: I9a7e5e03f89635a3790b3851f492a1a7aab58feb
8 years ago
Davanum Srinivas 33c1010c32 Option group for notifications
In change Ief6f95ea906bfd95b3218a930c9db5d8a764beb9, we 
decoupled RPC and Notifications a bit. We should take another
step and separate out the options for notifications into 
its own group.

Change-Id: Ib51e2839f9035d0cc0e3f459939d9f9003a8c810
8 years ago
Jenkins 9938f668f4 Merge "Add Warning when we cannot notify" 8 years ago
Davanum Srinivas 6dba2ed591 Add Warning when we cannot notify
Calling Notifier.audit() won't work with LogDriver as the
logger does not have a audit level anymore, So let's at
least not fail silently and let the operators know that
we are dropping stuff on the floor.

Closes-Bug: #1518170
Change-Id: I74002c72e6763ea8b5df7d97d722619bd4a1950b
8 years ago
Julien Danjou 47a906aff3 middleware: remove oslo.context usage
There's nothing interesting in the fake context we're passing. Let's
stop depending on oslo.context all together here.

Change-Id: I7784bc3b818e67118e03905857c39eac66765fad
8 years ago
Flavio Percoco 925eb734a9 Remove qpidd's driver from the tree
Back in liberty we marked this driver as deprecated. This patch removes
it from the tree. The patch also removes tests, options and other
references in the documentation. Note that one script is being kept
because it's required by the amqp driver.

Depends-On: If4b1773334e424d1f4a4e112bd1f10aca62682a9
Change-Id: I4a9cba314c4a2f24307504fa7b5427424268b114
8 years ago
Matt Riedemann 15cd99050c Provide alias to oslo_messaging.notify._impl_messaging
Ifb96c2ae9868426cac2700bf4917c27c02c90b15 moved the _impl_messaging
module to oslo_messaging.notify.messaging which breaks neutron.

Neutron is fixed on master for mitaka but neutron on stable/liberty
is broken, and changing neutron on stable/liberty to use the new
path would require a global-requirements minimum version bump for
oslo.messaging to 2.6.0, which we want to avoid for people that have
already shipped liberty.

So provide an alias to the moved module so neutron in stable/liberty
continues to work. We deprecate the module so consumers know they
need to upgrade and move off this. We may need to cap oslo.messaging
in global-requirements on stable/liberty at some point when we remove
the deprecated alias module.

Change-Id: I29453e0fbf30b0a571c2b1afc7cc81d1a11535f0
Closes-Bug: #1513630
8 years ago
Davanum Srinivas 6621b9010e Decouple transport for RPC and Notification
Add a new configuration option for setting up
an alternate notification_transport_url that
can be used for notifications. This allows
operators to separate the transport mechanisms
used for RPC and Notifications.


Closes-Bug: #1504622
Change-Id: Ief6f95ea906bfd95b3218a930c9db5d8a764beb9
8 years ago
Cyril Roelandt c4a7ac0b65 Use yaml.safe_load instead of yaml.load
We currently use yaml.load to read a user-written config file. This can
lead to malicious code execution, so we should use yaml.safe_load

Found using bandit.

Change-Id: I27792f0435bc3cb9b9d31846d07a8d47a1e7679d
8 years ago
Daisuke Fujita 64fb61d37c Fix typo in rpc/ and notify/
1. "messaging configuration configuration" to "messaging configuration"

Change-Id: Icda5d2df67cc4d2d0e0cb3a453d0b04dd998e5f4
8 years ago
Davanum Srinivas 1893c495f6 Allow custom notification drivers
Our class hierarchy hides classes and modules that so its hard
for folks to write a custom Notification driver. We should
make these public and document them

Closes-Bug: #1426046
Change-Id: Ifb96c2ae9868426cac2700bf4917c27c02c90b15
8 years ago
Jenkins b02dd25655 Merge "Mask passwords when logging messages" 8 years ago
Ryan Rossiter c990ee02fa Mask passwords when logging messages
When logging a message, any secrets and passwords should be masked. This
uses oslo_utils.strutils to mask any passwords that are to be logged.

Change-Id: I263d44c0f2e900c5f6e210cbd7ec56e48d0d5bb2
Closes-Bug: #1487038
8 years ago