- kombu4 wraps recoverable errors as OperationalErrors rather than
raising amqp errors
- also, raise a recoverable error and redeclare if for some reason a
message is double acknowledged... previously, this was hidden.
- ensure socket is not none
- use connect method to ensure connection
Depends-On: I9f980b51901ac31599b9651633956ad2eea6a1ac
Change-Id: I73958c8057353a2eefe1baaa7a41148193d507f7
Currently Kafka driver for an oslo.messaging uses kafka-python==0.9.5
and mostly broken. This package version supports only low level Kafka
producer and consumer API which are marked as deprecated now [1]. Using
of these interfaces bring a big concern to the message processing,
because current KafkaConsumer has not any consuming coordination. This
fact causes a message duplication for the several consumers of one
topic. This behavior is specific to Ceilometer and services which read
and process notifications from other services.
New version of kafka-python allows to use async thread safe message
producers and coordinated consumers [1].
[1] http://kafka-python.readthedocs.io/en/master/changelog.html#feb-15-2016
The driver is currently experimental, python-kafka<1.0.0 API have major
issue described above that can't make the oslo.messaging driver works,
so we prefer having a working driver with a non-synced dependencies, that the
reverse.
Co-Authored-By: Mehdi Abaakouk <sileht@redhat.com>
Change-Id: I29862ed7bf56b9d8878fa8e9fb1cbd9d643908a4
Adding constraints support to libraries is slightly more complex than
services as the libraries themselves are listed in upper-constraints.txt
which leads to errors that you can't install a specific version and a
constrained version.
This change adds constraints support by also adding a helper script to
edit the constraints to remove oslo.messaging.
Change-Id: I8be883215f27abb58d15b85e8542cbdf32000bac
1) Remove hardcoding 'redis' matchmaker from any zmq:// transport because it's
fails in cases zmq+etcd:// or zmq+sentinel:// urls.
2) Allow to use transport_url from config file by removing default value for
args.url option. In this case by default args.url is None and
transport_url will be used from config file.
Change-Id: I1edcabe46b1e92dd4e6eccc9151b50d03f394186
Do initialization of the generator only for rpc/notify clients.
No needs to load messages_length.yaml and calculate distribution for servers.
Change-Id: Ib1ac30181bec20eacf6d29ccd5f76c0b6b1e96f8
This patch provide the sync flag:
simulator.py rpc-client --sync call
simulator.py rpc-client --sync fanout
The --sync values means next:
- call: to send sync msg via rpc.call
- fanout: to broadcast sync msg via rpc.fanout to all servers on topic
When clients has sent all messages, the rpc-server will be stopped
automatically and dump statistics to a file (if --json-file is used).
This is much usefull than rough killing the process of the server in
benchmark frameworks.
Change-Id: I06fd8dbbcdc8b2b9f13029029f730b417ff128ce
--debug, --is-cast and --is-fanout are defined as args of type=bool.
This means that, for example, if we want to enable debug logging
level, we have to type '--debug True'. But we can also use
'--debug False' in order to do the same, which is very misleading
(in fact, any non-empty string will evaluate to True). This patch
tries to solve this problem by replacing type=bool and
default=False with action='store_true' for these args, so that
we will be able to enable them (they will remain False by default
as before) simply as '--debug' etc.
Change-Id: I8ee04c35427df446966161491da8d264b44975bf
When we calculate metrics like msg/sec, latency, etc
we expect the start and end time as time of
the first and last processed message:
RPC Server life timeline:
[----0..5 sec----][---5..10 sec---][---10..15 sec--]
waiting clients 10 msg recved wait sigint
expected: duration 5 sec, 2 msg/sec
actual (incorrect): duration 15 sec, 0.6 msg/sec
no reason to set the boundaries if server was idle few seconds
before running of clients and after.
Change-Id: I33e0a605b54ea7b89977504892528c41c3b00a68
ZeroMQ driver options are current stored into the DEFAULT group.
This change makes the zmq configuration clearer by putting its
options into oslo_messaging_zmq group.
Change-Id: Ia00fda005b1664750d2646f8c82ebdf295b156fb
Closes-bug: #1417040
Co-Authored-By: Oleksii Zamiatin <ozamiatin@mirantis.com>
Now it is possible to stop simulator client and server
by sending SIGINT or SIGTERM signals. Note that both stop
gracefully and it takes some time to do this.
Change-Id: Ie3fd1ea8b146070d61a247fd8ccc124df8d34848
Right now simulator.py expects messages_length.yaml to be in the
current directory and as a result, it is impossible to launch
simulator while you are at the repository root.
The change makes simulator to search for the file in the same
directory as simulator itself is in.
Change-Id: Ic2ca21d87e2d888432dc6204c7b4b995a06aecbd
Aligning message sending and stats collection to whole seconds.
This allows to merge data from client and server - useful
for visualization of message flow.
Change-Id: I507c021d851254d4d84e8922de687931f8545864
This patch introduces new parameter --json <file name>.
When it is specified the simulator stores stats in JSON file.
Change-Id: I0b7a87ee614f44baa920569fe0c7046269c90925
Calculate message latency: for RPC calls on both server-side and
on client-side (round-trip); for RPC cast and NOTIFY on server-side
only.
The message is extended with metadata: sequence id, creation time,
reception time and return time (for RPC calls). For every message
the metadata is stored into MessageStatCollector. To reduce memory
consumption the collector aggregates stats on every second and can
print the ongoing stats to stdout. At the finish the overall stats
are printed.
NOTIFY tests are changed to be similar to RPC:
* use pre-generated messages
* parameter "-w WAIT_BEFORE_ANSWER" for processing delay
* parameter "--requeue" to re-queue new messages (only those that
miss the cache)
Change-Id: I4c2fc11bfaee17b6825cdc7d6edf07b8e91ef83a
Dependency on scipy results in extra requirements to the system
(include the need of Fortran compiler). With this patch the
weighted random choice algorithm is implemented directly.
Change-Id: I60cbc6c452945000add5a65263d1ab4e42dd91f9
We need to use eventlet.sleep() in green thread loops
in order to have more chances to switch between them.
Change-Id: I7c08e82182b68b95c36265d58df0644c5ce2c171
"n-t1" as notification topic not really useful. Let's allow
it to be specified from the command line. Also make sure
we set the notification driver as "messaging" by default
Change-Id: I5daf09b9a01e4d642b0f53d7634d5b305d5a82cb
Added fanout option to target
Fixed timeout option
Removed setting config values as they are
reverted to defaults on driver load
Change-Id: Ia2d8a7bbcb04706deb3d701c1a0f93a0dd835a19
The function yaml.load() provides the ability to construct an arbitrary
Python object. For security, we use yaml.safe_load() instead which
limits this ability to simple Python objects(like integers or lists).
ref: https://en.wikipedia.org/wiki/YAML#Security
Change-Id: Iea2a2d79c764d635b02c5d6d36c9a5652010d716
* passing targets to rpc-client using -tg arg
so clients can send messages to different targets
* removed redis.flush_db() so we can start
several simulator processes for zmq.
Change-Id: I0d36c5d8cbec938c8fd85d1041d2a3bff02b49e5
There are files containing string format arguments inside
logging messages. Using logging function parameters should
be preferred.
Change-Id: Iaa15bc52d1d8832032cc46205effb49630cea494
Closes-Bug: #1321274
passing config with --config-file
passing list of topics to use with -tp arg;
passing server name with -s arg
Change-Id: I8dcce9e725117f8a046135b94d8d2047d24bbf76
Since xrange has been removed in Python 3, this patch let's replace xrange()
with six.moves.range() to provide py3 compatibility.
Change-Id: I85e0748786867334756a51dcb834f84ffd1e55a7
* Avoid string concatenation during logging, especially
when logging level is disabled.
* Initialize the Randomized strings we will be using
as payloads *before* we start sending the messages
as the string creation takes time and reduces
throughput
Change-Id: I546229fe7ade95572e11bcda95a587228d84bc28
Gnocchi performs better if measurements are write in batch
When Ceilometer is used with Gnocchi, this is not possible.
This change introduce a new notification listener that allows that.
On the driver side, a default batch implementation is provided.
It's just call the legacy poll method many times.
Driver can override it to provide a better implementation.
For example, kafka handles batch natively and take benefit of this.
Change-Id: I16184da24b8661aff7f4fba6196ecf33165f1a77
gord chung