a7112fd30b
The main improvement comes from using the Influxdb Line Protocol. The encoding methods in line_utils.py are like the ones used in the influxdb client but optimized for our data Additional improvement comes from avoiding calls to encode('utf8') as the influxdb client already does that. On my test system, these changes increased the number of measurements processed from about 2200/second to about 3700/second. Measurement processing time is now dominated by Kafka. Approximately, 35% of time is spent reading from Kafka and approximately 22% of time is committing offsets. Only 10% of the time is spent writing to Influxdb. About 30% of the time is spent converting messages from the json string read from Kafka into the Line Protocol format for Influxdb. Once monasca-common is modified to use the faster kafka library, performance should be even better. I did try using ujson, but my tests showed it wasn't any faster than the json package. Change-Id: I2acf76d9a5f583c74a272e18350b9c0ad5883f95 |
||
---|---|---|
.. | ||
__init__.py | ||
test_influxdb.py | ||
test_persister_main.py |