Commit Graph

3 Commits (37411e792a8ee36eb04832785044b048719e9fcf)

Author SHA1 Message Date
Witek Bedyk dc8956d7b8 Migrate from ujson to simplejson
The change updates the imports to use simplejson library and fixes three
unit tests. In particular TestUtils.test_parse_alarm_state_hist_message
has been slightly changed to make testing more robust.

Also, Python 2 specific implemetation has been removed from
influxdb.line_utils .

Additionally, standard library unittest.mock is used instead of the
third party mock lib.

Change-Id: I64b1a60e8be929c25c005a0f429b1274cb8570e6
Story: 2007549
Task: 39390
2020-04-21 12:35:08 +02:00
Jakub Wachowski 16d8e69a4c Use ujson instead of json for parsing messages
ujson is simply faster than json :)

Change-Id: I1f33dda01cc4ee09f0bae1c2d2a33fbda65c1cac
2017-08-09 04:44:13 +00:00
Craig Bryant a7112fd30b Increase Persister Performance
The main improvement comes from using the Influxdb Line Protocol. The
encoding methods in are like the ones used in the influxdb
client but optimized for our data

Additional improvement comes from avoiding calls to encode('utf8') as
the influxdb client already does that.

On my test system, these changes increased the number of measurements
processed from about 2200/second to about 3700/second. Measurement
processing time is now dominated by Kafka. Approximately, 35% of time
is spent reading from Kafka and approximately 22% of time is committing
offsets. Only 10% of the time is spent writing to Influxdb. About 30% of
the time is spent converting messages from the json string read from
Kafka into the Line Protocol format for Influxdb.

Once monasca-common is modified to use the faster kafka library,
performance should be even better.

I did try using ujson, but my tests showed it wasn't any faster than
the json package.

Change-Id: I2acf76d9a5f583c74a272e18350b9c0ad5883f95
2017-06-22 14:00:41 -06:00