openstack
/
monasca-persister
Code Issues Proposed changes
monasca-persister/monasca_persister
History
Craig Bryant a7112fd30b Increase Persister Performance 
The main improvement comes from using the Influxdb Line Protocol. The
encoding methods in line_utils.py are like the ones used in the influxdb
client but optimized for our data

Additional improvement comes from avoiding calls to encode('utf8') as
the influxdb client already does that.

On my test system, these changes increased the number of measurements
processed from about 2200/second to about 3700/second. Measurement
processing time is now dominated by Kafka. Approximately, 35% of time
is spent reading from Kafka and approximately 22% of time is committing
offsets. Only 10% of the time is spent writing to Influxdb. About 30% of
the time is spent converting messages from the json string read from
Kafka into the Line Protocol format for Influxdb.

Once monasca-common is modified to use the faster kafka library,
performance should be even better.

I did try using ujson, but my tests showed it wasn't any faster than
the json package.

Change-Id: I2acf76d9a5f583c74a272e18350b9c0ad5883f95
 		2017-06-22 14:00:41 -06:00
..
repositories Increase Persister Performance 2017-06-22 14:00:41 -06:00
tests Increase Persister Performance 2017-06-22 14:00:41 -06:00
LICENSE Add files for pypi. 2014-09-30 15:21:50 -06:00
README.md Refactor the python persister 2016-03-10 09:14:24 -07:00
__init__.py Refactor the python persister 2016-03-10 09:14:24 -07:00
persister.py Clean imports in code 2016-12-01 13:29:24 +07:00

README.md

Monasca Persister

A Monasca Persister written in Python.

Reads alarms and metrics from a Kafka queue and stores them in an InfluxDB database.

License

(C) Copyright 2014-2016 Hewlett Packard Enterprise Development Company LP

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.