The change updates the imports to use simplejson library and fixes three
unit tests. In particular TestUtils.test_parse_alarm_state_hist_message
has been slightly changed to make testing more robust.
Also, Python 2 specific implemetation has been removed from
Additionally, standard library unittest.mock is used instead of the
third party mock lib.
The new configuration option is used to control the maximum size of
batches written to InfluxDB and optimizes performance for high loads.
The default value is set to 10 000 data points per batch.
At the moment, all data is retained infinitely. This change allows users
to specify a default_retention_hours (defaults to 0 meaning unlimited
retention) in the influxdb section of monasca persister configuration so
that new projects are automatically assigned this retention policy when
the given hour is greater than 0 to preserve original behaviour.
At the moment, monasca persister assumes that an InfluxDB database
already exists. With this change, a new database is created when it does
data_points_class can be now declared as a List or a Dictionary. The
latter is used when writing measurements per tenant. This refactoring
avoids creating the dictionary and then chaining again its values in
case of ElasticSearch and Cassandra databases or if db_per_tenant option
is disabled. The change also fixes the events pipeline.
At present, all time series are accumulated in the same database in
InfluxDB. This makes queries slow for tenants that have less data. This
patch enables the option to use separate database per tenancy.
This task implements the changes on monasca-persister which handles
write requests to InfluxDB database.
The main improvement comes from using the Influxdb Line Protocol. The
encoding methods in line_utils.py are like the ones used in the influxdb
client but optimized for our data
Additional improvement comes from avoiding calls to encode('utf8') as
the influxdb client already does that.
On my test system, these changes increased the number of measurements
processed from about 2200/second to about 3700/second. Measurement
processing time is now dominated by Kafka. Approximately, 35% of time
is spent reading from Kafka and approximately 22% of time is committing
offsets. Only 10% of the time is spent writing to Influxdb. About 30% of
the time is spent converting messages from the json string read from
Kafka into the Line Protocol format for Influxdb.
Once monasca-common is modified to use the faster kafka library,
performance should be even better.
I did try using ujson, but my tests showed it wasn't any faster than
the json package.
This patch set modifies lines which are importing objects
instead of modules. As per openstack import guide lines, user should
import modules in a file not objects.