The new configuration option is used to control the maximum size of
batches written to InfluxDB and optimizes performance for high loads.
The default value is set to 10 000 data points per batch.
Change-Id: I82c3e4e64984a996e41fe2657ce36b032b40d3fd
Story: 2007191
Task: 38319
At the moment, all data is retained infinitely. This change allows users
to specify a default_retention_hours (defaults to 0 meaning unlimited
retention) in the influxdb section of monasca persister configuration so
that new projects are automatically assigned this retention policy when
the given hour is greater than 0 to preserve original behaviour.
Story: 2006331
Task: 37234
Change-Id: I4136df1d43954eb026a104f3f85b3a58197f5435
At the moment, monasca persister assumes that an InfluxDB database
already exists. With this change, a new database is created when it does
not exist.
Story: 2006331
Task: 37327
Change-Id: I3d88eb8b2cdbd81f9c84fe1703b2ee2d1b09e1ab
data_points_class can be now declared as a List or a Dictionary. The
latter is used when writing measurements per tenant. This refactoring
avoids creating the dictionary and then chaining again its values in
case of ElasticSearch and Cassandra databases or if db_per_tenant option
is disabled. The change also fixes the events pipeline.
Change-Id: I25ea80eea714acb167e70f188ed229cc90531596
Story: 2006331
Task: 37211
At present, all time series are accumulated in the same database in
InfluxDB. This makes queries slow for tenants that have less data. This
patch enables the option to use separate database per tenancy.
This task implements the changes on monasca-persister which handles
write requests to InfluxDB database.
Change-Id: I7de6e0faf069b889d3953583b29a876c3d82c62c
Story: 2006331
Task: 36072
The main improvement comes from using the Influxdb Line Protocol. The
encoding methods in line_utils.py are like the ones used in the influxdb
client but optimized for our data
Additional improvement comes from avoiding calls to encode('utf8') as
the influxdb client already does that.
On my test system, these changes increased the number of measurements
processed from about 2200/second to about 3700/second. Measurement
processing time is now dominated by Kafka. Approximately, 35% of time
is spent reading from Kafka and approximately 22% of time is committing
offsets. Only 10% of the time is spent writing to Influxdb. About 30% of
the time is spent converting messages from the json string read from
Kafka into the Line Protocol format for Influxdb.
Once monasca-common is modified to use the faster kafka library,
performance should be even better.
I did try using ujson, but my tests showed it wasn't any faster than
the json package.
Change-Id: I2acf76d9a5f583c74a272e18350b9c0ad5883f95
This patch set modifies lines which are importing objects
instead of modules. As per openstack import guide lines, user should
import modules in a file not objects.
http://docs.openstack.org/developer/hacking/#imports
Change-Id: I8af9278ffefcc20cf7447ecff7470217cd0e4632
Separate out the persister classes into their own python files
and clean up some unused code for start up.
Change-Id: Id1460baef9e06025ed57030438a0021c253cc99a
Adrian Czarnecki