Commit Graph

9 Commits (37411e792a8ee36eb04832785044b048719e9fcf)

Author SHA1 Message Date
Adrian Czarnecki df0fd4958f Remove six
Change-Id: I898c97fda4e1e93c64c6b75fbfea5af80ffd684a
Story: 2008305
Task: 41274
2020-11-24 15:10:57 +01:00
Witek Bedyk 0696cf54ee Add configuration option influxdb.batch_size
The new configuration option is used to control the maximum size of
batches written to InfluxDB and optimizes performance for high loads.
The default value is set to 10 000 data points per batch.

Change-Id: I82c3e4e64984a996e41fe2657ce36b032b40d3fd
Story: 2007191
Task: 38319
2020-01-21 11:07:22 +01:00
Bharat Kunwar 97aab2f004 Support default_retention_hours for influxdb
At the moment, all data is retained infinitely. This change allows users
to specify a default_retention_hours (defaults to 0 meaning unlimited
retention) in the influxdb section of monasca persister configuration so
that new projects are automatically assigned this retention policy when
the given hour is greater than 0 to preserve original behaviour.

Story: 2006331
Task: 37234

Change-Id: I4136df1d43954eb026a104f3f85b3a58197f5435
2019-11-15 13:47:47 +00:00
Bharat Kunwar df12bd8628 Support automatic db creation for InfluxDB
At the moment, monasca persister assumes that an InfluxDB database
already exists. With this change, a new database is created when it does
not exist.

Story: 2006331
Task: 37327

Change-Id: I3d88eb8b2cdbd81f9c84fe1703b2ee2d1b09e1ab
2019-10-31 23:12:00 +00:00
Witek Bedyk e883b5bcda Move data_points to repository class
data_points_class can be now declared as a List or a Dictionary. The
latter is used when writing measurements per tenant. This refactoring
avoids creating the dictionary and then chaining again its values in
case of ElasticSearch and Cassandra databases or if db_per_tenant option
is disabled. The change also fixes the events pipeline.

Change-Id: I25ea80eea714acb167e70f188ed229cc90531596
Story: 2006331
Task: 37211
2019-10-24 13:34:30 +02:00
Bharat Kunwar 774e981f87 Implement separate db per tenancy
At present, all time series are accumulated in the same database in
InfluxDB. This makes queries slow for tenants that have less data. This
patch enables the option to use separate database per tenancy.

This task implements the changes on monasca-persister which handles
write requests to InfluxDB database.

Change-Id: I7de6e0faf069b889d3953583b29a876c3d82c62c
Story: 2006331
Task: 36072
2019-09-27 08:31:07 +00:00
Craig Bryant a7112fd30b Increase Persister Performance
The main improvement comes from using the Influxdb Line Protocol. The
encoding methods in are like the ones used in the influxdb
client but optimized for our data

Additional improvement comes from avoiding calls to encode('utf8') as
the influxdb client already does that.

On my test system, these changes increased the number of measurements
processed from about 2200/second to about 3700/second. Measurement
processing time is now dominated by Kafka. Approximately, 35% of time
is spent reading from Kafka and approximately 22% of time is committing
offsets. Only 10% of the time is spent writing to Influxdb. About 30% of
the time is spent converting messages from the json string read from
Kafka into the Line Protocol format for Influxdb.

Once monasca-common is modified to use the faster kafka library,
performance should be even better.

I did try using ujson, but my tests showed it wasn't any faster than
the json package.

Change-Id: I2acf76d9a5f583c74a272e18350b9c0ad5883f95
2017-06-22 14:00:41 -06:00
Cao Xuan Hoang 6733c064d9 Clean imports in code
This patch set modifies lines which are importing objects
instead of modules. As per openstack import guide lines, user should
import modules in a file not objects.

Change-Id: I8af9278ffefcc20cf7447ecff7470217cd0e4632
2016-12-01 13:29:24 +07:00
Michael James Hoppal 5b1ffc3e2b Refactor the python persister
Separate out the persister classes into their own python files
and clean up some unused code for start up.

Change-Id: Id1460baef9e06025ed57030438a0021c253cc99a
2016-03-10 09:14:24 -07:00