Commit Graph

16 Commits (37411e792a8ee36eb04832785044b048719e9fcf)

Author SHA1 Message Date
Adrian Czarnecki df0fd4958f Remove six
Change-Id: I898c97fda4e1e93c64c6b75fbfea5af80ffd684a
Story: 2008305
Task: 41274
2020-11-24 15:10:57 +01:00
Witek Bedyk dc8956d7b8 Migrate from ujson to simplejson
The change updates the imports to use simplejson library and fixes three
unit tests. In particular TestUtils.test_parse_alarm_state_hist_message
has been slightly changed to make testing more robust.

Also, Python 2 specific implemetation has been removed from
influxdb.line_utils .

Additionally, standard library unittest.mock is used instead of the
third party mock lib.

Change-Id: I64b1a60e8be929c25c005a0f429b1274cb8570e6
Story: 2007549
Task: 39390
2020-04-21 12:35:08 +02:00
Witek Bedyk 0696cf54ee Add configuration option influxdb.batch_size
The new configuration option is used to control the maximum size of
batches written to InfluxDB and optimizes performance for high loads.
The default value is set to 10 000 data points per batch.

Change-Id: I82c3e4e64984a996e41fe2657ce36b032b40d3fd
Story: 2007191
Task: 38319
2020-01-21 11:07:22 +01:00
Bharat Kunwar 97aab2f004 Support default_retention_hours for influxdb
At the moment, all data is retained infinitely. This change allows users
to specify a default_retention_hours (defaults to 0 meaning unlimited
retention) in the influxdb section of monasca persister configuration so
that new projects are automatically assigned this retention policy when
the given hour is greater than 0 to preserve original behaviour.

Story: 2006331
Task: 37234

Change-Id: I4136df1d43954eb026a104f3f85b3a58197f5435
2019-11-15 13:47:47 +00:00
Bharat Kunwar df12bd8628 Support automatic db creation for InfluxDB
At the moment, monasca persister assumes that an InfluxDB database
already exists. With this change, a new database is created when it does
not exist.

Story: 2006331
Task: 37327

Change-Id: I3d88eb8b2cdbd81f9c84fe1703b2ee2d1b09e1ab
2019-10-31 23:12:00 +00:00
Witek Bedyk e883b5bcda Move data_points to repository class
data_points_class can be now declared as a List or a Dictionary. The
latter is used when writing measurements per tenant. This refactoring
avoids creating the dictionary and then chaining again its values in
case of ElasticSearch and Cassandra databases or if db_per_tenant option
is disabled. The change also fixes the events pipeline.

Change-Id: I25ea80eea714acb167e70f188ed229cc90531596
Story: 2006331
Task: 37211
2019-10-24 13:34:30 +02:00
Bharat Kunwar 774e981f87 Implement separate db per tenancy
At present, all time series are accumulated in the same database in
InfluxDB. This makes queries slow for tenants that have less data. This
patch enables the option to use separate database per tenancy.

This task implements the changes on monasca-persister which handles
write requests to InfluxDB database.

Change-Id: I7de6e0faf069b889d3953583b29a876c3d82c62c
Story: 2006331
Task: 36072
2019-09-27 08:31:07 +00:00
Arseni Lipinski e415ea71cb Change process_message() to static
Change-Id: I931a93f9de0df3373e5601a9d7abb29e7f90a8bc
2019-04-12 15:09:53 +02:00
Jui Chandwaskar 6abefff1b2 Update pep8 checks
* Update max line length to 100
* Clean up codes for pep8 checks from tox.ini

Change-Id: I974c0c31dc51784506cbf54b87bc450f2334845e
Signed-off-by: Jui Chandwaskar <>
2018-04-11 16:21:51 +02:00
Jakub Wachowski 16d8e69a4c Use ujson instead of json for parsing messages
ujson is simply faster than json :)

Change-Id: I1f33dda01cc4ee09f0bae1c2d2a33fbda65c1cac
2017-08-09 04:44:13 +00:00
Adrian Czarnecki 5f168778c6 Use oslo-config-generator
To generate the sample config file, do what other OpenStack projects do
and use the oslo.config provided mechanism.

Story: 2001009
Task:  4201

Change-Id: I490de4110843f7dfd618a6a8cc8d6a4e35db65ce
2017-07-28 13:48:01 +02:00
Jenkins 97a9a7032a Merge "Increase Persister Performance" 2017-06-26 05:19:10 +00:00
Craig Bryant a7112fd30b Increase Persister Performance
The main improvement comes from using the Influxdb Line Protocol. The
encoding methods in are like the ones used in the influxdb
client but optimized for our data

Additional improvement comes from avoiding calls to encode('utf8') as
the influxdb client already does that.

On my test system, these changes increased the number of measurements
processed from about 2200/second to about 3700/second. Measurement
processing time is now dominated by Kafka. Approximately, 35% of time
is spent reading from Kafka and approximately 22% of time is committing
offsets. Only 10% of the time is spent writing to Influxdb. About 30% of
the time is spent converting messages from the json string read from
Kafka into the Line Protocol format for Influxdb.

Once monasca-common is modified to use the faster kafka library,
performance should be even better.

I did try using ujson, but my tests showed it wasn't any faster than
the json package.

Change-Id: I2acf76d9a5f583c74a272e18350b9c0ad5883f95
2017-06-22 14:00:41 -06:00
Craig Bryant 7082ded836 Turn on bandit check as part of pep8
Add bandit job as part of pep8 in tox.ini

Had to fix one issue and mark two instances of try except pass as OK so that
bandit will pass

Change-Id: Ia1c96e27d1bae360c6ae0d4131665e2b712f573f
2017-02-08 11:22:18 -07:00
Cao Xuan Hoang 6733c064d9 Clean imports in code
This patch set modifies lines which are importing objects
instead of modules. As per openstack import guide lines, user should
import modules in a file not objects.

Change-Id: I8af9278ffefcc20cf7447ecff7470217cd0e4632
2016-12-01 13:29:24 +07:00
Michael James Hoppal 5b1ffc3e2b Refactor the python persister
Separate out the persister classes into their own python files
and clean up some unused code for start up.

Change-Id: Id1460baef9e06025ed57030438a0021c253cc99a
2016-03-10 09:14:24 -07:00