The change updates the imports to use simplejson library and fixes three
unit tests. In particular TestUtils.test_parse_alarm_state_hist_message
has been slightly changed to make testing more robust.
Also, Python 2 specific implemetation has been removed from
influxdb.line_utils .
Additionally, standard library unittest.mock is used instead of the
third party mock lib.
Depends-On: https://review.opendev.org/718014
Depends-On: https://review.opendev.org/720188
Change-Id: I64b1a60e8be929c25c005a0f429b1274cb8570e6
Story: 2007549
Task: 39390
The new configuration option is used to control the maximum size of
batches written to InfluxDB and optimizes performance for high loads.
The default value is set to 10 000 data points per batch.
Change-Id: I82c3e4e64984a996e41fe2657ce36b032b40d3fd
Story: 2007191
Task: 38319
At the moment, all data is retained infinitely. This change allows users
to specify a default_retention_hours (defaults to 0 meaning unlimited
retention) in the influxdb section of monasca persister configuration so
that new projects are automatically assigned this retention policy when
the given hour is greater than 0 to preserve original behaviour.
Story: 2006331
Task: 37234
Change-Id: I4136df1d43954eb026a104f3f85b3a58197f5435
At the moment, monasca persister assumes that an InfluxDB database
already exists. With this change, a new database is created when it does
not exist.
Story: 2006331
Task: 37327
Change-Id: I3d88eb8b2cdbd81f9c84fe1703b2ee2d1b09e1ab
data_points_class can be now declared as a List or a Dictionary. The
latter is used when writing measurements per tenant. This refactoring
avoids creating the dictionary and then chaining again its values in
case of ElasticSearch and Cassandra databases or if db_per_tenant option
is disabled. The change also fixes the events pipeline.
Change-Id: I25ea80eea714acb167e70f188ed229cc90531596
Story: 2006331
Task: 37211
At present, all time series are accumulated in the same database in
InfluxDB. This makes queries slow for tenants that have less data. This
patch enables the option to use separate database per tenancy.
This task implements the changes on monasca-persister which handles
write requests to InfluxDB database.
Change-Id: I7de6e0faf069b889d3953583b29a876c3d82c62c
Story: 2006331
Task: 36072
InfluxdbClientError has content attribute no message attribute.
Message attribute deprecated in Python 2.6 and it's go away
in Python 3.0
Change-Id: I5fe4bcb018a7c8c99df29d9798aa0e2736f33b6f
Story: 2005900
Task: 33762
The change introduces the possibility to run the persister with
the new confluent-kafka client. It has to be enabled in the
configuration file.
Story: 2003705
Task: 30117
Depends-On: https://review.opendev.org/675297
Change-Id: I05428b8ae9e0ba9af5b81d3b103434ebd5657108
metrics table in cassandra always contain row which has no metric_id.
it is casused by cassandra ttl.cassandra supports TTL column only,
so when update a row,metric_id must update as well.
Change-Id: Ia2d2e1585965dd31f9679e2a63497bf4cc68349e
Story: 2005832
Task: 33590
This change supports catching more parse errors which can jam
the persistor, such as:
`partial write: unable to parse 'string_with_invalid_char.%'`.
Change-Id: Ia12d04708a88308e88fcbd488d72f4313ba42a36
Story: 2005430
Task: 30468
persister get contact_points from configuration file
with "[ip.dest for ip in conf.cassandra.contact_points]".
cfg.dest replace "-" with "_".if user config cassandra
hosts cassandra-0,then persister crashed.
Task: 29957
Story: 2005195
Change-Id: I9ee29fdce781739f06683e4b4a80183669663189
some metrics get from kafka,may not a standard point that
influxdb can parse.in this case,points should be dropped.
Task: 29641
Story: 2005069
Change-Id: Ib907a280ced26ef2c228efe14b57440f894d8aad
- Change the default index prefix name for events.
Story: 2003955
Task: 26880
Depends-On: https://review.openstack.org/#/c/600030/
Change-Id: Idb6913229d3ac4e391eda0d6129eb2590bc7a678
* Update max line length to 100
* Clean up codes for pep8 checks from tox.ini
Change-Id: I974c0c31dc51784506cbf54b87bc450f2334845e
Signed-off-by: Jui Chandwaskar <jchandwaskar@op5.com>
This change extends the persister to read OpenStack
event objects from kafka topic and then stores
them into ElasticSearch.
Story: 2001112
Task: 4843
Change-Id: I35b0db67dc088e56ad281c84fc4b50fa7a064e44
To generate the sample config file, do what other OpenStack projects do
and use the oslo.config provided mechanism.
Story: 2001009
Task: 4201
Change-Id: I490de4110843f7dfd618a6a8cc8d6a4e35db65ce
The main improvement comes from using the Influxdb Line Protocol. The
encoding methods in line_utils.py are like the ones used in the influxdb
client but optimized for our data
Additional improvement comes from avoiding calls to encode('utf8') as
the influxdb client already does that.
On my test system, these changes increased the number of measurements
processed from about 2200/second to about 3700/second. Measurement
processing time is now dominated by Kafka. Approximately, 35% of time
is spent reading from Kafka and approximately 22% of time is committing
offsets. Only 10% of the time is spent writing to Influxdb. About 30% of
the time is spent converting messages from the json string read from
Kafka into the Line Protocol format for Influxdb.
Once monasca-common is modified to use the faster kafka library,
performance should be even better.
I did try using ujson, but my tests showed it wasn't any faster than
the json package.
Change-Id: I2acf76d9a5f583c74a272e18350b9c0ad5883f95
Add bandit job as part of pep8 in tox.ini
Had to fix one issue and mark two instances of try except pass as OK so that
bandit will pass
Change-Id: Ia1c96e27d1bae360c6ae0d4131665e2b712f573f
This patch set modifies lines which are importing objects
instead of modules. As per openstack import guide lines, user should
import modules in a file not objects.
http://docs.openstack.org/developer/hacking/#imports
Change-Id: I8af9278ffefcc20cf7447ecff7470217cd0e4632
Revival and adding some modification to support schema change.
See also: I914176d60bfce91fbe449702f7e78bb2f78706ce
Change-Id: I2152057ef4843b555a21118cb20bd9eff36c3b7a
Separate out the persister classes into their own python files
and clean up some unused code for start up.
Change-Id: Id1460baef9e06025ed57030438a0021c253cc99a
Adrian Czarnecki