Commit Graph

38 Commits (37411e792a8ee36eb04832785044b048719e9fcf)

Author SHA1 Message Date
Adrian Czarnecki df0fd4958f Remove six
Change-Id: I898c97fda4e1e93c64c6b75fbfea5af80ffd684a
Story: 2008305
Task: 41274
3 years ago
Hervé Beraud ac9d9b7351 Stop to use the __future__ module.
The __future__ module [1] was used in this context to ensure compatibility
between python 2 and python 3.

We previously dropped the support of python 2.7 [2] and now we only support
python 3 so we don't need to continue to use this module and the imports
listed below.

Imports commonly used and their related PEPs:
- `division` is related to PEP 238 [3]
- `print_function` is related to PEP 3105 [4]
- `unicode_literals` is related to PEP 3112 [5]
- `with_statement` is related to PEP 343 [6]
- `absolute_import` is related to PEP 328 [7]


Change-Id: I937b48769403d0c78af5069e29768bb7bcdb7b0b
3 years ago
Witek Bedyk dc8956d7b8 Migrate from ujson to simplejson
The change updates the imports to use simplejson library and fixes three
unit tests. In particular TestUtils.test_parse_alarm_state_hist_message
has been slightly changed to make testing more robust.

Also, Python 2 specific implemetation has been removed from
influxdb.line_utils .

Additionally, standard library unittest.mock is used instead of the
third party mock lib.

Change-Id: I64b1a60e8be929c25c005a0f429b1274cb8570e6
Story: 2007549
Task: 39390
3 years ago
Witek Bedyk 0696cf54ee Add configuration option influxdb.batch_size
The new configuration option is used to control the maximum size of
batches written to InfluxDB and optimizes performance for high loads.
The default value is set to 10 000 data points per batch.

Change-Id: I82c3e4e64984a996e41fe2657ce36b032b40d3fd
Story: 2007191
Task: 38319
3 years ago
Bharat Kunwar 97aab2f004 Support default_retention_hours for influxdb
At the moment, all data is retained infinitely. This change allows users
to specify a default_retention_hours (defaults to 0 meaning unlimited
retention) in the influxdb section of monasca persister configuration so
that new projects are automatically assigned this retention policy when
the given hour is greater than 0 to preserve original behaviour.

Story: 2006331
Task: 37234

Change-Id: I4136df1d43954eb026a104f3f85b3a58197f5435
4 years ago
Bharat Kunwar df12bd8628 Support automatic db creation for InfluxDB
At the moment, monasca persister assumes that an InfluxDB database
already exists. With this change, a new database is created when it does
not exist.

Story: 2006331
Task: 37327

Change-Id: I3d88eb8b2cdbd81f9c84fe1703b2ee2d1b09e1ab
4 years ago
Witek Bedyk e883b5bcda Move data_points to repository class
data_points_class can be now declared as a List or a Dictionary. The
latter is used when writing measurements per tenant. This refactoring
avoids creating the dictionary and then chaining again its values in
case of ElasticSearch and Cassandra databases or if db_per_tenant option
is disabled. The change also fixes the events pipeline.

Change-Id: I25ea80eea714acb167e70f188ed229cc90531596
Story: 2006331
Task: 37211
4 years ago
Bharat Kunwar 774e981f87 Implement separate db per tenancy
At present, all time series are accumulated in the same database in
InfluxDB. This makes queries slow for tenants that have less data. This
patch enables the option to use separate database per tenancy.

This task implements the changes on monasca-persister which handles
write requests to InfluxDB database.

Change-Id: I7de6e0faf069b889d3953583b29a876c3d82c62c
Story: 2006331
Task: 36072
4 years ago
zhangjianweibj c78d5ebc43 Fix InfluxdbClientError exception bug
InfluxdbClientError has content attribute no message attribute.
Message attribute deprecated in Python 2.6 and it's go away
in Python 3.0

Change-Id: I5fe4bcb018a7c8c99df29d9798aa0e2736f33b6f
Story: 2005900
Task: 33762
4 years ago
Witek Bedyk b88084003f Use Confluent Kafka client
The change introduces the possibility to run the persister with
the new confluent-kafka client. It has to be enabled in the
configuration file.

Story: 2003705
Task: 30117

Change-Id: I05428b8ae9e0ba9af5b81d3b103434ebd5657108
4 years ago
zhangjianweibj 506717e23e cassandra metrics_id contain null value bug
metrics table in cassandra always contain row which has no metric_id.
it is casused by cassandra ttl.cassandra supports TTL column only,
so when update a row,metric_id must update as well.

Change-Id: Ia2d2e1585965dd31f9679e2a63497bf4cc68349e
Story: 2005832
Task: 33590
4 years ago
Zuul 19d4358893 Merge "Minor refactoring for" 4 years ago
Arseni Lipinski f4ea2f9db0 Minor refactoring for
Change-Id: Ie40f22f768e67f112081524446174383845fb9b7
4 years ago
Zuul eda5e3ae79 Merge "Change process_message() to static" 4 years ago
Arseni Lipinski e415ea71cb Change process_message() to static
Change-Id: I931a93f9de0df3373e5601a9d7abb29e7f90a8bc
4 years ago
Doug Szumski f43d019a67 Widen exception catch for point parse failure
This change supports catching more parse errors which can jam
the persistor, such as:
`partial write: unable to parse 'string_with_invalid_char.%'`.

Change-Id: Ia12d04708a88308e88fcbd488d72f4313ba42a36
Story: 2005430
Task: 30468
4 years ago
zhangjianweibj 2bef4e500a cassandra contact_points bug
persister get contact_points from configuration file
with "[ip.dest for ip in conf.cassandra.contact_points]".
cfg.dest replace "-" with "_".if user config cassandra
hosts cassandra-0,then persister crashed.
Task: 29957
Story: 2005195

Change-Id: I9ee29fdce781739f06683e4b4a80183669663189
4 years ago
zhangjianweibj 0ab568481f some points unable to parse
some metrics get from kafka,may not a standard point that
influxdb can this case,points should be dropped.

Task: 29641
Story: 2005069

Change-Id: Ib907a280ced26ef2c228efe14b57440f894d8aad
4 years ago
Martin Chacon Piza 473dea341f Parse envelope fields from kafka to prepare the ES index
- Change the default index prefix name for events.

Story: 2003955
Task: 26880

Change-Id: Idb6913229d3ac4e391eda0d6129eb2590bc7a678
5 years ago
Martin Chacon Piza 71b4d50f6e Fixed the handling of metrics outside of the retention policy
Change-Id: I909c10eede6a0835ea02ca334914396dd1119801
Story: 2003202
Task: 23358
5 years ago
Jui Chandwaskar 6abefff1b2 Update pep8 checks
* Update max line length to 100
* Clean up codes for pep8 checks from tox.ini

Change-Id: I974c0c31dc51784506cbf54b87bc450f2334845e
Signed-off-by: Jui Chandwaskar <>
5 years ago
Zuul c923911cb5 Merge "Allow persister to handle retention policies in influx" 5 years ago
James Gu 20337572e3 Add Cassandra db support
Added cassandra db plugins in persister, inlcuding Java and
python version.

Change-Id: I4bbe48f8fe550385de6ed3e14120850a8542c7c9
story: 2001231
task: 5758
6 years ago
Jakub Wachowski 8273de7ca6 Add support for persisting events
This change extends the persister to read OpenStack
event objects from kafka topic and then stores
them into ElasticSearch.

Story: 2001112
Task: 4843

Change-Id: I35b0db67dc088e56ad281c84fc4b50fa7a064e44
6 years ago
ryan-brandt 3dde66dd33 Allow persister to handle retention policies in influx
Change-Id: I99f3bc265f347c805fc13cad5e857f70f791851a
6 years ago
Jakub Wachowski 16d8e69a4c Use ujson instead of json for parsing messages
ujson is simply faster than json :)

Change-Id: I1f33dda01cc4ee09f0bae1c2d2a33fbda65c1cac
6 years ago
Adrian Czarnecki 5f168778c6 Use oslo-config-generator
To generate the sample config file, do what other OpenStack projects do
and use the oslo.config provided mechanism.

Story: 2001009
Task:  4201

Change-Id: I490de4110843f7dfd618a6a8cc8d6a4e35db65ce
6 years ago
Jenkins 97a9a7032a Merge "Increase Persister Performance" 6 years ago
Craig Bryant a7112fd30b Increase Persister Performance
The main improvement comes from using the Influxdb Line Protocol. The
encoding methods in are like the ones used in the influxdb
client but optimized for our data

Additional improvement comes from avoiding calls to encode('utf8') as
the influxdb client already does that.

On my test system, these changes increased the number of measurements
processed from about 2200/second to about 3700/second. Measurement
processing time is now dominated by Kafka. Approximately, 35% of time
is spent reading from Kafka and approximately 22% of time is committing
offsets. Only 10% of the time is spent writing to Influxdb. About 30% of
the time is spent converting messages from the json string read from
Kafka into the Line Protocol format for Influxdb.

Once monasca-common is modified to use the faster kafka library,
performance should be even better.

I did try using ujson, but my tests showed it wasn't any faster than
the json package.

Change-Id: I2acf76d9a5f583c74a272e18350b9c0ad5883f95
6 years ago
Craig Bryant 7082ded836 Turn on bandit check as part of pep8
Add bandit job as part of pep8 in tox.ini

Had to fix one issue and mark two instances of try except pass as OK so that
bandit will pass

Change-Id: Ia1c96e27d1bae360c6ae0d4131665e2b712f573f
6 years ago
Cao Xuan Hoang 6733c064d9 Clean imports in code
This patch set modifies lines which are importing objects
instead of modules. As per openstack import guide lines, user should
import modules in a file not objects.

Change-Id: I8af9278ffefcc20cf7447ecff7470217cd0e4632
7 years ago
Jenkins a3acd792da Merge "Revival of Cassandra repository" 7 years ago
Shinya Kawabata 613bab70b8 Revival of Cassandra repository
Revival and adding some modification to support schema change.

See also: I914176d60bfce91fbe449702f7e78bb2f78706ce
Change-Id: I2152057ef4843b555a21118cb20bd9eff36c3b7a
7 years ago
Paula Madalina Crismaru 57e10bbcb7 Fixes H201: no 'except:' at least use 'except Exception:'
Enabled H201 on tox and changes made in relevant files.

Change-Id: I7e1f05d9f90b11ac89b371e66ed9a0010462ad01
7 years ago
Joachim Barheine 0bc26e4ea0 fix alarm history
camel-case conversion was missing for currentValues attribute

Change-Id: I6ad53e41b7fe94c2a359204aa37c772ddfddff57
7 years ago
Michael James Hoppal 7c77ad3e44 Enable the ability to run multiple consumer processes at once
Change-Id: Ibed328d0b2bc11206d9975e84ce957305f8b973a
7 years ago
Michael James Hoppal 22e18453dd Remove cassandra repository
Change-Id: I914176d60bfce91fbe449702f7e78bb2f78706ce
7 years ago
Michael James Hoppal 5b1ffc3e2b Refactor the python persister
Separate out the persister classes into their own python files
and clean up some unused code for start up.

Change-Id: Id1460baef9e06025ed57030438a0021c253cc99a
7 years ago