We change the default value of kafka_common.legacy_kafka_client_enabled
from True to False. The use of new Confluent Kafka client is
recommended.
Change-Id: I300958e1e945e5236b01f4e50cb01c18468cc524
Story: 2007924
Task: 40340
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.
Fix problems found.
Update local hacking checks for new flake8.
Remove hacking and friends from lower-constraints, it's not needed
there.
Change-Id: Iadf750a1e6d3181b645ceccdf75cf910bc23adfd
The change updates the imports to use simplejson library and fixes three
unit tests. In particular TestUtils.test_parse_alarm_state_hist_message
has been slightly changed to make testing more robust.
Also, Python 2 specific implemetation has been removed from
influxdb.line_utils .
Additionally, standard library unittest.mock is used instead of the
third party mock lib.
Depends-On: https://review.opendev.org/718014
Depends-On: https://review.opendev.org/720188
Change-Id: I64b1a60e8be929c25c005a0f429b1274cb8570e6
Story: 2007549
Task: 39390
The new configuration option is used to control the maximum size of
batches written to InfluxDB and optimizes performance for high loads.
The default value is set to 10 000 data points per batch.
Change-Id: I82c3e4e64984a996e41fe2657ce36b032b40d3fd
Story: 2007191
Task: 38319
At the moment, all data is retained infinitely. This change allows users
to specify a default_retention_hours (defaults to 0 meaning unlimited
retention) in the influxdb section of monasca persister configuration so
that new projects are automatically assigned this retention policy when
the given hour is greater than 0 to preserve original behaviour.
Story: 2006331
Task: 37234
Change-Id: I4136df1d43954eb026a104f3f85b3a58197f5435
At the moment, monasca persister assumes that an InfluxDB database
already exists. With this change, a new database is created when it does
not exist.
Story: 2006331
Task: 37327
Change-Id: I3d88eb8b2cdbd81f9c84fe1703b2ee2d1b09e1ab
data_points_class can be now declared as a List or a Dictionary. The
latter is used when writing measurements per tenant. This refactoring
avoids creating the dictionary and then chaining again its values in
case of ElasticSearch and Cassandra databases or if db_per_tenant option
is disabled. The change also fixes the events pipeline.
Change-Id: I25ea80eea714acb167e70f188ed229cc90531596
Story: 2006331
Task: 37211
At present, all time series are accumulated in the same database in
InfluxDB. Separate database per tenant would make queries faster for
tenants. It would also allow administrators to define retention policy
per tenancy.
This changeset adds a migration tool and a guide to allow users to
migrate existing data to a database per tenant.
Task: 36435
Story: 2006331
Change-Id: I3faf3d96b1121c8b8c383dee2fda27d34af5c853
At present, all time series are accumulated in the same database in
InfluxDB. This makes queries slow for tenants that have less data. This
patch enables the option to use separate database per tenancy.
This task implements the changes on monasca-persister which handles
write requests to InfluxDB database.
Change-Id: I7de6e0faf069b889d3953583b29a876c3d82c62c
Story: 2006331
Task: 36072
InfluxdbClientError has content attribute no message attribute.
Message attribute deprecated in Python 2.6 and it's go away
in Python 3.0
Change-Id: I5fe4bcb018a7c8c99df29d9798aa0e2736f33b6f
Story: 2005900
Task: 33762
The change introduces the possibility to run the persister with
the new confluent-kafka client. It has to be enabled in the
configuration file.
Story: 2003705
Task: 30117
Depends-On: https://review.opendev.org/675297
Change-Id: I05428b8ae9e0ba9af5b81d3b103434ebd5657108
metrics table in cassandra always contain row which has no metric_id.
it is casused by cassandra ttl.cassandra supports TTL column only,
so when update a row,metric_id must update as well.
Change-Id: Ia2d2e1585965dd31f9679e2a63497bf4cc68349e
Story: 2005832
Task: 33590
It was found in some installations that python-cachetools needed
to be present on the system. This can happen if the installation
was configured for the Java mon-persister but the recreate-metric-id
tool is written in Python. Added a bullet item to describe this
dependency and the quick workaround of installing the
python-cachetools package.
Change-Id: I25635c069a5ed21342780a28b516c74b61dae30b
Story: 2005305
Task: 30611
It was discovered that the example paths included in the README.rst
used monasca-persister rather than monasca_persister.
Also added a footnote with two error messages that occur when the
contact_points is not specified correctly.
Story: 2005305
Task: 30611
Change-Id: I0c53af817504e638bd82401ca0b62874cb591f36
In some rare cases it is possible for a row in Cassandra for
metrics to have no value for metric_id or created_at, though
they may still have updated_at and the other required columns.
This tool is for recreating the metric_id from the other required
columns.
An additional 'persister-check-missing-metric-id.py' tool is
provided which can be run to see if there are missing metric-id
values that need to be recreated.
Please see the README.rst for usage directions.
Story: 2005305
Task: 30611
Change-Id: I0593558407c8c773d728bbd035dde91310b59be3
(cherry picked from commit 09af9bff91)
This change supports catching more parse errors which can jam
the persistor, such as:
`partial write: unable to parse 'string_with_invalid_char.%'`.
Change-Id: Ia12d04708a88308e88fcbd488d72f4313ba42a36
Story: 2005430
Task: 30468
persister get contact_points from configuration file
with "[ip.dest for ip in conf.cassandra.contact_points]".
cfg.dest replace "-" with "_".if user config cassandra
hosts cassandra-0,then persister crashed.
Task: 29957
Story: 2005195
Change-Id: I9ee29fdce781739f06683e4b4a80183669663189
some metrics get from kafka,may not a standard point that
influxdb can parse.in this case,points should be dropped.
Task: 29641
Story: 2005069
Change-Id: Ib907a280ced26ef2c228efe14b57440f894d8aad
The new default configuration file name has changed to
monasca-persister.conf. This change updates documentation to use the new
default.
Change-Id: Ic271221f19cdc86e8297ed88cdcf4a86df4374b8
Story: 2004867
Task: 29534
persister.conf is the deprecated path for the configuration file.
The new path is /etc/monasca/monasca-persister.conf . This is inline
with other OpenStack projects.
Also fix the log setup and set the product name to 'monasca-persister'
which is inline with other OpenStack and Monasca projects.
Change-Id: Ida11c326b3e6771b5ccf994205c55874db05bc5c
Story: 2004867
Task: 29112
Hervé Beraud