DEPRECATED, Monasca Transform and Aggregation Engine
Go to file
Charles Short 5e2687f637 Switch to using stestr
According to Openstack summit session [1] stestr is
maintained project to which all Openstack projects
should migrate.
Let's switch it then.

[1] https://etherpad.openstack.org/p/YVR-python-pti

Change-Id: Ic0ddaae3bd0a4ad4e34264429f717a292e17a1c9
Signed-off-by: Charles Short <zulcss@gmail.com>
2018-06-29 09:45:40 +02:00
devstack Update monasca-transform devstack plugin 2018-04-17 13:05:35 -07:00
doc/source Remove vestigate HUDSON_PUBLISH_DOCS reference 2017-09-02 08:50:25 -05:00
docs Merge "Cleanup pre transform and transform specs" 2018-06-11 19:03:55 +00:00
etc Set region in metric meta from config file 2018-03-27 16:47:46 -07:00
monasca_transform Cleanup pre transform and transform specs 2018-05-29 16:35:33 -07:00
scripts Enhanced refresh monasca transform script 2017-04-19 17:16:50 -07:00
tests Cleanup pre transform and transform specs 2018-05-29 16:35:33 -07:00
tools/vagrant Update refresh monasca transfom script to systemd 2017-06-01 15:02:03 -07:00
.gitignore Switch to using stestr 2018-06-29 09:45:40 +02:00
.gitreview Added .gitreview 2016-04-08 07:58:56 +00:00
.stestr.conf Switch to using stestr 2018-06-29 09:45:40 +02:00
.testr.conf Corrected test environment 2016-11-22 10:24:07 +00:00
.zuul.yaml add lower-constraints job 2018-03-22 15:50:40 -04:00
LICENSE monasca-transform initial commit 2016-05-26 00:10:37 +00:00
README.md Refresh monasca transform docs 2018-04-16 08:22:17 -07:00
lower-constraints.txt Updated from global requirements 2018-05-04 11:48:45 -07:00
requirements.txt Updated from global requirements 2018-05-04 11:48:45 -07:00
setup.cfg Changes to setup cfg 2017-05-25 13:53:57 -07:00
setup.py Updated from global requirements 2017-03-17 15:59:01 -07:00
test-requirements.txt Switch to using stestr 2018-06-29 09:45:40 +02:00
tox.ini Switch to using stestr 2018-06-29 09:45:40 +02:00

README.md

Team and repository tags

Team and repository tags

Monasca Transform

monasca-transform is a data driven aggregation engine which collects, groups and aggregates existing individual Monasca metrics according to business requirements and publishes new transformed (derived) metrics to the Monasca Kafka queue.

  • Since the new transformed metrics are published as any other metric in Monasca, alarms can be set and triggered on the transformed metric.

  • Monasca Transform uses Apache Spark to aggregate data. Apache Spark is a highly scalable, fast, in-memory, fault tolerant and parallel data processing framework. All monasca-transform components are implemented in Python and use Spark's PySpark Python API to interact with Spark.

  • Monasca Transform does transformation and aggregation of incoming metrics in two phases.

    • In the first phase spark streaming application is set to retrieve in data from kafka at a configurable stream interval (default stream_inteval is 10 minutes) and write the data aggregated for stream interval to pre_hourly_metrics topic in kafka.

    • In the second phase, which is kicked off every hour, all metrics in metrics_pre_hourly topic in Kafka are aggregated again, this time over a larger interval of an hour. These hourly aggregated metrics published to metrics topic in kafka.

Use Cases handled by Monasca Transform

Please refer to Problem Description section on the Monasca/Transform wiki

Operation

Please refer to How Monasca Transform Operates section on the Monasca/Transform wiki

Architecture

Please refer to Architecture and Logical processing data flow sections on the Monasca/Transform wiki

To set up the development environment

The monasca-transform uses DevStack as a common dev environment. See the README.md in the devstack directory for details on how to include monasca-transform in a DevStack deployment.

Generic aggregation components

Monasca Transform uses a set of generic aggregation components which can be assembled in to an aggregation pipeline.

Please refer to generic aggregation components document for information on list of generic aggregation components available.

Create a new aggregation pipeline example

Generic aggregation components make it easy to build new aggregation pipelines for different Monasca metrics.

This create a new aggregation pipeline example shows how to create pre_transform_specs and transform_specs to create an aggregation pipeline for a new set of Monasca metrics, while leveraging existing set of generic aggregation components.

Original proposal and blueprint

Original proposal: Monasca/Transform-proposal

Blueprint: monasca-transform blueprint