DEPRECATED, Monasca Transform and Aggregation Engine
Go to file
melissaml 6e284fb710 Update links in README
Change the outdated links to the latest links in README.

Change-Id: Ic4f5cdb9f16f1a18eca75a2e2f08c7d02d9ffdfe
2018-09-07 12:57:34 -07:00
devstack Update monasca-transform devstack plugin 2018-04-17 13:05:35 -07:00
doc/source Remove vestigate HUDSON_PUBLISH_DOCS reference 2017-09-02 08:50:25 -05:00
docs Remove service_id from pre-transform spec 2018-07-17 15:39:24 -07:00
etc Set region in metric meta from config file 2018-03-27 16:47:46 -07:00
monasca_transform Remove service_id from pre-transform spec 2018-07-17 15:39:24 -07:00
scripts Enhanced refresh monasca transform script 2017-04-19 17:16:50 -07:00
tests Remove service_id from pre-transform spec 2018-07-17 15:39:24 -07:00
tools/vagrant Update refresh monasca transfom script to systemd 2017-06-01 15:02:03 -07:00
.gitignore Switch to using stestr 2018-06-29 09:45:40 +02:00
.gitreview Added .gitreview 2016-04-08 07:58:56 +00:00
.stestr.conf Switch to using stestr 2018-06-29 09:45:40 +02:00
.zuul.yaml add python 3.6 unit test job 2018-08-22 18:52:01 -04:00
LICENSE monasca-transform initial commit 2016-05-26 00:10:37 +00:00
README.rst Update links in README 2018-09-07 12:57:34 -07:00
lower-constraints.txt Remove testrepository and .testr.conf 2018-07-12 09:50:52 +07:00
requirements.txt Updated from global requirements 2018-05-04 11:48:45 -07:00
setup.cfg Convert README.md to ReStructuredTest format 2018-06-21 09:35:23 +02:00
setup.py Updated from global requirements 2017-03-17 15:59:01 -07:00
test-requirements.txt Remove testrepository and .testr.conf 2018-07-12 09:50:52 +07:00
tox.ini Fix README.rst and add syntax check 2018-08-08 15:04:35 +02:00

README.rst

Team and repository tags

image

Monasca Transform

monasca-transform is a data driven aggregation engine which collects, groups and aggregates existing individual Monasca metrics according to business requirements and publishes new transformed (derived) metrics to the Monasca Kafka queue.

  • Since the new transformed metrics are published as any other metric in Monasca, alarms can be set and triggered on the transformed metric.
  • Monasca Transform uses Apache Spark to aggregate data. Apache Spark is a highly scalable, fast, in-memory, fault tolerant and parallel data processing framework. All monasca-transform components are implemented in Python and use Sparks PySpark Python API to interact with Spark.
  • Monasca Transform does transformation and aggregation of incoming metrics in two phases.
    • In the first phase spark streaming application is set to retrieve in data from kafka at a configurable stream interval (default stream_inteval is 10 minutes) and write the data aggregated for stream interval to pre_hourly_metrics topic in kafka.
    • In the second phase, which is kicked off every hour, all metrics in metrics_pre_hourly topic in Kafka are aggregated again, this time over a larger interval of an hour. These hourly aggregated metrics published to metrics topic in kafka.

Use Cases handled by Monasca Transform

Please refer to Problem Description section on the Monasca/Transform wiki

Operation

Please refer to How Monasca Transform Operates section on the Monasca/Transform wiki

Architecture

Please refer to Architecture and Logical processing data flow sections on the Monasca/Transform wiki

To set up the development environment

The monasca-transform uses DevStack as a common dev environment. See the README.md in the devstack directory for details on how to include monasca-transform in a DevStack deployment.

Generic aggregation components

Monasca Transform uses a set of generic aggregation components which can be assembled in to an aggregation pipeline.

Please refer to the generic-aggregation-components document for information on list of generic aggregation components available.

Create a new aggregation pipeline example

Generic aggregation components make it easy to build new aggregation pipelines for different Monasca metrics.

This create a new aggregation pipeline example shows how to create pre_transform_specs and transform_specs to create an aggregation pipeline for a new set of Monasca metrics, while leveraging existing set of generic aggregation components.

Original proposal and blueprint

Original proposal: Monasca/Transform-proposal

Blueprint: monasca-transform blueprint