DEPRECATED, Monasca Transform and Aggregation Engine
Go to file
huang.zhiping ccd56bf0d4 fix tox python3 overrides
We want to default to running all tox environments under python 3, so
set the basepython value in each environment.

We do not want to specify a minor version number, because we do not
want to have to update the file every time we upgrade python.

We do not want to set the override once in testenv, because that
breaks the more specific versions used in default environments like
py35 and py36.

Change-Id: Iaa7c9b9fca48dd2e091049da711e70395f6ec28e
2018-10-01 09:26:32 +00:00
devstack Fix PEP8 tests for Python 3 2018-10-01 11:21:23 +02:00
doc/source Remove vestigate HUDSON_PUBLISH_DOCS reference 2017-09-02 08:50:25 -05:00
docs Remove service_id from pre-transform spec 2018-07-17 15:39:24 -07:00
etc Set region in metric meta from config file 2018-03-27 16:47:46 -07:00
monasca_transform Fix PEP8 tests for Python 3 2018-10-01 11:21:23 +02:00
scripts Enhanced refresh monasca transform script 2017-04-19 17:16:50 -07:00
tests Merge "Enable mutable config in monasca_transform" 2018-09-11 23:20:35 +00:00
tools/vagrant Update refresh monasca transfom script to systemd 2017-06-01 15:02:03 -07:00
.gitignore Switch to using stestr 2018-06-29 09:45:40 +02:00
.gitreview Added .gitreview 2016-04-08 07:58:56 +00:00
.stestr.conf Switch to using stestr 2018-06-29 09:45:40 +02:00
.zuul.yaml add python 3.6 unit test job 2018-08-22 18:52:01 -04:00
LICENSE monasca-transform initial commit 2016-05-26 00:10:37 +00:00
README.rst Update links in README 2018-09-07 12:57:34 -07:00
lower-constraints.txt Remove pykafka from lower-constraints 2018-07-24 16:40:21 +02:00
requirements.txt Updated from global requirements 2018-05-04 11:48:45 -07:00
setup.cfg Convert README.md to ReStructuredTest format 2018-06-21 09:35:23 +02:00
setup.py Updated from global requirements 2017-03-17 15:59:01 -07:00
test-requirements.txt Remove testrepository and .testr.conf 2018-07-12 09:50:52 +07:00
tox.ini fix tox python3 overrides 2018-10-01 09:26:32 +00:00

README.rst

Team and repository tags

image

Monasca Transform

monasca-transform is a data driven aggregation engine which collects, groups and aggregates existing individual Monasca metrics according to business requirements and publishes new transformed (derived) metrics to the Monasca Kafka queue.

  • Since the new transformed metrics are published as any other metric in Monasca, alarms can be set and triggered on the transformed metric.
  • Monasca Transform uses Apache Spark to aggregate data. Apache Spark is a highly scalable, fast, in-memory, fault tolerant and parallel data processing framework. All monasca-transform components are implemented in Python and use Sparks PySpark Python API to interact with Spark.
  • Monasca Transform does transformation and aggregation of incoming metrics in two phases.
    • In the first phase spark streaming application is set to retrieve in data from kafka at a configurable stream interval (default stream_inteval is 10 minutes) and write the data aggregated for stream interval to pre_hourly_metrics topic in kafka.
    • In the second phase, which is kicked off every hour, all metrics in metrics_pre_hourly topic in Kafka are aggregated again, this time over a larger interval of an hour. These hourly aggregated metrics published to metrics topic in kafka.

Use Cases handled by Monasca Transform

Please refer to Problem Description section on the Monasca/Transform wiki

Operation

Please refer to How Monasca Transform Operates section on the Monasca/Transform wiki

Architecture

Please refer to Architecture and Logical processing data flow sections on the Monasca/Transform wiki

To set up the development environment

The monasca-transform uses DevStack as a common dev environment. See the README.md in the devstack directory for details on how to include monasca-transform in a DevStack deployment.

Generic aggregation components

Monasca Transform uses a set of generic aggregation components which can be assembled in to an aggregation pipeline.

Please refer to the generic-aggregation-components document for information on list of generic aggregation components available.

Create a new aggregation pipeline example

Generic aggregation components make it easy to build new aggregation pipelines for different Monasca metrics.

This create a new aggregation pipeline example shows how to create pre_transform_specs and transform_specs to create an aggregation pipeline for a new set of Monasca metrics, while leveraging existing set of generic aggregation components.

Original proposal and blueprint

Original proposal: Monasca/Transform-proposal

Blueprint: monasca-transform blueprint