3feaf7400e
* Refreshed monasca transform README.md * Added new documentation on generic aggregation components * Added new documentation on creating a new aggregation pipeline * Added new documentation for internal data formats used by monasca transform namely record store data format and instance usage data format. Change-Id: Id4cfb422ace2c59103c658e58e4ffb0d61a303f5 Story: 2001765 Task: 12167 |
||
|---|---|---|
| devstack | ||
| doc/source | ||
| docs | ||
| etc | ||
| monasca_transform | ||
| scripts | ||
| tests | ||
| tools/vagrant | ||
| .gitignore | ||
| .gitreview | ||
| .testr.conf | ||
| .zuul.yaml | ||
| LICENSE | ||
| README.md | ||
| lower-constraints.txt | ||
| requirements.txt | ||
| setup.cfg | ||
| setup.py | ||
| test-requirements.txt | ||
| tox.ini | ||
README.md
Team and repository tags
Monasca Transform
monasca-transform is a data driven aggregation engine which collects, groups and aggregates existing individual Monasca metrics according to business requirements and publishes new transformed (derived) metrics to the Monasca Kafka queue.
-
Since the new transformed metrics are published as any other metric in Monasca, alarms can be set and triggered on the transformed metric.
-
Monasca Transform uses Apache Spark to aggregate data. Apache Spark is a highly scalable, fast, in-memory, fault tolerant and parallel data processing framework. All monasca-transform components are implemented in Python and use Spark's PySpark Python API to interact with Spark.
-
Monasca Transform does transformation and aggregation of incoming metrics in two phases.
-
In the first phase spark streaming application is set to retrieve in data from kafka at a configurable stream interval (default stream_inteval is 10 minutes) and write the data aggregated for stream interval to pre_hourly_metrics topic in kafka.
-
In the second phase, which is kicked off every hour, all metrics in metrics_pre_hourly topic in Kafka are aggregated again, this time over a larger interval of an hour. These hourly aggregated metrics published to metrics topic in kafka.
-
Use Cases handled by Monasca Transform
Please refer to Problem Description section on the Monasca/Transform wiki
Operation
Please refer to How Monasca Transform Operates section on the Monasca/Transform wiki
Architecture
Please refer to Architecture and Logical processing data flow sections on the Monasca/Transform wiki
To set up the development environment
The monasca-transform uses DevStack as a common dev environment. See the README.md in the devstack directory for details on how to include monasca-transform in a DevStack deployment.
Generic aggregation components
Monasca Transform uses a set of generic aggregation components which can be assembled in to an aggregation pipeline.
Please refer to generic aggregation components document for information on list of generic aggregation components available.
Create a new aggregation pipeline example
Generic aggregation components make it easy to build new aggregation pipelines for different Monasca metrics.
This create a new aggregation pipeline example shows how to create pre_transform_specs and transform_specs to create an aggregation pipeline for a new set of Monasca metrics, while leveraging existing set of generic aggregation components.
Original proposal and blueprint
Original proposal: Monasca/Transform-proposal
Blueprint: monasca-transform blueprint