Monasca Database Persister

Go to file

Craig Bryant dbab337d76 Commit kafka reads once the item was persisted In order to continue using the Kafka High Consumer API, the disruptor was removed. This allows a direct call to to kafka to commit the offsets when items are flushed. Different ConsumerConnectors had to be created for Metrics and Alarms so the offsets could be committed separately Changed configuration to match the new model. Remove configuration parameters that were no longer needed Changed the name Disruptor to Pipeline Allow only one EventHandler per pipeline Added code to flush the Metrics and Alarms, and shutdown the Kafka ConsumerConnections on a normal shutdown. This keeps the persister from losing Metrics and Alarms. Made measurementTimeStampSimpleDateFormat not static since SimpleDateFormat is not thread safe Changed some logging debug statements so Strings weren't created if debug not on Created FlushableHandler as a base class and moved duplicate code into it from MetricHandler and AlarmStateTransitionHistoryHandler Change-Id: Id31a1d148f8e796f5be483dd02544be49c009b18 Changed MetricHandler to take MetricEnvelope[] Change-Id: Ifabbe253cc0163f150ada2252a41a5d9fb9ab423		2014-07-31 22:15:24 -06:00
src	Commit kafka reads once the item was persisted	2014-07-31 22:15:24 -06:00
.gitignore	Commit kafka reads once the item was persisted	2014-07-31 22:15:24 -06:00
.gitreview	Get initial build to work.	2014-07-15 15:29:35 -06:00
LICENSE	Added copyright header and LICENSE file.	2014-05-01 12:45:08 -06:00
README.md	Change the build to use the mon-common from the local repository. Updated README.md with build instructions.	2014-07-08 13:01:36 -06:00
pom.xml	Commit kafka reads once the item was persisted	2014-07-31 22:15:24 -06:00

README.md

mon-persister

The Monitoring Persister consumes metrics and alarm state transitions from the Message Queue and stores them in the Metrics and Alarms database.

Although the Persister isn't primarly a Web service it uses DropWizard, https://dropwizard.github.io/dropwizard/, which provides a nice Web appliction framework to expose an http endpoint that provides an interface through which metrics about the Persister can be queried as well as health status.

The basic design of the Persister is to have one Kafka consumer publish to a Disruptor, https://github.com/LMAX-Exchange/disruptor, that has output processors. The output processors use prepared batch statements to write to the Metrics and Alarms database.

The number of output processors/threads in the Persister can be specified to scale to more messages. To horizontally scale and provide fault-tolerance any number of Persisters can be started as consumers from the the Message Queue.

Build

Requires mon-common from https://github.com/hpcloud-mon/mon-common. Download and build following instructions in its README.md. Then build mon-persister by:

mvn clean package

TODO

Purge metrics on shutdown
Add more robust offset management in Kafka. Currently, the offset is advanced as each message is read. If the Persister stops after the metric has been read and prior to it being committed to the Metrics and Alarms database, the metric will be lost.
Add better handling of SQL exceptions.
Complete health check.
Specify and document the names of the metrics that are available for monitoring of the Persister.
Document the yaml configuration parameters.

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.