openstack/monasca-notification
Code Issues Proposed changes
Notification Engine for Monasca
187 Commits 3 Branches 94 Tags 4.5 MiB
Python 91.4%
Jinja 5.3%
Shell 2.8%
Dockerfile 0.5%
6f75509f9c
Go to file
Nam Nguyen Hoai 6f75509f9c Replace yaml.load() with yaml.safe_load() 
Avoid dangerous file parsing and object serialization libraries.
yaml.load is the obvious function to use but it is dangerous[1]
Because yaml.load return Python object may be dangerous if you
receive a YAML document from an untrusted source such as the Internet.
The function yaml.safe_load limits this ability to simple Python
objects like integers or lists.

In addition, Bandit flags yaml.load() as security risk so replace
all occurrences with yaml.safe_load(). Thus I replace yaml.load()
with yaml.safe_load()

[1] https://security.openstack.org/guidelines/dg_avoid-dangerous-input-parsing-libraries.html

Change-Id: I3f1a756e80b617ba1bcf824ef4dee9a27eb6887a
Closes-Bug: #1634265
2017-01-18 09:31:10 +07:00
monasca_notification Replace yaml.load() with yaml.safe_load() 2017-01-18 09:31:10 +07:00
tests Add config option for statsd 2016-12-02 07:10:52 +01:00
tools Remove monasca_notification_offsets 2017-01-13 09:35:56 +01:00
.gitignore Fixed mysql reconnect on error 2015-08-21 14:37:53 -06:00
.gitreview Update .gitreview for new namespace 2015-10-17 22:30:54 +00:00
HACKING.rst Rename to monasca, setup for tox, removed legacy bits 2014-07-16 15:59:00 -06:00
LICENSE Added copyright header, LICENSE and HACKING.rst. 2014-05-01 12:27:06 -06:00
notification.yaml Add config option for statsd 2016-12-02 07:10:52 +01:00
README.md Add config option for statsd 2016-12-02 07:10:52 +01:00
requirements.txt Updated from global requirements 2016-12-12 03:40:57 +00:00
setup.cfg Remove monasca_notification_offsets 2017-01-13 09:35:56 +01:00
setup.py Updated from global requirements 2016-09-03 01:56:37 +00:00
test-requirements.txt Updated from global requirements 2016-09-03 01:56:37 +00:00
tox.ini Use constraints everywhere 2016-08-30 20:15:57 +02:00

README.md

Team and repository tags

Team and repository tags

Notification Engine

This engine reads alarms from Kafka and then notifies the customer using their configured notification method. Multiple notification and retry engines can run in parallel up to one per available Kafka partition. Zookeeper is used to negotiate access to the Kafka partitions whenever a new process joins or leaves the working set.

Architecture

The notification engine generates notifications using the following steps:

  1. Reads Alarms from Kafka, with no auto commit. - KafkaConsumer class
  2. Determine notification type for an alarm. Done by reading from mysql. - AlarmProcessor class
  3. Send Notification. - NotificationProcessor class
  4. Successful notifications are added to a sent notification topic. - NotificationEngine class
  5. Failed notifications are added to a retry topic. - NotificationEngine class
  6. Commit offset to Kafka - KafkaConsumer class

The notification engine uses three Kafka topics:

  1. alarm_topic: Alarms inbound to the notification engine.
  2. notification_topic: Successfully sent notifications.
  3. notification_retry_topic: Unsuccessful notifications.

A retry engine runs in parallel with the notification engine and gives any failed notification a configurable number of extra chances at succeess.

The retry engine generates notifications using the following steps:

  1. Reads Notification json data from Kafka, with no auto commit. - KafkaConsumer class
  2. Rebuild the notification that failed. - RetryEngine class
  3. Send Notification. - NotificationProcessor class
  4. Successful notifictions are added to a sent notification topic. - RetryEngine class
  5. Failed notifications that have not hit the retry limit are added back to the retry topic. - RetryEngine class
  6. Failed notifications that have hit the retry limit are discarded. - RetryEngine class
  7. Commit offset to Kafka - KafkaConsumer class

The retry engine uses two Kafka topics:

  1. notification_retry_topic: Notifications that need to be retried.
  2. notification_topic: Successfully sent notifications.

Fault Tolerance

When reading from the alarm topic no committing is done. The committing is done only after processing. This allows the processing to continue even though some notifications can be slow. In the event of a catastrophic failure some notifications could be sent but the alarms not yet acknowledged. This is an acceptable failure mode, better to send a notification twice than not at all.

The general process when a major error is encountered is to exit the daemon which should allow the other processes to renegotiate access to the Kafka partitions. It is also assumed the notification engine will be run by a process supervisor which will restart it in case of a failure. This way any errors which are not easy to recover from are automatically handled by the service restarting and the active daemon switching to another instance.

Though this should cover all errors there is risk that an alarm or set of alarms can be processed and notifications sent out multiple times. To minimize this risk a number of techniques are used:

  • Timeouts are implemented with all notification types.
  • An alarm TTL is utilized. Any alarm older than the TTL is not processed.

Operation

Yaml config file by default is in '/etc/monasca/notification.yaml', a sample is in this project.

Monitoring

statsd is incorporated into the daemon and will send all stats to statsd server launched by monasca-agent. Default host and port points at localhost:8125.

  • Counters
    • ConsumedFromKafka
    • AlarmsFailedParse
    • AlarmsNoNotification
    • NotificationsCreated
    • NotificationsSentSMTP
    • NotificationsSentWebhook
    • NotificationsSentPagerduty
    • NotificationsSentFailed
    • NotificationsInvalidType
    • AlarmsFinished
    • PublishedToKafka
  • Timers
    • ConfigDBTime
    • SendNotificationTime

Future Considerations

  • More extensive load testing is needed
    • How fast is the mysql db? How much load do we put on it. Initially I think it makes most sense to read notification details for each alarm but eventually I may want to cache that info.
    • How expensive are commits to Kafka for every message we read? Should we commit every N messages?
    • How efficient is the default Kafka consumer batch size?
    • Currently we can get ~200 notifications per second per NotificationEngine instance using webhooks to a local http server. Is that fast enough?
    • Are we putting too much load on Kafka at ~200 commits per second?

License

Copyright (c) 2014 Hewlett-Packard Development Company, L.P.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.