A data extraction and transformation library for OpenStack notifications.
Go to file
OpenDev Sysadmins fc637b2f52 OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:50:38 +00:00
bin Add PEP8 check and fix related issues 2015-05-05 10:20:59 -05:00
doc Add config file format docs 2015-04-15 22:10:46 +00:00
etc Inital commit. 2014-05-27 22:57:46 +00:00
stackdistiller Add PEP8 check and fix related issues 2015-05-05 10:20:59 -05:00
tests Add PEP8 check and fix related issues 2015-05-05 10:20:59 -05:00
.gitignore Add PEP8 check and fix related issues 2015-05-05 10:20:59 -05:00
.gitreview OpenDev Migration Patch 2019-04-19 19:50:38 +00:00
LICENSE Inital commit. 2014-05-27 22:57:46 +00:00
MANIFEST.in Inital commit. 2014-05-27 22:57:46 +00:00
README.md Add config file format docs 2015-04-15 22:10:46 +00:00
requirements.txt Add PEP8 check and fix related issues 2015-05-05 10:20:59 -05:00
setup.cfg Bump dev status 2015-08-17 22:46:04 +00:00
setup.py Add PEP8 check and fix related issues 2015-05-05 10:20:59 -05:00
test-requirements.txt Inital commit. 2014-05-27 22:57:46 +00:00
tox.ini Add PEP8 check and fix related issues 2015-05-05 10:20:59 -05:00

stackdistiller

A data extraction and transformation library for OpenStack notifications.

Stackdistiller is designed to extract data from openstack notifications and convert it into a form relevant to the application consuming the notification. It consists of two components, the Distiller, which extracts data from notifications according to a YAML config file, and the Condenser, which receives the data extracted by the Distiller, and formats it into an application-specific object, referred to as an Event. This could be a simple python dictionary, an XML document tree, or a set of ORM model objects.

Distiller

The Distiller reads a YAML config file to determine what data to extract from each notification, according to it's event type. event types can be wildcarded using shell glob syntax. The distiller will extract two types of data from each notification:

  • Metadata from the notifications envelope, including the event type, message id (uuid of the notification) and the timestamp showing when notification was generated by the source system.
  • A series of data items extracted from the notification's body. These are called Traits. Traits are basically just typed name-value pairs.

The distiller can also do some basic data massaging on traits extracted from the notification, such as splitting a value from a string. This is handled by trait plugins. These are just classes that implement the TraitPluginBase interface. They are referred to by name in the config, and are looked up in a trait_plugin_map passed to the distiller on init. The plugin map is just a dictionary, or dictionary-like object (such as a plugin manager) that maps names to plugin classes. If no map is passed to the distiller, it will use a default that just contains the builtin plugins bundled with stackdistiller.

If a notification does not match any event definition in the distiller's config file, the distiller's to_event method will return None, indicating it cannot extract that notification. This may be what you want (i.e. your application may only be interested in certain notifications.), but if you want to record basic informaton from any event type, you can pass "catchall=True" to the distiller, and it will generate a minimal event from any notification.

Condenser

The Condenser receives the data extracted from the notification by the Distiller and formats it into an appropriate type of Event object. An instance of a Condenser class is passed, along with the raw, deserialized notification, to the distiller object's to_event method. To create your own type of Event from the data extracted by the distiller, you just need to create a Condenser class to receive the data. Condenser classes don't have to subclass any particular class, as long as they implement the methods defined in stackdistiller.condenser.CondenserBase. If you do not pass a condenser to the distiller when you call to_event, it will create an instance of the default DictionaryCondenser for you. This just formats the event as a plain python dictionary.

Example:

import json
from stackdistiller import distiller
from stackdistiller import condenser

config_file_name = "events_i_want.yaml"
notification_string = open('a_notification_here.json', 'r').read()

notification = json.loads(notification_string)
config = distiller.load_config(config_file_name)

dist = distiller.Distiller(config, catchall=False)

#this is the default condenser.
cond = condenser.DictionaryCondenser()

if dist.to_event(notification, cond):
    # What get_event() returns is up to the condenser class. In this
    # case, it's a dictionary.
    event = cond.get_event()
    print "Yay! An Event: %s" % str(event)
else:
    print "Not something we are interested in. Ignoring."