From f01ecbb9ad6e9897864c2be84e40849efd63f239 Mon Sep 17 00:00:00 2001 From: Ashwin Agate Date: Wed, 18 Jul 2018 17:54:40 -0700 Subject: [PATCH] Monasca Events Publishing Spec Monasca Events API was developed to store events data in Elasticsearch. There is still a need to collect and publish Openstack Notifications as Monasca Events. This commit includes a specification for creating a new Monasca Events Listener service that would be independent of Ceilometer. This is being proposed for the Stein cycle. Monasca Ceilometer project currently publishes ceilometer samples to Monasca API. An alternate specification proposes to extend Monasca Ceilometer project and add a new events publisher which would publish Openstack notifications (or events) to Monasca Events API. This solution was proposed for Rocky but will not be implemented as the Ceilometer Events feature has been deprecated on master. Co-Authored-By: Joseph Davis Co-Authored-By: Ashwin Agate Story: 2003023 Task: 23047 Change-Id: I8ed7a19c0565a0cd613866934eb2540ae46c847d --- README.rst | 9 +- doc/source/specs/rocky/index.rst | 8 + doc/source/specs/rocky/not-implemented | 1 + .../monasca-events-publishing.rst | 705 ++++++++++++++++++ .../approved/monasca-events-listener.rst | 586 +++++++++++++++ 5 files changed, 1306 insertions(+), 3 deletions(-) create mode 120000 doc/source/specs/rocky/not-implemented create mode 100644 specs/rocky/not-implemented/monasca-events-publishing.rst create mode 100644 specs/stein/approved/monasca-events-listener.rst diff --git a/README.rst b/README.rst index ee3f8d2..11bfa73 100644 --- a/README.rst +++ b/README.rst @@ -25,11 +25,14 @@ The layout of this repository is:: priorities// specs// -Where there are two sub-directories in ``specs``: +Where there are three sub-directories in ``specs``: specs//approved - specifications approved, but not yet implemented + Specifications approved, but not yet implemented specs//implemented - implemented specifications + Implemented specifications +specs//not-implemented + Specifications that were approved but are not expected to be implemented. + These are kept for historical reference. diff --git a/doc/source/specs/rocky/index.rst b/doc/source/specs/rocky/index.rst index ccb3149..3c27fee 100644 --- a/doc/source/specs/rocky/index.rst +++ b/doc/source/specs/rocky/index.rst @@ -24,3 +24,11 @@ Rocky approved (but not implemented) specs: :maxdepth: 1 approved/* + +Rocky deprecated specs: + +.. toctree:: + :glob: + :maxdepth: 1 + + not-implemented/* diff --git a/doc/source/specs/rocky/not-implemented b/doc/source/specs/rocky/not-implemented new file mode 120000 index 0000000..93d2bbc --- /dev/null +++ b/doc/source/specs/rocky/not-implemented @@ -0,0 +1 @@ +../../../../specs/rocky/not-implemented \ No newline at end of file diff --git a/specs/rocky/not-implemented/monasca-events-publishing.rst b/specs/rocky/not-implemented/monasca-events-publishing.rst new file mode 100644 index 0000000..db67291 --- /dev/null +++ b/specs/rocky/not-implemented/monasca-events-publishing.rst @@ -0,0 +1,705 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +================================================ +Monasca Events Publishing - from Ceilometer +================================================ + +https://storyboard.openstack.org/#!/story/2003023 + +Monasca Events API [1] was developed to store Openstack Notification data in Elasticsearch. There is +still a need to collect and publish Openstack Notifications to Monasca Events API. Monasca +Ceilometer project[2] currently publishes ceilometer samples[3] to Monasca API. We are proposing to +extend Monasca Ceilometer project and add a new events publisher which will publish Openstack +notifications (or events)[3] to Monasca Events API. + +UPDATE: This spec is being superceded by the ../../stein/approved/monasca-events-listener.rst spec, +but is kept here for reference. + + +Problem description +=================== + +All Openstack services generate a lot of notifications or events which contain large amounts of +operational and state information about the service and its resources. This notification data is not +currently available in Monasca. + +Ceilometer data processing pipeline[3] provides an extensible mechanism of publishing samples and +events using a custom publisher. Ceilometer samples represent a quantity that can be measured (for +e.g. the size of a volume) and events represent an occurrence of an event and do not have any +associated quantity (e.g. volume was created). + +Monasca Ceilometer project currently provides a samples publisher. Monasca Ceilometer samples +publisher converts Ceilometer samples to Monasca Metrics format which are then published to Monasca +API. There is no corresponding events publisher to Monasca yet. + +By adding an event publisher to Monasca Ceilometer project we could take advantage of Ceilometer's +event publishing mechanism to publish events to Monasca Events API. + +Ceilometer consists of different data collection components - namely Polling Agent, Notification +Agent and Compute Agent. (Please see [7] for System Architecture diagram) Ceilometer also has a data +storage and retrieval component, which would be Monasca in our case. + +Samples publisher and new proposed events publisher run within Ceilometer's Notification Agent +component and are part of Notifcation Agent's data processing pipeline. Monasca +Ceilometer presumes the need to install and deploy Ceilometer Notification Agent component (doesn't +need Polling Agent or Compute Agent deployed) on all the control nodes. Ceilometer Notification +Agent is Highly Available (HA) and can run on multiple nodes. We will have to evaluate its +performance in terms of scaling for events, but we haven't run into performance/scale problems +with current samples publisher. + + +Use Cases +--------- + +#. Openstack notification data would be stored in Elasticsearch + via the Monasca Events API + + Example sequence from Nova notification to Monasca API + #. Nova completes the creation of a VM + #. Nova generates a Notification message to oslo.messaging + #. Ceilometer Notification agent receives the Notification message + #. Ceilometer translates the Notification to a Monasca API format according to the configuration + #. Ceilometer Event Publisher publishes formatted Notification to Monasca Events API + #. Monasca Events API receives and validates formatted Notification + #. Monasca Events stores event Notification in configured Elasticsearch instance + + +Proposed change +=============== + +#. Openstack Notifications consist of envelope and payload fields + + Example Openstack Notification data format: + + .. code-block:: javascript + + { + "_context_auth_token": "42630b3ea13242fcad20e0a92d0207f1", + "_context_domain": null, + "_context_instance_lock_checked": false, + "_context_is_admin": true, + "_context_project_domain": null, + "_context_project_id": "a4f77", + "_context_project_name": "admin", + "_context_quota_class": null, + "_context_read_deleted": "no", + "_context_read_only": false, + "_context_remote_address": "192.168.245.4", + "_context_request_id": "req-5948338c-f223-4fd8-9249-8769f7a3e460", + "_context_resource_uuid": null, + "_context_roles": [ + "monasca-user", + "admin", + "KeystoneAdmin" + ], + "_context_service_catalog": [ + { + "endpoints": [ + { + "adminURL": "http://192.168.245.8:8776/v2/a4f77", + "internalURL": "http://192.168.245.8:8776/v2/a4f77", + "publicURL": "http://192.168.245.9:8776/v2/a4f77", + "region": "region1" + } + ], + "name": "cinderv2", + "type": "volumev2" + }, + { + "endpoints": [ + { + "adminURL": "http://192.168.245.8:8776/v1/a4f77", + "internalURL": "http://192.168.245.8:8776/v1/a4f77", + "publicURL": "http://192.168.245.9:8776/v1/a4f77", + "region": "region1" + } + ], + "name": "cinder", + "type": "volume" + } + ], + "_context_show_deleted": false, + "_context_tenant": "a4f77", + "_context_timestamp": "2015-09-18T20:54:23.468522", + "_context_user": "be396488c7034811a200a3cb1d103a28", + "_context_user_domain": null, + "_context_user_id": "be396488c7034811a200a3cb1d103a28", + "_context_user_identity": "be396488c7034811a200a3cb1d103a28 a4f77 - - -", + "_context_user_name": "admin", + "_unique_id": "ff9699d587bf4283a3c367ab88be1541", + "event_type": "compute.instance.create.start", + "message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d", + "payload": { + "access_ip_v4": null, + "access_ip_v6": null, + "architecture": null, + "availability_zone": null, + "cell_name": "", + "created_at": "2015-09-18 20:55:25+00:00", + "deleted_at": "", + "disk_gb": 1, + "display_name": "testeee", + "ephemeral_gb": 0, + "host": null, + "hostname": "testeee", + "image_meta": { + "base_image_ref": "df0c8", + "container_format": "bare", + "disk_format": "qcow2", + "min_disk": "1", + "min_ram": "0" + }, + "image_name": "glanceaaa3", + "image_ref_url": "http://192.168.245.5:9292/images/df0c8", + "instance_flavor_id": "1", + "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6", + "instance_type": "m1.tiny", + "instance_type_id": 4, + "kernel_id": "", + "launched_at": "", + "memory_mb": 512, + "metadata": {}, + "node": null, + "os_type": null, + "progress": "", + "ramdisk_id": "", + "reservation_id": "r-1ghilddw", + "root_gb": 1, + "state": "building", + "state_description": "", + "tenant_id": "a4f77", + "terminated_at": "", + "user_id": "be396488c7034811a200a3cb1d103a28", + "vcpus": 1 + }, + "priority": "INFO", + "publisher_id": "compute.ccp-compute0001-mgmt", + "timestamp": "2015-09-18 20:55:37.639023" + } + +#. All the fields with the prefix of '_context" are the envelope fields, the + other interesting fields are + + #. 'message_id' - notification identifier + #. 'payload' - contains most of the relevant and useful information in JSON format + #. 'priority' - notification priority + #. 'publisher_id' - notification publisher + #. 'timestamp' - notification timestamp + +#. Ceilometer event publishing framework converts the Openstack notifications to events format[4]. + Event publishing framework also has the ability to extract only some of the 'payload' data into + a flat set of key-value pairs called 'traits' and publish the normalized 'event' with 'traits' + extracted from the payload using a custom publisher. + + Extraction of certain fields into traits from the payload is + driven by configuration file, but by default "publisher_id", + 'request_id', 'tenant_id', 'user_id' and 'project_id' + fields are always extracted and added as 'traits'. + + The event can also have an optional field called 'raw' which has original + notification, provided 'store_raw' option is set in ceilometer.conf + + Questions/TODO: + + * Q1: Does the store_raw field only apply to events, or to all notifications processed by + Ceilometer? + * We will have to find it out if it has any adverse impact on sample publisher. Though in the + case of samples, monasca sample publisher definitely does not submit raw payload, so it must + be getting dropped. + + Example Ceilometer Event data format: + + .. code-block:: javascript + + { + "event_type": "compute.instance.create.start", + "message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d", + "generated": "2015-09-18 20:55:37.639023", + "traits": { + "publisher_id": "compute.ccp-compute0001-mgmt", + "request_id": "req-5948338c-f223-4fd8-9249-8769f7a3e460", + "tenant_id": "a4f77", + "project_id": "a4f77", + "user_id": "be396488c7034811a200a3cb1d103a28" + }, + "raw": { "_context_auth_token": "42630b3ea13242fcad20e0a92d0207f1", + "_context_domain": null, + ... + ... + "event_type": "compute.instance.create.start", + "message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d", + "payload": { + "access_ip_v4": null, + "access_ip_v6": null, + "architecture": null, + "availability_zone": null, + "cell_name": "", + "created_at": "2015-09-18 20:55:25+00:00", + "deleted_at": "", + "disk_gb": 1, + "display_name": "testeee", + "ephemeral_gb": 0, + "host": null, + "hostname": "testeee", + "image_meta": { + "base_image_ref": "df0c8", + "container_format": "bare", + "disk_format": "qcow2", + "min_disk": "1", + "min_ram": "0" + }, + "image_name": "glanceaaa3", + "image_ref_url": "http://192.168.245.5:9292/images/df0c8", + "instance_flavor_id": "1", + "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6", + "instance_type": "m1.tiny", + "instance_type_id": 4, + "kernel_id": "", + "launched_at": "", + "memory_mb": 512, + "metadata": {}, + "node": null, + "os_type": null, + "progress": "", + "ramdisk_id": "", + "reservation_id": "r-1ghilddw", + "root_gb": 1, + "state": "building", + "state_description": "", + "tenant_id": "a4f77", + "terminated_at": "", + "user_id": "be396488c7034811a200a3cb1d103a28", + "vcpus": 1 + } + } + } + + +#. Key-Value pairs that can be extracted from 'payload' in form of traits + can be defined in events definitions file. + + For example the following events definitions yaml specifies that for + all events which have a prefix of "compute.instance.*" then + add "user_id", "instance_id", and "instance_type_id" as traits, + after extracting values from "payload.user_id", "payload.instance_id", + and "payload.instance_type_id" respectively. + + .. code-block:: yaml + + --- + - event_type: compute.instance.* + + traits: &instance_traits + user_id: + fields: payload.user_id + instance_id: + fields: payload.instance_id + instance_type_id: + type: int + fields: payload.instance_type_id + + We are for now proposing not to use this feature, of defining traits for each event + extracting since we have the ability to store entire payload, via + Monasca Events API. + + We can certainly look at enabling this feature in the future if we run into trouble storing + entire JSON "payload" in Elasticsearch. This is certainly a nifty way to trim the amount + of data that will be stored. + +#. The proposed new Custom Monasca Ceilometer event publisher will run within Ceilometer's + Notification Agent component. It will leverage Ceilometer's data processing pipeline[3] which + converts notifications to Ceilometer's event format. At the end of its processing, Monasca + Ceilometer event publisher will convert Ceilometer Event data into Monasca Event format[6] and + publish the monasca event to Monasca Events API. + +#. Monasca Events API allows a field called 'payload' which can be in an arbitrary + nested JSON format. Monasca-Ceilometer event publisher will extract JSON field called + 'payload' from 'raw' (JSON path notation: 'raw.payload'), publish the payload from the + original notification to Monasca Events API. + + Example Monasca Event Format: + + .. code-block:: javascript + + events: [ + { + dimensions": { + "service": "compute.ccp-compute0001-mgmt", + "topic": "notification.sample", + "hostname": "nova-compute:compute + }, + event: { + + "event_type": "compute.instance.create.start", + + "payload": { + "access_ip_v4": null, + "access_ip_v6": null, + "architecture": null, + "availability_zone": null, + "cell_name": "", + "created_at": "2015-09-18 20:55:25+00:00", + "deleted_at": "", + "disk_gb": 1, + "display_name": "testeee", + "ephemeral_gb": 0, + "host": null, + "hostname": "testeee", + "image_meta": { + "base_image_ref": "df0c8", + "container_format": "bare", + "disk_format": "qcow2", + "min_disk": "1", + "min_ram": "0" + }, + "image_name": "glanceaaa3", + "image_ref_url": "http://192.168.245.5:9292/images/df0c8", + "instance_flavor_id": "1", + "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6", + "instance_type": "m1.tiny", + "instance_type_id": 4, + "kernel_id": "", + "launched_at": "", + "memory_mb": 512, + "metadata": {}, + "node": null, + "os_type": null, + "progress": "", + "ramdisk_id": "", + "reservation_id": "r-1ghilddw", + "root_gb": 1, + "state": "building", + "state_description": "", + "tenant_id": "a4f77", + "terminated_at": "", + "user_id": "be396488c7034811a200a3cb1d103a28", + "vcpus": 1 + } + }, + publisher_id: "compute.ccp-compute0001-mgmt", + priority: "INFO" + } + ] + +#. If no traits are specified in events pipeline yaml configuration file for an event + Ceilometer's data processing pipeline will add the following default traits: + + * service: (All notifications should have this) notification’s publisher + * tenant_id + * request_id + * project_id + * user_id + + Note: "service" is not the service that produced the event as in say "compute", "glance", + "cinder" but rather notification RabbitMQ publisher that produced the event + e.g. "compute.ccp-compute0001-mgmt" so is not very useful. + +#. Ceilometer event data is converted to Monasca event data format before being published to Monasca + Event API. Following fields in Monasca Event data are not available in current Ceilometer Event + data format: + + * "service" + * "dimensions.topic" + * "event.priority" + + We are proposing removing these fields from Monasca Event format (will be done as a separate + spec/implementation process) for the following reasons: + + "service": Currently Openstack notifications do not specify a service, that + generated the notification in a consistent way. It might be possible to create an external + mapping file which maps event name to a service but its hard to maintain such mapping over a + period of time. + + "dimensions.topic": This field is not available in the source Openstack notification + + "event.priority": This field is not currently available in Ceilometer Event format. It is + available in the source Openstack notification. Note: If we think this field can be useful we can + propose adding it to the Ceilometer Event format. + +#. Following new fields will be added to Monasca Event data as dimensions: + + * "dimensions.publisher_id": Identifier for the publisher that generated the event. Maps to + "traits.publisher_id" in Ceilometer event data. + * "dimensions.user_id": Identifier for user that generated the event. Maps to "traits.user_id" in + Ceilometer event data. + * "dimensions.project_id": Identifier of the project that generated the event. Maps to + "traits.project_id" or "traits.tenant_id" in Ceilometer event data. + +#. hostname is available in the event payload, but its location might differ from event to event. We + can use Ceilometer's event definitions config to always add a trait called "hostname" to all + events. e.g. for compute.instance.* will have a trait called "hostname", which grabs data from + "payload.hostname" + + .. code-block:: yaml + + --- + - event_type: compute.instance.* + + traits: &instance_traits + user_id: + fields: payload.hostname + +#. The proposed new Monasca Ceilometer event publisher will have the ability to submit event + data in a batch and at a configurable frequency (similar to current samples publisher). The + event data will be published if the items in the current batch reach their maximum size + (config setting) or if certain time interval has elapsed since the last publish + (config setting). This will make sure that the batch does not get huge at the same time + there is no significant delay in publishing of the events to Monasca Events API. + +#. Monasca Ceilometer event publisher will use service credentials from ceilometer configuration + file (in "[monasca]" section) to get keystone token. + + Example "[monasca]" section in ceilometer config file + .. code-block:: text + + [monasca] + service_auth_url = https://localhost:5000/v3 + service_password = secretpassword + service_username = ceilometer + service_interface = internalURL + service_auth_type = password + # service_project_id may also be used + service_project_name = admin + service_domain_name = Default + service_region_name = RegionOne + + The publisher will then make a POST request to Monasca Events /v1.0/events REST api[8] to publish + events to Monasca Events API. The URL for the instance of Monasca Events API will be configured + in the Ceilometer 'events-pipeline.yaml' file. This has the added advantage of allowing + different events to be published differently (see Ceilometer pipeline documentation [10]). + +#. "tenant_id" and "user_id" that the notification relates to are available in "payload" section + of the notification, and these notifications are generated by each service itself. + + There is no additional "Openstack-operator-agent" like component or functionality required to + fetch that data from the service and publish to monasca event api on behalf of the original + tenant. + Ceilometer publishing pipeline simply extracts these "tenant_id" and "user_id" fields from the + "payload" and makes those fields available as "tenant_id" and "user_id" traits, which would then + be mapped to "dimensions.project_id" and "dimensions.user_id" fields in monasca events format. + + In other words, original "tenant_id" and "user_id" values are available in + the payload of the notification, and will make its way to "dimensions.tenant_id" + and "dimensions.user_id" in Monasca Event. + + Questions/TODO: + * Q: Do we need to do anything special to handle multi-tenancy in monasca-events api like being + done for metrics[9] ? Would original user_id and tenant_id in "dimensions.user_id" and + "dimensions.tenant_id" fields in dimensions serve this purpose? + * Q: In Ceilometer V2 API (which has been deprecated and removed), when querying data the role + "admin" could access data for all tenants, whereas a user with "ceilometer-admin" role could + access only data for a particular tenant. Can we implement something like this for + monasca-events api when querying for data? + +#. Monasca Ceilometer event publisher will also retry submitting a batch, in case Monasca + Events API is temporarily unavailable or down. The retry frequency, the number of retries + and the number of items that can be in the retry batch will also be set via configuration. + + +Alternative Solutions +--------------------- + +#. Standalone monasca event agent which reads Openstack notifications published to RabbitMQ + (on "notification" topic) and publishes them to Monasca Events API. + Pro: + * No dependency on Telemetry project. + * May be simple to develop if leverage the oslo.messaging functionality. + * Ceilometer has *deprecated* the events functionality in the Stein release. [13] + Con: + * Another agent to convince users to install on their systems. + * Reinventing work already done in the Ceilometer agent. The OpenStack community already uses Ceilometer and contributes updates when something fails. + This alternate solution will be detailed in a separate spec, as it is likely + the long term solution Monasca will need. + +#. Openstack Panko [5] is a event storage and REST API for Ceilometer. + Pro: + * An 'official' subproject within Telemetry, so there is some community recognition. + Con: + * Its primary storage is in a relational database which has problems with scale. + * It is not maintained actively and not ready for production. [11] + * It will be deprecated eventually. [12] + +Data model impact +----------------- + +None + +REST API impact +--------------- + +#. We are proposing to tweak the Monasca Event data format by removing and adding following + fields as mentioned in "Proposed change" section above. + + Remove fields (JSON path notation): "service", "dimensions.topic", + "dimensions.hostname" and "event.priority" + + Add fields (JSON path notation): "dimensions.publisher_id", "dimensions.user_id" and + "dimensions.project_id" + + This change will have an impact on Monasca Events API. + +Security impact +--------------- + +The proposed Monasca Ceilometer events publisher will collect and publish +Openstack event (notification) data to Monasca API. Openstack notification +data does not have any sensitive data like 'tokens'. +Notifications do contain 'user_id' and 'project_id' fields but do not +contain any Personally Identifiable Information (PII) for the user or +the project. + + +Other end user impact +--------------------- + +None. + +Performance Impact +------------------ + +#. The number of notifications(events) generated by different services will depend on the capacity + of the cloud along with the number of resources being created by the users. + + For example, if there was a large number of compute VM's being created or destroyed it could + lead to a surge in number of notifications (events) that would have to be published. Optimum + configuration options related to say event batch size and event batch interval would have to be + documented, to reduce any adverse affect on performance. + +#. Monasca Ceilometer publisher runs within Ceilometer Notification Agent component and invoked as a + last step in its data processing pipeline. It is an additional component that will have to to be + deployed on all the controller nodes. We will have to evaluate the performance impact of + Ceilometer Notification Agent when publishing events to Monasca Events API. + + +Other deployer impact +--------------------- + +#. The proposed new Monasca-Ceilometer events publisher will introduce + few new configuration options like + * events api endpoint + * events batch interval + * events batch size + * events retry interval + +#. Monasca Ceilometer Events publisher will have to to be added to Ceilometer's + "[ceilometer.event.publisher]" section entry_points.txt + + For example: + + [ceilometer.event.publisher] + monasca = ceilometer.publisher.monclient:MonascaEventsPublisher + +#. As part of developing new Monasca Ceilometer Events publisher devstack plugin would be updated to + add the above configuration changes. + + +Developer impact +---------------- + +#. The proposed change to Monasca Event Format will have an impact on existing Monasca Event API, + since Monasca Event Format will have to be tweaked. (See REST API Impact section above) + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + joadavis, aagate + +Other contributors: + + + +Work Items +---------- + +#. Implement new Monasca Ceilometer Events publisher. + +#. Implement monasca-ceilometer devstack plugin changes to deploy + new events publisher. + +#. Implement unit tests for Events publisher. + +#. Implement change to Monasca Event format in Monasca Events API. + + +Dependencies +============ + +#. Monasca Events API 1.0: https://storyboard.openstack.org/#!/story/2001654 + +#. Monasca Ceilometer project: https://github.com/openstack/monasca-ceilometer + +#. Ceilometer Data processing and pipelines: +https://docs.openstack.org/ceilometer/pike/admin/telemetry-data-pipelines.html + +Testing +======= + +#. New Monasca Ceilometer Event publisher unit tests will be added, which can test publishing with + various config options events batch size, events batch interval, handling retry when Monasca + Event API is not available. + +#. Adding tempest tests for Monasca Ceilometer events publisher could be looked at as part of + separate effort. + + Please note that current Monasca Ceilometer samples publisher does not have tempest tests either + so having tempest tests for both events and samples publisher could be considered in the future. + +Documentation Impact +==================== + +#. New Monasca Events Publisher config options will be documented + + * events api endpoint + * events batch interval + * events batch size + * events retry interval + +#. Recommended values for each of the config options will also be documented based on the size of + the cloud and resources for Cloud Operators. + +References +========== + +[1] Monasca Events API 1.0: https://storyboard.openstack.org/#!/story/2001654 + +[2] Monasca Ceilometer project: https://github.com/openstack/monasca-ceilometer + +[3] Ceilometer Data processing and pipelines: +https://docs.openstack.org/ceilometer/pike/admin/telemetry-data-pipelines.html + +[4] Ceilometer Events: https://docs.openstack.org/ceilometer/latest/admin/telemetry-events.html + +[5] Openstack Panko: https://github.com/openstack/panko + +[6] Monasca Event Format: +https://github.com/openstack/monasca-events-api/blob/master/doc/api-samples/v1/req_simple_event.json + +[7] Ceilometer System Architecture Diagram: +https://docs.openstack.org/ceilometer/ocata/architecture.html + +[8] Monasca Events POST v1.0 API: +https://github.com/openstack/monasca-events-api/blob/master/api-ref/source/events.inc + +[9] Cross-Tenant Metric Submission: +https://github.com/openstack/monasca-agent/blob/master/docs/MonascaMetrics.md#cross-tenant-metric-submission + +[10] Ceilometer pipeline yaml documentation: +https://docs.openstack.org/ceilometer/latest/admin/telemetry-data-pipelines.html + +[11] No future for Panko or Aodh: +https://julien.danjou.info/lessons-from-openstack-telemetry-deflation/ + +[12] Ceilometer Events deprecated means Panko also deprecated: +http://eavesdrop.openstack.org/irclogs/%23openstack-telemetry/%23openstack-telemetry.2018-10-10.log.html + +[13] Ceilometer Events marked as deprecated in Stein: +https://review.openstack.org/#/c/603336/ \ No newline at end of file diff --git a/specs/stein/approved/monasca-events-listener.rst b/specs/stein/approved/monasca-events-listener.rst new file mode 100644 index 0000000..c01d5b1 --- /dev/null +++ b/specs/stein/approved/monasca-events-listener.rst @@ -0,0 +1,586 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +======================= +Monasca Events Listener +======================= + +https://storyboard.openstack.org/#!/story/2003023 + +Monasca Events API [1]_ was developed to store event data in Elasticsearch. +A new application could use the Monasca Events API to post an event directly for processing +and storage. +However, a general collection service is needed to capture the existing OpenStack Notifications +already generated by OpenStack Services and pass them to Monasca for storage and processing. +This specification proposes creating a new Monasca Events Listener to capture events and pass +them to Monasca services. + + +Problem description +=================== + +All Openstack services generate a lot of notifications or events which contain large amounts of +operational and state information about the service and its resources. Services such as Nova [14]_ +already publish to a RabbitMQ topic. +This notification data is not currently available in Monasca. + +Ceilometer data processing pipeline [3]_ provides an extensible mechanism of publishing samples and +events using a custom publisher. Ceilometer samples represent a quantity that can be measured +(e.g. the size of a volume) and events represent an occurrence of an event and do not have any +associated quantity (e.g. volume was created). The Telemetry project also has the Panko [5]_ +service for indexing and storing these events. + +A previous `spec <../../rocky/not-implemented/monasca-events-publishing.html>`_ was created to +specify an enhancement to Ceilometer to allow collected events to be published to the Monasca +Events API. +However, with the recent deprecation of Ceilometer's Event functionality [13]_, this is no longer +a long-term option. + +This spec is for creating a new Monasca service which would listen for OpenStack Event messages +(often called notifications) and process them through Monasca by consuming the RabbitMQ message +and producing a post to the Monasca Events API with that event. + +This service could use Ceilometer, Vitrage, Watcher, or another service as an example of how to +listen to notifications from OpenStack services such as Nova [14]_. + +It is being proposed to make the Monasca Events Listener service a part of the Monasca Agent +code base and reuse code, including the existing monasca_setup script and config. + + +Use Cases +--------- + +#. Openstack notification data would be stored in Elasticsearch via the Monasca services + + Example sequence: + + #. Nova completes the creation of a VM + #. Nova generates a Notification message to oslo.messaging + #. oslo.messaging posts the message to RabbitMQ + #. Monasca Event Listener receives the Notification message from RabbitMQ + #. Monasca Event Listener validates and translates the Notification to a Monasca Events + API format according to the configuration + #. Monasca Event Listener publishes formatted Notification to Monasca Events API + #. Monasca Events API receives and validates formatted Notification + #. Monasca Events API stores event Notification in configured Elasticsearch instance + + +Proposed change +=============== + +#. Monasca Event Listener will be a new service which can be run in a Highly + Available configuration by running an instance of Monasca Event Listener + on each Controller node in a cloud. Each node will listen to OpenStack + notifications in RabbitMQ and convert notifications to a post to the Monasca + Events API. The Monasca Events API then passes the notification on as Kafka + messages on the 'monevents' topic, where the Monasca Persister will receive + and store them. By the nature of RabbitMQ clients, the load will be + distributed between the Monasca Events Listener instances (only one listener + will process each RabbitMQ message). + +#. Monasca Event Listener will filter messages from OpenStack services based on + specification of event_types to be collected. This will reduce 'noise' and + focus event collection on those events that are deemed valuable. + This filtering specification can be from a configuration file, or optionally + could be controlled through a new API implemented as part of the Monasca + Events API service. + +#. OpenStack Notifications consist of envelope and payload fields. See [15]_ and [16]_ for + examples. + + Example OpenStack Notification data format: + + .. code-block:: javascript + + { + "_context_auth_token": "42630b3ea13242fcad20e0a92d0207f1", + "_context_domain": null, + "_context_instance_lock_checked": false, + "_context_is_admin": true, + "_context_project_domain": null, + "_context_project_id": "a4f77", + "_context_project_name": "admin", + "_context_quota_class": null, + "_context_read_deleted": "no", + "_context_read_only": false, + "_context_remote_address": "192.168.245.4", + "_context_request_id": "req-5948338c-f223-4fd8-9249-8769f7a3e460", + "_context_resource_uuid": null, + "_context_roles": [ + "monasca-user", + "admin", + "KeystoneAdmin" + ], + "_context_service_catalog": [ + { + "endpoints": [ + { + "adminURL": "http://192.168.245.8:8776/v2/a4f77", + "internalURL": "http://192.168.245.8:8776/v2/a4f77", + "publicURL": "http://192.168.245.9:8776/v2/a4f77", + "region": "region1" + } + ], + "name": "cinderv2", + "type": "volumev2" + }, + { + "endpoints": [ + { + "adminURL": "http://192.168.245.8:8776/v1/a4f77", + "internalURL": "http://192.168.245.8:8776/v1/a4f77", + "publicURL": "http://192.168.245.9:8776/v1/a4f77", + "region": "region1" + } + ], + "name": "cinder", + "type": "volume" + } + ], + "_context_show_deleted": false, + "_context_tenant": "a4f77", + "_context_timestamp": "2015-09-18T20:54:23.468522", + "_context_user": "be396488c7034811a200a3cb1d103a28", + "_context_user_domain": null, + "_context_user_id": "be396488c7034811a200a3cb1d103a28", + "_context_user_identity": "be396488c7034811a200a3cb1d103a28 a4f77 - - -", + "_context_user_name": "admin", + "_unique_id": "ff9699d587bf4283a3c367ab88be1541", + "event_type": "compute.instance.create.start", + "message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d", + "payload": { + "access_ip_v4": null, + "access_ip_v6": null, + "architecture": null, + "availability_zone": null, + "cell_name": "", + "created_at": "2015-09-18 20:55:25+00:00", + "deleted_at": "", + "disk_gb": 1, + "display_name": "testeee", + "ephemeral_gb": 0, + "host": null, + "hostname": "testeee", + "image_meta": { + "base_image_ref": "df0c8", + "container_format": "bare", + "disk_format": "qcow2", + "min_disk": "1", + "min_ram": "0" + }, + "image_name": "glanceaaa3", + "image_ref_url": "http://192.168.245.5:9292/images/df0c8", + "instance_flavor_id": "1", + "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6", + "instance_type": "m1.tiny", + "instance_type_id": 4, + "kernel_id": "", + "launched_at": "", + "memory_mb": 512, + "metadata": {}, + "node": null, + "os_type": null, + "progress": "", + "ramdisk_id": "", + "reservation_id": "r-1ghilddw", + "root_gb": 1, + "state": "building", + "state_description": "", + "tenant_id": "a4f77", + "terminated_at": "", + "user_id": "be396488c7034811a200a3cb1d103a28", + "vcpus": 1 + }, + "priority": "INFO", + "publisher_id": "compute.ccp-compute0001-mgmt", + "timestamp": "2015-09-18 20:55:37.639023" + } + +#. All the fields with the prefix of '_context' are the envelope fields, the + other interesting fields are + + #. 'message_id' - notification identifier + #. 'payload' - contains most of the relevant and useful information in JSON format + #. 'priority' - notification priority + #. 'publisher_id' - notification publisher + #. 'timestamp' - notification timestamp + +#. Monasca Event Listener converts the OpenStack notifications to Monasca events format. + This format will be suitable for Kafka messaging and will match the expected data fields + of the Monasca Persister. This conversion and validation should be common between the + Monasca Event Listener and Monasca Event API. + +#. The Kafka client connection will handle communication issues such as reconnections and + resending as needed. + +#. Monasca Events API allows a field called 'payload' which can be in an arbitrary + nested JSON format. + TODO: mappings + + Example Monasca Event Format: + + .. code-block:: javascript + + events: [ + { + dimensions": { + "service": "compute.ccp-compute0001-mgmt", + "topic": "notification.sample", + "hostname": "nova-compute:compute + }, + event: { + + "event_type": "compute.instance.create.start", + + "payload": { + "access_ip_v4": null, + "access_ip_v6": null, + "architecture": null, + "availability_zone": null, + "cell_name": "", + "created_at": "2015-09-18 20:55:25+00:00", + "deleted_at": "", + "disk_gb": 1, + "display_name": "testeee", + "ephemeral_gb": 0, + "host": null, + "hostname": "testeee", + "image_meta": { + "base_image_ref": "df0c8", + "container_format": "bare", + "disk_format": "qcow2", + "min_disk": "1", + "min_ram": "0" + }, + "image_name": "glanceaaa3", + "image_ref_url": "http://192.168.245.5:9292/images/df0c8", + "instance_flavor_id": "1", + "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6", + "instance_type": "m1.tiny", + "instance_type_id": 4, + "kernel_id": "", + "launched_at": "", + "memory_mb": 512, + "metadata": {}, + "node": null, + "os_type": null, + "progress": "", + "ramdisk_id": "", + "reservation_id": "r-1ghilddw", + "root_gb": 1, + "state": "building", + "state_description": "", + "tenant_id": "a4f77", + "terminated_at": "", + "user_id": "be396488c7034811a200a3cb1d103a28", + "vcpus": 1 + } + }, + publisher_id: "compute.ccp-compute0001-mgmt", + priority: "INFO" + } + ] + +#. Following fields in Monasca Event data may not be available in the OpenStack notification + data format: + + * "service" + * "dimensions.topic" + * "event.priority" + + We are proposing removing these fields from Monasca Event format (will be done as a separate + spec/implementation process) for the following reasons: + + "service": Currently OpenStack notifications do not specify a service, that + generated the notification in a consistent way. It might be possible to create an external + mapping file which maps event name to a service but its hard to maintain such mapping over a + period of time. + + "dimensions.topic": This field is not available in the source OpenStack notification. + However, the Monasca Event Listener may be able to save the RabbitMQ topic that the notification + was collected from. In that case, this field should be used. + + "event.priority": This field is not currently available in Ceilometer Event format. It is + available in the source OpenStack notification. Note: If we think this field can be useful we can + propose adding it to the Monasca Event Listener format. + +#. Following new fields will be added to Monasca Event data as dimensions: + + * "dimensions.publisher_id": Identifier for the publisher that generated the event. + * "dimensions.user_id": Identifier for user that generated the event. + * "dimensions.project_id": Identifier of the project that generated the event. + +#. 'hostname' is available in the event payload, but its location might differ from event to event. + +#. The proposed new Monasca Event Listener will have the ability to submit event + data in a batch and at a configurable frequency (similar to current samples publisher). The + event data will be published if the items in the current batch reach their maximum size + (config setting) or if certain time interval has elapsed since the last publish + (config setting). This will make sure that the batch does not get huge at the same time + there is no significant delay in publishing of the events to Monasca Events API. + +#. Monasca Event Listener will have a configuration file to configure connection information for + both RabbitMQ and Monasca Events API. + +#. The "tenant_id" and "user_id" that the notification relates to are available in "payload" + section of the notification, and these notifications are generated by each service itself. + + There is no additional "OpenStack-operator-agent" like component or functionality required to + fetch that data from the service and publish to monasca event api on behalf of the original + tenant. + (Ceilometer publishing pipeline simply extracts these "tenant_id" and "user_id" fields from the + "payload" and makes those fields available as "tenant_id" and "user_id" traits, which would then + be mapped to "dimensions.project_id" and "dimensions.user_id" fields in monasca events format.) + + In other words, original "tenant_id" and "user_id" values are available in + the payload of the notification, and will make its way to "dimensions.tenant_id" + and "dimensions.user_id" in Monasca Event. + + Questions/TODO: + + * Q: Do we need to do anything special to handle multi-tenancy in monasca-events api like being + done for metrics [9]_ ? Would original user_id and tenant_id in "dimensions.user_id" and + "dimensions.tenant_id" fields in dimensions serve this purpose? + + * A: Monasca Events Listener can start with sending all events to a single "admin" tenant + and if required in the future some other process could copy select metrics back to tenant + projects. + + * Q: In Ceilometer V2 API (which has been deprecated and removed), when querying data the role + "admin" could access data for all tenants, whereas a user with "ceilometer-admin" role could + access only data for a particular tenant. Can we implement something like this for + monasca-events api when querying for data? + + * A: In Monasca API every request is scoped to the project, so there is no equivalent of + Ceilometer's "admin" role to query data for all projects. So placing all events in to + an "admin" project may be the best approach. + + * Q: How should services which generate notifications but do not include a tenant_id be + handled? For example Keystone [16]_. + How does Ceilometer handle such events? + + * A: If all events are in an "admin" project then admin metrics like shared ceph cluster + load or provider network load can be copied back to tenants so they may understand how + infrastructure is affecting their workload. + + * Note: Configuration of Elasticsearch cluster is out of scope for this spec. If needed + could assign a separate Elasticsearch cluster to the events API to avoid overloads. + + +Alternative Solutions +--------------------- + +#. Reuse the Ceilometer functionality to collect and publish events to the + Monasca Events API. While this may be less work initially, Ceilometer has + deprecated the Events functionality as of Stein. [13]_ + +#. An alternate Events Listener was proposed that would listen for RabbitMQ events + then publish them directly to the 'monevents' topic in Kafka. Discussion on this + can be seen in the git history for this document and in IRC logs [18]_. + + Pro: + + * A much simpler approach, more efficient that HTTP hop through another service. + * No need for batching in service, as RabbitMQ and Kafka clients would handle fast + throughput and short network interruptions. + + Con: + + * Nova Cells v2 each have their own RabbitMQ instances. While most deployments + would likely have a centralized RabbitMQ, it is not required in the documented + architecture. + * Regions may also cause separation of RabbitMQ instances that need to be monitored. + * While it might be possible to have a service/agent in each Cell publish back to + a centralized to Kafka directly, our authentication and networking for Kafka was + not designed to support that. + +#. OpenStack Panko [5]_ is a event storage and REST API for Ceilometer and could + be used instead of a Monasca solution. + + Pro: + + * An 'official' subproject within Telemetry, so there is some community recognition. + + Con: + + * Its primary storage is in a relational database which has problems with scale. + * It is not maintained actively and not ready for production. [11]_ + * It will be deprecated eventually. [12]_ + + +Data model impact +----------------- + +None + +REST API impact +--------------- + +#. We are proposing to tweak the Monasca Event data format by removing and adding following + fields as mentioned in "Proposed change" section above. + + Remove fields or make them optional (JSON path notation): "service", "dimensions.topic", + "dimensions.hostname" and "event.priority" + + Add fields (JSON path notation): "dimensions.publisher_id", "dimensions.user_id" and + "dimensions.project_id" + + This change will have an impact on Monasca Events API. + +Security impact +--------------- + +The proposed Monasca Event Listener will collect and publish OpenStack event +(notification) data to Monasca Events API. OpenStack notification data does not +have any sensitive data like 'tokens'. +Notifications do contain 'user_id' and 'project_id' fields but do not +contain any Personally Identifiable Information (PII) for the user or +the project. + + +Other end user impact +--------------------- + +None. + +Performance Impact +------------------ + +#. The number of notifications(events) generated by different services will depend on the capacity + of the cloud along with the number of resources being created by the users. + + For example, if there was a large number of compute VM's being created or destroyed it could + lead to a surge in number of notifications (events) that would have to be published. Optimum + configuration options related to say event batch size and event batch interval would have to be + documented, to reduce any adverse affect on performance. + +#. The proposed Monasca Event Listener is a new service, so performance is unknown. However, the + Monasca API has been shown to have a high performance throughput. + + +Other deployer impact +--------------------- + +#. The proposed Monasca Event Listener will introduce a + few new configuration options like + + * RabbitMQ connection information + * Monasca Events API endpoint URL + * events batch interval + * events batch size + * events retry interval + * Keystone credentials for obtaining a token + * Conversion options for OpenStack notifications to the Monasca Event format. This may + be stored in separate pipeline configuration files, similar to how transform specs + are configured in Monasca Transform. + +#. As part of developing new Monasca Event Listener devstack plugin would be + updated to add the above configuration changes. + + +Developer impact +---------------- + +#. The proposed change to Monasca Event Format will have an impact on existing Monasca Event API, + since Monasca Event Format will have to be tweaked. (See REST API Impact section above) + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + joadavis, aagate + +Other contributors: + + + +Work Items +---------- + +#. Implement new Monasca Event Listener. + + * Connection to RabbitMQ for OpenStack Notifications + * Add filtering of notifications for configured event_types + + * Specification in configuration file + * (Optional) Creation of a new API to configure event_type subscriptions + + * Validation of OpenStack Notification data and format + * Conversion of data format to meet Monasca Events requirements + * Publishing to Monasca Events API + * Configuration of conversion specifications per-event type + +#. Implement monasca devstack plugin changes to deploy + new Events Listener service. + +#. Implement unit tests for Monasca Event Listener. + + +Dependencies +============ + +None + +Testing +======= + +#. New Monasca Event Listener unit tests will be added, which can test publishing with + various config options events batch size, events batch interval, handling retry when Monasca + Event API is not available. + +#. Adding tempest tests for Monasca Event Listener could be looked at as part of + separate effort. + + +Documentation Impact +==================== + +#. New Monasca Event Listener config options will be documented + +#. Recommended values for each of the config options will also be documented based on the size of + the cloud and resources for Cloud Operators. + +References +========== + +.. [1] Monasca Events API 1.0: https://git.openstack.org/cgit/openstack/monasca-events-api/ + +[2] Monasca Ceilometer project: https://github.com/openstack/monasca-ceilometer + +.. [3] Ceilometer Data processing and pipelines: https://docs.openstack.org/ceilometer/pike/admin/telemetry-data-pipelines.html + +[4] Ceilometer Events: https://docs.openstack.org/ceilometer/latest/admin/telemetry-events.html + +.. [5] Openstack Panko: https://github.com/openstack/panko + +[6] Monasca Event Format: https://github.com/openstack/monasca-events-api/blob/master/doc/api-samples/v1/req_simple_event.json + +[7] Ceilometer System Architecture Diagram: https://docs.openstack.org/ceilometer/ocata/architecture.html + +[8] Monasca Events POST v1.0 API: https://github.com/openstack/monasca-events-api/blob/master/api-ref/source/events.inc + +.. [9] Cross-Tenant Metric Submission: https://github.com/openstack/monasca-agent/blob/master/docs/MonascaMetrics.md#cross-tenant-metric-submission + +[10] Ceilometer pipeline yaml documentation: https://docs.openstack.org/ceilometer/latest/admin/telemetry-data-pipelines.html + +.. [11] No future for Panko or Aodh: https://julien.danjou.info/lessons-from-openstack-telemetry-deflation/ + +.. [12] Ceilometer Events deprecated means Panko also deprecated: http://eavesdrop.openstack.org/irclogs/%23openstack-telemetry/%23openstack-telemetry.2018-10-10.log.html + +.. [13] Ceilometer Events marked as deprecated in Stein: https://review.openstack.org/#/c/603336/ + +.. [14] Nova notification version update lists services effected (see "Deprecating legacy notifications"): https://etherpad.openstack.org/p/nova-ptg-stein + +.. [15] Nova notification reference: https://docs.openstack.org/nova/latest/reference/notifications.html#existing-versioned-notifications + +.. [16] Keystone notification reference: https://docs.openstack.org/keystone/latest/advanced-topics/event_notifications.html#example-notification-project-create + +[17] Monasca Events API publisher to Kafka: https://github.com/openstack/monasca-events-api/blob/master/monasca_events_api/app/common/events_publisher.py + +.. [18] Monasca IRC meeting Dec 15, 2018: http://eavesdrop.openstack.org/meetings/monasca/2018/monasca.2018-12-12-15.00.log.html \ No newline at end of file