Merge "Monasca Events Publishing Spec"
This commit is contained in:
commit
abb3854784
|
@ -25,11 +25,14 @@ The layout of this repository is::
|
|||
priorities/<release>/
|
||||
specs/<release>/
|
||||
|
||||
Where there are two sub-directories in ``specs``:
|
||||
Where there are three sub-directories in ``specs``:
|
||||
|
||||
specs/<release>/approved
|
||||
specifications approved, but not yet implemented
|
||||
Specifications approved, but not yet implemented
|
||||
|
||||
specs/<release>/implemented
|
||||
implemented specifications
|
||||
Implemented specifications
|
||||
|
||||
specs/<release>/not-implemented
|
||||
Specifications that were approved but are not expected to be implemented.
|
||||
These are kept for historical reference.
|
||||
|
|
|
@ -24,3 +24,11 @@ Rocky approved (but not implemented) specs:
|
|||
:maxdepth: 1
|
||||
|
||||
approved/*
|
||||
|
||||
Rocky deprecated specs:
|
||||
|
||||
.. toctree::
|
||||
:glob:
|
||||
:maxdepth: 1
|
||||
|
||||
not-implemented/*
|
||||
|
|
|
@ -0,0 +1 @@
|
|||
../../../../specs/rocky/not-implemented
|
|
@ -0,0 +1,705 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
================================================
|
||||
Monasca Events Publishing - from Ceilometer
|
||||
================================================
|
||||
|
||||
https://storyboard.openstack.org/#!/story/2003023
|
||||
|
||||
Monasca Events API [1] was developed to store Openstack Notification data in Elasticsearch. There is
|
||||
still a need to collect and publish Openstack Notifications to Monasca Events API. Monasca
|
||||
Ceilometer project[2] currently publishes ceilometer samples[3] to Monasca API. We are proposing to
|
||||
extend Monasca Ceilometer project and add a new events publisher which will publish Openstack
|
||||
notifications (or events)[3] to Monasca Events API.
|
||||
|
||||
UPDATE: This spec is being superceded by the ../../stein/approved/monasca-events-listener.rst spec,
|
||||
but is kept here for reference.
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
All Openstack services generate a lot of notifications or events which contain large amounts of
|
||||
operational and state information about the service and its resources. This notification data is not
|
||||
currently available in Monasca.
|
||||
|
||||
Ceilometer data processing pipeline[3] provides an extensible mechanism of publishing samples and
|
||||
events using a custom publisher. Ceilometer samples represent a quantity that can be measured (for
|
||||
e.g. the size of a volume) and events represent an occurrence of an event and do not have any
|
||||
associated quantity (e.g. volume was created).
|
||||
|
||||
Monasca Ceilometer project currently provides a samples publisher. Monasca Ceilometer samples
|
||||
publisher converts Ceilometer samples to Monasca Metrics format which are then published to Monasca
|
||||
API. There is no corresponding events publisher to Monasca yet.
|
||||
|
||||
By adding an event publisher to Monasca Ceilometer project we could take advantage of Ceilometer's
|
||||
event publishing mechanism to publish events to Monasca Events API.
|
||||
|
||||
Ceilometer consists of different data collection components - namely Polling Agent, Notification
|
||||
Agent and Compute Agent. (Please see [7] for System Architecture diagram) Ceilometer also has a data
|
||||
storage and retrieval component, which would be Monasca in our case.
|
||||
|
||||
Samples publisher and new proposed events publisher run within Ceilometer's Notification Agent
|
||||
component and are part of Notifcation Agent's data processing pipeline. Monasca
|
||||
Ceilometer presumes the need to install and deploy Ceilometer Notification Agent component (doesn't
|
||||
need Polling Agent or Compute Agent deployed) on all the control nodes. Ceilometer Notification
|
||||
Agent is Highly Available (HA) and can run on multiple nodes. We will have to evaluate its
|
||||
performance in terms of scaling for events, but we haven't run into performance/scale problems
|
||||
with current samples publisher.
|
||||
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
|
||||
#. Openstack notification data would be stored in Elasticsearch
|
||||
via the Monasca Events API
|
||||
|
||||
Example sequence from Nova notification to Monasca API
|
||||
#. Nova completes the creation of a VM
|
||||
#. Nova generates a Notification message to oslo.messaging
|
||||
#. Ceilometer Notification agent receives the Notification message
|
||||
#. Ceilometer translates the Notification to a Monasca API format according to the configuration
|
||||
#. Ceilometer Event Publisher publishes formatted Notification to Monasca Events API
|
||||
#. Monasca Events API receives and validates formatted Notification
|
||||
#. Monasca Events stores event Notification in configured Elasticsearch instance
|
||||
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
#. Openstack Notifications consist of envelope and payload fields
|
||||
|
||||
Example Openstack Notification data format:
|
||||
|
||||
.. code-block:: javascript
|
||||
|
||||
{
|
||||
"_context_auth_token": "42630b3ea13242fcad20e0a92d0207f1",
|
||||
"_context_domain": null,
|
||||
"_context_instance_lock_checked": false,
|
||||
"_context_is_admin": true,
|
||||
"_context_project_domain": null,
|
||||
"_context_project_id": "a4f77",
|
||||
"_context_project_name": "admin",
|
||||
"_context_quota_class": null,
|
||||
"_context_read_deleted": "no",
|
||||
"_context_read_only": false,
|
||||
"_context_remote_address": "192.168.245.4",
|
||||
"_context_request_id": "req-5948338c-f223-4fd8-9249-8769f7a3e460",
|
||||
"_context_resource_uuid": null,
|
||||
"_context_roles": [
|
||||
"monasca-user",
|
||||
"admin",
|
||||
"KeystoneAdmin"
|
||||
],
|
||||
"_context_service_catalog": [
|
||||
{
|
||||
"endpoints": [
|
||||
{
|
||||
"adminURL": "http://192.168.245.8:8776/v2/a4f77",
|
||||
"internalURL": "http://192.168.245.8:8776/v2/a4f77",
|
||||
"publicURL": "http://192.168.245.9:8776/v2/a4f77",
|
||||
"region": "region1"
|
||||
}
|
||||
],
|
||||
"name": "cinderv2",
|
||||
"type": "volumev2"
|
||||
},
|
||||
{
|
||||
"endpoints": [
|
||||
{
|
||||
"adminURL": "http://192.168.245.8:8776/v1/a4f77",
|
||||
"internalURL": "http://192.168.245.8:8776/v1/a4f77",
|
||||
"publicURL": "http://192.168.245.9:8776/v1/a4f77",
|
||||
"region": "region1"
|
||||
}
|
||||
],
|
||||
"name": "cinder",
|
||||
"type": "volume"
|
||||
}
|
||||
],
|
||||
"_context_show_deleted": false,
|
||||
"_context_tenant": "a4f77",
|
||||
"_context_timestamp": "2015-09-18T20:54:23.468522",
|
||||
"_context_user": "be396488c7034811a200a3cb1d103a28",
|
||||
"_context_user_domain": null,
|
||||
"_context_user_id": "be396488c7034811a200a3cb1d103a28",
|
||||
"_context_user_identity": "be396488c7034811a200a3cb1d103a28 a4f77 - - -",
|
||||
"_context_user_name": "admin",
|
||||
"_unique_id": "ff9699d587bf4283a3c367ab88be1541",
|
||||
"event_type": "compute.instance.create.start",
|
||||
"message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d",
|
||||
"payload": {
|
||||
"access_ip_v4": null,
|
||||
"access_ip_v6": null,
|
||||
"architecture": null,
|
||||
"availability_zone": null,
|
||||
"cell_name": "",
|
||||
"created_at": "2015-09-18 20:55:25+00:00",
|
||||
"deleted_at": "",
|
||||
"disk_gb": 1,
|
||||
"display_name": "testeee",
|
||||
"ephemeral_gb": 0,
|
||||
"host": null,
|
||||
"hostname": "testeee",
|
||||
"image_meta": {
|
||||
"base_image_ref": "df0c8",
|
||||
"container_format": "bare",
|
||||
"disk_format": "qcow2",
|
||||
"min_disk": "1",
|
||||
"min_ram": "0"
|
||||
},
|
||||
"image_name": "glanceaaa3",
|
||||
"image_ref_url": "http://192.168.245.5:9292/images/df0c8",
|
||||
"instance_flavor_id": "1",
|
||||
"instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6",
|
||||
"instance_type": "m1.tiny",
|
||||
"instance_type_id": 4,
|
||||
"kernel_id": "",
|
||||
"launched_at": "",
|
||||
"memory_mb": 512,
|
||||
"metadata": {},
|
||||
"node": null,
|
||||
"os_type": null,
|
||||
"progress": "",
|
||||
"ramdisk_id": "",
|
||||
"reservation_id": "r-1ghilddw",
|
||||
"root_gb": 1,
|
||||
"state": "building",
|
||||
"state_description": "",
|
||||
"tenant_id": "a4f77",
|
||||
"terminated_at": "",
|
||||
"user_id": "be396488c7034811a200a3cb1d103a28",
|
||||
"vcpus": 1
|
||||
},
|
||||
"priority": "INFO",
|
||||
"publisher_id": "compute.ccp-compute0001-mgmt",
|
||||
"timestamp": "2015-09-18 20:55:37.639023"
|
||||
}
|
||||
|
||||
#. All the fields with the prefix of '_context" are the envelope fields, the
|
||||
other interesting fields are
|
||||
|
||||
#. 'message_id' - notification identifier
|
||||
#. 'payload' - contains most of the relevant and useful information in JSON format
|
||||
#. 'priority' - notification priority
|
||||
#. 'publisher_id' - notification publisher
|
||||
#. 'timestamp' - notification timestamp
|
||||
|
||||
#. Ceilometer event publishing framework converts the Openstack notifications to events format[4].
|
||||
Event publishing framework also has the ability to extract only some of the 'payload' data into
|
||||
a flat set of key-value pairs called 'traits' and publish the normalized 'event' with 'traits'
|
||||
extracted from the payload using a custom publisher.
|
||||
|
||||
Extraction of certain fields into traits from the payload is
|
||||
driven by configuration file, but by default "publisher_id",
|
||||
'request_id', 'tenant_id', 'user_id' and 'project_id'
|
||||
fields are always extracted and added as 'traits'.
|
||||
|
||||
The event can also have an optional field called 'raw' which has original
|
||||
notification, provided 'store_raw' option is set in ceilometer.conf
|
||||
|
||||
Questions/TODO:
|
||||
|
||||
* Q1: Does the store_raw field only apply to events, or to all notifications processed by
|
||||
Ceilometer?
|
||||
* We will have to find it out if it has any adverse impact on sample publisher. Though in the
|
||||
case of samples, monasca sample publisher definitely does not submit raw payload, so it must
|
||||
be getting dropped.
|
||||
|
||||
Example Ceilometer Event data format:
|
||||
|
||||
.. code-block:: javascript
|
||||
|
||||
{
|
||||
"event_type": "compute.instance.create.start",
|
||||
"message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d",
|
||||
"generated": "2015-09-18 20:55:37.639023",
|
||||
"traits": {
|
||||
"publisher_id": "compute.ccp-compute0001-mgmt",
|
||||
"request_id": "req-5948338c-f223-4fd8-9249-8769f7a3e460",
|
||||
"tenant_id": "a4f77",
|
||||
"project_id": "a4f77",
|
||||
"user_id": "be396488c7034811a200a3cb1d103a28"
|
||||
},
|
||||
"raw": { "_context_auth_token": "42630b3ea13242fcad20e0a92d0207f1",
|
||||
"_context_domain": null,
|
||||
...
|
||||
...
|
||||
"event_type": "compute.instance.create.start",
|
||||
"message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d",
|
||||
"payload": {
|
||||
"access_ip_v4": null,
|
||||
"access_ip_v6": null,
|
||||
"architecture": null,
|
||||
"availability_zone": null,
|
||||
"cell_name": "",
|
||||
"created_at": "2015-09-18 20:55:25+00:00",
|
||||
"deleted_at": "",
|
||||
"disk_gb": 1,
|
||||
"display_name": "testeee",
|
||||
"ephemeral_gb": 0,
|
||||
"host": null,
|
||||
"hostname": "testeee",
|
||||
"image_meta": {
|
||||
"base_image_ref": "df0c8",
|
||||
"container_format": "bare",
|
||||
"disk_format": "qcow2",
|
||||
"min_disk": "1",
|
||||
"min_ram": "0"
|
||||
},
|
||||
"image_name": "glanceaaa3",
|
||||
"image_ref_url": "http://192.168.245.5:9292/images/df0c8",
|
||||
"instance_flavor_id": "1",
|
||||
"instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6",
|
||||
"instance_type": "m1.tiny",
|
||||
"instance_type_id": 4,
|
||||
"kernel_id": "",
|
||||
"launched_at": "",
|
||||
"memory_mb": 512,
|
||||
"metadata": {},
|
||||
"node": null,
|
||||
"os_type": null,
|
||||
"progress": "",
|
||||
"ramdisk_id": "",
|
||||
"reservation_id": "r-1ghilddw",
|
||||
"root_gb": 1,
|
||||
"state": "building",
|
||||
"state_description": "",
|
||||
"tenant_id": "a4f77",
|
||||
"terminated_at": "",
|
||||
"user_id": "be396488c7034811a200a3cb1d103a28",
|
||||
"vcpus": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
#. Key-Value pairs that can be extracted from 'payload' in form of traits
|
||||
can be defined in events definitions file.
|
||||
|
||||
For example the following events definitions yaml specifies that for
|
||||
all events which have a prefix of "compute.instance.*" then
|
||||
add "user_id", "instance_id", and "instance_type_id" as traits,
|
||||
after extracting values from "payload.user_id", "payload.instance_id",
|
||||
and "payload.instance_type_id" respectively.
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
---
|
||||
- event_type: compute.instance.*
|
||||
|
||||
traits: &instance_traits
|
||||
user_id:
|
||||
fields: payload.user_id
|
||||
instance_id:
|
||||
fields: payload.instance_id
|
||||
instance_type_id:
|
||||
type: int
|
||||
fields: payload.instance_type_id
|
||||
|
||||
We are for now proposing not to use this feature, of defining traits for each event
|
||||
extracting since we have the ability to store entire payload, via
|
||||
Monasca Events API.
|
||||
|
||||
We can certainly look at enabling this feature in the future if we run into trouble storing
|
||||
entire JSON "payload" in Elasticsearch. This is certainly a nifty way to trim the amount
|
||||
of data that will be stored.
|
||||
|
||||
#. The proposed new Custom Monasca Ceilometer event publisher will run within Ceilometer's
|
||||
Notification Agent component. It will leverage Ceilometer's data processing pipeline[3] which
|
||||
converts notifications to Ceilometer's event format. At the end of its processing, Monasca
|
||||
Ceilometer event publisher will convert Ceilometer Event data into Monasca Event format[6] and
|
||||
publish the monasca event to Monasca Events API.
|
||||
|
||||
#. Monasca Events API allows a field called 'payload' which can be in an arbitrary
|
||||
nested JSON format. Monasca-Ceilometer event publisher will extract JSON field called
|
||||
'payload' from 'raw' (JSON path notation: 'raw.payload'), publish the payload from the
|
||||
original notification to Monasca Events API.
|
||||
|
||||
Example Monasca Event Format:
|
||||
|
||||
.. code-block:: javascript
|
||||
|
||||
events: [
|
||||
{
|
||||
dimensions": {
|
||||
"service": "compute.ccp-compute0001-mgmt",
|
||||
"topic": "notification.sample",
|
||||
"hostname": "nova-compute:compute
|
||||
},
|
||||
event: {
|
||||
|
||||
"event_type": "compute.instance.create.start",
|
||||
|
||||
"payload": {
|
||||
"access_ip_v4": null,
|
||||
"access_ip_v6": null,
|
||||
"architecture": null,
|
||||
"availability_zone": null,
|
||||
"cell_name": "",
|
||||
"created_at": "2015-09-18 20:55:25+00:00",
|
||||
"deleted_at": "",
|
||||
"disk_gb": 1,
|
||||
"display_name": "testeee",
|
||||
"ephemeral_gb": 0,
|
||||
"host": null,
|
||||
"hostname": "testeee",
|
||||
"image_meta": {
|
||||
"base_image_ref": "df0c8",
|
||||
"container_format": "bare",
|
||||
"disk_format": "qcow2",
|
||||
"min_disk": "1",
|
||||
"min_ram": "0"
|
||||
},
|
||||
"image_name": "glanceaaa3",
|
||||
"image_ref_url": "http://192.168.245.5:9292/images/df0c8",
|
||||
"instance_flavor_id": "1",
|
||||
"instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6",
|
||||
"instance_type": "m1.tiny",
|
||||
"instance_type_id": 4,
|
||||
"kernel_id": "",
|
||||
"launched_at": "",
|
||||
"memory_mb": 512,
|
||||
"metadata": {},
|
||||
"node": null,
|
||||
"os_type": null,
|
||||
"progress": "",
|
||||
"ramdisk_id": "",
|
||||
"reservation_id": "r-1ghilddw",
|
||||
"root_gb": 1,
|
||||
"state": "building",
|
||||
"state_description": "",
|
||||
"tenant_id": "a4f77",
|
||||
"terminated_at": "",
|
||||
"user_id": "be396488c7034811a200a3cb1d103a28",
|
||||
"vcpus": 1
|
||||
}
|
||||
},
|
||||
publisher_id: "compute.ccp-compute0001-mgmt",
|
||||
priority: "INFO"
|
||||
}
|
||||
]
|
||||
|
||||
#. If no traits are specified in events pipeline yaml configuration file for an event
|
||||
Ceilometer's data processing pipeline will add the following default traits:
|
||||
|
||||
* service: (All notifications should have this) notification’s publisher
|
||||
* tenant_id
|
||||
* request_id
|
||||
* project_id
|
||||
* user_id
|
||||
|
||||
Note: "service" is not the service that produced the event as in say "compute", "glance",
|
||||
"cinder" but rather notification RabbitMQ publisher that produced the event
|
||||
e.g. "compute.ccp-compute0001-mgmt" so is not very useful.
|
||||
|
||||
#. Ceilometer event data is converted to Monasca event data format before being published to Monasca
|
||||
Event API. Following fields in Monasca Event data are not available in current Ceilometer Event
|
||||
data format:
|
||||
|
||||
* "service"
|
||||
* "dimensions.topic"
|
||||
* "event.priority"
|
||||
|
||||
We are proposing removing these fields from Monasca Event format (will be done as a separate
|
||||
spec/implementation process) for the following reasons:
|
||||
|
||||
"service": Currently Openstack notifications do not specify a service, that
|
||||
generated the notification in a consistent way. It might be possible to create an external
|
||||
mapping file which maps event name to a service but its hard to maintain such mapping over a
|
||||
period of time.
|
||||
|
||||
"dimensions.topic": This field is not available in the source Openstack notification
|
||||
|
||||
"event.priority": This field is not currently available in Ceilometer Event format. It is
|
||||
available in the source Openstack notification. Note: If we think this field can be useful we can
|
||||
propose adding it to the Ceilometer Event format.
|
||||
|
||||
#. Following new fields will be added to Monasca Event data as dimensions:
|
||||
|
||||
* "dimensions.publisher_id": Identifier for the publisher that generated the event. Maps to
|
||||
"traits.publisher_id" in Ceilometer event data.
|
||||
* "dimensions.user_id": Identifier for user that generated the event. Maps to "traits.user_id" in
|
||||
Ceilometer event data.
|
||||
* "dimensions.project_id": Identifier of the project that generated the event. Maps to
|
||||
"traits.project_id" or "traits.tenant_id" in Ceilometer event data.
|
||||
|
||||
#. hostname is available in the event payload, but its location might differ from event to event. We
|
||||
can use Ceilometer's event definitions config to always add a trait called "hostname" to all
|
||||
events. e.g. for compute.instance.* will have a trait called "hostname", which grabs data from
|
||||
"payload.hostname"
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
---
|
||||
- event_type: compute.instance.*
|
||||
|
||||
traits: &instance_traits
|
||||
user_id:
|
||||
fields: payload.hostname
|
||||
|
||||
#. The proposed new Monasca Ceilometer event publisher will have the ability to submit event
|
||||
data in a batch and at a configurable frequency (similar to current samples publisher). The
|
||||
event data will be published if the items in the current batch reach their maximum size
|
||||
(config setting) or if certain time interval has elapsed since the last publish
|
||||
(config setting). This will make sure that the batch does not get huge at the same time
|
||||
there is no significant delay in publishing of the events to Monasca Events API.
|
||||
|
||||
#. Monasca Ceilometer event publisher will use service credentials from ceilometer configuration
|
||||
file (in "[monasca]" section) to get keystone token.
|
||||
|
||||
Example "[monasca]" section in ceilometer config file
|
||||
.. code-block:: text
|
||||
|
||||
[monasca]
|
||||
service_auth_url = https://localhost:5000/v3
|
||||
service_password = secretpassword
|
||||
service_username = ceilometer
|
||||
service_interface = internalURL
|
||||
service_auth_type = password
|
||||
# service_project_id may also be used
|
||||
service_project_name = admin
|
||||
service_domain_name = Default
|
||||
service_region_name = RegionOne
|
||||
|
||||
The publisher will then make a POST request to Monasca Events /v1.0/events REST api[8] to publish
|
||||
events to Monasca Events API. The URL for the instance of Monasca Events API will be configured
|
||||
in the Ceilometer 'events-pipeline.yaml' file. This has the added advantage of allowing
|
||||
different events to be published differently (see Ceilometer pipeline documentation [10]).
|
||||
|
||||
#. "tenant_id" and "user_id" that the notification relates to are available in "payload" section
|
||||
of the notification, and these notifications are generated by each service itself.
|
||||
|
||||
There is no additional "Openstack-operator-agent" like component or functionality required to
|
||||
fetch that data from the service and publish to monasca event api on behalf of the original
|
||||
tenant.
|
||||
Ceilometer publishing pipeline simply extracts these "tenant_id" and "user_id" fields from the
|
||||
"payload" and makes those fields available as "tenant_id" and "user_id" traits, which would then
|
||||
be mapped to "dimensions.project_id" and "dimensions.user_id" fields in monasca events format.
|
||||
|
||||
In other words, original "tenant_id" and "user_id" values are available in
|
||||
the payload of the notification, and will make its way to "dimensions.tenant_id"
|
||||
and "dimensions.user_id" in Monasca Event.
|
||||
|
||||
Questions/TODO:
|
||||
* Q: Do we need to do anything special to handle multi-tenancy in monasca-events api like being
|
||||
done for metrics[9] ? Would original user_id and tenant_id in "dimensions.user_id" and
|
||||
"dimensions.tenant_id" fields in dimensions serve this purpose?
|
||||
* Q: In Ceilometer V2 API (which has been deprecated and removed), when querying data the role
|
||||
"admin" could access data for all tenants, whereas a user with "ceilometer-admin" role could
|
||||
access only data for a particular tenant. Can we implement something like this for
|
||||
monasca-events api when querying for data?
|
||||
|
||||
#. Monasca Ceilometer event publisher will also retry submitting a batch, in case Monasca
|
||||
Events API is temporarily unavailable or down. The retry frequency, the number of retries
|
||||
and the number of items that can be in the retry batch will also be set via configuration.
|
||||
|
||||
|
||||
Alternative Solutions
|
||||
---------------------
|
||||
|
||||
#. Standalone monasca event agent which reads Openstack notifications published to RabbitMQ
|
||||
(on "notification" topic) and publishes them to Monasca Events API.
|
||||
Pro:
|
||||
* No dependency on Telemetry project.
|
||||
* May be simple to develop if leverage the oslo.messaging functionality.
|
||||
* Ceilometer has *deprecated* the events functionality in the Stein release. [13]
|
||||
Con:
|
||||
* Another agent to convince users to install on their systems.
|
||||
* Reinventing work already done in the Ceilometer agent. The OpenStack community already uses Ceilometer and contributes updates when something fails.
|
||||
This alternate solution will be detailed in a separate spec, as it is likely
|
||||
the long term solution Monasca will need.
|
||||
|
||||
#. Openstack Panko [5] is a event storage and REST API for Ceilometer.
|
||||
Pro:
|
||||
* An 'official' subproject within Telemetry, so there is some community recognition.
|
||||
Con:
|
||||
* Its primary storage is in a relational database which has problems with scale.
|
||||
* It is not maintained actively and not ready for production. [11]
|
||||
* It will be deprecated eventually. [12]
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
None
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
#. We are proposing to tweak the Monasca Event data format by removing and adding following
|
||||
fields as mentioned in "Proposed change" section above.
|
||||
|
||||
Remove fields (JSON path notation): "service", "dimensions.topic",
|
||||
"dimensions.hostname" and "event.priority"
|
||||
|
||||
Add fields (JSON path notation): "dimensions.publisher_id", "dimensions.user_id" and
|
||||
"dimensions.project_id"
|
||||
|
||||
This change will have an impact on Monasca Events API.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
The proposed Monasca Ceilometer events publisher will collect and publish
|
||||
Openstack event (notification) data to Monasca API. Openstack notification
|
||||
data does not have any sensitive data like 'tokens'.
|
||||
Notifications do contain 'user_id' and 'project_id' fields but do not
|
||||
contain any Personally Identifiable Information (PII) for the user or
|
||||
the project.
|
||||
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None.
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
#. The number of notifications(events) generated by different services will depend on the capacity
|
||||
of the cloud along with the number of resources being created by the users.
|
||||
|
||||
For example, if there was a large number of compute VM's being created or destroyed it could
|
||||
lead to a surge in number of notifications (events) that would have to be published. Optimum
|
||||
configuration options related to say event batch size and event batch interval would have to be
|
||||
documented, to reduce any adverse affect on performance.
|
||||
|
||||
#. Monasca Ceilometer publisher runs within Ceilometer Notification Agent component and invoked as a
|
||||
last step in its data processing pipeline. It is an additional component that will have to to be
|
||||
deployed on all the controller nodes. We will have to evaluate the performance impact of
|
||||
Ceilometer Notification Agent when publishing events to Monasca Events API.
|
||||
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
#. The proposed new Monasca-Ceilometer events publisher will introduce
|
||||
few new configuration options like
|
||||
* events api endpoint
|
||||
* events batch interval
|
||||
* events batch size
|
||||
* events retry interval
|
||||
|
||||
#. Monasca Ceilometer Events publisher will have to to be added to Ceilometer's
|
||||
"[ceilometer.event.publisher]" section entry_points.txt
|
||||
|
||||
For example:
|
||||
|
||||
[ceilometer.event.publisher]
|
||||
monasca = ceilometer.publisher.monclient:MonascaEventsPublisher
|
||||
|
||||
#. As part of developing new Monasca Ceilometer Events publisher devstack plugin would be updated to
|
||||
add the above configuration changes.
|
||||
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
#. The proposed change to Monasca Event Format will have an impact on existing Monasca Event API,
|
||||
since Monasca Event Format will have to be tweaked. (See REST API Impact section above)
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
joadavis, aagate
|
||||
|
||||
Other contributors:
|
||||
<launchpad-id or None>
|
||||
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
#. Implement new Monasca Ceilometer Events publisher.
|
||||
|
||||
#. Implement monasca-ceilometer devstack plugin changes to deploy
|
||||
new events publisher.
|
||||
|
||||
#. Implement unit tests for Events publisher.
|
||||
|
||||
#. Implement change to Monasca Event format in Monasca Events API.
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
#. Monasca Events API 1.0: https://storyboard.openstack.org/#!/story/2001654
|
||||
|
||||
#. Monasca Ceilometer project: https://github.com/openstack/monasca-ceilometer
|
||||
|
||||
#. Ceilometer Data processing and pipelines:
|
||||
https://docs.openstack.org/ceilometer/pike/admin/telemetry-data-pipelines.html
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
#. New Monasca Ceilometer Event publisher unit tests will be added, which can test publishing with
|
||||
various config options events batch size, events batch interval, handling retry when Monasca
|
||||
Event API is not available.
|
||||
|
||||
#. Adding tempest tests for Monasca Ceilometer events publisher could be looked at as part of
|
||||
separate effort.
|
||||
|
||||
Please note that current Monasca Ceilometer samples publisher does not have tempest tests either
|
||||
so having tempest tests for both events and samples publisher could be considered in the future.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
#. New Monasca Events Publisher config options will be documented
|
||||
|
||||
* events api endpoint
|
||||
* events batch interval
|
||||
* events batch size
|
||||
* events retry interval
|
||||
|
||||
#. Recommended values for each of the config options will also be documented based on the size of
|
||||
the cloud and resources for Cloud Operators.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
[1] Monasca Events API 1.0: https://storyboard.openstack.org/#!/story/2001654
|
||||
|
||||
[2] Monasca Ceilometer project: https://github.com/openstack/monasca-ceilometer
|
||||
|
||||
[3] Ceilometer Data processing and pipelines:
|
||||
https://docs.openstack.org/ceilometer/pike/admin/telemetry-data-pipelines.html
|
||||
|
||||
[4] Ceilometer Events: https://docs.openstack.org/ceilometer/latest/admin/telemetry-events.html
|
||||
|
||||
[5] Openstack Panko: https://github.com/openstack/panko
|
||||
|
||||
[6] Monasca Event Format:
|
||||
https://github.com/openstack/monasca-events-api/blob/master/doc/api-samples/v1/req_simple_event.json
|
||||
|
||||
[7] Ceilometer System Architecture Diagram:
|
||||
https://docs.openstack.org/ceilometer/ocata/architecture.html
|
||||
|
||||
[8] Monasca Events POST v1.0 API:
|
||||
https://github.com/openstack/monasca-events-api/blob/master/api-ref/source/events.inc
|
||||
|
||||
[9] Cross-Tenant Metric Submission:
|
||||
https://github.com/openstack/monasca-agent/blob/master/docs/MonascaMetrics.md#cross-tenant-metric-submission
|
||||
|
||||
[10] Ceilometer pipeline yaml documentation:
|
||||
https://docs.openstack.org/ceilometer/latest/admin/telemetry-data-pipelines.html
|
||||
|
||||
[11] No future for Panko or Aodh:
|
||||
https://julien.danjou.info/lessons-from-openstack-telemetry-deflation/
|
||||
|
||||
[12] Ceilometer Events deprecated means Panko also deprecated:
|
||||
http://eavesdrop.openstack.org/irclogs/%23openstack-telemetry/%23openstack-telemetry.2018-10-10.log.html
|
||||
|
||||
[13] Ceilometer Events marked as deprecated in Stein:
|
||||
https://review.openstack.org/#/c/603336/
|
|
@ -0,0 +1,586 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
=======================
|
||||
Monasca Events Listener
|
||||
=======================
|
||||
|
||||
https://storyboard.openstack.org/#!/story/2003023
|
||||
|
||||
Monasca Events API [1]_ was developed to store event data in Elasticsearch.
|
||||
A new application could use the Monasca Events API to post an event directly for processing
|
||||
and storage.
|
||||
However, a general collection service is needed to capture the existing OpenStack Notifications
|
||||
already generated by OpenStack Services and pass them to Monasca for storage and processing.
|
||||
This specification proposes creating a new Monasca Events Listener to capture events and pass
|
||||
them to Monasca services.
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
All Openstack services generate a lot of notifications or events which contain large amounts of
|
||||
operational and state information about the service and its resources. Services such as Nova [14]_
|
||||
already publish to a RabbitMQ topic.
|
||||
This notification data is not currently available in Monasca.
|
||||
|
||||
Ceilometer data processing pipeline [3]_ provides an extensible mechanism of publishing samples and
|
||||
events using a custom publisher. Ceilometer samples represent a quantity that can be measured
|
||||
(e.g. the size of a volume) and events represent an occurrence of an event and do not have any
|
||||
associated quantity (e.g. volume was created). The Telemetry project also has the Panko [5]_
|
||||
service for indexing and storing these events.
|
||||
|
||||
A previous `spec <../../rocky/not-implemented/monasca-events-publishing.html>`_ was created to
|
||||
specify an enhancement to Ceilometer to allow collected events to be published to the Monasca
|
||||
Events API.
|
||||
However, with the recent deprecation of Ceilometer's Event functionality [13]_, this is no longer
|
||||
a long-term option.
|
||||
|
||||
This spec is for creating a new Monasca service which would listen for OpenStack Event messages
|
||||
(often called notifications) and process them through Monasca by consuming the RabbitMQ message
|
||||
and producing a post to the Monasca Events API with that event.
|
||||
|
||||
This service could use Ceilometer, Vitrage, Watcher, or another service as an example of how to
|
||||
listen to notifications from OpenStack services such as Nova [14]_.
|
||||
|
||||
It is being proposed to make the Monasca Events Listener service a part of the Monasca Agent
|
||||
code base and reuse code, including the existing monasca_setup script and config.
|
||||
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
|
||||
#. Openstack notification data would be stored in Elasticsearch via the Monasca services
|
||||
|
||||
Example sequence:
|
||||
|
||||
#. Nova completes the creation of a VM
|
||||
#. Nova generates a Notification message to oslo.messaging
|
||||
#. oslo.messaging posts the message to RabbitMQ
|
||||
#. Monasca Event Listener receives the Notification message from RabbitMQ
|
||||
#. Monasca Event Listener validates and translates the Notification to a Monasca Events
|
||||
API format according to the configuration
|
||||
#. Monasca Event Listener publishes formatted Notification to Monasca Events API
|
||||
#. Monasca Events API receives and validates formatted Notification
|
||||
#. Monasca Events API stores event Notification in configured Elasticsearch instance
|
||||
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
#. Monasca Event Listener will be a new service which can be run in a Highly
|
||||
Available configuration by running an instance of Monasca Event Listener
|
||||
on each Controller node in a cloud. Each node will listen to OpenStack
|
||||
notifications in RabbitMQ and convert notifications to a post to the Monasca
|
||||
Events API. The Monasca Events API then passes the notification on as Kafka
|
||||
messages on the 'monevents' topic, where the Monasca Persister will receive
|
||||
and store them. By the nature of RabbitMQ clients, the load will be
|
||||
distributed between the Monasca Events Listener instances (only one listener
|
||||
will process each RabbitMQ message).
|
||||
|
||||
#. Monasca Event Listener will filter messages from OpenStack services based on
|
||||
specification of event_types to be collected. This will reduce 'noise' and
|
||||
focus event collection on those events that are deemed valuable.
|
||||
This filtering specification can be from a configuration file, or optionally
|
||||
could be controlled through a new API implemented as part of the Monasca
|
||||
Events API service.
|
||||
|
||||
#. OpenStack Notifications consist of envelope and payload fields. See [15]_ and [16]_ for
|
||||
examples.
|
||||
|
||||
Example OpenStack Notification data format:
|
||||
|
||||
.. code-block:: javascript
|
||||
|
||||
{
|
||||
"_context_auth_token": "42630b3ea13242fcad20e0a92d0207f1",
|
||||
"_context_domain": null,
|
||||
"_context_instance_lock_checked": false,
|
||||
"_context_is_admin": true,
|
||||
"_context_project_domain": null,
|
||||
"_context_project_id": "a4f77",
|
||||
"_context_project_name": "admin",
|
||||
"_context_quota_class": null,
|
||||
"_context_read_deleted": "no",
|
||||
"_context_read_only": false,
|
||||
"_context_remote_address": "192.168.245.4",
|
||||
"_context_request_id": "req-5948338c-f223-4fd8-9249-8769f7a3e460",
|
||||
"_context_resource_uuid": null,
|
||||
"_context_roles": [
|
||||
"monasca-user",
|
||||
"admin",
|
||||
"KeystoneAdmin"
|
||||
],
|
||||
"_context_service_catalog": [
|
||||
{
|
||||
"endpoints": [
|
||||
{
|
||||
"adminURL": "http://192.168.245.8:8776/v2/a4f77",
|
||||
"internalURL": "http://192.168.245.8:8776/v2/a4f77",
|
||||
"publicURL": "http://192.168.245.9:8776/v2/a4f77",
|
||||
"region": "region1"
|
||||
}
|
||||
],
|
||||
"name": "cinderv2",
|
||||
"type": "volumev2"
|
||||
},
|
||||
{
|
||||
"endpoints": [
|
||||
{
|
||||
"adminURL": "http://192.168.245.8:8776/v1/a4f77",
|
||||
"internalURL": "http://192.168.245.8:8776/v1/a4f77",
|
||||
"publicURL": "http://192.168.245.9:8776/v1/a4f77",
|
||||
"region": "region1"
|
||||
}
|
||||
],
|
||||
"name": "cinder",
|
||||
"type": "volume"
|
||||
}
|
||||
],
|
||||
"_context_show_deleted": false,
|
||||
"_context_tenant": "a4f77",
|
||||
"_context_timestamp": "2015-09-18T20:54:23.468522",
|
||||
"_context_user": "be396488c7034811a200a3cb1d103a28",
|
||||
"_context_user_domain": null,
|
||||
"_context_user_id": "be396488c7034811a200a3cb1d103a28",
|
||||
"_context_user_identity": "be396488c7034811a200a3cb1d103a28 a4f77 - - -",
|
||||
"_context_user_name": "admin",
|
||||
"_unique_id": "ff9699d587bf4283a3c367ab88be1541",
|
||||
"event_type": "compute.instance.create.start",
|
||||
"message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d",
|
||||
"payload": {
|
||||
"access_ip_v4": null,
|
||||
"access_ip_v6": null,
|
||||
"architecture": null,
|
||||
"availability_zone": null,
|
||||
"cell_name": "",
|
||||
"created_at": "2015-09-18 20:55:25+00:00",
|
||||
"deleted_at": "",
|
||||
"disk_gb": 1,
|
||||
"display_name": "testeee",
|
||||
"ephemeral_gb": 0,
|
||||
"host": null,
|
||||
"hostname": "testeee",
|
||||
"image_meta": {
|
||||
"base_image_ref": "df0c8",
|
||||
"container_format": "bare",
|
||||
"disk_format": "qcow2",
|
||||
"min_disk": "1",
|
||||
"min_ram": "0"
|
||||
},
|
||||
"image_name": "glanceaaa3",
|
||||
"image_ref_url": "http://192.168.245.5:9292/images/df0c8",
|
||||
"instance_flavor_id": "1",
|
||||
"instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6",
|
||||
"instance_type": "m1.tiny",
|
||||
"instance_type_id": 4,
|
||||
"kernel_id": "",
|
||||
"launched_at": "",
|
||||
"memory_mb": 512,
|
||||
"metadata": {},
|
||||
"node": null,
|
||||
"os_type": null,
|
||||
"progress": "",
|
||||
"ramdisk_id": "",
|
||||
"reservation_id": "r-1ghilddw",
|
||||
"root_gb": 1,
|
||||
"state": "building",
|
||||
"state_description": "",
|
||||
"tenant_id": "a4f77",
|
||||
"terminated_at": "",
|
||||
"user_id": "be396488c7034811a200a3cb1d103a28",
|
||||
"vcpus": 1
|
||||
},
|
||||
"priority": "INFO",
|
||||
"publisher_id": "compute.ccp-compute0001-mgmt",
|
||||
"timestamp": "2015-09-18 20:55:37.639023"
|
||||
}
|
||||
|
||||
#. All the fields with the prefix of '_context' are the envelope fields, the
|
||||
other interesting fields are
|
||||
|
||||
#. 'message_id' - notification identifier
|
||||
#. 'payload' - contains most of the relevant and useful information in JSON format
|
||||
#. 'priority' - notification priority
|
||||
#. 'publisher_id' - notification publisher
|
||||
#. 'timestamp' - notification timestamp
|
||||
|
||||
#. Monasca Event Listener converts the OpenStack notifications to Monasca events format.
|
||||
This format will be suitable for Kafka messaging and will match the expected data fields
|
||||
of the Monasca Persister. This conversion and validation should be common between the
|
||||
Monasca Event Listener and Monasca Event API.
|
||||
|
||||
#. The Kafka client connection will handle communication issues such as reconnections and
|
||||
resending as needed.
|
||||
|
||||
#. Monasca Events API allows a field called 'payload' which can be in an arbitrary
|
||||
nested JSON format.
|
||||
TODO: mappings
|
||||
|
||||
Example Monasca Event Format:
|
||||
|
||||
.. code-block:: javascript
|
||||
|
||||
events: [
|
||||
{
|
||||
dimensions": {
|
||||
"service": "compute.ccp-compute0001-mgmt",
|
||||
"topic": "notification.sample",
|
||||
"hostname": "nova-compute:compute
|
||||
},
|
||||
event: {
|
||||
|
||||
"event_type": "compute.instance.create.start",
|
||||
|
||||
"payload": {
|
||||
"access_ip_v4": null,
|
||||
"access_ip_v6": null,
|
||||
"architecture": null,
|
||||
"availability_zone": null,
|
||||
"cell_name": "",
|
||||
"created_at": "2015-09-18 20:55:25+00:00",
|
||||
"deleted_at": "",
|
||||
"disk_gb": 1,
|
||||
"display_name": "testeee",
|
||||
"ephemeral_gb": 0,
|
||||
"host": null,
|
||||
"hostname": "testeee",
|
||||
"image_meta": {
|
||||
"base_image_ref": "df0c8",
|
||||
"container_format": "bare",
|
||||
"disk_format": "qcow2",
|
||||
"min_disk": "1",
|
||||
"min_ram": "0"
|
||||
},
|
||||
"image_name": "glanceaaa3",
|
||||
"image_ref_url": "http://192.168.245.5:9292/images/df0c8",
|
||||
"instance_flavor_id": "1",
|
||||
"instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6",
|
||||
"instance_type": "m1.tiny",
|
||||
"instance_type_id": 4,
|
||||
"kernel_id": "",
|
||||
"launched_at": "",
|
||||
"memory_mb": 512,
|
||||
"metadata": {},
|
||||
"node": null,
|
||||
"os_type": null,
|
||||
"progress": "",
|
||||
"ramdisk_id": "",
|
||||
"reservation_id": "r-1ghilddw",
|
||||
"root_gb": 1,
|
||||
"state": "building",
|
||||
"state_description": "",
|
||||
"tenant_id": "a4f77",
|
||||
"terminated_at": "",
|
||||
"user_id": "be396488c7034811a200a3cb1d103a28",
|
||||
"vcpus": 1
|
||||
}
|
||||
},
|
||||
publisher_id: "compute.ccp-compute0001-mgmt",
|
||||
priority: "INFO"
|
||||
}
|
||||
]
|
||||
|
||||
#. Following fields in Monasca Event data may not be available in the OpenStack notification
|
||||
data format:
|
||||
|
||||
* "service"
|
||||
* "dimensions.topic"
|
||||
* "event.priority"
|
||||
|
||||
We are proposing removing these fields from Monasca Event format (will be done as a separate
|
||||
spec/implementation process) for the following reasons:
|
||||
|
||||
"service": Currently OpenStack notifications do not specify a service, that
|
||||
generated the notification in a consistent way. It might be possible to create an external
|
||||
mapping file which maps event name to a service but its hard to maintain such mapping over a
|
||||
period of time.
|
||||
|
||||
"dimensions.topic": This field is not available in the source OpenStack notification.
|
||||
However, the Monasca Event Listener may be able to save the RabbitMQ topic that the notification
|
||||
was collected from. In that case, this field should be used.
|
||||
|
||||
"event.priority": This field is not currently available in Ceilometer Event format. It is
|
||||
available in the source OpenStack notification. Note: If we think this field can be useful we can
|
||||
propose adding it to the Monasca Event Listener format.
|
||||
|
||||
#. Following new fields will be added to Monasca Event data as dimensions:
|
||||
|
||||
* "dimensions.publisher_id": Identifier for the publisher that generated the event.
|
||||
* "dimensions.user_id": Identifier for user that generated the event.
|
||||
* "dimensions.project_id": Identifier of the project that generated the event.
|
||||
|
||||
#. 'hostname' is available in the event payload, but its location might differ from event to event.
|
||||
|
||||
#. The proposed new Monasca Event Listener will have the ability to submit event
|
||||
data in a batch and at a configurable frequency (similar to current samples publisher). The
|
||||
event data will be published if the items in the current batch reach their maximum size
|
||||
(config setting) or if certain time interval has elapsed since the last publish
|
||||
(config setting). This will make sure that the batch does not get huge at the same time
|
||||
there is no significant delay in publishing of the events to Monasca Events API.
|
||||
|
||||
#. Monasca Event Listener will have a configuration file to configure connection information for
|
||||
both RabbitMQ and Monasca Events API.
|
||||
|
||||
#. The "tenant_id" and "user_id" that the notification relates to are available in "payload"
|
||||
section of the notification, and these notifications are generated by each service itself.
|
||||
|
||||
There is no additional "OpenStack-operator-agent" like component or functionality required to
|
||||
fetch that data from the service and publish to monasca event api on behalf of the original
|
||||
tenant.
|
||||
(Ceilometer publishing pipeline simply extracts these "tenant_id" and "user_id" fields from the
|
||||
"payload" and makes those fields available as "tenant_id" and "user_id" traits, which would then
|
||||
be mapped to "dimensions.project_id" and "dimensions.user_id" fields in monasca events format.)
|
||||
|
||||
In other words, original "tenant_id" and "user_id" values are available in
|
||||
the payload of the notification, and will make its way to "dimensions.tenant_id"
|
||||
and "dimensions.user_id" in Monasca Event.
|
||||
|
||||
Questions/TODO:
|
||||
|
||||
* Q: Do we need to do anything special to handle multi-tenancy in monasca-events api like being
|
||||
done for metrics [9]_ ? Would original user_id and tenant_id in "dimensions.user_id" and
|
||||
"dimensions.tenant_id" fields in dimensions serve this purpose?
|
||||
|
||||
* A: Monasca Events Listener can start with sending all events to a single "admin" tenant
|
||||
and if required in the future some other process could copy select metrics back to tenant
|
||||
projects.
|
||||
|
||||
* Q: In Ceilometer V2 API (which has been deprecated and removed), when querying data the role
|
||||
"admin" could access data for all tenants, whereas a user with "ceilometer-admin" role could
|
||||
access only data for a particular tenant. Can we implement something like this for
|
||||
monasca-events api when querying for data?
|
||||
|
||||
* A: In Monasca API every request is scoped to the project, so there is no equivalent of
|
||||
Ceilometer's "admin" role to query data for all projects. So placing all events in to
|
||||
an "admin" project may be the best approach.
|
||||
|
||||
* Q: How should services which generate notifications but do not include a tenant_id be
|
||||
handled? For example Keystone [16]_.
|
||||
How does Ceilometer handle such events?
|
||||
|
||||
* A: If all events are in an "admin" project then admin metrics like shared ceph cluster
|
||||
load or provider network load can be copied back to tenants so they may understand how
|
||||
infrastructure is affecting their workload.
|
||||
|
||||
* Note: Configuration of Elasticsearch cluster is out of scope for this spec. If needed
|
||||
could assign a separate Elasticsearch cluster to the events API to avoid overloads.
|
||||
|
||||
|
||||
Alternative Solutions
|
||||
---------------------
|
||||
|
||||
#. Reuse the Ceilometer functionality to collect and publish events to the
|
||||
Monasca Events API. While this may be less work initially, Ceilometer has
|
||||
deprecated the Events functionality as of Stein. [13]_
|
||||
|
||||
#. An alternate Events Listener was proposed that would listen for RabbitMQ events
|
||||
then publish them directly to the 'monevents' topic in Kafka. Discussion on this
|
||||
can be seen in the git history for this document and in IRC logs [18]_.
|
||||
|
||||
Pro:
|
||||
|
||||
* A much simpler approach, more efficient that HTTP hop through another service.
|
||||
* No need for batching in service, as RabbitMQ and Kafka clients would handle fast
|
||||
throughput and short network interruptions.
|
||||
|
||||
Con:
|
||||
|
||||
* Nova Cells v2 each have their own RabbitMQ instances. While most deployments
|
||||
would likely have a centralized RabbitMQ, it is not required in the documented
|
||||
architecture.
|
||||
* Regions may also cause separation of RabbitMQ instances that need to be monitored.
|
||||
* While it might be possible to have a service/agent in each Cell publish back to
|
||||
a centralized to Kafka directly, our authentication and networking for Kafka was
|
||||
not designed to support that.
|
||||
|
||||
#. OpenStack Panko [5]_ is a event storage and REST API for Ceilometer and could
|
||||
be used instead of a Monasca solution.
|
||||
|
||||
Pro:
|
||||
|
||||
* An 'official' subproject within Telemetry, so there is some community recognition.
|
||||
|
||||
Con:
|
||||
|
||||
* Its primary storage is in a relational database which has problems with scale.
|
||||
* It is not maintained actively and not ready for production. [11]_
|
||||
* It will be deprecated eventually. [12]_
|
||||
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
None
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
#. We are proposing to tweak the Monasca Event data format by removing and adding following
|
||||
fields as mentioned in "Proposed change" section above.
|
||||
|
||||
Remove fields or make them optional (JSON path notation): "service", "dimensions.topic",
|
||||
"dimensions.hostname" and "event.priority"
|
||||
|
||||
Add fields (JSON path notation): "dimensions.publisher_id", "dimensions.user_id" and
|
||||
"dimensions.project_id"
|
||||
|
||||
This change will have an impact on Monasca Events API.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
The proposed Monasca Event Listener will collect and publish OpenStack event
|
||||
(notification) data to Monasca Events API. OpenStack notification data does not
|
||||
have any sensitive data like 'tokens'.
|
||||
Notifications do contain 'user_id' and 'project_id' fields but do not
|
||||
contain any Personally Identifiable Information (PII) for the user or
|
||||
the project.
|
||||
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None.
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
#. The number of notifications(events) generated by different services will depend on the capacity
|
||||
of the cloud along with the number of resources being created by the users.
|
||||
|
||||
For example, if there was a large number of compute VM's being created or destroyed it could
|
||||
lead to a surge in number of notifications (events) that would have to be published. Optimum
|
||||
configuration options related to say event batch size and event batch interval would have to be
|
||||
documented, to reduce any adverse affect on performance.
|
||||
|
||||
#. The proposed Monasca Event Listener is a new service, so performance is unknown. However, the
|
||||
Monasca API has been shown to have a high performance throughput.
|
||||
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
#. The proposed Monasca Event Listener will introduce a
|
||||
few new configuration options like
|
||||
|
||||
* RabbitMQ connection information
|
||||
* Monasca Events API endpoint URL
|
||||
* events batch interval
|
||||
* events batch size
|
||||
* events retry interval
|
||||
* Keystone credentials for obtaining a token
|
||||
* Conversion options for OpenStack notifications to the Monasca Event format. This may
|
||||
be stored in separate pipeline configuration files, similar to how transform specs
|
||||
are configured in Monasca Transform.
|
||||
|
||||
#. As part of developing new Monasca Event Listener devstack plugin would be
|
||||
updated to add the above configuration changes.
|
||||
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
#. The proposed change to Monasca Event Format will have an impact on existing Monasca Event API,
|
||||
since Monasca Event Format will have to be tweaked. (See REST API Impact section above)
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
joadavis, aagate
|
||||
|
||||
Other contributors:
|
||||
<launchpad-id or None>
|
||||
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
#. Implement new Monasca Event Listener.
|
||||
|
||||
* Connection to RabbitMQ for OpenStack Notifications
|
||||
* Add filtering of notifications for configured event_types
|
||||
|
||||
* Specification in configuration file
|
||||
* (Optional) Creation of a new API to configure event_type subscriptions
|
||||
|
||||
* Validation of OpenStack Notification data and format
|
||||
* Conversion of data format to meet Monasca Events requirements
|
||||
* Publishing to Monasca Events API
|
||||
* Configuration of conversion specifications per-event type
|
||||
|
||||
#. Implement monasca devstack plugin changes to deploy
|
||||
new Events Listener service.
|
||||
|
||||
#. Implement unit tests for Monasca Event Listener.
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
None
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
#. New Monasca Event Listener unit tests will be added, which can test publishing with
|
||||
various config options events batch size, events batch interval, handling retry when Monasca
|
||||
Event API is not available.
|
||||
|
||||
#. Adding tempest tests for Monasca Event Listener could be looked at as part of
|
||||
separate effort.
|
||||
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
#. New Monasca Event Listener config options will be documented
|
||||
|
||||
#. Recommended values for each of the config options will also be documented based on the size of
|
||||
the cloud and resources for Cloud Operators.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
.. [1] Monasca Events API 1.0: https://git.openstack.org/cgit/openstack/monasca-events-api/
|
||||
|
||||
[2] Monasca Ceilometer project: https://github.com/openstack/monasca-ceilometer
|
||||
|
||||
.. [3] Ceilometer Data processing and pipelines: https://docs.openstack.org/ceilometer/pike/admin/telemetry-data-pipelines.html
|
||||
|
||||
[4] Ceilometer Events: https://docs.openstack.org/ceilometer/latest/admin/telemetry-events.html
|
||||
|
||||
.. [5] Openstack Panko: https://github.com/openstack/panko
|
||||
|
||||
[6] Monasca Event Format: https://github.com/openstack/monasca-events-api/blob/master/doc/api-samples/v1/req_simple_event.json
|
||||
|
||||
[7] Ceilometer System Architecture Diagram: https://docs.openstack.org/ceilometer/ocata/architecture.html
|
||||
|
||||
[8] Monasca Events POST v1.0 API: https://github.com/openstack/monasca-events-api/blob/master/api-ref/source/events.inc
|
||||
|
||||
.. [9] Cross-Tenant Metric Submission: https://github.com/openstack/monasca-agent/blob/master/docs/MonascaMetrics.md#cross-tenant-metric-submission
|
||||
|
||||
[10] Ceilometer pipeline yaml documentation: https://docs.openstack.org/ceilometer/latest/admin/telemetry-data-pipelines.html
|
||||
|
||||
.. [11] No future for Panko or Aodh: https://julien.danjou.info/lessons-from-openstack-telemetry-deflation/
|
||||
|
||||
.. [12] Ceilometer Events deprecated means Panko also deprecated: http://eavesdrop.openstack.org/irclogs/%23openstack-telemetry/%23openstack-telemetry.2018-10-10.log.html
|
||||
|
||||
.. [13] Ceilometer Events marked as deprecated in Stein: https://review.openstack.org/#/c/603336/
|
||||
|
||||
.. [14] Nova notification version update lists services effected (see "Deprecating legacy notifications"): https://etherpad.openstack.org/p/nova-ptg-stein
|
||||
|
||||
.. [15] Nova notification reference: https://docs.openstack.org/nova/latest/reference/notifications.html#existing-versioned-notifications
|
||||
|
||||
.. [16] Keystone notification reference: https://docs.openstack.org/keystone/latest/advanced-topics/event_notifications.html#example-notification-project-create
|
||||
|
||||
[17] Monasca Events API publisher to Kafka: https://github.com/openstack/monasca-events-api/blob/master/monasca_events_api/app/common/events_publisher.py
|
||||
|
||||
.. [18] Monasca IRC meeting Dec 15, 2018: http://eavesdrop.openstack.org/meetings/monasca/2018/monasca.2018-12-12-15.00.log.html
|
Loading…
Reference in New Issue