Browse Source

Monasca Events Publishing Spec

Monasca Events API was developed to store events
data in Elasticsearch.

There is still a need to collect and publish
Openstack Notifications as Monasca Events.

This commit includes a specification for creating a new Monasca
Events Listener service that would be independent of Ceilometer.
This is being proposed for the Stein cycle.

Monasca Ceilometer project currently publishes ceilometer samples
to Monasca API. An alternate specification proposes to extend Monasca
Ceilometer project and add a new events publisher which would publish
Openstack notifications (or events) to Monasca Events API.
This solution was proposed for Rocky but will not be implemented
as the Ceilometer Events feature has been deprecated on master.

Co-Authored-By: Joseph Davis <joseph.davis@suse.com>
Co-Authored-By: Ashwin Agate <ashwin.agate@suse.com>

Story: 2003023
Task: 23047

Change-Id: I8ed7a19c0565a0cd613866934eb2540ae46c847d
Ashwin Agate 10 months ago
parent
commit
f01ecbb9ad

+ 6
- 3
README.rst View File

@@ -25,11 +25,14 @@ The layout of this repository is::
25 25
   priorities/<release>/
26 26
   specs/<release>/
27 27
 
28
-Where there are two sub-directories in ``specs``:
28
+Where there are three sub-directories in ``specs``:
29 29
 
30 30
 specs/<release>/approved
31
-  specifications approved, but not yet implemented
31
+  Specifications approved, but not yet implemented
32 32
 
33 33
 specs/<release>/implemented
34
-  implemented specifications
34
+  Implemented specifications
35 35
 
36
+specs/<release>/not-implemented
37
+  Specifications that were approved but are not expected to be implemented.
38
+  These are kept for historical reference.

+ 8
- 0
doc/source/specs/rocky/index.rst View File

@@ -24,3 +24,11 @@ Rocky approved (but not implemented) specs:
24 24
    :maxdepth: 1
25 25
 
26 26
    approved/*
27
+
28
+Rocky deprecated specs:
29
+
30
+.. toctree::
31
+   :glob:
32
+   :maxdepth: 1
33
+
34
+   not-implemented/*

+ 1
- 0
doc/source/specs/rocky/not-implemented View File

@@ -0,0 +1 @@
1
+../../../../specs/rocky/not-implemented

+ 705
- 0
specs/rocky/not-implemented/monasca-events-publishing.rst View File

@@ -0,0 +1,705 @@
1
+..
2
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
3
+ License.
4
+
5
+ http://creativecommons.org/licenses/by/3.0/legalcode
6
+
7
+================================================
8
+Monasca Events Publishing - from Ceilometer
9
+================================================
10
+
11
+https://storyboard.openstack.org/#!/story/2003023
12
+
13
+Monasca Events API [1] was developed to store Openstack Notification data in Elasticsearch. There is
14
+still a need to collect and publish Openstack Notifications to Monasca Events API. Monasca
15
+Ceilometer project[2] currently publishes ceilometer samples[3] to Monasca API. We are proposing to
16
+extend Monasca Ceilometer project and add a new events publisher which will publish Openstack
17
+notifications (or events)[3] to Monasca Events API.
18
+
19
+UPDATE: This spec is being superceded by the ../../stein/approved/monasca-events-listener.rst spec,
20
+but is kept here for reference.
21
+
22
+
23
+Problem description
24
+===================
25
+
26
+All Openstack services generate a lot of notifications or events which contain large amounts of
27
+operational and state information about the service and its resources. This notification data is not
28
+currently available in Monasca.
29
+
30
+Ceilometer data processing pipeline[3] provides an extensible mechanism of publishing samples and
31
+events using a custom publisher.  Ceilometer samples represent a quantity that can be measured (for
32
+e.g. the size of a volume) and events represent an occurrence of an event and do not have any
33
+associated quantity (e.g. volume was created).
34
+
35
+Monasca Ceilometer project currently provides a samples publisher. Monasca Ceilometer samples
36
+publisher converts Ceilometer samples to Monasca Metrics format which are then published to Monasca
37
+API. There is no corresponding events publisher to Monasca yet.
38
+
39
+By adding an event publisher to Monasca Ceilometer project we could take advantage of Ceilometer's
40
+event publishing mechanism to publish events to Monasca Events API.
41
+
42
+Ceilometer consists of different data collection components - namely Polling Agent, Notification
43
+Agent and Compute Agent. (Please see [7] for System Architecture diagram) Ceilometer also has a data
44
+storage and retrieval component, which would be Monasca in our case.
45
+
46
+Samples publisher and new proposed events publisher run within Ceilometer's Notification Agent
47
+component and are part of Notifcation Agent's data processing pipeline. Monasca
48
+Ceilometer presumes the need to install and deploy Ceilometer Notification Agent component (doesn't
49
+need Polling Agent or Compute Agent deployed) on all the control nodes. Ceilometer Notification
50
+Agent is Highly Available (HA) and can run on multiple nodes. We will have to evaluate its
51
+performance in terms of scaling for events, but we haven't run into performance/scale problems
52
+with current samples publisher.
53
+
54
+
55
+Use Cases
56
+---------
57
+
58
+#. Openstack notification data would be stored in Elasticsearch
59
+   via the Monasca Events API
60
+
61
+   Example sequence from Nova notification to Monasca API
62
+   #. Nova completes the creation of a VM
63
+   #. Nova generates a Notification message to oslo.messaging
64
+   #. Ceilometer Notification agent receives the Notification message
65
+   #. Ceilometer translates the Notification to a Monasca API format according to the configuration
66
+   #. Ceilometer Event Publisher publishes formatted Notification to Monasca Events API
67
+   #. Monasca Events API receives and validates formatted Notification
68
+   #. Monasca Events stores event Notification in configured Elasticsearch instance
69
+
70
+
71
+Proposed change
72
+===============
73
+
74
+#. Openstack Notifications consist of envelope and payload fields
75
+
76
+   Example Openstack Notification data format:
77
+
78
+   .. code-block:: javascript
79
+
80
+        {
81
+            "_context_auth_token": "42630b3ea13242fcad20e0a92d0207f1",
82
+            "_context_domain": null,
83
+            "_context_instance_lock_checked": false,
84
+            "_context_is_admin": true,
85
+            "_context_project_domain": null,
86
+            "_context_project_id": "a4f77",
87
+            "_context_project_name": "admin",
88
+            "_context_quota_class": null,
89
+            "_context_read_deleted": "no",
90
+            "_context_read_only": false,
91
+            "_context_remote_address": "192.168.245.4",
92
+            "_context_request_id": "req-5948338c-f223-4fd8-9249-8769f7a3e460",
93
+            "_context_resource_uuid": null,
94
+            "_context_roles": [
95
+                "monasca-user",
96
+                "admin",
97
+                "KeystoneAdmin"
98
+            ],
99
+            "_context_service_catalog": [
100
+                {
101
+                    "endpoints": [
102
+                        {
103
+                            "adminURL": "http://192.168.245.8:8776/v2/a4f77",
104
+                            "internalURL": "http://192.168.245.8:8776/v2/a4f77",
105
+                            "publicURL": "http://192.168.245.9:8776/v2/a4f77",
106
+                            "region": "region1"
107
+                        }
108
+                    ],
109
+                    "name": "cinderv2",
110
+                    "type": "volumev2"
111
+                },
112
+                {
113
+                    "endpoints": [
114
+                        {
115
+                            "adminURL": "http://192.168.245.8:8776/v1/a4f77",
116
+                            "internalURL": "http://192.168.245.8:8776/v1/a4f77",
117
+                            "publicURL": "http://192.168.245.9:8776/v1/a4f77",
118
+                            "region": "region1"
119
+                        }
120
+                    ],
121
+                    "name": "cinder",
122
+                    "type": "volume"
123
+                }
124
+            ],
125
+            "_context_show_deleted": false,
126
+            "_context_tenant": "a4f77",
127
+            "_context_timestamp": "2015-09-18T20:54:23.468522",
128
+            "_context_user": "be396488c7034811a200a3cb1d103a28",
129
+            "_context_user_domain": null,
130
+            "_context_user_id": "be396488c7034811a200a3cb1d103a28",
131
+            "_context_user_identity": "be396488c7034811a200a3cb1d103a28 a4f77 - - -",
132
+            "_context_user_name": "admin",
133
+            "_unique_id": "ff9699d587bf4283a3c367ab88be1541",
134
+            "event_type": "compute.instance.create.start",
135
+            "message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d",
136
+            "payload": {
137
+                "access_ip_v4": null,
138
+                "access_ip_v6": null,
139
+                "architecture": null,
140
+                "availability_zone": null,
141
+                "cell_name": "",
142
+                "created_at": "2015-09-18 20:55:25+00:00",
143
+                "deleted_at": "",
144
+                "disk_gb": 1,
145
+                "display_name": "testeee",
146
+                "ephemeral_gb": 0,
147
+                "host": null,
148
+                "hostname": "testeee",
149
+                "image_meta": {
150
+                    "base_image_ref": "df0c8",
151
+                    "container_format": "bare",
152
+                    "disk_format": "qcow2",
153
+                    "min_disk": "1",
154
+                    "min_ram": "0"
155
+                },
156
+                "image_name": "glanceaaa3",
157
+                "image_ref_url": "http://192.168.245.5:9292/images/df0c8",
158
+                "instance_flavor_id": "1",
159
+                "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6",
160
+                "instance_type": "m1.tiny",
161
+                "instance_type_id": 4,
162
+                "kernel_id": "",
163
+                "launched_at": "",
164
+                "memory_mb": 512,
165
+                "metadata": {},
166
+                "node": null,
167
+                "os_type": null,
168
+                "progress": "",
169
+                "ramdisk_id": "",
170
+                "reservation_id": "r-1ghilddw",
171
+                "root_gb": 1,
172
+                "state": "building",
173
+                "state_description": "",
174
+                "tenant_id": "a4f77",
175
+                "terminated_at": "",
176
+                "user_id": "be396488c7034811a200a3cb1d103a28",
177
+                "vcpus": 1
178
+            },
179
+            "priority": "INFO",
180
+            "publisher_id": "compute.ccp-compute0001-mgmt",
181
+            "timestamp": "2015-09-18 20:55:37.639023"
182
+        }
183
+
184
+#. All the fields with the prefix of '_context" are the envelope fields, the
185
+   other interesting fields are
186
+
187
+   #. 'message_id' - notification identifier
188
+   #. 'payload' - contains most of the relevant and useful information in JSON format
189
+   #. 'priority' - notification priority
190
+   #. 'publisher_id' - notification publisher
191
+   #. 'timestamp' - notification timestamp
192
+
193
+#. Ceilometer event publishing framework converts the Openstack notifications to events format[4].
194
+   Event publishing framework also has the ability to extract only some of the 'payload' data into
195
+   a flat set of key-value pairs called 'traits' and publish the normalized 'event' with 'traits'
196
+   extracted from the payload using a custom publisher.
197
+
198
+   Extraction of certain fields into traits from the payload is
199
+   driven by configuration file, but by default "publisher_id",
200
+   'request_id', 'tenant_id', 'user_id' and 'project_id'
201
+   fields are always extracted and added as 'traits'.
202
+
203
+   The event can also have an optional field called 'raw' which has original
204
+   notification, provided 'store_raw' option is set in ceilometer.conf
205
+
206
+   Questions/TODO:
207
+
208
+   * Q1: Does the store_raw field only apply to events, or to all notifications processed by
209
+     Ceilometer?
210
+   * We will have to find it out if it has any adverse impact on sample publisher. Though in the
211
+     case of samples, monasca sample publisher definitely does not submit raw payload, so it must
212
+     be getting dropped.
213
+
214
+   Example Ceilometer Event data format:
215
+
216
+   .. code-block:: javascript
217
+
218
+      {
219
+        "event_type": "compute.instance.create.start",
220
+        "message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d",
221
+        "generated": "2015-09-18 20:55:37.639023",
222
+        "traits": {
223
+           "publisher_id": "compute.ccp-compute0001-mgmt",
224
+           "request_id": "req-5948338c-f223-4fd8-9249-8769f7a3e460",
225
+           "tenant_id": "a4f77",
226
+           "project_id": "a4f77",
227
+           "user_id": "be396488c7034811a200a3cb1d103a28"
228
+         },
229
+         "raw": {  "_context_auth_token": "42630b3ea13242fcad20e0a92d0207f1",
230
+                   "_context_domain": null,
231
+                   ...
232
+                   ...
233
+                   "event_type": "compute.instance.create.start",
234
+                   "message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d",
235
+                   "payload": {
236
+                       "access_ip_v4": null,
237
+                       "access_ip_v6": null,
238
+                       "architecture": null,
239
+                       "availability_zone": null,
240
+                       "cell_name": "",
241
+                       "created_at": "2015-09-18 20:55:25+00:00",
242
+                       "deleted_at": "",
243
+                       "disk_gb": 1,
244
+                       "display_name": "testeee",
245
+                       "ephemeral_gb": 0,
246
+                       "host": null,
247
+                       "hostname": "testeee",
248
+                       "image_meta": {
249
+                           "base_image_ref": "df0c8",
250
+                           "container_format": "bare",
251
+                           "disk_format": "qcow2",
252
+                           "min_disk": "1",
253
+                           "min_ram": "0"
254
+                       },
255
+                      "image_name": "glanceaaa3",
256
+                      "image_ref_url": "http://192.168.245.5:9292/images/df0c8",
257
+                      "instance_flavor_id": "1",
258
+                      "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6",
259
+                      "instance_type": "m1.tiny",
260
+                      "instance_type_id": 4,
261
+                      "kernel_id": "",
262
+                      "launched_at": "",
263
+                      "memory_mb": 512,
264
+                      "metadata": {},
265
+                      "node": null,
266
+                      "os_type": null,
267
+                      "progress": "",
268
+                      "ramdisk_id": "",
269
+                      "reservation_id": "r-1ghilddw",
270
+                      "root_gb": 1,
271
+                      "state": "building",
272
+                      "state_description": "",
273
+                      "tenant_id": "a4f77",
274
+                      "terminated_at": "",
275
+                      "user_id": "be396488c7034811a200a3cb1d103a28",
276
+                      "vcpus": 1
277
+                      }
278
+                }
279
+      }
280
+
281
+
282
+#. Key-Value pairs that can be extracted from 'payload' in form of traits
283
+   can be defined in events definitions file.
284
+
285
+   For example the following events definitions yaml specifies that for
286
+   all events which have a prefix of "compute.instance.*" then
287
+   add  "user_id", "instance_id", and "instance_type_id" as traits,
288
+   after extracting values from "payload.user_id", "payload.instance_id",
289
+   and "payload.instance_type_id" respectively.
290
+
291
+   .. code-block:: yaml
292
+
293
+   ---
294
+   - event_type: compute.instance.*
295
+
296
+     traits: &instance_traits
297
+      user_id:
298
+        fields: payload.user_id
299
+      instance_id:
300
+        fields: payload.instance_id
301
+      instance_type_id:
302
+        type: int
303
+        fields: payload.instance_type_id
304
+
305
+   We are for now proposing not to use this feature, of defining traits for each event
306
+   extracting since we have the ability to store entire payload, via
307
+   Monasca Events API.
308
+
309
+   We can certainly look at enabling this feature in the future if we run into trouble storing
310
+   entire JSON "payload" in Elasticsearch. This is certainly a nifty way to trim the amount
311
+   of data that will be stored.
312
+
313
+#. The proposed new Custom Monasca Ceilometer event publisher will run within Ceilometer's
314
+   Notification Agent component. It will leverage Ceilometer's data processing pipeline[3] which
315
+   converts notifications to Ceilometer's event format.  At the end of its processing, Monasca
316
+   Ceilometer event publisher will convert Ceilometer Event data into Monasca Event format[6] and
317
+   publish the monasca event to Monasca Events API.
318
+
319
+#. Monasca Events API allows a field called 'payload' which can be in an arbitrary
320
+   nested JSON format. Monasca-Ceilometer event publisher will extract JSON field called
321
+   'payload' from 'raw' (JSON path notation: 'raw.payload'), publish the payload from the
322
+   original notification to Monasca Events API.
323
+
324
+   Example Monasca Event Format:
325
+
326
+   .. code-block:: javascript
327
+
328
+        events: [
329
+        {
330
+          dimensions": {
331
+                "service": "compute.ccp-compute0001-mgmt",
332
+                "topic": "notification.sample",
333
+                "hostname": "nova-compute:compute
334
+          },
335
+          event: {
336
+
337
+                  "event_type": "compute.instance.create.start",
338
+
339
+                  "payload": {
340
+                       "access_ip_v4": null,
341
+                       "access_ip_v6": null,
342
+                       "architecture": null,
343
+                       "availability_zone": null,
344
+                       "cell_name": "",
345
+                       "created_at": "2015-09-18 20:55:25+00:00",
346
+                       "deleted_at": "",
347
+                       "disk_gb": 1,
348
+                       "display_name": "testeee",
349
+                       "ephemeral_gb": 0,
350
+                       "host": null,
351
+                       "hostname": "testeee",
352
+                       "image_meta": {
353
+                           "base_image_ref": "df0c8",
354
+                           "container_format": "bare",
355
+                           "disk_format": "qcow2",
356
+                           "min_disk": "1",
357
+                           "min_ram": "0"
358
+                       },
359
+                      "image_name": "glanceaaa3",
360
+                      "image_ref_url": "http://192.168.245.5:9292/images/df0c8",
361
+                      "instance_flavor_id": "1",
362
+                      "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6",
363
+                      "instance_type": "m1.tiny",
364
+                      "instance_type_id": 4,
365
+                      "kernel_id": "",
366
+                      "launched_at": "",
367
+                      "memory_mb": 512,
368
+                      "metadata": {},
369
+                      "node": null,
370
+                      "os_type": null,
371
+                      "progress": "",
372
+                      "ramdisk_id": "",
373
+                      "reservation_id": "r-1ghilddw",
374
+                      "root_gb": 1,
375
+                      "state": "building",
376
+                      "state_description": "",
377
+                      "tenant_id": "a4f77",
378
+                      "terminated_at": "",
379
+                      "user_id": "be396488c7034811a200a3cb1d103a28",
380
+                      "vcpus": 1
381
+                      }
382
+                 },
383
+            publisher_id: "compute.ccp-compute0001-mgmt",
384
+            priority: "INFO"
385
+         }
386
+        ]
387
+
388
+#. If no traits are specified in events pipeline yaml configuration file for an event
389
+   Ceilometer's data processing pipeline will add the following default traits:
390
+
391
+   * service: (All notifications should have this) notification’s publisher
392
+   * tenant_id
393
+   * request_id
394
+   * project_id
395
+   * user_id
396
+
397
+   Note: "service" is not the service that produced the event as in say "compute", "glance",
398
+   "cinder" but rather notification RabbitMQ publisher that produced the event
399
+   e.g. "compute.ccp-compute0001-mgmt" so is not very useful.
400
+
401
+#. Ceilometer event data is converted to Monasca event data format before being published to Monasca
402
+   Event API. Following fields in Monasca Event data are not available in current Ceilometer Event
403
+   data format:
404
+
405
+   * "service"
406
+   * "dimensions.topic"
407
+   * "event.priority"
408
+
409
+   We are proposing removing these fields from Monasca Event format (will be done as a separate
410
+   spec/implementation process) for the following reasons:
411
+
412
+   "service": Currently Openstack notifications do not specify a service, that
413
+   generated the notification in a consistent way. It might be possible to create an external
414
+   mapping file which maps event name to a service but its hard to maintain such mapping over a
415
+   period of time.
416
+
417
+   "dimensions.topic": This field is not available in the source Openstack notification
418
+
419
+   "event.priority": This field is not currently available in Ceilometer Event format. It is
420
+   available in the source Openstack notification. Note: If we think this field can be useful we can
421
+   propose adding it to the Ceilometer Event format.
422
+
423
+#. Following new fields will be added to Monasca Event data as dimensions:
424
+
425
+   * "dimensions.publisher_id": Identifier for the publisher that generated the event. Maps to
426
+     "traits.publisher_id" in Ceilometer event data.
427
+   * "dimensions.user_id": Identifier for user that generated the event. Maps to "traits.user_id" in
428
+     Ceilometer event data.
429
+   * "dimensions.project_id": Identifier of the project that generated the event. Maps to
430
+     "traits.project_id" or "traits.tenant_id" in Ceilometer event data.
431
+
432
+#. hostname is available in the event payload, but its location might differ from event to event. We
433
+   can use Ceilometer's event definitions config to always add a trait called "hostname" to all
434
+   events. e.g. for compute.instance.* will have a trait called "hostname", which grabs data from
435
+   "payload.hostname"
436
+
437
+   .. code-block:: yaml
438
+
439
+   ---
440
+   - event_type: compute.instance.*
441
+
442
+     traits: &instance_traits
443
+      user_id:
444
+        fields: payload.hostname
445
+
446
+#. The proposed new Monasca Ceilometer event publisher will have the ability to submit event
447
+   data in a batch and at a configurable frequency (similar to current samples publisher). The
448
+   event data will be published if the items in the current batch reach their maximum size
449
+   (config setting) or if certain time interval has elapsed since the last publish
450
+   (config setting). This will make sure that the batch does not get huge at the same time
451
+   there is no significant delay in publishing of the events to Monasca Events API.
452
+
453
+#. Monasca Ceilometer event publisher will use service credentials from ceilometer configuration
454
+   file (in "[monasca]" section) to get keystone token.
455
+
456
+   Example "[monasca]" section in ceilometer config file
457
+   .. code-block:: text
458
+
459
+   [monasca]
460
+   service_auth_url = https://localhost:5000/v3
461
+   service_password = secretpassword
462
+   service_username = ceilometer
463
+   service_interface = internalURL
464
+   service_auth_type = password
465
+   # service_project_id may also be used
466
+   service_project_name = admin
467
+   service_domain_name = Default
468
+   service_region_name = RegionOne
469
+
470
+   The publisher will then make a POST request to Monasca Events /v1.0/events REST api[8] to publish
471
+   events to  Monasca Events API.  The URL for the instance of Monasca Events API will be configured
472
+   in the Ceilometer 'events-pipeline.yaml' file.  This has the added advantage of allowing
473
+   different events to be published differently (see Ceilometer pipeline documentation [10]).
474
+
475
+#. "tenant_id" and "user_id" that the notification relates to are available in "payload" section
476
+   of the notification, and these notifications are generated by each service itself.
477
+
478
+   There is no additional "Openstack-operator-agent" like component or functionality required to
479
+   fetch that data from the service and publish to monasca event api on behalf of the original
480
+   tenant.
481
+   Ceilometer publishing pipeline simply extracts these "tenant_id" and "user_id" fields from the
482
+   "payload" and makes those fields available as "tenant_id" and "user_id" traits, which would then
483
+   be mapped to "dimensions.project_id" and "dimensions.user_id" fields in monasca events format.
484
+
485
+   In other words, original "tenant_id" and "user_id" values are available in
486
+   the payload of the notification, and will make its way to "dimensions.tenant_id"
487
+   and "dimensions.user_id" in Monasca Event.
488
+
489
+   Questions/TODO:
490
+   * Q: Do we need to do anything special to handle multi-tenancy in monasca-events api like being
491
+   done for metrics[9] ? Would original user_id and tenant_id in "dimensions.user_id" and
492
+   "dimensions.tenant_id" fields in dimensions serve this purpose?
493
+   * Q: In Ceilometer V2 API (which has been deprecated and removed), when querying data the role
494
+   "admin" could access data for all tenants, whereas a user with "ceilometer-admin" role could
495
+   access only data for a particular tenant. Can we implement something like this for
496
+   monasca-events api when querying for data?
497
+
498
+#. Monasca Ceilometer event publisher will also retry submitting a batch, in case Monasca
499
+   Events API is temporarily unavailable or down. The retry frequency, the number of retries
500
+   and the number of items that can be in the retry batch will also be set via configuration.
501
+
502
+
503
+Alternative Solutions
504
+---------------------
505
+
506
+#. Standalone monasca event agent which reads Openstack notifications published to RabbitMQ
507
+   (on "notification" topic) and publishes them to Monasca Events API.
508
+   Pro:
509
+   * No dependency on Telemetry project.
510
+   * May be simple to develop if leverage the oslo.messaging functionality.
511
+   * Ceilometer has *deprecated* the events functionality in the Stein release. [13]
512
+   Con:
513
+   * Another agent to convince users to install on their systems.
514
+   * Reinventing work already done in the Ceilometer agent.  The OpenStack community already uses Ceilometer and contributes updates when something fails.
515
+   This alternate solution will be detailed in a separate spec, as it is likely
516
+   the long term solution Monasca will need.
517
+
518
+#. Openstack Panko [5] is a event storage and REST API for Ceilometer.
519
+   Pro:
520
+   * An 'official' subproject within Telemetry, so there is some community recognition.
521
+   Con:
522
+   * Its primary storage is in a relational database which has problems with scale.
523
+   * It is not maintained actively and not ready for production. [11]
524
+   * It will be deprecated eventually. [12]
525
+
526
+Data model impact
527
+-----------------
528
+
529
+None
530
+
531
+REST API impact
532
+---------------
533
+
534
+#. We are proposing to tweak the Monasca Event data format by removing and adding following
535
+   fields as mentioned in "Proposed change" section above.
536
+
537
+   Remove fields (JSON path notation): "service", "dimensions.topic",
538
+   "dimensions.hostname" and "event.priority"
539
+
540
+   Add fields (JSON path notation): "dimensions.publisher_id", "dimensions.user_id" and
541
+   "dimensions.project_id"
542
+
543
+   This change will have an impact on Monasca Events API.
544
+
545
+Security impact
546
+---------------
547
+
548
+The proposed Monasca Ceilometer events publisher will collect and publish
549
+Openstack event (notification) data to Monasca API. Openstack notification
550
+data does not have any sensitive data like 'tokens'.
551
+Notifications do contain 'user_id' and 'project_id' fields but do not
552
+contain any Personally Identifiable Information (PII) for the user or
553
+the project.
554
+
555
+
556
+Other end user impact
557
+---------------------
558
+
559
+None.
560
+
561
+Performance Impact
562
+------------------
563
+
564
+#. The number of notifications(events) generated by different services will depend on the capacity
565
+   of the cloud along with the number of resources being created by the users.
566
+
567
+   For example, if there was a large number of compute VM's being created or destroyed it could
568
+   lead to a surge in number of notifications (events) that would have to be published.  Optimum
569
+   configuration options related to say event batch size and event batch interval would have to be
570
+   documented, to reduce any adverse affect on performance.
571
+
572
+#. Monasca Ceilometer publisher runs within Ceilometer Notification Agent component and invoked as a
573
+   last step in its data processing pipeline. It is an additional component that will have to to be
574
+   deployed on all the controller nodes.  We will have to evaluate the performance impact of
575
+   Ceilometer Notification Agent when publishing events to Monasca Events API.
576
+
577
+
578
+Other deployer impact
579
+---------------------
580
+
581
+#. The proposed new Monasca-Ceilometer events publisher will introduce
582
+   few new configuration options like
583
+   * events api endpoint
584
+   * events batch interval
585
+   * events batch size
586
+   * events retry interval
587
+
588
+#. Monasca Ceilometer Events publisher will have to to be added to Ceilometer's
589
+   "[ceilometer.event.publisher]" section  entry_points.txt
590
+
591
+   For example:
592
+
593
+   [ceilometer.event.publisher]
594
+   monasca = ceilometer.publisher.monclient:MonascaEventsPublisher
595
+
596
+#. As part of developing new Monasca Ceilometer Events publisher devstack plugin would be updated to
597
+   add the above configuration changes.
598
+
599
+
600
+Developer impact
601
+----------------
602
+
603
+#. The proposed change to Monasca Event Format will have an impact on existing Monasca Event API,
604
+   since Monasca Event Format will have to be tweaked.  (See REST API Impact section above)
605
+
606
+
607
+Implementation
608
+==============
609
+
610
+Assignee(s)
611
+-----------
612
+
613
+Primary assignee:
614
+  joadavis, aagate
615
+
616
+Other contributors:
617
+  <launchpad-id or None>
618
+
619
+
620
+Work Items
621
+----------
622
+
623
+#. Implement new Monasca Ceilometer Events publisher.
624
+
625
+#. Implement monasca-ceilometer devstack plugin changes to deploy
626
+   new events publisher.
627
+
628
+#. Implement unit tests for Events publisher.
629
+
630
+#. Implement change to Monasca Event format in Monasca Events API.
631
+
632
+
633
+Dependencies
634
+============
635
+
636
+#. Monasca Events API 1.0: https://storyboard.openstack.org/#!/story/2001654
637
+
638
+#. Monasca Ceilometer project: https://github.com/openstack/monasca-ceilometer
639
+
640
+#. Ceilometer Data processing and pipelines:
641
+https://docs.openstack.org/ceilometer/pike/admin/telemetry-data-pipelines.html
642
+
643
+Testing
644
+=======
645
+
646
+#. New Monasca Ceilometer Event publisher unit tests will be added, which can test publishing with
647
+   various config options events batch size, events batch interval, handling retry when Monasca
648
+   Event API is not available.
649
+
650
+#. Adding tempest tests for Monasca Ceilometer events publisher could be looked at as part of
651
+   separate effort.
652
+
653
+   Please note that current Monasca Ceilometer samples publisher does not have tempest tests either
654
+   so having tempest tests for both events and samples publisher could be considered in the future.
655
+
656
+Documentation Impact
657
+====================
658
+
659
+#. New Monasca Events Publisher config options will be documented
660
+
661
+   * events api endpoint
662
+   * events batch interval
663
+   * events batch size
664
+   * events retry interval
665
+
666
+#. Recommended values for each of the config options will also be documented based on the size of
667
+   the cloud and resources for Cloud Operators.
668
+
669
+References
670
+==========
671
+
672
+[1] Monasca Events API 1.0: https://storyboard.openstack.org/#!/story/2001654
673
+
674
+[2] Monasca Ceilometer project: https://github.com/openstack/monasca-ceilometer
675
+
676
+[3] Ceilometer Data processing and pipelines:
677
+https://docs.openstack.org/ceilometer/pike/admin/telemetry-data-pipelines.html
678
+
679
+[4] Ceilometer Events: https://docs.openstack.org/ceilometer/latest/admin/telemetry-events.html
680
+
681
+[5] Openstack Panko: https://github.com/openstack/panko
682
+
683
+[6] Monasca Event Format:
684
+https://github.com/openstack/monasca-events-api/blob/master/doc/api-samples/v1/req_simple_event.json
685
+
686
+[7] Ceilometer System Architecture Diagram:
687
+https://docs.openstack.org/ceilometer/ocata/architecture.html
688
+
689
+[8] Monasca Events POST v1.0 API:
690
+https://github.com/openstack/monasca-events-api/blob/master/api-ref/source/events.inc
691
+
692
+[9] Cross-Tenant Metric Submission:
693
+https://github.com/openstack/monasca-agent/blob/master/docs/MonascaMetrics.md#cross-tenant-metric-submission
694
+
695
+[10] Ceilometer pipeline yaml documentation:
696
+https://docs.openstack.org/ceilometer/latest/admin/telemetry-data-pipelines.html
697
+
698
+[11] No future for Panko or Aodh:
699
+https://julien.danjou.info/lessons-from-openstack-telemetry-deflation/
700
+
701
+[12] Ceilometer Events deprecated means Panko also deprecated:
702
+http://eavesdrop.openstack.org/irclogs/%23openstack-telemetry/%23openstack-telemetry.2018-10-10.log.html
703
+
704
+[13] Ceilometer Events marked as deprecated in Stein:
705
+https://review.openstack.org/#/c/603336/

+ 586
- 0
specs/stein/approved/monasca-events-listener.rst View File

@@ -0,0 +1,586 @@
1
+..
2
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
3
+ License.
4
+
5
+ http://creativecommons.org/licenses/by/3.0/legalcode
6
+
7
+=======================
8
+Monasca Events Listener
9
+=======================
10
+
11
+https://storyboard.openstack.org/#!/story/2003023
12
+
13
+Monasca Events API [1]_ was developed to store event data in Elasticsearch.
14
+A new application could use the Monasca Events API to post an event directly for processing
15
+and storage.
16
+However, a general collection service is needed to capture the existing OpenStack Notifications
17
+already generated by OpenStack Services and pass them to Monasca for storage and processing.
18
+This specification proposes creating a new Monasca Events Listener to capture events and pass
19
+them to Monasca services.
20
+
21
+
22
+Problem description
23
+===================
24
+
25
+All Openstack services generate a lot of notifications or events which contain large amounts of
26
+operational and state information about the service and its resources. Services such as Nova [14]_
27
+already publish to a RabbitMQ topic.
28
+This notification data is not currently available in Monasca.
29
+
30
+Ceilometer data processing pipeline [3]_ provides an extensible mechanism of publishing samples and
31
+events using a custom publisher.  Ceilometer samples represent a quantity that can be measured
32
+(e.g. the size of a volume) and events represent an occurrence of an event and do not have any
33
+associated quantity (e.g. volume was created).  The Telemetry project also has the Panko [5]_
34
+service for indexing and storing these events.
35
+
36
+A previous `spec <../../rocky/not-implemented/monasca-events-publishing.html>`_ was created to
37
+specify an enhancement to Ceilometer to allow collected events to be published to the Monasca
38
+Events API.
39
+However, with the recent deprecation of Ceilometer's Event functionality [13]_, this is no longer
40
+a long-term option.
41
+
42
+This spec is for creating a new Monasca service which would listen for OpenStack Event messages
43
+(often called notifications) and process them through Monasca by consuming the RabbitMQ message
44
+and producing a post to the Monasca Events API with that event.
45
+
46
+This service could use Ceilometer, Vitrage, Watcher, or another service as an example of how to
47
+listen to notifications from OpenStack services such as Nova [14]_.
48
+
49
+It is being proposed to make the Monasca Events Listener service a part of the Monasca Agent
50
+code base and reuse code, including the existing monasca_setup script and config.
51
+
52
+
53
+Use Cases
54
+---------
55
+
56
+#. Openstack notification data would be stored in Elasticsearch via the Monasca services
57
+
58
+   Example sequence:
59
+
60
+   #. Nova completes the creation of a VM
61
+   #. Nova generates a Notification message to oslo.messaging
62
+   #. oslo.messaging posts the message to RabbitMQ
63
+   #. Monasca Event Listener receives the Notification message from RabbitMQ
64
+   #. Monasca Event Listener validates and translates the Notification to a Monasca Events
65
+      API format according to the configuration
66
+   #. Monasca Event Listener publishes formatted Notification to Monasca Events API
67
+   #. Monasca Events API receives and validates formatted Notification
68
+   #. Monasca Events API stores event Notification in configured Elasticsearch instance
69
+
70
+
71
+Proposed change
72
+===============
73
+
74
+#. Monasca Event Listener will be a new service which can be run in a Highly
75
+   Available configuration by running an instance of Monasca Event Listener
76
+   on each Controller node in a cloud.  Each node will listen to OpenStack
77
+   notifications in RabbitMQ and convert notifications to a post to the Monasca
78
+   Events API.  The Monasca Events API then passes the notification on as Kafka
79
+   messages on the 'monevents' topic, where the Monasca Persister will receive
80
+   and store them. By the nature of RabbitMQ clients, the load will be
81
+   distributed between the Monasca Events Listener instances (only one listener
82
+   will process each RabbitMQ message).
83
+
84
+#. Monasca Event Listener will filter messages from OpenStack services based on
85
+   specification of event_types to be collected.  This will reduce 'noise' and
86
+   focus event collection on those events that are deemed valuable.
87
+   This filtering specification can be from a configuration file, or optionally
88
+   could be controlled through a new API implemented as part of the Monasca
89
+   Events API service.
90
+
91
+#. OpenStack Notifications consist of envelope and payload fields.  See [15]_ and [16]_ for
92
+   examples.
93
+
94
+   Example OpenStack Notification data format:
95
+
96
+   .. code-block:: javascript
97
+
98
+        {
99
+            "_context_auth_token": "42630b3ea13242fcad20e0a92d0207f1",
100
+            "_context_domain": null,
101
+            "_context_instance_lock_checked": false,
102
+            "_context_is_admin": true,
103
+            "_context_project_domain": null,
104
+            "_context_project_id": "a4f77",
105
+            "_context_project_name": "admin",
106
+            "_context_quota_class": null,
107
+            "_context_read_deleted": "no",
108
+            "_context_read_only": false,
109
+            "_context_remote_address": "192.168.245.4",
110
+            "_context_request_id": "req-5948338c-f223-4fd8-9249-8769f7a3e460",
111
+            "_context_resource_uuid": null,
112
+            "_context_roles": [
113
+                "monasca-user",
114
+                "admin",
115
+                "KeystoneAdmin"
116
+            ],
117
+            "_context_service_catalog": [
118
+                {
119
+                    "endpoints": [
120
+                        {
121
+                            "adminURL": "http://192.168.245.8:8776/v2/a4f77",
122
+                            "internalURL": "http://192.168.245.8:8776/v2/a4f77",
123
+                            "publicURL": "http://192.168.245.9:8776/v2/a4f77",
124
+                            "region": "region1"
125
+                        }
126
+                    ],
127
+                    "name": "cinderv2",
128
+                    "type": "volumev2"
129
+                },
130
+                {
131
+                    "endpoints": [
132
+                        {
133
+                            "adminURL": "http://192.168.245.8:8776/v1/a4f77",
134
+                            "internalURL": "http://192.168.245.8:8776/v1/a4f77",
135
+                            "publicURL": "http://192.168.245.9:8776/v1/a4f77",
136
+                            "region": "region1"
137
+                        }
138
+                    ],
139
+                    "name": "cinder",
140
+                    "type": "volume"
141
+                }
142
+            ],
143
+            "_context_show_deleted": false,
144
+            "_context_tenant": "a4f77",
145
+            "_context_timestamp": "2015-09-18T20:54:23.468522",
146
+            "_context_user": "be396488c7034811a200a3cb1d103a28",
147
+            "_context_user_domain": null,
148
+            "_context_user_id": "be396488c7034811a200a3cb1d103a28",
149
+            "_context_user_identity": "be396488c7034811a200a3cb1d103a28 a4f77 - - -",
150
+            "_context_user_name": "admin",
151
+            "_unique_id": "ff9699d587bf4283a3c367ab88be1541",
152
+            "event_type": "compute.instance.create.start",
153
+            "message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d",
154
+            "payload": {
155
+                "access_ip_v4": null,
156
+                "access_ip_v6": null,
157
+                "architecture": null,
158
+                "availability_zone": null,
159
+                "cell_name": "",
160
+                "created_at": "2015-09-18 20:55:25+00:00",
161
+                "deleted_at": "",
162
+                "disk_gb": 1,
163
+                "display_name": "testeee",
164
+                "ephemeral_gb": 0,
165
+                "host": null,
166
+                "hostname": "testeee",
167
+                "image_meta": {
168
+                    "base_image_ref": "df0c8",
169
+                    "container_format": "bare",
170
+                    "disk_format": "qcow2",
171
+                    "min_disk": "1",
172
+                    "min_ram": "0"
173
+                },
174
+                "image_name": "glanceaaa3",
175
+                "image_ref_url": "http://192.168.245.5:9292/images/df0c8",
176
+                "instance_flavor_id": "1",
177
+                "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6",
178
+                "instance_type": "m1.tiny",
179
+                "instance_type_id": 4,
180
+                "kernel_id": "",
181
+                "launched_at": "",
182
+                "memory_mb": 512,
183
+                "metadata": {},
184
+                "node": null,
185
+                "os_type": null,
186
+                "progress": "",
187
+                "ramdisk_id": "",
188
+                "reservation_id": "r-1ghilddw",
189
+                "root_gb": 1,
190
+                "state": "building",
191
+                "state_description": "",
192
+                "tenant_id": "a4f77",
193
+                "terminated_at": "",
194
+                "user_id": "be396488c7034811a200a3cb1d103a28",
195
+                "vcpus": 1
196
+            },
197
+            "priority": "INFO",
198
+            "publisher_id": "compute.ccp-compute0001-mgmt",
199
+            "timestamp": "2015-09-18 20:55:37.639023"
200
+        }
201
+
202
+#. All the fields with the prefix of '_context' are the envelope fields, the
203
+   other interesting fields are
204
+
205
+   #. 'message_id' - notification identifier
206
+   #. 'payload' - contains most of the relevant and useful information in JSON format
207
+   #. 'priority' - notification priority
208
+   #. 'publisher_id' - notification publisher
209
+   #. 'timestamp' - notification timestamp
210
+
211
+#. Monasca Event Listener converts the OpenStack notifications to Monasca events format.
212
+   This format will be suitable for Kafka messaging and will match the expected data fields
213
+   of the Monasca Persister.  This conversion and validation should be common between the
214
+   Monasca Event Listener and Monasca Event API.
215
+
216
+#. The Kafka client connection will handle communication issues such as reconnections and
217
+   resending as needed.
218
+
219
+#. Monasca Events API allows a field called 'payload' which can be in an arbitrary
220
+   nested JSON format.
221
+   TODO: mappings
222
+
223
+   Example Monasca Event Format:
224
+
225
+   .. code-block:: javascript
226
+
227
+        events: [
228
+        {
229
+          dimensions": {
230
+                "service": "compute.ccp-compute0001-mgmt",
231
+                "topic": "notification.sample",
232
+                "hostname": "nova-compute:compute
233
+          },
234
+          event: {
235
+
236
+                  "event_type": "compute.instance.create.start",
237
+
238
+                  "payload": {
239
+                       "access_ip_v4": null,
240
+                       "access_ip_v6": null,
241
+                       "architecture": null,
242
+                       "availability_zone": null,
243
+                       "cell_name": "",
244
+                       "created_at": "2015-09-18 20:55:25+00:00",
245
+                       "deleted_at": "",
246
+                       "disk_gb": 1,
247
+                       "display_name": "testeee",
248
+                       "ephemeral_gb": 0,
249
+                       "host": null,
250
+                       "hostname": "testeee",
251
+                       "image_meta": {
252
+                           "base_image_ref": "df0c8",
253
+                           "container_format": "bare",
254
+                           "disk_format": "qcow2",
255
+                           "min_disk": "1",
256
+                           "min_ram": "0"
257
+                       },
258
+                      "image_name": "glanceaaa3",
259
+                      "image_ref_url": "http://192.168.245.5:9292/images/df0c8",
260
+                      "instance_flavor_id": "1",
261
+                      "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6",
262
+                      "instance_type": "m1.tiny",
263
+                      "instance_type_id": 4,
264
+                      "kernel_id": "",
265
+                      "launched_at": "",
266
+                      "memory_mb": 512,
267
+                      "metadata": {},
268
+                      "node": null,
269
+                      "os_type": null,
270
+                      "progress": "",
271
+                      "ramdisk_id": "",
272
+                      "reservation_id": "r-1ghilddw",
273
+                      "root_gb": 1,
274
+                      "state": "building",
275
+                      "state_description": "",
276
+                      "tenant_id": "a4f77",
277
+                      "terminated_at": "",
278
+                      "user_id": "be396488c7034811a200a3cb1d103a28",
279
+                      "vcpus": 1
280
+                      }
281
+                 },
282
+            publisher_id: "compute.ccp-compute0001-mgmt",
283
+            priority: "INFO"
284
+         }
285
+        ]
286
+
287
+#. Following fields in Monasca Event data may not be available in the OpenStack notification
288
+   data format:
289
+
290
+   * "service"
291
+   * "dimensions.topic"
292
+   * "event.priority"
293
+
294
+   We are proposing removing these fields from Monasca Event format (will be done as a separate
295
+   spec/implementation process) for the following reasons:
296
+
297
+   "service": Currently OpenStack notifications do not specify a service, that
298
+   generated the notification in a consistent way. It might be possible to create an external
299
+   mapping file which maps event name to a service but its hard to maintain such mapping over a
300
+   period of time.
301
+
302
+   "dimensions.topic": This field is not available in the source OpenStack notification.
303
+   However, the Monasca Event Listener may be able to save the RabbitMQ topic that the notification
304
+   was collected from.  In that case, this field should be used.
305
+
306
+   "event.priority": This field is not currently available in Ceilometer Event format. It is
307
+   available in the source OpenStack notification. Note: If we think this field can be useful we can
308
+   propose adding it to the Monasca Event Listener format.
309
+
310
+#. Following new fields will be added to Monasca Event data as dimensions:
311
+
312
+   * "dimensions.publisher_id": Identifier for the publisher that generated the event.
313
+   * "dimensions.user_id": Identifier for user that generated the event.
314
+   * "dimensions.project_id": Identifier of the project that generated the event.
315
+
316
+#. 'hostname' is available in the event payload, but its location might differ from event to event.
317
+
318
+#. The proposed new Monasca Event Listener will have the ability to submit event
319
+   data in a batch and at a configurable frequency (similar to current samples publisher). The
320
+   event data will be published if the items in the current batch reach their maximum size
321
+   (config setting) or if certain time interval has elapsed since the last publish
322
+   (config setting). This will make sure that the batch does not get huge at the same time
323
+   there is no significant delay in publishing of the events to Monasca Events API.
324
+
325
+#. Monasca Event Listener will have a configuration file to configure connection information for
326
+   both RabbitMQ and Monasca Events API.
327
+
328
+#. The "tenant_id" and "user_id" that the notification relates to are available in "payload"
329
+   section of the notification, and these notifications are generated by each service itself.
330
+
331
+   There is no additional "OpenStack-operator-agent" like component or functionality required to
332
+   fetch that data from the service and publish to monasca event api on behalf of the original
333
+   tenant.
334
+   (Ceilometer publishing pipeline simply extracts these "tenant_id" and "user_id" fields from the
335
+   "payload" and makes those fields available as "tenant_id" and "user_id" traits, which would then
336
+   be mapped to "dimensions.project_id" and "dimensions.user_id" fields in monasca events format.)
337
+
338
+   In other words, original "tenant_id" and "user_id" values are available in
339
+   the payload of the notification, and will make its way to "dimensions.tenant_id"
340
+   and "dimensions.user_id" in Monasca Event.
341
+
342
+   Questions/TODO:
343
+
344
+   * Q: Do we need to do anything special to handle multi-tenancy in monasca-events api like being
345
+     done for metrics [9]_ ? Would original user_id and tenant_id in "dimensions.user_id" and
346
+     "dimensions.tenant_id" fields in dimensions serve this purpose?
347
+
348
+     * A: Monasca Events Listener can start with sending all events to a single "admin" tenant
349
+       and if required in the future some other process could copy select metrics back to tenant
350
+       projects.
351
+
352
+   * Q: In Ceilometer V2 API (which has been deprecated and removed), when querying data the role
353
+     "admin" could access data for all tenants, whereas a user with "ceilometer-admin" role could
354
+     access only data for a particular tenant. Can we implement something like this for
355
+     monasca-events api when querying for data?
356
+
357
+     * A: In Monasca API every request is scoped to the project, so there is no equivalent of
358
+       Ceilometer's "admin" role to query data for all projects.  So placing all events in to
359
+       an "admin" project may be the best approach.
360
+
361
+   * Q: How should services which generate notifications but do not include a tenant_id be
362
+     handled?  For example Keystone [16]_.
363
+     How does Ceilometer handle such events?
364
+
365
+     * A: If all events are in an "admin" project then admin metrics like shared ceph cluster
366
+       load or provider network load can be copied back to tenants so they may understand how
367
+       infrastructure is affecting their workload.
368
+
369
+   * Note: Configuration of Elasticsearch cluster is out of scope for this spec. If needed
370
+     could assign a separate Elasticsearch cluster to the events API to avoid overloads.
371
+
372
+
373
+Alternative Solutions
374
+---------------------
375
+
376
+#. Reuse the Ceilometer functionality to collect and publish events to the
377
+   Monasca Events API.  While this may be less work initially, Ceilometer has
378
+   deprecated the Events functionality as of Stein. [13]_
379
+
380
+#. An alternate Events Listener was proposed that would listen for RabbitMQ events
381
+   then publish them directly to the 'monevents' topic in Kafka.  Discussion on this
382
+   can be seen in the git history for this document and in IRC logs [18]_.
383
+
384
+   Pro:
385
+
386
+   * A much simpler approach, more efficient that HTTP hop through another service.
387
+   * No need for batching in service, as RabbitMQ and Kafka clients would handle fast
388
+     throughput and short network interruptions.
389
+
390
+   Con:
391
+
392
+   * Nova Cells v2 each have their own RabbitMQ instances. While most deployments
393
+     would likely have a centralized RabbitMQ, it is not required in the documented
394
+     architecture.
395
+   * Regions may also cause separation of RabbitMQ instances that need to be monitored.
396
+   * While it might be possible to have a service/agent in each Cell publish back to
397
+     a centralized to Kafka directly, our authentication and networking for Kafka was
398
+     not designed to support that.
399
+
400
+#. OpenStack Panko [5]_ is a event storage and REST API for Ceilometer and could
401
+   be used instead of a Monasca solution.
402
+
403
+   Pro:
404
+
405
+   * An 'official' subproject within Telemetry, so there is some community recognition.
406
+
407
+   Con:
408
+
409
+   * Its primary storage is in a relational database which has problems with scale.
410
+   * It is not maintained actively and not ready for production. [11]_
411
+   * It will be deprecated eventually. [12]_
412
+
413
+
414
+Data model impact
415
+-----------------
416
+
417
+None
418
+
419
+REST API impact
420
+---------------
421
+
422
+#. We are proposing to tweak the Monasca Event data format by removing and adding following
423
+   fields as mentioned in "Proposed change" section above.
424
+
425
+   Remove fields or make them optional (JSON path notation): "service", "dimensions.topic",
426
+   "dimensions.hostname" and "event.priority"
427
+
428
+   Add fields (JSON path notation): "dimensions.publisher_id", "dimensions.user_id" and
429
+   "dimensions.project_id"
430
+
431
+   This change will have an impact on Monasca Events API.
432
+
433
+Security impact
434
+---------------
435
+
436
+The proposed Monasca Event Listener will collect and publish OpenStack event
437
+(notification) data to Monasca Events API. OpenStack notification data does not
438
+have any sensitive data like 'tokens'.
439
+Notifications do contain 'user_id' and 'project_id' fields but do not
440
+contain any Personally Identifiable Information (PII) for the user or
441
+the project.
442
+
443
+
444
+Other end user impact
445
+---------------------
446
+
447
+None.
448
+
449
+Performance Impact
450
+------------------
451
+
452
+#. The number of notifications(events) generated by different services will depend on the capacity
453
+   of the cloud along with the number of resources being created by the users.
454
+
455
+   For example, if there was a large number of compute VM's being created or destroyed it could
456
+   lead to a surge in number of notifications (events) that would have to be published.  Optimum
457
+   configuration options related to say event batch size and event batch interval would have to be
458
+   documented, to reduce any adverse affect on performance.
459
+
460
+#. The proposed Monasca Event Listener is a new service, so performance is unknown.  However, the
461
+   Monasca API has been shown to have a high performance throughput.
462
+
463
+
464
+Other deployer impact
465
+---------------------
466
+
467
+#. The proposed Monasca Event Listener will introduce a
468
+   few new configuration options like
469
+
470
+   * RabbitMQ connection information
471
+   * Monasca Events API endpoint URL
472
+   * events batch interval
473
+   * events batch size
474
+   * events retry interval
475
+   * Keystone credentials for obtaining a token
476
+   * Conversion options for OpenStack notifications to the Monasca Event format.  This may
477
+     be stored in separate pipeline configuration files, similar to how transform specs
478
+     are configured in Monasca Transform.
479
+
480
+#. As part of developing new Monasca Event Listener devstack plugin would be
481
+   updated to add the above configuration changes.
482
+
483
+
484
+Developer impact
485
+----------------
486
+
487
+#. The proposed change to Monasca Event Format will have an impact on existing Monasca Event API,
488
+   since Monasca Event Format will have to be tweaked.  (See REST API Impact section above)
489
+
490
+Implementation
491
+==============
492
+
493
+Assignee(s)
494
+-----------
495
+
496
+Primary assignee:
497
+  joadavis, aagate
498
+
499
+Other contributors:
500
+  <launchpad-id or None>
501
+
502
+
503
+Work Items
504
+----------
505
+
506
+#. Implement new Monasca Event Listener.
507
+
508
+   * Connection to RabbitMQ for OpenStack Notifications
509
+   * Add filtering of notifications for configured event_types
510
+
511
+     * Specification in configuration file
512
+     * (Optional) Creation of a new API to configure event_type subscriptions
513
+
514
+   * Validation of OpenStack Notification data and format
515
+   * Conversion of data format to meet Monasca Events requirements
516
+   * Publishing to Monasca Events API
517
+   * Configuration of conversion specifications per-event type
518
+
519
+#. Implement monasca devstack plugin changes to deploy
520
+   new Events Listener service.
521
+
522
+#. Implement unit tests for Monasca Event Listener.
523
+
524
+
525
+Dependencies
526
+============
527
+
528
+None
529
+
530
+Testing
531
+=======
532
+
533
+#. New Monasca Event Listener unit tests will be added, which can test publishing with
534
+   various config options events batch size, events batch interval, handling retry when Monasca
535
+   Event API is not available.
536
+
537
+#. Adding tempest tests for Monasca Event Listener could be looked at as part of
538
+   separate effort.
539
+
540
+
541
+Documentation Impact
542
+====================
543
+
544
+#. New Monasca Event Listener config options will be documented
545
+
546
+#. Recommended values for each of the config options will also be documented based on the size of
547
+   the cloud and resources for Cloud Operators.
548
+
549
+References
550
+==========
551
+
552
+.. [1] Monasca Events API 1.0: https://git.openstack.org/cgit/openstack/monasca-events-api/
553
+
554
+[2] Monasca Ceilometer project: https://github.com/openstack/monasca-ceilometer
555
+
556
+.. [3] Ceilometer Data processing and pipelines: https://docs.openstack.org/ceilometer/pike/admin/telemetry-data-pipelines.html
557
+
558
+[4] Ceilometer Events: https://docs.openstack.org/ceilometer/latest/admin/telemetry-events.html
559
+
560
+.. [5] Openstack Panko: https://github.com/openstack/panko
561
+
562
+[6] Monasca Event Format: https://github.com/openstack/monasca-events-api/blob/master/doc/api-samples/v1/req_simple_event.json
563
+
564
+[7] Ceilometer System Architecture Diagram: https://docs.openstack.org/ceilometer/ocata/architecture.html
565
+
566
+[8] Monasca Events POST v1.0 API: https://github.com/openstack/monasca-events-api/blob/master/api-ref/source/events.inc
567
+
568
+.. [9] Cross-Tenant Metric Submission: https://github.com/openstack/monasca-agent/blob/master/docs/MonascaMetrics.md#cross-tenant-metric-submission
569
+
570
+[10] Ceilometer pipeline yaml documentation: https://docs.openstack.org/ceilometer/latest/admin/telemetry-data-pipelines.html
571
+
572
+.. [11] No future for Panko or Aodh: https://julien.danjou.info/lessons-from-openstack-telemetry-deflation/
573
+
574
+.. [12] Ceilometer Events deprecated means Panko also deprecated: http://eavesdrop.openstack.org/irclogs/%23openstack-telemetry/%23openstack-telemetry.2018-10-10.log.html
575
+
576
+.. [13] Ceilometer Events marked as deprecated in Stein: https://review.openstack.org/#/c/603336/
577
+
578
+.. [14] Nova notification version update lists services effected (see "Deprecating legacy notifications"): https://etherpad.openstack.org/p/nova-ptg-stein
579
+
580
+.. [15] Nova notification reference: https://docs.openstack.org/nova/latest/reference/notifications.html#existing-versioned-notifications
581
+
582
+.. [16] Keystone notification reference: https://docs.openstack.org/keystone/latest/advanced-topics/event_notifications.html#example-notification-project-create
583
+
584
+[17] Monasca Events API publisher to Kafka: https://github.com/openstack/monasca-events-api/blob/master/monasca_events_api/app/common/events_publisher.py
585
+
586
+.. [18] Monasca IRC meeting Dec 15, 2018: http://eavesdrop.openstack.org/meetings/monasca/2018/monasca.2018-12-12-15.00.log.html

Loading…
Cancel
Save