PaaS Event Format for Ceilometer Collection
Provides a payload specification to standadardize the format of PaaS integrations so that Ceilometer can consume metering data from any conforming application without custom integration. DocImpact Change-Id: Icbb141b7aaf402d18d0a7e9e08a7b816d9833e6c
This commit is contained in:
Phil Neal
parent
e225b5213b
commit
3b20308d52
424
specs/juno/paas-event-format-for-ceilometer.rst
Normal file
424
specs/juno/paas-event-format-for-ceilometer.rst
Normal file
@ -0,0 +1,424 @@
|
|||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
=================
|
||||||
|
PaaS Event Format
|
||||||
|
=================
|
||||||
|
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/ceilometer/+spec/paas-event-format-for-ceilometer
|
||||||
|
|
||||||
|
The general mechanism for setting up Ceilometer collection for a new service
|
||||||
|
has been to either create a targeted agent or use the existing service APIs to
|
||||||
|
meter these systems. While this approach has worked for the existing set of
|
||||||
|
services, it will run into a several problems as the number of PaaS and SaaS
|
||||||
|
services that need to be metered expand. This blueprint provides a proposal for
|
||||||
|
streamlining the format of integrations so that Ceilometer can consume
|
||||||
|
metering data from any conforming PaaS application without doing any custom
|
||||||
|
integration.
|
||||||
|
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
There are a number of PaaS services that are currently in development
|
||||||
|
and a growing number of applications running on top of OpenStack
|
||||||
|
infrastructure. Many of these new applications will need to be metered
|
||||||
|
for a variety of reasons: internal systems management, license
|
||||||
|
management, end customer billing. Ceilometer currently supports the
|
||||||
|
collection of metering information for many infrastructure-level services,
|
||||||
|
however it won't be viable to do custom integrations for the
|
||||||
|
potentially hundreds (and eventually thousands) of services that may be
|
||||||
|
hosted in OpenStack. We want to head off that issue by defining a
|
||||||
|
universal PaaS notification payload so Ceilometer can consume PaaS
|
||||||
|
notifications without significant integration work.
|
||||||
|
|
||||||
|
A few use cases come to mind:
|
||||||
|
**Database as a Service**
|
||||||
|
|
||||||
|
Database as a Service (DBaaS), i.e. Trove, has an architecture where a service
|
||||||
|
controller manages special Nova compute instances on behalf of the customer.
|
||||||
|
From their perspective they are running instances of a database, not compute
|
||||||
|
instances. Metering and billing will be based on hours that the database has
|
||||||
|
been running, not necessarily how long the hosting instance has been run. This
|
||||||
|
requires a unique set of metering records to be generated and stored to enable
|
||||||
|
usage tracking and billing of the individual database instances.
|
||||||
|
|
||||||
|
**DNS as a Service**
|
||||||
|
|
||||||
|
DNS as a service runs on a similar service controller architecture to DBaaS. In
|
||||||
|
this case, the meters that need to be measured are the existence of a DNS zone
|
||||||
|
and the number of DNS queries that have been served. Once again these are
|
||||||
|
application level meters that do not correlate directly to the host instances
|
||||||
|
that are running. Queries could be served by a variety of hosts and a DNS
|
||||||
|
zone's existence does not depend on a specific compute instance. Application
|
||||||
|
level metrics must be tracked by the DNS service and reported out so that these
|
||||||
|
systems can be tracked.
|
||||||
|
|
||||||
|
**Load Balancing as a Service**
|
||||||
|
|
||||||
|
A load balancer is a logical system that consists of some number of independent
|
||||||
|
backends that host the load balancing software. Meters that must be measured
|
||||||
|
would be things like the existence of a load balancer VIP, its members, and the
|
||||||
|
amount of data that is being transferred through the system. Once again, this
|
||||||
|
does not directly correlate to the underlying infrastructure but must be
|
||||||
|
reported at the application level
|
||||||
|
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
We need to standardize the content of the PaaS payload to include a set of data
|
||||||
|
that will include data needed for metering and billing. This essentially
|
||||||
|
constitutes a documentation change and a set of tests to verify key fields.
|
||||||
|
|
||||||
|
The following is the proposed schema:
|
||||||
|
::
|
||||||
|
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"Field": "event_type",
|
||||||
|
"Type": "enumeration",
|
||||||
|
"Description": "for event type records, this describes the actual event that occurred",
|
||||||
|
"Compliance": "required for events",
|
||||||
|
"Notes": "depends on service, defaults to create, exists, delete"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "timestamp",
|
||||||
|
"Type": "UTC DateTime",
|
||||||
|
"Description": "timestamp of when this event was generated at the resource",
|
||||||
|
"Compliance": "required",
|
||||||
|
"Notes": "ISO 8859 date YYYY-mm-ddThh:mm:ss"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "message_id",
|
||||||
|
"Type": "String",
|
||||||
|
"Description": "unique identifier for event",
|
||||||
|
"Compliance": "required",
|
||||||
|
"Notes": ""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"payload": [
|
||||||
|
{
|
||||||
|
"Field": "version",
|
||||||
|
"Type": "String",
|
||||||
|
"Description": "Version of event format",
|
||||||
|
"Compliance": "required",
|
||||||
|
"Notes": ""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "audit_period_beginning",
|
||||||
|
"Type": "UTC DateTime",
|
||||||
|
"Description": "Represents start time for metrics reported",
|
||||||
|
"Compliance": "required",
|
||||||
|
"Notes": "Format ISO 8859 date YYYY-mm-ddThh:mm:ss"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "audit_period_ending",
|
||||||
|
"Type": "UTC DateTime",
|
||||||
|
"Description": "Represents end time for metrics reported",
|
||||||
|
"Compliance": "required",
|
||||||
|
"Notes": "Format ISO 8859 date YYYY-mm-ddThh:mm:ss"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "record_type",
|
||||||
|
"Type": "enumeration ",
|
||||||
|
"Values": {
|
||||||
|
"event": "events describe some kind of state change in the service",
|
||||||
|
"quantity": "quantity describes a usage metric value"
|
||||||
|
},
|
||||||
|
"Compliance": "required",
|
||||||
|
"Notes": ""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "project_id",
|
||||||
|
"Type": "UUID",
|
||||||
|
"Description": "Keystone project_id identifies the owner of
|
||||||
|
the service instance",
|
||||||
|
"Compliance": "required",
|
||||||
|
"Notes": ""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "user_id",
|
||||||
|
"Type": "UUID",
|
||||||
|
"Description": "Keystone user_id identifies specific user",
|
||||||
|
"Compliance": "optional",
|
||||||
|
"Notes": ""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "service_id",
|
||||||
|
"Type": "UUID",
|
||||||
|
"Description": "Keystone service_id uniquely identifies a service",
|
||||||
|
"Compliance": "required",
|
||||||
|
"Notes": ""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "service_type",
|
||||||
|
"Type": "String",
|
||||||
|
"Description": "Keystone service_type uniquely identifies a service",
|
||||||
|
"Compliance": "required",
|
||||||
|
"Notes": ""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "instance_id",
|
||||||
|
"Type": "UUID",
|
||||||
|
"Description": "uniquely identifies an instance of the service",
|
||||||
|
"Compliance": "required",
|
||||||
|
"Notes": "assuming instance level reporting"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "display_name",
|
||||||
|
"Type": "String",
|
||||||
|
"Description": "text description of service",
|
||||||
|
"Compliance": "optional",
|
||||||
|
"Notes": "used if customer names instances"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "instance_type_id",
|
||||||
|
"Type": "enumeration",
|
||||||
|
"Description": "used to describe variations of a service",
|
||||||
|
"Compliance": "required",
|
||||||
|
"Notes": "needed if variations of service have different prices or
|
||||||
|
need to be broken out separately"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "instance_type",
|
||||||
|
"Type": "String",
|
||||||
|
"Description": "text description of service variations",
|
||||||
|
"Compliance": "optional",
|
||||||
|
"Notes": ""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "availability_zone",
|
||||||
|
"Type": "String",
|
||||||
|
"Description": "where the service is deployed",
|
||||||
|
"Compliance": "optional",
|
||||||
|
"Notes": "required if service is deployed at an AZ level"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "region",
|
||||||
|
"Type": "String",
|
||||||
|
"Description": "data center that the service is deployed in",
|
||||||
|
"Compliance": "optional",
|
||||||
|
"Notes": "required if service is billed at a regional level"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "state",
|
||||||
|
"Type": "enumeration",
|
||||||
|
"Description": "status of the service at the time of record generation",
|
||||||
|
"Compliance": "optional",
|
||||||
|
"Notes": "required for existence events"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "state_description",
|
||||||
|
"Type": "String",
|
||||||
|
"Description": "text description of state of service",
|
||||||
|
"Compliance": "",
|
||||||
|
"Notes": ""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "license_code",
|
||||||
|
"Type": "enumeration",
|
||||||
|
"Description": "value that describes a specific license model",
|
||||||
|
"Compliance": "optional",
|
||||||
|
"Notes": "this field is TBD depending on dev_pay design work"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"metrics": [
|
||||||
|
{
|
||||||
|
"Field": "metric_name",
|
||||||
|
"Type": "String",
|
||||||
|
"Description": "unique name for the metric that is represented
|
||||||
|
in this record",
|
||||||
|
"Compliance": "required",
|
||||||
|
"Notes": ""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "metric_type",
|
||||||
|
"Type": "enumeration",
|
||||||
|
"Description": "gauge, cumulative, delta",
|
||||||
|
"Compliance": "required",
|
||||||
|
"Notes": "describes the behavior of the metric, from ceilometer"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "metric_value",
|
||||||
|
"Type": "Float",
|
||||||
|
"Description": "value of metric for quantity type records",
|
||||||
|
"Compliance": "required for quantities",
|
||||||
|
"Notes": ""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Field": "metric_units",
|
||||||
|
"Type": "enumeration",
|
||||||
|
"Description": "describes the units for the quantity",
|
||||||
|
"Compliance": "optional",
|
||||||
|
"Notes": ""
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
Note: Required means that it must be present and described as in the specification.
|
||||||
|
Optional means it can be present or not, but if present it must be described as
|
||||||
|
in the specifications.
|
||||||
|
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
The alternative approach to this standardization is to allow PaaS service
|
||||||
|
providers to determine the content of the notifications and the associated
|
||||||
|
plugins, requiring missing data to be addressed post implementation through
|
||||||
|
patching.
|
||||||
|
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
* What new data objects and/or database schema changes is this going to
|
||||||
|
require?
|
||||||
|
|
||||||
|
This format should fit within the existing schema.
|
||||||
|
|
||||||
|
|
||||||
|
* What database migrations will accompany this change, treating the SQLAlchemy
|
||||||
|
and NoSQL cases separately.
|
||||||
|
|
||||||
|
No DB migrations will be required.
|
||||||
|
|
||||||
|
* Will this add to the on-the-fly data massaging cruft that we've accreted
|
||||||
|
over time?
|
||||||
|
|
||||||
|
The purpose of this change is to avoid the explosion of cruft that could
|
||||||
|
potentially be generated as PaaS services implement variations on the
|
||||||
|
notification payload.
|
||||||
|
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
The API should transparently handle the PaaS data, there should be no API
|
||||||
|
impact.
|
||||||
|
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
As this BP only specifies a standard format for PaaS samples (and one that
|
||||||
|
closely resembles the current sample content) current content of Ceilometer
|
||||||
|
samples) there is no security impact.
|
||||||
|
|
||||||
|
|
||||||
|
Pipeline impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
The implementation of the PaaS collection agent is documented in a separate BP;
|
||||||
|
this standardization requires no pipeline changes in itself.
|
||||||
|
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
|
||||||
|
Performance/Scalability Impacts
|
||||||
|
-------------------------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
There should be a positive impact to deployers from implementing this standard:
|
||||||
|
unifying the PaaS content up-front will reduce the likelihood of missing data
|
||||||
|
required for downstream processing.
|
||||||
|
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
There is the increase in up-front effort to define, for each PaaS service being
|
||||||
|
implemented, a set of data that fulfills the standard. There is also an
|
||||||
|
implication that there may be evolution of this standard, and therefore
|
||||||
|
evolution of the documentation/tests.
|
||||||
|
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
This is tied to the work documented in the PaaS Event Collection blueprint and
|
||||||
|
will be implemented in parallel with that. That said, there's certainly benefit
|
||||||
|
in opening this up to those who have an interest in PaaS events, especially if
|
||||||
|
there are data elements that we haven't considered here.
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
nealph (phil.neal@hp.com)
|
||||||
|
|
||||||
|
Other contributors:
|
||||||
|
fabiog (fabio.gianetti@hp.com)
|
||||||
|
|
||||||
|
Ongoing maintainer:
|
||||||
|
nealph (phil.neal@hp.com)
|
||||||
|
|
||||||
|
It's likely that the standard will expand: the intent here is to define and
|
||||||
|
clearly document the core elements required. For that reason, I expect
|
||||||
|
contributions to this documentation will be done by others beside myself.
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
1. Identify impacts to existing services (if any) and submit appropriate
|
||||||
|
requests (via Launchpad)
|
||||||
|
|
||||||
|
2. Create documentation changes
|
||||||
|
|
||||||
|
|
||||||
|
Future lifecycle
|
||||||
|
================
|
||||||
|
|
||||||
|
I expect there will be pressure to expand the specification in future release
|
||||||
|
cycles.
|
||||||
|
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
There is some implication for testing, since we want to check for existence of
|
||||||
|
the fields defined in the specification.
|
||||||
|
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
Documentation for this change is the core deliverable. While it won't require a
|
||||||
|
lot of documentation, the readability and descriptiveness of the documentation
|
||||||
|
is critical. We expect to leverage much of the content at
|
||||||
|
https://wiki.openstack.org/wiki/Ceilometer/blueprints/StandardPaaSEventFormat
|
||||||
|
as well as add updated event payload examples.
|
||||||
|
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
The full blueprint for this change is available at:
|
||||||
|
https://wiki.openstack.org/wiki/Ceilometer/blueprints/StandardPaaSEventFormat
|
||||||
|
|
||||||
|
The related blueprint outlining changes to the PaaS event collection mechanism
|
||||||
|
(that benefits from on this standard being in place)
|
||||||
|
is available at:
|
||||||
|
https://wiki.openstack.org/wiki/Ceilometer/blueprints/PaaSEventUsageCollection
|
||||||
Loading…
Reference in New Issue
Block a user