Spec for sending polled meters to notification agent
As requested, this is a spec that describes adjusting the polling agents such that they no longer perform transformations or other sink-side handling described by pipeline.yaml but instead merely push the results of their polling to the notification agent. blueprint pollsters-no-transform Change-Id: I61f6c17f2e561a5fa738f149a6b1f4d6f5bd5ea2
This commit is contained in:
Chris Dent
parent
048c949528
commit
a649270528
187
specs/liberty/pollsters-no-transform.rst
Normal file
187
specs/liberty/pollsters-no-transform.rst
Normal file
@ -0,0 +1,187 @@
|
||||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
=====================
|
||||
Pollsters No Tranform
|
||||
=====================
|
||||
|
||||
https://blueprints.launchpad.net/ceilometer/+spec/pollsters-no-transform
|
||||
|
||||
The ceilometer polling agents currently perform two roles: They poll various
|
||||
services to generate samples and they optionally transform those samples, using
|
||||
pipelines defined by ``pipeline.yaml``, to compose other samples eventually
|
||||
publishing them as measurements. This latter transformation functionality raises
|
||||
the complexity of the pollster code. An alternative would be for the pollsters
|
||||
to send notifications for the meters it creates and have them be processed
|
||||
by the notification agent (which has a transformation pipeline of its own) and
|
||||
then be published.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Including the transformation pipeline in the pollsters increases complexity
|
||||
in the pollsters and duplicates functionality found in the notification agent.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
One way to resolve this complexity is to not do transformations in the
|
||||
pollsters. Instead when new samples are polled, format them as notifications
|
||||
and push them onto the notification bus to be retreived by the notification
|
||||
agent. The pipeline within that agent will then do any required transformations.
|
||||
|
||||
Besides clarifying the focus of the polling agents it is likely this change
|
||||
will make it easier to test and build performance improvements both in code
|
||||
and in deployments. This is murky speculation but it is challenging to test the
|
||||
theory without doing the work.
|
||||
|
||||
We also get an opportunity to review and audit the polling code, which is a
|
||||
useful thing to do on a regular basis.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
The primary alternative is to leave things as they are. Obviously if we do this
|
||||
we lose the benefits described above.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
The proposal does not impact the structure of the data, but it would
|
||||
impact the flow of the data. Testing will be required to see what
|
||||
kind of impact this will have on the timeliness of data acquisition.
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
No direct impact on the resources presented by the API.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
None beyond existing concerns.
|
||||
|
||||
Pipeline impact
|
||||
---------------
|
||||
|
||||
Describing pipeline impact is somewhat complex because the word pipeline
|
||||
can be used:
|
||||
|
||||
* to describe the overall process whereby a suite of samples is
|
||||
gathered and transformed
|
||||
* to identify the pairing of a ``source`` with a ``sink`` in the
|
||||
``pipeline.yaml`` file
|
||||
* to point at the file ``pipeline.yaml`` which configures many
|
||||
aspects of gathering, transforming and publishing samples as well as
|
||||
transforming and publishing samples that have been produced from
|
||||
notifications
|
||||
|
||||
In this change transformation and publishing will be localized in the
|
||||
notification agents. Gathering and producing of samples will be done by
|
||||
polling agents, the api server (via posting meters) and the notification
|
||||
agents.
|
||||
|
||||
In order to ease migration the polling agents will continue to use
|
||||
the ``pipeline.yaml`` file to configure the sources they will poll
|
||||
and the intervals at which that polling will be done. The same file
|
||||
may be used, or a new file without any sinks. The idea is that both
|
||||
should work.
|
||||
|
||||
The most significant difference is that the existing validation
|
||||
which confirms that there is a pairing between ``source`` and
|
||||
``sink`` will be removed. The polling agents may poll whatever
|
||||
they like. If the notification agent doesn't want the samples that
|
||||
are generated it can simply drop them on the floor. This provides
|
||||
two benefits:
|
||||
|
||||
* it makes the pipeline (on the polling side) less complex
|
||||
* it focuses authority in the notification agent
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None.
|
||||
|
||||
Performance/Scalability Impacts
|
||||
-------------------------------
|
||||
|
||||
There is hope that this will decrease the footprint of the pollsters making it
|
||||
easier to deploy many of them, thus increasing the overall performance of the
|
||||
polling system.
|
||||
|
||||
On the negative side there could be increased load on both the messaging and the
|
||||
notification agents but this could be compensated for by:
|
||||
|
||||
* Having the pollsters publish their notifications on their own topic or even
|
||||
bus.
|
||||
* Deploying more notification agents (in a horizontal fashion).
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
Deployers will need to be aware of their options for deploying suitable numbers
|
||||
of notification agents.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
Testing sample transformation from point of sample creation to point of storage
|
||||
will become more complex as it will traverse several services. With the advent
|
||||
of in-tree functional testing there is a place where this testing can and
|
||||
should be done. Having those tests will be very helpful in the long run despite
|
||||
the initial pain in setting them up.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
chdent
|
||||
|
||||
Other contributors:
|
||||
you and all your friends
|
||||
|
||||
Ongoing maintainer:
|
||||
chdent
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
* Ensure existence of robust tests of both polling and notification agents.
|
||||
* Replace source and sink handling in the pollster code with a simpler read of
|
||||
the source descriptions in pipeline.yaml (leaving open the future option of
|
||||
using a different file). Create samples will be republished as notifications.
|
||||
* Update documenation to reflect new architecture.
|
||||
|
||||
Future lifecycle
|
||||
================
|
||||
|
||||
The members of the Telemetry program will do any required ongoing
|
||||
maintenance for this feature.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
No new dependencies.
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Testing as described above.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
Deployment documentation will need to be updated to reflect the connection
|
||||
between the polling and notification agents, the option to use a custom
|
||||
messaging bus and other opportunities for customization.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
None at this time.
|
||||
Loading…
Reference in New Issue
Block a user