delete verbose/redundant/deprecated text
the docs are a bit wordy in some places. our docs are confusing enough: - remove notification > polling in architecture page - remove redundant Events filtering from event_types - move shuffle/batch options to customisation page - add http publisher - remove db choices from architecture page, point to dbreco page - remove db background referencing single database route - remove warnings to say db might change, use api (obvious) - remove stuff about Ceilometer is part Openstack but not tied to it - move hbase configuration under common hbase section in install - remove tox compatibility comments since it's > 2 years - remove random non-devstack/historical text from devstack page - remove default cinder/nova configurations from devstack page Change-Id: Ifdbeee358cfcb5d3736608e39067ff2859cc0ed6
This commit is contained in:
gordon chung
parent
bdb291739a
commit
7fe87e904f
@ -67,27 +67,22 @@ How is data collected?
|
||||
This is a representation of how the collectors and agents gather data from
|
||||
multiple sources.
|
||||
|
||||
In a perfect world, each and every project that you want to instrument should
|
||||
send events on the Oslo bus about anything that could be of interest to
|
||||
you. Unfortunately, not all projects have implemented this and you will often
|
||||
need to instrument other tools which may not use the same bus as OpenStack has
|
||||
defined. The Ceilometer project created 2 methods to collect data:
|
||||
The Ceilometer project created 2 methods to collect data:
|
||||
|
||||
1. :term:`Bus listener agent` which takes events generated on the
|
||||
notification bus and transforms them into Ceilometer samples. This
|
||||
is the preferred method of data collection. If you are working on some
|
||||
is **the preferred method** of data collection. If you are working on some
|
||||
OpenStack related project and are using the Oslo library, you are kindly
|
||||
invited to come and talk to one of the project members to learn how you
|
||||
could quickly add instrumentation for your project.
|
||||
2. :term:`Polling agents`, which is the less preferred method, will poll
|
||||
some API or other tool to collect information at a regular interval.
|
||||
Where the option exists to gather the same data by consuming notifications,
|
||||
then the polling approach is less preferred due to the load it can impose
|
||||
The polling approach is less preferred due to the load it can impose
|
||||
on the API services.
|
||||
|
||||
The first method is supported by the ceilometer-notification agent, which
|
||||
monitors the message queues for notifications. Polling agents can be configured
|
||||
either to poll local hypervisor or remote APIs (public REST APIs exposed by
|
||||
either to poll the local hypervisor or remote APIs (public REST APIs exposed by
|
||||
services and host-level SNMP/IPMI daemons).
|
||||
|
||||
Notification Agents: Listening for data
|
||||
@ -125,9 +120,6 @@ expressed an interest in seeing. For example, a callback asking for
|
||||
events on the ``nova`` exchange using the ``notifications.info`` topic. Event
|
||||
matching can also work using wildcards e.g. ``compute.instance.*``.
|
||||
|
||||
Similarly, if enabled, notifications are converted into Events which can be
|
||||
filtered based on event_type declared by other services.
|
||||
|
||||
.. _polling:
|
||||
|
||||
Polling Agents: Asking for data
|
||||
@ -157,16 +149,7 @@ daemon is configured to run one or more *pollster* plugins using either the
|
||||
The agents periodically ask each pollster for instances of
|
||||
``Sample`` objects. The frequency of polling is controlled via the pipeline
|
||||
configuration. See :ref:`Pipeline-Configuration` for details.
|
||||
The agent framework then passes the samples to the
|
||||
pipeline for processing.
|
||||
|
||||
Please notice that there's an optional config called
|
||||
``shuffle_time_before_polling_task`` in ceilometer.conf. Enable this by
|
||||
setting an integer greater than zero to shuffle agents to start polling task,
|
||||
so as to add some random jitter to the time of sending requests to nova
|
||||
or other components to avoid large number of requests in short time.
|
||||
Additionally, there is an option to stream samples to minimise latency (at the
|
||||
expense of load) by setting ``batch_polled_samples`` to ``False`` in ceilometer.conf.
|
||||
The agent framework then passes the samples to the notification agent for processing.
|
||||
|
||||
|
||||
Processing the data
|
||||
@ -214,11 +197,12 @@ Publishing the data
|
||||
|
||||
This figure shows how a sample can be published to multiple destinations.
|
||||
|
||||
Currently, processed data can be published using 4 different transports:
|
||||
Currently, processed data can be published using 5 different transports:
|
||||
notifier, a notification based publisher which pushes samples to a message
|
||||
queue which can be consumed by the collector or an external system; udp, which
|
||||
publishes samples using UDP packets; and kafka, which publishes data to a Kafka
|
||||
message queue to be consumed by any system that supports Kafka.
|
||||
publishes samples using UDP packets; http, which targets a REST interface;
|
||||
and kafka, which publishes data to a Kafka message queue to be consumed by any
|
||||
system that supports Kafka.
|
||||
|
||||
|
||||
Storing the data
|
||||
@ -230,49 +214,9 @@ Collector Service
|
||||
The collector daemon gathers the processed event and metering data captured by
|
||||
the notification and polling agents. It validates the incoming data and (if
|
||||
the signature is valid) then writes the messages to a declared target:
|
||||
database, file, or http.
|
||||
database, file, gnocchi or http.
|
||||
|
||||
.. _which-db:
|
||||
|
||||
Supported databases
|
||||
-------------------
|
||||
|
||||
.. figure:: ./6-storagemodel.png
|
||||
:width: 100%
|
||||
:align: center
|
||||
:alt: Storage model
|
||||
|
||||
An overview of the Ceilometer storage model.
|
||||
|
||||
Since the beginning of the project, a plugin model has been put in place
|
||||
to allow for various types of database backends to be used. A list of supported
|
||||
backends can be found in the :ref:`choosing_db_backend` section of the
|
||||
documentation for more details.
|
||||
|
||||
In the Juno and Kilo release cycle, Ceilometer's database was divided into
|
||||
three separate connections: alarm, event, and metering. This allows
|
||||
deployers to either continue storing all data within a single database or to
|
||||
divide the data into their own databases, tailored for its purpose. For
|
||||
example, a deployer could choose to store alarms in an SQL backend while
|
||||
storing events and metering data in a NoSQL backend.
|
||||
|
||||
Ceilometer's storage service is designed to handle use cases where full-fidelity
|
||||
of the data is required (e.g. auditing). To handle responsive, long-term data
|
||||
queries, solutions that strip away some of the data's resolution, such as
|
||||
Gnocchi, are recommended.
|
||||
|
||||
.. note::
|
||||
|
||||
As of Liberty, alarming support, and subsequently its database, is handled
|
||||
by Aodh_.
|
||||
|
||||
.. note::
|
||||
|
||||
We do not guarantee that we won't change the DB schema, so it is
|
||||
highly recommended to access the database through the API and not use
|
||||
direct queries.
|
||||
|
||||
.. _Aodh: http://docs.openstack.org/developer/aodh/
|
||||
More details on database and Gnocchi targets can be found in the install guide.
|
||||
|
||||
|
||||
Accessing the data
|
||||
@ -282,14 +226,8 @@ API Service
|
||||
-----------
|
||||
|
||||
If the collected data from polling and notification agents are stored in Ceilometer's
|
||||
database(s) (see the section :ref:`which-db`), it is possible that the schema of
|
||||
these database(s) may evolve over time. For this reasons, we offer a REST API
|
||||
and recommend that you access the collected data via the API rather than by
|
||||
accessing the underlying database directly.
|
||||
|
||||
If the way in which you wish to access your data is not yet supported by the API,
|
||||
please contact us with your feedback, so that we can improve the API
|
||||
accordingly.
|
||||
database(s) (see the section :ref:`choosing_db_backend`), a REST API is available
|
||||
to access the collected data rather than by accessing the underlying database directly.
|
||||
|
||||
.. figure:: ./2-accessmodel.png
|
||||
:width: 100%
|
||||
@ -298,18 +236,6 @@ accordingly.
|
||||
|
||||
This is a representation of how to access data stored by Ceilometer
|
||||
|
||||
The :ref:`list of currently built in meters <measurements>` is available in
|
||||
the developer documentation, and it is also relatively easy to add your own
|
||||
(and eventually contribute it).
|
||||
|
||||
Ceilometer is part of OpenStack, but is not tied to OpenStack's definition of
|
||||
"users" and "tenants." The "source" field of each sample refers to the authority
|
||||
defining the user and tenant associated with the sample. Deployers can define
|
||||
custom sources through a configuration file, and then create agents to collect
|
||||
samples for new meters using those sources. This means that you can collect
|
||||
data for applications running on top of OpenStack, such as a PaaS or SaaS
|
||||
layer, and use the same tools for metering your entire cloud.
|
||||
|
||||
Moreover, end users can also
|
||||
:ref:`send their own application specific data <user-defined-data>` into the
|
||||
database through the REST API for a various set of use cases.
|
||||
|
||||
@ -22,47 +22,6 @@ order to set up the services please see the
|
||||
`Telemetry section <http://docs.openstack.org/trunk/config-reference/content/ch_configuring-openstack-telemetry.html>`_
|
||||
in the OpenStack Manuals Configuration Reference.
|
||||
|
||||
HBase
|
||||
===================
|
||||
|
||||
This storage implementation uses Thrift HBase interface. The default Thrift
|
||||
connection settings should be changed to support using ConnectionPool in HBase.
|
||||
To ensure proper configuration, please add the following lines to the
|
||||
`hbase-site.xml` configuration file::
|
||||
|
||||
<property>
|
||||
<name>hbase.thrift.minWorkerThreads</name>
|
||||
<value>200</value>
|
||||
</property>
|
||||
|
||||
For pure development purposes, you can use HBase from Apache_ or some other
|
||||
vendor like Cloudera or Hortonworks. To verify your installation, you can use
|
||||
the `list` command in `HBase shell`, to list the tables in your
|
||||
HBase server, as follows::
|
||||
|
||||
$ ${HBASE_HOME}/bin/hbase shell
|
||||
|
||||
hbase> list
|
||||
|
||||
.. note::
|
||||
This driver has been tested against HBase 0.94.2/CDH 4.2.0,
|
||||
HBase 0.94.4/HDP 1.2, HBase 0.94.18/Apache, HBase 0.94.5/Apache,
|
||||
HBase 0.96.2/Apache and HBase 0.98.0/Apache.
|
||||
Versions earlier than 0.92.1 are not supported due to feature incompatibility.
|
||||
|
||||
To find out more about supported storage backends please take a look on the
|
||||
:doc:`install/manual/` guide.
|
||||
|
||||
.. note::
|
||||
|
||||
If you are changing the configuration on the fly to use HBase, as a storage
|
||||
backend, you will need to restart the Ceilometer services that use the
|
||||
database to allow the changes to take affect, i.e. the collector and API
|
||||
services.
|
||||
|
||||
.. _Apache: https://hbase.apache.org/book/quickstart.html
|
||||
|
||||
|
||||
Sample Configuration file
|
||||
=========================
|
||||
|
||||
@ -72,15 +31,6 @@ the Icehouse release. For more details, please read the file
|
||||
etc/ceilometer/README-ceilometer.conf.txt. You can generate this sample
|
||||
configuration file by running ``tox -e genconfig``.
|
||||
|
||||
.. note::
|
||||
tox version 1.7.0 and 1.7.1 have a `backward compatibility issue`_
|
||||
with OpenStack projects. If you meet the "tox.ConfigError: ConfigError:
|
||||
substitution key 'posargs' not found" problem, run
|
||||
``sudo pip install -U "tox>=1.6.1,!=1.7.0,!=1.7.1"`` to get a proper
|
||||
version, then try ``tox -e genconfig`` again.
|
||||
|
||||
.. _`backward compatibility issue`: https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror
|
||||
|
||||
.. _Pipeline-Configuration:
|
||||
|
||||
Pipelines
|
||||
|
||||
@ -150,3 +150,16 @@ configuration settings should be added::
|
||||
.. note::
|
||||
If gnocchi dispatcher is enabled, Ceilometer api calls will return a 410 with
|
||||
an empty result. The Gnocchi Api should be used instead to access the data.
|
||||
|
||||
Efficient polling
|
||||
=================
|
||||
|
||||
- There is an optional config called ``shuffle_time_before_polling_task``
|
||||
in ceilometer.conf. Enable this by setting an integer greater than zero to
|
||||
shuffle polling time for agents. This will add some random jitter to the time
|
||||
of sending requests to Nova or other components to avoid large number of
|
||||
requests in a short time period.
|
||||
- There is an option to stream samples to minimise latency (at the
|
||||
expense of load) by setting ``batch_polled_samples`` to ``False`` in
|
||||
ceilometer.conf.
|
||||
|
||||
|
||||
@ -27,6 +27,13 @@
|
||||
for low latency use cases. For more responsive use cases, it's recommended
|
||||
to store data in an alternative source such as Gnocchi_
|
||||
|
||||
.. note::
|
||||
|
||||
As of Liberty, alarming support, and subsequently its database, is handled
|
||||
by Aodh_.
|
||||
|
||||
.. _Aodh: http://docs.openstack.org/developer/aodh/
|
||||
|
||||
Selecting a database backend for Ceilometer should not be done lightly for
|
||||
numerous reasons:
|
||||
|
||||
|
||||
@ -20,19 +20,12 @@
|
||||
|
||||
Ceilometer has several daemons. The basic are: :term:`polling agent` running
|
||||
either on the Nova compute node(s) or :term:`polling agent` running on the
|
||||
central management node(s), :term:`collector`
|
||||
and :term:`notification agent` running on the cloud's management node(s).
|
||||
central management node(s), :term:`collector` and :term:`notification agent`
|
||||
running on the cloud's management node(s).
|
||||
|
||||
In a development environment created by devstack_, these services are
|
||||
typically running on the same server. They do not have to be, though, so some
|
||||
of the instructions below are duplicated. Skip the steps you have already done.
|
||||
typically running on the same server.
|
||||
|
||||
.. note::
|
||||
|
||||
In fact, previously ceilometer had separated compute and central agents, and
|
||||
their support is implemented in devstack_ right now, not one agent variant.
|
||||
For now we do have deprecated cmd scripts emulating old compute/central
|
||||
behavior using namespaces option passed to polling agent, which will be
|
||||
maintained for a transitional period.
|
||||
|
||||
Configuring devstack
|
||||
====================
|
||||
@ -58,15 +51,7 @@ Configuring devstack
|
||||
# Enable the Ceilometer devstack plugin
|
||||
enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer.git
|
||||
|
||||
5. Nova does not generate the periodic notifications for all known
|
||||
instances by default. To enable these auditing events, set
|
||||
``instance_usage_audit`` to true in the nova configuration file and restart
|
||||
the service.
|
||||
|
||||
6. Cinder does not generate notifications by default. To enable
|
||||
these auditing events, set the following in the cinder configuration file
|
||||
and restart the service::
|
||||
|
||||
notification_driver=messagingv2
|
||||
By default, all ceilometer services except for ceilometer-ipmi agent will
|
||||
be enabled
|
||||
|
||||
.. _devstack: http://www.devstack.org/
|
||||
|
||||
@ -159,6 +159,14 @@ HBase
|
||||
[database]
|
||||
connection = hbase://hbase-thrift-host:9090?table_prefix=ceilo&table_prefix_separator=.
|
||||
|
||||
To ensure proper configuration, please add the following lines to the
|
||||
`hbase-site.xml` configuration file::
|
||||
|
||||
<property>
|
||||
<name>hbase.thrift.minWorkerThreads</name>
|
||||
<value>200</value>
|
||||
</property>
|
||||
|
||||
.. _`Gnocchi installation`: http://docs.openstack.org/developer/gnocchi/install.html
|
||||
.. _HappyBase: http://happybase.readthedocs.org/en/latest/index.html#
|
||||
.. _MongoDB: http://www.mongodb.org/
|
||||
|
||||
Loading…
Reference in New Issue
Block a user