From 7fe87e904f4c2bca39e0e643eeb7b4d0ff17ca08 Mon Sep 17 00:00:00 2001 From: gordon chung Date: Tue, 5 Apr 2016 16:45:38 -0400 Subject: [PATCH] delete verbose/redundant/deprecated text the docs are a bit wordy in some places. our docs are confusing enough: - remove notification > polling in architecture page - remove redundant Events filtering from event_types - move shuffle/batch options to customisation page - add http publisher - remove db choices from architecture page, point to dbreco page - remove db background referencing single database route - remove warnings to say db might change, use api (obvious) - remove stuff about Ceilometer is part Openstack but not tied to it - move hbase configuration under common hbase section in install - remove tox compatibility comments since it's > 2 years - remove random non-devstack/historical text from devstack page - remove default cinder/nova configurations from devstack page Change-Id: Ifdbeee358cfcb5d3736608e39067ff2859cc0ed6 --- doc/source/architecture.rst | 100 ++++------------------------- doc/source/configuration.rst | 50 --------------- doc/source/install/custom.rst | 13 ++++ doc/source/install/dbreco.rst | 7 ++ doc/source/install/development.rst | 27 ++------ doc/source/install/manual.rst | 8 +++ 6 files changed, 47 insertions(+), 158 deletions(-) diff --git a/doc/source/architecture.rst b/doc/source/architecture.rst index eb558cf0..2460efe8 100644 --- a/doc/source/architecture.rst +++ b/doc/source/architecture.rst @@ -67,27 +67,22 @@ How is data collected? This is a representation of how the collectors and agents gather data from multiple sources. -In a perfect world, each and every project that you want to instrument should -send events on the Oslo bus about anything that could be of interest to -you. Unfortunately, not all projects have implemented this and you will often -need to instrument other tools which may not use the same bus as OpenStack has -defined. The Ceilometer project created 2 methods to collect data: +The Ceilometer project created 2 methods to collect data: 1. :term:`Bus listener agent` which takes events generated on the notification bus and transforms them into Ceilometer samples. This - is the preferred method of data collection. If you are working on some + is **the preferred method** of data collection. If you are working on some OpenStack related project and are using the Oslo library, you are kindly invited to come and talk to one of the project members to learn how you could quickly add instrumentation for your project. 2. :term:`Polling agents`, which is the less preferred method, will poll some API or other tool to collect information at a regular interval. - Where the option exists to gather the same data by consuming notifications, - then the polling approach is less preferred due to the load it can impose + The polling approach is less preferred due to the load it can impose on the API services. The first method is supported by the ceilometer-notification agent, which monitors the message queues for notifications. Polling agents can be configured -either to poll local hypervisor or remote APIs (public REST APIs exposed by +either to poll the local hypervisor or remote APIs (public REST APIs exposed by services and host-level SNMP/IPMI daemons). Notification Agents: Listening for data @@ -125,9 +120,6 @@ expressed an interest in seeing. For example, a callback asking for events on the ``nova`` exchange using the ``notifications.info`` topic. Event matching can also work using wildcards e.g. ``compute.instance.*``. -Similarly, if enabled, notifications are converted into Events which can be -filtered based on event_type declared by other services. - .. _polling: Polling Agents: Asking for data @@ -157,16 +149,7 @@ daemon is configured to run one or more *pollster* plugins using either the The agents periodically ask each pollster for instances of ``Sample`` objects. The frequency of polling is controlled via the pipeline configuration. See :ref:`Pipeline-Configuration` for details. -The agent framework then passes the samples to the -pipeline for processing. - -Please notice that there's an optional config called -``shuffle_time_before_polling_task`` in ceilometer.conf. Enable this by -setting an integer greater than zero to shuffle agents to start polling task, -so as to add some random jitter to the time of sending requests to nova -or other components to avoid large number of requests in short time. -Additionally, there is an option to stream samples to minimise latency (at the -expense of load) by setting ``batch_polled_samples`` to ``False`` in ceilometer.conf. +The agent framework then passes the samples to the notification agent for processing. Processing the data @@ -214,11 +197,12 @@ Publishing the data This figure shows how a sample can be published to multiple destinations. -Currently, processed data can be published using 4 different transports: +Currently, processed data can be published using 5 different transports: notifier, a notification based publisher which pushes samples to a message queue which can be consumed by the collector or an external system; udp, which -publishes samples using UDP packets; and kafka, which publishes data to a Kafka -message queue to be consumed by any system that supports Kafka. +publishes samples using UDP packets; http, which targets a REST interface; +and kafka, which publishes data to a Kafka message queue to be consumed by any +system that supports Kafka. Storing the data @@ -230,49 +214,9 @@ Collector Service The collector daemon gathers the processed event and metering data captured by the notification and polling agents. It validates the incoming data and (if the signature is valid) then writes the messages to a declared target: -database, file, or http. +database, file, gnocchi or http. -.. _which-db: - -Supported databases -------------------- - -.. figure:: ./6-storagemodel.png - :width: 100% - :align: center - :alt: Storage model - - An overview of the Ceilometer storage model. - -Since the beginning of the project, a plugin model has been put in place -to allow for various types of database backends to be used. A list of supported -backends can be found in the :ref:`choosing_db_backend` section of the -documentation for more details. - -In the Juno and Kilo release cycle, Ceilometer's database was divided into -three separate connections: alarm, event, and metering. This allows -deployers to either continue storing all data within a single database or to -divide the data into their own databases, tailored for its purpose. For -example, a deployer could choose to store alarms in an SQL backend while -storing events and metering data in a NoSQL backend. - -Ceilometer's storage service is designed to handle use cases where full-fidelity -of the data is required (e.g. auditing). To handle responsive, long-term data -queries, solutions that strip away some of the data's resolution, such as -Gnocchi, are recommended. - -.. note:: - - As of Liberty, alarming support, and subsequently its database, is handled - by Aodh_. - -.. note:: - - We do not guarantee that we won't change the DB schema, so it is - highly recommended to access the database through the API and not use - direct queries. - -.. _Aodh: http://docs.openstack.org/developer/aodh/ +More details on database and Gnocchi targets can be found in the install guide. Accessing the data @@ -282,14 +226,8 @@ API Service ----------- If the collected data from polling and notification agents are stored in Ceilometer's -database(s) (see the section :ref:`which-db`), it is possible that the schema of -these database(s) may evolve over time. For this reasons, we offer a REST API -and recommend that you access the collected data via the API rather than by -accessing the underlying database directly. - -If the way in which you wish to access your data is not yet supported by the API, -please contact us with your feedback, so that we can improve the API -accordingly. +database(s) (see the section :ref:`choosing_db_backend`), a REST API is available +to access the collected data rather than by accessing the underlying database directly. .. figure:: ./2-accessmodel.png :width: 100% @@ -298,18 +236,6 @@ accordingly. This is a representation of how to access data stored by Ceilometer -The :ref:`list of currently built in meters ` is available in -the developer documentation, and it is also relatively easy to add your own -(and eventually contribute it). - -Ceilometer is part of OpenStack, but is not tied to OpenStack's definition of -"users" and "tenants." The "source" field of each sample refers to the authority -defining the user and tenant associated with the sample. Deployers can define -custom sources through a configuration file, and then create agents to collect -samples for new meters using those sources. This means that you can collect -data for applications running on top of OpenStack, such as a PaaS or SaaS -layer, and use the same tools for metering your entire cloud. - Moreover, end users can also :ref:`send their own application specific data ` into the database through the REST API for a various set of use cases. diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst index 742f45c9..8e074779 100644 --- a/doc/source/configuration.rst +++ b/doc/source/configuration.rst @@ -22,47 +22,6 @@ order to set up the services please see the `Telemetry section `_ in the OpenStack Manuals Configuration Reference. -HBase -=================== - -This storage implementation uses Thrift HBase interface. The default Thrift -connection settings should be changed to support using ConnectionPool in HBase. -To ensure proper configuration, please add the following lines to the -`hbase-site.xml` configuration file:: - - - hbase.thrift.minWorkerThreads - 200 - - -For pure development purposes, you can use HBase from Apache_ or some other -vendor like Cloudera or Hortonworks. To verify your installation, you can use -the `list` command in `HBase shell`, to list the tables in your -HBase server, as follows:: - - $ ${HBASE_HOME}/bin/hbase shell - - hbase> list - -.. note:: - This driver has been tested against HBase 0.94.2/CDH 4.2.0, - HBase 0.94.4/HDP 1.2, HBase 0.94.18/Apache, HBase 0.94.5/Apache, - HBase 0.96.2/Apache and HBase 0.98.0/Apache. - Versions earlier than 0.92.1 are not supported due to feature incompatibility. - -To find out more about supported storage backends please take a look on the -:doc:`install/manual/` guide. - -.. note:: - - If you are changing the configuration on the fly to use HBase, as a storage - backend, you will need to restart the Ceilometer services that use the - database to allow the changes to take affect, i.e. the collector and API - services. - -.. _Apache: https://hbase.apache.org/book/quickstart.html - - Sample Configuration file ========================= @@ -72,15 +31,6 @@ the Icehouse release. For more details, please read the file etc/ceilometer/README-ceilometer.conf.txt. You can generate this sample configuration file by running ``tox -e genconfig``. -.. note:: - tox version 1.7.0 and 1.7.1 have a `backward compatibility issue`_ - with OpenStack projects. If you meet the "tox.ConfigError: ConfigError: - substitution key 'posargs' not found" problem, run - ``sudo pip install -U "tox>=1.6.1,!=1.7.0,!=1.7.1"`` to get a proper - version, then try ``tox -e genconfig`` again. - -.. _`backward compatibility issue`: https://bitbucket.org/hpk42/tox/issue/150/posargs-configerror - .. _Pipeline-Configuration: Pipelines diff --git a/doc/source/install/custom.rst b/doc/source/install/custom.rst index 7d7b18a8..edb954db 100644 --- a/doc/source/install/custom.rst +++ b/doc/source/install/custom.rst @@ -150,3 +150,16 @@ configuration settings should be added:: .. note:: If gnocchi dispatcher is enabled, Ceilometer api calls will return a 410 with an empty result. The Gnocchi Api should be used instead to access the data. + +Efficient polling +================= + +- There is an optional config called ``shuffle_time_before_polling_task`` + in ceilometer.conf. Enable this by setting an integer greater than zero to + shuffle polling time for agents. This will add some random jitter to the time + of sending requests to Nova or other components to avoid large number of + requests in a short time period. +- There is an option to stream samples to minimise latency (at the + expense of load) by setting ``batch_polled_samples`` to ``False`` in + ceilometer.conf. + diff --git a/doc/source/install/dbreco.rst b/doc/source/install/dbreco.rst index f20bb6d2..3b940a50 100644 --- a/doc/source/install/dbreco.rst +++ b/doc/source/install/dbreco.rst @@ -27,6 +27,13 @@ for low latency use cases. For more responsive use cases, it's recommended to store data in an alternative source such as Gnocchi_ +.. note:: + + As of Liberty, alarming support, and subsequently its database, is handled + by Aodh_. + +.. _Aodh: http://docs.openstack.org/developer/aodh/ + Selecting a database backend for Ceilometer should not be done lightly for numerous reasons: diff --git a/doc/source/install/development.rst b/doc/source/install/development.rst index a0fe2d66..08a052d8 100644 --- a/doc/source/install/development.rst +++ b/doc/source/install/development.rst @@ -20,19 +20,12 @@ Ceilometer has several daemons. The basic are: :term:`polling agent` running either on the Nova compute node(s) or :term:`polling agent` running on the -central management node(s), :term:`collector` -and :term:`notification agent` running on the cloud's management node(s). +central management node(s), :term:`collector` and :term:`notification agent` +running on the cloud's management node(s). + In a development environment created by devstack_, these services are -typically running on the same server. They do not have to be, though, so some -of the instructions below are duplicated. Skip the steps you have already done. +typically running on the same server. -.. note:: - - In fact, previously ceilometer had separated compute and central agents, and - their support is implemented in devstack_ right now, not one agent variant. - For now we do have deprecated cmd scripts emulating old compute/central - behavior using namespaces option passed to polling agent, which will be - maintained for a transitional period. Configuring devstack ==================== @@ -58,15 +51,7 @@ Configuring devstack # Enable the Ceilometer devstack plugin enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer.git -5. Nova does not generate the periodic notifications for all known - instances by default. To enable these auditing events, set - ``instance_usage_audit`` to true in the nova configuration file and restart - the service. - -6. Cinder does not generate notifications by default. To enable - these auditing events, set the following in the cinder configuration file - and restart the service:: - - notification_driver=messagingv2 + By default, all ceilometer services except for ceilometer-ipmi agent will + be enabled .. _devstack: http://www.devstack.org/ diff --git a/doc/source/install/manual.rst b/doc/source/install/manual.rst index 6328a38b..fa543f39 100644 --- a/doc/source/install/manual.rst +++ b/doc/source/install/manual.rst @@ -159,6 +159,14 @@ HBase [database] connection = hbase://hbase-thrift-host:9090?table_prefix=ceilo&table_prefix_separator=. + To ensure proper configuration, please add the following lines to the + `hbase-site.xml` configuration file:: + + + hbase.thrift.minWorkerThreads + 200 + + .. _`Gnocchi installation`: http://docs.openstack.org/developer/gnocchi/install.html .. _HappyBase: http://happybase.readthedocs.org/en/latest/index.html# .. _MongoDB: http://www.mongodb.org/