Shuffle agents to send request

For each polling interval, Ceilometer compute agent will trigger
nova client directly to acquire the list of VMs.

However, there's a potential risk that the workload for nova client
will increases proportional to the number of Ceilometer compute
agents. In other word, there's a possibility that thousands of
ceilometer compute agents will call nova client in a very short time
period which may lead large number of requests to nova client in
short time. As a result, the nova client can not handle so many
requests in short time, and ceilometer compute agent can't fetch
metrics.

So we here shuffle agents to start polling task in order to fluff up
the time of sending request to nova or other components to avoid
large number of requests in short time. This would be more
performance effective and reliable.

DocImpact

Change-Id: I12e3f104fc92fe15adc05e2b981627f31ee5bfaa
Closes-bug: #1412613

ceilometer/agent/base.py

@@ -21,6 +21,7 @@
 import collections
 import fnmatch
 import itertools
+import random
 
 from oslo_config import cfg
 from oslo_context import context
@@ -37,6 +38,16 @@ from ceilometer import utils
 
 LOG = log.getLogger(__name__)
 
+OPTS = [
+    cfg.IntOpt('shuffle_time_before_polling_task',
+               default=0,
+               help='To reduce large requests at same time to Nova or other '
+                    'components from different compute agents, shuffle '
+                    'start time of polling task.'),
+]
+
+cfg.CONF.register_opts(OPTS)
+
 
 class PollsterListForbidden(Exception):
     def __init__(self):
@@ -237,10 +248,16 @@ class AgentManager(os_service.Service):
         # allow time for coordination if necessary
         delay_start = self.partition_coordinator.is_active()
 
+        # set shuffle time before polling task if necessary
+        delay_polling_time = random.randint(
+            0, cfg.CONF.shuffle_time_before_polling_task)
+
         for interval, task in six.iteritems(self.setup_polling_tasks()):
+            delay_time = (interval + delay_polling_time if delay_start
+                          else delay_polling_time)
             self.tg.add_timer(interval,
                               self.interval_task,
-                              initial_delay=interval if delay_start else None,
+                              initial_delay=delay_time,
                               task=task)
         self.tg.add_timer(cfg.CONF.coordination.heartbeat,
                           self.partition_coordinator.heartbeat)

ceilometer/opts.py

@@ -66,7 +66,8 @@ import ceilometer.volume.notifications
 def list_opts():
     return [
         ('DEFAULT',
-         itertools.chain(ceilometer.api.app.OPTS,
+         itertools.chain(ceilometer.agent.base.OPTS,
+                         ceilometer.api.app.OPTS,
                          ceilometer.cmd.polling.CLI_OPTS,
                          ceilometer.compute.notifications.OPTS,
                          ceilometer.compute.util.OPTS,

doc/source/architecture.rst

@@ -382,6 +382,12 @@ the publishers defined in the pipeline configuration. For example,
 the ``notifier`` publisher converts the Sample to metering messages, which it
 then signs and transmits on the metering message bus.
 
+Please notice that there's an optional config called
+``shuffle_time_before_polling_task`` in ceilometer.conf. Enable this by
+setting an integer greater than zero to shuffle agents to start polling task,
+so that fluff up the time of sending requests to nova or other components to
+avoid large number of requests in short time.
+
 The frequency of polling is controlled via the pipeline configuration.
 See :ref:`Pipeline-Configuration` for details.