Add event queue sizes to stats reporting

Especially in bigger systems the event queue sizes can be an important
performance indicator so add them to the stats.

Change-Id: Ib5e2926052302b151e505b0be1da66567a645634

doc/source/reference/monitoring.rst

@@ -361,6 +361,30 @@ These metrics are emitted by the Zuul :ref:`scheduler`:
       executor to run on.  This should ideally be at zero; persistent
       higher values indicate more executor resources would be useful.
 
+
+.. stat:: zuul.scheduler
+
+   Holds metrics related to the Zuul scheduler.
+
+   .. stat:: eventqueues
+
+      Holds metrics about the event queue lengths in the Zuul scheduler.
+
+      .. stat:: trigger
+         :type: gauge
+
+         The size of the current trigger event queue.
+
+      .. stat:: result
+         :type: gauge
+
+         The size of the current result event queue.
+
+      .. stat:: management
+         :type: gauge
+
+         The size of the current management event queue.
+
 .. stat:: zuul.geard
 
    Gearman job distribution statistics.  Gearman jobs encompass the

zuul/scheduler.py

@@ -455,6 +455,13 @@ class Scheduler(threading.Thread):
         self.statsd.gauge('zuul.executors.jobs_running', execute_running)
         self.statsd.gauge('zuul.executors.jobs_queued', execute_queue)
 
+        self.statsd.gauge('zuul.scheduler.eventqueues.trigger',
+                          self.trigger_event_queue.qsize())
+        self.statsd.gauge('zuul.scheduler.eventqueues.result',
+                          self.result_event_queue.qsize())
+        self.statsd.gauge('zuul.scheduler.eventqueues.management',
+                          self.management_event_queue.qsize())
+
     def addEvent(self, event):
         self.trigger_event_queue.put(event)
         self.wake_event.set()