Merge "Add gearman stats reference"
This commit is contained in:
commit
51ede31e84
|
@ -264,7 +264,10 @@ These metrics are emitted by the Zuul :ref:`scheduler`:
|
||||||
.. stat:: current_requests
|
.. stat:: current_requests
|
||||||
:type: gauge
|
:type: gauge
|
||||||
|
|
||||||
The number of outstanding nodepool requests from Zuul.
|
The number of outstanding nodepool requests from Zuul. Ideally
|
||||||
|
this will be at zero, meaning all requests are fulfilled.
|
||||||
|
Persistently high values indicate more testing node resources
|
||||||
|
would be helpful.
|
||||||
|
|
||||||
.. stat:: zuul.mergers
|
.. stat:: zuul.mergers
|
||||||
|
|
||||||
|
@ -283,7 +286,9 @@ These metrics are emitted by the Zuul :ref:`scheduler`:
|
||||||
.. stat:: jobs_queued
|
.. stat:: jobs_queued
|
||||||
:type: gauge
|
:type: gauge
|
||||||
|
|
||||||
The number of merge jobs queued.
|
The number of merge jobs waiting for a merger. This should
|
||||||
|
ideally be zero; persistent higher values indicate more merger
|
||||||
|
resources would be useful.
|
||||||
|
|
||||||
.. stat:: zuul.executors
|
.. stat:: zuul.executors
|
||||||
|
|
||||||
|
@ -307,8 +312,40 @@ These metrics are emitted by the Zuul :ref:`scheduler`:
|
||||||
.. stat:: jobs_queued
|
.. stat:: jobs_queued
|
||||||
:type: gauge
|
:type: gauge
|
||||||
|
|
||||||
The number of executor jobs queued.
|
The number of jobs allocated nodes, but queued waiting for an
|
||||||
|
executor to run on. This should ideally be at zero; persistent
|
||||||
|
higher values indicate more exectuor resources would be useful.
|
||||||
|
|
||||||
|
.. stat:: zuul.geard
|
||||||
|
|
||||||
|
Gearman job distribution statistics. Gearman jobs encompass the
|
||||||
|
wide variety of distributed jobs running within the scheduler and
|
||||||
|
across mergers and exectuors. These stats are emitted by the `gear
|
||||||
|
<https://pypi.org/project/gear/>`__ library.
|
||||||
|
|
||||||
|
.. stat:: running
|
||||||
|
:type: gauge
|
||||||
|
|
||||||
|
Jobs that Gearman has actively running. The longest running
|
||||||
|
jobs will usually relate to active job execution so you would
|
||||||
|
expect this to have a lower bound around there. Note this may
|
||||||
|
be lower than active nodes, as a multiple-node job will only
|
||||||
|
have one active Gearman job.
|
||||||
|
|
||||||
|
.. stat:: waiting
|
||||||
|
:type: gauge
|
||||||
|
|
||||||
|
Jobs waiting in the gearman queue. This would be expected to be
|
||||||
|
around zero; note that this is *not* related to the backlogged
|
||||||
|
queue of jobs waiting for a node allocation (node allocations
|
||||||
|
are via Zookeeper). If this is unexpectedly high, see
|
||||||
|
:ref:`debug_gearman` for queue debugging tips to find out which
|
||||||
|
particular function calls are waiting.
|
||||||
|
|
||||||
|
.. stat:: total
|
||||||
|
:type: gauge
|
||||||
|
|
||||||
|
The sum of the `running` and `waiting` jobs.
|
||||||
|
|
||||||
As an example, given a job named `myjob` in `mytenant` triggered by a
|
As an example, given a job named `myjob` in `mytenant` triggered by a
|
||||||
change to `myproject` on the `master` branch in the `gate` pipeline
|
change to `myproject` on the `master` branch in the `gate` pipeline
|
||||||
|
|
|
@ -1,10 +1,41 @@
|
||||||
Troubleshooting
|
Troubleshooting
|
||||||
---------------
|
---------------
|
||||||
|
|
||||||
You can use telnet to connect to gearman to check which Zuul
|
Some advanced troubleshooting options are provided below. These are
|
||||||
components are online::
|
generally very low-level and are not normally required.
|
||||||
|
|
||||||
|
.. _debug_gearman:
|
||||||
|
|
||||||
|
Gearman Jobs
|
||||||
|
============
|
||||||
|
|
||||||
|
Connecting to Gearman can allow you see if any Zuul components appear
|
||||||
|
to not be accepting requests correctly.
|
||||||
|
|
||||||
|
For unencrypted Gearman connections, you can use telnet to connect to
|
||||||
|
and check which Zuul components are online::
|
||||||
|
|
||||||
telnet <gearman_ip> 4730
|
telnet <gearman_ip> 4730
|
||||||
|
|
||||||
Useful commands are ``workers`` and ``status`` which you can run by just
|
For encrypted connections, you will need to provide suitable keys,
|
||||||
typing those commands once connected to gearman.
|
e.g::
|
||||||
|
|
||||||
|
openssl s_client -connect localhost:4730 -cert /etc/zuul/ssl/client.pem -key /etc/zuul/ssl/client.key
|
||||||
|
|
||||||
|
Commands available are discussed in the Gearman `administrative
|
||||||
|
protocol <http://gearman.org/protocol>`__. Useful commands are
|
||||||
|
``workers`` and ``status`` which you can run by just typing those
|
||||||
|
commands once connected to gearman.
|
||||||
|
|
||||||
|
For ``status`` you will see output for internal Zuul functions in the
|
||||||
|
form ``FUNCTION\tTOTAL\tRUNNING\tAVAILABLE_WORKERS``::
|
||||||
|
|
||||||
|
...
|
||||||
|
executor:resume:ze06.openstack.org 0 0 1
|
||||||
|
zuul:config_errors_list 0 0 1
|
||||||
|
zuul:status_get 0 0 1
|
||||||
|
executor:stop:ze11.openstack.org 0 0 1
|
||||||
|
zuul:job_list 0 0 1
|
||||||
|
zuul:tenant_sql_connection 0 0 1
|
||||||
|
executor:resume:ze09.openstack.org 0 0 1
|
||||||
|
...
|
||||||
|
|
Loading…
Reference in New Issue