From 9129ed72502ea35e4950d72a3c02cbe9a296822c Mon Sep 17 00:00:00 2001
From: Tobias Henkel <tobias.henkel@bmw.de>
Date: Mon, 2 Oct 2017 07:06:30 +0200
Subject: [PATCH] Increase ansible internal_poll_interval

The default internal polling interval [1] of ansible is optimized for
UI responsiveness. The default value of 0.001s comes with the cost of
a high cpu load.

The testing on my local machine shows that running a 'sleep 30' with
ansible takes about 3.6 cpu seconds. Increasing the internal poll
interval to 0.01s dropped the cpu costs to 0.9 cpu seconds which is
quite a bit lower. Further increasing this value drops the cpu use
further to 0.5 cpu seconds.

But the potential job runtime costs with many tasks also need to be
analyzed. There I tested a playbook with 100 debug tasks. In this case
the default interval is the fastest [5]. The 0.01s interval is
slightly slower but still fast [6] where the 0.1s interval is much
slower [7].

So all in all for our automation use case the 0.01s interval seems to
be the best fit with a greatly reduced cost of cpu time while still
being fast for playbooks with many tasks.

[1]: http://docs.ansible.com/ansible/2.3/intro_configuration.html#internal-poll-interval

[2]: Long task run with default interval (0.001s):
  $ time ansible -i hosts all -m shell -a 'sleep 30'
  nodepool | SUCCESS | rc=0 >>
  real 31.68
  user 3.11
  sys 0.49

[3]: Long task run with longer interval (0.01s):
  $ time ansible -i hosts all -m shell -a 'sleep 30'
  nodepool | SUCCESS | rc=0 >>
  real 30.54
  user 0.76
  sys 0.11

[4]: Long task run with much interval (0.1s):

  $ time ansible -i hosts all -m shell -a 'sleep 30'
  zuul | SUCCESS | rc=0 >>
  real 31.15
  user 0.44
  sys 0.10

[5]: 100 tasks run with default interval (0.001s):

  $ time ansible-playbook -i hosts test.yaml
  real 2.30
  user 1.81
  sys 0.62

[6]: 100 tasks run with longer interval (0.01s):

  $ time ansible-playbook -i hosts test.yaml
  real 2.77
  user 1.79
  sys 0.66

[7]: 100 tasks run with much longer interval (0.1s):

  $ time ansible-playbook -i hosts test.yaml
  real 11.40
  user 2.05
  sys 0.82

Change-Id: I569cb5dc445cfbca465735703c516f8acdab636b
---
 zuul/executor/server.py | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/zuul/executor/server.py b/zuul/executor/server.py
index b37a82e9e2..1475440448 100644
--- a/zuul/executor/server.py
+++ b/zuul/executor/server.py
@@ -1507,6 +1507,13 @@ class AnsibleJob(object):
             config.write('display_args_to_stdout = %s\n' %
                          str(not jobdir_playbook.secrets_content))
 
+            # Increase the internal poll interval of ansible.
+            # The default interval of 0.001s is optimized for interactive
+            # ui at the expense of CPU load. As we have a non-interactive
+            # automation use case a longer poll interval is more suitable
+            # and reduces CPU load of the ansible process.
+            config.write('internal_poll_interval = 0.01\n')
+
             config.write('[ssh_connection]\n')
             # NB: when setting pipelining = True, keep_remote_files
             # must be False (the default).  Otherwise it apparently