Implement Ansible fact cache for Mistral executor

Most of the playbooks executed by Mistral gather facts which, by default,
are stored in memory until the conclusion of the playbook execution. This
means that for every new playbook run the facts for each host are gathered
again. Fact gathering can take a long time, especially when the target host
has a lot of devices, which is often the case for compute hosts (many
network devices and many storage devices) and for controller hosts (many
network devices).

To optimise fact gathering, we set the default behaviour for Mistral's
Ansible execution to store gathered facts using the jsonfile format in a
temporary location, and we tell it to use smart gathering so that it will
only gather new facts when a new playbook executes.

The validations are run using the 'validations' user, it cannot share the
same fact cache because the /var/tmp/ansible_fact_cache folder will be owned
by the 'mistral' user, and is therefore not writable by the 'validations'
user. We therefore set the validations to use a different path.

This should cut down the deployment time a bit, which saves a little time in
CI and saves a lot of time in production environments.

Related: rhbz#1749406
Change-Id: Ie7adf41cef0fafde646f840b2bf3e01ba42a17d5
(cherry picked from commit bbc0d767bb)

scripts/run-validation

@@ -30,6 +30,14 @@ export ANSIBLE_RETRY_FILES_ENABLED=False
 # Add `-E` to the sudo flags to preserve the shell environment. The `-H -S -n`
 # options are Ansible's default values.
 export ANSIBLE_SUDO_FLAGS="-H -S -n -E"
+# Use a fact cache to improve performance
+# The defaults from tripleo_common/actions/ansible.py cannot be used
+# beause they will clash with the 'mistral' user owning the fact cache
+# path.
+export ANSIBLE_GATHERING=smart
+export ANSIBLE_CACHE_PLUGIN=jsonfile
+export ANSIBLE_CACHE_PLUGIN_CONNECTION=~/ansible_fact_cache
+export ANSIBLE_GATHER_TIMEOUT=7200 # 7200s = 2h
 
 export ANSIBLE_PRIVATE_KEY_FILE=$IDENTITY_FILE
 

tripleo_common/actions/ansible.py

@@ -46,6 +46,15 @@ def write_default_ansible_cfg(work_dir,
     config.set('defaults', 'timeout', '30')
     config.set('defaults', 'gather_timeout', '30')
 
+    # Setup fact cache to improve playbook execution speed
+    config.set('defaults', 'gathering', 'smart')
+    config.set('defaults', 'fact_caching', 'jsonfile')
+    config.set('defaults', 'fact_caching_connection',
+               '/var/tmp/ansible_fact_cache')
+
+    # Expire facts in the fact cache after 7200s (2h)
+    config.set('defaults', 'fact_caching_timeout', '7200')
+
     # mistral user has no home dir set, so no place to save a known hosts file
     config.set('ssh_connection', 'ssh_args',
                '-o UserKnownHostsFile=/dev/null '