Browse Source

puppet: don't run module install steps multiple times

It turns out you can't use "run_once" with the "free" strategy in
Ansible.  It actually warns you about this, if you're looking in the
right place.

The existing run-puppet role calls two things with "run_once:", both
delegated to localhost -- cloning the ansible-role-puppet repo (so we
can include_role: puppet) and installing the puppet modules (via
install-ansible-roles role), which are copied from bridge to the
remote side and run by ansible-role-puppet.

With remote_puppet_else.yaml we are running all the puppet hosts at
once with the "free" strategy.  This means that these two tasks, both
delegated to localhost (bridge) are actually running for every host.
install-ansible-roles does a git clone, and thus we often see one of
the clones bailing out with a git locking error, because the other
host is running similtaneously.
I8585a1af2dcc294c0e61fc45d9febb044e42151d tried to stop this with
"run_once:" -- but as noted because it's running under the "free"
strategy this is silently ignored.

To get around this, split out the two copying steps into a new role
"puppet-setup".  To maintain the namespace, the "run-puppet" module is
renamed to "puppet-run".  Before each call of (now) "puppet-run", make
sure we run "puppet-setup" just on localhost.

Remove the run_once and delegation on "install-ansible-roles"; because
this is now called from the playbook with localhost context.

Change-Id: I3b1cea5a25974f56ea9202e252af7b8420f4adc9
changes/31/749431/4
Ian Wienand 2 years ago
parent
commit
19ea4603f4
  1. 3
      inventory/service/group_vars/puppet.yaml
  2. 8
      playbooks/remote_puppet_afs.yaml
  3. 8
      playbooks/remote_puppet_else.yaml
  4. 2
      playbooks/roles/install-ansible-roles/tasks/main.yaml
  5. 13
      playbooks/roles/puppet-run/README.rst
  6. 10
      playbooks/roles/puppet-run/tasks/main.yaml
  7. 5
      playbooks/roles/puppet-setup-ansible/README.rst
  8. 10
      playbooks/roles/puppet-setup-ansible/tasks/main.yaml
  9. 12
      playbooks/roles/run-puppet/README.rst
  10. 8
      playbooks/service-codesearch.yaml
  11. 8
      playbooks/service-eavesdrop.yaml
  12. 8
      playbooks/service-nodepool.yaml

3
inventory/service/group_vars/puppet.yaml

@ -11,6 +11,3 @@ mgmt_hieradata: /etc/ansible/hosts
mgmt_puppet_module_dir: /etc/puppet/modules
puppet_hieradata_link_dest: /opt/system-config/hieradata
puppet_nolog_sync: '{{ silence_synchronize }}'
ansible_roles:
- puppet

8
playbooks/remote_puppet_afs.yaml

@ -1,8 +1,14 @@
- hosts: 'localhost:!disabled'
name: Install puppet role/modules
strategy: linear
roles:
- puppet-setup-ansible
- hosts: "afs:afsdb:!disabled"
name: "AFS: run puppet on the AFS servers"
strategy: free
roles:
- run-puppet
- puppet-run
- hosts: "mirror-update:!disabled"
name: "Create key for remote vos release"

8
playbooks/remote_puppet_else.yaml

@ -1,5 +1,11 @@
- hosts: 'localhost:!disabled'
name: Install puppet role/modules
strategy: linear
roles:
- puppet-setup-ansible
- hosts: 'puppet:!review:!afs:!afsdb:!puppetmaster*:!nb*:!codesearch:!eavesdrop:!disabled'
name: "Puppet-else: run puppet on all other servers"
strategy: free
roles:
- run-puppet
- puppet-run

2
playbooks/roles/install-ansible-roles/tasks/main.yaml

@ -3,8 +3,6 @@
repo: '{{ ansible_role_src_root }}/src/opendev.org/opendev/ansible-role-{{ ansible_role }}'
dest: '/etc/ansible/roles/{{ ansible_role }}'
force: yes
delegate_to: localhost
run_once: true
loop: '{{ ansible_roles }}'
loop_control:
loop_var: ansible_role

13
playbooks/roles/puppet-run/README.rst

@ -0,0 +1,13 @@
Run puppet on remote servers
Omnibus role that takes care of installing puppet and then running
puppet. Uses include_role so that the installation of the puppet role
can run as the first task, then the puppet role can be used in a
following task.
This role should run after ``puppet-setup-ansible``
.. zuul:rolevar:: manifest
:default: manifests/site.pp
Puppet manifest file to run.

10
playbooks/roles/run-puppet/tasks/main.yaml → playbooks/roles/puppet-run/tasks/main.yaml

@ -1,19 +1,9 @@
# Use include_role instead of roles: so that we can late-bind the roles list
- include_role:
name: iptables
- include_role:
name: install-ansible-roles
- include_role:
name: puppet-install
- include_role:
name: disable-puppet-agent
- name: Run puppet module install
delegate_to: localhost
run_once: true
command:
cmd: bash install_modules.sh
chdir: /etc/puppet
- include_role:
name: puppet

5
playbooks/roles/puppet-setup-ansible/README.rst

@ -0,0 +1,5 @@
Setup Ansible on this host to run puppet on remote hosts.
Import the ansible-roles-puppet role for running puppet on remote
hosts and bring in the repository of required puppet modules.

10
playbooks/roles/puppet-setup-ansible/tasks/main.yaml

@ -0,0 +1,10 @@
- include_role:
name: install-ansible-roles
vars:
ansible_roles:
- puppet
- name: Run puppet module install on bridge
command:
cmd: bash install_modules.sh
chdir: /etc/puppet

12
playbooks/roles/run-puppet/README.rst

@ -1,12 +0,0 @@
Run puppet on remote servers
Omnibus role that takes care of installing the puppet role,
installing puppet and then running puppet. Uses include_role
so that the installation of the puppet role can run as the
first task, then the puppet role can be used in a following
task.
.. zuul:rolevar:: manifest
:default: manifests/site.pp
Puppet manifest file to run.

8
playbooks/service-codesearch.yaml

@ -1,3 +1,9 @@
- hosts: 'localhost:!disabled'
name: Install puppet role/modules
strategy: linear
roles:
- puppet-setup-ansible
- hosts: 'codesearch:!disabled'
name: "codesearch: run puppet on codesearch"
strategy: free
@ -5,5 +11,5 @@
- iptables
- sync-project-config
- pip3
- name: run-puppet
- name: puppet-run
manifest: /opt/system-config/production/manifests/codesearch.pp

8
playbooks/service-eavesdrop.yaml

@ -1,3 +1,9 @@
- hosts: 'localhost:!disabled'
name: Install puppet role/modules
strategy: linear
roles:
- puppet-setup-ansible
- hosts: 'eavesdrop:!disabled'
name: "eavesdrop: run puppet on eavesdrop"
strategy: free
@ -8,5 +14,5 @@
- install-docker
- accessbot
- gerritbot
- name: run-puppet
- name: puppet-run
manifest: /opt/system-config/production/manifests/eavesdrop.pp

8
playbooks/service-nodepool.yaml

@ -8,6 +8,12 @@
- configure-openstacksdk
- nodepool-builder
- hosts: 'localhost:!disabled'
name: Install puppet role/modules
strategy: linear
roles:
- puppet-setup-ansible
- hosts: 'nb03.openstack.org:!disabled'
name: "run puppet on all older servers"
strategy: free
@ -16,7 +22,7 @@
- nodepool-base-legacy
- configure-openstacksdk
- configure-kubectl
- run-puppet
- puppet-run
- hosts: nodepool-launcher:!disabled
name: "Configure nodepool launchers"

Loading…
Cancel
Save