We are running into memory contention and ooming out on
ansible-playbook. Less workers = more ram, hope.
We can also move puppetmaster.o.o to a host with more ram (it only has
2G right now.) We can also disable the apache/passenger/puppet that is
running on the host.
Change-Id: Id5ade889748d5e8f65a8ea68cc64b0c071c6a627
Add separate playbook for infacloud nodes to ensure they run in the
correct order - baremetal -> controller -> compute.
Baremetal is intentionally left out, it is not ready yet.
All 'disabled' flags on infracloud hosts are turned off. This patch
landing turns on management of the infracloud.
Co-Authored-By: Yolanda Robla <info@ysoft.biz>
Co-Authored-By: Spencer Krum <nibz@spencerkrum.com>
Change-Id: Ieeda072d45f7454d6412295c2c6a0cf7ce61d952
There are a few things that are run as part of run_all.sh that are
not logged into puppet_run_all.log - namely git cloning, module installation
and ansible role installation. Let's go ahead and do those in a playbook
so that we can see their output while we're watching the log file.
Change-Id: I6982452f1e572b7bc5a7b7d167c1ccc159c94e66
If we're going to run puppet apply on all of our nodes, they need
the puppet modules installed on them first.
Change-Id: I84b80818fa54d1ddc4d46fead663ed4212bb6ff3
/etc/ansible/playbooks isn't actually a thing, it was just a convenient
place to put things. However, to enable puppet apply, we're going to
want a group_vars directory adjacent to the playbooks, so having them be
a subdirectory of the puppet module and installed by it is just extra
complexity. Also, if we run out of system-config, then it'll be easier
to work with things like what we do with puppet environments for testing
things.
Change-Id: I947521a73051a44036e7f4c45ce74a79637f5a8b
Our current puppet run_all.sh script takes almost 45 minutes to run
puppet agent on all of our nodes. We are using the default concurrency
of 5. Our puppet master should be able to handle a bit more than that.
Run the git/gerrit playbook with a concurrency of 10 and everything else
with a concurrency of 20.
Change-Id: Ia09abb6fa8c699e156aed38d86ce6fd193f3a42d
Ansible galaxy will not overwrite a role that already exists by default.
To keep our ansible puppet role up to date force its installation.
Change-Id: I75eda8600f666895f9be8711d089615e57b3f3c5
Similar to how we install puppet modules from standalone repos, start
using the ansible-galaxy command to install roles from standalone role
repos.
Change-Id: Iae7d8e4626479e565bc194496de289027a4668ed
Depends-On: I76d5cab55942beaff44ea5f289f93ff6ce772c5f
It's possible for connectivity to a server or manifest application
to break for indeterminate periods of time, so the playbooks should
be run without errexit.
Change-Id: Id4968de3ef8090faa8f97ae8bab29c282d595bbc
The current set of runs make the fleet depend on all git changes
working. The only thing we actually care about is that gerrit doesn't
get updated if git fails.
Change-Id: Id488e14c7dbaddfbffece7b1d8ef65f06b3688d8
Previously the run_all.sh script would run puppet even if git is not
properly updated or if our puppet modules fail to install. Now set -e in
the script so that any failure to update git or install puppet modules
causes run_all.sh to bail out early.
This is important to ensure that we get consistent and expected results
from puppet when it runs.
Change-Id: Icb3fb2a97d11675762b49c57978b08115bfcbc04
Move the install location of the git checkout of the config repo
to /opt/system-config to make things more consistent for operators.
Actually moving this on the puppetmaster will be a manual step.
Change-Id: Id9297088ae6c76c02e35414433aae2733f9f639c
If the logic is just in a role, it's hard to re-use it in a one-off
manner on the command line. By putting it into a module, we can
run:
ansible git0* -m puppet
To run puppet on the git farm, for instance.
Also, the file is completely not openstack specific, so do it in
such a way that we can submit it as a module upstream.
Change-Id: I35b2850e02ec5da2b41ad14eec9fd6d5a356bc93
ansible-playbook is in /usr/local, but that's not in the cron job path.
Also, although there is an ansible log setting in ansible.cfg, the
ansible-playbook command still outputs stuff. We don't want cronspam,
so add the redirection to the log file back in.
Change-Id: Id585c11cca4cbd7e1ba26adbfbe22af650ca2b50
Instead of a shell script looping over ssh calls, use a simple
ansible playbook. The benefit this gets is that we can then also
script ad-hoc admin tasks either via playbooks or on the command
line. We can also then get rid of the almost entirely unused
salt infrastructure.
Change-Id: I53112bd1f61d94c0521a32016c8a47c8cf9e50f7
When we want to watch run_all happen, it's hard, because there is
no logging. To fix that - make there be some logging. Then, rotate
the logs.
Change-Id: I0eed7aeeec0ff21e58d57d6385cc59b74bbf31fb
In anticipation of driving puppet over ssh, we need keys on the hosts
and the scripts on the master. Don't turn them on yet, because we want
to be able to do some by-hand testing of the mechanism.
Change-Id: I2c353777e2f8fb5a2e733ce405ba40427ce901e5