Commit Graph

28 Commits

Author SHA1 Message Date
Monty Taylor
b02c411166 Run puppet on infracloud in a different cron
It takes too long to run puppet on infracloud and it's blocking our
other servers.

Change-Id: I7202617acc5a04e18672b217db53510167d597bd
2016-08-31 14:39:53 +00:00
Spencer Krum
ec4d6cfbeb Run ansible-playbook in timeout
We need this in case it is oomkilled

Change-Id: Ia405dd800850ad46e3e11696079012dfc34a06ea
2016-05-02 18:48:38 -07:00
Spencer Krum
e84b1ddb2b Run less parallelism in ansible
We are running into memory contention and ooming out on
ansible-playbook. Less workers = more ram,  hope.

We can also move puppetmaster.o.o to a host with more ram (it only has
2G right now.) We can also disable the apache/passenger/puppet that is
running on the host.

Change-Id: Id5ade889748d5e8f65a8ea68cc64b0c071c6a627
2016-04-11 13:18:50 -07:00
Colleen Murphy
32f956f268 Add infracloud playbook
Add separate playbook for infacloud nodes to ensure they run in the
correct order - baremetal -> controller -> compute.

Baremetal is intentionally left out, it is not ready yet.

All 'disabled' flags on infracloud hosts are turned off. This patch
landing turns on management of the infracloud.

Co-Authored-By: Yolanda Robla <info@ysoft.biz>
Co-Authored-By: Spencer Krum <nibz@spencerkrum.com>
Change-Id: Ieeda072d45f7454d6412295c2c6a0cf7ce61d952
2016-02-08 18:03:02 -08:00
Monty Taylor
f1b9b864f7 Translate the rest of run_all.sh to ansible
There are a few things that are run as part of run_all.sh that are
not logged into puppet_run_all.log - namely git cloning, module installation
and ansible role installation. Let's go ahead and do those in a playbook
so that we can see their output while we're watching the log file.

Change-Id: I6982452f1e572b7bc5a7b7d167c1ccc159c94e66
2016-01-10 12:38:22 -05:00
Monty Taylor
8ff794f599 Copy system-config and puppet modules everywhere
If we're going to run puppet apply on all of our nodes, they need
the puppet modules installed on them first.

Change-Id: I84b80818fa54d1ddc4d46fead663ed4212bb6ff3
2015-11-24 16:32:00 -05:00
Monty Taylor
d039a62045 Move playbooks out of the puppet module
/etc/ansible/playbooks isn't actually a thing, it was just a convenient
place to put things. However, to enable puppet apply, we're going to
want a group_vars directory adjacent to the playbooks, so having them be
a subdirectory of the puppet module and installed by it is just extra
complexity. Also, if we run out of system-config, then it'll be easier
to work with things like what we do with puppet environments for testing
things.

Change-Id: I947521a73051a44036e7f4c45ce74a79637f5a8b
2015-10-30 11:31:05 +09:00
AzherKhan
ece86b546b Setting ansible playbooks path variable
Created a variable to manage the ansible
playbooks directory path.

Change-Id: Iabb74e9f1aa95828c01b1957849e2b68164d7d20
2015-10-01 12:07:38 +05:30
Clark Boylan
5e283fd6cc Run more puppet agents at a time with ansible
Our current puppet run_all.sh script takes almost 45 minutes to run
puppet agent on all of our nodes. We are using the default concurrency
of 5. Our puppet master should be able to handle a bit more than that.

Run the git/gerrit playbook with a concurrency of 10 and everything else
with a concurrency of 20.

Change-Id: Ia09abb6fa8c699e156aed38d86ce6fd193f3a42d
2015-04-23 09:48:24 -07:00
Clark Boylan
e13b91ee03 Need to force ansible role installs
Ansible galaxy will not overwrite a role that already exists by default.
To keep our ansible puppet role up to date force its installation.

Change-Id: I75eda8600f666895f9be8711d089615e57b3f3c5
2015-03-03 17:20:25 -08:00
Jenkins
ac7a36db8a Merge "Rename roles.yml to roles.yaml" 2015-03-04 00:44:05 +00:00
Jenkins
f41251935d Merge "Install standalone ansible roles" 2015-03-04 00:43:22 +00:00
James E. Blair
7faa62efb1 Rename roles.yml to roles.yaml
All our other YAML files end with .yaml and also:

  http://www.yaml.org/faq.html

Change-Id: I2ecf2e715f704d92861d34db1479fdd29ff816d8
2015-02-26 15:20:38 -08:00
Monty Taylor
b7cfc00620 Install standalone ansible roles
Similar to how we install puppet modules from standalone repos, start
using the ansible-galaxy command to install roles from standalone role
repos.

Change-Id: Iae7d8e4626479e565bc194496de289027a4668ed
Depends-On: I76d5cab55942beaff44ea5f289f93ff6ce772c5f
2015-02-25 20:07:16 -05:00
Gregory Haynes
b9945fccec Use ansible logging during puppet run_all.sh
When ansible-playbook outputs to stdout it does not include timestamps,
but ansible logging does.

Change-Id: Ifb63d34d1dcc7931d734d08dc31223b531d65aa2
2015-02-12 22:55:01 +00:00
Jenkins
84eefdc30a Merge "Put playbooks in a directory" 2015-01-24 20:39:19 +00:00
Jeremy Stanley
bba9b3eef7 Don't bail out in run_all.sh for ansible errors
It's possible for connectivity to a server or manifest application
to break for indeterminate periods of time, so the playbooks should
be run without errexit.

Change-Id: Id4968de3ef8090faa8f97ae8bab29c282d595bbc
2014-12-22 15:49:46 +00:00
Jenkins
40f3b17963 Merge "Bail out in run_all.sh runs when failures happen" 2014-12-19 13:43:28 +00:00
Monty Taylor
75c068f767 Put playbooks in a directory
Listing each of them individually in the puppetmaster.pp file is just
plain crazypants.

Change-Id: Ice621be7d62ec8ff5bc680cf24c237c38f8f30e5
2014-11-28 11:13:41 -05:00
Monty Taylor
6db6ba3724 Run puppet even if git servers fail
The current set of runs make the fleet depend on all git changes
working. The only thing we actually care about is that gerrit doesn't
get updated if git fails.

Change-Id: Id488e14c7dbaddfbffece7b1d8ef65f06b3688d8
2014-10-21 10:46:22 -05:00
Monty Taylor
b4b5724a0b Add AFS
I don't really think this needs any further explanation.

Change-Id: I41378bd320c6c6fad2c981d5cc773486af075c41
2014-10-20 15:13:14 -05:00
Clark Boylan
258d6233a2 Bail out in run_all.sh runs when failures happen
Previously the run_all.sh script would run puppet even if git is not
properly updated or if our puppet modules fail to install. Now set -e in
the script so that any failure to update git or install puppet modules
causes run_all.sh to bail out early.

This is important to ensure that we get consistent and expected results
from puppet when it runs.

Change-Id: Icb3fb2a97d11675762b49c57978b08115bfcbc04
2014-10-18 13:43:38 -07:00
James E. Blair
10c12fe979 Change opt/config to opt/system-config
Move the install location of the git checkout of the config repo
to /opt/system-config to make things more consistent for operators.

Actually moving this on the puppetmaster will be a manual step.

Change-Id: Id9297088ae6c76c02e35414433aae2733f9f639c
2014-10-17 12:14:35 -07:00
Monty Taylor
09697cffdf Move ansible puppet code into a module
If the logic is just in a role, it's hard to re-use it in a one-off
manner on the command line. By putting it into a module, we can
run:

  ansible git0* -m puppet

To run puppet on the git farm, for instance.

Also, the file is completely not openstack specific, so do it in
such a way that we can submit it as a module upstream.

Change-Id: I35b2850e02ec5da2b41ad14eec9fd6d5a356bc93
2014-07-05 10:17:56 -07:00
Monty Taylor
db57161a47 Fix run puppet cronjob.
ansible-playbook is in /usr/local, but that's not in the cron job path.

Also, although there is an ansible log setting in ansible.cfg, the
ansible-playbook command still outputs stuff. We don't want cronspam,
so add the redirection to the log file back in.

Change-Id: Id585c11cca4cbd7e1ba26adbfbe22af650ca2b50
2014-07-04 21:33:25 -07:00
Monty Taylor
034f37c32a Use ansible instead of direct ssh calls
Instead of a shell script looping over ssh calls, use a simple
ansible playbook. The benefit this gets is that we can then also
script ad-hoc admin tasks either via playbooks or on the command
line. We can also then get rid of the almost entirely unused
salt infrastructure.

Change-Id: I53112bd1f61d94c0521a32016c8a47c8cf9e50f7
2014-07-04 10:01:08 -07:00
Monty Taylor
21ab83c0b5 Add some system logging to run_all
When we want to watch run_all happen, it's hard, because there is
no logging. To fix that - make there be some logging. Then, rotate
the logs.

Change-Id: I0eed7aeeec0ff21e58d57d6385cc59b74bbf31fb
2014-04-18 14:16:16 -07:00
Monty Taylor
edaa31ebbd Add keys and script for puppet over ssh
In anticipation of driving puppet over ssh, we need keys on the hosts
and the scripts on the master. Don't turn them on yet, because we want
to be able to do some by-hand testing of the mechanism.

Change-Id: I2c353777e2f8fb5a2e733ce405ba40427ce901e5
2014-04-15 20:24:16 -07:00