44 Commits

Author SHA1 Message Date
Jeremy Stanley
49601213fe Allow DNS zone transfers from ns1/ns2.opendev.org
This was likely missed in bootstrapping. Temporarily allow all
authoritative slaves (opendev as well as openstack) to perform zone
transfers over 53/tcp on either silent master nameserver.

Change-Id: I68455a1d4fa5042da14b3c2e0747af00effad0da
2019-01-02 16:31:56 +00:00
James E. Blair
7f3963efca Add ze12.openstack.org
We believe the relative_priority change has altered our workload
such that we have smaller jobs starting more frequently.  Since
job starts are limited by the executors, we have developed a backlog
and need another executor to relieve the pressure.

Change-Id: I98052e0135c7ee615f1f187b9d0a250cdd1ff178
2018-12-05 14:08:17 -08:00
Clark Boylan
15b19ace2c Nodepool group no longer hosts zookeeper
Remove the zookeeper tcp firewall rules from the nodepool group vars
file as we have dedicated zookeeper servers now. These rules are not
helpful.

Change-Id: I08c2596b8f459fe59d45b0f01e002b9e4b4186d4
2018-11-28 16:47:19 -08:00
James E. Blair
6368113ec9 Add kube config to nodepool servers
This adds connection information for an experimental kubernetes
cluster hosted in vexxhost-sjc1 to the nodepool servers.

Change-Id: Ie7aad841df1779ddba69315ddd9e0ae96a1c8c53
2018-11-28 16:24:53 -08:00
James E. Blair
dae1a0351c Configure opendev nameservers using ansible
Change-Id: Ie6430053159bf5a09b2c002ad6a4f84334a5bca3
2018-11-02 13:49:38 -07:00
James E. Blair
90e6088881 Configure adns1.opendev.org server via ansible
Change-Id: Ib4d3cd7501a276bff62e3bc0998d93c41f3ab185
2018-11-02 13:49:38 -07:00
Clark Boylan
0f8e7a91bb Nodepool.o.o is no longer a thing, remove it
We've only been using nodepool.o.o as a zookeeper server for the past
year or so. Last week we transitioned to a three node zookeeper cluster
and stopped using nodepool.o.o. This server has since been deleted.

This is the last bit of cleanup to remove it from config management.

Change-Id: I9d0363393ed20ee59f40b210ea14fb105a492e20
2018-11-01 12:09:08 -07:00
Ian Wienand
97a3ab9bf3 Add statsd metrics for ansible runs
Add some coarse-grained statsd tracking for the global ansible runs.
Adds a timer for each step, along with an overall timer.

This adds a single argument so that we only try to run stats when
running from the cron job (so if we're debugging by hand or something,
this doesn't trigger).  Graphite also needs to accept stats from
bridge.o.o.  The plan is to present this via a simple grafana
dashboard.

Change-Id: I299c0ab5dc3dea4841e560d8fb95b8f3e7df89f2
2018-09-10 14:49:45 +10:00
James E. Blair
00a4b7ae30 Allow ns servers to connect to adns
These firewall rules were missed in the conversion from puppet
to ansible.

Change-Id: I38c348542a568dc6c1a175116753d16e02e7e2dc
2018-09-06 08:50:45 -07:00
Clark Boylan
09288c7c37 Manage clouds.yaml files in ansible
This manages the clouds.yaml files in ansible so that we can get them
updated automatically on bridge.openstack.org (which does not puppet).

Co-Authored-By: James E. Blair <jeblair@redhat.com>
Depends-On: https://review.openstack.org/598378
Change-Id: I2071f2593f57024bc985e18eaf1ffbf6f3d38140
2018-09-04 08:49:00 -07:00
Monty Taylor
eb086094a8 Install limestone CA on hosts using openstacksdk
In order to talk to limestone clouds we need to configure a custom CA.
Do this in ansible instead of puppet.

A followup should add writing out clouds.yaml files.

Change-Id: I355df1efb31feb31e039040da4ca6088ea632b7e
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-08-31 12:17:35 -07:00
James E. Blair
800397c3da base-test: iptables: allow zuul console streaming
This adds a group var which should normally be the empty list but
can be overridden by the test framework to inject additional iptables
rules.  It's used to add the zuul console streaming port.  To
accomplish this, the base+extras pattern is adopted for
iptables public tcp/udp ports.  This means all host/group vars should
use the "extra" form of the variable rather than the actual variable
defined by the role.

Change-Id: I33fe2b7de4a4ba79c25c0fb41a00e3437cee5463
2018-08-29 09:20:42 -07:00
James E. Blair
e37ad40de1 Correct iptables var names
This variable name was translated incorrectly during the transition
from puppet to ansible for iptables.

Change-Id: I865ba7122b215a7f653aa5ed5770a05edbd655a0
2018-08-28 08:45:58 -07:00
Monty Taylor
15663daaf7 Add iptables role
Co-Authored-By: James E. Blair <corvus@inaugust.com>
Change-Id: Id8b347483affd710759f9b225bfadb3ce851333c
Depends-On: https://review.openstack.org/596503
2018-08-27 14:33:32 +00:00
Monty Taylor
83ebf61e07
Be more explicit about puppet paths
puppet wants the code to be in /opt/system-config/production because of
the environment config. bridge just wants /opt/system-config because
it's an ansible server.

Rather than relying on inferring things, just be explicit about what we
want where.

Depends-On: https://review.openstack.org/593134
Change-Id: I9e749d2c50f7d8a7b0681fe48f38f4741c8a8d01
2018-08-17 14:26:22 -05:00
Monty Taylor
a634593a05
Set mgmt_hieradata in puppet group_vars
This is not a variable describing the system-under-management
bridge.openstack.org - it's a variable that is always true for all
systems in the puppet group.

As a result, update the puppet apply test to figure out which directory
we should be copying modules _from_ - since the puppet4 tests will be
unhappy otherwise.

Change-Id: Iddee83944bd85f69acf4fcfde83dc70304386baf
2018-08-17 14:25:50 -05:00
Monty Taylor
5380eb5b6e
Remove purge_apt_sources
This was a setting added for infra cloud that had to do with bootstrap
order. It seems to have been cargo-culted elsewhere. Remove it. Let's be
specific with our sources.list files.

Change-Id: Iefbd59ad20e9fdc450d9a0c4e58b9cf4a89ff5a3
2018-08-17 11:53:52 -05:00
Monty Taylor
92c9a7c869
Clean up puppet variables and playbooks
The puppet playbooks were some of the first we wrote, so they're
slightly wonky.

Remove '---' lines that are completely unnecessary.

Fix indentation.

Move some variables that are the same everywhere into
ansible variables.

Put puppet related variables into the puppet group_vars.

Stop running puppet on localhost in the git playbook.

Change-Id: I2d2a4acccd3523f1931ebec5977771d5a310a0c7
2018-08-17 09:41:12 -05:00
Monty Taylor
1a8c2f66da
Move /opt/system-config/production to /opt/system-config
The production directory is a relic from the puppet environment concept,
which we do not use. Remove it.

The puppet apply tests run puppet locally, where the production
environment is still needed, so don't update the paths in the
tools/prep-apply.sh.

Depends-On: https://review.openstack.org/592946
Change-Id: I82572cc616e3c994eab38b0de8c3c72cb5ec5413
2018-08-17 09:41:02 -05:00
Zuul
f3036203c3 Merge "Remove base.yaml things from openstack_project::server" 2018-08-17 10:43:53 +00:00
Monty Taylor
bab6fcad3c
Remove base.yaml things from openstack_project::server
Now that we've got base server stuff rewritten in ansible, remove the
old puppet versions.

Depends-On: https://review.openstack.org/588326
Change-Id: I5c82fe6fd25b9ddaa77747db377ffa7e8bf23c7b
2018-08-16 17:25:10 -05:00
Monty Taylor
4b7252ce8b
Add raw blocks to storyboard exim routers
The exim config chunk has a {{ in it, which makes the ansible jinja
very cranky. Add in a raw block so it doesn't try to understand the
exim.

Change-Id: If49d976e503b6ebe236a2d2c6077cce96783e102
2018-08-16 16:29:02 -05:00
James E. Blair
40c6e6d7ad Template all exim routers
So that we can have complete control of the router order, always
template the full set of routers, including the "default" ones.
So that it's easy to use the defaults but put them in a different
order, define each router in its own variable which can be used
in host or group vars to "copy" that router in.

Apply this change to lists, firehose, and storyboard, all of which
have custom exim routers.  Note that firehose intentionally has
its localuser router last.

Change-Id: I737942b8c15f7020b54e350db885e968a93f806a
2018-08-16 13:49:55 -07:00
Monty Taylor
f78f871afe
Make a firehose group with firehose01 in it
We want to configure firehose logically as the firehose service, but the
host that is in the group is called firehose01.openstack.org. Make a
group and put the config variables for firehose into it.

Change-Id: I17c8e8a72f41c5e2730af81f70cef81dd3ed7bca
2018-08-16 15:11:20 -05:00
Monty Taylor
b8f4081c2e
Use ansible group vars for futureparser flag
Now that we're running with ansible, we can set the futureparser varible
in the group_vars for the futureparser group and stop passing it as a
parameter explicitly.

Change-Id: I41fe283e96bb48a17f2acfe2ffd939223b5345e7
2018-08-16 14:02:50 -05:00
Monty Taylor
0d1f235fce
Add exim config for firehose and storyboard
In order to get puppet out of the business of mucking with exim and
fighting ansible, finish moving the config to ansible.

This introduces a storyboard group that we can use to apply the exim
config across both servers. It also splits the base playbook so that we
can avoid running exim on the backup servers. And we set
purge_apt_sources the same as was set in puppet. We should probably
remove it though, since none of us have any clue why it's here.

Change-Id: I43ee891a9c1beead7f97808208829b01a0a7ced6
2018-08-15 15:11:48 -05:00
Monty Taylor
4cca3f8d2a
Add lists exim config to ansible
The mailing list servers have a more complex exim config. Put the
routers and transports into ansible variables.

While we're doing it, role variables with an exim_ prefix - since 'routers'
as a global variable might be a little broad.

iteritems isn't a thing in python3, only items.

We need to escape the exim config with ${if or{{ - because of the {{
which looks like jinja. Wrap it in a {% raw %} block.

Getting the yaml indentation right for things here is non-trivial. Make
them strings instead.

Add a README.rst file - and use the zuul:rolevar construct in it,
because it's nice.

Change-Id: Ieccfce99a1d278440c5baa207479a1887898298e
2018-08-15 15:11:48 -05:00
Zuul
ee72c7e3c3 Merge "Remove old inactive users" 2018-08-10 21:16:21 +00:00
Zuul
411b2ccc93 Merge "Make integration tests works" 2018-08-10 19:30:23 +00:00
Monty Taylor
204b36fcfd
Remove old inactive users
We don't really need to keep these in here. We can put a user in the
remove group without them being in this list.

Change-Id: I321d489d4202272e36d25c5b8913ca7cdda25fdd
2018-08-10 12:21:39 -05:00
Monty Taylor
d587307aaf
Make integration tests works
Split base playbook into two plays

The update apt-cache handler from base-repos needs to fire before we run
base-server. Split into two plays so that the handler will fire.

Fix use of first_found

For include_vars, using the lookup version of first_found requires being
explicit about the path to search in as well. We also need to use query
together with loop to get skip to work right.

Extract the list of file locations we look for for distro and platform
specific variables into a variable so that we can reuse it instead of
copy-pasta.

The vim package is vim-nox on ubuntu and vim-minimal on debian.

ntpdate only needs to be enabled on boot, it does not need to be
immediately started. At least, that's what the old puppet was doing and
trying to start it immediately breaks centos integration tests.

emacs-nox is emacs23-nox on trusty.

Change-Id: If3db276a5f6a8f76d7ce8635da8d2cbc316af341
Depends-On: https://review.openstack.org/588326
2018-08-10 12:12:32 -05:00
Monty Taylor
4180e8be03
Move key type into the key string
In translating these from puppet, they key_type was messed up.

Change-Id: I28e9a203961cfc049c6fb0522f38e0a5d5647b16
2018-08-08 08:26:55 -05:00
Monty Taylor
60fecd508d
Install and configure ansible on bridge
There is a shared caching infrastructure in ansible now for inventory
and fact plugins. It needs to be configured so that our inventory access
isn't slow as dirt.

Unfortunately the copy of openstack.py in 2.6 is busted WRT to caching
because the internal API changed ... and we didn't have any test jobs
set up for it. This also includes a fixed copy of the plugin and
installs it into the a plugin dir.

Change-Id: Ie92e5d7eac4b7e4060a4e07cb29c5a6f2a16ae18
2018-08-03 09:05:07 -05:00
Monty Taylor
0bb4232586 Add base playbooks and roles to bootstrap a new server
We want to launch a new bastion host to run ansible on. Because we're
working on the transition to ansible, it seems like being able to do
that without needing puppet would be nice. This gets user management,
base repo setup and whatnot installed. It doesn't remove them from the
existing puppet, nor does it change the way we're calling anything that
currently exists.

Add bridge.openstack.org to the disabled group so that we don't try to
run puppet on it.

Change-Id: I3165423753009c639d9d2e2ed7d9adbe70360932
2018-08-01 14:57:44 -07:00
Paul Belanger
e420c72da7 Stop reporting to puppetdb
We are in the process of shutting down puppetdb.o.o, so stop pushing
reports to it.

Change-Id: Ib27b21c3fb2cd149e57432fd511129a5c8ecc3e9
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-06 16:35:56 -04:00
James E. Blair
820ac0b0bb Stop publishing to puppetdb
It is AWOL while under repair.  Stop publishing there until it
is back in reliable service.

Change-Id: I781bfb32090803edcc027d3bc72ea5719951e9d5
2016-10-31 12:20:08 -07:00
Monty Taylor
a97a3d4c7a Start namespacing ansible group_vars
It's fine right now with 5, but over time if we keep a flat namespae,
which is not necessary, it's just going to get ugly.

Change-Id: I07a143f45f2eb100c231ea1b7dd617b40f8f231c
2016-02-24 11:57:32 -06:00
Monty Taylor
658b0958ff Configure the host to report to puppetdb as
We need to plumb through a configured host setting to report to puppetdb
as so that certs work.

Change-Id: I290ad569283390bac2a74a9991331c9e86821ab7
2016-02-24 11:38:10 -06:00
Monty Taylor
63325581c1 Configure out puppet runs to log to syslog
The puppet ansible module is growing a flag to be able to send stdout to
syslog. It's growing that because we want to use it. Let's.

Change-Id: I22b1d0e1fb635f2c626d75a11764725c8753bf24
2016-01-21 18:36:16 -05:00
Monty Taylor
4e62f20007 Use puppet apply instead of puppet agent
At long last, the day of reckoning is here. Run puppet apply and then
copy the log files back and post them to puppetdb.

Change-Id: I919fea64df0fbb8681e91ac9425b4c43760bb3dd
2016-01-19 18:40:28 -05:00
Monty Taylor
b8b5cf748a Use /opt not /etc as the system-config location
/etc/system-config isn't really a thing.

Change-Id: I8b0598a7645e2dd3505ac239e6194e7f165d2ee7
2016-01-19 11:10:22 -05:00
Monty Taylor
2f9b98b3cd Use the puppet role to copy the puppet code
When we do it as a second playbook, the failure to copy updated code
cannot prevent puppet from running.

Change-Id: I94b06988a20da4c0c2cf492485997ec49c3dca13
Depends-On: I22b7a21778d514a0a1ab04a76f03fdc9c58a05b3
2016-01-19 08:09:01 -05:00
Monty Taylor
43d26acad8 Start copying hiera data everywhere
One step before flipping the switch, start copying hieradata, even
though we're still using agent, so that we can verify as much as we
want.

Change-Id: Iae63fd056cdb17aedd6526b9cbc1d83037ddcbb3
2015-11-24 19:17:35 -05:00
Monty Taylor
1e862a9ade Add some in-tree ansible group vars
As we're using these roles, we'll want to pass potentially different
values to different of our hosts over time. For instance, we may want to
set the jenkins servers to start using puppet apply before we get all
the hosts there. Since we run most of the hosts in a big matching
mechanism, the way we can pass different input values to each host.

Change-Id: I5698355df0c13cd11fe5987787e65ee85a384256
2015-10-30 02:33:27 +00:00