362 Commits

Author SHA1 Message Date
James E. Blair
76c7720df1 Install openshift module on bridge
This is so that we can use the ansible k8s_raw module on bridge
to interact with k8s clusters.

Change-Id: I1bc0de734a8635db528ce159bad6710246309fb9
2018-12-20 13:02:40 -08:00
Zuul
686651c75d Merge "Import install-docker role" 2018-12-18 17:08:15 +00:00
Zuul
97afa829ae Merge "Collect syslogs from nodes in ansible tests" 2018-12-18 17:08:12 +00:00
Zuul
1079fc5cbf Merge "Set iptables forward drop by default" 2018-12-18 17:08:11 +00:00
Clark Boylan
a8d35bb4bd Copy pasta the debian base server bits, don't include them
The arm64 nodes install special kernels so we have a dedicated
base-server task list for them. To reduce duplication we were then
include_tasks: Debian.yaml but this seems to result in the ansible play
crashing there and continuing with the next play in the playbook as if
there were no failure/error.

This is concerning but to deal with this in the present lets copy pasta
the debian bits so things hopefully work again then go from there.

Logs of this occurring:

  2018-12-14 20:54:28,515 p=11685 u=root |  TASK [base-server : Install HWE kernel for arm64] ******************************
  2018-12-14 20:54:28,515 p=11685 u=root |  Friday 14 December 2018  20:54:28 +0000 (0:00:14.672)       0:08:06.479 *******
  2018-12-14 20:54:32,564 p=11685 u=root |  ok: [mirror01.london.linaro-london.openstack.org]
  2018-12-14 20:54:32,747 p=11685 u=root |  ok: [nb03.openstack.org]
  2018-12-14 20:54:32,843 p=11685 u=root |  ok: [mirror01.nrt1.arm64ci.openstack.org]
  2018-12-14 20:54:33,727 p=11685 u=root |  ok: [mirror01.cn1.linaro.openstack.org]
  2018-12-14 20:54:33,777 p=11685 u=root |  TASK [base-server : Include generic Debian tasks] ******************************
  2018-12-14 20:54:33,778 p=11685 u=root |  Friday 14 December 2018  20:54:33 +0000 (0:00:05.262)       0:08:11.741 *******
  2018-12-14 20:54:34,023 p=11685 u=root |  PLAY [Base: configure OpenStackSDK on bridge] **********************************
  2018-12-14 20:54:34,052 p=11685 u=root |  TASK [include_role : configure-openstacksdk] ***********************************

Change-Id: I20dbd5b4c768c967c82f786a7cb1d5261bf5b494
2018-12-14 13:36:07 -08:00
Ian Wienand
f07bf2a507 Import install-docker role
This is a role for installing docker on our control-plane servers.

It is based on install-docker from zuul-jobs.

Basic testinfra tests are added; because docker fiddles the iptables
rules in magic ways, the firewall testing is moved out of the base
tests and modified to partially match our base firewall configuration.

Change-Id: Ia4de5032789ff0f2b07d4f93c0c52cf94aa9c25c
2018-12-14 11:30:47 -08:00
Ian Wienand
860b0f9773 Collect syslogs from nodes in ansible tests
This collects syslogs from nodes running in our ansible gate tests.
The node's logs are grouped under a "hosts" directory (the bridge.o.o
logs are moved there for consistentcy too).

Change-Id: I3869946888f09e189c61be4afb280673aa3a3f2e
2018-12-14 10:33:27 -08:00
Clark Boylan
94eb7e5d2b Set iptables forward drop by default
Docker wants to set FORWARD DROP but our existing rules set FORWARD
ACCEPT. To avoid these two services fighting over each other and to
simplify testing lets default to FORWARD DROP too.

None of our servers should act as routers currently. If we resurrect
infracloud or if we deploy k8s this may change but today this should be
fine and be a safer ruleset.

Change-Id: I5f19233129cf54eb70beb335c7b6224f0836096c
2018-12-14 10:33:26 -08:00
Jeremy Stanley
a507b6b401 Add lists.opendev.org to Mailman
Set up the initial boilerplate to enable addition of new
project-neutral Mailman mailing lists on lists.opendev.org.

Change-Id: I8cad4149bdd7b51d10f43b928cdb9362d4bde835
2018-12-13 20:36:08 +00:00
Zuul
2cfe6061a8 Merge "Enable ARA reports for system-config bridge CI jobs" 2018-12-12 23:19:02 +00:00
Zuul
5be026ccc7 Merge "Add support for enabling the ARA callback plugin in install-ansible" 2018-12-12 23:19:00 +00:00
Zuul
78f802715e Merge "Prefix install_openstacksdk variable" 2018-12-12 23:18:58 +00:00
Zuul
35e5e15ef6 Merge "Configure packages on ubuntu arm servers" 2018-12-10 20:23:36 +00:00
Zuul
f0c54c65c6 Merge "Don't install lxd on our servers" 2018-12-10 20:22:20 +00:00
James E. Blair
7f3963efca Add ze12.openstack.org
We believe the relative_priority change has altered our workload
such that we have smaller jobs starting more frequently.  Since
job starts are limited by the executors, we have developed a backlog
and need another executor to relieve the pressure.

Change-Id: I98052e0135c7ee615f1f187b9d0a250cdd1ff178
2018-12-05 14:08:17 -08:00
Clark Boylan
c44d47db43 Configure packages on ubuntu arm servers
We have an arm specific task here to install the HWE kernel. We use
first found to select these tasks which means the default Debian package
setup (unattended upgrades and cleanup) is not installed on our arm
servers.

Fix this by having the arm specific tasks include the generic Debian
tasks.

Change-Id: Ibb57e8b095a4cbd27cc14ef0c5ad45c61edc0679
2018-12-05 10:29:52 -08:00
Clark Boylan
5f6a1c33c0 Don't install lxd on our servers
We don't intend on using lxd on our servers and lxd is causing problems
for unattended upgrades. Lets just make sure these packages aren't
installed and avoid the problems entirely.

Change-Id: I9c6fcf8b0072c23ee0127245fa3bb6c3477dcaf5
2018-12-05 10:26:01 -08:00
Zuul
8c984dead9 Merge "Retire the interop-wg mailing list" 2018-12-05 15:10:39 +00:00
Ian Wienand
3bed6e0fd3
Enable ARA reports for system-config bridge CI jobs
This change takes the ARA report from the "inner" run of the base
playbooks on our bridge.o.o node and publishes it into the final log
output.  This is then displayed by the middleware.

Create a new log hierarchy with a "bridge.o.o" to make it clear the
logs here are related to the test running on that node.  Move the
ansible config under there too.

Change-Id: I74122db09f0f712836a0ee820c6fac87c3c9c734
2018-12-04 17:46:47 -05:00
David Moreau Simard
35e87d6879
Add support for enabling the ARA callback plugin in install-ansible
This change enables the installation of the ARA callback plugin in
the install-ansible role. It does not take care of any web reporting
capabilities.

ARA will not be installed and set up by default.
It can be installed and configured by setting
"install_ansible_enable_ara" to "true".

Co-Authored-By: David Moreau-Simard <dmsimard@redhat.com>
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
Change-Id: Iea84ec8e23ca2e3f021aafae4e89c764f2e05bd2
2018-12-04 17:46:47 -05:00
David Moreau Simard
dd554dbd02
Prefix install_openstacksdk variable
Rename install_openstacksdk to install_ansible_opensatcksdk to make it
clear this is part of the install-ansible role, and it's the
openstacksdk version used with ansible (might be important if we
switch to virtualenvs). This also clears up inconsistency when we add
ARA install options too.

Change-Id: Ie8cb3d5651322b3f6d2de9d6d80964b0d2822dce
2018-12-04 17:46:47 -05:00
Zuul
56ee3a67ba Merge "bridge.o.o : install ansible 2.7.3" 2018-12-04 20:37:49 +00:00
Zuul
bb4fa8335f Merge "Shut down openstack general, dev, ops and sigs mls" 2018-12-04 00:51:27 +00:00
Zuul
41fb4a9248 Merge "Tighten permissions on zone keys" 2018-12-03 23:38:07 +00:00
James E. Blair
3706754b6b Don't import tasks in iptables reload and use listen
This syntax doesn't work in Ansible 2.8.0.  Futher, we can use
"listen" to collapse the notify to a single item (at the
expense of duplicating the when clause in the handlers).

Change-Id: I05e2d32f4e1e692ac528a7254c6e3be2858ebacf
2018-12-03 08:59:30 -08:00
Monty Taylor
330ffb394b
Update the current-context to valid context
The current-context field needs to reference a defined context. The file
otherwise defines only one "vexxhost-sjc1". Set current-context to that
context.

Change-Id: I1d8991efb5d546f007146fd2fa86ce2b2aeed286
2018-11-30 15:00:08 -06:00
Jeremy Stanley
8017415779 Retire the interop-wg mailing list
This list's owners have asked for it to be shut down, as they will
be using an [interop-wg] tag on the new openstack-discuss ML for
future communication. Once this merges (so that Puppet won't
recreate it), the list can be removed with the `rmlist` utility
(this will still leave the archives available but will remove it
from the list index and no longer accept subscriptions/posts).

Set the old list address as an alias for the new openstack-discuss
ML so that replies to previous messages from the list will be routed
there for the foreseeable future.

Change-Id: Ib5fd5aece2465d569e0e7c180ee14ba94882f2b7
2018-11-30 18:39:16 +00:00
Jeremy Stanley
e9d49b4839 Shut down openstack general, dev, ops and sigs mls
The general openstack, openstack-dev, openstack-operators and
openstack-sigs mailing lists have been deprecated since November 19
and are slated to be removed on December 3. Merging this on that
date will ensure any further replies to messages from those lists
are rerouted to the new openstack-discuss mailing list for the
foreseeable future.

The openstack-tc list is included in this batch as it has already
been closed down with a recommendation to send further such
communications to the openstack-discuss ML.

Additionally remove the Puppet mailman resource for the
openstack-sigs ML so it won't be automatically recreated after it
gets deleted (the other lists predate our use of Puppet for this
purpose).

Clean up the corresponding -owner spam rejection aliases since these
addresses will no longer be accepting E-mail anyway.

Change-Id: I9a7fae465c3f6bdcf3ebbadb8926eb4feb8fad79
2018-11-30 18:22:00 +00:00
James E. Blair
2bc9bc8925 Disable openstack inventory plugin
We don't use this anymore, remove it from our config.

Change-Id: I561a6942978fca67d8f83059a957f45540ea52d7
2018-11-30 09:25:09 -08:00
Ian Wienand
77acd56dc7 bridge.o.o : install ansible 2.7.3
This installs Ansible 2.7.3 on bridge.o.o to incorporate fixes for [1]
which is currently stopping the cloud-launcher from running.

Currently every run it hits citycloud Lon1 and tries to delete it's
router

 TASK [cloud-launcher : Processing router openstackci-router1 for openstackci-citycloud Lon1] ***
 Monday 12 November 2018  04:07:48 +0000 (0:00:00.430)       0:07:45.811 *******
 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Error
 detaching interface from router c7197a8f-096a-4488-a3ae-16fdce0ea580
 ...  cannot be deleted, as it is required by one or more floating
 IPs."}

Although it doesn't succeed, it's probably better that it isn't even
trying...

A prior version of this installed the unreleased stable branch to
bring this in, but didn't end up with enough reviews.  I've left
behind how to do that as a breadcrumb should we need to do similar in
the future (we do seem to have a nack of tickling Ansible bugs :)

[1] 951572bec1

Change-Id: I8f112ba994040c52c7b3c7ee6fd6f5a69fd22919
2018-11-30 20:38:05 +11:00
Zuul
e4f569e7c3 Merge "Blackhole messages to openstack-ko-owner@l.o.o" 2018-11-29 16:13:07 +00:00
Clark Boylan
15b19ace2c Nodepool group no longer hosts zookeeper
Remove the zookeeper tcp firewall rules from the nodepool group vars
file as we have dedicated zookeeper servers now. These rules are not
helpful.

Change-Id: I08c2596b8f459fe59d45b0f01e002b9e4b4186d4
2018-11-28 16:47:19 -08:00
James E. Blair
6368113ec9 Add kube config to nodepool servers
This adds connection information for an experimental kubernetes
cluster hosted in vexxhost-sjc1 to the nodepool servers.

Change-Id: Ie7aad841df1779ddba69315ddd9e0ae96a1c8c53
2018-11-28 16:24:53 -08:00
Ian Y. Choi
72781811aa Blackhole messages to openstack-ko-owner@l.o.o
The OpenStack Korean mailing list's owner address have
become overrun by the same mass spam we've seen hitting our other ML
owner addresses. Add a blackhole alias for it.

Change-Id: Ia6c7e6701a69ee56076062aa85f8699121648501
2018-11-29 02:23:35 +09:00
Jeremy Stanley
33ec337b42 Blackhole messages to openstack-sigs-owner@l.o.o
The OpenStack SIGS mailing list's owner address is starting to
become overrun by the same mass spam we've seen hitting our other ML
owner addresses. Add a blackhole alias for it.

Change-Id: Iefc5b5fa600c5d1de75d3302c8ddf0e1a03301e5
2018-11-19 16:16:33 +00:00
Jeremy Stanley
6c406f825b Tighten permissions on zone keys
Remove world-readable/traversable bits from permissions on the BIND
DNSSEC keys directory and the keys themselves (not actually
necessary for the public key files, but added for consistency as
they share a directory with the private keys). Note that this
matches the permissions and ownership of the existing
adns1.openstack.org server.

Change-Id: I015777ee346fefcaa92e64ad2ee88a41c7ea9bde
2018-11-14 12:44:09 +00:00
James E. Blair
3bb6841b33 Fix key filename on master ns
The keys should have a 'K' at the start.

Change-Id: I873aed771448005877eb1fdf5dc739521bf39889
2018-11-14 10:39:57 +01:00
Jeremy Stanley
4fb2143f3c Blackhole messages to edge-computing-owner@l.o.o
The OpenStack edge-computing mailing list's owner address is
starting to become overrun by the same mass spam we've seen hitting
our other ML owner addresses. Add a blackhole alias for it.

Change-Id: I97a2db5d0565cc166604352e397f580ea2d9e767
2018-11-12 10:15:02 +00:00
Zuul
78c6860192 Merge "run_cloud_launcher.sh : generate runtime stats" 2018-11-09 05:23:02 +00:00
Ian Wienand
06da49c6e2 bridge.o.o: Use latest openstacksdk
Similar to the pinning introduced in
Ic465efb637c0a1eb475f04b0b0e356d8797ecdeb, use the "latest"
openstacksdk package and allow for passing of pinned versions if
required.

Update the devel test to also use the master of opensatcksdk

Change-Id: I4b437ca9024c87903bdd3569c8309cde725ce28e
2018-11-08 09:50:58 +11:00
Ian Wienand
24c81fb0c3 Pin bridge.o.o to ansible 2.7.0, add devel testing job
This adds arguments to "install-ansible" to allow us to specify the
package name and version.

This is used to pin bridge.o.o to 2.7.0 (see
I9cf4baf1b15893f0c677567f5afede0d0234f0b2).

A new job is added to test against the ansible-devel branch. Added as
voting for now, until it proves to be a concern.

Change-Id: Ic465efb637c0a1eb475f04b0b0e356d8797ecdeb
2018-11-08 09:50:53 +11:00
Ian Wienand
e32a9fdd0f run_cloud_launcher.sh : generate runtime stats
Similar to run_all.sh (I299c0ab5dc3dea4841e560d8fb95b8f3e7df89f2),
produce a runtime stat for each run of the cloud launcher.

Although it won't directly highlight errors, problems tend to end this
playbook early.  When graphed with grafana, we could have noticed a
large drop in the average runtime which would have suggested a
problem.

Change-Id: I8e5371cbc94e9a803ea5e64ae94aca293b834c73
2018-11-08 08:43:40 +11:00
Ian Wienand
f295b5b44a Update citycloud project details
These names were taken from the citycloud web interface RC file, but
actually match what we already have in
playbooks/templates/clouds/nodepool_clouds.yaml.j2

Testing with this I can authenticate to openstackzuul-citycloud

Change-Id: Ic7aeb5c3a96e5594b8c9c396daaad7e79c1f5c63
2018-11-07 11:48:17 +11:00
Monty Taylor
cad774e65a
interface is not an auth option
interface indicates which of the public, internal or admin endpoints a
user wants to consume.

Change-Id: I061200bbf4477ab53ec6431c71baa8dda6bea6b5
2018-11-06 15:49:23 -06:00
Monty Taylor
214662a424 Install latest openstacksdk on bridge
It's designed to always be used from the latest version.

This trips an ansible lint rule (ANSIBLE0010) which we can ignore, as
we often have pip things that we want to install the latest release
of automatically.

Change-Id: Ieac93ab3a555f2423d4fbcf101d6d9681ae0e497
2018-11-07 06:20:33 +11:00
Monty Taylor
20d966a9db
Add Lon1 and Sto2 to openstackzuul cloud entry
These are missing, so cloud launcher is sad.

Change-Id: Ie74259a3e6acc2d6fc6c40ac42e0c80997208f85
2018-11-06 11:19:44 -06:00
Zuul
9679f57225 Merge "Add Arm64 CI cloud" 2018-11-06 01:21:48 +00:00
James E. Blair
d15c6166c3 adns: Set zone directory permissions
Bind needs to be able to write to the zone directories in order
to sign the zones.

Change-Id: I5649c28c6f7d8d98e0eca3c9c4da5d7312198b5c
2018-11-05 09:02:55 -08:00
James E. Blair
dae1a0351c Configure opendev nameservers using ansible
Change-Id: Ie6430053159bf5a09b2c002ad6a4f84334a5bca3
2018-11-02 13:49:38 -07:00
James E. Blair
90e6088881 Configure adns1.opendev.org server via ansible
Change-Id: Ib4d3cd7501a276bff62e3bc0998d93c41f3ab185
2018-11-02 13:49:38 -07:00