opendev/system-config
Code Issues Proposed changes
e97d031d68
system-config/zuul.d/system-config-run.yaml

850 lines
26 KiB
YAML
Raw Normal View History

Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- job:
name: system-config-run
description: |
Run the "base" playbook for system-config hosts.
This is a parent job designed to be inherited.
abstract: true
pre-run: playbooks/zuul/run-base-pre.yaml
run: playbooks/zuul/run-base.yaml
post-run: playbooks/zuul/run-base-post.yaml
vars:
install_ansible_ara_enable: true
zuul_copy_output: "{{ copy_output | combine(host_copy_output | default({})) }}"
stage_dir: "{{ ansible_user_dir }}/zuul-output"
copy_output:
'/var/log/syslog': logs_txt
'/var/log/messages': logs_txt
'/var/log/docker': logs
Add iptables_extra_allowed_groups This adds a new variable for the iptables role that allows us to indicate all members of an ansible inventory group should have iptables rules added. It also removes the unused zuul-executor-opendev group, and some unused variables related to the snmp rule. Also, collect the generated iptables rules for debugging. Change-Id: I48746a6527848a45a4debf62fd833527cc392398 Depends-On: https://review.opendev.org/728952
2020-05-08 14:15:01 -07:00
'/etc/iptables/rules.v4': logs_txt
'/etc/iptables/rules.v6': logs_txt
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
host-vars:
bridge.openstack.org:
host_copy_output:
'{{ zuul.project.src_dir }}/junit.xml': logs
Fix junit error, add HTML report Specifying the family stops a deprecation warning being output. Add a HTML report and report it as an artifact as well; this is easier to read. Change-Id: I2bd6505c19cee2d51e9af27e9344cfe2e1110572
2020-07-13 14:27:58 +10:00
'{{ zuul.project.src_dir }}/test-results.html': logs
Copy generated inventory to bridge logs This is the inventory generated and used by bridge, copy it into the logs as well. Change-Id: I15d0ddc4c8340735c0332139ddedc06fc05b8269
2020-07-13 13:39:53 +10:00
'{{ zuul.project.src_dir }}/inventory/base/gate-hosts.yaml': logs
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
# Note: the following two jobs implement the variant-based multiple
# inheritance trick. Both of these variants will always apply,
# therefore both parents will appear in the inheritance hierarchy).
- job:
name: system-config-run-containers
parent: system-config-run
# Note: see above re multiple-inheritance.
- job:
name: system-config-run-containers
parent: opendev-buildset-registry-consumer
description: |
Run the "base" playbook for system-config hosts which use
containers.
This is a parent job designed to be inherited. Use this job if
the service in question is container-based. It expects a
buildset registry and pulls images from the intermediate
registry.
- job:
name: system-config-run-base
parent: system-config-run
description: |
Run the "base" playbook on each of the node types
currently in use.
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: xenial
label: ubuntu-xenial
- name: bionic
label: ubuntu-bionic
Add focal to system-config base job Follow-on to addition for executors in I0126f7c77d92deb91711f38a19384a9319955cf5 to keep the base roles focal clean. Change-Id: I40fdf1b2b7d012f4ce5d013528f4460d277e44d4
2020-05-06 06:07:16 +10:00
- name: focal
label: ubuntu-focal
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
files:
- playbooks/
- roles/
- testinfra/
- job:
name: system-config-run-base-ansible-devel
parent: system-config-run-base
description: |
Run the base playbook with the latest ansible.
required-projects:
- name: github.com/ansible/ansible
override-checkout: devel
- name: github.com/philpep/testinfra
- name: openstack/openstacksdk
- name: recordsansible/ara
# NOTE(ianw): 2019-06-11 pinned to stable branch until we handle 1.0
# http://lists.openstack.org/pipermail/openstack-infra/2019-June/006400.html
override-checkout: stable/0.x
vars:
bridge_ansible_name: '{{ ansible_user_dir}}/src/github.com/ansible/ansible'
bridge_ansible_version: null
bridge_openstacksdk_name: '{{ ansible_user_dir }}/src/opendev.org/openstack/openstacksdk'
bridge_openstacksdk_version: null
bridge_ara_name: '{{ ansible_user_dir}}/src/opendev.org/recordsansible/ara'
bridge_ara_version: null
# Although we don't have an arm64 based bridge; Zuul can't currently
# allocate a mixed x86/arm64 situation across clouds. Thus it helps
# to keep this clean so we can run the other tests.
- job:
name: system-config-run-base-arm64
parent: system-config-run
description: |
Run the "base" playbook on ARM64.
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic-arm64
- name: bionic
label: ubuntu-bionic-arm64
files:
- playbooks/
- roles/
- testinfra/
- job:
name: system-config-run-eavesdrop
parent: system-config-run-containers
description: |
Run the playbook for an eavesdrop server.
required-projects:
- opendev/ansible-role-puppet
- opendev/system-config
- openstack/project-config
Add ansible role to manage gerritbot This new ansible role deploys gerritbot with docker-compose on eavesdrop.openstack.org. This way we can run it where the other bots live. Testing is rudimentary for now as we don't really want to connect to a production gerrit and freenode. We check things the best we can. We will want to coordinate deployment of this change with disabling the running service on the gerrit server. Depends-On: https://review.opendev.org/745240 Change-Id: I008992978791ff0a38f92fb4bc529ff643f01dd6
2020-08-04 15:14:28 -07:00
requires:
- accessbot-container-image
- gerritbot-container-image
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: eavesdrop01.openstack.org
label: ubuntu-xenial
vars:
run_playbooks:
- playbooks/service-eavesdrop.yaml
files:
- playbooks/service-eavesdrop.yaml
- playbooks/run-accessbot.yaml
Split inventory into multiple dirs and move hostvars Make inventory/service for service-specific things, including the groups.yaml group definitions, and inventory/base for hostvars related to the base system, including the list of hosts. Move the exisitng host_vars into inventory/service, since most of them are likely service-specific. Move group_vars/all.yaml into base/group_vars as almost all of it is related to base things, with the execption of the gerrit public key. A followup patch will move host-specific values into equivilent files in inventory/base. This should let us override hostvars in gate jobs. It should also allow us to do better file matchers - and to be able to organize our playbooks move if we want to. Depends-On: https://review.opendev.org/731583 Change-Id: Iddf57b5be47c2e9de16b83a1bc83bee25db995cf
2020-05-26 15:46:41 -05:00
- inventory/service/group_vars/eavesdrop.yaml
- inventory/service/group_vars/puppet.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/roles/zuul-user
- playbooks/roles/install-docker
- playbooks/roles/puppet-install/
- playbooks/roles/disable-puppet-agent/
- playbooks/roles/accessbot
- playbooks/roles/logrotate
Add ansible role to manage gerritbot This new ansible role deploys gerritbot with docker-compose on eavesdrop.openstack.org. This way we can run it where the other bots live. Testing is rudimentary for now as we don't really want to connect to a production gerrit and freenode. We check things the best we can. We will want to coordinate deployment of this change with disabling the running service on the gerrit server. Depends-On: https://review.opendev.org/745240 Change-Id: I008992978791ff0a38f92fb4bc529ff643f01dd6
2020-08-04 15:14:28 -07:00
- playbooks/zuul/templates/group_vars/eavesdrop.yaml.j2
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- modules/openstack_project/manifests/eavesdrop.pp
- manifests/eavesdrop.pp
- docker/accessbot/
- testinfra/test_eavesdrop.py
- job:
name: system-config-run-codesearch
parent: system-config-run
description: |
Run the playbook for an codesearch server.
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: codesearch01.openstack.org
label: ubuntu-xenial
required-projects:
- opendev/ansible-role-puppet
- opendev/system-config
- openstack/project-config
files:
- playbooks/install-ansible.yaml
- playbooks/service-codesearch.yaml
Split inventory into multiple dirs and move hostvars Make inventory/service for service-specific things, including the groups.yaml group definitions, and inventory/base for hostvars related to the base system, including the list of hosts. Move the exisitng host_vars into inventory/service, since most of them are likely service-specific. Move group_vars/all.yaml into base/group_vars as almost all of it is related to base things, with the execption of the gerrit public key. A followup patch will move host-specific values into equivilent files in inventory/base. This should let us override hostvars in gate jobs. It should also allow us to do better file matchers - and to be able to organize our playbooks move if we want to. Depends-On: https://review.opendev.org/731583 Change-Id: Iddf57b5be47c2e9de16b83a1bc83bee25db995cf
2020-05-26 15:46:41 -05:00
- inventory/service/group_vars/puppet.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/roles/run-puppet/
- playbooks/roles/install-ansible-roles/
- playbooks/roles/sync-project-config
- playbooks/roles/puppet-install/
- playbooks/roles/disable-puppet-agent/
- modules/openstack_project/manifests/codesearch.pp
- modules/openstack_project/files/resync-hound-config.sh
- manifests/codesearch.pp
vars:
run_playbooks:
- playbooks/service-codesearch.yaml
- job:
name: system-config-run-letsencrypt
parent: system-config-run
description: |
Run the playbook for letsencrypt key acquisition
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: adns-letsencrypt.opendev.org
label: ubuntu-bionic
- name: letsencrypt01.opendev.org
label: ubuntu-bionic
- name: letsencrypt02.opendev.org
label: ubuntu-bionic
vars:
run_playbooks:
- playbooks/service-nameserver.yaml
Rename service-letsencrypt to just letsencrypt This isn't a service, it's a meta thing that we run for different hosts at different times. Change-Id: Ib65665c98afb3ddb94b15346931be88a4b1757d8
2020-05-28 16:43:14 -05:00
- playbooks/letsencrypt.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
host-vars:
Generate ssl check list directly from letsencrypt variables This autogenerates the list of ssl domains for the ssl-cert-check tool directly from the letsencrypt list. The first step is the install-certcheck role that replaces the puppet-ssl_cert_check module that does the same. The reason for this is so that during gate testing we can test this on the test bridge.openstack.org server, and avoid adding another node as a requirement for this test. letsencrypt-request-certs is updated to set a fact letsencrypt_certcheck_domains for each host that is generating a certificate. As described in the comments, this defaults to the first host specified for the certificate and the listening port can be indicated (if set, this new port value is stripped when generating certs as is not necessary for certificate generation). The new letsencrypt-config-certcheck role runs and iterates all letsencrypt hosts to build the final list of domains that should be checked. This is then extended with the letsencrypt_certcheck_additional_domains value that covers any hosts using certificates not provisioned by letsencrypt using this mechanism. These additional domains are pre-populated from the openstack.org domains in the extant check file, minus those openstack.org domain certificates we are generating via letsencrypt (see letsencrypt-create-certs/handlers/main.yaml). Additionally, we update some of the certificate variables in host_vars that are listening on port !443. As mentioned, bridge.openstack.org is placed in the new certcheck group for gate testing, so the tool and config file will be deployed to it. For production, cacti is added to the group, which is where the tool currently runs. The extant puppet installation is disabled, pending removal in a follow-on change. Change-Id: Idbe084f13f3684021e8efd9ac69b63fe31484606
2020-05-18 14:43:16 +10:00
bridge.openstack.org:
host_copy_output:
'/var/lib/certcheck': logs
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
letsencrypt01.opendev.org:
host_copy_output:
'/var/log/acme.sh': logs
letsencrypt02.opendev.org:
host_copy_output:
'/var/log/acme.sh': logs
files:
- playbooks/install-ansible.yaml
Split inventory into multiple dirs and move hostvars Make inventory/service for service-specific things, including the groups.yaml group definitions, and inventory/base for hostvars related to the base system, including the list of hosts. Move the exisitng host_vars into inventory/service, since most of them are likely service-specific. Move group_vars/all.yaml into base/group_vars as almost all of it is related to base things, with the execption of the gerrit public key. A followup patch will move host-specific values into equivilent files in inventory/base. This should let us override hostvars in gate jobs. It should also allow us to do better file matchers - and to be able to organize our playbooks move if we want to. Depends-On: https://review.opendev.org/731583 Change-Id: Iddf57b5be47c2e9de16b83a1bc83bee25db995cf
2020-05-26 15:46:41 -05:00
- inventory/service/group_vars/letsencrypt.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/roles/letsencrypt
- job:
name: system-config-run-lists
parent: system-config-run
description: |
Run the playbook for a list server.
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: lists.openstack.org
label: ubuntu-xenial
required-projects:
- opendev/ansible-role-puppet
- opendev/system-config
files:
- playbooks/install-ansible.yaml
- modules/openstack_project/manifests/lists.pp
Split inventory into multiple dirs and move hostvars Make inventory/service for service-specific things, including the groups.yaml group definitions, and inventory/base for hostvars related to the base system, including the list of hosts. Move the exisitng host_vars into inventory/service, since most of them are likely service-specific. Move group_vars/all.yaml into base/group_vars as almost all of it is related to base things, with the execption of the gerrit public key. A followup patch will move host-specific values into equivilent files in inventory/base. This should let us override hostvars in gate jobs. It should also allow us to do better file matchers - and to be able to organize our playbooks move if we want to. Depends-On: https://review.opendev.org/731583 Change-Id: Iddf57b5be47c2e9de16b83a1bc83bee25db995cf
2020-05-26 15:46:41 -05:00
- inventory/service/host_vars/lists.openstack.org.yaml
- inventory/service/group_vars/puppet
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/roles/run-puppet/
- playbooks/roles/install-ansible-roles/
Move base roles into a base subdir If we move these into a subdir, it cleans up the number of things we nave to files match on. Stop running disable-puppet-agent in base. We run it in run-puppet which should be fine. Change-Id: Ia16adb96b11d25a097490882c4c59a50a0b7b23d
2020-05-26 14:24:20 -05:00
- playbooks/roles/base/exim
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
vars:
run_playbooks:
- playbooks/remote_puppet_else.yaml
- job:
name: system-config-run-nodepool
parent: system-config-run
description: |
Run the playbook for nodepool.
timeout: 3600
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: zk01.opendev.org
label: ubuntu-bionic
- name: nl01.openstack.org
label: ubuntu-xenial
- name: nb04.opendev.org
label: ubuntu-bionic
required-projects:
- opendev/ansible-role-puppet
- opendev/system-config
- openstack/project-config
host-vars:
nl01.openstack.org:
host_copy_output:
'/etc/nodepool/nodepool.yaml': logs
Save zuul and nodepool logs from gate test jobs Let's save our debug logs so we can better observe the system in the gate. Change-Id: Ic80b646e0407d27e43cdb10cb573551999dd01d4
2020-05-20 07:26:27 -07:00
'/var/log/nodepool/launcher-debug.log': logs
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
nb04.opendev.org:
host_copy_output:
'/etc/nodepool/nodepool.yaml': logs
Save zuul and nodepool logs from gate test jobs Let's save our debug logs so we can better observe the system in the gate. Change-Id: Ic80b646e0407d27e43cdb10cb573551999dd01d4
2020-05-20 07:26:27 -07:00
'/var/log/nodepool/builder-debug.log': logs
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
vars:
run_playbooks:
Rename service-letsencrypt to just letsencrypt This isn't a service, it's a meta thing that we run for different hosts at different times. Change-Id: Ib65665c98afb3ddb94b15346931be88a4b1757d8
2020-05-28 16:43:14 -05:00
- playbooks/letsencrypt.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/service-zookeeper.yaml
- playbooks/service-nodepool.yaml
- playbooks/remote_puppet_else.yaml
Run restart playbooks to test they work This runs our zuul and nodepool restart playbooks after the initial service installs in the system-config-run jobs. This will help ensure that they work consistently over time. Fix nodepool restart playbook Change-Id: I953e7222218c5555bb44fccd913eaa5e9374c669
2020-06-16 12:02:11 -05:00
# Test our ad hoc restart playbook works
- playbooks/nodepool_restart.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
files:
- playbooks/install-ansible.yaml
Split inventory into multiple dirs and move hostvars Make inventory/service for service-specific things, including the groups.yaml group definitions, and inventory/base for hostvars related to the base system, including the list of hosts. Move the exisitng host_vars into inventory/service, since most of them are likely service-specific. Move group_vars/all.yaml into base/group_vars as almost all of it is related to base things, with the execption of the gerrit public key. A followup patch will move host-specific values into equivilent files in inventory/base. This should let us override hostvars in gate jobs. It should also allow us to do better file matchers - and to be able to organize our playbooks move if we want to. Depends-On: https://review.opendev.org/731583 Change-Id: Iddf57b5be47c2e9de16b83a1bc83bee25db995cf
2020-05-26 15:46:41 -05:00
- inventory/service/group_vars/nodepool.yaml
- inventory/service/group_vars/nodepool-builder.yaml
- inventory/service/group_vars/nodepool-launcher.yaml
- inventory/service/group_vars/puppet
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/roles/run-puppet/
- playbooks/roles/install-ansible-roles/
- playbooks/roles/configure-openstacksdk/
- playbooks/roles/nodepool
- playbooks/templates/clouds/
Run restart playbooks to test they work This runs our zuul and nodepool restart playbooks after the initial service installs in the system-config-run jobs. This will help ensure that they work consistently over time. Fix nodepool restart playbook Change-Id: I953e7222218c5555bb44fccd913eaa5e9374c669
2020-06-16 12:02:11 -05:00
- playbooks/nodepool_restart.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- testinfra/test_nodepool.py
- job:
name: system-config-run-dns
parent: system-config-run
description: |
Run the playbook for dns.
required-projects:
- opendev/zone-opendev.org
- opendev/zone-zuul-ci.org
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: adns1.opendev.org
label: ubuntu-bionic
- name: ns1.opendev.org
label: ubuntu-bionic
vars:
run_playbooks:
- playbooks/service-nameserver.yaml
host-vars:
adns1.opendev.org:
host_copy_output:
'/etc/bind/named.conf': logs
'/var/lib/bind/zones': logs
files:
- playbooks/install-ansible.yaml
Split inventory into multiple dirs and move hostvars Make inventory/service for service-specific things, including the groups.yaml group definitions, and inventory/base for hostvars related to the base system, including the list of hosts. Move the exisitng host_vars into inventory/service, since most of them are likely service-specific. Move group_vars/all.yaml into base/group_vars as almost all of it is related to base things, with the execption of the gerrit public key. A followup patch will move host-specific values into equivilent files in inventory/base. This should let us override hostvars in gate jobs. It should also allow us to do better file matchers - and to be able to organize our playbooks move if we want to. Depends-On: https://review.opendev.org/731583 Change-Id: Iddf57b5be47c2e9de16b83a1bc83bee25db995cf
2020-05-26 15:46:41 -05:00
- inventory/service/group_vars/adns.yaml
- inventory/service/group_vars/dns.yaml
- inventory/service/host_vars/(ad)?ns\d+.opendev.org.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/zuul/templates/group_vars/adns.yaml.j2
- playbooks/zuul/templates/group_vars/ns.yaml.j2
- playbooks/roles/master-nameserver/
- playbooks/roles/nameserver/
- testinfra/test_adns.py
- testinfra/test_ns.py
- job:
name: system-config-run-backup
parent: system-config-run
description: |
Run the playbook for backup configuration
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: backup01.region.provider.opendev.org
label: ubuntu-bionic
- name: backup-test01.opendev.org
label: ubuntu-bionic
- name: backup-test02.opendev.org
label: ubuntu-xenial
vars:
run_playbooks:
- playbooks/service-backup.yaml
files:
- playbooks/install-ansible.yaml
- playbooks/roles/backup
- playbooks/zuul/templates/host_vars/backup
- testinfra/test_backups.py
- job:
name: system-config-run-mirror-base
parent: system-config-run
abstract: true
description: |
Run the playbook for a mirror node
vars:
run_playbooks:
Rename service-letsencrypt to just letsencrypt This isn't a service, it's a meta thing that we run for different hosts at different times. Change-Id: Ib65665c98afb3ddb94b15346931be88a4b1757d8
2020-05-28 16:43:14 -05:00
- playbooks/letsencrypt.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/service-mirror.yaml
host-vars:
mirror01.openafs.provider.opendev.org:
host_copy_output:
'/var/log/apache2/': logs
'/var/log/acme.sh': logs
Enable ssl on all mirror vhosts Previously we had enabled SSL on our main vhost for the mirrors. Do similar for all of the proxy cache vhosts for docker and other external resources. As part of this change we improve the testing to ensure that the new vhosts are working as expected. One testing specific change to note is the testinfra node names did not match our existing system-config-run job nodenames. This has been corrected. Additionally RHRegistryMirror and QuayMirror may not be working and fixing those is left as a followup. Change-Id: I9dbbd4080c3a2cce4acc39d63244f7a645503553
2020-05-18 11:44:52 -07:00
'/etc/apache2/sites-available/mirror.conf': logs
mirror02.openafs.provider.opendev.org:
host_copy_output:
'/var/log/apache2/': logs
'/var/log/acme.sh': logs
'/etc/apache2/sites-available/mirror.conf': logs
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
files:
- playbooks/install-ansible.yaml
- roles/
Split inventory into multiple dirs and move hostvars Make inventory/service for service-specific things, including the groups.yaml group definitions, and inventory/base for hostvars related to the base system, including the list of hosts. Move the exisitng host_vars into inventory/service, since most of them are likely service-specific. Move group_vars/all.yaml into base/group_vars as almost all of it is related to base things, with the execption of the gerrit public key. A followup patch will move host-specific values into equivilent files in inventory/base. This should let us override hostvars in gate jobs. It should also allow us to do better file matchers - and to be able to organize our playbooks move if we want to. Depends-On: https://review.opendev.org/731583 Change-Id: Iddf57b5be47c2e9de16b83a1bc83bee25db995cf
2020-05-26 15:46:41 -05:00
- inventory/service/group_vars/mirror.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/roles/mirror/
- playbooks/roles/letsencrypt
Rename service-letsencrypt to just letsencrypt This isn't a service, it's a meta thing that we run for different hosts at different times. Change-Id: Ib65665c98afb3ddb94b15346931be88a4b1757d8
2020-05-28 16:43:14 -05:00
- playbooks/letsencrypt.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/service-mirror.yaml
Remove puppet mirror support Remove the separate "mirror_opendev" group and rename it to just "mirror". Update various parts to reflect that change. We no longer deploy any mirror hosts with puppet, remove the various configuration files. Depends-On: https://review.opendev.org/728345 Change-Id: Ia982fe9cb4357447989664f033df976b528aaf84
2020-05-15 13:29:38 +10:00
- playbooks/zuul/templates/group_vars/mirror.yaml.j2
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- testinfra/test_mirror.py
- job:
name: system-config-run-mirror-x86
parent: system-config-run-mirror-base
Run iptables in service playbooks instead of base It's the only part of base that's important to run when we run a service. Run it in the service playbooks and get rid of the dependency on infra-prod-base. Continue running it in base so that new nodes are brought up with iptables in place. Bump the timeout for the mirror job, because the iptables addition seems to have just bumped it over the edge. Change-Id: I4608216f7a59cfa96d3bdb191edd9bc7bb9cca39
2020-05-26 17:41:54 -05:00
timeout: 3600
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: mirror01.openafs.provider.opendev.org
label: ubuntu-bionic
Add focal testing for mirror nodes Change-Id: I64de9a61c5044b93f6ce7e2d31cf51d78fd4ec16
2020-05-12 08:53:25 +10:00
- name: mirror02.openafs.provider.opendev.org
label: ubuntu-focal
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- job:
name: system-config-run-mirror-arm64
parent: system-config-run-mirror-base
timeout: 3600
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic-arm64
- name: mirror01.openafs.provider.opendev.org
label: ubuntu-bionic-arm64
- job:
name: system-config-run-mirror-update
parent: system-config-run
description: |
Run the playbook for a mirror update node
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: mirror-update01.opendev.org
mirror-update: update to focal We want a more recent version of rsync, and upgrading to focal is one easy way to get it, and to also have a base OS with a longer support period. Test it in the gate. Change-Id: I1edf074e5fe788ef75693d2cd172370c05bf4732
2020-06-18 14:24:27 +10:00
label: ubuntu-focal
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
vars:
run_playbooks:
- playbooks/service-mirror-update.yaml
files:
- playbooks/install-ansible.yaml
- roles/
- playbooks/roles/mirror-update/
- playbooks/service-mirror-update.yaml
- testinfra/test_mirror-update.py
- job:
name: system-config-run-docker-registry
parent: system-config-run
description: |
Run the playbook for the docker registry.
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: insecure-ci-registry01.opendev.org
label: ubuntu-bionic
vars:
run_playbooks:
Rename service-letsencrypt to just letsencrypt This isn't a service, it's a meta thing that we run for different hosts at different times. Change-Id: Ib65665c98afb3ddb94b15346931be88a4b1757d8
2020-05-28 16:43:14 -05:00
- playbooks/letsencrypt.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/service-registry.yaml
host-vars:
insecure-ci-registry01.opendev.org:
host_copy_output:
'/var/registry/auth': logs
'/var/registry/certs': logs
files:
- playbooks/install-ansible.yaml
Split inventory into multiple dirs and move hostvars Make inventory/service for service-specific things, including the groups.yaml group definitions, and inventory/base for hostvars related to the base system, including the list of hosts. Move the exisitng host_vars into inventory/service, since most of them are likely service-specific. Move group_vars/all.yaml into base/group_vars as almost all of it is related to base things, with the execption of the gerrit public key. A followup patch will move host-specific values into equivilent files in inventory/base. This should let us override hostvars in gate jobs. It should also allow us to do better file matchers - and to be able to organize our playbooks move if we want to. Depends-On: https://review.opendev.org/731583 Change-Id: Iddf57b5be47c2e9de16b83a1bc83bee25db995cf
2020-05-26 15:46:41 -05:00
- inventory/service/group_vars/registry.yaml
- inventory/service/host_vars/insecure-ci-registry\d+.opendev.org.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/zuul/templates/group_vars/registry.yaml.j2
- playbooks/roles/letsencrypt-create-certs/handlers/restart_zuul_registry.yaml
- playbooks/roles/registry/
- playbooks/roles/pip3/
- playbooks/roles/install-docker/
- testinfra/test_registry.py
- job:
name: system-config-run-etherpad
parent: system-config-run-containers
description: |
Run the playbook for the etherpad servers.
timeout: 3600
requires: etherpad-container-image
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: etherpad01.opendev.org
label: ubuntu-bionic
vars:
run_playbooks:
Rename service-letsencrypt to just letsencrypt This isn't a service, it's a meta thing that we run for different hosts at different times. Change-Id: Ib65665c98afb3ddb94b15346931be88a4b1757d8
2020-05-28 16:43:14 -05:00
- playbooks/letsencrypt.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/service-etherpad.yaml
files:
- playbooks/bridge.yaml
Rename service-letsencrypt to just letsencrypt This isn't a service, it's a meta thing that we run for different hosts at different times. Change-Id: Ib65665c98afb3ddb94b15346931be88a4b1757d8
2020-05-28 16:43:14 -05:00
- playbooks/letsencrypt.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/service-etherpad.yaml
- playbooks/roles/etherpad/
- playbooks/roles/install-docker/
- playbooks/roles/pip3/
- docker/etherpad/
Test etherpad with testinfra This adds simple testing of the etherpad service to testinfra. Change-Id: I3c89a0a92a41cf69d075d6cef99fa12db68b17c6
2020-06-10 08:49:53 -07:00
- testinfra/test_etherpad.py
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- job:
name: system-config-run-gitea
parent: system-config-run-containers
description: |
Run the playbook for the gitea servers.
timeout: 3600
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: gitea-lb01.opendev.org
label: ubuntu-bionic
- name: gitea99.opendev.org
label: ubuntu-bionic
required-projects:
- openstack/project-config
- opendev/system-config
vars:
run_playbooks:
Rename service-letsencrypt to just letsencrypt This isn't a service, it's a meta thing that we run for different hosts at different times. Change-Id: Ib65665c98afb3ddb94b15346931be88a4b1757d8
2020-05-28 16:43:14 -05:00
- playbooks/letsencrypt.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/service-gitea-lb.yaml
- playbooks/service-gitea.yaml
- playbooks/manage-projects.yaml
Deal with gitea pagination of repo lists We list gitea repos to determine if we need to create a repo. If the repo isn't listed by gitea we create it. New gitea paginates these listings so we were only getting 30 repos listed when we had far more. This resulted in us trying to create repos which already exist which is a gitea http 409 error. Fix this by paging through the listings until we've seen all the repos. This should give us a complete listing. To test this we run our manage-projects playbook twice in the system-config-run-gitea job. The first pass creates all the new projects. Then the second pass should noop cleanly. Change-Id: I73b77b9ddaa0106d4dc0a49c4d4b7751a39a16f9 Co-Authored-By: Jeremy Stanley <fungi@yuggoth.org>
2020-06-24 15:32:34 -07:00
# Run twice to ensure that we noop properly when
# all projects are created in gitea.
- playbooks/manage-projects.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
run_test_playbook: playbooks/test-gitea.yaml
host-vars:
gitea99.opendev.org:
host_copy_output:
'/var/gitea/conf': logs
'/var/gitea/certs': logs
'/var/gitea/logs': logs
gitea-lb01.opendev.org:
host_copy_output:
'/var/haproxy/etc': logs
files:
- playbooks/install-ansible.yaml
Rename service-letsencrypt to just letsencrypt This isn't a service, it's a meta thing that we run for different hosts at different times. Change-Id: Ib65665c98afb3ddb94b15346931be88a4b1757d8
2020-05-28 16:43:14 -05:00
- playbooks/letsencrypt.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/service-gitea-lb.yaml
- playbooks/service-gitea.yaml
- playbooks/manage-projects.yaml
- playbooks/test-gitea.yaml
Split inventory into multiple dirs and move hostvars Make inventory/service for service-specific things, including the groups.yaml group definitions, and inventory/base for hostvars related to the base system, including the list of hosts. Move the exisitng host_vars into inventory/service, since most of them are likely service-specific. Move group_vars/all.yaml into base/group_vars as almost all of it is related to base things, with the execption of the gerrit public key. A followup patch will move host-specific values into equivilent files in inventory/base. This should let us override hostvars in gate jobs. It should also allow us to do better file matchers - and to be able to organize our playbooks move if we want to. Depends-On: https://review.opendev.org/731583 Change-Id: Iddf57b5be47c2e9de16b83a1bc83bee25db995cf
2020-05-26 15:46:41 -05:00
- inventory/service/group_vars/gitea.yaml
- inventory/service/group_vars/gitea-lb.yaml
- inventory/service/host_vars/gitea
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/zuul/templates/group_vars/gitea.yaml.j2
- playbooks/zuul/templates/group_vars/gitea-lb.yaml.j2
- playbooks/roles/pip3/
- playbooks/roles/install-docker/
- playbooks/roles/letsencrypt
- playbooks/roles/gerrit/
- playbooks/roles/gitea/
- playbooks/roles/gitea-git-repos/
- playbooks/roles/haproxy/
- playbooks/roles/letsencrypt-create-certs/handlers/restart_gitea.yaml
- testinfra/test_gitea.py
- testinfra/test_gitea_lb.py
# From gitea_files -- If we rebuild the image, we want to run
# this job as well.
- docker/gitea/
# From haproxy-statsd_files -- If we rebuild the image, we want
# to run this job as well.
- docker/haproxy-statsd/
Grafana container deployment This uses the Grafana container created with Iddfafe852166fe95b3e433420e2e2a4a6380fc64 to run the grafana.opendev.org service. We retain the old model of an Apache reverse-proxy; it's well tested and understood, it's much easier than trying to map all the SSL termination/renewal/etc. into the Grafana container and we don't have to convince ourselves the container is safe to be directly web-facing. Otherwise this is a fairly straight forward deployment of the container. As before, it uses the graph configuration kept in project-config which is loaded in with grafyaml, which is included in the container. Once nice advantage is that it makes it quite easy to develop graphs locally, using the container which can talk to the public graphite instance. The documentation has been updated with a reference on how to do this. Change-Id: I0cc76d29b6911aecfebc71e5fdfe7cf4fcd071a4
2020-06-23 11:41:27 +10:00
- job:
name: system-config-run-grafana
parent: system-config-run-containers
description: |
Run the playbook for the etherpad servers.
timeout: 3600
requires: grafana-container-image
required-projects:
- opendev/system-config
- openstack/project-config
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: grafana01.opendev.org
label: ubuntu-focal
vars:
run_playbooks:
- playbooks/letsencrypt.yaml
- playbooks/service-grafana.yaml
files:
- playbooks/bridge.yaml
- playbooks/letsencrypt.yaml
- playbooks/service-grafana.yaml
- playbooks/roles/grafana/
- playbooks/roles/install-docker/
- playbooks/roles/pip3/
- docker/grafana/
- testinfra/test_grafana.py
Graphite container deployment This deploys graphite from the upstream container. We override the statsd configuration to have it listen on ipv6. Similarly we override the ngnix config to listen on ipv6, enable ssl, forward port 80 to 443, block the /admin page (we don't use it). For production we will just want to put some cinder storage in /opt/graphite/storage on the production host and figure out how to migrate the old stats. The is also a bit of cleanup that will follow, because we half-converted grafana01.opendev.org -- so everything can't be in the same group till that is gone. Testing has been added to push some stats and ensure they are seen. Change-Id: Ie843b3d90a72564ef90805f820c8abc61a71017d
2020-06-26 10:23:16 +10:00
- job:
name: system-config-run-graphite
parent: system-config-run
description: |
Run the playbook for the graphite servers.
timeout: 3600
required-projects:
- opendev/system-config
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
# NOTE(ianw): 01 is a half-puppet opendev.org
# server
- name: graphite02.opendev.org
label: ubuntu-focal
vars:
run_playbooks:
- playbooks/letsencrypt.yaml
- playbooks/service-graphite.yaml
host-vars:
graphite02.opendev.org:
host_copy_output:
'/var/log/graphite': logs
files:
- playbooks/bridge.yaml
- playbooks/letsencrypt.yaml
- playbooks/service-graphite.yaml
- playbooks/roles/graphite
- playbooks/roles/install-docker/
- playbooks/roles/pip3/
- testinfra/test_graphite.py
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- job:
name: system-config-run-meetpad
parent: system-config-run-containers
requires: jitsi-meet-container-image
description: |
Run the playbook for jitsi-meet.
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: meetpad01.opendev.org
label: ubuntu-bionic
Add support for multiple jvbs behind meetpad The jitsi video bridge (jvb) appears to be the main component we'll need to scale up to handle more users on meetpad. Start preliminary ansiblification of scale out jvb hosts. Note this requires each new jvb to run on a separate host as the jvb docker images seem to rely on $HOSTNAME to uniquely identify each jvb. Change-Id: If6d055b6ec163d4a9d912bee9a9912f5a7b58125
2020-05-18 12:28:51 -07:00
- name: jvb01.opendev.org
label: ubuntu-bionic
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
vars:
run_playbooks:
Rename service-letsencrypt to just letsencrypt This isn't a service, it's a meta thing that we run for different hosts at different times. Change-Id: Ib65665c98afb3ddb94b15346931be88a4b1757d8
2020-05-28 16:43:14 -05:00
- playbooks/letsencrypt.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/service-meetpad.yaml
host-vars:
meetpad01.opendev.org:
host_copy_output:
Add support for multiple jvbs behind meetpad The jitsi video bridge (jvb) appears to be the main component we'll need to scale up to handle more users on meetpad. Start preliminary ansiblification of scale out jvb hosts. Note this requires each new jvb to run on a separate host as the jvb docker images seem to rely on $HOSTNAME to uniquely identify each jvb. Change-Id: If6d055b6ec163d4a9d912bee9a9912f5a7b58125
2020-05-18 12:28:51 -07:00
'/var/jitsi-meet': logs
jvb01.opendev.org:
host_copy_output:
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
'/var/jitsi-meet': logs
files:
- playbooks/install-ansible.yaml
Split inventory into multiple dirs and move hostvars Make inventory/service for service-specific things, including the groups.yaml group definitions, and inventory/base for hostvars related to the base system, including the list of hosts. Move the exisitng host_vars into inventory/service, since most of them are likely service-specific. Move group_vars/all.yaml into base/group_vars as almost all of it is related to base things, with the execption of the gerrit public key. A followup patch will move host-specific values into equivilent files in inventory/base. This should let us override hostvars in gate jobs. It should also allow us to do better file matchers - and to be able to organize our playbooks move if we want to. Depends-On: https://review.opendev.org/731583 Change-Id: Iddf57b5be47c2e9de16b83a1bc83bee25db995cf
2020-05-26 15:46:41 -05:00
- inventory/service/group_vars/meetpad.yaml
- inventory/service/host_vars/meetpad\d+.opendev.org.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/roles/letsencrypt-create-certs/handlers/restart_jitsi_meet.yaml
- playbooks/roles/jitsi-meet/
- playbooks/roles/pip3/
- playbooks/roles/install-docker/
- playbooks/zuul/templates/group_vars/meetpad.yaml.j2
- testinfra/test_meetpad.py
Run jobs prod test jobs when docker images update We build our own docker images for several services, but weren't triggering production test runs when the docker images are updated. Fix this and force chagnes to dockerfiles, which produce new images, to trigger production test runs of those services. Change-Id: I18962663f168bbf6380e315d96b18751a46ceb58
2020-05-08 09:05:23 -07:00
- docker/jitsi-meet/
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- job:
name: system-config-run-zookeeper
parent: system-config-run
description: |
Run the playbook for the zookeeper cluster.
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: zk01.opendev.org
label: ubuntu-bionic
vars:
run_playbooks:
- playbooks/service-zookeeper.yaml
files:
- playbooks/install-ansible.yaml
Split inventory into multiple dirs and move hostvars Make inventory/service for service-specific things, including the groups.yaml group definitions, and inventory/base for hostvars related to the base system, including the list of hosts. Move the exisitng host_vars into inventory/service, since most of them are likely service-specific. Move group_vars/all.yaml into base/group_vars as almost all of it is related to base things, with the execption of the gerrit public key. A followup patch will move host-specific values into equivilent files in inventory/base. This should let us override hostvars in gate jobs. It should also allow us to do better file matchers - and to be able to organize our playbooks move if we want to. Depends-On: https://review.opendev.org/731583 Change-Id: Iddf57b5be47c2e9de16b83a1bc83bee25db995cf
2020-05-26 15:46:41 -05:00
- inventory/service/group_vars/zookeeper.yaml
- ^inventory/service/host_vars/zk\d+\..*
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/roles/zookeeper/
- playbooks/roles/pip3/
- playbooks/roles/install-docker/
- testinfra/test_zookeeper.py
- job:
name: system-config-run-zuul-preview
parent: system-config-run
description: |
Run the playbook for the zuul-preview service.
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: zp01.opendev.org
label: ubuntu-bionic
vars:
run_playbooks:
- playbooks/service-zuul-preview.yaml
files:
- playbooks/install-ansible.yaml
- playbooks/roles/zuul-preview/
- playbooks/roles/pip3/
- playbooks/roles/install-docker/
- testinfra/test_zuul_preview.py
- job:
name: system-config-run-zuul
parent: system-config-run
description: |
Run the playbook for the main Zuul cluster.
timeout: 3600
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: zk01.opendev.org
label: ubuntu-bionic
- name: zm01.openstack.org
label: ubuntu-xenial
- name: ze01.opendev.org
label: ubuntu-focal
- name: ze01.openstack.org
label: ubuntu-xenial
- name: zuul01.openstack.org
label: ubuntu-xenial
required-projects:
- openstack/project-config
- opendev/system-config
vars:
run_playbooks:
Rename service-letsencrypt to just letsencrypt This isn't a service, it's a meta thing that we run for different hosts at different times. Change-Id: Ib65665c98afb3ddb94b15346931be88a4b1757d8
2020-05-28 16:43:14 -05:00
- playbooks/letsencrypt.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/service-zookeeper.yaml
- playbooks/service-zuul.yaml
Run restart playbooks to test they work This runs our zuul and nodepool restart playbooks after the initial service installs in the system-config-run jobs. This will help ensure that they work consistently over time. Fix nodepool restart playbook Change-Id: I953e7222218c5555bb44fccd913eaa5e9374c669
2020-06-16 12:02:11 -05:00
# Test our ad hoc restart playbook works
- playbooks/zuul_restart.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
host-vars:
zm01.openstack.org:
host_copy_output:
Stop using backend hostname in zuul testinfra tests Tests that call host.backend.get_hostname() to switch on test assertions are likely to fail open. Stop using this in zuul tests and instead add new files for each of the types of zuul hosts where we want to do additional verification. Share the iptables related code between all the tests that perform iptables checks. Also, some extra merger test and some negative assertions are added. Move multi-node-hosts-file to after set-hostname. multi-node-hosts-file is designed to append, and set-hostname is designed to write. When we write the gate version of the inventory, map the nodepool private_ipv4 address as the public_v4 address of the inventory host since that's what is written to /etc/hosts, and is therefore, in the context of a gate job, the "public" address. Change-Id: Id2dad08176865169272a8c135d232c2b58a7a2c1
2020-06-03 14:20:36 -07:00
'/etc/hosts': logs
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
'/etc/zuul/zuul.conf': logs
Save zuul and nodepool logs from gate test jobs Let's save our debug logs so we can better observe the system in the gate. Change-Id: Ic80b646e0407d27e43cdb10cb573551999dd01d4
2020-05-20 07:26:27 -07:00
'/var/log/zuul/merger-debug.log': logs
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
ze01.opendev.org:
host_copy_output:
Stop using backend hostname in zuul testinfra tests Tests that call host.backend.get_hostname() to switch on test assertions are likely to fail open. Stop using this in zuul tests and instead add new files for each of the types of zuul hosts where we want to do additional verification. Share the iptables related code between all the tests that perform iptables checks. Also, some extra merger test and some negative assertions are added. Move multi-node-hosts-file to after set-hostname. multi-node-hosts-file is designed to append, and set-hostname is designed to write. When we write the gate version of the inventory, map the nodepool private_ipv4 address as the public_v4 address of the inventory host since that's what is written to /etc/hosts, and is therefore, in the context of a gate job, the "public" address. Change-Id: Id2dad08176865169272a8c135d232c2b58a7a2c1
2020-06-03 14:20:36 -07:00
'/etc/hosts': logs
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
'/etc/zuul/zuul.conf': logs
Save zuul and nodepool logs from gate test jobs Let's save our debug logs so we can better observe the system in the gate. Change-Id: Ic80b646e0407d27e43cdb10cb573551999dd01d4
2020-05-20 07:26:27 -07:00
'/var/log/zuul/executor-debug.log': logs
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
ze01.openstack.org:
host_copy_output:
Stop using backend hostname in zuul testinfra tests Tests that call host.backend.get_hostname() to switch on test assertions are likely to fail open. Stop using this in zuul tests and instead add new files for each of the types of zuul hosts where we want to do additional verification. Share the iptables related code between all the tests that perform iptables checks. Also, some extra merger test and some negative assertions are added. Move multi-node-hosts-file to after set-hostname. multi-node-hosts-file is designed to append, and set-hostname is designed to write. When we write the gate version of the inventory, map the nodepool private_ipv4 address as the public_v4 address of the inventory host since that's what is written to /etc/hosts, and is therefore, in the context of a gate job, the "public" address. Change-Id: Id2dad08176865169272a8c135d232c2b58a7a2c1
2020-06-03 14:20:36 -07:00
'/etc/hosts': logs
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
'/etc/zuul/zuul.conf': logs
Save zuul and nodepool logs from gate test jobs Let's save our debug logs so we can better observe the system in the gate. Change-Id: Ic80b646e0407d27e43cdb10cb573551999dd01d4
2020-05-20 07:26:27 -07:00
'/var/log/zuul/executor-debug.log': logs
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
zuul01.openstack.org:
host_copy_output:
Stop using backend hostname in zuul testinfra tests Tests that call host.backend.get_hostname() to switch on test assertions are likely to fail open. Stop using this in zuul tests and instead add new files for each of the types of zuul hosts where we want to do additional verification. Share the iptables related code between all the tests that perform iptables checks. Also, some extra merger test and some negative assertions are added. Move multi-node-hosts-file to after set-hostname. multi-node-hosts-file is designed to append, and set-hostname is designed to write. When we write the gate version of the inventory, map the nodepool private_ipv4 address as the public_v4 address of the inventory host since that's what is written to /etc/hosts, and is therefore, in the context of a gate job, the "public" address. Change-Id: Id2dad08176865169272a8c135d232c2b58a7a2c1
2020-06-03 14:20:36 -07:00
'/etc/hosts': logs
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
'/etc/zuul/zuul.conf': logs
Save zuul and nodepool logs from gate test jobs Let's save our debug logs so we can better observe the system in the gate. Change-Id: Ic80b646e0407d27e43cdb10cb573551999dd01d4
2020-05-20 07:26:27 -07:00
'/var/log/zuul/debug.log': logs
Stop using backend hostname in zuul testinfra tests Tests that call host.backend.get_hostname() to switch on test assertions are likely to fail open. Stop using this in zuul tests and instead add new files for each of the types of zuul hosts where we want to do additional verification. Share the iptables related code between all the tests that perform iptables checks. Also, some extra merger test and some negative assertions are added. Move multi-node-hosts-file to after set-hostname. multi-node-hosts-file is designed to append, and set-hostname is designed to write. When we write the gate version of the inventory, map the nodepool private_ipv4 address as the public_v4 address of the inventory host since that's what is written to /etc/hosts, and is therefore, in the context of a gate job, the "public" address. Change-Id: Id2dad08176865169272a8c135d232c2b58a7a2c1
2020-06-03 14:20:36 -07:00
bridge.openstack.org:
host_copy_output:
'/etc/hosts': logs
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
files:
- playbooks/install-ansible.yaml
- playbooks/service-zookeeper.yaml
- playbooks/service-zuul.yaml
Split inventory into multiple dirs and move hostvars Make inventory/service for service-specific things, including the groups.yaml group definitions, and inventory/base for hostvars related to the base system, including the list of hosts. Move the exisitng host_vars into inventory/service, since most of them are likely service-specific. Move group_vars/all.yaml into base/group_vars as almost all of it is related to base things, with the execption of the gerrit public key. A followup patch will move host-specific values into equivilent files in inventory/base. This should let us override hostvars in gate jobs. It should also allow us to do better file matchers - and to be able to organize our playbooks move if we want to. Depends-On: https://review.opendev.org/731583 Change-Id: Iddf57b5be47c2e9de16b83a1bc83bee25db995cf
2020-05-26 15:46:41 -05:00
- inventory/service/group_vars/zuul
- inventory/service/group_vars/zookeeper.yaml
- inventory/service/host_vars/zk\d+
- inventory/service/host_vars/zuul01.openstack.org
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/roles/zookeeper/
Vendor the apt repo gpg keys used for Zuul We use several PPAs on the Zuul servers, and today the Ubuntu keyring servers are frequently failing. Rather than rely on them, store the GPG keys in this repo and install the files "manually" rather than using the apt_repo module. Change-Id: I009a1a38d3a5864a8d5b0d8f8be24a83d1924292
2020-05-19 14:25:41 -07:00
- playbooks/roles/install-apt-repo
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/roles/zuul
Fake zuul_connections for gate We can't establish Gerrit or Github connections in the gate, so Zuul fails to start. Reducing the set of connections in the gate to just smtp should allow it to start (albiet with tenant loading errors). But that should let us test basic system setup and internal connectivity. Change-Id: I39d648ac5dd6ee3e9bfbc026cd6d7142461c418c
2020-06-08 14:57:19 -07:00
- playbooks/zuul/templates/group_vars/zuul
- playbooks/zuul/templates/group_vars/zookeeper.yaml
- playbooks/zuul/templates/host_vars/zk\d+
- playbooks/zuul/templates/host_vars/zuul01.openstack.org
Run restart playbooks to test they work This runs our zuul and nodepool restart playbooks after the initial service installs in the system-config-run jobs. This will help ensure that they work consistently over time. Fix nodepool restart playbook Change-Id: I953e7222218c5555bb44fccd913eaa5e9374c669
2020-06-16 12:02:11 -05:00
- playbooks/zuul_restart.yaml
Stop using backend hostname in zuul testinfra tests Tests that call host.backend.get_hostname() to switch on test assertions are likely to fail open. Stop using this in zuul tests and instead add new files for each of the types of zuul hosts where we want to do additional verification. Share the iptables related code between all the tests that perform iptables checks. Also, some extra merger test and some negative assertions are added. Move multi-node-hosts-file to after set-hostname. multi-node-hosts-file is designed to append, and set-hostname is designed to write. When we write the gate version of the inventory, map the nodepool private_ipv4 address as the public_v4 address of the inventory host since that's what is written to /etc/hosts, and is therefore, in the context of a gate job, the "public" address. Change-Id: Id2dad08176865169272a8c135d232c2b58a7a2c1
2020-06-03 14:20:36 -07:00
- testinfra/test_zuul_executor.py
- testinfra/test_zuul_scheduler.py
- testinfra/test_zuul_merger.py
- testinfra/util.py
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- job:
name: system-config-run-review
parent: system-config-run-containers
description: |
Run the playbook for gerrit (in a container).
requires: gerrit-2.13-container-image
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: review01.openstack.org
label: ubuntu-xenial
- name: review-dev01.opendev.org
label: ubuntu-xenial
required-projects:
- openstack/project-config
- opendev/system-config
vars:
run_playbooks:
Rename service-letsencrypt to just letsencrypt This isn't a service, it's a meta thing that we run for different hosts at different times. Change-Id: Ib65665c98afb3ddb94b15346931be88a4b1757d8
2020-05-28 16:43:14 -05:00
- playbooks/letsencrypt.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/service-review-dev.yaml
- playbooks/service-review.yaml
host-vars:
review-dev01.opendev.org:
host_copy_output:
'/home/gerrit2/review_site/etc': logs
'/home/gerrit2/review_site/logs': logs
'/var/log/apache2/': logs
'/var/log/acme.sh': logs
review01.openstack.org:
host_copy_output:
'/home/gerrit2/review_site/etc': logs
'/home/gerrit2/review_site/logs': logs
'/var/log/apache2/': logs
'/var/log/acme.sh': logs
files:
- playbooks/install-ansible.yaml
Rename service-letsencrypt to just letsencrypt This isn't a service, it's a meta thing that we run for different hosts at different times. Change-Id: Ib65665c98afb3ddb94b15346931be88a4b1757d8
2020-05-28 16:43:14 -05:00
- playbooks/letsencrypt.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/service-review.*.yaml
Split inventory into multiple dirs and move hostvars Make inventory/service for service-specific things, including the groups.yaml group definitions, and inventory/base for hostvars related to the base system, including the list of hosts. Move the exisitng host_vars into inventory/service, since most of them are likely service-specific. Move group_vars/all.yaml into base/group_vars as almost all of it is related to base things, with the execption of the gerrit public key. A followup patch will move host-specific values into equivilent files in inventory/base. This should let us override hostvars in gate jobs. It should also allow us to do better file matchers - and to be able to organize our playbooks move if we want to. Depends-On: https://review.opendev.org/731583 Change-Id: Iddf57b5be47c2e9de16b83a1bc83bee25db995cf
2020-05-26 15:46:41 -05:00
- inventory/service/group_vars/review.yaml
- inventory/service/host_vars/review\d+.openstack.org.yaml
- inventory/service/group_vars/review-dev.yaml
- inventory/service/host_vars/review-dev\d+.opendev.org.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/roles/pip3/
- playbooks/roles/install-docker/
- playbooks/roles/letsencrypt
- playbooks/roles/gerrit/
- playbooks/zuul/templates/group_vars/review-dev.yaml.j2
- playbooks/zuul/templates/group_vars/review.yaml.j2
- testinfra/test_gerrit.py
Run jobs prod test jobs when docker images update We build our own docker images for several services, but weren't triggering production test runs when the docker images are updated. Fix this and force chagnes to dockerfiles, which produce new images, to trigger production test runs of those services. Change-Id: I18962663f168bbf6380e315d96b18751a46ceb58
2020-05-08 09:05:23 -07:00
- docker/gerrit/2.13/
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- job:
name: system-config-run-static
parent: system-config-run
description: |
Run the playbook for a static node.
timeout: 3600
nodeset:
nodes:
- name: bridge.openstack.org
label: ubuntu-bionic
- name: static01.opendev.org
label: ubuntu-bionic
vars:
run_playbooks:
Rename service-letsencrypt to just letsencrypt This isn't a service, it's a meta thing that we run for different hosts at different times. Change-Id: Ib65665c98afb3ddb94b15346931be88a4b1757d8
2020-05-28 16:43:14 -05:00
- playbooks/letsencrypt.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/service-static.yaml
files:
- playbooks/install-ansible.yaml
- playbooks/roles/static/
- playbooks/roles/letsencrypt
Rename service-letsencrypt to just letsencrypt This isn't a service, it's a meta thing that we run for different hosts at different times. Change-Id: Ib65665c98afb3ddb94b15346931be88a4b1757d8
2020-05-28 16:43:14 -05:00
- playbooks/letsencrypt.yaml
Organize zuul jobs in zuul.d/ dir Our .zuul.yaml file has grown quite large. Try to make this more manageable by splitting it into zuul.d/ directory with jobs organized by function. Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 12:37:26 -07:00
- playbooks/service-static.yaml
- testinfra/test_static.py
host-vars:
static01.opendev.org:
host_copy_output:
'/var/log/acme.sh/': logs
'/etc/apache2/': logs
'/var/log/apache2/': logs
Copy Permalink