36 Commits

Author SHA1 Message Date
Ian Wienand
500eba3634
configure-mirrors: make each compontent in 9-stream configurable
When we added 9-stream with I2da3a5e8a45598c6b3ded132ea61b86b4480e262
we had some components pointing to the mirrors and some not.  This is
basically because the OpenDev mirrors choose to mirror these parts of
the distribution.  It was an oversight to do things like this, as this
is nominally a generic configuration tool.

This proposes that we put each component behind a variable which can
be enabled or disabled.  This way users of the role can switch
on-or-off whatever their local mirrors carry, and fall-back to the
generic mirror metalink in other cases.

I imagine that with future distros like this we would have everything
disabled by default in zuul-jobs, and leave it up to sites to
explicitly opt-in the various components.  We could announce this via
our usual deprecation path and do similar for 9-stream; but for now
this keeps the defaults as it is currently written.

Change-Id: Ibfd88c137855c22b960099cc8d582b241c3b1db6
2022-09-16 10:32:29 +10:00
Zuul
c3a0026fa7 Merge "Update gpg key file for extras-common in CS9" 2022-09-14 23:10:21 +00:00
Clark Boylan
fc133bd165 Enable deb-src on Debian with a flag in configure-mirrors
This enables deb-src repos on Debian hosts with a flag. The flag
currently defaults to True to maintain backward compatible behavior with
the hardcoded deb-src repos that configure-mirrors previously set. We
intend on flipping this default to False after announcing this change.

The reasons for this are that source package repos are rarely needed
by CI systems, but these repos can consume large amounts of disk in
mirrors. To make it easier to people deploying mirrors we want to avoid
enabling source repos by default.

Change-Id: I7e9cd0ec1e3184c3c0561bbb7d3069feaf5f1ba5
2022-04-27 09:11:08 -07:00
Alfredo Moralejo
18021b8f0f Update gpg key file for extras-common in CS9
The actual signature for extras-common is the -sha512 file [1][2].

[1] https://gitlab.com/redhat/centos-stream/rpms/centos-release/-/blob/c9s/RPM-GPG-KEY-CentOS-SIG-Extras-SHA512
[2] https://gitlab.com/redhat/centos-stream/rpms/centos-release/-/blob/c9s/centos-addons.repo#L116

Change-Id: I6a8c3d737500696dafb66315aef38072a2d20d8d
2022-04-19 10:19:36 +02:00
Ian Wienand
c20a2435a6 configure-mirrors: fix stripped newline
I74b9de7092f182c942a58ac7a46b9fbd791889de hit a common ansible gotcha
where it likes to strip the trailing newline after a {% endif %}.
This has resulted in invalid lines in our sources.list.

Unfortunately we miss this because it still exits with 0.  Add a
simple test looking for warning output.

Change-Id: I46d393a5e67d10a52c4dcca803176ff368a4b5bd
2022-03-01 11:10:30 +11:00
Jonathan Rosser
5d01b68574 Allow some configure-mirrors repositories to be disabled
Out-of-the-box installs of centos-8 do not enable the PowerTools
or HighAvailability repositories. Debian/Ubuntu do not enable
the backports repository by default.

Having these repos enabled by default in CI has led to merging
broken code for OpenStack deployment tooling which is attempting
to also manage the presence/absence of these repositories. It
is challenging to remove these repositories on the running node
because the repo URL (rather than just the name) is required as
input to the apt_repository and yum_repository ansible modules.

This patch adds a role default variable to configure-mirrors to
allow a job to opt out of these extra repositories. The default
is set to 'True' to allow existing jobs to work as before.

Change-Id: I74b9de7092f182c942a58ac7a46b9fbd791889de
2022-02-16 20:39:31 +00:00
Alfredo Moralejo
25f110c399 Add CentOS Stream 9 to configure-mirrors role
So that we use AFS mirrors for CS9 nodes.

Closes-Bug: #1959181
Change-Id: I2da3a5e8a45598c6b3ded132ea61b86b4480e262
2022-01-27 15:05:46 +01:00
Jeremy Stanley
750be2e2de Add new Debian security mirror suite pattern
Starting with Debian 11 (bullseye), security packages are in
bullseye-security as opposed to older releases like buster/updates.
List the last several stable releases in hopes nobody is trying to
use this role to configure platforms older than Debian 8 (jessie,
the current "oldoldstable").

A followup change demonstrates this works in the test-base-roles
job, but because the job matrices have to be updated in one fell
swoop, and many of those jobs won't work without this change already
merged (due to protected use in our base job), it's not tested
directly within this change.

Change-Id: I2d7712cbfd037a65b9025980a6c0cccd917f8947
2021-05-03 18:39:34 +00:00
Javier Pena
cfada06486 Fix repo files for CentOS Stream
Some repo file names and ids do not match the files provided by
CentOS, which can cause duplicated repos [1].

[1] - https://8a51f3027f5f50a14ba7-fcfc85786424b7f5a3fcadff9da35f94.ssl.cf5.rackcdn.com/770771/4/check/packstack-centos8s-integration-scenario002/9dc0bd2/logs/etc/yum.repos.d/index.html

Change-Id: Ib89ad9f79cd5317b3178301a47462277004fa81a
2021-01-22 17:46:26 +01:00
5cfc37b300 Rename CentOS8 repo files to CentOS-Linux
With CentOS8.3 release repo files are now
renamed to CentOS-Linux*, update in zuul-jobs
too to sync, also update repo names as per
latest CentOS8 minor release.

The change is needed now as dib images are updated
to reflect new repo files post [1].

[1] https://review.opendev.org/c/openstack/diskimage-builder/+/765963

Change-Id: I7ed9bd582043717ae5bae303ea9b32db7f73008e
2020-12-18 14:23:08 +05:30
Carlos Goncalves
3f743e00fd Add CentOS 8 Stream testing
Add CentOS 8 Stream nodes to the testing regime.

Add repositories for CentOS 8 Stream to configure-mirrors

Depends-On: https://review.opendev.org/#/c/734788/
Change-Id: Ia2f2b22461b4f4eca19d294ae06f6e1c90ea8599
2020-10-07 07:11:35 +11:00
Clark Boylan
43aa849174 Drop suse mirror types in our zypper repo configs
zypper wants to autodetect these values and it appears that upstream
changes repo types because we are having errors now. Dropping this
config manually on a held node seems to fix things.

Change-Id: I8ad28da7164d9a0955f43d4864ba24c14f0bd4a3
2020-08-06 13:59:27 -07:00
Clark Boylan
6a2aaf7c80 Remove failovermethod from fedora dnf repo configs
The failovermethod option was removed in dnf and produces eye catching
warnings. Clean this up to clean up job output by configure-mirrors.

More details at: https://bugzilla.redhat.com/show_bug.cgi?id=1653831

Change-Id: Id646835e0fa9e5e99cde57382b3148caeb55e8bd
2020-05-20 15:16:34 -07:00
Alfredo Moralejo
5074bca828 Add CentOS8 High Availability repository
High Availability packages have been added as part of core CentOS
repositories in 8.1.

This patch add this repo and enables it as it provides corosync and
pacemaker based HA solution and some other packages required for it.

Change-Id: Idbddd81f251c1ade97892128e52f9214420bead7
2020-02-05 15:47:06 +01:00
Zuul
e21bed0c48 Merge "Add pypi_fqdn to differentiate it package mirrors" 2020-01-06 16:36:58 +00:00
Clark Boylan
b62c488eab Fix ansible use of filters and tests
Ansible 2.9 requires the use of "version" as a filter and
"version_compare" has been removed. Also tests cannot be used as
filters which means we cannot do when: foo | test and must do
when: foo is test instead. Make these fixes.

Both changes should be backward compatible to ansible 2.5:
https://docs.ansible.com/ansible/latest/user_guide/playbooks_tests.html#version-comparison
https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_2.5.html#jinja-tests-used-as-filters

Change-Id: Id95f674a485877db2a7924994366d1c6c591a684
2019-12-17 10:49:57 -08:00
Bogdan Dobrelya
77bc616dc5 Add pypi_fqdn to differentiate it package mirrors
There is mirror_fqdn that, if defined, overrides mirrors used for
packages and also mirrors for pypi. This is generally incorrect,
if one wants to use different mirrors for packages vs pypi eggs.

Add pypi_fqdn that defaults to mirror_fqdn, which allows users
to go with a custom pypi_mirror. Make pypi trusting the given
pypi_fqdn value instead of generally unrelated mirror_fqdn.

Change-Id: I12975b57951699351cfc0d40beaeb7c703651dd0
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
2019-11-28 09:00:07 +01:00
Monty Taylor
e79ca5ec3a Stop installing pydistutils.cfg
We install pydistutils.cfg to catch legacy cases where people
are doing things with easy_install. That code path is not
desirable and increasingly broken, as support for things like
allow-hosts is dropped upstream.

Stop installing the file. This will mean people using easy_install
won't get the benefit of the mirrors, but they shouldn't be
using easy_install anyway.

Remove pydistutils.cfg if it's there

Make sure we don't have any pydistutils.cfg around.

Change-Id: I24a05f456f87bd4cb57ebf89d4245477bf675f97
2019-11-24 12:03:39 -05:00
Ian Wienand
2e684a8da5 configure-mirrors: add CentOS 8
Add repositories for CentOS 8, add base test.

Co-Authored-By: Sorin Sbarnea <ssbarnea@redhat.com>
Change-Id: I24e194a2f4729046c8f521ffccd3b00055127516
2019-10-15 17:33:48 +11:00
Ian Wienand
2462d16322 configure-mirrors: make separate template directories for each platform
This keeps each platform's "/etc" template files in a separate
top-level directory.

Additionally, we add a distro major version match to the task import
and rename centos7 (in preparation for centos8).

Change-Id: If65d51a27e30311b1da20522afb6dbce7ee6cf35
2019-10-15 17:04:57 +11:00
Xinliang Liu
e330de0e09 Use debian-security mirror
debian-security has been mirrored by change[1], so we can use
it now.

[1] https://review.openstack.org/#/c/577316/

Change-Id: I2c88306564b04db3c6e061ab52545c5a63665e53
2018-06-27 11:03:34 +08:00
Paul Belanger
49711c6540
Update repo file for fedora-28
With the release of fedora-28, the default locations for
fedora-update.repo have changed. We still need to support fedora-27
until fedora-29 is released.

Change-Id: I8eacd659cf18a8dc571aea6531483610fcd91d1a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-05-02 12:28:36 -04:00
Paul Belanger
df364a46e0
Switch to http://security.debian.org/ for debian
Today the mirror system in openstack-infra isn't properly setup to
mirror debian-security. Until we can fix this, switch to the upstream
URL.

Change-Id: Icfa7dc874d14de245927d461595f90b9dacc7825
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-04-18 13:38:23 -04:00
Jens Harbott
9861bc1640 Allow updates from untrusted mirrors
The default behaviour of apt has been changed, see [0]. When using
unsigned mirrors, we need to set the "trusted" option explicitly in
order to allow updates from these.

[0] https://manpages.debian.org/stretch/apt/apt-secure.8.en.html

Depends-On: https://review.openstack.org/#/c/536615/
Change-Id: I1a6818b022cd34c8899179c36cae962a6c8ec5ed
2018-03-03 06:49:27 +00:00
Dirk Mueller
7172bd5dc8 Setup repositories for openSUSE Tumbleweed
Tumbleweed as a rolling release model distribution does not
have a sensible release version and is kept distinct from the
classing "release + update" distributions that are called "Leap".

Conditionalize the repository setup for tumbleweed properly,
the base path is different and it does not have a releaseversion
component (the release version is always changing, it is
the snapshot date in the form YYYYMMDD).

Change-Id: I700ba2279dc7aadc7051cad53898e1de9007a08d
2018-02-25 21:15:09 +01:00
David Moreau Simard
a4e9e12241
Remove the CentOS Plus mirror from the configured mirrors
CentOS Plus is not being mirrorred by openstack-infra and contains
mostly alternative kernel images which are not expected to be used in
jobs.

Let's reconsider if there is a use case for it in the future but in
the meantime let's remove it.

Change-Id: I9681b3b92c913f45791f610d845a94ccf3a16e48
2017-12-01 11:26:13 -05:00
Paul Belanger
afb6f902f1
Use sources.list.d for Debian with APT
Under debian, /etc/apt/sources.list.d is where the default list files
are stored. We need to also use that to properly re-write them.

Otherwise, we don't properly configure our AFS mirrors.

Change-Id: I1eb3ac19ba7081d3f4fd9b000e266d86449dee0e
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-10-12 20:25:35 -04:00
David Moreau-Simard
97e764be26 Add support for opensuse in configure-mirrors
Change-Id: Ic73b502277314555513e548aafb9231550fa1401
2017-09-07 20:22:41 +00:00
David Moreau-Simard
a3fd046cc8 Add support for fedora in configure-mirrors
Change-Id: I05d87e2a846c2b72eee80e06eeceeb1f92e39f42
2017-09-07 20:22:30 +00:00
David Moreau-Simard
b2ccb986a9 Add support for debian in configure-mirrors
Change-Id: Ib7230c69d31d075392d738cbf7e1685fd46480c4
2017-09-07 20:22:15 +00:00
David Moreau-Simard
8ac328ec21 Add support for centos in configure-mirrors
Change-Id: Ia62c60fb9bd7e8e19d9349fa9675aa379b872924
2017-09-07 20:22:08 +00:00
David Moreau-Simard
d45cfa287d Streamline configure-mirrors and Ubuntu repository setup
- Improve the structure to support multiple distributions
- Use one task to set up the different repository files
- Use handler to update the apt cache

Change-Id: I90933164cbca9d230cebeb3e2e1263c7806db840
2017-09-07 15:37:31 -04:00
Paul Belanger
e59abec7a3 Rename mirror_host to mirror_fqdn in configure-mirrors
It make more sense to revert to mirror_fqdn, since it is a FQDN.

Change-Id: I4a0749b64a71e551e4fbea5b416b46d2d6433d0b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-08-03 14:53:57 -04:00
Paul Belanger
05e6980baa Remove mirror_domain from configure-mirror
After talking to clarkb, we decided we can drop this in favor of just
using the mirror_host.

Change-Id: Ie4ce336174aa2a825d125f7e0d9a256de139908b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-08-03 14:49:24 -04:00
Monty Taylor
83acffb6b9
Replace hand-written slug with ansible_managed
There is an ansible_managed variable that can be referenced in templates
to add a comment about a file being managed by ansible. Use it.

Change-Id: I60c27cf0eb35dcd4805c3517b3b7e0c7d9a0b3ea
2017-07-12 06:26:04 -05:00
Paul Belanger
8638ac3e2a
Create configure-mirrors role
This will be used to replace our current confgure_mirror.sh script
that we today as a ready-script in nodepool.

This only adds support for ubuntu today. As we move forward and
configuration openstack-infra from JJB to ansible, we'll need to add
more distros.

Attach the role to the unittests base job for now. It should ultimately
be attached to the base job, but adding it to the unittests job lets us
test it more easily.

Co-Authored-By: Monty Taylor <mordred@inaugust.com>
Change-Id: I9bfa28c87390c09bb2c4cd0de6ce4c7890f8d81a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-07-11 14:56:12 -04:00