2373 Commits

Author SHA1 Message Date
Ian Wienand
a016a1a565 linters: standardise on newline at end of file
I noticed this by accident when I ran ansible-lint over this repo from
an outside context; it didn't use the .yamllint in here and started
compalining about eof whitespace.

After scratching my head for a bit as to why this didn't fail here, I
realised we've allowed various newlines since the initial commit
I936fe2c997597972d884c5fc62655d28e8aaf8c5.

Remove this and just use the default eof rules, and fixup the
whitespace as required.  This is fairly unimportant, but is nice for
consistency.

Change-Id: Idb46a1f39ba798b0bf70eaa27b4c6b4758ce3d26
2022-07-28 16:19:06 +10:00
Ian Wienand
73951559fc upload-git-mirror: no_log around key writing
Add no_log here as a fallback against ever leaving the output in the
logs.

Change-Id: Ia3d518d915705b40e0e12e25e0a0787a8cf614d7
2022-07-28 09:31:49 +10:00
Ian Wienand
ad0ea28b6a ensure-kubernetes: pull cri-dockerd systemd from tag
Pull from the tag, instead of master, to avoid going out of sync

Change-Id: I5d3c612b1f8daed6f54c24fd70dc449dbc83ec8a
2022-07-28 08:26:13 +10:00
Ian Wienand
255ed06075 ansible-lint: disable progressive mode
Progressive mode

 ... makes the linter return a success even if some failures are
 found, as long the total number of violations did not increase since
 the previous commit.

I have found what I think is inconsistent matching of the errors
between runs and I'm not sure it isn't hiding problems.  We are
linter-clean and gate on the linter passing, so we don't need to do
this two-pass system.

Change-Id: I47be01a095d80dfb4d15f90da7bce49c3d42a2dd
2022-07-27 17:16:19 +10:00
Ian Wienand
5a2c7bff96 linters: update to ansible-lint 6
This updates to ansible-lint 6.  Some prior changes have updated for
the bulk of the rules, a couple of noqa points are added here.  Some
updates to the skipped rules are added.  This should have no
operational change, the only updates are cosmetic.

Change-Id: I165677bbb904f92292df00f7b9b27f8f3573aeb0
2022-07-27 17:13:39 +10:00
Ian Wienand
6d23d20f2f linters: add names to blocks
This is preparation for a later version of ansbile-lint, which finds
missing names on blocks.  This seems a reasonable rule, and the
Ansible manual says [1]

  Names for blocks have been available since Ansible 2.3. We recommend
  using names in all tasks, within blocks or elsewhere, for better
  visibility into the tasks being executed when you run the playbook.

This simply adds a name tag for blocks that are missing it.  This
should have no operational change, but allows us to update the linter
in a follow-on change.

[1] https://docs.ansible.com/ansible/latest/user_guide/playbooks_blocks.html

Change-Id: I92ed4616775650aced352bc9088a07e919f1a25f
2022-07-27 17:13:39 +10:00
Ian Wienand
f76cfbab11 linters: fix spaces between filters
This fixes a number of places where we do not have spaces between
filters.  I think that this is a reasonable rule for readability (I
also think it probably was enforced, but maybe later versions got
better at detecting it?).

These are detected by a later version of Ansible lint; this change
should have no operational change to any roles but prepares us to
update in a follow-on change.

Change-Id: I07e1a109b87adce86f483d14d7e02fcecb8313d5
2022-07-27 17:13:39 +10:00
Ian Wienand
f9e38131a7 emit-job-header: noqa on error ignore
This marks this particular ignore_errors as safe.  Generally I think
the rule is good, in that tasks get marked as failed when using
ignore_errors which is confusing and hard to debug.  But in this case
it's because of missing hostnames
(I8d67d8aa284b6ce9ae012608e8f1b12784ce836b) and not something we can
catch easily.

Change-Id: I9b5d6a85254f689e97bca91951632b4dea60cfde
2022-07-27 17:06:59 +10:00
Ian Wienand
df942a2aea linters: rename loop variable
In this repo we name the loop variables.  Although this is a test
playbook, it's good for consistency.  This is picked up by a later
version of ansible-lint.  This should have no operational change.

Change-Id: I084a1e8515fe1fda039190fe6518512ebf03217e
2022-07-27 17:06:30 +10:00
Ian Wienand
0956167b8c upload-git-mirror: fix ssh key newline
I don't think this testing has been run of Focal before, and the ssh
there is more picky about trailing newlines in the ssh private key.
Ensure it has a newline to avoid problems.

While we're here, clean up an old unused argument

Change-Id: If300083b9bc8e5538dcb1eeeaa896ba22c21232e
2022-07-27 14:42:00 +10:00
Ian Wienand
95d53aa40e setup.py: override modules to stop auto-detection
Some of our jobs have started failing with

   error: Multiple top-level packages discovered in a flat-layout:
   ['roles', 'playbooks'].
   ... <and so on> ...

which seems to be a combination of jobs that don't run often and some
recent-ish version of setuptools/pip.  From what I can tell, what we
want to do here is disable autodetection as what we have isn't really
a python distributable package.

Also fix the two places this is monkey-patched in over the existing
file.

Change-Id: I2a0dfbbedbb9bddd34b6af691118cf7c422a82b0
2022-07-27 11:14:06 +10:00
Ian Wienand
e0a2b647a6 test/upload-pypi : install venv
When we added this test I used ensure-pip in the ensure-twine role to
pull in the pip dependency.  But we soon realised that ensure-twine
ran in a rootless context so couldn't install packages, and reverted
it with I2cf4224228860b8a2cddd360636bb8633967b07e.

However, this testing wasn't fixed up.  Use ensure-pip in the test
playbook before we run the test, and add a note to ensure-twine that
it runs without privileges.

Change-Id: I6aef14b188dce8c4de774b09ddad718fec8fbf6c
2022-07-27 10:54:30 +10:00
Ian Wienand
08c922fd98 ensure-kubernetes: install cri-dockerd; fix networking
For ... reasons ... kubernetes deprecated docker as a container
runtime, and requires this cri-dockerd daemon to make docker cri-ish
enough to work.  Install and start it so the docker path keeps
working, although long-term I guess they're saying to move away from
this (from what I read it the "none" driver will also have problems
with cgroupsv2, which makes it's future on Jaunty look interesting).

Honestly I don't really know why the cri-o now needs the
--network-plugin flag.  Without it I get

 X Exiting due to RUNTIME_ENABLE: unknown network plugin:

which isn't described anywhere I can see.  Improvements welcome :)

Change-Id: I8ff34fa116aca14abee7e71f510bc49ffc547524
2022-07-27 10:51:33 +10:00
Ian Wienand
ac056cd6da upload-artifactory: fix version comparision
This check fails when Ansible updated to two-digit point releases; use
the version filter.

Change-Id: I4be5543f21017b747333657416eac8c3cae89838
2022-07-27 05:53:07 +10:00
Ian Wienand
a46b34b9f7 ensure-twine : remove ensure-pip
This was added with I55ef79bf9c5af9e4d1da24ed654821277edb663b but
unfortunately breaks the OpenDev use where this runs on the executor
and doesn't have sudo.  Let's just leave it the way it was.

Change-Id: I2cf4224228860b8a2cddd360636bb8633967b07e
2022-07-19 08:15:26 +10:00
Ian Wienand
cf74d70961 upload-pypi: always test upload
Simplify the update procedure by tagging the checkout with the current
unix timestamp, so we should always having an increasing version
number.  This means we can do the test upload unconditionally in the
check gate.  Given this runs very infrequently, this shouldn't be a
burden.

Change-Id: I12c222802297d760296d35651db8d22fedb6dab4
2022-07-15 09:47:03 +10:00
Ian Wienand
673a499e7b upload-pypi: test sandbox upload
This does a full upload of the sandbox project to the test.pypi.org
service.

It uses the inline token with the caveats noted in the comment.  Since
you can't upload the same version twice, always running this in the
check job will mean it will work the first time, but then fail (unless
something has been committed to the sandbox to update the version
number).  As noted, the idea is that we develop the change and then do
one run that does a full test, and reviewers can check that before
committing.  There's no real way to use a secret here because we
actually want this to run at check time, not in any post pipeline
where it would already be in production.  We don't want to do
something like require committing something to sandbox every time you
run this in the check queue, etc.  Given how much this is actually
updated (most of it hasn't been touched since 2019) I think this is a
reasonable compromise.

Change-Id: Ida4ad07c82a6301107c938565656988aba3bf250
2022-07-13 18:52:52 +10:00
Ian Wienand
7adb0c3b4a upload-pypi: basic testing
Very basic testing to include the ensure-twine and upload-pypi role.
This really just does a syntax and sanity check.

Change-Id: Ie40ec32b3b2ffc4646301c6e145c527fe6a3bae2
2022-07-13 15:18:10 +10:00
Ian Wienand
ffe4f4fea5 ensure-twine: make python3 default, ensure pip installed
Make the default twine python installation use "python3".  Use the
ensure-pip role before calling pip.

This role currently isn't tested; it is added with upload-pypi testing
in the follow-on Ie40ec32b3b2ffc4646301c6e145c527fe6a3bae2

Change-Id: I55ef79bf9c5af9e4d1da24ed654821277edb663b
2022-07-13 15:18:10 +10:00
Ian Wienand
2c58a8ceed upload-pypi: support API token upload
Support an api_token argument.  Accounts that have two-factor auth
will at some point require using API tokens to upload.

Change-Id: Id0c47d088d6e8febbae3c96caecc81ebe247754e
2022-07-13 15:18:08 +10:00
Zuul
8e9d9c2f7d Merge "ensure-java role: Bump default java_version to 11" 2022-07-11 12:38:04 +00:00
Zuul
1f25bb5ea3 Merge "Revert "Install venv for all platforms in ensure-pip"" 2022-07-07 06:58:08 +00:00
Zuul
812e063513 Merge "[tox] Update readme regarding default values" 2022-06-27 08:59:45 +00:00
Szymon Datko
85ec622e11 [tox] Update readme regarding default values
In 2021 there was a change [1] in tox role default values.
However, the readme was not updated as part of this change.
This commit adds the missed note.

[1] https://review.opendev.org/c/zuul/zuul-jobs/+/807702

Change-Id: Ifa5b37c854eddbcda677ec5cd67a59bc3384ba34
2022-06-26 15:45:03 +00:00
David Ostrovsky
bc3e02bbc1 ensure-java role: Bump default java_version to 11
Premier Support for Java 8 was discontinued on 31 March 2022: [1]. Even
though Extended Support for Java 8 is still available until 2030, the
vast majority of projects switched to Java 11 already and forced to pass
java_version: 11.

Bump the default and ask the projects that still rely on Java 8 to pass
java_version: 8 to downgrade and use outdated Java releases.

[1] https://www.oracle.com/java/technologies/java-se-support-roadmap.html

Change-Id: I57d08039937d89881c7331293568dc510cabad19
2022-06-21 08:29:15 +02:00
Zuul
21faa1a9d8 Merge "Add the post-reboot-tasks role" 2022-06-17 15:46:53 +00:00
Ian Wienand
372f3af706 Revert "Install venv for all platforms in ensure-pip"
This reverts commit 0aca37281458b94b30fd124f885cadc18ad3c61e.

This uses the variable ensure_pip_from_upstream_interpreters to
install python-venv packages.

ensure_pip_from_upstream_interpreters is documented as being a list of
python interpreters to run "get-pip.py" with to install pip globally.
This only happens if ensure_pip_from_upstream is set to True.

It's unclear from the original discussion in the change at least, what
this was trying to fix.  It's also unclear why this would skip
packages, and what circumstances that is required in.  It's now
causing more confusion as we try to add fixes ontop of that.

Revert this -- whatever this is doing, we should find the right way to
do it, which isn't this.

Change-Id: Ieada7ec2d2c7320d7aacb9a5dc6de52394c11ed1
2022-06-17 14:52:20 +10:00
Clark Boylan
e6b747b32b Fix two testing problems
First up we need python2.7 specific msrest version as azure-storage-blob
depends on it and it doesn't properly specify python versions for its
releases.

Next we cannot use zuul/zuul to test ensure-pip can install packages
because zuul/zuul requires python3.8 or newer now. Some platforms we run
ensure-pip on still only have python3.6. We address this by using
zuul-client instead.

Change-Id: I3d91fd3f85e3d7845ab2d7615a1ebfb614064573
2022-06-16 10:30:00 -07:00
Ade Lee
25caf7ef5e Add the post-reboot-tasks role
This role will do basic checks to confirm that the node is
sufficiently up to continue afer a reboot.

Change-Id: Iebf474c9351e4246d7ab2072b48a50e93dbf0b94
2022-06-06 04:56:14 -07:00
James E. Blair
e69d879cae buildset registry: run socat in new session
We spawn socat processes in the background in buildset registry
related roles.  Ansible 5 is much better at killing all processes
in its session when the task is complete.  Work around that by
starting socat within a new session with setsid.

Change-Id: Iaab17f5d4068be6b08e3d89d2abe80199f0cd654
2022-05-24 10:46:42 -07:00
James E. Blair
f18bea4e09 ensure-podman: Remove kubic from Ubuntu 18.04 and drop 20.04
The kubic repo no longer has packages for Ubuntu 18.04.  Use the
PPA instead.

Kubic also dropped packages for 20.04, but the PPA is not available.
Since we have no source for podman packages on 20.04, drop support.

Change-Id: I0fa441cd2fe4ec35c1bbe6630324965e223d7512
2022-05-24 10:46:38 -07:00
James E. Blair
a1e6229171 Remove "include:" usage from multi-node-bridge
This is deprecated, and include_tasks is sufficient and present
in all supported versions.

Change-Id: If88d9d6ce9fb40a9a8e42e7a32d473ffd51b780a
2022-05-23 14:44:12 -07:00
James E. Blair
09d493916a Correct git config item name in mirror-workspace-git-repos
It appears this setting inadvertently had an extra word "ignore"
included in the key name, and later versions of the git module in
Ansible (5) throw an error in that case.  Remove it so that this
role will work in all versions.

This has been previously tested in a base-test cycle via
test-mirror-workspace-git-repos.

Change-Id: I597aa8d331e53955e826f09aa51a579e12c94f85
2022-05-23 13:13:00 -07:00
James E. Blair
0c980bbcb7 Make test-prepare-workspace-git role
This is a copy of prepare-workspace-git except that it imports
test-mirror-workspace-git-repos.  This is for base job testing.

Change-Id: I4ef3e4376c9e958761c165836c4fb546157e237a
2022-05-19 11:40:44 -07:00
James E. Blair
423a7d6044 Correct git config item name in mirror-workspace-git-repos
It appears this setting inadvertently had an extra word "ignore"
included in the key name, and later versions of the git module in
Ansible (5) throw an error in that case.  Remove it so that this
role will work in all versions.

Change-Id: If008cc5b12efb23f731f99806ddfb69ce9bb5d15
2022-05-19 09:33:00 -07:00
James E. Blair
2cc836a320 Re-sync test-mirror-workspace-git-repos
This brings it in-line with mirror-workspace-git-repos in preparation
for further changes.

Change-Id: Ia7d07f84c5bf48dc11e195738edbce7acece8fee
2022-05-19 09:32:15 -07:00
Zuul
904ca53b86 Merge "Make note of python_version being a string value" 2022-05-16 17:27:36 +00:00
Zuul
aff1f418f3 Merge "Switch py3.10 testing to Ubuntu Jammy" 2022-05-16 16:58:21 +00:00
Clark Boylan
52e84220e5 Make note of python_version being a string value
The type of the python_version variable matters when we interpret it
after yaml deserialization. If it is interpreted as a float then 3.1 ==
3.10 and those python versions are very different. Instead make note
that it should be a string and then update all of the tox jobs to set
this variable as a string.

Change-Id: I55bbc70526a13e6bd4c3c32da8590ad2af76fe7c
2022-05-16 09:16:35 -07:00
Zuul
61e9886477 Merge "Bump bazelisk version to 1.11.0" 2022-05-13 23:33:31 +00:00
Clark Boylan
baa7586095 Fix the disabling of deb src repos in configure-mirrors
The wrong flag was toggled to False. This revers the other flag back to
True and disables enable_deb_src_repos by setting it to false.

This was my mistake and also missed in review :(

Change-Id: Iffef2da4ae1ad4910be6d124e0cb0d019460fbb5
2022-05-13 10:49:52 -07:00
David Ostrovsky
5379244f9c Bump bazelisk version to 1.11.0
Change-Id: Iff82b373a7c45ff299a4559284027a6c4b243ed1
2022-05-13 05:05:18 +00:00
Zuul
d8184feded Merge "Switch enable_src_repos to False in configure-mirrors" 2022-05-12 17:12:24 +00:00
Ian Wienand
8e7d5e0404 add-gpgkey: trust incoming key
Add the incoming key to the trustdb with ultimate trust.  I noticed
this when using this role in a job that rechecks the signatures made
with an imported key (c.f Id624aa1ec6213be70809a8f911ab4aadc8a6ed53
and related changes).

Remove "--allow-secret-key-import" as it doesn't do anything any more,
per the man page.

Change-Id: I5fce163bce5c68342a444c36d9ba4af6e4af362c
2022-05-05 14:40:00 +10:00
Clark Boylan
e52559f527 Switch py3.10 testing to Ubuntu Jammy
We switch from Fedora 35 to Ubuntu Jammy for this testing because Ubuntu
Jammy is an LTS release that will be supported for 5 years, but Fedora
35's total supported time period is about 13 months some of which is now
gone.

Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/840355
Change-Id: I3025679a4747155a91e88a65dcb4f946a329fba1
2022-05-04 16:52:37 -07:00
Clark Boylan
88cd70f90c Switch enable_src_repos to False in configure-mirrors
This should only be landed after we have announced this changing default
and enough time has passed for people to reasonably update their CI
systems if necessary.

Change-Id: Ibf2e2ca0579e180db074b7ab342dae15108dad27
2022-04-27 09:14:11 -07:00
Clark Boylan
fc133bd165 Enable deb-src on Debian with a flag in configure-mirrors
This enables deb-src repos on Debian hosts with a flag. The flag
currently defaults to True to maintain backward compatible behavior with
the hardcoded deb-src repos that configure-mirrors previously set. We
intend on flipping this default to False after announcing this change.

The reasons for this are that source package repos are rarely needed
by CI systems, but these repos can consume large amounts of disk in
mirrors. To make it easier to people deploying mirrors we want to avoid
enabling source repos by default.

Change-Id: I7e9cd0ec1e3184c3c0561bbb7d3069feaf5f1ba5
2022-04-27 09:11:08 -07:00
Zuul
775729a4c0 Merge "Add per-build WinRM cert generation" 2022-04-26 21:52:35 +00:00
Chandan Kumar (raukadah)
8399dd1842 Donot Setup RDO gpg keys and repo for RHEL and Fedora
Currently in multi-node-bridge role, For RHEL and CentOS
distro, RDO repos are setted up and from where rdo-openvswitch
get pulled in RHEL deployment and causes unwanted failures.

Like a particular version of rdo-openvswitch is not yet available
in CentOS and it fails the job with nothing provides message.

Enabling RDO repos specifically for CentOS and other distros
except RHEL and Fedora will fix the
issue.

Signed-off-by: Chandan Kumar (raukadah) <chkumar@redhat.com>
Change-Id: Id68f5904c4ec3d667a16e9f4f195d53b02d29cec
2022-04-18 21:51:15 +05:30
James E. Blair
59d7af0e67 Add per-build WinRM cert generation
This adds roles that, similar to add-build-sshkey, create a per-build
WinRM certificate, install it on remote windows nodes, and then switch
to using the certificate in Ansible for authentication.  A second role
is included which can clean up the cert which is useful for static
nodes.

Since winrm certificates must be acessible within the bubblewrap
container, these roles can be used to restrict the system-wide winrm
cert to trusted playbooks while untrusted playbooks will only have access
to the per-build cert (with appropriate configuration of the executor).

Change-Id: I4efe25594c2f543886a000aa02fb0a38683a43cb
2022-04-13 15:04:51 -07:00