2412 Commits

Author SHA1 Message Date
Clark Boylan
fa1331e4a7 Remove redundant ssh known hosts prep
This removes redundant ssh known hosts prep in the
multi-node-known-hosts role as we've seen these tasks take significant
amounts of time due to the high cost of ansible looping. According to
the change [0] that added the system wide known_hosts setting this was
not originally done to avoid breaking testing in openstack-zuul-jobs.
Since then those tests have been moved to zuul-jobs [1] and I've updated
them with this change.

This should cut the time to run this role by about half.

[0] https://review.opendev.org/c/zuul/zuul-jobs/+/548642
[1] https://review.opendev.org/c/zuul/zuul-jobs/+/668767

Change-Id: I90492bbb5ae15fd3b36a54071c4eef891f53b405
2022-09-12 12:55:24 -07:00
Zuul
3c6ec6ec27 Merge "upload-npm : support authToken argument" 2022-09-08 01:13:45 +00:00
Ian Wienand
1520fb6474 upload-npm : support authToken argument
Support setting authToken to use a token for upload rather than
username/password.

This is "based" on Id0c47d088d6e8febbae3c96caecc81ebe247754e which
does the same for pypi.  There we run a test by uploading to the test
pypi server, I'm not aware of a similar thing for npm.

Change-Id: Ie8610bdfaa33b2120a61802c12e6ba534fe55464
2022-09-07 13:27:01 +10:00
Clark Boylan
e05b55fa67 Speed up log file fetching tasks
This replaces ansible loop tasks which rename log files to have a .txt
suffix with an ansible module task. The reason for this is each ansible
loop iteration can be very slow; up to 3 seconds per task. When
many log files need to be renamed this adds up quickly. The module can
do all of the renames in a single python process that should be much
quicker.

Note the old interface expected a list of extensions to rename. It then
converted that list of extensions into a regex used to find the files to
rename. The new code does not use regexes. It is possible someone abused
the old interface to hijack the regex and do something extra fun, but
that was not documented nor likely to be safe.

Change-Id: I7033a862d7e42b3acd8ad264aefca50685b317ff
2022-09-06 16:17:53 -07:00
Clark Boylan
1f78388328 Improve stage-output functional test
This adds extra . separators to the file we check is renamed. This came
up as a real world issue with devstack when testing the follow on update
to stage-output. We put this in a separate change to ensure the old and
new behavior is consistent.

Change-Id: I3b7504cfd90a4e7523ce88c50e90b2e9004e05ee
2022-09-06 16:16:12 -07:00
Clark Boylan
fbdd81f789 Exercise stage-output extensions_to_txt in testing
This updates our stage-output test to exercise the extensions_to_txt
behavior. Do this as we've seen the performance of these tasks is quite
poor. Testing will help us improve this without breaking things.

Change-Id: I35844fa438773b4789e7f1e4d223bd59d5df7ba7
2022-08-31 13:38:24 -07:00
Zuul
1513ab45bb Merge "Fedora : update to Fedora 36 release nodes" 2022-08-31 01:03:33 +00:00
Zuul
6e568f32d0 Merge "zuul-jobs-test-ensure-python-pyenv: update matchers" 2022-08-31 00:52:11 +00:00
Ian Wienand
aa85e13138 ansible-lint: pin to < 6.5
The new 6.5 release seems to have some issues with our code base (see
linked issue).  Pin to 6.4 for now.

Change-Id: I9516314f7bd2924bd50456a3f850f6151678e95f
2022-08-29 11:07:31 +10:00
Ian Wienand
d5ecbcebc4 Fedora : update to Fedora 36 release nodes
Change-Id: Id107ff96c75a11d03a0bde30f6011f2b44359f23
2022-08-22 15:27:11 +10:00
Ian Wienand
0741df3ef1 zuul-jobs-test-ensure-python-pyenv: update matchers
This was added with I2ab11bb45b6b2a49d54db39195228ab40141185c.  I
don't think we need to match on the
zuul-tests.d/python-roles-jobs.yaml file; it means that if you update
one job (e.g. say for a new platform like
Id107ff96c75a11d03a0bde30f6011f2b44359f23) it unnecessarily runs the
pyenv test for every platform.  Remove the matcher.

Change-Id: I8d24ed8b172304b829668cf768e66e7810c58ef2
2022-08-22 15:25:10 +10:00
James E. Blair
5ebf086442 Ensure-java: update apt cache
We should update the apt cache before attempting to install a package.

Change-Id: I6b1058e581815ceb194ca0cbb1c6e1b8e93928e6
2022-08-15 10:54:23 -07:00
Zuul
357e6fa6f9 Merge "Supply missing error message on S3 upload failure" 2022-08-14 22:33:08 +00:00
Zuul
23cff8eeb4 Merge "Add cli running example for s3 and swift log uploaders" 2022-08-12 07:25:47 +00:00
Zuul
1c9b84522a Merge "Use RDO wallaby repo to install openvswitch in CS9" 2022-08-12 07:25:44 +00:00
Simon Westphahl
f2ccc45c06 Supply missing error message on S3 upload failure
The `fail_json()` method of the Ansible module expects a msg argument.
https://docs.ansible.com/ansible/latest/reference_appendices/module_utils.html

    Traceback (most recent call last):
      File \\"<stdin>\\", line 102, in <module>
      File \\"<stdin>\\", line 94, in _ansiballz_main
      File \\"<stdin>\\", line 40, in invoke_module
      File \\"/usr/local/lib/python3.10/runpy.py\\", line 224, in run_module
        return _run_module_code(code, init_globals, run_name, mod_spec)
      File \\"/usr/local/lib/python3.10/runpy.py\\", line 96, in _run_module_code
        _run_code(code, mod_globals, init_globals,
      File \\"/usr/local/lib/python3.10/runpy.py\\", line 86, in _run_code
        exec(code, run_globals)
      File \\"/var/cache/zuul-executor/jobs/2f077ccd11b844e587a4d220ed22f36c/work/tmp/ansible_zuul_s3_upload_payload_l3h9evxc/ansible_zuul_s3_upload_payload.zip/ansible/modules/zuul_s3_upload.py\\", line 366, in <module>
      File \\"/var/cache/zuul-executor/jobs/2f077ccd11b844e587a4d220ed22f36c/work/tmp/ansible_zuul_s3_upload_payload_l3h9evxc/ansible_zuul_s3_upload_payload.zip/ansible/modules/zuul_s3_upload.py\\", line 313, in ansible_main
      File \\"/var/cache/zuul-executor/jobs/2f077ccd11b844e587a4d220ed22f36c/work/tmp/ansible_zuul_s3_upload_payload_l3h9evxc/ansible_zuul_s3_upload_payload.zip/ansible/module_utils/basic.py\\", line 2183, in fail_json
    AssertionError: implementation error -- msg to explain the error is required

Change-Id: I30a50ebd3c4a2553142a67ecb1bbb1faa2b3c527
2022-08-10 15:24:19 +02:00
Zuul
2bc6d7b9c3 Merge "linters: standardise on newline at end of file" 2022-08-05 09:06:19 +00:00
Zuul
6231ec4da9 Merge "upload-git-mirror: no_log around key writing" 2022-08-05 08:48:33 +00:00
Alfredo Moralejo
78f1dfaf8a Use RDO wallaby repo to install openvswitch in CS9
We should use the oldest release where we run centos stream 9 jobs which
is wallaby. We set up xena in the past as we backported cs9 jobs to xena
first but now we should switch to wallaby or we may hit issues in
wallaby jobs when the ovs version in xena is higher that in wallaby (as
now).

Change-Id: I18dab9701e96b25396b45df221f54fe94c1ab36b
2022-08-04 16:36:18 +02:00
James E. Blair
d1c36bb472 Test ensure-kubernetes on all Ubuntu platforms
Change-Id: I97057badf262a8931d1e48cab2987020dde12161
2022-07-28 12:00:11 -07:00
James E. Blair
300ee42863 Revert "Revert cri-dockerd changes"
This reverts commit c4a666991207f9434a819da9bac8b3603ad3d66e.

The change was originally reverted because it broke testing on Jammy.
Jammy broke because it was attempting to install cri-o using the Bionic
method via a PPA. This is no longer valid for newer Ubuntu, and Focal
uses a different process too. We update the cri-o installation for Jammy
to match Focal and make this method the default so that new Ubuntu
releases default to modern installation methods. Bionic continues to
install from a PPA.

Additionally we bump the cri-o version from 1.15 to 1.24 to get a
version that has packages for both Focal and Jammy. This new version
requires we also install cri-o-runc separately as they don't seem to
have proper package dependencies in place between these two packages.
The crio systemd service fails to start without cri-o-runc installed.

Change-Id: Ic29576e26be094744cc1b169a3c8f0bca814f089
2022-07-28 11:57:28 -07:00
James E. Blair
9101dd322c Support subsets of platforms in update-test-platforms
This lets us specify that we want to run a job on all "ubuntu"
platforms.

Change-Id: Ib466d719d3de241a6bd31ed4896cbeb96452aea8
2022-07-28 08:39:48 -07:00
James E. Blair
7b4b9b638f Sort supported platforms
This sorts the supported test platforms so that we can more easily
modify them in a future change.

Change-Id: I3a3a0347b89ab90d6ee74bc3070f7ad3604d59df
2022-07-28 08:39:47 -07:00
James E. Blair
62f6997e7f Add jammy testing
Change-Id: If24c92df128901170fa5dc47e693084001735fbd
2022-07-28 08:39:46 -07:00
James E. Blair
c4a6669912 Revert cri-dockerd changes
This was apparently not tested on Ubuntu jammy and has broken
Nodepool testing.

https://zuul.opendev.org/t/zuul/build/9ae631c37a384b4bb63515c6c5f04a00

Revert "ensure-kubernetes: pull cri-dockerd systemd from tag"

This reverts commit ad0ea28b6a2cbc8d68ee1a05e6a1d2712102dca4.

Revert "ensure-kubernetes: install cri-dockerd; fix networking"

This reverts commit 08c922fd988e98bd93d5a6d488a98e5b76797e18.

Change-Id: Ic20b8840c478b6d81626f728b8661ef5946e12d4
2022-07-28 07:22:40 -07:00
Ian Wienand
a016a1a565 linters: standardise on newline at end of file
I noticed this by accident when I ran ansible-lint over this repo from
an outside context; it didn't use the .yamllint in here and started
compalining about eof whitespace.

After scratching my head for a bit as to why this didn't fail here, I
realised we've allowed various newlines since the initial commit
I936fe2c997597972d884c5fc62655d28e8aaf8c5.

Remove this and just use the default eof rules, and fixup the
whitespace as required.  This is fairly unimportant, but is nice for
consistency.

Change-Id: Idb46a1f39ba798b0bf70eaa27b4c6b4758ce3d26
2022-07-28 16:19:06 +10:00
Zuul
4abf1a751c Merge "ensure-kubernetes: pull cri-dockerd systemd from tag" 2022-07-28 00:22:58 +00:00
Zuul
d51364c29c Merge "ansible-lint: disable progressive mode" 2022-07-28 00:22:56 +00:00
Zuul
7a5f297dcf Merge "linters: update to ansible-lint 6" 2022-07-28 00:22:53 +00:00
Zuul
f73c508cbc Merge "linters: add names to blocks" 2022-07-28 00:22:49 +00:00
Zuul
88741467fa Merge "linters: fix spaces between filters" 2022-07-28 00:07:28 +00:00
Zuul
a3afa7f594 Merge "emit-job-header: noqa on error ignore" 2022-07-28 00:06:41 +00:00
Zuul
443e461b9c Merge "linters: rename loop variable" 2022-07-27 23:53:28 +00:00
Zuul
50e17d253e Merge "upload-git-mirror: fix ssh key newline" 2022-07-27 23:53:26 +00:00
Zuul
110c62aae3 Merge "setup.py: override modules to stop auto-detection" 2022-07-27 23:53:21 +00:00
Zuul
632cb451cf Merge "test/upload-pypi : install venv" 2022-07-27 23:53:16 +00:00
Zuul
a4ccae41a2 Merge "ensure-kubernetes: install cri-dockerd; fix networking" 2022-07-27 23:53:12 +00:00
Ian Wienand
73951559fc upload-git-mirror: no_log around key writing
Add no_log here as a fallback against ever leaving the output in the
logs.

Change-Id: Ia3d518d915705b40e0e12e25e0a0787a8cf614d7
2022-07-28 09:31:49 +10:00
Ian Wienand
ad0ea28b6a ensure-kubernetes: pull cri-dockerd systemd from tag
Pull from the tag, instead of master, to avoid going out of sync

Change-Id: I5d3c612b1f8daed6f54c24fd70dc449dbc83ec8a
2022-07-28 08:26:13 +10:00
Zuul
1a63e95efd Merge "upload-artifactory: fix version comparision" 2022-07-27 21:22:36 +00:00
Ian Wienand
255ed06075 ansible-lint: disable progressive mode
Progressive mode

 ... makes the linter return a success even if some failures are
 found, as long the total number of violations did not increase since
 the previous commit.

I have found what I think is inconsistent matching of the errors
between runs and I'm not sure it isn't hiding problems.  We are
linter-clean and gate on the linter passing, so we don't need to do
this two-pass system.

Change-Id: I47be01a095d80dfb4d15f90da7bce49c3d42a2dd
2022-07-27 17:16:19 +10:00
Ian Wienand
5a2c7bff96 linters: update to ansible-lint 6
This updates to ansible-lint 6.  Some prior changes have updated for
the bulk of the rules, a couple of noqa points are added here.  Some
updates to the skipped rules are added.  This should have no
operational change, the only updates are cosmetic.

Change-Id: I165677bbb904f92292df00f7b9b27f8f3573aeb0
2022-07-27 17:13:39 +10:00
Ian Wienand
6d23d20f2f linters: add names to blocks
This is preparation for a later version of ansbile-lint, which finds
missing names on blocks.  This seems a reasonable rule, and the
Ansible manual says [1]

  Names for blocks have been available since Ansible 2.3. We recommend
  using names in all tasks, within blocks or elsewhere, for better
  visibility into the tasks being executed when you run the playbook.

This simply adds a name tag for blocks that are missing it.  This
should have no operational change, but allows us to update the linter
in a follow-on change.

[1] https://docs.ansible.com/ansible/latest/user_guide/playbooks_blocks.html

Change-Id: I92ed4616775650aced352bc9088a07e919f1a25f
2022-07-27 17:13:39 +10:00
Ian Wienand
f76cfbab11 linters: fix spaces between filters
This fixes a number of places where we do not have spaces between
filters.  I think that this is a reasonable rule for readability (I
also think it probably was enforced, but maybe later versions got
better at detecting it?).

These are detected by a later version of Ansible lint; this change
should have no operational change to any roles but prepares us to
update in a follow-on change.

Change-Id: I07e1a109b87adce86f483d14d7e02fcecb8313d5
2022-07-27 17:13:39 +10:00
Ian Wienand
f9e38131a7 emit-job-header: noqa on error ignore
This marks this particular ignore_errors as safe.  Generally I think
the rule is good, in that tasks get marked as failed when using
ignore_errors which is confusing and hard to debug.  But in this case
it's because of missing hostnames
(I8d67d8aa284b6ce9ae012608e8f1b12784ce836b) and not something we can
catch easily.

Change-Id: I9b5d6a85254f689e97bca91951632b4dea60cfde
2022-07-27 17:06:59 +10:00
Ian Wienand
df942a2aea linters: rename loop variable
In this repo we name the loop variables.  Although this is a test
playbook, it's good for consistency.  This is picked up by a later
version of ansible-lint.  This should have no operational change.

Change-Id: I084a1e8515fe1fda039190fe6518512ebf03217e
2022-07-27 17:06:30 +10:00
Ian Wienand
0956167b8c upload-git-mirror: fix ssh key newline
I don't think this testing has been run of Focal before, and the ssh
there is more picky about trailing newlines in the ssh private key.
Ensure it has a newline to avoid problems.

While we're here, clean up an old unused argument

Change-Id: If300083b9bc8e5538dcb1eeeaa896ba22c21232e
2022-07-27 14:42:00 +10:00
Ian Wienand
95d53aa40e setup.py: override modules to stop auto-detection
Some of our jobs have started failing with

   error: Multiple top-level packages discovered in a flat-layout:
   ['roles', 'playbooks'].
   ... <and so on> ...

which seems to be a combination of jobs that don't run often and some
recent-ish version of setuptools/pip.  From what I can tell, what we
want to do here is disable autodetection as what we have isn't really
a python distributable package.

Also fix the two places this is monkey-patched in over the existing
file.

Change-Id: I2a0dfbbedbb9bddd34b6af691118cf7c422a82b0
2022-07-27 11:14:06 +10:00
Ian Wienand
e0a2b647a6 test/upload-pypi : install venv
When we added this test I used ensure-pip in the ensure-twine role to
pull in the pip dependency.  But we soon realised that ensure-twine
ran in a rootless context so couldn't install packages, and reverted
it with I2cf4224228860b8a2cddd360636bb8633967b07e.

However, this testing wasn't fixed up.  Use ensure-pip in the test
playbook before we run the test, and add a note to ensure-twine that
it runs without privileges.

Change-Id: I6aef14b188dce8c4de774b09ddad718fec8fbf6c
2022-07-27 10:54:30 +10:00
Ian Wienand
08c922fd98 ensure-kubernetes: install cri-dockerd; fix networking
For ... reasons ... kubernetes deprecated docker as a container
runtime, and requires this cri-dockerd daemon to make docker cri-ish
enough to work.  Install and start it so the docker path keeps
working, although long-term I guess they're saying to move away from
this (from what I read it the "none" driver will also have problems
with cgroupsv2, which makes it's future on Jaunty look interesting).

Honestly I don't really know why the cri-o now needs the
--network-plugin flag.  Without it I get

 X Exiting due to RUNTIME_ENABLE: unknown network plugin:

which isn't described anywhere I can see.  Improvements welcome :)

Change-Id: I8ff34fa116aca14abee7e71f510bc49ffc547524
2022-07-27 10:51:33 +10:00