Commit Graph

1984 Commits

Author SHA1 Message Date
Sorin Sbarnea
0d886253f6 Add managed jobs to periodic-weekly
To avoid getting zuul-jobs getting rotten without known, we need
to run them weekly.

This should solve issue where some less common job get broken and
we find out much later, when we have to make other unrelated changes.

Change-Id: I3a95526fdb83b8aef12ee2ee09a089990e75cb59
Depends-On: https://review.opendev.org/#/c/748706
2020-09-17 08:33:50 +00:00
Ian Wienand
c5c9f7d862 Retire Fedora 31 for 32
We have Fedora 32 nodes now, retire the Fedora 31 testing and replace
with 32.

Docker isn't supported on Fedora 32.  There's a problem with cgroups
v2 and you can read all about it at [1].  Drop the install test.

Haskell upstream has dropped F32 from the COPR repo as it is now
packaged; a new package with the correct dependencies has been pushed
but isn't available yet (see the inline github issue), so workaround
the missing package.

[1] https://github.com/docker/for-linux/issues/955

Change-Id: I025613bee8c0a8513afa90331431d2c18b616464
2020-09-17 17:54:59 +10:00
Ian Wienand
4d2c441f40 ensure-kubernetes: workaround slow account creation
Per the referenced link, account creation can be slow and that's just
the way it is.  This should help tests that fail with

  Error from server (Forbidden): error when creating "test-pod.yaml":
  pods "test" is forbidden: error looking up service account
  default/default: serviceaccount "default" not found

Change-Id: I405aa7e58737c7061a471da2e2807c77756c76b8
2020-09-17 13:04:54 +10:00
Tristan Cacqueray
ca4111a586 render-diff: fix typo in failure messsage
This change fix a typo where `git_diff` is undefined and should be replaced
by render_diff.

Change-Id: I9f1322853b7abdf6abd8b0d287b4b538963e6674
2020-09-03 16:30:15 +00:00
Zuul
160ade087d Merge "Remove dependency on pkg_resources" 2020-09-02 17:06:09 +00:00
Sorin Sbarnea
0716fa10dc Remove dependency on pkg_resources
Fixes issue that breaks execution on systems with older setuptools.

Change-Id: I3f86b038872ad868cea659ccfd89e87c450fbf46
2020-09-01 19:03:13 +00:00
Haikel Guemar
54bf1abaa4 Add-sshkey: ensure that ssh home directory exists
Some images might not have pre-created that directory, hence failing the task.
This simple step will prevent that.

Change-Id: Id2b63e84ef047f02127772c4d45b55ec360436f8
2020-09-01 14:43:15 +02:00
Zuul
b13b925eff Merge "Update hashicorp jobs file with correct title" 2020-09-01 08:21:46 +00:00
Albin Vass
17d7322bd3 Update hashicorp jobs file with correct title
Change-Id: Ibc3a3f1f442e97d0c2d7edf3383694bcd5ad7cc0
2020-09-01 05:43:46 +00:00
Zuul
8fb0751ddc Merge "bindep: Fixed runtime warnings" 2020-08-27 08:24:12 +00:00
Zuul
4d3b20b351 Merge "ensure-pip: Clarify situation with ensure_pip_from_packages_with_python2" 2020-08-26 23:14:22 +00:00
Zuul
1769ab4049 Merge "Update upload-logs-s3 with access/secret key variable documentation" 2020-08-26 15:27:45 +00:00
Zuul
4380c461a6 Merge "Add python_version for tox27" 2020-08-26 15:27:43 +00:00
Zuul
293080869f Merge "Multi Node/Context support for mirroring Git workspaces via kubectl" 2020-08-26 15:24:48 +00:00
Ian Wienand
048aff6c98 Add ensure-rust role
Add a role to install Rust via the rustup tool.  It defaults to
installing globally, which avoids having to worry too much about
setting paths for follow-on jobs.

Packaged Rust and the upstream rustup install tool can live together,
and there's various documentation about it.  Thus I've made this such
that we can expand it with packaged Rust support if there is a need,
but I have not implemented that yet.

Change-Id: I32f9b285904a7036f9a80ada8a49fa9cf31b5163
2020-08-25 09:04:43 +10:00
Zuul
dab423acaf Merge "tox: include command output in log/error" 2020-08-24 19:05:22 +00:00
Zuul
31a9f85aef Merge "ara-report: add option for artifact prefix" 2020-08-24 19:05:21 +00:00
Sorin Sbarnea
a73d432078 bindep: Fixed runtime warnings
Resolve few ansible runtime warnings.

Change-Id: I3e2d5ec3b5b1fc6bbfaded4b8bece035f9880517
2020-08-24 18:32:16 +01:00
Ian Wienand
304874fb6a tox: include command output in log/error
There's a couple of places that call subprocess.check_output that
might raise and exception (I'm seeing one finding the "--name"); catch
the CalledProcessError and include it's output to give more info.

Change-Id: I53cfb9c1b37bd4dbf9e5bf04fbab83d3cdad795d
2020-08-21 12:56:36 +10:00
Paul Belanger
1e92a67db6 Revert "Ensure git is installed in prepare-workspace-git role"
This is breaking jobs in zuul.ansible.com, as there is an issue with
order with DNS.  In our base jobs, we first configure ensure git repos
are installed on the node, then configure the node for infra things
like dns, package mirrors, etc.  In our case, dns hasn't been fully
configured, which makes the package install fail.

I would propose we revert this change and re-release diskimage-builder. Or
have git considered an image based dependency.

A workaround for now has been to revert DIB, to rebuild images adding
back git.  However, this was exposed due to us missing one image.

This reverts commit 5d09503c88.

Change-Id: I533cbec83f7525175b9a94d70a61ba7424154aa6
2020-08-20 16:13:35 +00:00
5d09503c88 Ensure git is installed in prepare-workspace-git role
Since this role relies on git, let's ensure git is
installed. Issue got detected when git was made
build-only in [1], it's now being reverted[2].

[1] https://review.opendev.org/#/c/745678/
[2] https://review.opendev.org/#/c/747025/

Related-Bug: #1892326
Change-Id: I3115d581aa3bb8b22db1cd10e4180e8cc0e530f4
2020-08-20 13:49:39 +05:30
Ian Wienand
1dcda3ff36 ara-report: add option for artifact prefix
After https://review.opendev.org/#/c/742971 the report is generated on
the host.  For system-config, this means the report ends up copied to
"bridge.openstack.org/ara-report" in the logs directory; but the
artifact points to the non-existant top-directory.

Add a variable to allow a top-level dir to be set.

Needed-By: https://review.opendev.org/747101
Change-Id: I61a7d7328b5a45fc8dee09909f9411c397dd31b6
2020-08-20 14:39:05 +10:00
Denis Fortin
1a806f04e2 terraform: Add parameter for plan file
Add terraform_plan parameter to be defined when using apply
command. It allows to specify a plan file to apply.

Change-Id: I2858317e4dd560030a367b2cd2ac43db5585152a
2020-08-18 08:59:57 +00:00
Ian Wienand
620c95e4a3 Add focal testing
Add focal testing for automatically generated jobs.

Seems only one minor quoting change for ovs is required.

Change-Id: I5de5d5bd420092729de251d6bfcfe80b8af79f1a
2020-08-18 15:35:02 +10:00
Benjamin Schanzel
5b66292c55 Multi Node/Context support for mirroring Git workspaces via kubectl
The mirroring of git workspaces to k8s pod nodes is done by piping a git
push command through `kubectl exec`. Without specifying a k8s context
for kubectl, the `current-context` from the kubeconfig is used. This is
problematic in multi-node scenarios where different nodes reside in
different namespaces and thus, are reachable via separate contexts.
Ansible cannot handle the context switch automatically as it would via
the inventory and its kubectl connection plugin because we are running
the git push from the executor/localhost.
This change makes sure the correct context of the corresponding node is
passed to `kubectl exec` explicitly, thus pushing the git repos to the
correct k8s pod.

Change-Id: Ic0b11b5860bb0062b3a1c068f4f963fb76018715
2020-08-17 15:48:14 +02:00
Zuul
391f49c60d Merge "ensure-docker: remove amd64 architecture pin" 2020-08-14 17:57:13 +00:00
Zuul
0fd44e68b0 Merge "ensure-pip: add instructions for RedHat system" 2020-08-14 09:02:52 +00:00
Andreas Jaeger
495c5ed62e Disable E208 for now
ansible-lint throws an undocumented error "E208 File permissions not
mentioned" that looks too aggressive, for example it complains IMHO
wrongly about:

roles/add-build-sshkey/tasks/remote-linux.yaml:1: [E208] File permissions not mentioned
roles/add-build-sshkey/tasks/remote-linux.yaml:14: [E208] File permissions not mentioned
roles/add-build-sshkey/tasks/remote-linux.yaml:20: [E208] File permissions not mentioned
roles/add-build-sshkey/tasks/remote-linux.yaml:27: [E208] File permissions not mentioned

Disable it for now until ansible-lint is fixed - and the real problems
in zuul-jobs are fixed.

https://github.com/ansible/ansible-lint/pull/949 fixes the false
positives. Once that is merged and a new release is out, we should
revert and fix the real problems.

Change-Id: I3e3978bbc9458ad79623d22969344fb46f5fee43
2020-08-14 09:56:25 +02:00
Ian Wienand
bd992f3a76 ensure-docker: remove amd64 architecture pin
Just let this install the repo for whatever architecture we're running
on.

Change-Id: Ifdff9b731db5d5ab93dd7da6f7d8a311b3162113
2020-08-14 10:30:03 +10:00
00f4aab4d9 Fix url for ARA report
https://review.opendev.org/#/c/742971 changed
defaults for ara_report_path to be an absolute path but
missed changes in zuul artifacts. The url here
needs to be relative to log dir to be browsable.

Change-Id: I53511709227ea890eb417778be89e49e03771ff2
2020-08-12 12:27:17 +05:30
Zuul
7633994ed6 Merge "add-build-sshkey: call cmd with command" 2020-08-11 19:34:37 +00:00
Zuul
2975f163f5 Merge "Allow ara-report to run on any node" 2020-08-11 18:18:00 +00:00
Albin Vass
2e7b28ee86 add-build-sshkey: call cmd with command
Currently a new cmd shell is opened instead of running the command
supplied when adding the build key for ssh enabled windows nodes.

Fix this by adding a '/c' flag.

Change-Id: I582febfe91d5dc229cc6a1959477a461a237336b
2020-08-11 11:40:33 +02:00
Clark Boylan
4ab37d798a Loop over incomplete subunit files properly
We are still trying to get incomplete subunit files into our logs. This
is now failing because we assumed a single dict with a files entry to
iterate over but in reality we get a list of dicts with files entries to
iterate over. In order to iterate over this content we use
with_subelements to iterate over and copy the found files.

Change-Id: I160ac5caeec6fd7f5a35e805b7d4811b20405d14
2020-08-07 14:01:18 -07:00
Tristan Cacqueray
594aa7e6f5 ensure-pip: add instructions for RedHat system
This change adds instructions to help user figure out what to do when
unittest job fail since https://review.opendev.org/736070 requires
epel to be installed.

Change-Id: Ie69ba81405bc3fbc7666c7ee12ed4996c5323709
2020-08-07 12:27:36 +00:00
Zuul
155a880534 Merge "Fix partial subunit stream logging" 2020-08-06 21:56:45 +00:00
Clark Boylan
43aa849174 Drop suse mirror types in our zypper repo configs
zypper wants to autodetect these values and it appears that upstream
changes repo types because we are having errors now. Dropping this
config manually on a held node seems to fix things.

Change-Id: I8ad28da7164d9a0955f43d4864ba24c14f0bd4a3
2020-08-06 13:59:27 -07:00
Clark Boylan
f9475ddc41 Fix partial subunit stream logging
In a previous change I attempted to log in flight subunit streams that
may be left behind if a job times out. This unfortunately didn't work
because in the cases we leave those files behind we also assume that we
don't have a valid .stestr state due to the contents of this directly.

Address this by modifying the find-stestr.sh script to indicate via its
return code if .stestr or .testrepository are present separate from
which commands are valid to run. Then in ansible we can do steps that
always apply if (s)testr were used (like copying partial logs) and
separately do subunit parsing if the contents of the (s)testr db
directories are valid.

Change-Id: I1c8f2405d74484631f633065baf9764dbd0209ee
2020-08-06 11:40:46 -07:00
Zuul
1ba95015ac Merge "Avoid to use 'length' filter with null value" 2020-08-06 13:36:52 +00:00
Pierre-Louis Bonicoli
222056a26f
Avoid to use 'length' filter with null value
This error occurs when variables are null and
DEFAULT_JINJA2_NATIVE is enabled:

    object of type 'NoneType' has no len()

Change-Id: I7c69be60b846c03e30a0a6e8893bf2f3a3d5e5b8
2020-08-03 03:31:29 +02:00
James E. Blair
c24c09e82c Allow ara-report to run on any node
This role had some assumptions that it would run only on the
zuul-executor.  If we remove those assumptions, it can run on any
worker node, which allows it to be used in untrusted playbooks.

Note that this is backwards compatible only in the case that the
user did not provide a non-default value for ara_report_path.  If
a user overrode that (eg, to put the report in a directory other
than "ara" under the executor log dir), this will break.

Change-Id: I3a28368a049fa583398ece8a66f90783f53098f4
2020-07-24 18:56:02 +00:00
Zuul
fcf84346ef Merge "Collect partial subunit files" 2020-07-23 20:04:16 +00:00
Zuul
f2ad332b48 Merge "Disable base role testing that runs code on localhost" 2020-07-23 20:04:14 +00:00
Zuul
672ca9ecc3 Merge "Reinstate docker tag cleanup" 2020-07-23 18:58:09 +00:00
Albin Vass
f472ed9591 Update upload-logs-s3 with access/secret key variable documentation
Change-Id: If8d7986b0658df8cf548fb23bed1497ebf878fab
2020-07-23 18:51:39 +02:00
Ian Wienand
50375ac31c ensure-pip: Clarify situation with ensure_pip_from_packages_with_python2
With I450171aad5e31d2925239ab3d0641cd23f6815a2 we defaulted
ensure_pip_from_packages_with_python2 to True when Ansible was running
under Python 2.  Clarify in the documentation that it will be set in
this condition.  Also clarify that EPEL is required to install Python
2 pip packages on CentOS 7.

The RedHat.yaml started installing python2 packages unconditionally
when running under Python 2 with
I2ab11bb45b6b2a49d54db39195228ab40141185c.  This should have just
relied on ensure_pip_from_packages_with_python2 variable, update that.
Thus this will default to True when under Python 2, but will allow
people to turn off including the Python 2 packages if they know they
will not need them (e.g. they may run Ansible under Python 2, but only
ever use python3 in testing, and thus can avoid the epel dependency).

All of the distribution roles actually install the Python 3
pip/setuptools/wheel packages unconditionally when
ensure_pip_from_packages is set; they do *not* install it for the
``ansible_python_interpreter`` as implied by the documentation.

Change-Id: Ib948c8a156a0a96d610a310b6291781c5d3c7f96
2020-07-23 10:55:18 +10:00
Clark Boylan
8a08cb94ac Collect partial subunit files
If (s)testr are killed during a test run (possibly due to a timeout)
they will "leak" a tmp file containing the subunit stream to that point.
This file is useful for debugging so collect it. We don't process it
further as the file is a partial stream and humans may need to look at
it to make sense of it.

Change-Id: I6f0e61ec78f4dd9e2c4e52bc968b4f57586cb22e
2020-07-22 15:32:25 -07:00
Clark Boylan
44d2187e7f Disable base role testing that runs code on localhost
The workspace setup role requires local code execution on the executor.
This is not allowed from an untrusted context so we disable it.

The previous assertions about the zuul-cloner setup depend on the
workspace setup running successfully. Disable those extra assertions.

subunit fetching role assertions grepped the html file for validity on
localhost. Disable this as well.

Change-Id: I7449749f50b6e4a34c4615b00836a7148e01c768
2020-07-22 15:32:18 -07:00
James E. Blair
2dc89d8347 Reinstate docker tag cleanup
This uses a new method of calculating the cutoff time which does not
require executing code on the executor.

Change-Id: I92e71727e24281a31cb7caf0b78dd17a5aaae129
Co-Authored-By: Clark Boylan <clark.boylan@gmail.com>
2020-07-22 08:39:54 -07:00
Zuul
919fe2e8a7 Merge "dco-license: remove the empty nodeset" 2020-07-22 15:23:12 +00:00