2039 Commits

Author SHA1 Message Date
Sorin Sbarnea
856866fdde More E208 mode fixes
Change-Id: I8157ec1f31b8c5a064b63002e8311b91ef9ce9ab
See: https://ansible-lint.readthedocs.io/en/latest/default_rules.html#file-permissions-not-mentioned
2020-10-31 07:34:03 +00:00
Zuul
73a2da3b12 Merge "fetch-sphinx-tarball: don't run merge-output-to-logs" 2020-10-30 20:03:58 +00:00
Zuul
2223c8835d Merge "fetch-sphinx-tarball: explain what is happening" 2020-10-30 19:46:11 +00:00
Zuul
2efb34d589 Merge "Improve errors from update-test-platforms" 2020-10-25 23:26:48 +00:00
Sorin Sbarnea
d27c7ff9db Improve errors from update-test-platforms
Instead of getting a confusing stacktrace when someone adds a new
file that is missing the final project entry we now give a friendlier
error that also recommends way to address the issue.

Change-Id: I5a84d3a20e847eeb2d5d843ef970b5790532683c
2020-10-25 23:18:30 +00:00
Zuul
c5d8494e2f Merge "Install openswitch and firewall if need a bridge only" 2020-10-24 21:25:24 +00:00
Ian Wienand
6f43acde41 fetch-sphinx-tarball: don't run merge-output-to-logs
The merge-output-to-logs role is not doing anything in this test.  As
described inline, this merges files in
zuul.executor.work_root/<docs,artifacts> into the
zuul.executor.log_root directory so they are available in change
results.

Since this job doesn't publish anything there, this role is unused.

merge-output-to-logs currently can't run because it tries to run shell
scripts on the executor.  Thus we can remove this unused role and
restore the job.

Change-Id: I1afc905aa8d9c420bed316e99760ad7ad1d838ce
2020-10-15 14:44:03 +11:00
Ian Wienand
be12793c20 fetch-sphinx-tarball: explain what is happening
This documents the steps in the fetch-sphinx-tarball test to explain
what is happening.

Change-Id: I91d03beeb15395cf4f243988434112ae250b1742
2020-10-15 13:34:49 +11:00
Zuul
ac6119dd94 Merge "Revert "Refactor fetch-sphinx-tarball to be executor safe"" 2020-10-14 21:07:18 +00:00
James E. Blair
5603eb2291 Revert "Refactor fetch-sphinx-tarball to be executor safe"
This reverts commit 51a8ed8e95a2240547b0128701cc8acf6ba8bbcc.

This has a typo ("exector").  The fix is obvious, but the bigger
issue is that it was not caught in testing, even though the main
purpose of the change was to re-enable tests.  We should understand
why it wasn't caught in testing and resolve that before fixing and
unreverting.

Change-Id: I3ed407546fecc52d4a039f7959c0521511e6a00b
2020-10-14 13:54:22 -07:00
Tobias Henkel
e854c90f1f
Remove unneeded gce from upload_utils
This is gce specific and already existing in the gce specific part and
can be just removed.

Change-Id: I8838e3587cf5a2044bd4945f5ecea3f3290deddc
2020-10-14 22:43:04 +02:00
Zuul
4b19def07d Merge "Refactor fetch-sphinx-tarball to be executor safe" 2020-10-14 18:51:05 +00:00
Zuul
5d1662b56b Merge "Consolidate common log upload code into module_utils" 2020-10-14 18:28:17 +00:00
Sagi Shnaidman
7975e874e8 Install openswitch and firewall if need a bridge only
In case of nodeset which doesn't have peers and switch there is
no need to install openswitch and firewall rules, because nothing
will be installed. Let's skip it if no need.
Change-Id: I98cf5ec390ee22e538baa076c9ab87eea6a44c9e
2020-10-13 14:25:37 +03:00
Zuul
7e150858e4 Merge "Use ansible_distribution* facts instead of ansible_lsb" 2020-10-12 18:24:58 +00:00
Zuul
d7c6088d62 Merge "validate-host: skip linux nly tasks on windows machines" 2020-10-07 20:13:34 +00:00
Zuul
aba28fa570 Merge "Add CentOS 8 Stream testing" 2020-10-07 07:39:05 +00:00
Carlos Goncalves
3f743e00fd Add CentOS 8 Stream testing
Add CentOS 8 Stream nodes to the testing regime.

Add repositories for CentOS 8 Stream to configure-mirrors

Depends-On: https://review.opendev.org/#/c/734788/
Change-Id: Ia2f2b22461b4f4eca19d294ae06f6e1c90ea8599
2020-10-07 07:11:35 +11:00
Zuul
ffbfbebb02 Merge "ensure-docker: < 1500 MTU workaround" 2020-10-06 17:10:46 +00:00
Zuul
8d75b3d91c Merge "Pin openstacksdk to <0.48.0" 2020-10-06 16:53:21 +00:00
Tobias Henkel
9ff9fa5827
Pin openstacksdk to <0.48.0
The openstacksdk dropped py35 support in 0.48.0. Since we still have
py35 support we need to pin it temporarily until we've deprecated and
removed it.

Change-Id: Ia50ed54f43605fc1e12c95a2218b73abcadc66ed
2020-10-06 17:12:11 +02:00
Ian Wienand
664d068ef7 ensure-docker: < 1500 MTU workaround
As described inline, we should lower the MTU in the docker
configuration when we see the interface has a MTU lower than 1500 so
things "just work".  This particularly affects the Linaro ARM64 cloud
in OpenDev, but it is a generic issue.

Change-Id: I338616c41a65b007d56648fdab6da2a6a6b909f4
Story: https://storyboard.openstack.org/#!/story/2008230
2020-10-06 11:26:15 +00:00
Sorin Sbarnea
a1742afb60 ensure-docker: validate network connectivity
Verifies that installed docker can download and run containers
that need network access. This should prevent bugs where
service was installed but in a broken state.

Fixes bug which failed to run tests when tests were modified.

Change-Id: I309168719fd3cb7488bc2d0f4fec7785e1eb5d53
Story: https://storyboard.openstack.org/#!/story/2008215
2020-10-06 15:42:28 +11:00
Zuul
e59b68d909 Merge "update-json-file: add role to combine values into a .json" 2020-10-05 06:43:23 +00:00
Ian Wienand
0b9fad9583 update-json-file: add role to combine values into a .json
Ansible doens't really have a great built-in way to modify a json file
(unlike ini files).  The extant docker role does what seems to be the
usual standard, which is slurp in the file, parse it and then write it
back out.

In a follow-on change (I338616c41a65b007d56648fdab6da2a6a6b909f4) we
need to set some more values in the docker configuration .json file,
which made me think it's generic enough that we can have a role to
basically run read the file, |combine and write it back out.

This adds such a role with various options, and converts the existing
json configuration update in ensure-docker to use it.

Change-Id: I155a409945e0175249cf2dc630b839c7a97fb452
2020-10-05 15:18:58 +11:00
Ian Wienand
51a8ed8e95 Refactor fetch-sphinx-tarball to be executor safe
This reverts commit 69a238df46ca81e8890ebb2ace7addcbb4852911.

The role is re-written with executor-safe methods.

Depends-On: https://review.opendev.org/753222
Change-Id: I0b52eff66bfdca776e0e5c426bf1fc57deb3fc49
2020-10-05 15:14:31 +11:00
Zuul
ee7ff55598 Merge "Partial address ansible-lint E208" 2020-10-02 17:18:23 +00:00
Tobias Henkel
eb19f437f7
Consolidate common log upload code into module_utils
All log upload modules use a large portion of common code for finding
files and creating indexes. Move this into module_utils so this can be
shared between all.

Change-Id: I16c105bc70e07b0c8a4aa8e96119ab7451e00346
2020-10-02 12:40:14 +02:00
Zuul
1dbdc0e3fa Merge "Fix certificate issue with use buildset registry" 2020-10-02 00:26:58 +00:00
Ian Wienand
6f76d3f68a ensure-twine: remove readme_renderer
This dependency was added with
Iaf4da5aedaa3814b2ecebed4391da2324d3e388d to prevent a warning when
using "twine check" with text/markdown descriptions.

Per [1], this check doesn't really do anything and will never fail.

We are now having issues using this role with with Python 3.8 on the
executor as the md dependency pulls in cmarkgfm which has binary
dependencies but does not publish any wheels; meaning the executor
fails to install as it does not have a toolchain setup.  Unfortunately
upstream is not particularly active, so building good manylinux wheels
is not something we're going to be able to deal with in a timely
fashion.

Given that this is not a failure, and isn't really doing anything and
blocks Python 3.8, it seems our best course is to remove this
dependency.

[1] https://github.com/pypa/twine/pull/421

Change-Id: Iac5c9f63d41375889e4fdad67b9a45a24a644341
2020-10-02 09:08:58 +10:00
Zuul
effff59852 Merge "Merge upload logs modules into common role" 2020-10-01 16:00:31 +00:00
Zuul
5e5df2ae70 Merge "shake-build: add shake_target variable" 2020-09-30 15:41:35 +00:00
Andrii Ostapenko
fd2dc3bc91
Fix certificate issue with use buildset registry
Use buildset_registry_alias for certificate file name to address
'unknown certificate authority' issue on consumer side when using standalone
use-buildset-registry role.

Issue is not visible with opendev parent jobs because certificate is handled
in pull-from-intermediate-registry role there.

Change-Id: Ieaf43be0dd4ccb92a8240a493ee6636a23c9d484
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-09-29 10:24:23 -05:00
Clark Boylan
d49893f894 Partial address ansible-lint E208
- replaces ignore with a warn, which displays issue without affecting
  the linting outcome (allowing gradual fixing)
- bumps linter to enable the warn_list feature
- fixes a set of issues, others will be fixed in follow-up

Change-Id: I7d6f8c156b06f68f681943e88860930968e7c9f9
2020-09-29 10:29:01 +01:00
Zuul
cbe1b4a5aa Merge "Fix promote cleanup" 2020-09-29 06:42:44 +00:00
Zuul
d69c9f75b1 Merge "Make sure that we pass list in loop" 2020-09-29 01:35:06 +00:00
Dmitriy Rabotyagov
39776cfdd9 Make sure that we pass list in loop
map returns a generator that needs to be converted to a list for
the loop.

We also don't need unique as this is iterating over a list of
directory names that must already be unique.

Change-Id: Ibd22d79be29aaa9d3a7924319c59929e665f9cbc
2020-09-28 22:45:13 +00:00
Andrii Ostapenko
a8084e54ef
Fix promote cleanup
Change-Id: I1b639e5ed221301219f808a53dcb938cccefa019
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-09-28 16:58:19 -05:00
Zuul
08617aa7d8 Merge "Allow skip files when download logs" 2020-09-28 19:03:00 +00:00
Andrii Ostapenko
ef47a743b6
Add ability to use *-docker-image roles in periodic jobs
Use '{{ zuul.pipeline }}' tag prefix in *-docker-image instead of
'change_{{ zuul.change }}' one when zuul.change is not provided, that is
the case with periodic jobs. This allows to build, upload and promote images
using periodic jobs e.g:

- project:
    periodic:
      - project-buildset-registry

      - project-build-image1:
          dependencies:
            - name: project-buildset-registry
      - project-build-image2:
          dependencies:
            - name: project-buildset-registry

      # pulls from buildset registry and tests both images
      - project-test:
          dependencies:
            - name: project-build-image1
            - name: project-build-image2

      # pre-pulls images from buildset registry for fast build
      - project-upload-image1:
          dependencies:
            - name: project-test
      - project-upload-image2:
          dependencies:
            - name: project-test

      - project-promote:
          dependencies:
            - name: project-upload-image1
            - name: project-upload-image2

This fuctionality will allow to keep latest images up to date for the
case when image incorporates continuously updating code from multiple
repositories.

Using true ternary for tag evaluation because ternary filter requires
all passed to it variables be defined or defaulted [0].

[0] https://github.com/ansible/ansible/issues/51276

Change-Id: I8eb7d2baa24905e7aac51fce0b2f9b1f24f037f9
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-09-25 14:22:17 -05:00
Tobias Henkel
753f6157f4
Merge upload logs modules into common role
As a first step towards minimizing code duplication between the
various upload-logs roles move the uplaod modules into a common role
upload-logs-base. For easier review common code will be consolidated
in a followup change.

The google and s3 variant missed the unicode fix that swift log upload
received. Add this to make to make the test cases work with the same
fixtures.

Change-Id: I2d4474ae1023c3f3e03faaa5b888e705ee2ed0bc
2020-09-25 13:21:12 +02:00
Tristan Cacqueray
a086fb4333 ensure-zookeeper: add role to setup zookeeper
This role is lifted from https://src.fedoraproject.org/rpms/zuul/blob/master/f/tests/setup_zookeeper.yml

Co-Authored-By: Fabien Boucher <fboucher@redhat.com>
Change-Id: Iec21d12baddf3de580d1941adade107c7e24fdd9
2020-09-24 23:29:59 +00:00
Zuul
b7d151c911 Merge "Add support to use stow for ensure-python" 2020-09-24 08:01:34 +00:00
Dmitriy Rabotyagov
3faa27fd2d Add support to use stow for ensure-python
In case image is prepared with python-stow-versions element,
it make sense to use stow to just enable required python version instead
of building it for each job run.

Change-Id: Ie04a2bb59d351c1e5c79b2da79f7a094c44cdf86
2020-09-24 07:39:52 +00:00
Zuul
c8d0c1a017 Merge "explicit error when test_command is undef" 2020-09-23 17:49:39 +00:00
Zuul
801b925006 Merge "Use Train repo to install openvswitch from multi-node-bridge in CentOS 8" 2020-09-22 23:44:18 +00:00
Pierre-Louis Bonicoli
0eb09e7b4f
explicit error when test_command is undef
Don't use a shell builtin: shell builtins aren't available since
'command' ansible module is used. Instead, fail with an explicit
error when the required test_command variable is not set.

Fix this error:

    Ansible output: b'failed: [host] (item=exit 1) => {
        "ansible_loop_var": "item",
        "changed": false,
        "cmd": "exit 1",
        "item": "exit 1",
        "msg": "[Errno 2] No such file or directory: 'exit': 'exit'",
        "rc": 2
    }

Change-Id: I88303f7302d7354ffc8b18e607b28349a9860a57
2020-09-22 13:39:22 +02:00
Alfredo Moralejo
1dbba7f70e Use Train repo to install openvswitch from multi-node-bridge in CentOS 8
Currently it's using master repo for CentOS 8. RDO s updating the version of
OVS in master to 2.13 while previous releases use version 2.12 so it's a
problem to install 2.13 for jobs gating reviews for Train or Ussuri.

This review is moving to use Train repository to install openvswitch in
all CentOS 8 jobs as it's the older version supported on CentOS 8. Note
that CI jobs running with this multi-node setup configure proper repos for
the tested release (ussuri, victoria or master) at a later stage and
update the version of OpenvSwitch to the one included on each specific
release repo.

Closes-Bug: #1896469

Change-Id: Ie30997f360b285131cb5051d582da2a15cca7205
2020-09-22 11:12:30 +02:00
Zuul
1fc2a5c0a4 Merge "Add managed jobs to periodic-weekly" 2020-09-22 06:20:15 +00:00
Sorin Sbarnea
69a238df46 Disable broken fetch-sphinx-tarball test job
Recent security fixes made to zuul made impossible to run this
test. Disabling this test until we can find a way to test it as
this is not an easy change.

This is needed because there were at least two other changes that
failed to pass because triggered this broken job. This was broken
for more than a month.

Main challenge here is that Ansible does not have any `mv` module, so
we still need to use shell. Replacing mv with copy could risk producing
out of disk space failures sooner or later.

Change-Id: If09e0430033fd2f1d6fc5e35e5ec0b10dc7c2dfe
Needed-By: https://review.opendev.org/#/c/748480/
Needed-By: https://review.opendev.org/#/c/748682/
2020-09-21 15:00:01 +01:00