1162 Commits

Author SHA1 Message Date
Albin Vass
2ad252a217 Fixes tox log fetching when envlist is set to 'ALL'
Change-Id: I5cf5bbf44b9eb653856efb7e60561069f17a66d0
2019-11-28 12:24:16 +00:00
Bogdan Dobrelya
77bc616dc5 Add pypi_fqdn to differentiate it package mirrors
There is mirror_fqdn that, if defined, overrides mirrors used for
packages and also mirrors for pypi. This is generally incorrect,
if one wants to use different mirrors for packages vs pypi eggs.

Add pypi_fqdn that defaults to mirror_fqdn, which allows users
to go with a custom pypi_mirror. Make pypi trusting the given
pypi_fqdn value instead of generally unrelated mirror_fqdn.

Change-Id: I12975b57951699351cfc0d40beaeb7c703651dd0
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
2019-11-28 09:00:07 +01:00
Ian Wienand
cfdf251140 Use RDO trunk repos work for openvswitch on centos8
This is a bit of a hack, but allows installation of openvswitch for
testing the multinode roles.  Official CentOS 8 RDO builds are still a
work in progress, but this will let us get a head-start on any
multi-node testing.

Change-Id: I2eb1a16e9995a19d61e309aa59b232577184b527
2019-11-25 08:27:50 +01:00
Monty Taylor
e79ca5ec3a Stop installing pydistutils.cfg
We install pydistutils.cfg to catch legacy cases where people
are doing things with easy_install. That code path is not
desirable and increasingly broken, as support for things like
allow-hosts is dropped upstream.

Stop installing the file. This will mean people using easy_install
won't get the benefit of the mirrors, but they shouldn't be
using easy_install anyway.

Remove pydistutils.cfg if it's there

Make sure we don't have any pydistutils.cfg around.

Change-Id: I24a05f456f87bd4cb57ebf89d4245477bf675f97
2019-11-24 12:03:39 -05:00
James E. Blair
68ea061fb2 install-podman: also install slirp4netns
This is required in order to perform port forwarding when running
rootless.

Again, unclear why this isn't required in the package.

Change-Id: I61964a23ce7c80b1af149cb7f98897367e970d45
2019-11-21 14:27:54 -08:00
James E. Blair
3c4f23ec43 install-podman: also install uidmap
This package is required in order for UID mapping in rootless
operation to work. It is unclear why it isn't specified as a
dependency.

Change-Id: I7dd22b683c2fed9921de39a474de3bd9f2c3b0d4
2019-11-21 10:09:16 -08:00
vass
0ee411860a Adds go jobs and roles.
Adds jobs to test and build go packages.
Adds a role to download and install go from
the official download page (https://golang.org/dl/).
Adds a role to run commands with go e.g build, test, run.

Change-Id: Iad2d877fffa2530e9fdeec648a60755a80cf01f4
2019-11-20 20:27:21 +01:00
Ian Wienand
6228b71b07 install-devstack: switch to Python 3
Nova has switched to Python 3 only but devstack has not caught up (see
[1]).  Switch on Python 3 here; it will soon be the default and we can
shake out any errors before that.

[1] https://review.opendev.org/694891

Change-Id: Idf26a7af08219421c1a6cc381317e59fb6950b86
2019-11-19 15:20:39 +11:00
Zuul
e26a4d6163 Merge "Role to install podman" 2019-11-18 20:52:24 +00:00
Ian Wienand
26cadea6af install-docker: show installed version
Show full version details of install.  This can be useful when you're
trying to decide what features you may or may not have.

Change-Id: Iedc6aa1b73b0e7d7612c27d890273e84d4d04a22
2019-11-11 12:56:43 +11:00
Ian Wienand
cdff4b7835 Role to install podman
A simple role to install podman packages.

Change-Id: I135fab8474b3564bb4ab1f2b19d4645d4a86f560
2019-11-08 08:51:58 +01:00
Zuul
81d7e7a993 Merge "Remove argument to ssh-keygen for key size" 2019-11-05 20:49:24 +00:00
Ian Wienand
89ed14752b test-swift-log-upload: fix dry_run parameter
This was pointed out and fixed by jhesketh in
Ie4888606a8ca4ffe2eb99ddbbcd9d5cee8ceec44, but I forgot to add it to
the test role.  Sure enough, it fails as jhesketh predicted.

[1] https://review.opendev.org/#/c/592929/3..4/roles/upload-logs-swift/library/zuul_swift_upload.py

Change-Id: I8d25a234fb92cd1bcfcd0cb7ddf9b8222dbd5978
2019-11-01 16:06:22 +11:00
Ian Wienand
9a8cc2a1c0 test-upload-logs-swift: Add missing template for test
I forgot to "git add" this file in
Id91350ff1c531fd7266f3bf76681a8415941481f.

Change-Id: Ia6af42a7d9062fa7895f3b6901a0fa1e8646e8b9
2019-11-01 15:24:29 +11:00
David Shrewsbury
ce67cef122 run-buildset-registry: Fix python version check
The "Start the buildset registry" task, which uses the
docker_container Ansible module, was failing with:

    "No module named 'requests'"

This is because the earlier package install task was installing
the packages into the wrong python version because the
ansible_python_interpreter value was 'auto' and thus never
included the 'python3' value.

Change-Id: I887fdf8d000c8b916fdab281c531b7c98bdd5ae2
2019-10-31 16:17:27 -04:00
Zuul
d7b86832b2 Merge "upload-logs-swift: test bulk download script change" 2019-10-31 06:36:02 +00:00
Zuul
04f7283b27 Merge "upload-logs-swift: Stub out dry run in the uploader" 2019-10-31 06:26:59 +00:00
Ian Wienand
ddc0da55a1 upload-logs-swift: test bulk download script change
This implements the production change
I98c80f657f38c5e1ed5f28e5d36988a3429ad1f8 in the test role.  Review
comments should be left there; we can merge this and then parent a job
to base-test to test it.

Change-Id: Id91350ff1c531fd7266f3bf76681a8415941481f
2019-10-31 06:26:49 +00:00
Ian Wienand
a6f946472d upload-logs-swift: Stub out dry run in the uploader
By doing this, we're not constrained about where to run the uploader
while still providing some useful testing in dry-run mode.

Change-Id: Ie4888606a8ca4ffe2eb99ddbbcd9d5cee8ceec44
2019-10-31 15:43:44 +11:00
Clint Byrum
a997073141 Remove argument to ssh-keygen for key size
We don't need to be explicit, ssh-keygen will pick a good size for us.
Meanwhile, 1024 bit keys are generally seen as security problems and
thus ring bells and are refused by some servers/services.

Change-Id: Iaea82e0b394a5a6b1da3b59637fc4e0f541e1978
2019-10-30 16:43:40 -07:00
James E. Blair
91ae8e62de Test buildset registry with k8s and docker
This adds a test job for the buildset registry with k8s with the
docker runtime.

Change-Id: I0ec1d3d5980424f85f5e7619b9dec96c8f0f70f1
2019-10-28 15:45:39 -07:00
James E. Blair
dba2c101fd Adjust buildset registry container name
If we want to run a second registry on a different port, we'll need
a different name for the docker container.

Change-Id: I887d9015c0d21e6d7f95379e6fa7fb4211e58d3d
2019-10-28 15:42:29 -07:00
James E. Blair
a321a8270d Make the buildset registry port configurable
This will allow us to run the role twice on the same host.  This
will be useful when we test changes to this role.

Change-Id: I97baeb3172298648bcfef26c5be635ad4be036f0
2019-10-28 15:42:29 -07:00
James E. Blair
8766890012 Use zuul-registry as buildset registry
The proxy functionality is no longer needed so it is removed.

Change-Id: I29ff75d331b433ea4ad3b66ed723eee14a90b404
Depends-On: https://review.opendev.org/689829
2019-10-28 15:42:26 -07:00
Zuul
aa7dcaf69c Merge "fetch_subunit: Change variable" 2019-10-21 19:58:39 +00:00
Zuul
ff1c4648e1 Merge "fetch-subunit-output: collect additional subunits (2nd try)" 2019-10-21 19:48:51 +00:00
Andreas Jaeger
293b064d29 fetch_subunit: Change variable
Change new variable zuul_additional_subunit_dirs to
fetch_subunit_output_additional_dirs to name it after the role.

Change-Id: I7ac0b5d343701409a8741439334fe27f5374a258
2019-10-21 21:07:39 +02:00
Zuul
0d39fba5fe Merge "configure-mirrors: Consolidate dnf/yum handler" 2019-10-19 07:45:07 +00:00
David Shrewsbury
a37c21b772 Increase minikube logging output to maximum
Per settings mentioned at: https://minikube.sigs.k8s.io/docs/tasks/debug/

The current logging level is very minimal for tracking down any
potential k8s issues.

Change-Id: I4ebd694481de936f9df790a46e195b251a803c5f
2019-10-17 20:00:01 +00:00
James E. Blair
03e42f874b install-kubernetes: add support for cri-o runtime
This adds optional support for using the cri-o rather than the
(still default) docker runtime.

Change-Id: I0f7cbb758a21cb022ceac24654183138bebf83bf
2019-10-17 11:27:31 -07:00
Luigi Toscano
ab8f9fc403 fetch-subunit-output: collect additional subunits (2nd try)
In addition to the main subunit file from zuul_work_dir,
collect the subunit files from the elements (directories)
of the zuul_additional_subunit_dirs list.

The default behavior is unchanged.

While the documentation of this role states that zuul_work_dir
contains an absolute path, this is not always true.
So make sure to not make any assumption about zuul_work_dir
in order to not fail spectacularly as it happened with the
previous patch[0].

Add also some tests for the role: both the basic case
and with an additional test directory.

[0] https://review.opendev.org/673885

Change-Id: Iabf2e0cf6d86e36a174778367186bbd39a65c3dd
2019-10-17 00:42:28 +02:00
Zuul
6d5826844a Merge "Allow for overriding dns resolvers in install-kubernetes" 2019-10-16 19:30:17 +00:00
Sorin Sbarnea
d3a52f108d configure-mirrors: Consolidate dnf/yum handler
Simplify logic by using a single handler for redhat systems.

Change-Id: I328f0abb602f0d71372812ff64d8ad3f3764fbd0
2019-10-15 18:21:44 +01:00
Ian Wienand
2e684a8da5 configure-mirrors: add CentOS 8
Add repositories for CentOS 8, add base test.

Co-Authored-By: Sorin Sbarnea <ssbarnea@redhat.com>
Change-Id: I24e194a2f4729046c8f521ffccd3b00055127516
2019-10-15 17:33:48 +11:00
Ian Wienand
2462d16322 configure-mirrors: make separate template directories for each platform
This keeps each platform's "/etc" template files in a separate
top-level directory.

Additionally, we add a distro major version match to the task import
and rename centos7 (in preparation for centos8).

Change-Id: If65d51a27e30311b1da20522afb6dbce7ee6cf35
2019-10-15 17:04:57 +11:00
Clark Boylan
7755ef1b8d Use gzip to compress files uploaded to swift
We've discovered that rackspace swift seems to always want to gzip
encode files when clients request their contents. When our files are
deflate encoded this results in files that are first deflate encoded
then gzip encoded. Not all browers or layer 7 firewalls can handle this
(despite being perfectly valid according to the HTTP RFCs). We'll use
gzip to see if that causes rackspace to not double encode the files.

To do this with memory efficienty we vendor a tool from pypi called
gzip-stream which allows us to read chunks of the compressed data at a
time without loading the entire file into memory or writing multiple
gzip headers in a single file.

Change-Id: I9483cfdbd8e7d0683eeb24d28dd6d8b0c0e772fa
2019-10-11 11:32:17 -07:00
Clark Boylan
61c05a180f Test role for upload-logs-swift
This creates a testing role for upload-logs-swift so that we can test
chagnes to this role before moving them into the production role.

Change-Id: If55b0e2809d9309bc98e6763da32219d996dbfa4
2019-10-11 11:32:11 -07:00
Clark Boylan
3c60b35a19 Replace command with shell in persistent-firewall
We continue to see the odd task rc of -13 when running iptables-save to
store persistent set of firewall rules. Switch to shell to give us a bit
more debugging freedom if necessary (and to rule out the command
module).

Change-Id: I0c0208101cad985d4113de6b636c3816613b778c
2019-10-02 10:18:38 -07:00
Zuul
7ce53a7732 Merge "Add unit tests for tox_instalL_sibling_packages" 2019-10-01 19:42:04 +00:00
Mohammed Naser
0193198215 Add unit tests for tox_instalL_sibling_packages
This adds very basic framework and a simple unit test for a function
which had non-Python 3 code.

Change-Id: Ie22402aeacd21942c356d7d7c5c962b40a29caa4
2019-10-01 19:18:25 +00:00
Zuul
26c1592e77 Merge "Allow upload-logs to toggle compression of console logs" 2019-10-01 18:56:10 +00:00
Clark Boylan
121baa0d77 Revert "Use import_role on persistent-firewall"
This reverts commit 46b7b6e1c98a8b12647be4b30b5b54405379d6ec.

This didn't end up changing the incidence of the iptables-save command
task failures.

Change-Id: I02e725d7330bc9b438a9864ea49510cca7fee524
2019-10-01 09:11:02 -07:00
Zuul
2689c24414 Merge "Use import_role on persistent-firewall" 2019-09-30 23:17:19 +00:00
Clark Boylan
40d32838d5 Escape special filename characters in urls
The href url paths need to have quoted filenames to handle cases where
filenames have special characters like : in them.

Change-Id: I0bc0de8d27c6e45c4a6b8841985b8265f0219df2
2019-09-30 14:12:18 -07:00
Clark Boylan
46b7b6e1c9 Use import_role on persistent-firewall
Previously to persist the filewall we were including the
persistent-firewall role. This seems to occasionally break because the
second invocation of the role (on multinode jobs after setting up the
multinode bridge) fails with an RC of -13 when listing ipv4 iptables
roles. Then when we try to write them to disk the variable is empty.

One thought is that dynamically loading the role multiple times may be
confusing ansible. Use import_role to statically load the role instead
and see if this helps.

Change-Id: I2458f8eb4c2e4638336fa14e436e13b5a2263cce
2019-09-30 10:39:02 -07:00
Colleen Murphy
18285584f1 Use iptables for openSUSE
In openSUSE Tumbleweed, the SuSEfirewall2 package was removed in favor
of firewalld[1]. This commit updates the iptables persistance tasks to
avoid using SuSEfirewall2 and instead use rc.local to restore saved
rules upon restart, and undefines the iptables_service variable for SUSE
since there is no service to restart any more. See the related change
for image builds[2].

[1] https://lists.opensuse.org/opensuse-factory/2019-01/msg00490.html
[2] https://review.opendev.org/683236

Change-Id: I0f8d74dd00df192c20b96a9368b964839c306171
2019-09-27 15:45:01 -07:00
Zuul
01f6c26e29 Merge "Remove invalid argument from format_exc() calls" 2019-09-24 08:29:42 +00:00
Zuul
d5edece13c Merge "Add more languages for build-releasenotes" 2019-09-24 02:43:39 +00:00
Ian Wienand
20bffbcfc0 Remove invalid argument from format_exc() calls
The argument here is an integer "limit", not the exception.

I think that we only notice this on Python 3 because of exception
chaining.  It causes a real failure though because the exception
handler that is meant to fall into "pass" raises another exception
when ipv6 doesn't work.

Change-Id: I0908a0a3dbb2356caabbffd062379751a0b61c41
2019-09-24 12:15:28 +10:00
Andreas Jaeger
0f42cbb88b Add more languages for build-releasenotes
Update list of languages since we have some more translations.

Change-Id: Ib9636132549673908660e47ad5e33ad3d9386c0f
2019-09-23 14:01:05 +02:00