The `tox_parse_output` currently logs the entire output of tox
into the syslog as the natural Ansible behaviour, as part of doing
that, it also encodes it inside utf-8. This results in errors
in the case that the the data contains surrogates and crashes the
entire job.
Given that this issue really occurs during logging and we don't
have a reason to log all of the tox output to the syslog, this
patch sets the module parameter to `no_log` which should skip the
encoding and logging for that parameter[0].
[0]: https://github.com/ansible/ansible/blob/stable-2.9/lib/ansible/module_utils/basic.py#L1945-L1957
Change-Id: I470cd35111a63662c43f779fc9e119aa8d78c838
Retrieving file comments is not an essential part of the job and
should not fail the job if something goes wrong.
Change-Id: Ide13343c6eda3a1fd0f45cb33cba35f3b048ece6
This is an alternative to I98c80f657f38c5e1ed5f28e5d36988a3429ad1f8
which does not modify the upload script, but rather queries the API
and manifest for what to download.
The script is a hybrid of python and bash to not implement json API
parsing badly in bash, but not replicate curl badly in python either.
The script sanity checks for dependencies, which are considered pretty
standard for any developer who would be interested in downloading logs
like this.
The role writes out the script with the correct build values coded
into it, so a potential user just has to run it without any arguments
or modification.
Change-Id: Ic33732adbfd3210191bf4976c3ee316cfc50568e
Allow us to define which packages need to be removed before installing
the desired ones.
Fixes use of role on systems where it was desired to install upstream
docker but the system one was already installed.
Change-Id: Ib6ae0a8fba937901b29565e2d36486eccf4f67bb
Adds information that is can be key regarding the outcome of the job:
* distribution name
* distribution version
Change-Id: I8362ef2f0a89de4c318f13c874b6f083a4f1699f
We forgot to revert test-upload-logs-swift with the revert
4f13f7c07f101ee34ebfcfdf6175adb3857f8ac5 in the production role.
Change-Id: I4977091b1ab8e5188a71d16e609c43a2482496c4
The install-kubernetes role is now failing due to the fact that
minikube now requires conntrack. This patch solves this.
Change-Id: I1e59d5e9255064646bd8e8c6b6edb2c34d45b516
This reverts commit acde44818d0827174b22290192b3c728c440a1c5 and
the testing part of b3f417a6e6fb9539d4f7aee46a123e7ced76d0b5.
We'd like to obtain more consensus on the download script before
we commit to this. In particular, the new zuul manifest file may
make it possible to do this without adding the feature to the
log upload roles.
Change-Id: I959c44b4dac6cad6d1b3d82ba6bc0949c9c759ff
The uri module faithfully passes the entire string, even if it has
trailing whitespace such as a newline. The zuul encrypt_secret
command currently does not trim, so if echo was used instead of
echo -n this can fail in a hard to debug manner.
Change-Id: Ic9525ac2925b6639f58604ca40dc878d20511ff8
Everything's better with some unicode sprinkled in. Add a unicode
filename to keep unit testing on its toes.
Note this is duplicated across the test role too.
Change-Id: Iaefe9bea2c1a10d440ef75df3acd71fdd9a4157e
After we have determinted the root URL, create a download script from
a given jinja2 template. This is added to the file list at the root
and uploaded with the other files.
Generated index files are given a new flag so they can be
differentiated.
This is an impelementation of
Iea8abd4cd71ece26b51335642f73bd2e544c42dd for the swift-upload role.
Change-Id: I98c80f657f38c5e1ed5f28e5d36988a3429ad1f8
We can just find the basename of the template file, and return that --
the artifact will be a relative URL which will be right.
For additional help, add a metadata string you could copy-paste to
download.
Change-Id: I1a6f931fdd1613d1e51a20ed76a69c76bbb14dc1
Some time ago we merged Id91350ff1c531fd7266f3bf76681a8415941481f to
test the idea of a download script; proposed in
I98c80f657f38c5e1ed5f28e5d36988a3429ad1f8.
However, we never actually go to either merging or abandoning the
actual change in I98c80f657f38c5e1ed5f28e5d36988a3429ad1f8.
This adds the download script as an artifact, which brings the testing
role in-line with the latest thinking in the actual job proposed in
I98c80f657f38c5e1ed5f28e5d36988a3429ad1f8.
Once tested, we can then actually get reviews this time and decide to
either go with this, or revert the whole thing and remove it from the
testing role.
Change-Id: Id1d8b04ff65a4dc567b73f22bcfb2795a3f13d7d
CentOS8 deps should be compatible with RHEL and provide latest versions
of deps compared to the old rhel8 one. Let's switch to use it to install
openvswitch too.
Change-Id: I04090a179706c52e3db4e9118b74f7c7d5ad9776
In order to install particular python versions on distros that
don't otherwise have them, add an option for installing via
pyenv.
The packages to be installed to allow for Python to be built were
tested on the official Docker containers so they should provide
a fair amount of coverage from a minimal environment.
Co-Authored-By: Mohammed Naser <mnaser@vexxhost.com>
Change-Id: Ic3312458b499a4b743895fa5829bb25155f77654
Inventory hostnames like "abc/123" is valid in both ansible and zuul
but this role breaks since it uses it as part of a path. This sanitizes
the hostname from "abc/123" to "abc_123".
Change-Id: Ic89d595b6f004b5ca4805f1af8387e8ba56564aa
At the moment, the build registry is not used inside Kubernetes jobs
and it is required to override the entire pre.yaml just to enable it.
This patch adds an option of using docker_use_buildset_registry inside
install-docker which can be used in order to install Docker and letting
it use the buildset registry simply by adjusting the job.vars
Change-Id: I1b42eac6accbf7c350aee76d18a823ba6327548d
This refactors the code inside helm-template which waits for all
pods to become Ready to move inside it's own seperate role instead
which can be used for other tasks.
Change-Id: Ibb234c46c49fe656cf918a7a4af115e0d26f23f0
The role currently defaults to installing tox with python2 in order
to maintain compatibility with existing jobs. This change makes the
default behaviour use python3 as python2 is EOL.
Change-Id: Ic41b8fee7ae8a8f5df6c1107c6ed812c14b8c8be
minikube has added a new "feature" which pre-loads a blob of data
with everything it needs that we can't disable, with a fallback.
we're currently downloading ~500MB and failing to use it because
it's compressed by lz4 which we don't have in images, this change
adds that to the package list.
Change-Id: If74faf2b21d564fa23253ea2eb5378457f055735
The `mirror-workspace-git-repos` uses Git to mirror repos from the
executor to build nodes. This comes in handy when the node already
holds a copy of the repo(s), as Git can incrementally update them.
Currently, `mirror-workspace-git-repos` assumes the build node is
directly reachable via ssh. This is (usually) not the case when
running against a Kubernetes Pod build node, thus we can not
efficiently update repos on such nodes using Git.
This change allows pushing Git repos to Kubernetes Pods by bridging
the Git communication over `kubectl` and thereby allowing for the
same behaviour as for standard, non-container, ssh-reachable build
nodes.
(cf. https://git-scm.com/docs/git-remote-ext)
Change-Id: I71919d3bc93e48d5bb24234265fca9c84ce23b87
This patch enables the ability to set a value which prefers the
installation of tox within python2 over python3.
Change-Id: I29c9585aa9048c0e2855ec1eaf1f48041cfe46e2
We currently run 'git gc' on repos before creating the tarball for the
node cache. On busy repos this is not enough to cut down the size of
the repo. With more aggressive pruning and packing one of our repos in
the cache got down from 6GB to 1.8GB.
Change-Id: I7ef428c25fde713efabd966d59b5bd97ec248d7e
Implements workaround for the missing --nobest option in Ansible
package module in order to succeed installing docker-ce.
Change-Id: I96e133dc5c59b9c811ff69df7651adadfcdf3a97
This works around missing parent directories in AFS, since rsync
will create the target directory but not its parent directories. In
cases where the parent directory does not yet exist, it's nice to
create it automatically first, if we can.
Change-Id: Ia136ea3a416d6085d1f6e277b4c071573c5660e4
