499 Commits

Author SHA1 Message Date
Monty Taylor
f6a92ecb6c
Make subunit processing more resilient
Use subunit2html from tox envs instead of os-testr-env

We have an openstack-infra hard-coded location in this role. Rework it
to try looking in the tox env if one exists or to look on the system.

Leave /usr/os-testr-env in as a fallback but with a debug statement so
that we can track instances of the fallback being used.

tox_envlist can be a comma separated list. If it is, the "find
subunit2html in a tox env" logic isn't going to work.

Use a shell script so that we can do a first-found logic similar to what
we use in ensure-sphinx for finding requirements files.

It's a big bash blob. Put it in a file so it syntax highlights correctly
and execute it with the script module.

Change-Id: I0b468840fd04a379007fe5aca07feb2eaf7081ac
2018-01-21 10:39:49 -06:00
Monty Taylor
fa54ff21e1
Clean up when conditions in fetch-subunit-output
Review comments from earlier requested a cleaner approach to the whens
with using blocks. This patch started off as that, then realized we
could make it even clearer by collapsing the tox_envlist when as well.

Change-Id: I2a32f1aa802c0727b7d15f24f58322d5e26c8693
2018-01-16 12:32:49 -06:00
Monty Taylor
2a0f06147e
Adjust check for .stestr directory
stestr can leave a .stestr directory from which subunit cannot be
generated after an initial run that fails due to a python import/parse
error.

In order to determine if a run will be able to generate subunit output,
check for the '.stestr/failing' file.

NOTE: This is **NOT** an indication of a failing run. It's a file that
will be present in the .stestr directory if an entire run has happened
and acts as a marker that the directory has enough information to export
a subunit stream.

Change-Id: Id1dd696c81da09e9d2e5fe350337b73eb0646d2d
2018-01-16 11:58:24 -06:00
Monty Taylor
3d0550a96c
Revert "Revert "Add consolidated role for processing subunit""
This reverts commit 9f4efe98e10c9d553dc8a15bd7b84584e1118684.

Compress testr results.html before fetching it

The synchronize command has testr_results.html.gz in the file list, but
we don't ever gzip it so it fails on fetch.

Make the "Check for testr directory" task actually check for
.testrepository and not for .stestr a second time.

The compression bug broke only for people using stestr. For people
using testr, the path bug meant they don't get html reports.

Change-Id: I0cdfc66ee8b046affeb0b071fef38c21cb7a4948
2018-01-16 11:58:18 -06:00
Monty Taylor
9f4efe98e1 Revert "Add consolidated role for processing subunit"
This reverts commit ebdcf57fb088ba09db51a1f2bcbc915731d04bd2.

Change-Id: Icfa811571ec7c26617e0f8e9db086001fc0bc81f
2018-01-16 01:10:56 +00:00
Monty Taylor
ebdcf57fb0
Add consolidated role for processing subunit
We don't need two roles for testr and stestr. The roles have logic to
check to see if .testrepository or .stestr dirs exist, and the logic
after the subunit extraction is the same. Combine them into a single
role which should do the right thing.

Also, document it.

Change-Id: I833320cf9a932d8e119645eb798ce0c93d854321
2017-12-20 08:33:42 -06:00
Andreas Jaeger
57fbc6e96e Do not use --ignore-missing-args for rsync
rsync --ignore-missing-args is only supported with rsync 3.1. CentOS 7
uses rsync 3.0 and fails.

We ignore missing remote files, so remove the argument and let the role
ignore an error if the files do not exist.

Update fetch-tox-output to use failed_when as well. We really want the
role not to fail (failed_when) instead of showing a failure and then the
playbook ignore it (ignore_errors).

Change-Id: I24bfb5519acccd7fd36b4839bb598436b4c8f01c
2017-12-20 15:04:42 +01:00
Zuul
7d5f01c88e Merge "Dereference symlinks when staging artefacts" 2017-12-19 13:58:02 +00:00
Clark Boylan
bd579c2b5b Support autodoc_tree_index_modules in sphinx role
PBR supports two flags for autodoc in sphinx, autodoc_index_modules and
autodoc_tree_index_modules. We were only supporting
autodoc_index_modules but some projects like keystone use
autodoc_tree_index_modules. Address this by setting the autodoc flag to
true if either is set to true and defer to pbr to figure out what it
means.

Change-Id: If856e4b838d013781c0690b4ac1c2b37900d0fdd
2017-12-18 19:51:02 -08:00
Monty Taylor
1e1c54e23b Handle projects still using pbr autodoc_index_modules
There are a non-zero number of projects in OpenStack that are still
using pbr.autodoc_index_modules. Switching to the new sphinx build job
breaks them if they have warning-is-error turned on, as their docs are
expecting a reference to the autogenerated module doc indexes.

We should come up with a better solution for migrating them ... probably
a sphinx plugin that one can use with some options and whatnot. Or we
can commit a copy of the generated autoindex file and then treat it as
actual source code moving forward.

For now we need to run those projects with the pbr modified sphinx
builder. This adds support to the setup.cfg value lookup module
for looking up autodoc_index_modules and plumbs it in to the sphinx
invocation.

A note has also been left with an idea for how to simplify this and
remove the python module (we can ini lookup on the executor) but
implementing that is left as an exercise for later.

Change-Id: Ie0c9f24df09255e871f904e079b68809144b36b4
2017-12-18 21:31:56 +01:00
Monty Taylor
0dc2c9466d
Deal with lack of requirements file better
For projects that only need sphinx and no other actual depenedencies,
there's no real value in failing if we don't find a doc requirements
file. We know all things using ensure-sphinx will, in fact, need sphinx.
Past that, it's entirely reasonable for someone to have no other needs.

Change-Id: I07dfa12f4aceee50a2f2c000df937dcbb09dd557
2017-12-18 12:25:03 -06:00
Andrea Frittoli (andreaf)
68dec9ecc5 Dereference symlinks when staging artefacts
If an artefact is a symlink, dereference it before coying it as
just copying the link may render it broken one the staged folder
is synced to the executor.

Change-Id: I5ad4dfccbc1389da3afc53f3c866d3475e006db6
2017-12-18 16:06:41 +00:00
Zuul
aed3385b26 Merge "Fetch built web content in javascript jobs" 2017-12-17 19:11:45 +00:00
Monty Taylor
89f68c5ca0
Fetch built web content in javascript jobs
If a javascript job produces web content that should be deployed (such
as zuul-web or storyboard-webclient) we should collect that in the post
step of the job and put it into the html/ directory.

While in there, add a bunch of get_*: false entries for the stat calls.
Calculating the md5 and whatnot is expensive, and we should really
always set them to false unless we're *using* the hash values.

Change-Id: I9b96656aa5f225e4b069a78442cf0159cadc9009
2017-12-16 20:07:02 -06:00
Zuul
f4d0223639 Merge "translations: Check for translation-source first" 2017-12-16 16:17:47 +00:00
Zuul
6fa855b476 Merge "Half-Revert "Revert "Add ensure-reno and ensure-babel roles""" 2017-12-16 15:55:12 +00:00
Andreas Jaeger
85f7b0a793 translations: Check for translation-source first
Translation jobs might not produce the translation-source directory but
still succeed, e.g. if the branch is not translated.

Check for the translation-source directory and only fetch files if there
is really something to fetch.

Change-Id: I064873412bf8503c5e7d317739adb891315038f8
2017-12-16 15:45:36 +00:00
Monty Taylor
3bdb8d22bf Half-Revert "Revert "Add ensure-reno and ensure-babel roles""
Add the ensure-babel role back.

This reverts half of commit 89f8a9684c5da2ea5e09dd37e84dc612319c6559.

Change-Id: I1334ff1f469061884b222dd99e72a989d72c68be
2017-12-16 15:31:17 +01:00
Monty Taylor
8a86f7c7fc
Don't run npm shrinkwrap if using yarn
If we're using yarn, it means we started from a yarn.lock file in the
first place. If that's true, we don't need to *produce* a shrinkwrap
file.

Needed-By: I848f22dce6196b2df9de4b9f585e857e03cdb0d7
Change-Id: I6519ff5ea7ac4b79e46f29c6190d660718d03f1b
2017-12-15 19:26:30 -06:00
Zuul
96006880a7 Merge "npm-test needs xvfb etc" 2017-12-15 18:22:34 +00:00
Andreas Jaeger
b809cc556b npm-test needs xvfb etc
The npm-test run for most projects needs the special pre-test job that
the npm changes removed. Readd it to test job.

Also, do no base on nodejs-npm since we need to override the pre script
and do not want both pre scripts to run.

Needed-By: Ibb4d38f150ab8f2029ea704cab0c9719efd57396
Change-Id: Id533690cfad380ceaa88ad08f5c97f35112c8f79
2017-12-15 18:53:26 +01:00
Zuul
5a190a4c03 Merge "Install yarn if needed in javascript jobs" 2017-12-15 17:11:54 +00:00
Zuul
09b793cc69 Merge "Revert "Revert "Add sphinx_python variable to sphinx role and job""" 2017-12-15 14:59:30 +00:00
Akihiro Motoki
9cfd8244ce Fix npm-run-test job
npm-run-test is still broken. There is neither of parent and run.

Change-Id: I2bd961e902b1fbd269fc98517e09600290e6ef50
2017-12-15 21:25:11 +09:00
Andreas Jaeger
0e36e746d5 Fix npm-run-test job
Change I5f9d420f1618e9a196b4218bca7055f377349351 updated all npm tests
but forgot to update npm-run-test, jobs now fail with
"cannot find ... playbooks/javascript/run-test.yaml".

Update job the same way as other jobs.

Change-Id: I6fdf9f9c55ce0744012d42afaa940b325669c36e
2017-12-15 11:03:06 +00:00
Monty Taylor
a08002fd44
Install yarn if needed in javascript jobs
Some javascript projects use yarn for dependencies instead of raw npm.
If a yarn.lock file is detected in the zuul_work_dir, install yarn and
use yarn to install dependencies.

Needed-By: I5864e1b2f9952fbc35b08ebdef348c4917564c37
Change-Id: I9ae7dd3b33b2b607c1c5a17632e2aeb12e339d15
2017-12-14 10:08:22 -06:00
Monty Taylor
b125b917d9
Update javascript jobs to take npm_command variable
Similar to tox, the only difference in these is the command passed to
npm. Update the base job to do that, and the child jobs to use the
variable.

Change-Id: I5f9d420f1618e9a196b4218bca7055f377349351
2017-12-14 09:42:11 -06:00
Zuul
d3955a9d3f Merge "Convert back to zuul.projects" 2017-12-13 16:01:13 +00:00
Andreas Jaeger
88279bd6cc Revert "Revert "Add sphinx_python variable to sphinx role and job""
The first version of this change was broken, fix it with updating
roles/ensure-sphinx/tasks/main.yaml to run the test-requirements check
really on the remove node.

This reverts commit 68ded2251f34d2938eb7d91e846dd0e8ad6c892f.

Change-Id: I481e032834fdbf674157b2c9a8fa6f95fc570ddb
2017-12-13 16:31:26 +01:00
Zuul
cacbc30ee0 Merge "releasenotes: Remove package install" 2017-12-12 20:19:21 +00:00
Andreas Jaeger
ebb165eb69 releasenotes: Remove package install
The vast majority of changes to enable running releasenotes without
installating of the package itself are merged. Remove the package
install now. Note that install-if-python is the proper role for this,
the previous setting was a noop.

The remaining few changes can merge at any time, projects had enough
time.

See also
http://lists.openstack.org/pipermail/openstack-dev/2017-November/124815.html

Needed-By: I225d73a2f9321329b34a282e08aea6773622df5b
Change-Id: I6d461a69537d3f7dd44e9b9877f652df656eedd8
2017-12-11 14:16:08 +01:00
Jeremy Stanley
8b577d4106 Change to the executor workdir during npm upload
When using npm to install locally, it wants to write a node_modules
tree in the current path. Since the default path of a command is the
path where the playbook resides and this is not writeable on a zuul
executor, change to the writeable workdir first.

Change-Id: Ib4f632c0b9e3f5842e3cd649b6ca7ecb8042e7e3
2017-12-09 17:27:56 +00:00
Zuul
050bbf4692 Merge "Pin to npm4 until npm 5.6.0 comes out" 2017-12-09 16:03:53 +00:00
Zuul
192d736505 Merge "fetch-tox-output: remove --ignore-missing-args" 2017-12-09 11:20:42 +00:00
Zuul
10a30c8141 Merge "tox: make install_sibling_packages pip import optional" 2017-12-09 11:20:42 +00:00
Monty Taylor
0cd5075466
Pin to npm4 until npm 5.6.0 comes out
npm5 has an issue uploading tarballs it didn't build. For now, install
npm4 and use that to upload our tarballs.

The fix for https://github.com/npm/npm/issues/16723 should be in 5.6.0
which should allow us to revert this.

Change-Id: I8f633ee2119218799706b47ba3b471dcb04c1253
2017-12-08 17:20:51 -06:00
Zuul
23c9fde929 Merge "Revert "Add sphinx_python variable to sphinx role and job"" 2017-12-08 12:31:32 +00:00
Jens Harbott (frickler)
68ded2251f Revert "Add sphinx_python variable to sphinx role and job"
This reverts commit 1f8c4737651b1d64eda909e1a6641673f60ebbeb.

Change-Id: I98e7414337d58af0f4819163ee0e88702a52c051
2017-12-08 11:10:45 +00:00
Zuul
d50fa12a51 Merge "Add sphinx_python variable to sphinx role and job" 2017-12-08 00:11:39 +00:00
Zuul
ab60637718 Merge "Fix npm-run-test" 2017-12-07 16:33:27 +00:00
Zuul
e8967612be Merge "bindep: skip package install when not needed" 2017-12-07 16:33:27 +00:00
Zuul
21a2b108b3 Merge "bindep: fix install bindep_command fact name" 2017-12-07 16:33:26 +00:00
Zuul
f928de5a0d Merge "revoke-sudo: only revoke when zuul is sudoer" 2017-12-07 16:33:25 +00:00
Monty Taylor
2862cddb6a
Source nvm before using it
According to https://github.com/creationix/nvm#install-script,
we should, you know, actually source the script rather than just cloning
the repo and then ignoring it.

Change-Id: I046af8d38cf7288fcb3fac74cd6839d4e006ca90
2017-12-06 11:29:11 -06:00
David Moreau Simard
1f8c473765
Add sphinx_python variable to sphinx role and job
Some projects, such as Zuul itself, need to use python3 to build docs.

This adds a new parameter, sphinx_python, which defines the version
of python to build documentation with. The default is python2 and
projects can choose python3 on a need basis.

Remove the comment about the need for a leading comment, since there
is now a real need for the leading comment anyway.

Replace a few direct references to {{ ansible_user_dir }}/.venv that
should have been using zuul_work_virtualenv.

Update sphinx invocation to source the virtualenv activate first.
Some sphinx modules, such as sphinxcontrib.programoutput, attempt to
execute programs and only invoking sphinx-build with the relative
path causes the paths to not be set up properly.

Co-Authored-By: David Moreau Simard <dmsimard@redhat.com>
Change-Id: Ie5e2c93f88465f4aa746827ff88a585dbaa44fd5
2017-12-05 21:10:15 -05:00
Tristan Cacqueray
53ff9f166e fetch-tox-output: remove --ignore-missing-args
This is no longer needed since the task errors are now ignored.
Moreover this option doesn't work on el7.

Change-Id: Ic6b32999eef2fa26d63ed5d34b3b05b3ade60b32
2017-12-05 09:20:49 +00:00
Tristan Cacqueray
62015d9e15 tox: make install_sibling_packages pip import optional
This enable using the tox role without pip module installed on the host

Change-Id: Ic2c9e75ed967faf93267b6b9c58fdf3b02837ea9
2017-12-05 09:13:01 +00:00
Tristan Cacqueray
ea5c3d08da bindep: skip package install when not needed
When bindep test doesn't output package, there is no need
to run the package task.

Change-Id: I6f13e060a6d9657d3dec4f43a0b9c2dbff2d362d
2017-12-05 09:09:34 +00:00
Tristan Cacqueray
184f036fd8 bindep: fix install bindep_command fact name
After installing bindep, the role should set the bindep_command fact
instead of bindep_found_command. Otherwise the role fail later:

Task: Define bindep_command fact
The error was: 'bindep_command' is undefined

Change-Id: I11a21c9678c75b7887a8a58c25925dc87e7c9315
2017-12-05 08:52:31 +00:00
Tristan Cacqueray
892dc6a095 revoke-sudo: only revoke when zuul is sudoer
This change makes unittests jobs usable on read-only environment.

Change-Id: I36cfe7e5849687dbed510396a825dc0ec45542b3
2017-12-05 08:36:26 +00:00