385 Commits

Author SHA1 Message Date
Monty Taylor
f6a92ecb6c
Make subunit processing more resilient
Use subunit2html from tox envs instead of os-testr-env

We have an openstack-infra hard-coded location in this role. Rework it
to try looking in the tox env if one exists or to look on the system.

Leave /usr/os-testr-env in as a fallback but with a debug statement so
that we can track instances of the fallback being used.

tox_envlist can be a comma separated list. If it is, the "find
subunit2html in a tox env" logic isn't going to work.

Use a shell script so that we can do a first-found logic similar to what
we use in ensure-sphinx for finding requirements files.

It's a big bash blob. Put it in a file so it syntax highlights correctly
and execute it with the script module.

Change-Id: I0b468840fd04a379007fe5aca07feb2eaf7081ac
2018-01-21 10:39:49 -06:00
Monty Taylor
fa54ff21e1
Clean up when conditions in fetch-subunit-output
Review comments from earlier requested a cleaner approach to the whens
with using blocks. This patch started off as that, then realized we
could make it even clearer by collapsing the tox_envlist when as well.

Change-Id: I2a32f1aa802c0727b7d15f24f58322d5e26c8693
2018-01-16 12:32:49 -06:00
Monty Taylor
2a0f06147e
Adjust check for .stestr directory
stestr can leave a .stestr directory from which subunit cannot be
generated after an initial run that fails due to a python import/parse
error.

In order to determine if a run will be able to generate subunit output,
check for the '.stestr/failing' file.

NOTE: This is **NOT** an indication of a failing run. It's a file that
will be present in the .stestr directory if an entire run has happened
and acts as a marker that the directory has enough information to export
a subunit stream.

Change-Id: Id1dd696c81da09e9d2e5fe350337b73eb0646d2d
2018-01-16 11:58:24 -06:00
Monty Taylor
3d0550a96c
Revert "Revert "Add consolidated role for processing subunit""
This reverts commit 9f4efe98e10c9d553dc8a15bd7b84584e1118684.

Compress testr results.html before fetching it

The synchronize command has testr_results.html.gz in the file list, but
we don't ever gzip it so it fails on fetch.

Make the "Check for testr directory" task actually check for
.testrepository and not for .stestr a second time.

The compression bug broke only for people using stestr. For people
using testr, the path bug meant they don't get html reports.

Change-Id: I0cdfc66ee8b046affeb0b071fef38c21cb7a4948
2018-01-16 11:58:18 -06:00
Monty Taylor
9f4efe98e1 Revert "Add consolidated role for processing subunit"
This reverts commit ebdcf57fb088ba09db51a1f2bcbc915731d04bd2.

Change-Id: Icfa811571ec7c26617e0f8e9db086001fc0bc81f
2018-01-16 01:10:56 +00:00
Monty Taylor
ebdcf57fb0
Add consolidated role for processing subunit
We don't need two roles for testr and stestr. The roles have logic to
check to see if .testrepository or .stestr dirs exist, and the logic
after the subunit extraction is the same. Combine them into a single
role which should do the right thing.

Also, document it.

Change-Id: I833320cf9a932d8e119645eb798ce0c93d854321
2017-12-20 08:33:42 -06:00
Andreas Jaeger
57fbc6e96e Do not use --ignore-missing-args for rsync
rsync --ignore-missing-args is only supported with rsync 3.1. CentOS 7
uses rsync 3.0 and fails.

We ignore missing remote files, so remove the argument and let the role
ignore an error if the files do not exist.

Update fetch-tox-output to use failed_when as well. We really want the
role not to fail (failed_when) instead of showing a failure and then the
playbook ignore it (ignore_errors).

Change-Id: I24bfb5519acccd7fd36b4839bb598436b4c8f01c
2017-12-20 15:04:42 +01:00
Zuul
7d5f01c88e Merge "Dereference symlinks when staging artefacts" 2017-12-19 13:58:02 +00:00
Clark Boylan
bd579c2b5b Support autodoc_tree_index_modules in sphinx role
PBR supports two flags for autodoc in sphinx, autodoc_index_modules and
autodoc_tree_index_modules. We were only supporting
autodoc_index_modules but some projects like keystone use
autodoc_tree_index_modules. Address this by setting the autodoc flag to
true if either is set to true and defer to pbr to figure out what it
means.

Change-Id: If856e4b838d013781c0690b4ac1c2b37900d0fdd
2017-12-18 19:51:02 -08:00
Monty Taylor
1e1c54e23b Handle projects still using pbr autodoc_index_modules
There are a non-zero number of projects in OpenStack that are still
using pbr.autodoc_index_modules. Switching to the new sphinx build job
breaks them if they have warning-is-error turned on, as their docs are
expecting a reference to the autogenerated module doc indexes.

We should come up with a better solution for migrating them ... probably
a sphinx plugin that one can use with some options and whatnot. Or we
can commit a copy of the generated autoindex file and then treat it as
actual source code moving forward.

For now we need to run those projects with the pbr modified sphinx
builder. This adds support to the setup.cfg value lookup module
for looking up autodoc_index_modules and plumbs it in to the sphinx
invocation.

A note has also been left with an idea for how to simplify this and
remove the python module (we can ini lookup on the executor) but
implementing that is left as an exercise for later.

Change-Id: Ie0c9f24df09255e871f904e079b68809144b36b4
2017-12-18 21:31:56 +01:00
Monty Taylor
0dc2c9466d
Deal with lack of requirements file better
For projects that only need sphinx and no other actual depenedencies,
there's no real value in failing if we don't find a doc requirements
file. We know all things using ensure-sphinx will, in fact, need sphinx.
Past that, it's entirely reasonable for someone to have no other needs.

Change-Id: I07dfa12f4aceee50a2f2c000df937dcbb09dd557
2017-12-18 12:25:03 -06:00
Andrea Frittoli (andreaf)
68dec9ecc5 Dereference symlinks when staging artefacts
If an artefact is a symlink, dereference it before coying it as
just copying the link may render it broken one the staged folder
is synced to the executor.

Change-Id: I5ad4dfccbc1389da3afc53f3c866d3475e006db6
2017-12-18 16:06:41 +00:00
Zuul
aed3385b26 Merge "Fetch built web content in javascript jobs" 2017-12-17 19:11:45 +00:00
Monty Taylor
89f68c5ca0
Fetch built web content in javascript jobs
If a javascript job produces web content that should be deployed (such
as zuul-web or storyboard-webclient) we should collect that in the post
step of the job and put it into the html/ directory.

While in there, add a bunch of get_*: false entries for the stat calls.
Calculating the md5 and whatnot is expensive, and we should really
always set them to false unless we're *using* the hash values.

Change-Id: I9b96656aa5f225e4b069a78442cf0159cadc9009
2017-12-16 20:07:02 -06:00
Zuul
f4d0223639 Merge "translations: Check for translation-source first" 2017-12-16 16:17:47 +00:00
Zuul
6fa855b476 Merge "Half-Revert "Revert "Add ensure-reno and ensure-babel roles""" 2017-12-16 15:55:12 +00:00
Andreas Jaeger
85f7b0a793 translations: Check for translation-source first
Translation jobs might not produce the translation-source directory but
still succeed, e.g. if the branch is not translated.

Check for the translation-source directory and only fetch files if there
is really something to fetch.

Change-Id: I064873412bf8503c5e7d317739adb891315038f8
2017-12-16 15:45:36 +00:00
Monty Taylor
3bdb8d22bf Half-Revert "Revert "Add ensure-reno and ensure-babel roles""
Add the ensure-babel role back.

This reverts half of commit 89f8a9684c5da2ea5e09dd37e84dc612319c6559.

Change-Id: I1334ff1f469061884b222dd99e72a989d72c68be
2017-12-16 15:31:17 +01:00
Monty Taylor
8a86f7c7fc
Don't run npm shrinkwrap if using yarn
If we're using yarn, it means we started from a yarn.lock file in the
first place. If that's true, we don't need to *produce* a shrinkwrap
file.

Needed-By: I848f22dce6196b2df9de4b9f585e857e03cdb0d7
Change-Id: I6519ff5ea7ac4b79e46f29c6190d660718d03f1b
2017-12-15 19:26:30 -06:00
Zuul
5a190a4c03 Merge "Install yarn if needed in javascript jobs" 2017-12-15 17:11:54 +00:00
Zuul
09b793cc69 Merge "Revert "Revert "Add sphinx_python variable to sphinx role and job""" 2017-12-15 14:59:30 +00:00
Monty Taylor
a08002fd44
Install yarn if needed in javascript jobs
Some javascript projects use yarn for dependencies instead of raw npm.
If a yarn.lock file is detected in the zuul_work_dir, install yarn and
use yarn to install dependencies.

Needed-By: I5864e1b2f9952fbc35b08ebdef348c4917564c37
Change-Id: I9ae7dd3b33b2b607c1c5a17632e2aeb12e339d15
2017-12-14 10:08:22 -06:00
Zuul
d3955a9d3f Merge "Convert back to zuul.projects" 2017-12-13 16:01:13 +00:00
Andreas Jaeger
88279bd6cc Revert "Revert "Add sphinx_python variable to sphinx role and job""
The first version of this change was broken, fix it with updating
roles/ensure-sphinx/tasks/main.yaml to run the test-requirements check
really on the remove node.

This reverts commit 68ded2251f34d2938eb7d91e846dd0e8ad6c892f.

Change-Id: I481e032834fdbf674157b2c9a8fa6f95fc570ddb
2017-12-13 16:31:26 +01:00
Jeremy Stanley
8b577d4106 Change to the executor workdir during npm upload
When using npm to install locally, it wants to write a node_modules
tree in the current path. Since the default path of a command is the
path where the playbook resides and this is not writeable on a zuul
executor, change to the writeable workdir first.

Change-Id: Ib4f632c0b9e3f5842e3cd649b6ca7ecb8042e7e3
2017-12-09 17:27:56 +00:00
Zuul
050bbf4692 Merge "Pin to npm4 until npm 5.6.0 comes out" 2017-12-09 16:03:53 +00:00
Zuul
192d736505 Merge "fetch-tox-output: remove --ignore-missing-args" 2017-12-09 11:20:42 +00:00
Zuul
10a30c8141 Merge "tox: make install_sibling_packages pip import optional" 2017-12-09 11:20:42 +00:00
Monty Taylor
0cd5075466
Pin to npm4 until npm 5.6.0 comes out
npm5 has an issue uploading tarballs it didn't build. For now, install
npm4 and use that to upload our tarballs.

The fix for https://github.com/npm/npm/issues/16723 should be in 5.6.0
which should allow us to revert this.

Change-Id: I8f633ee2119218799706b47ba3b471dcb04c1253
2017-12-08 17:20:51 -06:00
Zuul
23c9fde929 Merge "Revert "Add sphinx_python variable to sphinx role and job"" 2017-12-08 12:31:32 +00:00
Jens Harbott (frickler)
68ded2251f Revert "Add sphinx_python variable to sphinx role and job"
This reverts commit 1f8c4737651b1d64eda909e1a6641673f60ebbeb.

Change-Id: I98e7414337d58af0f4819163ee0e88702a52c051
2017-12-08 11:10:45 +00:00
Zuul
d50fa12a51 Merge "Add sphinx_python variable to sphinx role and job" 2017-12-08 00:11:39 +00:00
Zuul
ab60637718 Merge "Fix npm-run-test" 2017-12-07 16:33:27 +00:00
Zuul
e8967612be Merge "bindep: skip package install when not needed" 2017-12-07 16:33:27 +00:00
Zuul
21a2b108b3 Merge "bindep: fix install bindep_command fact name" 2017-12-07 16:33:26 +00:00
Zuul
f928de5a0d Merge "revoke-sudo: only revoke when zuul is sudoer" 2017-12-07 16:33:25 +00:00
Monty Taylor
2862cddb6a
Source nvm before using it
According to https://github.com/creationix/nvm#install-script,
we should, you know, actually source the script rather than just cloning
the repo and then ignoring it.

Change-Id: I046af8d38cf7288fcb3fac74cd6839d4e006ca90
2017-12-06 11:29:11 -06:00
David Moreau Simard
1f8c473765
Add sphinx_python variable to sphinx role and job
Some projects, such as Zuul itself, need to use python3 to build docs.

This adds a new parameter, sphinx_python, which defines the version
of python to build documentation with. The default is python2 and
projects can choose python3 on a need basis.

Remove the comment about the need for a leading comment, since there
is now a real need for the leading comment anyway.

Replace a few direct references to {{ ansible_user_dir }}/.venv that
should have been using zuul_work_virtualenv.

Update sphinx invocation to source the virtualenv activate first.
Some sphinx modules, such as sphinxcontrib.programoutput, attempt to
execute programs and only invoking sphinx-build with the relative
path causes the paths to not be set up properly.

Co-Authored-By: David Moreau Simard <dmsimard@redhat.com>
Change-Id: Ie5e2c93f88465f4aa746827ff88a585dbaa44fd5
2017-12-05 21:10:15 -05:00
Tristan Cacqueray
53ff9f166e fetch-tox-output: remove --ignore-missing-args
This is no longer needed since the task errors are now ignored.
Moreover this option doesn't work on el7.

Change-Id: Ic6b32999eef2fa26d63ed5d34b3b05b3ade60b32
2017-12-05 09:20:49 +00:00
Tristan Cacqueray
62015d9e15 tox: make install_sibling_packages pip import optional
This enable using the tox role without pip module installed on the host

Change-Id: Ic2c9e75ed967faf93267b6b9c58fdf3b02837ea9
2017-12-05 09:13:01 +00:00
Tristan Cacqueray
ea5c3d08da bindep: skip package install when not needed
When bindep test doesn't output package, there is no need
to run the package task.

Change-Id: I6f13e060a6d9657d3dec4f43a0b9c2dbff2d362d
2017-12-05 09:09:34 +00:00
Tristan Cacqueray
184f036fd8 bindep: fix install bindep_command fact name
After installing bindep, the role should set the bindep_command fact
instead of bindep_found_command. Otherwise the role fail later:

Task: Define bindep_command fact
The error was: 'bindep_command' is undefined

Change-Id: I11a21c9678c75b7887a8a58c25925dc87e7c9315
2017-12-05 08:52:31 +00:00
Tristan Cacqueray
892dc6a095 revoke-sudo: only revoke when zuul is sudoer
This change makes unittests jobs usable on read-only environment.

Change-Id: I36cfe7e5849687dbed510396a825dc0ec45542b3
2017-12-05 08:36:26 +00:00
Sean McGinnis
a34de95685 Fix variable name in npm-publish job
Variable work_root was accidentally set as work_dir.

Change-Id: Ib0650027565308e04235b9b7ca48b6470f1f1a0c
2017-12-04 15:35:25 -06:00
Monty Taylor
28bf5c0dcc
Ensure ChangeLog exists in install-if-python
We install projects if they are python in the sphinx jobs because
autodoc needs to be able to import the package. Also, for pbr projects,
many projects have links to ChangeLog in the docs, which is generated by
pbr on install and sdist. Installing with pip install --no-deps does not
cause the ChangeLog to be generated, so trigger it via an sdist build
first.

Change-Id: I4528cc7f322d1fc753eb5bc259f70c47ee9d1d80
2017-12-01 13:36:05 -06:00
David Moreau Simard
a4e9e12241
Remove the CentOS Plus mirror from the configured mirrors
CentOS Plus is not being mirrorred by openstack-infra and contains
mostly alternative kernel images which are not expected to be used in
jobs.

Let's reconsider if there is a use case for it in the future but in
the meantime let's remove it.

Change-Id: I9681b3b92c913f45791f610d845a94ccf3a16e48
2017-12-01 11:26:13 -05:00
Ian Wienand
80f5a7eeb4 Convert back to zuul.projects
Zuul has transitioned zuul.projects to a dict that mirrors
zuul._projects.  Convert users back to zuul.projects so we can remove
the underscore version in zuul.

Change-Id: I1408a15d4da274f3fc911f03d5572f4818bc4ebb
Depends-On: I3c011f72933e98ccbf8badf0e9197c8659766c51
2017-12-01 15:02:20 +11:00
Zuul
fda87ae875 Merge "Remove tox-siblings role stub" 2017-11-30 19:30:11 +00:00
David Moreau Simard
5d751bab12
mirror-workspace-git-repos: Pass a dict, not a list containing a dict
This was a regression introduced in the commit
b1b2537de78ec80dc970001bf5aa0b61cfa93ccb

Change-Id: Ib5ed93491631f22472a3eb4913d29d20a78592cd
2017-11-30 10:29:34 -05:00
Ian Wienand
b1b2537de7
Move to dictionary list of projects zuul._projects (take 2)
This is a revert of commit f28cc33bc3bc7f8a30d868964ec7712c26cef8f9
which was a revert of the original attempt at moving to zuul._projects
(df987735730e712ea3bc57f99d3c96cb561c43a9).

The issue was that the order of projects in "Determine local HEAD" was
not the same as "Update remote repository state" and we ended up
checking out the wrong thing.

To avoid this, build the projects and their current heads on the
exectutor into a separate dict, and as we iterate the projects look up
the correct values in that.

Change-Id: I0d9a66435714efe42fb653aafc6cb71b08ca6b8a
2017-11-29 17:56:08 -06:00