wheel-copy.sh is the wrong place to do the index of the package
directory. Move it to wheel-index.sh where we are walking the final
Some things have become evident when generating the indexes requiring
some larger changes.
Firstly, the indexer script needs python3 on the host. Since we're
still building CentOS 7 wheels, we need to install Python 3 from EPEL
Secondly, because part of the PEP503 index page is the file hash,
reading all the files back over AFS is quite slow. It's also quite
slow having ansible loop a task each time, which all adds up to job
Instead, make the indexes on the local disk before we copy the results
to AFS. This requires copying both scripts to the host for execution
(rather than relying on "script:" ) so the wheel-copy.sh script can
While we are there, a small refactor on the wheel-indexer.py to use
os.walk() (which makes it easier to have this as a stand-alone
recursive script later, if something changes). Also update the output
to use <ul><li> for the filenames, so it looks a little better on the
From the ansible manual
"The difference between single quotes and double quotes is that in
double quotes you can use escapes:"
This turns out to be important when you're splitting based on \n ...
What I didn't notice is that on disk the projects are in a/ b/ c/ type
subdirectories, while from the mirror side we paper-over this with
mod_rewrite rules so the projects all look like they're at top-level.
This should make the script descend into each directory correctly to
generate the index.
PEP503  defines the simple respository API format that we should
follow for our mirrors publishing .whl files.
This loops the output directories and creates a index.html for each
project with the required tags.
We do not really have gate testing for this, so to avoid destroying
the whole mirror we write out a temporary index file first, and ignore
errors. When this works, we will revert to index.html and do a single
manual removal of the temporary indexes.
We saw what looks very much like the index generation racing, with
"mv" unable to close the index file ...
Move to a linear strategy so that each host finishes each task before
moving on. Then add run_once to the index generation so only one of
the hosts in a python2/3 situation creates the index file.
We only need the three wheel-X scripts from jenkins/scripts and need
them only for the playbook and its roles. Move the scripts in the
roles directory and copy them only if needed - using the scripts module.
Remove copying of complete jenkins/script for this playbook using the
role legacy-copy-project-config-scripts, it's not needed anymore.
Pass the wheel keytab across and aklog with it.
Also, split the build role into a build and a copy so that we only have
the token when we need it, and not while we're running the wheel builds
themselves. And split release from build so that we can release the
mirror once for a consistent state across python2 and python3.