In [1] some new safe_sudo checks were added, which lead to errors
in the project-config elements. We add tags to ignore these errors
for now.
[1] I161a5aea1d29dcdc7236f70d372c53246ec73749
Change-Id: If3ff10b8fbf20a8327895f439b955f9941818668
When cleaning up comments from lines of the form:
foo bar # this is a comment
You want to remove remove everything after the first comment character
not after the last comment character. This is because:
foo bar # this is # a comment
Should get everything after bar removed not everything after is.
Unfortunately when you use str.rfind() you remove everything after is
not everything after bar. Switch to regular trusty find to fix this.
Change-Id: I78aa6b51b5be03bd3b8ce7885415442171218977
This script is unnecessarily complex for what it is doing, which is
essentially pre-seeding the source-repositories cache for devstack.
As we can see from I41e81d6bac98875eecde2376e0865784626e11a8 it's very
confusing having large parts of the source-repositories script
copy-pasted as a separate element and has led several of us down the
wrong path.
Strip this script back to the simple thing it is doing, which is
checking out/updating the source-repositories devstack cache.
I have tested this by building an image with a warm cache and with a
cold cache, in both cases the checkout was found and the list of
images to cache in 55-cache-devstack-repos was found.
Change-Id: I6c686312de102cbe438585e26bf6986e06b6f41c
In another confusing addendum to
I41e81d6bac98875eecde2376e0865784626e11a8 (which was already a
confusing addendum to Ieb6a6e9f55bd93f63c3d0a71828c276c2d02e1b9), we
have decided that the refspec used to fetch here is not sufficient to
clear out remote branches everything when updating.
"+refs/heads/*:refs/heads/*" says to replicate everything from the
remote refs/heads into our local refs/heads, but leaves out
refs/remotes/*
The upshot of this is that I41e81d6bac98875eecde2376e0865784626e11a8
will remove the local branches (refs/heads/stable/icehouse, say) but
not remove the remote branches (refs/remotes/origin/stable/icehouse).
The devstack caching script keeps picking up these remote branches,
checking them out, and consequently trying to download old images.
*Nothing* ever removes these branches. In the main dib cache git
update, we also have --prune, but our refspec there is even more
limited (+master:master). This explains why these branches never seem
to die.
Note, an even better mirror would be "+refs/*:refs/*" (in fact, if you
do git clone --mirror, this is what the repo would be setup with to
fetch by default). However, this drags in "refs/changes/*" and all
sorts of other gerrit things. We don't really need them so we just
keep the limiting on.
Change-Id: Ia9c3ffdb2b5f72a45d629961338b415308d6dd21
In a rather confusing addendum to
Ieb6a6e9f55bd93f63c3d0a71828c276c2d02e1b9, we have actually mirrored a
version of the source-repo script from dib and munged it to cache
devstack early so we can use it to find the vm images to download and
cache.
However, we are not ensuring that we remove old branches in this
clone, which is leading to the problems of us picking up images from
old branches that don't exist any more.
Change-Id: I41e81d6bac98875eecde2376e0865784626e11a8
The release has happened, the migration worked, all's right with the
world.
This reverts commit 04eb36588d80dd77a9c692f53470b7d727567604.
Change-Id: I28a3e9cf3b420a6ccca20c52c4e67adc3f5710c5
This commit adds a temporary version cap on subunit2sql to be < 1.0.0.
The 1.0.0 release includes a very large database migration which will
be slow to execute. The python DB api from >=v1.0.0 will not work with
a database that doesn't have the updated schema. So while the migration
is running let's cap the version we install to prevent everything from
breaking while the migration is running. (which might take days)
Change-Id: Iab89beb5c7aba8b744a62f5063e513b72cab0ec2
Turns out we can't do this without dirtying our images by
preinstalling all the libraries/headers our requirements want to
link against. We should revisit the wheel mirror solution which has
already been written as a saner solution.
This reverts commit 1503ef327817bd42cbf888e773f5639e660169a7.
Change-Id: I69ea73b6eb4b620f1ac77467ee3784ec3c813ee2
This installs everything in upper-constraints in a throw away venv,
which has the knock on effect of fully populating the wheel cache for
root with wheels for everything. This hopefully speeds up all devstack
runs quite a bit.
Change-Id: I429f353c33d892f76552c83d34ac1329ed18f97f
Since I6c5a962260741dcf6f89da9a33b96372a719b7b0 dib has had a
standardised method for ensuring consistency of tracing and error
detection. Bring the tracing for these elements up to that standard,
but maintain the status-quo of flags such as "-e" and "pipefail" by
adding ignore flags where appropriate (we can update these separately
to avoid breakage)
Other minor changes are alphabetical-ordering in the element-deps
files and permissions on prepare-node script
With this, "tox -edib" passes
Change-Id: Ibba1dadb9e819f94294c9d583b83ff698252f93f
We are still working on the caching story for dnf-based builds
(i.e. Fedora 22). There are a couple of options which we will work
through on the linked page. We will sort this out before we move the
devstack job (yet to be created, because the nodes aren't there yet)
out of experimental.
In the mean time, disable the caching in these elements so we can get
Fedora 22 image builds (currently they're in a big looping failure
[1]).
[1] http://nodepool.openstack.org/image.log
Change-Id: I3a435889fc5109d7365240068047aac98abc605e
Some packages may depend on python-setuptools, which is not
installed and cannot be reinstalled on CentOS 6.x once yum has
erased them, so use --skip-broken to avoid aborting. Also on this
platform --downloadonly causes yum to return nonzero even when it
succeeds, so ignore its exit code.
Change-Id: Iaada39ae81e1e47fe9d0bedba80fd19e4e0e6f38
The --downloadonly option to yum is provided by the
yum-plugin-downloadonly package. This is merely a virtual package
satisfied by yum itself in newer releases, but an optional package
in older ones such as CentOS/RHEL 6.x. Install it just to be sure it
will work, since we use this to pre-cache RPMs on nodepool images.
Change-Id: I9e2e1605f3721c410180aa46a81b7b731d08503a
On CentOS 6.x the rpm executable is in /bin instead of /usr/bin, but
the cache-devstack element ends up running yum if it wants to
download RPM packages anyway. Look for yum instead of rpm as an
indication of which packages to install.
Commit fa18656 fixed this in cache_devstack.py for snapshot images,
but missed that it was affecting diskimage-builder elements as well.
Change-Id: I4b76a48564cd5d703d79cfda88bbbfe97216a70a
Since DIB creates a chroot but doesn't actually chroot in an
extra-data script, look for /etc/redhat-release relative to the
temporary chroot base path. Also given that the centos element
leaves DIB_RELEASE exported as am empty string in the calling
environment rather than unset, place it first in the conditional
list.
Change-Id: I0a51f88ff5ea71b6aa6e6406b656cefb0e135e88
Change fd9c80208c00021a49df6907973bbbb3fa242069 inadvertently
removed the subprocess import from 55-cache-devstack-repos even
though it was used in a function defined within that element, unlike
in its snapshot prep script counterpart. Readd it so that this
element works again.
Change-Id: I954a1e7e99af96b08e0a59e99e513902731fd773
In the DevStack caching script/element for Nodepool images, use
run_local instead of subprocess.check_output in the _find_images
function. The latter wasn't introduced until Python 2.7 and so won't
work on CentOS 6. The script called from this function returns
quickly and doesn't benefit from non-blocking I/O anyway.
Change-Id: I3129f1f5b3fece321ae132ea1a52b0e156e58365
The centos element (which is CentOS 6.x specific) doesn't provide
DIB_RELEASE so we have to identify it another way.
Change-Id: Ibf3b25b99f03c6077538dec6c3770dc0f690e3f5
Once the migration has been run manually we need to use
subunit2sql>=0.4.0 because that is the version of the DB schema running
on the infra db.
This reverts commit aa87c57ca7fe95f9785719acb6b626f12171e240.
Change-Id: I747083bc15ecda7e1926a87c2adcdaf29e565e23
Subunit2sql 0.4.0 was recently released which included a schema change
however the infra db migration takes too long and can't be automated.
(yet?) Until we run the migration we can't use 0.4.0 on any of the
tooling. This caps the version to be less than 0.4.0
Change-Id: I08f113fa904fa962e8f2dc04187ff44e764de47e
This reverts commit 1f84dbb2ce2ae7ed50401299db499adb6e836ec0.
Reverting because this runs a significant portion of devstack during
image builds installing packages necessary to build the wheels. Instead
we should be building wheels and hosting them on our mirror. Then image
builds and jobs themselves can both use this mirror to either cache
wheels or install wheels.
Conflicts:
nodepool/scripts/cache_devstack.py
Change-Id: I70ebc57076845995c539a42e4f87e241569211b4
There are some cases where packages (or its dependencies) are missing.
Cache them on a best-effort basis, and ignore any that are missing.
They will be downloaded and installed if and when they are needed.
Change-Id: I5cd7f5f35b4fef6002c6cc3d66543a33a6972ef8
This commit adds calling tools/build_wheels.sh to the cache devstack
step in the nodepool scripts. This will generate and cache the wheels
before the devstack run, which is a length process normally involving C
compilations.
Change-Id: I9064d7d9b9b879a05d665c3b002a631fcc953f52
Use GIT_BASE variable to be consistent with:
nodepool/elements/openstack-repos/extra-data.d/50-create-repo-list
Change-Id: I2ae933aef62cb1f9009483406bb15405efaf4001
pkgs-to-install is written to /tmp/pkgs-to-install not
~/tmp/pkgs-to-install/.
From nodepool logs:
2015-02-26 14:24:11,465 INFO nodepool.image.build.devstack-trusty:
/tmp/in_target.d/install.d/50-download-pkgs:
line 28: /root/tmp/pkgs-to-install: No such file or directory
Change-Id: If367be96bbc2761faa9bd4b4bb173a3ecbe9b406
When we build our images we attempt to checkout all of the valid
devstack branches so that we can list all of the images necessary to be
cached for each branch. Unfortunately this devstack repo is owned by
root during dib and snapshot image builds. This means we must use sudo
when updating the repo (either changing branches or updating the
current branch).
Change-Id: I8cd09cfed4d586648dcbd34fa04bfc030c31ee45
Because of the testrepository bug 1416512 using the aggregate stream
generated by sql2subunit the timing data would be ignored by the testr
scheduler. The long term fix is to address this in testrepository
eventually. But, in the short to get any utility out of the preseed
data this commit switches the preseed to not use the aggregate view
stream and instead take the 10 most recent runs and use those as
the preseed data. While this data isn't as good for performing the
optimization, but it will at least use the data.
Change-Id: Ib0e815ca1966f5b5806fb257b91ac7a25d4db18d
When the testrepository directory is created by calling the preseed
script the permissions are different from the rest of the tempest
repo. When the dir is rsync'd by devstack-gate it errors out with
permission denied on the .testrepository dir. This commit fixes this
by setting the ownership to be the same as the rest of the cached
repo.
Change-Id: Ie12f3f0d824401f2ed53a77f8f307ea0f178854e
We cache pip in cache_devstack but not in dib element cache-devstack
Fetching get-pip.py across the internet breaks surprisingly often in hp
cloud, so precache this.
Change-Id: I86d8e54b5a3129bedf7e472d4463009c139f0656
Related-Bug: #1413034
The DIB cache-devstack element's early-source-repos script was using a
hardcoded cache path rather than rooting at $DIB_IMAGE_CACHE. This meant
that for this portion of the DIB builds they insisted on writing to the
current users homedir and not the overridden cache path. Root the cache
path properly against $DIB_IMAGE_CACHE so that an overridden path is
properly used.
Change-Id: I9c84de7497bd9259b5660e78e84d2adc86f9d61c
When seeding tempest with testrepository subunit data use the correct
path to the tempest repo in the imgae build. Because we moved to
install.d we ues a path relative to the chroot and not relative to the
mount path.
Change-Id: I8e296595945c8cfe91a88f69ae46fe33c53df2db
DIB needs its scripts to be executable or they are not run. Update
permissions on two files that were missing the executable bit.
Change-Id: Ie172e50de57b5168264964644cd28530f023542a
Also rename the element to one which runs in the context of the
diskimage chroot rather than in the root context, and make it
executable.
Change-Id: I093d02706f2965e071e2a76ab1131ac29016d03c
This commit adds a new nodepool script to use subunit2sql to preseed
testrepository with timing data to use for scheduler optimization.
This uses aggregate data from all the tests stored in the subunit2sql
db from all the test runs to generate a subunit stream with test
execute times which equal the rolling average stored in the tests
table.
This also adds a call out to the new prepare_tempest_testrepository
script in order to pre-seed tempest's testrepository with data from
the subunit2sql DB. This will enable the testr scheduler to perform
some worker balance optimization based on the average run_times from
all the previous gate runs.
Change-Id: I04404a55baf4da2745760246b8606ed3db52c03a
A short time ago, devstack change the name of its lists of debian
packages from apts to debs. In order to properly pre-cache, we should
read these.
Change-Id: Id784e2876a2cbbede650b7e101ff90f7d74ec367
Add some filtering to 55-cache-devsatck-repos to handle centos7, which
actually gets setup with a DISTRO_NAME of rhel7 in devstack
Update the readme file to highlight the options for building other
images.
"DISTRO=centos7 ./tools/build-image.sh" creates a workable centos7
qcow image with this change and a d-i-b that includes
Ie24033468b78587ea87188ee1b843b26895798ff
Change-Id: Ida38cba0a303ab77432d5d354f1952e00ebfa00e
This repo was created from filter branching the openstack-infra/
config repo. This process brought a lot of cruft with it in the
form of directories that we no longer need. This patch removes
that cruft so we begin with a tidier repo.
Change-Id: Ibffad1b11c0c5f84eedfb0365369f60c4961a0f3
