After talking with clarkb, we decided it might be time to also stop
caching packages for devstack. Like the cache-bindep patch before, we
have a robust mirror infra in place right now, and believe we are
ready to start using it more.
Change-Id: I249f21a98fea3b963b7ffb8e3d0fce02cc540d46
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This commit adds Gentoo support to elements needed to build a Gentoo
nodepool image. The previous version of this commit had the default
section of the pkg-map in the wrong area, specifically in the following
file.
nodepool/elements/infra-package-needs/pkg-map
Change-Id: Ic686c325bc06564585a2e3ac50cadd7556612333
Our images have failed to build for the last 3 days, reverting until we can debug.
This reverts commit 4c515e20732a64d3be1768d776f69506cd36dd29.
Change-Id: I2e653bcd8a30a85ea46a9861bdf9f95413a76f64
This commit adds Gentoo support to elements needed to build a Gentoo
nodepool image.
Change-Id: I2ceeb915748a11d8e729069566e722a3fe30ba99
Signed-off-by: Matthew Thode <mthode@mthode.org>
In [1] some new safe_sudo checks were added, which lead to errors
in the project-config elements. We add tags to ignore these errors
for now.
[1] I161a5aea1d29dcdc7236f70d372c53246ec73749
Change-Id: If3ff10b8fbf20a8327895f439b955f9941818668
When cleaning up comments from lines of the form:
foo bar # this is a comment
You want to remove remove everything after the first comment character
not after the last comment character. This is because:
foo bar # this is # a comment
Should get everything after bar removed not everything after is.
Unfortunately when you use str.rfind() you remove everything after is
not everything after bar. Switch to regular trusty find to fix this.
Change-Id: I78aa6b51b5be03bd3b8ce7885415442171218977
This script is unnecessarily complex for what it is doing, which is
essentially pre-seeding the source-repositories cache for devstack.
As we can see from I41e81d6bac98875eecde2376e0865784626e11a8 it's very
confusing having large parts of the source-repositories script
copy-pasted as a separate element and has led several of us down the
wrong path.
Strip this script back to the simple thing it is doing, which is
checking out/updating the source-repositories devstack cache.
I have tested this by building an image with a warm cache and with a
cold cache, in both cases the checkout was found and the list of
images to cache in 55-cache-devstack-repos was found.
Change-Id: I6c686312de102cbe438585e26bf6986e06b6f41c
In another confusing addendum to
I41e81d6bac98875eecde2376e0865784626e11a8 (which was already a
confusing addendum to Ieb6a6e9f55bd93f63c3d0a71828c276c2d02e1b9), we
have decided that the refspec used to fetch here is not sufficient to
clear out remote branches everything when updating.
"+refs/heads/*:refs/heads/*" says to replicate everything from the
remote refs/heads into our local refs/heads, but leaves out
refs/remotes/*
The upshot of this is that I41e81d6bac98875eecde2376e0865784626e11a8
will remove the local branches (refs/heads/stable/icehouse, say) but
not remove the remote branches (refs/remotes/origin/stable/icehouse).
The devstack caching script keeps picking up these remote branches,
checking them out, and consequently trying to download old images.
*Nothing* ever removes these branches. In the main dib cache git
update, we also have --prune, but our refspec there is even more
limited (+master:master). This explains why these branches never seem
to die.
Note, an even better mirror would be "+refs/*:refs/*" (in fact, if you
do git clone --mirror, this is what the repo would be setup with to
fetch by default). However, this drags in "refs/changes/*" and all
sorts of other gerrit things. We don't really need them so we just
keep the limiting on.
Change-Id: Ia9c3ffdb2b5f72a45d629961338b415308d6dd21
In a rather confusing addendum to
Ieb6a6e9f55bd93f63c3d0a71828c276c2d02e1b9, we have actually mirrored a
version of the source-repo script from dib and munged it to cache
devstack early so we can use it to find the vm images to download and
cache.
However, we are not ensuring that we remove old branches in this
clone, which is leading to the problems of us picking up images from
old branches that don't exist any more.
Change-Id: I41e81d6bac98875eecde2376e0865784626e11a8
Since I6c5a962260741dcf6f89da9a33b96372a719b7b0 dib has had a
standardised method for ensuring consistency of tracing and error
detection. Bring the tracing for these elements up to that standard,
but maintain the status-quo of flags such as "-e" and "pipefail" by
adding ignore flags where appropriate (we can update these separately
to avoid breakage)
Other minor changes are alphabetical-ordering in the element-deps
files and permissions on prepare-node script
With this, "tox -edib" passes
Change-Id: Ibba1dadb9e819f94294c9d583b83ff698252f93f
On CentOS 6.x the rpm executable is in /bin instead of /usr/bin, but
the cache-devstack element ends up running yum if it wants to
download RPM packages anyway. Look for yum instead of rpm as an
indication of which packages to install.
Commit fa18656 fixed this in cache_devstack.py for snapshot images,
but missed that it was affecting diskimage-builder elements as well.
Change-Id: I4b76a48564cd5d703d79cfda88bbbfe97216a70a
Since DIB creates a chroot but doesn't actually chroot in an
extra-data script, look for /etc/redhat-release relative to the
temporary chroot base path. Also given that the centos element
leaves DIB_RELEASE exported as am empty string in the calling
environment rather than unset, place it first in the conditional
list.
Change-Id: I0a51f88ff5ea71b6aa6e6406b656cefb0e135e88
Change fd9c80208c00021a49df6907973bbbb3fa242069 inadvertently
removed the subprocess import from 55-cache-devstack-repos even
though it was used in a function defined within that element, unlike
in its snapshot prep script counterpart. Readd it so that this
element works again.
Change-Id: I954a1e7e99af96b08e0a59e99e513902731fd773
In the DevStack caching script/element for Nodepool images, use
run_local instead of subprocess.check_output in the _find_images
function. The latter wasn't introduced until Python 2.7 and so won't
work on CentOS 6. The script called from this function returns
quickly and doesn't benefit from non-blocking I/O anyway.
Change-Id: I3129f1f5b3fece321ae132ea1a52b0e156e58365
The centos element (which is CentOS 6.x specific) doesn't provide
DIB_RELEASE so we have to identify it another way.
Change-Id: Ibf3b25b99f03c6077538dec6c3770dc0f690e3f5
Use GIT_BASE variable to be consistent with:
nodepool/elements/openstack-repos/extra-data.d/50-create-repo-list
Change-Id: I2ae933aef62cb1f9009483406bb15405efaf4001
When we build our images we attempt to checkout all of the valid
devstack branches so that we can list all of the images necessary to be
cached for each branch. Unfortunately this devstack repo is owned by
root during dib and snapshot image builds. This means we must use sudo
when updating the repo (either changing branches or updating the
current branch).
Change-Id: I8cd09cfed4d586648dcbd34fa04bfc030c31ee45
The DIB cache-devstack element's early-source-repos script was using a
hardcoded cache path rather than rooting at $DIB_IMAGE_CACHE. This meant
that for this portion of the DIB builds they insisted on writing to the
current users homedir and not the overridden cache path. Root the cache
path properly against $DIB_IMAGE_CACHE so that an overridden path is
properly used.
Change-Id: I9c84de7497bd9259b5660e78e84d2adc86f9d61c
Also rename the element to one which runs in the context of the
diskimage chroot rather than in the root context, and make it
executable.
Change-Id: I093d02706f2965e071e2a76ab1131ac29016d03c
This commit adds a new nodepool script to use subunit2sql to preseed
testrepository with timing data to use for scheduler optimization.
This uses aggregate data from all the tests stored in the subunit2sql
db from all the test runs to generate a subunit stream with test
execute times which equal the rolling average stored in the tests
table.
This also adds a call out to the new prepare_tempest_testrepository
script in order to pre-seed tempest's testrepository with data from
the subunit2sql DB. This will enable the testr scheduler to perform
some worker balance optimization based on the average run_times from
all the previous gate runs.
Change-Id: I04404a55baf4da2745760246b8606ed3db52c03a
A short time ago, devstack change the name of its lists of debian
packages from apts to debs. In order to properly pre-cache, we should
read these.
Change-Id: Id784e2876a2cbbede650b7e101ff90f7d74ec367
Add some filtering to 55-cache-devsatck-repos to handle centos7, which
actually gets setup with a DISTRO_NAME of rhel7 in devstack
Update the readme file to highlight the options for building other
images.
"DISTRO=centos7 ./tools/build-image.sh" creates a workable centos7
qcow image with this change and a d-i-b that includes
Ie24033468b78587ea87188ee1b843b26895798ff
Change-Id: Ida38cba0a303ab77432d5d354f1952e00ebfa00e
This repo was created from filter branching the openstack-infra/
config repo. This process brought a lot of cruft with it in the
form of directories that we no longer need. This patch removes
that cruft so we begin with a tidier repo.
Change-Id: Ibffad1b11c0c5f84eedfb0365369f60c4961a0f3
