Use new locations following OpenDev changes.
Node: This changes the on-disk repo path to /opt/git/openstack/devstack
Change-Id: I7042913fefa64dcec4044779dbeb13f86daea858
We've patched stackviz to work under python3 properly but we are still
pulling an old tarball for stackviz that was built last year. The legacy
job that built the file at this location seems to have been removed.
Switch to the new dist/ location which appears to be correct based on
tarball file sizes.
Someone that understands stackviz better than me should confirm this new
locations is the correct one.
Change-Id: If659a6f1fb50d288afed75e3f4975f7a4d140d35
We are noticing on nb03 that the curl get of this file fails; but it
starts working when you use https. Switch to it.
Change-Id: I5fef4281547758f484cd8a93326b0cfeb790762a
The need for the distribution was removed in I249f21a98fea3b963b7ffb8e3d0fce02cc540d46
so we can remove the code around that was determining it but not
actually making use of it anywhere anymore.
Change-Id: I572943bbc0c2665a885b7a80cdb2d1817c61bc3c
Without the newline this falls into the old bash "read" gotcha trick
where the last line will be skipped silently. This means the client
isn't actualy downloaded.
dib-lint added a check in I23e293b957cd4f008611656cf9166391b1b537a2
and I tested it would have caught this.
Change-Id: Idbae2037f275fad91b87e1cf5ee80663c1f05ddc
Cache zanata cli tools during build, and up the prep-zanata role to
look for the cached version in preference to dowloading.
Change-Id: If70f3386d0350c94dd1548e7f3c60a1d02914824
git:// is not a secure protocol. It's also harder to do apache rewrite
tricks and whatnot when we eventually flatten namespaces. Change
occurances of git://git.openstack.org to https://git.openstack.org.
Change-Id: Ic54f1b84346e37a76baa87deadbb540662889bca
fix a "bug" in that the flake8 configuration in tox.ini was exclusively
selecting H231 as the only error it would report, so it was missing the
errors in the python modules (such as submit_log_processor_jobs). Due to
this being the case for a long time (since 2004) limit the more thorough
linting to the roles/ and playbooks/ directories where we'll be adding
ansible plugins/modules/etc. Also, lint in jenkins/script and nodepool.
Fix problems found.
We can lint everything with pep8 once the zuul v2 scripts are removed,
not worth patching them right now.
Change-Id: I479f010643cf3b67c183d763510f07a33400d38b
Co-Authored-By: Jesse Keating <omgjlk@us.ibm.com>
It seems this optimization is no longer needed. Once we address:
https://bugs.launchpad.net/testrepository/+bug/1416512
We could consider adding it back. However, this also means we stop
hitting logstash.openstack.org each time we build an image. So, we
should maybe consider caching that data some place over having our
image builds do it.
Change-Id: I16b56a13028c0f228d21dde423db3f70f07f4e1a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Add python3 support for these functions. Tested with a python3 based
dib in a Xenial environment. Add __future__ imports so we don't
regress on these bits.
Change-Id: Iddca88cb6c9fbff6383c3597eb81133948b3b420
For whatever reason, the source-repositories caching element is
running in the very early extra-data phase of disk-image-builder.
cache-devstack is instructing source-repositories to put the cached
urls into /home/jenkins/cache/files (55-cache-devstack-repos makes up
the list of images to download from devstack; the other two are just
pre-configured URLS).
This causes a problem when trying to add the jenkins user later in
jenkins-slave/install.d/20-jenkins-slave. Since source repositories
has already created /home/jenkins, useradd won't do it again and does
not copy in any of the skeleton files.
This moves the extra-data phase caching into /opt/cache, which is
probably a more logical place for it anyway. To maintain the status
quo, after we create the jenkins user, symlink ~jenkins/cache/files to
this new location.
This was probably going wrong with puppet too. I'm guessing it just
ignored the existing homedir when creating the user. I have tested
this by building trusty and stopping the build before and after the
useradd calls in 20-jenkins-slave. "/home" was empty, the useradd
creates the user, group & directory, the symlink is made correctly and
/opt/cache/files appears populated as usual.
Change-Id: Ibca6867f29b257a5110cb5522a5cca3a97fa9377
In an effor to remove the stackviz element from nodepool, we can now
use the periodic job and download the package from tarballs.o.o. This
mean, we can remove npm development libraries from our base images in
a follow commit.
Move periodic-package-stackviz-element to the periodic pipeline.
Change-Id: I798c98f9ed91e0e7b2a43bd9e8d0837ce223cb8b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
We gate on python3.5 for subunit2sql, we should allow our DIB install
to use it.
Change-Id: Ie411a6317c83687db14039820845576c7838c89d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
After talking with clarkb, we decided it might be time to also stop
caching packages for devstack. Like the cache-bindep patch before, we
have a robust mirror infra in place right now, and believe we are
ready to start using it more.
Change-Id: I249f21a98fea3b963b7ffb8e3d0fce02cc540d46
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This commit adds Gentoo support to elements needed to build a Gentoo
nodepool image. The previous version of this commit had the default
section of the pkg-map in the wrong area, specifically in the following
file.
nodepool/elements/infra-package-needs/pkg-map
Change-Id: Ic686c325bc06564585a2e3ac50cadd7556612333
Our images have failed to build for the last 3 days, reverting until we can debug.
This reverts commit 4c515e20732a64d3be1768d776f69506cd36dd29.
Change-Id: I2e653bcd8a30a85ea46a9861bdf9f95413a76f64
This commit adds Gentoo support to elements needed to build a Gentoo
nodepool image.
Change-Id: I2ceeb915748a11d8e729069566e722a3fe30ba99
Signed-off-by: Matthew Thode <mthode@mthode.org>
In [1] some new safe_sudo checks were added, which lead to errors
in the project-config elements. We add tags to ignore these errors
for now.
[1] I161a5aea1d29dcdc7236f70d372c53246ec73749
Change-Id: If3ff10b8fbf20a8327895f439b955f9941818668
When cleaning up comments from lines of the form:
foo bar # this is a comment
You want to remove remove everything after the first comment character
not after the last comment character. This is because:
foo bar # this is # a comment
Should get everything after bar removed not everything after is.
Unfortunately when you use str.rfind() you remove everything after is
not everything after bar. Switch to regular trusty find to fix this.
Change-Id: I78aa6b51b5be03bd3b8ce7885415442171218977
This script is unnecessarily complex for what it is doing, which is
essentially pre-seeding the source-repositories cache for devstack.
As we can see from I41e81d6bac98875eecde2376e0865784626e11a8 it's very
confusing having large parts of the source-repositories script
copy-pasted as a separate element and has led several of us down the
wrong path.
Strip this script back to the simple thing it is doing, which is
checking out/updating the source-repositories devstack cache.
I have tested this by building an image with a warm cache and with a
cold cache, in both cases the checkout was found and the list of
images to cache in 55-cache-devstack-repos was found.
Change-Id: I6c686312de102cbe438585e26bf6986e06b6f41c
In another confusing addendum to
I41e81d6bac98875eecde2376e0865784626e11a8 (which was already a
confusing addendum to Ieb6a6e9f55bd93f63c3d0a71828c276c2d02e1b9), we
have decided that the refspec used to fetch here is not sufficient to
clear out remote branches everything when updating.
"+refs/heads/*:refs/heads/*" says to replicate everything from the
remote refs/heads into our local refs/heads, but leaves out
refs/remotes/*
The upshot of this is that I41e81d6bac98875eecde2376e0865784626e11a8
will remove the local branches (refs/heads/stable/icehouse, say) but
not remove the remote branches (refs/remotes/origin/stable/icehouse).
The devstack caching script keeps picking up these remote branches,
checking them out, and consequently trying to download old images.
*Nothing* ever removes these branches. In the main dib cache git
update, we also have --prune, but our refspec there is even more
limited (+master:master). This explains why these branches never seem
to die.
Note, an even better mirror would be "+refs/*:refs/*" (in fact, if you
do git clone --mirror, this is what the repo would be setup with to
fetch by default). However, this drags in "refs/changes/*" and all
sorts of other gerrit things. We don't really need them so we just
keep the limiting on.
Change-Id: Ia9c3ffdb2b5f72a45d629961338b415308d6dd21
In a rather confusing addendum to
Ieb6a6e9f55bd93f63c3d0a71828c276c2d02e1b9, we have actually mirrored a
version of the source-repo script from dib and munged it to cache
devstack early so we can use it to find the vm images to download and
cache.
However, we are not ensuring that we remove old branches in this
clone, which is leading to the problems of us picking up images from
old branches that don't exist any more.
Change-Id: I41e81d6bac98875eecde2376e0865784626e11a8
The release has happened, the migration worked, all's right with the
world.
This reverts commit 04eb36588d80dd77a9c692f53470b7d727567604.
Change-Id: I28a3e9cf3b420a6ccca20c52c4e67adc3f5710c5
This commit adds a temporary version cap on subunit2sql to be < 1.0.0.
The 1.0.0 release includes a very large database migration which will
be slow to execute. The python DB api from >=v1.0.0 will not work with
a database that doesn't have the updated schema. So while the migration
is running let's cap the version we install to prevent everything from
breaking while the migration is running. (which might take days)
Change-Id: Iab89beb5c7aba8b744a62f5063e513b72cab0ec2
Turns out we can't do this without dirtying our images by
preinstalling all the libraries/headers our requirements want to
link against. We should revisit the wheel mirror solution which has
already been written as a saner solution.
This reverts commit 1503ef327817bd42cbf888e773f5639e660169a7.
Change-Id: I69ea73b6eb4b620f1ac77467ee3784ec3c813ee2
This installs everything in upper-constraints in a throw away venv,
which has the knock on effect of fully populating the wheel cache for
root with wheels for everything. This hopefully speeds up all devstack
runs quite a bit.
Change-Id: I429f353c33d892f76552c83d34ac1329ed18f97f
Since I6c5a962260741dcf6f89da9a33b96372a719b7b0 dib has had a
standardised method for ensuring consistency of tracing and error
detection. Bring the tracing for these elements up to that standard,
but maintain the status-quo of flags such as "-e" and "pipefail" by
adding ignore flags where appropriate (we can update these separately
to avoid breakage)
Other minor changes are alphabetical-ordering in the element-deps
files and permissions on prepare-node script
With this, "tox -edib" passes
Change-Id: Ibba1dadb9e819f94294c9d583b83ff698252f93f
We are still working on the caching story for dnf-based builds
(i.e. Fedora 22). There are a couple of options which we will work
through on the linked page. We will sort this out before we move the
devstack job (yet to be created, because the nodes aren't there yet)
out of experimental.
In the mean time, disable the caching in these elements so we can get
Fedora 22 image builds (currently they're in a big looping failure
[1]).
[1] http://nodepool.openstack.org/image.log
Change-Id: I3a435889fc5109d7365240068047aac98abc605e
Some packages may depend on python-setuptools, which is not
installed and cannot be reinstalled on CentOS 6.x once yum has
erased them, so use --skip-broken to avoid aborting. Also on this
platform --downloadonly causes yum to return nonzero even when it
succeeds, so ignore its exit code.
Change-Id: Iaada39ae81e1e47fe9d0bedba80fd19e4e0e6f38
The --downloadonly option to yum is provided by the
yum-plugin-downloadonly package. This is merely a virtual package
satisfied by yum itself in newer releases, but an optional package
in older ones such as CentOS/RHEL 6.x. Install it just to be sure it
will work, since we use this to pre-cache RPMs on nodepool images.
Change-Id: I9e2e1605f3721c410180aa46a81b7b731d08503a
On CentOS 6.x the rpm executable is in /bin instead of /usr/bin, but
the cache-devstack element ends up running yum if it wants to
download RPM packages anyway. Look for yum instead of rpm as an
indication of which packages to install.
Commit fa18656 fixed this in cache_devstack.py for snapshot images,
but missed that it was affecting diskimage-builder elements as well.
Change-Id: I4b76a48564cd5d703d79cfda88bbbfe97216a70a
Since DIB creates a chroot but doesn't actually chroot in an
extra-data script, look for /etc/redhat-release relative to the
temporary chroot base path. Also given that the centos element
leaves DIB_RELEASE exported as am empty string in the calling
environment rather than unset, place it first in the conditional
list.
Change-Id: I0a51f88ff5ea71b6aa6e6406b656cefb0e135e88
Change fd9c80208c00021a49df6907973bbbb3fa242069 inadvertently
removed the subprocess import from 55-cache-devstack-repos even
though it was used in a function defined within that element, unlike
in its snapshot prep script counterpart. Readd it so that this
element works again.
Change-Id: I954a1e7e99af96b08e0a59e99e513902731fd773
In the DevStack caching script/element for Nodepool images, use
run_local instead of subprocess.check_output in the _find_images
function. The latter wasn't introduced until Python 2.7 and so won't
work on CentOS 6. The script called from this function returns
quickly and doesn't benefit from non-blocking I/O anyway.
Change-Id: I3129f1f5b3fece321ae132ea1a52b0e156e58365
The centos element (which is CentOS 6.x specific) doesn't provide
DIB_RELEASE so we have to identify it another way.
Change-Id: Ibf3b25b99f03c6077538dec6c3770dc0f690e3f5
Once the migration has been run manually we need to use
subunit2sql>=0.4.0 because that is the version of the DB schema running
on the infra db.
This reverts commit aa87c57ca7fe95f9785719acb6b626f12171e240.
Change-Id: I747083bc15ecda7e1926a87c2adcdaf29e565e23
Subunit2sql 0.4.0 was recently released which included a schema change
however the infra db migration takes too long and can't be automated.
(yet?) Until we run the migration we can't use 0.4.0 on any of the
tooling. This caps the version to be less than 0.4.0
Change-Id: I08f113fa904fa962e8f2dc04187ff44e764de47e
This reverts commit 1f84dbb2ce2ae7ed50401299db499adb6e836ec0.
Reverting because this runs a significant portion of devstack during
image builds installing packages necessary to build the wheels. Instead
we should be building wheels and hosting them on our mirror. Then image
builds and jobs themselves can both use this mirror to either cache
wheels or install wheels.
Conflicts:
nodepool/scripts/cache_devstack.py
Change-Id: I70ebc57076845995c539a42e4f87e241569211b4
There are some cases where packages (or its dependencies) are missing.
Cache them on a best-effort basis, and ignore any that are missing.
They will be downloaded and installed if and when they are needed.
Change-Id: I5cd7f5f35b4fef6002c6cc3d66543a33a6972ef8
This commit adds calling tools/build_wheels.sh to the cache devstack
step in the nodepool scripts. This will generate and cache the wheels
before the devstack run, which is a length process normally involving C
compilations.
Change-Id: I9064d7d9b9b879a05d665c3b002a631fcc953f52
Use GIT_BASE variable to be consistent with:
nodepool/elements/openstack-repos/extra-data.d/50-create-repo-list
Change-Id: I2ae933aef62cb1f9009483406bb15405efaf4001
pkgs-to-install is written to /tmp/pkgs-to-install not
~/tmp/pkgs-to-install/.
From nodepool logs:
2015-02-26 14:24:11,465 INFO nodepool.image.build.devstack-trusty:
/tmp/in_target.d/install.d/50-download-pkgs:
line 28: /root/tmp/pkgs-to-install: No such file or directory
Change-Id: If367be96bbc2761faa9bd4b4bb173a3ecbe9b406
When we build our images we attempt to checkout all of the valid
devstack branches so that we can list all of the images necessary to be
cached for each branch. Unfortunately this devstack repo is owned by
root during dib and snapshot image builds. This means we must use sudo
when updating the repo (either changing branches or updating the
current branch).
Change-Id: I8cd09cfed4d586648dcbd34fa04bfc030c31ee45
