Since I6c5a962260741dcf6f89da9a33b96372a719b7b0 dib has had a
standardised method for ensuring consistency of tracing and error
detection. Bring the tracing for these elements up to that standard,
but maintain the status-quo of flags such as "-e" and "pipefail" by
adding ignore flags where appropriate (we can update these separately
to avoid breakage)
Other minor changes are alphabetical-ordering in the element-deps
files and permissions on prepare-node script
With this, "tox -edib" passes
Change-Id: Ibba1dadb9e819f94294c9d583b83ff698252f93f
We are still working on the caching story for dnf-based builds
(i.e. Fedora 22). There are a couple of options which we will work
through on the linked page. We will sort this out before we move the
devstack job (yet to be created, because the nodes aren't there yet)
out of experimental.
In the mean time, disable the caching in these elements so we can get
Fedora 22 image builds (currently they're in a big looping failure
[1]).
[1] http://nodepool.openstack.org/image.log
Change-Id: I3a435889fc5109d7365240068047aac98abc605e
Rather then explicitly forcing a security context for unbound, we
should restore the default selinux contexts defined by the OS.
Change-Id: I4bb21dcbbcbcff6a5458ebf9478f58d95ad7240a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This commit creates a venv for installing os-testr which will enable
all test jobs to have access to the subunit2html utility which has
been moved to live inside the os-testr package instead of as a slave
script.
Change-Id: I2050b54eb2def10438764f3eeb55ecf9caa874dc
This help when reading log files from nodepool. Otherwise we see the
following in the log files:
[1;31mWarning: Config file /etc/puppet/hiera.yaml not found, using
Hiera defaults[0m
Change-Id: I3a865e5107e2749ed44c144539af49e311e0125f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The nodepool base element really needs the puppet element, but we didn't
express that in the element deps. That makes a later reorg of elements
more difficult than it needs to be, because our consumption element list
is expressing internal dependency chains.
Change-Id: Ib2296bed1d7f289e7aa2720a38555275de1b6402
The nodepool-base element relies on a couple environement variables.
Describe NODEPOOL_SCRIPTDIR and NODEPOOL_STATIC_NAMESERVER in the
README.rst file.
Change-Id: I56f2aab095a0504e19598d7296d072e7a51b07c2
Invoke bindep from the bindir for its virtualenv in /usr/bindep-env,
and guard its exit code with true since its normal behavior is to
return zero only when all needed packages are already present.
While here, replace calls to tempfile with mktemp since the former
is not readily available on CentOS 6.x.
Change-Id: I926e73f305b1b54f6855cc4acf3407e91b88213a
While git show is being used to build up a list of candidate
packages to cache, it can cause the cache hook to abort if it
returns nonzero because of a nonexistent bindep list file. Guard
that with a fallback to true so that the loop can continue
normally.
Change-Id: Ic3b773ad2e8efa2c450d37a0115355c0e95da853
The version of git on CentOS 6.x is too ancient to support git
branch --list which is necessary to disambiguate -r with a name
pattern. Instead just list all remote branches and filter with grep
like a good Neanderthal.
Change-Id: I4e673e86acb70c205d159e22b5864cf60ae831df
A quirk of diskimage-builder is that no two elements you compose are
allowed to have parts with the same filename. Rename cache-bindep's
install.d/50-download-pkgs so that it doesn't conflict with the one
of the same name in the cache-devstack element.
Change-Id: I5709147fc5bd4c06fb1bba8ea0e068c0eda7d010
Passing a source-repositories ref of "*" signals fetching all heads
and tags rather than just a specific refname. This is desirable for
the warm cache on our job workers, since jobs may run against
arbitrary branches (or possibly even tags). The point is to have a
reasonably complete mirror of each repo, rather than a fairly
minimal clone.
Change-Id: I624ed1e259e007d4246afb45c3a0560598bfbe3f
Depends-On: I4562c9689a8d235ebe09b2f7178aa5890dbc85f1
We intend to use bindep to tell us what distro packages should be
installed at job run-time, so need to similarly identify those in
such a way that they can be cached on our worker images.
Install the latest bindep release from PyPI into /usr/bindep-env
(consistent with the way we provide virtualenvs for access to
zuul-cloner and libraries used by our log uploads to swift).
Concatenate bindep lists from all branches of every hosted Git repo
along with our fallback list in project-config. Run bindep to render
them into a platform-specific manifest of package names and feed
that to the relevant package management tool for the platform to
download this set into the package cache.
Note that similar tooling is not applied to snapshot nodes, since we
only intend to run jobs which use bindep on diskimage-builder
created nodes.
Change-Id: I0c408fee35dfe9b4f700c51565bfaae0a3d03beb
The build for node-devstack will fail because of missing scripts unless
nodepool-base is executed before, so add this as dependency.
Change-Id: Iabc06a542529500ad248ae067beed9639d24e506
Cleanup every use of `` for subshells in the nodepool and tools
directory , replacing them with $(), and finally making the scripts
consistent.
Change-Id: I2b05cd20f9c9a30ab88f8db235aa81da93b1fad3
Some packages may depend on python-setuptools, which is not
installed and cannot be reinstalled on CentOS 6.x once yum has
erased them, so use --skip-broken to avoid aborting. Also on this
platform --downloadonly causes yum to return nonzero even when it
succeeds, so ignore its exit code.
Change-Id: Iaada39ae81e1e47fe9d0bedba80fd19e4e0e6f38
When we're building nodepool images on top of minimal elements, there
will be no cloud-init, and therefore no need to disable cloud-init
datasources. In fact, trying to do so will be an error.
Change-Id: I98887c43566e07f2be9d2dc5fae6538078c7348e
If someone is building an image locally and don't want to do anything
special with nodepool scripts, they should not need to set an env var.
The env var obviously still needs to be honored for nodepool operation
and for override.
Change-Id: I08076a8eafe7019e715b1b46633e8f603031a1ea
The --downloadonly option to yum is provided by the
yum-plugin-downloadonly package. This is merely a virtual package
satisfied by yum itself in newer releases, but an optional package
in older ones such as CentOS/RHEL 6.x. Install it just to be sure it
will work, since we use this to pre-cache RPMs on nodepool images.
Change-Id: I9e2e1605f3721c410180aa46a81b7b731d08503a
On CentOS 6.x the rpm executable is in /bin instead of /usr/bin, but
the cache-devstack element ends up running yum if it wants to
download RPM packages anyway. Look for yum instead of rpm as an
indication of which packages to install.
Commit fa18656 fixed this in cache_devstack.py for snapshot images,
but missed that it was affecting diskimage-builder elements as well.
Change-Id: I4b76a48564cd5d703d79cfda88bbbfe97216a70a
Since DIB creates a chroot but doesn't actually chroot in an
extra-data script, look for /etc/redhat-release relative to the
temporary chroot base path. Also given that the centos element
leaves DIB_RELEASE exported as am empty string in the calling
environment rather than unset, place it first in the conditional
list.
Change-Id: I0a51f88ff5ea71b6aa6e6406b656cefb0e135e88
Change fd9c80208c00021a49df6907973bbbb3fa242069 inadvertently
removed the subprocess import from 55-cache-devstack-repos even
though it was used in a function defined within that element, unlike
in its snapshot prep script counterpart. Readd it so that this
element works again.
Change-Id: I954a1e7e99af96b08e0a59e99e513902731fd773
In the DevStack caching script/element for Nodepool images, use
run_local instead of subprocess.check_output in the _find_images
function. The latter wasn't introduced until Python 2.7 and so won't
work on CentOS 6. The script called from this function returns
quickly and doesn't benefit from non-blocking I/O anyway.
Change-Id: I3129f1f5b3fece321ae132ea1a52b0e156e58365
The centos element (which is CentOS 6.x specific) doesn't provide
DIB_RELEASE so we have to identify it another way.
Change-Id: Ibf3b25b99f03c6077538dec6c3770dc0f690e3f5
Rather than deleting cloud-init, which is going to take longer, just
disable ec2 metadata service. This will be a no-op on rackspace, which
already does this.
Change-Id: I5e8baee50800f7aae474288a914333c21466855a
Once the migration has been run manually we need to use
subunit2sql>=0.4.0 because that is the version of the DB schema running
on the infra db.
This reverts commit aa87c57ca7fe95f9785719acb6b626f12171e240.
Change-Id: I747083bc15ecda7e1926a87c2adcdaf29e565e23
Subunit2sql 0.4.0 was recently released which included a schema change
however the infra db migration takes too long and can't be automated.
(yet?) Until we run the migration we can't use 0.4.0 on any of the
tooling. This caps the version to be less than 0.4.0
Change-Id: I08f113fa904fa962e8f2dc04187ff44e764de47e
This reverts commit 1f84dbb2ce2ae7ed50401299db499adb6e836ec0.
Reverting because this runs a significant portion of devstack during
image builds installing packages necessary to build the wheels. Instead
we should be building wheels and hosting them on our mirror. Then image
builds and jobs themselves can both use this mirror to either cache
wheels or install wheels.
Conflicts:
nodepool/scripts/cache_devstack.py
Change-Id: I70ebc57076845995c539a42e4f87e241569211b4
There are some cases where packages (or its dependencies) are missing.
Cache them on a best-effort basis, and ignore any that are missing.
They will be downloaded and installed if and when they are needed.
Change-Id: I5cd7f5f35b4fef6002c6cc3d66543a33a6972ef8
Update manual dib mounting instructions to match changes made in the
tools/mount-image.sh script (I0deba1907f390f0d308ae6c8ed5448afbbf7688a)
and add a reference to the script itself in the documentation.
Change-Id: Ic72bf05f21cecb8bdff36956f3635c2f504c4dff
This commit adds calling tools/build_wheels.sh to the cache devstack
step in the nodepool scripts. This will generate and cache the wheels
before the devstack run, which is a length process normally involving C
compilations.
Change-Id: I9064d7d9b9b879a05d665c3b002a631fcc953f52
Use GIT_BASE variable to be consistent with:
nodepool/elements/openstack-repos/extra-data.d/50-create-repo-list
Change-Id: I2ae933aef62cb1f9009483406bb15405efaf4001
pkgs-to-install is written to /tmp/pkgs-to-install not
~/tmp/pkgs-to-install/.
From nodepool logs:
2015-02-26 14:24:11,465 INFO nodepool.image.build.devstack-trusty:
/tmp/in_target.d/install.d/50-download-pkgs:
line 28: /root/tmp/pkgs-to-install: No such file or directory
Change-Id: If367be96bbc2761faa9bd4b4bb173a3ecbe9b406
Allow supplying filename and paths with '**' recursive glob matches
to zuul-swift-upload. Since bash (or shell etc) will expand on any
filenames provided to the program this needs to be used in quotes.
Usage example:
./zuul_swift_upload.py my_results.txt '**/sdist/*.zip' output.log
The hierarchy is always flattened meaning the supplied list is
placed in the topmost generated index.html. Sub-folders still keep
their hierarchy.
Change-Id: I9ba04f7e46b579dcf3f8ad0bd188f41fa5dbcad9