227 Commits

Author SHA1 Message Date
Ramy Asselin
da4f071588 Ignore missing packages
There are some cases where packages (or its dependencies) are missing.
Cache them on a best-effort basis, and ignore any that are missing.
They will be downloaded and installed if and when they are needed.

Change-Id: I5cd7f5f35b4fef6002c6cc3d66543a33a6972ef8
2015-03-12 14:53:34 -07:00
Matthew Treinish
5e9046594f Log stdout from build_wheels in cache_devstack
This commit adds a print statement with the stdout from the run_local()
call which executes build_wheels.sh in devstack. Right now when the
script is executed all it does is print that it's running but we are
unable to debug anything that could potentially go wrong during the
execution of the script without the output from build_wheels.

Change-Id: I7433f054127a42fa73598db219c6e351efc98fb7
2015-03-12 16:22:41 -04:00
Jenkins
971251414b Merge "Build wheel house during nodepool devstack caching" 2015-03-11 19:37:13 +00:00
Jenkins
bae9ecaccc Merge "Support custom static nameserver during build" 2015-03-11 19:02:23 +00:00
Ramy Asselin
0d5c572d52 Update dib script usage
Update manual dib mounting instructions to match changes made in the
tools/mount-image.sh script (I0deba1907f390f0d308ae6c8ed5448afbbf7688a)
and add a reference to the script itself in the documentation.

Change-Id: Ic72bf05f21cecb8bdff36956f3635c2f504c4dff
2015-03-10 15:58:32 -07:00
Matthew Treinish
1f84dbb2ce Build wheel house during nodepool devstack caching
This commit adds calling tools/build_wheels.sh to the cache devstack
step in the nodepool scripts. This will generate and cache the wheels
before the devstack run, which is a length process normally involving C
compilations.

Change-Id: I9064d7d9b9b879a05d665c3b002a631fcc953f52
2015-03-06 18:27:51 -05:00
Ramy Asselin
2433522020 DIB project-config scripts requires DIB_RELEASE
Also add a helpful hint to run in offline mode to speed things up
while debugging.

Change-Id: I4727ff8f0199b4b190339409f8417566286d944e
2015-03-05 17:53:29 -08:00
Jenkins
1c716e0449 Merge "zuul-swift-upload support recursive globs" 2015-03-06 01:19:24 +00:00
Ramy Asselin
4a5e37d3ed Use GIT_BASE environment variable
Use GIT_BASE variable to be consistent with:
nodepool/elements/openstack-repos/extra-data.d/50-create-repo-list

Change-Id: I2ae933aef62cb1f9009483406bb15405efaf4001
2015-03-04 17:28:25 -08:00
Joe Gordon
fb74bd0d38 Fix pkgs-to-install path
pkgs-to-install is written to /tmp/pkgs-to-install not
~/tmp/pkgs-to-install/.

From nodepool logs:
2015-02-26 14:24:11,465 INFO nodepool.image.build.devstack-trusty:
/tmp/in_target.d/install.d/50-download-pkgs:
 line 28: /root/tmp/pkgs-to-install: No such file or directory

Change-Id: If367be96bbc2761faa9bd4b4bb173a3ecbe9b406
2015-03-04 15:16:15 -08:00
Joshua Hesketh
3691476925 zuul-swift-upload support recursive globs
Allow supplying filename and paths with '**' recursive glob matches
to zuul-swift-upload. Since bash (or shell etc) will expand on any
filenames provided to the program this needs to be used in quotes.

Usage example:
./zuul_swift_upload.py my_results.txt '**/sdist/*.zip' output.log

The hierarchy is always flattened meaning the supplied list is
placed in the topmost generated index.html. Sub-folders still keep
their hierarchy.

Change-Id: I9ba04f7e46b579dcf3f8ad0bd188f41fa5dbcad9
2015-03-04 09:33:17 +11:00
Clark Boylan
12dc33e30c Use sudo when manipulating git for image list
When we build our images we attempt to checkout all of the valid
devstack branches so that we can list all of the images necessary to be
cached for each branch. Unfortunately this devstack repo is owned by
root during dib and snapshot image builds. This means we must use sudo
when updating the repo (either changing branches or updating the
current branch).

Change-Id: I8cd09cfed4d586648dcbd34fa04bfc030c31ee45
2015-02-26 18:19:23 -08:00
Gregory Haynes
f504d47200 Support custom static nameserver during build
We statically set 8.8.8.8 for a dns resolver in prepare-node. Allow
setting a custom nameserver.

Change-Id: I888c4bcefdf36ad7064d66b6a1e75d996b93b375
2015-02-26 17:28:41 -08:00
Jenkins
f934d6e6fe Merge "Respect NODEPOOL_SSH_KEY in prepare-node for dib" 2015-02-26 19:59:18 +00:00
Jenkins
edd917e974 Merge "Remove wasteful tripleo workspace-cache cloning" 2015-02-24 20:51:07 +00:00
Jenkins
652eeb24de Merge "Use sudo when gem installing during image build" 2015-02-24 20:32:01 +00:00
Clark Boylan
1b9eea5da0 Use sudo when gem installing during image build
We need root permissions in order to gem install rspec during image
builds.

Change-Id: Ida6081d5b26fc168ec37f5d529ba862e1ae05c9a
2015-02-24 10:52:39 -08:00
Jeremy Stanley
a2551dfef6 Run restrict-memory element later in image build
Since grub isn't installed by DIB until finalise.d/51 we need to
make sure the restrict-memory element happens after that, so it can
find and alter the grub configs accordingly.

Also make it apply the updated configuration, similar to how
nodepool/scripts/restrict_memory.sh does it.

Change-Id: I854f3bd1850594811cc8957f7a9263c33dfe6826
2015-02-24 18:52:06 +00:00
Dan Prince
ccb783fb6d Remove wasteful tripleo workspace-cache cloning
There doesn't appear to be anything in TripleO that relies
on these clones. Also, this is a bit confusing since people
may expect that things copied into ~workspace-cache would
eventually end up at /opt/git/ (this is not the case).

The correct way to add projects to a job is to export a
PROJECTS variable like devstack-gate does.

Change-Id: I0a96855ed9a38bcade6e506b37621b6b7b348b49
2015-02-19 13:36:23 -08:00
Gregory Haynes
20891096e0 Respect NODEPOOL_SSH_KEY in prepare-node for dib
We dont respect this parameter when building dib images.

Change-Id: I403b59f150d11bfa01875a89de777023c15d2e71
2015-02-19 10:43:39 -08:00
Jenkins
c04e186b7d Merge "prepare_node.sh line lengths are too long" 2015-02-16 15:53:37 +00:00
Clark Boylan
ad623a7e86 Use sudo when editing /etc/pip.conf
Unlike before where the jenkins user was editing files that it owns in
its homedir the /etc/pip.conf editing needs root permissions since that
file is owned by root.

Change-Id: I485a08cb243cc0abf4c848e6061606b4320a59cb
2015-02-12 16:47:05 -08:00
Jenkins
8ce84a99ac Merge "Edit global pip.conf instead of per-user pip.conf" 2015-02-12 23:38:47 +00:00
Ramy Asselin
a6fbdd9ac4 prepare_node.sh line lengths are too long
These are are too long to be properly viewed in github.
Also, by putting each puppet argument on a separate line
will help visual clarity and future maintenance.

Change-Id: I4b04d54ef57fd10660eae0b217433880551e86dd
2015-02-12 10:46:27 -08:00
Jenkins
fca3b13721 Merge "set -x in cache-testrepository install" 2015-02-10 15:09:25 +00:00
Jenkins
ee7ad8b491 Merge "Add puppet tripleo check tests" 2015-02-10 15:05:15 +00:00
Gregory Haynes
bc3046bf01 set -x in cache-testrepository install
Were not set -x'ing here. This makes debugging failures near impossible.

Change-Id: Ib459c2897c191e10103efd3c3109c319dcc50723
2015-02-09 20:48:39 -08:00
Clark Boylan
4c4b50c1b7 Fix rspec installs during image builds
This change is adapted from Ief25f224fbe15530a503f7c53acef8759df281b4
to fix rspec installs when puppet runs during old school nodepool
snapshot builds.

Note we are not updating DIB and snapshot together because we do not yet
use DIB builds for the bare-* $THIN=false builds.

Change-Id: I4d0ab20d437da0c779bddc286664f9def98484e1
2015-02-09 10:33:28 -08:00
Dan Prince
209832cc03 Add puppet tripleo check tests
This patch adds check test coverage on tripleo-puppet-elements
and puppet-tripleo.

As part of this change we also drop the normal tripleo check
tests from tripleo-puppet-elements because it isn't needed
and is a waste of resources.  Only the puppet tests would
actually make use of this code.

Change-Id: I079e7b3bcbb36828390a034970092e3e88de4acf
2015-02-09 12:49:32 -05:00
Matthew Treinish
c6ff8a8c55 Preseed with a number of samples from db instead of aggregate
Because of the testrepository bug 1416512 using the aggregate stream
generated by sql2subunit the timing data would be ignored by the testr
scheduler. The long term fix is to address this in testrepository
eventually. But, in the short to get any utility out of the preseed
data this commit switches the preseed to not use the aggregate view
stream and instead take the 10 most recent runs and use those as
the preseed data. While this data isn't as good for performing the
optimization, but it will at least use the data.

Change-Id: Ib0e815ca1966f5b5806fb257b91ac7a25d4db18d
2015-02-04 15:01:51 -05:00
Jenkins
e12bd38597 Merge "Stop using py3k-precise nodes" 2015-02-03 21:02:54 +00:00
Jeremy Stanley
90e3c200bd Stop using py3k-precise nodes
These are no longer being used to run jobs, so clean up references
to them.

Change-Id: Iac7ffde66a2d5d1a1361ed9bcadb9144e034f10f
Depends-On: I40d5d09f28ef53583d239d2e852e9c50b5962cf8
2015-01-30 23:47:30 +00:00
James E. Blair
311aff2fce Fix centos6 image updates on hpcloud
The "yum repolist" command was failing with what I think was the
same error that the update command was designed to fix.  I am
unsure how or if that ever worked correctly, or what might have
changed to cause it to start impacting that command.

At any rate, specifying "epel*" as the repo name to disable will
cause the update command to work regardless of whether it is
installed.

Change-Id: I1d978ade76ec78e8895e00bbf64e5e40529223f8
2015-01-30 15:11:30 -08:00
Matthew Treinish
cfaf1f9daf Set proper ownership on tempest test repository
When the testrepository directory is created by calling the preseed
script the permissions are different from the rest of the tempest
repo. When the dir is rsync'd by devstack-gate it errors out with
permission denied on the .testrepository dir. This commit fixes this
by setting the ownership to be the same as the rest of the cached
repo.

Change-Id: Ie12f3f0d824401f2ed53a77f8f307ea0f178854e
2015-01-23 16:11:26 -05:00
Joe Gordon
58cbd080bd Cache get-pip.py in dib element cache-devstack
We cache pip in cache_devstack but not in dib element cache-devstack

Fetching get-pip.py across the internet breaks surprisingly often in hp
cloud, so precache this.

Change-Id: I86d8e54b5a3129bedf7e472d4463009c139f0656
Related-Bug: #1413034
2015-01-23 12:05:19 -08:00
Jenkins
8dec7c3655 Merge "Fix DIB image cache path in DIB element" 2015-01-16 08:14:08 +00:00
Clark Boylan
f89b832398 Install argparse and requests to swift venv
The zuul swift log uploader script needs argparse on python2.6 and
requests on all python versions to work properly. Install these two
packages to the virtualenv configured for this script at image build
time.

Note that `pip install argparse` on python >= 2.7 is not an error
it just reports that argparse is installed as part of the python
installation and moves on.

Change-Id: Ie79b59e1c687cce2b435c9704959b02911f94778
2015-01-15 10:31:15 -08:00
Clark Boylan
b8932165ba Fix DIB image cache path in DIB element
The DIB cache-devstack element's early-source-repos script was using a
hardcoded cache path rather than rooting at $DIB_IMAGE_CACHE. This meant
that for this portion of the DIB builds they insisted on writing to the
current users homedir and not the overridden cache path. Root the cache
path properly against $DIB_IMAGE_CACHE so that an overridden path is
properly used.

Change-Id: I9c84de7497bd9259b5660e78e84d2adc86f9d61c
2015-01-13 14:29:53 -08:00
Clark Boylan
87bba6b03f Use correct path for tempest in testr dib script
When seeding tempest with testrepository subunit data use the correct
path to the tempest repo in the imgae build. Because we moved to
install.d we ues a path relative to the chroot and not relative to the
mount path.

Change-Id: I8e296595945c8cfe91a88f69ae46fe33c53df2db
2015-01-12 22:08:22 -08:00
Clark Boylan
3d6303f3fc Set the executable bit on all dib element scripts
DIB needs its scripts to be executable or they are not run. Update
permissions on two files that were missing the executable bit.

Change-Id: Ie172e50de57b5168264964644cd28530f023542a
2015-01-12 16:15:27 -08:00
Jeremy Stanley
b70a27d6c4 Use PyMySQL and venv path when caching subunit2sql
Also rename the element to one which runs in the context of the
diskimage chroot rather than in the root context, and make it
executable.

Change-Id: I093d02706f2965e071e2a76ab1131ac29016d03c
2015-01-12 23:34:45 +00:00
Jeremy Stanley
05e6bfaa10 Correct tempest clone location in node prep
Change-Id: I4208637457bbf1e7562f358913ff71b9f3b0732d
2015-01-12 17:25:06 +00:00
Jenkins
1d113cbf24 Merge "Add nodepool script to preseed testrepository from subunit2sql" 2015-01-12 14:14:45 +00:00
Matthew Treinish
8fdd1ce943 Add nodepool script to preseed testrepository from subunit2sql
This commit adds a new nodepool script to use subunit2sql to preseed
testrepository with timing data to use for scheduler optimization.
This uses aggregate data from all the tests stored in the subunit2sql
db from all the test runs to generate a subunit stream with test
execute times which equal the rolling average stored in the tests
table.

This also adds a call out to the new prepare_tempest_testrepository
script in order to pre-seed tempest's testrepository with data from
the subunit2sql DB. This will enable the testr scheduler to perform
some worker balance optimization based on the average run_times from
all the previous gate runs.

Change-Id: I04404a55baf4da2745760246b8606ed3db52c03a
2015-01-07 12:25:06 -05:00
Joshua Hesketh
8bc108140e Set up venv for zuul-swift-logs
Set up a venv for zuul-swift-logs and install its dependencies
(python-magic) there instead of as system packages.

For DIB use 90 as uploading logs may be considered more important
than installing zuul.

Change-Id: Id0fc01729853e65c81cdd50e4ffa4d0d6de00ae9
2015-01-07 10:38:59 +11:00
Jenkins
46ea9288c5 Merge "Move to using package-installs.yaml" 2015-01-05 16:08:45 +00:00
Jenkins
b4f8a55a78 Merge "Only guard yum update against epel if enabled" 2015-01-05 15:58:05 +00:00
Monty Taylor
9001f4a399 Edit global pip.conf instead of per-user pip.conf
Now that there is a global pip.conf, just make the setting there.

Change-Id: I0af9d1cc6907e3977f4e7052070c72821bc1bf3a
2014-12-26 10:42:12 -06:00
James Polley
aeadb73010 Always use sudo -H when pip installing
Recent versions of pip will create ~/.cache if it doesn't already exist.

If this happens while running "sudo pip" the resulting dir will be owned
by root and 700 - which breaks anything else on the system that wants to
use ~/.cache

I *think* this finds all instances of "sudo pip" in this repo and fixes
them to use -H. This should mean they always run in the right ~

Change-Id: I47ddb3b591df6ac2100f09b38c9b8a03cb1ba1ff
closes-bug: #1405732
2014-12-26 10:41:58 +01:00
Mate Lakat
c7c3d68468 XenServer: Enable injection of XVA and ISO urls
This will enable the flexible configuration of the XVA and the URL
through nodepool, thus it's not needed to push an infra change for
changing those values. Also if you do not have those values configured,
sensible defaults are provided.

Change-Id: Ie2be50f842b4220d4e7f9d8e830d3df3262a6353
2014-12-18 18:05:58 +01:00