95 Commits

Author SHA1 Message Date
James E. Blair
8eb0732a00 Remove unused fix_pip.sh
This doesn't appear to be used any more.

Change-Id: I094c42e6eaaaaa779efa7a4978c2fd84557ccf79
2016-01-19 18:03:40 -08:00
Andreas Jaeger
1db085d5ec Wrap some overly long lines
To cut down on noise when running bashate, wrap some overlong lines.

Change-Id: I23dcffbae2539d3d6b11e5527f3d4cbf08a18064
2015-12-23 13:56:53 +01:00
James E. Blair
381d507b04 Increase memory restriction to 9023M
On OVH and Bluebox, the memory layout is such that there is still
significant (~900M) memory above the 8192M address.  Increase the
limit to encompass that, which will bring these providers up to
approximately 8G, while increasing hpcloud (which is the actual
target of this restriction) to about 8.5G.

Change-Id: I5c121be55cadad13ad5807968f33b492f9b1e215
2015-12-15 10:32:06 -08:00
Jenkins
cdd2dfda55 Merge "Remove now duplicate sync;sleep 5" 2015-12-07 13:10:20 +00:00
Andreas Jaeger
d8170deb07 Remove CentOS6 from nodepool scripts
With CentOS6 usage removed, remove special handling for it from nodepool
scripts.

Change-Id: Ife5ec9ade201dcd8f5969087dd2fc88be12dbebd
2015-12-04 22:11:22 +01:00
Andreas Jaeger
ca687cd8b4 Remove centos6 handling from cache_devstack
With CentOS6 support gone, we can remove special treatment from
nodepool scripts.

Change-Id: Ia50de3450c0f21c27740d2d6294d0f2451196e21
2015-12-04 21:52:49 +01:00
Monty Taylor
97c0e3b0b6 Remove now duplicate sync;sleep 5
Since https://review.openstack.org/#/c/196252/ merged, nodepool
injects a sync;sleep 5 after running the scripts, so having it in the
scripts is a double sleep.

Change-Id: Ie7a9315e6de67625eca15b45f1662e4c63f8bc10
2015-12-01 13:27:19 -06:00
Matthew Treinish
3ed5aa98c5 Revert "Set a temporary cap on subunit2sql before 1.0.0 release"
The release has happened, the migration worked, all's right with the
world.

This reverts commit 04eb36588d80dd77a9c692f53470b7d727567604.

Change-Id: I28a3e9cf3b420a6ccca20c52c4e67adc3f5710c5
2015-11-29 17:01:45 +00:00
Matthew Treinish
04eb36588d
Set a temporary cap on subunit2sql before 1.0.0 release
This commit adds a temporary version cap on subunit2sql to be < 1.0.0.
The 1.0.0 release includes a very large database migration which will
be slow to execute. The python DB api from >=v1.0.0 will not work with
a database that doesn't have the updated schema. So while the migration
is running let's cap the version we install to prevent everything from
breaking while the migration is running. (which might take days)

Change-Id: Iab89beb5c7aba8b744a62f5063e513b72cab0ec2
2015-11-23 15:49:58 -05:00
Cyril Roelandt
2554d974b8 Add python3-dev
The gate-neutron-dsvm-functional-py34 job needs Python.h, and it is
currently not available when running Python 3.

Change-Id: I00ce26864db682a4577dc5360519918007aa5499
Partial-Bug: #1500400
2015-10-22 15:08:28 +02:00
Matthew Treinish
76d164b1ba
Create an os-testr venv for running subunit2html
This commit creates a venv for installing os-testr which will enable
all test jobs to have access to the subunit2html utility which has
been moved to live inside the os-testr package instead of as a slave
script.

Change-Id: I2050b54eb2def10438764f3eeb55ecf9caa874dc
2015-08-14 12:37:22 -04:00
Paul Belanger
ebdd2553d0 Remove color output from puppet apply
This help when reading log files from nodepool. Otherwise we see the
following in the log files:

  [1;31mWarning: Config file /etc/puppet/hiera.yaml not found, using
  Hiera defaults[0m

Change-Id: I3a865e5107e2749ed44c144539af49e311e0125f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2015-08-06 15:38:35 -04:00
yanyanhu
da344ab915 Fix a typo in prepare_node_devstack.sh
Change-Id: I763b6efe57d37c1e2a1da299b9a2a0c32e0df32f
2015-08-04 02:36:07 -04:00
Jeremy Stanley
f954e8ccd4 Revert "Block installing of mock from wheels"
This reverts commit 71d10ea626f5a62d6b48133e19f469b19c9a7f14.

It was unnecessary once the offending wheel got removed from PyPI,
and this never worked anyway. It was preventing the ability to boot
new nodes, erroring thusly:

    ./configure_mirror.sh: line 25: /tmp/pip.conf: Permission denied

Deleting the images to which the change was applied returned
nodepool to being able to boot new nodes again.

Change-Id: I0841c6a5a26cf5be22e2d8fea861bdceb0393842
2015-07-12 01:54:12 +00:00
Monty Taylor
71d10ea626 Block installing of mock from wheels
This is needed until a new wheel of mock can be cut after the new pbr
release is cut.

Change-Id: I96a1794ee754f879514b34add3e36a4a42fd681b
2015-07-10 10:23:22 -04:00
Jeremy Stanley
62ad51e638 Remove managed stable/icehouse branch testing
Remove jobs which test stable/icehouse branches of repos tagged
release:managed as the branch has reached end-of-life and is being
removed from those repos.

Change-Id: I88a44cfa84597012af7da0bd22de02dc2349b1fa
2015-07-01 18:40:41 +00:00
Ian Wienand
877d93bb84 Remove cloud-init details
If we have booted with cloud-init, then this status directory is
populated.  Remove it so when we boot snapshot images, they behave as
if on a fresh system.

Change-Id: Idc9ce01290b659e3239d30be847221447a8e5e84
2015-06-18 14:05:50 +10:00
Ian Wienand
a9444cdad8 Remove mount of ephemeral disk
hpcloud has started sending metadata to cloud-init to mount ephemeral
disks.  This ends up writing a fstab entry for /dev/vdb.

---
$ curl http://169.254.169.254/2009-04-04/meta-data/block-device-mapping/ephemeral0
/dev/vdb
---

It's unclear why this just started happening, but it did.

devstack-gate later attempts to repartition this into swap & disk
space and mount it elsewhere.  So remove this mount from fstab -- it
shouldn't come back because the next thing we do here is tell
cloud-init to not use the metadata source.

Change-Id: I3787d0f7e5139e891686ffbb2970e65d09f112b1
2015-06-18 14:05:38 +10:00
Steve Kowalik
85a0f4f44f Switch to using $() for subshells, part 1
Cleanup every use of `` for subshells in the nodepool and tools
directory , replacing them with $(), and finally making the scripts
consistent.

Change-Id: I2b05cd20f9c9a30ab88f8db235aa81da93b1fad3
2015-06-04 15:30:55 +10:00
James E. Blair
8e5c538e5a Add allow_hosts to pydistutils.cfg
This should cause easy_install to only contact our pypi mirrors
when fetching packages.

https://pythonhosted.org/setuptools/easy_install.html#restricting-downloads-with-allow-hosts

Change-Id: I7f7847721c2e8c6299c7fe5ab20045e8bdbc5bd7
2015-06-02 10:50:03 -07:00
Jenkins
38ec65bd48 Merge "Deal with yum --downloadonly CentOS 6.x quirks" 2015-05-12 21:38:26 +00:00
Jeremy Stanley
3d5b04ed19 Deal with yum --downloadonly CentOS 6.x quirks
Some packages may depend on python-setuptools, which is not
installed and cannot be reinstalled on CentOS 6.x once yum has
erased them, so use --skip-broken to avoid aborting. Also on this
platform --downloadonly causes yum to return nonzero even when it
succeeds, so ignore its exit code.

Change-Id: Iaada39ae81e1e47fe9d0bedba80fd19e4e0e6f38
2015-05-12 15:25:01 +00:00
Clark Boylan
f4bb992006 Don't disable cloud-init if not installed part 2
During scripted snapshot image builds we remove python-setuptools on
rhel/centos 6. This uninstalls cloud-init on hpcloud centos6 builds
because cloud-init depends on setuptools. We don't actually need
cloud-init in hpcloud because they use dhcp and we have hard coded ssh
keys so just don't bother setting cloud-init config if the config dir in
/etc does not exist.

One could argue that we would possibly want cloud-init to regen host
keys for us, but centos' sshd service should do that for us if the host
keys do not exist.

Change-Id: I96621b0ab1574eb8db0f4394877d3c1fc8208576
2015-05-11 19:39:11 -07:00
Jeremy Stanley
0137c25b46 Install yum-plugin-downloadonly on nodes
The --downloadonly option to yum is provided by the
yum-plugin-downloadonly package. This is merely a virtual package
satisfied by yum itself in newer releases, but an optional package
in older ones such as CentOS/RHEL 6.x. Install it just to be sure it
will work, since we use this to pre-cache RPMs on nodepool images.

Change-Id: I9e2e1605f3721c410180aa46a81b7b731d08503a
2015-05-07 13:33:12 +00:00
Jeremy Stanley
fa18656f1c Look for yum instead of rpm when caching devstack
On CentOS 6.x the rpm executable is in /bin instead of /usr/bin, but
cache_devstack.py ends up running yum if it wants to download RPM
packages anyway. Look for yum instead of rpm as an indication of
which packages to install.

Change-Id: Iad79c0fcf66d2bd457195f007009c76f6e6aa2d2
2015-05-04 02:52:50 +00:00
Monty Taylor
c183c33710 Don't install redhat-lsb-core in prepare_node
We don't need it for anything other than detecting whether or not we
should update the sources.list file. Except that's obviously not
something we should do on systems with yum

Remove the call and defer it to install_puppet where it's also
installed.

Change-Id: Ie22aeac1e4c731e7ab61514cb4982c8c35c482e6
2015-05-03 12:55:55 -04:00
Jeremy Stanley
fd9c80208c Replace subprocess with run_local in _find_images
In the DevStack caching script/element for Nodepool images, use
run_local instead of subprocess.check_output in the _find_images
function. The latter wasn't introduced until Python 2.7 and so won't
work on CentOS 6. The script called from this function returns
quickly and doesn't benefit from non-blocking I/O anyway.

Change-Id: I3129f1f5b3fece321ae132ea1a52b0e156e58365
2015-05-02 12:17:56 +00:00
Jeremy Stanley
3020955e85 Provide lsb_release in snapshot builds
Some CentOS 6 images do not come with redhat-lsb-core preinstalled,
so the lsb_release command is not available when prepare_node.sh
wants it. Make sure we install it first if it's not already there.

Change-Id: Ieeae21538c237e069acbc4df051474071b81ba4a
2015-05-01 17:29:26 +00:00
Monty Taylor
8bc8d33f26 Avoid vendor supplied apt mirrors
They are nuking the world

Closes-Bug: #1286818
Change-Id: Id876c8a0dea500b90b3ede1502bc757bfb71c4d4
2015-04-28 08:29:33 -04:00
Mathew Odden
9da29144d1 Fix TODO in tempest testr prep script
technical_debt = technical_debt - 1

This function is now available from the subunit2sql 0.4.2 API,
so lets use it and clean up this TODO.

Change-Id: I5acf279b2e78dddaeb59489d01d92c00ee996f8d
2015-03-26 12:33:29 -05:00
Monty Taylor
fc554ab821 Disable metadata in cloud-init config
Rather than deleting cloud-init, which is going to take longer, just
disable ec2 metadata service. This will be a no-op on rackspace, which
already does this.

Change-Id: I5e8baee50800f7aae474288a914333c21466855a
2015-03-20 14:13:16 -04:00
Monty Taylor
6113b274e4 Revert "Remove ssh host keys during image build"
This reverts commit 1c04bff0d66b85cbdb74fa9ceaec6f1fa9f7ead4.

Change-Id: I097b1cbeed3beedec3f64260676641244b22a2ea
2015-03-20 12:47:10 -04:00
Monty Taylor
c9966cee0b Revert "Regenerate ssh host key on boot"
This reverts commit e60eb73a5d17b19a4beb53d67f7ef680f542a78b.

Change-Id: Iae667506e9878928f39f1518a26800f18f30a3eb
2015-03-20 12:47:05 -04:00
Monty Taylor
e60eb73a5d Regenerate ssh host key on boot
Ubuntu does not do this naturally, so we have to do it for it.

Change-Id: I27635a78ebd7154096ee9c89d7d2b9f4e01e2406
2015-03-19 16:35:36 -04:00
Monty Taylor
1c04bff0d6 Remove ssh host keys during image build
If we're going to remove cloud-init, then we need to make sure we don't
bake in ssh host keys.

Change-Id: Id67934924053cce77b440ddb521112fe1f96e76a
2015-03-19 13:13:30 -04:00
Jenkins
8ae55a6364 Merge "Revert "Cap the subunit2sql version in the nodepool scripts"" 2015-03-18 14:13:40 +00:00
Jenkins
aa56346513 Merge "Remove unneeded workaround" 2015-03-16 23:04:56 +00:00
Matthew Treinish
8f7a206987 Revert "Cap the subunit2sql version in the nodepool scripts"
Once the migration has been run manually we need to use
subunit2sql>=0.4.0 because that is the version of the DB schema running
on the infra db.

This reverts commit aa87c57ca7fe95f9785719acb6b626f12171e240.

Change-Id: I747083bc15ecda7e1926a87c2adcdaf29e565e23
2015-03-16 15:13:05 +00:00
Matthew Treinish
aa87c57ca7 Cap the subunit2sql version in the nodepool scripts
Subunit2sql 0.4.0 was recently released which included a schema change
however the infra db migration takes too long and can't be automated.
(yet?) Until we run the migration we can't use 0.4.0 on any of the
tooling. This caps the version to be less than 0.4.0

Change-Id: I08f113fa904fa962e8f2dc04187ff44e764de47e
2015-03-13 20:48:27 -04:00
K Jonathan Harker
0f12203ac8 Remove unneeded workaround
Now that the pull request has merged, this workaround is no longer
needed.

Change-Id: I6fce6eb13d3e7c8da9ed60a7b9226c6ddb740b2a
2015-03-13 12:15:57 -07:00
Clark Boylan
c1a8b3e76e Revert "Build wheel house during nodepool devstack caching"
This reverts commit 1f84dbb2ce2ae7ed50401299db499adb6e836ec0.

Reverting because this runs a significant portion of devstack during
image builds installing packages necessary to build the wheels. Instead
we should be building wheels and hosting them on our mirror. Then image
builds and jobs themselves can both use this mirror to either cache
wheels or install wheels.

Conflicts:
	nodepool/scripts/cache_devstack.py

Change-Id: I70ebc57076845995c539a42e4f87e241569211b4
2015-03-12 17:07:41 -07:00
Matthew Treinish
5e9046594f Log stdout from build_wheels in cache_devstack
This commit adds a print statement with the stdout from the run_local()
call which executes build_wheels.sh in devstack. Right now when the
script is executed all it does is print that it's running but we are
unable to debug anything that could potentially go wrong during the
execution of the script without the output from build_wheels.

Change-Id: I7433f054127a42fa73598db219c6e351efc98fb7
2015-03-12 16:22:41 -04:00
Jenkins
971251414b Merge "Build wheel house during nodepool devstack caching" 2015-03-11 19:37:13 +00:00
Matthew Treinish
1f84dbb2ce Build wheel house during nodepool devstack caching
This commit adds calling tools/build_wheels.sh to the cache devstack
step in the nodepool scripts. This will generate and cache the wheels
before the devstack run, which is a length process normally involving C
compilations.

Change-Id: I9064d7d9b9b879a05d665c3b002a631fcc953f52
2015-03-06 18:27:51 -05:00
Joshua Hesketh
3691476925 zuul-swift-upload support recursive globs
Allow supplying filename and paths with '**' recursive glob matches
to zuul-swift-upload. Since bash (or shell etc) will expand on any
filenames provided to the program this needs to be used in quotes.

Usage example:
./zuul_swift_upload.py my_results.txt '**/sdist/*.zip' output.log

The hierarchy is always flattened meaning the supplied list is
placed in the topmost generated index.html. Sub-folders still keep
their hierarchy.

Change-Id: I9ba04f7e46b579dcf3f8ad0bd188f41fa5dbcad9
2015-03-04 09:33:17 +11:00
Clark Boylan
12dc33e30c Use sudo when manipulating git for image list
When we build our images we attempt to checkout all of the valid
devstack branches so that we can list all of the images necessary to be
cached for each branch. Unfortunately this devstack repo is owned by
root during dib and snapshot image builds. This means we must use sudo
when updating the repo (either changing branches or updating the
current branch).

Change-Id: I8cd09cfed4d586648dcbd34fa04bfc030c31ee45
2015-02-26 18:19:23 -08:00
Jenkins
edd917e974 Merge "Remove wasteful tripleo workspace-cache cloning" 2015-02-24 20:51:07 +00:00
Clark Boylan
1b9eea5da0 Use sudo when gem installing during image build
We need root permissions in order to gem install rspec during image
builds.

Change-Id: Ida6081d5b26fc168ec37f5d529ba862e1ae05c9a
2015-02-24 10:52:39 -08:00
Dan Prince
ccb783fb6d Remove wasteful tripleo workspace-cache cloning
There doesn't appear to be anything in TripleO that relies
on these clones. Also, this is a bit confusing since people
may expect that things copied into ~workspace-cache would
eventually end up at /opt/git/ (this is not the case).

The correct way to add projects to a job is to export a
PROJECTS variable like devstack-gate does.

Change-Id: I0a96855ed9a38bcade6e506b37621b6b7b348b49
2015-02-19 13:36:23 -08:00
Jenkins
c04e186b7d Merge "prepare_node.sh line lengths are too long" 2015-02-16 15:53:37 +00:00