122 Commits

Author SHA1 Message Date
Jenkins
b460656a49 Merge "Move prepare-node to it's own element" 2016-05-31 15:01:32 +00:00
Jenkins
739565ba85 Merge "Initial commit of zuul-worker DIB element" 2016-05-25 23:47:05 +00:00
Matthew Thode
57d7524bbe
Move prepare-node to it's own element
This moves prepare-node out of node-devstack so that cache-devstack is
not included unnecessarilly.  We change openstack-repos and
node-devstack to depend on this element.  The evenual goal is to remove
node-devstack, which will be done in a followup commit.

Change-Id: I4d9ebc3659f5f59cab4d4ab0762e157b6ff11eda
2016-05-19 21:38:11 -05:00
Paul Belanger
d4bbb4570c
Initial commit of zuul-worker DIB element
Now that zuulv2.5 has landed, we can start building our DIBs using the
new zuul-worker element.  As part of the jenkins clean up, we'll
eventually be removing all things jenkins from our puppet manifests.

As we move forward, we'll be using DIB to bootstrap our zuul-workers
over puppet.

Change-Id: I0e76931fdb4ca0c7445b1e72dc348f0cf03eaee5
Depends-On: Idb4ef11576671180060fb5b2b1202f9bfec5fd47
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-05-14 12:28:41 -04:00
Austin Clark
b96649ed81 Create stackviz element and script in nodepool
In order to get Stackviz running against tempest gate run data, it
will need to be pulled in its pre-built form (i.e. with npm deps
installed) from the nodepool images. This change adds a couple steps
to build Stackviz so that it is in a ready state for retrieval and use
on the logs server.

A future devstack-gate patch will pull the stackviz directory from the
nodepool images onto the logs server. This nodepool patch is required
first in order to prepare stackviz for download. Installing the npm
modules takes a considerable amount of time (~5-10 mins), so it is
best that this work is done on the images before gate jobs are run.

Depends-On: I383e5bf99f6e9e2c7d5fa5c894ef573fa06facd7
Change-Id: I75fdad6584ee237dd07e25148a27ff4df3a5efe1
2016-05-11 15:05:45 -06:00
Paul Belanger
acf160a5a7
Revert "Create stackviz element and script in nodepool"
This reverts commit b4a933c18f4791d297a42a7d6f4fcff71b48b4a5.

Currently all our DIBs are failing to build, as a result we are
reverting stackviz until we can properly test.

Change-Id: Id336c1c8df7123c67f821a3a72da77c470a3a1f1
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-05-02 11:45:25 -04:00
Austin Clark
b4a933c18f Create stackviz element and script in nodepool
In order to get Stackviz running against tempest gate run data, it
will need to be pulled in its pre-built form (i.e. with npm deps
installed) from the nodepool images. This change adds a couple steps
to build Stackviz so that it is in a ready state for retrieval and use
on the logs server.

A future devstack-gate patch will pull the stackviz directory from the
nodepool images onto the logs server. This nodepool patch is required
first in order to prepare stackviz for download. Installing the npm
modules takes a considerable amount of time (~5-10 mins), so it is
best that this work is done on the images before gate jobs are run.

Change-Id: I9bc50b8eba696264d2b97ed20a37f28f4768abb8
2016-04-29 13:09:33 -04:00
Jenkins
efd502e981 Merge "Let dib fix selinux policy for us" 2016-04-22 13:30:05 +00:00
Jenkins
3c333ece51 Merge "Use setfiles to restore unbound selinux policy" 2016-04-21 22:24:12 +00:00
Clark Boylan
1b4537a133 Let dib fix selinux policy for us
New dib (see depends on) will fix selinux policy restoration and run it
at finalise level 90. Therefore we can stop doing it and move our
unbound stuff to level 89 and dib will do magical things for us.

Change-Id: I129297cfd85e2978631add99b7770e2839b459dc
Depends-On: Iae0afe850f52ec3b59c49507fa9bbcc1c8f8cfa1
2016-04-20 14:30:31 -07:00
Clark Boylan
c96ea14e82 Use setfiles to restore unbound selinux policy
We finalise the unbound install at run parts level 99, but the dib built
in selinux restore runs at level 11 so we need to handle this directly
for unbound.

We switch from restorecon to setfiles becaues restorecon does not do the
right thing in a chroot (it is chroot aware and essentially noops for us
because we are in a chroot).

Change-Id: I7af6a4925e191a7d1bc78fb2d66d9ce74b944a39
2016-04-20 14:14:44 -07:00
Jeremy Stanley
36ed0b0cec Revert "Lower our nodepool-base finalise.d script"
This reverts commit e62609f45ce008d9a3b40ea406e5a68d8420f8e1.

Change-Id: Ie9076d70d062a7c666e53966e03377c65d3aab8a
2016-04-20 13:40:56 +00:00
Paul Belanger
e62609f45c
Lower our nodepool-base finalise.d script
This is needed because we need 11-selinux-fixfiles-restore from
rpm-distro to run after us, so we can properly restore SELinux
security contexts.

We can also remove restorecon functionality, since it doesn't work in
chroots on ubuntu build servers.

Change-Id: Ic9ffb121d5950d9028061591a346a5228fe89214
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-04-20 08:44:54 -04:00
Jenkins
c8b6ae5158 Merge "Fix package and image list comment cleanup" 2016-04-20 06:09:02 +00:00
Ian Wienand
6cab81fec4 Fix dependencies for openstack-repos
openstack-repos:install.d/95-chown-jenkins relies on the jenkins user
being around, which isn't done until the puppet stuff in
node-devstack:install.d/20-prepare-node.

Express this dependency

Change-Id: I1cf006d709a54b9da31c2375a0941fe497c3c761
2016-04-20 13:36:57 +10:00
Clark Boylan
7f35178617 Fix package and image list comment cleanup
When cleaning up comments from lines of the form:

  foo bar # this is a comment

You want to remove remove everything after the first comment character
not after the last comment character. This is because:

  foo bar # this is # a comment

Should get everything after bar removed not everything after is.
Unfortunately when you use str.rfind() you remove everything after is
not everything after bar. Switch to regular trusty find to fix this.

Change-Id: I78aa6b51b5be03bd3b8ce7885415442171218977
2016-04-19 17:27:14 -07:00
Jenkins
51ec23046a Merge "Silence zuul_swift_upload" 2016-04-05 15:03:17 +00:00
Jeremy Stanley
565c807d3b Add traceroute to default preinstalled packageset
When building images with diskimage-builder, include traceroute so
that it can be used as an ever-present diagnostic in a variety of
jobs without having to declare it in job-specific or
project-specific dependency lists.

Change-Id: Ic0be90226d01b399f3ca0c26ad6d43bcaa2a0bdf
2016-04-04 18:19:36 +00:00
Clark Boylan
90bc24b54f Add removal of grub boot timeouts back in
When we removed the restrict memory grub stuff because hpcloud was no
longer in use we also removed the configuration updates that set the
grub timeout to 0. This can save many seconds per instance boot purely
waiting for grub to timeout and continue booting the host.

Change-Id: I2d79ec4ebd55f927d868e06c8f00cf50f34f3b92
2016-03-30 15:20:10 -07:00
Jeremy Stanley
678c1c81b5 Drop tempfile workaround for aggregate bindep list
Since support for using "-f -" to read a package list from stdin
is supported in bindep 0.1.0 and later, remove the hacky temporary
workaround which stashed it in a tempfile.

Change-Id: I02d153113a160028658e38828e113fcf02bc5ab1
2016-03-28 14:03:18 +00:00
Andreas Jaeger
84e76e3f72 Silence zuul_swift_upload
At the end of gate jobs, we get warnings from zuul_swift_upload that
are not actionable for us.

Use requestsexceptions to silence these warnings. Add requestsexceptions
to the venv that is created for zuul_swift_upload to run in.

Example file
http://logs.openstack.org/48/298048/1/check/gate-ha-guide-tox-checkniceness/62e2d16/console.html

The example shows 6 warnings starting with:
/usr/zuul-swift-logs-env/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:315:
SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject
Name Indication) extension to TLS is not available on this platform.
This may cause the server to present an incorrect TLS certificate, which
can cause validation failures. For more information, see
https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/zuul-swift-logs-env/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:120:
InsecurePlatformWarning: A true SSLContext object is not available. This
prevents urllib3 from configuring SSL appropriately and may cause
certain SSL connections to fail. For more information, see
https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
...

Change-Id: I02b4b6c7b426f3e9aa498941e4d75f67066d4d27
2016-03-27 21:23:31 +02:00
Clark Boylan
53cd995cf6 Add element to handle generic package installs
We end up needing to install a fair bit of stuff on top of the minimal
images in order to make them useable. Do this in a generic infra package
needs element that can grow to accomodate our needs.

For the future we can hopefully get some of this into dib itself so that
others can easily build instances they can ssh into.

Change-Id: I5826df829971e182cd0713b29df1cf70c119a0ab
2016-03-17 12:26:58 -07:00
Igor Belikov
e57fe8bdf7 Add curl package to puppet nodepool element
puppet element calls install_puppet.sh script from system-config repo,
which depends on either curl or wget being installed in the system. Some
minimal images like debian jessie or ubuntu xenial miss those packages
and cause puppet element to fail when building the image.

Change-Id: Ic5124906b19b82d134a611e68fc7c5dd6fc88702
2016-03-16 16:18:30 +03:00
Paul Belanger
458f2fdb64 Fix typo with $YUM variable
Like ianw suggested, we should fallback to yum if ${YUM} is not
defined.

Change-Id: I9e2c632a3c7225d13e6245e65fa4da29ce7541df
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-03-07 13:11:10 -05:00
Jenkins
547aa9c06b Merge "Fix cache-bindep breakage on Fedora / CentOS" 2016-03-07 13:32:26 +00:00
Paul Belanger
7158aa01f8 Fix cache-bindep breakage on Fedora / CentOS
Currently, we don't have 100% gate coverage for
other-requirements.txt. As such, it is possible for our DIB builds to
fail on bindep.  So, we only cache packages for Ubuntu Trusty ATM
until we can enable 100% coverage for centos and fedora for
other-requirements.txt per project.

Change-Id: I0f06619224ab59bc6165316226abeccb75c101cb
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-03-06 19:27:58 -05:00
Clark Boylan
69a073edbe Don't cache inactive repos
We have attic and stackforge projects all of which are now inactive.
Because they are inactive we don't need to be caching these repos on
every test slave image.

Change-Id: I4df78c3d542758ce6159c195c1407f1d56f565a0
2016-03-05 18:01:08 -08:00
Ian Wienand
2badcc1893 Add a dib-builddate file
Add a build-date timestamp file to the nodepool-base element and
output that as part of the "network-info" macro that we run at the
start of most tests.

This will allow non-priviledged users to quickly see the date the node
running their test was built on, which can be correlated to the logs
on nodepool.openstack.org to help debug issues that might have to do
with the underlying image build.

Change-Id: Id0c9f6203ed487350285031d3965bc6290370a27
2016-03-01 15:16:15 +11:00
Jenkins
8aa39b8d06 Merge "Cleanup the early devstack clone script" 2016-02-16 04:25:36 +00:00
Andreas Jaeger
71d84bed47 Remove restrict-memory
We do not have HP cloud anymore, remove the restrict-memory files.

Change-Id: I68f8fbd01927ead42e9fab1ff41360f035275d4a
2016-02-15 20:04:12 +01:00
Ian Wienand
abbbf317ce Cleanup the early devstack clone script
This script is unnecessarily complex for what it is doing, which is
essentially pre-seeding the source-repositories cache for devstack.
As we can see from I41e81d6bac98875eecde2376e0865784626e11a8 it's very
confusing having large parts of the source-repositories script
copy-pasted as a separate element and has led several of us down the
wrong path.

Strip this script back to the simple thing it is doing, which is
checking out/updating the source-repositories devstack cache.

I have tested this by building an image with a warm cache and with a
cold cache, in both cases the checkout was found and the list of
images to cache in 55-cache-devstack-repos was found.

Change-Id: I6c686312de102cbe438585e26bf6986e06b6f41c
2016-01-19 11:11:28 +11:00
Ian Wienand
c70cf77969 Update remote refs in early devstack clone
In another confusing addendum to
I41e81d6bac98875eecde2376e0865784626e11a8 (which was already a
confusing addendum to Ieb6a6e9f55bd93f63c3d0a71828c276c2d02e1b9), we
have decided that the refspec used to fetch here is not sufficient to
clear out remote branches everything when updating.

"+refs/heads/*:refs/heads/*" says to replicate everything from the
remote refs/heads into our local refs/heads, but leaves out
refs/remotes/*

The upshot of this is that I41e81d6bac98875eecde2376e0865784626e11a8
will remove the local branches (refs/heads/stable/icehouse, say) but
not remove the remote branches (refs/remotes/origin/stable/icehouse).
The devstack caching script keeps picking up these remote branches,
checking them out, and consequently trying to download old images.

*Nothing* ever removes these branches.  In the main dib cache git
update, we also have --prune, but our refspec there is even more
limited (+master:master).  This explains why these branches never seem
to die.

Note, an even better mirror would be "+refs/*:refs/*" (in fact, if you
do git clone --mirror, this is what the repo would be setup with to
fetch by default).  However, this drags in "refs/changes/*" and all
sorts of other gerrit things.  We don't really need them so we just
keep the limiting on.

Change-Id: Ia9c3ffdb2b5f72a45d629961338b415308d6dd21
2016-01-19 11:06:12 +11:00
Ian Wienand
53832d82cd Prune old branches in early devstack clone
In a rather confusing addendum to
Ieb6a6e9f55bd93f63c3d0a71828c276c2d02e1b9, we have actually mirrored a
version of the source-repo script from dib and munged it to cache
devstack early so we can use it to find the vm images to download and
cache.

However, we are not ensuring that we remove old branches in this
clone, which is leading to the problems of us picking up images from
old branches that don't exist any more.

Change-Id: I41e81d6bac98875eecde2376e0865784626e11a8
2016-01-18 17:27:31 +11:00
Jenkins
c6526e5480 Merge "Install puppet using build_git system-config" 2015-12-15 20:41:09 +00:00
James E. Blair
381d507b04 Increase memory restriction to 9023M
On OVH and Bluebox, the memory layout is such that there is still
significant (~900M) memory above the 8192M address.  Increase the
limit to encompass that, which will bring these providers up to
approximately 8G, while increasing hpcloud (which is the actual
target of this restriction) to about 8.5G.

Change-Id: I5c121be55cadad13ad5807968f33b492f9b1e215
2015-12-15 10:32:06 -08:00
Clark Boylan
bffc38d470 Install puppet using build_git system-config
When running disk image builder use the build specific checkout of
system-config to install puppet and puppet modules. This allows for easy
local testing of changes to those scripts. Without it master is always
used.

Change-Id: Ifc86d77f702633d630adc1ee619f58607cb56c82
2015-12-08 13:43:47 -08:00
Andreas Jaeger
d8170deb07 Remove CentOS6 from nodepool scripts
With CentOS6 usage removed, remove special handling for it from nodepool
scripts.

Change-Id: Ife5ec9ade201dcd8f5969087dd2fc88be12dbebd
2015-12-04 22:11:22 +01:00
Matthew Treinish
3ed5aa98c5 Revert "Set a temporary cap on subunit2sql before 1.0.0 release"
The release has happened, the migration worked, all's right with the
world.

This reverts commit 04eb36588d80dd77a9c692f53470b7d727567604.

Change-Id: I28a3e9cf3b420a6ccca20c52c4e67adc3f5710c5
2015-11-29 17:01:45 +00:00
Matthew Treinish
04eb36588d
Set a temporary cap on subunit2sql before 1.0.0 release
This commit adds a temporary version cap on subunit2sql to be < 1.0.0.
The 1.0.0 release includes a very large database migration which will
be slow to execute. The python DB api from >=v1.0.0 will not work with
a database that doesn't have the updated schema. So while the migration
is running let's cap the version we install to prevent everything from
breaking while the migration is running. (which might take days)

Change-Id: Iab89beb5c7aba8b744a62f5063e513b72cab0ec2
2015-11-23 15:49:58 -05:00
Jeremy Stanley
23da1ea04a Revert "warm pip wheel cache during image build"
Turns out we can't do this without dirtying our images by
preinstalling all the libraries/headers our requirements want to
link against. We should revisit the wheel mirror solution which has
already been written as a saner solution.

This reverts commit 1503ef327817bd42cbf888e773f5639e660169a7.

Change-Id: I69ea73b6eb4b620f1ac77467ee3784ec3c813ee2
2015-11-04 20:40:23 +00:00
Sean Dague
1503ef3278 warm pip wheel cache during image build
This installs everything in upper-constraints in a throw away venv,
which has the knock on effect of fully populating the wheel cache for
root with wheels for everything. This hopefully speeds up all devstack
runs quite a bit.

Change-Id: I429f353c33d892f76552c83d34ac1329ed18f97f
2015-11-04 12:52:22 -05:00
Cyril Roelandt
2554d974b8 Add python3-dev
The gate-neutron-dsvm-functional-py34 job needs Python.h, and it is
currently not available when running Python 3.

Change-Id: I00ce26864db682a4577dc5360519918007aa5499
Partial-Bug: #1500400
2015-10-22 15:08:28 +02:00
Ian Wienand
90c53a8ded diskimage-builder element cleanups for dib-lint
Since I6c5a962260741dcf6f89da9a33b96372a719b7b0 dib has had a
standardised method for ensuring consistency of tracing and error
detection.  Bring the tracing for these elements up to that standard,
but maintain the status-quo of flags such as "-e" and "pipefail" by
adding ignore flags where appropriate (we can update these separately
to avoid breakage)

Other minor changes are alphabetical-ordering in the element-deps
files and permissions on prepare-node script

With this, "tox -edib" passes

Change-Id: Ibba1dadb9e819f94294c9d583b83ff698252f93f
2015-10-08 11:33:03 +11:00
Ian Wienand
32dcd6af92 Skip caching for dnf (Fedora 22) builds
We are still working on the caching story for dnf-based builds
(i.e. Fedora 22). There are a couple of options which we will work
through on the linked page.  We will sort this out before we move the
devstack job (yet to be created, because the nodes aren't there yet)
out of experimental.

In the mean time, disable the caching in these elements so we can get
Fedora 22 image builds (currently they're in a big looping failure
[1]).

[1] http://nodepool.openstack.org/image.log

Change-Id: I3a435889fc5109d7365240068047aac98abc605e
2015-10-01 15:35:16 +10:00
Jenkins
5cda264308 Merge "Use restorecon over chcon for unbound selinux" 2015-08-27 11:50:30 +00:00
Paul Belanger
f30d863496 Use restorecon over chcon for unbound selinux
Rather then explicitly forcing a security context for unbound, we
should restore the default selinux contexts defined by the OS.

Change-Id: I4bb21dcbbcbcff6a5458ebf9478f58d95ad7240a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2015-08-19 10:15:13 -04:00
Matthew Treinish
76d164b1ba
Create an os-testr venv for running subunit2html
This commit creates a venv for installing os-testr which will enable
all test jobs to have access to the subunit2html utility which has
been moved to live inside the os-testr package instead of as a slave
script.

Change-Id: I2050b54eb2def10438764f3eeb55ecf9caa874dc
2015-08-14 12:37:22 -04:00
Paul Belanger
ebdd2553d0 Remove color output from puppet apply
This help when reading log files from nodepool. Otherwise we see the
following in the log files:

  [1;31mWarning: Config file /etc/puppet/hiera.yaml not found, using
  Hiera defaults[0m

Change-Id: I3a865e5107e2749ed44c144539af49e311e0125f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2015-08-06 15:38:35 -04:00
Jenkins
78d105a8a4 Merge "Express nodepool base's dependency on puppet" 2015-07-15 13:12:01 +00:00
Monty Taylor
e3ad2d85d2 Express nodepool base's dependency on puppet
The nodepool base element really needs the puppet element, but we didn't
express that in the element deps. That makes a later reorg of elements
more difficult than it needs to be, because our consumption element list
is expressing internal dependency chains.

Change-Id: Ib2296bed1d7f289e7aa2720a38555275de1b6402
2015-07-15 08:34:13 -04:00