133 Commits

Author SHA1 Message Date
Jenkins
5cda264308 Merge "Use restorecon over chcon for unbound selinux" 2015-08-27 11:50:30 +00:00
Paul Belanger
f30d863496 Use restorecon over chcon for unbound selinux
Rather then explicitly forcing a security context for unbound, we
should restore the default selinux contexts defined by the OS.

Change-Id: I4bb21dcbbcbcff6a5458ebf9478f58d95ad7240a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2015-08-19 10:15:13 -04:00
Matthew Treinish
76d164b1ba
Create an os-testr venv for running subunit2html
This commit creates a venv for installing os-testr which will enable
all test jobs to have access to the subunit2html utility which has
been moved to live inside the os-testr package instead of as a slave
script.

Change-Id: I2050b54eb2def10438764f3eeb55ecf9caa874dc
2015-08-14 12:37:22 -04:00
Paul Belanger
ebdd2553d0 Remove color output from puppet apply
This help when reading log files from nodepool. Otherwise we see the
following in the log files:

  [1;31mWarning: Config file /etc/puppet/hiera.yaml not found, using
  Hiera defaults[0m

Change-Id: I3a865e5107e2749ed44c144539af49e311e0125f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2015-08-06 15:38:35 -04:00
yanyanhu
da344ab915 Fix a typo in prepare_node_devstack.sh
Change-Id: I763b6efe57d37c1e2a1da299b9a2a0c32e0df32f
2015-08-04 02:36:07 -04:00
Jenkins
78d105a8a4 Merge "Express nodepool base's dependency on puppet" 2015-07-15 13:12:01 +00:00
Monty Taylor
e3ad2d85d2 Express nodepool base's dependency on puppet
The nodepool base element really needs the puppet element, but we didn't
express that in the element deps. That makes a later reorg of elements
more difficult than it needs to be, because our consumption element list
is expressing internal dependency chains.

Change-Id: Ib2296bed1d7f289e7aa2720a38555275de1b6402
2015-07-15 08:34:13 -04:00
Jenkins
4d5bc96fa0 Merge "nodepool: describe nodepool-base env variables" 2015-07-14 16:06:26 +00:00
Jenkins
e998bfcabf Merge "Add dependency on nodepool-base to node-devstack" 2015-07-14 13:50:55 +00:00
Jeremy Stanley
f954e8ccd4 Revert "Block installing of mock from wheels"
This reverts commit 71d10ea626f5a62d6b48133e19f469b19c9a7f14.

It was unnecessary once the offending wheel got removed from PyPI,
and this never worked anyway. It was preventing the ability to boot
new nodes, erroring thusly:

    ./configure_mirror.sh: line 25: /tmp/pip.conf: Permission denied

Deleting the images to which the change was applied returned
nodepool to being able to boot new nodes again.

Change-Id: I0841c6a5a26cf5be22e2d8fea861bdceb0393842
2015-07-12 01:54:12 +00:00
Monty Taylor
71d10ea626 Block installing of mock from wheels
This is needed until a new wheel of mock can be cut after the new pbr
release is cut.

Change-Id: I96a1794ee754f879514b34add3e36a4a42fd681b
2015-07-10 10:23:22 -04:00
Antoine Musso
9f5c63b0d3 nodepool: describe nodepool-base env variables
The nodepool-base element relies on a couple environement variables.
Describe NODEPOOL_SCRIPTDIR and NODEPOOL_STATIC_NAMESERVER in the
README.rst file.

Change-Id: I56f2aab095a0504e19598d7296d072e7a51b07c2
2015-07-03 21:28:08 +02:00
Jeremy Stanley
ff8a9f590e Clean up bindep invocation in cache-bindep element
Invoke bindep from the bindir for its virtualenv in /usr/bindep-env,
and guard its exit code with true since its normal behavior is to
return zero only when all needed packages are already present.

While here, replace calls to tempfile with mktemp since the former
is not readily available on CentOS 6.x.

Change-Id: I926e73f305b1b54f6855cc4acf3407e91b88213a
2015-07-03 01:31:40 +00:00
Jeremy Stanley
43e8e90cbf Don't fail to build cache on missing bindep lists
While git show is being used to build up a list of candidate
packages to cache, it can cause the cache hook to abort if it
returns nonzero because of a nonexistent bindep list file. Guard
that with a fallback to true so that the loop can continue
normally.

Change-Id: Ic3b773ad2e8efa2c450d37a0115355c0e95da853
2015-07-03 00:13:17 +00:00
Jeremy Stanley
0e079c607f Backward-compatible branch list for bindep element
The version of git on CentOS 6.x is too ancient to support git
branch --list which is necessary to disambiguate -r with a name
pattern. Instead just list all remote branches and filter with grep
like a good Neanderthal.

Change-Id: I4e673e86acb70c205d159e22b5864cf60ae831df
2015-07-02 23:41:52 +00:00
Jeremy Stanley
e700bd2820 Rename bindep package caching element for conflict
A quirk of diskimage-builder is that no two elements you compose are
allowed to have parts with the same filename. Rename cache-bindep's
install.d/50-download-pkgs so that it doesn't conflict with the one
of the same name in the cache-devstack element.

Change-Id: I5709147fc5bd4c06fb1bba8ea0e068c0eda7d010
2015-07-02 23:03:45 +00:00
Jenkins
bb486b5ebb Merge "Pre-cache distro packages from bindep lists" 2015-07-02 21:02:04 +00:00
Jeremy Stanley
62ad51e638 Remove managed stable/icehouse branch testing
Remove jobs which test stable/icehouse branches of repos tagged
release:managed as the branch has reached end-of-life and is being
removed from those repos.

Change-Id: I88a44cfa84597012af7da0bd22de02dc2349b1fa
2015-07-01 18:40:41 +00:00
Jeremy Stanley
a1f1ce370d Fetch all heads and tags when caching source repos
Passing a source-repositories ref of "*" signals fetching all heads
and tags rather than just a specific refname. This is desirable for
the warm cache on our job workers, since jobs may run against
arbitrary branches (or possibly even tags). The point is to have a
reasonably complete mirror of each repo, rather than a fairly
minimal clone.

Change-Id: I624ed1e259e007d4246afb45c3a0560598bfbe3f
Depends-On: I4562c9689a8d235ebe09b2f7178aa5890dbc85f1
2015-06-25 15:54:11 +00:00
Jeremy Stanley
fc39aea6d0 Pre-cache distro packages from bindep lists
We intend to use bindep to tell us what distro packages should be
installed at job run-time, so need to similarly identify those in
such a way that they can be cached on our worker images.

Install the latest bindep release from PyPI into /usr/bindep-env
(consistent with the way we provide virtualenvs for access to
zuul-cloner and libraries used by our log uploads to swift).
Concatenate bindep lists from all branches of every hosted Git repo
along with our fallback list in project-config. Run bindep to render
them into a platform-specific manifest of package names and feed
that to the relevant package management tool for the platform to
download this set into the package cache.

Note that similar tooling is not applied to snapshot nodes, since we
only intend to run jobs which use bindep on diskimage-builder
created nodes.

Change-Id: I0c408fee35dfe9b4f700c51565bfaae0a3d03beb
2015-06-24 18:56:09 +00:00
Ian Wienand
877d93bb84 Remove cloud-init details
If we have booted with cloud-init, then this status directory is
populated.  Remove it so when we boot snapshot images, they behave as
if on a fresh system.

Change-Id: Idc9ce01290b659e3239d30be847221447a8e5e84
2015-06-18 14:05:50 +10:00
Ian Wienand
a9444cdad8 Remove mount of ephemeral disk
hpcloud has started sending metadata to cloud-init to mount ephemeral
disks.  This ends up writing a fstab entry for /dev/vdb.

---
$ curl http://169.254.169.254/2009-04-04/meta-data/block-device-mapping/ephemeral0
/dev/vdb
---

It's unclear why this just started happening, but it did.

devstack-gate later attempts to repartition this into swap & disk
space and mount it elsewhere.  So remove this mount from fstab -- it
shouldn't come back because the next thing we do here is tell
cloud-init to not use the metadata source.

Change-Id: I3787d0f7e5139e891686ffbb2970e65d09f112b1
2015-06-18 14:05:38 +10:00
Jens Rosenboom
abbe9c6093 Add dependency on nodepool-base to node-devstack
The build for node-devstack will fail because of missing scripts unless
nodepool-base is executed before, so add this as dependency.

Change-Id: Iabc06a542529500ad248ae067beed9639d24e506
2015-06-11 10:22:08 +02:00
Steve Kowalik
85a0f4f44f Switch to using $() for subshells, part 1
Cleanup every use of `` for subshells in the nodepool and tools
directory , replacing them with $(), and finally making the scripts
consistent.

Change-Id: I2b05cd20f9c9a30ab88f8db235aa81da93b1fad3
2015-06-04 15:30:55 +10:00
James E. Blair
8e5c538e5a Add allow_hosts to pydistutils.cfg
This should cause easy_install to only contact our pypi mirrors
when fetching packages.

https://pythonhosted.org/setuptools/easy_install.html#restricting-downloads-with-allow-hosts

Change-Id: I7f7847721c2e8c6299c7fe5ab20045e8bdbc5bd7
2015-06-02 10:50:03 -07:00
Jenkins
38ec65bd48 Merge "Deal with yum --downloadonly CentOS 6.x quirks" 2015-05-12 21:38:26 +00:00
Jeremy Stanley
3d5b04ed19 Deal with yum --downloadonly CentOS 6.x quirks
Some packages may depend on python-setuptools, which is not
installed and cannot be reinstalled on CentOS 6.x once yum has
erased them, so use --skip-broken to avoid aborting. Also on this
platform --downloadonly causes yum to return nonzero even when it
succeeds, so ignore its exit code.

Change-Id: Iaada39ae81e1e47fe9d0bedba80fd19e4e0e6f38
2015-05-12 15:25:01 +00:00
Clark Boylan
f4bb992006 Don't disable cloud-init if not installed part 2
During scripted snapshot image builds we remove python-setuptools on
rhel/centos 6. This uninstalls cloud-init on hpcloud centos6 builds
because cloud-init depends on setuptools. We don't actually need
cloud-init in hpcloud because they use dhcp and we have hard coded ssh
keys so just don't bother setting cloud-init config if the config dir in
/etc does not exist.

One could argue that we would possibly want cloud-init to regen host
keys for us, but centos' sshd service should do that for us if the host
keys do not exist.

Change-Id: I96621b0ab1574eb8db0f4394877d3c1fc8208576
2015-05-11 19:39:11 -07:00
Monty Taylor
449d432feb Don't disable cloud-init if it is not installed
When we're building nodepool images on top of minimal elements, there
will be no cloud-init, and therefore no need to disable cloud-init
datasources. In fact, trying to do so will be an error.

Change-Id: I98887c43566e07f2be9d2dc5fae6538078c7348e
2015-05-10 12:58:08 -04:00
Monty Taylor
e60dc2a3c1 Don't require NODEPOOL_SCRIPTDIR to be set
If someone is building an image locally and don't want to do anything
special with nodepool scripts, they should not need to set an env var.
The env var obviously still needs to be honored for nodepool operation
and for override.

Change-Id: I08076a8eafe7019e715b1b46633e8f603031a1ea
2015-05-10 12:41:53 -04:00
Jenkins
47532ee5c3 Merge "Install yum-plugin-downloadonly on nodes" 2015-05-07 16:11:27 +00:00
Jeremy Stanley
0137c25b46 Install yum-plugin-downloadonly on nodes
The --downloadonly option to yum is provided by the
yum-plugin-downloadonly package. This is merely a virtual package
satisfied by yum itself in newer releases, but an optional package
in older ones such as CentOS/RHEL 6.x. Install it just to be sure it
will work, since we use this to pre-cache RPMs on nodepool images.

Change-Id: I9e2e1605f3721c410180aa46a81b7b731d08503a
2015-05-07 13:33:12 +00:00
Jeremy Stanley
572c35a827 Look for yum instead of rpm in elements too
On CentOS 6.x the rpm executable is in /bin instead of /usr/bin, but
the cache-devstack element ends up running yum if it wants to
download RPM packages anyway. Look for yum instead of rpm as an
indication of which packages to install.

Commit fa18656 fixed this in cache_devstack.py for snapshot images,
but missed that it was affecting diskimage-builder elements as well.

Change-Id: I4b76a48564cd5d703d79cfda88bbbfe97216a70a
2015-05-07 01:56:52 +00:00
Jenkins
4cd523714d Merge "Do a better job of identifying CentOS 6" 2015-05-06 14:07:16 +00:00
Jenkins
63dbb3d406 Merge "Look for yum instead of rpm when caching devstack" 2015-05-06 14:07:09 +00:00
Jeremy Stanley
6a8f64190d Do a better job of identifying CentOS 6
Since DIB creates a chroot but doesn't actually chroot in an
extra-data script, look for /etc/redhat-release relative to the
temporary chroot base path. Also given that the centos element
leaves DIB_RELEASE exported as am empty string in the calling
environment rather than unset, place it first in the conditional
list.

Change-Id: I0a51f88ff5ea71b6aa6e6406b656cefb0e135e88
2015-05-04 04:22:05 +00:00
Jeremy Stanley
08dc5cf613 Import subprocess in cache devstack element
Change fd9c80208c00021a49df6907973bbbb3fa242069 inadvertently
removed the subprocess import from 55-cache-devstack-repos even
though it was used in a function defined within that element, unlike
in its snapshot prep script counterpart. Readd it so that this
element works again.

Change-Id: I954a1e7e99af96b08e0a59e99e513902731fd773
2015-05-04 03:24:56 +00:00
Jeremy Stanley
fa18656f1c Look for yum instead of rpm when caching devstack
On CentOS 6.x the rpm executable is in /bin instead of /usr/bin, but
cache_devstack.py ends up running yum if it wants to download RPM
packages anyway. Look for yum instead of rpm as an indication of
which packages to install.

Change-Id: Iad79c0fcf66d2bd457195f007009c76f6e6aa2d2
2015-05-04 02:52:50 +00:00
Monty Taylor
c183c33710 Don't install redhat-lsb-core in prepare_node
We don't need it for anything other than detecting whether or not we
should update the sources.list file. Except that's obviously not
something we should do on systems with yum

Remove the call and defer it to install_puppet where it's also
installed.

Change-Id: Ie22aeac1e4c731e7ab61514cb4982c8c35c482e6
2015-05-03 12:55:55 -04:00
Jeremy Stanley
fd9c80208c Replace subprocess with run_local in _find_images
In the DevStack caching script/element for Nodepool images, use
run_local instead of subprocess.check_output in the _find_images
function. The latter wasn't introduced until Python 2.7 and so won't
work on CentOS 6. The script called from this function returns
quickly and doesn't benefit from non-blocking I/O anyway.

Change-Id: I3129f1f5b3fece321ae132ea1a52b0e156e58365
2015-05-02 12:17:56 +00:00
Jeremy Stanley
1023e20f41 Determine CentOS 6 platform in cache-devstack
The centos element (which is CentOS 6.x specific) doesn't provide
DIB_RELEASE so we have to identify it another way.

Change-Id: Ibf3b25b99f03c6077538dec6c3770dc0f690e3f5
2015-05-01 20:57:05 +00:00
Jeremy Stanley
3020955e85 Provide lsb_release in snapshot builds
Some CentOS 6 images do not come with redhat-lsb-core preinstalled,
so the lsb_release command is not available when prepare_node.sh
wants it. Make sure we install it first if it's not already there.

Change-Id: Ieeae21538c237e069acbc4df051474071b81ba4a
2015-05-01 17:29:26 +00:00
Monty Taylor
8bc8d33f26 Avoid vendor supplied apt mirrors
They are nuking the world

Closes-Bug: #1286818
Change-Id: Id876c8a0dea500b90b3ede1502bc757bfb71c4d4
2015-04-28 08:29:33 -04:00
Mathew Odden
9da29144d1 Fix TODO in tempest testr prep script
technical_debt = technical_debt - 1

This function is now available from the subunit2sql 0.4.2 API,
so lets use it and clean up this TODO.

Change-Id: I5acf279b2e78dddaeb59489d01d92c00ee996f8d
2015-03-26 12:33:29 -05:00
Monty Taylor
fc554ab821 Disable metadata in cloud-init config
Rather than deleting cloud-init, which is going to take longer, just
disable ec2 metadata service. This will be a no-op on rackspace, which
already does this.

Change-Id: I5e8baee50800f7aae474288a914333c21466855a
2015-03-20 14:13:16 -04:00
Monty Taylor
6113b274e4 Revert "Remove ssh host keys during image build"
This reverts commit 1c04bff0d66b85cbdb74fa9ceaec6f1fa9f7ead4.

Change-Id: I097b1cbeed3beedec3f64260676641244b22a2ea
2015-03-20 12:47:10 -04:00
Monty Taylor
c9966cee0b Revert "Regenerate ssh host key on boot"
This reverts commit e60eb73a5d17b19a4beb53d67f7ef680f542a78b.

Change-Id: Iae667506e9878928f39f1518a26800f18f30a3eb
2015-03-20 12:47:05 -04:00
Monty Taylor
e60eb73a5d Regenerate ssh host key on boot
Ubuntu does not do this naturally, so we have to do it for it.

Change-Id: I27635a78ebd7154096ee9c89d7d2b9f4e01e2406
2015-03-19 16:35:36 -04:00
Monty Taylor
1c04bff0d6 Remove ssh host keys during image build
If we're going to remove cloud-init, then we need to make sure we don't
bake in ssh host keys.

Change-Id: Id67934924053cce77b440ddb521112fe1f96e76a
2015-03-19 13:13:30 -04:00
Jenkins
8ae55a6364 Merge "Revert "Cap the subunit2sql version in the nodepool scripts"" 2015-03-18 14:13:40 +00:00