1740 Commits

Author SHA1 Message Date
Zuul
dd8f80623e Merge "Revert fixes for legacy boot jobs" 2018-02-21 14:58:41 +00:00
Paul Belanger
cee738d3af
Sort formats results from zookeeper
This is to help user readability, today dib-image-list can return
unsorted results for dib-image-list, making it difficult to read.

Change-Id: Iadf31347ee5310e9c021e985b61774011f77ed87
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-02-20 19:20:47 -05:00
Paul Belanger
126a4eabda
Use python3 for devstack testing
Nodepool only supports python3, so just default to python3 when
testing with devstack.

Change-Id: Ie6149a8fcdefc3c8e014ba8161caa95da41f6aaf
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-02-20 12:37:35 -05:00
Zuul
89f0d11a6e Merge "Avoid tools/nodepool-integration-setup.sh from zuul" 2018-02-20 10:24:38 +00:00
Zuul
9f5e5fbaa4 Merge "Remove copying of legacy install-distro script" 2018-02-20 10:24:25 +00:00
Zuul
910dfcee83 Merge "Fix nodepool-zuul-functional" 2018-02-20 09:51:14 +00:00
Zuul
42d40da565 Merge "Hack for test_delete_now rare failures" 2018-02-20 06:43:25 +00:00
David Shrewsbury
59d636740a Hack for test_delete_now rare failures
Until we have a better solution (either in kazoo or in nodepool)
for the problem described in the long inline comment, let's prevent
these rare test failures that have been plaguing us.

Change-Id: Id43c2dba88f8dd3b8ce7b70be551e071ca4b1dd1
2018-02-19 16:20:46 -05:00
Andreas Jaeger
a3ad923a7e Avoid tools/nodepool-integration-setup.sh from zuul
Fold in content from zuul/tools/nodepool-integration-setup.sh into our
playbook to reduce dependency of the repos and simplify job setup.

Change-Id: I168522ae2c214eac2f51410c67b3a79862a8787f
2018-02-16 08:56:18 +01:00
Andreas Jaeger
b692467182 Remove copying of legacy install-distro script
This is not needed anymore, the zuul job does not use it.

Change-Id: I4f9e41485ebe2c8b8e2303283b2937b8d3d67d49
Depends-On: https://review.openstack.org/545221
2018-02-16 08:40:12 +01:00
Ian Wienand
5195ccb70a Fix nodepool-zuul-functional
Several fixes to this job:

We need to run bindep twice - once for nodepool, once for zuul. Add
invocation of bindep role and also copy install-distro-packages so that
the job works, next step is remove the install-distro-packages from
zuul.

Add a post-run to copy the nodepool logs, so you can diagnose what's
going wrong if the jobs fails

Fix up a configuration issue, it tries to write build-logs to
/var/log/nodepool which it doesn't own, redirect to the temp area.

Add it as a non-voting check job

Depends-On: https://review.openstack.org/545163
Change-Id: I12db55d3e4c7a71b9af56567858df0a620ee3b73
2018-02-16 08:38:58 +01:00
Paul Belanger
1d3989ca35
Add opensuse-tumbleweed to nodepool dsvm testing
We'd like to bring opensuse-tumbleweed online for openstack-infra, so
enable testing of tumbleweed to help catch things.

Change-Id: Id0a1fbf6f3c7df63402670f8c54f7fb3af652ae5
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-02-15 20:16:36 -05:00
Paul Belanger
796324f15e
Use $NODEPOOL_KEY_NAME for dsvm testing
Follow up patch to previous changes and removes hardcoded 'root'
key-name.

Change-Id: If1ac2b0a55898607409216a3ffa1bb45831ad2fc
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-02-15 20:12:20 -05:00
Paul Belanger
04c586e644
Take 2: Add functional test for key-name and glean
Validate that glean will properly use the key-name from nodepool, and
allow a root user to ssh into the node.

It seems when we merge feature/zuulv3 back into master, we lost this
logic. It is still helpful to actually SSH into a node to validate
glean and diskimage-builder worked as expected.

See: https://review.openstack.org/455770/

Change-Id: I03f7f04be6c7889f94abed2d9e0a56d7e05ad90f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-02-15 16:53:21 -05:00
Paul Belanger
c857a38650 Switch to fedora-27 for nodepool dsvm
Fedora-27 is the latest supported version, switch to it. We also have
fedora-27 DIBs live in openstack-infra.

Change-Id: Iff9e01a8aab4ba50c42e5e72fabbe8cb20bc821c
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-02-15 08:41:03 +11:00
Paul Belanger
230c4ff792 Clean up old images for testing
Both fedora-25 and ubuntu-precise are nolonger tested. Remove dead
logic.

Change-Id: Ib54827bfea01cec39bd161a5a94fee350556102f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-02-15 08:41:03 +11:00
Zuul
41d2852824 Merge "Unpause Xenial build for non-src functional test" 2018-02-14 19:05:31 +00:00
Paul Belanger
5d580eeac6
Ensure only python3 is installed with bindep.txt
We no longer support python2.7 for nodepool, we can be more specific
and now only install python3 libraries.

Change-Id: I2a149a4b075fd63f382ed6b4a2e89f2bb8545da5
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-02-14 12:19:16 -05:00
Ian Wienand
a42a8746d9 Unpause Xenial build for non-src functional test
In I203d149a1d63ac8358e1c8b878d9c2bc0ba67c02 I forgot to unpause
Xenial for the non-src version of the functional testing.  This means
it effectively does nothing as all builds are paused.

Add it, along with a short note on the base job to remind you about
this.  Also slight layout change to keep all the functional-* jobs
together.

Change-Id: Ia78850e48fe232f380713fb0c9bef938099b7060
2018-02-14 10:21:05 +11:00
Ian Wienand
b1b1776101 Revert fixes for legacy boot jobs
These matches were only required for the legacy boot jobs, which are
now removed.

Change-Id: I2ab0d1172fb596d4f85481175cf754c17d406d12
Depends-On: https://review.openstack.org/543329
2018-02-13 19:05:55 +11:00
Ian Wienand
de24f25e58 Add native distro test jobs
This implements legacy-dsvm-nodepool-[distro]-src jobs as native jobs.
This seems like the best home for them, as they are run by multiple
externals such as glean and diskimage-builder.

Previously the defaults were set such that with no arguments, the
trusty test ran for nodepool-functional-py35.

To facilitate better templating, this turns all builds off by default.
The nodepool functional test is moved to a base template, and then
nodepool-functional-py35 explicitly builds and boots Xenial now
(trusty doesn't seem very useful).

The check_devstack_plugin.sh script runs after devstack, so needs to
source the stackrc file to pull in the variables about what images are
paused or not.

Additionally, the path for the script is fixed to nodepool so we can
run the job from other projects.

The redhat, ubuntu and suse legacy tests are re-implemented for their
respective builds.

This also highlighted that the opensuse test wasn't actually doing
anything.  This actually adds the configuration to build opensuse-423.

Needed-By: https://review.openstack.org/543270
Needed-By: https://review.openstack.org/543328
Needed-By: https://review.openstack.org/543329
Needed-By: https://review.openstack.org/543330
Change-Id: I203d149a1d63ac8358e1c8b878d9c2bc0ba67c02
2018-02-13 19:05:48 +11:00
Ian Wienand
d4f5a7bb9c Fix legacy dsvm jobs
The legacy-dsvm-nodepool-*-src jobs use this script, but don't pass
the argument added by I203d149a1d63ac8358e1c8b878d9c2bc0ba67c02.
Default it while we migrate everything.

Change-Id: I30943a3242d80ab87f6a89ec7f2bcfd3d62ad64b
2018-02-12 15:25:36 +11:00
Zuul
3bf920e9dd Merge "Store build logs automatically" 2018-02-09 17:04:26 +00:00
James E. Blair
baa831192f Store build logs automatically
This updates the builder to store individual build logs in dedicated
files, one per build, named for the image and build id.  Old logs are
automatically pruned.  By default, they are stored in
/var/log/nodepool/builds, but this can be changed.

This removes the need to specially configure logging handler for the
image build logs.

Change-Id: Ia7415d2fbbb320f8eddc4e46c3a055414df5f997
2018-02-09 07:50:20 -08:00
Zuul
a57a78657e Merge "Default min-ready to 0" 2018-02-09 06:53:27 +00:00
Zuul
d44bf47be3 Merge "Resolve paths before demonization" 2018-02-09 06:53:26 +00:00
Zuul
729406a965 Merge "Exercise pidfile before daemonizing" 2018-02-09 06:53:24 +00:00
Zuul
1039b0fca5 Merge "Exit launchers and builders immediately" 2018-02-09 06:53:23 +00:00
Zuul
8b7b3f3c86 Merge "Add default logging configuration" 2018-02-09 06:43:24 +00:00
James E. Blair
9f2230ab14 Default min-ready to 0
This is a more sensible default for most people, and friendlier
to new users.  Also, expand on the docs a bit.

Change-Id: I6f2c61ca423fba598c93fe5b26786dfa2711efb2
2018-02-08 17:06:43 -08:00
James E. Blair
8784158c75 Resolve paths before demonization
After daemonization, relative paths may be different, so resolve
all of the command line pathname arguments before then.

Change-Id: I2092c3f8e4595d8f874c769f41dacdf47554d681
2018-02-08 10:15:30 -08:00
James E. Blair
08409fb2fa Exercise pidfile before daemonizing
This is one of the few errors that can cause us to silently
fail to start.

Change-Id: Id5dbf737f55770d77e76fe94f72c2e39b9891dae
2018-02-08 10:15:21 -08:00
James E. Blair
4991bd5745 Exit launchers and builders immediately
Replace time.sleep with event.wait so that when the launcher is
stopped, it exits immediately.

The stop method of the main launcher process is altered to avoid
a race condition where it spawns a new poolworker while it is
shutting down.

Change-Id: Idbb6cc8f1e40ee2611cc73e27232f7db308a7230
2018-02-08 10:15:11 -08:00
Monty Taylor
0cb972b5b0
Update tox docs environment to match build-sphinx-docs
docs is a recommended developer convenience environment for doing the
sphinx build locally. The zuul job runs the sphinx-build command. Update
the docs env to do the same thing.

Move the job definitions in-tree - and also remove tox-docs.

Change-Id: Id810b8de1794b11b236eaa04940acbb754dd4b52
2018-02-07 17:55:58 -06:00
James E. Blair
782dca216c Add default logging configuration
Add a default logging configuration based on what was recently
added to Zuul.  This will use sane defaults for system logging,
though it does not yet address image build logging.

Change-Id: Ie86fc6d6839e2eb199c27515346b57b2ebede703
2018-02-07 15:49:19 -08:00
Zuul
e050908cf2 Merge "Convert from legacy to native devstack job" 2018-02-07 21:25:55 +00:00
Zuul
e359c6018b Merge "Add /node-list to the webapp" 2018-02-07 19:53:33 +00:00
David Shrewsbury
e054b74c52 Do not delete unused but allocated nodes
Let's assume an allocated node is *going* to be used (maybe zuul
hasn't got around to locking it quickly enough). We don't want
to assign a node, then delete it out from underneath the requestor.

Change-Id: I092cffbfd347684f66d6e7cbd0f910d40858580b
2018-02-06 12:01:54 -05:00
David Shrewsbury
f1f55e4638 Fix for age calculation on unused nodes
Setting the node state automatically changes the state_time
attribute for us, so our age calculation is always at 0 when
we go to log how old the node is at delete time. Fix it so
that we set state *after* logging the age.

Change-Id: I6e0510405f43842529c733110378b96decb88a07
2018-02-06 09:10:25 -05:00
Tristan Cacqueray
f2ad9ee0bc Add /node-list to the webapp
This change adds /node-list and /node-list.json to enable listing
current node list over http.

Change-Id: I256ebbba8692e78827a9104b08b433aafa26c71c
2018-02-06 13:21:08 +01:00
Zuul
0ff94fa98b Merge "Convert nodepool-zuul-functional job" 2018-02-06 11:53:47 +00:00
Andreas Jaeger
ed8fa74b8d Convert nodepool-zuul-functional job
Convert job to native Zuul v3 syntax.

Depends-On: https://review.openstack.org/540967
Change-Id: I1ab33417f3d1603163f280dab73778a293970f57
2018-02-05 20:29:57 +01:00
Zuul
b7efdb36f2 Merge "Refactor method node_list in status.py" 2018-02-04 12:42:30 +00:00
Zuul
4a4a61dcf0 Merge "Provider wedge fix" 2018-02-03 20:24:55 +00:00
David Shrewsbury
284e0b52bf Provider wedge fix
If a provider is going to pause due to insufficient quota, we attempt
to delete the oldest unused node (one that is READY and unlocked) to
guarantee that some quota will be freed.

To prevent us from issuing unnecessary delete requests, if we find a
node already being deleted within a recent timeframe (here, 5 minutes),
we do not issue another delete.

NOTE: This causes some assumptions in tests depending on the current
pause behavior (no proactive deletion) to no longer be valid, so steps
are taken in those tests to prevent the proactive delete from happening.

Change-Id: I8d94c60fe1ab184503592a02d6ca458f94a2ea3d
2018-02-02 16:42:13 -05:00
Zuul
e3af264950 Merge "Handle missing request during a decline." 2018-02-02 18:51:33 +00:00
Zuul
5a2f5cf804 Merge "Invalidate flavor and image cache on 400 errors" 2018-02-02 17:47:20 +00:00
David Shrewsbury
e3c952ef3a Handle missing request during a decline.
We've seen a handler continually decline a request because the
request went missing and we didn't handle that exception, so we
never set the handler to 'done'. This handles that exception,
and adds a safety mechanism to decline_request().

Change-Id: Ieb06846e0fe2aadfd9df0031e58b2df98bbfcc03
2018-02-02 12:46:49 -05:00
ChangBo Guo(gcb)
13ac2512ba Refactor method node_list in status.py
Extract same logics into one internal function, and call it
when get one node's values or iterate some nodes' value.

Change-Id: Idd536931a40f2369d3bfc4c623e94844c95f78a0
2018-02-02 17:41:48 +08:00
ChangBo Guo(gcb)
e9b29bab37 update supported python version in setup.cfg
Nodepool requires Python 3.5 or newer, so need
update the information for pypi.

Change-Id: I980116039fcd221fcb7bdfdd12dfbeeda8cbbe2f
2018-02-02 04:30:06 +00:00