Commit Graph

272 Commits (5c605b32400b4e45315f8654bb1d221d47ff4ce3)

Author SHA1 Message Date
Mohammed Naser f2a80ef185 Fix chroot type
The chroot is of type string, not int.  This patch updates the
documentation accordingly.

Change-Id: I35ca817013affeedd5aeaae661aa477e5972444b
4 years ago
Mohammed Naser aadff75476 Add extra spacing to avoid monospace rendering
The missing enter is resulting in the entire section rendered
with monospace font and the attr not properly resolving.

Change-Id: I657c62d572526d8cd06299c6d349806c4581677b
4 years ago
Mohammed Naser dcddbfaf9a Fix Kubernetes driver documentation
The labels section was mis-spaced therefore the documentation was
pointing that labels was a root configuration option for the provider
when it lives inside the pools section in reality.

Change-Id: If757cc23c694927bb6402d42eed171ad23b1e87e
4 years ago
Zuul 8efd70d0a8 Merge "Add a dib-cmd option for diskimages" 4 years ago
Ian Wienand 9367cf8ed8 Add a dib-cmd option for diskimages
This change allows you to specify a dib-cmd parameter for disk images,
which overrides the default call to "disk-image-create".  This allows
you to essentially decide the disk-image-create binary to be called
for each disk image configured.

It is inspired by a couple of things:

The "--fake" argument to nodepool-builder has always been a bit of a
wart; a case of testing-only functionality leaking across into the
production code.  It would be clearer if the tests used exposed
methods to configure themselves to use the fake builder.

Because disk-image-create is called from the $PATH, it makes it more
difficult to use nodepool from a virtualenv.  You can not just run
"nodepool-builder"; you have to ". activate" the virtualenv before
running the daemon so that the path is set to find the virtualenv

In addressing activation issues by automatically choosing the
in-virtualenv binary in Ie0e24fa67b948a294aa46f8164b077c8670b4025, it
was pointed out that others are already using wrappers in various ways
where preferring the co-installed virtualenv version would break.

With this, such users can ensure they call the "disk-image-create"
binary they want.  We can then make a change to prefer the
co-installed version without fear of breaking.

In theory, there's no reason why a totally separate
"/custom/venv/bin/disk-image-create" would not be valid if you
required a customised dib for some reason for just one image.  This is
not currently possible, even modulo PATH hacks, etc., all images will
use the same binary to build.  It is for this flexibility I think this
is best at the diskimage level, rather than as, say a global setting
for the whole builder instance.

Thus add a dib-cmd option for diskimages.  In the testing case, this
points to the fake-image-create script, and the --fake command-line
option and related bits are removed.

It should have no backwards compatibility effects; documentation and a
release note is added.

Change-Id: I6677e11823df72f8c69973c83039a987b67eb2af
4 years ago
Tristan Cacqueray 9694382c47 static: add host-key-checking toggle
This change adds a static node toggle to disable checkHost and

Change-Id: If3aefb81624b573faf1129bb82d46ee848ff27cf
4 years ago
Zuul 861288d41e Merge "Update documentation for k8s in-cluster config" 4 years ago
James E. Blair 676b6b07bd Update documentation for k8s in-cluster config
Update the docs and add a release note to let folks know that
in-cluster config is an option now.

Change-Id: Ifb4a575fb7e9b6a89e14f00a2e1cccee6fdabf72
4 years ago
Tristan Cacqueray 2436355ed1 static: document multiple labels shortcomming
When providing multiple static labels, user should create multiple pool
to avoid requests being queued.

Change-Id: I0fdca7f1409d80d82f00094c7eb889f7d92a8d59
4 years ago
Zuul a7dc2770b7 Merge "static: enable using a single host with different user or port" 4 years ago
Tristan Cacqueray 552140037f static: enable using a single host with different user or port
This change modifies the static driver registering logic to take
into account the username and connection-port when identifying
unique node so that a single hostname can be registered multiple
times with different user or port.

Change-Id: I8f770e13c0d00d65094abccee30e54ec0e6371b6
4 years ago
Zuul 210570cd49 Merge "openstack: document the key-name parameter" 4 years ago
Paul Belanger 3a5cabedcb Toggle host-key-checking for openstack provider.labels
This adds the ability for a pool.label to override the value, while having a label exist in the same
pool.  This is helpful because it is possible for 1 pool to mix network
configuration, and some nodes maybe missing a default gateway (making
them unroutable by default).

Change-Id: I934d42b8e48aedb0ebc03137b7305eb2af390fc7
Signed-off-by: Paul Belanger <>
4 years ago
Jeremy Stanley 74f960848c Update os-client-config references to openstacksdk
Nodepool's OpenStack driver relies on openstacksdk directly for
configuration handling now, no longer os-client-config. Update the
references in documentation and code comments accordingly.

Change-Id: I8850bce4a226aa550038f02cdb027b9613ce0ae2
4 years ago
Tristan Cacqueray 3fa7a45905 openstack: document the key-name parameter
Change-Id: I1b469dd8897033384dc3c59dbc6f058a9dd9b1bf
4 years ago
Tristan Cacqueray 159038503a Implement an OpenShift Pod provider
This change implements a single project OpenShift pod provider usable by a
regular user service account, without the need for a self-provisioner role.

Change-Id: I84e4bdda64716f9dd803eaa89e576c26a1667809
4 years ago
Tristan Cacqueray 76aa62230c Add python-path option to node
This change adds a new python_path Node attribute so that zuul executor
can remove the default hard-coded ansible_python_interpreter.

Change-Id: Iddf2cc6b2df579636ec39b091edcfe85a4a4ed10
4 years ago
Paul Belanger aaf36db8c6 Allow openstack provider labels to configure networks
We have a use case where we have a single pool, due to quota reasons,
but need the ability to selectively choose which network a label will
use. Now a nodepool operator will be able to define which networks are
attached to labels (in our case network appliances).

Change-Id: I3bfa32473c76b9fd59deee7d05b492e7cf67f69d
Signed-off-by: Paul Belanger <>
4 years ago
Zuul 2f66863db4 Merge "doc: add provider driver development instructions" 4 years ago
Zuul ba8dd4a354 Merge "Update dib stats" 4 years ago
Monty Taylor 34aae137fa Remove TaskManager and just use keystoneauth
Support for concurrency and rate limiting has been added to keystoneauth,
which is the library openstacksdk uses to talk to OpenStack. Instead
of managing concurrency in nodepool using the TaskManager and pool of
worker threads, let keystoneauth take over. This also means we no longer
have a hook into the request process, so we defer statsd reporting to
the openstacksdk layer as well.

Change-Id: If21a10c56f43a121d30aa802f2c89d31df97f121
4 years ago
David Shrewsbury 6c2c1d3aac Update docs for provider removal.
The docs for provider removal were a tad inaccurate and/or misleading.
This hopefully clarifies the procedure.

Change-Id: I8d4d88c45dc3cea3465e5bf508d83fd940e5fdec
4 years ago
Tristan Cacqueray 1ff1a253b5 doc: add provider driver development instructions
Change-Id: I31642883f2bfb99e30820ddf72be3feedb66943e
4 years ago
Zuul ff186a53d3 Merge "Use a pipeline for dib stats" 4 years ago
jabielecki c4f4279cb3 doc bugfix for static provider
Insert `nodes` level for driver `static`; it was missing.
Now it's `pools.[static].nodes.labels` etc.
The previous `pools.[static].labels` was failing a config.

Indent all the node's attributes.

Move `username` and `host-key` for readability.

Change-Id: Ia61fd73884f77f1d96f36038c1cd74eb37016f5c
4 years ago
Ian Wienand 6fa73eac26 Update dib stats
This updates dib stats after creating a dashboard to use them.

Firstly, the individual return codes and runtime for each image type
are unnecessary, because they call come from the same invocation of
dib.  While it is definitely useful to track the size of each output
image, the overall status for a build is only a single value.  This
moves these duplciated values to ".status.<rc|duration>".

Unfortunately, there's really no way to say "what was the time of the
last non-null value" in grafana+graphite [1].  This means you can't do
something useful like show a singlestat of the relative time of the
last build "X hours ago" using the timer value.  We can work around
this by putting the timestamp of the last build in a gauge value; this
monotonically increases and is easy to turn into a relative time.


Change-Id: Ia9518b6faecb30d45e0509bda4a9b2ab7fdc6261
4 years ago
Ian Wienand c68dbb9636 Use a pipeline for dib stats
I noticed in OpenStack production we don't seem to be getting all the
stats from dib, particularly from our very remote builder.  This is
likely because there is some packet loss quickly blasting out small
UDP packets with the stats.  A pipeline bundles the stats together
into the largest size packets it can (this has been a problem before;
see I3f68450c7164d1cf0f1f57f9a31e5dca2f72bc43).

Add some additional checks for the size stats which did not seem to be
covered by existing testing.

I also noticed that the documentation had an extra ".builder." in the
key which isn't actually there in the stats output.

Change-Id: Ib744f19385906d1e72231958d11c98f15b72d6bd
4 years ago
Monty Taylor 516e8cd176 Rename aws flavor-name to instance-type
The AWS term is instance-type, not flavor-name. Rename this while
we don't have a huge userbase.

While we're in there, rename a variable from image_name to image_id
since we use image_id everywhere else.

Change-Id: I1f7f16d2873982626d2434cf5ca1f6280adf739c
4 years ago
Clint Byrum 6a4a8f80db Add documentation for AWS/EC2 driver
a thing which illuminates,
the cloud, and nodepool.

Change-Id: I98ec01d188a3ee18175fe3ebc97d89668ec81a20
4 years ago
David Shrewsbury 890ea4975e Revert "Revert "Add a timeout for the image build""
This reverts commit ccf40a462a.

The previous version would not work properly when daemonized
because there was no stdout. This version maintains stdout and
uses select/poll with non-blocking stdout to capture the output
to a log file.


Change-Id: I7f0617b91e071294fe6051d14475ead1d7df56b7
4 years ago
Zuul f2c155821c Merge "Revert "Add a timeout for the image build"" 4 years ago
David Shrewsbury ccf40a462a Revert "Add a timeout for the image build"
This reverts commit 7225354ec0.

The disk-image-create command does not appear to be starting.

Change-Id: I81abe25a253a385cae08a57561129a678546f18f
4 years ago
Zuul 78d2476769 Merge "Revert "Revert "Cleanup down ports""" 4 years ago
Zuul 26c57ee5a9 Merge "Add a timeout for the image build" 4 years ago
David Shrewsbury 7225354ec0 Add a timeout for the image build
A builder thread can wedge if the build process wedges. Add a timeout
to the subprocess. Since it was the call to readline() that would block,
we change the process to have DIB write directly to the log. This allows
us to set a timeout in the Popen.wait() call. And we kill the dib
subprocess, as well.

The timeout value can be controlled in the diskimage configuration and
defaults to 8 hours.

Change-Id: I188e8a74dc39b55a4b50ade5c1a96832fea76a7d
4 years ago
Zuul 44ae87c310 Merge "Support userdata for instances in openstack" 4 years ago
Sagi Shnaidman d5027ff6a9 Support userdata for instances in openstack
Use "userdata" from Nova API to pass cloud-init config to nova
instances in openstack.

Change-Id: I1c6a1cbc5377d268901210631a376ca26f4887d8
4 years ago
Ian Wienand 0cf8144e8c
Revert "Revert "Cleanup down ports""
This reverts commit 7e1b8a7261.

openstacksdk >=0.19.0 fixes the filtering problems leading to all
ports being deleted. However openstacksdk <0.21.0 has problems with
dogpile.cache so use 0.21.0 as a minimum.

Change-Id: Id642d074cbb645ced5342dda4a1c89987c91a8fc
4 years ago
Tristan Cacqueray 301101d06c Fix OpenShift driver typo
This is a follow-up to fix typo introduced by:

Change-Id: I7178860f2b7a47cf58e96ee1d2ea32ceeeab5a18
4 years ago
Tristan Cacqueray c1378c4407 Implement an OpenShift resource provider
This change implements an OpenShift resource provider. The driver currently
supports project request and pod request to enable both containers as machine
and native containers workflow.

Change-Id: Id3770f2b22b80c2e3666b9ae5e1b2fc8092ed67c
4 years ago
David Shrewsbury 16325d5c4c Add arbitrary node attributes config option
This config option, available under each provider pool section, can
contain static key-value pairs that will be stored in ZooKeeper on
each Node znode. This will allow us to pass along abitrary data from
nodepool to any user of nodepool (specifically, zuul). Initially, this
will be used to pass along zone information to zuul executors.

Change-Id: I126d37a8c0a4f44dca59c11f76a583b9181ab653
5 years ago
James E. Blair dd551cda96 Normalize sidebar in docs
A new version of the alabaster theme changed how the logo is handled.
Correct that and add previous/next links.  This realigns the docs
with Zuul.

Change-Id: Icd19284e4d3c054a8ad4eeed42e1d93ca87f86bd
5 years ago
James E. Blair 744fb736bd Alter the metasyntactic variable in driver docs
We're trying to convey a hierarchy of options which may change
depending on what options are selected.  Given the config:

  - driver: openstack
    labels: [foo]

Rather than naming the option "providers.openstack.labels", name
it "providers.[openstack].labels" to indicate that the "openstack"
part of the option is not a normal name, but metasyntactic.  Use
square brackets to suggest it's an item in a list.

Also some minor style updates, and add 'kubernetes' to the list
of driver values.

Change-Id: I33db936475eaac207b722a9dbfcd8abe99cfc7ed
5 years ago
James E. Blair b1e383d127 Convert kubernetes config docs to zuul-sphinx
The rest of the documentation switched to this format while the
k8s driver was in development.

Change-Id: If39da0a0d51d00f289f6c5efbe9c25f7b07fdf40
5 years ago
Jeremy Stanley e0ebaa9799 Correct heading levels for Kubernetes config docs
Switch to the correct underline styles for the Kubernetes
configuration document sections so the driver entry appears at the
same heading level as other drivers in the ToC. Also use title
capitalization on the subsection names.

Change-Id: Ibba997102ea61950bae33a3382f9a1543eddf5c6
5 years ago
Tristan Cacqueray 4295ff6870 Implement a Kubernetes driver
This changes implements a Kubernetes resource provider.
The driver supports namespace request and pod request to enable both
containers as machine and native containers workflow.

Change-Id: I81b5dc5abe92b71cc98b0d71c8a2863cddff6027
5 years ago
Zuul 059606b9f3 Merge "Normalise more of the API stats calls" 5 years ago
Ian Wienand fc8827b55b Add overview of config options
This uses the attr-overview directive in the dependent change to add
an overview of the configuration options for each section.

Change-Id: I70c741c8192c7151af89e0902d6eca96da1c4891
5 years ago
Ian Wienand 8d54917488 Use zuul-sphinx for configuration layout
This moves the configuration documentation to a hierarchical layout
using the attr directives provided by zuul-sphinx.

Apart from making it look like the zuul documentation, this brings
consistency to things like required flags, default values and typing

There are no content changes but things have moved around somewhat to
accommodate the layout.


Change-Id: I831dfd8c9458a1f255aa05fa96cfc5c416ed3310
5 years ago
Ian Wienand f18e2e8c76 Normalise more of the API stats calls
We currently have keys like "ComputePostOs-volumes_boot" for providers
using boot-from-volume and other various "os-" keys depending on the
provider.  Normalise all these to regular CamelCase.  A basic
test-case is added.

Additionally add some documentation on the API call stats, pointing
out they reflect internal details so are subject to change.  A release
note is added for the updated stats.

Change-Id: If8398906a5a7ad3d96e985263b0c841e5dcaf7b5
5 years ago