The missing enter is resulting in the entire section rendered
with monospace font and the attr not properly resolving.
Change-Id: I657c62d572526d8cd06299c6d349806c4581677b
The labels section was mis-spaced therefore the documentation was
pointing that labels was a root configuration option for the provider
when it lives inside the pools section in reality.
Change-Id: If757cc23c694927bb6402d42eed171ad23b1e87e
This change allows you to specify a dib-cmd parameter for disk images,
which overrides the default call to "disk-image-create". This allows
you to essentially decide the disk-image-create binary to be called
for each disk image configured.
It is inspired by a couple of things:
The "--fake" argument to nodepool-builder has always been a bit of a
wart; a case of testing-only functionality leaking across into the
production code. It would be clearer if the tests used exposed
methods to configure themselves to use the fake builder.
Because disk-image-create is called from the $PATH, it makes it more
difficult to use nodepool from a virtualenv. You can not just run
"nodepool-builder"; you have to ". activate" the virtualenv before
running the daemon so that the path is set to find the virtualenv
disk-image-create.
In addressing activation issues by automatically choosing the
in-virtualenv binary in Ie0e24fa67b948a294aa46f8164b077c8670b4025, it
was pointed out that others are already using wrappers in various ways
where preferring the co-installed virtualenv version would break.
With this, such users can ensure they call the "disk-image-create"
binary they want. We can then make a change to prefer the
co-installed version without fear of breaking.
In theory, there's no reason why a totally separate
"/custom/venv/bin/disk-image-create" would not be valid if you
required a customised dib for some reason for just one image. This is
not currently possible, even modulo PATH hacks, etc., all images will
use the same binary to build. It is for this flexibility I think this
is best at the diskimage level, rather than as, say a global setting
for the whole builder instance.
Thus add a dib-cmd option for diskimages. In the testing case, this
points to the fake-image-create script, and the --fake command-line
option and related bits are removed.
It should have no backwards compatibility effects; documentation and a
release note is added.
Change-Id: I6677e11823df72f8c69973c83039a987b67eb2af
When providing multiple static labels, user should create multiple pool
to avoid requests being queued.
Change-Id: I0fdca7f1409d80d82f00094c7eb889f7d92a8d59
This change modifies the static driver registering logic to take
into account the username and connection-port when identifying
unique node so that a single hostname can be registered multiple
times with different user or port.
Change-Id: I8f770e13c0d00d65094abccee30e54ec0e6371b6
This adds the ability for a pool.label to override the
pool.host-key-checking value, while having a label exist in the same
pool. This is helpful because it is possible for 1 pool to mix network
configuration, and some nodes maybe missing a default gateway (making
them unroutable by default).
Change-Id: I934d42b8e48aedb0ebc03137b7305eb2af390fc7
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Nodepool's OpenStack driver relies on openstacksdk directly for
configuration handling now, no longer os-client-config. Update the
references in documentation and code comments accordingly.
Change-Id: I8850bce4a226aa550038f02cdb027b9613ce0ae2
This change implements a single project OpenShift pod provider usable by a
regular user service account, without the need for a self-provisioner role.
Change-Id: I84e4bdda64716f9dd803eaa89e576c26a1667809
This change adds a new python_path Node attribute so that zuul executor
can remove the default hard-coded ansible_python_interpreter.
Change-Id: Iddf2cc6b2df579636ec39b091edcfe85a4a4ed10
We have a use case where we have a single pool, due to quota reasons,
but need the ability to selectively choose which network a label will
use. Now a nodepool operator will be able to define which networks are
attached to labels (in our case network appliances).
Change-Id: I3bfa32473c76b9fd59deee7d05b492e7cf67f69d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Support for concurrency and rate limiting has been added to keystoneauth,
which is the library openstacksdk uses to talk to OpenStack. Instead
of managing concurrency in nodepool using the TaskManager and pool of
worker threads, let keystoneauth take over. This also means we no longer
have a hook into the request process, so we defer statsd reporting to
the openstacksdk layer as well.
Change-Id: If21a10c56f43a121d30aa802f2c89d31df97f121
The docs for provider removal were a tad inaccurate and/or misleading.
This hopefully clarifies the procedure.
Change-Id: I8d4d88c45dc3cea3465e5bf508d83fd940e5fdec
Insert `nodes` level for driver `static`; it was missing.
Now it's `pools.[static].nodes.labels` etc.
The previous `pools.[static].labels` was failing a config.
Indent all the node's attributes.
Move `username` and `host-key` for readability.
Change-Id: Ia61fd73884f77f1d96f36038c1cd74eb37016f5c
This updates dib stats after creating a dashboard to use them.
Firstly, the individual return codes and runtime for each image type
are unnecessary, because they call come from the same invocation of
dib. While it is definitely useful to track the size of each output
image, the overall status for a build is only a single value. This
moves these duplciated values to ".status.<rc|duration>".
Unfortunately, there's really no way to say "what was the time of the
last non-null value" in grafana+graphite [1]. This means you can't do
something useful like show a singlestat of the relative time of the
last build "X hours ago" using the timer value. We can work around
this by putting the timestamp of the last build in a gauge value; this
monotonically increases and is easy to turn into a relative time.
[1] https://github.com/grafana/grafana/issues/10550
Change-Id: Ia9518b6faecb30d45e0509bda4a9b2ab7fdc6261
I noticed in OpenStack production we don't seem to be getting all the
stats from dib, particularly from our very remote builder. This is
likely because there is some packet loss quickly blasting out small
UDP packets with the stats. A pipeline bundles the stats together
into the largest size packets it can (this has been a problem before;
see I3f68450c7164d1cf0f1f57f9a31e5dca2f72bc43).
Add some additional checks for the size stats which did not seem to be
covered by existing testing.
I also noticed that the documentation had an extra ".builder." in the
key which isn't actually there in the stats output.
Change-Id: Ib744f19385906d1e72231958d11c98f15b72d6bd
The AWS term is instance-type, not flavor-name. Rename this while
we don't have a huge userbase.
While we're in there, rename a variable from image_name to image_id
since we use image_id everywhere else.
Change-Id: I1f7f16d2873982626d2434cf5ca1f6280adf739c
This reverts commit ccf40a462a.
The previous version would not work properly when daemonized
because there was no stdout. This version maintains stdout and
uses select/poll with non-blocking stdout to capture the output
to a log file.
Depends-On: https://review.openstack.org/634266
Change-Id: I7f0617b91e071294fe6051d14475ead1d7df56b7
A builder thread can wedge if the build process wedges. Add a timeout
to the subprocess. Since it was the call to readline() that would block,
we change the process to have DIB write directly to the log. This allows
us to set a timeout in the Popen.wait() call. And we kill the dib
subprocess, as well.
The timeout value can be controlled in the diskimage configuration and
defaults to 8 hours.
Change-Id: I188e8a74dc39b55a4b50ade5c1a96832fea76a7d
This reverts commit 7e1b8a7261.
openstacksdk >=0.19.0 fixes the filtering problems leading to all
ports being deleted. However openstacksdk <0.21.0 has problems with
dogpile.cache so use 0.21.0 as a minimum.
Change-Id: Id642d074cbb645ced5342dda4a1c89987c91a8fc
This change implements an OpenShift resource provider. The driver currently
supports project request and pod request to enable both containers as machine
and native containers workflow.
Depends-On: https://review.openstack.org/608610
Change-Id: Id3770f2b22b80c2e3666b9ae5e1b2fc8092ed67c
This config option, available under each provider pool section, can
contain static key-value pairs that will be stored in ZooKeeper on
each Node znode. This will allow us to pass along abitrary data from
nodepool to any user of nodepool (specifically, zuul). Initially, this
will be used to pass along zone information to zuul executors.
Change-Id: I126d37a8c0a4f44dca59c11f76a583b9181ab653
A new version of the alabaster theme changed how the logo is handled.
Correct that and add previous/next links. This realigns the docs
with Zuul.
Change-Id: Icd19284e4d3c054a8ad4eeed42e1d93ca87f86bd
We're trying to convey a hierarchy of options which may change
depending on what options are selected. Given the config:
providers:
- driver: openstack
labels: [foo]
Rather than naming the option "providers.openstack.labels", name
it "providers.[openstack].labels" to indicate that the "openstack"
part of the option is not a normal name, but metasyntactic. Use
square brackets to suggest it's an item in a list.
Also some minor style updates, and add 'kubernetes' to the list
of driver values.
Change-Id: I33db936475eaac207b722a9dbfcd8abe99cfc7ed
Switch to the correct underline styles for the Kubernetes
configuration document sections so the driver entry appears at the
same heading level as other drivers in the ToC. Also use title
capitalization on the subsection names.
Change-Id: Ibba997102ea61950bae33a3382f9a1543eddf5c6
This changes implements a Kubernetes resource provider.
The driver supports namespace request and pod request to enable both
containers as machine and native containers workflow.
Depends-On: https://review.openstack.org/605823
Change-Id: I81b5dc5abe92b71cc98b0d71c8a2863cddff6027
This uses the attr-overview directive in the dependent change to add
an overview of the configuration options for each section.
Change-Id: I70c741c8192c7151af89e0902d6eca96da1c4891
Depends-On: https://review.openstack.org/604980
This moves the configuration documentation to a hierarchical layout
using the attr directives provided by zuul-sphinx.
Apart from making it look like the zuul documentation, this brings
consistency to things like required flags, default values and typing
info.
There are no content changes but things have moved around somewhat to
accommodate the layout.
Depends-On: https://review.openstack.org/604267
Change-Id: I831dfd8c9458a1f255aa05fa96cfc5c416ed3310
We currently have keys like "ComputePostOs-volumes_boot" for providers
using boot-from-volume and other various "os-" keys depending on the
provider. Normalise all these to regular CamelCase. A basic
test-case is added.
Additionally add some documentation on the API call stats, pointing
out they reflect internal details so are subject to change. A release
note is added for the updated stats.
Change-Id: If8398906a5a7ad3d96e985263b0c841e5dcaf7b5
Mohammed Naser