36 Commits

Author SHA1 Message Date
Dr. Jens Harbott
5aa792f1ae Start booting bookworm nodes
Image builds have been successful

Change-Id: If286eb3e1a75c643f67f3d6d3d7e2d31c205ac1b
2023-07-03 18:47:46 +02:00
Jeremy Stanley
8f916dc736 Restore rax-ord quota but lower max-concurrency
Looking at our graphs, we're still spiking up into the 30-60
concurrent building range at times, which seems to result in some
launches exceeding the already lengthy timeout and wasting quota,
but when things do manage to boot we effectively utilize most of
max-servers nicely. The variability is because max-concurrency is
the maximum number of in-flight node requests the launcher will
accept for a provider, but the number of nodes in a request can be
quite large sometimes.

Raise max-servers back to its earlier value reflecting our available
quota in this provider, but halve the max-concurrency so we don't
try to boot so many at a time.

Change-Id: I683cdf92edeacd7ccf7b550c5bf906e75dfc90e8
2023-03-16 19:53:55 +00:00
Jeremy Stanley
d0481326bf Limit rax-ord launch concurrency and don't retry
This region seems to take a very long time to launch nodes when we
have a burst of requests for them, like a thundering herd sort of
behavior causing launch times to increase substantially. We have a
lot of capacity in this region though, so want to boot as many
instances as we can here. Attempt to reduce the effect by limiting
the number of instances nodepool will launch at the same time.

Also, mitigate the higher timeout for this provider by not retrying
launch failures, so that we won't ever lock a request for multiples
of the timeout.

Change-Id: I179ab22df37b2f996288820074ec69b8e0a202a5
2023-03-10 18:09:33 +00:00
Jeremy Stanley
bc7d946ca2 Wait longer for rax-ord nodes and ease up API rate
We're still seeing a lot of timeouts waiting for instances to become
active in this provider, and are observing fairly long delays
between API calls at times. Increase the launch wait from 10 to 15
minutes, and increase the minimum delay between API calls by an
order of magnitude from 0.001 to 0.01 seconds.

Change-Id: Ib13ff03629481009a838a581d98d50accbf81de2
2023-03-08 14:39:38 +00:00
Jeremy Stanley
6f5c773b6e Try halving max-servers for rax-ord region
Reduce the max-servers in rax-ord from 195 to 100, and revert the
boot-timeout from the 300 we tried back down to 120 like the others.
We're continuing to see server create calls taking longer to report
active than nodepool is willing to wait, but also may be witnessing
the results of API rate limiting or systemic slowness. Reducing the
number of instances we attempt to boot there may give us a clearer
picture of whether that's the case.

Change-Id: Ife7035ba64b457d964c8497da0d9872e41769123
2023-03-07 18:39:00 +00:00
Jeremy Stanley
a177d641f2 Increase boot-timeout for rax-ord
For a while we've been seeing a lot of "Timeout waiting for instance
creation" in Rackspace's ORD region, but checking behind the
launcher it appears these instances do eventually boot, so we're
wasting significant resources discarding quota we never use.
Increase the timeout for this from 2 minutes to 5, but only in this
region as 2 minutes appears to be sufficient in the others.

Change-Id: I1cf91a606eefc4aa65507f491a20182770b99f09
2023-03-06 16:56:45 +00:00
Jeremy Stanley
b597a94b2e Add missing fedora-36 label to nl01
This seems to have been overlooked when the label was added to other
launchers, and is contributing to NODE_FAILURE results for some
jobs, particularly now that fedora-latest is relying on it.

Change-Id: Ifc0e5452ac0cf275463f6f1cfbe0d7fe350e3323
2022-09-19 15:05:52 +00:00
Ian Wienand
36b9c302e5 nodepool: Add Fedora 36
Add Fedora 36 builds

Change-Id: I64fac34945ea5c6ec91ddd442281fcaba2c53271
2022-08-22 11:25:09 +10:00
Neil Hanlon
705e8420a2
Add rockylinux 9 to nodepool
Change-Id: Iedf1c8eb2898cfc5771a5a695a53a39f9396edc9
2022-08-05 08:58:27 -04:00
wangxiyuan
30fca1376d Bump openEuler to 22.03 LTS
openEuler 20.03-LTS-SP2 was out of date in May 2022. 22.03 LTS
is the newest LTS version. It was release in March 2022 and
will be maintained for 2 years. This patch upgrades the LTS
version. It'll be used in Devstack, Kolla-ansible and so on
in CI jobs.

Change-Id: I23f2b397bc7f1d8c2a959e0e90f5058cf3bf104d
2022-08-03 14:40:34 +08:00
Dr. Jens Harbott
1bdccd42e5 Start launching Jammy images
The first image was built successfully, so we can start launching them.

Change-Id: Ie84d1700b6f4f7696e14dfe01bc887e422163d7e
2022-04-26 13:53:29 +02:00
Neil Hanlon
2ccc5241c8
Add rockylinux-8 to nodepool configuration
Signed-off-by: Neil Hanlon <neil@rockylinux.org>
Change-Id: Ic3344bc47ca56c27f7ec3427a0865bd6ce3349d3
Depends-On: https://review.opendev.org/c/openstack/project-config/+/829405
Depends-On: https://review.opendev.org/c/openstack/project-config/+/829712
2022-02-17 09:12:00 -05:00
Ian Wienand
dd58c496f8 Remove Fedora 34
The dependent changes should be the last references to Fedora 34 nodes

Depends-On: https://review.opendev.org/c/openstack/devstack/+/827576
Depends-On: https://review.opendev.org/c/openstack/devstack/+/827578
Depends-On: https://review.opendev.org/c/zuul/nodepool/+/827577

Change-Id: Ie14ea374808e5518588925de3a476f0bc6ff2ccf
2022-02-11 07:55:21 +11:00
Clark Boylan
dce378a6b4 Remove centos-8
This distro release reached its EOL December 31, 2021. We are removing
it from our CI system as people should really stop testing on it. They
can use CentOS 8 Stream or other alternatives instead.

Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/827181
Change-Id: I13e8185b7839371a9f9043b715dc39c6baf907d5
2022-02-02 09:48:36 +11:00
Clark Boylan
58e4543789 Set centos-8 min-ready to 0
This is in preparation for removing this label. This distro is no longer
supported and users will need to find alternatives.

Change-Id: I57b363671809afe415a376b0894041438140bdae
2022-01-31 13:31:24 -08:00
Clark Boylan
07e9803134 Remove opensuse-tumbleweed from nodepool
This removes the label, nodes, and images for opensuse-tumbleweed across
our cloud providers. We also update grafana to stop graphing stats for
the label.

Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/824068
Change-Id: Ic311af5d667c01c1845251270fd2fdda7d99ebcb
2022-01-10 14:02:55 -08:00
Clark Boylan
05afaaa9ea Set tumbleweed min-ready to 0
This is in preparation to remove the image and label entirely. Nothing
seems to use the image so clean it up.

Change-Id: I5ab3a0627874e302289deb442f80a782509df2c3
2022-01-10 14:02:55 -08:00
Ian Wienand
dc65009eaf nodepool: Remove . from openEuler name
We have a bug uploading images with "." in the name.  Work around this
for now by avoid it.

Change-Id: I20e1a926d02a632450b8114d84a0fa738b7ec639
2021-12-17 07:39:45 +11:00
wangxiyuan
d55da5dac0 Add openEuler 20.03 LTS SP2 node
openEuler diskimage element has been added[1]. This Patch plans
to build it in nodepool in order to support openEuler nodes in
OpenStack CI system.

This kind of nodes will be used as devstack CI job at least[2].

[1]: https://review.opendev.org/c/openstack/diskimage-builder/+/784363
[2]: https://review.opendev.org/c/openstack/devstack/+/760790

Change-Id: I05312e296eed2854c5fc45c94038d570028d08f9
2021-12-03 11:44:18 +08:00
Zuul
d7f8394115 Merge "Set Fedora 34 min-ready to zero" 2021-11-18 07:37:23 +00:00
Zuul
4a33b66493 Merge "Add Fedora 35" 2021-11-14 22:19:52 +00:00
Ian Wienand
b48f6b8f70 Remove debian-stretch nodes and builds
Change-Id: I62783d6180eddfff3c0e8014011e6bb90c819ed4
2021-11-10 15:10:06 +11:00
Ian Wienand
0cfaf9e476 Set debian-stretch to min-ready: 0
There should be no more references to these nodes

Change-Id: I5ccc7f07f91d64b485e3dd81be89ae60d243cfb1
2021-11-10 15:07:05 +11:00
Ian Wienand
5342520345 Set Fedora 34 min-ready to zero
In preparation for removing the nodes

Change-Id: Iae17bb366b02a665f10fcad7f2fcaa4c02044def
2021-11-08 15:20:17 +11:00
Ian Wienand
f8d93ea149 Add Fedora 35
DIB with support is deployed to builder nodes

Change-Id: I6828383643aee1d6dadc71dfd6c25ba91a1bf561
2021-11-08 15:20:13 +11:00
Alfredo Moralejo
5100cf7063 Add centos-9-stream nodepool image
CentOS Stream 9 repos are ready and support for it is included in
diskimage-builder [1]. This patch is adding centos-9-stream diskimages
and images to nodepool configuration in opendev.

[1] https://review.opendev.org/c/openstack/diskimage-builder/+/811392

Change-Id: I9b88baf422d947d5209d036766a86b09dca0c21a
2021-11-03 12:48:50 +01:00
Mohammed Naser
980f3ba06a Remove fedora-32 disk image config
This is a followup change to the previous change which removed fedora-32
node launches. Here we cleanup the disk image configs

Change-Id: I459ec47735550e4adcc912bd707836582223b075
2021-06-09 14:29:58 -04:00
Mohammed Naser
725135a002 Stop launch fedora-32 nodes nodepool
A followup change will remove the diskimage configuration for fedora-32.
The current fedora releases are 33 and 34 so we should clean up 32.

Depends-On: https://review.opendev.org/c/zuul/zuul-jobs/+/795636
Depends-On: https://review.opendev.org/c/zuul/zuul-operator/+/795638
Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/795639
Depends-On: https://review.opendev.org/c/zuul/nodepool/+/795642
Depends-On: https://review.opendev.org/c/x/tobiko/+/795641
Depends-On: https://review.opendev.org/c/openstack/devstack/+/795640
Change-Id: I4f8d0e0a1aee55c0b4c92bb11690045018f63bf0
2021-06-09 14:29:19 -04:00
Mohammed Naser
5dc75c5944 Build images for Fedora 34
This patch adds all the images to build Fedora 34 using the new
container element.

Depends-On: https://review.opendev.org/c/opendev/system-config/+/795602
Change-Id: Ifc650adbcb5ae6ed1f7ce50a4de58004ef020acc
2021-06-09 13:35:39 -04:00
Dmitriy Rabotyagov
4ac828fb7b Add Debian Bullseye nodepool images and wheels
Since Debian Buster can not be used with nova 23.0.0 because of the
min required libvirt version, we should make Bullseye available for CI
to ensure that OpenStack Wallaby release will run on it smoothly.

Depends-On: https://review.opendev.org/c/openstack/diskimage-builder/+/783790
Change-Id: I9c1bb7aaa02ba60ee52e2d7b990e2e6e1212317f
2021-04-19 14:16:38 +00:00
Clark Boylan
8c0185da95 Remove fedora-31 disk image config
This is a followup change to the previous change which removed fedora-31
node launches. Here we cleanup the disk image configs.

Change-Id: Ic72e324c65ee9d18e9c4cf6627ea6c147b9f484b
2021-03-29 11:15:29 +11:00
Clark Boylan
934b12a7e8 Stop launch fedora-31 nodes nodepool
A followup change will remove the diskimage configuration for fedora-31.
The current fedora releases are 32 and 33 so we should clean up 31.

Change-Id: I0dde34ab005f48ac521d91e407ac437d3cec965f
2021-03-29 11:15:29 +11:00
Clark Boylan
8932b81c5a Update nodepool zk configs to be a bit less confusing
Our system-config roles for nodepool update the zookeeper configuration
for our nodepool hosts. The content in the files here is merely a
placeholder. Make that more apparent via the addition of comments and
use of dummy data.

Change-Id: I4e35088a04f6393409963f841f2e9ba174c69598
2021-03-16 14:56:35 -07:00
Clark Boylan
e35112e17e Flip nl01.openstack.org to nl01.opendev.org
This flips the active control of the rax providers from
nl01.openstack.org to nl01.opendev.org. This change should only be
landed once we are happy with the deployment of a new nl01.opendev.org
instance.

Change-Id: Idc838c3cb2d631ef684f733b564f7d4713fc3a41
2021-03-10 12:45:27 -08:00
Clark Boylan
b704a87722 Startup nl01.opendev.org idle
This removes min ready settings for labels on nl01.opendev.org to
prevent it from generating extra min ready requests. We also set
max-servers to 0 on its providers so that it essentailly idles.

This is done to ensure the new server can be deployed properly. A
followon change will flip nl01.openstack.org to idle and set min-ready
and max-servers values on nl01.opendev.org to put it into use.

The old server can be removed once it idles.

Change-Id: Icb13ce9153e3d627b187ec8722539f20db51e266
2021-03-10 12:42:14 -08:00
Clark Boylan
f9d92fbea2 Add an nl01.opendev.org config
This is prep work for eventually deploying an nl01.opendev.org server.
First thing this allows us to do is find and deploy the right config in
testing. Once nl01.openstack.org has been completely replaced we can
remove its config from this repo.

Change-Id: Ib269cdc2bc6b9f96ba1e7a07a594267f49cbfcd5
2021-02-22 10:50:11 -08:00