66 Commits

Author SHA1 Message Date
Jeremy Stanley
1d497b0c7d Clean up unused labels from nl02 config
The provider-specific label variants designating nested virt
acceleration support or larger flavors are unused by nl02, so delete
them to reduce confusion.

Change-Id: Id3ac994216624e32d83ae9066d3e77f713cc7245
2024-05-22 15:33:34 +00:00
Jeremy Stanley
059f2785e5 Add Ubuntu 24.04 LTS (ubuntu-noble) nodes
Build images and boot ubuntu-noble everywhere we do for
ubuntu-jammy. Drop the kernel boot parameter override we use on
Jammy since it's default in the kernel versions included in Noble
now.

Change-Id: I3b9d01a111e66290cae16f7f4f58ba0c6f2cacd8
2024-05-21 19:37:55 +00:00
Zuul
371ec90145 Merge "Add warning to nodepool configs about changing cloud name" 2024-04-17 12:29:38 +00:00
Clark Boylan
aabaf95b49 Remove centos-7 nodepool image builds
This is the last step in cleaning centos-7 out of nodepool. The previous
change will have cleaned up uploads and now we can stop building the
images entirely.

Change-Id: Ie81d6d516cd6cd42ae9797025a39521ceede7b71
2024-03-13 08:30:16 -07:00
Clark Boylan
b8c53b9c03 Remove centos-7 image uploads from Nodepool
This removal of centos-7 image uploads should cause Nodepool to clean up
the existing images in the clouds. Once that is done we can completely
remove the image builds in a followup change.

We are performing this cleanup because CentOS 7 is near its EOL and
cleaning it up will create room on nodepool builders and our mirrors for
other more modern test platforms.

Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/912786
Change-Id: I48f6845bc7c97e0a8feb75fc0d540bdbe067e769
2024-03-13 08:21:46 -07:00
Clark Boylan
774ad69f33 Add warning to nodepool configs about changing cloud name
The cloud name is used to lookup cloud credentials in clouds.yaml, but
it is also used to determine names for things like mirrors within jobs.
As a result changing this value can impact running jobs as you need to
update DNS for mirrors (and possibly launch new mirrors) first. Add a
warning to help avoid problems like this in the future.

Change-Id: I9854ad47553370e6cc9ede843be3303dfa1f9f34
2024-03-07 11:28:17 -08:00
Clark Boylan
c41bc6e5c2 Remove debian-buster image builds from nodepool
This should be landed after the parent chagne has landed and nodepool
has successfully deleted all debian-buster image uploads from our cloud
providers. At this point it should be safe to remove the image builds
entirely.

Change-Id: I7fae65204ca825665c2e168f85d3630686d0cc75
2024-02-23 13:23:22 -08:00
Clark Boylan
feff36e424 Drop debian-buster image uploads from nodepool
Debian buster has been replaced by bullseye and bookworm, both of which
are releases we have images for. It is time to remove the unused debian
buster images as a result.

This change follows the process in nodepool docs for removing a provider
[0] (which isn't quite what we are doing) to properly remove images so
that they can be deleted by nodepool before we remove nodepool's
knowledge of them. The followup change will remove the image builds from
nodepool.

[0] https://zuul-ci.org/docs/nodepool/latest/operation.html#removing-a-provider

Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/910015
Change-Id: I37cb3779944ff9eb1b774ecaf6df3c6929596155
2024-02-23 13:19:49 -08:00
Clark Boylan
211fe14946 Remove opensuse-15 image builds from nodepool
This should be landed after the parent chagne has landed and nodepool
has successfully deleted all opensuse-15 image uploads from our cloud
providers. At this point it should be safe to remove the image builds
entirely.

Change-Id: Icc870ce04b0f0b26df673f85dd6380234979906f
2024-02-22 10:27:37 -08:00
Clark Boylan
5635e67866 Drop opensuse image uploads from nodepool
These images are old opensuse 15.2 and there doesn't seem to be interest
in keeping these images running (very few jobs ever ran on them and
rarely successfully and no one is trying to update to 15.5 or 15.6).

This change follows the process in nodepool docs for removing a provider
[0] (which isn't quite what we are doing) to properly remove images so
that they can be deleted by nodepool before we remove nodepool's
knowledge of them. The followup change will remove the image builds from
nodepool.

[0] https://zuul-ci.org/docs/nodepool/latest/operation.html#removing-a-provider

Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/909773
Change-Id: Id9373762ed5de5c7c5131811cec989c2e6e51910
2024-02-22 10:25:15 -08:00
Michal Nasiadka
4ba928c675 Add nested-virt-debian-bookworm
Change-Id: I17a202cc82ff19a788fde7b34415542c1b354fae
2023-10-04 14:47:23 +02:00
Clark Boylan
3b9c5d2f07 Remove fedora image builds
This removes the fedora image builds from nodepool. At this point
Nodepool should no longer have any knowledge of fedora.

There is potential for other cleanups for things like dib elements, but
leaving those in place doesn't hurt much.

Change-Id: I3e6984bc060e9d21f7ad851f3a64db8bb555b38a
2023-09-06 09:16:34 -07:00
Clark Boylan
d83736575e Remove fedora-35 and fedora-36 from nodepool providers
This will stop providing the node label entirely and should result in
nodepool cleaning up the existing images for these images in our cloud
providers. It does not remove the diskimages for fedora which will
happen next.

Change-Id: Ic1361ff4e159509103a6436c88c9f3b5ca447777
2023-09-06 09:12:33 -07:00
Dr. Jens Harbott
5aa792f1ae Start booting bookworm nodes
Image builds have been successful

Change-Id: If286eb3e1a75c643f67f3d6d3d7e2d31c205ac1b
2023-07-03 18:47:46 +02:00
Dr. Jens Harbott
ac5c9ccc5b Add nested-virt-debian-bullseye label to nodepool
kolla wants to have testing parity between Ubuntu and Debian, so add a
nested-virt-debian-bullseye label to nodepool matching the existing
nested-virt-ubuntu-bionic label.

Change-Id: I27766140120fb55a2eab35552f0321b1da9c67ff
2023-03-31 18:15:25 +02:00
Clark Boylan
53cdd2d990 Revert "Revert "Revert "Temporarily stop booting nodes in inmotion iad3"""
This reverts commit 4a2253aac3d3e7eda30b127ffbc7eec9bd3ce565.

We've made some modifications to the nova installation in this cloud
which should prevent nodes other than the mirror from launching on its
hypervisor. This should protect it from OOMs.

Change-Id: I9e0f6dac3c13f21e676f44c44206861cea289b34
2023-03-06 16:06:59 -08:00
Jeremy Stanley
4a2253aac3 Revert "Revert "Temporarily stop booting nodes in inmotion iad3""
The mirror server spontaneously powered off again. It's been booted
back up, but let's take the region out of service until someone has
a chance to investigate the reason and hopefully fix it so that it
doesn't keep happening.

This reverts commit f45f51fdd726462d16441668ec5ac1a2e6d6c0be.

Change-Id: If24a375f3b0cbf7f9d60157ae7597bb0b1c4835a
2023-03-03 14:23:03 +00:00
Jeremy Stanley
f45f51fdd7 Revert "Temporarily stop booting nodes in inmotion iad3"
Merge once the mirror for this provider has returned to service.

This reverts commit 17888a4a0352e16468384f3198ba2ba1fd0483b3.

Change-Id: I480d695a63f0a695631c97294740c8443dd6981c
2023-02-27 13:43:27 +00:00
Jeremy Stanley
17888a4a03 Temporarily stop booting nodes in inmotion iad3
The mirror server in the inmotion iad3 region is down. Don't boot
nodes there for now, since jobs run on them will almost certainly
fail. This can be reverted once the mirror is back in service.

Change-Id: I369b87f97446a3b927e98b59e2fd1ac1e772b8f8
2023-02-27 12:46:26 +00:00
Jeremy Stanley
92814f9b71
Remove empty limestone nodepool providers
Once the builders have a chance to clear out all uploaded images,
this will remove the remaining references in Nodepool. Then
system-config cleanup can proceed.

Change-Id: I69b96b690918a9145d2e7ccbc79968c5341480bb
2023-02-14 08:25:25 +11:00
Jeremy Stanley
7c81cf6eda
Farewell limestone
The mirror in our Limestone Networks donor environment is now
unreachable, but we ceased using this region years ago due to
persistent networking trouble and the admin hasn't been around for
roughly as long, so it's probably time to go ahead and say goodbye
to it.

In preparation for cleanup of credentials in system-config, first
remove configuration here except leave the nodepool provider with an
empty diskimages list so that it will have a chance to pick up after
itself.

Change-Id: I504682884a1439fac84d514880757c2cd041ada6
2023-02-14 08:25:10 +11:00
Jeremy Stanley
fcd02797ee Add missing rockylinux and openeuler entries
Our various nodepool launcher configs have gotten out of sync with
recent additions. Try to match them up better by adding rockylinux
and openeuler to places where they were missing. In particular, this
addresses an inconsistency on nl04 which caused the service to crash
at last restart due to rockylinux missing from the main diskimages
list.

Change-Id: Ie157ab7f667dbdc313b7fdbfc1df71874e8bd0fc
2022-09-20 12:35:16 +00:00
Michal Nasiadka
13552d20e5 Add nested-virt-rockylinux-9
Change-Id: I5c8c8350173d0aee0683c7fe40fafce7b5602a51
2022-08-24 17:47:11 +02:00
Ian Wienand
36b9c302e5 nodepool: Add Fedora 36
Add Fedora 36 builds

Change-Id: I64fac34945ea5c6ec91ddd442281fcaba2c53271
2022-08-22 11:25:09 +10:00
Neil Hanlon
705e8420a2
Add rockylinux 9 to nodepool
Change-Id: Iedf1c8eb2898cfc5771a5a695a53a39f9396edc9
2022-08-05 08:58:27 -04:00
Dr. Jens Harbott
0d47c4f241 Add nested-virt-ubuntu-jammy nodes
kolla-ansible has switched to Jammy in master, create nodes that allow
their kvm tests to continue to run.

Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I97b59a1dad032665db986ffaf9f7d6fdb45caf26
2022-08-04 12:09:58 +02:00
Dr. Jens Harbott
1bdccd42e5 Start launching Jammy images
The first image was built successfully, so we can start launching them.

Change-Id: Ie84d1700b6f4f7696e14dfe01bc887e422163d7e
2022-04-26 13:53:29 +02:00
Clark Boylan
3dc09d41d1 Remove airship-citycloud nodepool provider
Once we've settled on a max-servers value of 0 without any servers this
is the nextstep in removing the provider. This should completely remove
the provider from the launcher and the image builders.

We keep the airship-citycloud nodepool provider for historical
information purposes. We can clean this up later.

Change-Id: Icfb8fc6d2b15714ecb58960d8e44b199bedd6b0d
2022-03-01 08:18:08 -08:00
Clark Boylan
1c73cd826c Set airship-citycloud max-servers to 0
We've been notified that these resources won't be provided any longer.
This is the first step of setting max-servers to 0 and removing images
from the cloud. Once that is in we can remove the cloud more completely.

Change-Id: Iabcd6487a5bb3ed7fb6aae5dadf23a8171abcb7f
2022-03-01 08:10:23 -08:00
Neil Hanlon
2ccc5241c8
Add rockylinux-8 to nodepool configuration
Signed-off-by: Neil Hanlon <neil@rockylinux.org>
Change-Id: Ic3344bc47ca56c27f7ec3427a0865bd6ce3349d3
Depends-On: https://review.opendev.org/c/openstack/project-config/+/829405
Depends-On: https://review.opendev.org/c/openstack/project-config/+/829712
2022-02-17 09:12:00 -05:00
Ian Wienand
dd58c496f8 Remove Fedora 34
The dependent changes should be the last references to Fedora 34 nodes

Depends-On: https://review.opendev.org/c/openstack/devstack/+/827576
Depends-On: https://review.opendev.org/c/openstack/devstack/+/827578
Depends-On: https://review.opendev.org/c/zuul/nodepool/+/827577

Change-Id: Ie14ea374808e5518588925de3a476f0bc6ff2ccf
2022-02-11 07:55:21 +11:00
Clark Boylan
dce378a6b4 Remove centos-8
This distro release reached its EOL December 31, 2021. We are removing
it from our CI system as people should really stop testing on it. They
can use CentOS 8 Stream or other alternatives instead.

Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/827181
Change-Id: I13e8185b7839371a9f9043b715dc39c6baf907d5
2022-02-02 09:48:36 +11:00
Gregory Thiemonge
9b4d8e38fe Adding nested-virt pools for centos-9-stream
Change-Id: I938c163d8735eb9fd20142f4e2ff366728a5eec7
2022-01-20 09:05:45 +01:00
Clark Boylan
07e9803134 Remove opensuse-tumbleweed from nodepool
This removes the label, nodes, and images for opensuse-tumbleweed across
our cloud providers. We also update grafana to stop graphing stats for
the label.

Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/824068
Change-Id: Ic311af5d667c01c1845251270fd2fdda7d99ebcb
2022-01-10 14:02:55 -08:00
Zuul
4a33b66493 Merge "Add Fedora 35" 2021-11-14 22:19:52 +00:00
Ian Wienand
b48f6b8f70 Remove debian-stretch nodes and builds
Change-Id: I62783d6180eddfff3c0e8014011e6bb90c819ed4
2021-11-10 15:10:06 +11:00
Ian Wienand
f8d93ea149 Add Fedora 35
DIB with support is deployed to builder nodes

Change-Id: I6828383643aee1d6dadc71dfd6c25ba91a1bf561
2021-11-08 15:20:13 +11:00
Alfredo Moralejo
5100cf7063 Add centos-9-stream nodepool image
CentOS Stream 9 repos are ready and support for it is included in
diskimage-builder [1]. This patch is adding centos-9-stream diskimages
and images to nodepool configuration in opendev.

[1] https://review.opendev.org/c/openstack/diskimage-builder/+/811392

Change-Id: I9b88baf422d947d5209d036766a86b09dca0c21a
2021-11-03 12:48:50 +01:00
Zuul
fb716f1fcb Merge "Increasing max servers" 2021-10-26 21:55:07 +00:00
Yuriy Shyyan
69d4750d5b Increasing max servers
The cloud is very stable at 36, the maximum due to IPv4 limitations is
51 for now, so we're going with 51. I am actively monitoring the cloud workloads,
performance, workload spread, etc for any issues.

Change-Id: Ia5e0f04668739537da9d9ef58600be229df0e9b3
2021-10-26 16:15:47 -04:00
Zuul
7021447879 Merge "Reinstating InMotion cloud" 2021-10-26 18:18:47 +00:00
Yuriy Shyyan
06f9082b3f Reinstating InMotion cloud
IPs have been added, compute nodes have been added, nova-scheduler
configured for spread, placement configured for randomization of hosts
system capacity in place, placement optimized for maximum allocation of VMs.
Starting with 36, goal target is 51, depends on how CPU bottlenecks looks like with spread.

Change-Id: I6d3d00081a983be0fdf0adbe66b79751439fcce5
2021-10-26 13:54:19 -04:00
Zuul
03fac45522 Merge "Remove fedora-32 disk image config" 2021-10-20 04:40:05 +00:00
Yuriy Shyyan
dd8c47b0cb Disabling inmotion cloud scheduling for upgrades.
Going to be adding more hardware resources,
as well as applying some placement and nova-scheduler configuration options.

Change-Id: Ie8b956dd02129e25bc1c9c46c8b9e53040579681
2021-10-08 08:19:21 -04:00
Yuriy Shyyan
6c1b24c587 Adjusting tenancy limits for this cloud.
Still adjusting to Zuul as is just very explosive.
Goes from a load of 2 to 30 because codeset ends up being
tested in instances on the same node. This may help.

Change-Id: I0f07fb08e87a16e011952d0b755b9527f5a43a30
2021-09-28 06:50:57 -04:00
Yuriy Shyyan
81410020bc Improve CI performance and reduce infra load.
Scaling down once more, memory is doing good, but CPU
seems to get a bit explosive and impacts CI performance.
32 -> 24 at this point reduces risk and load potential by 25%.

Change-Id: Ic712c0ca5242d988204a38407c1dd5751c0ced41
2021-09-21 18:09:36 -04:00
Yuriy Shyyan
8a00690661 Scaling down InMotion nodepool resource.
Unfortunately with 40 we still saw OOM kills, Launch Errors as
well as reported long run times due to CPU overcommit.

Change-Id: I40a928fc910d557c9c846918037760ac39b0d74a
2021-09-21 08:00:29 -04:00
Yuriy Shyyan
c4b0d3fc1a Scaling down proposed instance count.
Unfortunately 50 instances on the current crashes out nova-compute.
Let's find a happy number, so the cloud is responsive. Setting to 40, but
anticipating that we may have to go even lower for consistency and responsiveness.

Change-Id: Ia726ed400bc881d1a97f0c39a05c481f38c6a0eb
2021-09-20 09:01:18 -04:00
Yuriy Shyyan
16e9d3a95a Scaling up Inmotion Cloud workloads.
We have added more IP space to the inmotion cloud.
Bump max servers to 50.

Change-Id: I2602d33f5deee92901405a1a6a7c86ffa9b42131
2021-09-17 15:53:46 -04:00
Clark Boylan
a11f4ca00b Disable inmotion for planned cloud work
We're going to be expanding the available IP addresses on this cloud and
I have been told we probably want the cloud to be disabled for this
work.

Change-Id: I5e1e8bbc934675795f9876e743dab59728a6f1f4
2021-09-17 07:22:13 -07:00