Some distros (gentoo) do not come with six in the base image (stage3),
it is installed later in the image build. Use the native
urlopen/URLError if the six version is not available.
Change-Id: Id1c52aa17e565c16369645508e43e4f485defa66
We haven't refreshed Gentoo images for the better part of a year,
and now the ones we're using are consistently failing jobs so we
need to update them. Un-pause initially to see if we can spot the
reason they fail (if they still fail).
Change-Id: I9bc26974f641eb934a85a730d0910cb170000e47
These builds are constantly failing (logs at [1], [2]) so pending
investigation, pause the builds to avoid filling the builders.
[1] https://nb01.opendev.org/
[2] https://nb02.opendev.org/
Change-Id: Ic280568c0c084f112cf496178456b898b8cdc18b
Although the servers are turned down to 0, the lack of response from
the cloud is causing problems for nodepool-builder trying to upload
images. Remove the entries completely to avoid this.
Change-Id: I8a82e852b389f3499e32f822cd7aa9e5792ea7c6
This reverts commit ef5cc14205.
We now automatically discard nodes which lack working IPv4
connectivity, the condition which was previously causing occasional
failures in this provider.
Change-Id: I7a7db0dbc1ff6203855eb5dee40d234a26335db0
We keep getting nodes than don't have an IPv4 address which
breaks various jobs. Disable until we can get that fixed.
Change-Id: I5e6b44cbe9a33fe29dc3c989b43c107a1a26599a
The previously observed problem (looks like rogue server instances
again) still appears to persist there.
This reverts commit 492e9eeba1.
Change-Id: I8c70268af93f4fe32a730834b405942602417a82
We're seeing a high incidence of job nodes return RESULT_UNREACHABLE
mid-build, enough to cause frequent RETRY_LIMIT results for changes.
Temporarily cease running jobs there until we/they can work out
whatever connectivity or rogue instance issues are causing this.
Change-Id: I4afd92ee64911f0611733ed9c92ec0a4bada54f5
The cluster has been upgraded to Bionic. Wore out SSDs have been
replaced. And all nodes are now configured with RAID0 instead of RAID1
to provide additional I/O to nodes.
This reverts commit dd62a5e81e.
Change-Id: I97237aecf2d00ebf8c23f931d7079333d90927fb
The launchers have had this label removed and if things look over there
the last step in the nodepool retirement of this image type is to remove
it from the builders.
Change-Id: Ic9e29cfdc6d8a7e2a009f524ecfb0f93b60e3724
This serves as a sanity check that we don't have any fedora-30 usage
hiding somewhere. If this goes in safely then we can remove the image
from the builders.
Change-Id: I09b21e812081f5855a069ca8ab1eedadf090c1b8
Nova CPU service fails to boot instances on ARM64 with the following
error log message:
qemu-kvm: cannot set up guest memory 'mach-virt.ram': Cannot allocate memory
This was observed while testing Octavia-enabled devstack deployments in
CI [1]. This patch adds a 16 GB RAM label for CentOS 8, similarly to
existing 16 GB RAM Bionic label introduced in [2].
[1] https://review.opendev.org/#/c/747629/
[2] https://review.opendev.org/#/c/714346/
Change-Id: If594fe44ad739b517ea56ebcd28008406b3b55ad
nb03.openstack.org and nb04.opendev.org have been cleaned up. Remove
their host specific configs.
Depends-On: https://review.opendev.org/752740
Change-Id: I310534a45a88546ae5537d3016a76693266212bc
We will be performing hardware maintenance.
This require us to shutdown some of the compute nodes,
therefore reducing effective compute capacity.
Change-Id: I017511874c98fa7ab27f811674cf67b8cdbd118d
We'd like to delete nb04.opendev.org and go back to running two x86
image builders. In order to do that gracefully we need to pause all
image builds on nb04 so that nb01 and nb02 take up the slack. Then once
nb04's images are no longer in use they can be deleted and the builder
can be removed.
Change-Id: Ibe00252de4f602a6c9f48ad1799d844710ddc6dc
This reverts commit 22479216b2.
We'll land this when the mirror has static ipv6 networking configured.
Change-Id: I9a037fc9db78f97cadccffb3440d9fb37fed221d
Our vexxhost mirror in ymq is getting rogue RAs. Disable this region in
nodepool so that we can statically configure addresses on the mirror
without impacting jobs.
Change-Id: I42e332dcc36d0e2a7a29e38d916de679e3b36b9c
While attempting to replace the Cinder volume attached to the
rax-dfw mirror server, the xenwatch reported a page allocation
failure. In order to ensure the new block device is correctly seen
by the guest OS we should reboot it. Temporarily wind down testing
there so as not to disrupt any running jobs when we do so. This can
be reverted after we complete the reboot.
Change-Id: I96f69bfa7e3bb2928f24755fd1ff2cb1b347548c
This reverts commit 6f992efbc5.
Setuptools 50.1.0 has been released which reverts the breaking behavior.
https://review.opendev.org/#/c/749766/ tests that these fixes work in
other venvs that exhibited the same problems. That change looks happy so
I think this revert is ready to go.
Change-Id: I31b62be4f85f40f4d99e463cd961dec0a3542f47
Carrier issues are preventing ca-ymq-1 instances, including the
mirror server, from reaching some IPv6 destinations such as PyPI.
Temporarily stop booting job nodes there until the routing problem
has been addressed.
Change-Id: Iea8b96f4a1551ff8bee5d61163eeb8ddfa57198b
We will be working to upgrade the nodes from Xenial to Bionic over
the coming weeks, as well as performing hardware maintenance to
replace wore out SSDs.
Change-Id: I8f12e4467b8f7fbc61ecaf975c6ed6cb6f16c2f0
The Leap 15.2 has happened and we should catch up. This depends on our
mirrors being updated first.
Depends-On: https://review.opendev.org/745251
Change-Id: I738f7037d85bd1898770af0a13509aa98a134720
These labels were not migrated when we moved to running inside our new
hardware inside the Montreal datacenter.
Those largely should disappear as soon as we move our operator
development to be multi-node to avoid having a 'unicorn' flavor. We'd
like to use them temporarily to unblock our progress for now.
Change-Id: Ie29a071b1b8916e0d09990b97bf589bf2b0fc50a
Ubuntu Focal has a newer libvirt version than Bionic (4.0.0 vs 6.0.0).
By adding a Focal-flavored nested-virt label, features made possible
by a more recent libvirt version can be tested in the gate.
Specifically, whitebox-tempest-plugin tests Nova's hw_video_type image
property. Support for the 'none' value was added in libvirt 4.6.0.
Change-Id: Id48fff64d13c258d9f22908debfad86c5f089bf5
Needed-by: https://review.opendev.org/#/c/742014/
Change [1] added nested virtualization labels for Ubuntu Bionic and
CentOS 7. This patch extends that to CentOS 8.
Additionally, we extend nl04 to include these labels too as OVH is a
nested-virt enabled nodepool provider.
[1] https://review.opendev.org/#/c/683431/
Change-Id: Ibf5ac5fa0371cc70dbe58806d147568278afcfea
These are no longer necessary after the merge of
I76befd9e2a054798e8e36500b723870f7c7ae328.
Depends-On: https://review.opendev.org/735774
Change-Id: Id98743d6d2884a15e8496bdefb764287f37cd248
After I76befd9e2a054798e8e36500b723870f7c7ae328 we don't need these
nodes, turn them down so none are ready.
Change-Id: I9a539010280ca9611ac32d9c3ba6a64e62d984bd
We've enabled https on our mirrors. Take advantage of this during our
image builds everywhere we set a mirror but xenial and stretch (they
don't do https out of the box).
Change-Id: I955f47392623c4af9a6f788d078f6d088a0d62cc
As discussed at [1] this removes the pip-and-virtualenv element from
images that still have it.
For now we leave the old templating bits so we can easily revert an
image if there are unavoidable issues.
[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-May/015157.html
Change-Id: I76befd9e2a054798e8e36500b723870f7c7ae328
Matthew Thode
Jeremy Stanley
Ian Wienand
Carlos Goncalves
Jens Harbott
donnydavis
Logan V
Tristan Cacqueray
Clark Boylan
Mathieu Gagné
Lee Yarwood
Mohammed Naser
Artom Lifshitz