Tobias Henkel
31c276c234
The quota calculations in nodepool never can be perfectly accurate because there still can be some races with other launchers in the tenant or by other workloads sharing the tenant. So we must be able to gracefully handle the case when the cloud refuses a launch because of quota. Currently we just invalidate the quota cache and immediately try again to launch the node. Of course that will fail again in most cases. This makes the node request and thus the zuul job fail with NODE_FAILURE. Instead we need to pause the handler like it would happen because of quota calculation. Instead we mark the node as aborted which is no failure indicator and pause the handler so it will automatically reschedule a new node launch as soon as the quota calculation allows it. Change-Id: I122268dc7723901c04a58fa6f5c1688a3fdab227
78 lines
1.5 KiB
Python
Executable File
78 lines
1.5 KiB
Python
Executable File
#!/usr/bin/env python
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
|
# not use this file except in compliance with the License. You may obtain
|
|
# a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
|
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
|
# License for the specific language governing permissions and limitations
|
|
# under the License.
|
|
|
|
|
|
class NotFound(Exception):
|
|
pass
|
|
|
|
|
|
class LaunchNodepoolException(Exception):
|
|
statsd_key = 'error.nodepool'
|
|
|
|
|
|
class LaunchStatusException(Exception):
|
|
statsd_key = 'error.status'
|
|
|
|
|
|
class LaunchNetworkException(Exception):
|
|
statsd_key = 'error.network'
|
|
|
|
|
|
class LaunchKeyscanException(Exception):
|
|
statsd_key = 'error.keyscan'
|
|
|
|
|
|
class BuilderError(RuntimeError):
|
|
pass
|
|
|
|
|
|
class BuilderInvalidCommandError(BuilderError):
|
|
pass
|
|
|
|
|
|
class DibFailedError(BuilderError):
|
|
pass
|
|
|
|
|
|
class QuotaException(Exception):
|
|
pass
|
|
|
|
|
|
class TimeoutException(Exception):
|
|
pass
|
|
|
|
|
|
class ConnectionTimeoutException(TimeoutException):
|
|
statsd_key = 'error.ssh'
|
|
|
|
|
|
class IPAddTimeoutException(TimeoutException):
|
|
statsd_key = 'error.ipadd'
|
|
|
|
|
|
class ServerDeleteException(TimeoutException):
|
|
statsd_key = 'error.serverdelete'
|
|
|
|
|
|
class ImageCreateException(TimeoutException):
|
|
statsd_key = 'error.imagetimeout'
|
|
|
|
|
|
class ZKException(Exception):
|
|
pass
|
|
|
|
|
|
class ZKLockException(ZKException):
|
|
pass
|