When cleaning fails, we power off the node, unless it has been running
a clean step already. This happens when aborting cleaning or on a boot
failure. This change makes sure that the power action does not wipe
the last_error field, resulting in a node with provision_state=CLEANFAIL
and last_error=None for several seconds. I've hit this in Metal3.
Also when aborting cleaning, make sure last_error is set during
the transition to CLEANFAIL, not when the clean up thread starts
running.
While here, make sure to log the current step in all cases, not only
when aborting a non-abortable step.
Change-Id: Id21dd7eb44dad149661ebe2d75a9b030aa70526f
Story: #2010603
Task: #47476
Currently when a conductor is stopped, the rpc service stops
responding to requests as soon as self.manager.del_host returns. This
means that until the hash ring is reset on the whole cluster, requests
can be sent to a service which is stopped.
This change waits for the remaining seconds to delay stopping until
CONF.hash_ring_reset_interval has elapsed. This will improve the
reliability of the cluster when scaling down or rolling out updates.
This delay only occurs when there is more than one online conductor,
to allow fast restarts on single-node ironic installs (bifrost,
metal3).
Change-Id: I643eb34f9605532c5c12dd2a42f4ea67bf3e0b40
This change adds the capability for the ironic-conductor
and standalone service process to transmit timer and counter
metrics to the message bus notifier which may be consumed by
a ceilometer, ironic-prometheus-exporter, or other consumer of
metrics event data on to the message bus.
This functionality is not presently supported on dedicated API
services such as those running as an ``ironic-api`` application
process, or Ironic WSGI application. This is due to the lack of
an internal trigger mechanism to transmit the data in a metrics
update to the message bus and/or notifier plugin.
This change requires ironic-lib 5.4.0 to collect and ship metrics via
the message bus.
Depends-On: https://review.opendev.org/c/openstack/ironic-lib/+/865311
Change-Id: If6941f970241a22d96e06d88365f76edc4683364
While developing some internal metrics collection capability,
and the realization that a lock was needed, we realized that
the lock activity itself would be a bit noisy. And image actions
also get lock logging, and it is just really noisy, but not super
helpful for troubleshooting.
So, set it to WARNING instead.
Discussion wise, see:
https://review.opendev.org/c/openstack/ironic-lib/+/865311
Change-Id: I3ab14ee5b5cc063784d26e3c760f1422c692060d
Follow-up to I6b830e5cc30f1fa1f1900e7c45e6f246fa1ec51c
Original changa introduced some errors such as mismatched
arguments for exceptions
Story: 2010275
Task: 46204
Change-Id: I550e048ab22a6cd25502b41d1c579819df369249
Follow-up to I74b19f7a42c1326d7ec04e6320176e81639ebfb4
Mention need of the maintenance mode to orphan swift
objects during node clean up
Story: 2010275
Task: 46204
Change-Id: Ie95a5bd333b0dab3e97254dfb4eb532bdbfd2650
The tl;dr is that we changed ``inspecting`` to include a
``inspect wait`` state. Unfortunately we never spotted the logic
inside of the db API. We never spotted it because our testing in
inspection code uses a mocked task manager... and we *really* don't
have intense db testing because we expect the objects and higher
level interactions to validate the lowest db level.
Unfortunately, because of the out of band inspection workflow,
we have to cover both cases in terms of what the starting state
and ending state could be, but we've added tests to
validate this is handled as we expect.
Change-Id: Icccbc6d65531e460c55555e021bf81d362f5fc8b
The dynamically allocated console port for a node is saved
into database and reused on subsequent console operations.
In certain code path the port record cann't be trusted and
we should do a re-allocation.
This patch fixes the issue by ignores previous allocation
record. The extra cleanup in the takeover is not required
anymore and removed as well.
Change-Id: I1a07ea9b30a2c760af7a6a4e39f3ff227df28fff
Story: 2010489
Task: 47061
Recently we hit an issue that the pid file is missing, current logic
simply removes pid file if the corresponding process is not found,
but if the pid file is lost then the console could never be stopped
and futher more, be restarted, regardless if the process is there or
not.
This patch captures FileNotFound to the exception handling to allow
console recovery.
Change-Id: I1a0b8347e960c6cff8aca10a22c67b710f7d617e
Follow-up to Ie174904420691be64ce6ca10bca3231f45a5bc58
which enables storage of inventory in Swift, but does not delete
the Swift entry when the node whose inventory is stored is deleted
Story: 2010275
Task: 46204
Change-Id: I74b19f7a42c1326d7ec04e6320176e81639ebfb4
- Basic support and testing for CRUD for node.shard.
- Policy checking for update node.shard.
- New API endpoint: GET /v1/shards
- Policy checking for GET /v1/shards
- Support for querying for nodes in a list of shards
Story: 2010378
Task: 46624
Change-Id: I385594339028c20cfc83fdcc4cbbec107efdacff
This request parameter will allow an operator to ask the question
"Do I need to assign shards to any of my nodes?".
Change-Id: I26b745e5ef2b320a8d8a0667ac61c080fcdcd576
The format string is expecting a dictionary with keys matching
those used in the format string. Any unused parameters will
cause an "not all arguments converted during string formatting"
exception.
The quote style is also changed from double to single quotes to
match the other logging statements in the code.
Change-Id: Ic9dea4f51d82866be8ac16242a79237c789b9745
Ports cannot be filtered by node, node_uuid, or portgroup at the same
time as other potential filters. Explicitly document this.
Change-Id: Ia875a6543eb8871ce70028c055de2f1832c3ecdb
