
Ironic-inspector documentation is now published on docs.openstack.org. Depends-On: Ic14cb73668544be27c6b96b384f93b239e49acfd Change-Id: Ic83534054a696474da2d89aebfef0d3cf1f8f7c6
3.4 KiB
Troubleshooting Node Management Failures
Where Are the Logs?
Some logs are stored in journald, but most are stored as
text files in /var/log
. Ironic and ironic-inspector logs
are stored in journald. Note that Ironic has 2 units:
openstack-ironic-api
and
openstack-ironic-conductor
. Similarly, ironic-inspector has
openstack-ironic-inspector
and
openstack-ironic-inspector-dnsmasq
. So for example to get
all ironic-inspector logs use:
sudo journalctl -u openstack-ironic-inspector -u openstack-ironic-inspector-dnsmasq
If something fails during the introspection ramdisk run,
ironic-inspector stores the ramdisk logs in
/var/log/ironic-inspector/ramdisk/
as gz-compressed tar
files. File names contain date, time and IPMI address of the node if it
was detected (only for bare metal).
Node Registration Problems
Any problems with node data registered into Ironic can be fixed using the Ironic CLI.
For example, a wrong MAC can be fixed in two steps:
Find out the assigned port UUID by running :
ironic node-port-list <NODE UUID>
Update the MAC address by running :
ironic port-update <PORT UUID> replace address=<NEW MAC>
A Wrong IPMI address can be fixed with the following command:
ironic node-update <NODE UUID> replace driver_info/ipmi_address=<NEW IPMI ADDRESS>
Hardware Introspection Problems
Introspection hangs and times out
ironic-inspector times out introspection process after some time (defaulting to 1 hour) if it never gets response from the introspection ramdisk. This can be a sign of a bug in the introspection ramdisk, but usually it happens due to environment misconfiguration, particularly BIOS boot settings. Please refer to ironic-inspector troubleshooting documentation for information on how to detect and fix such problems.
Refusing to introspect node with provision state "available"
If you're running introspection directly using ironic-inspector CLI (or in case of bugs in our scripts), a node can be in the "AVAILABLE" state, which is meant for deployment, not for introspection. You should advance node to the "MANAGEABLE" state before introspection and move it back before deployment. Please refer to upstream node states documentation for information on how to fix it.
How can introspection be stopped?
Currently ironic-inspector does not provide means for stopping
introspection. The recommended path is to wait until it times out.
Changing timeout
setting in
/etc/ironic-inspector/inspector.conf
may be used to reduce
this timeout from 1 hour (which usually too much, especially on virtual
environment).
If you do need to stop introspection for all nodes right now, do the following for each node:
ironic node-set-power-state UUID off
then remove ironic-inspector cache and restart it:
rm /var/lib/ironic-inspector/inspector.sqlite
sudo systemctl restart openstack-ironic-inspector