Usage ----- .. _usage_guide: Refer to the `API reference`_ for information on the HTTP API. Refer to the :python-ironic-inspector-client-doc:`client documentation <>` for information on how to use CLI and Python library. .. _API reference: https://docs.openstack.org/api-ref/baremetal-introspection/ Using from Ironic API ~~~~~~~~~~~~~~~~~~~~~ Ironic Kilo introduced support for hardware introspection under name of "inspection". **ironic-inspector** introspection is supported for some generic drivers, please refer to :ironic-doc:`Ironic inspection documentation ` for details. Node States ~~~~~~~~~~~ .. _node_states: * The nodes should be moved to ``MANAGEABLE`` provision state before introspection (requires *python-ironicclient* of version 0.5.0 or newer):: baremetal node manage * The introspection can be triggered by using the following command:: baremetal node inspect * After successful introspection and before deploying nodes should be made available to Nova, by moving them to ``AVAILABLE`` state:: baremetal node provide .. note:: Due to how Nova interacts with Ironic driver, you should wait 1 minute before Nova becomes aware of available nodes after issuing this command. Use ``nova hypervisor-stats`` command output to check it. Introspection Rules ~~~~~~~~~~~~~~~~~~~ .. _introspection_rules: Inspector supports a simple JSON-based DSL to define rules to run during introspection. Inspector provides an API to manage such rules, and will run them automatically after running all processing hooks. A rule consists of conditions to check, and actions to run. If conditions evaluate to true on the introspection data, then actions are run on a node. Please refer to the command below to import introspection rule:: baremetal introspection rule import Available conditions and actions are defined by plugins, and can be extended, see :ref:`contributing_link` for details. See the `API reference`_ for specific calls to define introspection rules. Conditions ^^^^^^^^^^ A condition is represented by an object with fields: ``op`` the type of comparison operation, default available operators include: * ``eq``, ``le``, ``ge``, ``ne``, ``lt``, ``gt`` - basic comparison operators; * ``in-net`` - checks that an IP address is in a given network; * ``matches`` - requires a full match against a given regular expression; * ``contains`` - requires a value to contain a given regular expression; * ``is-empty`` - checks that field is an empty string, list, dict or None value. ``field`` a `JSON path `_ to the field in the introspection data to use in comparison. Starting with the Mitaka release, you can also apply conditions to ironic node field. Prefix field with schema (``data://`` or ``node://``) to distinguish between values from introspection data and node. Both schemes use JSON path:: {"field": "node://property.path", "op": "eq", "value": "val"} {"field": "data://introspection.path", "op": "eq", "value": "val"} if scheme (node or data) is missing, condition compares data with introspection data. ``invert`` boolean value, whether to invert the result of the comparison. ``multiple`` how to treat situations where the ``field`` query returns multiple results (e.g. the field contains a list), available options are: * ``any`` (the default) require any to match, * ``all`` require all to match, * ``first`` require the first to match. All other fields are passed to the condition plugin, e.g. numeric comparison operations require a ``value`` field to compare against. Scope ^^^^^ By default, introspection rules are applied to all nodes being inspected. In order for the rule to be applied only to specific nodes, a matching scope variable must be set to both the rule and the node. To set the scope for a rule include field ``"scope"`` in JSON file before importing. For example:: cat { "description": "...", "actions": [...], "conditions": [...], "scope": "SCOPE" } Set the property ``inspection_scope`` on the node you want the rule to be applied to:: baremetal node set --property inspection_scope="SCOPE" Now, when inspecting, the rule will be applied only to nodes with matching scope value. It will also ignore nodes that do not have ``inspection_scope`` property set. Note that if a rule has no scope set, it will be applied to all nodes, regardless if they have ``inspection_scope`` set or not. Actions ^^^^^^^ An action is represented by an object with fields: ``action`` type of action. Possible values are defined by plugins. All other fields are passed to the action plugin. Default available actions include: * ``fail`` fail introspection. Requires a ``message`` parameter for the failure message. * ``set-attribute`` sets an attribute on an Ironic node. Requires a ``path`` field, which is the path to the attribute as used by ironic (e.g. ``/properties/something``), and a ``value`` to set. * ``set-capability`` sets a capability on an Ironic node. Requires ``name`` and ``value`` fields, which are the name and the value for a new capability accordingly. Existing value for this same capability is replaced. * ``extend-attribute`` the same as ``set-attribute``, but treats existing value as a list and appends value to it. If optional ``unique`` parameter is set to ``True``, nothing will be added if given value is already in a list. * ``add-trait`` adds a trait to an Ironic node. Requires a ``name`` field with the name of the trait to add. * ``remove-trait`` removes a trait from an Ironic node. Requires a ``name`` field with the name of the trait to remove. Starting from Mitaka release, ``value`` field in actions supports fetching data from introspection, using `python string formatting notation `_:: {"action": "set-attribute", "path": "/driver_info/ipmi_address", "value": "{data[inventory][bmc_address]}"} Note that any value referenced in this way will be converted to a string. If ``value`` is a dict or list, strings nested at any level within the structure will be formatted as well:: {"action": "set-attribute", "path": "/properties/root_device", "value": {"serial": "{data[root_device][serial]}"}} Plugins ~~~~~~~ .. _introspection_plugins: **ironic-inspector** heavily relies on plugins for data processing. Even the standard functionality is largely based on plugins. Set ``processing_hooks`` option in the configuration file to change the set of plugins to be run on introspection data. Note that order **does** matter in this option, especially for hooks that have dependencies on other hooks. These are plugins that are enabled by default and should not be disabled, unless you understand what you're doing: ``scheduler`` validates and updates basic hardware scheduling properties: CPU number and architecture, memory and disk size. .. note:: Diskless nodes have the disk size property ``local_gb == 0``. Always use node driver ``root_device`` hints to prevent unexpected HW failures passing silently. ``validate_interfaces`` validates network interfaces information. Creates new ports, optionally deletes ports that were not present in the introspection data. Also sets the ``pxe_enabled`` flag for the PXE-booting port and unsets it for all the other ports to avoid **nova** picking a random port to boot the node. .. note:: When the ``pxe_filter`` is configured to only open the DHCP server for known MAC addresses, i.e the ``[pxe_filter]deny_unknown_macs`` configuration option is enabled, it is not possible to rely on the ``validate_interfaces`` processing plug-in to create the PXE-booting port in ironic. Nodes must have at least one enrolled port prior to introspection in this case. The following plugins are enabled by default, but can be disabled if not needed: ``ramdisk_error`` reports error, if ``error`` field is set by the ramdisk, also optionally stores logs from ``logs`` field, see the `API reference`_ for details. ``capabilities`` detect node capabilities: CPU, boot mode, etc. See `Capabilities Detection`_ for more details. ``pci_devices`` gathers the list of all PCI devices returned by the ramdisk and compares to those defined in ``alias`` field(s) from ``pci_devices`` section of configuration file. The recognized PCI devices and their count are then stored in node properties. This information can be later used in nova flavors for node scheduling. Here are some plugins that can be additionally enabled: ``example`` example plugin logging it's input and output. ``raid_device`` gathers block devices from ramdisk and exposes root device in multiple runs. ``extra_hardware`` stores the value of the 'data' key returned by the ramdisk as a JSON encoded string in a Swift object. The plugin will also attempt to convert the data into a format usable by introspection rules. If this is successful then the new format will be stored in the 'extra' key. The 'data' key is then deleted from the introspection data, as unless converted it's assumed unusable by introspection rules. ``lldp_basic`` Processes LLDP data returned from inspection, parses TLVs from the Basic Management (802.1AB), 802.1Q, and 802.3 sets and stores the processed data back in the Ironic inspector database. To enable LLDP in the inventory from IPA, ``ipa-collect-lldp=1`` should be passed as a kernel parameter to the IPA ramdisk. ``local_link_connection`` Processes LLDP data returned from inspection, specifically looking for the port ID and chassis ID. If found, it configures the local link connection information on the Ironic ports with that data. To enable LLDP in the inventory from IPA, ``ipa-collect-lldp=1`` should be passed as a kernel parameter to the IPA ramdisk. In order to avoid processing the raw LLDP data twice, the ``lldp_basic`` plugin should also be installed and run prior to this plugin. ``physnet_cidr_map`` Configures the ``physical_network`` property of the nodes Ironic port when the IP address is in a configured CIDR mapping. CIDR to physical network mappings is set in configuration using the ``[port_physnet]/cidr_map`` option, for example:: [port_physnet] cidr_map = 10.10.10.0/24:physnet_a, 2001:db8::/64:physnet_b ``accelerators`` Processes PCI data returned from inspection and compares with the accelerator inventory, it will update accelerator device information to the properties field of the ironic node if any accelerator device is found, for example:: {'local_gb': '1115', 'cpus': '40', 'cpu_arch': 'x86_64', 'memory_mb': '32768', 'capabilities': 'boot_mode:bios,cpu_vt:true,cpu_aes:true,cpu_hugepages:true,cpu_hugepages_1g:true,cpu_txt:true', 'accel': [{'vendor_id': '10de', 'device_id': '1eb8', 'type': 'GPU', 'pci_address': '0000:82:00.0', 'device_info': 'NVIDIA Corporation Tesla T4'}] } Refer to :ref:`contributing_link` for information on how to write your own plugin. Discovery ~~~~~~~~~ Starting from Mitaka, **ironic-inspector** is able to register new nodes in Ironic. The existing ``node-not-found-hook`` handles what happens if **ironic-inspector** receives inspection data from a node it can not identify. This can happen if a node is manually booted without registering it with Ironic first. For discovery, the configuration file option ``node_not_found_hook`` should be set to load the hook called ``enroll``. This hook will enroll the unidentified node into Ironic using the ``fake-hardware`` hardware type. This is a configurable option: set ``enroll_node_driver`` in the **ironic-inspector** configuration file to the hardware type you want. You can also configure arbitrary fields to set on discovery, for example: .. code-block:: ini [discovery] enroll_node_driver = ipmi enroll_node_fields = management_interface:noop,resource_class:baremetal The ``enroll`` hook will also set the ``ipmi_address`` property on the new node, if its available in the introspection data we received, see `ramdisk callback`_. .. _ramdisk callback: https://docs.openstack.org/api-ref/baremetal-introspection/?expanded=ramdisk-callback-detail#ramdisk-callback Once the ``enroll`` hook is finished, **ironic-inspector** will process the introspection data in the same way it would for an identified node. It runs the processing :ref:`plugins `, and after that it runs introspection rules, which would allow for more customisable node configuration, see :ref:`rules `. A rule to set a node's Ironic driver to ``ipmi`` and populate the required ``driver_info`` for that driver would look like:: [{ "description": "Set IPMI driver_info if no credentials", "actions": [ {"action": "set-attribute", "path": "driver", "value": "ipmi"}, {"action": "set-attribute", "path": "driver_info/ipmi_username", "value": "username"}, {"action": "set-attribute", "path": "driver_info/ipmi_password", "value": "password"} ], "conditions": [ {"op": "is-empty", "field": "node://driver_info.ipmi_password"}, {"op": "is-empty", "field": "node://driver_info.ipmi_username"} ] },{ "description": "Set deploy info if not already set on node", "actions": [ {"action": "set-attribute", "path": "driver_info/deploy_kernel", "value": ""}, {"action": "set-attribute", "path": "driver_info/deploy_ramdisk", "value": ""} ], "conditions": [ {"op": "is-empty", "field": "node://driver_info.deploy_ramdisk"}, {"op": "is-empty", "field": "node://driver_info.deploy_kernel"} ] }] All nodes discovered and enrolled via the ``enroll`` hook, will contain an ``auto_discovered`` flag in the introspection data, this flag makes it possible to distinguish between manually enrolled nodes and auto-discovered nodes in the introspection rules using the rule condition ``eq``:: { "description": "Enroll auto-discovered nodes with ipmi hardware type", "actions": [ {"action": "set-attribute", "path": "driver", "value": "ipmi"} ], "conditions": [ {"op": "eq", "field": "data://auto_discovered", "value": true} ] } Reapplying introspection on stored data ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To allow correcting mistakes in introspection rules the API provides an entry point that triggers the introspection over stored data. The data to use for processing is kept in Swift separately from the data already processed. Reapplying introspection overwrites processed data in the store. Updating the introspection data through the endpoint isn't supported yet. Following preconditions are checked before reapplying introspection: * no data is being sent along with the request * Swift store is configured and enabled * introspection data is stored in Swift for the node UUID * node record is kept in database for the UUID * introspection is not ongoing for the node UUID Should the preconditions fail an immediate response is given to the user: * ``400`` if the request contained data or in case Swift store is not enabled in configuration * ``404`` in case Ironic doesn't keep track of the node UUID * ``409`` if an introspection is already ongoing for the node If the preconditions are met a background task is executed to carry out the processing and a ``202 Accepted`` response is returned to the endpoint user. As requested, these steps are performed in the background task: * preprocessing hooks * post processing hooks, storing result in Swift * introspection rules These steps are avoided, based on the feature requirements: * ``node_not_found_hook`` is skipped * power operations * roll-back actions done by hooks Limitations: * there's no way to update the unprocessed data atm. * the unprocessed data is never cleaned from the store * check for stored data presence is performed in background; missing data situation still results in a ``202`` response Capabilities Detection ~~~~~~~~~~~~~~~~~~~~~~ Starting with the Newton release, **Ironic Inspector** can optionally discover several node capabilities. A recent (Newton or newer) IPA image is required for it to work. Boot mode ^^^^^^^^^ The current boot mode (BIOS or UEFI) can be detected and recorded as ``boot_mode`` capability in Ironic. It will make some drivers to change their behaviour to account for this capability. Set the ``[capabilities]boot_mode`` configuration option to ``True`` to enable. CPU capabilities ^^^^^^^^^^^^^^^^ Several CPU flags are detected by default and recorded as following capabilities: * ``cpu_aes`` AES instructions. * ``cpu_vt`` virtualization support. * ``cpu_txt`` TXT support. * ``cpu_hugepages`` huge pages (2 MiB) support. * ``cpu_hugepages_1g`` huge pages (1 GiB) support. It is possible to define your own rules for detecting CPU capabilities. Set the ``[capabilities]cpu_flags`` configuration option to a mapping between a CPU flag and a capability, for example:: cpu_flags = aes:cpu_aes,svm:cpu_vt,vmx:cpu_vt See the default value of this option for a more detail example. InfiniBand support ^^^^^^^^^^^^^^^^^^ Starting with the Ocata release, **Ironic Inspector** supports detection of InfiniBand network interfaces. A recent (Ocata or newer) IPA image is required for that to work. When an InfiniBand network interface is discovered, the **Ironic Inspector** adds a ``client-id`` attribute to the ``extra`` attribute in the ironic port. The **Ironic Inspector** should be configured with ``iptables.ethoib_interfaces`` to indicate the Ethernet Over InfiniBand (EoIB) which are used for physical access to the DHCP network. For example if **Ironic Inspector** DHCP server is using ``br-inspector`` and the ``br-inspector`` has EoIB port e.g. ``eth0``, the ``iptables.ethoib_interfaces`` should be set to ``eth0``. The ``iptables.ethoib_interfaces`` allows to map the baremetal GUID to it's EoIB MAC based on the neighs files. This is needed for blocking DHCP traffic of the nodes (MACs) which are not part of the introspection. The format of the ``/sys/class/net//eth/neighs`` file:: # EMAC= IMAC= # For example: IMAC=97:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:29:26:52 qp number=97:fe lid=80:00:00:00:00:00:00 GUID=7c:fe:90:03:00:29:26:52 Example of content:: EMAC=02:00:02:97:00:01 IMAC=97:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:29:26:52 EMAC=02:00:00:61:00:02 IMAC=61:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:29:24:4f