Hardware introspection for OpenStack Ironic

This is an auxiliary service for discovering hardware properties for a node managed by OpenStack Ironic. Hardware introspection or hardware properties discovery is a process of getting hardware parameters required for scheduling from a bare metal node, given it's power management credentials (e.g. IPMI address, user name and password).

A special discovery ramdisk is required to collect the information on a node. The default one can be built using diskimage-builder and ironic-discoverd-ramdisk element (see Configuration below).

Support for ironic-discoverd is present in Tuskar UI --OpenStack Horizon plugin for TripleO.

ironic-discoverd requires OpenStack Juno (2014.2) release or newer.

Please use launchpad to report bugs and ask questions. Use PyPI for downloads and accessing the released version of this README. Refer to CONTRIBUTING.rst for instructions on how to contribute.

Workflow

Usual hardware introspection flow is as follows:

• Operator installs undercloud with ironic-discoverd (e.g. using instack-undercloud).
• Operator enrolls nodes into Ironic either manually or by uploading CSV file to Tuskar UI. Power management credentials should be provided to Ironic at this step.
• Operator sends nodes on introspection either manually using ironic-discoverd HTTP API or again via Tuskar UI.
• On receiving node UUID ironic-discoverd:
  • validates node power credentials, current power and provisioning states,
  • allows firewall access to PXE boot service for the nodes,
  • issues reboot command for the nodes, so that they boot the discovery ramdisk.
• The discovery ramdisk collects the required information and posts it back to ironic-discoverd.
• On receiving data from the discovery ramdisk, ironic-discoverd:
  • validates received data,
  • finds the node in Ironic database using it's BMC address (MAC address in case of SSH driver),
  • fills missing node properties with received data and creates missing ports.
• Separate HTTP API can be used to query introspection results for a given node.

Starting DHCP server and configuring PXE boot environment is not part of this package and should be done separately.

Installation

ironic-discoverd is available as an RPM from Fedora 22 repositories or from Juno RDO for Fedora 20, 21 and EPEL 7. It will be installed and preconfigured if you used instack-undercloud to build your undercloud. Otherwise after enabling required repositories install it using:

yum install openstack-ironic-discoverd

Alternatively (e.g. if you need the latest version), you can install package from PyPI (you may want to use virtualenv to isolate your environment):

pip install ironic-discoverd

The third way for RPM-based distros is to use ironic-discoverd copr which contains unstable git snapshots of ironic-discoverd.

Configuration

Copy example.conf to some permanent place (/etc/ironic-discoverd/discoverd.conf is what is used in the RPM). Fill in at least configuration values with names starting with os_. They configure how ironic-discoverd authenticates with Keystone.

Also set database option to where you want ironic-discoverd SQLite database to be placed.

See comments inside example.conf for the other possible configuration options.

Note

Configuration file contains a password and thus should be owned by root and should have access rights like 0600.

As for PXE boot environment, you'll need:

• TFTP server running and accessible (see below for using dnsmasq). Ensure pxelinux.0 is present in the TFTP root.

• Build and put into your TFTP directory kernel and ramdisk from the diskimage-builder ironic-discoverd-ramdisk element:

  ramdisk-image-create -o discovery fedora ironic-discoverd-ramdisk

  You need diskimage-builder 0.1.38 or newer to do it.

• You need PXE boot server (e.g. dnsmasq) running on the same machine as ironic-discoverd. Don't do any firewall configuration: ironic-discoverd will handle it for you. In ironic-discoverd configuration file set dnsmasq_interface to the interface your PXE boot server listens on. Here is an example dnsmasq.conf:

  port=0
  interface={INTERFACE}
  bind-interfaces
  dhcp-range={DHCP IP RANGE, e.g. 192.168.0.50,192.168.0.150}
  enable-tftp
  tftp-root={TFTP ROOT, e.g. /tftpboot}
  dhcp-boot=pxelinux.0

• Configure your $TFTPROOT/pxelinux.cfg/default with something like:

  default discover
  
  label discover
  kernel discovery.kernel
  append initrd=discovery.initramfs discoverd_callback_url=http://{IP}:5050/v1/continue
  
  ipappend 3

  Replace {IP} with IP of the machine (do not use loopback interface, it will be accessed by ramdisk on a booting machine).

  Note

  There are some prebuilt images which use obsolete ironic_callback_url instead of discoverd_callback_url. Modify pxelinux.cfg/default accordingly if you have one of these.

Here is discoverd.conf you may end up with:

[discoverd]
debug = false
os_auth_url = http://127.0.0.1:5000/v2.0
os_username = admin
os_password = password
os_tenant_name = admin
dnsmasq_interface = br-ctlplane

Note

Set debug = true if you want to see complete logs.

Running

If you installed ironic-discoverd from the RPM, you already have a systemd unit, so you can:

systemctl enable openstack-ironic-discoverd
systemctl start openstack-ironic-discoverd

Otherwise run as root:

ironic-discoverd --config-file /etc/ironic-discoverd/discoverd.conf

Note

Running as root is not required if ironic-discoverd does not manage the firewall (i.e. manage_firewall is set to false in the configuration file).

A good starting point for writing your own systemd unit should be one used in Fedora.

Usage

ironic-discoverd has a simple client library bundled within it. It provides functions:

• ironic_discoverd.client.introspect for starting introspection
• ironic_discoverd.client.get_status for querying introspection status

both accepting:

uuid

node UUID

base_url

optional ironic-discoverd service URL (defaults to 127.0.0.1:5050)

auth_token

optional Keystone token.

For testing purposes you can also use it from CLI:

python -m ironic_discoverd.client --auth-token TOKEN introspect UUID
python -m ironic_discoverd.client --auth-token TOKEN get_status UUID

Note

This CLI interface is not stable and may be changed without prior notice. Proper supported CLI is expected later.

HTTP API

By default ironic-discoverd listens on 0.0.0.0:5050, port can be changed in configuration. Protocol is JSON over HTTP.

The HTTP API consist of these endpoints:

• POST /v1/introspection/<UUID> initiate hardware discovery for node <UUID>. All power management configuration for this node needs to be done prior to calling the endpoint.

  Requires X-Auth-Token header with Keystone token for authentication.

  Optional parameters:

  • new_ipmi_password if set, ironic-discoverd will try to set IPMI password on the machine to this value. Power credentials validation will be skipped and manual power on will be required. See Setting IPMI credentials for details.
  • new_ipmi_username provides new IPMI user name in addition to password set by new_ipmi_password. Defaults to current ipmi_username in node driver_info field.

  Response:

  • 202 - accepted discovery request
  • 400 - bad request
  • 401, 403 - missing or invalid authentication
  • 404 - node cannot be found

  Client library function: ironic_discoverd.client.introspect.

• GET /v1/introspection/<UUID> get hardware discovery status.

  Requires X-Auth-Token header with Keystone token for authentication.

  Response:

  • 200 - OK
  • 400 - bad request
  • 401, 403 - missing or invalid authentication
  • 404 - node cannot be found

  Response body: JSON dictionary with keys:

  • finished (boolean) whether discovery is finished
  • error error string or null

  Client library function: ironic_discoverd.client.get_status.

• POST /v1/continue internal endpoint for the discovery ramdisk to post back discovered data. Should not be used for anything other than implementing the ramdisk. Request body: JSON dictionary with at least these keys:

  • cpus number of CPU
  • cpu_arch architecture of the CPU
  • memory_mb RAM in MiB
  • local_gb hard drive size in GiB
  • interfaces dictionary filled with data from all NIC's, keys being interface names, values being dictionaries with keys:
    • mac MAC address
    • ip IP address
  • boot_interface optional MAC address of the NIC that the machine PXE booted from either in standard format 11:22:33:44:55:66 or in PXELinux BOOTIF format 01-11-22-33-44-55-66.
  • block_devices optional block devices information for root_device_hint plugin, dictionary with keys:
    • serials list of serial numbers of block devices.

  Note

  This list highly depends on enabled plugins, provided above are expected keys for the default set of plugins. See Plugins for details.

  Response:

  • 200 - OK
  • 400 - bad request
  • 403 - node is not on introspection
  • 404 - node cannot be found or multiple nodes found

  Response body: JSON dictionary. If Setting IPMI Credentials is requested, body will contain the following keys:

  • ipmi_setup_credentials boolean True
  • ipmi_username new IPMI user name
  • ipmi_password new IPMI password

Setting IPMI Credentials

If you have physical access to your nodes, you can use ironic-discoverd to set IPMI credentials for them without knowing the original ones. The workflow is as follows:

• Ensure nodes will PXE boot on the right network by default.
• Set enable_setting_ipmi_credentials = true in the ironic-discoverd configuration file.
• Enroll nodes in Ironic with setting their ipmi_address only. This step allows ironic-discoverd to distinguish nodes.
• Set maintenance mode on nodes. That's an important step, otherwise Ironic might interfer with introspection process.
• Start introspection with providing additional parameters:
  • new_ipmi_password IPMI password to set,
  • new_ipmi_username IPMI user name to set, defaults to one in node driver_info.
• Manually power on the nodes and wait.
• After introspection is finished (watch nodes power state or use ironic-discoverd status API) you can turn maintenance mode off.

Note that due to various limitations on password value in different BMC, ironic-discoverd will only accept passwords with length between 1 and 20 consisting only of letters and numbers.

Plugins

ironic-discoverd heavily relies on plugins for data processing. Even the standard functionality is largely based on plugins. Set processing_hooks option in the configuration file to change the set of plugins to be run on introspection data. Note that order does matter in this option.

These are plugins that are enabled by default and should not be disabled, unless you understand what you're doing:

scheduler

validates and updates basic hardware scheduling properties: CPU number and architecture, memory and disk size.

validate_interfaces

validates network interfaces information.

Here are some plugins that can be additionally enabled:

ramdisk_error

reports error, if error field is set by the ramdisk.

example

example plugin logging it's input and output.

root_device_hint

gathers block devices from ramdisk and exposes root device in multiple runs.

Refer to CONTRIBUTING.rst for information on how to write your own plugin.

Release Notes

1.1 Series

See 1.1.0 release tracking page for details.

Upgrade Notes

• This version no longer supports ancient ramdisks that sent macs instead of interfaces. It also raises exception if no valid interfaces were found after processing.

Major Features

• Default to only creating a port for the NIC that the ramdisk was PXE booted from, if such information is provided by ramdisk as boot_interface field. Adjustable by only_pxe_booting_port option.

  See better-boot-interface-detection blueprint for details.

• Setting IPMI Credentials feature is considered stable now and is exposed in the client. It still needs to be enabled via configuration.

  See setup-ipmi-credentials-take2 blueprint for what changed since 1.0.0 (tl;dr: everything).

Other Changes

• Experimental plugin edeploy to use with eDeploy hardware detection and classification utilities.

  See eDeploy blueprint for details.

• Plugin root_device_hint for in-band root device discovery.

Known Issues

1.0 Series

1.0 is the first feature-complete release series. It's also the first series to follow standard OpenStack processes from the beginning. All 0.2 series users are advised to upgrade.

See 1.0.0 release tracking page for details.

Upgrade notes

Action required:

• Fill in database option in the configuration file before upgrading.
• Stop relying on ironic-discoverd setting maintenance mode itself.
• Stop relying on discovery_timestamp node extra field.

Action recommended:

• Switch your init scripts to use ironic-discoverd --config-file <path> instead of just ironic-discoverd <path>.
• Stop relying on on_discovery and newly_discovered being set in node extra field during and after introspection. Use new get status HTTP endpoint and client API instead.
• Switch from discover to introspect HTTP endpoint and client API.

Major features

• Introspection now times out by default, set timeout option to alter.

• New API GET /v1/introspection/<uuid> and client.get_status for getting discovery status.

  See get-status-api blueprint for details.

• New API POST /v1/introspection/<uuid> and client.introspect is now used to initiate discovery, /v1/discover is deprecated.

  See v1 API reform blueprint for details.

• /v1/continue is now sync:

  • Errors are properly returned to the caller
  • This call now returns value as a JSON dict (currently empty)

• Add support for plugins that hook into data processing pipeline. Refer to Plugins for information on bundled plugins and to CONTRIBUTING.rst for information on how to write your own.

  See plugin-architecture blueprint for details.

• Support for OpenStack Kilo release and new Ironic state machine -see Kilo state machine blueprint.

  As a side effect, no longer depend on maintenance mode for introspection. Stop putting node in maintenance mode before introspection.

• Cache nodes under introspection in a local SQLite database. database configuration option sets where to place this database. Improves performance by making less calls to Ironic API and makes possible to get results of introspection.

Other Changes

• Firewall management can be disabled completely via manage_firewall option.

• Experimental support for updating IPMI credentials from within ramdisk.

  Enable via configuration option enable_setting_ipmi_credentials. Beware that this feature lacks proper testing, is not supported officially yet and is subject to changes without keeping backward compatibility.

  See setup-ipmi-credentials blueprint for details.

Known Issues

• bug 1415040 it is required to set IP addresses instead of host names in ipmi_address/ilo_address/drac_host node driver_info field for ironic-discoverd to work properly.

0.2 Series

0.2 series is designed to work with OpenStack Juno release. The major changes are:

API

• Authentication via Keystone for /v1/discover.
• Expect interfaces instead of macs in post-back from the ramdisk [version 0.2.1].
• If interfaces is present, only add ports for NIC's with IP address set [version 0.2.1].
• /v1/discover now does some sync sanity checks [version 0.2.2].
• Nodes will be always put into maintenance mode before discovery [version 0.2.1].

Configuration

• Periodic firewall update is now configurable.
• On each start-up make several attempts to check that Ironic is available [version 0.2.2].

Misc

• Simple client in ironic_discoverd.client.
• Preliminary supported for Python 3.3 (real support depends on Eventlet).

0.1 Series

First stable release series. Not supported any more.