openstack/ironic
Code Issues Proposed changes
57037378be
ironic/doc/source/admin/notifications.rst
Jim Rollenhagen 52b8638193 Add conductor_group to node notifications 
Adds the field, bumps the versions, adds to docs.

Change-Id: I952a6c67135d84b93f178edce8e098a2d8200db4
Story: 2001795
Task: 23152
2018-07-25 21:14:28 +00:00

26 KiB
Raw Blame History

Notifications

Ironic, when configured to do so, will emit notifications over a message bus that indicate different events that occur within the service. These can be consumed by any external service. Examples may include a billing or usage system, a monitoring data store, or other OpenStack services. This page describes how to enable notifications and the different kinds of notifications that ironic may emit. The external consumer will see notifications emitted by ironic as JSON objects structured in the following manner:

{
    "priority": <string, defined by the sender>,
    "event_type": <string, defined by the sender>,
    "timestamp": <string, the isotime of when the notification emitted>,
    "publisher_id": <string, defined by the sender>,
    "message_id": <uuid, generated by oslo>,
    "payload": <json serialized dict, defined by the sender>
}

Configuration

To enable notifications with ironic, there are two configuration options in ironic.conf that must be adjusted.

The first option is the notification_level option in the [DEFAULT] section of the configuration file. This can be set to "debug", "info", "warning", "error", or "critical", and determines the minimum priority level for which notifications are emitted. For example, if the option is set to "warning", all notifications with priority level "warning", "error", or "critical" are emitted, but not notifications with priority level "debug" or "info". For information about the semantics of each log level, see the OpenStack logging standards1. If this option is unset, no notifications will be emitted. The priority level of each available notification is documented below.

The second option is the transport_url option in the [oslo_messaging_notifications] section of the configuration. This determines the message bus used when sending notifications. If this is unset, the default transport used for RPC is used.

All notifications are emitted on the "ironic_versioned_notifications" topic in the message bus. Generally, each type of message that traverses the message bus is associated with a topic describing what the message is about. For more information, see the documentation of your chosen message bus, such as the RabbitMQ documentation2.

Note that notifications may be lossy, and there's no guarantee that a notification will make it across the message bus to a consumer.

Versioning

Each notification has an associated version in the "ironic_object.version" field of the payload. Consumers are guaranteed that microversion bumps will add new fields, while macroversion bumps are backwards-incompatible and may have fields removed.

Available notifications

The notifications that ironic emits are described here. They are listed (alphabetically) by service first, then by event_type. All examples below show payloads before serialization to JSON.

ironic-api notifications

Resources CRUD notifications

These notifications are emitted from API service when ironic resources are modified as part of create, update, or delete (CRUD)3 procedures. All CRUD notifications are emitted at INFO level, except for "error" status that is emitted at ERROR level.

List of CRUD notifications for chassis:

  • baremetal.chassis.create.start
  • baremetal.chassis.create.end
  • baremetal.chassis.create.error
  • baremetal.chassis.update.start
  • baremetal.chassis.update.end
  • baremetal.chassis.update.error
  • baremetal.chassis.delete.start
  • baremetal.chassis.delete.end
  • baremetal.chassis.delete.error

Example of chassis CRUD notification:

{
 "priority": "info",
 "payload":{
     "ironic_object.namespace":"ironic",
     "ironic_object.name":"ChassisCRUDPayload",
     "ironic_object.version":"1.0",
     "ironic_object.data":{
         "created_at": "2016-04-10T10:13:03+00:00",
         "description": "bare 28",
         "extra": {},
         "updated_at": "2016-04-27T21:11:03+00:00",
         "uuid": "1910f669-ce8b-43c2-b1d8-cf3d65be815e"
     }
 },
 "event_type":"baremetal.chassis.update.end",
 "publisher_id":"ironic-api.hostname02"
}

List of CRUD notifications for node:

  • baremetal.node.create.start
  • baremetal.node.create.end
  • baremetal.node.create.error
  • baremetal.node.update.start
  • baremetal.node.update.end
  • baremetal.node.update.error
  • baremetal.node.delete.start
  • baremetal.node.delete.end
  • baremetal.node.delete.error

Example of node CRUD notification:

{
 "priority": "info",
 "payload":{
     "ironic_object.namespace":"ironic",
     "ironic_object.name":"NodeCRUDPayload",
     "ironic_object.version":"1.8",
     "ironic_object.data":{
         "chassis_uuid": "db0eef9d-45b2-4dc0-94a8-fc283c01171f",
         "clean_step": None,
         "conductor_group": "",
         "console_enabled": False,
         "created_at": "2016-01-26T20:41:03+00:00",
         "deploy_step": None,
         "driver": "ipmi",
         "driver_info": {
             "ipmi_address": "192.168.0.111",
             "ipmi_username": "root"},
         "extra": {},
         "inspection_finished_at": None,
         "inspection_started_at": None,
         "instance_info": {},
         "instance_uuid": None,
         "last_error": None,
         "maintenance": False,
         "maintenance_reason": None,
         "fault": None,
         "boot_interface": "pxe",
         "console_interface": "no-console",
         "deploy_interface": "iscsi",
         "inspect_interface": "no-inspect",
         "management_interface": "ipmitool",
         "network_interface": "flat",
         "power_interface": "ipmitool",
         "raid_interface": "no-raid",
         "rescue_interface": "no-rescue",
         "storage_interface": "noop",
         "vendor_interface": "no-vendor",
         "name": None,
         "power_state": "power off",
         "properties": {
             "memory_mb":  4096,
             "cpu_arch":  "x86_64",
             "local_gb":  10,
             "cpus":  8},
         "provision_state": "deploying",
         "provision_updated_at": "2016-01-27T20:41:03+00:00",
         "resource_class": None,
         "target_power_state": None,
         "target_provision_state": "active",
         "traits": [
             "CUSTOM_TRAIT1",
             "HW_CPU_X86_VMX"],
         "updated_at": "2016-01-27T20:41:03+00:00",
         "uuid": "1be26c0b-03f2-4d2e-ae87-c02d7f33c123"
     }
 },
 "event_type":"baremetal.node.update.end",
 "publisher_id":"ironic-api.hostname02"
}

List of CRUD notifications for port:

  • baremetal.port.create.start
  • baremetal.port.create.end
  • baremetal.port.create.error
  • baremetal.port.update.start
  • baremetal.port.update.end
  • baremetal.port.update.error
  • baremetal.port.delete.start
  • baremetal.port.delete.end
  • baremetal.port.delete.error

Example of port CRUD notification:

{
 "priority": "info",
 "payload":{
     "ironic_object.namespace":"ironic",
     "ironic_object.name":"PortCRUDPayload",
     "ironic_object.version":"1.2",
     "ironic_object.data":{
         "address": "77:66:23:34:11:b7",
         "created_at": "2016-02-11T15:23:03+00:00",
         "node_uuid": "5b236cab-ad4e-4220-b57c-e827e858745a",
         "extra": {},
         "local_link_connection": {},
         "physical_network": "physnet1",
         "portgroup_uuid": "bd2f385e-c51c-4752-82d1-7a9ec2c25f24",
         "pxe_enabled": True,
         "updated_at": "2016-03-27T20:41:03+00:00",
         "uuid": "1be26c0b-03f2-4d2e-ae87-c02d7f33c123"
     }
 },
 "event_type":"baremetal.port.update.end",
 "publisher_id":"ironic-api.hostname02"
}

List of CRUD notifications for port group:

  • baremetal.portgroup.create.start
  • baremetal.portgroup.create.end
  • baremetal.portgroup.create.error
  • baremetal.portgroup.update.start
  • baremetal.portgroup.update.end
  • baremetal.portgroup.update.error
  • baremetal.portgroup.delete.start
  • baremetal.portgroup.delete.end
  • baremetal.portgroup.delete.error

Example of portgroup CRUD notification:

{
 "priority": "info",
 "payload":{
     "ironic_object.namespace":"ironic",
     "ironic_object.name":"PortgroupCRUDPayload",
     "ironic_object.version":"1.0",
     "ironic_object.data":{
         "address": "11:44:32:87:61:e5",
         "created_at": "2017-01-11T11:33:03+00:00",
         "node_uuid": "5b236cab-ad4e-4220-b57c-e827e858745a",
         "extra": {},
         "mode": "7",
         "name": "portgroup-node-18",
         "properties": {},
         "standalone_ports_supported": True,
         "updated_at": "2017-01-31T11:41:07+00:00",
         "uuid": "db033a40-bfed-4c84-815a-3db26bb268bb",
     }
 },
 "event_type":"baremetal.portgroup.update.end",
 "publisher_id":"ironic-api.hostname02"
}

List of CRUD notifications for volume connector:

  • baremetal.volumeconnector.create.start
  • baremetal.volumeconnector.create.end
  • baremetal.volumeconnector.create.error
  • baremetal.volumeconnector.update.start
  • baremetal.volumeconnector.update.end
  • baremetal.volumeconnector.update.error
  • baremetal.volumeconnector.delete.start
  • baremetal.volumeconnector.delete.end
  • baremetal.volumeconnector.delete.error

Example of volume connector CRUD notification:

{
 "priority": "info",
 "payload": {
     "ironic_object.namespace": "ironic",
     "ironic_object.name": "VolumeConnectorCRUDPayload",
     "ironic_object.version": "1.0",
     "ironic_object.data": {
        "connector_id": "iqn.2017-05.org.openstack:01:d9a51732c3f",
        "created_at": "2017-05-11T05:57:36+00:00",
        "extra": {},
        "node_uuid": "4dbb4e69-99a8-4e13-b6e8-dd2ad4a20caf",
        "type": "iqn",
        "updated_at": "2017-05-11T08:28:58+00:00",
        "uuid": "19b9f3ab-4754-4725-a7a4-c43ea7e57360"
     }
 },
 "event_type": "baremetal.volumeconnector.update.end",
 "publisher_id":"ironic-api.hostname02"
}

List of CRUD notifications for volume target:

  • baremetal.volumetarget.create.start
  • baremetal.volumetarget.create.end
  • baremetal.volumetarget.create.error
  • baremetal.volumetarget.update.start
  • baremetal.volumetarget.update.end
  • baremetal.volumetarget.update.error
  • baremetal.volumetarget.delete.start
  • baremetal.volumetarget.delete.end
  • baremetal.volumetarget.delete.error

Example of volume target CRUD notification:

{
 "priority": "info",
 "payload": {
     "ironic_object.namespace": "ironic",
     "ironic_object.version": "1.0",
     "ironic_object.name": "VolumeTargetCRUDPayload"
     "ironic_object.data": {
         "boot_index": 0,
         "created_at": "2017-05-11T09:38:59+00:00",
         "extra": {},
         "node_uuid": "4dbb4e69-99a8-4e13-b6e8-dd2ad4a20caf",
         "properties": {
             "access_mode": "rw",
             "auth_method": "CHAP"
             "auth_password": "***",
             "auth_username": "urxhQCzAKr4sjyE8DivY",
             "encrypted": false,
             "qos_specs": null,
             "target_discovered": false,
             "target_iqn": "iqn.2010-10.org.openstack:volume-f0d9b0e6-b242-9105-91d4-a20331693ad8",
             "target_lun": 1,
             "target_portal": "192.168.12.34:3260",
             "volume_id": "f0d9b0e6-b042-4105-91d4-a20331693ad8",
         },
         "updated_at": "2017-05-11T09:52:04+00:00",
         "uuid": "82a45833-9c58-4ec1-943c-2091ab10e47b",
         "volume_id": "f0d9b0e6-b242-9105-91d4-a20331693ad8",
         "volume_type": "iscsi"
     }
 },
 "event_type": "baremetal.volumetarget.update.end",
 "publisher_id":"ironic-api.hostname02"
}

Node maintenance notifications

These notifications are emitted from API service when maintenance mode is changed via API service. List of maintenance notifications for a node:

  • baremetal.node.maintenance_set.start
  • baremetal.node.maintenance_set.end
  • baremetal.node.maintenance_set.error

"start" and "end" notifications have INFO level, "error" has ERROR. Example of node maintenance notification:

{
 "priority": "info",
 "payload":{
     "ironic_object.namespace":"ironic",
     "ironic_object.name":"NodePayload",
     "ironic_object.version":"1.10",
     "ironic_object.data":{
         "clean_step": None,
         "conductor_group": "",
         "console_enabled": False,
         "created_at": "2016-01-26T20:41:03+00:00",
         "driver": "ipmi",
         "extra": {},
         "inspection_finished_at": None,
         "inspection_started_at": None,
         "instance_info": {},
         "instance_uuid": None,
         "last_error": None,
         "maintenance": True,
         "maintenance_reason": "hw upgrade",
         "fault": None,
         "bios_interface": "no-bios",
         "boot_interface": "pxe",
         "console_interface": "no-console",
         "deploy_interface": "iscsi",
         "inspect_interface": "no-inspect",
         "management_interface": "ipmitool",
         "network_interface": "flat",
         "power_interface": "ipmitool",
         "raid_interface": "no-raid",
         "rescue_interface": "no-rescue",
         "storage_interface": "noop",
         "vendor_interface": "no-vendor",
         "name": None,
         "power_state": "power off",
         "properties": {
             "memory_mb":  4096,
             "cpu_arch":  "x86_64",
             "local_gb":  10,
             "cpus":  8},
         "provision_state": "available",
         "provision_updated_at": "2016-01-27T20:41:03+00:00",
         "resource_class": None,
         "target_power_state": None,
         "target_provision_state": None,
         "traits": [
             "CUSTOM_TRAIT1",
             "HW_CPU_X86_VMX"],
         "updated_at": "2016-01-27T20:41:03+00:00",
         "uuid": "1be26c0b-03f2-4d2e-ae87-c02d7f33c123"
     }
 },
 "event_type":"baremetal.node.maintenance_set.start",
 "publisher_id":"ironic-api.hostname02"
}

ironic-conductor notifications

Node console notifications

These notifications are emitted by the ironic-conductor service when conductor service starts or stops console for the node. The notification event types for a node console are:

  • baremetal.node.console_set.start
  • baremetal.node.console_set.end
  • baremetal.node.console_set.error
  • baremetal.node.console_restore.start
  • baremetal.node.console_restore.end
  • baremetal.node.console_restore.error

console_set action is used when start or stop console is initiated. The console_restore action is used when the console was already enabled, but a driver must restart the console because an ironic-conductor was restarted. This may also be sent when an ironic-conductor takes over a node that was being managed by another ironic-conductor. "start" and "end" notifications have INFO level, "error" has ERROR. Example of node console notification:

{
 "priority": "info",
 "payload":{
     "ironic_object.namespace":"ironic",
     "ironic_object.name":"NodePayload",
     "ironic_object.version":"1.10",
     "ironic_object.data":{
         "clean_step": None,
         "conductor_group": "",
         "console_enabled": True,
         "created_at": "2016-01-26T20:41:03+00:00",
         "driver": "ipmi",
         "extra": {},
         "inspection_finished_at": None,
         "inspection_started_at": None,
         "instance_info": {},
         "instance_uuid": None,
         "last_error": None,
         "maintenance": False,
         "maintenance_reason": None,
         "fault": None,
         "bios_interface": "no-bios",
         "boot_interface": "pxe",
         "console_interface": "no-console",
         "deploy_interface": "iscsi",
         "inspect_interface": "no-inspect",
         "management_interface": "ipmitool",
         "network_interface": "flat",
         "power_interface": "ipmitool",
         "raid_interface": "no-raid",
         "rescue_interface": "no-rescue",
         "storage_interface": "noop",
         "vendor_interface": "no-vendor",
         "name": None,
         "power_state": "power off",
         "properties": {
             "memory_mb":  4096,
             "cpu_arch":  "x86_64",
             "local_gb":  10,
             "cpus":  8},
         "provision_state": "available",
         "provision_updated_at": "2016-01-27T20:41:03+00:00",
         "resource_class": None,
         "target_power_state": None,
         "target_provision_state": None,
         "traits": [
             "CUSTOM_TRAIT1",
             "HW_CPU_X86_VMX"],
         "updated_at": "2016-01-27T20:41:03+00:00",
         "uuid": "1be26c0b-03f2-4d2e-ae87-c02d7f33c123"
     }
 },
 "event_type":"baremetal.node.console_set.end",
 "publisher_id":"ironic-conductor.hostname01"
}

baremetal.node.power_set

  • baremetal.node.power_set.start is emitted by the ironic-conductor service when it begins a power state change. It has notification level "info".
  • baremetal.node.power_set.end is emitted when ironic-conductor successfully completes a power state change task. It has notification level "info".
  • baremetal.node.power_set.error is emitted by ironic-conductor when it fails to set a node's power state. It has notification level "error". This can occur when ironic fails to retrieve the old power state prior to setting the new one on the node, or when it fails to set the power state if a change is requested.

Here is an example payload for a notification with this event type. The "to_power" payload field indicates the power state to which the ironic-conductor is attempting to change the node:

{
 "priority": "info",
 "payload":{
     "ironic_object.namespace":"ironic",
     "ironic_object.name":"NodeSetPowerStatePayload",
     "ironic_object.version":"1.10",
     "ironic_object.data":{
         "clean_step": None,
         "conductor_group": "",
         "console_enabled": False,
         "created_at": "2016-01-26T20:41:03+00:00",
         "deploy_step": None,
         "driver": "ipmi",
         "extra": {},
         "inspection_finished_at": None,
         "inspection_started_at": None,
         "instance_uuid": "d6ea00c1-1f94-4e95-90b3-3462d7031678",
         "last_error": None,
         "maintenance": False,
         "maintenance_reason": None,
         "fault": None,
         "boot_interface": "pxe",
         "console_interface": "no-console",
         "deploy_interface": "iscsi",
         "inspect_interface": "no-inspect",
         "management_interface": "ipmitool",
         "network_interface": "flat",
         "power_interface": "ipmitool",
         "raid_interface": "no-raid",
         "rescue_interface": "no-rescue",
         "storage_interface": "noop",
         "vendor_interface": "no-vendor",
         "name": None,
         "power_state": "power off",
         "properties": {
             "memory_mb":  4096,
             "cpu_arch":  "x86_64",
             "local_gb":  10,
             "cpus":  8},
         "provision_state": "available",
         "provision_updated_at": "2016-01-27T20:41:03+00:00",
         "resource_class": None,
         "target_power_state": None,
         "target_provision_state": None,
         "traits": [
             "CUSTOM_TRAIT1",
             "HW_CPU_X86_VMX"],
         "updated_at": "2016-01-27T20:41:03+00:00",
         "uuid": "1be26c0b-03f2-4d2e-ae87-c02d7f33c123",
         "to_power": "power on"
     }
 },
 "event_type":"baremetal.node.power_set.start",
 "publisher_id":"ironic-conductor.hostname01"
}

baremetal.node.power_state_corrected

  • baremetal.node.power_state_corrected.success is emitted by ironic-conductor when the power state on the baremetal hardware is different from the previous known power state of the node and the database is corrected to reflect this new power state. It has notification level "info".

Here is an example payload for a notification with this event_type. The "from_power" payload field indicates the previous power state on the node, prior to the correction:

{
 "priority": "info",
 "payload":{
     "ironic_object.namespace":"ironic",
     "ironic_object.name":"NodeCorrectedPowerStatePayload",
     "ironic_object.version":"1.10",
     "ironic_object.data":{
         "clean_step": None,
         "conductor_group": "",
         "console_enabled": False,
         "created_at": "2016-01-26T20:41:03+00:00",
         "deploy_step": None,
         "driver": "ipmi",
         "extra": {},
         "inspection_finished_at": None,
         "inspection_started_at": None,
         "instance_uuid": "d6ea00c1-1f94-4e95-90b3-3462d7031678",
         "last_error": None,
         "maintenance": False,
         "maintenance_reason": None,
         "fault": None,
         "boot_interface": "pxe",
         "console_interface": "no-console",
         "deploy_interface": "iscsi",
         "inspect_interface": "no-inspect",
         "management_interface": "ipmitool",
         "network_interface": "flat",
         "power_interface": "ipmitool",
         "raid_interface": "no-raid",
         "rescue_interface": "no-rescue",
         "storage_interface": "noop",
         "vendor_interface": "no-vendor",
         "name": None,
         "power_state": "power off",
         "properties": {
             "memory_mb":  4096,
             "cpu_arch":  "x86_64",
             "local_gb":  10,
             "cpus":  8},
         "provision_state": "available",
         "provision_updated_at": "2016-01-27T20:41:03+00:00",
         "resource_class": None,
         "target_power_state": None,
         "target_provision_state": None,
         "traits": [
             "CUSTOM_TRAIT1",
             "HW_CPU_X86_VMX"],
         "updated_at": "2016-01-27T20:41:03+00:00",
         "uuid": "1be26c0b-03f2-4d2e-ae87-c02d7f33c123",
         "from_power": "power on"
     }
 },
 "event_type":"baremetal.node.power_state_corrected.success",
 "publisher_id":"ironic-conductor.cond-hostname02"
}

baremetal.node.provision_set

  • baremetal.node.provision_set.start is emitted by the ironic-conductor service when it begins a provision state transition. It has notification level INFO.
  • baremetal.node.provision_set.end is emitted when ironic-conductor successfully completes a provision state transition. It has notification level INFO.
  • baremetal.node.provision_set.success is emitted when ironic-conductor successfully changes provision state instantly, without any intermediate work required (example is AVAILABLE to MANAGEABLE). It has notification level INFO.
  • baremetal.node.provision_set.error is emitted by ironic-conductor when it changes provision state as result of error event processing. It has notification level ERROR.

Here is an example payload for a notification with this event type. The "previous_provision_state" and "previous_target_provision_state" payload fields indicate a node's provision states before state change, "event" is the FSM (finite state machine) event that triggered the state change:

{
 "priority": "info",
 "payload":{
     "ironic_object.namespace":"ironic",
     "ironic_object.name":"NodeSetProvisionStatePayload",
     "ironic_object.version":"1.10",
     "ironic_object.data":{
         "clean_step": None,
         "conductor_group": "",
         "console_enabled": False,
         "created_at": "2016-01-26T20:41:03+00:00",
         "deploy_step": None,
         "driver": "ipmi",
         "extra": {},
         "inspection_finished_at": None,
         "inspection_started_at": None,
         "instance_info": {},
         "instance_uuid": None,
         "last_error": None,
         "maintenance": False,
         "maintenance_reason": None,
         "fault": None,
         "boot_interface": "pxe",
         "console_interface": "no-console",
         "deploy_interface": "iscsi",
         "inspect_interface": "no-inspect",
         "management_interface": "ipmitool",
         "network_interface": "flat",
         "power_interface": "ipmitool",
         "raid_interface": "no-raid",
         "rescue_interface": "no-rescue",
         "storage_interface": "noop",
         "vendor_interface": "no-vendor",
         "name": None,
         "power_state": "power off",
         "properties": {
             "memory_mb":  4096,
             "cpu_arch":  "x86_64",
             "local_gb":  10,
             "cpus":  8},
         "provision_state": "deploying",
         "provision_updated_at": "2016-01-27T20:41:03+00:00",
         "resource_class": None,
         "target_power_state": None,
         "target_provision_state": "active",
         "traits": [
             "CUSTOM_TRAIT1",
             "HW_CPU_X86_VMX"],
         "updated_at": "2016-01-27T20:41:03+00:00",
         "uuid": "1be26c0b-03f2-4d2e-ae87-c02d7f33c123",
         "previous_provision_state": "available",
         "previous_target_provision_state": None,
         "event": "deploy"
     }
 },
 "event_type":"baremetal.node.provision_set.start",
 "publisher_id":"ironic-conductor.hostname01"
}

  1. https://wiki.openstack.org/wiki/LoggingStandards#Log_level_definitions↩︎

  2. https://www.rabbitmq.com/documentation.html↩︎

  3. https://en.wikipedia.org/wiki/Create,_read,_update_and_delete↩︎