ironic/doc/source/admin/availability-zones.rst

Availability Zones and Resource Isolation

Overview

While Ironic does not implement traditional OpenStack Availability Zones like Nova and Neutron, it provides a three-tier approach for resource partitioning and isolation that achieves comprehensive availability zone functionality:

• Multiple Ironic Deployments: Completely separate Ironic services

  targeted by different Nova compute nodes

• Conductor Groups: Physical/geographical resource partitioning within

  a deployment

• Shards: Logical grouping for operational scaling within a deployment

This document explains how these mechanisms work together and how to achieve sophisticated availability zone functionality across your infrastructure. This document does not cover similar effect which can be achieved through the use of API level Role Based Access Control through the owner and lessee fields.

Table of Contents

Comparison with Other OpenStack Services

Service	Mechanism	Purpose
Nova	Availability Zones (host aggregates)	Instance placement across fault domains
Neutron	Agent AZs	Network service HA
Ironic	Multiple Deployments + Conductor Groups + Shards	Complete service isolation + physical partitioning + operational scaling

Ironic's Three-Tier Approach

Tier 1: Multiple Ironic Deployments

The highest level of isolation involves running completely separate Ironic services that Nova and other API users can target independently.

Use Cases:

• Complete geographic separation (different regions/countries)
• Regulatory compliance requiring full data isolation
• Independent upgrade cycles and operational teams

Implementation: Configure separate Nova compute services to target different Ironic deployments using Nova's Ironic driver configuration.

Benefits:

• Complete fault isolation - failure of one deployment doesn't affect others
• Independent scaling, upgrades, and maintenance
• Different operational policies per deployment
• Complete API endpoint separation

Tier 2: Conductor Groups (Physical Partitioning)

Within a single Ironic deployment, conductor groups provides physical resource partitioning.

Use Cases:

• Separate nodes by datacenter/availability zone within a region
• Separate nodes by conductor groups for conductor resource management
• Isolate hardware types or vendors
• Create fault domains for high availability
• Manage nodes with different network connectivity

Conductor groups control which conductor manages which nodes. Each conductor can be assigned to a specific group, and will only manage nodes that belong to the same group.

A classical challenge of Ironic is that it is able to manage far more Bare Metal nodes than a single nova-compute service is designed to support. The primary answer for this issue is to leverage Shards first, and then continue to evolve based upon operational needs.

See: conductor-groups for detailed configuration.

Tier 3: Shards (Logical Partitioning)

The finest level of granularity for operational and client-side grouping.

Use Cases:

• Horizontal scaling of operations
• Parallelize maintenance tasks
• Create logical groupings for different teams

Shards can be used by clients, including Nova, to limit the scope of their requests to a logical and declared subset of nodes which prevents multiple nova-compute services from being able to see and work with the same node on multiple nova-compute services.

Note

Shards are client-side constructs - Ironic itself does not use shard values internally.

1.82 Shard support was added in API version 1.82.

Warning

Once set, a shard should not be changed. Nova's model of leveraging the Ironic API does not permit this value to be changed after the fact.

Common Deployment Patterns

Pattern 1: Multi-Region with Complete Isolation

Use Case: Global deployment with regulatory compliance

Implementation:

• Multiple Deployments: ironic-us-east, ironic-eu-west, ironic-apac
• Nova Configuration: Separate compute services per region
• Conductor Groups: Optional within each deployment
• Shards: Operational grouping within regions

Example Nova Configuration:

# nova-compute for US East region
[ironic]
auth_url = https://keystone-us-east.example.com/v3
endpoint_override = https://ironic-us-east.example.com

# nova-compute for EU West region
[ironic]
auth_url = https://keystone-eu-west.example.com/v3
endpoint_override = https://ironic-eu-west.example.com

Note

The above indicated endpoint_override configuration is provided for illustrative purposes to stress endpoints would be distinctly different.

Pattern 2: Single Region with Datacenter Separation

Use Case: Metro deployment across multiple datacenters

Implementation:

• Single Deployment: One Ironic service
• Conductor Groups: datacenter-1, datacenter-2, datacenter-3
• Nova Configuration: Target specific conductor groups
• Shards: Optional operational grouping

In this case, we don't expect BMC management network access to occur between datacenters. Thus each datacenter is configured with it's own group of conductors.

Example Configuration:

# Configure Nova compute to target specific conductor group
[ironic]
conductor_group = datacenter-1

# Configure conductors (ironic.conf)
[conductor]
conductor_group = datacenter-1

# Assign nodes
baremetal node set --conductor-group datacenter-1 <node-uuid>

Note

Some larger operators who leverage conductor groups have suggested that it is sometimes logical to have a conductor set without a conductor_group set. This helps prevent orphaning nodes because Ironic routes all changes to the conductor which presently manages the node.

Pattern 3: Operational Scaling Within Datacenters

Use Case: Large deployment requiring parallel operations

Implementation:

• Single Deployment: One Ironic service
• Conductor Groups: By datacenter or hardware type
• Shards: Operational batches for maintenance/upgrades
• Nova Configuration: May target specific conductor groups

Example:

# Set up conductor groups by hardware
baremetal node set --conductor-group dell-servers <node-uuid-1>
baremetal node set --conductor-group hpe-servers <node-uuid-2>

# Create operational shards for maintenance
baremetal node set --shard maintenance-batch-1 <node-uuid-1>
baremetal node set --shard maintenance-batch-2 <node-uuid-2>

Pattern 4: Hybrid Multi-Tier Approach

Use Case: Complex enterprise deployment

Implementation: All three tiers working together

Example Architecture:

# Deployment 1: Production East Coast
# Nova compute service targets ironic-prod-east
[ironic]
endpoint_override = https://ironic-prod-east.example.com
conductor_group = datacenter-east

# Within this deployment:
baremetal node set --conductor-group datacenter-east --shard prod-batch-a <node-uuid>

# Deployment 2: Production West Coast
# Nova compute service targets ironic-prod-west
[ironic]
endpoint_override = https://ironic-prod-west.example.com
conductor_group = datacenter-west

Nova Integration and Configuration

Targeting Multiple Ironic Deployments

Nova's Ironic driver can be configured to target different Ironic services:

Per-Compute Service Configuration:

# /etc/nova/nova.conf on compute-service-1
[ironic]
auth_url = https://keystone-region1.example.com/v3
endpoint_override = https://ironic-region1.example.com
conductor_group = region1-zone1

# /etc/nova/nova.conf on compute-service-2
[ironic]
auth_url = https://keystone-region2.example.com/v3
endpoint_override = https://ironic-region2.example.com
conductor_group = region2-zone1

Advanced Options:

[ironic]
# Target specific conductor group within deployment
conductor_group = datacenter-east

# Target specific shard within deployment
shard = production-nodes

# Connection retry configuration
api_max_retries = 60
api_retry_interval = 2

Nova Ironic Hypervisor Configuration for complete Nova configuration details.

Scaling Considerations

Nova Compute Service Scaling:

• Single nova-compute can handle several hundred Ironic nodes efficiently.
• Consider multiple compute services for >1000 nodes per deployment. Nova-compute is modeled on keeping a relatively small number of "instances" per nova-compute process. For example, 250 baremetal nodes.
• One nova-compute process per conductor group or shard is expected.
• A conductor_group which is independent of a nova-compute service configuration can be changed at any time. A shard should never be changed once it has been introduced to a nova-compute process.

Multi-Deployment Benefits:

• Independent scaling per deployment
• Isolated failure domains
• Different operational schedules

Integration Considerations

Network Considerations

Ironic's partitioning works alongside physical network configuration:

• Physical networks can span multiple conductor groups
• Consider network topology when designing conductor group boundaries
• Ensure network connectivity between conductors and their assigned nodes

networking for detailed network configuration guidance

Nova Placement and Scheduling

When using Ironic with Nova:

• Nova's availability zones operate independently of Ironic's partitioning
• Use resource classes and traits for capability-based scheduling

../install/configure-nova-flavors for flavor and scheduling configuration

API Client Usage

Working Across Multiple Deployments

When managing multiple Ironic deployments, use separate client configurations:

# Configure client for deployment 1
export OS_AUTH_URL=https://keystone-east.example.com/v3
export OS_ENDPOINT_OVERRIDE=https://ironic-east.example.com
baremetal node list

# Configure client for deployment 2
export OS_AUTH_URL=https://keystone-west.example.com/v3
export OS_ENDPOINT_OVERRIDE=https://ironic-west.example.com
baremetal node list

Filtering by Conductor Group

# List nodes by conductor group
baremetal node list --conductor-group datacenter-east

# List ports by node conductor group
baremetal port list --conductor-group datacenter-east

Filtering by Shard

# List nodes by shard
baremetal node list --shard batch-a

# Get shard distribution
baremetal shard list

# Find nodes without a shard assignment
baremetal node list --unsharded

Combined Filtering Within Deployments

# Within a single deployment, filter by conductor group and shard
baremetal node list --conductor-group datacenter-1 --shard maintenance-batch-a

# Set both conductor group and shard on a node
baremetal node set --conductor-group datacenter-east --shard batch-a <node-uuid>

# Get overview of resource distribution
baremetal shard list
baremetal conductor list

Best Practices

Deployment Strategy Planning

1. Assess isolation requirements: Determine if you need complete service separation
2. Plan geographic distribution: Use multiple deployments for true regional separation
3. Design conductor groups: Align with physical/network boundaries
4. Implement shard strategy: Plan for operational efficiency
5. Configure Nova appropriately: Match Nova compute services to your architecture

Operational Considerations

Multiple Deployments:

• Maintain consistent tooling across deployments
• Plan for cross-deployment migrations if needed
• Monitor each deployment independently
• Coordinate upgrade schedules

Within Deployments:

• Monitor conductor distribution: baremetal shard list
• Ensure conductor redundancy per group
• Align network topology with conductor groups
• Automate shard management for balance

Nova Integration:

• Plan compute service distribution across deployments
• Monitor nova-compute to Ironic node ratios
• Test failover scenarios between compute services

Naming Conventions

Naming patterns can be defined by the infrastructure operator and below are some basic suggestions which may be relevant based upon operational requirements.

Conductor Groups:

• Geographic: datacenter-east, region-us-west, rack-01
• Hardware: dell-servers, hpe-gen10, gpu-nodes
• Network: vlan-100, isolated-network

Shards:

• Operational: maintenance-batch-1, upgrade-group-a
• Size-based: small-nodes, large-memory
• Temporal: weekend-maintenance, business-hours

Decision Matrix

Choose your approach based on requirements:

Requirement	Multiple Deployments	ConductorGroups**	Shards
Complete isolation	✓ Best	✓ Good	✗ No
Independent upgrades	✓ Complete	✓ Partial	✗ No
Geographic separation	✓ Best	✓ Good	✗ No
Operational scaling	✗ Overhead	✓ Good	✓ Best
Resource efficiency	✗ Lower	✓ Good	✓ Best

Troubleshooting

Multiple Deployment Issues

Connectivity Problems:

# Test connectivity to each deployment
baremetal --os-endpoint-override https://ironic-east.example.com node list
baremetal --os-endpoint-override https://ironic-west.example.com node list

Nova Configuration Issues:

# Check Nova compute service registration
openstack compute service list --service nova-compute

# Verify Nova can reach Ironic
grep -i ironic /var/log/nova/nova-compute.log

Cross-Deployment Node Migration:

# Export node data from source deployment
baremetal node show --fields all <node-uuid>

# Import to destination deployment (manual process)
# Note: Requires careful planning and may need custom tooling

Common Issues Within Deployments

Orphaned nodes: Nodes without matching conductor groups cannot be managed

# Find nodes without conductor groups
baremetal node list --conductor-group ""

# List available conductor groups
baremetal conductor list

Unbalanced shards: Monitor node distribution across shards

# Check shard distribution
baremetal shard list

# Find heavily loaded shards
baremetal node list --shard <shard-name> | wc -l

Missing conductor groups: Ensure all groups have active conductors

# Check conductor status
baremetal conductor list

# Verify conductor group configuration
# Check ironic.conf [conductor] conductor_group setting

Migration Scenarios

Moving nodes between conductor groups:

# Move node to different conductor group
baremetal node set --conductor-group new-group <node-uuid>

Reassigning shards:

# Change node shard assignment
baremetal node set --shard new-shard <node-uuid>

# Remove shard assignment
baremetal node unset --shard <node-uuid>

Warning

Shards should never be changed once a nova-compute service has identified a node in Ironic. Changing a shard at this point is an unsupported action. As such, Ironic's API RBAC policy restricts these actions to a "System-Scoped Admin" user. Normal Admin users are denied this capability due the restriction and requirement on the nova-compute side of the consumption of shards.

See Also

• conductor-groups - Detailed conductor group configuration
• networking - Physical network considerations
• ../install/refarch/index - Reference architectures
• multitenancy - Multi-tenant deployments
• tuning - Performance tuning considerations
• Nova Ironic Driver Documentation
• Nova Ironic Configuration Options