mistral: configure heartbeat parameters to avoid action timeout

This patch configures and increases the defaults heartbeat parameters in
Mistral so we don't hit timeouts when an action in a workflow takes
times to reply back in Mistral, when deploying an Overcloud.

Parameters added:

  MistralMaxMissedHeartbeats:
    type: number
    default: 15
    description: >
        The maximum amount of missed heartbeats to be allowed.
        If set to 0 then this feature is disabled. See check_interval for more
        details.
    constraints:
      - range: { min: 0 }
  MistralCheckInterval:
    type: number
    default: 20
    description: >
        How often (in seconds) action executions are checked.
        For example when check_interval is 10, check action
        executions every 10 seconds. When the checker runs it will
        transit all running action executions to error if the last
        heartbeat received is older than 10 * max_missed_heartbeats
        seconds. If set to 0 then this feature is disabled.
    constraints:
      - range: { min: 0 }
  MistralFirstHeartbeatTimeout:
    type: number
    default: 3600
    description: >
        The first heartbeat is handled differently, to provide a
        grace period in case there is no available executor to handle
        the action execution. For example when
        first_heartbeat_timeout = 3600, wait 3600 seconds before
        closing the action executions that never received a heartbeat.
    constraints:
      - range: { min: 0 }

Configuration applied to Undercloud:
Maximum missed heartbeats: 30 seconds
Time between interval checks: 40 seconds
First Heartbeat timeout after 7200 seconds

Depends-On: I7a2313bed58485e077ae210d222902f4f997f0f0
Change-Id: Id8663e76b61c9e09547c228da226b706383a3e20
Closes-Bug: #1821611
This commit is contained in:
Emilien Macchi 2019-03-25 15:48:47 -04:00
parent d89203603e
commit 374fafd66a
3 changed files with 45 additions and 0 deletions

View File

@ -71,6 +71,38 @@ parameters:
Messaging client subscriber parameter to specify Messaging client subscriber parameter to specify
an SSL connection to the messaging host. an SSL connection to the messaging host.
type: string type: string
MistralMaxMissedHeartbeats:
type: number
default: 15
description: >
The maximum amount of missed heartbeats to be allowed.
If set to 0 then this feature is disabled. See check_interval for more
details.
constraints:
- range: { min: 0 }
MistralCheckInterval:
type: number
default: 20
description: >
How often (in seconds) action executions are checked.
For example when check_interval is 10, check action
executions every 10 seconds. When the checker runs it will
transit all running action executions to error if the last
heartbeat received is older than 10 * max_missed_heartbeats
seconds. If set to 0 then this feature is disabled.
constraints:
- range: { min: 0 }
MistralFirstHeartbeatTimeout:
type: number
default: 3600
description: >
The first heartbeat is handled differently, to provide a
grace period in case there is no available executor to handle
the action execution. For example when
first_heartbeat_timeout = 3600, wait 3600 seconds before
closing the action executions that never received a heartbeat.
constraints:
- range: { min: 0 }
conditions: conditions:
service_debug_unset: {equals : [{get_param: MistralDebug}, '']} service_debug_unset: {equals : [{get_param: MistralDebug}, '']}
@ -103,6 +135,9 @@ outputs:
- {get_param: Debug } - {get_param: Debug }
- {get_param: MistralDebug } - {get_param: MistralDebug }
mistral::rpc_response_timeout: 120 mistral::rpc_response_timeout: 120
mistral::max_missed_heartbeats: {get_param: MistralMaxMissedHeartbeats}
mistral::check_interval: {get_param: MistralCheckInterval}
mistral::first_heartbeat_timeout: {get_param: MistralFirstHeartbeatTimeout}
mistral::keystone::authtoken::project_name: 'service' mistral::keystone::authtoken::project_name: 'service'
mistral::keystone::authtoken::user_domain_name: 'Default' mistral::keystone::authtoken::user_domain_name: 'Default'
mistral::keystone::authtoken::project_domain_name: 'Default' mistral::keystone::authtoken::project_domain_name: 'Default'

View File

@ -129,6 +129,10 @@ parameter_defaults:
MistralExecutionFieldSizeLimit: 16384 MistralExecutionFieldSizeLimit: 16384
MistralExecutorVolumes: MistralExecutorVolumes:
- /var/lib/config-data/nova/etc/nova:/etc/nova:ro - /var/lib/config-data/nova/etc/nova:/etc/nova:ro
# https://bugs.launchpad.net/tripleo/+bug/1821611
MistralMaxMissedHeartbeats: 30
MistralCheckInterval: 40
MistralFirstHeartBeatTimeout: 7200
NeutronServicePlugins: router,segments NeutronServicePlugins: router,segments
NeutronMechanismDrivers: ['openvswitch', 'baremetal'] NeutronMechanismDrivers: ['openvswitch', 'baremetal']
NeutronNetworkVLANRanges: 'physnet1:1000:2999' NeutronNetworkVLANRanges: 'physnet1:1000:2999'

View File

@ -0,0 +1,6 @@
---
fixes:
- |
Allow to configure Mistral parameters for Heartbeat and set sane defaults
for the Undercloud so we can deploy an Overcloud in Stein.
Fixes `bug 1821611 <https://bugs.launchpad.net/tripleo/+bug/1821611>`__.