Performance: Run common role in a separate play

The common role was previously added as a dependency to all other roles.
It would set a fact after running on a host to avoid running twice. This
had the nice effect that deploying any service would automatically pull
in the common services for that host. When using tags, any services with
matching tags would also run the common role. This could be both
surprising and sometimes useful.

When using Ansible at large scale, there is a penalty associated with
executing a task against a large number of hosts, even if it is skipped.
The common role introduces some overhead, just in determining that it
has already run.

This change extracts the common role into a separate play, and removes
the dependency on it from all other roles. New groups have been added
for cron, fluentd, and kolla-toolbox, similar to other services. This
changes the behaviour in the following ways:

* The common role is now run for all hosts at the beginning, rather than
  prior to their first enabled service
* Hosts must be in the necessary group for each of the common services
  in order to have that service deployed. This is mostly to avoid
  deploying on localhost or the deployment host
* If tags are specified for another service e.g. nova, the common role
  will *not* automatically run for matching hosts. The common tag must
  be specified explicitly

The last of these is probably the largest behaviour change. While it
would be possible to determine which hosts should automatically run the
common role, it would be quite complex, and would introduce some
overhead that would probably negate the benefit of splitting out the
common role.

Partially-Implements: blueprint performance-improvements

Change-Id: I6a4676bf6efeebc61383ec7a406db07c7a868b2a
This commit is contained in:
Mark Goddard 2020-06-30 11:05:01 +01:00
parent 904f1c9bd9
commit 56ae2db7ac
78 changed files with 159 additions and 260 deletions

View File

@ -20,10 +20,19 @@ localhost ansible_connection=local
# You can explicitly specify which hosts run each project by updating the
# groups in the sections below. Common services are grouped together.
[common:children]
control
network
compute
storage
monitoring
[chrony-server:children]
haproxy
[chrony:children]
control
network
compute
storage
@ -250,6 +259,19 @@ control
# function appropriately. For example, neutron-metadata-agent must run on the
# same host as the l3-agent and (depending on configuration) the dhcp-agent.
# Common
[cron:children]
common
[fluentd:children]
common
[kolla-logs:children]
common
[kolla-toolbox:children]
common
# Elasticsearch Curator
[elasticsearch-curator:children]
elasticsearch

View File

@ -44,6 +44,14 @@ control
# You can explicitly specify which hosts run each project by updating the
# groups in the sections below. Common services are grouped together.
[common:children]
control
network
compute
storage
monitoring
[chrony-server:children]
haproxy
@ -269,6 +277,19 @@ control
# function appropriately. For example, neutron-metadata-agent must run on the
# same host as the l3-agent and (depending on configuration) the dhcp-agent.
# Common
[cron:children]
common
[fluentd:children]
common
[kolla-logs:children]
common
[kolla-toolbox:children]
common
# Elasticsearch Curator
[elasticsearch-curator:children]
elasticsearch

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,14 +1,10 @@
---
project_name: "common"
# Due to the way we do our inventory, ansible does not pick up on the fact that
# this role has already run. We can track what has run with host facts.
common_run: False
common_services:
fluentd:
container_name: fluentd
group: all
group: fluentd
enabled: "{{ enable_fluentd | bool }}"
image: "{{ fluentd_image_full }}"
environment:
@ -17,7 +13,7 @@ common_services:
dimensions: "{{ fluentd_dimensions }}"
kolla-toolbox:
container_name: kolla_toolbox
group: all
group: kolla-toolbox
enabled: True
image: "{{ kolla_toolbox_image_full }}"
environment:
@ -29,7 +25,7 @@ common_services:
# DUMMY_ENVIRONMENT is needed because empty environment is not supported
cron:
container_name: cron
group: all
group: cron
enabled: True
image: "{{ cron_image_full }}"
environment:

View File

@ -5,6 +5,7 @@
action: "create_volume"
common_options: "{{ docker_common_options }}"
name: "kolla_logs"
when: inventory_hostname in groups['kolla-logs']
- name: Link kolla_logs volume to /var/log/kolla
become: true
@ -12,3 +13,4 @@
src: "{{ docker_runtime_directory or '/var/lib/docker' }}/volumes/kolla_logs/_data"
path: /var/log/kolla
state: link
when: inventory_hostname in groups['kolla-logs']

View File

@ -11,7 +11,7 @@
privileged: "{{ item.value.privileged | default(False) }}"
environment: "{{ item.value.environment }}"
when:
- item.value.enabled | bool
- item.value | service_enabled_and_mapped_to_host
with_dict: "{{ common_services }}"
notify:
- "Restart {{ item.key }} container"

View File

@ -1,30 +1,32 @@
---
- name: Ensuring config directories exist
vars:
service_name: "{{ item.0.service_name }}"
service: "{{ common_services[service_name] }}"
file:
path: "{{ node_config_directory }}/{{ item }}"
path: "{{ node_config_directory }}/{{ item.1 }}"
state: "directory"
owner: "{{ config_owner_user }}"
group: "{{ config_owner_group }}"
mode: "0770"
become: true
with_items:
- "kolla-toolbox"
- "cron"
- "cron/logrotate"
- name: Ensuring fluentd config directories exist
file:
path: "{{ node_config_directory }}/{{ item }}"
state: "directory"
mode: "0770"
become: true
with_items:
- "fluentd"
- "fluentd/input"
- "fluentd/output"
- "fluentd/format"
- "fluentd/filter"
when: enable_fluentd | bool
with_subelements:
- - service_name: "cron"
paths:
- "cron"
- "cron/logrotate"
- service_name: "fluentd"
paths:
- "fluentd"
- "fluentd/input"
- "fluentd/output"
- "fluentd/format"
- "fluentd/filter"
- service_name: "kolla-toolbox"
paths:
- "kolla-toolbox"
- paths
when: service | service_enabled_and_mapped_to_host
- name: Ensure fluentd image is present for label check
vars:
@ -35,7 +37,7 @@
action: "ensure_image"
common_options: "{{ docker_common_options }}"
image: "{{ service.image }}"
when: enable_fluentd | bool
when: service | service_enabled_and_mapped_to_host
- name: Fetch fluentd image labels
vars:
@ -45,12 +47,12 @@
docker_image_info:
name: "{{ service.image }}"
register: fluentd_labels
when: enable_fluentd | bool
when: service | service_enabled_and_mapped_to_host
- name: Set fluentd facts
set_fact:
fluentd_binary: "{{ fluentd_labels.images.0.ContainerConfig.Labels.fluentd_binary }}"
when: enable_fluentd | bool
when: common_services.fluentd | service_enabled_and_mapped_to_host
- include_tasks: copy-certs.yml
when:
@ -62,7 +64,7 @@
dest: "{{ node_config_directory }}/{{ item.key }}/config.json"
mode: "0660"
become: true
when: item.value.enabled | bool
when: item.value | service_enabled_and_mapped_to_host
with_dict: "{{ common_services }}"
notify:
- "Restart {{ item.key }} container"
@ -74,8 +76,7 @@
run_once: True
register: find_custom_fluentd_inputs
delegate_to: localhost
when:
- enable_fluentd | bool
when: common_services.fluentd | service_enabled_and_mapped_to_host
- name: Copying over fluentd input config files
vars:
@ -86,7 +87,7 @@
mode: "0660"
become: true
when:
- enable_fluentd | bool
- common_services.fluentd | service_enabled_and_mapped_to_host
- item ~ '.conf' not in customised_input_files
with_items:
- "00-global"
@ -107,7 +108,7 @@
dest: "{{ node_config_directory }}/fluentd/input/{{ item.path | basename }}"
mode: "0660"
when:
- enable_fluentd | bool
- common_services.fluentd | service_enabled_and_mapped_to_host
with_items: "{{ find_custom_fluentd_inputs.files }}"
notify:
- Restart fluentd container
@ -117,6 +118,8 @@
log_direct_to_elasticsearch: "{{ ( enable_elasticsearch | bool or
( elasticsearch_address != kolla_internal_vip_address )) and
not enable_monasca | bool }}"
when:
- common_services.fluentd | service_enabled_and_mapped_to_host
- name: Find custom fluentd output config files
find:
@ -126,7 +129,7 @@
register: find_custom_fluentd_outputs
delegate_to: localhost
when:
- enable_fluentd | bool
- common_services.fluentd | service_enabled_and_mapped_to_host
- name: Copying over fluentd output config files
vars:
@ -137,7 +140,7 @@
mode: "0660"
become: true
when:
- enable_fluentd | bool
- common_services.fluentd | service_enabled_and_mapped_to_host
- item.enabled | bool
- item.name ~ '.conf' not in customised_output_files
with_items:
@ -156,7 +159,7 @@
state: "absent"
become: true
when:
- enable_fluentd | bool
- common_services.fluentd | service_enabled_and_mapped_to_host
- item.disable | bool
with_items:
- name: "02-monasca"
@ -173,7 +176,7 @@
mode: "0660"
become: true
when:
- enable_fluentd | bool
- common_services.fluentd | service_enabled_and_mapped_to_host
with_items: "{{ find_custom_fluentd_outputs.files }}"
notify:
- Restart fluentd container
@ -186,7 +189,7 @@
register: find_custom_fluentd_format
delegate_to: localhost
when:
- enable_fluentd | bool
- common_services.fluentd | service_enabled_and_mapped_to_host
- name: Copying over fluentd format config files
vars:
@ -200,7 +203,7 @@
- "apache_access"
- "wsgi_access"
when:
- enable_fluentd | bool
- common_services.fluentd | service_enabled_and_mapped_to_host
- item ~ '.conf' not in customised_format_files
notify:
- Restart fluentd container
@ -211,7 +214,7 @@
dest: "{{ node_config_directory }}/fluentd/format/{{ item.path | basename }}"
mode: "0660"
when:
- enable_fluentd | bool
- common_services.fluentd | service_enabled_and_mapped_to_host
with_items: "{{ find_custom_fluentd_format.files }}"
notify:
- Restart fluentd container
@ -223,7 +226,7 @@
run_once: True
register: find_custom_fluentd_filters
delegate_to: localhost
when: enable_fluentd | bool
when: common_services.fluentd | service_enabled_and_mapped_to_host
- name: Copying over fluentd filter config files
vars:
@ -242,7 +245,7 @@
- src: 02-parser
dest: 02-parser
when:
- enable_fluentd | bool
- common_services.fluentd | service_enabled_and_mapped_to_host
- item.src ~ '.conf' not in customised_filter_files
notify:
- Restart fluentd container
@ -254,7 +257,8 @@
mode: "0660"
become: true
with_items: "{{ find_custom_fluentd_filters.files }}"
when: enable_fluentd | bool
when:
- common_services.fluentd | service_enabled_and_mapped_to_host
notify:
- Restart fluentd container
@ -266,7 +270,7 @@
become: true
with_items:
- "fluentd"
when: enable_fluentd | bool
when: common_services.fluentd | service_enabled_and_mapped_to_host
notify:
- Restart fluentd container
@ -276,7 +280,9 @@
dest: "{{ node_config_directory }}/cron/logrotate/{{ item.name }}.conf"
mode: "0660"
become: true
when: item.enabled | bool
when:
- common_services.cron | service_enabled_and_mapped_to_host
- item.enabled | bool
with_items:
- { name: "ansible", enabled: "yes" }
- { name: "aodh", enabled: "{{ enable_aodh }}" }
@ -348,7 +354,9 @@
content: "{{ rabbitmq_cluster_cookie }}"
dest: "{{ node_config_directory }}/kolla-toolbox/rabbitmq-erlang.cookie"
mode: "0660"
when: enable_rabbitmq | bool
when:
- common_services['kolla-toolbox'] | service_enabled_and_mapped_to_host
- enable_rabbitmq | bool
notify:
- Restart kolla-toolbox container
@ -361,7 +369,7 @@
mode: "0770"
ignore_errors: "{{ ansible_check_mode }}"
when:
- item.value.enabled | bool
- item.value | service_enabled_and_mapped_to_host
- item.key != "kolla-toolbox"
with_dict: "{{ common_services }}"
@ -374,7 +382,8 @@
mode: "0600"
become: true
when:
api_address_family == "ipv6"
- common_services['kolla-toolbox'] | service_enabled_and_mapped_to_host
- api_address_family == "ipv6"
- name: Copy rabbitmq erl_intr to kolla toolbox
copy:
@ -384,7 +393,8 @@
mode: "0600"
become: true
when:
api_address_family == "ipv6"
- common_services['kolla-toolbox'] | service_enabled_and_mapped_to_host
- api_address_family == "ipv6"
- include_tasks: check-containers.yml
when: kolla_action != "config"

View File

@ -1,10 +1,2 @@
---
- include_tasks: "{{ kolla_action }}.yml"
tags: common
when: not common_run
- name: Registering common role has run
set_fact:
common_run: True
tags: common
when: not common_run

View File

@ -6,5 +6,5 @@
common_options: "{{ docker_common_options }}"
image: "{{ item.value.image }}"
when:
- item.value.enabled | bool
- item.value | service_enabled_and_mapped_to_host
with_dict: "{{ common_services }}"

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -1,3 +0,0 @@
---
dependencies:
- { role: common }

View File

@ -92,6 +92,19 @@
roles:
- role: prechecks
- name: Apply role common
gather_facts: false
hosts:
- cron
- fluentd
- kolla-logs
- kolla-toolbox
serial: '{{ kolla_serial|default("0") }}'
tags:
- common
roles:
- role: common
- name: Apply role chrony
gather_facts: false
hosts:

View File

@ -0,0 +1,20 @@
---
features:
- |
Extracts the common role into a separate play. This provides a performance
benefit at scale, since the role dependency mechanism used previously had a
overhead. This change allows the only common role to be executed by
specifying the ``common`` tag.
upgrade:
- |
The common role is now executed in a separate play. This introduces a few
small changes in behaviour:
* the common role is now run for all hosts at the beginning, rather than
prior to their first enabled service
* hosts must be in the necessary group for each of the common services
(``cron``, ``fluentd``, ``kolla-logs``, ``kolla-toolbox``) in order to
have that service deployed
* if tags are specified for another service e.g. nova, the common role
will *not* automatically run for matching hosts. The common tag must
be specified explicitly

View File

@ -58,6 +58,14 @@ control
# You can explicitly specify which hosts run each project by updating the
# groups in the sections below. Common services are grouped together.
[common:children]
control
network
compute
storage
monitoring
[chrony-server:children]
haproxy
@ -283,6 +291,19 @@ control
# function appropriately. For example, neutron-metadata-agent must run on the
# same host as the l3-agent and (depending on configuration) the dhcp-agent.
# Common
[cron:children]
common
[fluentd:children]
common
[kolla-logs:children]
common
[kolla-toolbox:children]
common
# Elasticsearch Curator
[elasticsearch-curator:children]
elasticsearch

View File

@ -339,12 +339,12 @@ case "$1" in
;;
(mariadb_recovery)
ACTION="Attempting to restart mariadb cluster"
EXTRA_OPTS="$EXTRA_OPTS -e kolla_action=deploy -e common_run=true"
EXTRA_OPTS="$EXTRA_OPTS -e kolla_action=deploy"
PLAYBOOK="${BASEDIR}/ansible/mariadb_recovery.yml"
;;
(mariadb_backup)
ACTION="Backup MariaDB databases"
EXTRA_OPTS="$EXTRA_OPTS -e kolla_action=backup -e mariadb_backup_type=${BACKUP_TYPE} -e common_run=true"
EXTRA_OPTS="$EXTRA_OPTS -e kolla_action=backup -e mariadb_backup_type=${BACKUP_TYPE}"
PLAYBOOK="${BASEDIR}/ansible/mariadb_backup.yml"
;;
(destroy)