openstack/ironic-python-agent
Code Issues Proposed changes

Limit qemu-img execution arenas

Browse Source 
qemu-img attempts to launch multiple threads by default *and*
attempts to have multiple memory allocation arenas to operate
from. While multithreading can be good for performance, this
pattern and the memory footprint for process launch and
dependencies can turn the memory footprint for a cirros image
conversion (16MB) into 1.2GB of memory being asked for by the
qemu-img tool.

In order to limit this impact, as the default number of arenas
is governed by the number of CPUs times the number 8, it seems
reasonable to lower this to a more reasonable number which
also helps keep our possible memory footprint from being exceeded.

Change-Id: I71a28ec59ec31c691205eb34d9fcab63a2ccb682
Story: 2008928
Task: 42528
This commit is contained in:
Julia Kreger 2021-05-26 11:55:44 -07:00
parent 2172122b87
commit 9e4c7052a2
3 changed files with 21 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
ironic_python_agent/extensions/standby.py
View File

@ -206,7 +206,17 @@ def _write_whole_disk_image(image, image_info, device):
# TODO(dtantsur): switch to disk_utils.convert_image when it supports # TODO(dtantsur): switch to disk_utils.convert_image when it supports
# -t and -W flags and defaults to 2 GiB memory limit. # -t and -W flags and defaults to 2 GiB memory limit.
limits = processutils.ProcessLimits(address_space=2048 * units.Mi) limits = processutils.ProcessLimits(address_space=2048 * units.Mi)
utils.execute(*command, prlimit=limits) # TODO(TheJulia): qemu-img uses a default of 8 * nCPU to determine
# how many chunks of memory to allocate based upon the discussion in
# https://bugzilla.redhat.com/show_bug.cgi?id=1892773.
# Setting MALLOC_AREA_MAX=1 results in a process memory footprint of
# ~250mb. Setting it to 3 will allow for multiple areas, but won't
# greatly exceed the footprint.
env_vars = {'MALLOC_ARENA_MAX': '3'}
# Entirely disabling threading results in the same memory footprint
# as 1 areana, but multiple threads are useful for performance
# as the file could be fragmented/compressed.
utils.execute(*command, prlimit=limits, env_variables=env_vars)
except processutils.ProcessExecutionError as e: except processutils.ProcessExecutionError as e:
raise errors.ImageWriteError(device, e.exit_code, e.stdout, e.stderr) raise errors.ImageWriteError(device, e.exit_code, e.stdout, e.stderr)

4
ironic_python_agent/tests/unit/extensions/test_standby.py
View File

@ -182,7 +182,9 @@ class TestStandbyExtension(base.IronicAgentTest):
standby._write_image(image_info, device) standby._write_image(image_info, device)
execute_mock.assert_called_once_with(*command, prlimit=mock.ANY) execute_mock.assert_called_once_with(
*command, prlimit=mock.ANY,
env_variables={'MALLOC_ARENA_MAX': '3'})
wipe_mock.assert_called_once_with(device, '') wipe_mock.assert_called_once_with(device, '')
udev_mock.assert_called_once_with() udev_mock.assert_called_once_with()
rescan_mock.assert_called_once_with(device) rescan_mock.assert_called_once_with(device)

7
releasenotes/notes/limit-qemu-img-malloc-arena-025ed84115481eae.yaml Normal file
View File

@ -0,0 +1,7 @@
---
fixes:
- |
Fixes failures with disk image conversions which result in memory
allocation or input/output errors due to memory limitations by limiting
the number of available memory allocation pools to a non-dynamic
reasonable number which should not exceed the available system memory.