Browse Source

Limit qemu-img execution arenas

qemu-img attempts to launch multiple threads by default *and*
attempts to have multiple memory allocation arenas to operate
from. While multithreading can be good for performance, this
pattern and the memory footprint for process launch and
dependencies can turn the memory footprint for a cirros image
conversion (16MB) into 1.2GB of memory being asked for by the
qemu-img tool.

In order to limit this impact, as the default number of arenas
is governed by the number of CPUs times the number 8, it seems
reasonable to lower this to a more reasonable number which
also helps keep our possible memory footprint from being exceeded.

NOTE: This change is largely different than the original change
as an intermediate change converted write_image.sh to python.
As it is unlikely for us to backport the intermediate change,
it is logical for us to just modify the original script.
Otherwise the release note is ultimately what is backported for
release note tooling continutity.

Change-Id: I71a28ec59ec31c691205eb34d9fcab63a2ccb682
Story: 2008928
Task: 42528
(cherry picked from commit 9e4c7052a2fd9aac03858db696bf1ea9487f15e6)
(cherry picked from commit 9c20cca36284a2a17aa535bef92c828096c7d926)
(cherry picked from commit ee8118423b88f61ed58b0420356ceaed2fa53a40)
changes/33/794033/1
Julia Kreger 3 weeks ago
parent
commit
bc849e2f68
2 changed files with 16 additions and 0 deletions
  1. +9
    -0
      ironic_python_agent/shell/write_image.sh
  2. +7
    -0
      releasenotes/notes/limit-qemu-img-malloc-arena-025ed84115481eae.yaml

+ 9
- 0
ironic_python_agent/shell/write_image.sh View File

@ -48,6 +48,15 @@ log "Imaging $IMAGEFILE to $DEVICE"
# limit the memory usage for qemu-img to 2 GiB
ulimit -v 2097152
# NOTE(TheJulia): qemu-img uses multiple threads by default and in
# cross-thread memory allocation lock conflicts, glibc will ultimately
# attempt to provide it with an additional arena to allocate from, however
# the running default, when not overridden is 8 * ncpu * the footprint, which
# very quickly exceeds the ulimit. This is most observable on CI systems where
# cross-vcpu thread locking can result in a conflict that wouldn't normally be
# as likely on physical hardware.
# See discussion on https://bugzilla.redhat.com/show_bug.cgi?id=1892773
export MALLOC_ARENA_MAX=3
qemu-img convert -t directsync -O host_device $IMAGEFILE $DEVICE
sync


+ 7
- 0
releasenotes/notes/limit-qemu-img-malloc-arena-025ed84115481eae.yaml View File

@ -0,0 +1,7 @@
---
fixes:
- |
Fixes failures with disk image conversions which result in memory
allocation or input/output errors due to memory limitations by limiting
the number of available memory allocation pools to a non-dynamic
reasonable number which should not exceed the available system memory.

Loading…
Cancel
Save