systemd: Set a higher DefaultTasksMax value

systemd-2.28 introduced DefaultTasksMax which is used to control
the default TasksMax= setting for services and scopes running on the
system. (TasksMax= is the primary setting that exposes the "pids"
cgroup controller on systemd and was introduced in the previous
systemd release.) The setting now defaults to 512, which means
services that are not explicitly configured otherwise will only
be able to create 512 processes or threads at maximum, from this
version on. However, the 512 limit seems too strict and sometimes
leads to failures like the following one on busy containers

==> opensuse422: fatal: [container3]: FAILED! => {"changed": false, "cmd": "/usr/sbin/rabbitmqctl -q -n '' list_user_permissions guest", "failed": true, "msg": "/usr/sbin/rabbitmqctl: fork: retry: No child processes\n/usr/lib64/rabbitmq/lib/rabbitmq_server-3.6.6//sbin/rabbitmq-env: fork: retry: Resource temporarily unavailable\n/usr/lib64/rabbitmq/lib/rabbitmq_server-3.6.6//sbin/rabbitmq-env: fork: retry: No child processes\n/usr/lib64/rabbitmq/lib/rabbitmq_server-3.6.6//sbin/rabbitmq-env: fork: retry: No child processes\nFailed to create thread: Resource temporarily unavailable (11)\r\nAborted (core dumped)", "rc": 134, "stderr": "/usr/sbin/rabbitmqctl: fork: retry: No child processes\n/usr/lib64/rabbitmq/lib/rabbitmq_server-3.6.6//sbin/rabbitmq-env: fork: retry: Resource temporarily unavailable\n/usr/lib64/rabbitmq/lib/rabbitmq_server-3.6.6//sbin/rabbitmq-env: fork: retry: No child processes\n/usr/lib64/rabbitmq/lib/rabbitmq_server-3.6.6//sbin/rabbitmq-env: fork: retry: No child processes\nFailed to create thread: Resource temporarily unavailable (11)\r\nAborted (core dumped)\n", "stderr_lines": ["/usr/sbin/rabbitmqctl: fork: retry: No child processes", "/usr/lib64/rabbitmq/lib/rabbitmq_server-3.6.6//sbin/rabbitmq-env: fork: retry: Resource temporarily unavailable", "/usr/lib64/rabbitmq/lib/rabbitmq_server-3.6.6//sbin/rabbitmq-env: fork: retry: No child processes", "/usr/lib64/rabbitmq/lib/rabbitmq_server-3.6.6//sbin/rabbitmq-env: fork: retry: No child processes", "Failed to create thread: Resource temporarily unavailable (11)", "Aborted (core dumped)"], "stdout": "", "stdout_lines": []}

and with messages in the kernel log such as

[ 2925.999021] cgroup: fork rejected by pids controller in /init.scope/lxc/container1
[ 3083.704049] cgroup: fork rejected by pids controller in /init.scope/lxc/container2

As we see, even though the /init.scope/lxc/container1 as pids.max set to 'max', the /init.scope
has pids.max set to 512 and in cgroups we always respect the lowest
boundary

~> cat /sys/fs/cgroup/pids/init.scope/lxc/container1/pids.max
max
~> cat /sys/fs/cgroup/pids/init.scope/pids.max
512

As a result of which, the 512 limit is enforced.

As such, we add a new variable to make this limit configurable. The
default limit has now been increased to 8192.

Change-Id: I8b4143aac84d4c795cab9c0d978c9a97ebea1793
This commit is contained in:
Markos Chandras 2017-06-16 18:52:48 +01:00
parent 68d74b92c6
commit 82406ad958
3 changed files with 24 additions and 0 deletions

View File

@ -31,6 +31,10 @@ lxc_architecture_mapping:
# Set the volume size in gigabytes for the machine image caches.
lxc_host_machine_volume_size: 16
# DefaultTasksMax systemd value. It's not recommended to change this value as it
# could prevent new processes from starting on busy containers.
lxc_default_tasks_max: 8192
# lxc container rootfs directory and cache path
lxc_container_directory: "/var/lib/lxc"
lxc_container_cache_path: "/var/cache/lxc/download"

View File

@ -44,3 +44,10 @@
- name: Veth check
command: "/usr/local/bin/lxc-veth-check"
- name: Reload systemd units
systemd:
# TODO(hwoarang): We only want to reload the units so the 'name'
# parameter can be removed when we move to Ansible 2.4
name: it_does_not_matter
daemon_reload: yes

View File

@ -50,6 +50,19 @@
- lxc-files
- lxc-config
- name: Set systemd DefaultTasksMax value
lineinfile:
dest: /etc/systemd/system.conf
state: present
regexp: "^.*DefaultTasksMax.*$"
line: "DefaultTasksMax={{ lxc_default_tasks_max }} # Managed by Ansible"
when:
- ansible_service_mgr == 'systemd'
notify:
- Reload systemd units
tags:
- lxc-config
# Ensure apparmor reindex runs before other things that may fail
- meta: flush_handlers