The zuul executor has a number of governors that check system limits
which will result in the executor unregistering itself from performing
future work when near to these limits. One such limit is the number of
processes that is allowed by the runtime system. Add a governor that
will unregister the executor when near to the process limit.
To do this we rely on cgroups' pids.max and pids.current values. There
are two ways these manifest. The first is running within a container. In
this case the values will be present in the root cgroup directory. When
not running a container these values are not present. We could then
attempt to rely on ulimits and process listings, but it appears that
systemd creates a user.slice cgroup for each user and tracks the
pids.max and pids.current (and on my system these match my ulimits) so
we use this instead. Systemd is fairly ubiquitous and this prevents us
from having two different implementations in place.
This is important because running zuul jobs requires a number of
processes. When we are near the process limit starting a new job will
fork ansible and ansible forks ssh which also relies on control
persistence processes and an ssh-agent. Rather than failing the job due
to an inability to fork we should stop running new jobs until we have
backed away from the limit.
Change-Id: Ie24e386680087d7640fac13ceb34b1eb934d5146
4901d4c013