openstack/project-config
Code Issues Proposed changes

Nodepool elements: Add a script to initialize urandom

Browse Source 
In our Xenial images, we see unbound take a while to start because
it uses openssl which uses the getrandom call which can block during
early boot if the nonblocking random number generator is not yet
initialized.

This script uses haveged to quickly initialize the generator.

This commit only includes the script, a later commit will add the
rest of the necessary install steps to the element.

Change-Id: I09d18a0bad6c380fd149660ebfdaf6c12730dc74
This commit is contained in:
James E. Blair 2016-08-17 13:28:37 -07:00
parent 4dc6f2bc98
commit 6020537b2c
2 changed files with 272 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
nodepool/elements/initialize-urandom/README.rst Normal file
View File

@ -0,0 +1,2 @@
This uses haveged to quickly initialize the nonblocking kernel random
number generator at boot.

270
nodepool/elements/initialize-urandom/static/usr/local/bin/initialize-urandom.py Normal file
View File

@ -0,0 +1,270 @@
#!/usr/bin/env python
# Copyright 2016 Red Hat, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import ctypes
import errno
import fcntl
import os
import struct
import subprocess
"""Add entropy to the kernel until the nonblocking pool is
initialized.
The Linux kernel has 3 entropy pools: input, blocking, and
nonblocking. Normally entropy accumulates in the input pool and as it
is depleted by the other pools, it is transferred from the input pool
to the others.
The blocking pool corresponds to /dev/random, where reads from that
device return random numbers only as long as there is sufficient
entropy in the blocking pool. When that entropy is depleted, further
reads from /dev/random block until it is replenished.
The nonblocking pool corresponds to /dev/urandom, where reads never
block. Even if there is no entropy in the nonblocking pool, random
numbers are still returned.
The algorithms in use in Linux 3.17 require 128 bits of entropy in
order to initialize the random number generators associated with each
pool. Naturally, reads from /dev/random will not return until the
associated generator is initialized. Reads from /dev/urandom will not
block -- even if the generator is not initialized. The kernel will
output a notice[1] if this happens.
In order to avoid the situation where urandom is used when
uninitialized, the kernel diverts entropy from timers and interrupts
to the nonblocking pool (instead of the input pool) until it is
initialized. In this way, as the system boots, the nonblocking pool
accumulates entropy first, reducing the time period during which
urandom might produce numbers from an uninitialized generator, and
then the input and blocking pools are filled.
Beginning with Linux 3.17, the getrandom(2) syscall was added[2] so
that user-space programs that generally would like to use /dev/urandom
can do so without opening a file descriptor and, more relevant here,
can ensure that they do so only after the generator is initialized
(which otherwise is not possible with the /dev/urandom interface).
Unfortunately, programs which use this interface during early boot may
need to wait some time for the nonblocking pool to accumulate enough
entropy to initialize, and therefore for getrandom to return.
Particularly in the case of a VM, this may take considerable time.
There are many methods of addressing this shortcoming:
* Store data from /dev/random at shutdown and use it to seed the
entropy pool at the next boot. Most GNU/Linux distributions do
this. On Ubuntu Xenial, this task is performed by systemd[3].
Unfortunately, while writes to /dev/random (which is the method
systemd uses to seed the system at boot) do add data to the pool,
they do not increase the internal tracking of the amount of entropy
in the pool. Therefore, for the purposes of determining whether the
nonblocking pool has accumulated 128 bits of entropy, they are not
counted.
* Use haveged to maintain a sufficient amount of entropy. Haveged can
produce entropy very quickly, and when run at boot, will typically
immediately fill the entropy pool. Haveged performs an ioctl
operation on /dev/random rather than writing data to it, and this
ioctl allows it to specify how much entropy the data it supplies
contains. Therefore, unlike writes to /dev/random, ioctls do
increment the entropy counter. Unfortunately, data from ioctls are
*always* directed to the input pool. While entropy from timers and
interrupts are diverted to the nonblocking pool to speed its
initialization, data arriving from the ioctl instead end up in the
input pool for later use.
When more entropy than is needed is supplied to the input pool, the
kernel will preemptively transfer some of that entropy to the
secondary (including nonblocking) pools. Since haveged supplies so
much data on startup, some of this entropy should be able to spill
over into the nonblocking pool to aid it in achieving the
initialization threshold. Unfortunately, at the stage of early boot
we are considering, the input pool's generator also has not been
initialized. When the kernel receives a large amount of data from
haveged over the ioctl, it pushes the input pool's generator over
the 128 bit threshold, and initializes the input pool's generator.
When a pool's generator is initialized, the entropy counter for that
pool is reset to zero. This leaves no entropy to spill over to the
nonblocking pool. Haveged is only able to see the entropy count for
the input pool, and therefore is unaware that further contributions
of entropy would aid (via spill-over) in seeding the nonblocking
pool.
At this point it's worth discussing why the nonblocking pool is
still not initialized despite a full input pool. When a secondary
pool needs more entropy, it can pull from the input pool. However,
there is a timer that only allows the nonblocking pool to withdraw
entropy from the input pool every 60 seconds by default (this can be
adjusted via proc). If something during very early boot reads data
from /dev/urandom, a transfer (from the very likely empty) input
pool is initiated, starting the timer that will prevent another
transfer for 60 seconds, even if the input pool is later filled
(such as by haveged). This means that even with haveged running at
boot the delay due to a blocking getrandom(2) call may still be as
long as 60 seconds.
* Use rng-tools for the same purpose as haveged. rng-tools operates
in a similar manner to haveged, supplying entropic data to the
kernel via ioctl. However, it does so in smaller chunks. This
means that once the input pool's generator surpasses the 128 bit
threshold for initialization, entropy from the next ioctl from
rng-tools will be available to spill over to the nonblocking pool,
and may be sufficient to initialize it.
Because of this behavior, use of rng-tools may cause getrandom(2) to
return more quickly at boot, however, this may only happen due to a
quirk of implementation and relies on some specific values and
conditions for the amount of entropy in the input pool at the time
it is run.
This program speeds initialization of the nonblocking pool by adding
entropy to the input pool in small chunks. To determine when the
nonblocking pool is initialized, it performs the nonblocking
getrandom(2) syscall requesting one byte of random data. As long as
the nonblocking pool is uninitialized, that call will fail and set
errno to EAGAIN. In that case, the program reads 64 bytes of data
from haveged and sends it to the kernel using the ioctl interface,
then repeats this in a loop. That will cause entropy to accumulate in
the input pool until it is initialized and reaches the spill-over
threshold. Further data will accumulate in the nonblocking pool until
it is initialized. Once that occurs, the getrandom(2) call will
return successfully, and the program will exit the loop.
There are other ways this problem could be addressed (changes to
haveged or rng-tools to support behavior like this, or changes to the
kernel to direct entropy received via ioctl to the nonblocking pool
during initialization), however, this problem is likely to be
short-lived as the nonblocking generator is being replaced[4] in
current kernel versions and should not suffer from the same problem.
[1] http://lxr.free-electrons.com/source/drivers/char/random.c?v=3.17#L1385
[2] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c6e9d6f38894798696f23c8084ca7edbf16ee895
[3] https://www.freedesktop.org/software/systemd/man/systemd-random-seed.service.html
[4] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=e192be9d9a30555aae2ca1dc3aad37cba484cd4a
"""
class GeneratorNotInitializedError(Exception):
pass
class InterruptedError(Exception):
pass
class Pump(object):
# How much data, in bytes, to move at once. 64 is the size of the
# internal kernel buffer, so we match it.
CHUNK_SIZE = 64
# The syscall number for getrandom(2).
SYS_getrandom = 318
# The IOCTL to add entropy.
OP_RNDADDENTROPY = 0x40085203
# Flags for getrandom:
GRND_NONBLOCK = 0x0001 # Do not block
GRND_RANDOM = 0x0002 # Use /dev/random instead of urandom
def __init__(self):
# Use ctypes to invoke getrandom since it is not available in
# python. os.urandom may call getrandom in some versions of
# python3, however, the blocking on initialization behavior is
# seen as a bug and so os.urandom will never block, even if
# getrandom would. See http://bugs.python.org/issue26839
self._getrandom = ctypes.CDLL(None, use_errno=True).syscall
self._getrandom.restype = ctypes.c_long
# The arguments are syscall number, void *buf,
# size_t buflen, unsigned int flags.
self._getrandom.argtypes = (ctypes.c_long, ctypes.c_void_p,
ctypes.c_size_t, ctypes.c_uint)
def getrandom(self, length, random=False, nonblock=False):
flags = 0
if random:
flags |= self.GRND_RANDOM
if nonblock:
flags |= self.GRND_NONBLOCK
buf = ctypes.ARRAY(ctypes.c_char, length)()
r = self._getrandom(self.SYS_getrandom, buf, len(buf), flags)
if r == -1:
err = ctypes.get_errno()
if err == errno.EINVAL:
raise Exception("getrandom: Invalid argument")
elif err == errno.EFAULT:
raise Exception("getrandom: Buffer is outside "
"accessible address space")
elif err == errno.EAGAIN:
raise GeneratorNotInitializedError()
elif err == errno.EINTR:
raise InterruptedError()
return buf[:r]
def isInitialized(self):
# Read one byte from getrandom to determine whether the
# nonblocking pool is initialized.
try:
r = self.getrandom(1, nonblock=True)
if len(r) != 1:
raise Exception("No data returned from getrandom")
print("Nonblocking pool initilaized")
return True
except GeneratorNotInitializedError:
return False
def run(self):
"""Move data from haveged to the kernel until the nonblocking pool is
initialized.
"""
if self.isInitialized():
return
random_fd = os.open('/dev/random', os.O_RDWR)
# Start haveged and tell it to supply unlimited data on
# stdout, and print summary information.
p = subprocess.Popen(['/usr/sbin/haveged', '-f', '-', '-n', '0',
'-v', '1'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
while not self.isInitialized():
# Read a chunk from haveged.
data = b''
while len(data) < self.CHUNK_SIZE:
data += p.stdout.read(self.CHUNK_SIZE - len(data))
# The data structure is:
# struct rand_pool_info {
# int entropy_count;
# int buf_size;
# __u32 buf[0];
# };
arg = struct.pack('iis', len(data) * 8, len(data), data)
print("Moving %s bytes" % len(data))
fcntl.ioctl(random_fd, self.OP_RNDADDENTROPY, arg)
# Now that the generator is initialized, stop haveged and
# print the summary information.
p.send_signal(2)
p.stdout.read()
print(p.stderr.read().decode('utf-8'))
if __name__ == '__main__':
p = Pump()
p.run()