Disable timeouts on the log streaming connection

We set a 5 second timeout on our log callback connection from the
executor to the job node. This has created problems when there are quiet
periods during logging as there may be no data sent on the stream for
more than 5 seconds. The timeout is valuable when setting up the
initial connection as it quickly catches underlying network problems,
but is less useful when streaming the logs after connecting.

Address this by disabling the timeout once we are connected.

Change-Id: I66ab17c55d54a3ae0e3f4572bed72826315b4b22
This commit is contained in:
Clark Boylan
2018-06-05 10:46:08 -07:00
parent 57fa349ebf
commit fdf82d6418

View File

@@ -121,6 +121,11 @@ class CallbackModule(default.CallbackModule):
while True:
try:
s = socket.create_connection((ip, LOG_STREAM_PORT), 5)
# Disable the socket timeout after we have successfully
# connected to accomodate the fact that jobs may not be writing
# logs continously. Without this we can easily trip the 5
# second timeout.
s.settimeout(None)
except socket.timeout:
self._log(
"Timeout exception waiting for the logger. "