openstack
/
kuryr-kubernetes
Code Issues Proposed changes

Make ChunkedEncodingError less scary 

Queens Octavia Amphora has annoying issue of dropping every connection
after 50 seconds of inactivity. As Kuryr watchers connect to K8s API
through Octavia load balancer, this produces ChunkedEncodingError
exceptions all around in the logs. This commit makes K8sClient catch
those exceptions, log a nice warning explaining the situation and
restart the watcher using the last seen resourceVersion.

Change-Id: I18c79164ec18e6dbafc0f489f1428ff3aa72f162
Related-Bug: 1842689
This commit is contained in:
Michał Dulko 2019-09-04 18:56:34 +02:00
parent 51150ed2cd
commit 2ab2fbd5f6
1 changed files with 9 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
kuryr_kubernetes/k8s_client.py
View File

@ -278,3 +278,12 @@ class K8sClient(object):
LOG.warning('%ds without data received from watching %s. '
'Retrying the connection with resourceVersion=%s.',
timeouts[1], path, params.get('resourceVersion'))
except requests.exceptions.ChunkedEncodingError:
LOG.warning("Connection to %s closed when watching. This "
"mostly happens when Octavia's Amphora closes "
"connection due to lack of activity for 50s. "
"Since Rocky Octavia this is configurable and "
"should be set to at least 20m, so check timeouts "
"on Kubernetes API LB listener. Restarting "
"connection with resourceVersion=%s.", path,
params.get('resourceVersion'))