Browse Source

Workaround kubernetes python client deadlock issue

The kubernetes python client has a bug [1] which results in frequent
deadlocks while being cleaned up, which causes armada to hang at the
end of execution.

This patchset works around that issue by mocking out the associated
thread pools, since they are only needed for async kubernetes api
calls, which armada does not use.

[1]: https://github.com/kubernetes-client/python/issues/411

Change-Id: I71fbfbe355347ae2ddd02ffd26d881368320246b
Sean Eagan 4 months ago
parent
commit
673b1ed4bc
1 changed files with 8 additions and 0 deletions
  1. 8
    0
      armada/handlers/k8s.py

+ 8
- 0
armada/handlers/k8s.py View File

@@ -17,7 +17,9 @@ import re
17 17
 from kubernetes import client
18 18
 from kubernetes import config
19 19
 from kubernetes import watch
20
+from kubernetes.client import api_client
20 21
 from kubernetes.client.rest import ApiException
22
+from unittest.mock import Mock
21 23
 from oslo_config import cfg
22 24
 from oslo_log import log as logging
23 25
 
@@ -27,6 +29,12 @@ from armada.exceptions import k8s_exceptions as exceptions
27 29
 CONF = cfg.CONF
28 30
 LOG = logging.getLogger(__name__)
29 31
 
32
+# TODO: Remove after this bug is fixed and we have uplifted to a fixed version:
33
+#       https://github.com/kubernetes-client/python/issues/411
34
+# Avoid creating thread pools in kubernetes api_client.
35
+_dummy_pool = Mock()
36
+api_client.ThreadPool = lambda *args, **kwargs: _dummy_pool
37
+
30 38
 
31 39
 class K8s(object):
32 40
     '''

Loading…
Cancel
Save