Browse Source

Variable to set HTTP keepalive mode

The HTTP keepalive mode is currently hardcoded to "http-server-close"
for all HTTP services. This disables keepalive for HAProxy to backend
connections, but leaves it enabled for client connections to HAProxy.
This is problematic especially for service to service calls (e.g.
nova-api to neutron). If a request is made at the same time the HAProxy
keepalive timeout expires, the result of the request is undefined. This
leads to code 500 error responses from the nova-api because the request
from nova-api to neutron failed. "Connection aborted" error messages in
the logs are an indication of this issue.

There is also a bug report[1] about the same issue in devstack which was
solved by disabling keepalive and a script[2] to reproduce the issue in
devstack.

This adds a default and per service variables to set the HTTP keepalive
mode used by HAProxy. The default value is changed to "forceclose" to
disable HTTP keepalive on the server and client side. With HTTP
keepalive disabled the issue can no longer be reproduced.

[1] https://bugs.launchpad.net/devstack/+bug/1630664
[2] https://github.com/JordanP/openstack-snippets/blob/master/keepalive-race/keep-alive-race.py

Change-Id: If819912873270f0568974925490023310f9cbd66
Gaudenz Steinlin 1 month ago
parent
commit
e911f3f2d2
3 changed files with 8 additions and 1 deletions
  1. 4
    0
      defaults/main.yml
  2. 1
    0
      templates/haproxy.cfg.j2
  3. 3
    1
      templates/service.j2

+ 4
- 0
defaults/main.yml View File

@@ -103,6 +103,10 @@ haproxy_connect_timeout: "10s"
103 103
 haproxy_http_request_timeout: "5s"
104 104
 # Set the maximum inactivity time on the server side
105 105
 haproxy_server_timeout: "50s"
106
+# Set the HTTP keepalive mode to use
107
+# Disable persistent connections by default because they can cause issues when the server side closes the connection
108
+# at the same time a request is sent.
109
+haproxy_keepalive_mode: 'forceclose'
106 110
 
107 111
 
108 112
 ## haproxy tuning params

+ 1
- 0
templates/haproxy.cfg.j2 View File

@@ -28,6 +28,7 @@ defaults
28 28
         log global
29 29
         option dontlognull
30 30
         option redispatch
31
+        option {{ haproxy_keepalive_mode }}
31 32
         retries {{ haproxy_retries }}
32 33
         timeout client {{ haproxy_client_timeout }}
33 34
         timeout connect {{ haproxy_connect_timeout }}

+ 3
- 1
templates/service.j2 View File

@@ -44,7 +44,9 @@ frontend {{ item.service.haproxy_service_name }}-front-{{ loop.index }}
44 44
 {% if request_option == "http" %}
45 45
     option httplog
46 46
     option forwardfor except 127.0.0.0/8
47
-    option http-server-close
47
+{% if item.service.haproxy_http_keepalive_mode is defined %}
48
+    option {{ item.service.haproxy_http_keepalive_mode }}
49
+{% endif %}
48 50
 {% elif request_option == "tcp" %}
49 51
     option tcplog
50 52
 {% endif %}

Loading…
Cancel
Save