openstack/openstack-ansible-haproxy_server
Code Issues Proposed changes

Variable to set HTTP keepalive mode

Browse Source 
The HTTP keepalive mode is currently hardcoded to "http-server-close"
for all HTTP services. This disables keepalive for HAProxy to backend
connections, but leaves it enabled for client connections to HAProxy.
This is problematic especially for service to service calls (e.g.
nova-api to neutron). If a request is made at the same time the HAProxy
keepalive timeout expires, the result of the request is undefined. This
leads to code 500 error responses from the nova-api because the request
from nova-api to neutron failed. "Connection aborted" error messages in
the logs are an indication of this issue.

There is also a bug report[1] about the same issue in devstack which was
solved by disabling keepalive and a script[2] to reproduce the issue in
devstack.

This adds a default and per service variables to set the HTTP keepalive
mode used by HAProxy. The default value is changed to "forceclose" to
disable HTTP keepalive on the server and client side. With HTTP
keepalive disabled the issue can no longer be reproduced.

[1] https://bugs.launchpad.net/devstack/+bug/1630664
[2] https://github.com/JordanP/openstack-snippets/blob/master/keepalive-race/keep-alive-race.py

Change-Id: If819912873270f0568974925490023310f9cbd66
This commit is contained in:
Gaudenz Steinlin 2019-03-04 18:08:33 +01:00
parent ca23ec42ed
commit e911f3f2d2
3 changed files with 8 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
defaults/main.yml
View File

@ -103,6 +103,10 @@ haproxy_connect_timeout: "10s"
haproxy_http_request_timeout: "5s"
# Set the maximum inactivity time on the server side
haproxy_server_timeout: "50s"
# Set the HTTP keepalive mode to use
# Disable persistent connections by default because they can cause issues when the server side closes the connection
# at the same time a request is sent.
haproxy_keepalive_mode: 'forceclose'
## haproxy tuning params

1
templates/haproxy.cfg.j2
View File

@ -28,6 +28,7 @@ defaults
log global
option dontlognull
option redispatch
option {{ haproxy_keepalive_mode }}
retries {{ haproxy_retries }}
timeout client {{ haproxy_client_timeout }}
timeout connect {{ haproxy_connect_timeout }}

4
templates/service.j2
View File

@ -44,7 +44,9 @@ frontend {{ item.service.haproxy_service_name }}-front-{{ loop.index }}
{% if request_option == "http" %}
option httplog
option forwardfor except 127.0.0.0/8
option http-server-close
{% if item.service.haproxy_http_keepalive_mode is defined %}
option {{ item.service.haproxy_http_keepalive_mode }}
{% endif %}
{% elif request_option == "tcp" %}
option tcplog
{% endif %}