Commit Graph

54 Commits (65e132a734f005f090a384bfa129482d195c6d6e)

Author SHA1 Message Date
Michael Johnson 06ce4777c3 Fix multi-listener load balancers
Load balancers with multiple listeners, running on an amphora image
with HAProxy 1.8 or newer can experience excessive memory usage that
may lead to an ERROR provisioning_status.
This patch resolves this issue by consolidating the listeners into
a single haproxy process inside the amphora.

Story: 2005412
Task: 34744
Co-Authored-By: Adam Harwell <>
Change-Id: Idaccbcfa0126f1e26fbb3ad770c65c9266cfad5b
4 years ago
Zuul 6e6f3a1f1c Merge "Fix TCP listener logging bug" 4 years ago
Michael Johnson 83af850ec2 Fix TCP listener logging bug
HAProxy is not handling two of the HTTP log format variables correct
when the load balancer has a TCP listener.
This patch corrects that problem.

Change-Id: I2eb8a0b5de46ee56321bc0009b6ca2b3ad4caebf
4 years ago
German Eichberger 686303e79d Amphora logging
Configure rsyslog to forward logs to a target host

Co-Authored-By: Michael Johnson <>
Story: 1665069
Task: 33646

Change-Id: I00703f86555cbb574b943794b14a36fbc644f1b2
4 years ago
Michael Johnson 80ddbaeef4 Align logging in the amphora
This patch configures the primary components of the amphora to log
to syslog using consistent logging facilities.
By default, user traffic logs will go to LOG_LOCAL0 and the amphora
processes (haproxy, keepalived, etc.) will log to LOG_LOCAL1.

This is a patch supporting log offloading.

Change-Id: Ifda91e0310e812e34f1e398dd3176af8a9c58f89
Story: 1665069
Task: 5486
4 years ago
Michael Johnson 811ec10f32 Make sure the expected codes list is sorted
Previously the HTTP expected codes was using a set data type.
This patch changes the set to a sorted list to allow for consistent
testing and to make sure our expected codes list is always nicely

Change-Id: Id53c2552221bb133d69193f60516c6a785b8c7f7
4 years ago
ZhaoBo 44833d5d5e Support Host header inject for healthmonitor HTTP 1.1 health check
This patch adds 2 new options for healthmonitor HTTP health check.
'http_version' is for user to specify the HTTP version, 1.0 and 1.1 are
'domain_name' is for user to specify the HTTP host header inject to check
the HTTP backend health.
'domain_name' only available when HTTP version is 1.1

Story: 2002160
Task: 20010
Change-Id: Id3bf3962a02fbf77cf886c40ac64588cbacd3832
4 years ago
ZhaoBo 25fb7e4c32 Support L7policy redirect http code
Currently, L7Policy already support the redirection by url_prefix.
Then we can support the redirection with HTTP code.

This patch adds an new option 'redirect_http_code' to L7Policy API.

Story: 2003609
Task: 24941
Change-Id: Id0c9c376ffbc2fb10ddb988537d0ef1a8205e586
4 years ago
ZhaoBo acf6c36633 Amp driver support sni option to send the hostname to backend
If the Tls option of Pool is True, we default configure a sni option in
the associated members configuration, which is "sni ssl_fc_sni".

Story: 2003858
Task: 26684
Change-Id: Id61e0302dac3a5471b4fcb526b4edec50ec1a6fc
4 years ago
ZhaoBo e0e9af3b51 Add boolean tls_enabled option into Pool
Add "tls_enabled" option in Pool API.
This option will work on cert cases or no cert cases.

Story: 2003858
Task: 26672
Co-Authored-By: Michael Johnson <>
Change-Id: I62e31aaa66748ba652dfd5dbfd5a8b06d9ba0dfe
4 years ago
ZhaoBo 7aa115a553 Add 2 new fields into Pool API for support re-encryption
Add tls_ca_container_id and crl_container_id into Pool API.

Story: 2003858
Task: 26672
Co-Authored-By: Michael Johnson <>
Change-Id: I6cd6e2ca8e48a5df707a70d22505dec9d752c7eb
4 years ago
ZhaoBo aa7ac7ab73 Pool support sni cert for backend re-encryption
Add 1 fields like Listener does, which is 'tls_container_ref', this
field is introduced into Pool for storage the pool client certificate to
the backend servers, when the traffic willing to bring a cert to the
servers and check for tls connection.

Story: 2003859
Task: 26685
Change-Id: I29b7c7116e6087c942179ed9efdead494ef277a3
4 years ago
ZhaoBo f77d7d0220 L7rule support client certificate cases
This patch add 4 new types for SSL connection ACL configuration.
Which are:

The first type can just accept the compare type "EQUAL_TO" and value
"True" string.
The second can just accept the int value string to check the certificate
verify result, also just support "EQUAL_TO" compare type.
The third can accept key, the distinguished name field and a match string,
this one supports all kind compare types.

Story: 2002165
Task: 20025
Co-Authored-By: Michael Johnson <>
Change-Id: I71b57d0f32d4839a770396645d2b9945d24f2853
4 years ago
ZhaoBo aa1bca0271 Add new ssl header into Listener for client certificate
Add new ssl headers:
'X-SSL-Client-Verify', 'X-SSL-Client-Has-Cert', 'X-SSL-Client-DN',
'X-SSL-Client-CN', 'X-SSL-Issuer', 'X-SSL-Client-SHA1',
'X-SSL-Client-Not-Before', 'X-SSL-Client-Not-After'

Allow users to send to the backend with multiple choices when
tls_terminated is enabled for client certificate.

Story: 2002165
Task: 20020

Change-Id: I112936ee85c9e0dcfb87b962176ba7d623989a30
4 years ago
ZhaoBo 20509e2337 Add crl-file option for certification
Add crl-file in Listener side.

Story: 2002165
Co-Authored-By: Michael Johnson <>
Change-Id: I9e2ec06719fbbfd19482c2b8d39220e7e4ed81e3
4 years ago
ZhaoBo 7a8eb3ce22 Add an option to the Octavia V2 listener API for client cert
Listener API for client cerificate authentication with "None,
Optional, Mandatory" options

Story: 2002165
Task: 20019
Co-Authored-By: Michael Johnson <>
Change-Id: Ia753659981d99b315504f166c09afb8f5b14f195
4 years ago
ZhaoBo 0cc546a7c7 Add client_ca_tls_container_ref to listener API
This patch add 'client_ca_tls_container_ref' into listener API for front
client authentication.

Story: 2002165
Task: 20018
Co-Authored-By: Michael Johnson <>
Change-Id: I8a96d6fdfe53a16d1abcfd09bc6afedd6c490de2
4 years ago
Kamil Sambor 8923935df2 Add posibilities to set default timeouts
Default timeouts for backend member and frontend
client can be set now via config file.
Timeouts exposed in config in section haproxy_amphora:
 * timeout_client_data
 * timeout_member_connect
 * timeout_member_data
 * timeout_tcp_inspect

Change-Id: I6e1be42c5c15c4171b012734e4c2a1fded51dbc7
Partial-Bug: 1797130
Story: 2004042
Task: 27046
5 years ago
Zuul a0112c4b37 Merge "Support REDIRECT_PREFIX action for L7Policy" 5 years ago
Michael Johnson 9b6aa47c03 Fix an upgrade issue for CentOS 7 amphora
A recent patch[1] (stein master) added the http-reuse option to the
haproxy template for pools. This feature is not available in the HAProxy
version included with CentOS 7, 1.5.x. This could cause an upgrade issue
if the control plane was upgraded to Stein, but the cloud still had older
CentOS based amphora.

This patch corrects that issue by checking the HAProxy version in the
amphora and adjusting the template if it finds an older HAProxy.

This patch also updates the test_health_check_stale_amphora test to
not wait (sleep) for the full heartbeat_timeout.


Change-Id: I3d990d1d3cd93dbeced9edc53f9c166610dafcd0
Story: 2003901
Task: 26775
5 years ago
sapd 62192dba3a Support REDIRECT_PREFIX action for L7Policy
Currently, Octavia only support three actions for L7Policy,
in this patch we will implement new action for L7Policy.

Story: 2003700

Change-Id: Ie99591ede097b566294ebdb673c460442dd6d942
5 years ago
Michael Johnson 53772f5320 Set some amphora driver optimizations
This patch adds a few optimizations when using the amphora driver.

1. It increases the amp_active_retries from 10 to 30. This increases
the time we wait for nova to mark an instance "ACTIVE". The old default
of 10 was one minute forty seconds, but in some clouds it's been observed
that the nova schedule can get overloaded and take longer than a minute
forty to schedule the instance. Setting this to 30 means we will wait
five minutes for nova to schedule the instance.
2. It enables TCP kernel splicing in HAProxy. This has been shown to
reduce the CPU overhead for very high rate TCP load balancers.
3. Finally it enables "safe" HTTP keepalives on the backend member
connections [1]. This increases the request rate possible while using HTTP
protocol listeners and members.


Change-Id: I3af009cac9a9edc8aef793b52c6a1488fde2c59b
5 years ago
Nir Magnezi 100858fa79 Remove user_group option
In Pike[1], we introduced a user_group auto detection for haproxy.
The default user group name is auto-detected for any OS distribution
we support as a base for Amphorae.

user_group remained as an option for admins but was also
marked deprecated in Pike[2].

This patch removes that option altogether.

Story: 2003323
Task: 24357

[1] Ia8fede9d7da4709a48661d1fc595a16d04fcbfa9

Change-Id: Iddd4162674f116705d2b47062cbf7ca88f2677a6
5 years ago
Michael Johnson 294e0fc128 Fixes unlimited listener connection limit
When using the Octavia/amphora driver, unspecified or unlimited (-1)
settings would lead to a 2000 connection limit in HAproxy.
This patch updates that to be 1,000,000 connections.
1,000,000 was selected to amphora memory usage at a reasonable level.

Change-Id: Iddeb62412bb71b69cf1e9198be6131c59a3051b0
Story: 1635416
Task: 5159
5 years ago
Zuul 18da15188d Merge "Add lb_id comment to amp haproxy listener config" 5 years ago
Adam Harwell 0871dfcd2b Add lb_id comment to amp haproxy listener config
This is useful for debugging purposes, and could theoretically be
interesting data for custom elements.

Story: 2001236
Task: 5756

Change-Id: If5f7793bd16fd6c8841586f29cdcf5e53908b1fe
5 years ago
German Eichberger 1e4155f8ed Add a config to surpress amphora logging
Change-Id: I502e701acbaf0e92a64791a632eeb5ae61123bdd
5 years ago
Zuul f755657f17 Merge "Adding support for the octavia listener X-Forwarded-Proto header insertion." 5 years ago
Rafal Pietrzak d3f0a50014 Adding support for the octavia listener X-Forwarded-Proto header insertion.
Adding support for the octavia listener X-Forwarded-Proto header insertion.
  A X-Forwarded-Proto header is inserted into the end of request to the backend
  member. HTTP for the HTTP listener protocol type, HTTPS for
  the TERMINATED_HTTPS listener protocol type.

  Adding X-Forwarded-Proto to the list of HTTP headers which are supported.

  File: octavia/master/octavia/octavia/common/
      SUPPORTED_HTTP_HEADERS = ['X-Forwarded-For',

  Adding the lines to the macros.j2 realizing the the http/https insertion
  to the backend member according to listener protocol type:

  File:  octavia/common/jinja/haproxy/templates/macros.j2

    {% if listener.insert_headers.get('X-Forwarded-Proto',
                                          'False').lower() == 'true' %}
        {% if listener.protocol.lower() == constants.PROTOCOL_HTTP.lower() %}
    http-request set-header X-Forwarded-Proto http
        {% elif listener.protocol.lower() ==
                               constants.PROTOCOL_TERMINATED_HTTPS.lower() %}
    http-request set-header X-Forwarded-Proto https
        {% endif %}
    {% endif %}

Change-Id: Id017bb277eebae98f0441663e41d07b40b6e3e38
Story: 2002173
Task: 20038
5 years ago
Mohammed Naser 852191feb2 Allow using spaces for primary common name in SSL certificates
The common name is used as a file name inside the HAproxy
configuration file.  However, a common name can include spaces
and it will result in a configuration file that simply doesn't
work because of the spaces.

The patch changes the functionality so that it instead creates
a SHA1 hash of the certificate and uses that as the file name
to avoid those issues.

Change-Id: I039ed0b40df8b72a1238f8896548fe77086c530c
5 years ago
Adam Harwell c6ff8596db Create disabled members in haproxy
Members that were disabled / admin_state_up=False were simply excluded
from the haproxy configuration we pass to the amps. Instead, we should
be creating them in a disabled state, so they return in health messages
as status "maint", and can be marked OFFLINE via the standard health
mechanism, instead of just via override hacks.
This also resolves a bug introduced in an earlier change:
which caused admin-downed members to stay in NO_MONITOR always.

Change-Id: I6615b3ff89d7cef2af52d474aab3a03d947f98be
5 years ago
Adam Harwell bb0447e98b Expose timeout options
Various timeout options need to be exposed to enable use-cases more
complex than standard HTTP requests.

In this patch we expose four new timeout values:
* timeout_client_data
* timeout_member_connect
* timeout_member_data
* timeout_tcp_inspect

Change-Id: Id4667201c1bfaa06f7af9060c936ba00c2f314f9
Story: 1457556
Task: 5453
5 years ago
Adam Harwell e819e4521a Allow members to be set as "backup"
Setting a member as "backup" means no traffic will be sent to it unless
all non-backup members are marked as down.

This should be essentially the same in every backend provider AFAIU.

This was requested by at least one operator (me) and was agreed during
the PTG to add value.

Story: 2001777
Task: 12483
Change-Id: I953abe71a0988da78efc6b3961f7518c81c2a06d
5 years ago
Jacky Hu 649b33d247 Add license for empty
This also fix build-openstack-sphinx-docs, there was a change introduced
in sphinx 1.6.6:

If the size of is less than 2, then the module would be
skipped which will cause the sphinx consistency checking failing later.

Change-Id: I9d8764b6e907aceed8bb8a9b04711145d0eb32ad
5 years ago
Zuul 7947a43b5c Merge "Fix PING health monitor for old amphora" 5 years ago
Monty Taylor 967edebe0c Change how path munging works for sphinx docs
The 'import tools' line is fragile as it depends on how things are
executed as to whether or not '.' is in the python path.

Do the sphinx path munging before importing it.

Also, remove reference to modules/autoindex which does not exist and
thus causes sadness from warning-is-error.

Moves documentation requirements into doc/requirements.txt

Depends-On: Ib121961c5a953a434e7b333cd70f7838a2671f69
Change-Id: I23691aa1d0ea038ec1215e6199015529ddd92de4
6 years ago
Michael Johnson b9f5c47d63 Fix PING health monitor for old amphora
The previous PING health monitor patch had a bug that caused amphora
running an older version of the amphora-agent to fail even when running
non-ping health monitors.
This patch corrects that behavior to limit the issue to PING health
monitors as documented in the previous patch release note.

Change-Id: I83a3bf541c8c3c0f06544ef3228a5028756f51b3
6 years ago
Michael Johnson 2897b340f7 Fix health monitor type "PING" to use icmp ping
Currently with Octavia, if the user specifies a health monitor of type
"PING" we are still using a TCP connect to check for health.
This patch fixes that to actually ping the member to validate health.

Change-Id: I8a67efb7113ffa49b2805b37c3855373b17e5789
Story: 2001280
Task: 5826
6 years ago
Adam Harwell 897214a4ff SSL Health Monitors didn't actually ... check very much
Change HTTPS monitors to be a real check, and add TLS-HELLO type to
perform the older check functionality if desired.
The only reason you would need TLS-HELLO instead of HTTPS is if your
application does client-cert validation, as the HAProxy box won't have a
valid client cert.

Also add missing PING type to the DB, so PING monitors can be used.

Change-Id: I15a79b7fb0c2ff1020090b4057909a1f41a2c8ad
6 years ago
cheng 594a5935e8 Add monitor address and port to pool member
Closes-Bug: #1541579

Change-Id: I8b49726bc2b7fbc37e1cd203a405f4bcbd0b7125
Signed-off-by: cheng <>
6 years ago
cheng 7100872ddb Add new PROXY protocol to lbaas pool
This patch enable configure PROXY protocol informs to backend
server about the lay 3/4 address of the incoming connections.

Close-Bug: #1677987

Change-Id: Idc9a5718dddbaaaec251c9a0673c74e4132c5f54
Signed-off-by: cheng <>
6 years ago
Sindhu Devale fd5c800ff9 Align Octavia API to n-lbaasv2 for L7Rules
The endpoint are as follows:

- /v2.0/lbaas/l7policies/
- /v2.0/l7policies/

GET all - /<policy-id>/l7rules/
GET one - /<policy-id>/l7rules/<rule-id>
POST - /<policy-id>/l7rules/ {<body>}
PUT - /<policy-id>/l7rules/<rule-id> {<body>}
DELETE - /<policy-id>/l7rules/<rule-id>

Partially Closes-Bug: #1616701
Co-Authored-By: Shashank Kumar Shankar <>
Co-Authored-By: Adam Harwell <>

Change-Id: I247988a2ea19a92f827756504a0ee46679bbc53b
6 years ago
Lubosz "diltram" Kosnik 867b350988 Remove CONF.import_group
Remove unneeded import_group lines which are not doing anything and just makes
code harder to understand.

Change-Id: I673dd04dd31ae9771e6af982d184eee0e9cbf2d4
7 years ago
Stephen Balukoff ae5a3a99fb Update haproxy logging configuration
Use httplog for appropriate listeners in default amphora image, instead
of tcplog for everything.

Change-Id: I162862dace86556262d25cbe8ab10433139ffc89
7 years ago
Paul Glass ef11747a56 Switch HAProxy health check timeout to seconds
Change-Id: If8166b8e76ca6c1b15963ef99bec07ff2a6fb118
Closes-Bug: #1600326
7 years ago
Stephen Balukoff 5eb294e3fb Set fullconn parameter in haproxy backends
By default haproxy sets the connection limit of any backend to 10% of
the connection limits for all its front-ends. For long-lasting but
low-load connections to a small number of back-end servers through the
load balancer, this means that the haproxy-based back-ends have an
effective connection limit that is much smaller than what the user
expects it to be.

The commit explicitly sets the connection limit for haproxy back-ends to
be equal to that of their front-ends.

Change-Id: Ie87722318fd2bc0005b089d74363c0812253c27f
Partial-Bug: #1622793
7 years ago
Stephen Balukoff f37b3e5422 Set haproxy global connection limit
Haproxy uses two separate maxconn values: the maxconn setting in the
global context is used to adjust file descriptor limits and allocate
resources, while the maxconn setting in the frontend context controls if
additional connections are accepted on that frontend.

This patch adds a maxconn to the global context equal to the sum of the
connection limit on each listener. For listeners which no connection
limit specified, we don't set this value and leave it at haproxy's
default (2000).

Change-Id: I30f3cf5f469db953a0002721a315f02694621b7d
Partial-Bug: #1544861
7 years ago
Paul Glass e291a88210 Stop using bandit-baseline
bandit-baseline finds *new* issues introduced in a commit, by comparing
results between two git commits. If the git repository has uncommitted
changes, bandit-baseline refuses to run.

This switches over to using plain bandit instead of bandit-baseline,
and resolves or stifles existing bandit errors so we have a clean run.
These updates apply to bandit running as part of `tox -e pep8` or `tox
-e bandit`.

* Have bandit runs from tox ignore the octavia/tests directory
* Resolve several instances of `B701 jinja2_autoescape_false`
* Stifle several instances of `B303 md5`
* Resolve two instances of `B104 hardcoded_bind_all_interfaces`
* Stifle one instance of `B104 hardcoded_bind_all_interfaces` (see

Closes-Bug: #1621251
Related-Bug: #1489963

Change-Id: Iad3cbe5762949a6311bdd361b1f12c5a24c40633
7 years ago
Yang Li 4695d053ed Replace 'c = a if a else b' with 'c = a or b'
Change-Id: I66ed9cf8007243e2464837094a90b39449249e6a
7 years ago
Trevor Vardeman 7bfd1e20dc Fixing bug in single-create jinja config generation
Fix detailed generating the config regardless of the presence of the redirect
pool backend configuration, sending requests to the default pool backend

Change-Id: I84194e6b8cab765fc1c97c0acaaa0c3037320578
Closes-Bug: 1583341
7 years ago