Load balancers with multiple listeners, running on an amphora image
with HAProxy 1.8 or newer can experience excessive memory usage that
may lead to an ERROR provisioning_status.
This patch resolves this issue by consolidating the listeners into
a single haproxy process inside the amphora.
Story: 2005412
Task: 34744
Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: Idaccbcfa0126f1e26fbb3ad770c65c9266cfad5b
HAProxy is not handling two of the HTTP log format variables correct
when the load balancer has a TCP listener.
This patch corrects that problem.
Change-Id: I2eb8a0b5de46ee56321bc0009b6ca2b3ad4caebf
Configure rsyslog to forward logs to a target host
Co-Authored-By: Michael Johnson <johnsomor@gmail.com>
Story: 1665069
Task: 33646
Change-Id: I00703f86555cbb574b943794b14a36fbc644f1b2
This patch configures the primary components of the amphora to log
to syslog using consistent logging facilities.
By default, user traffic logs will go to LOG_LOCAL0 and the amphora
processes (haproxy, keepalived, etc.) will log to LOG_LOCAL1.
This is a patch supporting log offloading.
Change-Id: Ifda91e0310e812e34f1e398dd3176af8a9c58f89
Story: 1665069
Task: 5486
Previously the HTTP expected codes was using a set data type.
This patch changes the set to a sorted list to allow for consistent
testing and to make sure our expected codes list is always nicely
sorted.
Change-Id: Id53c2552221bb133d69193f60516c6a785b8c7f7
This patch adds 2 new options for healthmonitor HTTP health check.
'http_version' is for user to specify the HTTP version, 1.0 and 1.1 are
available.
'domain_name' is for user to specify the HTTP host header inject to check
the HTTP backend health.
'domain_name' only available when HTTP version is 1.1
Story: 2002160
Task: 20010
Change-Id: Id3bf3962a02fbf77cf886c40ac64588cbacd3832
Currently, L7Policy already support the redirection by url_prefix.
Then we can support the redirection with HTTP code.
This patch adds an new option 'redirect_http_code' to L7Policy API.
Story: 2003609
Task: 24941
Change-Id: Id0c9c376ffbc2fb10ddb988537d0ef1a8205e586
If the Tls option of Pool is True, we default configure a sni option in
the associated members configuration, which is "sni ssl_fc_sni".
Story: 2003858
Task: 26684
Change-Id: Id61e0302dac3a5471b4fcb526b4edec50ec1a6fc
Add "tls_enabled" option in Pool API.
This option will work on cert cases or no cert cases.
Story: 2003858
Task: 26672
Co-Authored-By: Michael Johnson <johnsomor@gmail.com>
Change-Id: I62e31aaa66748ba652dfd5dbfd5a8b06d9ba0dfe
Add tls_ca_container_id and crl_container_id into Pool API.
Story: 2003858
Task: 26672
Co-Authored-By: Michael Johnson <johnsomor@gmail.com>
Change-Id: I6cd6e2ca8e48a5df707a70d22505dec9d752c7eb
Add 1 fields like Listener does, which is 'tls_container_ref', this
field is introduced into Pool for storage the pool client certificate to
the backend servers, when the traffic willing to bring a cert to the
servers and check for tls connection.
Story: 2003859
Task: 26685
Change-Id: I29b7c7116e6087c942179ed9efdead494ef277a3
This patch add 4 new types for SSL connection ACL configuration.
Which are:
L7RULE_TYPE_SSL_CONN_HAS_CERT
L7RULE_TYPE_VERIFY_RESULT
L7RULE_TYPE_DN_FIELD
The first type can just accept the compare type "EQUAL_TO" and value
"True" string.
The second can just accept the int value string to check the certificate
verify result, also just support "EQUAL_TO" compare type.
The third can accept key, the distinguished name field and a match string,
this one supports all kind compare types.
Story: 2002165
Task: 20025
Co-Authored-By: Michael Johnson <johnsomor@gmail.com>
Change-Id: I71b57d0f32d4839a770396645d2b9945d24f2853
Add new ssl headers:
'X-SSL-Client-Verify', 'X-SSL-Client-Has-Cert', 'X-SSL-Client-DN',
'X-SSL-Client-CN', 'X-SSL-Issuer', 'X-SSL-Client-SHA1',
'X-SSL-Client-Not-Before', 'X-SSL-Client-Not-After'
Allow users to send to the backend with multiple choices when
tls_terminated is enabled for client certificate.
Story: 2002165
Task: 20020
Change-Id: I112936ee85c9e0dcfb87b962176ba7d623989a30
Add crl-file in Listener side.
Story: 2002165
Co-Authored-By: Michael Johnson <johnsomor@gmail.com>
Change-Id: I9e2ec06719fbbfd19482c2b8d39220e7e4ed81e3
Listener API for client cerificate authentication with "None,
Optional, Mandatory" options
Story: 2002165
Task: 20019
Co-Authored-By: Michael Johnson <johnsomor@gmail.com>
Change-Id: Ia753659981d99b315504f166c09afb8f5b14f195
This patch add 'client_ca_tls_container_ref' into listener API for front
client authentication.
Story: 2002165
Task: 20018
Co-Authored-By: Michael Johnson <johnsomor@gmail.com>
Change-Id: I8a96d6fdfe53a16d1abcfd09bc6afedd6c490de2
Default timeouts for backend member and frontend
client can be set now via config file.
Timeouts exposed in config in section haproxy_amphora:
* timeout_client_data
* timeout_member_connect
* timeout_member_data
* timeout_tcp_inspect
Change-Id: I6e1be42c5c15c4171b012734e4c2a1fded51dbc7
Partial-Bug: 1797130
Story: 2004042
Task: 27046
A recent patch[1] (stein master) added the http-reuse option to the
haproxy template for pools. This feature is not available in the HAProxy
version included with CentOS 7, 1.5.x. This could cause an upgrade issue
if the control plane was upgraded to Stein, but the cloud still had older
CentOS based amphora.
This patch corrects that issue by checking the HAProxy version in the
amphora and adjusting the template if it finds an older HAProxy.
This patch also updates the test_health_check_stale_amphora test to
not wait (sleep) for the full heartbeat_timeout.
[1] https://review.openstack.org/#/c/598379/
Change-Id: I3d990d1d3cd93dbeced9edc53f9c166610dafcd0
Story: 2003901
Task: 26775
Currently, Octavia only support three actions for L7Policy,
in this patch we will implement new action for L7Policy.
Story: 2003700
Change-Id: Ie99591ede097b566294ebdb673c460442dd6d942
This patch adds a few optimizations when using the amphora driver.
1. It increases the amp_active_retries from 10 to 30. This increases
the time we wait for nova to mark an instance "ACTIVE". The old default
of 10 was one minute forty seconds, but in some clouds it's been observed
that the nova schedule can get overloaded and take longer than a minute
forty to schedule the instance. Setting this to 30 means we will wait
five minutes for nova to schedule the instance.
2. It enables TCP kernel splicing in HAProxy. This has been shown to
reduce the CPU overhead for very high rate TCP load balancers.
3. Finally it enables "safe" HTTP keepalives on the backend member
connections [1]. This increases the request rate possible while using HTTP
protocol listeners and members.
[1] http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#4-http-reuse
Change-Id: I3af009cac9a9edc8aef793b52c6a1488fde2c59b
In Pike[1], we introduced a user_group auto detection for haproxy.
The default user group name is auto-detected for any OS distribution
we support as a base for Amphorae.
user_group remained as an option for admins but was also
marked deprecated in Pike[2].
This patch removes that option altogether.
Story: 2003323
Task: 24357
[1] Ia8fede9d7da4709a48661d1fc595a16d04fcbfa9
[2] https://review.openstack.org/#/c/429398/45/octavia/common/config.py@175
Change-Id: Iddd4162674f116705d2b47062cbf7ca88f2677a6
When using the Octavia/amphora driver, unspecified or unlimited (-1)
settings would lead to a 2000 connection limit in HAproxy.
This patch updates that to be 1,000,000 connections.
1,000,000 was selected to amphora memory usage at a reasonable level.
Change-Id: Iddeb62412bb71b69cf1e9198be6131c59a3051b0
Story: 1635416
Task: 5159
This is useful for debugging purposes, and could theoretically be
interesting data for custom elements.
Story: 2001236
Task: 5756
Change-Id: If5f7793bd16fd6c8841586f29cdcf5e53908b1fe
Adding support for the octavia listener X-Forwarded-Proto header insertion.
A X-Forwarded-Proto header is inserted into the end of request to the backend
member. HTTP for the HTTP listener protocol type, HTTPS for
the TERMINATED_HTTPS listener protocol type.
Adding X-Forwarded-Proto to the list of HTTP headers which are supported.
File: octavia/master/octavia/octavia/common/constants.py:
SUPPORTED_HTTP_HEADERS = ['X-Forwarded-For',
'X-Forwarded-Port',
'X-Forwarded-Proto']
Adding the lines to the macros.j2 realizing the the http/https insertion
to the backend member according to listener protocol type:
File: octavia/common/jinja/haproxy/templates/macros.j2
{% if listener.insert_headers.get('X-Forwarded-Proto',
'False').lower() == 'true' %}
{% if listener.protocol.lower() == constants.PROTOCOL_HTTP.lower() %}
http-request set-header X-Forwarded-Proto http
{% elif listener.protocol.lower() ==
constants.PROTOCOL_TERMINATED_HTTPS.lower() %}
http-request set-header X-Forwarded-Proto https
{% endif %}
{% endif %}
Change-Id: Id017bb277eebae98f0441663e41d07b40b6e3e38
Story: 2002173
Task: 20038
The common name is used as a file name inside the HAproxy
configuration file. However, a common name can include spaces
and it will result in a configuration file that simply doesn't
work because of the spaces.
The patch changes the functionality so that it instead creates
a SHA1 hash of the certificate and uses that as the file name
to avoid those issues.
Change-Id: I039ed0b40df8b72a1238f8896548fe77086c530c
Members that were disabled / admin_state_up=False were simply excluded
from the haproxy configuration we pass to the amps. Instead, we should
be creating them in a disabled state, so they return in health messages
as status "maint", and can be marked OFFLINE via the standard health
mechanism, instead of just via override hacks.
This also resolves a bug introduced in an earlier change:
https://review.openstack.org/#/c/567322/
which caused admin-downed members to stay in NO_MONITOR always.
Change-Id: I6615b3ff89d7cef2af52d474aab3a03d947f98be
Various timeout options need to be exposed to enable use-cases more
complex than standard HTTP requests.
In this patch we expose four new timeout values:
* timeout_client_data
* timeout_member_connect
* timeout_member_data
* timeout_tcp_inspect
Change-Id: Id4667201c1bfaa06f7af9060c936ba00c2f314f9
Story: 1457556
Task: 5453
Setting a member as "backup" means no traffic will be sent to it unless
all non-backup members are marked as down.
This should be essentially the same in every backend provider AFAIU.
This was requested by at least one operator (me) and was agreed during
the PTG to add value.
Story: 2001777
Task: 12483
Change-Id: I953abe71a0988da78efc6b3961f7518c81c2a06d
This also fix build-openstack-sphinx-docs, there was a change introduced
in sphinx 1.6.6:
https://github.com/sphinx-doc/sphinx/pull/4335/files
If the size of __init__.py is less than 2, then the module would be
skipped which will cause the sphinx consistency checking failing later.
Change-Id: I9d8764b6e907aceed8bb8a9b04711145d0eb32ad
The 'import tools' line is fragile as it depends on how things are
executed as to whether or not '.' is in the python path.
Do the sphinx path munging before importing it.
Also, remove reference to modules/autoindex which does not exist and
thus causes sadness from warning-is-error.
Moves documentation requirements into doc/requirements.txt
Depends-On: Ib121961c5a953a434e7b333cd70f7838a2671f69
Change-Id: I23691aa1d0ea038ec1215e6199015529ddd92de4
The previous PING health monitor patch had a bug that caused amphora
running an older version of the amphora-agent to fail even when running
non-ping health monitors.
This patch corrects that behavior to limit the issue to PING health
monitors as documented in the previous patch release note.
Change-Id: I83a3bf541c8c3c0f06544ef3228a5028756f51b3
Currently with Octavia, if the user specifies a health monitor of type
"PING" we are still using a TCP connect to check for health.
This patch fixes that to actually ping the member to validate health.
Change-Id: I8a67efb7113ffa49b2805b37c3855373b17e5789
Story: 2001280
Task: 5826
Change HTTPS monitors to be a real check, and add TLS-HELLO type to
perform the older check functionality if desired.
The only reason you would need TLS-HELLO instead of HTTPS is if your
application does client-cert validation, as the HAProxy box won't have a
valid client cert.
Also add missing PING type to the DB, so PING monitors can be used.
Change-Id: I15a79b7fb0c2ff1020090b4057909a1f41a2c8ad
This patch enable configure PROXY protocol informs to backend
server about the lay 3/4 address of the incoming connections.
Close-Bug: #1677987
Change-Id: Idc9a5718dddbaaaec251c9a0673c74e4132c5f54
Signed-off-by: cheng <tangch318@gmail.com>
The endpoint are as follows:
- /v2.0/lbaas/l7policies/
- /v2.0/l7policies/
GET all - /<policy-id>/l7rules/
GET one - /<policy-id>/l7rules/<rule-id>
POST - /<policy-id>/l7rules/ {<body>}
PUT - /<policy-id>/l7rules/<rule-id> {<body>}
DELETE - /<policy-id>/l7rules/<rule-id>
Partially Closes-Bug: #1616701
Co-Authored-By: Shashank Kumar Shankar <shashank.kumar.shankar@intel.com>
Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: I247988a2ea19a92f827756504a0ee46679bbc53b
Remove unneeded import_group lines which are not doing anything and just makes
code harder to understand.
Change-Id: I673dd04dd31ae9771e6af982d184eee0e9cbf2d4
By default haproxy sets the connection limit of any backend to 10% of
the connection limits for all its front-ends. For long-lasting but
low-load connections to a small number of back-end servers through the
load balancer, this means that the haproxy-based back-ends have an
effective connection limit that is much smaller than what the user
expects it to be.
The commit explicitly sets the connection limit for haproxy back-ends to
be equal to that of their front-ends.
Change-Id: Ie87722318fd2bc0005b089d74363c0812253c27f
Partial-Bug: #1622793
Haproxy uses two separate maxconn values: the maxconn setting in the
global context is used to adjust file descriptor limits and allocate
resources, while the maxconn setting in the frontend context controls if
additional connections are accepted on that frontend.
This patch adds a maxconn to the global context equal to the sum of the
connection limit on each listener. For listeners which no connection
limit specified, we don't set this value and leave it at haproxy's
default (2000).
Change-Id: I30f3cf5f469db953a0002721a315f02694621b7d
Partial-Bug: #1544861
bandit-baseline finds *new* issues introduced in a commit, by comparing
results between two git commits. If the git repository has uncommitted
changes, bandit-baseline refuses to run.
This switches over to using plain bandit instead of bandit-baseline,
and resolves or stifles existing bandit errors so we have a clean run.
These updates apply to bandit running as part of `tox -e pep8` or `tox
-e bandit`.
* Have bandit runs from tox ignore the octavia/tests directory
* Resolve several instances of `B701 jinja2_autoescape_false`
* Stifle several instances of `B303 md5`
* Resolve two instances of `B104 hardcoded_bind_all_interfaces`
* Stifle one instance of `B104 hardcoded_bind_all_interfaces` (see
https://bugs.launchpad.net/octavia/+bug/1489963)
Closes-Bug: #1621251
Related-Bug: #1489963
Change-Id: Iad3cbe5762949a6311bdd361b1f12c5a24c40633
Fix detailed generating the config regardless of the presence of the redirect
pool backend configuration, sending requests to the default pool backend
Change-Id: I84194e6b8cab765fc1c97c0acaaa0c3037320578
Closes-Bug: 1583341
Michael Johnson