Commit Graph

197 Commits (65e132a734f005f090a384bfa129482d195c6d6e)

Author SHA1 Message Date
Michael Johnson 06ce4777c3 Fix multi-listener load balancers
Load balancers with multiple listeners, running on an amphora image
with HAProxy 1.8 or newer can experience excessive memory usage that
may lead to an ERROR provisioning_status.
This patch resolves this issue by consolidating the listeners into
a single haproxy process inside the amphora.

Story: 2005412
Task: 34744
Co-Authored-By: Adam Harwell <>
Change-Id: Idaccbcfa0126f1e26fbb3ad770c65c9266cfad5b
4 years ago
Michael Johnson 0eaab450c1 Fix allocate_and_associate DB deadlock
In testing we saw a DB deadlock in allocate_and_associate.
This patch corrects that with a DB retry wrapper.

Change-Id: I20572505e448f9b7ff6314b5167cc4c519c445fb
Story: 2005893
Task: 33751
4 years ago
Sahid Orentino Ferdjaoui 7971b8ab60 db: add missing primary key in spares_pool table
In commit a205ab3ebe a new model
spares_pool has been introduced but the definition introduced by
0b468090e6 was missing the primary key.

In this commit we fixed the definition and add an update script for
already deployed env.

Note: without the PK this creates an error when Running Percona with
pxc_strict_mode == ENFORCING

Story: 2005531
Task: 30658
Change-Id: If70e5df5aea008e8998eaad912ba85dfb6bda77a
Signed-off-by: Sahid Orentino Ferdjaoui <>
4 years ago
Zuul 09020b6bfc Merge "Add Python 3.7 support" 4 years ago
Zuul 0573d2c609 Merge "Performance improvement for non-udp health checks" 4 years ago
Carlos Goncalves c4faac25de Add Python 3.7 support
In order to support Python 3.7, pylint has to be updated to 2.0.0
minimum. Newer versions of Pylint enforce additional checkers which can
be addressed with some code refactoring rather than silently ignoring
them in pylintrc; except useless-object-inheritance which is required to
be silented so that we stay compatible with Python 2.x.

Story: 2004073
Task: 27434

Change-Id: I52301d763797d619f195bd8a1c32bc47f1e68420
4 years ago
Adam Harwell 29d4340e9f Remove v1 API and associated code
Includes some updates to docs and configs and related files to remove
references to neutron-lbaas. Also remove handlers.

Change-Id: I3082962841d3b645f3cbd1a6b41fc7fb28dcf7e6
4 years ago
Adam Harwell 24e77d606d Performance improvement for non-udp health checks
When no UDP listeners are present, skip the UDP health-check code
branch, which prevents expensive and unnecessary DB calls.
Also optimise the UDP health-check code so it only fetches information
for relevant listeners.

Co-Authored-By: Adam Harwell <>
Change-Id: I7fde54084b39c1d0529cfb0bcfb79697d63ea6ae
4 years ago
Gregory Thiemonge 0b468090e6 Fix spare amphora check and creation
This patch fixes an issue when the SparesPool table is empty that blocks
spare amphorae creation. It creates a new spares pool entry if the table
is empty.

Story 2005352
Task 30306

Change-Id: I0ce2778277640ee9e509c709bf8621b8b025d6d3
4 years ago
Zuul a635dd6bc9 Merge "Limit spares pool to the spare_amphora_pool_size" 4 years ago
Michael Johnson a205ab3ebe Limit spares pool to the spare_amphora_pool_size
This patch fixes a bug where the housekeeping controllers may launch
more spare amphora than the spare_amphora_pool_size setting.

Story: 2003094
Task 23186

Change-Id: I4c98b3442d2471662488184fa7e91ac64ec33279
4 years ago
Zuul 7fc5406bd2 Merge "Fix performance of housekeeping DB clean up" 4 years ago
ZhaoBo 44833d5d5e Support Host header inject for healthmonitor HTTP 1.1 health check
This patch adds 2 new options for healthmonitor HTTP health check.
'http_version' is for user to specify the HTTP version, 1.0 and 1.1 are
'domain_name' is for user to specify the HTTP host header inject to check
the HTTP backend health.
'domain_name' only available when HTTP version is 1.1

Story: 2002160
Task: 20010
Change-Id: Id3bf3962a02fbf77cf886c40ac64588cbacd3832
4 years ago
ZhaoBo 25fb7e4c32 Support L7policy redirect http code
Currently, L7Policy already support the redirection by url_prefix.
Then we can support the redirection with HTTP code.

This patch adds an new option 'redirect_http_code' to L7Policy API.

Story: 2003609
Task: 24941
Change-Id: Id0c9c376ffbc2fb10ddb988537d0ef1a8205e586
4 years ago
ZhaoBo e0e9af3b51 Add boolean tls_enabled option into Pool
Add "tls_enabled" option in Pool API.
This option will work on cert cases or no cert cases.

Story: 2003858
Task: 26672
Co-Authored-By: Michael Johnson <>
Change-Id: I62e31aaa66748ba652dfd5dbfd5a8b06d9ba0dfe
4 years ago
ZhaoBo 7aa115a553 Add 2 new fields into Pool API for support re-encryption
Add tls_ca_container_id and crl_container_id into Pool API.

Story: 2003858
Task: 26672
Co-Authored-By: Michael Johnson <>
Change-Id: I6cd6e2ca8e48a5df707a70d22505dec9d752c7eb
4 years ago
ZhaoBo aa7ac7ab73 Pool support sni cert for backend re-encryption
Add 1 fields like Listener does, which is 'tls_container_ref', this
field is introduced into Pool for storage the pool client certificate to
the backend servers, when the traffic willing to bring a cert to the
servers and check for tls connection.

Story: 2003859
Task: 26685
Change-Id: I29b7c7116e6087c942179ed9efdead494ef277a3
4 years ago
ZhaoBo f77d7d0220 L7rule support client certificate cases
This patch add 4 new types for SSL connection ACL configuration.
Which are:

The first type can just accept the compare type "EQUAL_TO" and value
"True" string.
The second can just accept the int value string to check the certificate
verify result, also just support "EQUAL_TO" compare type.
The third can accept key, the distinguished name field and a match string,
this one supports all kind compare types.

Story: 2002165
Task: 20025
Co-Authored-By: Michael Johnson <>
Change-Id: I71b57d0f32d4839a770396645d2b9945d24f2853
4 years ago
ZhaoBo 20509e2337 Add crl-file option for certification
Add crl-file in Listener side.

Story: 2002165
Co-Authored-By: Michael Johnson <>
Change-Id: I9e2ec06719fbbfd19482c2b8d39220e7e4ed81e3
4 years ago
ZhaoBo 7a8eb3ce22 Add an option to the Octavia V2 listener API for client cert
Listener API for client cerificate authentication with "None,
Optional, Mandatory" options

Story: 2002165
Task: 20019
Co-Authored-By: Michael Johnson <>
Change-Id: Ia753659981d99b315504f166c09afb8f5b14f195
4 years ago
ZhaoBo 0cc546a7c7 Add client_ca_tls_container_ref to listener API
This patch add 'client_ca_tls_container_ref' into listener API for front
client authentication.

Story: 2002165
Task: 20018
Co-Authored-By: Michael Johnson <>
Change-Id: I8a96d6fdfe53a16d1abcfd09bc6afedd6c490de2
4 years ago
Carlos Goncalves be552d99a4 Fix performance of housekeeping DB clean up
The Housekeeping service grows in utilization of CPU as more amphorae
are created and/or marked as DELETED. The problem lays on the SELECT
statement constructed in get_all_deleted_expiring_amphora via the ORM --
it is joined eager loading all relationships.

The task does not need such amount of information, only the amphora ID.
The statement could be simplified by not loading any relationship or, at
most, lazy loading them.

This patch also fixes performance of cleaning up deleted and expired
load balancers. The code was doing multiple round-trips to the database

1. retrieving all deleted LBs
2. for each LB, retrieving it again from DB to check expired date
3. delete LB

Step 1 and 2 are now condensed in get_all_deleted_expiring(), making it
a single round-trip.

Story: 2004665
Task: 28643

Change-Id: Iffc960c7c3a986328cfded1b4e408931ab0a7877
4 years ago
Michael Johnson 8d45d48731 Fix VIP address DB storage size to support IPv6
An earlier patch[1] attempted to fix the vip address storage size in the
octavia database, however it issued the alembic "alter_column" command
incorrectly and the column size was not fixed.

This patch corrects that with another alembic migration that uses the correct


Change-Id: Ib577f92be38fa9a1c84571729b0236913da8d9d2
4 years ago
Michael Johnson 0b1fe6a526 Adds flavor support to the amphora driver
This patch adds support for flavor metadata validation by the amphora driver
and support for setting the load balancer topology via a flavor.
It also adds "flavor_id" to the load balancer table in the database.

Change-Id: I8eae870abdb20dc32917957e32606deef387ec88
4 years ago
Pradeep Kumar Singh 637009ecd0 Add flavor, flavor_profile table and their APIs
This patch adds flavor and flavor_profile tables.
It also implements flavors and flavorprofiles apis.

Partially-Implements: Blueprint octavia-lbaas-flavors
Co-Authored-By: Michael Johnson <>

Change-Id: I99a673438458757d0acdaa46dd8ee041edb3be9c
4 years ago
Michael Johnson 66298f9a48 Add amphora statistics to the admin API
This patch adds an admin API for getting per-amphora statistics.

Change-Id: Ib57b2136dbb41067d6b8949ee42f946f109616e7
5 years ago
Yang JianFeng ae8d6fb2de Add compute_flavor field for amphora api
Operators want to have the ability to see amphora flavor information.
But they haven't access permisson of octavia configuration file. So
it is necessary to show amphora flavor information as part of command
'openstack loadbalancer amphora list/show'.

Story: 2002896
Task: 22986

Change-Id: Ib3ca05d816747d08ef7055ec532b81746468cbf9
5 years ago
wangxiyuan d62189366c Tags support for lb resources
Add tags support for all lb related resources. It includes:
load balancer, listener, member, pool, L7rule, L7policy
and health-monitor

Change-Id: Ib33a002b3b59820db29897454e9d4303c73310b2
Story: 2003890
Task: 26757
5 years ago
Luis Tomas Bolivar feff3b472c Ensure pool object contains the listener_id if passed
When creating a pool with an associated listener_id, sometimes the
listener_id is not set. Problem comes from [1] not setting the listeners
property on db_pool pool object even if it is passed.


Story: 2003833
Change-Id: I414a3c6d2e97d30ff12eda4ecf71ec032998df7d
5 years ago
sapd 62192dba3a Support REDIRECT_PREFIX action for L7Policy
Currently, Octavia only support three actions for L7Policy,
in this patch we will implement new action for L7Policy.

Story: 2003700

Change-Id: Ie99591ede097b566294ebdb673c460442dd6d942
5 years ago
Zuul 7fa83ef8e6 Merge "Fix the API list performance regression" 5 years ago
Zuul 89d992773e Merge "Fix health manager performance regression" 5 years ago
Michael Johnson f15b43ddf4 Fix the API list performance regression
This patch fixes the Octavia v2 API "list" performance regression.

It also corrects some database model forward reference issues.

Co-Authored-By: Adam Harwell <>
Change-Id: Ic110e0e73938743c1aba01aa28f393bae7141cbd
Story: 2002933
Task: 22920
5 years ago
Carlos Goncalves 92473ce210 Make health checks resilient to DB outages
Octavia is struggling with proper handling of DB connectivity issues
bringing down all running loadbalancers. Octavia tries to failover
amphorae and can fail in one of the following stages:

1. Octavia can't create new amphora because Nova isn't ready yet after
   DB outage. Nova-API throws 500, Octavia nukes amphora instance and
   won't try to recreate it again.
2. Octavia tries to recreate amphora instance but it gets stuck in
   PENDING_CREATE forever.
3. Octavia fails completely reporting DB connection issues, leaving some
   amphoras in error, some in pending_delete as bellow: It affects also
   HA deployments.

This patch fixes that by wrapping the DB check for health, waiting for
the connection to be re-established and sleeping off the full
"heartbeat_timeout" interval.

Story: 2003575
Task: 24871

Change-Id: I7b30cd31e1ce0cf9dab61484f4404f1c6ccddd5e
5 years ago
Michael Johnson f13a2e6546 Fix health manager performance regression
When running stress tests against the Octavia Health Manager it was
observed that the scalability and performance of the health manager has
It was observed that the ORM layer was forming poorly optimized queries,
putting excessive load on the database engine and unnecessary code paths
were executing for each heartbeat message.
This patch optimizes the health manager processing of amphora-agent
heartbeat messages by optimizing the database requests, pool processing,
and event streamer code paths.

Change-Id: I2f75715b09430ad139306d9196df0ec5d7a63da8
Story: 2001896
Task: 14381
5 years ago
ZhaoBo 4c6846a568 UDP for [3][5][6]
Add new types into db table.
Extending the existing API, including Listener, Pool, HealthMonitor for
UDP fields support.

For healthmonitor part, need to wait for other patch to fix the default

Patch List:

[1] Finish keepalived LVS jinja template for UDP support
[2] Extend the ability of amp agent for upload/refresh the keepalived
[3] Extend the db model and db table with necessary fields for met the new
udp backend
[4] Add logic/workflow elements process in UDP cases
[5] Extend the existing API to access udp parameters in Listener API
[6] Extend the existing pool API to access the new option in
session_persistence fields

Story: 1657091
Task: 5484
Change-Id: If728705f142f4195fe624bd9ef17413722d54fe3
5 years ago
Zuul 3951eab3a7 Merge "Cleanup Octavia create VIP ports on LB delete" 5 years ago
Michael Johnson 5785b9755e Cleanup Octavia create VIP ports on LB delete
This patch updates the Octavia load balancer VIP handling to deallocate
the VIP if it was created by Octavia. User provided or provider driver
supplied VIPs will not be deallocated.

Change-Id: Idb62a53197975a4aa52fbc3fb5ee192f261b08d8
5 years ago
Zuul 26852b00de Merge "Improve resource quota response message" 5 years ago
huangshan 0ca412ec51 Improve resource quota response message
Story: 2002726
Task: 22569

Change-Id: Ib102a3a9d9938997acc2d5f0b48ed46afb2486b6
5 years ago
Zuul a74f8b4874 Merge "fix tox python3 overrides" 5 years ago
Carlos Goncalves fd20d06896 Fix fields translation on filtering
Filtering load balancers by VIP data would raise an internal server
error (500 HTTP) in octavia-api. This patch fixes that as well as pool
filtering by healthmonitor_id.

The patch also adds vip.qos_policy_id to the load balancer filtering

Story: 2001944
Task: 15061

Change-Id: Iaa4877f71e98689b7e92b3e2abb9d7da2c5ca521
5 years ago
Doug Hellmann 0322cbc5c3 fix tox python3 overrides
We want to default to running all tox environments under python 3, so
set the basepython value in each environment.

We do not want to specify a minor version number, because we do not
want to have to update the file every time we upgrade python.

We do not want to set the override once in testenv, because that
breaks the more specific versions used in default environments like
py35 and py36.

This patch also updates pylint to 1.5.6 which is compatible with
In updating pylint we have some issues to correct, this patch addresses
those issues so the Octavia code passes pylint 1.5.6.

Change-Id: Iec21f4c803a427059d595612336d67a35ebf9585
Signed-off-by: Doug Hellmann <>
5 years ago
Michael Johnson 5af7133a02 Align logging on oslo_log
This patch aligns all of Octavia to use oslo_log instead of the built
in python logging. This should provide consistent log formats.

It adds a hacking check to make sure "logging" doesn't come back into
the code.

Change-Id: I9b76c2bb5a5c396faf85df4606f2ca00f23de913
5 years ago
Michael Johnson 1417f6f0f8 Improve the error logging for zombie amphora
In the case that nova failed to delete an amphroa they will continue to send
health heartbeat messages the the health manager. This patch improves the
logging of these amphora.

It also optimizes the statistics update flow when event streaming is
disabled by removing two extra database calls.

This patch also removes the un-used BaseControllerTask class.

This patch also finally solidifies that there will be one LB per amphora.

Change-Id: Idf83b19216c680a4854c1239ed9c5bc5ce7364a7
5 years ago
Adam Harwell 96cce3ed74 Fix amp failover where failover already failed
If a failover ran on an amphora and was unsuccessful and reverted, it
would mark the amp status "DELETED" and un-busy the health record.
It would then be picked up on the next failover check, start failing
over, and break early since it appeared to be "unallocated".

Also, housekeeping can now clean up expired amphora records based on the
amphora's updated_at time instead of the health record's time, which
means the records won't be immediately cleaned up anymore after they go
through failover flows.

Change-Id: I848b7fc69b977fcb39f8a07e2ea5fc7bd37b5c7a
5 years ago
Michael Johnson 7b2621fe29 Implement provider drivers - Load Balancer
This patch adds provider driver support to the Octavia v2 API, starting
with the load balancer API.

This patch also creates a provider driver for Octavia, initially fully
implementing the load balancer methods.

Follow on patches will implement the remain parts of the API.

Change-Id: Ia15280827799d1800c23ed76d2af0e3596b9d2f7
Story: 1655768
Task: 5165
5 years ago
Michael Johnson 9b98197ca4 Mark LBs in failover PENDING_UPDATE
Load balancers should be in provisioning_status PENDING_UPDATE while an
amphora on the load balancer is being failed over.

Change-Id: I3557b5a1009d7f555860a7c9a6f4abaeb4a12584
Story: 2002067
Task: 19721
5 years ago
Michael Johnson 97c0eab918 Fix calls to "DELETED" items
Fixes the v2 API returning "DELETED" records until the amphora_expiry_age
timeout expired. The API will now immediately return a 404 HTTP status
code when deleted objects are requested. The API version has been raised
to v2.1 to reflect this change.

Change-Id: Iaf150240b0de32f75ba8cfe605293e3af086cc78
Story: 2001557
Task: 6501
5 years ago
Adam Harwell bb0447e98b Expose timeout options
Various timeout options need to be exposed to enable use-cases more
complex than standard HTTP requests.

In this patch we expose four new timeout values:
* timeout_client_data
* timeout_member_connect
* timeout_member_data
* timeout_tcp_inspect

Change-Id: Id4667201c1bfaa06f7af9060c936ba00c2f314f9
Story: 1457556
Task: 5453
5 years ago