Load balancers with multiple listeners, running on an amphora image
with HAProxy 1.8 or newer can experience excessive memory usage that
may lead to an ERROR provisioning_status.
This patch resolves this issue by consolidating the listeners into
a single haproxy process inside the amphora.
Story: 2005412
Task: 34744
Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: Idaccbcfa0126f1e26fbb3ad770c65c9266cfad5b
In testing we saw a DB deadlock in allocate_and_associate.
This patch corrects that with a DB retry wrapper.
Change-Id: I20572505e448f9b7ff6314b5167cc4c519c445fb
Story: 2005893
Task: 33751
In commit a205ab3ebe a new model
spares_pool has been introduced but the definition introduced by
0b468090e6 was missing the primary key.
In this commit we fixed the definition and add an update script for
already deployed env.
Note: without the PK this creates an error when Running Percona with
pxc_strict_mode == ENFORCING
Story: 2005531
Task: 30658
Change-Id: If70e5df5aea008e8998eaad912ba85dfb6bda77a
Signed-off-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@canonical.com>
In order to support Python 3.7, pylint has to be updated to 2.0.0
minimum. Newer versions of Pylint enforce additional checkers which can
be addressed with some code refactoring rather than silently ignoring
them in pylintrc; except useless-object-inheritance which is required to
be silented so that we stay compatible with Python 2.x.
Story: 2004073
Task: 27434
Change-Id: I52301d763797d619f195bd8a1c32bc47f1e68420
Includes some updates to docs and configs and related files to remove
references to neutron-lbaas. Also remove handlers.
Change-Id: I3082962841d3b645f3cbd1a6b41fc7fb28dcf7e6
When no UDP listeners are present, skip the UDP health-check code
branch, which prevents expensive and unnecessary DB calls.
Also optimise the UDP health-check code so it only fetches information
for relevant listeners.
Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: I7fde54084b39c1d0529cfb0bcfb79697d63ea6ae
This patch fixes an issue when the SparesPool table is empty that blocks
spare amphorae creation. It creates a new spares pool entry if the table
is empty.
Story 2005352
Task 30306
Change-Id: I0ce2778277640ee9e509c709bf8621b8b025d6d3
This patch fixes a bug where the housekeeping controllers may launch
more spare amphora than the spare_amphora_pool_size setting.
Story: 2003094
Task 23186
Change-Id: I4c98b3442d2471662488184fa7e91ac64ec33279
This patch adds 2 new options for healthmonitor HTTP health check.
'http_version' is for user to specify the HTTP version, 1.0 and 1.1 are
available.
'domain_name' is for user to specify the HTTP host header inject to check
the HTTP backend health.
'domain_name' only available when HTTP version is 1.1
Story: 2002160
Task: 20010
Change-Id: Id3bf3962a02fbf77cf886c40ac64588cbacd3832
Currently, L7Policy already support the redirection by url_prefix.
Then we can support the redirection with HTTP code.
This patch adds an new option 'redirect_http_code' to L7Policy API.
Story: 2003609
Task: 24941
Change-Id: Id0c9c376ffbc2fb10ddb988537d0ef1a8205e586
Add "tls_enabled" option in Pool API.
This option will work on cert cases or no cert cases.
Story: 2003858
Task: 26672
Co-Authored-By: Michael Johnson <johnsomor@gmail.com>
Change-Id: I62e31aaa66748ba652dfd5dbfd5a8b06d9ba0dfe
Add tls_ca_container_id and crl_container_id into Pool API.
Story: 2003858
Task: 26672
Co-Authored-By: Michael Johnson <johnsomor@gmail.com>
Change-Id: I6cd6e2ca8e48a5df707a70d22505dec9d752c7eb
Add 1 fields like Listener does, which is 'tls_container_ref', this
field is introduced into Pool for storage the pool client certificate to
the backend servers, when the traffic willing to bring a cert to the
servers and check for tls connection.
Story: 2003859
Task: 26685
Change-Id: I29b7c7116e6087c942179ed9efdead494ef277a3
This patch add 4 new types for SSL connection ACL configuration.
Which are:
L7RULE_TYPE_SSL_CONN_HAS_CERT
L7RULE_TYPE_VERIFY_RESULT
L7RULE_TYPE_DN_FIELD
The first type can just accept the compare type "EQUAL_TO" and value
"True" string.
The second can just accept the int value string to check the certificate
verify result, also just support "EQUAL_TO" compare type.
The third can accept key, the distinguished name field and a match string,
this one supports all kind compare types.
Story: 2002165
Task: 20025
Co-Authored-By: Michael Johnson <johnsomor@gmail.com>
Change-Id: I71b57d0f32d4839a770396645d2b9945d24f2853
Add crl-file in Listener side.
Story: 2002165
Co-Authored-By: Michael Johnson <johnsomor@gmail.com>
Change-Id: I9e2ec06719fbbfd19482c2b8d39220e7e4ed81e3
Listener API for client cerificate authentication with "None,
Optional, Mandatory" options
Story: 2002165
Task: 20019
Co-Authored-By: Michael Johnson <johnsomor@gmail.com>
Change-Id: Ia753659981d99b315504f166c09afb8f5b14f195
This patch add 'client_ca_tls_container_ref' into listener API for front
client authentication.
Story: 2002165
Task: 20018
Co-Authored-By: Michael Johnson <johnsomor@gmail.com>
Change-Id: I8a96d6fdfe53a16d1abcfd09bc6afedd6c490de2
The Housekeeping service grows in utilization of CPU as more amphorae
are created and/or marked as DELETED. The problem lays on the SELECT
statement constructed in get_all_deleted_expiring_amphora via the ORM --
it is joined eager loading all relationships.
The task does not need such amount of information, only the amphora ID.
The statement could be simplified by not loading any relationship or, at
most, lazy loading them.
This patch also fixes performance of cleaning up deleted and expired
load balancers. The code was doing multiple round-trips to the database
unecessarily:
1. retrieving all deleted LBs
2. for each LB, retrieving it again from DB to check expired date
3. delete LB
Step 1 and 2 are now condensed in get_all_deleted_expiring(), making it
a single round-trip.
Story: 2004665
Task: 28643
Change-Id: Iffc960c7c3a986328cfded1b4e408931ab0a7877
An earlier patch[1] attempted to fix the vip address storage size in the
octavia database, however it issued the alembic "alter_column" command
incorrectly and the column size was not fixed.
This patch corrects that with another alembic migration that uses the correct
syntax.
[1] https://review.openstack.org/#/c/339826
Change-Id: Ib577f92be38fa9a1c84571729b0236913da8d9d2
This patch adds support for flavor metadata validation by the amphora driver
and support for setting the load balancer topology via a flavor.
It also adds "flavor_id" to the load balancer table in the database.
Change-Id: I8eae870abdb20dc32917957e32606deef387ec88
This patch adds flavor and flavor_profile tables.
It also implements flavors and flavorprofiles apis.
Partially-Implements: Blueprint octavia-lbaas-flavors
Co-Authored-By: Michael Johnson <johnsomor@gmail.com>
Change-Id: I99a673438458757d0acdaa46dd8ee041edb3be9c
Operators want to have the ability to see amphora flavor information.
But they haven't access permisson of octavia configuration file. So
it is necessary to show amphora flavor information as part of command
'openstack loadbalancer amphora list/show'.
Story: 2002896
Task: 22986
Change-Id: Ib3ca05d816747d08ef7055ec532b81746468cbf9
Add tags support for all lb related resources. It includes:
load balancer, listener, member, pool, L7rule, L7policy
and health-monitor
Change-Id: Ib33a002b3b59820db29897454e9d4303c73310b2
Story: 2003890
Task: 26757
Currently, Octavia only support three actions for L7Policy,
in this patch we will implement new action for L7Policy.
Story: 2003700
Change-Id: Ie99591ede097b566294ebdb673c460442dd6d942
This patch fixes the Octavia v2 API "list" performance regression.
It also corrects some database model forward reference issues.
Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: Ic110e0e73938743c1aba01aa28f393bae7141cbd
Story: 2002933
Task: 22920
Octavia is struggling with proper handling of DB connectivity issues
bringing down all running loadbalancers. Octavia tries to failover
amphorae and can fail in one of the following stages:
1. Octavia can't create new amphora because Nova isn't ready yet after
DB outage. Nova-API throws 500, Octavia nukes amphora instance and
won't try to recreate it again.
2. Octavia tries to recreate amphora instance but it gets stuck in
PENDING_CREATE forever.
3. Octavia fails completely reporting DB connection issues, leaving some
amphoras in error, some in pending_delete as bellow: It affects also
HA deployments.
This patch fixes that by wrapping the DB check for health, waiting for
the connection to be re-established and sleeping off the full
"heartbeat_timeout" interval.
Story: 2003575
Task: 24871
Change-Id: I7b30cd31e1ce0cf9dab61484f4404f1c6ccddd5e
When running stress tests against the Octavia Health Manager it was
observed that the scalability and performance of the health manager has
degraded.
It was observed that the ORM layer was forming poorly optimized queries,
putting excessive load on the database engine and unnecessary code paths
were executing for each heartbeat message.
This patch optimizes the health manager processing of amphora-agent
heartbeat messages by optimizing the database requests, pool processing,
and event streamer code paths.
Change-Id: I2f75715b09430ad139306d9196df0ec5d7a63da8
Story: 2001896
Task: 14381
Add new types into db table.
Extending the existing API, including Listener, Pool, HealthMonitor for
UDP fields support.
For healthmonitor part, need to wait for other patch to fix the default
value.
Patch List:
[1] Finish keepalived LVS jinja template for UDP support
[2] Extend the ability of amp agent for upload/refresh the keepalived
process
[3] Extend the db model and db table with necessary fields for met the new
udp backend
[4] Add logic/workflow elements process in UDP cases
[5] Extend the existing API to access udp parameters in Listener API
[6] Extend the existing pool API to access the new option in
session_persistence fields
Story: 1657091
Task: 5484
Change-Id: If728705f142f4195fe624bd9ef17413722d54fe3
This patch updates the Octavia load balancer VIP handling to deallocate
the VIP if it was created by Octavia. User provided or provider driver
supplied VIPs will not be deallocated.
Change-Id: Idb62a53197975a4aa52fbc3fb5ee192f261b08d8
Filtering load balancers by VIP data would raise an internal server
error (500 HTTP) in octavia-api. This patch fixes that as well as pool
filtering by healthmonitor_id.
The patch also adds vip.qos_policy_id to the load balancer filtering
capabilities.
Story: 2001944
Task: 15061
Change-Id: Iaa4877f71e98689b7e92b3e2abb9d7da2c5ca521
We want to default to running all tox environments under python 3, so
set the basepython value in each environment.
We do not want to specify a minor version number, because we do not
want to have to update the file every time we upgrade python.
We do not want to set the override once in testenv, because that
breaks the more specific versions used in default environments like
py35 and py36.
This patch also updates pylint to 1.5.6 which is compatible with
python3.
In updating pylint we have some issues to correct, this patch addresses
those issues so the Octavia code passes pylint 1.5.6.
Change-Id: Iec21f4c803a427059d595612336d67a35ebf9585
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
This patch aligns all of Octavia to use oslo_log instead of the built
in python logging. This should provide consistent log formats.
It adds a hacking check to make sure "logging" doesn't come back into
the code.
Change-Id: I9b76c2bb5a5c396faf85df4606f2ca00f23de913
In the case that nova failed to delete an amphroa they will continue to send
health heartbeat messages the the health manager. This patch improves the
logging of these amphora.
It also optimizes the statistics update flow when event streaming is
disabled by removing two extra database calls.
This patch also removes the un-used BaseControllerTask class.
This patch also finally solidifies that there will be one LB per amphora.
Change-Id: Idf83b19216c680a4854c1239ed9c5bc5ce7364a7
If a failover ran on an amphora and was unsuccessful and reverted, it
would mark the amp status "DELETED" and un-busy the health record.
It would then be picked up on the next failover check, start failing
over, and break early since it appeared to be "unallocated".
Also, housekeeping can now clean up expired amphora records based on the
amphora's updated_at time instead of the health record's time, which
means the records won't be immediately cleaned up anymore after they go
through failover flows.
Change-Id: I848b7fc69b977fcb39f8a07e2ea5fc7bd37b5c7a
This patch adds provider driver support to the Octavia v2 API, starting
with the load balancer API.
This patch also creates a provider driver for Octavia, initially fully
implementing the load balancer methods.
Follow on patches will implement the remain parts of the API.
Change-Id: Ia15280827799d1800c23ed76d2af0e3596b9d2f7
Story: 1655768
Task: 5165
Load balancers should be in provisioning_status PENDING_UPDATE while an
amphora on the load balancer is being failed over.
Change-Id: I3557b5a1009d7f555860a7c9a6f4abaeb4a12584
Story: 2002067
Task: 19721
Fixes the v2 API returning "DELETED" records until the amphora_expiry_age
timeout expired. The API will now immediately return a 404 HTTP status
code when deleted objects are requested. The API version has been raised
to v2.1 to reflect this change.
Change-Id: Iaf150240b0de32f75ba8cfe605293e3af086cc78
Story: 2001557
Task: 6501
Various timeout options need to be exposed to enable use-cases more
complex than standard HTTP requests.
In this patch we expose four new timeout values:
* timeout_client_data
* timeout_member_connect
* timeout_member_data
* timeout_tcp_inspect
Change-Id: Id4667201c1bfaa06f7af9060c936ba00c2f314f9
Story: 1457556
Task: 5453
Michael Johnson