Now that we are python3 only, we should move to using the built
in version of mock that supports all of our testing needs and
remove the dependency on the "mock" package.
This patch moves all references to "import mock" to
"from unittest import mock". It also cleans up some new line
inconsistency.
Change-Id: I72520a2ca010c2c27315d9dff839a4f9d7540b6b
Code was not using the correct filenames for the 'route',
'route6', 'rule' and 'rule6' files on Red Hat images.
Changed to use config option 'agent_server_network_file'
if it's specified, else the file of the correct name, and
added unit tests for each.
Change-Id: I335287da66524d026f0c42086d885b478c568bbd
Task: 37881
Story: 2007051
Fix unit test test_cmd_get_version_of_installed_package_mapped when run
on a CentOS 7 system.
testtools.matchers._impl.MismatchError: !=:
reference = 'rpm -q --queryformat %{VERSION} haproxy'
actual = 'rpm -q --queryformat %{VERSION} haproxy18'
Change-Id: I81f76afc7e3fa6f190c30f33198197cb627cce26
The single process patch changed the way listeners and load balancers
are deployed inside the amphora. This caused listeners with SNI
enabled to load all of the certificates for all of the TLS enabled
listeners on a load balancer.
This patch corrects that by configuring each listener with a
specific list of certificates.
Change-Id: I2f3c7ab4137dbd84d77a6a6b675975af406249d0
Story: 2006758
Task: 37252
Mocks were done slightly incorrectly for pyroute2, correct this and add
a missing mock for os.path.isfile since it only coincidentally worked.
Change-Id: Id178784ebb4a9da348d295201fd6aa715c5b4abe
Re-factored code to not insert a log-format line in haproxy
config when 'no log' is set in the template. Adjusted tests
as necessary.
Story: 2006181
Task: 35706
Change-Id: I8a0cdf7ee912d39d9ac24244728094a1d1e15c63
Currently jinja_combo.build_config method expect to use single
tls cert, though with multiple listeners there could be multiple
certs. Also in case of HTTP and TERMINATED_HTTPS listeners on the
same loadbalancer - creation of the second listener will fail.
Change-Id: Iad3b55e5add4283256f7836c3d4a501aa57ffc2f
Story: 2006513
Task: 36510
Currently the amphora agent will lookup interfaces using the
interface name determined earlier in the plug method. This can
lead to a race condition with the udev interface renaming rule.
This patch changes the interface lookup to use the MAC address
directly and not rely on the interface name.
Story: 2006300
Task: 36013
Change-Id: I5bc21d5abdeb67a3a8ae88456735643463f15694
In some environments running older versions of gunicorn in the
amphora image, gunicorn can fail to start do to /dev/log socket
issues (timing, configuration, etc.).
This patch sets up a dedicated rsyslog socket /run/rsyslog/octavia/log
for gunicorn and haproxy to use. This should resolve any issues with
systemd overriding the /dev/log socket.
This also bumps the gunicorn minimum verison to 19.9.0.
Change-Id: I1e1ad8fde2ad8c1ffba95b1867afb130503b0a5b
Load balancers with multiple listeners, running on an amphora image
with HAProxy 1.8 or newer can experience excessive memory usage that
may lead to an ERROR provisioning_status.
This patch resolves this issue by consolidating the listeners into
a single haproxy process inside the amphora.
Story: 2005412
Task: 34744
Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: Idaccbcfa0126f1e26fbb3ad770c65c9266cfad5b
Pyroute2 recently changed the way they import their base namespace
ojects. Now they are somewhat platform dependent, so on OSX unit tests
will fail for lack of linux-specific libs. Using `create=true` allows
the mocks to be created even if the library doesn't think they exist.
b69999ec82
Change-Id: I8f9c25e979a7e86ff0b8e19267e4525817802ba2
An exception handler in the amphora-agent has a python3 string
comparison bug that will cause a TypeError.
This patch fixes that bug and adds test coverage for the
start_stop_listener.
Change-Id: I6f5d95c5f875edda530f54ae72386d6495235ca6
Story: 2005898
Task: 33760
Configure rsyslog to forward logs to a target host
Co-Authored-By: Michael Johnson <johnsomor@gmail.com>
Story: 1665069
Task: 33646
Change-Id: I00703f86555cbb574b943794b14a36fbc644f1b2
This patch configures the primary components of the amphora to log
to syslog using consistent logging facilities.
By default, user traffic logs will go to LOG_LOCAL0 and the amphora
processes (haproxy, keepalived, etc.) will log to LOG_LOCAL1.
This is a patch supporting log offloading.
Change-Id: Ifda91e0310e812e34f1e398dd3176af8a9c58f89
Story: 1665069
Task: 5486
Occasionally the test jobs[1] will fail with:
octavia.amphorae.drivers.haproxy.rest_api_driver [-]
Could not connect to instance. Read timed out. (read timeout=120.0)
This patch increases the default read timeout to 180 and changes the
directory copy that would subsequently fail to be more idempotent.
[1] http://logs.openstack.org/09/613709/14/check/ \
octavia-v2-dsvm-scenario-two-node/d83db12/controller2/logs/ \
screen-o-cw.txt.gz#_Feb_08_21_58_23_919928
Change-Id: Ia0bd6762c2605ce240a549b3e90e5c44b65897a5
This patch fixes an oversight in the addition of flavors support in the
amphora driver[1]. The amphora-agent configuration file was still getting
the topology selected in the configuration file as opposed to the
topology selected in the flavor.
This is an additional patch at the end of the chain as it leverages
changes that were made in later flavor patches that pass the flavor
into the taskflow flow.
A follow on patch will address spares pool amphora.
[1] https://review.openstack.org/#/c/621323
Change-Id: I4c2b398b562970f128e06794690ffd7c2977db08
Currently we calculate prefix based on netmask when writing the vip
interface file. Since netmask has been converted to prefix in ipv6,
this patch will avoid converting it to prefix twice which could
result in a wrong prefix length.
Also fix a bug in another test that relies on osutils, but wasn't
mocking correctly.
Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Change-Id: I9ee0cce12a975f4ab8f3df2707b355aab35c6cb3
Octavia is plugging member networks, but only bringing up the first
fixed IP address on that network. This can mean that a secondary
fixed IP on the network, such as the IPv6 address is not brought
up at member creation time.
Change-Id: Ic5b19a303e53ab62875c4fc4be6ac03f926a6832
Co-Authored-By: Adam Harwell <flux.adam@gmail.com>
Story: 2004113
Task: 27535
Load balancers with IPv6 VIP addresses would fail to create due to
a duplicate address detection issue. The keepalived process would also
crash with a segfault due to a known bug[1].
This patch resolves both issues and allows load balancers with IPv6
VIP addresses to be created in active/standby topology.
[1] https://github.com/acassen/keepalived/issues/457
Story: 2003451
Task: 24657
Co-Authored-By: Michael Johnson <johnsomor@gmail.com>
Change-Id: I15a4be05740e2657f998902d468e57763c3ed52e
This patch adds a few optimizations when using the amphora driver.
1. It increases the amp_active_retries from 10 to 30. This increases
the time we wait for nova to mark an instance "ACTIVE". The old default
of 10 was one minute forty seconds, but in some clouds it's been observed
that the nova schedule can get overloaded and take longer than a minute
forty to schedule the instance. Setting this to 30 means we will wait
five minutes for nova to schedule the instance.
2. It enables TCP kernel splicing in HAProxy. This has been shown to
reduce the CPU overhead for very high rate TCP load balancers.
3. Finally it enables "safe" HTTP keepalives on the backend member
connections [1]. This increases the request rate possible while using HTTP
protocol listeners and members.
[1] http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#4-http-reuse
Change-Id: I3af009cac9a9edc8aef793b52c6a1488fde2c59b
Provide a sane return value for get_keepalivedlvs_pid
to ensure that underlying Python libary code does not
throw a TypeError when joining values:
TypeError: join() argument must be str or bytes, not 'MagicMock'
Change-Id: I174bbaf618a845cc246f3b1926d600d03e710de3
In Pike[1], we introduced a user_group auto detection for haproxy.
The default user group name is auto-detected for any OS distribution
we support as a base for Amphorae.
user_group remained as an option for admins but was also
marked deprecated in Pike[2].
This patch removes that option altogether.
Story: 2003323
Task: 24357
[1] Ia8fede9d7da4709a48661d1fc595a16d04fcbfa9
[2] https://review.openstack.org/#/c/429398/45/octavia/common/config.py@175
Change-Id: Iddd4162674f116705d2b47062cbf7ca88f2677a6
1. Removes the misc_dynamic setting from the UDP-CONNECT health monitor
as our script does not use it.
2. Adds a release note for the UDP features.
3. Updates the API reference for UDP support.
4. Adds a comment to the keepalived config with the LB ID.
5. Updates the status message type to be the correct UDP protocol.
6. Fix error during deleting a listener if there are multiple amphoraes.
7. Refactors systemd service script handling.
Story: 2003306
Task: 24258
Change-Id: I09240023d066ac5a71836d01045cda6ce5678712
These files will split with the current Octavia repo, before other parts
are ok.
Patch List:
[1] Finish keepalived LVS jinja template for UDP support
[2] Extend the ability of amp agent for upload/refresh the keepalived
process
[3] Extend the db model and db table with necessary fields for met the new
udp backend
[4] Add logic/workflow elements process in UDP cases
[5] Extend the existing API to access udp parameters in Listener API
[6] Extend the existing pool API to access the new option in
session_persistence fields
Change-Id: Ib4924e602d450b1feadb29e830d715ae77f5bbfe
When using the Octavia/amphora driver, unspecified or unlimited (-1)
settings would lead to a 2000 connection limit in HAproxy.
This patch updates that to be 1,000,000 connections.
1,000,000 was selected to amphora memory usage at a reasonable level.
Change-Id: Iddeb62412bb71b69cf1e9198be6131c59a3051b0
Story: 1635416
Task: 5159
This is useful for debugging purposes, and could theoretically be
interesting data for custom elements.
Story: 2001236
Task: 5756
Change-Id: If5f7793bd16fd6c8841586f29cdcf5e53908b1fe
This patch fixes an issue that caused an internal error to the amphora
agent and led amphorae to transition to FAILURE state. The issue was
that on CentOS-based amphorae HAproxy package name is 'haproxy18' while
the expected package name is 'haproxy'.
The patch introduces a package name per distro map that is consulted
whenever a package has a different name than the one expected. If
there's a miss, it falls back to the expected package name.
Story: 2002958
Task: 22960
Change-Id: I59dfa3f9bd5974302ec72343444813232ec2ea02
Non-Debian based amphora agents raise internal server error when
querying for a package version. That is due to the fact that the agent
currently assumes dpkg-based distros ignoring other OS families such as
Red Hat, SUSE, etc where the output from package info querying output is
slightly different from dpkg.
Story: 2002961
Task: 22963
Change-Id: Ied37bb0191ca8af4966d097f47ec80536dfc843f
The builtin platform.linux_distribution[1] is deprecated and will be
removed in 3.8 and the recommended replacement is distro.
This also raises a "deprecated method" error in pylint.
This patch moves us over to the future by following the recommendation
in the python docs and switching to use the "distro" module.
[1] https://docs.python.org/2/library/platform.html \
#platform.linux_distribution
Depends-On: https://review.openstack.org/578983
Change-Id: I29e2673572eab75b553da6b01143b007701808fd
The common name is used as a file name inside the HAproxy
configuration file. However, a common name can include spaces
and it will result in a configuration file that simply doesn't
work because of the spaces.
The patch changes the functionality so that it instead creates
a SHA1 hash of the certificate and uses that as the file name
to avoid those issues.
Change-Id: I039ed0b40df8b72a1238f8896548fe77086c530c
Various timeout options need to be exposed to enable use-cases more
complex than standard HTTP requests.
In this patch we expose four new timeout values:
* timeout_client_data
* timeout_member_connect
* timeout_member_data
* timeout_tcp_inspect
Change-Id: Id4667201c1bfaa06f7af9060c936ba00c2f314f9
Story: 1457556
Task: 5453
This also fix build-openstack-sphinx-docs, there was a change introduced
in sphinx 1.6.6:
https://github.com/sphinx-doc/sphinx/pull/4335/files
If the size of __init__.py is less than 2, then the module would be
skipped which will cause the sphinx consistency checking failing later.
Change-Id: I9d8764b6e907aceed8bb8a9b04711145d0eb32ad
Currently with Octavia, if the user specifies a health monitor of type
"PING" we are still using a TCP connect to check for health.
This patch fixes that to actually ping the member to validate health.
Change-Id: I8a67efb7113ffa49b2805b37c3855373b17e5789
Story: 2001280
Task: 5826
This addition automates the process, as opposed to relying on review inputs.
Inspired by Ib51bd97dc4394ef2b46d4dbb7fb36a9aa9f8fe3d
Change-Id: I1d6051cf6678b6d5db774fc884390fec626c1f2c
Some unit tests were doing some not-so-unit things and actually reaching
out to OS / files for things. This leads to failures when running tests
on OSX or other distributions.
Change-Id: Iff2cb4d2742a5fd8a5cf4f1bea0ab08504f46016
Michael Johnson