Functional tests log to a file only if they inherit from
the Sudo tests base class. This patch changes the base
class for some test cases to make them log.
Related-Bug: #1567668
Change-Id: I494ad5410e48489f1fb3689cec44c5a29bbc42f3
Running the tests locally I'm seeing:
http://paste.openstack.org/show/493222/
The issue in all three tests is that the keepalived manager
process spawns asynchronously (sudo -> root_helper -> keepalived).
For the 'spawns' test, it was asserting that process.alive was
True when it should have used wait_until_true.
For the respawns* tests, the test grabs process.pid before the process
has necessarily spawned. Moving process.pid to after the wait_until_true(
process.active) loop guarantees that pid will return keepalived's
process. Otherwise the 'pid' variable will be None, then when we
use utils.execute we try to kill the 'None' pid, and that doesn't seem
to work.
Change-Id: Ie77d406eaaf7f77edd4f598947999be4adf3d249
Closes-Bug: #1567023
Extracting the test fixture that creates a new process and leaves it
running for a given amount of time into helpers where other fixtures for
functional tests live. This both keeps the fixtures at one place and
increases visibility of the fixture so that it can be reused in other
tests. At the same time, the fixture is fixed as the original code
omitted starting the process.
Change-Id: I97aeb8d1d5773ef3d59e8f908aea34ccceb38378
Related-Bug: 1561046
keepalived refuses to start and claims "daemon already started"
when there is already a process with the same PID as found in
either the VRRP or the main process PID file. This happens even
in case when the new process is not keepalived. The situation
can happen when the neutron node is reset and the obsolete PID
files are not cleaned before neutron is started.
This commit adds PID file cleanup before keepalived start.
Closes-Bug: 1561046
Change-Id: Ib6b6f2fe76fe82253f195c9eab6b243d9eb76fa2
When keepalived crashed unexpectedly, the vrrp process that
it associates with will be orphan process. This will make
the VIP unable to migrate to the router in the same host.
Also, neutron code is not able to respawn the keepalived
process, because keepalived thinks itself is still running,
according to [1-3]. As a result, neutron will report respawning
keepalived all the time. Restart l3-agent will not help.
This patch will check and delete the orphan vrrp process
if there is any, in the processmonitor of l3 agent.
More details can be found in the bug description and comments.
[1] https://goo.gl/W3GL9I
[2] https://goo.gl/F0Ixfb
[3] https://goo.gl/dUqhTo
Change-Id: Ia1759ed1365b845d404686a8cd25f882cce35caf
Closes-Bug: #1511311
keepalived fails to respawn after crash for > 1.2.11 version.
When keepalived starts, it spawns vrrp thread to monitor vrrp forked
process. It also creates a vrrp pid file. When the process is killed, and
it's restarted, the the new keepalived process runs with -P, so
when we validate whether we are already running, we check vrrp pid file.
Since we never clean up the file before starting the process, and the process
dies without a chance to clean up the file as part of its signal
handler, respawn never works.
keepalived_respawns test should be skipped until bug is resolved.
See also: https://bugzilla.redhat.com/show_bug.cgi?id=1286729
Change-Id: Ic111573e0cd5ad5bfe70b0f38ec0203c10d52e34
Related-Bug: #1511311
This test initial design is problematic: it spawns keepalived,
it asserts the process is up, then it attempts to kill it.
However, this is when problems may arise:
a) it does so by using the disable method on the process - we
should be more rude than that if we want to simulate a crash!
b) keepalived may be forking while it is starting and it is
possible that for a moment the ppid changes and the process
owner invoking the kill has no rights to kill the spawned
process. This is the most plausible explaination I could find
as to why kill returns 1 with no standard error
c) it does not verify that the process has indeed disappeared
(what if the pm.disable didn't work?) - this means that the
test can pass, and yet the monitor may not work.
Bottom line: this test relied on the correctness of the very code
that was meant to validate...and that's not cool. To this aim, we
wait for the process to be active, kill the process with a kill -9
and verify that the process after the kill is indeed different.
Closes-bug: #1490043
Change-Id: Idaf419a1464d9d0d75b9106a7acd5cd960a7c623
Current theory is that there's a bug in external_process.active,
it returns True when it shouldn't, then kill -15 on the process
pid fails because the process isn't up. Added ps -p output to
see if the process is up or not.
Change-Id: Ic062be829d5f05a1294f6b2fa54820422871ffcb
Related-Bug: #1490043
Previously, keepalived process itself was disabled that lead to
respawning of keepalived by KeepalivedManager. This patch disables
KeepalivedManager in cleanup thus no respawn happens.
Closes-Bug: #1466873
Change-Id: If5524116e5d4fc41600920d31481282c5b797f7b
This patch addresses the following.
1. removes the un-used variables.
2. process_monitor (argument to KeepalivedManager) is changed to
a non-default parameter as its used in spawn, disable methods.
Change-Id: I8b130b21965ed3387e994818be947eb95d73a423
Base test class adds a cleanup fixture to stop any
spawned process monitor via unit or functional tests,
which otherwise would keep running after the tests already
finished, and execution functions go unpatched.
Without this patch unit tests will randomly fail
when we enable process monitor by default at change:
I85fe31bee30714148168a293eea29fa0a37f9701
Co-Authored-By: Maru Newby <marun@redhat.com>
Change-Id: Ide799a52391b14ff921de25788e8b0f0463fb8f8
Adding process monitor to keepalived code, so that keepalived processes
launched by the L3 agent will get monitored the same as other processes
launched by the L3 agent.
Old monitoring code was removed since it's not needed anymore.
Implements: blueprint agent-child-processes-status
Change-Id: I94a889ee07286ab3c6cdab9ab15e5aee6fbd133a
Currently functional test classes requiring invocation of commands via a
root helper inherit from BaseSudoTestCase[1] and should call explicitly
check_sudo_enabled in order to skip the test if sudo is disabled[2].
This change performs such check during BaseSudoTestCase.setUp in order
to ensure the check is done and replaces parent class for functional
test cases not requiring invocation of commands via a root helper.
This change prepares to transform BaseOVSLinuxTestCase[3] methods into
helper methods.
[1] in neutron.tests.functional.base
[2] sudo is enabled if OS_SUDO_TESTING is set to '1' or 'True'
[3] in neutron.tests.functional.agent.linux.base
Change-Id: I91a3506524352211729ac8b3b07cf3082fd8f13a
Remove the root_helper arg from the L3 agent.
Change-Id: Ib66813d9823618cdc3caf48673721173f0bdf5fc
Partially-Implements: blueprint rootwrap-daemon-mode
Depends-On: I1e3b64e5a1d6cff2aebc638710487bbdbdba61d4
A recent patch introduced a new keepalived manager utility
function which returns the keepalived.conf file contents,
but the keepalived functional test was not updated to use it.
Also, a bunch of unneeded logging configuration was removed from
this test.
Additionally, the keepalived configuration functional test
in the L3 agent testing module compares expected configuration
to an in-memory representation of the configuration that will
be persisted to disk. It's better to compare the expected
configuation with what was actually written to disk.
Change-Id: I0d6f73c5326d2ba1960456a151a67517db5cce7a
Oslo project decided to move away from using oslo.* namespace for all their
libraries [1], so we should migrate to new import path.
This patch applies new paths for:
- oslo.config
- oslo.db
- oslo.i18n
- oslo.messaging
- oslo.middleware
- oslo.rootwrap
- oslo.serialization
- oslo.utils
Added hacking check to enforce new import paths for all oslo libraries.
Updated setup.cfg entry points.
We'll cleanup old imports from oslo-incubator modules on demand or
if/when oslo officially deprecates old namespace in one of the next
cycles.
[1]: https://blueprints.launchpad.net/oslo-incubator/+spec/drop-namespace-packages
Depends-On: https://review.openstack.org/#/c/147248/
Depends-On: https://review.openstack.org/#/c/152292/
Depends-On: https://review.openstack.org/#/c/147240/
Closes-Bug: #1409733
Change-Id: If0dce29a0980206ace9866112be529436194d47e
testr is occasionally failing with 'Length too long' for
neutron-dsvm-functional test cases.
The issue is seen when the size of the testr log file size goes into MBs.
(Please see the bug comments for the details on the testr logs)
This patch fixes the issue by disabling debug logs in the tests.
Change-Id: I48fc9d14f6886f86761038ee4d106fbbe59b6890
Closes-bug: #1382579
The cfg.CFG argument is ignored and misleading. This change removes it
and updates all callers.
Closes-Bug: #1355565
Change-Id: I2fcece85d1abe848e5c01125cfb62c02f2dcbc86
This patch introduces a keepalived manager
which will be used for the blueprint
blueprint l3-high-availability.
The manager can create a keepalived.conf compliant configuration,
start, stop and restart the service, as well as create keepalived
notification scripts.
Implements: blueprint l3-high-availability
Change-Id: I1ba9f332778f27de950d9e97d4fb4a337f6f26da
Co-Authored-By: Assaf Muller <amuller@redhat.com>
Assaf Muller
Hynek Mlnarik
ChangBo Guo(gcb)
Hong Hui Xiao
Arie Bregman
Akihiro Motoki
armando-migliaccio
Jakub Libosvar
sridhargaddam
Miguel Angel Ajo
Mike Kolesnik
Cedric Brandily
Terry Wilson
Ihar Hrachyshka
Numan Siddique
Angus Lees