Commit Graph

25 Commits (99f4495c940011293e3cabbb590770dc1e7b6900)

Author SHA1 Message Date
Hongbin Lu ae4b331725 fullstack: retry on updating port's IP address
If the update_port call failed with error IpAddressAlreadyAllocatedClient,
retry a few more times in order to find IP addresses that are available.

Change-Id: I7c5d51b01fa56083b1a689fa629a9a34c8b77012
Closes-Bug: #1808595
2018-12-21 16:36:18 +00:00
Slawek Kaplonski f787f12aa3 Make port binding attempt after agent is revived
In some cases it may happen that port is "binding_failed"
because L2 agent running on destination host was down but
this is "temporary" issue.
It is like that for example in case when using L3 HA and when
master and backup network nodes were e.g. rebooted.
L3 agent might start running before L2 agent on host in such case
and if it's new master node, router ports will have "binding_failed"

When agent sends heartbeat and is getting back to live,
ML2 plugin will try to bind all ports with "binding_failed"
from this host.

Change-Id: I3bedb7c22312884cc28aa78aa0f8fbe418f97090
Closes-Bug: #1794809
2018-10-22 18:43:33 +02:00
Miguel Lavalle 9237cf374e Improve error message in fullstack test
_assert_ping_during_agents_restart is used in tests of L2 and L3
agents. However, when it raises an exception due to a timeout, the
associated message assumes the agent under test is L2. This patch
fixes that

Change-Id: I3568c97a621e97699fcd93f09897e132d4db402a
2018-06-08 08:07:54 +00:00
Slawek Kaplonski 32f14aa8a0 [Fullstack] Change time waiting for async ping results
During agents restart there is async ping run and there is
called function to wait until all async ping workers will
finish their job.

In TestHAL3Agent.test_ha_router_restart_agents_no_packet_lost
there are 60 pings sent with 1 second timeout so default
wait_until_true timeout which is set to 60 seconds might not
be enough in some cases.
Because of that wait_until_true timeout is now set as
twice higher value than is needed to number of packets to send
with ping_timeout.

This should give enough time to finish all workers.

Change-Id: Ia7c3755c2ba5029bdab3c1dd30b305f3bde19740
Closes-Bug: #1775183
2018-06-06 15:22:37 +02:00
Slawek Kaplonski 3e9e2a5b4b Disable IPv6 forwarding by default on HA routers
In case of HA routers IPv6 forwarding is not disabled by default and
then enabled only on master node.
Before this patch it was done in opposite way, so forwarding was
enabled by default and then disabled on backup nodes.
When forwarding was enabled/disabled for qg- port, MLDv2 packets are
sent and that might lead to temportary packets loss as packets to
FIP were sent to this backup node instead of master one.

Related-Bug: #1771841

Change-Id: Ia6b772e91c1f94612ca29d7082eca999372e60d6
2018-05-31 20:19:21 +00:00
Brian Haley 4f627b4e8d Change ip_lib network namespace code to use pyroute2
Change network namespace add/delete/list code to use
pyroute2 library instead of calling /sbin/ip.

Also changed all in-tree callers to use the new calls.

Closes-bug: #1717582
Related-bug: #1492714

Change-Id: Id802e77543177fbb95ff15c2c7361172e8824633
2017-10-04 21:09:28 +00:00
Ihar Hrachyshka a99897ffb3 Use rootwrap for fullstack test runner
We plan to switch to devstack-gate for fullstack job, and it revokes
direct sudo calls before executing tests, so we can't rely on sudo
working anymore.

This also moves functional-testing.filters to a more generic filename
(testing.filters) because the filters are now deployed and used by
fullstack target too.

Related-Bug: #1557168
Related-Bug: #1693689

Change-Id: I1718ea51836adbb8ef8dea79822a722dcf111127
2017-06-08 18:42:47 +00:00
Jakub Libosvar bc979efdb8 fullstack: Don't let dhcp agents failover
It turned out dhcp tests work only because agents are considered dead
after 10 seconds while they report to server every 60 seconds. This led
to calling network resync after agent revival and hiding the fact dhcp
agent is not capable of receiving any amqp messages.

This patch sets the report interval of agents to the half of
agent_down_time on server side and uses eventlet dhcp agent in order to
trigger eventlet monkey patching code.

Eventlet was behind the failure with messages not getting processed. As
[1] notes: "Note: If the “eventlet” executor is used, the threading and
time library need to be monkeypatched."

Because each port calls dhclient to obtain IP address and each dhclient
instance overwrites /etc/resolv.conf there was added a script that
generates fullstack-dhclient-script from an existing dhclient-script
before starting fulltstack tests. This generated script is passed to
each dhclient process running in fake fullstack machine using -sf


Related-bug: 1453350

Change-Id: I0336176b9c364fe3a95be5cef9e7a3af1ef9d7e9
2017-04-20 14:04:50 +00:00
Jakub Libosvar fcb1ccf823 fullstack: Log testrunner again
Commit 546ba377db removed logging of
testrunner, this patch adds it back.

Closes-Bug: 1603363
Change-Id: I6075c903b4d233629224b77425b8fb5e793608ce
2016-07-15 05:31:27 -04:00
Mike Bayer 546ba377db Restore MySQL and Postgresql functional testing
The oslo.db opportunistic test fixtures were not being
used effectively and the MySQL / PG databases were not
being used.  This patch restores working patterns against
oslo.db.  Additionally, the migration level tests have also
been updated to make use of oslo.db provisioning functionality
and unused methods have been removed.

The current approach makes use of oslo.db provisioning constructs
directly, as well as some semi-private attribute access within
oslo.db enginefacade, in order to work around some issues
that have arisen in oslo.db's test_base.
A new release of oslo.db will provide
public API points to resolve pending issues, and to
allow neutron's use cases here which will
also be of general applicability to openstack projects.

Closes-bug: #1594898

Change-Id: Ie27cf174fa24c2f479af47335d9ae139fb7d159a
2016-06-29 13:20:17 -04:00
Assaf Muller 8adc737cd1 Delete post_test_hook.generate_test_logs
Log files as .txt files, don't zip them, and put them where
they need to be instead of copy them there in the post gate
hook. The benefit to doing this is that we'll get logs
for tests even if the job timed out.

Change-Id: I4bfd27534c827aed3cbd7b43d7d1289480ea4806
Related-Bug: #1567668
2016-05-04 11:11:46 -04:00
Assaf Muller 8603ea6340 Add fullstack logging
We currently don't log everything being output from the test
runner, only what the processes themselves are logging.

Change-Id: Id5fb9cd44a0ed677a03da1d153ee3079fd5b7975
2016-03-23 11:37:56 -04:00
Ihar Hrachyshka bb567e9b32 Reset RNG seed with current time and pid for each test started
This will hopefully fix fullstack failures where different process
fixtures running in parallel test processes and relying on the same
random.choice() generator seeded by the same initial value could pick up
the same value as a service free port, and spawn their respective
resources using the same port.

Which made one of those unlucky services to fail.

Change-Id: I13cfa9392fd138c5e1b1b7d397b9ea91b2a47ed2
Closes-Bug: #1551288
2016-03-14 17:04:20 +01:00
Jenkins cedbc942c5 Merge "fullstack: use migration scripts to create db schema" 2015-08-21 03:48:33 +00:00
Jakub Libosvar 99fd239ba6 fullstack: use migration scripts to create db schema
Previously, we used create_all() based on models. We don't use
create_all() in production code and there is no guarantee models and
scripts are in sync even though we have a good functional test that
validates that. There are still pieces that can't be compared by

Change-Id: I72fa67811f0763298416e6e084a8b9b86619795b
Closes-Bug: 1486528
2015-08-20 16:06:10 +00:00
John Schwarz 7aa3b2eace Add EnvironmentDescription, pass it down
* The EnvironmentDescription class describes an entire fullstack
  environment (as opposed to the currently implemented host-only
  descriptions). This will allow future patches to signify that a test
  should set up an environment that supports tunneling, l2pop, QoS and
* Now, most fullstack fixtures (config and process ones, at least),
  expect both the EnvironmentDescription for the current test and the
  HostDescription for the 'host' the config/process is on. This allows
  for easier and most robust future changes, as now adding a new
  parameter to one of the description objects doesn't mean adding that
  argument to a number of other objects which are using it.
* Changed HostDescription's default argument of l3_agent to False, since
  adding new configurations and defualting them to True forces the
  author to go through ALL the tests and explicitly turn them on/off.
  However, defaulting new configurations to False only requires
  explicitly turning them on, which we ought to do anyway.

Change-Id: Ib2f12016ba4371bfda76c82e11d0794acc759955
2015-08-20 15:34:25 +03:00
Cedric Brandily 25baeccac9 Define fullstack router/network/subnet management fixture
This change defines the generic fixture ClientFixture for managing
neutron resources:

* router create/cleanup
* network create/cleanup
* subnet create/cleanup
* router interface add/cleanup

And uses it in neutron.test.fullstack.test_l3_agent.

Change-Id: I09fe40d65db60aeee1ff57a7e45c1978a5433517
2015-07-21 23:33:19 +02:00
Cedric Brandily 278a5fce29 Use PyMySQL in MySQL related functional/fullstack tests
mysql-python driver has been replaced by PyMySQL driver[1] in neutron
code but MySQL related functional/fullstack tests try to use
mysql-python driver because of MySQLOpportunisticTestCase[2] and tests
are skipped because mysql-python driver is no more available.

This change provides a backend implementation for mysql+pymysql, a base
base testcase MySQLTestCase[2] using mysql+pymysql implementation
(currently oslo.db provides none of them but will in the future) and
replaces MySQLOpportunisticTestCase with MySQLTestCase.

[1] I73e0fdb6eca70e7d029a40a2f6f17a7c0797a21d
[2] neutron.tests.common.base

Closes-Bug: #1463980
Change-Id: Ic5c1d12ab75443e1cc290a7447eeb4b452b4a9dd
2015-06-15 11:17:47 +02:00
Jeremy Stanley f08e9f1f53 Switch from MySQL-python to PyMySQL
As discussed in the Liberty Design Summit "Moving apps to Python 3"
cross-project workshop, the way forward in the near future is to
switch to the pure-python PyMySQL library as a default.

Change-Id: I73e0fdb6eca70e7d029a40a2f6f17a7c0797a21d
2015-06-09 19:48:20 -04:00
Assaf Muller b0b9e8bcaf Remove reference to non-existent fullstack fixture
EnvironmentFixture doesn't actually exist, and environment
should be a mandatory parameter.

Change-Id: I5598e176f297bcfa3b2cab188b4b446ce18d54ab
2015-06-03 13:53:14 -04:00
John Schwarz 2fcbbf3031 Move full-stack logs post-tests
Currently, it's up to the developer who wants to run full-stack on his
machine to make the directory in question (/opt/stack/logs). However,
this also means that the files don't get compressed at the end of a gate
run. Now, each full-stack test will have each own log directory in /tmp.
Once the logs are there, can run 'gzip' on all the log
files before moving them to /opt/stack/logs on its own.

Change-Id: I5c04d0af0b9858722ae0c4baf0ee478ffb078e02
2015-05-28 12:58:40 +03:00
Jenkins 86669de211 Merge "Add full-stack test" 2015-04-18 13:15:15 +00:00
John Schwarz 304d68d974 Add full-stack test
Currently, the full-stack framework has only one test which only uses
the neutron-server. This patch adds an actual test which makes sure that
once a router is created, an actual namespace is create for it. Since
this test requires 3 processes (neutron-server, l3-agent, ovs-agent),
existing full-stack code is modified to add more streamlined support for
such code.

Partially-Implements: blueprint integration-tests
Change-Id: Id5a8852d38543590b90e4bbed261a7a458071a9a
2015-04-14 15:42:54 +03:00
John Schwarz e409575886 Revert connection option post full-stack tests
The full-stack framework overrides the database connection string before
every test is started, but after the test it doesn't revert the string
back to what it was originally. Since after the test the database is
deleted, the string is not actually valid once the test finished, and
this conflicts with tests which are ran on the same job (specifically
the retargetable tests - see associated bug). The proposed patch saves
the original connection string and reverts it after the test finishes.

Change-Id: I96c01483009084cbc2b81588a1283e84e6bcb4c4
Closes-bug: #1440797
2015-04-09 18:51:03 +03:00
John Schwarz e0ea5edc12 Add full-stack tests framework
This patch introduces the full-stack tests framework, as specified in
the blueprint. In short, this adds the neutron.tests.fullstack module,
which supports test-managed neutron daemons. Currently only
neutron-server is supported and follow-up patches will support for
multiple agents.

Implements: blueprint integration-tests
Co-Authored-By: Maru Newby <>
Change-Id: Iff24fc7cd428488e918c5f06bc7f923095760b07
2015-03-26 20:21:40 +02:00