If the update_port call failed with error IpAddressAlreadyAllocatedClient,
retry a few more times in order to find IP addresses that are available.
In some cases it may happen that port is "binding_failed"
because L2 agent running on destination host was down but
this is "temporary" issue.
It is like that for example in case when using L3 HA and when
master and backup network nodes were e.g. rebooted.
L3 agent might start running before L2 agent on host in such case
and if it's new master node, router ports will have "binding_failed"
When agent sends heartbeat and is getting back to live,
ML2 plugin will try to bind all ports with "binding_failed"
from this host.
_assert_ping_during_agents_restart is used in tests of L2 and L3
agents. However, when it raises an exception due to a timeout, the
associated message assumes the agent under test is L2. This patch
During agents restart there is async ping run and there is
called function to wait until all async ping workers will
finish their job.
there are 60 pings sent with 1 second timeout so default
wait_until_true timeout which is set to 60 seconds might not
be enough in some cases.
Because of that wait_until_true timeout is now set as
twice higher value than is needed to number of packets to send
This should give enough time to finish all workers.
In case of HA routers IPv6 forwarding is not disabled by default and
then enabled only on master node.
Before this patch it was done in opposite way, so forwarding was
enabled by default and then disabled on backup nodes.
When forwarding was enabled/disabled for qg- port, MLDv2 packets are
sent and that might lead to temportary packets loss as packets to
FIP were sent to this backup node instead of master one.
Change network namespace add/delete/list code to use
pyroute2 library instead of calling /sbin/ip.
Also changed all in-tree callers to use the new calls.
We plan to switch to devstack-gate for fullstack job, and it revokes
direct sudo calls before executing tests, so we can't rely on sudo
This also moves functional-testing.filters to a more generic filename
(testing.filters) because the filters are now deployed and used by
fullstack target too.
It turned out dhcp tests work only because agents are considered dead
after 10 seconds while they report to server every 60 seconds. This led
to calling network resync after agent revival and hiding the fact dhcp
agent is not capable of receiving any amqp messages.
This patch sets the report interval of agents to the half of
agent_down_time on server side and uses eventlet dhcp agent in order to
trigger eventlet monkey patching code.
Eventlet was behind the failure with messages not getting processed. As
 notes: "Note: If the “eventlet” executor is used, the threading and
time library need to be monkeypatched."
Because each port calls dhclient to obtain IP address and each dhclient
instance overwrites /etc/resolv.conf there was added a script that
generates fullstack-dhclient-script from an existing dhclient-script
before starting fulltstack tests. This generated script is passed to
each dhclient process running in fake fullstack machine using -sf
The oslo.db opportunistic test fixtures were not being
used effectively and the MySQL / PG databases were not
being used. This patch restores working patterns against
oslo.db. Additionally, the migration level tests have also
been updated to make use of oslo.db provisioning functionality
and unused methods have been removed.
The current approach makes use of oslo.db provisioning constructs
directly, as well as some semi-private attribute access within
oslo.db enginefacade, in order to work around some issues
that have arisen in oslo.db's test_base.
A new release of oslo.db will provide
public API points to resolve pending issues, and to
allow neutron's use cases here which will
also be of general applicability to openstack projects.
Log files as .txt files, don't zip them, and put them where
they need to be instead of copy them there in the post gate
hook. The benefit to doing this is that we'll get logs
for tests even if the job timed out.
This will hopefully fix fullstack failures where different process
fixtures running in parallel test processes and relying on the same
random.choice() generator seeded by the same initial value could pick up
the same value as a service free port, and spawn their respective
resources using the same port.
Which made one of those unlucky services to fail.
Previously, we used create_all() based on models. We don't use
create_all() in production code and there is no guarantee models and
scripts are in sync even though we have a good functional test that
validates that. There are still pieces that can't be compared by
* The EnvironmentDescription class describes an entire fullstack
environment (as opposed to the currently implemented host-only
descriptions). This will allow future patches to signify that a test
should set up an environment that supports tunneling, l2pop, QoS and
* Now, most fullstack fixtures (config and process ones, at least),
expect both the EnvironmentDescription for the current test and the
HostDescription for the 'host' the config/process is on. This allows
for easier and most robust future changes, as now adding a new
parameter to one of the description objects doesn't mean adding that
argument to a number of other objects which are using it.
* Changed HostDescription's default argument of l3_agent to False, since
adding new configurations and defualting them to True forces the
author to go through ALL the tests and explicitly turn them on/off.
However, defaulting new configurations to False only requires
explicitly turning them on, which we ought to do anyway.
mysql-python driver has been replaced by PyMySQL driver in neutron
code but MySQL related functional/fullstack tests try to use
mysql-python driver because of MySQLOpportunisticTestCase and tests
are skipped because mysql-python driver is no more available.
This change provides a backend implementation for mysql+pymysql, a base
base testcase MySQLTestCase using mysql+pymysql implementation
(currently oslo.db provides none of them but will in the future) and
replaces MySQLOpportunisticTestCase with MySQLTestCase.
Currently, it's up to the developer who wants to run full-stack on his
machine to make the directory in question (/opt/stack/logs). However,
this also means that the files don't get compressed at the end of a gate
run. Now, each full-stack test will have each own log directory in /tmp.
Once the logs are there, post_test_hook.sh can run 'gzip' on all the log
files before moving them to /opt/stack/logs on its own.
Currently, the full-stack framework has only one test which only uses
the neutron-server. This patch adds an actual test which makes sure that
once a router is created, an actual namespace is create for it. Since
this test requires 3 processes (neutron-server, l3-agent, ovs-agent),
existing full-stack code is modified to add more streamlined support for
Partially-Implements: blueprint integration-tests
The full-stack framework overrides the database connection string before
every test is started, but after the test it doesn't revert the string
back to what it was originally. Since after the test the database is
deleted, the string is not actually valid once the test finished, and
this conflicts with tests which are ran on the same job (specifically
the retargetable tests - see associated bug). The proposed patch saves
the original connection string and reverts it after the test finishes.
This patch introduces the full-stack tests framework, as specified in
the blueprint. In short, this adds the neutron.tests.fullstack module,
which supports test-managed neutron daemons. Currently only
neutron-server is supported and follow-up patches will support for
Implements: blueprint integration-tests
Co-Authored-By: Maru Newby <firstname.lastname@example.org>