This patch adds new fullstack test which spawns 2 "hosts" and 2 "VMs" on
those hosts. Both VMs are plugged to the vlan network with some
segmentation id. Next segmentation id of the network is updated and test
ensures that new vlan id is configured in the physical bridge on both
"hosts" and connectivity between VMs still works fine.
Test runs only with Openvswitch agents as Linuxbridge doesn't supports
live update of the segmentation_id in the network.
Change-Id: I459aac7f4e9afe679d8ece1c27d0be49cec8e4ff
Now that we are python3 only, we should move to using the built
in version of mock that supports all of our testing needs and
remove the dependency on the "mock" package.
This completes removal of all references to "import mock",
changing to "from unittest import mock" in fullstack and
functional tests.
Added a hacking check to enforce it in future patches.
Change-Id: Ifcaf1c21bea0ec3c35278e49cecc90a101a82113
In test_dscp_marking_packets, sender vm is sending ICMP packets to the
receiver_vm.
Before this patch even if first ICMP packet wasn't send properly, test
failed due to exception raised in execute method.
It can happend sometimes that first packet will fail to be send as there
can be e.g. some configuration on agent's side not done yet. So we
should try to ping receiver_vm for 60 seconds instead of failing on
first attempt.
This patch changes that behaviour in test.
Change-Id: Ic84e1ef507f017516d47a6b66e646e7e7b5ded8d
Closes-Bug: #1852724
Removed E125 (continuation line does not distinguish itself
from next logical line) from the ignore list and fixed all
the indentation issues. Didn't think it was going to be
close to 100 files when I started.
Change-Id: I0a6f5efec4b7d8d3632dd9dbb43e0ab58af9dff3
All of the externally consumed variables from neutron.common.constants
now live in neutron-lib. This patch removes neutron.common.constants
and switches all uses over to lib.
NeutronLibImpact
Depends-On: https://review.openstack.org/#/c/647836/
Change-Id: I3c2f28ecd18996a1cee1ae3af399166defe9da87
Fullstack test test_dscp_marking_packets spawns tcpdump
process inside one fake vm's namespace and then tries to
ping this instance from second one.
After that it checks if tcdump captured any packet marked with
specific DSCP mark.
Ping was done usually only once (1 packet) because
it was done by using vm.wait_until_ping() method, so it could
happen sometimes that ping was send before tcpdump actually
started captuting traffic.
In such case test failed because there was no any packet captured.
This patch changes this by:
1. Start tcpdump async process with block=True, so it should be
already really started before test will go to the next steps,
2. Send always 10 packets instead of (usually) only one.
In addition this patch adds logging of captured tcpdump's stdout
and stderr streams. It may help debugging issues with this test in
the future.
Change-Id: I23bbde59af0250267843623dde2c5407059d9db2
Closes-Bug: #1818335
async_process.py and ovsdb_monitor.py are now platform
independent, for which reason we can move them to
neutron/agent/common.
Note that a few subprojects are using async_process. We'll use
debtcollector so that we don't break those projects, while logging
a deprecation warning.
Change-Id: I6a7418cb8680cd71fe16c5d98b9b09ef2d260d37
This incorporates flake8 2.6.x and pycodestyle will be used
instead of older pep8. This ensures future python3 compatibility
and a bit better code styling.
Change-Id: Ia7c7c5a44727f615a151e1e68dd94c7ed42f974f
Bug 1733649 is now closed with patch
Ia3522237dc787edb90d162ac4a5535ff5d2a03d5
This commit removes decorator which marked fullstack test
test_qos.TestDscpMarkingQoSOvs.test_dscp_marking_packets
as unstable.
This commit removes also some additional logs of tcpdump's
output added only as temporary "debug" solution.
Change-Id: I0343b1b4fd8d6eed1c26838bf70bce9868e6ee0c
Related-Bug: #1733649
In fullstack test test_dscp_marking_packets two fake hosts are
created, each of them has got one fake instance (called sender and
receiver).
Instance called "sender" sends ICMP packets to instance called
"receiver". In receiver's namespace tcpdump process is spawned with
filter to match only packets marked with specified DSCP value.
After sender instance successfully pings receiver, tcpdump process
is killed and stdout from it was examined to search logged ICMP
packets which were send from sender's IP to receiver's IP address.
That check was failing sometimes as is described in bug report.
It was failing when tcpdump doesn't capture any packets so there was
nothing on stdout from it. But even in such case tcpdump reports on
stderr summary of packets and it was like below:
0 packets captured
6 packets received by filter
0 packets dropped by kernel
which means that packets matching our filter was received by tcpdump but
were probably not processed yet. See [1] for more details.
So this patch changes filter used in tcpdump and way how its output is
checked.
Now in filter expression there are added src and dst IP addresses also so
it will match only packets send from sender instance to receiver instance.
After that tcpdump's stderr output is examined with regex to check if
line like:
X packets received by filter
is there and if X value is different than 0. If so, it means that
tcpdump received packets with wanted DSCP mark and test should pass.
[1] https://unix.stackexchange.com/a/29369
Change-Id: Ia3522237dc787edb90d162ac4a5535ff5d2a03d5
Closes-Bug: #1733649
In fullstack test for dscp marking packets tcpdump is used to capture
packets send from one namespace (Fake VM) to another one.
When tcpdump is closed quickly with SIGKILL signal it may not displays
all packets and that might cause an issue with failed dscp marking test.
Now tcpdump will be kill with SIGINT signal so it should display
all captured packets to the stdout.
Also it should now return to stderr summary of all captured/dropped
packets which can be helpful in future debugging of issue with those
tests if that will not solve the problem.
Change-Id: Ib2d588d081a48f4f2b6e98a943bca95b9955a149
Related-Bug: #1733649
This patch adds additional logging of tcpdump stderr output in
DSCP marking packets tests.
It is required to figure out why those tests are failing sometimes.
Change-Id: I02fe64bf817b47970e4e073f48ea8dea53bdd4f5
Related-Bug: #1733649
This patch adds additional logging of captured tcpdump output during
dscp mark packets tests.
Change-Id: I1ab5727ea70afbe19d2557b9b5c4d77500658cbc
Related-Bug: #1733649
There is no need to translate help message for config option
which is added and used only in tests.
Change-Id: I8b8bdd4131e6183245bcd2556b2107d769fd6427
Add support for QoS ingress bandwidth limiting in
openvswitch agent.
It uses default ovs QoS policies on bandwidth limiting
mechanism.
DocImpact: Ingress bandwidth limit in QoS supported by
Openvswitch agent
Change-Id: I9d94e27db5d574b61061689dc99f12f095625ca0
Partial-Bug: #1560961
It turned out dhcp tests work only because agents are considered dead
after 10 seconds while they report to server every 60 seconds. This led
to calling network resync after agent revival and hiding the fact dhcp
agent is not capable of receiving any amqp messages.
This patch sets the report interval of agents to the half of
agent_down_time on server side and uses eventlet dhcp agent in order to
trigger eventlet monkey patching code.
Eventlet was behind the failure with messages not getting processed. As
[1] notes: "Note: If the “eventlet” executor is used, the threading and
time library need to be monkeypatched."
Because each port calls dhclient to obtain IP address and each dhclient
instance overwrites /etc/resolv.conf there was added a script that
generates fullstack-dhclient-script from an existing dhclient-script
before starting fulltstack tests. This generated script is passed to
each dhclient process running in fake fullstack machine using -sf
parameter.
[1] https://docs.openstack.org/developer/oslo.messaging/server.html
Related-bug: 1453350
Change-Id: I0336176b9c364fe3a95be5cef9e7a3af1ef9d7e9
Multiple SIGHUPs in quick succession might cause the master keepalived
to forfeit its mastership (which will cause keepalived to remove IPs of
its external devices, severing connectivity). This can happen when, for
example, associating or disassociating multiple floatingips.
The patch makes the agent throttle SIGHUP sent to keepalived: the very first
SIGHUP is always sent; as for subsequent signals, they are delayed till
agent threshold is reached. (It's 3 seconds by default.)
As an example, when three consequent router updates trigger keepalived
respawn then:
* the very first signal is sent as usual;
* the second signal is deferred and sent in up to 3 seconds since the
first signal;
* the third signal is ignored, though the change that triggered it will
be correctly applied by the second signal handler when it is triggered
after threshold delay.
If the last time a spawn request occurred is older than current-time
minus threshold then there is no delay.
Co-Authored-By: Jakub Libosvar <libosvar@redhat.com>
Co-Authored-By: Cedric Brandily <zzelle@gmail.com>
Co-Authored-By: Ihar Hrachyshka <ihrachys@redhat.com>
Closes-Bug: 1647432
Change-Id: I2955e0de835458a2eea4dd088addf33b656f8670
This patch adds fullstack tests for DHCP agent HA functionallity.
There are two tests added:
* One is checking if network will be properly
rescheduled to new agent if existing DHCP agent will be down,
* Second is checking if network handled by 2 DHCP agents will have properly
working DHCP service even if one DHCP agent will be down
Change-Id: Iaad373cafd6f83f2c1f8e7ac58dc70070e7aabaf
Linuxbridge agent uses iptable rules in POSTROUTING chain
in the mangle table to mark outgoing packets with the
DSCP mark value configured by the user in QoS policy.
DocImpact: DSCP Marking rule support is extended to the
Linuxbridge L2 agent
Closes-Bug: #1644369
Change-Id: I47e44cb2e67ab73bd5ee0aa4cca47cb3d07e43f3
This reverts commit eb14f66d39.
Turned out that fullstack uses those files to start services, instead of
relying on console scripts generated by pbr. So without those files
having +x permissions, fullstack tests using those agents completely
fail.
Change-Id: I14f98fdd244ecfedfa97fb31bff9763a49e5cfb0
New fullstack test is added to check if packets which
should be sent from port with DSCP mark set are really
sent to another port with this DSCP mark.
This test uses IPv4 only.
Change-Id: I4b26c3c644eb6f2f7813658c99d16fbc3cc61e06
Closes-Bug: #1625570
This patch set modified some lines of code,
which import object, it should import module as
Openstack's recommendation [1].
[1] http://docs.openstack.org/developer/hacking/#imports
Change-Id: I31ecf6a295ffdd089b3333dcec762483e7c081ad
We need to be able to re-use wait_until_true in tempest scenario tests.
There is tempest bug https://bugs.launchpad.net/tempest/+bug/1592345
that prevents us to do so.
Also wait_until_true is not linux specific so it makes more sense to
have it in common package.
Change-Id: Ib8b0e51dbd9edaa58391774d428a737836dfdf77
This patch introduces fullstack testing for the QoS/DSCP
Open vSwitch implementation. It depends on the python-neutronclient
patches, therefore it could not be merged in the main patch.
Co-Authored-By: Miguel Angel Ajo <majopela@redhat.com>
Change-Id: I0ab6a1a0d1430c5791fea1d5b54106c6cc93b937
Closes-Bug: #1468353
Depends-On: I25ad60c1b9a66e568276a772b8c496987d9f8299
Remove IPv4 restriction for local_ip configuration statement.
Check for IP version mismatch of local_ip and remote_ip before creating
tunnel.
Create hash of remote IPv6 address for OVS interface/port name with least
posibility for collissions.
Fix existing tests that fail because of the added check for IP version
and subsequently valid IP addresses in _setup_tunnel_port.
DocImpact
Change-Id: I9ec137ef8c688b678a0c61f07e9a01382acbeb13
Closes-Bug: #1525895
Reactor code:
* Reuse oslo_utils.encodeutils.to_utf8() instead of existing
isinstance(text, six.text_type) test
* Replace jsonutils.dumps(obj).encode('utf-8') with
jsonutils.dump_as_bytes(obj).
* Other minor bytes/Unicode changes
Change-Id: I03b8eff0fd70ab65ac66d6f3221e8ced0a56db17
This patch adds the front end and back end implementation of QoS DSCP.
Associated patches that are dependent on this one:
* python-neutronclient: https://review.openstack.org/#/c/254280
* openstack-manuals: https://review.openstack.org/#/c/273638
* API Guide: https://review.openstack.org/#/c/275253
* Heat:
* Spec: https://review.openstack.org/#/c/272173
* QoSDscpMarkingRule resource: https://review.openstack.org/#/c/277567
* Fullstack tests: https://review.openstack.org/#/c/288392/
APIImpact - The API now supports marking traffic egressing from a VM's
dscp field with a valid dscp value.
Co-Authored-By: Nate Johnston <nate_johnston@cable.comcast.com>
Co-Authored-By: Victor Howard <victor.r.howard@gmail.com>
Co-Authored-By: Margaret Frances <margaret_frances@cable.comcast.com>
Co-Authored-By: James Reeves <james.reeves5546@gmail.com>
Co-Authored-By: John Schwarz <jschwarz@redhat.com>
Needed-By: I25ad60c1b9a66e568276a772b8c496987d9f8299
Needed-By: I881b8f5bc9024c20275bc56062de72a1c70c8321
Needed-By: I48ead4b459183db795337ab729830a1b3c0022da
Needed-By: Ib92b172dce48276b90ec75ee5880ddd69040d7c8
Needed-By: I4eb21495e84feea46880caf3360759263e1e8f95
Needed-By: I0ab6a1a0d1430c5791fea1d5b54106c6cc93b937
Partial-Bug: #1468353
Change-Id: Ic3baefe176df05f049a2e06529c58fd65fe6b419
OVS agent tunnel interfaces are named via:
'%s-%s' % (tunnel_type, destination_ip)
This means that the tunnel interface name is not unique if
two OVS agents on the same machine try to form a tunnel with a
third agent. This happens during full stack tests that start
multiple copies of the OVS agent on the test machine.
Thus, for full stack tests, to make sure that the tunnel
interface names created by ovs agents are globally unique, they
will have the following format :
'%s-%s-%s' % (tunnel_type, hash of source IP, hash of dest IP)
Since this patch centralizes the formation of the tunnel interface
name in a dedicated method that is monkey patched by the full stack
framework, a unit test has been added for this method.
Co-Authored-By: Mathieu Rohon <mathieu.rohon@gmail.com>
Closes-Bug: #1467633
Change-Id: I991af6a5f982746cc297f0248454f803dfbb2daf
It seems that the Queue + QoS + linux-htb implementation was really
limiting ingress by default. So this patch switches the implementation
to the ovs ingress_policing_rate and ingress_policing_burst parameters
of the Interface table.
Later in time we may want to revise this, to make TC & queueing possible,
but this is good enough for egress limiting.
Also, removed the _update_bandwidth_limit del+set on OvS QoS driver for
the bandwidth limit rule update, since that's not needed anymore.
Change-Id: Ie802a235ae19bf679ba638563ac7377337448f2a
Partially-Implements: ml2-qos
Creates a port in a policy, and subsequently modifies the
bandwidth limit rule in the policy, then verifies that the
new limits are assigned to the port.
Change-Id: I23fe45ef08618ad91567feb1707028e0a0bfe0d6
Partially-Implements: ml2-qos
It seems like agents' package content was removed by commit
01a7ba19cf but package itself was left in
the tree.
Change-Id: I651f8010aa7c4af59ce403b099db7bc064364133
Creates classes representing the 3 types of namespaces handled by the L3 agent:
router, snat and fip.
The scope of this change is:
- Creation and deletion methods are provided for each namespace class
- Creation and deletion of router and snat namespaces are moved to the router
classes. These namespaces are now members of the corresponding router class
- Invocation of Fip namespace creation and deletion is left in the agent, since
the agent owns it
- A context manager is provided to move the namespaces (router and snat)
cleanup code out of the agent
A follow up patchset will add methods to create and delete interfaces in the
namespaces. These methods are intended to be used by the router classes
Change-Id: I54b14e593ded6b2990d57a3ae9d598a699ae133e
Partially-Implements: bp restructure-l3-agent
The idea behind the AdvancedServices (Metadata, *aaS) services
to be singletons was to (1) offer a way to get an instance of such
a service in a convinient manner, and to (2) ensure that only one
service registered per service type.
(1) Since the AdvancedService.instance method required an instance
of a L3 agent, this point was kind of missed. If you have a L3 agent
instance in your hand, just use it to access the service you're looking
for.
(2) This is now fulfilled by asserting that only one driver is registered
(per type) in the .add method.
The motivation to make them non-singletons is that:
a) They don't need to be
b) The code is simplified this way
c) I've been facing crazy issues during functional testing where we're
constantly instantiating L3 agents, and the (singleton) metadata
service was referencing the wrong configuration object. The service
was re-initialized during a test, screwing up the configuration of
other mid-run tests.
Closes-Bug: #1425759
Depends-On: I155aece9b7028f1bdd3d9709facef392b38aec27
Depends-On: I199f6a967c64e921624b397fa5b7dfc56572d63a
Change-Id: I192e4d12380346c53c8ae0246371ccc41a836c4c
L3 agent drivers are singletons. They're created once, and hold
self.l3_agent. During testing, the agent is tossed away and
re-built, but the drivers singletons are pointing at the old
agent, and its old configuration.
Change-Id: Ie8a15318e71ea47cccad3b788751d914d51cbf18
Closes-Bug: #1404662
The file l3_agent.py has become too large. This patch is a simple
pure refactor to move some of the functionality in to other files
where things aren't too tangled up. There is no functional change
with this patch and I avoided gratuitous other fixups in this patch in
order to make it easier to review.
I plan to follow up on the new l3_dvr and l3_agent_router modules with
more restructuring in the near future.
Partially-Implements: bp restructure-l3-agent
Change-Id: I3529fe4146c50c940f41eb26d0b5efc5870b3af9
HA failover testing requires the ability for two L3 agents to
co-exist on the same machine. The following changes were made
to support this:
* Add l3 agent ability to simulate nested namespaces by suffixing
each router namespace with '@host'. For example, if a router
with id '1' is created, its namespace will be:
'qrouter-1@host'
* Added test.common submodule which will hold common code which is
shared between the different kind of tests (unit and funtional).
* Added tests.common.agents submodule which holds test-only agents, and
added a test L3NAT agent to be used in the HA functional testings.
Co-Authored-By: Assaf Muller <amuller@redhat.com>
Closes-bug: #1374946
Change-Id: I2185fe276d9d32b34c407396e7ed5d6e2801340c
Slawek Kaplonski