Commit Graph

43 Commits (76c5d02d3de5175d54ca2e96e25e0536773bb10c)

Author SHA1 Message Date
Zuul ca0c2746a8 Merge "Dont use dict.get() to know certain key is in dict" 5 years ago
Yuki Nishiwaki b803195a99 Dont use dict.get() to know certain key is in dict
In CommonAgentLoop class, there is logic to detect tap device is changed
locally or not by comparing timestamp with previous.
Sometimes timestamp value could be None depending on the timing (see bug/1781129)

But current _get_devices_locally_modified logic can not detect local
change from None to something because _get_devices_locally_modified
function don't always compare if previous timestamp value was None.

In order not to miss updated device always, better not to use dict.get() to
know previous iteration have timestamp or not.

Change-Id: Ib0361ad5c281f88558e8e048cfec588b9f9b1de4
Closes-Bug: #1781129
5 years ago
Miguel Lavalle f7064f2b6c Add binding activation to the Linuxbridge agent
As part of the implementation of multiple port bindings [1], add binding
activation support to the linux bridge agent. This will enable the
execution with linux bridge agents of the complete sequence of steps
outlined in [1] during an instance migration:

1) Create inactive port bindings for destination host
2) Migrate the instance to the destination host and plug its VIFs
3) Activate the port bindings in the destination host
4) Delete the port bindings for the source host


Change-Id: I2c937cc0a551e5ce0e8534c4dd4384ec2ca92da1
Partial-Bug: #1580880
5 years ago
Brian Haley c3b83a9ca6 Fix all pep8 E265 errors
Fixed all pep8 E265 errors and changed tox.ini to no longer
ignore them.  Also removed an N536 comment missed from a
previous change.

Change-Id: Ie6db8406c3b884c95b2a54a7598ea83476b8dba1
5 years ago
Brian Haley 90cd939047 Fix W503 pep8 warnings
Fix W503 (line break before binary operator) pep8 warnings
and no longer ignore new failures.


Change-Id: I7539f3b7187f2ad40681781f74b6e05a01bac474
5 years ago
Boden R ef93f7e7f0 use common agent topics from neutron-lib
The neutron.common.topics module was rehomed into neutron-lib with
commit Ie88b84949cbd55a4e7ad06341aab77b286cdc485
This patch consumes it by removing the rehomed module from neutron
and using the module from neutron-lib instead.


Change-Id: Ia4a4604c259ce862597de80c6deeb3d408bf0e95
5 years ago
Sławek Kapłoński cbee0f9f88 Use same instance of iptables_manager in L2 agent and extensions
This commit adds common_agent_extension class which is agent API
for L2 extension drivers used e.g. by Linuxbridge agent.
This is necessary to be able to use instance of iptables_manager
used in firewall driver also in L2 extension drivers (like qos).

This patch refactors little bit iptables_manager code to make possible
to initialize e.g. mangle or nat table on demand, even if iptables
is created as "state_less"

Change-Id: I3b66e49b7f176124e8aea3eb96d0d465f1ab1ea0
Closes-Bug: #1736674
6 years ago
Boden R 0822b0aef4 consume common constants from lib
The neutron-lib commit I360545b6ee4291547e0c5c8e668ad03d3efa4725 moved
the externally consumed globals from neutron.common.constants into lib.
With the exception of PROVISIONAL_IPV6_PD_PREFIX all other constants
in neutron.common.constants should only be used in neutron, and will
hopefully remain that way. External consumers needing access to other
common constants should move them into lib first.


Change-Id: Ie4bcffccf626a6e1de84af01f3487feb825f8b65
6 years ago
Boden R 40866acd02 use new payload objects for *_INIT callbacks
Part of the work we did while rehoming the callback modules introduced
the notion of payload objects [1] to replace the unstructured kwargs
used today. When using payloads event sources need to use publish()
rather than notify() to trigger the callback(s) and pass along a payload
object (if needed).

This patch begins to move us onto the payload objects by updating
BEFORE_INIT and AFTER_INIT event types to use the payloads.

NB: This change needs to be in sync with consumers using the events
herein. Once publish() is used with payloads, callback functions must
also define the payload kwarg to follow suit. Therefore such consumers
need to depend on this patch.



Change-Id: I9194c7857f10392149159071cda8e080e93adc10
6 years ago
Brian Haley 40ef314590 Remove security_groups_provider_updated rpc code
In Pike, the agent side of security_groups_provider_updated()
RPC code was changed to a NOOP when the provider rules were
changed to be static,
Now that we're in Queens we can deprecate it.

Change-Id: Ie018ff653633d3524f0e80c5e172a5d01bdad437
6 years ago
Inessa Vasilevskaya 7322bd6efb Make code follow log translation guideline
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.

Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
6 years ago
Kevin Benton 01a97d926c Remove deprecated prevent_arp_spoofing option
This was deprecated over a year ago in [1] so let's
get rid of it to clean up some code.

1. Ib63ba8ae7050465a0786ea3d50c65f413f4ebe38

Change-Id: I6039fb7e743c5d9a1a313e3c174ada36c9874c70
6 years ago
Kevin Benton 5aa7192ffa Common Agent loop: Catch delete_port extension failures
Catch delete port extension failures like we do with normal port
removal to avoid interrupting the deletion processing for other

Related-Bug: #1697937
Change-Id: I42d86a86965d30e14c85ce975681a6b82be41ecc
6 years ago
Boden R 0e2b667bf1 use neutron-lib callbacks
The callback modules have been available in neutron-lib since commit [1]
and are ready for consumption.

As the callback registry is implemented with a singleton manager
instance, sync complications can arise ensuring all consumers switch to
lib's implementation at the same time. Therefore this consumption has
been broken down:
1) Shim neutron's callbacks using lib's callback system and remove
existing neutron internals related to callbacks (devref, UTs, etc.).
2) Switch all neutron's callback imports over to neutron-lib's.
3) Have all sub-projects using callbacks move their imports over to use
neutron-lib's callbacks implementation.
4) Remove the callback shims in neutron-lib once sub-projects are moved
over to lib's callbacks.
5) Follow-on patches moving our existing uses of callbacks to the new
event payload model provided by

This patch implements #2 from above, moving all neutron's callback
imports to use neutron-lib's callbacks.

There are also a few places in the UT code that still patch callbacks,
we can address those in step #4 which may need [2].


[1] fea8bb64ba7ff52632c2bd3e3298eaedf623ee4f
[2] I9966c90e3f90552b41ed84a68b19f3e540426432

Change-Id: I8dae56f0f5c009bdf3e8ebfa1b360756216ab886
6 years ago
Kevin Benton df320474c5 Set MTU on tap devices in Linux Bridge agent
Libvirt does not set the MTU of the tap device it creates when creating
a bridge interface. It also does not set the MTU of the bridge itself.
This cannot be fixed on the Nova side since libvirt doesn't appear to
have support for setting MTUs on bridges until version 3x.

This results in a situation where the first VM tap interface attached to
a bridge will always have an MTU of 1500. The Neutron agent will then
add in VLAN/VXLAN interfaces with the correct MTU; however, the bridge
MTU will still be pinned to the smallest interface MTU attached to it.
This breaks jumbo frames until all small MTU tap devices are removed
from the bridge.

This patch explicitly sets the MTU on tap devices to match the network
MTU when processing the device.

We will have to carry this workaround until we stop Nova from
plugging taps into bridges[1] or until we drop support for older
libvirts on the Nova side and have it set the MTU.

This bug was introduced by change
I53c0eb57da956b36f09731d25db989719e9bc9dc which reverted automatic
setting of tap MTUs to match those of the physical device.

1. I23c5faaeab69aede1fd038a36f4a0b8f928498ce
Closes-Bug: #1684038
Change-Id: Ia245a3e22339fce026901e24a82e836c8b27cc28
6 years ago
John Perkins 7f23ccce23 Agent common config
Refactoring Neutron configuration options for agent common config to be
in neutron/conf/agent/common. This will allow centralization of all
configuration options and provide an easy way to import.

Partial-Bug: #1563069
Change-Id: Iebac0cdd3bcfd0135349128921b7ad7a1a939ab8
Needed-By: Ib676003bbe909b5a9013a3178b12dbe291d936af
6 years ago
Armando Migliaccio ca751a1486 Spin off context module

Partially-implements: blueprint neutron-lib

Change-Id: I48cf45dc1b07035d952152eac2548a3bd9fc2832
6 years ago
Dariusz Smigiel 9733825896 Cleanup of SecurityGroup classes
Commit Ifb70a118cef48c3c4cd313e22e907aa47bc51ad0 intended to remove some
classes, but without DeprecationWarnings. To prevent from possible
problems, if it would be now removed, added Warnings and included info,
about future removal.

Change-Id: Iacb93abc363bf638efc3acb5b29c02a7508bc43a
Related-Bug: #1444112
7 years ago
Jenkins b7fdd64cda Merge "Implementation of vlan-aware-vms for Linux Bridge" 7 years ago
Kevin Benton 2fe2efc55d LinuxBridge: Pass host into get_devices_details_list
Pass the host into get_devices_details_list on the linux bridge
agent so the debug logs on the server side don't show "host None".
This is mainly just for cosmetics and consistency with the OVS agent
since the only thing the host is really used for on the server side
is special treatment of DVR ports, which does not currently apply
to linux bridge.

Change-Id: I700fa26982bdb087cf7ea4b3eb69aec2f2e099c8
Closes-Bug: #1622566
7 years ago
Kevin Benton 2e882a9496 Implementation of vlan-aware-vms for Linux Bridge
This is the agent-side implementation of vlan-aware-vms for
the Linux Bridge agent. It implements the feature using
vlan subinterfaces.

Whenever subports are required, the linux bridge trunk driver
will create vlan devices off of the parent port device following
the same naming scheme as normal ports. This allows the normal
agent loop to see these VLAN ports and wire them like any other
port so the trunk logic doesn't have to concern itself with things
like firewall rules, anti-spoofing, or encapsulation onto the
physical network.

How to try:

* enable the Linux Bridge mech driver and install the Linux Bridge agent
* enable the 'trunk' service plugin
* make a port, turn it into a trunk, attach it to a VM (or boot a VM with it)
* add subports and configure your guest on the corresponding VLAN
* don't forget these subports have security groups so add allow rules!

Partially-implements: blueprint vlan-aware-vms
Change-Id: I688d5b25885c1c3938185467b15502ccf65cf935
7 years ago
Jenkins 14ebfb527d Merge "Refactoring config options for plugin agent opts" 7 years ago
Armando Migliaccio e354599134 Add agent-side driver scaffolding for trunk functionality
The agent code is enhanced to allow the trunk agent-side counterpart
to be activated seamlessly by means or local registry notifications.
Some integration with the server side is provided by loading the
RPC agent-side skeleton. Basic unit testing provides some coverage.

More effective functional and system coverage will be provided once
everything comes together.

Partially-implements: blueprint vlan-aware-vms

Co-Authored-By: Adolfo Duarte <>
Change-Id: Id70553e8980593f99548a4d2b0a78355933f7c2c
7 years ago
Anindita Das 17f2df320c Refactoring config options for plugin agent opts
Refactoring neutron ml2 plugin agent driver config opts to be in
neutron/conf/plugins/ml2/drivers/agent so that all the
configuration options for agent drivers reside in a centralized
location. This simplifies the process of looking up the agent driver
config opts and provides an easy way to import.

Change-Id: I673c10dfa35014fea129f7030139a89bfc20b272
Partial-Bug: #1563069
7 years ago
Kevin Benton c17c78df7c Port device events for common agent
Some of the agent code may want to receive events whenever
a device is added/updated/deleted. Currently the only way
to do that is to modify the loop directly or create an
extension. Neither of these options are good if it's code
we don't want users to disable (the extensions) and it's
specific to one agent (so we don't want to modify the loop).

Partially-Implements: blueprint vlan-aware-vms
Change-Id: I3468c7f46cc1b4000cdd537e8f216d207a14727a
7 years ago
Kevin Benton e7e8a4e938 Don't interrupt device loop for missing device
If a device is removed in the middle of
treat_devices_added_updated it can cause failures which
will force all of the other unaffected devices to be
reprocessed. When they are reprocessed it will cause
the ports on the server side to go back into BUILD which
can cause failures in tests expecting ACTIVE ports.

This patch adds an exception catcher that examines if the
port is still present. If it's missing, it suppresses the
exception since the device will be treated as a removed
device on the next iteration.

Closes-Bug: #1605090
Change-Id: Ia774a7c1578f5aca71f3b706e47819b3fdc1cce2
7 years ago
Nate Johnston 01a6c9c426 Generalize agent extension mechanism
This change generalizes agent extension code so that all agents can take
advantage of a common mechanism.

Co-Authored-By: Margaret Frances <>

Partially-Implements: blueprint l3-agent-extensions

Change-Id: I9380343c09d28eec67077c9e6d77c33a195e516b
7 years ago
zhufl 80c3a7aba0 Remove unused LOG
This is to remove unused LOG to keep code clean.

Change-Id: I5623fb0b22464c39a28c6e8f28878e6db4504e24
7 years ago
Dina Belova 9a43f58f4d Add OSprofiler support
* Add osprofiler wsgi middleware. This middleware is used for 2 things:
  1) It checks that person who wants to trace is trusted and knows
     secret HMAC key.
  2) It starts tracing in case of proper trace headers
     and adds first wsgi trace point, with info about HTTP request

* Add initialization of osprofiler at start of service
  Currently that includes oslo.messaging notifer instance creation
  to send Ceilometer backend notifications.

Neutron client change: Ic11796889075b2a0e589b70398fc4d4ed6f3ef7c

Co-authored-by: Ryan Moats <>
Depends-On: I5102eb46a7a377eca31375a0d64951ba1fdd035d
Closes-Bug: #1335640
DocImpact Add devref and operator documentation on how to use this
Change-Id: I7fa2ad57dc5763ce72cba6945ebcadef2188e8bd
7 years ago
Henry Gessau 4148a347b3 Use constants from neutron-lib
With this we enable the deprecation warnings by default.

Related-Blueprint: neutron-lib

Change-Id: I5b9e53751dd164010e5bbeb15f534ac0fe2a5105
7 years ago
Akihiro Motoki 2131340d74 release note to deprecate prevent_arp_spoofing option
This option was originally intended to deprecate in Mitaka
and remove in Newton, but we missed to announce it in Mitaka
release note. It looks better to deprecate it in Newton release
and remove it in Ocata release.

Change-Id: Iad466abbb0716da77801315d1a8766ba2f7c06b9
Closes-Bug: #1546010
7 years ago
Oleg Bondarev e532ee3fcc Notify resource_versions from agents only when needed
resource_versions were included into agent state reports recently to
support rolling upgrades (commit 97a272a892)
The downside is that it brought additional processing when handling state
reports on server side: update of local resources versions cache and
more seriously rpc casts to all other servers to do the same.
All this led to a visible performance degradation at scale with hundreds
of agents constantly sending reports. Under load (rally test) agents
may start "blinking" which makes cluster very unstable.

In fact there is no need to send and update resource_versions in each state
report. I see two cases when it should be done:
 1) agent was restarted (after it was upgraded);
 2) agent revived - which means that server was not receiving or being able
    to process state reports for some time (agent_down_time). During that
    time agent might be upgraded and restarted.

So this patch makes agents include resource_versions info only on startup.
After agent revival server itself will update version_manager with
resource_versions taken from agent DB record - this is to avoid
version_manager being outdated.

Closes-Bug: #1567497
Change-Id: I47a9869801f4e8f8af2a656749166b6fb49bcd3b
7 years ago
Akihiro Motoki 44be13a2a6 Hacking rule to check i18n usage
* Detect neutron.i18n import (neutron._i18n is recommended)
* Check builtins _ usage
* 'builtins = _' in tox.ini is no longer required.
* Introduce hacking rule doctest framework.
  Newly added check_builtins_gettext() hacking check takes
  token as argument. It is not a good idea to pass a tokenized
  line manually. Instead it is reasonable to use docstring based
  tests used in hacking repo.

Change-Id: Ib7464658fc4c8a6f1b03af6ab46f0bd3ee0bfb18
7 years ago
Kevin Benton 7afbd3a6b8 Check tap bridge timestamps to detect local changes
During a quick instance rebuild on Nova, it may remove a tap
interface and then plug it in within the agent polling window.
In this scenario the agent will not realize a device has changed
and will therefore not ask the server for info an subsequently
update its status. This will prevent the notification from being
sent back to Nova that the vif plugging has finished so the
VM will never resume state.

This adds a new timestamp collection method to the common agent
manager interface for devices that is used by the common agent
loop to determine if a device has locally changed.

The linux bridge implementation of it checks the timestamps on the
tap interface's 'bridge' directory, which will change whenever
the tap is added to bridge.

Closes-Bug: #1531862
Change-Id: If172470e907848556b6a8aff13520f94245919bb
7 years ago
Jenkins 4ffab69af9 Merge "Removed static reference to LinuxBridge in logging" 7 years ago
Jenkins b380b15d4c Merge "RPC Callback rolling upgrades reporting, and integration" 7 years ago
Sławek Kapłoński 45410d76be Add support for QoS for LinuxBridge agent
There is a new QoS extension driver for the linuxbridge agent
being added. This driver provides support for QoS configuring
on the linuxbridge agent.
This patch introduces two new config options for Linuxbridge agent:
kernel_hz - HZ value of host kernel,
tbf_latency - value of latency in tbf qdisc to calculate size of queue.

Co-Authored-By: vikram.choudhary <>

Change-Id: I457ca2569b5d4a916ba09e71040505cd0ad3257b
Closes-Bug: #1500012
Closes-Bug: #1550514
DocImpact Update agent configuration to show settings related to
          QoS and bandwidth limiting
7 years ago
Miguel Angel Ajo 97a272a892 RPC Callback rolling upgrades reporting, and integration
This is the second patch to allow upgrades on RPC versioned
objects callbacks.

This enables resource version notifications from agents to all
neutron servers via fanout for updating the version sets in
memory, and via agent status updates for DB storage, so any
neutron server can retrieve such information at boot.

Closes-Bug: #1535247

Change-Id: I67c1323267aaf7e49f4a359ff50b94e52dba4380
7 years ago
gh159m 1ed0acd269 Removed static reference to LinuxBridge in logging
Removed hard-coded "LinuxBridge" from logging message in the common
agent since it could also be used by other agents and changed it to
log the agent type instead.  This should allow logs to reflect
which agent type is actually using the common agent.

Removed unused/leftover constants from the common agent.

Change-Id: I2e790bf052f18f7314bfd1677fde65b96b2a5edd
Closes-Bug: #1551219
7 years ago
Jenkins 727ea09d06 Merge "Deprecate ARP spoofing protection option" 7 years ago
Kevin Benton 7bbacd49a4 Deprecate ARP spoofing protection option
This protection should always be enabled unless its explicitly
shutoff via the port security extension via the API. The primary
reason it was a config option was because it was merged at the end
of Kilo development so it wasn't considered stable. Now that it
has been enabled by default for all of Liberty and the development
of Mitaka, it's a good idea to just get rid of the option completely.

DocImpact: Remove references to prevent_arp_spoofing and replace
           with pointer to port security extension for disabling
           security features.
Change-Id: Ib63ba8ae7050465a0786ea3d50c65f413f4ebe38
7 years ago
Andreas Scheuring 7d153a671b Moving Common Agent into separate module
Moving the CommonAgent and all it's unittests into a speparate module.

Closes-Bug: #1468803

Change-Id: Ifccc6ee1a77eef3928ad326cd5857092aeef4a17
7 years ago
Andreas Scheuring 6e29cdd6b6 lb: ml2-agt: Separate AgentLoop from LinuxBridge specific impl
The goal is to extract the common agent code from the linuxbridge agent
to share this code with other agents (e.g. sriov and new macvtap [1]).
This is a first step into the direction of a so called modular l2

Therefore all linuxbridge implementation specifics are moved into the
LinuxBridgeManager class. The manager class will be passed as argument
into the common agent loop instead of instantiating it in its
constructor. In addition the network_maps and the updated_devices map
has been moved into the rpc class.

A clear manager interface has been defined for the communication
between the common agent loop and the impl specific manager class.

In a follow up patchset, the common agent loop will be moved into a
new file. This has not yet happened to simplify tracking the code
changes during review.


Change-Id: Ia71f5a403b7029f8cc591f83df91ab2d3916f3f8
Partial-Bug: #1468803
Partial-Bug: #1480979
7 years ago