Running with a stricter .pylintrc generates a lot of
C0330 warnings (hanging/continued indentation). Fix
the ones in neutron/db.
Trivialfix
Change-Id: I9311cfe5efc51552008072d84aa238e5d0c9de60
The goal of this patch is to make the Neutron code compliant
with SQLAlchemy 2.0.
All SQL transactions must be executed inside an explicit
writer/reader context. SQLAlchemy no longer will create an
implicit transaction if the session has no active transaction.
A warning message, only available in debug mode, is added. When
an ORM session calls "do_orm_execute", if there is no active
transaction, a warning message with a traceback will be logged
to help to debug the regression introduced.
Related-Bug: #1964575
Change-Id: I3da37fee205b8d67d10673075b9130147d9eab5f
This patch switches over to callback payloads for PORT
AFTER_DELETE events.
Some shims were removed.
Change-Id: If69e37b84fe1b027777b1d673b3d08a6651a979e
This patch switches the code over to the payload style of callbacks [1]
for PORT AFTER_CREATE events. In addition it adds a branch/shim to the
dhcp_rpc_agent_api to support both payload and kwarg style callbacks.
NeutronLibImpact
[1]
https://docs.openstack.org/neutron-lib/latest/contributor/callbacks.html
Change-Id: I25d43d4f8f2390b07e0d11c631f894d88669bbe0
Patch [1] introduced new, read only field "normalized_cidr" in
the security groups' rules.
With this patch this new field is send as source or
destination_ip_prefix field in the rule's dict in case if
remote_ip_prefix and normalized_cidr are set in the rule.
[1] https://review.opendev.org/c/openstack/neutron/+/749158
Related-Bug: #1869129
Change-Id: Icebe0255ce11eae859c05a273335b027da3f1545
Support security group rules with remote_address_group_id in openvswitch
firewall. This change reuses most of the firewall functions handling remote
security groups to also process remote address groups. The conjunctive flows
for a rule with remote_adress_group_id are similar to others with
remote_group_id but have different conj_ids.
Change-Id: I8c69e62ba56b0d3204e9c12df3133126071b92f7
Implements: blueprint address-groups-in-sg-rules
When add allowed-address-pair 0.0.0.0/0 to one port, it will
unexpectedly open all others' protocol under same security
group. IPv6 has the same problem.
The root cause is the openflow rules calculation of the
security group, it will unexpectedly allow all IP(4&6)
traffic to get through.
For openvswitch openflow firewall, this patch adds a source
mac address match for the allowed-address-pair which has
prefix lenght 0, that means all ethernet packets from this
mac will be accepted. It exactly will meet the request of
accepting any IP address from the configured VM.
Test result shows that the remote security group and
allowed address pair works:
1. Port has 0.0.0.0/0 allowed-address-pair clould send any
IP (src) packet out.
2. Port has x.x.x.x/y allowed-address-pair could be accepted
for those VMs under same security group.
3. Ports under same network can reach each other (remote
security group).
4. Protocol port number could be accessed only when there
has related rule.
Closes-bug: #1867119
Change-Id: I2e3aa7c400d7bb17cc117b65faaa160b41013dde
The retry_if_session_inactive decorator was rehomed into neutron-lib
[1]. This patch consumes it by removing the function from neutron and
using neutron-libs version where appropriate.
NeutronLibImpact
[1] https://review.openstack.org/#/c/557040/
Change-Id: I3e3289f33e62d45933d0fbf165bb4b25078f22d5
notify_sg_on_port_change will cause unnecessary notification to
neutron-l2-agents when port update, for example like port's name
updating will also notify.
notify_sg_on_port_change will only be needed when port's
Security Group update or port IP change.
Change-Id: I5439adf2c4b7dcf832241201fd949f6930e65fdf
Closes-Bug: #1720322
Neutron lib contains the latest callbacks and thus this patch removes
the callbacks package from neutron entirely.
NeutronLibImpact
Change-Id: I14e45fd5d2d3c816bb39f8ace56f7be460bac0d6
In Pike, the agent side of security_groups_provider_updated()
RPC code was changed to a NOOP when the provider rules were
changed to be static, https://review.openstack.org/#/c/432506
Now that we're in Queens we can deprecate it.
Change-Id: Ie018ff653633d3524f0e80c5e172a5d01bdad437
Break apart the SG RPC API into sg info generation logic,
notification logic, and logic to retrieve the rules from
the database.
This allows a follow-up patch to re-use the same security
group info generation logic while retrieving the security
group info from a push notification cache on the agent rather
than a DB on the server.
Partially-Implements: blueprint push-notifications
Change-Id: Ib4359e4016b5a72f70f3fd4f9134887eef59a0bd
This is a simple code move within a class to prepare
for a class separation in the following patches.
TrivialFix
Partially-Implements: blueprint push-notifications
Change-Id: Ibe40973568f1c9def2ded2f3a750e3423f4bd416
This is a simple code move within a class to prepare
for a class separation in the following patches.
TrivialFix
Partially-Implements: blueprint push-notifications
Change-Id: I843b316b01ca67cfe4636b01f960e0ae751b37a6
This moves the retry decorators onto the DB querying
functions in preparation for a refactor that allows
the data structure generating functions to be backed
by the push notification cache on the agents.
Partially-Implements: blueprint push-notifications
Change-Id: Id43783a244e4bc121b7efc378b37d1f70ffd33b3
In [1] the setup of security group rules for DHCP was refactored.
However it assumed that DHCPv6 responses would go to the target IP,
instead the destination will be the link-local address of the instance,
so we will need to add this to the list of allowed destinations.
[1] Ibfbf011284cbde396f74db9d982993f994082731
Change-Id: I6bd24a98d7c72a1bac28999eb3c9a831ab2ab9bf
Closes-Bug: 1693771
This moves the notify_security_groups_member_updated calls
outside of ML2 into a callback in the security groups RPC
base.
This will allow us to ultimately stop overriding the
create/delete/update port inheritance in ML2.
This results in a small behavior change internally for bulk
port creation operations where the notifier will be called
once for each port rather than once for all ports. We can
revisit this in the future if it turns out to be problematic
but right now the bulk port creation API is rarely anyway
since it's mostly a hidden feature (lack of client support).
Partial-Bug: #1666424
Change-Id: I344a82d5a1a8bf1e2b3f6a3ce7725682bb03b007
Setting up rules to allow DHCPv6, DHCP, and RAs from specific
IP addresses based on Neutron resources has a few issues:
1. It violates separation of concerns. We are implementing logic to
calculate where an IPv6 RA advertisement or DHCP advertisement
should be coming from in the security group code. This code should
not be trying to guess IPv6 LLAs, know about subnet modes, DHCP server
implementations, or the type of L3 plugin being used. Currently all
of these assumptions are baked into code that should only be
filtering, which makes it very rigid and brittle when it comes to
other implementations for DHCP and/or RAs.
2. It has scaling issues on large networks. Every time one of these
provider rules is updated, it triggers every L2 agent to refresh
all of the security group rules for ports in that network, which puts
significant load on the server.
3. It's main purpose: preventing spoofing of RA[1,2] and DHCP packets,
has long been superceded by preventing VMs from acting as DHCP/RA
servers[3][4].
This patch completely removes all of this logic and just returns
static provider rules to the agents that allow all DHCP server
and RA traffic ingress to the client. This addresses the issues
highlighted above since the code is significantly simplified and
the provider rules don't require refreshes on the agents.
Now that the provider rules never change, the RPC notification
listener on the agent-side for 'notify_provider_updated' is now
just a NOOP that doesn't trigger any refreshes. The notification
was left in place on the server side for older version agents
that have stale IP-specific provider rules. The entire notification
can be removed in the future.
The one open concern with this approach is that VMs will now be
able to receive DHCP offers from other DHCP servers on the same
network that aren't being filtered (e.g. a VM with port security
disabled or another device on a provider network). In order to
address this for DHCP, this patch adds two rules that only allow
DHCP offers targeted to either the broadcast or the correct client
IP. This prevents incorrect offers from ever reaching the client.
For RAs, this patch just allows all RAs so we may pick up
advertisements from other v6 routers attached to a network;
however, the instance won't actually be allowed to use bad addresses.
1. https://bugs.launchpad.net/neutron/+bug/1262759
2. I1d5c7aaa8e4cf057204eb746c0faab2c70409a94
3. Ice1c9dd349864da28806c5053e38ef86f43b7771
4. https://git.openstack.org/cgit/openstack/neutron/tree/
neutron/agent/linux/iptables_firewall.py
?h=521b1074f17574a5234843bce68f3810995e0e1d#n475
Closes-Bug: #1653830
Closes-Bug: #1663077
Change-Id: Ibfbf011284cbde396f74db9d982993f994082731
Oslo.utils provides same method get_ipv6_addr_by_EUI64, let's use
it and deprecate method get_ipv6_addr_by_EUI64 in neutron.
Closes-Bug: #1597213
Change-Id: If1eda1283b4405e13ccb60dab530ce4cc1b22948
This patch moves the module into the canonical place. It does it in
backwards compatible way, by leaving the model available under the old
location, only emitting a deprecation warning on access from there. In
the future, the old module will be completely cleaned up.
This patch also makes head.py file that is used to import and register
all models in the tree to import everything from under neutron.db.models
without explicitly naming each module. In that way, we can avoid the
need to import each new module with models as long as it's located in
the designated place. The file is a well known abuser in terms of git
conflicts, and it's better to avoid manual work.
Change-Id: I084031e6b3c8d9113cc6ee6e3967f0e1edaee0b5
Partial-Bug: #1597913
This patch will separate security group db models from mixins for OVO
implementation work.
Change-Id: Ie38d3cc2efa5f580261f207748affb5840ba23c9
Partial-Bug: #1597913
Replace uses of IPV6_LLA_PREFIX from neutron.common.constants with
IPv6_LLA_PREFIX from neutron-lib. Remove IPV6_LLA_PREFIX constant as it
is no longer used in the code base:
http://codesearch.openstack.org/?q=IPV6_LLA_PREFIX
TrivialFix
Change-Id: Ic81f63670cd51181b349c5c26cc49b2b654860f2
This patch is dependent on commit I8d03528f8f45f5f50fa467b39245a513a37c5d89.
It integrates the VersionedObject with the existing code.
Integration revealed that using IPAddress is not correct for allowed
address pairs, because the address can also represent a subnet.
Another issue revealed by the integration is that we must retain the
original string format passed by users through API for MAC addresses.
Neither we can use IPNetworkField from oslo.versionedobjects for
ip_address field because it will then always append prefix length to
base network address, even if prefix length is maximum for the type of
IP network (meaning, the address actually represents a single host),
which is contradictory to how API currently behaves (returning mask-less
addresses for /32 - for ipv4 - and /128 - for ipv6 - prefix lengths).
To solve those issues, 'authentic' flavors for netaddr.EUI and
netaddr.IPNetwork types are introduced. Those 'authentic' flavors
attempt to retain the original string representation, as passed by the
caller.
Since base IPNetworkField recreates network object on coerce(), and
hence looses information about the original string representation, we
introduce our custom flavor of the field type that reuses the network
object passed by the caller.
The change for the type of ip_address field triggers hash change.
Anyway, we are safe to change it without considering backwards
compatibility, because the object is not used anywhere yet.
Co-Authored-By: Ihar Hrachyshka <ihrachys@redhat.com>
Change-Id: I3c937267ce789ed510373616713b3fa9517c18ac
Partial-Bug: #1541928
Python 3 deprecated the logger.warn method, see:
https://docs.python.org/3/library/logging.html#logging.warning
so we prefer to use warning to avoid DeprecationWarning.
Closes-Bugs: #1529913
Change-Id: Icc01ce5fbd10880440cf75a2e0833394783464a0
Co-Authored-By: Gary Kotton <gkotton@vmware.com>
Add these additional protocols listed below to
security groups brings convenience to operators
on configuring these protocols. In addition, make
the security group rules more readable.
The added protocols are: ah, dccp, egp, esp, gre,
ipv6-encap, ipv6-frag, ipv6-nonxt, ipv6-opts,
ipv6-route, ospf, pgm, rsvp, sctp, udplite, vrrp.
A related patch is submitted to neutron-lib project:
https://review.openstack.org/259037
DocImpact: You can specify protocol names rather than
protocol number in API and CLI commands. I'll update
the documentation when it is merged.
APIImpact
Change-Id: Iaef9b650449b4d9d362a59305c45e0aa3831507c
Closes-Bug: #1475717
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
Security group provider rules for RA is set for the VM ports
when a router interface is added or updated after the VM
instance is created.
In the case of DVR Routers the security group provider rule
to allow the RA packets to flow through the VM port input
chain was missing and so the VM was not able to get a
SLAAC/DHCP address when associated with a DVR Router.
This fix will add the security group rule to the VM port input
chain to allow the RA packets to flow into the VM and hence
the VM will obtain an IP address assigned by the Router.
Closes-Bug: #1501969
Change-Id: Ib0b3499d9c880fe1462734b2d4092debf4819509
when it deletes a security group all rules, it should
include this sg information in RPC method
'security_group_info_for_devices', otherwise the ports
in this sg can't corrcectly update their iptables and
ipset sets.
Change-Id: Ibb071ce84590bd46cda2c1e010a566e75e22b4d2
Closes-bug: #1460562
For TCP/UDP protocol, port valid range is 0 to 65535, so for a
security group rule, its valid range is also 0 to 65535. this
patch makes two changes:
1. if a security group rule port_range_min is 0, l2 agent also can
get port_range_min real value 0 when it gets this rule for a
device via RPC.
2. For IptablesFirewallDriver, if port range is [0, xxxx], l2 agent
also need add this rule to iptables.
Change-Id: If93c54a31d973187889ead2c2797ffdd40a4393d
Closes-bug: #1473965
Previously this function lived in SecurityGroupServerRpcMixin. This
patch moves this function to SecurityGroupDbMixin so plugins who don't
use SecurityGroupServerRpcMixin can leverage this function. This patch
has no affect to anyone using SecurityGroupServerRpcMixin as that inherits
from SecurityGroupDbMixin.
Change-Id: I6ce60dad222e1e244f0d4fac9680e73de2a9b2e9
The rename from Quantum to Neutron left a few q_ strings
around, let's go ahead and clean them up.
Change-Id: I06e6bdbd0c2f3a25bb90b5fa291009b9ec2d471d
Some housekeeping was done in
- SecurityGroupDbMixin:
- create_rule_bulk() calls to create_rule();
- registry notification is in create_rule();
- separate validation for a single rule and for a group of rules
- SecurityGroupServerRpcMixin:
- overriden methods call to corresponding super class methods;
Hopefully code is now self-documented enough
Closes-Bug: #1461024
Change-Id: Ia75d7e206716bbe74aae89e4cebd0c2c40af68a8
With ml2 plugin, when a port's IP or security group changes, it
should send 'security_groups_member_updated' message to other l2
agents which have same security group with this changed port.
Change-Id: I2e7622d2db4c173ac879a95a6e0adf92b858fe82
Closes-bug: #1448022
This patch changes the get_port_from_device_mac() and
get_sg_ids_grouped_by_port() methods in ML2 db.py module so that
they do not create a new database session (via get_session()), but
instead reuse the session associated with the caller's context.
In order to make the session that is associated with the caller's
context available to these ML2 DB methods, the
get_ports_from_devices plugin API in securitygroups_rps_base.py
needs to be modified so that the context can be passed down to the
ML2 plugin. (A similar change is made to the get_port_from_device
plugin API for consistency.)
Change-Id: I3f990895887e156de929bd7ac3732df114dd4a4b
Closes-Bug: 1441205
When a security_groups_provider_updated is received then a global
refresh of the firewall is performed. This can be avoided if the
plugins pass as parameter of the call the devices that belongs to
the network updated.
Partially-Implements: blueprint restructure-l2-agent
Change-Id: I1e78f3a5ec7e5c5bcba338a0097566422411ef7e
It's mostly a matter of changing imports to a new location.
Non-obvious changes needed:
* pass overwrite= argument to oslo_context since oslo.log reads context
from its thread local store and not local.store from incubator
* don't store context at local.store now that there is no code that
would consume it
* LOG.deprecated() -> versionutils.report_deprecated_feature()
* dropped LOG.audit check from hacking rule since now the method does
not exist
* WritableLogger is now located in oslo_log.loggers
Dropped log module from the tree. Also dropped local module that is now
of no use (and obsolete, as per oslo team).
Added versionutils back to openstack-common.conf since now we use the
module directly from neutron code and not just as a dependency of some
other oslo-incubator module.
Note: tempest tests are expected to be broken now, so instead of fixing
all the oslo.log related issues for the subtree in this patch, I only
added TODOs with directions for later fix.
Closes-Bug: #1425013
Change-Id: I310e059a815377579de6bb2aa204de168e72571e
Use set operations instead of using list.
Currently complexity of the method is O(n^2) where n is
amount of ips (amount of VMs in the network).
When amount of VM is big (large L2 domain), this method
can significantly load the controller.
Reduce method complexity to O(n) on average.
Change-Id: If1660e8227e5c5cd80d49ebcc6a2e06d33d31939
Closes-Bug: #1429753
This change allows combining multiple RPC calls for
security group member/provider updates that are needed
as a result of create bulk port operations.
Change-Id: Ibfe84f40e55a9847cbcaa4a7344988b2d3fb895e
Closes-Bug: #1373634
With ironic servers, a NIC can fail and be replaced with one that has a
different mac. The corresponding neutron port needs to be updated with
the new mac address so the NIC can be guaranteed to retain the same IP
address.
This change enables this feature in the ml2 plugin. There may need to
be changes to other plugins to disable or complete the implementation of
this feature.
Closes-Bug: #1341268
Partially-implements: blueprint allow-mac-to-be-updated
APIImpact: Ports
DocImpact: minor port update api change
Change-Id: I1864c0882cda7eddc9ced519ed3f96c91b2b63f3
Brian Haley