Commit Graph

90 Commits (63bac0fb435d66ab3d1e1775435a5b6ab6752c6e)

Author SHA1 Message Date
Rodolfo Alonso Hernandez 4b6c2246c0 Force "network_id" in "subnet" DB registers
The "subnet" OVO does not allow to have an empty (None) "network_id" but
the DB "subnet" table allows to have this parameter empty (NULL) in the
database. In order to avoid any problem like the one described in the
bug, this patch ensures the database "subnet" register does always have
a "network_id" value and if a "network" register is being deleted, all
related "subnet" registers are checked first.

Change-Id: Iad210f0585b4201fdb87187b44a9b42267b58db4
Closes-Bug: #1839658
2019-08-19 16:32:42 +00:00
Hongbin Lu c8b04827a9 Revert "Update network revision only when it needs"
Closes-Bug: #1787028
This reverts commit e9a7ed8c63.

Change-Id: If52eb8f7c6a801f22a67d2a5786f16ddaa7c5f0f
2018-08-14 21:06:21 +00:00 e9a7ed8c63 Update network revision only when it needs
Basically, in case of concurrent requests creating ports
on *one* network, only one request succeeds, other requests
mostly getting 'standardattribute' update error as network
revision_number changed about 8 times for all port updates.

This patch increases network revision_number in database,
only when a port is created in a network, instead of each
update on port.

Change-Id: Idffb4edda616677b9b071644d3835c85052091a5
Closes-Bug: #1782421
2018-08-12 16:17:50 +00:00
Ihar Hrachyshka f21c7e2851 Allow to set/modify network mtu
This patch adds ``net-mtu-writable`` API extension that allows to write
to network ``mtu`` attribute.

The patch also adds support for the extension to ml2, as well as covers
the feature with unit and tempest tests. Agent side implementation of
the feature is moved into a separate patch to ease review.

DocImpact: neutron controller now supports ``net-mtu-writable`` API
APIImpact: new ``net-mtu-writable`` API extension was added.

Related-Bug: #1671634
Change-Id: Ib232796562edd8fa69ec06b0cc5cb752c1467add
2017-08-11 00:57:34 +00:00
Kevin Benton 32814bb39e Bump network rev on RBAC change
Increment the revision number when RBAC policies are
changed since it impacts the calculation of the 'shared'

Closes-Bug: #1708079
Change-Id: I4c7eeff8745eff3761d54ef6d3665cf3dc6e6222
2017-08-02 16:44:01 +00:00
Hirofumi Ichihara 96f0142b80 Tag mechanism supports resources with standard attribute
Tag mechanism supports network, subnet, port, subnetpool
router resources only. This patch allow tag mechanism to support
 resources with standard attribute.

Two old extenions are kept because of backward compatibility.
They will be removed in Queens release.

APIImpact: Tag is supported by resources with standard attribute
DocImpact: allow users to set tags on resources with standard attribute

Change-Id: Id7bb13b5beb58c313eea94ca03835d3daf5c94bc
Closes-Bug: #1682775
2017-07-25 08:14:04 +09:00
Boden R 62576cabf7 use core resource attribute constants from neutron-lib
neutron-lib now contains the API definitions for neutron's core
resources. This patch removes the constant core resource and collection
variables and uses them from lib. Subsequent patches will consume the
actual core resource attribute definitions.


Change-Id: Ia9afdf620cd538b2aa420593277d6403a45c996b
2017-06-30 06:25:36 -06:00
Kevin Benton 6dee4c5942 Cleanup _find_related_obj
All of the additional lookup logic was essentially duplicating
a relationship property of 'load_on_pending=True', which tells
SQLAlchemy to lookup the relationship during object creation [1].
So we can dump all of this logic and just use that option.


Related: blueprint push-notifications
Change-Id: I0e495a50f5cab9b6449825039d7683d77de1e763
2017-02-25 06:29:33 +00:00
Jenkins 5a79873632 Merge "Avoid loading network and all rels for subnet query" 2017-02-21 07:09:16 +00:00
Jenkins 6f944bab1e Merge "Remove ORM relationship between ports and networks" 2017-02-20 19:00:59 +00:00
Kevin Benton bc306a5512 Avoid loading network and all rels for subnet query
The Subnet model has a relationship to Network that is only
used for bumping the revision of the network. However, loading
the network loads all of its relationships as well so we end
up wasting a bunch of time loading segments, DHCP agents, etc all
for no reason. During a subnet update, we get the subnet from the
DB several times (policy engine, before update, after update,
and dhcp agent scheduler) so the slowness in get_subnet is amplified.

The recent switch to subqueries[1] has made loading all of this
unrelated data much more expensive since each relationship requires
a query, which was likely what brought this problem to the surface.

This adjusts the relationship to just load the standard attr object
of the network directly so we can still bump the revision of the network
without having to load the network itself.

This resulted in a ~20%-30% reduction of time spent in single update-subnet
operations on my dev environment (1sec to 650-850ms).

1. I6952c48236153a8e2f2f155375b70573ddc2cf0f

Closes-Bug: #1665967
Change-Id: Ie5f956ddf0704a5dde6e4f801a446f48fea5f697
2017-02-20 01:22:33 -08:00
Kevin Benton 7dfa64bf41 Remove ORM relationship between ports and networks
This relationship doesn't serve a purpose and it results
in SQLAlchemy trying to set the network_id column on ports
to NULL when attempting to delete a network with ports in it.

This results in the following exception:

IntegrityError: (1048, u"Column 'network_id' cannot be null")

This exception is not treated as a DBReferenceError by olso.db
which makes detecting races between ports being created on a
network being deleted very noisy in the logs since oslo.db treats
this as a generic DBError.

By removing the relationship, deleting a network will now just
result in a boring standard foreign key constraint failure which
will propagate from oslo.db as a DBReferenceError that is easy
to catch.

The patch that depends on this one adjusts the delete network logic
to capture these ReferenceErrors.

Change-Id: I4166a9bbaded57a77576400929a150123b8f1b2e
2017-02-06 01:20:53 -08:00
Kevin Benton 3ffe006743 Switch to 'subquery' for 1-M relationships
This switches to the use of subqueries for 1-m relationships
which will result in a higher constant query factor but will
eliminate the potential for cross-product explosions.

Closes-Bug: #1649317
Change-Id: I6952c48236153a8e2f2f155375b70573ddc2cf0f
2017-02-06 09:01:31 +00:00
Henry Gessau 1c5bf09a03 Use DB field sizes instead of _MAX_LEN constants
Remove The following _MAX_LEN constants from
neutron/api/v2/ and use the corresponding DB field size
constants from neutron_lib.db.constants instead.

 NAME_MAX_LEN              -->  NAME_FIELD_SIZE

In alembic migration scripts, the raw numerical value is used.

For more information, see:


Change-Id: I734890372584fe27e5d6ec38c0cad2de882ff11c
2016-12-23 13:48:57 +00:00
Kevin Benton a802b382d3 Use subqueries for rbac_entries and subnets<->network
Loading subnets as part of the networks list and networks
as part of the subnets list appears to have a significant
impact when the network has tags and the subnets have
extra routes entries. This is even further compounded by
the network having rbac entries (likely due to the subnet
inheriting the RBAC entries of the network with the custom
join condition in the model).

This patch converts rbac_entries on both subnet and network
to use a subquery and converts the network and subnets
relationships on the subnet and network models (respectively)
to use subqueries as well.

On my dev environment after running the script in the report,
a network list took 5 minutes. Converting just the rbac_entries
or just the network/subnet relationship to subqueries reduced it
to 3-5 seconds. Converting both (as this patch does), reduces it
back down to a couple of hundred milliseconds (normal perf of my
development env with the current network count).

Subqueries will just cost us a constant number of queries and
won't scale up with result count so this should not impact scalability
in any way.

None of these fields are queryable from the API, so we don't need
to worry about breaking queries against the models.

Partial-Bug: #1649317
Change-Id: Ic1947e3d78d58a79b21344b10cb7ab0e573e419f
2016-12-19 18:33:43 +00:00
Henry Gessau 55bda46c47 Remove model_base deprecations
The model_base mixins were deprecated from neutron a long time ago.


Change-Id: I4b9a069d4e36472a74daa38053a64109c3985473
2016-12-13 14:26:29 +00:00
Kevin Benton bd1c454c4f Deterministic ordering of fixed_ips
This adds an order_by clause to the fixed_ips relationship
on the port object to ensure that the fixed_ip ordering is
consistent between a create, an update, and a get request
for a port. Without it we were at the mercy of the sql backend
to determine how it felt like ordering them on the join condition.

Closes-Bug: #1630410
Change-Id: I523e0ab6e376f5ff6205b1cc1748aa6d546919cb
2016-10-07 00:40:01 -07:00
Ihar Hrachyshka dcd78423aa Introduce ovo objects for ports
Those objects are intentionally not integrated into the database code so
far. This is to quicken access to their definitions to implement
push-notifications for security groups and ports.

The object embeds segmentation information in addition to what's
available through the model. Specifically, binding_levels field exposes
all ml2 binding levels, that from their side load corresponding network
segment object. The order for level objects in binding_levels list field
is guaranteed to be in the order of level. So the consumers can eg.
access the bottom binding info with:


For PortBindingLevel object, we want to expose segmentation info. This
is achieved through a 'segment' ObjectField. The database model itself
contains segment_id too. There is no reason though to expose it for
Level object in two places (one as a model field, another one through
the ObjectField), so we avoid adding ID field. The base class that
handles loading for ObjectField based synthetic fields was assuming that
objects always have a field per model attribute, so it needed a slight
adjustment to support this case, where we extract foreign_keys
attributes from the model itself if the field is not present on the

Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Partially-Implements: blueprint push-notifications

Change-Id: I25de14e42e345d9235dbf4097c298ef5d606de51
Co-Authored-By: Martin Hickey <>
Co-Authored-By: Rossella Sblendido <>
Co-Authored-By: Manjeet Singh Bhatia <>
Co-Authored-By: Brandon Logan <>
Co-Authored-By: Victor Morales <>
2016-09-28 20:46:19 +00:00
Jenkins 4c4a55ce6b Merge "Add common way to extend standard attribute models" 2016-09-16 16:35:06 +00:00
Kevin Benton 465d22180e Add common way to extend standard attribute models
This adds a way for standard attribute models to declare
the API resources they show up in. It then adds a utility
function to the standard_attr module to grab a map of all
API resources and their corresponding models.

This can be used by any processing code that wants to add
fields to standard attribute resources.

This also adjusts the existing extensions to leverage this
new functionality.

Partially-Implements: blueprint add-neutron-extension-resource-timestamp
Change-Id: Idc8923d0e983fcb0690f8cb5b55a5aff8690154f
2016-09-15 01:42:24 +00:00
Jenkins 2c12add84d Merge "Remove availability range code and model" 2016-09-14 02:14:46 +00:00
Carl Baldwin b6a90df2ac Mark unaddressed ports with none in ip_allocation field
This is the Neutron side of the unaddressed ports blueprint. In order
to allow unaddressed ports, Nova wants the port to explicitly say it
is okay that it doesn't have any IP addresses.

In Neutron, an unaddressed port is one that was created by explicitly
passing [] in fixed_ips to create the port.  A new DB field is added
to the port to distinguish the unaddressed port case from the deferred
IP allocation case where routed networks is involved.

Change-Id: Ia61af4c14e955697a7d3fcc0bf4826a6d9475c98
Implements: blueprint vm-without-l3-address
APIImpact: port now has ip_allocation attribute, set on port create
2016-09-09 12:43:35 -06:00
Carl Baldwin 774792681d Remove availability range code and model
These models are effectively obsolete [1] and should've been removed
in a previous patch [2] but some of it was left behind.


Change-Id: Ib381c24f37e787b4912e28d98ec77473c0448c2b
Related-Bug: #1543094
Closes-Bug: #1620746
2016-09-06 17:35:12 +00:00
Jenkins b293bc2e44 Merge "Switch to neutron-lib for model_base" 2016-09-01 07:47:30 +00:00
Jenkins 9c4d048d87 Merge "Using revision_number to ensure no overlap in *one* network" 2016-09-01 07:38:27 +00:00
Henry Gessau 61cc14fd67 Switch to neutron-lib for model_base
Change-Id: If5b2b4cc0346515ddef3da1255ab49327c8e5732
2016-08-31 11:12:18 -04:00
Kevin Benton bdd7298284 Move standard attr out of model_base
The model_base file is going to move to Neutron lib in
I2087c6f5f66154cdaa4d8fa3d86f5e493f1d24d9. This will mainly leave
behind only the standard attributes related stuff so the name
'model_base' won't make much sense. This moves the standard attribute
related things into its own module so model_base can conceivably be
eliminated entirely.

Change-Id: Icaf3220fbc5723f2b5421a494371ef274d7073c7
2016-08-27 17:46:38 -04:00
Gary Kotton 9f09f27c5d Fix deprecation warnings
Remove deprecation warnings for various constants
and exceptions that have moved to neutron_lib.

Fix miscellaneous other deprecations.

Uses constants instead of l3_constants when importing
neutron-lib constants.

Co-Authored By: Henry Gessau <>
Co-Authored By: Gary Kotton <>

Change-Id: Ib0e8ff5c3e23677c1009241a1818cbc8a3430c38
2016-08-26 22:16:06 -04:00
Nam Nguyen Hoai 5264ab966d Using revision_number to ensure no overlap in *one* network
This patch uses revision_number in database. When creating
a subnet in a network, the revision_number of the network
will be increased. That will prevent overlapping CIDR
(overlapping CIDR means some subnets' cidrs are overlapping)
on *one* network.

Basically, in case of concurrent requests creating subnets
on *one* network, only one request successes, other requests
needs retrying request.

Change-Id: Id6548535075bed87a4b36e1462db546ab9163f29
Closes-Bug: #1532695
2016-08-24 14:27:48 +00:00
Ester Niclos Ferreras 460cb650d8 Fix neutron_lib.constants DeprecationWarning from db.models_v2
This patch addresses all the DeprecationWarning errors related
to constants in neutron.db.models_v2.

Change-Id: I54402d4441b3ccb15bab63adf830fa196bba62c3
2016-08-12 15:10:44 +02:00
Kevin Benton 4e8cc68349 Increment revision numbers on object changes
This adds the logic to increment the revision numbers
for objects whenever there are changes and it exposes
the revision number via a field in the API.

This is handled with a new default service plugin that
subscribes to DB events and bumps revision numbers for
any objects that were modified.

It also handles the logic for bumping the revision number
of a parent in a relationship where the children aren't
top-level neutron objects that would be tracked individually.
This is accomplished with a 'revises_on_change' attribute
on the child models that the service plugin will use to
find the parent and bump its revision.

API tests are included to test the revision numbers
added to each standard attribute enabled object.

Partially-Implements: bp/push-notifications
Change-Id: I476d3e03c8ee763cc4be6d679fe9f501eb3a19b5
2016-07-27 12:01:55 -07:00
Bin Yu 64f5fc8259 Refactor DNS integration out of DB core plugin
This patch set aims to move all the code related to DNS integration
from the DB core plugin to the DNS ML2 extension module.

By doing this, this patchset removes the dns related code in
db_base_plugin_v2 and the dns exteions module talks with core plugin
only through the method extension_manager and apply_dict_extend_functions

By properly implementing the generation of the dns_assignment attribute
for ports in the DNS ML2 extension module, this patchset also fixes

Change-Id: I63afb1a1bfeeb14eefb54681dc64959144deeb25
Closes-Bug: #1579601
Closes-Bug: #1579977
2016-07-26 07:33:50 +00:00
Ihar Hrachyshka a984f9554c Calculate MTU on every network fetch instead of on create
Today, existing networks may not reflect MTU configured for
neutron-server, if they were created when neutron-server was using
different MTU setup for its infrastructure, or when it was using bad
default values for network MTUs (specifically, before Mitaka, all networks
were getting MTU = 0 by default, disabling both advertisement and data
path MTU size enforcement).

This patch stops persisting MTU in the database on network create and
instead calculate it on every network resource fetch.

DocImpact Now changes to MTU configuration options immediately affect
          existing network MTUs, not just new networks.

UpgradeImpact Existing networks with invalid MTU persisted in database
              may change their MTU values to reflect configuration.

Change-Id: Iee4f5037bf10b73ba98464143b183aacb59c22f2
Closes-Bug: #1556182
2016-07-20 13:26:50 +02:00
Jakub Libosvar 766abb752a Make pep8 job great again
There is a bug in pep8, when 'select' used, it omits all default checks
and runs only those specified by 'select'.  We got hit by this issue
since I2d26534230ffe5d01aa0aab6ec902f81cfba774d was merged which lead to
almost no static checks in pep8 job.

Also note that off_by_default decorator has no effect for now because
factory in hacking is triggered after ignored checks are collected.
There will be a follow-up patch for that in order to make pep8 doing
its job quickly.


Related-Bug: 1594756
Change-Id: I8e27f40908e1bb4307cc7c893169a9d99f3433c4
2016-06-21 16:23:51 +00:00
tonytan4ever 63fc967a8f Refactor NetworkDhcpAgentBinding
This patch set is for breaking the circular dependency between
Agent/NetworkDhcpAgentBinding. The goal of this is to implement
Agent OVO

Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Change-Id: I3f2a8bcc6f8644e94e377dc916fba6743cb230bc
2016-06-15 12:19:03 -04:00
Carl Baldwin f494de47fc Associate subnets to segments through subnet API
Change-Id: Ia1084a94ac659332c126eb9d4787b04a89a4ba90
DocImpact: Need to add segment_id to API docs
Partially-Implements: blueprint routed-networks
2016-05-10 15:56:48 -06:00
Ilya Chukhnakov 3fb07b6629 Add device_id index to Port
Some 'Port' queries use 'device_id' column for lookup.
Such queries could be observed in database query log (at least) during
instance launch. In the absence of 'device_id' index that leads to full
table scan. That causes unnecessary database load and impacts query
response time.

Change-Id: If42b7d3265e216d393d3ab8c172b97637af908cc
Closes-Bug: #1574750
2016-04-25 16:28:54 -07:00
Jenkins a2b3344361 Merge "Add IPAllocation object to session info to stop GC" 2016-03-19 14:22:11 +00:00
Jenkins f161318cf7 Merge "Add uselist=True to subnet rbac_entries relationship" 2016-03-18 03:47:18 +00:00
Kevin Benton 691f8f5ea5 Add uselist=True to subnet rbac_entries relationship
Because the join conditions for Subnet rbac entries
are manually specified, SQLAlchemy is not
automatically detecting that this relationship is a list.
This adds the uselist=True kwarg to the relationship to
ensure that it's always handled as a list.

Change-Id: Ia4ae57ddd932260691584ae74c0305a79b2e60a9
Closes-Bug: #1557959
2016-03-16 01:35:26 -07:00
Cedric Brandily 79bc54a2d8 Fix add_is_default_to_subnetpool migration
add_is_default_to_subnetpool migration[1] is not working with a
non-empty subnetpool table because it adds a new non-nullable column
(is_default) to subnetpool table without a default value.

This change adds a server_default value to is_default column in
subnetpool table and updates associated db migration.

[1] neutron/db/migration/alembic_migrations/versions/mitaka/expand/\

Closes-Bug: #1557757
Change-Id: Ib0b2ac061d21a7d908f14b399b0bba06d94b7241
2016-03-15 22:55:32 +01:00
Kevin Benton 7d9169967f Add IPAllocation object to session info to stop GC
This adds the IPAllocation object created in the _store_ip_allocation
method to the session info dictionary to prevent it from being
immediately garbage collected. This is necessary because otherwise a
new persistent object will be created when the fixed_ips relationship
is referenced during the rest of the port create/update opertions.
This persistent object will then interfere with a retry operation
that uses the same session if it tries to create a conflicting record.

By preventing the object from being garbage collected, the reference
to fixed IPs will re-use the newly created sqlalchemy object instead
which will properly be cleaned up on a rollback.

This also removes the 'passive_delete' option from the fixed_ips
relationship on ports because IPAllocation objects would now be
left in the session after port deletes. At first glance, this might
look like a performance penalty because fixed_ips would be looked
up before port deletes; however, we already do that in the IPAM
code as well as the ML2 code so this relationship is already being
loaded on the delete_port operation.

Closes-Bug: #1556178
Change-Id: Ieee1343bb90cf111c55e00b9cabc27943b46c350
2016-03-14 01:23:20 -07:00
Kevin Benton 5dacbba701 Add a description field to all standard resources
In order to give users and operators more flexibility in
annotating the purpose of various Neutron resources, this patch
adds a description field limited to 255 chars to all of the
Neutron resources that reference the standard attribute table.
The resource that reference the table are the following:
security_group_rules, security_groups, ports, subnets,
networks, routers, floatingips, subnetpools

This patch adds a description table related to standard attributes
and migrates over the existing security group description to the new
table as well.

Co-Authored-By: James Dempsey <>

DocImpact: Adds a description field to all resources outline in
           commit message.
Closes-Bug: #1483480
Change-Id: I6e1ef53d7aae7d04a5485810cc1db0a8eb125953
2016-03-05 02:29:35 +00:00
Hong Hui Xiao 635581912f Correlate address scope with network
With address scope being enabled, networks now are in one ipv4
address scope and one ipv6 address scope.

This patch adds derived attributes to the network as part of the
address scopes extension that will show related address scopes
when viewing a network through the API.


Change-Id: Ib1657636033ad2c0009d50ebe7c5ae4f72f6b175
Closes-Bug: #1547380
2016-02-22 22:46:18 -05:00
Kevin Benton 89f96e627f Ensure agent binding modules are loaded
This patch ensures that models_v2 and the router db module
import the agent binding modules so their models are defined
in the sqlalchemy metadata. Otherwise, using an agentless
core/L3 plugin may result in a failure to initialize if nothing
imported the agent module.

Change-Id: I77240462b618d6e95c7c5862441d6d179e371f4f
Closes-Bug: #1529022
2016-01-10 14:09:38 +00:00
Kevin Benton 57964df5c6 Use a joined relationship for AZ info on networks
The previous code was doing a DB lookup for each network's
availability zone which was significantly impacting the
performance of network listings.

This patch adjusts the network model to be automatically joined
to the DHCP agents table that the AZ code uses to populate the
AZs for the network.

Change-Id: I908ceb1a68e0eed7c304e3ff82279ad6fa406167
Closes-Bug: #1525740
2015-12-14 14:45:30 -08:00
Manjeet Singh Bhatia ae40bf1fd9 Reuse constants defined in attributes
This change reuses constants defined in attributes module in db

Change-Id: Idbb5f21a485a41a814b6b8c5744348d25443239e
2015-12-04 05:14:55 +00:00
Hirofumi Ichihara 6e50027819 Add availability_zone support for network
This patch adds the availability_zone support for network.


Change-Id: I9259d9679c74d3b3658771290e920a7896631e62
Co-Authored-By: IWAMOTO Toshihiro <>
Partially-implements: blueprint add-availability-zone
2015-11-25 22:34:09 +09:00
Kevin Benton 3c8570f3fc Make Neutron resources reference standard attr table
This adds a new 'standardattributes' table and adds a foreign-key
references from ports, subnets, networks, subnetpools, routers,
securitygroups, and floatingips to this table.

This will make it easy to add new things to the schema like
timestamp fields or anything else that applies to multiple types
of Neutron resources. The new fields would just be added to the
'neutronresources' table instead of being duplicated across each
resource's table. Or, if the the relationship is 1-to-many (e.g. tags),
the new association table would be related to the 'standardattribute'

Related-Bug: #1496802
Change-Id: Iaa3ba81a7e9cae09cea153720b29879d8cc9a080
2015-11-23 07:41:26 -08:00
John Davidge 6ee91e56c8 Replace subnetpool config options with admin-only API
This patch adds a new boolean 'is_default' property to subnetpools. This
allows the admin to set the default v4/v6 subnetpools via the API rather
than the existing neutron.conf options - which are deprecated by this patch.

Only one subnetpool per IP family can be set to default.


Co-Authored-By: Carl Baldwin <>

Change-Id: I5daba2347cfb91fac0b155b2c1b459ee7d9e4505
Closes-Bug: 1501328
2015-11-06 17:16:31 +00:00