The "subnet" OVO does not allow to have an empty (None) "network_id" but
the DB "subnet" table allows to have this parameter empty (NULL) in the
database. In order to avoid any problem like the one described in the
bug, this patch ensures the database "subnet" register does always have
a "network_id" value and if a "network" register is being deleted, all
related "subnet" registers are checked first.
Basically, in case of concurrent requests creating ports
on *one* network, only one request succeeds, other requests
mostly getting 'standardattribute' update error as network
revision_number changed about 8 times for all port updates.
This patch increases network revision_number in database,
only when a port is created in a network, instead of each
update on port.
This patch adds ``net-mtu-writable`` API extension that allows to write
to network ``mtu`` attribute.
The patch also adds support for the extension to ml2, as well as covers
the feature with unit and tempest tests. Agent side implementation of
the feature is moved into a separate patch to ease review.
DocImpact: neutron controller now supports ``net-mtu-writable`` API
APIImpact: new ``net-mtu-writable`` API extension was added.
Tag mechanism supports network, subnet, port, subnetpool
router resources only. This patch allow tag mechanism to support
resources with standard attribute.
Two old extenions are kept because of backward compatibility.
They will be removed in Queens release.
APIImpact: Tag is supported by resources with standard attribute
DocImpact: allow users to set tags on resources with standard attribute
neutron-lib now contains the API definitions for neutron's core
resources. This patch removes the constant core resource and collection
variables and uses them from lib. Subsequent patches will consume the
actual core resource attribute definitions.
All of the additional lookup logic was essentially duplicating
a relationship property of 'load_on_pending=True', which tells
SQLAlchemy to lookup the relationship during object creation .
So we can dump all of this logic and just use that option.
Related: blueprint push-notifications
The Subnet model has a relationship to Network that is only
used for bumping the revision of the network. However, loading
the network loads all of its relationships as well so we end
up wasting a bunch of time loading segments, DHCP agents, etc all
for no reason. During a subnet update, we get the subnet from the
DB several times (policy engine, before update, after update,
and dhcp agent scheduler) so the slowness in get_subnet is amplified.
The recent switch to subqueries has made loading all of this
unrelated data much more expensive since each relationship requires
a query, which was likely what brought this problem to the surface.
This adjusts the relationship to just load the standard attr object
of the network directly so we can still bump the revision of the network
without having to load the network itself.
This resulted in a ~20%-30% reduction of time spent in single update-subnet
operations on my dev environment (1sec to 650-850ms).
This relationship doesn't serve a purpose and it results
in SQLAlchemy trying to set the network_id column on ports
to NULL when attempting to delete a network with ports in it.
This results in the following exception:
IntegrityError: (1048, u"Column 'network_id' cannot be null")
This exception is not treated as a DBReferenceError by olso.db
which makes detecting races between ports being created on a
network being deleted very noisy in the logs since oslo.db treats
this as a generic DBError.
By removing the relationship, deleting a network will now just
result in a boring standard foreign key constraint failure which
will propagate from oslo.db as a DBReferenceError that is easy
The patch that depends on this one adjusts the delete network logic
to capture these ReferenceErrors.
This switches to the use of subqueries for 1-m relationships
which will result in a higher constant query factor but will
eliminate the potential for cross-product explosions.
Remove The following _MAX_LEN constants from
neutron/api/v2/attributes.py and use the corresponding DB field size
constants from neutron_lib.db.constants instead.
NAME_MAX_LEN --> NAME_FIELD_SIZE
TENANT_ID_MAX_LEN --> PROJECT_ID_FIELD_SIZE
DESCRIPTION_MAX_LEN --> DESCRIPTION_FIELD_SIZE
LONG_DESCRIPTION_MAX_LEN --> LONG_DESCRIPTION_FIELD_SIZE
DEVICE_ID_MAX_LEN --> DEVICE_ID_FIELD_SIZE
DEVICE_OWNER_MAX_LEN --> DEVICE_NAME_FIELD_SIZE
In alembic migration scripts, the raw numerical value is used.
For more information, see:
Loading subnets as part of the networks list and networks
as part of the subnets list appears to have a significant
impact when the network has tags and the subnets have
extra routes entries. This is even further compounded by
the network having rbac entries (likely due to the subnet
inheriting the RBAC entries of the network with the custom
join condition in the model).
This patch converts rbac_entries on both subnet and network
to use a subquery and converts the network and subnets
relationships on the subnet and network models (respectively)
to use subqueries as well.
On my dev environment after running the script in the report,
a network list took 5 minutes. Converting just the rbac_entries
or just the network/subnet relationship to subqueries reduced it
to 3-5 seconds. Converting both (as this patch does), reduces it
back down to a couple of hundred milliseconds (normal perf of my
development env with the current network count).
Subqueries will just cost us a constant number of queries and
won't scale up with result count so this should not impact scalability
in any way.
None of these fields are queryable from the API, so we don't need
to worry about breaking queries against the models.
This adds an order_by clause to the fixed_ips relationship
on the port object to ensure that the fixed_ip ordering is
consistent between a create, an update, and a get request
for a port. Without it we were at the mercy of the sql backend
to determine how it felt like ordering them on the join condition.
Those objects are intentionally not integrated into the database code so
far. This is to quicken access to their definitions to implement
push-notifications for security groups and ports.
The object embeds segmentation information in addition to what's
available through the model. Specifically, binding_levels field exposes
all ml2 binding levels, that from their side load corresponding network
segment object. The order for level objects in binding_levels list field
is guaranteed to be in the order of level. So the consumers can eg.
access the bottom binding info with:
For PortBindingLevel object, we want to expose segmentation info. This
is achieved through a 'segment' ObjectField. The database model itself
contains segment_id too. There is no reason though to expose it for
Level object in two places (one as a model field, another one through
the ObjectField), so we avoid adding ID field. The base class that
handles loading for ObjectField based synthetic fields was assuming that
objects always have a field per model attribute, so it needed a slight
adjustment to support this case, where we extract foreign_keys
attributes from the model itself if the field is not present on the
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Partially-Implements: blueprint push-notifications
Co-Authored-By: Martin Hickey <firstname.lastname@example.org>
Co-Authored-By: Rossella Sblendido <email@example.com>
Co-Authored-By: Manjeet Singh Bhatia <firstname.lastname@example.org>
Co-Authored-By: Brandon Logan <email@example.com>
Co-Authored-By: Victor Morales <firstname.lastname@example.org>
This adds a way for standard attribute models to declare
the API resources they show up in. It then adds a utility
function to the standard_attr module to grab a map of all
API resources and their corresponding models.
This can be used by any processing code that wants to add
fields to standard attribute resources.
This also adjusts the existing extensions to leverage this
Partially-Implements: blueprint add-neutron-extension-resource-timestamp
This is the Neutron side of the unaddressed ports blueprint. In order
to allow unaddressed ports, Nova wants the port to explicitly say it
is okay that it doesn't have any IP addresses.
In Neutron, an unaddressed port is one that was created by explicitly
passing  in fixed_ips to create the port. A new DB field is added
to the port to distinguish the unaddressed port case from the deferred
IP allocation case where routed networks is involved.
Implements: blueprint vm-without-l3-address
APIImpact: port now has ip_allocation attribute, set on port create
The model_base file is going to move to Neutron lib in
I2087c6f5f66154cdaa4d8fa3d86f5e493f1d24d9. This will mainly leave
behind only the standard attributes related stuff so the name
'model_base' won't make much sense. This moves the standard attribute
related things into its own module so model_base can conceivably be
Remove deprecation warnings for various constants
and exceptions that have moved to neutron_lib.
Fix miscellaneous other deprecations.
Uses constants instead of l3_constants when importing
Co-Authored By: Henry Gessau <email@example.com>
Co-Authored By: Gary Kotton <firstname.lastname@example.org>
This patch uses revision_number in database. When creating
a subnet in a network, the revision_number of the network
will be increased. That will prevent overlapping CIDR
(overlapping CIDR means some subnets' cidrs are overlapping)
on *one* network.
Basically, in case of concurrent requests creating subnets
on *one* network, only one request successes, other requests
needs retrying request.
This adds the logic to increment the revision numbers
for objects whenever there are changes and it exposes
the revision number via a field in the API.
This is handled with a new default service plugin that
subscribes to DB events and bumps revision numbers for
any objects that were modified.
It also handles the logic for bumping the revision number
of a parent in a relationship where the children aren't
top-level neutron objects that would be tracked individually.
This is accomplished with a 'revises_on_change' attribute
on the child models that the service plugin will use to
find the parent and bump its revision.
API tests are included to test the revision numbers
added to each standard attribute enabled object.
This patch set aims to move all the code related to DNS integration
from the DB core plugin to the DNS ML2 extension module.
By doing this, this patchset removes the dns related code in
db_base_plugin_v2 and the dns exteions module talks with core plugin
only through the method extension_manager and apply_dict_extend_functions
By properly implementing the generation of the dns_assignment attribute
for ports in the DNS ML2 extension module, this patchset also fixes
Today, existing networks may not reflect MTU configured for
neutron-server, if they were created when neutron-server was using
different MTU setup for its infrastructure, or when it was using bad
default values for network MTUs (specifically, before Mitaka, all networks
were getting MTU = 0 by default, disabling both advertisement and data
path MTU size enforcement).
This patch stops persisting MTU in the database on network create and
instead calculate it on every network resource fetch.
DocImpact Now changes to MTU configuration options immediately affect
existing network MTUs, not just new networks.
UpgradeImpact Existing networks with invalid MTU persisted in database
may change their MTU values to reflect configuration.
There is a bug in pep8, when 'select' used, it omits all default checks
and runs only those specified by 'select'. We got hit by this issue
since I2d26534230ffe5d01aa0aab6ec902f81cfba774d was merged which lead to
almost no static checks in pep8 job.
Also note that off_by_default decorator has no effect for now because
factory in hacking is triggered after ignored checks are collected.
There will be a follow-up patch for that in order to make pep8 doing
its job quickly.
This patch set is for breaking the circular dependency between
Agent/NetworkDhcpAgentBinding. The goal of this is to implement
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Some 'Port' queries use 'device_id' column for lookup.
Such queries could be observed in database query log (at least) during
instance launch. In the absence of 'device_id' index that leads to full
table scan. That causes unnecessary database load and impacts query
Because the join conditions for Subnet rbac entries
are manually specified, SQLAlchemy is not
automatically detecting that this relationship is a list.
This adds the uselist=True kwarg to the relationship to
ensure that it's always handled as a list.
add_is_default_to_subnetpool migration is not working with a
non-empty subnetpool table because it adds a new non-nullable column
(is_default) to subnetpool table without a default value.
This change adds a server_default value to is_default column in
subnetpool table and updates associated db migration.
This adds the IPAllocation object created in the _store_ip_allocation
method to the session info dictionary to prevent it from being
immediately garbage collected. This is necessary because otherwise a
new persistent object will be created when the fixed_ips relationship
is referenced during the rest of the port create/update opertions.
This persistent object will then interfere with a retry operation
that uses the same session if it tries to create a conflicting record.
By preventing the object from being garbage collected, the reference
to fixed IPs will re-use the newly created sqlalchemy object instead
which will properly be cleaned up on a rollback.
This also removes the 'passive_delete' option from the fixed_ips
relationship on ports because IPAllocation objects would now be
left in the session after port deletes. At first glance, this might
look like a performance penalty because fixed_ips would be looked
up before port deletes; however, we already do that in the IPAM
code as well as the ML2 code so this relationship is already being
loaded on the delete_port operation.
In order to give users and operators more flexibility in
annotating the purpose of various Neutron resources, this patch
adds a description field limited to 255 chars to all of the
Neutron resources that reference the standard attribute table.
The resource that reference the table are the following:
security_group_rules, security_groups, ports, subnets,
networks, routers, floatingips, subnetpools
This patch adds a description table related to standard attributes
and migrates over the existing security group description to the new
table as well.
Co-Authored-By: James Dempsey <email@example.com>
DocImpact: Adds a description field to all resources outline in
With address scope being enabled, networks now are in one ipv4
address scope and one ipv6 address scope.
This patch adds derived attributes to the network as part of the
address scopes extension that will show related address scopes
when viewing a network through the API.
This patch ensures that models_v2 and the router db module
import the agent binding modules so their models are defined
in the sqlalchemy metadata. Otherwise, using an agentless
core/L3 plugin may result in a failure to initialize if nothing
imported the agent module.
The previous code was doing a DB lookup for each network's
availability zone which was significantly impacting the
performance of network listings.
This patch adjusts the network model to be automatically joined
to the DHCP agents table that the AZ code uses to populate the
AZs for the network.
This patch adds the availability_zone support for network.
Co-Authored-By: IWAMOTO Toshihiro <firstname.lastname@example.org>
Partially-implements: blueprint add-availability-zone
This adds a new 'standardattributes' table and adds a foreign-key
references from ports, subnets, networks, subnetpools, routers,
securitygroups, and floatingips to this table.
This will make it easy to add new things to the schema like
timestamp fields or anything else that applies to multiple types
of Neutron resources. The new fields would just be added to the
'neutronresources' table instead of being duplicated across each
resource's table. Or, if the the relationship is 1-to-many (e.g. tags),
the new association table would be related to the 'standardattribute'
This patch adds a new boolean 'is_default' property to subnetpools. This
allows the admin to set the default v4/v6 subnetpools via the API rather
than the existing neutron.conf options - which are deprecated by this patch.
Only one subnetpool per IP family can be set to default.
Co-Authored-By: Carl Baldwin <email@example.com>