The "subnet" OVO does not allow to have an empty (None) "network_id" but
the DB "subnet" table allows to have this parameter empty (NULL) in the
database. In order to avoid any problem like the one described in the
bug, this patch ensures the database "subnet" register does always have
a "network_id" value and if a "network" register is being deleted, all
related "subnet" registers are checked first.
Change-Id: Iad210f0585b4201fdb87187b44a9b42267b58db4
Closes-Bug: #1839658
Basically, in case of concurrent requests creating ports
on *one* network, only one request succeeds, other requests
mostly getting 'standardattribute' update error as network
revision_number changed about 8 times for all port updates.
This patch increases network revision_number in database,
only when a port is created in a network, instead of each
update on port.
Change-Id: Idffb4edda616677b9b071644d3835c85052091a5
Closes-Bug: #1782421
This patch adds ``net-mtu-writable`` API extension that allows to write
to network ``mtu`` attribute.
The patch also adds support for the extension to ml2, as well as covers
the feature with unit and tempest tests. Agent side implementation of
the feature is moved into a separate patch to ease review.
DocImpact: neutron controller now supports ``net-mtu-writable`` API
extension.
APIImpact: new ``net-mtu-writable`` API extension was added.
Related-Bug: #1671634
Change-Id: Ib232796562edd8fa69ec06b0cc5cb752c1467add
Increment the revision number when RBAC policies are
changed since it impacts the calculation of the 'shared'
field.
Closes-Bug: #1708079
Change-Id: I4c7eeff8745eff3761d54ef6d3665cf3dc6e6222
Tag mechanism supports network, subnet, port, subnetpool
router resources only. This patch allow tag mechanism to support
resources with standard attribute.
Two old extenions are kept because of backward compatibility.
They will be removed in Queens release.
APIImpact: Tag is supported by resources with standard attribute
DocImpact: allow users to set tags on resources with standard attribute
Change-Id: Id7bb13b5beb58c313eea94ca03835d3daf5c94bc
Closes-Bug: #1682775
neutron-lib now contains the API definitions for neutron's core
resources. This patch removes the constant core resource and collection
variables and uses them from lib. Subsequent patches will consume the
actual core resource attribute definitions.
NeutronLibImpact
Change-Id: Ia9afdf620cd538b2aa420593277d6403a45c996b
All of the additional lookup logic was essentially duplicating
a relationship property of 'load_on_pending=True', which tells
SQLAlchemy to lookup the relationship during object creation [1].
So we can dump all of this logic and just use that option.
1. http://docs.sqlalchemy.org/en/latest/orm/relationship_api.html
#sqlalchemy.orm.relationship.params.load_on_pending
Related: blueprint push-notifications
Change-Id: I0e495a50f5cab9b6449825039d7683d77de1e763
The Subnet model has a relationship to Network that is only
used for bumping the revision of the network. However, loading
the network loads all of its relationships as well so we end
up wasting a bunch of time loading segments, DHCP agents, etc all
for no reason. During a subnet update, we get the subnet from the
DB several times (policy engine, before update, after update,
and dhcp agent scheduler) so the slowness in get_subnet is amplified.
The recent switch to subqueries[1] has made loading all of this
unrelated data much more expensive since each relationship requires
a query, which was likely what brought this problem to the surface.
This adjusts the relationship to just load the standard attr object
of the network directly so we can still bump the revision of the network
without having to load the network itself.
This resulted in a ~20%-30% reduction of time spent in single update-subnet
operations on my dev environment (1sec to 650-850ms).
1. I6952c48236153a8e2f2f155375b70573ddc2cf0f
Closes-Bug: #1665967
Change-Id: Ie5f956ddf0704a5dde6e4f801a446f48fea5f697
This relationship doesn't serve a purpose and it results
in SQLAlchemy trying to set the network_id column on ports
to NULL when attempting to delete a network with ports in it.
This results in the following exception:
IntegrityError: (1048, u"Column 'network_id' cannot be null")
This exception is not treated as a DBReferenceError by olso.db
which makes detecting races between ports being created on a
network being deleted very noisy in the logs since oslo.db treats
this as a generic DBError.
By removing the relationship, deleting a network will now just
result in a boring standard foreign key constraint failure which
will propagate from oslo.db as a DBReferenceError that is easy
to catch.
The patch that depends on this one adjusts the delete network logic
to capture these ReferenceErrors.
Change-Id: I4166a9bbaded57a77576400929a150123b8f1b2e
This switches to the use of subqueries for 1-m relationships
which will result in a higher constant query factor but will
eliminate the potential for cross-product explosions.
Closes-Bug: #1649317
Change-Id: I6952c48236153a8e2f2f155375b70573ddc2cf0f
Remove The following _MAX_LEN constants from
neutron/api/v2/attributes.py and use the corresponding DB field size
constants from neutron_lib.db.constants instead.
NAME_MAX_LEN --> NAME_FIELD_SIZE
TENANT_ID_MAX_LEN --> PROJECT_ID_FIELD_SIZE
DESCRIPTION_MAX_LEN --> DESCRIPTION_FIELD_SIZE
LONG_DESCRIPTION_MAX_LEN --> LONG_DESCRIPTION_FIELD_SIZE
DEVICE_ID_MAX_LEN --> DEVICE_ID_FIELD_SIZE
DEVICE_OWNER_MAX_LEN --> DEVICE_NAME_FIELD_SIZE
In alembic migration scripts, the raw numerical value is used.
For more information, see:
http://lists.openstack.org/pipermail/openstack-dev/2016-October/105789.html
NeutronLibImpact
Change-Id: I734890372584fe27e5d6ec38c0cad2de882ff11c
Loading subnets as part of the networks list and networks
as part of the subnets list appears to have a significant
impact when the network has tags and the subnets have
extra routes entries. This is even further compounded by
the network having rbac entries (likely due to the subnet
inheriting the RBAC entries of the network with the custom
join condition in the model).
This patch converts rbac_entries on both subnet and network
to use a subquery and converts the network and subnets
relationships on the subnet and network models (respectively)
to use subqueries as well.
On my dev environment after running the script in the report,
a network list took 5 minutes. Converting just the rbac_entries
or just the network/subnet relationship to subqueries reduced it
to 3-5 seconds. Converting both (as this patch does), reduces it
back down to a couple of hundred milliseconds (normal perf of my
development env with the current network count).
Subqueries will just cost us a constant number of queries and
won't scale up with result count so this should not impact scalability
in any way.
None of these fields are queryable from the API, so we don't need
to worry about breaking queries against the models.
Partial-Bug: #1649317
Change-Id: Ic1947e3d78d58a79b21344b10cb7ab0e573e419f
This adds an order_by clause to the fixed_ips relationship
on the port object to ensure that the fixed_ip ordering is
consistent between a create, an update, and a get request
for a port. Without it we were at the mercy of the sql backend
to determine how it felt like ordering them on the join condition.
Closes-Bug: #1630410
Change-Id: I523e0ab6e376f5ff6205b1cc1748aa6d546919cb
Those objects are intentionally not integrated into the database code so
far. This is to quicken access to their definitions to implement
push-notifications for security groups and ports.
The object embeds segmentation information in addition to what's
available through the model. Specifically, binding_levels field exposes
all ml2 binding levels, that from their side load corresponding network
segment object. The order for level objects in binding_levels list field
is guaranteed to be in the order of level. So the consumers can eg.
access the bottom binding info with:
port_obj.binding_levels[-1].segment
For PortBindingLevel object, we want to expose segmentation info. This
is achieved through a 'segment' ObjectField. The database model itself
contains segment_id too. There is no reason though to expose it for
Level object in two places (one as a model field, another one through
the ObjectField), so we avoid adding ID field. The base class that
handles loading for ObjectField based synthetic fields was assuming that
objects always have a field per model attribute, so it needed a slight
adjustment to support this case, where we extract foreign_keys
attributes from the model itself if the field is not present on the
object.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Partially-Implements: blueprint push-notifications
Change-Id: I25de14e42e345d9235dbf4097c298ef5d606de51
Co-Authored-By: Martin Hickey <martin.hickey@ie.ibm.com>
Co-Authored-By: Rossella Sblendido <rsblendido@suse.com>
Co-Authored-By: Manjeet Singh Bhatia <manjeet.s.bhatia@intel.com>
Co-Authored-By: Brandon Logan <brandon.logan@rackspace.com>
Co-Authored-By: Victor Morales <victor.morales@intel.com>
This adds a way for standard attribute models to declare
the API resources they show up in. It then adds a utility
function to the standard_attr module to grab a map of all
API resources and their corresponding models.
This can be used by any processing code that wants to add
fields to standard attribute resources.
This also adjusts the existing extensions to leverage this
new functionality.
Partially-Implements: blueprint add-neutron-extension-resource-timestamp
Change-Id: Idc8923d0e983fcb0690f8cb5b55a5aff8690154f
This is the Neutron side of the unaddressed ports blueprint. In order
to allow unaddressed ports, Nova wants the port to explicitly say it
is okay that it doesn't have any IP addresses.
In Neutron, an unaddressed port is one that was created by explicitly
passing [] in fixed_ips to create the port. A new DB field is added
to the port to distinguish the unaddressed port case from the deferred
IP allocation case where routed networks is involved.
Change-Id: Ia61af4c14e955697a7d3fcc0bf4826a6d9475c98
Implements: blueprint vm-without-l3-address
APIImpact: port now has ip_allocation attribute, set on port create
The model_base file is going to move to Neutron lib in
I2087c6f5f66154cdaa4d8fa3d86f5e493f1d24d9. This will mainly leave
behind only the standard attributes related stuff so the name
'model_base' won't make much sense. This moves the standard attribute
related things into its own module so model_base can conceivably be
eliminated entirely.
Change-Id: Icaf3220fbc5723f2b5421a494371ef274d7073c7
Remove deprecation warnings for various constants
and exceptions that have moved to neutron_lib.
Fix miscellaneous other deprecations.
Uses constants instead of l3_constants when importing
neutron-lib constants.
Co-Authored By: Henry Gessau <gessau@gmail.com>
Co-Authored By: Gary Kotton <gkotton@vmware.com>
Change-Id: Ib0e8ff5c3e23677c1009241a1818cbc8a3430c38
This patch uses revision_number in database. When creating
a subnet in a network, the revision_number of the network
will be increased. That will prevent overlapping CIDR
(overlapping CIDR means some subnets' cidrs are overlapping)
on *one* network.
Basically, in case of concurrent requests creating subnets
on *one* network, only one request successes, other requests
needs retrying request.
Change-Id: Id6548535075bed87a4b36e1462db546ab9163f29
Closes-Bug: #1532695
This patch addresses all the DeprecationWarning errors related
to constants in neutron.db.models_v2.
Change-Id: I54402d4441b3ccb15bab63adf830fa196bba62c3
This adds the logic to increment the revision numbers
for objects whenever there are changes and it exposes
the revision number via a field in the API.
This is handled with a new default service plugin that
subscribes to DB events and bumps revision numbers for
any objects that were modified.
It also handles the logic for bumping the revision number
of a parent in a relationship where the children aren't
top-level neutron objects that would be tracked individually.
This is accomplished with a 'revises_on_change' attribute
on the child models that the service plugin will use to
find the parent and bump its revision.
API tests are included to test the revision numbers
added to each standard attribute enabled object.
Partially-Implements: bp/push-notifications
Change-Id: I476d3e03c8ee763cc4be6d679fe9f501eb3a19b5
This patch set aims to move all the code related to DNS integration
from the DB core plugin to the DNS ML2 extension module.
By doing this, this patchset removes the dns related code in
db_base_plugin_v2 and the dns exteions module talks with core plugin
only through the method extension_manager and apply_dict_extend_functions
By properly implementing the generation of the dns_assignment attribute
for ports in the DNS ML2 extension module, this patchset also fixes
https://bugs.launchpad.net/neutron/+bug/1579977
Change-Id: I63afb1a1bfeeb14eefb54681dc64959144deeb25
Closes-Bug: #1579601
Closes-Bug: #1579977
Today, existing networks may not reflect MTU configured for
neutron-server, if they were created when neutron-server was using
different MTU setup for its infrastructure, or when it was using bad
default values for network MTUs (specifically, before Mitaka, all networks
were getting MTU = 0 by default, disabling both advertisement and data
path MTU size enforcement).
This patch stops persisting MTU in the database on network create and
instead calculate it on every network resource fetch.
DocImpact Now changes to MTU configuration options immediately affect
existing network MTUs, not just new networks.
UpgradeImpact Existing networks with invalid MTU persisted in database
may change their MTU values to reflect configuration.
Change-Id: Iee4f5037bf10b73ba98464143b183aacb59c22f2
Closes-Bug: #1556182
There is a bug in pep8, when 'select' used, it omits all default checks
and runs only those specified by 'select'. We got hit by this issue
since I2d26534230ffe5d01aa0aab6ec902f81cfba774d was merged which lead to
almost no static checks in pep8 job.
Also note that off_by_default decorator has no effect for now because
factory in hacking is triggered after ignored checks are collected.
There will be a follow-up patch for that in order to make pep8 doing
its job quickly.
[1] https://github.com/PyCQA/pycodestyle/issues/390
Related-Bug: 1594756
Change-Id: I8e27f40908e1bb4307cc7c893169a9d99f3433c4
This patch set is for breaking the circular dependency between
Agent/NetworkDhcpAgentBinding. The goal of this is to implement
Agent OVO
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Change-Id: I3f2a8bcc6f8644e94e377dc916fba6743cb230bc
Some 'Port' queries use 'device_id' column for lookup.
Such queries could be observed in database query log (at least) during
instance launch. In the absence of 'device_id' index that leads to full
table scan. That causes unnecessary database load and impacts query
response time.
Change-Id: If42b7d3265e216d393d3ab8c172b97637af908cc
Closes-Bug: #1574750
Because the join conditions for Subnet rbac entries
are manually specified, SQLAlchemy is not
automatically detecting that this relationship is a list.
This adds the uselist=True kwarg to the relationship to
ensure that it's always handled as a list.
Change-Id: Ia4ae57ddd932260691584ae74c0305a79b2e60a9
Closes-Bug: #1557959
add_is_default_to_subnetpool migration[1] is not working with a
non-empty subnetpool table because it adds a new non-nullable column
(is_default) to subnetpool table without a default value.
This change adds a server_default value to is_default column in
subnetpool table and updates associated db migration.
[1] neutron/db/migration/alembic_migrations/versions/mitaka/expand/\
13cfb89f881a_add_is_default_to_subnetpool.py
Closes-Bug: #1557757
Change-Id: Ib0b2ac061d21a7d908f14b399b0bba06d94b7241
This adds the IPAllocation object created in the _store_ip_allocation
method to the session info dictionary to prevent it from being
immediately garbage collected. This is necessary because otherwise a
new persistent object will be created when the fixed_ips relationship
is referenced during the rest of the port create/update opertions.
This persistent object will then interfere with a retry operation
that uses the same session if it tries to create a conflicting record.
By preventing the object from being garbage collected, the reference
to fixed IPs will re-use the newly created sqlalchemy object instead
which will properly be cleaned up on a rollback.
This also removes the 'passive_delete' option from the fixed_ips
relationship on ports because IPAllocation objects would now be
left in the session after port deletes. At first glance, this might
look like a performance penalty because fixed_ips would be looked
up before port deletes; however, we already do that in the IPAM
code as well as the ML2 code so this relationship is already being
loaded on the delete_port operation.
Closes-Bug: #1556178
Change-Id: Ieee1343bb90cf111c55e00b9cabc27943b46c350
In order to give users and operators more flexibility in
annotating the purpose of various Neutron resources, this patch
adds a description field limited to 255 chars to all of the
Neutron resources that reference the standard attribute table.
The resource that reference the table are the following:
security_group_rules, security_groups, ports, subnets,
networks, routers, floatingips, subnetpools
This patch adds a description table related to standard attributes
and migrates over the existing security group description to the new
table as well.
Co-Authored-By: James Dempsey <jamesd@catalyst.net.nz>
APIImpact
DocImpact: Adds a description field to all resources outline in
commit message.
Closes-Bug: #1483480
Change-Id: I6e1ef53d7aae7d04a5485810cc1db0a8eb125953
With address scope being enabled, networks now are in one ipv4
address scope and one ipv6 address scope.
This patch adds derived attributes to the network as part of the
address scopes extension that will show related address scopes
when viewing a network through the API.
APIImpact
Change-Id: Ib1657636033ad2c0009d50ebe7c5ae4f72f6b175
Closes-Bug: #1547380
This patch ensures that models_v2 and the router db module
import the agent binding modules so their models are defined
in the sqlalchemy metadata. Otherwise, using an agentless
core/L3 plugin may result in a failure to initialize if nothing
imported the agent module.
Change-Id: I77240462b618d6e95c7c5862441d6d179e371f4f
Closes-Bug: #1529022
The previous code was doing a DB lookup for each network's
availability zone which was significantly impacting the
performance of network listings.
This patch adjusts the network model to be automatically joined
to the DHCP agents table that the AZ code uses to populate the
AZs for the network.
Change-Id: I908ceb1a68e0eed7c304e3ff82279ad6fa406167
Closes-Bug: #1525740
This patch adds the availability_zone support for network.
APIImpact
DocImpact
Change-Id: I9259d9679c74d3b3658771290e920a7896631e62
Co-Authored-By: IWAMOTO Toshihiro <iwamoto@valinux.co.jp>
Partially-implements: blueprint add-availability-zone
This adds a new 'standardattributes' table and adds a foreign-key
references from ports, subnets, networks, subnetpools, routers,
securitygroups, and floatingips to this table.
This will make it easy to add new things to the schema like
timestamp fields or anything else that applies to multiple types
of Neutron resources. The new fields would just be added to the
'neutronresources' table instead of being duplicated across each
resource's table. Or, if the the relationship is 1-to-many (e.g. tags),
the new association table would be related to the 'standardattribute'
table.
Related-Bug: #1496802
Change-Id: Iaa3ba81a7e9cae09cea153720b29879d8cc9a080
This patch adds a new boolean 'is_default' property to subnetpools. This
allows the admin to set the default v4/v6 subnetpools via the API rather
than the existing neutron.conf options - which are deprecated by this patch.
Only one subnetpool per IP family can be set to default.
DocImpact
ApiImpact
Co-Authored-By: Carl Baldwin <carl@ecbaldwin.net>
Change-Id: I5daba2347cfb91fac0b155b2c1b459ee7d9e4505
Closes-Bug: 1501328
Rodolfo Alonso Hernandez