Commit Graph

263 Commits (99f4495c940011293e3cabbb590770dc1e7b6900)

Author SHA1 Message Date
Boden R e4aa5902f7 use context manager from neutron-lib
The neutron.db.api.context_manager already references neutron-lib's
context manager; so consumers of it are already using neutron-lib. This
patch switches neutron's references to the context_manager over to
use neutron-lib's directly rather than that in neutron.db.api.


Change-Id: I97120faeec73690592ed21a5ec3c6202f61e1429
5 years ago
ZhaoBo 35d945e92f Add ext_parent policy check
Add common parent owner check for the resources which introduced by
service plugin.

Then port forwarding resource will share the same tenant_id with
floatingip. That means only the fip owner can create/update/get/delete
the associated port forwarding resource.

Partially-Implements: blueprint port-forwarding
Partial-Bug: #1491317
Change-Id: I450c674e55ca15e1d9a6a6224138f3305427da68
5 years ago
Hongbin Lu 2b1d8ea4a2 Implement filter validation
Enforce validation on filter parameters on list requests.
If an API request contains an unknown or unsupported parameter,
the server will return a 400 response instead of silently ignoring
the invalid input.

In resource attributes map, all filter parameters are annotated by
the ``is_filter`` keyword. Attributes with is_filter set to True
are candidates for validation.

Enabling filter validation requires support from core plugin and
all service plugins so each plugin need to indicate if it supports
the validation by setting ``__filter_validation_support`` to True.
If this field is not set, the default is False and validation is
turned off. Right now, the ML2 plugin and all the in-tree service
plugin support filter validation. Out-of-tree plugins will have
filter validation disabled by default.

An API extension is introduced to allow API users to discover this
new API behavior. This feature can be disabled by cloud operators
if they choose to do that. If it is disabled, the extension won't
be presented.

Depends-On: Ic3ab5b3ffdc378d570678b9c967cb42b0c7a8a9b
Depends-On: I4397df1c35463a8b532afdc9c5d28b37224a37b4
Depends-On: I3f2e6e861adaeef81a1a5819a57b28f5c6281d80
Depends-On: I1189bc9a50308df5c7e18c329f3a1262c90b9e12
Depends-On: I057cd917628c77dd20c0ff7747936c3fec7b4844
Depends-On: I0b24a304cc3466a2c05426cdbb6f9d99f1797edd

Change-Id: I21bf8a752813802822fd9966dda6ab3b6c4abfdc
Partial-Bug: #1749820
5 years ago
Zuul 414c0d40c9 Merge "use retry_db_errors from neutron-lib" 5 years ago
Boden R e4348eb1e1 use retry_db_errors from neutron-lib
The externally consumed APIs from neutron.db.api were rehomed into
neutron-lib with

This patch consumes the retry_db_errors function from lib by:
- Removing retry_db_errors from neutron.db.api
- Updating the imports for retry_db_errors to use it from lib
- Using the DB API retry fixture from lib in the UTs where applicable
- Removing the UTs for neutron.db.api as they are now covered in lib


Change-Id: I1feb842d3e0e92c945efb01ece29856335a398fe
5 years ago
Hongbin Lu d950949b90 Ensure request's object type is dict
Handle an edge case that API users send a POST/PUT request with
invalid data in request body.

Closes-Bug: #1780327
Change-Id: I3877c18a18ac506dc8f4a9ded2a18b53b9f6cfae
5 years ago
Brian Haley 7cfdf4aa81 Fix all pep8 E129 errors
Fixed all pep8 E129 errors and changed tox.ini to no longer
ignore them.

Change-Id: I0b06d99ce1d473b79a4cfdd173baa4f02e653847
5 years ago
Boden R 44292f4c9a consume neutron-lib resources attr map
Today we shim the RESOURCE_ATTRIBUTE_MAP in neutron; it references the
equivelant in neutron-lib named RESOURCES.

This patch removes neutron's RESOURCE_ATTRIBUTE_MAP and cleans up
neutron.api.v2.attributes in prep to delete it.
To do so:
- CORE_RESOURCES and RESOURCE_FOREIGN_KEYS are moved to the single
module that references them respectively and the are made private (no
consumers use them).
- get_collection_info is removed and instead the 2 uses in neutron
just use the get() method of the RESOURCES map. There are no external
uses of get_collection_info.
- References using RESOURCE_ATTRIBUTE_MAP are switched over to
neutron-lib's RESOURCES.
- The neutron.api.v2.attributes module is removed as it's empty now.
- A few api attribute UTs are removed; there's nothing to test as per
this patch.


Change-Id: Iaacee584d499c4d33d6d2dd9609c7ac0f2cfc386
5 years ago
Boden R 6141c73f09 remove SHARED constant that now lives in neutron-lib
The SHARED constant was rehomed into neutron-lib with commit
This patch removes the SHARED constant from neutron.


Change-Id: I9304ca7bdec6698e549c294c124c60230b6b9eaf
5 years ago
Boden R 3f1a9846d2 use callback payloads for REQUEST/RESPONSE events
This patch switches callbacks over to the payload object style events
[1] for BEFORE_RESPONSE and AFTER_REQUEST based notifications. To do
so an APIEventPayload object is used with the publish() method to
pass along the API related data. In addition a few UTs are updated to
work with the changes.



Change-Id: Ibd8559e0db9dcc995abf8937a0cb764b21a18531
6 years ago
Boden R 9e67ba5052 use core resource api defs from lib
The core resource API definitions are in neutron-lib and are already
setup in a RESOURCES map, similar to neutron's global
RESOURCE_ATTRIBUTE_MAP. However, a number of consumers directly use
RESOURCE_ATTRIBUTE_MAP and moreover can perform some interesting
operations on it while mockin" for tests. For that reason this
patch proposes we phase in the use of neutron's RESOURCES map rather
than a rip and replace of RESOURCE_ATTRIBUTE_MAP.

This patch removes the API definitions for the core neutron resources
by updating neutron's global RESOURCE_ATTRIBUTE_MAP to reference lib's
RESOURCE map. In addition the AttributeMapMemento class is removed; it
no longer servers a purpose and neutron-lib's fixture should be used
in its place.

This patch should be safe to merge without worry of impacting consumers.
No one is using AttributeMapMemento [1] and the global map only changed
its reference value.


Change-Id: Ib04fddcbd4465074452b71b16befa4e33b27259e
6 years ago
Boden R 0e25b403a5 use FAULT_MAP from neutron-lib
Idfd956fa213e1ba6c3b1c983f246c5b515951e6d rehomed the FAULT_MAP into
neutron-lib. This patch consumes it from lib and removes its definition
from neutron.


Change-Id: Ib051555ea8496ed7eec8ed30733c964012e31c32
6 years ago
Boden R 7e0d37699f shim FAULT_MAP from neutron-lib
Idfd956fa213e1ba6c3b1c983f246c5b515951e6d rehomed the FAULT_MAP into
neutron-lib. Since consumers expect to access the latest faults in this
map which are updated by neutron at runtime, we can't just move
consumers over to lib's version (it won't be updated by neutron).

Therefore this patch shims neutron's FAULT_MAP to use lib's. Once
this shim lands consumers can move to lib's FAULT_MAP (neutron will
ref it via shim) and finally we can then consume in neutron and remove
in a subsequent patch.

The shim herein shouldn't impact consumers so should be free to land
without worry of their impact.

Change-Id: I8e79f29206ec244d870e19f99c18ccaaa50b4761
6 years ago
Kevin Benton a3dff9769b Delete dead API v2 router code
This code is no longer used now that we have switched to
Pecan and have no option to switch to the legacy API

Change-Id: I0701dfe13024132cf5b9696560b0d027712e7739
6 years ago
Kevin Benton db1058a499 Switch to Pecan for unit tests
This will drop Pecan in place of the old APIRouter for all
unit tests. This will significantly increase the UT coverage
of pecan and will allow us to drop the old API routing logic

The rest of the inline test changes are due to slight plugin loading

Implements: blueprint wsgi-pecan-switch
Change-Id: I76dc23fb7b96d82b0da50285bd0aac76142e81e5
6 years ago
Kevin Benton e2ea0b4652 Drop the web_framework option
This option is no longer needed as we have switched to pecan
and will be deleting the legacy API layer.

Implements: blueprint wsgi-pecan-switch
Change-Id: Ib153d75bb10375e048a8417139873bdf9dca8769
6 years ago
Inessa Vasilevskaya 7322bd6efb Make code follow log translation guideline
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.

Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
6 years ago
Kevin Benton 9662e2b170 Use context interface for constraint
Use the new constraint interface on the context rather than
setting an ugly attribute.

Depends-On: I6bc2539a1ddbf7990164abeb8bb951ddcb45c993

Related-Bug: #1493714
Change-Id: I9142ca96a40092b2a4c94920c4ded9bbc3a0b35b
6 years ago
Jenkins 59006183e1 Merge "API compare-and-swap updates based on revision_number" 6 years ago
Boden R 62576cabf7 use core resource attribute constants from neutron-lib
neutron-lib now contains the API definitions for neutron's core
resources. This patch removes the constant core resource and collection
variables and uses them from lib. Subsequent patches will consume the
actual core resource attribute definitions.


Change-Id: Ia9afdf620cd538b2aa420593277d6403a45c996b
6 years ago
Jenkins d76c45d546 Merge "use attribute functions/operations from neutron-lib" 6 years ago
Kevin Benton 7f17b4759e API compare-and-swap updates based on revision_number
Allows posting revision number matching in the If-Match header
so updates/deletes will only be satisfied if the current revision
number of the object matches.

DocImpact: The Neutron API now supports conditional updates to resources
           that contain the standard 'revision_number' attribute by
           setting the revision_number in an HTTP If-Match header.

Partial-Bug: #1493714
Partially-Implements: blueprint push-notifications
Change-Id: I7d97d6044378eb59cb2c7bdc788dc6c174783299
6 years ago
Boden R 0a1405794f use service type constants from neutron_lib plugins
The well known service type constants are in
neutron_lib.plugins.constants, but for legacy reasons a few still exist
and are referenced from neutron_lib.constants that we'd like to remove.
This patch switches references over to neutron_lib's plugin constants.

Change-Id: I1861448cec303725b30cef8f42029f467f9e03a3
6 years ago
Boden R cbb0970b36 use attribute functions/operations from neutron-lib
neutron-lib now contains the functionality from neutron's attributes
module. This patch moves the local references over to neturon-lib's
implementation and removes the functionality from the attributes


Change-Id: I773702ebd4b66d28059cb3ced64b2b477ceff8af
6 years ago
Roey Chen f5f8a75963 Mask password when logging request body
Change-Id: I825ab268c140b991e39583cd2c2d557a202b7d97
Closes-Bug: #1697408
6 years ago
Ihar Hrachyshka 527468be33 api: work around Routes cutting off suffix from resource id
Routes allows for auxiliary format suffix. Sadly it doesn't distinguish
between an actual format suffix (.json) and any other suffix that may be
part of the id. (like for first.second resource tag). To work this
behavior around, we will reattach the 'format' suffix if it is not of a
supported format (json only at the time of writing).

This of course leaves a corner case where there is a tag where .json is
a part of its id. This seems to be a reasonable balance to leave it
unfixed, because an alternative would probably be not backwards

Closes-Bug: #1694897
Change-Id: I271107150166f0ee680faaa2e3ca6044cf4e8d4f
6 years ago
Boden R 0e2b667bf1 use neutron-lib callbacks
The callback modules have been available in neutron-lib since commit [1]
and are ready for consumption.

As the callback registry is implemented with a singleton manager
instance, sync complications can arise ensuring all consumers switch to
lib's implementation at the same time. Therefore this consumption has
been broken down:
1) Shim neutron's callbacks using lib's callback system and remove
existing neutron internals related to callbacks (devref, UTs, etc.).
2) Switch all neutron's callback imports over to neutron-lib's.
3) Have all sub-projects using callbacks move their imports over to use
neutron-lib's callbacks implementation.
4) Remove the callback shims in neutron-lib once sub-projects are moved
over to lib's callbacks.
5) Follow-on patches moving our existing uses of callbacks to the new
event payload model provided by

This patch implements #2 from above, moving all neutron's callback
imports to use neutron-lib's callbacks.

There are also a few places in the UT code that still patch callbacks,
we can address those in step #4 which may need [2].


[1] fea8bb64ba7ff52632c2bd3e3298eaedf623ee4f
[2] I9966c90e3f90552b41ed84a68b19f3e540426432

Change-Id: I8dae56f0f5c009bdf3e8ebfa1b360756216ab886
6 years ago
fpxie 574312165b Replace six.iteritems with dict.items(Part-2)
according to, now we should avoid
using six.iteritems and replace it with dict.items.

Change-Id: I58a399baa2275f280acc0e6d649f81838648ce5c
Closes-Bug: #1680761
6 years ago
Jenkins 79bf76d73c Merge "Pecan: /v2.0/ views response with resources" 6 years ago
Matthew Edmonds 2ae14cc9ad fix overaggressive 403->404 conversion
When a user is not authorized to see a given resource, we need to
convert HTTP 403s into HTTP 404s to avoid giving away information
that the resource exists. However, the previous code was being
overaggressive and doing this conversion even in some cases where
the user is allowed to see the resource and really needs to know
that what they were trying to do is forbidden, not be told that the
resource doesn't exist. This fixes that logic to only do the 403
to 404 conversion when truly appropriate.

Change-Id: I7a5b0a9e89c8a71490dd74497794a52489f46cd2
Closes-Bug: 1682621
6 years ago
tonytan4ever 5da865c1a8 Pecan: /v2.0/ views response with resources
The legacy framework return a list of resources as the response to:

GET /v2.0/

Current pecan 2.0 controller seems return the same as root controller
content. This patch will make it return the same response as legacy

Change-Id: Icae8c1b417d4d5a65a98eeafe80a26886aa254a3
6 years ago
Ann Kamyshnikova 9195c66cbf Use new enginefacade for quota and provisioning blocks
Use reader and writer for db operations.

Partially-Implements blueprint: enginefacade-switch

Change-Id: I3adaec4cae814c1feb88aa646b99823de9c0eb9e
6 years ago
tonytan4ever ebc7e1fb2f Pecan: Get loaded by paste deploy
This sets up the factory methods needed to have paste deploy create the
pecan app and return it.  It also changes the legacy factory methods to
conditionally use the pecan factory methods if the web_framework config
option is set to 'pecan'.  This way, all deployments of neutron will not
need to change their api-paste.ini files to get pecan toggled on.  It
should just happen without notice once pecan becomes the default.

Also, by moving this to be loaded by paste deploy, there is a good chunk of
code that has been removed because it is no longer necessary.

Co-Authored-By: Brandon Logan <>
Change-Id: I8b1bbea8d90fdc62715cd8b6738ad955df53d7cd
6 years ago
Jenkins 521b1074f1 Merge "Pecan: Fix tags handling for pecan" 6 years ago
Brandon Logan 96a45cadcc Pecan: Fix tags handling for pecan
Tags controller was not being picked up from the extension.  This also
allows the usage of collection_methods that tags uses.  Also, the tags
extension made use of action_status on a resource, which allowed
overriding the default status codes, so this patch handles action_status
as well.

Change-Id: Ic7f5cfddd30f26737469516235053656f1ba6a24
6 years ago
Jenkins 96bffbf6d2 Merge "Remove unused params named "context" in filter_attributes" 6 years ago
Henry Gessau 1c5bf09a03 Use DB field sizes instead of _MAX_LEN constants
Remove The following _MAX_LEN constants from
neutron/api/v2/ and use the corresponding DB field size
constants from neutron_lib.db.constants instead.

 NAME_MAX_LEN              -->  NAME_FIELD_SIZE

In alembic migration scripts, the raw numerical value is used.

For more information, see:


Change-Id: I734890372584fe27e5d6ec38c0cad2de882ff11c
7 years ago
Jenkins 7e9986411c Merge "Remove allow_pagination and allow_sorting config options" 7 years ago
Ihar Hrachyshka 61eb74af31 Remove allow_pagination and allow_sorting config options
They were deprecated in Newton. This patch cleans them up.

Note: it does not mean that the features will not work anymore. On the
contrary, now API will consistently sort and paginate for plugins that
honour the relevant sorting/pagination parameters.

Note2: base resource controller still allows to pass
allow_pagination=False and allow_sorting=False parameters to disable the
features if a registered plugin does not support the features yet.

Change-Id: I5fd30b20f645846d9366740372c4815c4b33e2eb
Related-Bug: #1566514
7 years ago
Henry Gessau f33bada2a9 Remove PLURALS
It's written to but never read from!

Related-Bug: #1642426

This patch is required by neutron-lib adoption because it is part of
rehoming neutron/api/v2/ All subprojects that reference
PLURALS must be patched to remove the reference.


Change-Id: Ifdf29b8fc4c824c0ee840c4f51593a5aa8a22127
7 years ago
Henry Gessau 4151486bd4 Remove REVERSED_PLURALS and get_resource_info()
By changing pecan to store plugins by collection (plural) instead of
resource (singular) we don't need REVERSED_PLURALS and

These are not used outside of neutron core.

Consequently, this exposed a bug with some controllers not honoring an
extension's intention of disallowing pagination and sorting.  This is
also fixed in this patch to get successful test runs.

Co-Authored-By: Brandon Logan <>

Change-Id: I9d529d0a2ad369e2be0a8df3c6f06a6532e8b13d
7 years ago
Armando Migliaccio 17563a802e Adopt neutron-lib plugin directory
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.

Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.

This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.

Usage pattern [1,2] can be translated to [3,4] respectively.

[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()

The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.

Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.

Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.


Partially-implements: blueprint neutron-lib

Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
7 years ago
zhangyanxian 1a0da09b4b Fix typos in &

Change-Id: I220262991e2af9cb9ec649c1a15f2923765fbf2f
7 years ago
Henry Gessau b0e56ac646 Retire deprecations for constants, exceptions, attributes
Subprojects have had enough time to switch to neutron-lib for
the deprecated items from these modules.

Change-Id: Ib2f4acf7e0153667389c57a52f2db33a23758b24
7 years ago
Dariusz Smigiel 6d85a51e2a Removed deprecation warning for converters and validators
Converters and Validators were moved to neutron_lib in change

Change-Id: I37c1e2ee1634a7fb72c401caee9122dadd669a64
7 years ago
Gábor Antal 1628fb72f8 Improved readibility and fixed indentation
In some places, indentation was wrong.
In this patchset, they are fixed.
Also, I did some readibility improvements which I've found
during indentation fixing.

Change-Id: I298b9b0c557c07848c69ed23b0faa3a3147d055d
7 years ago
Dariusz Smigiel dd5976b7ab Accept and return project_id for API calls
Update the API to accept project_id in requests and return
project_id in responses.

For now, the API treats tenant_id and project_id equivalently.
It accepts either or both in requests.
It returns both in responses, depending on filters.

We include an extension to indicate that support for project_id
is enabled in the API.

Completes: blueprint keystone-v3

APIImpact: Describe how the Networking API supports Keystone V3.

Co-Authored-By: Henry Gessau <>
Co-Authored-By: Akihiro Motoki <>

Change-Id: I8775aa8a477191ef21e7c3c6da31d098befefc3c
7 years ago
Kevin Benton 09c87425fa Prepare retry decorator to move to plugin level
Retrying mutating operations at the API layer caused a
couple of problems. First, when components would call
the core plugin using the neutron manager, they would
have to handle the retriable failures on their own or
undo any work they had done so far and allow retriable
failures to be propagated up to the API. Second, retrying
at the API makes composite operations (e.g. auto allocate,
add_router_interface, etc) painful because they have to
consider if exceptions are retriable before raising
fatal exceptions on failures of core plugin calls.

This patch begins the process of moving them down to the
core operations with a new decorator called
'retry_if_session_inactive', which ensures that the
retry logic isn't triggered if there is an ongoing transaction
since retrying inside of a transaction is normally ineffective.
Follow-up patches apply them to various parts of the code-base.

Additionally, the args and kwargs of the method are automatically
deep copied in retries to ensure that any mangling the methods
do to their arguments don't impact their retriability.

Finally, since we are leaving the API decorators in place for now,
the retry logic will not be triggered by another decorator if an
exception has already been retried. This prevents an exponential
explosion of retries on nested retry decorators.

The ultimate goal will be to get rid of the API decorators entirely
so retries are up to each individual plugin.

Partial-Bug: #1596075
Partial-Bug: #1612798
Change-Id: I7b8a4a105aabfa1b5f5dd7a638099007b0933e66
7 years ago
Henry Gessau 99cfd671be debtcollector for globals
The deprecation shim created by Doug turns out to be rather useful.
It emits a warning when an global (attribute of a module) is
referenced but that global has been moved to another module.

This update makes the following changes to the shim:

 - Rename it to _MovedGlobals to better describe what it is a
   debtcollector for.

 - Use inspect to get the original reference and to check that
   _MovedGlobals is called from the last line of a module.

 - Save the old reference automatically in the instance to prevent it
   from getting garbage collected.

 - Beef up the _moved_global() method for moving/renaming individual
   globals, allowing it to move and rename or rename in place.

Change-Id: I868aa4a3129dd05467a103364088efbb86bc5d0f
7 years ago
Jenkins 8eaa1808e5 Merge "Pecan: Handle member actions" 7 years ago