Commit Graph

85 Commits (99f4495c940011293e3cabbb590770dc1e7b6900)

Author SHA1 Message Date
Doug Hellmann 2530ccff8f Upgrade pylint to a version that works with python3
Also fixed some six.moves errors as well.

Change-Id: I9783ee588de1ec4b91a665b9dc89fdba3e2d90df
Signed-off-by: Doug Hellmann <>
5 years ago
Boden R 4bd2f0e8f7 shim neutron rpc with neutron-lib
The common rpc and exceptions were rehomed into
neutron-lib with [1]. This patch shims those rehomed
modules in neutron to switch over to neutron-lib's
versions under the covers.

To do so:
- The rpc and common exceptions are changed to
reference their counterpart in neutron-lib effectively
swapping the impl over to neutron-lib.
- The fake_notifier is removed from neutron and lib's
version is used instead.
- The rpc tests are removed; they live in lib now.
- A few unit test related changes are required
including changing mock.patch to mock.patch.object,
changing the mock checks for a few UTs as they don't
quite work the same with the shim in place.
- Using the RPC fixture from neutron-lib rather than
that setup in neutron's base test class.

With this shim in place, consumers are effectively using
neutron-lib's RPC plumbing and thus we can move consumers
over to neutron-lib's version at will. Once all
consumers are moved over we can come back and remove
the RPC logic from neutron and follow-up with a consumption



Change-Id: I87685be8764a152ac24366f13e190de9d4f6f8d8
5 years ago
Hongbin Lu 2b1d8ea4a2 Implement filter validation
Enforce validation on filter parameters on list requests.
If an API request contains an unknown or unsupported parameter,
the server will return a 400 response instead of silently ignoring
the invalid input.

In resource attributes map, all filter parameters are annotated by
the ``is_filter`` keyword. Attributes with is_filter set to True
are candidates for validation.

Enabling filter validation requires support from core plugin and
all service plugins so each plugin need to indicate if it supports
the validation by setting ``__filter_validation_support`` to True.
If this field is not set, the default is False and validation is
turned off. Right now, the ML2 plugin and all the in-tree service
plugin support filter validation. Out-of-tree plugins will have
filter validation disabled by default.

An API extension is introduced to allow API users to discover this
new API behavior. This feature can be disabled by cloud operators
if they choose to do that. If it is disabled, the extension won't
be presented.

Depends-On: Ic3ab5b3ffdc378d570678b9c967cb42b0c7a8a9b
Depends-On: I4397df1c35463a8b532afdc9c5d28b37224a37b4
Depends-On: I3f2e6e861adaeef81a1a5819a57b28f5c6281d80
Depends-On: I1189bc9a50308df5c7e18c329f3a1262c90b9e12
Depends-On: I057cd917628c77dd20c0ff7747936c3fec7b4844
Depends-On: I0b24a304cc3466a2c05426cdbb6f9d99f1797edd

Change-Id: I21bf8a752813802822fd9966dda6ab3b6c4abfdc
Partial-Bug: #1749820
5 years ago
Hongbin Lu d950949b90 Ensure request's object type is dict
Handle an edge case that API users send a POST/PUT request with
invalid data in request body.

Closes-Bug: #1780327
Change-Id: I3877c18a18ac506dc8f4a9ded2a18b53b9f6cfae
5 years ago
Hongbin Lu f213ba487b Populate project info before using it
Sorting and filtering will rely on the attributes information. It is
necessary to populate project info before using it to sort/filter.

Closes-Bug: #1765452
Change-Id: Ife90268530b6e86a0b0d213e4742a2ef81cb2395
5 years ago
Hongbin Lu a732bbf19e Support filter attribute with empty string
This will enable users to filter list of results with attributes
with empty value. For example, the request below will list
all unbound ports (unbound ports have blank device_id).

  GET "/ports?device_id="

Change-Id: I9001214de126eb888c2425b6a6275f59ec8478e7
Closes-Bug: #1749304
5 years ago
Isaku Yamahata 0063aa33c6 pecan.jsonify v1.3 adjustment
From pecan v1.3, pecan.jsonify uses json.Encoder unconditionally so that
jsonifying mock fails.
Pecan v1.2 uses simplejson.Encoder as encoder which accidentally encodes
MagicMock as {} due to check of '_asdict' attributes.
Since pecan.jsonify uses __json__ magic method when it's defined,
add __json__ method to mocks to return {} as json encode.

Closes-bug: #1754978
Change-Id: Iac955a7de2a1b66b5b56a73f5cc8a4e75150f82e
5 years ago
Boden R 44292f4c9a consume neutron-lib resources attr map
Today we shim the RESOURCE_ATTRIBUTE_MAP in neutron; it references the
equivelant in neutron-lib named RESOURCES.

This patch removes neutron's RESOURCE_ATTRIBUTE_MAP and cleans up
neutron.api.v2.attributes in prep to delete it.
To do so:
- CORE_RESOURCES and RESOURCE_FOREIGN_KEYS are moved to the single
module that references them respectively and the are made private (no
consumers use them).
- get_collection_info is removed and instead the 2 uses in neutron
just use the get() method of the RESOURCES map. There are no external
uses of get_collection_info.
- References using RESOURCE_ATTRIBUTE_MAP are switched over to
neutron-lib's RESOURCES.
- The neutron.api.v2.attributes module is removed as it's empty now.
- A few api attribute UTs are removed; there's nothing to test as per
this patch.


Change-Id: Iaacee584d499c4d33d6d2dd9609c7ac0f2cfc386
5 years ago
Boden R 3f1a9846d2 use callback payloads for REQUEST/RESPONSE events
This patch switches callbacks over to the payload object style events
[1] for BEFORE_RESPONSE and AFTER_REQUEST based notifications. To do
so an APIEventPayload object is used with the publish() method to
pass along the API related data. In addition a few UTs are updated to
work with the changes.



Change-Id: Ibd8559e0db9dcc995abf8937a0cb764b21a18531
6 years ago
Boden R 9e67ba5052 use core resource api defs from lib
The core resource API definitions are in neutron-lib and are already
setup in a RESOURCES map, similar to neutron's global
RESOURCE_ATTRIBUTE_MAP. However, a number of consumers directly use
RESOURCE_ATTRIBUTE_MAP and moreover can perform some interesting
operations on it while mockin" for tests. For that reason this
patch proposes we phase in the use of neutron's RESOURCES map rather
than a rip and replace of RESOURCE_ATTRIBUTE_MAP.

This patch removes the API definitions for the core neutron resources
by updating neutron's global RESOURCE_ATTRIBUTE_MAP to reference lib's
RESOURCE map. In addition the AttributeMapMemento class is removed; it
no longer servers a purpose and neutron-lib's fixture should be used
in its place.

This patch should be safe to merge without worry of impacting consumers.
No one is using AttributeMapMemento [1] and the global map only changed
its reference value.


Change-Id: Ib04fddcbd4465074452b71b16befa4e33b27259e
6 years ago
Boden R 95f1e03446 use plugin constants from neutron-lib
neutron-lib contains a number of the plugin related constants from
neutron.plugins.common.constants. This patch consumes those constants
from neutron-lib and removes them from neutron. In addition the notion
of the dummy plugin service type is moved strictly into the test
package of neutron since it's not a real service plugin.


Change-Id: I767c626f3fe6159ab3abd6a7ae3cb9893b79bf66
6 years ago
Kevin Benton a3dff9769b Delete dead API v2 router code
This code is no longer used now that we have switched to
Pecan and have no option to switch to the legacy API

Change-Id: I0701dfe13024132cf5b9696560b0d027712e7739
6 years ago
Kevin Benton db1058a499 Switch to Pecan for unit tests
This will drop Pecan in place of the old APIRouter for all
unit tests. This will significantly increase the UT coverage
of pecan and will allow us to drop the old API routing logic

The rest of the inline test changes are due to slight plugin loading

Implements: blueprint wsgi-pecan-switch
Change-Id: I76dc23fb7b96d82b0da50285bd0aac76142e81e5
6 years ago
Kevin Benton e2ea0b4652 Drop the web_framework option
This option is no longer needed as we have switched to pecan
and will be deleting the legacy API layer.

Implements: blueprint wsgi-pecan-switch
Change-Id: Ib153d75bb10375e048a8417139873bdf9dca8769
6 years ago
Zhengwei Gao 9e185bca08 Allow unprivileged users to get their quota usage
When user try to get quota usage detail for their own project, it will
return null. As process logic of details method is incorrect, it only
allow admin to get other's project quota usage detail.

Change-Id: I2e21dac497a6c5bffba6b55cb4456820900449df
Closes-Bug: #1708133
6 years ago
Sergey Belous a8109af65f Extend Quota API to report usage statistics
Extend existing quota api to report a quota set. The quota set
will contain a set of resources and its corresponding reservation,
limits and in_use count for each tenant.

DocImpact:Documentation describing the new API as well as the new
information that it exposes.

Co-Authored-By: Prince Boateng<>
Change-Id: Ief2a6a4d2d7085e2a9dcd901123bc4fe6ac7ca22
Related-bug: #1599488
6 years ago
Boden R cbb0970b36 use attribute functions/operations from neutron-lib
neutron-lib now contains the functionality from neutron's attributes
module. This patch moves the local references over to neturon-lib's
implementation and removes the functionality from the attributes


Change-Id: I773702ebd4b66d28059cb3ced64b2b477ceff8af
6 years ago
Boden R 0e2b667bf1 use neutron-lib callbacks
The callback modules have been available in neutron-lib since commit [1]
and are ready for consumption.

As the callback registry is implemented with a singleton manager
instance, sync complications can arise ensuring all consumers switch to
lib's implementation at the same time. Therefore this consumption has
been broken down:
1) Shim neutron's callbacks using lib's callback system and remove
existing neutron internals related to callbacks (devref, UTs, etc.).
2) Switch all neutron's callback imports over to neutron-lib's.
3) Have all sub-projects using callbacks move their imports over to use
neutron-lib's callbacks implementation.
4) Remove the callback shims in neutron-lib once sub-projects are moved
over to lib's callbacks.
5) Follow-on patches moving our existing uses of callbacks to the new
event payload model provided by

This patch implements #2 from above, moving all neutron's callback
imports to use neutron-lib's callbacks.

There are also a few places in the UT code that still patch callbacks,
we can address those in step #4 which may need [2].


[1] fea8bb64ba7ff52632c2bd3e3298eaedf623ee4f
[2] I9966c90e3f90552b41ed84a68b19f3e540426432

Change-Id: I8dae56f0f5c009bdf3e8ebfa1b360756216ab886
6 years ago
fpxie d2976d46d0 Replace six.iteritems with dict.items(Part-1)
according to,
now we should avoid using six.iteritems and replace
it with dict.items.

Change-Id: I8753e80b34c0f86cf70aebc3bcbd3392ee933f62
Partial-Bug: #1680761
6 years ago
Armando Migliaccio ca751a1486 Spin off context module

Partially-implements: blueprint neutron-lib

Change-Id: I48cf45dc1b07035d952152eac2548a3bd9fc2832
6 years ago
tonytan4ever ebc7e1fb2f Pecan: Get loaded by paste deploy
This sets up the factory methods needed to have paste deploy create the
pecan app and return it.  It also changes the legacy factory methods to
conditionally use the pecan factory methods if the web_framework config
option is set to 'pecan'.  This way, all deployments of neutron will not
need to change their api-paste.ini files to get pecan toggled on.  It
should just happen without notice once pecan becomes the default.

Also, by moving this to be loaded by paste deploy, there is a good chunk of
code that has been removed because it is no longer necessary.

Co-Authored-By: Brandon Logan <>
Change-Id: I8b1bbea8d90fdc62715cd8b6738ad955df53d7cd
6 years ago
Cedric Brandily c88debc44f Use addCleanup instead of tearDown
We should use addCleanup instead of tearDown because:
- tearDown is executed only if the test succeeds
- cleanups (defined with addCleanup) are executed even if the test fails

This change removes useless tearDown and transforms remaining ones into

Change-Id: I44bd26dcb5c8456126a35cb807f0bafc772c0ab0
6 years ago
ZhaoBo 22707fb3d2 Add filter check for quering
Neutron rasises 500 error if the filter is the resources'
object property.

This patch add the check to prevent querying with base_db property.

Change-Id: I1aec4c1537e687e9087c468a592802779f370cb3
Closes-Bug: #1654195
6 years ago
Jenkins 85f748a14c Merge "Correctly print --limit value passed via API" 7 years ago
Brian Haley e45d2a904d Correctly print --limit value passed via API
Passing an invalid limit value, for example '--limit -1'
in neutronclient, generates this error from the API:

  Bad limit request: Limit must be an integer 0 or
  greater and not '%d'.

It should print what was passed.

Closes-bug: #1649665

Change-Id: I9b8bd67d344cd91e008f7c897f1ea6a020a9c30b
7 years ago
Ihar Hrachyshka 61eb74af31 Remove allow_pagination and allow_sorting config options
They were deprecated in Newton. This patch cleans them up.

Note: it does not mean that the features will not work anymore. On the
contrary, now API will consistently sort and paginate for plugins that
honour the relevant sorting/pagination parameters.

Note2: base resource controller still allows to pass
allow_pagination=False and allow_sorting=False parameters to disable the
features if a registered plugin does not support the features yet.

Change-Id: I5fd30b20f645846d9366740372c4815c4b33e2eb
Related-Bug: #1566514
7 years ago
Henry Gessau 4151486bd4 Remove REVERSED_PLURALS and get_resource_info()
By changing pecan to store plugins by collection (plural) instead of
resource (singular) we don't need REVERSED_PLURALS and

These are not used outside of neutron core.

Consequently, this exposed a bug with some controllers not honoring an
extension's intention of disallowing pagination and sorting.  This is
also fixed in this patch to get successful test runs.

Co-Authored-By: Brandon Logan <>

Change-Id: I9d529d0a2ad369e2be0a8df3c6f06a6532e8b13d
7 years ago
Armando Migliaccio 17563a802e Adopt neutron-lib plugin directory
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.

Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.

This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.

Usage pattern [1,2] can be translated to [3,4] respectively.

[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()

The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.

Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.

Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.


Partially-implements: blueprint neutron-lib

Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
7 years ago
Dariusz Smigiel dd5976b7ab Accept and return project_id for API calls
Update the API to accept project_id in requests and return
project_id in responses.

For now, the API treats tenant_id and project_id equivalently.
It accepts either or both in requests.
It returns both in responses, depending on filters.

We include an extension to indicate that support for project_id
is enabled in the API.

Completes: blueprint keystone-v3

APIImpact: Describe how the Networking API supports Keystone V3.

Co-Authored-By: Henry Gessau <>
Co-Authored-By: Akihiro Motoki <>

Change-Id: I8775aa8a477191ef21e7c3c6da31d098befefc3c
7 years ago
Jenkins 7e450dc9f4 Merge "Use from_dict to load context params" 7 years ago
Sławek Kapłoński 38b1b47fe7 Fix internal server error during updating QoS rule
QoS rule can be edited or created only by admin users. Because
rule object don't have tenant_id attribute there was error
in checking tenant_id of object and neutron server got internal
server error.

Now if updating object doesn't have tenant_id at all (like in
case of QoS rules), 404 error will be returned to the user.

New API tests for checking this case are added for QoS.

Change-Id: Ia82ad84a3a07df4df8eaeaed6c47d31be9493cbd
Closes-Bug: #1515564
7 years ago
hobo.kengo e99274397c Check content type by completely match instead of partial match
Currently, 'get_content_type' method in neutron/ checks
specified format by 'in' statement.
Thus, string(e.g. 'application/j') that is partial matched to
'application/json' is returned.

However, we cannot find valid serializer from the format,
and request fails with unexpected error.

Change-Id: I75aac0308d0fd0321973c4bda3bc07ca0224e1c1
Closes-Bug: #1612485
7 years ago
Kumar Acharya e92b68dd82 delete event payload
This fix will allow the delete event to have the required data
in the notification payload.

Change-Id: I57a001ca2fddc2a750026e7da7980bfd8e5aab40
Closes-Bug: 1593127
7 years ago
Jamie Lennox 8ad224d4e6 Use from_dict to load context params
The context objects accepts and ignores unknown keyword arguments. This
was to allow it to handle the deserialization of parameters it didn't
understand from the base class' to_dict method. This will make it
difficult to pass unknown attributes to the base class so fix the
from_dict method to only accept params it knows about.

Related-Bug: #1602081
Change-Id: Ic58a2025680e8e1ba4f8a177d898be457e2c3160
7 years ago
Ihar Hrachyshka f3816cb8bd ml2: postpone exception logs to when retry mechanism fails to recover
Since Ia2d911a6a90b1baba1d5cc36f7c625e156a2bc33, we use version_id_col
SQLAlchemy feature to routinely bump revision numbers for resources.  By
doing so, SQLAlchemy also enforces the expected number on any UPDATE and
DELETE, which may not be valid when the transaction is actually applied.
In that case, the library will raise StaleDataError:

The exception is then logged by ml2 and bubbles up to API layer where
retry mechanism will correctly catch it and issue another attempt.

If API layer does not retry on exception, it already logs the error,
including the traceback.

In ml2, it's too early to decide if an exception is worth being logged.
Plugin instead should just silently allow all unexpected exceptions to
bubble up and be dealt by API layer.

At the same time, there are some details that are known at the plugin
level only, that are not easily deducible from the API request details.
That's why we save details about the error on the exception object that
bubbles up into API layer, where we are ready to decide if those details
are worth being logged.

Change-Id: I848df0aef5381e50dfb58e46d7a652113ac27a49
Closes-Bug: #1593719
7 years ago
Armando Migliaccio 877778ee4c Move DHCP notification logic out of API controller
Bug 1591766 unveiled an issue where calling the plugin API does not trigger
DHCP notifications. This is required by the auto-allocated-topology service
plugin that calls core_plugin.update_network(), and expect notifications
to be sent out on state changes. To accomplish this, the logic has been
encapsulated in the DHCP module, and leveraged via callback mechanisms.

For this reason, new events have been introduced, AFTER_REQUEST, and
BEFORE_RESPONSE. The latter in particular is the one needed to hook up
dhcp notifications in order to preserve backward compatibility.

More precisely, core plugins that use DHCP as is or implement their own,
(with or without an agent) should already instantiate their own notifier,
and if they do not, this should be rectified.

A search on reveals that out-of-tree plugins
already specify their own notifiers, and the default initialization is
clearly redundant now.

Related-bug: #1591766

Change-Id: I7440becb6d30af7159ecaeba09d7a28eceb71bea
7 years ago
Henry Gessau 78fff41ee3 Use converters and validators from neutron-lib
Related-Blueprint: neutron-lib

Change-Id: I6b9079e9e703c6fd75adbed3846e7257685433e8
7 years ago
Henry Gessau 4148a347b3 Use constants from neutron-lib
With this we enable the deprecation warnings by default.

Related-Blueprint: neutron-lib

Change-Id: I5b9e53751dd164010e5bbeb15f534ac0fe2a5105
7 years ago
Henry Gessau ae5bad49cc Use exceptions from neutron-lib
Related-Blueprint: neutron-lib

Change-Id: Ia014468bd621c4ee6aea95bf19328c61070174c4
7 years ago
Hirofumi Ichihara 0ae3c172ae Make API framework more flexible for various extensions
This patch adds a function to Neutron extension mechanism
so that we can add API methods to "/resources" path rather than
"/resources/action" path.

This patch also allow extentions to control correspondences
between action and status in API request.

Change-Id: I862086f7528583e65d7bee794f011ddff6ae8901
Partial-Implements: blueprint add-tags-to-core-resources
Related-Bug: #1489291
7 years ago
Jenkins 25ab9ee30b Merge "Add hacking check for assertEqual HTTP code" 7 years ago
Jenkins 1efe482358 Merge "Protect 'show' and 'index' with Retry decorator" 7 years ago
lzklibj 0f50d8c06f Add hacking check for assertEqual HTTP code
Add a hacking check for wrong usage, like:
  assertEqual(observed.code, webob.exc.HTTP**.code)
the correct usage should be:
  assertEqual(webob.exc.HTTP**.code, observed.code)

Change-Id: I8c038bae61c528d30af43e904a9bb98ac4cd4d3a
7 years ago
Kevin Benton 2a27361cf5 Protect 'show' and 'index' with Retry decorator
Commit 77de9653fd added a RetryRequest
exception to the policy engine for when items disappeared during policy
enforcement lookups. However, the API was not catching them for the
show and list operations.

This patch adds the decorators to the two methods to catch any
retry exception that may be emitted from the policy engine or
wherever else.

Closes-Bug: #1528031
Change-Id: If4aea5245cdbb2ea545e9a96d73386e3c21a3696
7 years ago
Yatin Kumbhare 6b4be76e14 Fix params order in assertEqual
Fix params order to correspond to real signature:
assertEqual(expected, actual)

Change-Id: I722b998f6eae47076f3d10213073296a0a9a2081
Closes-Bug: #1277104
7 years ago
Bhagyashri Shewale 88e899f7a0 Fix module's import order
Made corrections in import order for built-in, third party and
project specific modules as per OpenStack import standards [1].


Change-Id: I899deefd6ee4732d6c0afd17a5afbe42b0fa37ba
7 years ago
Salvatore Orlando 0f9a607cf6 Pecan: Streamline request body processing
This patch simplifies the process for validating the request body for
POST and PUT requests and preparing it for dispatching to plugins.

In particular it removes the need for parsing the URL to find the
resource identifier, and instead leverages Pecan's routing engine
to extract it. As a result the attribute population hook now simply
deals with request body validation, and has been renamed accordingly.

The logic for loading the current state of the object from the plugin
in order to enforce authZ policy on PUT requests has been moved to
the appropriate hook, PolicyEnforcementHook.

The logic for managing plural/singular names for resource has also been
improved, and two helper functions to retrieve a resource's attributes
have been added to neutron.api.v2.attributes

The logic for aborting requests with unsupported HTTP methods has also
been moved to the REST controllers. It used to be in PolicyEnforcementHook,
which was probably not the right thing to do.

This patch also unskips a functional test concerning ownership checks,
and add functional tests for verifying correct request processing.
Unit tests for the newly added helper functions are also provided.

Related blueprint wsgi-pecan-switch

Change-Id: Ib26998b37bdeec8af7a97f77b66d421b8cd271da
7 years ago
Jenkins 5a299de6ca Merge "Unify assertEqual for empty usages" 7 years ago
lzklibj 3491cbc0c5 Unify assertEqual for empty usages
Update previous assertEqual(observed, *empty) usages to
assertEqual(*empty*, observed).

This patch aslo update hacking check for assertEqual with
empty types.

Change-Id: I981277618f92254a5beb9d3308a317d8c14e125c
7 years ago
Mathieu Rohon 5d53dfb8d6 Avoid duplicating tenant check when creating resources
The check of the tenant done in the method _get_tenant_id_for_create()
is already did by the Neutron Controller in prepare_request_body(),
with a call to attributes.populate_tenant_id().
Moreover, when the Controller processes a "create" requests, it
will add the 'tenant_id' to the resource dict.
Thus, _get_tenant_id_for_create() can be deleted.
Calls to this method are replaced by the res['tenant_id'].

Changes have to be done in UT to explicitly add the tenant_id while
creating resources, since the UT framework is bypassing the controller code
that automatically adds the tenant_id to the resource.

Co-Authored-By: Hong Hui Xiao <>
Closes-Bug: #1513825
Change-Id: Icea06dc81344e1120bdf986a97a6b1094bbb765e
Depends-On: I31022e9230fc5404c6a94edabbb08d2b079c3a09
Depends-On: Iea3f014ef17a1e1b755cd2efe99afd1a36ebbc6a
Depends-On: I604602d023e0cbf7f6591149f914d73217d7a574
8 years ago