The common rpc and exceptions were rehomed into
neutron-lib with [1]. This patch shims those rehomed
modules in neutron to switch over to neutron-lib's
versions under the covers.
To do so:
- The rpc and common exceptions are changed to
reference their counterpart in neutron-lib effectively
swapping the impl over to neutron-lib.
- The fake_notifier is removed from neutron and lib's
version is used instead.
- The rpc tests are removed; they live in lib now.
- A few unit test related changes are required
including changing mock.patch to mock.patch.object,
changing the mock checks for a few UTs as they don't
quite work the same with the shim in place.
- Using the RPC fixture from neutron-lib rather than
that setup in neutron's base test class.
With this shim in place, consumers are effectively using
neutron-lib's RPC plumbing and thus we can move consumers
over to neutron-lib's version at will. Once all
consumers are moved over we can come back and remove
the RPC logic from neutron and follow-up with a consumption
patch.
NeutronLibImpact
[1] https://review.openstack.org/#/c/319328/
Change-Id: I87685be8764a152ac24366f13e190de9d4f6f8d8
Enforce validation on filter parameters on list requests.
If an API request contains an unknown or unsupported parameter,
the server will return a 400 response instead of silently ignoring
the invalid input.
In resource attributes map, all filter parameters are annotated by
the ``is_filter`` keyword. Attributes with is_filter set to True
are candidates for validation.
Enabling filter validation requires support from core plugin and
all service plugins so each plugin need to indicate if it supports
the validation by setting ``__filter_validation_support`` to True.
If this field is not set, the default is False and validation is
turned off. Right now, the ML2 plugin and all the in-tree service
plugin support filter validation. Out-of-tree plugins will have
filter validation disabled by default.
An API extension is introduced to allow API users to discover this
new API behavior. This feature can be disabled by cloud operators
if they choose to do that. If it is disabled, the extension won't
be presented.
Depends-On: Ic3ab5b3ffdc378d570678b9c967cb42b0c7a8a9b
Depends-On: I4397df1c35463a8b532afdc9c5d28b37224a37b4
Depends-On: I3f2e6e861adaeef81a1a5819a57b28f5c6281d80
Depends-On: I1189bc9a50308df5c7e18c329f3a1262c90b9e12
Depends-On: I057cd917628c77dd20c0ff7747936c3fec7b4844
Depends-On: I0b24a304cc3466a2c05426cdbb6f9d99f1797edd
Change-Id: I21bf8a752813802822fd9966dda6ab3b6c4abfdc
Partial-Bug: #1749820
Handle an edge case that API users send a POST/PUT request with
invalid data in request body.
Closes-Bug: #1780327
Change-Id: I3877c18a18ac506dc8f4a9ded2a18b53b9f6cfae
Sorting and filtering will rely on the attributes information. It is
necessary to populate project info before using it to sort/filter.
Closes-Bug: #1765452
Change-Id: Ife90268530b6e86a0b0d213e4742a2ef81cb2395
This will enable users to filter list of results with attributes
with empty value. For example, the request below will list
all unbound ports (unbound ports have blank device_id).
GET "/ports?device_id="
APIImpact
Change-Id: I9001214de126eb888c2425b6a6275f59ec8478e7
Closes-Bug: #1749304
From pecan v1.3, pecan.jsonify uses json.Encoder unconditionally so that
jsonifying mock fails.
Pecan v1.2 uses simplejson.Encoder as encoder which accidentally encodes
MagicMock as {} due to check of '_asdict' attributes.
Since pecan.jsonify uses __json__ magic method when it's defined,
add __json__ method to mocks to return {} as json encode.
Closes-bug: #1754978
Change-Id: Iac955a7de2a1b66b5b56a73f5cc8a4e75150f82e
Today we shim the RESOURCE_ATTRIBUTE_MAP in neutron; it references the
equivelant in neutron-lib named RESOURCES.
This patch removes neutron's RESOURCE_ATTRIBUTE_MAP and cleans up
neutron.api.v2.attributes in prep to delete it.
To do so:
- CORE_RESOURCES and RESOURCE_FOREIGN_KEYS are moved to the single
module that references them respectively and the are made private (no
consumers use them).
- get_collection_info is removed and instead the 2 uses in neutron
just use the get() method of the RESOURCES map. There are no external
uses of get_collection_info.
- References using RESOURCE_ATTRIBUTE_MAP are switched over to
neutron-lib's RESOURCES.
- The neutron.api.v2.attributes module is removed as it's empty now.
- A few api attribute UTs are removed; there's nothing to test as per
this patch.
NeutronLibImpact
Change-Id: Iaacee584d499c4d33d6d2dd9609c7ac0f2cfc386
This patch switches callbacks over to the payload object style events
[1] for BEFORE_RESPONSE and AFTER_REQUEST based notifications. To do
so an APIEventPayload object is used with the publish() method to
pass along the API related data. In addition a few UTs are updated to
work with the changes.
NeutronLibImpact
[1] https://docs.openstack.org/neutron-lib/latest/contributor/callbacks.html#event-payloads
Change-Id: Ibd8559e0db9dcc995abf8937a0cb764b21a18531
The core resource API definitions are in neutron-lib and are already
setup in a RESOURCES map, similar to neutron's global
RESOURCE_ATTRIBUTE_MAP. However, a number of consumers directly use
RESOURCE_ATTRIBUTE_MAP and moreover can perform some interesting
operations on it while mockin" for tests. For that reason this
patch proposes we phase in the use of neutron's RESOURCES map rather
than a rip and replace of RESOURCE_ATTRIBUTE_MAP.
This patch removes the API definitions for the core neutron resources
by updating neutron's global RESOURCE_ATTRIBUTE_MAP to reference lib's
RESOURCE map. In addition the AttributeMapMemento class is removed; it
no longer servers a purpose and neutron-lib's fixture should be used
in its place.
This patch should be safe to merge without worry of impacting consumers.
No one is using AttributeMapMemento [1] and the global map only changed
its reference value.
[1] http://codesearch.openstack.org/?q=AttributeMapMemento
Change-Id: Ib04fddcbd4465074452b71b16befa4e33b27259e
neutron-lib contains a number of the plugin related constants from
neutron.plugins.common.constants. This patch consumes those constants
from neutron-lib and removes them from neutron. In addition the notion
of the dummy plugin service type is moved strictly into the test
package of neutron since it's not a real service plugin.
NeutronLibImpact
Change-Id: I767c626f3fe6159ab3abd6a7ae3cb9893b79bf66
This code is no longer used now that we have switched to
Pecan and have no option to switch to the legacy API
framework.
Change-Id: I0701dfe13024132cf5b9696560b0d027712e7739
This will drop Pecan in place of the old APIRouter for all
unit tests. This will significantly increase the UT coverage
of pecan and will allow us to drop the old API routing logic
completely.
The rest of the inline test changes are due to slight plugin loading
differences.
Implements: blueprint wsgi-pecan-switch
Change-Id: I76dc23fb7b96d82b0da50285bd0aac76142e81e5
This option is no longer needed as we have switched to pecan
and will be deleting the legacy API layer.
Implements: blueprint wsgi-pecan-switch
Change-Id: Ib153d75bb10375e048a8417139873bdf9dca8769
When user try to get quota usage detail for their own project, it will
return null. As process logic of details method is incorrect, it only
allow admin to get other's project quota usage detail.
Change-Id: I2e21dac497a6c5bffba6b55cb4456820900449df
Closes-Bug: #1708133
Extend existing quota api to report a quota set. The quota set
will contain a set of resources and its corresponding reservation,
limits and in_use count for each tenant.
DocImpact:Documentation describing the new API as well as the new
information that it exposes.
APIImpact
Co-Authored-By: Prince Boateng<prince.a.owusu.boateng@intel.com>
Change-Id: Ief2a6a4d2d7085e2a9dcd901123bc4fe6ac7ca22
Related-bug: #1599488
neutron-lib now contains the functionality from neutron's attributes
module. This patch moves the local references over to neturon-lib's
implementation and removes the functionality from the attributes
module.
NeutronLibImpact
Change-Id: I773702ebd4b66d28059cb3ced64b2b477ceff8af
The callback modules have been available in neutron-lib since commit [1]
and are ready for consumption.
As the callback registry is implemented with a singleton manager
instance, sync complications can arise ensuring all consumers switch to
lib's implementation at the same time. Therefore this consumption has
been broken down:
1) Shim neutron's callbacks using lib's callback system and remove
existing neutron internals related to callbacks (devref, UTs, etc.).
2) Switch all neutron's callback imports over to neutron-lib's.
3) Have all sub-projects using callbacks move their imports over to use
neutron-lib's callbacks implementation.
4) Remove the callback shims in neutron-lib once sub-projects are moved
over to lib's callbacks.
5) Follow-on patches moving our existing uses of callbacks to the new
event payload model provided by neutron-lib.callback.events
This patch implements #2 from above, moving all neutron's callback
imports to use neutron-lib's callbacks.
There are also a few places in the UT code that still patch callbacks,
we can address those in step #4 which may need [2].
NeutronLibImpact
[1] fea8bb64ba7ff52632c2bd3e3298eaedf623ee4f
[2] I9966c90e3f90552b41ed84a68b19f3e540426432
Change-Id: I8dae56f0f5c009bdf3e8ebfa1b360756216ab886
according to https://wiki.openstack.org/wiki/Python3,
now we should avoid using six.iteritems and replace
it with dict.items.
Change-Id: I8753e80b34c0f86cf70aebc3bcbd3392ee933f62
Partial-Bug: #1680761
This sets up the factory methods needed to have paste deploy create the
pecan app and return it. It also changes the legacy factory methods to
conditionally use the pecan factory methods if the web_framework config
option is set to 'pecan'. This way, all deployments of neutron will not
need to change their api-paste.ini files to get pecan toggled on. It
should just happen without notice once pecan becomes the default.
Also, by moving this to be loaded by paste deploy, there is a good chunk of
code that has been removed because it is no longer necessary.
Co-Authored-By: Brandon Logan <brandon.logan@rackspace.com>
Change-Id: I8b1bbea8d90fdc62715cd8b6738ad955df53d7cd
We should use addCleanup instead of tearDown because:
- tearDown is executed only if the test succeeds
- cleanups (defined with addCleanup) are executed even if the test fails
This change removes useless tearDown and transforms remaining ones into
addCleanup.
Change-Id: I44bd26dcb5c8456126a35cb807f0bafc772c0ab0
Neutron rasises 500 error if the filter is the resources'
object property.
This patch add the check to prevent querying with base_db property.
Change-Id: I1aec4c1537e687e9087c468a592802779f370cb3
Closes-Bug: #1654195
Passing an invalid limit value, for example '--limit -1'
in neutronclient, generates this error from the API:
Bad limit request: Limit must be an integer 0 or
greater and not '%d'.
It should print what was passed.
Closes-bug: #1649665
Change-Id: I9b8bd67d344cd91e008f7c897f1ea6a020a9c30b
They were deprecated in Newton. This patch cleans them up.
Note: it does not mean that the features will not work anymore. On the
contrary, now API will consistently sort and paginate for plugins that
honour the relevant sorting/pagination parameters.
Note2: base resource controller still allows to pass
allow_pagination=False and allow_sorting=False parameters to disable the
features if a registered plugin does not support the features yet.
Change-Id: I5fd30b20f645846d9366740372c4815c4b33e2eb
Related-Bug: #1566514
By changing pecan to store plugins by collection (plural) instead of
resource (singular) we don't need REVERSED_PLURALS and
get_resource_info().
These are not used outside of neutron core.
Consequently, this exposed a bug with some controllers not honoring an
extension's intention of disallowing pagination and sorting. This is
also fixed in this patch to get successful test runs.
Co-Authored-By: Brandon Logan <brandon.logan@rackspace.com>
Change-Id: I9d529d0a2ad369e2be0a8df3c6f06a6532e8b13d
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.
Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.
This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.
Usage pattern [1,2] can be translated to [3,4] respectively.
[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()
The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.
Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.
Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.
NeutronLibImpact
Partially-implements: blueprint neutron-lib
Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
Update the API to accept project_id in requests and return
project_id in responses.
For now, the API treats tenant_id and project_id equivalently.
It accepts either or both in requests.
It returns both in responses, depending on filters.
We include an extension to indicate that support for project_id
is enabled in the API.
Completes: blueprint keystone-v3
APIImpact: Describe how the Networking API supports Keystone V3.
Co-Authored-By: Henry Gessau <HenryG@gessau.net>
Co-Authored-By: Akihiro Motoki <amotoki@gmail.com>
Change-Id: I8775aa8a477191ef21e7c3c6da31d098befefc3c
QoS rule can be edited or created only by admin users. Because
rule object don't have tenant_id attribute there was error
in checking tenant_id of object and neutron server got internal
server error.
Now if updating object doesn't have tenant_id at all (like in
case of QoS rules), 404 error will be returned to the user.
New API tests for checking this case are added for QoS.
Change-Id: Ia82ad84a3a07df4df8eaeaed6c47d31be9493cbd
Closes-Bug: #1515564
Currently, 'get_content_type' method in neutron/wsgi.py checks
specified format by 'in' statement.
Thus, string(e.g. 'application/j') that is partial matched to
'application/json' is returned.
However, we cannot find valid serializer from the format,
and request fails with unexpected error.
Change-Id: I75aac0308d0fd0321973c4bda3bc07ca0224e1c1
Closes-Bug: #1612485
This fix will allow the delete event to have the required data
in the notification payload.
Change-Id: I57a001ca2fddc2a750026e7da7980bfd8e5aab40
Closes-Bug: 1593127
The context objects accepts and ignores unknown keyword arguments. This
was to allow it to handle the deserialization of parameters it didn't
understand from the base class' to_dict method. This will make it
difficult to pass unknown attributes to the base class so fix the
from_dict method to only accept params it knows about.
Related-Bug: #1602081
Change-Id: Ic58a2025680e8e1ba4f8a177d898be457e2c3160
Since Ia2d911a6a90b1baba1d5cc36f7c625e156a2bc33, we use version_id_col
SQLAlchemy feature to routinely bump revision numbers for resources. By
doing so, SQLAlchemy also enforces the expected number on any UPDATE and
DELETE, which may not be valid when the transaction is actually applied.
In that case, the library will raise StaleDataError:
http://docs.sqlalchemy.org/en/latest/orm/exceptions.html#sqlalchemy.orm.exc.StaleDataError
The exception is then logged by ml2 and bubbles up to API layer where
retry mechanism will correctly catch it and issue another attempt.
If API layer does not retry on exception, it already logs the error,
including the traceback.
In ml2, it's too early to decide if an exception is worth being logged.
Plugin instead should just silently allow all unexpected exceptions to
bubble up and be dealt by API layer.
At the same time, there are some details that are known at the plugin
level only, that are not easily deducible from the API request details.
That's why we save details about the error on the exception object that
bubbles up into API layer, where we are ready to decide if those details
are worth being logged.
Change-Id: I848df0aef5381e50dfb58e46d7a652113ac27a49
Closes-Bug: #1593719
Bug 1591766 unveiled an issue where calling the plugin API does not trigger
DHCP notifications. This is required by the auto-allocated-topology service
plugin that calls core_plugin.update_network(), and expect notifications
to be sent out on state changes. To accomplish this, the logic has been
encapsulated in the DHCP module, and leveraged via callback mechanisms.
For this reason, new events have been introduced, AFTER_REQUEST, and
BEFORE_RESPONSE. The latter in particular is the one needed to hook up
dhcp notifications in order to preserve backward compatibility.
More precisely, core plugins that use DHCP as is or implement their own,
(with or without an agent) should already instantiate their own notifier,
and if they do not, this should be rectified.
A search on codesearch.openstack.org reveals that out-of-tree plugins
already specify their own notifiers, and the default initialization is
clearly redundant now.
Related-bug: #1591766
Change-Id: I7440becb6d30af7159ecaeba09d7a28eceb71bea
This patch adds a function to Neutron extension mechanism
so that we can add API methods to "/resources" path rather than
"/resources/action" path.
This patch also allow extentions to control correspondences
between action and status in API request.
Change-Id: I862086f7528583e65d7bee794f011ddff6ae8901
Partial-Implements: blueprint add-tags-to-core-resources
Related-Bug: #1489291
Add a hacking check for wrong usage, like:
assertEqual(observed.code, webob.exc.HTTP**.code)
the correct usage should be:
assertEqual(webob.exc.HTTP**.code, observed.code)
Change-Id: I8c038bae61c528d30af43e904a9bb98ac4cd4d3a
Commit 77de9653fd added a RetryRequest
exception to the policy engine for when items disappeared during policy
enforcement lookups. However, the API was not catching them for the
show and list operations.
This patch adds the decorators to the two methods to catch any
retry exception that may be emitted from the policy engine or
wherever else.
Closes-Bug: #1528031
Change-Id: If4aea5245cdbb2ea545e9a96d73386e3c21a3696
Fix params order to correspond to real signature:
assertEqual(expected, actual)
Change-Id: I722b998f6eae47076f3d10213073296a0a9a2081
Closes-Bug: #1277104
This patch simplifies the process for validating the request body for
POST and PUT requests and preparing it for dispatching to plugins.
In particular it removes the need for parsing the URL to find the
resource identifier, and instead leverages Pecan's routing engine
to extract it. As a result the attribute population hook now simply
deals with request body validation, and has been renamed accordingly.
The logic for loading the current state of the object from the plugin
in order to enforce authZ policy on PUT requests has been moved to
the appropriate hook, PolicyEnforcementHook.
The logic for managing plural/singular names for resource has also been
improved, and two helper functions to retrieve a resource's attributes
have been added to neutron.api.v2.attributes
The logic for aborting requests with unsupported HTTP methods has also
been moved to the REST controllers. It used to be in PolicyEnforcementHook,
which was probably not the right thing to do.
This patch also unskips a functional test concerning ownership checks,
and add functional tests for verifying correct request processing.
Unit tests for the newly added helper functions are also provided.
Related blueprint wsgi-pecan-switch
Change-Id: Ib26998b37bdeec8af7a97f77b66d421b8cd271da
The check of the tenant done in the method _get_tenant_id_for_create()
is already did by the Neutron Controller in prepare_request_body(),
with a call to attributes.populate_tenant_id().
Moreover, when the Controller processes a "create" requests, it
will add the 'tenant_id' to the resource dict.
Thus, _get_tenant_id_for_create() can be deleted.
Calls to this method are replaced by the res['tenant_id'].
Changes have to be done in UT to explicitly add the tenant_id while
creating resources, since the UT framework is bypassing the controller code
that automatically adds the tenant_id to the resource.
Co-Authored-By: Hong Hui Xiao <xiaohhui@cn.ibm.com>
Closes-Bug: #1513825
Change-Id: Icea06dc81344e1120bdf986a97a6b1094bbb765e
Depends-On: I31022e9230fc5404c6a94edabbb08d2b079c3a09
Depends-On: Iea3f014ef17a1e1b755cd2efe99afd1a36ebbc6a
Depends-On: I604602d023e0cbf7f6591149f914d73217d7a574
Doug Hellmann