231 Commits

Author SHA1 Message Date
Julia Kreger
c2df29e52f Phase 2 - SQLAlchemy 2.0 Compatability
* Changed common exception imports from SQLAlchemy for ORM
  query types which are now originated from the main exception
  definition set.

* Changed base join option usage to use objects instead of labels,
  and defaulted all multi-row result sets to return data using
  "selectinload" as opposed to operating with a join query to
  avoid need to de-duplicate all result sets.

* Changed DeployTemplates to utilize objects instead of
  field names for queries, and updated the associated join ORM
  model's relationship record between DeployTemplate and
  DeployTemplateSteps.

* Changed Ports, Chassis, Conductor, Volume Target/Connector
  queries to lean towards use of select/update/delete queries as
  opposed to ORM queries. Most of these changes revolved around
  references of field names as opposed to model objects.

* This change also labels a few lines as "noqa", which is a result
  of the style check rules getting triggered on statements as
  needed for SQLAlchemy.

Change-Id: I651ec4b50c79be6aa8c798ee27957ed720a578d8
2022-10-14 07:58:27 -07:00
Julia Kreger
49e085583d Phase 1 - SQLAlchemy 2.0 Compatability
One of the major changes in SQLAlchemy 2.0 is the removal
of autocommit support. It turns out Ironic was using this quite
aggressively without even really being aware of it.

* Moved the declaritive_base to ORM, as noted in the SQLAlchemy 2.0
  changes[0].

* Console testing caused us to become aware of issues around locking
  where session synchronization, when autocommit was enabled, was
  defaulted to False. The result of this is that you could have two
  sessions have different results, which could results on different
  threads, and where one could still attempt to lock based upon prior
  information. Inherently, while this basically worked, it was
  also sort of broken behavior. This resulted in locking being
  rewritten to use the style mandated in SQLAlchemy 2.0 migration
  documentation. This ultimately is due to locking, which is *heavily*
  relied upon in Ironic, and in unit testing with sqlite, there are
  no transactions, which means we can get some data inconsistency
  in unit testing as well if we're reliant upon the database to
  precisely and exactly return what we committed.[1]

* Begins changing the query.one()/query.all() style to use explicit
  select statements as part of the new style mandated for migration
  to SQLAlchemy 2.0.

* Instead of using field label strings for joined queries, use the
  object format, which makes much more sense now, and is part of
  the items required for eventual migration to 2.0.

* DB queries involving Traits are now loaded using SelectInLoad
  as opposed to Joins. The now deprecated ORM queries were quietly
  and silently de-duplicating rows and providing consistent sets
  from the resulting joined table responses, however putting much
  higher CPU load on the processing of results on the client.
  Prior performance testing has informed us this should be a minimal
  overhead impact, however these queries should no longer be in
  transactions with the Database Servers which should offset the
  shift in load pattern. The reason we cannot continue to deduplicate
  locally in our code is because we carry Dict data sets which cannot
  be hashed for deduplication. Most projects have handled this by
  treating them as Text and then converting, but without a massive
  rewrite, this seems to be the viable middle ground.

* Adds an explict mapping for traits and tags on the Node object
  to point directly to the NodeTrait and NodeTag classes. This
  superceeds the prior usage of a backref to make the association.

* Splits SQLAlchemy class model Node into Node and NodeBase, which
  allows for high performance queries to skip querying for ``tags``
  and ``traits``. Otherwise with the afrormentioned lookups would
  always execute as they are now properties as well on the Node
  class. This more common of a SQLAlchemy model, but Ironic's model
  has been a bit more rigid to date.

* Adds a ``start_consoles`` and ``start_allocations`` option to the
  conductor ``init_host`` method. This allows unit tests to be
  executed and launched with the service context, while *not* also
  creating race conditions which resulted in failed tests.

* The db API ``_paginate_query`` wrapper now contains additional
  logic to handle traditional ORM query responses and the newer style
  of unified query responses. Due to differences in queries and handling,
  which also was part of the driver for the creation of ``NodeBase``,
  as SQLAlchemy will only create an object if a base object is referenced.
  Also, by default, everything returned is a tuple in 1.4 with the
  unified interface.

* Also modified one unit test which counted time.sleep calls, which is
  a known pattern which can create failures which are ultimately noise.

Ultimately, I have labelled the remaining places which SQLAlchemy
warnings are raised at for deprecation/removal of functionality,
which needs to be addressed.

[0] https://docs.sqlalchemy.org/en/14/changelog/migration_20.html
[1] https://docs.sqlalchemy.org/en/14/dialects/sqlite.html#transaction-isolation-level-autocommit

Change-Id: Ie0f4b8a814eaef1e852088d12d33ce1eab408e23
2022-10-13 21:21:40 +00:00
Stephen Finucane
25b3e6796d tests: Add a WarningsFixture
We will use this in the future to prepare for SQLAlchemy 2.0. For now,
we're simply using it to filter out some of the more annoying warnings
and to highlight general SQLAlchemy issues we need to address.

Change-Id: I7c26c20e4b36c4f3b98873939677b966ec6186a5
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2022-09-08 10:58:42 +01:00
Ghanshyam Mann
2e94aa4241 Drop lower-constraints.txt and its testing
As discussed in TC PTG[1] and TC resolution[2], we are
dropping the lower-constraints.txt file and its testing.
We will keep lower bounds in the requirements.txt file but
with a note that these are not tested lower bounds and we
try our best to keep them updated.

[1] https://etherpad.opendev.org/p/tc-zed-ptg#L326
[2] https://governance.openstack.org/tc/resolutions/20220414-drop-lower-constraints.html#proposal

Change-Id: Ide6b3e9d6c1171f1ae568c256a3e9affa45de17b
2022-04-30 17:58:31 -05:00
Zuul
e5a1997df8 Merge "Create API documentation from docstrings" 2022-03-17 15:35:42 +00:00
Mahnoor Asghar
3e631a5931 Create API documentation from docstrings
Create a new Sphinx extension called 'web_api_docstring' to process
docstrings from the API classes, in order to generate API
documentation.

Story: 2009785
Task: 44291
Change-Id: Ia6b2b3741e2b1cbd29531c21795df4f0f0dc70ca
2022-03-17 01:22:44 +05:00
Steve Baker
b8ef35a984 Ensure tox.ini is ASCII
The left/right single quotation marks cause an ascii decode error in
some python3.6 environments.

Change-Id: I80d1b3e7f9e2e23ddaca640714a342b63ae3a5fa
2022-02-18 12:50:30 +13:00
Riccardo Pittau
47b98d44c5 Use default test dir
The test dir is already defined in .stestr.conf

Change-Id: I0d620ac5e9f7eeaae86243a9235f431a9114804b
2021-11-19 10:39:41 +01:00
Riccardo Pittau
e5d35ba65c Add lower-constraints job to current development branch
The lower-constraints test was removed becuase of an issue where pip
could not correctly determine the required packages versions to install,
ending in an almost infinite loop that would end up in timeout, failure,
and general mayhem.
Recently the issue has been fixed and, if properly configured, the
lower-constraints test can provide good indication of which minimum
versions are required to support the current code.
This patch adds the test back to the current development branch.
The long term goal is to keep the lower-constraints file in the stable
branches, but remove the test job to avoid issues during backports.

Change-Id: I5fff32ec5dd1a045116bcf02349650b1f5e3a196
2021-07-30 14:20:28 +02:00
Riccardo Pittau
7aec74d4e6 Increase version of hacking and pycodestyle
Fix errors in unit tests

Change-Id: Ib5a75fc5e5b6b5661d36e3a27796c3c02ed90056
2021-07-28 11:33:15 +02:00
Riccardo Pittau
f1c9ff0e92 Update min version of tox to use allowlist
The min version of tox is updated to 3.18.0 to replace whitelist_externals
with allowist_externals option [1]

[1] https://github.com/tox-dev/tox/blob/master/docs/changelog.rst#v3180-2020-07-23

Change-Id: Icf821d49d2543be2a37dffc686ae7ab4163b04cc
2021-05-18 11:13:22 +02:00
Zuul
b16fae4740 Merge "Enforce autospec in test_console_utils" 2021-03-17 06:06:34 +00:00
Zuul
c6ec9d0b79 Merge "Enforce autospec in test_portgroup" 2021-03-17 05:53:48 +00:00
Julia Kreger
88673f1e94 Allocation support for project scoped RBAC
Adds policy scope based RBAC handling for the allocations
endpoing which enables admins to create allocations if
they have baremetal nodes which are available to them.

Change-Id: I60e273afaf344fded9bdb8c4c8e143efc9971fc1
2021-03-12 08:37:51 -08:00
Riccardo Pittau
0b920d2a08 Enforce autospec in test_console_utils
Adjust unit tests and filters in tox.ini

Change-Id: I05a3a1b37d0ce3cf33b3b5b9637350f0fd1f0afb
2021-03-08 19:16:08 +01:00
Riccardo Pittau
94fae9ac23 Enforce autospec in test_portgroup
Adapt unit tests and remove filter from tox.ini

Change-Id: I9962afd71bb2a72374559dc1d07bba4137754eb0
2021-03-05 14:29:43 +01:00
Riccardo Pittau
00eb6dcfa2 Enforce autospec in test_port
Remove filter from tox.ini

Change-Id: Ic8990ac3cb2ddb6e243ce00918793fc96684fde0
2021-03-04 17:21:31 +01:00
Riccardo Pittau
b0696ea0a8 Enforce autospec in test_volume_connector
Adjust unit tests and remove filter from tox.ini

Change-Id: Ide7ac1a29f6e07842462d6e39fe797a309218884
2021-03-04 16:06:31 +01:00
Riccardo Pittau
b4cae9b828 Enforce autospec in test_volume_target
Adapt unit tests and remove filter from tox.ini

Change-Id: I67fe1dd81ae7fe384c869e73d88182fcde402826
2021-03-03 18:24:29 +01:00
Riccardo Pittau
f18fec2329 Enforce autospec in test_driver
And remove filter from tox.ini

Change-Id: I90efb9f309df46c9b0ef073e789e1f1aaa1b1a1c
2021-02-23 10:55:18 +01:00
Riccardo Pittau
52c01f87cc Enforce autospec in test_notification_utils module
And remove tox.ini filter

Change-Id: I29257c805ce0afb01ae9422671d71d225f4b2573
2021-02-19 13:16:29 +00:00
Riccardo Pittau
6ed75bc6cf Enforce autospec in test_node
And remove filter from tox.ini

Also sets noqa for 2 mocks to keep ignoring autospec.

Change-Id: I0eadacf87baf68bc1c6547aac562f85edcad1aa3
2021-02-19 14:16:01 +01:00
Zuul
3858b95204 Merge "Report the slowest tests after a test run" 2021-02-18 05:27:54 +00:00
Riccardo Pittau
437cae984d Enforce autospec in test_chassis
And remove corresponding tox.ini filter

Change-Id: Id52e736c73f491faf04d2691a1f2ef0441b94111
2021-02-17 11:56:17 +01:00
Dmitry Tantsur
e36d7af097 Report the slowest tests after a test run
Change-Id: Id9cf476f56f2f78b9e8adf6e736148e91916777d
2021-02-11 12:44:32 +01:00
Riccardo Pittau
5165edaf93 Update minversion of tox
The minimum version since when tox accepts having inline comments
in deps is 3.9.0

https://tox.readthedocs.io/en/latest/changelog.html#v3-9-0-2019-04-17

Change-Id: I4d2aaf97459344ca01e5ccd6e563474c094a4678
2021-01-21 14:57:07 +01:00
likui
443694b186 remove lower-constraints in tox
Change-Id: If798d35d47bbc13d2e6e751467d16848c44d2975
2020-12-17 09:35:24 +08:00
Dmitry Tantsur
5c7158f017 Avoid a full install in tox environments that do not need it
Usage of usedevelop=True results in all requirements.txt dependencies
always installed (without constraints) even when requirements.txt
is not explicitly added. Override usedevelop for these environments.

Add requirements.txt to the 'docs' environment since it imports
the whole ironic in the process.

Change-Id: Ibab7d9cff1bd6eccc022873d1c5f59e27e54abaf
2020-12-14 14:05:39 +01:00
Riccardo Pittau
acb1c6b483 Do not pin Python version in tox config
We want to use the default for each system.

Change-Id: I80e411e11b3b1db990ff91a1cf90c662170dcab6
2020-12-14 09:51:01 +01:00
Iury Gregory Melo Ferreira
98732623b2 Fix lower-constraints with the new pip resolver
* move pep8 dependencies from test-requirements to tox.ini,
  they're not needed there and are hard to constraint properly.
* add oslo.cache to l-c to avoid bump of dependencies

Change-Id: Ia5330f3d5778ee62811da081c28a16965e512b55
2020-12-11 13:34:24 -08:00
Riccardo Pittau
6f92e2b90e Enforce autospec in some api controllers modules
And modify corresponding filter in tox.ini for:
v1/test_utils.py

Change-Id: I48736c7281e7c2b4deed64d6763a5c633d9f27b0
2020-11-03 16:20:08 +01:00
Zuul
731f0768c4 Merge "Enforce autospec in some api controllers modules" 2020-11-03 11:17:50 +00:00
likui
64df3702ce Update TOX_CONSTRAINTS_FILE
UPPER_CONSTRAINTS_FILE is old name and deprecated
-https://zuul-ci.org/docs/zuul-jobs/python-roles.html#rolevar-tox.tox_constraints_file
This allows to use upper-constraints file as more
readable way instead of UPPER_CONSTRAINTS_FILE=<lower-constraints file>.

[1] https://review.opendev.org/#/c/722814/

Change-Id: I85c10d5969ed03e4f27b73a76eb9da5fffa3b485
2020-11-03 13:15:56 +08:00
Riccardo Pittau
3e5f3df91c Enforce autospec in some api controllers modules
And modify corresponding filter in tox.ini for:
test_base.py
v1/test_allocation.py

Change-Id: I02f1ee5093872ca8521646cf49bbae5744a386ff
2020-11-02 18:01:46 +01:00
wu.shiming
d875500490 Remove install unnecessary packages
The docs and releasenotes requirements migrated to doc/requirements.txt
we need not install things from requirements.txt.

Change-Id: I2633cd68b45d27ef3c964216a7d97cd442919101
2020-09-14 11:10:44 +08:00
Shivanand Tendulker
8fff98a6f9 Enforce autospec in ilo tests
And remove corresponding filter in tox.ini

Change-Id: Ib47a5894d7b05346726c5b8ae8668dc8512cc322
2020-08-25 13:06:59 -04:00
Riccardo Pittau
269ada268b Enforce autospec in common tests
And remove corresponding filter in tox.ini

Change-Id: I0664fec6477227f5fc3f91518a4e09f5a43be6d3
2020-08-24 18:24:15 +02:00
Riccardo Pittau
0fc1ca7a23 Enforce autospec in irmc tests
And remove corresponding filter from tox.ini

Change-Id: I33864a06c0436b85ec2ed7cc1b798b700b2bf3ab
2020-08-12 17:33:22 +02:00
Riccardo Pittau
c6c79d1437 Enforce autospec in xclarity tests
And remove corresponding filter from tox.ini

Change-Id: I2fcca0c697970264b711f90000087991bd6125ac
2020-08-07 17:47:14 +02:00
Riccardo Pittau
15fa7ac816 Enforce autospec in test_cinder
And remove corresponding filter from tox.ini

Change-Id: Ia8f608fbf2c6feffcf23da84ae91ccf3d6ea4baa
2020-08-06 18:17:50 +02:00
Riccardo Pittau
2bc99efe48 Enforce autospec in test_iscsi_deploy
And remove corresponding filter from tox.ini

Change-Id: I726dce65b9306b5925bc0b0d8a1187c1c473125d
2020-07-29 20:37:04 +02:00
Riccardo Pittau
7a839569aa Enforce autospec in some api tests
And remove corresping filters.

Change-Id: Idb7fe3fc8b220f223d137eaced47a9ded13eba13
2020-07-24 23:23:21 +02:00
Zuul
d83e21e1bf Merge "Enforce autospec in test_notification_utils" 2020-07-24 15:06:26 +00:00
Zuul
94dfa8967d Merge "Enforce autospec in test_deployments" 2020-07-24 15:06:23 +00:00
Riccardo Pittau
a7ee205019 Enforce autospec in test_notification_utils
And remove corresponding filter from tox.ini

Change-Id: I4724a5efd522c313d2f6dbb14dcdf19700f8cda7
2020-07-17 07:08:07 +00:00
Riccardo Pittau
5abb3aedce Enforce autospec in test_deployments
And remove corresponding filter from tox.ini

Change-Id: I5eae1cbed3abbb6316f67619c499f040f43d11b1
2020-07-17 09:07:37 +02:00
Riccardo Pittau
d430d1bdb9 Set min version of tox to 3.2.1
As recommended, since version 3.2.0 tox switches pip invocations
to use the module -m pip instead of direct invocation.
We set min version to 3.2.1 [1] to also fix the behavior of
--parallel--safe-build

[1] https://tox.readthedocs.io/en/latest/changelog.html#v3-2-1-2018-08-10

Change-Id: Ia7209934d30f6a55879319ab6ca94d8bf8886073
2020-07-15 15:25:09 +02:00
Riccardo Pittau
359d6bca92 Enforce autospec in test_task_manager
And remove corresponding filter from tox.ini

Change-Id: I226e329151e0f71e793ade432c80a1628aa21511
2020-07-07 15:45:15 +02:00
Riccardo Pittau
00519ba619 Enforce autospec in test_manager
And remove the corresponding H210 filter.

Change-Id: Ib53af28596dc1392f68c956efc6d8750de611c13
2020-07-02 11:30:42 +02:00
Riccardo Pittau
a58ca1f7a6 Enforce autospec in test_utils
And remove the corresponding H210 filter.

Change-Id: Iaec44cc66ba654e478381260044776fd838b7527
2020-06-24 09:06:52 +02:00