Commit Graph

257 Commits

Author SHA1 Message Date
Jay Faulkner
3db454f0ea Enable WSGI module entrypoint for Ironic
This adds a wsgi entrypoint module which can be used with a wsgi runner,
such as uwsgi, to launch Ironic API processes without the need of a
separate script.

The legacy WSGI script is currently being installed by PBR, and as part
of the migration to a pyproject.yaml-compatible PBR, we cannot use the
wsgi-scripts plugin anymore, and will be removing the script installed
by it in a future Ironic release.

The new WSGI script, because it has statements at the module top-level,
cannot be autodocumented; we now exclude it.

Also we don't treat all warnings as errors in pdf docs builds to allow
the use of mock autosummary, starting with including the wsgi module.

Co-Authored-By: Doug Goldstein <cardoe@cardoe.com>
Change-Id: I584ac6a25c4e6cd9744a609b50d12b434a930dc6
2024-10-25 16:57:05 +02:00
Takashi Kajinami
eb27c6e219 Drop SQLALCHEMY_WARN_20
This environment was used by SQLAlchemy 1.4 and is no longer necessary
since SQLAlchemy was bumped to 2.0 .

Change-Id: I0e01f61529b633251f99d5a1a3e00ffca6c8837f
2024-09-16 06:25:25 +00:00
Jay Faulkner
3dcbc19541 Add tox env for local dev; update documentation
This is a quick and easy way to get Ironic up for testing; updated to be
even quicker and easier if you don't need multiprocess or mysql.

Co-Authored-By: CID <cid@gr-oss.io>
Change-Id: Ibef8a24868fd1f507e69e6d615d6327031d11495
2024-07-01 18:28:12 +00:00
Jay Faulkner
b44cad8e5e Fix CI issues in codespell/pep8
Codespell upgrade caused failures, fixed spelling where
appropriate, added ignores where appropriate.

Some new package release broke pep8 runs; fixed by no
longer pinning Pygments version.

Change-Id: I670bbb170823d6a0ace8eeb9d9e486e8e9bf7404
2024-05-28 02:38:31 -07:00
Jay Faulkner
74ecde7ea0 Add servicing states to states doc, fix state diagram
Good news, pydot (original) is maintained again and pydot2 apparently is
not. By switching to pydot instead of the fork, svg generation works
now.

This adds states for servicing, and swaps us back to svg for the
regenerated diagram.

Change-Id: I410182ee04293434d889747ddec229870c908d91
2024-05-10 14:31:55 -07:00
Zuul
5fabff87d1 Merge "Use .coveragerc file" 2024-04-18 07:59:33 +00:00
Zuul
4aa096877e Merge "[codespell] Adding Tox Target for Codespell" 2024-03-14 17:13:10 +00:00
Sharpz7
d9283af213 [codespell] Adding Tox Target for Codespell
Second in a series of commits to add Codespell to Ironic Repos. This one adds the command that was used to fix the spelling errors.

Future Commits will add CI support and potentially a git-blame-ignore-revs file if their are lots of spelling mistakes that could clutter git blame.

Change-Id: I206f51f277d19bbcec450ed5312cd30d6fba8432
2024-02-12 19:59:06 +00:00
Riccardo Pittau
5c781a916c Force constraints when installing a package during tox test
The tox deps option grants installation of single dependencies and
requirements, optionally pinned using constraints, before installing
a package, therefore not granting installation of the correct
constraint during the package installation.
To fix that tox 4.4.0 has introduced the constrain_package_deps
option [1]

[1] https://tox.wiki/en/4.12.1/faq.html#using-constraint-files

Change-Id: I94f02c99d1301e9dcdecb8b5565ef6a24204dc69
2024-02-12 14:54:38 +01:00
Takashi Kajinami
b6f062a8e8 tox: Drop envdir
tox now always recreates an env although the env is shared using envdir
options.
~~~
$ tox -e genpolicy
genpolicy: recreate env because env type changed from
{'name': 'genconfig', 'type': 'VirtualEnvRunner'} to
{'name': 'genpolicy', 'type': 'VirtualEnvRunner'}
~~~

According to the maintainer of tox, this functionality is not intended
to be supported.
https://github.com/tox-dev/tox/issues/425#issuecomment-1011944293

Change-Id: I71faa02e183ab29511768bca5f84461bcd3b1fe3
2024-02-02 23:29:33 +09:00
Takashi Kajinami
055224c5e2 Use .coveragerc file
... instead of defining the omit options multiple times.

Change-Id: I14c921454dd50854da64bd1ed039d0cdcbb27f75
2024-02-02 23:17:12 +09:00
Jay Faulkner
fe48440326 Update to latest pep8/code style versions
Update to latest versions for pep8 tox environment, correct any failures
created by change.

Change-Id: I605193dacc5e6be5c664b8daab44be16065b154d
2023-12-28 14:23:05 -08:00
likui
3eca0d8713 tox: Remove basepython
Python 2 is EOL. No environment should be defaulting to it. Our CI
environments certainly aren't.

Change-Id: I7b59edcb2e258b774bc71ad8e9873653d3e0b9a4
2023-08-02 16:57:32 +08:00
Zuul
6233276b20 Merge "Unit tests: Isolate mysql test migrations" 2023-07-11 01:13:06 +00:00
Julia Kreger
c4e3100d5c Add hold steps
* Updates API version to 1.85 to permit an ``unhold`` verb
* Adds the ``deploy hold`` and ``clean hold`` provision states
  to the internal state machine.
* Adds on documentation on steps to help provide greater clarity
  to Ironic's users on how to utilize steps. It should be noted
  this documentation also includes the power state reserved step
  names from the DPU functionality patch.
* Fixes the state machine diagram. Changes type to PNG as SVG
  rendering is broken due to python libraries utilized for SVG
  generation which do not work on more recent Python versions.

Change-Id: I34f58f4e77e7757b89247fd64f5fcde26f679453
2023-06-30 14:34:26 -07:00
Riccardo Pittau
f69e9da1d0 Fix multiple things in CI
Disable irmc virtual media test_prepare_instance_with_secure_boot
and test_clean_up_instance_with_secure_boot
Related-Bug: #2025424

Explicitly close out test connection
When creating a test database, we should follow the same
pattern we know to be good, where an orphaned handler is not
left in memory to close out a connection.
Change connection style in dbTestBase to mirror how we do
database connections so they close out when we are done
with them.

Change migrations timeout to be >60 seoncds
In local testing, I found the migrations tended to take an
average of 70 seconds. Granted, my test machine is old, and slow
but the performance is very similar to a busy cloud provider.
As such, increase the timeout to a larger value so we can enable
the double migration test again.
Also use BASE_TEST_TIMEOUT as time limit for unit tests, failing
hard if that's passed.

Co-Authored-By: Julia Kreger <juliaashleykreger@gmail.com>
Change-Id: I84802be2e75751fe44ba2e1b60e60563cd276483
2023-06-30 14:45:45 +02:00
Julia Kreger
5cad8ac773 Unit tests: Isolate mysql test migrations
All database migration testing in opestack is done through
an opportunistic worker model, where if the database is available
and correctly configured for testing, i.e. openstack-citest user
and access appropriately granted, then the tests will create and
test migrations.

However, this has been problematic with mysql as of recent, as we
have seen a long standing migration issue boil to the surface often
with tests.

As a result, we're isolating that test down to it's own job so we
can limit the blast damage. This also helps us isolate is it all
of the tests, or is it just soley isolated down to the mysql test
run class, which is an additional data point.

By default, we continue to run Postgres migration tests in the
main jobs, as they haven't been impacted by this issue.

Change-Id: Iefc044c31ef029e400a7dad294504175a4462638
2023-06-29 09:23:34 -07:00
Riccardo Pittau
048bff1daf Use tox env variables in coverage tests
We should try to run coverage tests in the same conditions as
unit tests using the same tox environment.

Change-Id: I07341a5e16264964f4d165b9b5ef80dc35293a17
2023-06-29 15:53:09 +00:00
Julia Kreger
a600895ba3 Execute tests by class, not randomly
The default test runner behavior is to distribute various classes
across many runners, which is fine. But when you have lots of setup
to ensure an environment is correct for tests in a class, you
generally want to execute that together instead of separately.

As such, set the --parallel-class execution option for stestr.

Change-Id: I5a65accfb7e2033690b2934d874141db7f4bf383
2023-06-29 02:03:55 +00:00
Riccardo Pittau
dfb7f05ff0 Allow setting migrations timeout value from tox
The MIGRATIONS_TIMEOUT value is the timeout for the migrations tests
in seconds.
Can now be set ad environment variable, default to 60 seconds.

Change-Id: I20045a7c0c08a37038b4707791f6b67f715c4877
2023-06-20 17:34:28 +02:00
Riccardo Pittau
f052bde94f Add test timeout to tox config
Fails a test after OS_TEST_TIMEOUT seconds, default to 30 seconds.
Based on OS_TEST_TIMEOUT from oslotest
See https://github.com/openstack/oslotest/blob/master/oslotest/base.py#L35-L45
for more info

Also increase log output for improved troubleshooting.

Change-Id: Ibcdca2c449970b5a2ecdf2e3bb3bb900881b6d7c
2023-06-12 09:17:29 +02:00
Jay Faulkner
153d4d8385 Upgrade to latest hacking - v6
Required minor changes to existing files to comply with new flake rules.

Change-Id: Ia0bff27ab4a7ec98c533ea66357a3c0529026102
2023-04-21 12:15:07 -07:00
Riccardo Pittau
ebc1053cee Exclude all files starting with . from flake8 tests
Some tests tool and IDE may create a .local directory inside the repo
with virtualenvs for dependencies, other tools may create . directories
or files for temporary reports.
While they can be removed in a second time, or configured differently,
it's advisable to just exclude all file starting with . from the
flake8 tests to avoid confusion and possible unexpected errors.

Change-Id: I5dc2fd2dec9690c77babe0c2b1dc5d0991413b32
2023-03-28 18:11:35 +02:00
Dmitry Tantsur
c0460efd06 Add missing include for inventory API reference
Also fix up the title and make sure the linter checks API ref files.

Change-Id: I360fd4fab699e732ac03dc07faab33e18fe2bf13
2023-02-23 18:38:19 +01:00
Riccardo Pittau
6b84fbf8f2 Fix CI
- Remove skipsdist that it was never supported and causes breakage
when used with usedevelop.
- add script to allowlist for pep8 test
- disable setuptools autodiscovery
- Increase base VM memory according to new requirements for CS9
based IPA

Change-Id: I0bfef09a5312a17be54ce5c09805f06b7c349026
2022-12-29 17:10:53 +01:00
Jay Faulkner
ef772c2c1e Fixes for tox 4.0
Formatting changes in config file required for tox 4.0.

Change-Id: I84202ac10e9195647162f0b5737ebb610ef1ef93
2022-12-15 09:29:38 -08:00
Julia Kreger
c2df29e52f Phase 2 - SQLAlchemy 2.0 Compatability
* Changed common exception imports from SQLAlchemy for ORM
  query types which are now originated from the main exception
  definition set.

* Changed base join option usage to use objects instead of labels,
  and defaulted all multi-row result sets to return data using
  "selectinload" as opposed to operating with a join query to
  avoid need to de-duplicate all result sets.

* Changed DeployTemplates to utilize objects instead of
  field names for queries, and updated the associated join ORM
  model's relationship record between DeployTemplate and
  DeployTemplateSteps.

* Changed Ports, Chassis, Conductor, Volume Target/Connector
  queries to lean towards use of select/update/delete queries as
  opposed to ORM queries. Most of these changes revolved around
  references of field names as opposed to model objects.

* This change also labels a few lines as "noqa", which is a result
  of the style check rules getting triggered on statements as
  needed for SQLAlchemy.

Change-Id: I651ec4b50c79be6aa8c798ee27957ed720a578d8
2022-10-14 07:58:27 -07:00
Julia Kreger
49e085583d Phase 1 - SQLAlchemy 2.0 Compatability
One of the major changes in SQLAlchemy 2.0 is the removal
of autocommit support. It turns out Ironic was using this quite
aggressively without even really being aware of it.

* Moved the declaritive_base to ORM, as noted in the SQLAlchemy 2.0
  changes[0].

* Console testing caused us to become aware of issues around locking
  where session synchronization, when autocommit was enabled, was
  defaulted to False. The result of this is that you could have two
  sessions have different results, which could results on different
  threads, and where one could still attempt to lock based upon prior
  information. Inherently, while this basically worked, it was
  also sort of broken behavior. This resulted in locking being
  rewritten to use the style mandated in SQLAlchemy 2.0 migration
  documentation. This ultimately is due to locking, which is *heavily*
  relied upon in Ironic, and in unit testing with sqlite, there are
  no transactions, which means we can get some data inconsistency
  in unit testing as well if we're reliant upon the database to
  precisely and exactly return what we committed.[1]

* Begins changing the query.one()/query.all() style to use explicit
  select statements as part of the new style mandated for migration
  to SQLAlchemy 2.0.

* Instead of using field label strings for joined queries, use the
  object format, which makes much more sense now, and is part of
  the items required for eventual migration to 2.0.

* DB queries involving Traits are now loaded using SelectInLoad
  as opposed to Joins. The now deprecated ORM queries were quietly
  and silently de-duplicating rows and providing consistent sets
  from the resulting joined table responses, however putting much
  higher CPU load on the processing of results on the client.
  Prior performance testing has informed us this should be a minimal
  overhead impact, however these queries should no longer be in
  transactions with the Database Servers which should offset the
  shift in load pattern. The reason we cannot continue to deduplicate
  locally in our code is because we carry Dict data sets which cannot
  be hashed for deduplication. Most projects have handled this by
  treating them as Text and then converting, but without a massive
  rewrite, this seems to be the viable middle ground.

* Adds an explict mapping for traits and tags on the Node object
  to point directly to the NodeTrait and NodeTag classes. This
  superceeds the prior usage of a backref to make the association.

* Splits SQLAlchemy class model Node into Node and NodeBase, which
  allows for high performance queries to skip querying for ``tags``
  and ``traits``. Otherwise with the afrormentioned lookups would
  always execute as they are now properties as well on the Node
  class. This more common of a SQLAlchemy model, but Ironic's model
  has been a bit more rigid to date.

* Adds a ``start_consoles`` and ``start_allocations`` option to the
  conductor ``init_host`` method. This allows unit tests to be
  executed and launched with the service context, while *not* also
  creating race conditions which resulted in failed tests.

* The db API ``_paginate_query`` wrapper now contains additional
  logic to handle traditional ORM query responses and the newer style
  of unified query responses. Due to differences in queries and handling,
  which also was part of the driver for the creation of ``NodeBase``,
  as SQLAlchemy will only create an object if a base object is referenced.
  Also, by default, everything returned is a tuple in 1.4 with the
  unified interface.

* Also modified one unit test which counted time.sleep calls, which is
  a known pattern which can create failures which are ultimately noise.

Ultimately, I have labelled the remaining places which SQLAlchemy
warnings are raised at for deprecation/removal of functionality,
which needs to be addressed.

[0] https://docs.sqlalchemy.org/en/14/changelog/migration_20.html
[1] https://docs.sqlalchemy.org/en/14/dialects/sqlite.html#transaction-isolation-level-autocommit

Change-Id: Ie0f4b8a814eaef1e852088d12d33ce1eab408e23
2022-10-13 21:21:40 +00:00
Stephen Finucane
25b3e6796d tests: Add a WarningsFixture
We will use this in the future to prepare for SQLAlchemy 2.0. For now,
we're simply using it to filter out some of the more annoying warnings
and to highlight general SQLAlchemy issues we need to address.

Change-Id: I7c26c20e4b36c4f3b98873939677b966ec6186a5
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2022-09-08 10:58:42 +01:00
Ghanshyam Mann
2e94aa4241 Drop lower-constraints.txt and its testing
As discussed in TC PTG[1] and TC resolution[2], we are
dropping the lower-constraints.txt file and its testing.
We will keep lower bounds in the requirements.txt file but
with a note that these are not tested lower bounds and we
try our best to keep them updated.

[1] https://etherpad.opendev.org/p/tc-zed-ptg#L326
[2] https://governance.openstack.org/tc/resolutions/20220414-drop-lower-constraints.html#proposal

Change-Id: Ide6b3e9d6c1171f1ae568c256a3e9affa45de17b
2022-04-30 17:58:31 -05:00
Zuul
e5a1997df8 Merge "Create API documentation from docstrings" 2022-03-17 15:35:42 +00:00
Mahnoor Asghar
3e631a5931 Create API documentation from docstrings
Create a new Sphinx extension called 'web_api_docstring' to process
docstrings from the API classes, in order to generate API
documentation.

Story: 2009785
Task: 44291
Change-Id: Ia6b2b3741e2b1cbd29531c21795df4f0f0dc70ca
2022-03-17 01:22:44 +05:00
Steve Baker
b8ef35a984 Ensure tox.ini is ASCII
The left/right single quotation marks cause an ascii decode error in
some python3.6 environments.

Change-Id: I80d1b3e7f9e2e23ddaca640714a342b63ae3a5fa
2022-02-18 12:50:30 +13:00
Riccardo Pittau
47b98d44c5 Use default test dir
The test dir is already defined in .stestr.conf

Change-Id: I0d620ac5e9f7eeaae86243a9235f431a9114804b
2021-11-19 10:39:41 +01:00
Riccardo Pittau
e5d35ba65c Add lower-constraints job to current development branch
The lower-constraints test was removed becuase of an issue where pip
could not correctly determine the required packages versions to install,
ending in an almost infinite loop that would end up in timeout, failure,
and general mayhem.
Recently the issue has been fixed and, if properly configured, the
lower-constraints test can provide good indication of which minimum
versions are required to support the current code.
This patch adds the test back to the current development branch.
The long term goal is to keep the lower-constraints file in the stable
branches, but remove the test job to avoid issues during backports.

Change-Id: I5fff32ec5dd1a045116bcf02349650b1f5e3a196
2021-07-30 14:20:28 +02:00
Riccardo Pittau
7aec74d4e6 Increase version of hacking and pycodestyle
Fix errors in unit tests

Change-Id: Ib5a75fc5e5b6b5661d36e3a27796c3c02ed90056
2021-07-28 11:33:15 +02:00
Riccardo Pittau
f1c9ff0e92 Update min version of tox to use allowlist
The min version of tox is updated to 3.18.0 to replace whitelist_externals
with allowist_externals option [1]

[1] https://github.com/tox-dev/tox/blob/master/docs/changelog.rst#v3180-2020-07-23

Change-Id: Icf821d49d2543be2a37dffc686ae7ab4163b04cc
2021-05-18 11:13:22 +02:00
Zuul
b16fae4740 Merge "Enforce autospec in test_console_utils" 2021-03-17 06:06:34 +00:00
Zuul
c6ec9d0b79 Merge "Enforce autospec in test_portgroup" 2021-03-17 05:53:48 +00:00
Julia Kreger
88673f1e94 Allocation support for project scoped RBAC
Adds policy scope based RBAC handling for the allocations
endpoing which enables admins to create allocations if
they have baremetal nodes which are available to them.

Change-Id: I60e273afaf344fded9bdb8c4c8e143efc9971fc1
2021-03-12 08:37:51 -08:00
Riccardo Pittau
0b920d2a08 Enforce autospec in test_console_utils
Adjust unit tests and filters in tox.ini

Change-Id: I05a3a1b37d0ce3cf33b3b5b9637350f0fd1f0afb
2021-03-08 19:16:08 +01:00
Riccardo Pittau
94fae9ac23 Enforce autospec in test_portgroup
Adapt unit tests and remove filter from tox.ini

Change-Id: I9962afd71bb2a72374559dc1d07bba4137754eb0
2021-03-05 14:29:43 +01:00
Riccardo Pittau
00eb6dcfa2 Enforce autospec in test_port
Remove filter from tox.ini

Change-Id: Ic8990ac3cb2ddb6e243ce00918793fc96684fde0
2021-03-04 17:21:31 +01:00
Riccardo Pittau
b0696ea0a8 Enforce autospec in test_volume_connector
Adjust unit tests and remove filter from tox.ini

Change-Id: Ide7ac1a29f6e07842462d6e39fe797a309218884
2021-03-04 16:06:31 +01:00
Riccardo Pittau
b4cae9b828 Enforce autospec in test_volume_target
Adapt unit tests and remove filter from tox.ini

Change-Id: I67fe1dd81ae7fe384c869e73d88182fcde402826
2021-03-03 18:24:29 +01:00
Riccardo Pittau
f18fec2329 Enforce autospec in test_driver
And remove filter from tox.ini

Change-Id: I90efb9f309df46c9b0ef073e789e1f1aaa1b1a1c
2021-02-23 10:55:18 +01:00
Riccardo Pittau
52c01f87cc Enforce autospec in test_notification_utils module
And remove tox.ini filter

Change-Id: I29257c805ce0afb01ae9422671d71d225f4b2573
2021-02-19 13:16:29 +00:00
Riccardo Pittau
6ed75bc6cf Enforce autospec in test_node
And remove filter from tox.ini

Also sets noqa for 2 mocks to keep ignoring autospec.

Change-Id: I0eadacf87baf68bc1c6547aac562f85edcad1aa3
2021-02-19 14:16:01 +01:00
Zuul
3858b95204 Merge "Report the slowest tests after a test run" 2021-02-18 05:27:54 +00:00
Riccardo Pittau
437cae984d Enforce autospec in test_chassis
And remove corresponding tox.ini filter

Change-Id: Id52e736c73f491faf04d2691a1f2ef0441b94111
2021-02-17 11:56:17 +01:00