Commit Graph

399 Commits

Author SHA1 Message Date
Ghanshyam Mann
3f6507d851 Remove default override for RBAC config options
RBAC config options enforce_scope and enforce_new_defaults
were disabled by default in oslo.policy and Ironic had to override
the default value to enable those by default. Now oslo.policy
(4.4.0 onwards[1]) changed the default values[2][3] and enabled
by default for all the services. OpenStack service does not need
to override the default anymore.

NOTE: There is no change in behaviour here, oslo.policy provides the
same configuration that Ironic has overridden till now.

[1] https://review.opendev.org/c/openstack/releases/+/925032
[2] https://review.opendev.org/c/openstack/oslo.policy/+/924283
[3] https://review.opendev.org/c/openstack/requirements/+/925464

Change-Id: I280ae374048b16f1d27a55b09a4d7729de43f469
2024-09-05 12:46:12 -07:00
Steve Baker
91d4bacbec Replace cinderclient usage with openstacksdk
Change-Id: Ib4a533584da85281d425fdbffa12a52d4838e185
Closes-Bug: #2042494
2024-05-01 07:58:41 +12:00
Takashi Kajinami
1e4ef9395a Remove old excludes
These are detected as errors since the clean up was done[1] in
the requirements repository.

[1] 314734e938f107cbd5ebcc7af4d9167c11347406

Also remove the note about old pip's behavior because the resolver
in recent pip no longer requires specific order.

Change-Id: I742ea0192398b9e9b78b969fa81f65621d9490de
2024-04-30 22:43:13 +09:00
Zuul
b6b8ee07ce Merge "Replace glanceclient usage with openstacksdk" 2024-04-29 15:26:56 +00:00
Steve Baker
978afbd5a1 Replace glanceclient usage with openstacksdk
Closes-Bug: #2042495
Change-Id: Ic8421bd937a3a1ab6c3b86c259cd929810c0532e
2024-04-29 12:30:33 +12:00
Dmitry Tantsur
f7fd7961c0
Trivial: use canonical name of microversion-parse
Change-Id: I8b457b531cc181bbb43e8e21bcd29929ce5725f7
2024-04-24 09:13:47 +02:00
cid
5552e54d8b Use the new microversion-parse library to parse the version headers
Closes-Bug: #1584718

Change-Id: Icb2e21775d1500878bb6d29f45e82e32f14332aa
2024-04-05 09:28:52 +00:00
Winicius Silva
6d046ad7b4 Allow usage of virtual media via System
DMTF now changes their Virtual Media URI to support Systems. Redfish driver now
support this resource to boot, so it is needed that Ironic have a way to use it.

Closes-Bug: #2039458
Change-Id: I66e8edb847e93f96374072525222f05e7561fb07
2024-03-13 22:02:08 -03:00
Dmitry Tantsur
4b31cc1c48
Switch to qemu-img functions from ironic-lib 6.0.0
Change-Id: I0010d8710db4f2859dabe8ccfa5bf478d4f039f6
2024-03-11 17:16:12 +01:00
Julia Kreger
768f585603 packaging: reduce jsonschema revision
The VTEP switch support patch merged with a constraint of jsonschema
version 4.19 or above.

Except Debian only currently has 4.10, Centos 9 Stream only has 4.16,
and at present launchpad and the ubuntu mirror list is non-functional.

So in the interest of of packagers, we'll lower the version.

Note: I was able to successfully execute the unit tests with jsonschema
4.0.0 installed in the py3 virtualenv.

Change-Id: Ic3667a7663b7bd5dfad4665321d9c82cc08cc885
2024-02-05 07:27:35 -08:00
Zuul
67b198a29a Merge "Basic support for OVN VTEP switches" 2024-02-05 12:40:24 +00:00
Zuul
8fe3c0f35b Merge "Remove dependency on pytz" 2024-01-29 09:15:13 +00:00
Takashi Kajinami
c6e0553588 Remove dependency on pytz
pytz will be removed from RHEL/CentOS 10 because of the built-in
zoneinfo[1].

Because the current usage of pytz can be very easily replaced, this
removes the dependency on pytz.

[1] https://issues.redhat.com/browse/RHEL-219

Change-Id: Ia72c528eadeccf6075894ff58477fecade65ad71
2024-01-27 19:40:34 +09:00
Zuul
cd17f5e61a Merge "Drop rootwrap support" 2024-01-23 20:14:36 +00:00
Boushra Bettir
ed946c4d55 Basic support for OVN VTEP switches
Adds basic support for passing OVN VTEP switch metadata to
neutron via Ironic's port.local_link_connection field.

Adds microversion 1.90 to Ironic's API, adding support for
new schema in port.local_link_connection

Bump version of the jsonschema library to ensure consistent
behavior with new schema configurations.

Add documentation warning: This has not been tested as no
Ironic developers have access to the hardware in question.

Closes-bug: #2034953
Co-Authored-By: Austin Cormier <acormier@juniper.net>
Co-Authored-By: Jay Faulkner <jay@jvf.cc>
Change-Id: Ie98dc4552ec2ea16db1e2d382aed54ce9dfef41b
2024-01-19 13:37:12 -08:00
Dmitry Tantsur
be09717be2
Drop rootwrap support
After removing the iSCSI deploy and changing ISO parsing code to use
a corresponding library, Ironic no longer executes any commands as root
and it should stay this way.

Change-Id: I47d2bab9b94345fbcf89a2a80028853050a041ea
2024-01-08 18:02:27 +01:00
Julia Kreger
041a7d7064 Redfish UefiHttp boot support
Adds a redfish-https boot interface, based upon the
redfish-virtual-media boot interface, however substantially copies
some base methods because of simplification offered to use by
putting "attach/detach" logic into how the sushy library handles
the application and reset of a URL as a boot setting.

This feature also increases the requirement for the Sushy library
to version 4.7.0 which includes support to set the HttpBootUri
field in the BMC and automatically unset it as well.

Closes-Bug: #2032380
Change-Id: I991611cd67cb91aea21fc30bbae7cd24409dbbfa
2024-01-04 07:12:20 -08:00
Steve Baker
2db444bce1 Replace swiftclient usage with openstacksdk
Object create/delete operations translate clearly from swiftclient to
the SDK. Switching the temp URL handling is a little more disruptive but
the result is slightly more centralized and enables key rotation.

Change-Id: I8df2f032224bd5e540139a798a7ab76a1aeebb06
Closes-Bug: #2042493
2023-11-29 12:44:30 +13:00
Mahnoor Asghar
c3ee90ddac Add inspection hooks
Adds the 'local-link-connection' and 'parse-lldp' inspection hooks in
the agent inspect interface for processing data received from the
ramdisk at the /v1/continue_inspection endpoint.

Change-Id: I540f03b961b858e8fc00cd4abbc905faa8f0c6c5
Story: #2010275
2023-10-17 09:31:33 -02:00
Zuul
dffcc69e89 Merge "Fully monkey patch eventlet for consistent behavior" 2023-08-31 18:28:54 +00:00
Jay Faulkner
69bdbfb6e9 Fully monkey patch eventlet for consistent behavior
Eventlet expects to have green versions of every module loaded, including OS. The reasons we originally did not patch os are lost
to time, but there have been many releases of eventlet since, and we should return to a reasonable baseline.

Change-Id: Ia4113124b415bd647e3c984e587828eb5b612eee
2023-08-30 20:08:54 +00:00
Julia Kreger
c84fe147a3 Utilize the JSON-RPC port
Adds storage of the json-rpc port number to the conductor hostname
to enable rpc clients to understand which rpc servies they need to
connect to.

Depends-On: https://review.opendev.org/c/openstack/ironic-lib/+/879211
Change-Id: I6021152c83ab5025a9a9e6d8d24c64278c4c1053
2023-08-30 08:56:17 -07:00
Julia Kreger
4ba418716e Retool sqlite retries
... To not try instantly, but also not to wait forever to retry.

Also, the maximum delay is also now the proper setting to cause
the attempt to exit, and is only set to 10 seconds, with a fairly
tight interval for retries to occur within.

This change also doesn't abort retries for releasing a node lock
and updating a node, both actions if they halt due to the close
out of a task, can be catastrophic to the underlying operation
and state, because internal actions around locking can't be retried
with a long interval, otherwise things break in very bad ways.

Change-Id: I2041e90bb0f7f522bde4338eceda97f0ae8b2c35
2023-08-17 07:25:19 -07:00
Julia Kreger
82b8ec7a39 Get conductor metric data
This change adds the capability for the ironic-conductor
and standalone service process to transmit timer and counter
metrics to the message bus notifier which may be consumed by
a ceilometer, ironic-prometheus-exporter, or other consumer of
metrics event data on to the message bus.

This functionality is not presently supported on dedicated API
services such as those running as an ``ironic-api`` application
process, or Ironic WSGI application. This is due to the lack of
an internal trigger mechanism to transmit the data in a metrics
update to the message bus and/or notifier plugin.

This change requires ironic-lib 5.4.0 to collect and ship metrics via
the message bus.

Depends-On: https://review.opendev.org/c/openstack/ironic-lib/+/865311
Change-Id: If6941f970241a22d96e06d88365f76edc4683364
2023-02-23 11:39:07 -08:00
Tobias Urdin
f8608dbd95 Use new get_rpc_client API from oslo.messaging
Use the new API that is consistent with
the existing API instead of instantiating the client
class directly.

This was introduced in release 14.1.0 here [1] and
added into oslo.messaging here [2]

[1] https://review.opendev.org/c/openstack/requirements/+/869340
[2] https://review.opendev.org/c/openstack/oslo.messaging/+/862419

Change-Id: I2aea1d4fb59e7440586d8d9cd27bc61d29f4530c
2023-01-19 20:31:19 +00:00
Julia Kreger
49e085583d Phase 1 - SQLAlchemy 2.0 Compatability
One of the major changes in SQLAlchemy 2.0 is the removal
of autocommit support. It turns out Ironic was using this quite
aggressively without even really being aware of it.

* Moved the declaritive_base to ORM, as noted in the SQLAlchemy 2.0
  changes[0].

* Console testing caused us to become aware of issues around locking
  where session synchronization, when autocommit was enabled, was
  defaulted to False. The result of this is that you could have two
  sessions have different results, which could results on different
  threads, and where one could still attempt to lock based upon prior
  information. Inherently, while this basically worked, it was
  also sort of broken behavior. This resulted in locking being
  rewritten to use the style mandated in SQLAlchemy 2.0 migration
  documentation. This ultimately is due to locking, which is *heavily*
  relied upon in Ironic, and in unit testing with sqlite, there are
  no transactions, which means we can get some data inconsistency
  in unit testing as well if we're reliant upon the database to
  precisely and exactly return what we committed.[1]

* Begins changing the query.one()/query.all() style to use explicit
  select statements as part of the new style mandated for migration
  to SQLAlchemy 2.0.

* Instead of using field label strings for joined queries, use the
  object format, which makes much more sense now, and is part of
  the items required for eventual migration to 2.0.

* DB queries involving Traits are now loaded using SelectInLoad
  as opposed to Joins. The now deprecated ORM queries were quietly
  and silently de-duplicating rows and providing consistent sets
  from the resulting joined table responses, however putting much
  higher CPU load on the processing of results on the client.
  Prior performance testing has informed us this should be a minimal
  overhead impact, however these queries should no longer be in
  transactions with the Database Servers which should offset the
  shift in load pattern. The reason we cannot continue to deduplicate
  locally in our code is because we carry Dict data sets which cannot
  be hashed for deduplication. Most projects have handled this by
  treating them as Text and then converting, but without a massive
  rewrite, this seems to be the viable middle ground.

* Adds an explict mapping for traits and tags on the Node object
  to point directly to the NodeTrait and NodeTag classes. This
  superceeds the prior usage of a backref to make the association.

* Splits SQLAlchemy class model Node into Node and NodeBase, which
  allows for high performance queries to skip querying for ``tags``
  and ``traits``. Otherwise with the afrormentioned lookups would
  always execute as they are now properties as well on the Node
  class. This more common of a SQLAlchemy model, but Ironic's model
  has been a bit more rigid to date.

* Adds a ``start_consoles`` and ``start_allocations`` option to the
  conductor ``init_host`` method. This allows unit tests to be
  executed and launched with the service context, while *not* also
  creating race conditions which resulted in failed tests.

* The db API ``_paginate_query`` wrapper now contains additional
  logic to handle traditional ORM query responses and the newer style
  of unified query responses. Due to differences in queries and handling,
  which also was part of the driver for the creation of ``NodeBase``,
  as SQLAlchemy will only create an object if a base object is referenced.
  Also, by default, everything returned is a tuple in 1.4 with the
  unified interface.

* Also modified one unit test which counted time.sleep calls, which is
  a known pattern which can create failures which are ultimately noise.

Ultimately, I have labelled the remaining places which SQLAlchemy
warnings are raised at for deprecation/removal of functionality,
which needs to be addressed.

[0] https://docs.sqlalchemy.org/en/14/changelog/migration_20.html
[1] https://docs.sqlalchemy.org/en/14/dialects/sqlite.html#transaction-isolation-level-autocommit

Change-Id: Ie0f4b8a814eaef1e852088d12d33ce1eab408e23
2022-10-13 21:21:40 +00:00
Jay Faulkner
211b25f307 Zed Ironic requires Sushy >4
Sushy 4 includes enhancements including support for hardware Ironic
should work with in Zed.

Story: #2009865
Task: #44548
Change-Id: Ib82bd4d1442bf7d9b135d1c1553c39cfef87548a
2022-09-12 16:09:00 -07:00
Takashi Kajinami
f0935c182e Add audit middleware options to ironic.conf
Ironic supports enabling the audit middleware. This change adds
the option for the middleware to the ironic.conf file generated by
oslo-config-generator.

Depends-on: https://review.opendev.org/804316
Change-Id: Ic7adb755f47ea65fe975dfbc7cca905a376d728e
2022-07-07 15:51:07 +00:00
Ghanshyam Mann
2e94aa4241 Drop lower-constraints.txt and its testing
As discussed in TC PTG[1] and TC resolution[2], we are
dropping the lower-constraints.txt file and its testing.
We will keep lower bounds in the requirements.txt file but
with a note that these are not tested lower bounds and we
try our best to keep them updated.

[1] https://etherpad.opendev.org/p/tc-zed-ptg#L326
[2] https://governance.openstack.org/tc/resolutions/20220414-drop-lower-constraints.html#proposal

Change-Id: Ide6b3e9d6c1171f1ae568c256a3e9affa45de17b
2022-04-30 17:58:31 -05:00
Julia Kreger
9df7e67e69 Grenade: Change to use bios because we have funky networking
Grenade, for some confusing reason, creates a separate network,
and uses that for upgrade testing as opposed to the original network
the VMs were bound to. If Julia's memory is correct, this was for
multinode upgrade testing.

Anyway, When in UEFI mode, it appears that the TFTP packets
don't get tracked nor cross the boundrary. We likley need to
explicitly address this, but first, lets get the job working as
it was and can then update it.

Also, update requirements because markupsafe removed soft_unicode
method taht was deprecated since a while. Jinja2 started using the
new soft_str method since version 3.0.0

Change-Id: Iaebe966569962b0d3d43774d57b570469479f159
2022-04-04 14:13:58 +02:00
Dmitry Tantsur
76e99a30f7 Enable Redfish by default
Redfish has become an established technology with growing popularity.
By enabling it by default we make it easier for operators to use Ironic,
especially its advanced features, out of box.

Bump stevedore to match sushy.

Change-Id: I94dd4066ac598a7e5c2e0812e1ff286de5d164a2
2021-11-05 13:51:06 +01:00
Riccardo Pittau
06124bae73 Update requirements
Update minimum required versions of python packages.
The updates is based on crosscheck requirements from required
packages:
oslo-log 4.3.0 depends on oslo.context>=2.20.0
oslo-config 6.8.0 depends on rfc3986>=1.2.0
oslo-config 6.8.0 depends on requests>=2.18.0
oslo-log 4.3.0 depends on pbr>=3.1.1
oslo-log 4.3.0 depends on oslo.serialization>=2.25.0
oslo-policy 3.7.0 depends on oslo.context>=2.22.0

Also update minimum versions of ddt and stestr to avoid compatibility
issues with recent Python versions (3.8 and higher).

Change-Id: I34b558f4e7b93618130af1ba2250f6ab8575d4d6
2021-07-13 14:37:28 +02:00
Zuul
7d6e936cfe Merge "Add support for configdrive in anaconda interface" 2021-07-06 17:21:06 +00:00
Ghanshyam Mann
34337f75fb Fix oslo policy DeprecatedRule warnings
Since 3.7.0, oslo policy started the DeprecationWarning[1] if
deprecated_reason and deprecated_since param are not passed
in DeprecatedRule or they are passed in RuleDefault object.

These warnings are logged for every test which increase the
log size and sometime can full the log buffer and fail the
job.

This fixes the 1431 warnings
- https://zuul.opendev.org/t/openstack/build/cace355f7c0d4cbcb1df7856a03194d7/log/job-output.txt#1029

[1] https://github.com/openstack/oslo.policy/blob/3.7.0/oslo_policy/policy.py#L1538

Change-Id: I98d20aea80d547e1f301c150564f732619c9e744
2021-07-05 10:57:21 -05:00
Arun S A G
4ac8340fb6 Add support for configdrive in anaconda interface
Config drive is processed by the deploy interface and
written to the disk during post install of the deployment.

The config drive is written to /var/lib/cloud/seed/config_drive
on the disk. cloud-init should be able to process read config
drive contents and take actions on firstboot.

Change-Id: Id5a949e777e242d2f20966ec045ac319c61636ba
2021-07-04 10:36:18 -07:00
Kamlesh Chauvhan
e3a4c7d77d Upgrade oslo.db version
oslo.db 9.1.0 extends capabilities to handle changes
in duplicate key error information introduced since
MySQL 8.0.19 and fixes issue arose during inspection
of node created with idrac driver, inspect interface
as idrac-redfish.

Story: 2008901
Task: 42468
Change-Id: Ic18e485d4eb14566554d25379a719669ac362d70
2021-06-28 13:24:22 +00:00
Dmitry Tantsur
f2e8c031cc Switch to JSON RPC from ironic-lib
Change-Id: I8b438861780c85faae7ff18646960723a1fd9876
2021-03-10 16:48:07 +01:00
Zuul
3f2431001e Merge "Update oslo.policy requirement to version 3.6.2" 2021-03-01 17:25:38 +00:00
Dmitry Tantsur
6e09a008ab Replace pysendfile with os.sendfile
Change-Id: I1ef33d41fd3784f55929fa6a086fca0c335212e5
2021-02-18 10:16:57 +01:00
Boden R
b0607a2668 Replace retrying with tenacity
We are replacing all usages of the 'retrying' package with
'tenacity' as the author of retrying is not actively maintaining
the project.
Unit tests will be added/removed where applicable.

Tenacity [1] is a fork of retrying, but has improved the
interface and extensibility.
Our end goal here is removing the retrying package from our
requirements.

Tenacity provides the same functionality as retrying, but has the
following major differences to account for:
- Tenacity uses seconds rather than ms as retrying did.
- Tenacity has different kwargs for the decorator and
Retrying class itself.
- Tenacity has a different approach for retrying args by
using classes for its stop/wait/retry kwargs.
- By default tenacity raises a RetryError if a retried callable
times out; retrying raises the last exception from the callable.
Tenacity provides backwards compatibility here by offering
the 'reraise' kwarg.
- For retries that check a result, tenacity will raise if the retried
function raises, whereas retrying retried on all exceptions.

[1] https://github.com/jd/tenacity

Co-Authored-By: Dmitry Tantsur <dtantsur@protonmail.com>
Co-Authored-By: Riccardo Pittau <elfosardo@gmail.com>
Story: #1635390
Task: #10528
Change-Id: Ie5eb3ddc196505e8f58ed14de9952284598586fb
2021-02-15 17:11:33 +01:00
Lance Bragstad
94ba2641e3 Update oslo.policy requirement to version 3.6.2
This version of oslo.policy includes fixes that ensure the Enforcer only
modifies copies of the rules, making it safer in environments that run
tests in parallel where the Enforcer is configured differently.

Change-Id: I8d7a06558cbf073487707fc33219e43beb5fc043
2021-02-05 21:03:39 +00:00
Lance Bragstad
6ea73bdfbb Bump oslo.log requirement to 4.3.0
This ensures we have the WALLABY marker available for deprecation
noticies.

Change-Id: Ic91f34a90fdd0c22c06c669551eaf6f9a398956b
2021-01-19 10:59:48 -08:00
Julia Kreger
2404d486ac Policy json to yaml migration
Adds the status upgrade check for the JSON to YAML migration
effort and updates the documentation where it seems appropriate
to move from "policy.json" to "policy.yaml"

Mostly shamelessly copied from https://review.opendev.org/#/c/748059/
however is in-line with ironic's configuration and patching methods.

Related Blueprint: policy-json-to-yaml

Change-Id: I1d5b3892451579ebfd4d75a0f7185e0ef3c984c8
2021-01-04 13:40:54 -08:00
Iury Gregory Melo Ferreira
19d97e6562 Fix lower-constraints for Ubuntu Focal
Change-Id: Id3cc2d1b619790813b94b0cfd52ff6590aff060c
2020-09-11 04:23:12 +00:00
Harald Jensås
ff00047b4a Switch Ironic to openstacksdk for Neutron
This patch removes the dependency from ironicclient to
communicate with neutron in favor of openstacksdk.

Also:
 * Use import keystoneauth1.loading as ks_loading accross
   the project.
 * Refactor to have one 'get_client' function, removing the
   '_get_config_client' method. Setting config_client=True
   when calling 'get_client' returns a client using auth
   options values from conf parameters.

Depends-On: https://review.opendev.org/735601
Change-Id: Ib6c0fa2acfc33deb9c5b36ae724d5d8304d1dd29
2020-08-19 10:48:17 +02:00
Zuul
3670be1283 Merge "Deprecate http_basic_username and http_basic_password in [json_rpc]" 2020-07-28 19:14:00 +00:00
Dmitry Tantsur
74e9e1d82a Deprecate http_basic_username and http_basic_password in [json_rpc]
It's very confusing that we use username/password everywhere, except
for [json_rpc]. Just use the standard options.

Also the version if keystoneauth is bumpted to one that supports
http_basic.

Change-Id: Icc834c3f8febd45c2548314ee00b85a7f9cebd2c
2020-07-24 11:51:41 +02:00
Zuul
39a38ecf4e Merge "Add an option to choose the hash ring algorithm" 2020-07-23 17:31:00 +00:00
Dmitry Tantsur
9189b4bb26 Add an option to choose the hash ring algorithm
MD5 is not available in FIPS mode, we need a way to use something else.

Change-Id: Ie6e09ac66028cbe18717a7ea7a4c23730e3cb642
2020-07-22 18:27:19 +02:00
Zuul
b5ae75a406 Merge "Use native oslo.concurrency execution timeout in ipmitool" 2020-07-22 15:58:19 +00:00