Use pure json instead of jsonutils.
Borrow encode function from oslo.serialization to be used in the
utils module.
Change-Id: Ied9a2259a4329a86b4f0853bd1fb187563c0a036
Removes multipath base devices from consideration by
default, and instead allows the device-mapper device
managed by multipath to be picked up and utilized
instead.
In effect, allowing us to ignore standby paths *and*
leverage multiple concurrent IO paths if so offered
via ALUA.
In reality, anyone who has previously built IPA with
multipath tooling might not have encountered issues
previously because they used Active/Active SAN storage
environments. They would have worked because the IO lock
would have been exchanged between controllers and paths.
However, Active/Passive environments will block passive
paths from access, ultimately preventing new locks from
being established without proper negotiation. Ultimately
requiring multipathing *and* the agent to be smart enough
to know to disqualify underlying paths to backend storage
volumes.
An additional benefit of this is active/active MPIO devices
will, as long as ``multipath`` is present inside the ramdisk,
no longer possibly result in duplicate IO wipes occuring
accross numerous devices.
Story: #2010003
Task: #45108
Resolves: rhbz#2076622
Resolves: rhbz#2070519
Change-Id: I0fd6356f036d5ff17510fb838eaf418164cdfc92
The existing lsblk call is very handy for an overview, but there a lot
more useful pairs to collect. Collect them in a machine-readable format
to be able to use in debugging and further development.
Change-Id: Ib27843524421944ee93de975d275e93276a5597a
Even if journald is present, there is no guarantee that IPA logs there
(this is the case in container-based ramdisks).
Change-Id: Iceeab0010827728711e19e5b031ccac55fe1efde
In order to make sure we have the correct time early, e.g.
by the time we create a TLS certificate, this patch proposes
to force an immediate NTP update when using chronyd. While
the previous approach uses the passed NTP server as well, the
update may happen only after chronyd has performed measurements
(which may be too late).
Story: #2009058
Task: #42843
Change-Id: I6edafe8edeb8549f324959e7a1ec175c3049a515
_early_log prints to stdout, which is fine in some cases,
however in other cases it gets lost in the shuffle of process
launch by things like systemd.
Lets try to save everything, and re-log it so it is easy to
debug early issues.
Change-Id: I334a9073d17cccec4c669fae82edc3e388debc5c
Follow-up to 8dd6589e66d03e45e1d510601da9531a30842cff: PATH is not a
valid lsblk tag, we need to use KNAME with -p flag.
Also add a vmedia job to avoid breakages in the future. It's added
non-voting because we have a deadlock with this change:
https://review.opendev.org/c/openstack/ironic/+/783722
Change-Id: Ifffeac9c1c4d394526d655eaa14c9fe7bd3a1e5e
Virtual media devices based logic needs to be
guarded from being used or considered based upon
if the machine actually booted from virtual media,
or not.
At the same time, actual devices need to be checked
in order to make sure they align with what we expect
in order to prevent consideration of content which
should not be leveraged.
Change-Id: If2d5c6f4815c9e42798a2d96d59015e1b1dbd457
Story: 2008749
Task: 42108
This change adds a deploy step inject_files that adds a flexible
way to inject files into the instance.
Change-Id: I0e70a2cbc13744195c9493a48662e465ec010dbe
Story: #2008611
Task: #41794
Glean mounts the configdrive and does not unmount it afterwards.
If a mount point already exists, just use it.
Change-Id: Ia62279afbb9fd9770864942dc40629b69ae8f4ae
For ramdisk TLS (and other potential future enhancements) we need
to be able to inject configuration and certificates into the ramdisk.
Since we cannot pass files through kernel parameters, we need to
put them on the generated ISO or (in the future) config drive.
This change detects IPA configuration and copies it into the ramdisk
early enough for any configuration files to get picked.
Changed /dev/disk/by-label to blkid since the former may not exist
on all ramdisks (e.g. tinyIPA).
Change-Id: Ic64d7842a59795bbf02f194221dedc07c6b56e8c
DIB builds instance images with EFI partitions that only have the boot
flag, but not esp. According to parted documentation, boot is an alias
for esp on GPT, so accept it as well.
To avoid complexities when parsing parted output, the implementation
is switched to existing utils and ironic-lib functions.
Change-Id: I5f57535e5a89528c38d0879177b59db6c0f5c06e
Story: #2007455
Task: #39423
The proposed changes concern two steps:
First, when creating the RAID configuration, have a GPT partition
table type (this is not necessary, but more natural with UEFI).
Also, leave some space, either for the EFI partitions or the BIOS
boot partitions, outside the Software RAID.
Secondly, when installing the bootloader, make sure the correct
boot partitions are created or relocated.
Change-Id: Icf0a76b0de89e7a8494363ec91b2f1afda4faa3b
Story: #2006379
Task: #37635
Attempt to sync the clock and save it to the hardware clock.
This feature supports use of chrony or ntpdate.
Sem-Ver: feature
Change-Id: I178d7614429d582e742d9cba6d0fa3ae099775e3
Story: 1619054
Task: 11591
This patch changes the workflow for whole disk images when using uefi.
If we can identify the bootloader and it's valid we can update using
efibootmgr since grub2-install have problems specially on secure boot
mode.
We also updated the regex to search for the uefi partition on the disk,
since in some cases the parted command output can be without the FS
for the partition with esp Flag.
Change-Id: I7167e71e5d2352a045565289b200e5530d0ba11d
Story: #2006847
Task: #37435
This patch adds a function that will be responsible to identify
the efi partition on a give device, this is necessary on the Software
Raid scenario and when installing bootloader.
Change-Id: I5f326db2d37b2a15090ec84e477e63f7d92e7447
Co-Authored-By: Raphael Glon <raphael.glon@corp.ovh.com>
Since we've dropped support for Python 2.7, it's time to look at
the bright future that Python 3.x will bring and stop forcing
compatibility with older versions.
This patch removes the six library from requirements, not
looking back.
Change-Id: I4795417aa649be75ba7162a8cf30eacbb88c7b5e
The lshw output is huge even on virtual machines, and it pollutes
the debug logging. This change silences it. Instead, the lshw output
is collected as part of the ramdisk logs.
Depends-On: https://review.opendev.org/#/c/665635/
Change-Id: I6a3015b2d8d09f6f48b5cbd39dc84bd75b72f909
As some BIOSes try to boot only from the "first" disk, Ironic
should order potential disks not only by size, but also by name.
This patch proposes to add secondary sorting by device name when
identifying the root disk.
Change-Id: I4017c839eeb9d00d2b4ad5b90e4e9b65b74296c7
Story: #2004976
Task: #29434
Increases the amount of ram for CoreOS IPA to 2GB
as the base CoreOS image is now 310MB.
Bumped CPU count for CoreOS runs to 2 CPUs as the
concurrency helps boot times for the CoreOS ramdisk.
Adds netbase, udev, and open-iscsi to debian jessie container
as they are no longer present in the default container.
Explicitly set path variable for execution in the debian
container as udevadm is in /sbin, and we may not have
/sbin on the path that is passed through to the
chroot.
Also fixed new pep8 test failures.
Story: #1600228
Task: #16287
Change-Id: I488445dfd261b7bca322a0be7b4d8ca6105750a3
Improve the catching of calls to 'execute' related functions in unit
tests. Before we only caught calls to utils.execute(). Now we catch
calls to:
ironic_lib.utils.execute()
processutils.execute()
subprocess.call()
subprocess.check_call()
subprocess.check_output()
utils.execute()
Change-Id: If4720ebed00f15c2a19cb8badbe4dc3c808eeece
This change introduces a new base test class that mocks out
utils.execute and forces an exception if it gets called.
This has rooted out many tests that were doing this as a side effect of
calling other functions, doing things like modprobe and running iscsi
on the host's actual machine.
The tests are all now appropriately patched in places where this was
happening, and the new base class permanently prevents this from
accidentally happening again.
If you really want to call utils.execute() then you need to re-mock it
in your unit test.
Change-Id: Idf87d09a9c01a6bfe2767f8becabe65c02983518
Add missing 'autospec' keyword argument to mock.patch and
mock.patch.object calls. Use 'autospec=True' except for a few cases
where it fails because the mocked function is a @classmethod and it
doesn't work. In that case explicity set it to 'autospec=False'
Change-Id: I620dce91abaa4440e1803aeefb3e93c0b65d1419
Make sure that logs are encoded as text (not byte strings) in
python3 to allow theirs serialization in JSON.
Closes-Bug: #1668533
Change-Id: I3d0d87ca766146457d905f89c26331a415bd54f8
Use the flake8 plugin flake8-import-order to check import ordering. It
can do it automatically and don't need reviewers to check it.
Change-Id: I946457e9079ce0b54c7fe0ad554d024a1c61dce0
This patch adds standard SSL options to IPA config and makes use of them
when making HTTP requests.
For now, a single set of certificates is used when needed.
In the future configuration can be expanded to allow per-service
certificates.
Besides, the 'insecure' option (defaults to False) can be overridden
through kernel command line parameter 'ipa-insecure'.
This will allow running IPA in CI-like environments with self-signed SSL
certificates.
Change-Id: I259d9b3caa9ba1dc3d7382f375b8e086a5348d80
Closes-Bug: #1642515
This patch replaces python standard base64 library call to
oslo_serialization.base64 to follow OpenStack Python3 porting
standard [1].
Use base64 encoding takes 8-bit binary byte data and encodes it. On
Python3, A string is a sequence of Unicode characters thus base64 has
no idea what to do with Unicode data, it's not 8-bit[2]. We use
oslo_serialization.base64 for both python2 and python3.
[1] https://wiki.openstack.org/wiki/Python3
[2] http://stackoverflow.com/questions/8908287/base64-encoding-in-python-3
Change-Id: I1dafe878a71f37d223eaf58ea8e2e8c6b4422a14
This patch is replacing the "-ax" options of the ps command when
collecting the logs to "au". The "-x" option is not supported by the
BusyBox in the TinyIPA ramdisk.
Closes-Bug: #1599839
Change-Id: Id0c86db6aa16c63550890e1cd488994025f5dd68
The log extension is responsible for retrieving logs from the system,
if journalctl is present the logs will come from it, otherwise we
fallback to getting the logs from the /var/log directory + dmesg logs.
In the coreos ramdisk, we need to bind mount /run/log in the container
so the IPA service can have access to the journal.
For the tinyIPA ramdisk, the logs from IPA are now being redirected to
/var/logs/ironic-python-agent.log instead of only going to the default
stdout.
Inspector now shares the same method of collecting logs, extending its
capabilities for non-systemd systems.
Partial-Bug: #1587143
Change-Id: Ie507e2e5c58cffa255bbfb2fa5ffb95cb98ed8c4
This patch replace assertRaisesRegexp with
assertRaisesRegex which is deprecated in python3
https://docs.python.org/3.2/library/unittest.html
Also it update the base tests to be oslotest
BaseTestCase for python2.7 and python3 compatibility
Change-Id: I02571946f0643247e208d98dc91ea78cd9d351ee
Tests should use:
self.assertIn(value, list)
self.assertNotIn(value, list)
instead of:
self.assertTrue(value in list)
self.assertFalse(value in list)
because assertIn and assertNotIn raise more meaningful errors:
self.assertIn(3, [1, 2]
>>> MismatchError: 3 not in [1, 2]
self.assertTrue(3 in [1, 2])
>>> AssertionError: False is not true
Change-Id: I2d1c78fe71fe03e350b1035123b0a48b7186a6ec
Closes-Bug: #1510007
In order to support a more complex syntax for root device hints (e.g
operators: greater than, less than, in, etc...) we need to stop relying
on the kernel command line for passing the root device hints. This patch
changes this approach by getting the root device hints from a cached
node object that was set in the hardware module.
Two new functions: "cache_node" and "get_cached_node" were added to the
hardware module. The idea is to facilitate the access to a node object
representation from the hardware extension methods without changing
method signatures, which would break compatibility with out-of-tree
hardware managers.
Note that the new "get_cached_node" is just a guard function to
facilitate the tests for the code.
The function parse_root_device_hints() and its tests were removed since
it's not used/needed anymore.
Partial-Bug: #1561137
Change-Id: I830fe7da1a59b46e348213b6f451c2ee55f6008c
This patch enables virtual media deploy even if the expected virtual
floppy device name "/dev/disk/by-label/ir-vfd-dev" is capitalized to
"/dev/disk/by-label/IR-VFD-DEV".
This change is necessary because some bare metal hardware such as
FUJITSU RX1330 series create the capitalized virtual floppy device
name.
Change-Id: I436b3f9524a83814e082f8dd421df3e53b33f792
Closes-Bug: #1541167
Fix params order to correspond to real signature:
assertEqual(expected, actual)
Change-Id: I129aad5d8bb7ca0541c07913cb5edd8eef9e90ba
Closes-Bug: #1277104
Bring ironic-python-agent in line with the other ironic projects.
Stop ignoring all E12* errors except E129
Stop ignoring E711
Change-Id: Icb9bc198473d1b5e807c20869eb2af7f4d7ac360
Adds a new module ironic_python_agent.inspector and new entry point
for extensions, which will allow vendor-specific inspection.
Inspection is run on service start up just before the lookup.
Due to this early start, and due to the fact we don't even know
MAC address of nodes on inspection (to say nothing about IP addresses),
exception handling is a bit different from other agent features:
we try hard not to error out until we send at least something to inspector.
Change-Id: I00932463d41819fd0a050782e2c88eddf6fc08c6
