88 Commits

Author SHA1 Message Date
Uros Orozel
9a95e14512 Include partiton name and flags from parted output
Currently list_partitions method fails if partition
name is defined on any GPT partition, it also only
extract one partition flag.

Updated _PARTED_PRINT_RE is now set to capture
partition name/description and partition flags.

Change-Id: I40410622829453a485aa33047848922ab134d725
Task: 30250
Story: 2005322
2019-03-30 06:11:35 +00:00
Dmitry Tantsur
653b59a09d Run sync and partprobe after adding a configdrive partition
We're hitting a problem in the CI where /dev/sda2 does not exist after
being created. The logs suggest running partprobe may fix the problem,
so let's try it.

With this, we can also encounter a case where there refresh will fail
because the partition was created using fsync calls to force the write
of the partition table. However, filesystem VFS layer does impact this
behavior, and fsync on files in a ramdisk is a noop, so we must force
everything to be flushed to disk before attempting to execute partprobe.

Also run 'sgdisk -v' after partitioning to leave some traces in the logs
for debugging when something goes wrong.

Story: #2004744
Task: #28815
Change-Id: I9db48a3462422749290bbb887c14816734ab0478
Depends-On: I8d6f8c99e1ab5f26e3537630d0f6086e02a2b1ec
2019-01-09 10:40:28 -08:00
Zuul
dd7e1fec83 Merge "Allow to return unpopulated root partition from work_on_disk" 2018-10-11 17:31:54 +00:00
Dmitry Tantsur
38f5193cf9 Allow to return unpopulated root partition from work_on_disk
To support streaming raw partition images we need to fetch the image
directly from HTTP. To avoid copying this logic from IPA to ironic-lib,
enable work_on_disk to skip flashing the image by passing image_path=None.
To simplify life for the caller, return partitions dictionary in result.

Change-Id: I3189a2de115ae4bf47ad911161c2f5f3f062ef3c
Story: #2003809
Task: #26760
2018-10-09 11:59:41 +02:00
Dmitry Tantsur
8f6870b451 Prevent configuration changes in unit tests from affecting each other
Currently some of our tests set overrides, but nothing restores them.
This changes adds a config fixture to the bass class and also switches
some tests to using set_override instead of direct assignment.

Change-Id: I178061608e407047fb87d547c277ea38db353c26
Story: #2003891
Task: #26758
2018-09-27 12:52:31 +02:00
Zuul
66b098ee6d Merge "Accepts option for conv flags in dd command" 2018-09-21 11:15:15 +00:00
Yolanda Robla
217edfeca5 Accepts option for conv flags in dd command
When using dd command, it may be needed to specify conversion
options, such as sparse for optimization. This change allows to
accept this option, in preparation for ironic to sent it when
enabled on configuration.

Change-Id: I166abbf513557a836de3cce4fc9ea6b7fedbb562
Story: #2003755
Task: #26443
2018-09-21 09:39:51 +02:00
Zuul
1562dcf9de Merge "Check GPT table with sgdisk insread of partprobe" 2018-09-21 05:28:19 +00:00
Yolanda Robla
65b3df1aa8 Make search for config drive partition case insensitive
Currently the search for config drive is just done for config-2
in lowercase. If we pass the label with vfat, the label will
be in uppercase by convention. So the comparison will fail, and
this needs to be case insensitive to work.

Change-Id: I1dd196841fc3599333d68459eac606bd3b104b7b
Story: #2003547
Task: #24834
2018-08-24 10:27:15 +02:00
cheng
b9a109c65f Check GPT table with sgdisk insread of partprobe
On centos7.2, it doesn't happen every time, but sometimes, partprobe
fails with error `Device or resource busy` in _fix_gpt_structs. In fact,
this is not a problem, I tried ignoring this error and ran remain code.
Things went well. The target of running `partprobe` is to check GPT
table, we can run `sgdisk -v` instead.

This patch is to replace `partprobe` command in _fix_gpt_structs to
avoid `Device or resource busy` error.

Change-Id: I43182f17b1c6229132814313f7582ab30245f6bb
2018-08-22 18:39:31 +08:00
Zuul
344161bd50 Merge "Fall back to PARTUUID if UUID returns nothing." 2018-07-18 10:28:45 +00:00
Zuul
fbb8ce58aa Merge "Add logic to create PReP partition for ppc64* arch" 2018-07-17 10:06:30 +00:00
Michael Turek
2b540a00bb Add logic to create PReP partition for ppc64* arch
When booting a partition image localy, ppc64* expects an 8 MiB
partition, with the prep and boot flags set, containing the
bootloader [0]. This patch adds logic to ironic-lib to create
the parition with these flags when the node has a ppc64* arch.

[0] https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/PowerLinux%20Boot%20howto

Change-Id: I8f9748dd58146bfb2411c229b02969e0faf18222
Story: #1749057
Task: #22998
2018-07-16 14:26:50 +00:00
Matthew Thode
ca1fe71cf9 Fall back to PARTUUID if UUID returns nothing.
blkid does not always return a UUID when querying partitions (sda2 in my
case).  It does seem to return a PARTUUID though, so use that if needed.

Change-Id: I6f8dbd8fb8a02f6246b7ab00801159544f3288d6
Story: 2002052
Task: 19701
2018-07-16 06:36:14 -07:00
Olivier Bourdon
a809787783 Expose GPT partitioning fixing method
As suggested by TheJulia, the `_fix_gpt_structs` code
which is currently not accessible/callable from outside of ironic-lib
should be exposed so that other code like ironic-python-agent
can use it such as https://review.openstack.org/#/c/568524/

This patchset now exposes this code as `fix_gpt_structs`
(no leading underscore).

Change-Id: Icd824982d23966605fb179270bbf428cab23a41d
2018-07-02 11:39:03 +02:00
Michael Turek
4e31bc5a77 Remove unneccessary lambda from _test_make_partitions
There's a lambda here that seems to just return the addition
of two ints. Replacing it with a simple addition seems to produce
the same result.

Change-Id: I4b188986407c4ab83848d392e76ea71c95740737
2018-06-07 17:34:53 +00:00
fpxie
d2ade05278 Add retry attempts for the partprobe command
The partprobe command may fail because the device is busy some
times. Add retry attempts to resolve this problem.

Closes-Bug: #1756760
Change-Id: Ic888e7b6c93e83511a76bf8000619531b1eec9e1
2018-04-04 15:53:46 +08:00
John L. Villalovos
cdea1086d6 Use six.wraps() for Metrics so decorated methods can be inspected
Previously it was using functools.wraps() to create the decorators.
The problem is that we can't use the
oslo_utils.reflection.get_signature() function to get the correct
signature.

Change it to use six.wraps() which will add the '__wrapped__'
attribute which will be used when calling get_signature() and return
the signature of the decorated function.

Added unit tests to show we are able to see the signature of a wrapped
function.

Closes-Bug: #1746730
Change-Id: I75428e948b64b3b6758d31678a80d158a11c6beb
2018-02-01 08:47:01 -08:00
John L. Villalovos
ce0655bcda Add missing 'self' parameter to class methods
The 'self' parameter was missing in the class methods of the
DoNotCallPopen class. Add the 'self' parameter to the class methods.

Change-Id: I29bed962a97c8b28b3feb266d76243da4b4cab7a
2018-01-24 12:17:44 -08:00
John L. Villalovos
f3279dffe2 Simplify logic in wait_for_disk_to_become_available
* Simplify the logic in utils.py/wait_for_disk_to_become_available
 * Correct unit tests to not use None as a return value for calls to
   utils.execute() that return no output. In reality utils.execute()
   will return empty strings for example: ('', '')
 * Add unit tests to test 'psmisc' and 'busybox' versions of output
   data from 'fuser'
 * Add a shorter check device interval as test was taking over 2
   seconds and showing up in longest test run time list.

Change-Id: Ibc481e4a76c63612e1774b5c7a8144ee934eb2f5
2018-01-17 13:56:16 -08:00
Nikolay Fedotov
0d65451cfc Add wait_for_disk to destroy_disk_metadata function
* Move _wait_for_disk_to_become_available function
  from DiskPartitioner class to utils.py
* Call the wait_for_disk_to_become_available during
  the destroy_disk_metadata metadata to ensure that
  nothing is using disk and preventing metadata
  from being destroyed.

Change-Id: I9134b4cfdf15ba9610b4d1f844e64858d69713a5
Closes-Bug: #1739421
2018-01-17 08:45:36 -08:00
John L. Villalovos
35eb2af3f4 Fix issue with double mocking of utils.execute functions
An issue was discovered if we mock an already mock-ed function. This
was happening in our execute() detection code. Change it to not use a
mock and instead a function.

Add test cases to ensure it works as expected.

Change-Id: If58071b41c7e69bc182ea033bf647924f996f1f3
2017-12-11 07:26:42 -08:00
John L. Villalovos
5d1d7c4cc7 Update version of flake8-import-order package
Use latest version of flake8-import-order package, like we do in
openstack/ironic

Fix two issues detected by new version.

Change-Id: I3d2f5499f3a863167419469d313e06be79f51841
2017-12-08 14:13:32 -08:00
Ilya Etingof
37eccedd84 Added by_path root device hint
With the introduction of the `by_path` introspection
attribute at the IPA [1], it is now possible to leverage
the stable block device names for root device hinting.

This patch adds the `by_path` attribute to the collection
of valid root device hints.

1. https://review.openstack.org/#/c/498489/

Change-Id: I6f1c6b5ec5face7180aeb2d1aeebe50a6eb33f24
2017-09-05 22:46:48 +02:00
Jenkins
af06d06da5 Merge "Adjust test with option 'backend'" 2017-06-19 13:59:52 +00:00
ChangBo Guo(gcb)
f0a2e9791a Adjust test with option 'backend'
Config option 'backend' with parameter choices =['noop', 'statsd'],
that means oslo.config will ensure value is one of them, otherwise
a ValueError will be raised, so don't need handle and test config
option 'backend' with invalid value.

Change-Id: Ibda809416c565d57f19179d7f13f4cee8f8145ff
2017-06-16 11:26:02 +08:00
Jenkins
56e7619366 Merge "Support for NVMe drives for config drives" 2017-06-15 13:11:44 +00:00
Ramamani Yeleswarapu
af50acd65a Support for NVMe drives for config drives
NVMe drives use different device naming convention for partitions:

> lsblk /dev/nvme0n1
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
nvme0n1     259:0    0  1.8T  0 disk
|-nvme0n1p1 259:2    0  1.3G  0 part
`-nvme0n1p2 259:1    0   64M  0 part

The ironic-python-agent fails to provision NVMe drives,
because it does not expect the 'p' letter before the partition number.

This patch fixes that.

Change-Id: I5061f6d5b00f404d4aaadc6b18675039ba2ed854
Closes-Bug: #1695265
2017-06-08 10:11:54 -07:00
lingyongxu
9ab65835e5 Remove unnecessary setUp function in testcase
In testcase, setUp will be called automatically. This patch used to
remove setUp functions that do nothing. Besides, it will keep code clean.

Change-Id: I015e4faa2e185a49609078c281f449c65a0d8b27
2017-06-08 17:16:29 +08:00
Dmitry Tantsur
4491a52c90 Add 'sgdisk -Z' to destroy_disk_metadata
Apparently, wipefs still leaves some metadata around, which can break ceph.
Use sgdisk's zapping feature to be absolutely sure.

Change-Id: I51aea8fb5ba02d63a05a3794405caf000ff28e75
Closes-Bug: #1690458
2017-05-30 17:05:22 +02:00
John L. Villalovos
61a0c2582b Prevent tests from using utils.execute()
This change introduces a new base test class that mocks out
utils.execute() and similar functions and forces an exception if it
gets called. If utils.execute() is mocked by the test case then it
will work. What this does is prevent un-mocked access to
utils.execute() / processutils.execute() and similar subprocess
library functions.

Inspired by Julian Edwards' patch to ironic-python-agent

Change-Id: Ie5bda8f4c65cf4dfb40b91c11b61485c954addde
2017-05-17 09:21:26 -07:00
John L. Villalovos
e36a11a7c9 Remove unit tests that test oslo_concurrency.processutils.execute
Remove the unit tests that were testing the functionality of
oslo_concurrency.processutils.execute. It is not our job to unit test
3rd party libraries.

Change-Id: I1516af623346a590e25f4f09226c784031ca78a7
2017-05-11 12:51:00 -07:00
Javier Pena
87c196d670 Use enforce_type=False in test_nonexisting_backend
Config option metrics/backend only accepts values "noop" or "statsd" [1],
however test_nonexisting_backend() checks for a different value.
oslo.config 4.0 will check overridden value type by default[2], so we
need to set enforce_type in CONF.set_override to allow unit tests to
pass.

[1] 36109e58f2/ironic_lib/metrics_utils.py (L25-L28)
[2] https://review.openstack.org/#/c/328692/

Change-Id: Ia597da0d03cfcc5f34b271a12cc42600c6a6a2d3
2017-04-18 13:34:48 +02:00
Jenkins
ccfb67bb82 Merge "Test that configdrive disk exist with test -e" 2017-03-23 12:00:40 +00:00
Vasyl Saienko
810257d2d1 Test that configdrive disk exist with test -e
We waiting for device is ready by using 'udevadm settle
--exit-if-exist=/dev/blah' which should wait for disk before
start writing image. In the bug attached we can see that,
dd complains that target device doesn't exit.
When tested locally udevadm returns 0 when target device doesn't
exist:

$ udevadm settle --exit-if-exists=/dev/blah; echo $?
0
$ ls /dev/blah
ls: cannot access '/dev/blah': No such file or directory

This patch replaces calls 'udevadm settle' without arguments that waits
until udev event queue is empty.
Also add simple 'test -e' check, with 15 retries and random delay
(0.2 - 2sec) between each retry [0], to wait for configrive device.

[0] https://github.com/openstack/oslo.concurrency/blob/7afbc99/oslo_concurrency/processutils.py#L439
Change-Id: I6d2d485dd543fa8975187862d3e4b8d6567e0224
Closes-Bug: #1673731
2017-03-21 16:47:53 +02:00
John L. Villalovos
866a34ccab Add missing 'autospec' statements to unit test mocks
Add missing 'autospec' statements to unit test mocks. This helps to make
sure we don't accidentally call invalid assert methods.

Change-Id: I8dd2a3165b983dcc105e412717453b3bf3293546
2017-03-18 22:12:48 -07:00
John L. Villalovos
30c946d330 Use flake8-import-order
Use the flake8 plugin flake8-import-order to check import ordering. It
can do it automatically and don't need reviewers to check it.

Change-Id: Ib3fde54d8baa2c6e3567ff3f1a902de4b25f9f70
2017-02-16 09:51:31 -08:00
Lucas Alvares Gomes
4ae48d0b21 Limit memory usage when running qemu-img
With multiple qemu-img processes running at the same time the host machine
can quickly run out of memory. This patch is limiting the memory space
to 1 GiB when invoking qemu-img.

Change-Id: If791452edc38c4732ba2b96220e7969c72b234dd
Closes-Bug: #1657808
2017-01-19 16:57:59 +00:00
Ruby Loo
4e167186b1 Fix exception for multiple devices with same label
This fixes the exception message that is raised when there are
multiple devices with the same label. Instead of including the
original device path, it was showing one of the device's name
& label.

The message now includes the original device path along with
the two devices that have the same label.

Change-Id: Id114b25e95492686bc6679e217d4963d6a3ed0c4
Related-Bug: #1654269
2017-01-12 19:29:06 +00:00
Jenkins
7ae45cfe2e Merge "Fix retrieval of config-2 existing partition" 2017-01-12 08:13:40 +00:00
Yolanda Robla Mota
2d8f1d91c7 Fix retrieval of config-2 existing partition
Previously it was checked using blkid -t LABEL=config-2, but
this was not finding the partition, causing that images with a
pre-existing config-drive partition had an extra one created
by ironic.

Switched to using the 'lsblk' command and parsing the output of that to
get the information on the partition.

Change-Id: Ic5228ce91d1a133fd846ceb7be547a227292dd38
Closes-Bug: #1654269
Depends-On: I3676cd8e5c5e61fe4e3f67ab9e121c4cd0cd599b
2017-01-11 14:55:40 -08:00
Jay Faulkner
2f0ea84488 Create configdrive partitions as vfat
This fixes a bug where ironic-lib would create configdrive partitions as
ext2, which is incompatible with any OS but Linux and goes against the
configdrive standard as written in cloud-init docs.

Change-Id: Ib4ad8e0b4d6dbef9d441d9ae13fd82f8f51957e6
Closes-bug: #1655389
2017-01-10 08:50:21 -08:00
Luong Anh Tuan
2a3288c69c Use oslo_serialization.base64 to follow OpenStack Python3
This patch replaces python standard base64 library call to
oslo_serialization.base64 to follow OpenStack Python3 porting
standard [1].

Use base64 encoding takes 8-bit binary byte data and encodes it. On
Python3, A string is a sequence of Unicode characters thus base64 has
no idea what to do with Unicode data, it's not 8-bit[2]. We use
oslo_serialization.base64 for python2 and python3.

[1] https://wiki.openstack.org/wiki/Python3
[2] http://stackoverflow.com/questions/8908287/base64-encoding-in-python-3

Change-Id: I9d453ab6ebc826cad60c89d2a74c5764626eba01
2016-12-15 10:35:41 +00:00
Lucas Alvares Gomes
1411e3d315 Check for HCTL as part of root device hints
This patch is adding support for comparing the HCTL (Host, Channel,
Target, Lun) attribute of a block device as part of the root device
hints mechanism.

Partial-Bug: #1648036
Depends-On: If8897c68609e0df0378ee919b803ca5e497def02
Change-Id: I2d4a8b2dcef5cab95c27be12ddb8d08b62896869
2016-12-07 11:46:25 +00:00
John L. Villalovos
337ae96819 Add missing autospec=True to mocks
Some mocks have snuck in without having autospec=True. Add that to
them.

Change-Id: I79f8d68c78446f384e1b455b2835ef34602fd591
2016-10-20 08:23:53 -07:00
Jenkins
e282b96be8 Merge "Add more tests for work_on_disk" 2016-10-20 13:55:20 +00:00
Vladyslav Drok
939c026950 Wait for a config drive device to be available
A failure may occur if the dd to config drive partition happens
before it actually becomes available. This change adds a udevadm
settle call that ensures that config drive device is present in
/dev folder.

Closes-Bug: #1633063
Change-Id: I5526f791371e7a6f4cdad151bb163707504e76cf
2016-10-19 16:37:45 +03:00
Mario Villaplana
40e2765690 Add more tests for work_on_disk
ironic has some test cases that check the internals of what ironic-lib
does. This adds some test cases to ironic-lib's work_on_disk method to
ensure coverage will stay up to par when those tests are purged from
ironic proper.

Change-Id: If9499982156e349056cffd87d62d1da172494204
Related-Bug: 1552843
2016-10-18 17:58:44 +00:00
Lucas Alvares Gomes
17d1b06c13 Correctly count the number of primary partitions
This patch is adding a new method called count_msdos_partitions()
responsible for counting the number of primary and logical partitions
in a MBR partition table.

Prior to this patch, the code checking if there were more than 4 primary
partitions present in the partiton table was using the list_partitions()
method which do not distinguish between the types*. So, even if the disk
had 2 primary and 5 logical partitions the check would fail thinking
that 7 primary partitions were present (which is impossible btw).

This patch make use of "partprobe" to count the partitions, which is
already required in ironic-lib.

* Not distinguishing types is a pro on the list_partitions() method
because primary and logical does not apply for others disk labels such
as GPT.

This patch also fix a small typo in a non-related string :-)

Change-Id: I7042cd12775b0d715c15f466db2d4c5dc9e1a60a
Closes-Bug: #1629926
2016-10-05 14:41:38 -03:00
John L. Villalovos
c10121bc26 Remove unneeded disk_utils.mkfs() function
There is no need for the disk_utils.mkfs() function. Remove it and
update code to not use it.

Change-Id: I5609a8b80d23af23823ef0f0d7143efb209ef324
2016-09-22 17:49:47 -07:00