693 Commits

Author SHA1 Message Date
Zuul
12b62d6c3a Merge "Collect lsblk and /proc/mdstat with ramdisk logs" 2020-01-10 09:22:29 +00:00
Julia Kreger
de90f54b9b Permit hashed passwords to be sent to IPA.
Change-Id: Icfbad84a5a456ea06876c4829400b2f626fb24fe
Story: 2006777
Task: 37300
2019-12-19 11:19:48 -08:00
Zuul
436d857593 Merge "Software RAID: Use UUID to find root fs" 2019-12-19 02:55:36 +00:00
Julia Kreger
8aa9b79c46 Stop the API so rescue can exit and succeed
The change in frameworks with the API changed the overall execution
behavior from single threaded serialized execution threaded to
multithreaded through eventlet.

But we signaled ourselves to wait for the API to stop, but never
actually stopped the API.

As setting the flag to stop the API is a hard notification that
the API must now exit, there is no reason to wait and block exit,
hence the simple nature of this change.

Now the threads should exit, IPA should exit, and rescue should
ideally work.

Change-Id: I3faf2d169d4edac6a2321f8649bb9ce478885ea1
Story: 2006998
Task: 37768
2019-12-07 09:45:32 -08:00
Zuul
33fa473551 Merge "Replace WSME and Pecan with Werkzeug" 2019-12-06 19:12:18 +00:00
Iury Gregory Melo Ferreira
966356e58c Search for efi partition
This patch adds a function that will be responsible to identify
the efi partition on a give device, this is necessary on the Software
Raid scenario and when installing bootloader.

Change-Id: I5f326db2d37b2a15090ec84e477e63f7d92e7447
Co-Authored-By: Raphael Glon <raphael.glon@corp.ovh.com>
2019-12-04 20:09:59 +01:00
Dmitry Tantsur
f1b2df908a Replace WSME and Pecan with Werkzeug
WSME is no longer maintained and Pecan is an overkill for our (purely
internal) API. This change rewrites the API in Werkzeug (the library
underneath Flask). I don't use Flask here since it's also an overkill
for API with 4 meaningful endpoints.

Change-Id: Ifed45f70869adf00e795202a53a2a53c9c57ef30
2019-12-04 16:50:47 +01:00
Arne Wiebalck
13a26553e5 Software RAID: Use UUID to find root fs
Rather than assuming that the root fs is in the first partition
of the deployed image, use the UUID passed from the conductor.

Change-Id: Ie2299372e94386902e0a8be0597250ec52e1fec9
Story: #2006649
Task: #36887
Depends-on: https://review.opendev.org/686580
2019-12-04 15:32:50 +01:00
Zuul
6032643a04 Merge "Stop using six library" 2019-12-03 13:09:30 +00:00
Dmitry Tantsur
4354bc04f9 Replace netaddr dependency with stdlib ipaddress
netaddr is quite a big library, and all we need is covered by
the built-in ipaddress module (available in python 3).

Also add a safeguard for invalid 'ip route' output.

Change-Id: I9d76a8d1c1b6b1585e301a9c63b37fab3b98746f
2019-12-02 12:13:04 +01:00
Riccardo Pittau
ca7a46b113 Stop using six library
Since we've dropped support for Python 2.7, it's time to look at
the bright future that Python 3.x will bring and stop forcing
compatibility with older versions.
This patch removes the six library from requirements, not
looking back.

Change-Id: I4795417aa649be75ba7162a8cf30eacbb88c7b5e
2019-11-29 10:18:14 +01:00
Zuul
8e246f4482 Merge "RAID 5/6" 2019-11-28 11:18:02 +00:00
Julia Kreger
e4659c94cf RAID 5/6
Looks like adding RAID 5/6 support may be easier than most
could immagine. The code, as written appears to be safe and
logical creating a RAID 5 or RAID 6 volume.

Not that we can really test this in CI, but it seems only
validation code needs to be changed to loosen the constraint.

Change-Id: Ib891b3c97f0bfb02af3b59581a451c4b25e03b85
2019-11-27 20:30:28 +00:00
Kaifeng Wang
4097847a10 Clean up options deprecated prehistory
Configuration options like api-url had been deprecated long time ago[1],
this patch removes it.

[1] https://review.opendev.org/#/c/131632/

Change-Id: Ie448b35a4423066ef44dca7616e716cb5c118881
2019-11-19 08:46:21 +08:00
Zuul
60637c02c6 Merge "Bump up ipa-ip-lookup-attempts to 6" 2019-11-18 10:53:58 +00:00
Derek Higgins
c4bb694082 Bump up ipa-ip-lookup-attempts to 6
To accommodate network setup that takes longer then
30 seconds increase the number of IP lookup attempts. This
allows for IPv6 setup that occurs after DHCP has time out.

Change-Id: I1351e150a63c6247210ca0cbc8ce0abfe82129cd
2019-11-12 03:38:38 +00:00
Dmitry Tantsur
1d12505fa0 Do not try to read raw disk data as unicode
Fixes regression in commit 9f8fa2853af00045118ccd44180e7053e6d027af:
utils.execute returns unicode by default, but this data is binary.

Change-Id: I5b54c1f6cfac5fff672245c523b9fb647478edbd
2019-11-11 16:03:59 +01:00
Julia Kreger
9f8fa2853a Don't try to put a bootloader in place when bootable
Lets not do silly things and if the disk looks bootable,
and we're not trying to do UEFI, then let us assume the
proper thing will occur upon power-up.

Looks at the boot sector data and if an executable is
found in the first 218 bytes, then it bypasses loading
a boot loader.

Also adds a dependency on the "file" linux distribution
package.

Change-Id: I11bc26670a08ee13174a43d7cd0f1ab9c1bd35cf
Story: 2006474
Task: 36410
2019-11-01 13:58:55 +00:00
Julia Kreger
c5956bdada Suppress errors from iscsi session cleanup
In some cases, where the rts library is not installed, IPA
was recently changed to try and tear down the local side of
the iscsi connection by trying to tear down bond and target
being offered. The whole attempt with this is to ensure that
no disk locks are in place which can prevent partition table
updates.

Since we added this logic, in some cases these commands can
fail and cause the deployment process to fail when it would
have otherwise succeeded. As such, suppress the errors.

Change-Id: I0e04936ad337b394dd68e9b0396a9f1203218f9f
2019-11-01 06:57:58 -07:00
Zuul
358a46c4fe Merge "[Trivial]: Typo in comment for _get_partition" 2019-10-29 15:02:44 +00:00
Zuul
a158b862ea Merge "Set rd.md.uuid kernel parameter when deploying on software raid" 2019-10-28 11:43:04 +00:00
Arne Wiebalck
cb96cec794 [Trivial]: Typo in comment for _get_partition
Change-Id: I29a3ea86dd710fc206d9c25cb12b08324538d224
2019-10-25 18:48:16 +02:00
Zuul
20abc992fb Merge "Correct string formatting in logging" 2019-10-18 09:51:54 +00:00
Dmitry Tantsur
d61887e744 Correct string formatting in logging
Change-Id: Ibc3656e32b94004da7e895a27860c1fa3ac0bbb1
2019-10-17 16:14:10 +02:00
Andrei Nistor
1975478097 Set rd.md.uuid kernel parameter when deploying on software raid
When deploying an image to a software raid array, it is currently
required that the deployed image assembles the md arrays automatically
so that the rootfs can be mounted. In order to remove this
requirement/limitation on the deployed image we can add rd.md.uuid to
the kernel command line with the raid array's uuid.

Story: 2006648
Task: 36884
Change-Id: I42cb198753ecd84b7eaef6b5aa7c2064535bfe0e
2019-10-17 11:14:04 +00:00
Julia Kreger
3ee17e8624 Last resort fallback to find a partition
Falls back to attempt to use findfs to locate
a UUID or PARTUUID match as opposed to trying to
list and enumerate through lsblk output.

Can confirm that tinycore 8.x's findfs binary works
expected.

Story: 2006724
Task: 37141

Change-Id: I4d488e4a1ab680eb1353b158c3339cb30b056ada
2019-10-15 16:35:17 +00:00
Dmitry Tantsur
11976c9d2b Collect lsblk and /proc/mdstat with ramdisk logs
This should improve debugability of partitioning problems.

Change-Id: I3c7ae3f2831c9900a3f0d24daec6dd6b8bea6a60
2019-10-14 15:28:08 +02:00
Dmitry Tantsur
386654f602 Fix compatibility with Pint 0.5
Conversion from Quantity to int/long was added only in 0.5.2.

Change-Id: I23f93d29a0624b77f4624c6af530971eb6a4c01b
2019-09-25 11:53:11 +02:00
Zuul
74c8cd581b Merge "Attempt tgtd session detach" 2019-09-24 12:00:00 +00:00
Zuul
3968ec9d5a Merge "Delete_configuration, consider removed raid members as well" 2019-09-24 11:51:06 +00:00
raphael.glon
316dd3f952 Revert "Software raid: mbr/gpt partition table alternative"
This reverts commit 258d963e406c512bb90295c700ee22e4609abcd0.

Remove the mbr/gpt choice from softraid features for now, as it cannot be
directly used without additional commits, to be pushed soon. Furthermore, it
could even lead to instance spawn misconfiguration, if the disk label
specified in instance_info cannot fit with the boot mode + partitioning layout
(example: you build softraid over gpt, then you lose the mbr gap. Thus you need
an additional bios boot partition, in BIOS boot mode, or an esp, in UEFI boot
mode, outside the raid, for grub to install itself inthere, to be able to
assemble raid and find root device correctly).

Change-Id: I3a0a704ea99a40eb3fc8e879270dfbd356951488
2019-09-23 12:26:37 +02:00
Julia Kreger
8be4cb689c Attempt tgtd session detach
With raid, we were able to observe a condition where
the file is still open for non-exlcusive access which
blocks the software raid from being shutdown... which
is realistically needed to force a rescan in that case.

Change-Id: I8dbfae091267f8af5340ff5ebeebcba375d05542
2019-09-19 15:44:42 -07:00
Raphael Glon
37385dd9dd Delete_configuration, consider removed raid members as well
Change-Id: Ie4f62d8855d3f30a55f7032918dfed1f1d8b5b31
Story: #2006535
Task: #36591
2019-09-16 11:27:27 +02:00
Zuul
6d3d1367a9 Merge "Handle floppy disk controllers" 2019-08-27 07:53:00 +00:00
Julia Kreger
b98bc678b9 Handle floppy disk controllers
Cleaning presently fails on floopy disk controllers.

While they may be uncommon in server hardware, they
can exist in virtual machines, and even as virtual
devices on some hardware chassises.

Change-Id: I8ba07bfd5ca1e503f46c1bac4fffb5f509186939
Story: 2006419
Task: 36309
2019-08-22 08:58:24 -04:00
Zuul
0eff77ac8d Merge "Softraid: partitioning fixes" 2019-08-21 11:58:22 +00:00
Zuul
187d5da057 Merge "Software RAID: Ignore missing component devices or holder disks" 2019-08-19 13:27:41 +00:00
Arne Wiebalck
9dca5d76b3 Software RAID: Ignore missing component devices or holder disks
With the addition of devices with type 'MD' during
list_all_block_devices (for "empty" devices), partitions on top
of md devices are now returned as well. These don't need deletion,
so skip these during software RAID cleanup.

Story: #2006355
Task: #36121

Change-Id: Idff9b9b0cbe42b57aa315ae07c0b51c35c9ade1e
2019-08-16 11:35:07 +02:00
Raphael Glon
8ad9267aa0 Softraid: partitioning fixes
Fixes:
- Softraid creation: size_gb unit conversion to default parted unit.
- Softraid creation issue when we specify several logical disks with
  more than one having its size specified (not 'MAX').
- Softraid creation issue when a logical disk with size_gb 'MAX' is
  not last in the list.

Change-Id: I13e087611354a35acc2925037f1b10047477d972
Story: 2006352
Task: 36118
2019-08-15 23:48:15 +02:00
Zuul
680143a4da Merge "Fixes get_holder disks with nvme drives" 2019-08-14 09:14:55 +00:00
Zuul
ac6542cab8 Merge "Software RAID: Handle empty target_raid_config" 2019-08-13 10:01:19 +00:00
Raphael Glon
c546749423 Fixes get_holder disks with nvme drives
Change-Id: I195ffdeeb3c13bdec5fc1735b82efa53c8d9d3de
2019-08-13 10:37:18 +02:00
Zuul
6b778e171b Merge "Fixes physical memory fetching for aarch64" 2019-08-12 16:02:44 +00:00
Arne Wiebalck
5dd548dc7b Software RAID: Handle empty target_raid_config
If the target_raid_config is empty, we should not try
to create software RAID devices.

Change-Id: I5a29dc34a07903be0cc1b19594f160ac63c6ed2d
2019-08-12 11:38:28 +02:00
Dmitry Tantsur
3d5fc82ece Ignore /dev/ramX devices for any purposes
While most modern Linux distributions do not create RAM disks, newer
TinyCoreLinux actually does. Ignore them since we don't want to deploy
to RAM.

Change-Id: Ia810c7fb47f7098b8173d765ff9c33b48a223ff1
2019-08-12 09:22:52 +02:00
Dmitry Tantsur
c7c307c497 Software RAID: try to assemble RAID on start-up
TinyIPA does not do it, which prevents deployment on RAID devices.

Story: #2004581
Task: #36196
Change-Id: I6c1d90b4669102ab59588ab15f7166626c4d72be
2019-08-09 13:28:25 +02:00
Raphael Glon
258d963e40 Software raid: mbr/gpt partition table alternative
GPT/MBR partition table type alternative on softraid creation.

Change-Id: Ib3d00034fe687328a1ea3e038f6f86d2746e63a0
2019-08-08 11:44:25 +02:00
Dmitry Tantsur
88796a8936 Do not try to use zRAM devices for anything
These devices are mapped into memory, but nonetheless can appear
in lsblk output. Ignore them explicitly.

Change-Id: I2bb595ef29abdc98bd887bdf41c3c7f64cc3a7e6
2019-08-07 16:10:51 +02:00
Raphael Glon
570584f9ab Adds software raid creation support on nvme drives.
Change-Id: Ia725fcaee6018255af3dd7ff0eb2fb1cfbd2b322
Story: #2006341
Task: #36091
2019-08-06 13:23:08 +02:00
liyingjun
cd50fd1151 Fixes physical memory fetching for aarch64
The output for aarch64[1] `lshw -quiet -json` is a bit different from
x86_64, this will lead to 0 physical memory calculated currently. This
patch aims to make it compatible for aarch64 when getting physical memory

[1]: http://paste.openstack.org/show/754789/

Change-Id: Ib7b4ee857d8550e3d75ac8a53c61f0ecab45feff
2019-07-30 15:51:17 +08:00