765 Commits

Author SHA1 Message Date
Zuul
bb730f37cd Merge "Check paths before extending device" 2023-08-25 16:32:11 +00:00
whoami-rajat
71627c56ac Fix unit tests when hostid file exists
After merging change I0b60f9078f23f8464d8234841645ed520e8ba655, we
noticed an issue with existing unit tests which started failing.
The reason is 'nvme_hostid' was an additional parameter returned
in the response while fetching connector properties from nvme
connector.
This is environment specific and won't occur in environments where
'/etc/nvme/hostid' file doesn't exist due to which these tests
passed in gate but failed in the local run when hostid file
was present.
This patch mocks the get_nvme_host_id method for tests so the
hostid is never returned irrespective of the environment.

Closes-Bug: #2032941
Change-Id: I8b1aaedfdb9bef6e34813e39dede9afe98371d2b
2023-08-24 15:28:35 +00:00
Rajat Dhasmana
1435059da5 Check paths before extending device
In a multipath enabled deployment, when we try to extend a volume
and some paths are down, we fail to extend the multipath device and
leave the environment in an inconsistent state. See LP Bug #2032177
for more details.

To handle this, we check if all the paths are up before trying to
extend the device and fail fast if any path is down. This ensures
we don't partially extend some paths and leave the other to the
original size leading to inconsistent state in the environment.

Closes-Bug: 2032177

Change-Id: I5fc02efc5e9657821a1335f1c1ac5fe036e9329a
2023-08-24 11:40:32 +00:00
Zuul
eb76f8283e Merge "NVMe-oF: Create /etc/nvme/hostid" 2023-08-24 08:35:34 +00:00
Zuul
94f1f6ef59 Merge "SCSI: Support non SAM LUN addressing" 2023-08-24 08:19:36 +00:00
Gorka Eguileor
16c90d5fe9 NVMe-oF: Create /etc/nvme/hostid
The NVMe-oF connector currently create the `/etc/nvme/hostnqn` file if
it doesn't exist, but it may still be missing the `/etc/nvme/hostid`
value.

Some distribution packages create the file on installation but others
may not.

It is recommended for the file to be present so that nvme doesn't
randomly generate it.

Randomly generating it means that the value will be different for the
same storage array and the same volume if we connect, disconnect, and
connect the volume again.

This patch ensures that the file will exist and will try to use the
system's UUID as reported by DMI or a randomly generated one.

Closes-Bug: #2016029
Change-Id: I0b60f9078f23f8464d8234841645ed520e8ba655
2023-08-23 19:02:42 +02:00
Gorka Eguileor
00caa73776 RBD: Improve close and flush in IOWrapper
On Change-Id Ib3b066a7da071b1c2de78a1a4e569676539bd335 we improved the
RBDVolumeIOWrapper's flush and close methods, and this patch improves
them even further.

If the IOWrapper's close is not explicitly called and it's just
dereferenced (happens in unit tests) then during Garbage Collection the
wrapped image may be destroyed before the wrapper, which would trigger
the image being closed without the wrapper knowing, so when the wrapper
gets destroyed it will fail because it calls its close method, which
calls its flush, which calls the underlying image's flush, which will
fail because the underlying image was already closed.

We need to check if the underlying image has already being flushed
before calling the flush.

Calling the underlying close method for the Image or IOWrapper classes
is not a problem because they are idempotent.

Change-Id: Ib5a517d58427df0d1d1b22ad3dc66f673da147fe
2023-08-23 12:33:37 +02:00
Gorka Eguileor
59961647d3 SCSI: Support non SAM LUN addressing
This patch adds non SAM addressing modes for LUNs, specifically for
SAM-2 and SAM-3 flat addressing.

Code has been manually verified on Pure storage systems that uses SAM-2
addressing mode, because it's unusual for CI jobs to have more than
256 LUNs on a single target.

Closes-Bug: #2006960
Change-Id: If32d054e8f944f162bdc8700d842134a80049877
2023-08-23 12:19:11 +02:00
Zuul
0a5d4c1506 Merge "Fix multipath resize map with friendly names" 2023-08-17 00:56:26 +00:00
Rajat Dhasmana
75714711f2 Fix multipath resize map with friendly names
When multipath is enabled and friendly names are ON and we try
to extend a volume, we pass the SCSI WWID to the multipath_resize_map
method which is not correct.

There are 2 things we can pass to the multipathd resize map command:

1) Multipath device (eg: dm-0)
2) Alias (eg: mpath1) or UUID (eg: 36005076303ffc56200000000000010aa)

The value should be an alias (mpath1) when friendly names are ON
and UUID (36005076303ffc56200000000000010aa) when friendly names
are OFF. However, we only pass the UUID irrespective of the value
set for friendly names.

This patch passes the multipath device path (to multipathd resize
map command) which is the real path of the multipath device (/dev/dm-*).
This fixes the issue as it passes the same value irrespective of if
the friendly names are ON or OFF.

-> self.multipath_resize_map(os.path.realpath(mpath_device))

(Pdb) mpath_device
'/dev/disk/by-id/dm-uuid-mpath-3600140522774ce73be84f9cb9537e0c9'

(Pdb) os.path.realpath(mpath_device)
'/dev/dm-5'

Closes-Bug: 1609753
Change-Id: I1c60af19c2ebaa9de878cd07cfc0077c5ea56fe3
2023-07-27 21:12:52 +00:00
Zuul
bfb55b7b58 Merge "nvmeof: Call findmnt with -v" 2023-07-21 17:07:17 +00:00
Eric Harney
edc1bc433b nvmeof: Call findmnt with -v
The -v arg suppresses printing of [/dir] with the
device for bind mounts and btrfs volumes, which is
what we want for this usage.

This fixes _get_host_uuid() failing when using
a btrfs rootfs.

Closes-Bug: #2026257
Change-Id: I2d8f24193ecf821843bf8f4ea14b445561d6225c
2023-07-12 12:23:39 -04:00
Zuul
6d70ace1bb Merge "linuxrbd: Remove rados_connect_timeout parameter" 2023-07-10 08:35:56 +00:00
Zuul
5c840f2815 Merge "mypy: Fix failing mypy job" 2023-07-07 19:02:47 +00:00
Zuul
f7432b56f0 Merge "LVM: Remove outdated comment" 2023-07-03 16:29:38 +00:00
Eric Harney
4d41c2986c mypy: Fix failing mypy job
Currently fails w/ "Unused type: ignore comment" errors.

Change-Id: I7cced8b9d8704c1782ac8583ce227efcc21b2847
2023-06-30 08:59:19 -04:00
Zuul
8daafd376b Merge "Support force disconnect for FC" 2023-05-10 20:03:25 +00:00
Zuul
f7adadef4f Merge "Revert "Fix iSCSI disconnect_volume when flush fails"" 2023-04-27 15:56:55 +00:00
Brian Rosmaita
1b739ed2d5 Revert "Fix iSCSI disconnect_volume when flush fails"
This reverts commit 8070ac3bd903a443fbfd02abf8b0554d5d05cac1.

Reason for revert: This requires some more discussion, I should not have ninja-approved it.

Change-Id: I25917b95a32da4fd831d669cd21988f400f258e0
2023-04-27 12:19:16 +00:00
Zuul
7f780b62f5 Merge "Fix iSCSI disconnect_volume when flush fails" 2023-04-26 23:21:09 +00:00
Gorka Eguileor
570df49db9 Support force disconnect for FC
This patch adds support for the force and ignore_errors on the
disconnect_volume of the FC connector like we have in the iSCSI
connector.

Related-Bug: #2004555
Change-Id: Ia74ecfba03ba23de9d30eb33706245a7f85e1d66
2023-04-24 11:39:37 +02:00
Rajat Dhasmana
8070ac3bd9 Fix iSCSI disconnect_volume when flush fails
The purpose of providing force=True and ignore_errors=True
is to tell os-brick that we want to disconnect the volume
even if flush fails (force) and not raise any exceptions
(ignore_errors). Currently, in an iSCSI multipath environment,
disconnect_volume can fail when both parameters are True.

The current flow when disconnecting an iSCSI volume is
that if flushing a multipath device fails, we manually
remove the device, logout from the target portals,
and try the flush again.

There are two problems here:

1) code problem: The second flush is not wrapped by
ExceptionChainer. This causes it to raise the exception
immediately after flush fails irrespective of the value
of the ignore_errors flag.

2) conceptual problem: In this situation, there is no point
in making the second flush attempt. Instead, we should just
remove the multipath map from multipathd monitoring since
we have already removed the paths manually.

This patch fixes the conceptual problem, as we don't make a second
flush call and ignore any errors on the execution ``multipathd del map``
thereby also fixing the code problem.

Closes-Bug: #2012251
Change-Id: I828911495a2de550ea997e6f51cc039a7b7fa8cd
2023-04-21 06:32:09 +00:00
Zuul
42cbce38c1 Merge "LVM: Fix supports_full_pool_create" 2023-04-17 20:48:04 +00:00
Eric Harney
5926a4d4e2 LVM: Remove outdated comment
-l100%FREE was implemented years ago and is
already used by our code when the LVM version
meets the supports_full_pool_create check.

Change-Id: I8f26e3880429e648e7e985f9c6cb7dff56715727
2023-04-12 11:31:45 -04:00
Zuul
28ffcdbfa1 Merge "LVM: Fix bare raise on LVM command error" 2023-03-23 16:34:16 +00:00
Eric Harney
9496bb2e47 LVM: Fix supports_full_pool_create
mypy detected this problem -- the
supports_full_pool_create property will always
return True when referenced via the class instead
of via an instance.

Change-Id: Iefc691ae8d5eb001c9abbf554600a8f8132e038c
2023-03-21 09:49:51 -04:00
Zuul
4f335d410a Merge "Bump bandit to release 1.7.0" 2023-02-09 22:36:06 +00:00
Eric Harney
05052a1375 LVM: Fix bare raise on LVM command error
This should have an associated message, and is
detected as an error by pylint.

Change-Id: Icc312cb960971ecf2a7754b9056a869c9ec8098a
2023-02-09 11:10:19 -05:00
Zuul
2aa838ae75 Merge "Fix wrong assertion methods" 2023-02-08 20:03:23 +00:00
Eric Harney
acd0265f8d linuxrbd: Remove rados_connect_timeout parameter
This parameter doesn't do anything, we should skip it here.

https://docs.ceph.com/en/latest/rados/api/python/#rados.Rados.connect
https://github.com/ceph/ceph/blob/974339d1f/src/pybind/rados/rados.pyx#L674

Change-Id: I819e32f807bcba4c9d6f928f2ad04c4dd27f22c2
2023-02-06 15:49:31 -05:00
Takashi Natsume
e7b1426e31 Fix wrong assertion methods
Replace 'called_once_with' with 'assert_called_once_with'.

Change-Id: I18b55da0d1f142818e7ea14f6eebcc0f0f0cd23f
Closes-Bug: 1989280
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
2023-02-05 22:33:35 +00:00
Gorka Eguileor
a82b2ffce5 Support separate privsep logging levels
Currently we don't have os_brick DEBUG log levels in Nova when setting
the service to debug mode.

That happens because Nova is forcefully setting oslo.privsep.daemon
levels to INFO to prevent leaking instance XML details (bug #1784062).

Oslo Privsep now supports per-context debug log levels, so this patch
sets the log level name for its only os_brick privsep context to
"os_brick.privileged" to differentiate it from the service it runs under
which uses the default "oslo_privsep.daemon".

This way even though Nova is still disabling its own privileged calls it
won't affect os-brick privileged calls, allowing us to properly debug
block device attach/detach operations.

Closes-Bug: #1922052
Related-Bug: #1784062
Change-Id: I0de32021eb90ca045845a6c7c7e3d27e52895948
2023-01-26 17:02:22 +01:00
Zuul
655fcc41b3 Merge "Handle FileNotFoundError on get_system_uuid()" 2023-01-25 17:40:42 +00:00
Eric Harney
e43cdf6e1e Bump bandit to release 1.7.0
Use bandit 1.7.0.

Disables B101 globally and allows one instance of B604.

Change-Id: I82a39400520784adb8d26a355cd0e077ad58a530
2023-01-25 10:17:21 -05:00
Zuul
5c36bce2ab Merge "mypy: Annotate utils.get_device_size" 2023-01-20 17:37:30 +00:00
Eric Harney
dc953012b7 mypy: Annotate utils.get_device_size
Add annotations for parameters of this method.

Change-Id: I3149e0067f97d373333e0901e49725401fc6c068
2022-11-15 13:03:21 -05:00
Felipe Reyes
bc96120b7e Handle FileNotFoundError on get_system_uuid()
On some platforms there is no 'dmidecode' program (e.g. s390x), when
privsep attempts to run a program that's not found, it raises a
FileNotFoundError instead of ProcessExecutionError.

Closes-Bug: #1994083
Change-Id: I6fc43ab7e243d50b74036d1af5f9e8f880401cc6
2022-10-24 20:37:49 -03:00
Rajat Dhasmana
22830d7077 nit: correct spelling of Rescanning in debug log
Change-Id: Iba31b709ef0009891d0098600387e3fe6b1af26d
2022-09-24 18:48:14 +00:00
Eric Harney
47dda4b487 mypy: lvm.py
Change-Id: I27a565b8ca250f6790061932ab000946dedd97c5
2022-08-25 14:00:42 +00:00
Eric Harney
f5d5371cbc mypy: initiator/linuxrbd
Add type coverage for initiator/linuxrbd.

Change-Id: Id9a3bf8b1e8aea14cd2cf1eae6907c9719aad3b0
2022-08-25 09:59:32 -04:00
Zuul
0e3ab3bd74 Merge "RBD: Improve IOWrapper's close & flush methods" 2022-08-25 13:57:27 +00:00
Gorka Eguileor
fafd1a6f76 RBD: Improve IOWrapper's close & flush methods
The RBDVolumeIOWrapper class inherits from io.RawIOBase but its close
method doesn't flush and after being called the "closed" attribute still
returns False.

Trying to set the "closed" attribute ourselves in
RBDVolumeIOWrapper.close results in an error, so we call the parent's
implementation to do it.

The parent's close method will also call the flush method.

This change ensures that rbd_image.close is only called once, that the
closed attribute is correct, and that the data is flushed on close.

Without these changes we could have, unintentionally, multiple calls to
the rbd_image.close method.  If we called close on the wrapper
ourselves, then when the Garbage Collector is cleaning up the wrapper it
will call the io.RawIOBase finalizer that will see that the file is not
closed and call close again.

Now that we correctly set the closed flag on the instance, the flush
method should also behave like the parent implementation and fail if it
has already been closed.

Change-Id: Ib3b066a7da071b1c2de78a1a4e569676539bd335
2022-08-24 21:43:28 +02:00
Zuul
5b62f84c63 Merge "NVMe-oF: Get system uuid in privsep" 2022-08-24 14:18:27 +00:00
Gorka Eguileor
da4561165d NVMe-oF: Get system uuid in privsep
Move the _get_system_uuid method from the NVMeOFConnector connector to
os_brick/privsep/nvmeof.py renaming it to get_system_uuid and running it
as privileged.

This allows us to read sys/class/dmi/id/product_uuid file in Python
instead of running a subprocess to execute cat.

It also allows the connector to have one less privsep calls if the file
doesn't exist, because it can execute the dmidecode command directly
without making another request to the privsep daemon.

Change-Id: I8f2edef6fda97af0ff3f92e39c8b24a85b6c3402
2022-08-24 10:50:37 +02:00
Zuul
4e56e046cb Merge "NVMe-oF: read mdstat in Python" 2022-08-23 23:21:14 +00:00
Zuul
4acfd6bc7e Merge "mypy: Update format to future __annotations__" 2022-08-23 19:15:18 +00:00
Eric Harney
6de226ed18 mypy: Update format to future __annotations__
Use dict[], list[] etc. instead of Dict[], List[].

This works on all versions of Python that we support.

Change-Id: If9c737277c041ffd2d3bca1863d300ae7d9bd52e
2022-08-23 15:21:17 +00:00
Eric Harney
794f006f93 mypy: os_brick/encryptors/__init__.py
This coverage is somewhat limited because this
uses classes (context, keymgr) that live outside
of os-brick and therefore don't have type definitons
here.

Change-Id: I7827a870c04a7a6e02510b7909430feebb70e6f1
2022-08-23 15:12:04 +00:00
Eric Harney
aa97f4455c mypy: set no_implicit_optional
The current version of PEP-484 indicates that
type checkers should not allow implicit optional
parameters.

- https://www.python.org/dev/peps/pep-0484/#union-types
- https://github.com/python/mypy/issues/9091

Change-Id: Ie3301798462a89af9fc0053070055a77a2437415
2022-08-23 15:11:56 +00:00
Gorka Eguileor
da2bfa41f5 NVMe-oF: read mdstat in Python
The get_md_name uses the contents of mdstat using cat and piping it to
grep and then to awk using a privileged shell, but there's no need to do
that.

Actually the mdstat file has 0444 access, so we can read it in Python
directly instead of having to call the privsep daemon to create the
subprocess with the shell.

This patch changes the current implementation of the method to do the
reading and searching in Python.

Change-Id: Idbba6f9a6d928aa94d2920fc1bf8b6e2a0626a07
2022-08-23 12:18:17 +02:00