from an image
This patch fixes the issue of password getting writen in plain text in
logs while creating a new volume. It created a new logger with default
log level at error.
Closes-Bug: #2003179
Change-Id: I0292a30f402e5acddd8bbc31dfaef12ce24bf0b9
Dell Powerflex 4.x changed the error code of VOLUME_ALREADY_MAPPED_ERROR
to 4037. This patch adds that error code.
Closes-Bug: #2013749
Change-Id: I928c97ea977f6d0a0b654f15c80c00523c141406
Add file to the reno documentation build to show release notes for
stable/2023.2.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.2.
Sem-Ver: feature
Change-Id: I25bef272ded6c7c963c6ad0f95103fe421fa8fe7
In a multipath enabled deployment, when we try to extend a volume
and some paths are down, we fail to extend the multipath device and
leave the environment in an inconsistent state. See LP Bug #2032177
for more details.
To handle this, we check if all the paths are up before trying to
extend the device and fail fast if any path is down. This ensures
we don't partially extend some paths and leave the other to the
original size leading to inconsistent state in the environment.
Closes-Bug: 2032177
Change-Id: I5fc02efc5e9657821a1335f1c1ac5fe036e9329a
The NVMe-oF connector currently create the `/etc/nvme/hostnqn` file if
it doesn't exist, but it may still be missing the `/etc/nvme/hostid`
value.
Some distribution packages create the file on installation but others
may not.
It is recommended for the file to be present so that nvme doesn't
randomly generate it.
Randomly generating it means that the value will be different for the
same storage array and the same volume if we connect, disconnect, and
connect the volume again.
This patch ensures that the file will exist and will try to use the
system's UUID as reported by DMI or a randomly generated one.
Closes-Bug: #2016029
Change-Id: I0b60f9078f23f8464d8234841645ed520e8ba655
This patch adds non SAM addressing modes for LUNs, specifically for
SAM-2 and SAM-3 flat addressing.
Code has been manually verified on Pure storage systems that uses SAM-2
addressing mode, because it's unusual for CI jobs to have more than
256 LUNs on a single target.
Closes-Bug: #2006960
Change-Id: If32d054e8f944f162bdc8700d842134a80049877
When multipath is enabled and friendly names are ON and we try
to extend a volume, we pass the SCSI WWID to the multipath_resize_map
method which is not correct.
There are 2 things we can pass to the multipathd resize map command:
1) Multipath device (eg: dm-0)
2) Alias (eg: mpath1) or UUID (eg: 36005076303ffc56200000000000010aa)
The value should be an alias (mpath1) when friendly names are ON
and UUID (36005076303ffc56200000000000010aa) when friendly names
are OFF. However, we only pass the UUID irrespective of the value
set for friendly names.
This patch passes the multipath device path (to multipathd resize
map command) which is the real path of the multipath device (/dev/dm-*).
This fixes the issue as it passes the same value irrespective of if
the friendly names are ON or OFF.
-> self.multipath_resize_map(os.path.realpath(mpath_device))
(Pdb) mpath_device
'/dev/disk/by-id/dm-uuid-mpath-3600140522774ce73be84f9cb9537e0c9'
(Pdb) os.path.realpath(mpath_device)
'/dev/dm-5'
Closes-Bug: 1609753
Change-Id: I1c60af19c2ebaa9de878cd07cfc0077c5ea56fe3
The -v arg suppresses printing of [/dir] with the
device for bind mounts and btrfs volumes, which is
what we want for this usage.
This fixes _get_host_uuid() failing when using
a btrfs rootfs.
Closes-Bug: #2026257
Change-Id: I2d8f24193ecf821843bf8f4ea14b445561d6225c
This patch adds support for the force and ignore_errors on the
disconnect_volume of the FC connector like we have in the iSCSI
connector.
Related-Bug: #2004555
Change-Id: Ia74ecfba03ba23de9d30eb33706245a7f85e1d66
Add file to the reno documentation build to show release notes for
stable/2023.1.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.1.
Sem-Ver: feature
Change-Id: Ic3ac791f083aa097412dc9075a6b20f2b148db02
Currently we don't have os_brick DEBUG log levels in Nova when setting
the service to debug mode.
That happens because Nova is forcefully setting oslo.privsep.daemon
levels to INFO to prevent leaking instance XML details (bug #1784062).
Oslo Privsep now supports per-context debug log levels, so this patch
sets the log level name for its only os_brick privsep context to
"os_brick.privileged" to differentiate it from the service it runs under
which uses the default "oslo_privsep.daemon".
This way even though Nova is still disabling its own privileged calls it
won't affect os-brick privileged calls, allowing us to properly debug
block device attach/detach operations.
Closes-Bug: #1922052
Related-Bug: #1784062
Change-Id: I0de32021eb90ca045845a6c7c7e3d27e52895948
Add file to the reno documentation build to show release notes for
stable/zed.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/zed.
Sem-Ver: feature
Change-Id: Ia775c42636307fa35d0612937e4c949c0cc2193b
Cinder microversion 3.69 adds an additional value to shared_targets
beyond true and false. Now null/None is also a valid value that can be
used to force locking.
So we now have 3 possible values:
- True ==> Lock if iSCSI initiator doesn't support manual scans
- False ==> Never lock.
- None ==> Always lock.
This patch updates the guard_connection context manager to support the
3 possible values.
With this Cinder can now force locking for NVMe-oF drivers that share
subsystems.
Closes-Bug: #1961102
Depends-On: I8cda6d9830f39e27ac700b1d8796fe0489fd7c0a
Change-Id: Id872543ce08c934cefbbbdaff6ddc61e3828b1f1
We currently have 2 different connection information formats: The
initial one and the new one.
Within the new format we have 2 variants: Replicated and Not replicated.
Currently the nvmeof connector has multiple code paths and we end up
finding bugs that affect one path and not the other, and bugs that are
present in both may end up getting fixed in only one of them.
This patch consolidates both formats by converting the connection
information into a common internal representation, thus reducing the
number of code paths.
Thanks to this consolidation the Cinder drivers are less restricted on
how they can identify a volume in the connection information. They are
no longer forced to pass the NVMe UUID (in case they cannot get that
information or the backend doesn't support it) and they can provide
other information instead (nguid or namespace id).
This connection properties format consolidation is explained in the new
NVMeOFConnProps class docstring.
By consolidating the code paths a number of bugs get fixed
automatically, because they were broken in one path but not in the
other. As examples, the old code path didn't have rescans but the new
one did, and the old code path had device flush but the new one didn't.
Given the big refactoring needed to consolidate everything this patch
also accomplishes the following things:
- Uses Portal, Target, and NVMeOFConnProps classes instead of using the
harder to read and error prone dictionaries and tuples.
- Adds method annotations.
- Documents most methods (exect the raid ones which I'm not familiar
with)
- Adds the connector to the docs.
- Makes method signatures conform with Cinder team standards: no more
static methods passing self and no more calling class methods using
the class name instead of self.
Closes-Bug: #1964590
Closes-Bug: #1964395
Closes-Bug: #1964388
Closes-Bug: #1964385
Closes-Bug: #1964380
Closes-Bug: #1964383
Closes-Bug: #1965954
Closes-Bug: #1903032
Related-Bug: #1961102
Change-Id: I6c2a952f7e286f3e3890e3f2fcb2fdd1063f0c17
Patch fixing bug #1861071 resolved the issue of extending LUKS v1
volumes when nova connects them via libvirt instead of through os-brick,
but nova side still fails to extend LUKSv2 in-use volumes when they
don't go through libvirt.
The logs will show a very similar error, but the user won't know that
this has happened and Cinder will show the new size:
libvirt.libvirtError: internal error: unable to execute QEMU command
'block_resize': Cannot grow device files
There are 2 parts to this problem:
- The device mapper device is not automatically extended.
- Nova tries to use the encrypted block device size as the size of the
decrypted device.
This patch adds new functionality to the encryptors so that they can
extend decrypted volumes to match the size of the encrypted device.
New method added to encryptors is called "extend_volume", and should be
called after the homonymous method in the connector has been called, and
the value returned by the encryptor's extend_volume method is the real
size of the decrypted volume (encrypted volume - headers).
The patch only adds functionality for LUKS and LUKSv2 volumes, not to
cryptsetup volumes.
Related-Bug: #1967157
Change-Id: I351f1a7769c9f915e4cd280f05a8b8b87f40df84
OS-Brick uses file locks to ensure single access to critital sections,
and uses the oslo concurrency "lock_path" configuration option to
determine where to create these locks.
This is fine when each host has a single service using os-brick, which
is the case of Compute nodes and Controller nodes where Glance is not
using Cinder as its backend.
The problem happens when we have an HCI deployment where Cinder and Nova
are collocated on the same host or when Glance uses Cinder as its
backend and is running on the same host. In those scenarios os-brick
will create file locks in different paths for each service, which is not
the intended behavior.
A possible solutions is to set the "lock_path" of all the services to
the same location, which is not great, not only because we'll have all
nova, cinder, glance, and os-brick locks mixed in a single location
(service prefixes help a bit here), but also because then Cinder will
have permissions on the Nova and Glance locks, which doesn't seem right.
This patch introduces a new mechanism in os-brick to have its own
"lock_path" configuration option in the "os_brick" group. It defaults
to the current behavior if not explicitly defined, so it will use olso
concurrency's "lock_path".
To do this the patch introduces the oslo.config dependency and adds a
new "setup" method that sets the default of the os_brick "lock_path".
This new "setup" method is required because the order in which
configuration options for the different namespaces are imported cannot
be guaranteed, so the setup must be called *after* the service has
already imported all (or at least the ones os-brick cares about)
configuration option namespaces.
In other words, when os-brick files are loaded the value for oslo
concurrency's "lock_path" is not yet known, so it cannot be used as a
default, and the oslo_config substitution feature does not support
values from other namespaces, so we cannot default the "lock_path" from
the os_brick namespace to the value in "oslo_concurrency".
Since the value that is going to be used as the "lock_path" is not known
until the "setup" method is called, we cannot use the standard
"synchronized" method from "oslo_concurrency.lock_utils" and we need to
use our own.
In the current os-brick code, each connector that requires a lock is
importing and creating its own "synchronized" instance, but this patch
changes this behavior and creates a single instance, just like Cinder
does.
This feature requires changes in multiple projects -os-brick, cinder,
nova, glance, glance_store, devstack- to be fully supported, but this is
the base of all this effort.
Change-Id: Ic52338278eb5bb3d90ce582fe6b23f37eb5568c4
After change Ie373ab050dcc0a35c749d9a53b6cf5ca060bcb58 closing bugs
connector was broken for encrypted volumes:
<== disconnect_volume:
exception (0ms) AttributeError("'RBDVolumeIOWrapper' object has
no attribute 'startswith'") trace_logging_wrapper
/usr/lib/python3.9/site-packages/os_brick/utils.py:162
This hapens because "_device_path_from_symlink" doesn't take into
account of the file handle, and we were missing a unit test for this
case.
Closes-Bug: #1981455
Change-Id: Ib001e2b4d1347754c2b46730bc10d86e8cdab7ad
We were gathering HBA information using the systool command line tool,
but this tool is no longer going to be packaged in some Operating
Systems, for example in RHEL9 where the sysfsutils package is being
removed.
To prevent os-brick from breaking in those systems this patch removes
the usage of the command and reads the necessary information from sysfs.
Conveniently reading from sysfs should also be faster because we no
longer need to make a privsep call to run a command, instead we are
reading the minimum number of files and information possible.
Change-Id: Idb8b0c22a30e52c7f84a54dd9f410ff657f502a4
This patch fixes 2 issues related to the symlinks, or lack of, that
connectors' connect_volume methods return.
Some connectors always return the block device instead of a symlink for
encrypted volumes, and other connectors return a symlink that is owned
by the system's udev rules. Both cases are problematic
Returning the real device can prevent the encryptor connect_volume to
complete successfully, and in other cases (such as nvmeof) it completes,
but on the connector's disconnect volume it will leave the device behind
(i.e., /dev/nvme0n1) preventing new connections that would use that same
device name.
Returning a symlink owned by the system's udev rules means that they can
be reclaimed by those rules at any time. This can happen with
cryptsetup, because when it creates a new mapping it triggers udev rules
for the device that can reclaim the symlink after os-brick has replaced
it.
This patch creates a couple of decorators to facilitate this for all
connectors. These decorators transform the paths so that the callers
gets the expected symlink, but the connector doesn't need to worry about
it and will always see the value it returns regardless of what symlink
the caller gets.
From this moment onwards we use our own custom symlink that starts with
"/dev/disk/by-id/os-brick".
The patch fixes bugs in other connectors (such as the RBD local
connection), but since there are no open bugs they have not been
reported.
Closes-Bug: #1964379
Closes-Bug: #1967790
Change-Id: Ie373ab050dcc0a35c749d9a53b6cf5ca060bcb58
When disconnecting a non-replicated volume that uses the newer
connection properties format we'll get some exception trace logs that
could make us think that something went wrong, when in reality this is
ok.
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[95eb2bcc-e4eb-4f5b-b823-56fd7aa338c6]: Unexpected error while running command.
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: Command: blkid /dev/nvme0n2 -s TYPE -o value
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: Exit code: 2
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: Stdout: ''
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: Stderr: '' {{(pid=988326) _process_cmd /usr/local/lib/python3.6/site-packages/oslo_privsep/daemon.py:481}}
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: Traceback (most recent call last):
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: File "/usr/local/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: ret = func(*f_args, **f_kwargs)
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: File "/usr/local/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 274, in _wrap
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: return func(*args, **kwargs)
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: File "/opt/remote_brick/os_brick/privileged/rootwrap.py", line 197, in execute_root
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: File "/opt/remote_brick/os_brick/privileged/rootwrap.py", line 146, in custom_execute
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: on_completion=on_completion, *cmd, **kwargs)
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: File "/usr/local/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 441, in execute
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: cmd=sanitized_cmd)
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: Command: blkid /dev/nvme0n2 -s TYPE -o value
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: Exit code: 2
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: Stdout: ''
Mar 09 11:14:42 localhost.localdomain nova-compute[988198]: Stderr: ''
This patch calls the command execution in a way that no longer shows
such exception.
Closes-Bug: #1964389
Change-Id: I1c54e1ff3026cffe49b303f01e43d1b90d0bf3bc
Add file to the reno documentation build to show release notes for
stable/yoga.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/yoga.
Sem-Ver: feature
Change-Id: I8f2e1f690e3f47f7f34c7dfe53886be129e3d891
Environment:
- openstack version: Xena
- kolla on ubuntu focal 20.04
the installed version of nvme is 1.9, this version doesn't have the
command "show-hostnqn".
In case of missing hostnqn file, it will fail to create hostnqn
the exit error code is 231 and is not caught by the
try-except block.
Here we try to parse the stdout to check if the
subcommand (show-hostnqn) is not supported.
if its not, we continue to try and create the hostnqn.
Change-Id: I318f167baa0ba7789f4ca2c7c12a8de5568195e0
nvmeof connector methods handling volumes with volume_replicas should
utilize replica_count parameter in connection_information to
determine method of device exposure to host
Closes-Bug: #1938870
Change-Id: I629b80b208295c0e6bc9f18ad8e6905759e6593e
Use the '/sys/class/block/<ctrl>n*' form for finding nvme devices.
This way is more generic, and is mainly beneficial because the device name format does not change as it does in '/sys/class/nvme-fabrics/ctl/*' - where it is 'nvme*c*n*' vs 'nvme*n*' for multipath-enabled vs disabled kernels.
Closes-Bug: #1943615
Change-Id: Ib8e5d79e7dbae0867c794f35ff3350419d2cdf09
Currently os-brick is using in-process locks that will only prevent concurrent
access to critical sections to threads within a single process.
But based on the comment from iSCSI it seems like the code assumed that
these were file based locks that prevented concurrent access from
multiple processes.
Mentioned iSCSI comment is being removed because it's not correct that
our current retry mechanism will work with connect and disconnect
concurrency issues.
The reason why we haven't seen errors in Nova is because it runs a
single process and locks will be effective.
This is probably also not an issue in some transport protocols, such as
FC and RBD, and it wouldn't be an issue in iSCSI connections that don't
share targets.
But for others, such as iSCSI with shared targets and NVMe-OF, not
using file locks will create race conditions in the following cases:
- More than 1 cinder backend: Because we can have one backend doing a
detach in a create volume from image and the other an attach for an
offline migration.
- Backup/Restore if backup and volume services are running on the same
host.
- HCI scenarios where cinder volume and nova compute are running on the
same host, even if the same lock path if configured.
- Glance using Cinder as backend and is running on the same node as
cinder-volume or cinder-backup.
The problematic race conditions happen because the disconnect will do a
logout of the iSCSI target once the connect call has already confirmed
that the session to the target exists.
We could just add the file locks to iSCSI and NVMe, but I think it's
safer to add it to all the connectors and then, after proper testing, we
can can change back the locks that can be changed, and remove or reduce
the critical section in others.
Closes-Bug: #1947370
Change-Id: I6f7f7d19540361204d4ae3ead2bd6dcddb8fcd68
After compute host reboot, in an iSCSI/multipath environment, some
of the connections to the iSCSI portal are not reinitiated and missing
iSCSI devices are observed. This patchset introduced retries for this
particular scenario.
Closes-Bug: #1944474
Change-Id: I60ee7421f7b792e8324286908a9fdd8fb53e433e
Add file to the reno documentation build to show release notes for
stable/xena.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/xena.
Sem-Ver: feature
Change-Id: Ia92a0a3bb448f1de77038d83ca81056589a05cbc
Zuul
OpenStack Release Bot