OpenStack Block Storage (Cinder)
Go to file
Gorka Eguileor 0b2c0d9238 LVM: Fix delete volume error due to lvs failure
Sometimes we get an unexpected failure on the lvs command where it exits
with code 139, which is not one of the 5 predefined ones in the code
ECMD_PROCESSED, ENO_SUCH_CMD, EINVALID_CMD_LINE, EINIT_FAILED, or
ECMD_FAILED.

We've seen this happen mostly on CI jobs when deleting a volume, on the
check to see if the volume is present (``_volume_not_present``) and
makes the whole operation unexpectedly fail.

When looking at the logs we can find other instances of this same exit
code happening in other calls, but those are fortunately to be covered
by retries for other unexpected errors such as bug #1335905 that seem to
make the call eventually succeed.

The stderr of the failure is polluted with debug and warning messages
such as:

  [1] /dev/sda1: stat failed: No such file or directory

      This has been removed [2] from the LVM code indicating it's
      somewhat troublesome, but doesn't explain how.

  [3] Path /dev/sda1 no longer valid for device(8,1)

  [4] WARNING: Scan ignoring device 8:0 with no paths.

But the real error is most likely:

  [5]: Device open /dev/sda 8:0 failed errno 2

On failure we see that error twice, because the code retries it in LVM
trying to workaround some kind of unknown udev race [6].

Since the LVM code indicates that a retry can help, we'll retry on error
139 when calling ``get_lv_info``.

To narrow down the retry we'll only do it on error 139, so we modify the
existing ``retry`` decorator to accept the ``retry`` parameter (same as
the tenacity library) and create our own retry if the
ProcessExecutionError fails with a specific error.

This pattern seems better than blindly retrying all
ProcessExecutionError cases.

[1]: 17f5572bc9/lib/filters/filter-persistent.c (L132)
[2]: 22c5467add
[3]: b84a9927b7/lib/device/dev-cache.c (L1396-L1402)
[4]: b84a9927b7/lib/label/label.c (L798)
[5]: b84a9927b7/lib/label/label.c (L550)
[6]: b84a9927b7/lib/label/label.c (L562-L567)

Closes-Bug: #1901783
Change-Id: I6824ba4fbcb6fd8f57f8ff86ad7132446ac6c504
2021-03-29 16:43:10 +02:00
api-ref/source Merge "api-ref: Don't list HTTP 500 for backup create" 2021-03-19 22:09:11 +00:00
cinder LVM: Fix delete volume error due to lvs failure 2021-03-29 16:43:10 +02:00
contrib/block-box Add note that block-box is not supported 2020-01-10 08:37:54 -06:00
doc Use os-brick 4.3.0 2021-03-25 13:15:43 -04:00
etc/cinder Delete TSM Backup driver 2020-11-03 18:53:39 +02:00
playbooks zuul: collect cinderlib logs from tempest node(s) only 2020-07-16 14:42:29 +02:00
rally-jobs Small doc updates: canonical URLs, rally doc, etc 2019-06-04 17:57:16 +02:00
releasenotes LVM: Fix delete volume error due to lvs failure 2021-03-29 16:43:10 +02:00
tools Merge "tox mypy: Call mypywrap.sh directly" 2021-02-19 21:26:28 +00:00
.coveragerc Update .coveragerc after the removal of openstack directory 2016-10-17 19:09:37 +05:30
.gitignore Add mypy tox env 2020-10-14 08:24:13 -04:00
.gitreview OpenDev Migration Patch 2019-04-19 19:26:48 +00:00
.pylintrc Pylint: Filter out cinder.objects and requests E1101 2019-02-21 13:28:44 -05:00
.stestr.conf Add .stestr.conf configuration 2017-10-10 00:46:42 +00:00
.zuul.yaml Merge "add openstack-python3-wallaby-jobs-arm64 job" 2021-03-20 07:49:56 +00:00
CONTRIBUTING.rst Ussuri contrib docs community goal 2020-03-05 14:11:48 -05:00
HACKING.rst Update HACKING document to match current checks 2020-04-17 15:09:13 +00:00
LICENSE Initial fork out of Nova. 2012-05-03 10:48:26 -07:00
README.rst Merge "Refactor README links" 2020-01-22 11:44:36 +00:00
bindep.txt bindep: Track qemu-img dependency 2020-08-19 15:00:35 +00:00
driver-requirements.txt Add ceph iscsi volume driver 2021-01-28 08:22:13 -05:00
lower-constraints.txt Use os-brick 4.3.0 2021-03-25 13:15:43 -04:00
mypy-files.txt mypy: annotate volume manager 2021-02-10 12:27:47 -05:00
reno.yaml Reno: Use customized template 2020-10-06 13:28:45 +02:00
requirements.txt Use os-brick 4.3.0 2021-03-25 13:15:43 -04:00
setup.cfg Add ceph iscsi volume driver 2021-01-28 08:22:13 -05:00
setup.py Cleanup py27 support 2020-04-17 14:47:10 +02:00
test-requirements.txt Use os-brick 4.3.0 2021-03-25 13:15:43 -04:00
tox.ini Changed whitelist to allowlist in tox 2021-03-17 11:08:56 +05:30

README.rst

OpenStack Cinder

image

OpenStack Cinder is a storage service for an open cloud computing service.

You can learn more about Cinder at:

Getting Started

If you'd like to run from the master branch, you can clone the git repo:

git clone https://opendev.org/openstack/cinder

If you'd like to contribute, please see the information in CONTRIBUTING.rst

You can raise bugs on Launchpad

Python client

Python Cinderclient