If a 'keyring' key is found in the connection info passed to
connect_volume() use its value as the path to the keyring instead of the
default location (/etc/ceph/<cluster>.client.<user>.keyring).
This allows services such as cinder's RBD and Ceph backup drivers to
make use of a custom keyring path that an admin has defined.
Change-Id: Ib1230d3e40f56371567e1aead40db59667bad295
Closes-bug: #1668304
iSCSI multipath rescan uses iscsiadm --rescan option for nodes and
sessions, which can end up recreating devices that had just been removed
if there's a race condition between the removal of a SCSI device and the
connection of a volume.
The race condition happens if a rescan done when attaching happens right
between us removing the path and removing the exported lun, because the
rescan will add not only the new path we are attaching, but the old path
we are removing, since the lun still hasn't been removed.
This would leave orphaned devices that unnecessarily pollute our
environment,
This patch narrows the rescan to only rescan for the specific target id,
channel, and lun number if we can find this information.
When we cannot find this information we do the scan as we were doing it
before.
Closes-Bug: #1664032
Change-Id: I1b3bd34db260165a6ea9ca061f946d6dfcf8553f
When flushing a multipath we are ignoring map in use transient error, so
we log a warning that the flush has not been successful and that we have
received an exit code 1 and we continue to remove the individual paths.
This error is usually transient and a simple retry will succeed in
flushing the multipath.
Failure to retry will leave an empty multipath in our system.
Closes-Bug: #1663936
Change-Id: I710792bd707ad933ef60d11d25f530dddfb6fb2f
When we are using friendly names for multipath the multipaths are not
getting flushed, which may lead to data loss on slow connections and
multipath entries with no actual paths.
This happens in both iSCSI and FC connections, and it is due to the
flush being requested on the WWN instead of the actual name of the
device.
So when we are not using friendly names the WWN and the device name are
the same and our call to multipath -f will successfully flush remaining
data, but when we are using friendly names they will not match, and the
call to multipath -f will silently fail (return code 0) and the flush
will not actually go through. When the flush doesn't happen, if there is
remaining data, then the multipath will stay once the individual paths
have been removed.
Closes-Bug: #1663925
Change-Id: Ib93d945a5b5fca57bcac4e176d62d1412b95f2da
The change 916cfaf "Introduce encryption provider constants"
intended to issue deprecation messages for encryptors that
are no longer going to be supported in Pike, and continue
loading the current os_brick encryptors instead.
However, since it removed the "'LuksEncryptor' in provider"
style checks, any config values that were not explicitly listed
in LEGACY_PROVIDER_CLASS_TO_FORMAT_MAP will not be translated.
This results in a configuration using a provider of
"nova.volume.encryptors.luks.LuksEncryptor" to attempt to
import this module directly into Cinder when previously it
would use the os_brick encryptor. This is generally wrong
and causes a number of problems.
Closes-Bug: #1658755
Change-Id: I3ec6e3fe919bc03d158da04a18fb8b651002ed52
This fix solves a LUN attachment error for Ubuntu on s390 host setups. Fiber
Channel requires a path for discovery of new devices, and Ubuntu uses a
different format than previously assumed. This can be solved by simply adding
the Ubuntu-style path to the list of possible paths, since they get quietly
dropped if they lead nowhere in FibreChannelConnector._get_possible_volume_paths
anyways.
Change-Id: I0b07572903263122213f2ea5dc42151a7b69d99f
Closes-Bug: #1655047
These constants detail the supported encryption formats and their
associated in tree encryption provider implementations.
The use of out of tree and direct use of these in tree implementations
is now deprecated and will be blocked in the Pike release of os-brick.
Change-Id: Ic155bd29d46059832cce970bf60375e7e472eca6
Partial-bug: #1639293
The Hyper-V Nova driver will pass os-brick the device rescan
interval. This change indends to honor this for the Windows
connectors.
Other connectors use either a fixed rescan interval, either a
exponentially growing interval. If needed, this patch can be
updated so that other connectors may use this argument as well.
Change-Id: I3d6b4ed2ac27269241420ca03ec7075a977e86e8
The following file(s) added utf-8 encoding but never used. So we can
remove them at all.
os_brick/tests/base.py
os_brick/tests/remotefs/test_remotefs.py
os_brick/tests/test_brick.py
Change-Id: Icf7275285e2024fbfd577e95a5d88a50734858bb
This ensures that if a failure occurs with any of the linuxrbd calls,
the temp config file is still removed.
This also gives it a prefix so it is identifiable.
Change-Id: Ief3d907092e77c48a531a6ae78b10c58cc6db56c
Prior to Ib563b0ea the passphrase used by CryptsetupEncryptor and
LuksEncryptor had any leading zeros per hexadecimal digit removed, for
example 0x04 or 04 would turn into 0x4 or 4. As a result any volume
encrypted prior to the release of Newton used a modified passphrase that
was different to that stored by the key manager being used in the
environment.
To correct this for LuksEncryptor volumes permission denied errors are
now caught when attempting to open a volume. A second attempt to open
the volume is then made using a mangled passphrase. If successful the
correct passphrase is then added to the volume before the mangled
passphrase is finally removed. This workaround can be removed in a
future release once it is safe to assume that all LuksEncryptor volumes
have had any mangled passphrases replaced in this way.
This isn't possible for CryptsetupEncryptor volumes as the plain mode
used by cryptsetup does not provide a way for adding and removing keys.
As such on a permission denied error a second attempt is made to open
the volume using a mangled passphrase. Unlike the above workaround this
cannot be removed in a future release.
Change-Id: I7096463c5eba951dd6322ee6965435e877ca0371
Partial-bug: #1633518
Previously the encryptor test classes would mock the privileged rootwrap
executor directly in the setUp method of each class. This will
potentially race and is not required when mocking out execute calls
within os_brick.
This change also introduces a BaseEncryptorTestCase test class to avoid
duplicate runs of the various get_encryptors test methods.
Change-Id: I170fbcc07672c9c77b613c1eb84bcefbcd42ce77
When using ipv6, the IP address must be enclosed in square brackets
according to RFC 3986, Section 3.2.2. Without this, an ipv6 cloud will
fail to connect to the ceph cluster prior to backup because the address
and port are not parsed correctly.
Closes-Bug: #1640914
Change-Id: I6501c7cdac38fe545d12e786448a8e0c0a4f6a25
The utils.trace helper is logging the args list to
the decorated function but is not masking passwords
in those args. This change adds a call to mask passwords
in the function args list.
Change-Id: I79480c6f9c3e3a9a917854139461650780e6e8b4
Closes-Bug: #1640251
Nova's libvirt driver attempts to load up a set of volume
drivers which use os-brick connectors on start of the
nova-compute service. If there is a missing or invalid
mapping, like ISER + s390x, it results in a ValueError and
nova-compute fails to start.
This change raises a specific exception, which is still a
ValueError for backward compatibility, so that Nova can
handle this situation gracefully and not crash.
Change-Id: I1d016c226416cf2b77d6e0393498905f56fd8cde
Partial-Bug: #1639239
When resizing a in-use volume, if multipath is enabled, function
extend_volume only rescans ONE SCSI device and then run the command:
"multipathd reconfigure"
"multipathd resize map mpath_id"
The SCSI device resizes successfully, but the multipath device is
still the old size. This patch fixes it by rescanning all SCSI devices
of the multipath device.
Change-Id: I3a7c7d5e86defedfacd71067f2e5a89bca6aa35b
Closes-Bug: #1611659
In order to support running inside a Scale-Out File Server cluster,
when 'mounting' a local share, we're creating a symlink to the local
export path without actually creating a SMB mapping.
The issue is that instead of creating a symlink using the hash of
the share UNC path, we're using the local export path instead. This
is undesired behavior, in which case the caller will not be able
to find the 'mountpoint'.
This change addresses this issue, ensuring that we're using the
mountpoint expected by the caller (e.g. the Cinder SMB driver)
when mounting SMB shares.
Closes-Bug: #1631351
Change-Id: I6bf909da559a745596f7d4c30d12e794b0b156de
This patch adds a simple check to see if the kernel
even supports Fibre Channel before calling systool -c fc_host -v
to find the active HBA's on the system.
The problem is that privsep can't quietly run systool without puking out
a bunch of errors to stderr. We know that most systems don't have FC and
that's why we always wrapped out call to putils executing systool
with a try block. putils allows callers to expect exceptions and then
handle the exception logging or not themselves. privsep doesn't allow that.
Change-Id: Idd254303c9cca2b57d358de2fdbaf8abdd76b553
If we don't close connection to ceph cluster,
The connection count increases with cinder bakcup until exceed max open files.
Then rise OSError: [Errno 24] Too many open files
This patch is to close connection to ceph cluster after cinder backup.
Change-Id: I2fb243d2a57771dc3589e96db54e998e2c1c8ef7
Closes-bug: #1628626
In unittesttools, assertDictEqual() and assertEqual()
are implemented by using '!=' operator. But assertEqual()
can handle dict, list, set and so on. So we just call
assertEqual() to make the tests simpler.
Change-Id: If4b7524c07cfac7f705ee466bef02b7de161f8bb
Looking into this more there isn't any documented reason why we do this,
and on Ubuntu 16.04 there are issues with timing and devices/symlinks
getting messed up when we do the reload of device maps. We shouldn't
need to be forcing multipathd to do this, it loads devices on its own.
We'll leave in the one in 'wait_for_rw(..)' for now because there is
some evidence that you may need to call it to update the rw state of
the multipath devices, see:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise
_Linux/6/html/Storage_Administration_Guide/ch37s04s02.html
Change-Id: Iec58284abdc9bcbf99df5d07289bb9d60a3554d7
Closes-Bug: #1623700
Test local_dev.test_brick_lvm.BrickLvmTestCase.test_lv_deactivate_timeout
doesn't mock time.sleep which is used for retries.
It increases test execution time to ~2s instead of ~0.02.
Change-Id: Id3704d90de0b3d04401c0b7b7686470685dbc1cc
Closes-Bug: #1620986
Some of tests use different method of assertTrue(isinstance(A, B))
The correct way is to use assertIsInstance(A, B) provided by testtools
Change-Id: Iaa27afbd59ed00a8254df96794f0c38bc408a239
If we run a command that returns non ascii character on stdout or stderr
using _execute attribute from Executor class or any of the inheriting
classes and we try to use stdout or stderr in the logs without decoding
it first, it will results in an exception like:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
19: ordinal not in range(128)
In existing code this can happen in quite a lot of places, and not only
where we are logging an error but also when we are logging progress or
debug information.
For example when running dd we could get in stdout:
1+0 registros leídos
1+0 registros escritos
512 bytes (512 B) copiados, 0,000368406 s, 1,4 MB/s
Instead of decoding stdout and stderr whenever we want to use them, this
patch takes another approach, capturing ProcessExecutionError exceptions
and converting string field to unicode and converting returned values of
the execution as well. That way we can safely use the exception
contents and returned values anywhere in the code.
Closes-Bug: #1602346
Change-Id: I46d8a47ae4f638dbb4b0dc125008bceb0f29fa15
This patch ensures that tracing log output is masking out passwords
to the log file.
Closes-Bug: 1616527
Change-Id: I5452ab8b993a184406331ad34abb9ceff24e4180
Current RBD connector assumes ceph cluster name to be 'ceph', for
cluster has a different name, the backup manager won't be able
to connect to the ceph cluster.
This fix creates temporary ceph conf file to include mon_hosts and
client section for cinder user to specify keyring path, the same conf
file will be passed to linux rbd initiator to connect to ceph cluster
and the temp file will be deleted.
The temporary conf will have required info required to
connect to ceph cluster like
mon_hosts = <ip>:<port>
[client.<user>]
keyring = <user_keyring_path>
This fix assumes that the keyring for ceph cinder volume
user will exist in /etc/ceph/ directory
Change-Id: Ie7583fff22e2b54324e70f89d9df98a3a6b99e7b
Closes-Bug: #1609659
The Cinder backup manager now uses initiator connectors for
creating and restoring backups for all volume backends, so
this patch adds a connector for GPFS volumes to fix the
backup-create & backup-restore for the GPFS driver.
DocImpact
Change-Id: I49cc52d6df5d0c1b470ced531075d851aa30208d
Implements: blueprint os-brick-gpfs-protocol-support
Jon Bernard