The Sheepdog project is no longer active and the driver has been removed
from Cinder. This cleans up os-brick related code for the driver.
Change-Id: I9684db3ab134aeccce9f2ca47c29cdccaa8c5697
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Makes Python 3 the base python for all tox venvs and removes
py2 jobs from zuul.yaml.
Remaining:
- move the legacy gate jobs from openstack-zuul-jobs to the os-brick
repo and make them py3 jobs
- adjust setup.cfg to require py3 and remove py2 requirements from
test-requirements.txt and doc/requirements.txt
Partial-bug: #1853372
Change-Id: Ib447656aa213914bafd50380b3821090f93776f8
After some changes to the FC connector we have introduced a regression
on the way we do the scans, and we end up scanning using wildcards even
though we shouldn't.
The targets in the "initiator_target_map" don't mean that they are all
connected, so we must take that into account.
With the current code, if we have the following connections:
HBA host7 ---- SWITCH ---- port W (channel 0, target 2)
\--- SWITCH ---- port X (channel 0, target 3)
HBA host8 ---- SWITCH ---- port Y (channel 0, target 2)
\--- SWITCH ---- port Z (channel 0, target 3)
We will end up with the following scans 8 scans for LUN L:
- - L > host7
- - L > host7
0 2 L > host7
0 3 L > host7
0 2 L > host8
0 3 L > host8
- - L > host8
- - L > host8
Which correspond to the responses from _get_hba_channel_scsi_target like
this:
port Y port Z port W port X
host7 ... ['-','-',L] ['-','-',L] ['0','2',L] ['0','3',L]
host8 ... ['0','2',L] ['0','3',L] ['-','-',L] ['-','-',L]
And we should only be doing 4 scans:
0 2 L > host7
0 3 L > host7
0 2 L > host8
0 3 L > host8
Most storage arrays get their target ports automatically detected by the
Linux FC initiator and sysfs gets populated with that information, but
there are some that don't.
We'll do a narrow scan using the channel, target, and LUN for the former
and a wider scan for the latter.
If all paths to a former type of array were down on the system boot the
array could look like it's of the latter type and make us bring us
unwanted volumes into the system by doing a broad scan.
To prevent this from happening Cinder drivers can use the
"enable_wildcard_scan" key in the connection information to let us know
they don't want us to do broad scans even if no target ports are found
(because they know the cause is there's no connection).
Close-Bug: #1849504
Related-Bug: #1765000
Related-Bug: #1828440
Change-Id: I5dbefaff43fb902b15117b443fc92f7b6a6ad8c9
Add file to the reno documentation build to show release notes for
stable/train.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/train.
Change-Id: I5b7dad8b64ab837ef9bb82ef7c48c26ee878a589
Sem-Ver: feature
LUKS2 support was introduced into cryptsetup 2.0.0 [1] and offers various
improvements over the original format now referred to as LUKS1.
This change introduces an encryptor to os-brick mostly using the
existing LuksEncryptor class with the only difference being the `--type`
switch supplied to cryptsetup when formatting a volume. As such the bulk
of the _format_volume method from the original class has been extracted
into a new _format_luks_volume method both the original and new
Luks2Encryptor class can now reuse.
[1] https://www.saout.de/pipermail/dm-crypt/2017-December/005771.html
Change-Id: I09fb2b2be1e376f8ec0f49741c855cfd54ee27f0
This encryptor and the underlying plain cryptsetup encryption format it
provides is not widely used, inflexible (no key rotation) and as a
result no longer required in os-brick. This change starts the
deprecation process.
Additional changes will be posted to ensure the retype workflow between
this encryptor and the LUKS based encryptors is well tested ahead of any
removal.
Change-Id: Ibb560da269a2f330526af6761fa509c262e3d361
Some options are now automatically configured by the version 1.20:
- project
- html_last_updated_fmt
- latex_engine
- latex_elements
- version
- release.
Change-Id: I2281904c98f574dc5f42f31a12ebf5d1a4bdb48c
1. Sync sphinx dependency with global requirements. It caps python 2 since
sphinx 2.0 no longer supports Python 2.7.
2. Remove unncessary "=="
Change-Id: I2c87490faba283c4d4bcb3fbe386955a49601945
Current FC OS-Brick code only checks for single WWNN to exclude some
HBAs from scanning when we don't receive an initiator_target_map from
the backend.
There are storage arrays,like XtremIO, that due to their architecture
and design have all target ports automatically mapped to LUNs and are
returning the initiator_target_map, but some of those ports may not be
connected to our HBAs, so we end up with another case of bug #1765000.
This patch makes sure that we always check if the target implements
single WWNN, not only when we don't receive the initiator_target_map.
With this we decrease the likelihood of ending with unexpected devices
in our system, because now we will ignore unconnected HBAs (even if
reported in the initiator_target_map) if we are working with single WWNN
target.
Related-Bug: #1765000
Closes-Bug: #1828440
Change-Id: I02c208142f5b342f894666831449243863eccbfe
Add file to the reno documentation build to show release notes for
stable/stein.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/stein.
Change-Id: I54570e4d32416def24d0f73364a3e415085724b3
Sem-Ver: feature
This reverts commit a7f7abc5b8.
The Cinder ScaleIO driver has been rebranded to VxFlexOS, which is fine.
What shouldn't have happened is that the rebranding made it's way
into os-brick, resulting in a new connector protocol and mapping.
The new VxFlexOS driver in cinder should just use the existing
scaleio connector protocol, as it's not customer/user facing, and
ensures backwards and forwards compatibility.
Change-Id: Ia1e119c16091bbf6ff79e0acea8c1a7c656d6dd1
Dell EMC ScaleIO has been rebranded to VxFlex OS.
To follow this changes "scaleio" protocol renamed to "vxflexos",
"ScaleIOConnector" class renamed to "VxFlexOsConnector".
Old names will continue to work but will be removed in the Train
release.
Change-Id: I3bb1c985c64007c6960b6fa6f4cd893760a9a342
If a WWN has been provided in connection_properties, verify that the
found devices have that expected WWN. Fail the attachment if not. This
prevents cases where a device already exists on the host from an old
attachment that wasn't fully cleaned up, and the new attachment got
the same LUN. Using this old device could lead to data corruption.
Change-Id: I45a2221c0518213dc8132831c0bde9db4734da2b
Closes-Bug: 1813609
* When execute `nvme connect`, "host-nqn" is needed in RSD environment.
So add it as an optional parameter.
* It is observed that the first `nvme list` can miss the newly connected
nvme device, right after a success call of `nvme connect`. The
solution is to add the retry&sleep logic back.
* Sometimes there can be more than 10 nvme devices, e.g.
"/dev/nvme10n10". Need to update the device re pattern.
* In RSD environment, the connector needs additional information like
"system uuid", to let cinder-volume know the nova-cpu service is
running on which RSD node.
* Add a new protocol "NVMEOF" mapping to allow both Cinder and Nova to
identify and use the nvme connector.
Implements: blueprint nvme-volume-driver
Change-Id: I26e3dc140b2cf30a97665679fdc4d2f897cd4872
Under certain conditions detaching a multipath device may result on
failure when flushing one of the individual paths, but the disconnect
should have succeeded, because there were other paths available to flush
all the data.
OS-Brick is currently following standard recommended disconnect
mechanism for multipath devices:
- Release all device holders
- Flush multipath
- Flush single paths
- Delete single devices
The problem is that this procedure does an innecessary step, flushing
individual single paths, that may result in an error.
Originally it was thought that the individual flushes were necessary to
prevent data loss, but upon further study of the multipath-tools and the
device-mapper code it was discovered that this is not really the case.
After the multipath flushing has been completed we can be sure that the
data has been successfully sent and acknowledge by the device.
Closes-Bug: #1785669
Change-Id: I10f7fea2d69d5d9011f0d5486863a8d9d8a9696e
Implementing necessary mathods and rescaning storage
as necessary to support extending ScaleIO volumes
which are currently attached.
Change-Id: Icb9fc4fe1c16c34d8de9287046045d0837b7e80e
In the case that a compute host with multiple NICs can be connectted to
iscsi storage network, we expect the storage traffic only be
transimitted via storage NIC. This need to define an custom iface and set the
iface.net_ifacename value to the storage interface name.
Unfortunately, the current iscsi connector does not support to use a
custom iface. The _validate_iface_transport will change it to default.
After add tcp to the supported_transports, it is possible to use a
custom iface.
Please get the details as below:
1. CMP001 has three NICs: NIC1 is for management network, NIC2 is
for tenant network, and NIC3 is for storage network. NIC3 can access
storage device via layer 2 network. And NIC1 can access storage device
too, but it can only access via layer 3 network. We hope all the storage
traffic only be transimitted via NIC3, because they are in the same
layer 2 network.
2. Get the NIC3 MAC:
fa:16:3e:79:fd:63
3. Add new iscsi iface:
# iscsiadm -m iface --op=new -I tcp.fa:16:3e:79:fd:63
# iscsiadm -m iface -I tcp.fa:16:3e:79:fd:63 --op=update
-n iface.hwaddress -v fa:16:3e:79:fd:63
4. Edit nova.conf, and change iscsi_iface value:
iscsi_iface = tcp.fa:16:3e:79:fd:63
5. Restart nova compute service:
# systemctl restart openstack-nova-compute
Without this change, _validate_iface_transport will change the
custom iface tcp.fa:16:3e:79:fd:63 to default due to it cannot find tcp
transport.
According to this change, we should have a corresponding change to the
nova docs, here is change to nova
https://review.openstack.org/#/c/524443/
Change-Id: I85a62cb6e7f8d8982e97e792d647d38ce5641060
Closes-Bug:#1722432
Release notes are version independent, so remove version/release values.
We've found that most projects now require the service package to be
installed in order to build release notes, and this is entirely due to
the current convention of pulling in the version information.
Release notes should not need installation in order to build, so this
unnecessary version setting needs to be removed.
Change-Id: I0ca29d19436422a382bce8c23bba24a6e677c434
Needed-by: I56909152975f731a9d2c21b2825b972195e48ee8
These constants were moved over a year ago and we have had several releases
since then. The old ones that were kept for backwards compatibility can now
be removed.
Depends-on: I8682804d2299db793c1e4397a07ff67608a8bda6
Change-Id: Ia18de03880ca5b04d31dbdf7891fa6d3240ae9b5
Clean up both the releasenotes 'conf.py' and some remaining pieces of
the main 'conf.py'. In the case of the former, it looks like this was
copied from Cinder's config, and retains a lot of unnecessary config for
same. Remove it all.
It is also necessary to remove a docstring from 'os_brick/__init__.py',
as this was causing build issues. No idea how this was building
previously.
Change-Id: I2ca913c2ebcf2030642b18c72af180bb8f03b288
It was recently added to open-iscsi the functionality to disable
automatic LUN scans on iscsid start, on login, and on reception of
AEN/AER messages reporting LUN data has changed.
Those 3 cases were one of the causes why Nova-CPU and Cinder-Volumes
nodes would have unexpected devices. With this new feature we can
prevent them from appearing unnexpectedly.
This patch adds the mechanism required to configure our sessions for
manual scans in a backward compatible way.
Manual scans are enabled setting `node.session.scan` to `manual`.
Change-Id: I146a74f9f79c68a89677b9b26a324e06a35886f2
This patch refactors iSCSI connect code changing the approach to one
that relies primarily on sysfs, instead of CLI tools, to retrieve all
the required information: devices from the connection, multipath system
device name, multipath name, the WWN for the block devices...
By doing so, not only do we fix a good number of bugs, but we also
improve the reliability and speed of the mechanism.
A good example of improvements and benefits achieved by this patch are:
- Clean all leftovers on exceptions on a connection.
- Parallelize logins on multipath to increase speed on flaky network
connections.
- As long as there is at least one good path when working with multipath
we will return a multipath device instead of a single path device,
which helps with temporary flaky connections.
- Don't use the rescan retry parameter as log in retry on multipath
connections so both single and multipath cases are consistent.
- We no longer rely on just one device to get the wwn and look for the
multipath. This would be problematic with flaky connections.
- No more querying iSCSI devices for their WWN (page 0x83) removing
delays and issue on flaky connections.
- It's no longer a problem for the mechanism the fact that a device
exists but is not accessible.
- We use links in `/dev/disk/by-id` to get the WWID on connect, so we
make sure there are no leftovers on disconnect, but we no longer use
symlinks from `/dev/disk/by-path`, `/dev/disk/by-id`, or `/dev/mapper`
to find devices.
- We no longer need to rely on the WWN to determine the multipath, we
have the session and the LUN, so we trace the devices and from those
we get if they belong to a multipath.
- Stop polluting the logs with unnecessary exceptions from checking if
the node or session exist.
- Action retries will now only log the final exception instead of
logging all the exceptions.
- Warn when a multipath could not be formed and a single device is being
used, as performance will be degraded.
- We no longer do global rescans on single connections that could be
bringing in unrelated and unwanted devices (`iscsiadm -T iqn -p portal
--rescan`).
- Fix scan mechanism that would not request all scans when the same iqn
was shareed between portals and could send a scan request to the wrong
IP if they shared the same iqn.
- When doing single pathed connections we could end with a multipath
because we didn't clean up unfruitful connections.
It's worth mentioning that this patch does not touch the extend
operation.
Change-Id: Ia1c47bfaa7bc3544a5feba6a8a30faf2f132b8d7
This patch refactors iSCSI disconnect code changing the approach to one
that just uses `iscsiadm -m session` and sysfs to get all the required
information: devices from the connection, multipath system device name,
multipath name, the WWN for the block devices...
By doing so, not only do we fix a good number of bugs, but we also
improve the reliability and speed of the mechanism.
A good example of improvements and benefits achieved by this patch are:
- Common code for multipath and single path disconnects.
- No more querying iSCSI devices for their WWN (page 0x83) removing
delays and issue on flaky connections.
- All devices are properly cleaned even if they are not part of the
multipath.
- We wait for device removal and do it in parallel if there are
multiple.
- Removed usage of `multipath -l` to find devices which is really slow
with flaky connections and didn't work when called with a device from
a path that is down.
- Prevent losing data when detaching, currently if the multipath flush
fails for any other reason than "in use" we silently continue with the
removal. That is the case when all paths are momentarily down.
- Adds a new mechanism for the caller of the disconnect to specify that
it's acceptable to lose data and that it's more important to leave a
clean system. That is the case if we are creating a volume from an
image, since the volume will just be set to error, but we don't want
leftovers. Optionally we can tell os-brick to ignore errors and don't
raise an exception if the flush fails.
- Add a warning when we could be leaving leftovers behind due to
disconnect issues.
- Action retries (like multipath flush) will now only log the final
exception instead of logging all the exceptions.
- Flushes of individual paths now use exponential backoff retries
instead of random retries between 0.2 and 2 seconds (from oslo
library).
- We no longer use symlinks from `/dev/disk/by-path`, `/dev/disk/by-id`,
or `/dev/mapper` to find devices or multipaths, as they could be
leftovers from previous runs.
- With high failure rates (above 30%) some CLI calls will enter into a
weird state where they wait forever, so we add a timeout mechanism in
our `execute` method and add it to those specific calls.
Closes-Bug: #1502534
Change-Id: I058ff0a0e5ad517507dc3cda39087c913558561d
Initially Ic155bd29d46059832cce970bf60375e7e472eca6 had aimed to remove these
legacy provider names during the Pike cycle. While removing references
to these names in Tempest via Id221414d74af8413084c7935b762f93b7ce43c42 it was
highlighted that this wouldn't be possible until Queens and the eventual EOL of
Newton.
This change simply updates the logged deprecation warnings and adds a
new releasenote highlighting that these legancy provider names will
remain until Queens.
Change-Id: Ib8ca7ecb5494cd4fcae0657525d4c17bfc4ae37e
Implementation of an os-brick connector for Veritas HyperScale.
The Nova volume driver implementation is being reviewed here:
https://review.openstack.org/#/c/443951/
Change-Id: I01e26642f8fe4bc737c69493bb6ea6628bf72108
Implements: blueprint veritas-hyperscale-cinder-driver
Depends-On: Ie1af5f5d54b0115974a4024a1756e4e0aa07399a
If a 'keyring' key is found in the connection info passed to
connect_volume() use its value as the path to the keyring instead of the
default location (/etc/ceph/<cluster>.client.<user>.keyring).
This allows services such as cinder's RBD and Ceph backup drivers to
make use of a custom keyring path that an admin has defined.
Change-Id: Ib1230d3e40f56371567e1aead40db59667bad295
Closes-bug: #1668304
These constants detail the supported encryption formats and their
associated in tree encryption provider implementations.
The use of out of tree and direct use of these in tree implementations
is now deprecated and will be blocked in the Pike release of os-brick.
Change-Id: Ic155bd29d46059832cce970bf60375e7e472eca6
Partial-bug: #1639293
Releasenote translation publishing is being prepared. 'locale_dirs'
needs to be defined in conf.py to generate translated version of the
release notes.
Note that this repository might not get translated release notes - or
no translations at all - but we add the entry here nevertheless to
prepare for it.
Change-Id: I5e8b3d2865cfa96e45591b98846e03a9f9ec718f
In order to support automatically updating the release notes when we
create stable branches, we want the pages to be in a standard order.
This patch updates the order to be reverse chronological, so the most
recent notes appear at the top.
Change-Id: Ib364dcc8eb31275a31c83b68d7914263b183e393
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Change I743e676372703e74178c79683dd622d530981e04 removed volume
backend driver calls for creating and restoring volume backups.
The overridden methods for creating and restoring backups in
the VMDK driver is no longer called and this breaks the backup-
restore of volumes created by the VMDK driver. The Cinder backup
manager now uses initiator connectors for creating and restoring
backups for all volume backends. The patch adds a connector for
vmdk volumes to fix the backup-restore for the VMDK driver.
DocImpact
Change-Id: Ia1a20f93780593b1efbb74484c3fdd3ca3564290
Partial-bug: #1602660
This patch adds a Windows Fibre Channel connector.
The patch using Windows os-brick connectors in the Hyper-V
Nova driver: https://review.openstack.org/#/c/273504/
Change-Id: Iec263e5d5803fcceb315e17d16d2b154e0214584
Partial-Implements: blueprint os-brick-windows-support
This patch adds a Windows SMBFS connector. Also, a Windows
RemoteFS class is added, being used by this connector, having
a similar interface with the Linux RemoteFS client class.
The patch using Windows os-brick connectors in the Hyper-V
Nova driver: https://review.openstack.org/#/c/273504/
The Windows connector factory function has been removed as it's not
needed anymore.
Change-Id: I0c753a667d58391da7a903d11ab1f4729e68461a
Implements: blueprint os-brick-windows-support
Brian Rosmaita
OpenStack Release Bot