yuval brave beb2879793 Lightbits LightOS driver
This commit introduces the LightOS driver for cinder.
LightOS is a software-defined disaggregated clustered storage solution
running on commodity servers with commodity SSDs.
It is developed by Lightbits Labs (
and is actively developed and maintained.
LightOS is proprietary but the openstack drivers are
licensed under Apache v2.0.

The Cinder driver for LightOS currently supports the
 following functionality:

Create volume
Delete volume
Attach volume
Detach volume
Create image from volume
create volume from image
Live migration
Volume replication
Thin provisioning
Extend volume
Create snapshot
Delete snapshot
Create volume from snapshot
Create volume from volume (clone)

This driver has been developed and has been in use for a couple
of years by Lightbits and our clients.
We have tested it extensively internally with multiple openstack
versions, including Queens, Rocky, Stein, and Train.
We have also tested it with master (19.1 xena) and we are working to
extend testing to cover additional openstack releases.

We are glad to join the openstack community and hope to get your
feedback and comments on this driver, and if it is acceptable,
to see it merged into the tree.

Signed-off-by: Yuval Brave

Depends-On: I2e86fa84049053b7c75421d33ad1a1af459ef4e0
Change-Id: I8e6eab23419a4e7f092bb49aa85a4aaab5a8ba38
2022-01-26 22:13:26 +02:00

4.5 KiB

Lightbits LightOS Cinder Driver

The Lightbits(TM) LightOS(R) OpenStack driver enables OpenStack clusters to use LightOS clustered storage servers. This documentation explains how to configure Cinder for use with the Lightbits LightOS storage backend system.

Supported operations

  • Create volume
  • Delete volume
  • Attach volume
  • Detach volume
  • Create image from volume
  • Live migration
  • Volume replication
  • Thin provisioning
  • Multi-attach
  • Supported vendor driver
  • Extend volume
  • Create snapshot
  • Delete snapshot
  • Create volume from snapshot
  • Create volume from volume (clone)

LightOS OpenStack Driver Components

The LightOS OpenStack driver has three components: - Cinder driver - Nova libvirt volume driver - os_brick initiator connector

In addition, it requires the LightOS discovery-client, provided with LightOS. The os_brick connector uses the LightOS discovery-client to communicate with LightOS NVMe/TCP discovery services.

The Cinder Driver

The Cinder driver integrates with Cinder and performs REST operations against the LightOS cluster. To enable the driver, add the following to Cinder's configuration file

enabled_backends = lightos,<any other storage backend you use>


volume_driver = cinder.volume.drivers.lightos.LightOSVolumeDriver
volume_backend_name = lightos
lightos_api_address = <TARGET_ACCESS_IPS>
lightos_api_port = 443
lightos_default_num_replicas = 3
lightos_default_compression_enabled = False
  • TARGET_ACCESS_IPS are the LightOS cluster nodes access IPs. Multiple nodes should be separated by commas. For example: lightos_api_address =,, These IPs are where the driver looks for the LightOS clusters REST API servers.
  • LIGHTOS_JWT is the JWT (JSON Web Token) that is located at the LightOS installation controller. You can find the jwt at ~/lightos-default-admin-jwt.
  • The default number of replicas for volumes is 3, and valid values for lightos_default_num_replicas are 1, 2, or 3.
  • The default compression setting is False (i.e., data is uncompressed). The default compression setting can also be True. This can also be changed on a per-volume basis.
  • The default time to wait for API service response is 30 seconds per API endpoint.

Creating volumes with non-default compression and number of replicas settings can be done through the volume types mechanism. To create a new volume type with compression enabled:

$ openstack volume type create --property compression='<is> True' volume-with-compression

To create a new volume type with one replica:

$ openstack volume type create --property lightos:num_replicas=1 volume-with-one-replica

To create a new type for a compressed volume with three replicas:

$ openstack volume type create --property compression='<is> True' --property lightos:num_replicas=3 volume-with-three-replicas-and-compression

Then create a new volume with one of these volume types:

$ openstack volume create --size <size> --type <type name> <vol name>

NVNe/TCP and Asymmetric Namespace Access (ANA)

The LightOS clusters expose their volumes using NVMe/TCP Asynchronous Namespace Access (ANA). ANA is a relatively new feature in the NVMe/TCP stack in Linux but it is fully supported in Ubuntu 20.04. Each compute host in the OpenStack cluster needs to be ANA-capable to provide OpenStack VMs with LightOS volumes over NVMe/TCP. For more information on how to set up the compute nodes to use ANA, see the CentOS Linux Cluster Client Software Installation section of the Lightbits(TM) LightOS(R) Cluster Installation and Initial Configuration Guide.


In the current version, if any of the cluster nodes changes its access IPs, the Cinder driver's configuration file should be updated with the cluster nodes access IPs and restarted. As long as the Cinder driver can access at least one cluster access IP it will work, but will be susceptible to cluster node failures.

Driver options

The following table contains the configuration options supported by the Lightbits LightOS Cinder driver.