Commit Graph

219 Commits (master)

Author SHA1 Message Date
Ade Lee 933a7b7e6c Add fips check jobs
This patch adds two new FIPS enabled jobs to determine if there are any
issues when FIPS is enabled.

Because the FIPS jobs currently run on centos, code is added to the
test setup script to set up the databases correctly.  Also had
to increase the swap space on the nodes; see [0] for an explanation.



Change-Id: Ib85b6ecc6f1b12eb8afa866e56afbfb13aad0cba
2023-04-18 19:45:55 +00:00
Zuul d5bf53304c Merge "Support os-brick specific lock_path" 2022-09-10 00:04:36 +00:00
Gorka Eguileor ecefc7d67a Support os-brick specific lock_path
As a new feature, os-brick now supports setting the location of file
locks in a different location from the locks of the service.

The functionality is intended for HCI deployments and hosts that are
running Cinder and Glance using Cinder backend.  In those scenarios the
service can use a service specific location for its file locks while
only sharing the location of os-brick with the other services.

To leverage this functionality the new os-brick code is needed and
method ``os_brick.setup`` needs to be called once the service
configuration options have been loaded.

The default value of the os-brick ``lock_path`` is the one set in

This patch adds support for this new feature in a non backward
compatible way, so it requires an os-brick version bump in the

The patch also ensures that ``tox -egenconfig`` includes the os-brick
configuration options when generating the sample config.

Change-Id: I9f3e46c262f5b2b72fed5a4927165b3834a1c383
2022-09-09 19:47:00 +00:00
Eric Harney d5058ed598 mypy: work around mypy bug #13214
Turn off in-project stubs for now until this bug
is fixed and in a mypy release we can consume.

Change-Id: I079802bea4ae86ed29c57bd5273ef8e0e4bb6b3c
2022-07-26 09:50:56 -04:00
Luigi Toscano 1c0c25babb Native multibackend-matrix Zuul v3 job
Port the legacy legacy-tempest-dsvm-multibackend-matrix job to the
native Zuul v3 syntax, and rename it following the guidelines

This job tests the migration between two different backends
specified through the volume.backend_names configuration
key in tempest.conf.

Now the job leverages the existing zuul code, namely the
run-tempest role, which is called multiple times with all
the possible combinations of the 3 tested backends
(lvm, ceph, nfs) where the source and the destination differ.

The final JUnitXML output summarizes the test results
for each of the tested combinations.

Change-Id: I34e7e48ee63c4c269f82ae178a7118ed402cad6d
2021-09-08 01:05:44 +02:00
Brian Rosmaita c8c4fc953c Add installation of mypy stubs packages
The cinder-mypy job is failing because library stubs aren't installed
for requests [0].  Modify the to accept options specified
in an environment variable named OS_MYPY_OPTS to the mypy invocation,
and set this var in tox.ini.

The value is "--install-types --non-interactive" which is suitable for
CI purposes, and seems to make sense for local tox use as well [1].
The downside is it basically runs mypy twice, once to determine
whether there are any library stubs missing and then install them, and
againto do the actual check.  If we don't want this setting in
tox.ini, we can move it to .zuul.yaml for the cinder-mypy job run.

Also, update the version of mypy in test-requirements to a version
that supports the above options.

And, run mypy in its own env (instead of reusing pep8) so that the
tox logs are preserved during CI runs.


Change-Id: Id69cb519ee7300b33ff087de4e7d46cdad67d162
2021-07-15 10:33:36 -04:00
Zuul 3d67140cb7 Merge "tox mypy: Call directly" 2021-02-19 21:26:28 +00:00
Eric Harney d9c90cf488 tox mypy: Call directly
Tox issues a warning because bash is not
in whitelist_externals and indicates that this
will fail in tox 4+.

Just call directly to prevent this.

Change-Id: I3c1747b255338d189806b331ae6a3c1b37e6d744
2021-01-05 09:50:07 -05:00
Eric Harney ce060323f0 pylint: run with bash
[[ is a bash extension and therefore not
available on platforms where sh is not bash.
Use bash explicitly.

Change-Id: I3f31acd5fc3d7422270be53510962dcbc08db602
2021-01-04 08:46:50 -05:00
Eric Harney 6e53c264b6 Add mypy tox env
Add a "mypy" tox environment which runs mypy
type checking against Cinder code.

Taken from Stephen Finucane's Nova work at

Added "show_error_codes" and "pretty" options.

Generates an html report in ./mypy-report/

This adds stubs for oslo.i18n, so that _() calls
are annotated as intended.  It may be possible to
do this with less .pyi files carried along here.

Change-Id: I2589d22c1f16f2e177d34730a520591743c0c1e3
2020-10-14 08:24:13 -04:00
Sean McGinnis f766c76f2a
Make compatible with mysql8
Starting from mysql version 8 it's not possible to create a user
implictly when using GRANT.

This patch makes the behavior compatible with that.

Change-Id: Ia13402213bc82027e3d658918a22003be26576d6
Signed-off-by: Sean McGinnis <>
2020-06-22 14:27:34 -05:00
Thomas Goirand 2e89c931a6 Stop invoking Python 2
There's some remainings of Python 2 in a few .py that this patch
intends to fix. This patch has been in Debian for the last 2
years, it's probably about time to get this in upstream Cinder too.

Change-Id: I9c08c3ff1177c323a1c8cfd234ad58bd607e3c1a
2020-04-24 09:23:52 +02:00
Sean McGinnis 3eb9b422f4
Introduce flake8-import-order extension
This adds usage of the flake8-import-order extension to our flake8
checks to enforce consistency on our import ordering to follow the
overall OpenStack code guidelines.

Since we have now dropped Python 2, this also cleans up a few cases for
things that were third party libs but became part of the standard
library such as mock, which is now a standard part of unittest.

Some questions, in order of importance:

Q: Are you insane?
A: Potentially.

Q: Why should we touch all of these files?
A: This adds consistency to our imports. The extension makes sure that
   all imports follow our published guidelines of having imports ordered
   by standard lib, third party, and local. This will be a one time
   churn, then we can ensure consistency over time.

Q: Why bother. this doesn't really matter?
A: I agree - but...

We have the issue that we have less people actively involved and less
time to perform thorough code reviews. This will make it objective and
automated to catch these kinds of issues.

But part of this, even though it maybe seems a little annoying, is for
making it easier for contributors. Right now, we may or may not notice
if something is following the guidelines or not. And we may or may not
comment in a review to ask for a contributor to make adjustments to
follow the guidelines.

But then further along into the review process, someone decides to be
thorough, and after the contributor feels like they've had to deal with
other change requests and things are in really good shape, they get a -1
on something mostly meaningless as far as the functionality of their
code. It can be a frustrating and disheartening thing.

I believe this actually helps avoid that by making it an objective thing
that they find out right away up front - either the code is following
the guidelines and everything is happy, or it's not and running local
jobs or the pep8 CI job will let them know right away and they can fix
it. No guessing on whether or not someone is going to take a stand on
following the guidelines or not.

This will also make it easier on the code reviewers. The more we can
automate, the more time we can spend in code reviews making sure the
logic of the change is correct and less time looking at trivial coding
and style things.

Q: Should we use our hacking extensions for this?
A: Hacking has had to keep back linter requirements for a long time now.
   Current versions of the linters actually don't work with the way
   we've been hooking into them for our hacking checks. We will likely
   need to do away with those at some point so we can move on to the
   current linter releases. This will help ensure we have something in
   place when that time comes to make sure some checks are automated.

Q: Didn't you spend more time on this than the benefit we'll get from
A: Yeah, probably.

Change-Id: Ic13ba238a4a45c6219f4de131cfe0366219d722f
Signed-off-by: Sean McGinnis <>
2020-01-06 09:59:35 -06:00
Zuul 372de4e490 Merge "Fix pylint env for ancient versions of git" 2019-10-14 20:56:59 +00:00
Zuul b095236639 Merge "Move hacking checks to tests dir" 2019-10-07 21:23:19 +00:00
Zuul 819f5f8544 Merge "Add "service token" documentation" 2019-10-02 18:17:47 +00:00
Eric Harney 6f2efb871b Port to Python 3
This allows "tox -e pep8" to succeed on
CentOS 8.

This is only called from the pep8 env which
already specifies a basepython of python3, so
it should be ok.

Change-Id: Id11e137bc8e1fb50e551639c2be067647c5bc9d4
2019-09-26 10:13:53 -04:00
Brian Rosmaita bdf59306f7 Add "service token" documentation
Adds a doc about configuring and troubleshooting a service user
to send service tokens.

Also adds the config necessary for successful use of a service token
to the sample configuration file.  Only adds keystone v3 options
because v3 is required for service tokens.  Lists the service user
options before the session conf opts because the former are required
while the latter are less likely to be changed.

Closes-bug: #1837449

Change-Id: I4223624d00c713ab6b129ed278147f1c89995567
2019-09-26 10:26:21 +02:00
Eric Harney 3855eaca27 Move hacking checks to tests dir
This is not cinder service code, would be better
to move it into "tests".

Change-Id: I7b18453269cfc57bfbc9bd143d72948e88e60836
2019-09-13 10:06:48 -04:00
Eric Harney ddb42e3a0f Fix pylint env for ancient versions of git
On el7 w/git 1.8.3, "tox -e pylint" just returns:

fatal: ambiguous argument '*.py': unknown revision
or path not in the working tree.

Before this change.

Change-Id: I63bf975f85dda88ab5e76d8df46ba9a176bb2a3b
2019-09-13 10:00:51 -04:00
Eric Harney 1af55ce64a Pylint: use -j 0 arg
"-j 0" runs on the number of CPUs available, just
pass that instead of counting CPUs ourselves.

(This removes a stealthy python2 dependency for us,

Change-Id: Ib7d90efdbc692fa8e1f42615edec58ba3cef926b
2019-09-11 10:26:23 -04:00
Luigi Toscano 29bb1f7b4a Remove outdated scripts from tools/
- stestr (and ostestr) can be used instead,
  as they include its code;
- no references in the documentation,
  it looks like a relic;
- its only usage was removed when
  was removed, and it's being phased out from other repositories
  as well.

Change-Id: I566f893ae754687555fb3c4cb581e1b8a4eed95d
2019-07-21 22:59:39 +02:00
Jay S. Bryant b56dc7e152 Remove scanning upgrade checker from opts generation
The upgrade checker is going to need to be able to load
configuration settings to check for things that will
interfere with an upgrade.  This will cause genopts
to incorrectly think an entry needs to be added in
cinder/ .  This change add to the
exception list.

Change-Id: Iab71497b5e9405429976b385dde2f6b7d585abae
2019-04-02 21:52:26 -05:00
Walter A. Boring IV fb3b843b66 Add get_driver_options method
This patch adds a new static method to drivers to expose their
configuration options.  This also adds the same call to backup drivers
as well as the zone manager drivers.
Updated the generate_driver_list to expose the options as well.

This patch also orders the driver list alphabetically to make it easier
and consistent to find drivers.   The driver list is now broken down
into supported and unsupported drivers as well.

Change-Id: I3e7db26ef3df24a12e3bfa219fe25bfb315335ec
2019-03-07 09:18:01 -05:00
Ben Nemec 1542275a7c Add oslo.privsep to config-generator list
We recently exposed the privsep opts for config generator use, so
projects that depend on oslo.privsep should include them in their
sample configs.

Change-Id: I59702c8b84bbcf21d32150ede0a3758d1a456bd6
2019-02-27 19:15:15 +00:00
Eric Harney e8fe5aaf6a fast8 improvements
Determines how many commits to check based
on the $FAST8_NUM_COMMITS env variable.

If set to "smart", it uses git to try to run
against all unsubmitted commits.  This allows
fast8 to be more useful when actively developing
a series of patches.

Change-Id: I0adfba626ba2ea3479faf0f6ade712f14d3080d3
2019-01-22 14:36:23 -05:00
Chuck Short adcfb4f97e Simplify running pylint
This commit does several things:

- Setup and run pylint directly rather than running
  through a script. This allows the user to see what is happening
  while the user is running through pylint.
- Allow the user to either run pylint on a particular changeset,
  or the entire cinder tree.
- Allow the user to run on a particular changeset. Using like
  HEAD~1, etc.
- Since the pylint gate check I disabled the tests that were
  reported by pylint. The thought here would be go through
  the failures and correct them.
- Update pylint to 2.1.1.
- Removed old pylintrc.

Change-Id: I708c93843d991e8c0ac114b0b2d6ede373b96d0a
Signed-off-by: Chuck Short <>
2018-09-06 10:00:07 -04:00
wangxiyuan 1f91f7acba Add keystoneauth options to list_opts
The [service_user] group contains keystoneauth's session
options as well. This patch add these options back to list_opts.

Change-Id: I5a890c37b1bc23befb1f5792aa6129ca0c250917
2018-07-23 12:02:08 +08:00
Sean McGinnis 1c741636c7 Increase pylint processes
The default process concurrency for pylint is 1. Since our gate
images used to run the job have 8 cores, this results in slower
job execution with many cores sitting idle. To speed things up,
this sets the process count to match the number of cores available.

Change-Id: If4ba3333a1cb3b8bb6b4797f9149f412d32047ef
2018-06-20 10:35:26 -05:00
wanghao e396560f33 Keep ORM names matching their VO counterparts
Now cinder had some Versioned Objects which names do not
match their ORM counterparts. In method: get_model_for_versioned_object,
we handles those exceptions.

This patch fix this issue to keep the names match.

Change-Id: Icf709d87be99df95e5b52204032b730cd790096c
Closes-Bug: #1493112
2018-06-19 17:50:47 +08:00
Zuul f266406325 Merge "Clean output of driver list generation" 2018-06-14 22:03:37 +00:00
Sean McGinnis d675de18de Run pylint job under py3
This switches the pylint target to run under python3. In
order to work right, it also raises the version of pylint
used to a newer version.

Related-bug: #1761800

Change-Id: I0bc68a5aee24a53b9d3a08c20549b89d08989678
2018-06-06 16:28:34 -05:00
Sean McGinnis 37510ac034 Default tox jobs to python3
The long term goal for OpenStack is to support python3 for
all things except explicit python2 testing by the T release.
Part of preparing for that is ensuring all ancillary things
like docs jobs and other tooling work with python3.

This switches the default for our docs jobs to python3 so
anything that does not specify another version will end up
using it.

Further work is necessary for the pylint job due to changes
between the runtimes. That will be done in a follow up patch.

There was also a difference in behavior with the genopts job
where it ends up trying to recreate the venv under which it
is currently running under, resulting in a corrupted venv
and a failure. This cleans up that script and changes it so
rather than a tox job calling a tox job it just runs the

Change-Id: Ie79528b33070e6c60f9f5e1f2219084a2f0cb039
2018-06-05 14:16:34 -05:00
Sean McGinnis 113b383196 Clean output of driver list generation
The driver list generation takes driver docstrings and attempts
to parse that into rst formatted output for publication. With the
way this content is formatted within the docstring, this would
cause some slightly off formatting in the html output.

This attempts to better extract that information to make the
output cleaner and more readable.

Change-Id: I3b7e0817ade1c86b6d157c96872f00bb1b99c3d8
2018-04-25 17:15:23 -05:00
TommyLike 988f79479e Add policy documentation and sample file
This patch adds documentation and sample
file for default policy in code feature.

Change-Id: I597971a29ec61a1bf8c991b2715ec7644b2e2692
Partial-Implements: blueprint policy-in-code
2017-10-20 10:47:34 +08:00
Gyorgy Szombathelyi db6564bb74 Add another hack from NOVA_GROUP to doesn't really handle indirect variables
in group names, so add manual handlig of NOVA_GROUP.

Change-Id: I30a4666a058ebbf108fd42028fb0c39e5cb8bcae
Closes-Bug: #1717008
2017-09-14 10:19:17 +02:00
Sean McGinnis b4485ec09f Update generate_cinder_opts path
The location of the generate_cinder_opts script was moved in
d3abafdee4. There is a check in
the script to exclude the script to its parsing, but this path
was not updated when it was moved. This updated the path to
the new location.

Change-Id: I369dec534c69f67fea72959bc9d626dddd8768ff
2017-09-11 14:40:35 -06:00
Sorin Sbarnea 236f57e6ae Unhide errors from
Displays execution errors instead of hiding them.

Change-Id: I8f723d17dffb96f2aa07e7b584fa124af1441592
Signed-off-by: Sorin Sbarnea <>
2017-08-30 16:36:03 +01:00
Jay S. Bryant 8db6335bb5 [DOC BLD FIX] Make opts sections lowercase
The sections in the cinder/ file that are currently
upper case (I.E. BACKEND) cause a sphinx warning to be
generated.  When we switch to treating warnings as errors
this will break our doc build.

This patch updates the code that generates the cinder/
file to make sure that all of the section names are lower-case
with the exception of the 'DEFAULT' section.  With this
change in place we will not getting Sphinx warnings during
a doc build from the automatic config file generation.

Change-Id: Ibabbcaf7618623219690a2b27a2e548ea00be9d1
2017-08-29 10:53:48 -05:00
Eric Harney d3abafdee4 Move config-generator to tools
Moving this out of cinder/ separates it from the
actual run-time Cinder code.

Change-Id: I056f40120fdef60a0dc1a5926729748fd2835938
2017-08-16 16:18:48 -04:00
Eric Harney 6460050c5c Add releasenotes to check_exec list
Prevent addition of executable files in
the releasenotes/notes directory.

Change-Id: I80ac0b7fce7b16a453fe9139f0dacf88325af8e7
2017-08-16 15:50:44 -04:00
junboli db4327b6fe Keep consistent of naming convention
In the project, some of the terminology, like URL, URLs, API, APIs, OpenStack,
UUID, Cinder are neglectfully written as url, api, openstack, uuid, cinder.
This patch is to keep consistent of naming convention.

Change-Id: I98777fb4748cbc58b6e2fd1aca058d3e44069d07
2017-07-10 10:46:44 +08:00
Jenkins a58cb0c693 Merge "Add driver supported status to dict output format" 2017-03-27 20:58:49 +00:00
Erlon R. Cruz 183dca0f91 Fix multibackend-matrix confs
The command to edit tempest.conf should be run as the same
user that owns that file. This is causing the scripts to run always
with the same backends[1].


Change-Id: Ie865042c052746b5001f6132c159bc32165021a3
2017-02-15 16:01:32 -02:00
Erlon R. Cruz 91229faf59 Adds attached migration to multibackend-matrix
This test was left behind as it wasn't passing the LVM -> NFS
migration setup. As it wasn't possible to reproduce and fix localy,
let's add this to the gate so we can reproduce the error scenario.
Additionaly to that, some bug fixes need this to be fully tested.

Change-Id: Icd9138c1bc1335e9465184749bf6dd5733c605a3
2017-02-15 16:01:32 -02:00
Walter A. Boring IV 76304ef091 Output the driver info as json
This patch fixes the output of the dict format to json, so it can be
consumed a bit easier.

Change-Id: I7574c6eefddb8f085422387f9d4892b8dbe9977d
2017-02-10 21:24:10 +00:00
Andreas Jaeger afeb6f5aeb Minor cleanup on tools/
Remove unused variable.
Fix comment nit.

This addresses a review comment in

Change-Id: I47e75a68c7e97340dab8b51b9fab9ed2983061ef
2017-02-07 11:40:16 +01:00
Andreas Jaeger 1c91c0f033 Prepare for using standard python tests
Add simple script to setup mysql and postgresql databases, this script
can be run by users during testing and will be run by CI systems for
specific setup before running unit tests. This is exactly what is
currently done by OpenStack CI in project-config.

This allows to change in project-config the python-db jobs to
python-jobs since python-jobs will call this script initially.

See also

Update devref for this.

Needed-By: I628f825d9200f7951eae46b7be78b111b1a8141c
Change-Id: If549e6f108ec6184a432d5050da4386efe794a4e
2017-02-03 07:34:35 +01:00
Jenkins 275be41863 Merge "Gate migration tests: Add Cinder tempest hook" 2017-01-24 16:04:23 +00:00
Vipin Balachandran d34cd5cbac Add driver supported status to dict output format
Adding driver 'supported' status to the 'dict' output format
of driver listing.

Change-Id: I5ac1b151bd561991f12e4697b73fbbc3b8e5999b
2017-01-24 16:48:14 +05:30