Commit Graph

672 Commits

Author SHA1 Message Date
Zuul
43802e5170 Merge "Fix user creation with GRANT in MySQL 8.0(Ubuntu Focal)" 2020-07-02 18:40:46 +00:00
Ghanshyam Mann
e0d541073d Fix user creation with GRANT in MySQL 8.0(Ubuntu Focal)
In Ubuntu Bionic (18.04) mysql 5.7 version used to create
the user implicitly when using using the GRANT.

Ubuntu Focal (20.04) has mysql 8.0 and with mysql 8.0 there
is no implicit user creation with GRANT. We need to
create the user first before using GRANT command.

Nova unit and functional tests job using tools/test-setup.sh
script start failing when running on Ubuntu Focal

https://zuul.opendev.org/t/openstack/build/8b0f4fcc21854655a638c413b6fe1a91

Error log:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1064 (42000) at line 4: You have an error in your SQL syntax; check the manual
that corresponds to your MySQL server version for the right syntax to use near
'identified by 'openstack_citest' WITH GRANT OPTION' at line 2

Story: #2007865
Task: #40200

Change-Id: I97b0dcbb88c6ef7c22e3c55970211bed792bbd0d
Closes-Bug: #1885825
2020-06-30 21:21:39 -05:00
Elod Illes
c7c48c6f52 Fix cherry-pick check for merge patch
Cherry-pick check script validates the proposed patch's commit message.
If a patch is not on top of the given branch then Zuul rebases it to
the top and the patch becomes a merge patch. In this case the script
validates the merge patch's commit message instead of the original
patch's commit message and fails.

This fix selects the parent of the patch if it is a merge patch.

Change-Id: I8e4e5afc773d53dee9c1c24951bb07a45ddc2f1a
2020-06-26 20:31:20 +02:00
Dan Smith
aebc829c4e Check cherry-pick hashes in pep8 tox target
This adds a tools/ script that checks any cherry-picked hashes
on the current commit (or a provided commit) to make sure that
all the hashes exist on at least master or stable/.* branches.
This should help avoid accidentally merging stable backports
where one of the hashes along the line has changed due to conflicts.

Change-Id: I4afaa0808b75cc31a8dd14663912c162281a1a42
2020-06-15 11:15:31 -07:00
Stephen Finucane
34ecf5ab91 tox: Integrate mypy
mypy is an experimental optional static type checker for Python that
aims to combine the benefits of dynamic (or "duck") typing and static
typing. While still in development, most features are supported and it's
already being used by real world projects like Sphinx. Let's start small
by integrating it into some of the interfaces that nova exposes. We can
further build upon this if it works out.

This change sets up the boilerplate necessary to use mypy in nova. Type
annotations are not included for any module - these will be added
separately. We're calling mypy by way of a script, as this allows us to
store a list of files that we have converted while we're in the process
of adding type annotations where necessary.

Change-Id: I75ab46a6768c4ca2050fdde2b7f8eeb90724c8c6
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2020-05-15 15:59:53 +01:00
Stephen Finucane
1515a229e3 Remove future imports
These particular imports are no longer needed in a Python 3-only world.

Change-Id: Ia1b60ce238713b86f126e2d404199d102fdbc5bc
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2020-03-24 15:05:36 +00:00
Monty Taylor
e7cc5efd20 Keep pre-commit inline with hacking and fix whitespace
The flake8 hook for pre-commit installs its own flake8 which is
not what hacking is pinned to. This results in a bunch of errors.
By doing a local hook we can cause these to be the same.

There are flake8 exclusions in tox.ini that need to be used
in pre-commit. Also we should exclude svg files from the whitespace
modifications, because who cares.

Once those are fixed, running pre-commit -a fixes some whitespace
issues. While the normal use isn't to run -a - if someone does make
a patch that touches any of these they'll get erroneous unrelated
errors compared to their lower work - so fix all of the existing
issues, which are mostly space-at-end-of-line issues it seems.

Change-Id: I78e753492e99c03b30a0a691db3bd75ee3d289c9
2019-12-12 14:56:39 +00:00
Stephen Finucane
5f8dd65f6c Integrate 'pre-commit'
This nifty tool provides a extensible way to configure pre-commit hooks
to do everything from running linters to spellchecking [1] and has
already been adopted by a few OpenStack projects [2]. Let's start using
it as a faster, more automatic alternative to the 'fast8' tox target and
iterate on it as we find more things to use it for. The 'fast8' target
is not deprecated though a note is included for end-users to alert them
to this great new tool.

Note that the 'check-executables-have-shebangs' actually failed and
highlighted that the 'contributor/development-environment.rst' document
was marked as executable when it shouldn't have been. This is corrected.

[1] https://pre-commit.com/
[2] https://review.opendev.org/#/q/file:.pre-commit-config.yaml

Change-Id: I256e75eceeb3495dce04885ca42d16adb4160cf3
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2019-11-28 12:46:55 +00:00
Adam Spiers
5df748b2ed Make it easier to run a selection of tests relevant to ongoing work
During development of a new git commit, locally running a whole unit
or functional test suite to check every minor code change is
prohibitively expensive.  For maximum developer productivity and
happiness, it's generally desirable to make the feedback loop of the
traditional red/green cycle as quick as possible.

So add run-tests-for-diff.sh and run-tests.py to the tools/
subdirectory, using a few tricks as explained below to help with this.

run-tests.py takes a list of files on STDIN, filters the list for
tests which can be run in the current tox virtualenv, and then runs
them with the correct stestr options.

run-tests-for-diff.sh is a simple wrapper around run-tests.py which
determines which tests to run using output from "git diff".  This
allows running only the test files changed/added in the working tree:

    tools/run-tests-for-diff.sh

or by a single commit:

    tools/run-tests-for-diff.sh mybranch^!

or a range of commits, e.g. a branch containing a whole patch series
for a blueprint:

    tools/run-tests-for-diff.sh gerrit/master..bp/my-blueprint

It supports the same "-HEAD" invocation syntax as flake8wrap.sh (as
used by the "fast8" tox environment):

    tools/run-tests-for-diff.sh -HEAD

run-tests.py uses two tricks to make test runs as quick as possible:

  1. It's (already) possible to speed up running of tests by
     source'ing the "activate" file for the desired tox virtualenv,
     e.g.

        source .tox/py36/bin/activate

     and then running stestr directly.  This saves a few seconds by
     skipping the overhead introduced by running tox.

  2. When only one test file needs to be run, specifying the -n option
     to stestr will skip the costly test discovery phase, saving
     several more valuable seconds.

Future commits could build on top of this work, harnessing a framework
such as watchdog / watchmedo[0] or Guard[1] in order to automatically
run relevant tests every time your editor saves changes to a .py file.

[0] https://github.com/gorakhargosh/watchdog - Python-based
[1] https://guardgem.org - probably best in class, but Ruby-based so
    maybe unacceptable for use within Nova.

Change-Id: I9a9bda5d29bbb8d8d77f769cd1abf7c42a18c36b
2019-08-19 17:48:39 +01:00
ZhongShengping
7ecaa3fcf8 Replace git.openstack.org URLs with opendev.org URLs
Thorough replacement of git.openstack.org URLs with their opendev.org
counterparts.

Change-Id: I3e0af55e0707f04428a422b973d016ad30c82a12
2019-04-24 13:59:57 +08:00
Matthew Edmonds
d7ed9a550b Make Xen code py3-compatible
A couple places in the Xen code have been using syntax that is
python2-specific. This replaces those instances with code that will
work in both python2 and python3.

PEP 3108 [1] moves urllib.urlopen to urllib.request.urlopen. The six
module is used in order to work in both python2 and python3 [2].

PEP 3110 [3] changed the grammar for except clauses, such that the
'as' keyword is now required in place of a comma when specifying an
exception variable.

[1] https://www.python.org/dev/peps/pep-3108/
[2] https://pythonhosted.org/six/#module-six.moves.urllib.request
[3] https://www.python.org/dev/peps/pep-3110/

Change-Id: I1235d767718a4207f4cef3e5b140319d003ad7b0
2018-08-10 20:04:19 +00:00
Stephen Finucane
a34731dad1 Convert 'placement_api_docs' into a Sphinx extension
This ensures it will get run in the gate.

Change-Id: I923c39d3115b595b8de94e85d2977eb7782d98e2
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2018-07-02 11:21:47 +01:00
Stephen Finucane
de4ab973d5 trivial: Remove 'tools/releasenotes_tox.sh'
This is not needed since reno 2.1.0 (commit f8fc8f97) [1].

[1] f8fc8f97ff

Change-Id: I66ba0e0835970b965acce00ba69d143bd4471037
2018-06-28 15:06:46 +01:00
Matt Riedemann
9f48aee9b0 Remove deprecated monkey_patch config options
These were deprecated in Queens:

  Id793ac2c5cdc5dc473f95eac53b77617a1e389da

And can now be removed.

Change-Id: I27518d1154ec09c57700905bc58ae6a3ce1d8049
2018-05-16 11:40:41 -04:00
Matthew Treinish
9c56c78513
Finish stestr migration
Nova switched to using stestr for all it's in tree testing except for
the coverage job. This commit updates that job so it also uses stestr
and cleans up the last bits of testr setup left in tree. As part of
this change this moves all the one off per job configuration into the
tox job definitions instead of hiding it in either the .stestr.conf or
in a wrapper script.

Change-Id: I36e0e791d485b44641a1ed8770eca8e95aca19be
2017-11-24 16:51:12 -05:00
naichuans
693ace79fb xenapi: cached images should be cleaned up by time
For xenapi driver, there needs to be some way to delete cached
images based on when they were created. add an optional arg to
control delete operation.

Change-Id: I24fc45e989aa951aab55a261fce77f7e3667d988
Closes-bug: 1481689
2017-10-23 16:29:16 +00:00
melanie witt
53f244da12 Make setenv consistent for unit, func, and api-samples
Since ostestr switched to running stestr under the covers, we lost the
old magic setting of the environment variables via .testr.conf for
capturing stderr/stdout and the test timeout. This makes the unit,
functional, and api-samples envs consistent with the py27 env that was
already updated to set those variables.

This also updates the pretty_tox3.sh script to run stestr directly.

Change-Id: I27fa9b7e25c1a1dc921653eec84864423f898a85
2017-10-05 21:45:46 +00:00
Jenkins
823b118b8c Merge "trivial: Remove files from 'tools'" 2017-09-06 18:31:35 +00:00
Chris Dent
c324551015 Prevent blank line at start of migration placeholders
Because those blank lines sure are annoying.

Change-Id: I7b73536a89d18e0752651c198018eff6cc8b0e57
2017-08-25 18:43:19 +01:00
Andrey Volkov
fa5749727f [placement] Make placement_api_docs.py failing
If there is no api reference for some placement API method
tox -e placement-api-ref will fail now.

Change-Id: Ifefedcd51f1f8d5e741ef6d02b7a0879a185e05f
2017-08-10 11:27:13 +00:00
Stephen Finucane
017b5b869a trivial: Remove files from 'tools'
Two files are removed.

'ebtables.workaround' was required for libvirt < 1.2.11 for the reasons
described in the file itself. We no longer support this version of
libvirt, thus, the workaround can be removed.

'regression_tester' is a tool for running tests in a patch against old
code, to ensure the tests validate what the patch is supposed to be
fixing. This is unmaintained and is not referenced anywhere, and should
therefore be removed as it's likely bitrotted (or will do so).

Change-Id: Iac7f16ff4c178b5e71b786c332ca46a78bef5e3c
TrivialFix
2017-08-07 16:01:43 +01:00
Stephen Finucane
21d5dfacb4 tools/xenserver: Remove 'cleanup_sm_locks'
XenServer 6.0 has long since been replaced and the bug was fixed in 6.1
[1]. No need to keep this legacy script around.

[1] https://bugs.launchpad.net/nova/+bug/1103158/comments/1

Change-Id: I8c41e88281aaf4250859622142a2f03f4452e1be
2017-08-07 12:26:53 +00:00
Andrey Volkov
90cabd0c8c [placement] Fix placement-api-ref check tool
Some time ago the placement-api-ref source structure was changed
(Ia2fd62ae7f401cad34ee7c2b355c9a5ab1c93f6b) to use inc-files istead of
putting all in index.rst, but that change broke placement_api_docs.py.

This change modifies placement_api_docs.py to go through all *.inc files
in placement-api-ref directory and scan for the "rest method" header.

Change-Id: I60150ab51173934f848662499dcf6af18625697e
2017-05-24 12:15:28 +03:00
Jenkins
4613b33674 Merge "Structure for simply managing placement-api-ref" 2017-03-24 20:17:52 +00:00
Chris Dent
8c59991363 Structure for simply managing placement-api-ref
This consists of a duplicate of the [nova-]api-ref setup and
conf.py along with tooling to fail the tox -edocs target when
a route that is defined in
nova.api.openstack.placement.handler.ROUTE_DECLARATIONS is not
present in placement-api-ref/source/index.rst.
tools/placement_api_docs.py will report which routes are missing.

Though completely gameable (as demonstrated in the current lame
index.rst) it's better than nothing and provides some useful
structuring on what to do next. It's also the case that the 'docs'
target in tox is not part of gating.

The response for GET / is in place with the necessary
parameters.yaml for it to be correctly described. The 'get-root.json'
file provides the JSON of the expected response. The expectation is
that later commits will add information for other urls and their
JSON files will be named method-path-separated-by-dash.json with a
request/response qualifier as necessary.

Followup patches will add other routes.

A new parameters.yaml is used instead of reusing the one from
api-ref as there isn't a lot of expected overlap and having a
separate file will ease eventual extraction.

Running tox -eplacement-api-ref will generate the docs for review,
with output in placement-api-ref/build/html/index.html.

This will be hooked up with CI to deploy the generated docs,
eventually.

Change-Id: Ifb4d91d39db0e49b55952e37cdfc9f63dcd37aa3
2017-03-24 15:15:53 +03:00
Jenkins
e3eac21382 Merge "More usage of ostestr and cleanup an unused dependency" 2017-03-20 16:04:47 +00:00
ChangBo Guo(gcb)
6b73c281bc Remove usage of config option verbose
oslo.log deprecate config option verbose since aug 1, 2015 in review:
in https://review.openstack.org/#/c/206437/

That was a long time ago so it should be possible to remove it now. This
was already merged once but had to be reverted because some projects
were still relying on it. Oslo team plan to remove it, so we need clean
up its usage before removing it.

Closes-Bug: #1663570

Change-Id: I072e1078486f50cebea92695ff46c1af879eb23d
2017-02-27 10:02:45 +08:00
Luigi Toscano
8e34aece1a More usage of ostestr and cleanup an unused dependency
Use ostestr as runner instead of the custome pretty_tox.sh script
(pretty much removed everywhere). pretty_tox3.sh is just a wrapper
for ostestr but it could be probably removed as well going forward.
Also remove the tempest-lib dependency, which originally provided
subunit-trace (now from os-testr).

Change-Id: I4390964ce83167d634b80f696e82a0e199582ce6
2017-02-23 18:25:08 +01:00
Jenkins
a026af51dc Merge "tools: Remove 'colorizer'" 2017-02-23 12:42:44 +00:00
Jenkins
71f92d4f3a Merge "tools: Remove 'with_venv'" 2017-02-23 12:42:17 +00:00
Jenkins
b6c32ade17 Merge "tools: Remove 'install_venv', 'install_venv_common'" 2017-02-23 12:41:45 +00:00
Jenkins
3cd849a162 Merge "tools: Remove 'clean-vlans'" 2017-02-23 12:41:19 +00:00
Jenkins
1b5cbe3c95 Merge "tools: Remove 'enable-pre-commit-hook'" 2017-02-23 12:37:32 +00:00
Andreas Jaeger
d60dffc6be Prepare for using standard python tests
Add simple script to setup mysql and postgresql databases, this script
can be run by users during testing and will be run by CI systems for
specific setup before running unit tests. This is exactly what is
currently done by OpenStack CI in project-config.

This allows to change in project-config the python-db jobs to
python-jobs since python-jobs will call this script initially.

See also
http://lists.openstack.org/pipermail/openstack-dev/2016-November/107784.html

Update devref for this.

Needed-By: Iea42a0525b2c5a5cdbf8604eb23a6e7b029f6b48
Change-Id: Ie9bae659077dbe299eea131572117036065bdccf
2017-02-09 17:52:55 +00:00
Huan Xie
062065ca9e XenAPI Use os-xenapi lib for nova
XenServer has released os-xenapi lib on pypi, this patch is to
use os-xenapi in nova project. In this patch, we only change
the usage of XenAPI to os-xenapi and fix unit tests. In the next
patch, we will delete those unused files.

Partially-Implements: blueprint add-os-xenapi-library

Depends-On: Ic327135b893a77672fd42af919f47f181e932773

Change-Id: I424dfcd507c4b7fbeea5245cc1c234dec01d5781
2017-01-10 17:26:23 -08:00
Sean Dague
95441ef896 move gate hooks to gate/
We prevent a lot of tests from getting run on tools/ changes given
that most of that is unrelated to running any tests. By having the
gate hooks in that directory it made for somewhat odd separation of
what is test sensitive and what is not.

This moves things to the gate/ top level directory, and puts a symlink
in place to handle project-config compatibility until that can be
updated.

Change-Id: Iec9e89f0380256c1ae8df2d19c547d67bbdebd65
2017-01-04 11:05:16 +00:00
Stephen Finucane
08009b3485 tools: Remove 'colorizer'
This was used by the 'run_tests.sh' script, which has long since been
removed. There appears to be something similar built into ostestr [1]
should people care about this (I suspect they don't).

[1] https://github.com/openstack/os-testr/blob/0.8.0/os_testr/utils/colorizer.py

Change-Id: I5634b3f75fa2b392a2f49a61b7b46a299bbcd2fd
2017-01-04 11:04:39 +00:00
Stephen Finucane
98b9001a7b tools: Remove 'with_venv'
Nothing appears to use this any more. No point in keeping it around.

Change-Id: I7150c78f55fd2eb639ed85c868d38461219fda12
2017-01-04 10:45:41 +00:00
Stephen Finucane
4559faffd9 tools: Remove 'install_venv', 'install_venv_common'
tox has long since usurped this as the preferred approach to development
and testing.

Change-Id: I4baa1688b88135b71e1e04b4aede6d6d0197504b
2017-01-04 10:45:41 +00:00
Stephen Finucane
4404ffa109 tools: Remove 'clean-vlans'
Whatever function this script served in the past, it has long outlived
its usefulness in the age of neutron.

Change-Id: I0e5b0e0a49ee26bddbfce456aa91ffe78f62f34d
2017-01-04 10:45:41 +00:00
Stephen Finucane
0fc1fc5655 tools: Remove 'enable-pre-commit-hook'
This relies on the 'run_tests.sh' script, which has long since been
removed. git-review will do this for us so there's not much point in
keeping this around now.

Change-Id: Ie3edfa1cefeea94c36cf5646fcfae6ad4b3d8d84
2017-01-04 10:45:36 +00:00
ChangBo Guo(gcb)
2738e25f7e Add Python 3.5 functional tests in tox.ini
Add Python 3.5 functional tests tox envlist, this is
needed by python 3.5 functional tests Jenkins job.

Change-Id: I1bba877fcc2c1f2140cbbba2bc6e6c20d7972615
2016-12-14 13:33:07 +08:00
Jenkins
fa4dc18e41 Merge "Add Apache 2.0 license to source file" 2016-10-10 12:38:13 +00:00
Jenkins
290c31beaa Merge "Make releasenotes reminder detect added and untracked notes" 2016-10-06 19:55:00 +00:00
Dan Smith
85bf71b2ae Make db archival return a meaningful result code
This helps with automating to know when it's done. Update our own
post-test-hook to use it.

Change-Id: I61c3b187edab3e0e612470735eecd2db59028413
2016-09-29 15:38:18 -07:00
Cao Xuan Hoang
9a5748de21 Add Apache 2.0 license to source file
As per OpenStack licensing guide lines [1]:
[H102 H103] Newly contributed Source Code should be licensed under
the Apache 2.0 license.
[H104] Files with no code shouldn't contain any license header nor
comments, and must be left completely empty.

[1] http://docs.openstack.org/developer/hacking/#openstack-licensing

Change-Id: I74d25d88ac285c456cbb8874deb000ed782c2192
2016-09-29 11:15:32 +07:00
melanie witt
7c3fd3c59d Make releasenotes reminder detect added and untracked notes
When 'tox -ereleasenotes' is run, the releasenotes are built from
the committed notes. We have a script that displays a reminder when
it detects uncommitted notes. It doesn't, however, detect newly
added notes or untracked notes.

This adds detection of added and untracked notes for displaying
the reminder message.

Change-Id: I5164f0089eae4a95bc3e68cd3b515ef23556b227
2016-09-28 21:52:28 +00:00
Dan Smith
6f988105ac Add a post-test-hook to run the archive command
The archive_deleted_rows command in nova-manage is often-broken and
not well tested by us. We can test it to some degree in functional tests,
but running it against a real database with real deleted stuff in it
is a good idea. This adds a post-test hook and runs the archive so that
after a full test run in the gate, we'll see the output. Later, we should
make a failed run of this fatal, but for now, just run it so we can see
how close we are to being able to gate on it.

Change-Id: I16b2e00eede6af455cb74ca4e6ca951d56fdbcbc
2016-09-28 13:28:55 +00:00
Luong Anh Tuan
b9610fb0a3 Remove locals() for formatting strings.
Following OpenStack Style Guidelines:
http://docs.openstack.org/developer/hacking/#dictionaries-lists

Using locals() for formmatting strings is not clear as using explicit dictionaries
and can hire errors during refactoring.

Change-Id: If3930d62fea166760161eaa3bdf85efca95be121
2016-08-22 17:32:38 +07:00
Matt Riedemann
1e0b2b5822 Fix busted release notes
A bad release note slipped through because the bash script
wasn't returing the actual results. This fixes the script
and the busted release note.

Change-Id: Id32557f34121fe82fac21d089e4117a72ff585e5
Closes-Bug: #1610312
2016-08-05 12:14:19 -04:00