2863 Commits

Author SHA1 Message Date
Takashi Kajinami
3fbd05078f Use consistent credential for Redis and Redis Sentinel
We generally expects that users enabled authentication both in Redis
and Redis Sentinel at the same time, and it'd be more reasonable to
use the same credential by default. This behavior is more consistent
with the other libraries used in tooz or dogpile.cache, which are used
in multiple OpenStack services.

Change-Id: Ie8c96415f72bbd5f11f908bb9f96242d57b2bd4e
2024-02-05 00:26:18 +09:00
Takashi Kajinami
14444acab8 redis: Enable SSL for sentinel
Sentinel class supports ssl arguments since v4.0.0[1]. Setting these
are required to use SSL in Redis Sentinel.

[1] fea7b85dde

Related-Bug: #2052372
Change-Id: I2fdbb2cbd26ffd277b18adf45ebb7312ec6f0a24
2024-02-04 20:07:38 +09:00
Takashi Kajinami
78f3ef26a1 redis: Add username
Redis introduced ACL feature in 4.0.0, and this feature is supported by
redis-py since 3.4.0[1]. When ACL is enabled, authentication requires
username in addition to password.

[1] 8df8cd54d1

Closes-Bug: #2052371
Change-Id: Ie2de917603e752e45377edd81496cc46f5675c2c
2024-02-04 19:52:00 +09:00
Zuul
6aa9a431da Merge "Cleanup setup.py and requirements" 2024-01-29 12:46:04 +00:00
Zuul
b46a6c8b6d Merge "Delete the job from backend if it cannot be consumed" 2024-01-29 12:46:02 +00:00
Zuul
a0dc4553cb Merge "Bump hacking" 2024-01-29 10:50:08 +00:00
Takashi Kajinami
0000c79203 Bump hacking
hacking 3.0.x is too old.

Change-Id: Idbc349cbd348fb73a45c68c5f5b884c6a877ef4f
2024-01-28 16:04:41 +09:00
Zuul
05a14400c3 Merge "Replace deprecated perl-style regex" 2024-01-23 14:48:48 +00:00
Zuul
5ff3e51192 Merge "Avoid endless backtraces on StorageFailure" 2024-01-23 13:49:52 +00:00
Takashi Kajinami
60c01f2c9a Replace deprecated perl-style regex
Zuul deprecated usage of perl-style regex.

```
  All regular expressions must conform to RE2 syntax, but an
  expression using the deprecated Perl-style syntax has been detected.
  Adjust the configuration to conform to RE2 syntax.
```

Change-Id: I0dd4596cb0138e44122486e699b8f64714e8418a
2024-01-22 15:00:02 +09:00
Zuul
c56b292c9e Merge "Fix incorrect handling of storage exceptions" 2024-01-12 10:19:19 +00:00
Takashi Kajinami
0d5c948a74 Cleanup setup.py and requirements
- Python 2 is no longer supported
- setup.py is no longer managed by the global tooling
- Recent pip does not require appropriate order in requirement files

Change-Id: I8853a4221c6da78f860e821be4a063176c5734f3
2024-01-12 17:55:09 +09:00
Ghanshyam Mann
1c4af17623 Update python classifier in setup.cfg
As per the current release tested runtime, we test
python version from 3.8 to 3.11 so updating the
same in python classifier in setup.cfg

Change-Id: Ib88b74861cbd1260183f668b6cf616751f6b03e7
2024-01-11 16:37:06 -08:00
Takashi Kajinami
76d7ce091b coveragerc: Remove non-existent path
The taskflow/openstack directory does not exist.

Change-Id: If5efa6e190745cbfbb627f812fefb60c570a9893
2023-12-19 20:56:21 +09:00
Takashi Kajinami
78d8f7f741 Fix python shebang
The current shebang requires /usr/bin/python which is not available in
Ubuntu Jammy by default.

Change-Id: I8658c1e5c6c2c05df69865663a0fca12c12d4568
2023-10-17 16:04:41 +00:00
Gregory Thiemonge
981052a6cb Avoid endless backtraces on StorageFailure
If the storage is down (for instance sqlalchemy), the logbook cannot be
loaded, then taskflow goes into a endless loop of exceptions. To
mitigate this issue, use the retry feature of tenacity, the backtrace
will be displayed only every 5 seconds

Partial-Bug: #2037050

Change-Id: I656abdf7325c46d3afb2cc7ca905f1a335fb0d2f
2023-09-22 08:26:09 -04:00
Gregory Thiemonge
07a1a3f417 Fix incorrect handling of storage exceptions
Since [0] _save_flow_detail and _save_atom_detail use tenacious.retry,
it means that they no longer raise StorageException but a
tenacious.RetryError which is not caught by the callers.

Some jobs may have been incorrectly consumed (instead of being
rescheduled) when StorageException was raised.
Now tenacious reraises only the last exception (StorageException), which
is correctly handled.

[0] Ic6b0a78d20124cc027468ecc6aeff189c25d1a8a

Closes-Bug: #2037050
Change-Id: I030dd8fc5b65833243cf0948af53dc1aeabf41d9
2023-09-22 03:25:00 -04:00
Zuul
cd80fc96de Merge "bindep: Use new mysql-* package names" 2023-09-14 03:27:52 +00:00
Stephen Finucane
db27bef4a4 bindep: Use new mysql-* package names
Debian Bookworm switched from MySQL to MariaDB.

Change-Id: Iffacf1bb001eca68937debe8e688002348f4a818
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2023-09-08 12:56:37 +01:00
b329ec7ead Update master for stable/2023.2
Add file to the reno documentation build to show release notes for
stable/2023.2.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.2.

Sem-Ver: feature
Change-Id: I4c568a542aab8b4e45899124f30fbf7e9041749f
2023-09-07 09:39:30 +00:00
Tobias Urdin
3b40c04594 Update redis dep to >=3.0.0
This was not done in [1] when we removed the
StrictRedis alias usage.

[1] https://review.opendev.org/c/openstack/taskflow/+/859541

Change-Id: I8b7f02c090862ea117ba37da90ae6942e57184af
5.4.0
2023-08-16 15:57:37 +02:00
Stephen Finucane
4a29c44699 Add job to test with SQLAlchemy master (2.x)
Change-Id: I3486492c00b8294bfe67fa81a09d6f241f1a6057
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
5.3.0
2023-07-14 14:19:31 +01:00
Stephen Finucane
17a6c571dc db: Don't rely on autocommit behavior
Resolve the following RemovedIn20Warning warning:

  The current statement is being autocommitted using implicit
  autocommit, which will be removed in SQLAlchemy 2.0. Use the .begin()
  method of Engine or Connection in order to use an explicit transaction
  for DML and DDL statements.

There's only a single case of this, though we take the opportunity to
remove some unnecessary contextlib.closing calls: the Connection object
already supports being called as a context manager.

Change-Id: Ic8594773acbc3ff52469d6585d348076d13da67e
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2023-07-14 14:16:33 +01:00
Stephen Finucane
384e630e16 tests: Enable SQLAlchemy 2.0 deprecation warnings
Well, sort of. We enable them but immediately filter out the ones we're
actually seeing, the rationale being that we can address these in a
piecemeal fashion without the risk of introducing new issues.

There's a lot more to be done here. However, the work done in oslo.db
[1], nova [2], cinder [2] etc. should provide a guide for how to resolve
the outstanding issues.

[1] https://review.opendev.org/q/topic:sqlalchemy-20+project:openstack/oslo.db
[2] https://review.opendev.org/q/topic:sqlalchemy-20+project:openstack/nova
[3] https://review.opendev.org/q/topic:sqlalchemy-20+project:openstack/cinder

Change-Id: I5b11e63451b0e76969421fa8fe39ac2b920f90ab
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2023-07-14 14:12:05 +01:00
Stephen Finucane
171580c4d3 tests: Use valid IPv6 address
Python 3.11.4 [1] includes a fix for gh-103848 [2] and
urllib.parse.urlsplit will now validate that bracketed IP addresses are
valid IPv6 address. Fix this.

[1] https://docs.python.org/release/3.11.4/whatsnew/changelog.html#python-3-11-4
[2] https://github.com/python/cpython/issues/103848

Change-Id: Ibd3d24e07f0c5670224b3e186b329c207666a2ab
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2023-07-14 14:11:10 +01:00
Zuul
b18848b394 Merge "Upgrade development status" 2023-05-10 21:30:49 +00:00
Zuul
8fc904f5a0 Merge "Revert "Moves supported python runtimes from version 3.8 to 3.10"" 5.2.0 2023-05-05 13:43:14 +00:00
Ghanshyam
352cdb36f3 Revert "Moves supported python runtimes from version 3.8 to 3.10"
This reverts commit 8535f841487968be7105c280201ca275c6934333.

Keeping Python 3.10 in setup.cfg classifier. 

Reason for revert:

Needed-By: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/882175

TC has been discussing about re-adding the python 3.8
testing in current master 2023.2 release testing.

- https://meetings.opendev.org/meetings/tc/2023/tc.2023-04-25-18.00.log.html#l-191
- https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033469.html

While governance changes are under review, TC agreed to add py3.8 testing
so that we do not see more project/lib dropping python 3.8 and make them
uninstalable on python 3.8

- https://meetings.opendev.org/meetings/tc/2023/tc.2023-05-02-18.00.log.html#l-17
- https://review.opendev.org/c/openstack/governance/+/882165

Also adding py3.8 testing back in job https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/882175

Change-Id: Ie1d0a1ce0439ca5ffe3d3d377f3ac14a4fa6702f
2023-05-04 20:20:24 +00:00
Zuul
5c1b7e58dd Merge "Fix parsing of zookeeper jobboard backend options" 2023-04-19 00:11:52 +00:00
Zuul
314118bd61 Merge "Prepare taskflow for sqlalchemy2" 2023-04-13 19:28:27 +00:00
Zuul
d3dc83e523 Merge "Fix doc building with Sphinx 6.0" 2023-04-13 16:04:44 +00:00
Takashi Kajinami
8535f84148 Moves supported python runtimes from version 3.8 to 3.10
Within 2023.2 python version 3.9 and 3.10 are the
supported python runtimes [1].
[1] https: //review.opendev.org/c/openstack/governance/+/872232

Change-Id: I7f5323109ab03cdf3bd36fabee1bee37e7887679
2023-04-10 13:12:05 +09:00
Takashi Kajinami
aa2980f9d3 Upgrade development status
This library has been used by several OpenStack projects and we can
safely assume this is not mature and can be used for production.

Change-Id: I26540b2d887d2a5e9208a08dc3d2b4ce149924ac
2023-04-10 11:00:20 +09:00
Eric Harney
a2078f4e09 Fix doc building with Sphinx 6.0
Fix Sphinx 6.0 doc build error.

Change-Id: I9a1571cf416d32d040d8ff05d1c8b1769d1d0558
2023-03-21 11:28:33 -04:00
Gregory Thiemonge
77bc3cb7f8 Prepare taskflow for sqlalchemy2
* convert_unicode engine option is no longer supported
* use positional arguments for select, lists are no longer supported
* insert no longer accepts keyword constructor arguments
* explicitly convert Rows to dicts

Change-Id: I858d60b3d142e8ade8b2c0154872c8bfc19a42ba
2023-02-27 02:18:54 -05:00
dd73b125d1 Update master for stable/2023.1
Add file to the reno documentation build to show release notes for
stable/2023.1.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.1.

Sem-Ver: feature
Change-Id: I11c1dfdc64a0905df5e22c979579cdf5f55cd517
2023-02-24 15:26:38 +00:00
Zuul
325d45629d Merge "Replace abc.abstractproperty with property and abc.abstractmethod" 2023-02-17 15:08:24 +00:00
Zuul
e5e572ab8e Merge "Change StrictRedis usage to Redis" 2023-02-17 13:27:20 +00:00
Gregory Thiemonge
086b3e4335 Fix parsing of zookeeper jobboard backend options
Fix the zookeeper backend options when values are passed as strings,
a "False" string is now treated as the False boolean.

Closes-Bug: #1999174
Change-Id: I048faf06d89ebf980efe0598e647f2ec89f65ada
2023-01-12 08:59:29 +00:00
Gregory Thiemonge
5bcac4c7d4 Fix test_while_is_not with python 3.11
The test_while_is_not function relied on strings and literals to test
the 'while_is_not' function. But the while_is_not function uses the 'is'
operator to test the equivalency of 2 objects.
This triggers a bug with python 3.11 where using 'is' with literals is
not advised (it is not recommended since python 3.8 [0]).
The test now uses objects from a specific class to evaluate the
while_is_not function.

[0] https://docs.python.org/3/whatsnew/3.8.html#changes-in-python-behavior

Change-Id: I38a0135aaf73e25aa20a11c0685d5c2a7b587a07
5.1.0
2022-12-05 18:21:48 +01:00
Zuul
012b7c5aa9 Merge "Adapt to new jsonschema versions" 2022-11-21 10:31:01 +00:00
Tobias Urdin
4ff2d00178 Change StrictRedis usage to Redis
The StrictRedis class is only an alias for
Redis in >= 3.0.0

Change-Id: Ief27531f120a50805053c214cb61bb4151678d70
2022-09-29 08:37:05 +00:00
niuke
299c2afe14 remove unicode prefix from code
Change-Id: I59e8ae94eb83ae3a1ddcc99091e53686d2d55186
2022-08-24 15:17:18 +08:00
Corey Bryant
4bd0ab3327 Adapt to new jsonschema versions
This change provides fixes that were dectected by unit
test failures with new jsonschema (and py310).

The types argument has been removed in favor of providing a
type_checker to jsonschema.validators.extend:
https://github.com/python-jsonschema/jsonschema/issues/577

Closes-Bug: #1983412
Change-Id: I86f12b3d264320308e7f4841910fc21a6e8b3fa9
2022-08-03 15:16:02 -04:00
ljhuang
56413aa3c5 Replace abc.abstractproperty with property and abc.abstractmethod
Replace abc.abstractproperty with property and abc.abstractmethod,
as abc.abstractproperty has been deprecated since python3.3[1]

[1]https://docs.python.org/3.8/whatsnew/3.3.html?highlight=deprecated#abc

Change-Id: I1bcecd99d8856c26621a5304d9f7f01f8f111918
2022-08-03 20:37:13 +08:00
Zuul
3bb577e338 Merge "Fix formattiing of release list" 2022-06-21 16:29:28 +00:00
Zuul
6dfc18013a Merge "Quote string representations" 5.0.0 2022-06-13 16:32:31 +00:00
Michael Johnson
2521e3ee00 Quote string representations
networkx 2.8.3 will raise errors if certain characters, such as a colon, in the node names and the string is not quoted.
This patch double quotes flow, node, and task string representations to make sure there are not issues with these characters occuring in the names.

Story: 2010083
Change-Id: Ib0941cddf14dde5d6b9f97fe0224d6e6f3975226
2022-06-10 11:45:32 +02:00
Zuul
c04efe3d7f Merge "Remove six" 2022-06-10 01:20:55 +00:00
Hervé Beraud
acc630737f Fix formattiing of release list
Change-Id: Ia68da313c393ee506911b73bc7ff9bc37677d607
2022-05-27 14:06:31 +02:00