2666 Commits

Author SHA1 Message Date
Lukas Kranz
d8ec17cab0 Remove get_md5 parameter from stat module.
The get_md5 parameter was removed with ansible 9.
https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_9.html#id44

If it is being used the following error appears:
"Unsupported parameters for (stat) module: get_md5..."

Unrelated, but also blocking testing/merging of this change, the
Ansible version specs for older python versions is loosened
to allow installing older versions of Ansible on test nodes (like
focal) that have older pythons that are unsupported by newer Ansible.

Change-Id: I99dd4f16fde659d84eb3dfa191557b3d9508b0fb
2024-08-01 07:12:17 -07:00
Zuul
bd1d95ccdd Merge "Make prepare-workspace-git behavior more consistent" 2024-07-24 20:04:29 +00:00
e7d31aa59d Fix wheel_mirror for Debian
The default wheel_mirror(including major.minor) no longer works for Debian,
Fix it by using just major version like debian-11-x86_64. Similar was
fixed in [1] but missed fixing configure-mirrors.

[1] https://review.opendev.org/c/openstack/project-config/+/897545

Change-Id: I4194f18a06527d8af8922f3baf8766a7148e23fa
2024-07-24 10:31:19 +05:30
James E. Blair
51fe46231f Make prepare-workspace-git behavior more consistent
This runs the same commands on the git repo regardless of whether
it was cloned or if the role found it there already.  Since the
purpose of the role is to mirror the workspace repos from the
executor to the remote node, this will produce more consistent
behavior.

Note that anyone somehow relying on, say, the origin being set
outside of this role may encounter a behavior change.  It is expected
that anyone manipulating a repo that is also managed by this role
would perform those manipulations idempotently after running this
role.

Change-Id: I428bf2980a526919d5b154c585943be92d4c1cfa
2024-07-23 17:19:26 -07:00
James E. Blair
78276a58c5 Fix prepare-workspace-git operating on existing repos
Commit 8003cdc76ca177061b1a462d07efaff83e18491b causes problems
if the remote repo already exists (e.g., the worker node is static
and not ephemeral) because it unconditionally removes settings
which are only conditionally set if the workspace is newly cloned.

Fix that by remove the Ansible "creates" check from the task and
executing the set calls unconditionally (but also, recreate the
functionality of the create check for the cloning part of the
task, which is what we're really trying to avoid).

This will run a few extra command such as clearing the bare flag
and also resetting the origin remote.  That should be fine in
this role since we expect it to do whatever it takes to make the
remote repo the same as the local one.

Also, resync test-prepare-workspace-git.

Change-Id: Ife12992df9ce2b0ce199b3980a4baa255cb0f28a
2024-07-23 17:18:03 -07:00
Zuul
8c90dd472b Merge "prepare-workspace-git: Add allow deleting current branch" 2024-07-23 18:21:43 +00:00
bernhardbergpartner
8003cdc76c prepare-workspace-git: Add allow deleting current branch
In cases where the HEAD branch of a repository is deleted,
prepare-workspace-git will be unable to propagate that deletion from
the executor repos to a clone of a cached repo on a remote node.
Correct this by unsetting the receive.delyDeleteCurrent flag.  This was
correctly set in mirror-workspace-git, but did not make it into
prepare-workspace-git.

Change-Id: I159fb1f9f9cee873466f11be2f155bed6892472e
2024-07-17 14:43:08 +02:00
Aurelio Jargas
524b7e7b95 Add ensure-poetry role
Poetry (https://python-poetry.org) is not declared as a dependency for a
Python project, it must be available somehow in the system. This role
installs it if missing.

- Latest version is installed, unless `ensure_poetry_version` is
  informed.

- The installed executable path is set as the `poetry_executable` fact.

- The `/usr/local/bin/poetry` symlink can also be created if
  `ensure_poetry_global_symlink: true`.

This new role is basically a copy of the `ensure-nox` role with the
symlink creation snippet taken from the `ensure-tox` role.

The commit adding `ensure-nox` (77b1b24) has been taken as an example of
the necessary changes when adding a new role.

Change-Id: I5592d38d415a9d74055348406653b69f110541ae
2024-07-02 08:00:55 -07:00
Clark Boylan
ec242bac78 Switch fips fole testing to CentOS 9 Stream
CentOS 8 Stream is EOL and these jobs are no longer functional. We would
like to clean up the test nodes entirely as a result of this. Part of
this cleanup is either deleting existing jobs that rely on CentOS 8
Stream or converting them to run on a different platform. In this case
we update the fips role job to run on CentOS 9 Stream.

Change-Id: Ifb7d6deaf529e21c257590d63938026584bee7ae
2024-06-26 15:07:32 -07:00
Clark Boylan
a9a2f5ab50 Be more cautious initing mimetypes
We use mimetypes to set file mime types for upload to log server
locations. We also override yaml files mime type to text/plain when
doing so. If we then call mimetypes.init() again this overrides the
previous yaml update to text/plain. This wasn't a problem until
python3.12 but python3.12 (on Ubuntu Noble specifically) seems to import
test cases in a different order which results in the mimetypes.init()
call in generate_manifest.py overriding the yaml mimetype set by the log
upload test cases.

Simply check if mimetypes is already inited before we init it again
which should avoid the problem entirely.

Note that this is likely to only ever by a testing issues as typically
ansible wouldn't import all of this code together but the unittest
runner does.

Change-Id: Ifb9137ddd89713cad546129c462ad94315100940
2024-06-26 15:07:32 -07:00
Monty Taylor
7a58814cda Support .python-version files in ensure-python
We have support for installing python from pyenv, but it currently
requires setting the python version explicitly as an argument. If
the repo in question has a .python-version file, we shouldn't need to
require the user to provide that version a second time. Instead, we can
read from the file for the install step.

Change-Id: Ic4c2d3fc7f55169cec5211010fc3a9622fa324d1
2024-06-25 14:30:46 -07:00
Zuul
5663544e8d Merge "Add a job for publishing a site to netlify" 2024-06-11 13:45:21 +00:00
Clark Boylan
1d71eb349e Add nox and tox py312 jobs
Python 3.12 is a thing now and is present in a stable Linux distro
release (Ubuntu Noble). It seems reasonable for people to want to run
nox and tox against py312 targets. Let's make it easy for them and add
jobs to do that for them.

Change-Id: I9d644cfbe65b92207a5fe2ad6dd950093bda87dd
2024-06-07 10:40:53 +00:00
Clark Boylan
1f3567396e Update ansible versions used in unittesting
We were pinning ansible to 2.8 for unittesting but Zuul currently only
supports Ansible 8 and 9. Pin Ansible to 8 in order to better test what
we expect people are using in the wild. This will also enable the
testing of Zuul and Ansible and Zuul Jobs with newer python versions
like 3.12.

In order to run testing with Ansible 8 intsead of 2.8 we drop testing
against python2.7 and python3.8 as neither of them can install an
Ansible version this new.

Change-Id: Icd563def65dcfd40b174218cc4e2b94e0230c374
2024-06-07 10:40:33 +00:00
Dr. Jens Harbott
917e3b880d Drop outdated testing platforms
The package repos for centos-8-stream are empty now, so it can no longer
be deployed. Ubuntu Bionic is EOL and about to be deleted from the
opendev infrastructure, too.

Change-Id: I5e14957f2c866fc3e337ac4051692c68b450ceba
2024-06-07 11:42:27 +02:00
Monty Taylor
9858cb5848 Add a job for publishing a site to netlify
Netlify has support for publishing built sites via a CLI tool. Add
support for using that.

Change-Id: Ib47ac48a386e5e93f93455ef1202fc2af970b1c9
2024-06-03 11:58:34 -07:00
Jeremy Stanley
04f9d83059 Add ubuntu-noble testing
OpenDev has recently added ubuntu-noble nodes, so let's test our
standard library on that platform now.

Change-Id: I8d49696812e6b0c67723fba9242ea6a4f5cd5026
2024-05-23 10:54:39 -07:00
Clark Boylan
298354d2e3 Ombibus set of fixups for Ubuntu Noble
There are a number of updates we make for Ubuntu Noble in this commit.

1. Remove python2-dev from bindep for Noble as Noble appears to have no
   python2 runtime options.
2. Add libjpeg-dev to bindep for noble because Pillow doesn't build
   python3.12 wheels for Pillow<10 which we currently depend on. This
   means we need to build from source and that depends on libjpeg-dev.
3. We remove double bracket wrappers from ansible vars in ansible
   assertion blocks. Having them results in errors like:
     Conditional is marked as unsafe, and cannot be evaluated.
4. We update rust testing to explicitly install pkg-config before
   building python cryptography. This tool is required to build
   cryptography from source and is no longer being pulled in either
   by the base images or build-essential meta pacakge.
5. Add an Ubuntu-24.04 tasks file for the ensure-skopeo roles so that
   we try to install skopeo using distro packages or build from source
   and don't use Kubic which only has packages for old Ubuntu releases.

Change-Id: I388710ce40dc757ada4de819a9c3c59fc32fb07a
2024-05-23 10:54:36 -07:00
ricolin
bb9c303fab Fix collection import error
We need this specific fix [1] to target error:

from collections import Sequence, Mapping
ImportError: cannot import name 'Sequence' from 'collections'

[1] https://github.com/23andMe/Yamale/commit/
0ddd618c8ba7aa6df146a840b15a1285c476e10e

Change-Id: I04f3221624bdf88803b9e5f4a9fa3eabade756b1
2024-04-18 22:44:11 +08:00
Zuul
2220a7da8d Merge "Make prepare-workspace-git fail faster." 2024-03-26 09:04:24 +00:00
Radosław Piliszek
32edb51a7e Reenable crio jobs
We switch the crio buildset registry job to run on debian bookworm for
the same reason we switched the other buildset registry jobs:
compatibility between skopeo and docker.

Additionally, we fix the crio/minikube route by avoiding the use of
the removed repo. [1]
kubectl from minikube will be used as it should have always been.

[1] https://kubernetes.io/blog/2023/08/31/legacy-package-repository-deprecation/#can-i-continue-to-use-the-legacy-package-repositories

Change-Id: I25b1f3ba7ba34b0dc0043c659111ca8405b375da
2024-03-21 21:09:37 +00:00
Radosław Piliszek
bae449c42c Reenable buildset-registry jobs
As a followup to I4d05f9b187f9e40c3dcb2597e08c5bb50c261b17

We switch buildset-registry jobs to debian bookworm which has new enough
golang to build the latest skopeo version. Latest skopeo is used in
order to get api version negotiation behavior which is necessary for
talking to modern docker (version 25 or newer).

Change-Id: Ie673ef6724b0a40e3cfb2ba83e90d566e1f1837c
Co-Authored-By: Clark Boylan <cboylan@sapwetik.org>
2024-03-21 21:09:01 +00:00
Dr. Jens Harbott
90332e6439 Revert "Override DOCKER_MIN_API_VERSION for skopeo when installing docker"
This reverts *parts* of commit 9519fafd102017e54f4d543f25ca4398f1ec67b8.

Specifically, this reverts the main fix regarding DOCKER_MIN_API_VERSION.

Reason for revert: Docker 26.0.0 is out, which drops the old API

This patch additionally modifies the Zuul config in two ways:

* disables buildset-registry jobs because they are children of
  opendev-buildset-registry (a base-job in opendev) and thus
  do not pick up new changes - they are reenabled in a followup
* disables crio jobs because they have their repo seemingly broken

Change-Id: I4d05f9b187f9e40c3dcb2597e08c5bb50c261b17
Co-Authored-By: Radosław Piliszek <radek@piliszek.it>
2024-03-21 13:30:29 +00:00
Lukas Kranz
be72e626e8 Make prepare-workspace-git fail faster.
In case of aws spot instances, the node can drop the connection anytime.
Ansible loops do not fail when the first item fails, instead they execute all items before reporting the failure.
This leads to a big overhead, if we sync repositories, when the node has dropped the connection beforehand.
Therefore i used include_tasks, which fails for the first unsuccessful item.

Change-Id: Id6079a2cda10a825384d52d47750d9c05d323e00
2024-03-21 07:34:30 +01:00
Zuul
8374db870b Merge "prepare-workspace-git: Add ability to define synced pojects" 2024-03-19 14:59:28 +00:00
Zuul
703d0b1a70 Merge "Drop CentOS 7 test jobs" 2024-03-12 18:23:24 +00:00
Clark Boylan
a021a9bba2 Pin requests-oauthlib<1.4.0 under python2.7
While requests-oauthlib 1.4.0 reports compatibility with python2.7
evidence points to the contrary. Specifically:

      import requests_oauthlib as oauth
    File "/home/zuul/src/opendev.org/zuul/zuul-jobs/.tox/py27/lib/python2.7/site-packages/requests_oauthlib/__init__.py", line 5, in <module>
      from .oauth1_session import OAuth1Session
    File "/home/zuul/src/opendev.org/zuul/zuul-jobs/.tox/py27/lib/python2.7/site-packages/requests_oauthlib/oauth1_session.py", line 1, in <module>
      from urllib.parse import urlparse
  ImportError: No module named parse

The azure-storage-blob library depends on this oauthlib which is needed
for log upload role testing. We address that by pinning the lib under
python2.7.

Change-Id: Ia27f2df166ccb33dcc93334d884411670ac859f7
2024-03-12 10:19:51 -07:00
Clark Boylan
f2f5220561 Drop CentOS 7 test jobs
CentOS 7 is going EOL in the near future. Due to a lack of demand/need
within OpenDev and the upcoming EOL OpenDev has decided to drop CentOS 7
test nodes entirely. This will free up room for OpenDev to build and
mirror distro releases that are more relevant today. In preparation for
this removal we drop the zuul-jobs CentOS 7 testing.

Change-Id: Id49d28dd426ecbb3470e234d3b67875a6b097112
2024-03-08 08:20:27 -08:00
Lukas Kranz
5eca8feda9 prepare-workspace-git: Add ability to define synced pojects
The prepare_workspace_sync_required_projects_only variable allows
users to define which projects to sync to the node. This can prevent
syncing of unnecessary repositories. For some builds e.g. the
depends-on repositories dont need to be synced. The projects are
filtered based on the 'required' flag present in each zuul.project
entry and the required projects list also does not contain projects
which are present due to Depends-On or gate queue sequencing.
Having unnecessary repos in the workspace can for example also break
the analysis phase of bazel.

Change-Id: I3cc36cbfc60c81956caf5137da63973aeade4e21
Co-Authored-By: James E. Blair <jim@acmegating.com>
Co-Authored-By: Bernhard Berg <bernhard.berg@bearingpoint.com>
2024-02-28 10:56:48 -08:00
Clark Boylan
b81ef9e329 Drop debian buster testing
Debian buster is the old old release of Debian (it has been replaced by
bullseye and bookworm). Drop testing of this release so that OpenDev can
drop image builds and mirroring of content.

Change-Id: I7244c045d3346ff8c222ab5afc77dfbe05420cae
2024-02-23 08:48:40 -08:00
Clark Boylan
6e0af61097 Drop opensuse-15 jobs
The OpenDev team is planning to remove OpenSUSE LEAP 15 images as our
node builds and mirrors are for 15.2 which is ancient and no one is
currently working to modernize these test environments. On top of that
LEAP is apparently going away in the future and will be replaced with
another distro.

Change-Id: I7b35561f2577d0d2a3f951199e10c509bf7de421
2024-02-21 09:17:32 -08:00
James E. Blair
8074d20b8f Fix ensure-docker for Ansible 6
The systemd_service module is known as "systemd" in Ansible 6.  Use
the backwards compat format until Ansible 6 is removed from Zuul.

Change-Id: Ifff1cdfdf4d03426f289355673ee1472f0d37dd6
2024-02-16 06:55:05 -08:00
James E. Blair
509880073e Remove command.warn usage
This is no longer present in Ansible 9.

Removing these upsets ansible-lint, so those errors are ignored.

The base roles job has bitrotted on centos-7 and bionic due to
a bad voluptuous release used in an stestr test.  That is fixed in
this change as well.

Change-Id: I67886d5ad82ab590979f82bd102d6f974b9d4421
2024-02-14 15:17:52 -08:00
Clark Boylan
9519fafd10 Override DOCKER_MIN_API_VERSION for skopeo when installing docker
Skopeo hardcoded the docker api version for image manipulation to
version 1.22 of the api until very recently. Docker 25 sets 1.24 as a
minimum version by default introducing an incompatibility between the
tools. It isn't straightforward to install an updated skopeo everywhere
we need it (due to golang requirements). As a workaround we override
the min version to 1.22 when installing the docker daemon. This should
work until Docker 26 is released and removed the override option.

Note we also pin microk8s from latest/stable (which is currently
1.29/stable) to 1.28/stable to workaround
https://github.com/canonical/microk8s/issues/4361. This is necessary to
get the CI jobs for docker/registry/k8s testing working in order to land
this fixup.

Change-Id: I377ac84d532749eba578c4b32eb2ed6a5ce7a0c9
2024-02-14 10:42:47 -08:00
Clark Boylan
e3006463aa Remove openshift + zuul registry testing
The openshift testing relied on openshift 3.11 as this version of
openshift can be deployed locally without using baremetal management.
Unfortunately, docker images for this version of openshift are no longer
available leading to errors like:

  pod_workers.go:186] Error syncing pod b14b1b90-cb64-11ee-816f-bc764e2013fc ("openshift-service-cert-signer-operator-6d477f986b-qx6cs_openshift-core-operators(b14b1b90-cb64-11ee-816f-bc764e2013fc)"), skipping: failed to "StartContainer" for "operator" with ImagePullBackOff: "Back-off pulling image \"openshift/origin-service-serving-cert-signer:v3.11\""

When deploying this version of openshift. Since openshift 4 is not
generally installable in a similar manner we drop the testing. The
testing of the registry with other versions of kubernetes should give us
reasonable coverage.

Change-Id: I6ac87edee2123a5105f3b8138157faa7ecd94b0d
2024-02-14 10:42:12 -08:00
Zuul
df5756f765 Merge "Add zuul-tenant-conf-check role/job" 2024-02-13 16:09:34 +00:00
Zuul
d2998dfbcd Merge "Introduce LogJuicer roles" 2024-02-13 15:41:51 +00:00
James E. Blair
73bdf1f2df Add zuul-tenant-conf-check role/job
This performs static validation of Zuul tenant config files.

Change-Id: I5d439d6cfb963e55d07b2a0058de76f030fe47b3
2024-02-01 15:56:29 -08:00
Tristan Cacqueray
26db5b3b24 Introduce LogJuicer roles
This change adds new roles to run logjuicer in zuul jobs:
  https://github.com/logjuicer/logjuicer

Change-Id: I02824a18285a16c8f0be6bb96b5404aa0d601c16
2024-01-08 16:09:17 +00:00
Felix Edel
7761396303 mirror-workspace-git-repos: Retry on failure in git update task
We occasionally see the this task fail for the first element in the
zuul.projects list with a MODULE FAILURE and a return code of -13
(SIGPIPE) [1]. So far we couldn't identify the root cause, so try to
mitigate this issue by retrying on failure. This solution is similar to
the one used for the "Synchronize repos" task[2].

There is a bug report in Ansible that fits

Since it's only the first element in the loop that is failing while
subsequent elements are successful, we currently have two assumptions:

  1. As the task before is using a `delegate_to: localhost' [3],
     there might be a problem with Ansible when switching the connection
     from localhost to the remote host (node).
  2. Since the task before is using the same SSH connection [4] that is
     used by Ansible to push the git repository, there might be some
     "leftovers" on the connection that make the next task fail.
  3. There is also a bug report in Ansible [5] which might be causing
     that error.

[1]:
    {
        "ansible_loop_var": "zj_project",
        "changed": false,
        "failed": true,
        "module_stderr": "",
        "module_stdout": "",
        "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
        "rc": -13,
        "zj_project": {...}
    }

[2]: 3b3495e255/roles/mirror-workspace-git-repos/tasks/main.yaml (L32)
[3]: 3b3495e255/roles/mirror-workspace-git-repos/tasks/main.yaml (L25)
[4]: 3b3495e255/roles/mirror-workspace-git-repos/tasks/main.yaml (L16)
[5]: https://github.com/ansible/ansible/issues/81777

Change-Id: I0c4cb87bb076b9b40c9c446dbe5db437daff5897
2023-12-08 06:37:55 -08:00
Roman Kuznecov
6710f84a90 tox: Do not concat stdout and stderr in getting siblings
Several packages in calling "python setup.py --name ..." may return
warning message, e.g.:

  ...
  _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
  !!

          ********************************************************************************
          Requirements should be satisfied by a PEP 517 installer.
          If you are using pip, you can try `pip install --use-pep517`.
          ********************************************************************************

  !!
  dist.fetch_build_eggs(dist.setup_requires)
  WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
  <PACKAGE_NAME>

and then this huge log places into the package_name variable. But
script expects that package_name will contain only package name. Because of this situation could not find siblings packages and
install them.

Change-Id: I5bf9a19233c48d1260b5ab17d749bfc58a8ef2fa
2023-11-24 06:32:19 +00:00
Zuul
3b3495e255 Merge "Deprecate mirror-workspace-git-repos" 2023-09-26 23:24:36 +00:00
Lukas Kranz
ce2bea51d4 Deprecate mirror-workspace-git-repos
This change is preparation for https://review.opendev.org/c/zuul/zuul-jobs/+/887917

In the beginning, there was only prepare-workspace[0] which rsynced repos.

Then we added mirror-workspace-git[1] to make it more efficient by using git operation, but it required some openstack-specific code in project-config to work.

Then we added prepare-workspace-git[2] which completed the git-based sync solution by locating everything requried in zuul-jobs.  It used mirror-workspace-git by reference and added this TODO:

  # TODO(tobiash): we might want to deprecate the role mirror-workspace-git-repos
  # and move it here.

This change completes that TODO by moving the mirror-workspace-git-repos code into prepare-workspace-git and places the repo in a sensible and maintainable state with two simple and good options:

 * prepare-workspace (rsync)
 * prepare-workspace-git (git)

In the unlikely event anyone is still using mirror-workspace-git-repos standalone (OpenStack/OpenDev is not, and that solution was haphazard as described above) they would be well served by a notification that there is a better alternative which is what most of the community actually uses now.

[0] cfffd4431b8efc2f4df1999ecb89384a29c59238
[1] 348598e96aac742954fa326a15a4ef8fd7f71b8b
[2] 7cee7156bcde8bc396ac4b6581bf2cae02eea0e9

Change-Id: Ib80e0447d49363182fd0d4c4d4e269841bc3aa95
2023-09-25 14:51:26 -07:00
James E. Blair
2cf566f363 Unpin stestr and python-subunit in fetch-subunit-output test
It is unclear why stestr is pinned, so let's unpin to see if
latest works.

Change-Id: Ia3735fb7b6efc2d0f64b88b8c048db798fc7cb0b
2023-09-18 10:36:11 -07:00
James E. Blair
fbd9ba8144 Revert "Disable base role testing that runs code on localhost"
We can now run untrusted code on the executor, so we can re-enable
these tests.

This reverts commit 44d2187e7fcea0683c75db530458088b8371bb4e.

Change-Id: I65960af09a3cf3da1e780a96b6d13ddba3ce73d8
2023-09-18 10:36:11 -07:00
James E. Blair
10ffca9ba8 Pin python-subunit in fetch-subunit-output test
This test pins stestr (reason unclear) but does not pin python-subunit.
Python-subunit has made a release incompatible with that version of
stestr, so pin it.

Separately, it may be useful to see if we can unpin both.

Change-Id: Ia7cc45dc53ff0697e7ec84479c08c93e7d872a76
2023-09-18 10:36:11 -07:00
Zuul
f2208ed3c3 Merge "Remove the nox-py27 job" 2023-09-06 20:46:32 +00:00
Zuul
6ad6ee7a65 Merge "roles/ensure-python: Fix 'python_use_stow' option" 2023-09-05 14:52:59 +00:00
Zuul
5aca055b8b Merge "Drop Helm v2 support to fix v3 issue" 2023-09-04 14:07:26 +00:00
Clark Boylan
378e039dba Remove the nox-py27 job
We added nox somewhat recently and set it up to mimic existing tox jobs.
This meant adding py27 jobs. Since then (in OpenDev at least) only a
single project has used the py27 job: Bindep. Bindep is dropping support
for python2.7 as the need for it has come to an end. Additionally, nox
doesn't work with python2.7 out of the box due to a virtualenv
dependency that ends up being too new for python2.7 venv creation.
Rather than hack around that let's drop python2.7 job support.

Change-Id: I52c07b01ad173304c19b13a10927fdadf9d84170
2023-08-22 14:28:36 -07:00