[CI] Use compatible version of heat-tempest-plugin

This is a combination of four patch to fix the gate:

1. Use compatible version of heat-tempest-plugin

heat-tempest-plugin dropped py38 support, hence it cannot be installed
from recent master, so we need to override the checkout to the version
that still supports py38 and compatible to the actual branch.

This is needed both for the grenade job and the functional job, because
these are using the heat-tempest-plugin.

2. Try archive path to download Fedora image

This is a temporal workaround to allow downloading Fedora 37 image
which was moved to the archive path.

3. Remove reference to devstack-gate

devstack-gate was deprecated in xena and is being retired now[1].

4. [stable-only] Cap setuptools <71.0.0

py39 jobs (on ubuntu-focal) started to fail due to recent virtualenv
release (20.26.4) on Yoga (which bundles setuptools), because we have
'packaging==21.3' in this branch that is not compatible with newer
setuptools [2].

setuptools is bundled in virtualenv, so it has to be capped via the
virtualenv package. tox also needed to be capped (<4) as gate uses
tox 3.28.0, but with capping virtualenv we pull in latest tox as well,
which would cause other errors.

[1] https://review.opendev.org/c/openstack/governance/+/919629
[2] https://github.com/pypa/setuptools/issues/4483

Changes:
  .zuul.yaml

NOTE(elod.illes): change in .zuul.yaml is to adapt the patch to the
current branch ('<series>-last' needs to be used).

Change-Id: I9b1702749976a2cea42a24130e5fec2931b75ce1
(cherry picked from commit a806b400cf)
(cherry picked from commit ae6225890e)
(cherry picked from commit 4fd9953a55)
(cherry picked from commit 81a8b93254)
(cherry picked from commit 665ccfc262)
(cherry picked from commit c18b344591)

.zuul.yaml

@@ -10,11 +10,11 @@
       - zuul: opendev.org/openstack/devstack
       - zuul: opendev.org/openstack/tempest
     required-projects:
-      - openstack/devstack-gate
       - openstack/barbican
       - openstack/heat
       - openstack/heat-templates
-      - openstack/heat-tempest-plugin
+      - name: opendev.org/openstack/heat-tempest-plugin
+        override-checkout: yoga-last
       - openstack/octavia
       - openstack/neutron
       - openstack/oslo.messaging
@@ -135,7 +135,8 @@
     parent: grenade-multinode
     required-projects:
       - opendev.org/openstack/heat
-      - opendev.org/openstack/heat-tempest-plugin
+      - name: opendev.org/openstack/heat-tempest-plugin
+        override-checkout: yoga-last
       - opendev.org/openstack/python-heatclient
     vars:
       grenade_devstack_localrc:

devstack/lib/heat

@@ -463,11 +463,14 @@ function configure_tempest_for_heat {
         source /etc/ci/mirror_info.sh
     fi
     HEAT_TEST_FEDORA_IMAGE_UPSTREAM=https://download.fedoraproject.org/pub/fedora/linux
+    HEAT_TEST_FEDORA_IMAGE_UPSTREAM_ARCHIVE=https://download.fedoraproject.org/pub/archive/fedora/linux/
     HEAT_TEST_FEDORA_IMAGE_PATH=releases/37/Cloud/x86_64/images/Fedora-Cloud-Base-37-1.7.x86_64.qcow2
-    if curl --output /dev/null --silent --head --fail "${NODEPOOL_FEDORA_MIRROR}/${HEAT_TEST_FEDORA_IMAGE_PATH}"; then
+    if curl --location --output /dev/null --silent --head --fail "${NODEPOOL_FEDORA_MIRROR}/${HEAT_TEST_FEDORA_IMAGE_PATH}"; then
         export HEAT_TEST_FEDORA_IMAGE="${NODEPOOL_FEDORA_MIRROR}/${HEAT_TEST_FEDORA_IMAGE_PATH}"
-    else
+    elif curl --location --output /dev/null --silent --head --fail "${HEAT_TEST_FEDORA_IMAGE_UPSTREAM}/${HEAT_TEST_FEDORA_IMAGE_PATH}"; then
         export HEAT_TEST_FEDORA_IMAGE="${HEAT_TEST_FEDORA_IMAGE_UPSTREAM}/${HEAT_TEST_FEDORA_IMAGE_PATH}"
+    else
+        export HEAT_TEST_FEDORA_IMAGE="${HEAT_TEST_FEDORA_IMAGE_UPSTREAM_ARCHIVE}/${HEAT_TEST_FEDORA_IMAGE_PATH}"
     fi
     TOKEN=$(openstack token issue -c id -f value)
     local image_exists=$( openstack image list | grep "Fedora-Cloud-Base-37-1.7.x86_64" )

tox.ini

@@ -3,6 +3,12 @@ envlist = py36,py37,py38,py39,pep8
 ignore_basepython_conflict = True
 minversion = 3.1.0
 skipsdist = True
+# Cap setuptools via virtualenv to prevent compatibility issue with yoga
+# branch's upper constraint of 'packaging' package (21.3).
+requires =
+  virtualenv<20.26.4
+  tox<4
+  setuptools<71.0.0
 
 [testenv]
 basepython = python3