Fix amphora image build

DIB dropped Python 2 support in its version 3.0.0 major release and as
so we had to pin it to 2.30.0 in stable/stein and stable/train branches.
However, upstream get-pip.sh recently introduced a regression where it
does not work well with Python <= 3.5. Sadly, DIB 2.30.0 points to
master get-pip.sh [1] and DIB is branchless so the fix cannot be
backported and we are in a deadlock scenario where Python <= 3.5 amphora
images can't be built.

To work-around this issue on Stein and Train releases, we have to make a
compromise: either build amphora images using master DIB and Python
>=3.6 only or be stuck (unless someone else can come up with another
approach, all ears). Ussuri and newer releases are not impacted by this
issue as they are Python >=3.6 only releases.

Additionally, in the process of fixing Stein CI jobs we noticed that
octavia-v1-dsvm-scenario jobs were still running on Ubuntu Xenial
(Python 3.5!) but should have been on Bionic. The nodeset was updated to
Bionic-based but yet they continued to fail. For unknown reasons, Zuul
didn't collect CI log files so we couldn't troubleshoot the issue
further (infra team was pinged on IRC, no response). Given that Stein is
in extended maintenance and these jobs are neutron-lbaas (EOLd), we have
reasons to believe there is no one interested in maintaining these jobs
to the level worth keeping them as voting.

[1] https://review.opendev.org/c/openstack/diskimage-builder/+/772254

Change-Id: I13da9a49ace803e957dc52e4882b78141bc7bef3

devstack/plugin.sh

@@ -83,7 +83,20 @@ function build_octavia_worker_image {
         fi
         sudo mkdir -m755 ${dib_logs}
         sudo chown $STACK_USER ${dib_logs}
-        $OCTAVIA_DIR/diskimage-create/diskimage-create.sh -l ${dib_logs}/$(basename $OCTAVIA_AMP_IMAGE_FILE).log $octavia_dib_tracing_arg -o $OCTAVIA_AMP_IMAGE_FILE ${PARAM_OCTAVIA_AMP_BASE_OS:-} ${PARAM_OCTAVIA_AMP_DISTRIBUTION_RELEASE_ID:-} ${PARAM_OCTAVIA_AMP_IMAGE_SIZE:-}
+        # Build amphora image with master DIB in a Python 3 virtual environment
+        (
+            DIB_VENV_DIR=$(mktemp -d)
+            DIB_GIT_DIR=/tmp/dib-octavia
+            python3 -m venv $DIB_VENV_DIR
+            export USE_PYTHON3=True
+            export PATH=$DIB_VENV_DIR/bin:$PATH
+            if ! [ -d $DIB_GIT_DIR ]; then
+                git clone ${GITREPO["diskimage-builder"]} $DIB_GIT_DIR
+            fi
+            (cd $REQUIREMENTS_DIR && git show origin/master:upper-constraints.txt) | sed '/diskimage-builder/d' > $DIB_VENV_DIR/u-c.txt
+            pip install -c $DIB_VENV_DIR/u-c.txt $DIB_GIT_DIR
+            $OCTAVIA_DIR/diskimage-create/diskimage-create.sh -l ${dib_logs}/$(basename $OCTAVIA_AMP_IMAGE_FILE).log $octavia_dib_tracing_arg -o $OCTAVIA_AMP_IMAGE_FILE ${PARAM_OCTAVIA_AMP_BASE_OS:-} ${PARAM_OCTAVIA_AMP_DISTRIBUTION_RELEASE_ID:-} ${PARAM_OCTAVIA_AMP_IMAGE_SIZE:-}
+        )
     fi
     upload_image file://${OCTAVIA_AMP_IMAGE_FILE} $TOKEN
 

zuul.d/projects.yaml

@@ -40,8 +40,6 @@
               - ^doc/.*$
               - ^etc/.*$
               - ^releasenotes/.*$
-        - octavia-v1-dsvm-scenario
-        - octavia-v1-dsvm-py3x-scenario
         - octavia-v2-dsvm-noop-api
         - octavia-v2-dsvm-noop-py2-api
         - octavia-v2-dsvm-scenario:
@@ -83,6 +81,10 @@
               - ^octavia/tests/unit/.*$
               - ^releasenotes/.*$
             voting: false
+        - octavia-v1-dsvm-scenario:
+            voting: false
+        - octavia-v1-dsvm-py3x-scenario:
+            voting: false
     gate:
       queue: octavia
       jobs:
@@ -107,8 +109,6 @@
               - ^doc/.*$
               - ^etc/.*$
               - ^releasenotes/.*$
-        - octavia-v1-dsvm-scenario
-        - octavia-v1-dsvm-py3x-scenario
         - neutron-lbaasv2-dsvm-api
         - neutron-lbaasv2-dsvm-py3x-api
         - octavia-v2-dsvm-noop-api