Fix queens gates, multiple fixes

1. Make grenade jobs experimental for EM branches

As discussed in ML thread[1], we are going to
make grenade jobs as non voting for all EM stable and
oldest stable. grenade jobs are failing not and it might take
time to fix those if we are able to fix. Once it jobs are
working depends on project team, they can bring them back to
voting or keep non-voting.

If those jobs are failing consistently and no one is fixing them
then removing those n-v jobs in future also fine.

Additionally, it was proposed in neutron CI meeting [2] that non-voting
jobs would be moved to experimental, so move grenade jobs there instead
of keeping them non-voting

[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015499.html
[2] http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-07-01-15.00.log.html#l-101

StableOnly

Conflicts:
	.zuul.yaml

(cherry picked from commit 9313dce459)

2. Install pip2 for functional, fullstack,
   neutron-tempest-iptables_hybrid, rally

Else both jobs fail with "sudo: pip: command not found"

3. Add ensure-tox for functional, fullstack,
   neutron-tempest-iptables_hybrid, rally

Similar error message for tox

4. Disable OVS compilation for fullstack and move job to experimental

Compilation fails similarly to recent master failures:
/opt/stack/new/ovs/datapath/linux/geneve.c:943:15: error: ‘const struct ipv6_stub’ has no member named ‘ipv6_dst_lookup’

But branch 2.9 is not updated anymore. Use official package

This triggers a few tests failures, so move it to experimental (instead
of marking non-voting), same as grenade jobs

Conflicts:
	.zuul.yaml
	neutron/tests/contrib/gate_hook.sh

Depends-On: https://review.opendev.org/737940/
Change-Id: Ie846a8cb481da65999b12f5547b407cc7bdc3138
(cherry picked from commit 57f527a748)

.zuul.yaml

@@ -12,16 +12,12 @@
     check:
       jobs:
         - neutron-functional
-        - neutron-fullstack
         - neutron-rally-neutron
         - neutron-tempest-dvr
         - neutron-tempest-linuxbridge
-        - neutron-grenade-multinode
-        - neutron-grenade-dvr-multinode
         - neutron-tempest-multinode-full
         - neutron-tempest-dvr-ha-multinode-full
         - neutron-tempest-ovsfw
-        - neutron-grenade
         - openstack-tox-cover
         - tempest-full:
             irrelevant-files:
@@ -39,16 +35,16 @@
         - neutron-functional
         - neutron-tempest-dvr
         - neutron-tempest-linuxbridge
-        - neutron-grenade-multinode
-        - neutron-grenade-dvr-multinode
-        - neutron-grenade
-
         - openstack-tox-cover
     post:
       jobs:
         - openstack-tox-cover
     experimental:
       jobs:
+        - neutron-fullstack
+        - neutron-grenade
+        - neutron-grenade-multinode
+        - neutron-grenade-dvr-multinode
         - legacy-grenade-dsvm-neutron-linuxbridge-multinode:
             irrelevant-files:
               - ^(test-|)requirements.txt$
@@ -137,6 +133,8 @@
     required-projects:
       - openstack/devstack-gate
       - openstack/neutron
+    vars:
+      ensure_pip_from_packages_with_python2: yes
     irrelevant-files:
       - ^.*\.rst$
       - ^doc/.*$
@@ -152,6 +150,8 @@
     required-projects:
       - openstack/devstack-gate
       - openstack/neutron
+    vars:
+      ensure_pip_from_packages_with_python2: yes
     irrelevant-files:
       - ^.*\.rst$
       - ^doc/.*$
@@ -188,6 +188,8 @@
       - openstack/tripleo-image-elements
       - openstack/watcher
       - openstack/zaqar-ui
+    vars:
+      ensure_pip_from_packages_with_python2: yes
     irrelevant-files:
       - ^.*\.rst$
       - ^doc/.*$
@@ -340,6 +342,8 @@
       - openstack/devstack-gate
       - openstack/neutron
       - openstack/tempest
+    vars:
+      ensure_pip_from_packages_with_python2: yes
     irrelevant-files:
       - ^(test-|)requirements.txt$
       - ^.*\.rst$

neutron/tests/contrib/gate_hook.sh

@@ -70,19 +70,7 @@ case $VENV in
 
     configure_host_for_func_testing
 
-    # Because of bug present in current Ubuntu Xenial kernel version
-    # we need a fix for VXLAN local tunneling.
-    if [[ "$VENV" =~ "dsvm-fullstack" ]]; then
-        # The OVS_BRANCH variable is used by git checkout. In the case below,
-        # we use openvswitch commit 175be4bf23a206b264719b5661707af186b31f32
-        # that contains a fix for usage of VXLAN tunnels on a single node
-        # (commit 741f47cf35df2bfc7811b2cff75c9bb8d05fd26f) and is compatible
-        # with kernel 4.4.0-145
-        # NOTE(slaweq): Replace with a release tag when one is available.
-        # See commit 138df3e563de9da0e5a4155b3534a69621495742 (on the ovs repo).
-        OVS_BRANCH="175be4bf23a206b264719b5661707af186b31f32"
-        compile_ovs_kernel_module
-    elif [[ "$VENV" =~ "dsvm-functional" ]]; then
+    if [[ "$VENV" =~ "dsvm-functional" ]]; then
         # NOTE(slaweq): there is some bug in keepalived
         # 1:1.2.24-1ubuntu0.16.04.1, and because of that we have to use older
         # version for tests as workaround. For details check

playbooks/legacy/neutron-fullstack/run.yaml

@@ -1,7 +1,11 @@
 - hosts: all
   name: Autoconverted job legacy-neutron-dsvm-fullstack from old job gate-neutron-dsvm-fullstack-ubuntu-xenial
   tasks:
-
+    - name: Ensure tox is installed
+      include_role:
+        name: ensure-tox
+      vars:
+        ensure_global_symlinks: True
     - name: Ensure legacy workspace directory
       file:
         path: '{{ ansible_user_dir }}/workspace'

playbooks/legacy/neutron-functional/run.yaml

@@ -1,7 +1,11 @@
 - hosts: all
   name: Autoconverted job legacy-neutron-dsvm-functional from old job gate-neutron-dsvm-functional-ubuntu-xenial-nv
   tasks:
-
+    - name: Ensure tox is installed
+      include_role:
+        name: ensure-tox
+      vars:
+        ensure_global_symlinks: True
     - name: Ensure legacy workspace directory
       file:
         path: '{{ ansible_user_dir }}/workspace'

playbooks/legacy/neutron-rally-neutron/run.yaml

@@ -1,7 +1,11 @@
 - hosts: all
   name: Autoconverted job legacy-rally-dsvm-neutron-neutron from old job gate-rally-dsvm-neutron-neutron-ubuntu-xenial
   tasks:
-
+    - name: Ensure tox is installed
+      include_role:
+        name: ensure-tox
+      vars:
+        ensure_global_symlinks: True
     - name: Ensure legacy workspace directory
       file:
         path: '{{ ansible_user_dir }}/workspace'

playbooks/legacy/neutron-tempest-ovsfw/run.yaml

@@ -1,7 +1,11 @@
 - hosts: all
   name: Autoconverted job legacy-tempest-dsvm-neutron-ovsfw from old job gate-tempest-dsvm-neutron-ovsfw-ubuntu-xenial-nv
   tasks:
-
+    - name: Ensure tox is installed
+      include_role:
+        name: ensure-tox
+      vars:
+        ensure_global_symlinks: True
     - name: Ensure legacy workspace directory
       file:
         path: '{{ ansible_user_dir }}/workspace'