Fix ansible-devel job for Ansible 2.10 changes

The Ansible devel branch has pulled in some major changes that has
broken our -devel testing job.

Firstly, installing from source checkout now installs the package
"ansible-base"; this means when we install ARA, which has a dependency
on just "ansible" it pulls in the old 2.9 release (which is what the
-devel test is currently testing with -- the reason for this change).

We could remove ARA, but we quite like it's reports for the nested
Ansible runs.  So make a dummy "ansible" 2.9 package and install that
to satisfy the dependency.

Secondly, Ansible devel has split out a lot of things into "community
modules".  To keep testing the -devel branch into the future, we need
to pull in the community modules for testing as well [1].

After some very useful discussion with jborean93 in #ansible I believe
the best way to do this is to clone the community projects into place
in the ansible configuration directory.  Longer term, we should make
Zuul check these out and use that, then we can speculatively test
changes too -- but for now just KISS.

[1] For reference, upstream bundles all this into the "Ansible
Community Distribution" or ACD, which is what you will get when you
download "ansible" from PyPi or similar.  But this job should be
pulling the bleeding edge of ansible and the community modules we use
-- that's what it's for.

Depends-On: https://review.opendev.org/747337
Change-Id: I781e275acb6af85f816ebcaf57a9825b50ca1196

playbooks/roles/install-ansible/tasks/install_ansible_collection.yaml

@@ -0,0 +1,10 @@
+- name: 'Ensure {{ item.namespace }}/{{ item.name }} directory'
+  file:
+    path: '/root/.ansible/collections/ansible_collections/{{ item.namespace }}/{{ item.name }}'
+    state: directory
+    mode: 0755
+
+- name: 'Clone  {{ item.namespace }}/{{ item.name }} collection'
+  git:
+    repo: '{{ item.repo }}'
+    dest: '/root/.ansible/collections/ansible_collections/{{ item.namespace }}/{{ item.name }}'

playbooks/roles/install-ansible/tasks/install_ansible_stub.yaml

@@ -0,0 +1,22 @@
+- name: Create build dir
+  tempfile:
+    state: directory
+    suffix: fake-ansible
+  register: _build_dir
+
+- name: Install fake setup.py
+  blockinfile:
+    create: yes
+    path: '{{ _build_dir.path }}/setup.py'
+    block: |
+      import setuptools
+
+      setuptools.setup(name="ansible",
+                 url="http://fake.com",
+                 maintainer="nobody@nobody.com",
+                 version="2.9.0",
+                 description="Fake ansible")
+
+- name: Install stub ansible
+  pip:
+    name: '{{ _build_dir.path }}'

playbooks/roles/install-ansible/tasks/main.yaml

@@ -1,11 +1,12 @@
-# If ansible_install_version is not defined it should be "latest"
+# If install_ansible_version is not defined (note; not *empty*) it
+# should be "latest"
 - name: Set ansible default version to latest
   set_fact:
     install_ansible_version: latest
   when: install_ansible_version is not defined
 
 # If a version is not explicitly set we want to make sure to
-# completely omit the version argument to pip, as it will be coming
+# completely omit the version argument to pip:, as it will be coming
 # from the long-form install_ansible_name variable.  Additionally, if
 # the version is the special value "latest", then we also want to omit
 # any version number, but also set the package state to "latest".
@@ -19,7 +20,46 @@
     _install_ansible_state: latest
   when: install_ansible_version == 'latest'
 
-- name: Ensure required build packages for non-wheel architectures
+# Since Ansible 2.10 (~2020-08) the devel core package is split into
+# "ansible-base".  The PyPi the package "ansible" has everything in it
+# (the so called Ansible Community Distribution) but we specifically
+# want to test against devel.  However, ARA still depends on the
+# "ansible" package -- but we like ARA, because we can see the nested
+# runs with it.  To to keep ARA but avoid it bringing in either an old
+# version of Ansible, or the ACD, install a dummy package.
+- name: Check if running devel branch
+  set_fact:
+    _install_ansible_from_devel: true
+  when: '"github.com/ansible/ansible" in install_ansible_name'
+
+- name: Setup Ansible stub for post 2.10 compat
+  include_tasks: install_ansible_stub.yaml
+  when: _install_ansible_from_devel is defined
+
+# From Ansible 2.10 >= most of the fun stuff is in collections.  Clone
+# our required collections here.  Note, in production, we use ACD
+# which bundles most of this.
+#
+# TODO(ianw): we should add these to zuul and link the speculative
+# copies into ansible, then we could test changes in the collections!
+- name: Install Ansible collections
+  include_tasks: install_ansible_collection.yaml
+  when: _install_ansible_from_devel is defined
+  loop:
+    - namespace: ansible
+      name: posix
+      repo: https://github.com/ansible-collections/ansible.posix
+    - namespace: community
+      name: general
+      repo: https://github.com/ansible-collections/community.general
+    - namespace: community
+      name: crypto
+      repo: https://github.com/ansible-collections/community.crypto
+    - namespace: ansible
+      name: netcommon
+      repo: https://github.com/ansible-collections/ansible.netcommon
+
+- name: Ensure required Ansible build packages for non-wheel architectures
   apt:
     update_cache: yes
     name:
@@ -34,6 +74,26 @@
     version: '{{ _install_ansible_version | default(omit) }}'
     state: '{{ _install_ansible_state | default(omit) }}'
 
+- name: Ansible version check
+  command: 'ansible-playbook --version'
+  register: _ansible_version_check
+
+- name: Sanity check Ansible version
+  debug:
+    msg: '{{ _ansible_version_check.stdout }}'
+
+- name: Ansible cmd version check
+  command: 'ansible --version'
+  register: _ansible_version_check
+
+- name: Sanity check Ansible version
+  debug:
+    msg: '{{ _ansible_version_check.stdout }}'
+
+- name: Set up the ARA callback
+  include_tasks: install_ara.yaml
+  when: install_ansible_ara_enable
+
 # For use by k8s_raw ansible module
 # - name: Install openshift client
 #   pip:
@@ -89,10 +149,6 @@
     group: root
     mode: 0775
 
-- name: Set up the ARA callback
-  include_tasks: install_ara.yaml
-  when: install_ansible_ara_enable
-
 - name: Copy ansible.cfg in to place
   template:
     src: ansible.cfg.j2