infra-prod-bootstrap-bridge: run directly on bridge

In discussion of other changes, I realised that the bridge bootstrap
job is running via zuul/run-production-playbook.yaml.  This means it
uses the Ansible installed on bridge to run against itself -- which
isn't much of a bootstrap.

What should happen is that the bootstrap-bridge.yaml playbook, which
sets up ansible and keys on the bridge node, should run directly from
the executor against the bridge node.

To achieve this we reparent the job to opendev-infra-prod-setup-keys,
which sets up the executor to be able to log into the bridge node.  We
then add the host dynamically and run the bootstrap-bridge.yaml
playbook against it.

This is similar to the gate testing path; where bootstrap-bridge.yaml
is run from the exeuctor against the ephemeral bridge testing node
before the nested-Ansible is used.

The root key deployment is updated to use the nested Ansible directly,
so that it can read the variable from the on-host secrets.

Change-Id: Iebaeed5028050d890ab541818f405978afd60124

playbooks/bootstrap-bridge.yaml

@@ -23,6 +23,49 @@
         install_ansible_ara_name: '{{ bridge_ara_name | default("ara[server]") }}'
         install_ansible_ara_version: '{{ bridge_ara_version | default("latest") }}'
 
-    - name: Install root keys
-      include_role:
-        name: root-keys
+    # This is the key that bridge uses to log into remote hosts.
+    #
+    # For production, this root-key variable is kept with the others
+    # in the Ansible production secrets.  Thus we need to deploy via
+    # the local Ansible we just installed that will load these
+    # variables.  Remote hosts have trusted this from their bringup
+    # procedure.
+    #
+    # In testing, we have been called with "root_rsa_key" variable set
+    # with an ephemeral key.  In this case, we pass it in as a "-e"
+    # variable directly from the file written on disk.  The testing
+    # ephemeral nodes have been made to trust this by the multinode
+    # setup.
+    #
+    # NOTE(ianw) : Another option here is to keep the root key as a
+    # secret directly in Zuul, which could be written out directly
+    # here.  Maybe one day we will do something like this.
+    - name: Create root key variable when testing
+      when: root_rsa_key is defined
+      block:
+        - name: Create vars dict
+          set_fact:
+            _root_rsa_key_dict:
+              root_rsa_key: '{{ root_rsa_key }}'
+
+        - name: Save extra-vars
+          copy:
+            content: '{{ _root_rsa_key_dict | to_nice_json }}'
+            dest: '/home/zuul/root-rsa-key.json'
+
+    - name: Make ansible log directory
+      file:
+        path: '/var/log/ansible'
+        state: directory
+        owner: root
+        mode: 0755
+              
+    - name: Install root key
+      shell: >-
+          ansible-playbook -v ${ROOT_RSA_KEY} -i "localhost,"
+          /home/zuul/src/opendev.org/opendev/system-config/playbooks/zuul/run-production-bootstrap-bridge-add-rootkey.yaml
+            > /var/log/ansible/install-root-key.{{ lookup('pipe', 'date +%Y-%m-%dT%H:%M:%S') }}.log 2>&1
+      environment:
+        ROOT_RSA_KEY: '{{ "-e @/home/zuul/root-rsa-key.json" if root_rsa_key else "" }}'
+        ANSIBLE_ROLES_PATH: '/home/zuul/src/opendev.org/opendev/system-config/playbooks/roles'
+      no_log: true

playbooks/service-bridge.yaml

@@ -33,13 +33,6 @@
       include_role:
         name: rax-dns-backup
 
-    - name: Make ansible log directory
-      file:
-        path: '/var/log/ansible'
-        state: directory
-        owner: root
-        mode: 0755
-
     - name: Automated Zuul cluster reboots and updates
       # Note this is run via cron because a zuul job can't run this playbook
       # as the playbook relies on all jobs ending for graceful stops on the

playbooks/zuul/run-production-bootstrap-bridge-add-rootkey.yaml

@@ -0,0 +1,6 @@
+- hosts: localhost
+  connection: local
+  tasks:
+    - name: Install root keys
+      include_role:
+        name: root-keys

playbooks/zuul/run-production-bootstrap-bridge.yaml

@@ -0,0 +1,16 @@
+- hosts: localhost
+  tasks:
+    - name: Add bridge.o.o to inventory for playbook
+      add_host:
+        name: bridge.openstack.org
+        ansible_python_interpreter: python3
+        ansible_user: zuul
+        # Without setting ansible_host directly, mirror-workspace-git-repos
+        # gets sad because if delegate_to localhost and with add_host that
+        # ends up with ansible_host being localhost.
+        ansible_host: bridge.openstack.org
+        ansible_port: 22
+        # Port 19885 is firewalled
+        zuul_console_disabled: true
+
+- import_playbook: ../bootstrap-bridge.yaml

zuul.d/infra-prod.yaml

@@ -33,33 +33,27 @@
 
 - job:
     name: infra-prod-bootstrap-bridge
-    parent: infra-prod-playbook
+    parent: opendev-infra-prod-setup-keys
     description: |
         Configure the bastion host (bridge)
 
         This job does minimal configuration on the bastion host
         (bridge.openstack.org) to allow it to run system-config
-        playbooks against our production hosts.  It sets up Ansible on
-        the host.
+        playbooks against our production hosts.  It sets up Ansible
+        and root keys on the host.
 
         Note that this is separate to infra-prod-service-bridge;
         bridge in it's role as the bastion host actaully runs that
         against itself; it includes things not strictly needed to make
         the host able to deploy system-config.
-    vars:
-      playbook_name: bootstrap-bridge.yaml
+    run: playbooks/zuul/run-production-boostrap-bridge.yaml
     files:
-      - inventory/
-      - roles/
-      - install_modules.sh
-      - modules.env
       - playbooks/bootstrap-bridge.yaml
-      - playbooks/roles/pip3/
+      - playbooks/zuul/run-production-bootstrap-bridge.yaml
+      - playbooks/zuul/run-production-bootstrap-bridge-add-rootkey.yaml
       - playbooks/roles/install-ansible/
-      - playbooks/roles/logrotate/
       - playbooks/roles/root-keys/
       - inventory/service/host_vars/bridge.openstack.org.yaml
-      - playbooks/zuul/run-production-playbook.yaml
 
 - job:
     name: infra-prod-base