Add a meta log upload role with a failover mechanism

This adds a new role "upload-logs-failover" which allows to upload logs
to different platform specific targets (s3, swift, ...). It loops
through a configured list of upload targets until one of the uploads
succeeded.

Change-Id: I8daff4105a8de70c4f84db9927a85df63da7d274

doc/source/log-roles.rst

@@ -14,6 +14,7 @@ Log Roles
 .. zuul:autorole:: set-zuul-log-path-fact
 .. zuul:autorole:: upload-logs
 .. zuul:autorole:: upload-logs-azure
+.. zuul:autorole:: upload-logs-failover
 .. zuul:autorole:: upload-logs-gcs
 .. zuul:autorole:: upload-logs-s3
 .. zuul:autorole:: upload-logs-swift

roles/upload-logs-failover/README.rst

@@ -0,0 +1,54 @@
+Upload logs to swift/s3/azure/... with a failover mechanism.
+
+This role calls the ``upload-logs-[target]`` roles according to the list of
+upload targets passed as role vars. If the current upload target fails, it
+proceeds with the next item in its list until one of the uploads succeeded or
+no more targets are available.
+
+It expects the corresponding role vars of ``upload-logs-[target]`` to be
+configured in the list of upload targets ``zuul_log_targets``. An example
+configuration would look like the following and would ideally be passed as
+a Zuul secret as it may hold passwords and access keys.
+
+.. code-block:: yaml
+
+    zuul_log_targets:
+      - target: s3
+        args:
+          upload_logs_s3_endpoint: https://objectstore.example.com/
+          zuul_log_bucket: log-bucket
+          zuul_log_aws_access_key: foo
+          zuul_log_aws_secret_key: bar
+      - target: swift
+        args:
+          zuul_log_cloud_config:
+            auth:
+              username: foo
+              password: bar
+              user_domain_name: example
+              auth_url: https://openstack.example.com:5000/v3/
+          zuul_log_container: log-container
+
+Requirements and dependencies of called ``upload-logs-[target]`` roles apply
+accordingly, e.g. ``boto3`` for the s3 role. Please refer to their
+corresponding documentation.
+
+**Role Variables**
+
+.. zuul:rolevar:: zuul_log_targets
+   :type: list
+
+   List of upload targets
+
+.. zuul:rolevar:: zuul_log_targets.target
+   :type: string
+
+   Choice of "azure", "gcs", "s3", "swift", and any other platform specific
+   "upload-logs-\*" role that might be availble.
+
+.. zuul:rolevar:: zuul_log_targets.args
+   :type: dict
+
+   Complex argument which holds the variables required by the respective
+   "upload-logs-\*" role specified by the "target" var. Please refer to their
+   respective documentation.

roles/upload-logs-failover/tasks/main.yaml

@@ -0,0 +1,15 @@
+- name: Initialize control vars
+  set_fact:
+    upload_succeeded: false
+
+- name: Perform log upload
+  include_tasks: upload.yaml
+  loop: "{{ zuul_log_targets }}"
+  loop_control:
+    loop_var: zj_upload_item
+  no_log: true
+
+- name: Fail if no successful upload
+  fail:
+    msg: "Upload to all targets failed"
+  when: not upload_succeeded

roles/upload-logs-failover/tasks/upload.yaml

@@ -0,0 +1,35 @@
+- name: Upload if not succeeded yet
+  when: not upload_succeeded
+  block:
+    # NOTE: ideally, we could set these vars in the below 'include_role' like
+    #         include_role:
+    #           ...
+    #         vars: "{{ zj_upload_item.args }}"
+    #       but this does not work because of a known bug in Ansible, cf.
+    #       https://github.com/ansible/ansible/blob/stable-2.9/test/integration/targets/include_import/role/test_include_role.yml#L96
+    - name: "Unpack vars for {{ zj_upload_item.target }} upload"
+      set_fact: { "{{ zj_args.key }}": "{{ zj_args.value }}" }
+      loop: "{{ zj_upload_item.args | dict2items }}"
+      loop_control:
+        loop_var: zj_args
+      # The facts set here usually contain credentials
+      no_log: true
+      # ansible lint complains about "var-naming Task uses 'set_fact' to define
+      # variables that violates variable naming standards"
+      tags:
+        - skip_ansible_lint
+
+    - name: "Upload logs to {{ zj_upload_item.target }}"
+      include_role:
+        name: "upload-logs-{{ zj_upload_item.target }}"
+    # we expect any necessary role vars to be passed via the current
+    # target.args here
+
+    - name: Mark upload successful
+      set_fact:
+        upload_succeeded: true
+
+  rescue:
+    - name: Mark upload failed
+      set_fact:
+        upload_succeeded: false