Retry previously failed deployments

Previously, when the server pre and post deployments were run, if one
failed, it would not be automatically retried since the deployed file
for the deployment exists under /var/lib/heat-config/deployed.

This could be overridden by passing -e force=true to ansible-playbook,
however for deployments that have previously failed, I don't think this
should be required, and we should just retry them automatically.

This new behavior more closely matches the traditional behavior with
Heat, where a stack update would automatically attempt to reapply and
deployment that was in a failed state.

Change-Id: I18df88297abe7be50364233ed8f2514de6d3d434
Closes-Bug: #1756921
(cherry picked from commit 7b359af52a)

tripleo_common/releasenotes/notes/automatically-retry-failed-deployments-baf0c701e6d1ad4a.yaml

@@ -0,0 +1,5 @@
+---
+fixes:
+  - When using config-download, previously failed server deployments will be
+    automatically retried on subsequent runs with ansible-playbook. Previously,
+    -e force=true would have to be passed to trigger this behavior.

tripleo_common/templates/deployments.yaml

@@ -8,12 +8,31 @@
     dest: "/var/lib/heat-config/tripleo-config-download/{{ item }}"
   become: true
 
+- name: "Check if deployed file exists for {{ item }}"
+  stat:
+    path: /var/lib/heat-config/deployed/{{ deployment_uuid }}.json
+  register: deployed_file_stat
+
+- name: "Check previous deployment rc for {{ item }}"
+  shell: |
+    exit $(jq .deploy_status_code /var/lib/heat-config/deployed/{{ deployment_uuid }}.notify.json)
+  register: previous_deployment_result
+  ignore_errors: false
+  when: deployed_file_stat.stat.exists
+
+- name: "Remove deployed file for {{ item }} when previous deployment failed"
+  file:
+    path: /var/lib/heat-config/deployed/{{ deployment_uuid }}.json
+    state: absent
+  become: true
+  when: deployed_file_stat.stat.exists and previous_deployment_result.rc not in (0, 2)
+
 - name: "Force remove deployed file for {{ item }}"
   file:
     path: /var/lib/heat-config/deployed/{{ deployment_uuid }}.json
     state: absent
   become: true
-  when: force | bool
+  when: (force | bool)
 
 - name: "Run deployment {{ item }}"
   shell: |