[train/backport] Prevent nftables to interfere with tripleo firewall

If for some reason (we've hit this via
https://bugzilla.redhat.com/show_bug.cgi?id=1694723) /etc/nftables/*
rules get populated and the nftables service is started and enabled
(which it is by puppet)
we'll effectively end up having two separate firewalls: the iptables one
managed by puppet and the nftables one left in the hands of the rpm
defaults.

We need to make sure that nftables is not set up, since that will
effectively interfere with the puppet-firewall/tripleo-ansible firewall
modules.

To do so we empty /etc/sysconfig/nftables.conf (that way if this
code runs before the nftables rpm is installed it won't be
overwritten), then if the nftables.conf files has changed we flush
the nft rulesets and immediately reload the iptables services

Tested by deploying UC and OC on both rhel 8.1 and rhel8.2 and
then redeploying the UC and confirming that the nftables flush+
iptables service reload is not triggered on redeploy:
TASK [Prevent Nftables to set up any rules] ****************************
Thursday 26 March 2020  14:55:01 +0000 (0:00:00.127)       0:01:46.572 *
ok: [undercloud-0]

TASK [Flush Nftables rules when nftables.conf changed] *****************
Thursday 26 March 2020  14:55:02 +0000 (0:00:00.480)       0:01:47.052 *
skipping: [undercloud-0]

TASK [Restart iptables to restore firewall after flushing nftables] ****
Thursday 26 March 2020  14:55:02 +0000 (0:00:00.130)       0:01:47.183 *
skipping: [undercloud-0] => (item=iptables.service)
skipping: [undercloud-0] => (item=ip6tables.service)

NB: The cherry pick from master is not clean due to the tripleo-firewall
    moving from puppet to ansible in master

Closes-Bug: #1869166
Depends-On: https://review.opendev.org/715173
Change-Id: Ia4a2a58aada3b893fa23e04722f0a7d77e05a981

deployment/tripleo-firewall/tripleo-firewall-baremetal-puppet.yaml

@@ -68,16 +68,35 @@ outputs:
         include ::tripleo::firewall
 
       host_prep_tasks:
-        if:
-        - no_ctlplane
-        -
-          name: Ensure ctlplane subnet is set
-          fail:
-            msg: |
-              No CIDRs found in the ctlplane network tags.
-              Please refer to the documentation in order to
-              set the correct network tags in DeployedServerPortMap.
-        - null
+        list_concat:
+          - - name: Prevent Nftables to set up any rules
+              copy:
+                dest: /etc/sysconfig/nftables.conf
+                content: |
+                  # This file has been explicitely emptied and disabled by TripleO
+                  # so that nftables and iptables do not race each other
+              register: nftablesconf
+            - when: nftablesconf is changed
+              block:
+                - name: Flush Nftables rules when nftables.conf changed
+                  command: /usr/sbin/nft flush ruleset
+                - name: Restart iptables to restore firewall after flushing nftables
+                  systemd:
+                    state: reloaded
+                    name: "{{item}}"
+                  loop:
+                     - iptables.service
+                     - ip6tables.service
+          - if:
+            - no_ctlplane
+            - -
+                name: Ensure ctlplane subnet is set
+                fail:
+                msg: |
+                  No CIDRs found in the ctlplane network tags.
+                  Please refer to the documentation in order to
+                  set the correct network tags in DeployedServerPortMap.
+            - null
 
       deploy_steps_tasks:
         - when: step|int == 0