Enforce firewall rules before pacemaker-auth

We want to make sure that any firewall rule set to open pacemaker ports
is executed before we run any commands that invoke pcs to
authenticate remote nodes.

It simply makes sense from a high-level POV to explicitely open
up firewall rules before we invoke pcs commands that will talk to
remote nodes.

I have actually seen one case in the wild where during a scaleup
the node being scaled up was waiting on Exec['wait-for-settle']
and the bootstrap node failed to contact pcs on the scaled up node
because there the firewall rules were never opened up as it was
waiting on the 'wait-for-settle' step.

Note that we *cannot* impose the ordering via a too-generic
Firewall<||> collector because in tripleo::firewall we have

    Service<||> -> Class['tripleo::firewall::post']

and we would create a circular dependency.

Tested a queens deploy with this change and we are correctly
guaranteed to open up firewalling before invoking pcs:
Mar 05 16:22:51 controller-0. puppet-user[18840]: (/Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[swift_storage]/Tripleo::Firewall::Rule[123 swift storage]/Firewall[123 swift storage ipv4]/ensure) created
Mar 05 16:22:52 controller-0. puppet-user[18840]: (/Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[swift_storage]/Tripleo::Firewall::Rule[123 swift storage]/Firewall[123 swift storage ipv6]/ensure) created
Mar 05 16:22:52 controller-0. puppet-user[18840]: (/Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[tripleo_firewall]/Tripleo::Firewall::Rule[003 accept ssh from any]/Firewall[003 accept ssh from any ipv4]/ensure) created
Mar 05 16:22:52 controller-0. puppet-user[18840]: (/Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[tripleo_firewall]/Tripleo::Firewall::Rule[003 accept ssh from any]/Firewall[003 accept ssh from any ipv6]/ensure) created
Mar 05 16:22:52 controller-0. puppet-user[18840]: (Exec[reauthenticate-across-all-nodes](provider=posix)) Executing '/sbin/pcs cluster auth controller-0 controller-1 controller-2 database-0 database-1 database-2 messaging-0 messaging-1 messag
ing-2 -u hacluster -p foobar --force'
Mar 05 16:22:52 controller-0. puppet-user[18840]: Executing: '/sbin/pcs cluster auth controller-0 controller-1 controller-2 database-0 database-1 database-2 messaging-0 messaging-1 messaging-2 -u hacluster -p AQtEeE6e3FDEqrfm --force'
Mar 05 16:22:55 controller-0. puppet-user[18840]: (Exec[Create Cluster tripleo_cluster](provider=posix)) Executing '/sbin/pcs cluster setup --wait --name tripleo_cluster controller-0 controller-1 controller-2 database-0 database-1 database-2
messaging-0 messaging-1 messaging-2 --token 10000 --encryption 1'
Mar 05 16:22:55 controller-0. puppet-user[18840]: Executing: '/sbin/pcs cluster setup --wait --name tripleo_cluster controller-0 controller-1 controller-2 database-0 database-1 database-2 messaging-0 messaging-1 messaging-2 --token 10000 --en
cryption 1'
Mar 05 16:23:20 controller-0. puppet-user[18840]: (Exec[Start Cluster tripleo_cluster](provider=posix)) Executing check '/sbin/pcs status >/dev/null 2>&1'
Mar 05 16:23:20 controller-0. puppet-user[18840]: Executing: '/sbin/pcs status >/dev/null 2>&1'
Mar 05 16:23:21 controller-0. puppet-user[18840]: (Exec[Start Cluster tripleo_cluster](provider=posix)) Executing '/sbin/pcs cluster start --all'
Mar 05 16:23:21 controller-0. puppet-user[18840]: Executing: '/sbin/pcs cluster start --all'

Change-Id: I775ad1abf87368d013054e9a5dab22931f21f86c
Closes-Bug: #1866209
(cherry picked from commit 88e119d747)

manifests/profile/base/pacemaker.pp

@@ -163,6 +163,10 @@ class tripleo::profile::base::pacemaker (
       $pacemaker_node_ips_real = []
     }
 
+    # (bandini) We want to make sure that any rule that opens up services takes place
+    # before we invoke pcs commands (see LP#1866209)
+    Firewall<|tag == 'tripleo-firewall-rule'|> -> Exec <|tag == 'pacemaker-auth'|>
+
     if $encryption {
       $cluster_setup_extras = merge($cluster_setup_extras_pre, {'--encryption' => '1'})
     } else {