There are certain HA clustered services (e.g. galera) that don't
have the ability natively to reload their TLS certificate without
being restarted. If too many replicas are restarted concurrently
this might result in full service disruption.
To ensure service availability, provide a means to ensure that
only one service replica is restarted at a time in the cluster.
This works by using pacemaker's CIB to implement a cluster-wide
restart lock for a service. The lock has a TTL so it's guaranteed
to be eventually released without requiring complex contingency
cleanup in case of failures.
Tested locally by running the following:
1. force recreate certificate on all nodes at once for galera
(ipa-cert resubmit -i mysql), and verify that the resources
restart one after the other
2. create a lock manually in pacemaker, recreate certificate for
galera on all nodes, and verify that no resource is restarted
before the manually created lock expires.
3. create a lock manually, let it expires, recreate a certificate,
and verify that the resource is restarted appropriately and the
lock gets cleaned up from pacemaker once the restart finished.
Closes-Bug: #1885113
Change-Id: Ib2b62e33b34cf72edfdae6299cf432259bf960a2
0f54889408