Correct fernet token sync race condition

The fernet token rotation is subject to a race condition when using
aggressive rotation in a high volume, high traffic, high capacity cloud.
This change addresses the potential race condition by converting our
fernet token sync method from rsync to scp and by sorting the fernet
keys in reverse version ordering. This will ensure that the key with
the highest index is always synchronized first and will ensure that
the underlying file structure of a given target node always remains
intact during a sync operation.

Related-Bug: 1816927
Change-Id: I9087d953f7dabe04a2ad19af6121dae71544e5b2
Signed-off-by: Kevin Carter <kevin@cloudnull.com>

templates/keystone-fernet-rotate.sh.j2

@@ -31,6 +31,11 @@ function autorotate {
     {% if inventory_hostname != host %}
 
     # Fernet sync job to "{{ host }}"
+    scp -o UserKnownHostsFile=/dev/null \
+        -o StrictHostKeyChecking=no \
+        $(ls -dtr {{ keystone_fernet_tokens_key_repository }}/* | sort -Vr) \
+        {{ keystone_system_user_name }}@{{ hostvars[host]['ansible_host'] }}:{{ keystone_fernet_tokens_key_repository }}/
+
     rsync -e 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' \
           -avz \
           --delete \