Add configurable delay to connection rety attempts with IPA

Presently, when novajoin fails to make a connection with the IPA
server, for any reason, it will immediately re-attempt to make
the connection when the backoff is unset (it is off by default).
As a result, any timing related issues could be the source of
the connection issues will likely result in no connection at all.

This change adds a new configuration option, retry_delay, which
will halt subsequent connection attempts for N seconds where N
is the retry_delay. By default this is set to 5 seconds, mirroring
internal ipalib behavior[1].

[1] - https://github.com/freeipa/freeipa/blob/master/ipalib/install/kinit.py#L29-L30

Change-Id: Iec96e4bd6643c0a657c8db424cc72deb10f170bd

README.rst

@@ -194,6 +194,8 @@ section.  It provides the following options:
 - domain: The domain to associate with IPA hosts.
 - connect_retries: The number of times to attempt to contact the IPA
           server before failing.
+- retry_delay: How many seconds to wait before retrying to make a connection
+          with the IPA server.
 - project_subdomain: Use the project the instance is created in as the
           subddomain for the fully-qualified domain name. For example if
           the project is admin and the domain is example.com and the

doc/source/configuration.rst

@@ -35,6 +35,11 @@ connect_retries
 ~~~~~~~~~~~~~~~
 The number of times to attempt to contact the IPA server before failing.
 
+retry_delay
+~~~~~~~~~~~
+How many seconds to wait before retrying to make a connection with the IPA
+server.
+
 service_credentials
 -------------------
 

novajoin/config.py

@@ -35,6 +35,9 @@ service_opts = [
     cfg.IntOpt('connect_retries', default=4,
                help='How many times to attempt to retry '
                'the connection to IPA before giving up'),
+    cfg.IntOpt('retry_delay', default=5,
+               help='How many seconds to wait before retrying '
+               'to make a connection with the IPA server'),
     cfg.IntOpt('connect_backoff', default=0,
                help='Initial number of seconds to backoff before '
                'retrying the connection to IPA.  The backoff '

novajoin/ipa.py

@@ -61,6 +61,7 @@ class IPANovaJoinBase(object):
             return
 
         self.ntries = CONF.connect_retries
+        self.retry_delay = CONF.retry_delay
         self.initial_backoff = CONF.connect_backoff
         self.ccache = "MEMORY:" + str(uuid.uuid4())
         LOG.debug("cache: %s", self.ccache)
@@ -183,6 +184,11 @@ class IPANovaJoinBase(object):
                 # successful connection
                 self.__reset_backoff(message_id)
                 return
+            if not self.backoff:
+                LOG.info("[%s] Waiting %s seconds before next retry.",
+                         message_id,
+                         self.retry_delay)
+                time.sleep(self.retry_delay)
 
         LOG.error("[%s] Failed to connect to IPA after %d attempts",
                   message_id, self.ntries)

releasenotes/notes/add-retry-delay-config-aa12130f9a032559.yaml

@@ -0,0 +1,8 @@
+---
+features:
+  - |
+    Adds configurable delay to connection retry attempts with IPA through new
+    config parameter, retry_delay (default: 5 seconds). Enabled by default
+    but ignored when connect_backoff is set, failed attempts to connect to the
+    IPA server will wait N seconds, where N is the retry_delay, before
+    attempting subsequent connections.