Add config option for neutron client retries

Nova can occasionally fail to carry out server actions which require
calls to neutron API if haproxy happens to close a connection after
idle time if an incoming request attempts to re-use the connection
while it is being torn down.

In order to be more resilient to this scenario, we can add a config
option for neutron client to retry requests, similar to our existing
CONF.cinder.http_retries and CONF.glance.num_retries options.

Because we create our neutron client [1] using a keystoneauth1 session
[2], we can set the 'connect_retries' keyword argument to let
keystoneauth1 handle connection retries.

Closes-Bug: #1866937

[1] 57459c3429/nova/network/neutron.py (L226-L237)
[2] https://docs.openstack.org/keystoneauth/latest/api/keystoneauth1.session.html#keystoneauth1.session.Session

Change-Id: Ifb3afb13aff7e103c2e80ade817d0e63b624604a
(cherry picked from commit 0e34ed9733)

nova/conf/neutron.py

@@ -93,6 +93,20 @@ Related options:
   all tunneled networks
 * ``[neutron_physnet_$PHYSNET] numa_nodes`` must be configured for each value
   of ``$PHYSNET`` specified by this option
+"""),
+    cfg.IntOpt('http_retries',
+               default=3,
+               min=0,
+               help="""
+Number of times neutronclient should retry on any failed http call.
+
+0 means connection is attempted only once. Setting it to any positive integer
+means that on failure connection is retried that many times e.g. setting it
+to 3 means total attempts to connect will be 4.
+
+Possible values:
+
+* Any integer value. 0 means connection is attempted only once
 """),
 ]
 

nova/network/neutronv2/api.py

@@ -173,7 +173,8 @@ def get_client(context, admin=False):
     session = _get_session()
     client_args = dict(session=session,
                        auth=auth_plugin,
-                       global_request_id=context.global_id)
+                       global_request_id=context.global_id,
+                       connect_retries=CONF.neutron.http_retries)
 
     # NOTE(efried): We build an adapter
     #               to pull conf options

nova/tests/unit/network/test_neutronv2.py

@@ -294,6 +294,15 @@ class TestNeutronClient(test.NoDBTestCase):
             exception.Unauthorized,
             client.list_networks)
 
+    def test_neutron_http_retries(self):
+        retries = 42
+        self.flags(http_retries=retries, group='neutron')
+        my_context = context.RequestContext('userid',
+                                            uuids.my_tenant,
+                                            auth_token='token')
+        cl = neutronapi.get_client(my_context)
+        self.assertEqual(retries, cl.httpclient.connect_retries)
+
 
 class TestNeutronv2Base(test.TestCase):
 

releasenotes/notes/neutron-connection-retries-c276010afe238abc.yaml

@@ -0,0 +1,12 @@
+---
+fixes:
+  - |
+    A new config option ``[neutron]http_retries`` is added which defaults to
+    3. It controls how many times to retry a Neutron API call in response to a
+    HTTP connection failure. An example scenario where it will help is when a
+    deployment is using HAProxy and connections get closed after idle time. If
+    an incoming request tries to re-use a connection that is simultaneously
+    being torn down, a HTTP connection failure will occur and previously Nova
+    would fail the entire request. With retries, Nova can be more resilient in
+    this scenario and continue the request if a retry succeeds. Refer to
+    https://launchpad.net/bugs/1866937 for more details.