Add config option for neutron client retries
Nova can occasionally fail to carry out server actions which require calls to neutron API if haproxy happens to close a connection after idle time if an incoming request attempts to re-use the connection while it is being torn down. In order to be more resilient to this scenario, we can add a config option for neutron client to retry requests, similar to our existing CONF.cinder.http_retries and CONF.glance.num_retries options. Because we create our neutron client [1] using a keystoneauth1 session [2], we can set the 'connect_retries' keyword argument to let keystoneauth1 handle connection retries. Closes-Bug: #1866937 [1]57459c3429/nova/network/neutron.py (L226-L237)
[2] https://docs.openstack.org/keystoneauth/latest/api/keystoneauth1.session.html#keystoneauth1.session.Session Change-Id: Ifb3afb13aff7e103c2e80ade817d0e63b624604a (cherry picked from commit0e34ed9733
)
This commit is contained in:
parent
dd1643035e
commit
71971c2062
|
@ -93,6 +93,20 @@ Related options:
|
||||||
all tunneled networks
|
all tunneled networks
|
||||||
* ``[neutron_physnet_$PHYSNET] numa_nodes`` must be configured for each value
|
* ``[neutron_physnet_$PHYSNET] numa_nodes`` must be configured for each value
|
||||||
of ``$PHYSNET`` specified by this option
|
of ``$PHYSNET`` specified by this option
|
||||||
|
"""),
|
||||||
|
cfg.IntOpt('http_retries',
|
||||||
|
default=3,
|
||||||
|
min=0,
|
||||||
|
help="""
|
||||||
|
Number of times neutronclient should retry on any failed http call.
|
||||||
|
|
||||||
|
0 means connection is attempted only once. Setting it to any positive integer
|
||||||
|
means that on failure connection is retried that many times e.g. setting it
|
||||||
|
to 3 means total attempts to connect will be 4.
|
||||||
|
|
||||||
|
Possible values:
|
||||||
|
|
||||||
|
* Any integer value. 0 means connection is attempted only once
|
||||||
"""),
|
"""),
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
|
@ -173,7 +173,8 @@ def get_client(context, admin=False):
|
||||||
session = _get_session()
|
session = _get_session()
|
||||||
client_args = dict(session=session,
|
client_args = dict(session=session,
|
||||||
auth=auth_plugin,
|
auth=auth_plugin,
|
||||||
global_request_id=context.global_id)
|
global_request_id=context.global_id,
|
||||||
|
connect_retries=CONF.neutron.http_retries)
|
||||||
|
|
||||||
# NOTE(efried): We build an adapter
|
# NOTE(efried): We build an adapter
|
||||||
# to pull conf options
|
# to pull conf options
|
||||||
|
|
|
@ -294,6 +294,15 @@ class TestNeutronClient(test.NoDBTestCase):
|
||||||
exception.Unauthorized,
|
exception.Unauthorized,
|
||||||
client.list_networks)
|
client.list_networks)
|
||||||
|
|
||||||
|
def test_neutron_http_retries(self):
|
||||||
|
retries = 42
|
||||||
|
self.flags(http_retries=retries, group='neutron')
|
||||||
|
my_context = context.RequestContext('userid',
|
||||||
|
uuids.my_tenant,
|
||||||
|
auth_token='token')
|
||||||
|
cl = neutronapi.get_client(my_context)
|
||||||
|
self.assertEqual(retries, cl.httpclient.connect_retries)
|
||||||
|
|
||||||
|
|
||||||
class TestNeutronv2Base(test.TestCase):
|
class TestNeutronv2Base(test.TestCase):
|
||||||
|
|
||||||
|
|
|
@ -0,0 +1,12 @@
|
||||||
|
---
|
||||||
|
fixes:
|
||||||
|
- |
|
||||||
|
A new config option ``[neutron]http_retries`` is added which defaults to
|
||||||
|
3. It controls how many times to retry a Neutron API call in response to a
|
||||||
|
HTTP connection failure. An example scenario where it will help is when a
|
||||||
|
deployment is using HAProxy and connections get closed after idle time. If
|
||||||
|
an incoming request tries to re-use a connection that is simultaneously
|
||||||
|
being torn down, a HTTP connection failure will occur and previously Nova
|
||||||
|
would fail the entire request. With retries, Nova can be more resilient in
|
||||||
|
this scenario and continue the request if a retry succeeds. Refer to
|
||||||
|
https://launchpad.net/bugs/1866937 for more details.
|
Loading…
Reference in New Issue