Use TCP_NODELAY on outgoing connections

On a loopback device (e.g., when proxy-server and object-server are on
same node), PUTs in the range 64-200K may experience a delay due to the
effect of Nagel interacting with the loopback MTU of 64K.

This effect has been directly seen by Mark Seger and Rick Jones on a
proxy-server to object-server PUT. However, you could expect to see a
similar effect on replication via ssync if the object being replicated
is on a different drive on the same node.

A prior change [1] related to Nagel set TCP_NODELAY on responses. This change
sets it on all outgoing connections.

[1] I11f86df1f56fba1c6ab6084dc1f580c395f072dc

Change-Id: Ife8885a42b289a5eb4ac7e4698f8889858bc8b7e
Closes-bug: 1408622

swift/common/bufferedhttp.py

@@ -30,6 +30,7 @@ from swift import gettext_ as _
 from urllib import quote
 import logging
 import time
+import socket
 
 from eventlet.green.httplib import CONTINUE, HTTPConnection, HTTPMessage, \
     HTTPResponse, HTTPSConnection, _UNKNOWN
@@ -105,7 +106,9 @@ class BufferedHTTPConnection(HTTPConnection):
 
     def connect(self):
         self._connected_time = time.time()
-        return HTTPConnection.connect(self)
+        ret = HTTPConnection.connect(self)
+        self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
+        return ret
 
     def putrequest(self, method, url, skip_host=0, skip_accept_encoding=0):
         self._method = method

test/unit/common/test_bufferedhttp.py

@@ -15,6 +15,8 @@
 
 import unittest
 
+import socket
+
 from eventlet import spawn, Timeout, listen
 
 from swift.common import bufferedhttp
@@ -60,6 +62,8 @@ class TestBufferedHTTP(unittest.TestCase):
                             'x-header': 'value'},
                         query_string='omg&no=%7f')
                     conn.send('REQUEST\r\n')
+                    self.assertTrue(conn.sock.getsockopt(socket.IPPROTO_TCP,
+                                                         socket.TCP_NODELAY))
                     resp = conn.getresponse()
                     body = resp.read()
                     conn.close()