memcache: Prevent possible pool exhaustion

Normally, the _exception_occurred() handling in _get_conns() would
repopulate the pool -- but it won't if there's a MemecachePoolTimeout.
The assumption in _get_conns() is that we'll be waiting on the
super().get() when the timeout pops, but there's a chance you see
something like

   * caller starts a 1s MemecachePoolTimeout, calls get()
   * after 0.9s, super().get() returns (None, None), so we call
     create(), which blocks on connect()
   * the pool timeout lapses before the connect timeout

Previously, nothing would repopulate the pool when that happened.
Anecdotally, having it happen once seemed to increase the likelihood
that it would happen again (as more in-progress requests vie for fewer
available connections), which could completely deplete the connection
pool.

Now, catch the MemcachePoolError during create() and put back another
dead-connection sentinel before re-raising.

Change-Id: I601d92d8079ef75c4dd8af0eea8968bdad44c870

swift/common/memcached.py

@@ -149,10 +149,18 @@ class MemcacheConnPool(Pool):
 
     def get(self):
         fp, sock = super(MemcacheConnPool, self).get()
+        try:
             if fp is None:
                 # An error happened previously, so we need a new connection
                 fp, sock = self.create()
             return fp, sock
+        except MemcachePoolTimeout:
+            # This is the only place that knows an item was successfully taken
+            # from the pool, so it has to be responsible for repopulating it.
+            # Any other errors should get handled in _get_conns(); see the
+            # comment about timeouts during create() there.
+            self.put((None, None))
+            raise
 
 
 class MemcacheRing(object):