Browse Source

Pass `flush_on_reconnect` to memcache pooled backend

If a memcache server disappears and then reconnects when multiple memcache
servers are used (specific to the python-memcached based backends) it is
possible that the server will contain stale data. The default is now to
supply the ``flush_on_reconnect`` optional argument to the backend. This
means that when the service connects to a memcache server, it will flush
all cached data in the server. The pooled backend is more likely to
run into issues with this as it does not explicitly use a thread.local
for the client. The non-pooled backend was not touched, it is not
the recommended production use-case.

See the help from python-memcached:

    @param flush_on_reconnect: optional flag which prevents a
	scenario that can cause stale data to be read: If there's more
	than one memcached server and the connection to one is
	interrupted, keys that mapped to that server will get
	reassigned to another. If the first server comes back, those
	keys will map to it again. If it still has its data, get()s
	can read stale data that was overwritten on another
	server. This flag is off by default for backwards
	compatibility.

Change-Id: I3e335261f749ad065e8abe972f4ac476d334e6b3
closes-bug: #1819957
tags/1.34.0
Morgan Fainberg 2 months ago
parent
commit
1192f185a5
2 changed files with 45 additions and 1 deletions
  1. 21
    1
      oslo_cache/_memcache_pool.py
  2. 24
    0
      releasenotes/notes/bug-1819957-ccff6b0ec9d1cbf2.yaml

+ 21
- 1
oslo_cache/_memcache_pool.py View File

@@ -215,7 +215,27 @@ class MemcacheClientPool(ConnectionPool):
215 215
         self._hosts_deaduntil = [0] * len(urls)
216 216
 
217 217
     def _create_connection(self):
218
-        return _MemcacheClient(self.urls, **self._arguments)
218
+        # NOTE(morgan): Explicitly set flush_on_reconnect for pooled
219
+        # connections. This should ensure that stale data is never consumed
220
+        # from a server that pops in/out due to a network partition
221
+        # or disconnect.
222
+        #
223
+        # See the help from python-memcached:
224
+        #
225
+        # param flush_on_reconnect: optional flag which prevents a
226
+        #        scenario that can cause stale data to be read: If there's more
227
+        #        than one memcached server and the connection to one is
228
+        #        interrupted, keys that mapped to that server will get
229
+        #        reassigned to another. If the first server comes back, those
230
+        #        keys will map to it again. If it still has its data, get()s
231
+        #        can read stale data that was overwritten on another
232
+        #        server. This flag is off by default for backwards
233
+        #        compatibility.
234
+        #
235
+        # The normal non-pooled clients connect explicitly on each use and
236
+        # does not need the explicit flush_on_reconnect
237
+        return _MemcacheClient(self.urls, flush_on_reconnect=True,
238
+                               **self._arguments)
219 239
 
220 240
     def _destroy_connection(self, conn):
221 241
         conn.disconnect_all()

+ 24
- 0
releasenotes/notes/bug-1819957-ccff6b0ec9d1cbf2.yaml View File

@@ -0,0 +1,24 @@
1
+---
2
+fixes:
3
+  - |
4
+    [`bug 1819957 <https://bugs.launchpad.net/keystone/+bug/1819957>`_]
5
+    If a memcache server disappears and then reconnects when multiple memcache
6
+    servers are used (specific to the python-memcached based backends) it is
7
+    possible that the server will contain stale data. The default is now to
8
+    supply the ``flush_on_reconnect`` optional argument to the backend. This
9
+    means that when the service connects to a memcache server, it will flush
10
+    all cached data in the server. This change only impacts the pooled backend
11
+    as it is the most likely (with heavy use of greenlet) to be impacted
12
+    by the problem and is the recommended production configuration.
13
+
14
+    See the help from python-memcached:
15
+
16
+    @param flush_on_reconnect: optional flag which prevents a
17
+            scenario that can cause stale data to be read: If there's more
18
+            than one memcached server and the connection to one is
19
+            interrupted, keys that mapped to that server will get
20
+            reassigned to another. If the first server comes back, those
21
+            keys will map to it again. If it still has its data, get()s
22
+            can read stale data that was overwritten on another
23
+            server. This flag is off by default for backwards
24
+            compatibility.

Loading…
Cancel
Save