788d3c4969
In past days there were discussions about various issues with memcached connections [1][2][3]. After investigation it looks like common root cause for above problems is keystonemiddleware. More precisely said the way how keystonemiddleware is caching tokens. Currently it's using some home-made CachePool with direct usage of memcached library, moreover it looks like its approach is not eventlet-safe. Discussion can be mainly found in [4]. Fortunately keystonemiddleware can use "advanced cache pool", which is oslo.cache's implementation and was added long time ago [5], but it is turned on only if memcache_use_advanced_pool=True. This patch is switching to more elaborated oslo.cache CachePool and adding deprecation warning about eventlet-unsafe variant of keystonemiddleware's memcache pool. How to reproduce ? with memcache_use_advanced_pool=False 1. Build clean ENV of openstack 2. Deploy core projects (keystone,glance,nova,placement...) 3. Run while true; do COMMAND FOR SERVICE; done - several bashes, in parallel (5-7) COMMAND FOR SERVICE: - openstack network list - openstack volume list - openstack server list - openstack image list 4. Check memcached connections (which will grow up): - ss | grep 11211 | wc -l every second How to fix and test it ? Repeat above, to fix: - with memcache_use_advanced_pool=True OR - apply this patch Compare measurements in graph. [1] https://bugs.launchpad.net/keystonemiddleware/+bug/1892852 [2] https://bugs.launchpad.net/oslo.cache/+bug/1888394 [3] https://bugs.launchpad.net/keystonemiddleware/+bug/1883659 [4] https://review.opendev.org/c/openstack/oslo.cache/+/742193 [5] https://review.opendev.org/c/openstack/keystonemiddleware/+/268664 Closes-Bug: #1883659 Closes-Bug: #1892852 Closes-Bug: #1888394 Change-Id: I0e96334b65a0bf369ebf1d88651d13feb8d2ecac |
||
---|---|---|
.. | ||
__init__.py | ||
_auth.py | ||
_base.py | ||
_cache.py | ||
_exceptions.py | ||
_identity.py | ||
_memcache_crypt.py | ||
_opts.py | ||
_request.py | ||
_user_plugin.py |