From 20419bc6ed9ecb2861bb905ad0cb490823306d93 Mon Sep 17 00:00:00 2001 From: Morgan Fainberg Date: Tue, 15 Jul 2014 18:40:15 -0700 Subject: [PATCH] cache: Convert oslo.cache to use dogpile.cache Specification for converting the currently unused openstack.common.cache module to leverage dogpile.cache as the underlying caching library. Change-Id: Ifc214da8cbef188d16b8fdd925ded2e308a4d7d0 --- specs/kilo/oslo-cache-using-dogpile.rst | 282 ++++++++++++++++++++++++ 1 file changed, 282 insertions(+) create mode 100644 specs/kilo/oslo-cache-using-dogpile.rst diff --git a/specs/kilo/oslo-cache-using-dogpile.rst b/specs/kilo/oslo-cache-using-dogpile.rst new file mode 100644 index 0000000..c801771 --- /dev/null +++ b/specs/kilo/oslo-cache-using-dogpile.rst @@ -0,0 +1,282 @@ +.. + This template should be in ReSTructured text. For help with syntax, + see http://sphinx-doc.org/rest.html + + To test out your formatting, build the docs using tox, or see: + http://rst.ninjs.org + + The filename in the git repository should match the launchpad URL, + for example a URL of + https://blueprints.launchpad.net/oslo/+spec/awesome-thing should be + named awesome-thing.rst. + + Please wrap text at 79 columns. + + Please do not delete any of the sections in this + template. If you have nothing to say for a whole section, just write: None + + If you would like to provide a diagram with your spec, ascii diagrams are + required. http://asciiflow.com/ is a very nice tool to assist with making + ascii diagrams. The reason for this is that the tool used to review specs is + based purely on plain text. Plain text will allow review to proceed without + having to look at additional files which can not be viewed in gerrit. It + will also allow inline feedback on the diagram itself. + +======================================= +Oslo Cache Updated to use dogpile.cache +======================================= + +`proposed bp oslo-cache-using-dogpile +`_ + +Currently the various OpenStack projects implement caching (memoization, +key-value-store, etc) in a number of various ways. All of the mechanisms for +caching should be unified under a single Oslo library ``oslo.cache``. + +Problem description +=================== + +With the many implementations (and varying degree of flexibility) OpenStack +projects should standardize on a single library for caching (memoization, +key-value-store, etc) within the code base. Currently, Keystone has used +`dogpile.cache `_ very +successfully. With the flexible backend implementation, ``dogpile.cache`` is +a natural fit to replace the many different implementations of caching. + +The new ``oslo.cache`` library would replace the following: + +* Keystone Caching Layer + +* Keystone Key-Value-Store Implementation + +* Oslo MemoryCache Module + +* Caching within the Keystone Middleware + +* Marconi use of oslo.cache (incubator) module + +* (potentially) Swift Ring Memcache + + +This new module would also open the door for other projects to more easily +adopt memoization or other forms of caching. + +The ``oslo.cache`` module is mostly targeted to be an OpenStack friendly +wrapper for ``dogpile.cache``. This will provide a consistent way to cache and +memoize data within the OpenStack ecosystem. + + +Proposed change +=============== + +The current ``oslo-incubator`` +`oslo openstack.common.cache module +`_ +would be replaced with an implementation that leverages ``dogpile.cache`` +library instead of a custom-built system. + +This new implementation will provide a mechanism to handle key-value-store +(e.g. traditional Memcached) and memoization to begin with. + +The Base backends that will be supported are: + +* Memcached + + * BMemcached + + * Standard Memcached + + * Pylibmc + +* In-Memory (Python dict-based) + +* Redis + +* MongoDB - There is a MongoDB backend that was developed for Keystone, a + similar (based upon the code in Keystone) implementation will be submitted + as an enhancement to the ``dogpile.cache`` library if it is not already + supported at the time of implementation. If the backend is not accepted + by the ``dogpile.cache`` library, it will be maintained in the + to-be-determined method of maintaining OpenStack specific backend modules. + + +The ``oslo.cache`` module is intended to provide 3 distinct features above +and beyond raw use of ``dogpile.cache``: + +* OpenStack style configuration of ``dogpile.cache``. It is not expected that + ``dogpile.cache`` would accept patches that include ``oslo.config`` as the + basis. Use of ``oslo.config`` would not be appropriate in the case of + a more general-purpose library such as ``dogpile.cache``. + +* Updates to ``dogpile.cache``. Any updates that are needed to support the + OpenStack specific configuration will be submitted to the upstream library. + This may include enhancements to the decorators for memoization and the + key-value-store interfaces. These potential enhancements will be submitted + with the plan to use them as they become available (oslo.cache may have a + little extra conditional code added until the new features are released and + OpenStack accepts the minimum ``dogpile.cache`` version increase to the + global requirements), but it should not block development or use of the + ``oslo.cache`` module. + +* Backend maintenance. If the MongoDB backend is not accepted by the + ``dogpile.cache`` library a clear path for maintaining OpenStack specific + backends will be documented and laid out (either within ``oslo.cache`` module + or an associated module). + + +Each project that implements either it's own version of caching using +``dogpile.cache`` or other implementations of caching will be converted to use +the new ``oslo.cache`` module. + + +Alternatives +------------ + +This could be left to each project to implement their own caching or develop +an OpenStack specific set of modules. Neither of these options are optimal as +it requires specific knowledge of the various (or OpenStack specific) caching +system to implement a custom backend. With ``dogpile.cache`` it is possible to +implement a very simple backend and configure it for use with all OpenStack +services that make use of the ``oslo.cache`` module. + +Impact on Existing APIs +----------------------- + +Current APIs should remain unchanged (both in Oslo and in other projects) until +caching is implemented. Caching implementation may impact the APIs in some +regard (e.g. memoization requires proper invalidation of data). + +The current ``oslo.cache`` module will remain available but be marked as +deprecated. The new ``oslo.cache`` implementation will be provided side-by-side +for a period of time to allow time for any current consumers of ``oslo.cache`` +to change-over to the ``dogpile.cache`` based implementation. + +``oslo.cache`` will provide a fairly simple constructor to access/instantiate +the CacheRegion object(s) based upon the configuration built with +``oslo.cache``. There will also be an option to extract the configuration +dictionary that can be directly passed to ``dogpile.cache`` if a developer +chooses to do so. + +Security impact +--------------- + +No direct security implications. Use of the new caching module requires all +cached data to be properly invalidated (on change, etc). Stale data could +cause security related impact (and thus should be closely reviewed). + +Performance Impact +------------------ + +Ideally the performance impact of using caching should be only positive, +however maintenance of the cache coherency can have overhead when caching +is implemented. + +It is likely ``dogpile.cache`` will have a better performance profile than +the current ``MemoryCache`` module. + +Configuration Impact +-------------------- + +New options for caching will be added. To leverage caching the configuration +for a service using ``oslo.cache`` will need to have the values set. + +By default caching will be disabled (can be overrided by a project) to ensure +that memory leaking / improper caching / negative performance impact waiting on +non-existent external services will not impact an OpenStack project. + +Developer Impact +---------------- + +Developers will need to become familiar with ``dogpile.cache`` and how to +implement usage of memoization and/or key-value-store regions. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + Morgan Fainberg + +Other contributors: + Michael Bayer + Flavio Percoco + +Milestones +---------- + +Target Milestone for completion: + Kilo-2 + +Work Items +---------- + +* Convert ``oslo.cache`` module over to use ``dogpile.cache`` + +* Provide a clean / easy way to configure the cache region (in-memory + cache object that provides access to the key-value-store and memoization + decorators) + +* Ensure design allows for future expansion into cryptographic signing and/or + encryption of data stored within the key-value-store backend. + + +Incubation +========== + +Lifecycle will be incubation, and adoption in projects that leverage either +``dogpile.cache`` directly (Keystone) or the oslo.MemoryCache module. Once +the interfaces are stable (and clearly documented) it is expected this module +can quickly move to graduation. + +Adoption +-------- + +Keystone will be the primary (first target) to adopt the new module (replacing +the custom ``dogpile.cache`` implementation. + +The oslo.MemoryCache module will be updated to leverage the new ``oslo.cache`` +module. + +The direct use of the oslo.MemoryCache module will be deprecated in favor of +directly using ``oslo.cache``. + + +Library +------- + +The Library will graduate into a top-level ``oslo.cache`` library. + +Anticipated API Stabilization +----------------------------- + +I expect that this library should be able to stabilize within a single +development cycle. Adoption via incubator for Kilo and L, release as a library +in either K or L. + +Documentation Impact +==================== + +* Documentation on configuring the cache region will be required. + +* Developer documentation on implementing key-value-store and memoization + within an OpenStack project will be needed. + +Dependencies +============ + +All dependencies should already be in the global requirements. No external +blueprints should be needed. + +References +========== + + +.. note:: + + This work is licensed under a Creative Commons Attribution 3.0 + Unported License. + http://creativecommons.org/licenses/by/3.0/legalcode +