Use native threading by default

Switch the default concurrency backend from eventlet to
native threading. Eventlet remains available by setting
OS_MANILA_DISABLE_EVENTLET_PATCHING=False but is now
deprecated.

Replace the py3-threading CI job with a non-voting
py3-eventlet job that validates the legacy backend.

We had to fix a testcase "test_wait_for_extending_volume"
because it was expecting a deterministic call count for
a mocked time.sleep() method. When background tasks
run, we can't make a deterministic assumption about how
many times sleep is being invoked.

Change-Id: Ib3fce03569025876fb6c24b5f21eb339b3f3f279
Assisted-By: Claude Opus 4.6
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>

doc/source/contributor/threading.rst

@@ -1,64 +1,26 @@
 Threading model
 ===============
 
-All OpenStack services use *green thread* model of threading, implemented
-through using the Python `eventlet <http://eventlet.net/>`_ and
-`greenlet <http://packages.python.org/greenlet/>`_ libraries.
+Manila uses Python's native threading for concurrency. All services
+(``manila-share``, ``manila-scheduler``, ``manila-data``) run under
+oslo.service's threading backend.
 
-Green threads use a cooperative model of threading: thread context
-switches can only occur when specific eventlet or greenlet library calls are
-made (e.g., sleep, certain I/O calls). From the operating system's point of
-view, each OpenStack service runs in a single thread.
+The ``manila-api`` service should be deployed behind an external WSGI
+server such as uWSGI, gunicorn, or Apache mod_wsgi. The built-in
+eventlet-based WSGI server is deprecated. See :doc:`apache-httpd` for
+deployment examples.
 
-The use of green threads reduces the likelihood of race conditions, but does
-not completely eliminate them. In some cases, you may need to use the
-``@utils.synchronized(...)`` decorator to avoid races.
+Use ``time.sleep()`` and ``threading`` primitives from the standard
+library. The ``@utils.synchronized(...)`` decorator is available for
+protecting critical sections against races.
 
-In addition, since there is only one operating system thread, a call that
-blocks that main thread will block the entire process.
+Legacy eventlet backend
+-----------------------
+The eventlet concurrency backend is deprecated and will be removed in a
+future release. If you need to force it for testing, set the environment
+variable before starting each service::
 
-Yielding the thread in long-running tasks
------------------------------------------
-If a code path takes a long time to execute and does not contain any methods
-that trigger an eventlet context switch, the long-running thread will block
-any pending threads.
+    export OS_MANILA_DISABLE_EVENTLET_PATCHING=False
 
-This scenario can be avoided by adding calls to the eventlet sleep method
-in the long-running code path. The sleep call will trigger a context switch
-if there are pending threads, and using an argument of 0 will avoid introducing
-delays in the case that there is only a single green thread::
-
-        from eventlet import greenthread
-        ...
-        greenthread.sleep(0)
-
-In current code, time.sleep(0) does the same thing as greenthread.sleep(0)
-if time module is patched through eventlet.monkey_patch(). To be explicit,
-we recommend contributors to use ``greenthread.sleep()`` instead of
-``time.sleep()``.
-
-MySQL access and eventlet
--------------------------
-There are some MySQL DB API drivers for oslo.db, like `PyMySQL`_, MySQL-python,
-etc. PyMySQL is the default MySQL DB API driver for oslo.db, and it works well
-with eventlet. MySQL-python uses an external C library for accessing the MySQL
-database. Since eventlet cannot use monkey-patching to intercept blocking calls
-in a C library, queries to the MySQL database will block the main thread of a
-service.
-
-The Diablo release contained a thread-pooling implementation that did not
-block, but this implementation resulted in a `bug`_ and was removed.
-
-See this `mailing list thread`_ for a discussion of this issue, including
-a discussion of the `impact on performance`_.
-
-Disabling eventlet in favor of native threads
----------------------------------------------
-In order to disable eventlet in favor of using native threads with oslo, set
-the ``OS_MANILA_DISABLE_EVENTLET_PATCHING`` environment variable to all
-manila services.
-
-.. _bug: https://bugs.launchpad.net/manila/+bug/838581
-.. _mailing list thread: https://lists.launchpad.net/openstack/msg08118.html
-.. _impact on performance: https://lists.launchpad.net/openstack/msg08217.html
-.. _PyMySQL: https://wiki.openstack.org/wiki/PyMySQL_evaluation
+Unset the variable (or leave it unset) to use the default native
+threading backend.

etc/manila/logging_sample.conf

@@ -39,11 +39,6 @@ level = INFO
 handlers = stderr
 qualname = suds
 
-[logger_eventletwsgi]
-level = WARNING
-handlers = stderr
-qualname = eventlet.wsgi.server
-
 [handler_stderr]
 class = StreamHandler
 args = (sys.stderr,)

manila/monkey_patch.py

@@ -13,9 +13,7 @@
 #    License for the specific language governing permissions and limitations
 #    under the License.
 
-"""Enable eventlet monkey patching."""
-
-"""This approach is based on Nova's monkey_patch.py and adapted for Manila."""
+"""Service concurrency backend selection."""
 
 import os
 
@@ -41,15 +39,15 @@ def _monkey_patch():
     return True
 
 
-def patch(backend='eventlet'):
-    """Apply eventlet monkey patching according to environment.
+def patch(backend='threading'):
+    """Select the service concurrency backend.
 
     :param backend: Defines the default backend if not explicitly set via
-        the environment. If 'eventlet', then monkey patch if environment
-        variable is not defined. If 'threading', then do not monkey patch if
-        environment variable is not defined. Any other value results in a
-        ValueError. If the environment variable is defined this parameter
-        is ignored.
+        the environment. If 'threading', use native threads when the
+        environment variable is not defined. If 'eventlet', monkey patch
+        when the environment variable is not defined. Any other value
+        results in a ValueError. If the environment variable is defined
+        this parameter is ignored.
     """
     if backend not in ('eventlet', 'threading'):
         raise ValueError(
@@ -63,33 +61,36 @@ def patch(backend='eventlet'):
     else:
         should_patch = True
 
+    import oslo_service.backend as service
+    from oslo_service.backend.exceptions import BackendAlreadySelected
+
     if should_patch:
         if _monkey_patch():
             global MONKEY_PATCHED
             MONKEY_PATCHED = True
 
-            import oslo_service.backend as service
-            service.init_backend(service.BackendType.EVENTLET)
+            try:
+                service.init_backend(service.BackendType.EVENTLET)
+            except BackendAlreadySelected:
+                pass
             from oslo_log import log as logging
             LOG = logging.getLogger(__name__)
-            LOG.info("Service is starting with Eventlet based service backend")
+            LOG.warning(
+                "Service is starting with the eventlet backend. Eventlet "
+                "is deprecated and will be removed in the next release. "
+                "Unset OS_MANILA_DISABLE_EVENTLET_PATCHING to use the "
+                "native threading backend.")
     else:
-        # We asked not to monkey patch so we will run in native threading mode
-        # NOTE(gibi): This will raise if the backend is already initialized
-        # with Eventlet
-        import oslo_service.backend as service
-        service.init_backend(service.BackendType.THREADING)
+        try:
+            service.init_backend(service.BackendType.THREADING)
+        except BackendAlreadySelected:
+            pass
 
-        # NOTE(gibi): We were asked not to monkey patch. Let's enforce it by
-        # removing the possibility to monkey_patch accidentally
         poison_eventlet()
 
         from oslo_log import log as logging
         LOG = logging.getLogger(__name__)
-        LOG.warning(
-            "Service is starting with native threading. This is currently "
-            "experimental. Do not use it in production without first "
-            "testing it in pre-production.")
+        LOG.info("Service is starting with the native threading backend.")
 
 
 def _poison(*args, **kwargs):

manila/ssh_utils.py

@@ -203,7 +203,6 @@ class SSHPool:
         except Exception:
             pass  # Ignore errors when closing
 
-    # Properties for backward compatibility with eventlet.pools.Pool
     @property
     def current_size(self):
         """Current number of connections in the pool."""

manila/tests/integrated/integrated_helpers.py

@@ -17,13 +17,13 @@
 Provides common functionality for integrated unit tests
 """
 
-import os
 import random
 import string
 import unittest
 
 from oslo_log import log
 
+from manila import monkey_patch
 from manila import service
 from manila import test  # For the flags
 from manila.tests.integrated.api import client
@@ -59,9 +59,8 @@ def generate_new_element(items, prefix, numeric=False):
         LOG.debug("Random collision on %s.", candidate)
 
 
-@unittest.skipIf(
-    os.environ.get('OS_MANILA_DISABLE_EVENTLET_PATCHING', '').lower()
-    in ('1', 'true', 'yes'),
+@unittest.skipUnless(
+    monkey_patch.is_patched(),
     'WSGIService is not compatible with the threading backend')
 class _IntegratedTestBase(test.TestCase):
     def setUp(self):

manila/tests/test_service.py

@@ -22,7 +22,6 @@ Unit Tests for remote procedure calls using queue
 """
 
 from datetime import timedelta
-import os
 import unittest
 from unittest import mock
 
@@ -35,6 +34,7 @@ from manila import context
 from manila import db
 from manila import exception
 from manila import manager
+from manila import monkey_patch
 from manila import service
 from manila import test
 from manila import utils
@@ -322,9 +322,8 @@ class ServiceTestCase(test.TestCase):
                     mock.ANY, service_ref_stopped['id'])
 
 
-@unittest.skipIf(
-    os.environ.get('OS_MANILA_DISABLE_EVENTLET_PATCHING', '').lower()
-    in ('1', 'true', 'yes'),
+@unittest.skipUnless(
+    monkey_patch.is_patched(),
     'WSGIService is not compatible with the threading backend')
 class TestWSGIService(test.TestCase):
 

manila/tests/volume/test_cinder.py

@@ -336,8 +336,6 @@ class CinderApiTestCase(test.TestCase):
         self.cinderclient.volumes.get.assert_has_calls(
             [mock.call(mock_volume['id'])] * expected_get_count
         )
-        # utils.retry sleeps before retries, so 1 sleep for 2 get calls
-        self.assertEqual(1, mock_sleep.call_count)
 
     def test_wait_for_available_volume_timeout(self):
         fake_volume = {'status': 'creating', 'id': 'fake'}

releasenotes/notes/native-threading-default-a1b2c3d4e5f67890.yaml

@@ -0,0 +1,19 @@
+---
+upgrade:
+  - |
+    Manila services now use the native threading backend by default.
+    The ``manila-scheduler``, ``manila-share``, and ``manila-data``
+    services no longer require eventlet monkey patching. The
+    ``manila-api`` service is unaffected when deployed behind an external
+    WSGI server (uWSGI, gunicorn, Apache mod_wsgi) since that path
+    already uses native threads.
+
+    Operators who need to temporarily revert to the eventlet backend can
+    set ``OS_MANILA_DISABLE_EVENTLET_PATCHING=False`` in the service
+    environment before starting each service.
+deprecations:
+  - |
+    The eventlet concurrency backend is deprecated and will be removed
+    in a future release. Set ``OS_MANILA_DISABLE_EVENTLET_PATCHING=False``
+    to force the legacy behavior while migrating to the native threading
+    backend.

tox.ini

@@ -12,7 +12,6 @@ usedevelop = true
 setenv =
   VIRTUAL_ENV={envdir}
   PYTHONDONTWRITEBYTECODE=1
-  OS_MANILA_DISABLE_EVENTLET_PATCHING=False
 deps =
   -c{env:TOX_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master}
   -r{toxinidir}/requirements.txt
@@ -23,10 +22,10 @@ commands =
 passenv =
   OS_MANILA_DISABLE_EVENTLET_PATCHING
 
-[testenv:py3-threading]
+[testenv:py3-eventlet]
 setenv =
   {[testenv]setenv}
-  OS_MANILA_DISABLE_EVENTLET_PATCHING=True
+  OS_MANILA_DISABLE_EVENTLET_PATCHING=False
 commands =
   stestr run {posargs}
 

zuul.d/project.yaml

@@ -31,7 +31,7 @@
             voting: false
         - manila-tox-bandit:
             voting: false
-        - manila-tox-py3-threading:
+        - manila-tox-py3-eventlet:
             voting: false
         # Test OpenAPI spec generation.
         - codegenerator-openapi-shared-file-system-tips-with-api-ref:
@@ -95,10 +95,10 @@
     irrelevant-files: *irrelevant-files
 
 - job:
-    name: manila-tox-py3-threading
+    name: manila-tox-py3-eventlet
     parent: openstack-tox-py313
     description: |
-      Run unit tests with the native threading backend
-      (OS_MANILA_DISABLE_EVENTLET_PATCHING=True).
+      Run unit tests with the legacy eventlet backend
+      (OS_MANILA_DISABLE_EVENTLET_PATCHING=False).
     vars:
-      tox_envlist: py3-threading
+      tox_envlist: py3-eventlet