Document native threading mode and tuneables

Change-Id: I003177de3a9f69c71c19eb8eaa7232785e03e669
Signed-off-by: Balazs Gibizer <gibi@redhat.com>

doc/source/admin/concurrency.rst

@@ -0,0 +1,106 @@
+Nova service concurrency
+========================
+
+For a long time nova services relied almost exclusively on the Eventlet library
+for processing multiple API requests, RPC requests and other tasks that needed
+concurrency. Since Eventlet is not expected to support the next major cPython
+version the OpenStack TC set a `goal`__ to replace Eventlet and therefore Nova
+has started transitioning its concurrency model to native threads. During this
+transition Nova maintains the Eventlet based concurrency mode while building
+up support for the native threading mode.
+
+.. __: https://governance.openstack.org/tc/goals/selected/remove-eventlet.html
+
+.. note::
+
+   The native threading mode is not ready yet. Do not use it in production.
+
+Selecting concurrency mode for a service
+----------------------------------------
+
+Nova still uses Eventlet by default, but allows switching services to native
+threading mode at service startup via setting the environment variable
+``OS_NOVA_DISABLE_EVENTLET_PATCHING=true``.
+
+.. note::
+
+   Since nova 32.0.0 (2025.2 Flamingo) the nova-scheduler can be switched to
+   native threading mode.
+
+
+Tunables for the native threading mode
+--------------------------------------
+As native threads are more expensive resources than greenthreads Nova provides
+a set of configuration options to allow fine tuning the deployment based on
+load and resource constraints. The default values are selected to support a
+basic, small deployment without consuming substantially  more memory resources,
+than the legacy Eventlet mode. Increasing the size of the below thread pools
+means that the given service will consume more memory but will also allow more
+tasks to be executed concurrently.
+
+* :oslo.config:option:`cell_worker_thread_pool_size`: Used to execute tasks
+  across all the cells within the deployment.
+
+  E.g. To generate the result of the ``openstack server list`` CLI command, the
+  nova-api service will use one native thread for each cell to load the nova
+  instances from the related cell database.
+
+  So if the deployment has many cells then the size of this pool probably needs
+  to be increased.
+
+  This option is only relevant for nova-api, nova-metadata, nova-scheduler, and
+  nova-conductor as these are the services doing cross cell operations.
+
+* :oslo.config:option:`executor_thread_pool_size`: Used to handle incoming RPC
+  requests. Services with many more inbound requests will need larger pools.
+  For example, a single conductor serves requests from many computes as well
+  as the scheduler. A compute node only serves requests from the API for
+  lifecycle operations and other computes during migrations.
+
+  This option is only relevant for nova-scheduler, nova-conductor, and
+  nova-compute as these are the services acting as RPC servers.
+
+* :oslo.config:option:`default_thread_pool_size`: Used by various concurrent
+  tasks in the service that are not categorized into the above pools.
+
+  This option is relevant to every nova service using ``nova.utils.spawn()``.
+
+Seeing the usage of the pools
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When new work is submitted to any of these pools in both concurrency modes
+Nova logs the statistics of the pool (work executed, threads available,
+work queued, etc).
+This can be useful when fine tuning of the pool size is needed.
+The parameter :oslo.config:option:`thread_pool_statistic_period` defines how
+frequently such logging happens from a specific pool in seconds. A value of
+60 seconds means that stats will be logged from a pool maximum once every
+60 seconds. The value 0 means that logging happens every time work is submitted
+to the pool. The default value is -1 meaning that the stats logging is
+disabled.
+
+Preventing hanging threads
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Threads from a pool are not cancellable once they are executing a task,
+therefore it is important to ensure external dependencies cannot hold up a
+task execution indefinitely as that will lead to having fewer threads in the
+pool available for incoming work and therefore reduced overall capacity.
+
+Nova's RPC interface already uses proper timeout handling to avoid hanging
+threads. But adding timeout handling to the Nova's database interface is
+database server and database client library dependent.
+
+For mysql-server the `max_execution_time`__ configuration option can be used
+to limit the execution time of a database query on the server side. Similar
+options exist for other database servers.
+
+.. __: https://dev.mysql.com/doc/refman/8.4/en/server-system-variables.html#sysvar_max_execution_time
+
+For the pymysql database client a client side timeout can be implemented by
+adding the `read_timeout`__ connection parameter to the connection string.
+
+.. __: https://pymysql.readthedocs.io/en/latest/modules/connections.html#module-pymysql.connections
+
+We recommend using both in deployments where Nova services are running in
+native threading mode.

doc/source/admin/index.rst

@@ -105,6 +105,9 @@ the defaults from the :doc:`install guide </install/index>` will be sufficient.
 * :doc:`Running nova-api on wsgi </user/wsgi>`: Considerations for using a real
   WSGI container instead of the baked-in eventlet web server.
 
+* :doc:`Nova service concurrency </admin/concurrency>`: Considerations on how
+  to use and tune Nova services in threading mode.
+
 .. toctree::
    :maxdepth: 2
 
@@ -113,6 +116,7 @@ the defaults from the :doc:`install guide </install/index>` will be sufficient.
    default-ports
    availability-zones
    configuration/index
+   concurrency
 
 
 Basic configuration

doc/source/reference/threading.rst

@@ -1,8 +1,11 @@
 Threading model
 ===============
 
-All OpenStack services use *green thread* model of threading, implemented
+Eventlet
-through using the Python `eventlet <http://eventlet.net/>`_ and
+--------
+Before the Flamingo release all OpenStack services used the *green thread*
+model of threading, implemented through using the Python
+`eventlet <http://eventlet.net/>`_ and
 `greenlet <http://packages.python.org/greenlet/>`_ libraries.
 
 Green threads use a cooperative model of threading: thread context
@@ -18,7 +21,7 @@ In addition, since there is only one operating system thread, a call that
 blocks that main thread will block the entire process.
 
 Yielding the thread in long-running tasks
------------------------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 If a code path takes a long time to execute and does not contain any methods
 that trigger an eventlet context switch, the long-running thread will block
 any pending threads.
@@ -37,7 +40,7 @@ time module is patched through eventlet.monkey_patch(). To be explicit, we recom
 contributors use ``greenthread.sleep()`` instead of ``time.sleep()``.
 
 MySQL access and eventlet
--------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~
 There are some MySQL DB API drivers for oslo.db, like `PyMySQL`_, MySQL-python
 etc. PyMySQL is the default MySQL DB API driver for oslo.db, and it works well with
 eventlet. MySQL-python uses an external C library for accessing the MySQL database.
@@ -54,3 +57,21 @@ a discussion of the `impact on performance`_.
 .. _mailing list thread: https://lists.launchpad.net/openstack/msg08118.html
 .. _impact on performance: https://lists.launchpad.net/openstack/msg08217.html
 .. _PyMySQL: https://wiki.openstack.org/wiki/PyMySQL_evaluation
+
+Native threading
+----------------
+Since the Flamingo release OpenStack started to transition away form
+``eventlet``. During this transition Nova maintains support for running
+services with ``eventlet`` while working to add support for running services
+with ``native threading``.
+
+To support both modes with the same codebase Nova started using the
+`futurist`_ library. In native threading mode ``futurist.ThreadPoolsExecutors``
+are used to run concurrent tasks and both the oslo.service and the
+oslo.messaging libraries are configured to use native threads to execute tasks
+like periodics and RPC message handlers.
+
+.. _futurist: https://docs.openstack.org/futurist/latest/
+
+To see how to configure and tune the native threading mode read the
+:doc:`/admin/concurrency` guide.