Include docstrings for parallel engine types/strings supported

When a 'executor' option is passed to a action engine requested
to run in parallel mode that option will internally be examined
and it will affect the internally used execution model that the
engine will use; to make it understandable what the valid options
are include a docstring + table(s) that describes the options and
there valid values.

Change-Id: I9a1852427bae22a01f5993862617e384f10ec005
This commit is contained in:
Joshua Harlow
2015-01-10 14:23:34 -08:00
parent 316b453558
commit 2f7d86ac3e
3 changed files with 44 additions and 12 deletions

View File

@@ -161,16 +161,10 @@ Parallel
**Engine type**: ``'parallel'``
A parallel engine schedules tasks onto different threads/processes to allow for
running non-dependent tasks simultaneously.
Additional supported keyword arguments:
* ``executor``: a object that implements a :pep:`3148` compatible `executor`_
interface; it will be used for scheduling tasks. You can use instances of a
`thread pool executor`_ or a :py:class:`green executor
<taskflow.types.futures.GreenThreadPoolExecutor>` (which internally uses
`eventlet <http://eventlet.net/>`_ and greenthread pools) or a
`process pool executor`_.
running non-dependent tasks simultaneously. See the documentation of
:py:class:`~taskflow.engines.action_engine.engine.ParallelActionEngine` for
supported arguments that can be used to construct a parallel engine that runs
using your desired execution model.
.. tip::
@@ -340,6 +334,7 @@ Interfaces
.. automodule:: taskflow.engines.action_engine.compiler
.. automodule:: taskflow.engines.action_engine.completer
.. automodule:: taskflow.engines.action_engine.engine
.. automodule:: taskflow.engines.action_engine.executor
.. automodule:: taskflow.engines.action_engine.runner
.. automodule:: taskflow.engines.action_engine.runtime
.. automodule:: taskflow.engines.action_engine.scheduler
@@ -360,5 +355,4 @@ Hierarchy
.. _executor: https://docs.python.org/dev/library/concurrent.futures.html#concurrent.futures.Executor
.. _networkx: https://networkx.github.io/
.. _futures backport: https://pypi.python.org/pypi/futures
.. _thread pool executor: https://docs.python.org/dev/library/concurrent.futures.html#threadpoolexecutor
.. _process pool executor: https://docs.python.org/dev/library/concurrent.futures.html#processpoolexecutor

View File

@@ -239,7 +239,43 @@ class _ExecutorTextMatch(collections.namedtuple('_ExecutorTextMatch',
class ParallelActionEngine(ActionEngine):
"""Engine that runs tasks in parallel manner."""
"""Engine that runs tasks in parallel manner.
Supported keyword arguments:
* ``executor``: a object that implements a :pep:`3148` compatible executor
interface; it will be used for scheduling tasks. The following
type are applicable (other unknown types passed will cause a type
error to be raised).
========================= ===============================================
Type provided Executor used
========================= ===============================================
|cft|.ThreadPoolExecutor :class:`~.executor.ParallelThreadTaskExecutor`
|cfp|.ProcessPoolExecutor :class:`~.executor.ParallelProcessTaskExecutor`
|cf|._base.Executor :class:`~.executor.ParallelThreadTaskExecutor`
========================= ===============================================
* ``executor``: a string that will be used to select a :pep:`3148`
compatible executor; it will be used for scheduling tasks. The following
string are applicable (other unknown strings passed will cause a value
error to be raised).
=========================== ===============================================
String (case insensitive) Executor used
=========================== ===============================================
``process`` :class:`~.executor.ParallelProcessTaskExecutor`
``processes`` :class:`~.executor.ParallelProcessTaskExecutor`
``thread`` :class:`~.executor.ParallelThreadTaskExecutor`
``threaded`` :class:`~.executor.ParallelThreadTaskExecutor`
``threads`` :class:`~.executor.ParallelThreadTaskExecutor`
=========================== ===============================================
.. |cfp| replace:: concurrent.futures.process
.. |cft| replace:: concurrent.futures.thread
.. |cf| replace:: concurrent.futures
"""
_storage_factory = atom_storage.MultiThreadedStorage
# One of these types should match when a object (non-string) is provided

View File

@@ -373,6 +373,7 @@ class ParallelTaskExecutor(TaskExecutor):
to concurrent.Futures.Executor.
"""
#: Options this executor supports (passed in from engine options).
OPTIONS = frozenset(['max_workers'])
def __init__(self, executor=None, max_workers=None):
@@ -431,6 +432,7 @@ class ParallelProcessTaskExecutor(ParallelTaskExecutor):
the parent are executed on events in the child.
"""
#: Options this executor supports (passed in from engine options).
OPTIONS = frozenset(['max_workers', 'dispatch_periodicity'])
def __init__(self, executor=None, max_workers=None,