openstack/taskflow
Code Issues Proposed changes

Doc adjustments

Browse Source 
- Cleanup some grammar and adjust some wording usage.
- Add more docs to utils about what should and should
  not be used.
- Add more engine docs about how each one is used and
  tips and notes about each.
- Line length adjustments (might as well keep it somewhat
  in the normal range of what openstack code expects)

Change-Id: Ice6711f00e2b50e0bee777388c0555d79cc6e1b0
This commit is contained in:
Joshua Harlow
2014-04-11 12:35:01 -07:00
parent df3ca0083c
commit f2c82f0929
4 changed files with 114 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
doc/source/atoms.rst
View File

@@ -3,10 +3,10 @@ Atoms, Tasks and Retries
------------------------
An atom is the smallest unit in taskflow which acts as the base for other
classes. Atoms have a name and a version (if applicable). Atom is expected
classes. Atoms have a name and a version (if applicable). An atom is expected
to name desired input values (requirements) and name outputs (provided
values), see :doc:`arguments_and_results` page for complete reference
about it.
values), see :doc:`arguments_and_results` page for a complete reference
about these inputs and outputs.
.. automodule:: taskflow.atom
@@ -22,7 +22,9 @@ Retry
=====
A retry (derived from an atom) is a special unit that handles flow errors,
controlls flow execution and can retry it with another parameters if needed.
controls flow execution and can retry atoms with another parameters if needed.
It is useful to allow for alternate ways of retrying atoms when they fail so
the whole flow can proceed even when a group of atoms fail.
.. automodule:: taskflow.retry

66
doc/source/engines.rst
View File

@@ -5,32 +5,33 @@ Engines
Overview
========
Engines are what **really** runs your tasks and flows.
Engines are what **really** runs your atoms.
An *engine* takes a flow structure (described by :doc:`patterns`) and uses it to
decide which :doc:`atom <atoms>` to run and when.
TaskFlow provides different implementation of engines. Some may be easier to
use (ie, require no additional infrastructure setup) and understand, others
TaskFlow provides different implementations of engines. Some may be easier to
use (ie, require no additional infrastructure setup) and understand; others
might require more complicated setup but provide better scalability. The idea
and *ideal* is that deployers or developers of a service that uses TaskFlow can
select an engine that suites their setup best without modifying the code of
said service.
Engines might have different capabilities and configuration, but all of them
**must** implement same interface and preserve semantics of patterns (e.g.
Engines usually have different capabilities and configuration, but all of them
**must** implement the same interface and preserve the semantics of patterns (e.g.
parts of :py:class:`linear flow <taskflow.patterns.linear_flow.Flow>` are run
one after another, in order, even if engine is *capable* run tasks in
one after another, in order, even if engine is *capable* of running tasks in
parallel).
Creating Engines
================
All engines are mere classes that implement same interface, and of course it is
possible to import them and create their instances just like with any classes
in Python. But easier (and recommended) way for creating engine is using
engine helpers. All of them are imported into `taskflow.engines` module, so the
typical usage of them might look like::
All engines are mere classes that implement the same interface, and of course
it is possible to import them and create instances just like with any classes
in Python. But the easier (and recommended) way for creating an engine is using
the engine helper functions. All of these functions are imported into the
`taskflow.engines` module namespace, so the typical usage of these functions
might look like::
from taskflow import engines
@@ -46,7 +47,7 @@ Engine Configuration
====================
To select which engine to use and pass parameters to an engine you should use
``engine_conf`` parameter any helper factory function accepts. It may be:
the ``engine_conf`` parameter any helper factory function accepts. It may be:
* a string, naming engine type;
* a dictionary, holding engine type with key ``'engine'`` and possibly
@@ -62,6 +63,12 @@ Single-Threaded Engine
Runs all tasks on the single thread -- the same thread `engine.run()` is called
on. This engine is used by default.
.. tip::
If eventlet is used then this engine will not block other threads
from running as eventlet automatically creates a co-routine system (using
greenthreads and monkey patching). See `eventlet <http://eventlet.net/>`_
and `greenlet <http://greenlet.readthedocs.org/>`_ for more details.
Parallel Engine
---------------
@@ -75,10 +82,17 @@ Additional configuration parameters:
* ``executor``: a class that provides ``concurrent.futures.Executor``-like
interface; it will be used for scheduling tasks. You can use instances
of ``concurrent.futures.ThreadPoolExecutor`` or
``taskflow.utils.eventlet_utils.GreenExecutor``. Sharing executor between
engine instances provides better scalability.
``taskflow.utils.eventlet_utils.GreenExecutor`` (which internally uses
`eventlet <http://eventlet.net/>`_ and greenthread pools).
.. tip::
Sharing executor between engine instances provides better
scalability by reducing thread creation and teardown as well as by reusing
existing pools (which is a good practice in general).
.. note::
Running tasks with ``concurrent.futures.ProcessPoolExecutor`` is not
supported now.
@@ -88,14 +102,32 @@ Worker-Based Engine
**Engine type**: ``'worker-based'``
This is engine that schedules tasks to **workers** -- separate processes
dedicated for tasks execution, possibly running on other machines.
dedicated for certain tasks execution, possibly running on other machines,
connected via `amqp <http://www.amqp.org/>`_ (or other supported
`kombu <http://kombu.readthedocs.org/>`_ transports). For more information,
please see `wiki page`_ for more details on how the worker based engine
operates.
This engine is under active development and is not recommended for production
use yet. For more information, please see `wiki page`_ for more details.
.. note::
This engine is under active development and is experimental but it is
useable and does work but is missing some features (please check the
`blueprint page`_ for known issues and plans) that will make it more
production ready.
.. _wiki page: https://wiki.openstack.org/wiki/TaskFlow/Worker-based_Engine
.. _blueprint page: https://blueprints.launchpad.net/taskflow
Engine Interface
================
.. automodule:: taskflow.engines.base
Hierarchy
=========
.. inheritance-diagram::
taskflow.engines.base
taskflow.engines.action_engine.engine
taskflow.engines.worker_based.engine
:parts: 1

40
doc/source/inputs_and_outputs.rst
View File

@@ -12,11 +12,21 @@ use :doc:`persistence` directly.
Flow Inputs and Outputs
-----------------------
Tasks accept inputs via task arguments and provide outputs via task results (see :doc:`arguments_and_results` for more details). This the standard and recommended way to pass data from one task to another. Of course not every task argument needs to be provided to some other task of a flow, and not every task result should be consumed by every task.
Tasks accept inputs via task arguments and provide outputs via task results
(see :doc:`arguments_and_results` for more details). This is the standard and
recommended way to pass data from one task to another. Of course not every task
argument needs to be provided to some other task of a flow, and not every task
result should be consumed by every task.
If some value is required by one or more tasks of a flow, but is not provided by any task, it is considered to be flow input, and **must** be put into the storage before the flow is run. A set of names required by a flow can be retrieved via that flow's ``requires`` property. These names can be used to determine what names may be applicable for placing in storage ahead of time and which names are not applicable.
If some value is required by one or more tasks of a flow, but is not provided
by any task, it is considered to be flow input, and **must** be put into the
storage before the flow is run. A set of names required by a flow can be
retrieved via that flow's ``requires`` property. These names can be used to
determine what names may be applicable for placing in storage ahead of time
and which names are not applicable.
All values provided by tasks of the flow are considered to be flow outputs; the set of names of such values is available via ``provides`` property of the flow.
All values provided by tasks of the flow are considered to be flow outputs; the
set of names of such values is available via ``provides`` property of the flow.
.. testsetup::
@@ -52,12 +62,17 @@ As you can see, this flow does not require b, as it is provided by the fist task
Engine and Storage
------------------
The storage layer is how an engine persists flow and task details. For more in-depth design details see :doc:`persistence` and :doc:`storage`.
The storage layer is how an engine persists flow and task details. For more
in-depth design details see :doc:`persistence` and :doc:`storage`.
Inputs
------
As mentioned above, if some value is required by one or more tasks of a flow, but is not provided by any task, it is considered to be flow input, and **must** be put into the storage before the flow is run. On failure to do so :py:class:`~taskflow.exceptions.MissingDependencies` is raised by engine:
As mentioned above, if some value is required by one or more tasks of a flow,
but is not provided by any task, it is considered to be flow input, and
**must** be put into the storage before the flow is run. On failure to do
so :py:class:`~taskflow.exceptions.MissingDependencies` is raised by the engine
prior to running:
.. doctest::
@@ -80,7 +95,9 @@ As mentioned above, if some value is required by one or more tasks of a flow, bu
taskflow.exceptions.MissingDependencies: taskflow.patterns.linear_flow.Flow: cat-dog;
2 requires ['meow', 'woof'] but no other entity produces said requirements
The recommended way to provide flow inputs is to use ``store`` parameter of engine helpers (:py:func:`~taskflow.engines.helpers.run` or :py:func:`~taskflow.engines.helpers.load`):
The recommended way to provide flow inputs is to use the ``store`` parameter
of the engine helpers (:py:func:`~taskflow.engines.helpers.run` or
:py:func:`~taskflow.engines.helpers.load`):
.. doctest::
@@ -102,7 +119,10 @@ The recommended way to provide flow inputs is to use ``store`` parameter of engi
woof
{'meow': 'meow', 'woof': 'woof', 'dog': 'dog'}
You can also directly interact with the engine storage layer to add additional values, also you can't use :py:func:`~taskflow.engines.helpers.run` in this case:
You can also directly interact with the engine storage layer to add
additional values, note that if this route is used you can't use
:py:func:`~taskflow.engines.helpers.run` in this case to run your engine (instead
your must activate the engines run method directly):
.. doctest::
@@ -118,7 +138,11 @@ You can also directly interact with the engine storage layer to add additional v
Outputs
-------
As you can see from examples above, run method returns all flow outputs in a ``dict``. This same data can be fetched via :py:meth:`~taskflow.storage.Storage.fetch_all` method of the storage. You can also get single results using :py:meth:`~taskflow.storage.Storage.fetch_all`. For example:
As you can see from examples above, the run method returns all flow outputs in
a ``dict``. This same data can be fetched via
:py:meth:`~taskflow.storage.Storage.fetch_all` method of the storage. You can
also get single results using :py:meth:`~taskflow.storage.Storage.fetch_all`. For
example:
.. doctest::

27
doc/source/utils.rst
View File

@@ -2,4 +2,31 @@
Utils
-----
There are various helper utils that are part of taskflows internal usage (and
external/public usage of these helpers should be kept to a minimum as these
utility functions may be altered more often in the future).
External usage
==============
The following classes and modules are *recommended* for external usage:
.. autoclass:: taskflow.utils.misc.Failure
:members:
.. autoclass:: taskflow.utils.eventlet_utils.GreenExecutor
:members:
.. autofunction:: taskflow.utils.graph_utils.pformat
.. autofunction:: taskflow.utils.graph_utils.export_graph_to_dot
.. autofunction:: taskflow.utils.persistence_utils.temporary_log_book
.. autofunction:: taskflow.utils.persistence_utils.temporary_flow_detail
.. autofunction:: taskflow.utils.persistence_utils.pformat
.. autofunction:: taskflow.utils.persistence_utils.pformat_flow_detail
.. autofunction:: taskflow.utils.persistence_utils.pformat_atom_detail