Merge "Documentation tune-ups"
This commit is contained in:
@@ -23,8 +23,8 @@ parts of :py:class:`linear flow <taskflow.patterns.linear_flow.Flow>` are run
|
|||||||
one after another, in order, even if engine is *capable* of running tasks in
|
one after another, in order, even if engine is *capable* of running tasks in
|
||||||
parallel).
|
parallel).
|
||||||
|
|
||||||
Creating Engines
|
Creating
|
||||||
================
|
========
|
||||||
|
|
||||||
All engines are mere classes that implement the same interface, and of course
|
All engines are mere classes that implement the same interface, and of course
|
||||||
it is possible to import them and create instances just like with any classes
|
it is possible to import them and create instances just like with any classes
|
||||||
@@ -43,8 +43,8 @@ might look like::
|
|||||||
|
|
||||||
.. automodule:: taskflow.engines.helpers
|
.. automodule:: taskflow.engines.helpers
|
||||||
|
|
||||||
Engine Configuration
|
Usage
|
||||||
====================
|
=====
|
||||||
|
|
||||||
To select which engine to use and pass parameters to an engine you should use
|
To select which engine to use and pass parameters to an engine you should use
|
||||||
the ``engine_conf`` parameter any helper factory function accepts. It may be:
|
the ``engine_conf`` parameter any helper factory function accepts. It may be:
|
||||||
@@ -53,10 +53,8 @@ the ``engine_conf`` parameter any helper factory function accepts. It may be:
|
|||||||
* a dictionary, holding engine type with key ``'engine'`` and possibly
|
* a dictionary, holding engine type with key ``'engine'`` and possibly
|
||||||
type-specific engine parameters.
|
type-specific engine parameters.
|
||||||
|
|
||||||
Known engine types are listed below.
|
Single-Threaded
|
||||||
|
---------------
|
||||||
Single-Threaded Engine
|
|
||||||
----------------------
|
|
||||||
|
|
||||||
**Engine type**: ``'serial'``
|
**Engine type**: ``'serial'``
|
||||||
|
|
||||||
@@ -70,8 +68,8 @@ on. This engine is used by default.
|
|||||||
greenthreads and monkey patching). See `eventlet <http://eventlet.net/>`_
|
greenthreads and monkey patching). See `eventlet <http://eventlet.net/>`_
|
||||||
and `greenlet <http://greenlet.readthedocs.org/>`_ for more details.
|
and `greenlet <http://greenlet.readthedocs.org/>`_ for more details.
|
||||||
|
|
||||||
Parallel Engine
|
Parallel
|
||||||
---------------
|
--------
|
||||||
|
|
||||||
**Engine type**: ``'parallel'``
|
**Engine type**: ``'parallel'``
|
||||||
|
|
||||||
@@ -96,8 +94,8 @@ Additional configuration parameters:
|
|||||||
Running tasks with ``concurrent.futures.ProcessPoolExecutor`` is not
|
Running tasks with ``concurrent.futures.ProcessPoolExecutor`` is not
|
||||||
supported now.
|
supported now.
|
||||||
|
|
||||||
Worker-Based Engine
|
Worker-Based
|
||||||
-------------------
|
------------
|
||||||
|
|
||||||
**Engine type**: ``'worker-based'``
|
**Engine type**: ``'worker-based'``
|
||||||
|
|
||||||
@@ -118,8 +116,8 @@ operates.
|
|||||||
.. _wiki page: https://wiki.openstack.org/wiki/TaskFlow/Worker-based_Engine
|
.. _wiki page: https://wiki.openstack.org/wiki/TaskFlow/Worker-based_Engine
|
||||||
.. _blueprint page: https://blueprints.launchpad.net/taskflow
|
.. _blueprint page: https://blueprints.launchpad.net/taskflow
|
||||||
|
|
||||||
Engine Interface
|
Interfaces
|
||||||
================
|
==========
|
||||||
|
|
||||||
.. automodule:: taskflow.engines.base
|
.. automodule:: taskflow.engines.base
|
||||||
|
|
||||||
|
|||||||
@@ -19,8 +19,29 @@ claiming them, and only remove them from the queue when they're done with the
|
|||||||
work. If the consumer fails, the lock is *automatically* released and the item
|
work. If the consumer fails, the lock is *automatically* released and the item
|
||||||
is back on the queue for further consumption.
|
is back on the queue for further consumption.
|
||||||
|
|
||||||
|
For more information, please see `wiki page`_ for more details.
|
||||||
|
|
||||||
|
Definitions
|
||||||
|
===========
|
||||||
|
|
||||||
|
Jobs
|
||||||
|
A :py:class:`job <taskflow.jobs.job.Job>` consists of a unique identifier, name,
|
||||||
|
and a reference to a :py:class:`logbook <taskflow.persistence.logbook.LogBook>`
|
||||||
|
which contains the details of the work that has been or should be/will be
|
||||||
|
completed to finish the work that has been created for that job.
|
||||||
|
|
||||||
|
Jobboards
|
||||||
|
A :py:class:`jobboard <taskflow.jobs.jobboard.JobBoard>` is responsible for managing
|
||||||
|
the posting, ownership, and delivery of jobs. It acts as the location where jobs
|
||||||
|
can be posted, claimed and searched for; typically by iteration or notification.
|
||||||
|
Jobboards may be backed by different *capable* implementations (each with potentially differing
|
||||||
|
configuration) but all jobboards implement the same interface and semantics so
|
||||||
|
that the backend usage is as transparent as possible. This allows deployers or
|
||||||
|
developers of a service that uses TaskFlow to select a jobboard implementation
|
||||||
|
that fits their setup (and there intended usage) best.
|
||||||
|
|
||||||
Features
|
Features
|
||||||
--------
|
========
|
||||||
|
|
||||||
- High availability
|
- High availability
|
||||||
|
|
||||||
@@ -58,29 +79,8 @@ Features
|
|||||||
user to poll for status (similar in concept to a shipping *tracking*
|
user to poll for status (similar in concept to a shipping *tracking*
|
||||||
identifier created by fedex or UPS).
|
identifier created by fedex or UPS).
|
||||||
|
|
||||||
For more information, please see `wiki page`_ for more details.
|
Usage
|
||||||
|
=====
|
||||||
Jobs
|
|
||||||
----
|
|
||||||
|
|
||||||
A job consists of a unique identifier, name, and a reference to a logbook
|
|
||||||
which contains the details of the work that has been or should be/will be
|
|
||||||
completed to finish the work that has been created for that job.
|
|
||||||
|
|
||||||
Jobboards
|
|
||||||
---------
|
|
||||||
|
|
||||||
A jobboard is responsible for managing the posting, ownership, and delivery
|
|
||||||
of jobs. It acts as the location where jobs can be posted, claimed and searched
|
|
||||||
for; typically by iteration or notification. Jobboards may be backed by
|
|
||||||
different *capable* implementations (each with potentially differing
|
|
||||||
configuration) but all jobboards implement the same interface and semantics so
|
|
||||||
that the backend usage is as transparent as possible. This allows deployers or
|
|
||||||
developers of a service that uses TaskFlow to select a jobboard implementation
|
|
||||||
that fits their setup (and there intended usage) best.
|
|
||||||
|
|
||||||
Using Jobboards
|
|
||||||
===============
|
|
||||||
|
|
||||||
All engines are mere classes that implement same interface, and of course it is
|
All engines are mere classes that implement same interface, and of course it is
|
||||||
possible to import them and create their instances just like with any classes
|
possible to import them and create their instances just like with any classes
|
||||||
@@ -156,11 +156,6 @@ might look like:
|
|||||||
time.sleep(coffee_break_time)
|
time.sleep(coffee_break_time)
|
||||||
...
|
...
|
||||||
|
|
||||||
Jobboard Configuration
|
|
||||||
======================
|
|
||||||
|
|
||||||
Known engine types are listed below.
|
|
||||||
|
|
||||||
Zookeeper
|
Zookeeper
|
||||||
---------
|
---------
|
||||||
|
|
||||||
@@ -239,15 +234,11 @@ the claim by then, therefore both would be *working* on a job.
|
|||||||
.. _idempotent: http://en.wikipedia.org/wiki/Idempotence
|
.. _idempotent: http://en.wikipedia.org/wiki/Idempotence
|
||||||
.. _preemptable: http://en.wikipedia.org/wiki/Preemption_%28computing%29
|
.. _preemptable: http://en.wikipedia.org/wiki/Preemption_%28computing%29
|
||||||
|
|
||||||
Job Interface
|
Interfaces
|
||||||
=============
|
==========
|
||||||
|
|
||||||
.. automodule:: taskflow.jobs.backends
|
.. automodule:: taskflow.jobs.backends
|
||||||
.. automodule:: taskflow.jobs.job
|
.. automodule:: taskflow.jobs.job
|
||||||
|
|
||||||
Jobboard Interface
|
|
||||||
==================
|
|
||||||
|
|
||||||
.. automodule:: taskflow.jobs.jobboard
|
.. automodule:: taskflow.jobs.jobboard
|
||||||
|
|
||||||
.. _wiki page: https://wiki.openstack.org/wiki/TaskFlow/Paradigm_shifts#Workflow_ownership_transfer
|
.. _wiki page: https://wiki.openstack.org/wiki/TaskFlow/Paradigm_shifts#Workflow_ownership_transfer
|
||||||
|
|||||||
@@ -67,7 +67,7 @@ To receive notification on flow state changes use
|
|||||||
woof
|
woof
|
||||||
Flow 'cat-dog' transition to state SUCCESS
|
Flow 'cat-dog' transition to state SUCCESS
|
||||||
|
|
||||||
Task notifications
|
Task Notifications
|
||||||
------------------
|
------------------
|
||||||
|
|
||||||
To receive notification on task state changes use
|
To receive notification on task state changes use
|
||||||
|
|||||||
@@ -105,8 +105,8 @@ A few scenarios come to mind:
|
|||||||
(when data is saved -- every time it changes or at some particular moments
|
(when data is saved -- every time it changes or at some particular moments
|
||||||
or simply never).
|
or simply never).
|
||||||
|
|
||||||
Persistence Configuration
|
Usage
|
||||||
=========================
|
=====
|
||||||
|
|
||||||
To select which persistence backend to use you should use the
|
To select which persistence backend to use you should use the
|
||||||
:py:meth:`fetch() <taskflow.persistence.backends.fetch>` function which uses
|
:py:meth:`fetch() <taskflow.persistence.backends.fetch>` function which uses
|
||||||
@@ -137,13 +137,17 @@ the following:
|
|||||||
``'connection'`` and possibly type-specific backend parameters as other
|
``'connection'`` and possibly type-specific backend parameters as other
|
||||||
keys.
|
keys.
|
||||||
|
|
||||||
Known engine types are listed below.
|
Memory
|
||||||
|
------
|
||||||
|
|
||||||
**Connection**: ``'memory'``
|
**Connection**: ``'memory'``
|
||||||
|
|
||||||
Retains all data in local memory (not persisted to reliable storage). Useful
|
Retains all data in local memory (not persisted to reliable storage). Useful
|
||||||
for scenarios where persistence is not required (and also in unit tests).
|
for scenarios where persistence is not required (and also in unit tests).
|
||||||
|
|
||||||
|
Files
|
||||||
|
-----
|
||||||
|
|
||||||
**Connection**: ``'dir'`` or ``'file'``
|
**Connection**: ``'dir'`` or ``'file'``
|
||||||
|
|
||||||
Retains all data in a directory & file based structure on local disk. Will be
|
Retains all data in a directory & file based structure on local disk. Will be
|
||||||
@@ -152,6 +156,9 @@ from the same local machine only). Useful for cases where a *more* reliable
|
|||||||
persistence is desired along with the simplicity of files and directories (a
|
persistence is desired along with the simplicity of files and directories (a
|
||||||
concept everyone is familiar with).
|
concept everyone is familiar with).
|
||||||
|
|
||||||
|
Sqlalchemy
|
||||||
|
----------
|
||||||
|
|
||||||
**Connection**: ``'mysql'`` or ``'postgres'`` or ``'sqlite'``
|
**Connection**: ``'mysql'`` or ``'postgres'`` or ``'sqlite'``
|
||||||
|
|
||||||
Retains all data in a `ACID`_ compliant database using the `sqlalchemy`_ library
|
Retains all data in a `ACID`_ compliant database using the `sqlalchemy`_ library
|
||||||
@@ -163,6 +170,9 @@ does not apply when using sqlite).
|
|||||||
.. _sqlalchemy: http://www.sqlalchemy.org/docs/
|
.. _sqlalchemy: http://www.sqlalchemy.org/docs/
|
||||||
.. _ACID: https://en.wikipedia.org/wiki/ACID
|
.. _ACID: https://en.wikipedia.org/wiki/ACID
|
||||||
|
|
||||||
|
Zookeeper
|
||||||
|
---------
|
||||||
|
|
||||||
**Connection**: ``'zookeeper'``
|
**Connection**: ``'zookeeper'``
|
||||||
|
|
||||||
Retains all data in a `zookeeper`_ backend (zookeeper exposes operations on
|
Retains all data in a `zookeeper`_ backend (zookeeper exposes operations on
|
||||||
@@ -176,8 +186,8 @@ as the database connection types listed previously).
|
|||||||
.. _zookeeper: http://zookeeper.apache.org
|
.. _zookeeper: http://zookeeper.apache.org
|
||||||
.. _kazoo: http://kazoo.readthedocs.org/
|
.. _kazoo: http://kazoo.readthedocs.org/
|
||||||
|
|
||||||
Persistence Backend Interfaces
|
Interfaces
|
||||||
==============================
|
==========
|
||||||
|
|
||||||
.. automodule:: taskflow.persistence.backends
|
.. automodule:: taskflow.persistence.backends
|
||||||
.. automodule:: taskflow.persistence.backends.base
|
.. automodule:: taskflow.persistence.backends.base
|
||||||
|
|||||||
Reference in New Issue
Block a user