diff --git a/doc/source/engines.rst b/doc/source/engines.rst index 62d3a1a3..359cea6b 100644 --- a/doc/source/engines.rst +++ b/doc/source/engines.rst @@ -23,8 +23,8 @@ parts of :py:class:`linear flow ` are run one after another, in order, even if engine is *capable* of running tasks in parallel). -Creating Engines -================ +Creating +======== All engines are mere classes that implement the same interface, and of course it is possible to import them and create instances just like with any classes @@ -43,8 +43,8 @@ might look like:: .. automodule:: taskflow.engines.helpers -Engine Configuration -==================== +Usage +===== To select which engine to use and pass parameters to an engine you should use the ``engine_conf`` parameter any helper factory function accepts. It may be: @@ -53,10 +53,8 @@ the ``engine_conf`` parameter any helper factory function accepts. It may be: * a dictionary, holding engine type with key ``'engine'`` and possibly type-specific engine parameters. -Known engine types are listed below. - -Single-Threaded Engine ----------------------- +Single-Threaded +--------------- **Engine type**: ``'serial'`` @@ -70,8 +68,8 @@ on. This engine is used by default. greenthreads and monkey patching). See `eventlet `_ and `greenlet `_ for more details. -Parallel Engine ---------------- +Parallel +-------- **Engine type**: ``'parallel'`` @@ -96,8 +94,8 @@ Additional configuration parameters: Running tasks with ``concurrent.futures.ProcessPoolExecutor`` is not supported now. -Worker-Based Engine -------------------- +Worker-Based +------------ **Engine type**: ``'worker-based'`` @@ -118,8 +116,8 @@ operates. .. _wiki page: https://wiki.openstack.org/wiki/TaskFlow/Worker-based_Engine .. _blueprint page: https://blueprints.launchpad.net/taskflow -Engine Interface -================ +Interfaces +========== .. automodule:: taskflow.engines.base diff --git a/doc/source/jobs.rst b/doc/source/jobs.rst index 0d021696..374eb3c9 100644 --- a/doc/source/jobs.rst +++ b/doc/source/jobs.rst @@ -19,8 +19,29 @@ claiming them, and only remove them from the queue when they're done with the work. If the consumer fails, the lock is *automatically* released and the item is back on the queue for further consumption. +For more information, please see `wiki page`_ for more details. + +Definitions +=========== + +Jobs + A :py:class:`job ` consists of a unique identifier, name, + and a reference to a :py:class:`logbook ` + which contains the details of the work that has been or should be/will be + completed to finish the work that has been created for that job. + +Jobboards + A :py:class:`jobboard ` is responsible for managing + the posting, ownership, and delivery of jobs. It acts as the location where jobs + can be posted, claimed and searched for; typically by iteration or notification. + Jobboards may be backed by different *capable* implementations (each with potentially differing + configuration) but all jobboards implement the same interface and semantics so + that the backend usage is as transparent as possible. This allows deployers or + developers of a service that uses TaskFlow to select a jobboard implementation + that fits their setup (and there intended usage) best. + Features --------- +======== - High availability @@ -58,29 +79,8 @@ Features user to poll for status (similar in concept to a shipping *tracking* identifier created by fedex or UPS). -For more information, please see `wiki page`_ for more details. - -Jobs ----- - -A job consists of a unique identifier, name, and a reference to a logbook -which contains the details of the work that has been or should be/will be -completed to finish the work that has been created for that job. - -Jobboards ---------- - -A jobboard is responsible for managing the posting, ownership, and delivery -of jobs. It acts as the location where jobs can be posted, claimed and searched -for; typically by iteration or notification. Jobboards may be backed by -different *capable* implementations (each with potentially differing -configuration) but all jobboards implement the same interface and semantics so -that the backend usage is as transparent as possible. This allows deployers or -developers of a service that uses TaskFlow to select a jobboard implementation -that fits their setup (and there intended usage) best. - -Using Jobboards -=============== +Usage +===== All engines are mere classes that implement same interface, and of course it is possible to import them and create their instances just like with any classes @@ -156,11 +156,6 @@ might look like: time.sleep(coffee_break_time) ... -Jobboard Configuration -====================== - -Known engine types are listed below. - Zookeeper --------- @@ -239,15 +234,11 @@ the claim by then, therefore both would be *working* on a job. .. _idempotent: http://en.wikipedia.org/wiki/Idempotence .. _preemptable: http://en.wikipedia.org/wiki/Preemption_%28computing%29 -Job Interface -============= +Interfaces +========== .. automodule:: taskflow.jobs.backends .. automodule:: taskflow.jobs.job - -Jobboard Interface -================== - .. automodule:: taskflow.jobs.jobboard .. _wiki page: https://wiki.openstack.org/wiki/TaskFlow/Paradigm_shifts#Workflow_ownership_transfer diff --git a/doc/source/notifications.rst b/doc/source/notifications.rst index 8f92317d..88969bdd 100644 --- a/doc/source/notifications.rst +++ b/doc/source/notifications.rst @@ -67,7 +67,7 @@ To receive notification on flow state changes use woof Flow 'cat-dog' transition to state SUCCESS -Task notifications +Task Notifications ------------------ To receive notification on task state changes use diff --git a/doc/source/persistence.rst b/doc/source/persistence.rst index f7fe810c..75d1fab3 100644 --- a/doc/source/persistence.rst +++ b/doc/source/persistence.rst @@ -105,8 +105,8 @@ A few scenarios come to mind: (when data is saved -- every time it changes or at some particular moments or simply never). -Persistence Configuration -========================= +Usage +===== To select which persistence backend to use you should use the :py:meth:`fetch() ` function which uses @@ -137,13 +137,17 @@ the following: ``'connection'`` and possibly type-specific backend parameters as other keys. -Known engine types are listed below. +Memory +------ **Connection**: ``'memory'`` Retains all data in local memory (not persisted to reliable storage). Useful for scenarios where persistence is not required (and also in unit tests). +Files +----- + **Connection**: ``'dir'`` or ``'file'`` Retains all data in a directory & file based structure on local disk. Will be @@ -152,6 +156,9 @@ from the same local machine only). Useful for cases where a *more* reliable persistence is desired along with the simplicity of files and directories (a concept everyone is familiar with). +Sqlalchemy +---------- + **Connection**: ``'mysql'`` or ``'postgres'`` or ``'sqlite'`` Retains all data in a `ACID`_ compliant database using the `sqlalchemy`_ library @@ -163,6 +170,9 @@ does not apply when using sqlite). .. _sqlalchemy: http://www.sqlalchemy.org/docs/ .. _ACID: https://en.wikipedia.org/wiki/ACID +Zookeeper +--------- + **Connection**: ``'zookeeper'`` Retains all data in a `zookeeper`_ backend (zookeeper exposes operations on @@ -176,8 +186,8 @@ as the database connection types listed previously). .. _zookeeper: http://zookeeper.apache.org .. _kazoo: http://kazoo.readthedocs.org/ -Persistence Backend Interfaces -============================== +Interfaces +========== .. automodule:: taskflow.persistence.backends .. automodule:: taskflow.persistence.backends.base