openstack/deb-python-taskflow
Code Issues Proposed changes
Files
19f9674877902e7e67b6c48df04571b040fdef50
deb-python-taskflow/doc/source/conductors.rst
Joshua Harlow bb384577bc Move implementation(s) to there own sections 
Instead of putting implementation(s) under the interfaces
section put the implementation(s) under there own section.

This also includes some other tweaks to refer to those
implementation(s) where appropriate.

Change-Id: Iffdc0439c843e7f70cf873e5a75501feb51f96c7
2015-01-16 22:18:57 +00:00

2.3 KiB
Raw Blame History

Conductors

Overview

Conductors in TaskFlow provide a mechanism that unifies the various TaskFlow concepts under a single easy to use (as plug-and-play as we can make it) construct.

They are responsible for the following:

  • Interacting with jobboards <jobs> (examining and claiming jobs <jobs>).
  • Creating engines <engines> from the claimed jobs (using factories <resumption factories> to reconstruct the contained tasks and flows to be executed).
  • Dispatching the engine using the provided persistence <persistence> layer and engine configuration.
  • Completing or abandoning the claimed job (depending on dispatching and execution outcome).
  • Rinse and repeat.

Note

They are inspired by and have similar responsibilities as railroad conductors.

Considerations

Some usage considerations should be used when using a conductor to make sure it's used in a safe and reliable manner. Eventually we hope to make these non-issues but for now they are worth mentioning.

Endless cycling

What: Jobs that fail (due to some type of internal error) on one conductor will be abandoned by that conductor and then another conductor may experience those same errors and abandon it (and repeat). This will create a job abandonment cycle that will continue for as long as the job exists in an claimable state.

Example:

Conductor cycling

Alleviate by:

  1. Forcefully delete jobs that have been failing continuously after a given number of conductor attempts. This can be either done manually or automatically via scripts (or other associated monitoring).
  2. Resolve the internal error's cause (storage backend failure, other...).
  3. Help implement jobboard garbage binning.

Interfaces

taskflow.conductors.base

Implementations

taskflow.conductors.single_threaded

Hierarchy

taskflow.conductors.base taskflow.conductors.single_threaded