Retire Packaging Deb project repos

This commit is part of a series to retire the Packaging Deb
project. Step 2 is to remove all content from the project
repos, replacing it with a README notification where to find
ongoing work, and how to recover the repo if needed at some
future point (as in
https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project).

Change-Id: I6f54f00b686a16e9b3ea96642c78842db12af9d3
This commit is contained in:
Tony Breeds 2017-09-12 16:12:17 -06:00
parent 72db32b752
commit 057e46bf1e
303 changed files with 14 additions and 48254 deletions

View File

@ -1,8 +0,0 @@
[run]
branch = True
source = taskflow
omit = taskflow/tests/*,taskflow/openstack/*,taskflow/test.py
[report]
ignore_errors = True

64
.gitignore vendored
View File

@ -1,64 +0,0 @@
*.py[cod]
# C extensions
*.so
# Packages
*.egg
*.egg-info
dist
build
eggs
parts
bin
var
sdist
develop-eggs
.installed.cfg
lib
lib64
# Installer logs
pip-log.txt
# Unit test / coverage reports
.coverage*
.diagram-tools/*
.tox
nosetests.xml
.venv
cover
.testrepository
htmlcov
# Translations
*.mo
# Mr Developer
.mr.developer.cfg
.project
.pydevproject
.settings
# DS_STORE
.DS_Store
# Sqlite databases
*.sqlite
# Modified Files
*.swp
# PBR
build
AUTHORS
ChangeLog
# doc
doc/build/
.idea
env
# files created by releasenotes build
releasenotes/build

View File

@ -1,4 +0,0 @@
[gerrit]
host=review.openstack.org
port=29418
project=openstack/taskflow.git

View File

@ -1,12 +0,0 @@
Anastasia Karpinska <akarpinska@griddynamics.com>
Angus Salkeld <asalkeld@redhat.com>
Changbin Liu <changbl@research.att.com>
Changbin Liu <changbl@research.att.com> <changbin.liu@gmail.com>
Ivan A. Melnikov <imelnikov@griddynamics.com>
Jessica Lucci <jessica.lucci@rackspace.com>
Jessica Lucci <jessica.lucci@rackspace.com> <jessicalucci14@gmail.com>
Joshua Harlow <harlowja@yahoo-inc.com>
Joshua Harlow <harlowja@yahoo-inc.com> <harlowja@gmail.com>
Kevin Chen <kevin.chen@rackspace.com>
Kevin Chen <kevin.chen@rackspace.com> <kevi6362@mj4efudty3.rackspace.corp>
Kevin Chen <kevin.chen@rackspace.com> <kevin.chen.weijie@utexas.edu>

View File

@ -1,11 +0,0 @@
[DEFAULT]
test_command=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
OS_DEBUG=${OS_DEBUG:-TRACE} \
OS_LOG_CAPTURE=${OS_LOG_CAPTURE:-1} \
${PYTHON:-python} -m subunit.run discover -t ./ ./taskflow/tests $LISTOPT $IDOPTION
test_id_option=--load-list $IDFILE
test_list_option=--list

View File

@ -1,16 +0,0 @@
If you would like to contribute to the development of OpenStack,
you must follow the steps documented at:
http://docs.openstack.org/infra/manual/developers.html#development-workflow
Once those steps have been completed, changes to OpenStack
should be submitted for review via the Gerrit tool, following
the workflow documented at:
http://docs.openstack.org/infra/manual/developers.html#development-workflow
Pull requests submitted through GitHub will be ignored.
Bugs should be filed on Launchpad, not GitHub:
https://bugs.launchpad.net/taskflow

176
LICENSE
View File

@ -1,176 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.

14
README Normal file
View File

@ -0,0 +1,14 @@
This project is no longer maintained.
The contents of this repository are still available in the Git
source code management system. To see the contents of this
repository before it reached its end of life, please check out the
previous commit with "git checkout HEAD^1".
For ongoing work on maintaining OpenStack packages in the Debian
distribution, please see the Debian OpenStack packaging team at
https://wiki.debian.org/OpenStack/.
For any further questions, please email
openstack-dev@lists.openstack.org or join #openstack-dev on
Freenode.

View File

@ -1,77 +0,0 @@
========================
Team and repository tags
========================
.. image:: http://governance.openstack.org/badges/taskflow.svg
:target: http://governance.openstack.org/reference/tags/index.html
.. Change things from this point on
TaskFlow
========
.. image:: https://img.shields.io/pypi/v/taskflow.svg
:target: https://pypi.python.org/pypi/taskflow/
:alt: Latest Version
.. image:: https://img.shields.io/pypi/dm/taskflow.svg
:target: https://pypi.python.org/pypi/taskflow/
:alt: Downloads
A library to do [jobs, tasks, flows] in a highly available, easy to understand
and declarative manner (and more!) to be used with OpenStack and other
projects.
* Free software: Apache license
* Documentation: https://docs.openstack.org/taskflow/latest/
* Source: https://git.openstack.org/cgit/openstack/taskflow
* Bugs: https://bugs.launchpad.net/taskflow/
Join us
-------
- http://launchpad.net/taskflow
Testing and requirements
------------------------
Requirements
~~~~~~~~~~~~
Because this project has many optional (pluggable) parts like persistence
backends and engines, we decided to split our requirements into two
parts: - things that are absolutely required (you can't use the project
without them) are put into ``requirements.txt``. The requirements
that are required by some optional part of this project (you can use the
project without them) are put into our ``test-requirements.txt`` file (so
that we can still test the optional functionality works as expected). If
you want to use the feature in question (`eventlet`_ or the worker based engine
that uses `kombu`_ or the `sqlalchemy`_ persistence backend or jobboards which
have an implementation built using `kazoo`_ ...), you should add
that requirement(s) to your project or environment.
Tox.ini
~~~~~~~
Our ``tox.ini`` file describes several test environments that allow to test
TaskFlow with different python versions and sets of requirements installed.
Please refer to the `tox`_ documentation to understand how to make these test
environments work for you.
Developer documentation
-----------------------
We also have sphinx documentation in ``docs/source``.
*To build it, run:*
::
$ python setup.py build_sphinx
.. _kazoo: http://kazoo.readthedocs.org/
.. _sqlalchemy: http://www.sqlalchemy.org/
.. _kombu: http://kombu.readthedocs.org/
.. _eventlet: http://eventlet.net/
.. _tox: http://tox.testrun.org/
.. _developer documentation: https://docs.openstack.org/taskflow/latest/

Binary file not shown.

Binary file not shown.

Binary file not shown.

View File

@ -1,98 +0,0 @@
# -*- coding: utf-8 -*-
import datetime
import os
import subprocess
import sys
import warnings
sys.path.insert(0, os.path.abspath('../..'))
# -- General configuration ----------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.doctest',
'sphinx.ext.extlinks',
'sphinx.ext.inheritance_diagram',
'sphinx.ext.viewcode',
'openstackdocstheme'
]
# openstackdocstheme options
repository_name = 'openstack/taskflow'
bug_project = 'taskflow'
bug_tag = ''
html_last_updated_fmt = '%Y-%m-%d %H:%M'
# autodoc generation is a bit aggressive and a nuisance when doing heavy
# text edit cycles.
# execute "export SPHINX_DEBUG=1" in your terminal to disable
# Add any paths that contain templates here, relative to this directory.
templates_path = ['templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['_build']
# General information about the project.
project = u'TaskFlow'
copyright = u'%s, OpenStack Foundation' % datetime.date.today().year
source_tree = 'https://git.openstack.org/cgit/openstack/taskflow/tree'
# If true, '()' will be appended to :func: etc. cross-reference text.
add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
add_module_names = True
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# Prefixes that are ignored for sorting the Python module index
modindex_common_prefix = ['taskflow.']
# Shortened external links.
extlinks = {
'example': (source_tree + '/taskflow/examples/%s.py', ''),
'pybug': ('http://bugs.python.org/issue%s', ''),
}
# -- Options for HTML output --------------------------------------------------
# The theme to use for HTML and HTML Help pages. Major themes that come with
# Sphinx are currently 'default' and 'sphinxdoc'.
# html_theme_path = ["."]
html_theme = 'openstackdocs'
# html_static_path = ['static']
# Output file base name for HTML help builder.
htmlhelp_basename = '%sdoc' % project
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass
# [howto/manual]).
latex_documents = [
('index',
'%s.tex' % project,
'%s Documentation' % project,
'OpenStack Foundation', 'manual'),
]
# -- Options for autoddoc ----------------------------------------------------
# Keep source order
autodoc_member_order = 'bysource'
# Always include members
autodoc_default_flags = ['members', 'show-inheritance']

View File

@ -1,29 +0,0 @@
==========
TaskFlow
==========
*TaskFlow is a Python library that helps to make task execution easy,
consistent and reliable.* [#f1]_
.. note::
If you are just getting started or looking for an overview please
visit: http://wiki.openstack.org/wiki/TaskFlow which provides better
introductory material, description of high level goals and related content.
.. toctree::
:maxdepth: 2
user/index
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
.. [#f1] It should be noted that even though it is designed with OpenStack
integration in mind, and that is where most of its *current*
integration is it aims to be generally usable and useful in any
project.

View File

@ -1,15 +0,0 @@
{% extends "!layout.html" %}
{% block sidebarrel %}
<h3>{{ _('Navigation')}}</h3>
<ul>
{% if pagename != "index" %}
<li><a href="{{ pathto(master_doc) }}">{{ _('Table Of Contents') }}</a></li>
{% endif %}
{% if next %}
<li><a href="{{ next.link|e }}" title="{{ _('next chapter') }}">{{ _('Next topic') }}: {{ next.title }}</a></li>
{% endif %}
{% if prev %}
<li><a href="{{ prev.link|e }}" title="{{ _('previous chapter') }}">{{ _('Previous topic') }}: {{ prev.title }}</a></li>
{% endif %}
</ul>
{% endblock %}

View File

@ -1,428 +0,0 @@
=====================
Arguments and results
=====================
.. |task.execute| replace:: :py:meth:`~taskflow.atom.Atom.execute`
.. |task.revert| replace:: :py:meth:`~taskflow.atom.Atom.revert`
.. |retry.execute| replace:: :py:meth:`~taskflow.retry.Retry.execute`
.. |retry.revert| replace:: :py:meth:`~taskflow.retry.Retry.revert`
.. |Retry| replace:: :py:class:`~taskflow.retry.Retry`
.. |Task| replace:: :py:class:`Task <taskflow.task.Task>`
In TaskFlow, all flow and task state goes to (potentially persistent) storage
(see :doc:`persistence <persistence>` for more details). That includes all the
information that :doc:`atoms <atoms>` (e.g. tasks, retry objects...) in the
workflow need when they are executed, and all the information task/retry
produces (via serializable results). A developer who implements tasks/retries
or flows can specify what arguments a task/retry accepts and what result it
returns in several ways. This document will help you understand what those ways
are and how to use those ways to accomplish your desired usage pattern.
.. glossary::
Task/retry arguments
Set of names of task/retry arguments available as the ``requires``
and/or ``optional`` property of the task/retry instance. When a task or
retry object is about to be executed values with these names are
retrieved from storage and passed to the ``execute`` method of the
task/retry. If any names in the ``requires`` property cannot be
found in storage, an exception will be thrown. Any names in the
``optional`` property that cannot be found are ignored.
Task/retry results
Set of names of task/retry results (what task/retry provides) available
as ``provides`` property of task or retry instance. After a task/retry
finishes successfully, its result(s) (what the ``execute`` method
returns) are available by these names from storage (see examples
below).
.. testsetup::
from taskflow import task
Arguments specification
=======================
There are different ways to specify the task argument ``requires`` set.
Arguments inference
-------------------
Task/retry arguments can be inferred from arguments of the |task.execute|
method of a task (or the |retry.execute| of a retry object).
.. doctest::
>>> class MyTask(task.Task):
... def execute(self, spam, eggs, bacon=None):
... return spam + eggs
...
>>> sorted(MyTask().requires)
['eggs', 'spam']
>>> sorted(MyTask().optional)
['bacon']
Inference from the method signature is the ''simplest'' way to specify
arguments. Special arguments like ``self``, ``*args`` and ``**kwargs`` are
ignored during inference (as these names have special meaning/usage in python).
.. doctest::
>>> class UniTask(task.Task):
... def execute(self, *args, **kwargs):
... pass
...
>>> sorted(UniTask().requires)
[]
.. make vim sphinx highlighter* happy**
Rebinding
---------
**Why:** There are cases when the value you want to pass to a task/retry is
stored with a name other than the corresponding arguments name. That's when the
``rebind`` constructor parameter comes in handy. Using it the flow author
can instruct the engine to fetch a value from storage by one name, but pass it
to a tasks/retries ``execute`` method with another name. There are two possible
ways of accomplishing this.
The first is to pass a dictionary that maps the argument name to the name
of a saved value.
For example, if you have task::
class SpawnVMTask(task.Task):
def execute(self, vm_name, vm_image_id, **kwargs):
pass # TODO(imelnikov): use parameters to spawn vm
and you saved ``'vm_name'`` with ``'name'`` key in storage, you can spawn a vm
with such ``'name'`` like this::
SpawnVMTask(rebind={'vm_name': 'name'})
The second way is to pass a tuple/list/dict of argument names. The length of
the tuple/list/dict should not be less then number of required parameters.
For example, you can achieve the same effect as the previous example with::
SpawnVMTask(rebind_args=('name', 'vm_image_id'))
This is equivalent to a more elaborate::
SpawnVMTask(rebind=dict(vm_name='name',
vm_image_id='vm_image_id'))
In both cases, if your task (or retry) accepts arbitrary arguments
with the ``**kwargs`` construct, you can specify extra arguments.
::
SpawnVMTask(rebind=('name', 'vm_image_id', 'admin_key_name'))
When such task is about to be executed, ``name``, ``vm_image_id`` and
``admin_key_name`` values are fetched from storage and value from ``name`` is
passed to |task.execute| method as ``vm_name``, value from ``vm_image_id`` is
passed as ``vm_image_id``, and value from ``admin_key_name`` is passed as
``admin_key_name`` parameter in ``kwargs``.
Manually specifying requirements
--------------------------------
**Why:** It is often useful to manually specify the requirements of a task,
either by a task author or by the flow author (allowing the flow author to
override the task requirements).
To accomplish this when creating your task use the constructor to specify
manual requirements. Those manual requirements (if they are not functional
arguments) will appear in the ``kwargs`` of the |task.execute| method.
.. doctest::
>>> class Cat(task.Task):
... def __init__(self, **kwargs):
... if 'requires' not in kwargs:
... kwargs['requires'] = ("food", "milk")
... super(Cat, self).__init__(**kwargs)
... def execute(self, food, **kwargs):
... pass
...
>>> cat = Cat()
>>> sorted(cat.requires)
['food', 'milk']
.. make vim sphinx highlighter happy**
When constructing a task instance the flow author can also add more
requirements if desired. Those manual requirements (if they are not functional
arguments) will appear in the ``kwargs`` parameter of the |task.execute|
method.
.. doctest::
>>> class Dog(task.Task):
... def execute(self, food, **kwargs):
... pass
>>> dog = Dog(requires=("water", "grass"))
>>> sorted(dog.requires)
['food', 'grass', 'water']
.. make vim sphinx highlighter happy**
If the flow author desires she can turn the argument inference off and override
requirements manually. Use this at your own **risk** as you must be careful to
avoid invalid argument mappings.
.. doctest::
>>> class Bird(task.Task):
... def execute(self, food, **kwargs):
... pass
>>> bird = Bird(requires=("food", "water", "grass"), auto_extract=False)
>>> sorted(bird.requires)
['food', 'grass', 'water']
.. make vim sphinx highlighter happy**
Results specification
=====================
In python, function results are not named, so we can not infer what a
task/retry returns. This is important since the complete result (what the
task |task.execute| or retry |retry.execute| method returns) is saved
in (potentially persistent) storage, and it is typically (but not always)
desirable to make those results accessible to others. To accomplish this
the task/retry specifies names of those values via its ``provides`` constructor
parameter or by its default provides attribute.
Examples
--------
Returning one value
+++++++++++++++++++
If task returns just one value, ``provides`` should be string -- the
name of the value.
.. doctest::
>>> class TheAnswerReturningTask(task.Task):
... def execute(self):
... return 42
...
>>> sorted(TheAnswerReturningTask(provides='the_answer').provides)
['the_answer']
Returning a tuple
+++++++++++++++++
For a task that returns several values, one option (as usual in python) is to
return those values via a ``tuple``.
::
class BitsAndPiecesTask(task.Task):
def execute(self):
return 'BITs', 'PIECEs'
Then, you can give the value individual names, by passing a tuple or list as
``provides`` parameter:
::
BitsAndPiecesTask(provides=('bits', 'pieces'))
After such task is executed, you (and the engine, which is useful for other
tasks) will be able to get those elements from storage by name:
::
>>> storage.fetch('bits')
'BITs'
>>> storage.fetch('pieces')
'PIECEs'
Provides argument can be shorter then the actual tuple returned by a task --
then extra values are ignored (but, as expected, **all** those values are saved
and passed to the task |task.revert| or retry |retry.revert| method).
.. note::
Provides arguments tuple can also be longer then the actual tuple returned
by task -- when this happens the extra parameters are left undefined: a
warning is printed to logs and if use of such parameter is attempted a
:py:class:`~taskflow.exceptions.NotFound` exception is raised.
Returning a dictionary
++++++++++++++++++++++
Another option is to return several values as a dictionary (aka a ``dict``).
::
class BitsAndPiecesTask(task.Task):
def execute(self):
return {
'bits': 'BITs',
'pieces': 'PIECEs'
}
TaskFlow expects that a dict will be returned if ``provides`` argument is a
``set``:
::
BitsAndPiecesTask(provides=set(['bits', 'pieces']))
After such task executes, you (and the engine, which is useful for other tasks)
will be able to get elements from storage by name:
::
>>> storage.fetch('bits')
'BITs'
>>> storage.fetch('pieces')
'PIECEs'
.. note::
If some items from the dict returned by the task are not present in the
provides arguments -- then extra values are ignored (but, of course, saved
and passed to the |task.revert| method). If the provides argument has some
items not present in the actual dict returned by the task -- then extra
parameters are left undefined: a warning is printed to logs and if use of
such parameter is attempted a :py:class:`~taskflow.exceptions.NotFound`
exception is raised.
Default provides
++++++++++++++++
As mentioned above, the default base class provides nothing, which means
results are not accessible to other tasks/retries in the flow.
The author can override this and specify default value for provides using
the ``default_provides`` class/instance variable:
::
class BitsAndPiecesTask(task.Task):
default_provides = ('bits', 'pieces')
def execute(self):
return 'BITs', 'PIECEs'
Of course, the flow author can override this to change names if needed:
::
BitsAndPiecesTask(provides=('b', 'p'))
or to change structure -- e.g. this instance will make tuple accessible
to other tasks by name ``'bnp'``:
::
BitsAndPiecesTask(provides='bnp')
or the flow author may want to return default behavior and hide the results of
the task from other tasks in the flow (e.g. to avoid naming conflicts):
::
BitsAndPiecesTask(provides=())
Revert arguments
================
To revert a task the :doc:`engine <engines>` calls the tasks
|task.revert| method. This method should accept the same arguments
as the |task.execute| method of the task and one more special keyword
argument, named ``result``.
For ``result`` value, two cases are possible:
* If the task is being reverted because it failed (an exception was raised
from its |task.execute| method), the ``result`` value is an instance of a
:py:class:`~taskflow.types.failure.Failure` object that holds the exception
information.
* If the task is being reverted because some other task failed, and this task
finished successfully, ``result`` value is the result fetched from storage:
ie, what the |task.execute| method returned.
All other arguments are fetched from storage in the same way it is done for
|task.execute| method.
To determine if a task failed you can check whether ``result`` is instance of
:py:class:`~taskflow.types.failure.Failure`::
from taskflow.types import failure
class RevertingTask(task.Task):
def execute(self, spam, eggs):
return do_something(spam, eggs)
def revert(self, result, spam, eggs):
if isinstance(result, failure.Failure):
print("This task failed, exception: %s"
% result.exception_str)
else:
print("do_something returned %r" % result)
If this task failed (ie ``do_something`` raised an exception) it will print
``"This task failed, exception:"`` and a exception message on revert. If this
task finished successfully, it will print ``"do_something returned"`` and a
representation of the ``do_something`` result.
Retry arguments
===============
A |Retry| controller works with arguments in the same way as a |Task|. But it
has an additional parameter ``'history'`` that is itself a
:py:class:`~taskflow.retry.History` object that contains what failed over all
the engines attempts (aka the outcomes). The history object can be
viewed as a tuple that contains a result of the previous retries run and a
table/dict where each key is a failed atoms name and each value is
a :py:class:`~taskflow.types.failure.Failure` object.
Consider the following implementation::
class MyRetry(retry.Retry):
default_provides = 'value'
def on_failure(self, history, *args, **kwargs):
print(list(history))
return RETRY
def execute(self, history, *args, **kwargs):
print(list(history))
return 5
def revert(self, history, *args, **kwargs):
print(list(history))
Imagine the above retry had returned a value ``'5'`` and then some task ``'A'``
failed with some exception. In this case ``on_failure`` method will receive
the following history (printed as a list)::
[('5', {'A': failure.Failure()})]
At this point (since the implementation returned ``RETRY``) the
|retry.execute| method will be called again and it will receive the same
history and it can then return a value that subsequent tasks can use to alter
their behavior.
If instead the |retry.execute| method itself raises an exception,
the |retry.revert| method of the implementation will be called and
a :py:class:`~taskflow.types.failure.Failure` object will be present in the
history object instead of the typical result.
.. note::
After a |Retry| has been reverted, the objects history will be cleaned.

View File

@ -1,221 +0,0 @@
------------------------
Atoms, tasks and retries
------------------------
Atom
====
An :py:class:`atom <taskflow.atom.Atom>` is the smallest unit in TaskFlow which
acts as the base for other classes (its naming was inspired from the
similarities between this type and `atoms`_ in the physical world). Atoms
have a name and may have a version. An atom is expected to name desired input
values (requirements) and name outputs (provided values).
.. note::
For more details about atom inputs and outputs please visit
:doc:`arguments and results <arguments_and_results>`.
.. automodule:: taskflow.atom
.. _atoms: http://en.wikipedia.org/wiki/Atom
Task
=====
A :py:class:`task <taskflow.task.Task>` (derived from an atom) is a
unit of work that can have an execute & rollback sequence associated with
it (they are *nearly* analogous to functions). Your task objects should all
derive from :py:class:`~taskflow.task.Task` which defines what a task must
provide in terms of properties and methods.
**For example:**
.. image:: img/tasks.png
:width: 525px
:align: left
:alt: Task outline.
Currently the following *provided* types of task subclasses are:
* :py:class:`~taskflow.task.Task`: useful for inheriting from and creating your
own subclasses.
* :py:class:`~taskflow.task.FunctorTask`: useful for wrapping existing
functions into task objects.
.. note::
:py:class:`~taskflow.task.FunctorTask` task types can not currently be used
with the :doc:`worker based engine <workers>` due to the fact that
arbitrary functions can not be guaranteed to be correctly
located (especially if they are lambda or anonymous functions) on the
worker nodes.
Retry
=====
A :py:class:`retry <taskflow.retry.Retry>` (derived from an atom) is a special
unit of work that handles errors, controls flow execution and can (for
example) retry other atoms with other parameters if needed. When an associated
atom fails, these retry units are *consulted* to determine what the resolution
*strategy* should be. The goal is that with this consultation the retry atom
will suggest a *strategy* for getting around the failure (perhaps by retrying,
reverting a single atom, or reverting everything contained in the retries
associated `scope`_).
Currently derivatives of the :py:class:`retry <taskflow.retry.Retry>` base
class must provide a :py:func:`~taskflow.retry.Retry.on_failure` method to
determine how a failure should be handled. The current enumeration(s) that can
be returned from the :py:func:`~taskflow.retry.Retry.on_failure` method
are defined in an enumeration class described here:
.. autoclass:: taskflow.retry.Decision
To aid in the reconciliation process the
:py:class:`retry <taskflow.retry.Retry>` base class also mandates
:py:func:`~taskflow.retry.Retry.execute`
and :py:func:`~taskflow.retry.Retry.revert` methods (although subclasses
are allowed to define these methods as no-ops) that can be used by a retry
atom to interact with the runtime execution model (for example, to track the
number of times it has been called which is useful for
the :py:class:`~taskflow.retry.ForEach` retry subclass).
To avoid recreating common retry patterns the following provided retry
subclasses are provided:
* :py:class:`~taskflow.retry.AlwaysRevert`: Always reverts subflow.
* :py:class:`~taskflow.retry.AlwaysRevertAll`: Always reverts the whole flow.
* :py:class:`~taskflow.retry.Times`: Retries subflow given number of times.
* :py:class:`~taskflow.retry.ForEach`: Allows for providing different values
to subflow atoms each time a failure occurs (making it possibly to resolve
the failure by altering subflow atoms inputs).
* :py:class:`~taskflow.retry.ParameterizedForEach`: Same as
:py:class:`~taskflow.retry.ForEach` but extracts values from storage
instead of the :py:class:`~taskflow.retry.ForEach` constructor.
.. _scope: http://en.wikipedia.org/wiki/Scope_%28computer_science%29
.. note::
They are *similar* to exception handlers but are made to be *more* capable
due to their ability to *dynamically* choose a reconciliation strategy,
which allows for these atoms to influence subsequent execution(s) and the
inputs any associated atoms require.
Area of influence
-----------------
Each retry atom is associated with a flow and it can *influence* how the
atoms (or nested flows) contained in that flow retry or revert (using
the previously mentioned patterns and decision enumerations):
*For example:*
.. image:: img/area_of_influence.svg
:width: 325px
:align: left
:alt: Retry area of influence
In this diagram retry controller (1) will be consulted if task ``A``, ``B``
or ``C`` fail and retry controller (2) decides to delegate its retry decision
to retry controller (1). If retry controller (2) does **not** decide to
delegate its retry decision to retry controller (1) then retry
controller (1) will be oblivious of any decisions. If any of
task ``1``, ``2`` or ``3`` fail then only retry controller (1) will be
consulted to determine the strategy/pattern to apply to resolve there
associated failure.
Usage examples
--------------
.. testsetup::
import taskflow
from taskflow import task
from taskflow import retry
from taskflow.patterns import linear_flow
from taskflow import engines
.. doctest::
>>> class EchoTask(task.Task):
... def execute(self, *args, **kwargs):
... print(self.name)
... print(args)
... print(kwargs)
...
>>> flow = linear_flow.Flow('f1').add(
... EchoTask('t1'),
... linear_flow.Flow('f2', retry=retry.ForEach(values=['a', 'b', 'c'], name='r1', provides='value')).add(
... EchoTask('t2'),
... EchoTask('t3', requires='value')),
... EchoTask('t4'))
In this example the flow ``f2`` has a retry controller ``r1``, that is an
instance of the default retry controller :py:class:`~taskflow.retry.ForEach`,
it accepts a collection of values and iterates over this collection when
each failure occurs. On each run :py:class:`~taskflow.retry.ForEach` retry
returns the next value from the collection and stops retrying a subflow if
there are no more values left in the collection. For example if tasks ``t2`` or
``t3`` fail, then the flow ``f2`` will be reverted and retry ``r1`` will retry
it with the next value from the given collection ``['a', 'b', 'c']``. But if
the task ``t1`` or the task ``t4`` fails, ``r1`` won't retry a flow, because
tasks ``t1`` and ``t4`` are in the flow ``f1`` and don't depend on
retry ``r1`` (so they will not *consult* ``r1`` on failure).
.. doctest::
>>> class SendMessage(task.Task):
... def execute(self, message):
... print("Sending message: %s" % message)
...
>>> flow = linear_flow.Flow('send_message', retry=retry.Times(5)).add(
... SendMessage('sender'))
In this example the ``send_message`` flow will try to execute the
``SendMessage`` five times when it fails. When it fails for the sixth time (if
it does) the task will be asked to ``REVERT`` (in this example task reverting
does not cause anything to happen but in other use cases it could).
.. doctest::
>>> class ConnectToServer(task.Task):
... def execute(self, ip):
... print("Connecting to %s" % ip)
...
>>> server_ips = ['192.168.1.1', '192.168.1.2', '192.168.1.3' ]
>>> flow = linear_flow.Flow('send_message',
... retry=retry.ParameterizedForEach(rebind={'values': 'server_ips'},
... provides='ip')).add(
... ConnectToServer(requires=['ip']))
In this example the flow tries to connect a server using a list (a tuple
can also be used) of possible IP addresses. Each time the retry will return
one IP from the list. In case of a failure it will return the next one until
it reaches the last one, then the flow will be reverted.
Interfaces
==========
.. automodule:: taskflow.task
.. autoclass:: taskflow.retry.Retry
.. autoclass:: taskflow.retry.History
.. autoclass:: taskflow.retry.AlwaysRevert
.. autoclass:: taskflow.retry.AlwaysRevertAll
.. autoclass:: taskflow.retry.Times
.. autoclass:: taskflow.retry.ForEach
.. autoclass:: taskflow.retry.ParameterizedForEach
Hierarchy
=========
.. inheritance-diagram::
taskflow.atom
taskflow.task
taskflow.retry.Retry
taskflow.retry.AlwaysRevert
taskflow.retry.AlwaysRevertAll
taskflow.retry.Times
taskflow.retry.ForEach
taskflow.retry.ParameterizedForEach
:parts: 1

View File

@ -1,95 +0,0 @@
----------
Conductors
----------
.. image:: img/conductor.png
:width: 97px
:alt: Conductor
Overview
========
Conductors provide a mechanism that unifies the various
concepts under a single easy to use (as plug-and-play as we can make it)
construct.
They are responsible for the following:
* Interacting with :doc:`jobboards <jobs>` (examining and claiming
:doc:`jobs <jobs>`).
* Creating :doc:`engines <engines>` from the claimed jobs (using
:ref:`factories <resumption factories>` to reconstruct the contained
tasks and flows to be executed).
* Dispatching the engine using the provided :doc:`persistence <persistence>`
layer and engine configuration.
* Completing or abandoning the claimed :doc:`job <jobs>` (depending on
dispatching and execution outcome).
* *Rinse and repeat*.
.. note::
They are inspired by and have similar responsibilities
as `railroad conductors`_ or `musical conductors`_.
Considerations
==============
Some usage considerations should be used when using a conductor to make sure
it's used in a safe and reliable manner. Eventually we hope to make these
non-issues but for now they are worth mentioning.
Endless cycling
---------------
**What:** Jobs that fail (due to some type of internal error) on one conductor
will be abandoned by that conductor and then another conductor may experience
those same errors and abandon it (and repeat). This will create a job
abandonment cycle that will continue for as long as the job exists in an
claimable state.
**Example:**
.. image:: img/conductor_cycle.png
:scale: 70%
:alt: Conductor cycling
**Alleviate by:**
#. Forcefully delete jobs that have been failing continuously after a given
number of conductor attempts. This can be either done manually or
automatically via scripts (or other associated monitoring) or via
the jobboards :py:func:`~taskflow.jobs.base.JobBoard.trash` method.
#. Resolve the internal error's cause (storage backend failure, other...).
Interfaces
==========
.. automodule:: taskflow.conductors.base
.. automodule:: taskflow.conductors.backends
.. automodule:: taskflow.conductors.backends.impl_executor
Implementations
===============
Blocking
--------
.. automodule:: taskflow.conductors.backends.impl_blocking
Non-blocking
------------
.. automodule:: taskflow.conductors.backends.impl_nonblocking
Hierarchy
=========
.. inheritance-diagram::
taskflow.conductors.base
taskflow.conductors.backends.impl_blocking
taskflow.conductors.backends.impl_nonblocking
taskflow.conductors.backends.impl_executor
:parts: 1
.. _musical conductors: http://en.wikipedia.org/wiki/Conducting
.. _railroad conductors: http://en.wikipedia.org/wiki/Conductor_%28transportation%29

View File

@ -1,475 +0,0 @@
-------
Engines
-------
Overview
========
Engines are what **really** runs your atoms.
An *engine* takes a flow structure (described by :doc:`patterns <patterns>`)
and uses it to decide which :doc:`atom <atoms>` to run and when.
TaskFlow provides different implementations of engines. Some may be easier to
use (ie, require no additional infrastructure setup) and understand; others
might require more complicated setup but provide better scalability. The idea
and *ideal* is that deployers or developers of a service that use TaskFlow can
select an engine that suites their setup best without modifying the code of
said service.
.. note::
Engines usually have different capabilities and configuration, but all of
them **must** implement the same interface and preserve the semantics of
patterns (e.g. parts of a :py:class:`.linear_flow.Flow`
are run one after another, in order, even if the selected
engine is *capable* of running tasks in parallel).
Why they exist
--------------
An engine being *the* core component which actually makes your flows progress
is likely a new concept for many programmers so let's describe how it operates
in more depth and some of the reasoning behind why it exists. This will
hopefully make it more clear on their value add to the TaskFlow library user.
First though let us discuss something most are familiar already with; the
difference between `declarative`_ and `imperative`_ programming models. The
imperative model involves establishing statements that accomplish a programs
action (likely using conditionals and such other language features to do this).
This kind of program embeds the *how* to accomplish a goal while also defining
*what* the goal actually is (and the state of this is maintained in memory or
on the stack while these statements execute). In contrast there is the
declarative model which instead of combining the *how* to accomplish a goal
along side the *what* is to be accomplished splits these two into only
declaring what the intended goal is and not the *how*. In TaskFlow terminology
the *what* is the structure of your flows and the tasks and other atoms you
have inside those flows, but the *how* is not defined (the line becomes blurred
since tasks themselves contain imperative code, but for now consider a task as
more of a *pure* function that executes, reverts and may require inputs and
provide outputs). This is where engines get involved; they do the execution of
the *what* defined via :doc:`atoms <atoms>`, tasks, flows and the relationships
defined there-in and execute these in a well-defined manner (and the engine is
responsible for any state manipulation instead).
This mix of imperative and declarative (with a stronger emphasis on the
declarative model) allows for the following functionality to become possible:
* Enhancing reliability: Decoupling of state alterations from what should be
accomplished allows for a *natural* way of resuming by allowing the engine to
track the current state and know at which point a workflow is in and how to
get back into that state when resumption occurs.
* Enhancing scalability: When an engine is responsible for executing your
desired work it becomes possible to alter the *how* in the future by creating
new types of execution backends (for example the `worker`_ model which does
not execute locally). Without the decoupling of the *what* and the *how* it
is not possible to provide such a feature (since by the very nature of that
coupling this kind of functionality is inherently very hard to provide).
* Enhancing consistency: Since the engine is responsible for executing atoms
and the associated workflow, it can be one (if not the only) of the primary
entities that is working to keep the execution model in a consistent state.
Coupled with atoms which *should* be immutable and have have limited (if any)
internal state the ability to reason about and obtain consistency can be
vastly improved.
* With future features around locking (using `tooz`_ to help) engines can
also help ensure that resources being accessed by tasks are reliably
obtained and mutated on. This will help ensure that other processes,
threads, or other types of entities are also not executing tasks that
manipulate those same resources (further increasing consistency).
Of course these kind of features can come with some drawbacks:
* The downside of decoupling the *how* and the *what* is that the imperative
model where functions control & manipulate state must start to be shifted
away from (and this is likely a mindset change for programmers used to the
imperative model). We have worked to make this less of a concern by creating
and encouraging the usage of :doc:`persistence <persistence>`, to help make
it possible to have state and transfer that state via a argument input and
output mechanism.
* Depending on how much imperative code exists (and state inside that code)
there *may* be *significant* rework of that code and converting or
refactoring it to these new concepts. We have tried to help here by allowing
you to have tasks that internally use regular python code (and internally can
be written in an imperative style) as well as by providing
:doc:`examples <examples>` that show how to use these concepts.
* Another one of the downsides of decoupling the *what* from the *how* is that
it may become harder to use traditional techniques to debug failures
(especially if remote workers are involved). We try to help here by making it
easy to track, monitor and introspect the actions & state changes that are
occurring inside an engine (see :doc:`notifications <notifications>` for how
to use some of these capabilities).
.. _declarative: http://en.wikipedia.org/wiki/Declarative_programming
.. _imperative: http://en.wikipedia.org/wiki/Imperative_programming
.. _tooz: https://github.com/openstack/tooz
Creating
========
.. _creating engines:
All engines are mere classes that implement the same interface, and of course
it is possible to import them and create instances just like with any classes
in Python. But the easier (and recommended) way for creating an engine is using
the engine helper functions. All of these functions are imported into the
``taskflow.engines`` module namespace, so the typical usage of these functions
might look like::
from taskflow import engines
...
flow = make_flow()
eng = engines.load(flow, engine='serial', backend=my_persistence_conf)
eng.run()
...
.. automodule:: taskflow.engines.helpers
Usage
=====
To select which engine to use and pass parameters to an engine you should use
the ``engine`` parameter any engine helper function accepts and for any engine
specific options use the ``kwargs`` parameter.
Types
=====
Serial
------
**Engine type**: ``'serial'``
Runs all tasks on a single thread -- the same thread
:py:meth:`~taskflow.engines.base.Engine.run` is called from.
.. note::
This engine is used by **default**.
.. tip::
If eventlet is used then this engine will not block other threads
from running as eventlet automatically creates a implicit co-routine
system (using greenthreads and monkey patching). See
`eventlet <http://eventlet.net/>`_ and
`greenlet <http://greenlet.readthedocs.org/>`_ for more details.
Parallel
--------
**Engine type**: ``'parallel'``
A parallel engine schedules tasks onto different threads/processes to allow for
running non-dependent tasks simultaneously. See the documentation of
:py:class:`~taskflow.engines.action_engine.engine.ParallelActionEngine` for
supported arguments that can be used to construct a parallel engine that runs
using your desired execution model.
.. tip::
Sharing an executor between engine instances provides better
scalability by reducing thread/process creation and teardown as well as by
reusing existing pools (which is a good practice in general).
.. warning::
Running tasks with a `process pool executor`_ is **experimentally**
supported. This is mainly due to the `futures backport`_ and
the `multiprocessing`_ module that exist in older versions of python not
being as up to date (with important fixes such as :pybug:`4892`,
:pybug:`6721`, :pybug:`9205`, :pybug:`16284`,
:pybug:`22393` and others...) as the most recent python version (which
themselves have a variety of ongoing/recent bugs).
Workers
-------
.. _worker:
**Engine type**: ``'worker-based'`` or ``'workers'``
.. note:: Since this engine is significantly more complicated (and
different) then the others we thought it appropriate to devote a
whole documentation :doc:`section <workers>` to it.
How they run
============
To provide a peek into the general process that an engine goes through when
running lets break it apart a little and describe what one of the engine types
does while executing (for this we will look into the
:py:class:`~taskflow.engines.action_engine.engine.ActionEngine` engine type).
Creation
--------
The first thing that occurs is that the user creates an engine for a given
flow, providing a flow detail (where results will be saved into a provided
:doc:`persistence <persistence>` backend). This is typically accomplished via
the methods described above in `creating engines`_. The engine at this point
now will have references to your flow and backends and other internal variables
are setup.
Compiling
---------
During this stage (see :py:func:`~taskflow.engines.base.Engine.compile`) the
flow will be converted into an internal graph representation using a
compiler (the default implementation for patterns is the
:py:class:`~taskflow.engines.action_engine.compiler.PatternCompiler`). This
class compiles/converts the flow objects and contained atoms into a
`networkx`_ directed graph (and tree structure) that contains the equivalent
atoms defined in the flow and any nested flows & atoms as well as the
constraints that are created by the application of the different flow
patterns. This graph (and tree) are what will be analyzed & traversed during
the engines execution. At this point a few helper object are also created and
saved to internal engine variables (these object help in execution of
atoms, analyzing the graph and performing other internal engine
activities). At the finishing of this stage a
:py:class:`~taskflow.engines.action_engine.runtime.Runtime` object is created
which contains references to all needed runtime components and its
:py:func:`~taskflow.engines.action_engine.runtime.Runtime.compile` is called
to compile a cache of frequently used execution helper objects.
Preparation
-----------
This stage (see :py:func:`~taskflow.engines.base.Engine.prepare`) starts by
setting up the storage needed for all atoms in the compiled graph, ensuring
that corresponding :py:class:`~taskflow.persistence.models.AtomDetail` (or
subclass of) objects are created for each node in the graph.
Validation
----------
This stage (see :py:func:`~taskflow.engines.base.Engine.validate`) performs
any final validation of the compiled (and now storage prepared) engine. It
compares the requirements that are needed to start execution and
what is currently provided or will be produced in the future. If there are
*any* atom requirements that are not satisfied (no known current provider or
future producer is found) then execution will **not** be allowed to continue.
Execution
---------
The graph (and helper objects) previously created are now used for guiding
further execution (see :py:func:`~taskflow.engines.base.Engine.run`). The
flow is put into the ``RUNNING`` :doc:`state <states>` and a
:py:class:`~taskflow.engines.action_engine.builder.MachineBuilder` state
machine object and runner object are built (using the `automaton`_ library).
That machine and associated runner then starts to take over and begins going
through the stages listed below (for a more visual diagram/representation see
the :ref:`engine state diagram <engine states>`).
.. note::
The engine will respect the constraints imposed by the flow. For example,
if an engine is executing a :py:class:`~taskflow.patterns.linear_flow.Flow`
then it is constrained by the dependency graph which is linear in this
case, and hence using a parallel engine may not yield any benefits if one
is looking for concurrency.
Resumption
^^^^^^^^^^
One of the first stages is to analyze the :doc:`state <states>` of the tasks in
the graph, determining which ones have failed, which one were previously
running and determining what the intention of that task should now be
(typically an intention can be that it should ``REVERT``, or that it should
``EXECUTE`` or that it should be ``IGNORED``). This intention is determined by
analyzing the current state of the task; which is determined by looking at the
state in the task detail object for that task and analyzing edges of the graph
for things like retry atom which can influence what a tasks intention should be
(this is aided by the usage of the
:py:class:`~taskflow.engines.action_engine.selector.Selector` helper
object which was designed to provide helper methods for this analysis). Once
these intentions are determined and associated with each task (the intention is
also stored in the :py:class:`~taskflow.persistence.models.AtomDetail` object)
the :ref:`scheduling <scheduling>` stage starts.
.. _scheduling:
Scheduling
^^^^^^^^^^
This stage selects which atoms are eligible to run by using a
:py:class:`~taskflow.engines.action_engine.scheduler.Scheduler` implementation
(the default implementation looks at their intention, checking if predecessor
atoms have ran and so-on, using a
:py:class:`~taskflow.engines.action_engine.selector.Selector` helper
object as needed) and submits those atoms to a previously provided compatible
`executor`_ for asynchronous execution. This
:py:class:`~taskflow.engines.action_engine.scheduler.Scheduler` will return a
`future`_ object for each atom scheduled; all of which are collected into a
list of not done futures. This will end the initial round of scheduling and at
this point the engine enters the :ref:`waiting <waiting>` stage.
.. _waiting:
Waiting
^^^^^^^
In this stage the engine waits for any of the future objects previously
submitted to complete. Once one of the future objects completes (or fails) that
atoms result will be examined and finalized using a
:py:class:`~taskflow.engines.action_engine.completer.Completer` implementation.
It typically will persist results to a provided persistence backend (saved
into the corresponding :py:class:`~taskflow.persistence.models.AtomDetail`
and :py:class:`~taskflow.persistence.models.FlowDetail` objects via the
:py:class:`~taskflow.storage.Storage` helper) and reflect
the new state of the atom. At this point what typically happens falls into two
categories, one for if that atom failed and one for if it did not. If the atom
failed it may be set to a new intention such as ``RETRY`` or
``REVERT`` (other atoms that were predecessors of this failing atom may also
have there intention altered). Once this intention adjustment has happened a
new round of :ref:`scheduling <scheduling>` occurs and this process repeats
until the engine succeeds or fails (if the process running the engine dies the
above stages will be restarted and resuming will occur).
.. note::
If the engine is suspended while the engine is going through the above
stages this will stop any further scheduling stages from occurring and
all currently executing work will be allowed to finish (see
:ref:`suspension <suspension>`).
Finishing
---------
At this point the machine (and runner) that was built using the
:py:class:`~taskflow.engines.action_engine.builder.MachineBuilder` class has
now finished successfully, failed, or the execution was suspended. Depending on
which one of these occurs will cause the flow to enter a new state (typically
one of ``FAILURE``, ``SUSPENDED``, ``SUCCESS`` or ``REVERTED``).
:doc:`Notifications <notifications>` will be sent out about this final state
change (other state changes also send out notifications) and any failures that
occurred will be reraised (the failure objects are wrapped exceptions). If no
failures have occurred then the engine will have finished and if so desired the
:doc:`persistence <persistence>` can be used to cleanup any details that were
saved for this execution.
Special cases
=============
.. _suspension:
Suspension
----------
Each engine implements a :py:func:`~taskflow.engines.base.Engine.suspend`
method that can be used to *externally* (or in the future *internally*) request
that the engine stop :ref:`scheduling <scheduling>` new work. By default what
this performs is a transition of the flow state from ``RUNNING`` into a
``SUSPENDING`` state (which will later transition into a ``SUSPENDED`` state).
Since an engine may be remotely executing atoms (or locally executing them)
and there is currently no preemption what occurs is that the engines
:py:class:`~taskflow.engines.action_engine.builder.MachineBuilder` state
machine will detect this transition into ``SUSPENDING`` has occurred and the
state machine will avoid scheduling new work (it will though let active work
continue). After the current work has finished the engine will
transition from ``SUSPENDING`` into ``SUSPENDED`` and return from its
:py:func:`~taskflow.engines.base.Engine.run` method.
.. note::
When :py:func:`~taskflow.engines.base.Engine.run` is returned from at that
point there *may* (but does not have to be, depending on what was active
when :py:func:`~taskflow.engines.base.Engine.suspend` was called) be
unfinished work in the flow that was not finished (but which can be
resumed at a later point in time).
Scoping
=======
During creation of flows it is also important to understand the lookup
strategy (also typically known as `scope`_ resolution) that the engine you
are using will internally use. For example when a task ``A`` provides
result 'a' and a task ``B`` after ``A`` provides a different result 'a' and a
task ``C`` after ``A`` and after ``B`` requires 'a' to run, which one will
be selected?
Default strategy
----------------
When an engine is executing it internally interacts with the
:py:class:`~taskflow.storage.Storage` class
and that class interacts with the a
:py:class:`~taskflow.engines.action_engine.scopes.ScopeWalker` instance
and the :py:class:`~taskflow.storage.Storage` class uses the following
lookup order to find (or fail) a atoms requirement lookup/request:
#. Transient injected atom specific arguments.
#. Non-transient injected atom specific arguments.
#. Transient injected arguments (flow specific).
#. Non-transient injected arguments (flow specific).
#. First scope visited provider that produces the named result; note that
if multiple providers are found in the same scope the *first* (the scope
walkers yielded ordering defines what *first* means) that produced that
result *and* can be extracted without raising an error is selected as the
provider of the requested requirement.
#. Fails with :py:class:`~taskflow.exceptions.NotFound` if unresolved at this
point (the ``cause`` attribute of this exception may have more details on
why the lookup failed).
.. note::
To examine this information when debugging it is recommended to
enable the ``BLATHER`` logging level (level 5). At this level the storage
and scope code/layers will log what is being searched for and what is
being found.
.. _scope: http://en.wikipedia.org/wiki/Scope_%28computer_science%29
Interfaces
==========
.. automodule:: taskflow.engines.base
Implementations
===============
.. automodule:: taskflow.engines.action_engine.engine
Components
----------
.. warning::
External usage of internal engine functions, components and modules should
be kept to a **minimum** as they may be altered, refactored or moved to
other locations **without** notice (and without the typical deprecation
cycle).
.. automodule:: taskflow.engines.action_engine.builder
.. automodule:: taskflow.engines.action_engine.compiler
.. automodule:: taskflow.engines.action_engine.completer
.. automodule:: taskflow.engines.action_engine.deciders
.. automodule:: taskflow.engines.action_engine.executor
.. automodule:: taskflow.engines.action_engine.process_executor
.. automodule:: taskflow.engines.action_engine.runtime
.. automodule:: taskflow.engines.action_engine.scheduler
.. automodule:: taskflow.engines.action_engine.selector
.. autoclass:: taskflow.engines.action_engine.scopes.ScopeWalker
:special-members: __iter__
.. automodule:: taskflow.engines.action_engine.traversal
Hierarchy
=========
.. inheritance-diagram::
taskflow.engines.action_engine.engine.ActionEngine
taskflow.engines.base.Engine
taskflow.engines.worker_based.engine.WorkerBasedActionEngine
:parts: 1
.. _automaton: https://docs.openstack.org/automaton/latest/
.. _multiprocessing: https://docs.python.org/2/library/multiprocessing.html
.. _future: https://docs.python.org/dev/library/concurrent.futures.html#future-objects
.. _executor: https://docs.python.org/dev/library/concurrent.futures.html#concurrent.futures.Executor
.. _networkx: https://networkx.github.io/
.. _futures backport: https://pypi.python.org/pypi/futures
.. _process pool executor: https://docs.python.org/dev/library/concurrent.futures.html#processpoolexecutor

View File

@ -1,390 +0,0 @@
==========
Examples
==========
While developing TaskFlow the team has worked *hard* to make sure the various
concepts are explained by *relevant* examples. Here are a few selected examples
to get started (ordered by *perceived* complexity):
To explore more of these examples please check out the `examples`_ directory
in the TaskFlow `source tree`_.
.. note::
If the examples provided are not satisfactory (or up to your
standards) contributions are welcome and very much appreciated to help
improve them. The higher the quality and the clearer the examples are the
better and more useful they are for everyone.
.. _examples: http://git.openstack.org/cgit/openstack/taskflow/tree/taskflow/examples
.. _source tree: http://git.openstack.org/cgit/openstack/taskflow/
Hello world
===========
.. note::
Full source located at :example:`hello_world`.
.. literalinclude:: ../../../taskflow/examples/hello_world.py
:language: python
:linenos:
:lines: 16-
Passing values from and to tasks
================================
.. note::
Full source located at :example:`simple_linear_pass`.
.. literalinclude:: ../../../taskflow/examples/simple_linear_pass.py
:language: python
:linenos:
:lines: 16-
Using listeners
===============
.. note::
Full source located at :example:`echo_listener`.
.. literalinclude:: ../../../taskflow/examples/echo_listener.py
:language: python
:linenos:
:lines: 16-
Using listeners (to watch a phone call)
=======================================
.. note::
Full source located at :example:`simple_linear_listening`.
.. literalinclude:: ../../../taskflow/examples/simple_linear_listening.py
:language: python
:linenos:
:lines: 16-
Dumping a in-memory backend
===========================
.. note::
Full source located at :example:`dump_memory_backend`.
.. literalinclude:: ../../../taskflow/examples/dump_memory_backend.py
:language: python
:linenos:
:lines: 16-
Making phone calls
==================
.. note::
Full source located at :example:`simple_linear`.
.. literalinclude:: ../../../taskflow/examples/simple_linear.py
:language: python
:linenos:
:lines: 16-
Making phone calls (automatically reverting)
============================================
.. note::
Full source located at :example:`reverting_linear`.
.. literalinclude:: ../../../taskflow/examples/reverting_linear.py
:language: python
:linenos:
:lines: 16-
Building a car
==============
.. note::
Full source located at :example:`build_a_car`.
.. literalinclude:: ../../../taskflow/examples/build_a_car.py
:language: python
:linenos:
:lines: 16-
Iterating over the alphabet (using processes)
=============================================
.. note::
Full source located at :example:`alphabet_soup`.
.. literalinclude:: ../../../taskflow/examples/alphabet_soup.py
:language: python
:linenos:
:lines: 16-
Watching execution timing
=========================
.. note::
Full source located at :example:`timing_listener`.
.. literalinclude:: ../../../taskflow/examples/timing_listener.py
:language: python
:linenos:
:lines: 16-
Distance calculator
===================
.. note::
Full source located at :example:`distance_calculator`
.. literalinclude:: ../../../taskflow/examples/distance_calculator.py
:language: python
:linenos:
:lines: 16-
Table multiplier (in parallel)
==============================
.. note::
Full source located at :example:`parallel_table_multiply`
.. literalinclude:: ../../../taskflow/examples/parallel_table_multiply.py
:language: python
:linenos:
:lines: 16-
Linear equation solver (explicit dependencies)
==============================================
.. note::
Full source located at :example:`calculate_linear`.
.. literalinclude:: ../../../taskflow/examples/calculate_linear.py
:language: python
:linenos:
:lines: 16-
Linear equation solver (inferred dependencies)
==============================================
``Source:`` :example:`graph_flow.py`
.. literalinclude:: ../../../taskflow/examples/graph_flow.py
:language: python
:linenos:
:lines: 16-
Linear equation solver (in parallel)
====================================
.. note::
Full source located at :example:`calculate_in_parallel`
.. literalinclude:: ../../../taskflow/examples/calculate_in_parallel.py
:language: python
:linenos:
:lines: 16-
Creating a volume (in parallel)
===============================
.. note::
Full source located at :example:`create_parallel_volume`
.. literalinclude:: ../../../taskflow/examples/create_parallel_volume.py
:language: python
:linenos:
:lines: 16-
Summation mapper(s) and reducer (in parallel)
=============================================
.. note::
Full source located at :example:`simple_map_reduce`
.. literalinclude:: ../../../taskflow/examples/simple_map_reduce.py
:language: python
:linenos:
:lines: 16-
Sharing a thread pool executor (in parallel)
============================================
.. note::
Full source located at :example:`share_engine_thread`
.. literalinclude:: ../../../taskflow/examples/share_engine_thread.py
:language: python
:linenos:
:lines: 16-
Storing & emitting a bill
=========================
.. note::
Full source located at :example:`fake_billing`
.. literalinclude:: ../../../taskflow/examples/fake_billing.py
:language: python
:linenos:
:lines: 16-
Suspending a workflow & resuming
================================
.. note::
Full source located at :example:`resume_from_backend`
.. literalinclude:: ../../../taskflow/examples/resume_from_backend.py
:language: python
:linenos:
:lines: 16-
Creating a virtual machine (resumable)
======================================
.. note::
Full source located at :example:`resume_vm_boot`
.. literalinclude:: ../../../taskflow/examples/resume_vm_boot.py
:language: python
:linenos:
:lines: 16-
Creating a volume (resumable)
=============================
.. note::
Full source located at :example:`resume_volume_create`
.. literalinclude:: ../../../taskflow/examples/resume_volume_create.py
:language: python
:linenos:
:lines: 16-
Running engines via iteration
=============================
.. note::
Full source located at :example:`run_by_iter`
.. literalinclude:: ../../../taskflow/examples/run_by_iter.py
:language: python
:linenos:
:lines: 16-
Controlling retries using a retry controller
============================================
.. note::
Full source located at :example:`retry_flow`
.. literalinclude:: ../../../taskflow/examples/retry_flow.py
:language: python
:linenos:
:lines: 16-
Distributed execution (simple)
==============================
.. note::
Full source located at :example:`wbe_simple_linear`
.. literalinclude:: ../../../taskflow/examples/wbe_simple_linear.py
:language: python
:linenos:
:lines: 16-
Distributed notification (simple)
=================================
.. note::
Full source located at :example:`wbe_event_sender`
.. literalinclude:: ../../../taskflow/examples/wbe_event_sender.py
:language: python
:linenos:
:lines: 16-
Distributed mandelbrot (complex)
================================
.. note::
Full source located at :example:`wbe_mandelbrot`
Output
------
.. image:: img/mandelbrot.png
:height: 128px
:align: right
:alt: Generated mandelbrot fractal
Code
----
.. literalinclude:: ../../../taskflow/examples/wbe_mandelbrot.py
:language: python
:linenos:
:lines: 16-
Jobboard producer/consumer (simple)
===================================
.. note::
Full source located at :example:`jobboard_produce_consume_colors`
.. literalinclude:: ../../../taskflow/examples/jobboard_produce_consume_colors.py
:language: python
:linenos:
:lines: 16-
Conductor simulating a CI pipeline
==================================
.. note::
Full source located at :example:`tox_conductor`
.. literalinclude:: ../../../taskflow/examples/tox_conductor.py
:language: python
:linenos:
:lines: 16-
Conductor running 99 bottles of beer song requests
==================================================
.. note::
Full source located at :example:`99_bottles`
.. literalinclude:: ../../../taskflow/examples/99_bottles.py
:language: python
:linenos:
:lines: 16-

View File

@ -1,9 +0,0 @@
----------
Exceptions
----------
.. inheritance-diagram::
taskflow.exceptions
:parts: 1
.. automodule:: taskflow.exceptions

View File

@ -1,2 +0,0 @@
.. include:: ../../../ChangeLog

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 7.1 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 9.4 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 67 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 24 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 26 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 108 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 22 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 22 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 20 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 236 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 16 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 25 KiB

View File

@ -1,80 +0,0 @@
================
Using TaskFlow
================
Considerations
==============
Things to consider before (and during) development and integration with
TaskFlow into your project:
* Read over the `paradigm shifts`_ and engage the team in `IRC`_ (or via the
`openstack-dev`_ mailing list) if these need more explanation (prefix
``[Oslo][TaskFlow]`` to your emails subject to get an even faster
response).
* Follow (or at least attempt to follow) some of the established
`best practices`_ (feel free to add your own suggested best practices).
* Keep in touch with the team (see above); we are all friendly and enjoy
knowing your use cases and learning how we can help make your lives easier
by adding or adjusting functionality in this library.
.. _IRC: irc://chat.freenode.net/openstack-state-management
.. _best practices: http://wiki.openstack.org/wiki/TaskFlow/Best_practices
.. _paradigm shifts: http://wiki.openstack.org/wiki/TaskFlow/Paradigm_shifts
.. _openstack-dev: mailto:openstack-dev@lists.openstack.org
User Guide
==========
.. toctree::
:maxdepth: 2
atoms
arguments_and_results
inputs_and_outputs
patterns
engines
workers
notifications
persistence
resumption
jobs
conductors
examples
Miscellaneous
=============
.. toctree::
:maxdepth: 2
exceptions
states
types
utils
Bookshelf
=========
A useful collection of links, documents, papers, similar
projects, frameworks and libraries.
.. note::
Please feel free to submit your own additions and/or changes.
.. toctree::
:maxdepth: 1
shelf
Release notes
=============
.. toctree::
:maxdepth: 2
history

View File

@ -1,169 +0,0 @@
==================
Inputs and outputs
==================
In TaskFlow there are multiple ways to provide inputs for your tasks and flows
and get information from them. This document describes one of them, that
involves task arguments and results. There are also :doc:`notifications
<notifications>`, which allow you to get notified when a task or flow changes
state. You may also opt to use the :doc:`persistence <persistence>` layer
itself directly.
-----------------------
Flow inputs and outputs
-----------------------
Tasks accept inputs via task arguments and provide outputs via task results
(see :doc:`arguments and results <arguments_and_results>` for more details).
This is the standard and recommended way to pass data from one task to another.
Of course not every task argument needs to be provided to some other task of a
flow, and not every task result should be consumed by every task.
If some value is required by one or more tasks of a flow, but it is not
provided by any task, it is considered to be flow input, and **must** be put
into the storage before the flow is run. A set of names required by a flow can
be retrieved via that flow's ``requires`` property. These names can be used to
determine what names may be applicable for placing in storage ahead of time
and which names are not applicable.
All values provided by tasks of the flow are considered to be flow outputs; the
set of names of such values is available via the ``provides`` property of the
flow.
.. testsetup::
from taskflow import task
from taskflow.patterns import linear_flow
from taskflow import engines
from pprint import pprint
For example:
.. doctest::
>>> class MyTask(task.Task):
... def execute(self, **kwargs):
... return 1, 2
...
>>> flow = linear_flow.Flow('test').add(
... MyTask(requires='a', provides=('b', 'c')),
... MyTask(requires='b', provides='d')
... )
>>> flow.requires
frozenset(['a'])
>>> sorted(flow.provides)
['b', 'c', 'd']
.. make vim syntax highlighter happy**
As you can see, this flow does not require b, as it is provided by the fist
task.
.. note::
There is no difference between processing of
:py:class:`Task <taskflow.task.Task>` and
:py:class:`~taskflow.retry.Retry` inputs and outputs.
------------------
Engine and storage
------------------
The storage layer is how an engine persists flow and task details (for more
in-depth details see :doc:`persistence <persistence>`).
Inputs
------
As mentioned above, if some value is required by one or more tasks of a flow,
but is not provided by any task, it is considered to be flow input, and
**must** be put into the storage before the flow is run. On failure to do
so :py:class:`~taskflow.exceptions.MissingDependencies` is raised by the engine
prior to running:
.. doctest::
>>> class CatTalk(task.Task):
... def execute(self, meow):
... print meow
... return "cat"
...
>>> class DogTalk(task.Task):
... def execute(self, woof):
... print woof
... return "dog"
...
>>> flo = linear_flow.Flow("cat-dog")
>>> flo.add(CatTalk(), DogTalk(provides="dog"))
<taskflow.patterns.linear_flow.Flow object at 0x...>
>>> engines.run(flo)
Traceback (most recent call last):
...
taskflow.exceptions.MissingDependencies:
taskflow.patterns.linear_flow.Flow: cat-dog;
2 requires ['meow', 'woof'] but no other entity produces said requirements
The recommended way to provide flow inputs is to use the ``store`` parameter
of the engine helpers (:py:func:`~taskflow.engines.helpers.run` or
:py:func:`~taskflow.engines.helpers.load`):
.. doctest::
>>> class CatTalk(task.Task):
... def execute(self, meow):
... print meow
... return "cat"
...
>>> class DogTalk(task.Task):
... def execute(self, woof):
... print woof
... return "dog"
...
>>> flo = linear_flow.Flow("cat-dog")
>>> flo.add(CatTalk(), DogTalk(provides="dog"))
<taskflow.patterns.linear_flow.Flow object at 0x...>
>>> result = engines.run(flo, store={'meow': 'meow', 'woof': 'woof'})
meow
woof
>>> pprint(result)
{'dog': 'dog', 'meow': 'meow', 'woof': 'woof'}
You can also directly interact with the engine storage layer to add additional
values, note that if this route is used you can't use the helper method
:py:func:`~taskflow.engines.helpers.run`. Instead,
you must activate the engine's run method directly
:py:func:`~taskflow.engines.base.EngineBase.run`:
.. doctest::
>>> flo = linear_flow.Flow("cat-dog")
>>> flo.add(CatTalk(), DogTalk(provides="dog"))
<taskflow.patterns.linear_flow.Flow object at 0x...>
>>> eng = engines.load(flo, store={'meow': 'meow'})
>>> eng.storage.inject({"woof": "bark"})
>>> eng.run()
meow
bark
Outputs
-------
As you can see from examples above, the run method returns all flow outputs in
a ``dict``. This same data can be fetched via
:py:meth:`~taskflow.storage.Storage.fetch_all` method of the engines storage
object. You can also get single results using the
engines storage objects :py:meth:`~taskflow.storage.Storage.fetch` method.
For example:
.. doctest::
>>> eng = engines.load(flo, store={'meow': 'meow', 'woof': 'woof'})
>>> eng.run()
meow
woof
>>> pprint(eng.storage.fetch_all())
{'dog': 'dog', 'meow': 'meow', 'woof': 'woof'}
>>> print(eng.storage.fetch("dog"))
dog

View File

@ -1,372 +0,0 @@
----
Jobs
----
Overview
========
Jobs and jobboards are a **novel** concept that TaskFlow provides to allow for
automatic ownership transfer of workflows between capable owners (those owners
usually then use :doc:`engines <engines>` to complete the workflow). They
provide the necessary semantics to be able to atomically transfer a job from a
producer to a consumer in a reliable and fault tolerant manner. They are
modeled off the concept used to post and acquire work in the physical world
(typically a job listing in a newspaper or online website serves a similar
role).
**TLDR:** It's similar to a queue, but consumers lock items on the queue when
claiming them, and only remove them from the queue when they're done with the
work. If the consumer fails, the lock is *automatically* released and the item
is back on the queue for further consumption.
.. note::
For more information, please visit the `paradigm shift`_ page for
more details.
Definitions
===========
Jobs
A :py:class:`job <taskflow.jobs.base.Job>` consists of a unique identifier,
name, and a reference to a :py:class:`logbook
<taskflow.persistence.models.LogBook>` which contains the details of the
work that has been or should be/will be completed to finish the work that has
been created for that job.
Jobboards
A :py:class:`jobboard <taskflow.jobs.base.JobBoard>` is responsible for
managing the posting, ownership, and delivery of jobs. It acts as the
location where jobs can be posted, claimed and searched for; typically by
iteration or notification. Jobboards may be backed by different *capable*
implementations (each with potentially differing configuration) but all
jobboards implement the same interface and semantics so that the backend
usage is as transparent as possible. This allows deployers or developers of a
service that uses TaskFlow to select a jobboard implementation that fits
their setup (and their intended usage) best.
High level architecture
=======================
.. figure:: img/jobboard.png
:height: 350px
:align: right
**Note:** This diagram shows the high-level diagram (and further
parts of this documentation also refer to it as well) of the zookeeper
implementation (other implementations will typically have
different architectures).
Features
========
- High availability
- Guarantees workflow forward progress by transferring partially complete
work or work that has not been started to entities which can either resume
the previously partially completed work or begin initial work to ensure
that the workflow as a whole progresses (where progressing implies
transitioning through the workflow :doc:`patterns <patterns>` and
:doc:`atoms <atoms>` and completing their associated
:doc:`states <states>` transitions).
- Atomic transfer and single ownership
- Ensures that only one workflow is managed (aka owned) by a single owner at
a time in an atomic manner (including when the workflow is transferred to
a owner that is resuming some other failed owners work). This avoids
contention and ensures a workflow is managed by one and only one entity at
a time.
- *Note:* this does not mean that the owner needs to run the
workflow itself but instead said owner could use an engine that runs the
work in a distributed manner to ensure that the workflow progresses.
- Separation of workflow construction and execution
- Jobs can be created with logbooks that contain a specification of the work
to be done by a entity (such as an API server). The job then can be
completed by a entity that is watching that jobboard (not necessarily the
API server itself). This creates a disconnection between work
formation and work completion that is useful for scaling out horizontally.
- Asynchronous completion
- When for example a API server posts a job for completion to a
jobboard that API server can return a *tracking* identifier to the user
calling the API service. This *tracking* identifier can be used by the
user to poll for status (similar in concept to a shipping *tracking*
identifier created by fedex or UPS).
Usage
=====
All jobboards are mere classes that implement same interface, and of course
it is possible to import them and create instances of them just like with any
other class in Python. But the easier (and recommended) way for creating
jobboards is by using the :py:meth:`fetch() <taskflow.jobs.backends.fetch>`
function which uses entrypoints (internally using `stevedore`_) to fetch and
configure your backend.
Using this function the typical creation of a jobboard (and an example posting
of a job) might look like:
.. code-block:: python
from taskflow.persistence import backends as persistence_backends
from taskflow.jobs import backends as job_backends
...
persistence = persistence_backends.fetch({
"connection': "mysql",
"user": ...,
"password": ...,
})
book = make_and_save_logbook(persistence)
board = job_backends.fetch('my-board', {
"board": "zookeeper",
}, persistence=persistence)
job = board.post("my-first-job", book)
...
Consumption of jobs is similarly achieved by creating a jobboard and using
the iteration functionality to find and claim jobs (and eventually consume
them). The typical usage of a jobboard for consumption (and work completion)
might look like:
.. code-block:: python
import time
from taskflow import exceptions as exc
from taskflow.persistence import backends as persistence_backends
from taskflow.jobs import backends as job_backends
...
my_name = 'worker-1'
coffee_break_time = 60
persistence = persistence_backends.fetch({
"connection': "mysql",
"user": ...,
"password": ...,
})
board = job_backends.fetch('my-board', {
"board": "zookeeper",
}, persistence=persistence)
while True:
my_job = None
for job in board.iterjobs(only_unclaimed=True):
try:
board.claim(job, my_name)
except exc.UnclaimableJob:
pass
else:
my_job = job
break
if my_job is not None:
try:
perform_job(my_job)
except Exception:
LOG.exception("I failed performing job: %s", my_job)
board.abandon(my_job, my_name)
else:
# I finished it, now cleanup.
board.consume(my_job)
persistence.get_connection().destroy_logbook(my_job.book.uuid)
time.sleep(coffee_break_time)
...
There are a few ways to provide arguments to the flow. The first option is to
add a ``store`` to the flowdetail object in the
:py:class:`logbook <taskflow.persistence.models.LogBook>`.
You can also provide a ``store`` in the
:py:class:`job <taskflow.jobs.base.Job>` itself when posting it to the
job board. If both ``store`` values are found, they will be combined,
with the :py:class:`job <taskflow.jobs.base.Job>` ``store``
overriding the :py:class:`logbook <taskflow.persistence.models.LogBook>`
``store``.
.. code-block:: python
from oslo_utils import uuidutils
from taskflow import engines
from taskflow.persistence import backends as persistence_backends
from taskflow.persistence import models
from taskflow.jobs import backends as job_backends
...
persistence = persistence_backends.fetch({
"connection': "mysql",
"user": ...,
"password": ...,
})
board = job_backends.fetch('my-board', {
"board": "zookeeper",
}, persistence=persistence)
book = models.LogBook('my-book', uuidutils.generate_uuid())
flow_detail = models.FlowDetail('my-job', uuidutils.generate_uuid())
book.add(flow_detail)
connection = persistence.get_connection()
connection.save_logbook(book)
flow_detail.meta['store'] = {'a': 1, 'c': 3}
job_details = {
"flow_uuid": flow_detail.uuid,
"store": {'a': 2, 'b': 1}
}
engines.save_factory_details(flow_detail, flow_factory,
factory_args=[],
factory_kwargs={},
backend=persistence)
jobboard = get_jobboard(zk_client)
jobboard.connect()
job = jobboard.post('my-job', book=book, details=job_details)
# the flow global parameters are now the combined store values
# {'a': 2, 'b': 1', 'c': 3}
...
Types
=====
Zookeeper
---------
**Board type**: ``'zookeeper'``
Uses `zookeeper`_ to provide the jobboard capabilities and semantics by using
a zookeeper directory, ephemeral, non-ephemeral nodes and watches.
Additional *kwarg* parameters:
* ``client``: a class that provides ``kazoo.client.KazooClient``-like
interface; it will be used for zookeeper interactions, sharing clients
between jobboard instances will likely provide better scalability and can
help avoid creating to many open connections to a set of zookeeper servers.
* ``persistence``: a class that provides a :doc:`persistence <persistence>`
backend interface; it will be used for loading jobs logbooks for usage at
runtime or for usage before a job is claimed for introspection.
Additional *configuration* parameters:
* ``path``: the root zookeeper path to store job information (*defaults* to
``/taskflow/jobs``)
* ``hosts``: the list of zookeeper hosts to connect to (*defaults* to
``localhost:2181``); only used if a client is not provided.
* ``timeout``: the timeout used when performing operations with zookeeper;
only used if a client is not provided.
* ``handler``: a class that provides ``kazoo.handlers``-like interface; it will
be used internally by `kazoo`_ to perform asynchronous operations, useful
when your program uses eventlet and you want to instruct kazoo to use an
eventlet compatible handler.
.. note::
See :py:class:`~taskflow.jobs.backends.impl_zookeeper.ZookeeperJobBoard`
for implementation details.
Redis
-----
**Board type**: ``'redis'``
Uses `redis`_ to provide the jobboard capabilities and semantics by using
a redis hash data structure and individual job ownership keys (that can
optionally expire after a given amount of time).
.. note::
See :py:class:`~taskflow.jobs.backends.impl_redis.RedisJobBoard`
for implementation details.
Considerations
==============
Some usage considerations should be used when using a jobboard to make sure
it's used in a safe and reliable manner. Eventually we hope to make these
non-issues but for now they are worth mentioning.
Dual-engine jobs
----------------
**What:** Since atoms and engines are not currently `preemptable`_ we can not
force an engine (or the threads/remote workers... it is using to run) to stop
working on an atom (it is general bad behavior to force code to stop without
its consent anyway) if it has already started working on an atom (short of
doing a ``kill -9`` on the running interpreter). This could cause problems
since the points an engine can notice that it no longer owns a claim is at any
:doc:`state <states>` change that occurs (transitioning to a new atom or
recording a result for example), where upon noticing the claim has been lost
the engine can immediately stop doing further work. The effect that this causes
is that when a claim is lost another engine can immediately attempt to acquire
the claim that was previously lost and it *could* begin working on the
unfinished tasks that the later engine may also still be executing (since that
engine is not yet aware that it has *lost* the claim).
**TLDR:** not `preemptable`_, possible to become aware of losing a claim
after the fact (at the next state change), another engine could have acquired
the claim by then, therefore both would be *working* on a job.
**Alleviate by:**
#. Ensure your atoms are `idempotent`_, this will cause an engine that may be
executing the same atom to be able to continue executing without causing
any conflicts/problems (idempotency guarantees this).
#. On claiming jobs that have been claimed previously enforce a policy that
happens before the jobs workflow begins to execute (possibly prior to an
engine beginning the jobs work) that ensures that any prior work has been
rolled back before continuing rolling forward. For example:
* Rolling back the last atom/set of atoms that finished.
* Rolling back the last state change that occurred.
#. Delay claiming partially completed work by adding a wait period (to allow
the previous engine to coalesce) before working on a partially completed job
(combine this with the prior suggestions and *most* dual-engine issues
should be avoided).
.. _idempotent: http://en.wikipedia.org/wiki/Idempotence
.. _preemptable: http://en.wikipedia.org/wiki/Preemption_%28computing%29
Interfaces
==========
.. automodule:: taskflow.jobs.base
.. automodule:: taskflow.jobs.backends
Implementations
===============
Zookeeper
---------
.. automodule:: taskflow.jobs.backends.impl_zookeeper
Redis
-----
.. automodule:: taskflow.jobs.backends.impl_redis
Hierarchy
=========
.. inheritance-diagram::
taskflow.jobs.base
taskflow.jobs.backends.impl_redis
taskflow.jobs.backends.impl_zookeeper
:parts: 1
.. _paradigm shift: https://wiki.openstack.org/wiki/TaskFlow/Paradigm_shifts#Workflow_ownership_transfer
.. _zookeeper: http://zookeeper.apache.org/
.. _kazoo: http://kazoo.readthedocs.org/
.. _stevedore: https://docs.openstack.org/stevedore/latest
.. _redis: http://redis.io/

View File

@ -1,202 +0,0 @@
---------------------------
Notifications and listeners
---------------------------
.. testsetup::
from taskflow import task
from taskflow.patterns import linear_flow
from taskflow import engines
from taskflow.types import notifier
ANY = notifier.Notifier.ANY
Overview
========
Engines provide a way to receive notification on task and flow state
transitions (see :doc:`states <states>`), which is useful for
monitoring, logging, metrics, debugging and plenty of other tasks.
To receive these notifications you should register a callback with
an instance of the :py:class:`~taskflow.types.notifier.Notifier`
class that is attached to :py:class:`~taskflow.engines.base.Engine`
attributes ``atom_notifier`` and ``notifier``.
TaskFlow also comes with a set of predefined :ref:`listeners <listeners>`, and
provides means to write your own listeners, which can be more convenient than
using raw callbacks.
Receiving notifications with callbacks
======================================
Flow notifications
------------------
To receive notification on flow state changes use the
:py:class:`~taskflow.types.notifier.Notifier` instance available as the
``notifier`` property of an engine.
A basic example is:
.. doctest::
>>> class CatTalk(task.Task):
... def execute(self, meow):
... print(meow)
... return "cat"
...
>>> class DogTalk(task.Task):
... def execute(self, woof):
... print(woof)
... return 'dog'
...
>>> def flow_transition(state, details):
... print("Flow '%s' transition to state %s" % (details['flow_name'], state))
...
>>>
>>> flo = linear_flow.Flow("cat-dog").add(
... CatTalk(), DogTalk(provides="dog"))
>>> eng = engines.load(flo, store={'meow': 'meow', 'woof': 'woof'})
>>> eng.notifier.register(ANY, flow_transition)
>>> eng.run()
Flow 'cat-dog' transition to state RUNNING
meow
woof
Flow 'cat-dog' transition to state SUCCESS
Task notifications
------------------
To receive notification on task state changes use the
:py:class:`~taskflow.types.notifier.Notifier` instance available as the
``atom_notifier`` property of an engine.
A basic example is:
.. doctest::
>>> class CatTalk(task.Task):
... def execute(self, meow):
... print(meow)
... return "cat"
...
>>> class DogTalk(task.Task):
... def execute(self, woof):
... print(woof)
... return 'dog'
...
>>> def task_transition(state, details):
... print("Task '%s' transition to state %s" % (details['task_name'], state))
...
>>>
>>> flo = linear_flow.Flow("cat-dog")
>>> flo.add(CatTalk(), DogTalk(provides="dog"))
<taskflow.patterns.linear_flow.Flow object at 0x...>
>>> eng = engines.load(flo, store={'meow': 'meow', 'woof': 'woof'})
>>> eng.atom_notifier.register(ANY, task_transition)
>>> eng.run()
Task 'CatTalk' transition to state RUNNING
meow
Task 'CatTalk' transition to state SUCCESS
Task 'DogTalk' transition to state RUNNING
woof
Task 'DogTalk' transition to state SUCCESS
.. _listeners:
Listeners
=========
TaskFlow comes with a set of predefined listeners -- helper classes that can be
used to do various actions on flow and/or tasks transitions. You can also
create your own listeners easily, which may be more convenient than using raw
callbacks for some use cases.
For example, this is how you can use
:py:class:`~taskflow.listeners.printing.PrintingListener`:
.. doctest::
>>> from taskflow.listeners import printing
>>> class CatTalk(task.Task):
... def execute(self, meow):
... print(meow)
... return "cat"
...
>>> class DogTalk(task.Task):
... def execute(self, woof):
... print(woof)
... return 'dog'
...
>>>
>>> flo = linear_flow.Flow("cat-dog").add(
... CatTalk(), DogTalk(provides="dog"))
>>> eng = engines.load(flo, store={'meow': 'meow', 'woof': 'woof'})
>>> with printing.PrintingListener(eng):
... eng.run()
...
<taskflow.engines.action_engine.engine.SerialActionEngine object at ...> has moved flow 'cat-dog' (...) into state 'RUNNING' from state 'PENDING'
<taskflow.engines.action_engine.engine.SerialActionEngine object at ...> has moved task 'CatTalk' (...) into state 'RUNNING' from state 'PENDING'
meow
<taskflow.engines.action_engine.engine.SerialActionEngine object at ...> has moved task 'CatTalk' (...) into state 'SUCCESS' from state 'RUNNING' with result 'cat' (failure=False)
<taskflow.engines.action_engine.engine.SerialActionEngine object at ...> has moved task 'DogTalk' (...) into state 'RUNNING' from state 'PENDING'
woof
<taskflow.engines.action_engine.engine.SerialActionEngine object at ...> has moved task 'DogTalk' (...) into state 'SUCCESS' from state 'RUNNING' with result 'dog' (failure=False)
<taskflow.engines.action_engine.engine.SerialActionEngine object at ...> has moved flow 'cat-dog' (...) into state 'SUCCESS' from state 'RUNNING'
Interfaces
==========
.. automodule:: taskflow.listeners.base
Implementations
===============
Printing and logging listeners
------------------------------
.. autoclass:: taskflow.listeners.logging.LoggingListener
.. autoclass:: taskflow.listeners.logging.DynamicLoggingListener
.. autoclass:: taskflow.listeners.printing.PrintingListener
Timing listeners
----------------
.. autoclass:: taskflow.listeners.timing.DurationListener
.. autoclass:: taskflow.listeners.timing.PrintingDurationListener
.. autoclass:: taskflow.listeners.timing.EventTimeListener
Claim listener
--------------
.. autoclass:: taskflow.listeners.claims.CheckingClaimListener
Capturing listener
------------------
.. autoclass:: taskflow.listeners.capturing.CaptureListener
Formatters
----------
.. automodule:: taskflow.formatters
Hierarchy
=========
.. inheritance-diagram::
taskflow.listeners.base.DumpingListener
taskflow.listeners.base.Listener
taskflow.listeners.capturing.CaptureListener
taskflow.listeners.claims.CheckingClaimListener
taskflow.listeners.logging.DynamicLoggingListener
taskflow.listeners.logging.LoggingListener
taskflow.listeners.printing.PrintingListener
taskflow.listeners.timing.PrintingDurationListener
taskflow.listeners.timing.EventTimeListener
taskflow.listeners.timing.DurationListener
:parts: 1

View File

@ -1,34 +0,0 @@
--------
Patterns
--------
.. automodule:: taskflow.flow
Linear flow
~~~~~~~~~~~
.. automodule:: taskflow.patterns.linear_flow
Unordered flow
~~~~~~~~~~~~~~
.. automodule:: taskflow.patterns.unordered_flow
Graph flow
~~~~~~~~~~
.. automodule:: taskflow.patterns.graph_flow
.. automodule:: taskflow.deciders
Hierarchy
~~~~~~~~~
.. inheritance-diagram::
taskflow.flow
taskflow.patterns.linear_flow
taskflow.patterns.unordered_flow
taskflow.patterns.graph_flow
:parts: 2

View File

@ -1,330 +0,0 @@
===========
Persistence
===========
Overview
========
In order to be able to receive inputs and create outputs from atoms (or other
engine processes) in a fault-tolerant way, there is a need to be able to place
what atoms output in some kind of location where it can be re-used by other
atoms (or used for other purposes). To accommodate this type of usage TaskFlow
provides an abstraction (provided by pluggable `stevedore`_ backends) that is
similar in concept to a running programs *memory*.
This abstraction serves the following *major* purposes:
* Tracking of what was done (introspection).
* Saving *memory* which allows for restarting from the last saved state
which is a critical feature to restart and resume workflows (checkpointing).
* Associating additional metadata with atoms while running (without having
those atoms need to save this data themselves). This makes it possible to
add-on new metadata in the future without having to change the atoms
themselves. For example the following can be saved:
* Timing information (how long a task took to run).
* User information (who the task ran as).
* When a atom/workflow was ran (and why).
* Saving historical data (failures, successes, intermediary results...)
to allow for retry atoms to be able to decide if they should should continue
vs. stop.
* *Something you create...*
.. _stevedore: https://docs.openstack.org/stevedore/latest/
How it is used
==============
On :doc:`engine <engines>` construction typically a backend (it can be
optional) will be provided which satisfies the
:py:class:`~taskflow.persistence.base.Backend` abstraction. Along with
providing a backend object a
:py:class:`~taskflow.persistence.models.FlowDetail` object will also be
created and provided (this object will contain the details about the flow to be
ran) to the engine constructor (or associated :py:meth:`load()
<taskflow.engines.helpers.load>` helper functions). Typically a
:py:class:`~taskflow.persistence.models.FlowDetail` object is created from a
:py:class:`~taskflow.persistence.models.LogBook` object (the book object acts
as a type of container for :py:class:`~taskflow.persistence.models.FlowDetail`
and :py:class:`~taskflow.persistence.models.AtomDetail` objects).
**Preparation**: Once an engine starts to run it will create a
:py:class:`~taskflow.storage.Storage` object which will act as the engines
interface to the underlying backend storage objects (it provides helper
functions that are commonly used by the engine, avoiding repeating code when
interacting with the provided
:py:class:`~taskflow.persistence.models.FlowDetail` and
:py:class:`~taskflow.persistence.base.Backend` objects). As an engine
initializes it will extract (or create)
:py:class:`~taskflow.persistence.models.AtomDetail` objects for each atom in
the workflow the engine will be executing.
**Execution:** When an engine beings to execute (see :doc:`engine <engines>`
for more of the details about how an engine goes about this process) it will
examine any previously existing
:py:class:`~taskflow.persistence.models.AtomDetail` objects to see if they can
be used for resuming; see :doc:`resumption <resumption>` for more details on
this subject. For atoms which have not finished (or did not finish correctly
from a previous run) they will begin executing only after any dependent inputs
are ready. This is done by analyzing the execution graph and looking at
predecessor :py:class:`~taskflow.persistence.models.AtomDetail` outputs and
states (which may have been persisted in a past run). This will result in
either using their previous information or by running those predecessors and
saving their output to the :py:class:`~taskflow.persistence.models.FlowDetail`
and :py:class:`~taskflow.persistence.base.Backend` objects. This
execution, analysis and interaction with the storage objects continues (what is
described here is a simplification of what really happens; which is quite a bit
more complex) until the engine has finished running (at which point the engine
will have succeeded or failed in its attempt to run the workflow).
**Post-execution:** Typically when an engine is done running the logbook would
be discarded (to avoid creating a stockpile of useless data) and the backend
storage would be told to delete any contents for a given execution. For certain
use-cases though it may be advantageous to retain logbooks and their contents.
A few scenarios come to mind:
* Post runtime failure analysis and triage (saving what failed and why).
* Metrics (saving timing information associated with each atom and using it
to perform offline performance analysis, which enables tuning tasks and/or
isolating and fixing slow tasks).
* Data mining logbooks to find trends (in failures for example).
* Saving logbooks for further forensics analysis.
* Exporting logbooks to `hdfs`_ (or other no-sql storage) and running some type
of map-reduce jobs on them.
.. _hdfs: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html
.. note::
It should be emphasized that logbook is the authoritative, and, preferably,
the **only** (see :doc:`inputs and outputs <inputs_and_outputs>`) source of
run-time state information (breaking this principle makes it
hard/impossible to restart or resume in any type of automated fashion).
When an atom returns a result, it should be written directly to a logbook.
When atom or flow state changes in any way, logbook is first to know (see
:doc:`notifications <notifications>` for how a user may also get notified
of those same state changes). The logbook and a backend and associated
storage helper class are responsible to store the actual data. These
components used together specify the persistence mechanism (how data is
saved and where -- memory, database, whatever...) and the persistence
policy (when data is saved -- every time it changes or at some particular
moments or simply never).
Usage
=====
To select which persistence backend to use you should use the :py:meth:`fetch()
<taskflow.persistence.backends.fetch>` function which uses entrypoints
(internally using `stevedore`_) to fetch and configure your backend. This makes
it simpler than accessing the backend data types directly and provides a common
function from which a backend can be fetched.
Using this function to fetch a backend might look like:
.. code-block:: python
from taskflow.persistence import backends
...
persistence = backends.fetch(conf={
"connection': "mysql",
"user": ...,
"password": ...,
})
book = make_and_save_logbook(persistence)
...
As can be seen from above the ``conf`` parameter acts as a dictionary that
is used to fetch and configure your backend. The restrictions on it are
the following:
* a dictionary (or dictionary like type), holding backend type with key
``'connection'`` and possibly type-specific backend parameters as other
keys.
Types
=====
Memory
------
**Connection**: ``'memory'``
Retains all data in local memory (not persisted to reliable storage). Useful
for scenarios where persistence is not required (and also in unit tests).
.. note::
See :py:class:`~taskflow.persistence.backends.impl_memory.MemoryBackend`
for implementation details.
Files
-----
**Connection**: ``'dir'`` or ``'file'``
Retains all data in a directory & file based structure on local disk. Will be
persisted **locally** in the case of system failure (allowing for resumption
from the same local machine only). Useful for cases where a *more* reliable
persistence is desired along with the simplicity of files and directories (a
concept everyone is familiar with).
.. note::
See :py:class:`~taskflow.persistence.backends.impl_dir.DirBackend`
for implementation details.
SQLAlchemy
----------
**Connection**: ``'mysql'`` or ``'postgres'`` or ``'sqlite'``
Retains all data in a `ACID`_ compliant database using the `sqlalchemy`_
library for schemas, connections, and database interaction functionality.
Useful when you need a higher level of durability than offered by the previous
solutions. When using these connection types it is possible to resume a engine
from a peer machine (this does not apply when using sqlite).
Schema
^^^^^^
*Logbooks*
========== ======== =============
Name Type Primary Key
========== ======== =============
created_at DATETIME False
updated_at DATETIME False
uuid VARCHAR True
name VARCHAR False
meta TEXT False
========== ======== =============
*Flow details*
=========== ======== =============
Name Type Primary Key
=========== ======== =============
created_at DATETIME False
updated_at DATETIME False
uuid VARCHAR True
name VARCHAR False
meta TEXT False
state VARCHAR False
parent_uuid VARCHAR False
=========== ======== =============
*Atom details*
=========== ======== =============
Name Type Primary Key
=========== ======== =============
created_at DATETIME False
updated_at DATETIME False
uuid VARCHAR True
name VARCHAR False
meta TEXT False
atom_type VARCHAR False
state VARCHAR False
intention VARCHAR False
results TEXT False
failure TEXT False
version TEXT False
parent_uuid VARCHAR False
=========== ======== =============
.. _sqlalchemy: http://www.sqlalchemy.org/docs/
.. _ACID: https://en.wikipedia.org/wiki/ACID
.. note::
See :py:class:`~taskflow.persistence.backends.impl_sqlalchemy.SQLAlchemyBackend`
for implementation details.
.. warning::
Currently there is a size limit (not applicable for ``sqlite``) that the
``results`` will contain. This size limit will restrict how many prior
failures a retry atom can contain. More information and a future fix
will be posted to bug `1416088`_ (for the meantime try to ensure that
your retry units history does not grow beyond ~80 prior results). This
truncation can also be avoided by providing ``mysql_sql_mode`` as
``traditional`` when selecting your mysql + sqlalchemy based
backend (see the `mysql modes`_ documentation for what this implies).
.. _1416088: http://bugs.launchpad.net/taskflow/+bug/1416088
.. _mysql modes: http://dev.mysql.com/doc/refman/5.0/en/sql-mode.html
Zookeeper
---------
**Connection**: ``'zookeeper'``
Retains all data in a `zookeeper`_ backend (zookeeper exposes operations on
files and directories, similar to the above ``'dir'`` or ``'file'`` connection
types). Internally the `kazoo`_ library is used to interact with zookeeper
to perform reliable, distributed and atomic operations on the contents of a
logbook represented as znodes. Since zookeeper is also distributed it is also
able to resume a engine from a peer machine (having similar functionality
as the database connection types listed previously).
.. note::
See :py:class:`~taskflow.persistence.backends.impl_zookeeper.ZkBackend`
for implementation details.
.. _zookeeper: http://zookeeper.apache.org
.. _kazoo: http://kazoo.readthedocs.org/
Interfaces
==========
.. automodule:: taskflow.persistence.backends
.. automodule:: taskflow.persistence.base
.. automodule:: taskflow.persistence.path_based
Models
======
.. automodule:: taskflow.persistence.models
Implementations
===============
Memory
------
.. automodule:: taskflow.persistence.backends.impl_memory
Files
-----
.. automodule:: taskflow.persistence.backends.impl_dir
SQLAlchemy
----------
.. automodule:: taskflow.persistence.backends.impl_sqlalchemy
Zookeeper
---------
.. automodule:: taskflow.persistence.backends.impl_zookeeper
Storage
=======
.. automodule:: taskflow.storage
Hierarchy
=========
.. inheritance-diagram::
taskflow.persistence.base
taskflow.persistence.backends.impl_dir
taskflow.persistence.backends.impl_memory
taskflow.persistence.backends.impl_sqlalchemy
taskflow.persistence.backends.impl_zookeeper
:parts: 2

View File

@ -1,163 +0,0 @@
----------
Resumption
----------
Overview
========
**Question**: *How can we persist the flow so that it can be resumed, restarted
or rolled-back on engine failure?*
**Answer:** Since a flow is a set of :doc:`atoms <atoms>` and relations between
atoms we need to create a model and corresponding information that allows us to
persist the *right* amount of information to preserve, resume, and rollback a
flow on software or hardware failure.
To allow for resumption TaskFlow must be able to re-create the flow and
re-connect the links between atom (and between atoms->atom details and so on)
in order to revert those atoms or resume those atoms in the correct ordering.
TaskFlow provides a pattern that can help in automating this process (it does
**not** prohibit the user from creating their own strategies for doing this).
.. _resumption factories:
Factories
=========
The default provided way is to provide a `factory`_ function which will create
(or recreate your workflow). This function can be provided when loading a flow
and corresponding engine via the provided :py:meth:`load_from_factory()
<taskflow.engines.helpers.load_from_factory>` method. This `factory`_ function
is expected to be a function (or ``staticmethod``) which is reimportable (aka
has a well defined name that can be located by the ``__import__`` function in
python, this excludes ``lambda`` style functions and ``instance`` methods). The
`factory`_ function name will be saved into the logbook and it will be imported
and called to create the workflow objects (or recreate it if resumption
happens). This allows for the flow to be recreated if and when that is needed
(even on remote machines, as long as the reimportable name can be located).
.. _factory: https://en.wikipedia.org/wiki/Factory_%28object-oriented_programming%29
Names
=====
When a flow is created it is expected that each atom has a unique name, this
name serves a special purpose in the resumption process (as well as serving a
useful purpose when running, allowing for atom identification in the
:doc:`notification <notifications>` process). The reason for having names is
that an atom in a flow needs to be somehow matched with (a potentially)
existing :py:class:`~taskflow.persistence.models.AtomDetail` during engine
resumption & subsequent running.
The match should be:
* stable if atoms are added or removed
* should not change when service is restarted, upgraded...
* should be the same across all server instances in HA setups
Names provide this although they do have weaknesses:
* the names of atoms must be unique in flow
* it becomes hard to change the name of atom since a name change causes other
side-effects
.. note::
Even though these weaknesses names were selected as a *good enough*
solution for the above matching requirements (until something better is
invented/created that can satisfy those same requirements).
Scenarios
=========
When new flow is loaded into engine, there is no persisted data for it yet, so
a corresponding :py:class:`~taskflow.persistence.models.FlowDetail` object
will be created, as well as a
:py:class:`~taskflow.persistence.models.AtomDetail` object for each atom that
is contained in it. These will be immediately saved into the persistence
backend that is configured. If no persistence backend is configured, then as
expected nothing will be saved and the atoms and flow will be ran in a
non-persistent manner.
**Subsequent run:** When we resume the flow from a persistent backend (for
example, if the flow was interrupted and engine destroyed to save resources or
if the service was restarted), we need to re-create the flow. For that, we will
call the function that was saved on first-time loading that builds the flow for
us (aka; the flow factory function described above) and the engine will run.
The following scenarios explain some expected structural changes and how they
can be accommodated (and what the effect will be when resuming & running).
Same atoms
++++++++++
When the factory function mentioned above returns the exact same the flow and
atoms (no changes are performed).
**Runtime change:** Nothing should be done -- the engine will re-associate
atoms with :py:class:`~taskflow.persistence.models.AtomDetail` objects by name
and then the engine resumes.
Atom was added
++++++++++++++
When the factory function mentioned above alters the flow by adding a new atom
in (for example for changing the runtime structure of what was previously ran
in the first run).
**Runtime change:** By default when the engine resumes it will notice that a
corresponding :py:class:`~taskflow.persistence.models.AtomDetail` does not
exist and one will be created and associated.
Atom was removed
++++++++++++++++
When the factory function mentioned above alters the flow by removing a new
atom in (for example for changing the runtime structure of what was previously
ran in the first run).
**Runtime change:** Nothing should be done -- flow structure is reloaded from
factory function, and removed atom is not in it -- so, flow will be ran as if
it was not there, and any results it returned if it was completed before will
be ignored.
Atom code was changed
+++++++++++++++++++++
When the factory function mentioned above alters the flow by deciding that a
newer version of a previously existing atom should be ran (possibly to perform
some kind of upgrade or to fix a bug in a prior atoms code).
**Factory change:** The atom name & version will have to be altered. The
factory should replace this name where it was being used previously.
**Runtime change:** This will fall under the same runtime adjustments that
exist when a new atom is added. In the future TaskFlow could make this easier
by providing a ``upgrade()`` function that can be used to give users the
ability to upgrade atoms before running (manual introspection & modification of
a :py:class:`~taskflow.persistence.models.LogBook` can be done before engine
loading and running to accomplish this in the meantime).
Atom was split in two atoms or merged
+++++++++++++++++++++++++++++++++++++
When the factory function mentioned above alters the flow by deciding that a
previously existing atom should be split into N atoms or the factory function
decides that N atoms should be merged in <N atoms (typically occurring during
refactoring).
**Runtime change:** This will fall under the same runtime adjustments that
exist when a new atom is added or removed. In the future TaskFlow could make
this easier by providing a ``migrate()`` function that can be used to give
users the ability to migrate atoms previous data before running (manual
introspection & modification of a
:py:class:`~taskflow.persistence.models.LogBook` can be done before engine
loading and running to accomplish this in the meantime).
Flow structure was changed
++++++++++++++++++++++++++
If manual links were added or removed from graph, or task requirements were
changed, or flow was refactored (atom moved into or out of subflows, linear
flow was replaced with graph flow, tasks were reordered in linear flow, etc).
**Runtime change:** Nothing should be done.

View File

@ -1,60 +0,0 @@
Libraries & frameworks
----------------------
* `APScheduler`_ (Python)
* `Async`_ (Python)
* `Celery`_ (Python)
* `Graffiti`_ (Python)
* `JobLib`_ (Python)
* `Luigi`_ (Python)
* `Mesos`_ (C/C++)
* `Papy`_ (Python)
* `Parallel Python`_ (Python)
* `RQ`_ (Python)
* `Spiff`_ (Python)
* `TBB Flow`_ (C/C++)
Languages
---------
* `Ani`_
* `Make`_
* `Plaid`_
Services
--------
* `Cloud Dataflow`_
* `Mistral`_
Papers
------
* `Advances in Dataflow Programming Languages`_
Related paradigms
-----------------
* `Dataflow programming`_
* `Programming paradigm(s)`_
.. _APScheduler: http://pythonhosted.org/APScheduler/
.. _Async: http://pypi.python.org/pypi/async
.. _Celery: http://www.celeryproject.org/
.. _Graffiti: http://github.com/SegFaultAX/graffiti
.. _JobLib: http://pythonhosted.org/joblib/index.html
.. _Luigi: http://github.com/spotify/luigi
.. _RQ: http://python-rq.org/
.. _Mistral: http://wiki.openstack.org/wiki/Mistral
.. _Mesos: http://mesos.apache.org/
.. _Parallel Python: http://www.parallelpython.com/
.. _Spiff: http://github.com/knipknap/SpiffWorkflow
.. _Papy: http://code.google.com/p/papy/
.. _Make: http://www.gnu.org/software/make/
.. _Ani: http://code.google.com/p/anic/
.. _Programming paradigm(s): http://en.wikipedia.org/wiki/Programming_paradigm
.. _Plaid: http://www.cs.cmu.edu/~aldrich/plaid/
.. _Advances in Dataflow Programming Languages: http://www.cs.ucf.edu/~dcm/Teaching/COT4810-Spring2011/Literature/DataFlowProgrammingLanguages.pdf
.. _Cloud Dataflow: https://cloud.google.com/dataflow/
.. _TBB Flow: https://www.threadingbuildingblocks.org/tutorial-intel-tbb-flow-graph
.. _Dataflow programming: http://en.wikipedia.org/wiki/Dataflow_programming

View File

@ -1,241 +0,0 @@
------
States
------
.. _engine states:
.. note::
The code contains explicit checks during transitions using the models
described below. These checks ensure that a transition is valid, if the
transition is determined to be invalid the transitioning code will raise
a :py:class:`~taskflow.exceptions.InvalidState` exception. This exception
being triggered usually means there is some kind of bug in the code or some
type of misuse/state violation is occurring, and should be reported as such.
Engine
======
.. image:: img/engine_states.svg
:width: 660px
:align: center
:alt: Action engine state transitions
**RESUMING** - Prepares flow & atoms to be resumed.
**SCHEDULING** - Schedules and submits atoms to be worked on.
**WAITING** - Wait for atoms to finish executing.
**ANALYZING** - Analyzes and processes result/s of atom completion.
**SUCCESS** - Completed successfully.
**FAILURE** - Completed unsuccessfully.
**REVERTED** - Reverting was induced and all atoms were **not** completed
successfully.
**SUSPENDED** - Suspended while running.
**UNDEFINED** - *Internal state.*
**GAME_OVER** - *Internal state.*
Flow
====
.. image:: img/flow_states.svg
:width: 660px
:align: center
:alt: Flow state transitions
**PENDING** - A flow starts (or
via :py:meth:`~taskflow.engines.base.Engine.reset`) its execution lifecycle
in this state (it has no state prior to being ran by an engine, since
flow(s) are just pattern(s) that define the semantics and ordering of their
contents and flows gain state only when they are executed).
**RUNNING** - In this state the engine running a flow progresses through the
flow.
**SUCCESS** - Transitioned to once all of the flows atoms have finished
successfully.
**REVERTED** - Transitioned to once all of the flows atoms have been reverted
successfully after a failure.
**FAILURE** - The engine will transition the flow to this state when it can not
be reverted after a single failure or after multiple failures (greater than
one failure *may* occur when running in parallel).
**SUSPENDING** - In the ``RUNNING`` state the engine running the flow can be
suspended. When this happens, the engine attempts to transition the flow
to the ``SUSPENDING`` state immediately. In that state the engine running the
flow waits for running atoms to finish (since the engine can not preempt
atoms that are actively running).
**SUSPENDED** - When no atoms are running and all results received so far have
been saved, the engine transitions the flow from the ``SUSPENDING`` state
to the ``SUSPENDED`` state.
.. note::
The engine may transition the flow to the ``SUCCESS`` state (from the
``SUSPENDING`` state) if all atoms were in fact running (and completed)
before the suspension request was able to be honored (this is due to the lack
of preemption) or to the ``REVERTED`` state if the engine was reverting and
all atoms were reverted while the engine was waiting for running atoms to
finish or to the ``FAILURE`` state if atoms were running or reverted and
some of them had failed.
**RESUMING** - When the engine running a flow is interrupted *'in a
hard way'* (e.g. server crashed), it can be loaded from storage in *any*
state (this is required since it is can not be known what state was last
successfully saved). If the loaded state is not ``PENDING`` (aka, the flow was
never ran) or ``SUCCESS``, ``FAILURE`` or ``REVERTED`` (in which case the flow
has already finished), the flow gets set to the ``RESUMING`` state for the
short time period while it is being loaded from backend storage [a database, a
filesystem...] (this transition is not shown on the diagram). When the flow is
finally loaded, it goes to the ``SUSPENDED`` state.
From the ``SUCCESS``, ``FAILURE`` or ``REVERTED`` states the flow can be ran
again; therefore it is allowable to go back into the ``RUNNING`` state
immediately. One of the possible use cases for this transition is to allow for
alteration of a flow or flow details associated with a previously ran flow
after the flow has finished, and client code wants to ensure that each atom
from this new (potentially updated) flow has its chance to run.
Task
====
.. image:: img/task_states.svg
:width: 660px
:align: center
:alt: Task state transitions
**PENDING** - A task starts its execution lifecycle in this state (it has no
state prior to being ran by an engine, since tasks(s) are just objects that
represent how to accomplish a piece of work). Once it has been transitioned to
the ``PENDING`` state by the engine this means it can be executed immediately
or if needed will wait for all of the atoms it depends on to complete.
.. note::
An engine running a task also transitions the task to the ``PENDING`` state
after it was reverted and its containing flow was restarted or retried.
**IGNORE** - When a conditional decision has been made to skip (not
execute) the task the engine will transition the task to
the ``IGNORE`` state.
**RUNNING** - When an engine running the task starts to execute the task, the
engine will transition the task to the ``RUNNING`` state, and the task will
stay in this state until the tasks :py:meth:`~taskflow.atom.Atom.execute`
method returns.
**SUCCESS** - The engine running the task transitions the task to this state
after the task has finished successfully (ie no exception/s were raised during
running its :py:meth:`~taskflow.atom.Atom.execute` method).
**FAILURE** - The engine running the task transitions the task to this state
after it has finished with an error (ie exception/s were raised during
running its :py:meth:`~taskflow.atom.Atom.execute` method).
**REVERT_FAILURE** - The engine running the task transitions the task to this
state after it has finished with an error (ie exception/s were raised during
running its :py:meth:`~taskflow.atom.Atom.revert` method).
**REVERTING** - The engine running a task transitions the task to this state
when the containing flow the engine is running starts to revert and
its :py:meth:`~taskflow.atom.Atom.revert` method is called. Only tasks in
the ``SUCCESS`` or ``FAILURE`` state can be reverted. If this method fails (ie
raises an exception), the task goes to the ``REVERT_FAILURE`` state.
**REVERTED** - The engine running the task transitions the task to this state
after it has successfully reverted the task (ie no exception/s were raised
during running its :py:meth:`~taskflow.atom.Atom.revert` method).
Retry
=====
.. note::
A retry has the same states as a task and one additional state.
.. image:: img/retry_states.svg
:width: 660px
:align: center
:alt: Retry state transitions
**PENDING** - A retry starts its execution lifecycle in this state (it has no
state prior to being ran by an engine, since retry(s) are just objects that
represent how to retry an associated flow). Once it has been transitioned to
the ``PENDING`` state by the engine this means it can be executed immediately
or if needed will wait for all of the atoms it depends on to complete (in the
retry case the retry object will also be consulted when failures occur in the
flow that the retry is associated with by consulting its
:py:meth:`~taskflow.retry.Decider.on_failure` method).
.. note::
An engine running a retry also transitions the retry to the ``PENDING`` state
after it was reverted and its associated flow was restarted or retried.
**IGNORE** - When a conditional decision has been made to skip (not
execute) the retry the engine will transition the retry to
the ``IGNORE`` state.
**RUNNING** - When an engine starts to execute the retry, the engine
transitions the retry to the ``RUNNING`` state, and the retry stays in this
state until its :py:meth:`~taskflow.retry.Retry.execute` method returns.
**SUCCESS** - The engine running the retry transitions it to this state after
it was finished successfully (ie no exception/s were raised during
execution).
**FAILURE** - The engine running the retry transitions the retry to this state
after it has finished with an error (ie exception/s were raised during
running its :py:meth:`~taskflow.retry.Retry.execute` method).
**REVERT_FAILURE** - The engine running the retry transitions the retry to
this state after it has finished with an error (ie exception/s were raised
during its :py:meth:`~taskflow.retry.Retry.revert` method).
**REVERTING** - The engine running the retry transitions to this state when
the associated flow the engine is running starts to revert it and its
:py:meth:`~taskflow.retry.Retry.revert` method is called. Only retries
in ``SUCCESS`` or ``FAILURE`` state can be reverted. If this method fails (ie
raises an exception), the retry goes to the ``REVERT_FAILURE`` state.
**REVERTED** - The engine running the retry transitions the retry to this state
after it has successfully reverted the retry (ie no exception/s were raised
during running its :py:meth:`~taskflow.retry.Retry.revert` method).
**RETRYING** - If flow that is associated with the current retry was failed and
reverted, the engine prepares the flow for the next run and transitions the
retry to the ``RETRYING`` state.
Jobs
====
.. image:: img/job_states.svg
:width: 500px
:align: center
:alt: Job state transitions
**UNCLAIMED** - A job (with details about what work is to be completed) has
been initially posted (by some posting entity) for work on by some other
entity (for example a :doc:`conductor <conductors>`). This can also be a state
that is entered when some owning entity has manually abandoned (or
lost ownership of) a previously claimed job.
**CLAIMED** - A job that is *actively* owned by some entity; typically that
ownership is tied to jobs persistent data via some ephemeral connection so
that the job ownership is lost (typically automatically or after some
timeout) if that ephemeral connection is lost.
**COMPLETE** - The work defined in the job has been finished by its owning
entity and the job can no longer be processed (and it *may* be removed at
some/any point in the future).

View File

@ -1,50 +0,0 @@
-----
Types
-----
.. note::
Even though these types **are** made for public consumption and usage
should be encouraged/easily possible it should be noted that these may be
moved out to new libraries at various points in the future. If you are
using these types **without** using the rest of this library it is
**strongly** encouraged that you be a vocal proponent of getting these made
into *isolated* libraries (as using these types in this manner is not
the expected and/or desired usage).
Entity
======
.. automodule:: taskflow.types.entity
Failure
=======
.. automodule:: taskflow.types.failure
Graph
=====
.. automodule:: taskflow.types.graph
Notifier
========
.. automodule:: taskflow.types.notifier
:special-members: __call__
Sets
====
.. automodule:: taskflow.types.sets
Timing
======
.. automodule:: taskflow.types.timing
Tree
====
.. automodule:: taskflow.types.tree

View File

@ -1,69 +0,0 @@
---------
Utilities
---------
.. warning::
External usage of internal utility functions and modules should be kept
to a **minimum** as they may be altered, refactored or moved to other
locations **without** notice (and without the typical deprecation cycle).
Async
~~~~~
.. automodule:: taskflow.utils.async_utils
Banner
~~~~~~
.. automodule:: taskflow.utils.banner
Eventlet
~~~~~~~~
.. automodule:: taskflow.utils.eventlet_utils
Iterators
~~~~~~~~~
.. automodule:: taskflow.utils.iter_utils
Kazoo
~~~~~
.. automodule:: taskflow.utils.kazoo_utils
Kombu
~~~~~
.. automodule:: taskflow.utils.kombu_utils
Miscellaneous
~~~~~~~~~~~~~
.. automodule:: taskflow.utils.misc
Mixins
~~~~~~
.. automodule:: taskflow.utils.mixins
Persistence
~~~~~~~~~~~
.. automodule:: taskflow.utils.persistence_utils
Redis
~~~~~
.. automodule:: taskflow.utils.redis_utils
Schema
~~~~~~
.. automodule:: taskflow.utils.schema_utils
Threading
~~~~~~~~~
.. automodule:: taskflow.utils.threading_utils

View File

@ -1,438 +0,0 @@
Overview
========
This is engine that schedules tasks to **workers** -- separate processes
dedicated for certain atoms execution, possibly running on other machines,
connected via `amqp`_ (or other supported `kombu`_ transports).
.. note::
This engine is under active development and is usable and **does** work
but is missing some features (please check the `blueprint page`_ for
known issues and plans) that will make it more production ready.
.. _blueprint page: https://blueprints.launchpad.net/taskflow?searchtext=wbe
Terminology
-----------
Client
Code or program or service (or user) that uses this library to define
flows and run them via engines.
Transport + protocol
Mechanism (and `protocol`_ on top of that mechanism) used to pass information
between the client and worker (for example amqp as a transport and a json
encoded message format as the protocol).
Executor
Part of the worker-based engine and is used to publish task requests, so
these requests can be accepted and processed by remote workers.
Worker
Workers are started on remote hosts and each has a list of tasks it can
perform (on request). Workers accept and process task requests that are
published by an executor. Several requests can be processed simultaneously
in separate threads (or processes...). For example, an `executor`_ can be
passed to the worker and configured to run in as many threads (green or
not) as desired.
Proxy
Executors interact with workers via a proxy. The proxy maintains the
underlying transport and publishes messages (and invokes callbacks on message
reception).
Requirements
------------
* **Transparent:** it should work as ad-hoc replacement for existing
*(local)* engines with minimal, if any refactoring (e.g. it should be
possible to run the same flows on it without changing client code if
everything is set up and configured properly).
* **Transport-agnostic:** the means of transport should be abstracted so that
we can use `oslo.messaging`_, `gearmand`_, `amqp`_, `zookeeper`_, `marconi`_,
`websockets`_ or anything else that allows for passing information between a
client and a worker.
* **Simple:** it should be simple to write and deploy.
* **Non-uniformity:** it should support non-uniform workers which allows
different workers to execute different sets of atoms depending on the workers
published capabilities.
.. _marconi: https://wiki.openstack.org/wiki/Marconi
.. _zookeeper: http://zookeeper.org/
.. _gearmand: http://gearman.org/
.. _oslo.messaging: https://wiki.openstack.org/wiki/Oslo/Messaging
.. _websockets: http://en.wikipedia.org/wiki/WebSocket
.. _amqp: http://www.amqp.org/
.. _executor: https://docs.python.org/dev/library/concurrent.futures.html#executor-objects
.. _protocol: http://en.wikipedia.org/wiki/Communications_protocol
Design
======
There are two communication sides, the *executor* (and associated engine
derivative) and *worker* that communicate using a proxy component. The proxy
is designed to accept/publish messages from/into a named exchange.
High level architecture
-----------------------
.. image:: img/worker-engine.svg
:height: 340px
:align: right
Executor and worker communication
---------------------------------
Let's consider how communication between an executor and a worker happens.
First of all an engine resolves all atoms dependencies and schedules atoms that
can be performed at the moment. This uses the same scheduling and dependency
resolution logic that is used for every other engine type. Then the atoms which
can be executed immediately (ones that are dependent on outputs of other tasks
will be executed when that output is ready) are executed by the worker-based
engine executor in the following manner:
1. The executor initiates task execution/reversion using a proxy object.
2. :py:class:`~taskflow.engines.worker_based.proxy.Proxy` publishes task
request (format is described below) into a named exchange using a routing
key that is used to deliver request to particular workers topic. The
executor then waits for the task requests to be accepted and confirmed by
workers. If the executor doesn't get a task confirmation from workers within
the given timeout the task is considered as timed-out and a timeout
exception is raised.
3. A worker receives a request message and starts a new thread for processing
it.
1. The worker dispatches the request (gets desired endpoint that actually
executes the task).
2. If dispatched succeeded then the worker sends a confirmation response
to the executor otherwise the worker sends a failed response along with
a serialized :py:class:`failure <taskflow.types.failure.Failure>` object
that contains what has failed (and why).
3. The worker executes the task and once it is finished sends the result
back to the originating executor (every time a task progress event is
triggered it sends progress notification to the executor where it is
handled by the engine, dispatching to listeners and so-on).
4. The executor gets the task request confirmation from the worker and the task
request state changes from the ``PENDING`` to the ``RUNNING`` state. Once a
task request is in the ``RUNNING`` state it can't be timed-out (considering
that the task execution process may take an unpredictable amount of time).
5. The executor gets the task execution result from the worker and passes it
back to the executor and worker-based engine to finish task processing (this
repeats for subsequent tasks).
.. note::
:py:class:`~taskflow.types.failure.Failure` objects are not directly
json-serializable (they contain references to tracebacks which are not
serializable), so they are converted to dicts before sending and converted
from dicts after receiving on both executor & worker sides (this
translation is lossy since the traceback can't be fully retained, due
to its contents containing internal interpreter references and
details).
Protocol
~~~~~~~~
.. automodule:: taskflow.engines.worker_based.protocol
Examples
~~~~~~~~
Request (execute)
"""""""""""""""""
* **task_name** - full task name to be performed
* **task_cls** - full task class name to be performed
* **action** - task action to be performed (e.g. execute, revert)
* **arguments** - arguments the task action to be called with
* **result** - task execution result (result or
:py:class:`~taskflow.types.failure.Failure`) *[passed to revert only]*
Additionally, the following parameters are added to the request message:
* **reply_to** - executor named exchange workers will send responses back to
* **correlation_id** - executor request id (since there can be multiple request
being processed simultaneously)
**Example:**
.. code:: json
{
"action": "execute",
"arguments": {
"x": 111
},
"task_cls": "taskflow.tests.utils.TaskOneArgOneReturn",
"task_name": "taskflow.tests.utils.TaskOneArgOneReturn",
"task_version": [
1,
0
]
}
Request (revert)
""""""""""""""""
When **reverting:**
.. code:: json
{
"action": "revert",
"arguments": {},
"failures": {
"taskflow.tests.utils.TaskWithFailure": {
"exc_type_names": [
"RuntimeError",
"StandardError",
"Exception"
],
"exception_str": "Woot!",
"traceback_str": " File \"/homes/harlowja/dev/os/taskflow/taskflow/engines/action_engine/executor.py\", line 56, in _execute_task\n result = task.execute(**arguments)\n File \"/homes/harlowja/dev/os/taskflow/taskflow/tests/utils.py\", line 165, in execute\n raise RuntimeError('Woot!')\n",
"version": 1
}
},
"result": [
"failure",
{
"exc_type_names": [
"RuntimeError",
"StandardError",
"Exception"
],
"exception_str": "Woot!",
"traceback_str": " File \"/homes/harlowja/dev/os/taskflow/taskflow/engines/action_engine/executor.py\", line 56, in _execute_task\n result = task.execute(**arguments)\n File \"/homes/harlowja/dev/os/taskflow/taskflow/tests/utils.py\", line 165, in execute\n raise RuntimeError('Woot!')\n",
"version": 1
}
],
"task_cls": "taskflow.tests.utils.TaskWithFailure",
"task_name": "taskflow.tests.utils.TaskWithFailure",
"task_version": [
1,
0
]
}
Worker response(s)
""""""""""""""""""
When **running:**
.. code:: json
{
"data": {},
"state": "RUNNING"
}
When **progressing:**
.. code:: json
{
"details": {
"progress": 0.5
},
"event_type": "update_progress",
"state": "EVENT"
}
When **succeeded:**
.. code:: json
{
"data": {
"result": 666
},
"state": "SUCCESS"
}
When **failed:**
.. code:: json
{
"data": {
"result": {
"exc_type_names": [
"RuntimeError",
"StandardError",
"Exception"
],
"exception_str": "Woot!",
"traceback_str": " File \"/homes/harlowja/dev/os/taskflow/taskflow/engines/action_engine/executor.py\", line 56, in _execute_task\n result = task.execute(**arguments)\n File \"/homes/harlowja/dev/os/taskflow/taskflow/tests/utils.py\", line 165, in execute\n raise RuntimeError('Woot!')\n",
"version": 1
}
},
"state": "FAILURE"
}
Request state transitions
-------------------------
.. image:: img/wbe_request_states.svg
:width: 520px
:align: center
:alt: WBE request state transitions
**WAITING** - Request placed on queue (or other `kombu`_ message bus/transport)
but not *yet* consumed.
**PENDING** - Worker accepted request and is pending to run using its
executor (threads, processes, or other).
**FAILURE** - Worker failed after running request (due to task exception) or
no worker moved/started executing (by placing the request into ``RUNNING``
state) with-in specified time span (this defaults to 60 seconds unless
overridden).
**RUNNING** - Workers executor (using threads, processes...) has started to
run requested task (once this state is transitioned to any request timeout no
longer becomes applicable; since at this point it is unknown how long a task
will run since it can not be determined if a task is just taking a long time
or has failed).
**SUCCESS** - Worker finished running task without exception.
.. note::
During the ``WAITING`` and ``PENDING`` stages the engine keeps track
of how long the request has been *alive* for and if a timeout is reached
the request will automatically transition to ``FAILURE`` and any further
transitions from a worker will be disallowed (for example, if a worker
accepts the request in the future and sets the task to ``PENDING`` this
transition will be logged and ignored). This timeout can be adjusted and/or
removed by setting the engine ``transition_timeout`` option to a
higher/lower value or by setting it to ``None`` (to remove the timeout
completely). In the future this will be improved to be more dynamic
by implementing the blueprints associated with `failover`_ and
`info/resilence`_.
.. _failover: https://blueprints.launchpad.net/taskflow/+spec/wbe-worker-failover
.. _info/resilence: https://blueprints.launchpad.net/taskflow/+spec/wbe-worker-info
Usage
=====
Workers
-------
To use the worker based engine a set of workers must first be established on
remote machines. These workers must be provided a list of task objects, task
names, modules names (or entrypoints that can be examined for valid tasks) they
can respond to (this is done so that arbitrary code execution is not possible).
For complete parameters and object usage please visit
:py:class:`~taskflow.engines.worker_based.worker.Worker`.
**Example:**
.. code:: python
from taskflow.engines.worker_based import worker as w
config = {
'url': 'amqp://guest:guest@localhost:5672//',
'exchange': 'test-exchange',
'topic': 'test-tasks',
'tasks': ['tasks:TestTask1', 'tasks:TestTask2'],
}
worker = w.Worker(**config)
worker.run()
Engines
-------
To use the worker based engine a flow must be constructed (which contains tasks
that are visible on remote machines) and the specific worker based engine
entrypoint must be selected. Certain configuration options must also be
provided so that the transport backend can be configured and initialized
correctly. Otherwise the usage should be mostly transparent (and is nearly
identical to using any other engine type).
For complete parameters and object usage please see
:py:class:`~taskflow.engines.worker_based.engine.WorkerBasedActionEngine`.
**Example with amqp transport:**
.. code:: python
flow = lf.Flow('simple-linear').add(...)
eng = taskflow.engines.load(flow, engine='worker-based',
url='amqp://guest:guest@localhost:5672//',
exchange='test-exchange',
topics=['topic1', 'topic2'])
eng.run()
**Example with filesystem transport:**
.. code:: python
flow = lf.Flow('simple-linear').add(...)
eng = taskflow.engines.load(flow, engine='worker-based',
exchange='test-exchange',
topics=['topic1', 'topic2'],
transport='filesystem',
transport_options={
'data_folder_in': '/tmp/in',
'data_folder_out': '/tmp/out',
})
eng.run()
Additional supported keyword arguments:
* ``executor``: a class that provides a
:py:class:`~taskflow.engines.worker_based.executor.WorkerTaskExecutor`
interface; it will be used for executing, reverting and waiting for remote
tasks.
Limitations
===========
* Atoms inside a flow must receive and accept parameters only from the ways
defined in :doc:`persistence <persistence>`. In other words, the task
that is created when a workflow is constructed will not be the same task that
is executed on a remote worker (and any internal state not passed via the
:doc:`input and output <inputs_and_outputs>` mechanism can not be
transferred). This means resource objects (database handles, file
descriptors, sockets, ...) can **not** be directly sent across to remote
workers (instead the configuration that defines how to fetch/create these
objects must be instead).
* Worker-based engines will in the future be able to run lightweight tasks
locally to avoid transport overhead for very simple tasks (currently it will
run even lightweight tasks remotely, which may be non-performant).
* Fault detection, currently when a worker acknowledges a task the engine will
wait for the task result indefinitely (a task may take an indeterminate
amount of time to finish). In the future there needs to be a way to limit
the duration of a remote workers execution (and track their liveness) and
possibly spawn the task on a secondary worker if a timeout is reached (aka
the first worker has died or has stopped responding).
Implementations
===============
.. automodule:: taskflow.engines.worker_based.engine
Components
----------
.. warning::
External usage of internal engine functions, components and modules should
be kept to a **minimum** as they may be altered, refactored or moved to
other locations **without** notice (and without the typical deprecation
cycle).
.. automodule:: taskflow.engines.worker_based.dispatcher
.. automodule:: taskflow.engines.worker_based.endpoint
.. automodule:: taskflow.engines.worker_based.executor
.. automodule:: taskflow.engines.worker_based.proxy
.. automodule:: taskflow.engines.worker_based.worker
.. automodule:: taskflow.engines.worker_based.types
.. _kombu: http://kombu.readthedocs.org/

View File

@ -1,34 +0,0 @@
[MESSAGES CONTROL]
# Disable the message(s) with the given id(s).
disable=C0111,I0011,R0201,R0922,W0142,W0511,W0613,W0622,W0703
[BASIC]
# Variable names can be 1 to 31 characters long, with lowercase and underscores
variable-rgx=[a-z_][a-z0-9_]{0,30}$
# Argument names can be 2 to 31 characters long, with lowercase and underscores
argument-rgx=[a-z_][a-z0-9_]{1,30}$
# Method names should be at least 3 characters long
# and be lowercased with underscores
method-rgx=[a-z_][a-z0-9_]{2,50}$
# Don't require docstrings on tests.
no-docstring-rgx=((__.*__)|([tT]est.*)|setUp|tearDown)$
[DESIGN]
max-args=10
max-attributes=20
max-branchs=30
max-public-methods=100
max-statements=60
min-public-methods=0
[REPORTS]
output-format=parseable
include-ids=yes
[VARIABLES]
additional-builtins=_

View File

@ -1,286 +0,0 @@
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# taskflow Release Notes documentation build configuration file, created by
# sphinx-quickstart on Tue Nov 3 17:40:50 2015.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
# sys.path.insert(0, os.path.abspath('.'))
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'openstackdocstheme',
'reno.sphinxext',
]
# openstackdocstheme options
repository_name = 'openstack/taskflow'
bug_project = 'taskflow'
bug_tag = ''
html_last_updated_fmt = '%Y-%m-%d %H:%M'
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
# source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'taskflow Release Notes'
copyright = u'2016, taskflow Developers'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
from taskflow import version as taskflow_version
# The full version, including alpha/beta/rc tags.
release = taskflow_version.version_string()
# The short X.Y version.
version = release
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
# language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
# today = ''
# Else, today_fmt is used as the format for a strftime call.
# today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = []
# The reST default role (used for this markup: `text`) to use for all
# documents.
# default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
# add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
# add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
# show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
# modindex_common_prefix = []
# If true, keep warnings as "system message" paragraphs in the built documents.
# keep_warnings = False
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'openstackdocs'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
# html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
# html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
# html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
# html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
# html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
# html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
# directly to the root of the documentation.
# html_extra_path = []
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
# html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
# html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
# html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
# html_additional_pages = {}
# If false, no module index is generated.
# html_domain_indices = True
# If false, no index is generated.
# html_use_index = True
# If true, the index is split into individual pages for each letter.
# html_split_index = False
# If true, links to the reST sources are added to the pages.
# html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
# html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
# html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
# html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
# html_file_suffix = None
# Output file base name for HTML help builder.
htmlhelp_basename = 'taskflowReleaseNotesdoc'
# -- Options for LaTeX output ---------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
# 'preamble': '',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
('index', 'taskflowReleaseNotes.tex',
u'taskflow Release Notes Documentation',
u'taskflow Developers', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
# latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
# latex_use_parts = False
# If true, show page references after internal links.
# latex_show_pagerefs = False
# If true, show URL addresses after external links.
# latex_show_urls = False
# Documents to append as an appendix to all manuals.
# latex_appendices = []
# If false, no module index is generated.
# latex_domain_indices = True
# -- Options for manual page output ---------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
('index', 'taskflowreleasenotes',
u'taskflow Release Notes Documentation',
[u'taskflow Developers'], 1)
]
# If true, show URL addresses after external links.
# man_show_urls = False
# -- Options for Texinfo output -------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
('index', 'taskflowReleaseNotes',
u'taskflow Release Notes Documentation',
u'taskflow Developers', 'taskflowReleaseNotes',
'An OpenStack library for parsing configuration options from the command'
' line and configuration files.',
'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
# texinfo_appendices = []
# If false, no module index is generated.
# texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
# texinfo_show_urls = 'footnote'
# If true, do not generate a @detailmenu in the "Top" node's menu.
# texinfo_no_detailmenu = False
# -- Options for Internationalization output ------------------------------
locale_dirs = ['locale/']

View File

@ -1,9 +0,0 @@
===========================
taskflow Release Notes
===========================
.. toctree::
:maxdepth: 1
unreleased
ocata

View File

@ -1,6 +0,0 @@
===================================
Ocata Series Release Notes
===================================
.. release-notes::
:branch: origin/stable/ocata

View File

@ -1,5 +0,0 @@
==========================
Unreleased Release Notes
==========================
.. release-notes::

View File

@ -1,49 +0,0 @@
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
# See: https://bugs.launchpad.net/pbr/+bug/1384919 for why this is here...
pbr!=2.1.0,>=2.0.0 # Apache-2.0
# Packages needed for using this library.
# Python 2->3 compatibility library.
six>=1.9.0 # MIT
# Enum library made for <= python 3.3
enum34;python_version=='2.7' or python_version=='2.6' or python_version=='3.3' # BSD
# For async and/or periodic work
futurist!=0.15.0,>=0.11.0 # Apache-2.0
# For reader/writer + interprocess locks.
fasteners>=0.7 # Apache-2.0
# Very nice graph library
networkx>=1.10 # BSD
# For contextlib new additions/compatibility for <= python 3.3
contextlib2>=0.4.0 # PSF License
# Used for backend storage engine loading.
stevedore>=1.20.0 # Apache-2.0
# Backport for concurrent.futures which exists in 3.2+
futures>=3.0;python_version=='2.7' or python_version=='2.6' # BSD
# Used for structured input validation
jsonschema!=2.5.0,<3.0.0,>=2.0.0 # MIT
# For the state machine we run with
automaton>=0.5.0 # Apache-2.0
# For common utilities
oslo.utils>=3.20.0 # Apache-2.0
oslo.serialization!=2.19.1,>=1.10.0 # Apache-2.0
tenacity>=3.2.1 # Apache-2.0
# For lru caches and such
cachetools>=1.1.0 # MIT License
# For deprecation of things
debtcollector>=1.2.0 # Apache-2.0

View File

@ -1,88 +0,0 @@
#!/bin/bash
function usage {
echo "Usage: $0 [OPTION]..."
echo "Run Taskflow's test suite(s)"
echo ""
echo " -f, --force Force a clean re-build of the virtual environment. Useful when dependencies have been added."
echo " -p, --pep8 Just run pep8"
echo " -P, --no-pep8 Don't run static code checks"
echo " -v, --verbose Increase verbosity of reporting output"
echo " -h, --help Print this usage message"
echo ""
exit
}
function process_option {
case "$1" in
-h|--help) usage;;
-p|--pep8) let just_pep8=1;;
-P|--no-pep8) let no_pep8=1;;
-f|--force) let force=1;;
-v|--verbose) let verbose=1;;
*) pos_args="$pos_args $1"
esac
}
verbose=0
force=0
pos_args=""
just_pep8=0
no_pep8=0
tox_args=""
tox=""
for arg in "$@"; do
process_option $arg
done
py=`which python`
if [ -z "$py" ]; then
echo "Python is required to use $0"
echo "Please install it via your distributions package management system."
exit 1
fi
py_envs=`python -c 'import sys; print("py%s%s" % (sys.version_info[0:2]))'`
py_envs=${PY_ENVS:-$py_envs}
function run_tests {
local tox_cmd="${tox} ${tox_args} -e $py_envs ${pos_args}"
echo "Running tests for environments $py_envs via $tox_cmd"
bash -c "$tox_cmd"
}
function run_flake8 {
local tox_cmd="${tox} ${tox_args} -e pep8 ${pos_args}"
echo "Running flake8 via $tox_cmd"
bash -c "$tox_cmd"
}
if [ $force -eq 1 ]; then
tox_args="$tox_args -r"
fi
if [ $verbose -eq 1 ]; then
tox_args="$tox_args -v"
fi
tox=`which tox`
if [ -z "$tox" ]; then
echo "Tox is required to use $0"
echo "Please install it via \`pip\` or via your distributions" \
"package management system."
echo "Visit http://tox.readthedocs.org/ for additional installation" \
"instructions."
exit 1
fi
if [ $just_pep8 -eq 1 ]; then
run_flake8
exit
fi
run_tests || exit
if [ $no_pep8 -eq 0 ]; then
run_flake8
fi

105
setup.cfg
View File

@ -1,105 +0,0 @@
[metadata]
name = taskflow
summary = Taskflow structured state management library.
description-file =
README.rst
author = OpenStack
author-email = openstack-dev@lists.openstack.org
home-page = https://docs.openstack.org/taskflow/latest/
keywords = reliable,tasks,execution,parallel,dataflow,workflows,distributed
classifier =
Development Status :: 4 - Beta
Environment :: OpenStack
Intended Audience :: Developers
Intended Audience :: Information Technology
License :: OSI Approved :: Apache Software License
Operating System :: POSIX :: Linux
Programming Language :: Python
Programming Language :: Python :: 2
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
Programming Language :: Python :: 3.4
Programming Language :: Python :: 3.5
Topic :: Software Development :: Libraries
Topic :: System :: Distributed Computing
[global]
setup-hooks =
pbr.hooks.setup_hook
[files]
packages =
taskflow
[entry_points]
taskflow.jobboards =
zookeeper = taskflow.jobs.backends.impl_zookeeper:ZookeeperJobBoard
redis = taskflow.jobs.backends.impl_redis:RedisJobBoard
taskflow.conductors =
blocking = taskflow.conductors.backends.impl_blocking:BlockingConductor
nonblocking = taskflow.conductors.backends.impl_nonblocking:NonBlockingConductor
taskflow.persistence =
dir = taskflow.persistence.backends.impl_dir:DirBackend
file = taskflow.persistence.backends.impl_dir:DirBackend
memory = taskflow.persistence.backends.impl_memory:MemoryBackend
mysql = taskflow.persistence.backends.impl_sqlalchemy:SQLAlchemyBackend
postgresql = taskflow.persistence.backends.impl_sqlalchemy:SQLAlchemyBackend
sqlite = taskflow.persistence.backends.impl_sqlalchemy:SQLAlchemyBackend
zookeeper = taskflow.persistence.backends.impl_zookeeper:ZkBackend
taskflow.engines =
default = taskflow.engines.action_engine.engine:SerialActionEngine
serial = taskflow.engines.action_engine.engine:SerialActionEngine
parallel = taskflow.engines.action_engine.engine:ParallelActionEngine
worker-based = taskflow.engines.worker_based.engine:WorkerBasedActionEngine
workers = taskflow.engines.worker_based.engine:WorkerBasedActionEngine
[extras]
zookeeper =
kazoo>=2.2 # Apache-2.0
zake>=0.1.6 # Apache-2.0
redis =
redis>=2.10.0 # MIT
workers =
kombu!=4.0.2,>=4.0.0 # BSD
eventlet =
eventlet!=0.18.3,!=0.20.1,<0.21.0,>=0.18.2 # MIT
doc =
sphinx>=1.6.2 # BSD
openstackdocstheme>=1.11.0 # Apache-2.0
database =
SQLAlchemy!=1.1.5,!=1.1.6,!=1.1.7,!=1.1.8,>=1.0.10 # MIT
alembic>=0.8.10 # MIT
SQLAlchemy-Utils # BSD License
PyMySQL>=0.7.6 # MIT License
psycopg2>=2.5 # LGPL/ZPL
test =
pydotplus>=2.0.2 # MIT License
hacking<0.11,>=0.10.0
oslotest>=1.10.0 # Apache-2.0
mock>=2.0 # BSD
testtools>=1.4.0 # MIT
testscenarios>=0.4 # Apache-2.0/BSD
doc8 # Apache-2.0
reno!=2.3.1,>=1.8.0 # Apache-2.0
[nosetests]
cover-erase = true
verbosity = 2
[pbr]
warnerrors = True
[wheel]
universal = 1
[build_sphinx]
source-dir = doc/source
build-dir = doc/build
all_files = 1
warning-is-error = 1
[upload_sphinx]
upload-dir = doc/build/html

View File

@ -1,29 +0,0 @@
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT
import setuptools
# In python < 2.7.4, a lazy loading of package `pbr` will break
# setuptools if some other modules registered functions in `atexit`.
# solution from: http://bugs.python.org/issue15881#msg170215
try:
import multiprocessing # noqa
except ImportError:
pass
setuptools.setup(
setup_requires=['pbr>=2.0.0'],
pbr=True)

View File

View File

@ -1,385 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2013 Rackspace Hosting Inc. All Rights Reserved.
# Copyright (C) 2013 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import abc
import collections
import itertools
from oslo_utils import reflection
import six
from six.moves import zip as compat_zip
from taskflow.types import sets
from taskflow.utils import misc
# Helper types tuples...
_sequence_types = (list, tuple, collections.Sequence)
_set_types = (set, collections.Set)
# the default list of revert arguments to ignore when deriving
# revert argument mapping from the revert method signature
_default_revert_args = ('result', 'flow_failures')
def _save_as_to_mapping(save_as):
"""Convert save_as to mapping name => index.
Result should follow storage convention for mappings.
"""
# TODO(harlowja): we should probably document this behavior & convention
# outside of code so that it's more easily understandable, since what an
# atom returns is pretty crucial for other later operations.
if save_as is None:
return collections.OrderedDict()
if isinstance(save_as, six.string_types):
# NOTE(harlowja): this means that your atom will only return one item
# instead of a dictionary-like object or a indexable object (like a
# list or tuple).
return collections.OrderedDict([(save_as, None)])
elif isinstance(save_as, _sequence_types):
# NOTE(harlowja): this means that your atom will return a indexable
# object, like a list or tuple and the results can be mapped by index
# to that tuple/list that is returned for others to use.
return collections.OrderedDict((key, num)
for num, key in enumerate(save_as))
elif isinstance(save_as, _set_types):
# NOTE(harlowja): in the case where a set is given we will not be
# able to determine the numeric ordering in a reliable way (since it
# may be an unordered set) so the only way for us to easily map the
# result of the atom will be via the key itself.
return collections.OrderedDict((key, key) for key in save_as)
else:
raise TypeError('Atom provides parameter '
'should be str, set or tuple/list, not %r' % save_as)
def _build_rebind_dict(req_args, rebind_args):
"""Build a argument remapping/rebinding dictionary.
This dictionary allows an atom to declare that it will take a needed
requirement bound to a given name with another name instead (mapping the
new name onto the required name).
"""
if rebind_args is None:
return collections.OrderedDict()
elif isinstance(rebind_args, (list, tuple)):
# Attempt to map the rebound argument names position by position to
# the required argument names (if they are the same length then
# this determines how to remap the required argument names to the
# rebound ones).
rebind = collections.OrderedDict(compat_zip(req_args, rebind_args))
if len(req_args) < len(rebind_args):
# Extra things were rebound, that may be because of *args
# or **kwargs (or some other reason); so just keep all of them
# using 1:1 rebinding...
rebind.update((a, a) for a in rebind_args[len(req_args):])
return rebind
elif isinstance(rebind_args, dict):
return rebind_args
else:
raise TypeError("Invalid rebind value '%s' (%s)"
% (rebind_args, type(rebind_args)))
def _build_arg_mapping(atom_name, reqs, rebind_args, function, do_infer,
ignore_list=None):
"""Builds an input argument mapping for a given function.
Given a function, its requirements and a rebind mapping this helper
function will build the correct argument mapping for the given function as
well as verify that the final argument mapping does not have missing or
extra arguments (where applicable).
"""
# Build a list of required arguments based on function signature.
req_args = reflection.get_callable_args(function, required_only=True)
all_args = reflection.get_callable_args(function, required_only=False)
# Remove arguments that are part of ignore list.
if ignore_list:
for arg in ignore_list:
if arg in req_args:
req_args.remove(arg)
else:
ignore_list = []
# Build the required names.
required = collections.OrderedDict()
# Add required arguments to required mappings if inference is enabled.
if do_infer:
required.update((a, a) for a in req_args)
# Add additional manually provided requirements to required mappings.
if reqs:
if isinstance(reqs, six.string_types):
required.update({reqs: reqs})
else:
required.update((a, a) for a in reqs)
# Update required mappings values based on rebinding of arguments names.
required.update(_build_rebind_dict(req_args, rebind_args))
# Determine if there are optional arguments that we may or may not take.
if do_infer:
opt_args = sets.OrderedSet(all_args)
opt_args = opt_args - set(itertools.chain(six.iterkeys(required),
iter(ignore_list)))
optional = collections.OrderedDict((a, a) for a in opt_args)
else:
optional = collections.OrderedDict()
# Check if we are given some extra arguments that we aren't able to accept.
if not reflection.accepts_kwargs(function):
extra_args = sets.OrderedSet(six.iterkeys(required))
extra_args -= all_args
if extra_args:
raise ValueError('Extra arguments given to atom %s: %s'
% (atom_name, list(extra_args)))
# NOTE(imelnikov): don't use set to preserve order in error message
missing_args = [arg for arg in req_args if arg not in required]
if missing_args:
raise ValueError('Missing arguments for atom %s: %s'
% (atom_name, missing_args))
return required, optional
@six.add_metaclass(abc.ABCMeta)
class Atom(object):
"""An unit of work that causes a flow to progress (in some manner).
An atom is a named object that operates with input data to perform
some action that furthers the overall flows progress. It usually also
produces some of its own named output as a result of this process.
:param name: Meaningful name for this atom, should be something that is
distinguishable and understandable for notification,
debugging, storing and any other similar purposes.
:param provides: A set, string or list of items that
this will be providing (or could provide) to others, used
to correlate and associate the thing/s this atom
produces, if it produces anything at all.
:param inject: An *immutable* input_name => value dictionary which
specifies any initial inputs that should be automatically
injected into the atoms scope before the atom execution
commences (this allows for providing atom *local* values
that do not need to be provided by other atoms/dependents).
:param rebind: A dict of key/value pairs used to define argument
name conversions for inputs to this atom's ``execute``
method.
:param revert_rebind: The same as ``rebind`` but for the ``revert``
method. If unpassed, ``rebind`` will be used
instead.
:param requires: A set or list of required inputs for this atom's
``execute`` method.
:param revert_requires: A set or list of required inputs for this atom's
``revert`` method. If unpassed, ```requires`` will
be used.
:ivar version: An *immutable* version that associates version information
with this atom. It can be useful in resuming older versions
of atoms. Standard major, minor versioning concepts
should apply.
:ivar save_as: An *immutable* output ``resource`` name
:py:class:`.OrderedDict` this atom produces that other
atoms may depend on this atom providing. The format is
output index (or key when a dictionary is returned from
the execute method) to stored argument name.
:ivar rebind: An *immutable* input ``resource`` :py:class:`.OrderedDict`
that can be used to alter the inputs given to this atom. It
is typically used for mapping a prior atoms output into
the names that this atom expects (in a way this is like
remapping a namespace of another atom into the namespace
of this atom).
:ivar revert_rebind: The same as ``rebind`` but for the revert method. This
should only differ from ``rebind`` if the ``revert``
method has a different signature from ``execute`` or
a different ``revert_rebind`` value was received.
:ivar inject: See parameter ``inject``.
:ivar Atom.name: See parameter ``name``.
:ivar requires: A :py:class:`~taskflow.types.sets.OrderedSet` of inputs
this atom requires to function.
:ivar optional: A :py:class:`~taskflow.types.sets.OrderedSet` of inputs
that are optional for this atom to ``execute``.
:ivar revert_optional: The ``revert`` version of ``optional``.
:ivar provides: A :py:class:`~taskflow.types.sets.OrderedSet` of outputs
this atom produces.
"""
priority = 0
"""A numeric priority that instances of this class will have when running,
used when there are multiple *parallel* candidates to execute and/or
revert. During this situation the candidate list will be stably sorted
based on this priority attribute which will result in atoms with higher
priorities executing (or reverting) before atoms with lower
priorities (higher being defined as a number bigger, or greater tha
an atom with a lower priority number). By default all atoms have the same
priority (zero).
For example when the following is combined into a
graph (where each node in the denoted graph is some task)::
a -> b
b -> c
b -> e
b -> f
When ``b`` finishes there will then be three candidates that can run
``(c, e, f)`` and they may run in any order. What this priority does is
sort those three by their priority before submitting them to be
worked on (so that instead of say a random run order they will now be
ran by there sorted order). This is also true when reverting (in that the
sort order of the potential nodes will be used to determine the
submission order).
"""
default_provides = None
def __init__(self, name=None, provides=None, requires=None,
auto_extract=True, rebind=None, inject=None,
ignore_list=None, revert_rebind=None, revert_requires=None):
if provides is None:
provides = self.default_provides
self.name = name
self.version = (1, 0)
self.inject = inject
self.save_as = _save_as_to_mapping(provides)
self.provides = sets.OrderedSet(self.save_as)
if ignore_list is None:
ignore_list = []
self.rebind, exec_requires, self.optional = self._build_arg_mapping(
self.execute,
requires=requires,
rebind=rebind, auto_extract=auto_extract,
ignore_list=ignore_list
)
revert_ignore = ignore_list + list(_default_revert_args)
revert_mapping = self._build_arg_mapping(
self.revert,
requires=revert_requires or requires,
rebind=revert_rebind or rebind,
auto_extract=auto_extract,
ignore_list=revert_ignore
)
(self.revert_rebind, addl_requires,
self.revert_optional) = revert_mapping
self.requires = exec_requires.union(addl_requires)
def _build_arg_mapping(self, executor, requires=None, rebind=None,
auto_extract=True, ignore_list=None):
required, optional = _build_arg_mapping(self.name, requires, rebind,
executor, auto_extract,
ignore_list=ignore_list)
# Form the real rebind mapping, if a key name is the same as the
# key value, then well there is no rebinding happening, otherwise
# there will be.
rebind = collections.OrderedDict()
for (arg_name, bound_name) in itertools.chain(six.iteritems(required),
six.iteritems(optional)):
rebind.setdefault(arg_name, bound_name)
requires = sets.OrderedSet(six.itervalues(required))
optional = sets.OrderedSet(six.itervalues(optional))
if self.inject:
inject_keys = frozenset(six.iterkeys(self.inject))
requires -= inject_keys
optional -= inject_keys
return rebind, requires, optional
def pre_execute(self):
"""Code to be run prior to executing the atom.
A common pattern for initializing the state of the system prior to
running atoms is to define some code in a base class that all your
atoms inherit from. In that class, you can define a ``pre_execute``
method and it will always be invoked just prior to your atoms running.
"""
@abc.abstractmethod
def execute(self, *args, **kwargs):
"""Activate a given atom which will perform some operation and return.
This method can be used to perform an action on a given set of input
requirements (passed in via ``*args`` and ``**kwargs``) to accomplish
some type of operation. This operation may provide some named
outputs/results as a result of it executing for later reverting (or for
other atoms to depend on).
NOTE(harlowja): the result (if any) that is returned should be
persistable so that it can be passed back into this atom if
reverting is triggered (especially in the case where reverting
happens in a different python process or on a remote machine) and so
that the result can be transmitted to other atoms (which may be local
or remote).
:param args: positional arguments that atom requires to execute.
:param kwargs: any keyword arguments that atom requires to execute.
"""
def post_execute(self):
"""Code to be run after executing the atom.
A common pattern for cleaning up global state of the system after the
execution of atoms is to define some code in a base class that all your
atoms inherit from. In that class, you can define a ``post_execute``
method and it will always be invoked just after your atoms execute,
regardless of whether they succeeded or not.
This pattern is useful if you have global shared database sessions
that need to be cleaned up, for example.
"""
def pre_revert(self):
"""Code to be run prior to reverting the atom.
This works the same as :meth:`.pre_execute`, but for the revert phase.
"""
def revert(self, *args, **kwargs):
"""Revert this atom.
This method should undo any side-effects caused by previous execution
of the atom using the result of the :py:meth:`execute` method and
information on the failure which triggered reversion of the flow the
atom is contained in (if applicable).
:param args: positional arguments that the atom required to execute.
:param kwargs: any keyword arguments that the atom required to
execute; the special key ``'result'`` will contain
the :py:meth:`execute` result (if any) and
the ``**kwargs`` key ``'flow_failures'`` will contain
any failure information.
"""
def post_revert(self):
"""Code to be run after reverting the atom.
This works the same as :meth:`.post_execute`, but for the revert phase.
"""
def __str__(self):
return "%s==%s" % (self.name, misc.get_version_string(self))
def __repr__(self):
return '<%s %s>' % (reflection.get_class_name(self), self)

View File

@ -1,45 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2014 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import logging
import stevedore.driver
from taskflow import exceptions as exc
# NOTE(harlowja): this is the entrypoint namespace, not the module namespace.
CONDUCTOR_NAMESPACE = 'taskflow.conductors'
LOG = logging.getLogger(__name__)
def fetch(kind, name, jobboard, namespace=CONDUCTOR_NAMESPACE, **kwargs):
"""Fetch a conductor backend with the given options.
This fetch method will look for the entrypoint 'kind' in the entrypoint
namespace, and then attempt to instantiate that entrypoint using the
provided name, jobboard and any board specific kwargs.
"""
LOG.debug('Looking for %r conductor driver in %r', kind, namespace)
try:
mgr = stevedore.driver.DriverManager(
namespace, kind,
invoke_on_load=True,
invoke_args=(name, jobboard),
invoke_kwds=kwargs)
return mgr.driver
except RuntimeError as e:
raise exc.NotFound("Could not find conductor %s" % (kind), e)

View File

@ -1,41 +0,0 @@
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import futurist
from taskflow.conductors.backends import impl_executor
class BlockingConductor(impl_executor.ExecutorConductor):
"""Blocking conductor that processes job(s) in a blocking manner."""
MAX_SIMULTANEOUS_JOBS = 1
"""
Default maximum number of jobs that can be in progress at the same time.
"""
@staticmethod
def _executor_factory():
return futurist.SynchronousExecutor()
def __init__(self, name, jobboard,
persistence=None, engine=None,
engine_options=None, wait_timeout=None,
log=None, max_simultaneous_jobs=MAX_SIMULTANEOUS_JOBS):
super(BlockingConductor, self).__init__(
name, jobboard,
persistence=persistence, engine=engine,
engine_options=engine_options,
wait_timeout=wait_timeout, log=log,
max_simultaneous_jobs=max_simultaneous_jobs)

View File

@ -1,357 +0,0 @@
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import abc
import functools
import itertools
import threading
try:
from contextlib import ExitStack # noqa
except ImportError:
from contextlib2 import ExitStack # noqa
from debtcollector import removals
from oslo_utils import excutils
from oslo_utils import timeutils
import six
from taskflow.conductors import base
from taskflow import exceptions as excp
from taskflow.listeners import logging as logging_listener
from taskflow import logging
from taskflow import states
from taskflow.types import timing as tt
from taskflow.utils import iter_utils
from taskflow.utils import misc
LOG = logging.getLogger(__name__)
@six.add_metaclass(abc.ABCMeta)
class ExecutorConductor(base.Conductor):
"""Dispatches jobs from blocking :py:meth:`.run` method to some executor.
This conductor iterates over jobs in the provided jobboard (waiting for
the given timeout if no jobs exist) and attempts to claim them, work on
those jobs using an executor (potentially blocking further work from being
claimed and consumed) and then consume those work units after
completion. This process will repeat until the conductor has been stopped
or other critical error occurs.
NOTE(harlowja): consumption occurs even if a engine fails to run due to
a atom failure. This is only skipped when an execution failure or
a storage failure occurs which are *usually* correctable by re-running on
a different conductor (storage failures and execution failures may be
transient issues that can be worked around by later execution). If a job
after completing can not be consumed or abandoned the conductor relies
upon the jobboard capabilities to automatically abandon these jobs.
"""
LOG = None
"""
Logger that will be used for listening to events (if none then the module
level logger will be used instead).
"""
REFRESH_PERIODICITY = 30
"""
Every 30 seconds the jobboard will be resynced (if for some reason
a watch or set of watches was not received) using the `ensure_fresh`
option to ensure this (for supporting jobboard backends only).
"""
#: Default timeout used to idle/wait when no jobs have been found.
WAIT_TIMEOUT = 0.5
MAX_SIMULTANEOUS_JOBS = -1
"""
Default maximum number of jobs that can be in progress at the same time.
Negative or zero values imply no limit (do note that if a executor is
used that is built on a queue, as most are, that this will imply that the
queue will contain a potentially large & unfinished backlog of
submitted jobs). This *may* get better someday if
https://bugs.python.org/issue22737 is ever implemented and released.
"""
#: Exceptions that will **not** cause consumption to occur.
NO_CONSUME_EXCEPTIONS = tuple([
excp.ExecutionFailure,
excp.StorageFailure,
])
_event_factory = threading.Event
"""This attribute *can* be overridden by subclasses (for example if
an eventlet *green* event works better for the conductor user)."""
EVENTS_EMITTED = tuple([
'compilation_start', 'compilation_end',
'preparation_start', 'preparation_end',
'validation_start', 'validation_end',
'running_start', 'running_end',
'job_consumed', 'job_abandoned',
])
"""Events will be emitted for each of the events above. The event is
emitted to listeners registered with the conductor.
"""
def __init__(self, name, jobboard,
persistence=None, engine=None,
engine_options=None, wait_timeout=None,
log=None, max_simultaneous_jobs=MAX_SIMULTANEOUS_JOBS):
super(ExecutorConductor, self).__init__(
name, jobboard, persistence=persistence,
engine=engine, engine_options=engine_options)
self._wait_timeout = tt.convert_to_timeout(
value=wait_timeout, default_value=self.WAIT_TIMEOUT,
event_factory=self._event_factory)
self._dead = self._event_factory()
self._log = misc.pick_first_not_none(log, self.LOG, LOG)
self._max_simultaneous_jobs = int(
misc.pick_first_not_none(max_simultaneous_jobs,
self.MAX_SIMULTANEOUS_JOBS))
self._dispatched = set()
def _executor_factory(self):
"""Creates an executor to be used during dispatching."""
raise excp.NotImplementedError("This method must be implemented but"
" it has not been")
@removals.removed_kwarg('timeout', version="0.8", removal_version="2.0")
def stop(self, timeout=None):
"""Requests the conductor to stop dispatching.
This method can be used to request that a conductor stop its
consumption & dispatching loop.
The method returns immediately regardless of whether the conductor has
been stopped.
"""
self._wait_timeout.interrupt()
@property
def dispatching(self):
"""Whether or not the dispatching loop is still dispatching."""
return not self._dead.is_set()
def _listeners_from_job(self, job, engine):
listeners = super(ExecutorConductor, self)._listeners_from_job(
job, engine)
listeners.append(logging_listener.LoggingListener(engine,
log=self._log))
return listeners
def _dispatch_job(self, job):
engine = self._engine_from_job(job)
listeners = self._listeners_from_job(job, engine)
with ExitStack() as stack:
for listener in listeners:
stack.enter_context(listener)
self._log.debug("Dispatching engine for job '%s'", job)
consume = True
details = {
'job': job,
'engine': engine,
'conductor': self,
}
def _run_engine():
has_suspended = False
for _state in engine.run_iter():
if not has_suspended and self._wait_timeout.is_stopped():
self._log.info("Conductor stopped, requesting "
"suspension of engine running "
"job %s", job)
engine.suspend()
has_suspended = True
try:
for stage_func, event_name in [(engine.compile, 'compilation'),
(engine.prepare, 'preparation'),
(engine.validate, 'validation'),
(_run_engine, 'running')]:
self._notifier.notify("%s_start" % event_name, details)
stage_func()
self._notifier.notify("%s_end" % event_name, details)
except excp.WrappedFailure as e:
if all((f.check(*self.NO_CONSUME_EXCEPTIONS) for f in e)):
consume = False
if self._log.isEnabledFor(logging.WARNING):
if consume:
self._log.warn(
"Job execution failed (consumption being"
" skipped): %s [%s failures]", job, len(e))
else:
self._log.warn(
"Job execution failed (consumption"
" proceeding): %s [%s failures]", job, len(e))
# Show the failure/s + traceback (if possible)...
for i, f in enumerate(e):
self._log.warn("%s. %s", i + 1,
f.pformat(traceback=True))
except self.NO_CONSUME_EXCEPTIONS:
self._log.warn("Job execution failed (consumption being"
" skipped): %s", job, exc_info=True)
consume = False
except Exception:
self._log.warn(
"Job execution failed (consumption proceeding): %s",
job, exc_info=True)
else:
if engine.storage.get_flow_state() == states.SUSPENDED:
self._log.info("Job execution was suspended: %s", job)
consume = False
else:
self._log.info("Job completed successfully: %s", job)
return consume
def _try_finish_job(self, job, consume):
try:
if consume:
self._jobboard.consume(job, self._name)
self._notifier.notify("job_consumed", {
'job': job,
'conductor': self,
'persistence': self._persistence,
})
else:
self._jobboard.abandon(job, self._name)
self._notifier.notify("job_abandoned", {
'job': job,
'conductor': self,
'persistence': self._persistence,
})
except (excp.JobFailure, excp.NotFound):
if consume:
self._log.warn("Failed job consumption: %s", job,
exc_info=True)
else:
self._log.warn("Failed job abandonment: %s", job,
exc_info=True)
def _on_job_done(self, job, fut):
consume = False
try:
consume = fut.result()
except KeyboardInterrupt:
with excutils.save_and_reraise_exception():
self._log.warn("Job dispatching interrupted: %s", job)
except Exception:
self._log.warn("Job dispatching failed: %s", job, exc_info=True)
try:
self._try_finish_job(job, consume)
finally:
self._dispatched.discard(fut)
def _can_claim_more_jobs(self, job):
if self._wait_timeout.is_stopped():
return False
if self._max_simultaneous_jobs <= 0:
return True
if len(self._dispatched) >= self._max_simultaneous_jobs:
return False
else:
return True
def _run_until_dead(self, executor, max_dispatches=None):
total_dispatched = 0
if max_dispatches is None:
# NOTE(TheSriram): if max_dispatches is not set,
# then the conductor will run indefinitely, and not
# stop after 'n' number of dispatches
max_dispatches = -1
dispatch_gen = iter_utils.iter_forever(max_dispatches)
is_stopped = self._wait_timeout.is_stopped
try:
# Don't even do any work in the first place...
if max_dispatches == 0:
raise StopIteration
fresh_period = timeutils.StopWatch(
duration=self.REFRESH_PERIODICITY)
fresh_period.start()
while not is_stopped():
any_dispatched = False
if fresh_period.expired():
ensure_fresh = True
fresh_period.restart()
else:
ensure_fresh = False
job_it = itertools.takewhile(
self._can_claim_more_jobs,
self._jobboard.iterjobs(ensure_fresh=ensure_fresh))
for job in job_it:
self._log.debug("Trying to claim job: %s", job)
try:
self._jobboard.claim(job, self._name)
except (excp.UnclaimableJob, excp.NotFound):
self._log.debug("Job already claimed or"
" consumed: %s", job)
else:
try:
fut = executor.submit(self._dispatch_job, job)
except RuntimeError:
with excutils.save_and_reraise_exception():
self._log.warn("Job dispatch submitting"
" failed: %s", job)
self._try_finish_job(job, False)
else:
fut.job = job
self._dispatched.add(fut)
any_dispatched = True
fut.add_done_callback(
functools.partial(self._on_job_done, job))
total_dispatched = next(dispatch_gen)
if not any_dispatched and not is_stopped():
self._wait_timeout.wait()
except StopIteration:
# This will be raised from 'dispatch_gen' if it reaches its
# max dispatch number (which implies we should do no more work).
with excutils.save_and_reraise_exception():
if max_dispatches >= 0 and total_dispatched >= max_dispatches:
self._log.info("Maximum dispatch limit of %s reached",
max_dispatches)
def run(self, max_dispatches=None):
self._dead.clear()
self._dispatched.clear()
try:
self._jobboard.register_entity(self.conductor)
with self._executor_factory() as executor:
self._run_until_dead(executor,
max_dispatches=max_dispatches)
except StopIteration:
pass
except KeyboardInterrupt:
with excutils.save_and_reraise_exception():
self._log.warn("Job dispatching interrupted")
finally:
self._dead.set()
# Inherit the docs, so we can reference them in our class docstring,
# if we don't do this sphinx gets confused...
run.__doc__ = base.Conductor.run.__doc__
def wait(self, timeout=None):
"""Waits for the conductor to gracefully exit.
This method waits for the conductor to gracefully exit. An optional
timeout can be provided, which will cause the method to return
within the specified timeout. If the timeout is reached, the returned
value will be ``False``, otherwise it will be ``True``.
:param timeout: Maximum number of seconds that the :meth:`wait` method
should block for.
"""
return self._dead.wait(timeout)

View File

@ -1,69 +0,0 @@
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import futurist
import six
from taskflow.conductors.backends import impl_executor
from taskflow.utils import threading_utils as tu
class NonBlockingConductor(impl_executor.ExecutorConductor):
"""Non-blocking conductor that processes job(s) using a thread executor.
NOTE(harlowja): A custom executor factory can be provided via keyword
argument ``executor_factory``, if provided it will be
invoked at
:py:meth:`~taskflow.conductors.base.Conductor.run` time
with one positional argument (this conductor) and it must
return a compatible `executor`_ which can be used
to submit jobs to. If ``None`` is a provided a thread pool
backed executor is selected by default (it will have
an equivalent number of workers as this conductors
simultaneous job count).
.. _executor: https://docs.python.org/dev/library/\
concurrent.futures.html#executor-objects
"""
MAX_SIMULTANEOUS_JOBS = tu.get_optimal_thread_count()
"""
Default maximum number of jobs that can be in progress at the same time.
"""
def _default_executor_factory(self):
max_simultaneous_jobs = self._max_simultaneous_jobs
if max_simultaneous_jobs <= 0:
max_workers = tu.get_optimal_thread_count()
else:
max_workers = max_simultaneous_jobs
return futurist.ThreadPoolExecutor(max_workers=max_workers)
def __init__(self, name, jobboard,
persistence=None, engine=None,
engine_options=None, wait_timeout=None,
log=None, max_simultaneous_jobs=MAX_SIMULTANEOUS_JOBS,
executor_factory=None):
super(NonBlockingConductor, self).__init__(
name, jobboard,
persistence=persistence, engine=engine,
engine_options=engine_options, wait_timeout=wait_timeout,
log=log, max_simultaneous_jobs=max_simultaneous_jobs)
if executor_factory is None:
self._executor_factory = self._default_executor_factory
else:
if not six.callable(executor_factory):
raise ValueError("Provided keyword argument 'executor_factory'"
" must be callable")
self._executor_factory = executor_factory

View File

@ -1,180 +0,0 @@
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import abc
import os
import threading
import fasteners
import six
from taskflow import engines
from taskflow import exceptions as excp
from taskflow.types import entity
from taskflow.types import notifier
from taskflow.utils import misc
@six.add_metaclass(abc.ABCMeta)
class Conductor(object):
"""Base for all conductor implementations.
Conductors act as entities which extract jobs from a jobboard, assign
there work to some engine (using some desired configuration) and then wait
for that work to complete. If the work fails then they abandon the claimed
work (or if the process they are running in crashes or dies this
abandonment happens automatically) and then another conductor at a later
period of time will finish up the prior failed conductors work.
"""
#: Entity kind used when creating new entity objects
ENTITY_KIND = 'conductor'
def __init__(self, name, jobboard,
persistence=None, engine=None, engine_options=None):
self._name = name
self._jobboard = jobboard
self._engine = engine
self._engine_options = misc.safe_copy_dict(engine_options)
self._persistence = persistence
self._lock = threading.RLock()
self._notifier = notifier.Notifier()
@misc.cachedproperty
def conductor(self):
"""Entity object that represents this conductor."""
hostname = misc.get_hostname()
pid = os.getpid()
name = '@'.join([self._name, hostname + ":" + str(pid)])
metadata = {
'hostname': hostname,
'pid': pid,
}
return entity.Entity(self.ENTITY_KIND, name, metadata)
@property
def notifier(self):
"""The conductor actions (or other state changes) notifier.
NOTE(harlowja): different conductor implementations may emit
different events + event details at different times, so refer to your
conductor documentation to know exactly what can and what can not be
subscribed to.
"""
return self._notifier
def _flow_detail_from_job(self, job):
"""Extracts a flow detail from a job (via some manner).
The current mechanism to accomplish this is the following choices:
* If the job details provide a 'flow_uuid' key attempt to load this
key from the jobs book and use that as the flow_detail to run.
* If the job details does not have have a 'flow_uuid' key then attempt
to examine the size of the book and if it's only one element in the
book (aka one flow_detail) then just use that.
* Otherwise if there is no 'flow_uuid' defined or there are > 1
flow_details in the book raise an error that corresponds to being
unable to locate the correct flow_detail to run.
"""
book = job.book
if book is None:
raise excp.NotFound("No book found in job")
if job.details and 'flow_uuid' in job.details:
flow_uuid = job.details["flow_uuid"]
flow_detail = book.find(flow_uuid)
if flow_detail is None:
raise excp.NotFound("No matching flow detail found in"
" jobs book for flow detail"
" with uuid %s" % flow_uuid)
else:
choices = len(book)
if choices == 1:
flow_detail = list(book)[0]
elif choices == 0:
raise excp.NotFound("No flow detail(s) found in jobs book")
else:
raise excp.MultipleChoices("No matching flow detail found (%s"
" choices) in jobs book" % choices)
return flow_detail
def _engine_from_job(self, job):
"""Extracts an engine from a job (via some manner)."""
flow_detail = self._flow_detail_from_job(job)
store = {}
if flow_detail.meta and 'store' in flow_detail.meta:
store.update(flow_detail.meta['store'])
if job.details and 'store' in job.details:
store.update(job.details["store"])
engine = engines.load_from_detail(flow_detail, store=store,
engine=self._engine,
backend=self._persistence,
**self._engine_options)
return engine
def _listeners_from_job(self, job, engine):
"""Returns a list of listeners to be attached to an engine.
This method should be overridden in order to attach listeners to
engines. It will be called once for each job, and the list returned
listeners will be added to the engine for this job.
:param job: A job instance that is about to be run in an engine.
:param engine: The engine that listeners will be attached to.
:returns: a list of (unregistered) listener instances.
"""
# TODO(dkrause): Create a standard way to pass listeners or
# listener factories over the jobboard
return []
@fasteners.locked
def connect(self):
"""Ensures the jobboard is connected (noop if it is already)."""
if not self._jobboard.connected:
self._jobboard.connect()
@fasteners.locked
def close(self):
"""Closes the contained jobboard, disallowing further use."""
self._jobboard.close()
@abc.abstractmethod
def run(self, max_dispatches=None):
"""Continuously claims, runs, and consumes jobs (and repeat).
:param max_dispatches: An upper bound on the number of jobs that will
be dispatched, if none or negative this implies
there is no limit to the number of jobs that
will be dispatched, otherwise if positive this
run method will return when that amount of jobs
has been dispatched (instead of running
forever and/or until stopped).
"""
@abc.abstractmethod
def _dispatch_job(self, job):
"""Dispatches a claimed job for work completion.
Accepts a single (already claimed) job and causes it to be run in
an engine. Returns a future object that represented the work to be
completed sometime in the future. The future should return a single
boolean from its result() method. This boolean determines whether the
job will be consumed (true) or whether it should be abandoned (false).
:param job: A job instance that has already been claimed by the
jobboard.
"""

View File

@ -1,99 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2012 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import six
from taskflow.utils import misc
class Depth(misc.StrEnum):
"""Enumeration of decider(s) *area of influence*."""
ALL = 'ALL'
"""
**Default** decider depth that affects **all** successor atoms (including
ones that are in successor nested flows).
"""
FLOW = 'FLOW'
"""
Decider depth that affects **all** successor tasks in the **same**
flow (it will **not** affect tasks/retries that are in successor
nested flows).
.. warning::
While using this kind we are allowed to execute successors of
things that have been ignored (for example nested flows and the
tasks they contain), this may result in symbol lookup errors during
running, user beware.
"""
NEIGHBORS = 'NEIGHBORS'
"""
Decider depth that affects only **next** successor tasks (and does
not traverse past **one** level of successor tasks).
.. warning::
While using this kind we are allowed to execute successors of
things that have been ignored (for example nested flows and the
tasks they contain), this may result in symbol lookup errors during
running, user beware.
"""
ATOM = 'ATOM'
"""
Decider depth that affects only **targeted** atom (and does
**not** traverse into **any** level of successor atoms).
.. warning::
While using this kind we are allowed to execute successors of
things that have been ignored (for example nested flows and the
tasks they contain), this may result in symbol lookup errors during
running, user beware.
"""
@classmethod
def translate(cls, desired_depth):
"""Translates a string into a depth enumeration."""
if isinstance(desired_depth, cls):
# Nothing to do in the first place...
return desired_depth
if not isinstance(desired_depth, six.string_types):
raise TypeError("Unexpected desired depth type, string type"
" expected, not %s" % type(desired_depth))
try:
return cls(desired_depth.upper())
except ValueError:
pretty_depths = sorted([a_depth.name for a_depth in cls])
raise ValueError("Unexpected decider depth value, one of"
" %s (case-insensitive) is expected and"
" not '%s'" % (pretty_depths, desired_depth))
# Depth area of influence order (from greater influence to least).
#
# Order very much matters here...
_ORDERING = tuple([
Depth.ALL, Depth.FLOW, Depth.NEIGHBORS, Depth.ATOM,
])
def pick_widest(depths):
"""Pick from many depths which has the **widest** area of influence."""
return _ORDERING[min(_ORDERING.index(d) for d in depths)]

View File

@ -1,32 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2012 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from oslo_utils import eventletutils as _eventletutils
# Give a nice warning that if eventlet is being used these modules
# are highly recommended to be patched (or otherwise bad things could
# happen).
_eventletutils.warn_eventlet_not_patched(
expected_patched_modules=['time', 'thread'])
# Promote helpers to this module namespace (for easy access).
from taskflow.engines.helpers import flow_from_detail # noqa
from taskflow.engines.helpers import load # noqa
from taskflow.engines.helpers import load_from_detail # noqa
from taskflow.engines.helpers import load_from_factory # noqa
from taskflow.engines.helpers import run # noqa
from taskflow.engines.helpers import save_factory_details # noqa

View File

@ -1,55 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2015 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import abc
import six
from taskflow import states
@six.add_metaclass(abc.ABCMeta)
class Action(object):
"""An action that handles executing, state changes, ... of atoms."""
NO_RESULT = object()
"""
Sentinel use to represent lack of any result (none can be a valid result)
"""
#: States that are expected to have a result to save...
SAVE_RESULT_STATES = (states.SUCCESS, states.FAILURE,
states.REVERTED, states.REVERT_FAILURE)
def __init__(self, storage, notifier):
self._storage = storage
self._notifier = notifier
@abc.abstractmethod
def schedule_execution(self, atom):
"""Schedules atom execution."""
@abc.abstractmethod
def schedule_reversion(self, atom):
"""Schedules atom reversion."""
@abc.abstractmethod
def complete_reversion(self, atom, result):
"""Completes atom reversion."""
@abc.abstractmethod
def complete_execution(self, atom, result):
"""Completes atom execution."""

View File

@ -1,104 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2012-2013 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from taskflow.engines.action_engine.actions import base
from taskflow import retry as retry_atom
from taskflow import states
from taskflow.types import failure
class RetryAction(base.Action):
"""An action that handles executing, state changes, ... of retry atoms."""
def __init__(self, storage, notifier, retry_executor):
super(RetryAction, self).__init__(storage, notifier)
self._retry_executor = retry_executor
def _get_retry_args(self, retry, revert=False, addons=None):
if revert:
arguments = self._storage.fetch_mapped_args(
retry.revert_rebind,
atom_name=retry.name,
optional_args=retry.revert_optional
)
else:
arguments = self._storage.fetch_mapped_args(
retry.rebind,
atom_name=retry.name,
optional_args=retry.optional
)
history = self._storage.get_retry_history(retry.name)
arguments[retry_atom.EXECUTE_REVERT_HISTORY] = history
if addons:
arguments.update(addons)
return arguments
def change_state(self, retry, state, result=base.Action.NO_RESULT):
old_state = self._storage.get_atom_state(retry.name)
if state in self.SAVE_RESULT_STATES:
save_result = None
if result is not self.NO_RESULT:
save_result = result
self._storage.save(retry.name, save_result, state)
# TODO(harlowja): combine this with the save to avoid a call
# back into the persistence layer...
if state == states.REVERTED:
self._storage.cleanup_retry_history(retry.name, state)
else:
if state == old_state:
# NOTE(imelnikov): nothing really changed, so we should not
# write anything to storage and run notifications.
return
self._storage.set_atom_state(retry.name, state)
retry_uuid = self._storage.get_atom_uuid(retry.name)
details = {
'retry_name': retry.name,
'retry_uuid': retry_uuid,
'old_state': old_state,
}
if result is not self.NO_RESULT:
details['result'] = result
self._notifier.notify(state, details)
def schedule_execution(self, retry):
self.change_state(retry, states.RUNNING)
return self._retry_executor.execute_retry(
retry, self._get_retry_args(retry))
def complete_reversion(self, retry, result):
if isinstance(result, failure.Failure):
self.change_state(retry, states.REVERT_FAILURE, result=result)
else:
self.change_state(retry, states.REVERTED, result=result)
def complete_execution(self, retry, result):
if isinstance(result, failure.Failure):
self.change_state(retry, states.FAILURE, result=result)
else:
self.change_state(retry, states.SUCCESS, result=result)
def schedule_reversion(self, retry):
self.change_state(retry, states.REVERTING)
arg_addons = {
retry_atom.REVERT_FLOW_FAILURES: self._storage.get_failures(),
}
return self._retry_executor.revert_retry(
retry, self._get_retry_args(retry, addons=arg_addons, revert=True))
def on_failure(self, retry, atom, last_failure):
self._storage.save_retry_failure(retry.name, atom.name, last_failure)
arguments = self._get_retry_args(retry)
return retry.on_failure(**arguments)

View File

@ -1,146 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2012-2013 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import functools
from taskflow.engines.action_engine.actions import base
from taskflow import logging
from taskflow import states
from taskflow import task as task_atom
from taskflow.types import failure
LOG = logging.getLogger(__name__)
class TaskAction(base.Action):
"""An action that handles scheduling, state changes, ... of task atoms."""
def __init__(self, storage, notifier, task_executor):
super(TaskAction, self).__init__(storage, notifier)
self._task_executor = task_executor
def _is_identity_transition(self, old_state, state, task, progress=None):
if state in self.SAVE_RESULT_STATES:
# saving result is never identity transition
return False
if state != old_state:
# changing state is not identity transition by definition
return False
# NOTE(imelnikov): last thing to check is that the progress has
# changed, which means progress is not None and is different from
# what is stored in the database.
if progress is None:
return False
old_progress = self._storage.get_task_progress(task.name)
if old_progress != progress:
return False
return True
def change_state(self, task, state,
progress=None, result=base.Action.NO_RESULT):
old_state = self._storage.get_atom_state(task.name)
if self._is_identity_transition(old_state, state, task,
progress=progress):
# NOTE(imelnikov): ignore identity transitions in order
# to avoid extra write to storage backend and, what's
# more important, extra notifications.
return
if state in self.SAVE_RESULT_STATES:
save_result = None
if result is not self.NO_RESULT:
save_result = result
self._storage.save(task.name, save_result, state)
else:
self._storage.set_atom_state(task.name, state)
if progress is not None:
self._storage.set_task_progress(task.name, progress)
task_uuid = self._storage.get_atom_uuid(task.name)
details = {
'task_name': task.name,
'task_uuid': task_uuid,
'old_state': old_state,
}
if result is not self.NO_RESULT:
details['result'] = result
self._notifier.notify(state, details)
if progress is not None:
task.update_progress(progress)
def _on_update_progress(self, task, event_type, details):
"""Should be called when task updates its progress."""
try:
progress = details.pop('progress')
except KeyError:
pass
else:
try:
self._storage.set_task_progress(task.name, progress,
details=details)
except Exception:
# Update progress callbacks should never fail, so capture and
# log the emitted exception instead of raising it.
LOG.exception("Failed setting task progress for %s to %0.3f",
task, progress)
def schedule_execution(self, task):
self.change_state(task, states.RUNNING, progress=0.0)
arguments = self._storage.fetch_mapped_args(
task.rebind,
atom_name=task.name,
optional_args=task.optional
)
if task.notifier.can_be_registered(task_atom.EVENT_UPDATE_PROGRESS):
progress_callback = functools.partial(self._on_update_progress,
task)
else:
progress_callback = None
task_uuid = self._storage.get_atom_uuid(task.name)
return self._task_executor.execute_task(
task, task_uuid, arguments,
progress_callback=progress_callback)
def complete_execution(self, task, result):
if isinstance(result, failure.Failure):
self.change_state(task, states.FAILURE, result=result)
else:
self.change_state(task, states.SUCCESS,
result=result, progress=1.0)
def schedule_reversion(self, task):
self.change_state(task, states.REVERTING, progress=0.0)
arguments = self._storage.fetch_mapped_args(
task.revert_rebind,
atom_name=task.name,
optional_args=task.revert_optional
)
task_uuid = self._storage.get_atom_uuid(task.name)
task_result = self._storage.get(task.name)
failures = self._storage.get_failures()
if task.notifier.can_be_registered(task_atom.EVENT_UPDATE_PROGRESS):
progress_callback = functools.partial(self._on_update_progress,
task)
else:
progress_callback = None
return self._task_executor.revert_task(
task, task_uuid, arguments, task_result, failures,
progress_callback=progress_callback)
def complete_reversion(self, task, result):
if isinstance(result, failure.Failure):
self.change_state(task, states.REVERT_FAILURE, result=result)
else:
self.change_state(task, states.REVERTED, progress=1.0,
result=result)

View File

@ -1,370 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2012 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from concurrent import futures
import weakref
from automaton import machines
from oslo_utils import timeutils
from taskflow import logging
from taskflow import states as st
from taskflow.types import failure
from taskflow.utils import iter_utils
# Default waiting state timeout (in seconds).
WAITING_TIMEOUT = 60
# Meta states the state machine uses.
UNDEFINED = 'UNDEFINED'
GAME_OVER = 'GAME_OVER'
META_STATES = (GAME_OVER, UNDEFINED)
# Event name constants the state machine uses.
SCHEDULE = 'schedule_next'
WAIT = 'wait_finished'
ANALYZE = 'examine_finished'
FINISH = 'completed'
FAILED = 'failed'
SUSPENDED = 'suspended'
SUCCESS = 'success'
REVERTED = 'reverted'
START = 'start'
# Internal enums used to denote how/if a atom was completed."""
FAILED_COMPLETING = 'failed_completing'
WAS_CANCELLED = 'was_cancelled'
SUCCESSFULLY_COMPLETED = 'successfully_completed'
# For these states we will gather how long (in seconds) the
# state was in-progress (cumulatively if the state is entered multiple
# times)
TIMED_STATES = (st.ANALYZING, st.RESUMING, st.SCHEDULING, st.WAITING)
LOG = logging.getLogger(__name__)
class MachineMemory(object):
"""State machine memory."""
def __init__(self):
self.next_up = set()
self.not_done = set()
self.failures = []
self.done = set()
def cancel_futures(self):
"""Attempts to cancel any not done futures."""
for fut in self.not_done:
fut.cancel()
class MachineBuilder(object):
"""State machine *builder* that powers the engine components.
NOTE(harlowja): the machine (states and events that will trigger
transitions) that this builds is represented by the following
table::
+--------------+------------------+------------+----------+---------+
| Start | Event | End | On Enter | On Exit |
+--------------+------------------+------------+----------+---------+
| ANALYZING | completed | GAME_OVER | . | . |
| ANALYZING | schedule_next | SCHEDULING | . | . |
| ANALYZING | wait_finished | WAITING | . | . |
| FAILURE[$] | . | . | . | . |
| GAME_OVER | failed | FAILURE | . | . |
| GAME_OVER | reverted | REVERTED | . | . |
| GAME_OVER | success | SUCCESS | . | . |
| GAME_OVER | suspended | SUSPENDED | . | . |
| RESUMING | schedule_next | SCHEDULING | . | . |
| REVERTED[$] | . | . | . | . |
| SCHEDULING | wait_finished | WAITING | . | . |
| SUCCESS[$] | . | . | . | . |
| SUSPENDED[$] | . | . | . | . |
| UNDEFINED[^] | start | RESUMING | . | . |
| WAITING | examine_finished | ANALYZING | . | . |
+--------------+------------------+------------+----------+---------+
Between any of these yielded states (minus ``GAME_OVER`` and ``UNDEFINED``)
if the engine has been suspended or the engine has failed (due to a
non-resolveable task failure or scheduling failure) the machine will stop
executing new tasks (currently running tasks will be allowed to complete)
and this machines run loop will be broken.
NOTE(harlowja): If the runtimes scheduler component is able to schedule
tasks in parallel, this enables parallel running and/or reversion.
"""
def __init__(self, runtime, waiter):
self._runtime = weakref.proxy(runtime)
self._selector = runtime.selector
self._completer = runtime.completer
self._scheduler = runtime.scheduler
self._storage = runtime.storage
self._waiter = waiter
def build(self, statistics, timeout=None, gather_statistics=True):
"""Builds a state-machine (that is used during running)."""
if gather_statistics:
watches = {}
state_statistics = {}
statistics['seconds_per_state'] = state_statistics
watches = {}
for timed_state in TIMED_STATES:
state_statistics[timed_state.lower()] = 0.0
watches[timed_state] = timeutils.StopWatch()
statistics['discarded_failures'] = 0
statistics['awaiting'] = 0
statistics['completed'] = 0
statistics['incomplete'] = 0
memory = MachineMemory()
if timeout is None:
timeout = WAITING_TIMEOUT
# Cache some local functions/methods...
do_complete = self._completer.complete
do_complete_failure = self._completer.complete_failure
get_atom_intention = self._storage.get_atom_intention
def do_schedule(next_nodes):
with self._storage.lock.write_lock():
return self._scheduler.schedule(
sorted(next_nodes,
key=lambda node: getattr(node, 'priority', 0),
reverse=True))
def iter_next_atoms(atom=None, apply_deciders=True):
# Yields and filters and tweaks the next atoms to run...
maybe_atoms_it = self._selector.iter_next_atoms(atom=atom)
for atom, late_decider in maybe_atoms_it:
if apply_deciders:
proceed = late_decider.check_and_affect(self._runtime)
if proceed:
yield atom
else:
yield atom
def resume(old_state, new_state, event):
# This reaction function just updates the state machines memory
# to include any nodes that need to be executed (from a previous
# attempt, which may be empty if never ran before) and any nodes
# that are now ready to be ran.
with self._storage.lock.write_lock():
memory.next_up.update(
iter_utils.unique_seen((self._completer.resume(),
iter_next_atoms())))
return SCHEDULE
def game_over(old_state, new_state, event):
# This reaction function is mainly a intermediary delegation
# function that analyzes the current memory and transitions to
# the appropriate handler that will deal with the memory values,
# it is *always* called before the final state is entered.
if memory.failures:
return FAILED
with self._storage.lock.read_lock():
leftover_atoms = iter_utils.count(
# Avoid activating the deciders, since at this point
# the engine is finishing and there will be no more further
# work done anyway...
iter_next_atoms(apply_deciders=False))
if leftover_atoms:
# Ok we didn't finish (either reverting or executing...) so
# that means we must of been stopped at some point...
LOG.trace("Suspension determined to have been reacted to"
" since (at least) %s atoms have been left in an"
" unfinished state", leftover_atoms)
return SUSPENDED
elif self._runtime.is_success():
return SUCCESS
else:
return REVERTED
def schedule(old_state, new_state, event):
# This reaction function starts to schedule the memory's next
# nodes (iff the engine is still runnable, which it may not be
# if the user of this engine has requested the engine/storage
# that holds this information to stop or suspend); handles failures
# that occur during this process safely...
with self._storage.lock.write_lock():
current_flow_state = self._storage.get_flow_state()
if current_flow_state == st.RUNNING and memory.next_up:
not_done, failures = do_schedule(memory.next_up)
if not_done:
memory.not_done.update(not_done)
if failures:
memory.failures.extend(failures)
memory.next_up.intersection_update(not_done)
elif current_flow_state == st.SUSPENDING and memory.not_done:
# Try to force anything not cancelled to now be cancelled
# so that the executor that gets it does not continue to
# try to work on it (if the future execution is still in
# its backlog, if it's already being executed, this will
# do nothing).
memory.cancel_futures()
return WAIT
def complete_an_atom(fut):
# This completes a single atom saving its result in
# storage and preparing whatever predecessors or successors will
# now be ready to execute (or revert or retry...); it also
# handles failures that occur during this process safely...
atom = fut.atom
try:
outcome, result = fut.result()
do_complete(atom, outcome, result)
if isinstance(result, failure.Failure):
retain = do_complete_failure(atom, outcome, result)
if retain:
memory.failures.append(result)
else:
# NOTE(harlowja): avoid making any intention request
# to storage unless we are sure we are in DEBUG
# enabled logging (otherwise we will call this all
# the time even when DEBUG is not enabled, which
# would suck...)
if LOG.isEnabledFor(logging.DEBUG):
intention = get_atom_intention(atom.name)
LOG.debug("Discarding failure '%s' (in response"
" to outcome '%s') under completion"
" units request during completion of"
" atom '%s' (intention is to %s)",
result, outcome, atom, intention)
if gather_statistics:
statistics['discarded_failures'] += 1
if gather_statistics:
statistics['completed'] += 1
except futures.CancelledError:
# Well it got cancelled, skip doing anything
# and move on; at a further time it will be resumed
# and something should be done with it to get it
# going again.
return WAS_CANCELLED
except Exception:
memory.failures.append(failure.Failure())
LOG.exception("Engine '%s' atom post-completion"
" failed", atom)
return FAILED_COMPLETING
else:
return SUCCESSFULLY_COMPLETED
def wait(old_state, new_state, event):
# TODO(harlowja): maybe we should start doing 'yield from' this
# call sometime in the future, or equivalent that will work in
# py2 and py3.
if memory.not_done:
done, not_done = self._waiter(memory.not_done, timeout=timeout)
memory.done.update(done)
memory.not_done = not_done
return ANALYZE
def analyze(old_state, new_state, event):
# This reaction function is responsible for analyzing all nodes
# that have finished executing/reverting and figuring
# out what nodes are now ready to be ran (and then triggering those
# nodes to be scheduled in the future); handles failures that
# occur during this process safely...
next_up = set()
with self._storage.lock.write_lock():
while memory.done:
fut = memory.done.pop()
# Force it to be completed so that we can ensure that
# before we iterate over any successors or predecessors
# that we know it has been completed and saved and so on...
completion_status = complete_an_atom(fut)
if (not memory.failures
and completion_status != WAS_CANCELLED):
atom = fut.atom
try:
more_work = set(iter_next_atoms(atom=atom))
except Exception:
memory.failures.append(failure.Failure())
LOG.exception(
"Engine '%s' atom post-completion"
" next atom searching failed", atom)
else:
next_up.update(more_work)
current_flow_state = self._storage.get_flow_state()
if (current_flow_state == st.RUNNING
and next_up and not memory.failures):
memory.next_up.update(next_up)
return SCHEDULE
elif memory.not_done:
if current_flow_state == st.SUSPENDING:
memory.cancel_futures()
return WAIT
else:
return FINISH
def on_exit(old_state, event):
LOG.trace("Exiting old state '%s' in response to event '%s'",
old_state, event)
if gather_statistics:
if old_state in watches:
w = watches[old_state]
w.stop()
state_statistics[old_state.lower()] += w.elapsed()
if old_state in (st.SCHEDULING, st.WAITING):
statistics['incomplete'] = len(memory.not_done)
if old_state in (st.ANALYZING, st.SCHEDULING):
statistics['awaiting'] = len(memory.next_up)
def on_enter(new_state, event):
LOG.trace("Entering new state '%s' in response to event '%s'",
new_state, event)
if gather_statistics and new_state in watches:
watches[new_state].restart()
state_kwargs = {
'on_exit': on_exit,
'on_enter': on_enter,
}
m = machines.FiniteMachine()
m.add_state(GAME_OVER, **state_kwargs)
m.add_state(UNDEFINED, **state_kwargs)
m.add_state(st.ANALYZING, **state_kwargs)
m.add_state(st.RESUMING, **state_kwargs)
m.add_state(st.REVERTED, terminal=True, **state_kwargs)
m.add_state(st.SCHEDULING, **state_kwargs)
m.add_state(st.SUCCESS, terminal=True, **state_kwargs)
m.add_state(st.SUSPENDED, terminal=True, **state_kwargs)
m.add_state(st.WAITING, **state_kwargs)
m.add_state(st.FAILURE, terminal=True, **state_kwargs)
m.default_start_state = UNDEFINED
m.add_transition(GAME_OVER, st.REVERTED, REVERTED)
m.add_transition(GAME_OVER, st.SUCCESS, SUCCESS)
m.add_transition(GAME_OVER, st.SUSPENDED, SUSPENDED)
m.add_transition(GAME_OVER, st.FAILURE, FAILED)
m.add_transition(UNDEFINED, st.RESUMING, START)
m.add_transition(st.ANALYZING, GAME_OVER, FINISH)
m.add_transition(st.ANALYZING, st.SCHEDULING, SCHEDULE)
m.add_transition(st.ANALYZING, st.WAITING, WAIT)
m.add_transition(st.RESUMING, st.SCHEDULING, SCHEDULE)
m.add_transition(st.SCHEDULING, st.WAITING, WAIT)
m.add_transition(st.WAITING, st.ANALYZING, ANALYZE)
m.add_reaction(GAME_OVER, FINISH, game_over)
m.add_reaction(st.ANALYZING, ANALYZE, analyze)
m.add_reaction(st.RESUMING, START, resume)
m.add_reaction(st.SCHEDULING, SCHEDULE, schedule)
m.add_reaction(st.WAITING, WAIT, wait)
m.freeze()
return (m, memory)

View File

@ -1,399 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2014 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import threading
import fasteners
from oslo_utils import excutils
import six
from taskflow import flow
from taskflow import logging
from taskflow import task
from taskflow.types import graph as gr
from taskflow.types import tree as tr
from taskflow.utils import iter_utils
from taskflow.utils import misc
from taskflow.flow import (LINK_INVARIANT, LINK_RETRY) # noqa
LOG = logging.getLogger(__name__)
# Constants attached to node attributes in the execution graph (and tree
# node metadata), provided as constants here and constants in the compilation
# class (so that users will not have to import this file to access them); but
# provide them as module constants so that internal code can more
# easily access them...
TASK = 'task'
RETRY = 'retry'
FLOW = 'flow'
FLOW_END = 'flow_end'
# Quite often used together, so make a tuple everyone can share...
ATOMS = (TASK, RETRY)
FLOWS = (FLOW, FLOW_END)
class Terminator(object):
"""Flow terminator class."""
def __init__(self, flow):
self._flow = flow
self._name = "%s[$]" % (self._flow.name,)
@property
def flow(self):
"""The flow which this terminator signifies/marks the end of."""
return self._flow
@property
def name(self):
"""Useful name this end terminator has (derived from flow name)."""
return self._name
def __str__(self):
return "%s[$]" % (self._flow,)
class Compilation(object):
"""The result of a compilers ``compile()`` is this *immutable* object."""
#: Task nodes will have a ``kind`` metadata key with this value.
TASK = TASK
#: Retry nodes will have a ``kind`` metadata key with this value.
RETRY = RETRY
FLOW = FLOW
"""
Flow **entry** nodes will have a ``kind`` metadata key with
this value.
"""
FLOW_END = FLOW_END
"""
Flow **exit** nodes will have a ``kind`` metadata key with
this value (only applicable for compilation execution graph, not currently
used in tree hierarchy).
"""
def __init__(self, execution_graph, hierarchy):
self._execution_graph = execution_graph
self._hierarchy = hierarchy
@property
def execution_graph(self):
"""The execution ordering of atoms (as a graph structure)."""
return self._execution_graph
@property
def hierarchy(self):
"""The hierarchy of patterns (as a tree structure)."""
return self._hierarchy
def _overlap_occurrence_detector(to_graph, from_graph):
"""Returns how many nodes in 'from' graph are in 'to' graph (if any)."""
return iter_utils.count(node for node in from_graph.nodes_iter()
if node in to_graph)
def _add_update_edges(graph, nodes_from, nodes_to, attr_dict=None):
"""Adds/updates edges from nodes to other nodes in the specified graph.
It will connect the 'nodes_from' to the 'nodes_to' if an edge currently
does *not* exist (if it does already exist then the edges attributes
are just updated instead). When an edge is created the provided edge
attributes dictionary will be applied to the new edge between these two
nodes.
"""
# NOTE(harlowja): give each edge its own attr copy so that if it's
# later modified that the same copy isn't modified...
for u in nodes_from:
for v in nodes_to:
if not graph.has_edge(u, v):
if attr_dict:
graph.add_edge(u, v, attr_dict=attr_dict.copy())
else:
graph.add_edge(u, v)
else:
# Just update the attr_dict (if any).
if attr_dict:
graph.add_edge(u, v, attr_dict=attr_dict.copy())
class TaskCompiler(object):
"""Non-recursive compiler of tasks."""
def compile(self, task, parent=None):
graph = gr.DiGraph(name=task.name)
graph.add_node(task, kind=TASK)
node = tr.Node(task, kind=TASK)
if parent is not None:
parent.add(node)
return graph, node
class FlowCompiler(object):
"""Recursive compiler of flows."""
def __init__(self, deep_compiler_func):
self._deep_compiler_func = deep_compiler_func
def compile(self, flow, parent=None):
"""Decomposes a flow into a graph and scope tree hierarchy."""
graph = gr.DiGraph(name=flow.name)
graph.add_node(flow, kind=FLOW, noop=True)
tree_node = tr.Node(flow, kind=FLOW, noop=True)
if parent is not None:
parent.add(tree_node)
if flow.retry is not None:
tree_node.add(tr.Node(flow.retry, kind=RETRY))
decomposed = dict(
(child, self._deep_compiler_func(child, parent=tree_node)[0])
for child in flow)
decomposed_graphs = list(six.itervalues(decomposed))
graph = gr.merge_graphs(graph, *decomposed_graphs,
overlap_detector=_overlap_occurrence_detector)
for u, v, attr_dict in flow.iter_links():
u_graph = decomposed[u]
v_graph = decomposed[v]
_add_update_edges(graph, u_graph.no_successors_iter(),
list(v_graph.no_predecessors_iter()),
attr_dict=attr_dict)
# Insert the flow(s) retry if needed, and always make sure it
# is the **immediate** successor of the flow node itself.
if flow.retry is not None:
graph.add_node(flow.retry, kind=RETRY)
_add_update_edges(graph, [flow], [flow.retry],
attr_dict={LINK_INVARIANT: True})
for node in graph.nodes_iter():
if node is not flow.retry and node is not flow:
graph.node[node].setdefault(RETRY, flow.retry)
from_nodes = [flow.retry]
attr_dict = {LINK_INVARIANT: True, LINK_RETRY: True}
else:
from_nodes = [flow]
attr_dict = {LINK_INVARIANT: True}
# Ensure all nodes with no predecessors are connected to this flow
# or its retry node (so that the invariant that the flow node is
# traversed through before its contents is maintained); this allows
# us to easily know when we have entered a flow (when running) and
# do special and/or smart things such as only traverse up to the
# start of a flow when looking for node deciders.
_add_update_edges(graph, from_nodes, [
node for node in graph.no_predecessors_iter()
if node is not flow
], attr_dict=attr_dict)
# Connect all nodes with no successors into a special terminator
# that is used to identify the end of the flow and ensure that all
# execution traversals will traverse over this node before executing
# further work (this is especially useful for nesting and knowing
# when we have exited a nesting level); it allows us to do special
# and/or smart things such as applying deciders up to (but not
# beyond) a flow termination point.
#
# Do note that in a empty flow this will just connect itself to
# the flow node itself... and also note we can not use the flow
# object itself (primarily because the underlying graph library
# uses hashing to identify node uniqueness and we can easily create
# a loop if we don't do this correctly, so avoid that by just
# creating this special node and tagging it with a special kind); we
# may be able to make this better in the future with a multidigraph
# that networkx provides??
flow_term = Terminator(flow)
graph.add_node(flow_term, kind=FLOW_END, noop=True)
_add_update_edges(graph, [
node for node in graph.no_successors_iter()
if node is not flow_term
], [flow_term], attr_dict={LINK_INVARIANT: True})
return graph, tree_node
class PatternCompiler(object):
"""Compiles a flow pattern (or task) into a compilation unit.
Let's dive into the basic idea for how this works:
The compiler here is provided a 'root' object via its __init__ method,
this object could be a task, or a flow (one of the supported patterns),
the end-goal is to produce a :py:class:`.Compilation` object as the result
with the needed components. If this is not possible a
:py:class:`~.taskflow.exceptions.CompilationFailure` will be raised.
In the case where a **unknown** type is being requested to compile
a ``TypeError`` will be raised and when a duplicate object (one that
has **already** been compiled) is encountered a ``ValueError`` is raised.
The complexity of this comes into play when the 'root' is a flow that
contains itself other nested flows (and so-on); to compile this object and
its contained objects into a graph that *preserves* the constraints the
pattern mandates we have to go through a recursive algorithm that creates
subgraphs for each nesting level, and then on the way back up through
the recursion (now with a decomposed mapping from contained patterns or
atoms to there corresponding subgraph) we have to then connect the
subgraphs (and the atom(s) there-in) that were decomposed for a pattern
correctly into a new graph and then ensure the pattern mandated
constraints are retained. Finally we then return to the
caller (and they will do the same thing up until the root node, which by
that point one graph is created with all contained atoms in the
pattern/nested patterns mandated ordering).
Also maintained in the :py:class:`.Compilation` object is a hierarchy of
the nesting of items (which is also built up during the above mentioned
recusion, via a much simpler algorithm); this is typically used later to
determine the prior atoms of a given atom when looking up values that can
be provided to that atom for execution (see the scopes.py file for how this
works). Note that although you *could* think that the graph itself could be
used for this, which in some ways it can (for limited usage) the hierarchy
retains the nested structure (which is useful for scoping analysis/lookup)
to be able to provide back a iterator that gives back the scopes visible
at each level (the graph does not have this information once flattened).
Let's take an example:
Given the pattern ``f(a(b, c), d)`` where ``f`` is a
:py:class:`~taskflow.patterns.linear_flow.Flow` with items ``a(b, c)``
where ``a`` is a :py:class:`~taskflow.patterns.linear_flow.Flow` composed
of tasks ``(b, c)`` and task ``d``.
The algorithm that will be performed (mirroring the above described logic)
will go through the following steps (the tree hierarchy building is left
out as that is more obvious)::
Compiling f
- Decomposing flow f with no parent (must be the root)
- Compiling a
- Decomposing flow a with parent f
- Compiling b
- Decomposing task b with parent a
- Decomposed b into:
Name: b
Nodes: 1
- b
Edges: 0
- Compiling c
- Decomposing task c with parent a
- Decomposed c into:
Name: c
Nodes: 1
- c
Edges: 0
- Relinking decomposed b -> decomposed c
- Decomposed a into:
Name: a
Nodes: 2
- b
- c
Edges: 1
b -> c ({'invariant': True})
- Compiling d
- Decomposing task d with parent f
- Decomposed d into:
Name: d
Nodes: 1
- d
Edges: 0
- Relinking decomposed a -> decomposed d
- Decomposed f into:
Name: f
Nodes: 3
- c
- b
- d
Edges: 2
c -> d ({'invariant': True})
b -> c ({'invariant': True})
"""
def __init__(self, root, freeze=True):
self._root = root
self._history = set()
self._freeze = freeze
self._lock = threading.Lock()
self._compilation = None
self._matchers = [
(flow.Flow, FlowCompiler(self._compile)),
(task.Task, TaskCompiler()),
]
self._level = 0
def _compile(self, item, parent=None):
"""Compiles a item (pattern, task) into a graph + tree node."""
item_compiler = misc.match_type(item, self._matchers)
if item_compiler is not None:
self._pre_item_compile(item)
graph, node = item_compiler.compile(item, parent=parent)
self._post_item_compile(item, graph, node)
return graph, node
else:
raise TypeError("Unknown object '%s' (%s) requested to compile"
% (item, type(item)))
def _pre_item_compile(self, item):
"""Called before a item is compiled; any pre-compilation actions."""
if item in self._history:
raise ValueError("Already compiled item '%s' (%s), duplicate"
" and/or recursive compiling is not"
" supported" % (item, type(item)))
self._history.add(item)
if LOG.isEnabledFor(logging.TRACE):
LOG.trace("%sCompiling '%s'", " " * self._level, item)
self._level += 1
def _post_item_compile(self, item, graph, node):
"""Called after a item is compiled; doing post-compilation actions."""
self._level -= 1
if LOG.isEnabledFor(logging.TRACE):
prefix = ' ' * self._level
LOG.trace("%sDecomposed '%s' into:", prefix, item)
prefix = ' ' * (self._level + 1)
LOG.trace("%sGraph:", prefix)
for line in graph.pformat().splitlines():
LOG.trace("%s %s", prefix, line)
LOG.trace("%sHierarchy:", prefix)
for line in node.pformat().splitlines():
LOG.trace("%s %s", prefix, line)
def _pre_compile(self):
"""Called before the compilation of the root starts."""
self._history.clear()
self._level = 0
def _post_compile(self, graph, node):
"""Called after the compilation of the root finishes successfully."""
self._history.clear()
self._level = 0
@fasteners.locked
def compile(self):
"""Compiles the contained item into a compiled equivalent."""
if self._compilation is None:
self._pre_compile()
try:
graph, node = self._compile(self._root, parent=None)
except Exception:
with excutils.save_and_reraise_exception():
# Always clear the history, to avoid retaining junk
# in memory that isn't needed to be in memory if
# compilation fails...
self._history.clear()
else:
self._post_compile(graph, node)
if self._freeze:
graph.freeze()
node.freeze()
self._compilation = Compilation(graph, node)
return self._compilation

View File

@ -1,224 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2014 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import abc
import weakref
from oslo_utils import reflection
from oslo_utils import strutils
import six
from taskflow.engines.action_engine import compiler as co
from taskflow.engines.action_engine import executor as ex
from taskflow import logging
from taskflow import retry as retry_atom
from taskflow import states as st
LOG = logging.getLogger(__name__)
@six.add_metaclass(abc.ABCMeta)
class Strategy(object):
"""Failure resolution strategy base class."""
strategy = None
def __init__(self, runtime):
self._runtime = runtime
@abc.abstractmethod
def apply(self):
"""Applies some algorithm to resolve some detected failure."""
def __str__(self):
base = reflection.get_class_name(self, fully_qualified=False)
if self.strategy is not None:
strategy_name = self.strategy.name
else:
strategy_name = "???"
return base + "(strategy=%s)" % (strategy_name)
class RevertAndRetry(Strategy):
"""Sets the *associated* subflow for revert to be later retried."""
strategy = retry_atom.RETRY
def __init__(self, runtime, retry):
super(RevertAndRetry, self).__init__(runtime)
self._retry = retry
def apply(self):
tweaked = self._runtime.reset_atoms([self._retry], state=None,
intention=st.RETRY)
tweaked.extend(self._runtime.reset_subgraph(self._retry, state=None,
intention=st.REVERT))
return tweaked
class RevertAll(Strategy):
"""Sets *all* nodes/atoms to the ``REVERT`` intention."""
strategy = retry_atom.REVERT_ALL
def __init__(self, runtime):
super(RevertAll, self).__init__(runtime)
def apply(self):
return self._runtime.reset_atoms(
self._runtime.iterate_nodes(co.ATOMS),
state=None, intention=st.REVERT)
class Revert(Strategy):
"""Sets atom and *associated* nodes to the ``REVERT`` intention."""
strategy = retry_atom.REVERT
def __init__(self, runtime, atom):
super(Revert, self).__init__(runtime)
self._atom = atom
def apply(self):
tweaked = self._runtime.reset_atoms([self._atom], state=None,
intention=st.REVERT)
tweaked.extend(self._runtime.reset_subgraph(self._atom, state=None,
intention=st.REVERT))
return tweaked
class Completer(object):
"""Completes atoms using actions to complete them."""
def __init__(self, runtime):
self._runtime = weakref.proxy(runtime)
self._storage = runtime.storage
self._undefined_resolver = RevertAll(self._runtime)
self._defer_reverts = strutils.bool_from_string(
self._runtime.options.get('defer_reverts', False))
self._resolve = not strutils.bool_from_string(
self._runtime.options.get('never_resolve', False))
def resume(self):
"""Resumes atoms in the contained graph.
This is done to allow any previously completed or failed atoms to
be analyzed, there results processed and any potential atoms affected
to be adjusted as needed.
This should return a set of atoms which should be the initial set of
atoms that were previously not finished (due to a RUNNING or REVERTING
attempt not previously finishing).
"""
atoms = list(self._runtime.iterate_nodes(co.ATOMS))
atom_states = self._storage.get_atoms_states(atom.name
for atom in atoms)
if self._resolve:
for atom in atoms:
atom_state, _atom_intention = atom_states[atom.name]
if atom_state == st.FAILURE:
self._process_atom_failure(
atom, self._storage.get(atom.name))
for retry in self._runtime.iterate_retries(st.RETRYING):
retry_affected_atoms_it = self._runtime.retry_subflow(retry)
for atom, state, intention in retry_affected_atoms_it:
if state:
atom_states[atom.name] = (state, intention)
unfinished_atoms = set()
for atom in atoms:
atom_state, _atom_intention = atom_states[atom.name]
if atom_state in (st.RUNNING, st.REVERTING):
unfinished_atoms.add(atom)
LOG.trace("Resuming atom '%s' since it was left in"
" state %s", atom, atom_state)
return unfinished_atoms
def complete_failure(self, node, outcome, failure):
"""Performs post-execution completion of a nodes failure.
Returns whether the result should be saved into an accumulator of
failures or whether this should not be done.
"""
if outcome == ex.EXECUTED and self._resolve:
self._process_atom_failure(node, failure)
# We resolved something, carry on...
return False
else:
# Reverting failed (or resolving was turned off), always
# retain the failure...
return True
def complete(self, node, outcome, result):
"""Performs post-execution completion of a node result."""
handler = self._runtime.fetch_action(node)
if outcome == ex.EXECUTED:
handler.complete_execution(node, result)
else:
handler.complete_reversion(node, result)
def _determine_resolution(self, atom, failure):
"""Determines which resolution strategy to activate/apply."""
retry = self._runtime.find_retry(atom)
if retry is not None:
# Ask retry controller what to do in case of failure.
handler = self._runtime.fetch_action(retry)
strategy = handler.on_failure(retry, atom, failure)
if strategy == retry_atom.RETRY:
return RevertAndRetry(self._runtime, retry)
elif strategy == retry_atom.REVERT:
# Ask parent retry and figure out what to do...
parent_resolver = self._determine_resolution(retry, failure)
# In the future, this will be the only behavior. REVERT
# should defer to the parent retry if it exists, or use the
# default REVERT_ALL if it doesn't.
if self._defer_reverts:
return parent_resolver
# Ok if the parent resolver says something not REVERT, and
# it isn't just using the undefined resolver, assume the
# parent knows best.
if parent_resolver is not self._undefined_resolver:
if parent_resolver.strategy != retry_atom.REVERT:
return parent_resolver
return Revert(self._runtime, retry)
elif strategy == retry_atom.REVERT_ALL:
return RevertAll(self._runtime)
else:
raise ValueError("Unknown atom failure resolution"
" action/strategy '%s'" % strategy)
else:
return self._undefined_resolver
def _process_atom_failure(self, atom, failure):
"""Processes atom failure & applies resolution strategies.
On atom failure this will find the atoms associated retry controller
and ask that controller for the strategy to perform to resolve that
failure. After getting a resolution strategy decision this method will
then adjust the needed other atoms intentions, and states, ... so that
the failure can be worked around.
"""
resolver = self._determine_resolution(atom, failure)
LOG.debug("Applying resolver '%s' to resolve failure '%s'"
" of atom '%s'", resolver, failure, atom)
tweaked = resolver.apply()
# Only show the tweaked node list when trace is on, otherwise
# just show the amount/count of nodes tweaks...
if LOG.isEnabledFor(logging.TRACE):
LOG.trace("Modified/tweaked %s nodes while applying"
" resolver '%s'", tweaked, resolver)
else:
LOG.debug("Modified/tweaked %s nodes while applying"
" resolver '%s'", len(tweaked), resolver)

View File

@ -1,184 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2015 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import abc
import itertools
import six
from taskflow import deciders
from taskflow.engines.action_engine import compiler
from taskflow.engines.action_engine import traversal
from taskflow import logging
from taskflow import states
LOG = logging.getLogger(__name__)
@six.add_metaclass(abc.ABCMeta)
class Decider(object):
"""Base class for deciders.
Provides interface to be implemented by sub-classes.
Deciders check whether next atom in flow should be executed or not.
"""
@abc.abstractmethod
def tally(self, runtime):
"""Tally edge deciders on whether this decider should allow running.
The returned value is a list of edge deciders that voted
'nay' (do not allow running).
"""
@abc.abstractmethod
def affect(self, runtime, nay_voters):
"""Affects associated atoms due to at least one 'nay' edge decider.
This will alter the associated atom + some set of successor atoms by
setting there state and intention to ``IGNORE`` so that they are
ignored in future runtime activities.
"""
def check_and_affect(self, runtime):
"""Handles :py:func:`~.tally` + :py:func:`~.affect` in right order.
NOTE(harlowja): If there are zero 'nay' edge deciders then it is
assumed this decider should allow running.
Returns boolean of whether this decider allows for running (or not).
"""
nay_voters = self.tally(runtime)
if nay_voters:
self.affect(runtime, nay_voters)
return False
return True
def _affect_all_successors(atom, runtime):
execution_graph = runtime.compilation.execution_graph
successors_iter = traversal.depth_first_iterate(
execution_graph, atom, traversal.Direction.FORWARD)
runtime.reset_atoms(itertools.chain([atom], successors_iter),
state=states.IGNORE, intention=states.IGNORE)
def _affect_successor_tasks_in_same_flow(atom, runtime):
execution_graph = runtime.compilation.execution_graph
successors_iter = traversal.depth_first_iterate(
execution_graph, atom, traversal.Direction.FORWARD,
# Do not go through nested flows but do follow *all* tasks that
# are directly connected in this same flow (thus the reason this is
# called the same flow decider); retries are direct successors
# of flows, so they should also be not traversed through, but
# setting this explicitly ensures that.
through_flows=False, through_retries=False)
runtime.reset_atoms(itertools.chain([atom], successors_iter),
state=states.IGNORE, intention=states.IGNORE)
def _affect_atom(atom, runtime):
runtime.reset_atoms([atom], state=states.IGNORE, intention=states.IGNORE)
def _affect_direct_task_neighbors(atom, runtime):
def _walk_neighbors():
execution_graph = runtime.compilation.execution_graph
for node in execution_graph.successors_iter(atom):
node_data = execution_graph.node[node]
if node_data['kind'] == compiler.TASK:
yield node
successors_iter = _walk_neighbors()
runtime.reset_atoms(itertools.chain([atom], successors_iter),
state=states.IGNORE, intention=states.IGNORE)
class IgnoreDecider(Decider):
"""Checks any provided edge-deciders and determines if ok to run."""
_depth_strategies = {
deciders.Depth.ALL: _affect_all_successors,
deciders.Depth.ATOM: _affect_atom,
deciders.Depth.FLOW: _affect_successor_tasks_in_same_flow,
deciders.Depth.NEIGHBORS: _affect_direct_task_neighbors,
}
def __init__(self, atom, edge_deciders):
self._atom = atom
self._edge_deciders = edge_deciders
def tally(self, runtime):
voters = {
'run_it': [],
'do_not_run_it': [],
'ignored': [],
}
history = {}
if self._edge_deciders:
# Gather all atoms (the ones that were not ignored) results so
# that those results can be used by the decider(s) that are
# making a decision as to pass or not pass...
states_intentions = runtime.storage.get_atoms_states(
ed.from_node.name for ed in self._edge_deciders
if ed.kind in compiler.ATOMS)
for atom_name in six.iterkeys(states_intentions):
atom_state, _atom_intention = states_intentions[atom_name]
if atom_state != states.IGNORE:
history[atom_name] = runtime.storage.get(atom_name)
for ed in self._edge_deciders:
if (ed.kind in compiler.ATOMS and
# It was an ignored atom (not included in history and
# the only way that is possible is via above loop
# skipping it...)
ed.from_node.name not in history):
voters['ignored'].append(ed)
continue
if not ed.decider(history=history):
voters['do_not_run_it'].append(ed)
else:
voters['run_it'].append(ed)
if LOG.isEnabledFor(logging.TRACE):
LOG.trace("Out of %s deciders there were %s 'do no run it'"
" voters, %s 'do run it' voters and %s 'ignored'"
" voters for transition to atom '%s' given history %s",
sum(len(eds) for eds in six.itervalues(voters)),
list(ed.from_node.name
for ed in voters['do_not_run_it']),
list(ed.from_node.name for ed in voters['run_it']),
list(ed.from_node.name for ed in voters['ignored']),
self._atom.name, history)
return voters['do_not_run_it']
def affect(self, runtime, nay_voters):
# If there were many 'nay' edge deciders that were targeted
# at this atom, then we need to pick the one which has the widest
# impact and respect that one as the decider depth that will
# actually affect things.
widest_depth = deciders.pick_widest(ed.depth for ed in nay_voters)
affector = self._depth_strategies[widest_depth]
return affector(self._atom, runtime)
class NoOpDecider(Decider):
"""No-op decider that says it is always ok to run & has no effect(s)."""
def tally(self, runtime):
"""Always good to go."""
return []
def affect(self, runtime, nay_voters):
"""Does nothing."""

View File

@ -1,637 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2012 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import collections
import contextlib
import itertools
import threading
from automaton import runners
from concurrent import futures
import fasteners
import networkx as nx
from oslo_utils import excutils
from oslo_utils import strutils
from oslo_utils import timeutils
import six
from taskflow.engines.action_engine import builder
from taskflow.engines.action_engine import compiler
from taskflow.engines.action_engine import executor
from taskflow.engines.action_engine import process_executor
from taskflow.engines.action_engine import runtime
from taskflow.engines import base
from taskflow import exceptions as exc
from taskflow import logging
from taskflow import states
from taskflow import storage
from taskflow.types import failure
from taskflow.utils import misc
LOG = logging.getLogger(__name__)
@contextlib.contextmanager
def _start_stop(task_executor, retry_executor):
# A teenie helper context manager to safely start/stop engine executors...
task_executor.start()
try:
retry_executor.start()
try:
yield (task_executor, retry_executor)
finally:
retry_executor.stop()
finally:
task_executor.stop()
def _pre_check(check_compiled=True, check_storage_ensured=True,
check_validated=True):
"""Engine state precondition checking decorator."""
def decorator(meth):
do_what = meth.__name__
@six.wraps(meth)
def wrapper(self, *args, **kwargs):
if check_compiled and not self._compiled:
raise exc.InvalidState("Can not %s an engine which"
" has not been compiled" % do_what)
if check_storage_ensured and not self._storage_ensured:
raise exc.InvalidState("Can not %s an engine"
" which has not had its storage"
" populated" % do_what)
if check_validated and not self._validated:
raise exc.InvalidState("Can not %s an engine which"
" has not been validated" % do_what)
return meth(self, *args, **kwargs)
return wrapper
return decorator
class ActionEngine(base.Engine):
"""Generic action-based engine.
This engine compiles the flow (and any subflows) into a compilation unit
which contains the full runtime definition to be executed and then uses
this compilation unit in combination with the executor, runtime, machine
builder and storage classes to attempt to run your flow (and any
subflows & contained atoms) to completion.
NOTE(harlowja): during this process it is permissible and valid to have a
task or multiple tasks in the execution graph fail (at the same time even),
which will cause the process of reversion or retrying to commence. See the
valid states in the states module to learn more about what other states
the tasks and flow being ran can go through.
**Engine options:**
+----------------------+-----------------------+------+------------+
| Name/key | Description | Type | Default |
+======================+=======================+======+============+
| ``defer_reverts`` | This option lets you | bool | ``False`` |
| | safely nest flows | | |
| | with retries inside | | |
| | flows without retries | | |
| | and it still behaves | | |
| | as a user would | | |
| | expect (for example | | |
| | if the retry gets | | |
| | exhausted it reverts | | |
| | the outer flow unless | | |
| | the outer flow has a | | |
| | has a separate retry | | |
| | behavior). | | |
+----------------------+-----------------------+------+------------+
| ``never_resolve`` | When true, instead | bool | ``False`` |
| | of reverting | | |
| | and trying to resolve | | |
| | a atom failure the | | |
| | engine will skip | | |
| | reverting and abort | | |
| | instead of reverting | | |
| | and/or retrying. | | |
+----------------------+-----------------------+------+------------+
| ``inject_transient`` | When true, values | bool | ``True`` |
| | that are local to | | |
| | each atoms scope | | |
| | are injected into | | |
| | storage into a | | |
| | transient location | | |
| | (typically a local | | |
| | dictionary), when | | |
| | false those values | | |
| | are instead persisted | | |
| | into atom details | | |
| | (and saved in a non- | | |
| | transient manner). | | |
+----------------------+-----------------------+------+------------+
"""
NO_RERAISING_STATES = frozenset([states.SUSPENDED, states.SUCCESS])
"""
States that if the engine stops in will **not** cause any potential
failures to be reraised. States **not** in this list will cause any
failure/s that were captured (if any) to get reraised.
"""
IGNORABLE_STATES = frozenset(
itertools.chain([states.SCHEDULING, states.WAITING, states.RESUMING,
states.ANALYZING], builder.META_STATES))
"""
Informational states this engines internal machine yields back while
running, not useful to have the engine record but useful to provide to
end-users when doing execution iterations via :py:meth:`.run_iter`.
"""
MAX_MACHINE_STATES_RETAINED = 10
"""
During :py:meth:`~.run_iter` the last X state machine transitions will
be recorded (typically only useful on failure).
"""
def __init__(self, flow, flow_detail, backend, options):
super(ActionEngine, self).__init__(flow, flow_detail, backend, options)
self._runtime = None
self._compiled = False
self._compilation = None
self._compiler = compiler.PatternCompiler(flow)
self._lock = threading.RLock()
self._storage_ensured = False
self._validated = False
# Retries are not *currently* executed out of the engines process
# or thread (this could change in the future if we desire it to).
self._retry_executor = executor.SerialRetryExecutor()
self._inject_transient = strutils.bool_from_string(
self._options.get('inject_transient', True))
self._gather_statistics = strutils.bool_from_string(
self._options.get('gather_statistics', True))
self._statistics = {}
@_pre_check(check_compiled=True,
# NOTE(harlowja): We can alter the state of the
# flow without ensuring its storage is setup for
# its atoms (since this state change does not affect
# those units).
check_storage_ensured=False,
check_validated=False)
def suspend(self):
self._change_state(states.SUSPENDING)
@property
def statistics(self):
return self._statistics
@property
def compilation(self):
"""The compilation result.
NOTE(harlowja): Only accessible after compilation has completed (None
will be returned when this property is accessed before compilation has
completed successfully).
"""
if self._compiled:
return self._compilation
else:
return None
@misc.cachedproperty
def storage(self):
"""The storage unit for this engine.
NOTE(harlowja): the atom argument lookup strategy will change for
this storage unit after
:py:func:`~taskflow.engines.base.Engine.compile` has
completed (since **only** after compilation is the actual structure
known). Before :py:func:`~taskflow.engines.base.Engine.compile`
has completed the atom argument lookup strategy lookup will be
restricted to injected arguments **only** (this will **not** reflect
the actual runtime lookup strategy, which typically will be, but is
not always different).
"""
def _scope_fetcher(atom_name):
if self._compiled:
return self._runtime.fetch_scopes_for(atom_name)
else:
return None
return storage.Storage(self._flow_detail,
backend=self._backend,
scope_fetcher=_scope_fetcher)
def run(self, timeout=None):
"""Runs the engine (or die trying).
:param timeout: timeout to wait for any atoms to complete (this timeout
will be used during the waiting period that occurs when
unfinished atoms are being waited on).
"""
with fasteners.try_lock(self._lock) as was_locked:
if not was_locked:
raise exc.ExecutionFailure("Engine currently locked, please"
" try again later")
for _state in self.run_iter(timeout=timeout):
pass
def run_iter(self, timeout=None):
"""Runs the engine using iteration (or die trying).
:param timeout: timeout to wait for any atoms to complete (this timeout
will be used during the waiting period that occurs after the
waiting state is yielded when unfinished atoms are being waited
on).
Instead of running to completion in a blocking manner, this will
return a generator which will yield back the various states that the
engine is going through (and can be used to run multiple engines at
once using a generator per engine). The iterator returned also
responds to the ``send()`` method from :pep:`0342` and will attempt to
suspend itself if a truthy value is sent in (the suspend may be
delayed until all active atoms have finished).
NOTE(harlowja): using the ``run_iter`` method will **not** retain the
engine lock while executing so the user should ensure that there is
only one entity using a returned engine iterator (one per engine) at a
given time.
"""
self.compile()
self.prepare()
self.validate()
# Keep track of the last X state changes, which if a failure happens
# are quite useful to log (and the performance of tracking this
# should be negligible).
last_transitions = collections.deque(
maxlen=max(1, self.MAX_MACHINE_STATES_RETAINED))
with _start_stop(self._task_executor, self._retry_executor):
self._change_state(states.RUNNING)
if self._gather_statistics:
self._statistics.clear()
w = timeutils.StopWatch()
w.start()
else:
w = None
try:
closed = False
machine, memory = self._runtime.builder.build(
self._statistics, timeout=timeout,
gather_statistics=self._gather_statistics)
r = runners.FiniteRunner(machine)
for transition in r.run_iter(builder.START):
last_transitions.append(transition)
_prior_state, new_state = transition
# NOTE(harlowja): skip over meta-states
if new_state in builder.META_STATES:
continue
if new_state == states.FAILURE:
failure.Failure.reraise_if_any(memory.failures)
if closed:
continue
try:
try_suspend = yield new_state
except GeneratorExit:
# The generator was closed, attempt to suspend and
# continue looping until we have cleanly closed up
# shop...
closed = True
self.suspend()
except Exception:
# Capture the failure, and ensure that the
# machine will notice that something externally
# has sent an exception in and that it should
# finish up and reraise.
memory.failures.append(failure.Failure())
closed = True
else:
if try_suspend:
self.suspend()
except Exception:
with excutils.save_and_reraise_exception():
LOG.exception("Engine execution has failed, something"
" bad must of happened (last"
" %s machine transitions were %s)",
last_transitions.maxlen,
list(last_transitions))
self._change_state(states.FAILURE)
else:
if last_transitions:
_prior_state, new_state = last_transitions[-1]
if new_state not in self.IGNORABLE_STATES:
self._change_state(new_state)
if new_state not in self.NO_RERAISING_STATES:
e_failures = self.storage.get_execute_failures()
r_failures = self.storage.get_revert_failures()
er_failures = itertools.chain(
six.itervalues(e_failures),
six.itervalues(r_failures))
failure.Failure.reraise_if_any(er_failures)
finally:
if w is not None:
w.stop()
self._statistics['active_for'] = w.elapsed()
@staticmethod
def _check_compilation(compilation):
"""Performs post compilation validation/checks."""
seen = set()
dups = set()
execution_graph = compilation.execution_graph
for node, node_attrs in execution_graph.nodes_iter(data=True):
if node_attrs['kind'] in compiler.ATOMS:
atom_name = node.name
if atom_name in seen:
dups.add(atom_name)
else:
seen.add(atom_name)
if dups:
raise exc.Duplicate(
"Atoms with duplicate names found: %s" % (sorted(dups)))
return compilation
def _change_state(self, state):
moved, old_state = self.storage.change_flow_state(state)
if moved:
details = {
'engine': self,
'flow_name': self.storage.flow_name,
'flow_uuid': self.storage.flow_uuid,
'old_state': old_state,
}
self.notifier.notify(state, details)
def _ensure_storage(self):
"""Ensure all contained atoms exist in the storage unit."""
self.storage.ensure_atoms(
self._runtime.iterate_nodes(compiler.ATOMS))
for atom in self._runtime.iterate_nodes(compiler.ATOMS):
if atom.inject:
self.storage.inject_atom_args(atom.name, atom.inject,
transient=self._inject_transient)
@fasteners.locked
@_pre_check(check_validated=False)
def validate(self):
# At this point we can check to ensure all dependencies are either
# flow/task provided or storage provided, if there are still missing
# dependencies then this flow will fail at runtime (which we can avoid
# by failing at validation time).
if LOG.isEnabledFor(logging.TRACE):
execution_graph = self._compilation.execution_graph
LOG.trace("Validating scoping and argument visibility for"
" execution graph with %s nodes and %s edges with"
" density %0.3f", execution_graph.number_of_nodes(),
execution_graph.number_of_edges(),
nx.density(execution_graph))
missing = set()
# Attempt to retain a chain of what was missing (so that the final
# raised exception for the flow has the nodes that had missing
# dependencies).
last_cause = None
last_node = None
missing_nodes = 0
for atom in self._runtime.iterate_nodes(compiler.ATOMS):
exec_missing = self.storage.fetch_unsatisfied_args(
atom.name, atom.rebind, optional_args=atom.optional)
revert_missing = self.storage.fetch_unsatisfied_args(
atom.name, atom.revert_rebind,
optional_args=atom.revert_optional)
atom_missing = (('execute', exec_missing),
('revert', revert_missing))
for method, method_missing in atom_missing:
if method_missing:
cause = exc.MissingDependencies(atom,
sorted(method_missing),
cause=last_cause,
method=method)
last_cause = cause
last_node = atom
missing_nodes += 1
missing.update(method_missing)
if missing:
# For when a task is provided (instead of a flow) and that
# task is the only item in the graph and its missing deps, avoid
# re-wrapping it in yet another exception...
if missing_nodes == 1 and last_node is self._flow:
raise last_cause
else:
raise exc.MissingDependencies(self._flow,
sorted(missing),
cause=last_cause)
self._validated = True
@fasteners.locked
@_pre_check(check_storage_ensured=False, check_validated=False)
def prepare(self):
if not self._storage_ensured:
# Set our own state to resuming -> (ensure atoms exist
# in storage) -> suspended in the storage unit and notify any
# attached listeners of these changes.
self._change_state(states.RESUMING)
self._ensure_storage()
self._change_state(states.SUSPENDED)
self._storage_ensured = True
# Reset everything back to pending (if we were previously reverted).
if self.storage.get_flow_state() == states.REVERTED:
self.reset()
@fasteners.locked
@_pre_check(check_validated=False)
def reset(self):
# This transitions *all* contained atoms back into the PENDING state
# with an intention to EXECUTE (or dies trying to do that) and then
# changes the state of the flow to PENDING so that it can then run...
self._runtime.reset_all()
self._change_state(states.PENDING)
@fasteners.locked
def compile(self):
if self._compiled:
return
self._compilation = self._check_compilation(self._compiler.compile())
self._runtime = runtime.Runtime(self._compilation,
self.storage,
self.atom_notifier,
self._task_executor,
self._retry_executor,
options=self._options)
self._runtime.compile()
self._compiled = True
class SerialActionEngine(ActionEngine):
"""Engine that runs tasks in serial manner."""
def __init__(self, flow, flow_detail, backend, options):
super(SerialActionEngine, self).__init__(flow, flow_detail,
backend, options)
self._task_executor = executor.SerialTaskExecutor()
class _ExecutorTypeMatch(collections.namedtuple('_ExecutorTypeMatch',
['types', 'executor_cls'])):
def matches(self, executor):
return isinstance(executor, self.types)
class _ExecutorTextMatch(collections.namedtuple('_ExecutorTextMatch',
['strings', 'executor_cls'])):
def matches(self, text):
return text.lower() in self.strings
class ParallelActionEngine(ActionEngine):
"""Engine that runs tasks in parallel manner.
**Additional engine options:**
* ``executor``: a object that implements a :pep:`3148` compatible executor
interface; it will be used for scheduling tasks. The following
type are applicable (other unknown types passed will cause a type
error to be raised).
========================= ===============================================
Type provided Executor used
========================= ===============================================
|cft|.ThreadPoolExecutor :class:`~.executor.ParallelThreadTaskExecutor`
|cfp|.ProcessPoolExecutor :class:`~.|pe|.ParallelProcessTaskExecutor`
|cf|._base.Executor :class:`~.executor.ParallelThreadTaskExecutor`
========================= ===============================================
* ``executor``: a string that will be used to select a :pep:`3148`
compatible executor; it will be used for scheduling tasks. The following
string are applicable (other unknown strings passed will cause a value
error to be raised).
=========================== ===============================================
String (case insensitive) Executor used
=========================== ===============================================
``process`` :class:`~.|pe|.ParallelProcessTaskExecutor`
``processes`` :class:`~.|pe|.ParallelProcessTaskExecutor`
``thread`` :class:`~.executor.ParallelThreadTaskExecutor`
``threaded`` :class:`~.executor.ParallelThreadTaskExecutor`
``threads`` :class:`~.executor.ParallelThreadTaskExecutor`
``greenthread`` :class:`~.executor.ParallelThreadTaskExecutor`
(greened version)
``greedthreaded`` :class:`~.executor.ParallelThreadTaskExecutor`
(greened version)
``greenthreads`` :class:`~.executor.ParallelThreadTaskExecutor`
(greened version)
=========================== ===============================================
* ``max_workers``: a integer that will affect the number of parallel
workers that are used to dispatch tasks into (this number is bounded
by the maximum parallelization your workflow can support).
* ``wait_timeout``: a float (in seconds) that will affect the
parallel process task executor (and therefore is **only** applicable when
the executor provided above is of the process variant). This number
affects how much time the process task executor waits for messages from
child processes (typically indicating they have finished or failed). A
lower number will have high granularity but *currently* involves more
polling while a higher number will involve less polling but a slower time
for an engine to notice a task has completed.
.. |pe| replace:: process_executor
.. |cfp| replace:: concurrent.futures.process
.. |cft| replace:: concurrent.futures.thread
.. |cf| replace:: concurrent.futures
"""
# One of these types should match when a object (non-string) is provided
# for the 'executor' option.
#
# NOTE(harlowja): the reason we use the library/built-in futures is to
# allow for instances of that to be detected and handled correctly, instead
# of forcing everyone to use our derivatives (futurist or other)...
_executor_cls_matchers = [
_ExecutorTypeMatch((futures.ThreadPoolExecutor,),
executor.ParallelThreadTaskExecutor),
_ExecutorTypeMatch((futures.ProcessPoolExecutor,),
process_executor.ParallelProcessTaskExecutor),
_ExecutorTypeMatch((futures.Executor,),
executor.ParallelThreadTaskExecutor),
]
# One of these should match when a string/text is provided for the
# 'executor' option (a mixed case equivalent is allowed since the match
# will be lower-cased before checking).
_executor_str_matchers = [
_ExecutorTextMatch(frozenset(['processes', 'process']),
process_executor.ParallelProcessTaskExecutor),
_ExecutorTextMatch(frozenset(['thread', 'threads', 'threaded']),
executor.ParallelThreadTaskExecutor),
_ExecutorTextMatch(frozenset(['greenthread', 'greenthreads',
'greenthreaded']),
executor.ParallelGreenThreadTaskExecutor),
]
# Used when no executor is provided (either a string or object)...
_default_executor_cls = executor.ParallelThreadTaskExecutor
def __init__(self, flow, flow_detail, backend, options):
super(ParallelActionEngine, self).__init__(flow, flow_detail,
backend, options)
# This ensures that any provided executor will be validated before
# we get to far in the compilation/execution pipeline...
self._task_executor = self._fetch_task_executor(self._options)
@classmethod
def _fetch_task_executor(cls, options):
kwargs = {}
executor_cls = cls._default_executor_cls
# Match the desired executor to a class that will work with it...
desired_executor = options.get('executor')
if isinstance(desired_executor, six.string_types):
matched_executor_cls = None
for m in cls._executor_str_matchers:
if m.matches(desired_executor):
matched_executor_cls = m.executor_cls
break
if matched_executor_cls is None:
expected = set()
for m in cls._executor_str_matchers:
expected.update(m.strings)
raise ValueError("Unknown executor string '%s' expected"
" one of %s (or mixed case equivalent)"
% (desired_executor, list(expected)))
else:
executor_cls = matched_executor_cls
elif desired_executor is not None:
matched_executor_cls = None
for m in cls._executor_cls_matchers:
if m.matches(desired_executor):
matched_executor_cls = m.executor_cls
break
if matched_executor_cls is None:
expected = set()
for m in cls._executor_cls_matchers:
expected.update(m.types)
raise TypeError("Unknown executor '%s' (%s) expected an"
" instance of %s" % (desired_executor,
type(desired_executor),
list(expected)))
else:
executor_cls = matched_executor_cls
kwargs['executor'] = desired_executor
try:
for (k, value_converter) in executor_cls.constructor_options:
try:
kwargs[k] = value_converter(options[k])
except KeyError:
pass
except AttributeError:
pass
return executor_cls(**kwargs)

View File

@ -1,236 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2013 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import abc
import futurist
import six
from taskflow import task as ta
from taskflow.types import failure
from taskflow.types import notifier
# Execution and reversion outcomes.
EXECUTED = 'executed'
REVERTED = 'reverted'
def _execute_retry(retry, arguments):
try:
result = retry.execute(**arguments)
except Exception:
result = failure.Failure()
return (EXECUTED, result)
def _revert_retry(retry, arguments):
try:
result = retry.revert(**arguments)
except Exception:
result = failure.Failure()
return (REVERTED, result)
def _execute_task(task, arguments, progress_callback=None):
with notifier.register_deregister(task.notifier,
ta.EVENT_UPDATE_PROGRESS,
callback=progress_callback):
try:
task.pre_execute()
result = task.execute(**arguments)
except Exception:
# NOTE(imelnikov): wrap current exception with Failure
# object and return it.
result = failure.Failure()
finally:
task.post_execute()
return (EXECUTED, result)
def _revert_task(task, arguments, result, failures, progress_callback=None):
arguments = arguments.copy()
arguments[ta.REVERT_RESULT] = result
arguments[ta.REVERT_FLOW_FAILURES] = failures
with notifier.register_deregister(task.notifier,
ta.EVENT_UPDATE_PROGRESS,
callback=progress_callback):
try:
task.pre_revert()
result = task.revert(**arguments)
except Exception:
# NOTE(imelnikov): wrap current exception with Failure
# object and return it.
result = failure.Failure()
finally:
task.post_revert()
return (REVERTED, result)
class SerialRetryExecutor(object):
"""Executes and reverts retries."""
def __init__(self):
self._executor = futurist.SynchronousExecutor()
def start(self):
"""Prepare to execute retries."""
self._executor.restart()
def stop(self):
"""Finalize retry executor."""
self._executor.shutdown()
def execute_retry(self, retry, arguments):
"""Schedules retry execution."""
fut = self._executor.submit(_execute_retry, retry, arguments)
fut.atom = retry
return fut
def revert_retry(self, retry, arguments):
"""Schedules retry reversion."""
fut = self._executor.submit(_revert_retry, retry, arguments)
fut.atom = retry
return fut
@six.add_metaclass(abc.ABCMeta)
class TaskExecutor(object):
"""Executes and reverts tasks.
This class takes task and its arguments and executes or reverts it.
It encapsulates knowledge on how task should be executed or reverted:
right now, on separate thread, on another machine, etc.
"""
@abc.abstractmethod
def execute_task(self, task, task_uuid, arguments,
progress_callback=None):
"""Schedules task execution."""
@abc.abstractmethod
def revert_task(self, task, task_uuid, arguments, result, failures,
progress_callback=None):
"""Schedules task reversion."""
def start(self):
"""Prepare to execute tasks."""
def stop(self):
"""Finalize task executor."""
class SerialTaskExecutor(TaskExecutor):
"""Executes tasks one after another."""
def __init__(self):
self._executor = futurist.SynchronousExecutor()
def start(self):
self._executor.restart()
def stop(self):
self._executor.shutdown()
def execute_task(self, task, task_uuid, arguments, progress_callback=None):
fut = self._executor.submit(_execute_task,
task, arguments,
progress_callback=progress_callback)
fut.atom = task
return fut
def revert_task(self, task, task_uuid, arguments, result, failures,
progress_callback=None):
fut = self._executor.submit(_revert_task,
task, arguments, result, failures,
progress_callback=progress_callback)
fut.atom = task
return fut
class ParallelTaskExecutor(TaskExecutor):
"""Executes tasks in parallel.
Submits tasks to an executor which should provide an interface similar
to concurrent.Futures.Executor.
"""
constructor_options = [
('max_workers', lambda v: v if v is None else int(v)),
]
"""
Optional constructor keyword arguments this executor supports. These will
typically be passed via engine options (by a engine user) and converted
into the correct type before being sent into this
classes ``__init__`` method.
"""
def __init__(self, executor=None, max_workers=None):
self._executor = executor
self._max_workers = max_workers
self._own_executor = executor is None
@abc.abstractmethod
def _create_executor(self, max_workers=None):
"""Called when an executor has not been provided to make one."""
def _submit_task(self, func, task, *args, **kwargs):
fut = self._executor.submit(func, task, *args, **kwargs)
fut.atom = task
return fut
def execute_task(self, task, task_uuid, arguments, progress_callback=None):
return self._submit_task(_execute_task, task, arguments,
progress_callback=progress_callback)
def revert_task(self, task, task_uuid, arguments, result, failures,
progress_callback=None):
return self._submit_task(_revert_task, task, arguments, result,
failures, progress_callback=progress_callback)
def start(self):
if self._own_executor:
self._executor = self._create_executor(
max_workers=self._max_workers)
def stop(self):
if self._own_executor:
self._executor.shutdown(wait=True)
self._executor = None
class ParallelThreadTaskExecutor(ParallelTaskExecutor):
"""Executes tasks in parallel using a thread pool executor."""
def _create_executor(self, max_workers=None):
return futurist.ThreadPoolExecutor(max_workers=max_workers)
class ParallelGreenThreadTaskExecutor(ParallelThreadTaskExecutor):
"""Executes tasks in parallel using a greenthread pool executor."""
DEFAULT_WORKERS = 1000
"""
Default number of workers when ``None`` is passed; being that
greenthreads don't map to native threads or processors very well this
is more of a guess/somewhat arbitrary, but it does match what the eventlet
greenpool default size is (so at least it's consistent with what eventlet
does).
"""
def _create_executor(self, max_workers=None):
if max_workers is None:
max_workers = self.DEFAULT_WORKERS
return futurist.GreenThreadPoolExecutor(max_workers=max_workers)

View File

@ -1,737 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2015 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import asyncore
import binascii
import collections
import errno
import functools
import hmac
import math
import os
import pickle
import socket
import struct
import time
import futurist
from oslo_utils import excutils
import six
from taskflow.engines.action_engine import executor as base
from taskflow import logging
from taskflow import task as ta
from taskflow.types import notifier as nt
from taskflow.utils import iter_utils
from taskflow.utils import misc
from taskflow.utils import schema_utils as su
from taskflow.utils import threading_utils
LOG = logging.getLogger(__name__)
# Internal parent <-> child process protocol schema, message constants...
MAGIC_HEADER = 0xDECAF
CHALLENGE = 'identify_yourself'
CHALLENGE_RESPONSE = 'worker_reporting_in'
ACK = 'ack'
EVENT = 'event'
SCHEMAS = {
# Basic jsonschemas for verifying that the data we get back and
# forth from parent <-> child observes at least a basic expected
# format.
CHALLENGE: {
"type": "string",
"minLength": 1,
},
ACK: {
"type": "string",
"minLength": 1,
},
CHALLENGE_RESPONSE: {
"type": "string",
"minLength": 1,
},
EVENT: {
"type": "object",
"properties": {
'event_type': {
"type": "string",
},
'sent_on': {
"type": "number",
},
},
"required": ['event_type', 'sent_on'],
"additionalProperties": True,
},
}
# See http://bugs.python.org/issue1457119 for why this is so complex...
_DECODE_ENCODE_ERRORS = [pickle.PickleError, TypeError]
try:
import cPickle
_DECODE_ENCODE_ERRORS.append(cPickle.PickleError)
del cPickle
except (ImportError, AttributeError):
pass
_DECODE_ENCODE_ERRORS = tuple(_DECODE_ENCODE_ERRORS)
# Use the best pickle from here on out...
from six.moves import cPickle as pickle
class UnknownSender(Exception):
"""Exception raised when message from unknown sender is recvd."""
class ChallengeIgnored(Exception):
"""Exception raised when challenge has not been responded to."""
class Reader(object):
"""Reader machine that streams & parses messages that it then dispatches.
TODO(harlowja): Use python-suitcase in the future when the following
are addressed/resolved and released:
- https://github.com/digidotcom/python-suitcase/issues/28
- https://github.com/digidotcom/python-suitcase/issues/29
Binary format format is the following (no newlines in actual format)::
<magic-header> (4 bytes)
<mac-header-length> (4 bytes)
<mac> (1 or more variable bytes)
<identity-header-length> (4 bytes)
<identity> (1 or more variable bytes)
<msg-header-length> (4 bytes)
<msg> (1 or more variable bytes)
"""
#: Per state memory initializers.
_INITIALIZERS = {
'magic_header_left': 4,
'mac_header_left': 4,
'identity_header_left': 4,
'msg_header_left': 4,
}
#: Linear steps/transitions (order matters here).
_TRANSITIONS = tuple([
'magic_header_left',
'mac_header_left',
'mac_left',
'identity_header_left',
'identity_left',
'msg_header_left',
'msg_left',
])
def __init__(self, auth_key, dispatch_func, msg_limit=-1):
if not six.callable(dispatch_func):
raise ValueError("Expected provided dispatch function"
" to be callable")
self.auth_key = auth_key
self.dispatch_func = dispatch_func
msg_limiter = iter_utils.iter_forever(msg_limit)
self.msg_count = six.next(msg_limiter)
self._msg_limiter = msg_limiter
self._buffer = misc.BytesIO()
self._state = None
# Local machine variables and such are stored in here.
self._memory = {}
self._transitions = collections.deque(self._TRANSITIONS)
# This is the per state callback handler set. The first entry reads
# the data and the second entry is called after reading is completed,
# typically to save that data into object memory, or to validate
# it.
self._handlers = {
'magic_header_left': (self._read_field_data,
self._save_and_validate_magic),
'mac_header_left': (self._read_field_data,
functools.partial(self._save_pos_integer,
'mac_left')),
'mac_left': (functools.partial(self._read_data, 'mac'),
functools.partial(self._save_data, 'mac')),
'identity_header_left': (self._read_field_data,
functools.partial(self._save_pos_integer,
'identity_left')),
'identity_left': (functools.partial(self._read_data, 'identity'),
functools.partial(self._save_data, 'identity')),
'msg_header_left': (self._read_field_data,
functools.partial(self._save_pos_integer,
'msg_left')),
'msg_left': (functools.partial(self._read_data, 'msg'),
self._dispatch_and_reset),
}
# Force transition into first state...
self._transition()
def _save_pos_integer(self, key_name, data):
key_val = struct.unpack("!i", data)[0]
if key_val <= 0:
raise IOError("Invalid %s length received for key '%s', expected"
" greater than zero length" % (key_val, key_name))
self._memory[key_name] = key_val
return True
def _save_data(self, key_name, data):
self._memory[key_name] = data
return True
def _dispatch_and_reset(self, data):
self.dispatch_func(
self._memory['identity'],
# Lazy evaluate so the message can be thrown out as needed
# (instead of the receiver discarding it after the fact)...
functools.partial(_decode_message, self.auth_key, data,
self._memory['mac']))
self.msg_count = six.next(self._msg_limiter)
self._memory.clear()
def _transition(self):
try:
self._state = self._transitions.popleft()
except IndexError:
self._transitions.extend(self._TRANSITIONS)
self._state = self._transitions.popleft()
try:
self._memory[self._state] = self._INITIALIZERS[self._state]
except KeyError:
pass
self._handle_func, self._post_handle_func = self._handlers[self._state]
def _save_and_validate_magic(self, data):
magic_header = struct.unpack("!i", data)[0]
if magic_header != MAGIC_HEADER:
raise IOError("Invalid magic header received, expected 0x%x but"
" got 0x%x for message %s" % (MAGIC_HEADER,
magic_header,
self.msg_count + 1))
self._memory['magic'] = magic_header
return True
def _read_data(self, save_key_name, data):
data_len_left = self._memory[self._state]
self._buffer.write(data[0:data_len_left])
if len(data) < data_len_left:
data_len_left -= len(data)
self._memory[self._state] = data_len_left
return ''
else:
self._memory[self._state] = 0
buf_data = self._buffer.getvalue()
self._buffer.reset()
self._post_handle_func(buf_data)
self._transition()
return data[data_len_left:]
def _read_field_data(self, data):
return self._read_data(self._state, data)
@property
def bytes_needed(self):
return self._memory.get(self._state, 0)
def feed(self, data):
while len(data):
data = self._handle_func(data)
class BadHmacValueError(ValueError):
"""Value error raised when an invalid hmac is discovered."""
def _create_random_string(desired_length):
if desired_length <= 0:
return b''
data_length = int(math.ceil(desired_length / 2.0))
data = os.urandom(data_length)
hex_data = binascii.hexlify(data)
return hex_data[0:desired_length]
def _calculate_hmac(auth_key, body):
mac = hmac.new(auth_key, body).hexdigest()
if isinstance(mac, six.text_type):
mac = mac.encode("ascii")
return mac
def _encode_message(auth_key, message, identity, reverse=False):
message = pickle.dumps(message, 2)
message_mac = _calculate_hmac(auth_key, message)
pieces = [
struct.pack("!i", MAGIC_HEADER),
struct.pack("!i", len(message_mac)),
message_mac,
struct.pack("!i", len(identity)),
identity,
struct.pack("!i", len(message)),
message,
]
if reverse:
pieces.reverse()
return tuple(pieces)
def _decode_message(auth_key, message, message_mac):
tmp_message_mac = _calculate_hmac(auth_key, message)
if tmp_message_mac != message_mac:
raise BadHmacValueError('Invalid message hmac')
return pickle.loads(message)
class Channel(object):
"""Object that workers use to communicate back to their creator."""
def __init__(self, port, identity, auth_key):
self.identity = identity
self.port = port
self.auth_key = auth_key
self.dead = False
self._sent = self._received = 0
self._socket = None
self._read_pipe = None
self._write_pipe = None
def close(self):
if self._socket is not None:
self._socket.close()
self._socket = None
self._read_pipe = None
self._write_pipe = None
def _ensure_connected(self):
if self._socket is None:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setblocking(1)
try:
s.connect(("", self.port))
except socket.error as e:
with excutils.save_and_reraise_exception():
s.close()
if e.errno in (errno.ECONNREFUSED, errno.ENOTCONN,
errno.ECONNRESET):
# Don't bother with further connections...
self.dead = True
read_pipe = s.makefile("rb", 0)
write_pipe = s.makefile("wb", 0)
try:
msg = self._do_recv(read_pipe=read_pipe)
su.schema_validate(msg, SCHEMAS[CHALLENGE])
if msg != CHALLENGE:
raise IOError("Challenge expected not received")
else:
pieces = _encode_message(self.auth_key,
CHALLENGE_RESPONSE,
self.identity)
self._do_send_and_ack(pieces, write_pipe=write_pipe,
read_pipe=read_pipe)
except Exception:
with excutils.save_and_reraise_exception():
s.close()
else:
self._socket = s
self._read_pipe = read_pipe
self._write_pipe = write_pipe
def recv(self):
self._ensure_connected()
return self._do_recv()
def _do_recv(self, read_pipe=None):
if read_pipe is None:
read_pipe = self._read_pipe
msg_capture = collections.deque(maxlen=1)
msg_capture_func = (lambda _from_who, msg_decoder_func:
msg_capture.append(msg_decoder_func()))
reader = Reader(self.auth_key, msg_capture_func, msg_limit=1)
try:
maybe_msg_num = self._received + 1
bytes_needed = reader.bytes_needed
while True:
blob = read_pipe.read(bytes_needed)
if len(blob) != bytes_needed:
raise EOFError("Read pipe closed while reading %s"
" bytes for potential message %s"
% (bytes_needed, maybe_msg_num))
reader.feed(blob)
bytes_needed = reader.bytes_needed
except StopIteration:
pass
msg = msg_capture[0]
self._received += 1
return msg
def _do_send(self, pieces, write_pipe=None):
if write_pipe is None:
write_pipe = self._write_pipe
for piece in pieces:
write_pipe.write(piece)
write_pipe.flush()
def _do_send_and_ack(self, pieces, write_pipe=None, read_pipe=None):
self._do_send(pieces, write_pipe=write_pipe)
self._sent += 1
msg = self._do_recv(read_pipe=read_pipe)
su.schema_validate(msg, SCHEMAS[ACK])
if msg != ACK:
raise IOError("Failed receiving ack for sent"
" message %s" % self._metrics['sent'])
def send(self, message):
self._ensure_connected()
self._do_send_and_ack(_encode_message(self.auth_key, message,
self.identity))
class EventSender(object):
"""Sends event information from a child worker process to its creator."""
def __init__(self, channel):
self._channel = channel
self._pid = None
def __call__(self, event_type, details):
if not self._channel.dead:
if self._pid is None:
self._pid = os.getpid()
message = {
'event_type': event_type,
'details': details,
'sent_on': time.time(),
}
LOG.trace("Sending %s (from child %s)", message, self._pid)
self._channel.send(message)
class DispatcherHandler(asyncore.dispatcher):
"""Dispatches from a single connection into a target."""
#: Read/write chunk size.
CHUNK_SIZE = 8192
def __init__(self, sock, addr, dispatcher):
if six.PY2:
asyncore.dispatcher.__init__(self, map=dispatcher.map, sock=sock)
else:
super(DispatcherHandler, self).__init__(map=dispatcher.map,
sock=sock)
self.blobs_to_write = list(dispatcher.challenge_pieces)
self.reader = Reader(dispatcher.auth_key, self._dispatch)
self.targets = dispatcher.targets
self.tied_to = None
self.challenge_responded = False
self.ack_pieces = _encode_message(dispatcher.auth_key, ACK,
dispatcher.identity,
reverse=True)
self.addr = addr
def handle_close(self):
self.close()
def writable(self):
return bool(self.blobs_to_write)
def handle_write(self):
try:
blob = self.blobs_to_write.pop()
except IndexError:
pass
else:
sent = self.send(blob[0:self.CHUNK_SIZE])
if sent < len(blob):
self.blobs_to_write.append(blob[sent:])
def _send_ack(self):
self.blobs_to_write.extend(self.ack_pieces)
def _dispatch(self, from_who, msg_decoder_func):
if not self.challenge_responded:
msg = msg_decoder_func()
su.schema_validate(msg, SCHEMAS[CHALLENGE_RESPONSE])
if msg != CHALLENGE_RESPONSE:
raise ChallengeIgnored("Discarding connection from %s"
" challenge was not responded to"
% self.addr)
else:
LOG.trace("Peer %s (%s) has passed challenge sequence",
self.addr, from_who)
self.challenge_responded = True
self.tied_to = from_who
self._send_ack()
else:
if self.tied_to != from_who:
raise UnknownSender("Sender %s previously identified as %s"
" changed there identity to %s after"
" challenge sequence" % (self.addr,
self.tied_to,
from_who))
try:
task = self.targets[from_who]
except KeyError:
raise UnknownSender("Unknown message from %s (%s) not matched"
" to any known target" % (self.addr,
from_who))
msg = msg_decoder_func()
su.schema_validate(msg, SCHEMAS[EVENT])
if LOG.isEnabledFor(logging.TRACE):
msg_delay = max(0, time.time() - msg['sent_on'])
LOG.trace("Dispatching message from %s (%s) (it took %0.3f"
" seconds for it to arrive for processing after"
" being sent)", self.addr, from_who, msg_delay)
task.notifier.notify(msg['event_type'], msg.get('details'))
self._send_ack()
def handle_read(self):
data = self.recv(self.CHUNK_SIZE)
if len(data) == 0:
self.handle_close()
else:
try:
self.reader.feed(data)
except (IOError, UnknownSender):
LOG.warning("Invalid received message", exc_info=True)
self.handle_close()
except _DECODE_ENCODE_ERRORS:
LOG.warning("Badly formatted message", exc_info=True)
self.handle_close()
except (ValueError, su.ValidationError):
LOG.warning("Failed validating message", exc_info=True)
self.handle_close()
except ChallengeIgnored:
LOG.warning("Failed challenge sequence", exc_info=True)
self.handle_close()
class Dispatcher(asyncore.dispatcher):
"""Accepts messages received from child worker processes."""
#: See https://docs.python.org/2/library/socket.html#socket.socket.listen
MAX_BACKLOG = 5
def __init__(self, map, auth_key, identity):
if six.PY2:
asyncore.dispatcher.__init__(self, map=map)
else:
super(Dispatcher, self).__init__(map=map)
self.identity = identity
self.challenge_pieces = _encode_message(auth_key, CHALLENGE,
identity, reverse=True)
self.auth_key = auth_key
self.targets = {}
@property
def port(self):
if self.socket is not None:
return self.socket.getsockname()[1]
else:
return None
def setup(self):
self.targets.clear()
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.bind(("", 0))
LOG.trace("Accepting dispatch requests on port %s", self.port)
self.listen(self.MAX_BACKLOG)
def writable(self):
return False
@property
def map(self):
return self._map
def handle_close(self):
if self.socket is not None:
self.close()
def handle_accept(self):
pair = self.accept()
if pair is not None:
sock, addr = pair
addr = "%s:%s" % (addr[0], addr[1])
LOG.trace("Potentially accepted new connection from %s", addr)
DispatcherHandler(sock, addr, self)
class ParallelProcessTaskExecutor(base.ParallelTaskExecutor):
"""Executes tasks in parallel using a process pool executor.
NOTE(harlowja): this executor executes tasks in external processes, so that
implies that tasks that are sent to that external process are pickleable
since this is how the multiprocessing works (sending pickled objects back
and forth) and that the bound handlers (for progress updating in
particular) are proxied correctly from that external process to the one
that is alive in the parent process to ensure that callbacks registered in
the parent are executed on events in the child.
"""
#: Default timeout used by asyncore io loop (and eventually select/poll).
WAIT_TIMEOUT = 0.01
constructor_options = [
('max_workers', lambda v: v if v is None else int(v)),
('wait_timeout', lambda v: v if v is None else float(v)),
]
"""
Optional constructor keyword arguments this executor supports. These will
typically be passed via engine options (by a engine user) and converted
into the correct type before being sent into this
classes ``__init__`` method.
"""
def __init__(self, executor=None,
max_workers=None, wait_timeout=None):
super(ParallelProcessTaskExecutor, self).__init__(
executor=executor, max_workers=max_workers)
self._auth_key = _create_random_string(32)
self._dispatcher = Dispatcher({}, self._auth_key,
_create_random_string(32))
if wait_timeout is None:
self._wait_timeout = self.WAIT_TIMEOUT
else:
if wait_timeout <= 0:
raise ValueError("Provided wait timeout must be greater"
" than zero and not '%s'" % wait_timeout)
self._wait_timeout = wait_timeout
# Only created after starting...
self._worker = None
def _create_executor(self, max_workers=None):
return futurist.ProcessPoolExecutor(max_workers=max_workers)
def start(self):
if threading_utils.is_alive(self._worker):
raise RuntimeError("Worker thread must be stopped via stop()"
" before starting/restarting")
super(ParallelProcessTaskExecutor, self).start()
self._dispatcher.setup()
self._worker = threading_utils.daemon_thread(
asyncore.loop, map=self._dispatcher.map,
timeout=self._wait_timeout)
self._worker.start()
def stop(self):
super(ParallelProcessTaskExecutor, self).stop()
self._dispatcher.close()
if threading_utils.is_alive(self._worker):
self._worker.join()
self._worker = None
def _submit_task(self, func, task, *args, **kwargs):
"""Submit a function to run the given task (with given args/kwargs).
NOTE(harlowja): Adjust all events to be proxies instead since we want
those callbacks to be activated in this process, not in the child,
also since typically callbacks are functors (or callables) we can
not pickle those in the first place...
To make sure people understand how this works, the following is a
lengthy description of what is going on here, read at will:
So to ensure that we are proxying task triggered events that occur
in the executed subprocess (which will be created and used by the
thing using the multiprocessing based executor) we need to establish
a link between that process and this process that ensures that when a
event is triggered in that task in that process that a corresponding
event is triggered on the original task that was requested to be ran
in this process.
To accomplish this we have to create a copy of the task (without
any listeners) and then reattach a new set of listeners that will
now instead of calling the desired listeners just place messages
for this process (a dispatcher thread that is created in this class)
to dispatch to the original task (using a common accepting socket and
per task sender socket that is used and associated to know
which task to proxy back too, since it is possible that there many
be *many* subprocess running at the same time).
Once the subprocess task has finished execution, the executor will
then trigger a callback that will remove the task + target from the
dispatcher (which will stop any further proxying back to the original
task).
"""
progress_callback = kwargs.pop('progress_callback', None)
clone = task.copy(retain_listeners=False)
identity = _create_random_string(32)
channel = Channel(self._dispatcher.port, identity, self._auth_key)
def rebind_task():
# Creates and binds proxies for all events the task could receive
# so that when the clone runs in another process that this task
# can receive the same notifications (thus making it look like the
# the notifications are transparently happening in this process).
proxy_event_types = set()
for (event_type, listeners) in task.notifier.listeners_iter():
if listeners:
proxy_event_types.add(event_type)
if progress_callback is not None:
proxy_event_types.add(ta.EVENT_UPDATE_PROGRESS)
if nt.Notifier.ANY in proxy_event_types:
# NOTE(harlowja): If ANY is present, just have it be
# the **only** event registered, as all other events will be
# sent if ANY is registered (due to the nature of ANY sending
# all the things); if we also include the other event types
# in this set if ANY is present we will receive duplicate
# messages in this process (the one where the local
# task callbacks are being triggered). For example the
# emissions of the tasks notifier (that is running out
# of process) will for specific events send messages for
# its ANY event type **and** the specific event
# type (2 messages, when we just want one) which will
# cause > 1 notify() call on the local tasks notifier, which
# causes more local callback triggering than we want
# to actually happen.
proxy_event_types = set([nt.Notifier.ANY])
if proxy_event_types:
# This sender acts as our forwarding proxy target, it
# will be sent pickled to the process that will execute
# the needed task and it will do the work of using the
# channel object to send back messages to this process for
# dispatch into the local task.
sender = EventSender(channel)
for event_type in proxy_event_types:
clone.notifier.register(event_type, sender)
return bool(proxy_event_types)
def register():
if progress_callback is not None:
task.notifier.register(ta.EVENT_UPDATE_PROGRESS,
progress_callback)
self._dispatcher.targets[identity] = task
def deregister(fut=None):
if progress_callback is not None:
task.notifier.deregister(ta.EVENT_UPDATE_PROGRESS,
progress_callback)
self._dispatcher.targets.pop(identity, None)
should_register = rebind_task()
if should_register:
register()
try:
fut = self._executor.submit(func, clone, *args, **kwargs)
except RuntimeError:
with excutils.save_and_reraise_exception():
if should_register:
deregister()
fut.atom = task
if should_register:
fut.add_done_callback(deregister)
return fut

View File

@ -1,328 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2014 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import collections
import functools
from futurist import waiters
from taskflow import deciders as de
from taskflow.engines.action_engine.actions import retry as ra
from taskflow.engines.action_engine.actions import task as ta
from taskflow.engines.action_engine import builder as bu
from taskflow.engines.action_engine import compiler as com
from taskflow.engines.action_engine import completer as co
from taskflow.engines.action_engine import scheduler as sched
from taskflow.engines.action_engine import scopes as sc
from taskflow.engines.action_engine import selector as se
from taskflow.engines.action_engine import traversal as tr
from taskflow import exceptions as exc
from taskflow import logging
from taskflow import states as st
from taskflow.utils import misc
from taskflow.flow import (LINK_DECIDER, LINK_DECIDER_DEPTH) # noqa
# Small helper to make the edge decider tuples more easily useable...
_EdgeDecider = collections.namedtuple('_EdgeDecider',
'from_node,kind,decider,depth')
LOG = logging.getLogger(__name__)
class Runtime(object):
"""A aggregate of runtime objects, properties, ... used during execution.
This object contains various utility methods and properties that represent
the collection of runtime components and functionality needed for an
action engine to run to completion.
"""
def __init__(self, compilation, storage, atom_notifier,
task_executor, retry_executor,
options=None):
self._atom_notifier = atom_notifier
self._task_executor = task_executor
self._retry_executor = retry_executor
self._storage = storage
self._compilation = compilation
self._atom_cache = {}
self._options = misc.safe_copy_dict(options)
def _walk_edge_deciders(self, graph, atom):
"""Iterates through all nodes, deciders that alter atoms execution."""
# This is basically a reverse breadth first exploration, with
# special logic to further traverse down flow nodes as needed...
predecessors_iter = graph.predecessors_iter
nodes = collections.deque((u_node, atom)
for u_node in predecessors_iter(atom))
visited = set()
while nodes:
u_node, v_node = nodes.popleft()
u_node_kind = graph.node[u_node]['kind']
u_v_data = graph.adj[u_node][v_node]
try:
decider = u_v_data[LINK_DECIDER]
decider_depth = u_v_data.get(LINK_DECIDER_DEPTH)
if decider_depth is None:
decider_depth = de.Depth.ALL
yield _EdgeDecider(u_node, u_node_kind,
decider, decider_depth)
except KeyError:
pass
if u_node_kind == com.FLOW and u_node not in visited:
# Avoid re-exploring the same flow if we get to this same
# flow by a different *future* path...
visited.add(u_node)
# Since we *currently* jump over flow node(s), we need to make
# sure that any prior decider that was directed at this flow
# node also gets used during future decisions about this
# atom node.
nodes.extend((u_u_node, u_node)
for u_u_node in predecessors_iter(u_node))
def compile(self):
"""Compiles & caches frequently used execution helper objects.
Build out a cache of commonly used item that are associated
with the contained atoms (by name), and are useful to have for
quick lookup on (for example, the change state handler function for
each atom, the scope walker object for each atom, the task or retry
specific scheduler and so-on).
"""
change_state_handlers = {
com.TASK: functools.partial(self.task_action.change_state,
progress=0.0),
com.RETRY: self.retry_action.change_state,
}
schedulers = {
com.RETRY: self.retry_scheduler,
com.TASK: self.task_scheduler,
}
check_transition_handlers = {
com.TASK: st.check_task_transition,
com.RETRY: st.check_retry_transition,
}
actions = {
com.TASK: self.task_action,
com.RETRY: self.retry_action,
}
graph = self._compilation.execution_graph
for node, node_data in graph.nodes_iter(data=True):
node_kind = node_data['kind']
if node_kind in com.FLOWS:
continue
elif node_kind in com.ATOMS:
check_transition_handler = check_transition_handlers[node_kind]
change_state_handler = change_state_handlers[node_kind]
scheduler = schedulers[node_kind]
action = actions[node_kind]
else:
raise exc.CompilationFailure("Unknown node kind '%s'"
" encountered" % node_kind)
metadata = {}
deciders_it = self._walk_edge_deciders(graph, node)
walker = sc.ScopeWalker(self.compilation, node, names_only=True)
metadata['scope_walker'] = walker
metadata['check_transition_handler'] = check_transition_handler
metadata['change_state_handler'] = change_state_handler
metadata['scheduler'] = scheduler
metadata['edge_deciders'] = tuple(deciders_it)
metadata['action'] = action
LOG.trace("Compiled %s metadata for node %s (%s)",
metadata, node.name, node_kind)
self._atom_cache[node.name] = metadata
# TODO(harlowja): optimize the different decider depths to avoid
# repeated full successor searching; this can be done by searching
# for the widest depth of parent(s), and limiting the search of
# children by the that depth.
@property
def compilation(self):
return self._compilation
@property
def storage(self):
return self._storage
@property
def options(self):
return self._options
@misc.cachedproperty
def selector(self):
return se.Selector(self)
@misc.cachedproperty
def builder(self):
return bu.MachineBuilder(self, waiters.wait_for_any)
@misc.cachedproperty
def completer(self):
return co.Completer(self)
@misc.cachedproperty
def scheduler(self):
return sched.Scheduler(self)
@misc.cachedproperty
def task_scheduler(self):
return sched.TaskScheduler(self)
@misc.cachedproperty
def retry_scheduler(self):
return sched.RetryScheduler(self)
@misc.cachedproperty
def retry_action(self):
return ra.RetryAction(self._storage,
self._atom_notifier,
self._retry_executor)
@misc.cachedproperty
def task_action(self):
return ta.TaskAction(self._storage,
self._atom_notifier,
self._task_executor)
def _fetch_atom_metadata_entry(self, atom_name, metadata_key):
return self._atom_cache[atom_name][metadata_key]
def check_atom_transition(self, atom, current_state, target_state):
"""Checks if the atom can transition to the provided target state."""
# This does not check if the name exists (since this is only used
# internally to the engine, and is not exposed to atoms that will
# not exist and therefore doesn't need to handle that case).
check_transition_handler = self._fetch_atom_metadata_entry(
atom.name, 'check_transition_handler')
return check_transition_handler(current_state, target_state)
def fetch_edge_deciders(self, atom):
"""Fetches the edge deciders for the given atom."""
# This does not check if the name exists (since this is only used
# internally to the engine, and is not exposed to atoms that will
# not exist and therefore doesn't need to handle that case).
return self._fetch_atom_metadata_entry(atom.name, 'edge_deciders')
def fetch_scheduler(self, atom):
"""Fetches the cached specific scheduler for the given atom."""
# This does not check if the name exists (since this is only used
# internally to the engine, and is not exposed to atoms that will
# not exist and therefore doesn't need to handle that case).
return self._fetch_atom_metadata_entry(atom.name, 'scheduler')
def fetch_action(self, atom):
"""Fetches the cached action handler for the given atom."""
metadata = self._atom_cache[atom.name]
return metadata['action']
def fetch_scopes_for(self, atom_name):
"""Fetches a walker of the visible scopes for the given atom."""
try:
return self._fetch_atom_metadata_entry(atom_name, 'scope_walker')
except KeyError:
# This signals to the caller that there is no walker for whatever
# atom name was given that doesn't really have any associated atom
# known to be named with that name; this is done since the storage
# layer will call into this layer to fetch a scope for a named
# atom and users can provide random names that do not actually
# exist...
return None
# Various helper methods used by the runtime components; not for public
# consumption...
def iterate_retries(self, state=None):
"""Iterates retry atoms that match the provided state.
If no state is provided it will yield back all retry atoms.
"""
if state:
atoms = list(self.iterate_nodes((com.RETRY,)))
atom_states = self._storage.get_atoms_states(atom.name
for atom in atoms)
for atom in atoms:
atom_state, _atom_intention = atom_states[atom.name]
if atom_state == state:
yield atom
else:
for atom in self.iterate_nodes((com.RETRY,)):
yield atom
def iterate_nodes(self, allowed_kinds):
"""Yields back all nodes of specified kinds in the execution graph."""
graph = self._compilation.execution_graph
for node, node_data in graph.nodes_iter(data=True):
if node_data['kind'] in allowed_kinds:
yield node
def is_success(self):
"""Checks if all atoms in the execution graph are in 'happy' state."""
atoms = list(self.iterate_nodes(com.ATOMS))
atom_states = self._storage.get_atoms_states(atom.name
for atom in atoms)
for atom in atoms:
atom_state, _atom_intention = atom_states[atom.name]
if atom_state == st.IGNORE:
continue
if atom_state != st.SUCCESS:
return False
return True
def find_retry(self, node):
"""Returns the retry atom associated to the given node (or none)."""
graph = self._compilation.execution_graph
return graph.node[node].get(com.RETRY)
def reset_atoms(self, atoms, state=st.PENDING, intention=st.EXECUTE):
"""Resets all the provided atoms to the given state and intention."""
tweaked = []
for atom in atoms:
if state or intention:
tweaked.append((atom, state, intention))
if state:
change_state_handler = self._fetch_atom_metadata_entry(
atom.name, 'change_state_handler')
change_state_handler(atom, state)
if intention:
self.storage.set_atom_intention(atom.name, intention)
return tweaked
def reset_all(self, state=st.PENDING, intention=st.EXECUTE):
"""Resets all atoms to the given state and intention."""
return self.reset_atoms(self.iterate_nodes(com.ATOMS),
state=state, intention=intention)
def reset_subgraph(self, atom, state=st.PENDING, intention=st.EXECUTE):
"""Resets a atoms subgraph to the given state and intention.
The subgraph is contained of **all** of the atoms successors.
"""
execution_graph = self._compilation.execution_graph
atoms_it = tr.depth_first_iterate(execution_graph, atom,
tr.Direction.FORWARD)
return self.reset_atoms(atoms_it, state=state, intention=intention)
def retry_subflow(self, retry):
"""Prepares a retrys + its subgraph for execution.
This sets the retrys intention to ``EXECUTE`` and resets all of its
subgraph (its successors) to the ``PENDING`` state with an ``EXECUTE``
intention.
"""
tweaked = self.reset_atoms([retry], state=None, intention=st.EXECUTE)
tweaked.extend(self.reset_subgraph(retry))
return tweaked

View File

@ -1,103 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2014 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import weakref
from taskflow import exceptions as excp
from taskflow import states as st
from taskflow.types import failure
class RetryScheduler(object):
"""Schedules retry atoms."""
def __init__(self, runtime):
self._runtime = weakref.proxy(runtime)
self._retry_action = runtime.retry_action
self._storage = runtime.storage
def schedule(self, retry):
"""Schedules the given retry atom for *future* completion.
Depending on the atoms stored intention this may schedule the retry
atom for reversion or execution.
"""
intention = self._storage.get_atom_intention(retry.name)
if intention == st.EXECUTE:
return self._retry_action.schedule_execution(retry)
elif intention == st.REVERT:
return self._retry_action.schedule_reversion(retry)
elif intention == st.RETRY:
self._retry_action.change_state(retry, st.RETRYING)
# This will force the subflow to start processing right *after*
# this retry atom executes (since they will be blocked on their
# predecessor getting out of the RETRYING/RUNNING state).
self._runtime.retry_subflow(retry)
return self._retry_action.schedule_execution(retry)
else:
raise excp.ExecutionFailure("Unknown how to schedule retry with"
" intention: %s" % intention)
class TaskScheduler(object):
"""Schedules task atoms."""
def __init__(self, runtime):
self._storage = runtime.storage
self._task_action = runtime.task_action
def schedule(self, task):
"""Schedules the given task atom for *future* completion.
Depending on the atoms stored intention this may schedule the task
atom for reversion or execution.
"""
intention = self._storage.get_atom_intention(task.name)
if intention == st.EXECUTE:
return self._task_action.schedule_execution(task)
elif intention == st.REVERT:
return self._task_action.schedule_reversion(task)
else:
raise excp.ExecutionFailure("Unknown how to schedule task with"
" intention: %s" % intention)
class Scheduler(object):
"""Safely schedules atoms using a runtime ``fetch_scheduler`` routine."""
def __init__(self, runtime):
self._runtime = weakref.proxy(runtime)
def schedule(self, atoms):
"""Schedules the provided atoms for *future* completion.
This method should schedule a future for each atom provided and return
a set of those futures to be waited on (or used for other similar
purposes). It should also return any failure objects that represented
scheduling failures that may have occurred during this scheduling
process.
"""
futures = set()
for atom in atoms:
scheduler = self._runtime.fetch_scheduler(atom)
try:
futures.add(scheduler.schedule(atom))
except Exception:
# Immediately stop scheduling future work so that we can
# exit execution early (rather than later) if a single atom
# fails to schedule correctly.
return (futures, [failure.Failure()])
return (futures, [])

View File

@ -1,118 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2014 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from taskflow.engines.action_engine import compiler as co
from taskflow.engines.action_engine import traversal as tr
from taskflow import logging
LOG = logging.getLogger(__name__)
class ScopeWalker(object):
"""Walks through the scopes of a atom using a engines compilation.
NOTE(harlowja): for internal usage only.
This will walk the visible scopes that are accessible for the given
atom, which can be used by some external entity in some meaningful way,
for example to find dependent values...
"""
def __init__(self, compilation, atom, names_only=False):
self._node = compilation.hierarchy.find(atom)
if self._node is None:
raise ValueError("Unable to find atom '%s' in compilation"
" hierarchy" % atom)
self._level_cache = {}
self._atom = atom
self._execution_graph = compilation.execution_graph
self._names_only = names_only
self._predecessors = None
def __iter__(self):
"""Iterates over the visible scopes.
How this works is the following:
We first grab all the predecessors of the given atom (lets call it
``Y``) by using the :py:class:`~.compiler.Compilation` execution
graph (and doing a reverse breadth-first expansion to gather its
predecessors), this is useful since we know they *always* will
exist (and execute) before this atom but it does not tell us the
corresponding scope *level* (flow, nested flow...) that each
predecessor was created in, so we need to find this information.
For that information we consult the location of the atom ``Y`` in the
:py:class:`~.compiler.Compilation` hierarchy/tree. We lookup in a
reverse order the parent ``X`` of ``Y`` and traverse backwards from
the index in the parent where ``Y`` exists to all siblings (and
children of those siblings) in ``X`` that we encounter in this
backwards search (if a sibling is a flow itself, its atom(s)
will be recursively expanded and included). This collection will
then be assumed to be at the same scope. This is what is called
a *potential* single scope, to make an *actual* scope we remove the
items from the *potential* scope that are **not** predecessors
of ``Y`` to form the *actual* scope which we then yield back.
Then for additional scopes we continue up the tree, by finding the
parent of ``X`` (lets call it ``Z``) and perform the same operation,
going through the children in a reverse manner from the index in
parent ``Z`` where ``X`` was located. This forms another *potential*
scope which we provide back as an *actual* scope after reducing the
potential set to only include predecessors previously gathered. We
then repeat this process until we no longer have any parent
nodes (aka we have reached the top of the tree) or we run out of
predecessors.
"""
graph = self._execution_graph
if self._predecessors is None:
predecessors = set(
node for node in graph.bfs_predecessors_iter(self._atom)
if graph.node[node]['kind'] in co.ATOMS)
self._predecessors = predecessors.copy()
else:
predecessors = self._predecessors.copy()
last = self._node
for lvl, parent in enumerate(self._node.path_iter(include_self=False)):
if not predecessors:
break
last_idx = parent.index(last.item)
try:
visible, removals = self._level_cache[lvl]
predecessors = predecessors - removals
except KeyError:
visible = []
removals = set()
atom_it = tr.depth_first_reverse_iterate(
parent, start_from_idx=last_idx)
for atom in atom_it:
if atom in predecessors:
predecessors.remove(atom)
removals.add(atom)
visible.append(atom)
if not predecessors:
break
self._level_cache[lvl] = (visible, removals)
if LOG.isEnabledFor(logging.TRACE):
visible_names = [a.name for a in visible]
LOG.trace("Scope visible to '%s' (limited by parent '%s'"
" index < %s) is: %s", self._atom,
parent.item.name, last_idx, visible_names)
if self._names_only:
yield [a.name for a in visible]
else:
yield visible
last = parent

View File

@ -1,228 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2013 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import operator
import weakref
from taskflow.engines.action_engine import compiler as co
from taskflow.engines.action_engine import deciders
from taskflow.engines.action_engine import traversal
from taskflow import logging
from taskflow import states as st
from taskflow.utils import iter_utils
LOG = logging.getLogger(__name__)
class Selector(object):
"""Selector that uses a compilation and aids in execution processes.
Its primary purpose is to get the next atoms for execution or reversion
by utilizing the compilations underlying structures (graphs, nodes and
edge relations...) and using this information along with the atom
state/states stored in storage to provide other useful functionality to
the rest of the runtime system.
"""
def __init__(self, runtime):
self._runtime = weakref.proxy(runtime)
self._storage = runtime.storage
self._execution_graph = runtime.compilation.execution_graph
def iter_next_atoms(self, atom=None):
"""Iterate next atoms to run (originating from atom or all atoms)."""
if atom is None:
return iter_utils.unique_seen((self._browse_atoms_for_execute(),
self._browse_atoms_for_revert()),
seen_selector=operator.itemgetter(0))
state = self._storage.get_atom_state(atom.name)
intention = self._storage.get_atom_intention(atom.name)
if state == st.SUCCESS:
if intention == st.REVERT:
return iter([
(atom, deciders.NoOpDecider()),
])
elif intention == st.EXECUTE:
return self._browse_atoms_for_execute(atom=atom)
else:
return iter([])
elif state == st.REVERTED:
return self._browse_atoms_for_revert(atom=atom)
elif state == st.FAILURE:
return self._browse_atoms_for_revert()
else:
return iter([])
def _browse_atoms_for_execute(self, atom=None):
"""Browse next atoms to execute.
This returns a iterator of atoms that *may* be ready to be
executed, if given a specific atom, it will only examine the successors
of that atom, otherwise it will examine the whole graph.
"""
if atom is None:
atom_it = self._runtime.iterate_nodes(co.ATOMS)
else:
# NOTE(harlowja): the reason this uses breadth first is so that
# when deciders are applied that those deciders can be applied
# from top levels to lower levels since lower levels *may* be
# able to run even if top levels have deciders that decide to
# ignore some atoms... (going deeper first would make this
# problematic to determine as top levels can have their deciders
# applied **after** going deeper).
atom_it = traversal.breadth_first_iterate(
self._execution_graph, atom, traversal.Direction.FORWARD)
for atom in atom_it:
is_ready, late_decider = self._get_maybe_ready_for_execute(atom)
if is_ready:
yield (atom, late_decider)
def _browse_atoms_for_revert(self, atom=None):
"""Browse next atoms to revert.
This returns a iterator of atoms that *may* be ready to be be
reverted, if given a specific atom it will only examine the
predecessors of that atom, otherwise it will examine the whole
graph.
"""
if atom is None:
atom_it = self._runtime.iterate_nodes(co.ATOMS)
else:
atom_it = traversal.breadth_first_iterate(
self._execution_graph, atom, traversal.Direction.BACKWARD,
# Stop at the retry boundary (as retries 'control' there
# surronding atoms, and we don't want to back track over
# them so that they can correctly affect there associated
# atoms); we do though need to jump through all tasks since
# if a predecessor Y was ignored and a predecessor Z before Y
# was not it should be eligible to now revert...
through_retries=False)
for atom in atom_it:
is_ready, late_decider = self._get_maybe_ready_for_revert(atom)
if is_ready:
yield (atom, late_decider)
def _get_maybe_ready(self, atom, transition_to, allowed_intentions,
connected_fetcher, ready_checker,
decider_fetcher, for_what="?"):
def iter_connected_states():
# Lazily iterate over connected states so that ready checkers
# can stop early (vs having to consume and check all the
# things...)
for atom in connected_fetcher():
# TODO(harlowja): make this storage api better, its not
# especially clear what the following is doing (mainly
# to avoid two calls into storage).
atom_states = self._storage.get_atoms_states([atom.name])
yield (atom, atom_states[atom.name])
# NOTE(harlowja): How this works is the following...
#
# 1. First check if the current atom can even transition to the
# desired state, if not this atom is definitely not ready to
# execute or revert.
# 2. Check if the actual atoms intention is in one of the desired/ok
# intentions, if it is not there we are still not ready to execute
# or revert.
# 3. Iterate over (atom, atom_state, atom_intention) for all the
# atoms the 'connected_fetcher' callback yields from underlying
# storage and direct that iterator into the 'ready_checker'
# callback, that callback should then iterate over these entries
# and determine if it is ok to execute or revert.
# 4. If (and only if) 'ready_checker' returns true, then
# the 'decider_fetcher' callback is called to get a late decider
# which can (if it desires) affect this ready result (but does
# so right before the atom is about to be scheduled).
state = self._storage.get_atom_state(atom.name)
ok_to_transition = self._runtime.check_atom_transition(atom, state,
transition_to)
if not ok_to_transition:
LOG.trace("Atom '%s' is not ready to %s since it can not"
" transition to %s from its current state %s",
atom, for_what, transition_to, state)
return (False, None)
intention = self._storage.get_atom_intention(atom.name)
if intention not in allowed_intentions:
LOG.trace("Atom '%s' is not ready to %s since its current"
" intention %s is not in allowed intentions %s",
atom, for_what, intention, allowed_intentions)
return (False, None)
ok_to_run = ready_checker(iter_connected_states())
if not ok_to_run:
return (False, None)
else:
return (True, decider_fetcher())
def _get_maybe_ready_for_execute(self, atom):
"""Returns if an atom is *likely* ready to be executed."""
def ready_checker(pred_connected_it):
for pred in pred_connected_it:
pred_atom, (pred_atom_state, pred_atom_intention) = pred
if (pred_atom_state in (st.SUCCESS, st.IGNORE) and
pred_atom_intention in (st.EXECUTE, st.IGNORE)):
continue
LOG.trace("Unable to begin to execute since predecessor"
" atom '%s' is in state %s with intention %s",
pred_atom, pred_atom_state, pred_atom_intention)
return False
LOG.trace("Able to let '%s' execute", atom)
return True
decider_fetcher = lambda: \
deciders.IgnoreDecider(
atom, self._runtime.fetch_edge_deciders(atom))
connected_fetcher = lambda: \
traversal.depth_first_iterate(self._execution_graph, atom,
# Whether the desired atom
# can execute is dependent on its
# predecessors outcomes (thus why
# we look backwards).
traversal.Direction.BACKWARD)
# If this atoms current state is able to be transitioned to RUNNING
# and its intention is to EXECUTE and all of its predecessors executed
# successfully or were ignored then this atom is ready to execute.
LOG.trace("Checking if '%s' is ready to execute", atom)
return self._get_maybe_ready(atom, st.RUNNING, [st.EXECUTE],
connected_fetcher, ready_checker,
decider_fetcher, for_what='execute')
def _get_maybe_ready_for_revert(self, atom):
"""Returns if an atom is *likely* ready to be reverted."""
def ready_checker(succ_connected_it):
for succ in succ_connected_it:
succ_atom, (succ_atom_state, _succ_atom_intention) = succ
if succ_atom_state not in (st.PENDING, st.REVERTED, st.IGNORE):
LOG.trace("Unable to begin to revert since successor"
" atom '%s' is in state %s", succ_atom,
succ_atom_state)
return False
LOG.trace("Able to let '%s' revert", atom)
return True
noop_decider = deciders.NoOpDecider()
connected_fetcher = lambda: \
traversal.depth_first_iterate(self._execution_graph, atom,
# Whether the desired atom
# can revert is dependent on its
# successors states (thus why we
# look forwards).
traversal.Direction.FORWARD)
decider_fetcher = lambda: noop_decider
# If this atoms current state is able to be transitioned to REVERTING
# and its intention is either REVERT or RETRY and all of its
# successors are either PENDING or REVERTED then this atom is ready
# to revert.
LOG.trace("Checking if '%s' is ready to revert", atom)
return self._get_maybe_ready(atom, st.REVERTING, [st.REVERT, st.RETRY],
connected_fetcher, ready_checker,
decider_fetcher, for_what='revert')

View File

@ -1,126 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2015 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import collections
import enum
from taskflow.engines.action_engine import compiler as co
class Direction(enum.Enum):
"""Traversal direction enum."""
#: Go through successors.
FORWARD = 1
#: Go through predecessors.
BACKWARD = 2
def _extract_connectors(execution_graph, starting_node, direction,
through_flows=True, through_retries=True,
through_tasks=True):
if direction == Direction.FORWARD:
connected_iter = execution_graph.successors_iter
else:
connected_iter = execution_graph.predecessors_iter
connected_to_functors = {}
if through_flows:
connected_to_functors[co.FLOW] = connected_iter
connected_to_functors[co.FLOW_END] = connected_iter
if through_retries:
connected_to_functors[co.RETRY] = connected_iter
if through_tasks:
connected_to_functors[co.TASK] = connected_iter
return connected_iter(starting_node), connected_to_functors
def breadth_first_iterate(execution_graph, starting_node, direction,
through_flows=True, through_retries=True,
through_tasks=True):
"""Iterates connected nodes in execution graph (from starting node).
Does so in a breadth first manner.
Jumps over nodes with ``noop`` attribute (does not yield them back).
"""
initial_nodes_iter, connected_to_functors = _extract_connectors(
execution_graph, starting_node, direction,
through_flows=through_flows, through_retries=through_retries,
through_tasks=through_tasks)
q = collections.deque(initial_nodes_iter)
while q:
node = q.popleft()
node_attrs = execution_graph.node[node]
if not node_attrs.get('noop'):
yield node
try:
node_kind = node_attrs['kind']
connected_to_functor = connected_to_functors[node_kind]
except KeyError:
pass
else:
q.extend(connected_to_functor(node))
def depth_first_iterate(execution_graph, starting_node, direction,
through_flows=True, through_retries=True,
through_tasks=True):
"""Iterates connected nodes in execution graph (from starting node).
Does so in a depth first manner.
Jumps over nodes with ``noop`` attribute (does not yield them back).
"""
initial_nodes_iter, connected_to_functors = _extract_connectors(
execution_graph, starting_node, direction,
through_flows=through_flows, through_retries=through_retries,
through_tasks=through_tasks)
stack = list(initial_nodes_iter)
while stack:
node = stack.pop()
node_attrs = execution_graph.node[node]
if not node_attrs.get('noop'):
yield node
try:
node_kind = node_attrs['kind']
connected_to_functor = connected_to_functors[node_kind]
except KeyError:
pass
else:
stack.extend(connected_to_functor(node))
def depth_first_reverse_iterate(node, start_from_idx=-1):
"""Iterates connected (in reverse) **tree** nodes (from starting node).
Jumps through nodes with ``noop`` attribute (does not yield them back).
"""
# Always go left to right, since right to left is the pattern order
# and we want to go backwards and not forwards through that ordering...
if start_from_idx == -1:
# All of them...
children_iter = node.reverse_iter()
else:
children_iter = reversed(node[0:start_from_idx])
for child in children_iter:
if child.metadata.get('noop'):
# Jump through these...
for grand_child in child.dfs_iter(right_to_left=False):
if grand_child.metadata['kind'] in co.ATOMS:
yield grand_child.item
else:
yield child.item

View File

@ -1,135 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2013 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import abc
import six
from taskflow.types import notifier
from taskflow.utils import misc
@six.add_metaclass(abc.ABCMeta)
class Engine(object):
"""Base for all engines implementations.
:ivar Engine.notifier: A notification object that will dispatch
events that occur related to the flow the engine
contains.
:ivar atom_notifier: A notification object that will dispatch events that
occur related to the atoms the engine contains.
"""
def __init__(self, flow, flow_detail, backend, options):
self._flow = flow
self._flow_detail = flow_detail
self._backend = backend
self._options = misc.safe_copy_dict(options)
self._notifier = notifier.Notifier()
self._atom_notifier = notifier.Notifier()
@property
def notifier(self):
"""The flow notifier."""
return self._notifier
@property
def atom_notifier(self):
"""The atom notifier."""
return self._atom_notifier
@property
def options(self):
"""The options that were passed to this engine on construction."""
return self._options
@abc.abstractproperty
def storage(self):
"""The storage unit for this engine."""
@abc.abstractproperty
def statistics(self):
"""A dictionary of runtime statistics this engine has gathered.
This dictionary will be empty when the engine has never been
ran. When it is running or has ran previously it should have (but
may not) have useful and/or informational keys and values when
running is underway and/or completed.
.. warning:: The keys in this dictionary **should** be some what
stable (not changing), but there existence **may**
change between major releases as new statistics are
gathered or removed so before accessing keys ensure that
they actually exist and handle when they do not.
"""
@abc.abstractmethod
def compile(self):
"""Compiles the contained flow into a internal representation.
This internal representation is what the engine will *actually* use to
run. If this compilation can not be accomplished then an exception
is expected to be thrown with a message indicating why the compilation
could not be achieved.
"""
@abc.abstractmethod
def reset(self):
"""Reset back to the ``PENDING`` state.
If a flow had previously ended up (from a prior engine
:py:func:`.run`) in the ``FAILURE``, ``SUCCESS`` or ``REVERTED``
states (or for some reason it ended up in an intermediary state) it
can be desirable to make it possible to run it again. Calling this
method enables that to occur (without causing a state transition
failure, which would typically occur if :py:meth:`.run` is called
directly without doing a reset).
"""
@abc.abstractmethod
def prepare(self):
"""Performs any pre-run, but post-compilation actions.
NOTE(harlowja): During preparation it is currently assumed that the
underlying storage will be initialized, the atoms will be reset and
the engine will enter the ``PENDING`` state.
"""
@abc.abstractmethod
def validate(self):
"""Performs any pre-run, post-prepare validation actions.
NOTE(harlowja): During validation all final dependencies
will be verified and ensured. This will by default check that all
atoms have satisfiable requirements (satisfied by some other
provider).
"""
@abc.abstractmethod
def run(self):
"""Runs the flow in the engine to completion (or die trying)."""
@abc.abstractmethod
def suspend(self):
"""Attempts to suspend the engine.
If the engine is currently running atoms then this will attempt to
suspend future work from being started (currently active atoms can
not currently be preempted) and move the engine into a suspend state
which can then later be resumed from.
"""

View File

@ -1,286 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2013 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import contextlib
from oslo_utils import importutils
from oslo_utils import reflection
import six
import stevedore.driver
from taskflow import exceptions as exc
from taskflow import logging
from taskflow.persistence import backends as p_backends
from taskflow.utils import misc
from taskflow.utils import persistence_utils as p_utils
LOG = logging.getLogger(__name__)
# NOTE(imelnikov): this is the entrypoint namespace, not the module namespace.
ENGINES_NAMESPACE = 'taskflow.engines'
# The default entrypoint engine type looked for when it is not provided.
ENGINE_DEFAULT = 'default'
def _extract_engine(engine, **kwargs):
"""Extracts the engine kind and any associated options."""
kind = engine
if not kind:
kind = ENGINE_DEFAULT
# See if it's a URI and if so, extract any further options...
options = {}
try:
uri = misc.parse_uri(kind)
except (TypeError, ValueError):
pass
else:
kind = uri.scheme
options = misc.merge_uri(uri, options.copy())
# Merge in any leftover **kwargs into the options, this makes it so
# that the provided **kwargs override any URI/engine specific
# options.
options.update(kwargs)
return (kind, options)
def _fetch_factory(factory_name):
try:
return importutils.import_class(factory_name)
except (ImportError, ValueError) as e:
raise ImportError("Could not import factory %r: %s"
% (factory_name, e))
def _fetch_validate_factory(flow_factory):
if isinstance(flow_factory, six.string_types):
factory_fun = _fetch_factory(flow_factory)
factory_name = flow_factory
else:
factory_fun = flow_factory
factory_name = reflection.get_callable_name(flow_factory)
try:
reimported = _fetch_factory(factory_name)
assert reimported == factory_fun
except (ImportError, AssertionError):
raise ValueError('Flow factory %r is not reimportable by name %s'
% (factory_fun, factory_name))
return (factory_name, factory_fun)
def load(flow, store=None, flow_detail=None, book=None,
backend=None, namespace=ENGINES_NAMESPACE,
engine=ENGINE_DEFAULT, **kwargs):
"""Load a flow into an engine.
This function creates and prepares an engine to run the provided flow. All
that is left after this returns is to run the engine with the
engines :py:meth:`~taskflow.engines.base.Engine.run` method.
Which engine to load is specified via the ``engine`` parameter. It
can be a string that names the engine type to use, or a string that
is a URI with a scheme that names the engine type to use and further
options contained in the URI's host, port, and query parameters...
Which storage backend to use is defined by the backend parameter. It
can be backend itself, or a dictionary that is passed to
:py:func:`~taskflow.persistence.backends.fetch` to obtain a
viable backend.
:param flow: flow to load
:param store: dict -- data to put to storage to satisfy flow requirements
:param flow_detail: FlowDetail that holds the state of the flow (if one is
not provided then one will be created for you in the provided backend)
:param book: LogBook to create flow detail in if flow_detail is None
:param backend: storage backend to use or configuration that defines it
:param namespace: driver namespace for stevedore (or empty for default)
:param engine: string engine type or URI string with scheme that contains
the engine type and any URI specific components that will
become part of the engine options.
:param kwargs: arbitrary keyword arguments passed as options (merged with
any extracted ``engine``), typically used for any engine
specific options that do not fit as any of the
existing arguments.
:returns: engine
"""
kind, options = _extract_engine(engine, **kwargs)
if isinstance(backend, dict):
backend = p_backends.fetch(backend)
if flow_detail is None:
flow_detail = p_utils.create_flow_detail(flow, book=book,
backend=backend)
LOG.debug('Looking for %r engine driver in %r', kind, namespace)
try:
mgr = stevedore.driver.DriverManager(
namespace, kind,
invoke_on_load=True,
invoke_args=(flow, flow_detail, backend, options))
engine = mgr.driver
except RuntimeError as e:
raise exc.NotFound("Could not find engine '%s'" % (kind), e)
else:
if store:
engine.storage.inject(store)
return engine
def run(flow, store=None, flow_detail=None, book=None,
backend=None, namespace=ENGINES_NAMESPACE,
engine=ENGINE_DEFAULT, **kwargs):
"""Run the flow.
This function loads the flow into an engine (with the :func:`load() <load>`
function) and runs the engine.
The arguments are interpreted as for :func:`load() <load>`.
:returns: dictionary of all named
results (see :py:meth:`~.taskflow.storage.Storage.fetch_all`)
"""
engine = load(flow, store=store, flow_detail=flow_detail, book=book,
backend=backend, namespace=namespace,
engine=engine, **kwargs)
engine.run()
return engine.storage.fetch_all()
def save_factory_details(flow_detail,
flow_factory, factory_args, factory_kwargs,
backend=None):
"""Saves the given factories reimportable attributes into the flow detail.
This function saves the factory name, arguments, and keyword arguments
into the given flow details object and if a backend is provided it will
also ensure that the backend saves the flow details after being updated.
:param flow_detail: FlowDetail that holds state of the flow to load
:param flow_factory: function or string: function that creates the flow
:param factory_args: list or tuple of factory positional arguments
:param factory_kwargs: dict of factory keyword arguments
:param backend: storage backend to use or configuration
"""
if not factory_args:
factory_args = []
if not factory_kwargs:
factory_kwargs = {}
factory_name, _factory_fun = _fetch_validate_factory(flow_factory)
factory_data = {
'factory': {
'name': factory_name,
'args': factory_args,
'kwargs': factory_kwargs,
},
}
if not flow_detail.meta:
flow_detail.meta = factory_data
else:
flow_detail.meta.update(factory_data)
if backend is not None:
if isinstance(backend, dict):
backend = p_backends.fetch(backend)
with contextlib.closing(backend.get_connection()) as conn:
conn.update_flow_details(flow_detail)
def load_from_factory(flow_factory, factory_args=None, factory_kwargs=None,
store=None, book=None, backend=None,
namespace=ENGINES_NAMESPACE, engine=ENGINE_DEFAULT,
**kwargs):
"""Loads a flow from a factory function into an engine.
Gets flow factory function (or name of it) and creates flow with
it. Then, the flow is loaded into an engine with the :func:`load() <load>`
function, and the factory function fully qualified name is saved to flow
metadata so that it can be later resumed.
:param flow_factory: function or string: function that creates the flow
:param factory_args: list or tuple of factory positional arguments
:param factory_kwargs: dict of factory keyword arguments
Further arguments are interpreted as for :func:`load() <load>`.
:returns: engine
"""
_factory_name, factory_fun = _fetch_validate_factory(flow_factory)
if not factory_args:
factory_args = []
if not factory_kwargs:
factory_kwargs = {}
flow = factory_fun(*factory_args, **factory_kwargs)
if isinstance(backend, dict):
backend = p_backends.fetch(backend)
flow_detail = p_utils.create_flow_detail(flow, book=book, backend=backend)
save_factory_details(flow_detail,
flow_factory, factory_args, factory_kwargs,
backend=backend)
return load(flow=flow, store=store, flow_detail=flow_detail, book=book,
backend=backend, namespace=namespace,
engine=engine, **kwargs)
def flow_from_detail(flow_detail):
"""Reloads a flow previously saved.
Gets the flow factories name and any arguments and keyword arguments from
the flow details metadata, and then calls that factory to recreate the
flow.
:param flow_detail: FlowDetail that holds state of the flow to load
"""
try:
factory_data = flow_detail.meta['factory']
except (KeyError, AttributeError, TypeError):
raise ValueError('Cannot reconstruct flow %s %s: '
'no factory information saved.'
% (flow_detail.name, flow_detail.uuid))
try:
factory_fun = _fetch_factory(factory_data['name'])
except (KeyError, ImportError):
raise ImportError('Could not import factory for flow %s %s'
% (flow_detail.name, flow_detail.uuid))
args = factory_data.get('args', ())
kwargs = factory_data.get('kwargs', {})
return factory_fun(*args, **kwargs)
def load_from_detail(flow_detail, store=None, backend=None,
namespace=ENGINES_NAMESPACE, engine=ENGINE_DEFAULT,
**kwargs):
"""Reloads an engine previously saved.
This reloads the flow using the
:func:`flow_from_detail() <flow_from_detail>` function and then calls
into the :func:`load() <load>` function to create an engine from that flow.
:param flow_detail: FlowDetail that holds state of the flow to load
Further arguments are interpreted as for :func:`load() <load>`.
:returns: engine
"""
flow = flow_from_detail(flow_detail)
return load(flow, flow_detail=flow_detail,
store=store, backend=backend,
namespace=namespace, engine=engine, **kwargs)

View File

@ -1,167 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2014 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from kombu import exceptions as kombu_exc
from taskflow import exceptions as excp
from taskflow import logging
from taskflow.utils import kombu_utils as ku
LOG = logging.getLogger(__name__)
class Handler(object):
"""Component(s) that will be called on reception of messages."""
__slots__ = ['_process_message', '_validator']
def __init__(self, process_message, validator=None):
self._process_message = process_message
self._validator = validator
@property
def process_message(self):
"""Main callback that is called to process a received message.
This is only called after the format has been validated (using
the ``validator`` callback if applicable) and only after the message
has been acknowledged.
"""
return self._process_message
@property
def validator(self):
"""Optional callback that will be activated before processing.
This callback if present is expected to validate the message and
raise :py:class:`~taskflow.exceptions.InvalidFormat` if the message
is not valid.
"""
return self._validator
class TypeDispatcher(object):
"""Receives messages and dispatches to type specific handlers."""
def __init__(self, type_handlers=None, requeue_filters=None):
if type_handlers is not None:
self._type_handlers = dict(type_handlers)
else:
self._type_handlers = {}
if requeue_filters is not None:
self._requeue_filters = list(requeue_filters)
else:
self._requeue_filters = []
@property
def type_handlers(self):
"""Dictionary of message type -> callback to handle that message.
The callback(s) will be activated by looking for a message
property 'type' and locating a callback in this dictionary that maps
to that type; if one is found it is expected to be a callback that
accepts two positional parameters; the first being the message data
and the second being the message object. If a callback is not found
then the message is rejected and it will be up to the underlying
message transport to determine what this means/implies...
"""
return self._type_handlers
@property
def requeue_filters(self):
"""List of filters (callbacks) to request a message to be requeued.
The callback(s) will be activated before the message has been acked and
it can be used to instruct the dispatcher to requeue the message
instead of processing it. The callback, when called, will be provided
two positional parameters; the first being the message data and the
second being the message object. Using these provided parameters the
filter should return a truthy object if the message should be requeued
and a falsey object if it should not.
"""
return self._requeue_filters
def _collect_requeue_votes(self, data, message):
# Returns how many of the filters asked for the message to be requeued.
requeue_votes = 0
for i, cb in enumerate(self._requeue_filters):
try:
if cb(data, message):
requeue_votes += 1
except Exception:
LOG.exception("Failed calling requeue filter %s '%s' to"
" determine if message %r should be requeued.",
i + 1, cb, message.delivery_tag)
return requeue_votes
def _requeue_log_error(self, message, errors):
# TODO(harlowja): Remove when http://github.com/celery/kombu/pull/372
# is merged and a version is released with this change...
try:
message.requeue()
except errors as exc:
# This was taken from how kombu is formatting its messages
# when its reject_log_error or ack_log_error functions are
# used so that we have a similar error format for requeuing.
LOG.critical("Couldn't requeue %r, reason:%r",
message.delivery_tag, exc, exc_info=True)
else:
LOG.debug("Message '%s' was requeued.", ku.DelayedPretty(message))
def _process_message(self, data, message, message_type):
handler = self._type_handlers.get(message_type)
if handler is None:
message.reject_log_error(logger=LOG,
errors=(kombu_exc.MessageStateError,))
LOG.warning("Unexpected message type: '%s' in message"
" '%s'", message_type, ku.DelayedPretty(message))
else:
if handler.validator is not None:
try:
handler.validator(data)
except excp.InvalidFormat as e:
message.reject_log_error(
logger=LOG, errors=(kombu_exc.MessageStateError,))
LOG.warning("Message '%s' (%s) was rejected due to it"
" being in an invalid format: %s",
ku.DelayedPretty(message), message_type, e)
return
message.ack_log_error(logger=LOG,
errors=(kombu_exc.MessageStateError,))
if message.acknowledged:
LOG.debug("Message '%s' was acknowledged.",
ku.DelayedPretty(message))
handler.process_message(data, message)
else:
message.reject_log_error(logger=LOG,
errors=(kombu_exc.MessageStateError,))
def on_message(self, data, message):
"""This method is called on incoming messages."""
LOG.debug("Received message '%s'", ku.DelayedPretty(message))
if self._collect_requeue_votes(data, message):
self._requeue_log_error(message,
errors=(kombu_exc.MessageStateError,))
else:
try:
message_type = message.properties['type']
except KeyError:
message.reject_log_error(
logger=LOG, errors=(kombu_exc.MessageStateError,))
LOG.warning("The 'type' message property is missing"
" in message '%s'", ku.DelayedPretty(message))
else:
self._process_message(data, message, message_type)

View File

@ -1,49 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2014 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from oslo_utils import reflection
from taskflow.engines.action_engine import executor
class Endpoint(object):
"""Represents a single task with execute/revert methods."""
def __init__(self, task_cls):
self._task_cls = task_cls
self._task_cls_name = reflection.get_class_name(task_cls)
self._executor = executor.SerialTaskExecutor()
def __str__(self):
return self._task_cls_name
@property
def name(self):
return self._task_cls_name
def generate(self, name=None):
# NOTE(skudriashev): Note that task is created here with the `name`
# argument passed to its constructor. This will be a problem when
# task's constructor requires any other arguments.
return self._task_cls(name=name)
def execute(self, task, **kwargs):
event, result = self._executor.execute_task(task, **kwargs).result()
return result
def revert(self, task, **kwargs):
event, result = self._executor.revert_task(task, **kwargs).result()
return result

View File

@ -1,85 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2014 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from taskflow.engines.action_engine import engine
from taskflow.engines.worker_based import executor
from taskflow.engines.worker_based import protocol as pr
class WorkerBasedActionEngine(engine.ActionEngine):
"""Worker based action engine.
Specific backend options (extracted from provided engine options):
:param exchange: broker exchange exchange name in which executor / worker
communication is performed
:param url: broker connection url (see format in kombu documentation)
:param topics: list of workers topics to communicate with (this will also
be learned by listening to the notifications that workers
emit).
:param transport: transport to be used (e.g. amqp, memory, etc.)
:param transition_timeout: numeric value (or None for infinite) to wait
for submitted remote requests to transition out
of the (PENDING, WAITING) request states. When
expired the associated task the request was made
for will have its result become a
:py:class:`~taskflow.exceptions.RequestTimeout`
exception instead of its normally returned
value (or raised exception).
:param transport_options: transport specific options (see:
http://kombu.readthedocs.org/ for what these
options imply and are expected to be)
:param retry_options: retry specific options
(see: :py:attr:`~.proxy.Proxy.DEFAULT_RETRY_OPTIONS`)
:param worker_expiry: numeric value (or negative/zero/None for
infinite) that defines the number of seconds to
continue to send messages to workers that
have **not** responded back to a prior
notification/ping request (this defaults
to 60 seconds).
"""
def __init__(self, flow, flow_detail, backend, options):
super(WorkerBasedActionEngine, self).__init__(flow, flow_detail,
backend, options)
# This ensures that any provided executor will be validated before
# we get to far in the compilation/execution pipeline...
self._task_executor = self._fetch_task_executor(self._options,
self._flow_detail)
@classmethod
def _fetch_task_executor(cls, options, flow_detail):
try:
e = options['executor']
if not isinstance(e, executor.WorkerTaskExecutor):
raise TypeError("Expected an instance of type '%s' instead of"
" type '%s' for 'executor' option"
% (executor.WorkerTaskExecutor, type(e)))
return e
except KeyError:
return executor.WorkerTaskExecutor(
uuid=flow_detail.uuid,
url=options.get('url'),
exchange=options.get('exchange', 'default'),
retry_options=options.get('retry_options'),
topics=options.get('topics', []),
transport=options.get('transport'),
transport_options=options.get('transport_options'),
transition_timeout=options.get('transition_timeout',
pr.REQUEST_TIMEOUT),
worker_expiry=options.get('worker_expiry',
pr.EXPIRES_AFTER),
)

View File

@ -1,284 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2014 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import functools
import threading
from oslo_utils import timeutils
import six
from taskflow.engines.action_engine import executor
from taskflow.engines.worker_based import dispatcher
from taskflow.engines.worker_based import protocol as pr
from taskflow.engines.worker_based import proxy
from taskflow.engines.worker_based import types as wt
from taskflow import exceptions as exc
from taskflow import logging
from taskflow.task import EVENT_UPDATE_PROGRESS # noqa
from taskflow.utils import kombu_utils as ku
from taskflow.utils import misc
from taskflow.utils import threading_utils as tu
LOG = logging.getLogger(__name__)
class WorkerTaskExecutor(executor.TaskExecutor):
"""Executes tasks on remote workers."""
def __init__(self, uuid, exchange, topics,
transition_timeout=pr.REQUEST_TIMEOUT,
url=None, transport=None, transport_options=None,
retry_options=None, worker_expiry=pr.EXPIRES_AFTER):
self._uuid = uuid
self._ongoing_requests = {}
self._ongoing_requests_lock = threading.RLock()
self._transition_timeout = transition_timeout
self._proxy = proxy.Proxy(uuid, exchange,
on_wait=self._on_wait, url=url,
transport=transport,
transport_options=transport_options,
retry_options=retry_options)
# NOTE(harlowja): This is the most simplest finder impl. that
# doesn't have external dependencies (outside of what this engine
# already requires); it though does create periodic 'polling' traffic
# to workers to 'learn' of the tasks they can perform (and requires
# pre-existing knowledge of the topics those workers are on to gather
# and update this information).
self._finder = wt.ProxyWorkerFinder(uuid, self._proxy, topics,
worker_expiry=worker_expiry)
self._proxy.dispatcher.type_handlers.update({
pr.RESPONSE: dispatcher.Handler(self._process_response,
validator=pr.Response.validate),
pr.NOTIFY: dispatcher.Handler(
self._finder.process_response,
validator=functools.partial(pr.Notify.validate,
response=True)),
})
# Thread that will run the message dispatching (and periodically
# call the on_wait callback to do various things) loop...
self._helper = None
self._messages_processed = {
'finder': self._finder.messages_processed,
}
def _process_response(self, response, message):
"""Process response from remote side."""
LOG.debug("Started processing response message '%s'",
ku.DelayedPretty(message))
try:
request_uuid = message.properties['correlation_id']
except KeyError:
LOG.warning("The 'correlation_id' message property is"
" missing in message '%s'",
ku.DelayedPretty(message))
else:
request = self._ongoing_requests.get(request_uuid)
if request is not None:
response = pr.Response.from_dict(response)
LOG.debug("Extracted response '%s' and matched it to"
" request '%s'", response, request)
if response.state == pr.RUNNING:
request.transition_and_log_error(pr.RUNNING, logger=LOG)
elif response.state == pr.EVENT:
# Proxy the event + details to the task notifier so
# that it shows up in the local process (and activates
# any local callbacks...); thus making it look like
# the task is running locally (in some regards).
event_type = response.data['event_type']
details = response.data['details']
request.task.notifier.notify(event_type, details)
elif response.state in (pr.FAILURE, pr.SUCCESS):
if request.transition_and_log_error(response.state,
logger=LOG):
with self._ongoing_requests_lock:
del self._ongoing_requests[request.uuid]
request.set_result(result=response.data['result'])
else:
LOG.warning("Unexpected response status '%s'",
response.state)
else:
LOG.debug("Request with id='%s' not found", request_uuid)
@staticmethod
def _handle_expired_request(request):
"""Handle a expired request.
When a request has expired it is removed from the ongoing requests
dictionary and a ``RequestTimeout`` exception is set as a
request result.
"""
if request.transition_and_log_error(pr.FAILURE, logger=LOG):
# Raise an exception (and then catch it) so we get a nice
# traceback that the request will get instead of it getting
# just an exception with no traceback...
try:
request_age = timeutils.now() - request.created_on
raise exc.RequestTimeout(
"Request '%s' has expired after waiting for %0.2f"
" seconds for it to transition out of (%s) states"
% (request, request_age, ", ".join(pr.WAITING_STATES)))
except exc.RequestTimeout:
with misc.capture_failure() as failure:
LOG.debug(failure.exception_str)
request.set_result(failure)
return True
return False
def _clean(self):
if not self._ongoing_requests:
return
with self._ongoing_requests_lock:
ongoing_requests_uuids = set(six.iterkeys(self._ongoing_requests))
waiting_requests = {}
expired_requests = {}
for request_uuid in ongoing_requests_uuids:
try:
request = self._ongoing_requests[request_uuid]
except KeyError:
# Guess it got removed before we got to it...
pass
else:
if request.expired:
expired_requests[request_uuid] = request
elif request.current_state == pr.WAITING:
waiting_requests[request_uuid] = request
if expired_requests:
with self._ongoing_requests_lock:
while expired_requests:
request_uuid, request = expired_requests.popitem()
if self._handle_expired_request(request):
del self._ongoing_requests[request_uuid]
if waiting_requests:
finder = self._finder
new_messages_processed = finder.messages_processed
last_messages_processed = self._messages_processed['finder']
if new_messages_processed > last_messages_processed:
# Some new message got to the finder, so we can see
# if any new workers match (if no new messages have been
# processed we might as well not do anything).
while waiting_requests:
_request_uuid, request = waiting_requests.popitem()
worker = finder.get_worker_for_task(request.task)
if (worker is not None and
request.transition_and_log_error(pr.PENDING,
logger=LOG)):
self._publish_request(request, worker)
self._messages_processed['finder'] = new_messages_processed
def _on_wait(self):
"""This function is called cyclically between draining events."""
# Publish any finding messages (used to locate workers).
self._finder.maybe_publish()
# If the finder hasn't heard from workers in a given amount
# of time, then those workers are likely dead, so clean them out...
self._finder.clean()
# Process any expired requests or requests that have no current
# worker located (publish messages for those if we now do have
# a worker located).
self._clean()
def _submit_task(self, task, task_uuid, action, arguments,
progress_callback=None, result=pr.NO_RESULT,
failures=None):
"""Submit task request to a worker."""
request = pr.Request(task, task_uuid, action, arguments,
timeout=self._transition_timeout,
result=result, failures=failures)
# Register the callback, so that we can proxy the progress correctly.
if (progress_callback is not None and
task.notifier.can_be_registered(EVENT_UPDATE_PROGRESS)):
task.notifier.register(EVENT_UPDATE_PROGRESS, progress_callback)
request.future.add_done_callback(
lambda _fut: task.notifier.deregister(EVENT_UPDATE_PROGRESS,
progress_callback))
# Get task's worker and publish request if worker was found.
worker = self._finder.get_worker_for_task(task)
if worker is not None:
if request.transition_and_log_error(pr.PENDING, logger=LOG):
with self._ongoing_requests_lock:
self._ongoing_requests[request.uuid] = request
self._publish_request(request, worker)
else:
LOG.debug("Delaying submission of '%s', no currently known"
" worker/s available to process it", request)
with self._ongoing_requests_lock:
self._ongoing_requests[request.uuid] = request
return request.future
def _publish_request(self, request, worker):
"""Publish request to a given topic."""
LOG.debug("Submitting execution of '%s' to worker '%s' (expecting"
" response identified by reply_to=%s and"
" correlation_id=%s) - waited %0.3f seconds to"
" get published", request, worker, self._uuid,
request.uuid, timeutils.now() - request.created_on)
try:
self._proxy.publish(request, worker.topic,
reply_to=self._uuid,
correlation_id=request.uuid)
except Exception:
with misc.capture_failure() as failure:
LOG.critical("Failed to submit '%s' (transitioning it to"
" %s)", request, pr.FAILURE, exc_info=True)
if request.transition_and_log_error(pr.FAILURE, logger=LOG):
with self._ongoing_requests_lock:
del self._ongoing_requests[request.uuid]
request.set_result(failure)
def execute_task(self, task, task_uuid, arguments,
progress_callback=None):
return self._submit_task(task, task_uuid, pr.EXECUTE, arguments,
progress_callback=progress_callback)
def revert_task(self, task, task_uuid, arguments, result, failures,
progress_callback=None):
return self._submit_task(task, task_uuid, pr.REVERT, arguments,
result=result, failures=failures,
progress_callback=progress_callback)
def wait_for_workers(self, workers=1, timeout=None):
"""Waits for geq workers to notify they are ready to do work.
NOTE(harlowja): if a timeout is provided this function will wait
until that timeout expires, if the amount of workers does not reach
the desired amount of workers before the timeout expires then this will
return how many workers are still needed, otherwise it will
return zero.
"""
return self._finder.wait_for_workers(workers=workers,
timeout=timeout)
def start(self):
"""Starts message processing thread."""
if self._helper is not None:
raise RuntimeError("Worker executor must be stopped before"
" it can be started")
self._helper = tu.daemon_thread(self._proxy.start)
self._helper.start()
self._proxy.wait()
def stop(self):
"""Stops message processing thread."""
if self._helper is not None:
self._proxy.stop()
self._helper.join()
self._helper = None
with self._ongoing_requests_lock:
while self._ongoing_requests:
_request_uuid, request = self._ongoing_requests.popitem()
self._handle_expired_request(request)
self._finder.reset()
self._messages_processed['finder'] = self._finder.messages_processed

View File

@ -1,571 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2014 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import abc
import collections
import threading
from automaton import exceptions as machine_excp
from automaton import machines
import fasteners
import futurist
from oslo_serialization import jsonutils
from oslo_utils import reflection
from oslo_utils import timeutils
import six
from taskflow.engines.action_engine import executor
from taskflow import exceptions as excp
from taskflow import logging
from taskflow.types import failure as ft
from taskflow.utils import schema_utils as su
# NOTE(skudriashev): This is protocol states and events, which are not
# related to task states.
WAITING = 'WAITING'
PENDING = 'PENDING'
RUNNING = 'RUNNING'
SUCCESS = 'SUCCESS'
FAILURE = 'FAILURE'
EVENT = 'EVENT'
# During these states the expiry is active (once out of these states the expiry
# no longer matters, since we have no way of knowing how long a task will run
# for).
WAITING_STATES = (WAITING, PENDING)
# Once these states have been entered a request can no longer be
# automatically expired.
STOP_TIMER_STATES = (RUNNING, SUCCESS, FAILURE)
# Remote task actions.
EXECUTE = 'execute'
REVERT = 'revert'
# Remote task action to event map.
ACTION_TO_EVENT = {
EXECUTE: executor.EXECUTED,
REVERT: executor.REVERTED
}
# NOTE(skudriashev): A timeout which specifies request expiration period.
REQUEST_TIMEOUT = 60
# NOTE(skudriashev): A timeout which controls for how long a queue can be
# unused before it is automatically deleted. Unused means the queue has no
# consumers, the queue has not been redeclared, the `queue.get` has not been
# invoked for a duration of at least the expiration period. In our case this
# period is equal to the request timeout, once request is expired - queue is
# no longer needed.
QUEUE_EXPIRE_TIMEOUT = REQUEST_TIMEOUT
# Workers notify period.
NOTIFY_PERIOD = 5
# When a worker hasn't notified in this many seconds, it will get expired from
# being used/targeted for further work.
EXPIRES_AFTER = 60
# Message types.
NOTIFY = 'NOTIFY'
REQUEST = 'REQUEST'
RESPONSE = 'RESPONSE'
# Object that denotes nothing (none can actually be valid).
NO_RESULT = object()
LOG = logging.getLogger(__name__)
def make_an_event(new_state):
"""Turns a new/target state into an event name."""
return ('on_%s' % new_state).lower()
def build_a_machine(freeze=True):
"""Builds a state machine that requests are allowed to go through."""
m = machines.FiniteMachine()
for st in (WAITING, PENDING, RUNNING):
m.add_state(st)
for st in (SUCCESS, FAILURE):
m.add_state(st, terminal=True)
# When a executor starts to publish a request to a selected worker but the
# executor has not recved confirmation from that worker that anything has
# happened yet.
m.default_start_state = WAITING
m.add_transition(WAITING, PENDING, make_an_event(PENDING))
# When a request expires (isn't able to be processed by any worker).
m.add_transition(WAITING, FAILURE, make_an_event(FAILURE))
# Worker has started executing a request.
m.add_transition(PENDING, RUNNING, make_an_event(RUNNING))
# Worker failed to construct/process a request to run (either the worker
# did not transition to RUNNING in the given timeout or the worker itself
# had some type of failure before RUNNING started).
#
# Also used by the executor if the request was attempted to be published
# but that did publishing process did not work out.
m.add_transition(PENDING, FAILURE, make_an_event(FAILURE))
# Execution failed due to some type of remote failure.
m.add_transition(RUNNING, FAILURE, make_an_event(FAILURE))
# Execution succeeded & has completed.
m.add_transition(RUNNING, SUCCESS, make_an_event(SUCCESS))
# No further changes allowed.
if freeze:
m.freeze()
return m
def failure_to_dict(failure):
"""Attempts to convert a failure object into a jsonifyable dictionary."""
failure_dict = failure.to_dict()
try:
# it's possible the exc_args can't be serialized as JSON
# if that's the case, just get the failure without them
jsonutils.dumps(failure_dict)
return failure_dict
except (TypeError, ValueError):
return failure.to_dict(include_args=False)
@six.add_metaclass(abc.ABCMeta)
class Message(object):
"""Base class for all message types."""
def __repr__(self):
return ("<%s object at 0x%x with contents %s>"
% (reflection.get_class_name(self, fully_qualified=False),
id(self), self.to_dict()))
@abc.abstractmethod
def to_dict(self):
"""Return json-serializable message representation."""
class Notify(Message):
"""Represents notify message type."""
#: String constant representing this message type.
TYPE = NOTIFY
# NOTE(harlowja): the executor (the entity who initially requests a worker
# to send back a notification response) schema is different than the
# worker response schema (that's why there are two schemas here).
#: Expected notify *response* message schema (in json schema format).
RESPONSE_SCHEMA = {
"type": "object",
'properties': {
'topic': {
"type": "string",
},
'tasks': {
"type": "array",
"items": {
"type": "string",
},
}
},
"required": ["topic", 'tasks'],
"additionalProperties": False,
}
#: Expected *sender* request message schema (in json schema format).
SENDER_SCHEMA = {
"type": "object",
"additionalProperties": False,
}
def __init__(self, **data):
self._data = data
@property
def topic(self):
return self._data.get('topic')
@property
def tasks(self):
return self._data.get('tasks')
def to_dict(self):
return self._data
@classmethod
def validate(cls, data, response):
if response:
schema = cls.RESPONSE_SCHEMA
else:
schema = cls.SENDER_SCHEMA
try:
su.schema_validate(data, schema)
except su.ValidationError as e:
cls_name = reflection.get_class_name(cls, fully_qualified=False)
if response:
excp.raise_with_cause(excp.InvalidFormat,
"%s message response data not of the"
" expected format: %s" % (cls_name,
e.message),
cause=e)
else:
excp.raise_with_cause(excp.InvalidFormat,
"%s message sender data not of the"
" expected format: %s" % (cls_name,
e.message),
cause=e)
_WorkUnit = collections.namedtuple('_WorkUnit', ['task_cls', 'task_name',
'action', 'arguments'])
class Request(Message):
"""Represents request with execution results.
Every request is created in the WAITING state and is expired within the
given timeout if it does not transition out of the (WAITING, PENDING)
states.
State machine a request goes through as it progresses (or expires)::
+------------+------------+---------+----------+---------+
| Start | Event | End | On Enter | On Exit |
+------------+------------+---------+----------+---------+
| FAILURE[$] | . | . | . | . |
| PENDING | on_failure | FAILURE | . | . |
| PENDING | on_running | RUNNING | . | . |
| RUNNING | on_failure | FAILURE | . | . |
| RUNNING | on_success | SUCCESS | . | . |
| SUCCESS[$] | . | . | . | . |
| WAITING[^] | on_failure | FAILURE | . | . |
| WAITING[^] | on_pending | PENDING | . | . |
+------------+------------+---------+----------+---------+
"""
#: String constant representing this message type.
TYPE = REQUEST
#: Expected message schema (in json schema format).
SCHEMA = {
"type": "object",
'properties': {
# These two are typically only sent on revert actions (that is
# why are are not including them in the required section).
'result': {},
'failures': {
"type": "object",
},
'task_cls': {
'type': 'string',
},
'task_name': {
'type': 'string',
},
'task_version': {
"oneOf": [
{
"type": "string",
},
{
"type": "array",
},
],
},
'action': {
"type": "string",
"enum": list(six.iterkeys(ACTION_TO_EVENT)),
},
# Keyword arguments that end up in the revert() or execute()
# method of the remote task.
'arguments': {
"type": "object",
},
},
'required': ['task_cls', 'task_name', 'task_version', 'action'],
}
def __init__(self, task, uuid, action,
arguments, timeout=REQUEST_TIMEOUT, result=NO_RESULT,
failures=None):
self._action = action
self._event = ACTION_TO_EVENT[action]
self._arguments = arguments
self._result = result
self._failures = failures
self._watch = timeutils.StopWatch(duration=timeout).start()
self._lock = threading.Lock()
self._machine = build_a_machine()
self._machine.initialize()
self.task = task
self.uuid = uuid
self.created_on = timeutils.now()
self.future = futurist.Future()
self.future.atom = task
@property
def current_state(self):
"""Current state the request is in."""
return self._machine.current_state
def set_result(self, result):
"""Sets the responses futures result."""
self.future.set_result((self._event, result))
@property
def expired(self):
"""Check if request has expired.
When new request is created its state is set to the WAITING, creation
time is stored and timeout is given via constructor arguments.
Request is considered to be expired when it is in the WAITING/PENDING
state for more then the given timeout (it is not considered to be
expired in any other state).
"""
if self._machine.current_state in WAITING_STATES:
return self._watch.expired()
return False
def to_dict(self):
"""Return json-serializable request.
To convert requests that have failed due to some exception this will
convert all `failure.Failure` objects into dictionaries (which will
then be reconstituted by the receiver).
"""
request = {
'task_cls': reflection.get_class_name(self.task),
'task_name': self.task.name,
'task_version': self.task.version,
'action': self._action,
'arguments': self._arguments,
}
if self._result is not NO_RESULT:
result = self._result
if isinstance(result, ft.Failure):
request['result'] = ('failure', failure_to_dict(result))
else:
request['result'] = ('success', result)
if self._failures:
request['failures'] = {}
for atom_name, failure in six.iteritems(self._failures):
request['failures'][atom_name] = failure_to_dict(failure)
return request
def transition_and_log_error(self, new_state, logger=None):
"""Transitions *and* logs an error if that transitioning raises.
This overlays the transition function and performs nearly the same
functionality but instead of raising if the transition was not valid
it logs a warning to the provided logger and returns False to
indicate that the transition was not performed (note that this
is *different* from the transition function where False means
ignored).
"""
if logger is None:
logger = LOG
moved = False
try:
moved = self.transition(new_state)
except excp.InvalidState:
logger.warn("Failed to transition '%s' to %s state.", self,
new_state, exc_info=True)
return moved
@fasteners.locked
def transition(self, new_state):
"""Transitions the request to a new state.
If transition was performed, it returns True. If transition
was ignored, it returns False. If transition was not
valid (and will not be performed), it raises an InvalidState
exception.
"""
old_state = self._machine.current_state
if old_state == new_state:
return False
try:
self._machine.process_event(make_an_event(new_state))
except (machine_excp.NotFound, machine_excp.InvalidState) as e:
raise excp.InvalidState("Request transition from %s to %s is"
" not allowed: %s" % (old_state,
new_state, e))
else:
if new_state in STOP_TIMER_STATES:
self._watch.stop()
LOG.debug("Transitioned '%s' from %s state to %s state", self,
old_state, new_state)
return True
@classmethod
def validate(cls, data):
try:
su.schema_validate(data, cls.SCHEMA)
except su.ValidationError as e:
cls_name = reflection.get_class_name(cls, fully_qualified=False)
excp.raise_with_cause(excp.InvalidFormat,
"%s message response data not of the"
" expected format: %s" % (cls_name,
e.message),
cause=e)
else:
# Validate all failure dictionaries that *may* be present...
failures = []
if 'failures' in data:
failures.extend(six.itervalues(data['failures']))
result = data.get('result')
if result is not None:
result_data_type, result_data = result
if result_data_type == 'failure':
failures.append(result_data)
for fail_data in failures:
ft.Failure.validate(fail_data)
@staticmethod
def from_dict(data, task_uuid=None):
"""Parses **validated** data into a work unit.
All :py:class:`~taskflow.types.failure.Failure` objects that have been
converted to dict(s) on the remote side will now converted back
to py:class:`~taskflow.types.failure.Failure` objects.
"""
task_cls = data['task_cls']
task_name = data['task_name']
action = data['action']
arguments = data.get('arguments', {})
result = data.get('result')
failures = data.get('failures')
# These arguments will eventually be given to the task executor
# so they need to be in a format it will accept (and using keyword
# argument names that it accepts)...
arguments = {
'arguments': arguments,
}
if task_uuid is not None:
arguments['task_uuid'] = task_uuid
if result is not None:
result_data_type, result_data = result
if result_data_type == 'failure':
arguments['result'] = ft.Failure.from_dict(result_data)
else:
arguments['result'] = result_data
if failures is not None:
arguments['failures'] = {}
for task, fail_data in six.iteritems(failures):
arguments['failures'][task] = ft.Failure.from_dict(fail_data)
return _WorkUnit(task_cls, task_name, action, arguments)
class Response(Message):
"""Represents response message type."""
#: String constant representing this message type.
TYPE = RESPONSE
#: Expected message schema (in json schema format).
SCHEMA = {
"type": "object",
'properties': {
'state': {
"type": "string",
"enum": list(build_a_machine().states) + [EVENT],
},
'data': {
"anyOf": [
{
"$ref": "#/definitions/event",
},
{
"$ref": "#/definitions/completion",
},
{
"$ref": "#/definitions/empty",
},
],
},
},
"required": ["state", 'data'],
"additionalProperties": False,
"definitions": {
"event": {
"type": "object",
"properties": {
'event_type': {
'type': 'string',
},
'details': {
'type': 'object',
},
},
"required": ["event_type", 'details'],
"additionalProperties": False,
},
# Used when sending *only* request state changes (and no data is
# expected).
"empty": {
"type": "object",
"additionalProperties": False,
},
"completion": {
"type": "object",
"properties": {
# This can be any arbitrary type that a task returns, so
# thats why we can't be strict about what type it is since
# any of the json serializable types are allowed.
"result": {},
},
"required": ["result"],
"additionalProperties": False,
},
},
}
def __init__(self, state, **data):
self.state = state
self.data = data
@classmethod
def from_dict(cls, data):
state = data['state']
data = data['data']
if state == FAILURE and 'result' in data:
data['result'] = ft.Failure.from_dict(data['result'])
return cls(state, **data)
def to_dict(self):
return dict(state=self.state, data=self.data)
@classmethod
def validate(cls, data):
try:
su.schema_validate(data, cls.SCHEMA)
except su.ValidationError as e:
cls_name = reflection.get_class_name(cls, fully_qualified=False)
excp.raise_with_cause(excp.InvalidFormat,
"%s message response data not of the"
" expected format: %s" % (cls_name,
e.message),
cause=e)
else:
state = data['state']
if state == FAILURE and 'result' in data:
ft.Failure.validate(data['result'])

View File

@ -1,231 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2014 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import collections
import threading
import kombu
from kombu import exceptions as kombu_exceptions
import six
from taskflow.engines.worker_based import dispatcher
from taskflow import logging
LOG = logging.getLogger(__name__)
# NOTE(skudriashev): A timeout of 1 is often used in environments where
# the socket can get "stuck", and is a best practice for Kombu consumers.
DRAIN_EVENTS_PERIOD = 1
# Helper objects returned when requested to get connection details, used
# instead of returning the raw results from the kombu connection objects
# themselves so that a person can not mutate those objects (which would be
# bad).
_ConnectionDetails = collections.namedtuple('_ConnectionDetails',
['uri', 'transport'])
_TransportDetails = collections.namedtuple('_TransportDetails',
['options', 'driver_type',
'driver_name', 'driver_version'])
class Proxy(object):
"""A proxy processes messages from/to the named exchange.
For **internal** usage only (not for public consumption).
"""
DEFAULT_RETRY_OPTIONS = {
# The number of seconds we start sleeping for.
'interval_start': 1,
# How many seconds added to the interval for each retry.
'interval_step': 1,
# Maximum number of seconds to sleep between each retry.
'interval_max': 1,
# Maximum number of times to retry.
'max_retries': 3,
}
"""Settings used (by default) to reconnect under transient failures.
See: http://kombu.readthedocs.org/ (and connection ``ensure_options``) for
what these values imply/mean...
"""
# This is the only provided option that should be an int, the others
# are allowed to be floats; used when we check that the user-provided
# value is valid...
_RETRY_INT_OPTS = frozenset(['max_retries'])
def __init__(self, topic, exchange,
type_handlers=None, on_wait=None, url=None,
transport=None, transport_options=None,
retry_options=None):
self._topic = topic
self._exchange_name = exchange
self._on_wait = on_wait
self._running = threading.Event()
self._dispatcher = dispatcher.TypeDispatcher(
# NOTE(skudriashev): Process all incoming messages only if proxy is
# running, otherwise requeue them.
requeue_filters=[lambda data, message: not self.is_running],
type_handlers=type_handlers)
ensure_options = self.DEFAULT_RETRY_OPTIONS.copy()
if retry_options is not None:
# Override the defaults with any user provided values...
for k in set(six.iterkeys(ensure_options)):
if k in retry_options:
# Ensure that the right type is passed in...
val = retry_options[k]
if k in self._RETRY_INT_OPTS:
tmp_val = int(val)
else:
tmp_val = float(val)
if tmp_val < 0:
raise ValueError("Expected value greater or equal to"
" zero for 'retry_options' %s; got"
" %s instead" % (k, val))
ensure_options[k] = tmp_val
self._ensure_options = ensure_options
self._drain_events_timeout = DRAIN_EVENTS_PERIOD
if transport == 'memory' and transport_options:
polling_interval = transport_options.get('polling_interval')
if polling_interval is not None:
self._drain_events_timeout = polling_interval
# create connection
self._conn = kombu.Connection(url, transport=transport,
transport_options=transport_options)
# create exchange
self._exchange = kombu.Exchange(name=self._exchange_name,
durable=False, auto_delete=True)
@property
def dispatcher(self):
"""Dispatcher internally used to dispatch message(s) that match."""
return self._dispatcher
@property
def connection_details(self):
"""Details about the connection (read-only)."""
# The kombu drivers seem to use 'N/A' when they don't have a version...
driver_version = self._conn.transport.driver_version()
if driver_version and driver_version.lower() == 'n/a':
driver_version = None
if self._conn.transport_options:
transport_options = self._conn.transport_options.copy()
else:
transport_options = {}
transport = _TransportDetails(
options=transport_options,
driver_type=self._conn.transport.driver_type,
driver_name=self._conn.transport.driver_name,
driver_version=driver_version)
return _ConnectionDetails(
uri=self._conn.as_uri(include_password=False),
transport=transport)
@property
def is_running(self):
"""Return whether the proxy is running."""
return self._running.is_set()
def _make_queue(self, routing_key, exchange, channel=None):
"""Make a named queue for the given exchange."""
queue_name = "%s_%s" % (self._exchange_name, routing_key)
return kombu.Queue(name=queue_name,
routing_key=routing_key, durable=False,
exchange=exchange, auto_delete=True,
channel=channel)
def publish(self, msg, routing_key, reply_to=None, correlation_id=None):
"""Publish message to the named exchange with given routing key."""
if isinstance(routing_key, six.string_types):
routing_keys = [routing_key]
else:
routing_keys = routing_key
# Filter out any empty keys...
routing_keys = [r_k for r_k in routing_keys if r_k]
if not routing_keys:
LOG.warning("No routing key/s specified; unable to send '%s'"
" to any target queue on exchange '%s'", msg,
self._exchange_name)
return
def _publish(producer, routing_key):
queue = self._make_queue(routing_key, self._exchange)
producer.publish(body=msg.to_dict(),
routing_key=routing_key,
exchange=self._exchange,
declare=[queue],
type=msg.TYPE,
reply_to=reply_to,
correlation_id=correlation_id)
def _publish_errback(exc, interval):
LOG.exception('Publishing error: %s', exc)
LOG.info('Retry triggering in %s seconds', interval)
LOG.debug("Sending '%s' message using routing keys %s",
msg, routing_keys)
with kombu.connections[self._conn].acquire(block=True) as conn:
with conn.Producer() as producer:
ensure_kwargs = self._ensure_options.copy()
ensure_kwargs['errback'] = _publish_errback
safe_publish = conn.ensure(producer, _publish, **ensure_kwargs)
for routing_key in routing_keys:
safe_publish(producer, routing_key)
def start(self):
"""Start proxy."""
def _drain(conn, timeout):
try:
conn.drain_events(timeout=timeout)
except kombu_exceptions.TimeoutError:
pass
def _drain_errback(exc, interval):
LOG.exception('Draining error: %s', exc)
LOG.info('Retry triggering in %s seconds', interval)
LOG.info("Starting to consume from the '%s' exchange.",
self._exchange_name)
with kombu.connections[self._conn].acquire(block=True) as conn:
queue = self._make_queue(self._topic, self._exchange, channel=conn)
callbacks = [self._dispatcher.on_message]
with conn.Consumer(queues=queue, callbacks=callbacks) as consumer:
ensure_kwargs = self._ensure_options.copy()
ensure_kwargs['errback'] = _drain_errback
safe_drain = conn.ensure(consumer, _drain, **ensure_kwargs)
self._running.set()
try:
while self._running.is_set():
safe_drain(conn, self._drain_events_timeout)
if self._on_wait is not None:
self._on_wait()
finally:
self._running.clear()
def wait(self):
"""Wait until proxy is started."""
self._running.wait()
def stop(self):
"""Stop proxy."""
self._running.clear()

View File

@ -1,263 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2014 Yahoo! Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import functools
from oslo_utils import reflection
from oslo_utils import timeutils
from taskflow.engines.worker_based import dispatcher
from taskflow.engines.worker_based import protocol as pr
from taskflow.engines.worker_based import proxy
from taskflow import logging
from taskflow.types import failure as ft
from taskflow.types import notifier as nt
from taskflow.utils import kombu_utils as ku
from taskflow.utils import misc
LOG = logging.getLogger(__name__)
class Server(object):
"""Server implementation that waits for incoming tasks requests."""
def __init__(self, topic, exchange, executor, endpoints,
url=None, transport=None, transport_options=None,
retry_options=None):
type_handlers = {
pr.NOTIFY: dispatcher.Handler(
self._delayed_process(self._process_notify),
validator=functools.partial(pr.Notify.validate,
response=False)),
pr.REQUEST: dispatcher.Handler(
self._delayed_process(self._process_request),
validator=pr.Request.validate),
}
self._executor = executor
self._proxy = proxy.Proxy(topic, exchange,
type_handlers=type_handlers,
url=url, transport=transport,
transport_options=transport_options,
retry_options=retry_options)
self._topic = topic
self._endpoints = dict([(endpoint.name, endpoint)
for endpoint in endpoints])
def _delayed_process(self, func):
"""Runs the function using the instances executor (eventually).
This adds a *nice* benefit on showing how long it took for the
function to finally be executed from when the message was received
to when it was finally ran (which can be a nice thing to know
to determine bottle-necks...).
"""
func_name = reflection.get_callable_name(func)
def _on_run(watch, content, message):
LOG.trace("It took %s seconds to get around to running"
" function/method '%s' with"
" message '%s'", watch.elapsed(), func_name,
ku.DelayedPretty(message))
return func(content, message)
def _on_receive(content, message):
LOG.debug("Submitting message '%s' for execution in the"
" future to '%s'", ku.DelayedPretty(message), func_name)
watch = timeutils.StopWatch()
watch.start()
try:
self._executor.submit(_on_run, watch, content, message)
except RuntimeError:
LOG.error("Unable to continue processing message '%s',"
" submission to instance executor (with later"
" execution by '%s') was unsuccessful",
ku.DelayedPretty(message), func_name,
exc_info=True)
return _on_receive
@property
def connection_details(self):
return self._proxy.connection_details
@staticmethod
def _parse_message(message):
"""Extracts required attributes out of the messages properties.
This extracts the `reply_to` and the `correlation_id` properties. If
any of these required properties are missing a `ValueError` is raised.
"""
properties = []
for prop in ('reply_to', 'correlation_id'):
try:
properties.append(message.properties[prop])
except KeyError:
raise ValueError("The '%s' message property is missing" %
prop)
return properties
def _reply(self, capture, reply_to, task_uuid, state=pr.FAILURE, **kwargs):
"""Send a reply to the `reply_to` queue with the given information.
Can capture failures to publish and if capturing will log associated
critical errors on behalf of the caller, and then returns whether the
publish worked out or did not.
"""
response = pr.Response(state, **kwargs)
published = False
try:
self._proxy.publish(response, reply_to, correlation_id=task_uuid)
published = True
except Exception:
if not capture:
raise
LOG.critical("Failed to send reply to '%s' for task '%s' with"
" response %s", reply_to, task_uuid, response,
exc_info=True)
return published
def _on_event(self, reply_to, task_uuid, event_type, details):
"""Send out a task event notification."""
# NOTE(harlowja): the executor that will trigger this using the
# task notification/listener mechanism will handle logging if this
# fails, so thats why capture is 'False' is used here.
self._reply(False, reply_to, task_uuid, pr.EVENT,
event_type=event_type, details=details)
def _process_notify(self, notify, message):
"""Process notify message and reply back."""
try:
reply_to = message.properties['reply_to']
except KeyError:
LOG.warning("The 'reply_to' message property is missing"
" in received notify message '%s'",
ku.DelayedPretty(message), exc_info=True)
else:
response = pr.Notify(topic=self._topic,
tasks=list(self._endpoints.keys()))
try:
self._proxy.publish(response, routing_key=reply_to)
except Exception:
LOG.critical("Failed to send reply to '%s' with notify"
" response '%s'", reply_to, response,
exc_info=True)
def _process_request(self, request, message):
"""Process request message and reply back."""
try:
# NOTE(skudriashev): parse broker message first to get
# the `reply_to` and the `task_uuid` parameters to have
# possibility to reply back (if we can't parse, we can't respond
# in the first place...).
reply_to, task_uuid = self._parse_message(message)
except ValueError:
LOG.warn("Failed to parse request attributes from message '%s'",
ku.DelayedPretty(message), exc_info=True)
return
else:
# prepare reply callback
reply_callback = functools.partial(self._reply, True, reply_to,
task_uuid)
# Parse the request to get the activity/work to perform.
try:
work = pr.Request.from_dict(request, task_uuid=task_uuid)
except ValueError:
with misc.capture_failure() as failure:
LOG.warning("Failed to parse request contents"
" from message '%s'",
ku.DelayedPretty(message), exc_info=True)
reply_callback(result=pr.failure_to_dict(failure))
return
# Now fetch the task endpoint (and action handler on it).
try:
endpoint = self._endpoints[work.task_cls]
except KeyError:
with misc.capture_failure() as failure:
LOG.warning("The '%s' task endpoint does not exist, unable"
" to continue processing request message '%s'",
work.task_cls, ku.DelayedPretty(message),
exc_info=True)
reply_callback(result=pr.failure_to_dict(failure))
return
else:
try:
handler = getattr(endpoint, work.action)
except AttributeError:
with misc.capture_failure() as failure:
LOG.warning("The '%s' handler does not exist on task"
" endpoint '%s', unable to continue processing"
" request message '%s'", work.action, endpoint,
ku.DelayedPretty(message), exc_info=True)
reply_callback(result=pr.failure_to_dict(failure))
return
else:
try:
task = endpoint.generate(name=work.task_name)
except Exception:
with misc.capture_failure() as failure:
LOG.warning("The '%s' task '%s' generation for request"
" message '%s' failed", endpoint,
work.action, ku.DelayedPretty(message),
exc_info=True)
reply_callback(result=pr.failure_to_dict(failure))
return
else:
if not reply_callback(state=pr.RUNNING):
return
# Associate *any* events this task emits with a proxy that will
# emit them back to the engine... for handling at the engine side
# of things...
if task.notifier.can_be_registered(nt.Notifier.ANY):
task.notifier.register(nt.Notifier.ANY,
functools.partial(self._on_event,
reply_to, task_uuid))
elif isinstance(task.notifier, nt.RestrictedNotifier):
# Only proxy the allowable events then...
for event_type in task.notifier.events_iter():
task.notifier.register(event_type,
functools.partial(self._on_event,
reply_to, task_uuid))
# Perform the task action.
try:
result = handler(task, **work.arguments)
except Exception:
with misc.capture_failure() as failure:
LOG.warning("The '%s' endpoint '%s' execution for request"
" message '%s' failed", endpoint, work.action,
ku.DelayedPretty(message), exc_info=True)
reply_callback(result=pr.failure_to_dict(failure))
else:
# And be done with it!
if isinstance(result, ft.Failure):
reply_callback(result=result.to_dict())
else:
reply_callback(state=pr.SUCCESS, result=result)
def start(self):
"""Start processing incoming requests."""
self._proxy.start()
def wait(self):
"""Wait until server is started."""
self._proxy.wait()
def stop(self):
"""Stop processing incoming requests."""
self._proxy.stop()

Some files were not shown because too many files have changed in this diff Show More