2b094fb02c
Add a taskflow executor to be able to support more complex execution models. Change-Id: I9fe96a5e39059fbe571cc39021bacd6993c50f25
151 lines
3.7 KiB
ReStructuredText
151 lines
3.7 KiB
ReStructuredText
..
|
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
|
License.
|
|
|
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
|
|
|
====================
|
|
Taskflow Integration
|
|
====================
|
|
|
|
https://blueprints.launchpad.net/glance/+spec/taskflow-integration
|
|
|
|
Add a new task executor using the taskflow library.
|
|
|
|
"TaskFlow is a Python library for OpenStack (and other projects) that helps make
|
|
task execution easy, consistent, scalable and reliable. It allows the creation
|
|
of lightweight task objects and/or functions that are combined together into
|
|
flows (aka: workflows) in a declarative manner.": `Taskflow Wiki <https://wiki.openstack.org/wiki/TaskFlow>`_
|
|
|
|
Problem description
|
|
===================
|
|
|
|
Glance currently comes with an eventlet executor which is not easily extensible
|
|
by nature and doesn't support many features that taskflow support out of the
|
|
box such as execution on remote workers.
|
|
|
|
Proposed change
|
|
===============
|
|
|
|
We propose here a new executor (self-contained) implementing the interfaces of the
|
|
executor API. This executor will route the tasks to the eventlet executor of taskflow.
|
|
We will be using the `Taskflow Green Thread Pool Executor <http://docs.openstack.org/developer/taskflow/types.html#taskflow.types.futures.GreenThreadPoolExecutor>`_ which ensures that eventlet green threads are used when
|
|
using the taskflow engine.
|
|
The initial implementation should provide the same result as the eventlet executor
|
|
already contained in Glance. However, subsequent blueprints will come to leverage
|
|
more advanced functionalities.
|
|
|
|
Alternatives
|
|
------------
|
|
|
|
Use the existing eventlet executor. This approach is likely to become rewriting
|
|
taskflow.
|
|
|
|
Data model impact
|
|
-----------------
|
|
|
|
None.
|
|
|
|
REST API impact
|
|
---------------
|
|
|
|
None.
|
|
|
|
Security impact
|
|
---------------
|
|
|
|
None.
|
|
|
|
Notifications impact
|
|
--------------------
|
|
|
|
None with this spec. However, in the future, it will be possible to plug-in to
|
|
the taskflow notification engine and potentially drop its messages onto a
|
|
notification bus.
|
|
|
|
Other end user impact
|
|
---------------------
|
|
|
|
The end user should be able to transparently execute tasks with all the
|
|
executors.
|
|
|
|
Performance Impact
|
|
------------------
|
|
|
|
For serial execution of tasks, the performance of the evenlet executor and
|
|
the taskflow executor should be close to similar.
|
|
However, for more complex workflows, we should be able to achieve performance
|
|
improvements by parallelizing the work, and also distributing it with taskflow.
|
|
|
|
Other deployer impact
|
|
---------------------
|
|
|
|
The deployer will have to update glance-api.conf and specify 'taskflow' as the
|
|
executor.
|
|
Also, it will be possible to choose the engine mode 'serial' or 'parallel' and
|
|
the maximum number of workers.
|
|
Remote workers (not supported with this spec) will require more infrastructure.
|
|
|
|
Developer impact
|
|
----------------
|
|
|
|
None.
|
|
|
|
Implementation
|
|
==============
|
|
|
|
Assignee(s)
|
|
-----------
|
|
|
|
Primary assignee:
|
|
arnaud
|
|
|
|
Other contributors:
|
|
harlowja
|
|
|
|
Reviewers
|
|
---------
|
|
|
|
Core reviewer(s):
|
|
nikhil, zhiyan
|
|
|
|
Other reviewer(s):
|
|
harlowja
|
|
|
|
Work Items
|
|
----------
|
|
|
|
Implementation of the taskflow executor with unit tests.
|
|
|
|
Dependencies
|
|
============
|
|
|
|
None.
|
|
|
|
Testing
|
|
=======
|
|
|
|
It will be possible to add a specifc configuration in DevStack leverage this
|
|
new executor. Ultimately, this is the executor that should be used at the gate.
|
|
|
|
Documentation Impact
|
|
====================
|
|
|
|
Initially, the documentation should explain how to configure glance-api.conf and
|
|
what is taskflow. Later on, it should be explained how to achieve more complex
|
|
scenario.
|
|
|
|
References
|
|
==========
|
|
|
|
* https://wiki.openstack.org/wiki/TaskFlow
|
|
|
|
* https://github.com/openstack/taskflow
|
|
|
|
* https://pypi.python.org/pypi/taskflow/
|
|
|
|
* https://review.openstack.org/#/c/85211/14 (needs to be rebased once the spec
|
|
is approved)
|
|
|
|
* Discussions at the OpenStack Summit in Paris
|