Add a new task executor using the taskflow library.
"TaskFlow is a Python library for OpenStack (and other projects) that helps make task execution easy, consistent, scalable and reliable. It allows the creation of lightweight task objects and/or functions that are combined together into flows (aka: workflows) in a declarative manner.": Taskflow Wiki
Glance currently comes with an eventlet executor which is not easily extensible by nature and doesn't support many features that taskflow support out of the box such as execution on remote workers.
We propose here a new executor (self-contained) implementing the interfaces of the executor API. This executor will route the tasks to the eventlet executor of taskflow. We will be using the Taskflow Green Thread Pool Executor which ensures that eventlet green threads are used when using the taskflow engine. The initial implementation should provide the same result as the eventlet executor already contained in Glance. However, subsequent blueprints will come to leverage more advanced functionalities.
Use the existing eventlet executor. This approach is likely to become rewriting taskflow.
Data model impact
REST API impact
None with this spec. However, in the future, it will be possible to plug-in to the taskflow notification engine and potentially drop its messages onto a notification bus.
Other end user impact
The end user should be able to transparently execute tasks with all the executors.
For serial execution of tasks, the performance of the evenlet executor and the taskflow executor should be close to similar. However, for more complex workflows, we should be able to achieve performance improvements by parallelizing the work, and also distributing it with taskflow.
Other deployer impact
The deployer will have to update glance-api.conf and specify 'taskflow' as the executor. Also, it will be possible to choose the engine mode 'serial' or 'parallel' and the maximum number of workers. Remote workers (not supported with this spec) will require more infrastructure.
- Primary assignee:
- Other contributors:
- Core reviewer(s):
- Other reviewer(s):
Implementation of the taskflow executor with unit tests.
It will be possible to add a specifc configuration in DevStack leverage this new executor. Ultimately, this is the executor that should be used at the gate.
Initially, the documentation should explain how to configure glance-api.conf and what is taskflow. Later on, it should be explained how to achieve more complex scenario.