openstack
/
oslo-specs
Code Issues Proposed changes

Remove specs not completed in juno 

These specs were not completed in juno and can be resubmitted for kilo.

Change-Id: I8c481d50ad0a00567361d59c9fb78d9513163ebb
This commit is contained in:
Doug Hellmann 2014-09-15 13:00:44 -04:00
parent ef70f6c522
commit 79518fba6e
6 changed files with 0 additions and 1478 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

179
specs/juno/app-agnostic-logging-parameters.rst
View File

@ -1,179 +0,0 @@
===================================================================
Provide application-agnostic logging parameters in format strings
===================================================================
https://blueprints.launchpad.net/oslo/+spec/app-agnostic-logging-parameters
We still have some nova-specific names in the default logging format
string, and we want to eliminate those to make oslo.log more generally
useful across projects.
Problem description
===================
The default ``logging_context_format_string`` and
``logging_default_format_string`` option values include
``%(instance)s``, which is not useful in all projects. We should do
something like what we did with "user_identity" and provide a generic
name, which the projects can fill in with their desired value.
Proposed change
===============
Make :class:`RequestContext` from :mod:`openstack.common.context` into
an abstract base class for other applications to use for their own
application-specific request contexts. In the new base class, define
some abstract properties with generic names like ``user_identity``,
``resource_id``, etc. that the subclass can override.
Change the default for the logging format strings to refer to these
generic names.
Add a new method to the base class to return values useful for
logging. We cannot use the existing :meth:`to_dict` because we expect
the logging values to contain generated properties not used for things
like reconstructing the context when it passes through RPC calls.
::
def get_logging_values(self):
"""Return a dict containing values for logging using this context.
"""
values = self.to_dict()
values.update({
'user_identity': self.user_identity,
'resource_id': self.resource_id,
})
return values
Remove the other functions in the :mod:`context` module for creating
and testing contexts. The applications all have their own version of
these functions and the versions provided in :mod:`context` are not
useful when subclasses of :class:`RequestContext` are used.
Update the logging code to use :meth:`get_logging_values` instead of
:meth:`to_dict`.
Alternatives
------------
We had previously talked about removing this module entirely, but
given changes to logging to make the user identity parameters log
consistently across projects, I think making it a useful base class is
a better approach.
Impact on Existing APIs
-----------------------
Existing context classes will be updated to be subclasses of the base
class, which may allow us to save some repeated code in the
constructor.
Security impact
---------------
When we talk about logging and contexts together we typically worry
about exposing secret details. I don't think any of these proposed
changes expose any information beyond what we are exposing already.
Performance Impact
------------------
Possibly a slight impact creating :class:`RequestContext` instances in
the application. If an app sees that as a problem, they could opt to
simply copy the base class API into their local class rather than
using a subclass, but it would be up to them to keep up with API
changes at that point. I don't think this is a significant performance
issue.
Configuration Impact
--------------------
The defaults for the configuration options might change, but the
*output* should be the same and the old values will still work as well
as they did before.
Developer Impact
----------------
The idea is for the other projects to define their context as a
subclass of the :class:`RequestContext` in Oslo, implementing or
overriding private methods or properties in order to meet the API
needed by the logging module.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Doug Hellmann (doug-hellmann)
Other contributors:
None
Milestones
----------
Target Milestone for completion: Juno-1
Work Items
----------
1. Remove unused functions from :mod:`context`.
2. Add new :meth:`get_logging_values` to :class:`RequestContext`.
3. Add abstract properties to :class:`RequestContext`.
4. Update default format strings in :mod:`log`.
5. Update :mod:`log` to use :meth:`get_logging_values`.
Incubation
==========
N/A
Adoption
--------
N/A
Library
-------
N/A
Anticipated API Stabilization
-----------------------------
I expect :meth:`get_logging_values` to be stable.
We may add more generated properties to :class:`RequestContext` over
time, but we will have to add those as normal properties (not
abstract) to provide backwards compatibility.
Documentation Impact
====================
The defaults for the config options will need to be updated in any
documentation generated from the option definitions.
Dependencies
============
None
References
==========
* Discussion at the juno summit:
https://etherpad.openstack.org/p/juno-oslo-release-plan
* Mailing list thread that mentions the domain support in Oslo's
context as a potential issue for nova:
http://lists.openstack.org/pipermail/openstack-dev/2014-February/027634.html
.. note::
This work is licensed under a Creative Commons Attribution 3.0
Unported License.
http://creativecommons.org/licenses/by/3.0/legalcode

195
specs/juno/enable-mysql-connector.rst
View File

@ -1,195 +0,0 @@
=========================================
Enable MySQL Connector driver (phase 1)
=========================================
https://blueprints.launchpad.net/oslo/+spec/enable-mysql-connector
Switch from MySQLdb to MySQL Connector [#pypi]_ which provides multiple
benefits, including resilience of code and potential performance advantages.
Note: this phase includes adding optional support to run Devstack against MySQL
Connector and is not meant to change the default recommended database driver
for running against MySQL. Choice of default driver will be revisited the next
cycle.
Problem description
===================
The current MySQL client library we use (MySQLdb) plays badly with eventlet and
may result in db deadlocks (see [#dead_locks]_). It also blocks execution of
other green threads while we're deep in the library code (see [#one_thread]_).
To avoid those issues, we need to be able to switch to a pure python library
with better eventlet support. MySQL Connector [#official_page]_ is an official
MySQL Python client library and has the needed qualities.
Proposed change
===============
Oslo.db already supports using other MySQL drivers [#dialects]_. Oslo.db uses
SQLAlchemy which supports switching client libraries by modifying the SQL
connection URI that we pass to a service as a configuration option.
We can specify the driver to use when connecting to MySQL as follows:
connection = mysql+mysqlconnector://...
So most work is around oslo.db, devstack, and documentation. (For details, see
'Work Items' section below.)
Alternatives
------------
Some database lock timeouts can be mitigated by refactoring code
[#hack_around_notifications]_. We may also introduce special mechanisms that
would reduce the chances for the current thread to yield under transaction that
are one of the main causes of db deadlocks. It is not realistic to expect that
we're able to nail down and fix all occurances of possible yields under
transactions.
We could also switch to some other library that is also supported by SQLAlchemy
[#dialects]_. Though those other libraries do not provide clear benefits in
comparison to MySQL Connector, while the latter is the official driver for the
MySQL project.
We could also introduce a per-kernel-thread (or global) Python/eventlet lock
around any calls into MySQLdb that may block. Not a particularly attractive
option, because the impact on parallel operations is expected to be enormous.
Finally, we could just drop eventlet from all our projects, though this option
is not realistic.
Impact on Existing APIs
-----------------------
None.
Security impact
---------------
None.
Performance Impact
------------------
Though MySQL Connector is a pure Python library, while MySQLdb is largely
written in C, and we could expect that the new module is a bit slower than the
current one, performance may actually be improved. This is because the new
module is eventlet aware, meaning threads will be able to switch while waiting
for I/O from a database server.
This expectation is proved by the following measurements [#measurements]_.
A simple performance test for Neutron was run locally on devstack to check
impact of the change. Four scenarios were checked:
* create 200 ports in serial
* create 200 ports in parallel using 'futures' module with 10 thread workers
* create 2000 networks in serial
* create 2000 networks in parallel using 'futures' module with 10 thread workers
Each scenario was run three times.
For serial scenarios, the average result is pretty the same:
* mysqldb (ports): 59.43 sec
* mysql connector (ports): 62.81 sec
* mysqldb (networks): 211.94 sec
* mysql connector (networks): 206.80 sec
For parallel scenarios, there is a very significant benefit:
* mysqldb (ports): 58.37 sec
* mysql connector (ports): 35.32 sec
* mysqldb (networks): 215.81 sec
* mysql connector (networks): 88.66 sec
More detailed benchmarking and scalability testing will be done in the second
phase which belongs to Kilo cycle. Details on what constitutes a proper
benchmark for global switch are to be decided during that cycle.
Configuration Impact
--------------------
From configuration point of view, this is just a matter of changing connection
string for each service. Old values should continue to work, so migration to
the new library can be delayed or done iteratively. Also, we don't enforce or
even recommend operators to switch their database driver of choice, so unless
they explicitly opt-in, there is no change for them whatsoever.
Developer Impact
----------------
More strict rules will be applied for migration code to facilitate using
multiple database drivers from the same code (like not passing multiple SQL
statements in single call to engine.execute()).
Implementation
==============
Assignee(s)
-----------
Primary assignee:
ihar-hrachyshka
Secondary assignee:
gus (Angus Lees)
Milestones
----------
Juno-3
Work Items
----------
The following items should be handled before claiming the switch to MySQL
Connector to be completed:
* oslo.db requires a few small changes for testing and feature parity with
non-default MySQL drivers (development tracked by Angus Lees).
* update devstack to enable optional support for running it against MySQL
Connector.
* create a separate gate check to run tempest against the new driver.
Incubation
==========
None.
Documentation Impact
====================
We won't recommend users to switch database module in Juno. We may still be
interested in notifying them that there is now an option to run their
deployments against the new driver, which has its own benefits though.
Dependencies
============
MySQL Connector must be posted to PyPI to be able to introduce it as part of
global requirements list.
MySQL Connector is published under the terms of the same license as for MySQLdb
(GPLv2), so there should be no legal issues with using that. Also, MySQL
Connector provides FOSS exception for a vast number of open source licenses
[#exceptions]_, so it can be considered as more liberal than MySQLdb.
References
==========
.. [#pypi] https://pypi.python.org/pypi/mysql-connector-python
.. [#dead_locks] https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad
.. [#one_thread] http://docs.openstack.org/developer/nova/devref/threading.html
.. [#official_page] http://www.mysql.com/products/connector/
.. [#dialects] http://docs.sqlalchemy.org/en/rel_0_9/dialects/
.. [#hack_around_notifications] https://review.openstack.org/100934/
.. [#measurements] http://www.diamondtin.com/2014/sqlalchemy-gevent-mysql-python-drivers-comparison/
.. [#exceptions] http://www.mysql.com/about/legal/licensing/foss-exception/
.. note::
This work is licensed under a Creative Commons Attribution 3.0
Unported License.
http://creativecommons.org/licenses/by/3.0/legalcode

258
specs/juno/oslo-messaging-greenio-executor.rst
View File

@ -1,258 +0,0 @@
===============================
oslo.messaging greenio executor
===============================
https://blueprints.launchpad.net/oslo.messaging/+spec/greenio-executor
Add a new oslo.messaging executor which adds a new capability to the
eventlet executor - the ability to dispatch requests to asyncio
coroutines running on greenio's greenlet based asyncio event loop.
Problem description
===================
We're attempting to take baby-steps towards moving completely from
eventlet to asyncio/trollius.
Any OpenStack service code is run in response to various I/O events like
REST API requests, RPC calls, notifications received, periodic timers,
etc. We eventually want the asyncio event loop to be what schedules
code to run in response to these events. Right now, it is eventlet doing
that.
Now, because we're using eventlet, the code that is run in response to
these events looks like synchronous code that makes a bunch of
synchronous calls. For example, the code might do some_sync_op() and
that will cause a context switch to a different greenthread (within the
same native thread) where we might handle another I/O event (like a REST
API request) while we're waiting for some_sync_op() to return::
def foo(self):
result = some_sync_op() # this may yield to another greenlet
return do_stuff(result)
Eventlet's infamous monkey patching is what make this magic happen.
When we switch to asyncio's event loop, all of this code needs to be
ported to asyncio's explicitly asynchronous approach. We might do::
@asyncio.coroutine
def foo(self):
result = yield from some_async_op(...)
return do_stuff(result)
or::
@asyncio.coroutine
def foo(self):
fut = Future()
some_async_op(callback=fut.set_result)
...
result = yield from fut
return do_stuff(result)
Porting from eventlet's implicit async approach to asyncio's explicit
async API will be seriously time consuming and we need to be able to do
it piece-by-piece.
The problem this spec addresses is how to allow a single oslo.messaging
RPC endpoint method to be ported to asyncio's explicit async approach.
Proposed change
===============
The plan is:
#. Stick with eventlet; everything gets monkey patched as normal.
#. We register the greenio event loop with asyncio - this means that
e.g. when you schedule an asyncio coroutine, greenio runs it in a
greenlet using eventlet's event loop.
#. oslo.messaging will need a new variant of eventlet executor which
knows how to dispatch an asyncio coroutine. It's important that even
with a coroutine endpoint method, we send the reply from a
greenthread so that the dispatch greenthread doesn't get blocked if
the incoming.reply() call causes a greenlet context switch.
#. When all of ceilometer has been ported over to asyncio coroutines,
we can stop monkey patching, stop using greenio and switch to the
asyncio event loop
#. When we make this change, we'll want a completely native asyncio
oslo.messaging executor. Unless the oslo.messaging drivers support
asyncio themselves, that executor will probably need a separate
native thread to poll for messages and send replies.
This spec specifically proposes the addition of a 'greenio' executor
which would do something roughly like this::
while True:
incoming = self.listener.poll()
method = dispatcher.get_endpoint_method(incoming)
if asyncio.iscoroutinefunc(method):
fut = asyncio.Task(method())
fut.add_done_callback(lambda fut: incoming.reply(fut.result()))
else:
self._greenpool.spawn_n(method)
If the endpoint method is just a normal python function, we dispatch
it in a new greenthread just like the current eventlet executor does.
If the endpoint method is an asyncio coroutine, we schedule that through
the greenio event loop which means the coroutine runs in a greenlet.
When the coroutine completes, a callback is invoked in another greenlet
to send the reply back to the client.
Alternatives
------------
The alternative is to add an asyncio executor which can only schedule asyncio
coroutines to the asyncio eventloop. This would mean all code in a service
would need to be ported and eventlet replaced with asyncio in one atomic
change. That would be bonkers.
Impact on Existing APIs
-----------------------
Two changes to the public oslo.messaging API:
#. The get_rpc_server() and get_notification_listener() will now accept
executor='greenio'.
#. RPC server or notification listener endpoint methods can now be asyncio
coroutines (i.e. methods annotated with @async.coroutine that use
constructs like 'yield from' or asyncio.Task()).
Security impact
---------------
None.
Performance Impact
------------------
In theory, there could be differences in the performance of endpoint methods
ported to be asyncio coroutines but we have no idea yet.
Configuration Impact
--------------------
None.
Developer Impact
----------------
Developers will need to understand the difference between implicitly async
code relying on eventlet's monkey patching versus explicitly async code using
asyncio constructs. This will be a far-ranging change for OpenStack, but this
blueprint is only about one small feature in oslo.messaging which will enable
this work to begin.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
victor-stinner
Other contributors:
markmc
flaper87
Milestones
----------
Target Milestone for completion:
juno-3
Work Items
----------
#. greenio needs to be added to openstack/requirements. See `Review Criteria`_
for the requirements repo. In particular, the availability of greenio in
distros and greenio's commitment to API stability are worth considering.
#. Add a greenio executor.
#. Include examples of asyncio coroutine endpoint methods in the docs.
#. Include greenio based unit tests which dispatch both types of endpoint
methods with both RPC servers and notification listeners. Basing these tests
the rabbit and/or fake drivers probably makes sense.
.. _Review Criteria: https://wiki.openstack.org/wiki/Requirements#Review_Criteria
Incubation
==========
Adoption
--------
Ceilometer is likely to be the first service to use this executor.
Library
-------
oslo.messaging.
Anticipated API Stabilization
-----------------------------
The API should be stable from introduction.
Documentation Impact
====================
oslo.messaging developer docs needs some small additions.
Dependencies
============
- This introduces a dependency on trollius and greenio.
References
==========
Victor's excellent docs on asyncio and trollius:
https://docs.python.org/3/library/asyncio.html
http://trollius.readthedocs.org/
Victor's proposed asyncio executor:
https://review.openstack.org/70948
The case for adopting asyncio in OpenStack:
https://wiki.openstack.org/wiki/Oslo/blueprints/asyncio
Victor's current status on trollius in OpenStack:
http://haypo-notes.readthedocs.org/openstack.html#trollius-in-openstack
A blog post on the subject from Victor:
http://techs.enovance.com/6562/asyncio-openstack-python3
Summary of the discussion at the Paris Juno Sprint which lead to this design:
http://lists.openstack.org/pipermail/openstack-dev/2014-July/039291.html
A previous email I wrote about an asyncio executor:
http://lists.openstack.org/pipermail/openstack-dev/2013-June/009934.html
The mock-up of an asyncio executor Mark wrote (and never tested):
https://github.com/markmc/oslo-incubator/blob/8509b8b/openstack/common/messaging/_executors/impl_tulip.py
Mark's blog post on async I/O and Python:
http://blogs.gnome.org/markmc/2013/06/04/async-io-and-python/
greenio - greelets support for asyncio:
https://github.com/1st1/greenio/
.. note::
This work is licensed under a Creative Commons Attribution 3.0
Unported License.
http://creativecommons.org/licenses/by/3.0/legalcode

145
specs/juno/remove-context-adapter.rst
View File

@ -1,145 +0,0 @@
====================================
Remove ContextAdapter from logging
====================================
Include the URL of your launchpad blueprint:
https://blueprints.launchpad.net/oslo/+spec/remove-context-adapter
We want to remove the :class:`openstack.common.log.ContextAdapter`
class as part of graduating ``oslo.log``, to reduce the API footprint.
Problem description
===================
We use :class:`ContextAdapter` to add request context information to
log messages we output. Requiring a specially adapted logging handle
limits the scope of where that context information can be added, and
unnecessarily funnels all logging calls through the oslo logging
modules.
Proposed change
===============
1. Ensure :class:`ContextHandler` implements all of the same behaviors
as :class:`ContextAdapter`, with respect to the values being
output and their sources.
2. Update :func:`getLogger` to return a :class:`BaseLoggerAdapter`
(probably renamed to something like :class:`LoggerAdapter` in the
process). This will allow modules that use the :meth:`audit`,
:meth:`warn`, and :meth:`deprecated` methods to still work until we
can update the callers.
3. Remove the :class:`ContextAdapter` class.
4. Remove the :func:`getLazyLogger` function.
Alternatives
------------
None
Impact on Existing APIs
-----------------------
By changing :func:`getLogger` to return a different adapter type, we
can keep the API as it is for now to buy time to update the callers
that use the methods that would otherwise be removed. Modules that do
not use those methods can switch to using the Python standard library
:mod:`logging` module to get a logger.
Security impact
---------------
None
Performance Impact
------------------
None
Configuration Impact
--------------------
None
Developer Impact
----------------
See "Impact on Existing APIs" above.
Implementation
==============
Assignee(s)
-----------
Who is leading the writing of the code? Or is this a blueprint where you're
throwing it out there to see who picks it up?
If more than one person is working on the implementation, please designate the
primary author and contact.
Primary assignee:
Doug Hellmann (doug-hellmann)
Other contributors:
None
Milestones
----------
Target Milestone for completion:
Juno-2
Work Items
----------
1. Verify that the :class:`ContextHandler` works properly with
:class:`Message`, and update it to make it work if it does not.
2. See "Proposed Change" above.
Incubation
==========
None
Adoption
--------
None
Library
-------
oslo.log
Anticipated API Stabilization
-----------------------------
This change is part of stabilizing the API for oslo.log before
graduation.
Documentation Impact
====================
None
Dependencies
============
* We need to remove the import cycle between log and versionutils
before implementing this
change. https://blueprints.launchpad.net/oslo/+spec/app-agnostic-logging-parameters
References
==========
* Discussion from the Juno summit: https://etherpad.openstack.org/p/juno-oslo-release-plan
* Related blueprint on using our context as a base class: https://blueprints.launchpad.net/oslo/+spec/app-agnostic-logging-parameters
* Related blueprint for graduating oslo.log: https://blueprints.launchpad.net/oslo/+spec/graduate-oslo-log
.. note::
This work is licensed under a Creative Commons Attribution 3.0
Unported License.
http://creativecommons.org/licenses/by/3.0/legalcode

304
specs/juno/taskflow-conditional-execution.rst
View File

@ -1,304 +0,0 @@
=======================
Conditional execution
=======================
Include the URL of your launchpad blueprint:
https://blueprints.launchpad.net/taskflow/+spec/conditional-flow-choices
Allow a level of conditional execution of compiled `atoms`_ (typically
`tasks`_) and associated `flows`_ (aka workflows) that can be used to alter
the execution *path* based on some type of condition and a decided outcome.
.. _atoms: http://docs.openstack.org/developer/taskflow/atoms.html#atom
.. _flows: http://docs.openstack.org/developer/taskflow/patterns.html
.. _tasks: http://docs.openstack.org/developer/taskflow/atoms.html#task
Problem description
===================
Mistral and cinder (and others?) would like to be able to conditionally have
a task or subflow execute based on some type of selection made by either a
prior atom or an error or a function (for example). This kind of
conditional execution right now is difficult currently in taskflow since the
current model is more of a declaratively *fixed* pipeline that is verified
before execution and only completes execution when all atoms have ``EXECUTED``,
``FAILED`` or ``REVERTED`` (when retries are involved an atom or group of
atoms can flip between ``FAILED``, ``REVERTING`` and ``RUNNING`` as
the associated `retry`_ controller tries to resolve the execution
state). Conditional execution alters this since now we cannot, at compile time,
make a decision about whether an atom will or will not execute (since it is
not known until runtime whether the atom will execute) and we will be required
skip parts of the workflow based on a conditional outcome. Due to this these
*skipped* pieces of the workflow will now not have a ``EXECUTED``
or ``REVERTED`` state but will be placed in a new state ``IGNORED``.
Overall, to accommodate this feature we will try to adjust taskflow to provide
at least a *basic* level of support to accomplish a *type* of conditional
execution. In the future this support can be expanded as the idea and
implementation matures and proves itself.
Proposed change
===============
In order to support conditionals we first need to determine what a conditional
means in a workflow, what it looks like and how it would be incorporated into
a user's workflow.
The following is an example of how this *may* look when a user decides to add
a conditional choice into a workflow:
::
nfs_flow = linear_flow.Flow("nfs-work")
nfs_flow.add(
ConfigureNFS(),
StartNFS(),
AttachNFS(),
...
)
nfs_matcher = lambda volume_type: volume_type == 'nfs'
ceph_flow = linear_flow.Flow("ceph-work")
ceph_flow.add(
ConfigureCeph(),
AttachCeph(),
...
)
ceph_matcher = lambda volume_type: volume_type == 'ceph'
root_flow = linear_flow.Flow("my-work")
root_flow.add(
PopulateDatabase(),
# Only one of these two will run (this is equivalent to an if/elif)
LambdaGate(requires="volume_type",
decider=nfs_matcher).add(nfs_flow),
LambdaGate(requires="volume_type",
decider=ceph_matcher).add(ceph_flow),
SendNotification(),
...
)
**Note:** that after this workflow has been constructed that the user will then
hand it off to taskflow for reliable, consistent execution using one of the
supported `engines`_ types, parallel, threaded or other...
Implementation
--------------
To make it simple (to start) let's limit the scope and suggest that a
conditional (a gate, gate seems to be a common name for these according
to `advances in dataflow programming`_) is restricted to being a runtime
scheduling decision that causes the path of execution to allow (aka pass) or
ignore (aka discard) the components that are contained with-in the gate.
This means that at the `compile time`_ phase when a gate is inserted into a
flow that any output symbols produced by the subgraph under the gate can
not be depended upon by any successors that follow the gate decision.
For example the following will **not** be valid since it introduces a symbol
dependency on a conditional gate (in a future change we could set these
symbols to some defaults when they will not be produced, for now though we
will continue *more* being strict):
::
nfs_flow = linear_flow.Flow("nfs-work")
nfs_flow.add(
ConfigureNFS(provides=["where_configured"]),
AttachNFS(),
...
)
nfs_matcher = lambda volume_type: volume_type == 'nfs'
root_flow = linear_flow.Flow("my-work")
root_flow.add(
PopulateDatabase(),
LambdaGate(requires="volume_type",
decider=nfs_matcher).add(nfs_flow),
SendNotification(requires=['where_configured']),
...
)
The next thing a conditional then needs a mechanism to influence
the *outcome* which should be executed to either allow its subgraph to pass
or to be discarded. When a gate is encountered while executing (assume there
is some way to know that we have *hit* a gate) we need to first freeze
execution of any successors (nothing must execute in any outcomes subgraph
ahead of time, this would violate the conditional constraint).
**Note:** that this does *not* mean that a subgraph executing that is not
connected to this conditional will have to be stopped (its execution is not
dependent on this outcome decision).
So assuming we can freeze dependent execution (which we currently can do, since
taskflow `engines`_ are responsible for scheduling further work) we now need
to provide the gate a way to make a decision; usually the decision will be
based on some prior execution or other external state. We have a mechanism for
doing this already so we will continue using the
existing `inputs and outputs`_ mechanism to communicate any
state (local or otherwise) to the gate. Now the gate just needs to be able to
make a boolean (or `truthy`_ value) decision about whether what is contained
after the gate should run or should not run so that the runtime can continue
down that execution path. To accommodate this we will require the gate object
to provide an ``execute()`` method (this allows gates themselves to
be an `atom`_ derived type) that returns a `truthy`_ value. If the value
returned is ``true`` then the gate will have been determined to have been
passed and the contained subgraph is then eligible for execution and
further scheduling. Otherwise if the value that is
returned is ``false`` (or falsey) then the contained subgraphs nodes will be
put into the ``IGNORE`` state before further scheduling occurs.
**Note:** this occurs *before* further scheduling so that if a failure
occurs (``kill -9`` for example) during saving those atoms ``IGNORE``
states that a resuming entity can attempt to make forward progress in saving
those same ``IGNORE`` states; without having to worry about an outcomes
subgraph having started to execute.
After this completes scheduling will resume and the nodes marked ``IGNORE``
will not be executed by current & future scheduling decisions (and the engine
will continue scheduling and completing atoms and all will be merry...).
Retries
#######
Conditionals have another an interesting interaction with `retry`_ logic, in
that when a subgraph is retried, we must decide what to do about the prior
outcome which may have been traversed and decide if we should allow the prior
outcome decision to be altered. This means that when a execution graph is
retried it can be possible to alter the gates decision and enter a new
subgraph (which may contain its own new set of retry strategies and
so-on). To start we will *clear* the outcome of a gate when a retry
resets/unwinds the graph that a retry object has *control* over (this will
cause the gate to be executed again). It will also be required to flip the
``IGNORED`` state back to the ``PENDING`` state so that the gates contained
nodes *could* be rescheduled. The gate would then have to use whatever provided
symbol inputs to recalculate its decision and decide on a new outcome (this
then could cause a new subgraph to be executed and so-on). This way will allow
for the most natural integration with the existing codebase (and is likely what
users would expect to happen).
.. _engines: http://docs.openstack.org/developer/taskflow/engines.html
.. _advances in dataflow programming: http://www.cs.ucf.edu/~dcm/Teaching/COT4810-Spring2011/Literature/DataFlowProgrammingLanguages.pdf
.. _inputs and outputs: http://docs.openstack.org/developer/taskflow/inputs_and_outputs.html
.. _retry: http://docs.openstack.org/developer/taskflow/atoms.html#retry
.. _truthy: https://docs.python.org/release/2.5.2/lib/truth.html
.. _compile time: http://docs.openstack.org/developer/taskflow/engines.html#compiling
.. _atom: http://docs.openstack.org/developer/taskflow/atoms.html#atom
Alternatives
------------
Some of the current and prior research was investigated to understand the
different strategies others have done to make this kind of conditionals
possible in similar languages and libraries:
* http://www.cs.ucf.edu/~dcm/Teaching/COT4810-Spring2011/Literature/DataFlowProgrammingLanguages.pdf
* http://paginas.fe.up.pt/~prodei/dsie12/papers/paper_17.pdf
* http://dl.acm.org/citation.cfm?id=3885
* And a few others...
Various pipelining/workflow like pypi libraries were looked at. None that I
could find actually provide conditional execution primitives. A not fully
inclusive list:
* https://pypi.python.org/pypi/DAGPype
* https://github.com/SegFaultAX/graffiti
* And a few others...
Impact on Existing APIs
-----------------------
This will require a new type/s that when encountered can be used to decide
which outcome should taken (for now a *gate* type and possibly a *lambda gate*
derived class). These new types will be publicly useable types that can be
depended upon working as expected (observing and operating by the constraints
described above).
Security impact
---------------
N/A
Performance Impact
------------------
N/A
Configuration Impact
--------------------
N/A
Developer Impact
----------------
A new way of using taskflow would be introduced that would hopefully receive
usage and mature as taskflow continues to progress, mature and get more
innovative usage.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
harlowja
Milestones
----------
Juno/2
Work Items
----------
* Introduce new gate types.
* Connect types into compilation routine.
* Connect types into scheduling/runtime routine.
* Test like crazy.
Incubation
==========
N/A
Adoption
--------
N/A
Library
-------
N/A
Anticipated API Stabilization
-----------------------------
N/A
Documentation Impact
====================
New developer docs explaining the concepts, how to use this and examples will
be provided and updated accordingly.
Dependencies
============
None
References
==========
* Brainstorm: https://etherpad.openstack.org/p/BrainstormFlowConditions
* Prototype: https://review.openstack.org/#/c/87417/
.. note::
This work is licensed under a Creative Commons Attribution 3.0
Unported License.
http://creativecommons.org/licenses/by/3.0/legalcode

397
specs/juno/taskflow-improved-scoping.rst
View File

@ -1,397 +0,0 @@
=====================================
Better scoping for atoms and flows.
=====================================
Include the URL of your launchpad blueprint:
https://blueprints.launchpad.net/taskflow/+spec/improved-scoping
Better scoping for atoms and flows.
Problem description
===================
Currently `atoms`_ in taskflow have the ability to declare, rebind, and receive
automatically needed runtime symbols (see `arguments and results`_) and provide
named output symbols. This feature is required to enable an external entity to
arrange execution ordering as well as to allow for state transfer/retainment
to be performed by a workflow runtime (in this case the `engine`_
concept). This is quite useful to allow for automated resuming (and other
features, such as parallel execution) since when an atom does not
maintain state (or maintains very little) internally, the workflow runtime
can track the execution flow and the output symbols and input
symbols (with associated values) of atoms using
various `persistence strategies`_ (it also allows the engine to notify others
about state transitions and other nifty features...).
**Note:** In a way this externalized state is equivalent to a
workflows *memory* (without actually retaining that state on the runtime stack
and/or heap). This externalization of workflow execution & state enables
many innovative strategies that can be explored in the future and is one of the
key design patterns that taskflow was built in-mind with.
In general though we need to increase the usefulness & usability of the current
mechanism. It currently has some of the following drawbacks (included with
each is an example to make this more clear as to why it's a drawback):
* Overly complicated ``rebinding/requires/provides`` manipulation to avoid
symbol naming conflicts. For example, when one atom in a `flow`_ produces an
output that has a name of another atoms input and a third atom in a subflow
also produces the same output there is required to be
a `rebinding`_ application to ensure that the right output goes into the
right input. This results in a bad (not horrible, just bad) user
interaction & experience.
**Example:**
::
from taskflow import task
from taskflow.patterns import linear_flow
class Dumper(task.Task):
def execute(self, *args, **kwargs):
print(self)
print(args)
print(kwargs)
r = linear_flow.Flow("root")
r.add(
Dumper(provides=['a']),
Dumper(requires=['a'],
provides=['c']),
Dumper(requires=['c']),
)
sr = linear_flow.Flow("subroot")
sr.add(Dumper(provides=['c']))
sr.add(Dumper(requires=['c']))
# This line fails with the following message:
#
# subroot provides ['c'] but is already being provided by root and
# duplicate producers are disallowed
#
# It can be resolved by renaming provides=['c'] -> provides=['c_1'] (and
# subsequently renaming the following requires in the subroot flow) which
# instead should be resolved by proper lookup & scoping of inputs and
# outputs.
r.add(sr)
As can be seen in this example we have created a scoping *like* mechanism via
the nested flow concept & implementation but we have not made it as easy as it
should be to nest flows that have *conflicting* symbols for inputs (aka,
required bound symbols) and outputs (aka, provided bound symbols). This should
be resolved so that it becomes much easier to combine arbitrary flows together
without having to worry about symbol naming errors & associated issues.
**Note:** we can (and certain folks are) using the ability to inject symbols
that are requirements of atoms before running to create a atom local scope.
This helps in avoiding some of the runtime naming issues but does not solve the
full problem.
* Loss of nesting post-compilation/post-runtime. This makes it hard to do
extraction of results post-execution since it limits how those results can
be fetched (making it unintuitive to users why they can not extract results
for a given nesting hierarchy). This results in a bad user experience (and
likely is not what users expect).
**Example:**
::
from taskflow.engines.action_engine import engine
from taskflow.patterns import linear_flow
from taskflow.persistence.backends import impl_memory
from taskflow import task
from taskflow.utils import persistence_utils as pu
class Dumper(task.Task):
def execute(self, *args, **kwargs):
print("Executing: %s" % self)
print(args)
print(kwargs)
return (self.name,)
r = linear_flow.Flow("root")
r.add(
Dumper(name="r-0", provides=['a']),
Dumper(name="r-1", requires=['a'], provides=['c']),
Dumper(name="r-2", requires=['c']),
)
sr = linear_flow.Flow("subroot")
sr.add(Dumper(name="sr-0", provides=['c_1']))
sr.add(Dumper(name="sr-1", requires=['c_1']))
r.add(sr)
# Create needed persistence layers/backends...
storage_backend = impl_memory.MemoryBackend()
detail_book, flow_detail = pu.temporary_flow_detail(storage_backend)
# Create an engine and run.
engine_conf = {}
e = engine.SingleThreadedActionEngine(
r, flow_detail, storage_backend, engine_conf)
e.compile()
e.run()
print("Done:")
print e.storage.fetch_all()
# Output produced is the following:
#
# Executing: r-0==1.0
# ()
# {}
# Executing: r-1==1.0
# ()
# {'a': 'r-0'}
# Executing: r-2==1.0
# ()
# {'c': 'r-1'}
# Executing: sr-0==1.0
# ()
# {}
# Executing: sr-1==1.0
# ()
# {'c_1': 'sr-0'}
# Done:
# {'a': 'r-0', 'c_1': 'sr-0', 'c': 'r-1'}
#
# No exposed API to get just the results of 'subroot', the only exposed
# API is to get by atom name or all, this makes it hard for users that just
# want to extract individual results from a given segment of the
# overall hierarchy.
To increase the usefulness of the storage, persistence and workflow concept
we need to expand the inference, validation, input and output, storage and
runtime lookup mechanism to better account for the `scope`_ a atom resides
in.
.. _atoms: http://docs.openstack.org/developer/taskflow/atoms.html#atom
.. _arguments and results: http://docs.openstack.org/developer/taskflow/arguments_and_results.html#arguments-specification
.. _engine: http://docs.openstack.org/developer/taskflow/engines.html
.. _scope: https://en.wikipedia.org/wiki/Scope_%28computer_science%29
.. _rebinding: http://docs.openstack.org/developer/taskflow/arguments_and_results.html#rebinding
.. _flow: http://docs.openstack.org/developer/taskflow/patterns.html#taskflow.flow.Flow
.. _persistence strategies: http://docs.openstack.org/developer/taskflow/persistence.html
Proposed user facing change
===========================
To ensure the case where a subflow produces output symbols that conflict with a
contained parent flow we will allow for a subflow to provide the same output
as a prior sibling/parent instead of denying that addition. This means that if
a parent flow contains a atom/flow ``X`` that produces symbol ``a`` and it
contains another atom or subflow ``Y`` that also produces ``a`` the ``a`` which
will be visible to items following ``Y`` will be the ``a`` produced
by ``Y`` and not by ``X``. For the items inside ``Y`` the ``a`` that will be
visible will be determined by the location in ``Y`` where ``a`` is
produced (the items that use ``a`` before ``a`` is produced in ``Y`` will use
the ``a`` produced by ``X`` and the items after ``a`` is produced in ``Y`` will
use the new ``a``). This type of *shadowing* reflects a concept how people
familiar with programming already use (`variable name shadowing`_).
To allow a flow to retain even *more* control of its exposed input and output
symbols we will introduce the following new flow constructor parameter.
* ``contain=<CONSTANT>``: when set on a flow object this attribute will cause
the flow to behave differently when intermixed with other flows. One of the
constants to be will be ``contain=REQUIRES`` which will denote that this
flow will use only requirements that are produced by the atoms contained
in itself and **not** try to require any symbols from its parent or prior
sibling flows or atoms. This attribute literally means the scope of
the flow will be completly self contained. A second constant (these
constants can be *ORed* together to combine them in various ways) will
be ``contain=PROVIDES`` which will denote that the symbols this
flow *may* produce will **not** be consumable by any subsequent sibling
flows or atoms. This attribute literally means that the scope of the flow
will be restricted to **only** using requirements from prior sibling or
parent flows and the produced output symbols will **not** be visible to
subsequent sibling flows or atoms.
When no constant is provided we will assume the standard routine of not
restricting input and output symbols and only applying the shadowing rule
defined previously.
**Note:** depending on time constraints we have the ability to just skip the
different ``contain`` constants and just do the shadowing approach (and later
add in the other various constants as time permits).
.. _variable name shadowing: https://en.wikipedia.org/wiki/Variable_shadowing
Proposed runtime change
=======================
During runtime we will be required to create a logical structure which retains
the same user facing constraints. To do this we will retain information about
the atom and flow `symbol table`_ like hierarchy at runtime in a secondary
tree structure (so now instead of *just* retaining a directed graph of the
atoms and flows prior structure we will retain a directed graph and a tree
hierarchical structure).
This tree structure will contain a representation of the hierarchy that
atoms were composed in and the symbols being produced at the different levels.
For example an atom in a top level flow will be at a higher level in that tree
and a atom in a subflow will be at a lower level in that tree. The leaf nodes
of the tree will be the individual atom objects + any associated metadata and
the non-leaf nodes will be the flow objects + any associated metadata (the main
piece of metadata in flow nodes will be a symbol table, also known as a
dictionary). This structure & associated metadata will be constructed
at compilation time where we presently construct the directed graph of
nodes to run.
This approach allows the lookup of an atoms requirements to become a symbol
table & tree traversal problem where the atoms (now a node in the tree) parents
will be traversed until an atom that produces a needed symbol is located (this
information is verified at preparation time, which happens right before
execution, so it can be assumed there are no atoms that have symbols that are
*not* provided by some other atom).
At compilation time the ``contain=<CONSTANT>`` attribute will also be examined
and metadata will be associated with the created tree node to signify what the
visiblity of the symbol table for that node is. This metadata will be used
during the runtime symbol name lookup process to ensure we restrict the lookup
of symbols to the constraints imposed by the selected attribute/s.
At runtime when a symbol is needed for an atom we will locate the node that
is associated with the atom in the tree and walk upwards until we find the
correct symbol (obeying the ``contain`` constraints as needed) and value. When
saving we will save values into the parent flow nodes symbol table instead of
into the single symbol table that is saved into currently.
Finally, this addition makes it possible for post-execution extraction of
individual tree segments (by allowing for fetching a tree nodes symbol table
and allowing for users to traverse it as they desire). This is often useful
for examining the results flows and atoms produced after the workflow runtime
has finished executed (and doing any further function/method... calls that an
application may wish to do with those results).
.. _symbol table: http://en.wikipedia.org/wiki/Symbol_table
Alternatives
------------
The alternative is not to change anything and require that users go through
a painful symbol renaming (and extraction) process. This works *ok* for
workflows that are controlled and where it is possible to define the flow in a
single function where all the various symbol names can be adjusted at flow
creation time. It does not work well for arbitrary gluing of various workflows
together from arbitrary sources (a use-case that would be expected to be common
in the OpenStack projects, where drivers *could* provide components of an
overall workflow). Without this change it would likely mean that there would be
various functions created by users that would have *messy* and
*complicated* symbol renaming algorithms to resolve the issue that taskflow
should instead resolve itself. This results in a bad user experience (and
likely is not what users expect).
Impact on Existing APIs
-----------------------
The existing API's will continue operating as before, when the new options
are set the functionalty will change accordingly to be less strict. Now instead
of duplicate names causing errors a new mode will be enabled by default, the
variable shadowing mode. This will allow flows that would have not been
allowed to be created before now to be created. In general this will be an
additive change that enables new usage that errored out before this change.
Security impact
---------------
N/A
Performance Impact
------------------
N/A
Configuration Impact
--------------------
N/A
Developer Impact
----------------
This scoping should make it easier to implement flows in a manner that
conceptually makes sense for programmers used to the standard scoping
strategies that programming languages come built-in with.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
* harlowja
Other contributors:
* dkrause?
Milestones
----------
J/3
Work Items
----------
* Add a tree type [https://review.openstack.org/#/c/97325/]
* Add ``contains`` constraints to flows and adjust pattern ``add()`` methods
to accept and verify those constraints at atom/subflow addition time.
* Retain symbol hierarchy at compilation time by constructing a tree instance
and during the directed graph creation routine adding nodes to this tree as
needed (along with any other metadata needed).
* Adjust the compilation routine to retain this ``contains`` attribute in the
tree nodes metadata so that it can be using at runtime.
* Adjust the action engine implementation to use this new source of information
during symbol lookup so that this new information is used during runtime.
* Expose the results of running via a new api that allows for fetching a named
atom/flows storage resultant ``node`` (this allows for traversing over the
symbol tables for children nodes contained there-in).
* Test like crazy.
Future ideas
------------
* In a future change we could support the ability to have automatic symbol
names that would be populated at compilation time. This would allow the flow
creator to associate a ``<anonymous>`` like object as the symbol that will
be transferred between tasks/atoms (which right now is required to be
a string). The ``<anonymous>`` object instance will be translated into
a *actual* generated symbol name at compilation time (the runtime symbol
lookup mechanism will then be unaffected by this change). This would help
those users that can not use the above new capabilities. It would allow those
users to have a way to transfer symbols between scopes without
being *as* restricted by literal string names.
Incubation
==========
N/A
Documentation Impact
====================
Developer docs, examples will be updated to explain the new change and provide
examples of how this new change can be used.
Dependencies
============
N/A
References
==========
N/A
.. note::
This work is licensed under a Creative Commons Attribution 3.0
Unported License.
http://creativecommons.org/licenses/by/3.0/legalcode