Normally when exceptions are caught in DependencyTaskGroup, we cancel with
a grace period and then re-raise the first exception once all subtasks are
stopped. This means that e.g. when a resource fails, the exception the
stack gets back is from the first resource failure and not e.g. any
resulting exceptions from cancelled in-progress resources.
However, for exceptions thrown in to the task from the outside (as opposed
to raised by one of the subtasks), we generally want the *last* one to be
re-raised. Specifically, if a cancel-without-rollback exception is thrown
followed by a cancel-with-rollback, we want the with-rollback exception to
be re-raised so that the stack knows to roll back.
This allows the caller to specify an arbitrary function to be called after
each sleep() when running a task. Callers can use this to keep track of
progress or keep track of external state. Any exception raised by the
callback function will be thrown into the task.
This will be useful in numerous places where we use a TaskRunner to provide
a timeout when we are already inside a wrappertask. It eliminates the
need to manually advance the TaskRunner, and ensures correct exception
In some cases we may want to have different grace periods for different
tasks when we cancel a DependencyTaskGroup due to an error. This change
allows us to pass a function instead of just a value, so that we can
compute the correct grace period for each task.
TaskRunner.cancel() can (and, in the case of a running Resource action with
grace_period=None, will) raise an exception. To ensure that all of the
tasks in the DependencyTaskGroup get cancelled, catch and ignore any
exceptions raised when cancelling each task.
This change allows a scheduler co-routine to yield an integer n, such
that only every nth call to TaskRunner.step() results in the co-routine
being advanced. We'll be able to use this to selectively slow down
polling of resources where and when we know it doesn't make sense to
poll every second.
If we use unicode as resource name or parameter,
we are likely to encounter unicode problems.
This patch aims to fix unicode problems, and
there are several things to be clear:
1. For the class wrap with six.python_2_unicode_compatible, in
python2 their original __str__ function will map to __unicode__,
and new __str__ function comes from __unicode__.encode.
So we should always return unicode in __str__() if wrap
2. python_2_unicode_compatible will not handle __repr__,
__repr__ should return str in all versions of python.
Co-Authored-By: Ethan Lynn <email@example.com>
For an update that involves e.g. stepping through the whole graph and
verifying that nothing needs to be updated, we spend a lot of time sleeping
unnecessarilty. Every task will exit without yielding (i.e.
it will be complete after calling TaskRunner.start()), yet the
DependencyTaskGroup yields after each set of tasks so the minimum sleep
time in seconds is the maximum path length in the graph minus one.
This change fixes that by removing nodes from the graph immediately if they
are done immediately after having been started. Since the _ready() call
returns an iterator, this allows any later tasks that were blocking only on
this one to start immediately. To ensure that any tasks that are only
blocking on this one _do_ appear later, iterate over the graph in
topologically sorted order.
The potential downside to this would be any time that actions complete
quickly (i.e. without yielding), but we still need to throttle them. An
obvious case might be a resource type with no check_create_complete()
function - creating a lot of these in a row could result in quota failures
on the target API. However, the Resource.action_handler_task() task always
yields at least once even if there is no check, so this patch should not
change its behaviour.
* Tests were skipped due to a mox3 bug. The bug has now been fixed
and there's a new release of mox3. Unskip it.
* Remove the tool run-py3-tests.sh which isn't needed anymore.
We're close to the end of porting all unit tests
to be compatible with python34. However, ~8 tests are skipped
specifically in python34 because of bugs mostly in mox3.
Those tests will likely have to be converted to mock in
the future. Note that from now on, every test added will have
to be python34 compatible.
Also, fix __str__ function of ExceptionGroup in test_scheduler.
It intentionally uses str so as to not flatten the unicode
literal in python2.7 and also be compatible with python3.4
Use volume progress objects instead (as in volume attachment resources).
Module volume_tasks and class scheduler.PollingTaskGroup have been removed,
as they are no longer used anywhere.