This is a partial revert of ae65d94e34
which did two things: made over-quota errors not fatal, and invalidated
the quota cache.
The quota cache invalidation had a typo which caused it to fail. This
error actually meant that we retained the old behavior of marking
a launch attempt as failed if we encountered an over-quota error.
This change adds a test to exercise this and fixes the typo.
In writing the test, it was observed that a launch which encounteres
an over-quota situation will continue to retry indefinitely, every
5 seconds, and will emit an exception into the log each time. This is
probably not the best experience for an operator. Further, because
there is no discernable state change within nodepool, it is very
difficult to test this situation.
Due to these issues, we should discuss a more robust way to handle
unexpected over-quota errors. In the mean time, reverting to the old
(and indeed, still current due to the typo) behavior of failing on
over-quota error, seems the safest way to proceed.
Also, change time.monotonic to time.time. The reference point for
monotonic is undefined, and can be 0, or other values less than 300,
which causes the cache timeout check to fail.
Change-Id: Id488c070df991a554570c5717dc85aec351fed45
5aac0a361e