Currently this is how reconnect works:
- pyngus detects failure and invokes callback
Controller.connection_failed() which in turn calls
Controller._handle_connection_loss()
- The first thing that _handle_connection_loss does is to set
self.addresser to None (important later)
- Then it defers _do_reconnect after a delay (normally 1 second)
- (1 second passes)
- _do_reconnect calls _hard_reset which resets the controller state
However, there is a race here. This can happen:
- The above, up until it defers and waits for 1 second
- Controller.send() is invoked on a task
- A new Sender is created, and critically because self.reply_link
still exists and is active, we call sender.attach and pass in
self.addresser. Remember _handle_connection_loss sets
self.addresser to None.
- Eventually Sender.attach throws an AttributeError because it
attempts to call addresser.resolve() but addresser is None
The reason this happens is because although the connection is dead,
the controller state is still half-alive because _hard_reset hasn't
been called yet since it's deferred one second in _do_reconnect.
The fix here is to move _hard_reset out of _do_reconnect and directly
into _handle_connection_loss. The eventloop is woken up immediately
to process _hard_reset but _do_reconnect is still deferred as before
so as to retain the desired reconnect backoff behavior.
Closes-Bug: #1941652
Change-Id: Ife62a7d76022908f0dc6a77f1ad607cb2fbd3e8f
02a38f507d