wsgi: Reap stale workers (after a timeout) following a reload

Add a new tunable, `stale_worker_timeout`, defaulting to 86400 (i.e. 24
hours). Once this time elapses following a reload, the manager process
will issue SIGKILLs to any remaining stale workers.

This gives operators a way to configure a limit for how long old code
and configs may still be running in their cluster.

To enable this, the temporary reload child (which waits for the reload
to complete then closes the accept socket on all the old workers) has
grown the ability to send state to the re-exec'ed manager. Currently,
this is limited to just the set of pre-re-exec child PIDs and their
reload times, though it was designed to be reasonably extensible.

This allows the new manager to recognize stale workers as they exit
instead of logging

   Ignoring wait() result from unknown PID ...

With the improved knowledge of subprocesses, we can kick the log level
for the above message up from info to warning; we no longer expect it
to trigger in practice.

Drive-by: Add logging to ServersPerPortStrategy.register_worker_exit
that's comparable to what WorkersStrategy does.

Change-Id: I8227939d04fda8db66fb2f131f2c71ce8741c7d9
This commit is contained in:
Tim Burke
2021-04-30 13:33:37 -07:00
committed by Matthew Oliver
parent 734ed9cdd8
commit ae6300af86
9 changed files with 315 additions and 4 deletions

View File

@@ -330,6 +330,12 @@ use = egg:swift#proxy
# ionice_class =
# ionice_priority =
#
# When reloading servers with SIGUSR1, workers running with old config/code
# are allowed some time to finish serving in-flight requests. Use this to
# configure the grace period (in seconds), after which the reloaded server
# will issue SIGKILLs to remaining stale workers.
# stale_worker_timeout = 86400
#
# When upgrading from liberasurecode<=1.5.0, you may want to continue writing
# legacy CRCs until all nodes are upgraded and capabale of reading fragments
# with zlib CRCs. liberasurecode>=1.6.2 checks for the environment variable