Enqueue SSH commands through a thread pool

We now use a thread pool to queue up and run received SSH commands.
This way the server has a maximum number of threads it will permit
to execute on the CPUs at once, and over-subscription is impossible
as additional work requests will be queued up and serviced only
when a thread becomes available.

We seed the SSH thread pool at 1.5x the number of CPUs in the system,
under the assumption that the threads may wind up blocking for IO
and therefore would be able to yield and share the CPUs.  This can
be changed by the administrator with the sshd.threads parameter.

Threads in the SSH thread pool run at MIN_PRIORITY so they get
pushed into the background relative to HTTP service threads.
The SSH transport tends to be for larger bulk operations with longer
executions, so it should take a back seat to HTTP requests which
we want to service in sub 200 millisecond times.

To aid debugging a worker thread in the SSH thread pool changes
its name to the command it is executing.  This makes stack dumps
easier to inspect when looking for deadlock.

Admin-only commands always bypass the thread pool and execute on
their own unique thread.  These commands tend to be executed a lot
less often then user commands, and may be used by an administrator
to inspect current server state.  We don't want them to block and
wait for a thread to become available.

Bug: issue 320
Change-Id: I4060182d0662594c0caea28b6915df1d3740aa19
Signed-off-by: Shawn O. Pearce <sop@google.com>

gerrit-sshd/src/main/java/com/google/gerrit/sshd/SshModule.java

@@ -25,6 +25,7 @@ import com.google.gerrit.server.IdentifiedUser;
 import com.google.gerrit.server.RemotePeer;
 import com.google.gerrit.server.config.FactoryModule;
 import com.google.gerrit.server.config.GerritRequestModule;
+import com.google.gerrit.server.git.WorkQueue;
 import com.google.gerrit.server.project.ProjectControl;
 import com.google.gerrit.server.ssh.SshInfo;
 import com.google.gerrit.sshd.args4j.AccountGroupIdHandler;
@@ -75,6 +76,9 @@ public class SshModule extends FactoryModule {
     bind(CommandFactoryProvider.class);
     bind(CommandFactory.class).toProvider(CommandFactoryProvider.class);
 
+    bind(WorkQueue.Executor.class).annotatedWith(CommandExecutor.class)
+        .toProvider(CommandExecutorProvider.class).in(SINGLETON);
+
     bind(PublickeyAuthenticator.class).to(DatabasePubKeyAuth.class);
     bind(KeyPairProvider.class).toProvider(HostKeyProvider.class).in(SINGLETON);