Parallelize static resource cleanup

Similar to the parallel registration of static nodes on startup we also
need to optimize the leaked resource cleanup. The problem here is the
same in that node registration was serialized and could be slowed down
significantly by a lot of timeouts.

This in turn blocked node re-registration in the node deleted
notification, which also needs the register lock.

Change-Id: Ibb4f759b1a98d564fe5eab065824239cec72364b

nodepool/driver/static/provider.py

@@ -443,12 +443,19 @@ class StaticNodeProvider(Provider, QuotaSupport):
     def cleanupLeakedResources(self):
         if self._idle:
             return
+
         with self._register_lock:
             self.getRegisteredNodes()
+            with ThreadPoolExecutor() as executor:
                 for pool in self.provider.pools.values():
+                    synced_nodes = []
                     for static_node in pool.nodes:
+                        synced_nodes.append((static_node, executor.submit(
+                            self.syncNodeCount, static_node, pool)))
+
+                    for static_node, result in synced_nodes:
                         try:
-                        self.syncNodeCount(static_node, pool)
+                            result.result()
                         except StaticNodeError as exc:
                             self.log.warning("Couldn't sync node: %s", exc)
                             continue