Pre-cache all of the things devstack might need