We've seen in production a huge delay between image builds that have
been requested until the build has actually been started. This was
despite several free builders.
Profiling showed that all of our builders are extremely busy with
reloading and comparing their config [1]. This is with a ~2MB sized
nodepool.yaml and accordingly many diskimages.
The reason for this bottleneck is that after every image we check
we're reloading the config and re-comparing it with the currently
active config. This config checking currently effectively grows
quadratically with the number of diskimages (more iterations and
slower config check).
To solve this problem we should perform this check less often.
[1] Yappi data:
name ncall tsub ttot tavg
..er/__init__.py:852 AwsLabel.__eq__ 468.. 144.6263 252.2575 0.000054
..odepool/config.py:276 Label.__eq__ 1962 0.015242 251.5035 0.128187
..py:866 AwsProviderDiskImage.__ne__ 234.. 3.761890 86.93310 0.000037
../nodepool/config.py:425 loadConfig 4 0.000625 22.47684 5.619211
..-packages/yaml/__init__.py:74 load 20 0.128956 21.65417 1.082708
Change-Id: I786daa20ca428039a44d14b1e389d4d3fd62a735
00d25045ea