Add a disallow_robots parameter which can be used to disallow
indexing of all site content by robots. This is particularly useful
for test deployments where you don't want extra (often stale) copies
of your content to show up in search engines.
Change-Id: Ic62a72555315bd344db338809920a3605f17c8c6
