afa87385cc
And slow down bing (msnbot). Change-Id: Id8361047abc2cfb52260b3d0ef01275ec3a923f5 Reviewed-on: https://review.openstack.org/32435 Reviewed-by: Jeremy Stanley <fungi@yuggoth.org> Reviewed-by: Elizabeth Krumbach Joseph <lyz@princessleia.com> Reviewed-by: Anita Kuno <anita.kuno@enovance.com> Approved: James E. Blair <corvus@inaugust.com> Tested-by: Jenkins
12 lines
206 B
Plaintext
12 lines
206 B
Plaintext
# Directions for web crawlers.
|
|
# See http://www.robotstxt.org/wc/norobots.html.
|
|
|
|
User-agent: HTTrack
|
|
User-agent: puf
|
|
User-agent: MSIECrawler
|
|
User-agent: Nutch
|
|
Disallow: /
|
|
|
|
User-agent: msnbot
|
|
Crawl-delay: 1
|