4de5f79599
When we run gerrit, we also need to run Apache. Change-Id: Ia2f1494808bd29d83e041e224cb2eb5fc406a93b
12 lines
206 B
Plaintext
12 lines
206 B
Plaintext
# Directions for web crawlers.
|
|
# See http://www.robotstxt.org/wc/norobots.html.
|
|
|
|
User-agent: HTTrack
|
|
User-agent: puf
|
|
User-agent: MSIECrawler
|
|
User-agent: Nutch
|
|
Disallow: /
|
|
|
|
User-agent: msnbot
|
|
Crawl-delay: 1
|