9 lines
171 B
Plaintext
9 lines
171 B
Plaintext
# Directions for web crawlers.
|
|
# See http://www.robotstxt.org/wc/norobots.html.
|
|
|
|
User-agent: HTTrack
|
|
User-agent: puf
|
|
User-agent: MSIECrawler
|
|
User-agent: Nutch
|
|
Disallow: /
|