227839e8ad
Copy static files directly into the container image instead of managing them dynamically with Ansible. Change-Id: I0ebe40ad2a97e87b00137af7c93a3ffa84929a2e
12 lines
206 B
Plaintext
12 lines
206 B
Plaintext
# Directions for web crawlers.
|
|
# See http://www.robotstxt.org/wc/norobots.html.
|
|
|
|
User-agent: HTTrack
|
|
User-agent: puf
|
|
User-agent: MSIECrawler
|
|
User-agent: Nutch
|
|
Disallow: /
|
|
|
|
User-agent: msnbot
|
|
Crawl-delay: 1
|