048a918a61
We move robots.txt to custom/ instead of custom/public/ as custom/public/ is now served at /assets/ via the gitea webserver and we need robots.txt at the root. Related to this we update image urls to be prefixed with AssetUrlPrefix in their paths so that if this path changes against in the future we should automatically accomodate that. Change-Id: I8ce5fe8ff342617ff156a401be8418d593fd35c4
36 lines
755 B
Plaintext
36 lines
755 B
Plaintext
# This was kindly seeded with a mix of
|
|
# https://git.lelux.fi/theel0ja/gitea-robots.txt/src/branch/master/robots.txt
|
|
# and
|
|
# https://github.com/robots.txt
|
|
# at 2020-07-01
|
|
#
|
|
# Some commented out items are left to indicate we have considered
|
|
# them and would like to explicitly allow them for indexing while they
|
|
# are not causing problems.
|
|
|
|
User-agent: *
|
|
|
|
# Disallow: /avatars
|
|
# Disallow: /user/*
|
|
# Disallow: /*/*/src/commit/*
|
|
# Disallow: /*/*/commit/*
|
|
|
|
Disallow: /*/*/activity/*
|
|
Disallow: /vendor/librejs.html
|
|
Disallow: /api/swagger
|
|
Disallow: /swagger.*.json
|
|
|
|
# Language spam
|
|
Disallow: /*?lang=
|
|
|
|
# From github
|
|
Disallow: */archive/
|
|
Disallow: */blame/
|
|
# Disallow: /raw/*
|
|
Disallow: /.git/
|
|
Disallow: */.git/
|
|
Disallow: /*.git$
|
|
Disallow: /*q=
|
|
|
|
Crawl-delay: 2
|