openstack/openstack-manuals
Code Issues Proposed changes

Warn crawlers not to touch the tripwire

Browse Source 
OpenDev's static content sites implement web application firewall
rules designed to catch and block malicious crawlers by instructing
them not to index a particular non-existent file and then blocking
any client that tries to reach it. Add a corresponding robots.txt
directive to the docs.openstack.org site similar to the one used in
https://docs.opendev.org/robots.txt already.

Change-Id: I3fa133f06998d69245ce34191771604a7dce451a
Signed-off-by: Jeremy Stanley <fungi@yuggoth.org>
This commit is contained in:
Jeremy Stanley
2026-02-26 17:50:39 +00:00
parent 3dbe0fd35e
commit 70bd2f0785
1 changed files with 3 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
www/static/robots.txt
View File

@@ -15,3 +15,6 @@ Disallow: /ironic/pike
Disallow: /ironic/queens
Disallow: /ironic/rocky
Disallow: /ironic/stein
# Tripwire URL designed to catch and block misbehaving crawlers
Disallow: /83572085-b2cf-4be3-b8ed-abff38714d87.html