From 70bd2f0785a0eb695fad9c2ed806d6602950a51a Mon Sep 17 00:00:00 2001 From: Jeremy Stanley Date: Thu, 26 Feb 2026 17:50:39 +0000 Subject: [PATCH] Warn crawlers not to touch the tripwire OpenDev's static content sites implement web application firewall rules designed to catch and block malicious crawlers by instructing them not to index a particular non-existent file and then blocking any client that tries to reach it. Add a corresponding robots.txt directive to the docs.openstack.org site similar to the one used in https://docs.opendev.org/robots.txt already. Change-Id: I3fa133f06998d69245ce34191771604a7dce451a Signed-off-by: Jeremy Stanley --- www/static/robots.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/www/static/robots.txt b/www/static/robots.txt index 30e40ba62d..68a3df28ab 100644 --- a/www/static/robots.txt +++ b/www/static/robots.txt @@ -15,3 +15,6 @@ Disallow: /ironic/pike Disallow: /ironic/queens Disallow: /ironic/rocky Disallow: /ironic/stein + +# Tripwire URL designed to catch and block misbehaving crawlers +Disallow: /83572085-b2cf-4be3-b8ed-abff38714d87.html