system-config/testinfra
Clark Boylan c499b57e16 Add robots.txt to our list servers
We've noticed that our uwsgi queues are filling up and a lot of requests
are being made to robots.txt which ends up 500/503 erroring. Add a
robots.txt file which allows crawling of our lists and archives with a
delay value in hopes this will cause bots to cache results and not fill
up the queue with repetetive requests.

Change-Id: I660d8d43f6b2d96663212d93ec48e67d86e9e761
2024-04-23 08:51:50 -07:00
..
conftest.py More yaml.safe_load() in testinfra/conftest.py 2021-10-13 21:58:41 +00:00
test_adns.py dns: abstract names 2023-04-19 09:53:10 +10:00
test_base.py logrotate: don't use filename to generate config file 2023-02-14 07:03:17 +11:00
test_borg_backups.py system-config-run-borg-backup: rename hosts to distro 2022-08-11 13:32:49 +10:00
test_bridge.py Farewell limestone 2023-02-13 23:54:59 +00:00
test_codesearch.py hound: enable detect-ref 2022-02-25 17:27:35 +11:00
test_eavesdrop.py ptgbot: rename site to ptg.opendev.org 2021-10-07 09:43:39 +11:00
test_etherpad.py Upgrade Etherpad's MariaDB to 10.11 2024-03-04 15:12:22 -08:00
test_gerrit.py Block access to Gitiles 2022-02-18 21:39:27 +00:00
test_gitea_lb.py Test gitea haproxy maxconn limits 2024-01-02 18:12:00 +00:00
test_gitea.py Increase gitea db connection limit 2024-02-05 10:40:03 -08:00
test_grafana.py testinfra: refactor screenshot taking 2021-09-07 12:54:26 +10:00
test_graphite.py graphite: fix xFilesFactor 2022-06-28 18:41:17 +10:00
test_keycloak.py Clarify testinfra socket name for keycloak rdbms 2024-02-09 17:35:03 +00:00
test_letsencrypt.py Update a few s/bridge01/bridge99 references 2022-11-11 15:05:39 +11:00
test_lists_opendev_org.py Add robots.txt to our list servers 2024-04-23 08:51:50 -07:00
test_meetpad.py Use our jitsi-meet image for meetpad 2020-03-27 09:43:41 -07:00
test_mirror-update.py Cleanup opensuse mirroring configs entirely 2024-03-18 15:49:43 -07:00
test_mirror.py More completely disable ansible galaxy proxy testing 2024-04-05 13:40:14 -07:00
test_nodepool.py Add new opendev.org nodepool launchers 2021-03-16 15:21:58 -07:00
test_ns.py dns: abstract names 2023-04-19 09:53:10 +10:00
test_paste.py paste : move testing host to paste99, remove https hacks 2022-07-07 10:02:46 +10:00
test_refstack.py refstack: Edit URL of public RefStackAPI 2021-03-10 14:09:20 +11:00
test_registry.py Deploy insecure-ci-registry.opendev.org on jammy 2023-06-23 13:21:48 -07:00
test_static.py Revert registry.zuul-ci.org 2023-11-14 16:05:28 -08:00
test_tracing.py Add Jaeger tracing server 2022-09-15 19:21:33 -07:00
test_zookeeper.py Add zk test to check myid is set in service 2023-10-17 09:31:16 -07:00
test_zuul_db.py Add a standalone zuul db server 2024-04-04 12:25:23 -07:00
test_zuul_executor.py Remove gearman from Zuul 2022-02-01 13:52:47 -08:00
test_zuul_lb.py Add Zuul load balancer 2022-02-10 13:24:42 -08:00
test_zuul_merger.py Remove gearman from Zuul 2022-02-01 13:52:47 -08:00
test_zuul_preview.py Add zp02 zuul-preview server to replace zp01 2023-06-01 10:05:46 -07:00
test_zuul_scheduler.py Remove gearman from Zuul 2022-02-01 13:52:47 -08:00
util.py testinfra: Update selenium calls 2022-10-20 09:00:43 +11:00