6 Commits (master)

Author SHA1 Message Date
  Christian Berendt 8f05ac1b0a [sitemap] respect the robots.txt policies 4 years ago
  Christian Berendt b7ed3589d3 [sitemap] remove deprecated URL from settings 4 years ago
  Christian Berendt 02d0060144 [sitemap] increase the DOWNLOAD_WARNSIZE 4 years ago
  Christian Berendt 8a2df7b6ef [sitemap] resolve several ScrapyDeprecationWarning exceptions 4 years ago
  Christian Berendt 31c25891c1 script to generate the sitemap.xml for docs.openstack.org 5 years ago