Christian Berendt 44243238ac [sitemap] do not add empty URLs to the list of start URLs ...

This will solve the 'ValueError: Missing scheme in request url'
exception.

Change-Id: I295b2a519668b36413ec2ba4ef3ba9dfaac6e859

2016-01-22 14:58:44 +01:00

..

spiders

[sitemap] do not add empty URLs to the list of start URLs

2016-01-22 14:58:44 +01:00

__init__.py

script to generate the sitemap.xml for docs.openstack.org

2014-05-29 01:29:18 +02:00

items.py

script to generate the sitemap.xml for docs.openstack.org

2014-05-29 01:29:18 +02:00

pipelines.py

[sitemap] resolve remaining ScrapyDeprecationWarning exceptions

2015-10-02 10:34:13 +02:00

settings.py

Merge "[sitemap] respect the robots.txt policies"

2015-10-20 12:15:50 +00:00