openstack/openstack-doc-tools
Code Issues Proposed changes

Merge "[sitemap] introduce attribute to define start URLs"

Browse Source
This commit is contained in:
Jenkins 2015-10-20 12:15:38 +00:00 committed by Gerrit Code Review
commit 9c5b3e7171
2 changed files with 9 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
sitemap/README.rst
View File

@ -24,6 +24,11 @@ The result will be available in the file ``sitemap_developer.openstack.org.xml``
To write log messages into a file append the parameter ``-s LOG_FILE=scrapy.log``.
It is possible to define a set of additional start URLs using the attribute
``urls``. Separate multiple URLs with ``,``.
$ scrapy crawl sitemap -a domain=developer.openstack.org -a urls="http://developer.openstack.org/de/api-guide/quick-start/"
Dependencies
============

8
sitemap/generator/spiders/sitemap.py
View File

@ -42,13 +42,13 @@ class SitemapSpider(spiders.CrawlSpider):
)
]
def __init__(self, domain='docs.openstack.org', *args, **kwargs):
def __init__(self, domain='docs.openstack.org', urls='', *args, **kwargs):
super(SitemapSpider, self).__init__(*args, **kwargs)
self.domain = domain
self.allowed_domains = [domain]
self.start_urls = [
'http://%s/index.html' % domain,
]
self.start_urls = ['http://%s/index.html' % domain]
for url in urls.split(','):
self.start_urls.append(url)
def parse_item(self, response):
item = items.SitemapItem()