openstack/openstack-doc-tools
Code Issues Proposed changes
512d64fd78
openstack-doc-tools/sitemap
History
Christian Berendt 02d0060144 [sitemap] increase the DOWNLOAD_WARNSIZE 
The default value for the DOWNLOAD_WARNSIZE setting is 33554432 (32MB).
This patch will double the value. This solves the following warning:

---snip---
WARNING: Received (37889579) bytes larger than download warn size
(33554432).
---snap---

Change-Id: I5be4bcefa6fe2eb474901bb958669978dd6e1fef
2015-10-02 10:38:45 +02:00
..
generator [sitemap] increase the DOWNLOAD_WARNSIZE 2015-10-02 10:38:45 +02:00
README.rst [sitemap] add a requirements file 2015-10-02 08:28:37 +02:00
requirements.txt [sitemap] add a requirements file 2015-10-02 08:28:37 +02:00
scrapy.cfg script to generate the sitemap.xml for docs.openstack.org 2014-05-29 01:29:18 +02:00
transform-sitemap.xslt Remove /draft from sitemap 2015-04-18 09:43:10 +02:00

README.rst

Sitemap Generator

This script crawls all available sites on http://docs.openstack.org and extracts all URLs. Based on the URLs the script generates a sitemap for search engines according to the protocol described at http://www.sitemaps.org/protocol.html.

Usage

To generate a new sitemap file simply run the spider using the following command. It will take several minutes to crawl all available sites on http://docs.openstack.org. The result will be available in the file sitemap_docs.openstack.org.xml.

$ scrapy crawl sitemap

It's also possible to crawl other sites using the attribute domain.

For example to crawl http://developer.openstack.org use the following command. The result will be available in the file sitemap_developer.openstack.org.xml.

$ scrapy crawl sitemap -a domain=developer.openstack.org

To write log messages into a file append the parameter -s LOG_FILE=scrapy.log.

Dependencies

To install the needed modules you can use pip or the package management system included in your distribution. When using the package management system maybe the name of the packages differ. When using pip it's maybe necessary to install some development packages.

$ pip install -r requirements.txt