Commit Graph

30 Commits (master)

Author SHA1 Message Date
Andreas Jaeger 8cca4479b6 Update sitemap
* Update releases: Newton is unmainted; train, ussuri are relased
* We have redirects to some sites, ignore them
* Remove python2 code
* Use https as starting point

Switch repo also to victoria PTI jobs.

Change-Id: I744e725b32a797b0e208ac7e81afe4b8d3578751
2020-06-21 21:48:53 +02:00
Andreas Jaeger b9d280acce Update sitemap for Python3
Fix sitemap code to work under Python3.

Change-Id: I7aad21f612047ce264b93460f7604b63fa951cab
2020-03-24 12:04:33 +01:00
Andreas Jaeger a3fdc40659 Update sitemap file
Add new releases, add already stein to not forget about it.

Update regex for releases, we do not publish only to /ocata/
but also to /PROJECT/ocata - tread all these the same.

Change-Id: I8ef454f779904d91e5e79c83c488dcba42ba2150
2018-09-09 12:00:39 +02:00
Nguyen Hung Phuong 8625096bce Clean imports in code
In some part in the code we import objects. In the Openstack style guidelines
they recommend to import only modules. [1]: "Do not import objects, only modules".


Change-Id: I49452e41f9674e6ebfcc1577d59d21750e957677
2018-02-21 13:38:16 +07:00
Andreas Jaeger fb838cf897 Do not add EOL releases to sitemap
Block out EOL releases from the sitemap.

also, lower value for lowest prio.

Change-Id: I1500ae1d602b9471cbacb62854f9c53f8378cb79
2017-07-28 20:30:33 +02:00
Andreas Jaeger 6898821aeb Change priorities for sitemap
Downgrade old EOL releases, give current maintained releases and
development versions a higher priority.

this needs changes to the test framework, update mocking. Also, rename
URL to be and not .com.

Change-Id: I2c2c0408e203a65a9541baaf55ffe60694463975
2017-07-28 20:30:06 +02:00
Andreas Jaeger d4c18f5683 sitemap: Add ocata
Handle ocata as older release.

Change-Id: I7437faf717ca277e2ba72ef38e65a2979af11346
2017-07-25 20:02:40 +02:00
Brian Moss d3bc42483a Update sitemap tests
- add py3 to tox.ini (gate already tests py3)
- move all tests to $GITROOT/test so they can all run
  through testr
- add scrapy to test-requirements.txt to support sitemap tests
- move tests from to
- fix broken sitemap tests
- add newton to list of old releases in
- ignore flake8 H101 as it returns false positives for Sphinx
- Use openstackdocstheme for docs
- Update sitemap README
- Restructure repo docs
- fix minor style issues

Change-Id: I22c018149b2eefde6ca5c38c22ac06886fe9a7a8
2017-04-29 10:36:27 +10:00
Andreas Jaeger c99bb681b2 Mitaka is an old release now
Update settings for mitaka

Change-Id: I122aafd4a3136da209ccd26d13d9060a134ce727
2016-10-06 14:33:34 +02:00
Christian Berendt 92a4fb6527 Move SitemapItem class into generator.spiders.sitemap_file
This will solve the following issue:

ImportError: No module named sitemap.generator

Change-Id: I449f0fb0c87613c36447cca94f4f5857f3d31afa
2016-10-06 13:49:43 +02:00
Percila c050836a8f doc-tools unit tests
Renamed sitemap file to avoid module name conflict
when importing at the sitemap unittest

Added py.test tox environment

Change-Id: I94480e374b29802414b62591a51c04ecd804905e
Closes-Bug: #1387716
2016-08-03 07:05:51 +00:00
Rebecca Finn ad80212a90 Remove Icehouse and Juno content from sitemap
Icehouse and Juno docs should not be accesed, so this updates the sitemap
generator tool to skip them. The sitemap can then be regenerated.

Partial-Bug: #1602647
Change-Id: I3117b114881c3a12b6a1a9565a0a720c9c8678b1
2016-07-14 16:49:28 +00:00
Christian Berendt f6715a3ba5 [sitemap] mark liberty as an old release
Change-Id: I8a16a4dd1051a351750b448e59e518e8223629b9
2016-04-05 12:59:58 +02:00
Christian Berendt 44243238ac [sitemap] do not add empty URLs to the list of start URLs
This will solve the 'ValueError: Missing scheme in request url'

Change-Id: I295b2a519668b36413ec2ba4ef3ba9dfaac6e859
2016-01-22 14:58:44 +01:00
Christian Berendt 8ebb376bb8 [sitemap] set higher priority for files of the current release
Change-Id: I9dbaa787354582f2f766fcce58aff95766d242c7
2016-01-13 11:40:52 +01:00
Jenkins c47df24295 Merge "[sitemap] respect the robots.txt policies" 2015-10-20 12:15:50 +00:00
Jenkins 9c5b3e7171 Merge "[sitemap] introduce attribute to define start URLs" 2015-10-20 12:15:38 +00:00
Jenkins 47c1209f61 Merge "[sitemap] if last-modified not set in header use current date" 2015-10-20 01:12:12 +00:00
Christian Berendt 815fcb5bb6 [sitemap] introduce attribute to define start URLs
Change-Id: I09018a824310ef48f6b62f04f09d6fd704808119
Closes-bug: #1506601
2015-10-16 11:22:05 +02:00
Christian Berendt 8f05ac1b0a [sitemap] respect the robots.txt policies
Change-Id: I219eff6241479c3c883c55079b48a88591a2a99d
2015-10-16 07:59:33 +02:00
Christian Berendt b7ed3589d3 [sitemap] remove deprecated URL from settings
Change-Id: I2c3b1a7757b0a01b5822b827b3445cd46bdbc57c
2015-10-16 07:32:49 +02:00
Christian Berendt 6b5b7d530a [sitemap] if last-modified not set in header use current date
This solves the following issue when last-modified is not set in the
response header:

KeyError: 'Last-Modified'

Change-Id: I372f9638f6b5006ff250b9dbe0e75cc1556d18c2
2015-10-16 07:28:10 +02:00
Christian Berendt 02d0060144 [sitemap] increase the DOWNLOAD_WARNSIZE
The default value for the DOWNLOAD_WARNSIZE setting is 33554432 (32MB).
This patch will double the value. This solves the following warning:

WARNING: Received (37889579) bytes larger than download warn size

Change-Id: I5be4bcefa6fe2eb474901bb958669978dd6e1fef
2015-10-02 10:38:45 +02:00
Christian Berendt 94d69f1868 [sitemap] resolve remaining ScrapyDeprecationWarning exceptions
* SgmlLinkExtractor is deprecated
* Module `scrapy.contrib.exporter` is deprecated

Change-Id: Ie3207a537abfd2b75602000fb4fef8f4c1e8ba56
2015-10-02 10:34:13 +02:00
Christian Berendt 8a2df7b6ef [sitemap] resolve several ScrapyDeprecationWarning exceptions
Change-Id: Ie87e6c316eecd45006332088912b31809b4c9679
2015-08-22 10:06:59 +02:00
Andreas Jaeger a21cf5f375 Remove /draft from sitemap
Do not include /draft in sitemap.

Paritally implements bp draft-publishing

Change-Id: Iddfef2933c7a397da6f084da1285716b0078ea06
2015-04-18 09:43:10 +02:00
Anne Gentle b4f43bdcaa Update sitemap generator
- Removes grizzly, adds juno
- Removes /api/ just like /trunk/ now that <service>-api repos have moved
to openstack-attic

Change-Id: I4b5dea2379d58b3ed506f70c4a9539a64f892f8e
2015-02-13 23:46:09 -06:00
Andreas Jaeger a6fbc259b6 Improve sitemap generation
Filter out any trunk links

Change-Id: Icf35370237e0209c836ed8883040bf6fe4c9a7c0
2014-06-27 14:25:14 +02:00
Christian Berendt d449a6bcd2 Bump hacking to 0.9.x series
Require at least 0.9.1 because 0.9.0. had a minor bug.

This change also fixes all found issues.

Rewrote some docstrings to fit H405.

Change-Id: Ifeef11b783dbe70b2652d2b9ea29d5b20f69f2ce
2014-06-12 09:17:22 +02:00
Christian Berendt 31c25891c1 script to generate the sitemap.xml for
This script crawls all available sites on and extracts
all URLs. Based on the URLs the script generates a sitemap for search engines
according to the protocol described at

Change-Id: Id7839d2048989da503d31e436455aea9bb4cdc1d
2014-05-29 01:29:18 +02:00