openstack/devstack
Code Issues Proposed changes
18ce57e55c
devstack/tools/generate-devstack-plugins-list.py

62 lines
1.9 KiB
Python
Raw Normal View History

Generate plugins list through https requests The devstack plugins list can be generated through web requests in environments (such as the proposal slave) that lack copies of all the relevant git repositories. One downside to this is that there is no way of getting the last modification time of the plugin. Change-Id: I2c5c9282a8ad80014cad171a4dfbdc8f26044cd1
2016-02-18 14:46:35 -05:00
#! /usr/bin/env python
# Copyright 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
# This script is intended to be run as part of a periodic proposal bot
# job in OpenStack infrastructure.
#
# In order to function correctly, the environment in which the
# script runs must have
# * network access to the review.openstack.org Gerrit API
# working directory
# * network access to https://git.openstack.org/cgit
Plugin autogen list: handle variable-width plugin names We've had a couple of cases where plugin names are longer than our table width. Take the fixed-with table-header out of the header file, and generate it dynamically based on first-column width. To simplify, take advantage that RST allows a variable-length last column and so don't specify it's width. Add a link to the cgit URL for each project you can click on to browse the source (link text remains the git:// URL). Add some logging so you can see what the python generator is doing, should you run it. Change-Id: I5d5e692039bbb30b2508119412472dac1d105c08
2016-03-21 13:03:34 +11:00
import logging
Generate plugins list through https requests The devstack plugins list can be generated through web requests in environments (such as the proposal slave) that lack copies of all the relevant git repositories. One downside to this is that there is no way of getting the last modification time of the plugin. Change-Id: I2c5c9282a8ad80014cad171a4dfbdc8f26044cd1
2016-02-18 14:46:35 -05:00
import json
import requests
Plugin autogen list: handle variable-width plugin names We've had a couple of cases where plugin names are longer than our table width. Take the fixed-with table-header out of the header file, and generate it dynamically based on first-column width. To simplify, take advantage that RST allows a variable-length last column and so don't specify it's width. Add a link to the cgit URL for each project you can click on to browse the source (link text remains the git:// URL). Add some logging so you can see what the python generator is doing, should you run it. Change-Id: I5d5e692039bbb30b2508119412472dac1d105c08
2016-03-21 13:03:34 +11:00
logging.basicConfig(level=logging.DEBUG)
Generate plugins list through https requests The devstack plugins list can be generated through web requests in environments (such as the proposal slave) that lack copies of all the relevant git repositories. One downside to this is that there is no way of getting the last modification time of the plugin. Change-Id: I2c5c9282a8ad80014cad171a4dfbdc8f26044cd1
2016-02-18 14:46:35 -05:00
url = 'https://review.openstack.org/projects/'
# This is what a project looks like
'''
"openstack-attic/akanda": {
"id": "openstack-attic%2Fakanda",
"state": "READ_ONLY"
},
'''
def is_in_openstack_namespace(proj):
Plugin autogen list: handle variable-width plugin names We've had a couple of cases where plugin names are longer than our table width. Take the fixed-with table-header out of the header file, and generate it dynamically based on first-column width. To simplify, take advantage that RST allows a variable-length last column and so don't specify it's width. Add a link to the cgit URL for each project you can click on to browse the source (link text remains the git:// URL). Add some logging so you can see what the python generator is doing, should you run it. Change-Id: I5d5e692039bbb30b2508119412472dac1d105c08
2016-03-21 13:03:34 +11:00
# only interested in openstack namespace (e.g. not retired
# stackforge, etc)
Generate plugins list through https requests The devstack plugins list can be generated through web requests in environments (such as the proposal slave) that lack copies of all the relevant git repositories. One downside to this is that there is no way of getting the last modification time of the plugin. Change-Id: I2c5c9282a8ad80014cad171a4dfbdc8f26044cd1
2016-02-18 14:46:35 -05:00
return proj.startswith('openstack/')
Fix return code detection in plugin list generation As can be seen in logs of the periodic generation job, our cgit does a weird thing where sometimes it returns a 404 page with content, and sometimes a zero response (see [1] for example, the last number is response size). This appears to be an openstack CI issue; possibly due to cgit caching or similar (see [2] for manual test). It will have to be investigated with the host apache logs. This is resulting in a lot of projects incorrectly being picked up as having plugins (I7116571d2a2b1fc3a61e5f1ed46ac2cbc244775a). I'm not sure if this problem is also releated to the original status-code issues mentioned in the code, but testing shows that cgit is correctly returning 404's for missing files (you can see in the logs [1]). Thus switch the logic to examine the return code which avoids this issue. [1] http://logs.openstack.org/periodic/propose-devstack-plugins-list/e55790c/console.html.gz#_2016-05-04_06_46_51_660 [2] http://paste.openstack.org/show/496434/ Change-Id: I6a06347d91d091441f6f7b70f99aba6d8e9add4b
2016-05-09 13:19:09 +10:00
# Check if this project has a plugin file
Generate plugins list through https requests The devstack plugins list can be generated through web requests in environments (such as the proposal slave) that lack copies of all the relevant git repositories. One downside to this is that there is no way of getting the last modification time of the plugin. Change-Id: I2c5c9282a8ad80014cad171a4dfbdc8f26044cd1
2016-02-18 14:46:35 -05:00
def has_devstack_plugin(proj):
r = requests.get("https://git.openstack.org/cgit/%s/plain/devstack/plugin.sh" % proj)
Fix return code detection in plugin list generation As can be seen in logs of the periodic generation job, our cgit does a weird thing where sometimes it returns a 404 page with content, and sometimes a zero response (see [1] for example, the last number is response size). This appears to be an openstack CI issue; possibly due to cgit caching or similar (see [2] for manual test). It will have to be investigated with the host apache logs. This is resulting in a lot of projects incorrectly being picked up as having plugins (I7116571d2a2b1fc3a61e5f1ed46ac2cbc244775a). I'm not sure if this problem is also releated to the original status-code issues mentioned in the code, but testing shows that cgit is correctly returning 404's for missing files (you can see in the logs [1]). Thus switch the logic to examine the return code which avoids this issue. [1] http://logs.openstack.org/periodic/propose-devstack-plugins-list/e55790c/console.html.gz#_2016-05-04_06_46_51_660 [2] http://paste.openstack.org/show/496434/ Change-Id: I6a06347d91d091441f6f7b70f99aba6d8e9add4b
2016-05-09 13:19:09 +10:00
return r.status_code == 200
Generate plugins list through https requests The devstack plugins list can be generated through web requests in environments (such as the proposal slave) that lack copies of all the relevant git repositories. One downside to this is that there is no way of getting the last modification time of the plugin. Change-Id: I2c5c9282a8ad80014cad171a4dfbdc8f26044cd1
2016-02-18 14:46:35 -05:00
Plugin autogen list: handle variable-width plugin names We've had a couple of cases where plugin names are longer than our table width. Take the fixed-with table-header out of the header file, and generate it dynamically based on first-column width. To simplify, take advantage that RST allows a variable-length last column and so don't specify it's width. Add a link to the cgit URL for each project you can click on to browse the source (link text remains the git:// URL). Add some logging so you can see what the python generator is doing, should you run it. Change-Id: I5d5e692039bbb30b2508119412472dac1d105c08
2016-03-21 13:03:34 +11:00
logging.debug("Getting project list from %s" % url)
Generate plugins list through https requests The devstack plugins list can be generated through web requests in environments (such as the proposal slave) that lack copies of all the relevant git repositories. One downside to this is that there is no way of getting the last modification time of the plugin. Change-Id: I2c5c9282a8ad80014cad171a4dfbdc8f26044cd1
2016-02-18 14:46:35 -05:00
r = requests.get(url)
projects = sorted(filter(is_in_openstack_namespace, json.loads(r.text[4:])))
Plugin autogen list: handle variable-width plugin names We've had a couple of cases where plugin names are longer than our table width. Take the fixed-with table-header out of the header file, and generate it dynamically based on first-column width. To simplify, take advantage that RST allows a variable-length last column and so don't specify it's width. Add a link to the cgit URL for each project you can click on to browse the source (link text remains the git:// URL). Add some logging so you can see what the python generator is doing, should you run it. Change-Id: I5d5e692039bbb30b2508119412472dac1d105c08
2016-03-21 13:03:34 +11:00
logging.debug("Found %d projects" % len(projects))
Generate plugins list through https requests The devstack plugins list can be generated through web requests in environments (such as the proposal slave) that lack copies of all the relevant git repositories. One downside to this is that there is no way of getting the last modification time of the plugin. Change-Id: I2c5c9282a8ad80014cad171a4dfbdc8f26044cd1
2016-02-18 14:46:35 -05:00
found_plugins = filter(has_devstack_plugin, projects)
for project in found_plugins:
Plugin autogen list: handle variable-width plugin names We've had a couple of cases where plugin names are longer than our table width. Take the fixed-with table-header out of the header file, and generate it dynamically based on first-column width. To simplify, take advantage that RST allows a variable-length last column and so don't specify it's width. Add a link to the cgit URL for each project you can click on to browse the source (link text remains the git:// URL). Add some logging so you can see what the python generator is doing, should you run it. Change-Id: I5d5e692039bbb30b2508119412472dac1d105c08
2016-03-21 13:03:34 +11:00
# strip of openstack/
print statment has become a function in py3 Change-Id: Ia616e1fdd4c3fc8e1a36b026e0c3e2a568dc6517
2016-04-05 16:18:56 +03:00
print(project[10:])
Copy Permalink