openstack/project-config
Code Issues Proposed changes

selective caching of git repos

Browse Source 
Adding the ability to selectively cache only some git repos in
nodepool slave.
This is useful in some environments where it is needless to cache
all the git repos that appear in gerrit.
For example, in a 3rd party CI environment, each nodepool slave
needs only few openstack projects in order to check commits to
a specific project. So it is needless to cache > 1500 projects
currently in gerrit

Change-Id: Ie7302594b24d2bdcc157d3cf64b1f219e7ef0205
This commit is contained in:
Isaac Beckman 2016-05-29 13:36:17 +03:00
parent 3860fdfb9c
commit 918a386596
2 changed files with 36 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
nodepool/elements/openstack-repos/README.rst
View File

@ -1 +1,19 @@
Download all repos and packages that a devstack run might need.
===============
openstack-repos
===============
Download all repos and packages that might be needed.
Environment variables:
----------------------
DIB_CUSTOM_PROJECTS_LIST_URL
:Required: No
:Default: None
:Description: Url to a yaml file contains custom list of repos.
The custom yaml file has the same structure as the default file:
'https://git.openstack.org/cgit/openstack-infra/project-config/plain/gerrit/projects.yaml'
Download only the repos that appear in the custom file rather than
downloading all openstack repos that appear in the default file.
:Example:
DIB_CUSTOM_PROJECTS_LIST_URL='file:///etc//project-config//gerrit//custom_projects.yaml'

18
nodepool/elements/openstack-repos/extra-data.d/50-create-repo-list
View File

@ -18,6 +18,7 @@
import os
import urllib2
from urllib2 import URLError
import yaml
URL = ('https://git.openstack.org/cgit/openstack-infra/project-config/'
@ -27,9 +28,24 @@ TMP_HOOKS_PATH=os.environ['TMP_HOOKS_PATH']
PROJECTS_REPOS=os.path.join(TMP_HOOKS_PATH, 'source-repository-projects-yaml')
GIT_BASE=os.environ.get('GIT_BASE', 'git://git.openstack.org')
CUSTOM_PROJECTS_LIST_URL=os.environ.get('DIB_CUSTOM_PROJECTS_LIST_URL')
def get_project_list(url):
try:
projects = [f['project'] for f in yaml.load(urllib2.urlopen(url))]
return projects
except URLError:
print "Could not open project list url: '%s'" % url
return None
def main():
projects = [f['project'] for f in yaml.load(urllib2.urlopen(URL))]
projects = []
if CUSTOM_PROJECTS_LIST_URL:
projects = get_project_list(CUSTOM_PROJECTS_LIST_URL)
if not projects:
projects = get_project_list(URL)
with open(PROJECTS_REPOS, 'w') as projects_list:
for project in projects:
# Skip repos that are inactive