Statically cache devstack images and packages
We had been running a script to generate a list of things to cache for devstack. Unfortunately, we've discovered that this attempts to perform unsafe actions which illustrates that this is unsafe (and creates unnecessary relationship between opendev images and openstack/devstack). Address this by providing a static list of things to cache. Note this does not do anything for arm64 images (that will need to be addressed in a follow on but they are largely not running devstack there yet). On a Bionic node this is what we have in /opt/cache/files/: cirros-0.3.2-i386-disk.vmdk cirros-0.3.4-x86_64-disk.img cirros-0.3.4-x86_64-disk.vhd.tgz cirros-0.3.4-x86_64-uec.tar.gz cirros-0.3.5-x86_64-disk.img cirros-0.3.5-x86_64-disk.vhd.tgz cirros-0.3.5-x86_64-uec.tar.gz cirros-0.4.0-x86_64-disk.img cirros-0.4.0-x86_64-uec.tar.gz etcd-v3.1.10-linux-amd64.tar.gz etcd-v3.2.17-linux-amd64.tar.gz etcd-v3.3.12-linux-amd64.tar.gz get-pip.py stackviz-latest.tar.gz zanata-cli-4.3.3-dist.tar.gz I've trimmed out the vmdk, vhd, and tarball based images as we should all be using qcow2s. Everything under etcd is provided by preexisting static lists. Change-Id: Iff741e8ed4c517ccabae6e6d6ba730f0aa37a272
This commit is contained in:
Clark Boylan
parent
03edecbaf5
commit
9e1c44d9a4
|
|
@ -1,88 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Copyright (c) 2012-2014 Hewlett-Packard Development Company, L.P.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
# implied.
|
||||
#
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# dib-lint: disable=setpipefail
|
||||
if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
|
||||
set -x
|
||||
fi
|
||||
set -eu
|
||||
|
||||
# We need a copy of the devstack repo to query for the list of image
|
||||
# files to cached by the source-repositories element (see
|
||||
# 55-cache-devstack-repos)
|
||||
#
|
||||
# This script is a hack to use the source-repositories repo cache to
|
||||
# get early access to the devstack checkout (rather than doing a whole
|
||||
# separate clone of the devstack tree). In essence, this script knows
|
||||
# the path of the devstack in the source-repositories cache and
|
||||
# creates/updates it. To get the gist of where this comes from, see
|
||||
# dib's
|
||||
# elements/source-repositories/extra-data.d/98-source-repositories
|
||||
#
|
||||
# This could be removed if we could ever source and call the
|
||||
# clone/update functions asynchronously from the source-repositories
|
||||
# dib element.
|
||||
|
||||
function get_devstack_from_cache {
|
||||
|
||||
local REPONAME=devstack
|
||||
local REPOTYPE=git
|
||||
local REPOPATH=/opt/git/openstack/devstack
|
||||
local REPOLOCATION=https://opendev.org/openstack/devstack.git
|
||||
local REPOREF=master
|
||||
|
||||
local REPO_DEST=$TMP_MOUNT_PATH$REPOPATH
|
||||
local REPO_SUB_DIRECTORY=$(dirname $REPO_DEST)
|
||||
sudo mkdir -p $REPO_SUB_DIRECTORY # dib-lint: safe_sudo
|
||||
|
||||
# build the same cache name as source-repositories
|
||||
CACHE_NAME=$(echo "${REPOTYPE}_${REPOLOCATION}" | sha1sum | awk '{ print $1 }' )
|
||||
CACHE_NAME=${REPONAME//[^A-Za-z0-9]/_}_${CACHE_NAME}
|
||||
CACHE_PATH=${CACHE_BASE}/$CACHE_NAME
|
||||
|
||||
# clone repo if not in cache
|
||||
if [ ! -e "$CACHE_PATH" ] ; then
|
||||
echo "early-source-repo: Caching $REPONAME from $REPOLOCATION in $CACHE_PATH"
|
||||
git clone $REPOLOCATION $CACHE_PATH.tmp
|
||||
mv ${CACHE_PATH}{.tmp,}
|
||||
fi
|
||||
|
||||
# update cached repo
|
||||
if [ -z "$DIB_OFFLINE" ] ; then
|
||||
echo "early-source-repo: Updating cache of $REPOLOCATION in $CACHE_PATH with ref $REPOREF"
|
||||
# note, we could update *all* refs here (+refs/*:refs/* and
|
||||
# create a really full mirror. but that brings in a lot of
|
||||
# change refs from gerrit which traditionally we haven't used.
|
||||
# Also, we only really care about refs/remotes/* as the
|
||||
# devstack caching script goes through these branches to
|
||||
# decide what to checkout. We keep refs/heads/* for
|
||||
# historical reasons...)
|
||||
git --git-dir=$CACHE_PATH/.git fetch --prune --update-head-ok \
|
||||
$REPOLOCATION \
|
||||
+refs/heads/*:refs/heads/* +refs/remotes/*:refs/remotes/*
|
||||
fi
|
||||
|
||||
# clone the updated repo to our destination
|
||||
sudo git clone $CACHE_PATH $REPO_DEST # dib-lint: safe_sudo
|
||||
}
|
||||
|
||||
# in case source-repositories hasn't run yet, make the cache
|
||||
CACHE_BASE=$DIB_IMAGE_CACHE/source-repositories
|
||||
mkdir -p $CACHE_BASE
|
||||
|
||||
get_devstack_from_cache
|
||||
|
|
@ -1,149 +0,0 @@
|
|||
#!/usr/bin/env python
|
||||
|
||||
# Copyright (C) 2011-2013 OpenStack Foundation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
# implied.
|
||||
#
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from __future__ import print_function
|
||||
from __future__ import unicode_literals
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
|
||||
TMP_MOUNT_PATH = os.environ['TMP_MOUNT_PATH']
|
||||
TMP_HOOKS_PATH = os.environ['TMP_HOOKS_PATH']
|
||||
|
||||
DEVSTACK = os.path.join(TMP_MOUNT_PATH, 'opt/git/openstack/devstack')
|
||||
CACHEDIR = os.path.join(TMP_MOUNT_PATH, 'tmp')
|
||||
IMAGES = os.path.join(TMP_HOOKS_PATH, 'source-repository-images')
|
||||
|
||||
|
||||
def run_local(cmd, status=False, cwd='.', env={}):
|
||||
print("Running:", cmd)
|
||||
newenv = os.environ
|
||||
newenv.update(env)
|
||||
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, cwd=cwd,
|
||||
stderr=subprocess.STDOUT, env=newenv,
|
||||
universal_newlines=True)
|
||||
(out, nothing) = p.communicate()
|
||||
if status:
|
||||
return (p.returncode, out.strip())
|
||||
return out.strip()
|
||||
|
||||
|
||||
def git_branches():
|
||||
branches = []
|
||||
for branch in run_local(['git', 'branch', '-a'], cwd=DEVSTACK).split("\n"):
|
||||
branch = branch.strip()
|
||||
if not branch.startswith('remotes/origin'):
|
||||
continue
|
||||
branches.append(branch)
|
||||
return branches
|
||||
|
||||
|
||||
def _legacy_find_images(basedir):
|
||||
"""Divine what images we should use based on parsing stackrc."""
|
||||
images = []
|
||||
for line in open(os.path.join(basedir, 'stackrc')):
|
||||
line = line.strip()
|
||||
if line.startswith('IMAGE_URLS'):
|
||||
if '#' in line:
|
||||
line = line[:line.find('#')]
|
||||
if line.endswith(';;'):
|
||||
line = line[:-2]
|
||||
line = line.split('=', 1)[1].strip()
|
||||
if line.startswith('${IMAGE_URLS:-'):
|
||||
line = line[len('${IMAGE_URLS:-'):]
|
||||
if line.endswith('}'):
|
||||
line = line[:-1]
|
||||
if not line:
|
||||
continue
|
||||
if line[0] == line[-1] == '"':
|
||||
line = line[1:-1]
|
||||
# Add image to the list to be downloaded, but
|
||||
# skip downloading giant vmware images
|
||||
images += [x.strip() for x in line.split(',')
|
||||
if not x.strip().endswith('vmdk')]
|
||||
return images
|
||||
|
||||
|
||||
def _find_images(basedir):
|
||||
images = []
|
||||
image_tool = os.path.join(basedir, 'tools', 'image_list.sh')
|
||||
if os.path.exists(image_tool):
|
||||
returncode, out = run_local(image_tool, status=True)
|
||||
if returncode:
|
||||
print("%s failed" % image_tool)
|
||||
print("Exit: %s, Output: %s" % (returncode, out))
|
||||
# reset images so we'll fall back
|
||||
images = []
|
||||
else:
|
||||
images = out.split('\n')
|
||||
return images
|
||||
|
||||
|
||||
def local_prep():
|
||||
branches = []
|
||||
for branch in git_branches():
|
||||
# Ignore branches of the form 'somestring -> someotherstring'
|
||||
# as this denotes a symbolic reference and the entire string
|
||||
# as is cannot be checked out. We can do this safely as the
|
||||
# reference will refer to one of the other branches returned
|
||||
# by git_branches.
|
||||
if ' -> ' in branch:
|
||||
continue
|
||||
branch_data = {'name': branch}
|
||||
print('Branch: ', branch)
|
||||
run_local(['sudo', 'git', 'checkout', branch], cwd=DEVSTACK)
|
||||
run_local(['sudo', 'git', 'pull', '--ff-only', 'origin'], cwd=DEVSTACK)
|
||||
|
||||
images = _find_images(DEVSTACK)
|
||||
if not images:
|
||||
images = _legacy_find_images(DEVSTACK)
|
||||
|
||||
branch_data['images'] = images
|
||||
branches.append(branch_data)
|
||||
return branches
|
||||
|
||||
|
||||
def main():
|
||||
|
||||
branches = local_prep()
|
||||
|
||||
image_filenames = []
|
||||
line_template = "%(name)s file %(location)s %(url)s\n"
|
||||
with open(IMAGES, 'w') as images_list:
|
||||
for branch_data in branches:
|
||||
for url in branch_data['images']:
|
||||
fname = url.split('/')[-1].strip()
|
||||
if fname == '':
|
||||
continue
|
||||
# TODO: There is an image in devstack that starts with
|
||||
# $HEAT_FETCHED_TEST_IMAGE
|
||||
if fname.startswith('$'):
|
||||
continue
|
||||
if fname in image_filenames:
|
||||
continue
|
||||
image_filenames.append(fname)
|
||||
args = dict(
|
||||
name=fname,
|
||||
location=os.path.join('/opt/cache/files', fname),
|
||||
url=url)
|
||||
|
||||
images_list.write(line_template % args)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Copyright (c) 2014 Hewlett-Packard Development Company, L.P.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
# implied.
|
||||
#
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Remove the copy of devstack that we cloned in for getting devstack
|
||||
# cache data so that the later cloning of all of the OpenStack repos
|
||||
# won't bomb out
|
||||
|
||||
# dib-lint: disable=setpipefail
|
||||
if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then
|
||||
set -x
|
||||
fi
|
||||
set -eu
|
||||
|
||||
sudo rm -rf $TMP_MOUNT_PATH/opt/git/openstack/devstack
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
cirros-0.3.4-x86_64-disk.img file /opt/cache/files/cirros-0.3.4-x86_64-disk.img https://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img
|
||||
cirros-0.3.5-x86_64-disk.img file /opt/cache/files/cirros-0.3.5-x86_64-disk.img https://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img
|
||||
cirros-0.4.0-x86_64-disk.img file /opt/cache/files/cirros-0.4.0-x86_64-disk.img https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
|
||||
etcd-v3.1.10-linux-amd64.tar.gz file /opt/cache/files/etcd-v3.1.10-linux-amd64.tar.gz https://github.com/etcd-io/etcd/releases/download/v3.1.10/etcd-v3.1.10-linux-amd64.tar.gz
|
||||
etcd-v3.2.17-linux-amd64.tar.gz file /opt/cache/files/etcd-v3.2.17-linux-amd64.tar.gz https://github.com/etcd-io/etcd/releases/download/v3.2.17/etcd-v3.2.17-linux-amd64.tar.gz
|
||||
etcd-v3.3.12-linux-amd64.tar.gz file /opt/cache/files/etcd-v3.3.12-linux-amd64.tar.gz https://github.com/etcd-io/etcd/releases/download/v3.3.12/etcd-v3.3.12-linux-amd64.tar.gz
|
||||
Loading…
Reference in New Issue