starlingx/tools
Code Issues Proposed changes
2048253ec7
tools/centos-mirror-tools/utils_tests.sh
Davlet Panech ac49ff342c use curl + avoid partial downloads 
Mirror scripts sometimes leave corrupted/partial files behind.

Problems
========

1) wget is called with the -O flag, and the server returns an HTTP
error for the requested URL (404 etc). Wget leaves a zero-length file
behind. This doesn't seem to happen without the -O flag.

2) wget starts the download which stalls & times out half-way; wget
gives up and requests the same file with a byte offset of the form
"Range: bytes=1234-", and the web server doesn't support open-ended
ranges. In this case wget prints out a warning, leaves a partial file
behind and returns success.

3) Sites like GitHub generate repo tarballs on the fly, eg:
https://github.com/kubernetes/kubernetes/archive/refs/tags/v1.19.3.tar.gz
Since tags can move, downloading such a file twice may result in a
different file. Therefore HTTP "resume download" may corrupt files in
this case.

4) Git "keyword expansion" feature may result in differences in source
files being downloaded. For example, this file:

  https://github.com/kubernetes/kubernetes/blob/v1.19.3/staging/src/k8s.io/component-base/version/base.go

contains lines similar to:

  gitVersion  = "v0.0.0-master+$Format:%h$"

where %h is replaced with a short SHA when the tar file is
exported/downloaded.  How short the SHA is depends on git history and
sometimes results in shortened SHAs of different lengths. So
downloading that file may result in different files.

Therefore HTTP "Range" header may corrupt files in this case as
well.

5) Curl is invoked with the "--retry" option and starts the download;
connection stalls; curl gives up, connects again, skips the 1st N
bytes and appends to the partial file. If the file changes while we
are doing this, it will end up corrupting the file. This is very
unlikely to happen and I haven't been able to reproduce this case.

Problems with HTTP Range header
===============================
Curl/wget "resume/continue download" feature has no way of verifying
whether the partial file on disk, and the one being re-requested, are in
fact the same file.  If the file changes on the server between
downloads, "resume download" will corrupt it.

Some web servers don't support this at all, which triggers case (2)
with wget.

Some web servers support the Range header, but require that the end
byte position is present. This is not compatible with wget & curl.
For example curl & wget add headers similar to: "Range: bytes=1234-"
means give me the file starting at offset 1234 and till EOF. This also
triggers case (2).

This patch
==========

* Always download the file to a temporary name, then rename into place

* Use curl instead wget (better error handling). The only exception is
"recursive downloads", which curl doesn't support.

Bug: https://bugs.launchpad.net/starlingx/+bug/1950017
Change-Id: Iaa89009ce23efe5b73ecb8163556ce6db932028b
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
2021-11-10 14:25:47 -05:00

93 lines
2.9 KiB
Bash
Raw Blame History

#!/bin/bash
#
# SPDX-License-Identifier: Apache-2.0
#
# Copyright (C) 2019 Intel Corporation
#
# Set of unit tests for dl_rpms.sh
set -o errexit
set -o nounset
YUMCONFOPT=""
RELEASEVER="--releasever=7"
source utils.sh
check_result() {
local _res="$1"
local _expect="$2"
if [[ "$_res" != $_expect ]]; then
echo "Fail"
echo "expected $_expect"
echo "returned $_res"
exit 1
fi
echo "Success"
}
# get_download_file_command
res=$(get_download_file_command "https://libvirt.org/sources/python/libvirt-python-3.5.0-1.fc24.src.rpm")
expect="curl* https://libvirt.org/sources/python/libvirt-python-3.5.0-1.fc24.src.rpm"
check_result "$res" "$expect"
res=$(get_download_file_command --quiet "python2-httpbin-0.5.0-6.el7.noarch.rpm")
expect="curl*--silent* https://kojipkgs.fedoraproject.org/packages/python2-httpbin/0.5.0/6.el7/noarch/python2-httpbin-0.5.0-6.el7.noarch.rpm"
check_result "$res" "$expect"
# get_url
res=$(get_url "acpid-2.0.19-9.el7.x86_64.rpm" "L1")
expect="https://vault.centos.org/centos/7.4.1708/cr/x86_64/Packages/acpid-2.0.19-9.el7.x86_64.rpm"
check_result "$res" "$expect"
res=$(get_url "python2-httpbin-0.5.0-6.el7.noarch.rpm#http://cbs.centos.org/kojifiles/packages/python-httpbin/0.5.0/6.el7/noarch/python2-httpbin-0.5.0-6.el7.noarch.rpm" "L1")
expect="http://cbs.centos.org/kojifiles/packages/python-httpbin/0.5.0/6.el7/noarch/python2-httpbin-0.5.0-6.el7.noarch.rpm"
check_result "$res" "$expect"
res=$(get_url "python2-httpbin-0.5.0-6.el7.noarch.rpm" "K1")
expect="https://kojipkgs.fedoraproject.org/packages/python2-httpbin/0.5.0/6.el7/noarch/python2-httpbin-0.5.0-6.el7.noarch.rpm"
check_result "$res" "$expect"
# get_yum_command
res=$(get_yum_command "anaconda-21.48.22.147-1.el7.centos.src.rpm" "L1")
expect="yumdownloader -q -C --releasever=7 --source anaconda-21.48.22.147-1.el7.centos"
check_result "$res" "$expect"
res=$(get_yum_command "acpid-2.0.19-9.el7.x86_64.rpm" "L1")
expect="yumdownloader -q -C --releasever=7 --archlist=noarch,x86_64 acpid-2.0.19-9.el7"
check_result "$res" "$expect"
# get_rpm_level_name
res=$(get_rpm_level_name "acl-2.2.51-12.el7.x86_64.rpm" "L1")
expect="acl-2.2.51-12.el7"
check_result "$res" "$expect"
res=$(get_rpm_level_name "acl-2.2.51-12.el7.x86_64.rpm" "L3")
expect="acl"
check_result "$res" "$expect"
res=$(get_rpm_level_name "anaconda-21.48.22.147-1.el7.centos.src.rpm" "L2")
expect="anaconda-21.48.22.147"
check_result "$res" "$expect"
res=$(get_arch_from_rpm "acl-2.2.51-12.el7.x86_64.rpm")
expect="x86_64"
check_result "$res" "$expect"
res=$(get_arch_from_rpm "acl-2.2.51-12.el7.noarch.rpm")
expect="noarch"
check_result "$res" "$expect"
res=$(get_arch_from_rpm "acl-2.2.51-12.el7.src.rpm")
expect="src"
check_result "$res" "$expect"
res=$(get_arch_from_rpm "acl-2.2.51-12.el7.src.rpm#https://someurl.com/acl-2.2.51-12.el7.src.rpm")
expect="src"
check_result "$res" "$expect"
View Git Blame Copy Permalink