Browse Source

download_mirror.sh fails with 404 errors

The 'yum makecache' step is failing. The issue is seen when the fast
mirror plugin for yum is in use (default).

The metadata for a yum repo as scattered across several files. The
master file is repomd.xml, and it may list several supporting files,
e.g.
   filelists.xml.gz
   primary.xml.gz
   filelists.sqlite.bz2
   other.sqlite.bz2
   other.xml.gz
   primary.sqlite.bz2

When fast mirror plugin is in use, each file might download from a
separate source. This opens the door for a race condition when a repo
update is rolling out across the mirrors.

The error suggests we have already downloaded a repomd.xml, but it is
obsolete and references a supporting file that no longer exists.

A second possible source of inconsistent data is caching proxies.

Solution:
1) Disabling the yum fast cache plugin is undesirable, and there
is no option forcing metadata iof a given repo to be single sourced.
So this update adds a limited number of retries for the
'yum makecache' command.

So far we've never seen 'yum makecache' fail twice in a row... 
largely because the 'fastest mirror' plugin rarely downloaded from the
same source twice.

2) Add 'http_caching=packages' to yum.conf, which asks upstream proxies
to not cache repodata.

Change-Id: I3fa3f61e38d868fb14f4175b87c0d323abb57def
Closes-bug: 1818911
Signed-off-by: Scott Little <scott.little@windriver.com>
changes/95/641495/2
Scott Little 2 months ago
parent
commit
cf435e6c2a
3 changed files with 21 additions and 1 deletions
  1. 2
    0
      Dockerfile
  2. 18
    1
      centos-mirror-tools/dl_rpms.sh
  3. 1
    0
      centos-mirror-tools/yum.conf.sample

+ 2
- 0
Dockerfile View File

@@ -21,6 +21,8 @@ FROM centos:7.4.1708
21 21
 #    echo -e "export http_proxy=$http_proxy\nexport https_proxy=$https_proxy\n\
22 22
 #export ftp_proxy=$ftp_proxy" >> /root/.bashrc
23 23
 
24
+RUN echo "http_caching=packages" >> /etc/yum.conf
25
+
24 26
 # username you will docker exec into the container as.
25 27
 # It should NOT be your host username so you can easily tell
26 28
 # if you are in our out of the container.

+ 18
- 1
centos-mirror-tools/dl_rpms.sh View File

@@ -251,7 +251,24 @@ download () {
251 251
 
252 252
 
253 253
 # Prime the cache
254
-${SUDOCMD} yum ${YUMCONFOPT} ${RELEASEVER} makecache
254
+loop_count=0
255
+max_loop_count=5
256
+echo "${SUDOCMD} yum ${YUMCONFOPT} ${RELEASEVER} makecache"
257
+while ! ${SUDOCMD} yum ${YUMCONFOPT} ${RELEASEVER} makecache ; do
258
+    # To protect against intermittent 404 errors, we'll retry
259
+    # a few times.  The suspected issue is pulling repodata
260
+    # from multiple source that are temporarily inconsistent.
261
+    loop_count=$((loop_count + 1))
262
+    if [ $loop_count -gt $max_loop_count ]; then
263
+        break
264
+    fi
265
+    echo "makecache retry: $loop_count"
266
+
267
+    # Wipe the inconsistent data from the last try
268
+    echo "yum ${YUMCONFOPT} ${RELEASEVER} clean all"
269
+    yum ${YUMCONFOPT} ${RELEASEVER} clean all
270
+done
271
+
255 272
 
256 273
 # Download files
257 274
 if [ -s "$rpms_list" ];then

+ 1
- 0
centos-mirror-tools/yum.conf.sample View File

@@ -13,6 +13,7 @@ bugtracker_url=http://bugs.centos.org/set_project.php?project_id=23&ref=http://b
13 13
 distroverpkg=centos-release
14 14
 override_install_langs=en_US.utf8
15 15
 tsflags=nodocs
16
+http_caching=packages
16 17
 reposdir=yum.repos.d
17 18
 
18 19
 

Loading…
Cancel
Save