starlingx/tools
Code Issues Proposed changes

Improve generate-cgcs-centos-repo.sh efficiency

Browse Source 
The generate-cgcs-centos-repo.sh utility sets up symlinks
in the cgcs-centos-repo directory for each entry in the
defined LST files, pointing to files from a mirror. In the
processing of the LST files, a complete "find" of the
specified mirror is done for each file, looking for a match.
Given there are more than 2000 such entries, this causes
the tool to take a long time to do its job.

This update enhances the tool to do a single find upfront,
caching the result in a temporary file. During processing of
the LST files, then, the tool can grep this file to find the
corresponding file, rather than repeating the find.

In addition, this update fixes an issue with error reporting,
where a flag was used inside the processing loop to indicate
missing files. However, because the loop runs in a subshell,
this status flag is no longer set when checked outside the loop.
This is replaced with a check of the missing file report.

Change-Id: I0eb41f9da3d5afd010ade9308474e537c409f540
Closes-Bug: 1831488
Signed-off-by: Don Penney <don.penney@windriver.com>
This commit is contained in:
Don Penney 2019-06-03 14:45:28 -04:00
parent 78258f737f
commit cac209d677
1 changed files with 10 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
toCOPY/generate-cgcs-centos-repo.sh
View File

@ -59,17 +59,18 @@ for t in "Binary" "Source" ; do
fi fi
done done
unsuccessful_file=0 mirror_content=$(mktemp -t centos-repo-XXXXXX)
find ${mirror_dir} -type f > ${mirror_content}
for lst_file in ${rpm_lst_files} ; do for lst_file in ${rpm_lst_files} ; do
grep -v "^#" ${lst_file_dir}/${lst_file} | while IFS="#" read rpmname extrafields; do grep -v "^#" ${lst_file_dir}/${lst_file} | while IFS="#" read rpmname extrafields; do
if [ -z "${rpmname}" ]; then if [ -z "${rpmname}" ]; then
continue continue
fi fi
mirror_file=$(find ${mirror_dir} -name ${rpmname}) mirror_file=$(grep "/${rpmname}$" ${mirror_content})
if [ -z "${mirror_file}" ]; then if [ -z "${mirror_file}" ]; then
echo "Error -- could not find requested ${rpmname} in ${mirror_dir}" echo "Error -- could not find requested ${rpmname} in ${mirror_dir}"
echo ${rpmname} >> ${missing_rpms_file} echo ${rpmname} >> ${missing_rpms_file}
unsuccessful_file=1
continue continue
fi fi
@ -87,11 +88,11 @@ for lst_file in ${rpm_lst_files} ; do
ln -sf "${mirror_dir}/$ff" "${dest_dir}/${sub_dir}" ln -sf "${mirror_dir}/$ff" "${dest_dir}/${sub_dir}"
if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then
echo "Failed ${mirror_file}: ln -sf \"${mirror_dir}/$ff\" \"${dest_dir}/${sub_dir}\"" echo "Failed ${mirror_file}: ln -sf \"${mirror_dir}/$ff\" \"${dest_dir}/${sub_dir}\""
unsuccessful_file=1
fi fi
done done
done done
rm -f ${mirror_content}
if [ ! -f "$mock_cfg_file" ]; then if [ ! -f "$mock_cfg_file" ]; then
echo "Cannot find mock.cfg.proto file!" echo "Cannot find mock.cfg.proto file!"
@ -130,6 +131,10 @@ cat ${lst_file_dir}/${other_lst_file} | grep -v "#" | while IFS=":" read targett
done done
echo "Done creating repo directory" echo "Done creating repo directory"
if [ ${unsuccessful_file} -ne 0 ]; then declare -i missing_rpms_file_count=$(wc -l ${missing_rpms_file} 2>/dev/null | awk '{print $1}')
if [ ${missing_rpms_file_count} -gt 0 ]; then
echo "WARNING: Some targets could not be found. Your repo may be incomplete." echo "WARNING: Some targets could not be found. Your repo may be incomplete."
echo "Missing targets:"
cat ${missing_rpms_file}
exit 1
fi fi