mirror.starlingx.cengn.ca no longer exists. CENGN is kindly forwarding
requests to the new location mirror.starlingx.windriver.com for now, but
that will only last a few months. We need to replace all the references
with the new URL.
I will also remove as many 'cengn' references as possible, replacing them
with 'stx_mirror'
Partial-Bug: 2033555
Signed-off-by: Scott Little <scott.little@windriver.com>
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: I9d1396d603dcd3fc54bd4496fd332de278aa70da
The original code would download all verions names in our lst files,
upload them one at a time to Aptly, and as part of each upload it
would delete other versions of the package. The end result is that
the last uploaded version is the only one remaining in Aptly.
Fix is to delete old versions of a package from Aptly as a
separate step, with full knowledge of all versions we need to keep.
Testing:
1) Add multiple versions of a package to a lst file.
Run downloader. All expected versions are found
in Aptly.
2) Remove the extra version of the package.
Run downloader. The extra, no longer needed package
version are removed from Aptly leaving only the one we want.
Story: 2010797
Task: 48697
Change-Id: I6de33d8d1c2bb5161c905765e7757102ea9b8cac
Signed-off-by: Scott Little <scott.little@windriver.com>
(cherry picked from commit 9b0a1394661bc0e8c9c6f89437f5648cd5a8d532)
This commit fixes various minor issues in utils.py Python module's
run_shell_cmd function:
- Fix "local variable 'process' referenced before assignment" exception
that can occur while handling exceptions raised by subprocess.Popen().
This appears to only occur when a forked/child process cannot execute
a program, which is a programming error, which in turn is a minor
issue.
Before this commit, attempting to run a non-existent program with, for
example: 'run_shell_command("/bin/abc", ...)' would result in the
exception "local variable 'process' referenced before assignment"
while handling the exception "[Errno 2] No such file or directory:
'/bin/abc'".
With this commit, only the latter exception is reported, but in the
following form, as part of a single/non-nested exception:
[ Failed to execute command: "['/bin/abc']" Exception: "[Errno 2] \
No such file or directory: '/bin/abc'" ]
- Avoid making Python print two tracebacks when handling exceptions
raised by subprocess.Popen(). Logging a one-line description of the
exception encountered by subprocess.Popen() (via "{e}", where "e" is
an Exception object) is sufficient on its own to debug the encountered
issue. (Please see above for an example.)
- Always log standard error of the command. While most of the messages
logged to the standard error are debugging-oriented or point at minor
issues, ignoring standard error still causes us to miss non-critical
warning messages. In addition, change the log level for standard error
output from "error" to "debug" to avoid panicking build system users
due to the non-critical messages that will appear, such as the
following:
dpkg-architecture: warning: cannot determine CC system type, falling \
back to default (native compilation)
dpkg-source --before-build .
dpkg-buildpackage: warning: building a source package without \
cleaning up as you asked; it might contain undesired files
dpkg-source -b .
dpkg-genbuildinfo --build=source
dpkg-genchanges --build=source >../mlnx-ofed-kernel_....changes
dpkg-genchanges: info: not including original source code in upload
- In the logs, prefix the command's standard output with "stdout: " and
its standard error with "stderr: ". This allows distinguishing between
the command's messages emitted via standard output and standard error.
- Note that adding the "stdout: " prefix to each line of each command's
standard output increases the log file's size by about 190 KiB,
according to the following results from a fresh build with
'build-pkgs -b std,rt -c -a':
$ grep -e '2022-09-1[67] .*stdout: ' builder.log | wc -l
24391 # num_lines
$ echo $(( 24391 * 8 )) # i.e., num_lines * strlen("stdout: ")
195128 # bytes
Logging each command's standard error (along with prefixing each line
with "stderr: ") adds about 1.87 MiB of output to the build logs,
which corresponds to an increase of approximately 21 percent:
$ grep -e '2022-09-1[67] .*stderr: ' builder.log | wc -c
1963024
$ grep -e '2022-09-1[67] ' builder.log | wc -c
11164207
$ echo 'scale=2; print 1963024.0/(11164207-1963024) * 100,"%\n";' \
| bc
21.00%
Despite the increase in the log file size, logging the standard error
uncovers some issues, mostly relating to the debian/changelog file
formatting used by StarlingX developers, which are admittedly minor.
Examples include:
... stderr: dpkg-genchanges: warning: debian/changelog(l...): \
badly formatted trailer line
... stderr: dpkg-genchanges: warning: debian/changelog(l...): \
found end of file where expected more change data or trailer
- Log the return code of the command if it fails. This provides
additional information regarding why a command failed, in case the
command's output is not sufficient to determine the reason.
- Prefer to log the command as a list/tuple if it is provided with that
type, for easier correlation with the code and for consistency with
the rest of the logging in the function.
- Improve logging and reduce the duplication involved in log message
construction.
Verification
- 'build-pkgs -b rt,std -c -a' runs without issues with this patch, but
emits additional output due to logging the standard error streams of
executed commands.
Closes-Bug: 1989009
Closes-Bug: 1990177
Change-Id: I6b3c4dc70fb1404280ea15e8dcc573c83792f9b1
Signed-off-by: M. Vefa Bicakci <vefa.bicakci@windriver.com>
downloader tries to download binaries from CENGN first, if fails, then
tries to download from Debian url.
Move url_to_cengn and get_download_url to utils.py, so that
downloader can use these functions to convert an url to CENGN url.
Test Plan:
Pass: downloader -b
Pass: See the url is converted to CENGN url in log.
Closes-bug: 1986961
Signed-off-by: Yue Tao <Yue.Tao@windriver.com>
Change-Id: Ic5cbdc70535d5f1f68362cc009d8007afb7b8a35
When using a download URL that includes the character '&' (as part of a
query string), the following build failure is encountered, because the
'&' character was is interpreted by the shell as a special character for
backgrounding the 'wget' command.
debrepack - ERROR: 2022-06-29 20:04:31 (3.79 MB/s) - \
'...' saved [...]
debrepack - ERROR: [ Failed - "wget -t 5 --wait=15 \
http://a_url_with_& -O ..." ]
This issue is resolved by not using the shell for spawning download
commands. (That is, by setting "shell=False" in Python's
subprocess.Popen constructor.) This commit implements an in-place
solution that detects the type of the argument and sets the "shell="
keyword argument to subprocess.Popen accordingly.
Verification:
- Downloading from URLs with '&' characters works as expected as part of
the "debrepack" phase of build-pkgs.
- No negative behaviour observed with a build from scratch.
Closes-Bug: 1988343
Change-Id: I2a529f60b9a57b6d139f95d31bcd18c51a0fbecb
Signed-off-by: M. Vefa Bicakci <vefa.bicakci@windriver.com>
Move run_shell_cmd to utils.py, so that it can be called by other
modules.
Test Plan:
Pass: build-pkgs -c -a
Signed-off-by: Yue Tao <yue.tao@windriver.com>
Change-Id: Ib114fe498cc0303846099f803f79b280aef3f64b
Add timestamps to logs (console and file), updates debrepack
to use utils.py for logging and removes color from file handler
builder.log.
Test Plan:
Pass: build-pkgs -a
Pass: build-image
Pass: repo_manage list
Pass: downloader -s -b
Story: 2008846
Task: 45086
Signed-off-by: Luis Sampaio <luis.sampaio@windriver.com>
Change-Id: If2d76d0d1a6ddd47b6ecc8ed72f9adecbcd354b5
Remove hard coded lists of build types and layers.
Discover all buildable layers and build types by
scanning the source code base for config files.
Discover the build order of layers and build types from
config files
Allow options to reduce the build set to specific
subset of layers or build-types
Story: 2008862
Task: 43154
Depends-On: https://review.opendev.org/c/starlingx/root/+/832145
Depends-On: https://review.opendev.org/c/starlingx/tools/+/832704
Signed-off-by: Scott Little <scott.little@windriver.com>
Signed-off-by: hbai <haiqing.bai@windriver.com>
Change-Id: I3e6ab41ed79877a926f3adc25c4058436bbccc17
Remove hard coded lists of build types and layers.
Discover all buildable layers and build types by
scanning the source code base for config files.
Discover the build order of layers and build types from
config files
Allow options to reduce the build set to specific
subset of layers or build-types
Story: 2008862
Task: 43154
Depends-On: https://review.opendev.org/c/starlingx/root/+/832145
Depends-On: https://review.opendev.org/c/starlingx/tools/+/832704
Change-Id: I3eae4888e3d3e9d65a4bcd1410bfc2d4cb6c31d4
Signed-off-by: Scott Little <scott.little@windriver.com>
Create the initial script 'build-pkgs' which helps to launch
the building of debian packages in build container.
The depend function modules 'dsccache.py', 'debsentry.py' and
'utils.py' are also created.
Story: 2008846
Task: 43120
Signed-off-by: hbai <haiqing.bai@windriver.com>
Change-Id: If1924e72aa2a5563da1c00ac328ca1e560c65cbd