12 Commits

Author SHA1 Message Date
Jackie Huang
d9705916d1 iutils.py: fix typo for upstream url
'urstx_mirror' doesn't exist and was introduced
in previous commit.

Closes-bug: 2044987

Change-Id: I020acd6c8fb0c0aab0911542f8f1af4d27be9b2c
Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
2023-12-08 16:00:59 +08:00
Scott Little
130c43619b cengn reference removal
mirror.starlingx.cengn.ca no longer exists. CENGN is kindly forwarding
requests to the new location mirror.starlingx.windriver.com for now, but
that will only last a few months. We need to replace all the references
with the new URL.

I will also remove as many 'cengn' references as possible, replacing them
with 'stx_mirror'

Partial-Bug: 2033555
Signed-off-by: Scott Little <scott.little@windriver.com>
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Change-Id: I9d1396d603dcd3fc54bd4496fd332de278aa70da
2023-11-07 17:06:46 -05:00
Scott Little
ef7fa75417 Allow our repositories to have multiple versions of a package
The original code would download all verions names in our lst files,
upload them one at a time to Aptly, and as part of each upload it
would delete other versions of the package.  The end result is that
the last uploaded version is the only one remaining in Aptly.

Fix is to delete old versions of a package from Aptly as a
separate step, with full knowledge of all versions we need to keep.

Testing:

1) Add multiple versions of a package to a lst file.
Run downloader.  All expected versions are found
in Aptly.

2) Remove the extra version of the package.
Run downloader.  The extra, no longer needed package
version are removed from Aptly leaving only the one we want.

Story: 2010797
Task: 48697

Change-Id: I6de33d8d1c2bb5161c905765e7757102ea9b8cac
Signed-off-by: Scott Little <scott.little@windriver.com>
(cherry picked from commit 9b0a1394661bc0e8c9c6f89437f5648cd5a8d532)
2023-10-05 00:41:20 +00:00
M. Vefa Bicakci
818830b25e Debian/stx: utils: Minor run_shell_cmd fix-ups
This commit fixes various minor issues in utils.py Python module's
run_shell_cmd function:

- Fix "local variable 'process' referenced before assignment" exception
  that can occur while handling exceptions raised by subprocess.Popen().
  This appears to only occur when a forked/child process cannot execute
  a program, which is a programming error, which in turn is a minor
  issue.

  Before this commit, attempting to run a non-existent program with, for
  example: 'run_shell_command("/bin/abc", ...)' would result in the
  exception "local variable 'process' referenced before assignment"
  while handling the exception "[Errno 2] No such file or directory:
  '/bin/abc'".

  With this commit, only the latter exception is reported, but in the
  following form, as part of a single/non-nested exception:

    [ Failed to execute command: "['/bin/abc']" Exception: "[Errno 2] \
      No such file or directory: '/bin/abc'" ]

- Avoid making Python print two tracebacks when handling exceptions
  raised by subprocess.Popen(). Logging a one-line description of the
  exception encountered by subprocess.Popen() (via "{e}", where "e" is
  an Exception object) is sufficient on its own to debug the encountered
  issue. (Please see above for an example.)

- Always log standard error of the command. While most of the messages
  logged to the standard error are debugging-oriented or point at minor
  issues, ignoring standard error still causes us to miss non-critical
  warning messages. In addition, change the log level for standard error
  output from "error" to "debug" to avoid panicking build system users
  due to the non-critical messages that will appear, such as the
  following:

  dpkg-architecture: warning: cannot determine CC system type, falling \
    back to default (native compilation)
  dpkg-source --before-build .
  dpkg-buildpackage: warning: building a source package without \
    cleaning up as you asked; it might contain undesired files
  dpkg-source -b .
  dpkg-genbuildinfo --build=source
  dpkg-genchanges --build=source >../mlnx-ofed-kernel_....changes
  dpkg-genchanges: info: not including original source code in upload

- In the logs, prefix the command's standard output with "stdout: " and
  its standard error with "stderr: ". This allows distinguishing between
  the command's messages emitted via standard output and standard error.

- Note that adding the "stdout: " prefix to each line of each command's
  standard output increases the log file's size by about 190 KiB,
  according to the following results from a fresh build with
  'build-pkgs -b std,rt -c -a':

    $ grep -e '2022-09-1[67] .*stdout: ' builder.log | wc -l
    24391 # num_lines
    $ echo $(( 24391 * 8 )) # i.e., num_lines * strlen("stdout: ")
    195128 # bytes

  Logging each command's standard error (along with prefixing each line
  with "stderr: ") adds about 1.87 MiB of output to the build logs,
  which corresponds to an increase of approximately 21 percent:

    $ grep -e '2022-09-1[67] .*stderr: ' builder.log | wc -c
    1963024
    $ grep -e '2022-09-1[67] ' builder.log | wc -c
    11164207
    $ echo 'scale=2; print 1963024.0/(11164207-1963024) * 100,"%\n";' \
        | bc
    21.00%

  Despite the increase in the log file size, logging the standard error
  uncovers some issues, mostly relating to the debian/changelog file
  formatting used by StarlingX developers, which are admittedly minor.
  Examples include:

  ... stderr: dpkg-genchanges: warning: debian/changelog(l...): \
    badly formatted trailer line
  ... stderr: dpkg-genchanges: warning: debian/changelog(l...): \
    found end of file where expected more change data or trailer

- Log the return code of the command if it fails. This provides
  additional information regarding why a command failed, in case the
  command's output is not sufficient to determine the reason.

- Prefer to log the command as a list/tuple if it is provided with that
  type, for easier correlation with the code and for consistency with
  the rest of the logging in the function.

- Improve logging and reduce the duplication involved in log message
  construction.

Verification
- 'build-pkgs -b rt,std -c -a' runs without issues with this patch, but
  emits additional output due to logging the standard error streams of
  executed commands.

Closes-Bug: 1989009
Closes-Bug: 1990177
Change-Id: I6b3c4dc70fb1404280ea15e8dcc573c83792f9b1
Signed-off-by: M. Vefa Bicakci <vefa.bicakci@windriver.com>
2022-09-19 17:01:13 +00:00
Yue Tao
49840b466a downloader: binary download supports CENGN
downloader tries to download binaries from CENGN first, if fails, then
tries to download from Debian url.

Move url_to_cengn and get_download_url to utils.py, so that
downloader can use these functions to convert an url to CENGN url.

Test Plan:

Pass: downloader -b
Pass: See the url is converted to CENGN url in log.

Closes-bug: 1986961

Signed-off-by: Yue Tao <Yue.Tao@windriver.com>
Change-Id: Ic5cbdc70535d5f1f68362cc009d8007afb7b8a35
2022-09-08 11:19:04 +08:00
M. Vefa Bicakci
c0c89d5f51 stx: Do not use shell for curl and wget
When using a download URL that includes the character '&' (as part of a
query string), the following build failure is encountered, because the
'&' character was is interpreted by the shell as a special character for
backgrounding the 'wget' command.

  debrepack  - ERROR: 2022-06-29 20:04:31 (3.79 MB/s) - \
    '...' saved [...]
  debrepack  - ERROR: [ Failed - "wget -t 5 --wait=15 \
    http://a_url_with_& -O ..." ]

This issue is resolved by not using the shell for spawning download
commands. (That is, by setting "shell=False" in Python's
subprocess.Popen constructor.) This commit implements an in-place
solution that detects the type of the argument and sets the "shell="
keyword argument to subprocess.Popen accordingly.

Verification:
- Downloading from URLs with '&' characters works as expected as part of
  the "debrepack" phase of build-pkgs.
- No negative behaviour observed with a build from scratch.

Closes-Bug: 1988343
Change-Id: I2a529f60b9a57b6d139f95d31bcd18c51a0fbecb
Signed-off-by: M. Vefa Bicakci <vefa.bicakci@windriver.com>
2022-08-31 20:57:58 +00:00
Yue Tao
43c4dc3134 Debian: move run_shell_cmd to utils.py
Move run_shell_cmd to utils.py, so that it can be called by other
modules.

Test Plan:

Pass: build-pkgs -c -a

Signed-off-by: Yue Tao <yue.tao@windriver.com>
Change-Id: Ib114fe498cc0303846099f803f79b280aef3f64b
2022-07-08 10:27:31 +08:00
Luis Sampaio
e118936e89 Add timestamp to build logs
Add timestamps to logs (console and file), updates debrepack
to use utils.py for logging and removes color from file handler
builder.log.

Test Plan:
Pass: build-pkgs -a
Pass: build-image
Pass: repo_manage list
Pass: downloader -s -b

Story: 2008846
Task: 45086

Signed-off-by: Luis Sampaio <luis.sampaio@windriver.com>
Change-Id: If2d76d0d1a6ddd47b6ecc8ed72f9adecbcd354b5
2022-05-02 10:30:58 -07:00
hbai
c2cc10133c stx: discover buildable packages
Remove hard coded lists of build types and layers.

Discover all buildable layers and build types by
scanning the source code base for config files.

Discover the build order of layers and build types from
config files

Allow options to reduce the build set to specific
subset of layers or build-types

Story: 2008862
Task: 43154
Depends-On: https://review.opendev.org/c/starlingx/root/+/832145
Depends-On: https://review.opendev.org/c/starlingx/tools/+/832704
Signed-off-by: Scott Little <scott.little@windriver.com>

Signed-off-by: hbai <haiqing.bai@windriver.com>
Change-Id: I3e6ab41ed79877a926f3adc25c4058436bbccc17
2022-04-19 17:42:35 +08:00
hbai
9a223b9598 Revert "stx: discover buildable packages"
This reverts commit 64189c22c468ee37f5e9d65ce228c2ed63332c17.

Signed-off-by: hbai <haiqing.bai@windriver.com>
Change-Id: Iff6b7fa087aa3a6dcc9ead6389721ed42de41094
2022-04-09 12:09:36 +08:00
Scott Little
64189c22c4 stx: discover buildable packages
Remove hard coded lists of build types and layers.

Discover all buildable layers and build types by
scanning the source code base for config files.

Discover the build order of layers and build types from
config files

Allow options to reduce the build set to specific
subset of layers or build-types

Story: 2008862
Task: 43154
Depends-On: https://review.opendev.org/c/starlingx/root/+/832145
Depends-On: https://review.opendev.org/c/starlingx/tools/+/832704
Change-Id: I3eae4888e3d3e9d65a4bcd1410bfc2d4cb6c31d4
Signed-off-by: Scott Little <scott.little@windriver.com>
2022-03-22 14:53:48 -04:00
hbai
44da1da1f9 debian:build-pkgs: Create the initial scripts to build packages
Create the initial script 'build-pkgs' which helps to launch
the building of debian packages in build container.
The depend function modules 'dsccache.py', 'debsentry.py' and
'utils.py' are also created.

Story: 2008846
Task: 43120

Signed-off-by: hbai <haiqing.bai@windriver.com>
Change-Id: If1924e72aa2a5563da1c00ac328ca1e560c65cbd
2021-09-27 05:15:13 -04:00