The original code would download all verions names in our lst files,
upload them one at a time to Aptly, and as part of each upload it
would delete other versions of the package. The end result is that
the last uploaded version is the only one remaining in Aptly.
Fix is to delete old versions of a package from Aptly as a
separate step, with full knowledge of all versions we need to keep.
Testing:
1) Add multiple versions of a package to a lst file.
Run downloader. All expected versions are found
in Aptly.
2) Remove the extra version of the package.
Run downloader. The extra, no longer needed package
version are removed from Aptly leaving only the one we want.
Story: 2010797
Task: 48697
Change-Id: I6de33d8d1c2bb5161c905765e7757102ea9b8cac
Signed-off-by: Scott Little <scott.little@windriver.com>
(cherry picked from commit 9b0a1394661bc0e8c9c6f89437f5648cd5a8d532)
The '--reuse|--reuse_maximum' feature mirrors the remote
shared repository and imports all debs from the mirror
to the local repository, there are no deb packages in the
local build directory for the reused packages which make
some tasks like building docker images and secure boot
signing fail for the missing deb packages.
This commit supports the below functions to fix the above
issues:
a. If '--dl_reused' option is enabled for option '--reuse'
or '--reuse_maximum', all the reused deb packages will be
downloaded to their local build directory.
b. 'never_reuse.lst' will be checked and the packages listed
in it will be built locally instead of reusing them if the
option '--reuse' is enabled. And it will be ignored if the
option '--reuse_maximum' is enabled.
Test Plan:
Pass: build-pkgs (make sure the normal build-pkgs works)
Pass: export STX_SHARED_REPO=<url to shard repo>
export STX_SHARED_SOURCE=<url to shared source>
build-pkgs --reuse
Pass: export STX_SHARED_REPO=<url to shard repo>
export STX_SHARED_SOURCE=<url to shared source>
build-pkgs --reuse --dl_reused
Pass: export STX_SHARED_REPO=<url to shard repo>
export STX_SHARED_SOURCE=<url to shared source>
build-pkgs --clean --reuse --dl_reused
Run the secure boot signing script
Pass: export STX_SHARED_REPO=<url to shard repo>
export STX_SHARED_SOURCE=<url to shared source>
build-pkgs --clean --reuse --dl_reused
build-pkgs (Make sure this build will not build from scratch)
Pass: export STX_SHARED_REPO=<url to shard repo>
export STX_SHARED_SOURCE=<url to shared source>
build-pkgs --clean --reuse_maximum --dl_reused
Pass: export STX_SHARED_REPO=<url to shard repo>
export STX_SHARED_SOURCE=<url to shared source>
build-pkgs --reuse_maximum --dl_reused
Partial-Bug: 2017763
Signed-off-by: hqbai <haiqing.bai@windriver.com>
Change-Id: I8cd84dbe6fe8f0262dde12befb0b16367e261968
Update false to False if the mirror creation
fails.
Story: 2008862
Task: 46340
Signed-off-by: Luis Sampaio <luis.sampaio@windriver.com>
Change-Id: Id0d898e7cabea583c4dbb91b94a7ee7415c43844
This commit updates code to handle the below things:
a. For these reused packages, updates the local debsentry cache
after successfully copying the reused debs from the shared
repository, this fixed the issue that the second project can not
get the subpackages from the shared debsentry of the first reused
project.
b. If the subpackages can't be got or failed to copy reused debs from
the shared repository, the reused packages will be reclaimed and
switch to the local build.
c. Opiton '--dl_reused' is provided to download these reused debs to
the package's local build directory if this is really needed
Test Plan:
1. Creates project A with reuse disabled and shared the build output
Creates project B with reuse enabled, the shared project is A
Creates project C with reuse enabled, the shared project is B
Project C built successfully
2. Create project D with reuse enabled and '--dl_reused' option:
$build-pkgs -a --parallel 10 --reuse --dl_reused
All the resued .deb will be downloaded to the local directory:
${MY_WORKSPACE}/reused_debs/downloads/binary
Closes-Bug: 1993224
Signed-off-by: Haiqing Bai <haiqing.bai@windriver.com>
Change-Id: I03267aab66f17afdb540f7d407bb4c9d97805886
Find and copy a set of packages from one repository to another.
Multi-version package is not supported. For each package, any
repository can only contain one binary version and one source
version. While different repositories can contain different
versions.
Test Plan:
Pass: copy a set of binary packages
Pass: copy a set of source packages
Pass: copy packages does not exist in source repo.(report error)
Pass: copy packages already exist in destination repo.
The package is 100% same, ignore silently;
The package is not same, in "overwrite" mode, overwrite
the original one in source repository;
The package is not same, not in "overwrite" mode, ignore.
Story: 2008862
Task: 46340
Signed-off-by: Zhang Xiao <xiao.zhang@windriver.com>
Change-Id: I6769481290ecd11162737e71b0d863e5987d5340
Add support to use host's sources.list;
Enlarge the APTFECTH_JOBS from 10 to 20;
For download and sync, rename parameter "name" to "repository" to align
with other sub-commands
Test Plan:
- Pass: Download with different APTFECTH_JOBS.
- Pass: download binary/source packages with host's source into a
local repo, all packages been downloaded and deployed.
- Pass: sync a local repository with binary/source package list files.
No duplicated package been downloaded. All packages been deployed
through specified local repository
- Pass: Insert error package/version into package list file, get
exception and related error message.
Story: 2008862
Task: 45283
Signed-off-by: Zhang Xiao <xiao.zhang@windriver.com>
Change-Id: I35391e6b64db018c0718eac43d9854692a5b826a
Add a method to quickly deploy local repository. In such cases,
no snapshot been created, the publish based on local repository
directly.
Add a function __wait_for_slow_task for tasks may spend more than
60 seconds.
Test Plan:
- Pass: quickly deploy repository not exist, failed as designed
- Pass: quickly deploy a repository with different suffix, with
the very repository upgraded. All works in order.
- Pass: Deploy a huge enough repository with timeout_count as 1,
timeout and deploy failed.
- Pass: Deploy a huge repository with default timeout_count(10),
work in order.
Story: 2008862
Task: 45612
Signed-off-by: Zhang Xiao <xiao.zhang@windriver.com>
Change-Id: I42ac3f1da03d2bf9c98a198bc3ed4945ad14e588
* build-docker-images/stx-debian/stx.preferences.part.in
* build-docker-images/build-base-image.sh
This file sets base image priorities for apt repos that match certain
properties. Formerly we used higher priority for repos hosted by the
build server, as opposed to debian.org or others. This doesn't work
when using a Debian mirror hosted on the build server itself, since
the hostname of both repos are equal.
Solution: increase priority for repos whose "Release" file contains
the field "Origin: $REPOMGR_ORIGIN" and make sure aptly adds that
field to its Release file. The value comes from the environment and
should be set by the build container.
* stx/aptly_deb_usage.py:
Add an "Origin" field to non-mirror publications, value taken from
environment REPOMGR_ORIGIN
* build-docker-images/stx-debian/Dockerfile.stable
Improvements to package conflict resolution and docker FS
caching-related issues:
- Upgrade base packages to versions in managed repos before doing
anything else
- Install packages provided by upstream debian in a separate RUN
command/docker FS layer
- Make sure each "apt-get install" is in its own RUN command and is
preceded with "apt-get update" -- to avoid using stale metadata due
to "docker build" FS layer caching
TESTS
======================
- Define REPOMGR_ORIGIN in container environment
- Run downloader & build-pkgs & make sure generated repos' Release file
contains "Origin: starlingx"
- Build base image & make sure its apt.preferences contains the priority
rule for "Origin: starlingx"
Story: 2010055
Task: 45729
Change-Id: Ibaafbfbeef408904d216265168daa466d90fc7f2
Signed-off-by: Davlet Panech <davlet.panech@windriver.com>
Optimization purpose. Deploy local repo each time the repo been
modified takes lots of CPU cycles. Add a parameter in function
delete_pkg and upload_pkg to let caller decide if the repository
will be deployed. Add an interface called deploy_repo to manually
deploy a local repository.
This feature only effect on binary packages, for source packages,
we still need to deploy the repository immediately, to support
feature "Check orig tarballs when uploading source packages" of
commit 21d0995ee0.
Remote repository doesn't need this interface for it can only be
deployed once.
Test Plan:
- Pass: upload/delete binary package without deploying repo
- Pass: upload/delete binary package and deploy repo
- Pass: upload/delete source package without deploying repo
- Pass: deploy a repo does not exist, return false and get
related error message.
- Pass: Upload a bundle of packages more than 2GB in size
and deploy related repo at last.
Story: 2008862
Task: 45281
Depends-On: https://review.opendev.org/c/starlingx/root/+/841701
Signed-off-by: Zhang Xiao <xiao.zhang@windriver.com>
Change-Id: Ifc49e0bac2f04819a0fd71e8f485367ce258648e
function upload_pkg_local of class Deb_aptly only deals with one
single package file. That is not good for source package and
uploading a bundle of binary packages. Adjust it to let it
input a bundle of package files.
Test Plan:
- Pass: upload binary packages one by one
- Pass: upload source packages one by one
- Pass: upload several binary packages in one function call
- Pass: upload several source packages in one function call
- Pass: upload several package files with both binary and source
packages in one function call
- Pass: Try to upload a package file without read permission, nothing
uploaded and get warning message.
- Pass: Call function upload_pkg_local with "pkg_files" as an empty
list, just return False and warning message.
Story: 20082862
Task: 45134
Signed-off-by: Zhang Xiao <xiao.zhang@windriver.com>
Change-Id: Id0b0f69dd0a4caedafb670ebc1bc849e6f92d9e8
Fix 8550b600b05d839. A single quote in wrong posithin may cause
an exeption as below:
ValueError: unsupported format character ''' (0x27) at index 27
'%'s Vs '%s'
Story: 2008862
Task: 45010
Test Plan:
Pass: Remove repo does not exist
Signed-off-by: ZhangXiao <xiao.zhang@windriver.com>
Change-Id: I9f3c9606470a83754a3d21f396a9547018855b46
Extend the repo manager with a new subcommand
to list all the packages in an aptly repository.
Testing
repo_manage.py list_pkgs -r deb-local-build
stx repomgr list_pkgs -r deb-local-build
Story: 2008862
Task: 44695
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: I30cf5f540dcc20306f8d0fce32e1b715df0ff4c2
Repo merge is only used for LAT, source packages are useless.
In some special cases, there maybe duplicated packages in different
repositories, find them and output warning messages.
Story: 2008846
Task: 44319
Test Plan:
Pass: Merge repos contain source packages, no source package
in merged publication.
Merge repos contain duplicated binary packages, all such
duplication should be outputed as WARNNING.
Signed-off-by: ZhangXiao <xiao.zhang@windriver.com>
Change-Id: I349174e5b864ad966e2d56b9c329ba764df86ae5
This will not create a real repository but just a publication,
contains all packages of some other repositories. Only used for
debootstrap:
https://wiki.debian.org/Debootstrap
... Debootstrap can only use one repository for its packages...
Story: 2008846
Task: 44318
Test Plan:
Pass: Merge several local repos, merge several remote repos(mirror),
merge several local and remote repos.
Signed-off-by: ZhangXiao <xiao.zhang@windriver.com>
Change-Id: If271d431f75716417379321228a49f79e3aaf220
Sometimes, developers may modify source code and construct orig.* files
with different content. When uploading such a source package, if there
is already such a file in the repository with different content, aptly
will raise an error like:
ERROR: unable to publish: unable to process packages: error linking file
to /var/aptly/public/prefix=deb-local-build/pool/main/a\
/armada-helm-toolkit/armada-helm-toolkit_1.0.orig.tar.gz:
file already exists and is different
Check it at the very beginning and stop the upload operation to avoid
the error from lower level from aptly.
Story: 2008846
Task: 43868
Signed-off-by: ZhangXiao <xiao.zhang@windriver.com>
Change-Id: I55f014f7b3638811517d0ba45f417644ecdc2d76
Be careful to use it without paramter "--package_version", this
will delete all versions of specified packages.
Story: 2008846
Task: 43864
Signed-off-by: ZhangXiao <xiao.zhang@windriver.com>
Change-Id: I5914141018b9d46af37d08309776577a18be837f
list_local/list_remote always outputs "info" level messages for
interactive usage. While those messages seems verbose in other
use cases. Add a "quiet" parameter to make it.
Story: 2008846
Task: 43692
Change-Id: If492e40d74d2433d5809b6628deaf0bd20fc1186
Signed-off-by: Zhang Xiao <xiao.zhang@windriver.com>
StarlingX-Debian use Debian style repositories to manage packages.
Currently, the repo_manage is based on aptly to manage repository and
packages.
Story: 2008846
Task: 43122
Signed-off-by: ZhangXiao <xiao.zhang@windriver.com>
Change-Id: Ie5272e2ecc5e3722d1925fdb2a108f669dfa9218
