This updates test-prepare-workspace-git to use a new Ansible module
for its work. This module supports each of the three main tasks
with a single Python invocation for all projects, rather than
using Ansible to loop over them.
Within the python invocation, a ThreadPoolExecutor with 10 workers
is used to execute the setup processes with increased parallelism.
This should greatly improve the workspace sync speed for jobs with
large numbers of repos.
A simple local test with the 16 Zuul project repos reduces the
workspace-prep time to less than 50% of the current time.
This removes the mirror_workspace_quiet role variable, since detailed
information about the repo state can now be found in the repo state
JSON file.
Otherwise, the actual git commands used to prepare the repos should
be identical.
Change-Id: Ib4b6bb379fe656ac9109b8e6073eff8c28c7eaee
This runs the same commands on the git repo regardless of whether
it was cloned or if the role found it there already. Since the
purpose of the role is to mirror the workspace repos from the
executor to the remote node, this will produce more consistent
behavior.
Note that anyone somehow relying on, say, the origin being set
outside of this role may encounter a behavior change. It is expected
that anyone manipulating a repo that is also managed by this role
would perform those manipulations idempotently after running this
role.
Change-Id: I428bf2980a526919d5b154c585943be92d4c1cfa
Commit 8003cdc76c causes problems
if the remote repo already exists (e.g., the worker node is static
and not ephemeral) because it unconditionally removes settings
which are only conditionally set if the workspace is newly cloned.
Fix that by remove the Ansible "creates" check from the task and
executing the set calls unconditionally (but also, recreate the
functionality of the create check for the cloning part of the
task, which is what we're really trying to avoid).
This will run a few extra command such as clearing the bare flag
and also resetting the origin remote. That should be fine in
this role since we expect it to do whatever it takes to make the
remote repo the same as the local one.
Also, resync test-prepare-workspace-git.
Change-Id: Ife12992df9ce2b0ce199b3980a4baa255cb0f28a
In cases where the HEAD branch of a repository is deleted,
prepare-workspace-git will be unable to propagate that deletion from
the executor repos to a clone of a cached repo on a remote node.
Correct this by unsetting the receive.delyDeleteCurrent flag. This was
correctly set in mirror-workspace-git, but did not make it into
prepare-workspace-git.
Change-Id: I159fb1f9f9cee873466f11be2f155bed6892472e
The prepare_workspace_sync_required_projects_only variable allows
users to define which projects to sync to the node. This can prevent
syncing of unnecessary repositories. For some builds e.g. the
depends-on repositories dont need to be synced. The projects are
filtered based on the 'required' flag present in each zuul.project
entry and the required projects list also does not contain projects
which are present due to Depends-On or gate queue sequencing.
Having unnecessary repos in the workspace can for example also break
the analysis phase of bazel.
Change-Id: I3cc36cbfc60c81956caf5137da63973aeade4e21
Co-Authored-By: James E. Blair <jim@acmegating.com>
Co-Authored-By: Bernhard Berg <bernhard.berg@bearingpoint.com>
This change is preparation for https://review.opendev.org/c/zuul/zuul-jobs/+/887917
In the beginning, there was only prepare-workspace[0] which rsynced repos.
Then we added mirror-workspace-git[1] to make it more efficient by using git operation, but it required some openstack-specific code in project-config to work.
Then we added prepare-workspace-git[2] which completed the git-based sync solution by locating everything requried in zuul-jobs. It used mirror-workspace-git by reference and added this TODO:
# TODO(tobiash): we might want to deprecate the role mirror-workspace-git-repos
# and move it here.
This change completes that TODO by moving the mirror-workspace-git-repos code into prepare-workspace-git and places the repo in a sensible and maintainable state with two simple and good options:
* prepare-workspace (rsync)
* prepare-workspace-git (git)
In the unlikely event anyone is still using mirror-workspace-git-repos standalone (OpenStack/OpenDev is not, and that solution was haphazard as described above) they would be well served by a notification that there is a better alternative which is what most of the community actually uses now.
[0] cfffd4431b
[1] 348598e96a
[2] 7cee7156bc
Change-Id: Ib80e0447d49363182fd0d4c4d4e269841bc3aa95
Use zuul_workspace_root as the destination folder for preparing and
mirroring the workspace, which points to ansible_user_dir by default.
(As is currently the case).
The override of this variable can be useful when using a static node
with multiple slots to use a separate workspace for each slot.
Change-Id: I2014b9c876907cfc17a0ec7fa2b6d4e67cb2331c
The child change to prepare-workspace-git explains this in more detail.
We're reducing the number of ansible loops these roles run to reduce
ansible task startup overhead and speed up the role. This change is made
first to the test role so that we can ensure this all works properly
before affecting production zuul installs.
Change-Id: I3d33bb0001e65f449e1026ed323514097088f664
This is a copy of prepare-workspace-git except that it imports
test-mirror-workspace-git-repos. This is for base job testing.
Change-Id: I4ef3e4376c9e958761c165836c4fb546157e237a