A change in Ansible 6, discussed at [1], highlights that Ansible is
using the #! line to determine using it's own heuristics the
interpreter to use to run the file, rather than the more common idea
that this file is being interpreted by the kernel.
The best solution seems to be to have no interpreter line, which is
done here.
To avoid confusion this removes the executable bit; if you want to run
it manually you should run it under the python binary as an argument.
I couldn't find any other instances of this same behaviour in
system-config.
[1] https://github.com/ansible/ansible/issues/78809
[1] 9142be2f6c
[2] 9e22cfdb0f
Change-Id: I1d37e485a877982c72fa19acad907b682858c15b
This reverts commit a39a939e0352741d0b2c43e96e660f52eac22245.
Turns out that ansible module args don't get typed the way we expect
them. This means having a Boolean or List type argument just ends up in
confusion and always_update being truthy every which way. Revert until
we can fix this properly.
Change-Id: I596fe6883098ba636b1cad5196d1fdd76ff19076
Setting the gitea_always_update var for the gitea-git-repos role to
a list will filter metadata updates to only the project names
included in the supplied list. False and True still have their prior
meanings of do no metadata updates or force metadata updates for
every project we host.
Add testing for this, and also actually test that the rename
playbook renamed something.
Get rid of the git clone in the playbook since it's no longer
relevant to how we run things anyway, we'll instead want to rely on
the Zuul supplied projects.yaml path.
Change-Id: Id8238b232caffc242c6bda9fe39eb7e65fe5e059
Setting gitea project settings like wiki and issue tracker settings was
previously done via hijacking web ui requests. We now have a REST API
that is capable of setting things items. Using this API should be more
reliable as the API is versioned.
Update the gitea project creation code to use this API for more
stability. As a nice side effect the code is simplified quite a bit as
we can combine a few actions that were previously separate like updating
descriptions and default branches.
As a side note this fixes a bug where we hardcoded setting master as the
default branch despite making that configurable.
Change-Id: I101dd8f81a2cb91655f6de878bc94350aeb1fc0c
There is some correlation that running the manage-projects playbook
gives our gitea fits. The bulk of the work done here is in trying to
update the descriptions of all projects. There isn't a good way to see
if the description is already set first so we just try and ignore
errors. This creates potentially thousands of operations all at once and
could be why things are sad.
We move these operations under the always update flag which is not set
on normal runs. If we really need to converge to a good updated state we
can manually run the playbook/role with always update set.
We also don't set a limit on the number of ThreadPoolExecutor workers
which will default to 5 * NumProcs. Could be that tuning this down would
make gitea happier.
One other thought is that we may not be using request sessions properly
for connection reuse. In particular requests notes that you need to set
stream to False or read request content to return a connection back to
the pool for reuse. We might look into this for further improvements.
Change-Id: I6e6fb1eb08303e9da7e38cf493d1871364340000
There appears to be a gitea bug that causes PATCH updates to projects to
fail when the cache is in a bad state for that project. We use PATCH
updates to a project to set the project descriptions. Since project
descriptions are not critical to gitea functionality (we weren't
updating them until last week) we can treat this as best effort and
ignore these failures.
We'll log these cases to aid in further debugging but continue on. The
next pass can retry.
Change-Id: I625bdc0856caaccb6b55931b0cdc6cf11a0bf3e1
When we decide we don't need to create a project we set the project
description. The reason for this is that humans like to see their
project descriptions update when they change them.
Rather than get, compare, and set the descrition we just set it under
the assumption this will be fewer requests and thus quicker. The impact
on the db likely plays into this too but our gitea dbs are mostly idle
so should be fine.
Change-Id: I04bdd747f8934d0b35bf76aec5d70be01b921285
Note this shouldn't be used until we can configure Gerrit to do similar
with jeepyb. Otherwise we'll end up with mismatched branches between our
canonical source (Gerrit) and our mirrors (Gitea).
Change-Id: I8d353cbc90c2d354e7cdebfc4e247f3f73d97d86
According to gitea swagger definitions all of these GET requests for
lists of items are paginated with a max limit of 50 items per request.
Update our ansible machinery to properly page these items to avoid
problems in the future.
Note we should try and confirm that this is how it works for production
gitea.
Change-Id: I5df13288b497fb4fb716b4223b3dd61c698a7739
We list gitea repos to determine if we need to create a repo. If the
repo isn't listed by gitea we create it. New gitea paginates these
listings so we were only getting 30 repos listed when we had far more.
This resulted in us trying to create repos which already exist which is
a gitea http 409 error.
Fix this by paging through the listings until we've seen all the
repos. This should give us a complete listing.
To test this we run our manage-projects playbook twice in the
system-config-run-gitea job. The first pass creates all the new
projects. Then the second pass should noop cleanly.
Change-Id: I73b77b9ddaa0106d4dc0a49c4d4b7751a39a16f9
Co-Authored-By: Jeremy Stanley <fungi@yuggoth.org>
We're making these requests to localhost over an ssh connection.
The password warning, on the other hand, is a real thing. Let's not
log the gitea password when we run this in prod.
Change-Id: I2157e4027dce5ab9ebceb3f78dbeff22a83d9fad
This runs repo creation across two orgs at the same time. It doesn't
help to parallelize more than 2 since openstack runs the entire time
in one thread (so the other thread handles all the other orgs).
Parallelizing by org avoids database contention for updating the user
table, since each org is a different user. However, there's a weird
locking thing going on with the first update to the settings table,
so this does some extra work to serialize actions until we perform
that first update, then switches to parallel.
This is the maximum we can parallelize repo creation at the moment,
and it also maximizes settings updates (the settings updates take less
time than repo creation, so no further optimization helps).
Change-Id: I7f83dcdb4531a547ae5281434d7cda825dd50059
This keeps repo creation serialized (because of a bug in gitea),
but it parallelizes updating the settings. This should reduce
our time by about half.
It also uses a requests session, though I'm not sure if that
really gets us anything.
It eliminates a couple of extraneous GET calls following 302
redirect responses from the POSTs on setting updates.
This will automatically paralellize to nproc * 5 threads.
Change-Id: I5549562d667c0939d0af1151d44b9190774196f9
When determining whether a project exists, we need to compare to
just the name, not the full data structure about the project.
Also, if the project exists, don't try to create it again; that
will return a 409 conflict error.
Change-Id: I0b8affac96b17fa73253082b1b87d4c00bf23463
Add the full remote_puppet_git playbook that we actually use in
production so that we can test the whole kit and caboodle. For
now don't add a review.o.o server to the mix, because we aren't
testing anything about it.
Change-Id: If1112a363e96148c06f8edf1e3adeaa45fc7271c
Sadly, as readable as the use of the uri module to do the interactions
with gitea is, more reent ansible changed how subprocesses are forked
and this makes iterating over all the projects in projects.yaml take
an incredibly long amount of time.
Instead of doing it in yaml, make a python module that takes the list
one time and does looping and requests calls. This should make it be
possible to run the actual gitea creation playbook in integration tests.
Change-Id: Ifff3291c1092e6df09ae339c9e7dddb5ee692685
