We're making these requests to localhost over an ssh connection.
The password warning, on the other hand, is a real thing. Let's not
log the gitea password when we run this in prod.
Change-Id: I2157e4027dce5ab9ebceb3f78dbeff22a83d9fad
This runs repo creation across two orgs at the same time. It doesn't
help to parallelize more than 2 since openstack runs the entire time
in one thread (so the other thread handles all the other orgs).
Parallelizing by org avoids database contention for updating the user
table, since each org is a different user. However, there's a weird
locking thing going on with the first update to the settings table,
so this does some extra work to serialize actions until we perform
that first update, then switches to parallel.
This is the maximum we can parallelize repo creation at the moment,
and it also maximizes settings updates (the settings updates take less
time than repo creation, so no further optimization helps).
Change-Id: I7f83dcdb4531a547ae5281434d7cda825dd50059
This keeps repo creation serialized (because of a bug in gitea),
but it parallelizes updating the settings. This should reduce
our time by about half.
It also uses a requests session, though I'm not sure if that
really gets us anything.
It eliminates a couple of extraneous GET calls following 302
redirect responses from the POSTs on setting updates.
This will automatically paralellize to nproc * 5 threads.
Change-Id: I5549562d667c0939d0af1151d44b9190774196f9
When determining whether a project exists, we need to compare to
just the name, not the full data structure about the project.
Also, if the project exists, don't try to create it again; that
will return a 409 conflict error.
Change-Id: I0b8affac96b17fa73253082b1b87d4c00bf23463
Add the full remote_puppet_git playbook that we actually use in
production so that we can test the whole kit and caboodle. For
now don't add a review.o.o server to the mix, because we aren't
testing anything about it.
Change-Id: If1112a363e96148c06f8edf1e3adeaa45fc7271c
Sadly, as readable as the use of the uri module to do the interactions
with gitea is, more reent ansible changed how subprocesses are forked
and this makes iterating over all the projects in projects.yaml take
an incredibly long amount of time.
Instead of doing it in yaml, make a python module that takes the list
one time and does looping and requests calls. This should make it be
possible to run the actual gitea creation playbook in integration tests.
Change-Id: Ifff3291c1092e6df09ae339c9e7dddb5ee692685
I ran our global gitea project sync playbook across all eight gitea
hosts and one failed with a 404 against a specific project. Rerunning
the playbook against that one gitea server worked fine.
Until we sort out why this might happen lets retry our HTTP POSTs up to
3 times until they succeed.
Some numbers: We have ~2k repos and 8 servers and make two http requests
per repo for a total of 32k requests. If one fails out of that the
success rate is very high so retrying a few times should be fine.
Change-Id: I937a4f852f6713a419c03a17c3b4984a97eae0d8
Some projects use storyboard and some use launchpad as their issue
tracker. Handle this dynamically based on the projects.yaml content when
we create projects and set the appropriate urls.
Change-Id: Ie1d79bf1a662c00078689f2c752231e610610eda
Story: 2004627
Task: 29849
The web page from which this call is taken has a readme selector
and a default value of "Default" that we're not sending in this
request. Send it to avoid gitea not being able to find the empty
readme.
Also, add gitea-git-repos to the files section of system-config-run-gitea
so that we actually test it.
Change-Id: Ieec94aadb63fa097f10a3f325dd105b30e610dd9
This reverts commit b076aefa0864dc12269a4454a921f4830b2a2d9a.
This may still not work. The last time it got 404s when
running as part of project creation. So possibly it was
failing because there was no git content yet and thus
master was an invalid value. If that's the case, we may
need to add auto_init to true just to get a default repo
and metadata and count on gerrit force-pushing during first
replication.
Use auto_init to create a repo with a master branch
It's not possible to set the default_branch to master if there is
no repo, and without that first replication sets the branch
incorrectly.
Set auto_init to true to get an empty repo which will be replaced
by the initial gerrit force-push anyway.
Change-Id: I66de43a4d00583c06cf92bb7083ce28d4b8d4df9
Story: 2004627
Task: 29847
Ansible task execution can be a bit slow - so the noop case where
we don't end up doing anything can still be costly. Instead, put
the when on the loop call, which will apply to each iteration of
the loop, not running the loop itself. This way we should only
include_tasks if we need to.
In order for the utility playbook to keep working, we also run all
of the iterations of the loop if gitea_always_update is true. This
will make a sync run take a long time but be comprehensive.
Change-Id: Ib60c736d46d8253e603de097eb80bc84b3366310
We normally only do project settings when we create the project.
Add a playbook we can use to do a manual sync from project-config.
Change-Id: I5260f2de697420a01d796acc3128be00705a53ee
This is 404ing. It's not strictly needed for the moment, let's
come back to it.
This reverts commit 266b2dd3fc4bcbe2416f77f34540705a6ba6cdde.
Change-Id: Id9ab3c233753025d7a01eb4664ecee6d102bbf19
If, for some reason, a repo gets created via gerrit replication
instead of via the api, the default branch setting can be incorrectly
inferred. While we're setting things, just set master to be the default
branch everywhere.
Change-Id: I388afe670221bd8ec28fbba71041ab460e66411b
This uses the form post approach also used in repo renaming. There
is no official REST API for this yet, so the form post values were
taken from what the web ui does.
We should submit a patch upstream to get an actual api call for this.
Run it idempotently, submitting the settings for every repo every time.
Change-Id: I9265837039df962e85f11d16419e043fb9a56ff8
We are currently attempting to create repos which already exist. This
fails.
The reason for this is we set the gitea_org_repos fact for each org
overriding the last org. This means only the last org processed has any
projects in this list. We then check against this list when creating
projects so that we only create projects if they aren't in the list.
Meaning any project for repos not in the last org attempts to get
recreated.
We can address this by keeping a global list of repos regardless of org
then checking against that.
An alternative solution would be to process projects for each org
separately. Or to have gitea give us the global list so we don't have to
build it ourselves.
Change-Id: Id9a480634918dad2160a4e040a41ce6226ae67d8
Switch to a heredoc for the gitea sql_statement so that we don't have to
figure out multiple levels of "" quoting.
Change-Id: I734640936a9d15d03026fc3b05b5fbc221957b94
Go ahead and actually clone project-config on bridge, because
we're going to read the projects.yaml file with a lookup. Because
it's a local action, not a remote action.
Change-Id: I77454bcb10b797ce5b48018caef7fecb31947b97
We need to make sure repos are created in gitea before gerrit so
that we can safely create to gerrit and have it replicate.
Change-Id: If3efc7ed0d7995f1ef6f52cd2eefb9260193e020
Story: 2004627
Task: 29703