26 Commits

Author SHA1 Message Date
Zuul
cb86492fa4 Merge "Silence InsecureRequestWarning and password warning" 2019-07-17 17:15:36 +00:00
Monty Taylor
b58bc86c89 Silence InsecureRequestWarning and password warning
We're making these requests to localhost over an ssh connection.

The password warning, on the other hand, is a real thing. Let's not
log the gitea password when we run this in prod.

Change-Id: I2157e4027dce5ab9ebceb3f78dbeff22a83d9fad
2019-07-17 15:57:57 +00:00
James E. Blair
13c7c8bb7e Parallelize repo creation by org
This runs repo creation across two orgs at the same time.  It doesn't
help to parallelize more than 2 since openstack runs the entire time
in one thread (so the other thread handles all the other orgs).

Parallelizing by org avoids database contention for updating the user
table, since each org is a different user.  However, there's a weird
locking thing going on with the first update to the settings table,
so this does some extra work to serialize actions until we perform
that first update, then switches to parallel.

This is the maximum we can parallelize repo creation at the moment,
and it also maximizes settings updates (the settings updates take less
time than repo creation, so no further optimization helps).

Change-Id: I7f83dcdb4531a547ae5281434d7cda825dd50059
2019-07-16 14:24:44 -07:00
James E. Blair
1e18651565 Provide better module return info from gitea create repos
Be more correct about changed and failures.

Change-Id: I0b37b1bd85efc35233d864ca7801a8862806467f
2019-07-16 13:31:18 -07:00
James E. Blair
47bd535d60 Use a thread pool to update gitea repos faster
This keeps repo creation serialized (because of a bug in gitea),
but it parallelizes updating the settings.  This should reduce
our time by about half.

It also uses a requests session, though I'm not sure if that
really gets us anything.

It eliminates a couple of extraneous GET calls following 302
redirect responses from the POSTs on setting updates.

This will automatically paralellize to nproc * 5 threads.

Change-Id: I5549562d667c0939d0af1151d44b9190774196f9
2019-07-16 10:29:24 -07:00
James E. Blair
892596373f Improve idempotency of gitea-git-repos
When determining whether a project exists, we need to compare to
just the name, not the full data structure about the project.

Also, if the project exists, don't try to create it again; that
will return a 409 conflict error.

Change-Id: I0b8affac96b17fa73253082b1b87d4c00bf23463
2019-07-15 16:05:20 -07:00
Monty Taylor
5c6b3411b7 Run actual full project creation in gitea test
Add the full remote_puppet_git playbook that we actually use in
production so that we can test the whole kit and caboodle. For
now don't add a review.o.o server to the mix, because we aren't
testing anything about it.

Change-Id: If1112a363e96148c06f8edf1e3adeaa45fc7271c
2019-07-11 13:39:22 -07:00
Monty Taylor
f9358173a3 Add some logging to repo creation
So that we can verify what was done, we should emit some things to
log and return them.

Change-Id: I9c48e94fe099002335113aed296bfc9a52d4c10e
2019-07-11 11:54:47 -04:00
Monty Taylor
caebf387b4 Translate gitea project creation to python
Sadly, as readable as the use of the uri module to do the interactions
with gitea is, more reent ansible changed how subprocesses are forked
and this makes iterating over all the projects in projects.yaml take
an incredibly long amount of time.

Instead of doing it in yaml, make a python module that takes the list
one time and does looping and requests calls. This should make it be
possible to run the actual gitea creation playbook in integration tests.

Change-Id: Ifff3291c1092e6df09ae339c9e7dddb5ee692685
2019-07-11 08:21:35 -04:00
Clark Boylan
177edc0abb Retry gitea repo setting HTTP POSTs
I ran our global gitea project sync playbook across all eight gitea
hosts and one failed with a 404 against a specific project. Rerunning
the playbook against that one gitea server worked fine.

Until we sort out why this might happen lets retry our HTTP POSTs up to
3 times until they succeed.

Some numbers: We have ~2k repos and 8 servers and make two http requests
per repo for a total of 32k requests. If one fails out of that the
success rate is very high so retrying a few times should be fine.

Change-Id: I937a4f852f6713a419c03a17c3b4984a97eae0d8
2019-03-15 13:01:39 -07:00
Clark Boylan
f41d9da163 Handle launchpad projects in our gitea project creation
Some projects use storyboard and some use launchpad as their issue
tracker. Handle this dynamically based on the projects.yaml content when
we create projects and set the appropriate urls.

Change-Id: Ie1d79bf1a662c00078689f2c752231e610610eda
Story: 2004627
Task: 29849
2019-03-12 15:14:34 -07:00
Monty Taylor
683b73b4fb Send readme parameter when creating projects
The web page from which this call is taken has a readme selector
and a default value of "Default" that we're not sending in this
request. Send it to avoid gitea not being able to find the empty
readme.

Also, add gitea-git-repos to the files section of system-config-run-gitea
so that we actually test it.

Change-Id: Ieec94aadb63fa097f10a3f325dd105b30e610dd9
2019-03-07 22:17:53 +00:00
Monty Taylor
d447f7d2df Revert "Revert "Set default branch for repo""
This reverts commit b076aefa0864dc12269a4454a921f4830b2a2d9a.

This may still not work. The last time it got 404s when
running as part of project creation. So possibly it was
failing because there was no git content yet and thus
master was an invalid value. If that's the case, we may
need to add auto_init to true just to get a default repo
and metadata and count on gerrit force-pushing during first
replication.

Use auto_init to create a repo with a master branch

It's not possible to set the default_branch to master if there is
no repo, and without that first replication sets the branch
incorrectly.

Set auto_init to true to get an empty repo which will be replaced
by the initial gerrit force-push anyway.

Change-Id: I66de43a4d00583c06cf92bb7083ce28d4b8d4df9
Story: 2004627
Task: 29847
2019-03-06 18:42:54 +00:00
Monty Taylor
e29a62d87f Filter setup-repos loop before include_tasks
Ansible task execution can be a bit slow - so the noop case where
we don't end up doing anything can still be costly. Instead, put
the when on the loop call, which will apply to each iteration of
the loop, not running the loop itself. This way we should only
include_tasks if we need to.

In order for the utility playbook to keep working, we also run all
of the iterations of the loop if gitea_always_update is true. This
will make a sync run take a long time but be comprehensive.

Change-Id: Ib60c736d46d8253e603de097eb80bc84b3366310
2019-03-04 20:39:17 +00:00
Monty Taylor
d3220a7ade Add utility playbook for fixing gitea project settings
We normally only do project settings when we create the project.
Add a playbook we can use to do a manual sync from project-config.

Change-Id: I5260f2de697420a01d796acc3128be00705a53ee
2019-03-04 16:06:49 +00:00
Monty Taylor
1b7d793f7d Only update gitea project settings during creation
It takes too long to update them all every time. Only update them
when we create the project.

Change-Id: I2a2b2515b57a4e6a494a7c73dd86e562816a31ea
2019-03-04 16:03:57 +00:00
Monty Taylor
626bc0459a Limit project description to 255 characters
Otherwise, gitea is sad.

Change-Id: I36e466d06daab8147c052ba3b8fe6754799205b5
2019-03-04 15:57:09 +00:00
Monty Taylor
b076aefa08 Revert "Set default branch for repo"
This is 404ing. It's not strictly needed for the moment, let's
come back to it.

This reverts commit 266b2dd3fc4bcbe2416f77f34540705a6ba6cdde.

Change-Id: Id9ab3c233753025d7a01eb4664ecee6d102bbf19
2019-03-03 21:55:11 +00:00
Monty Taylor
ef42937daf Parse the CRSF token before using
We're missing a step here.

Change-Id: I01ea5e1570afe8f59e96953dc7dc30372a490bf0
2019-03-03 17:57:31 +00:00
Monty Taylor
266b2dd3fc Set default branch for repo
If, for some reason, a repo gets created via gerrit replication
instead of via the api, the default branch setting can be incorrectly
inferred. While we're setting things, just set master to be the default
branch everywhere.

Change-Id: I388afe670221bd8ec28fbba71041ab460e66411b
2019-03-03 16:31:19 +00:00
Monty Taylor
72c3904717 Use HTTP for repo settings
This uses the form post approach also used in repo renaming. There
is no official REST API for this yet, so the form post values were
taken from what the web ui does.

We should submit a patch upstream to get an actual api call for this.

Run it idempotently, submitting the settings for every repo every time.

Change-Id: I9265837039df962e85f11d16419e043fb9a56ff8
2019-03-03 15:47:15 +00:00
Clark Boylan
0fe4c04172 Aggregate all gitea repos and check against them
We are currently attempting to create repos which already exist. This
fails.

The reason for this is we set the gitea_org_repos fact for each org
overriding the last org. This means only the last org processed has any
projects in this list. We then check against this list when creating
projects so that we only create projects if they aren't in the list.
Meaning any project for repos not in the last org attempts to get
recreated.

We can address this by keeping a global list of repos regardless of org
then checking against that.

An alternative solution would be to process projects for each org
separately. Or to have gitea give us the global list so we don't have to
build it ourselves.

Change-Id: Id9a480634918dad2160a4e040a41ce6226ae67d8
2019-03-01 20:49:30 -08:00
Clark Boylan
c0b64767a2 Use heredoc in gitea sql_statement
Switch to a heredoc for the gitea sql_statement so that we don't have to
figure out multiple levels of "" quoting.

Change-Id: I734640936a9d15d03026fc3b05b5fbc221957b94
2019-03-01 17:54:02 -08:00
Monty Taylor
e836fb7af6 Remember that lookups are local
Go ahead and actually clone project-config on bridge, because
we're going to read the projects.yaml file with a lookup. Because
it's a local action, not a remote action.

Change-Id: I77454bcb10b797ce5b48018caef7fecb31947b97
2019-03-01 21:55:18 +00:00
James E. Blair
2a6de1e8bf Clone project-config to /opt on gitea servers
We don't need to clone to /tmp, /opt is safer.

Change-Id: Ib41e5e22f6de9048db12f9814dc4af4621e1637a
2019-03-01 10:07:59 -08:00
Monty Taylor
a53f333162 Create git repos on gitea servers before gerrit
We need to make sure repos are created in gitea before gerrit so
that we can safely create to gerrit and have it replicate.

Change-Id: If3efc7ed0d7995f1ef6f52cd2eefb9260193e020
Story: 2004627
Task: 29703
2019-03-01 18:07:33 +00:00