Commit Graph

42 Commits

Author SHA1 Message Date
Clark Boylan
20b9ce5907 Put jgit pack settings in jgit.config
We were setting these values in gerrit.config but it isn't clear if
these now need to go in jgit.config instead. I've tried to clarify with
upstream maintainers as the documentation is quite confusing. While we
wait for clarification why don't we just set the values in both files to
ensure we are covered.

This converts jgit.config to a jinja2 template so that we can use the
variable number of pack files setting.

Change-Id: I70c1e6b738ed6e9fdb72d86e7cf3fb8cfecf1323
2020-12-07 12:13:09 -08:00
Zuul
1426c8e283 Merge "gerrit: fix db/ mount for gate testing" 2020-12-02 18:29:09 +00:00
Zuul
c93ed30955 Merge "gerrit: set ownership on ~gerrit2/.ssh directory" 2020-12-02 18:25:28 +00:00
Jeremy Stanley
cc04abd39d Move jgit autogc disablement into correct config
The receive.autogc configuration apparently needs to be in its own
file, not in the general gerrit.conf. Move it to the correct
location.

https://review.opendev.org/Documentation/config-gerrit.html#jgit-receive

While we're here, correct the filename on the gerrit.config and
secure.config templates to make it clear they're jinja2 files, and
add a file mode to the replication.config where it was missing.

Change-Id: I9243bccac103c51ee435725aae482731642a37cc
2020-12-01 23:15:57 +00:00
Ian Wienand
27c3f7ac1e gerrit: fix db/ mount for gate testing
The gate testing sets gerrit_database_type to 'h2', but the container
does not setup /var/gerrit/db.  It ends up owned by root in the
container and gerrit can't write to it, causing traceback errors for
account_patch_reviews when anything is updated.

Add a db peer directory with the other config dirs, and make sure it
is bind mounted into the image in the same way.

While we are here, alphabetise this list of directories for
consistency.

Change-Id: Ib24e3027622a711db76d24172b27cdebe6152cb6
2020-11-27 11:15:30 +11:00
Ian Wienand
f0e5d78ba7 gerrit: set ownership on ~gerrit2/.ssh directory
Currently is is 0700 and owned by root, so the gerrit2 user can't
access the keys in there by default (in production this is already
correct).  This is the key used to push replication to gitea for
example.

Change-Id: Ie0230cd74aeb3e759a597c999b2507260f2b6c17
2020-11-27 10:48:26 +11:00
Zuul
b412c96f9e Merge "Clean up cron tab entry from ansible once removed from host" 2020-11-23 06:42:08 +00:00
Clark Boylan
52094c441d Clean up old Gerrit html theming and commentlinks
When we get to Gerrit 3.x the old html theming, hideci.js and
commentlinks that parse html and rely on urls no longer work. Lets clean
that up when we get there.

We can add back in similar things that polygerrit does support later
but we don't want that to make the upgrade even more difficult to do.

This should only be merged once we are running Gerrit 3.x.

Change-Id: I838840e6cbf09ca28faeb2cf06290e298a4a1f74
2020-11-17 16:04:58 -08:00
Clark Boylan
a6cc2d5d8e Clean up cron tab entry from ansible once removed from host
This is a followup to the previous change that cleans up a now unused
crontab resource. THis should be merged once the actual crontab entry
has been removed from the host.

Change-Id: Ieb8df262d9b047e860a22cfac5569c4ca0b7111b
2020-10-16 09:41:49 -07:00
Clark Boylan
9011096d49 Stop managing gerrit's local git mirror dir
We stopped serving this content and the next step is to stop managing it
internally. This depends on a change to jeepyb that makes the local git
dir management on the jeepyb side optional. Once that lands we can
update our configs to tell jeepyb to stop managing it.

We also stop doing garbage collection, mounting it into containers that
don't need it, etc.

Depends-On: https://review.opendev.org/758597
Change-Id: I2185e90edfcac71941bc29a4e11b7b2d4c7c2e13
2020-10-16 09:41:07 -07:00
Clark Boylan
388ec27173 Add gerrit static files that were lost in ansiblification
The title.svg logo for opendev and two jquery js files are no longer
managed by ansible nor do they appear to be in our docker image. They
appear to have been lost when we converted from puppet to ansible +
docker. Add them back in. We are also missing the icla html content (but
not the other clas) add this one back in.

We vendor the js contents even though in the past we copied them from a
git repo clone and a distro package installation. This way we don't have
unexpected surprises, record that the files are used, and can always
update them later.

Change-Id: I981b4b0f233ece45d03a80dc1724a4e496f66eb8
2020-10-09 10:38:45 -07:00
Monty Taylor
4aa28fee13 Update host_vars and sync-to-review-test playbook
The host is review-test.opendev.org, so hostvars for
review-test.openstack.org are not so much going to do anything.

It's easier if we just ssh as root from review to gerrit2
on review-test.

review-test needs to be in letsencrypt group and have a
handler.

We need to install mysql - it's on the existing review
servers but not in ansible, it's just left over from
puppet.

The db credentials are in /root/.gerrit_db.cnf

Change-Id: I90e3c9d1b398cc16fea9f7056cfb059c7140160e
2020-07-12 08:09:46 -05:00
Monty Taylor
4d26d9cb40 Don't install the track-upstream cron on review-test
This is just spawning containers that never die.

Change-Id: I1f5215c6e60ac59d1eb224bef9032785938dfc70
2020-07-07 14:40:24 -05:00
Monty Taylor
2574113870 Make a review-test that we run ansible on
Private host_vars exist on bridge separate from review01.openstack.org
with their own mysql database, rest key and email key. Add in a
conditional on accepting our own hostkey since we're not actually
going to run manage-projects on this host so we don't need to do that.

Change-Id: If7f45250e7ead62f2117f3e6c0fcfc0b4e7f71aa
2020-06-15 13:32:05 -05:00
Zuul
5e4901b7c6 Merge "Install docker-compose from pypi" 2020-04-17 19:11:19 +00:00
Clark Boylan
8eb981b47f Install docker-compose from pypi
We want to use stop_grace_period to manage gerrit service stops. This
feature was added in docker-compose 1.10 but the distro provides 1.5.
Work around this by installing docker-compose from pypi.

This seems like a useful feature and we want to manage docker-compose
the same way globally so move docker-compose installation into the
install-docker role.

New docker-compose has slightly different output that we must check for
in the gitea start/stop machinery. We also need to check for different
container name formatting in our test cases. We should pause here and
consider if this has any upgrade implications for our existing services.

Change-Id: Ia8249a2b84a2ef167ee4ffd66d7a7e7cff8e21fb
2020-04-16 12:08:00 -07:00
Monty Taylor
ebae022d07 Use project-config from zuul instead of direct clones
We use project-config for gerrit, gitea and nodepool config. That's
cool, because can clone that from zuul too and make sure that each
prod run we're doing runs with the contents of the patch in question.

Introduce a flag file that can be touched in /home/zuulcd that will
block zuul from running prod playbooks. By default, if the file is
there, zuul will wait for an hour before giving up.

Rename zuulcd to zuul

To better align prod and test, name the zuul user zuul.

Change-Id: I83c38c9c430218059579f3763e02d6b9f40c7b89
2020-04-15 12:29:33 -05:00
Zuul
ff821692ed Merge "Write out db config for root user" 2020-04-15 01:11:07 +00:00
Monty Taylor
842ba4d592 Write out db config for root user
We use this in some utlity scripts, but we currently don't
write it out anywhere. It was an old puppet artifact.

Change-Id: Ib6fdfc4f4a9c5d1befdb6d256989450996dd2a3d
2020-04-14 15:21:03 -05:00
Monty Taylor
014b3004c0 Add self host keys to known_hosts on gerrit
We run some utility scripts which ssh to ourselves, but we aren't
setting host keys for them. We should fix that.

Change-Id: I2aa5d5e65b15c5c151767377dbc5ead1e442b3ce
2020-04-13 11:33:16 -05:00
Zuul
95184b5454 Merge "Add root cron jobs to gerrit" 2020-04-11 17:46:32 +00:00
Monty Taylor
d475302682 Actually install patchset-created hook
Change-Id: I85a7dcc31f8991da2765a60b0d8f236aecc3024a
2020-04-11 09:44:06 -05:00
Monty Taylor
38cb5da4bb Add root cron jobs to gerrit
These were managed by puppet and got missed in the original
transition.

Change-Id: Ica1e6cf1ccbcc66e58fcb1f47748022dc03aa8b8
2020-04-10 13:39:33 -05:00
Monty Taylor
27dfd250af Set env vars pointing to correct file locations
Files are bind-mounted into the container in different locations.
Set envvars pointing to the right places.

Also - we need to bind-mount the projects.yaml and projects.ini
files into the container.

While we're at it, move patchset-created to be a regular file.

Change-Id: Iacd3e921464b24479db13bbf7ae998b8d8e2103d
2020-04-10 13:29:50 -05:00
Monty Taylor
3b2f87105d Add cron jobs that were managed by puppet
These still exist, because puppet put them there. We should delete
them by hand on the server and then land this.

Change-Id: I7ee0c60fade0b29a52898a7edf64554d1aa61cae
2020-04-03 10:53:30 -05:00
Monty Taylor
bde5860f57 Parameterize manage-projects logging output
Change-Id: If4c456957fdce308ccafee0dfad8b627a85ea796
2020-03-30 13:53:39 -05:00
Monty Taylor
cc1929d075 Add an image pull to manage-projects playbook
When we run manage-projects, we always want the latest image.

Add a task file to the gerrit role so that it can have access
to role default variables. Then call it from the playbook
with tasks_from.

Change-Id: I08bd10539d7f08e8759f1c22d66dea18821c9d42
2020-03-27 15:58:41 -05:00
Monty Taylor
2d208f0df9 Actually write out the projects.ini config file
We have a template for it in tree, but we're not, you know,
templating it.

Change-Id: Ic6b3ecc62cdfc7c486a912ba4863c7c4c7734e0a
2020-03-24 14:39:12 -05:00
Monty Taylor
e6d98f0181 Install utility scripts for running jeepyb commands
jeepyb is installed in the gerrit image because of hook scripts.
To run manage-projects and track-upstream, make wrapper scripts
in /usr/local that runs the commands from in the container image
bind-mounting the appropriate dirs and files into the container.

Change-Id: I7ef1f00e69d4c310d69d83c80ca210e8f340878d
2020-03-22 10:39:47 -05:00
Monty Taylor
906e6a72a1 Use dev subdir on review-dev for project-config things
This is in anticipation of running manage-projects, which wants to
use the dev subdir of project-config when run on dev, but we're
currently not doing anything with that. Point at root or dev as
appropriate. Then we can do a similar thing - probably just
bind-mounting the root or the dev when we docker run for
manage-projects.

Change-Id: Ia13bbb0bf8dbe1f7e7c0f378ba9b41bef3ecd5c1
2020-03-21 11:34:28 -05:00
Monty Taylor
63d8f7af48 Base 2.13 image on gerrit-base
We install jeepyb and launchpadlib in gerrit-base. Those are
important. We also need to add cgi for gitweb.

The gerrit init command does two things that we don't actually
want it to do at runtime. It extracts the plugins into the
plugins dir, and it downloads the right database library.

We can extract the plugins for it during image creation, and
then we can also download the plugin it would have downloaded.

We can also download the mysql library for it:

https://gerrit.googlesource.com/gerrit/+/refs/heads/stable-2.13/gerrit-pgm/src/main/resources/com/google/gerrit/pgm/init/libraries.config

Finally, we tell it to not download or expand anything during
init, because we're running in a container and next time we run
the process that dir isn't going to be there.

Our gerrit integration tests don't depend on our gerrit image builds.
Put in image depends between run-review and gerrit builds.

We also need to depend directly on opendev-buildset-registry.

Add java.security.egd setting to java invocation

This tells java to be secure.

https://stackoverflow.com/questions/58991966/what-java-security-egd-option-is-for

Add support for setting heap limit properly

The gerrit init script does this based on the value in
container.javaOptions. We could, but then we'd have to
run an entrypoint script. Instead, set the value via
the JAVA_OPTIONS env var setting based on a value from
ansible.

Finally, make gerrit-master image build non-voting

It looks like there might be a real issue, but debugging that
is not important for us at this moment.

Depends-On: https://review.opendev.org/714216
Change-Id: I01e94c10f470fb3c8ddfce7b0e201357e5050679
2020-03-20 16:37:18 -05:00
Monty Taylor
9b999423b7 Add an use a utility service for gerrit commands
We have a need to run ad-hoc commands with the various things
mounted, such as the gerrit init, as well as our friends like
manage-projects. Make a second "service" that doesn't do anything,
but allows us to run "docker-compose run".

Change-Id: Ic9bafe85b90af413c891b969273437be9df6a79f
2020-03-20 09:51:29 -05:00
Monty Taylor
6cc2d0c176 Copy the contents of static and etc
We don't want to copy static and etc into the static and etc dirs
creating static/static and etc/etc.

It's possible this could also be done with trailing /'s?

Change-Id: I054158a677261743ed67107894e207f7b615ea7f
2020-03-20 03:25:02 +00:00
Monty Taylor
2e5b95a7f3 Add flag to allow skipping docker-compose up
For our rollout, we need to be able to run this without actually
running the up.

Also, split out startup tasks so that we can run them from a
dedicated start playbook by themselves.

Change-Id: I08d994e496fbd8d5adbfa1ce344b0ae52f46535c
2020-03-19 15:51:33 -05:00
Monty Taylor
e5e925d715 Switch back to docker for gerrit and nodepool-builder
We rolled out review-dev with podman and it worked fine for us. It
worked less fine for nodepool-builder, although we still might be
able to solve it. Maybe right now isn't the time to do this switch.
Gitea, gitea-lb and zuul-registry all use docker instead of podman.

The only thing running with podman right now is review-dev. We can
do a manual cleanup of podman there before runnign this to keep
things simple:

  - stop gerrit service
  - uninstall podman and podman-compose
  - uninstall podman ppa config
  - uninstall pip3

Then let ansible install docker and docker compose up.

Story: #2007407
Task: #39062
Change-Id: I9bf99b18559d49d11ba99a96f02a4a45a4f65a86
2020-03-15 23:26:49 +00:00
Monty Taylor
bbe8086726 Use LE certs for Apache
We're getting LE certs for the hosts now, use them in the apache
config. Also add the redirects.

Change-Id: I67d33b4c542182a2474ac0d2416357541b1c3a47
2020-02-13 10:31:59 -06:00
Monty Taylor
4de5f79599 Add Apache to Ansible for Gerrit
When we run gerrit, we also need to run Apache.

Change-Id: Ia2f1494808bd29d83e041e224cb2eb5fc406a93b
2020-02-03 07:57:36 -06:00
Monty Taylor
4449612d20 Plumb through secure.config contents
While we're in there - fix a misspelling.

Remove auth.restTokenPrivateKey from config file. It hasn't been
used since 2.6: https://gerrit-review.googlesource.com/c/gerrit/+/70770

Change-Id: I94405cf870d57780b86f30c2bddb573ff15c05bc
2020-01-07 17:36:10 -05:00
Monty Taylor
a3d91d4df3 Clean up review comments
Make image and volume list in compose file templated.

Rename the gerrit-podman directory to not be based on tool.

Make sure we run the job on changes to the playbooks.

Update the job name - it's not just review-dev anymore.

Change-Id: I0341fa95caff656a2176cc2026ec0ac8903fb24e
2019-12-17 08:13:34 -05:00
Monty Taylor
cc65640ef4 Add replication config to gerrit ansible
We need to configure replication.

Change-Id: I1bb61969a5adf37b07e43701d9a867b409a5ccf0
2019-12-17 08:13:34 -05:00
Monty Taylor
0a208bd1a9 Add launchpadlib credentials to gerrit ansible
Gerrit needs to be able to update bugs on launchpad. For that we need
credentials.

Change-Id: I967ee8715e03298a2ab021f37e17dc5fbde6fee6
2019-12-17 08:13:34 -05:00
James E. Blair
4f9720e76e Run a gerrit container on review-dev01
This runs gerrit in a container on review-dev01 using podman.

Remove an unused web_server.py file that we found from copying it
from puppet to ansible.

Change-Id: I399d3cf8471bc8063022b0db0ff81718b2ee2941
2019-10-29 08:29:17 +09:00