system-config/doc/source/gitea.rst
Clark Boylan 6f178c2737 Add docs on restoring a gitea repository
We have discovered that it is possible for a gitea repository to be come
corrupted. Since gitea is not the source of truth the easiest way to
handle this is to replace the repo with a new empty repository and have
Gerrit replicate back to it. This adds documentation that walks through
the process of doing this.

Change-Id: Ief990adaaf3cbb3c748bc9ee6ceb466a1104915a
2022-03-02 12:03:01 -08:00

181 lines
5.7 KiB
ReStructuredText

:title: Gitea
.. _gitea:
Gitea
#####
Gitea is running on opendev.org
At a Glance
===========
:Hosts:
* https://opendev.org/
* gitea*.opendev.org
:Ansible:
* :git_file:`playbooks/roles/gitea`
* :git_file:`playbooks/roles/haproxy`
:Configuration:
* :git_file:`inventory/service/group_vars/gitea-lb.yaml`
:Projects:
* https://gitea.io/
:Bugs:
* https://storyboard.openstack.org/#!/project/748
* https://github.com/go-gitea/gitea/issues
Overview
========
The OpenDev Git repositories are hosted on a pool of servers. They
are served via https using Gitea behind HAProxy which handles load
balancing across the nodes.
.. _gitea-maintenance:
Backend Maintenance
===================
To temporarily remove a git backend from the HAProxy load balancer,
you can put it in "maintenance" mode. This can be done interactively
on the HAProxy host. Note that long-term changes to the topology
should be made via configuration management. These commands must be
run as root.
To see the current status of all servers::
echo "show stat" | socat /var/haproxy/run/stats stdio
To disable a server (eg, gitea08)::
echo "disable server balance_git_http/gitea08.opendev.org" | socat /var/haproxy/run/stats stdio
echo "disable server balance_git_https/gitea08.opendev.org" | socat /var/haproxy/run/stats stdio
To re-enable a server::
echo "enable server balance_git_http/gitea08.opendev.org" | socat /var/haproxy/run/stats stdio
echo "enable server balance_git_https/gitea08.opendev.org" | socat /var/haproxy/run/stats stdio
To run these commands and others interactively, issue the prompt
command to haproxy::
socat readline /var/haproxy/run/stats
prompt
Deploy a New Backend
====================
Our gitea servers do maintain a small amount of state (they remember
repo rename redirects) so there is a small amount of process required
to deploy a new Gitea backend.
To deploy a new Gitea backend we add it to the ansible inventory, but
exclude it from the manage-projects.yaml playbook. This will fully
provision an empty Gitea server with running Gitea and database processes.
Then we can manually restore the database from another node, create all of
the bare git repos, and replicate all of the repo content.
When these steps are done the new gitea backend can be added to the
haproxy config and its exclusion from "Create repos on gitea servers"
can be removed.
Restore the Gitea Database
--------------------------
The first step in restoring the database is to determine which container
is running the database::
docker ps -a
Make note of the container id for the container running the mariadb image.
Next we stop the gitea services::
docker stop $GITEA_CONTAINER_IDS
With services stopped and the container id captured we are ready to
restore the database. First find the database to restore; it is
backed up in ``/var/backups/gitea-mariadb`` on gitea hosts. You can
copy and uncompress it.
Then restore the database (note we use ``docker exec`` and not
``docker-compose exec`` for performance reasons)::
docker exec -i $DB_CONTAINER_ID bash -c '/usr/bin/mysql -uroot -p"$MYSQL_ROOT_PASSWORD"' < /root/gitea-mariadb.sql
When that is completed you can restart the docker containers that were
stopped::
# Check that containers are still stopped
docker ps -a
docker start $GITEA_CONTAINER_IDS
Create All Bare Git Repos
-------------------------
Gitea's admin dashboard includes a useful button to create all missing
git repos. At this point in the deployment we have recovered the DB
contents so Gitea knows there are missing repos and will happily create
empty replacements if we ask it to.
Login to Gitea as ``root`` via ``https://giteaXY.opendev.org:3000/user/login``.
The credentials can be found in hiera's group vars for the gitea group.
Navigate to ``https://giteaXY.opendev.org:3000/admin`` and click the run
button for ``Reinitialize all missing Git repositories for which records exist``.
Replicate Git Repo Content from Gerrit
--------------------------------------
First we must accept the RSA host key for the Gerrit server (not the
ecdsa key)::
gerrit2@review.opendev.org$ ssh -o HostKeyAlgorithms=ssh-rsa -p222 git@giteaXY.opendev.org
Then we can ask Gerrit to replicate all repo content into our new empty repos::
ssh -p 29418 $USER@review.opendev.org gerrit replication start --url giteaXY.opendev.org
You can monitor the progress of this with::
ssh -p 29418 $USER@review.opendev.org gerrit show-queue
Once this is complete, add the server back into the haproxy as
discussed above.
Reset a Corrupted Git Repo
==========================
It is possible for the repos Gitea hosts to become corrupted. Since Gerrit
is our source of truth the easiest way to handle this is have Gerrit replicate
the data back to Gitea. Unfortunately, replication will fail with a corrupted
repo on the destination. To work around this we replace the repo with a new
empty bare repository and then replicate.
First thing to do is :ref:`remove the backend from the haproxy rotation<gitea-maintenance>`.
Next we need to stop gitea on the backend::
cd /etc/gitea-docker
docker-compose down
Then move aside the old repo and replace it with a new empty bare repo::
cd /root/corrupted_repos
mv /var/gitea/data/git/repositories/org/example.git ./example.git.bak
git init --bare example.git
chown -R 1000:1000 example.git
mv ./example.git /var/gitea/data/git/repositories/org/example.git
Now start the gitea service back up again::
cd /etc/gitea-docker
docker-compose up -d mariadb gitea-web
# Wait for web to be responseive
docker-compose up -d gitea-ssh
Finally trigger a Gerrit replication::
ssh -p 29418 user.admin@review.opendev.org replication start --url $giteabackendname org/example
Once replication is complete you can add the backend to the haproxy rotation.