Gitea is running on opendev.org
The OpenDev Git repositories are hosted on a pool of servers. They are served via https using Gitea behind HAProxy which handles load balancing across the nodes.
To temporarily remove a git backend from the HAProxy load balancer, you can put it in "maintenance" mode. This can be done interactively on the HAProxy host. Note that long-term changes to the topology should be made via configuration management. These commands must be run as root.
To see the current status of all servers:
echo "show stat" | socat /var/haproxy/run/stats stdio
To disable a server (eg, gitea08):
echo "disable server balance_git_http/gitea08.opendev.org" | socat /var/haproxy/run/stats stdio echo "disable server balance_git_https/gitea08.opendev.org" | socat /var/haproxy/run/stats stdio
To re-enable a server:
echo "enable server balance_git_http/gitea08.opendev.org" | socat /var/haproxy/run/stats stdio echo "enable server balance_git_https/gitea08.opendev.org" | socat /var/haproxy/run/stats stdio
To run these commands and others interactively, issue the prompt command to haproxy:
socat readline /var/haproxy/run/stats prompt
Our gitea servers do maintain a small amount of state (they remember repo rename redirects) so there is a small amount of process required to deploy a new Gitea backend.
To deploy a new Gitea backend we add it to the ansible inventory, but exclude it from the manage-projects.yaml playbook. This will full provision an empty Gitea server with running Gitea and database processes. Then we can manually restore the database from another node, create all of the bare git repos, and replicate all of the repo content.
When these steps are done the new gitea backend can be added to the haproxy config and its exclusion from "Create repos on gitea servers" can be removed.
The first step in restoring the database is to determine which container is running the database:
docker ps -a
Make note of the container id for the container running the mariadb image.
Next we stop the gitea services:
docker stop $GITEA_CONTAINER_IDS
With services stopped and the container id captured we are ready to restore the database. First find the database to restore; it is backed up in
/var/backups/gitea-mariadb on gitea hosts. You can copy and uncompress it.
Then restore the database (note we use
docker exec and not
docker-compose exec for performance reasons):
docker exec -i $DB_CONTAINER_ID bash -c '/usr/bin/mysql -uroot -p"$MYSQL_ROOT_PASSWORD"' < /root/gitea-mariadb.sql
When that is completed you can restart the docker containers that were stopped:
# Check that containers are still stopped docker ps -a docker start $GITEA_CONTAINER_IDS
Gitea's admin dashboard includes a useful button to create all missing git repos. At this point in the deployment we have recovered the DB contents so Gitea knows there are missing repos and will happily create empty replacements if we ask it to.
Login to Gitea as
https://giteaXY.opendev.org:3000/user/login. The credentials can be found in hiera's group vars for the gitea group. Navigate to
https://giteaXY.opendev.org:3000/admin and click the run button for
Reinitialize all missing Git repositories for which records exist.
First we must accept the RSA host key for the Gerrit server (not the ecdsa key):
email@example.com$ ssh -o HostKeyAlgorithms=ssh-rsa -p222 git@giteaXY.opendev.org
Then we can ask Gerrit to replicate all repo content into our new empty repos:
ssh -p 29418 $USER@review.opendev.org gerrit replication start --url giteaXY.opendev.org
You can monitor the progress of this with:
ssh -p 29418 $USER@review.opendev.org gerrit show-queue
Once this is complete, add the server back into the haproxy as discussed above.