b8703c3f54
I had to figure out this command to restore some of our reprepro mirrors, add notes on it. Cleanup some other things I noticed; quotes around REPREPRO command, clarify the situation with pointing mirrors at the RO/RW volumes and refactor discussion. Change-Id: I3e3f763929bf74e0c4faf29b13973dcf1df36975
173 lines
6.2 KiB
ReStructuredText
173 lines
6.2 KiB
ReStructuredText
:title: reprepro
|
|
|
|
.. _reprepro:
|
|
|
|
Reprepro
|
|
########
|
|
|
|
Debian package mirroring tool
|
|
|
|
At a Glance
|
|
===========
|
|
|
|
:Hosts:
|
|
* http://mirror-update.openstack.org
|
|
:Puppet:
|
|
* :file:`modules/openstack_project/manifests/mirror_update.pp`
|
|
:Projects:
|
|
* https://mirrorer.alioth.debian.org
|
|
:Documentation:
|
|
* http://git.debian.org/?p=mirrorer/reprepro.git;a=blob_plain;hb=HEAD;f=docs/manual.html
|
|
* https://github.com/esc/reprepro/blob/master/docs/recovery
|
|
:Bugs:
|
|
* https://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=reprepro;dist=unstable
|
|
|
|
Overview
|
|
========
|
|
|
|
reprepro is the tool we use to mirror Debian repositories (including
|
|
Ubuntu) to the AFS mirrors.
|
|
|
|
Repository signing
|
|
==================
|
|
|
|
Note our repositories are not signed. ``apt`` will require
|
|
``--no-check-gpg`` or similar settings in configuration to use
|
|
OpenStack mirrors.
|
|
|
|
Normal operation
|
|
================
|
|
|
|
Repository syncs are driven from ``cron`` on the
|
|
``mirror-update.openstack.org`` host using the
|
|
``/usr/local/bin/reprepro-mirror-update`` script. Repositories will
|
|
update, remove old references and perform the ``vos release``.
|
|
|
|
Advanced Recovery Techniques
|
|
============================
|
|
|
|
For a small repository, corruption is probably best handled by
|
|
removing the entire repository and re-syncing. This is undesirable
|
|
for larger repositories, however.
|
|
|
|
.. note::
|
|
|
|
Be careful with ``vos release`` which is done as part of
|
|
``/usr/local/bin/reprepo-mirror-update`` to avoid inadvertently
|
|
releasing in progress work. Also be aware the commands in that
|
|
script by default run under ``timeout`` which you may not want in
|
|
recovery.
|
|
|
|
Corrupt ``reprepo`` databases will halt mirroring with often obscure
|
|
symptoms. For example, this has been seen in production with
|
|
``reprepo`` ending up hung in an silent infinite loop. In this case,
|
|
using ``strace`` revealed the last operation was on a file-descriptor
|
|
related to a ``.db`` file, which gave a clue the databases were
|
|
corrupt. Other failures may be possible, of course.
|
|
|
|
The following assumes you have a root shell with the correct AFS
|
|
permissions for the mirror volumes, drop into something like::
|
|
|
|
k5start -t -f /etc/reprepro.keytab service/reprepro -- bash
|
|
|
|
We will use the Ubuntu repository as an example below.
|
|
|
|
In a crisis, you want to stop the cron job running to update the repo.
|
|
You can either edit it out with ``crontab -e`` and put the host in the
|
|
emergency file (so puppet doesn't replace it) or, in a pinch, take the
|
|
lock in a infinite loop like ::
|
|
|
|
flock -n /var/run/reprepro/ubuntu.lock bash -c while true; do sleep 1000; done
|
|
|
|
Firstly check in ``dmesg`` for AFS related errors. It is quite likely
|
|
any corruption has happened due to issues at this layer, so ensure
|
|
stability here before continuing to further recovery.
|
|
|
|
The databases are in the ``db`` directory in the mirror::
|
|
|
|
# ls /afs/.openstack.org/mirror/ubuntu/db
|
|
checksums.db contents.cache.db packages.db references.db release.caches.db version
|
|
|
|
It is best to make backup copies before any recovery operations.
|
|
Although AFS /should/ keep up, you should do any recovery of the
|
|
``db`` directory on a local copy to avoid any intermittent issues
|
|
there further corrupting the database, then copy back the updated
|
|
files when complete.::
|
|
|
|
# cp -r /afs/.openstack.org/mirror/ubuntu/db ~/db
|
|
|
|
For convenience, setup the common ``reprepo`` options for verbose
|
|
logging, the configuration directory and to work on the local
|
|
database::
|
|
|
|
# export REPREPRO="reprepro -VVV --confdir /etc/reprepro/ubuntu --dbdir ~/db"
|
|
|
|
From the upstream recovery document, the ``references.db`` can be
|
|
removed and recreated quickly with::
|
|
|
|
$REPREPRO rereference
|
|
|
|
The ``checksums.db`` can also be recreated. You can rebuild with::
|
|
|
|
cd /afs/.openstack.org/mirror/ubuntu
|
|
find -type f -printf "pool/%P\n" > /tmp/file-list
|
|
$REPREPRO -b . _detect < /tmp/file-list
|
|
|
|
* This will take several hours (~6 hours in 2017) as it touches all
|
|
the repo files.
|
|
|
|
Note that if the ``.deb`` files on disk are corrupt, this may lead to
|
|
errors on update about mismatching checksums which have been stored in
|
|
the database. Likely you want to remove these files from disk and
|
|
from the checksums database with a command similar to::
|
|
|
|
$REPREPRO _forget pool/main/p/package/the_package_1.2.3.deb
|
|
rm pool/main/p/package/the_package_1.2.3.deb
|
|
|
|
They should come back with the next update.
|
|
|
|
In some situations where things are very out of sync, it may be easier
|
|
to remove and replace an entire section of the repository. For
|
|
example, if during updates files within ``xenial-security`` are seen
|
|
to be corrupt, you can remove ``xenial-security`` from
|
|
``/etc/reprepro/ubuntu/distributions`` and run the following::
|
|
|
|
# remove old
|
|
$REPREPRO --delete clearvanished
|
|
# run an update
|
|
$REPREPRO update
|
|
|
|
You can then re-add the entries and run another update, which should
|
|
resync everything from fresh.
|
|
|
|
You may also see errors relating to individual packages not being
|
|
referenced correctly::
|
|
|
|
checking references to 'bionic|main|arm64' for 'texlive-latex-base': pool/main/t/texlive-base/texlive-latex-base_2017.20180305-1_all.deb
|
|
Missing reference to 'pool/main/t/texlive-base/texlive-latex-base_2017.20180305-1_all.deb' by 'bionic|main|arm64'
|
|
...
|
|
There have been errors!
|
|
|
|
In this case, the ``_addreference`` command can be useful. The
|
|
parameters are the *filekey*, which is the path to the file, and the
|
|
*identifier*, which is the tuple ``bionic|main|arm64`` above. To
|
|
restore the reference try::
|
|
|
|
# $REPREPRO _addreference pool/main/t/texlive-base/texlive-latex-base_2017.20180305-1_all.deb 'bionic|main|arm64'
|
|
Adding reference to 'pool/main/t/texlive-base/texlive-latex-base_2017.20180305-1_all.deb' by 'bionic|main|arm64'
|
|
|
|
Remember to put the databases back in place::
|
|
|
|
# mv /afs/.openstack.org/mirror/ubuntu/db /afs/.openstack.org/mirror/ubuntu/db.old
|
|
# cp -r ~/db /afs/.openstack.org/mirror/ubuntu/
|
|
|
|
To stage a recovery prior to release, you can modify the
|
|
``mirror_root`` argument in ``openstack_project::mirror`` puppet to
|
|
point Apache to the RW mirror ``/afs/.openstack.org/mirror`` where
|
|
fixes are deployed, rather than the released RO
|
|
``/afs/openstack.org/mirror``. This way you can avoid having to
|
|
release the RO mirror and switch back quickly if things don't work.
|
|
|
|
When fixed, you can either manually run ``vos release``, or restore
|
|
cron and let the next ``reprepro-mirror-update`` run do it.
|