From 23dc00f2e4e14f19d78b224a34c6d252466732ce Mon Sep 17 00:00:00 2001
From: Clark Boylan <clark.boylan@gmail.com>
Date: Fri, 11 Jul 2014 09:39:28 -0700
Subject: [PATCH] Document use of bandersnatch for pypi mirroring

We use bandersnatch to mirror pypi now and that should be documented.
Add all the usual header information with quick links to useful places
like local config and upstream bug trackers. Also, note how we run
bandersnatch and details on detecting and correcting stale packages on
pypi preventing local syncs.

Change-Id: I98db04c19c427335a4786a7923982fabcf57051b
---
 doc/source/bandersnatch.rst | 51 +++++++++++++++++++++++++++++++++++++
 doc/source/systems.rst      |  1 +
 2 files changed, 52 insertions(+)
 create mode 100644 doc/source/bandersnatch.rst

diff --git a/doc/source/bandersnatch.rst b/doc/source/bandersnatch.rst
new file mode 100644
index 0000000000..99a3f5e659
--- /dev/null
+++ b/doc/source/bandersnatch.rst
@@ -0,0 +1,51 @@
+:title: Bandersnatch
+
+.. _bandersnatch:
+
+Bandersnatch
+############
+
+A pypi mirror tool
+
+At a Glance
+===========
+
+:Hosts:
+  * http://pypi.openstack.org
+:Puppet:
+  * :file:`modules/openstack_project/manifests/static.pp`
+:Projects:
+  * https://pypi.python.org/pypi/bandersnatch
+:Documentation:
+  * https://pypi.python.org/pypi/bandersnatch#configuration
+:Bugs:
+  * http://bugs.launchpad.net/openstack-ci
+  * https://bitbucket.org/pypa/bandersnatch/issues?status=new&status=open
+
+Overview
+========
+
+Bandersnatch is a tool we run on the static.openstack.org host to
+build a complete mirror of pypi.python.org. Cron execs bandersnatch
+on an interval with logs going to ``/var/log/bandersnatch``.
+
+Stale Packages
+==============
+
+There is an issue with pypi.python.org syncing to its CDN occasionally
+resulting in stale package artifacts. You will notice this in the
+bandersnatch logs as::
+
+  2014-07-11 01:30:04,592 INFO: Syncing package: python-novaclient (serial 1154164)
+  2014-07-11 01:30:04,592 DEBUG: Getting /pypi/python-novaclient/json (serial 1154164)
+  2014-07-11 01:30:04,599 DEBUG: Expected PyPI serial 1154164 for request https://pypi.python.org/pypi/python-novaclient/json but got 1154163
+  2014-07-11 01:30:04,599 ERROR: Stale serial for package python-novaclient
+  2014-07-11 01:30:04,599 ERROR: Stale serial for python-novaclient (1154164) not updating. Giving up.
+
+The fix for this is to issue a PURGE against the url specified above::
+
+  curl -X PURGE https://pypi.python.org/pypi/python-novaclient/json
+
+The next run of bandersnatch will sync the package. Note this PURGE
+step should be performed automatically by our bandersnatch wrapper
+script, but can be performed by hand safely if necessary.
diff --git a/doc/source/systems.rst b/doc/source/systems.rst
index 7993199d53..ec034f1e83 100644
--- a/doc/source/systems.rst
+++ b/doc/source/systems.rst
@@ -22,6 +22,7 @@ Major Systems
    planet
    puppet
    static
+   bandersnatch
    lists
    wiki
    git