Permit use of wheel mirrors in pypi element.

pypi-mirror creates a separate mirror index for wheels (one per OS
that mirrors are built on). To be able to use it one then needs to be
able to export multiple mirrors for inclusion in pip.conf. As a drive
by I made it possible to disable the use of the pypi.python.org index
without using --offline (as --offline has larger impact).

Change-Id: I3e85a8069b18cafd7eae4cd0591821acc3b5a739
This commit is contained in:
Robert Collins 2014-03-25 16:44:46 +13:00
parent 2f24062753
commit ee18fe41da
2 changed files with 54 additions and 27 deletions

View File

@ -8,11 +8,25 @@ temporarily overwrites /root/.pip.conf and .pydistutils.cfg to use it.
When online, the official pypi.python.org pypi index is supplied as an When online, the official pypi.python.org pypi index is supplied as an
extra-url, so uncached dependencies will still be available. When offline, only extra-url, so uncached dependencies will still be available. When offline, only
the mirror is used - be warned that a stale mirror will cause build failures. the mirror is used - be warned that a stale mirror will cause build failures.
To disable the pypi.python.org index without using --offline (e.g. when working
behind a corporate firewall that prohibits pypi.python.org) set
DIB\_NO\_PYPI\_PIP to any non-empty value.
To use an arbitrary mirror set PYPI\_MIRROR\_URL=http[s]://somevalue/ To use an arbitrary mirror set PYPI\_MIRROR\_URL=http[s]://somevalue/
[pypi-mirror](https://git.openstack.org/cgit/openstack-infra/pypi-mirror) can be useful in making a Additional mirrors can be added by exporting PYPI\_MIRROR\_URL\_1=... etc. Only
partial PyPI mirror suitable for building images. For instance: the one mirror can be used by easy-install, but since wheels need to be in the
first mirror to be used, the last listed mirror is used as the pydistutils
index. NB: The sort order for these variables is a simple string sort - if
you have more than 9 additional mirrors, some care will be needed.
A typical use of this element is thus:
export PYPI\_MIRROR\_URL=http://site/pypi/Ubuntu-13.10
export PYPI\_MIRROR\_URL\_1=http://site/pypi/
[pypi-mirror](https://git.openstack.org/cgit/openstack-infra/pypi-mirror) can
be useful in making a partial PyPI mirror suitable for building images. For
instance:
* sudo apt-get install libxml2-dev libxslt-dev libmysqlclient-dev libpq-dev \ * sudo apt-get install libxml2-dev libxslt-dev libmysqlclient-dev libpq-dev \
libnspr4-dev pkg-config libsqlite3-dev libzmq-dev libffi-dev libldap2-dev \ libnspr4-dev pkg-config libsqlite3-dev libzmq-dev libffi-dev libldap2-dev \

View File

@ -1,32 +1,45 @@
#!/bin/bash #!/usr/bin/env python
set -eu import os.path
PYPIURL=${PYPI_MIRROR_URL:-'file:///tmp/pypi'} def main():
mkdir -p ~/.pip home = os.path.expanduser("~")
backup_configs(home)
indices = []
if os.environ.get('PYPI_MIRROR_URL'):
candidates = [k for k in os.environ if k.startswith('PYPI_MIRROR_URL')]
indices = list(map(os.environ.get, sorted(candidates)))
else:
indices = ['file:///tmp/pypi']
easy_index = indices[-1]
use_pypi_python_org = True
if os.environ.get('DIB_OFFLINE'):
use_pypi_python_org = False
if os.environ.get('DIB_NO_PYPI_PIP'):
use_pypi_python_org = False
if use_pypi_python_org:
indices.append('https://pypi.python.org/simple')
with file(home + '/.pip/pip.conf', 'wt') as output:
output.write('[global]\n')
output.write('log = %s/pip.log\n' % (home,))
output.write('index-url = %s\n' % (indices[0],))
for index in indices[1:]:
output.write('extra-index-url = %s\n' % (index,))
with file(home + '/.pydistutils.cfg', 'wt') as output:
output.write('[easy_install]\n')
output.write('index_url = %s\n' % (easy_index,))
if [ -e ~/.pip/pip.conf ]; then
mv ~/.pip/pip.conf{,.orig}
fi
if [ -e ~/.pydistutils.cfg ]; then def backup_configs(home):
mv ~/.pydistutils.cfg{,.orig} pip_dir = home + '/.pip'
fi if not os.path.exists(pip_dir):
os.mkdir(pip_dir)
if os.path.exists(pip_dir + '/pip.conf'):
os.rename(pip_dir + '/pip.conf', pip_dir + '/pip.conf.orig')
if os.path.exists(home + '/.pydistutils.cfg'):
os.rename(home + '/.pydistutils.cfg', home + '/.pydistutils.cfg.orig')
if [ -z $DIB_OFFLINE ]; then
ONLINE="extra-index-url = https://pypi.python.org/simple"
else
ONLINE=""
fi
cat <<EOF > ~/.pip/pip.conf if __name__ == '__main__':
[global] main()
index-url = $PYPIURL
$ONLINE
log = $HOME/pip.log
EOF
cat <<EOF > ~/.pydistutils.cfg
[easy_install]
index_url = $PYPIURL
EOF