diskimage-builder/elements/pypi
Ian Wienand 672705831f Add a best-effort sudo safety check
As motivation for this; we have had two breakouts of dib in recent
memory.  One was a failure to unmount through symlinks in the core
code (I335316019ef948758392b03e91f9869102a472b9) and the other was
removing host keys on the build-system
(Ib01d71ff9415a0ae04d963f6e380aab9ac2260ce).

For the most part, dib runs unprivileged.  Bits of the core code are
hopefully well tested (modulo bugs like the first one!).  We give free
reign inside the chroot (although there is still some potential there
for adverse external affects via bind mounts).  Where we could be a
bit safer (and could have prevented at least the second of these
breakouts) is with some better checking that the "sudo" calls
*outside* the chroot at least looked sane.

This adds a basic check that we're using chroot or image paths when
calling sudo in those parts of elements that run *outside* the chroot.
Various files are updated to accomodate this check; mostly by just
ignoring it for existing code (I have not audited these calls).

Nobody is pretending this type of checking makes dib magically safe,
or removes the issues with it needing to do things as root during the
build.  But this can help find egregious errors like the key removal.

Change-Id: I161a5aea1d29dcdc7236f70d372c53246ec73749
2016-05-09 15:41:38 +10:00
..
extra-data.d Add a best-effort sudo safety check 2016-05-09 15:41:38 +10:00
post-install.d Standarise tracing for scripts 2015-02-12 10:41:32 +11:00
pre-install.d Allow setting DIB_PIP_RETRIES 2015-02-19 15:29:36 -08:00
README.rst Allow setting DIB_PIP_RETRIES 2015-02-19 15:29:36 -08:00

pypi

Inject a PyPI mirror

Use a custom PyPI mirror to build images. The default is to bind mount one from ~/.cache/image-create/pypi/mirror into the build environment as mirror URL file:///tmp/pypi. The element temporarily overwrites /root/.pip.conf and .pydistutils.cfg to use it.

When online, the official pypi.python.org pypi index is supplied as an extra-url, so uncached dependencies will still be available. When offline, only the mirror is used - be warned that a stale mirror will cause build failures. To disable the pypi.python.org index without using --offline (e.g. when working behind a corporate firewall that prohibits pypi.python.org) set DIB_NO_PYPI_PIP to any non-empty value.

To use an arbitrary mirror set DIB_PYPI_MIRROR_URL=http[s]://somevalue/

Additional mirrors can be added by exporting DIB_PYPI_MIRROR_URL_1=... etc. Only the one mirror can be used by easy-install, but since wheels need to be in the first mirror to be used, the last listed mirror is used as the pydistutils index. NB: The sort order for these variables is a simple string sort - if you have more than 9 additional mirrors, some care will be needed.

You can also set the number of retries that occur on failure by setting the DIB_PIP_RETRIES environment variable. If setting fallback pip mirrors you typically want to set this to 0 to prevent the need to fail multiple times before falling back.

A typical use of this element is thus: export DIB_PYPI_MIRROR_URL=http://site/pypi/Ubuntu-13.10 export DIB_PYPI_MIRROR_URL_1=http://site/pypi/ export DIB_PYPI_MIRROR_URL_2=file:///tmp/pypi export DIB_PIP_RETRIES=0

[devpi-server](https://git.openstack.org/cgit/openstack-infra/pypi-mirro://pypi.python.org/pypi/devpi-server) can be useful in making a partial PyPI mirror suitable for building images. For instance:

  • Re-export your variables to point at the new mirror:

    export DIB_PYPI_MIRROR_URL=http://machinename:3141/ unset DIB_PYPI__MIRROR_URL_1 unset DIB_PYPI__MIRROR_URL_2

The next time packages are installed, they'll be cached on the local devpi server; subsequent runs pointed at the same mirror will use the local cache if the upstream can't be contacted.

Note that this process only has the server running temporarily; see [Quickstart: Permanent install on server/laptop](http://doc.devpi.net/latest/quickstart-server.html) guide from the devpi developers for more information on a more permanent setup.