A recent change in pip wheel cache behavior had upstream pip indicating
that we really should be using pip wheel instead. The reason we weren't
using pip wheel appears to be that we wanted to infer what top level
wheel to install via contents of a dir separate from our wheel output
dir/wheel cache. Using pip wheel implies everything gets flattened into
one location. We deal with this by having the build tool write all of
the top level wheels we care about into a separate location. Later we
can install all of the top level wheels while pointing find links at the
larger set of deps in the dir created by pip wheel.
Change-Id: Id9c674c1ec6fe5e72534549082e3adda9e286fd5
Python 3.11 has been released. Once the parent commit of this commit
lands we will have removed our python3.8 images making room for
python3.11 in our image list. Add these new images which will make way
for running and testing our software on this new version of python.
Change-Id: Idcea3d6fa22839390f63cd1722bc4cb46a6ccd53
This is useful for debugging when pip does updates and we need to be
sure that we ran with a new (or old) version of pip.
Change-Id: I556bb68e255ee0bdbcbd2c72dc537e2a6c7d64b6
It is a good idea ot periodically update our base python images. Now is
a good time to do it as we've got debian bullseye updates and python
minor releases. The bullseye updates fix a glibc bug that was affecting
Ansible in the zuul images. With this update we'll be able to remove the
workaround for that issue.
We also update the builder image's apt-get process to include a clean to
match tbe base image. This is more for consistency than anything else.
Finally update job timeouts for builds as it seems we occasionally need
more time particularly for emulated arm64 builds.
Change-Id: I31483ff434f19f408aef3b63cb2cd24044a8bf29
Our python builder's assemble script was used as guidance for building
wheels to workaround git's new permission checking behavior. When it was
called out that setup.py is deprecated and using `build` would be better
we decided that using setup.py was fine since all our images currently
use it.
Now that we've cleaned up our old buster images and python3.7 it is the
perfect time to update to modern python build tooling as well. Update
the assembel script to use `build` instead of setup.py.
As far as I can tell `build` maintains the old setup.py behavior of
emitting artifacts to the output directory separate of any dependency
packages that may have been pulled down to build the package. As the old
comment indicates this behavior is desireable.
It may also be worth testing this update against Zuul and other
relatively complicated python packages to ensure we haven't changed
behavior in some unexpected way.
Change-Id: Ib2a7335219842413e507c0593ceed187819d83b3
Now that we've cleaned up the old unused images we can look forward to
new Python. Add Python 3.10 base images based on Bullseye.
As part of this process we update the default var values in our
Dockerfiles to set Bullseye and Python3.10 as our defaults as these
should be valid for some time. We also tidy up some yaml anchor names to
make future copy and paste for new versions of images easier to perform
text replacement on.
Change-Id: I4943a9178334c4bdf10ee5601e39004d6783b34c
There's some more work before our consumers can switch to bullseye.
To make this process more tractable, revert the recent backport
addition, and specify that we want bullseye images from upstream.
That gets us back to where we were at the start of this. Next,
we can start building 2x images of python-base/builder and tag
them with bullseye or buster. Then the consumers can specify
which tags, then start switching.
Revert "Add backports repos to base and builder images"
This reverts commit b217e38904da3ccab6eb96251376f1635ee55d21.
Revert "Update matrix-eavesdrop for bullseye"
This reverts commit fc38c6975367c09d003474ebd7bfefc465459a06.
Change-Id: Id21681342fe5268296128c1b09436a80c46e3169
These don't install anything by default, but allow people
to easily opt-in to a backport package if they need to without
lots of extra lines of boilerplate.
The base python image is on bullseye now instead of buster. That
means the libffi version is 7 not 6.
Change-Id: I0e0c2669d838fb622422f696f73e96e409157270
We had an image promote failure for python-base:3.8. Due to docker
hub making it very difficult to know if old tags have been cleaned up we
are not sure that reenqueing the previous chagne to zuul will do the
right thing. It may downgrade the latest tag on some of our images. To
avoid any confusion over what is latest we just have zuul build new
images and promote them again.
Change-Id: Iaa859396030f1110b43788e73e9644a97e2ada9b
We are looking ahead to rebuilding a number of our images for services
like Zuul, Gerrit, and Gitea to do things like check zuul v5 efforts,
fix gerrit bugs, and upgrade gitea to a new version. Ensuring that we
have an up to date base platform seems like a good idea as a result.
Change-Id: I4262b8aa1759eaae85e5429c5a5097397152afa5
It seems I added this --force with the original commit
(I4943ae723b06b0ad808e7c7f20788109e21aa8bf) but I'm not really sure
why. If we have built any siblings their wheels should have higher
version numbers (e.g. like pbr versioning with a "dev" on it).
Thus we shouldn't need to force the wheels to be installed. The
--force here causes a lot of uninstalls that take up quite a bit of
time, especialy under emulation.
Change-Id: I88b824058dc1cee90bfe4c8c4fd43a86472bc478
Because Id68080575a30e4a08c99df0af603fbb65a0983bd didn't touch any of
the docker files (but just added new 3.9 builds) they didn't get
promoted. Update timestamp to trigger this.
Change-Id: I6bf33936d4da773329900a2a52d09654087313d4
This change adds comments to the python-base and python-builder
dockerfiles to force a rebuild of these images. We do this periodically
to pull in updates.
Change-Id: I109a46603a74a376cc36fdfbd35734f6dc835abe
The pip install -f flag is for find-links which is a list of locations
to look for python packages. What we hvae at this path for extras
installation is a list of packages themselves not locations to find
them. We need to use the -r flag for requirements lists to specify this
instead.
This change should update our zuul and nodepool images to include useful
debugging extras.
Change-Id: I647bb835d0c85c3772e1593866a54cfc5ea1db2f
python-builder is for building docker images. As such, we almost
certainly don't want to be prompted for debconf questions.
Set DEBIAN_FRONTEND=noninteractive so that things like krb5-user
can be installed.
Change-Id: Ieff0555a57e7dbc1b7c4a059b1d0351d016a0c86
Our docker python-base image is a python3 image where python and python3
point to the same version of python which happens to be a 3.x depending
on the specific base image.
One of the things that came up at the PTG is that we'd like to start
identifying where we use python2 in order to port to python3. That is
made a bit more difficult when we use `python` assuming it is `python3`
as we do in our docker images.
Update the images to use `python3` to make this a lot more explicit and
clear that we don't need to port these items.
Change-Id: I54e8f128f0cb8fbbdf54e3748384470262bef3a9
If the extras includes a '#' inside of it, it means that it is
probably followed up with a comment such as license for that package and
it will fail to install because pip will try to use that literal #.
Change-Id: I31c5ffe3dda3ced4b6a412cc955c73321d4af507
This lets you build an image to install a package or set of packages
without having to build a fake pbr install.
Change-Id: I6c171881283b89a14a532f31aedb33e223a0ea1f
We produce lists of extras requirements to install, but installing
them takes an extra step. Allow it to be done all at once by accepting
names of extras on the install-from-bindep command line.
Change-Id: Ie75b9a668569c759f78b70617e311ed6f025039a
base and builder are currently pinned to 3.7, which makes it hard
to consider upgrading to python-3.8. To help with that, make
python-3.7 and python-3.8 images, but point latest as 3.7 for the
time being. Then add version-specific provides and FROM lines
so that we can start being deliberate and clear about our version
choice.
Change-Id: Ibf1d846d5c4a005547785124567ce2900e272a7a
We disable these on our VM images, so might as well make our
container images follow suit. Could make them a smidge smaller.
Also do it in the gerrit-base image.
Change-Id: Iba60cf5c7009d57c4910f9e4464aff9231598ad6
There was an issue in pip that prevented correctly caching locally
built wheels [1]. This has been fixed in recent pip versions so
upgrade pip in both images so image caching works correctly. This is
needed to unbreak nodepool images that fail to install the locally
built netifaces package.
[1] https://github.com/pypa/pip/issues/6852
Change-Id: Ibbe12bcc53253a80d0bafa3d09a20c49a3a2b784
To make it clear that docker hub is but one of many possible registries,
update our usage of FROM and image: lines to include docker.io in the
path.
There are a few other FROM lines for the gitea images which are handled
in a separate stack.
Change-Id: I6fafd5f659ad19de6951574afc9a6b6a4cf184df
We need to run bindep before installing git, because otherwise if
a project needs git in its bindep, it won't show up because it'll
be on the build host.
Split the function in two and call them before and after the git
installation.
Change-Id: I316b1bc643eb9293500b31e676361eec7060701d
In the dependent change, the docker roles will add sibling packages to
the .zuul-siblings directory of the checked-out source.
Refactor the "assemble" script to handle this. Essentially we build
the wheel for "." and then iterate over ZUUL_SIBLINGS subdirectories
(set in a --build-arg by the role in dependent change) to also build
the sibling packages. Note we concatenate the bindep.txt files, so
that we end up with the complete package list required by the main
code and its dependencies.
"install-from-bindep" now installs all the wheels, using --force to
make sure we re-install the speculatively built packages.
This means that a single Dockerfile works under Zuul when
ZUUL_SIBLINGS is set, pointing to Zuul's checkouts; but it also works
stand-alone -- in this case ZUUL_SIBLINGS is empty and we just install
from upstream as usual.
Depends-On: https://review.opendev.org/696987
Change-Id: I4943ae723b06b0ad808e7c7f20788109e21aa8bf
Apparently, an update to the python-base image caused a new image
to be uploaded to dockerhub. Because python-builder was not updated,
it is running a slightly older base image which we believe is causing
issues with python wheel packaging due to possible different python
versions.
Change-Id: I38948882131e30c7358a970292621b0280b75aac
This helps to reduce the image size but not creating /root/.cache folder
for pip related files.
Change-Id: I1162d7d6fd2e4c7dd9cde44c964aec610a4dd6d3
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
By removing /var/lib/apt/lists it reduces the image size.
Change-Id: Icbe118a2725700b9e5e8da97d062161c5b9a5d2d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
There's no real need to tie these together into a multi-stage
Dockerfile as they don't really share anything. Split them.
Change-Id: Ifd7ccadcd8048eeb57797d60356aec2f9f0d2c80
Depends-On: https://review.openstack.org/641805
It really doesn't actually matter, but if we do the apt-get install
before the COPY, it can increase the caching efficiency by a billionth
of a percent.
Depends-On: https://review.openstack.org/632528
Change-Id: I875812c956af1c4739e0986d267a47652bb5e6f3
There are cases when an issue has been fixed upstream in a
dependency but not yet released and it is desirable to install
that version in a built image. pbr does not support such urls
in requirements files because releasing software that depends
on unreleased software is quite dangerous.
Update assemble and install-from-bindep to install -r requirements.txt
before installing the actual software so that image builds with
such requirements lines will work.
Change-Id: Ie17a66bdc4ea0609ce4e66051f659cbd829fdd5d
This is a builder image intended to make it easy to build
images for projects that use pbr and bindep.
Also, build a python-base image for use with python-builder
pbrx installed dumb-init in a base image and used it as an
entrypoint for the image. This is a good pattern that we should
continue. Make a python-base image that can be used in the FROM
line of consuming final images.
Change-Id: I8da39e680e9c5ca0d8f603a2645e83fc3be87a8b
