Merge "Create Docker Registry with Pull Through Cache"
This commit is contained in:
Zuul
Gerrit Code Review
commit
b8d3ae4f68
@ -13,8 +13,23 @@ This will also speed up the bootstrap process as images will be downloaded only
|
||||
once. This guide assumes that you are installing this local external docker
|
||||
registry on a Linux system with Docker installed and configured.
|
||||
|
||||
Optionally, you can also create multiple registries with pull-through cache,
|
||||
skipping the need to pre-populate the registry and keeping the images and tags
|
||||
up-to-date.
|
||||
|
||||
.. rubric:: |proc|
|
||||
|
||||
.. tabs::
|
||||
.. group-tab:: Private Registry
|
||||
|
||||
.. note::
|
||||
In this section, you will pre-populate a local registry with multiple
|
||||
images. Currently, the generated images list will not contain all
|
||||
images, making it necessary to manually pull some images used by the
|
||||
kubeadm tool. Alternatively, you can follow the "pull-through cache"
|
||||
section, allowing docker to get the image from remote, if it doesn't
|
||||
exists locally.
|
||||
|
||||
#. Create folders to store your local registry images (``storage``) and to
|
||||
place setup files that will be used later on (``images``):
|
||||
|
||||
@ -64,16 +79,16 @@ registry on a Linux system with Docker installed and configured.
|
||||
registry:2
|
||||
|
||||
.. note::
|
||||
*Optional*: the ``-p`` parameter configures a mapping between the host
|
||||
port and the container port. If you want to listen on another port on
|
||||
your host, say 9000, change from ``-p 5000:5000 \`` to
|
||||
*Optional*: the ``-p`` parameter configures a mapping between the
|
||||
host port and the container port. If you want to listen on another
|
||||
port on your host, say 9000, change from ``-p 5000:5000 \`` to
|
||||
``-p 9000:5000 \``.
|
||||
|
||||
#. Create the list of images that will populate the registry.
|
||||
|
||||
Obtain the Kubernetes version your |prod| uses. This can be found in the
|
||||
``fresh_install_k8s_version`` value of the `Kubernetes versions`_ file.
|
||||
Use the branches and tags to find the value for your version.
|
||||
Obtain the Kubernetes version your |prod| uses. This can be found in
|
||||
the ``fresh_install_k8s_version`` value of the `Kubernetes versions`_
|
||||
file. Use the branches and tags to find the value for your version.
|
||||
|
||||
With the Kubernetes version, you can find the corresponding folder in
|
||||
`system images`_ and locate the ``system-images.yml`` file. This file
|
||||
@ -92,9 +107,26 @@ registry on a Linux system with Docker installed and configured.
|
||||
|
||||
curl -s ${IMAGES_YAML_RAW_FILE} | grep -v '\-\-\-' | grep -v '^#' | cut -d ':' -f2,3 | tr -d ' ' > $HOME/docker-registry/images/list.lst
|
||||
|
||||
.. note:: *Optional*: If you have a running |prod| setup, you can run
|
||||
the following to create an Ansible Playbook to get the exact images
|
||||
you will need instead:
|
||||
The expected image list will be presented in this format:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
<REGISTRY_URL/IMAGE_NAME>:<IMAGE_TAG>
|
||||
<REGISTRY_URL/IMAGE_NAME>:<IMAGE_TAG>
|
||||
<REGISTRY_URL/IMAGE_NAME>:<IMAGE_TAG>...
|
||||
|
||||
.. important::
|
||||
Due to ``kubeadm`` dynamically pulling the necessary images for
|
||||
creating the k8s cluster, based on the ``Kubernetes version`` used,
|
||||
this list doesn't contain all necessary images.
|
||||
You'll need to manually add the extra images to the ``list.lst``
|
||||
file, using `kubeadm`_. You can check the official k8s docs on how
|
||||
to get them.
|
||||
|
||||
.. note::
|
||||
*Optional*: If you have a running |prod| setup, you can run the
|
||||
following to create an Ansible Playbook to get the exact images you
|
||||
will need instead:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
@ -122,8 +154,8 @@ registry on a Linux system with Docker installed and configured.
|
||||
You will find the Kubernetes version to assign to the `K8S_VERSION`
|
||||
variable on the aforementioned `Kubernetes versions`_ file.
|
||||
|
||||
#. Create and run a script that will populate the registry based on the list
|
||||
of images:
|
||||
#. Create and run a script that will populate the registry based on the
|
||||
list of images:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
@ -169,9 +201,117 @@ registry on a Linux system with Docker installed and configured.
|
||||
$REG_SCRIPT $HOME/docker-registry/images/list.lst
|
||||
|
||||
.. note::
|
||||
The ``populate_registry.sh`` script checks if each image in the list is
|
||||
already present, which means you can update the list and re-run the script
|
||||
to get new images whenever necessary.
|
||||
The ``populate_registry.sh`` script checks if each image in the
|
||||
list is already present, which means you can update the list and
|
||||
re-run the script to get new images whenever necessary.
|
||||
|
||||
.. group-tab:: With Pull-through cache
|
||||
|
||||
#. Create folders to store your local registry images (``storage``) and to place setup files that will be used later on (``images``):
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
mkdir -p $HOME/docker-registry/storage
|
||||
mkdir -p $HOME/docker-registry/images
|
||||
mkdir -p $HOME/docker-registry/config
|
||||
|
||||
#. Create a configuration file that will be used by Docker's official
|
||||
registry image later on. The script will replace the PORT and
|
||||
REGISTRY values:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
cat > $HOME/docker-registry/config.yml << EOF
|
||||
version: 0.1
|
||||
log:
|
||||
fields:
|
||||
service: registry
|
||||
storage:
|
||||
cache:
|
||||
blobdescriptor: inmemory
|
||||
filesystem:
|
||||
rootdirectory: /var/lib/registry
|
||||
http:
|
||||
addr: <PORT>
|
||||
headers:
|
||||
X-Content-Type-Options: [nosniff]
|
||||
health:
|
||||
storagedriver:
|
||||
enabled: true
|
||||
interval: 10s
|
||||
threshold: 3
|
||||
proxy:
|
||||
remoteurl: https://<REGISTRY>
|
||||
EOF
|
||||
|
||||
#. Create and run a script that will create several private docker
|
||||
registries:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
export REG_CACHE_SCRIPT=$HOME/docker-registry/images/create_registries.sh
|
||||
cat > $REG_CACHE_SCRIPT <<'EOF'
|
||||
#!/bin/bash
|
||||
|
||||
if [ -z "$1" ]; then
|
||||
echo "No argument supplied"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
export LOCAL_REG=$HOME/docker-registry
|
||||
INSECURE_REGISTRY_LIST=()
|
||||
LOCALHOST_IP=$1
|
||||
[ -z "$2" ] && BASE_PORT=5000 || BASE_PORT=$2
|
||||
REGISTRY_LIST=("quay.io" "gcr.io" "k8s.gcr.io" "registry-1.docker.io"
|
||||
"docker.elastic.co" "ghcr.io" "registry.k8s.io" "icr.io")
|
||||
CONFIG_BASE=$(cat "$LOCAL_REG"/config.yml)
|
||||
|
||||
for cur_registry in "${REGISTRY_LIST[@]}"; do
|
||||
CONTAINER_NAME="registry_$cur_registry"
|
||||
NEW_CONFIG_FILE_NAME="config_$cur_registry.yml"
|
||||
NEW_CONFIG_FILE="$LOCAL_REG/configs/$NEW_CONFIG_FILE_NAME"
|
||||
NEW_CONFIG=$(echo "$CONFIG_BASE" | sed "s/<PORT>/$BASE_PORT/" | sed "s/<REGISTRY>/$cur_registry/")
|
||||
echo "$NEW_CONFIG" >"$NEW_CONFIG_FILE"
|
||||
|
||||
docker run -d \
|
||||
--restart=always \
|
||||
--name "$CONTAINER_NAME" \
|
||||
-v "$LOCAL_REG"/storage:/var/lib/registry \
|
||||
-v "$NEW_CONFIG_FILE":/etc/docker/registry/config.yml \
|
||||
-e REGISTRY_HTTP_ADDR=0.0.0.0:"$BASE_PORT" \
|
||||
-p "$BASE_PORT":"$BASE_PORT" \
|
||||
registry:2.6.2
|
||||
|
||||
INSECURE_REGISTRY_LIST+=("$cur_registry - $LOCALHOST_IP:$BASE_PORT")
|
||||
((BASE_PORT++))
|
||||
|
||||
done
|
||||
|
||||
for reg in "${INSECURE_REGISTRY_LIST[@]}"; do
|
||||
echo "$reg"
|
||||
done
|
||||
EOF
|
||||
|
||||
chmod +x $REG_CACHE_SCRIPT
|
||||
$REG_CACHE_SCRIPT <your ip address> <starting port (default 5000)>
|
||||
|
||||
.. important::
|
||||
The ``create_registries.sh`` script create 8 registry
|
||||
containers. You'll need 8 free ports, in sequence, starting from
|
||||
the specified starting port.
|
||||
Ex.: Passing port ``6000`` as the second argument, will create containers on ``6000-6007`` port range.
|
||||
|
||||
.. note::
|
||||
On it's current version, Docker CLI throws a warning when
|
||||
pulling images that're still using the v1 manifest format.
|
||||
The official Docker image registry, on it's latest version, does
|
||||
not allow images that still uses them.
|
||||
A few images used by the StarlingX OS are outdated, so, as of
|
||||
now, it'll be necessary to use a older version of the registry
|
||||
image (v2.6.2).
|
||||
Check the official `registry as a pull through cache`_ docker
|
||||
guide for more info.
|
||||
|
||||
|
||||
.. note::
|
||||
The Docker CLI exclusively permits insecure (HTTP) registries when on
|
||||
@ -188,6 +328,9 @@ Your registry is ready! On your next |prod| installation, update your
|
||||
``/home/sysadmin/localhost.yml`` bootstrap overrides file with the
|
||||
following lines to use it:
|
||||
|
||||
.. tabs::
|
||||
.. group-tab:: Private Registry
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
docker_registries:
|
||||
@ -211,6 +354,31 @@ following lines to use it:
|
||||
type: docker
|
||||
secure: false
|
||||
|
||||
.. group-tab:: With Pull-through cache
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
docker_registries:
|
||||
quay.io:
|
||||
url: <your IP address>:<port_1>
|
||||
gcr.io:
|
||||
url: <your IP address>:<port_2>
|
||||
k8s.gcr.io:
|
||||
url: <your IP address>:<port_3>
|
||||
docker.io:
|
||||
url: <your IP address>:<port_4>
|
||||
docker.elastic.co:
|
||||
url: <your IP address>:<port_5>
|
||||
ghcr.io:
|
||||
url: <your IP address>:<port_6>
|
||||
registry.k8s.io:
|
||||
url: <your IP address>:<port_7>
|
||||
icr.io:
|
||||
url: <your IP address>:<port_8>
|
||||
defaults:
|
||||
type: docker
|
||||
secure: false
|
||||
|
||||
.. note::
|
||||
This procedure configured |prod| to use an insecure registry via the
|
||||
``docker_registries.defaults.secure`` parameter set to ``false`` in the
|
||||
@ -219,3 +387,5 @@ following lines to use it:
|
||||
|
||||
.. _Kubernetes versions: https://opendev.org/starlingx/ansible-playbooks/src/branch/master/playbookconfig/src/playbooks/roles/bootstrap/validate-config/vars/main.yml
|
||||
.. _system images: https://opendev.org/starlingx/ansible-playbooks/src/branch/master/playbookconfig/src/playbooks/roles/common/load-images-information/vars
|
||||
.. _registry as a pull through cache: https://docs.docker.com/docker-hub/mirror/
|
||||
.. _kubeadm: https://kubernetes.io/pt-br/docs/reference/setup-tools/kubeadm/kubeadm-config/#cmd-config-images-list
|
||||
|
||||
Loading…
Reference in New Issue
Block a user