diff --git a/README.md b/README.md index 74aad50..9e3c5bf 100644 --- a/README.md +++ b/README.md @@ -1,65 +1,185 @@ -# docker-aptly +# Airship-utils -## Features +Airship-utils is a collection of tools that can accompany other airship-* projects. +Currently, airship-utils contains the following components: -- Packages are downloaded during the docker image build -- GPG keys for signature may be generated during the docker image build or existing ones are used -- Nginx blacklist support at runtime +- miniMirror -## Quickstart +## miniMirror -The main difference with the upstream repo is packages saved inside a docker image. -During the image building /opt/update_mirror_ubuntu.sh is called to create mirrors, update them, -merge all in one snapshot and publish it. By default, a new GPG key is generated for making a signature for repo. +miniMirror is a combination of package mirroring tool (Aptly) and a web server (Nginx) +packed into a container and a helm chart for k8s deployment. See [1] for additional info. -There are two modes: filtered build that fetches only packages specified in assets/packages and -unfiltered build that fetches all packages. The filtered build is used by default. +### Features -To fetch all packages the following command can be used: +- Packages are downloaded during the docker image build. + A list of packages can be specified with particular versions or + without them to get the current snapshots. Currently miniMirror + focuses on Ubuntu repositories only. +- GPG key for signature can be generated during the docker image build + or existing one can be used. To make a signature for packages Aptly + requires a private key, it is not possible to use a signature from + original mirror. +- Packages blacklist support at runtime. An additional Nginx + configuration can be provided to block specific package + installation. By default packages contains the following regexp in name are blocked: + - telnet + - ftp + - \brsh\b + - \bnis\b + +### How to build miniMirror image? + +#### General desription + +As Debian packages are downloaded and saved inside a docker image, it +may take some time to build the image. The process of building the image +includes the following steps: + +- Prepare GPG environment (see assets/startup.sh for details). + - Put into right places or generate GPG key depending on the build + configuration. See "Step by step guide" step 2 for details. + - Update GPG keyring. +- Create packages infrastructure (see assets/update_mirror_ubuntu.sh). + - Create Aptly mirrors. + - Fetch packages from upstream repositories according to the mirror + configurations. + - Merge repositories. For example, by default xenial, xenial-updates, + and xenial-security are used. Packages from each repository are + merged into one with latest wins strategy. + - Publish repository to directory Nginx will serve static files + from. + +#### Configuration + +The following build args are available: + +Repository configuration: + +- UPSTREAM_URL - a URL packages are downloaded from (mirror URL). +- UPSTREAM_KEY_URL - a URL for public GPG key if it is not on a default location. +- UBUNTU_RELEASE - a release name for a Ubuntu distribution. +- COMPONENTS - a list of repository components separated by space. + For example, values can be main, universe, restricted, multiverse [2]. +- REPOS - a list of repository types separated by space. + For example xenial, xenial-updates, xenial-security, xenial-backports. + +Packages configuration: +- MODE - a string determining if all packages should be downloaded or + specific only. Possible values: packages or all. +- PACKAGE_FILE - a file name where a list of packages is saved. If + MODE=packages the file must be available in assets/packages + directory. + +GPG key configuration: +By default GPG key for making package signature is generated during +the build. If you have a GPG key already you can put private and +public keys in assets/gpg dir. Keys must have special names: aptly.sec +and aptly.pub. You may configure GPG key params via the following arguments: +- FULL_NAME - a full name for a GPG key. +- EMAIL_ADDRESS - an email for a GPG key. +- GPG_PASSWORD - a passphrase for a GPG key. This can be used both for + GPG key generation and GPG key usage. + +Nginx configuration: +- HOSTNAME - server_name configuration for Nginx. + +Example: ```bash -git clone https://github.com/urpylka/docker-aptly.git -docker build docker-aptly --build-arg MODE=all +git clone https://git.openstack.org/openstack/airship-utils +docker build airship-utils \ + --UBUNTU_RELEASE=bionic \ + --build-arg FULL_NAME="John Smith" \ + --build-arg EMAIL_ADDRESS="john.smith@example.com" \ + --build-arg GPG_PASSWORD="PickAPassword" \ + --build-arg HOSTNAME=_ ``` -By default GPG key for making package signature are generated during the build. -You may configure GPG key params via build arguments: FULL_NAME, EMAIL_ADDRESS, and GPG_PASSWORD, like: +### Step by step guide + +This is an example of how miniMirror can be used. + +1) Prepare a list of packages needed for a miniMirror image. ```bash -docker build docker-aptly \ - --build-arg FULL_NAME="First Last" \ - --build-arg EMAIL_ADDRESS="youremail@example.com" \ - --build-arg GPG_PASSWORD="PickAPassword" +cd airship-utils +cat << 'EOF' > assets/packages/my_packages +mysql-client-5.7 (= 5.7.24-0ubuntu0.16.04.1) +mysql-client-core-5.7 +postgresql-client-9.5 (= 9.5.14-0ubuntu0.16.04) +postgresql-client-common +EOF ``` -If you have a GPG key already you can put private and public key in assets/gpg dir. -Keys must have special names: aptly.sec and aptly.pub -For example: +2) Prepare a GPG key for making package signature. + +GPG public and private keys should be named as assets/gpg/aptly.pub and assets/gpg/aptly.key. ```bash -cp docker-aptly/assets/gpg/aptly.sec -cp docker-aptly/assets/gpg/aptly.pub - -docker build docker-aptly \ - --build-arg GPG_PASSWORD="GPG passphrase for my private key" +mkdir -p /opt/aptly +export FULL_NAME='John Smith' +export EMAIL_ADDRESS='john.smith@example.com' +export GPG_PASSWORD='my_passphrase' +bash assets/gpg_batch.sh +gpg -v --batch --gen-key /opt/gpg_batch +mv /opt/aptly/* assets/gpg/ +rm /opt/gpg_batch ``` -To use the Nginx blacklist feature a volume with Nginx config has to be mounted at runtime. -By default, the following keywords are blocked: telnet, ftp. -If no volume is mounted then no blacklist will be used. +3) Build docker image. + +```bash +docker build . -t mini-mirror \ + --build-arg PACKAGE_FILE=my_packages \ + --build-arg GPG_PASSWORD="$GPG_PASSWORD" +``` + +4) Test miniMirror container. + +Start miniMirror container. + +```bash +docker run -d \ + --publish 8080:80 \ + --volume $(pwd)/assets/nginx:/opt/nginx \ + --name mini-mirror \ + mini-mirror +``` + +Run another container and install packages there. + +```bash +docker run --network host \ +--env PACKAGES='mysql-client-5.7 postgresql-client-9.5' \ +--name target \ +--volume $(pwd)/tools:/opt \ +ubuntu:16.04 /opt/install_packages.sh +``` + +### How to blacklist miniMirror packages + +To use the Nginx blacklist feature a volume with Nginx config has to +be mounted at runtime. If no volume is mounted then no blacklist will +be used. ```bash docker run \ - --name aptly \ + --name mini-mirror \ --detach \ --publish 8080:80 \ --volume $(pwd)/assets/nginx:/opt/nginx \ - aptly:test + mini-mirror ``` -___ -For additional docs see https://github.com/amadev/docker-aptly +## References +* [1] https://review.openstack.org/#/c/611376/ +* [2] https://help.ubuntu.com/community/Repositories + +## Copyright + +* Copyright 2018 AT&T Intellectual Property * Copyright 2018 Artem B. Smirnov * Copyright 2016 Bryan J. Hong * Licensed under the Apache License, Version 2.0 diff --git a/mini-mirror/assets/nginx/blacklist.conf b/mini-mirror/assets/nginx/blacklist.conf index cc74f61..0b44f3d 100644 --- a/mini-mirror/assets/nginx/blacklist.conf +++ b/mini-mirror/assets/nginx/blacklist.conf @@ -1,3 +1,3 @@ -location ~ (telnet|ftp) { +location ~ (telnet|ftp|\brsh\b|\bnis\b) { return 404; } diff --git a/tools/install_packages.sh b/tools/install_packages.sh index 502d672..aaafba8 100755 --- a/tools/install_packages.sh +++ b/tools/install_packages.sh @@ -9,5 +9,8 @@ apt install -y curl curl -s localhost:8080/aptly_repo_signing.key | apt-key add - echo 'deb http://localhost:8080 xenial main' > /etc/apt/sources.list apt-get update -apt-cache policy accountsservice -apt-get install -y accountsservice +PACKAGES=${PACKAGES:-accountsservice} +for package in $PACKAGES; do + apt-cache policy "$package" + apt-get install -y "$package" +done