zuul/zuul
Code Issues Proposed changes

Reorganize admin manual

Browse Source 
This makes a number of changes to the installation/configuration
documentation in the admin manual.

Remove quick-start guide.  The process of quick-starting is
covered by the installation and setup tutorial, which is now the
first of the installation sections.  The reference material from
quick-start is now in the tutorial.

Rename the tutorial quick-start.  It's nice to have something
named quick-start, and the tutorial fits the bill.

Rename the installation section "Installation Reference".  This
now has more detailed information about installation and
deployment choices, but has very little procedural documentation.

Make zuul-from-scratch more internally consistent in style (use
code-block:: shell and heredocs wherever possible).

Change-Id: I7e4714ce5e775dc9ac0988c3470eef1f74fb36d6
This commit is contained in:
James E. Blair
2018-10-10 09:54:03 -07:00
parent 8571b52ba8
commit a1d8fc7a51
14 changed files with 695 additions and 663 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/admin/centos7_setup.rst
+15 -15
View File
@@ -20,9 +20,9 @@ Zuul and Nodepool.
To get started, ssh to your machine as the ``centos`` user.
.. code-block:: console
.. code-block:: shell
$ ssh centos@<ip_address>
ssh centos@<ip_address>
Environment Setup
-----------------
@@ -41,15 +41,15 @@ dependencies are handled by the bindep program, but a few additional
dependencies are needed to install bindep, and for other commands
which we will use in these instructions.
.. code-block:: console
.. code-block:: shell
$ sudo yum update -y
$ sudo systemctl reboot
$ sudo yum install -y https://centos7.iuscommunity.org/ius-release.rpm
$ sudo yum install -y git python35u python35u-pip python35u-devel java-1.8.0-openjdk
$ sudo alternatives --install /usr/bin/python3 python3 /usr/bin/python3.5 10
$ sudo alternatives --install /usr/bin/pip3 pip3 /usr/bin/pip3.5 10
$ sudo pip3 install python-openstackclient bindep
sudo yum update -y
sudo systemctl reboot
sudo yum install -y https://centos7.iuscommunity.org/ius-release.rpm
sudo yum install -y git python35u python35u-pip python35u-devel java-1.8.0-openjdk
sudo alternatives --install /usr/bin/python3 python3 /usr/bin/python3.5 10
sudo alternatives --install /usr/bin/pip3 pip3 /usr/bin/pip3.5 10
sudo pip3 install python-openstackclient bindep
Install Zookeeper
-----------------
@@ -60,7 +60,7 @@ Nodepool for nodes.
.. code-block:: console
$ sudo bash -c "cat << EOF > /etc/yum.repos.d/bigtop.repo
sudo bash -c "cat << EOF > /etc/yum.repos.d/bigtop.repo
[bigtop]
name=Bigtop
enabled=1
@@ -69,7 +69,7 @@ Nodepool for nodes.
baseurl=http://repos.bigtop.apache.org/releases/1.2.1/centos/7/x86_64
gpgkey=https://dist.apache.org/repos/dist/release/bigtop/KEYS
EOF"
$ sudo yum install -y zookeeper zookeeper-server
$ sudo systemctl start zookeeper-server.service
$ sudo systemctl status zookeeper-server.service
$ sudo systemctl enable zookeeper-server.service
sudo yum install -y zookeeper zookeeper-server
sudo systemctl start zookeeper-server.service
sudo systemctl status zookeeper-server.service
sudo systemctl enable zookeeper-server.service
doc/source/admin/components.rst
+1
View File
@@ -22,6 +22,7 @@ which is described below.
Merger -- Gearman
Executor -- Gearman
Web -- Gearman
Finger -- Gearman
Gearman -- Scheduler;
Scheduler -- Gerrit;
doc/source/admin/gerrit_setup.rst
+29 -16
View File
@@ -23,17 +23,22 @@ ACL enabled.
.. TODO: Instructions to create the ssh key used here
As the admin user, create the ``zuul`` user, and import an SSH key for ``zuul``::
As the admin user, create the ``zuul`` user, and import an SSH key for
``zuul``:
$ cat $PUBKEY | ssh -p 29418 $USER@localhost gerrit create-account \
.. code-block:: shell
cat $PUBKEY | ssh -p 29418 $USER@localhost gerrit create-account \
--group "'Registered Users'" --ssh-key - zuul
``$PUBKEY`` is the location of the SSH public key for the ``zuul``
user. ``$USER`` is the username for the admin user.
The ``zuul`` user should now be able to stream events::
The ``zuul`` user should now be able to stream events:
$ ssh -p 29418 zuul@localhost gerrit stream-events
.. code-block:: shell
ssh -p 29418 zuul@localhost gerrit stream-events
Configure Gerrit
----------------
@@ -64,27 +69,35 @@ The admin user can create new projects in Gerrit, which users can then clone
and use to submit code changes. Zuul will monitor the Gerrit event stream for
these submissions.
To create a new project named 'demo-project'::
To create a new project named 'demo-project':
$ ssh -p 29418 $USER@localhost gerrit create-project demo-project --empty-commit
.. code-block:: shell
ssh -p 29418 $USER@localhost gerrit create-project demo-project --empty-commit
Modify the Project
------------------
* Clone the project::
* Clone the project:
$ git clone ssh://$USER@localhost:29418/demo-project.git
.. code-block:: shell
* Install the change ID hook that Gerrit requires::
git clone ssh://$USER@localhost:29418/demo-project.git
$ cd demo-project
$ scp -p -P 29418 $USER@localhost:hooks/commit-msg .git/hooks/
* Install the change ID hook that Gerrit requires:
* Now you are ready to modify the project and push the changes to Gerrit::
.. code-block:: shell
$ echo "test" > README.txt
$ git add .
$ git commit -m "First commit"
$ git push origin HEAD:refs/for/master
cd demo-project
scp -p -P 29418 $USER@localhost:hooks/commit-msg .git/hooks/
* Now you are ready to modify the project and push the changes to Gerrit:
.. code-block:: shell
echo "test" > README.txt
git add .
git commit -m "First commit"
git push origin HEAD:refs/for/master
You should now be able to see your change in Gerrit.
doc/source/admin/github_setup.rst
+30 -15
View File
@@ -60,22 +60,29 @@ Go back to the `General` settings page for the app,
https://github.com/organizations/my-org/settings/apps/my-org-zuul
and look for the app `ID` number, under the `About` section.
Edit ``/etc/zuul/zuul.conf`` to add the following::
Edit ``/etc/zuul/zuul.conf`` to add the following:
[connection github]
driver=github
app_id=<APP ID NUMBER>
app_key=/etc/zuul/github.pem
webhook_token=<WEBHOOK SECRET>
.. code-block:: shell
sudo bash -c "cat >> /etc/zuul/zuul.conf <<EOF
[connection github]
driver=github
app_id=<APP ID NUMBER>
app_key=/etc/zuul/github.pem
webhook_token=<WEBHOOK SECRET>
EOF"
Upload the private key which was generated earlier, and save it in
``/etc/zuul/github.pem``.
Restart all of Zuul::
Restart all of Zuul:
sudo systemctl restart zuul-executor.service
sudo systemctl restart zuul-web.service
sudo systemctl restart zuul-scheduler.service
.. code-block:: shell
sudo systemctl restart zuul-executor.service
sudo systemctl restart zuul-web.service
sudo systemctl restart zuul-scheduler.service
Go to the `Advanced` tab for the app in GitHub,
https://github.com/organizations/my-org/settings/apps/my-org-zuul/advanced,
@@ -92,7 +99,9 @@ and ``zuul-test``, respectively.
Visit the public app page on GitHub,
https://github.com/apps/my-org-zuul, and install the app into your org.
Edit ``/etc/zuul/main.yaml`` so that it looks like this::
Edit ``/etc/zuul/main.yaml`` so that it looks like this:
.. code-block:: yaml
- tenant:
name: quickstart
@@ -114,13 +123,17 @@ you can immediately use in your Zuul installation.
The second section is your GitHub configuration.
After updating the file, restart the Zuul scheduler::
After updating the file, restart the Zuul scheduler:
sudo systemctl restart zuul-scheduler.service
.. code-block:: shell
sudo systemctl restart zuul-scheduler.service
Add an initial pipeline configuration to the `zuul-test-config`
repository. Inside that project, create a ``zuul.yaml`` file with the
following contents::
following contents:
.. code-block:: yaml
- pipeline:
name: check
@@ -152,7 +165,9 @@ following contents::
Merge that commit into the repository.
In the `zuul-test` project, create a `.zuul.yaml` file with the
following contents::
following contents:
.. code-block:: yaml
- project:
check:
doc/source/admin/index.rst
+2 -2
View File
@@ -11,12 +11,12 @@ provides to in-project configuration.
.. toctree::
:maxdepth: 2
tutorial
quick-start
zuul-from-scratch
installation
zuul-from-scratch
components
connections
tenants
monitoring
client
troubleshooting
doc/source/admin/installation.rst
+105 -9
View File
@@ -1,5 +1,5 @@
Installation
============
Installation Reference
======================
Install Zuul
------------
@@ -17,9 +17,42 @@ interaction with other python packages installed on a system, you may
wish to install Zuul within a Python virtualenv.
Zuul has several system-level dependencies as well. You can find a
list of operating system packages in `bindep.txt` in Zuul's source
list of operating system packages in ``bindep.txt`` in Zuul's source
directory.
Zuul Components
---------------
Zuul provides the following components:
- **zuul-scheduler**: The main Zuul process. Handles receiving
events, executing jobs, collecting results and posting reports.
Coordinates the work of the other components. It also provides
a gearman daemon which the other components use for
coordination.
- **zuul-merger**: Scale-out component that performs git merge
operations. Zuul performs a large number of git operations in
the course of its work. Adding merger processes can help speed
Zuul's processing. This component is optional (zero or more of
these can be run).
- **zuul-executor**: Scale-out component for executing jobs. At
least one of these is required. Depending on system
configuration, you can expect a single executor to handle up to
about 100 simultaneous jobs. Can handle the functions of a
merger if dedicated mergers are not provided. One or more of
these must be run.
- **zuul-web**: A web server that receives "webhook" events from
external providers, supplies a web dashboard, and provides
websocket access to live streaming of logs.
- **zuul-fingergw**: A gateway which provides finger protocol
access to live streaming of logs.
For more detailed information about these, see :ref:`components`.
External Dependencies
---------------------
@@ -68,10 +101,73 @@ the correct version will be installed automatically with Zuul.
Because of the close integration of Zuul and Ansible, attempting to
use other versions of Ansible with Zuul is not recommended.
Zuul Setup
----------
At minimum you need to provide ``zuul.conf`` and ``main.yaml`` placed
in ``/etc/zuul/``. The following example uses the builtin gearman
service in Zuul, and a connection to Gerrit.
**zuul.conf**::
[scheduler]
tenant_config=/etc/zuul/main.yaml
[gearman_server]
start=true
[gearman]
server=127.0.0.1
[connection my_gerrit]
driver=gerrit
server=git.example.com
port=29418
baseurl=https://git.example.com/gerrit/
user=zuul
sshkey=/home/zuul/.ssh/id_rsa
See :ref:`components` and :ref:`connections` for more details.
The following tells Zuul to read its configuration from and operate on
the *example-project* project:
**main.yaml**::
- tenant:
name: example-tenant
source:
my_gerrit:
untrusted-projects:
- example-project
Starting Zuul
-------------
You can run any zuul process with the **-d** option to make it not
daemonize. It's a good idea at first to confirm there's no issues with
your configuration.
To start, simply run::
zuul-scheduler
Once run you should have two zuul-scheduler processes (if using the
built-in gearman server, or one process otherwise).
Before Zuul can run any jobs, it needs to load its configuration, most
of which is in the git repositories that Zuul operates on. Start an
executor to allow zuul to do that::
zuul-executor
Zuul should now be able to read its configuration from the configured
repo and process any jobs defined therein.
.. _web-deployment-options:
Web Deployment Options
======================
----------------------
The ``zuul-web`` service provides an web dashboard, a REST API and a websocket
log streaming service as a single holistic web application. For production use
@@ -104,7 +200,7 @@ Reverse Proxy. Where rewrite rule examples are given, they will be given
with Apache syntax, but any other Reverse Proxy should work just fine.
Basic Reverse Proxy
-------------------
~~~~~~~~~~~~~~~~~~~
Using Apache as the Reverse Proxy requires the ``mod_proxy``,
``mod_proxy_http`` and ``mod_proxy_wstunnel`` modules to be installed and
@@ -119,7 +215,7 @@ simplest reverse-proxy case is::
Static Offload
--------------
~~~~~~~~~~~~~~
To have the Reverse Proxy serve the static html/javascript assets instead of
proxying them to the REST layer, register the location where you unpacked
@@ -146,7 +242,7 @@ the web application as the document root and add rewrite rules::
Sub directory serving
---------------------
~~~~~~~~~~~~~~~~~~~~~
The web application needs to be rebuild to update the internal location of
the static files. Set the homepage setting in the package.json to an
@@ -188,7 +284,7 @@ add the following rewrite rules::
White Labeled Tenant
--------------------
~~~~~~~~~~~~~~~~~~~~
Running a white-labeled tenant is similar to the offload case, but adds a
rule to ensure connection webhooks don't try to get put into the tenant scope.
@@ -223,7 +319,7 @@ Assuming the zuul tenant name is "example", the rewrite rules are::
Static External
---------------
~~~~~~~~~~~~~~~
.. note::
doc/source/admin/nodepool_install.rst
+23 -23
View File
@@ -10,15 +10,15 @@ First we'll create the nodepool user and set up some directories it
needs. We also need to create an SSH key for Zuul to use when it logs
into the nodes that Nodepool provides.
.. code-block:: console
.. code-block:: shell
$ sudo groupadd --system nodepool
$ sudo useradd --system nodepool --home-dir /var/lib/nodepool --create-home -g nodepool
$ ssh-keygen -t rsa -b 2048 -f nodepool_rsa # don't enter a passphrase
$ sudo mkdir /etc/nodepool/
$ sudo mkdir /var/log/nodepool
$ sudo chgrp -R nodepool /var/log/nodepool/
$ sudo chmod 775 /var/log/nodepool/
sudo groupadd --system nodepool
sudo useradd --system nodepool --home-dir /var/lib/nodepool --create-home -g nodepool
ssh-keygen -t rsa -b 2048 -f nodepool_rsa # don't enter a passphrase
sudo mkdir /etc/nodepool/
sudo mkdir /var/log/nodepool
sudo chgrp -R nodepool /var/log/nodepool/
sudo chmod 775 /var/log/nodepool/
Installation
------------
@@ -27,21 +27,21 @@ Clone the Nodepool git repository and install it. The ``bindep``
program is used to determine any additional binary dependencies which
are required.
.. code-block:: console
.. code-block:: shell
# All:
$ git clone https://git.zuul-ci.org/nodepool
$ pushd nodepool/
git clone https://git.zuul-ci.org/nodepool
pushd nodepool/
# For Fedora and CentOS:
$ sudo yum -y install $(bindep -b compile)
sudo yum -y install $(bindep -b compile)
# For openSUSE:
$ sudo zypper install -y $(bindep -b compile)
sudo zypper install -y $(bindep -b compile)
# All:
$ sudo pip3 install .
$ popd
sudo pip3 install .
popd
Service File
------------
@@ -49,18 +49,18 @@ Service File
Nodepool includes a systemd service file for nodepool-launcher in the ``etc``
source directory. To use it, do the following steps.
.. code-block:: console
.. code-block:: shell
$ sudo cp etc/nodepool-launcher.service /etc/systemd/system/nodepool-launcher.service
$ sudo chmod 0644 /etc/systemd/system/nodepool-launcher.service
sudo cp etc/nodepool-launcher.service /etc/systemd/system/nodepool-launcher.service
sudo chmod 0644 /etc/systemd/system/nodepool-launcher.service
If you are installing Nodepool on ``CentOS 7`` and copied the provided service
file in previous step, please follow the steps below to use corresponding
systemd drop-in file so Nodepool service can be managed by systemd.
.. code-block:: console
.. code-block:: shell
$ sudo mkdir /etc/systemd/system/nodepool-launcher.service.d
$ sudo cp etc/nodepool-launcher.service.d/centos.conf \
/etc/systemd/system/nodepool-launcher.service.d/centos.conf
$ sudo chmod 0644 /etc/systemd/system/nodepool-launcher.service.d/centos.conf
sudo mkdir /etc/systemd/system/nodepool-launcher.service.d
sudo cp etc/nodepool-launcher.service.d/centos.conf \
/etc/systemd/system/nodepool-launcher.service.d/centos.conf
sudo chmod 0644 /etc/systemd/system/nodepool-launcher.service.d/centos.conf
doc/source/admin/nodepool_openstack.rst
+7 -3
View File
@@ -10,14 +10,18 @@ Before starting on this, you need to download your `openrc`
configuration from your OpenStack cloud. Put it on your server in the
staging user's home directory. It should be called
``<username>-openrc.sh``. Once that is done, create a new keypair
that will be installed when instantiating the servers::
that will be installed when instantiating the servers:
.. code-block:: shell
cd ~
source <username>-openrc.sh # this may prompt for password - enter it
openstack keypair create --public-key nodepool_rsa.pub nodepool
We'll use the private key later wheen configuring Zuul. In the same
session, configure nodepool to talk to your cloud::
session, configure nodepool to talk to your cloud:
.. code-block:: shell
umask 0066
sudo mkdir -p ~nodepool/.config/openstack
@@ -48,7 +52,7 @@ configuration file:
* flavor-name
* image-name - from your cloud
::
.. code-block:: shell
sudo bash -c "cat >/etc/nodepool/nodepool.yaml <<EOF
zookeeper-servers:
doc/source/admin/nodepool_static.rst
+26 -22
View File
@@ -19,38 +19,42 @@ the following requirements:
When setting up your nodepool.yaml file, you will need the host keys
for each node for the ``host-key`` value. This can be obtained with
the command::
the command:
$ ssh-keyscan -t ed25519 <HOST>
.. code-block:: shell
ssh-keyscan -t ed25519 <HOST>
Nodepool Configuration
----------------------
Below is a sample Nodepool configuration file that sets up static
nodes. Place this file in ``/etc/nodepool/nodepool.yaml``.
nodes. Place this file in ``/etc/nodepool/nodepool.yaml``:
::
.. code-block:: shell
zookeeper-servers:
- host: localhost
sudo bash -c "cat > /etc/nodepool/nodepool.yaml <<EOF
zookeeper-servers:
- host: localhost
labels:
- name: ubuntu-xenial
labels:
- name: ubuntu-xenial
providers:
- name: static-vms
driver: static
pools:
- name: main
nodes:
- name: 192.168.1.10
labels: ubuntu-xenial
host-key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGXqY02bdYqg1BcIf2x08zs60rS6XhlBSQ4qE47o5gb"
username: zuul
- name: 192.168.1.11
labels: ubuntu-xenial
host-key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGXqY02bdYqg1BcIf2x08zs60rS6XhlBSQ5sE47o5gc"
username: zuul
providers:
- name: static-vms
driver: static
pools:
- name: main
nodes:
- name: 192.168.1.10
labels: ubuntu-xenial
host-key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGXqY02bdYqg1BcIf2x08zs60rS6XhlBSQ4qE47o5gb"
username: zuul
- name: 192.168.1.11
labels: ubuntu-xenial
host-key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGXqY02bdYqg1BcIf2x08zs60rS6XhlBSQ5sE47o5gc"
username: zuul
EOF"
Make sure that ``username``, ``host-key``, IP addresses and label names are
customized for your environment.
doc/source/admin/opensuse_leap15_setup.rst
+4 -3
View File
@@ -22,13 +22,14 @@ dependencies are handled by the bindep program, but a few additional
dependencies are needed to install bindep, and for other commands
which we will use in these instructions.
::
.. code-block:: shell
sudo zypper install -y git python3-pip
Then install bindep
::
.. code-block:: shell
pip3 install --user bindep
# Add it to your path
PATH=~/.local/bin:$PATH
@@ -48,7 +49,7 @@ To download follow the directions on `Zookeeper's releases page
<https://zookeeper.apache.org/releases.html>`_ to grab the latest
release of zookeeper. Then:
::
.. code-block:: shell
sudo zypper install -y java-1_8_0-openjdk
tar -xzf zookeeper-3.4.12.tar.gz # Tarball downloaded from Zookeeper
doc/source/admin/quick-start.rst
+427 -84
View File
@@ -1,123 +1,466 @@
Quick Start Guide
=================
.. _quick_start:
This provides a very simple overview of Zuul. It is recommended to
read the following sections for more details.
Quick-Start Installation and Tutorial
=====================================
Install Zuul
------------
Zuul is not like other CI or CD systems. It is a project gating
system designed to assist developers in taking a change from proposal
through deployment. Zuul can support any number of workflow processes
and systems, but to help you get started with Zuul, this tutorial will
walk through setting up a basic gating configuration which protects
projects from merging broken code.
You can get zuul from pypi via::
This tutorial is entirely self-contained and may safely be run on a
workstation. The only requirements are a network connection and the
ability to run Docker containers. For code review, it provides an
instance of Gerrit, though the concepts you will learn apply equally
to GitHub. Even if you don't ultimately intend to use Gerrit, you are
encouraged to follow this tutorial to learn how to set up Zuul and
then consult further documentation to configure your Zuul to interact
with GitHub.
pip install zuul
Start Zuul Containers
---------------------
Zuul Components
---------------
Before you start, ensure that some needed packages are installed.
Zuul provides the following components:
.. code-block:: shell
- **zuul-scheduler**: The main Zuul process. Handles receiving
events, executing jobs, collecting results and posting reports.
Coordinates the work of the other components.
# Debian / Ubuntu:
- **zuul-merger**: Scale-out component that performs git merge
operations. Zuul performs a large number of git operations in
the course of its work. Adding merger processes can help speed
Zuul's processing. This component is optional (zero or more of
these can be run).
sudo apt-get install docker-compose git git-review
- **zuul-executor**: Scale-out component for executing jobs. At
least one of these is required. Depending on system
configuration, you can expect a single executor to handle up to
about 100 simultaneous jobs. Can handle the functions of a
merger if dedicated mergers are not provided. One or more of
these must be run.
# Red Hat / Fedora / SUSE:
- **zuul-web**: A web server that currently provides websocket access to
live-streaming of logs.
sudo yum install docker-compose git git-review
- **gearman**: optional builtin gearman daemon provided by zuul-scheduler
Clone the Zuul repository:
External components:
.. code-block:: shell
- **gearman**: A gearman daemon if the built-in daemon is not used.
git clone https://git.zuul-ci.org/zuul
- **zookeeper**: A zookeeper cluster (or single host) for
communicating with Nodepool.
Then cd into the directory containing this document, and run
docker-compose in order to start Zuul, Nodepool and Gerrit.
- **nodepool**: Provides nodes for Zuul to use when executing jobs.
.. code-block:: shell
cd zuul/doc/source/admin/examples
docker-compose up
Zuul Setup
----------
All of the services will be started with debug-level logging sent to
the standard output of the terminal where docker-compose is running.
You will see a considerable amount of information scroll by, including
some errors. Zuul will immediately attempt to connect to Gerrit and
begin processing, even before Gerrit has fully initialized. The
docker composition includes scripts to configure Gerrit and create an
account for Zuul. Once this has all completed, the system should
automatically connect, stabilize and become idle. When this is
complete, you will have the following services running:
At minimum you need to provide **zuul.conf** and **main.yaml** placed
in **/etc/zuul/**. The following example uses the builtin gearman
service in Zuul, and a connection to Gerrit.
* Zookeeper
* Gerrit
* Nodepool Launcher
* Zuul Scheduler
* Zuul Web Server
* Zuul Executor
* Apache HTTPD
**zuul.conf**::
And a long-running static test node used by Nodepool and Zuul upon
which to run tests.
[scheduler]
tenant_config=/etc/zuul/main.yaml
The Zuul scheduler is configured to connect to Gerrit via a connection
named ``gerrit``. Zuul can interact with as many systems as
necessary, each such connection is assigned a name for use in the Zuul
configuration.
[gearman_server]
start=true
Zuul is a multi-tenant application, so that differing needs of
independent work-groups can be supported from one system. This
example configures a single tenant named ``example-tenant``. Assigned
to this tenant are three projects: ``zuul-config``, ``test1`` and
``test2``. These have already been created in Gerrit and are ready
for us to begin using.
[gearman]
server=127.0.0.1
Add Your Gerrit Account
-----------------------
[connection my_gerrit]
driver=gerrit
server=git.example.com
port=29418
baseurl=https://git.example.com/gerrit/
user=zuul
sshkey=/home/zuul/.ssh/id_rsa
Before you can interact with Gerrit, you will need to create an
account. The initialization script has already created an account for
Zuul, but has left the task of creating your own account to you so
that you can provide your own SSH key. You may safely use any
existing SSH key on your workstation, or you may create a new one by
running ``ssh-keygen``.
See :ref:`components` and :ref:`connections` for more details.
Gerrit is configured in a development mode where passwords are not
required in the web interface and you may become any user in the
system at any time.
The following tells Zuul to read its configuration from and operate on
the *example-project* project:
To create your Gerrit account, visit http://localhost:8080 in your
browser and click `Become`.
**main.yaml**::
.. image:: images/become.png
:align: center
- tenant:
name: example-tenant
source:
my_gerrit:
untrusted-projects:
- example-project
Then click `New Account` under `Register`.
Starting Zuul
-------------
.. image:: images/register.png
:align: center
You can run any zuul process with the **-d** option to make it not
daemonize. It's a good idea at first to confirm there's no issues with
your configuration.
Enter your full name and click `Save Changes`.
To start, simply run::
.. image:: images/name.png
:align: center
zuul-scheduler
Enter the username you use to log into your workstation in the
`Username` field and click `Select Username`.
Once run you should have two zuul-scheduler processes (if using the
built-in gearman server, or one process otherwise).
.. image:: images/username.png
:align: center
Before Zuul can run any jobs, it needs to load its configuration, most
of which is in the git repositories that Zuul operates on. Start an
executor to allow zuul to do that::
Copy and paste the contents of ``~/.ssh/id_rsa.pub`` into the SSH key
field and click `Add`.
zuul-executor
.. image:: images/sshkey.png
:align: center
Zuul should now be able to read its configuration from the configured
repo and process any jobs defined therein.
Click the `Continue` link at the bottom of the page.
Troubleshooting
---------------
.. image:: images/continue.png
:align: center
You can use telnet to connect to gearman to check which Zuul
components are online::
At this point you have created and logged into your personal account
in Gerrit and are ready to begin configuring Zuul.
telnet <gearman_ip> 4730
Configure Zuul Pipelines
------------------------
Useful commands are **workers** and **status** which you can run by just
typing those commands once connected to gearman.
Zuul recognizes two types of projects: :term:`config
projects<config-project>` and :term:`untrusted
projects<untrusted-project>`. An *untrusted project* is a normal
project from Zuul's point of view. In a gating system, it contains
the software under development and/or most of the job content that
Zuul will run. A *config project* is a special project that contains
the Zuul's configuration. Because it has access to normally
restricted features in Zuul, changes to this repository are not
dynamically evaluated by Zuul. The security and functionality of the
rest of the system depends on this repository, so it is best to limit
what is contained within it to the minimum, and ensure thorough code
review practices when changes are made.
Zuul has no built-in workflow definitions, so in order for it to do
anything, you will need to begin by making changes to a *config
project*. The initialization script has already created a project
named ``zuul-config`` which you should now clone onto your workstation:
.. code-block:: shell
git clone http://localhost:8080/zuul-config
You will find that this repository is empty. Zuul reads its
configuration from either a single file or a directory. In a *Config
Project* with substantial Zuul configuration, you may find it easiest
to use the ``zuul.d`` directory for Zuul configuration. Later, in
*Untrusted Projects* you will use a single file for in-repo
configuration. Make the directory:
.. code-block:: shell
cd zuul-config
mkdir zuul.d
The first type of configuration items we need to add are the Pipelines
we intend to use. In Zuul, a Pipeline represents a workflow action.
It is triggered by some action on a connection. Projects are able to
attach jobs to run in that pipeline, and when they complete, the
results are reported along with actions which may trigger further
Pipelines. In a gating system two pipelines are required:
:term:`check` and :term:`gate`. In our system, ``check`` will be
triggered when a patch is uploaded to Gerrit, so that we are able to
immediately run tests and report whether the change works and is
therefore able to merge. The ``gate`` pipeline is triggered when a code
reviewer approves the change in Gerrit. It will run test jobs again
(in case other changes have merged since the change in question was
uploaded) and if these final tests pass, will automatically merge the
change. To configure these pipelines, copy the following file into
`zuul.d/pipelines.yaml`:
.. literalinclude:: examples/zuul-config/zuul.d/pipelines.yaml
:language: yaml
Once we have bootstrapped our initial Zuul configuration, we will want
to use the gating process on this repository too, so we need to attach
the ``zuul-config`` repository to the ``check`` and ``gate`` pipelines
we are about to create. There are no jobs defined yet, so we must use
the internally defined ``noop`` job, which always returns success.
Later on we will be configuring some other projects, and while we will
be able to dynamically add jobs to their pipelines, those projects
must first be attached to the pipelines in order for that to work. In
our system, we want all of the projects in Gerrit to participate in
the check and gate pipelines, so we can use a regular expression to
apply this to all projects. To configure the ``check`` and ``gate``
pipelines for ``zuul-config`` to run the ``noop`` job, and add all
projects to those pipelines (with no jobs), copy the following file
into ``zuul.d/projects.yaml``:
.. literalinclude:: examples/zuul-config/zuul.d/projects.yaml
:language: yaml
Every real job (i.e., all jobs other than ``noop``) must inherit from a
:term:`base job`, and base jobs may only be defined in a
:term:`config-project`. Let's go ahead and add a simple base job that
we can build on later. Copy the following into ``zuul.d/jobs.yaml``:
.. literalinclude:: examples/zuul-config/zuul.d/jobs.yaml
:language: yaml
Commit the changes and push them up for review:
.. code-block:: shell
git add zuul.d
git commit -m "Add initial Zuul configuration"
git review
Because Zuul is currently running with no configuration whatsoever, it
will ignore this change. For this initial change which bootstraps the
entire system, we will need to bypass code review (hopefully for the
last time). To do this, you need to switch to the Administrator
account in Gerrit. Visit http://localhost:8080 in your browser and
then:
Click on your name in the top right corner then click `Switch
Account`.
.. image:: images/switch-example.png
:align: center
Click `admin` to log in as the `admin` user.
.. image:: images/become-select.png
:align: center
In the top left corner of the page, click `All` and `Open` to see the
list of open changes, then click on the change you uploaded.
.. image:: images/open-changes.png
:align: center
Click `Reply...` at the top center of the change screen. This will
open a dialog where you can leave a review message and vote on the
change. As the administrator, you have access to vote in all of the
review categories, even `Verified` which is normally reserved for
Zuul. Vote Code-Review: +2, Verified: +2, Workflow: +1, and then
click `Post` to leave your approval votes.
.. image:: images/review-1001.png
:align: center
Once the required votes have been set, the `Submit` button will
appear; click it. This will cause the change to be merged
immediately. This is normally handled by Zuul, but as the
administrator you can bypass Zuul to forcibly merge a change.
.. image:: images/submit-1001.png
:align: center
Now that the initial configuration has been bootstrapped, you should
not need to bypass testing and code review again, so switch back to
the account you created for yourself. Click on `Administrator` in the
top right corner then click `Switch Account`.
.. image:: images/switch-admin.png
:align: center
Click your username.
.. image:: images/become-select.png
:align: center
Test Zuul Pipelines
-------------------
Zuul is now running with a basic :term:`check` and :term:`gate`
configuration. We can now begin adding Zuul configuration to one of
our :term:`untrusted projects<untrusted-project>`. Start by cloning
the `test1` project which was created by the setup script.
.. code-block:: shell
cd ..
git clone http://localhost:8080/test1
Every Zuul job that runs needs a playbook, so let's create a
sub-directory in the project to hold playbooks:
.. code-block:: shell
cd test1
mkdir playbooks
Start with a simple playbook which just outputs a debug message. Copy
the following to ``playbooks/testjob.yaml``:
.. literalinclude:: examples/test1/playbooks/testjob.yaml
:language: yaml
Now define a Zuul job which runs that playbook. Zuul will read its
configuration from any of ``zuul.d/`` or ``.zuul.d/`` directories, or
the files ``zuul.yaml`` or ``.zuul.yaml``. Generally in an *untrusted
project* which isn't dedicated entirely to Zuul, it's best to put
Zuul's configuration in a hidden file. Copy the following to
``.zuul.yaml`` in the root of the project:
.. literalinclude:: examples/test1/zuul.yaml
:language: yaml
Commit the changes and push them up to Gerrit for review:
.. code-block:: shell
git add .zuul.yaml playbooks
git commit -m "Add test Zuul job"
git review
Zuul will dynamically evaluate proposed changes to its configuration
in *untrusted projects* immediately, so shortly after your change is
uploaded, Zuul will run the new job and report back on the change.
Visit http://localhost:8080/#/dashboard/self and open the change you
just uploaded. If the build is complete, Zuul should have left a
Verified: +1 vote on the change, along with a comment at the bottom.
Expand the comments and you should see that the job succeeded, but
there are no logs and no way to see the output, only a `finger` URL
(which is what Zuul reports when it doesn't know where build logs are
stored).
.. image:: images/check1-1002.png
:align: center
This means everything is working so far, but we need to configure a
bit more before we have a useful job.
Configure a Base Job
--------------------
Every Zuul tenant needs at least one base job. Zuul administrators
can use a base job to customize Zuul to the local environment. This
may include tasks which run both before jobs, such as setting up
package mirrors or networking configuration, or after jobs, such as
artifact and log storage.
Zuul doesn't take anything for granted, and even tasks such as copying
the git repos for the project being tested onto the remote node must
be explicitly added to a base job (and can therefore be customized as
needed). The Zuul in this tutorial is pre-configured to use the `zuul
jobs`_ repository which is the "standard library" of Zuul jobs and
roles. We will make use of it to quickly create a base job which
performs the necessary set up actions and stores build logs.
.. _zuul jobs: https://zuul-ci.org/docs/zuul-jobs/
Return to the ``zuul-config`` repo that you were working in earlier.
We're going to add some playbooks to the empty base job we created
earlier. Start by creating a directory to store those playbooks:
.. code-block:: shell
cd ..
cd zuul-config
mkdir -p playbooks/base
Zuul supports running any number of playbooks before a job (called
*pre-run* playbooks) or after a job (called *post-run* playbooks).
We're going to add a single *pre-run* playbook now. Copy the
following to ``playbooks/base/pre.yaml``:
.. literalinclude:: examples/zuul-config/playbooks/base/pre.yaml
:language: yaml
This playbook does two things; first it creates a new SSH key and adds
it to all of the hosts in the inventory, and removes the private key
that Zuul normally uses to log into nodes from the running SSH agent.
This is just an extra bit of protection which ensures that if Zuul's
SSH key has access to any important systems, normal Zuul jobs can't
use it. The second thing the playbook does is copy the git
repositories that Zuul has prepared (which may have one or more
changes being tested) to all of the nodes used in the job.
Next, add a *post-run* playbook to remove the per-build SSH key. Copy
the following to ``playbooks/base/post-ssh.yaml``:
.. literalinclude:: examples/zuul-config/playbooks/base/post-ssh.yaml
:language: yaml
This is the complement of the `add-build-sshkey` role in the pre-run
playbook -- it simply removes the per-build ssh key from any remote
systems. Zuul always tries to run all of the post-run playbooks
regardless of whether any previous playbooks have failed. Because we
always want log collection to run and we want it to run last, we
create a second post-run playbook for it. Copy the following to
``playbooks/base/post-logs.yaml``:
.. literalinclude:: examples/zuul-config/playbooks/base/post-logs.yaml
:language: yaml
This tutorial is running an Apache webserver in a container which will
serve build logs from a volume that is shared with the Zuul executor.
That volume is mounted at `/srv/static/logs`, which is the default
location in the `upload-logs`_ role. The role also supports copying
files to a remote server via SCP; see the role documentation for how
to configure it. For this simple case, the only option we need to
provide is the URL where the logs can ultimately be found.
.. note:: Zuul-jobs also contains a `role
<https://zuul-ci.org/docs/zuul-jobs/roles.html#role-upload-logs-swift>`_
to upload logs to an OpenStack Object Storage (swift)
container. If you create a role to upload logs to another
system, please feel free to contribute it to the zuul-jobs
repository for others to use.
.. _upload-logs: https://zuul-ci.org/docs/zuul-jobs/roles.html#role-upload-logs
Now that the new playbooks are in place, update the ``base`` job
definition to include them. Overwrite ``zuul.d/jobs.yaml`` with the
following:
.. literalinclude:: examples/zuul-config/zuul.d/jobs2.yaml
:language: yaml
Then commit the change and upload it to Gerrit for review:
.. code-block:: shell
git add playbooks zuul.d/jobs.yaml
git commit -m "Update Zuul base job"
git review
Visit http://localhost:8080/#/dashboard/self and open the
``zuul-config`` change you just uploaded.
You should see a Verified +1 vote from Zuul. Click `Reply` then vote
Code-Review: +2 and Workflow: +1 then click 'Post'.
.. image:: images/review-1003.png
:align: center
Wait a few moments for Zuul to process the event, and then reload the
page. The change should have been merged.
Visit http://localhost:8080/#/dashboard/self and return to the
``test1`` change you uploaded earlier. Click `Reply` then type
`recheck` into the text field and click `Post`.
.. image:: images/recheck-1002.png
:align: center
This will cause Zuul to re-run the test job we created earlier. This
time it will run with the updated base job configuration, and when
complete, it will report the published log location as a comment on
the change:
.. image:: images/check2-1002.png
:align: center
Follow the link and you will be able to browse the console log for the
job. In the middle of the log, you should see the "Hello, world!"
output from the job's playbook.
doc/source/admin/troubleshooting.rst
+10
View File
@@ -0,0 +1,10 @@
Troubleshooting
---------------
You can use telnet to connect to gearman to check which Zuul
components are online::
telnet <gearman_ip> 4730
Useful commands are ``workers`` and ``status`` which you can run by just
typing those commands once connected to gearman.
doc/source/admin/tutorial.rst
-464
View File
@@ -1,464 +0,0 @@
Installation and Setup Tutorial
===============================
Zuul is not like other CI or CD systems. It is a project gating
system designed to assist developers in taking a change from proposal
through deployment. Zuul can support any number of workflow processes
and systems, but to help you get started with Zuul, this tutorial will
walk through setting up a basic gating configuration which protects
projects from merging broken code.
This tutorial is entirely self-contained and may safely be run on a
workstation. The only requirements are a network connection and the
ability to run Docker containers. For code review, it provides an
instance of Gerrit, though the concepts you will learn apply equally
to GitHub. Even if you don't ultimately intend to use Gerrit, you are
encouraged to follow this tutorial to learn how to set up Zuul and
then consult further documentation to configure your Zuul to interact
with GitHub.
Start Zuul Containers
---------------------
Before you start, ensure that some needed packages are installed.
.. code-block:: shell
# Debian / Ubuntu:
sudo apt-get install docker-compose git git-review
# Red Hat / Fedora / SUSE:
sudo yum install docker-compose git git-review
Clone the Zuul repository:
.. code-block:: shell
git clone https://git.zuul-ci.org/zuul
Then cd into the directory containing this document, and run
docker-compose in order to start Zuul, Nodepool and Gerrit.
.. code-block:: shell
cd zuul/doc/source/admin/examples
docker-compose up
All of the services will be started with debug-level logging sent to
the standard output of the terminal where docker-compose is running.
You will see a considerable amount of information scroll by, including
some errors. Zuul will immediately attempt to connect to Gerrit and
begin processing, even before Gerrit has fully initialized. The
docker composition includes scripts to configure Gerrit and create an
account for Zuul. Once this has all completed, the system should
automatically connect, stabilize and become idle. When this is
complete, you will have the following services running:
* Zookeeper
* Gerrit
* Nodepool Launcher
* Zuul Scheduler
* Zuul Web Server
* Zuul Executor
* Apache HTTPD
And a long-running static test node used by Nodepool and Zuul upon
which to run tests.
The Zuul scheduler is configured to connect to Gerrit via a connection
named ``gerrit``. Zuul can interact with as many systems as
necessary, each such connection is assigned a name for use in the Zuul
configuration.
Zuul is a multi-tenant application, so that differing needs of
independent work-groups can be supported from one system. This
example configures a single tenant named ``example-tenant``. Assigned
to this tenant are three projects: ``zuul-config``, ``test1`` and
``test2``. These have already been created in Gerrit and are ready
for us to begin using.
Add Your Gerrit Account
-----------------------
Before you can interact with Gerrit, you will need to create an
account. The initialization script has already created an account for
Zuul, but has left the task of creating your own account to you so
that you can provide your own SSH key. You may safely use any
existing SSH key on your workstation, or you may create a new one by
running ``ssh-keygen``.
Gerrit is configured in a development mode where passwords are not
required in the web interface and you may become any user in the
system at any time.
To create your Gerrit account, visit http://localhost:8080 in your
browser and click `Become`.
.. image:: images/become.png
:align: center
Then click `New Account` under `Register`.
.. image:: images/register.png
:align: center
Enter your full name and click `Save Changes`.
.. image:: images/name.png
:align: center
Enter the username you use to log into your workstation in the
`Username` field and click `Select Username`.
.. image:: images/username.png
:align: center
Copy and paste the contents of ``~/.ssh/id_rsa.pub`` into the SSH key
field and click `Add`.
.. image:: images/sshkey.png
:align: center
Click the `Continue` link at the bottom of the page.
.. image:: images/continue.png
:align: center
At this point you have created and logged into your personal account
in Gerrit and are ready to begin configuring Zuul.
Configure Zuul Pipelines
------------------------
Zuul recognizes two types of projects: :term:`config
projects<config-project>` and :term:`untrusted
projects<untrusted-project>`. An *untrusted project* is a normal
project from Zuul's point of view. In a gating system, it contains
the software under development and/or most of the job content that
Zuul will run. A *config project* is a special project that contains
the Zuul's configuration. Because it has access to normally
restricted features in Zuul, changes to this repository are not
dynamically evaluated by Zuul. The security and functionality of the
rest of the system depends on this repository, so it is best to limit
what is contained within it to the minimum, and ensure thorough code
review practices when changes are made.
Zuul has no built-in workflow definitions, so in order for it to do
anything, you will need to begin by making changes to a *config
project*. The initialization script has already created a project
named ``zuul-config`` which you should now clone onto your workstation:
.. code-block:: shell
git clone http://localhost:8080/zuul-config
You will find that this repository is empty. Zuul reads its
configuration from either a single file or a directory. In a *Config
Project* with substantial Zuul configuration, you may find it easiest
to use the ``zuul.d`` directory for Zuul configuration. Later, in
*Untrusted Projects* you will use a single file for in-repo
configuration. Make the directory:
.. code-block:: shell
cd zuul-config
mkdir zuul.d
The first type of configuration items we need to add are the Pipelines
we intend to use. In Zuul, a Pipeline represents a workflow action.
It is triggered by some action on a connection. Projects are able to
attach jobs to run in that pipeline, and when they complete, the
results are reported along with actions which may trigger further
Pipelines. In a gating system two pipelines are required:
:term:`check` and :term:`gate`. In our system, ``check`` will be
triggered when a patch is uploaded to Gerrit, so that we are able to
immediately run tests and report whether the change works and is
therefore able to merge. The ``gate`` pipeline is triggered when a code
reviewer approves the change in Gerrit. It will run test jobs again
(in case other changes have merged since the change in question was
uploaded) and if these final tests pass, will automatically merge the
change. To configure these pipelines, copy the following file into
`zuul.d/pipelines.yaml`:
.. literalinclude:: examples/zuul-config/zuul.d/pipelines.yaml
:language: yaml
Once we have bootstrapped our initial Zuul configuration, we will want
to use the gating process on this repository too, so we need to attach
the ``zuul-config`` repository to the ``check`` and ``gate`` pipelines
we are about to create. There are no jobs defined yet, so we must use
the internally defined ``noop`` job, which always returns success.
Later on we will be configuring some other projects, and while we will
be able to dynamically add jobs to their pipelines, those projects
must first be attached to the pipelines in order for that to work. In
our system, we want all of the projects in Gerrit to participate in
the check and gate pipelines, so we can use a regular expression to
apply this to all projects. To configure the ``check`` and ``gate``
pipelines for ``zuul-config`` to run the ``noop`` job, and add all
projects to those pipelines (with no jobs), copy the following file
into ``zuul.d/projects.yaml``:
.. literalinclude:: examples/zuul-config/zuul.d/projects.yaml
:language: yaml
Every real job (i.e., all jobs other than ``noop``) must inherit from a
:term:`base job`, and base jobs may only be defined in a
:term:`config-project`. Let's go ahead and add a simple base job that
we can build on later. Copy the following into ``zuul.d/jobs.yaml``:
.. literalinclude:: examples/zuul-config/zuul.d/jobs.yaml
:language: yaml
Commit the changes and push them up for review:
.. code-block:: shell
git add zuul.d
git commit -m "Add initial Zuul configuration"
git review
Because Zuul is currently running with no configuration whatsoever, it
will ignore this change. For this initial change which bootstraps the
entire system, we will need to bypass code review (hopefully for the
last time). To do this, you need to switch to the Administrator
account in Gerrit. Visit http://localhost:8080 in your browser and
then:
Click on your name in the top right corner then click `Switch
Account`.
.. image:: images/switch-example.png
:align: center
Click `admin` to log in as the `admin` user.
.. image:: images/become-select.png
:align: center
In the top left corner of the page, click `All` and `Open` to see the
list of open changes, then click on the change you uploaded.
.. image:: images/open-changes.png
:align: center
Click `Reply...` at the top center of the change screen. This will
open a dialog where you can leave a review message and vote on the
change. As the administrator, you have access to vote in all of the
review categories, even `Verified` which is normally reserved for
Zuul. Vote Code-Review: +2, Verified: +2, Workflow: +1, and then
click `Post` to leave your approval votes.
.. image:: images/review-1001.png
:align: center
Once the required votes have been set, the `Submit` button will
appear; click it. This will cause the change to be merged
immediately. This is normally handled by Zuul, but as the
administrator you can bypass Zuul to forcibly merge a change.
.. image:: images/submit-1001.png
:align: center
Now that the initial configuration has been bootstrapped, you should
not need to bypass testing and code review again, so switch back to
the account you created for yourself. Click on `Administrator` in the
top right corner then click `Switch Account`.
.. image:: images/switch-admin.png
:align: center
Click your username.
.. image:: images/become-select.png
:align: center
Test Zuul Pipelines
-------------------
Zuul is now running with a basic :term:`check` and :term:`gate`
configuration. We can now begin adding Zuul configuration to one of
our :term:`untrusted projects<untrusted-project>`. Start by cloning
the `test1` project which was created by the setup script.
.. code-block:: shell
cd ..
git clone http://localhost:8080/test1
Every Zuul job that runs needs a playbook, so let's create a
sub-directory in the project to hold playbooks:
.. code-block:: shell
cd test1
mkdir playbooks
Start with a simple playbook which just outputs a debug message. Copy
the following to ``playbooks/testjob.yaml``:
.. literalinclude:: examples/test1/playbooks/testjob.yaml
:language: yaml
Now define a Zuul job which runs that playbook. Zuul will read its
configuration from any of ``zuul.d/`` or ``.zuul.d/`` directories, or
the files ``zuul.yaml`` or ``.zuul.yaml``. Generally in an *untrusted
project* which isn't dedicated entirely to Zuul, it's best to put
Zuul's configuration in a hidden file. Copy the following to
``.zuul.yaml`` in the root of the project:
.. literalinclude:: examples/test1/zuul.yaml
:language: yaml
Commit the changes and push them up to Gerrit for review:
.. code-block:: shell
git add .zuul.yaml playbooks
git commit -m "Add test Zuul job"
git review
Zuul will dynamically evaluate proposed changes to its configuration
in *untrusted projects* immediately, so shortly after your change is
uploaded, Zuul will run the new job and report back on the change.
Visit http://localhost:8080/#/dashboard/self and open the change you
just uploaded. If the build is complete, Zuul should have left a
Verified: +1 vote on the change, along with a comment at the bottom.
Expand the comments and you should see that the job succeeded, but
there are no logs and no way to see the output, only a `finger` URL
(which is what Zuul reports when it doesn't know where build logs are
stored).
.. image:: images/check1-1002.png
:align: center
This means everything is working so far, but we need to configure a
bit more before we have a useful job.
Configure a Base Job
--------------------
Every Zuul tenant needs at least one base job. Zuul administrators
can use a base job to customize Zuul to the local environment. This
may include tasks which run both before jobs, such as setting up
package mirrors or networking configuration, or after jobs, such as
artifact and log storage.
Zuul doesn't take anything for granted, and even tasks such as copying
the git repos for the project being tested onto the remote node must
be explicitly added to a base job (and can therefore be customized as
needed). The Zuul in this tutorial is pre-configured to use the `zuul
jobs`_ repository which is the "standard library" of Zuul jobs and
roles. We will make use of it to quickly create a base job which
performs the necessary set up actions and stores build logs.
.. _zuul jobs: https://zuul-ci.org/docs/zuul-jobs/
Return to the ``zuul-config`` repo that you were working in earlier.
We're going to add some playbooks to the empty base job we created
earlier. Start by creating a directory to store those playbooks:
.. code-block:: shell
cd ..
cd zuul-config
mkdir -p playbooks/base
Zuul supports running any number of playbooks before a job (called
*pre-run* playbooks) or after a job (called *post-run* playbooks).
We're going to add a single *pre-run* playbook now. Copy the
following to ``playbooks/base/pre.yaml``:
.. literalinclude:: examples/zuul-config/playbooks/base/pre.yaml
:language: yaml
This playbook does two things; first it creates a new SSH key and adds
it to all of the hosts in the inventory, and removes the private key
that Zuul normally uses to log into nodes from the running SSH agent.
This is just an extra bit of protection which ensures that if Zuul's
SSH key has access to any important systems, normal Zuul jobs can't
use it. The second thing the playbook does is copy the git
repositories that Zuul has prepared (which may have one or more
changes being tested) to all of the nodes used in the job.
Next, add a *post-run* playbook to remove the per-build SSH key. Copy
the following to ``playbooks/base/post-ssh.yaml``:
.. literalinclude:: examples/zuul-config/playbooks/base/post-ssh.yaml
:language: yaml
This is the complement of the `add-build-sshkey` role in the pre-run
playbook -- it simply removes the per-build ssh key from any remote
systems. Zuul always tries to run all of the post-run playbooks
regardless of whether any previous playbooks have failed. Because we
always want log collection to run and we want it to run last, we
create a second post-run playbook for it. Copy the following to
``playbooks/base/post-logs.yaml``:
.. literalinclude:: examples/zuul-config/playbooks/base/post-logs.yaml
:language: yaml
This tutorial is running an Apache webserver in a container which will
serve build logs from a volume that is shared with the Zuul executor.
That volume is mounted at `/srv/static/logs`, which is the default
location in the `upload-logs`_ role. The role also supports copying
files to a remote server via SCP; see the role documentation for how
to configure it. For this simple case, the only option we need to
provide is the URL where the logs can ultimately be found.
.. note:: Zuul-jobs also contains a `role
<https://zuul-ci.org/docs/zuul-jobs/roles.html#role-upload-logs-swift>`_
to upload logs to an OpenStack Object Storage (swift)
container. If you create a role to upload logs to another
system, please feel free to contribute it to the zuul-jobs
repository for others to use.
.. _upload-logs: https://zuul-ci.org/docs/zuul-jobs/roles.html#role-upload-logs
Now that the new playbooks are in place, update the ``base`` job
definition to include them. Overwrite ``zuul.d/jobs.yaml`` with the
following:
.. literalinclude:: examples/zuul-config/zuul.d/jobs2.yaml
:language: yaml
Then commit the change and upload it to Gerrit for review:
.. code-block:: shell
git add playbooks zuul.d/jobs.yaml
git commit -m "Update Zuul base job"
git review
Visit http://localhost:8080/#/dashboard/self and open the
``zuul-config`` change you just uploaded.
You should see a Verified +1 vote from Zuul. Click `Reply` then vote
Code-Review: +2 and Workflow: +1 then click 'Post'.
.. image:: images/review-1003.png
:align: center
Wait a few moments for Zuul to process the event, and then reload the
page. The change should have been merged.
Visit http://localhost:8080/#/dashboard/self and return to the
``test1`` change you uploaded earlier. Click `Reply` then type
`recheck` into the text field and click `Post`.
.. image:: images/recheck-1002.png
:align: center
This will cause Zuul to re-run the test job we created earlier. This
time it will run with the updated base job configuration, and when
complete, it will report the published log location as a comment on
the change:
.. image:: images/check2-1002.png
:align: center
Follow the link and you will be able to browse the console log for the
job. In the middle of the log, you should see the "Hello, world!"
output from the job's playbook.
doc/source/admin/zuul-from-scratch.rst
+16 -7
View File
@@ -1,9 +1,14 @@
Zuul From Scratch
=================
.. note:: This is a work in progress that attempts to walk through all
of the steps needed to run Zuul on a all-in-one server, and
demonstrate running against either a GitHub or Gerrit project.
This document details a fully manual installation of Zuul on a
all-in-one server. If you want to learn all the details about how to
install Zuul without the aid of any existing installation tools, you
may find this a useful reference.
If, instead, you want to get Zuul running quickly, see the
:ref:`quick_start` which runs all of the Zuul services in containers
with a single command.
Environment Setup
-----------------
@@ -40,7 +45,7 @@ server to listen on all public addresses. This is so that Zuul may
receive webhook events from GitHub. You may wish to proxy this or
further restrict public access.
::
.. code-block:: shell
sudo bash -c "cat > /etc/zuul/zuul.conf <<EOF
[gearman]
@@ -68,7 +73,7 @@ Starting Services
-----------------
After you have Zookeeper, Nodepool, and Zuul installed and configured, you can
start Nodepool and Zuul services with::
start Nodepool and Zuul services with::
sudo systemctl daemon-reload
@@ -93,7 +98,9 @@ Zuul provides a `standard library`_ of jobs and roles. To take advantage
of these jobs, add the ``zuul-jobs`` repo, which is hosted by the Zuul
project, to your system.
Add to ``/etc/zuul/zuul.conf``::
Add to ``/etc/zuul/zuul.conf``:
.. code-block:: shell
sudo bash -c "cat >> /etc/zuul/zuul.conf <<EOF
@@ -102,7 +109,9 @@ Add to ``/etc/zuul/zuul.conf``::
baseurl=https://git.zuul-ci.org/
EOF"
Restart executor and scheduler::
Restart executor and scheduler:
.. code-block:: shell
sudo systemctl restart zuul-executor.service
sudo systemctl restart zuul-scheduler.service