Add zuul from scratch doc
This is just a skeleton for now, and should be expanded, but it should be correct and up to date for what we expect in the v3 release. Change-Id: I305200cbfc128cc1bc09e809540a873f60e0fe5c Co-Authored-By: Leif Madsen <lmadsen@redhat.com> Co-Authored-By: Monty Taylor <mordred@inaugust.com> Co-Authored-By: Paul Belanger <pabelanger@redhat.com>
This commit is contained in:
James E. Blair
parent
687f4df835
commit
f09c238185
|
|
@ -40,60 +40,43 @@ Then in the zuul.conf, set webhook_token and api_token.
|
|||
Application
|
||||
...........
|
||||
|
||||
.. NOTE Duplicate content here and in zuul-from-scratch.rst. Keep them
|
||||
in sync.
|
||||
|
||||
To create a `GitHub application
|
||||
<https://developer.github.com/apps/building-integrations/setting-up-and-registering-github-apps/registering-github-apps/>`_:
|
||||
|
||||
* Go to your organization settings page to create the application, e.g.:
|
||||
https://github.com/organizations/my-org/settings/apps/new
|
||||
|
||||
* Set GitHub App name to "my-org-zuul"
|
||||
|
||||
* Set Setup URL to your setup documentation, when user install the application
|
||||
they are redirected to this url
|
||||
|
||||
* Set Webhook URL to
|
||||
``http://<zuul-hostname>/connection/<connection-name>/payload``.
|
||||
|
||||
* Create a Webhook secret
|
||||
|
||||
* Set permissions:
|
||||
|
||||
* Commit statuses: Read & Write
|
||||
|
||||
* Issues: Read & Write
|
||||
|
||||
* Pull requests: Read & Write
|
||||
|
||||
* Repository contents: Read & Write (write to let zuul merge change)
|
||||
* Repository administration: Read
|
||||
|
||||
* Set events subscription:
|
||||
|
||||
* Label
|
||||
|
||||
* Status
|
||||
|
||||
* Issue comment
|
||||
|
||||
* Issues
|
||||
|
||||
* Pull request
|
||||
|
||||
* Pull request review
|
||||
|
||||
* Pull request review comment
|
||||
|
||||
* Commit comment
|
||||
|
||||
* Create
|
||||
|
||||
* Push
|
||||
|
||||
* Release
|
||||
|
||||
* Set Where can this GitHub App be installed to "Any account"
|
||||
|
||||
* Create the App
|
||||
|
||||
* Generate a Private key in the app settings page
|
||||
|
||||
Then in the zuul.conf, set webhook_token, app_id and app_key.
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@ provides to in-project configuration.
|
|||
:maxdepth: 2
|
||||
|
||||
quick-start
|
||||
zuul-from-scratch
|
||||
installation
|
||||
components
|
||||
connections
|
||||
|
|
|
|||
|
|
@ -0,0 +1,505 @@
|
|||
Zuul From Scratch
|
||||
=================
|
||||
|
||||
.. note:: This is a work in progress that attempts to walk through all
|
||||
of the steps needed to run Zuul on a cloud server against
|
||||
GitHub projects.
|
||||
|
||||
Environment Setup
|
||||
-----------------
|
||||
|
||||
We're going to be using Fedora 27 on a cloud server for this
|
||||
installation.
|
||||
|
||||
Login to your environment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Since we'll be using a cloud image for Fedora 27, our login user will
|
||||
be ``fedora`` which will also be the staging user for installation of
|
||||
Zuul and Nodepool.
|
||||
|
||||
To get started, ssh to your machine as the ``fedora`` user::
|
||||
|
||||
ssh fedora@<ip_address>
|
||||
|
||||
Environment Setup
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
::
|
||||
|
||||
sudo dnf update -y
|
||||
sudo systemctl reboot
|
||||
sudo dnf install git redhat-lsb-core python3 python3-pip python3-devel make gcc openssl-devel python-openstackclient -y
|
||||
pip3 install --user bindep
|
||||
|
||||
Zuul and Nodepool Installation
|
||||
------------------------------
|
||||
|
||||
Install Zookeeper
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
::
|
||||
|
||||
sudo dnf install zookeeper -y
|
||||
|
||||
Install Nodepool
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
::
|
||||
|
||||
sudo adduser --system nodepool --home-dir /var/lib/nodepool --create-home
|
||||
git clone https://git.openstack.org/openstack-infra/nodepool
|
||||
cd nodepool/
|
||||
sudo dnf -y install $(bindep -b)
|
||||
sudo pip3 install .
|
||||
|
||||
Install Zuul
|
||||
~~~~~~~~~~~~
|
||||
|
||||
::
|
||||
|
||||
sudo adduser --system zuul --home-dir /var/lib/zuul --create-home
|
||||
git clone https://git.openstack.org/openstack-infra/zuul
|
||||
cd zuul/
|
||||
sudo dnf install $(bindep -b) -y
|
||||
sudo pip3 install git+https://github.com/sigmavirus24/github3.py.git@develop#egg=Github3.py
|
||||
sudo pip3 install .
|
||||
|
||||
Setup
|
||||
-----
|
||||
|
||||
Zookeeper Setup
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
.. TODO recommended reading for zk clustering setup
|
||||
|
||||
::
|
||||
|
||||
sudo bash -c 'echo "1" > /etc/zookeeper/myid'
|
||||
sudo bash -c 'echo "tickTime=2000
|
||||
dataDir=/var/lib/zookeeper
|
||||
clientPort=2181" > /etc/zookeeper/zoo.cfg'
|
||||
|
||||
Nodepool Setup
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
Before starting on this, you need to download your `openrc`
|
||||
configuration from your OpenStack cloud. Put it on your server in the
|
||||
fedora user's home directory. It should be called
|
||||
``<username>-openrc.sh``. Once that is done, create a new keypair
|
||||
that will be installed when instantiating the servers::
|
||||
|
||||
cd ~
|
||||
source <username>-openrc.sh # this will prompt for password - enter it
|
||||
umask 0066
|
||||
|
||||
ssh-keygen -t rsa -b 2048 -f nodepool_rsa # don't enter a passphrase
|
||||
openstack keypair create --public-key nodepool_rsa.pub nodepool
|
||||
|
||||
We'll use the private key later wheen configuring Zuul. In the same
|
||||
session, configure nodepool to talk to your cloud::
|
||||
|
||||
sudo mkdir -p ~nodepool/.config/openstack
|
||||
cat > clouds.yaml <<EOF
|
||||
clouds:
|
||||
mycloud:
|
||||
auth:
|
||||
username: $OS_USERNAME
|
||||
password: $OS_PASSWORD
|
||||
project_name: ${OS_PROJECT_NAME:-$OS_TENANT_NAME}
|
||||
auth_url: $OS_AUTH_URL
|
||||
region_name: $OS_REGION_NAME
|
||||
EOF
|
||||
sudo mv clouds.yaml ~nodepool/.config/openstack/
|
||||
sudo chown -R nodepool.nodepool ~nodepool/.config
|
||||
umask 0002
|
||||
|
||||
Once you've written out the file, double check all the required fields have been filled out.
|
||||
|
||||
::
|
||||
|
||||
sudo mkdir /etc/nodepool/
|
||||
sudo mkdir /var/log/nodepool
|
||||
sudo chgrp -R nodepool /var/log/nodepool/
|
||||
sudo chmod 775 /var/log/nodepool/
|
||||
|
||||
Nodepool Configuration
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Inputs needed for this file:
|
||||
|
||||
* cloud name / region name - from clouds.yaml
|
||||
* flavor-name
|
||||
* image-name - from your cloud
|
||||
|
||||
::
|
||||
|
||||
sudo bash -c "cat >/etc/nodepool/nodepool.yaml <<EOF
|
||||
zookeeper-servers:
|
||||
- host: localhost
|
||||
port: 2181
|
||||
|
||||
providers:
|
||||
- name: myprovider # this is a nodepool identifier for this cloud provider (cloud+region combo)
|
||||
region-name: regionOne # this needs to match the region name in clouds.yaml but is only needed if there is more than one region
|
||||
cloud: mycloud # This needs to match the name in clouds.yaml
|
||||
cloud-images:
|
||||
- name: centos-7 # Defines a cloud-image for nodepool
|
||||
image-name: CentOS-7-x86_64-GenericCloud-1706 # name of image from cloud
|
||||
username: centos # The user Zuul should log in as
|
||||
pools:
|
||||
- name: main
|
||||
max-servers: 4 # nodepool will never create more than this many servers
|
||||
labels:
|
||||
- name: centos-7-small # defines label that will be used to get one of these in a job
|
||||
flavor-name: 'm1.small' # name of flavor from cloud
|
||||
cloud-image: centos-7 # matches name from cloud-images
|
||||
key-name: nodepool # name of the keypair to use for authentication
|
||||
|
||||
labels:
|
||||
- name: centos-7-small # defines label that will be used in jobs
|
||||
min-ready: 2 # nodepool will always keep this many booted and ready to go
|
||||
EOF"
|
||||
|
||||
.. warning::
|
||||
|
||||
`min-ready:2` may incur costs in your cloud provider
|
||||
|
||||
|
||||
Zuul Setup
|
||||
~~~~~~~~~~
|
||||
|
||||
::
|
||||
|
||||
sudo mkdir /etc/zuul/
|
||||
sudo mkdir /var/log/zuul/
|
||||
sudo chown zuul.zuul /var/log/zuul/
|
||||
sudo mkdir /var/lib/zuul/.ssh
|
||||
sudo chmod 0700 /var/lib/zuul/.ssh
|
||||
sudo mv nodepool_rsa /var/lib/zuul/.ssh
|
||||
sudo chown -R zuul.zuul /var/lib/zuul/.ssh
|
||||
|
||||
Zuul Configuration
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Write the Zuul config file. Note that this configures Zuul's web
|
||||
server to listen on all public addresses. This is so that Zuul may
|
||||
receive webhook events from GitHub. You may wish to proxy this or
|
||||
further restrict public access.
|
||||
|
||||
::
|
||||
|
||||
sudo bash -c "cat > /etc/zuul/zuul.conf <<EOF
|
||||
[gearman]
|
||||
server=127.0.0.1
|
||||
|
||||
[gearman_server]
|
||||
start=true
|
||||
|
||||
[executor]
|
||||
private_key_file=/home/zuul/.ssh/nodepool_rsa
|
||||
|
||||
[web]
|
||||
listen_address=0.0.0.0
|
||||
|
||||
[scheduler]
|
||||
tenant_config=/etc/zuul/main.yaml
|
||||
EOF"
|
||||
|
||||
sudo bash -c "cat > /etc/zuul/main.yaml <<EOF
|
||||
- tenant:
|
||||
name: quickstart
|
||||
EOF"
|
||||
|
||||
Service Management
|
||||
------------------
|
||||
|
||||
Zookeeper Service Management
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
::
|
||||
|
||||
sudo systemctl start zookeeper.service
|
||||
|
||||
::
|
||||
|
||||
sudo systemctl status zookeeper.service
|
||||
● zookeeper.service - Apache ZooKeeper
|
||||
Loaded: loaded (/usr/lib/systemd/system/zookeeper.service; disabled; vendor preset: disabled)
|
||||
Active: active (running) since Wed 2018-01-03 14:53:47 UTC; 5s ago
|
||||
Process: 4153 ExecStart=/usr/bin/zkServer.sh start zoo.cfg (code=exited, status=0/SUCCESS)
|
||||
Main PID: 4160 (java)
|
||||
Tasks: 17 (limit: 4915)
|
||||
CGroup: /system.slice/zookeeper.service
|
||||
└─4160 java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/share/java/
|
||||
|
||||
::
|
||||
|
||||
sudo systemctl enable zookeeper.service
|
||||
|
||||
|
||||
Nodepool Service Management
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
::
|
||||
|
||||
sudo bash -c "cat > /etc/systemd/system/nodepool-launcher.service <<EOF
|
||||
[Unit]
|
||||
Description=Nodepool Launcher Service
|
||||
After=syslog.target network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
# Options to pass to nodepool-launcher.
|
||||
Group=nodepool
|
||||
User=nodepool
|
||||
RuntimeDirectory=nodepool
|
||||
ExecStart=/usr/local/bin/nodepool-launcher
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF"
|
||||
|
||||
sudo chmod 0644 /etc/systemd/system/nodepool-launcher.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl start nodepool-launcher.service
|
||||
sudo systemctl status nodepool-launcher.service
|
||||
sudo systemctl enable nodepool-launcher.service
|
||||
|
||||
Zuul Service Management
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
::
|
||||
|
||||
sudo bash -c "cat > /etc/systemd/system/zuul-scheduler.service <<EOF
|
||||
[Unit]
|
||||
Description=Zuul Scheduler Service
|
||||
After=syslog.target network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
Group=zuul
|
||||
User=zuul
|
||||
RuntimeDirectory=zuul
|
||||
ExecStart=/usr/local/bin/zuul-scheduler
|
||||
ExecStop=/usr/local/bin/zuul-scheduler stop
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF"
|
||||
|
||||
sudo bash -c "cat > /etc/systemd/system/zuul-executor.service <<EOF
|
||||
[Unit]
|
||||
Description=Zuul Executor Service
|
||||
After=syslog.target network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
Group=zuul
|
||||
User=zuul
|
||||
RuntimeDirectory=zuul
|
||||
ExecStart=/usr/local/bin/zuul-executor
|
||||
ExecStop=/usr/local/bin/zuul-executor stop
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF"
|
||||
|
||||
sudo bash -c "cat > /etc/systemd/system/zuul-web.service <<EOF
|
||||
[Unit]
|
||||
Description=Zuul Web Service
|
||||
After=syslog.target network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
Group=zuul
|
||||
User=zuul
|
||||
RuntimeDirectory=zuul
|
||||
ExecStart=/usr/local/bin/zuul-web
|
||||
ExecStop=/usr/local/bin/zuul-web stop
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF"
|
||||
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl start zuul-scheduler.service
|
||||
sudo systemctl status zuul-scheduler.service
|
||||
sudo systemctl enable zuul-scheduler.service
|
||||
sudo systemctl start zuul-executor.service
|
||||
sudo systemctl status zuul-executor.service
|
||||
sudo systemctl enable zuul-executor.service
|
||||
sudo systemctl start zuul-web.service
|
||||
sudo systemctl status zuul-web.service
|
||||
sudo systemctl enable zuul-web.service
|
||||
|
||||
Use Zuul Jobs
|
||||
-------------
|
||||
|
||||
Add to ``/etc/zuul/zuul.conf``::
|
||||
|
||||
sudo bash -c "cat >> /etc/zuul/zuul.conf <<EOF
|
||||
|
||||
[connection zuul-git]
|
||||
driver=git
|
||||
baseurl=https://git.openstack.org/
|
||||
EOF"
|
||||
|
||||
Restart executor and scheduler::
|
||||
|
||||
sudo systemctl restart zuul-executor.service
|
||||
sudo systemctl restart zuul-scheduler.service
|
||||
|
||||
Configure GitHub
|
||||
----------------
|
||||
|
||||
You'll need an organization in Github for this, so create one if you
|
||||
haven't already. In this example we will use `my-org`.
|
||||
|
||||
.. NOTE Duplicate content here and in drivers/github.rst. Keep them
|
||||
in sync.
|
||||
|
||||
Create a `GitHub application
|
||||
<https://developer.github.com/apps/building-integrations/setting-up-and-registering-github-apps/registering-github-apps/>`_:
|
||||
|
||||
* Go to your organization settings page to create the application, e.g.:
|
||||
https://github.com/organizations/my-org/settings/apps/new
|
||||
* Set GitHub App name to "my-org-zuul"
|
||||
* Set Setup URL to your setup documentation, when users install the application
|
||||
they are redirected to this url
|
||||
* Set Webhook URL to
|
||||
``http://<IP ADDRESS>/connection/github/payload``.
|
||||
* Create a Webhook secret, and record it for later use
|
||||
* Set permissions:
|
||||
|
||||
* Commit statuses: Read & Write
|
||||
* Issues: Read & Write
|
||||
* Pull requests: Read & Write
|
||||
* Repository contents: Read & Write (write to let zuul merge change)
|
||||
* Repository administration: Read
|
||||
|
||||
* Set events subscription:
|
||||
|
||||
* Label
|
||||
* Status
|
||||
* Issue comment
|
||||
* Issues
|
||||
* Pull request
|
||||
* Pull request review
|
||||
* Pull request review comment
|
||||
* Commit comment
|
||||
* Create
|
||||
* Push
|
||||
* Release
|
||||
|
||||
* Set Where can this GitHub App be installed to "Any account"
|
||||
* Create the App
|
||||
* Generate a Private key in the app settings page and save the file for later
|
||||
|
||||
.. TODO See if we can script this using GitHub API
|
||||
|
||||
Go back to the `General` settings page for the app,
|
||||
https://github.com/organizations/my-org/settings/apps/my-org-zuul
|
||||
and look for the app `ID` number, under the `About` section.
|
||||
|
||||
Edit ``/etc/zuul/zuul.conf`` to add the following::
|
||||
|
||||
[connection github]
|
||||
driver=github
|
||||
app_id=<APP ID NUMBER>
|
||||
app_key=/etc/zuul/github.pem
|
||||
webhook_token=<WEBHOOK TOKEN>
|
||||
|
||||
Upload the private key which was generated earlier, and save it in
|
||||
``/etc/zuul/github.pem``.
|
||||
|
||||
Restart all of Zuul::
|
||||
|
||||
sudo systemctl restart zuul-executor.service
|
||||
sudo systemctl restart zuul-web.service
|
||||
sudo systemctl restart zuul-scheduler.service
|
||||
|
||||
Go to the `Advanced` tab for the app in GitHub,
|
||||
https://github.com/organizations/my-org/settings/apps/my-org-zuul/advanced,
|
||||
and look for the initial ping from the app. It probably wasn't
|
||||
delivered since Zuul wasn't configured at the time, so click
|
||||
``Resend`` and verify that it is delivered now that Zuul is
|
||||
configured.
|
||||
|
||||
Visit the public app page on GitHub,
|
||||
https://github.com/apps/my-org-zuul, and install the app into your org.
|
||||
|
||||
Create two new repositories in your org. One will hold the
|
||||
configuration for this tenant in Zuul, the other should be a normal
|
||||
project repo to use for testing. We'll call them `zuul-test-config`
|
||||
and `zuul-test`, respectively.
|
||||
|
||||
Edit ``/etc/zuul/main.yaml`` so that it looks like this::
|
||||
|
||||
- tenant:
|
||||
name: quickstart
|
||||
source:
|
||||
zuul-git:
|
||||
config-projects:
|
||||
- openstack-infra/zuul-base-jobs
|
||||
untrusted-projects:
|
||||
- openstack-infra/zuul-jobs
|
||||
github:
|
||||
config-projects:
|
||||
- my-org/zuul-test-config
|
||||
untrusted-projects:
|
||||
- my-org/zuul-test
|
||||
|
||||
The first section, under 'zuul-git' imports the "standard library" of
|
||||
Zuul jobs, a collection of jobs that can be used by any Zuul
|
||||
installation.
|
||||
|
||||
The second section is your GitHub configuration.
|
||||
|
||||
After updating the file, restart the Zuul scheduler::
|
||||
|
||||
sudo systemctl restart zuul-scheduler.service
|
||||
|
||||
Add an initial pipeline configuration to the `zuul-test-config`
|
||||
repository. Inside that project, create a ``zuul.yaml`` file with the
|
||||
following contents::
|
||||
|
||||
- pipeline:
|
||||
name: check
|
||||
description: |
|
||||
Newly opened pull requests enter this pipeline to receive an
|
||||
initial verification
|
||||
manager: independent
|
||||
trigger:
|
||||
github:
|
||||
- event: pull_request
|
||||
action:
|
||||
- opened
|
||||
- changed
|
||||
- reopened
|
||||
- event: pull_request
|
||||
action: comment
|
||||
comment: (?i)^\s*recheck\s*$
|
||||
start:
|
||||
github:
|
||||
status: pending
|
||||
comment: false
|
||||
success:
|
||||
github:
|
||||
status: 'success'
|
||||
failure:
|
||||
github:
|
||||
status: 'failure'
|
||||
|
||||
Merge that commit into the repository.
|
||||
|
||||
In the `zuul-test` project, create a `.zuul.yaml` file with the
|
||||
following contents::
|
||||
|
||||
- project:
|
||||
check:
|
||||
jobs:
|
||||
- noop
|
||||
|
||||
Open a new pull request with that commit against the `zuul-test`
|
||||
project and verify that Zuul reports a successful run of the `noop`
|
||||
job.
|
||||
Loading…
Reference in New Issue