inaugust
/
inaugust.com
Code Issues Proposed changes
inaugust.com/src/zuulv3/tutorial.rst

2290 lines
48 KiB
ReStructuredText
Raw Permalink Blame History

. display in 68x24
.. display in 88x24
.. pygments yaml? (only file breaks (---) tinted)
.. slide on high level v3 changes
.. slide on nodepool
.. transition:: dissolve
:duration: 0.4
Test Slide
==========
.. hidetitle::
.. ansi:: images/testslide.ans
Preshow
=======
.. hidetitle::
.. ansi:: images/cursor.ans images/cursor2.ans
Zuul
====
.. hidetitle::
.. ansi:: images/title.ans
Important Links
===============
* https://opendev.org/zuul
* https://zuul-ci.org/docs/zuul
* https://zuul-ci.org/docs/zuul-jobs/
* https://docs.openstack.org/infra/openstack-zuul-jobs/
* freenode:#zuul
* https://opendev.org/inaugust/inaugust.com/src/branch/master/src/zuulv3/tutorial.rst
Overview
========
* Discussion of concepts
* Installation of software
* Configurting Zuul
* Writing jobs
Please Ask Questions
====================
Installation of Software
========================
Ways to Install Zuul
====================
* Windmill: https://opendev.org/windmill/windmill/
* Software Factory: https://softwarefactory-project.io/
* Puppet: https://opendev.org/opendev/puppet-zuul/
* Containers: https://hub.docker.com/_/zuul/
Today we'll be using Containers with docker-compose
===================================================
Getting Started
===============
While we talk about other things ...
* Install docker, docker-compose, git-review
Debian/Ubuntu:
::
apt-get install docker-compose git git-review
Red Hat / SuSE
::
yum install docker-compose git git-review
* mkdir -p ~/src/opendev.org/zuul
* cd ~/src/opendev.org/zuul
* git clone https://opendev.org/zuul/zuul
* cd zuul
* cd doc/source/admin/examples
* docker-compose up
Output in docker-compose window
===============================
* All services running with debug logging to stdout
* Tons of information will have been output - including some errors
* Zuul connects to Gerrit before it's fully configured
* As it becomes configured, Zuul notices and becomes happy
* Once happy, it should stablize and become idle
We'll come back to this
=======================
It's going to do a bunch of network - and this is a conference.
Red Hat
=======
.. hidetitle::
.. container:: handout
i work for
.. ansi:: images/redhat.ans
OpenStack
=========
.. hidetitle::
.. ansi:: images/openstack.ans
OpenDev
=======
::
"most insane CI infrastructure I've ever been a part of"
-- Alex Gaynor
"OpenStack Infra are like the SpaceX of CI"
-- Emily Dunham
Zuul
====
.. hidetitle::
.. ansi:: images/zuul.ans
What Zuul does
==============
* "Speculative Future State"
* multiple repositories
* integrated deliverable
* gated commits
* testing like deployment
Our Use Case
============
OpenStack Is
============
* Federated
* Distributed
* Large
Federated
=========
* Hundreds of involved companies
* Hundreds of sub-projects
* "One" deliverable
* Union of priorities/use cases
* "Decisions are made by those who show up"
Impact of being Federated
=========================
* No company can appoint people to positions in the project
* The project cannot fire anyone
* Heavy reliance on consensus
* CI system doesn't assume anyone is "in charge"
Distributed
===========
* There is no office
* Contributor base is global
* Multitude of contributor backgrounds
Impact of being Distributed
===========================
* Tooling must empower all contributors, regardless of background,
skill level or cultural context
* Heavy preference for text-based communication
* Cannot assume US-centric needs or solutions
Sound like anybody's Day Job?
=============================
* Multiple silos
* Competing management chains
* Junior/Senior Devs, "Architects", Security Teams, Frontend/Backend
* Spread across locations
* "One" product
Large numbers of
================
* Contributors (\~2k in any given 6 month period)
* Changes
* Code Repositories (2139 as of this morning)
Not Bragging About Scale
========================
OpenStack Scale Comparison
==========================
* 2KJPH (2,000 jobs per hour)
* Build Nodes from 16 Regions of 5 Public and 4 Private OpenStack Clouds
* Rackspace, Internap, OVH, Vexxhost, CityCloud
* Linaro (ARM), Limestone, Packethost, Fortnebula
* 10,000 changes merged per month
OpenStack Scale Comparison
==========================
* 2KJPH (2,000 jobs per hour)
* Build Nodes from 16 Regions of 5 Public and 4 Private OpenStack Clouds
* Rackspace, Internap, OVH, Vexxhost, CityCloud
* Linaro (ARM), Limestone, Packethost, Fortnebula
* 10,000 changes merged per month
* By comparison, our friends at the amazing project Ansible received
13,000 changes and had merged 8,000 of them in its first 4 years.
Impact of scale
===============
* Empower teams to take care of themselves (distributed)
* Efficiency gained from shared solutions (centralized)
* Empower teams to do what they want (distributed)
* Enforce common standards or requirements (centralized)
* Zuul supports per-repo config, central config, and multiple tenants
One Zuul install is all you need for an entire Enterprise
=========================================================
One Zuul install is all you need for an entire Enterprise
=========================================================
* 7 "admins" run Zuul for OpenDev supporting 2500+ devs
* None are 100% full-time on Zuul
* Team could handle another 2500+ devs
Who Is Running Zuul?
====================
* Zuul is in production for OpenStack for 7 years (in OpenStack VMs)
Also running at:
* BMW (control plane in OpenShift)
* GoDaddy (control plane in Kubernetes)
* GoodMoney (control plane in EKS, adding GKE)
* Le Bon Coin
* Volvo
* Western Digital
* Easystack
* TungstenFabric
* Huawei OpenLab
* IBM
* Red Hat
* others ...
Developer Process In a Nutshell
===============================
* Code Review - nobody has direct commit/push access
* Gated Commits - nobody has submit permission
* Every change gated on Code Analysis, Unit Tests, Functional Tests
and End to End Integration Tests
* Run all tests (at least) twice:
* on patchset upload
* between change approval and merge
OpenStack Developer Workflow
============================
.. container:: handout
* Who has submitted a patch?
* Who wants to?
* (Who is here because the name of this talk is weird?)
::
Hack Review Test
========= ========== ==========
push approve
+-------------+ +-------------+
| | | |
+------+--+ +--v----+--+ +--v-------+
| | | | | |
| $EDITOR | | Gerrit | | Zuul |
| | | | | |
+------^--+ +--+----^--+ +--+-------+
| | | |
+-------------+ +-------------+
clone merge
Gerrit
======
.. hidetitle::
.. container:: handout
explain patch upload, zuul runs, test results displayed in gerrit
this is all the interface to zuul users need to see
switch to actual gertty screenshot
also show zuul status page
but zuul is doing a lot of work behind the scenes, and if you look
closer, this is what you see
.. ansi:: images/color-gertty.ans
Zuul in a nutshell
==================
* Listens for code events
* Prepares appropriate job config and git repo states
* Allocates nodes for test jobs
* Pushes git repo states to nodes
* Runs user-defined Ansible playbooks
* Collects/reports results
* Potentially merges change
All in Service of Gating
========================
No Tests / Manual Tests
=======================
* No test automation exists or ...
* Developer runs test suite before pushing code
* Prone to developer skipping tests for "trivial" changes
* Doesn't scale organizationally
Periodic Testing
================
* Developers push changes directly to shared branch
* CI system runs tests from time to time - report if things still work
* "Who broke the build?"
* Leads to hacks like NVIE model
Post-Merge Testing
==================
* Developers push changes directly to shared branch
* CI system is triggered by push - reports if push broke something
* Frequently batched / rolled up
* Easier to diagnose which change broke things
* Reactive - the bad changes are already in
Pre-Review Testing
==================
* Changes are pushed to code review (Gerrit Change, GitHub PR, etc)
* CI system is triggered by code review change creation
* Test results inform review decisions
* Proactive - testing code before it lands
* Reviewers can get bored waiting for tests
* Only tests code as written, not potential result of merging code
Gating
======
* Changes are pushed to code review
* Gating system is triggered by code review approval
* Gating system merges code IFF tests pass
* Proactive - testing code before it lands
* Future state resulting from merge of code is tested
* Reviewers can fire-and-forget safely
Mix and Match
=============
* Zuul supports all of those modes
* Zuul users frequently combine them
* Run pre-review (check) and gating (gate) on each change
* Post-merge/post-tag for release/publication automation
* Periodic for catching bitrot
Pipelines in Zuul
=================
* In other systems, a "pipeline" is per-job - triger, content, report
* In Zuul, a Pipeline describes a shared general workflow for changes
* Triggers
* Requirements
* Reporters
* Job *content* is separate from triggering/reporting
* Projects add Jobs to Pipelines
* Same Job can be used in multiple Pipelines - and by multiple Projects
Check Jobs
==========
* Run on patchset upload
* Verify patch as written
* Avoid wasting reviewer time on broken changes
check pipeline
==============
.. code:: yaml
- pipeline:
name: check
manager: independent
precedence: low
require:
gerrit:
open: True
current-patchset: True
trigger:
gerrit:
- event: patchset-created
- event: change-restored
- event: comment-added
comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck
- event: comment-added
require-approval:
- Verified: [-1, -2]
username: zuul
approval:
- Workflow: 1
success:
gerrit:
Verified: 1
failure:
gerrit:
Verified: -1
Gate Triggering in Gerrit
=========================
* Gate jobs run between Code Review Approval and Merging
* "Workflow" Label in Gerrit
* Approvers have ability to vote +1 in Workflow
* Nobody sees the Submit button
* Zuul runs Gate jobs on Workflow+1 - Merges on Success
gate pipeline
=============
.. code:: yaml
- pipeline:
name: gate
manager: dependent
post-review: True
require:
gerrit:
open: True
current-patchset: True
approval:
- Workflow: 1
trigger:
gerrit:
- event: comment-added
approval:
- Workflow: 1
start:
gerrit:
Verified: 0
success:
gerrit:
Verified: 2
submit: true
failure:
gerrit:
Verified: -2
Multi-repository integration
============================
* Multiple source repositories are needed for deliverable
* Future state to be tested is the future state of all involved repos
To test proposed future state
=============================
* Get tip of each project. Merge appropriate change(s). Test.
* Changes must be serialized, otherwise state under test is invalid.
* Integrated deliverable repos share serialized queue
Speculative Execution
=====================
* Correct parallel processing of serialized future states
* Create virtual serial queue of changes for each deliverable
* Assume each change will pass its tests
* Test successive changes with previous changes applied to starting state
Nearest Non-Failing Change
==========================
(aka 'The Jim Blair Algorithm')
* If a change fails, move it aside
* Cancel all test jobs behind it in the queue
* Reparent queue items on the nearest non-failing change
* Restart tests with new state
Zuul Simulation
===============
.. transition:: pan
.. container:: handout
* todo
.. ansi:: images/zsim-00.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-01.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-02.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-03.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-04.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-05.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-06.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-07.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-08.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-09.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-10.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-11.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-12.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-13.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-14.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-15.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-16.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-17.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-18.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-19.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-20.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-21.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-22.ans
Explicit Cross-Project Dependencies
===================================
* Developers can mark changes as being dependent
* Depends-On: footer - in commit or PR
* Zuul uses depends-on when constructing virtual serial queue
* Will not merge changes in gate before depends-on changes
* Works cross-repo AND cross-source
Sources
=======
* Gerrit
* Github
* Pagure
* Git
* Bitbucket (in-review) - https://review.opendev.org/#/c/657837
* Gitlab (in-development) - https://review.opendev.org/#/c/685990/
Cross Source
============
.. code:: yaml
trigger:
gerrit:
- event: patchset-created
- event: change-restored
- event: comment-added
comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck
github:
- event: pull_request
action:
- opened
- changed
- reopened
- event: pull_request
action: comment
comment: (?i)^\s*recheck\s*$
start:
github:
status: pending
comment: false
success:
gerrit:
Verified: 1
github:
status: 'success'
failure:
gerrit:
Verified: -1
github:
status: 'failure'
Cross Source
============
* Explicit Dependency between projects from different sources
* Change to Zuul that depends on change to Gerrit
* https://review.opendev.org/#/c/680778/
::
commit 737d61c116ff5f32770ef72e2dd82a031ab32591
Author: James E. Blair <jeblair@redhat.com>
Date: Mon Aug 19 14:58:20 2020 -0700
Add Support for Gerrit Checks Plugin
Depends-On: https://gerrit-review.googlesource.com/c/plugins/checks/+/232079
Change-Id: I8e5903f4429c5a1273a6120e0d09c57169e8f938
Lock Step Changes
=================
* Circular Dependencies are not supported on purpose
* Rolling upgrades across interdependent services
* HOWEVER - many valid use cases (go/rust/c++) - support expected
Configuring Zuul
================
* Zuul Manages Git Repos
* Zuul Jobs are configured in Git Repos
* Zuul Job Content is self-testing
Projects
========
* A "Project" is a git repo
Config Project vs. Untrusted Project
====================================
config project
* special project containing project admin content
* has access to features that are normally restricted
* job changes are not applied speculatively
untrusted project
* most projects
* some actions (like executing code on localhost) are blocked
* job changes are applied speculatively
Config Files vs. Directories
============================
* Zuul reads in-repo config from:
``.zuul.yaml``, ``zuul.yaml``, ``zuul.d`` or ``.zuul.d``
* For projects with substantial zuul config, like ``zuul-config``
``zuul.d`` directory is likely best.
* The directories are read run-parts style.
* Recommended practice is splitting by type of object
Live Configuration Changes
==========================
.. container:: handout
Zuul is a distributed system, with a distributed configuration.
Zuul's config is in git - but something has to tell it where the repos are
.. code:: yaml
- tenant:
name: openstack
source:
gerrit:
config-projects:
- opendev/project-config
untrusted-projects:
- zuul/zuul-jobs
- zuul/zuul
- zuul/nodepool
- openstack/openstacksdk
github:
untrusted-projects:
- include: []
projects:
- ansible/ansible
Zuul Startup
============
* Read config file (main.yaml)
Zuul Startup
============
* Read config file
* Ask mergers for branches of each repo
.. ansi:: images/startup1.ans
Zuul Startup
============
* Read config file
* Ask mergers for branches of each repo
* Ask mergers for .zuul.yaml for each branch
of each repo
.. ansi:: images/startup2.ans
When .zuul.yaml Changes
=======================
.. container:: progressive
* Zuul looks for changes to .zuul.yaml
* Asks mergers for updated content
* Splices into configuration used for that change
* Works with cross-repo dependencies
("This change depends on a change to the job definition")
Zuul Architecture
=================
Zuul is comprised of several services (mostly python3)
* zuul-scheduler
* zuul-executor
* zuul-merger
* zuul-web
* zuul-fingergw
* zuul-dashboard (javascript/react)
* zuul-proxy (c++)
* nodepool-launcher
* nodepool-builder
* RDBMS
* Gearman
* Zookeeper
* zuul-registry (coming soon)
Where Does Job Content Run?
===========================
Nodepool
========
* A separate service that works very closely with *Zuul*
* *Zuul* requires *Nodepool* but *Nodepool* can be used independently
* Creates and destroys zero or more node resources
* Resources can include VMs, Containers, COE contexts or Bare Metals
* Static driver for allocating pre-existing nodes to jobs
* Optionally periodically builds images and uploads to clouds
(Remember that 2,000 jobs per hour number?)
Nodepool Launcher
=================
Where build nodes should come from
* OpenStack
* Static
* Kubernetes Pod
* Kubernetes Namespace
* AWS
In work / coming soon:
* Azure
* GCE
Nodepool Builder
================
Optionally periodically build and upload new base images
* OpenStack
How do you use this thing?
==========================
.. transition:: tilt
.. hidetitle::
.. figlet:: Configuration
Human Roles
===========
* Deployer
* Project Admin
* End User
Deployer Config
===============
Zuul: Connection, Triggers, Reporters
Nodepool: Launcher and Builder Config
Connection Plugins
==================
Describes how Zuul connects to external systems
* Gerrit
* Github
* Pagure
* git
* mqtt
* smtp
* sql
Trigger Plugins
===============
Input to Zuul for use in causing jobs to run
* Gerrit
* Github
* Pagure
* git
* zuul
Reporter Plugins
================
Where Zuul should send information about jobs
* Gerrit
* Github
* Pagure
* mqtt
* smtp
* sql
Admin Role
==========
* What sources, triggers and reporters
* What projects Zuul manages
* What node labels are available
* *Everything* else is Job Content
Demo Installation using docker-compose
======================================
Remember this?
* Install docker, docker-compose, git-review
Debian/Ubuntu:
::
apt-get install docker-compose git git-review
Red Hat / SuSE
::
yum install docker-compose git git-review
* mkdir -p ~/src/opendev.org/zuul
* cd ~/src/opendev.org/zuul
* git clone https://opendev.org/zuul/zuul
* cd zuul
* cd doc/source/admin/examples
* docker-compose up
What's Running
==============
* Zookeeper
* Gerrit
* Nodepool Launcher
* Zuul Scheduler
* Zuul Web Server
* Zuul Executor
* Apache HTTPD
* A container to use as a 'static' build node
How they're connected
=====================
* End Users talk to Gerrit and Apache HTTPD
* Zuul Scheduler talks to Gerrit
* Nodepool Launcher, Zuul Scheduler, Zuul Web talk to Zookeeper
* Zuul Executor talks to Zuul Scheduler (using Gearman)
Initial provided config
=======================
* docker-compose has plumbed in basic config ``etc_zuul/zuul.conf``
and ``etc_zuul/main.yaml``
* Gerrit Connection named "gerrit"
* Zuul user for that connection
* Git connection named "opendev.org" for ``zuul-jobs`` standard library
Initial tenant
==============
* Zuul is (always) multi-tenant
* Example config contains a tenant called ``example-tenant``
* Three projects in the ``example-tenant`` tenant:
``zuul-config``, ``test1``, ``test2``
* Three projects are also in gerrit ready to use
zuul.conf
=========
::
[gearman]
server=scheduler
[gearman_server]
start=true
[zookeeper]
hosts=zk
[scheduler]
tenant_config=/etc/zuul/main.yaml
[web]
listen_address=0.0.0.0
[executor]
private_key_file=/var/ssh/nodepool
default_username=root
zuul.conf part 2
================
::
[connection "gerrit"]
name=gerrit
driver=gerrit
server=gerrit
sshkey=/var/ssh/zuul
user=zuul
password=secret
baseurl=http://gerrit:8080
auth_type=basic
[connection "zuul-ci.org"]
name=zuul-ci
driver=git
baseurl=https://opendev.org/
main.yaml
=========
::
- tenant:
name: example-tenant
source:
gerrit:
config-projects:
- zuul-config
untrusted-projects:
- test1
- test2
zuul-ci.org:
untrusted-projects:
- zuul-jobs:
include:
- job
Running Dashboard
=================
http://localhost:9000
Jobs
====
* Define node types needed from nodepool
* Define which ansible playbooks to run
* Jobs may be defined centrally or in the repo being tested
* Jobs have contextual variants that simplify configuration
* Jobs definitions support inheritance
Job Libraries
=============
* Zuul jobs are all defined in git repositories
* Designed to be directly shared across zuul installations
* Standard library: https://opendev.org/zuul/zuul-jobs
* Zuul installs should add ``zuul-jobs`` to their config
* As changes land in ``zuul-jobs`` - Zuul installs will get them automatically
Local Job Libraries
===================
* Jobs specific to local project, but meant to be shared
* https://opendev.org/openstack/openstack-zuul-jobs
* https://github.com/ansible/zuul-jobs
* https://opendev.org/openstack/devstack (here be dragons)
Simple Job
==========
.. code:: yaml
- job:
name: tox
pre-run: playbooks/setup-tox.yaml
run: playbooks/tox.yaml
post-run: playbooks/fetch-tox-output.yaml
What about job content?
=======================
* Written in Ansible
* Ansible is excellent at running one or more tasks in one or more places
* The answer to "how do I" is almost always "Ansible"
Playbooks
=========
* Jobs run playbooks
* Allocated nodes from nodepool are provided in Ansible Inventory
* Playbooks may be defined centrally or in the repo being tested
* Playbooks can use roles from current or other Zuul repos
* Playbooks are not allowed to execute content on 'localhost'
* *Note*: localhost restriction may go away soon
Simple Playbook Example
=======================
* playbooks/tox.yaml
.. code:: yaml
- hosts: ubuntu-bionic
tasks:
- name: Run tox
args:
chdir: "{{ zuul_work_dir }}"
shell: "tox -e{{ tox_envlist }}"
Job Inheritance
===============
All Zuuls have a base job
.. code:: yaml
- job:
name: base
parent: null
description: |
The base job for Zuul.
timeout: 1800
nodeset:
nodes:
- name: primary
label: ubuntu-bionic
pre-run: playbooks/base/pre.yaml
post-run:
- playbooks/base/post-ssh.yaml
- playbooks/base/post-logs.yaml
secrets:
- site_logs
Logging
=======
* Zuul doesn't know about logging - only about URLs to report
* Job Ansible content does *something* with logs
* Job content tells Zuul what URL(s) to report to user
* Allows for some really cool patterns
Live Web Preview
================
* For static websites, publish built content and report url
https://review.opendev.org/#/c/635716/
https://zuul.opendev.org/t/zuul/build/2ef4ceb232a2467ab56be726f31928ef
https://470ff2d7584ecc192d95-20fcf28638b4f38abf523a12606af55d.ssl.cf1.rackcdn.com/635716/12/check/zuul-build-dashboard/2ef4ceb/npm/html
Project Admin Role
==================
* Base job(s)
* Pre-playbooks for local environment
* Post-playbooks - where do logs and artifacts go?
* Shared trusted publication jobs
Simple Job Inheritance
======================
.. code:: yaml
- job:
name: tox-py36
parent: tox
vars:
tox_envlist: py36
Inheritance Works Like An Onion
===============================
* pre-run playbooks run in order of inheritance
* run playbook of job runs
* post-run playbooks run in reverse order of inheritance
* If pre-run playbooks fail, job is re-tried
* All post-run playbooks run - as far as pre-run playbooks got
Inheritance Example
===================
For tox-py36 job
* base pre-run playbooks/base/pre.yaml
* tox pre-run playbooks/setup-tox.yaml
* tox run playbooks/tox.yaml
* tox post-run playbooks/fetch-tox-output.yaml
* base post-run playbooks/base/post-ssh.yaml
* base post-run playbooks/base/post-logs.yaml
Simple Job Variant
==================
.. code:: yaml
- job:
name: tox-py36
branches: stable/mitaka
nodeset:
- name: ubuntu-trusty
label: ubuntu-trusty
Nodesets for Multi-node Jobs
============================
.. code:: yaml
- nodeset:
name: ceph-cluster
nodes:
- name: controller
label: centos-7
- name: compute1
label: fedora-28
- name: compute2
label: fedora-28
groups:
- name: ceph-osd
nodes:
- controller
- name: ceph-monitor
nodes:
- controller
- compute1
- compute2
Multi-node Job
==============
* nodesets are provided to Ansible for jobs in inventory
.. code:: yaml
- job:
name: ceph-multinode
nodeset: ceph-cluster
run: playbooks/install-ceph.yaml
Multi-node Ceph Job Content
===========================
.. code:: yaml
- hosts: all
roles:
- install-ceph
- hosts: ceph-osd
roles:
- start-ceph-osd
- hosts: ceph-monitor
roles:
- start-ceph-monitor
- hosts: all
roles:
- do-something-interesting
Projects
========
* Projects are git repositories
* Specify a set of jobs for each pipeline
* golang git repo naming as been adopted:
::
zuul@ubuntu-bionic:~$ find /home/zuul/src -mindepth 3 -maxdepth 3 -type d
/home/zuul/src/opendev.org/openstack/openstacksdk
/home/zuul/src/opendev.org/zuul/zuul
/home/zuul/src/github.com/ansible/ansible
/home/zuul/src/gerrit.googlesource.com/gerrit
Project Config
==============
* Specify a set of jobs for each pipeline
.. code:: yaml
- project:
check:
jobs:
- openstack-tox-py36
- openstack-tox-docs
gate:
jobs:
- openstack-tox-py36
- openstack-tox-docs
Project with Local Variant
==========================
.. code:: yaml
- project:
check:
jobs:
- openstack-tox-py27
- openstack-tox-py35
- openstack-tox-py36:
voting: false
- openstack-tox-docs
gate:
jobs:
- openstack-tox-py27
- openstack-tox-py35
- openstack-tox-docs
Project with More Local Variants
================================
.. code:: yaml
- project:
check:
jobs:
- openstack-tox-py27
- openstack-tox-py35
- openstack-tox-py36:
voting: false
- openstack-tox-docs:
files: '^docs/.*$'
Project with Many Local Variants
================================
.. code:: yaml
- project:
check:
jobs:
- openstack-tox-py27:
nodeset:
- name: centos-7
label: centos-7
- openstack-tox-py27:
branches: stable/newton
nodeset:
- name: ubuntu-trusty
label: ubuntu-trusty
- openstack-tox-py35
- openstack-tox-py36:
voting: false
- openstack-tox-docs:
files: '^docs/.*$'
Project With Central and Local Config
=====================================
.. code:: yaml
# In opendev.org/openstack/project-config:
- project:
name: openstack/nova
templates:
- openstack-tox-jobs
.. code:: yaml
# In opendev.org/openstack/nova/.zuul.yaml:
- project:
check:
- nova-placement-functional-devstack
Project with Job Dependencies
=============================
Full DAG of Job Dependencies
.. code:: yaml
- project:
release:
jobs:
- build-artifacts
- upload-tarball:
dependencies: build-artifacts
- upload-pypi:
dependencies: build-artifacts
- notify-mirror:
dependencies:
- upload-tarball
- upload-pypi
Speculative Container Images
============================
* Build container images based on non-published images from other changes
* Test built container images in gate jobs
* Promote exact image from gate after merge
* Soft Dependencies - use artifact from another job only if it was built
* https://zuul-ci.org/docs/zuul-jobs/docker-image.html
Playbooks
=========
* Jobs run playbooks
* Playbooks may be defined centrally or in the repo being tested
* Playbooks can use roles from current or other Zuul repos or Galaxy
* Playbooks are not allowed to execute content on 'localhost'
Test Like Production
====================
If you use Ansible for deployment, your test and deployment processes
and playbooks are the same
What if you don't use Ansible?
==============================
OpenStack Infra Control Plane uses Puppet (for now)
===================================================
.. code:: yaml
OpenDev Control Plane uses some Puppet (for now)
================================================
.. code:: yaml
- name: Install puppet
shell: ./install_puppet.sh
args:
chdir: "{{ ansible_user_dir }}/src/opendev.org/opendev/system-config"
become: yes
- name: Copy manifest
copy:
src: manifest.pp
dest: "{{ ansible_user_dir }}/manifest.pp"
- name: Run puppet
puppet:
manifest: "{{ ansible_user_dir }}/manifest.pp"
become: yes
Secrets
=======
* Inspired by Kubernetes Secrets API
* Projects can add named encrypted secrets to their .zuul.yaml file
* Jobs can request to use secrets by name
* Jobs using secrets are not reconfigured speculatively
* Secrets can only be used by the same project they are defined in
* Public key per project:
``{{ zuul_url }}/{{ tenant }}/{{ project }}.pub``
::
GET https://zuul.opendev.org/api/tenant/openstack/key/openstack/openstacksdk.pub
Secret Example (note, no admins had to enable this)
===================================================
.. code:: yaml
# In opendev.org/openstack/loci/.zuul.yaml:
- secret:
name: loci_docker_login
data:
user: loci-username
password: !encrypted/pkcs1-oaep
- gUEX4eY3JAk/Xt7Evmf/hF7xr6HpNRXTibZjrKTbmI4QYHlzEBrBbHey27Pt/eYvKKeKw
hk8MDQ4rNX7ZK1v+CKTilUfOf4AkKYbe6JFDd4z+zIZ2PAA7ZedO5FY/OnqrG7nhLvQHE
5nQrYwmxRp4O8eU5qG1dSrM9X+bzri8UnsI7URjqmEsIvlUqtybQKB9qQXT4d6mOeaKGE
5h6Ydkb9Zdi4Qh+GpCGDYwHZKu1mBgVK5M1G6NFMy1DYz+4NJNkTRe9J+0TmWhQ/KZSqo
4ck0x7Tb0Nr7hQzV8SxlwkaCTLDzvbiqmsJPLmzXY2jry6QsaRCpthS01vnj47itoZ/7p
taH9CoJ0Gl7AkaxsrDSVjWSjatTQpsy1ub2fuzWHH4ASJFCiu83Lb2xwYts++r8ZSn+mA
hbEs0GzPI6dIWg0u7aUsRWMOB4A+6t2IOJibVYwmwkG8TjHRXxVCLH5sY+i3MR+NicR9T
IZFdY/AyH6vt5uHLQDU35+5n91pUG3F2lyiY5aeMOvBL05p27GTMuixR5ZoHcvSoHHtCq
7Wnk21iHqmv/UnEzqUfXZOque9YP386RBWkshrHd0x3OHUfBK/WrpivxvIGBzGwMr2qAj
/AhJsfDXKBBbhGOGk1u5oBLjeC4SRnAcIVh1+RWzR4/cAhOuy2EcbzxaGb6VTM=
Secret Example
==============
.. code:: yaml
# In opendev.org/openstack/loci/.zuul.yaml:
- job:
name: publish-loci-cinder
parent: loci-cinder
post-run: playbooks/push
secrets:
- loci_docker_login
# In opendev.org/openstack/loci/playbooks/push.yaml:
- hosts: all
tasks:
- include_vars: vars.yaml
- name: Push project to DockerHub
block:
- command: docker login -u {{ loci_docker_login.user }} -p {{ loci_docker_login.password }}
no_log: True
- command: docker push openstackloci/{{ project }}:{{ branch }}-{{ item.name }}
with_items: "{{ distros }}"
Back to our Install
===================
Zuul Containers
===============
* Published on every commit
* Application/Process containers
* Built using Speculative Containers
* Config / Data should be bind-mounted in
Container Philosophy
====================
* As minimal as possible
* OS inside of container does not matter
zuul/zuul-executor
==================
* In k8s, zuul-executor must be run in privileged pod
* Uses bubblewrap for unprivileged sanboxing
* Restriction may be lifted in the future
Release Management
==================
* Zuul is a CI system
* C stands for "Continuous"
* It is run Continuously Delivered and Deployed upstream
* Releases are tagged from code run upstream
* There is no intent to have a 'stable' release
* 'stable' is a synonym for "old and buggy"
* **NOTE** container images are not currently tagged
zuul/zuul-scheduler
===================
* SPOF
* We're working on it
* Recommend running scheduler from tags
Let's Do Something with Our Zuul
================================
Gerrit Account
==============
* Need a user account to interact with Gerrit
* Gerrit is configured in dev mode - no passwords required
* Visit http://localhost:8080
* Click "New Account"
* Skip entering anything, click "settings"
* Enter username in Username field (match your local laptop user)
* Enter Full Name
* Click "Save Changes"
More Gerrit Account
===================
* Scroll down to Email Addresses
* Enter email address in "New email address" field
* Click "Send Verification"
* Gerrit is in Developer Mode, will not send email
* Scroll down to "SSH Keys"
* Copy ``~/.ssh/id_rsa.pub`` contents into "New SSH Key" field
* Click "Add New SSH Key"
* Reload the page in your browser
Config Repo
===========
* ``zuul-config`` is a trusted ``config-repo``
* Security and functionality of system depend on this repo
* Limit its contents to minimum required
Setting up Gating
=================
* We want to have changes to ``zuul-config`` be gated
* We need to define pipelines: ``check`` and ``gate``
* Need to attach ``zuul-config`` to them
* Start with builtin ``noop`` job (always return success)
* Use regex to attach all projects to ``check`` and ``gate``
Pipeline Definitions
====================
* Zuul has no built-in workflow definitions, let's add ``check`` and ``gate``
check pipeline
==============
::
- pipeline:
name: check
description: |
Newly uploaded patchsets enter this pipeline to receive an
initial +/-1 Verified vote.
manager: independent
require:
gerrit:
open: True
current-patchset: True
trigger:
gerrit:
- event: patchset-created
- event: change-restored
comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck
success:
gerrit:
Verified: 1
mysql:
failure:
gerrit:
Verified: -1
mysql:
gate pipeline
=============
::
- pipeline:
name: gate
description: |
Changes that have been approved are enqueued in order in this
pipeline, and if they pass tests, will be merged.
manager: dependent
post-review: True
require:
gerrit:
open: True
current-patchset: True
approval:
- Workflow: 1
trigger:
gerrit:
- event: comment-added
approval:
- Workflow: 1
start:
gerrit:
Verified: 0
success:
gerrit:
Verified: 2
submit: true
mysql:
failure:
gerrit:
Verified: -2
mysql:
Add the pipeline definitions
============================
.. code-block:: bash
cd ~/src
git clone http://localhost:8080/zuul-config
cd zuul-config
mkdir zuul.d
cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/zuul-config/zuul.d/pipelines.yaml zuul.d
Shared Project Pipeline Definition
==================================
In ``examples/zuul-config/zuul.d/projects.yaml``
.. code-block:: yaml
- project:
name: ^.*$
check:
jobs: []
gate:
jobs: []
- project:
name: zuul-config
check:
jobs:
- noop
gate:
jobs:
- noop
Attach the projects to the pipelines
====================================
.. code-block:: bash
cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/zuul-config/zuul.d/projects.yaml zuul.d
Commit the changes and push up for review
=========================================
.. code-block:: bash
git add zuul.d
git commit
git review
Force merging bootstrap config
==============================
* Zuul is running with no config, so it won't do anything
* For this change (and this change only) we will bypass gating
Reviewing normally
==================
* visit http://localhost:8080/#/c/zuul-config/+/1001/
* click reply
* vote +2 Code Review +1 Approved
Verified +2 is Missing
======================
Verified +2 is what we have zuul configured to do.
::
success:
gerrit:
Verified: 2
submit: true
Bypassing Gating
================
* visit http://localhost:8080/
* Click avatar in Upper Right
* Click Sign Out
* Click Sign In
* Click 'admin'
* Click on change you uploaded
* Click Reply...
* Vote +2 Verified (normal users do not see this)
* Click submit (normal users do not see this)
* Click avatar in Upper Right
* Click Sign Out
* Click Sign In
* click your username
Dashboard
=========
http://localhost:9000/t/example-tenant/status
Add Job to a repo
=================
Clone the test1 repo
====================
::
cd ~/src
git clone http://localhodsts:8080/test1
cd test1
Simple debug playbook
=====================
.. code:: yaml
- hosts: all
tasks:
- debug:
msg: Hello world!
Add the playbook to our repo
============================
::
mkdir playbooks
cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/test1/playbooks/testjob.yaml playbooks
git add playbooks/testjob.yaml
Simple Zuul Job
===============
::
- job:
name: testjob
run: playbooks/testjob.yaml
- project:
check:
jobs:
- testjob
gate:
jobs:
- testjob
Add it to the .zuul.yaml
========================
This is a normal repo, let's use .zuul.yaml
::
cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/test1/zuul.yaml .zuul.yaml
git add .zuul.yaml
Submit the change for review
============================
.. code-block:: shell
git add .zuul.yaml playbooks
git commit -m "Add test Zuul job"
git review
Then check http://localhost:8080/dashboard/self
Where are the logs?
===================
Base Job
========
* Every Zuul installation must define a ``base`` job
* Push git repos to build node
* Publish logs/artifacts
* Any local specific setup
* Goes in config repo - because it impacts EVERY job
Minimal Base Job
================
.. code:: yaml
- job:
name: base
parent: null
nodeset:
nodes:
- name: ubuntu-bionic
label: ubuntu-bionic
Minimal Base Job
================
::
cd ~/src/zuul-config
cp ~/src/examples/zuul-config/zuul.d/jobs.yaml zuul.d
Base pre Playbook
=================
* Using roles from zuul-jobs repo
* Creates a per-build ssh key
* Copies git repos to test node
.. code:: yaml
- hosts: all
roles:
- add-build-sshkey
- prepare-workspace
Base pre Playbook
=================
::
mkdir -p playbooks/base
cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/zuul-config/playbooks/base/pre.yaml playbooks/base
Base post-ssh playbook
======================
* Removes the per-build ssh key from the pre playbook
.. code:: yaml
- hosts: all
roles:
- remove-build-sshkey
Base post-ssh playbook
======================
::
cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/zuul-config/playbooks/base/post-ssh playbooks/base
Base post-logs playbook
=======================
* Quickstart is running Apache server for log content
* Volume mounted at /srv/static/logs (default for upload-logs)
* Tells Zuul where the logs can be found
.. code:: yaml
- hosts: localhost
gather_facts: False
roles:
- role: upload-logs
zuul_log_url: "http://localhost:8000"
Base post-ssh playbook
======================
::
cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/zuul-config/playbooks/base/post-logs playbooks/base
Update Base Job
===============
.. code:: yaml
- job:
name: base
parent: null
description: |
The recommended base job.
All jobs ultimately inherit from this. It runs a pre-playbook
which copies all of the job's prepared git repos on to all of
the nodes in the nodeset.
It also sets a default timeout value (which may be overidden).
pre-run: playbooks/base/pre.yaml
post-run:
- playbooks/base/post-ssh.yaml
- playbooks/base/post-logs.yaml
roles:
- zuul: zuul/zuul-jobs
timeout: 1800
nodeset:
nodes:
- name: ubuntu-bionic
label: ubuntu-bionic
Update Base Job
===============
::
cp ~/src/examples/zuul-config/zuul.d/jobs2.yaml zuul.d
Commit and Push for Review
==========================
::
git add playbooks zuul.d/jobs.yaml
git commit -m "Update Zuul base job"
git review
Land Change
===========
* Visit http://localhost:8080/dashboard/self
* Should be a Verified +1 vote from Zuul
* Click Reply
* Vote: Code-Review +2 and Workflow +1
* Click Send
* Wait a bit, then reload, change should be merged
Recheck the test1 change
========================
* Visit http://localhost:8080/dashboard/self
* Click Reply
" Comment "recheck"
* Click Send
Job Now Running with Updated Base Job
=====================================
* Should get a link to log location
* Check out Zuul Dashboard
Zuul tests syntax automatically
===============================
* Edit zuul/jobs.yaml
* Change ``parent: null`` to ``parent: broken``
* git commit ; git review
* Check out the review in gerrit ... there should be errors!
Job configs are shared
======================
::
cd ~/src
git clone http://localhost:8080/test2
cd test2
Add testjob to test2 repo
=========================
::
cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/test1/zuul.yaml .zuul.yaml
git add .zuul.yaml
git commit -a -m"Added testjob to test2 repo"
git review
Where's the job?
================
* Check http://localhost:8080/dashboard/self
* Look at error from zuul
* This is a job for cross-project depends!
Add Depends-On footer
=====================
::
git commit -a --amend
* Add a blank line then
::
Depends-On: http://localhost:8080/c/test1/+/1
* Then resubmit
::
git review
Let's Add a Secret
==================
* Edit .zuul.yaml - name the secret and the field
::
cd ~/src/test1
echo "my-secret" | ~/src/opendev.org/zuul/zuul/tools/encrypt_secret.py --tenant example-tenant http://localhost:9000 test1 >> .zuul.yaml
Secret Name
===========
.. code-block:: yaml
- secret:
name: amazing_secret
data:
name: !encrypted/pkcs1-oaep
Add secret to job
=================
.. code-block:: yaml
- job:
name: testjob
run: playbooks/testjob.yaml
secrets:
- amazing_secret
Print the secret in the job
===========================
* *NOTE* This is dumb
* Edit playbooks/testjob.yaml
.. code-block:: yaml
- hosts: all
tasks:
- debug:
msg: "Hello {{ amazing_secret.name }}!"
Commit and push for review
==========================
::
git commit -a -m"Added a secret"
git review
Change test2 to depend on the secret change
===========================================
::
git commit -a --amend
* Change the Depends-On line
::
Depends-On: http://localhost:8080/c/test1/+/2
* Then resubmit
::
git review
Let's go look at the outputs!
=============================
Add a job with run-test-command
===============================
* What if you don't want a playbook?
* Role 'run-test-command' in zuul-jobs
.. code-block:: yaml
- job:
parent: run-test-command
name: simple-test-job
vars:
test_command: exit 0
Add to the project pipeline config
==================================
.. code-block:: yaml
- project:
check:
jobs:
- testjob
- simple-test-job
gate:
jobs:
- testjob
- simple-test-job
Commit and Push Up
==================
::
git commit -a -m"Added a simple test job"
git review
What's wrong?
=============
Add required-projects
=====================
* Edit .zuul.yaml
.. code-block:: yaml
- job:
parent: run-test-command
name: simple-test-job
vars:
test_command: "find {{ ansible_user_dir }}"
required-projects:
- test1
- test2
Important Links
===============
* https://opendev.org/zuul
* https://zuul-ci.org/docs/zuul
* https://zuul-ci.org/docs/zuul-jobs/
* https://docs.openstack.org/infra/openstack-zuul-jobs/
* freenode:#zuul
* https://opendev.org/inaugust/inaugust.com/src/branch/master/src/zuulv3/tutorial.rst
View Git Blame Copy Permalink