inaugust
/
inaugust.com
Code Issues Proposed changes
inaugust.com/src/zuulv3/tutorial.rst

48 KiB
Raw Blame History

. display in 68x24 .. display in 88x24

dissolve

Test Slide

images/testslide.ans

Preshow

images/cursor.ans images/cursor2.ans

Zuul

images/title.ans

Important Links

Overview

  • Discussion of concepts
  • Installation of software
  • Configurting Zuul
  • Writing jobs

Please Ask Questions

Installation of Software

Ways to Install Zuul

Today we'll be using Containers with docker-compose

Getting Started

While we talk about other things ...

  • Install docker, docker-compose, git-review

Debian/Ubuntu:

apt-get install docker-compose git git-review

Red Hat / SuSE

yum install docker-compose git git-review
  • mkdir -p ~/src/opendev.org/zuul
  • cd ~/src/opendev.org/zuul
  • git clone https://opendev.org/zuul/zuul
  • cd zuul
  • cd doc/source/admin/examples
  • docker-compose up

Output in docker-compose window

  • All services running with debug logging to stdout
  • Tons of information will have been output - including some errors
  • Zuul connects to Gerrit before it's fully configured
  • As it becomes configured, Zuul notices and becomes happy
  • Once happy, it should stablize and become idle

We'll come back to this

It's going to do a bunch of network - and this is a conference.

Red Hat

images/redhat.ans

OpenStack

images/openstack.ans

OpenDev

"most insane CI infrastructure I've ever been a part of"

  -- Alex Gaynor

"OpenStack Infra are like the SpaceX of CI"

  -- Emily Dunham

Zuul

images/zuul.ans

What Zuul does

  • "Speculative Future State"
  • multiple repositories
  • integrated deliverable
  • gated commits
  • testing like deployment

Our Use Case

OpenStack Is

  • Federated
  • Distributed
  • Large

Federated

  • Hundreds of involved companies
  • Hundreds of sub-projects
  • "One" deliverable
  • Union of priorities/use cases
  • "Decisions are made by those who show up"

Impact of being Federated

  • No company can appoint people to positions in the project
  • The project cannot fire anyone
  • Heavy reliance on consensus
  • CI system doesn't assume anyone is "in charge"

Distributed

  • There is no office
  • Contributor base is global
  • Multitude of contributor backgrounds

Impact of being Distributed

  • Tooling must empower all contributors, regardless of background, skill level or cultural context
  • Heavy preference for text-based communication
  • Cannot assume US-centric needs or solutions

Sound like anybody's Day Job?

  • Multiple silos
  • Competing management chains
  • Junior/Senior Devs, "Architects", Security Teams, Frontend/Backend
  • Spread across locations
  • "One" product

Large numbers of

  • Contributors (~2k in any given 6 month period)
  • Changes
  • Code Repositories (2139 as of this morning)

Not Bragging About Scale

OpenStack Scale Comparison

  • 2KJPH (2,000 jobs per hour)
  • Build Nodes from 16 Regions of 5 Public and 4 Private OpenStack Clouds
  • Rackspace, Internap, OVH, Vexxhost, CityCloud
  • Linaro (ARM), Limestone, Packethost, Fortnebula
  • 10,000 changes merged per month

OpenStack Scale Comparison

  • 2KJPH (2,000 jobs per hour)
  • Build Nodes from 16 Regions of 5 Public and 4 Private OpenStack Clouds
  • Rackspace, Internap, OVH, Vexxhost, CityCloud
  • Linaro (ARM), Limestone, Packethost, Fortnebula
  • 10,000 changes merged per month
    • By comparison, our friends at the amazing project Ansible received 13,000 changes and had merged 8,000 of them in its first 4 years.

Impact of scale

  • Empower teams to take care of themselves (distributed)
  • Efficiency gained from shared solutions (centralized)
  • Empower teams to do what they want (distributed)
  • Enforce common standards or requirements (centralized)
  • Zuul supports per-repo config, central config, and multiple tenants

One Zuul install is all you need for an entire Enterprise

One Zuul install is all you need for an entire Enterprise

  • 7 "admins" run Zuul for OpenDev supporting 2500+ devs
  • None are 100% full-time on Zuul
  • Team could handle another 2500+ devs

Who Is Running Zuul?

  • Zuul is in production for OpenStack for 7 years (in OpenStack VMs)

Also running at:

  • BMW (control plane in OpenShift)
  • GoDaddy (control plane in Kubernetes)
  • GoodMoney (control plane in EKS, adding GKE)
  • Le Bon Coin
  • Volvo
  • Western Digital
  • Easystack
  • TungstenFabric
  • Huawei OpenLab
  • IBM
  • Red Hat
  • others ...

Developer Process In a Nutshell

  • Code Review - nobody has direct commit/push access
  • Gated Commits - nobody has submit permission
  • Every change gated on Code Analysis, Unit Tests, Functional Tests and End to End Integration Tests
  • Run all tests (at least) twice:
    • on patchset upload
    • between change approval and merge

OpenStack Developer Workflow

  • Who has submitted a patch?
  • Who wants to?
  • (Who is here because the name of this talk is weird?)
Hack             Review              Test
=========         ==========         ==========

        push              approve
   +-------------+    +-------------+
   |             |    |             |
+------+--+       +--v----+--+       +--v-------+
|         |       |          |       |          |
| $EDITOR |       |  Gerrit  |       |   Zuul   |
|         |       |          |       |          |
+------^--+       +--+----^--+       +--+-------+
   |             |    |             |
   +-------------+    +-------------+
        clone              merge

Gerrit

explain patch upload, zuul runs, test results displayed in gerrit this is all the interface to zuul users need to see

switch to actual gertty screenshot

also show zuul status page

but zuul is doing a lot of work behind the scenes, and if you look closer, this is what you see

images/color-gertty.ans

Zuul in a nutshell

  • Listens for code events
  • Prepares appropriate job config and git repo states
  • Allocates nodes for test jobs
  • Pushes git repo states to nodes
  • Runs user-defined Ansible playbooks
  • Collects/reports results
  • Potentially merges change

All in Service of Gating

No Tests / Manual Tests

  • No test automation exists or ...
  • Developer runs test suite before pushing code
  • Prone to developer skipping tests for "trivial" changes
  • Doesn't scale organizationally

Periodic Testing

  • Developers push changes directly to shared branch
  • CI system runs tests from time to time - report if things still work
  • "Who broke the build?"
  • Leads to hacks like NVIE model

Post-Merge Testing

  • Developers push changes directly to shared branch
  • CI system is triggered by push - reports if push broke something
  • Frequently batched / rolled up
  • Easier to diagnose which change broke things
  • Reactive - the bad changes are already in

Pre-Review Testing

  • Changes are pushed to code review (Gerrit Change, GitHub PR, etc)
  • CI system is triggered by code review change creation
  • Test results inform review decisions
  • Proactive - testing code before it lands
  • Reviewers can get bored waiting for tests
  • Only tests code as written, not potential result of merging code

Gating

  • Changes are pushed to code review
  • Gating system is triggered by code review approval
  • Gating system merges code IFF tests pass
  • Proactive - testing code before it lands
  • Future state resulting from merge of code is tested
  • Reviewers can fire-and-forget safely

Mix and Match

  • Zuul supports all of those modes
  • Zuul users frequently combine them
  • Run pre-review (check) and gating (gate) on each change
  • Post-merge/post-tag for release/publication automation
  • Periodic for catching bitrot

Pipelines in Zuul

  • In other systems, a "pipeline" is per-job - triger, content, report
  • In Zuul, a Pipeline describes a shared general workflow for changes
  • Triggers
  • Requirements
  • Reporters
  • Job content is separate from triggering/reporting
  • Projects add Jobs to Pipelines
  • Same Job can be used in multiple Pipelines - and by multiple Projects

Check Jobs

  • Run on patchset upload
  • Verify patch as written
  • Avoid wasting reviewer time on broken changes

check pipeline

- pipeline:
    name: check
    manager: independent
    precedence: low
    require:
      gerrit:
        open: True
        current-patchset: True
    trigger:
      gerrit:
        - event: patchset-created
        - event: change-restored
        - event: comment-added
          comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck
        - event: comment-added
          require-approval:
            - Verified: [-1, -2]
              username: zuul
          approval:
            - Workflow: 1
    success:
      gerrit:
        Verified: 1
    failure:
      gerrit:
        Verified: -1

Gate Triggering in Gerrit

  • Gate jobs run between Code Review Approval and Merging
  • "Workflow" Label in Gerrit
  • Approvers have ability to vote +1 in Workflow
  • Nobody sees the Submit button
  • Zuul runs Gate jobs on Workflow+1 - Merges on Success

gate pipeline

- pipeline:
    name: gate
    manager: dependent
    post-review: True
    require:
      gerrit:
        open: True
        current-patchset: True
        approval:
          - Workflow: 1
    trigger:
      gerrit:
        - event: comment-added
          approval:
            - Workflow: 1
    start:
      gerrit:
        Verified: 0
    success:
      gerrit:
        Verified: 2
        submit: true
    failure:
      gerrit:
        Verified: -2

Multi-repository integration

  • Multiple source repositories are needed for deliverable
  • Future state to be tested is the future state of all involved repos

To test proposed future state

  • Get tip of each project. Merge appropriate change(s). Test.
  • Changes must be serialized, otherwise state under test is invalid.
  • Integrated deliverable repos share serialized queue

Speculative Execution

  • Correct parallel processing of serialized future states
  • Create virtual serial queue of changes for each deliverable
  • Assume each change will pass its tests
  • Test successive changes with previous changes applied to starting state

Nearest Non-Failing Change

(aka 'The Jim Blair Algorithm')

  • If a change fails, move it aside
  • Cancel all test jobs behind it in the queue
  • Reparent queue items on the nearest non-failing change
  • Restart tests with new state

Zuul Simulation

pan

  • todo

images/zsim-00.ans

Zuul Simulation

cut

  • todo

images/zsim-01.ans

Zuul Simulation

cut

  • todo

images/zsim-02.ans

Zuul Simulation

cut

  • todo

images/zsim-03.ans

Zuul Simulation

cut

  • todo

images/zsim-04.ans

Zuul Simulation

cut

  • todo

images/zsim-05.ans

Zuul Simulation

cut

  • todo

images/zsim-06.ans

Zuul Simulation

cut

  • todo

images/zsim-07.ans

Zuul Simulation

cut

  • todo

images/zsim-08.ans

Zuul Simulation

cut

  • todo

images/zsim-09.ans

Zuul Simulation

cut

  • todo

images/zsim-10.ans

Zuul Simulation

cut

  • todo

images/zsim-11.ans

Zuul Simulation

cut

  • todo

images/zsim-12.ans

Zuul Simulation

cut

  • todo

images/zsim-13.ans

Zuul Simulation

cut

  • todo

images/zsim-14.ans

Zuul Simulation

cut

  • todo

images/zsim-15.ans

Zuul Simulation

cut

  • todo

images/zsim-16.ans

Zuul Simulation

cut

  • todo

images/zsim-17.ans

Zuul Simulation

cut

  • todo

images/zsim-18.ans

Zuul Simulation

cut

  • todo

images/zsim-19.ans

Zuul Simulation

cut

  • todo

images/zsim-20.ans

Zuul Simulation

cut

  • todo

images/zsim-21.ans

Zuul Simulation

cut

  • todo

images/zsim-22.ans

Explicit Cross-Project Dependencies

  • Developers can mark changes as being dependent
  • Depends-On: footer - in commit or PR
  • Zuul uses depends-on when constructing virtual serial queue
  • Will not merge changes in gate before depends-on changes
  • Works cross-repo AND cross-source

Sources

Cross Source

trigger:
  gerrit:
    - event: patchset-created
    - event: change-restored
    - event: comment-added
      comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck
  github:
    - event: pull_request
      action:
        - opened
        - changed
        - reopened
    - event: pull_request
      action: comment
      comment: (?i)^\s*recheck\s*$
start:
  github:
    status: pending
    comment: false
success:
  gerrit:
    Verified: 1
  github:
    status: 'success'
failure:
  gerrit:
    Verified: -1
  github:
    status: 'failure'

Cross Source

commit 737d61c116ff5f32770ef72e2dd82a031ab32591
Author: James E. Blair <jeblair@redhat.com>
Date:   Mon Aug 19 14:58:20 2020 -0700

Add Support for Gerrit Checks Plugin

Depends-On: https://gerrit-review.googlesource.com/c/plugins/checks/+/232079
Change-Id: I8e5903f4429c5a1273a6120e0d09c57169e8f938

Lock Step Changes

  • Circular Dependencies are not supported on purpose
  • Rolling upgrades across interdependent services
  • HOWEVER - many valid use cases (go/rust/c++) - support expected

Configuring Zuul

  • Zuul Manages Git Repos
  • Zuul Jobs are configured in Git Repos
  • Zuul Job Content is self-testing

Projects

  • A "Project" is a git repo

Config Project vs. Untrusted Project

config project

  • special project containing project admin content
  • has access to features that are normally restricted
  • job changes are not applied speculatively

untrusted project

  • most projects
  • some actions (like executing code on localhost) are blocked
  • job changes are applied speculatively

Config Files vs. Directories

  • Zuul reads in-repo config from: .zuul.yaml, zuul.yaml, zuul.d or .zuul.d
  • For projects with substantial zuul config, like zuul-config zuul.d directory is likely best.
  • The directories are read run-parts style.
  • Recommended practice is splitting by type of object

Live Configuration Changes

Zuul is a distributed system, with a distributed configuration.

Zuul's config is in git - but something has to tell it where the repos are

- tenant:
    name: openstack
    source:
      gerrit:
        config-projects:
          - opendev/project-config
        untrusted-projects:
          - zuul/zuul-jobs
          - zuul/zuul
          - zuul/nodepool
          - openstack/openstacksdk
      github:
        untrusted-projects:
          - include: []
            projects:
              - ansible/ansible

Zuul Startup

  • Read config file (main.yaml)

Zuul Startup

  • Read config file
  • Ask mergers for branches of each repo

images/startup1.ans

Zuul Startup

  • Read config file

  • Ask mergers for branches of each repo

  • Ask mergers for .zuul.yaml for each branch

    of each repo

images/startup2.ans

When .zuul.yaml Changes

  • Zuul looks for changes to .zuul.yaml

  • Asks mergers for updated content

  • Splices into configuration used for that change

  • Works with cross-repo dependencies

    ("This change depends on a change to the job definition")

Zuul Architecture

Zuul is comprised of several services (mostly python3)

  • zuul-scheduler
  • zuul-executor
  • zuul-merger
  • zuul-web
  • zuul-fingergw
  • zuul-dashboard (javascript/react)
  • zuul-proxy (c++)
  • nodepool-launcher
  • nodepool-builder
  • RDBMS
  • Gearman
  • Zookeeper
  • zuul-registry (coming soon)

Where Does Job Content Run?

Nodepool

  • A separate service that works very closely with Zuul
  • Zuul requires Nodepool but Nodepool can be used independently
  • Creates and destroys zero or more node resources
  • Resources can include VMs, Containers, COE contexts or Bare Metals
  • Static driver for allocating pre-existing nodes to jobs
  • Optionally periodically builds images and uploads to clouds

(Remember that 2,000 jobs per hour number?)

Nodepool Launcher

Where build nodes should come from

  • OpenStack
  • Static
  • Kubernetes Pod
  • Kubernetes Namespace
  • AWS

In work / coming soon:

  • Azure
  • GCE

Nodepool Builder

Optionally periodically build and upload new base images

  • OpenStack

How do you use this thing?

tilt

Configuration

Human Roles

  • Deployer
  • Project Admin
  • End User

Deployer Config

Zuul: Connection, Triggers, Reporters Nodepool: Launcher and Builder Config

Connection Plugins

Describes how Zuul connects to external systems

  • Gerrit
  • Github
  • Pagure
  • git
  • mqtt
  • smtp
  • sql

Trigger Plugins

Input to Zuul for use in causing jobs to run

  • Gerrit
  • Github
  • Pagure
  • git
  • zuul

Reporter Plugins

Where Zuul should send information about jobs

  • Gerrit
  • Github
  • Pagure
  • mqtt
  • smtp
  • sql

Admin Role

  • What sources, triggers and reporters
  • What projects Zuul manages
  • What node labels are available
  • Everything else is Job Content

Demo Installation using docker-compose

Remember this?

  • Install docker, docker-compose, git-review

Debian/Ubuntu:

apt-get install docker-compose git git-review

Red Hat / SuSE

yum install docker-compose git git-review
  • mkdir -p ~/src/opendev.org/zuul
  • cd ~/src/opendev.org/zuul
  • git clone https://opendev.org/zuul/zuul
  • cd zuul
  • cd doc/source/admin/examples
  • docker-compose up

What's Running

  • Zookeeper
  • Gerrit
  • Nodepool Launcher
  • Zuul Scheduler
  • Zuul Web Server
  • Zuul Executor
  • Apache HTTPD
  • A container to use as a 'static' build node

How they're connected

  • End Users talk to Gerrit and Apache HTTPD
  • Zuul Scheduler talks to Gerrit
  • Nodepool Launcher, Zuul Scheduler, Zuul Web talk to Zookeeper
  • Zuul Executor talks to Zuul Scheduler (using Gearman)

Initial provided config

  • docker-compose has plumbed in basic config etc_zuul/zuul.conf and etc_zuul/main.yaml
  • Gerrit Connection named "gerrit"
  • Zuul user for that connection
  • Git connection named "opendev.org" for zuul-jobs standard library

Initial tenant

  • Zuul is (always) multi-tenant
  • Example config contains a tenant called example-tenant
  • Three projects in the example-tenant tenant: zuul-config, test1, test2
  • Three projects are also in gerrit ready to use

zuul.conf

[gearman]
server=scheduler

[gearman_server]
start=true

[zookeeper]
hosts=zk

[scheduler]
tenant_config=/etc/zuul/main.yaml

[web]
listen_address=0.0.0.0

[executor]
private_key_file=/var/ssh/nodepool
default_username=root

zuul.conf part 2

[connection "gerrit"]
name=gerrit
driver=gerrit
server=gerrit
sshkey=/var/ssh/zuul
user=zuul
password=secret
baseurl=http://gerrit:8080
auth_type=basic

[connection "zuul-ci.org"]
name=zuul-ci
driver=git
baseurl=https://opendev.org/

main.yaml

- tenant:
    name: example-tenant
    source:
      gerrit:
        config-projects:
          - zuul-config
        untrusted-projects:
          - test1
          - test2
      zuul-ci.org:
        untrusted-projects:
          - zuul-jobs:
              include:
                - job

Running Dashboard

http://localhost:9000

Jobs

  • Define node types needed from nodepool
  • Define which ansible playbooks to run
  • Jobs may be defined centrally or in the repo being tested
  • Jobs have contextual variants that simplify configuration
  • Jobs definitions support inheritance

Job Libraries

  • Zuul jobs are all defined in git repositories
  • Designed to be directly shared across zuul installations
  • Standard library: https://opendev.org/zuul/zuul-jobs
  • Zuul installs should add zuul-jobs to their config
  • As changes land in zuul-jobs - Zuul installs will get them automatically

Local Job Libraries

Simple Job

- job:
   name: tox
   pre-run: playbooks/setup-tox.yaml
   run: playbooks/tox.yaml
   post-run: playbooks/fetch-tox-output.yaml

What about job content?

  • Written in Ansible
  • Ansible is excellent at running one or more tasks in one or more places
  • The answer to "how do I" is almost always "Ansible"

Playbooks

  • Jobs run playbooks
  • Allocated nodes from nodepool are provided in Ansible Inventory
  • Playbooks may be defined centrally or in the repo being tested
  • Playbooks can use roles from current or other Zuul repos
  • Playbooks are not allowed to execute content on 'localhost'
  • Note: localhost restriction may go away soon

Simple Playbook Example

  • playbooks/tox.yaml
- hosts: ubuntu-bionic
  tasks:
    - name: Run tox
      args:
        chdir: "{{ zuul_work_dir }}"
      shell: "tox -e{{ tox_envlist }}"

Job Inheritance

All Zuuls have a base job

- job:
    name: base
    parent: null
    description: |
      The base job for Zuul.
    timeout: 1800
    nodeset:
      nodes:
        - name: primary
          label: ubuntu-bionic
    pre-run: playbooks/base/pre.yaml
    post-run:
      - playbooks/base/post-ssh.yaml
      - playbooks/base/post-logs.yaml
    secrets:
      - site_logs

Logging

  • Zuul doesn't know about logging - only about URLs to report
  • Job Ansible content does something with logs
  • Job content tells Zuul what URL(s) to report to user
  • Allows for some really cool patterns

Live Web Preview

  • For static websites, publish built content and report url

https://review.opendev.org/#/c/635716/

https://zuul.opendev.org/t/zuul/build/2ef4ceb232a2467ab56be726f31928ef

https://470ff2d7584ecc192d95-20fcf28638b4f38abf523a12606af55d.ssl.cf1.rackcdn.com/635716/12/check/zuul-build-dashboard/2ef4ceb/npm/html

Project Admin Role

  • Base job(s)
  • Pre-playbooks for local environment
  • Post-playbooks - where do logs and artifacts go?
  • Shared trusted publication jobs

Simple Job Inheritance

- job:
    name: tox-py36
    parent: tox
    vars:
      tox_envlist: py36

Inheritance Works Like An Onion

  • pre-run playbooks run in order of inheritance
  • run playbook of job runs
  • post-run playbooks run in reverse order of inheritance
  • If pre-run playbooks fail, job is re-tried
  • All post-run playbooks run - as far as pre-run playbooks got

Inheritance Example

For tox-py36 job

  • base pre-run playbooks/base/pre.yaml
  • tox pre-run playbooks/setup-tox.yaml
  • tox run playbooks/tox.yaml
  • tox post-run playbooks/fetch-tox-output.yaml
  • base post-run playbooks/base/post-ssh.yaml
  • base post-run playbooks/base/post-logs.yaml

Simple Job Variant

- job:
    name: tox-py36
    branches: stable/mitaka
    nodeset:
      - name: ubuntu-trusty
        label: ubuntu-trusty

Nodesets for Multi-node Jobs

- nodeset:
    name: ceph-cluster
    nodes:
      - name: controller
        label: centos-7
      - name: compute1
        label: fedora-28
      - name: compute2
        label: fedora-28
    groups:
      - name: ceph-osd
        nodes:
          - controller
      - name: ceph-monitor
        nodes:
          - controller
          - compute1
          - compute2

Multi-node Job

  • nodesets are provided to Ansible for jobs in inventory
- job:
    name: ceph-multinode
    nodeset: ceph-cluster
    run: playbooks/install-ceph.yaml

Multi-node Ceph Job Content

- hosts: all
  roles:
    - install-ceph

- hosts: ceph-osd
  roles:
    - start-ceph-osd

- hosts: ceph-monitor
  roles:
    - start-ceph-monitor

- hosts: all
  roles:
    - do-something-interesting

Projects

  • Projects are git repositories
  • Specify a set of jobs for each pipeline
  • golang git repo naming as been adopted:
zuul@ubuntu-bionic:~$ find /home/zuul/src -mindepth 3 -maxdepth 3 -type d
/home/zuul/src/opendev.org/openstack/openstacksdk
/home/zuul/src/opendev.org/zuul/zuul
/home/zuul/src/github.com/ansible/ansible
/home/zuul/src/gerrit.googlesource.com/gerrit

Project Config

  • Specify a set of jobs for each pipeline
- project:
    check:
      jobs:
        - openstack-tox-py36
        - openstack-tox-docs
    gate:
      jobs:
        - openstack-tox-py36
        - openstack-tox-docs

Project with Local Variant

- project:
    check:
      jobs:
        - openstack-tox-py27
        - openstack-tox-py35
        - openstack-tox-py36:
            voting: false
        - openstack-tox-docs
    gate:
      jobs:
        - openstack-tox-py27
        - openstack-tox-py35
        - openstack-tox-docs

Project with More Local Variants

- project:
    check:
      jobs:
        - openstack-tox-py27
        - openstack-tox-py35
        - openstack-tox-py36:
            voting: false
        - openstack-tox-docs:
            files: '^docs/.*$'

Project with Many Local Variants

- project:
    check:
      jobs:
        - openstack-tox-py27:
           nodeset:
             - name: centos-7
               label: centos-7
        - openstack-tox-py27:
            branches: stable/newton
            nodeset:
              - name: ubuntu-trusty
                label: ubuntu-trusty
        - openstack-tox-py35
        - openstack-tox-py36:
            voting: false
        - openstack-tox-docs:
            files: '^docs/.*$'

Project With Central and Local Config

# In opendev.org/openstack/project-config:
- project:
    name: openstack/nova
    templates:
      - openstack-tox-jobs
# In opendev.org/openstack/nova/.zuul.yaml:
- project:
    check:
      - nova-placement-functional-devstack

Project with Job Dependencies

Full DAG of Job Dependencies

- project:
    release:
      jobs:
        - build-artifacts
        - upload-tarball:
            dependencies: build-artifacts
        - upload-pypi:
            dependencies: build-artifacts
        - notify-mirror:
            dependencies:
              - upload-tarball
              - upload-pypi

Speculative Container Images

  • Build container images based on non-published images from other changes
  • Test built container images in gate jobs
  • Promote exact image from gate after merge
  • Soft Dependencies - use artifact from another job only if it was built
  • https://zuul-ci.org/docs/zuul-jobs/docker-image.html

Playbooks

  • Jobs run playbooks
  • Playbooks may be defined centrally or in the repo being tested
  • Playbooks can use roles from current or other Zuul repos or Galaxy
  • Playbooks are not allowed to execute content on 'localhost'

Test Like Production

If you use Ansible for deployment, your test and deployment processes and playbooks are the same

What if you don't use Ansible?

OpenStack Infra Control Plane uses Puppet (for now)

OpenDev Control Plane uses some Puppet (for now)

- name: Install puppet
  shell: ./install_puppet.sh
  args:
    chdir: "{{ ansible_user_dir }}/src/opendev.org/opendev/system-config"
  become: yes

- name: Copy manifest
  copy:
    src: manifest.pp
    dest: "{{ ansible_user_dir }}/manifest.pp"

- name: Run puppet
  puppet:
    manifest: "{{ ansible_user_dir }}/manifest.pp"
  become: yes

Secrets

  • Inspired by Kubernetes Secrets API
  • Projects can add named encrypted secrets to their .zuul.yaml file
  • Jobs can request to use secrets by name
  • Jobs using secrets are not reconfigured speculatively
  • Secrets can only be used by the same project they are defined in
  • Public key per project: {{ zuul_url }}/{{ tenant }}/{{ project }}.pub
::

GET https://zuul.opendev.org/api/tenant/openstack/key/openstack/openstacksdk.pub

Secret Example (note, no admins had to enable this)

# In opendev.org/openstack/loci/.zuul.yaml:
- secret:
    name: loci_docker_login
    data:
      user: loci-username
      password: !encrypted/pkcs1-oaep
        - gUEX4eY3JAk/Xt7Evmf/hF7xr6HpNRXTibZjrKTbmI4QYHlzEBrBbHey27Pt/eYvKKeKw
          hk8MDQ4rNX7ZK1v+CKTilUfOf4AkKYbe6JFDd4z+zIZ2PAA7ZedO5FY/OnqrG7nhLvQHE
          5nQrYwmxRp4O8eU5qG1dSrM9X+bzri8UnsI7URjqmEsIvlUqtybQKB9qQXT4d6mOeaKGE
          5h6Ydkb9Zdi4Qh+GpCGDYwHZKu1mBgVK5M1G6NFMy1DYz+4NJNkTRe9J+0TmWhQ/KZSqo
          4ck0x7Tb0Nr7hQzV8SxlwkaCTLDzvbiqmsJPLmzXY2jry6QsaRCpthS01vnj47itoZ/7p
          taH9CoJ0Gl7AkaxsrDSVjWSjatTQpsy1ub2fuzWHH4ASJFCiu83Lb2xwYts++r8ZSn+mA
          hbEs0GzPI6dIWg0u7aUsRWMOB4A+6t2IOJibVYwmwkG8TjHRXxVCLH5sY+i3MR+NicR9T
          IZFdY/AyH6vt5uHLQDU35+5n91pUG3F2lyiY5aeMOvBL05p27GTMuixR5ZoHcvSoHHtCq
          7Wnk21iHqmv/UnEzqUfXZOque9YP386RBWkshrHd0x3OHUfBK/WrpivxvIGBzGwMr2qAj
          /AhJsfDXKBBbhGOGk1u5oBLjeC4SRnAcIVh1+RWzR4/cAhOuy2EcbzxaGb6VTM=

Secret Example

# In opendev.org/openstack/loci/.zuul.yaml:
- job:
    name: publish-loci-cinder
    parent: loci-cinder
    post-run: playbooks/push
    secrets:
      - loci_docker_login

# In opendev.org/openstack/loci/playbooks/push.yaml:
- hosts: all
  tasks:
    - include_vars: vars.yaml

- name: Push project to DockerHub
  block:
    - command: docker login -u {{ loci_docker_login.user }} -p {{ loci_docker_login.password }}
      no_log: True
    - command: docker push openstackloci/{{ project }}:{{ branch }}-{{ item.name }}
      with_items: "{{ distros }}"

Back to our Install

Zuul Containers

  • Published on every commit
  • Application/Process containers
  • Built using Speculative Containers
  • Config / Data should be bind-mounted in

Container Philosophy

  • As minimal as possible
  • OS inside of container does not matter

zuul/zuul-executor

  • In k8s, zuul-executor must be run in privileged pod
  • Uses bubblewrap for unprivileged sanboxing
  • Restriction may be lifted in the future

Release Management

  • Zuul is a CI system
  • C stands for "Continuous"
  • It is run Continuously Delivered and Deployed upstream
  • Releases are tagged from code run upstream
  • There is no intent to have a 'stable' release
  • 'stable' is a synonym for "old and buggy"
  • NOTE container images are not currently tagged

zuul/zuul-scheduler

  • SPOF
  • We're working on it
  • Recommend running scheduler from tags

Let's Do Something with Our Zuul

Gerrit Account

  • Need a user account to interact with Gerrit
  • Gerrit is configured in dev mode - no passwords required
  • Visit http://localhost:8080
  • Click "New Account"
  • Skip entering anything, click "settings"
  • Enter username in Username field (match your local laptop user)
  • Enter Full Name
  • Click "Save Changes"

More Gerrit Account

  • Scroll down to Email Addresses
  • Enter email address in "New email address" field
  • Click "Send Verification"
  • Gerrit is in Developer Mode, will not send email
  • Scroll down to "SSH Keys"
  • Copy ~/.ssh/id_rsa.pub contents into "New SSH Key" field
  • Click "Add New SSH Key"
  • Reload the page in your browser

Config Repo

  • zuul-config is a trusted config-repo
  • Security and functionality of system depend on this repo
  • Limit its contents to minimum required

Setting up Gating

  • We want to have changes to zuul-config be gated
  • We need to define pipelines: check and gate
  • Need to attach zuul-config to them
  • Start with builtin noop job (always return success)
  • Use regex to attach all projects to check and gate

Pipeline Definitions

  • Zuul has no built-in workflow definitions, let's add check and gate

check pipeline

- pipeline:
    name: check
    description: |
      Newly uploaded patchsets enter this pipeline to receive an
      initial +/-1 Verified vote.
    manager: independent
    require:
      gerrit:
        open: True
        current-patchset: True
    trigger:
      gerrit:
        - event: patchset-created
        - event: change-restored
          comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck
    success:
      gerrit:
        Verified: 1
      mysql:
    failure:
      gerrit:
        Verified: -1
      mysql:

gate pipeline

- pipeline:
    name: gate
    description: |
      Changes that have been approved are enqueued in order in this
      pipeline, and if they pass tests, will be merged.
    manager: dependent
    post-review: True
    require:
      gerrit:
        open: True
        current-patchset: True
        approval:
          - Workflow: 1
    trigger:
      gerrit:
        - event: comment-added
          approval:
            - Workflow: 1
    start:
      gerrit:
        Verified: 0
    success:
      gerrit:
        Verified: 2
        submit: true
      mysql:
    failure:
      gerrit:
        Verified: -2
      mysql:

Add the pipeline definitions

cd ~/src
git clone http://localhost:8080/zuul-config
cd zuul-config
mkdir zuul.d
cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/zuul-config/zuul.d/pipelines.yaml zuul.d

Shared Project Pipeline Definition

In examples/zuul-config/zuul.d/projects.yaml

- project:
    name: ^.*$
    check:
      jobs: []
    gate:
      jobs: []

- project:
    name: zuul-config
    check:
      jobs:
        - noop
    gate:
      jobs:
        - noop

Attach the projects to the pipelines

cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/zuul-config/zuul.d/projects.yaml zuul.d

Commit the changes and push up for review

git add zuul.d
git commit
git review

Force merging bootstrap config

  • Zuul is running with no config, so it won't do anything
  • For this change (and this change only) we will bypass gating

Reviewing normally

Verified +2 is Missing

Verified +2 is what we have zuul configured to do.

::
success:
gerrit:

Verified: 2 submit: true

Bypassing Gating

  • visit http://localhost:8080/
  • Click avatar in Upper Right
  • Click Sign Out
  • Click Sign In
  • Click 'admin'
  • Click on change you uploaded
  • Click Reply...
  • Vote +2 Verified (normal users do not see this)
  • Click submit (normal users do not see this)
  • Click avatar in Upper Right
  • Click Sign Out
  • Click Sign In
  • click your username

Dashboard

http://localhost:9000/t/example-tenant/status

Add Job to a repo

Clone the test1 repo

cd ~/src
git clone http://localhodsts:8080/test1
cd test1

Simple debug playbook

- hosts: all
  tasks:
    - debug:
        msg: Hello world!

Add the playbook to our repo

mkdir playbooks
cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/test1/playbooks/testjob.yaml playbooks
git add playbooks/testjob.yaml

Simple Zuul Job

- job:
    name: testjob
    run: playbooks/testjob.yaml

- project:
    check:
      jobs:
        - testjob
    gate:
      jobs:
        - testjob

Add it to the .zuul.yaml

This is a normal repo, let's use .zuul.yaml

::

cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/test1/zuul.yaml .zuul.yaml git add .zuul.yaml

Submit the change for review

git add .zuul.yaml playbooks
git commit -m "Add test Zuul job"
git review

Then check http://localhost:8080/dashboard/self

Where are the logs?

Base Job

  • Every Zuul installation must define a base job
  • Push git repos to build node
  • Publish logs/artifacts
  • Any local specific setup
  • Goes in config repo - because it impacts EVERY job

Minimal Base Job

- job:
    name: base
    parent: null
    nodeset:
      nodes:
        - name: ubuntu-bionic
          label: ubuntu-bionic

Minimal Base Job

cd ~/src/zuul-config
cp ~/src/examples/zuul-config/zuul.d/jobs.yaml zuul.d

Base pre Playbook

  • Using roles from zuul-jobs repo
  • Creates a per-build ssh key
  • Copies git repos to test node
- hosts: all
  roles:
    - add-build-sshkey
    - prepare-workspace

Base pre Playbook

mkdir -p playbooks/base
cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/zuul-config/playbooks/base/pre.yaml playbooks/base

Base post-ssh playbook

  • Removes the per-build ssh key from the pre playbook
- hosts: all
  roles:
    - remove-build-sshkey

Base post-ssh playbook

cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/zuul-config/playbooks/base/post-ssh playbooks/base

Base post-logs playbook

  • Quickstart is running Apache server for log content
  • Volume mounted at /srv/static/logs (default for upload-logs)
  • Tells Zuul where the logs can be found
- hosts: localhost
  gather_facts: False
  roles:
    - role: upload-logs
      zuul_log_url: "http://localhost:8000"

Base post-ssh playbook

cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/zuul-config/playbooks/base/post-logs playbooks/base

Update Base Job

- job:
    name: base
    parent: null
    description: |
      The recommended base job.

      All jobs ultimately inherit from this.  It runs a pre-playbook
      which copies all of the job's prepared git repos on to all of
      the nodes in the nodeset.

      It also sets a default timeout value (which may be overidden).
    pre-run: playbooks/base/pre.yaml
    post-run:
      - playbooks/base/post-ssh.yaml
      - playbooks/base/post-logs.yaml
    roles:
      - zuul: zuul/zuul-jobs
    timeout: 1800
    nodeset:
      nodes:
        - name: ubuntu-bionic
          label: ubuntu-bionic

Update Base Job

cp ~/src/examples/zuul-config/zuul.d/jobs2.yaml zuul.d

Commit and Push for Review

git add playbooks zuul.d/jobs.yaml
git commit -m "Update Zuul base job"
git review

Land Change

  • Visit http://localhost:8080/dashboard/self
  • Should be a Verified +1 vote from Zuul
  • Click Reply
  • Vote: Code-Review +2 and Workflow +1
  • Click Send
  • Wait a bit, then reload, change should be merged

Recheck the test1 change

* Click Reply " Comment "recheck" * Click Send

Job Now Running with Updated Base Job

  • Should get a link to log location
  • Check out Zuul Dashboard

Zuul tests syntax automatically

  • Edit zuul/jobs.yaml
  • Change parent: null to parent: broken
  • git commit ; git review
  • Check out the review in gerrit ... there should be errors!

Job configs are shared

cd ~/src
git clone http://localhost:8080/test2
cd test2

Add testjob to test2 repo

cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/test1/zuul.yaml .zuul.yaml
git add .zuul.yaml
git commit -a -m"Added testjob to test2 repo"
git review

Where's the job?

Add Depends-On footer

git commit -a --amend
  • Add a blank line then
Depends-On: http://localhost:8080/c/test1/+/1
  • Then resubmit
git review

Let's Add a Secret

  • Edit .zuul.yaml - name the secret and the field
cd ~/src/test1
echo "my-secret" | ~/src/opendev.org/zuul/zuul/tools/encrypt_secret.py --tenant example-tenant http://localhost:9000 test1 >> .zuul.yaml

Secret Name

- secret:
    name: amazing_secret
    data:
      name: !encrypted/pkcs1-oaep

Add secret to job

- job:
    name: testjob
    run: playbooks/testjob.yaml
    secrets:
      - amazing_secret

Print the secret in the job

  • NOTE This is dumb
  • Edit playbooks/testjob.yaml
- hosts: all
  tasks:
    - debug:
        msg: "Hello {{ amazing_secret.name }}!"

Commit and push for review

git commit -a -m"Added a secret"
git review

Change test2 to depend on the secret change

git commit -a --amend
  • Change the Depends-On line
Depends-On: http://localhost:8080/c/test1/+/2
  • Then resubmit
git review

Let's go look at the outputs!

Add a job with run-test-command

  • What if you don't want a playbook?
  • Role 'run-test-command' in zuul-jobs
- job:
    parent: run-test-command
    name: simple-test-job
    vars:
      test_command: exit 0

Add to the project pipeline config

- project:
    check:
      jobs:
        - testjob
        - simple-test-job
    gate:
      jobs:
        - testjob
        - simple-test-job

Commit and Push Up

git commit -a -m"Added a simple test job"
git review

What's wrong?

Add required-projects

  • Edit .zuul.yaml
- job:
    parent: run-test-command
    name: simple-test-job
    vars:
      test_command: "find {{ ansible_user_dir }}"
    required-projects:
      - test1
      - test2

Important Links