inaugust
/
inaugust.com
Code Issues Proposed changes
inaugust.com/src/zuulv3/getting-started.rst

19 KiB
Raw Blame History

. display in 68x24 .. display in 88x24

dissolve

Test Slide

images/testslide.ans

Preshow

images/cursor.ans images/cursor2.ans

Zuul

images/title.ans

Red Hat

images/redhat.ans

Ansible

images/ansible.ans

Zuul

images/zuul.ans

What Zuul Does

  • gated changes
  • one or more git repositories
  • integrated deliverable
  • testing like deployment

Underlying Philosophy

  • All changes flow through code review
  • Changes only land if they pass all tests
  • Computers are cheaper than humans

Ramifications of Philosophy

  • No direct push access for anyone
  • Software should be installable from source
  • Testing should be automated and repeatable
  • Developers write tests with their patches
  • Code always works

Getting to Gating

No Tests / Manual Tests

  • No test automation exists or ...
  • Developer runs test suite before pushing code
  • Prone to developer skipping tests for "trivial" changes
  • Doesn't scale organizationally

Periodic Testing

  • Developers push changes directly to shared branch
  • CI system runs tests from time to time - report if things still work
  • "Who broke the build?"
  • Leads to hacks like NVIE model

Post-Merge Testing

  • Developers push changes directly to shared branch
  • CI system is triggered by push - reports if push broke something
  • Frequently batched / rolled up
  • Easier to diagnose which change broke things
  • Reactive - the bad changes are already in

Pre-review Testing

  • Changes are pushed to code review (Gerrit Change, GitHub PR, etc)
  • CI system is triggered by code review change creation
  • Test results inform review decisions
  • Proactive - testing code before it lands
  • Reviewers can get bored waiting for tests
  • Only tests code as written, not potential result of merging code

Gating

  • Changes are pushed to code review
  • CI system is triggered by code review approval
  • CI system merges code IFF tests pass
  • Proactive - testing code before it lands
  • Future state resulting from merge of code is tested
  • Reviewers can fire-and-forget safely

Mix and Match

  • Zuul supports all of those modes
  • Zuul users frequently combine them
  • Run pre-review (check) and gating (gate) on each change
  • Post-merge/post-tag for release/publication automation
  • Periodic for catching bitrot

Multi-repository integration

  • Multiple source repositories are needed for deliverable
  • Future state to be tested is the future state of all involved repos

To test proposed future state

  • Get tip of each project. Merge appropriate change. Test.
  • Changes must be serialized, otherwise state under test is invalid.
  • Integrated deliverable repos share serialized queue

Speculative Execution

  • Correct parallel processing of serialized future states
  • Create virtual serial queue of changes for each deliverable
  • Assume each change will pass its tests
  • Test successive changes with previous changes applied to starting state

Nearest Non-Failing Change

(aka 'The Jim Blair Algorithm')

  • If a change fails, move it aside
  • Cancel all test jobs behind it in the queue
  • Reparent queue items on the nearest non-failing change
  • Restart tests with new state

Zuul Simulation

pan

  • todo

images/zsim-00.ans

Zuul Simulation

cut

  • todo

images/zsim-01.ans

Zuul Simulation

cut

  • todo

images/zsim-02.ans

Zuul Simulation

cut

  • todo

images/zsim-03.ans

Zuul Simulation

cut

  • todo

images/zsim-04.ans

Zuul Simulation

cut

  • todo

images/zsim-05.ans

Zuul Simulation

cut

  • todo

images/zsim-06.ans

Zuul Simulation

cut

  • todo

images/zsim-07.ans

Zuul Simulation

cut

  • todo

images/zsim-08.ans

Zuul Simulation

cut

  • todo

images/zsim-09.ans

Zuul Simulation

cut

  • todo

images/zsim-10.ans

Zuul Simulation

cut

  • todo

images/zsim-11.ans

Zuul Simulation

cut

  • todo

images/zsim-12.ans

Zuul Simulation

cut

  • todo

images/zsim-13.ans

Zuul Simulation

cut

  • todo

images/zsim-14.ans

Zuul Simulation

cut

  • todo

images/zsim-15.ans

Zuul Simulation

cut

  • todo

images/zsim-16.ans

Zuul Simulation

cut

  • todo

images/zsim-17.ans

Zuul Simulation

cut

  • todo

images/zsim-18.ans

Zuul Simulation

cut

  • todo

images/zsim-19.ans

Zuul Simulation

cut

  • todo

images/zsim-20.ans

Zuul Simulation

cut

  • todo

images/zsim-21.ans

Zuul Simulation

cut

  • todo

images/zsim-22.ans

Cross-Project Dependencies

Testing or gating dependencies manually specified by developers

Live Configuration Changes

Zuul is a distributed system, with a distributed configuration.

- tenant:
    name: openstack
    source:
      gerrit:
        config-repos:
          - openstack-infra/project-config
        project-repos:
          - openstack/nova
          - openstack/keystone
          - openstack-infra/devstack-gate

Zuul Startup

  • Read config file

Zuul Startup

  • Read config file
  • Ask mergers for branches of each repo

images/startup1.ans

Zuul Startup

  • Read config file

  • Ask mergers for branches of each repo

  • Ask mergers for .zuul.yaml for each branch

    of each repo

images/startup2.ans

When .zuul.yaml Changes

  • Zuul looks for changes to .zuul.yaml

  • Asks mergers for updated content

  • Splices into configuration used for that change

  • Works with cross-repo dependencies

    ("This change depends on a change to the job definition")

Zuul Architecture

images/architecture.ans

Nodepool

  • A separate program that works very closely with Zuul
  • Creates and destroys zero or more node resources
  • Resources can include VMs, Containers, COE contexts or Bare Metals
  • Static driver for allocating pre-existing nodes to jobs
  • Optionally periodically builds images and uploads to clouds

Nodepool Launcher

Where build nodes should come from

  • OpenStack
  • Static

In review:

  • Kubernetes
  • OpenShift
  • AWS

In work / coming soon:

  • Azure
  • GCE
  • Mac Stadium

What about test/job content?

  • Written in Ansible
  • Ansible is excellent at running one or more tasks in one or more places
  • The answer to "how do I" is almost always "Ansible"

What Zuul Does

  • Listens for code events
  • Prepares appropriate job config and git repo states
  • Allocates nodes for test jobs
  • Pushes git repo states to nodes
  • Runs user-defined Ansible playbooks
  • Collects/reports results
  • Potentially merges change

OpenStack Infra - Largest Known Zuul

  • 2KJPH (2,000 jobs per hour)
  • Build Nodes from 16 Regions of 5 Public and 3 Private OpenStack Clouds
  • Rackspace, Internap, OVH, Vexxhost, CityCloud
  • Linaro, Limestone, Packethost
  • 10,000 changes merged per month

Zuul is not New

  • Has been in Production for OpenStack for Six Years
  • Zuul is now a top-level effort of OpenStack Foundation
  • Zuul is in production for OpenStack (Control Plane in OpenStack VMs)
  • Zuul v3 first release where not-OpenStack is first-class use case

Not just for OpenStack

  • BMW (control plane in OpenShift)
  • GoDaddy (control plane in Kubernetes)
  • Easystack
  • OpenContrail
  • OpenLab
  • Red Hat
  • others ...

Code Review Systems

  • Gerrit
  • GitHub (Public and Enterprise)

In work / coming soon:

  • GitLab
  • Bitbucket

Support for non-git

  • Nope
  • helix4git may work for perforce, but is untested

Installation of Software

Ways to Install Zuul

Zuul Containers

  • Published on every commit
  • Application/Process containers
  • Config / Data should be bind-mounted in

zuul/zuul-executor

  • In k8s, zuul-executor must be run privileged
  • Uses bubblewrap for unprivileged sanboxing
  • Restriction may be lifted in the future

Release Management

  • Zuul is a CI system
  • C stands for "Continuous"
  • It is run Continuously Delivered and Deployed upstream
  • Releases are tagged from code run upstream
  • There is no intent to have a 'stable' release
  • 'stable' is a synonym for "old and buggy"

zuul/zuul-scheduler

  • SPOF
  • We're working on it
  • Recommend running scheduler from tags

Quick Start

https://zuul-ci.org/docs/zuul/admin/quick-start.html

Important Links

Questions

images/questions.ans

Quick Start Prereq

  • Install docker, docker-compose, git-review

Debian/Ubuntu:

sudo apt-get install docker-compose git git-review

RHEL / CentOS / Fedora:

sudo yum install docker docker-compose git git-review

OpenSuse:

sudo zypper install docker docker-compose git git-review

RHEL / CentOS / Fedora / OpenSuse

sudo systemctl enable docker.service
sudo systemctl start docker.service

Actual Quick Start

What's Running

  • Zookeeper
  • Gerrit
  • Nodepool Launcher
  • Zuul Scheduler
  • Zuul Web Server
  • Zuul Executor
  • Apache HTTPD
  • A container to use as a 'static' build node

How they're connected

  • End Users talk to Gerrit and Apache HTTPD
  • Zuul Scheduler talks to Gerrit
  • Nodepool Launcher, Zuul Scheduler, Zuul Web talk to Zookeeper
  • Zuul Executor talks to Zuul Scheduler (using Gearman)

Initial provided config

  • docker-compose has plumbed in basic config etc_zuul/zuul.conf and etc_zuul/main.yaml
  • Gerrit Connection named "gerrit"
  • Zuul user for that connection
  • Git connection named "opendev.org" for zuul-jobs standard library

Initial tenant

  • Zuul is (always) multi-tenant
  • Example config contains a tenant called example-tenant
  • Three projects in the example-tenant tenant: zuul-config, test1, test2
  • Three projects are also in gerrit ready to use

zuul.conf

[gearman]
server=scheduler

[gearman_server]
start=true

[zookeeper]
hosts=zk

[scheduler]
tenant_config=/etc/zuul/main.yaml

[web]
listen_address=0.0.0.0

[executor]
private_key_file=/var/ssh/nodepool
default_username=root

zuul.conf part 2

[connection "gerrit"]
name=gerrit
driver=gerrit
server=gerrit
sshkey=/var/ssh/zuul
user=zuul
password=secret
baseurl=http://gerrit:8080
auth_type=basic

[connection "opendev.org"]
name=opendev
driver=git
baseurl=https://opendev.org/

main.yaml

- tenant:
    name: example-tenant
    source:
      gerrit:
        config-projects:
          - zuul-config
        untrusted-projects:
          - test1
          - test2
      opendev.org:
        untrusted-projects:
          - zuul/zuul-jobs:
              include:
                - job

Gerrit Account

  • Need a user account to interact with Gerrit
  • Gerrit is configured in dev mode - no passwords required
  • Visit http://localhost:8080
  • Click "Become"
  • Click "New Account"
  • Click "Register"
  • Enter Full Name
  • Click "Save Changes"
  • Enter username in Username field (match your local laptop user)
  • Copy ~/.ssh/id_rsa.pub contents into SSH Key field
  • Click Continue

Config Repo

  • zuul-config is a trusted config-repo
  • Security and functionality of system depend on this repo
  • Limit its contents to minimum required

Config Files vs. Directories

  • Zuul reads config from: .zuul.yaml, zuul.yaml, zuul.d or .zuul.d
  • For projects with substantial zuul config, like zuul-config zuul.d directory is likely best.
  • The directories are read run-parts style.
  • Recommended practice is splitting by type of object

Setting up Gating

  • We want to have changes to zuul-config be gated
  • We need to define pipelines: check and gate
  • Need to attach zuul-config to them
  • Start with builtin noop job (always return success)
  • Use regex to attach all projects to check and gate

Pipeline Definitions

  • Zuul has no built-in workflow definitions, let's add check and gate

check pipeline

- pipeline:
    name: check
    description: |
      Newly uploaded patchsets enter this pipeline to receive an
      initial +/-1 Verified vote.
    manager: independent
    require:
      gerrit:
        open: True
        current-patchset: True
    trigger:
      gerrit:
        - event: patchset-created
        - event: change-restored
    success:
      gerrit:
        Verified: 1
    failure:
      gerrit:
        Verified: -1

gate pipeline

::
  • pipeline:

    name: gate description: | Changes that have been approved are enqueued in order in this pipeline, and if they pass tests, will be merged. manager: dependent post-review: True require: gerrit: open: True current-patchset: True approval: - Workflow: 1 trigger: gerrit: - event: comment-added approval: - Workflow: 1 start: gerrit: Verified: 0 success: gerrit: Verified: 2 submit: true failure: gerrit: Verified: -2

Add the pipeline definitions

git clone http://localhost:8080/zuul-config
cd zuul-config
mkdir zuul.d
cp ../examples/zuul-config/zuul.d/pipelines.yaml .

Shared Project Pipeline Definition

In examples/zuul-config/zuul.d/projects.yaml

- project:
    name: ^.*$
    check:
      jobs: []
    gate:
      jobs: []

- project:
    name: zuul-config
    check:
      jobs:
        - noop
    gate:
      jobs:
        - noop

Attach the projects to the pipelines

cp ../examples/zuul-config/zuul.d/projects.yaml .

Commit the changes and push up for review

git add zuul.d
git commit
git review

Force merging bootstrap config

  • Zuul is running with no config, so it won't do anything
  • For this change (and this change only) we will bypass gating

Reviewing normally

Verified +2 is Missing

Verified +2 is what we have zuul configured to do.

::
success:
gerrit:

Verified: 2 submit: true

Bypassing Gating

Base Job

  • Every Zuul installation must define a base job
  • Push git repos to build node
  • Publish logs/artifacts
  • Any local specific setup
  • Goes in config repo - because it impacts EVERY job

Add Base Job to zuul-config

cp ../examples/zuul-config/zuul.d/jobs.yaml .
git add jobs.yaml
git commit
git review

Then go to http://localhost:8080/#/c/zuul-config/+/1002/ and approve it

Zuul should merge the patch

zuul-config is configured to use the noop job

Zuul tests syntax automatically

  • Edit jobs.yaml
  • Change parent: null to parent: broken
  • git commit ; git review
  • Check out the review in gerrit ... there should be errors!

Presentty

pan

Presentty