25 KiB

Raw Blame History

. display in 68x24 .. display in 88x24

dissolve

Test Slide

images/testslide.ans

Preshow

images/cursor.ans images/cursor2.ans

Zuul

images/title.ans

Red Hat

images/redhat.ans

OpenStack

images/openstack.ans

OpenDev

"most insane CI infrastructure I've ever been a part of"

  -- Alex Gaynor

"like the SpaceX of CI"

  -- Emily Dunham

Zuul

images/zuul.ans

Ansible

images/ansible.ans

What Zuul does

"Speculative Future State"

multiple repositories

integrated deliverable

gated commits

open tooling

nobody is special

testing like deployment

OpenStack Is

Federated

Distributed

Large

Open

Not Alone

Federated

Hundreds of involved companies

No 'main' company

"Decisions are made by those who show up"

Union of priorities/use cases

Impact of being Federated

No company can appoint people to positions in the project

The project cannot fire anyone

Variable background of contributors

Heavy reliance on consensus

Distributed

There is no office

Contributor base is global

Multitude of contributor backgrounds

Impact of being Distributed

Tooling must empower all contributors, regardless of background, skill level or cultural context

Heavy preference for text-based communication

Cannot assume US-centric needs or solutions

"Accept patches from random people on the internet"

Large numbers of

Contributors (~2k in any given 6 month period)

Changes

Code Repositories (2082 as of this morning)

Not Bragging About Scale

OpenStack Scale Comparison

2KJPH (2,000 jobs per hour)

Build Nodes from 16 Regions of 5 Public and 3 Private OpenStack Clouds

Rackspace, Internap, OVH, Vexxhost, CityCloud

Linaro (ARM), Limestone, Packethost

10,000 changes merged per month

OpenStack Scale Comparison

2KJPH (2,000 jobs per hour)

Build Nodes from 16 Regions of 5 Public and 3 Private OpenStack Clouds

Rackspace, Internap, OVH, Vexxhost, CityCloud

Linaro (ARM), Limestone, Packethost

10,000 changes merged per month

By comparison, our friends at the amazing project Ansible received 13,000 changes and had merged 8,000 of them in its first 4 years.

Impact of scale

Empower teams to take care of themselves (distributed)

Efficiency gained from shared solutions (centralized)

Zuul supports per-repo config, central config, and multiple tenants

One Zuul install is all you need for all of BMW or ExxonMobil or IBM

Four Opens

Open Source (we don't hold back Enterprise features, we don't cripple things)

Open Design (design process open to all, decisions are not made inside company doors)

Open Development (public source code, public code review, all code is reviewed and gated)

Open Community (lazy consensus, democratic leadership from participants, public logged meetings in IRC, public archived mailing lists)

Impact of Four Opens

Tooling is not exempt

Fifth Open - Open Operations - all infrastructure is run in the Open, via GitOps

We're Not Alone

Dependencies (libvirt/kvm/xen, mysql/pg, rabbit, python/javascript, ceph/gluster, ansible/salt/puppet/chef, ovs/odl)

Adjacencies (kubernetes, ansible, terraform, opnfv, spinnaker)

Vendors (plugins, products, services, distros)

Developer Process In a Nutshell

Code Review - nobody has direct commit/push access

3rd-Party CI for vendors

Gated Commits

Developer Workflow

Who has submitted a patch?
Who wants to?
(Who is here because the name of this talk is weird?)

Hack             Review              Test
=========         ==========         ==========

        push              approve
   +-------------+    +-------------+
   |             |    |             |
+------+--+       +--v----+--+       +--v-------+
|         |       |          |       |          |
| $EDITOR |       |  Gerrit  |       |   Zuul   |
|         |       |          |       |          |
+------^--+       +--+----^--+       +--+-------+
   |             |    |             |
   +-------------+    +-------------+
        clone              merge

Gerrit

explain patch upload, zuul runs, test results displayed in gerrit this is all the interface to zuul users need to see

switch to actual gertty screenshot

also show zuul status page

but zuul is doing a lot of work behind the scenes, and if you look closer, this is what you see

images/color-gertty.ans

Zuul is not New

Has been in Production for OpenStack for Six Years

Zuul v3 first release where not-OpenStack is first-class use case

Zuul is now an OpenStack Foundation Pilot Project

Not just for OpenStack

Zuul is in production for OpenStack (in OpenStack VMs)

Also running at:

BMW (control plane in OpenShift)

GoDaddy (control plane in Kubernetes)

GoodMoney (control plane in EKS, adding GKE)

Le Bon Coin

Easystack

TungstenFabric

OpenLab

Red Hat

others ...

Zuul in a nutshell

Listens for code events

Prepares appropriate job config and git repo states

Allocates nodes for test jobs

Pushes git repo states to nodes

Runs user-defined Ansible playbooks

Collects/reports results

Potentially merges change

All in Service of Gating

No Tests / Manual Tests

No test automation exists or ...

Developer runs test suite before pushing code

Prone to developer skipping tests for "trivial" changes

Doesn't scale organizationally

Periodic Testing

Developers push changes directly to shared branch

CI system runs tests from time to time - report if things still work

"Who broke the build?"

Leads to hacks like NVIE model

Post-Merge Testing

Developers push changes directly to shared branch

CI system is triggered by push - reports if push broke something

Frequently batched / rolled up

Easier to diagnose which change broke things

Reactive - the bad changes are already in

Pre-Review Testing

Changes are pushed to code review (Gerrit Change, GitHub PR, etc)

CI system is triggered by code review change creation

Test results inform review decisions

Proactive - testing code before it lands

Reviewers can get bored waiting for tests

Only tests code as written, not potential result of merging code

Gating

Changes are pushed to code review

Gating system is triggered by code review approval

Gating system merges code IFF tests pass

Proactive - testing code before it lands

Future state resulting from merge of code is tested

Reviewers can fire-and-forget safely

Mix and Match

Zuul supports all of those modes

Zuul users frequently combine them

Run pre-review (check) and gating (gate) on each change

Post-merge/post-tag for release/publication automation

Periodic for catching bitrot

Multi-repository integration

Multiple source repositories are needed for deliverable

Future state to be tested is the future state of all involved repos

To test proposed future state

Get tip of each project. Merge appropriate change(s). Test.

Changes must be serialized, otherwise state under test is invalid.

Integrated deliverable repos share serialized queue

Speculative Execution

Correct parallel processing of serialized future states

Create virtual serial queue of changes for each deliverable

Assume each change will pass its tests

Test successive changes with previous changes applied to starting state

Nearest Non-Failing Change

(aka 'The Jim Blair Algorithm')

If a change fails, move it aside

Cancel all test jobs behind it in the queue

Reparent queue items on the nearest non-failing change

Restart tests with new state

Zuul Simulation

pan

todo

images/zsim-00.ans

Zuul Simulation

cut

todo

images/zsim-01.ans

Zuul Simulation

cut

todo

images/zsim-02.ans

Zuul Simulation

cut

todo

images/zsim-03.ans

Zuul Simulation

cut

todo

images/zsim-04.ans

Zuul Simulation

cut

todo

images/zsim-05.ans

Zuul Simulation

cut

todo

images/zsim-06.ans

Zuul Simulation

cut

todo

images/zsim-07.ans

Zuul Simulation

cut

todo

images/zsim-08.ans

Zuul Simulation

cut

todo

images/zsim-09.ans

Zuul Simulation

cut

todo

images/zsim-10.ans

Zuul Simulation

cut

todo

images/zsim-11.ans

Zuul Simulation

cut

todo

images/zsim-12.ans

Zuul Simulation

cut

todo

images/zsim-13.ans

Zuul Simulation

cut

todo

images/zsim-14.ans

Zuul Simulation

cut

todo

images/zsim-15.ans

Zuul Simulation

cut

todo

images/zsim-16.ans

Zuul Simulation

cut

todo

images/zsim-17.ans

Zuul Simulation

cut

todo

images/zsim-18.ans

Zuul Simulation

cut

todo

images/zsim-19.ans

Zuul Simulation

cut

todo

images/zsim-20.ans

Zuul Simulation

cut

todo

images/zsim-21.ans

Zuul Simulation

cut

todo

images/zsim-22.ans

Explicit Cross-Project Dependencies

Developers can mark changes as being dependent

Depends-On: footer - in commit or PR

Zuul uses depends-on when constructing virtual serial queue

Will not merge changes in gate before depends-on changes

Works cross-repo AND cross-source

Lock Step Changes

Circular Dependencies are not supported on purpose

Rolling upgrades across interdependent services

HOWEVER - many valid use cases (go/rust/c++) - support will be coming

Live Configuration Changes

Zuul is a distributed system, with a distributed configuration.

- tenant:
    name: openstack
    source:
      gerrit:
        config-repos:
          - opendev/project-config
        project-repos:
          - zuul/zuul-jobs
          - zuul/zuul
          - zuul/nodepool
          - ansible/ansible
          - openstack/openstacksdk

Zuul Startup

Read config file

Zuul Startup

Read config file
Ask mergers for branches of each repo

images/startup1.ans

Zuul Startup

Read config file
Ask mergers for branches of each repo
Ask mergers for .zuul.yaml for each branch

of each repo

images/startup2.ans

When .zuul.yaml Changes

Zuul looks for changes to .zuul.yaml
Asks mergers for updated content
Splices into configuration used for that change
Works with cross-repo dependencies

("This change depends on a change to the job definition")

Zuul Architecture

Zuul is comprised of several services (mostly python3)

zuul-scheduler

zuul-executor

zuul-merger

zuul-web

zuul-fingergw

zuul-dashboard (javascript/react)

zuul-proxy (c++)

nodepool-launcher

nodepool-builder

RDBMS

Gearman

Zookeeper

Where Does Job Content Run?

Nodepool

A separate service that works very closely with Zuul

Zuul requires Nodepool but Nodepool can be used independently

Creates and destroys zero or more node resources

Resources can include VMs, Containers, COE contexts or Bare Metals

Static driver for allocating pre-existing nodes to jobs

Optionally periodically builds images and uploads to clouds

Nodepool Launcher

Where build nodes should come from

OpenStack

Static

Kubernetes Pod

Kubernetes Namespace

AWS

In work / coming soon:

Azure

GCE

What about job content?

Written in Ansible

Ansible is excellent at running one or more tasks in one or more places

The answer to "how do I" is almost always "Ansible"

Logging

Zuul doesn't know about logging - only about URLs to report
Job Ansible content does something with logs
Job content tells Zuul what URL to report to user
Allows for some really cool patterns

Live Web Preview

For static websites, publish built content and report url

https://review.openstack.org/#/c/648838 http://logs.openstack.org/38/648838/7/check/zuul-build-dashboard/96253e1/npm/html/

Jobs

Define node types needed from nodepool
Define which ansible playbooks to run
Jobs may be defined centrally or in the repo being tested
Jobs have contextual variants that simplify configuration
Jobs definitions support inheritance

Important Links

Questions

images/questions.ans

How do you use this thing?

tilt

Configuration

Job

- job:
    name: base
    parent: null
    description: |
      The base job for Zuul.
    timeout: 1800
    nodeset:
      nodes:
        - name: primary
          label: ubuntu-xenial
    pre-run: playbooks/base/pre.yaml
    post-run:
      - playbooks/base/post-ssh.yaml
      - playbooks/base/post-logs.yaml
    secrets:
      - site_logs

Simple Job

- job:
   name: tox
   pre-run: playbooks/setup-tox.yaml
   run: playbooks/tox.yaml
   post-run: playbooks/fetch-tox-output.yaml

Simple Job Inheritance

- job:
    name: tox-py36
    parent: tox
    vars:
      tox_envlist: py36

Inheritance Works Like An Onion

pre-run playbooks run in order of inheritance

run playbook of job runs

post-run playbooks run in reverse order of inheritance

If pre-run playbooks fail, job is re-tried

All post-run playbooks run - as far as pre-run playbooks got

Inheritance Example

For tox-py36 job

base pre-run playbooks/base/pre.yaml

tox pre-run playbooks/setup-tox.yaml

tox run playbooks/tox.yaml

tox post-run playbooks/fetch-tox-output.yaml

base post-run playbooks/base/post-ssh.yaml

base post-run playbooks/base/post-logs.yaml

Simple Job Variant

- job:
    name: tox-py27
    branches: stable/mitaka
    nodeset:
      - name: primary
        label: ubuntu-trusty

Nodesets for Multi-node Jobs

- nodeset:
    name: ceph-cluster
    nodes:
      - name: controller
        label: centos-7
      - name: compute1
        label: fedora-28
      - name: compute2
        label: fedora-28
    groups:
      - name: ceph-osd
        nodes:
          - controller
      - name: ceph-monitor
        nodes:
          - controller
          - compute1
          - compute2

Multi-node Job

nodesets are provided to Ansible for jobs in inventory

- job:
    name: ceph-multinode
    nodeset: ceph-cluster
    run: playbooks/install-ceph.yaml

Multi-node Ceph Job Content

- hosts: all
  roles:
    - install-ceph

- hosts: ceph-osd
  roles:
    - start-ceph-osd

- hosts: ceph-monitor
  roles:
    - start-ceph-monitor

- hosts: all
  roles:
    - do-something-interesting

Projects

Projects are git repositories
Specify a set of jobs for each pipeline
golang git repo naming as been adopted:

zuul@ubuntu-xenial:~$ find /home/zuul/src -mindepth 3 -maxdepth 3 -type d
/home/zuul/src/git.openstack.org/openstack-infra/shade
/home/zuul/src/git.openstack.org/openstack/keystoneauth
/home/zuul/src/git.openstack.org/openstack/os-client-config
/home/zuul/src/github.com/ansible/ansible

Project Config

Specify a set of jobs for each pipeline

- project:
    check:
      jobs:
        - openstack-tox-py27
        - openstack-tox-py35
        - openstack-tox-docs
    gate:
      jobs:
        - openstack-tox-py27
        - openstack-tox-py35
        - openstack-tox-docs

Project with Local Variant

- project:
    check:
      jobs:
        - openstack-tox-py27
        - openstack-tox-py35
        - openstack-tox-py36:
            voting: false
        - openstack-tox-docs
    gate:
      jobs:
        - openstack-tox-py27
        - openstack-tox-py35
        - openstack-tox-docs

Project with More Local Variants

- project:
    check:
      jobs:
        - openstack-tox-py27
        - openstack-tox-py35
        - openstack-tox-py36:
            voting: false
        - openstack-tox-docs:
            files: '^docs/.*$'

Project with Many Local Variants

- project:
    check:
      jobs:
        - openstack-tox-py27:
           nodeset:
             - name: centos-7
               label: centos-7
        - openstack-tox-py27:
            branches: stable/newton
            nodeset:
              - name: ubuntu-trusty
                label: ubuntu-trusty
        - openstack-tox-py35
        - openstack-tox-py36:
            voting: false
        - openstack-tox-docs:
            files: '^docs/.*$'

Project With Central and Local Config

# In git.openstack.org/openstack-infra/project-config:
- project:
    name: openstack/nova
    templates:
      - openstack-tox-jobs

# In git.openstack.org/openstack/nova/.zuul.yaml:
- project:
    check:
      - nova-placement-functional-devstack

Project with Job Dependencies

- project:
    release:
      jobs:
        - build-artifacts
        - upload-tarball:
            dependencies: build-artifacts
        - upload-pypi:
            dependencies: build-artifacts
        - notify-mirror:
            dependencies:
              - upload-tarball
              - upload-pypi

Playbooks

Jobs run playbooks
Playbooks may be defined centrally or in the repo being tested
Playbooks can use roles from current or other Zuul repos or Galaxy
Playbooks are not allowed to execute content on 'localhost'

devstack-tempest Run Playbook

# Changes that run through devstack-tempest are likely to have an impact on
# the devstack part of the job, so we keep devstack in the main play to
# avoid zuul retrying on legitimate failures.
- hosts: all
  roles:
    - run-devstack

# We run tests only on one node, regardless how many nodes are in the system
- hosts: tempest
  roles:
    - setup-tempest-run-dir
    - setup-tempest-data-dir
    - acl-devstack-files
    - run-tempest

Simple Shell Playbook

hosts: controller
roles:
 - shell: |
     cd {{ zuul.project.src_dir }}
     ./run_tests.sh

Test Like Production

If you use Ansible for deployment, your test and deployment processes and playbooks are the same

What if you don't use Ansible?

OpenStack Infra Control Plane uses Puppet (for now)

# In git.openstack.org/openstack-infra/project-config/roles/legacy-install-afs-with-puppet/tasks/main.yaml
- name: Install puppet
  shell: ./install_puppet.sh
  args:
    chdir: "{{ ansible_user_dir }}/src/git.openstack.org/openstack-infra/system-config"
  environment:
    # Skip setting up pip, our images have already done this.
    SETUP_PIP: "false"
  become: yes

- name: Copy manifest
  copy:
    src: manifest.pp
    dest: "{{ ansible_user_dir }}/manifest.pp"

- name: Run puppet
  puppet:
    manifest: "{{ ansible_user_dir }}/manifest.pp"
  become: yes

Secrets

Inspired by Kubernetes Secrets API
Projects can add named encrypted secrets to their .zuul.yaml file
Jobs can request to use secrets by name
Jobs using secrets are not reconfigured speculatively
Secrets can only be used by the same project they are defined in
Public key per project: {{ zuul_url }}/{{ tenant }}/{{ project }}.pub

::: GET https://zuul.openstack.org/openstack-infra/shade.pub

Secret Example (note, no admins had to enable this)

# In git.openstack.org/openstack/loci/.zuul.yaml:
- secret:
    name: loci_docker_login
    data:
      user: loci-username
      password: !encrypted/pkcs1-oaep
        - gUEX4eY3JAk/Xt7Evmf/hF7xr6HpNRXTibZjrKTbmI4QYHlzEBrBbHey27Pt/eYvKKeKw
          hk8MDQ4rNX7ZK1v+CKTilUfOf4AkKYbe6JFDd4z+zIZ2PAA7ZedO5FY/OnqrG7nhLvQHE
          5nQrYwmxRp4O8eU5qG1dSrM9X+bzri8UnsI7URjqmEsIvlUqtybQKB9qQXT4d6mOeaKGE
          5h6Ydkb9Zdi4Qh+GpCGDYwHZKu1mBgVK5M1G6NFMy1DYz+4NJNkTRe9J+0TmWhQ/KZSqo
          4ck0x7Tb0Nr7hQzV8SxlwkaCTLDzvbiqmsJPLmzXY2jry6QsaRCpthS01vnj47itoZ/7p
          taH9CoJ0Gl7AkaxsrDSVjWSjatTQpsy1ub2fuzWHH4ASJFCiu83Lb2xwYts++r8ZSn+mA
          hbEs0GzPI6dIWg0u7aUsRWMOB4A+6t2IOJibVYwmwkG8TjHRXxVCLH5sY+i3MR+NicR9T
          IZFdY/AyH6vt5uHLQDU35+5n91pUG3F2lyiY5aeMOvBL05p27GTMuixR5ZoHcvSoHHtCq
          7Wnk21iHqmv/UnEzqUfXZOque9YP386RBWkshrHd0x3OHUfBK/WrpivxvIGBzGwMr2qAj
          /AhJsfDXKBBbhGOGk1u5oBLjeC4SRnAcIVh1+RWzR4/cAhOuy2EcbzxaGb6VTM=

Secret Example

# In git.openstack.org/openstack/loci/.zuul.yaml:
- job:
    name: publish-loci-cinder
    parent: loci-cinder
    post-run: playbooks/push
    secrets:
      - loci_docker_login

# In git.openstack.org/openstack/loci/playbooks/push.yaml:
- hosts: all
  tasks:
    - include_vars: vars.yaml

- name: Push project to DockerHub
  block:
    - command: docker login -u {{ loci_docker_login.user }} -p {{ loci_docker_login.password }}
      no_log: True
    - command: docker push openstackloci/{{ project }}:{{ branch }}-{{ item.name }}
      with_items: "{{ distros }}"

Important Links

Questions

images/questions.ans

Presentty

pan

Presentty

Console presentations written in reStructuredText
Cross-fade, pan, tilt, cut transitions
Figlet, cowsay!
https://pypi.python.org/pypi/presentty

25 KiB Raw Blame History

Test Slide

Preshow

Zuul

Red Hat

OpenStack

OpenDev

Zuul

Ansible

What Zuul does

OpenStack Is

Federated

Impact of being Federated

Distributed

Impact of being Distributed

Large numbers of

Not Bragging About Scale

OpenStack Scale Comparison

OpenStack Scale Comparison

Impact of scale

Four Opens

Impact of Four Opens

We're Not Alone

Developer Process In a Nutshell

Developer Workflow

Gerrit

Zuul is not New

Not just for OpenStack

Zuul in a nutshell

All in Service of Gating

No Tests / Manual Tests

Periodic Testing

Post-Merge Testing

Pre-Review Testing

Gating

Mix and Match

Multi-repository integration

To test proposed future state

Speculative Execution

Nearest Non-Failing Change

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Zuul Simulation

Explicit Cross-Project Dependencies

Lock Step Changes

Live Configuration Changes

Zuul Startup

Zuul Startup

Zuul Startup

When .zuul.yaml Changes

Zuul Architecture

Where Does Job Content Run?

Nodepool

Nodepool Launcher

What about job content?

Logging

Live Web Preview

Jobs

Important Links

Questions

25 KiB

Raw Blame History