inaugust
/
inaugust.com
Code Issues Proposed changes
inaugust.com/src/zuulv3/zuul.rst

22 KiB
Raw Blame History

. display in 68x24 .. display in 88x24

dissolve

Test Slide

images/testslide.ans

Preshow

images/cursor.ans images/cursor2.ans

Zuul

images/title.ans

Red Hat

images/redhat.ans

OpenStack

images/openstack.ans

OpenStack Infra

"most insane CI infrastructure I've ever been a part of"

  -- Alex Gaynor

"OpenStack Infra are like the SpaceX of CI"

  -- Emily Dunham

Zuul

images/zuul.ans

Ansible

images/ansible.ans

Presentation Checklist

[X] Logos

Spoilers

  • What Zuul does
    • multiple repositories
    • integrated deliverable
    • gated commits
    • open tooling
    • nobody is special
    • testing like deployment

OpenStack Is

  • Federated
  • Distributed
  • Large
  • Open
  • Not Alone

Federated

  • Hundreds of involved companies
  • No 'main' company
  • "Decisions are made by those who show up"
  • Union of priorities/use cases

Impact of being Federated

  • No company can appoint people to positions in the project
  • The project cannot fire anyone
  • Variable background of contributors
  • Heavy reliance on consensus

Distributed

  • There is no office
  • Contributor base is global
  • Multitude of contributor backgrounds

Impact of being Distributed

  • Tooling must empower all contributors, regardless of background, skill level or cultural context
  • Heavy preference for text-based communication
  • Cannot assume US-centric needs or solutions

Large numbers of

  • Contributors (~2k in any given 6 month period)
  • Changes
  • Code Repositories (1955 as of this morning)

Not Bragging About Scale

OpenStack Scale Comparison

  • 2KJPH (2,000 jobs per hour)
  • Build Nodes from 13 Regions of 5 Public and 2 Private OpenStack Clouds
  • Rackspace, Internap, OVH, Vexxhost, CityCloud and Linaro, Limestone
  • 10,000 changes merged per month

OpenStack Scale Comparison

  • 2KJPH (2,000 jobs per hour)
  • Build Nodes from 13 Regions of 5 Public and 2 Private OpenStack Clouds
  • Rackspace, Internap, OVH, Vexxhost, CityCloud and Linaro, Limestone
  • 10,000 changes merged per month
    • By comparison, our friends at the amazing project Ansible received 13,000 changes and had merged 8,000 of them in its first 4 years.

Four Opens

  • Open Source (we don't hold back Enterprise features, we don't cripple things)
  • Open Design (design process open to all, decisions are not made inside company doors)
  • Open Development (public source code, public code review, all code is reviewed and gated)
  • Open Community (lazy consensus, democratic leadership from participants, public logged meetings in IRC, public archived mailing lists)

We're Not Alone

  • Dependencies (libvirt/kvm/xen, mysql/pg, rabbit, python/javascript, ceph/gluster, ansible/salt/puppet/chef, ovs/odl)
  • Adjacencies (kubernetes, ansible, terraform, opnfv, spinnaker)
  • Vendors (plugins, products, services, distros)

Developer Process In a Nutshell

  • Code Review - nobody has direct commit/push access
  • 3rd-Party CI for vendors
  • Gated Commits

OpenStack Developer Workflow

  • Who has submitted a patch?
  • Who wants to?
  • (Who is here because the name of this talk is weird?)
Hack             Review              Test
=========         ==========         ==========

        push              approve
   +-------------+    +-------------+
   |             |    |             |
+------+--+       +--v----+--+       +--v-------+
|         |       |          |       |          |
| $EDITOR |       |  Gerrit  |       |   Zuul   |
|         |       |          |       |          |
+------^--+       +--+----^--+       +--+-------+
   |             |    |             |
   +-------------+    +-------------+
        clone              merge

Gerrit

explain patch upload, zuul runs, test results displayed in gerrit this is all the interface to zuul users need to see

switch to actual gertty screenshot

also show zuul status page

but zuul is doing a lot of work behind the scenes, and if you look closer, this is what you see

images/color-gertty.ans

Zuul Architecture

images/architecture.ans

Presentation Checklist

[x] Logos
[x] Architecture diagram

Nodepool

  • A separate program that works very closely with Zuul

  • Builds images daily and uploads to clouds

  • Creates and destroys (at least) a vm for every job

    (Remember that 2,000 jobs per hour number?)

Zuul is not New

  • Has been in Production for OpenStack for Six Years
  • Zuul v3 first release where not-OpenStack is first-class use case
  • Zuul is now a top-level effort of OpenStack Foundation

Not just for OpenStack

  • Zuul is in production for OpenStack (in OpenStack VMs)

Also running at:

  • BMW (control plane in OpenShift)
  • Easystack
  • GoDaddy (control plane in Kubernetes)
  • OpenContrail
  • OpenLab
  • Red Hat
  • others ...

Zuul in a nutshell

  • Listens for code events
  • Prepares appropriate job config and git repo states
  • Allocates nodes for test jobs
  • Pushes git repo states to nodes
  • Runs user-defined Ansible playbooks
  • Collects/reports results
  • Potentially merges change

All in Service of Gating

Gating

Every change proposed for a repository is tested before it merges.

Co-gating

Changes to a set of repositories merge monotonically such that each change is tested with the current state of all the other related repositories before it merges.

Parallel Co-gating

Changes are serialized such that each change is tested with all of the changes ahead of it to satisfy the co-gating requirement while being able to run tests for multiple changes simultaneously.

Presentation Checklist

[x] Logos
[x] Architecture diagram
[x] Cows

Zuul Simulation

pan

  • todo

images/zsim-00.ans

Zuul Simulation

cut

  • todo

images/zsim-01.ans

Zuul Simulation

cut

  • todo

images/zsim-02.ans

Zuul Simulation

cut

  • todo

images/zsim-03.ans

Zuul Simulation

cut

  • todo

images/zsim-04.ans

Zuul Simulation

cut

  • todo

images/zsim-05.ans

Zuul Simulation

cut

  • todo

images/zsim-06.ans

Zuul Simulation

cut

  • todo

images/zsim-07.ans

Zuul Simulation

cut

  • todo

images/zsim-08.ans

Zuul Simulation

cut

  • todo

images/zsim-09.ans

Zuul Simulation

cut

  • todo

images/zsim-10.ans

Zuul Simulation

cut

  • todo

images/zsim-11.ans

Zuul Simulation

cut

  • todo

images/zsim-12.ans

Zuul Simulation

cut

  • todo

images/zsim-13.ans

Zuul Simulation

cut

  • todo

images/zsim-14.ans

Zuul Simulation

cut

  • todo

images/zsim-15.ans

Zuul Simulation

cut

  • todo

images/zsim-16.ans

Zuul Simulation

cut

  • todo

images/zsim-17.ans

Zuul Simulation

cut

  • todo

images/zsim-18.ans

Zuul Simulation

cut

  • todo

images/zsim-19.ans

Zuul Simulation

cut

  • todo

images/zsim-20.ans

Zuul Simulation

cut

  • todo

images/zsim-21.ans

Zuul Simulation

cut

  • todo

images/zsim-22.ans

Cross-Project Dependencies

Testing or gating dependencies manually specified by developers

Live Configuration Changes

Zuul is a distributed system, with a distributed configuration.

- tenant:
    name: openstack
    source:
      gerrit:
        config-repos:
          - openstack-infra/project-config
        project-repos:
          - openstack/nova
          - openstack/keystone
          - openstack-infra/devstack-gate

Zuul Startup

  • Read config file

Zuul Startup

  • Read config file
  • Ask mergers for branches of each repo

images/startup1.ans

Zuul Startup

  • Read config file

  • Ask mergers for branches of each repo

  • Ask mergers for .zuul.yaml for each branch

    of each repo

images/startup2.ans

When .zuul.yaml Changes

  • Zuul looks for changes to .zuul.yaml

  • Asks mergers for updated content

  • Splices into configuration used for that change

  • Works with cross-repo dependencies

    ("This change depends on a change to the job definition")

How do you use this thing?

tilt

Configuration

Pipelines

  • A process definition that connects git repositories, jobs, and reporting mechanisms.
  • A context to fix a set of jobs to each project.

Check Pipeline

- pipeline:
    name: check
    manager: independent
    source: gerrit
    trigger:
      gerrit:
        - event: patchset-created
        - event: change-restored
    success:
      gerrit:
        verified: 1

Gate Pipeline

- pipeline:
    name: gate
    manager: dependent
    source: gerrit
    trigger:
      gerrit:
        - event: comment-added
          approval:
          - workflow: 1
    success:
      gerrit:
        verified: 2
        submit: true

Jobs

  • Jobs run on nodes from nodepool (static or dynamic)
  • Metadata defined in Zuul's configuration
  • Execution content in Ansible
  • Jobs may be defined centrally or in the repo being tested
  • Jobs have contextual variants that simplify configuration

Job

- job:
    name: base
    parent: null
    description: |
      The base job for Zuul.
    timeout: 1800
    nodeset:
      nodes:
        - name: primary
          label: centos-7
    pre-run: playbooks/base/pre.yaml
    post-run:
      - playbooks/base/post-ssh.yaml
      - playbooks/base/post-logs.yaml
    secrets:
      - site_logs

Simple Job

- job:
   name: tox
   pre-run: playbooks/setup-tox.yaml
   run: playbooks/tox.yaml
   post-run: playbooks/fetch-tox-output.yaml

Simple Job Inheritance

- job:
    name: tox-py36
    parent: tox
    vars:
      tox_envlist: py36

Inheritance Works Like An Onion

  • pre-run playbooks run in order of inheritance
  • run playbook of job runs
  • post-run playbooks run in reverse order of inheritance
  • If pre-run playbooks fail, job is re-tried
  • All post-run playbooks run - as far as pre-run playbooks got

Inheritance Example

For tox-py36 job

  • base pre-run playbooks/base/pre.yaml
  • tox pre-run playbooks/setup-tox.yaml
  • tox run playbooks/tox.yaml
  • tox post-run playbooks/fetch-tox-output.yaml
  • base post-run playbooks/base/post-ssh.yaml
  • base post-run playbooks/base/post-logs.yaml

Simple Job Variant

- job:
    name: tox-py27
    branches: stable/mitaka
    nodeset:
      - name: ubuntu-trusty
        label: ubuntu-trusty

Nodesets for Multi-node Jobs

- nodeset:
    name: ceph-cluster
    nodes:
      - name: controller
        label: centos-7
      - name: compute1
        label: fedora-28
      - name: compute2
        label: fedora-28
    groups:
      - name: ceph-osd
        nodes:
          - controller
      - name: ceph-monitor
        nodes:
          - controller
          - compute1
          - compute2

Multi-node Job

  • nodesets are provided to Ansible for jobs in inventory
- job:
    name: ceph-multinode
    nodeset: ceph-cluster
    run: playbooks/install-ceph.yaml

Multi-node Ceph Job Content

- hosts: all
  roles:
    - install-ceph

- hosts: ceph-osd
  roles:
    - start-ceph-osd

- hosts: ceph-monitor
  roles:
    - start-ceph-monitor

- hosts: all
  roles:
    - do-something-interesting

Projects

  • Projects are git repositories
  • Specify a set of jobs for each pipeline
  • golang git repo naming as been adopted:
zuul@ubuntu-xenial:~$ find /home/zuul/src -mindepth 3 -maxdepth 3 -type d
/home/zuul/src/git.openstack.org/openstack-infra/shade
/home/zuul/src/git.openstack.org/openstack/keystoneauth
/home/zuul/src/git.openstack.org/openstack/os-client-config
/home/zuul/src/github.com/ansible/ansible

Project Config

  • Specify a set of jobs for each pipeline
- project:
    check:
      jobs:
        - openstack-tox-py27
        - openstack-tox-py35
        - openstack-tox-docs
    gate:
      jobs:
        - openstack-tox-py27
        - openstack-tox-py35
        - openstack-tox-docs

Project with Local Variant

- project:
    check:
      jobs:
        - openstack-tox-py27
        - openstack-tox-py35
        - openstack-tox-py36:
            voting: false
        - openstack-tox-docs
    gate:
      jobs:
        - openstack-tox-py27
        - openstack-tox-py35
        - openstack-tox-docs

Project with More Local Variants

- project:
    check:
      jobs:
        - openstack-tox-py27
        - openstack-tox-py35
        - openstack-tox-py36:
            voting: false
        - openstack-tox-docs:
            files: '^docs/.*$'

Project with Many Local Variants

- project:
    check:
      jobs:
        - openstack-tox-py27:
           nodeset:
             - name: centos-7
               label: centos-7
        - openstack-tox-py27:
            branches: stable/newton
            nodeset:
              - name: ubuntu-trusty
                label: ubuntu-trusty
        - openstack-tox-py35
        - openstack-tox-py36:
            voting: false
        - openstack-tox-docs:
            files: '^docs/.*$'

Project With Central and Local Config

# In git.openstack.org/openstack-infra/project-config:
- project:
    name: openstack/nova
    templates:
      - openstack-tox-jobs
# In git.openstack.org/openstack/nova/.zuul.yaml:
- project:
    check:
      - nova-placement-functional-devstack

Project with Job Dependencies

- project:
    release:
      jobs:
        - build-artifacts
        - upload-tarball:
            dependencies: build-artifacts
        - upload-pypi:
            dependencies: build-artifacts
        - notify-mirror:
            dependencies:
              - upload-tarball
              - upload-pypi

Playbooks

  • Jobs run playbooks
  • Playbooks may be defined centrally or in the repo being tested
  • Playbooks can use roles from current or other Zuul repos or Galaxy
  • Playbooks are not allowed to execute content on 'localhost'

devstack-tempest Run Playbook

# Changes that run through devstack-tempest are likely to have an impact on
# the devstack part of the job, so we keep devstack in the main play to
# avoid zuul retrying on legitimate failures.
- hosts: all
  roles:
    - run-devstack

# We run tests only on one node, regardless how many nodes are in the system
- hosts: tempest
  roles:
    - setup-tempest-run-dir
    - setup-tempest-data-dir
    - acl-devstack-files
    - run-tempest

Simple Shell Playbook

hosts: controller
roles:
 - shell: |
     cd {{ zuul.project.src_dir }}
     ./run_tests.sh

Test Like Production

If you use Ansible for deployment, your test and deployment processes and playbooks are the same

What if you don't use Ansible?

OpenStack Infra Control Plane uses Puppet (for now)

# In git.openstack.org/openstack-infra/project-config/roles/legacy-install-afs-with-puppet/tasks/main.yaml
- name: Install puppet
  shell: ./install_puppet.sh
  args:
    chdir: "{{ ansible_user_dir }}/src/git.openstack.org/openstack-infra/system-config"
  environment:
    # Skip setting up pip, our images have already done this.
    SETUP_PIP: "false"
  become: yes

- name: Copy manifest
  copy:
    src: manifest.pp
    dest: "{{ ansible_user_dir }}/manifest.pp"

- name: Run puppet
  puppet:
    manifest: "{{ ansible_user_dir }}/manifest.pp"
  become: yes

Secrets

  • Inspired by Kubernetes Secrets API
  • Projects can add named encrypted secrets to their .zuul.yaml file
  • Jobs can request to use secrets by name
  • Jobs using secrets are not reconfigured speculatively
  • Secrets can only be used by the same project they are defined in
  • Public key per project: {{ zuul_url }}/{{ tenant }}/{{ project }}.pub
::

GET https://zuul.openstack.org/openstack-infra/shade.pub

Secret Example (note, no admins had to enable this)

# In git.openstack.org/openstack/loci/.zuul.yaml:
- secret:
    name: loci_docker_login
    data:
      user: loci-username
      password: !encrypted/pkcs1-oaep
        - gUEX4eY3JAk/Xt7Evmf/hF7xr6HpNRXTibZjrKTbmI4QYHlzEBrBbHey27Pt/eYvKKeKw
          hk8MDQ4rNX7ZK1v+CKTilUfOf4AkKYbe6JFDd4z+zIZ2PAA7ZedO5FY/OnqrG7nhLvQHE
          5nQrYwmxRp4O8eU5qG1dSrM9X+bzri8UnsI7URjqmEsIvlUqtybQKB9qQXT4d6mOeaKGE
          5h6Ydkb9Zdi4Qh+GpCGDYwHZKu1mBgVK5M1G6NFMy1DYz+4NJNkTRe9J+0TmWhQ/KZSqo
          4ck0x7Tb0Nr7hQzV8SxlwkaCTLDzvbiqmsJPLmzXY2jry6QsaRCpthS01vnj47itoZ/7p
          taH9CoJ0Gl7AkaxsrDSVjWSjatTQpsy1ub2fuzWHH4ASJFCiu83Lb2xwYts++r8ZSn+mA
          hbEs0GzPI6dIWg0u7aUsRWMOB4A+6t2IOJibVYwmwkG8TjHRXxVCLH5sY+i3MR+NicR9T
          IZFdY/AyH6vt5uHLQDU35+5n91pUG3F2lyiY5aeMOvBL05p27GTMuixR5ZoHcvSoHHtCq
          7Wnk21iHqmv/UnEzqUfXZOque9YP386RBWkshrHd0x3OHUfBK/WrpivxvIGBzGwMr2qAj
          /AhJsfDXKBBbhGOGk1u5oBLjeC4SRnAcIVh1+RWzR4/cAhOuy2EcbzxaGb6VTM=

Secret Example

# In git.openstack.org/openstack/loci/.zuul.yaml:
- job:
    name: publish-loci-cinder
    parent: loci-cinder
    post-run: playbooks/push
    secrets:
      - loci_docker_login

# In git.openstack.org/openstack/loci/playbooks/push.yaml:
- hosts: all
  tasks:
    - include_vars: vars.yaml

- name: Push project to DockerHub
  block:
    - command: docker login -u {{ loci_docker_login.user }} -p {{ loci_docker_login.password }}
      no_log: True
    - command: docker push openstackloci/{{ project }}:{{ branch }}-{{ item.name }}
      with_items: "{{ distros }}"

Important Links

Questions

images/questions.ans

Presentty

pan

Presentty