. display in 68x24 .. display in 88x24 .. pygments yaml? (only file breaks (---) tinted) .. slide on high level v3 changes .. slide on nodepool .. transition:: dissolve :duration: 0.4 Test Slide ========== .. hidetitle:: .. ansi:: images/testslide.ans Preshow ======= .. hidetitle:: .. ansi:: images/cursor.ans images/cursor2.ans Zuul ==== .. hidetitle:: .. ansi:: images/title.ans Important Links =============== * https://opendev.org/zuul * https://zuul-ci.org/docs/zuul * https://zuul-ci.org/docs/zuul-jobs/ * https://docs.openstack.org/infra/openstack-zuul-jobs/ * freenode:#zuul * https://opendev.org/inaugust/inaugust.com/src/branch/master/src/zuulv3/tutorial.rst Overview ======== * Discussion of concepts * Installation of software * Configurting Zuul * Writing jobs Please Ask Questions ==================== Installation of Software ======================== Ways to Install Zuul ==================== * Windmill: https://opendev.org/windmill/windmill/ * Software Factory: https://softwarefactory-project.io/ * Puppet: https://opendev.org/opendev/puppet-zuul/ * Containers: https://hub.docker.com/_/zuul/ Today we'll be using Containers with docker-compose =================================================== Getting Started =============== While we talk about other things ... * Install docker, docker-compose, git-review Debian/Ubuntu: :: apt-get install docker-compose git git-review Red Hat / SuSE :: yum install docker-compose git git-review * mkdir -p ~/src/opendev.org/zuul * cd ~/src/opendev.org/zuul * git clone https://opendev.org/zuul/zuul * cd zuul * cd doc/source/admin/examples * docker-compose up Output in docker-compose window =============================== * All services running with debug logging to stdout * Tons of information will have been output - including some errors * Zuul connects to Gerrit before it's fully configured * As it becomes configured, Zuul notices and becomes happy * Once happy, it should stablize and become idle We'll come back to this ======================= It's going to do a bunch of network - and this is a conference. Red Hat ======= .. hidetitle:: .. container:: handout i work for .. ansi:: images/redhat.ans OpenStack ========= .. hidetitle:: .. ansi:: images/openstack.ans OpenDev ======= :: "most insane CI infrastructure I've ever been a part of" -- Alex Gaynor "OpenStack Infra are like the SpaceX of CI" -- Emily Dunham Zuul ==== .. hidetitle:: .. ansi:: images/zuul.ans What Zuul does ============== * "Speculative Future State" * multiple repositories * integrated deliverable * gated commits * testing like deployment Our Use Case ============ OpenStack Is ============ * Federated * Distributed * Large Federated ========= * Hundreds of involved companies * Hundreds of sub-projects * "One" deliverable * Union of priorities/use cases * "Decisions are made by those who show up" Impact of being Federated ========================= * No company can appoint people to positions in the project * The project cannot fire anyone * Heavy reliance on consensus * CI system doesn't assume anyone is "in charge" Distributed =========== * There is no office * Contributor base is global * Multitude of contributor backgrounds Impact of being Distributed =========================== * Tooling must empower all contributors, regardless of background, skill level or cultural context * Heavy preference for text-based communication * Cannot assume US-centric needs or solutions Sound like anybody's Day Job? ============================= * Multiple silos * Competing management chains * Junior/Senior Devs, "Architects", Security Teams, Frontend/Backend * Spread across locations * "One" product Large numbers of ================ * Contributors (\~2k in any given 6 month period) * Changes * Code Repositories (2139 as of this morning) Not Bragging About Scale ======================== OpenStack Scale Comparison ========================== * 2KJPH (2,000 jobs per hour) * Build Nodes from 16 Regions of 5 Public and 4 Private OpenStack Clouds * Rackspace, Internap, OVH, Vexxhost, CityCloud * Linaro (ARM), Limestone, Packethost, Fortnebula * 10,000 changes merged per month OpenStack Scale Comparison ========================== * 2KJPH (2,000 jobs per hour) * Build Nodes from 16 Regions of 5 Public and 4 Private OpenStack Clouds * Rackspace, Internap, OVH, Vexxhost, CityCloud * Linaro (ARM), Limestone, Packethost, Fortnebula * 10,000 changes merged per month * By comparison, our friends at the amazing project Ansible received 13,000 changes and had merged 8,000 of them in its first 4 years. Impact of scale =============== * Empower teams to take care of themselves (distributed) * Efficiency gained from shared solutions (centralized) * Empower teams to do what they want (distributed) * Enforce common standards or requirements (centralized) * Zuul supports per-repo config, central config, and multiple tenants One Zuul install is all you need for an entire Enterprise ========================================================= One Zuul install is all you need for an entire Enterprise ========================================================= * 7 "admins" run Zuul for OpenDev supporting 2500+ devs * None are 100% full-time on Zuul * Team could handle another 2500+ devs Who Is Running Zuul? ==================== * Zuul is in production for OpenStack for 7 years (in OpenStack VMs) Also running at: * BMW (control plane in OpenShift) * GoDaddy (control plane in Kubernetes) * GoodMoney (control plane in EKS, adding GKE) * Le Bon Coin * Volvo * Western Digital * Easystack * TungstenFabric * Huawei OpenLab * IBM * Red Hat * others ... Developer Process In a Nutshell =============================== * Code Review - nobody has direct commit/push access * Gated Commits - nobody has submit permission * Every change gated on Code Analysis, Unit Tests, Functional Tests and End to End Integration Tests * Run all tests (at least) twice: * on patchset upload * between change approval and merge OpenStack Developer Workflow ============================ .. container:: handout * Who has submitted a patch? * Who wants to? * (Who is here because the name of this talk is weird?) :: Hack Review Test ========= ========== ========== push approve +-------------+ +-------------+ | | | | +------+--+ +--v----+--+ +--v-------+ | | | | | | | $EDITOR | | Gerrit | | Zuul | | | | | | | +------^--+ +--+----^--+ +--+-------+ | | | | +-------------+ +-------------+ clone merge Gerrit ====== .. hidetitle:: .. container:: handout explain patch upload, zuul runs, test results displayed in gerrit this is all the interface to zuul users need to see switch to actual gertty screenshot also show zuul status page but zuul is doing a lot of work behind the scenes, and if you look closer, this is what you see .. ansi:: images/color-gertty.ans Zuul in a nutshell ================== * Listens for code events * Prepares appropriate job config and git repo states * Allocates nodes for test jobs * Pushes git repo states to nodes * Runs user-defined Ansible playbooks * Collects/reports results * Potentially merges change All in Service of Gating ======================== No Tests / Manual Tests ======================= * No test automation exists or ... * Developer runs test suite before pushing code * Prone to developer skipping tests for "trivial" changes * Doesn't scale organizationally Periodic Testing ================ * Developers push changes directly to shared branch * CI system runs tests from time to time - report if things still work * "Who broke the build?" * Leads to hacks like NVIE model Post-Merge Testing ================== * Developers push changes directly to shared branch * CI system is triggered by push - reports if push broke something * Frequently batched / rolled up * Easier to diagnose which change broke things * Reactive - the bad changes are already in Pre-Review Testing ================== * Changes are pushed to code review (Gerrit Change, GitHub PR, etc) * CI system is triggered by code review change creation * Test results inform review decisions * Proactive - testing code before it lands * Reviewers can get bored waiting for tests * Only tests code as written, not potential result of merging code Gating ====== * Changes are pushed to code review * Gating system is triggered by code review approval * Gating system merges code IFF tests pass * Proactive - testing code before it lands * Future state resulting from merge of code is tested * Reviewers can fire-and-forget safely Mix and Match ============= * Zuul supports all of those modes * Zuul users frequently combine them * Run pre-review (check) and gating (gate) on each change * Post-merge/post-tag for release/publication automation * Periodic for catching bitrot Pipelines in Zuul ================= * In other systems, a "pipeline" is per-job - triger, content, report * In Zuul, a Pipeline describes a shared general workflow for changes * Triggers * Requirements * Reporters * Job *content* is separate from triggering/reporting * Projects add Jobs to Pipelines * Same Job can be used in multiple Pipelines - and by multiple Projects Check Jobs ========== * Run on patchset upload * Verify patch as written * Avoid wasting reviewer time on broken changes check pipeline ============== .. code:: yaml - pipeline: name: check manager: independent precedence: low require: gerrit: open: True current-patchset: True trigger: gerrit: - event: patchset-created - event: change-restored - event: comment-added comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck - event: comment-added require-approval: - Verified: [-1, -2] username: zuul approval: - Workflow: 1 success: gerrit: Verified: 1 failure: gerrit: Verified: -1 Gate Triggering in Gerrit ========================= * Gate jobs run between Code Review Approval and Merging * "Workflow" Label in Gerrit * Approvers have ability to vote +1 in Workflow * Nobody sees the Submit button * Zuul runs Gate jobs on Workflow+1 - Merges on Success gate pipeline ============= .. code:: yaml - pipeline: name: gate manager: dependent post-review: True require: gerrit: open: True current-patchset: True approval: - Workflow: 1 trigger: gerrit: - event: comment-added approval: - Workflow: 1 start: gerrit: Verified: 0 success: gerrit: Verified: 2 submit: true failure: gerrit: Verified: -2 Multi-repository integration ============================ * Multiple source repositories are needed for deliverable * Future state to be tested is the future state of all involved repos To test proposed future state ============================= * Get tip of each project. Merge appropriate change(s). Test. * Changes must be serialized, otherwise state under test is invalid. * Integrated deliverable repos share serialized queue Speculative Execution ===================== * Correct parallel processing of serialized future states * Create virtual serial queue of changes for each deliverable * Assume each change will pass its tests * Test successive changes with previous changes applied to starting state Nearest Non-Failing Change ========================== (aka 'The Jim Blair Algorithm') * If a change fails, move it aside * Cancel all test jobs behind it in the queue * Reparent queue items on the nearest non-failing change * Restart tests with new state Zuul Simulation =============== .. transition:: pan .. container:: handout * todo .. ansi:: images/zsim-00.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-01.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-02.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-03.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-04.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-05.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-06.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-07.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-08.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-09.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-10.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-11.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-12.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-13.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-14.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-15.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-16.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-17.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-18.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-19.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-20.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-21.ans Zuul Simulation =============== .. transition:: cut .. container:: handout * todo .. ansi:: images/zsim-22.ans Explicit Cross-Project Dependencies =================================== * Developers can mark changes as being dependent * Depends-On: footer - in commit or PR * Zuul uses depends-on when constructing virtual serial queue * Will not merge changes in gate before depends-on changes * Works cross-repo AND cross-source Sources ======= * Gerrit * Github * Pagure * Git * Bitbucket (in-review) - https://review.opendev.org/#/c/657837 * Gitlab (in-development) - https://review.opendev.org/#/c/685990/ Cross Source ============ .. code:: yaml trigger: gerrit: - event: patchset-created - event: change-restored - event: comment-added comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck github: - event: pull_request action: - opened - changed - reopened - event: pull_request action: comment comment: (?i)^\s*recheck\s*$ start: github: status: pending comment: false success: gerrit: Verified: 1 github: status: 'success' failure: gerrit: Verified: -1 github: status: 'failure' Cross Source ============ * Explicit Dependency between projects from different sources * Change to Zuul that depends on change to Gerrit * https://review.opendev.org/#/c/680778/ :: commit 737d61c116ff5f32770ef72e2dd82a031ab32591 Author: James E. Blair Date: Mon Aug 19 14:58:20 2020 -0700 Add Support for Gerrit Checks Plugin Depends-On: https://gerrit-review.googlesource.com/c/plugins/checks/+/232079 Change-Id: I8e5903f4429c5a1273a6120e0d09c57169e8f938 Lock Step Changes ================= * Circular Dependencies are not supported on purpose * Rolling upgrades across interdependent services * HOWEVER - many valid use cases (go/rust/c++) - support expected Configuring Zuul ================ * Zuul Manages Git Repos * Zuul Jobs are configured in Git Repos * Zuul Job Content is self-testing Projects ======== * A "Project" is a git repo Config Project vs. Untrusted Project ==================================== config project * special project containing project admin content * has access to features that are normally restricted * job changes are not applied speculatively untrusted project * most projects * some actions (like executing code on localhost) are blocked * job changes are applied speculatively Config Files vs. Directories ============================ * Zuul reads in-repo config from: ``.zuul.yaml``, ``zuul.yaml``, ``zuul.d`` or ``.zuul.d`` * For projects with substantial zuul config, like ``zuul-config`` ``zuul.d`` directory is likely best. * The directories are read run-parts style. * Recommended practice is splitting by type of object Live Configuration Changes ========================== .. container:: handout Zuul is a distributed system, with a distributed configuration. Zuul's config is in git - but something has to tell it where the repos are .. code:: yaml - tenant: name: openstack source: gerrit: config-projects: - opendev/project-config untrusted-projects: - zuul/zuul-jobs - zuul/zuul - zuul/nodepool - openstack/openstacksdk github: untrusted-projects: - include: [] projects: - ansible/ansible Zuul Startup ============ * Read config file (main.yaml) Zuul Startup ============ * Read config file * Ask mergers for branches of each repo .. ansi:: images/startup1.ans Zuul Startup ============ * Read config file * Ask mergers for branches of each repo * Ask mergers for .zuul.yaml for each branch of each repo .. ansi:: images/startup2.ans When .zuul.yaml Changes ======================= .. container:: progressive * Zuul looks for changes to .zuul.yaml * Asks mergers for updated content * Splices into configuration used for that change * Works with cross-repo dependencies ("This change depends on a change to the job definition") Zuul Architecture ================= Zuul is comprised of several services (mostly python3) * zuul-scheduler * zuul-executor * zuul-merger * zuul-web * zuul-fingergw * zuul-dashboard (javascript/react) * zuul-proxy (c++) * nodepool-launcher * nodepool-builder * RDBMS * Gearman * Zookeeper * zuul-registry (coming soon) Where Does Job Content Run? =========================== Nodepool ======== * A separate service that works very closely with *Zuul* * *Zuul* requires *Nodepool* but *Nodepool* can be used independently * Creates and destroys zero or more node resources * Resources can include VMs, Containers, COE contexts or Bare Metals * Static driver for allocating pre-existing nodes to jobs * Optionally periodically builds images and uploads to clouds (Remember that 2,000 jobs per hour number?) Nodepool Launcher ================= Where build nodes should come from * OpenStack * Static * Kubernetes Pod * Kubernetes Namespace * AWS In work / coming soon: * Azure * GCE Nodepool Builder ================ Optionally periodically build and upload new base images * OpenStack How do you use this thing? ========================== .. transition:: tilt .. hidetitle:: .. figlet:: Configuration Human Roles =========== * Deployer * Project Admin * End User Deployer Config =============== Zuul: Connection, Triggers, Reporters Nodepool: Launcher and Builder Config Connection Plugins ================== Describes how Zuul connects to external systems * Gerrit * Github * Pagure * git * mqtt * smtp * sql Trigger Plugins =============== Input to Zuul for use in causing jobs to run * Gerrit * Github * Pagure * git * zuul Reporter Plugins ================ Where Zuul should send information about jobs * Gerrit * Github * Pagure * mqtt * smtp * sql Admin Role ========== * What sources, triggers and reporters * What projects Zuul manages * What node labels are available * *Everything* else is Job Content Demo Installation using docker-compose ====================================== Remember this? * Install docker, docker-compose, git-review Debian/Ubuntu: :: apt-get install docker-compose git git-review Red Hat / SuSE :: yum install docker-compose git git-review * mkdir -p ~/src/opendev.org/zuul * cd ~/src/opendev.org/zuul * git clone https://opendev.org/zuul/zuul * cd zuul * cd doc/source/admin/examples * docker-compose up What's Running ============== * Zookeeper * Gerrit * Nodepool Launcher * Zuul Scheduler * Zuul Web Server * Zuul Executor * Apache HTTPD * A container to use as a 'static' build node How they're connected ===================== * End Users talk to Gerrit and Apache HTTPD * Zuul Scheduler talks to Gerrit * Nodepool Launcher, Zuul Scheduler, Zuul Web talk to Zookeeper * Zuul Executor talks to Zuul Scheduler (using Gearman) Initial provided config ======================= * docker-compose has plumbed in basic config ``etc_zuul/zuul.conf`` and ``etc_zuul/main.yaml`` * Gerrit Connection named "gerrit" * Zuul user for that connection * Git connection named "opendev.org" for ``zuul-jobs`` standard library Initial tenant ============== * Zuul is (always) multi-tenant * Example config contains a tenant called ``example-tenant`` * Three projects in the ``example-tenant`` tenant: ``zuul-config``, ``test1``, ``test2`` * Three projects are also in gerrit ready to use zuul.conf ========= :: [gearman] server=scheduler [gearman_server] start=true [zookeeper] hosts=zk [scheduler] tenant_config=/etc/zuul/main.yaml [web] listen_address=0.0.0.0 [executor] private_key_file=/var/ssh/nodepool default_username=root zuul.conf part 2 ================ :: [connection "gerrit"] name=gerrit driver=gerrit server=gerrit sshkey=/var/ssh/zuul user=zuul password=secret baseurl=http://gerrit:8080 auth_type=basic [connection "zuul-ci.org"] name=zuul-ci driver=git baseurl=https://opendev.org/ main.yaml ========= :: - tenant: name: example-tenant source: gerrit: config-projects: - zuul-config untrusted-projects: - test1 - test2 zuul-ci.org: untrusted-projects: - zuul-jobs: include: - job Running Dashboard ================= http://localhost:9000 Jobs ==== * Define node types needed from nodepool * Define which ansible playbooks to run * Jobs may be defined centrally or in the repo being tested * Jobs have contextual variants that simplify configuration * Jobs definitions support inheritance Job Libraries ============= * Zuul jobs are all defined in git repositories * Designed to be directly shared across zuul installations * Standard library: https://opendev.org/zuul/zuul-jobs * Zuul installs should add ``zuul-jobs`` to their config * As changes land in ``zuul-jobs`` - Zuul installs will get them automatically Local Job Libraries =================== * Jobs specific to local project, but meant to be shared * https://opendev.org/openstack/openstack-zuul-jobs * https://github.com/ansible/zuul-jobs * https://opendev.org/openstack/devstack (here be dragons) Simple Job ========== .. code:: yaml - job: name: tox pre-run: playbooks/setup-tox.yaml run: playbooks/tox.yaml post-run: playbooks/fetch-tox-output.yaml What about job content? ======================= * Written in Ansible * Ansible is excellent at running one or more tasks in one or more places * The answer to "how do I" is almost always "Ansible" Playbooks ========= * Jobs run playbooks * Allocated nodes from nodepool are provided in Ansible Inventory * Playbooks may be defined centrally or in the repo being tested * Playbooks can use roles from current or other Zuul repos * Playbooks are not allowed to execute content on 'localhost' * *Note*: localhost restriction may go away soon Simple Playbook Example ======================= * playbooks/tox.yaml .. code:: yaml - hosts: ubuntu-bionic tasks: - name: Run tox args: chdir: "{{ zuul_work_dir }}" shell: "tox -e{{ tox_envlist }}" Job Inheritance =============== All Zuuls have a base job .. code:: yaml - job: name: base parent: null description: | The base job for Zuul. timeout: 1800 nodeset: nodes: - name: primary label: ubuntu-bionic pre-run: playbooks/base/pre.yaml post-run: - playbooks/base/post-ssh.yaml - playbooks/base/post-logs.yaml secrets: - site_logs Logging ======= * Zuul doesn't know about logging - only about URLs to report * Job Ansible content does *something* with logs * Job content tells Zuul what URL(s) to report to user * Allows for some really cool patterns Live Web Preview ================ * For static websites, publish built content and report url https://review.opendev.org/#/c/635716/ https://zuul.opendev.org/t/zuul/build/2ef4ceb232a2467ab56be726f31928ef https://470ff2d7584ecc192d95-20fcf28638b4f38abf523a12606af55d.ssl.cf1.rackcdn.com/635716/12/check/zuul-build-dashboard/2ef4ceb/npm/html Project Admin Role ================== * Base job(s) * Pre-playbooks for local environment * Post-playbooks - where do logs and artifacts go? * Shared trusted publication jobs Simple Job Inheritance ====================== .. code:: yaml - job: name: tox-py36 parent: tox vars: tox_envlist: py36 Inheritance Works Like An Onion =============================== * pre-run playbooks run in order of inheritance * run playbook of job runs * post-run playbooks run in reverse order of inheritance * If pre-run playbooks fail, job is re-tried * All post-run playbooks run - as far as pre-run playbooks got Inheritance Example =================== For tox-py36 job * base pre-run playbooks/base/pre.yaml * tox pre-run playbooks/setup-tox.yaml * tox run playbooks/tox.yaml * tox post-run playbooks/fetch-tox-output.yaml * base post-run playbooks/base/post-ssh.yaml * base post-run playbooks/base/post-logs.yaml Simple Job Variant ================== .. code:: yaml - job: name: tox-py36 branches: stable/mitaka nodeset: - name: ubuntu-trusty label: ubuntu-trusty Nodesets for Multi-node Jobs ============================ .. code:: yaml - nodeset: name: ceph-cluster nodes: - name: controller label: centos-7 - name: compute1 label: fedora-28 - name: compute2 label: fedora-28 groups: - name: ceph-osd nodes: - controller - name: ceph-monitor nodes: - controller - compute1 - compute2 Multi-node Job ============== * nodesets are provided to Ansible for jobs in inventory .. code:: yaml - job: name: ceph-multinode nodeset: ceph-cluster run: playbooks/install-ceph.yaml Multi-node Ceph Job Content =========================== .. code:: yaml - hosts: all roles: - install-ceph - hosts: ceph-osd roles: - start-ceph-osd - hosts: ceph-monitor roles: - start-ceph-monitor - hosts: all roles: - do-something-interesting Projects ======== * Projects are git repositories * Specify a set of jobs for each pipeline * golang git repo naming as been adopted: :: zuul@ubuntu-bionic:~$ find /home/zuul/src -mindepth 3 -maxdepth 3 -type d /home/zuul/src/opendev.org/openstack/openstacksdk /home/zuul/src/opendev.org/zuul/zuul /home/zuul/src/github.com/ansible/ansible /home/zuul/src/gerrit.googlesource.com/gerrit Project Config ============== * Specify a set of jobs for each pipeline .. code:: yaml - project: check: jobs: - openstack-tox-py36 - openstack-tox-docs gate: jobs: - openstack-tox-py36 - openstack-tox-docs Project with Local Variant ========================== .. code:: yaml - project: check: jobs: - openstack-tox-py27 - openstack-tox-py35 - openstack-tox-py36: voting: false - openstack-tox-docs gate: jobs: - openstack-tox-py27 - openstack-tox-py35 - openstack-tox-docs Project with More Local Variants ================================ .. code:: yaml - project: check: jobs: - openstack-tox-py27 - openstack-tox-py35 - openstack-tox-py36: voting: false - openstack-tox-docs: files: '^docs/.*$' Project with Many Local Variants ================================ .. code:: yaml - project: check: jobs: - openstack-tox-py27: nodeset: - name: centos-7 label: centos-7 - openstack-tox-py27: branches: stable/newton nodeset: - name: ubuntu-trusty label: ubuntu-trusty - openstack-tox-py35 - openstack-tox-py36: voting: false - openstack-tox-docs: files: '^docs/.*$' Project With Central and Local Config ===================================== .. code:: yaml # In opendev.org/openstack/project-config: - project: name: openstack/nova templates: - openstack-tox-jobs .. code:: yaml # In opendev.org/openstack/nova/.zuul.yaml: - project: check: - nova-placement-functional-devstack Project with Job Dependencies ============================= Full DAG of Job Dependencies .. code:: yaml - project: release: jobs: - build-artifacts - upload-tarball: dependencies: build-artifacts - upload-pypi: dependencies: build-artifacts - notify-mirror: dependencies: - upload-tarball - upload-pypi Speculative Container Images ============================ * Build container images based on non-published images from other changes * Test built container images in gate jobs * Promote exact image from gate after merge * Soft Dependencies - use artifact from another job only if it was built * https://zuul-ci.org/docs/zuul-jobs/docker-image.html Playbooks ========= * Jobs run playbooks * Playbooks may be defined centrally or in the repo being tested * Playbooks can use roles from current or other Zuul repos or Galaxy * Playbooks are not allowed to execute content on 'localhost' Test Like Production ==================== If you use Ansible for deployment, your test and deployment processes and playbooks are the same What if you don't use Ansible? ============================== OpenStack Infra Control Plane uses Puppet (for now) =================================================== .. code:: yaml OpenDev Control Plane uses some Puppet (for now) ================================================ .. code:: yaml - name: Install puppet shell: ./install_puppet.sh args: chdir: "{{ ansible_user_dir }}/src/opendev.org/opendev/system-config" become: yes - name: Copy manifest copy: src: manifest.pp dest: "{{ ansible_user_dir }}/manifest.pp" - name: Run puppet puppet: manifest: "{{ ansible_user_dir }}/manifest.pp" become: yes Secrets ======= * Inspired by Kubernetes Secrets API * Projects can add named encrypted secrets to their .zuul.yaml file * Jobs can request to use secrets by name * Jobs using secrets are not reconfigured speculatively * Secrets can only be used by the same project they are defined in * Public key per project: ``{{ zuul_url }}/{{ tenant }}/{{ project }}.pub`` :: GET https://zuul.opendev.org/api/tenant/openstack/key/openstack/openstacksdk.pub Secret Example (note, no admins had to enable this) =================================================== .. code:: yaml # In opendev.org/openstack/loci/.zuul.yaml: - secret: name: loci_docker_login data: user: loci-username password: !encrypted/pkcs1-oaep - gUEX4eY3JAk/Xt7Evmf/hF7xr6HpNRXTibZjrKTbmI4QYHlzEBrBbHey27Pt/eYvKKeKw hk8MDQ4rNX7ZK1v+CKTilUfOf4AkKYbe6JFDd4z+zIZ2PAA7ZedO5FY/OnqrG7nhLvQHE 5nQrYwmxRp4O8eU5qG1dSrM9X+bzri8UnsI7URjqmEsIvlUqtybQKB9qQXT4d6mOeaKGE 5h6Ydkb9Zdi4Qh+GpCGDYwHZKu1mBgVK5M1G6NFMy1DYz+4NJNkTRe9J+0TmWhQ/KZSqo 4ck0x7Tb0Nr7hQzV8SxlwkaCTLDzvbiqmsJPLmzXY2jry6QsaRCpthS01vnj47itoZ/7p taH9CoJ0Gl7AkaxsrDSVjWSjatTQpsy1ub2fuzWHH4ASJFCiu83Lb2xwYts++r8ZSn+mA hbEs0GzPI6dIWg0u7aUsRWMOB4A+6t2IOJibVYwmwkG8TjHRXxVCLH5sY+i3MR+NicR9T IZFdY/AyH6vt5uHLQDU35+5n91pUG3F2lyiY5aeMOvBL05p27GTMuixR5ZoHcvSoHHtCq 7Wnk21iHqmv/UnEzqUfXZOque9YP386RBWkshrHd0x3OHUfBK/WrpivxvIGBzGwMr2qAj /AhJsfDXKBBbhGOGk1u5oBLjeC4SRnAcIVh1+RWzR4/cAhOuy2EcbzxaGb6VTM= Secret Example ============== .. code:: yaml # In opendev.org/openstack/loci/.zuul.yaml: - job: name: publish-loci-cinder parent: loci-cinder post-run: playbooks/push secrets: - loci_docker_login # In opendev.org/openstack/loci/playbooks/push.yaml: - hosts: all tasks: - include_vars: vars.yaml - name: Push project to DockerHub block: - command: docker login -u {{ loci_docker_login.user }} -p {{ loci_docker_login.password }} no_log: True - command: docker push openstackloci/{{ project }}:{{ branch }}-{{ item.name }} with_items: "{{ distros }}" Back to our Install =================== Zuul Containers =============== * Published on every commit * Application/Process containers * Built using Speculative Containers * Config / Data should be bind-mounted in Container Philosophy ==================== * As minimal as possible * OS inside of container does not matter zuul/zuul-executor ================== * In k8s, zuul-executor must be run in privileged pod * Uses bubblewrap for unprivileged sanboxing * Restriction may be lifted in the future Release Management ================== * Zuul is a CI system * C stands for "Continuous" * It is run Continuously Delivered and Deployed upstream * Releases are tagged from code run upstream * There is no intent to have a 'stable' release * 'stable' is a synonym for "old and buggy" * **NOTE** container images are not currently tagged zuul/zuul-scheduler =================== * SPOF * We're working on it * Recommend running scheduler from tags Let's Do Something with Our Zuul ================================ Gerrit Account ============== * Need a user account to interact with Gerrit * Gerrit is configured in dev mode - no passwords required * Visit http://localhost:8080 * Click "New Account" * Skip entering anything, click "settings" * Enter username in Username field (match your local laptop user) * Enter Full Name * Click "Save Changes" More Gerrit Account =================== * Scroll down to Email Addresses * Enter email address in "New email address" field * Click "Send Verification" * Gerrit is in Developer Mode, will not send email * Scroll down to "SSH Keys" * Copy ``~/.ssh/id_rsa.pub`` contents into "New SSH Key" field * Click "Add New SSH Key" * Reload the page in your browser Config Repo =========== * ``zuul-config`` is a trusted ``config-repo`` * Security and functionality of system depend on this repo * Limit its contents to minimum required Setting up Gating ================= * We want to have changes to ``zuul-config`` be gated * We need to define pipelines: ``check`` and ``gate`` * Need to attach ``zuul-config`` to them * Start with builtin ``noop`` job (always return success) * Use regex to attach all projects to ``check`` and ``gate`` Pipeline Definitions ==================== * Zuul has no built-in workflow definitions, let's add ``check`` and ``gate`` check pipeline ============== :: - pipeline: name: check description: | Newly uploaded patchsets enter this pipeline to receive an initial +/-1 Verified vote. manager: independent require: gerrit: open: True current-patchset: True trigger: gerrit: - event: patchset-created - event: change-restored comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck success: gerrit: Verified: 1 mysql: failure: gerrit: Verified: -1 mysql: gate pipeline ============= :: - pipeline: name: gate description: | Changes that have been approved are enqueued in order in this pipeline, and if they pass tests, will be merged. manager: dependent post-review: True require: gerrit: open: True current-patchset: True approval: - Workflow: 1 trigger: gerrit: - event: comment-added approval: - Workflow: 1 start: gerrit: Verified: 0 success: gerrit: Verified: 2 submit: true mysql: failure: gerrit: Verified: -2 mysql: Add the pipeline definitions ============================ .. code-block:: bash cd ~/src git clone http://localhost:8080/zuul-config cd zuul-config mkdir zuul.d cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/zuul-config/zuul.d/pipelines.yaml zuul.d Shared Project Pipeline Definition ================================== In ``examples/zuul-config/zuul.d/projects.yaml`` .. code-block:: yaml - project: name: ^.*$ check: jobs: [] gate: jobs: [] - project: name: zuul-config check: jobs: - noop gate: jobs: - noop Attach the projects to the pipelines ==================================== .. code-block:: bash cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/zuul-config/zuul.d/projects.yaml zuul.d Commit the changes and push up for review ========================================= .. code-block:: bash git add zuul.d git commit git review Force merging bootstrap config ============================== * Zuul is running with no config, so it won't do anything * For this change (and this change only) we will bypass gating Reviewing normally ================== * visit http://localhost:8080/#/c/zuul-config/+/1001/ * click reply * vote +2 Code Review +1 Approved Verified +2 is Missing ====================== Verified +2 is what we have zuul configured to do. :: success: gerrit: Verified: 2 submit: true Bypassing Gating ================ * visit http://localhost:8080/ * Click avatar in Upper Right * Click Sign Out * Click Sign In * Click 'admin' * Click on change you uploaded * Click Reply... * Vote +2 Verified (normal users do not see this) * Click submit (normal users do not see this) * Click avatar in Upper Right * Click Sign Out * Click Sign In * click your username Dashboard ========= http://localhost:9000/t/example-tenant/status Add Job to a repo ================= Clone the test1 repo ==================== :: cd ~/src git clone http://localhodsts:8080/test1 cd test1 Simple debug playbook ===================== .. code:: yaml - hosts: all tasks: - debug: msg: Hello world! Add the playbook to our repo ============================ :: mkdir playbooks cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/test1/playbooks/testjob.yaml playbooks git add playbooks/testjob.yaml Simple Zuul Job =============== :: - job: name: testjob run: playbooks/testjob.yaml - project: check: jobs: - testjob gate: jobs: - testjob Add it to the .zuul.yaml ======================== This is a normal repo, let's use .zuul.yaml :: cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/test1/zuul.yaml .zuul.yaml git add .zuul.yaml Submit the change for review ============================ .. code-block:: shell git add .zuul.yaml playbooks git commit -m "Add test Zuul job" git review Then check http://localhost:8080/dashboard/self Where are the logs? =================== Base Job ======== * Every Zuul installation must define a ``base`` job * Push git repos to build node * Publish logs/artifacts * Any local specific setup * Goes in config repo - because it impacts EVERY job Minimal Base Job ================ .. code:: yaml - job: name: base parent: null nodeset: nodes: - name: ubuntu-bionic label: ubuntu-bionic Minimal Base Job ================ :: cd ~/src/zuul-config cp ~/src/examples/zuul-config/zuul.d/jobs.yaml zuul.d Base pre Playbook ================= * Using roles from zuul-jobs repo * Creates a per-build ssh key * Copies git repos to test node .. code:: yaml - hosts: all roles: - add-build-sshkey - prepare-workspace Base pre Playbook ================= :: mkdir -p playbooks/base cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/zuul-config/playbooks/base/pre.yaml playbooks/base Base post-ssh playbook ====================== * Removes the per-build ssh key from the pre playbook .. code:: yaml - hosts: all roles: - remove-build-sshkey Base post-ssh playbook ====================== :: cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/zuul-config/playbooks/base/post-ssh playbooks/base Base post-logs playbook ======================= * Quickstart is running Apache server for log content * Volume mounted at /srv/static/logs (default for upload-logs) * Tells Zuul where the logs can be found .. code:: yaml - hosts: localhost gather_facts: False roles: - role: upload-logs zuul_log_url: "http://localhost:8000" Base post-ssh playbook ====================== :: cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/zuul-config/playbooks/base/post-logs playbooks/base Update Base Job =============== .. code:: yaml - job: name: base parent: null description: | The recommended base job. All jobs ultimately inherit from this. It runs a pre-playbook which copies all of the job's prepared git repos on to all of the nodes in the nodeset. It also sets a default timeout value (which may be overidden). pre-run: playbooks/base/pre.yaml post-run: - playbooks/base/post-ssh.yaml - playbooks/base/post-logs.yaml roles: - zuul: zuul/zuul-jobs timeout: 1800 nodeset: nodes: - name: ubuntu-bionic label: ubuntu-bionic Update Base Job =============== :: cp ~/src/examples/zuul-config/zuul.d/jobs2.yaml zuul.d Commit and Push for Review ========================== :: git add playbooks zuul.d/jobs.yaml git commit -m "Update Zuul base job" git review Land Change =========== * Visit http://localhost:8080/dashboard/self * Should be a Verified +1 vote from Zuul * Click Reply * Vote: Code-Review +2 and Workflow +1 * Click Send * Wait a bit, then reload, change should be merged Recheck the test1 change ======================== * Visit http://localhost:8080/dashboard/self * Click Reply " Comment "recheck" * Click Send Job Now Running with Updated Base Job ===================================== * Should get a link to log location * Check out Zuul Dashboard Zuul tests syntax automatically =============================== * Edit zuul/jobs.yaml * Change ``parent: null`` to ``parent: broken`` * git commit ; git review * Check out the review in gerrit ... there should be errors! Job configs are shared ====================== :: cd ~/src git clone http://localhost:8080/test2 cd test2 Add testjob to test2 repo ========================= :: cp ~/src/opendev.org/zuul/zuul/doc/source/admin/examples/test1/zuul.yaml .zuul.yaml git add .zuul.yaml git commit -a -m"Added testjob to test2 repo" git review Where's the job? ================ * Check http://localhost:8080/dashboard/self * Look at error from zuul * This is a job for cross-project depends! Add Depends-On footer ===================== :: git commit -a --amend * Add a blank line then :: Depends-On: http://localhost:8080/c/test1/+/1 * Then resubmit :: git review Let's Add a Secret ================== * Edit .zuul.yaml - name the secret and the field :: cd ~/src/test1 echo "my-secret" | ~/src/opendev.org/zuul/zuul/tools/encrypt_secret.py --tenant example-tenant http://localhost:9000 test1 >> .zuul.yaml Secret Name =========== .. code-block:: yaml - secret: name: amazing_secret data: name: !encrypted/pkcs1-oaep Add secret to job ================= .. code-block:: yaml - job: name: testjob run: playbooks/testjob.yaml secrets: - amazing_secret Print the secret in the job =========================== * *NOTE* This is dumb * Edit playbooks/testjob.yaml .. code-block:: yaml - hosts: all tasks: - debug: msg: "Hello {{ amazing_secret.name }}!" Commit and push for review ========================== :: git commit -a -m"Added a secret" git review Change test2 to depend on the secret change =========================================== :: git commit -a --amend * Change the Depends-On line :: Depends-On: http://localhost:8080/c/test1/+/2 * Then resubmit :: git review Let's go look at the outputs! ============================= Add a job with run-test-command =============================== * What if you don't want a playbook? * Role 'run-test-command' in zuul-jobs .. code-block:: yaml - job: parent: run-test-command name: simple-test-job vars: test_command: exit 0 Add to the project pipeline config ================================== .. code-block:: yaml - project: check: jobs: - testjob - simple-test-job gate: jobs: - testjob - simple-test-job Commit and Push Up ================== :: git commit -a -m"Added a simple test job" git review What's wrong? ============= Add required-projects ===================== * Edit .zuul.yaml .. code-block:: yaml - job: parent: run-test-command name: simple-test-job vars: test_command: "find {{ ansible_user_dir }}" required-projects: - test1 - test2 Important Links =============== * https://opendev.org/zuul * https://zuul-ci.org/docs/zuul * https://zuul-ci.org/docs/zuul-jobs/ * https://docs.openstack.org/infra/openstack-zuul-jobs/ * freenode:#zuul * https://opendev.org/inaugust/inaugust.com/src/branch/master/src/zuulv3/tutorial.rst