inaugust
/
inaugust.com
Code Issues Proposed changes

Add getting-started talk

This commit is contained in:
Monty Taylor 2018-10-24 06:10:20 +01:00
parent 63d06e732b
commit 6dddf88024
No known key found for this signature in database
GPG Key ID: 7BAE94BC7141A594
1 changed files with 989 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

989
src/zuulv3/getting-started.rst Normal file
View File

@ -0,0 +1,989 @@
. display in 68x24
.. display in 88x24
.. pygments yaml? (only file breaks (---) tinted)
.. slide on high level v3 changes
.. slide on nodepool
.. transition:: dissolve
:duration: 0.4
Test Slide
==========
.. hidetitle::
.. ansi:: images/testslide.ans
Preshow
=======
.. hidetitle::
.. ansi:: images/cursor.ans images/cursor2.ans
Zuul
====
.. hidetitle::
.. ansi:: images/title.ans
Red Hat
=======
.. hidetitle::
.. container:: handout
i work for
.. ansi:: images/redhat.ans
Ansible
=======
.. hidetitle::
.. ansi:: images/ansible.ans
Zuul
====
.. hidetitle::
.. ansi:: images/zuul.ans
What Zuul Does
==============
* gated changes
* one or more git repositories
* integrated deliverable
* testing like deployment
Underlying Philosophy
=====================
* All changes flow through code review
* Changes only land if they pass all tests
* Computers are cheaper than humans
Ramifications of Philosophy
===========================
* No direct push access for anyone
* Software should be installable from source
* Testing should be automated and repeatable
* Developers write tests with their patches
* Code always works
Getting to Gating
=================
No Tests / Manual Tests
=======================
* No test automation exists or ...
* Developer runs test suite before pushing code
* Prone to developer skipping tests for "trivial" changes
* Doesn't scale organizationally
Periodic Testing
================
* Developers push changes directly to shared branch
* CI system runs tests from time to time - report if things still work
* "Who broke the build?"
* Leads to hacks like NVIE model
Post-Merge Testing
==================
* Developers push changes directly to shared branch
* CI system is triggered by push - reports if push broke something
* Frequently batched / rolled up
* Easier to diagnose which change broke things
* Reactive - the bad changes are already in
Pre-review Testing
==================
* Changes are pushed to code review (Gerrit Change, GitHub PR, etc)
* CI system is triggered by code review change creation
* Test results inform review decisions
* Proactive - testing code before it lands
* Reviewers can get bored waiting for tests
* Only tests code as written, not potential result of merging code
Gating
======
* Changes are pushed to code review
* CI system is triggered by code review approval
* CI system merges code IFF tests pass
* Proactive - testing code before it lands
* Future state resulting from merge of code is tested
* Reviewers can fire-and-forget safely
Mix and Match
=============
* Zuul supports all of those modes
* Zuul users frequently combine them
* Run pre-review (check) and gating (gate) on each change
* Post-merge/post-tag for release/publication automation
* Periodic for catching bitrot
Multi-repository integration
============================
* Multiple source repositories are needed for deliverable
* Future state to be tested is the future state of all involved repos
To test proposed future state
=============================
* Get tip of each project. Merge appropriate change. Test.
* Changes must be serialized, otherwise state under test is invalid.
* Integrated deliverable repos share serialized queue
Speculative Execution
=====================
* Correct parallel processing of serialized future states
* Create virtual serial queue of changes for each deliverable
* Assume each change will pass its tests
* Test successive changes with previous changes applied to starting state
Nearest Non-Failing Change
==========================
(aka 'The Jim Blair Algorithm')
* If a change fails, move it aside
* Cancel all test jobs behind it in the queue
* Reparent queue items on the nearest non-failing change
* Restart tests with new state
Zuul Simulation
===============
.. transition:: pan
.. container:: handout
* todo
.. ansi:: images/zsim-00.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-01.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-02.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-03.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-04.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-05.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-06.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-07.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-08.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-09.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-10.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-11.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-12.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-13.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-14.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-15.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-16.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-17.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-18.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-19.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-20.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-21.ans
Zuul Simulation
===============
.. transition:: cut
.. container:: handout
* todo
.. ansi:: images/zsim-22.ans
Cross-Project Dependencies
==========================
Testing or gating dependencies manually specified by developers
.. container:: progressive
* nodepool https://review.openstack.org/612168
Make functional src jobs actually install from source
* openstacksdk https://review.openstack.org/612186
Don't start task managers passed in to Connection
Depends-On: https://review.openstack.org/612168
* openstacksdk https://review.openstack.org/604521
Add support for per-service rate limits
(git parent is 612186)
* nodepool https://review.openstack.org/612169
Consume rate limiting task manager from openstacksdk
Depends-On: https://review.openstack.org/604521
(nodepool-functional-py35-src should pass, but
nodepool-functional-py35 should not fail until openstacksdk release)
Live Configuration Changes
==========================
.. container:: handout
Zuul is a distributed system, with a distributed configuration.
.. code:: yaml
- tenant:
name: openstack
source:
gerrit:
config-repos:
- openstack-infra/project-config
project-repos:
- openstack/nova
- openstack/keystone
- openstack-infra/devstack-gate
Zuul Startup
============
* Read config file
Zuul Startup
============
* Read config file
* Ask mergers for branches of each repo
.. ansi:: images/startup1.ans
Zuul Startup
============
* Read config file
* Ask mergers for branches of each repo
* Ask mergers for .zuul.yaml for each branch
of each repo
.. ansi:: images/startup2.ans
When .zuul.yaml Changes
=======================
.. container:: progressive
* Zuul looks for changes to .zuul.yaml
* Asks mergers for updated content
* Splices into configuration used for that change
* Works with cross-repo dependencies
("This change depends on a change to the job definition")
Zuul Architecture
=================
.. ansi:: images/architecture.ans
Nodepool
========
* A separate program that works very closely with *Zuul*
* Creates and destroys zero or more node resources
* Resources can include VMs, Containers, COE contexts or Bare Metals
* Static driver for allocating pre-existing nodes to jobs
* Optionally periodically builds images and uploads to clouds
Nodepool Launcher
=================
Where build nodes should come from
* OpenStack
* Static
In review:
* Kubernetes
* OpenShift
* AWS
In work / coming soon:
* Azure
* GCE
* Mac Stadium
What about test/job content?
============================
* Written in Ansible
* Ansible is excellent at running one or more tasks in one or more places
* The answer to "how do I" is almost always "Ansible"
What Zuul Does
==============
* Listens for code events
* Prepares appropriate job config and git repo states
* Allocates nodes for test jobs
* Pushes git repo states to nodes
* Runs user-defined Ansible playbooks
* Collects/reports results
* Potentially merges change
OpenStack Infra - Largest Known Zuul
====================================
* 2KJPH (2,000 jobs per hour)
* Build Nodes from 16 Regions of 5 Public and 3 Private OpenStack Clouds
* Rackspace, Internap, OVH, Vexxhost, CityCloud
* Linaro, Limestone, Packethost
* 10,000 changes merged per month
Zuul is not New
===============
* Has been in Production for OpenStack for Six Years
* Zuul is now a top-level effort of OpenStack Foundation
* Zuul is in production for OpenStack (Control Plane in OpenStack VMs)
* Zuul v3 first release where not-OpenStack is first-class use case
Not just for OpenStack
======================
* BMW (control plane in OpenShift)
* GoDaddy (control plane in Kubernetes)
* Easystack
* OpenContrail
* OpenLab
* Red Hat
* others ...
Code Review Systems
===================
* Gerrit
* GitHub (Public and Enterprise)
In work / coming soon:
* GitLab
* Bitbucket
Support for non-git
===================
.. container:: progressive
* Nope
* helix4git *may* work for perforce, but is untested
Installation of Software
========================
Ways to Install Zuul
====================
* Windmill: http://git.openstack.org/cgit/openstack/windmill
* Software Factory: https://softwarefactory-project.io/
* Puppet: http://git.openstack.org/cgit/openstack-infra/puppet-zuul
* Containers: https://hub.docker.com/_/zuul/
Zuul Containers
===============
* Published on every commit
* Application/Process containers
* Config / Data should be bind-mounted in
zuul/zuul-executor
==================
* In k8s, zuul-executor must be run privileged
* Uses bubblewrap for unprivileged sanboxing
* Restriction may be lifted in the future
Release Management
==================
* Zuul is a CI system
* C stands for "Continuous"
* It is run Continuously Delivered and Deployed upstream
* Releases are tagged from code run upstream
* There is no intent to have a 'stable' release
* 'stable' is a synonym for "old and buggy"
zuul/zuul-scheduler
===================
* SPOF
* We're working on it
* Recommend running scheduler from tags
Quick Start
===========
https://zuul-ci.org/docs/zuul/admin/quick-start.html
Important Links
===============
* https://zuul-ci.org/
* https://git.zuul-ci.org/cgit/zuul
* https://zuul-ci.org/docs/zuul
* https://zuul-ci.org/docs/zuul-jobs/
* https://docs.openstack.org/infra/openstack-zuul-jobs/
* freenode:#zuul
Questions
=========
.. ansi:: images/questions.ans
Quick Start Prereq
==================
* Install docker, docker-compose, git-review
Debian/Ubuntu:
::
sudo apt-get install docker-compose git git-review
RHEL / CentOS / Fedora:
::
sudo yum install docker docker-compose git git-review
OpenSuse:
::
sudo zypper install docker docker-compose git git-review
RHEL / CentOS / Fedora / OpenSuse
::
sudo systemctl enable docker.service
sudo systemctl start docker.service
Actual Quick Start
==================
* git clone https://git.zuul-ci.org/zuul
* cd zuul
* cd doc/source/admin/examples
* docker-compose up
What's Running
==============
* Zookeeper
* Gerrit
* Nodepool Launcher
* Zuul Scheduler
* Zuul Web Server
* Zuul Executor
* Apache HTTPD
* A container to use as a 'static' build node
How they're connected
=====================
* End Users talk to Gerrit and Apache HTTPD
* Zuul Scheduler talks to Gerrit
* Nodepool Launcher, Zuul Scheduler, Zuul Web talk to Zookeeper
* Zuul Executor talks to Zuul Scheduler (using Gearman)
Initial provided config
=======================
* docker-compose has plumbed in basic config ``etc_zuul/zuul.conf``
and ``etc_zuul/main.yaml``
* Gerrit Connection named "gerrit"
* Zuul user for that connection
* Git connection named "zuul-ci.org" for ``zuul-jobs`` standard library
Initial tenant
==============
* Zuul is (always) multi-tenant
* Example config contains a tenant called ``example-tenant``
* Three projects in the ``example-tenant`` tenant:
``zuul-config``, ``test1``, ``test2``
* Three projects are also in gerrit ready to use
zuul.conf
=========
::
[gearman]
server=scheduler
[gearman_server]
start=true
[zookeeper]
hosts=zk
[scheduler]
tenant_config=/etc/zuul/main.yaml
[web]
listen_address=0.0.0.0
[executor]
private_key_file=/var/ssh/nodepool
default_username=root
zuul.conf part 2
================
::
[connection "gerrit"]
name=gerrit
driver=gerrit
server=gerrit
sshkey=/var/ssh/zuul
user=zuul
password=secret
baseurl=http://gerrit:8080
auth_type=basic
[connection "zuul-ci.org"]
name=zuul-ci
driver=git
baseurl=https://git.zuul-ci.org/
main.yaml
=========
::
- tenant:
name: example-tenant
source:
gerrit:
config-projects:
- zuul-config
untrusted-projects:
- test1
- test2
zuul-ci.org:
untrusted-projects:
- zuul-jobs:
include:
- job
Gerrit Account
==============
* Need a user account to interact with Gerrit
* Gerrit is configured in dev mode - no passwords required
* Visit http://localhost:8080
* Click "Become"
* Click "New Account"
* Click "Register"
* Enter Full Name
* Click "Save Changes"
* Enter username in Username field (match your local laptop user)
* Copy ``~/.ssh/id_rsa.pub`` contents into SSH Key field
* Click Continue
Config Repo
===========
* ``zuul-config`` is a trusted ``config-repo``
* Security and functionality of system depend on this repo
* Limit its contents to minimum required
Config Files vs. Directories
============================
* Zuul reads config from:
``.zuul.yaml``, ``zuul.yaml``, ``zuul.d`` or ``.zuul.d``
* For projects with substantial zuul config, like ``zuul-config``
``zuul.d`` directory is likely best.
* The directories are read run-parts style.
* Recommended practice is splitting by type of object
Setting up Gating
=================
* We want to have changes to ``zuul-config`` be gated
* We need to define pipelines: ``check`` and ``gate``
* Need to attach ``zuul-config`` to them
* Start with builtin ``noop`` job (always return success)
* Use regex to attach all projects to ``check`` and ``gate``
Pipeline Definitions
====================
* Zuul has no built-in workflow definitions, let's add ``check`` and ``gate``
check pipeline
==============
::
- pipeline:
name: check
description: |
Newly uploaded patchsets enter this pipeline to receive an
initial +/-1 Verified vote.
manager: independent
require:
gerrit:
open: True
current-patchset: True
trigger:
gerrit:
- event: patchset-created
- event: change-restored
success:
gerrit:
Verified: 1
failure:
gerrit:
Verified: -1
gate pipeline
=============
::
- pipeline:
name: gate
description: |
Changes that have been approved are enqueued in order in this
pipeline, and if they pass tests, will be merged.
manager: dependent
post-review: True
require:
gerrit:
open: True
current-patchset: True
approval:
- Workflow: 1
trigger:
gerrit:
- event: comment-added
approval:
- Workflow: 1
start:
gerrit:
Verified: 0
success:
gerrit:
Verified: 2
submit: true
failure:
gerrit:
Verified: -2
Add the pipeline definitions
============================
.. code-block:: bash
git clone http://localhost:8080/zuul-config
cd zuul-config
mkdir zuul.d
cp ../examples/zuul-config/zuul.d/pipelines.yaml .
Shared Project Pipeline Definition
==================================
In ``examples/zuul-config/zuul.d/projects.yaml``
.. code-block:: yaml
- project:
name: ^.*$
check:
jobs: []
gate:
jobs: []
- project:
name: zuul-config
check:
jobs:
- noop
gate:
jobs:
- noop
Attach the projects to the pipelines
====================================
.. code-block:: bash
cp ../examples/zuul-config/zuul.d/projects.yaml .
Commit the changes and push up for review
=========================================
.. code-block:: bash
git add zuul.d
git commit
git review
Force merging bootstrap config
==============================
* Zuul is running with no config, so it won't do anything
* For this change (and this change only) we will bypass gating
Reviewing normally
==================
* visit http://localhost:8080/#/c/zuul-config/+/1001/
* click reply
* vote +2 Code Review +1 Approved
Verified +2 is Missing
======================
Verified +2 is what we have zuul configured to do.
::
success:
gerrit:
Verified: 2
submit: true
Bypassing Gating
================
* visit http://localhost:8080/
* click 'switch account'
* click 'admin'
* visit http://localhost:8080/#/c/zuul-config/+/1001/
* click reply
* vote +2 Verified (normal users do not see this)
* click submit (normal users do not see this)
* click 'switch account'
* click your username
Base Job
========
* Every Zuul installation must define a ``base`` job
* Push git repos to build node
* Publish logs/artifacts
* Any local specific setup
* Goes in config repo - because it impacts EVERY job
Add Base Job to zuul-config
===========================
::
cp ../examples/zuul-config/zuul.d/jobs.yaml .
git add jobs.yaml
git commit
git review
Then go to http://localhost:8080/#/c/zuul-config/+/1002/ and approve it
Zuul should merge the patch
===========================
zuul-config is configured to use the ``noop`` job
Zuul tests syntax automatically
===============================
* Edit jobs.yaml
* Change ``parent: null`` to ``parent: broken``
* git commit ; git review
* Check out the review in gerrit ... there should be errors!
Presentty
=========
.. hidetitle::
.. transition:: pan
.. figlet:: Presentty
* Console presentations written in reStructuredText
* Cross-fade, pan, tilt, cut transitions
* https://pypi.python.org/pypi/presentty