diff --git a/doc/source/devstack-gate.rst b/doc/source/devstack-gate.rst index df6e540b8d..15c974f31d 100644 --- a/doc/source/devstack-gate.rst +++ b/doc/source/devstack-gate.rst @@ -2,6 +2,12 @@ .. _devstack-gate: +.. warning:: + + devstack-gate is deprecated in favor of Zuul v3 native jobs defined + in the devstack repo itself. This document describes how devstack-gate + works as there are still legacy jobs using it. + Devstack Gate ############# @@ -43,10 +49,10 @@ How It Works The devstack test starts with an essentially bare virtual machine made available by :ref:`nodepool` and prepares the testing -environment. This is driven by the the devstack-gate repository which -holds several scripts that are run by Jenkins. +environment. This is driven by the devstack-gate repository which +holds several scripts that are run by Zuul. -When a proposed change is approved by the core reviewers, Jenkins +When a proposed change is approved by the core reviewers, Zuul triggers the devstack gate test itself. This job runs on one of the previously configured nodes and invokes the devstack-vm-gate-wrap.sh script which checks out code from all of the involved repositories, and @@ -54,7 +60,7 @@ merges the proposed change. That script then calls devstack-vm-gate.sh which installs a devstack configuration file, and invokes devstack. Once devstack is finished, it runs Tempest, which performs integration testing. After everything is done, devstack-gate copies and formats all of the logs -for archival. A jenkins jobs then copies these logs to the log archive. +for archival. Zuul then copies these logs to the log archive. How to Debug a Devstack Gate Failure ==================================== diff --git a/doc/source/gerrit.rst b/doc/source/gerrit.rst index d93e40bd19..f47ad49678 100644 --- a/doc/source/gerrit.rst +++ b/doc/source/gerrit.rst @@ -88,7 +88,7 @@ access to external testing tools on a sandbox project. The `Voting Third-Party CI` group is used to grant +/-1 Verified access to external testing tools for all projects. -The `Continuous Integration Tools` group contains Jenkins and any +The `Continuous Integration Tools` group contains Zuul and any other CI tools that get +2/-2 access on reviews. The `Release Managers` group is used for release managers. @@ -218,9 +218,10 @@ High level goals: #. Anonymous users can read all projects. #. All registered users can perform informational code review (+/-1) on any project. -#. Jenkins can perform verification (blocking or approving: +/-1). +#. Zuul can perform verification (blocking or approving: +/-2). +#. Third Party CI systems can perform informational verification (+/-1). #. All registered users can create changes. -#. The OpenStack Release Manager and Jenkins can tag releases (push +#. The OpenStack Release Manager and Zuul can tag releases (push annotated tags). #. Members of $PROJECT-core group can perform full code review (blocking or approving: +/- 2), and submit changes to be merged. diff --git a/doc/source/grafana.rst b/doc/source/grafana.rst index d5d34e1ba4..ab2bb8e9a4 100644 --- a/doc/source/grafana.rst +++ b/doc/source/grafana.rst @@ -7,7 +7,7 @@ Grafana Grafana is an open source, feature rich metrics dashboard and graph editor for Graphite, InfluxDB & OpenTSDB. OpenStack runs Graphite which stores all the -metrics related to Nodepool, Zuul and Jenkins (to name a few). +metrics related to Nodepool and Zuul (to name a few). At a Glance =========== diff --git a/doc/source/irc.rst b/doc/source/irc.rst index d77598e7d7..f8f63d505f 100644 --- a/doc/source/irc.rst +++ b/doc/source/irc.rst @@ -60,7 +60,7 @@ your channel. Join #openstack-infra if you have any trouble with any of these commands. NOTE: Channel admin should issue the access commands above BEFORE adding -channel to gerritbot and accessbot, otherwise Jenkins will fail tests. +channel to gerritbot and accessbot, otherwise Zuul will fail tests. Meetbot ======= diff --git a/doc/source/logstash.rst b/doc/source/logstash.rst index fc007758c7..83c4b75a17 100644 --- a/doc/source/logstash.rst +++ b/doc/source/logstash.rst @@ -22,6 +22,7 @@ At a Glance :Configuration: * :cgit_file:`modules/openstack_project/files/logstash` * :cgit_file:`modules/openstack_project/templates/logstash` + * `submit-logstash-jobs defaults`_ :Projects: * http://logstash.net/ * http://kibana.org/ @@ -36,7 +37,7 @@ At a Glance Overview ======== -Logs from Jenkins test runs are sent to logstash where they are +Logs from Zuul test runs are sent to logstash where they are indexed and stored. Logstash facilitates reviewing logs from multiple sources in a single test run, searching for errors or particular events within a test run, as well as searching for log event trends @@ -47,14 +48,9 @@ System Architecture There are four major layers in our Logstash setup. -1. Log Pusher Scripts. - Subscribes to the Jenkins ZeroMQ Event Publisher listening for build - finished events. When a build finishes an event is received from - Jenkins which is then converted into Gearman jobs specific to that - event for each log file we care about. These jobs trigger Gearman - workers that then fetch the logs generated by that build, chop them - up, annotate them with Jenkins build info and finally sends them to a - Logstash indexer process. +1. Submit Logstash Jobs. + The `logs post-playbook`_ in the Zuul ``base`` job submit logs defined + in the `submit-logstash-jobs defaults`_ to a Logstash Indexer. 2. Logstash Indexer. Reads these log events from the log pusher, filters them to remove unwanted lines, collapses multiline events together, and parses @@ -76,7 +72,7 @@ bottleneck very quickly. This looks something like: :: - jenkins + zuul post-logs playbook | | gearman-client --------------- @@ -98,14 +94,12 @@ bottleneck very quickly. This looks something like: Log Pusher ---------- -This is a pair of simple Python scripts. The first listens to Jenkins -build events and converts them into Gearman jobs and the second performs -Gearman jobs to push log files into logstash. +This is an ansible module in the `submit-log-processor-jobs role`_. It +submits Gearman jobs to push log files into logstash. Log pushing looks like this: -* Jenkins publishes build complete notifications. -* Receive notification from Jenkins and convert to Gearman jobs. +* Zuul runs post-playbook on job completion. * Using info in the Gearman job log files are retrieved. * Log files are processed then shipped to Logstash. @@ -188,7 +182,7 @@ are used to tell logstash how to parse a given file's messages, based on the file's message format. When adding a new file to be indexed to -:cgit_file:`modules/openstack_project/files/logstash/jenkins-log-client.yaml` +http://git.openstack.org/cgit/openstack-infra/project-config/tree/roles/submit-logstash-jobs/defaults/main.yaml at least one tag from the indexer.conf.erb file should be associated with the new file. One can expect to see '{%logmessage%}' instead of actual message data if indexing is not working properly. @@ -299,7 +293,7 @@ General query tips: fields require specific information. * This system is growing fast and may not always keep up with the load. Be patient. If expected logs do not show up immediately after the - Jenkins job completes wait a few minutes. + Zuul job completes wait a few minutes. crm114 ======= @@ -329,3 +323,6 @@ be expected. Should the crm114 settings need to be tuned or expanded, a patch may be submitted for this file, which controls the process: https://git.openstack.org/cgit/openstack-infra/puppet-log_processor/tree/files/classify-log.crm +.. _logs post-playbook: http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/base/post-logs.yaml +.. _submit-logstash-jobs defaults: http://git.openstack.org/cgit/openstack-infra/project-config/tree/roles/submit-logstash-jobs/defaults/main.yaml +.. _submit-log-processor-jobs role: http://git.openstack.org/cgit/openstack-infra/project-config/tree/roles/submit-log-processor-jobs diff --git a/doc/source/translate.rst b/doc/source/translate.rst index de37db4b5b..ff37523153 100644 --- a/doc/source/translate.rst +++ b/doc/source/translate.rst @@ -33,8 +33,10 @@ instance of Zanata running on the `Wildfly JBoss Application Server before being applied in production. Translators work through the Zanata web UI or with the zanata-cli tool to do -their translations. A series of Jenkins jobs handle translations proposals -on the proposal slave. +their translations. Zuul proposes updates to translations with the +``upstream-translation-update`` and ``propose-translation-update`` jobs +defined in `project-config zuul jobs`_. + Projects are added for translations by modifying :config:`gerrit/projects.yaml` and adding the following to @@ -49,3 +51,5 @@ changes. Finally, the translations jobs must be added to the project in :config:`jenkins/jobs/projects.yaml` and :config:`zuul/layout.yaml`. + +.. _project-config zuul jobs: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/jobs.yaml diff --git a/modules/openstack_project/files/logstash/jenkins-log-client.yaml b/modules/openstack_project/files/logstash/jenkins-log-client.yaml deleted file mode 100644 index 2e177eb933..0000000000 --- a/modules/openstack_project/files/logstash/jenkins-log-client.yaml +++ /dev/null @@ -1,618 +0,0 @@ -source-url: http://logs.openstack.org - -# List of zmq event inputs. -zmq-publishers: [] - -subunit-files: - - name: logs/testrepository.subunit - build-queue-filter: 'gate' - - name: logs/old/testrepository.subunit - job-filter: '.*grenade-dsvm.*' - build-queue-filter: 'gate' - - name: testrepository.subunit - build-queue-filter: 'gate' - - name: logs/karma.subunit - job-filter: '.*npm-run.*' - build-queue-filter: 'gate' - - name: karma.subunit - job-filter: '.*npm-run.*' - build-queue-filter: 'gate' - - name: testrepository.subunit - build-queue-filter: 'periodic' - - name: logs/testrepository.subunit - build-queue-filter: 'periodic' - - name: testrepository.subunit - build-queue-filter: 'post' - - name: logs/testrepository.subunit - build-queue-filter: 'post' -# List of files to source logs from. -source-files: - - name: console.html - tags: - - console - - name: logs/devstack-gate-cleanup-host.txt - tags: - - console - - console.html - - name: logs/devstack-gate-setup-host.txt - tags: - - console - - console.html - - name: logs/devstack-gate-setup-workspace-new.txt - tags: - - console - - console.html - - name: logs/devstack-gate-setup-workspace-old.txt - tags: - - console - - console.html - job-filter: '.*grenade.*' - - name: logs/grenade.sh.txt - tags: - - console - - console.html - job-filter: '.*grenade.*' - - name: logs/devstacklog.txt - tags: - - console - - console.html - - name: logs/screen-c-api.txt - tags: - - screen - - oslofmt - - name: logs/screen-c-bak.txt - tags: - - screen - - oslofmt - - name: logs/screen-c-sch.txt - tags: - - screen - - oslofmt - - name: logs/screen-c-vol.txt - tags: - - screen - - oslofmt - - name: logs/screen-designate-agent.txt - tags: - - screen - - oslofmt - job-filter: '.*designate.*' - - name: logs/screen-designate-api.txt - tags: - - screen - - oslofmt - job-filter: '.*designate.*' - - name: logs/screen-designate-central.txt - tags: - - screen - - oslofmt - job-filter: '.*designate.*' - - name: logs/screen-designate-mdns.txt - tags: - - screen - - oslofmt - job-filter: '.*designate.*' - - name: logs/screen-designate-pool-manager.txt - tags: - - screen - - oslofmt - job-filter: '.*designate.*' - - name: logs/screen-designate-sink.txt - tags: - - screen - - oslofmt - job-filter: '.*designate.*' - - name: logs/screen-designate-zone-manager.txt - tags: - - screen - - oslofmt - job-filter: '.*designate.*' - - name: logs/screen-g-api.txt - tags: - - screen - - oslofmt - - name: logs/screen-g-reg.txt - tags: - - screen - - oslofmt - - name: logs/screen-h-api-cfn.txt - tags: - - screen - - oslofmt - - name: logs/screen-h-api-cw.txt - tags: - - screen - - oslofmt - - name: logs/screen-h-api.txt - tags: - - screen - - oslofmt - - name: logs/screen-h-eng.txt - tags: - - screen - - oslofmt - - name: logs/screen-ir-api.txt - tags: - - screen - - oslofmt - - name: logs/screen-ir-cond.txt - tags: - - screen - - oslofmt - # This file is used on branches >= pike. - - name: logs/screen-keystone.txt - tags: - - screen - - oslofmt - # even though keystone is going through apache, it modifies log - # format to oslo format. This file is used on branches < pike. - # Note that it is also present on grenade jobs and does not have - # a new vs old split. - - name: logs/apache/keystone.txt - tags: - - screen - - oslofmt - - name: logs/screen-m-api.txt - tags: - - screen - - oslofmt - - name: logs/screen-m-sch.txt - tags: - - screen - - oslofmt - - name: logs/screen-m-shr.txt - tags: - - screen - - oslofmt - - name: logs/screen-n-api.txt - tags: - - screen - - oslofmt - - name: logs/screen-n-api-meta.txt - tags: - - screen - - oslofmt - - name: logs/screen-n-cell-child.txt - tags: - - screen - - oslofmt - job-filter: '.*cells.*' - - name: logs/screen-n-cell-region.txt - tags: - - screen - - oslofmt - job-filter: '.*cells.*' - # Only nova-conductor in a singleconductor deployment. - - name: logs/screen-n-cond.txt - tags: - - screen - - oslofmt - # Top-level nova-conductor in a superconductor deployment. - - name: logs/screen-n-super-cond.txt - tags: - - screen - - oslofmt - # Cell-level conductor in a superconductor deployment. - - name: logs/screen-n-cond-cell1.txt - tags: - - screen - - oslofmt - - name: logs/screen-n-cpu.txt - tags: - - screen - - oslofmt - - name: logs/screen-n-cpu-1.txt - tags: - - screen - - oslofmt - - name: logs/screen-n-crt.txt - tags: - - screen - - oslofmt - - name: logs/screen-n-net.txt - tags: - - screen - - oslofmt - - name: logs/screen-n-obj.txt - tags: - - screen - - oslofmt - - name: logs/screen-n-sch.txt - tags: - - screen - - oslofmt - - name: logs/screen-o-api.txt - tags: - - screen - - oslofmt - - name: logs/screen-o-cw.txt - tags: - - screen - - oslofmt - - name: logs/screen-o-hk.txt - tags: - - screen - - oslofmt - - name: logs/screen-o-hm.txt - tags: - - screen - - oslofmt - - name: logs/screen-q-agt.txt - tags: - - screen - - oslofmt - - name: logs/screen-q-dhcp.txt - tags: - - screen - - oslofmt - - name: logs/screen-q-l3.txt - tags: - - screen - - oslofmt - - name: logs/screen-q-lbaas.txt - tags: - - screen - - oslofmt - - name: logs/screen-q-meta.txt - tags: - - screen - - oslofmt - - name: logs/screen-q-metering.txt - tags: - - screen - - oslofmt - - name: logs/screen-q-svc.txt - tags: - - screen - - oslofmt - - name: logs/screen-s-account.txt - tags: - - screen - - apachecombined - - name: logs/screen-s-container.txt - tags: - - screen - - apachecombined - - name: logs/screen-s-object.txt - tags: - - screen - - apachecombined - - name: logs/screen-sahara.txt - tags: - - screen - - oslofmt - - name: logs/screen-tr-api.txt - tags: - - screen - - oslofmt - - name: logs/screen-tr-cond.txt - tags: - - screen - - oslofmt - - name: logs/screen-tr-tmgr.txt - tags: - - screen - - oslofmt - - name: logs/apache/horizon_error.txt - tags: - - apacheerror -# TODO(clarkb) Add swift proxy logs here. - - name: logs/syslog.txt - tags: - - syslog - - name: logs/libvirtd.txt - tags: - - libvirt - - name: logs/tempest.txt - tags: - - screen - - oslofmt - - name: logs/javelin.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - -# Neutron index log files (files with messages from all test cases) - - name: logs/dsvm-functional-index.txt - tags: - - oslofmt - job-filter: '.*neutron-dsvm-functional.*' - - name: logs/dsvm-fullstack-index.txt - tags: - - oslofmt - job-filter: '.*neutron-dsvm-fullstack.*' - -# Grenade new logs below - - name: logs/new/screen-c-api.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-c-bak.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-c-sch.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-c-vol.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-g-api.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-g-reg.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-n-api.txt - tags: - - screen - - oslofmt - - name: logs/new/screen-n-api-meta.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-n-cond.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-n-cpu.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-n-crt.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-n-net.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-n-obj.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-n-sched.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-q-agt.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-q-dhcp.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-q-l3.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-q-lbaas.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-q-meta.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-q-metering.txt - job-filter: '.*grenade.*' - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-q-svc.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/new/screen-s-account.txt - tags: - - screen - - apachecombined - job-filter: '.*grenade.*' - - name: logs/new/screen-s-container.txt - tags: - - screen - - apachecombined - job-filter: '.*grenade.*' - - name: logs/new/screen-s-object.txt - tags: - - screen - - apachecombined - job-filter: '.*grenade.*' -# Grenade old logs below - - name: logs/old/screen-c-api.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-c-bak.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-c-sch.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-c-vol.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-g-api.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-g-reg.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-n-api.txt - tags: - - screen - - oslofmt - - name: logs/old/screen-n-api-meta.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-n-cond.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-n-cpu.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-n-crt.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-n-net.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-n-obj.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-n-sched.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-q-agt.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-q-dhcp.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-q-l3.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-q-lbaas.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-q-meta.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-q-metering.txt - job-filter: '.*grenade.*' - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-q-svc.txt - tags: - - screen - - oslofmt - job-filter: '.*grenade.*' - - name: logs/old/screen-s-account.txt - tags: - - screen - - apachecombined - job-filter: '.*grenade.*' - - name: logs/old/screen-s-container.txt - tags: - - screen - - apachecombined - job-filter: '.*grenade.*' - - name: logs/old/screen-s-object.txt - tags: - - screen - - apachecombined - job-filter: '.*grenade.*' -# multinode subnode logs - - name: logs/subnode-2/devstacklog.txt - tags: - - console - - console.html - - name: logs/subnode-2/screen-n-api-meta.txt - tags: - - screen - - oslofmt - job-filter: '.*multinode.*' - - name: logs/subnode-2/screen-n-cpu.txt - tags: - - screen - - oslofmt - job-filter: '.*multinode.*' - - name: logs/subnode-2/screen-n-net.txt - tags: - - screen - - oslofmt - job-filter: '.*multinode.*' - - name: logs/subnode-2/screen-o-api.txt - tags: - - screen - - oslofmt - job-filter: '.*multinode.*' - - name: logs/subnode-2/screen-o-cw.txt - tags: - - screen - - oslofmt - job-filter: '.*multinode.*' - - name: logs/subnode-2/screen-o-hk.txt - tags: - - screen - - oslofmt - job-filter: '.*multinode.*' - - name: logs/subnode-2/screen-o-hm.txt - tags: - - screen - - oslofmt - job-filter: '.*multinode.*' -# tripleo logs - - name: logs/postci.txt - tags: - - console - - postci - job-filter: '.*-tripleo-ci-.*' - - name: logs/undercloud/var/log/extra/logstash.txt - tags: - - console - - postci - job-filter: '.*-tripleo-ci-.*'