Browse Source

Ectomy some Jenkins out of the docs

There are a bunch of places where the narrative text say things like
"jenkins runs jobs". This hasn't been true for a while and it's getting
less true.

Left third-party and running-your-own alone because those are
instructions for other people to do things - and we are not yet at the
point where we are suggesting anyone do their things like we do our
things.

The devstack-gate document describes an old deprecated system, but the
system is still in use (sadly) so it was mostly left intact. A warning
was added so that people would be clear that it was deprecated.

Also removed the logstash client config file. It's the only change in
here that actually affects running code and became unnecessary when it
was switched to geard with Ie3f814e6d3278d87f2a20a72e40b6b92217684fc

Change-Id: Iaf2128c3f953976180c71cb599fcbff7bc06c28a
changes/52/436452/5
Monty Taylor 4 years ago
parent
commit
55ddc7bf88
No known key found for this signature in database GPG Key ID: 7BAE94BC7141A594
7 changed files with 36 additions and 646 deletions
  1. +10
    -4
      doc/source/devstack-gate.rst
  2. +4
    -3
      doc/source/gerrit.rst
  3. +1
    -1
      doc/source/grafana.rst
  4. +1
    -1
      doc/source/irc.rst
  5. +14
    -17
      doc/source/logstash.rst
  6. +6
    -2
      doc/source/translate.rst
  7. +0
    -618
      modules/openstack_project/files/logstash/jenkins-log-client.yaml

+ 10
- 4
doc/source/devstack-gate.rst View File

@ -2,6 +2,12 @@
.. _devstack-gate:
.. warning::
devstack-gate is deprecated in favor of Zuul v3 native jobs defined
in the devstack repo itself. This document describes how devstack-gate
works as there are still legacy jobs using it.
Devstack Gate
#############
@ -43,10 +49,10 @@ How It Works
The devstack test starts with an essentially bare virtual machine
made available by :ref:`nodepool` and prepares the testing
environment. This is driven by the the devstack-gate repository which
holds several scripts that are run by Jenkins.
environment. This is driven by the devstack-gate repository which
holds several scripts that are run by Zuul.
When a proposed change is approved by the core reviewers, Jenkins
When a proposed change is approved by the core reviewers, Zuul
triggers the devstack gate test itself. This job runs on one of the
previously configured nodes and invokes the devstack-vm-gate-wrap.sh
script which checks out code from all of the involved repositories, and
@ -54,7 +60,7 @@ merges the proposed change. That script then calls devstack-vm-gate.sh
which installs a devstack configuration file, and invokes devstack. Once
devstack is finished, it runs Tempest, which performs integration testing.
After everything is done, devstack-gate copies and formats all of the logs
for archival. A jenkins jobs then copies these logs to the log archive.
for archival. Zuul then copies these logs to the log archive.
How to Debug a Devstack Gate Failure
====================================


+ 4
- 3
doc/source/gerrit.rst View File

@ -88,7 +88,7 @@ access to external testing tools on a sandbox project.
The `Voting Third-Party CI` group is used to grant +/-1 Verified
access to external testing tools for all projects.
The `Continuous Integration Tools` group contains Jenkins and any
The `Continuous Integration Tools` group contains Zuul and any
other CI tools that get +2/-2 access on reviews.
The `Release Managers` group is used for release managers.
@ -218,9 +218,10 @@ High level goals:
#. Anonymous users can read all projects.
#. All registered users can perform informational code review (+/-1)
on any project.
#. Jenkins can perform verification (blocking or approving: +/-1).
#. Zuul can perform verification (blocking or approving: +/-2).
#. Third Party CI systems can perform informational verification (+/-1).
#. All registered users can create changes.
#. The OpenStack Release Manager and Jenkins can tag releases (push
#. The OpenStack Release Manager and Zuul can tag releases (push
annotated tags).
#. Members of $PROJECT-core group can perform full code review
(blocking or approving: +/- 2), and submit changes to be merged.


+ 1
- 1
doc/source/grafana.rst View File

@ -7,7 +7,7 @@ Grafana
Grafana is an open source, feature rich metrics dashboard and graph editor for
Graphite, InfluxDB & OpenTSDB. OpenStack runs Graphite which stores all the
metrics related to Nodepool, Zuul and Jenkins (to name a few).
metrics related to Nodepool and Zuul (to name a few).
At a Glance
===========


+ 1
- 1
doc/source/irc.rst View File

@ -60,7 +60,7 @@ your channel.
Join #openstack-infra if you have any trouble with any of these commands.
NOTE: Channel admin should issue the access commands above BEFORE adding
channel to gerritbot and accessbot, otherwise Jenkins will fail tests.
channel to gerritbot and accessbot, otherwise Zuul will fail tests.
Meetbot
=======


+ 14
- 17
doc/source/logstash.rst View File

@ -22,6 +22,7 @@ At a Glance
:Configuration:
* :cgit_file:`modules/openstack_project/files/logstash`
* :cgit_file:`modules/openstack_project/templates/logstash`
* `submit-logstash-jobs defaults`_
:Projects:
* http://logstash.net/
* http://kibana.org/
@ -36,7 +37,7 @@ At a Glance
Overview
========
Logs from Jenkins test runs are sent to logstash where they are
Logs from Zuul test runs are sent to logstash where they are
indexed and stored. Logstash facilitates reviewing logs from multiple
sources in a single test run, searching for errors or particular
events within a test run, as well as searching for log event trends
@ -47,14 +48,9 @@ System Architecture
There are four major layers in our Logstash setup.
1. Log Pusher Scripts.
Subscribes to the Jenkins ZeroMQ Event Publisher listening for build
finished events. When a build finishes an event is received from
Jenkins which is then converted into Gearman jobs specific to that
event for each log file we care about. These jobs trigger Gearman
workers that then fetch the logs generated by that build, chop them
up, annotate them with Jenkins build info and finally sends them to a
Logstash indexer process.
1. Submit Logstash Jobs.
The `logs post-playbook`_ in the Zuul ``base`` job submit logs defined
in the `submit-logstash-jobs defaults`_ to a Logstash Indexer.
2. Logstash Indexer.
Reads these log events from the log pusher, filters them to remove
unwanted lines, collapses multiline events together, and parses
@ -76,7 +72,7 @@ bottleneck very quickly. This looks something like:
::
jenkins
zuul post-logs playbook
|
|
gearman-client ---------------
@ -98,14 +94,12 @@ bottleneck very quickly. This looks something like:
Log Pusher
----------
This is a pair of simple Python scripts. The first listens to Jenkins
build events and converts them into Gearman jobs and the second performs
Gearman jobs to push log files into logstash.
This is an ansible module in the `submit-log-processor-jobs role`_. It
submits Gearman jobs to push log files into logstash.
Log pushing looks like this:
* Jenkins publishes build complete notifications.
* Receive notification from Jenkins and convert to Gearman jobs.
* Zuul runs post-playbook on job completion.
* Using info in the Gearman job log files are retrieved.
* Log files are processed then shipped to Logstash.
@ -188,7 +182,7 @@ are used to tell logstash how to parse a given file's messages, based
on the file's message format.
When adding a new file to be indexed to
:cgit_file:`modules/openstack_project/files/logstash/jenkins-log-client.yaml`
http://git.openstack.org/cgit/openstack-infra/project-config/tree/roles/submit-logstash-jobs/defaults/main.yaml
at least one tag from the indexer.conf.erb file should be associated
with the new file. One can expect to see '{%logmessage%}' instead of
actual message data if indexing is not working properly.
@ -299,7 +293,7 @@ General query tips:
fields require specific information.
* This system is growing fast and may not always keep up with the load.
Be patient. If expected logs do not show up immediately after the
Jenkins job completes wait a few minutes.
Zuul job completes wait a few minutes.
crm114
=======
@ -329,3 +323,6 @@ be expected. Should the crm114 settings need to be tuned or expanded,
a patch may be submitted for this file, which controls the process:
https://git.openstack.org/cgit/openstack-infra/puppet-log_processor/tree/files/classify-log.crm
.. _logs post-playbook: http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/base/post-logs.yaml
.. _submit-logstash-jobs defaults: http://git.openstack.org/cgit/openstack-infra/project-config/tree/roles/submit-logstash-jobs/defaults/main.yaml
.. _submit-log-processor-jobs role: http://git.openstack.org/cgit/openstack-infra/project-config/tree/roles/submit-log-processor-jobs

+ 6
- 2
doc/source/translate.rst View File

@ -33,8 +33,10 @@ instance of Zanata running on the `Wildfly JBoss Application Server
before being applied in production.
Translators work through the Zanata web UI or with the zanata-cli tool to do
their translations. A series of Jenkins jobs handle translations proposals
on the proposal slave.
their translations. Zuul proposes updates to translations with the
``upstream-translation-update`` and ``propose-translation-update`` jobs
defined in `project-config zuul jobs`_.
Projects are added for translations by
modifying :config:`gerrit/projects.yaml` and adding the following to
@ -49,3 +51,5 @@ changes.
Finally, the translations jobs must be added to the project in
:config:`jenkins/jobs/projects.yaml` and :config:`zuul/layout.yaml`.
.. _project-config zuul jobs: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/jobs.yaml

+ 0
- 618
modules/openstack_project/files/logstash/jenkins-log-client.yaml View File

@ -1,618 +0,0 @@
source-url: http://logs.openstack.org
# List of zmq event inputs.
zmq-publishers: []
subunit-files:
- name: logs/testrepository.subunit
build-queue-filter: 'gate'
- name: logs/old/testrepository.subunit
job-filter: '.*grenade-dsvm.*'
build-queue-filter: 'gate'
- name: testrepository.subunit
build-queue-filter: 'gate'
- name: logs/karma.subunit
job-filter: '.*npm-run.*'
build-queue-filter: 'gate'
- name: karma.subunit
job-filter: '.*npm-run.*'
build-queue-filter: 'gate'
- name: testrepository.subunit
build-queue-filter: 'periodic'
- name: logs/testrepository.subunit
build-queue-filter: 'periodic'
- name: testrepository.subunit
build-queue-filter: 'post'
- name: logs/testrepository.subunit
build-queue-filter: 'post'
# List of files to source logs from.
source-files:
- name: console.html
tags:
- console
- name: logs/devstack-gate-cleanup-host.txt
tags:
- console
- console.html
- name: logs/devstack-gate-setup-host.txt
tags:
- console
- console.html
- name: logs/devstack-gate-setup-workspace-new.txt
tags:
- console
- console.html
- name: logs/devstack-gate-setup-workspace-old.txt
tags:
- console
- console.html
job-filter: '.*grenade.*'
- name: logs/grenade.sh.txt
tags:
- console
- console.html
job-filter: '.*grenade.*'
- name: logs/devstacklog.txt
tags:
- console
- console.html
- name: logs/screen-c-api.txt
tags:
- screen
- oslofmt
- name: logs/screen-c-bak.txt
tags:
- screen
- oslofmt
- name: logs/screen-c-sch.txt
tags:
- screen
- oslofmt
- name: logs/screen-c-vol.txt
tags:
- screen
- oslofmt
- name: logs/screen-designate-agent.txt
tags:
- screen
- oslofmt
job-filter: '.*designate.*'
- name: logs/screen-designate-api.txt
tags:
- screen
- oslofmt
job-filter: '.*designate.*'
- name: logs/screen-designate-central.txt
tags:
- screen
- oslofmt
job-filter: '.*designate.*'
- name: logs/screen-designate-mdns.txt
tags:
- screen
- oslofmt
job-filter: '.*designate.*'
- name: logs/screen-designate-pool-manager.txt
tags:
- screen
- oslofmt
job-filter: '.*designate.*'
- name: logs/screen-designate-sink.txt
tags:
- screen
- oslofmt
job-filter: '.*designate.*'
- name: logs/screen-designate-zone-manager.txt
tags:
- screen
- oslofmt
job-filter: '.*designate.*'
- name: logs/screen-g-api.txt
tags:
- screen
- oslofmt
- name: logs/screen-g-reg.txt
tags:
- screen
- oslofmt
- name: logs/screen-h-api-cfn.txt
tags:
- screen
- oslofmt
- name: logs/screen-h-api-cw.txt
tags:
- screen
- oslofmt
- name: logs/screen-h-api.txt
tags:
- screen
- oslofmt
- name: logs/screen-h-eng.txt
tags:
- screen
- oslofmt
- name: logs/screen-ir-api.txt
tags:
- screen
- oslofmt
- name: logs/screen-ir-cond.txt
tags:
- screen
- oslofmt
# This file is used on branches >= pike.
- name: logs/screen-keystone.txt
tags:
- screen
- oslofmt
# even though keystone is going through apache, it modifies log
# format to oslo format. This file is used on branches < pike.
# Note that it is also present on grenade jobs and does not have
# a new vs old split.
- name: logs/apache/keystone.txt
tags:
- screen
- oslofmt
- name: logs/screen-m-api.txt
tags:
- screen
- oslofmt
- name: logs/screen-m-sch.txt
tags:
- screen
- oslofmt
- name: logs/screen-m-shr.txt
tags:
- screen
- oslofmt
- name: logs/screen-n-api.txt
tags:
- screen
- oslofmt
- name: logs/screen-n-api-meta.txt
tags:
- screen
- oslofmt
- name: logs/screen-n-cell-child.txt
tags:
- screen
- oslofmt
job-filter: '.*cells.*'
- name: logs/screen-n-cell-region.txt
tags:
- screen
- oslofmt
job-filter: '.*cells.*'
# Only nova-conductor in a singleconductor deployment.
- name: logs/screen-n-cond.txt
tags:
- screen
- oslofmt
# Top-level nova-conductor in a superconductor deployment.
- name: logs/screen-n-super-cond.txt
tags:
- screen
- oslofmt
# Cell-level conductor in a superconductor deployment.
- name: logs/screen-n-cond-cell1.txt
tags:
- screen
- oslofmt
- name: logs/screen-n-cpu.txt
tags:
- screen
- oslofmt
- name: logs/screen-n-cpu-1.txt
tags:
- screen
- oslofmt
- name: logs/screen-n-crt.txt
tags:
- screen
- oslofmt
- name: logs/screen-n-net.txt
tags:
- screen
- oslofmt
- name: logs/screen-n-obj.txt
tags:
- screen
- oslofmt
- name: logs/screen-n-sch.txt
tags:
- screen
- oslofmt
- name: logs/screen-o-api.txt
tags:
- screen
- oslofmt
- name: logs/screen-o-cw.txt
tags:
- screen
- oslofmt
- name: logs/screen-o-hk.txt
tags:
- screen
- oslofmt
- name: logs/screen-o-hm.txt
tags:
- screen
- oslofmt
- name: logs/screen-q-agt.txt
tags:
- screen
- oslofmt
- name: logs/screen-q-dhcp.txt
tags:
- screen
- oslofmt
- name: logs/screen-q-l3.txt
tags:
- screen
- oslofmt
- name: logs/screen-q-lbaas.txt
tags:
- screen
- oslofmt
- name: logs/screen-q-meta.txt
tags:
- screen
- oslofmt
- name: logs/screen-q-metering.txt
tags:
- screen
- oslofmt
- name: logs/screen-q-svc.txt
tags:
- screen
- oslofmt
- name: logs/screen-s-account.txt
tags:
- screen
- apachecombined
- name: logs/screen-s-container.txt
tags:
- screen
- apachecombined
- name: logs/screen-s-object.txt
tags:
- screen
- apachecombined
- name: logs/screen-sahara.txt
tags:
- screen
- oslofmt
- name: logs/screen-tr-api.txt
tags:
- screen
- oslofmt
- name: logs/screen-tr-cond.txt
tags:
- screen
- oslofmt
- name: logs/screen-tr-tmgr.txt
tags:
- screen
- oslofmt
- name: logs/apache/horizon_error.txt
tags:
- apacheerror
# TODO(clarkb) Add swift proxy logs here.
- name: logs/syslog.txt
tags:
- syslog
- name: logs/libvirtd.txt
tags:
- libvirt
- name: logs/tempest.txt
tags:
- screen
- oslofmt
- name: logs/javelin.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
# Neutron index log files (files with messages from all test cases)
- name: logs/dsvm-functional-index.txt
tags:
- oslofmt
job-filter: '.*neutron-dsvm-functional.*'
- name: logs/dsvm-fullstack-index.txt
tags:
- oslofmt
job-filter: '.*neutron-dsvm-fullstack.*'
# Grenade new logs below
- name: logs/new/screen-c-api.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-c-bak.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-c-sch.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-c-vol.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-g-api.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-g-reg.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-n-api.txt
tags:
- screen
- oslofmt
- name: logs/new/screen-n-api-meta.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-n-cond.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-n-cpu.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-n-crt.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-n-net.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-n-obj.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-n-sched.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-q-agt.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-q-dhcp.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-q-l3.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-q-lbaas.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-q-meta.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-q-metering.txt
job-filter: '.*grenade.*'
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-q-svc.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/new/screen-s-account.txt
tags:
- screen
- apachecombined
job-filter: '.*grenade.*'
- name: logs/new/screen-s-container.txt
tags:
- screen
- apachecombined
job-filter: '.*grenade.*'
- name: logs/new/screen-s-object.txt
tags:
- screen
- apachecombined
job-filter: '.*grenade.*'
# Grenade old logs below
- name: logs/old/screen-c-api.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-c-bak.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-c-sch.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-c-vol.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-g-api.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-g-reg.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-n-api.txt
tags:
- screen
- oslofmt
- name: logs/old/screen-n-api-meta.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-n-cond.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-n-cpu.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-n-crt.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-n-net.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-n-obj.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-n-sched.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-q-agt.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-q-dhcp.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-q-l3.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-q-lbaas.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-q-meta.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-q-metering.txt
job-filter: '.*grenade.*'
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-q-svc.txt
tags:
- screen
- oslofmt
job-filter: '.*grenade.*'
- name: logs/old/screen-s-account.txt
tags:
- screen
- apachecombined
job-filter: '.*grenade.*'
- name: logs/old/screen-s-container.txt
tags:
- screen
- apachecombined
job-filter: '.*grenade.*'
- name: logs/old/screen-s-object.txt
tags:
- screen
- apachecombined
job-filter: '.*grenade.*'
# multinode subnode logs
- name: logs/subnode-2/devstacklog.txt
tags:
- console
- console.html
- name: logs/subnode-2/screen-n-api-meta.txt
tags:
- screen
- oslofmt
job-filter: '.*multinode.*'
- name: logs/subnode-2/screen-n-cpu.txt
tags:
- screen
- oslofmt
job-filter: '.*multinode.*'
- name: logs/subnode-2/screen-n-net.txt
tags:
- screen
- oslofmt
job-filter: '.*multinode.*'
- name: logs/subnode-2/screen-o-api.txt
tags:
- screen
- oslofmt
job-filter: '.*multinode.*'
- name: logs/subnode-2/screen-o-cw.txt
tags:
- screen
- oslofmt
job-filter: '.*multinode.*'
- name: logs/subnode-2/screen-o-hk.txt
tags:
- screen
- oslofmt
job-filter: '.*multinode.*'
- name: logs/subnode-2/screen-o-hm.txt
tags:
- screen
- oslofmt
job-filter: '.*multinode.*'
# tripleo logs
- name: logs/postci.txt
tags:
- console
- postci
job-filter: '.*-tripleo-ci-.*'
- name: logs/undercloud/var/log/extra/logstash.txt
tags:
- console
- postci
job-filter: '.*-tripleo-ci-.*'

Loading…
Cancel
Save