Remove Jenkins documentation; replace with zuul-launcher
Change-Id: Ie6d077906a4d6e204d4cd1847486dc098de2f8f6
This commit is contained in:
James E. Blair
@@ -1,217 +0,0 @@
|
||||
:title: Jenkins
|
||||
|
||||
.. _jenkins:
|
||||
|
||||
Jenkins
|
||||
#######
|
||||
|
||||
Jenkins is a Continuous Integration system that runs tests and
|
||||
automates some parts of project operations. It is controlled for the
|
||||
most part by :ref:`zuul` which determines what jobs are run when.
|
||||
|
||||
At a Glance
|
||||
===========
|
||||
|
||||
:Hosts:
|
||||
* http://jenkins.openstack.org
|
||||
* http://jenkins-dev.openstack.org
|
||||
:Puppet:
|
||||
* https://git.openstack.org/cgit/openstack-infra/puppet-jenkins/tree/
|
||||
* :file:`modules/openstack_project/manifests/jenkins.pp`
|
||||
* :file:`modules/openstack_project/manifests/jenkins_dev.pp`
|
||||
:Configuration:
|
||||
* :config:`jenkins/jobs`
|
||||
:Projects:
|
||||
* http://jenkins-ci.org/
|
||||
* :ref:`zuul`
|
||||
* :ref:`jjb`
|
||||
:Bugs:
|
||||
* https://storyboard.openstack.org/#!/project/748
|
||||
* https://wiki.jenkins-ci.org/display/JENKINS/Issue+Tracking
|
||||
:Resources:
|
||||
* :ref:`zuul`
|
||||
* :ref:`jjb`
|
||||
|
||||
Overview
|
||||
========
|
||||
|
||||
A large number and variety of jobs are defined in Jenkins. The
|
||||
configuration of all of those jobs is stored in git in the
|
||||
openstack-infra/project-config repository. They are defined in YAML
|
||||
files that are read by :ref:`jjb` which configures the actual jobs in
|
||||
Jenkins.
|
||||
|
||||
Anyone may submit a change to the openstack-infra/project-config
|
||||
repository that defines a new job or alters an existing job by editing
|
||||
the appropriate YAML files. See :ref:`jjb` for more information.
|
||||
|
||||
Because of the large number of builds that Jenkins executes, the
|
||||
OpenStack project favors the following approach in configuring Jenkins
|
||||
jobs:
|
||||
|
||||
* Minimal use of plugins: the more post-processing work that Jenkins
|
||||
needs to perform on a job, the more likely we are to run into
|
||||
compatibility problems among plugins, and contention for shared
|
||||
resources on the Jenkins master. A number of popular plugins
|
||||
will cause all builds of a job to be serialized even if the jobs
|
||||
otherwise run in parallel.
|
||||
* Minimal build history: Jenkins stores build history in individual
|
||||
XML files on disk, and accessing a large build history can cause
|
||||
the Jenkins master to be unresponsive for a significant time while
|
||||
loading them. It also increases memory usage. Instead, we
|
||||
generally keep no more than a day's worth of builds.
|
||||
* Move artifacts off of Jenkins: Jenkins is not efficient at serving
|
||||
static information such as build artifacts (e.g., tarballs) or
|
||||
logs. Instead, we copy them to a static webserver which is far
|
||||
more efficient.
|
||||
|
||||
Authorization
|
||||
=============
|
||||
|
||||
Jenkins is set up to use OpenID in a Single Sign On mode with Launchpad.
|
||||
This means that all of the user and group information is managed via
|
||||
Launchpad users and teams. In the Jenkins Security Matrix, a Launchpad team
|
||||
name can be specified and any members of that team will be granted those
|
||||
permissions. However, because of the way the information is processed, a
|
||||
user will need to re-log in upon changing either team membership on
|
||||
Launchpad, or changing that team's authorization in Jenkins for the new
|
||||
privileges to take effect.
|
||||
|
||||
Devstack Gate
|
||||
=============
|
||||
|
||||
OpenStack integration testing is performed by the devstack gate test
|
||||
framework. This framework runs the devstack exercises and Tempest
|
||||
smoketests against a devstack install on single use cloud servers. The
|
||||
devstack gate source can be found on `git.openstack.org
|
||||
<https://git.openstack.org/cgit/openstack-infra/devstack-gate>`_ and the `Readme
|
||||
<https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/README.rst>`_
|
||||
describes the process of using devstack gate to run your own devstack
|
||||
based tests.
|
||||
|
||||
For management of the devstack and other instances, a tool called
|
||||
:ref:`nodepool` creates and deletes Jenkins slaves as needed in order to
|
||||
maintain the pool.
|
||||
|
||||
Sysadmin
|
||||
========
|
||||
|
||||
Jenkins is largely hidden, and has no sensitive data exposed
|
||||
publicly, so we use self-signed certs for Jenkins masters.
|
||||
|
||||
After bringing up a jenkins node (16G memory instance if you use the
|
||||
stock jenkins.default) with puppet, log in and configure Jenkins by
|
||||
hand:
|
||||
|
||||
#. Configure the site so it knows it's correct url.
|
||||
(Jenkins URL in global config). This is needed to complete an SSO
|
||||
sign-in.
|
||||
|
||||
#. Configure the OpenID plugin for your SSO site (e.g. Launchpad)
|
||||
|
||||
#. Do not set CSRF protection - that causes problems with various components
|
||||
such as nodepool and swift log uploader.
|
||||
|
||||
#. Login.
|
||||
|
||||
#. Setup matrix security: add the 'authenticated' pseudo user and
|
||||
grant Admin access to your own user.
|
||||
|
||||
#. Setup one account per `http://docs.openstack.org/infra/jenkins-job-builder/installation.html#configuration-file`
|
||||
and grab the API token for it.
|
||||
|
||||
#. Configure the number of executors you want on the Jenkins Master
|
||||
(e.g. 1)
|
||||
|
||||
#. Configure a maven environment (if you have Maven projects to test).
|
||||
|
||||
#. Enable the gearman plugin globally. Your gearman server is
|
||||
zuul.$project. If Test Connection fails, do a puppet run (puppet
|
||||
agent --test) on the zuul machine, as gearman wouldn't have started
|
||||
with no workers configured.
|
||||
|
||||
#. Configure the timestamper plugin. E.g. to
|
||||
'<b>'yyyy-MM-dd HH:mm:ss'</b> '
|
||||
|
||||
#. Enable the zmq plugin globally if it is visible. No settings were
|
||||
visible when writing this doc.
|
||||
|
||||
#. You will configure global scp and ftp credentials for static and
|
||||
docs sites respectively later, but as we haven't setup those sites
|
||||
yet, that's not possible :).
|
||||
|
||||
Puppet takes care of the rest.
|
||||
|
||||
Quirks
|
||||
------
|
||||
|
||||
Note that jenkins talks to its slaves via ssh, the
|
||||
modules/openstack_project/manifests/init.pp file contains the ssh
|
||||
public key that puppet installs on the slaves.
|
||||
|
||||
Slaves
|
||||
------
|
||||
|
||||
Statically provisioned slaves have labels assigned by hand. E.g.
|
||||
centos6, and are added to a chosen Jenkins master by hand. Adding a
|
||||
slave is then:
|
||||
|
||||
#. Launch a slave
|
||||
|
||||
#. Add it to Jenkins
|
||||
Add your jenkins master key for the credentials (make it global,
|
||||
one-time operation).
|
||||
Set the jenkins home to /home/jenkins
|
||||
|
||||
#. Set appropriate labels on it
|
||||
|
||||
#. Profit!
|
||||
|
||||
Safe Master Restarts
|
||||
====================
|
||||
|
||||
Jenkins masters periodically leak threads reducing their job
|
||||
throughput and eventually leading to crashes. We work around this
|
||||
by performing weekly rolling restarts of the Jenkins masters with
|
||||
an ansible playbook.
|
||||
|
||||
If you need to perform a safe restart against a single master you
|
||||
can do this by running the same playbook and limiting it to a
|
||||
specific jenkins master
|
||||
|
||||
To do this::
|
||||
|
||||
root@puppetmaster# ansible-playbook -f1 --limit $server_fqdn \
|
||||
/opt/system-config/production/playbooks/restart_jenkins_masters.yaml \
|
||||
--extra-vars "user=hudson-openstack \
|
||||
password=$(/opt/system-config/production/tools/hieraedit.py \
|
||||
--yaml /etc/puppet/hieradata/production/fqdn/nodepool.openstack.org.yaml jenkins_api_key)"
|
||||
|
||||
Consider running this in screen as the worst case run time is as
|
||||
long as our longest running job.
|
||||
|
||||
How to manually run jenkins job builder
|
||||
=======================================
|
||||
|
||||
Jenkins job builder may need to be run manually under certain situations. If the expected
|
||||
jobs are not being created in jenkins masters, running jjb manually on the masters where
|
||||
it failed is suggested. To do this::
|
||||
|
||||
user@jenkins01# sudo -H jenkins-jobs --conf /etc/jenkins_jobs/jenkins_jobs.ini \
|
||||
update --delete-old /etc/jenkins_jobs/config
|
||||
|
||||
Consider running this in screen as the worst case run time can be of several hours.
|
||||
|
||||
In the case of incorrect jobs configuration caused by some jjb malfunction, all jobs
|
||||
will need to be regenerated. As jjb uses a local cache, to force the regeneration
|
||||
of all jobs, the cache needs to be ignored. To do this::
|
||||
|
||||
user@jenkins01# sudo -H jenkins-jobs --ignore-cache --conf \
|
||||
/etc/jenkins_jobs/jenkins_jobs.ini update --delete-old /etc/jenkins_jobs/config
|
||||
|
||||
In order to speed up the massive job reconfiguration, it may be desired to set jenkins
|
||||
on shutdown mode, visiting this link::
|
||||
|
||||
`https://jenkins[xx].openstack.org/quietDown`
|
||||
|
||||
And make Jenkins alive again after job reconfiguration finishes.
|
||||
@@ -12,10 +12,7 @@ At a Glance
|
||||
===========
|
||||
|
||||
:Hosts:
|
||||
* http://jenkins.openstack.org
|
||||
* http://jenkins-dev.openstack.org
|
||||
:Puppet:
|
||||
* https://git.openstack.org/cgit/openstack-infra/puppet-jenkins/tree/manifests/job_builder.pp
|
||||
* http://zm*.openstack.org
|
||||
:Configuration:
|
||||
* :config:`jenkins/jobs/`
|
||||
:Projects:
|
||||
@@ -228,44 +225,11 @@ values.
|
||||
Notice that example1 makes use of the job group and example2 makes use of the
|
||||
job template.
|
||||
|
||||
Job Caching
|
||||
-----------
|
||||
Zuul
|
||||
====
|
||||
|
||||
The Jenkins Jobs builder maintains a special `cache`_ that
|
||||
contains an MD5 of every generated XML that it builds. If
|
||||
it finds the XML is different then it will proceed to send this
|
||||
to Jenkins, otherwise it is skipped. If a job is accidentally deleted
|
||||
then this file should be modified or removed.
|
||||
|
||||
.. _cache: http://docs.openstack.org/infra/jenkins-job-builder/installation.html#running
|
||||
|
||||
Sending a Job to Jenkins
|
||||
------------------------
|
||||
|
||||
The Jenkins Jobs builder talks to Jenkins using the Jenkins API. This
|
||||
means that it can create and modify jobs directly without the need to
|
||||
restart or reload the Jenkins server. It also means that Jenkins will
|
||||
verify the XML and cause the Jenkins Jobs builder to fail if there is
|
||||
a problem.
|
||||
|
||||
For this to work a configuration file is needed. There is an erb
|
||||
template for this configuration file at
|
||||
:file:`modules/jenkins/templates/jenkins_jobs.ini.erb`. The contents
|
||||
of this template are:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[jenkins]
|
||||
user=<%= username %>
|
||||
password=<%= password %>
|
||||
url=<%= url %>
|
||||
|
||||
The values for user and url are hardcoded in the Puppet repo in
|
||||
:file:`modules/openstack_project/manifests/jenkins.pp`, but the
|
||||
password is stored in hiera. Make sure you have it defined as
|
||||
``jenkins_jobs_password`` in the hiera DB.
|
||||
|
||||
The password can be obtained by logging into the Jenkins user,
|
||||
clicking on your username in the top-right, clicking on `Configure`
|
||||
and then `Show API Token`. This API Token is your password for the
|
||||
API.
|
||||
In our environment, we no longer use Jenkins to execute jobs. Zuul
|
||||
itself, via Ansible, runs the actual workload. Zuul reads JJB config
|
||||
files in order to define its jobs, so, aside from the detail of not
|
||||
actually using Jenkins or creating any jobs in it, the use of JJB to
|
||||
configure jobs in Zuul is the same.
|
||||
|
||||
@@ -309,7 +309,7 @@ Stage 5 - Jenkins Master(s)
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
For Zuul to schedule work, it needs one or more Gearman connected Jenkins
|
||||
masters. See :ref:`jenkins` for details.
|
||||
masters.
|
||||
|
||||
The minimum setup is one master, but if you will be permitting any code
|
||||
submitter to trigger test runs, we recommend having two: one untrusted and one
|
||||
@@ -360,7 +360,7 @@ which get the most load (as they run jobs from anyone).
|
||||
|
||||
#. Launch the node with a size larger than the jenkins size you specified.
|
||||
|
||||
#. Setup Jenkins per :ref:`jenkins`.
|
||||
#. Setup Jenkins.
|
||||
|
||||
At this stage doing a 'recheck' should still report LOST on a change.
|
||||
But in the zuul debug.log in /var/log/zuul you should see a 'build xxx not
|
||||
|
||||
@@ -10,7 +10,6 @@ Major Systems
|
||||
gerrit
|
||||
grafana
|
||||
grafyaml
|
||||
jenkins
|
||||
zuul
|
||||
jjb
|
||||
logstash
|
||||
|
||||
@@ -15,6 +15,8 @@ At a Glance
|
||||
* http://status.openstack.org/zuul
|
||||
* http://zuul.openstack.org
|
||||
* http://zuul-dev.openstack.org
|
||||
* zm*.openstack.org
|
||||
* zl*.openstack.org
|
||||
:Puppet:
|
||||
* https://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/
|
||||
* :file:`modules/openstack_project/manifests/zuul_prod.pp`
|
||||
@@ -35,12 +37,11 @@ The OpenStack project uses a number of pipelines in Zuul:
|
||||
|
||||
**check**
|
||||
Newly uploaded patchsets enter this pipeline to receive an initial
|
||||
+/-1 Verified vote from Jenkins.
|
||||
+/-1 Verified vote.
|
||||
|
||||
**gate**
|
||||
Changes that have been approved by core developers are enqueued in
|
||||
order in this pipeline, and if they pass tests in Jenkins, will be
|
||||
merged.
|
||||
order in this pipeline, and if they pass tests, will be merged.
|
||||
|
||||
**post**
|
||||
This pipeline runs jobs that operate after each change is merged.
|
||||
@@ -146,3 +147,31 @@ before the restart::
|
||||
You may watch the `Zuul Status Page
|
||||
<http://status.openstack.org/zuul/>`_ to confirm that changes are
|
||||
returning to the queues.
|
||||
|
||||
Mergers
|
||||
-------
|
||||
|
||||
Servers with names matching the pattern zm*.openstack.org are Zuul
|
||||
Mergers. These are horizontally scalable components of Zuul which
|
||||
perform git operations for the benefit of jobs. They serve git
|
||||
repositories via Apache over http, and jobs fetch changes to test from
|
||||
them. They can be started and stopped at will, and new ones added as
|
||||
necessary to accommodate load. If you remove a merger, be sure to
|
||||
leave Apache running for several hours until the last job that may
|
||||
have been launched with instructions to fetch from that merger has
|
||||
completed.
|
||||
|
||||
Launchers
|
||||
---------
|
||||
|
||||
We use an Ansible based launcher in Zuul to actually run jobs. This
|
||||
component runs on a horizontally scalable set of servers named
|
||||
zl*.openstack.org. It reads job configuration from Jenkins Job
|
||||
Builder files in the project-config repository and translates that
|
||||
into Ansible playbooks which it runs on our workers. Our jobs are
|
||||
configured to upload as much information as possible along with their
|
||||
logs, but if there is an error which can not be diagnosed in that
|
||||
manner, Ansible logs are available in the launcher-debug log file on
|
||||
the launcher host. You may use the Zuul build UUID to track
|
||||
assignment of a given job from the Zuul scheduler to the Zuul launcher
|
||||
used by that job.
|
||||
|
||||
Reference in New Issue
Block a user