421 Commits

Author SHA1 Message Date
Clark Boylan
9fa6186cc8 Move slave scripts to new slave script location
We apparently didn't rebase all changes properly and some new scripts
ended up in jenkins/files/slave_scripts when they should all go into
openstack_project/files/slave_scripts. Move them now.

Change-Id: If6eec70f9da393193497d4d52d6a69a34cdb4db2
2014-05-30 15:15:51 -07:00
Jenkins
7b374cd372 Merge "Add slavescript to grab console log" 2014-05-30 20:57:30 +00:00
Monty Taylor
a207facd23 Create and upload wheels
We want to make wheels for pre-release and release events. We will only
build wheels on python 2.7. If the project supports python 3 as well, the
project will have marked itself as supporting universal wheels. If it does
that, python 2.7 will build a wheel that can be used in both places. If it
does not support python 3, then we don't have to build on on python 3.

Change-Id: I060b59d8adc586ff37065fa4f6172a592ee3f903
2014-05-28 12:52:39 -07:00
Monty Taylor
ad574bedee Changes to get nodepool working with trusty
There are few things that are different for our slaves on trusty. Of
note, we don't want to pin puppet, and we need to provide a version
for postgres.

Change-Id: Ibee78cd4fbeef2e6af43379d2bc3a0f0e9767a06
2014-05-28 10:09:35 -07:00
K Jonathan Harker
34407052cc Correct source file location
The jenkins-sudo file resources were moved from the jenkins module to
the openstack_project module, and the source paths in the file resource
were changed to reflect the move, but the source *.sudo files were not
moved to the new module.

Change-Id: I54c288fe73cacd8e0bd09db3d6f855e06fc65434
2014-05-23 14:14:42 -07:00
Jenkins
5f1f5b2167 Merge "Move jenkins-sudo bits to openstack_project" 2014-05-23 20:11:18 +00:00
K Jonathan Harker
541fac117b Move jenkins-sudo bits to openstack_project
The jenkins-sudo privileges are specific to the slave_scripts directory,
which has already been moved to the openstack_project. Let's keep these
two things together.

Change-Id: I0ef16850bf97d54f1ac92e41e5bd31e7f6f40ea9
2014-05-23 11:55:20 -07:00
K Jonathan Harker
f9c4c6f64c Remove jenkins slave_scripts from jenkins masters
There are no jobs running on the masters anymore, they don't need to
have the slave scripts.

Change-Id: Ia4666e396f0d69369729ce4475062c57d0763a91
2014-05-23 11:45:01 -07:00
Joshua Hesketh
d89c72fbcb Add slavescript to grab console log
The console logs are at
http://buildserver/jenkins/job/MyJobName/666/consoleText

Add in a shell script to grab the console log using $BUILD_URL
(https://wiki.jenkins-ci.org/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-JenkinsSetEnvironmentVariables)

Change-Id: I481fdec77d42918a68ba13645912704948f86a36
2014-05-23 10:47:50 +10:00
K Jonathan Harker
8e7f9e3458 Move openstack-specific config out of ::jenkins
The ::jenkins::slave class contained a lot of openstack-specific
configuration rather than configuration of a generic jenkins slave.

The term "bare slave" is overloaded and confusing: create simple_slave
and thick_slave to differentiate between the two meanings of "bare".
Some portions of ::jenkins::slave will move to simple_slave, some
portions to thick_slave, and some portions to slave_common (all in the
openstack_project module).

Change-Id: I5281a03a7f6da3f98714bcc59ae840ace8435578
2014-05-20 14:39:51 -07:00
Andreas Jaeger
017c4505b4 Only download nearly finished translated files
The daily "Imported Translations from Transifex" patch includes all po
files, even the ones that are completely untranslated or only have a partial
translation. There's no real need for these files.

Therefore, only download files that are at least 75 per cent translated.

There is one exception: For openstack-manuals the common directory
contains many files where only a few need to be translated - so allow
here 8 per cent translation. This currently downloads the Japanese and
French files and thus covers the languages that we need.

Partial-Bug: #1317794

Change-Id: I763c21768e3876aaaf4734bf2d769de20b49ff42
2014-05-16 14:19:39 -04:00
Clark Boylan
4e4da023c4 Fix zero tests ran unittest check.
The unittests attempt to check that more than 0 tests run; however, in
some cases 0 tests will be run and testr last won't list the number of
tests as a number. This causes the sed filter to fail to match resulting
in a rancount of "". If we get a rancount of "" or "0" fail the test.

Change-Id: I1f7ca358c73c071e60ba4b651cf75db18b8fd487
2014-05-07 20:00:18 -07:00
Jenkins
835d365dce Merge "JJB can take longer than 5 minutes to run" 2014-05-06 22:12:08 +00:00
Jenkins
ed801eca82 Merge "add subunit log test sanity" 2014-05-01 01:43:55 +00:00
K Jonathan Harker
35eea3a673 JJB can take longer than 5 minutes to run
Double the timeout on the jenkins_jobs_update exec (from 5m to 10m)
because jenkins-jobs can take longer than 5 minutes to run when a
reasonably large number of jobs are defined.

Change-Id: Iff74a322a2f2e605c3c9e78f36cd8d12571e944a
2014-04-30 16:52:36 -07:00
Jenkins
1a336a75d3 Merge "Create propose_translation_update for horizon" 2014-04-30 14:55:04 +00:00
Jenkins
0a91df49ed Merge "Remove accumulated python cruft" 2014-04-30 13:52:45 +00:00
James E. Blair
5f12e27897 Drop the ref from the tag fetch in GGP
If you start with an out of date repo (we do -- we clone from cached
git repos on our images) and the git remote update fails (eg due to
mirror sync issues) and you then do a fetch with a ref argument
(eg refs/tags/foo), it will fail with:

  error: no such remote ref refs/tags/foo

However, git fetch --tags will get all the tags from the zuul merger,
which is what we really care about, and the subsequent checkout and
reset will make sure that we really did get the tag we wanted.
Therefore, remove the ref argument that can cause these occasional
failures.

Change-Id: Iaed8b82204a3eddc7ee418ebe94637f2a371d25b
2014-04-29 13:20:57 -07:00
Monty Taylor
d0061640be Remove accumulated python cruft
We nolonger need or want setuptools_git installed anywhere.
get-pip no longer needs ez_setup.py first. Also, github broke
redirection on Fedora, so change the URL.

Change-Id: I16d64695bf05e672fdc12236424b62e5cd5e5dc7
2014-04-29 10:10:01 -07:00
Monty Taylor
8895d64289 Fix up ssh key usage
We need for the keys to only have the key content. Also, we've learned
that the ssh_authorized_key title is important.

Change-Id: Ie772572e041ea6c1587dddb02937b89541a37e1b
2014-04-25 12:35:25 -07:00
Jenkins
b6f3fd8d6e Merge "Move to using ssh_authorized_key type" 2014-04-25 09:03:46 +00:00
Clark Boylan
8159d5acde Handle additional reqs push failure as non failing
When pushing changes to gerrit for requirements proposals there are
cases where we want to ignore push errors. Specifically when gerrit is
complaining because you are pushing the same commit or one with
identical comments. We should just ignore these errors as it means the
current reqs proposal is up to date.

See
https://review.openstack.org/Documentation/error-no-changes-made.html
for more info.

Change-Id: I5d7a1991449a80f570191eaad49dc3f9a0ff1a6a
2014-04-23 12:38:03 -07:00
Jenkins
415842d361 Merge "Add a slave-script to Jenkins for pushing to swift" 2014-04-23 03:10:48 +00:00
Clark Boylan
e0eb6f9130 Set config prior to git review -s.
Requirements proposal jobs need to set git configs for each git repo it
is proposing to. This is necessary because it shouldn't set the git
config globally for the Jenkins user on the slaves.

Change-Id: Id69ba218065022f69d2dd9d140825699d78604e3
2014-04-17 16:17:09 -07:00
Jenkins
3beec123b3 Merge "Add language spepcific font package to Jenkins slave" 2014-04-17 21:15:57 +00:00
James E. Blair
4baaf915df Override user interaction in merge tags job
Add the "-y" flag to avoid the "Is this really what you meant to do?"
prompt.

Change-Id: I3397a6e1266c6c3a5f652d3b343e37073f968dc7
2014-04-17 07:02:46 -07:00
Andreas Jaeger
18cf0b63b4 Create propose_translation_update for horizon
Add slave-file to handle horizon translations.
Setup periodic job to import horizon translations the same way its done
for other projects.

Change-Id: I0e28f0268e7f3fdea9d5f22d2e896b0079410ea6
2014-04-16 20:09:25 +02:00
Monty Taylor
6c1486ea92 Move to using ssh_authorized_key type
There is a builtin type for adding authorized_keys. We started
using it for the restricted access for the puppet triggering key,
and it seems to make sense to be consistent across the board.

Change-Id: I76ffb0136c58c05fa7c8abb793cb8e3b03cb5f6c
2014-04-15 20:24:23 -07:00
Clark Boylan
8c76adbad3 Use proposal bot username in proposal jobs.
After the switch to using the proposal-bot gerrit user in proposal jobs
some jobs continued to attempted to use the jenkins user. Fix that and
use the proposal-bot user.

Change-Id: I146b6ea29d66bd171072e6529413efcc5cae3d8d
2014-04-15 11:11:51 -07:00
Jenkins
fdddcd6b97 Merge "Propose translation update: Handle PO-Revision-Date" 2014-04-15 17:17:47 +00:00
Jenkins
2a848f9ff3 Merge "Make jenkins proposal jobs use dedicated user." 2014-04-15 16:58:53 +00:00
Andreas Jaeger
fcc53182f0 Propose translation update: Handle PO-Revision-Date
Include PO-Revision-Date in list of lines to ignore so that we do
not get new proposed files where just the date has changed.

Change-Id: I3ce13d0cdd9fdbf1711d40ebbdef408c93d49523
2014-04-15 09:00:36 +02:00
Jenkins
546fb9eecd Merge "Reduce number of files send for propose_translation" 2014-04-14 23:15:28 +00:00
Clark Boylan
7d1a297e4b Make jenkins proposal jobs use dedicated user.
Switch all jenkins proposal jobs to a dedicated user with dedicated
credentials. This is being done to be more flexible and secure when it
comes to managing the scripts that make proposals to gerrit.

Change-Id: I2dbdd530bf5b64c14207f645512a1eb319681166
2014-04-14 15:38:31 -07:00
Andreas Jaeger
a853bed193 Reduce number of files send for propose_translation
If a single file has changed, all other .pot files are send as well
since they have a new timestamp.

Check for files that only have a new timestamp and revert their change
so that they do not get committed at all.

Also, handle Project-Id-Version the same way as the timestamp.

Change-Id: I49488d01a7bb9242508ec7d378240449124608de
2014-04-14 21:54:07 +02:00
Andreas Jaeger
ba1020368a Do not generate obsolete translations
Call update_catalog with option --ignore-obsolete=true to not generate
any obsolete translation entries.

Change-Id: Ia898149f6203ebf024a64b8bf3ecbffd75261ce8
Related-Bug: #1299349
2014-04-08 11:33:31 +02:00
Jenkins
528c5af1d5 Merge "Make transifex the only source of translations" 2014-04-08 08:26:43 +00:00
Sean Dague
b6ed068a48 add subunit log test sanity
subunit logs should not be more than 50MB uncompressed. If they
are you are doing something terribly wrong, and both consuming a
ton of infra resources, as well as local user memory.

Change-Id: I0f93684c0b9055bc95808cfae827e171fa30f6d1
2014-04-07 15:48:43 -04:00
Thierry Carrez
b20f71adfe Fix milestone-proposed requirements update
milestone-proposed was checked out in a detached state, which prevents
git-review from successfully proposing changes to it. It works with
master because master is checked out by default on the initial clone.

This commit makes sure the branch is checked out with proper origin
tracking before proceeding with changes.

Fixes bug #1302490

Change-Id: Iff4f653d2d68fd36bb076ed8ae9fa5b0bb03edda
2014-04-04 15:49:01 +02:00
Jeremy Stanley
9b0176d091 Requirements check should tolerate existing cruft
* .../slave_scripts/project-requirements-change.py: When parsing
requirements files, only perform style checks like missing final
newline or duplicate entries on the new/proposed version of a file
and ignore them on the original version of the file in the
repository. This makes it possible to actually correct errors on
imported requirements files, even with the job running this script
already voting on the correcting change. Also annotate this with
more commentary so people have some hope of understanding what the
heck it's doing.

Change-Id: I8ea2800ecc8eb24937e69a573e925b4daf64ff3f
Closes-Bug: #1300843
2014-04-03 15:25:48 +00:00
Joshua Hesketh
7b4cfd7dd1 Add a slave-script to Jenkins for pushing to swift
I042cdd2dd2407f381cafcabc5c6b83d9b9a9eb00 adds instructions to
workers on how to send logs to swift using the FormPost middleware.
This slave-script allows you to pass in a directory to push to
swift using these instructions and returns a URL where they are
available.

Change-Id: I7ccafa408a88ef80317b1c546b3451c0033dce61
2014-04-01 14:01:36 +11:00
Jenkins
32d3abef76 Merge "Revert "Enable Gearman as default on Jenkins slaves"" 2014-03-31 22:54:06 +00:00
James E. Blair
8af61da62f Revert "Enable Gearman as default on Jenkins slaves"
SERVER: Invalid relationship: File[/var/lib/jenkins/hudson.plugins.gearman.GearmanPluginConfig.xml] { require => Class[Zuul] }, because Class[Zuul]doesn't seem to be in the catalog

This reverts commit 2a9cd677e4eb6a60787ebc73bd8fd9e1b6f26b34
2014-03-31 22:49:41 +00:00
Akihiro Motoki
e18cb36161 Add language spepcific font package to Jenkins slave
Localized documents start to be generated in gate jobs and
they require language specific font packages. This commit
add language_fonts_packages parameter to accept language fonts.

It adds Japanese and Korean font packages to Ubuntu build
slave because gate jobs are now run on Ubuntu based hosts.
Pakcages for RedHat flavor will be added in future if needed.

Change-Id: Id4a0363e119f4458c270fb32bd428ebcfcc75225
Closes-Bug: #1278327
2014-04-01 03:15:43 +09:00
Clark Boylan
a8c6385871 Make transifex the only source of translations
Prior to this change we attempted to push translations made direclty in
the git tree to transifex allowing translations to be made via git
directly or in transifex. Simplify the process to avoid cases where the
transifex client will refuse to update local translations because they
are newer than those in transifex. This makes transifex the only source
of translation updates. In addition to simplifying the process force tx
pull -a operations as a way to force the updates if we end up in this
state.

Change-Id: I171b3b1a29cbeaab0c4d9de3ab1bd3a6b933ea53
2014-03-31 11:05:52 -07:00
Jenkins
d302b1b698 Merge "Enable Gearman as default on Jenkins slaves" 2014-03-31 16:29:50 +00:00
Jenkins
753837e3bd Merge "Fetch tag refs from zuul in gerrit-git-prep" 2014-03-30 23:23:18 +00:00
Jeremy Stanley
8bc51a6007 Fetch tag refs from zuul in gerrit-git-prep
* gerrit-git-prep.sh: If the zuul ref is a tag, then fetch the tag
from zuul. Otherwise there's a risk that it may not have replicated
to the origin yet when initially setting up the workspace.

Change-Id: Iadd8385e4a0d0957fcac1cf89d0b5a1cfa848b49
2014-03-26 23:58:02 +00:00
Jeremy Stanley
2e21b966c9 Allow BRANCH instead of ZUUL_REF in ggp
* modules/jenkins/files/slave_scripts/gerrit-git-prep.sh: Allow
callers to supply BRANCH in the environment for cases where there
will be no ZUUL_REF environment variable. This is mainly to support
use in periodic jobs, so that we can transition away from the
Jenkins SCM plug-in.

Change-Id: I86c77695d8c374c94932664026ab7a541686ae74
2014-03-25 21:03:35 +00:00
Jeremy Stanley
044ded07b4 Pin rake gem to 10.1.1 so it works with ruby 1.8.x
Without this, we get "Error installing puppetlabs_spec_helper: rake
requires Ruby version >= 1.9."

Change-Id: I81a32380cee1a35a50e1c3baf2e1afcd69b1614f
2014-03-25 12:57:22 +00:00