507 Commits

Author SHA1 Message Date
Jenkins
c0a7436c33 Merge "Implement propose_rdo_promote CI periodic job" 2016-06-03 16:19:24 +00:00
Emilien Macchi
41235561e6 Implement propose_rdo_promote CI periodic job
Background: during a cycle, we deploy OpenStack on centos7 from trunk
using RDO repositories, automatically built at each commit in OpenStack.

Currently, Puppet OpenStack CI manually promotes
puppet-openstack-integration every week, to latest URL. We would like to
automate this process with a daily periodic job so every day we would
have a promotion and we could detect which commit in OpenStack would
break our CI, and we could work on it quicker than usual, because
feedback would be daily.

This patch adds a periodic job, called
"propose-puppet-openstack-rdo-promote" that will execute
jenkins/scripts/propose_rdo_promote.sh script. The script has been
tested and already used manually.

Change-Id: I9b19a440325f609746a197676d3799aa0f60ce66
2016-06-03 08:34:19 -04:00
Andreas Jaeger
ffbf1c1e58 Fix VENV temporary path
Remove wrong setup of temporary path. mktemp creates the directory in
/tmp, so no need to add current directory to it.

Change-Id: Ie8e9ed90d1bb038a877ff9caf6d2d339f04766f0
2016-06-03 09:18:10 +02:00
Jenkins
068cfa5a53 Merge "Fix translation proposals" 2016-06-03 05:07:27 +00:00
Andreas Jaeger
129937a406 Fix translation proposals
Translation proposals have been changing files in the VENV directory
since the pull_from_zanata function downloads all files and it's not
clear which directories to use. The find there is from the top level.

Move the VENV to outside of the git tree to the tmp directory - like
it's done in all other cases for jenkins/scripts/. This also allows to
simplify the upstream_translation_update script.

Change-Id: Idf344d89691189c7c9ed46ef885433d3db14dff2
2016-06-02 19:18:37 +02:00
Andreas Jaeger
ea5be3a3a0 [translations] Fix path for firstapp
The directory is called firstapp, not openstack-firstapp. Fix the script
to handle it.

Change-Id: Icc7c59276b722a3f5f21afb4b0b653dd5e3aa32f
2016-06-02 09:16:46 +02:00
Jeremy Stanley
ffc57fc87a Use virtualenv isolation when building wheels
Run each round of wheel building in a fresh virtualenv. This solves
needing to have pip installed into the system python3 module search
path.

Change-Id: I7c58826e12bd5747930510af0c5c8c4134c9b815
Closes-Bug: #2000593
2016-05-27 20:37:38 +00:00
Victor Stinner
8cbb585777 Fix wheel-build.sh for Python 3
Replace pip3 command with "python3 -m pip": the pip3 command doesn't exist on
the builder. Moreover, "python -m pip" is now recommended over "pip".

Replace also the pip command with "python -m pip".

Change-Id: Idbc6e53bf2f54a9da952fd940efb8e41e6b7c253
Related-Bug: 1582257
2016-05-20 15:45:46 +02:00
Jenkins
4a779af3da Merge "Build also wheel packages for Python 3" 2016-05-18 09:56:53 +00:00
Victor Stinner
c7ee1a5898 Build also wheel packages for Python 3
Currently, tox gets many prebuilt wheel packages on Python 2 which
make the creation of the Python 2 virtual environments fast. But
Python 3 requires to build much more dependencies and so the creation
of the Python 3 virtual environments is slower.

This change builds also wheel packages on Python 3.

Change-Id: I60c5b2c55eaf3145c3028ede1cf93a76e766b587
Related-Bug: 1582257
2016-05-16 16:34:55 +02:00
Emilien Macchi
673a839e33 generate_puppetfile: add missing $DIR
Complete a script to generate a clean Puppetfile, by adding a missing
$DIR so we generate Puppetfile with all modules.

Change-Id: I0d760e155f8b45aaab579323da5446a143e4f249
2016-05-16 09:05:18 +02:00
Emilien Macchi
8e4a940b93 Fix generate_puppetfile.sh script
The script was generating a puppetfile in the workspace, and not in the
puppet-openstcak-integration, so git didn't find any diff after the
script run.
This patch aims to generate the Puppetfile in the right directory.

Change-Id: Iaac200fa5a520f79e102171a2a1b02ca2b3c3801
2016-05-13 15:25:04 +02:00
Andreas Jaeger
0182437c5a Use "bindep test"
For testing purposes use the "test" profile to install packages. This
installs also any packages that have no selector.

Change-Id: Ia1890185d2cded484de5ceb6a4b4001510aa33da
2016-05-10 15:43:07 +02:00
Andreas Jaeger
fa58d76612 Really fix translations copy
We cannot create the .translation-source directory before pushing to
zanata, otherwise the zanata-cli will push the .translation-source
directory as well.

Instead collect modulenames and do copying at end.

Reference:
http://logs.openstack.org/e4/e49717dd71bbaf0e9ca2303206037edc56fbe689/post/nova-upstream-translation-update/3d5f20e/console.html#_2016-05-06_21_30_33_017

Change-Id: I21ad03d1d227964e0709ef99613eaa060fee7158
2016-05-06 23:44:48 +02:00
Andreas Jaeger
b2bb88e4c5 Improve pot publishing
Some projects like horizon have several modules and each contains the
same pot file, just in different locations:
* horizon/locale/djangojs.pot
* horizon/locale/django.pot
* openstack_dashboard/locale/djangojs.pot
* openstack_dashboard/locale/django.pot

so, the previous logic copied all in the exact same directory, just
resulting only in one of each - and not both.

Change the logic to use exactly same path for publishing. This is also
the path that is shown for each file in Zanata, so it's easy to
recognize and consistent accross git repository, Zanata and
tarballs.openstack.org.
Instead of a final find, run find only in the modules that we handle.

So, files are published to:
tarballs.openstack.org/translation-source/horizon/master/horizon/locale/djangojs.pot
etc

Change-Id: I2b5f6383b2d918ffdbbf4adc1dcccd06d721c65f
2016-05-06 22:52:59 +02:00
Jenkins
a72fb19b03 Merge "Do not store source translation files in git" 2016-05-06 15:34:12 +00:00
Jenkins
ca750dfe26 Merge "Implement periodic job to update Puppet OpenStack constraints" 2016-05-06 00:34:00 +00:00
Andreas Jaeger
5988eab4ac Fix publishing location for translations
We want to publish to
tarballs.openstack.org/translation-source/PROJECT/VERSION . Correctly
construct path for it.

Change-Id: I9802f50cae5dd461f76d8ba016a5f404fe74f4f3
2016-05-05 22:48:16 +02:00
Jenkins
448740641b Merge "Translations: Publish pot files to tarballs.o.o" 2016-05-05 19:22:14 +00:00
Andreas Jaeger
f856488f3a Do not store source translation files in git
Since we publish the source translation files (*.pot) now at
tarballs.openstack.org, we can remove the storing of source translation
files in git completly.

Note that we still need to generate pot files - to check percentage of
translated files.

Change-Id: Ia9b3a65fbb902c2644e81d7567f5e5e166a66b6a
2016-05-05 20:58:06 +02:00
Andreas Jaeger
02528fc319 translations: Ignore version failures
python setup.py --version fails on tacker repo since it has an extra
setup_hook. Instead of aborting the script, use "unknown" as version.

Change-Id: If67cd1c41e006a8505683a004c7cbce93064b3a1
2016-05-04 23:59:26 +02:00
Jenkins
30a859871b Merge "Update puppet logic for run-tarball.sh" 2016-05-04 14:45:53 +00:00
Jenkins
934aa94854 Merge "Install constrainted version of babel" 2016-05-04 13:00:05 +00:00
Tony Breeds
06d4b4ccac Sort the freeze_venv output
When looking at the console log it's more human friendly to sort this output
then for example all the oslo libraries are togther.

Change-Id: Iefe9184c9746e0c6ad487418ceda578bf7b363ff
2016-05-04 03:27:18 +00:00
Emilien Macchi
697a5df0a1 Implement periodic job to update Puppet OpenStack constraints
In Puppet OpenStack CI, we use r10k to clone Puppet OpenStack modules
and all their dependencies from different places.
The Puppetfile is useful for both unit and functional tests, so we have
a consistency in what we test.

Until now, we were managing this Puppetfile without automation.
From time to time, we were updating the file but it does not scale,
regarding the work that needs to be done and the amount of dependencies
that never stop to grow up.

We recently implemented a script that is called
generate_puppetfile.sh (see dependency). This script aims to generate a
Puppetfile by cloning OpenStack & External modules.
The script will fetch tags and take the most recent one.
The script works as it is now, please look:
https://review.openstack.org/300696
And the Puppetfile that is generated.

This patch is an attempt to automate the run of this script, by using
the periodic job that is already in place for many other projects.

More features will come in next iterations:
* support stable branches
* add more logic in the script to detect whether or not tags exist (all
  our current modules have tags, but we never know...)

Change-Id: Id23ba86487d2223861b21941fa04acac0d49e3f3
2016-05-03 16:31:16 -04:00
Andreas Jaeger
06609659c0 Install constrainted version of babel
Instead of using a hardcoded version of Babel, let's use the constrained
version. This is still 2.2.0 right now but once it changes, it changes
here as well. We can make this change now since Babel is tested as part
of the requirements repository.

When installing horizon, use the same branch as the project uses
and use constraints here.

Test babel changes in requirements now using a tox target. The job is
only run when upper-constraints or one of the test files is changed -
and only run for master and newton or newer.

Change-Id: I15fa6a706323615d6072d8acaa859fea9bf35883
Depends-On: Iab7844ba748ecd84ec07ad4280b2317bd626fe9a
2016-05-02 19:39:27 +02:00
Paul Belanger
2ab55fb249
Update puppet logic for run-tarball.sh
Here we are making sure our tarballs that we produce are named
puppet-foo.tar.gz.  Additionally, we don't need to rename tagged
tarballs, since puppet build module does the right thing.

Change-Id: Ib16327fe436e5dd84c773be19892f3eca4a09f55
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-05-02 10:45:11 -04:00
Jenkins
d9b546aee7 Merge "copy_puppet_logs.sh: grab openvswitch logs" 2016-04-23 11:46:32 +00:00
Emilien Macchi
b60950913f run-tarball/puppet: fix module name for -tarball jobs
Fix the regex that find the actual Puppet module name.

Change-Id: I0fb69aee716114bc789440b1a6a60ca4dd86cf00
2016-04-22 16:29:25 -04:00
Emilien Macchi
88bf325628 copy_puppet_logs.sh: grab openvswitch logs
Add OVS logs, very useful to debug networking issues.

Change-Id: I6dd2997c6448f0f97b41b3caa557733a6d9f7645
2016-04-22 16:05:54 -04:00
Andreas Jaeger
4a36d40fc3 Translations: Publish pot files to tarballs.o.o
Publish all pot files to
tarballs.openstack.org/translation-source/{name}/VERSION

This way we have always the most recent pot files available for
reviewers and can remove them from the git repositories in a followup.

Change-Id: I3c5e145115206b773466a74d63fad84da17f25e5
2016-04-22 21:19:46 +02:00
James E. Blair
aca7a864a7 Don't check for proposed branches in propose_update
These are not used anymore, so I am reliably informed.

Change-Id: I9a4da1d350d1884ba2b263f6945a01ac4e2c33b4
2016-04-21 07:54:41 -07:00
James E. Blair
a86d97f497 Revert "Fix comment on requirements job"
This reverts commit 4f96b2727c2afd3c3d6b888c21c8d18ba1cb7376.

The correct fix for this is not to stop using the feature.

Change-Id: I4b35a6ca13153257984074b5145a5e691564e0ca
2016-04-21 07:23:51 -07:00
Ian Wienand
4f96b2727c Fix comment on requirements job
Currently failures of this job leave a broken comment that doesn't
match the test-result regexes, meaning that you can't see the result
in the callout box.  See [1].

I863d88e9a7ed2fd41924b8fc4a12dbea3ee2b205 fixes the root cause, but
this message is still a bit wrong because it doesn't have the
[PASSED|FAILURE] bit.

I don't really see why we need this special result at all.  If it
fails, you have to read the log anyway to figure out what is wrong, so
move the message pointing to the docs to where the failure actually
happens in the logs.

[1] https://review.openstack.org/#/c/292193/

Change-Id: Ie5e7d2318a510a5811c9131d56594e856da9dcc5
2016-04-21 13:57:59 +10:00
Emilien Macchi
ffa7d793d4 Add tarball jobs for all Puppet OpenStack modules
We created the job for puppet-aodh first, so we could test.
Now it's in place, we would like the jobs for all modules.

Change-Id: I03e0235b0cd8f216b952c03145d158577862c0fc
2016-04-20 18:20:13 +00:00
Emilien Macchi
67e650f8f8 run-tarball/puppet: fix MODULE_NAME
drop "-tarball" in MODULE_NAME:
Because we're cloning the module in puppet-<modulename>-tarball, and we
want the tarball puppet-<modulename>-<version>.tar.gz, we need to drop
"-tarball" from the basename.

Change-Id: I77680a3ed0647c458900f605f1e8e50bf70369d3
2016-04-20 14:16:07 -04:00
Andreas Jaeger
717ca3b9bf Propose translations: Fix invalid bash
Remove extra $s from local declaration.

Change-Id: I259bfa1f52349223ddd77d7aa0fd26257721748c
2016-04-20 09:24:29 +02:00
Jenkins
05b0a07052 Merge "Translations: Refactor" 2016-04-19 13:30:02 +00:00
Andreas Jaeger
f5a2ef4748 Translations: Fix sync of releasenotes
Only generate releasenotes from master branch and only if
they exist.

Change-Id: Iaff4375fc370592484df2d6917bcdd9f15d380b6
2016-04-18 20:11:51 +02:00
Andreas Jaeger
cb645e72af Translations: Refactor
Merge extract_messages and extract_messages_log into one function
extract_messages_python since these are always used together.

Merge propose_python and propose_django in one function propose,
call the extract_messages_* function outside of it.

Refactor python and django handling in a single function that can
be invoked twice instead of duplicating lines.

Change-Id: Ib92df685f9268e2429a119f619c1ab7e9ff1b82a
2016-04-18 19:37:35 +02:00
Jenkins
e9432c01c4 Merge "Translations: Cleanup django modules" 2016-04-18 14:28:10 +00:00
Jenkins
adc444a019 Merge "Translations: Fix releasenotes" 2016-04-18 14:22:58 +00:00
Andreas Jaeger
67e91f7801 Translations: Handle non-translated module
extract_messages removes the pot file if it does not contain any
translations. But then git add will fail since nothing is to add.

This currently breaks keystone proposal:
http://logs.openstack.org/periodic/keystone-propose-translation-update/2ef243f

Add a test for empty directory and exist early if it exists.

Change-Id: I3cce7f81856d579dde36445f722ba513a29fae61
2016-04-18 09:19:58 +02:00
Andreas Jaeger
5cb5247c5c Translations: Fix releasenotes
Since we have now first translations of releasenotes, we can test
downloading of these. Testing barbican, nova, and openstack-manuals on
my local system found a couple of problems that are fixed with this
patch.

Refactor how we handle releasenotes, let's handle them like python and
django projects with specific functions.

Run the same cleanup on releasenotes like on other pot and po
files.

Fix download of releasenotes translations for openstack-manuals, previously
zanata-cli complained about a missing rule for these.

Remove from setup_project the special reno_resource setup, this is not
needed, the general rule works just fine.

Also, openstack-manuals does not set up the venv, so no need to remove
it.

Include minor cleanup for cleanup_pot_files.

Change-Id: I8c9a018b859b983f39c5239224bb530ce7500cb3
2016-04-18 08:53:09 +02:00
Richard Jones
d7598e96ba Fix script to include repos in PYTHONPATH
The repos checkout needs to be in the PYTHONPATH for the
import of the xstatic module to work. Since we invoke
the xstatic_check_version.py by absolute path, Python
does not include the cwd() in the PYTHONPATH.

Change-Id: Idd4f8db6334c9f29168e3bc39de3ed95a4e1c60f
2016-04-18 15:17:08 +10:00
Andreas Jaeger
2e256f8f91 Translations: Cleanup django modules
Remove untranslated and unused po/pot files for django modules - the
same way we do for python projects.

Change-Id: I06e71e29a727a40c960b3721333dda9e0bde7b11
2016-04-17 21:46:02 +02:00
Andreas Jaeger
0af0f35a2a translations: Do not merge untranslated pot files
Do not storate pot files that are not translated currently, remove
them from any proposals.

Change-Id: I9ded8b892a4034bccf27c43d0d3721a97eab6bb0
2016-04-17 19:29:38 +02:00
Andreas Jaeger
4a63f9bcdd Add translation metadata
pybabel does not set by default PROJECT, VERSION or BUG-address - unlike
"python setup.py extract". Set these values now.

Change-Id: I889899034208f2861216b0d7944db8480b2dd7fb
2016-04-16 20:16:41 +02:00
Jenkins
e3f67080f7 Merge "run-wheel.sh: only run wheel for Python projects" 2016-04-15 19:43:17 +00:00
Andreas Jaeger
a1dd4fee2f Remove VENV before pushing to Zanata
For horizon, we pushed 10+ files to Zanata, since all pot files in the
VENV were pushed as well.

Remove the VENV before we push to the translation server.

Change-Id: I236af881b2fa46bf1f676ebe06513fb27accaf5f
2016-04-15 19:59:50 +02:00