54 Commits

Author SHA1 Message Date
Andreas Jaeger
483db9e7b5 translations: Always setup venv
With healthcheck generation, we need the venv containing the subunit
files always to be set up.

Change-Id: I82531fff49e0258c2934315f11f2dd2064f308ea
2016-06-07 10:11:11 +02:00
Andreas Jaeger
e6cb1cd8b7 Simplify translation sync
Since we do not store any pot files in git anymore, the optimization to
only push to the translation server if strings have changed, is not
possible anymore. Remove this code now, this includes removing
some git adds for pot files.

Change-Id: I28ad9edc49b0f840cd51354334d7ae08aadab87a
2016-06-06 21:57:49 +02:00
Andreas Jaeger
24ef9bb2a2 Get translation jobs into healthcheck
Add subunit testr output at end of run of the translation scripts, so
that healthcheck will have information about success/failure of the job.

This is implemented as an exit trap since we abort on every error and
need to provide the failure message.

Also, upload the healthcheck data to logs.openstack.org so that
healthcheck can import it.

This should be a temporary solution until healthcheck can get data from
Zuul about failure/success of job runs using database access.

Change-Id: I74bba9659a9767027e2daa649ec6f38bf822b65a
2016-06-03 17:29:24 +02:00
Andreas Jaeger
129937a406 Fix translation proposals
Translation proposals have been changing files in the VENV directory
since the pull_from_zanata function downloads all files and it's not
clear which directories to use. The find there is from the top level.

Move the VENV to outside of the git tree to the tmp directory - like
it's done in all other cases for jenkins/scripts/. This also allows to
simplify the upstream_translation_update script.

Change-Id: Idf344d89691189c7c9ed46ef885433d3db14dff2
2016-06-02 19:18:37 +02:00
Andreas Jaeger
fa58d76612 Really fix translations copy
We cannot create the .translation-source directory before pushing to
zanata, otherwise the zanata-cli will push the .translation-source
directory as well.

Instead collect modulenames and do copying at end.

Reference:
http://logs.openstack.org/e4/e49717dd71bbaf0e9ca2303206037edc56fbe689/post/nova-upstream-translation-update/3d5f20e/console.html#_2016-05-06_21_30_33_017

Change-Id: I21ad03d1d227964e0709ef99613eaa060fee7158
2016-05-06 23:44:48 +02:00
Andreas Jaeger
b2bb88e4c5 Improve pot publishing
Some projects like horizon have several modules and each contains the
same pot file, just in different locations:
* horizon/locale/djangojs.pot
* horizon/locale/django.pot
* openstack_dashboard/locale/djangojs.pot
* openstack_dashboard/locale/django.pot

so, the previous logic copied all in the exact same directory, just
resulting only in one of each - and not both.

Change the logic to use exactly same path for publishing. This is also
the path that is shown for each file in Zanata, so it's easy to
recognize and consistent accross git repository, Zanata and
tarballs.openstack.org.
Instead of a final find, run find only in the modules that we handle.

So, files are published to:
tarballs.openstack.org/translation-source/horizon/master/horizon/locale/djangojs.pot
etc

Change-Id: I2b5f6383b2d918ffdbbf4adc1dcccd06d721c65f
2016-05-06 22:52:59 +02:00
Andreas Jaeger
5988eab4ac Fix publishing location for translations
We want to publish to
tarballs.openstack.org/translation-source/PROJECT/VERSION . Correctly
construct path for it.

Change-Id: I9802f50cae5dd461f76d8ba016a5f404fe74f4f3
2016-05-05 22:48:16 +02:00
Jenkins
448740641b Merge "Translations: Publish pot files to tarballs.o.o" 2016-05-05 19:22:14 +00:00
Andreas Jaeger
06609659c0 Install constrainted version of babel
Instead of using a hardcoded version of Babel, let's use the constrained
version. This is still 2.2.0 right now but once it changes, it changes
here as well. We can make this change now since Babel is tested as part
of the requirements repository.

When installing horizon, use the same branch as the project uses
and use constraints here.

Test babel changes in requirements now using a tox target. The job is
only run when upper-constraints or one of the test files is changed -
and only run for master and newton or newer.

Change-Id: I15fa6a706323615d6072d8acaa859fea9bf35883
Depends-On: Iab7844ba748ecd84ec07ad4280b2317bd626fe9a
2016-05-02 19:39:27 +02:00
Andreas Jaeger
4a36d40fc3 Translations: Publish pot files to tarballs.o.o
Publish all pot files to
tarballs.openstack.org/translation-source/{name}/VERSION

This way we have always the most recent pot files available for
reviewers and can remove them from the git repositories in a followup.

Change-Id: I3c5e145115206b773466a74d63fad84da17f25e5
2016-04-22 21:19:46 +02:00
Jenkins
05b0a07052 Merge "Translations: Refactor" 2016-04-19 13:30:02 +00:00
Andreas Jaeger
f5a2ef4748 Translations: Fix sync of releasenotes
Only generate releasenotes from master branch and only if
they exist.

Change-Id: Iaff4375fc370592484df2d6917bcdd9f15d380b6
2016-04-18 20:11:51 +02:00
Andreas Jaeger
cb645e72af Translations: Refactor
Merge extract_messages and extract_messages_log into one function
extract_messages_python since these are always used together.

Merge propose_python and propose_django in one function propose,
call the extract_messages_* function outside of it.

Refactor python and django handling in a single function that can
be invoked twice instead of duplicating lines.

Change-Id: Ib92df685f9268e2429a119f619c1ab7e9ff1b82a
2016-04-18 19:37:35 +02:00
Andreas Jaeger
5cb5247c5c Translations: Fix releasenotes
Since we have now first translations of releasenotes, we can test
downloading of these. Testing barbican, nova, and openstack-manuals on
my local system found a couple of problems that are fixed with this
patch.

Refactor how we handle releasenotes, let's handle them like python and
django projects with specific functions.

Run the same cleanup on releasenotes like on other pot and po
files.

Fix download of releasenotes translations for openstack-manuals, previously
zanata-cli complained about a missing rule for these.

Remove from setup_project the special reno_resource setup, this is not
needed, the general rule works just fine.

Also, openstack-manuals does not set up the venv, so no need to remove
it.

Include minor cleanup for cleanup_pot_files.

Change-Id: I8c9a018b859b983f39c5239224bb530ce7500cb3
2016-04-18 08:53:09 +02:00
Andreas Jaeger
a1dd4fee2f Remove VENV before pushing to Zanata
For horizon, we pushed 10+ files to Zanata, since all pot files in the
VENV were pushed as well.

Remove the VENV before we push to the translation server.

Change-Id: I236af881b2fa46bf1f676ebe06513fb27accaf5f
2016-04-15 19:59:50 +02:00
Andreas Jaeger
adb9486bb8 Use pybabel instead of python setup.py, enable sync
We use already pybabel for django projects, let's use it also for python
projects. This works around a bug in Babel 2.3.3 with keyword option
parsing and unifies on using pybabel for both python and django.

Also extract the installation of horizon so that it's only done once for
modules that have multiple django modules.

This change uses Babel 2.2.0 now but works with 2.3.3 as well.

Reenable syncing of translation.

Change-Id: Ie220fc9418ae31941d182885e44237dc7fafe10a
2016-04-15 07:04:52 +02:00
Andreas Jaeger
5e64905611 Disable translations completely
Babel 2.3.3 has broken all python projects. I expect that with the way
Zanata works, we will get back all strings but right now all log-level
strings are extracted in the wrong way.

Temporarily disable translations completely. We do not loose any data
due to this and win time to discuss a proper plan.

Change-Id: Iae8d6b32736b1dd9cff3142c00a27399e4862304
2016-04-13 10:34:44 +02:00
Akihiro Motoki
2ce411f7f6 Translation job scripts support multiple python modules
At now, horizon repo has two python modules and we have
special handling in the translation job scripts.
This commit handles multiple python modules per repository
and locale directory will be prepared per python module.

After this we no longer need to have dedicated functions
for horizon.

Change-Id: Id32fbf37489c0056e5d7722fc3e7c71a95213e08
2016-02-22 15:00:17 +09:00
Andreas Jaeger
24cadad668 Cleanup translation scripts
Remove now unused functions and rename new functions:
* Removed setup_project, renamed setup_django to setup_project
* Removed extract_messages, renamed extract_messages_new to
  extract_messages
* Removed extract_messages_log, renamed extract_messages_log_new to
  extract_messages_log.

Remove quotation marks around 0/1 for integer comparisons.

Change-Id: Iacdd8bd3a1f715bda14b402af7ba36044c345465
2016-02-03 08:04:39 +01:00
Andreas Jaeger
8e9b206191 Re-enable repos for translation
The python-client, networking, and neutron repos are setup
properly again, so we can enable translation syncing again.

Depends-On: I836f7a11943a3d76fb0a119b401ead112680d7a7
Change-Id: Id2a5c00ddf46d43330e28fa60f8850c96de13f0c
Story: 2000452
2016-02-03 08:03:14 +01:00
Andreas Jaeger
a97afd8b83 Re-Enable django_openstack_auth/designate-dashboard translations
Also, remove liberty jobs for these so that we do not need to backport
or handle them extra.

Backporting would break packaging by third parties, therefore not an
option.

Designate-dashboard setup is broken in liberty anyhow, so no need to
handle it extra. django_openstack_auth is good translated, and we're at
the end of translation.

Besides re-enabling the translations using the new setup, we can now
remove the old function setup_django_openstack_auth since it's not used
anymore.

Story: 2000452
Change-Id: I8b72a148d0f0e89d61807eee50a772e1fbcecaa2
2016-02-03 08:02:48 +01:00
Jenkins
5e96e9aeda Merge "Use new translation setup for all python projects" 2016-02-02 09:20:51 +00:00
Andreas Jaeger
5738b0dba5 Use new translation setup for all python projects
Use the new translation setup for all python projects. This now uses
it for most python projects and those will just work without any change
with the new setup since for them modulename and reponame are the same.
For example for nova, both are "nova".

In addition, enable new setup for vmware-nsx and ironic-inspector - the
projects are ready now.

Note this change temporarily disables python-novaclient and
python-neutronclient - instead of making the script complicated, we wait
until all python projects are ready and then reenable all.

This also removes the old propose_python function that is not needed
anymore and renames propose_python_new to propose_python.

Remove also some outdated comments.

Story: 2000452
Change-Id: I5b89f7b168a862773282dc0400763b1a94ea9e10
2016-02-02 08:40:04 +01:00
Jenkins
2476c7a3fb Merge "Do not sync translations: django_openstack_auth/designate-dashbaord" 2016-02-01 13:49:37 +00:00
Andreas Jaeger
a1063fe298 Re-enable oslo translations
All changes for olso.* have been merged and the translations renamed
on our translation server, we can now handle translations the "new" way.
Remove temporary disabling of them and use the new functions.

This also adds a missing invocation of setup_log_level_vars to the setup
of python projects. Without it, extract_messages_log_level will fail.

Change-Id: I66311726c93b54099ad149e6d06a01ec2f9b2edc
Story: 2000452
2016-01-31 17:41:17 +00:00
Akihiro Motoki
a504a6de76 Do not sync translations: django_openstack_auth/designate-dashbaord
Disable syncing of translations for django_openstack_auth and
designate-dashboard until we have renamed file in zanata and the repo for it.

Change-Id: Ie2462614ce92ee09a4d84481b4df9410f57f597b
Story: 2000452
2016-01-30 19:37:10 +01:00
Akihiro Motoki
2da8eb7f10 Use new translation setup for all horizon plugin repos
New django translation setup scripts can applied to all
horizon related repositories. The exceptions are horizon
and django_openstack_auth.

This fixes the failure in the re-enabled trove-dashboard
translation jobs. We should do this before re-enabling it.

Change-Id: I90015ccf531db9f62a54d5b5c022886798db73be
Story: 2000452
2016-01-29 18:04:37 +01:00
Jenkins
47313b62d5 Merge "Prepare new translation functions for python projects" 2016-01-29 16:53:52 +00:00
Andreas Jaeger
a25c9013f6 Blacklist some more repos for translation sync
These repositories need to get stopped as well, since we will rename
them also.

Before we can enable them again, we need to update Zanata and the
repositories itself.

Change-Id: I9986833819566f39280f5a3944a49c8f7de9ebe5
2016-01-28 14:12:18 +01:00
Akihiro Motoki
f49e0112a4 Prepare new translation functions for python projects
python-neutronclient and python-novaclient repos should work with
the new scripts
(though the current locale files still exist in the repo).
The current script will be replaced after the migration completes.

This script supports $modulename/locale/$modulename.pot as POT filename.

Part of infra-spec Improve Translation Setup

Story: 2000452
Change-Id: I67bfab533a8fd48f038c10e82509a2e626205a33
2016-01-28 13:52:20 +01:00
Andreas Jaeger
3a67c34373 Do not sync translations for some projects
Disable syncing of translations for python-* and oslo.* projects until
we have renamed files in zanata for them.

Change-Id: I511d9bebda56cdc16b769cc5c0c60e62ae00e97c
Story: 2000452
2016-01-28 09:09:25 +01:00
Akihiro Motoki
db00502c88 magnum-ui use generic django translation processing
New generic django translation functions can be used with magnum-ui.
magnum-ui specific functions are no longer required.

Also add a note on django_openstack_auth.

Change-Id: I71a1bc78612adefa5f81b89d9fe7e0a1517e08d5
2016-01-26 06:29:52 +09:00
Andreas Jaeger
8af2b26682 Add new django extraction
- Create a new extract_django function that can be used
  for all django repositories.
- Create a new setup_django function to setup translation server
  for django repositories.
- Create a new propose_django function to propose translations.
- Extra a module name from setup.cfg.

Use these functions for murano-dashboard.

Co-Authored-By: Akihiro Motoki <amotoki@gmail.com>
Change-Id: I75c61ca709e05740c8932218b132752828b90b91
Story: 2000452
2016-01-26 06:22:55 +09:00
Andreas Jaeger
edcac0194f Make translation scripts more robust
In case of new projects, the locale directory might not exist. Create it
so that the jobs do not fail.

Change-Id: I9b8af9ae61a026d351d00ae93f1d74877e5925b2
2016-01-24 19:55:24 +01:00
KATO Tomoyuki
cc392a5a8f Add training-guides translation jobs and scripts
Change-Id: I129197fa9a711d2d1ee6b6b24889331b96af1514
2015-11-16 07:40:39 +00:00
Andreas Jaeger
66e39fd25f Fix up magnum-ui setup
Magnum-ui is setup similar to horizon with two django pot files. Invoke
run_tests.sh here.

Change-Id: I09350bfd384f84d132dc03d9bae2a5015acb6b0f
2015-11-03 08:58:03 +01:00
Steve Kowalik
00ad7446f1 Properly deal with magnum-ui for Zanata
magnum-ui is a special snowflake in that its project name is magnum-ui,
but the directory base for source and translations is magnum_ui. Since
it's a Django project, it also isn't treated like a general Python
project anyway, so treat it like django_openstack_auth, by performing
a little refactoring.

Since the repo now contains files, and is marked for translations in
projects.yaml anyway, enable it for translations jobs.

Change-Id: I7eb73181cccae6d4bfb81848d39099d3502a044c
2015-10-23 16:43:29 +11:00
Steve Kowalik
c7836c092e Add a new script that queries for a Zanata version
To allow us to check if a version exists for a Zanata project without
requiring us to create a zanata.xml first, add a small script that
uses the Zanata REST API to query for its existance.

Make use it instead of the zanata-cli in upstream_translation_update,
since we can't create a zanata.xml if the version doesn't exist.

Change-Id: I93a72845aa705ce6b72acec9b5d62b5565368ab0
2015-09-24 16:23:54 +10:00
Clark Boylan
1a06e58eae Push version specific .pots to Zanata
As our release versions change keep Zanata up to date with those
changes by pushing them to specific versions in Zanata.

Note that because Zanata can't handle /'s in version names we replace
/'s with -'s and map between git and Zanata that way. We also will exit
0 for success if the current git branch does not have a corresponding
version in Zanata.

Change-Id: I806b2a01c8d4fec2d149be3647931f95b559e76f
2015-09-23 10:34:04 -07:00
Jenkins
143bdd6226 Merge "Translations: Refaktor" 2015-09-22 20:31:31 +00:00
Andreas Jaeger
56ccfa16cf Translations: Refaktor
Create extract_messages function that can be used by both python and
django_openstack_auth repositories.

This enables the extra keywords for django_openstack_auth as well.

Change-Id: Ica998700215db5a0eae6aa88f4462ae416af2bdd
2015-09-22 20:08:36 +02:00
Jenkins
692f3ccb7d Merge "Disable translation copy" 2015-09-22 17:19:49 +00:00
Andreas Jaeger
325d4b435f Disable translation copy
By default Zanata copies translations from other versions, let's disable
this for now, it currently leads to undesired effects.

See also
http://docs.zanata.org/en/latest/user-guide/translation-reuse/copy-trans/

Change-Id: I4ed5ad34c1c42ac70926a379b58e2dd72cb6a750
2015-09-22 18:57:22 +02:00
Steve Kowalik
8575d3ce8e Collapse two cases in upstream_translation_update
Building on the excellent work done in
I54d560398daa67af8effeb794fce11e53affa1a6, collapse two case blocks
into one.

Change-Id: I9dfe79ed12fa93c30aab3e52a8357c44d1cba938
2015-09-22 16:05:44 +10:00
Andreas Jaeger
1c6307887d Merge upstream_translation scripts
Merge upstream_translation_update_manuals.sh,
upstream_translation_horizon.sh, and
upstream_translation_django_openstack_auth.sh into
upstream_translation_update.sh.

Adjust job definition for these projects so that the new scripts
get used.

Adjust layout.yaml for change, remove now obsolete condition.

Change-Id: I54d560398daa67af8effeb794fce11e53affa1a6
2015-09-21 20:56:59 +02:00
Andreas Jaeger
0f78e7bf19 Remove usage of transifex client
Remove all invocations of transifex client "tx".

This removes also the script populate-zanata-project.sh that is not
needed anymore, since we have switch to Zanata and started translating
there. The script is broken with the removal of "tx" support in other
places and not needed anymore.

Change-Id: I74350c92da425a6bf32ea1debd217a531dc2f400
2015-09-18 21:29:59 +02:00
Steve Kowalik
7111d2d7c4 Stop pushing to Transifex
This removes the code from all four upstream_translation_* scripts
that pushes new .pot files up to Transifex, and forces the comments
around them to be in line.

This leaves Transifex setup in place for the moment, since the
propose jobs that pull from Transifex are still utilising it.

Change-Id: I702e6073f92faaa79a7aa53c44022f9e437721af
2015-09-11 12:30:23 +10:00
Andreas Jaeger
152efbc3bc Handle empty PO files
Some projects like oslo.messaging and oslo.reports have now empty PO
files. Pushing empty files to transifex results in an error.

Check before pushing of the main file whether it's empty and only
push if it's non-empty - like it's done for the log-level PO
files already.

Change-Id: Ia757e9d4b88bc8eb7b1ce800a25fe0ed1ebe3e76
2015-08-26 14:53:55 +02:00
Steve Kowalik
66d1b67ae7 No longer ignore exit status of zanata-cli push
Now that the self-signed certificate for the Zanata server is stored
in the default Java keystore, zanata-cli push for the translation
upstream update jobs is working nicely, so we can stop using if to
guard the non-zero exit status from the script.

Change-Id: I75d892c9fe207b19ef263bb09b9beaf25848fbaf
2015-08-05 11:42:47 +10:00
Steve Kowalik
b8883366c9 Ignore zanata-cli return code
Running zanata-cli push on the proposal slave is turning up some
errors that require investigation. To give us some time to investigate
them, ignore errors from zanata-cli push.

Change-Id: I56a59027a9f8931e1a226383107b526fe7b90e68
2015-07-24 19:20:47 +10:00