Commit Graph

57 Commits

Author SHA1 Message Date
Andreas Jaeger
717ca3b9bf Propose translations: Fix invalid bash
Remove extra $s from local declaration.

Change-Id: I259bfa1f52349223ddd77d7aa0fd26257721748c
2016-04-20 09:24:29 +02:00
Andreas Jaeger
cb645e72af Translations: Refactor
Merge extract_messages and extract_messages_log into one function
extract_messages_python since these are always used together.

Merge propose_python and propose_django in one function propose,
call the extract_messages_* function outside of it.

Refactor python and django handling in a single function that can
be invoked twice instead of duplicating lines.

Change-Id: Ib92df685f9268e2429a119f619c1ab7e9ff1b82a
2016-04-18 19:37:35 +02:00
Jenkins
e9432c01c4 Merge "Translations: Cleanup django modules" 2016-04-18 14:28:10 +00:00
Jenkins
adc444a019 Merge "Translations: Fix releasenotes" 2016-04-18 14:22:58 +00:00
Andreas Jaeger
67e91f7801 Translations: Handle non-translated module
extract_messages removes the pot file if it does not contain any
translations. But then git add will fail since nothing is to add.

This currently breaks keystone proposal:
http://logs.openstack.org/periodic/keystone-propose-translation-update/2ef243f

Add a test for empty directory and exist early if it exists.

Change-Id: I3cce7f81856d579dde36445f722ba513a29fae61
2016-04-18 09:19:58 +02:00
Andreas Jaeger
5cb5247c5c Translations: Fix releasenotes
Since we have now first translations of releasenotes, we can test
downloading of these. Testing barbican, nova, and openstack-manuals on
my local system found a couple of problems that are fixed with this
patch.

Refactor how we handle releasenotes, let's handle them like python and
django projects with specific functions.

Run the same cleanup on releasenotes like on other pot and po
files.

Fix download of releasenotes translations for openstack-manuals, previously
zanata-cli complained about a missing rule for these.

Remove from setup_project the special reno_resource setup, this is not
needed, the general rule works just fine.

Also, openstack-manuals does not set up the venv, so no need to remove
it.

Include minor cleanup for cleanup_pot_files.

Change-Id: I8c9a018b859b983f39c5239224bb530ce7500cb3
2016-04-18 08:53:09 +02:00
Andreas Jaeger
2e256f8f91 Translations: Cleanup django modules
Remove untranslated and unused po/pot files for django modules - the
same way we do for python projects.

Change-Id: I06e71e29a727a40c960b3721333dda9e0bde7b11
2016-04-17 21:46:02 +02:00
Andreas Jaeger
0af0f35a2a translations: Do not merge untranslated pot files
Do not storate pot files that are not translated currently, remove
them from any proposals.

Change-Id: I9ded8b892a4034bccf27c43d0d3721a97eab6bb0
2016-04-17 19:29:38 +02:00
Andreas Jaeger
adb9486bb8 Use pybabel instead of python setup.py, enable sync
We use already pybabel for django projects, let's use it also for python
projects. This works around a bug in Babel 2.3.3 with keyword option
parsing and unifies on using pybabel for both python and django.

Also extract the installation of horizon so that it's only done once for
modules that have multiple django modules.

This change uses Babel 2.2.0 now but works with 2.3.3 as well.

Reenable syncing of translation.

Change-Id: Ie220fc9418ae31941d182885e44237dc7fafe10a
2016-04-15 07:04:52 +02:00
Andreas Jaeger
5e64905611 Disable translations completely
Babel 2.3.3 has broken all python projects. I expect that with the way
Zanata works, we will get back all strings but right now all log-level
strings are extracted in the wrong way.

Temporarily disable translations completely. We do not loose any data
due to this and win time to discuss a proper plan.

Change-Id: Iae8d6b32736b1dd9cff3142c00a27399e4862304
2016-04-13 10:34:44 +02:00
Akihiro Motoki
2ce411f7f6 Translation job scripts support multiple python modules
At now, horizon repo has two python modules and we have
special handling in the translation job scripts.
This commit handles multiple python modules per repository
and locale directory will be prepared per python module.

After this we no longer need to have dedicated functions
for horizon.

Change-Id: Id32fbf37489c0056e5d7722fc3e7c71a95213e08
2016-02-22 15:00:17 +09:00
Andreas Jaeger
1e8c6eedf0 Make propose_translation_update more robust
If there're no translations at all, the locale directory is empty
but we try to add content and that fails like:
 + git add 'ironic_ui/locale/*'
 | fatal: pathspec 'ironic_ui/locale/*' did not match any files

Do not use a wildcard, just add the directory. That way the git command
and will not fail - and the rest of the script works fine.

Change-Id: I2a67b692f437a1bced9507e66cce217765d5d913
2016-02-11 21:20:24 +01:00
Andreas Jaeger
24cadad668 Cleanup translation scripts
Remove now unused functions and rename new functions:
* Removed setup_project, renamed setup_django to setup_project
* Removed extract_messages, renamed extract_messages_new to
  extract_messages
* Removed extract_messages_log, renamed extract_messages_log_new to
  extract_messages_log.

Remove quotation marks around 0/1 for integer comparisons.

Change-Id: Iacdd8bd3a1f715bda14b402af7ba36044c345465
2016-02-03 08:04:39 +01:00
Andreas Jaeger
8e9b206191 Re-enable repos for translation
The python-client, networking, and neutron repos are setup
properly again, so we can enable translation syncing again.

Depends-On: I836f7a11943a3d76fb0a119b401ead112680d7a7
Change-Id: Id2a5c00ddf46d43330e28fa60f8850c96de13f0c
Story: 2000452
2016-02-03 08:03:14 +01:00
Andreas Jaeger
a97afd8b83 Re-Enable django_openstack_auth/designate-dashboard translations
Also, remove liberty jobs for these so that we do not need to backport
or handle them extra.

Backporting would break packaging by third parties, therefore not an
option.

Designate-dashboard setup is broken in liberty anyhow, so no need to
handle it extra. django_openstack_auth is good translated, and we're at
the end of translation.

Besides re-enabling the translations using the new setup, we can now
remove the old function setup_django_openstack_auth since it's not used
anymore.

Story: 2000452
Change-Id: I8b72a148d0f0e89d61807eee50a772e1fbcecaa2
2016-02-03 08:02:48 +01:00
Jenkins
221de77380 Merge "Test translation imports" 2016-02-02 23:31:48 +00:00
Jenkins
5e96e9aeda Merge "Use new translation setup for all python projects" 2016-02-02 09:20:51 +00:00
Andreas Jaeger
5738b0dba5 Use new translation setup for all python projects
Use the new translation setup for all python projects. This now uses
it for most python projects and those will just work without any change
with the new setup since for them modulename and reponame are the same.
For example for nova, both are "nova".

In addition, enable new setup for vmware-nsx and ironic-inspector - the
projects are ready now.

Note this change temporarily disables python-novaclient and
python-neutronclient - instead of making the script complicated, we wait
until all python projects are ready and then reenable all.

This also removes the old propose_python function that is not needed
anymore and renames propose_python_new to propose_python.

Remove also some outdated comments.

Story: 2000452
Change-Id: I5b89f7b168a862773282dc0400763b1a94ea9e10
2016-02-02 08:40:04 +01:00
Jenkins
2476c7a3fb Merge "Do not sync translations: django_openstack_auth/designate-dashbaord" 2016-02-01 13:49:37 +00:00
Andreas Jaeger
a1063fe298 Re-enable oslo translations
All changes for olso.* have been merged and the translations renamed
on our translation server, we can now handle translations the "new" way.
Remove temporary disabling of them and use the new functions.

This also adds a missing invocation of setup_log_level_vars to the setup
of python projects. Without it, extract_messages_log_level will fail.

Change-Id: I66311726c93b54099ad149e6d06a01ec2f9b2edc
Story: 2000452
2016-01-31 17:41:17 +00:00
Akihiro Motoki
a504a6de76 Do not sync translations: django_openstack_auth/designate-dashbaord
Disable syncing of translations for django_openstack_auth and
designate-dashboard until we have renamed file in zanata and the repo for it.

Change-Id: Ie2462614ce92ee09a4d84481b4df9410f57f597b
Story: 2000452
2016-01-30 19:37:10 +01:00
Andreas Jaeger
aa3d342839 Test translation imports
Test imported files whether those are valid (can be compiled by
gettext) and do not propose them if they are invalid.

If there are invalid files, a proposal of valid files will still happen.
The script will fail at the end so that these invalid files can be
detected.

Change-Id: Ia48181bc79903dd47be054a6f7c7ee535506d282
2016-01-30 17:05:26 +01:00
Akihiro Motoki
2da8eb7f10 Use new translation setup for all horizon plugin repos
New django translation setup scripts can applied to all
horizon related repositories. The exceptions are horizon
and django_openstack_auth.

This fixes the failure in the re-enabled trove-dashboard
translation jobs. We should do this before re-enabling it.

Change-Id: I90015ccf531db9f62a54d5b5c022886798db73be
Story: 2000452
2016-01-29 18:04:37 +01:00
Jenkins
47313b62d5 Merge "Prepare new translation functions for python projects" 2016-01-29 16:53:52 +00:00
Andreas Jaeger
a25c9013f6 Blacklist some more repos for translation sync
These repositories need to get stopped as well, since we will rename
them also.

Before we can enable them again, we need to update Zanata and the
repositories itself.

Change-Id: I9986833819566f39280f5a3944a49c8f7de9ebe5
2016-01-28 14:12:18 +01:00
Akihiro Motoki
f49e0112a4 Prepare new translation functions for python projects
python-neutronclient and python-novaclient repos should work with
the new scripts
(though the current locale files still exist in the repo).
The current script will be replaced after the migration completes.

This script supports $modulename/locale/$modulename.pot as POT filename.

Part of infra-spec Improve Translation Setup

Story: 2000452
Change-Id: I67bfab533a8fd48f038c10e82509a2e626205a33
2016-01-28 13:52:20 +01:00
Andreas Jaeger
3a67c34373 Do not sync translations for some projects
Disable syncing of translations for python-* and oslo.* projects until
we have renamed files in zanata for them.

Change-Id: I511d9bebda56cdc16b769cc5c0c60e62ae00e97c
Story: 2000452
2016-01-28 09:09:25 +01:00
Akihiro Motoki
db00502c88 magnum-ui use generic django translation processing
New generic django translation functions can be used with magnum-ui.
magnum-ui specific functions are no longer required.

Also add a note on django_openstack_auth.

Change-Id: I71a1bc78612adefa5f81b89d9fe7e0a1517e08d5
2016-01-26 06:29:52 +09:00
Andreas Jaeger
8af2b26682 Add new django extraction
- Create a new extract_django function that can be used
  for all django repositories.
- Create a new setup_django function to setup translation server
  for django repositories.
- Create a new propose_django function to propose translations.
- Extra a module name from setup.cfg.

Use these functions for murano-dashboard.

Co-Authored-By: Akihiro Motoki <amotoki@gmail.com>
Change-Id: I75c61ca709e05740c8932218b132752828b90b91
Story: 2000452
2016-01-26 06:22:55 +09:00
Andreas Jaeger
edcac0194f Make translation scripts more robust
In case of new projects, the locale directory might not exist. Create it
so that the jobs do not fail.

Change-Id: I9b8af9ae61a026d351d00ae93f1d74877e5925b2
2016-01-24 19:55:24 +01:00
Andreas Jaeger
0e44a256d2 Remove update_po from translation proposal script
So far we have updated some po files (files in a specific language) whenever
we downloaded files from our translation server. This was done
for python projects only but not for manuals and horizon - and
was broken for django_openstack_auth.

It was done to ensure that POT files (translation source) and PO files
(translated files in specific language) are in sync. But since we're
pushing to Zanata and downloading *all* translation files from there,
Zanata does the update step for us and we get all changes.

Remove update_po now, it's not needed.

Change-Id: I9a0dac230dbb8c47ae6d82aebbf7727c54f2d2cd
2016-01-15 10:55:04 +01:00
KATO Tomoyuki
e510f9b683 Fix wrong path name for training-guide translations
Change-Id: Icd69e35a2f9639f0fd370691f3a8b2c22da68709
2015-11-25 14:57:13 +00:00
KATO Tomoyuki
cc392a5a8f Add training-guides translation jobs and scripts
Change-Id: I129197fa9a711d2d1ee6b6b24889331b96af1514
2015-11-16 07:40:39 +00:00
Andreas Jaeger
66e39fd25f Fix up magnum-ui setup
Magnum-ui is setup similar to horizon with two django pot files. Invoke
run_tests.sh here.

Change-Id: I09350bfd384f84d132dc03d9bae2a5015acb6b0f
2015-11-03 08:58:03 +01:00
Steve Kowalik
00ad7446f1 Properly deal with magnum-ui for Zanata
magnum-ui is a special snowflake in that its project name is magnum-ui,
but the directory base for source and translations is magnum_ui. Since
it's a Django project, it also isn't treated like a general Python
project anyway, so treat it like django_openstack_auth, by performing
a little refactoring.

Since the repo now contains files, and is marked for translations in
projects.yaml anyway, enable it for translations jobs.

Change-Id: I7eb73181cccae6d4bfb81848d39099d3502a044c
2015-10-23 16:43:29 +11:00
Jenkins
463a2458ef Merge "Translations: Merge pull_from_zanata" 2015-09-25 18:49:05 +00:00
Clark Boylan
624832c5fa Make translation pushes to gerrit branch aware
To support multiple translation versions in Zanata we need to make the
reverse push into Gerrit branch aware as well. Do this by setting up the
existing propose translation jobs to accept a branch parameter which
tells them which version to pull from Zanata and where to push into
Gerrit.

This will add a new periodic jenkins job for each release/branch.

Change-Id: Ia206bfaa7dd6eaad6aa1f6e5ddbbbf9cdadf8db0
2015-09-23 10:35:01 -07:00
Andreas Jaeger
f40bcd2213 Translations: Merge pull_from_zanata
Merge the two pull_from_zanata functions. We do not need the special
50 % rule for manuals anymore.

Change-Id: I80ed31b5d59020ece3abcfc4bd9e6f8145dd566a
2015-09-23 08:40:02 +02:00
Andreas Jaeger
56ccfa16cf Translations: Refaktor
Create extract_messages function that can be used by both python and
django_openstack_auth repositories.

This enables the extra keywords for django_openstack_auth as well.

Change-Id: Ica998700215db5a0eae6aa88f4462ae416af2bdd
2015-09-22 20:08:36 +02:00
Andreas Jaeger
4dc0ca560e Merge translation proposal scripts
Merge
jenkins/scripts/propose_translation_update_django_openstack_auth.sh,
jenkins/scripts/propose_translation_update_horizon.sh, and
jenkins/scripts/propose_translation_update_manuals.sh into
jenkins/scripts/propose_translation_update.sh.

Update Jenkins and Zuul files, we can use now the same job
in all cases.

Change-Id: Idb9e8534dfc0454d75fe186e1a4a454832c8fdae
2015-09-22 20:07:52 +02:00
Andreas Jaeger
0f78e7bf19 Remove usage of transifex client
Remove all invocations of transifex client "tx".

This removes also the script populate-zanata-project.sh that is not
needed anymore, since we have switch to Zanata and started translating
there. The script is broken with the removal of "tx" support in other
places and not needed anymore.

Change-Id: I74350c92da425a6bf32ea1debd217a531dc2f400
2015-09-18 21:29:59 +02:00
Steve Kowalik
28eb4ce6f5 Remove support for pulling from Transifex
Switch the translation scripts to only pull from Zanata, no matter
what extra arguments they are passed on the command line. This does
not completly remove support for Transifex, which will be cleaned up
in a future commit.

Change-Id: I21bd54d4205693315da1152a310ff4faf22531a5
2015-09-16 15:53:14 +10:00
Steve Kowalik
f54eb1ed69 Redo function that pulls translations from Zanata
The existing function pull_from_zanata would only work for the simple
Python project case of pulling translations from Zanata -- with only
locales directory and trivial rules to apply, with no real hope of
working for Horizon or OpenStack Manuals, given the current
functionality of the Zanata CLI utility.

Rewrite it to download every translation from Zanata and perform some
checks by hand if we keep the file or remove it. This also required
refactoring a large portion of cleanup_po_files so both functions could
leverage the same code.  OpenStack Manuals treats downloaded
translations sufficiently different that a similar function was required.

This also adds full support for Horizon and OpenStack Manuals projects,
including jobs, as well as new job templates to make it easier to
transition projects to pulling from Zanata. Adding -zanata jobs for the
Python projects and django_openstack_auth is not required, they are
already present.

Change-Id: I5b9f2eca7f099c56057fc5d09b4bbfe5ea5c596d
2015-09-08 17:10:45 +00:00
Steve Kowalik
3ca25f7204 Add support for pulling translations from Zanata
To help move to Zanata, we should pull down translations from
the server. Add a function that should behave the same way as
the existing Transifex function, as well as changing two existing
scripts to call it based on a command-line parameter. Add two new
jobs with a -zanata suffix so we can switch projects one at a time.

This change deals with all of the Python projects and
django_openstack_auth. Horizon and OpenStack manuals will be dealt with
in a future patch.

Change-Id: I8003ac37aeb89d59ceeba2e75eed2f8100b82367
2015-08-06 18:12:50 +10:00
Steve Kowalik
85f99e2d18 Refactor pulling translations into a common func
In preperation for switching projects one at a time to pulling
translations from Zanata, refactor the two calls to tx pull into
a common function.

Change-Id: I21051fbdcf7bd210606333ba46e5abb0c64e7540
2015-07-09 16:45:42 +10:00
Steve Kowalik
e3060b094c Switch to using $() for subshells, part 2
Cleanup every use of `` for subshells in the jenkins/scripts
directory, replacing them with $(), and finally making the
scripts consistent.

Change-Id: Id3e45fb873c1ebeae6ab63a0caa4c427a7ccbb62
2015-06-04 15:33:48 +10:00
Jenkins
c563b2d6f4 Merge "Minor translation cleanups" 2015-05-05 06:44:29 +00:00
Andreas Jaeger
fe09040a32 Minor translation cleanups
Remove .tx file at the end to have a clean repository.

Run setup.py with parameter --quiet, the long list of "extracting
messages from file XXX" are not needed.

Change-Id: I735f02a968aed8b362803781498ccc8085fa0e03
2015-05-03 09:55:00 +02:00
Andreas Jaeger
19e84b070b Fix translation steps
Today's nova import had some file changes where only the
POT-Creation-Date was changed which we don't need to import.

With the new compression step, we need to filter out commits *after* the
compression, otherwise counting is wrong.

Change-Id: Ib8fe6ca5f77c805147b73fe94905c72ccf3369d4
2015-05-03 09:42:42 +02:00
Andreas Jaeger
e569eb9045 Handle empty translation files
We have some strange files in the tree that contain no or little
translation after merging - and these are downloaded from transifex.
So, these files are not known to git and git rm fails.
To not special case every git rm and git reset for both old and new
files, re-add a "git add" at the beginning so that we can operate on
files known to git.

Note: This happens when we have a file like glance-log-critical.po that has no
strings - but we had to create it with initial content in transifex,
since empty files are not allowed there.

Change-Id: Ie9bb3e59560d50829e56928b2ed79e3999dd39cb
2015-05-02 09:10:01 +02:00