109 Commits

Author SHA1 Message Date
Steve Kowalik
28eb4ce6f5 Remove support for pulling from Transifex
Switch the translation scripts to only pull from Zanata, no matter
what extra arguments they are passed on the command line. This does
not completly remove support for Transifex, which will be cleaned up
in a future commit.

Change-Id: I21bd54d4205693315da1152a310ff4faf22531a5
2015-09-16 15:53:14 +10:00
Elizabeth K. Joseph
d23e14688d Change translations proposal topic for zanata
Translations proposals are being moved to being submitted by Zanata rather than
Transifex, reflect this in the Gerrit topic.

Change-Id: I9bd7dc8c6cd761cdcd50b7ec0dcd34e57b253f47
2015-09-11 10:28:59 -07:00
Steve Kowalik
7b8769c2c9 Set DocFolder correctly for Zanata
Since Zanata has trouble mapping .pot files that don't start with ./
for projects that set trans-dir and src-dir to ., make use of a new
variable to set them correctly for all manuals projects.

Change-Id: I729633db3e76cc07e0ec3d5083c15fa73770a21c
2015-09-09 16:47:19 +10:00
Jenkins
b408a630ee Merge "Change rule for django_openstack_auth" 2015-09-09 04:03:21 +00:00
Steve Kowalik
bd1308d3e9 Change rule for django_openstack_auth
The default rule for django_openstack_auth does not match anything since
the .po files are not named the same as the .pot file, set the rule
manually.

Change-Id: I32971259531ed18141a2011621b69d12085cbe5f
2015-09-09 13:25:20 +10:00
Steve Kowalik
b8d6965fc5 Fix the rule pattern for Horizon
The rule pattern for finding .pot files within Horizon is far too
locked down, and will not match anything. Correct it to use paths
that will match.

Change-Id: I7ca45dec0ea4067d8d88a6ac99f11cf9da4a8b35
2015-09-09 13:07:31 +10:00
Steve Kowalik
f54eb1ed69 Redo function that pulls translations from Zanata
The existing function pull_from_zanata would only work for the simple
Python project case of pulling translations from Zanata -- with only
locales directory and trivial rules to apply, with no real hope of
working for Horizon or OpenStack Manuals, given the current
functionality of the Zanata CLI utility.

Rewrite it to download every translation from Zanata and perform some
checks by hand if we keep the file or remove it. This also required
refactoring a large portion of cleanup_po_files so both functions could
leverage the same code.  OpenStack Manuals treats downloaded
translations sufficiently different that a similar function was required.

This also adds full support for Horizon and OpenStack Manuals projects,
including jobs, as well as new job templates to make it easier to
transition projects to pulling from Zanata. Adding -zanata jobs for the
Python projects and django_openstack_auth is not required, they are
already present.

Change-Id: I5b9f2eca7f099c56057fc5d09b4bbfe5ea5c596d
2015-09-08 17:10:45 +00:00
Steve Kowalik
fdab16cc23 Remove errant } in create-zanata-xml for Horizon
This causes the Zanata CLI utility to print an error message about
accepted variables, and it may cause problems for the parser.

Change-Id: If46bd2baf8ee12394b61a9d505fd9f7e690f1b23
2015-09-03 14:46:53 +10:00
Steve Kowalik
029594205e Do not ignore the exit status creating Zanata XML
Since we're getting closer to switching over to Zanata, we want to
be notified of problems, and most, if not all projects do not fail
when generating the zanata.xml file, no longer mask the exit status
by using an if block.

Change-Id: I0e013df1048ab564ce72124dbdacce2065807d52
2015-08-28 14:37:16 +10:00
Steve Kowalik
b15242b47c Really add common exclusion for manuals repos
*/source/common/* will only match one path element either side, so
we need to open the glob a little wider to match correctly. It also
appears that the other manuals repos also have source/common as a
symlink, so remove the openstack-manuals guard.

Change-Id: Id6e883ec388d9d5f6d471fa8b23f8e3f2ed1885a
2015-08-27 17:25:08 +10:00
Steve Kowalik
f55193d3b9 Add common exclusion for Zanata openstack-manuals
*/source/common in the openstack-manuals repository is a symbolic
link to common-rst, which results in pushing the files twice, and
may result in confusion. Add that path to the excludes for
openstack-manuals only.

Change-Id: I6067d94d71d3d025fc81cf331c84d5b214752218
2015-08-26 16:12:35 +10:00
Steve Kowalik
770b73d35e Switch locale marker to locale_with_underscore
Zanata 3.7.1 has moved much of the locale logic to the server, which
means we can no longer make use of client side transforms, such as
map-from, so set the locale to locale_with_underscore so dashes are
replaced with underscores.

Change-Id: I0bde5b84388f2be39c85600870ec530e8b5c07a5
2015-08-20 13:39:11 +10:00
Jenkins
50eaf49461 Merge "Add support for pulling translations from Zanata" 2015-08-12 19:14:40 +00:00
Steve Kowalik
3ca25f7204 Add support for pulling translations from Zanata
To help move to Zanata, we should pull down translations from
the server. Add a function that should behave the same way as
the existing Transifex function, as well as changing two existing
scripts to call it based on a command-line parameter. Add two new
jobs with a -zanata suffix so we can switch projects one at a time.

This change deals with all of the Python projects and
django_openstack_auth. Horizon and OpenStack manuals will be dealt with
in a future patch.

Change-Id: I8003ac37aeb89d59ceeba2e75eed2f8100b82367
2015-08-06 18:12:50 +10:00
Andreas Jaeger
7268fd329d Fix glossary recognition
Fix last change to be applied at proper place.

The glossary test needs to be in the !RST case, move it over.

Change-Id: Ib629ad85edd7f57288c94056154d1359126a4eac
2015-08-05 20:32:13 +02:00
Andreas Jaeger
a9af3157bf Fix translation sync with Transifex
Transifex now reserves glossary as part of the name, so rename it (as
done in transifex) to glossary-1.

Glossary is only used for openstack-manuals, so we do this change only
at that place.

Change-Id: I6e0e678b9c19c3a5fec882ede40679a7c8b73c5a
2015-08-05 11:33:48 +02:00
Peng Wu
ce43c75f24 Update jenkins jobs for more-gettext-support blueprint
Refer URL:
https://blueprints.launchpad.net/oslo.i18n/+spec/more-gettext-support
https://review.openstack.org/#q,topic:bp/more-gettext-support,n,z

Implements: blueprint more-gettext-support

Depends-on: I258eac447ecc7b71fb02952077cf3ef3ecfe12bb

Change-Id: I493e4698cd430792395fd77325fc0efab0ea917d
2015-07-28 16:22:52 +08:00
Jenkins
d3dfbb8f88 Merge "Add excludes support for create-zanata-xml" 2015-07-14 22:00:24 +00:00
Steve Kowalik
990b4d7737 Add excludes support for create-zanata-xml
For projects such as Manuals and Horizon, the source directory in
zanata.xml is set to '.', which may have the unintended side-effect
of the Zanata client finding .po or .pot files under dot directories.
Add a new option to create-zanata-xml, and use it for those two
projects.

Change-Id: I5fc26069c288eca130a45c6e9c5e3dd00ad2c784
2015-07-13 15:24:05 +10:00
Steve Kowalik
85f99e2d18 Refactor pulling translations into a common func
In preperation for switching projects one at a time to pulling
translations from Zanata, refactor the two calls to tx pull into
a common function.

Change-Id: I21051fbdcf7bd210606333ba46e5abb0c64e7540
2015-07-09 16:45:42 +10:00
Steve Kowalik
17011a2620 Add Zanata support for openstack-manuals projects
Building on existing work for supporting pushing translations
to Zanata for other projects, add support for pushing translations
for the four openstack-manuals projects.

Change-Id: Ic18331b7ee402c4c89b6dacdc4edc02c3ef07cdc
2015-07-03 15:23:23 +10:00
Jenkins
eec80fadb5 Merge "Change Zanata project for django_openstack_auth" 2015-06-30 15:58:49 +00:00
Jenkins
0b9dd9d83f Merge "Handle missing pot files in translation proposal" 2015-06-30 15:26:23 +00:00
Steve Kowalik
7222ffe07b Change Zanata project for django_openstack_auth
It turns out that Zanata can not share a project like Transifex
does, the client gets all jealous and removes all other translation
files when you push the second project. To that end, change the
project of django_openstack_auth to its own name rather than horizon.

Change-Id: I8ddcf18f3fd2549ee0d948041bb62da7f669e76b
2015-06-30 17:03:13 +10:00
Steve Kowalik
544eb51c73 Sprinkle in Horizon translations for Zanata
Building on work landed in II3bfb188c8b0c0e65f22d7edc30721b163f084fff,
add support for pushing source documents of Horizon into Zanata.
This has also necessitated adding support for parsing and adding
rules for create-zanata-xml.py.

As a drive-by correct some incorrect docstrings in ZanataUtils, split
some ludicrously long lines calling create-zanata-xml.py, and correct
usage of an undefined variable in setup_django_openstack_auth.

Change-Id: Id7a2c82e979d1f878022b4982f51ef4ef06ecf4b
2015-06-24 18:47:06 +10:00
Andreas Jaeger
cac641f921 Translations: Update horizon
Horizon uses pot files now as all other projects, adjust scripts for the
change.

Add new Javascript translation for openstack_dashboard.

Remove now unused function compress_non_en_po_files.

Change-Id: Ibd7069c68e458d655a879e552f69f7146b47abc9
Depends-On: I2ae68207994ba549b5d8e551a40a468d63e0a4bb
2015-06-22 19:27:36 +02:00
Andreas Jaeger
b2c02a9e52 Handle missing pot files in translation proposal
searchlight project had only one pot file but not the other five that
are expected. The filter_commit function was filtering out the newly
added file and run "git checkout -- NEWFILE" which failed.

Make the script more robust so that it handles newly added files
correctly.

Change-Id: I4a7a8ab9b75f310992b7232920206d4ddc041268
2015-06-20 09:26:38 +02:00
Steve Kowalik
9ec48c03fe Support pushing translations to Zanata
In order to migrate to Zanata for translations, we need to modify
the proposal scripts to also upload translations to the Zanata
server. This commit supports most projects (those that make use of
upstream_translation_update.sh), as well as django_openstack_auth.

This does not impact the existing push and pull of
translations to Transifex, and also untouched are the translations
for Horizon or the manuals, which are all handled in separate scripts.

Co-Authored-By: stephane <stephane@alum.mit.edu>
Change-Id: I3bfb188c8b0c0e65f22d7edc30721b163f084fff
2015-06-19 00:19:51 +10:00
Andreas Jaeger
a0e27a5933 Fix filter_commit in common_translation_update.sh
This fixes a problem with propose_translation_update.sh on
oslo.concurrency:

+ git reset -q oslo.concurrency/locale/en_GB/LC_MESSAGES/oslo.concurrency-log-error.po
fatal: ambiguous argument
'oslo.concurrency/locale/en_GB/LC_MESSAGES/oslo.concurrency-log-error.po': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

Add the "--" to separate paths to handle deleted files as well.

Change-Id: Ifbce50bd851ea0028ebbb948e5173bda5777fc43
2015-06-16 20:11:07 +02:00
Steve Kowalik
e2c58de6fe Drop usage of `` in if statements
Subshells are not required for if statements, so remove them. This
also drops all usage of `` in scripts.

Change-Id: I44c476ddfb67e5b3ebd3aaba3b26561f02708477
2015-06-04 15:44:14 +10:00
Steve Kowalik
e3060b094c Switch to using $() for subshells, part 2
Cleanup every use of `` for subshells in the jenkins/scripts
directory, replacing them with $(), and finally making the
scripts consistent.

Change-Id: Id3e45fb873c1ebeae6ab63a0caa4c427a7ccbb62
2015-06-04 15:33:48 +10:00
Jenkins
34f35f003a Merge "Switch to using test -n" 2015-06-01 23:18:20 +00:00
Jenkins
5dc343c2c0 Merge "Don't repeat the commit message" 2015-06-01 23:18:13 +00:00
Jenkins
cc44e097d8 Merge "Move git config to a common function/file" 2015-06-01 23:16:23 +00:00
Jenkins
16b4dc7ba9 Merge "Ignore last-translator for translation update trigger" 2015-06-01 22:31:47 +00:00
Andreas Jaeger
7166dc85d8 Sort PO files to avoid random reorder
Changes like
https://review.openstack.org/#/c/185848/1/horizon/locale/ca/LC_MESSAGES/djangojs.po
just reordered entries with no content change. Pass --sort-output to
msgattrib to filter the entries in alphabetical order so that any
rearrangements of the source files will not have an effect on the order
of the entries.

Change-Id: I2d9c91f3d1fbcf61ae21521758836793742389ed
2015-05-27 16:43:11 +02:00
Steve Kowalik
f4a2c9802d Switch to using test -n
Using the construct of x$var != x is unnecessary, and has been
unnecessary for quite some time. We trust our shell, so switch to
using test -n.

Change-Id: I8eea943e5b641ba3d8781212877da7ddec167a6c
2015-05-21 13:39:55 -07:00
Steve Kowalik
c7ac961329 Move git config to a common function/file
There are a number of calls to git config in the slave scripts are
all the same, and we should use a shell function to not repeat
ourselves.

Change-Id: Ib2fa3d5ca12d5d08e2d21a734fea13aa9848b19e
2015-05-20 14:17:05 -07:00
Steve Kowalik
1201673176 Don't repeat the commit message
When we edit the commit message for sending the patch for updating
translations, we repeat the entire text of the commit message,
rather than using the existing variable (which is fine, because the
shell will expand the variable before assigning it).

Change-Id: I2840935d2174b1931de0cb782e52cfed4c049312
2015-05-20 11:53:52 -07:00
Andreas Jaeger
05b95e5434 Ignore last-translator for translation update trigger
Do not import a file if the Last-Translator line has changed (together
with other ignored lines).

This will ignore imports like
https://review.openstack.org/#/c/183771/1/doc/glossary/locale/ja.po

If there are other changes, including real translations, the file will
get imported as usual.

Change-Id: I6f56f64c107e4f93b92460612e656b91341a8338
2015-05-16 10:00:48 +02:00
Andreas Jaeger
a848b7e577 Propose translations: Fix handling of deleted files
Today's propose_translation_update.sh run failed for ceilometer since
files were deleted but the loop in filter_commit considered these
as zero changing lines. Change the loop to check only new and modified
files.

Change-Id: I94f5389b9fe0c16d529860f57bb63ef8a3c5c64a
2015-05-11 09:16:44 +02:00
Jenkins
fb9472006b Merge "Only import files if there are new translations" 2015-05-08 14:31:27 +00:00
Jenkins
ee71a44768 Merge "Decrease PO files for imports (horizon+friends)" 2015-05-07 19:01:36 +00:00
Andreas Jaeger
fe09040a32 Minor translation cleanups
Remove .tx file at the end to have a clean repository.

Run setup.py with parameter --quiet, the long list of "extracting
messages from file XXX" are not needed.

Change-Id: I735f02a968aed8b362803781498ccc8085fa0e03
2015-05-03 09:55:00 +02:00
Andreas Jaeger
f2a7e45591 Only import files if there are new translations
Do not import files if only content is deleted but no new translation
added. If there's at least one file with added translations, files with
removed content are also imported.

This should reduce the number of imports, since a project should see an
import only if new strings are in transifex.

Change-Id: Id521a2209bf9cad3b90acf81447db027beed7b64
2015-05-02 13:39:07 +02:00
Andreas Jaeger
c294627405 Decrease PO files for imports (horizon+friends)
Remove untranslated strings and location information from PO files
for horizon and django_openstack_auth. For horizon, the
location information is still in the English PO files as reference for easy
regeneration using "msgmerge EnglishPOFILE POFILE -o COMPLETEPOFILE".

Change-Id: Ibb440320e0fd882c86ff79ec295411c3d062a521
2015-04-29 16:09:18 +02:00
Andreas Jaeger
9ee5653f21 Fix translation compress for manuals
Not all manual repositories have a doc directory, adjust the paths.
Not all files should be compressed, we need to exclude in all
repositories - besides openstack-manuals - the glossary.

Change-Id: I358db42585699b05e893b66e1ac6d2927fb43ee3
2015-04-24 08:51:56 +02:00
Andreas Jaeger
41fe0097a4 Decrease PO files for Doc team imports
Remove untranslated strings and location information from PO files. The
location information is still in the POT files as reference for easy
regeneration using "msgmerge POTFILE POFILE -o COMPLETEPOFILE".

Our tools handle the smaller PO files just fine.

Enable this for Documentation projects.

Also, quote file name arguments in cleanup_po_files.

Change-Id: I4649314f68332d9db46a10aac0d7550bddc46d0e
2015-04-23 07:24:08 +02:00
Andreas Jaeger
8e31e6cd26 Fix path of RST locale file
locale files in RST folders are under $DOCNAME/source/locale/, fix one
error.

Also, small refactor of an condition.

Change-Id: I9a61b2d0e26bd19e2b0c1b8b95b2369530b05fcf
2015-03-09 18:38:17 +01:00
Andreas Jaeger
e8e61d6107 Support translation of RST files
The OpenStack manuals are switching to some RST files. Enhance our
scripts so that they can handle RST files in manual projects.

Change-Id: Ie3e9591d2aad88aced3092aa2a8919af186ce83d
Depends-On: I3124cf65b254c1c5aefce59447bb94240d96a742
2015-03-06 22:23:42 +01:00