From 78a21fa83087f39fe8deea00e872f0f2408c29a8 Mon Sep 17 00:00:00 2001 From: Akihiro Motoki Date: Thu, 4 Feb 2021 09:12:59 +0900 Subject: [PATCH] Document tips on debugging translation job scripts The need for this information is rare but when I develop the translation job scripts I always research the same thing so it is useful at least for me. Change-Id: I83f38fd673456e0d1179c50d9df903e16b01bd5f --- doc/source/index.rst | 1 + doc/source/infra-debug.rst | 152 +++++++++++++++++++++++++++++++++++++ doc/source/infra.rst | 5 ++ 3 files changed, 158 insertions(+) create mode 100644 doc/source/infra-debug.rst diff --git a/doc/source/index.rst b/doc/source/index.rst index 7168936..f61cd34 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -35,6 +35,7 @@ Contents tools i18n_team_meeting infra + infra-debug project_setup reviewing-translation-import release_management diff --git a/doc/source/infra-debug.rst b/doc/source/infra-debug.rst new file mode 100644 index 0000000..6684a2f --- /dev/null +++ b/doc/source/infra-debug.rst @@ -0,0 +1,152 @@ +===================== +Debugging job scripts +===================== + +While it would be rare, you may would like to debug the translation +job scripts. This section describes tips for such cases. +``propose_translation_update.sh`` is mainly covered. + +.. note:: + + It is not a complete document for debugging. + It was just written as a note. Feel free to add more topics. + +Environment +----------- + +It is better to use the same environment (e.g., Linux distribution version) +used in the translation job. To find the information, check ``nodeset`` of the +``propose-translation-update`` job definition in +`zuul configuration `__ +in openstack/project-config repository. +As of Feb 2021, ``ubuntu-bionic`` is used. + +Preparations +------------ + +You can prepare dependencies by following +`prepare-zanata-client role `__. + +What you need are: + +* Installing Zanata CLI and its dependencies. + Ensure ``zanata-cli`` command is found in your path. +* Prepare ``~/.config/zanata.ini``. + See :ref:`Zanata CLI ` for more detail. +* Copy the translation scripts to your working directory from + `openstack-zuul-jobs/roles/prepare-zanata-client/files `__. + In addition, you need to copy ``common.sh`` from + `project-config/roles/copy-proposal-common-scripts/files `__ + to the same working directory. + +Commenting out CI-specific codes +-------------------------------- + +The job scripts contain codes specific to OpenStack CI. +For example, there is a code to communicate with OpenStack gerrit +and the account is hardcoded. +It looks convenient to comment out such code blocks +to debug the scripts in a local env. It should be useful +unless you are debuging a CI-specific issue. + +Code blocks which look better to be commented out are: + +* common_translation_update.sh + + * ``trap "finish" EXIT`` (It depends on testrepository and usually fails in + local envs) + +* propose_translation_update.sh + + * ``setup_review`` and a logic to check an existing change + (It is unnecessary as we do not propose a real change to gerrit. + The account used is hardcoded and it always fails in local runs.) + * ``send_patch`` (It is unnecessary as we do not want to propose + a real change to gerrit.) + * (optional) + The main logic (from ``case "$PROJECT" in`` to ``filter_commits``) + (we will debug the main logic, so perhaps we would like to run the script + piece by piece.) + +The diff would be like http://paste.openstack.org/show/802260/. + +Copying upper-constraints.txt +----------------------------- + +The job scripts assume that upper-constraints.txt exists in the top directory of +the target project repository. + +.. code-block:: console + + $ cd $PROJECT_DIR + $ wget https://releases.openstack.org/constraints/upper/master + +Note that you need to adjust the URL of the upper-constraints file +when you work on a stable branch. + +Creating ~/.venv virtualenv +--------------------------- + +The job scripts assume that required python modules are installed in +``~/.venv`` virtualenv. In the zuul job, this virtualenv is prepared via +``ensure-sphinx`` and ``ensure-babel`` roles in zuul/zuul-jobs. +You need to do the same. + +Note that doc/requirements.txt in most projects are almost samer so perhaps you +can reuse this virtualenv for most projects. If it does not work, consider +recreating the virtualenv. + +.. code-block:: bash + + $ python3 -m venv ~/.venv + $ . ~/.venv/bin/activate + # $PROJECT_DIR is a target project repository like horizon, ironic-ui .... + (.venv) $ cd $PROJECT_DIR + # Install sphinx related modules ensure-sphinx installs + (.venv) $ pip install -r doc/requirements.txt -c ../requirements/upper-constraints.txt + # Install modules ensure-babel installs + (.venv) $ pip install Babel lxml pbr requests -c ../requirements/upper-constraints.txt + (.venv) $ deactivate + +Loading scripts +--------------- + +.. code-block:: bash + + # It is better to start a new shell when debugging the script + $ bash + > cd $PROJECT_REPO + # The arguments are: {{ zuul.project.short_name }} {{ zuul.branch }} {{ zuul.job }} $HORIZON_REPO + > . $WORKDIR/propose_translation_update.sh ironic-ui master propose-translation-update ../horizon + > + +Simulating translation changes +------------------------------ + +If you would like to simulate translation changes, you can download translations +in advance (using zanata-cli) and modify them as you want. +In such case, you can modify the script as follows: + +.. code-block:: console + + --- a/roles/prepare-zanata-client/files/common_translation_update.sh + +++ b/roles/prepare-zanata-client/files/common_translation_update.sh + @@ -734,7 +732,8 @@ function pull_from_zanata { + # Since Zanata does not currently have an option to not download new + # files, we download everything, and then remove new files that are not + # translated enough. + - zanata-cli -B -e pull + + #zanata-cli -B -e pull + + cp -r $DOWNLOAD_TRANSLATIONS/$project/* . + + # We skip directories starting with '.' because they never contain + # translations for the project (in particular, '.tox'). Likewise + +You can download translations as below. +``zanata.xml`` will be created once you run the propose_translation_update.sh +(or you can prepare it by following :ref:`zanata-cli` "Project configuration"). + +.. code-block:: console + + $ cd $DOWNLOAD_TRANSLATIONS/$project + $ zanata-cli -B -e pull --project-config $PROJECT_DIR/zanata.xml diff --git a/doc/source/infra.rst b/doc/source/infra.rst index 6b44fc3..8b47422 100644 --- a/doc/source/infra.rst +++ b/doc/source/infra.rst @@ -95,3 +95,8 @@ repository. Note that the scripts in the tasks use `zanata-cli `__ to pull and push translation content. + +Debugging job scripts +--------------------- + +See :doc:`infra-debug`.