diff --git a/goals/pike/python35.rst b/goals/pike/python35.rst index 8484fbee7..f725fe416 100644 --- a/goals/pike/python35.rst +++ b/goals/pike/python35.rst @@ -1,5 +1,7 @@ .. -*- mode: rst -*- +.. _goal-support-python-3.5: + ==================== Support Python 3.5 ==================== diff --git a/goals/stein/python3-first.rst b/goals/stein/python3-first.rst new file mode 100644 index 000000000..0b9d9d3c0 --- /dev/null +++ b/goals/stein/python3-first.rst @@ -0,0 +1,393 @@ +.. -*- encoding: utf-8 -*- + +=============================== + Run under Python 3 by default +=============================== + +As we agreed in the :ref:`python2-deprecation-timeline` resolution, +the next phase of our adoption of python 3 is to begin running all +jobs using python 3 by default, and only using python 2 to test +operating under python 2 (via unit, functional, or integration +tests). This goal describes the activities needed to move us to this +"python 3 first" state. + +:Storyboard Story: https://storyboard.openstack.org/#!/story/2002586 + +Champions +========= + +* Doug Hellmann (doug@doughellmann.com) +* Nguyễn Trí Hải (nguyentrihai93@gmail.com) +* Ma Lei (ma.lei@99cloud.net) +* Huang Zhiping (huang.zhiping@99cloud.net) + +Gerrit Topic +============ + +`python3-first `__ + +Completion Criteria +=================== + +1. The Zuul settings to attach jobs have been moved from the + ``openstack-infra/project-config`` repository into each project + repository. +2. Documentation build and publish jobs use python 3. +3. Release notes build and publish jobs use python 3. +4. Source code linter jobs (pep8, pylint, bandit, etc.) should use + python 3. +5. Release artifact build and publish jobs use python 3 and publish to + PyPI. +6. There are functional test jobs running under python 3. +7. The wiki tracking page is up to date for the project. +8. Projects are running python 3.6 unit test jobs. + +The goal champions will start by preparing the patches for steps +1-5. That will give project teams more time to focus on steps 6-8, +which will need the expertise of someone on the team. Project teams +should be prepared to review the changes quickly, and assist if there +are any questions about jobs with branch-specifiers (see below). Step +7, updating the wiki tracking page, is the responsibility of the +project team. Steps 6 and 8 involve adding more test jobs and may also +require code changes or tox settings updates, so those will be left to +the project team as well. + +Details +======= + +Moving settings from project-config +----------------------------------- + +A big part of the work for this goal for each project is to configure +different sets of jobs to run the same tasks differently on different +branches. Although Zuul supports doing this using branch specifiers, +it will be easier to maintain the job settings over time if we place +them in each repository. This lets us set different jobs, or variants +of jobs, to run against each branch just by modifying the Zuul +settings in that branch. + +The first step, therefore, is to take the project settings that +control the branch-aware jobs out of the +``openstack-infra/project-config`` repository and put them into each +git repository in the appropriate branch. + +https://review.openstack.org/580736/ is an example of adding the +project settings to the master branch of ``openstack/whereto``. The +patch was constructed by using the ``python3-first`` command from the +``openstack/goal-tools`` repository (see the ``README.rst`` there for +directions). + +.. note:: + + Not all of the job settings should be moved into the project tree: + + 1. The line applying the ``translation-jobs`` project template (or + any of its branch-specific variants) should stay in + ``openstack-infra/project-config``. + + 2. Settings for jobs that should only ever run on the ``master`` + branch should stay in ``openstack-infra/project-config``. + + 3. Any project templates that are not "branch-aware" (because they + run based on a tag event) need to remain configured in + ``zuul.d/projects.yaml`` in ``openstack-infra/project-config``. + +Each repository needs a patch like 574393 on the master branch. The +settings can go into any of the Zuul configuration files +(``.zuul.yaml``, ``zuul.yaml``, ``.zuul.d/project.yaml``, etc.) and +should be organized based on the way the project team has their Zuul +configuration set up already. + +Similar patches should then be created for the ``stable/rocky``, +``stable/queens``, ``stable/pike``, and ``stable/ocata`` branches. +Sometimes it will be possible to just backport the change from the +master branch, but many projects have jobs configured with branch +specifiers to control which branches run the tests. We do not want to +mix branch specifiers with branch-specific configuration, so the +settings will need to be modified or removed. This may mean that the +versions of the change on stable branches are different from the +version on master. The ``python3-first jobs extract`` command in the +``goal-tools`` repository should correctly compute the set of jobs to +move into each branch. + +After all of the settings are in place within the branches of a +repository, they can be removed from ``openstack-infra/project-config``. +https://review.openstack.org/580737/ shows how to remove the settings +for whereto. + +.. note:: + + Due to the large number of changes involved, we want to update the + ``openstack-infra/project-config`` repository with 1 patch per + project team. That patch **must not** be approved before all of the + settings are added to each repository in all of the relevant + branches. Normally we would manage that requirement using + ``Depends-On`` to link the patches together, but for some teams + that will produce a very very long list of inter-patch dependencies + (Oslo would take 128 patches). Rather than listing them all as + dependencies, therefore, we will wait to submit the update to + project-config until *after* all of the changes to the project + repositories have been approved. + +At this point, most of the job configuration for each repository can +be managed directly by the project team, and each branch will have a +separate list of jobs. This will make future job management much +simpler and should make it possible for the project team to do the +rest of the work for the goal without waiting for anyone else to +review their patches. + +Documentation Jobs +------------------ + +The ``publish-openstack-docs-pti`` project template defines a set of +jobs for testing and building documentation using tox as a wrapper +around Sphinx, and replaces the old ``publish-openstack-sphinx-docs`` +project template. The new jobs use a ``docs`` tox environment, defined +in each repository, so the tox settings can be used to provide a +self-testing patch to the repository to switch the version of python +used. + +https://review.openstack.org/580738 shows how to change the +project settings for a repository to use the new documentation +jobs. The change will be self-testing, and should only be made on the +master branch. + +https://review.openstack.org/572895 shows how to update the +tox.ini settings in the project to set the ``basepython`` variable for +the ``docs`` environment, used for developer testing. This change +should only be made on the master branch. + +Common issues to anticipate: + +* Under python 3 the output of subprocess.check_output() is a bytes + instance, but sphinx expects values for version and timestamps to be + str objects. Doc builds that do things in conf.py (or extensions) + like extract the modification date from the most recent commit will + need to properly decode the return + values. https://review.openstack.org/#/c/575483 shows one example of + how to fix this sort of problem. + +Release Notes Jobs +------------------ + +The ``release-notes-jobs-python3`` project template defines a set of +jobs for testing and building release notes using python 3. The +release notes jobs do not use tox, but the tox settings should still +be updated. + +In the Zuul configuration on the master branch of the repository, +change the project template ``release-notes-jobs`` to +``release-notes-jobs-python3``. If the patch to change the project +template does not run the new job, it may be necessary to add a dummy +release note to make the patch self-testing. + +https://review.openstack.org/#/c/572895/ shows how to update the +tox.ini settings in the project to set the ``basepython`` variable for +the ``releasenotes`` environment, used for developer testing. This +change should only be made on the master branch. + +Common issues to anticipate: + +* Under python 3 the output of subprocess.check_output() is a bytes + instance, but sphinx expects values for version and timestamps to be + str objects. Doc builds that do things in conf.py (or extensions) + like extract the modification date from the most recent commit will + need to properly decode the return values. + +Source Code Linter Jobs +----------------------- + +Most of the jobs we have that run source code linters *do* use tox to +control the versions of the linter tool. These jobs typically have +names like ``openstack-tox-linters`` or ``openstack-tox-pep8``. + +https://review.openstack.org/#/c/572895/ shows how to update the +tox.ini settings in the project to set the ``basepython`` variable for +the environments. All of the linter jobs running against python source +code in the master branch should be updated. These changes should be +self-testing. + +Any linter jobs that use python-based tools to check other sorts of +source should also be updated to use python 3, if possible. + +Common issues to anticipate: + +* The built-in ``file``() no longer exists under python 3, so using it + causes pylint to report an undefined name. Use ``open()`` instead. + +* The built-in ``unicode`` no longer exists under python 3, so using + it causes pylint to report an undefined name. Use ``six.text_type`` + instead. + +* Under python 3 the flake8/hacking/pep8/pylint tools run different or + additional checks. This may mean new code formatting issues will + have to be fixed as part of changing the linter jobs over. + +* There is a bug in the older version of pylint that many projects are + using that prevents it from working correctly under python 3. Pylint + will have to be upgraded as part of this transition; version 1.9.2 + is known to work. + + The error message from the broken version is:: + + AttributeError: 'Call' object has no attribute 'starargs' + + Updating the version of pylint brings new rules, and will require + modifications either to source code or to the pylint configuration. + https://review.openstack.org/#/c/573024/ is an example of updating + to the latest version of pylint in the freezer repository, with a + combination of fixes and disabling rules. + +Release Artifact Publishing +--------------------------- + +We will be making several changes to artifact publishing for +Python-based projects simultaneously. The job settings for the release +artifact publishing need to be defined in +``openstack-infra/project-config/zuul.d/projects.yaml`` rather than in +each project repository, because those jobs are not "branch aware" and +therefore we do not want different versions of the jobs on different +branches. + +First, a new job that uses ``setuptools`` to validate the packaging +metadata for a repository will run in the check and gate queues when +``README.rst``, ``setup.cfg``, or ``setup.py`` are modified. This will +be an early warning for issues that may come up as part of publishing +the build artifacts, and runs the same step that was added recently to +the validation job in ``openstack/releases`` as well as actually +building an sdist and a wheel. + +Second, the new packaging test, build, and publish jobs will all run +under python 3. + +Third, all python-based deliverables will have their sdists and wheels +published to PyPI. This will simplify dependency management between +plugins and server projects and will streamline the number of +variations of release jobs that we have. + +In order to make this change, project teams may first need to register +their project name on PyPI. Refer to `the creator's guide in the infra +manual`_ for details of how to do this. + +After the name is configured on PyPI, change any existing release +project template to +``publish-to-pypi-python3``. https://review.openstack.org/580740 shows +an example of changing the job setting. + +It is not possible to test the job change, because it needs to be made +in the ``openstack-infra/project-config`` repository. Therefore, after +the first change merges it will be useful to create a second patch in +the project repository with a whitespace or other typo-fix change in +the ``README.rst`` to trigger the packaging test job in this patch to +ensure everything works properly. https://review.openstack.org/580741 +shows an example of such a change. + +.. note:: + + Teams using release jobs that rely on python to publish artifacts + for projects not written in python (and therefore not covered by + ``publish-to-pypi-python3``) should work with the release and infra + teams to update their release jobs to use python 3. + +Common issues to anticipate: + +* Projects that have not published to PyPI before may need to fix + their ``README.rst`` file if it uses RST directives only defined by + Sphinx and not by docutils. The new test job will catch any issues. + +* Projects that cannot reserve their project name on PyPI because it + is owned by another community may need to change the sdist name in + their ``setup.cfg`` in order to be able to publish to PyPI under a + different name. That will not change how the code is imported, but + it will change package names and may require setting + ``tarball-base`` in the release settings managed in + ``openstack/releases``. The release management team can help if you + end up needing to change names, so contact them before starting to + make the change. + +.. _the creator's guide in the infra manual: https://docs.openstack.org/infra/manual/creators.html#give-openstack-permission-to-publish-releases + +.. _on the mailing list: http://lists.openstack.org/pipermail/openstack-dev/2018-June/131193.html + +Functional Test Jobs +-------------------- + +Updating the functional test jobs for a project will require more +knowledge of the jobs that exist, which ones need to be duplicated +under python 3, and which can be changed to python 3 without being run +under python 2. Changing the job configuration will require knowledge +of the job implementation details. For these reasons, the analysis and +implementation work for updating the functional test jobs is left up +to each project team. + +Libraries used by services that run in the default integrated gate can +add the ``lib-forward-testing-python3`` project template to ensure +they have full integration tests run. +https://review.openstack.org/#/c/575927/ shows an example of doing +this for oslo.config. + +Where possible, when modifying existing jobs, a variable should be +added to control the version of python so that the same job +implementation (playbooks, roles, etc.) can be used instead of +duplicating the entire job definition. This will simplify cleaning up +the old job definitions when python 2 support is finally dropped. + +It should be possible to update functional and integration test jobs +that run through tox by setting ``basepython = python3`` for the +appropriate tox environment, as in +https://review.openstack.org/#/c/572895/. + +Wiki Tracking Page +------------------ + +We have been using https://wiki.openstack.org/wiki/Python3 to track +the status of support in each project. Teams should keep the page up +to date with information about blockers, test jobs, etc. as they work +on this goal (and after, ideally). + +Python 3.6 Unit Test Jobs +------------------------- + +`On the mailing list`_ Zane proposed updating to test with Python 3.6 +when it is available. Adding those test jobs will be easier after the +Zuul configuration is moved out of the project-config repository, so +this step is left for last. Because adding the test job may require +code changes, it will be up to each project team to take this step by +adding ``openstack-python36-jobs`` to the list of templates associated +with the project on the master branch. The change will be +self-testing, and can either be structured to include the code changes +(if they are trivial) or end a series of patches (if the code changes +are significant). + +.. note:: + + We do not plan to update the minimum version of python 3 we support + as part of this goal. Projects already running python 3.5 jobs + should continue to do so. + +References +========== + +* :ref:`goal-support-python-3.5` +* `Updating python packaging jobs `__ +* `Configuring library forward testing jobs `__ +* `Planning etherpad `__ +* `Status of OpenStack projects + `__ + from the Python3 wiki page. + +Current State / Anticipated Impact +================================== + +A significant number of patches to update the tox settings for +projects have already been proposed and many have been merged: + +https://review.openstack.org/#/q/topic:python3-first + +Some of the Oslo libraries are using the python 3 versions of these +jobs already. + +Because the goal champion team will prepare a lot of the patches to +move the Zuul settings, we expect project teams to be able to focus on +unique aspects of their testing such as branch-specific jobs or +functional jobs. diff --git a/resolutions/20180529-python2-deprecation-timeline.rst b/resolutions/20180529-python2-deprecation-timeline.rst index c01f206ec..846959cdf 100644 --- a/resolutions/20180529-python2-deprecation-timeline.rst +++ b/resolutions/20180529-python2-deprecation-timeline.rst @@ -1,3 +1,5 @@ +.. _python2-deprecation-timeline: + ========================================= 2018-05-29 Python2 Deprecation Timeline =========================================