From 34b622de8d862d0cf8c8e817fb3047b699eb7f54 Mon Sep 17 00:00:00 2001 From: Tristan Cacqueray Date: Thu, 31 May 2018 08:49:21 +0000 Subject: [PATCH] docs: add Project Testing Interface guide Change-Id: I7b4c85d2d4a07e2a431246d0499a2a6cbccb5f18 --- doc/source/user/howtos.rst | 12 + .../user/howtos/cross-project-gating.rst | 24 ++ doc/source/user/howtos/pti.rst | 214 ++++++++++++++++++ doc/source/user/index.rst | 1 + 4 files changed, 251 insertions(+) create mode 100644 doc/source/user/howtos.rst create mode 100644 doc/source/user/howtos/cross-project-gating.rst create mode 100644 doc/source/user/howtos/pti.rst diff --git a/doc/source/user/howtos.rst b/doc/source/user/howtos.rst new file mode 100644 index 0000000000..c1f4f3fdb0 --- /dev/null +++ b/doc/source/user/howtos.rst @@ -0,0 +1,12 @@ +:title: HowTos + +.. _howtos: + +HowTos: Recommended Practices +============================= + +.. toctree:: + :maxdepth: 2 + + howtos/pti + howtos/cross-project-gating diff --git a/doc/source/user/howtos/cross-project-gating.rst b/doc/source/user/howtos/cross-project-gating.rst new file mode 100644 index 0000000000..342c44cc81 --- /dev/null +++ b/doc/source/user/howtos/cross-project-gating.rst @@ -0,0 +1,24 @@ +:title: Cross Project Gating + +.. _cross-project-gating: + +Cross Project Gating +==================== + +The following sections describe how to test future state of cross projects +dependencies. + +Python +------ + +Python project's requirements are pulled from external source by default, and +the zuul-jobs tox playbook implement a special task to ensure the job required +projects are installed from the zuul src_root so that the gate effectively test +in flight project's state. + +Packaging +--------- + +Similarly, packaging project, where tests are performed using the final binary +artifacts, would need to inject a local repository to ensure the test is using +the change as prepared by Zuul. diff --git a/doc/source/user/howtos/pti.rst b/doc/source/user/howtos/pti.rst new file mode 100644 index 0000000000..c5fbfba7bb --- /dev/null +++ b/doc/source/user/howtos/pti.rst @@ -0,0 +1,214 @@ +:title: Project Testing Interface + +.. _pti: + +Project Testing Interface +========================= + +The following sections describe an example PTI (Project Testing Interface) +implementation. The goal is to setup a consistent interface for driving tests +and other necessary tasks to succesfully configure :ref:`project_gating` within +your organization. + + +Projects layout +--------------- + +A proper PTI needs at least two projects: + +* org-config: a :term:`config-project` curated by administrators, and +* org-jobs: a :term:`untrusted-project` to hold common jobs. + +The projects that are being tested or deployed are also +:term:`untrusted-project`, and for the purpose of this example we will use a +couple of integrated projects named: + +* org-server +* org-client + + +org-config +~~~~~~~~~~ + +The config project needs careful scrutiny as it defines priviledged Zuul +configurations that are shared by all your projects: + +:ref:`pipeline` triggers and requirements let you define when a change is +tested and what are the conditions for merging code. Approval from core +members or special labels to indicate a change is good to go are pipelines +configuration. + +The base job let you define how the test environment is validated before +the actual job is executed. The base job also defines how and where the job +artifacts are stored. + +More importantly, a config-project may enforce a set of integration jobs to +be executed on behalf of the other projects. A regular (untrusted-project) can +only manage its own configuration, and as part of a PTI implementation, you +want to ensure your projects' changes undergo validation that are defined +globally by your organization. + +Because the nature of those configuration settings are priviledged, +config-projects changes are only effective when merged. + + +org-jobs +~~~~~~~~ + +Jobs definition content are not priviledged Zuul settings and jobs can be +defined in a regular :term:`untrusted-project`. +As a matter of fact, it is recommended to define jobs outside of the config +project so that job updates can be tested before being merged. + +In this example, we are using a dedicated org-jobs project. + + +Projects content +---------------- + +In this example PTI, the organization requirements are a consistent code style +and an integration test to validate org-client and org-server works according +to a reference implementation. + +In the org-jobs project, we define a couple of jobs: + +.. code-block:: yaml + + # org-jobs/zuul.yaml + - job: + name: org-codestyle + parent: run-test-command + vars: + test_command: $code-style-tool $org-check-argument + # e.g.: linters --max-column 120 + + - job: + name: org-integration-test + run: integration-tests.yaml + required-projects: + - org-server + - org-client + +The integration-tests.yaml playbook needs to implement an integration test +that checks both the server and client code. + + +In the org-config project, we define a project template: + +.. code-block:: yaml + + # org-config/zuul.d/pti.yaml + - project-template: + name: org-pti + queue: integrated + check: + jobs: + - org-codestyle + - org-integration-test + gate: + jobs: + - org-codestyle + - org-integration-test + + +Finaly, in the org-config project, we setup the PTI template on both projects: + +.. code-block:: yaml + + # org-config/zuul.d/projects.yaml + - project: + name: org-server + templates: + - org-pti + + - project: + name: org-client + templates: + - org-pti + + +Usage +----- + +With the above layout, the organization projects use a consistent testing +interface. +The org-client or org-server does not need extra settings, all new +contribution shall pass the codestyle and integration-test as defined by +the organization admin. + + +Project tests +~~~~~~~~~~~~~ + +Projects may add extra jobs on top of the PTI. +For example, the org-client project can add a user interface test: + +.. code-block:: yaml + + # org-client/.zuul.yaml + - job: + name: org-client-ui-validation + + - project: + check: + jobs: + - org-client-ui-validation + gate: + jobs: + - org-client-ui-validation + +In this example, new org-client change will run the PTI's jobs as well as the +org-client-ui-validation job. + + +Updating PTI test +~~~~~~~~~~~~~~~~~ + +Once the PTI is in place, if a project needs adjustment, +it can proceed as follow: + +First a change on org-jobs is proposed to modify a job. For example, update a +codestyle check using such commit: + +.. code-block:: text + + # org-jobs/change-url + + Update codestyle to enforce CamelCase. + +Then, without merging this proposal, it can be tested accross the projects using +such commit: + +.. code-block:: text + + # org-client/change-url + + Validate new codestyle. + Depends-On: org-jobs/change-url + +Lastly the org-jobs may be enriched with: + +.. code-block:: text + + # org-jobs/change-url + + Update codestyle to enforce CamelCase. + Needed-By: org-client/change-url + + +.. note:: Extra care is required when updating PTI jobs as they affects all + the projects. Ideally, the org-jobs project would use a org-jobs-check + to run PTI jobs change on every projects. + + +Cross project gating +-------------------- + +The org-pti template is using the "integrated" queue to ensure projects change +are gated by the zuul scheduler. Though, the jobs need extra care to properly +test projects as they are prepared by Zuul. For example, the +org-integration-test playbook need to ensure the client and server are installed +from the zuul src_root. + +This is called sibbling installation, and it is critical piece to ensure cross +project gating. Check out the recommended practices :ref:`cross-project-gating`. diff --git a/doc/source/user/index.rst b/doc/source/user/index.rst index 4cf4c668c7..31ed7b6eb0 100644 --- a/doc/source/user/index.rst +++ b/doc/source/user/index.rst @@ -17,3 +17,4 @@ configure it to meet your needs. jobs encryption badges + howtos