Added Docs for TripleO Zuul Job parenting

Signed-off-by: Chandan Kumar (raukadah) <chkumar@redhat.com>
Change-Id: I96594e510208d12b682afc706b071e3a4c1b564d
This commit is contained in:
Chandan Kumar (raukadah) 2022-03-30 12:13:34 +05:30
parent 39ddc0ce71
commit 9b85a666ea
2 changed files with 417 additions and 0 deletions

View File

@ -18,3 +18,4 @@ TripleO CI Guide
third_party_dependencies_ci
content_provider_jobs
tripleo_dependencies_pipelines
tripleo_ci_job_parenting

View File

@ -0,0 +1,416 @@
TripleO CI Zuul Jobs Parenting
==============================
When a developer submits a patch to TripleO repositories, their code is
tested against a series of different TripleO CI jobs.
Each job creates a different scenario for testing purposes.
The TripleO CI jobs are Zuul jobs, defined within TripleO projects under
one of several locations: `zuul.d`_ directory, .zuul.yaml or zuul.yaml.
A Zuul job can be inherited in various child jobs as `parent`_.
Zuul Job Parenting
++++++++++++++++++
In order to re-use a particular Zuul job, we create
a set of standard base jobs, which contain
ansible variables, required projects, pre-run, run,
post-run steps and Zuul related variables.
These base job definitions are used as `parent`_ in various tripleo-ci
jobs. The child job inherits attributes from the parent unless
these are overridden by the child.
A child job can override the variable which is also defined
in parent job.
TripleO CI Base jobs
++++++++++++++++++++
TripleO CI base jobs are defined in `zuul.d/base-upstream.yaml`_ file
in tripleo-ci repo.
Below is the list of base jobs and each is explained in a little more detail
in subsequent sections:
* tripleo-ci-base-common-required-projects
* tripleo-ci-base-standard
* tripleo-ci-base-multinode-standard
* tripleo-ci-base-singlenode-standard
* tripleo-ci-base-standalone-standard
* tripleo-ci-base-standalone-upgrade-standard
* tripleo-ci-base-ovb-standard
* tripleo-ci-base-containers-standard
* tripleo-ci-base-images-standard
* tripleo-ci-content-provider-standard
tripleo-ci-base-common-required-projects
----------------------------------------
It contains a list of common required projects and ansible roles
which are needed to start the deployment. It is used in
upstream, RDO and Downstream.
If a new project is needed in all types of deployment
(upstream, RDO and Downstream) it can be added here.
tripleo-ci-base-standard
------------------------
It contains a set of ansible variables and playbooks used in
most deployments.
tripleo-ci-base-multinode-standard
----------------------------------
It contains a set of ansible variables and playbooks used in
most containers multinode and scenarios job.
It is used in those jobs where the user needs to deploy
OpenStack using one undercloud and one controller.
tripleo-ci-base-singlenode-standard
-----------------------------------
It contains a set of ansible variables and playbooks used in
most single node jobs.
It is used in those jobs where user needs to build containers
and overcloud images which later can be used in another deployment.
It can also be used for undercloud deployment.
tripleo-ci-base-standalone-standard
-----------------------------------
It contains a set of ansible variables and playbooks used in
most standalone and standalone scenario jobs.
The standalone job consists of single node overcloud deployment.
tripleo-ci-base-standalone-upgrade-standard
-------------------------------------------
It contains a set of ansible variables and playbooks used in
the standalone upgrade job.
The singlenode job consists of single node overcloud deployment
where we upgrade a deployment from an older release to a newer one.
tripleo-ci-base-ovb-standard
----------------------------
It contains a set of ansible variables and playbooks used in
the virtual baremetal deployment.
The ovb job consists of one undercloud and multiple overcloud
deployment having one compute and multiple controllers
on virtual baremetals nodes. It is a replica of
real world customer deployments.
It is used in RDO and downstream jobs.
tripleo-ci-base-containers-standard
-----------------------------------
It contains a set of ansible variables and playbooks used
during build containers and pushing it to specific registry.
tripleo-ci-base-images-standard
-------------------------------
It contains a set of ansible variables and playbooks used
during build overcloud images and pushing it to image server.
tripleo-ci-content-provider-standard
------------------------------------
It contains a set of ansible variables and playbooks used for
building containers and pushing it to local registry,
building depends-on patches into respective rpm packages via DLRN and
serving a local yum repos.
The job is paused to serve container registry and yum repos which later
on used in dependent jobs.
Currently these jobs are running in Upstream and Downstream.
Required Project Jobs
+++++++++++++++++++++
It contains the list of required projects needed for specific type
of deployment.
For example: `tripleo-ci-build-containers-required-projects-upstream`_ have
required projects like ansible-role-container-registry,
kolla, python-tripleoclient, tripleo-ansible used in Upstream for building containers.
In RDO, we have `tripleo-ci-build-containers-required-projects-rdo`_ for the same.
Many Upstream OpenStack projects are forked downstream and have different
branches.
To accommodate the downstream namespace and branches we can use a downstream
specific required project job (*required-projects-downstream*)
as a base job with proper branches and override-checkout.
Below is one of the example of container multinode required projects job.
`Upstream`_ ::
- job:
name: tripleo-ci-base-required-projects-multinode-upstream
description: |
Base abstract job to add required-projects for Upstream Multinode Jobs
abstract: true
parent: tripleo-ci-base-multinode-standard
required-projects:
- opendev.org/openstack/tripleo-ansible
- opendev.org/openstack/tripleo-common
- opendev.org/openstack/tripleo-operator-ansible
- name: opendev.org/openstack/ansible-config_template
override-checkout: master
`RDO`_ ::
- job:
name: tripleo-ci-base-required-projects-multinode-rdo
abstract: true
description: |
Base abstract job for multinode in RDO CI zuulv3 jobs
parent: tripleo-ci-base-multinode-standard
pre-run:
- playbooks/tripleo-rdo-base/pre.yaml
- playbooks/tripleo-rdo-base/container-login.yaml
roles:
- zuul: opendev.org/openstack/ansible-role-container-registry
- zuul: opendev.org/openstack/tripleo-ansible
required-projects:
- opendev.org/openstack/ansible-role-container-registry
- opendev.org/openstack/tripleo-ansible
secrets:
- rdo_registry
vars:
registry_login_enabled: true
Downstream ::
- job:
name: tripleo-ci-base-required-projects-multinode-internal
description: |
Base abstract job to add required-projects for multinode downstream job
abstract: true
override-checkout: <downstream branch name>
parent: tripleo-ci-base-multinode-standard
required-projects:
- name: tripleo-ansible
branch: <downstream-branch>
- ansible-config_template
- tripleo-operator-ansible
- rdo-jobs
- tripleo-environments
roles:
- zuul: rdo-jobs
pre-run:
- playbooks/configure-mirrors.yaml
- playbooks/tripleo-rdo-base/cert-install.yaml
- playbooks/tripleo-rdo-base/pre-keys.yaml
vars:
mirror_locn: <downstream mirror address>
featureset_override:
artg_repos_dir: /home/zuul/src/<downstream-url>/openstack
Distribution Jobs
+++++++++++++++++
The TripleO deployment is supported on multiple distro versions.
Here is the current supported martrix in RDO. Downstream and Upstream.
+----------+------------------------------+-------------+
| Release | CentOS/CentOS Stream Version |RHEL Version |
+==========+==============================+=============+
| Master | 9-Stream |- |
+----------+------------------------------+-------------+
| Wallaby | 8-Stream, 9-Stream |8.x, 9 |
+----------+------------------------------+-------------+
| Victoria | 8-Stream |- |
+----------+------------------------------+-------------+
| Ussuri | 8-Stream |- |
+----------+------------------------------+-------------+
| Train | 7, 8-Stream |8.x |
+----------+------------------------------+-------------+
Each of these distros have different settings which are used in deployment.
It's easier to maintain separate variables based on distributions.
Below is the example of distro jobs for containers multinode at different levels.
`Upstream Distro Jobs`_ ::
- job:
name: tripleo-ci-base-multinode
abstract: true
description: |
Base abstract job for multinode TripleO CI C7 zuulv3 jobs
parent: tripleo-ci-base-required-projects-multinode-upstream
nodeset: two-centos-7-nodes
- job:
name: tripleo-ci-base-multinode-centos-8
abstract: true
description: |
Base abstract job for multinode TripleO CI centos-8 zuulv3 jobs
parent: tripleo-ci-base-required-projects-multinode-upstream
nodeset: two-centos-8-nodes
- job:
name: tripleo-ci-base-multinode-centos-9
abstract: true
description: |
Base abstract job for multinode TripleO CI centos-9 zuulv3 jobs
parent: tripleo-ci-base-required-projects-multinode-upstream
nodeset: two-centos-9-nodes
`RDO Distro Jobs`_ ::
- job:
name: tripleo-ci-base-multinode-periodic
parent: tripleo-ci-base-multinode-rdo
pre-run: playbooks/tripleo-ci-periodic-base/pre.yaml
post-run: playbooks/tripleo-ci-periodic-base/post.yaml
required-projects:
- config
- rdo-infra/ci-config
roles:
- zuul: rdo-infra/ci-config
secrets:
- dlrnapi
- job:
name: tripleo-ci-base-multinode-periodic-centos-8
parent: tripleo-ci-base-multinode-rdo-centos-8
pre-run: playbooks/tripleo-ci-periodic-base/pre.yaml
post-run: playbooks/tripleo-ci-periodic-base/post.yaml
required-projects:
- config
- rdo-infra/ci-config
roles:
- zuul: rdo-infra/ci-config
vars:
promote_source: tripleo-ci-testing
secrets:
- dlrnapi
- job:
name: tripleo-ci-base-multinode-periodic-centos-9
parent: tripleo-ci-base-multinode-rdo-centos-9
pre-run: playbooks/tripleo-ci-periodic-base/pre.yaml
post-run: playbooks/tripleo-ci-periodic-base/post.yaml
required-projects:
- config
- rdo-infra/ci-config
roles:
- zuul: rdo-infra/ci-config
vars:
promote_source: tripleo-ci-testing
secrets:
- dlrnapi
Zuul Job Inheritance Order
++++++++++++++++++++++++++
Here is the example of Upstream inheritance of tripleo-ci-centos-9-containers-multinode_ job.::
tripleo-ci-base-common-required-projects
|
v
tripleo-ci-base-standard
|
v
tripleo-ci-base-multinode-standard
|
v
tripleo-ci-base-required-projects-multinode-upstream
|
v
tripleo-ci-base-multinode-centos-9
|
v
tripleo-ci-centos-9-containers-multinode
Here is the another example of RDO job periodic-tripleo-ci-centos-8-containers-multinode-master_ ::
tripleo-ci-base-multinode-standard
|
v
tripleo-ci-base-required-projects-multinode-rdo
|
v
tripleo-ci-base-multinode-rdo-centos-8
|
v
tripleo-ci-base-multinode-periodic-centos-8
|
v
periodic-tripleo-ci-centos-8-containers-multinode-master
TripleO CI Zuul Job Repos
+++++++++++++++++++++++++
Below is the list of repos where tripleo-ci related Zuul jobs are defined.
Upstream
--------
* `tripleo-ci <https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d>`_
RDO
---
* `config <https://github.com/rdo-infra/review.rdoproject.org-config/tree/master/zuul.d>`_: Jobs which needs secrets are defined here.
* `rdo-jobs <https://github.com/rdo-infra/rdo-jobs/tree/master/zuul.d>`_
FAQs regarding tripleo CI jobs
++++++++++++++++++++++++++++++
* If we have a new project, which needs to be tested at all places
and installed from source but
- cloned from upstream source, then it needs to be added under required-projects
at tripleo-ci-base-common-required-projects job.
- the project namespace is different in Upstream and downstream, then it needs to be
added under required-projects at
Downstream (tripleo-ci-base-required-projects-multinode-internal) or
Upstream (tripleo-ci-base-required-projects-multinode-upstream) specific
required-projects parent job.
- if the project is only developed at downstream or RDO or Upstream, then it needs
to be added under required project at downstream or RDO or Upstream required-projects
parent job.
* In order to add support for new distros, please use required-projects job as a
parent and then create distro version specific child job with required nodeset.
* If a project with different branch is re-added in child job required-projects,
then the child job project will be used in the deployment.
* If a playbook (which calls another role, exists in different repo ) is called at
pre-run step in Zuul job, then role specific required projects and roles needs
to be added at that job level. For example: In `tripleo-ci-containers-rdo-upstream-pre`_
job, ansible-role-container-registry and triple-ansible is needed for pre.yaml playbook.
So both projects are added in roles and required-projects.
* If a job having pre/post run playbook needs zuul secrets and playbook depends on
distros, then the job needs to be defined in config repo.
* We should not use branches attributies in Zuul Distro jobs or options jobs.
.. _`zuul.d`: https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d
.. _`parent`: https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.parent
.. _`zuul.d/base-upstream.yaml`: https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/base-upstream.yaml
.. _`tripleo-ci-build-containers-required-projects-rdo`: https://github.com/rdo-infra/rdo-jobs/commit/86e7e63ce6da27c2815afa845a6878cf96acdb47#diff-4897e02c92e2979a54f09d6eb383dba74c9a9211b065a52f9ecc4efbcce19637R17
.. _`tripleo-ci-build-containers-required-projects-upstream`: https://opendev.org/openstack/tripleo-ci/commit/1d640d09fd808caa33b82f0bdd5622120cebef09
.. _`Upstream`: https://opendev.org/openstack/tripleo-ci/src/commit/9e270ea7f8c19fc3902a38d87a7ea4ace8219cd9/zuul.d/multinode-jobs.yaml#L17
.. _`RDO`: https://github.com/rdo-infra/review.rdoproject.org-config/commit/b96b916fb2446171f5040ba8168c470a79f1befa#diff-80b60a19d10a7b56e22da7bfc1926e4e8d2143670b3ec3f26d009bda8e8910bfR527
.. _`Upstream Distro Jobs`: https://github.com/openstack/tripleo-ci/commit/9e270ea7f8c19fc3902a38d87a7ea4ace8219cd9#diff-7653508e44c2cd8de8b5140648d7583c5efb27f0012155ff21f83c22edad69a3R29-R57
.. _`RDO Distro Jobs`: https://github.com/rdo-infra/review.rdoproject.org-config/commit/b96b916fb2446171f5040ba8168c470a79f1befa#diff-80b60a19d10a7b56e22da7bfc1926e4e8d2143670b3ec3f26d009bda8e8910bfR574-R616
.. _`periodic-tripleo-ci-centos-8-containers-multinode-master`: https://review.rdoproject.org/zuul/job/periodic-tripleo-ci-centos-8-containers-multinode-master
.. _`tripleo-ci-centos-9-containers-multinode`: https://zuul.openstack.org/job/tripleo-ci-centos-9-containers-multinode
.. _`tripleo-ci-containers-rdo-upstream-pre`: https://opendev.org/openstack/tripleo-ci/commit/05366af2930d76b4791a0fcb1f8ed9fddb132721