openstack
/
tripleo-docs
Code Issues Proposed changes

Add initial doc for compose pinning dependency jobs 

This patch adds documentation for CentOS compose pinning dependency
work. It explains how users can create their own jobs to use
compose-repos dependency to test against a CentOS compose.
It also removes two deprecated dependency pipelines.

Change-Id: I39091a656bef91978a2fd08e4a4a6cfcf32a4c9f
Signed-off-by: Douglas Viroel <dviroel@redhat.com>
This commit is contained in:
Douglas Viroel 2022-03-23 15:05:26 -03:00
parent 39ddc0ce71
commit 2bc0dbe511
1 changed files with 78 additions and 52 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

130
doc/source/ci/tripleo_dependencies_pipelines.rst
View File

@ -14,10 +14,9 @@ Currently we have `promotion and component pipeline`_ set up to detect
OpenStack projects related issues early.
In order to detect the breakages from non-openstack projects, TripleO
dependency pipeline comes into existance. we have added two types of
pipeline:
dependency pipeline has come into existence. Currently we have a single
type of pipeline enabled:
* packages coming from specific module
* packages coming from specific repo
The configurations for each pipeline can be found under
@ -25,56 +24,15 @@ tripleo-quickstart/src/branch/master/config/release/dependency_ci/<module/repo n
Current OpenStack Dependency Pipeline jobs
------------------------------------------
* openstack-dependencies-containertools - for testing container tools dependencies
* openstack-dependencies-centos8stream - for testing base operating system dependencies coming from CentOS-8 stream repo.
* openstack-dependencies-openvswitch - for testing OVS and OVN dependencies coming from NFV sig repo.
* openstack-dependencies-centos-compose - for testing jobs pinned to a specific CentOS compose build.
UnderStanding Module dependency pipeline
----------------------------------------
.. note::
The following pipelines were deprecated in favor of CentOS Stream 9 adoption:
- openstack-dependencies-containertools - for testing container tools dependencies
- openstack-dependencies-centos8stream - for testing base operating system dependencies coming from CentOS-8 stream repo.
Under openstack-dependencies-containertools pipeline,
We test the latest podman/buildah version comes from container tools RHEL-8 module.
The release specific configuration for `containertools`_ pipeline can be found here:
.. code-block:: yaml
dependency_modules:
- module_name: container-tools
control_version: 2.0
test_version: rhel8
dep_repo_cmd_after: |
sudo dnf repolist;
sudo dnf module list;
{% for item in dependency_modules %}
sudo dnf module disable {{ item.module_name }}:{{ item.control_version }} -y;
sudo dnf module enable {{ item.module_name }}:{{ item.test_version }} -y;
{% endfor %}
sudo dnf clean metadata
sudo dnf clean all;
sudo dnf update -y;
What do the above terms mean:
* `module_name`: Name of the DNF module to pull packages from
* `control_version`: Name of the version to disable for the particular module
* `test_version`: Name of the version to enable for the particular module
In the above cases, it is going to test container-tools module in which
it will disable container-tools:2.0 and enable container-tools:rhel8 module.
All the container-tools packages will be coming here.
The above config is used by `get-dependency-module-content.yaml`_ to enable or
disable the above module during the deployment.
- jobs running in `openstack-dependencies-containertools`_ pipeline on review.rdoproject.org.
* periodic-tripleo-ci-centos-8-standalone-container-tools-master
* periodic-tripleo-ci-centos-8-standalone-container-tools-container-build-master
The job definitions are defined under `rdo-jobs/zuul.d/dependencies-jobs.yaml`_ and
their triggers are defined here: `rdo-jobs/zuul.d/project-templates-dependencies.yaml`_.
UnderStanding package dependency pipeline
Understanding Package Dependency Pipeline
-----------------------------------------
openstack-dependencies-openvswitch is a package dependency pipeline where we
@ -135,8 +93,70 @@ Below are the jobs running in `openstack-dependencies-openvswitch`_ pipeline on
vars:
force_periodic: false
Understanding CentOS Compose Pinning Dependency
-----------------------------------------------
The dependency `compose-repos`_ works in the same line as package dependency jobs, with the difference that instead of
setting up a single repository at a time, it consumes metadata from the provided compose URL and generates a set of
repos as specified in the configuration snippet below:
Ensuring correct module or repo is used
.. code-block:: yaml
...
add_repos:
- type: compose_repos
compose_url: "https://odcs.stream.centos.org/production/latest-CentOS-Stream/compose/"
release: centos-stream-9
disable_conflicting: true
variants:
- AppStream
- BaseOS
- HighAvailability
- CRB
disable_repos:
- quickstart-centos-appstreams.repo
- quickstart-centos-base.repo
- quickstart-centos-highavailability.repo
- quickstart-centos-crb.repo
The `compose_repos` repo's type was created to generate a set of repos based on a compose repos URL and information about
variants and conflicting repos. The `variants` will define which repos should be created from compose metadata, while
`disable_conflicting` and `disable_repos` should guarantee that conflicting repos will be disabled in favor of the new
ones.
For more details on how repos are generated, please check `yum-config-compose`_ in setup-role and `yum-config`_ tool.
.. note::
The process of setting up compose-repos starts earlier in the job, before any call to `repo-setup`, in one of the pre
playbooks defined in base jobs. You shall see the `centos-compose-repos.yml`_ playbook running in jobs that have
`dependency` set to *centos-compose*, which sets up those repos using the same tools mentioned above. The purpose of
the dependency config here is to keep those repos enabled when any other playbook or role calls `repo-setup`.
Testing Jobs Using Compose Pinning Dependency
---------------------------------------------
In order to test any job against a CentOS compose build, which can be a compose newer or older than the available on
CentOS mirrors, you will need to setup a new job definition and provide the following job variables:
.. code-block:: yaml
- job:
name: tripleo-ci-centos-9-standalone-compose-pinning
parent: tripleo-ci-centos-9-standalone
vars:
dependency: centos-compose
centos_compose_url: https://odcs.stream.centos.org/production/latest-CentOS-Stream/compose/
build_container_images: true
containers_base_image: quay.io/centos/centos:stream9
* `dependency`: need to be set to *centos-compose*.
* `centos_compose_url`: CentOS compose URL to be tested. Note that the full URL ends with `compose`, because it is where
compose `metadata` lives, required by `yum-config` tool to generate the repos. The default value is set to latest
compose, which might be ahead of mirror's compose.
.. note::
In the example above, there is an enabled flag for `build_container_images`. It means that process of building
containers will also use compose repositories.
Ensuring Correct Module or Repo is Used
---------------------------------------
Once a jobs runs and finishes in the dependency pipeline, we need to navigate
@ -149,9 +169,11 @@ two log files:
All the above operation is done `rdo-jobs/playbooks/dependency/diff-control-test.yaml`_ playbook which uses
`compare_rpms`_ project from ci-config/ci-scripts/infra-setup/roles/rrcockpit/files.
.. note::
The dependency `compose-repos` doesn't support rpm diff control test yet.
.. _`promotion and component pipeline`: https://docs.openstack.org/tripleo-docs/latest/ci/stages-overview.html
.. _`openvswitch dependency pipeline`: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/dependency_ci/openvswitch/repo_config.yaml
.. _`containertools`: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/dependency_ci/container-tools/repo_config.yaml
.. _`openstack-dependencies-containertools`: https://review.rdoproject.org/zuul/builds?pipeline=openstack-dependencies-containertools
.. _`openstack-dependencies-openvswitch`: https://review.rdoproject.org/zuul/builds?pipeline=openstack-dependencies-openvswitch
.. _`rdo-jobs/zuul.d/dependencies-jobs.yaml`: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/dependencies-jobs.yaml
@ -160,3 +182,7 @@ All the above operation is done `rdo-jobs/playbooks/dependency/diff-control-test
.. _`get-dependency-module-content.yaml`: https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/dependency/get-dependency-module-content.yaml
.. _`rdo-jobs/dependency/get-dependency-repo-content.yaml`: https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/dependency/get-dependency-repo-content.yaml
.. _`compare_rpms`: https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/rrcockpit/files/compare_rpms
.. _`compose-repos`: https://github.com/openstack/tripleo-quickstart/blob/3f3f93da95c8531a4542c9a9aaa6424f2f6364c6/config/release/dependency_ci/centos-compose/repo_config.yaml
.. _`yum-config-compose`: https://github.com/openstack/tripleo-quickstart/blob/3f3f93da95c8531a4542c9a9aaa6424f2f6364c6/roles/repo-setup/tasks/yum-config-compose.yml
.. _`yum-config`: https://github.com/openstack/tripleo-repos/blob/cbbdde6cb6c73692b3ce9d0f6931f1b6e6fe6c91/plugins/modules/yum_config.py
.. _`centos-compose-repos.yml`: https://github.com/openstack/tripleo-ci/blob/5246cc282819f0248d997db79bc16c0f00a9e2f8/playbooks/tripleo-ci/centos-compose-repos.yml