Browse Source

Merge "Add spec for single sourcing repo setup"

Zuul 2 months ago
committed by Gerrit Code Review
  1. 339


@ -0,0 +1,339 @@
This work is licensed under a Creative Commons Attribution 3.0 Unported
TripleO Repos Single Source
This proposal lays out the plan to use tripleo-repos as a single source
to install and configure non-base OS repos for TripleO - including
setting the required DLRN hashes.
Problem Description
Reviewing the code base, there are multiple places where repos are
specified. For example,in the release files we are setting the
configuration that is applied by `repo setup role`_.
Some of the other repo/version configurations are included in:
* `tripleo repos`_
* `repo setup role`_
* `release config files`_
* `container tooling (base tcib file)`_
* `tripleo-ansible`_
* `rdo config`_ (example)
* `tripleo-heat-templates`_
* `tripleo-operator-ansible`_
.. _`tripleo repos`:
.. _`repo setup role`:
.. _`release config files`:
.. _`container tooling (base tcib file)`:
.. _`tripleo-ansible`:
.. _`rdo config`:
.. _`tripleo-heat-templates`:
.. _`tripleo-operator-ansible`:
The process of setting repo versions requires getting and
transforming DLRN hashes, for example resolving 'current-tripleo'
to a particular DLRN build ID and specifying the correct proxies.
Currently a large portion of this work is done in the release files
resulting in sections of complicated and fragile Bash scripts -
duplicated across numerous release files.
This duplication, coupled with the various locations in use
for setting repo configurations, modules and supported versions
is confusing and error prone.
There should be one source of truth for which repos are installed
within a tripleo deployment and how they are installed.
Single-sourcing all these functions will avoid the current
problems of duplication, over-writing settings and version confusion.
Proposed Change
This proposal puts forward using tripleo-repos as the 'source of truth'
for setting repo configurations, modules and supported versions -
including setting the DLRN hashes required to specify exact repo
versions to install for upstream development/CI workflows.
Having a single source of truth for repo config, modules, etc. will make
development and testing more consistent, reliable and easier to debug.
The intent is to use the existing tripleo-repos repo for this work and
not to create a new repo. It is as yet to be determined if we will add
a v2/versioned api or how we will handle the integration with the
existing functionality there.
We aim to modularize the design and implementation of the proposed tripleo-repos
work. Two sub systems in particular have been identified that can be
implemented independently of, and ultimately to be consumed by, tripleo-repos;
the resolution of delorean build hashes from known tags (i.e. resolving
'current-tripleo' to a particular DLRN build ID) and the configuration of dnf
repos and modules will be implemented as independent python modules, with
their own unit tests, clis, ansible modules etc.
Integration Points
The new work in tripleo-repos will have to support with all
the cases currently in use and will have to integrate with:
* DLRN Repos
* release files
* container and overcloud image builds
* rdo config
* yum/dnf repos and modules
* Ansible (Ansible module)
* promotion pipeline - ensuring the correct DLRN hashes
Incorporating the DLRN hash functionality makes the tool
more complex. Unit tests will be required to guard
against frequent breakages. This is one of the reasons that we decided to split
this DLRN hash resolution into its own dedicated python module
'tripleo-get-hash' for which we can have independent unit tests.
The scope of the new tripleo-repos tool will be limited to upstream
development/CI workflows.
Functionality to set repos, modules and versions is already available today.
It would be possible to leave the status quo or:
* Use rdo config to set one version per release - however, this would not
address the issue of changing DLRN hashes
* Create an rpm that lays down /etc/tripleo-release where container-tools could
be meta data in with that, similar to /etc/os-release
Security Impact
No security impact is anticipated. The work is currently in the tripleo
open-source repos and will remain there - just in a consolidated
place and format.
Upgrade Impact
Currently there will be no upgrade impact. The new CLI will support
all release versions under support and in use. At a later date,
when the old CLI is deprecated there may be some update
However,there may be work to make the emit_releases_file
functionality compatible with the new CLI.
Other End User Impact
Work done on the new project branch will offer a different version of CLI, v2.
End users would be able to select which version of the CLI to use - until
the old CLI is deprecated.
Performance Impact
No performance impact is expected. Possible performance improvements could
result from ensuring that proxy handling (release file, mirrors, rdoproject)
is done correctly and consistently.
Other Deployer Impact
Developer Impact
See ```Other End User Impact``` section.
The functionality added to tripleo-repos will be writen as a Python module
with a CLI and will be able to perform the following primary functions:
* Single source the installation of all TripleO related repos
* Include the functionality current available in the repo-setup role
including creating repos from templates and files
* Perform proxy handling such as is done in the release files
(mirrors, using rdoproject for DLRN repos)
* Get and transform human-readable DLRN hashes - to be implemented as an
independent module.
* Support setting yum modules such as container-tools - to be implemented
as an independent module.
* Support enabling and disabling repos and setting their priorities
The repo-setup role shall remain but it will invoke tripleo-repos.
All options required to be passed to tripleo-repos should be in the
release file.
Work done on the new project branch will offer a different version of CLI, v2.
Unit tests will be added on this branch to test the new CLI directly.
CI would be flipped to run in the new branch when approved by TripleO teams.
All current unit tests should pass with the new code.
An Ansible module will be added to call the tripleo-repos
options from Ansible directly without requiring the end
user to invoke the Python CLI from within Ansible.
The aim is for tripleo-repos to be the single source for all repo related
configuration. In particular the goal is to serve the following 3 personas:
* Upstream/OpenStack CI jobs
* Downstream/OSP/RHEL jobs
* Customer installations
The configuration required to serve each of these use cases is slightly
different. In upstream CI jobs we need to configure the latest current-tripleo
promoted content repos. In downstream/OSP jobs we need to use rhos-release
and in customer installations we need to use subscription manager.
Because of these differing requirements we are leaning towards storing the
configuration for each in their intended locations, with the upstream config
being the 'base' and the downstream config building ontop of that (the
implication is that some form of inheritance will be used to avoid duplication).
This was discussed during the `Xena PTG session`_
.. _`Xena PTG session`:
* sshnaidm (DF and CI)
* marios (CI and W-release PTL)
* weshay
* chandankumar
* ysandeep
* arxcruz
* rlandy
* other DF members (cloudnull)
Work Items
Proposed Schedule
Investigative work will be begin in the W-release cycle on a project branch
in tripleo-repos. The spec will be put forward for approval in the X-release
cycle and impactful and integration work will be visible once the spec
is approved.
This work has a dependency on the `DLRN API`_ and on yum/dnf.
.. _`DLRN API`:
Specific unit tests will be added with the python-based code built.
All current CI tests will run through this work and will
test it on all releases and in various aspects such as:
* container build
* overcloud image build
* TripleO deployments (standalone, multinode, scenarios, OVB)
* updates and upgrades
CLI Design
Here is an abstract sketch of the intended cli design for the
new tripleo-repos.
It covers most of the needs discussed at multiple places.
Scenario 1
The goal is to construct a repo with the correct hash for an integration
or a component pipeline.
For this scenario:
* Any combination of `hash, distro, commit, release, promotion, url` parameters can passed
* Use the `tripleo-get-hash`_ module to determine the DLRN build ID
* Use the calculated DLRN build ID to create and add a repo
.. _`tripleo-get-hash`:
Scenario 2
The goal is to construct any type of yum/dnf repo.
For this scenario:
* Construct and add a yum/dnf repo using a combination of the following parameters
* filename - filename for saving the resulting repo (mandatory)
* reponame - name of repository (mandatory)
* baseurl - base URL of the repository (mandatory)
* down_url - URL to download repo file from (mandatory/multually exclusive to baseurl)
* priority - priority of resulting repo (optional)
* enabled - 0/1 whether the repo is enabled or not (default: 1 - enabled)
* gpgcheck - whether to check GPG keys for repo (default: 0 - don't check)
* module_hotfixes - whether to make all RPMs from the repository available (default: 0)
* sslverify - whether to use a cert to use repo metadata (default: 1)
* type - type of the repo(default: generic, others: custom and file)
Scenario 3
The goal is to enable or disable specific dnf module and also install or
remove a specific package.
For this scenario:
* Specify
* module name
* which version to disable
* which version to enable
* which specific package from the module to install (optional)
Scenario 4
The goal is to enable or disable some repos,
remove any associated repo files no longer needed,
and then perform a system update.
For this scenario:
* Specify
* repo names to be disabled
* repo names to be enabled
* the files to be removed
* whether to perform the system update
Documentation Impact
tripleo-docs will be updated to point to the new supported
repo/modules/versions setting workflow in tripleo-repos.
References to old sources of settings such as tripleo-ansible,
release files in tripleo-quickstart and the repo-setup role
will have to be removed and replaced to point to the new