openstack/tripleo-docs
Code Issues Proposed changes

Add documentation for TripleO/RDO/OSP Pipelines

Browse Source 
This documentation covers the promotion pipelines
and the use of the DLRN API Promoter script to
promote and update images, repos and symlinks.

Change-Id: Idcd8f4f4cd9bb8468ede95f4e879f0e2f710c9ba
This commit is contained in:
Ronelle Landy
2017-10-09 14:54:02 -04:00
parent 9e6170f498
commit 2f9d4b96c2
3 changed files with 242 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

141
doc/source/contributor/dlrn-promoter-overview.rst Normal file
View File

@@ -0,0 +1,141 @@
How the TripleO-RDO Pipelines' Promotions Work
==============================================
Building consumable RDO repos and images involves various stages.
Each stage takes inputs and outputs artifacts. This document explains the
stages comprising the promotion pipelines, and the tools used to create
and manage the resulting artifacts.
What is DLRN?
-------------
DLRN is a tool to build RPM packages from each commit to a set of
OpenStack-related git repositories that are included in RDO.
DLRN builds are run through CI and to detect packaging issues with the
upstream branches of these Openstack projects.
DLRN Artifacts - Hashes and Repos
---------------------------------
When a DLRN build completes, it produces a new hash and related repo version.
For example, the Pike builds on CentOS are available at:
https://trunk.rdoproject.org/centos7-pike/.
The builds are placed in directories by DLRN hash. Each directory contains
the RPMs as well as a repo file
https://trunk.rdoproject.org/centos7-pike/current-tripleo/delorean.repo
and a ``commit.yaml`` file
https://trunk.rdoproject.org/centos7-pike/current-tripleo/commit.yaml.
There are some standard links that are updated as the builds complete and pass
stages of CI. Examples are these links are:
- https://trunk.rdoproject.org/centos7-pike/current/
- https://trunk.rdoproject.org/centos7-pike/consistent/
- https://trunk.rdoproject.org/centos7-pike/current-tripleo/
- https://trunk.rdoproject.org/centos7-pike/current-tripleo-rdo/
- https://trunk.rdoproject.org/centos7-pike/current-tripleo-rdo-internal/
- https://trunk.rdoproject.org/centos7-pike/tripleo-ci-testing/
The above links will be referenced in the sections below.
Promoting through the Stages - DLRN API
---------------------------------------
DLRN API Client
```````````````
`The DLRN API
<https://github.com/softwarefactory-project/DLRN/blob/master/doc/source/api.rst>`_
`client <https://github.com/softwarefactory-project/dlrnapi_client/>`_
enables users to query repo status, upload new hashes and create promotions.
Calls to the dlrnapi_client retrieve the inputs to stages and upload artifacts
after stages.
For example:
::
$ dlrnapi --url https://trunk.rdoproject.org/api-centos-master-uc \
promotion-get --promote-name tripleo-ci-testing
[{'commit_hash': 'ec650aa2c8ce952e4a33651190301494178ac562',
'distro_hash': '9a7acc684265872ff288a11610614c3b5739939b',
'promote_name': 'tripleo-ci-testing',
'timestamp': 1506427440},
{'commit_hash': 'ec650aa2c8ce952e4a33651190301494178ac562',
[..]
$ dlrnapi --url https://trunk.rdoproject.org/api-centos-master-uc \
repo-status --commit-hash ec650aa2c8ce952e4a33651190301494178ac562 \
--distro-hash 9a7acc684265872ff288a11610614c3b5739939b
[{'commit_hash': 'ec650aa2c8ce952e4a33651190301494178ac562',
'distro_hash': '9a7acc684265872ff288a11610614c3b5739939b',
'in_progress': False,
'job_id': 'consistent',
'notes': '',
'success': True,
'timestamp': 1506409403,
'url': ''},
{'commit_hash': 'ec650aa2c8ce952e4a33651190301494178ac562',
'distro_hash': '9a7acc684265872ff288a11610614c3b5739939b',
'in_progress': False,
'job_id': 'periodic-singlenode-featureset023',
'notes': '',
'success': True,
'timestamp': 1506414726,
'url': 'https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-centos-7-master-containers-build/8a76883'},
{'commit_hash': 'ec650aa2c8ce952e4a33651190301494178ac562',
[..]
DLRN API Promoter
`````````````````
`The DLRN API Promoter script
<https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/dlrnapi_promoter>`_
is a Python script that, based on the information in an input config file,
will promote an existing DLRN link to another link, provided the required tests
return successful results.
For example,
`the master ini config file
<https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/master.ini>`_
is passed to the `promoter script
<https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/dlrnapi_promoter.py>`_
to promote the ``current-tripleo`` link to ``current-tripleo-rdo``. See the
sections above where both these links (for Pike) were shown.
In the RDO Phase 1 pipeline, the tests listed under the ``[current-tripleo-rdo]``
are run with the ``current-tripleo`` hash. Each test reports its ``success`` status to the
DLRN API endpoint for the Master release, ``api-centos-master-uc``.
If each test reports ``SUCCESS: true``, the content of the ``current-tripleo``
will become the new content of the ``current-tripleo-rdo`` hash.
For complete documentation on how to run the Promoter script see:
https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/README.md
Pushing RDO containers to ``docker.io``
```````````````````````````````````````
The DLRN Promotor script calls the `container push playbook
<https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/container-push/container-push.yml>`_
to push the RDO containers at each stage to `docker.io
<https://hub.docker.com/r/tripleopike/centos-binary-heat-api/tags/>`_.
Note that the the above ``docker.io`` link shows containers tagged with ``tripleo-ci-testing``,
``current-tripleo`` and ``current-tripleo-rdo``.
DLRN API Promoter Server
````````````````````````
It is recommended that the Promoter script is run from a dedicated server.
`The promoter-setup repo
<https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/promoter-setup>`_
contains the Ansible playbook used to setup the promoter-server in the RDO
Cloud environment. This playbook allows the promoter script server to be
rebuilt as required.

2
doc/source/contributor/index.rst
View File

@@ -8,3 +8,5 @@ TripleO Contributor Guide
new_developers
contributions
check_gates
dlrn-promoter-overview
stages-overview

99
doc/source/contributor/stages-overview.rst Normal file
View File

@@ -0,0 +1,99 @@
Promotion Stages
================
The list below shows each stage within the RDO promotion workflow.
Each stage shows the inputs taken and the artifacts produced.
.. note:: All the links shown in the diagram refer to the Master release.
Links for stable branches would include the stable release name,
for example, `Pike stable release <https://trunk.rdoproject.org/centos7-pike/tripleo-ci-testing/>`_.
1. **Upstream TripleO**
`CI DLRN Master consistent
<https://trunk.rdoproject.org/centos7-master/consistent/>`_
is generated every 30 mins from upstream commits in case of no packaging errors.
The ``current`` build contains the latest packages that can be built,
even if some other packages are failing to build. The ``current`` and
``consistent`` builds will be equivalent where there are no problems with
the build. See https://trunk.rdoproject.org/centos7-master/report.html
(for Master) for the result of builds.
2. **Upstream Promotion Pipeline**
`rdoproject.org zuul <https://review.rdoproject.org/zuul/>`_
*Update* from consistent -> tripleo-ci-testing
https://trunk.rdoproject.org/centos7-master/tripleo-ci-testing/
*Push containers* to ``docker.io`` tagged with ``tripleo-ci-testing``
https://hub.docker.com/r/tripleomaster/centos-binary-heat-api/tags/
Run tests - if tests report success ->
*Promote* from tripleo-ci-testing -> current-tripleo using DLRN Promoter
https://trunk.rdoproject.org/centos7-master/current-tripleo/
*Push containers* to trunk.registry.rdoproject.org/master/ (tripleo-ci-testing)
and upon promotion, also tag them with ``current-tripleo``
https://review.rdoproject.org/jenkins/job/periodic-tripleo-centos-7-master-containers-build
*Update images* using DLRN Promoter
https://images.rdoproject.org/master/rdo_trunk/current-tripleo/stable/
periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-master-upload
See artifacts referenced:
https://github.com/openstack/tripleo-quickstart/blob/master/config/release/master.yml
https://images.rdoproject.org/master/rdo_trunk/current-tripleo
RH1 mirror server for images http://66.187.229.139/
*Push containers* to ``docker.io`` tagged with ``current-tripleo``
https://hub.docker.com/r/tripleomaster/centos-binary-heat-api/tags/
Logs from the DLRN Promoter can be accessed on http://38.145.33.13/
`A Sova instance <http://38.145.34.234/>`_ is used to monitor these jobs
3. **RDO Phase 1**
https://ci.centos.org/job/rdo_trunk-promote-master-current-tripleo/
Get https://trunk-primary.rdoproject.org/centos7-pike/current-tripleo/delorean.repo
Run tests - if tests report success ->
*Promote* from current-tripleo -> current-tripleo-rdo using DLRN Promoter
https://trunk.rdoproject.org/centos7-master/current-tripleo-rdo/
*Promote the images* to
https://images.rdoproject.org/master/rdo_trunk/current-tripleo-rdo/
*Tag the containers* as ``current-tripleo-rdo``
*Push containers* to ``docker.io`` tagged with ``current-tripleo-rdo``
https://hub.docker.com/r/tripleomaster/centos-binary-heat-api/tags/
4. **RDO Phase 2**
Get RDO Phase 1 generated image (RDO on CentOS) from
https://images.rdoproject.org/master/$BUILD_SYS/$PIN/$hash_id
and cache internally
Build http://<internal>/ci-images/master (RDO on RHEL)
Run tests (RDO on CentOS for baremetal and RDO on RHEL)
If tests report success ->
*Update symlink* using DLRN Promoter
https://trunk.rdoproject.org/centos7-master/current-tripleo-rdo-internal/
*Update the images link* as
https://images.rdoproject.org/master/rdo_trunk/current-tripleo-rdo-internal
*Tag the containers* as ``current-tripleo-rdo-internal``
Run futher downstream jobs (scale etc.)
rdo-promote-master-rdo_trunk-nonvoting
5. **OSP Phase 0**
*Hand off* to RHOSP builds
https://access.redhat.com/documentation/en/red-hat-openstack-platform/