diff --git a/doc/source/ci/component_integration_pipelines.mmd b/doc/source/ci/component_integration_pipelines.mmd new file mode 100644 index 00000000..da0605da --- /dev/null +++ b/doc/source/ci/component_integration_pipelines.mmd @@ -0,0 +1,5 @@ +graph TD + compute/consistent-->compute/component-ci-testing-->compute/promoted-components-->promoted-components + cinder/consistent-->cinder/component-ci-testing-->cinder/promoted-components-->promoted-components + security/consistent-->security/component-ci-testing-->security/promoted-components-->promoted-components + promoted-components-->tripleo-ci-testing-->current-tripleo diff --git a/doc/source/ci/component_pipeline_tags_flow.mmd b/doc/source/ci/component_pipeline_tags_flow.mmd new file mode 100644 index 00000000..01345573 --- /dev/null +++ b/doc/source/ci/component_pipeline_tags_flow.mmd @@ -0,0 +1,6 @@ +graph TD + A[consistent] -->|periodically promote to| B[component-ci-testing] + B -->|Run periodic promotion jobs| C{criteria jobs passing?} + C -->|no| E[reject content] + C -->|yes| D[promoted-components] + D --> F[start of integration pipeline] diff --git a/doc/source/ci/promotions.mmd b/doc/source/ci/promotions.mmd index a9a83abe..e6441776 100644 --- a/doc/source/ci/promotions.mmd +++ b/doc/source/ci/promotions.mmd @@ -1,5 +1,6 @@ graph TD - master --> |build-packages|current-consistent - current-consistent --> |tripleo-ci promotion|current-tripleo - current-tripleo --> |rdo phase1 promotion|current-tripleo-rdo - current-tripleo-rdo --> |rdo phase2 promotion|current-tripleo-rdo-internal + A[promoted-components] --> |periodically promote to| B[tripleo-ci-testing] + B --> |Build images & containers| C[images in RDO cloud] + C --> |Run periodic promotion jobs| D{criteria jobs passing?} + D-->|no| F[reject content] + D-->|yes| E[current-tripleo] diff --git a/doc/source/ci/stages-overview.rst b/doc/source/ci/stages-overview.rst index d3386574..06ac4bbc 100644 --- a/doc/source/ci/stages-overview.rst +++ b/doc/source/ci/stages-overview.rst @@ -1,117 +1,266 @@ -Promotion Stages -================ +TripleO CI Promotions +===================== -The list below shows each stage within the RDO promotion workflow. -Each stage shows the inputs taken and the artifacts produced. +This section introduces the concept of promotions in TripleO. +In short, a promotion happens when we can certify the latest version of all +packages required for a TripleO deployment of OpenStack as being in a good +state and without regressions. -.. note:: All of the shown links refer to the Master release. +The certification consists of running Zuul CI jobs with the latest packages +built from source for TripleO code (list of TripleO repos at [1]_) and +the latest packages built from source for non-tripleo code. If the tests are +successful, then the result is certified as **current-tripleo**, ready to be +consumed by the TripleO CI check and gate jobs (see [2]_ for more information +about check and gate). - Links for stable branches would include the stable release name, - for example, `Pike stable release `_. +This process is continuous as new code is merged into the various repos. Every +time we get a successful completion of the promotion CI jobs, the tested content +is 'promoted' to be the new **current-tripleo**, hence the name this workflow +is known by. At a given time, the latest **current-tripleo** is the baseline by +which we test all new code submissions to the TripleO project. + +TripleO vs non-tripleo repos +---------------------------- + +All proposed code submissions across the various tripleo repos are gated by the +TripleO community which owns and manages the zuul check and gate jobs for those +repos. + +However, we cannot gate changes to anything outside TripleO, including all +the OpenStack projects used by TripleO as well as any dependencies such +as Open vSwitch or Pacemaker. + +Even though we cannot gate on those external repos, the promotion process +allows us to test our TripleO code with their latest versions. If there are +regressions or any other bugs (and assuming ideal test coverage) the promotion +jobs will fail accordingly allowing the TripleO CI team to investigate and file +launchpad bugs so the issue(s) can be addressed. + +RDO DLRN & Promotion Criteria +----------------------------- + +TripleO CI jobs consume packages built by the RDO DLRN service ('delorean') so +we first introduce it here. An overview is given on the RDO project site at +[3]_. + +In short, RDO DLRN builds RPMs from source and publishes the resulting packages +and repos. Each build or repo is identifiable using a unique build ID. + +RDO DLRN assigns named tags to particular build IDs. You can see all of these +named tags by browsing at the RDO DLRN package root, for example for Centos8 +master branch at [4]_. Of particular importance to the TripleO promotion +workflow are:: + +* current +* consistent +* component-ci-testing +* promoted-components +* tripleo-ci-testing +* current-tripleo + +The list of tags in the order given above gives the logical progression +through the TripleO promotion workflow. + +The build ID referenced by each of those named tags is constantly updated as +new content is 'promoted' to become the new named tag. + +A general pattern in DLRN is that **current** is applied to the very latest +build, that is, the latest commits to a particular repo. A new **current** +build is generated periodically (e.g. every half hour). The **consistent** tag +represents the latest version of packages where there were no errors +encountered during the build for any of those (i.e. all packages were built +successfully). The **consistent** build is what TripleO consumes as the entry +point to the TripleO promotion workflow. + +One last point to be made about RDO DLRN is that after the TripleO promotion +CI jobs are executed against a particular DLRN build ID, the results are +reported back to DLRN. For example, you can query using the build ID at [5]_ +to get the list of jobs that were executed +against that specific content, together with the results for each. + +The list of jobs that are required to pass before we can promote a particular +build is known as the 'promotion criteria'. In order to promote, TripleO +queries the DLRN API to get the results for a particular build and compares the +passing jobs to the promotion criteria, before promoting or rejecting that +content accordingly. You can find the master centos8 promotion criteria at [6]_ +for example. + +The TripleO Promotion Pipelines +------------------------------- + +A pipeline refers to a series of Zuul CI jobs and what we refer to as the +TripleO promotion workflow is actually a number of interconnected pipelines. +At the highest level conceptually these are grouped into either *Component* +or *Integration* pipelines. The output of the Component pipeline serves as +input to the Integration pipeline. + +A Component is a conceptual grouping of packages related by functional area +(with respect to an OpenStack deployment). This grouping is enforced in +practice by the RDO DLRN server and the current list of all components can be +found at [7]_. For example, you can expect to find the 'openstack-nova-' +packages within the Compute component. + +The Component pipeline actually consists of a number of individual +pipelines, one for each of the components. The starting point for each of these +is the latest **consistent** build of the component packages and we will go +into more detail about the flow inside the component pipelines in the following +section. + +A successful run of the jobs for the given component allows us to certify that +content as being the new **promoted-components**, ready to be used as input to +the Integration pipeline. The Integration pipeline qualifies the result of the +components tested together and when that is successful we can promote to a new +current-tripleo. This is shown conceptually for a subset of components here: + +.. only:: html + + .. mermaid:: component_integration_pipelines.mmd + +In the diagram above, you can see the component pipeline at the top with the +compute, cinder and security components. This feeds into the integration +pipeline in the bottom half of the diagram where promoted-components will be +tested together and if successful produce the new **current-tripleo**. + +The Component Promotion Pipeline +-------------------------------- + +As noted above, the "Component pipeline" is actually a series of individual +pipelines, one for each component. While these all operate and promote +in the same way, they do so independently of each other. +So the latest **compute/promoted-components** may be much newer than the latest +**security/promoted-components**, if the latter is failing to promote for +example. The following flowchart shows the progression of the RDO DLRN tags +through a single component pipeline while in practice this flow is repeated in +parallel per component. + +.. only:: html + + .. mermaid:: component_pipeline_tags_flow.mmd -.. only:: latex +As illustrated above, the entry point to the component pipelines +is the latest **consistent** build from RDO DLRN. Once a day a periodic job +tags the latest **consistent** build as **component-ci-testing**. For example +you can see the history for the baremetal component job at [8]_ descriptively +named +**periodic-tripleo-centos-8-master-component-baremetal-promote-consistent-to-component-ci-testing**. - .. parsed-literal:: +After this job has completed the content marked as **component-ci-testing** +becomes the new candidate for promotion to be passed through the component CI +jobs. The **component-ci-testing** repo content is tested with the latest +**current-tripleo** repos of everything else. Remember that at a given time +**current-tripleo** is the known good baseline by which we test all new +content and the same applies to new content tested in the component pipelines. - master --**build-packages**--> - current-consistent -->-- `tripleo-ci promotion `_ --> - current-tripleo -->-- `rdo phase1 promotion `_ --> - current-tripleo-rdo -->-- `rdo phase2 promotion `_ --> - current-tripleo-rdo-internal +As an example of the component CI jobs, you can see the history for the +baremetal component standalone job at [9]_. If you navigate to the +*logs/undercloud/etc/yum.repos.d/* +directory for one of those job runs you will see (at least) the following +repos: + +* delorean.repo - which provides the latest current-tripleo content +* baremetal-component.repo - which provides the 'component-ci-testing' content + that we are trying to promote. + +You may notice that the trick allowing the baremetal-component.repo to have +precedence for the packages it provides is to set the repo priority accordingly +(*1* for the component and *20* for delorean.repo). + +Another periodic job checks the result of the **component-ci-testing** job runs +and if the component promotion criteria is satisfied the candidate content is +promoted and tagged as the new **promoted-components**. You can find the +promotion criteria for Centos8 master components at [10]_. + +As an example the history for the zuul job that handles promotion to +promoted-components for the cinder component can be found at [11]_ + +You can explore the latest content tagged as **promoted-components** for the +compute component at [12]_. All the component **promoted-components** are +aggregated into one repo that can be found at [13]_ and looks +like the following:: + + [delorean-component-baremetal] + name=delorean-openstack-ironic-9999119f737cd39206df3d73e23e5f47933a6f32 + baseurl=https://trunk.rdoproject.org/centos8/component/baremetal/99/99/9999119f737cd39206df3d73e23e5f47933a6f32_1b0aff0d + enabled=1 + gpgcheck=0 + priority=1 + + [delorean-component-cinder] + name=delorean-openstack-cinder-482e6a3cc5cca697b54ee1d853a4eca6e6f3cfc7 + baseurl=https://trunk.rdoproject.org/centos8/component/cinder/48/2e/482e6a3cc5cca697b54ee1d853a4eca6e6f3cfc7_ae00ff8c + enabled=1 + gpgcheck=0 + priority=1 + +Every time a component promotes a new **component/promoted-components** the +aggregated **promoted-components** delorean.repo on the RDO DLRN server is +updated with the new content. + +This **promoted-components** repo is used as the starting point for the TripleO +Integration promotion pipeline. + +The Integration Promotion Pipeline +---------------------------------- + +The Integration pipeline as the name suggests is the integration point where +we test new content from all components together. The consolidated +**promoted-components** delorean.repo produced by the component pipeline +is tested with a series of CI jobs. If the jobs listed in the promotion +criteria pass, we promote that content and tag it as **current-tripleo**. .. only:: html .. mermaid:: promotions.mmd -*Overview of the tags used during the tripleo promotions stages* +As can be seen in the flowchart above, the **promoted-components** content +is periodically promoted (pinned) to **tripleo-ci-testing**, which becomes the +new promotion candidate to be tested. You can find the build history +for the job that promotes to **tripleo-ci-testing** for Centos 8 master, +descriptively named +**periodic-tripleo-centos-8-master-promote-promoted-components-to-tripleo-ci-testing**, +at [14]_. -1. **Upstream TripleO** +First the **tripleo-ci-testing** content is used to build containers and +overcloud deployment images and these are pushed to RDO cloud to be used by +the rest of the jobs in the integration pipeline. - `CI DLRN Master consistent - `_ - is generated every 30 mins from upstream commits in case of no packaging errors. +The periodic promotion jobs are then executed with the results being reported +back to DLRN. If the right jobs pass according to the promotion criteria +then the **tripleo-ci-testing** content is promoted and tagged to become the +new **current-tripleo**. - The ``current`` build contains the latest packages that can be built, - even if some other packages are failing to build. The ``current`` and - ``consistent`` builds will be equivalent where there are no problems with - the build. See https://trunk.rdoproject.org/centos7-master/report.html - (for Master) for the result of builds. +An important distinction in the integration pipeline compared to the promotion +pipeline is in the final promotion of content. In the component pipeline +the **promoted-components** content is tagged by a periodic Zuul job as +described above. For the Integration pipeline however, the promotion to +**current-tripleo** happens with the use of a dedicated service. This service +is known to the tripleo-ci squad by a few names including +'the promotion server', 'the promoter server' and 'the promoter'. -2. **Upstream Promotion Pipeline** +In short the promoter periodically queries delorean for the results of the last +few tripleo-ci-testing runs. It compares the results to the promotion criteria +and if successful it re-tags the container and overcloud deployment images as +**current-tripleo** and pushes back to RDO cloud (as well as to the quay.io and +docker registries). It also talks to the DLRN server and retags the +successful **tripleo-ci-testing** repo as the new **current-tripleo**. +You can read more about the promoter with links to the code at [15]_. - `rdoproject.org zuul `_ - - *Update* from consistent -> tripleo-ci-testing - https://trunk.rdoproject.org/centos7-master/tripleo-ci-testing/ - - *Push containers* to ``docker.io`` tagged with ``tripleo-ci-testing`` - https://hub.docker.com/r/tripleomaster/centos-binary-heat-api/tags/ - - Run tests - if tests report success -> - *Promote* from tripleo-ci-testing -> current-tripleo using DLRN Promoter - https://trunk.rdoproject.org/centos7-master/current-tripleo/ - - *Push containers* to trunk.registry.rdoproject.org/master/ (tripleo-ci-testing) - and upon promotion, also tag them with ``current-tripleo`` - https://review.rdoproject.org/jenkins/job/periodic-tripleo-centos-7-master-containers-build - - *Update images* using DLRN Promoter - https://images.rdoproject.org/master/rdo_trunk/current-tripleo/stable/ - periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-master-upload - See artifacts referenced: - https://github.com/openstack/tripleo-quickstart/blob/master/config/release/master.yml - https://images.rdoproject.org/master/rdo_trunk/current-tripleo - RH1 mirror server for images http://66.187.229.139/ - - *Push containers* to ``docker.io`` tagged with ``current-tripleo`` - https://hub.docker.com/r/tripleomaster/centos-binary-heat-api/tags/ - - Logs from the DLRN Promoter can be accessed on http://38.145.34.55//. - We also use a `Grafana dashboard `_ - and sova for `CI Status `_ to monitor jobs. - -3. **RDO Phase 1** - - https://ci.centos.org/job/rdo_trunk-promote-master-current-tripleo/ - - Get https://trunk.rdoproject.org/centos7-master/current-tripleo/delorean.repo - - Run tests - if tests report success -> - *Promote* from current-tripleo -> current-tripleo-rdo using DLRN Promoter - https://trunk.rdoproject.org/centos7-master/current-tripleo-rdo/ - - *Promote the images* to - https://images.rdoproject.org/master/rdo_trunk/current-tripleo-rdo/ - - *Tag the containers* as ``current-tripleo-rdo`` - - *Push containers* to ``docker.io`` tagged with ``current-tripleo-rdo`` - https://hub.docker.com/r/tripleomaster/centos-binary-heat-api/tags/ - -4. **RDO Phase 2** - - Get RDO Phase 1 generated image (RDO on CentOS) from - https://images.rdoproject.org/master/$BUILD_SYS/$PIN/$hash_id - and cache internally - - Build http:///ci-images/master (RDO on RHEL) - - Run tests (RDO on CentOS for baremetal and RDO on RHEL) - If tests report success -> - *Update symlink* using DLRN Promoter - https://trunk.rdoproject.org/centos7-master/current-tripleo-rdo-internal/ - - *Update the images link* as - https://images.rdoproject.org/master/rdo_trunk/current-tripleo-rdo-internal - - *Tag the containers* as ``current-tripleo-rdo-internal`` - - Run further downstream jobs (scale etc.) - rdo-promote-master-rdo_trunk-nonvoting - -5. **OSP Phase 0** - - *Hand off* to RHOSP builds - https://access.redhat.com/documentation/en/red-hat-openstack-platform/ +References +~~~~~~~~~~ +.. [1] `List of TripleO repos `_ +.. [2] `TripleO Check and Gate jobs `_ +.. [3] `RDO DLRN Overview @ rdoproject.org `_ +.. [4] `Index of RDO DLRN builds for Centos 8 master @ rdoproject.org `_ +.. [5] `Query RDO DLRN by build ID @ rdoproject.org `_ +.. [6] `Centos8 current-tripleo promotion criteria at time of writing `_ +.. [7] `Centos8 RDO DLRN components @ rdoproject.org `_ +.. [8] `Zuul job history "periodic-tripleo-centos-8-master-component-baremetal-promote-consistent-to-component-ci-testing" `_ +.. [9] `Zuul job history "periodic-tripleo-ci-centos-8-standalone-baremetal-master" `_ +.. [10] `Centos8 master promoted-components promotion critiera at time of writing `_ +.. [11] `Zuul job history "periodic-tripleo-centos-8-master-component-cinder-promote-to-promoted-components" `_ +.. [12] `Compute promoted-components @ rdoproject.org `_ +.. [13] `Centos8 master promoted-components delorean.repo @ rdoproject.org `_ +.. [14] `Zuul job history "periodic-tripleo-centos-8-master-promote-promoted-components-to-tripleo-ci-testing" `_ +.. [15] `TripleO CI docs "Promotion Server and Criteria" `_