In a lot of placeses we are using word "benchmark" which can mean workload, subtask, or test case which is very confusing. This patch partially address wrong usage of "benchamrk" word Change-Id: Id3b2b7ae841a5243684c12cc51c96f005dbe7544
5.4 KiB
Rally OpenStack Gates
Gate jobs
The OpenStack CI system uses the so-called "Gate jobs" to control merges of patches submitted for review on Gerrit. These Gate jobs usually just launch a set of tests -- unit, functional, integration, style -- that check that the proposed patch does not break the software and can be merged into the target branch, thus providing additional guarantees for the stability of the software.
Create a custom Rally Gate job
You can create a Rally Gate job for your project to run Rally tasks against the patchsets proposed to be merged into your project.
To create a rally-gate job, you should create a rally-jobs/ directory at the root of your project.
As a rule, this directory contains only {projectname}.yaml, but more scenarios and jobs can be added as well. This yaml file is in fact an input Rally task file specifying scenarios that should be run in your gate job.
To make {projectname}.yaml run in gates, you need to add "rally-jobs" to the "jobs" section of projects.yaml in openstack-infra/project-config.
Example: Rally Gate job for Glance
Let's take a look at an example for the Glance project:
Edit jenkins/jobs/projects.yaml:
- project:
-
name: glance node: 'bare-precise || bare-trusty' tarball-site: tarballs.openstack.org doc-publisher-site: docs.openstack.org
- jobs:
-
- python-jobs
- python-icehouse-bitrot-jobs
- python-juno-bitrot-jobs
- openstack-publish-jobs
- translation-jobs - rally-jobs
Also add gate-rally-dsvm-{projectname} to zuul/layout.yaml:
- name: openstack/glance template: - name: merge-check - name: python26-jobs - name: python-jobs - name: openstack-server-publish-jobs - name: openstack-server-release-jobs - name: periodic-icehouse - name: periodic-juno - name: check-requirements - name: integrated-gate - name: translation-jobs - name: large-ops - name: experimental-tripleo-jobs check: - check-devstack-dsvm-cells - gate-rally-dsvm-glance gate: - gate-devstack-dsvm-cells experimental: - gate-grenade-dsvm-forward
To add one more scenario and job, you need to add {scenarioname}.yaml file here, and gate-rally-dsvm-{scenarioname} to projects.yaml.
For example, you can add myscenario.yaml to rally-jobs directory in your project and then edit jenkins/jobs/projects.yaml in this way:
- project:
-
name: glance github-org: openstack node: bare-precise tarball-site: tarballs.openstack.org doc-publisher-site: docs.openstack.org
- jobs:
-
- python-jobs
- python-havana-bitrot-jobs
- openstack-publish-jobs
- translation-jobs
- rally-jobs - 'gate-rally-dsvm-{name}': name: myscenario
Finally, add gate-rally-dsvm-myscenario to zuul/layout.yaml:
- name: openstack/glance template: - name: python-jobs - name: openstack-server-publish-jobs - name: periodic-havana - name: check-requirements - name: integrated-gate check: - check-devstack-dsvm-cells - check-tempest-dsvm-postgres-full - gate-tempest-dsvm-large-ops - gate-tempest-dsvm-neutron-large-ops - gate-rally-dsvm-myscenario
It is also possible to arrange your input task files as templates
based on Jinja2
. Say, you want to set the image names used
throughout the myscenario.yaml task file as a variable
parameter. Then, replace concrete image names in this file with a
variable:
...
NovaServers.boot_and_delete_server:
-
args:
image:
name: {{image_name}}
...
NovaServers.boot_and_list_server:
-
args:
image:
name: {{image_name}}
...
and create a file named myscenario_args.yaml that will define the parameter values:
---
image_name: "^cirros.*-disk$"
this file will be automatically used by Rally to substitute the variables in myscenario.yaml.
Plugins & Extras in Rally Gate jobs
Along with scenario configs in yaml, the rally-jobs directory can also contain two subdirectories:
- plugins:
Plugins <plugins>
needed for your gate job; - extra: auxiliary files like bash scripts or images.
Both subdirectories will be copied to ~/.rally/ before the job gets started.