Removing unclean and wrong document "third_party.rst" from the docs tree
Change-Id: I971bbd5691422e046ce7dd0acf9deb447f1bae90 Signed-off-by: Igor Shishkin <me@teran.ru>
This commit is contained in:
Igor Shishkin
parent
0a3a744bd5
commit
c4e9301be9
|
|
@ -16,6 +16,5 @@ Table of contents
|
|||
infra/overview
|
||||
infra/puppet_master_deployment
|
||||
infra/seed_server_deployment
|
||||
infra/third_party
|
||||
infra/zabbix_server_deployment
|
||||
packaging
|
||||
|
|
|
|||
|
|
@ -1,310 +0,0 @@
|
|||
.. _third-party-testing:
|
||||
|
||||
Third Party Testing
|
||||
===================
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
Gerrit has an event stream which can be subscribed to, using this it
|
||||
is possible to test commits against testing systems beyond those
|
||||
supplied by Fuel's Jenkins setup. It is also possible for these
|
||||
systems to feed information back into Gerrit. What's more, they
|
||||
can leave non-gating votes on Gerrit review requests.
|
||||
|
||||
There are several examples of systems that read the Gerrit event stream
|
||||
and run their own tests on the commits
|
||||
`on this page <https://wiki.openstack.org/wiki/ThirdPartySystems>`_.
|
||||
For each patch set the third party system tests, the system adds a comment
|
||||
in Gerrit with a summary of the test result and links to the test artifacts.
|
||||
|
||||
Requirements
|
||||
------------
|
||||
|
||||
* Until a third party testing system operates in a stable fashion, third
|
||||
party tests can comment on patches but not vote on them.
|
||||
|
||||
* A system can also be set up to only do '+1' reviews and leave all the
|
||||
'-1's to be manually confirmed.
|
||||
|
||||
* A third-party system may only leave one comment per patch set
|
||||
(unless it is retriggered).
|
||||
|
||||
* The maintainers are responsible for re-triggering tests when their third
|
||||
party testing system breaks.
|
||||
|
||||
* Support recheck to request re-running a test.
|
||||
|
||||
* Support the following syntaxes: ``recheck``.
|
||||
* Recheck means recheck everything. A single recheck comment should
|
||||
re-trigger all testing systems.
|
||||
|
||||
* Publish contact information for the maintainers.
|
||||
|
||||
* All accounts must be previously set by posting launchpad bug to add your
|
||||
system.
|
||||
* Maintainers are encouraged to be in IRC regularly to make it
|
||||
faster to contact them.
|
||||
|
||||
* Include a public link to all test artifacts to make debugging failed tests
|
||||
easier (using a dns name over a hardcoded ip is recommended).
|
||||
This should include:
|
||||
|
||||
* Environment details
|
||||
|
||||
* This must include a utc timestamp of the test run
|
||||
* Test configuration
|
||||
|
||||
* Skipped tests
|
||||
* logs should include a trace of the commands used
|
||||
* OpenStack logs
|
||||
* Tempest logs (including ``testr_results.html.gz``)
|
||||
|
||||
* logs must be browsable; logs requiring download, installation or login
|
||||
to access are not acceptable
|
||||
|
||||
.. note:: All test artifacts must be retained for one month.
|
||||
|
||||
Reading the Event Stream
|
||||
------------------------
|
||||
|
||||
It is possible to use ssh to connect to ``review.fuel-infra.org`` on port 29418
|
||||
with your ssh key if you have a normal reviewer account in Gerrit.
|
||||
|
||||
This will give you a real-time JSON stream of events happening inside Gerrit.
|
||||
For example:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ ssh -p 29418 USERNAME@review.fuel-infra.org gerrit stream-events
|
||||
|
||||
Will give a stream with an output like this (line breaks and
|
||||
indentation added in this document for readability, the real JSON will
|
||||
be all one line per event):
|
||||
|
||||
.. code-block:: javascript
|
||||
|
||||
{"type":"comment-added","change":
|
||||
{"project":"openstack/keystone","branch":"stable/essex","topic":"bug/969088","id":"I18ae38af62b4c2b2423e20e436611fc30f844ae1","number":"7385","subject":"Make import_nova_auth only create roles which don\u0027t already exist","owner":
|
||||
{"name":"Chuck Short","email":"chuck.short@canonical.com","username":"zulcss"},"url":"https://review.fuel-infra.org/7385"},
|
||||
"patchSet":
|
||||
{"number":"1","revision":"aff45d69a73033241531f5e3542a8d1782ddd859","ref":"refs/changes/85/7385/1","uploader":
|
||||
{"name":"Chuck Short","email":"chuck.short@canonical.com","username":"zulcss"},
|
||||
"createdOn":1337002189},
|
||||
"author":
|
||||
{"name":"Mark McLoughlin","email":"markmc@redhat.com","username":"markmc"},
|
||||
"approvals":
|
||||
[{"type":"CRVW","description":"Code Review","value":"2"},{"type":"APRV","description":"Approved","value":"0"}],
|
||||
"comment":"Hmm, I actually thought this was in Essex already.\n\nIt\u0027s a pretty annoying little issue for folks migrating for nova auth. Fix is small and pretty safe. Good choice for backporting"}
|
||||
|
||||
For most purposes you will want to trigger on ``patchset-created`` for when a
|
||||
new patchset has been uploaded.
|
||||
|
||||
Further documentation on how to use the events stream can be found in `Gerrit's stream event documentation page <http://gerrit-documentation.googlecode.com/svn/Documentation/2.3/cmd-stream-events.html>`_.
|
||||
|
||||
Posting Result To Gerrit
|
||||
------------------------
|
||||
|
||||
External testing systems can give non-gating votes to Gerrit by means
|
||||
of a -1/+1 verify vote. Comments should also be provided to explain what kind
|
||||
of test failed. We do also ask that the comments contain public links to the
|
||||
failure so that the developer can see what caused the failure.
|
||||
|
||||
An example of how to post this is as follows:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ ssh -p 29418 USERNAME@review.fuel-infra.org gerrit review -m '"Test failed on MegaTestSystem <http://megatestsystem.org/tests/1234>"' --verified=-1 c0ff33
|
||||
|
||||
In this example ``c0ff33`` is the commit ID for the review. You can
|
||||
set the verified to either `-1` or `+1` depending on whether or not it
|
||||
passed the tests.
|
||||
|
||||
Further documentation on the `review` command in Gerrit can be found in the `Gerrit review documentation page <http://gerrit-documentation.googlecode.com/svn/Documentation/2.3/cmd-review.html>`_.
|
||||
|
||||
We do suggest cautious testing of these systems and have a development Gerrit
|
||||
setup to test on if required. In SmokeStack's case all failures are manually
|
||||
reviewed before getting pushed to OpenStack, while this may not scale it is
|
||||
advisable during the initial testing of the setup.
|
||||
|
||||
There are several triggers that gerrit will match to alter the
|
||||
formatting of comments. The raw regular expressions can be seen in
|
||||
`gerrit.pp <https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/gerrit.pp>`_.
|
||||
For example, to have your test results formatted in the same manner as
|
||||
the upstream Jenkins results, use a template for each result matching::
|
||||
|
||||
* test-name-no-spaces http://link.to/result : [SUCCESS|FAILURE] some comment about the test
|
||||
|
||||
.. _request-account-label:
|
||||
|
||||
Creating a Service Account
|
||||
--------------------------
|
||||
|
||||
In order to post comments as a Third Party CI System and eventually verify
|
||||
your build status on Gerrit patches, you will need a dedicated Gerrit
|
||||
CI account. You will need to create this account in our OpenID provider
|
||||
`Launchpad <https://launchpad.net>`_. You may already have an existing
|
||||
personal account in Launchpad, but you should create a new and entirely
|
||||
separate account for this purpose.
|
||||
|
||||
Once you have created this account with the OpenID provider you can log
|
||||
into Gerrit with that new account as you would with your normal user
|
||||
account. Once logged in you will need to do several things:
|
||||
|
||||
1. Set an SSH username at https://review.fuel-infra.org/#/settings/ if
|
||||
it isn't already set. This is the username your CI system will use to
|
||||
SSH to Gerrit in order to read the event stream.
|
||||
|
||||
2. Set the account's fullname at https://review.fuel-infra.org/#/settings/contact
|
||||
This name should follow a few rules in order to make it clear in Gerrit
|
||||
comments what this CI system exists to test. The name should have three
|
||||
pieces ``Organization`` ``Product/technology`` ``CI designator``. The
|
||||
organization value should be your company name or other organization
|
||||
affiliation. Product/technology should describe the product or technology
|
||||
you are testing in conjunction with OpenStack. This should be the name of
|
||||
a component which cannot be tested in the official OpenStack
|
||||
infrastructure (requires particular physical hardware, proprietary
|
||||
software, some hypervisor feature not available in public clouds,
|
||||
et cetera). Note this should not be the name of an OpenStack project but
|
||||
rather the thing you are testing with OpenStack projects. And finally
|
||||
the CI designator is used to denote this is a CI system so that automatic
|
||||
Gerrit comment parsers can filter these comments out. This value should
|
||||
be ``CI`` for most CI systems but can be ``Bot`` if you are not
|
||||
performing continuous integration. An example of a proper name would be
|
||||
something like ``IBM DB2 CI``.
|
||||
|
||||
3. Add the SSH public key you will be using to the Gerrit account at
|
||||
https://review.fuel-infra.org/#/settings/ssh-keys You can generate an
|
||||
ssh key using ``ssh-keygen``. You want to give Gerrit the contents of
|
||||
the generated id_rsa.pub file.
|
||||
|
||||
Once you have done this you will have everything you need to comment on
|
||||
Gerrit changes from our CI system but you will not be able to vote +/-1
|
||||
Verified on changes. To get voting rights you will need to get the release
|
||||
group of the project you are testing to add you to their project specific
|
||||
<project>-ci group. Please contact the project in question when you are
|
||||
ready to start voting and they can add you to this group.
|
||||
|
||||
|
||||
|
||||
The Jenkins Gerrit Trigger Plugin Way
|
||||
-------------------------------------
|
||||
|
||||
There is a Gerrit Trigger plugin for Jenkins which automates all of the
|
||||
processes described in this document. So if your testing system is Jenkins
|
||||
based you can use it to simplify things. You will still need an account to do
|
||||
this as described in the :ref:`request-account-label` section above.
|
||||
|
||||
The Gerrit Trigger plugin for Jenkins can be found on `the Jenkins
|
||||
repository`_. You can install it using the Advanced tab in the
|
||||
Jenkins Plugin Manager.
|
||||
|
||||
.. _the Jenkins repository: http://repo.jenkins-ci.org/repo/com/sonyericsson/hudson/plugins/gerrit/gerrit-trigger/
|
||||
|
||||
Once installed Jenkins will have a new `Gerrit Trigger` option in the `Manage
|
||||
Jenkins` menu. This should be given the following options::
|
||||
|
||||
Hostname: review.fuel-infra.org
|
||||
Frontend URL: https://review.fuel-infra.org/
|
||||
SSH Port: 29418
|
||||
Username: (the Gerrit user)
|
||||
SSH Key File: (path to the user SSH key)
|
||||
|
||||
Verify
|
||||
------
|
||||
Started: 0
|
||||
Successful: 1
|
||||
Failed: -1
|
||||
Unstable: 0
|
||||
|
||||
Code Review
|
||||
-----------
|
||||
Started: 0
|
||||
Successful: 0
|
||||
Failed: 0
|
||||
Unstable: 0
|
||||
|
||||
(under Advanced Button):
|
||||
|
||||
Stated: (blank)
|
||||
Successful: gerrit approve <CHANGE>,<PATCHSET> --message 'Build Successful <BUILDS_STATS>' --verified <VERIFIED> --code-review <CODE_REVIEW>
|
||||
Failed: gerrit approve <CHANGE>,<PATCHSET> --message 'Build Failed <BUILDS_STATS>' --verified <VERIFIED> --code-review <CODE_REVIEW>
|
||||
Unstable: gerrit approve <CHANGE>,<PATCHSET> --message 'Build Unstable <BUILDS_STATS>' --verified <VERIFIED> --code-review <CODE_REVIEW>
|
||||
|
||||
Note that it is useful to include something in the messages about what testing
|
||||
system is supplying these messages.
|
||||
|
||||
When creating jobs in Jenkins you will have the option to add triggers. You
|
||||
should configure as follows::
|
||||
|
||||
Trigger on Patchset Uploaded: ticked
|
||||
(the rest unticked)
|
||||
|
||||
Type: Plain
|
||||
Pattern: openstack/project-name (where project-name is the name of the project)
|
||||
Branches:
|
||||
Type: Path
|
||||
Pattern: **
|
||||
|
||||
This job will now automatically trigger when a new patchset is
|
||||
uploaded and will report the results to Gerrit automatically.
|
||||
|
||||
Testing your CI setup
|
||||
---------------------
|
||||
|
||||
You can use the ``fuel-external/test`` project to test your external CI
|
||||
infrastructure with OpenStack's Gerrit. By using the sandbox project you
|
||||
can test your CI system without affecting regular OpenStack reviews.
|
||||
|
||||
Once you confirm your CI system works as you expect, change your
|
||||
configuration of the gerrit trigger plugin or zuul to subscribe to gerrit
|
||||
events from your target project.
|
||||
|
||||
Permissions on your Third Party System
|
||||
--------------------------------------
|
||||
|
||||
When you create your CI account it will have no special permissions.
|
||||
This means it can comment on changes but generally not vote +/-1
|
||||
Verified on any changes. The exception to this is on the
|
||||
``fuel-external/test`` project. Any account is able to vote +/-1
|
||||
Verified on that account and it provides a way to test your CI's voting
|
||||
abilities before you vote on other projects.
|
||||
|
||||
.. _openstack-dev/ci-sandbox: https://review.fuel-infra.org/[ADDME]
|
||||
|
||||
The Fuel Infrastructure team disables mis-behaving third-party ci
|
||||
accounts at its discretion. This documentation endeavours to outline specific
|
||||
circumstances that may lead to an account being disabled. There have been
|
||||
times when third-party ci systems behave in ways we didn't envision
|
||||
and therefore were unable to document prior to the event. If your
|
||||
third-party ci system has been disabled, please don't hesitate to contact
|
||||
devops team.
|
||||
|
||||
In order to get your Third Pary CI account to have voting permissions on
|
||||
repos in gerrit in addition to ``fuel-external/test`` you have a greater
|
||||
chance of success if you follow these steps:
|
||||
|
||||
* Set up your system and test it according to "Testing your CI setup" outlined
|
||||
above (this will create a history of activity associated with your account
|
||||
which will be evaluated when you apply for voting permissions).
|
||||
|
||||
* Post comments, that adhere to the "Requirements" listed above, that
|
||||
demonstrate the format for your system communication to the repos
|
||||
you want your system to test.
|
||||
|
||||
* Once your Third Party Account has a history on gerrit so that others
|
||||
can evaluate your format for comments, and the stability of your
|
||||
voting pattern (in the sandbox repo):
|
||||
|
||||
* send an email to the fuel-devops mailing list nominating your
|
||||
system for voting permissions
|
||||
|
||||
* fuel-devops@mirantis.com
|
||||
|
||||
* present your account history
|
||||
* address any questions and concerns with your system
|
||||
|
||||
* If the members of the program you want voting permissions from agree
|
||||
your system should be able to vote, the release group for that program
|
||||
or project can add you to the <project>-ci group specific to that
|
||||
program/project.
|
||||
Loading…
Reference in New Issue