opendev/system-config
Code Issues Proposed changes
c5b95b55fa
system-config/doc/source/gerrit.rst
Clark Boylan bdeef4e4ce Add info on running jstack against gerrit to docs 
This is meant to help debug gerrit in some circumstances particularly
now that the Java Melody plugin is not installed.

Change-Id: Ifedb7abd08c7fe1281ac510c6872fe8d9fe700a1
2022-01-31 12:06:05 -08:00

585 lines
19 KiB
ReStructuredText
Raw Blame History

:title: Gerrit
.. _gerrit:
Gerrit
######
Gerrit is the code review system used by the OpenStack project. For a
full description of how the system fits into the OpenStack workflow,
see `the development workflow guide
<https://docs.opendev.org/opendev/infra-manual/latest/developers.html#development-workflow>`_.
This section describes how Gerrit is configured for use in the
OpenStack project and the tools used to manage that configuration.
At a Glance
===========
:Hosts:
* https://review.opendev.org
:Ansible:
* :git_file:`playbooks/service-review.yaml`
* :git_file:`playbooks/roles/gerrit`
:Configuration:
* :git_file:`playbooks/roles/gerrit/templates/projects.ini.j2`
* :config:`gerrit/projects.yaml`
:Projects:
* http://code.google.com/p/gerrit/
:Bugs:
* https://storyboard.openstack.org/#!/project/715
* http://code.google.com/p/gerrit/issues/list
:Resources:
* `Gerrit Documentation <https://review.opendev.org/Documentation/index.html>`_
Installation
============
Gerrit is installed and configured by Ansible, using a Docker container
that contains the Java WAR file.
Cinder Volumes
--------------
The Gerrit installation at /home/gerrit2 is located on a Cinder
volume. See :ref:`cinder` for details on volume management. Note
that SSD volumes are used (and they have a minimum size of 100G).
Gerrit Configuration
--------------------
Most of Gerrit's configuration is in configuration files or Git
repositories (and in our case, managed by Puppet), but a few items
must be configured in the database. The following is a record of
these changes:
Add information about the CLA:
.. code-block:: mysql
sudo -u root
mysql
use reviewdb;
insert into contributor_agreements values (
'Y', 'Y', 'Y', 'ICLA',
'OpenStack Individual Contributor License Agreement',
'static/cla.html', 2);
Groups
------
A number of system-wide groups are configured in Gerrit (rather than
via Puppet). When installing a new Gerrit, you should create these by
hand (and capture their UUID - you will need them to setup the ACLs
later).
The `Project Bootstrappers` group grants all the permissions needed to
set up a new project. Normally, the OpenStack Project Creater account
is the only member of this group, but members of the `Administrators`
group may temporarily add themselves in order to correct problems with
automatic project creation.
The `Third-Party CI` group is used to grant +/-1 Verified
access to external testing tools on a sandbox project.
The `Voting Third-Party CI` group is used to grant +/-1 Verified
access to external testing tools for all projects.
The `Continuous Integration Tools` group contains Zuul and any
other CI tools that get +2/-2 access on reviews.
Users
-----
The first user to log in becomes an administrator. Be sure to set an
account name and add ssh keys - you'll need those.
Once you've created your groups you should create the
``openstack-project-creator`` account by hand (the account name is
referenced from
:git_file:`playbooks/roles/gerrit/templates/projects.ini.j2`)
using::
cat $pubkey | ssh -p 29418 $USER@$HOST gerrit create-account \
--group "'Project Bootstrappers'" \
--group Administrators \
--full-name "'Project Creator'" \
--email openstack-infra@lists.openstack.org \
--ssh-key - openstack-project-creator
.. _gerrit_github_integration:
GitHub Integration
==================
Gerrit replicates to GitHub by pushing to a standard Git remote. The
GitHub projects are configured to allow only the Gerrit user to push.
Pull requests can not be disabled for a project in Github, so instead
we have a script that runs from cron to close any open pull requests
with instructions to use Gerrit.
These are both handled automatically by :ref:`jeepyb`.
Note that the user running Gerrit will need to accept the GitHub host
keys. e.g.::
sudo su - gerrit2
ssh github.com
Troubleshooting
---------------
When creating a new project, there can be times where the :ref:`jeepyb`
automation to create the GitHub project can fail, and leave the project
improperly configured.
This can cause replication to GitHub to fail. The project in GitHub will
be created, but will appear empty. When trying replication from Gerrit,
it will show a `Permission denied` error when trying to push content.
To solve that, following steps are needed:
#. Login into github.com, using openstack-project-creator user.
#. Navigate to the failed repository, and enter on Settings > Collaborators
& teams option.
#. Add Gerrit as Team member to that project.
After the team has been added, project will start replicating successfully
to GitHub.
Gerrit IRC Bot
==============
Gerritbot consumes the Gerrit event stream and announces relevant
events on IRC. :ref:`gerritbot` is an openstack-infra project and is
also available on Pypi.
Launchpad Bug Integration
=========================
In addition to the hyperlinks provided by the regex in gerrit.config,
we use a Gerrit hook to update Launchpad bugs when changes referencing
them are applied. This is managed by the :ref:`jeepyb`
openstack-infra project.
Storyboard Integration
======================
We use the Gerrit its-storyboard_ plugin to update :ref:`storyboard`
stories and tasks when changes referencing them are applied.
.. _its-storyboard: https://review.opendev.org/plugins/its-storyboard/Documentation/index.html
New Project Creation
====================
Gerrit project creation is now managed through changes to the
openstack/project-config repository. :ref:`jeepyb` handles
automatically creating any new projects defined in the configuration
files.
Local Git Replica
=================
Gerrit replicates all repos to a local directory so that Apache can
serve the anonymous http requests out directly. This is automatically
configured by :ref:`jeepyb`.
.. _acl:
Access Controls
===============
High level goals:
#. Anonymous users can read all projects.
#. All registered users can perform informational code review (+/-1)
on any project.
#. Zuul can perform verification (blocking or approving: +/-2).
#. Third Party CI systems can perform informational verification (+/-1).
#. All registered users can create changes.
#. Members of $PROJECT-core group can perform full code review
(blocking or approving: +/- 2), and submit changes to be merged.
#. Drivers (PTL and delegates) of client library projects should be
able to add tags (which are automatically used to trigger
releases).
The global Gerrit permissions set out the high level goals (and
manage-projects can then override this on a per project basis as
needed). To setup the global permissions, first create the groups
covered above under Groups.
You need to grant yourself enough access to replace the ACLs over ssh (we use
SSH because it's fast, and it gets syntax checked).
#. Visit ``https://$HOST/#/admin/projects/All-Projects,access`` and click on Edit.
#. Look for the reference to 'refs/meta/config', click on the drop-box
for 'add permission' and choose 'PUSH'.
#. Type in Administrators as the group name
#. Click on Add
#. Click on Save Changes
Then... we need to fetch the All-Projects ACLs, update them, then push the
updates back into Gerrit::
export USER=$your_gerrit_user
export HOST=$your_gerrit_host
cd $anywhereyoulike
mkdir All-Projects-ACLs
cd All-Projects-ACLs
git init
git remote add gerrit ssh://$USER@$HOST:29418/All-Projects.git
git fetch gerrit +refs/meta/*:refs/remotes/gerrit-meta/*
git checkout -b config remotes/gerrit-meta/config
There will be two interesting files, `groups` and `project.config`.
`groups` contains UUIDs and names of groups that will be referenced
in `project.config`. UUIDs can be found on the group page in Gerrit.
Next, edit `project.config` to look like::
[access "refs/*"]
create = group Project Bootstrappers
forgeAuthor = group Registered Users
forgeCommitter = group Project Bootstrappers
push = +force group Project Bootstrappers
pushMerge = group Project Bootstrappers
pushSignedTag = group Project Bootstrappers
pushTag = group Continuous Integration Tools
pushTag = group Project Bootstrappers
read = group Anonymous Users
editTopicName = group Registered Users
[access "refs/drafts/*"]
push = block group Registered Users
[access "refs/for/refs/*"]
push = group Registered Users
[access "refs/for/refs/zuul/*"]
pushMerge = group Continuous Integration Tools
[access "refs/heads/*"]
label-Code-Review = -2..+2 group Project Bootstrappers
label-Code-Review = -1..+1 group Registered Users
label-Verified = -2..+2 group Continuous Integration Tools
label-Verified = -2..+2 group Project Bootstrappers
label-Verified = -1..+1 group Continuous Integration Tools Development
label-Verified = -1..+1 group Voting Third-Party CI
label-Workflow = -1..+0 group Change Owner
label-Workflow = -1..+1 group Project Bootstrappers
rebase = group Registered Users
submit = group Continuous Integration Tools
submit = group Project Bootstrappers
[access "refs/meta/config"]
read = group Project Owners
[access "refs/meta/openstack/*"]
create = group Continuous Integration Tools
push = group Continuous Integration Tools
read = group Continuous Integration Tools
[access "refs/zuul/*"]
create = group Continuous Integration Tools
push = +force group Continuous Integration Tools
pushMerge = group Continuous Integration Tools
[capability]
accessDatabase = group Administrators
administrateServer = group Administrators
createProject = group Project Bootstrappers
emailReviewers = deny group Third-Party CI
priority = batch group Service Users
runAs = group Project Bootstrappers
streamEvents = group Registered Users
[contributor-agreement "ICLA"]
accepted = group CLA Accepted - ICLA
agreementUrl = static/cla.html
autoVerify = group CLA Accepted - ICLA
description = OpenStack Individual Contributor License Agreement
[contributor-agreement "System CLA"]
accepted = group System CLA
agreementUrl = static/system-cla.html
description = DON'T SIGN THIS: System CLA (externally managed)
[contributor-agreement "USG CLA"]
accepted = group USG CLA
agreementUrl = static/usg-cla.html
description = DON'T SIGN THIS: U.S. Government CLA (externally managed)
[label "Code-Review"]
abbreviation = R
copyAllScoresOnTrivialRebase = true
copyMinScore = true
function = MaxWithBlock
value = -2 Do not merge
value = -1 This patch needs further work before it can be merged
value = 0 No score
value = +1 Looks good to me, but someone else must approve
value = +2 Looks good to me (core reviewer)
[label "Verified"]
function = MaxWithBlock
value = -2 Fails
value = -1 Doesn't seem to work
value = 0 No score
value = +1 Works for me
value = +2 Verified
[label "Workflow"]
function = MaxWithBlock
value = -1 Work in progress
value = 0 Ready for reviews
value = +1 Approved
[plugin "its-storyboard"]
enabled = true
[project]
description = Rights inherited by all other projects
Now edit the groups file. The format is::
#UUID Group Name
1234567890123456789012345678901234567890 group-foo
Each of the groups listed above under 'Groups' should have an entry as well as
the built in groups such as 'Service Users' which may or may not be
present in the initial groups file. You can find the UUID values by navigating
to Admin -> Groups -> Group Name -> General in the Web UI.
Finally, commit the changes and push the config back up to Gerrit::
git commit -am "Initial All-Projects config"
git push gerrit HEAD:refs/meta/config
Manual Administrative Tasks
===========================
The following sections describe tasks that individuals with root
access may need to perform on rare occasions.
Renaming a Project
------------------
Renaming a project is not automated and is disruptive to developers,
so it should be avoided. Allow for an hour of downtime for the
project in question, and about 10 minutes of downtime for all of
Gerrit. All Gerrit changes, merged and open, will carry over, so
in-progress changes do not need to be merged before the move.
To rename a project:
#. Prepare a change to the project-config repo to update things like
projects.yaml, Gerrit ACLs, zuul and gerritbot for the new name.
#. Prepare a yaml file called repos.yaml that has a single dictionary called
`repos` with a list of dictionaries each having an old and new entry.
Optionally also add a `gerrit_groups` dict of the same form if groups
are being renamed::
repos:
- old: stackforge/awesome-repo
new: openstack/awesome-repo
- old: openstack/foo
new: openstack/bar
gerrit_groups:
- old: old-core-group
new: new-core-group
Add this file to the ``renames/`` directory in the
``opendev/project-config`` repository.
#. An hour in advance of the maintenance (if possible), put
``review02.opendev.org``, ``gitea01-8.opendev.org``, and
``storyboard01.opendev.org`` into the emergency file on bridge.
#. Check that all servers involved in the rename playbook
(review, zuul-scheduler, storyboard, storyboard-dev, and the giteas) are
responding to ssh to ensure the next step can run successfully.
#. Run the ansible rename repos playbook, passing in the path to your yaml
file::
sudo ansible-playbook -f 10 /home/zuul/srv/opendev.org/opendev/system-config/playbooks/rename_repos.yaml -e repolist=ABSOLUTE_PATH_TO_VARS_FILE
#. :ref:`Force-merge <force-merging-a-change>` the prepared configuration
changes.
#. Wait for the changes merged above to replicate to the giteas.
.. warning::
Not waiting at this step can cause manage-projects to run with
our old pre rename state causing the project to be created under
its old name.
#. Remove ``review02.opendev.org``, ``gitea01-8.opendev.org``, and
``storyboard01.opendev.org`` from the emergency file.
#. Ensure that the next manage-projects run does not update the giteas
or review servers. It should be a noop.
Developers will either need to re-clone a new copy of the repository,
or manually update their remotes with something like::
git remote set-url origin https://opendev.org/$ORG/$PROJECT
Third-Party Testing Access
--------------------------
The command to add an account for an automated system which gets -1/+1
code verify voting rights (as outlined in :ref:`third-party-testing`)
looks like:
.. code-block:: bash
ssh -p 29418 review.opendev.org "gerrit create-account \
--group 'Third-Party CI' \
--full-name 'Some CI Bot' \
--email ci-bot@third-party.org \
--ssh-key 'ssh-rsa AAAAB3Nz...zaUCse1P ci-bot@third-party.org' \
some-ci-bot"
Details on the create-account_ command can be found in the Gerrit
API documentation.
.. _create-account: https://review.opendev.org/Documentation/cmd-create-account.html
Deleting Accounts in Gerrit
---------------------------
We can not delete accounts. They can be made inactive.
Duplicate Accounts in Gerrit
----------------------------
If a user has two accounts, we can not combine them. We can only
deactivate one of them.
For example, user ``foo`` has an account ``foo@company.com`` and moves
to a new job, creating a new account ``foo@new.com``. They log-in
with ``foo@new.com``, but then realise what they really wanted to do
was *add* this new address to their existing account
(i.e. ``foo@company.com``).
The first step to resolve this is to confirm the ID of the unwanted
account, . As an admin user with a HTTP password set, search for the
new account:
..
curl -u you.admin -i -H "Accept: application/json" 'https://review.opendev.org/a/accounts/foo@new.com'
That will return an ``_acount_id``. For this example, assume it is
``12345``. The user should check in their settings they are *not*
using this account.
Clone ``All-Users`` to modify the account, and checkout the account
config, which is sharded by the last two digits of the account-id.
..
git clone ssh://you.admin@review.opendev.org:29418/All-Users
git fetch refs/users/45/12345
git checkout FETCH_HEAD
Edit the ``[account]`` section of ``account.conf`` to remove
``preferredEmail`` and have a line ``active = false``. Put your admin
account into ``Project Bootstrappers`` (see :ref:`sysadmin`) and
commit this
..
git commit -m "Make duplicate account inactive" --author <your@email.com>
git push origin HEAD:refs/users/45/12345
There will still be an OpenID external ID associated with this now
inactive account. This will prevent adding ``foo@new.com`` to another
account until this is removed.
Check this via the API with
..
curl -u you.admin -i -H "Accept: application/json" https://review.opendev.org/a/accounts/12345/external.ids
This will give a json result with an ``identity`` URL like
``"identity":"https://login.ubuntu.com/+id/RaND0m``. Use this to
delete the record with another call
..
curl -XPOST -u you.admin -i -H "Content-Type: application/json" -d '["https://login.ubuntu.com/+id/RaND0m"]' https://review.opendev.org/a/accounts/12345/external.ids:delete
The user should now be able to add ``foo@new.com`` to their old
account.
Deactivating a Gerrit account
-----------------------------
To deactivate a Gerrit account (use case can be a failing Third Party CI), you
must follow that steps:
1. Identify the account ID of the Third Party CI you need to deactivate. Third-Party CI
members can be found on: https://review.opendev.org/#/admin/groups/270,members
That will give you the name and email of all members. Then you can get the matching
numerical account ID with the help of REST API::
curl -i -H "Accept: application/json" --digest --user <<gerrit_user>>:<<http_pass>> -X GET https://review.opendev.org/a/accounts/{email}
This will return a JSON dictionary, that will contain _account_id field.
2. Mark the account as inactive using gerrit ssh api, with::
ssh -p 29418 review.opendev.org gerrit set-account --inactive {account-id}
Alternatively you can use REST API, sending a DELETE for::
curl -i -H "Accept: application/json" --digest --user <<gerrit_user>>:<<http_pass>> -X DELETE https://review.opendev.org/a/accounts/{account-id}/active
3. Check if there are active gerrit ssh connections::
ssh -p 29418 review.opendev.org gerrit show-connections -n | grep {account-id}
And kill all of them with subsequent::
ssh -p 29418 review.opendev.org gerrit close-connection {connection-id}
4. You can check if the account is properly marked as inactive using REST API,
sending a GET for::
curl -i -H "Accept: application/json" --digest --user <<gerrit_user>>:<<http_pass>> -X GET https://review.opendev.org/a/accounts/{account-id}/active
A 200 return code means the account is active, and 204 means account inactive.
4. In the case of a failing Third Party CI, if the account caused a loop of comments in
a change, you can delete them with following query::
delete from change_messages where author_id={account-id} and change_id={change-id};
Generating a Thread Dump for Debugging
--------------------------------------
We removed the Java Melody plugin the wake of the Log4Shell vulnerability.
This removed an easy way to acquire a thread dump but dumping threads is
still possible with java command line tools. You may find yourself wanting
to do this if Gerrit is suffering from poor performance or you are trying to
debug odd Gerrit behavior.
To run jstack and produce a thread dump do::
review02# docker exec -it gerrit-compose_gerrit_1 bash
gerrit@review02:/$ ps -ef | grep java # find the Gerrit java process PID
gerrit@review02:/$ jstack $PID > /tmp/dump.yearmonthday
View Git Blame Copy Permalink