Update some docs for opendev

There's a lot of these, so doing them in chunks. This fixes
the custom roles.

Remove the git and jjb docs, since we don't use them anymore.

Change-Id: I0c5b74f7b73315dac93bce6be0d920cddb94fb58
This commit is contained in:
Monty Taylor 2019-04-20 13:59:34 +00:00 committed by James E. Blair
parent c6d129a108
commit e01ed4f066
24 changed files with 104 additions and 419 deletions

View File

@ -27,7 +27,7 @@ At a Glance
* afs02.dfw.openstack.org (a second fileserver in DFW)
* afs01.ord.openstack.org (a fileserver in ORD)
:Puppet:
* https://git.openstack.org/cgit/openstack-infra/puppet-openafs/tree/
* https://opendev.org/opendev/puppet-openafs
* :cgit_file:`modules/openstack_project/manifests/afsdb.pp`
* :cgit_file:`modules/openstack_project/manifests/afsfs.pp`
:Projects:
@ -508,7 +508,7 @@ Reverse Proxy Cache
^^^^^^^^^^^^^^^^^^^
* `modules/openstack_project/templates/mirror.vhost.erb
<https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/mirror.vhost.erb>`__
<https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/templates/mirror.vhost.erb>`__
Each of the region-local mirror hosts exposes a limited reverse HTTP
proxy on port 8080. These proxies run within the same Apache setup as

View File

@ -14,7 +14,7 @@ At a Glance
* https://ask.openstack.org
* https://ask-staging.openstack.org
:Puppet:
* https://git.openstack.org/cgit/openstack-infra/puppet-askbot/tree/
* https://opendev.org/opendev/puppet-askbot
* https://github.com/vamsee/puppet-solr
* :cgit_file:`modules/openstack_project/manifests/ask.pp`
* :cgit_file:`modules/openstack_project/manifests/ask-staging.pp`
@ -28,7 +28,7 @@ Overview
The site ask.openstack.org based on the officially released askbot pip distribution.
The stable deployment is extended with a custom OpenStack theme available at
https://git.openstack.org/cgit/openstack-infra/askbot-theme.
https://opendev.org/opendev/askbot-theme.
The ask-staging.openstack.org site based on master branch of
https://github.com/askbot/askbot-devel repository, and deploys askbot

View File

@ -13,7 +13,7 @@ At a Glance
:Hosts:
* sip://pbx.opendev.org
:Puppet:
* https://git.openstack.org/cgit/openstack-infra/puppet-asterisk/tree/
* https://opendev.org/opendev/puppet-asterisk
* :cgit_file:`modules/openstack_project/manifests/pbx.pp`
:Projects:
* http://www.asterisk.org

View File

@ -14,7 +14,7 @@ At a Glance
:Hosts:
* http://codesearch.openstack.org
:Puppet:
* https://git.openstack.org/cgit/openstack-infra/puppet-hound/tree/
* https://opendev.org/opendev/puppet-hound
* :cgit_file:`modules/openstack_project/manifests/codesearch.pp`
:Projects:
* https://github.com/etsy/Hound

View File

@ -150,7 +150,7 @@ The public portions can be proposed via the standard review process at
any time by anyone. Exact details of cloud configuration changes from
time to time; the best way to begin the addition is to clone the
``system-configuration`` repository (i.e. this repo) with ``git clone
https://git.openstack.org/openstack-infra/system-config`` and ``grep``
https://opendev.org/opendev/system-config`` and ``grep``
for an existing cloud (or go through ``git log`` and find the last
cloud added) and follow the pattern. After posting the review, CI
tests and reviewers will help with any issues.
@ -174,7 +174,7 @@ and the cloud will become active.
Once active, ``bridge.openstack.org`` will begin regularly running
`ansible-role-cloud-launcher
<http://git.openstack.org/cgit/openstack/ansible-role-cloud-launcher/>`__
<http://opendev.org/opendev/ansible-role-cloud-launcher/>`__
against the new cloud to configure keys, upload base images, setup
security groups and such.
@ -191,7 +191,7 @@ to start running testing nodes.
At this point, the cloud needs to be added to nodepool configuration
in `project-config
<https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool>`__.
<https://opendev.org/openstack/project-config/src/branch/master/nodepool>`__.
Again existing entries provide useful templates for the initial review
proposal, which can be done by anyone. Some clouds provision
particular flavors for CI nodes; these need to be present at this
@ -201,7 +201,7 @@ checks and reviewers will help with any fine details.
Once this is committed, nodepool will upload images into the new
region and start running nodes automatically. Don't forget to add the
region to the `grafana
<https://git.openstack.org/cgit/openstack-infra/project-config/tree/grafana>`__
<https://opendev.org/openstack/project-config/src/branch/master/grafana>`__.
configuration to ensure we have a dashboard for the region's health.
Ongoing operation

View File

@ -39,8 +39,8 @@ def cgit_file_role(name, rawtext, text, lineno, inliner,
:param content: The directive content for customization.
"""
ref = ('https://git.openstack.org/cgit/openstack-infra/'
'system-config/tree/%s' % text)
ref = ('https://opendev.org/opendev/'
'system-config/src/branch/master/%s' % text)
linktext = 'system-config: %s' % text
node = nodes.reference(rawtext, linktext, refuri=ref, **options)
return [node], []
@ -63,8 +63,8 @@ def config_role(name, rawtext, text, lineno, inliner,
:param content: The directive content for customization.
"""
ref = ('https://git.openstack.org/cgit/openstack-infra/'
'project-config/tree/%s' % text)
ref = ('https://opendev.org/openstack/'
'project-config/src/branch/master/%s' % text)
linktext = 'project-config: %s' % text
node = nodes.reference(rawtext, linktext, refuri=ref, **options)
return [node], []

View File

@ -19,11 +19,11 @@ At a Glance
===========
:Projects:
* https://git.openstack.org/cgit/openstack-infra/devstack-gate
* https://opendev.org/openstack/devstack-gate
:Bugs:
* https://storyboard.openstack.org/#!/project/712
:Resources:
* `Devstack-gate README <https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/README.rst>`_
* `Devstack-gate README <https://opendev.org/openstack/devstack-gate/src/branch/master/README.rst>`_
Overview
========
@ -66,11 +66,11 @@ How to Debug a Devstack Gate Failure
====================================
Instructions for debugging a failure can be found in the
`Devstack-gate README <https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/README.rst>`_
`Devstack-gate README <https://opendev.org/openstack/devstack-gate/src/branch/master/README.rst>`_
Developer Setup
===============
If you'd like to work on the devstack-gate scripts and test process,
see the `Devstack-gate README <https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/README.rst>`_
see the `Devstack-gate README <https://opendev.org/openstack/devstack-gate/src/branch/master/README.rst>`_
for specific instructions.

View File

@ -13,10 +13,10 @@ At a Glance
:Hosts:
* http://status.openstack.org
:Puppet:
* https://git.openstack.org/cgit/openstack-infra/puppet-elastic_recheck/tree/
* https://opendev.org/opendev/puppet-elastic_recheck
* :cgit_file:`modules/openstack_project/manifests/status.pp`
:Projects:
* https://git.openstack.org/cgit/openstack-infra/elastic-recheck
* https://opendev.org/opendev/elastic-recheck
:Bugs:
* https://storyboard.openstack.org/#!/project/713
:Resources:

View File

@ -16,7 +16,7 @@ At a Glance
:Hosts:
* http://etherpad.openstack.org
:Puppet:
* https://git.openstack.org/cgit/openstack-infra/puppet-etherpad_lite/tree/
* https://opendev.org/opendev/puppet-etherpad_lite/tree/
* :cgit_file:`modules/openstack_project/manifests/etherpad.pp`
* :cgit_file:`modules/openstack_project/manifests/etherpad_dev.pp`
:Projects:

View File

@ -13,14 +13,14 @@ At a Glance
:Hosts:
* firehose*.openstack.org
:Puppet:
* https://git.openstack.org/cgit/openstack-infra/puppet-mosquitto
* https://git.openstack.org/cgit/openstack-infra/puppet-germqtt
* https://git.openstack.org/cgit/openstack-infra/puppet-lpmqtt
* https://opendev.org/opendev/puppet-mosquitto
* https://opendev.org/opendev/puppet-germqtt
* https://opendev.org/opendev/puppet-lpmqtt
* :cgit_file:`modules/openstack_project/manifests/firehose.pp`
:Projects:
* https://mosquitto.org/
* http://git.openstack.org/cgit/openstack-infra/germqtt/
* http://git.openstack.org/cgit/openstack-infra/lpmqtt/
* http://opendev.org/opendev/germqtt/
* http://opendev.org/opendev/lpmqtt/
Overview
========
@ -81,11 +81,11 @@ As of right now the following services publish messages to the firehose:
| logstash worker | gearman-logstash | `logstash-gearman-worker`_ |
+-----------------+------------------+----------------------------+
.. _germqtt: http://git.openstack.org/cgit/openstack-infra/germqtt/
.. _lpmqtt: http://git.openstack.org/cgit/openstack-infra/lpmqtt/
.. _subunit-gearman-worker: http://git.openstack.org/cgit/openstack-infra/puppet-subunit2sql/tree/files/subunit-gearman-worker.py
.. _ansible_mqtt_plugin: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/puppetmaster/mqtt.py
.. _logstash-gearman-worker: https://git.openstack.org/cgit/openstack-infra/puppet-log_processor/tree/files/log-gearman-worker.py
.. _germqtt: http://opendev.org/opendev/germqtt/
.. _lpmqtt: http://opendev.org/opendev/lpmqtt/
.. _subunit-gearman-worker: https://opendev.org/opendev/puppet-subunit2sql/src/branch/master/files/subunit-gearman-worker.py
.. _ansible_mqtt_plugin: https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/puppetmaster/mqtt.py
.. _logstash-gearman-worker: https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-worker.py
For a full schema description see :ref:`firehose_schema`

View File

@ -16,7 +16,7 @@ Messages on firehose for gerrit are generated using the `germqtt`_ project. For
the most part these are basically identical to what gerrit returns on it's
native event stream except over MQTT.
.. _germqtt: http://git.openstack.org/cgit/openstack-infra/germqtt/
.. _germqtt: http://opendev.org/opendev/germqtt/
Topics
------
@ -54,14 +54,14 @@ doc here just refer to gerrit's docs on the `JSON payload`_ which documents the
contents of each JSON object and refer to the doc on `Gerrit events`_ for which
JSON objects are included with which event type.
.. _JSON payload: https://review.openstack.org/Documentation/json.html
.. _Gerrit events: https://review.openstack.org/Documentation/cmd-stream-events.html#events
.. _JSON payload: https://review.opendev.org/Documentation/json.html
.. _Gerrit events: https://review.opendev.org/Documentation/cmd-stream-events.html#events
Launchpad
=========
The messages sent to firehose for launchpad are generated using `lpmqtt`_
.. _lpmqtt: http://git.openstack.org/cgit/openstack-infra/lpmqtt/
.. _lpmqtt: http://opendev.org/opendev/lpmqtt/
Topics
------
@ -121,7 +121,7 @@ Subunit Workers
The messages for the subunit workers are generated directly in the
`subunit gearman worker scripts`_.
.. _subunit gearman worker scripts: http://git.openstack.org/cgit/openstack-infra/puppet-subunit2sql/tree/files/subunit-gearman-worker.py
.. _subunit gearman worker scripts: https://opendev.org/opendev/puppet-subunit2sql/src/branch/master/files/subunit-gearman-worker.py
Topics
------
@ -156,7 +156,7 @@ Ansible
We have mqtt events emitted from ansible being run on :ref:`puppet-master`.
These events are generated using a `MQTT Ansible Callback Plugin`_.
.. _MQTT Ansible Callback Plugin: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/puppetmaster/mqtt.py
.. _MQTT Ansible Callback Plugin: https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/puppetmaster/mqtt.py
Topics
------
@ -402,7 +402,7 @@ Logstash Workers
The messages for the subunit workers are generated directly in the
`logstash gearman worker scripts`_.
.. _logstash gearman worker scripts: http://git.openstack.org/cgit/openstack-infra/puppet-log_processor/tree/files/log-gearman-worker.py
.. _logstash gearman worker scripts: https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-worker.py
Topics
------

View File

@ -17,10 +17,10 @@ At a Glance
===========
:Hosts:
* http://review.openstack.org
* http://review.opendev.org
* http://review-dev.openstack.org
:Puppet:
* https://git.openstack.org/cgit/openstack-infra/puppet-gerrit/tree/
* https://opendev.org/opendev/puppet-gerrit
* :cgit_file:`modules/openstack_project/manifests/review.pp`
* :cgit_file:`modules/openstack_project/manifests/review_dev.pp`
:Configuration:
@ -32,7 +32,7 @@ At a Glance
* https://storyboard.openstack.org/#!/project/715
* http://code.google.com/p/gerrit/issues/list
:Resources:
* `Gerrit Documentation <https://review.openstack.org/Documentation/index.html>`_
* `Gerrit Documentation <https://review.opendev.org/Documentation/index.html>`_
Installation
============
@ -189,7 +189,7 @@ Storyboard Integration
We use the Gerrit its-storyboard_ plugin to update :ref:`storyboard`
stories and tasks when changes referencing them are applied.
.. _its-storyboard: https://review.openstack.org/plugins/its-storyboard/Documentation/index.html
.. _its-storyboard: https://review.opendev.org/plugins/its-storyboard/Documentation/index.html
New Project Creation
====================
@ -478,7 +478,7 @@ To rename a project:
Developers will either need to re-clone a new copy of the repository,
or manually update their remotes with something like::
git remote set-url origin https://git.openstack.org/$ORG/$PROJECT
git remote set-url origin https://opendev.org/$ORG/$PROJECT
Third-Party Testing Access
@ -490,7 +490,7 @@ looks like:
.. code-block:: bash
ssh -p 29418 review.openstack.org "gerrit create-account \
ssh -p 29418 review.opendev.org "gerrit create-account \
--group 'Third-Party CI' \
--full-name 'Some CI Bot' \
--email ci-bot@third-party.org \
@ -500,7 +500,7 @@ looks like:
Details on the create-account_ command can be found in the Gerrit
API documentation.
.. _create-account: https://review.openstack.org/Documentation/cmd-create-account.html
.. _create-account: https://review.opendev.org/Documentation/cmd-create-account.html
Resetting a Username in Gerrit
------------------------------
@ -558,14 +558,14 @@ Then be sure to set the old account to inactive:
.. code-block:: bash
ssh review.openstack.org -p29418 gerrit set-account --inactive OLD
ssh review.opendev.org -p29418 gerrit set-account --inactive OLD
Finally, flush Gerrit's caches so any immediate account lookups will hit
the current DB contents:
.. code-block:: bash
ssh review.openstack.org -p29418 gerrit flush-caches --all
ssh review.opendev.org -p29418 gerrit flush-caches --all
Combining Gerrit Accounts
@ -621,7 +621,7 @@ against the current DB contents:
.. code-block:: bash
ssh review.openstack.org -p29418 gerrit flush-caches --all
ssh review.opendev.org -p29418 gerrit flush-caches --all
Make the user aware that these steps have also removed any group
memberships, preferences, SSH keys, CLA
@ -665,35 +665,35 @@ To deactivate a Gerrit account (use case can be a failing Third Party CI), you
must follow that steps:
1. Identify the account ID of the Third Party CI you need to deactivate. Third-Party CI
members can be found on: https://review.openstack.org/#/admin/groups/270,members
members can be found on: https://review.opendev.org/#/admin/groups/270,members
That will give you the name and email of all members. Then you can get the matching
numerical account ID with the help of REST API::
curl -i -H "Accept: application/json" --digest --user <<gerrit_user>>:<<http_pass>> -X GET https://review.openstack.org/a/accounts/{email}
curl -i -H "Accept: application/json" --digest --user <<gerrit_user>>:<<http_pass>> -X GET https://review.opendev.org/a/accounts/{email}
This will return a JSON dictionary, that will contain _account_id field.
2. Mark the account as inactive using gerrit ssh api, with::
ssh -p 29418 review.openstack.org gerrit set-account --inactive {account-id}
ssh -p 29418 review.opendev.org gerrit set-account --inactive {account-id}
Alternatively you can use REST API, sending a DELETE for::
curl -i -H "Accept: application/json" --digest --user <<gerrit_user>>:<<http_pass>> -X DELETE https://review.openstack.org/a/accounts/{account-id}/active
curl -i -H "Accept: application/json" --digest --user <<gerrit_user>>:<<http_pass>> -X DELETE https://review.opendev.org/a/accounts/{account-id}/active
3. Check if there are active gerrit ssh connections::
ssh -p 29418 review.openstack.org gerrit show-connections -n | grep {account-id}
ssh -p 29418 review.opendev.org gerrit show-connections -n | grep {account-id}
And kill all of them with subsequent::
ssh -p 29418 review.openstack.org gerrit close-connection {connection-id}
ssh -p 29418 review.opendev.org gerrit close-connection {connection-id}
4. You can check if the account is properly marked as inactive using REST API,
sending a GET for::
curl -i -H "Accept: application/json" --digest --user <<gerrit_user>>:<<http_pass>> -X GET https://review.openstack.org/a/accounts/{account-id}/active
curl -i -H "Accept: application/json" --digest --user <<gerrit_user>>:<<http_pass>> -X GET https://review.opendev.org/a/accounts/{account-id}/active
A 200 return code means the account is active, and 204 means account inactive.

View File

@ -1,77 +0,0 @@
:title: Git
.. _git:
Git
########
The web frontend cgit is running on git.openstack.org.
At a Glance
===========
:Hosts:
* https://git.openstack.org
* git*.openstack.org
:Puppet:
* https://git.openstack.org/cgit/openstack-infra/puppet-cgit/tree/
* :cgit_file:`modules/openstack_project/manifests/git.pp`
:Configuration:
* :cgit_file:`modules/openstack_project/files/git/cgitrc`
:Projects:
* http://git.zx2c4.com/cgit/
:Bugs:
* https://storyboard.openstack.org/#!/project/748
* http://lists.zx2c4.com/mailman/listinfo/cgit
Overview
========
The OpenStack git repositories are hosted on a pool of CentOS servers with the
EPEL repository that includes the cgit package. They are served up via https
using cgit and via git:// by git-daemon behind HAProxy which handles load
balancing across the nodes.
SELinux is enabled on these nodes and requires restorecon to be run against
/var/lib/git to set the appropriate SELinux security context. SELinux rules are
also in place to allow for Apache to run on a non-standard port so it can sit
behind HAProxy. This is all handled by puppet.
In order to mitigate potential problems with HTTP(S) responses, HAProxy is
configured using the source balance method so that every request from a single
host will be served by one backend node unless nodes are added or removed.
The jeepyb script create-cgitrepos runs against projects.yaml to generate the
/etc/cgitrepos file listing all the git repositories. The git repositories are
synced to all the nodes from the Gerrit server.
Backend Maintenance
===================
To temporarily remove a git backend from the HAProxy load balancer,
you can put it in "maintenance" mode. This can be done interactively
on the HAProxy host. Note that long-term changes to the topology
should be made via configuration management. These commands must be
run as root.
To see the current status of all servers::
echo "show stat" | socat /var/lib/haproxy/stats stdio
To disable a server (eg, git08)::
echo "disable server balance_git_daemon/git08.openstack.org" | socat /var/lib/haproxy/stats stdio
echo "disable server balance_git_http/git08.openstack.org" | socat /var/lib/haproxy/stats stdio
echo "disable server balance_git_https/git08.openstack.org" | socat /var/lib/haproxy/stats stdio
To re-enable a server::
echo "enable server balance_git_daemon/git08.openstack.org" | socat /var/lib/haproxy/stats stdio
echo "enable server balance_git_http/git08.openstack.org" | socat /var/lib/haproxy/stats stdio
echo "enable server balance_git_https/git08.openstack.org" | socat /var/lib/haproxy/stats stdio
To run these commands and others interactively, issue the prompt
command to haproxy::
socat readline /var/lib/haproxy/stats
prompt

View File

@ -13,14 +13,14 @@ At a Glance
===========
:Hosts:
* review.openstack.org
* review.opendev.org
:Puppet:
* https://git.openstack.org/cgit/openstack-infra/system-config/tree/
* https://opendev.org/opendev/system-config
* :cgit_file:`modules/openstack_project/manifests/gerrit.pp`
* :cgit_file:`hiera/group/zuul-scheduler.yaml`
:Projects:
* https://git.openstack.org/cgit/openstack-infra/zuul
* https://git.openstack.org/cgit/openstack-infra/jeepyb
* https://opendev.org/zuul/zuul
* https://opendev.org/opendev/jeepyb
:Chat:
* #openstack-infra on freenode

View File

@ -18,7 +18,7 @@ At a Glance
:Configuration:
* :config:`grafana`
:Projects:
* https://git.openstack.org/cgit/openstack-infra/grafyaml
* https://opendev.org/opendev/grafyaml
:Bugs:
* https://storyboard.openstack.org/#!/project/818
:Resources:

View File

@ -13,14 +13,14 @@ At a Glance
:Hosts:
* http://eavesdrop.openstack.org/
* http://review.openstack.org/
* http://review.opendev.org/
* https://wiki.openstack.org/wiki/Infrastructure_Status
* http://ptg.openstack.org/
:Puppet:
* https://git.openstack.org/cgit/openstack-infra/puppet-meetbot/tree/
* https://git.openstack.org/cgit/openstack-infra/puppet-statusbot/tree/
* https://git.openstack.org/cgit/openstack-infra/puppet-gerritbot/tree/
* https://git.openstack.org/cgit/openstack-infra/puppet-ptgbot/tree/
* https://opendev.org/opendev/puppet-meetbot
* https://opendev.org/opendev/puppet-statusbot
* https://opendev.org/opendev/puppet-gerritbot
* https://opendev.org/opendev/puppet-ptgbot
* :cgit_file:`modules/openstack_project/manifests/eavesdrop.pp`
* :cgit_file:`modules/openstack_project/manifests/review.pp`
:Configuration:
@ -29,10 +29,10 @@ At a Glance
:Projects:
* http://wiki.debian.org/MeetBot
* http://sourceforge.net/projects/supybot/
* https://git.openstack.org/cgit/openstack-infra/meetbot
* https://git.openstack.org/cgit/openstack-infra/gerritbot
* https://git.openstack.org/cgit/openstack-infra/statusbot
* https://git.openstack.org/cgit/openstack/ptgbot
* https://opendev.org/opendev/meetbot
* https://opendev.org/opendev/gerritbot
* https://opendev.org/opendev/statusbot
* https://opendev.org/openstack/ptgbot
:Bugs:
* https://storyboard.openstack.org/#!/project/748
@ -93,7 +93,7 @@ Meetbot
-------
The OpenStack Infrastructure Meetbot fork can be found at
https://git.openstack.org/cgit/openstack-infra/meetbot. Manual installation of the Meetbot
https://opendev.org/opendev/meetbot. Manual installation of the Meetbot
plugin is straightforward and documented in that repository's README.
OpenStack Infrastructure installs and configures Meetbot through Puppet.
@ -259,16 +259,16 @@ PTG Bot
Bot that `Project Teams Gathering <https://www.openstack.org/ptg>`_
room moderators use to surface what's currently happening at the
event. Usage instructions are provided in its `README.rst file
<https://git.openstack.org/cgit/openstack/ptgbot/tree/README.rst>`_.
<https://opendev.org/openstack/ptgbot/src/branch/master/README.rst>`_.
It writes some static content into ``/var/lib/ptgbot/www`` on the
eavesdrop.openstack.org server and then serves that from a
http://ptg.openstack.org/ Apache vhost.
Code for the PTG bot lives in the openstack-infra/ptgbot respository
(https://git.openstack.org/cgit/openstack-infra/ptgbot), while the
Code for the PTG bot lives in the openstack/ptgbot respository
(https://opendev.org/openstack/ptgbot), while the
puppet module used to deploy it (including the template used for its
configuration) lives in the openstack-infra/puppet-ptgbot repository
(https://git.openstack.org/cgit/openstack-infra/puppet-ptgbot).
configuration) lives in the opendev/puppet-ptgbot repository
(https://opendev.org/opendev/puppet-ptgbot).
Basic Channel Operator Commands
===============================

View File

@ -13,7 +13,7 @@ At a Glance
===========
:Hosts:
* http://review.openstack.org
* http://review.opendev.org
* http://review-dev.openstack.org
:Puppet:
* https://git.openstack.org/cgit/openstack-infra/puppet-jeepyb/tree/

View File

@ -1,235 +0,0 @@
:title: Jenkins Job Builder
.. _jjb:
Jenkins Job Builder
###################
Jenkins Job Builder is a system for configuring Jenkins jobs using
simple YAML files stored in Git.
At a Glance
===========
:Hosts:
* http://zm*.openstack.org
:Configuration:
* :config:`jenkins/jobs/`
:Projects:
* https://git.openstack.org/cgit/openstack-infra/jenkins-job-builder
:Bugs:
* https://storyboard.openstack.org/#!/project/723
:Resources:
* `Reference Manual <http://docs.openstack.org/infra/jenkins-job-builder>`_
Overview
========
In order to make the process of managing thousands of Jenkins jobs
easier, Jenkins Job Builder was designed to take YAML based
configurations and convert those into jobs that are injected into
Jenkins.
The documentation below describes how the OpenStack Infrastructure
team uses the Jenkins Job Builder in our environment.
Configuring Projects
====================
The YAML scripts to make this work are stored in the
:config:`jenkins/jobs/` directory of the project-config repository.
In this directory you can have four different types of yaml config
files:
* Jenkins Jobs Defaults in ``defaults.yaml``.
* Jenkins Jobs Macros to give larger config sections meaningful names in
``macros.yaml``.
* Project specific configurations in ``project_name.yaml``.
* Job template configurations. Need a ``projects.yaml`` file to
specify how the templates should be filled out and templates go in
``template_name.yaml``.
YAML Format
===========
Defaults
--------
Example defaults config:
.. code-block:: yaml
- defaults:
name: global
project-type: freestyle
concurrent: true
wrappers:
- timeout:
timeout: 30
fail: true
- timestamps
logrotate:
daysToKeep: 1
numToKeep: -1
artifactDaysToKeep: -1
artifactNumToKeep: -1
This config starts with the ``- defaults::`` line. This specifies that this
section contains default values rather than job specifications. In this
section we specify a useful set of defaults including a default description
indicating Puppet manages these jobs, jobs are allowed to run concurrently,
and a thirty minute job timeout.
Macros
------
Macros exist to give meaningful names to blocks of configuration that can be
used in job configs in place of the blocks they name. For example:
.. code-block:: yaml
- builder:
name: git-prep
builders:
- shell: "/slave_scripts/git-prep.sh"
- builder:
name: docs
builders:
- shell: "/slave_scripts/run-docs.sh"
- publisher:
name: console-log
publishers:
- scp:
site: 'scp-server'
files:
- target: 'logs/$JOB_NAME/$BUILD_NUMBER'
copy-console: true
copy-after-failure: true
In this block of code we define two builder macros and one publisher macro.
Each macro has a name and using that name in a job config is equivalent to
having the yaml below the name in place of the name in the job config. The next
section shows how you can use these macros.
Job Config
----------
Example job config:
.. code-block:: yaml
- job:
name: example-docs
node: node-label
triggers:
- zuul
builders:
- git-prep
- docs
publishers:
- scp:
site: 'scp-server'
files:
- target: 'dir/ectory'
source: 'build/html/foo'
keep-hierarchy: true
- console-log
Each job specification begins with ``-job:``. Under this section you can
specify the job details like name, node, etc. Any detail defined in the
defaults section that is not defined under this job will be included as well.
In addition to attribute details you can also specify how jenkins should
perform this job. What trigger methods should be used, the build steps,
jenkins publishing steps and so on. The macros defined earlier make this easy
and simple.
Job Templates
-------------
Job templates allow you to specify a job config once with arguments that are
replaced with the values specified in ``projects.yaml``. This allows you to
reuse job configs across many projects. First you need a templated job config:
.. code-block:: yaml
- job-template:
name: '{name}-docs'
triggers:
- zuul
builders:
- git-prep
- docs
publishers:
- scp:
site: 'scp-server'
files:
- target: 'dir/ectory'
source: 'build/html/foo'
keep-hierarchy: true
- console-log
node: '{node}'
- job-group:
name: python-jobs
jobs:
- '{name}-docs'
This takes the previous ``example-docs`` job and templatizes it. This will
allow us to easily create ``example1-docs`` and ``example2-docs`` jobs.
Each job template begins with ``- job-template:`` and the job specification is
identical to the previous one, but we have introduced variable arguments. In
this case ``{name}`` is a variable value that will be replaced. The values for
name will be defined in the ``projects.yaml`` file.
The ``- job-group:`` section is not strictly necessary but allows you to group
many job templates with the same variable arguments under one name.
The ``projects.yaml`` pulls all of the magic together. It specifies the
arguments to and instantiates the job templates as real jobs. For example:
.. code-block:: yaml
- project:
name: example1
node: bare-trusty
jobs:
- python-jobs
- project:
name: example2
node: bare-centos6
jobs:
- {name}-docs
Each project using templated jobs should have its own ``- project:`` section.
Under this sections there should be a ``jobs:`` section with a list of job
templates or job groups to be used by this project. Other values under the
``- project:`` section define the arguments to the templates listed under
``jobs:``. In this case we are giving the docs template ``name`` and ``node``
values.
Notice that example1 makes use of the job group and example2 makes use of the
job template.
Zuul
====
In our environment, we no longer use Jenkins to execute jobs. Zuul
itself, via Ansible, runs the actual workload. Zuul reads JJB config
files in order to define its jobs, so, aside from the detail of not
actually using Jenkins or creating any jobs in it, the use of JJB to
configure jobs in Zuul is the same.

View File

@ -15,7 +15,7 @@ At a Glance
* logstash-worker\*.openstack.org
* elasticsearch\*.openstack.org
:Puppet:
* https://git.openstack.org/cgit/openstack-infra/puppet-logstash/tree/
* https://opendev.org/opendev/puppet-logstash
* :cgit_file:`modules/openstack_project/manifests/logstash.pp`
* :cgit_file:`modules/openstack_project/manifests/logstash_worker.pp`
* :cgit_file:`modules/openstack_project/manifests/elasticsearch.pp`
@ -110,8 +110,8 @@ the Gearman server.
If you are interested in technical details the source of these scripts
can be found at
* https://git.openstack.org/cgit/openstack-infra/puppet-log_processor/tree/files/log-gearman-client.py
* https://git.openstack.org/cgit/openstack-infra/puppet-log_processor/tree/files/log-gearman-worker.py
* https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-client.py
* https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-worker.py
Subunit Gearman Worker
----------------------
@ -124,8 +124,7 @@ from gate and periodic queue tempest runs.
If you are interested in technical details the source of this script can be
found at:
* https://git.openstack.org/cgit/openstack-infra/puppet-subunit2sql/tree/files/subunit-gearman-worker.py
* https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/subunit-gearman-worker.py
Logstash
--------
@ -172,23 +171,23 @@ schema.
The config file that tells Logstash how to do this flattening can be
found at
https://git.openstack.org/cgit/openstack-infra/logstash-filters/tree/filters/openstack-filters.conf
https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf
This works via the tags that are associated with a given message.
The tags in
https://git.openstack.org/cgit/openstack-infra/logstash-filters/tree/filters/openstack-filters.conf
https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf
are used to tell logstash how to parse a given file's messages, based
on the file's message format.
When adding a new file to be indexed to
http://git.openstack.org/cgit/openstack-infra/project-config/tree/roles/submit-logstash-jobs/defaults/main.yaml
https://opendev.org/opendev/base-jobs/src/branch/master/roles/submit-logstash-jobs/defaults/main.yaml
at least one tag from the openstack-filters.conf file should be associated
with the new file. One can expect to see '{%logmessage%}' instead of
actual message data if indexing is not working properly.
In the event a new file's format is not covered, a patch for
https://git.openstack.org/cgit/openstack-infra/logstash-filters/tree/filters/openstack-filters.conf
https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf
should be submitted with an appropriate parsing pattern.
ElasticSearch
@ -321,8 +320,8 @@ Used in a kibana query, it would be structured like this:
This is still an early effort and additional tuning and refinement should
be expected. Should the crm114 settings need to be tuned or expanded,
a patch may be submitted for this file, which controls the process:
https://git.openstack.org/cgit/openstack-infra/puppet-log_processor/tree/files/classify-log.crm
https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/classify-log.crm
.. _logs post-playbook: http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/base/post-logs.yaml
.. _submit-logstash-jobs defaults: http://git.openstack.org/cgit/openstack-infra/project-config/tree/roles/submit-logstash-jobs/defaults/main.yaml
.. _submit-log-processor-jobs role: http://git.openstack.org/cgit/openstack-infra/project-config/tree/roles/submit-log-processor-jobs
.. _logs post-playbook: https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/post-logs.yaml
.. _submit-logstash-jobs defaults: https://opendev.org/opendev/base-jobs/src/branch/master/roles/submit-logstash-jobs/defaults/main.yaml
.. _submit-log-processor-jobs role: https://opendev.org/opendev/base-jobs/src/branch/master/roles/submit-log-processor-jobs

View File

@ -86,7 +86,7 @@ For adding a new node to your puppet master, you can either use the
(see :cgit_file:`launch/README` for full details) or bootstrap puppet manually.
For manual bootstrap, you need to run on the new server connecting
(for example, review.openstack.org) to the puppet master:
(for example, review.opendev.org) to the puppet master:
.. code-block:: bash

View File

@ -14,7 +14,6 @@ Major Systems
grafyaml
zuul
zuulv3
jjb
logstash
elastic-recheck
devstack-gate
@ -29,7 +28,6 @@ Major Systems
reprepro
lists
wiki
git
openstackid
storyboard
kerberos

View File

@ -79,7 +79,7 @@ Requirements
Reading the Event Stream
------------------------
It is possible to use ssh to connect to ``review.openstack.org`` on port 29418
It is possible to use ssh to connect to ``review.opendev.org`` on port 29418
with your ssh key if you have a normal reviewer account in Gerrit.
This will give you a real-time JSON stream of events happening inside Gerrit.
@ -87,7 +87,7 @@ For example:
.. code-block:: bash
$ ssh -p 29418 USERNAME@review.openstack.org gerrit stream-events
$ ssh -p 29418 USERNAME@review.opendev.org gerrit stream-events
Will give a stream with an output like this (line breaks and
indentation added in this document for readability, the real JSON will
@ -97,7 +97,7 @@ be all one line per event):
{"type":"comment-added","change":
{"project":"openstack/keystone","branch":"stable/essex","topic":"bug/969088","id":"I18ae38af62b4c2b2423e20e436611fc30f844ae1","number":"7385","subject":"Make import_nova_auth only create roles which don\u0027t already exist","owner":
{"name":"Chuck Short","email":"chuck.short@canonical.com","username":"zulcss"},"url":"https://review.openstack.org/7385"},
{"name":"Chuck Short","email":"chuck.short@canonical.com","username":"zulcss"},"url":"https://review.opendev.org/7385"},
"patchSet":
{"number":"1","revision":"aff45d69a73033241531f5e3542a8d1782ddd859","ref":"refs/changes/85/7385/1","uploader":
{"name":"Chuck Short","email":"chuck.short@canonical.com","username":"zulcss"},
@ -111,7 +111,7 @@ be all one line per event):
For most purposes you will want to trigger on ``patchset-created`` for when a
new patchset has been uploaded.
Further documentation on how to use the events stream can be found in `Gerrit's stream event documentation page <https://review.openstack.org/Documentation/cmd-stream-events.html>`_.
Further documentation on how to use the events stream can be found in `Gerrit's stream event documentation page <https://review.opendev.org/Documentation/cmd-stream-events.html>`_.
Posting Result To Gerrit
------------------------
@ -127,13 +127,13 @@ An example of how to post this is as follows:
.. code-block:: bash
$ ssh -p 29418 USERNAME@review.openstack.org gerrit review -m '"Test failed on MegaTestSystem <http://megatestsystem.org/tests/1234>"' --verified=-1 c0ff33
$ ssh -p 29418 USERNAME@review.opendev.org gerrit review -m '"Test failed on MegaTestSystem <http://megatestsystem.org/tests/1234>"' --verified=-1 c0ff33
In this example ``c0ff33`` is the commit ID for the review. You can
set the verified to either `-1` or `+1` depending on whether or not it
passed the tests.
Further documentation on the `review` command in Gerrit can be found in the `Gerrit review documentation page <https://review.openstack.org/Documentation/cmd-review.html>`_.
Further documentation on the `review` command in Gerrit can be found in the `Gerrit review documentation page <https://review.opendev.org/Documentation/cmd-review.html>`_.
We do suggest cautious testing of these systems and have a development Gerrit
setup to test on if required. In SmokeStack's case all failures are manually
@ -166,11 +166,11 @@ Once you have created this account with the OpenID provider you can log
into Gerrit with that new account as you would with your normal user
account. Once logged in you will need to do several things:
1. Set an SSH username at https://review.openstack.org/#/settings/ if
1. Set an SSH username at https://review.opendev.org/#/settings/ if
it isn't already set. This is the username your CI system will use to
SSH to Gerrit in order to read the event stream.
2. Set the account's fullname at https://review.openstack.org/#/settings/contact
2. Set the account's fullname at https://review.opendev.org/#/settings/contact
This name should follow a few rules in order to make it clear in Gerrit
comments what this CI system exists to test. The name should have three
pieces ``Organization`` ``Product/technology`` ``CI designator``. The
@ -189,7 +189,7 @@ account. Once logged in you will need to do several things:
something like ``IBM DB2 CI``.
3. Add the SSH public key you will be using to the Gerrit account at
https://review.openstack.org/#/settings/ssh-keys You can generate an
https://review.opendev.org/#/settings/ssh-keys You can generate an
ssh key using ``ssh-keygen``. You want to give Gerrit the contents of
the generated id_rsa.pub file.
@ -204,9 +204,9 @@ nodepool), you should subscribe to the
list to keep on top of announcements there.
It would also be a good idea to contact the `Third Party Coordinators
<https://review.openstack.org/#/admin/groups/440>`_ asking to add your account
<https://review.opendev.org/#/admin/groups/440>`_ asking to add your account
to the `Third Party CI mail filter list
<https://review.openstack.org/#/admin/groups/270>`_. This is necessary to keep
<https://review.opendev.org/#/admin/groups/270>`_. This is necessary to keep
Gerrit from sending email messages every time an account comments on a patch.
Once you have done this you will have everything you need to comment on
@ -233,8 +233,8 @@ Jenkins Plugin Manager.
Once installed Jenkins will have a new `Gerrit Trigger` option in the `Manage
Jenkins` menu. This should be given the following options::
Hostname: review.openstack.org
Frontend URL: https://review.openstack.org/
Hostname: review.opendev.org
Frontend URL: https://review.opendev.org/
SSH Port: 29418
Username: (the Gerrit user)
SSH Key File: (path to the user SSH key)

View File

@ -74,7 +74,7 @@ talks through git and ssh so you will need to manually check ssh host
keys as the zuul user. e.g.::
sudo su - zuul
ssh -p 29418 review.openstack.org
ssh -p 29418 review.opendev.org
To debug Zuul's gearman server, SSL is required. Use the following
command::

View File

@ -124,7 +124,7 @@ host keys as the zuul user.
e.g.::
sudo su - zuul
ssh -p 29418 review.openstack.org
ssh -p 29418 review.opendev.org
The Zuul Scheduler talks to Nodepool using Zookeeper and distributes work to
the executors using gear.