5a8bd7e162
Using wget can lead to problems because it's not default. We should use curl instead Change-Id: Idc859c57582ed3b83c6177e91e14ba4828311826 Closes-Bug: #1847344
327 lines
12 KiB
Django/Jinja
327 lines
12 KiB
Django/Jinja
<!DOCTYPE HTML>
|
|
<html lang="en-US">
|
|
<head>
|
|
<title>WARNING - WIP - NOT FOR GENERAL USE - README for Quickstart Job Zuul-based Reproducer Script</title>
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
<style>
|
|
.collapsible {
|
|
background-color: #777;
|
|
color: white;
|
|
cursor: pointer;
|
|
padding: 18px;
|
|
width: 100%;
|
|
border: none;
|
|
text-align: left;
|
|
outline: none;
|
|
font-size: 15px;
|
|
}
|
|
|
|
.active, .collapsible:hover {
|
|
background-color: #555;
|
|
}
|
|
|
|
.content {
|
|
padding: 0 18px;
|
|
max-height: 0;
|
|
overflow: hidden;
|
|
transition: max-height 0.2s ease-out;
|
|
background-color: #f1f1f1;
|
|
}
|
|
</style>
|
|
</head>
|
|
</head>
|
|
<body>
|
|
<h1> WARNING - WIP - NOT FOR GENERAL USE - How to reproduce a job result using the Zuul-based reproducer playbooks and script</h1>
|
|
<p>Check the top-level logs directory for a tar file <b>reproducer-zuul-based-quickstart.tar</b>.
|
|
This tar archive contains three files:
|
|
<ul>
|
|
<li><code>launcher-env-setup-playbook.yaml</code> - a playbook to clone repos and set up
|
|
an Ansible environment to launch jobs</li>
|
|
<li><code>launcher-playbook.yaml - a playbook that installs and starts up Zuul, Gerrit and
|
|
other required services, and also creates and launches a reproducer job using
|
|
the same dlrn hashes as in CI</code></li>
|
|
<li><code>reproducer-zuul-based-quickstart.sh - a Bash script to provide an interface
|
|
to run the above playbooks with various options</code></li>
|
|
</ul>
|
|
Running these playbooks will set up Zuul, Gerrit, and related services in containers
|
|
on the local system, and will then create a reproducer job which can be run on a
|
|
cloud personal tenant or via libvirt on the local system.
|
|
If no <b>reproducer-zuul-based-quickstart.tar</b> file is found, that usually means an
|
|
infra failure before Quickstart could start or a problem in collecting logs.
|
|
Check on IRC Freenode channel <i>#tripleo</i> to see if there's an ongoing infra
|
|
issue.</p>
|
|
|
|
<button class="collapsible">Setting up to run the Zuul-based reproducer</button>
|
|
<div class="content">
|
|
<p>
|
|
To setup and run the reproducer you must first decide <b>where</b> it will run
|
|
- on a <b>pet</b> (your laptop) or on <b>cattle</b> (rdo-cloud vm or other
|
|
temporary environment)? For each case see notes below for required setup.
|
|
|
|
<p>
|
|
<b>If this is the first time you are running the reproducer</b> you must
|
|
ensure you have an externally routable network in your RDO-cloud tenant called
|
|
'private'. That is you will need a router attached to the external network that
|
|
is also attached to your network called 'private' (which you may need to create).
|
|
Assuming you have a clouds.yaml that let's you talk to RDO cloud you can use
|
|
the following commands to create the required setup:
|
|
<pre>
|
|
<code>
|
|
openstack --os-cloud rdo-cloud network create private
|
|
openstack --os-cloud rdo-cloud subnet create private --network private --gateway 192.168.0.1 --subnet-range 192.168.0.0/24
|
|
openstack --os-cloud rdo-cloud router create privaterouter
|
|
openstack --os-cloud rdo-cloud router set --external-gateway 38.145.32.0/22 privaterouter
|
|
openstack --os-cloud rdo-cloud router add subnet privaterouter private
|
|
</code>
|
|
</pre>
|
|
</p>
|
|
|
|
Alternatively log in to <a href="https://phx2.cloud.rdoproject.org/dashboard/auth/login/?next=/dashboard/project/networks/">RDO cloud horizon</a> and create the network and router
|
|
that way.
|
|
|
|
<hr></hr>
|
|
<h1>Cattle</h1>
|
|
If you're using a temporary environment like RDO-cloud vm to run the
|
|
reproducer you can use the script inside the reproducer-quickstart/ directory
|
|
from the logs of the job you are reproducing. The script is acalled
|
|
<code>reproducer-fresh-box-setup.sh</code>:
|
|
|
|
<pre><code><b>
|
|
curl -o reproducer-fresh-box-setup.sh http://logs.openstack.org/79/657779/2/check/tripleo-ci-centos-7-containers-multinode/67d8eb6/logs/reproducer-quickstart/reproducer-fresh-box-setup.sh
|
|
chmod 775 reproducer-fresh-box-setup.sh
|
|
./reproducer-fresh-box-setup.sh -u marios -p password -c reprozuul
|
|
</b></code></pre>
|
|
|
|
<p><b>note:</b> If you're using an RDO-cloud vm be sure to use at least m1.large2
|
|
flavour as we've hit hard disk limits with m1.large</p>
|
|
<p> There are two required parameters <code>-u USER</code/> and <code>-p PASS</code>
|
|
which correspond to the RDO cloud username and password. These are written to
|
|
the generated <code>$HOME/.config/openstack/clouds.yaml</code> file expected by the
|
|
reproducer.
|
|
</p>
|
|
<p>This script creates the default user 'reprozuul' with passwordless sudo
|
|
privileges, creates $HOME/.config/openstack/clouds.yaml with the -u USER and
|
|
-p PASS parameters and creates the required ssh keypair. It must run as root,
|
|
or as a user with passwordless sudo.
|
|
</p>
|
|
<p> The script finally switches to the newly created user and outputs the newly
|
|
created public key. You must upload this key to gerrit (<b>both</b>
|
|
<a href="https://review.opendev.org/#/settings/ssh-keys">opendev.org</a> as well
|
|
as <a href="https://review.rdoproject.org/r/#/settings/ssh-keys">rdoproject.org</a>).
|
|
</p>
|
|
<hr></hr>
|
|
<h1>Pet</h1>
|
|
|
|
If you are running the reproducer on a non-transient machine, that is your
|
|
laptop or any other box which isn't temporary (a pet), then you need:
|
|
|
|
<ul>
|
|
<li> Ensure user running reproducer has passwordless sudo </li>
|
|
<li> That user has $HOME/.config/openstack/clouds.yaml with "rdo-cloud" entry </li>
|
|
<li> A ssh keypair generated with <b><code>-m PEM -t rsa</code></b> and uploaded
|
|
to gerrit (<b>both </b><a href="https://review.opendev.org/#/settings/ssh-keys">opendev.org</a>
|
|
as well as <a href="https://review.rdoproject.org/r/#/settings/ssh-keys">rdoproject.org</a>).
|
|
</li>
|
|
<li> A externally routable network in your RDO-cloud tenant named 'private'.
|
|
See example commands above if you need to create this or log in to
|
|
<a href="https://phx2.cloud.rdoproject.org/dashboard/auth/login/?next=/dashboard/project/networks/"> RDO cloud horizon</a> and verify your setup or create as needed.
|
|
</ul>
|
|
|
|
You only need to do these things once on your pet and then rerun reproducer or
|
|
update as needed.
|
|
|
|
<ul>
|
|
<li>Example clouds.yaml:</li>
|
|
<pre>
|
|
<code>CLOUDS_YAML_PATH=$HOME/.config/openstack/
|
|
mkdir -p $CLOUDS_YAML_PATH
|
|
/bin/bash -c "cat <<EOF>>$CLOUDS_YAML_PATH/clouds.yaml
|
|
clouds:
|
|
rdo-cloud:
|
|
identity_api_version: 3
|
|
region_name: regionOne
|
|
auth:
|
|
auth_url: https://phx2.cloud.rdoproject.org:13000/v3
|
|
password: $RDO_PASS
|
|
project_name: $RDO_USER
|
|
username: $RDO_USER
|
|
user_domain_name: Default
|
|
project_domain_name: Default
|
|
regions:
|
|
- name: regionOne
|
|
values:
|
|
networks:
|
|
- name: 38.145.32.0/22
|
|
routes_externally: true
|
|
- name: private
|
|
EOF"
|
|
</code>
|
|
</pre>
|
|
</ul>
|
|
<ul>
|
|
<br/>
|
|
<li> Create an ssh key pair without a pass phrase:</li>
|
|
<ul>
|
|
<li>The ssh key pair is configurable - see variables <code>ssh_path</code>
|
|
<code>user_pri_key</code>. The default key is:<code>~/.ssh/id_rsa[.pub]</code></li>
|
|
<li> <b>Note:</b> Use the following to create keys: <b><code> ssh-keygen -m PEM -t rsa </code></b>
|
|
See FAQ below for more information about ssh keys requirements.
|
|
</ul>
|
|
</ul>
|
|
</br>
|
|
</div>
|
|
|
|
<button class="collapsible">Running the Zuul-based reproducer - using the Bash script</button>
|
|
<div class="content">
|
|
<p>
|
|
wget or curl the <b>reproducer-quickstart/reproducer-zuul-based-quickstart.tar</b> file.
|
|
<pre><code><b>
|
|
curl -Lo reproducer-zuul-based-quickstart.tar http://logs.openstack.org/79/657779/2/check/tripleo-ci-centos-7-standalone-upgrade/92fd476/logs/reproducer-quickstart/reproducer-zuul-based-quickstart.tar
|
|
tar -xvf reproducer-zuul-based-quickstart.tar
|
|
</b></code></pre>
|
|
One of the extracted files is the script called <code>./reproducer-zuul-based-quickstart.sh</code>
|
|
and you can run it with <code>--help</code> to see available options.
|
|
</br>
|
|
</br>
|
|
<b>Note:</b> If the local $USERNAME does not match the gerrit user IDs for
|
|
<a href="https://review.opendev.org/#/settings/ssh-keys">opendev.org</a> or
|
|
<a href="https://review.rdoproject.org/r/#/settings/ssh-keys">rdoproject.org</a>)
|
|
then you will need to supply the -ug (upstream gerrit) and -rg (rdo gerrit)
|
|
parameters:
|
|
|
|
<ul>
|
|
</ul>
|
|
<pre><code><b>
|
|
mkdir WORKSPACE
|
|
./reproducer-zuul-based-quickstart.sh --workspace WORKSPACE -ug slim_shady -rg slim_shadier
|
|
</b></code></pre>
|
|
</p>
|
|
Monitor the output and follow any instructions - for example the script will
|
|
exit and require re-run once the user is added to the docker group.
|
|
</br>
|
|
</br>
|
|
</div>
|
|
|
|
<button class="collapsible">Running the Zuul-based reproducer - using the playbooks directly</button>
|
|
<div class="content">
|
|
<p>
|
|
The Bash script is not required although it is the supported path.
|
|
The playbooks can be run directly using <code>ansible-playbook</code>.
|
|
The Bash script contains some package installation and Docker user/group setup
|
|
that you will need to cover prior to running the playbooks directly.
|
|
Please see the Bash scripts for details, especially if you are running the
|
|
reproducer for the first time on the local environment.
|
|
The playbooks can be run as follows:
|
|
<ul>
|
|
<li> Note that the <code>launcher-env-setup-playbook.yaml</code> will clone
|
|
repos and set up 'roles', 'playbooks', and 'library' directories
|
|
from which the tasks will be executed. </li>
|
|
<li><code>ansible-playbook ./launcher-env-setup-playbook.yaml</code></li>
|
|
<li><code>ansible-playbook ./launcher-playbook.yaml $OPTIONS</code></li>
|
|
</ul>
|
|
The <code>launcher-playbook.yaml</code> includes comments on how to run the playbook
|
|
with the same options listed in the Bash script section above.
|
|
</p>
|
|
</div>
|
|
|
|
<button class="collapsible">Running the Zuul-based reproducer - autohold nodes for debugging</button>
|
|
<div class="content">
|
|
<p>
|
|
The reproducer can be launched such that if there is an error the environment will
|
|
remain up and running available to the developer to debug.
|
|
|
|
<ul>
|
|
<li>For example:</li>
|
|
|
|
</ul>
|
|
<li><code> cd ~/tripleo-ci-reproducer </code></li>
|
|
<li><code>docker-compose exec scheduler zuul autohold --tenant tripleo-ci-reproducer --job tripleo-ci-centos-7-standalone --reason debug --project test1</code></li>
|
|
</p>
|
|
</div>
|
|
|
|
<button class="collapsible">Ansible tags and running repeatedly</button>
|
|
<div class="content">
|
|
<p>
|
|
|
|
<ul>
|
|
<li> The Bash scripts runs with <code>--tags all</code> passed to the
|
|
<code>launcher-playbook.yaml</code>.
|
|
</li>
|
|
<li>
|
|
After the initial docker setup has run, it is not necessary to run through
|
|
those setup steps for every reproducer. The <code>launcher-playbook.yaml</code>
|
|
can be run with <code>--skip-tags start</code> after the initial run.
|
|
</li>
|
|
</p>
|
|
</div>
|
|
|
|
<button class="collapsible">Monitoring the launched job</button>
|
|
<div class="content">
|
|
<p>
|
|
The <code>launcher-playbook.yaml</code> will create and push a job
|
|
to a test1 testproject repo in a temp directory. This job will start and the
|
|
related Zuul infrastructure logs can be viewed in:</br>
|
|
<pre>
|
|
<code>cd ~/tripleo-ci-reproducer
|
|
docker-compose logs --tail=10 -f
|
|
# Just view the scheduler and executor logs:
|
|
docker-compose logs --tail=10 -f scheduler executor</code>
|
|
</pre>
|
|
</br>
|
|
To see the job progress, you can access the web consoles of:
|
|
<ul>
|
|
<li>Gerrit: http://localhost:8080</li>
|
|
<li>Zuul: http://localhost:9000</li>
|
|
<li>Logs: http://localhost:8000</li>
|
|
</ul>
|
|
If you are accessing a remote machine, replace <code>localhost</code> with the ip address.
|
|
</p>
|
|
</div>
|
|
|
|
<button class="collapsible">Known Issues and FAQ</button>
|
|
<div class="content">
|
|
<p>
|
|
<ul>
|
|
<li>Gerrit keys<br/>
|
|
If your ssh keys have being created with a new version of OpenSSH, Zuul will
|
|
fail when running with these keys as there is a Paramiko bug [1].
|
|
It is recommended that keys be generated as follows:
|
|
<code>ssh-keygen -m PEM -t rsa</code> - accepting the passwordless option.
|
|
Then update the public keys used with Gerrit upstream and RDO Gerrit.</br>
|
|
[1] https://storyboard.openstack.org/#!/story/2004842</li>
|
|
</ul>
|
|
<ul>
|
|
<li>If you currently do not have access to upstream-cloudinit-[centos-7, fedora-28] : </li>
|
|
<code>
|
|
<pre>
|
|
source the openrc from the openstack-nodepool tenant
|
|
openstack image list | grep upstream-cloudinit
|
|
glance member-create ${IMAGE_ID} ${YOUR_TENANT_ID}
|
|
glance member-create ${IMAGE_ID} ${YOUR_TENANT_ID}
|
|
</code>
|
|
</pre>
|
|
</ul>
|
|
</p>
|
|
</p>
|
|
</div>
|
|
|
|
<script>
|
|
var coll = document.getElementsByClassName("collapsible");
|
|
var i;
|
|
|
|
for (i = 0; i < coll.length; i++) {
|
|
coll[i].addEventListener("click", function() {
|
|
this.classList.toggle("active");
|
|
var content = this.nextElementSibling;
|
|
if (content.style.maxHeight){
|
|
content.style.maxHeight = null;
|
|
} else {
|
|
content.style.maxHeight = content.scrollHeight + "px";
|
|
}
|
|
});
|
|
}
|
|
</script>
|
|
|
|
</body>
|
|
</html>
|