Add reproducer script for OVB and multinode jobs
This review adds functionality to create a reproducer script in the logs. The reproducer script will allow users to recreate failing OVB and multinode jobs in personal cloud tenants. User documentation for the reproducer-quickstart script is added. Change-Id: I9fe8550a75c3ffb6d1271b01b1144bfbdc82c95d
This commit is contained in:
Ronelle Landy
Sagi Shnaidman
parent
7bc877db04
commit
ed220f5b98
@ -50,7 +50,7 @@ artcl_exclude_list:
|
|||||||
- /etc/pki/ca-trust/extracted
|
- /etc/pki/ca-trust/extracted
|
||||||
- /etc/alternatives
|
- /etc/alternatives
|
||||||
- /var/log/journal
|
- /var/log/journal
|
||||||
artcl_collect_dir: "{{ local_working_dir }}/collected_files"
|
# artcl_collect_dir is defaulted in extras-common
|
||||||
artcl_gzip_only: true
|
artcl_gzip_only: true
|
||||||
artcl_tar_gz: false
|
artcl_tar_gz: false
|
||||||
|
|
||||||
|
|||||||
@ -47,6 +47,11 @@
|
|||||||
/(\/var\/log\/|\/etc\/)[^ \/\.]+\.gz$/ { rename($0) }';
|
/(\/var\/log\/|\/etc\/)[^ \/\.]+\.gz$/ { rename($0) }';
|
||||||
when: artcl_txt_rename|bool
|
when: artcl_txt_rename|bool
|
||||||
|
|
||||||
|
- name: Create the reproducer script
|
||||||
|
include_role:
|
||||||
|
name: create-reproducer-script
|
||||||
|
when: lookup('env', 'TOCI_JOBTYPE') != ''
|
||||||
|
|
||||||
- name: upload to the artifact server using pubkey auth
|
- name: upload to the artifact server using pubkey auth
|
||||||
shell: rsync -av --quiet -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" {{ artcl_collect_dir }}/ {{ artcl_rsync_path }}/{{ lookup('env', 'BUILD_TAG') }}
|
shell: rsync -av --quiet -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" {{ artcl_collect_dir }}/ {{ artcl_rsync_path }}/{{ lookup('env', 'BUILD_TAG') }}
|
||||||
async: "{{ artcl_publish_timeout }}"
|
async: "{{ artcl_publish_timeout }}"
|
||||||
|
|||||||
@ -0,0 +1,82 @@
|
|||||||
|
<!DOCTYPE HTML>
|
||||||
|
<html lang="en-US">
|
||||||
|
<head>
|
||||||
|
<title>README for Quickstart Job Reproducer Script</title>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<h1>How to reproduce a job result using the reproducer-quickstart.sh script</h1>
|
||||||
|
<p>Check the top-level logs directory for a file <b>reproducer-quickstart.sh</b>.
|
||||||
|
Running this file will set up a stack on a personal tenant where the job run
|
||||||
|
can be reproduced.
|
||||||
|
If no <b>reproducer-quickstart.sh</b> file is found, that usually means an
|
||||||
|
infra failure before Quickstart could start or a problem in collecting logs.
|
||||||
|
Check on IRC Freenode channel <i>#tripleo</i> to see if there's an ongoing infra
|
||||||
|
issue.</p>
|
||||||
|
|
||||||
|
<h2>What is the reproducer-quickstart.sh script?</h2>
|
||||||
|
<p>A <b>reproducer-quickstart.sh</b> script is generated at the top-level log directory
|
||||||
|
for jobs that run TripleO-Quickstart and toci_gate_test-oooq.sh. Running the script
|
||||||
|
from a host machine will set up a stack to the point where the user can ssh to the
|
||||||
|
undercloud, source a variable file, and run <b>toci_gate_test-oooq.sh</b>. As such,
|
||||||
|
reproducing the complete jobs result requires some manual interaction.</p>
|
||||||
|
|
||||||
|
<p>The <b>reproducer-quickstart.sh</b> script is generated to be specific to that
|
||||||
|
job and <b>reproducer-quickstart.sh</b> scripts from other jobs will be different.</p>
|
||||||
|
|
||||||
|
<h2>Running the reproducer-quickstart.sh script</h2>
|
||||||
|
<p>Use <code>cURL</code>, <code>wget</code> or a similar tool to copy the
|
||||||
|
<b>reproducer-quickstart.sh</b> to a clean working directory on a host from
|
||||||
|
where you can access the cloud tenant you want to run the reproducer on.
|
||||||
|
Make sure that the copied script is executable (has +x permissions) before you
|
||||||
|
attempt to run it.
|
||||||
|
|
||||||
|
<b>Source the tenant credentials before running the script.</b>
|
||||||
|
|
||||||
|
Also, it is a good idea to check that the tenant has enough capacity
|
||||||
|
and resources to reproduce the job before you begin.</p>
|
||||||
|
|
||||||
|
<p>Note that, by default, the <b>reproducer-quickstart.sh</b>
|
||||||
|
script is written to run an a personal RDO Cloud tenant. Instructions on how to
|
||||||
|
modify the script to run on another host cloud are included in a section below.</p>
|
||||||
|
|
||||||
|
<p>When the script completes, you will see a list of instructions printed out on how
|
||||||
|
to run <b>toci_gate_test-oooq.sh</b> from the undercloud:</p>
|
||||||
|
|
||||||
|
<ul>
|
||||||
|
<li><code>ssh to the undercloud: $ ssh zuul@$ansible_host</code></li>
|
||||||
|
<li><code>Source the environment settings file: $ source /home/zuul/env_vars_to_src.sh</code></li>
|
||||||
|
<li><code>Run the toci gate script: $ /opt/stack/tripleo-ci/toci_gate_test-oooq.sh</code></li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h2>Input argument options in the reproducer-quickstart.sh script</h2>
|
||||||
|
<p>Running <code>reproducer-quickstart.sh --help</code> will show a list of the
|
||||||
|
input options. All the options have a default and none are required.
|
||||||
|
The options are listed below with explanations:</p>
|
||||||
|
|
||||||
|
<ul>
|
||||||
|
<li><code>-w, --workspace</code>: directory where the virtualenv, inventory files, etc.
|
||||||
|
Defaults to creating a directory in /tmp</li>
|
||||||
|
<li><code>-v, --create-virtualenv</code>: create a virtualenv to install Ansible and dependencies.
|
||||||
|
Defaults to true.</li>
|
||||||
|
<li><code>-r, --remove-stacks-keypairs</code>: delete all stacks in the tenant before deployment.
|
||||||
|
will also delete associated keypairs if they exist.
|
||||||
|
Defaults to false.</li>
|
||||||
|
<li><code>-p, --nodestack-prefix</code>: add a unique prefix for multinode and singlenode stacks
|
||||||
|
Defaults to empty.</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h2>Using an alternative host cloud</h2>
|
||||||
|
<p>The <b>reproducer-quickstart.sh</b> script is written to run an a personal RDO Cloud tenant.
|
||||||
|
If you want to run on a different host cloud, modify the <b>reproducer-quickstart.sh</b> as follows:</p>
|
||||||
|
|
||||||
|
<ul>
|
||||||
|
<li>Add git clone of <code>tripleo-environments</code> or any other repository containing
|
||||||
|
the cloud details, after cloning <code>tripleo-quickstart-extras</code></li>
|
||||||
|
<li>Add the repository to quickstart-extras-requirements.txt:
|
||||||
|
<code>echo "file://$WORKSPACE/tripleo-environments/#egg=tripleo-environments" >>
|
||||||
|
$WORKSPACE/tripleo-quickstart/quickstart-extras-requirements.txt</code></li>
|
||||||
|
<li>Use that host cloud's environment file in place of RDO Cloud's environment file.</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
40
roles/create-reproducer-script/README.md
Normal file
40
roles/create-reproducer-script/README.md
Normal file
@ -0,0 +1,40 @@
|
|||||||
|
create-reproducer-script
|
||||||
|
========================
|
||||||
|
|
||||||
|
This role creates a script to reproduce OVB and multinode jobs.
|
||||||
|
|
||||||
|
Role Variables
|
||||||
|
--------------
|
||||||
|
|
||||||
|
For the defaults of these variables, see the defaults/main.yml file in this role.
|
||||||
|
|
||||||
|
* env_vars_to_source_file: env_vars_to_src.sh
|
||||||
|
* reproducer_quickstart_script: reproducer-quickstart.sh.j2
|
||||||
|
From the extras-common role:
|
||||||
|
* artcl_collect_dir: "{{ local_working_dir }}/collected_files"
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
------------
|
||||||
|
|
||||||
|
The role is run within the collect-logs role.
|
||||||
|
|
||||||
|
Example Playbook
|
||||||
|
----------------
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
---
|
||||||
|
- name: Create a file to reproduce the job
|
||||||
|
hosts: localhost
|
||||||
|
roles:
|
||||||
|
- create-reproducer-script
|
||||||
|
```
|
||||||
|
|
||||||
|
License
|
||||||
|
-------
|
||||||
|
|
||||||
|
Apache 2.0
|
||||||
|
|
||||||
|
Author Information
|
||||||
|
------------------
|
||||||
|
|
||||||
|
OpenStack
|
||||||
3
roles/create-reproducer-script/defaults/main.yml
Normal file
3
roles/create-reproducer-script/defaults/main.yml
Normal file
@ -0,0 +1,3 @@
|
|||||||
|
env_vars_to_source_file: env_vars_to_src.sh
|
||||||
|
reproducer_quickstart_script: reproducer-quickstart.sh.j2
|
||||||
|
reproducer_quickstart_readme_file: "{{ artcl_collect_dir }}/README-reproducer-quickstart.html"
|
||||||
2
roles/create-reproducer-script/meta/main.yml
Normal file
2
roles/create-reproducer-script/meta/main.yml
Normal file
@ -0,0 +1,2 @@
|
|||||||
|
dependencies:
|
||||||
|
- extras-common
|
||||||
21
roles/create-reproducer-script/tasks/main.yml
Normal file
21
roles/create-reproducer-script/tasks/main.yml
Normal file
@ -0,0 +1,21 @@
|
|||||||
|
---
|
||||||
|
- name: Set fact for environment variables
|
||||||
|
set_fact:
|
||||||
|
zuul_changes: "{{ lookup('env', 'ZUUL_CHANGES') }}"
|
||||||
|
nodes_config: "{{ lookup('env', 'NODES_FILE') }}"
|
||||||
|
toci_jobtype: "{{ lookup('env', 'TOCI_JOBTYPE') }}"
|
||||||
|
|
||||||
|
- name: Set fact for stable branch
|
||||||
|
set_fact:
|
||||||
|
stable_release: "{{ lookup('env', 'STABLE_RELEASE') }}"
|
||||||
|
|
||||||
|
- name: Create the reproducer file from template
|
||||||
|
template:
|
||||||
|
src: "{{ reproducer_quickstart_script }}"
|
||||||
|
dest: "{{ artcl_collect_dir }}/reproducer-quickstart.sh"
|
||||||
|
mode: 0755
|
||||||
|
|
||||||
|
- name: Create reproducer script documentation from template
|
||||||
|
template:
|
||||||
|
src: README-reproducer-quickstart.html.j2
|
||||||
|
dest: "{{ reproducer_quickstart_readme_file }}"
|
||||||
@ -0,0 +1,213 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# See documentation for using the reproducer script:
|
||||||
|
# README-reproducer-quickstart.html
|
||||||
|
# (in the same top-level logs directory as this reproducer script).
|
||||||
|
|
||||||
|
: ${WORKSPACE:=$(mktemp -d -t reproduce-tmp.XXXXX)}
|
||||||
|
{% if 'ovb' in toci_jobtype %}
|
||||||
|
: ${CREATE_VIRTUALENV:=true}
|
||||||
|
{% else %}
|
||||||
|
: ${CREATE_VIRTUALENV:=false}
|
||||||
|
{% endif %}
|
||||||
|
: ${REMOVE_STACKS_KEYPAIRS:=false}
|
||||||
|
: ${NODESTACK_PREFIX:=""}
|
||||||
|
|
||||||
|
usage () {
|
||||||
|
echo "Usage: $0 [options]"
|
||||||
|
echo ""
|
||||||
|
echo "Options:"
|
||||||
|
echo " -w, --workspace <dir>"
|
||||||
|
echo " directory where the virtualenv, inventory files, etc."
|
||||||
|
echo " are created. Defaults to creating a directory in /tmp"
|
||||||
|
echo " -v, --create-virtualenv"
|
||||||
|
echo " create a virtualenv to install Ansible and dependencies."
|
||||||
|
echo " Options to pass true/false. Defaults to true for OVB. "
|
||||||
|
echo " Defaults to false for other deployment types."
|
||||||
|
echo " -r, --remove-stacks-keypairs"
|
||||||
|
echo " delete all Heat stacks (both Multinode and OVB created) "
|
||||||
|
echo " in the tenant before deployment."
|
||||||
|
echo " Will also delete associated keypairs if they exist."
|
||||||
|
echo " Options to pass true/false.Defaults to false."
|
||||||
|
echo " -p, --nodestack-prefix"
|
||||||
|
echo " add a unique prefix for multinode and singlenode stacks"
|
||||||
|
echo " Defaults to empty."
|
||||||
|
echo " -h, --help print this help and exit"
|
||||||
|
}
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# Check that tenant credentials have been sourced
|
||||||
|
if [[ ! -v OS_TENANT_NAME ]]; then
|
||||||
|
echo "Tenant credentials are not sourced."
|
||||||
|
exit 1;
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Input argument assignments
|
||||||
|
while [ "x$1" != "x" ]; do
|
||||||
|
|
||||||
|
case "$1" in
|
||||||
|
--workspace|-w)
|
||||||
|
WORKSPACE=$(realpath $2)
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
|
||||||
|
--create-virtualenv|-v)
|
||||||
|
CREATE_VIRTUALENV=$2
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
|
||||||
|
--remove-stacks-keypairs|-r)
|
||||||
|
REMOVE_STACKS_KEYPAIRS=$2
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
|
||||||
|
--nodestack-prefix|-p)
|
||||||
|
NODESTACK_PREFIX=$2
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
|
||||||
|
--help|-h)
|
||||||
|
usage
|
||||||
|
exit
|
||||||
|
;;
|
||||||
|
|
||||||
|
--) shift
|
||||||
|
break
|
||||||
|
;;
|
||||||
|
|
||||||
|
-*) echo "ERROR: unknown option: $1" >&2
|
||||||
|
usage >&2
|
||||||
|
exit 2
|
||||||
|
;;
|
||||||
|
|
||||||
|
*) break
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
shift
|
||||||
|
done
|
||||||
|
|
||||||
|
set -x
|
||||||
|
# Exit if running ovb-fakeha-caserver
|
||||||
|
# This test is not converted to run with tripleo-quickstart
|
||||||
|
export TOCI_JOBTYPE="{{ toci_jobtype }}"
|
||||||
|
if [[ "$TOCI_JOBTYPE" == *"ovb-fakeha-caserver"* ]]; then
|
||||||
|
echo "
|
||||||
|
ovb-fakeha-caserver is not run with tripleo-quickstart.
|
||||||
|
It can not be reproduced using this script.
|
||||||
|
"
|
||||||
|
exit 1;
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Start from a clean workspace
|
||||||
|
export WORKSPACE
|
||||||
|
cd $WORKSPACE
|
||||||
|
rm -rf tripleo-quickstart tripleo-quickstart-extras
|
||||||
|
|
||||||
|
# Clone quickstart and quickstart-extras
|
||||||
|
git clone https://github.com/openstack/tripleo-quickstart
|
||||||
|
git clone https://github.com/openstack/tripleo-quickstart-extras
|
||||||
|
|
||||||
|
# Set up a virtual env if requested
|
||||||
|
if [ "$CREATE_VIRTUALENV" = "true" ]; then
|
||||||
|
virtualenv --system-site-packages $WORKSPACE/venv_ansible
|
||||||
|
source $WORKSPACE/venv_ansible/bin/activate
|
||||||
|
pip install --upgrade setuptools pip
|
||||||
|
pip install -r $WORKSPACE/tripleo-quickstart/requirements.txt
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$REMOVE_STACKS_KEYPAIRS" = "true" ]; then
|
||||||
|
# The cleanup templates expects there to be in a /bin dir in the workspace # from quickstart setup.
|
||||||
|
# To use the clients sourced from venv
|
||||||
|
sed -i "s#{.*/bin/##g" $WORKSPACE/tripleo-quickstart-extras/roles/ovb-manage-stack/templates/cleanup-stacks-keypairs.sh.j2
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Export ZUUL_CHANGES in the Ansible host if there are changes in
|
||||||
|
# tripleo-quickstart, tripleo-quickstart-extras or tripleo-ci repos
|
||||||
|
# before running playbooks
|
||||||
|
export ZUUL_CHANGES="{{ zuul_changes }}"
|
||||||
|
|
||||||
|
# Export our roles path so that we can use the roles from our workspace
|
||||||
|
export ANSIBLE_ROLES_PATH=$ANSIBLE_ROLES_PATH:$WORKSPACE/tripleo-quickstart/roles:$WORKSPACE/tripleo-quickstart-extras/roles
|
||||||
|
|
||||||
|
# Export a node config for the topology you need ie:
|
||||||
|
export NODES_FILE="{{ nodes_config }}"
|
||||||
|
|
||||||
|
{% if 'ovb' in toci_jobtype %}
|
||||||
|
ansible-playbook tripleo-quickstart-extras/playbooks/ovb-create-stack.yml \
|
||||||
|
-e local_working_dir=$WORKSPACE \
|
||||||
|
-e virthost=localhost \
|
||||||
|
-e @$WORKSPACE/tripleo-quickstart-extras/config/environments/rdocloud.yml \
|
||||||
|
-e ssh_extra_args="" \
|
||||||
|
-e ovb_dump_hosts=true \
|
||||||
|
-e ovb_setup_user=true \
|
||||||
|
-e cleanup_stacks_keypairs=$REMOVE_STACKS_KEYPAIRS \
|
||||||
|
-e @$WORKSPACE/tripleo-quickstart/$NODES_FILE
|
||||||
|
|
||||||
|
# Run the playbook to setup the undercloud/subnodes to look like nodepool nodes
|
||||||
|
ansible-playbook -i $WORKSPACE/ovb_hosts $WORKSPACE/tripleo-quickstart-extras/playbooks/nodepool-setup.yml
|
||||||
|
|
||||||
|
# Copy the nodes.json file to the undercloud
|
||||||
|
export $(awk '/subnode-0/ {print $2}' ovb_hosts)
|
||||||
|
scp $WORKSPACE/nodes.json zuul@$ansible_host:/home/zuul/
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
{% if 'multinode' in toci_jobtype or 'singlenode' in toci_jobtype %}
|
||||||
|
# Calculate subnode_count
|
||||||
|
if [[ -z "$NODES_FILE" ]]; then
|
||||||
|
SUBNODE_COUNT=1
|
||||||
|
else
|
||||||
|
SUBNODE_COUNT=$(( $( awk '/node_count: / {print $2}' $WORKSPACE/tripleo-quickstart/$NODES_FILE ) +1 ))
|
||||||
|
fi
|
||||||
|
|
||||||
|
ansible-playbook tripleo-quickstart-extras/playbooks/provision_multinodes.yml \
|
||||||
|
-e local_working_dir=$WORKSPACE \
|
||||||
|
-e subnode_count=$SUBNODE_COUNT \
|
||||||
|
-e prefix=$NODESTACK_PREFIX
|
||||||
|
|
||||||
|
# Run the playbook to setup the undercloud/subnodes to look like nodepool nodes
|
||||||
|
ansible-playbook -i $WORKSPACE/multinode_hosts $WORKSPACE/tripleo-quickstart-extras/playbooks/nodepool-setup.yml
|
||||||
|
|
||||||
|
# Get ansible_host
|
||||||
|
export $(awk '/subnode-0/ {print $2}' multinode_hosts)
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
{% if dlrn_hash_newest is defined %}
|
||||||
|
EXTRA_VARS="$EXTRA_VARS --extra-vars dlrn_hash_tag_newest={{ hostvars['undercloud'].dlrn_hash_newest }} "
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
# Create the env_vars_to_source file and copy it to the undercloud
|
||||||
|
cat >"{{ env_vars_to_source_file }}" <<EOF
|
||||||
|
export ZUUL_CHANGES="{{ zuul_changes }}"
|
||||||
|
export NODES_FILE="{{ nodes_config }}"
|
||||||
|
export TOCI_JOBTYPE="{{ toci_jobtype }}"
|
||||||
|
export EXTRA_VARS="$EXTRA_VARS --extra-vars dlrn_hash_tag={{ hostvars['undercloud'].dlrn_hash }} "
|
||||||
|
EOF
|
||||||
|
|
||||||
|
{% if stable_release != '' %}
|
||||||
|
cat >>"{{ env_vars_to_source_file }}" <<EOF
|
||||||
|
export STABLE_RELEASE="{{ stable_release }}"
|
||||||
|
EOF
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
{% if 'ovb' in toci_jobtype %}
|
||||||
|
cat >>"{{ env_vars_to_source_file }}" <<EOF
|
||||||
|
export TE_DATAFILE=/home/zuul/nodes.json
|
||||||
|
EOF
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
scp "$WORKSPACE/{{ env_vars_to_source_file }}" zuul@$ansible_host:/home/zuul/
|
||||||
|
|
||||||
|
# Remove -x so that the instructions don't print twice
|
||||||
|
set +x
|
||||||
|
|
||||||
|
# Instruct the user to execute toci_gate_test-oooq.sh on the undercloud
|
||||||
|
echo "
|
||||||
|
Now complete the test excution on the undercloud:
|
||||||
|
- ssh to the undercloud: $ ssh zuul@$ansible_host
|
||||||
|
- Source the environment settings file: $ source /home/zuul/env_vars_to_src.sh
|
||||||
|
- Run the toci gate script: $ /opt/stack/tripleo-ci/toci_gate_test-oooq.sh
|
||||||
|
|
||||||
|
To avoid timeouts, you can start a screen session before executing the commands: $ screen -S ci
|
||||||
|
"
|
||||||
@ -25,3 +25,4 @@ timestamper_cmd: >-
|
|||||||
enable_libvirt_tripleo_ui: false
|
enable_libvirt_tripleo_ui: false
|
||||||
composable_scenario: ""
|
composable_scenario: ""
|
||||||
upgrade_composable_scenario: ""
|
upgrade_composable_scenario: ""
|
||||||
|
artcl_collect_dir: "{{ local_working_dir }}/collected_files"
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user