How to recreate this job

Please refer to the recreation instructions

How to figure out what went wrong?

Check the console log and search for PLAY RECAP. There are sometimes multiple ansible runs in a job, usually the last one is the relevant. If no PLAY RECAP text is found that usually means an infra failure before Quickstart could even start. Try rechecking or asking on #tripleo if there's an ongoing infra issue.

Look for a line above the PLAY RECAP that starts with "fatal:". If no such line is found, try searching for other PLAY RECAP lines or other error outputs.

If this "fatal" line contains the execution of a shell script and redirects to a log, check which machine that task ran on. Look under that node's directory in the logs to find the file.

Example output:

fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "set -o pipefail && /home/zuul/overcloud-prep-images.sh 2>&1 | awk '{ print strftime(\"%Y-%m-%d %H:%M:%S |\"), $0; fflush(); }' > /home/stack/overcloud_prep_images.log", "failed": true, "rc": 1}

PLAY RECAP *********************************************************************


In this case the overcloud-prep-images.sh script failed, which is redirected to /home/zuul/overcloud_prep_images.log on the undercloud.

If this is a different Ansible error, that could mean either an infra problem (often has UNREACHABLE in the line) or a bug in Quickstart. Ask on #tripleo to get help or open a bug on Launchpad. Add the "ci" tag if it's a CI issue and "quickstart" if you suspect that the bug is in Quickstart itself.

Links to common log files

• undercloud/home/zuul/ - the source and log output of all templated shell scripts
• undercloud/var/log/ - directories and files copied from /var/log on the undercloud. If other overcloud/subnodes exist, similar $node/var/log directories will also exist in these logs.
• undercloud/var/log/extra/ - extra system details like package list, and cpu info gathered from the undercloud
• undercloud/var/lib/mistral - output of all ansible used by config-download to drive the overcloud deployment
• stackwiz - tempest test results
• docs/build/ - autogenerated documentation

Variables used in the job run

The logs contain files showing variables used in the job run.

• undercloud/var/log/extra/dump_variables_vars.json.txt.gz - contains the variables used in the running the actual test
• releases.sh - the output of the script setting release-related variables
• playbook_executions.log - prints out the complete commands, with all expanded arguments, to run the playbooks

Dry Run option

As a debugging step, a job can be run manually with '-dryrun' appended to the job name. When the "playbook dry run" option is invoked, the playbooks will not execute and collect logs will not run but certain log files, including 'toci_env_args_output.log', which contains the environment variables used in the job, and playbook_executions.log will still be produced in the logs directory for inspection. This option serves to assist with debugging and to test the testing scripts themselves.