To setup and run the reproducer you must first decide where it will run - on a pet (your laptop) or on cattle (rdo-cloud vm or other temporary environment)? For each case see notes below for required setup.

If this is the first time you are running the reproducer you must ensure you have an externally routable network in your RDO-cloud tenant called 'private'. That is you will need a router attached to the external network that is also attached to your network called 'private' (which you may need to create). Assuming you have a clouds.yaml that let's you talk to RDO cloud you can use the following commands to create the required setup:

    openstack --os-cloud rdo-cloud network create private
    openstack --os-cloud rdo-cloud subnet create private --network private --gateway 192.168.0.1 --subnet-range 192.168.0.0/24
    openstack --os-cloud rdo-cloud router create privaterouter
    openstack --os-cloud rdo-cloud router set --external-gateway 38.145.32.0/22  privaterouter
    openstack --os-cloud rdo-cloud router add subnet privaterouter private

Alternatively log in to RDO cloud horizon and create the network and router that way.

---

Cattle

If you're using a temporary environment like RDO-cloud vm to run the reproducer you can use the script inside the reproducer-quickstart/ directory from the logs of the job you are reproducing. The script is acalled reproducer-fresh-box-setup.sh:

    curl -o reproducer-fresh-box-setup.sh http://logs.openstack.org/79/657779/2/check/tripleo-ci-centos-7-containers-multinode/67d8eb6/logs/reproducer-quickstart/reproducer-fresh-box-setup.sh
    chmod 775 reproducer-fresh-box-setup.sh
    ./reproducer-fresh-box-setup.sh -u marios -p password -c reprozuul

note: If you're using an RDO-cloud vm be sure to use at least m1.large2 flavour as we've hit hard disk limits with m1.large

There are two required parameters -u USER and -p PASS which correspond to the RDO cloud username and password. These are written to the generated $HOME/.config/openstack/clouds.yaml file expected by the reproducer.

This script creates the default user 'reprozuul' with passwordless sudo privileges, creates $HOME/.config/openstack/clouds.yaml with the -u USER and -p PASS parameters and creates the required ssh keypair. It must run as root, or as a user with passwordless sudo.

The script finally switches to the newly created user and outputs the newly created public key. You must upload this key to gerrit (both opendev.org as well as rdoproject.org).

---

Pet

If you are running the reproducer on a non-transient machine, that is your laptop or any other box which isn't temporary (a pet), then you need:

• Ensure user running reproducer has passwordless sudo
• That user has $HOME/.config/openstack/clouds.yaml with "rdo-cloud" entry
• A ssh keypair generated with -m PEM -t rsa and uploaded to gerrit (both opendev.org as well as rdoproject.org).
• A externally routable network in your RDO-cloud tenant named 'private'. See example commands above if you need to create this or log in to RDO cloud horizon and verify your setup or create as needed.

You only need to do these things once on your pet and then rerun reproducer or update as needed.

• Example clouds.yaml:

CLOUDS_YAML_PATH=$HOME/.config/openstack/
mkdir -p $CLOUDS_YAML_PATH
/bin/bash -c "cat <>$CLOUDS_YAML_PATH/clouds.yaml
clouds:
  rdo-cloud:
    identity_api_version: 3
    region_name: regionOne
    auth:
      auth_url: https://phx2.cloud.rdoproject.org:13000/v3
      password: $RDO_PASS
      project_name: $RDO_USER
      username: $RDO_USER
      user_domain_name: Default
      project_domain_name: Default
    regions:
    - name: regionOne
      values:
        networks:
        - name: 38.145.32.0/22
          routes_externally: true
        - name: private
EOF"

• Create an ssh key pair without a pass phrase:
• The ssh key pair is configurable - see variables ssh_path user_pri_key. The default key is:~/.ssh/id_rsa[.pub]
• Note: Use the following to create keys: ssh-keygen -m PEM -t rsa See FAQ below for more information about ssh keys requirements.

wget or curl the reproducer-quickstart/reproducer-zuul-based-quickstart.tar file.

    curl -Lo reproducer-zuul-based-quickstart.tar http://logs.openstack.org/79/657779/2/check/tripleo-ci-centos-7-standalone-upgrade/92fd476/logs/reproducer-quickstart/reproducer-zuul-based-quickstart.tar
    tar -xvf reproducer-zuul-based-quickstart.tar

One of the extracted files is the script called ./reproducer-zuul-based-quickstart.sh and you can run it with --help to see available options.

Note: If the local $USERNAME does not match the gerrit user IDs for opendev.org or rdoproject.org) then you will need to supply the -ug (upstream gerrit) and -rg (rdo gerrit) parameters:

    mkdir WORKSPACE
    ./reproducer-zuul-based-quickstart.sh --workspace WORKSPACE -ug slim_shady -rg slim_shadier

Monitor the output and follow any instructions - for example the script will exit and require re-run once the user is added to the docker group.

The Bash script is not required although it is the supported path. The playbooks can be run directly using ansible-playbook. The Bash script contains some package installation and Docker user/group setup that you will need to cover prior to running the playbooks directly. Please see the Bash scripts for details, especially if you are running the reproducer for the first time on the local environment. The playbooks can be run as follows:

• Note that the launcher-env-setup-playbook.yaml will clone repos and set up 'roles', 'playbooks', and 'library' directories from which the tasks will be executed.
• ansible-playbook ./launcher-env-setup-playbook.yaml
• ansible-playbook ./launcher-playbook.yaml $OPTIONS

The launcher-playbook.yaml includes comments on how to run the playbook with the same options listed in the Bash script section above.

The reproducer can be launched such that if there is an error the environment will remain up and running available to the developer to debug.

• For example:

cd ~/tripleo-ci-reproducer

docker-compose exec scheduler zuul autohold --tenant tripleo-ci-reproducer --job tripleo-ci-centos-7-standalone --reason debug --project test1

• The Bash scripts runs with --tags all passed to the launcher-playbook.yaml.
• After the initial docker setup has run, it is not necessary to run through those setup steps for every reproducer. The launcher-playbook.yaml can be run with --skip-tags start after the initial run.

The launcher-playbook.yaml will create and push a job to a test1 testproject repo in a temp directory. This job will start and the related Zuul infrastructure logs can be viewed in:

cd ~/tripleo-ci-reproducer
docker-compose logs --tail=10 -f
# Just view the scheduler and executor logs:
docker-compose logs --tail=10 -f scheduler executor

To see the job progress, you can access the web consoles of:

• Gerrit: http://localhost:8080
• Zuul: http://localhost:9000
• Logs: http://localhost:8000

If you are accessing a remote machine, replace localhost with the ip address.

• Gerrit keys
  If your ssh keys have being created with a new version of OpenSSH, Zuul will fail when running with these keys as there is a Paramiko bug [1]. It is recommended that keys be generated as follows: ssh-keygen -m PEM -t rsa - accepting the passwordless option. Then update the public keys used with Gerrit upstream and RDO Gerrit.
  [1] https://storyboard.openstack.org/#!/story/2004842

• If you currently do not have access to upstream-cloudinit-[centos-7, fedora-28] :

source the openrc from the openstack-nodepool tenant
openstack image list | grep upstream-cloudinit
glance member-create ${IMAGE_ID} ${YOUR_TENANT_ID}
glance member-create ${IMAGE_ID} ${YOUR_TENANT_ID}