openstack/tripleo-quickstart-extras
Code Issues Proposed changes

Add fresh-box-setup script for zuul-reproducer first run

Browse Source 
On first run or a fresh setup there are some required steps like
creating a user, a clouds.yaml and ssh keys for gerrit. This adds
a script into the create reproducer role which can be used for
this initial setup

Change-Id: I6d70a87006800447a12b4609ccc81400741290db
This commit is contained in:
Marios Andreou
2019-05-08 13:55:45 +03:00
parent 1b88449454
commit a04f3c986b
4 changed files with 258 additions and 94 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
roles/create-zuul-based-reproducer/defaults/main.yml
View File

@@ -1,5 +1,6 @@
---
reproducer_zuul_based_quickstart_script: reproducer-zuul-based-quickstart.sh.j2
reproducer_fresh_box_setup_script: reproducer-fresh-box-setup.sh.j2
reproducer_zuul_based_quickstart_readme_file: "{{ artcl_collect_dir }}/README-reproducer.html"
reproducer_zuul_based_launcher_playbook: launcher-playbook.yaml.j2
launcher_env_setup_playbook: launcher-env-setup-playbook.yaml.j2

6
roles/create-zuul-based-reproducer/tasks/main.yml
View File

@@ -76,6 +76,12 @@
dest: "{{ artcl_collect_dir }}/{{ zuul_reproducer_dir }}/reproducer-zuul-based-quickstart.sh"
mode: 0755
- name: Create the reproducer-fresh-box-setup script
template:
src: "{{ reproducer_fresh_box_setup_script }}"
dest: "{{ artcl_collect_dir }}/{{ zuul_reproducer_dir }}/reproducer-fresh-box-setup.sh"
mode: 0755
- name: Copy the tripleo-quickstart install-deps script
copy:
src: "/home/{{ undercloud_user }}/src/opendev.org/openstack/tripleo-quickstart/{{ item }}"

218
roles/create-zuul-based-reproducer/templates/README-reproducer-zuul-based-quickstart.html.j2
View File

@@ -54,125 +54,152 @@ issue.</p>
<button class="collapsible">Setting up to run the Zuul-based reproducer</button>
<div class="content">
<p>
Prior to running the Zuul-based reproducer, the following setup steps are needed -
( only need to be done once per local system):
To setup and run the reproducer you must first decide <b>where</b> it will run
- on a <b>pet</b> (your laptop) or on <b>cattle</b> (rdo-cloud vm or other
temporary environment)? For each case see notes below for required setup.
<p>
<b>If this is the first time you are running the reproducer</b> you must
ensure you have an externally routable network in your RDO-cloud tenant called
'private'. That is you will need a router attached to the external network that
is also attached to your network called 'private' (which you may need to create).
Assuming you have a clouds.yaml that let's you talk to RDO cloud you can use
the following commands to create the required setup:
<pre>
<code>
openstack --os-cloud rdo-cloud network create private
openstack --os-cloud rdo-cloud subnet create private --network private --gateway 192.168.0.1 --subnet-range 192.168.0.0/24
openstack --os-cloud rdo-cloud router create privaterouter
openstack --os-cloud rdo-cloud router set --external-gateway 38.145.32.0/22 privaterouter
openstack --os-cloud rdo-cloud router add subnet privaterouter private
</code>
</pre>
</p>
Alternatively log in to <a href="https://phx2.cloud.rdoproject.org/dashboard/auth/login/?next=/dashboard/project/networks/">RDO cloud horizon</a> and create the network and router
that way.
<hr></hr>
<h1>Cattle</h1>
If you're using a temporary environment like RDO-cloud vm to run the
reproducer you can use the script inside the reproducer-quickstart/ directory
from the logs of the job you are reproducing. The script is acalled
<code>reproducer-fresh-box-setup.sh</code>:
<pre><code><b>
curl -o reproducer-fresh-box-setup.sh http://logs.openstack.org/79/657779/2/check/tripleo-ci-centos-7-containers-multinode/67d8eb6/logs/reproducer-quickstart/reproducer-fresh-box-setup.sh
chmod 775 reproducer-fresh-box-setup.sh
./reproducer-fresh-box-setup.sh -u marios -p password -c reprozuul
</b></code></pre>
<p><b>note:</b> If you're using an RDO-cloud vm be sure to use at least m1.large2
flavour as we've hit hard disk limits with m1.large</p>
<p> There are two required parameters <code>-u USER</code/> and <code>-p PASS</code>
which correspond to the RDO cloud username and password. These are written to
the generated <code>$HOME/.config/openstack/clouds.yaml</code> file expected by the
reproducer.
</p>
<p>This script creates the default user 'reprozuul' with passwordless sudo
privileges, creates $HOME/.config/openstack/clouds.yaml with the -u USER and
-p PASS parameters and creates the required ssh keypair. It must run as root,
or as a user with passwordless sudo.
</p>
<p> The script finally switches to the newly created user and outputs the newly
created public key. You must upload this key to gerrit (<b>both</b>
<a href="https://review.opendev.org/#/settings/ssh-keys">opendev.org</a> as well
as <a href="https://review.rdoproject.org/r/#/settings/ssh-keys">rdoproject.org</a>).
</p>
<hr></hr>
<h1>Pet</h1>
If you are running the reproducer on a non-transient machine, that is your
laptop or any other box which isn't temporary (a pet), then you need:
<ul>
<li>Create a clouds.yaml - for example:
<li> Ensure user running reproducer has passwordless sudo </li>
<li> That user has $HOME/.config/openstack/clouds.yaml with "rdo-cloud" entry </li>
<li> A ssh keypair generated with <b><code>-m PEM -t rsa</code></b> and uploaded
to gerrit (<b>both </b><a href="https://review.opendev.org/#/settings/ssh-keys">opendev.org</a>
as well as <a href="https://review.rdoproject.org/r/#/settings/ssh-keys">rdoproject.org</a>).
</li>
<li> A externally routable network in your RDO-cloud tenant named 'private'.
See example commands above if you need to create this or log in to
<a href="https://phx2.cloud.rdoproject.org/dashboard/auth/login/?next=/dashboard/project/networks/"> RDO cloud horizon</a> and verify your setup or create as needed.
</ul>
You only need to do these things once on your pet and then rerun reproducer or
update as needed.
<ul>
<li>Example clouds.yaml:</li>
<pre>
<code>mkdir -p ~/.config/openstack
vi ~/.config/openstack/clouds.yaml
<code>CLOUDS_YAML_PATH=$HOME/.config/openstack/
mkdir -p $CLOUDS_YAML_PATH
/bin/bash -c "cat <<EOF>>$CLOUDS_YAML_PATH/clouds.yaml
clouds:
rdo-cloud:
identity_api_version: 3
region_name: regionOne
auth:
auth_url: https://phx2.cloud.rdoproject.org:13000/v3
password: xxxxxxx
project_name: user_name
username:  user_name
password: $RDO_PASS
project_name: $RDO_USER
username: $RDO_USER
user_domain_name: Default
project_domain_name: Default
regions:
- name: regionOne
values:
networks:
- name: 38.145.32.0/22
routes_externally: true
- name: private
- name: regionOne
values:
networks:
- name: 38.145.32.0/22
routes_externally: true
- name: private
EOF"
</code>
</pre>
</ul>
<ul>
<li> Test your cloud.yaml by running:
<pre>
<code>
openstack --os-cloud rdo-cloud image list
</code>
</pre>
</ul>
<ul>
<li>Note that you will need a network named 'private' in the tenant connected to an
router that is connected to the external network.</li>
<li>Create an ssh key pair without a pass phrase:</li>
<br/>
<li> Create an ssh key pair without a pass phrase:</li>
<ul>
<li>The ssh key pair is configurable - see variables <code>ssh_path</code></li>
<li><code>user_pri_key</code>. The default key is:<code>~/.ssh/id_rsa[.pub]</code></li>
<li> <b>Note:</b> Reference FAQ below when creating ssh keys, some keys fail when used w/ gerrit
<li>The ssh key pair is configurable - see variables <code>ssh_path</code>
<code>user_pri_key</code>. The default key is:<code>~/.ssh/id_rsa[.pub]</code></li>
<li> <b>Note:</b> Use the following to create keys: <b><code> ssh-keygen -m PEM -t rsa </code></b>
See FAQ below for more information about ssh keys requirements.
</ul>
</p>
</ul>
</br>
</div>
<button class="collapsible">Running the Zuul-based reproducer - using the Bash script</button>
<div class="content">
<p>
<ul>
<li> wget or curl the <b>reproducer-quickstart/reproducer-zuul-based-quickstart.tar</b> file. </li>
<li> Note that the <code>launcher-env-setup-playbook.yaml</code> will set up
clone repos and set up 'roles', 'playbooks', and 'library' directories
from which the tasks will be executed.</li>
<li> Once the files are extracted from the archive, you can run the Bash script
with the various options. Run the script with <code>--help</code> option to
output a list of available run options. </li>
</ul>
<b>Note:</b> the local $USERNAME idealy matches the openstack and rdo gerrit user id's
If not pay extra attention to the -ug and -rg options.
<ul>
<code>
<pre>
Options:
-w, --workspace dir
directory where the repos and directories
are created. Defaults to creating a directory in /tmp
-l, --libvirt
Runs a 2-node multinode job or singlenode job on a
single virthost using libvirt to create the nodes.
If a singlenode reproducer is run, two VMs will still be created.
-c, --cloud-name
Host cloud, specified in the clouds.yaml, to target
Defaults to rdo-cloud
-cp, --clouds-yaml-path
Full path to the clouds.yaml file
Defaults to /home/$USERNAME/.config/openstack/clouds.yaml
-ok, --ovb-key-name
Name of the key to use in the host tenant for OVB deployments
Defaults to tripleo-ci-team
-f, --force-post-failure
Force job to fail so that the nodes will be held.
Temporary solution
-ug, --upstream-gerrit-user
Set the upstream gerrit user required to clone repos.
Defaults to the current $USERNAME
-rg, --rdo-gerrit-user
Set the upstream RDO user required to clone RDO-related repos.
Defaults to the current $USERNAME
-k, --ssh-key
private ssh key used to set up an access nodes.
Defaults to id_rsa.
-kp, --ssh-key-public
public ssh key used to set up an access nodes.
Defaults to id_rsa.pub
-skp, --ssh-key-path
Path to directory where user ssh keys are stored.
Defaults to /home/$USERNAME/.ssh
-ugk, --upstream-gerrit-key
Set the upstream gerrit private key.
Defaults to the id_rsa.
-rgk, --rdo-gerrit-key
Set the upstream RDO user key.
Defaults to the id_rsa.
-e, --extra-params
File or/and parameters used to override default
parameters for playbooks. Multiple files
can be passed [-e @file1.yml -e @file2.yml ...]
and arguments [-e var=value -e var2=value2 ...]
-h, --help print this help and exit
wget or curl the <b>reproducer-quickstart/reproducer-zuul-based-quickstart.tar</b> file.
<pre><code><b>
wget http://logs.openstack.org/79/657779/2/check/tripleo-ci-centos-7-standalone-upgrade/92fd476/logs/reproducer-quickstart/reproducer-zuul-based-quickstart.tar
tar -xvf reproducer-zuul-based-quickstart.tar
</b></code></pre>
One of the extracted files is the script called <code>./reproducer-zuul-based-quickstart.sh</code>
and you can run it with <code>--help</code> to see available options.
</br>
</br>
<b>Note:</b> If the local $USERNAME does not match the gerrit user IDs for
<a href="https://review.opendev.org/#/settings/ssh-keys">opendev.org</a> or
<a href="https://review.rdoproject.org/r/#/settings/ssh-keys">rdoproject.org</a>)
then you will need to supply the -ug (upstream gerrit) and -rg (rdo gerrit)
parameters:
</code>
</pre>
<ul>
</ul>
<pre><code><b>
mkdir WORKSPACE
./reproducer-zuul-based-quickstart.sh --workspace WORKSPACE -ug slim_shady -rg slim_shadier
</b></code></pre>
</p>
Monitor the output and follow any instructions - for example the script will
exit and require re-run once the user is added to the docker group.
</br>
</br>
</div>
<button class="collapsible">Running the Zuul-based reproducer - using the playbooks directly</button>
@@ -183,9 +210,12 @@ The playbooks can be run directly using <code>ansible-playbook</code>.
The Bash script contains some package installation and Docker user/group setup
that you will need to cover prior to running the playbooks directly.
Please see the Bash scripts for details, especially if you are running the
reproducer for the first time on teh local environment.
reproducer for the first time on the local environment.
The playbooks can be run as follows:
<ul>
<li> Note that the <code>launcher-env-setup-playbook.yaml</code> will clone
repos and set up 'roles', 'playbooks', and 'library' directories
from which the tasks will be executed. </li>
<li><code>ansible-playbook ./launcher-env-setup-playbook.yaml</code></li>
<li><code>ansible-playbook ./launcher-playbook.yaml $OPTIONS</code></li>
</ul>

127
roles/create-zuul-based-reproducer/templates/reproducer-fresh-box-setup.sh.j2 Normal file
View File

@@ -0,0 +1,127 @@
#!/bin/bash
REPRO_USER=${REPRO_USER:=reprozuul}
REPRO_DEPS="tar wget screen"
RDO_USER=""
RDO_PASS=""
CLOUDS_YAML_PATH=".config/openstack/"
usage () {
echo "Usage: $0 [options]"
echo ""
echo "Options:"
echo " -u, --rdo-user"
echo " Username for RDO cloud to be written to"
echo " $CLOUDS_YAML_PATH/clouds.yaml. REQUIRED."
echo " -p, --rdo-pass"
echo " Password for RDO cloud to be written to"
echo " $CLOUDS_YAML_PATH/clouds.yaml. REQUIRED."
echo " -c, --create-user"
echo " Local user to create for running reproducer."
echo " Defaults to $REPRO_USER"
echo " -h, --help print this help and exit"
}
set -e
# Input argument assignments
while [ "x$1" != "x" ]; do
case "$1" in
--rdo-user|-u)
RDO_USER=$2
shift
;;
--rdo-pass|-p)
RDO_PASS=$2
shift
;;
--create-user|-c)
REPRO_USER=$2
shift
;;
--help|-h)
usage
exit
;;
--) shift
break
;;
-*) echo "ERROR: unknown option: $1" >&2
usage >&2
exit 2
;;
*) break
;;
esac
shift
done
# check required params - rdo user/pass
for param in "$RDO_USER" "$RDO_PASS" ; do
if [[ $param = "" ]]; then
echo "error --rdo-user and --rdo-pass are required parameters"
exit 1
fi
done
# create user
sudo adduser $REPRO_USER
echo "$REPRO_USER ALL=(root) NOPASSWD:ALL" | sudo tee -a /etc/sudoers.d/$REPRO_USER
# install packages
sudo yum -y install $REPRO_DEPS
# write clouds.yaml
CLOUDS_YAML_PATH="/home/$REPRO_USER/.config/openstack/"
sudo mkdir -p $CLOUDS_YAML_PATH
sudo /bin/bash -c "cat <<EOF>>$CLOUDS_YAML_PATH/clouds.yaml
clouds:
rdo-cloud:
identity_api_version: 3
region_name: regionOne
auth:
auth_url: https://phx2.cloud.rdoproject.org:13000/v3
password: $RDO_PASS
project_name: $RDO_USER
username: $RDO_USER
user_domain_name: Default
project_domain_name: Default
regions:
- name: regionOne
values:
networks:
- name: 38.145.32.0/22
routes_externally: true
- name: private
EOF"
sudo chown -R $REPRO_USER:$REPRO_USER /home/$REPRO_USER/.config
# create keys
mkdir keystemp
pushd keystemp
ssh-keygen -m PEM -t rsa -N "" -f id_rsa
ssh_path=/home/$REPRO_USER/.ssh
sudo mkdir $ssh_path
sudo mv id_rsa* $ssh_path
sudo chmod 700 $ssh_path
sudo chmod 600 $ssh_path/id_rsa
sudo chmod 644 $ssh_path/id_rsa.pub
sudo chown -R $REPRO_USER:$REPRO_USER $ssh_path
popd
rm -rf keystemp
echo "Fresh box setup complete - now go upload the public key in"
echo "$CLOUDS_YAML_PATH to Gerrit - both upstream and rdo. Then you are ready "
echo "to run the reproducer. Here is the public key you must upload:"
sudo cat /home/$REPRO_USER/.ssh/id_rsa.pub
echo "Switching to $REPRO_USER"
sudo su - $REPRO_USER