Add fresh-box-setup script for zuul-reproducer first run
On first run or a fresh setup there are some required steps like creating a user, a clouds.yaml and ssh keys for gerrit. This adds a script into the create reproducer role which can be used for this initial setup Change-Id: I6d70a87006800447a12b4609ccc81400741290db
This commit is contained in:
Marios Andreou
parent
1b88449454
commit
a04f3c986b
@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
reproducer_zuul_based_quickstart_script: reproducer-zuul-based-quickstart.sh.j2
|
reproducer_zuul_based_quickstart_script: reproducer-zuul-based-quickstart.sh.j2
|
||||||
|
reproducer_fresh_box_setup_script: reproducer-fresh-box-setup.sh.j2
|
||||||
reproducer_zuul_based_quickstart_readme_file: "{{ artcl_collect_dir }}/README-reproducer.html"
|
reproducer_zuul_based_quickstart_readme_file: "{{ artcl_collect_dir }}/README-reproducer.html"
|
||||||
reproducer_zuul_based_launcher_playbook: launcher-playbook.yaml.j2
|
reproducer_zuul_based_launcher_playbook: launcher-playbook.yaml.j2
|
||||||
launcher_env_setup_playbook: launcher-env-setup-playbook.yaml.j2
|
launcher_env_setup_playbook: launcher-env-setup-playbook.yaml.j2
|
||||||
|
|||||||
@ -76,6 +76,12 @@
|
|||||||
dest: "{{ artcl_collect_dir }}/{{ zuul_reproducer_dir }}/reproducer-zuul-based-quickstart.sh"
|
dest: "{{ artcl_collect_dir }}/{{ zuul_reproducer_dir }}/reproducer-zuul-based-quickstart.sh"
|
||||||
mode: 0755
|
mode: 0755
|
||||||
|
|
||||||
|
- name: Create the reproducer-fresh-box-setup script
|
||||||
|
template:
|
||||||
|
src: "{{ reproducer_fresh_box_setup_script }}"
|
||||||
|
dest: "{{ artcl_collect_dir }}/{{ zuul_reproducer_dir }}/reproducer-fresh-box-setup.sh"
|
||||||
|
mode: 0755
|
||||||
|
|
||||||
- name: Copy the tripleo-quickstart install-deps script
|
- name: Copy the tripleo-quickstart install-deps script
|
||||||
copy:
|
copy:
|
||||||
src: "/home/{{ undercloud_user }}/src/opendev.org/openstack/tripleo-quickstart/{{ item }}"
|
src: "/home/{{ undercloud_user }}/src/opendev.org/openstack/tripleo-quickstart/{{ item }}"
|
||||||
|
|||||||
@ -54,125 +54,152 @@ issue.</p>
|
|||||||
<button class="collapsible">Setting up to run the Zuul-based reproducer</button>
|
<button class="collapsible">Setting up to run the Zuul-based reproducer</button>
|
||||||
<div class="content">
|
<div class="content">
|
||||||
<p>
|
<p>
|
||||||
Prior to running the Zuul-based reproducer, the following setup steps are needed -
|
To setup and run the reproducer you must first decide <b>where</b> it will run
|
||||||
( only need to be done once per local system):
|
- on a <b>pet</b> (your laptop) or on <b>cattle</b> (rdo-cloud vm or other
|
||||||
|
temporary environment)? For each case see notes below for required setup.
|
||||||
|
|
||||||
|
<p>
|
||||||
|
<b>If this is the first time you are running the reproducer</b> you must
|
||||||
|
ensure you have an externally routable network in your RDO-cloud tenant called
|
||||||
|
'private'. That is you will need a router attached to the external network that
|
||||||
|
is also attached to your network called 'private' (which you may need to create).
|
||||||
|
Assuming you have a clouds.yaml that let's you talk to RDO cloud you can use
|
||||||
|
the following commands to create the required setup:
|
||||||
|
<pre>
|
||||||
|
<code>
|
||||||
|
openstack --os-cloud rdo-cloud network create private
|
||||||
|
openstack --os-cloud rdo-cloud subnet create private --network private --gateway 192.168.0.1 --subnet-range 192.168.0.0/24
|
||||||
|
openstack --os-cloud rdo-cloud router create privaterouter
|
||||||
|
openstack --os-cloud rdo-cloud router set --external-gateway 38.145.32.0/22 privaterouter
|
||||||
|
openstack --os-cloud rdo-cloud router add subnet privaterouter private
|
||||||
|
</code>
|
||||||
|
</pre>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
Alternatively log in to <a href="https://phx2.cloud.rdoproject.org/dashboard/auth/login/?next=/dashboard/project/networks/">RDO cloud horizon</a> and create the network and router
|
||||||
|
that way.
|
||||||
|
|
||||||
|
<hr></hr>
|
||||||
|
<h1>Cattle</h1>
|
||||||
|
If you're using a temporary environment like RDO-cloud vm to run the
|
||||||
|
reproducer you can use the script inside the reproducer-quickstart/ directory
|
||||||
|
from the logs of the job you are reproducing. The script is acalled
|
||||||
|
<code>reproducer-fresh-box-setup.sh</code>:
|
||||||
|
|
||||||
|
<pre><code><b>
|
||||||
|
curl -o reproducer-fresh-box-setup.sh http://logs.openstack.org/79/657779/2/check/tripleo-ci-centos-7-containers-multinode/67d8eb6/logs/reproducer-quickstart/reproducer-fresh-box-setup.sh
|
||||||
|
chmod 775 reproducer-fresh-box-setup.sh
|
||||||
|
./reproducer-fresh-box-setup.sh -u marios -p password -c reprozuul
|
||||||
|
</b></code></pre>
|
||||||
|
|
||||||
|
<p><b>note:</b> If you're using an RDO-cloud vm be sure to use at least m1.large2
|
||||||
|
flavour as we've hit hard disk limits with m1.large</p>
|
||||||
|
<p> There are two required parameters <code>-u USER</code/> and <code>-p PASS</code>
|
||||||
|
which correspond to the RDO cloud username and password. These are written to
|
||||||
|
the generated <code>$HOME/.config/openstack/clouds.yaml</code> file expected by the
|
||||||
|
reproducer.
|
||||||
|
</p>
|
||||||
|
<p>This script creates the default user 'reprozuul' with passwordless sudo
|
||||||
|
privileges, creates $HOME/.config/openstack/clouds.yaml with the -u USER and
|
||||||
|
-p PASS parameters and creates the required ssh keypair. It must run as root,
|
||||||
|
or as a user with passwordless sudo.
|
||||||
|
</p>
|
||||||
|
<p> The script finally switches to the newly created user and outputs the newly
|
||||||
|
created public key. You must upload this key to gerrit (<b>both</b>
|
||||||
|
<a href="https://review.opendev.org/#/settings/ssh-keys">opendev.org</a> as well
|
||||||
|
as <a href="https://review.rdoproject.org/r/#/settings/ssh-keys">rdoproject.org</a>).
|
||||||
|
</p>
|
||||||
|
<hr></hr>
|
||||||
|
<h1>Pet</h1>
|
||||||
|
|
||||||
|
If you are running the reproducer on a non-transient machine, that is your
|
||||||
|
laptop or any other box which isn't temporary (a pet), then you need:
|
||||||
|
|
||||||
<ul>
|
<ul>
|
||||||
<li>Create a clouds.yaml - for example:
|
<li> Ensure user running reproducer has passwordless sudo </li>
|
||||||
|
<li> That user has $HOME/.config/openstack/clouds.yaml with "rdo-cloud" entry </li>
|
||||||
|
<li> A ssh keypair generated with <b><code>-m PEM -t rsa</code></b> and uploaded
|
||||||
|
to gerrit (<b>both </b><a href="https://review.opendev.org/#/settings/ssh-keys">opendev.org</a>
|
||||||
|
as well as <a href="https://review.rdoproject.org/r/#/settings/ssh-keys">rdoproject.org</a>).
|
||||||
|
</li>
|
||||||
|
<li> A externally routable network in your RDO-cloud tenant named 'private'.
|
||||||
|
See example commands above if you need to create this or log in to
|
||||||
|
<a href="https://phx2.cloud.rdoproject.org/dashboard/auth/login/?next=/dashboard/project/networks/"> RDO cloud horizon</a> and verify your setup or create as needed.
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
You only need to do these things once on your pet and then rerun reproducer or
|
||||||
|
update as needed.
|
||||||
|
|
||||||
|
<ul>
|
||||||
|
<li>Example clouds.yaml:</li>
|
||||||
<pre>
|
<pre>
|
||||||
<code>mkdir -p ~/.config/openstack
|
<code>CLOUDS_YAML_PATH=$HOME/.config/openstack/
|
||||||
vi ~/.config/openstack/clouds.yaml
|
mkdir -p $CLOUDS_YAML_PATH
|
||||||
|
/bin/bash -c "cat <<EOF>>$CLOUDS_YAML_PATH/clouds.yaml
|
||||||
clouds:
|
clouds:
|
||||||
rdo-cloud:
|
rdo-cloud:
|
||||||
identity_api_version: 3
|
identity_api_version: 3
|
||||||
region_name: regionOne
|
region_name: regionOne
|
||||||
auth:
|
auth:
|
||||||
auth_url: https://phx2.cloud.rdoproject.org:13000/v3
|
auth_url: https://phx2.cloud.rdoproject.org:13000/v3
|
||||||
password: xxxxxxx
|
password: $RDO_PASS
|
||||||
project_name: user_name
|
project_name: $RDO_USER
|
||||||
username: user_name
|
username: $RDO_USER
|
||||||
user_domain_name: Default
|
user_domain_name: Default
|
||||||
project_domain_name: Default
|
project_domain_name: Default
|
||||||
regions:
|
regions:
|
||||||
- name: regionOne
|
- name: regionOne
|
||||||
values:
|
values:
|
||||||
networks:
|
networks:
|
||||||
- name: 38.145.32.0/22
|
- name: 38.145.32.0/22
|
||||||
routes_externally: true
|
routes_externally: true
|
||||||
- name: private
|
- name: private
|
||||||
|
EOF"
|
||||||
</code>
|
</code>
|
||||||
</pre>
|
</pre>
|
||||||
</ul>
|
</ul>
|
||||||
<ul>
|
<ul>
|
||||||
<li> Test your cloud.yaml by running:
|
<br/>
|
||||||
<pre>
|
<li> Create an ssh key pair without a pass phrase:</li>
|
||||||
<code>
|
|
||||||
openstack --os-cloud rdo-cloud image list
|
|
||||||
</code>
|
|
||||||
</pre>
|
|
||||||
</ul>
|
|
||||||
<ul>
|
|
||||||
<li>Note that you will need a network named 'private' in the tenant connected to an
|
|
||||||
router that is connected to the external network.</li>
|
|
||||||
<li>Create an ssh key pair without a pass phrase:</li>
|
|
||||||
<ul>
|
<ul>
|
||||||
<li>The ssh key pair is configurable - see variables <code>ssh_path</code></li>
|
<li>The ssh key pair is configurable - see variables <code>ssh_path</code>
|
||||||
<li><code>user_pri_key</code>. The default key is:<code>~/.ssh/id_rsa[.pub]</code></li>
|
<code>user_pri_key</code>. The default key is:<code>~/.ssh/id_rsa[.pub]</code></li>
|
||||||
<li> <b>Note:</b> Reference FAQ below when creating ssh keys, some keys fail when used w/ gerrit
|
<li> <b>Note:</b> Use the following to create keys: <b><code> ssh-keygen -m PEM -t rsa </code></b>
|
||||||
|
See FAQ below for more information about ssh keys requirements.
|
||||||
</ul>
|
</ul>
|
||||||
</p>
|
</ul>
|
||||||
|
</br>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<button class="collapsible">Running the Zuul-based reproducer - using the Bash script</button>
|
<button class="collapsible">Running the Zuul-based reproducer - using the Bash script</button>
|
||||||
<div class="content">
|
<div class="content">
|
||||||
<p>
|
<p>
|
||||||
<ul>
|
wget or curl the <b>reproducer-quickstart/reproducer-zuul-based-quickstart.tar</b> file.
|
||||||
<li> wget or curl the <b>reproducer-quickstart/reproducer-zuul-based-quickstart.tar</b> file. </li>
|
<pre><code><b>
|
||||||
<li> Note that the <code>launcher-env-setup-playbook.yaml</code> will set up
|
wget http://logs.openstack.org/79/657779/2/check/tripleo-ci-centos-7-standalone-upgrade/92fd476/logs/reproducer-quickstart/reproducer-zuul-based-quickstart.tar
|
||||||
clone repos and set up 'roles', 'playbooks', and 'library' directories
|
tar -xvf reproducer-zuul-based-quickstart.tar
|
||||||
from which the tasks will be executed.</li>
|
</b></code></pre>
|
||||||
<li> Once the files are extracted from the archive, you can run the Bash script
|
One of the extracted files is the script called <code>./reproducer-zuul-based-quickstart.sh</code>
|
||||||
with the various options. Run the script with <code>--help</code> option to
|
and you can run it with <code>--help</code> to see available options.
|
||||||
output a list of available run options. </li>
|
</br>
|
||||||
</ul>
|
</br>
|
||||||
<b>Note:</b> the local $USERNAME idealy matches the openstack and rdo gerrit user id's
|
<b>Note:</b> If the local $USERNAME does not match the gerrit user IDs for
|
||||||
If not pay extra attention to the -ug and -rg options.
|
<a href="https://review.opendev.org/#/settings/ssh-keys">opendev.org</a> or
|
||||||
<ul>
|
<a href="https://review.rdoproject.org/r/#/settings/ssh-keys">rdoproject.org</a>)
|
||||||
<code>
|
then you will need to supply the -ug (upstream gerrit) and -rg (rdo gerrit)
|
||||||
<pre>
|
parameters:
|
||||||
Options:
|
|
||||||
-w, --workspace dir
|
|
||||||
directory where the repos and directories
|
|
||||||
are created. Defaults to creating a directory in /tmp
|
|
||||||
-l, --libvirt
|
|
||||||
Runs a 2-node multinode job or singlenode job on a
|
|
||||||
single virthost using libvirt to create the nodes.
|
|
||||||
If a singlenode reproducer is run, two VMs will still be created.
|
|
||||||
-c, --cloud-name
|
|
||||||
Host cloud, specified in the clouds.yaml, to target
|
|
||||||
Defaults to rdo-cloud
|
|
||||||
-cp, --clouds-yaml-path
|
|
||||||
Full path to the clouds.yaml file
|
|
||||||
Defaults to /home/$USERNAME/.config/openstack/clouds.yaml
|
|
||||||
-ok, --ovb-key-name
|
|
||||||
Name of the key to use in the host tenant for OVB deployments
|
|
||||||
Defaults to tripleo-ci-team
|
|
||||||
-f, --force-post-failure
|
|
||||||
Force job to fail so that the nodes will be held.
|
|
||||||
Temporary solution
|
|
||||||
-ug, --upstream-gerrit-user
|
|
||||||
Set the upstream gerrit user required to clone repos.
|
|
||||||
Defaults to the current $USERNAME
|
|
||||||
-rg, --rdo-gerrit-user
|
|
||||||
Set the upstream RDO user required to clone RDO-related repos.
|
|
||||||
Defaults to the current $USERNAME
|
|
||||||
-k, --ssh-key
|
|
||||||
private ssh key used to set up an access nodes.
|
|
||||||
Defaults to id_rsa.
|
|
||||||
-kp, --ssh-key-public
|
|
||||||
public ssh key used to set up an access nodes.
|
|
||||||
Defaults to id_rsa.pub
|
|
||||||
-skp, --ssh-key-path
|
|
||||||
Path to directory where user ssh keys are stored.
|
|
||||||
Defaults to /home/$USERNAME/.ssh
|
|
||||||
-ugk, --upstream-gerrit-key
|
|
||||||
Set the upstream gerrit private key.
|
|
||||||
Defaults to the id_rsa.
|
|
||||||
-rgk, --rdo-gerrit-key
|
|
||||||
Set the upstream RDO user key.
|
|
||||||
Defaults to the id_rsa.
|
|
||||||
-e, --extra-params
|
|
||||||
File or/and parameters used to override default
|
|
||||||
parameters for playbooks. Multiple files
|
|
||||||
can be passed [-e @file1.yml -e @file2.yml ...]
|
|
||||||
and arguments [-e var=value -e var2=value2 ...]
|
|
||||||
-h, --help print this help and exit
|
|
||||||
|
|
||||||
</code>
|
<ul>
|
||||||
</pre>
|
|
||||||
</ul>
|
</ul>
|
||||||
|
<pre><code><b>
|
||||||
|
mkdir WORKSPACE
|
||||||
|
./reproducer-zuul-based-quickstart.sh --workspace WORKSPACE -ug slim_shady -rg slim_shadier
|
||||||
|
</b></code></pre>
|
||||||
</p>
|
</p>
|
||||||
|
Monitor the output and follow any instructions - for example the script will
|
||||||
|
exit and require re-run once the user is added to the docker group.
|
||||||
|
</br>
|
||||||
|
</br>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<button class="collapsible">Running the Zuul-based reproducer - using the playbooks directly</button>
|
<button class="collapsible">Running the Zuul-based reproducer - using the playbooks directly</button>
|
||||||
@ -183,9 +210,12 @@ The playbooks can be run directly using <code>ansible-playbook</code>.
|
|||||||
The Bash script contains some package installation and Docker user/group setup
|
The Bash script contains some package installation and Docker user/group setup
|
||||||
that you will need to cover prior to running the playbooks directly.
|
that you will need to cover prior to running the playbooks directly.
|
||||||
Please see the Bash scripts for details, especially if you are running the
|
Please see the Bash scripts for details, especially if you are running the
|
||||||
reproducer for the first time on teh local environment.
|
reproducer for the first time on the local environment.
|
||||||
The playbooks can be run as follows:
|
The playbooks can be run as follows:
|
||||||
<ul>
|
<ul>
|
||||||
|
<li> Note that the <code>launcher-env-setup-playbook.yaml</code> will clone
|
||||||
|
repos and set up 'roles', 'playbooks', and 'library' directories
|
||||||
|
from which the tasks will be executed. </li>
|
||||||
<li><code>ansible-playbook ./launcher-env-setup-playbook.yaml</code></li>
|
<li><code>ansible-playbook ./launcher-env-setup-playbook.yaml</code></li>
|
||||||
<li><code>ansible-playbook ./launcher-playbook.yaml $OPTIONS</code></li>
|
<li><code>ansible-playbook ./launcher-playbook.yaml $OPTIONS</code></li>
|
||||||
</ul>
|
</ul>
|
||||||
|
|||||||
@ -0,0 +1,127 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
REPRO_USER=${REPRO_USER:=reprozuul}
|
||||||
|
REPRO_DEPS="tar wget screen"
|
||||||
|
RDO_USER=""
|
||||||
|
RDO_PASS=""
|
||||||
|
CLOUDS_YAML_PATH=".config/openstack/"
|
||||||
|
usage () {
|
||||||
|
echo "Usage: $0 [options]"
|
||||||
|
echo ""
|
||||||
|
echo "Options:"
|
||||||
|
echo " -u, --rdo-user"
|
||||||
|
echo " Username for RDO cloud to be written to"
|
||||||
|
echo " $CLOUDS_YAML_PATH/clouds.yaml. REQUIRED."
|
||||||
|
echo " -p, --rdo-pass"
|
||||||
|
echo " Password for RDO cloud to be written to"
|
||||||
|
echo " $CLOUDS_YAML_PATH/clouds.yaml. REQUIRED."
|
||||||
|
echo " -c, --create-user"
|
||||||
|
echo " Local user to create for running reproducer."
|
||||||
|
echo " Defaults to $REPRO_USER"
|
||||||
|
echo " -h, --help print this help and exit"
|
||||||
|
}
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# Input argument assignments
|
||||||
|
while [ "x$1" != "x" ]; do
|
||||||
|
|
||||||
|
case "$1" in
|
||||||
|
--rdo-user|-u)
|
||||||
|
RDO_USER=$2
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
|
||||||
|
--rdo-pass|-p)
|
||||||
|
RDO_PASS=$2
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
|
||||||
|
--create-user|-c)
|
||||||
|
REPRO_USER=$2
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
|
||||||
|
--help|-h)
|
||||||
|
usage
|
||||||
|
exit
|
||||||
|
;;
|
||||||
|
|
||||||
|
--) shift
|
||||||
|
break
|
||||||
|
;;
|
||||||
|
|
||||||
|
-*) echo "ERROR: unknown option: $1" >&2
|
||||||
|
usage >&2
|
||||||
|
exit 2
|
||||||
|
;;
|
||||||
|
|
||||||
|
*) break
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
shift
|
||||||
|
done
|
||||||
|
|
||||||
|
# check required params - rdo user/pass
|
||||||
|
for param in "$RDO_USER" "$RDO_PASS" ; do
|
||||||
|
if [[ $param = "" ]]; then
|
||||||
|
echo "error --rdo-user and --rdo-pass are required parameters"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# create user
|
||||||
|
sudo adduser $REPRO_USER
|
||||||
|
echo "$REPRO_USER ALL=(root) NOPASSWD:ALL" | sudo tee -a /etc/sudoers.d/$REPRO_USER
|
||||||
|
|
||||||
|
# install packages
|
||||||
|
sudo yum -y install $REPRO_DEPS
|
||||||
|
|
||||||
|
# write clouds.yaml
|
||||||
|
CLOUDS_YAML_PATH="/home/$REPRO_USER/.config/openstack/"
|
||||||
|
sudo mkdir -p $CLOUDS_YAML_PATH
|
||||||
|
sudo /bin/bash -c "cat <<EOF>>$CLOUDS_YAML_PATH/clouds.yaml
|
||||||
|
clouds:
|
||||||
|
rdo-cloud:
|
||||||
|
identity_api_version: 3
|
||||||
|
region_name: regionOne
|
||||||
|
auth:
|
||||||
|
auth_url: https://phx2.cloud.rdoproject.org:13000/v3
|
||||||
|
password: $RDO_PASS
|
||||||
|
project_name: $RDO_USER
|
||||||
|
username: $RDO_USER
|
||||||
|
user_domain_name: Default
|
||||||
|
project_domain_name: Default
|
||||||
|
regions:
|
||||||
|
- name: regionOne
|
||||||
|
values:
|
||||||
|
networks:
|
||||||
|
- name: 38.145.32.0/22
|
||||||
|
routes_externally: true
|
||||||
|
- name: private
|
||||||
|
EOF"
|
||||||
|
|
||||||
|
sudo chown -R $REPRO_USER:$REPRO_USER /home/$REPRO_USER/.config
|
||||||
|
|
||||||
|
# create keys
|
||||||
|
mkdir keystemp
|
||||||
|
pushd keystemp
|
||||||
|
ssh-keygen -m PEM -t rsa -N "" -f id_rsa
|
||||||
|
ssh_path=/home/$REPRO_USER/.ssh
|
||||||
|
sudo mkdir $ssh_path
|
||||||
|
sudo mv id_rsa* $ssh_path
|
||||||
|
sudo chmod 700 $ssh_path
|
||||||
|
sudo chmod 600 $ssh_path/id_rsa
|
||||||
|
sudo chmod 644 $ssh_path/id_rsa.pub
|
||||||
|
sudo chown -R $REPRO_USER:$REPRO_USER $ssh_path
|
||||||
|
popd
|
||||||
|
rm -rf keystemp
|
||||||
|
|
||||||
|
echo "Fresh box setup complete - now go upload the public key in"
|
||||||
|
echo "$CLOUDS_YAML_PATH to Gerrit - both upstream and rdo. Then you are ready "
|
||||||
|
echo "to run the reproducer. Here is the public key you must upload:"
|
||||||
|
sudo cat /home/$REPRO_USER/.ssh/id_rsa.pub
|
||||||
|
echo "Switching to $REPRO_USER"
|
||||||
|
sudo su - $REPRO_USER
|
||||||
|
|
||||||
Loading…
Reference in New Issue
Block a user