openstack/project-config
Code Issues Proposed changes
b51894c602
project-config/nodepool/scripts/prepare_node_devstack.sh

45 lines
1.4 KiB
Bash
Raw Normal View History

Add nodepool host Change-Id: Ib216bd400aa269ccdb17848b3870ab65400cf3d2
2013-08-15 17:52:52 +00:00
#!/bin/bash -xe
# Copyright (C) 2011-2013 OpenStack Foundation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
#
# See the License for the specific language governing permissions and
# limitations under the License.
HOSTNAME=$1
Export prepare_node variables rather than pass them around The prepare_node scripts are primarily called from within nodepool, which only passes in a single argument: the hostname of the node. The rest of the values that we want to pass in to the underlying prepare_node script can be shared through the environment, making order no longer important and alleviating the need to duplicate setting defaults in order to pad positional arguments to a newly created variable. Change-Id: I0a13e27524b75a50a482870414ebe2fd6ac8315e
2014-07-02 16:45:56 -07:00
export SUDO='true'
export THIN='true'
Correct tempest clone location in node prep Change-Id: I4208637457bbf1e7562f358913ff71b9f3b0732d
2015-01-12 17:25:06 +00:00
TEMPEST_DIR=${TEMPEST_DIR:-/opt/git/openstack/tempest}
Export prepare_node variables rather than pass them around The prepare_node scripts are primarily called from within nodepool, which only passes in a single argument: the hostname of the node. The rest of the values that we want to pass in to the underlying prepare_node script can be shared through the environment, making order no longer important and alleviating the need to duplicate setting defaults in order to pad positional arguments to a newly created variable. Change-Id: I0a13e27524b75a50a482870414ebe2fd6ac8315e
2014-07-02 16:45:56 -07:00
./prepare_node.sh "$HOSTNAME"
sudo -u jenkins -i /opt/nodepool-scripts/prepare_devstack.sh "$HOSTNAME"
Add sync calls to all prepare scripts Strictly speaking, we only need sync calls at the end of the script that is directly invoked by nodepool. But just so we don't have to think about it too much, add them to all prepare scripts. Without it, clouds will happily snapshot a system without all the data written to disk! Closes-Bug: 1282136 Change-Id: Id7dcf6667d6e48eb5ae95fd42b09f0651eeb38f9
2014-02-19 09:03:19 -08:00
Add nodepool script to preseed testrepository from subunit2sql This commit adds a new nodepool script to use subunit2sql to preseed testrepository with timing data to use for scheduler optimization. This uses aggregate data from all the tests stored in the subunit2sql db from all the test runs to generate a subunit stream with test execute times which equal the rolling average stored in the tests table. This also adds a call out to the new prepare_tempest_testrepository script in order to pre-seed tempest's testrepository with data from the subunit2sql DB. This will enable the testr scheduler to perform some worker balance optimization based on the average run_times from all the previous gate runs. Change-Id: I04404a55baf4da2745760246b8606ed3db52c03a
2014-11-20 23:01:37 -05:00
# Setup venv and install deps for prepare_tempest_testrepository.py
sudo virtualenv -p python2 /opt/git/subunit2sql-env
Wrap some overly long lines To cut down on noise when running bashate, wrap some overlong lines. Change-Id: I23dcffbae2539d3d6b11e5527f3d4cbf08a18064
2015-12-23 13:36:33 +01:00
sudo -H /opt/git/subunit2sql-env/bin/pip install -U testrepository \
subunit2sql PyMySQL
Add nodepool script to preseed testrepository from subunit2sql This commit adds a new nodepool script to use subunit2sql to preseed testrepository with timing data to use for scheduler optimization. This uses aggregate data from all the tests stored in the subunit2sql db from all the test runs to generate a subunit stream with test execute times which equal the rolling average stored in the tests table. This also adds a call out to the new prepare_tempest_testrepository script in order to pre-seed tempest's testrepository with data from the subunit2sql DB. This will enable the testr scheduler to perform some worker balance optimization based on the average run_times from all the previous gate runs. Change-Id: I04404a55baf4da2745760246b8606ed3db52c03a
2014-11-20 23:01:37 -05:00
# Pre-seed tempest testrepository with data from subunit2sql
Wrap some overly long lines To cut down on noise when running bashate, wrap some overlong lines. Change-Id: I23dcffbae2539d3d6b11e5527f3d4cbf08a18064
2015-12-23 13:36:33 +01:00
sudo -i env PATH=/opt/git/subunit2sql-env/bin:$PATH \
/opt/git/subunit2sql-env/bin/python2 \
/opt/nodepool-scripts/prepare_tempest_testrepository.py \
$TEMPEST_DIR
Add nodepool script to preseed testrepository from subunit2sql This commit adds a new nodepool script to use subunit2sql to preseed testrepository with timing data to use for scheduler optimization. This uses aggregate data from all the tests stored in the subunit2sql db from all the test runs to generate a subunit stream with test execute times which equal the rolling average stored in the tests table. This also adds a call out to the new prepare_tempest_testrepository script in order to pre-seed tempest's testrepository with data from the subunit2sql DB. This will enable the testr scheduler to perform some worker balance optimization based on the average run_times from all the previous gate runs. Change-Id: I04404a55baf4da2745760246b8606ed3db52c03a
2014-11-20 23:01:37 -05:00
Preseed with a number of samples from db instead of aggregate Because of the testrepository bug 1416512 using the aggregate stream generated by sql2subunit the timing data would be ignored by the testr scheduler. The long term fix is to address this in testrepository eventually. But, in the short to get any utility out of the preseed data this commit switches the preseed to not use the aggregate view stream and instead take the 10 most recent runs and use those as the preseed data. While this data isn't as good for performing the optimization, but it will at least use the data. Change-Id: Ib0e815ca1966f5b5806fb257b91ac7a25d4db18d
2015-01-30 13:27:53 -05:00
sudo chown -R jenkins:jenkins $TEMPEST_DIR/preseed-streams
Set proper ownership on tempest test repository When the testrepository directory is created by calling the preseed script the permissions are different from the rest of the tempest repo. When the dir is rsync'd by devstack-gate it errors out with permission denied on the .testrepository dir. This commit fixes this by setting the ownership to be the same as the rest of the cached repo. Change-Id: Ie12f3f0d824401f2ed53a77f8f307ea0f178854e
2015-01-23 16:02:19 -05:00
Fix a typo in prepare_node_devstack.sh Change-Id: I763b6efe57d37c1e2a1da299b9a2a0c32e0df32f
2015-08-04 02:28:37 -04:00
# Delete the venv after the script is called
Add nodepool script to preseed testrepository from subunit2sql This commit adds a new nodepool script to use subunit2sql to preseed testrepository with timing data to use for scheduler optimization. This uses aggregate data from all the tests stored in the subunit2sql db from all the test runs to generate a subunit stream with test execute times which equal the rolling average stored in the tests table. This also adds a call out to the new prepare_tempest_testrepository script in order to pre-seed tempest's testrepository with data from the subunit2sql DB. This will enable the testr scheduler to perform some worker balance optimization based on the average run_times from all the previous gate runs. Change-Id: I04404a55baf4da2745760246b8606ed3db52c03a
2014-11-20 23:01:37 -05:00
sudo rm -rf /opt/git/subunit2sql-env
Limit non tripleo nodepool nodes to 8GB of RAM Limit the non tripleo nodepool nodes to 8GB of RAM regardless of the flavor used. This allows us to use arbitrary flavors that may have the CPU resources that we require without risk that we become dependent on the vast amounts of RAM that typically come with more CPU. While we are at it set grub timeout to 0 saving 5 seconds of boottime for every server we boot. Change-Id: Ia38341239e0189c84a21a4f58dd7ca637490a27b
2014-03-05 14:33:39 -08:00
./restrict_memory.sh
Copy Permalink