93 lines
3.3 KiB
Bash
Raw Normal View History

Implement local scripts for all gate tests This patch implements the following: - scripts-library.sh which includes commonly used functions, variables and other preparation commands for all other scripts - bootstrap-ansible.sh which only deploys a selected version of ansible and ensures that any other requirements are prepared on the deployment host - bootstrap-aio.sh which runs all host preparation actions for an all-in-one build - gate-check-lint.sh which runs a lint and syntax check - gate-check-commit.sh which runs all actions required for a gate commit check, utilising the other scripts where required - run-smoke-test.sh which runs tempest from inside the utility container - run-playbooks.sh which runs the playbooks - the existing conf.d/swift.yml is renamed to be an example configuration - the example configurations can be used as documentation - etc/network/interfaces.d/aio_interfaces.cfg, etc/rpc_deploy/conf.d/swift.yml and etc/rpc_deploy/rpc_user_config.yml are now configurations used for the AIO deployment - a workaround for https://bugs.launchpad.net/bugs/1244589 to ensure that DHCP checksums are implemented by the host which is required for the smoke tests to work - the removal of the rpc heat templates as they're unusable in their current state - setting MAX_RETRIES to 0, ensuring that any failures cause an immediate commit check failure in the gate - this prevents the masking of failures by retry attempts DocImpact Co-Authored-By: Kevin Carter <kevin.carter@rackspace.com> Closes-Bug: #1415883 Closes-Bug: #1417999 Closes-Bug: #1419807 Change-Id: I95242d48ad0fb055f16510803c8aa14dc183ac17
2015-02-04 15:51:06 +00:00
#!/usr/bin/env bash
# Copyright 2014, Rackspace US, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
## Shell Opts ----------------------------------------------------------------
set -e -u -x
Implement local scripts for all gate tests This patch implements the following: - scripts-library.sh which includes commonly used functions, variables and other preparation commands for all other scripts - bootstrap-ansible.sh which only deploys a selected version of ansible and ensures that any other requirements are prepared on the deployment host - bootstrap-aio.sh which runs all host preparation actions for an all-in-one build - gate-check-lint.sh which runs a lint and syntax check - gate-check-commit.sh which runs all actions required for a gate commit check, utilising the other scripts where required - run-smoke-test.sh which runs tempest from inside the utility container - run-playbooks.sh which runs the playbooks - the existing conf.d/swift.yml is renamed to be an example configuration - the example configurations can be used as documentation - etc/network/interfaces.d/aio_interfaces.cfg, etc/rpc_deploy/conf.d/swift.yml and etc/rpc_deploy/rpc_user_config.yml are now configurations used for the AIO deployment - a workaround for https://bugs.launchpad.net/bugs/1244589 to ensure that DHCP checksums are implemented by the host which is required for the smoke tests to work - the removal of the rpc heat templates as they're unusable in their current state - setting MAX_RETRIES to 0, ensuring that any failures cause an immediate commit check failure in the gate - this prevents the masking of failures by retry attempts DocImpact Co-Authored-By: Kevin Carter <kevin.carter@rackspace.com> Closes-Bug: #1415883 Closes-Bug: #1417999 Closes-Bug: #1419807 Change-Id: I95242d48ad0fb055f16510803c8aa14dc183ac17
2015-02-04 15:51:06 +00:00
## Variables -----------------------------------------------------------------
Convert existing roles into galaxy roles This change implements the blueprint to convert all roles and plays into a more generic setup, following upstream ansible best practices. Items Changed: * All tasks have tags. * All roles use namespaced variables. * All redundant tasks within a given play and role have been removed. * All of the repetitive plays have been removed in-favor of a more simplistic approach. This change duplicates code within the roles but ensures that the roles only ever run within their own scope. * All roles have been built using an ansible galaxy syntax. * The `*requirement.txt` files have been reformatted follow upstream Openstack practices. * Dynamically generated inventory is now more organized, this should assist anyone who may want or need to dive into the JSON blob that is created. In the inventory a properties field is used for items that customize containers within the inventory. * The environment map has been modified to support additional host groups to enable the seperation of infrastructure pieces. While the old infra_hosts group will still work this change allows for groups to be divided up into seperate chunks; eg: deployment of a swift only stack. * The LXC logic now exists within the plays. * etc/openstack_deploy/user_variables.yml has all password/token variables extracted into the separate file etc/openstack_deploy/user_secrets.yml in order to allow seperate security settings on that file. Items Excised: * All of the roles have had the LXC logic removed from within them which should allow roles to be consumed outside of the `os-ansible-deployment` reference architecture. Note: * the directory rpc_deployment still exists and is presently pointed at plays containing a deprecation warning instructing the user to move to the standard playbooks directory. * While all of the rackspace specific components and variables have been removed and or were refactored the repository still relies on an upstream mirror of Openstack built python files and container images. This upstream mirror is hosted at rackspace at "http://rpc-repo.rackspace.com" though this is not locked to and or tied to rackspace specific installations. This repository contains all of the needed code to create and/or clone your own mirror. DocImpact Co-Authored-By: Jesse Pretorius <jesse.pretorius@rackspace.co.uk> Closes-Bug: #1403676 Implements: blueprint galaxy-roles Change-Id: I03df3328b7655f0cc9e43ba83b02623d038d214e
2015-02-14 10:06:50 -06:00
export BOOTSTRAP_ANSIBLE=${BOOTSTRAP_ANSIBLE:-"yes"}
export BOOTSTRAP_AIO=${BOOTSTRAP_AIO:-"yes"}
export RUN_PLAYBOOKS=${RUN_PLAYBOOKS:-"yes"}
export RUN_TEMPEST=${RUN_TEMPEST:-"yes"}
# Ansible options
export ANSIBLE_PARAMETERS=${ANSIBLE_PARAMETERS:-"-v"}
# Deployment options
export DEPLOY_HOST=${DEPLOY_HOST:-"yes"}
export DEPLOY_LB=${DEPLOY_LB:-"yes"}
export DEPLOY_INFRASTRUCTURE=${DEPLOY_INFRASTRUCTURE:-"yes"}
export DEPLOY_LOGGING=${DEPLOY_LOGGING:-"yes"}
export DEPLOY_OPENSTACK=${DEPLOY_OPENSTACK:-"yes"}
export DEPLOY_SWIFT=${DEPLOY_SWIFT:-"yes"}
export DEPLOY_TEMPEST=${DEPLOY_TEMPEST:-"yes"}
# Limit the gate check to only performing one attempt, unless already set
export MAX_RETRIES=${MAX_RETRIES:-"2"}
# tempest and testr options, default is to run tempest in serial
export RUN_TEMPEST_OPTS=${RUN_TEMPEST_OPTS:-'--serial'}
export TESTR_OPTS=${TESTR_OPTS:-''}
Implement local scripts for all gate tests This patch implements the following: - scripts-library.sh which includes commonly used functions, variables and other preparation commands for all other scripts - bootstrap-ansible.sh which only deploys a selected version of ansible and ensures that any other requirements are prepared on the deployment host - bootstrap-aio.sh which runs all host preparation actions for an all-in-one build - gate-check-lint.sh which runs a lint and syntax check - gate-check-commit.sh which runs all actions required for a gate commit check, utilising the other scripts where required - run-smoke-test.sh which runs tempest from inside the utility container - run-playbooks.sh which runs the playbooks - the existing conf.d/swift.yml is renamed to be an example configuration - the example configurations can be used as documentation - etc/network/interfaces.d/aio_interfaces.cfg, etc/rpc_deploy/conf.d/swift.yml and etc/rpc_deploy/rpc_user_config.yml are now configurations used for the AIO deployment - a workaround for https://bugs.launchpad.net/bugs/1244589 to ensure that DHCP checksums are implemented by the host which is required for the smoke tests to work - the removal of the rpc heat templates as they're unusable in their current state - setting MAX_RETRIES to 0, ensuring that any failures cause an immediate commit check failure in the gate - this prevents the masking of failures by retry attempts DocImpact Co-Authored-By: Kevin Carter <kevin.carter@rackspace.com> Closes-Bug: #1415883 Closes-Bug: #1417999 Closes-Bug: #1419807 Change-Id: I95242d48ad0fb055f16510803c8aa14dc183ac17
2015-02-04 15:51:06 +00:00
## Functions -----------------------------------------------------------------
Convert existing roles into galaxy roles This change implements the blueprint to convert all roles and plays into a more generic setup, following upstream ansible best practices. Items Changed: * All tasks have tags. * All roles use namespaced variables. * All redundant tasks within a given play and role have been removed. * All of the repetitive plays have been removed in-favor of a more simplistic approach. This change duplicates code within the roles but ensures that the roles only ever run within their own scope. * All roles have been built using an ansible galaxy syntax. * The `*requirement.txt` files have been reformatted follow upstream Openstack practices. * Dynamically generated inventory is now more organized, this should assist anyone who may want or need to dive into the JSON blob that is created. In the inventory a properties field is used for items that customize containers within the inventory. * The environment map has been modified to support additional host groups to enable the seperation of infrastructure pieces. While the old infra_hosts group will still work this change allows for groups to be divided up into seperate chunks; eg: deployment of a swift only stack. * The LXC logic now exists within the plays. * etc/openstack_deploy/user_variables.yml has all password/token variables extracted into the separate file etc/openstack_deploy/user_secrets.yml in order to allow seperate security settings on that file. Items Excised: * All of the roles have had the LXC logic removed from within them which should allow roles to be consumed outside of the `os-ansible-deployment` reference architecture. Note: * the directory rpc_deployment still exists and is presently pointed at plays containing a deprecation warning instructing the user to move to the standard playbooks directory. * While all of the rackspace specific components and variables have been removed and or were refactored the repository still relies on an upstream mirror of Openstack built python files and container images. This upstream mirror is hosted at rackspace at "http://rpc-repo.rackspace.com" though this is not locked to and or tied to rackspace specific installations. This repository contains all of the needed code to create and/or clone your own mirror. DocImpact Co-Authored-By: Jesse Pretorius <jesse.pretorius@rackspace.co.uk> Closes-Bug: #1403676 Implements: blueprint galaxy-roles Change-Id: I03df3328b7655f0cc9e43ba83b02623d038d214e
2015-02-14 10:06:50 -06:00
info_block "Checking for required libraries." 2> /dev/null || source $(dirname ${0})/scripts-library.sh
Implement local scripts for all gate tests This patch implements the following: - scripts-library.sh which includes commonly used functions, variables and other preparation commands for all other scripts - bootstrap-ansible.sh which only deploys a selected version of ansible and ensures that any other requirements are prepared on the deployment host - bootstrap-aio.sh which runs all host preparation actions for an all-in-one build - gate-check-lint.sh which runs a lint and syntax check - gate-check-commit.sh which runs all actions required for a gate commit check, utilising the other scripts where required - run-smoke-test.sh which runs tempest from inside the utility container - run-playbooks.sh which runs the playbooks - the existing conf.d/swift.yml is renamed to be an example configuration - the example configurations can be used as documentation - etc/network/interfaces.d/aio_interfaces.cfg, etc/rpc_deploy/conf.d/swift.yml and etc/rpc_deploy/rpc_user_config.yml are now configurations used for the AIO deployment - a workaround for https://bugs.launchpad.net/bugs/1244589 to ensure that DHCP checksums are implemented by the host which is required for the smoke tests to work - the removal of the rpc heat templates as they're unusable in their current state - setting MAX_RETRIES to 0, ensuring that any failures cause an immediate commit check failure in the gate - this prevents the masking of failures by retry attempts DocImpact Co-Authored-By: Kevin Carter <kevin.carter@rackspace.com> Closes-Bug: #1415883 Closes-Bug: #1417999 Closes-Bug: #1419807 Change-Id: I95242d48ad0fb055f16510803c8aa14dc183ac17
2015-02-04 15:51:06 +00:00
## Main ----------------------------------------------------------------------
# Disable Ansible color output
sed -i 's/nocolor.*/nocolor = 1/' $(dirname ${0})/../playbooks/ansible.cfg
# Make the /openstack/log directory for openstack-infra gate check log publishing
mkdir -p /openstack/log
# Implement the log directory link for openstack-infra log publishing
ln -sf /openstack/log $(dirname ${0})/../logs
# Create ansible logging directory and add in a log file entry into ansible.cfg
mkdir -p /openstack/log/ansible-logging
sed -i '/\[defaults\]/a log_path = /openstack/log/ansible-logging/ansible.log' $(dirname ${0})/../playbooks/ansible.cfg
Implement local scripts for all gate tests This patch implements the following: - scripts-library.sh which includes commonly used functions, variables and other preparation commands for all other scripts - bootstrap-ansible.sh which only deploys a selected version of ansible and ensures that any other requirements are prepared on the deployment host - bootstrap-aio.sh which runs all host preparation actions for an all-in-one build - gate-check-lint.sh which runs a lint and syntax check - gate-check-commit.sh which runs all actions required for a gate commit check, utilising the other scripts where required - run-smoke-test.sh which runs tempest from inside the utility container - run-playbooks.sh which runs the playbooks - the existing conf.d/swift.yml is renamed to be an example configuration - the example configurations can be used as documentation - etc/network/interfaces.d/aio_interfaces.cfg, etc/rpc_deploy/conf.d/swift.yml and etc/rpc_deploy/rpc_user_config.yml are now configurations used for the AIO deployment - a workaround for https://bugs.launchpad.net/bugs/1244589 to ensure that DHCP checksums are implemented by the host which is required for the smoke tests to work - the removal of the rpc heat templates as they're unusable in their current state - setting MAX_RETRIES to 0, ensuring that any failures cause an immediate commit check failure in the gate - this prevents the masking of failures by retry attempts DocImpact Co-Authored-By: Kevin Carter <kevin.carter@rackspace.com> Closes-Bug: #1415883 Closes-Bug: #1417999 Closes-Bug: #1419807 Change-Id: I95242d48ad0fb055f16510803c8aa14dc183ac17
2015-02-04 15:51:06 +00:00
# Bootstrap an AIO setup if required
if [ "${BOOTSTRAP_AIO}" == "yes" ]; then
source $(dirname ${0})/bootstrap-aio.sh
fi
Convert existing roles into galaxy roles This change implements the blueprint to convert all roles and plays into a more generic setup, following upstream ansible best practices. Items Changed: * All tasks have tags. * All roles use namespaced variables. * All redundant tasks within a given play and role have been removed. * All of the repetitive plays have been removed in-favor of a more simplistic approach. This change duplicates code within the roles but ensures that the roles only ever run within their own scope. * All roles have been built using an ansible galaxy syntax. * The `*requirement.txt` files have been reformatted follow upstream Openstack practices. * Dynamically generated inventory is now more organized, this should assist anyone who may want or need to dive into the JSON blob that is created. In the inventory a properties field is used for items that customize containers within the inventory. * The environment map has been modified to support additional host groups to enable the seperation of infrastructure pieces. While the old infra_hosts group will still work this change allows for groups to be divided up into seperate chunks; eg: deployment of a swift only stack. * The LXC logic now exists within the plays. * etc/openstack_deploy/user_variables.yml has all password/token variables extracted into the separate file etc/openstack_deploy/user_secrets.yml in order to allow seperate security settings on that file. Items Excised: * All of the roles have had the LXC logic removed from within them which should allow roles to be consumed outside of the `os-ansible-deployment` reference architecture. Note: * the directory rpc_deployment still exists and is presently pointed at plays containing a deprecation warning instructing the user to move to the standard playbooks directory. * While all of the rackspace specific components and variables have been removed and or were refactored the repository still relies on an upstream mirror of Openstack built python files and container images. This upstream mirror is hosted at rackspace at "http://rpc-repo.rackspace.com" though this is not locked to and or tied to rackspace specific installations. This repository contains all of the needed code to create and/or clone your own mirror. DocImpact Co-Authored-By: Jesse Pretorius <jesse.pretorius@rackspace.co.uk> Closes-Bug: #1403676 Implements: blueprint galaxy-roles Change-Id: I03df3328b7655f0cc9e43ba83b02623d038d214e
2015-02-14 10:06:50 -06:00
# Bootstrap ansible if required
if [ "${BOOTSTRAP_ANSIBLE}" == "yes" ]; then
source $(dirname ${0})/bootstrap-ansible.sh
fi
# Enable debug logging for all services to make failure debugging easier
echo "debug: True" | tee -a /etc/openstack_deploy/user_variables.yml
Implement local scripts for all gate tests This patch implements the following: - scripts-library.sh which includes commonly used functions, variables and other preparation commands for all other scripts - bootstrap-ansible.sh which only deploys a selected version of ansible and ensures that any other requirements are prepared on the deployment host - bootstrap-aio.sh which runs all host preparation actions for an all-in-one build - gate-check-lint.sh which runs a lint and syntax check - gate-check-commit.sh which runs all actions required for a gate commit check, utilising the other scripts where required - run-smoke-test.sh which runs tempest from inside the utility container - run-playbooks.sh which runs the playbooks - the existing conf.d/swift.yml is renamed to be an example configuration - the example configurations can be used as documentation - etc/network/interfaces.d/aio_interfaces.cfg, etc/rpc_deploy/conf.d/swift.yml and etc/rpc_deploy/rpc_user_config.yml are now configurations used for the AIO deployment - a workaround for https://bugs.launchpad.net/bugs/1244589 to ensure that DHCP checksums are implemented by the host which is required for the smoke tests to work - the removal of the rpc heat templates as they're unusable in their current state - setting MAX_RETRIES to 0, ensuring that any failures cause an immediate commit check failure in the gate - this prevents the masking of failures by retry attempts DocImpact Co-Authored-By: Kevin Carter <kevin.carter@rackspace.com> Closes-Bug: #1415883 Closes-Bug: #1417999 Closes-Bug: #1419807 Change-Id: I95242d48ad0fb055f16510803c8aa14dc183ac17
2015-02-04 15:51:06 +00:00
# Run the ansible playbooks if required
if [ "${RUN_PLAYBOOKS}" == "yes" ]; then
# Set-up our tiny awk script.
strip_debug="
!/(^[ 0-9|:.-]+<[0-9.]|localhost+>)|Extracting/ {
gsub(/{.*/, \"\");
gsub(/\\n.*/, \"\");
gsub(/\=\>.*/, \"\");
print
}
"
set -o pipefail
bash $(dirname ${0})/run-playbooks.sh | awk "${strip_debug}"
set +o pipefail
Implement local scripts for all gate tests This patch implements the following: - scripts-library.sh which includes commonly used functions, variables and other preparation commands for all other scripts - bootstrap-ansible.sh which only deploys a selected version of ansible and ensures that any other requirements are prepared on the deployment host - bootstrap-aio.sh which runs all host preparation actions for an all-in-one build - gate-check-lint.sh which runs a lint and syntax check - gate-check-commit.sh which runs all actions required for a gate commit check, utilising the other scripts where required - run-smoke-test.sh which runs tempest from inside the utility container - run-playbooks.sh which runs the playbooks - the existing conf.d/swift.yml is renamed to be an example configuration - the example configurations can be used as documentation - etc/network/interfaces.d/aio_interfaces.cfg, etc/rpc_deploy/conf.d/swift.yml and etc/rpc_deploy/rpc_user_config.yml are now configurations used for the AIO deployment - a workaround for https://bugs.launchpad.net/bugs/1244589 to ensure that DHCP checksums are implemented by the host which is required for the smoke tests to work - the removal of the rpc heat templates as they're unusable in their current state - setting MAX_RETRIES to 0, ensuring that any failures cause an immediate commit check failure in the gate - this prevents the masking of failures by retry attempts DocImpact Co-Authored-By: Kevin Carter <kevin.carter@rackspace.com> Closes-Bug: #1415883 Closes-Bug: #1417999 Closes-Bug: #1419807 Change-Id: I95242d48ad0fb055f16510803c8aa14dc183ac17
2015-02-04 15:51:06 +00:00
fi
# Run the tempest tests if required
if [ "${RUN_TEMPEST}" == "yes" ]; then
source $(dirname ${0})/run-tempest.sh
fi
Convert existing roles into galaxy roles This change implements the blueprint to convert all roles and plays into a more generic setup, following upstream ansible best practices. Items Changed: * All tasks have tags. * All roles use namespaced variables. * All redundant tasks within a given play and role have been removed. * All of the repetitive plays have been removed in-favor of a more simplistic approach. This change duplicates code within the roles but ensures that the roles only ever run within their own scope. * All roles have been built using an ansible galaxy syntax. * The `*requirement.txt` files have been reformatted follow upstream Openstack practices. * Dynamically generated inventory is now more organized, this should assist anyone who may want or need to dive into the JSON blob that is created. In the inventory a properties field is used for items that customize containers within the inventory. * The environment map has been modified to support additional host groups to enable the seperation of infrastructure pieces. While the old infra_hosts group will still work this change allows for groups to be divided up into seperate chunks; eg: deployment of a swift only stack. * The LXC logic now exists within the plays. * etc/openstack_deploy/user_variables.yml has all password/token variables extracted into the separate file etc/openstack_deploy/user_secrets.yml in order to allow seperate security settings on that file. Items Excised: * All of the roles have had the LXC logic removed from within them which should allow roles to be consumed outside of the `os-ansible-deployment` reference architecture. Note: * the directory rpc_deployment still exists and is presently pointed at plays containing a deprecation warning instructing the user to move to the standard playbooks directory. * While all of the rackspace specific components and variables have been removed and or were refactored the repository still relies on an upstream mirror of Openstack built python files and container images. This upstream mirror is hosted at rackspace at "http://rpc-repo.rackspace.com" though this is not locked to and or tied to rackspace specific installations. This repository contains all of the needed code to create and/or clone your own mirror. DocImpact Co-Authored-By: Jesse Pretorius <jesse.pretorius@rackspace.co.uk> Closes-Bug: #1403676 Implements: blueprint galaxy-roles Change-Id: I03df3328b7655f0cc9e43ba83b02623d038d214e
2015-02-14 10:06:50 -06:00
exit_success