The `successerator` has been re-enabled using a single retry which will also help
the gate process to ensure that we're not having to deal with so many transient
failures.
Change-Id: I7ca374ce18b7a78b0a1563d5244b1d9ac52c4d92
The gate-check-commit.sh script defined the SYMLINK_DIR variable, but it
was only used in bootstrap-aio.sh. This meant that using
bootstrap-aio.sh by itself would fail due to the undefined variable.
This change moves the declaration to boostrap-aio.sh so that an AIO can
be boostrapped without necessitating a gate check.
Change-Id: Id1b20379c5837cdd4b3df03de42f0ea612e3fbb5
Closes-Bug: #1434315
This change removes the temporary rabbitmq_package_url override set in
gate-check-commit.sh and adds a group_vars entry so that o-a-d
installations will download rabbitmq-server from mirror.rackspace.com.
The role's default will remain pointed at the upstream rabbitmq.com
mirror.
Change-Id: I55b28f073a3ed3e0924cdd0007149bb1d9de21f7
Closes-Bug: #1436240
We're currently get a number of gate failures due to timeouts
downloading from http://www.rabbitmq.com/. This commit temporarily
overrides rabbitmq_package_url so we can clear backlog of reviews.
Change-Id: I7fa91501049e22184707787b7125017e5b63859b
Partial-Bug: #1436240
This change reverts the default setting for fatal_deprecations which
is presently breaks all of openstack due to the requirement of the
cinder / glance v1 apis. The variables have been left in the roles
but the exports in the gate commit scripts have been removed.
Change-Id: Ica6128cba01df22d9aaffd05758902c04c4fe5d4
Partial-Bug: #1428412
This patch implements the following changes:
'commit_aio' tests are now called 'scenario' tests
- this is now set as the default test set
'commit_multinode' tests are now called 'api' tests
- network, compute, object_storage tests have been added
'nightly_heat_multinode' tests are now called 'smoke' tests
Change-Id: Ic415eab199f647a3a4029c535aeceb86291da162
Implements: blueprint additional-tempest-checks
Introduced namespaced variables for all OpenStack services supporting
this setting as defined through oslo libraries. Default value is False
in each case. Gating commit checks now enable the fatal_deprecations
setting for each supporting service.
Closes Bug: 1428412
Change-Id: I5f41d3fdfa1cc876efc0c33c657c9dad18a8ba51
Currently the output of run-playbooks is filtered through an awk
statement to reduce the verbosity of the output. The problem is that
bash uses the return code from the right most command to determine the
return code for the whole pipe. Awk successfully processes the
run-playbooks output, even if run-playbooks fails, so awk returns 0, and
the pipeline is assumed to have completed successfully.
From the bash manual:
"Each command in a pipeline is executed in its own subshell (see Command
Execution Environment). The exit status of a pipeline is the exit status
of the last command in the pipeline, unless the pipefail option is
enabled (see The Set Builtin). If pipefail is enabled, the pipeline’s
return status is the value of the last (rightmost) command to exit with
a non-zero status, or zero if all commands exit successfully. If the
reserved word ‘!’ precedes the pipeline, the exit status is the logical
negation of the exit status as described above. The shell waits for all
commands in the pipeline to terminate before returning a value."
By enabling pipefail, we can ensure that the pipeline exits non-zero if
the run-playbooks script fails.
Closes-bug: #1432295
Change-Id: I019141a7bfbcfa817b90e39f4ae1a68eb3f269a5
Currently the verbose shell builtin is used, resulting in the entire
scripts contents being output in the logs. The instance information is
also output to stdout. This is unnecessarily verbose and prone to
failure. To ensure that the instance information is correctly rendered
failures from within the instance information commands will now always
return true and be logged within the `/openstack/log/ansible-logging`
directory. Each time that the function is called a new log will be
created tagged with the datecode in seconds. This commit also removes
the verbose builtin from the scripts, removes the unused
os-ansible-aio-check script.
Partial-Bug: #1425482
Change-Id: I59952d077a1a8dda12278e57a165d4bac272ed5f
This patch adjusts the method introduced in
I98fb57a56c1b7ddd6558e265cf6e65b159a05123 to trim the debug information
from the ansible output.
This adjustment allows the output to come to the console immediately,
instead of piping to a file and doing the processing after the ansible
run is complete.
Change-Id: I515adcad34ef76b8ea47fe90bc6572fb3aec1d2c
Partial-bug: #1425482
Since I2786f73334b99bfa98d344c50337c86f75fae273 ansible's output is
logged properly to a file and published at the end of a gate check.
This trims the output from ansible to the console to make the console
log that is published far shorter and easier to read.
Co-Authored-By: Ian Cordasco <ian.cordasco@rackspace.com>
Change-Id: I98fb57a56c1b7ddd6558e265cf6e65b159a05123
Partial-bug: 1425482
This change implements the change to allow testr to run in serial which
should resolve most issues with tempest on a resource constrained host.
The problem that is being resolved here is that testr will, by default,
run in multi-threaded and has a relitively high probability to result in
a 503. This primarily impacts the gate jobs as they are built on a hosts
with 4 CPUs and 8GB of ram.
By default if the tempest script is run, testr will run multi-threaded.
However, if the environment variable RUN_TEMPEST_OPTS is set to
'--serial' then it will run in serial. This is done for the
gate-check-commit script so that the change only affects the AIO gate
check.
An additional environment variable TESTR_OPTS is added for further
flexibility.
Co-Authored-By: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
Closes-Bug: #1428847
Change-Id: I8c7ac72ef46ae3f3af3d3503081bec0cbe7276e9
This patch creates the /openstack/log directory within the AIO bootstrap
script to prevent 'unstable' results in openstack-infra's jenkins.
It also moves the symlink to that directory to much earlier in the
gate-check script for the same purpose.
Co-Authored-By: Ian Cordasco <ian.cordasco@rackspace.com>
Closes-Bug: #1428744
Partial-Bug: #1425482
Change-Id: Ib74b5fd216d43e74ca264c14818cfef6e2dc0c70
Ant needs a path that is local to the directory from which it operates.
To be able to publish our logs, we need to symlink /openstack/log to
/home/jenkins/workspace/os-ansible-deployment-dvsm-check-commit/logs.
This also allows us to remove the log publisher we wrote and just use
the devstack-logs publisher instead.
Related-Bug: 1425482
Depends-On: If94e6b97c159000c83fadc6e4dd72d56ee90d3b9
Change-Id: Iee055beb853cbb76a073c8060d6920d317f5bd64
This change implements the blueprint to convert all roles and plays into
a more generic setup, following upstream ansible best practices.
Items Changed:
* All tasks have tags.
* All roles use namespaced variables.
* All redundant tasks within a given play and role have been removed.
* All of the repetitive plays have been removed in-favor of a more
simplistic approach. This change duplicates code within the roles but
ensures that the roles only ever run within their own scope.
* All roles have been built using an ansible galaxy syntax.
* The `*requirement.txt` files have been reformatted follow upstream
Openstack practices.
* Dynamically generated inventory is now more organized, this should assist
anyone who may want or need to dive into the JSON blob that is created.
In the inventory a properties field is used for items that customize containers
within the inventory.
* The environment map has been modified to support additional host groups to
enable the seperation of infrastructure pieces. While the old infra_hosts group
will still work this change allows for groups to be divided up into seperate
chunks; eg: deployment of a swift only stack.
* The LXC logic now exists within the plays.
* etc/openstack_deploy/user_variables.yml has all password/token
variables extracted into the separate file
etc/openstack_deploy/user_secrets.yml in order to allow seperate
security settings on that file.
Items Excised:
* All of the roles have had the LXC logic removed from within them which
should allow roles to be consumed outside of the `os-ansible-deployment`
reference architecture.
Note:
* the directory rpc_deployment still exists and is presently pointed at plays
containing a deprecation warning instructing the user to move to the standard
playbooks directory.
* While all of the rackspace specific components and variables have been removed
and or were refactored the repository still relies on an upstream mirror of
Openstack built python files and container images. This upstream mirror is hosted
at rackspace at "http://rpc-repo.rackspace.com" though this is
not locked to and or tied to rackspace specific installations. This repository
contains all of the needed code to create and/or clone your own mirror.
DocImpact
Co-Authored-By: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
Closes-Bug: #1403676
Implements: blueprint galaxy-roles
Change-Id: I03df3328b7655f0cc9e43ba83b02623d038d214e
This patch removes and/or renames anything that is Rackspace specific
from the playbooks, roles and variables.
It also removes items which appear to be orphaned/unused and flattens
the playbooks into a single directory in order to better match ansible
best practise (and remove some horrible fiddles we were doing).
The following have been removed due to RAX/RPC naming or RAX/RPC
specific usage:
- playbooks/monitoring
- playbooks/rax*
- playbooks/rpc*
- roles/maas*
- roles/rax*
- roles/rpc*
- scripts/f5-*
- scripts/maas*
- scripts/rpc*
- scripts/*lab*
- vars/repo_packages/rackspace*
- vars/repo_packages/rax*
- vars/repo_packages/rpc*
- vars/repo_packages/holland.yml
The following have been removed as they are unused:
- playbooks/setup/host-network-setup.yml
- roles/openssl_pem_request
- roles/host_interfaces
- scripts/elsa*
- ssh/
- vars/repo_packages/turbolift.yml
The following directories have been renamed:
- etc/rpc_deploy > etc/openstack_deploy
- rpc_deployment > playbooks
The playbooks have all been moved into a single directory:
- rpc_deployment/playbooks/infrastructure/* > playbooks/
- rpc_deployment/playbooks/openstack/* > playbooks/
- rpc_deployment/playbooks/setup/* > playbooks/
The following files have been renamed:
- lxc-rpc > lxc-openstack
- lxc-rpc.conf > lxc-openstack.conf
- rpc_environment > openstack_environment
- rpc_release > openstack_release (etc and pip)
- rpc_tempest_gate.sh > openstack_tempest_gate.sh
- rpc_user_config > openstack_user_config
The following variables have been renamed:
- rpc_release > openstack_release
- rpc_repo_url > openstack_repo_url
The following variables have been introduced:
- openstack_code_name: The code name of the upstream OpenStack release
(eg: Juno)
Notable variable/template value changes:
- rabbit_cluster_name: rpc > openstack
- wsrep_cluster_name: rpc_galera_cluster > openstack_galera_cluster
DocImpact
Closes-Bug: #1403676
Implements: blueprint rackspace-namesake
Change-Id: Ib480fdad500b03c7cb90684aa444da9946ba8032
This patch implements the following:
- scripts-library.sh which includes commonly used functions, variables
and other preparation commands for all other scripts
- bootstrap-ansible.sh which only deploys a selected version of ansible
and ensures that any other requirements are prepared on the
deployment host
- bootstrap-aio.sh which runs all host preparation actions for an
all-in-one build
- gate-check-lint.sh which runs a lint and syntax check
- gate-check-commit.sh which runs all actions required for a gate
commit check, utilising the other scripts where required
- run-smoke-test.sh which runs tempest from inside the utility container
- run-playbooks.sh which runs the playbooks
- the existing conf.d/swift.yml is renamed to be an example
configuration - the example configurations can be used as
documentation
- etc/network/interfaces.d/aio_interfaces.cfg,
etc/rpc_deploy/conf.d/swift.yml and
etc/rpc_deploy/rpc_user_config.yml are now configurations used for
the AIO deployment
- a workaround for https://bugs.launchpad.net/bugs/1244589 to ensure
that DHCP checksums are implemented by the host which is required for
the smoke tests to work
- the removal of the rpc heat templates as they're unusable in their
current state
- setting MAX_RETRIES to 0, ensuring that any failures cause an
immediate commit check failure in the gate - this prevents the
masking of failures by retry attempts
DocImpact
Co-Authored-By: Kevin Carter <kevin.carter@rackspace.com>
Closes-Bug: #1415883
Closes-Bug: #1417999
Closes-Bug: #1419807
Change-Id: I95242d48ad0fb055f16510803c8aa14dc183ac17