openstack/nova
Code Issues Proposed changes

Verify archive_deleted_rows --all-cells in post test hook

Browse Source 
We are already running archive_deleted_rows in the gate, but we are
not verifying whether all instance records, for example, were actually
successfully removed from the databases (cell0 and cell1).

This adds the --all-cells option to our archive_deleted_rows runs and
verifies that instance records were successfully removed from all cell
databases.

It is not sufficient to check only for return code 0 because
archive_deleted_rows will still return 0 when it misses archiving
records in cell databases.

Related-Bug: #1719487

Change-Id: If133b12bf02d708c099504a88b474dce0bdb0f00
This commit is contained in:
melanie witt 2019-07-26 00:06:18 +00:00
parent 97b8cb3f58
commit 1c9de9c777
1 changed files with 20 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

28
gate/post_test_hook.sh
View File

@ -10,7 +10,14 @@ function archive_deleted_rows {
return 1
fi
for i in `seq 30`; do
$MANAGE $* db archive_deleted_rows --verbose --max_rows 1000 --before "$(date -d tomorrow)"
if [[ $i -eq 1 ]]; then
# This is just a test wrinkle to make sure we're covering the
# non-all-cells (cell0) case, as we're not passing in the cell1
# config.
$MANAGE $* db archive_deleted_rows --verbose --max_rows 50 --before "$(date -d tomorrow)"
else
$MANAGE $* db archive_deleted_rows --verbose --max_rows 1000 --before "$(date -d tomorrow)" --all-cells
fi
RET=$?
if [[ $RET -gt 1 ]]; then
echo Archiving failed with result $RET
@ -36,17 +43,11 @@ function purge_db {
BASE=${BASE:-/opt/stack}
source ${BASE}/devstack/functions-common
source ${BASE}/devstack/lib/nova
cell_conf=$(conductor_conf 1)
# NOTE(danms): We need to pass the main config to get the api db
# bits, and then also the cell config for the cell1 db (instead of
# the cell0 config that is in the main config file). Later files
# take precedence.
conf="--config-file $NOVA_CONF --config-file $cell_conf"
# This needs to go before 'set -e' because otherwise the intermediate runs of
# 'nova-manage db archive_deleted_rows' returning 1 (normal and expected) would
# cause this script to exit and fail.
archive_deleted_rows $conf
archive_deleted_rows
set -e
@ -59,6 +60,17 @@ set +x
source $BASE/devstack/openrc admin
set -x
# Verify whether instances were archived from all cells. Admin credentials are
# needed to list deleted instances across all projects.
echo "Verifying that instances were archived from all cells"
deleted_servers=$(openstack server list --deleted --all-projects -c ID -f value)
# Fail if any deleted servers were found.
if [[ -n "$deleted_servers" ]]; then
echo "There were unarchived instances found after archiving; failing."
exit 1
fi
# TODO(mriedem): Consider checking for instances in ERROR state because
# if there are any, we would expect them to retain allocations in Placement
# and therefore we don't really need to check for leaked allocations.