From c62ed6ccba4b09b454cc1992582a2955ea309aeb Mon Sep 17 00:00:00 2001 From: Chris Dent Date: Wed, 1 May 2019 16:05:23 -0600 Subject: [PATCH] Package db migration scripts in placement pypi dist Having the db migration scripts within the openstack-placement pypi distribution is desirable for deployment tools, such as openstack-ansible. It provides a known good location for the script, available with a pip install. There are several ways to distribute files with a python package. The method used here was chosen because it works both with tarballs and wheels (the files are already in the tarball, as a result of the way pbr works, but not in the wheel). Here's what's done: * The db migrate scripts are put in their own direcory, placement_db_tools, so that only they are packaged, not the other tools. * To preserve how grenade interacts with these files as well as not disrupt the docs, symlinks from tools to placement_db_tools have been created. * placement_db_tools is added to the list of packages included in the openstack-placement distro. This means that when 'pip install openstack-placement' happens, the python environment will then include placement and placement_db_tools directories. The end result is that the true path to the script can be found with: pkg_resources.resource_filename('placement_db_tools', 'mysql-migrate-db.sh') This has been noted in the to-stein.rst document. A different package was chosen to not muddy the waters of what is "actually placement". Similarly, the 'data_files' functionality provided by pbr was not used, because that requires the file be written to a location on the local filesystem, relative to the install prefix. Dirtying the filesystem outside the python lib with this sort of thing is inappropriate. Change-Id: Ie326ce8a2a0692d20793bc18be606e034fa94a44 Story: 2005535 Task: 30671 --- doc/source/upgrade/to-stein.rst | 7 + placement_db_tools/__init__.py | 21 ++ placement_db_tools/mysql-migrate-db.sh | 301 +++++++++++++++++++ placement_db_tools/postgresql-migrate-db.sh | 287 +++++++++++++++++++ setup.cfg | 1 + tools/mysql-migrate-db.sh | 302 +------------------- tools/postgresql-migrate-db.sh | 288 +------------------ 7 files changed, 619 insertions(+), 588 deletions(-) create mode 100644 placement_db_tools/__init__.py create mode 100755 placement_db_tools/mysql-migrate-db.sh create mode 100755 placement_db_tools/postgresql-migrate-db.sh mode change 100755 => 120000 tools/mysql-migrate-db.sh mode change 100755 => 120000 tools/postgresql-migrate-db.sh diff --git a/doc/source/upgrade/to-stein.rst b/doc/source/upgrade/to-stein.rst index 2d89a4d09..1e4fee831 100644 --- a/doc/source/upgrade/to-stein.rst +++ b/doc/source/upgrade/to-stein.rst @@ -62,6 +62,13 @@ There are database migrations scripts in the placement code repository which may be used to copy the data or as models for your own tooling: `mysql-migrate-db.sh`_ and `postgresql-migrate-db.sh`_. +.. note:: Starting in the Train release, these migration scripts are also + packaged with the `openstack-placement`_ package on PyPI. Their + filenames may be discovered using ``pkg_resources`` to look in the + ``placement_db_tools`` package:: + + pkg_resources.resource_filename('placement_db_tools', 'mysql-migrate-db.sh') + For best results run the database migration on your database host. If you are unable to do this, you will need to take some additional steps below. diff --git a/placement_db_tools/__init__.py b/placement_db_tools/__init__.py new file mode 100644 index 000000000..01363eedb --- /dev/null +++ b/placement_db_tools/__init__.py @@ -0,0 +1,21 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. +""" +This package provides a way to distribute useful database management tools as +part of the Python package of the placement service: openstack-placement. + +Their filenames may be discovered using ``pkg_resources`` to look in the +``placement_db_tools`` package: + + pkg_resources.resource_filename('placement_db_tools', + 'mysql-migrate-db.sh') +""" diff --git a/placement_db_tools/mysql-migrate-db.sh b/placement_db_tools/mysql-migrate-db.sh new file mode 100755 index 000000000..7d4343ba1 --- /dev/null +++ b/placement_db_tools/mysql-migrate-db.sh @@ -0,0 +1,301 @@ +#!/bin/bash + +# This script will attempt to migrate your nova-api placement data to +# a new placement database. Run it with --help for usage, and --mkconfig +# to write a template config file to use. + +# Defaults we can guess +DEFAULT_MIGRATE_TABLES="allocations placement_aggregates consumers inventories projects " +DEFAULT_MIGRATE_TABLES+="resource_classes resource_provider_aggregates resource_provider_traits " +DEFAULT_MIGRATE_TABLES+="resource_providers traits users " +MIGRATE_TABLES=${MIGRATE_TABLES:-$DEFAULT_MIGRATE_TABLES} +PLACEMENT_DB_HOST=${PLACEMENT_DB_HOST:-localhost} +PLACEMENT_DB=${PLACEMENT_DB:-placement} +NOVA_API_DB_HOST=${NOVA_API_DB_HOST:-localhost} +NOVA_API_DB=${NOVA_API_DB:-nova_api} +TMPDIR=${TMPDIR:-/tmp} +LAST_MYSQL_ERR=${TMPDIR}/migrate-mysql-db.err +INITIAL_PLACEMENT_DB_VERSION=${INITIAL_DB_VERSION:-b4ed3a175331} +ME=$(basename "$0") + +declare -a ARGS +declare -a OPTS + +function getflag() { + # Return true if --$flag is present on the command line + # Usage: getflag help -> 0 + local flag="$1" + for opt in ${OPTS[*]}; do + if [ "$opt" == "--${flag}" ]; then + return 0 + fi + done + return 1 +} + +function parse_argv() { + # Parse command line arguments into positional arguments and + # option flags. Store each in $ARGS, $OPTS. + # Usage: parse_argv $* + for item in $*; do + if echo $item | grep -q -- '^--'; then + OPTS+=($item) + else + ARGS+=($item) + fi + done +} + +function db_var() { + # Return an attribute of database config based on the symbolic + # name + # Usage: db_var PLACEMENT USER -> $PLACEMENT_USER + local db="$1" + local var="$2" + + eval echo "\$${db}_${var}" +} + +function mysql_command() { + # Run a mysql command with the usual connection information taken + # from a symbolic configuration name + # Usage: mysql_command PLACEMENT [command] [args..] -> stdout + local whichdb="$1" + shift + local command=mysql + if [ "$2" ]; then + command=${1:-mysql} + shift + fi + local db=$(db_var $whichdb DB) + local host=$(db_var $whichdb DB_HOST) + local user=$(db_var $whichdb USER) + local pass=$(db_var $whichdb PASS) + + if [ "$command" = "mysql" ]; then + command="mysql --skip-column-names" + fi + + $command -h$host -u$user -p$pass $db $* 2>$LAST_MYSQL_ERR +} + +function show_error() { + # Prints the last error (if present) and removes the temporary + # file + if [ -f $LAST_MYSQL_ERR ]; then + cat $LAST_MYSQL_ERR + rm -f $LAST_MYSQL_ERR + fi +} + +function check_db() { + # Check a DB to see if it's missing, present, filled with data + # Returns 0 if it is present with data, 1 if present but no data + # or 2 if not present (or unable to connect) + # Usage: check_db PLACEMENT -> 0 + local whichdb="$1" + + local inv + local inv_count + + if ! echo "SELECT DATABASE()" | mysql_command $whichdb >/dev/null 2>&1; then + echo "Failed to connect to $whichdb database" + show_error + return 2 + fi + + inv=$(echo "SELECT COUNT(id) FROM inventories" | + mysql_command $whichdb) + if [ $? -ne 0 ]; then + # No schema + return 1 + fi + + inv_count=$(echo $inv | tail -n1) + if [ $inv_count -gt 0 ]; then + # Data found + return 0 + else + # No data found, but schema intact + return 1 + fi +} + +function check_cli() { + # Returns 0 if placement cli is installed and configured, + # 1 if it is not installed, or 2 if the access to the + # placement database fails + # Usage: check_cli -> 0 + placement-manage --version > /dev/null 2>&1 + + if [ $? -ne 0 ]; then + # placement not installed + return 1 + fi + + placement-manage db version > /dev/null 2>&1 + + if [ $? -ne 0 ]; then + # DB connection fails + return 2 + fi +} + +function migrate_data() { + # Actually migrate data from a source to destination symbolic + # database. Returns 1 if failure, 0 otherwise. + # Usage: migrate_data NOVA_API PLACEMENT -> 0 + local source="$1" + local dest="$2" + local tmpdir=$(mktemp -d migrate-db.XXXXXXXX) + local tmpfile="${tmpdir}/from-nova.sql" + + echo "Dumping from $source to $tmpfile" + mysql_command $source mysqldump $MIGRATE_TABLES > $tmpfile || { + echo 'Failed to dump source database:' + show_error + return 1 + } + echo "Loading to $dest from $tmpfile" + mysql_command $dest < $tmpfile || { + echo 'Failed to load destination database:' + show_error + return 1 + } +} + +function sanity_check_env() { + # Check that we have everything we need to examine the situation + # and potentially do the migration. Loads values from the rcfile, + # if present. Returns 1 if a config was not found, 2 if that + # config is incomplete or 0 if everything is good. + # Usage: sanity_check_env $rcfile -> 0 + + RCFILE="${1:-migrate-db.rc}" + if [ "$RCFILE" = '-' ]; then + # Don't require a file and assume everything is already + # set in the environment + true + elif [ ! -f "$RCFILE" ]; then + echo -n 'ERROR: Specify an RC file on the command line or create ' + echo 'migrate-db.rc in the current directory' + echo + show_help + else + source $RCFILE + fi + + required="NOVA_API_DB NOVA_API_USER NOVA_API_PASS PLACEMENT_DB PLACEMENT_USER PLACEMENT_PASS" + for var in $required; do + value=$(eval echo "\$$var") + if [ -z "$value" ]; then + echo "A value for $var was not provided but is required" + return 2 + fi + done +} + +function make_config() { + # Create or update a config file with defaults we know. Either use + # the default migrate-db.rc or the file specified on the command + # line. + RCFILE="${1:-migrate-db.rc}" + if [ -f "$RCFILE" ]; then + source $RCFILE + fi + + vars="NOVA_API_DB NOVA_API_USER NOVA_API_PASS NOVA_API_DB_HOST " + vars+="PLACEMENT_DB PLACEMENT_USER PLACEMENT_PASS PLACEMENT_DB_HOST " + vars+="MIGRATE_TABLES" + + (for var in $vars; do + val=$(eval echo "\$$var") + echo "${var}=\"$val\"" + done) > $RCFILE + + echo Wrote $(readlink -f $RCFILE) +} + +function show_help() { + echo "Usage: $ME [flags] [rcfile]" + echo + echo "Flags:" + echo " --help: this text" + echo " --migrate: actually do data migration" + echo " --mkconfig: write/update config to \$rcfile" + echo + echo "Pass '-' as \$rcfile if all config values are set in" + echo "the environment." + echo + echo "Exit codes:" + echo " 0: Success" + echo " 1: Usage error" + echo " 2: Configuration missing or incomplete" + echo " 3: Migration already completed" + echo " 4: No data to migrate from nova (new deployment)" + echo " 5: Unable to connect to one or both databases" + echo " 6: Unable to execute placement's CLI commands" + exit 0 +} + +parse_argv $* + +if getflag help; then + show_help +fi + +if getflag mkconfig; then + make_config $ARGS + exit 0 +fi + +# +# Actual migration logic starts here +# + +# Sanity check that we have what we need or bail +sanity_check_env $ARGS || exit $? + +# Check the state of each database we care about +check_db NOVA_API +nova_present=$? +check_db PLACEMENT +placement_present=$? +check_cli +placement_cli=$? + +# Try to come up with a good reason to refuse to migrate +if [ $nova_present -eq 0 -a $placement_present -eq 0 ]; then + echo "Migration has already completed. The placement database appears to have data." + exit 3 +elif [ $nova_present -eq 1 ]; then + echo "No data present in nova database - nothing to migrate (new deployment?)" + exit 4 +elif [ $nova_present -eq 2 ]; then + echo "Unable to proceed without connection to nova database" + exit 5 +elif [ $placement_present -eq 2 ]; then + echo "Unable to proceed without connection to placement database" + exit 5 +elif [ $placement_cli -eq 1 ]; then + echo "Unable to proceed without placement installed" + exit 6 +elif [ $placement_cli -eq 2 ]; then + echo "The 'placement-manage db version' command fails" + echo "Is placement.conf configured to access the new database?" + exit 6 +fi + +# If we get here, we expect to be able to migrate. Require them to opt into +# actual migration before we do anything. + +echo Nova database contains data, placement database does not. Okay to proceed with migration + +if getflag migrate $*; then + migrate_data NOVA_API PLACEMENT + placement-manage db stamp $INITIAL_PLACEMENT_DB_VERSION +else + echo "To actually migrate, run me with --migrate" +fi + +rm -f $LAST_MYSQL_ERR diff --git a/placement_db_tools/postgresql-migrate-db.sh b/placement_db_tools/postgresql-migrate-db.sh new file mode 100755 index 000000000..9bd6732cf --- /dev/null +++ b/placement_db_tools/postgresql-migrate-db.sh @@ -0,0 +1,287 @@ +#!/bin/bash + +# This script will attempt to migrate your nova-api placement data to +# a new placement database. Run it with --help for usage, and --mkconfig +# to write a template config file to use. + +# Defaults we can guess +DEFAULT_MIGRATE_TABLES="allocations placement_aggregates consumers inventories projects " +DEFAULT_MIGRATE_TABLES+="resource_classes resource_provider_aggregates resource_provider_traits " +DEFAULT_MIGRATE_TABLES+="resource_providers traits users " +MIGRATE_TABLES=${MIGRATE_TABLES:-$DEFAULT_MIGRATE_TABLES} +PG_MIGRATE_TABLES=${MIGRATE_TABLES// /|}} +PLACEMENT_DB_HOST=${PLACEMENT_DB_HOST:-localhost} +PLACEMENT_DB=${PLACEMENT_DB:-placement} +NOVA_API_DB_HOST=${NOVA_API_DB_HOST:-localhost} +NOVA_API_DB=${NOVA_API_DB:-nova_api} +TMPDIR=${TMPDIR:-/tmp} +LAST_PSQL_ERR=${TMPDIR}/migrate-psql-db.err +INITIAL_PLACEMENT_DB_VERSION=${INITIAL_DB_VERSION:-b4ed3a175331} + +declare -a ARGS +declare -a OPTS + +function getflag() { + # Return true if --$flag is present on the command line + # Usage: getflag help -> 0 + local flag="$1" + for opt in ${OPTS[*]}; do + if [ "$opt" == "--${flag}" ]; then + return 0 + fi + done + return 1 +} + +function parse_argv() { + # Parse command line arguments into positional arguments and + # option flags. Store each in $ARGS, $OPTS. + # Usage: parse_argv $* + for item in $*; do + if echo $item | grep -q -- '^--'; then + OPTS+=($item) + else + ARGS+=($item) + fi + done +} + +function db_var() { + # Return an attribute of database config based on the symbolic + # name + # Usage: db_var PLACEMENT USER -> $PLACEMENT_USER + local db="$1" + local var="$2" + + eval echo "\$${db}_${var}" +} + +function psql_command() { + # Run a psql command with the usual connection information taken + # from a symbolic configuration name + # Usage: psql_command PLACEMENT [command] [args..] -> stdout + local whichdb="$1" + shift + local command=psql + if [ "$2" ]; then + command=${1:-psql} + shift + fi + local db=$(db_var $whichdb DB) + local host=$(db_var $whichdb DB_HOST) + local user=$(db_var $whichdb USER) + local pass=$(db_var $whichdb PASS) + + if [ "$command" = "psql" ]; then + command="psql -t" + fi + + PGPASSWORD=$pass $command -h$host -U$user $db $* 2>$LAST_PSQL_ERR +} + +function check_db() { + # Check a DB to see if it's missing, present, filled with data + # Returns 0 if it is present with data, 1 if present but no data + # or 2 if not present (or unable to connect) + # Usage: check_db PLACEMENT -> 0 + local whichdb="$1" + + local inv + local inv_count + local error_found + + if ! echo "SELECT CURRENT_DATABASE()" | psql_command $whichdb >/dev/null 2>&1; then + echo "Failed to connect to $whichdb database" + return 2 + fi + + inv=$(echo "SELECT COUNT(id) FROM inventories" | + psql_command $whichdb) + if [ $? -ne 0 ]; then + # No DB + return 1 + fi + + error_found=$(cat $LAST_PSQL_ERR | grep ERROR) + if [ $? -eq 0 ]; then + # No schema + return 1 + fi + + inv_count=$(echo $inv | tail -n1) + if [ $inv_count -gt 0 ]; then + # Data found + return 0 + else + # No data found, but schema intact + return 1 + fi +} + +function check_cli() { + # Returns 0 if placement cli is installed and configured, + # 1 if it is not installed, or 2 if the access to the + # placement database fails + # Usage: check_cli -> 0 + placement-manage --version > /dev/null 2>&1 + + if [ $? -ne 0 ]; then + # placement not installed + return 1 + fi + + placement-manage db version > /dev/null 2>&1 + + if [ $? -ne 0 ]; then + # DB connection fails + return 2 + fi +} + +function migrate_data() { + # Actually migrate data from a source to destination symbolic + # database. Returns 1 if failure, 0 otherwise. + # Usage: migrate_data NOVA_API PLACEMENT -> 0 + local source="$1" + local dest="$2" + local tmpdir=$(mktemp -d migrate-db.XXXXXXXX) + local tmpfile="${tmpdir}/from-nova.sql" + + echo "Dumping from $source to $tmpfile" + psql_command $source pg_dump -t $PG_MIGRATE_TABLES > $tmpfile || { + echo 'Failed to dump source database:' + cat $LAST_PSQL_ERR + return 1 + } + echo "Loading to $dest from $tmpfile" + # There is some output when loading file, so we redirect it to /dev/null. + psql_command $dest < $tmpfile >/dev/null || { + echo 'Failed to load destination database:' + cat $LAST_PSQL_ERR + return 1 + } +} + +function sanity_check_env() { + # Check that we have everything we need to examine the situation + # and potentially do the migration. Loads values from the rcfile, + # if present. Returns 1 if a config was not found, 2 if that + # config is incomplete or 0 if everything is good. + # Usage: sanity_check_env $rcfile -> 0 + + RCFILE="${1:-migrate-db.rc}" + if [ ! -f "$RCFILE" ]; then + echo 'Specify an RC file on the command line or create migrate-db.rc in the current directory' + return 1 + fi + + source $RCFILE + + required="NOVA_API_DB NOVA_API_USER NOVA_API_PASS PLACEMENT_DB PLACEMENT_USER PLACEMENT_PASS" + for var in $required; do + value=$(eval echo "\$$var") + if [ -z "$value" ]; then + echo "A value for $var was not provided but is required" + return 2 + fi + done + +} + +function make_config() { + # Create or update a config file with defaults we know. Either use + # the default migrate-db.rc or the file specified on the command + # line. + RCFILE="${1:-migrate-db.rc}" + if [ -f "$RCFILE" ]; then + source $RCFILE + fi + + vars="NOVA_API_DB NOVA_API_USER NOVA_API_PASS NOVA_API_DB_HOST " + vars+="PLACEMENT_DB PLACEMENT_USER PLACEMENT_PASS PLACEMENT_DB_HOST " + vars+="MIGRATE_TABLES" + + (for var in $vars; do + val=$(eval echo "\$$var") + echo "${var}=\"$val\"" + done) > $RCFILE + + echo Wrote $(readlink -f $RCFILE) +} + +parse_argv $* + +if getflag help; then + echo "Usage: $0 [flags] [rcfile]" + echo + echo "Flags:" + echo " --help: this text" + echo " --migrate: actually do data migration" + echo " --mkconfig: write/update config to \$rcfile" + echo + echo "Exit codes:" + echo " 0: Success" + echo " 1: Usage error" + echo " 2: Configuration missing or incomplete" + echo " 3: Migration already completed" + echo " 4: No data to migrate from nova (new deployment)" + echo " 5: Unable to connect to one or both databases" + echo " 6: Unable to execute placement's CLI commands" + exit 0 +fi + +if getflag mkconfig; then + make_config $ARGS + exit 0 +fi + +# +# Actual migration logic starts here +# + +# Sanity check that we have what we need or bail +sanity_check_env $ARGS || exit $? + +# Check the state of each database we care about +check_db NOVA_API +nova_present=$? +check_db PLACEMENT +placement_present=$? +check_cli +placement_cli=$? + +# Try to come up with a good reason to refuse to migrate +if [ $nova_present -eq 0 -a $placement_present -eq 0 ]; then + echo "Migration has already completed. The placement database appears to have data." + exit 3 +elif [ $nova_present -eq 1 ]; then + echo "No data present in nova database - nothing to migrate (new deployment?)" + exit 4 +elif [ $nova_present -eq 2 ]; then + echo "Unable to proceed without connection to nova database" + exit 5 +elif [ $placement_present -eq 2 ]; then + echo "Unable to proceed without connection to placement database" + exit 5 +elif [ $placement_cli -eq 1 ]; then + echo "Unable to proceed without placement installed" + exit 6 +elif [ $placement_cli -eq 2 ]; then + echo "The 'placement-manage db version' command fails" + echo "Is placement.conf configured to access the new database?" + exit 6 +fi + +# If we get here, we expect to be able to migrate. Require them to opt into +# actual migration before we do anything. + +echo Nova database contains data, placement database does not. Okay to proceed with migration + +if getflag migrate $*; then + migrate_data NOVA_API PLACEMENT + placement-manage db stamp $INITIAL_PLACEMENT_DB_VERSION +else + echo "To actually migrate, run me with --migrate" +fi + +rm -f $LAST_PSQL_ERR diff --git a/setup.cfg b/setup.cfg index 09102998e..c5d9ad7d2 100644 --- a/setup.cfg +++ b/setup.cfg @@ -25,6 +25,7 @@ setup-hooks = [files] packages = placement + placement_db_tools [entry_points] oslo.config.opts = diff --git a/tools/mysql-migrate-db.sh b/tools/mysql-migrate-db.sh deleted file mode 100755 index 7d4343ba1..000000000 --- a/tools/mysql-migrate-db.sh +++ /dev/null @@ -1,301 +0,0 @@ -#!/bin/bash - -# This script will attempt to migrate your nova-api placement data to -# a new placement database. Run it with --help for usage, and --mkconfig -# to write a template config file to use. - -# Defaults we can guess -DEFAULT_MIGRATE_TABLES="allocations placement_aggregates consumers inventories projects " -DEFAULT_MIGRATE_TABLES+="resource_classes resource_provider_aggregates resource_provider_traits " -DEFAULT_MIGRATE_TABLES+="resource_providers traits users " -MIGRATE_TABLES=${MIGRATE_TABLES:-$DEFAULT_MIGRATE_TABLES} -PLACEMENT_DB_HOST=${PLACEMENT_DB_HOST:-localhost} -PLACEMENT_DB=${PLACEMENT_DB:-placement} -NOVA_API_DB_HOST=${NOVA_API_DB_HOST:-localhost} -NOVA_API_DB=${NOVA_API_DB:-nova_api} -TMPDIR=${TMPDIR:-/tmp} -LAST_MYSQL_ERR=${TMPDIR}/migrate-mysql-db.err -INITIAL_PLACEMENT_DB_VERSION=${INITIAL_DB_VERSION:-b4ed3a175331} -ME=$(basename "$0") - -declare -a ARGS -declare -a OPTS - -function getflag() { - # Return true if --$flag is present on the command line - # Usage: getflag help -> 0 - local flag="$1" - for opt in ${OPTS[*]}; do - if [ "$opt" == "--${flag}" ]; then - return 0 - fi - done - return 1 -} - -function parse_argv() { - # Parse command line arguments into positional arguments and - # option flags. Store each in $ARGS, $OPTS. - # Usage: parse_argv $* - for item in $*; do - if echo $item | grep -q -- '^--'; then - OPTS+=($item) - else - ARGS+=($item) - fi - done -} - -function db_var() { - # Return an attribute of database config based on the symbolic - # name - # Usage: db_var PLACEMENT USER -> $PLACEMENT_USER - local db="$1" - local var="$2" - - eval echo "\$${db}_${var}" -} - -function mysql_command() { - # Run a mysql command with the usual connection information taken - # from a symbolic configuration name - # Usage: mysql_command PLACEMENT [command] [args..] -> stdout - local whichdb="$1" - shift - local command=mysql - if [ "$2" ]; then - command=${1:-mysql} - shift - fi - local db=$(db_var $whichdb DB) - local host=$(db_var $whichdb DB_HOST) - local user=$(db_var $whichdb USER) - local pass=$(db_var $whichdb PASS) - - if [ "$command" = "mysql" ]; then - command="mysql --skip-column-names" - fi - - $command -h$host -u$user -p$pass $db $* 2>$LAST_MYSQL_ERR -} - -function show_error() { - # Prints the last error (if present) and removes the temporary - # file - if [ -f $LAST_MYSQL_ERR ]; then - cat $LAST_MYSQL_ERR - rm -f $LAST_MYSQL_ERR - fi -} - -function check_db() { - # Check a DB to see if it's missing, present, filled with data - # Returns 0 if it is present with data, 1 if present but no data - # or 2 if not present (or unable to connect) - # Usage: check_db PLACEMENT -> 0 - local whichdb="$1" - - local inv - local inv_count - - if ! echo "SELECT DATABASE()" | mysql_command $whichdb >/dev/null 2>&1; then - echo "Failed to connect to $whichdb database" - show_error - return 2 - fi - - inv=$(echo "SELECT COUNT(id) FROM inventories" | - mysql_command $whichdb) - if [ $? -ne 0 ]; then - # No schema - return 1 - fi - - inv_count=$(echo $inv | tail -n1) - if [ $inv_count -gt 0 ]; then - # Data found - return 0 - else - # No data found, but schema intact - return 1 - fi -} - -function check_cli() { - # Returns 0 if placement cli is installed and configured, - # 1 if it is not installed, or 2 if the access to the - # placement database fails - # Usage: check_cli -> 0 - placement-manage --version > /dev/null 2>&1 - - if [ $? -ne 0 ]; then - # placement not installed - return 1 - fi - - placement-manage db version > /dev/null 2>&1 - - if [ $? -ne 0 ]; then - # DB connection fails - return 2 - fi -} - -function migrate_data() { - # Actually migrate data from a source to destination symbolic - # database. Returns 1 if failure, 0 otherwise. - # Usage: migrate_data NOVA_API PLACEMENT -> 0 - local source="$1" - local dest="$2" - local tmpdir=$(mktemp -d migrate-db.XXXXXXXX) - local tmpfile="${tmpdir}/from-nova.sql" - - echo "Dumping from $source to $tmpfile" - mysql_command $source mysqldump $MIGRATE_TABLES > $tmpfile || { - echo 'Failed to dump source database:' - show_error - return 1 - } - echo "Loading to $dest from $tmpfile" - mysql_command $dest < $tmpfile || { - echo 'Failed to load destination database:' - show_error - return 1 - } -} - -function sanity_check_env() { - # Check that we have everything we need to examine the situation - # and potentially do the migration. Loads values from the rcfile, - # if present. Returns 1 if a config was not found, 2 if that - # config is incomplete or 0 if everything is good. - # Usage: sanity_check_env $rcfile -> 0 - - RCFILE="${1:-migrate-db.rc}" - if [ "$RCFILE" = '-' ]; then - # Don't require a file and assume everything is already - # set in the environment - true - elif [ ! -f "$RCFILE" ]; then - echo -n 'ERROR: Specify an RC file on the command line or create ' - echo 'migrate-db.rc in the current directory' - echo - show_help - else - source $RCFILE - fi - - required="NOVA_API_DB NOVA_API_USER NOVA_API_PASS PLACEMENT_DB PLACEMENT_USER PLACEMENT_PASS" - for var in $required; do - value=$(eval echo "\$$var") - if [ -z "$value" ]; then - echo "A value for $var was not provided but is required" - return 2 - fi - done -} - -function make_config() { - # Create or update a config file with defaults we know. Either use - # the default migrate-db.rc or the file specified on the command - # line. - RCFILE="${1:-migrate-db.rc}" - if [ -f "$RCFILE" ]; then - source $RCFILE - fi - - vars="NOVA_API_DB NOVA_API_USER NOVA_API_PASS NOVA_API_DB_HOST " - vars+="PLACEMENT_DB PLACEMENT_USER PLACEMENT_PASS PLACEMENT_DB_HOST " - vars+="MIGRATE_TABLES" - - (for var in $vars; do - val=$(eval echo "\$$var") - echo "${var}=\"$val\"" - done) > $RCFILE - - echo Wrote $(readlink -f $RCFILE) -} - -function show_help() { - echo "Usage: $ME [flags] [rcfile]" - echo - echo "Flags:" - echo " --help: this text" - echo " --migrate: actually do data migration" - echo " --mkconfig: write/update config to \$rcfile" - echo - echo "Pass '-' as \$rcfile if all config values are set in" - echo "the environment." - echo - echo "Exit codes:" - echo " 0: Success" - echo " 1: Usage error" - echo " 2: Configuration missing or incomplete" - echo " 3: Migration already completed" - echo " 4: No data to migrate from nova (new deployment)" - echo " 5: Unable to connect to one or both databases" - echo " 6: Unable to execute placement's CLI commands" - exit 0 -} - -parse_argv $* - -if getflag help; then - show_help -fi - -if getflag mkconfig; then - make_config $ARGS - exit 0 -fi - -# -# Actual migration logic starts here -# - -# Sanity check that we have what we need or bail -sanity_check_env $ARGS || exit $? - -# Check the state of each database we care about -check_db NOVA_API -nova_present=$? -check_db PLACEMENT -placement_present=$? -check_cli -placement_cli=$? - -# Try to come up with a good reason to refuse to migrate -if [ $nova_present -eq 0 -a $placement_present -eq 0 ]; then - echo "Migration has already completed. The placement database appears to have data." - exit 3 -elif [ $nova_present -eq 1 ]; then - echo "No data present in nova database - nothing to migrate (new deployment?)" - exit 4 -elif [ $nova_present -eq 2 ]; then - echo "Unable to proceed without connection to nova database" - exit 5 -elif [ $placement_present -eq 2 ]; then - echo "Unable to proceed without connection to placement database" - exit 5 -elif [ $placement_cli -eq 1 ]; then - echo "Unable to proceed without placement installed" - exit 6 -elif [ $placement_cli -eq 2 ]; then - echo "The 'placement-manage db version' command fails" - echo "Is placement.conf configured to access the new database?" - exit 6 -fi - -# If we get here, we expect to be able to migrate. Require them to opt into -# actual migration before we do anything. - -echo Nova database contains data, placement database does not. Okay to proceed with migration - -if getflag migrate $*; then - migrate_data NOVA_API PLACEMENT - placement-manage db stamp $INITIAL_PLACEMENT_DB_VERSION -else - echo "To actually migrate, run me with --migrate" -fi - -rm -f $LAST_MYSQL_ERR diff --git a/tools/mysql-migrate-db.sh b/tools/mysql-migrate-db.sh new file mode 120000 index 000000000..178d0f68f --- /dev/null +++ b/tools/mysql-migrate-db.sh @@ -0,0 +1 @@ +../placement_db_tools/mysql-migrate-db.sh \ No newline at end of file diff --git a/tools/postgresql-migrate-db.sh b/tools/postgresql-migrate-db.sh deleted file mode 100755 index 9bd6732cf..000000000 --- a/tools/postgresql-migrate-db.sh +++ /dev/null @@ -1,287 +0,0 @@ -#!/bin/bash - -# This script will attempt to migrate your nova-api placement data to -# a new placement database. Run it with --help for usage, and --mkconfig -# to write a template config file to use. - -# Defaults we can guess -DEFAULT_MIGRATE_TABLES="allocations placement_aggregates consumers inventories projects " -DEFAULT_MIGRATE_TABLES+="resource_classes resource_provider_aggregates resource_provider_traits " -DEFAULT_MIGRATE_TABLES+="resource_providers traits users " -MIGRATE_TABLES=${MIGRATE_TABLES:-$DEFAULT_MIGRATE_TABLES} -PG_MIGRATE_TABLES=${MIGRATE_TABLES// /|}} -PLACEMENT_DB_HOST=${PLACEMENT_DB_HOST:-localhost} -PLACEMENT_DB=${PLACEMENT_DB:-placement} -NOVA_API_DB_HOST=${NOVA_API_DB_HOST:-localhost} -NOVA_API_DB=${NOVA_API_DB:-nova_api} -TMPDIR=${TMPDIR:-/tmp} -LAST_PSQL_ERR=${TMPDIR}/migrate-psql-db.err -INITIAL_PLACEMENT_DB_VERSION=${INITIAL_DB_VERSION:-b4ed3a175331} - -declare -a ARGS -declare -a OPTS - -function getflag() { - # Return true if --$flag is present on the command line - # Usage: getflag help -> 0 - local flag="$1" - for opt in ${OPTS[*]}; do - if [ "$opt" == "--${flag}" ]; then - return 0 - fi - done - return 1 -} - -function parse_argv() { - # Parse command line arguments into positional arguments and - # option flags. Store each in $ARGS, $OPTS. - # Usage: parse_argv $* - for item in $*; do - if echo $item | grep -q -- '^--'; then - OPTS+=($item) - else - ARGS+=($item) - fi - done -} - -function db_var() { - # Return an attribute of database config based on the symbolic - # name - # Usage: db_var PLACEMENT USER -> $PLACEMENT_USER - local db="$1" - local var="$2" - - eval echo "\$${db}_${var}" -} - -function psql_command() { - # Run a psql command with the usual connection information taken - # from a symbolic configuration name - # Usage: psql_command PLACEMENT [command] [args..] -> stdout - local whichdb="$1" - shift - local command=psql - if [ "$2" ]; then - command=${1:-psql} - shift - fi - local db=$(db_var $whichdb DB) - local host=$(db_var $whichdb DB_HOST) - local user=$(db_var $whichdb USER) - local pass=$(db_var $whichdb PASS) - - if [ "$command" = "psql" ]; then - command="psql -t" - fi - - PGPASSWORD=$pass $command -h$host -U$user $db $* 2>$LAST_PSQL_ERR -} - -function check_db() { - # Check a DB to see if it's missing, present, filled with data - # Returns 0 if it is present with data, 1 if present but no data - # or 2 if not present (or unable to connect) - # Usage: check_db PLACEMENT -> 0 - local whichdb="$1" - - local inv - local inv_count - local error_found - - if ! echo "SELECT CURRENT_DATABASE()" | psql_command $whichdb >/dev/null 2>&1; then - echo "Failed to connect to $whichdb database" - return 2 - fi - - inv=$(echo "SELECT COUNT(id) FROM inventories" | - psql_command $whichdb) - if [ $? -ne 0 ]; then - # No DB - return 1 - fi - - error_found=$(cat $LAST_PSQL_ERR | grep ERROR) - if [ $? -eq 0 ]; then - # No schema - return 1 - fi - - inv_count=$(echo $inv | tail -n1) - if [ $inv_count -gt 0 ]; then - # Data found - return 0 - else - # No data found, but schema intact - return 1 - fi -} - -function check_cli() { - # Returns 0 if placement cli is installed and configured, - # 1 if it is not installed, or 2 if the access to the - # placement database fails - # Usage: check_cli -> 0 - placement-manage --version > /dev/null 2>&1 - - if [ $? -ne 0 ]; then - # placement not installed - return 1 - fi - - placement-manage db version > /dev/null 2>&1 - - if [ $? -ne 0 ]; then - # DB connection fails - return 2 - fi -} - -function migrate_data() { - # Actually migrate data from a source to destination symbolic - # database. Returns 1 if failure, 0 otherwise. - # Usage: migrate_data NOVA_API PLACEMENT -> 0 - local source="$1" - local dest="$2" - local tmpdir=$(mktemp -d migrate-db.XXXXXXXX) - local tmpfile="${tmpdir}/from-nova.sql" - - echo "Dumping from $source to $tmpfile" - psql_command $source pg_dump -t $PG_MIGRATE_TABLES > $tmpfile || { - echo 'Failed to dump source database:' - cat $LAST_PSQL_ERR - return 1 - } - echo "Loading to $dest from $tmpfile" - # There is some output when loading file, so we redirect it to /dev/null. - psql_command $dest < $tmpfile >/dev/null || { - echo 'Failed to load destination database:' - cat $LAST_PSQL_ERR - return 1 - } -} - -function sanity_check_env() { - # Check that we have everything we need to examine the situation - # and potentially do the migration. Loads values from the rcfile, - # if present. Returns 1 if a config was not found, 2 if that - # config is incomplete or 0 if everything is good. - # Usage: sanity_check_env $rcfile -> 0 - - RCFILE="${1:-migrate-db.rc}" - if [ ! -f "$RCFILE" ]; then - echo 'Specify an RC file on the command line or create migrate-db.rc in the current directory' - return 1 - fi - - source $RCFILE - - required="NOVA_API_DB NOVA_API_USER NOVA_API_PASS PLACEMENT_DB PLACEMENT_USER PLACEMENT_PASS" - for var in $required; do - value=$(eval echo "\$$var") - if [ -z "$value" ]; then - echo "A value for $var was not provided but is required" - return 2 - fi - done - -} - -function make_config() { - # Create or update a config file with defaults we know. Either use - # the default migrate-db.rc or the file specified on the command - # line. - RCFILE="${1:-migrate-db.rc}" - if [ -f "$RCFILE" ]; then - source $RCFILE - fi - - vars="NOVA_API_DB NOVA_API_USER NOVA_API_PASS NOVA_API_DB_HOST " - vars+="PLACEMENT_DB PLACEMENT_USER PLACEMENT_PASS PLACEMENT_DB_HOST " - vars+="MIGRATE_TABLES" - - (for var in $vars; do - val=$(eval echo "\$$var") - echo "${var}=\"$val\"" - done) > $RCFILE - - echo Wrote $(readlink -f $RCFILE) -} - -parse_argv $* - -if getflag help; then - echo "Usage: $0 [flags] [rcfile]" - echo - echo "Flags:" - echo " --help: this text" - echo " --migrate: actually do data migration" - echo " --mkconfig: write/update config to \$rcfile" - echo - echo "Exit codes:" - echo " 0: Success" - echo " 1: Usage error" - echo " 2: Configuration missing or incomplete" - echo " 3: Migration already completed" - echo " 4: No data to migrate from nova (new deployment)" - echo " 5: Unable to connect to one or both databases" - echo " 6: Unable to execute placement's CLI commands" - exit 0 -fi - -if getflag mkconfig; then - make_config $ARGS - exit 0 -fi - -# -# Actual migration logic starts here -# - -# Sanity check that we have what we need or bail -sanity_check_env $ARGS || exit $? - -# Check the state of each database we care about -check_db NOVA_API -nova_present=$? -check_db PLACEMENT -placement_present=$? -check_cli -placement_cli=$? - -# Try to come up with a good reason to refuse to migrate -if [ $nova_present -eq 0 -a $placement_present -eq 0 ]; then - echo "Migration has already completed. The placement database appears to have data." - exit 3 -elif [ $nova_present -eq 1 ]; then - echo "No data present in nova database - nothing to migrate (new deployment?)" - exit 4 -elif [ $nova_present -eq 2 ]; then - echo "Unable to proceed without connection to nova database" - exit 5 -elif [ $placement_present -eq 2 ]; then - echo "Unable to proceed without connection to placement database" - exit 5 -elif [ $placement_cli -eq 1 ]; then - echo "Unable to proceed without placement installed" - exit 6 -elif [ $placement_cli -eq 2 ]; then - echo "The 'placement-manage db version' command fails" - echo "Is placement.conf configured to access the new database?" - exit 6 -fi - -# If we get here, we expect to be able to migrate. Require them to opt into -# actual migration before we do anything. - -echo Nova database contains data, placement database does not. Okay to proceed with migration - -if getflag migrate $*; then - migrate_data NOVA_API PLACEMENT - placement-manage db stamp $INITIAL_PLACEMENT_DB_VERSION -else - echo "To actually migrate, run me with --migrate" -fi - -rm -f $LAST_PSQL_ERR diff --git a/tools/postgresql-migrate-db.sh b/tools/postgresql-migrate-db.sh new file mode 120000 index 000000000..98f9d1880 --- /dev/null +++ b/tools/postgresql-migrate-db.sh @@ -0,0 +1 @@ +../placement_db_tools/postgresql-migrate-db.sh \ No newline at end of file