openstack/nova
Code Issues Proposed changes

Add nova-status upgrade check command framework

Browse Source 
This adds the basic framework for the nova-status
upgrade check commands. Follow up patches will flesh
this out to perform cells v2 and placement API related
upgrade status checks.

A man page is added for the new CLI and as part of that
the man page index is sorted.

Part of blueprint cells-scheduling-interaction
Part of blueprint resource-providers-scheduler-db-filters

Change-Id: I687dd7317703a1bb76c197ebba849ca368c0872e
This commit is contained in:
Matt Riedemann 2016-12-15 16:01:41 -05:00
parent 4b71d7e221
commit a1f3a5946a
7 changed files with 490 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
doc/source/man/index.rst
View File

@ -41,7 +41,7 @@ Reference
nova-novncproxy
nova-rootwrap
nova-scheduler
nova-spicehtml5proxy
nova-xvpvncproxy
nova-serialproxy
nova-spicehtml5proxy
nova-status
nova-xvpvncproxy

96
doc/source/man/nova-status.rst Normal file
View File

@ -0,0 +1,96 @@
===========
nova-status
===========
--------------------------------------
CLI interface for nova status commands
--------------------------------------
:Author: openstack@lists.openstack.org
:Date: 2016-12-16
:Copyright: OpenStack Foundation
:Version: 15.0.0
:Manual section: 1
:Manual group: cloud computing
SYNOPSIS
========
nova-status <category> <action> [<args>]
DESCRIPTION
===========
The nova-status command provides routines for checking the status of a Nova
deployment.
OPTIONS
=======
The standard pattern for executing a nova-status command is::
nova-status <category> <command> [<args>]
Run without arguments to see a list of available command categories::
nova-status
Categories are:
* upgrade
Detailed descriptions are below.
You can also run with a category argument such as "upgrade" to see a list of
all commands in that category::
nova-status upgrade
These sections describe the available categories and arguments for nova-status.
Upgrade
~~~~~~~
``nova-status upgrade check``
Performs a release-specific readiness check before restarting services with
new code. This command expects to have complete configuration and access
to databases and services within a cell. For example, this check may query
the Nova API database and one or more cell databases. It may also make
requests to other services such as the Placement REST API via the Keystone
service catalog.
**Return Codes**
0) All upgrade readiness checks passed successfully and there is nothing
to do.
1) At least one check encountered an issue and requires further
investigation. This is considered a warning but the upgrade may be OK.
2) There was an upgrade status check failure that needs to be
investigated. This should be considered something that stops an
upgrade.
**History of Checks**
**15.0.0 (Ocata)**
* Checks are added for cells v2 so ``nova-status upgrade check`` should be
run *after* running the ``nova-manage cell_v2 simple_cell_setup``
command.
* Checks are added for the Placement API such that there is an endpoint in
the Keystone service catalog, the service is running and the check can
make a successful request to the endpoint. The command also checks to
see that there are compute node resource providers checking in with the
Placement service. More information on the Placement service can be found
at: `<http://docs.openstack.org/developer/nova/placement.html>`_
SEE ALSO
========
* OpenStack Nova Docs: `<http://docs.openstack.org/developer/nova/>`_
BUGS
====
* Nova bugs are managed at Launchpad: `<https://bugs.launchpad.net/nova>`_

209
nova/cmd/status.py Normal file
View File

@ -0,0 +1,209 @@
# Copyright 2016 IBM Corp.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""
CLI interface for nova status commands.
"""
from __future__ import print_function
import functools
import sys
import textwrap
import traceback
# enum comes from the enum34 package if python < 3.4, else it's stdlib
import enum
from oslo_config import cfg
import prettytable
from nova.cmd import common as cmd_common
import nova.conf
from nova import config
from nova.i18n import _
from nova import version
CONF = nova.conf.CONF
class UpgradeCheckCode(enum.IntEnum):
"""These are the status codes for the nova-status upgrade check command
and internal check commands.
"""
# All upgrade readiness checks passed successfully and there is
# nothing to do.
SUCCESS = 0
# At least one check encountered an issue and requires further
# investigation. This is considered a warning but the upgrade may be OK.
WARNING = 1
# There was an upgrade status check failure that needs to be
# investigated. This should be considered something that stops an upgrade.
FAILURE = 2
UPGRADE_CHECK_MSG_MAP = {
UpgradeCheckCode.SUCCESS: _('Success'),
UpgradeCheckCode.WARNING: _('Warning'),
UpgradeCheckCode.FAILURE: _('Failure'),
}
class UpgradeCheckResult(object):
"""Class used for 'nova-status upgrade check' results.
The 'code' attribute is an UpgradeCheckCode enum.
The 'details' attribute is a translated message generally only used for
checks that result in a warning or failure code. The details should provide
information on what issue was discovered along with any remediation.
"""
def __init__(self, code, details=None):
super(UpgradeCheckResult, self).__init__()
self.code = code
self.details = details
class UpgradeCommands(object):
"""Commands related to upgrades.
The subcommands here must not rely on the nova object model since they
should be able to run on n-1 data. Any queries to the database should be
done through the sqlalchemy query language directly like the database
schema migrations.
"""
def _check_cellsv2(self):
# TODO(mriedem): perform the same checks as the 030_require_cell_setup
# API DB migration.
return UpgradeCheckResult(UpgradeCheckCode.SUCCESS)
def _check_placement(self):
# TODO(mriedem): check to see that the placement API service is running
# and we can connect to it, and that the number of resource providers
# in the placement service is greater than or equal to the number of
# compute nodes in the database.
return UpgradeCheckResult(UpgradeCheckCode.SUCCESS)
# The format of the check functions is to return an UpgradeCheckResult
# object with the appropriate UpgradeCheckCode and details set. If the
# check hits warnings or failures then those should be stored in the
# returned UpgradeCheckResult's "details" attribute. The summary will
# be rolled up at the end of the check() function.
_upgrade_checks = {
# Added in Ocata
_('Cells v2'): _check_cellsv2,
# Added in Ocata
_('Placement API'): _check_placement,
}
def _get_details(self, upgrade_check_result):
if upgrade_check_result.details is not None:
# wrap the text on the details to 60 characters
return '\n'.join(textwrap.wrap(upgrade_check_result.details, 60,
subsequent_indent=' '))
def check(self):
"""Performs checks to see if the deployment is ready for upgrade.
These checks are expected to be run BEFORE services are restarted with
new code. These checks also require access to potentially all of the
Nova databases (nova, nova_api, nova_api_cell0) and external services
such as the placement API service.
:returns: UpgradeCheckCode
"""
return_code = UpgradeCheckCode.SUCCESS
# This is a list if 2-item tuples for the check name and it's results.
check_results = []
# Sort the checks by name so that we have predictable test results.
for name in sorted(self._upgrade_checks.keys()):
func = self._upgrade_checks[name]
result = func(self)
# store the result of the check for the summary table
check_results.append((name, result))
# we want to end up with the highest level code of all checks
if result.code > return_code:
return_code = result.code
# We're going to build a summary table that looks like:
# +----------------------------------------------------+
# | Upgrade Check Results |
# +----------------------------------------------------+
# | Check: Cells v2 |
# | Result: Success |
# | Details: None |
# +----------------------------------------------------+
# | Check: Placement API |
# | Result: Failure |
# | Details: There is no placement-api endpoint in the |
# | service catalog. |
# +----------------------------------------------------+
t = prettytable.PrettyTable([_('Upgrade Check Results')],
hrules=prettytable.ALL)
t.align = 'l'
for name, result in check_results:
cell = (
_('Check: %(name)s\n'
'Result: %(result)s\n'
'Details: %(details)s') %
{
'name': name,
'result': UPGRADE_CHECK_MSG_MAP[result.code],
'details': self._get_details(result),
}
)
t.add_row([cell])
print(t)
return return_code
CATEGORIES = {
'upgrade': UpgradeCommands,
}
add_command_parsers = functools.partial(cmd_common.add_command_parsers,
categories=CATEGORIES)
category_opt = cfg.SubCommandOpt('category',
title='Command categories',
help='Available categories',
handler=add_command_parsers)
def main():
"""Parse options and call the appropriate class/method."""
CONF.register_cli_opt(category_opt)
config.parse_args(sys.argv)
if CONF.category.name == "version":
print(version.version_string_with_package())
return 0
if CONF.category.name == "bash-completion":
cmd_common.print_bash_completion(CATEGORIES)
return 0
try:
fn, fn_args, fn_kwargs = cmd_common.get_action_fn()
ret = fn(*fn_args, **fn_kwargs)
return(ret)
except Exception:
print(_('Error:\n%s') % traceback.format_exc())
# This is 10 so it's not confused with the upgrade check exit codes.
return 10

1
nova/hacking/checks.py
View File

@ -672,6 +672,7 @@ def check_config_option_in_central_place(logical_line, filename):
# CLI opts are allowed to be outside of nova/conf directory
'nova/cmd/manage.py',
'nova/cmd/policy_check.py',
'nova/cmd/status.py',
# config options should not be declared in tests, but there is
# another checker for it (N320)
'nova/tests',

179
nova/tests/unit/cmd/test_status.py Normal file
View File

@ -0,0 +1,179 @@
# Copyright 2016 IBM Corp.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""
Unit tests for the nova-status CLI interfaces.
"""
import fixtures
import mock
from six.moves import StringIO
from nova.cmd import status
import nova.conf
from nova import test
CONF = nova.conf.CONF
class TestNovaStatusMain(test.NoDBTestCase):
"""Tests for the basic nova-status command infrastructure."""
def setUp(self):
super(TestNovaStatusMain, self).setUp()
self.output = StringIO()
self.useFixture(fixtures.MonkeyPatch('sys.stdout', self.output))
@mock.patch.object(status.config, 'parse_args')
@mock.patch.object(status, 'CONF')
def _check_main(self, mock_CONF, mock_parse_args,
category_name='check', expected_return_value=0):
mock_CONF.category.name = category_name
return_value = status.main()
self.assertEqual(expected_return_value, return_value)
mock_CONF.register_cli_opt.assert_called_once_with(
status.category_opt)
@mock.patch.object(status.version, 'version_string_with_package',
return_value="x.x.x")
def test_main_version(self, mock_version_string):
self._check_main(category_name='version')
self.assertEqual("x.x.x\n", self.output.getvalue())
@mock.patch.object(status.cmd_common, 'print_bash_completion')
def test_main_bash_completion(self, mock_print_bash):
self._check_main(category_name='bash-completion')
mock_print_bash.assert_called_once_with(status.CATEGORIES)
@mock.patch.object(status.cmd_common, 'get_action_fn')
def test_main(self, mock_get_action_fn):
mock_fn = mock.Mock()
mock_fn_args = [mock.sentinel.arg]
mock_fn_kwargs = {'key': mock.sentinel.value}
mock_get_action_fn.return_value = (mock_fn, mock_fn_args,
mock_fn_kwargs)
self._check_main(expected_return_value=mock_fn.return_value)
mock_fn.assert_called_once_with(mock.sentinel.arg,
key=mock.sentinel.value)
@mock.patch.object(status.cmd_common, 'get_action_fn')
def test_main_error(self, mock_get_action_fn):
mock_fn = mock.Mock(side_effect=Exception('wut'))
mock_get_action_fn.return_value = (mock_fn, [], {})
self._check_main(expected_return_value=10)
output = self.output.getvalue()
self.assertIn('Error:', output)
# assert the traceback is in the output
self.assertIn('wut', output)
class TestUpgradeCheckBasic(test.NoDBTestCase):
"""Tests for the nova-status upgrade check command.
The tests in this class should just test basic logic and use mock. Real
checks which require more elaborate fixtures or the database should be done
in separate test classes as they are more or less specific to a particular
release and may be removed in a later release after they are no longer
needed.
"""
def setUp(self):
super(TestUpgradeCheckBasic, self).setUp()
self.output = StringIO()
self.useFixture(fixtures.MonkeyPatch('sys.stdout', self.output))
self.cmd = status.UpgradeCommands()
def test_check_success(self):
fake_checks = {
'good': mock.Mock(return_value=status.UpgradeCheckResult(
status.UpgradeCheckCode.SUCCESS
)),
}
with mock.patch.object(self.cmd, '_upgrade_checks', fake_checks):
self.assertEqual(status.UpgradeCheckCode.SUCCESS, self.cmd.check())
expected = """\
+-----------------------+
| Upgrade Check Results |
+-----------------------+
| Check: good |
| Result: Success |
| Details: None |
+-----------------------+
"""
self.assertEqual(expected, self.output.getvalue())
def test_check_warning(self):
fake_checks = {
'good': mock.Mock(return_value=status.UpgradeCheckResult(
status.UpgradeCheckCode.SUCCESS
)),
'warn': mock.Mock(return_value=status.UpgradeCheckResult(
status.UpgradeCheckCode.WARNING, 'there might be a problem'
)),
}
with mock.patch.object(self.cmd, '_upgrade_checks', fake_checks):
self.assertEqual(status.UpgradeCheckCode.WARNING, self.cmd.check())
expected = """\
+-----------------------------------+
| Upgrade Check Results |
+-----------------------------------+
| Check: good |
| Result: Success |
| Details: None |
+-----------------------------------+
| Check: warn |
| Result: Warning |
| Details: there might be a problem |
+-----------------------------------+
"""
self.assertEqual(expected, self.output.getvalue())
def test_check_failure(self):
# make the error details over 60 characters so we test the wrapping
error_details = 'go back to bed' + '!' * 60
fake_checks = {
'good': mock.Mock(return_value=status.UpgradeCheckResult(
status.UpgradeCheckCode.SUCCESS
)),
'warn': mock.Mock(return_value=status.UpgradeCheckResult(
status.UpgradeCheckCode.WARNING, 'there might be a problem'
)),
'fail': mock.Mock(return_value=status.UpgradeCheckResult(
status.UpgradeCheckCode.FAILURE, error_details
)),
}
with mock.patch.object(self.cmd, '_upgrade_checks', fake_checks):
self.assertEqual(status.UpgradeCheckCode.FAILURE, self.cmd.check())
expected = """\
+-----------------------------------------------------------------------+
| Upgrade Check Results |
+-----------------------------------------------------------------------+
| Check: fail |
| Result: Failure |
| Details: go back to bed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
| !!!!!!!!!!!!!! |
+-----------------------------------------------------------------------+
| Check: good |
| Result: Success |
| Details: None |
+-----------------------------------------------------------------------+
| Check: warn |
| Result: Warning |
| Details: there might be a problem |
+-----------------------------------------------------------------------+
"""
self.assertEqual(expected, self.output.getvalue())

1
requirements.txt
View File

@ -22,6 +22,7 @@ netaddr!=0.7.16,>=0.7.13 # BSD
netifaces>=0.10.4 # MIT
paramiko>=2.0 # LGPLv2.1+
Babel>=2.3.4 # BSD
enum34;python_version=='2.7' or python_version=='2.6' or python_version=='3.3' # BSD
iso8601>=0.1.11 # MIT
jsonschema!=2.5.0,<3.0.0,>=2.0.0 # MIT
python-cinderclient!=1.7.0,!=1.7.1,>=1.6.0 # Apache-2.0

1
setup.cfg
View File

@ -67,6 +67,7 @@ console_scripts =
nova-scheduler = nova.cmd.scheduler:main
nova-serialproxy = nova.cmd.serialproxy:main
nova-spicehtml5proxy = nova.cmd.spicehtml5proxy:main
nova-status = nova.cmd.status:main
nova-xvpvncproxy = nova.cmd.xvpvncproxy:main
wsgi_scripts =
nova-placement-api = nova.api.openstack.placement.wsgi:init_application