Add nova-status upgrade check command framework

This adds the basic framework for the nova-status
upgrade check commands. Follow up patches will flesh
this out to perform cells v2 and placement API related
upgrade status checks.

A man page is added for the new CLI and as part of that
the man page index is sorted.

Part of blueprint cells-scheduling-interaction
Part of blueprint resource-providers-scheduler-db-filters

Change-Id: I687dd7317703a1bb76c197ebba849ca368c0872e

doc/source/man/index.rst

@@ -41,7 +41,7 @@ Reference
    nova-novncproxy
    nova-rootwrap
    nova-scheduler
-   nova-spicehtml5proxy
-   nova-xvpvncproxy
    nova-serialproxy
-
+   nova-spicehtml5proxy
+   nova-status
+   nova-xvpvncproxy

doc/source/man/nova-status.rst

@@ -0,0 +1,96 @@
+===========
+nova-status
+===========
+
+--------------------------------------
+CLI interface for nova status commands
+--------------------------------------
+
+:Author: openstack@lists.openstack.org
+:Date:   2016-12-16
+:Copyright: OpenStack Foundation
+:Version: 15.0.0
+:Manual section: 1
+:Manual group: cloud computing
+
+SYNOPSIS
+========
+
+  nova-status <category> <action> [<args>]
+
+DESCRIPTION
+===========
+
+The nova-status command provides routines for checking the status of a Nova
+deployment.
+
+OPTIONS
+=======
+
+The standard pattern for executing a nova-status command is::
+
+    nova-status <category> <command> [<args>]
+
+Run without arguments to see a list of available command categories::
+
+    nova-status
+
+Categories are:
+
+* upgrade
+
+Detailed descriptions are below.
+
+You can also run with a category argument such as "upgrade" to see a list of
+all commands in that category::
+
+    nova-status upgrade
+
+These sections describe the available categories and arguments for nova-status.
+
+Upgrade
+~~~~~~~
+
+``nova-status upgrade check``
+
+  Performs a release-specific readiness check before restarting services with
+  new code. This command expects to have complete configuration and access
+  to databases and services within a cell. For example, this check may query
+  the Nova API database and one or more cell databases. It may also make
+  requests to other services such as the Placement REST API via the Keystone
+  service catalog.
+
+  **Return Codes**
+
+  0) All upgrade readiness checks passed successfully and there is nothing
+     to do.
+  1) At least one check encountered an issue and requires further
+     investigation. This is considered a warning but the upgrade may be OK.
+  2) There was an upgrade status check failure that needs to be
+     investigated. This should be considered something that stops an
+     upgrade.
+
+  **History of Checks**
+
+  **15.0.0 (Ocata)**
+
+  * Checks are added for cells v2 so ``nova-status upgrade check`` should be
+    run *after* running the ``nova-manage cell_v2 simple_cell_setup``
+    command.
+  * Checks are added for the Placement API such that there is an endpoint in
+    the Keystone service catalog, the service is running and the check can
+    make a successful request to the endpoint. The command also checks to
+    see that there are compute node resource providers checking in with the
+    Placement service. More information on the Placement service can be found
+    at: `<http://docs.openstack.org/developer/nova/placement.html>`_
+
+
+SEE ALSO
+========
+
+* OpenStack Nova Docs: `<http://docs.openstack.org/developer/nova/>`_
+
+BUGS
+====
+
+* Nova bugs are managed at Launchpad: `<https://bugs.launchpad.net/nova>`_

nova/cmd/status.py

@@ -0,0 +1,209 @@
+# Copyright 2016 IBM Corp.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""
+CLI interface for nova status commands.
+"""
+
+from __future__ import print_function
+import functools
+import sys
+import textwrap
+import traceback
+
+# enum comes from the enum34 package if python < 3.4, else it's stdlib
+import enum
+from oslo_config import cfg
+import prettytable
+
+from nova.cmd import common as cmd_common
+import nova.conf
+from nova import config
+from nova.i18n import _
+from nova import version
+
+CONF = nova.conf.CONF
+
+
+class UpgradeCheckCode(enum.IntEnum):
+    """These are the status codes for the nova-status upgrade check command
+    and internal check commands.
+    """
+
+    # All upgrade readiness checks passed successfully and there is
+    # nothing to do.
+    SUCCESS = 0
+
+    # At least one check encountered an issue and requires further
+    # investigation. This is considered a warning but the upgrade may be OK.
+    WARNING = 1
+
+    # There was an upgrade status check failure that needs to be
+    # investigated. This should be considered something that stops an upgrade.
+    FAILURE = 2
+
+
+UPGRADE_CHECK_MSG_MAP = {
+    UpgradeCheckCode.SUCCESS: _('Success'),
+    UpgradeCheckCode.WARNING: _('Warning'),
+    UpgradeCheckCode.FAILURE: _('Failure'),
+}
+
+
+class UpgradeCheckResult(object):
+    """Class used for 'nova-status upgrade check' results.
+
+    The 'code' attribute is an UpgradeCheckCode enum.
+    The 'details' attribute is a translated message generally only used for
+    checks that result in a warning or failure code. The details should provide
+    information on what issue was discovered along with any remediation.
+    """
+
+    def __init__(self, code, details=None):
+        super(UpgradeCheckResult, self).__init__()
+        self.code = code
+        self.details = details
+
+
+class UpgradeCommands(object):
+    """Commands related to upgrades.
+
+    The subcommands here must not rely on the nova object model since they
+    should be able to run on n-1 data. Any queries to the database should be
+    done through the sqlalchemy query language directly like the database
+    schema migrations.
+    """
+
+    def _check_cellsv2(self):
+        # TODO(mriedem): perform the same checks as the 030_require_cell_setup
+        # API DB migration.
+        return UpgradeCheckResult(UpgradeCheckCode.SUCCESS)
+
+    def _check_placement(self):
+        # TODO(mriedem): check to see that the placement API service is running
+        # and we can connect to it, and that the number of resource providers
+        # in the placement service is greater than or equal to the number of
+        # compute nodes in the database.
+        return UpgradeCheckResult(UpgradeCheckCode.SUCCESS)
+
+    # The format of the check functions is to return an UpgradeCheckResult
+    # object with the appropriate UpgradeCheckCode and details set. If the
+    # check hits warnings or failures then those should be stored in the
+    # returned UpgradeCheckResult's "details" attribute. The summary will
+    # be rolled up at the end of the check() function.
+    _upgrade_checks = {
+        # Added in Ocata
+        _('Cells v2'): _check_cellsv2,
+        # Added in Ocata
+        _('Placement API'): _check_placement,
+    }
+
+    def _get_details(self, upgrade_check_result):
+        if upgrade_check_result.details is not None:
+            # wrap the text on the details to 60 characters
+            return '\n'.join(textwrap.wrap(upgrade_check_result.details, 60,
+                                           subsequent_indent='  '))
+
+    def check(self):
+        """Performs checks to see if the deployment is ready for upgrade.
+
+        These checks are expected to be run BEFORE services are restarted with
+        new code. These checks also require access to potentially all of the
+        Nova databases (nova, nova_api, nova_api_cell0) and external services
+        such as the placement API service.
+
+        :returns: UpgradeCheckCode
+        """
+        return_code = UpgradeCheckCode.SUCCESS
+        # This is a list if 2-item tuples for the check name and it's results.
+        check_results = []
+        # Sort the checks by name so that we have predictable test results.
+        for name in sorted(self._upgrade_checks.keys()):
+            func = self._upgrade_checks[name]
+            result = func(self)
+            # store the result of the check for the summary table
+            check_results.append((name, result))
+            # we want to end up with the highest level code of all checks
+            if result.code > return_code:
+                return_code = result.code
+
+        # We're going to build a summary table that looks like:
+        # +----------------------------------------------------+
+        # | Upgrade Check Results                              |
+        # +----------------------------------------------------+
+        # | Check: Cells v2                                    |
+        # | Result: Success                                    |
+        # | Details: None                                      |
+        # +----------------------------------------------------+
+        # | Check: Placement API                               |
+        # | Result: Failure                                    |
+        # | Details: There is no placement-api endpoint in the |
+        # |          service catalog.                          |
+        # +----------------------------------------------------+
+        t = prettytable.PrettyTable([_('Upgrade Check Results')],
+                                    hrules=prettytable.ALL)
+        t.align = 'l'
+        for name, result in check_results:
+            cell = (
+                _('Check: %(name)s\n'
+                  'Result: %(result)s\n'
+                  'Details: %(details)s') %
+                {
+                    'name': name,
+                    'result': UPGRADE_CHECK_MSG_MAP[result.code],
+                    'details': self._get_details(result),
+                }
+            )
+            t.add_row([cell])
+        print(t)
+
+        return return_code
+
+
+CATEGORIES = {
+    'upgrade': UpgradeCommands,
+}
+
+
+add_command_parsers = functools.partial(cmd_common.add_command_parsers,
+                                        categories=CATEGORIES)
+
+
+category_opt = cfg.SubCommandOpt('category',
+                                 title='Command categories',
+                                 help='Available categories',
+                                 handler=add_command_parsers)
+
+
+def main():
+    """Parse options and call the appropriate class/method."""
+    CONF.register_cli_opt(category_opt)
+    config.parse_args(sys.argv)
+
+    if CONF.category.name == "version":
+        print(version.version_string_with_package())
+        return 0
+
+    if CONF.category.name == "bash-completion":
+        cmd_common.print_bash_completion(CATEGORIES)
+        return 0
+
+    try:
+        fn, fn_args, fn_kwargs = cmd_common.get_action_fn()
+        ret = fn(*fn_args, **fn_kwargs)
+        return(ret)
+    except Exception:
+        print(_('Error:\n%s') % traceback.format_exc())
+        # This is 10 so it's not confused with the upgrade check exit codes.
+        return 10

nova/hacking/checks.py

@@ -672,6 +672,7 @@ def check_config_option_in_central_place(logical_line, filename):
         # CLI opts are allowed to be outside of nova/conf directory
         'nova/cmd/manage.py',
         'nova/cmd/policy_check.py',
+        'nova/cmd/status.py',
         # config options should not be declared in tests, but there is
         # another checker for it (N320)
         'nova/tests',

nova/tests/unit/cmd/test_status.py

@@ -0,0 +1,179 @@
+# Copyright 2016 IBM Corp.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""
+Unit tests for the nova-status CLI interfaces.
+"""
+
+import fixtures
+import mock
+from six.moves import StringIO
+
+from nova.cmd import status
+import nova.conf
+from nova import test
+
+CONF = nova.conf.CONF
+
+
+class TestNovaStatusMain(test.NoDBTestCase):
+    """Tests for the basic nova-status command infrastructure."""
+
+    def setUp(self):
+        super(TestNovaStatusMain, self).setUp()
+        self.output = StringIO()
+        self.useFixture(fixtures.MonkeyPatch('sys.stdout', self.output))
+
+    @mock.patch.object(status.config, 'parse_args')
+    @mock.patch.object(status, 'CONF')
+    def _check_main(self, mock_CONF, mock_parse_args,
+                    category_name='check', expected_return_value=0):
+        mock_CONF.category.name = category_name
+        return_value = status.main()
+
+        self.assertEqual(expected_return_value, return_value)
+        mock_CONF.register_cli_opt.assert_called_once_with(
+            status.category_opt)
+
+    @mock.patch.object(status.version, 'version_string_with_package',
+                       return_value="x.x.x")
+    def test_main_version(self, mock_version_string):
+        self._check_main(category_name='version')
+        self.assertEqual("x.x.x\n", self.output.getvalue())
+
+    @mock.patch.object(status.cmd_common, 'print_bash_completion')
+    def test_main_bash_completion(self, mock_print_bash):
+        self._check_main(category_name='bash-completion')
+        mock_print_bash.assert_called_once_with(status.CATEGORIES)
+
+    @mock.patch.object(status.cmd_common, 'get_action_fn')
+    def test_main(self, mock_get_action_fn):
+        mock_fn = mock.Mock()
+        mock_fn_args = [mock.sentinel.arg]
+        mock_fn_kwargs = {'key': mock.sentinel.value}
+        mock_get_action_fn.return_value = (mock_fn, mock_fn_args,
+                                           mock_fn_kwargs)
+
+        self._check_main(expected_return_value=mock_fn.return_value)
+        mock_fn.assert_called_once_with(mock.sentinel.arg,
+                                        key=mock.sentinel.value)
+
+    @mock.patch.object(status.cmd_common, 'get_action_fn')
+    def test_main_error(self, mock_get_action_fn):
+        mock_fn = mock.Mock(side_effect=Exception('wut'))
+        mock_get_action_fn.return_value = (mock_fn, [], {})
+
+        self._check_main(expected_return_value=10)
+        output = self.output.getvalue()
+        self.assertIn('Error:', output)
+        # assert the traceback is in the output
+        self.assertIn('wut', output)
+
+
+class TestUpgradeCheckBasic(test.NoDBTestCase):
+    """Tests for the nova-status upgrade check command.
+
+    The tests in this class should just test basic logic and use mock. Real
+    checks which require more elaborate fixtures or the database should be done
+    in separate test classes as they are more or less specific to a particular
+    release and may be removed in a later release after they are no longer
+    needed.
+    """
+
+    def setUp(self):
+        super(TestUpgradeCheckBasic, self).setUp()
+        self.output = StringIO()
+        self.useFixture(fixtures.MonkeyPatch('sys.stdout', self.output))
+        self.cmd = status.UpgradeCommands()
+
+    def test_check_success(self):
+        fake_checks = {
+            'good': mock.Mock(return_value=status.UpgradeCheckResult(
+                status.UpgradeCheckCode.SUCCESS
+            )),
+        }
+        with mock.patch.object(self.cmd, '_upgrade_checks', fake_checks):
+            self.assertEqual(status.UpgradeCheckCode.SUCCESS, self.cmd.check())
+        expected = """\
++-----------------------+
+| Upgrade Check Results |
++-----------------------+
+| Check: good           |
+| Result: Success       |
+| Details: None         |
++-----------------------+
+"""
+        self.assertEqual(expected, self.output.getvalue())
+
+    def test_check_warning(self):
+        fake_checks = {
+            'good': mock.Mock(return_value=status.UpgradeCheckResult(
+                status.UpgradeCheckCode.SUCCESS
+            )),
+            'warn': mock.Mock(return_value=status.UpgradeCheckResult(
+                status.UpgradeCheckCode.WARNING, 'there might be a problem'
+            )),
+        }
+        with mock.patch.object(self.cmd, '_upgrade_checks', fake_checks):
+            self.assertEqual(status.UpgradeCheckCode.WARNING, self.cmd.check())
+        expected = """\
++-----------------------------------+
+| Upgrade Check Results             |
++-----------------------------------+
+| Check: good                       |
+| Result: Success                   |
+| Details: None                     |
++-----------------------------------+
+| Check: warn                       |
+| Result: Warning                   |
+| Details: there might be a problem |
++-----------------------------------+
+"""
+        self.assertEqual(expected, self.output.getvalue())
+
+    def test_check_failure(self):
+        # make the error details over 60 characters so we test the wrapping
+        error_details = 'go back to bed' + '!' * 60
+        fake_checks = {
+            'good': mock.Mock(return_value=status.UpgradeCheckResult(
+                status.UpgradeCheckCode.SUCCESS
+            )),
+            'warn': mock.Mock(return_value=status.UpgradeCheckResult(
+                status.UpgradeCheckCode.WARNING, 'there might be a problem'
+            )),
+            'fail': mock.Mock(return_value=status.UpgradeCheckResult(
+                status.UpgradeCheckCode.FAILURE, error_details
+            )),
+        }
+        with mock.patch.object(self.cmd, '_upgrade_checks', fake_checks):
+            self.assertEqual(status.UpgradeCheckCode.FAILURE, self.cmd.check())
+        expected = """\
++-----------------------------------------------------------------------+
+| Upgrade Check Results                                                 |
++-----------------------------------------------------------------------+
+| Check: fail                                                           |
+| Result: Failure                                                       |
+| Details: go back to bed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
+|   !!!!!!!!!!!!!!                                                      |
++-----------------------------------------------------------------------+
+| Check: good                                                           |
+| Result: Success                                                       |
+| Details: None                                                         |
++-----------------------------------------------------------------------+
+| Check: warn                                                           |
+| Result: Warning                                                       |
+| Details: there might be a problem                                     |
++-----------------------------------------------------------------------+
+"""
+        self.assertEqual(expected, self.output.getvalue())

requirements.txt

@@ -22,6 +22,7 @@ netaddr!=0.7.16,>=0.7.13 # BSD
 netifaces>=0.10.4 # MIT
 paramiko>=2.0 # LGPLv2.1+
 Babel>=2.3.4 # BSD
+enum34;python_version=='2.7' or python_version=='2.6' or python_version=='3.3'  # BSD
 iso8601>=0.1.11 # MIT
 jsonschema!=2.5.0,<3.0.0,>=2.0.0 # MIT
 python-cinderclient!=1.7.0,!=1.7.1,>=1.6.0 # Apache-2.0

setup.cfg

@@ -67,6 +67,7 @@ console_scripts =
     nova-scheduler = nova.cmd.scheduler:main
     nova-serialproxy = nova.cmd.serialproxy:main
     nova-spicehtml5proxy = nova.cmd.spicehtml5proxy:main
+    nova-status = nova.cmd.status:main
     nova-xvpvncproxy = nova.cmd.xvpvncproxy:main
 wsgi_scripts =
     nova-placement-api = nova.api.openstack.placement.wsgi:init_application