Exit 1 when db sync runs before api_db sync

Since cells v2 was introduced, nova operators must run two commands to
migrate the database schemas of nova's databases - nova-manage api_db
sync and nova-manage db sync. It is necessary to run them in this order,
since the db sync may depend on schema changes made to the api database
in the api_db sync. Executing the db sync first may fail, for example
with the following seen in a Queens to Rocky upgrade:

nova-manage db sync
ERROR: Could not access cell0.
Has the nova_api database been created?
Has the nova_cell0 database been created?
Has "nova-manage api_db sync" been run?
Has "nova-manage cell_v2 map_cell0" been run?
Is [api_database]/connection set in nova.conf?
Is the cell0 database connection URL correct?
Error: (pymysql.err.InternalError) (1054, u"Unknown column
        'cell_mappings.disabled' in 'field list'") [SQL: u'SELECT
cell_mappings.created_at AS cell_mappings_created_at,
cell_mappings.updated_at AS cell_mappings_updated_at,
cell_mappings.id AS cell_mappings_id, cell_mappings.uuid AS
cell_mappings_uuid, cell_mappings.name AS cell_mappings_name,
cell_mappings.transport_url AS cell_mappings_transport_url,
cell_mappings.database_connection AS
cell_mappings_database_connection, cell_mappings.disabled AS
cell_mappings_disabled \nFROM cell_mappings \nWHERE
cell_mappings.uuid = %(uuid_1)s \n LIMIT %(param_1)s'] [parameters:
{u'uuid_1': '00000000-0000-0000-0000-000000000000', u'param_1': 1}]
(Background on this error at: http://sqlalche.me/e/2j85)

Despite this error, the command actually exits zero, so deployment tools
are likely to continue with the upgrade, leading to issues down the
line.

This change modifies the command to exit 1 if the cell0 sync fails.

This change also clarifies this ordering in the upgrade and nova-manage
documentation, and adds information on exit codes for the command.

Change-Id: Iff2a23e09f2c5330b8fc0e9456860b65bd6ac149
Closes-Bug: #1832860

doc/source/cli/nova-manage.rst

@@ -56,7 +56,10 @@ Nova Database
     the cell0 database). If ``--local_cell`` is specified, then only the main
     database in the current cell is upgraded. The local database connection is
     determined by ``[database]/connection`` in the configuration file passed to
-    nova-manage.
+    nova-manage. This command should be run after ``nova-manage api_db sync``.
+
+    Returns exit code 0 if the database schema was synced successfully, or 1 if
+    cell0 cannot be accessed.
 
 ``nova-manage db archive_deleted_rows [--max_rows <number>] [--verbose] [--until-complete] [--before <date>] [--purge]``
     Move deleted rows from production tables to shadow tables. Note that the
@@ -180,6 +183,9 @@ Nova API Database
     optional placement database if ``[placement_database]/connection`` is
     configured.
 
+    Returns exit code 0 if the database schema was synced successfully. This
+    command should be run before ``nova-manage db sync``.
+
 .. _man-page-cells-v2:
 
 Nova Cells v2

doc/source/user/upgrade.rst

@@ -69,10 +69,11 @@ same time.
      version of Nova, either in a venv or a separate control plane node,
      including all the python dependencies.
 
-   * Using the newly installed nova code, run the DB sync.
-     (``nova-manage api_db sync``; ``nova-manage db sync``). These schema
-     change operations should have minimal or no effect on performance, and
-     should not cause any operations to fail.
+   * Using the newly installed nova code, run the DB sync. First run
+     ``nova-manage api_db sync``, then ``nova-manage db sync``. In a multi-cell
+     environment, ``nova-manage db sync`` must currently be run in each cell.
+     These schema change operations should have minimal or no effect on
+     performance, and should not cause any operations to fail.
 
    * At this point, new columns and tables may exist in the database. These
      DB schema changes are done in a way that both the N and N+1 release can

nova/cmd/manage.py

@@ -476,6 +476,7 @@ Has "nova-manage cell_v2 map_cell0" been run?
 Is [api_database]/connection set in nova.conf?
 Is the cell0 database connection URL correct?
 Error: %s""") % six.text_type(e))
+                return 1
         return migration.db_sync(version)
 
     def version(self):

nova/tests/unit/test_nova_manage.py

@@ -728,7 +728,8 @@ Cell %s: 456
         """Asserts that a detailed error message is given when an unknown
         error occurs trying to get the cell0 cell mapping.
         """
-        self.commands.sync()
+        result = self.commands.sync()
+        self.assertEqual(1, result)
         mock_get_by_uuid.assert_called_once_with(
             test.MatchType(context.RequestContext),
             objects.CellMapping.CELL0_UUID)