Fix series-upgrade issue where seeded file goes missing

During resume on a non-leader unit, the 'seeded' file can go missing
when the package syncs over to the /var/lib/mysql or
/var/lib/percona-xtradb-cluster directories (vivid+).  It's not really
clear why it doesn't do this every time (i.e. not every non-leader unit
fails), but this fix ensures that if the unit *is* seeded prior to the
series-upgrade, then it stays seeded after the series upgrade.

The related zaza-openstack-tests change [1] is about fixing the
series-upgrade test.

Note there is no trusty-mitaka test here as the charm doesn't support
trusty.  The last version of percona-cluster in the charm store that
supports trusty is rev. 276

[1]: https://github.com/openstack-charmers/zaza-openstack-tests/pull/406

Change-Id: I628be1c24081d7e0e150e5064c5fa4ab694632e9
Closes-bug: #1868326

hooks/percona_utils.py

@@ -682,11 +682,23 @@ def cluster_in_sync():
     return False
 
 
-def charm_check_func():
+def charm_check_func(ensure_seeded=False):
     """Custom function to assess the status of the current unit
 
-    @returns (status, message) - tuple of strings if an issue
+    :param ensure_seeded: if set, then ensure that the seeded file exists.
+        This is to work around bug: #1868326   where the resume action to
+        manage_payload_services causes the seeded file to be removed under
+        certain circumstances.  This is one of the only places where we can
+        actually re-assert it prior to the resume failing, as this function
+        gets a chance to run after management_payload_services() but prior to
+        the actual
+    :type ensure_seeded: bool
+    :returns: (status, message)
+    :rtype: Tuple[str, str]
     """
+    if ensure_seeded and not seeded():
+        log("'seeded' file is missing but should exists; putting it back.")
+        mark_seeded()
     if is_unit_upgrading_set():
         # Avoid looping through attempting to determine cluster_in_sync
         return ("blocked", "Unit upgrading.")
@@ -830,9 +842,14 @@ def assess_status_func(configs):
     @param configs: a templating.OSConfigRenderer() object
     @return f() -> None : a function that assesses the unit's workload status
     """
+    # BUG: 1868326 - the resume action on the mysql service can wipe out the
+    # seeded file; if it is seeded, then we put it back. This function is
+    # called as part of the resume_unit() call and is one of the only places to
+    # actually do and fix this check.
+    is_seeded = seeded()
     return make_assess_status_func(
         configs, REQUIRED_INTERFACES,
-        charm_func=lambda _: charm_check_func(),
+        charm_func=lambda _: charm_check_func(ensure_seeded=is_seeded),
         services=services(), ports=None)
 
 

tests/bundles/overlays/xenial-queens-ha-series-upgrade.yaml.j2

@@ -0,0 +1 @@
+ha.yaml

tests/bundles/xenial-queens-ha-series-upgrade.yaml

@@ -0,0 +1,15 @@
+series: xenial
+relations:
+- - keystone:shared-db
+  - percona-cluster:shared-db
+applications:
+  percona-cluster:
+    series: xenial
+    charm: ../../../percona-cluster
+    num_units: 1
+  keystone:
+    charm: cs:~openstack-charmers-next/keystone
+    num_units: 1
+    options:
+      openstack-origin: cloud:xenial-queens
+      token-expiration: 60

tests/tests.yaml

@@ -7,6 +7,8 @@ tests:
  - xenial_model:
    - zaza.openstack.charm_tests.mysql.tests.PerconaClusterCharmTests
    - zaza.openstack.charm_tests.mysql.tests.PerconaClusterScaleTests
+ - xenial_series_upgrade_model:
+   - zaza.openstack.charm_tests.series_upgrade.tests.XenialBionicSeriesUpgrade
 configure:
   - zaza.model.block_until_all_units_idle
 target_deploy_status:
@@ -16,3 +18,4 @@ gate_bundles:
 smoke_bundles:
   - bionic_model: bionic-ha
 dev_bundles:
+  - xenial_series_upgrade_model: xenial-queens-ha-series-upgrade