Allow for invalid packet sequence in keepalive

In the SQLAlchemy keep_alive class, MariaDB is failing
as pymysql reports an invalid packet sequence.
MariaDB seems to timeout the client in a different
way than MySQL and PXC, which manifests itself as the
aforementioned invalid sequence.  It is now handled
as a special-case exception.

With this fix, the MariaDB scenario tests now pass.

The scenario tests were also tweaked a bit, which aided
in the testing of the fix.  'group=instance' was created,
plus instance_error properly interleaved with
instance_create.  _has_status now calls get_instance with
the admin client so that any faults are accompanied by
a relevant stack trace.  Cases where the result code
was being checked out-of-sequence were removed, and explicit
calls to check the http code for the right client were added.

The replication error messages for promote and eject were
enhanced as well to attempt to debug spurious failures.
One of those failures was 'Replication is not on after 60 seconds.'
This was fixed by setting 'MASTER_CONNECT_RETRY' in the mariadb
gtid replication strategy as was done in:
https://review.openstack.org/#/c/188933

Finally, backup_incremental was added to MariaDB supported
groups and cleaned up elsewhere.

Closes-Bug: #1621702
Change-Id: Id6bde5a34e1d79eece3084f761dcd153c38ccbad
This commit is contained in:
Peter Stachowski
2016-08-29 19:47:47 +00:00
parent 8ba72a5f3f
commit bd761989ee
18 changed files with 240 additions and 152 deletions

View File

@@ -25,6 +25,7 @@ import uuid
from oslo_log import log as logging
from oslo_utils import encodeutils
from pymysql import err as pymysql_err
from six.moves import urllib
import sqlalchemy
from sqlalchemy import exc
@@ -568,6 +569,14 @@ class BaseKeepAliveConnection(interfaces.PoolListener):
raise exc.DisconnectionError()
else:
raise
# MariaDB seems to timeout the client in a different
# way than MySQL and PXC, which manifests itself as
# an invalid packet sequence. Handle it as well.
except pymysql_err.InternalError as ex:
if "Packet sequence number wrong" in ex.message:
raise exc.DisconnectionError()
else:
raise
@six.add_metaclass(abc.ABCMeta)