From d77a3cdf6fafe3e9ed3988c35f9fa6eeee11bf28 Mon Sep 17 00:00:00 2001 From: Mike Bayer Date: Wed, 18 Nov 2020 15:14:07 -0500 Subject: [PATCH] Force MySQL / MariaDB log_warnings to 1 This MySQL / MariaDB server value was changed from the value 1 to 2 between MariaDB 10.1 and 10.2 [1]. The result of this change is that any database connection which is not gracefully closed results in a log message "Got an error reading communication packets" in the MySQL server log, which is misleading as it does not usually refer to any actionable issue; real connectivity issues are always seen in application logs and most of these messages in the server logs are likely to be false positives due to the behavior of HAProxy. While applications can reduce the occurence of this error by ensuring that database connections are gracefully closed, this is already the behavior of oslo.db and SQLAlchemy which maintains a connection pool that closes out stale connections explicitly when requests are made. The majority of these warnings are likely the result of normal HAProxy operation, where the settings "timeout client" and "timeout server" are set to 90 minutes, such that any connection older than this time will be non-gracefully closed by the proxy, generating the warning. An idle application server process will not have attended to connections that are older than the timeout period, leading to these connections being left for HAProxy to handle; HAProxy's timeout behavior leading to this message in the logs has been confirmed in local experimentation. The application server itself is never exposed to this as upon the start of work will always recycle any connection that is older than its own timeout, which defaults to 60 minutes for applications using oslo.config + oslo.db. Without HAProxy having the capability to close out these connections using MySQL's protocol, the messages are unavoidable. The message will also occur anytime an Openstack process is stopped or killed for all connections that are pooled in that process. The correct way to diagnose if an application is having connectivity issues is to look in the application server log itself for error messages and stack traces that have much more detail as to the context that produced a particular error message. This warning is also known to occur when an application server is not able to respond to packets quickly enough as has been observed with services such as Cinder where eventlet monkeypatching causes the PyMySQL client to be blocked; however when this occurs, there is an informative stack trace and error message in the application logs that shows what's going on. As this particular warning message is not useful in that most occurences will refer to normal behavior as designed, the log level should be forced to "1" to prevent these messages as they are causing confusion in downstream environments. [1] https://mariadb.com/kb/en/upgrading-from-mariadb-101-to-mariadb-102/#incompatible-changes-between-101-and-102 Change-Id: I0efb4f77aaceda635c8983d6b7a240171a7accdc --- manifests/profile/base/database/mysql.pp | 1 + manifests/profile/pacemaker/database/mysql.pp | 1 + manifests/profile/pacemaker/database/mysql_bundle.pp | 1 + 3 files changed, 3 insertions(+) diff --git a/manifests/profile/base/database/mysql.pp b/manifests/profile/base/database/mysql.pp index 73a814d49..121ac186a 100644 --- a/manifests/profile/base/database/mysql.pp +++ b/manifests/profile/base/database/mysql.pp @@ -171,6 +171,7 @@ class tripleo::profile::base::database::mysql ( 'innodb_file_per_table' => 'ON', 'innodb_log_file_size' => $innodb_log_file_size, 'innodb_lock_wait_timeout' => $innodb_lock_wait_timeout, + 'log_warnings' => '1', 'table_open_cache' => $table_open_cache, 'innodb_flush_method' => $innodb_flush_method, 'query_cache_size' => '0', diff --git a/manifests/profile/pacemaker/database/mysql.pp b/manifests/profile/pacemaker/database/mysql.pp index d3902d373..c60efd1a5 100644 --- a/manifests/profile/pacemaker/database/mysql.pp +++ b/manifests/profile/pacemaker/database/mysql.pp @@ -194,6 +194,7 @@ class tripleo::profile::pacemaker::database::mysql ( 'query_cache_size' => '0', 'query_cache_type' => '0', 'bind-address' => $bind_address, + 'log_warnings' => '1', 'max_connections' => hiera('mysql_max_connections'), 'open_files_limit' => '65536', 'wsrep_on' => 'ON', diff --git a/manifests/profile/pacemaker/database/mysql_bundle.pp b/manifests/profile/pacemaker/database/mysql_bundle.pp index 2cb78a5b3..d9994d2e9 100644 --- a/manifests/profile/pacemaker/database/mysql_bundle.pp +++ b/manifests/profile/pacemaker/database/mysql_bundle.pp @@ -296,6 +296,7 @@ class tripleo::profile::pacemaker::database::mysql_bundle ( 'query_cache_size' => '0', 'query_cache_type' => '0', 'bind-address' => $bind_address, + 'log_warnings' => '1', 'max_connections' => hiera('mysql_max_connections'), 'open_files_limit' => '65536', 'wsrep_on' => 'ON',