openstack-manuals/doc/high-availability-guide/controller/section_mysql.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE section [
<!ENTITY % openstack SYSTEM "../../common/entities/openstack.ent">
%openstack;
]>
<section xmlns="http://docbook.org/ns/docbook"
  xmlns:xi="http://www.w3.org/2001/XInclude"
  xmlns:xlink="http://www.w3.org/1999/xlink"
  version="5.0"
  xml:id="s-mysql">

          <title>Highly available MySQL</title>

        <para>MySQL is the default database server used by many OpenStack
services. Making the MySQL service highly available involves</para>
        <itemizedlist>
          <listitem>
            <para>
Configure a DRBD device for use by MySQL,
</para>
          </listitem>
          <listitem>
            <para>
Configure MySQL to use a data directory residing on that DRBD
  device,
</para>
          </listitem>
          <listitem>
            <para>
Select and assign a virtual IP address (VIP) that can freely
  float between cluster nodes,
</para>
          </listitem>
          <listitem>
            <para>
Configure MySQL to listen on that IP address,
</para>
          </listitem>
          <listitem>
            <para>
Manage all resources, including the MySQL daemon itself, with
  the Pacemaker cluster manager.
</para>
          </listitem>
        </itemizedlist>
        <note>
          <para><link xlink:href="http://galeracluster.com/">MySQL/Galera</link> is an
alternative method of configuring MySQL for high availability. It is
likely to become the preferred method of achieving MySQL high
availability once it has sufficiently matured. At the time of writing,
however, the Pacemaker/DRBD based approach remains the recommended one
for OpenStack environments.</para>
        </note>
        <section xml:id="_configure_drbd">

            <title>Configure DRBD</title>

          <para>The Pacemaker based MySQL server requires a DRBD resource from
which it mounts the <literal>/var/lib/mysql</literal> directory. In this example,
the DRBD resource is simply named <literal>mysql</literal>:</para>
          <formalpara>

              <title><literal>mysql</literal> DRBD resource configuration (<filename>/etc/drbd.d/mysql.res</filename>)</title>

            <para>
              <programlisting>resource mysql {
  device    minor 0;
  disk      "/dev/data/mysql";
  meta-disk internal;
  on node1 {
    address ipv4 10.0.42.100:7700;
  }
  on node2 {
    address ipv4 10.0.42.254:7700;
  }
}</programlisting>
            </para>
          </formalpara>
          <para>This resource uses an underlying local disk (in DRBD terminology, a
<emphasis>backing device</emphasis>) named <literal>/dev/data/mysql</literal> on both cluster nodes,
<literal>node1</literal> and <literal>node2</literal>. Normally, this would be an LVM Logical Volume
specifically set aside for this purpose. The DRBD <literal>meta-disk</literal> is
<literal>internal</literal>, meaning DRBD-specific metadata is being stored at the end
of the <literal>disk</literal> device itself. The device is configured to communicate
between IPv4 addresses 10.0.42.100 and 10.0.42.254, using TCP port
7700. Once enabled, it will map to a local DRBD block device with the
device minor number 0, that is, <filename>/dev/drbd0</filename>.</para>
          <para>Enabling a DRBD resource is explained in detail in
<link xlink:href="http://www.drbd.org/users-guide-8.3/s-first-time-up.html">the DRBD
User’s Guide</link>. In brief, the proper sequence of commands is this:</para>
          <screen><prompt>#</prompt> <userinput>drbdadm create-md mysql</userinput><co xml:id="CO3-1"/>
<prompt>#</prompt> <userinput>drbdadm up mysql</userinput><co xml:id="CO3-2"/>
<prompt>#</prompt> <userinput>drbdadm -- --force primary mysql</userinput><co xml:id="CO3-3"/></screen>
          <calloutlist>
            <callout arearefs="CO3-1">
              <para>
Initializes DRBD metadata and writes the initial set of metadata
to <literal>/dev/data/mysql</literal>. Must be completed on both nodes.
</para>
            </callout>
            <callout arearefs="CO3-2">
              <para>
Creates the <literal>/dev/drbd0</literal> device node, <emphasis>attaches</emphasis> the DRBD device
to its backing store, and <emphasis>connects</emphasis> the DRBD node to its peer. Must
be completed on both nodes.
</para>
            </callout>
            <callout arearefs="CO3-3">
              <para>
Kicks off the initial device synchronization, and puts the device
into the <literal>primary</literal> (readable and writable) role. See
<link xlink:href="http://www.drbd.org/users-guide-8.3/ch-admin.html#s-roles">Resource
roles</link> (from the DRBD User’s Guide) for a more detailed description of
the primary and secondary roles in DRBD. Must be completed <emphasis>on one
node only,</emphasis> namely the one where you are about to continue with
creating your filesystem.
</para>
            </callout>
          </calloutlist>
        </section>
        <section xml:id="_creating_a_file_system">

            <title>Creating a file system</title>

          <para>Once the DRBD resource is running and in the primary role (and
potentially still in the process of running the initial device
synchronization), you may proceed with creating the filesystem for
MySQL data. XFS is the generally recommended filesystem due to its journaling, efficient allocation, and performance:</para>
          <screen><prompt>#</prompt> <userinput>mkfs -t xfs /dev/drbd0</userinput></screen>
          <para>You may also use the alternate device path for the DRBD device, which
may be easier to remember as it includes the self-explanatory resource
name:</para>
          <screen><prompt>#</prompt> <userinput>mkfs -t xfs /dev/drbd/by-res/mysql</userinput></screen>
          <para>Once completed, you may safely return the device to the secondary
role. Any ongoing device synchronization will continue in the
background:</para>
          <screen><prompt>#</prompt> <userinput>drbdadm secondary mysql</userinput></screen>
        </section>
        <section xml:id="_prepare_mysql_for_pacemaker_high_availability">

            <title>Prepare MySQL for Pacemaker high availability</title>

          <para>In order for Pacemaker monitoring to function properly, you must
ensure that MySQL’s database files reside on the DRBD device. If you
already have an existing MySQL database, the simplest approach is to
just move the contents of the existing <literal>/var/lib/mysql</literal> directory into
the newly created filesystem on the DRBD device.</para>
          <warning>
            <para>You must complete the next step while the MySQL database
server is shut down.</para>
          </warning>
          <screen><prompt>#</prompt> <userinput>mount /dev/drbd/by-res/mysql /mnt</userinput>
<prompt>#</prompt> <userinput>mv /var/lib/mysql/* /mnt</userinput>
<prompt>#</prompt> <userinput>umount /mnt</userinput></screen>
          <para>For a new MySQL installation with no existing data, you may also run
the <literal>mysql_install_db</literal> command:</para>
          <screen><prompt>#</prompt> <userinput>mount /dev/drbd/by-res/mysql /mnt</userinput>
<prompt>#</prompt> <userinput>mysql_install_db --datadir=/mnt</userinput>
<prompt>#</prompt> <userinput>umount /mnt</userinput></screen>
          <para>Regardless of the approach, the steps outlined here must be completed
on only one cluster node.</para>
        </section>
        <section xml:id="_add_mysql_resources_to_pacemaker">

            <title>Add MySQL resources to Pacemaker</title>

          <para>You can now add the Pacemaker configuration for
MySQL resources. Connect to the Pacemaker cluster with <literal>crm
configure</literal>, and add the following cluster resources:</para>
          <programlisting>primitive p_ip_mysql ocf:heartbeat:IPaddr2 \
  params ip="192.168.42.101" cidr_netmask="24" \
  op monitor interval="30s"
primitive p_drbd_mysql ocf:linbit:drbd \
  params drbd_resource="mysql" \
  op start timeout="90s" \
  op stop timeout="180s" \
  op promote timeout="180s" \
  op demote timeout="180s" \
  op monitor interval="30s" role="Slave" \
  op monitor interval="29s" role="Master"
primitive p_fs_mysql ocf:heartbeat:Filesystem \
  params device="/dev/drbd/by-res/mysql" \
    directory="/var/lib/mysql" \
    fstype="xfs" \
    options="relatime" \
  op start timeout="60s" \
  op stop timeout="180s" \
  op monitor interval="60s" timeout="60s"
primitive p_mysql ocf:heartbeat:mysql \
  params additional_parameters="--bind-address=192.168.42.101"
    config="/etc/mysql/my.cnf" \
    pid="/var/run/mysqld/mysqld.pid" \
    socket="/var/run/mysqld/mysqld.sock" \
    log="/var/log/mysql/mysqld.log" \
  op monitor interval="20s" timeout="10s" \
  op start timeout="120s" \
  op stop timeout="120s"
group g_mysql p_ip_mysql p_fs_mysql p_mysql
ms ms_drbd_mysql p_drbd_mysql \
  meta notify="true" clone-max="2"
colocation c_mysql_on_drbd inf: g_mysql ms_drbd_mysql:Master
order o_drbd_before_mysql inf: ms_drbd_mysql:promote g_mysql:start</programlisting>
          <para>This configuration creates</para>
          <itemizedlist>
            <listitem>
              <para><literal>p_ip_mysql</literal>, a virtual IP address for use by MySQL
  (192.168.42.101),
</para>
            </listitem>
            <listitem>
              <para><literal>p_fs_mysql</literal>, a Pacemaker managed filesystem mounted to
  <literal>/var/lib/mysql</literal> on whatever node currently runs the MySQL
  service,
</para>
            </listitem>
            <listitem>
              <para><literal>ms_drbd_mysql</literal>, the <emphasis>master/slave set</emphasis> managing the <literal>mysql</literal>
  DRBD resource,
</para>
            </listitem>
            <listitem>
              <para>
a service <literal>group</literal> and <literal>order</literal> and <literal>colocation</literal> constraints to ensure
  resources are started on the correct nodes, and in the correct sequence.
</para>
            </listitem>
          </itemizedlist>
          <para><literal>crm configure</literal> supports batch input, so you may copy and paste the
above into your live pacemaker configuration, and then make changes as
required. For example, you may enter <literal>edit p_ip_mysql</literal> from the
<literal>crm configure</literal> menu and edit the resource to match your preferred
virtual IP address.</para>
          <para>Once completed, commit your configuration changes by entering <literal>commit</literal>
from the <literal>crm configure</literal> menu. Pacemaker will then start the MySQL
service, and its dependent resources, on one of your nodes.</para>
        </section>
        <section xml:id="_configure_openstack_services_for_highly_available_mysql">

            <title>Configure OpenStack services for highly available MySQL</title>

          <para>Your OpenStack services must now point their MySQL configuration to
the highly available, virtual cluster IP address&mdash;rather than a
MySQL server’s physical IP address as you normally would.</para>
          <para>For OpenStack Image, for example, if your MySQL service IP address is
192.168.42.101 as in the configuration explained here, you would use
the following line in your OpenStack Image registry configuration file
(<filename>glance-registry.conf</filename>):</para>
          <programlisting language="ini">sql_connection = mysql://glancedbadmin:&lt;password&gt;@192.168.42.101/glance</programlisting>
          <para>No other changes are necessary to your OpenStack configuration. If the
node currently hosting your database experiences a problem
necessitating service failover, your OpenStack services may experience
a brief MySQL interruption, as they would in the event of a network
hiccup, and then continue to run normally.</para>
        </section>
      </section>