openstack-manuals/doc/arch-design/compute_focus/section_prescriptive_examples_compute_focus.xml

<?xml version="1.0" encoding="UTF-8"?>
<section xmlns="http://docbook.org/ns/docbook"
  xmlns:xi="http://www.w3.org/2001/XInclude"
  xmlns:xlink="http://www.w3.org/1999/xlink"
  version="5.0"
  xml:id="prescriptive-example-compute-focus">
    <?dbhtml stop-chunking?>
    <title>Prescriptive examples</title>
    <para>The Conseil Européen pour la Recherche Nucléaire (CERN),
        also known as the European Organization for Nuclear Research,
        provides particle accelerators and other infrastructure for
        high-energy physics research.</para>
    <para>As of 2011 CERN operated these two compute centers in Europe
        with plans to add a third.</para>
    <informaltable rules="all">
        <col width="40%" />
        <col width="60%" />
        <thead>
            <tr><th>Data center</th><th>Approximate capacity</th></tr>
            </thead>
        <tbody>
            <tr>
                <td>Geneva, Switzerland</td>
                <td>
                    <itemizedlist>
                        <listitem><para>3.5 Mega Watts</para></listitem>
                        <listitem><para>91000 cores</para></listitem>
                        <listitem><para>120 PB HDD</para></listitem>
                        <listitem><para>100 PB Tape</para></listitem>
                        <listitem><para>310 TB Memory</para></listitem>
                    </itemizedlist>
                </td>
            </tr>
            <tr>
                <td>Budapest, Hungary</td>
                <td>
                    <itemizedlist>
                        <listitem><para>2.5 Mega Watts</para></listitem>
                        <listitem><para>20000 cores</para></listitem>
                        <listitem><para>6 PB HDD</para></listitem>
                    </itemizedlist>
                </td>
            </tr>
        </tbody>
    </informaltable>
    <para>To support a growing number of compute-heavy users of
        experiments related to the Large Hadron Collider (LHC), CERN
        ultimately elected to deploy an OpenStack cloud using
        Scientific Linux and RDO. This effort aimed to simplify the
        management of the center's compute resources with a view to
        doubling compute capacity through the addition of a
        data center in 2013 while maintaining the same
        levels of compute staff.</para>
    <para>The CERN solution uses <glossterm baseform="cell">cells</glossterm>
        for segregation of compute
        resources and for transparently scaling between different data
        centers. This decision meant trading off support for security
        groups and live migration. In addition, they must manually replicate
        some details, like flavors, across cells. In
        spite of these drawbacks cells provide the
        required scale while exposing a single public API endpoint to
        users.</para>
    <para>CERN created a compute cell for each of the two original data
        centers and created a third when it added a new data center
        in 2013. Each cell contains three availability zones to
        further segregate compute resources and at least three
        RabbitMQ message brokers configured for clustering with
        mirrored queues for high availability.</para>
    <para>The API cell, which resides behind a HAProxy load balancer,
        is in the data center in Switzerland and directs API
        calls to compute cells using a customized variation of the
        cell scheduler. The customizations allow certain workloads to
        route to a specific data center or all data centers,
        with cell RAM availability determining cell selection in the
        latter case.</para>
    <mediaobject>
        <imageobject>
            <imagedata contentwidth="4in" fileref="../figures/Generic_CERN_Example.png"/>
        </imageobject>
    </mediaobject>
    <para>There is also some customization of the filter scheduler
        that handles placement within the cells:</para>
    <variablelist>
      <varlistentry><term>ImagePropertiesFilter</term>
        <listitem>
          <para>Provides special handling
                depending on the guest operating system in use
                (Linux-based or Windows-based).</para>
        </listitem>
      </varlistentry>
      <varlistentry><term>ProjectsToAggregateFilter</term>
        <listitem><para>Provides special
                handling depending on which project the instance is
                associated with.</para>
        </listitem>
      </varlistentry>
      <varlistentry><term>default_schedule_zones</term>
        <listitem><para>Allows the selection of
                multiple default availability zones, rather than a
                single default.</para>
        </listitem>
      </varlistentry>
    </variablelist>
    <para>A central database team manages the MySQL database server in each cell
        in an active/passive configuration with a NetApp storage back end.
        Backups run every 6 hours.</para>

    <section xml:id="network-architecture">
      <title>Network architecture</title>
    <para>To integrate with existing networking infrastructure, CERN
        made customizations to legacy networking (nova-network). This was in the
        form of a driver to integrate with CERN's existing database
        for tracking MAC and IP address assignments.</para>
    <para>The driver facilitates selection of a MAC address and IP for
        new instances based on the compute node where the scheduler places
        the instance.</para>
    <para>The driver considers the compute node where the scheduler
        placed an instance and selects a MAC address and IP
        from the pre-registered list associated with that node in the
        database. The database updates to reflect the address assignment to
        that instance.</para>
    </section>

    <section xml:id="storage-architecture">
      <title>Storage architecture</title>
    <para>CERN deploys the OpenStack Image service in the API cell and
        configures it to expose version 1 (V1) of the API. This also requires
        the image registry. The storage back end in
        use is a 3 PB Ceph cluster.</para>
    <para>CERN maintains a small set of Scientific Linux 5 and 6 images onto
        which orchestration tools can place applications. Puppet manages
        instance configuration and customization.</para>
    </section>

    <section xml:id="monitoring">
      <title>Monitoring</title>
    <para>CERN does not require direct billing, but uses the Telemetry module
        to perform metering for the purposes of adjusting
        project quotas. CERN uses a sharded, replicated, MongoDB back-end.
        To spread API load, CERN deploys instances of the nova-api service
        within the child cells for Telemetry to query
        against. This also requires the configuration of supporting services
        such as keystone, glance-api, and glance-registry in the child cells.
    </para>
    <mediaobject>
        <imageobject>
            <imagedata contentwidth="4in"
                fileref="../figures/Generic_CERN_Architecture.png"/>
        </imageobject>
    </mediaobject>
    <para>
        Additional monitoring tools in use include <link
        xlink:href="http://flume.apache.org/">Flume</link>, <link
        xlink:href="http://www.elasticsearch.org/">Elastic
        Search</link>, <link
        xlink:href="http://www.elasticsearch.org/overview/kibana/">Kibana</link>,
        and the CERN developed <link
        xlink:href="http://lemon.web.cern.ch/lemon/index.shtml">Lemon</link>
        project.
    </para>
    </section>
</section>