236 lines
14 KiB
XML
236 lines
14 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
||
<section xmlns="http://docbook.org/ns/docbook"
|
||
xmlns:xi="http://www.w3.org/2001/XInclude"
|
||
xmlns:xlink="http://www.w3.org/1999/xlink"
|
||
version="5.0"
|
||
xml:id="section_objectstorage-components">
|
||
<title>Components</title>
|
||
<para>The components that enable Object Storage to deliver high availability, high
|
||
durability, and high concurrency are:</para>
|
||
<itemizedlist>
|
||
<listitem>
|
||
<para><emphasis role="bold">Proxy servers.</emphasis> Handle all of the incoming
|
||
API requests.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Rings.</emphasis> Map logical names of data to
|
||
locations on particular disks.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Zones.</emphasis> Isolate data from other zones. A
|
||
failure in one zone doesn’t impact the rest of the cluster because data is
|
||
replicated across zones.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Accounts and containers.</emphasis> Each account and
|
||
container are individual databases that are distributed across the cluster. An
|
||
account database contains the list of containers in that account. A container
|
||
database contains the list of objects in that container.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Objects.</emphasis> The data itself.</para>
|
||
</listitem>
|
||
<listitem>
|
||
<para><emphasis role="bold">Partitions.</emphasis> A partition stores objects,
|
||
account databases, and container databases and helps manage locations where data
|
||
lives in the cluster.</para>
|
||
</listitem>
|
||
</itemizedlist>
|
||
<figure>
|
||
<title>Object Storage building blocks</title>
|
||
<mediaobject>
|
||
<imageobject>
|
||
<imagedata fileref="../common/figures/objectstorage-buildingblocks.png"/>
|
||
</imageobject>
|
||
</mediaobject>
|
||
</figure>
|
||
<section xml:id="section_proxy-servers">
|
||
<title>Proxy servers</title>
|
||
<para>Proxy servers are the public face of Object Storage and handle all of the incoming API
|
||
requests. Once a proxy server receives a request, it determines the storage node based
|
||
on the object's URL, for example, https://swift.example.com/v1/account/container/object.
|
||
Proxy servers also coordinate responses, handle failures, and coordinate
|
||
timestamps.</para>
|
||
<para>Proxy servers use a shared-nothing architecture and can be scaled as needed based on
|
||
projected workloads. A minimum of two proxy servers should be deployed for redundancy.
|
||
If one proxy server fails, the others take over.</para>
|
||
</section>
|
||
<section xml:id="section_ring">
|
||
<title>Rings</title>
|
||
<para>A ring represents a mapping between the names of entities stored on disk and their
|
||
physical locations. There are separate rings for accounts, containers, and objects. When
|
||
other components need to perform any operation on an object, container, or account, they
|
||
need to interact with the appropriate ring to determine their location in the
|
||
cluster.</para>
|
||
<para>The ring maintains this mapping using zones, devices, partitions, and replicas. Each
|
||
partition in the ring is replicated, by default, three times across the cluster, and
|
||
partition locations are stored in the mapping maintained by the ring. The ring is also
|
||
responsible for determining which devices are used for handoff in failure
|
||
scenarios.</para>
|
||
<para>Data can be isolated into zones in the ring. Each partition replica is guaranteed to
|
||
reside in a different zone. A zone could represent a drive, a server, a cabinet, a
|
||
switch, or even a data center.</para>
|
||
<para>The partitions of the ring are equally divided among all of the devices in the Object
|
||
Storage installation. When partitions need to be moved around (for example, if a device
|
||
is added to the cluster), the ring ensures that a minimum number of partitions are moved
|
||
at a time, and only one replica of a partition is moved at a time.</para>
|
||
<para>You can use weights to balance the distribution of partitions on drives across the
|
||
cluster. This can be useful, for example, when differently sized drives are used in a
|
||
cluster.</para>
|
||
<para>The ring is used by the proxy server and several background processes (like
|
||
replication).</para>
|
||
<figure>
|
||
<title>The <emphasis role="bold">ring</emphasis></title>
|
||
<mediaobject>
|
||
<imageobject>
|
||
<imagedata fileref="../common/figures/objectstorage-ring.png"/>
|
||
</imageobject>
|
||
</mediaobject>
|
||
</figure>
|
||
<para>These rings are externally managed, in that the server processes themselves do not
|
||
modify the rings, they are instead given new rings modified by other tools.</para>
|
||
<para>The ring uses a configurable number of bits from an
|
||
MD5 hash for a path as a partition index that designates a
|
||
device. The number of bits kept from the hash is known as
|
||
the partition power, and 2 to the partition power
|
||
indicates the partition count. Partitioning the full MD5
|
||
hash ring allows other parts of the cluster to work in
|
||
batches of items at once which ends up either more
|
||
efficient or at least less complex than working with each
|
||
item separately or the entire cluster all at once.</para>
|
||
<para>Another configurable value is the replica count, which indicates how many of the
|
||
partition-device assignments make up a single ring. For a given partition number, each
|
||
replica’s device will not be in the same zone as any other replica's device. Zones can
|
||
be used to group devices based on physical locations, power separations, network
|
||
separations, or any other attribute that would improve the availability of multiple
|
||
replicas at the same time.</para>
|
||
</section>
|
||
<section xml:id="section_zones">
|
||
<title>Zones</title>
|
||
<para>Object Storage allows configuring zones in order to isolate failure boundaries.
|
||
Each data replica resides in a separate zone, if possible. At the smallest level, a zone
|
||
could be a single drive or a grouping of a few drives. If there were five object storage
|
||
servers, then each server would represent its own zone. Larger deployments would have an
|
||
entire rack (or multiple racks) of object servers, each representing a zone. The goal of
|
||
zones is to allow the cluster to tolerate significant outages of storage servers without
|
||
losing all replicas of the data.</para>
|
||
<para>As mentioned earlier, everything in Object Storage is stored, by default, three
|
||
times. Swift will place each replica "as-uniquely-as-possible" to ensure both high
|
||
availability and high durability. This means that when chosing a replica location,
|
||
Object Storage chooses a server in an unused zone before an unused server in a zone that
|
||
already has a replica of the data.</para>
|
||
<figure>
|
||
<title>Zones</title>
|
||
<mediaobject>
|
||
<imageobject>
|
||
<imagedata fileref="../common/figures/objectstorage-zones.png"/>
|
||
</imageobject>
|
||
</mediaobject>
|
||
</figure>
|
||
<para>When a disk fails, replica data is automatically distributed to the other zones to
|
||
ensure there are three copies of the data.</para>
|
||
</section>
|
||
<section xml:id="section_accounts-containers">
|
||
<title>Accounts and containers</title>
|
||
<para>Each account and container is an individual SQLite
|
||
database that is distributed across the cluster. An
|
||
account database contains the list of containers in
|
||
that account. A container database contains the list
|
||
of objects in that container.</para>
|
||
<figure>
|
||
<title>Accounts and containers</title>
|
||
<mediaobject>
|
||
<imageobject>
|
||
<imagedata fileref="../common/figures/objectstorage-accountscontainers.png"/>
|
||
</imageobject>
|
||
</mediaobject>
|
||
</figure>
|
||
<para>To keep track of object data locations, each account in the system has a database
|
||
that references all of its containers, and each container database references each
|
||
object.</para>
|
||
</section>
|
||
<section xml:id="section_partitions">
|
||
<title>Partitions</title>
|
||
<para>A partition is a collection of stored data, including account databases, container
|
||
databases, and objects. Partitions are core to the replication system.</para>
|
||
<para>Think of a partition as a bin moving throughout a fulfillment center warehouse.
|
||
Individual orders get thrown into the bin. The system treats that bin as a cohesive
|
||
entity as it moves throughout the system. A bin is easier to deal with than many little
|
||
things. It makes for fewer moving parts throughout the system.</para>
|
||
<para>System replicators and object uploads/downloads operate on partitions. As the
|
||
system scales up, its behavior continues to be predictable because the number of
|
||
partitions is a fixed number.</para>
|
||
<para>Implementing a partition is conceptually simple, a partition is just a
|
||
directory sitting on a disk with a corresponding hash table of what it contains.</para>
|
||
<figure>
|
||
<title>Partitions</title>
|
||
<mediaobject>
|
||
<imageobject>
|
||
<imagedata fileref="../common/figures/objectstorage-partitions.png"/>
|
||
</imageobject>
|
||
</mediaobject>
|
||
</figure>
|
||
</section>
|
||
<section xml:id="section_replicators">
|
||
<title>Replicators</title>
|
||
<para>In order to ensure that there are three copies of the data everywhere, replicators
|
||
continuously examine each partition. For each local partition, the replicator compares
|
||
it against the replicated copies in the other zones to see if there are any
|
||
differences.</para>
|
||
<para>The replicator knows if replication needs to take place by examining hashes. A hash
|
||
file is created for each partition, which contains hashes of each directory in the
|
||
partition. Each of the three hash files is compared. For a given partition, the hash
|
||
files for each of the partition's copies are compared. If the hashes are different, then
|
||
it is time to replicate, and the directory that needs to be replicated is copied
|
||
over.</para>
|
||
<para>This is where partitions come in handy. With fewer things in the system, larger
|
||
chunks of data are transferred around (rather than lots of little TCP connections, which
|
||
is inefficient) and there is a consistent number of hashes to compare.</para>
|
||
<para>The cluster eventually has a consistent behavior where the newest data has a
|
||
priority.</para>
|
||
<figure>
|
||
<title>Replication</title>
|
||
<mediaobject>
|
||
<imageobject>
|
||
<imagedata fileref="../common/figures/objectstorage-replication.png"/>
|
||
</imageobject>
|
||
</mediaobject>
|
||
</figure>
|
||
<para>If a zone goes down, one of the nodes containing a replica notices and proactively
|
||
copies data to a handoff location.</para>
|
||
</section>
|
||
<section xml:id="section_usecases">
|
||
<title>Use cases</title>
|
||
<para>The following sections show use cases for object uploads and downloads and introduce the components.</para>
|
||
<section xml:id="upload">
|
||
<title>Upload</title>
|
||
<para>A client uses the REST API to make a HTTP request to PUT an object into an existing
|
||
container. The cluster receives the request. First, the system must figure out where
|
||
the data is going to go. To do this, the account name, container name, and object
|
||
name are all used to determine the partition where this object should live.</para>
|
||
<para>Then a lookup in the ring figures out which storage nodes contain the partitions in
|
||
question.</para>
|
||
<para>The data is then sent to each storage node where it is placed in the appropriate
|
||
partition. At least two of the three writes must be successful before the client is
|
||
notified that the upload was successful.</para>
|
||
<para>Next, the container database is updated asynchronously to reflect that there is a new
|
||
object in it.</para>
|
||
<figure>
|
||
<title>Object Storage in use</title>
|
||
<mediaobject>
|
||
<imageobject>
|
||
<imagedata fileref="../common/figures/objectstorage-usecase.png"/>
|
||
</imageobject>
|
||
</mediaobject>
|
||
</figure>
|
||
</section>
|
||
<section xml:id="section_swift-component-download">
|
||
<title>Download</title>
|
||
<para>A request comes in for an account/container/object. Using the same consistent hashing,
|
||
the partition name is generated. A lookup in the ring reveals which storage nodes
|
||
contain that partition. A request is made to one of the storage nodes to fetch the
|
||
object and, if that fails, requests are made to the other nodes.</para>
|
||
</section>
|
||
</section>
|
||
</section>
|