openstack/swift
Code Issues Proposed changes

Add fallocate_reserve to account and container servers.

Browse Source 
The object server can be configured to leave a certain amount of disk
space free; default is 1%. This is useful in avoiding 100%-full
filesystems, as those can get Swift in a state where the filesystem is
too full to write tombstones, so you can't delete objects to free up
space.

When a cluster has accounts/containers and objects on the same disks,
then you can wind up with a 100%-full disk since account and container
servers don't respect fallocate_reserve. This commit makes account and
container servers respect fallocate_reserve so that disks shared
between account/container and object rings won't get 100% full.

When a disk's free space falls below the configured reserve, account
and container PUT, POST, and REPLICATE requests will fail with a 507
status code. These are the operations that can significantly increase
the disk space used by a given database.

I called the parameter "fallocate_reserve" for consistency with the
object server. No actual fallocate() call happens under Swift's
control in the account or container servers (sqlite3 might make such a
call, but it's out of our hands).

Change-Id: I083442eef14bf83c0ea717b1decb3e6b56dbf1d0
This commit is contained in:
Samuel Merritt
2018-02-05 14:06:18 -08:00
committed by Matthew Oliver
parent bea74a3260
commit 8e651a2d3d
10 changed files with 321 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
doc/source/admin_guide.rst
View File

@@ -298,10 +298,27 @@ Preventing Disk Full Scenarios
Prevent disk full scenarios by ensuring that the ``proxy-server`` blocks PUT
requests and rsync prevents replication to the specific drives.
You can prevent `proxy-server` PUT requests to low space disks by ensuring
``fallocate_reserve`` is set in the ``object-server.conf``. By default,
``fallocate_reserve`` is set to 1%. This blocks PUT requests that leave the
free disk space below 1% of the disk.
You can prevent `proxy-server` PUT requests to low space disks by
ensuring ``fallocate_reserve`` is set in ``account-server.conf``,
``container-server.conf``, and ``object-server.conf``. By default,
``fallocate_reserve`` is set to 1%. In the object server, this blocks
PUT requests that would leave the free disk space below 1% of the
disk. In the account and container servers, this blocks operations
that will increase account or container database size once the free
disk space falls below 1%.
Setting ``fallocate_reserve`` is highly recommended to avoid filling
disks to 100%. When Swift's disks are completely full, all requests
involving those disks will fail, including DELETE requests that would
otherwise free up space. This is because object deletion includes the
creation of a zero-byte tombstone (.ts) to record the time of the
deletion for replication purposes; this happens prior to deletion of
the object's data. On a completely-full filesystem, that zero-byte .ts
file cannot be created, so the DELETE request will fail and the disk
will remain completely full. If ``fallocate_reserve`` is set, then the
filesystem will have enough space to create the zero-byte .ts file,
and thus the deletion of the object will succeed and free up some
space.
In order to prevent rsync replication to specific drives, firstly
setup ``rsync_module`` per disk in your ``object-replicator``.