+Document method to avoid rsync filling root drive

When rsync pushes to a remote node with an unmounted drive and if
certain steps are not taken, rsync may attempt to write files to
the local drive at the location where the drive was mounted.

There are two suggested solutions for this issue:
  1) Set the permissions for all mount points in /srv/node/
       to root:root 755
  2) Mount the drives elsewhere and symlink the drives to /srv/.../

The first method ensures that only root and not the swift user
can write in the /srv/.../ directories.

The second method will prompt a broken link issue if rsync
attempts to write to an unmounted drive.

Change-Id: I60ce4ed9ef8401768d5f78b6806cbb2e2a65303e
Closes-Bug: #1470576

doc/source/admin_guide.rst

@@ -154,6 +154,10 @@ until it has been resolved.  If the drive is going to be replaced immediately,
 then it is just best to replace the drive, format it, remount it, and let
 replication fill it up.
 
+After the drive is unmounted, make sure the mount point is owned by root
+(root:root 755). This ensures that rsync will not try to replicate into the
+root drive once the failed drive is unmounted.
+
 If the drive can't be replaced immediately, then it is best to leave it
 unmounted, and set the device weight to 0. This will allow all the
 replicas that were on that drive to be replicated elsewhere until the drive

doc/source/deployment_guide.rst

@@ -1229,6 +1229,10 @@ For a standard swift install, all data drives are mounted directly under
 be sure to set the `devices` config option in all of the server configs to
 point to the correct directory.
 
+The mount points for each drive in /srv/node/ should be owned by the root user
+almost exclusively (root:root 755). This is required to prevent rsync from
+syncing files into the root drive in the event a drive is unmounted.
+
 Swift uses system calls to reserve space for new objects being written into
 the system. If your filesystem does not support `fallocate()` or
 `posix_fallocate()`, be sure to set the `disable_fallocate = true` config

doc/source/development_saio.rst

@@ -95,6 +95,16 @@ another device when creating the VM, and follow these instructions:
         # **Make sure to include the trailing slash after /srv/$x/**
         for x in {1..4}; do sudo chown -R ${USER}:${USER} /srv/$x/; done
 
+     Note: We create the mount points and mount the storage disk under
+     /mnt/sdb1. This disk will contain one directory per simulated swift node,
+     each owned by the current swift user.
+
+     We then create symlinks to these directories under /srv.
+     If the disk sdb is unmounted, files will not be written under
+     /srv/\*, because the symbolic link destination /mnt/sdb1/* will not
+     exist. This prevents disk sync operations from writing to the root
+     partition in the event a drive is unmounted.
+
   #. Next, skip to :ref:`common-dev-section`.
 
 
@@ -135,6 +145,15 @@ these instructions:
         # **Make sure to include the trailing slash after /srv/$x/**
         for x in {1..4}; do sudo chown -R ${USER}:${USER} /srv/$x/; done
 
+     Note: We create the mount points and mount the loopback file under
+     /mnt/sdb1. This file will contain one directory per simulated swift node,
+     each owned by the current swift user.
+
+     We then create symlinks to these directories under /srv.
+     If the loopback file is unmounted, files will not be written under
+     /srv/\*, because the symbolic link destination /mnt/sdb1/* will not
+     exist. This prevents disk sync operations from writing to the root
+     partition in the event a drive is unmounted.
 
 .. _common-dev-section: