set sysvar vm_min_free_kbytes as function of total mem

Problem description:
During times of high memory pressure, the kernel will drop
into synchronous memory allocation mode if the amount of free
memory in the normal zone drops below the min watermark.
Synchronous mode reclaims some memory before allocating any,
and is thus vastly slower than asynchronous mode. If under high
memory pressure we drop to synchronous mode, and there's a lot
of memory allocation requests being made ie. contention for
locks in the memory allocator, and one of the tasks contending
is being monitored by pmon, then it can be blocked long enough
to trigger a watchdog timeout which brings down the entire node.

Solution:
One way to reduce the chances of dropping below the min
watermark in the normal zone is to increase the amount of
kernel reserved memory via sysctl variable vm_min_free_kbytes.
The three watermarks, min, low, and high are all calculated and
set by the kernel based on the amount of kernel reserved memory.
While all three of the watermarks will be increased, the spread
between them will also be increased.  When the number of pages
in the zone drops below low, the kswapd will run and reclaim
pages (typically from the buffer cache) until the high
watermark is met.  This operates a lot like a water pressure
tank.  So by increasing the amount of kernel reserved memory,
kswapd should engage sooner and thus increase the odds that
we don't hit the min watermark.

Currently the amount of kernel reserved memory is set statically
to 128 MB except on a storage node where we set it to 256 MB.
This is really too low on systems with a lot of memory.  It
should be set as a function of total system memory.  There are
many possibilities for such a function, such as just setting it
to a percentage of total memory such as 0.5% or 1% as examples.

Here we take a quantum approach to the function, reserving
128 MB for every 25 GB of system memory, with also a minimum
amount specified that we won't go below.  This is approximately
0.5% based on the step function.  Why this approach?  Because it
has no impact on typical small vbox nodes used in development.
Those nodes are typically around 25 GB in size, and it isn't
a great idea to be reclaiming pages from the buffer cache just
to have them sit unused, as these small emulated nodes need all
i/o performance help they can get.

This approach to the function is not absolute and is subject to
change as we learn more about how to best tune the settings.

Change-Id: I6244497b262c217dc9467010501a3041e102a288
Closes-Bug: 1940855
Signed-off-by: Jim Somerville <Jim.Somerville@windriver.com>

puppet-manifests/src/modules/platform/manifests/sysctl.pp

@@ -2,6 +2,19 @@ class platform::sysctl::params (
   $low_latency = false,
 ) inherits ::platform::params {}
 
+class platform::sysctl::vm_min_free_kbytes (
+  $minimum_kb = 131072,
+  $per_every_gb = 25,
+  $reserve_mb = 128,
+) inherits ::platform::sysctl::params {
+
+  # Try to keep reserve_mb free per_every_gb of memory
+  $want_min_free_kbytes = (floor($::memorysize_mb) / ($per_every_gb * 1024)) * $reserve_mb * 1024
+  $min_free_kbytes = max($want_min_free_kbytes, $minimum_kb)
+  sysctl::value { 'vm.min_free_kbytes':
+    value => String($min_free_kbytes)
+  }
+}
 
 class platform::sysctl
   inherits ::platform::sysctl::params {
@@ -118,11 +131,7 @@ class platform::sysctl::controller
 
   include ::platform::sysctl
   include ::platform::sysctl::controller::reserve_ports
-
-  # Increase min_free_kbytes to 128 MiB from 88 MiB, helps prevent OOM
-  sysctl::value { 'vm.min_free_kbytes':
-    value => '131072'
-  }
+  include ::platform::sysctl::vm_min_free_kbytes
 
   # Engineer VM page cache tunables to prevent significant IO delays that may
   # occur if we flush a buildup of dirty pages.  Engineer VM settings to make
@@ -170,11 +179,8 @@ class platform::sysctl::controller
 class platform::sysctl::compute {
   include ::platform::sysctl
   include ::platform::sysctl::compute::reserve_ports
+  include ::platform::sysctl::vm_min_free_kbytes
 
-  # Increase min_free_kbytes to 128 MiB from 88 MiB, helps prevent OOM
-  sysctl::value { 'vm.min_free_kbytes':
-    value => '131072'
-  }
 }
 
 class platform::sysctl::compute::reserve_ports
@@ -194,9 +200,8 @@ class platform::sysctl::compute::reserve_ports
 class platform::sysctl::storage {
   include ::platform::sysctl
 
-  # Increase min_free_kbytes to 256 MiB for storage node, helps prevent OOM
-  sysctl::value { 'vm.min_free_kbytes':
-    value => '262144'
+  class { 'platform::sysctl::vm_min_free_kbytes':
+    minimum_kb => 262144
   }
 }