Allow optional, temporary healthcheck failure.

A deployer may want to remove a Swift node from a load balancer for
maintenance or upgrade.  This patch provides an optional mechanism for
this.  The healthcheck filter config can specify "disable_path" which is
a filesystem path.  If a file is present at that location, the
healthcheck middleware returns a 503 with a body of "DISABLED BY FILE".

So a deployer can configure "disable_path" and then touch that
filesystem path, wait for the proxy to be removed from the load balancer
pool, perform maintenance/upgrade, and then remove the "disable_path"
file.

Also cleaned up the conf file man pages a bit.

Change-Id: I1759c78c74910a54c720f298d4d8e6fa57a4dab4
This commit is contained in:
Darrell Bishop 2012-12-03 16:05:44 -08:00
parent 4f617f49b6
commit b8e3e9e1c2
12 changed files with 484 additions and 356 deletions

View File

@ -89,8 +89,8 @@ are acceptable within this section.
.IP "\fBpipeline\fR"
It is used when you need apply a number of filters. It is a list of filters
ended by an application. The default should be "healthcheck
account-server"
ended by an application. The normal pipeline is "healthcheck
recon account-server".
.RE
.PD
@ -103,7 +103,7 @@ This is indicated by section name [app:account-server]. Below are the parameters
that are acceptable within this section.
.IP "\fBuse\fR"
Entry point for paste.deploy for the account server. This is the reference to the installed python egg.
The default is \fBegg:swift#account\fR.
This is normally \fBegg:swift#account\fR.
.IP "\fBset log_name\fR
Label used when logging. The default is account-server.
.IP "\fBset log_facility\fR
@ -130,7 +130,22 @@ Below are the filters available and respective acceptable parameters.
.RS 3
.IP "\fBuse\fR"
Entry point for paste.deploy for the healthcheck middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#healthcheck\fR.
This is normally \fBegg:swift#healthcheck\fR.
.IP "\fBdisable_path\fR"
An optional filesystem path which, if present, will cause the healthcheck
URL to return "503 Service Unavailable" with a body of "DISABLED BY FILE".
.RE
.RS 0
.IP "\fB[filter:recon]\fR"
.RS 3
.IP "\fBuse\fR"
Entry point for paste.deploy for the recon middleware. This is the reference to the installed python egg.
This is normally \fBegg:swift#recon\fR.
.IP "\fBrecon_cache_path\fR"
The recon_cache_path simply sets the directory where stats for a few items will be stored.
Depending on the method of deployment you may need to create this directory manually
and ensure that swift has read/write. The default is /var/cache/swift.
.RE
.PD

View File

@ -91,8 +91,8 @@ are acceptable within this section.
.IP "\fBpipeline\fR"
It is used when you need to apply a number of filters. It is a list of filters
ended by an application. The default should be \fB"healthcheck
container-server"\fR
ended by an application. The normal pipeline is "healthcheck
recon container-server".
.RE
.PD
@ -105,7 +105,7 @@ This is indicated by section name [app:container-server]. Below are the paramete
that are acceptable within this section.
.IP "\fBuse\fR"
Entry point for paste.deploy for the container server. This is the reference to the installed python egg.
The default is \fBegg:swift#container\fR.
This is normally \fBegg:swift#container\fR.
.IP "\fBset log_name\fR
Label used when logging. The default is container-server.
.IP "\fBset log_facility\fR
@ -136,7 +136,22 @@ Below are the filters available and respective acceptable parameters.
.RS 3
.IP "\fBuse\fR"
Entry point for paste.deploy for the healthcheck middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#healthcheck\fR.
This is normally \fBegg:swift#healthcheck\fR.
.IP "\fBdisable_path\fR"
An optional filesystem path which, if present, will cause the healthcheck
URL to return "503 Service Unavailable" with a body of "DISABLED BY FILE".
.RE
.RS 0
.IP "\fB[filter:recon]\fR"
.RS 3
.IP "\fBuse\fR"
Entry point for paste.deploy for the recon middleware. This is the reference to the installed python egg.
This is normally \fBegg:swift#recon\fR.
.IP "\fBrecon_cache_path\fR"
The recon_cache_path simply sets the directory where stats for a few items will be stored.
Depending on the method of deployment you may need to create this directory manually
and ensure that swift has read/write. The default is /var/cache/swift.
.RE
.PD

View File

@ -91,8 +91,8 @@ are acceptable within this section.
.IP "\fBpipeline\fR"
It is used when you need to apply a number of filters. It is a list of filters
ended by an application. The default should be \fB"healthcheck recon
object-server"\fR
ended by an application. The normal pipeline is "healthcheck recon
object-server".
.RE
.PD
@ -105,7 +105,7 @@ This is indicated by section name [app:object-server]. Below are the parameters
that are acceptable within this section.
.IP "\fBuse\fR"
Entry point for paste.deploy for the object server. This is the reference to the installed python egg.
The default is \fBegg:swift#object\fR.
This is normally \fBegg:swift#object\fR.
.IP "\fBset log_name\fR
Label used when logging. The default is object-server.
.IP "\fBset log_facility\fR
@ -136,7 +136,10 @@ Below are the filters available and respective acceptable parameters.
.RS 3
.IP "\fBuse\fR"
Entry point for paste.deploy for the healthcheck middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#healthcheck\fR.
This is normally \fBegg:swift#healthcheck\fR.
.IP "\fBdisable_path\fR"
An optional filesystem path which, if present, will cause the healthcheck
URL to return "503 Service Unavailable" with a body of "DISABLED BY FILE".
.RE
.RS 0
@ -144,8 +147,8 @@ The default is \fBegg:swift#healthcheck\fR.
.RE
.RS 3
.IP "\fBuse\fR"
Entry point for paste.deploy for the healthcheck middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#recon\fR.
Entry point for paste.deploy for the recon middleware. This is the reference to the installed python egg.
This is normally \fBegg:swift#recon\fR.
.IP "\fBrecon_cache_path\fR"
The recon_cache_path simply sets the directory where stats for a few items will be stored.
Depending on the method of deployment you may need to create this directory manually

View File

@ -91,8 +91,8 @@ are acceptable within this section.
.IP "\fBpipeline\fR"
It is used when you need apply a number of filters. It is a list of filters
ended by an application. The default should be \fB"catch_errors healthcheck
cache ratelimit tempauth proxy-server"\fR
ended by an application. The normal pipeline is "catch_errors healthcheck
cache ratelimit tempauth proxy-logging proxy-server".
.RE
.PD
@ -109,7 +109,10 @@ Below are the filters available and respective acceptable parameters.
.RS 3
.IP "\fBuse\fR"
Entry point for paste.deploy for the healthcheck middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#healthcheck\fR.
This is normally \fBegg:swift#healthcheck\fR.
.IP "\fBdisable_path\fR"
An optional filesystem path which, if present, will cause the healthcheck
URL to return "503 Service Unavailable" with a body of "DISABLED BY FILE".
.RE
@ -119,7 +122,7 @@ The default is \fBegg:swift#healthcheck\fR.
.RS 3
.IP \fBuse\fR
Entry point for paste.deploy for the tempauth middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#tempauth\fR.
This is normally \fBegg:swift#tempauth\fR.
.IP "\fBset log_name\fR"
Label used when logging. The default is tempauth.
.IP "\fBset log_facility\fR"
@ -170,27 +173,6 @@ Here are example entries, required for running the tests:
.RE
.PD
.RS 0
.IP "\fB[filter:healthcheck]\fR"
.RE
.RS 3
.IP \fBuse\fR
Entry point for paste.deploy for the healthcheck middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#healthcheck\fR.
.IP "\fBset log_name\fR"
Label used when logging. The default is healthcheck.
.IP "\fBset log_facility\fR"
Syslog log facility. The default is LOG_LOCAL0.
.IP "\fBset log_level\fR "
Logging level. The default is INFO.
.IP "\fBset log_address\fR"
Logging address. The default is /dev/log.
.IP "\fBset log_headers\fR "
Enables the ability to log request headers. The default is False.
.RE
.RS 0
.IP "\fB[filter:cache]\fR"
.RE
@ -200,7 +182,7 @@ Caching middleware that manages caching in swift.
.RS 3
.IP \fBuse\fR
Entry point for paste.deploy for the memcache middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#memcache\fR.
This is normally \fBegg:swift#memcache\fR.
.IP "\fBset log_name\fR"
Label used when logging. The default is memcache.
.IP "\fBset log_facility\fR"
@ -241,7 +223,7 @@ Rate limits requests on both an Account and Container level. Limits are configu
.RS 3
.IP \fBuse\fR
Entry point for paste.deploy for the ratelimit middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#ratelimit\fR.
This is normally \fBegg:swift#ratelimit\fR.
.IP "\fBset log_name\fR"
Label used when logging. The default is ratelimit.
.IP "\fBset log_facility\fR"
@ -289,7 +271,7 @@ Middleware that translates container and account parts of a domain to path param
.RS 3
.IP \fBuse\fR
Entry point for paste.deploy for the domain_remap middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#domain_remap\fR.
This is normally \fBegg:swift#domain_remap\fR.
.IP "\fBset log_name\fR"
Label used when logging. The default is domain_remap.
.IP "\fBset log_address\fR"
@ -318,7 +300,7 @@ by this middleware. Defaults to 'AUTH'.
.RS 3
.IP \fBuse\fR
Entry point for paste.deploy for the catch_errors middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#catch_errors\fR.
This is normally \fBegg:swift#catch_errors\fR.
.IP "\fBset log_name\fR"
Label used when logging. The default is catch_errors.
.IP "\fBset log_facility\fR"
@ -342,7 +324,7 @@ Note: this middleware requires python-dnspython
.RS 3
.IP \fBuse\fR
Entry point for paste.deploy for the cname_lookup middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#cname_lookup\fR.
This is normally \fBegg:swift#cname_lookup\fR.
.IP "\fBset log_name\fR"
Label used when logging. The default is cname_lookup.
.IP "\fBset log_facility\fR"
@ -371,7 +353,7 @@ Note: Put staticweb just after your auth filter(s) in the pipeline
.RS 3
.IP \fBuse\fR
Entry point for paste.deploy for the staticweb middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#staticweb\fR.
This is normally \fBegg:swift#staticweb\fR.
.IP \fBcache_timeout\fR
Seconds to cache container x-container-meta-web-* header values. The default is 300 seconds.
.IP "\fBset log_name\fR"
@ -423,7 +405,7 @@ Note: Put formpost just before your auth filter(s) in the pipeline
.RS 3
.IP \fBuse\fR
Entry point for paste.deploy for the formpost middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#formpost\fR.
This is normally \fBegg:swift#formpost\fR.
.RE
@ -437,7 +419,7 @@ Note: Just needs to be placed before the proxy-server in the pipeline.
.RS 3
.IP \fBuse\fR
Entry point for paste.deploy for the name_check middleware. This is the reference to the installed python egg.
The default is \fBegg:swift#name_check\fR.
This is normally \fBegg:swift#name_check\fR.
.IP \fBforbidden_chars\fR
Characters that will not be allowed in a name.
.IP \fBmaximum_length\fR
@ -447,6 +429,49 @@ Python regular expressions of substrings that will not be allowed in a name.
.RE
.RS 0
.IP "\fB[filter:proxy_logging]\fR"
.RE
Logging for the proxy server now lives in this middleware.
If the access_* variables are not set, logging directives from [DEFAULT]
without "access_" will be used.
.RS 3
.IP \fBuse\fR
Entry point for paste.deploy for the proxy_logging middleware. This is the reference to the installed python egg.
This is normally \fBegg:swift#proxy_logging\fR.
.IP "\fBaccess_log_name\fR"
Label used when logging. The default is proxy-server.
.IP "\fBaccess_log_facility\fR"
Syslog log facility. The default is LOG_LOCAL0.
.IP "\fBaccess_log_level\fR "
Logging level. The default is INFO.
.IP \fBaccess_log_address\fR
Default is /dev/log.
.IP \fBaccess_log_udp_host\fR
If set, access_log_udp_host will override access_log_address. Default is
unset.
.IP \fBaccess_log_udp_port\fR
Default is 514.
.IP \fBaccess_log_statsd_host\fR
You can use log_statsd_* from [DEFAULT], or override them here.
Default is localhost.
.IP \fBaccess_log_statsd_port\fR
Default is 8125.
.IP \fBaccess_log_statsd_default_sample_rate\fR
Default is 1.
.IP \fBaccess_log_statsd_metric_prefix =
Default is "" (empty-string)
.IP \fBaccess_log_headers\fR
Default is False.
.IP \fBlog_statsd_valid_http_methods\fR
What HTTP methods are allowed for StatsD logging (comma-sep); request methods
not in this list will have "BAD_METHOD" for the <verb> portion of the metric.
Default is "GET,HEAD,POST,PUT,DELETE,COPY,OPTIONS".
.RE
.PD
@ -459,23 +484,17 @@ This is indicated by section name [app:proxy-server]. Below are the parameters
that are acceptable within this section.
.IP \fBuse\fR
Entry point for paste.deploy for the proxy server. This is the reference to the installed python egg.
The default is \fBegg:swift#proxy\fR.
.IP "\fBset log_name\fR
This is normally \fBegg:swift#proxy\fR.
.IP \fBset log_name\fR
Label used when logging. The default is proxy-server.
.IP "\fBset log_facility\fR
.IP \fBset log_facility\fR
Syslog log facility. The default is LOG_LOCAL0.
.IP "\fB set log_level\fR
.IP \fB set log_level\fR
Logging level. The default is INFO.
.IP "\fB set log_address\fR
.IP \fB set log_address\fR
Logging address. The default is /dev/log.
.IP "\fBset access_log_name\fR"
Label used when logging. The default is proxy-server.
.IP "\fBset access_log_facility\fR"
Syslog log facility. The default is LOG_LOCAL0.
.IP "\fBset access_log_level\fR "
Logging level. The default is INFO.
.IP "\fB set log_requests\fR
Enables request logging. The default is False.
.IP \fBlog_handoffs\fR
Log when handoff locations are used. Default is True.
.IP \fBrecheck_account_existence\fR
Cache timeout in seconds to send memcached for account existence. The default is 60 seconds.
.IP \fBrecheck_container_existence\fR
@ -508,10 +527,11 @@ container sync won't be able to sync posts. The default is True.
If set to 'true' authorized accounts that do not yet exist within the Swift cluster
will be automatically created. The default is set to false.
.IP \fBrate_limit_after_segment\fR
Rate limit the download of large object segments after this segment is
downloaded. The default is 10 segments.
Start rate-limiting object segments after the Nth segment of a segmented
object. The default is 10 segments.
.IP \fBrate_limit_segments_per_sec\fR
Rate limit large object downlods at this rate. The default is 1.
Once segment rate-limiting kicks in for an object, limit segments served to N
per second. The default is 1.
.RE
.PD

View File

@ -30,7 +30,7 @@
# db_preallocation = off
[pipeline:main]
pipeline = recon account-server
pipeline = healthcheck recon account-server
[app:account-server]
use = egg:swift#account
@ -42,6 +42,12 @@ use = egg:swift#account
# set log_address = /dev/log
# auto_create_account_prefix = .
[filter:healthcheck]
use = egg:swift#healthcheck
# An optional filesystem path, which if present, will cause the healthcheck
# URL to return "503 Service Unavailable" with a body of "DISABLED BY FILE"
# disable_path =
[filter:recon]
use = egg:swift#recon
# recon_cache_path = /var/cache/swift

View File

@ -33,7 +33,7 @@
# db_preallocation = off
[pipeline:main]
pipeline = recon container-server
pipeline = healthcheck recon container-server
[app:container-server]
use = egg:swift#container
@ -48,6 +48,12 @@ use = egg:swift#container
# allow_versions = False
# auto_create_account_prefix = .
[filter:healthcheck]
use = egg:swift#healthcheck
# An optional filesystem path, which if present, will cause the healthcheck
# URL to return "503 Service Unavailable" with a body of "DISABLED BY FILE"
# disable_path =
[filter:recon]
use = egg:swift#recon
#recon_cache_path = /var/cache/swift

View File

@ -28,7 +28,7 @@
# log_statsd_metric_prefix =
[pipeline:main]
pipeline = recon object-server
pipeline = healthcheck recon object-server
[app:object-server]
use = egg:swift#object
@ -57,6 +57,12 @@ use = egg:swift#object
# allowed_headers = Content-Disposition, Content-Encoding, X-Delete-At, X-Object-Manifest
# auto_create_account_prefix = .
[filter:healthcheck]
use = egg:swift#healthcheck
# An optional filesystem path, which if present, will cause the healthcheck
# URL to return "503 Service Unavailable" with a body of "DISABLED BY FILE"
# disable_path =
[filter:recon]
use = egg:swift#recon
#recon_cache_path = /var/cache/swift

View File

@ -162,12 +162,12 @@ user_test_tester3 = testing3
[filter:healthcheck]
use = egg:swift#healthcheck
# You can override the default log routing for this filter here:
# set log_name = healthcheck
# set log_facility = LOG_LOCAL0
# set log_level = INFO
# set log_headers = False
# set log_address = /dev/log
# An optional filesystem path, which if present, will cause the healthcheck
# URL to return "503 Service Unavailable" with a body of "DISABLED BY FILE".
# This facility may be used to temporarily remove a Swift node from a load
# balancer pool during maintenance or upgrade (remove the file to allow the
# node back into the load balancer pool).
# disable_path =
[filter:cache]
use = egg:swift#memcache
@ -315,7 +315,7 @@ use = egg:swift#name_check
[filter:proxy-logging]
use = egg:swift#proxy_logging
# If not set, logging directives from [DEFAULT] without "access_" will be # used
# If not set, logging directives from [DEFAULT] without "access_" will be used
# access_log_name = swift
# access_log_facility = LOG_LOCAL0
# access_log_level = INFO

View File

@ -13,6 +13,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
import os
from swift.common.swob import Request, Response
@ -20,21 +22,35 @@ class HealthCheckMiddleware(object):
"""
Healthcheck middleware used for monitoring.
If the path is /healthcheck, it will respond with "OK" in the body
If the path is /healthcheck, it will respond 200 with "OK" as the body.
If the optional config parameter "disable_path" is set, and a file is
present at that path, it will respond 503 with "DISABLED BY FILE" as the
body.
"""
def __init__(self, app, *args, **kwargs):
def __init__(self, app, conf):
self.app = app
self.conf = conf
self.disable_path = conf.get('disable_path', '')
def GET(self, req):
"""Returns a 200 response with "OK" in the body."""
return Response(request=req, body="OK", content_type="text/plain")
def DISABLED(self, req):
"""Returns a 503 response with "DISABLED BY FILE" in the body."""
return Response(request=req, status=503, body="DISABLED BY FILE",
content_type="text/plain")
def __call__(self, env, start_response):
req = Request(env)
try:
if req.path == '/healthcheck':
return self.GET(req)(env, start_response)
handler = self.GET
if self.disable_path and os.path.exists(self.disable_path):
handler = self.DISABLED
return handler(req)(env, start_response)
except UnicodeError:
# definitely, this is not /healthcheck
pass
@ -42,6 +58,9 @@ class HealthCheckMiddleware(object):
def filter_factory(global_conf, **local_conf):
conf = global_conf.copy()
conf.update(local_conf)
def healthcheck_filter(app):
return HealthCheckMiddleware(app)
return HealthCheckMiddleware(app, conf)
return healthcheck_filter

View File

@ -21,7 +21,8 @@ or that exceed a defined length.
Place in proxy filter before proxy, e.g.
[pipeline:main]
pipeline = catch_errors healthcheck name_check cache tempauth sos proxy-server
pipeline = catch_errors healthcheck name_check cache ratelimit tempauth sos
proxy-logging proxy-server
[filter:name_check]
use = egg:swift#name_check

View File

@ -28,7 +28,8 @@ added. For example::
...
[pipeline:main]
pipeline = healthcheck cache tempauth staticweb proxy-server
pipeline = catch_errors healthcheck cache ratelimit tempauth staticweb
proxy-logging proxy-server
...

View File

@ -13,32 +13,68 @@
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import shutil
import tempfile
import unittest
from swift.common.swob import Request
from swift.common.swob import Request, Response
from swift.common.middleware import healthcheck
class FakeApp(object):
def __call__(self, env, start_response):
return "FAKE APP"
req = Request(env)
return Response(request=req, body='FAKE APP')(
env, start_response)
def start_response(*args):
pass
class TestHealthCheck(unittest.TestCase):
def setUp(self):
self.app = healthcheck.HealthCheckMiddleware(FakeApp())
self.tempdir = tempfile.mkdtemp()
self.disable_path = os.path.join(self.tempdir, 'dont-taze-me-bro')
self.got_statuses = []
def tearDown(self):
shutil.rmtree(self.tempdir, ignore_errors=True)
def get_app(self, app, global_conf, **local_conf):
factory = healthcheck.filter_factory(global_conf, **local_conf)
return factory(app)
def start_response(self, status, headers):
self.got_statuses.append(status)
def test_healthcheck(self):
req = Request.blank('/healthcheck', environ={'REQUEST_METHOD': 'GET'})
resp = self.app(req.environ, start_response)
app = self.get_app(FakeApp(), {})
resp = app(req.environ, self.start_response)
self.assertEquals(['200 OK'], self.got_statuses)
self.assertEquals(resp, ['OK'])
def test_healtcheck_pass(self):
req = Request.blank('/', environ={'REQUEST_METHOD': 'GET'})
resp = self.app(req.environ, start_response)
self.assertEquals(resp, 'FAKE APP')
app = self.get_app(FakeApp(), {})
resp = app(req.environ, self.start_response)
self.assertEquals(['200 OK'], self.got_statuses)
self.assertEquals(resp, ['FAKE APP'])
def test_healthcheck_pass_not_disabled(self):
req = Request.blank('/healthcheck', environ={'REQUEST_METHOD': 'GET'})
app = self.get_app(FakeApp(), {}, disable_path=self.disable_path)
resp = app(req.environ, self.start_response)
self.assertEquals(['200 OK'], self.got_statuses)
self.assertEquals(resp, ['OK'])
def test_healthcheck_pass_disabled(self):
open(self.disable_path, 'w')
req = Request.blank('/healthcheck', environ={'REQUEST_METHOD': 'GET'})
app = self.get_app(FakeApp(), {}, disable_path=self.disable_path)
resp = app(req.environ, self.start_response)
self.assertEquals(['503 Service Unavailable'], self.got_statuses)
self.assertEquals(resp, ['DISABLED BY FILE'])
if __name__ == '__main__':
unittest.main()