9.0 KiB

Raw Blame History

Gear: Asynchronous Event-Driven Gearman Interface

gear

This module implements an asynchronous event-driven interface to Gearman. It provides interfaces to build a client or worker, and access to the administrative protocol. The design approach is to keep it simple, with a relatively thin abstration of the Gearman protocol itself. It should be easy to use to build a client or worker that operates either synchronously or asynchronously.

The module also provides a simple Gearman server for use as a convenience in unit tests. The server is not designed for production use under load.

Client Example

To use the client interface, instantiate a :pyClient, and submit a :pyJob. For example:

import gear
client = gear.Client()
client.addServer('gearman.example.com')
client.waitForServer()  # Wait for at least one server to be connected

job = gear.Job("reverse", "test string")
client.submitJob(job)

The waitForServer() call is only necessary when running in a synchronous context. When running asynchronously, it is probably more desirable to omit that call and instead handle the :pyNoConnectedServersError exception that submitJob may raise if no servers are connected at the time.

When Gearman returns data to the client, the :pyJob object is updated immediately. Event handlers are called on the :pyClient object so that subclasses have ample facilities for reacting to events synchronously.

Worker Example

To use the worker interface, create a :pyWorker, register at least one function that the worker supports, and then wait for a Job to be dispatched to the worker.

An example of a Gearman worker:

import gear
worker = gear.Worker('reverser')
worker.addServer('gearman.example.com')
worker.registerFunction("reverse")

while True:
    job = worker.getJob()
    job.sendWorkComplete(job.arguments[::-1])

Server Example

You can run the Gearman server by executing the geard command. For help execute geard --help

SSL Connections

For versions of Gearman supporting SSL connections, specify the files containing the SSL private key, public certificate, and CA certificate in the addServer() call. For example:

ssl_key = '/path/to/key.pem'
ssl_cert = '/path/to/cert.pem'
ssl_ca = '/path/to/ca.pem'
client.addServer('gearman.example.com', 4730, ssl_key, ssl_cert, ssl_ca)

All three files must be specified for SSL to be used.

API Reference

The following sections document the module's public API. It is divided into sections focusing on implementing a client, a worker, using the administrative protocol, and then the classes that are common to all usages of the module.

Client Usage

The classes in this section should be all that are needed in order to implement a Gearman client.

Client Objects

gear.Client

Job Objects

gear.Job

Worker Usage

The classes in this section should be all that are needed in order to implement a Gearman worker.

Worker Objects

gear.Worker

FunctionRecord Objects

gear.FunctionRecord

WorkerJob Objects

gear.WorkerJob

Administrative Protocol

Gearman provides an administrative protocol that is multiplexed on the same connection as the normal binary protocol for jobs. The classes in this section are useful for working with that protocol. They need to be used with an existing :pyConnection object; either one obtained via a :pyClient or :pyWorker, or via direct instantiation of :pyConnection to a Gearman server.

AdminRequest Objects

gear.AdminRequest

gear.StatusAdminRequest

gear.ShowJobsAdminRequest

gear.ShowUniqueJobsAdminRequest

gear.CancelJobAdminRequest

gear.VersionAdminRequest

Server Usage

Logging

To enable Gearman server logging you can setup a log configuration file and pass it to geard (i.e. geard --log-config=logging.config)

Example logging.config:

[loggers]
keys=root,gear

[handlers]
keys=console,debug,info

[formatters]
keys=simple

[logger_root]
level=WARNING
handlers=console

[logger_gear]
level=INFO
handlers=debug,info
qualname=gear

[handler_console]
level=WARNING
class=StreamHandler
formatter=simple
args=(sys.stdout,)

[handler_debug]
level=DEBUG
class=logging.handlers.TimedRotatingFileHandler
formatter=simple
args=('/var/log/gear/debug.log', 'midnight', 1, 30,)

[handler_info]
level=INFO
class=logging.handlers.TimedRotatingFileHandler
formatter=simple
args=('/var/log/gear/info.log', 'midnight', 1, 30,)

[formatter_simple]
format=%(asctime)s %(levelname)s %(name)s: %(message)s
datefmt=

ACL

The syntax of the optional ACL file consists of a number of sections identified by the SSL certificate Common Name Subject, and the arguments to the :pyACLEntry constructor as key-value pairs:

[<subject>]
register=<regex>
invoke=<regex>
grant=<boolean>

For example:

[my_worker]
register=.*

[my_client]
invoke=.*

[my_node_manager]
grant=True

Server Objects

gear.Server

Access Control

The gear server supports authorization via access control lists. When an :pyACL object is supplied to the server (or a file on the command line), gear changes from the normal Gearman mode of allow-by-default to deny-by-default and only clients with ACL entries will be able to perform actions such as registering functions or submitting jobs. Authorization is based on the SSL certificate Common Name Subject associated with the connection. An :pyACL object may be modified programatically at run-time.

The administrative protocol supports modifying ACLs with the following commands:

acl list

List the current acls:

acl list
client  register=None   invoke=.*   grant=True
worker  register=.* invoke=None grant=True
.

acl grant <verb> <subject> <pattern>

Grant the <verb> action for functions matching <pattern> to <subject>. Verbs can be one of register, invoke, or grant. This requires the current connection have the grant permission. Example:

acl grant register worker .*
OK

acl revoke <verb> <subject>

Revoke the <verb> action from <subject>. Verbs can be one of register, invoke, grant, or all to indicate all permissions for the subject should be revoked. This requires the grant permission, except that a subject may always revoke its own permissions. Example:

acl revoke register worker
OK

acl self-revoke <verb>

Revoke the <verb> action from connections associted with the current certificate subject. Verbs can be one of register, invoke, grant, or all to indicate all permissions for the subject should be revoked. This is similar to acl revoke but is a convenience method so that a subject does not need to know its own common name. A subject always has permission to revoke its own permissions. Example:

acl self-revoke register
OK

ACL Objects

gear.ACL

ACLEntry Objects

gear.ACLEntry

Common

These classes do not normally need to be directly instatiated to use the gear API, but they may be returned or otherwise be accessible from other classes in this module. They generally operate at a lower level, but still form part of the public API.

Connection Objects

gear.Connection

Packet Objects

gear.Packet

Exceptions

gear.ConnectionError

gear.InvalidDataError

gear.ConfigurationError

gear.NoConnectedServersError

gear.UnknownJobError

gear.InterruptedError

Constants

These constants are used by public API classes.

Normal job precedence.

Low job precedence.

High job precedence.

gear.constants

Indices and tables

genindex
modindex
search

9.0 KiB Raw Blame History