A short introduction to Django and Gearman

*Note: This is a recap of a lightning talk I gave at [DjangoCon 2009] [djangocon]. Here are [the slides](http://www.slideshare.net/cmheisel/gearman-and-django-working-later)*

Problem: You’ve got resource or time intensive work to be done but you don’t want to do it within the Django request/response cycle. Jobs like fetching remote resources, resizing images, expensive database queries, etc. are good candidates.

At work, whenever we have a problem with scaling, or really any problem at all, [Johnny][johnny] always asks **”What would [Brad Fitzpatrick][brad] do?”**

What he did was build [Gearman][gearman]. It is a work queuing and distribution system. From your client code you send a request off for work to be done, either synchronously or asynchronously. Worker instances pull jobs off the stack and perform them.

The Python bindings are [stupidly simple to use](http://samuel.github.com/python-gearman/docs/).

# client.py
from gearman import GearmanClient
client = GearmanClient(["127.0.0.1"])
client.dispatch_background_task("echo", "foo")

# worker.py
worker = GearmanWorker(["127.0.0.1"])
worker.register_function("echo", lambda job:job.arg)
worker.work()

It’s conceptually similar to [RabbitMQ][rabbitmq]. Both are message queues but Gearman feels a little more specialized for use as a task queue, while RabbitMQ is more a general messaging framework.

RabbitMQ is nice as well but I’ve got one particular pet peeve with it. When you ask “How do I scale RabbitMQ?” the answer is pretty much [“Use the power of Erlang”][rabbit-scale].

When you ask “How do I scale Gearman?” the answer is “just like memcached, add more servers to the cluster and specify them in your connection.”

from gearman import GearmanClient

client = GearmanClient(["workserver.yourdomain.com",
"procrastinate.yourdomain.com"])

If you’re looking for a task queue, I highly recommend [Gearman][gearman] — it’s easy to use and easy to scale!

[djangocon]: http://www.djangocon.org/
[brad]: http://bradfitz.com/
[johnny]: http://revjohnnyhealey.wordpress.com/
[gearman]: http://gearman.org/
[rabbitmq]: http://www.rabbitmq.com/
[rabbit-scale]: http://browsertoolkit.com/fault-tolerance.png