*Note: This is a recap of a lightning talk I gave at [DjangoCon 2009] [djangocon]. Here are [the slides](http://www.slideshare.net/cmheisel/gearman-and-django-working-later)*
Problem: You’ve got resource or time intensive work to be done but you don’t want to do it within the Django request/response cycle. Jobs like fetching remote resources, resizing images, expensive database queries, etc. are good candidates.
At work, whenever we have a problem with scaling, or really any problem at all, [Johnny][johnny] always asks **”What would [Brad Fitzpatrick][brad] do?”**
What he did was build [Gearman][gearman]. It is a work queuing and distribution system. From your client code you send a request off for work to be done, either synchronously or asynchronously. Worker instances pull jobs off the stack and perform them.
The Python bindings are [stupidly simple to use](http://samuel.github.com/python-gearman/docs/).
# client.py
from gearman import GearmanClient
client = GearmanClient(["127.0.0.1"])
client.dispatch_background_task("echo", "foo")
# worker.py
worker = GearmanWorker(["127.0.0.1"])
worker.register_function("echo", lambda job:job.arg)
worker.work()
It’s conceptually similar to [RabbitMQ][rabbitmq]. Both are message queues but Gearman feels a little more specialized for use as a task queue, while RabbitMQ is more a general messaging framework.
RabbitMQ is nice as well but I’ve got one particular pet peeve with it. When you ask “How do I scale RabbitMQ?” the answer is pretty much [“Use the power of Erlang”][rabbit-scale].
When you ask “How do I scale Gearman?” the answer is “just like memcached, add more servers to the cluster and specify them in your connection.”
from gearman import GearmanClient client = GearmanClient(["workserver.yourdomain.com", "procrastinate.yourdomain.com"])
If you’re looking for a task queue, I highly recommend [Gearman][gearman] — it’s easy to use and easy to scale!
[djangocon]: http://www.djangocon.org/
[brad]: http://bradfitz.com/
[johnny]: http://revjohnnyhealey.wordpress.com/
[gearman]: http://gearman.org/
[rabbitmq]: http://www.rabbitmq.com/
[rabbit-scale]: http://browsertoolkit.com/fault-tolerance.png
