Asynchronous Processing with Ruby on Rails (RailsConf 2008)

Posted by Jon
on Tuesday, June 03

As promised, here are the slides from my RailsConf talk on Asynchronous Processing. The slides don’t always make sense without the context, so I’ve added a few labels here and there. If something doesn’t make sense, feel free to ask about it.

Summary: if your web app has any sort of complexity, you’ll likely have some operations that are best processed asynchronously. In particular, any operation that takes a long time, is resource-intensive, or isn’t naturally triggered by http or by a human is a good candidate for asynchronous processing. You can build your own solution, but before you do, check out Bj for most uses, workling/starling for high performance or scalability, Amazon web services for distributed processing, and plain old cron for basic cases.

I also gave a lightning talk called “How not to present at RailsConf in 14 easy steps” – it’s just 2-3 minutes long, so I think I’ll turn it into a screencast. Look for that later this week.

Comments

Leave a response

  1. DustinJune 03, 2008 @ 12:23 PM

    I started using [beanstalk]http://xph.us/software/beanstalkd/ a while back mostly because of [async observer]http://async-observer.rubyforge.org/

    These fit very well into rails, scale really well and can turn synchronous jobs into asynchronous jobs with just a few letters on an existing after_save (or any other arbitrary code point).

  2. DustinJune 03, 2008 @ 12:25 PM

    (ugh, I never mastered textile, however)

    See: beanstalk and async observer

  3. TomekJune 03, 2008 @ 03:18 PM

    What about backgroundfu?

    http://github.com/ncr/background-fu/

  4. ScottJune 03, 2008 @ 03:31 PM

    Thanks for the talk. It was a clear summary of major options + recommendations based on experience. Exactly what I was looking for.

  5. Jacek BecelaJune 03, 2008 @ 03:36 PM

    Have you tried BackgroundFu ?

  6. Jon DahlJune 04, 2008 @ 10:01 AM

    Dustin: beanstalkd + async observer is definitely an interesting option. I haven’t had a chance to use it in production, but I may give it a try. It appears to be a good alternative to workling/starling. As a downside, I don’t think the queue is persisted, so you may need to store the state of each job somewhere and sweep for lost jobs occasionally, but that’s just good practice anyway.

    Scott: thanks! Glad you found it helpful.

    Tomek and Jacek: I haven’t used backgroundfu either. It looks to me like it started with some problems but has come a long way. Does it have any specific advantages over the alternatives? Any cases where it isn’t appropriate?

  7. Jacek BecelaJune 04, 2008 @ 02:52 PM

    BJ looks great but it’s missing (or I missed it) cron-like features. I recently added simple scheduling for future execution and plan to add periodic execution but with simpler interface than cron. BJ also assumes that one worker on one machine is ok, but there are types of tasks which are better suited to be run on multiple workers. BJ has a lot of features which in my opinion are very rarely used (stdin, stdout, shell similarities, etc.).

    When writing BackgroundFu I focused mostly on simplicity, and the effect is that there are no external dependencies or configuration needed to make it work. The learning curve is almost flat (Job is just an AR class). These problems you mentioned are also long gone.

    Workling doesn’t offer persistence like B-Fu and BJ do. Also the learning curve is steeper, and writing workers has more requirements than writing just the code that should be run.

    And old good cron – imagine spawning whole rails environment every time you want to execute a task in backgorund (say 5.times a minute)

    My opinions may be a little biased towards BackgroundFu but I checked BJ and Workling thorougly. Give it a try.

  8. RanyJune 09, 2008 @ 02:00 PM

    @Jacek: workling has persistence over starling. The only requirements for workers is that your class extends Workling::Base. It basically just follows rails conventions in this way (models in app/models, subclass AR::Base, workers in app/workers, subclass Workling::Base).

    Setting up is super simple. Check out this article: (http://playtype.net/past/2008/2/6/starling_and_asynchrous_tasks_in_ruby_on_rails/)

    /r

  9. BillJune 25, 2008 @ 02:21 AM

    I would recommend avoiding BackgroundRb. We’ve been trying to use it for almost four months, and have experienced a series of hard to solve problems. The two biggest drawbacks are that its logging is such that it doesn’t report exceptions by default, and after an exception occurs, the worker will often stop working until restarting the backgroundrb server. This is one of a litany of problems we’ve had, but long story short, my recommendation for future async processors would be anything but Bdrb.