MapReduce at RailsConf Europe

Posted by Jon
on Thursday, July 03

This September, I’ll be presenting at RailsConf Europe on EC2, MapReduce, and Distributed Processing. The talk will explain the MapReduce approach to distributed processing, will show a few example implementations, and will discuss MapReduce vs. other distributed processing techniques.

Whether you’ll be there or not, if you’re interested in learning more about MapReduce, here are some resources. I’ll write a few more posts on the subject before the conference, so watch this space as well.

Cluster Computing and MapReduce is a great series of video lectures given to Google interns in 2007. The first two are the most appropriate: the first introduces distributed processing concept, while the second covers MapReduce itself.

MapReduce: Simplified Data Processing on Large Clusters is the paper by Jeffrey Dean and Sanjay Ghemawat of Google that got things going in the first place.

MapReduce for Ruby: Ridiculously Easy Distributed Programming discusses MapReduce and introduces Starfish, a Ruby library for distributed processing. Starfish is not a MapReduce implementation, however – it takes a somewhat different approach to distributed processing.

Skynet (a few writeups: InfoQ, Dion Almaer) is another Ruby-based distributed processing system inspired by MapReduce.

Writing Ruby Map-Reduce programs for Hadoop discusses using Ruby to wrap Hadoop, a MapReduce-like system built in Java.

Introduction to Parallel Programming and MapReduce at Google Code University, a good overview of distributed processing and the MapReduce approach.

And finally, one article that you should avoid:

MapReduce: A major step backwards compares MapReduce to relational databases, and says that MapReduces loses out because it doesn’t support database indices, database views, Crystal reports, etc. Basically, the complaint is that MapReduce isn’t SQL compliant. WTF? Clearly, the author(s) didn’t understand what MapReduce is. The problem, as explained elsewhere, is that the authors thought that MapReduce == CouchDB/SimpleDB. Which is obviously not true. %s/MapReduce/SimpleDB the original article and it makes some sense. But long story short, this article will teach you nothing about MapReduce, and will likely confuse you further. So stay away.

Just say 'no' to Vegas, RailsConf

Posted by Jon
on Friday, June 27

If you’re just going to get one thing out of this article, it’s this: Vegas is a bad idea for RailsConf ‘09.

Here’s the story. At the end of RailsConf ‘08, in Portland for its second year, Chad Fowler tentatively announced the location of next year’s RailsConf, saying something like “We’re not sure what you’ll think about this, but what about Las Vegas?” Upon which the crowd erupted in cheering, thereby supposedly confirming the Vegas idea. But afterwards, I talked to a dozen people who said they thought Vegas was a bad idea. And in the month since then, I haven’t talked to a single person who was excited about it. That’s how crowds work, I guess – the 10% of Rails developers who enjoy gambling, strippers, and steak applauded loudly, and the 30% of don’t have a strong opinion one way or another got swept up in the excitement.

Of course, it doesn’t really matter – holding the next RailsConf in Las Vegas won’t kill Rails, and won’t set the Rails community on a future of drunkenness, adultery, and gambling. I’m sure most conference-goers will fly in, attend sessions, have dinner, and hack in their hotel lobbies, just like any other conference.

But it’s still a bad idea.

DARE to keep RailsConf different

If you haven’t already, check out Giles Bowkett’s recent post on the situation. He talks about Ruby Central’s desire to “keep RailsConf weird.” What’s especially confusing, as Giles points out, is that the Vegas announcement came after DHH’s keynote, which said that we should get more sleep and use our advantages for good, not evil (in the form of hookers and fur coats). Charles Nutter made a similar point, saying that Rails + enterprise doesn’t have to mean steak and strippers.

As Giles puts it,

DHH is saying, “No hookers! Choose a life well-lived!” And RailsConf is like, “Screw Portland! We’re going to Vegas!”

Portland was a great place for a conference. Portland has great food & beer, cool hotels, the world’s biggest bookstore, and a dozen movie-theater-pubs. Every night after the conference, a thousand Rails geeks would descend upon the city’s first-rate brewpubs and coffee shops to meet people, eat and drink, and discuss programming, politics, philosophy, or whatever. $5.99 steak buffets just won’t be the same. I really enjoyed my twice annual trip to Portland, and would love to go back again. If Ruby Central wants to keep RailsConf edgy, it can’t do much better than Portland.

Of course, if we don’t want to do RailsConfs ‘09, ‘10, ‘11, and ‘12 in the same place, there is no lack of great cities to consider. What about Seattle? Boston? Austin? San Diego? San Francisco? New York? Kansas City? Toronto? Vancouver? Minneapolis?

Is RailsConf worth it?

So enough about Vegas. I get asked from time to time if RailsConf is worthwhile. My answer is: “Yes (I think).”

The first RailsConf was small(er), high energy, and novel. Rails was just on the brink of mainstream acceptance, which is a fun time in the life of a technology – it’s growing rapidly but is still edgy. The sessions were generally high quality, and the keynotes were excellent (Dave Thomas, Martin Fowler, David Heinemeier Hansson, Paul Graham, and _why).

RailsConf 2007 was a completely different conference, what with O’Reilly and 1600 attendees. Ze Frank was great, but the overall session quality was pretty weak. There were some great talks, to be sure; but there weren’t enough really deep technical talks, and some of the presenters didn’t seem to have really practiced. Things were probably set off on the wrong note from the beginning: Day 0’s 3-hour tutorials were mostly disappointing. Also, there seemed to be an abundance of non-programmer business folks, probably there to check out this new thing called Rails.

This year’s conference was quite a bit better. It had the same polished feel that O’Reilly brings, which is both good and bad. Keynotes were mixed. But mainly, the sessions were mostly really good. Whatever David Black, Chad Fowler, and Rich Kilmer did to improve the session quality, it worked. Interestingly enough, it seemed like the non-programmers were gone this year.

RubyConf 2007 (the only one I’ve attended) felt a lot like RailsConf 2006. It had a similar size, a similar venue, and somewhat similar atmosphere. Sessions were good, and I learned quite a bit. Most of the people there were Rails developers, but they were the ones who were interested in Ruby as a language and not just as the technology behind Rails.

So my answer is that RailsConf is worth it, as long as it follows the 2008 path. It will never look like 2006 again; big RailsConfs are here to stay. But 2008 was reasonably graceful for a big conference. I just hope it doesn’t fall into the 2007 trap, which will happen if it tries to cater to managers and the mainstream. It’s impossible to know what next year’s conference will look like, though I think the success of 2008 was a conscious rejection of some of the failures of 2007. And if you’re looking for something smaller and edgier, there’s always RubyConf, RubyFringe, and the regional conferences.

(Two updates.

First, the title of this article was ambiguous, so I changed it from “Just say ‘no’ to RailsConf Las Vegas” to “Just say ‘no’ to Vegas, RailsConf”. The title was supposed to say “Vegas is a bad idea for RailsConf,” not “Don’t attend if it is in Las Vegas.” If you decide not to attend because of the location, that’s fine – but I’ll consider being there either way.

Second, thanks to Chad Fowler and David A. Black for weighing in. Conferences are a lot of work, as Luke knows on a smaller scale, and I can’t imagine what kind of work Ruby Central puts into RailsConf each year. Especially when they’d probably be hacking. :) So thanks for the work, and the conferences that result.)

Lightning talk: How not to present at RailsConf, in 14 easy steps

Posted by Jon
on Friday, June 06

I gave this Lightning Talk on the last day of RailsConf.

Asynchronous Processing with Ruby on Rails (RailsConf 2008)

Posted by Jon
on Tuesday, June 03

As promised, here are the slides from my RailsConf talk on Asynchronous Processing. The slides don’t always make sense without the context, so I’ve added a few labels here and there. If something doesn’t make sense, feel free to ask about it.

Summary: if your web app has any sort of complexity, you’ll likely have some operations that are best processed asynchronously. In particular, any operation that takes a long time, is resource-intensive, or isn’t naturally triggered by http or by a human is a good candidate for asynchronous processing. You can build your own solution, but before you do, check out Bj for most uses, workling/starling for high performance or scalability, Amazon web services for distributed processing, and plain old cron for basic cases.

I also gave a lightning talk called “How not to present at RailsConf in 14 easy steps” – it’s just 2-3 minutes long, so I think I’ll turn it into a screencast. Look for that later this week.

Get more sleep

Posted by Luke
on Saturday, May 31

There was one thing David Heinemeier Hansson said at his RailsConf keynote that I unequivocally agreed with. But besides the value of the 4 day work week, he also brought up the topic of sleep.

I’ve been reading the book Brain Rules by molecular biologist John Medina. It’s puts the latest research about how the brain works into a popularity accessible form, and makes recommendations for individuals, schools, and businesses to better harness human potential for learning.

Brain Rule #7 is Sleep well, think well. One of the interesting facts in the book is about the cumulative effects of sleep deprivation. After a few days of sleeping less than needed, human performance is equivaalent to going 48 hours without any sleep at all. Also, do you ever get the urge for an afternoon nap? According to Brain Rules, a nap can improve cognitive performance almost 30%.

So, yeah. Get more sleep.

Hello Portland!

Posted by Luke
on Thursday, May 29

The whole Rail Spikes crew is here in Portland for RailsConf! Feel free to say “hi.”

Jon Dahl from Slantwise Design

Jon will be speaking about Asynchronous Processing with Rails at 11:45 on Saturday. Come check it out.

Tea time at Mariage Freres

I (Luke) am going to be checking out the Advanced Rails Recipes book signing during lunch on Friday. Come say hi and meet some other developers.

Eric Chapweske

This is Eric’s first RailsConf, so be gentle. He plans on taking up the Portland microbrew circuit.

See you at RailsConf '08

Posted by Jon
on Monday, February 25

It looks like I will be speaking at RailsConf again. This time, I’ll be discussing asynchronous processing.

If you’ve worked on a Rails app that strays beyond basic CRUD, you’ve probably had to think about running some actions in the background – that is, outside of the HTTP request/response cycle. I’ve had to worry about this on quite a few projects, and contributed a recipe to the forthcoming Advanced Rails Recipes book. The recipe discussed how to package a Ruby script as a daemon, which is a simple and powerful approach to background processing. But there are quite a few others, including (but not limited to): AP4R, cron, starling, workling, beanstalk, backgroundrb, spawn, background_fu, and Amazon SQS. And I’ve left out quite a few.

So as I prepare for the conference, I’d love your feedback. What options have you tried? What has worked well or worked poorly? What would you like to see at the conference?

RailsConf slides (video transcoding)

Posted by Jon
on Friday, May 18

Here are the slides from my RailsConf presentation today. I’ve taken out the video; if you’re really interested in seeing it, send me an email. I’ve also created a version without any images at all in order to save space.

Video transcoding slides, with images (4.5M)

Video transcoding slides, no images (207K)

I’d also love your feedback from the talk if you were there.

Slantwise goes global

Posted by Luke
on Wednesday, May 02

Two of us from Slantwise (and Rail Spikes) will be going on the speaking circuit this summer.

Jon will be speaking about video transcoding and Rails at Rails Conf 2007. The series of blog posts here at Rail Spikes will form the basis for his talk. Jon will be speaking on the first day of the conference, May 18th.

My talk on mobile integration with Rails was accepted to Ostrava on Rails in the Czech Republic, June 22-23.

I’ll be speaking about how to connect your Rails application to mobile phones with MMS2R and some other techniques. It looks like I’ll have some good company, too!

Both of these presentations have grown out of some recent projects we’ve been working on that we’ll be making public shortly. So stay tuned!