Music and programming: interviews with Chad Fowler and Dave Thomas

Posted by Jon
on Thursday, April 30

I’ll be speaking at RailsConf 2009 this year on music and software development (Five musical patterns for programmers). The basic premise is that software development and music actually have quite a bit in common. This may be surprising to some people, who see programming as a cold, rational left-brain sort of thing, like science. But we programmers know that this is not really the case at all.

So as a prelude to my talk, I decided to interview two programmer-musicians on the subject: Chad Fowler and Dave Thomas. Both compose and perform music, and both are noted programmers. Here is the interview.

Rail Spikes: Tell us a little about your background with both programming and music.

Chad Fowler: I started my professional life as a saxophonist in Memphis. I played the Beale street clubs and all the typical Memphis professional musician stuff. Among others, I played for a while with Ann Peebles and her husband Don Bryant with the rhythm section from all the old Hi Records recordings. I did mostly R&B and jazz professionally but I was probably most well known in the Memphis community for making “strange” music. Before playing music professionally, I played guitar in punk bands in high school. I was a fan of punk, heavy metal, hip hop, pop, (new) classical and pretty much everything else. As I immersed myself in the world of jazz, it became quickly clear that the jazz community doesn’t like punk and other less “serious” types of music and has an almost religious negative reaction to jazz musicians who do.

It was almost as if any deviation from the “normal” world of jazz made you a traitor. So I did the natural thing: started a group called The Jazz Traitors, which played music that 1) we loved and 2) offended the jazz community (not necessarily in that order).

I was also very interested in composing “classical” music. I studied with a composer named Kamran Ince, who is still my favorite such composer.

As for programming, I’ve been interested in programming since I was a young child using my commodore 64. I wasn’t really that good at it as a kid but I played around a lot. I didn’t get serious until I picked up programming again as a hobby while I was a professional musician. After a late night gig at a bar, it was relaxing to go home and unwind to some C programming tutorials. I didn’t have a need to program, nor did I have a project in mind (except that I have always loved video games and wanted to learn how they worked). But I got so into it, that I ended up getting a job in computer support because a friend filled out an application for me.

Being the gamer I am, as soon as I started in computer support, I naturally wanted to “level up”. That meant becoming a network administrator. Then a system administrator. Then a programmer, then a designer, then an architect, then a CTO, etc. Now here I am. It’s been fun.

Dave Thomas: There was always a lot of music in our house. My father liked to play the piano and the organ (I learned to solder as he built a Heathkit organ from a kit in the late 60s). My mother liked Broadway musicals. So we’d often experience alternating hours of Chopin and South Pacific. My brother was also musical. I wasn’t particularly, but I enjoyed noodling on the piano, and spent hours just playing with chords and progressions.

I’ve been programming since I was 15 or so.

Rail Spikes: Some developers – yourself included – have suggested a similarity between programming and music composition or performance. How exactly are music and programming similar?

Dave Thomas: I’m not sure, but I think it might be something to do with the discovery of patterns. Both music and code consist of nested sets of variations and repetitions. There’s a rythm to executing code, in the same way there’s a rythm to music. It is never exact, but it’s there. After a while, I found I could imagine the rythm and structure of my programs as they run, in the same way you can pick apart the structure of a piece of music as you listen to it. And, jsut as with music, it takes experience to be able to feel the deeper structures and notice the more extreme variations. But being able to spot them in programs makes coding simpler and more interesting. The basic coding structures—loops, method calls, and so on—provide the framework for composing in the same way that staff and bar lines do for music. Algorithms are like the progressions, and data becomes the notes. And in the same way that good music takes all these things and then surprises you, good code does the same thing. It isn’t mechanical and repetitive: instead it uses the constraints to build something bigger and more interesting.

Chad Fowler: It’s hard for me to put my finger on. There’s something similar in the way I think when I do each.

I think it all boils down to language, though. In all of these cases (including learning actual language), you take a bunch of tokens (notes, sounds, grunts, functions, classes) and combine them into a grammar which you use to express ideas. The way you do that is totally up to you as long as the intended ideas are communicated. With computer programs, they have to do what they’re meant to do. With music, they express or evoke emotions, paint pictures, cause anxiety or whatever.

Some computer programs evoke emotions and cause anxiety as well.

Rail Spikes: Is Ruby development more like improvised jazz or composed classical music?

Chad Fowler: I think it’s both. And I don’t think Ruby is any different in this than other languages. Much of the discussion about the relationship between programming and music focuses on the more obvious idea of programming as composition. It makes sense, since programmers tend to sit and type their ideas into an editor and then eventually execute it. The programs can be checked, tested, refactored, etc. before the actual performance. This is how classical composition works as well.

But the less obvious angle is that in many situations, programming is like performance. In fact, even in music, improvisation is really just real time composition. You don’t get a chance to refactor because your “code” is executed as you write it.

I’ve had this same feeling while debugging production problems, hacking new features on a tight deadline, or sometimes during the initial creation of an application. The same synapses are firing as when I was trying to play Cherokee at 200 beats per minute. Mistakes can’t be erased, so they have to be nuanced into (worst case) insignificant events or (best case) important drivers behind the work.

From a purely development-oriented perspective, TDD is more like improvisation than composition. I think that’s what I like about it. It’s motivating and creative in an exciting, time-sensitive way. You take small steps and see where they lead you. Sure, you can always revert your changes if you paint yourself into a corner but part of the fun and challenge is to not paint yourself into a corner.

One thing jazz musicians like to say is that every wrong note is just a half step away from a right note. TDD is like that. You might take a slightly wrong turn. It’s fun to see if you can course-correct without starting over.

Rail Spikes: Do developers need to be musically inclined? Does it help?

Chad Fowler: Obviously not. Some of the best programmers I know are not musicians. I can’t tell if it helps, but I would guess that developers who are also musicians are different than developers who aren’t. I don’t think that’s because being a musician changes people, though. I think it’s because the people who are both are the kind of people who need to do both.

This usually means they’re “right brain” people. This leads to a way of thinking that changes how they approach programming problems.

I think learning music (or another right brain discipline) is a good way to exercise your mind. So I wouldn’t be surprised if leaning music helps people exercise their thought processes in ways that will benefit their work as programmers (or authors, or lawyers, or doctors or whatever).

I also think, though, that if we were all musicians at heart, we wouldn’t get much done. I rely heavily on my less artsy colleagues to ground me and be sometimes more pragmatic than I am. So I don’t think we all need to be a “right brain” programmer. It would be disasterous if we were.

Dave Thomas: Do they need to be? No. But many of the good ones I know are. I’d guess that density of musicians in software development is many times the population norm. But that means you could also ask the question “Do musicians have to know software development?”

I think the more interesting question is to ask “how can people best express what they enjoy doing?” because both music and software development are outlets for this.

Rail Spikes: What sort of music do you listen to? Any recommendations for Ruby developers looking to expand their musical horizons?

Chad Fowler: As I mentioned earlier, I like all kinds of music (with a few exceptions). Lately I’ve been listening to a lot of instrumental hip hop, such as DJ Qbert and Mixmaster Mike. I’ve also been getting into a genre of electronic music called “electro”, which sounds like the bleeps and bloops that are the soundtrack of my dreams (if a computer is going to generate music I always like it to sound like a computer generated it).

As for recommendations, here are a few ideas for things that most developers probably haven’t listened to:

  • Kamran Ince – He was my composition teacher and, I think, an accessible introduction to the world of “new music”, which is what we call new composed “classical” music. The term “classical” is a widely spread misnomer. It actually refers to music written in the late 18th and early 19th centuries, but most people use it to mean high brow music written for instruments like violins. So whatever you call it, Kamran Ince writes some beautiful instances of it. Specifically check out his chamber music, such as Domes and Arches.
  • Charlie Wood – I have had the pleasure of playing with Charlie on a few occasions. He is a R&B singer/organist/composer from Memphis and writes some of the most intelligent songs you’ll hear. My favorite album of his is “Who I Am”.
  • John Zorn – Zorn has been around for a long time and is a leader in the world of Avant Garde music. He’s also one of the most amazing saxophonists ever. If you’re new to this kind of thing, his Masada quartet (“radical Jewish music”) produces some great stuff that’s accessible to first time listeners. If you’re looking for something to shock your aural taste buds, try Painkiller (metal-tinged noise) or Naked City.

Dave Thomas: I listen to just about anything that’s interesting. My playlist here is very varied, and I try to add new stuff to it farily regularly. I know people who are trained as musicians, and I tend to ask them what they’re listening to. Sometimes that leads to challenges: my ear isn’t as developed as their ears. But often it leads to whole new areas of cool stuff. So I’d recommend everyone should find a friend who knows more than you do about music and ask them to surprise and challenge you. (That advice probably applies to just about everything, thinking about it.) It’s easy to find music that stimulates your lizard brain. Get into the habit of looking for the stuff that engages at a higher level too. And, like everything, have fun with it.

MapReduce at RailsConf Europe

Posted by Jon
on Thursday, July 03

This September, I’ll be presenting at RailsConf Europe on EC2, MapReduce, and Distributed Processing. The talk will explain the MapReduce approach to distributed processing, will show a few example implementations, and will discuss MapReduce vs. other distributed processing techniques.

Whether you’ll be there or not, if you’re interested in learning more about MapReduce, here are some resources. I’ll write a few more posts on the subject before the conference, so watch this space as well.

Cluster Computing and MapReduce is a great series of video lectures given to Google interns in 2007. The first two are the most appropriate: the first introduces distributed processing concept, while the second covers MapReduce itself.

MapReduce: Simplified Data Processing on Large Clusters is the paper by Jeffrey Dean and Sanjay Ghemawat of Google that got things going in the first place.

MapReduce for Ruby: Ridiculously Easy Distributed Programming discusses MapReduce and introduces Starfish, a Ruby library for distributed processing. Starfish is not a MapReduce implementation, however – it takes a somewhat different approach to distributed processing.

Skynet (a few writeups: InfoQ, Dion Almaer) is another Ruby-based distributed processing system inspired by MapReduce.

Writing Ruby Map-Reduce programs for Hadoop discusses using Ruby to wrap Hadoop, a MapReduce-like system built in Java.

Introduction to Parallel Programming and MapReduce at Google Code University, a good overview of distributed processing and the MapReduce approach.

And finally, one article that you should avoid:

MapReduce: A major step backwards compares MapReduce to relational databases, and says that MapReduces loses out because it doesn’t support database indices, database views, Crystal reports, etc. Basically, the complaint is that MapReduce isn’t SQL compliant. WTF? Clearly, the author(s) didn’t understand what MapReduce is. The problem, as explained elsewhere, is that the authors thought that MapReduce == CouchDB/SimpleDB. Which is obviously not true. %s/MapReduce/SimpleDB the original article and it makes some sense. But long story short, this article will teach you nothing about MapReduce, and will likely confuse you further. So stay away.

Just say 'no' to Vegas, RailsConf

Posted by Jon
on Friday, June 27

If you’re just going to get one thing out of this article, it’s this: Vegas is a bad idea for RailsConf ‘09.

Here’s the story. At the end of RailsConf ‘08, in Portland for its second year, Chad Fowler tentatively announced the location of next year’s RailsConf, saying something like “We’re not sure what you’ll think about this, but what about Las Vegas?” Upon which the crowd erupted in cheering, thereby supposedly confirming the Vegas idea. But afterwards, I talked to a dozen people who said they thought Vegas was a bad idea. And in the month since then, I haven’t talked to a single person who was excited about it. That’s how crowds work, I guess – the 10% of Rails developers who enjoy gambling, strippers, and steak applauded loudly, and the 30% of don’t have a strong opinion one way or another got swept up in the excitement.

Of course, it doesn’t really matter – holding the next RailsConf in Las Vegas won’t kill Rails, and won’t set the Rails community on a future of drunkenness, adultery, and gambling. I’m sure most conference-goers will fly in, attend sessions, have dinner, and hack in their hotel lobbies, just like any other conference.

But it’s still a bad idea.

DARE to keep RailsConf different

If you haven’t already, check out Giles Bowkett’s recent post on the situation. He talks about Ruby Central’s desire to “keep RailsConf weird.” What’s especially confusing, as Giles points out, is that the Vegas announcement came after DHH’s keynote, which said that we should get more sleep and use our advantages for good, not evil (in the form of hookers and fur coats). Charles Nutter made a similar point, saying that Rails + enterprise doesn’t have to mean steak and strippers.

As Giles puts it,

DHH is saying, “No hookers! Choose a life well-lived!” And RailsConf is like, “Screw Portland! We’re going to Vegas!”

Portland was a great place for a conference. Portland has great food & beer, cool hotels, the world’s biggest bookstore, and a dozen movie-theater-pubs. Every night after the conference, a thousand Rails geeks would descend upon the city’s first-rate brewpubs and coffee shops to meet people, eat and drink, and discuss programming, politics, philosophy, or whatever. $5.99 steak buffets just won’t be the same. I really enjoyed my twice annual trip to Portland, and would love to go back again. If Ruby Central wants to keep RailsConf edgy, it can’t do much better than Portland.

Of course, if we don’t want to do RailsConfs ‘09, ‘10, ‘11, and ‘12 in the same place, there is no lack of great cities to consider. What about Seattle? Boston? Austin? San Diego? San Francisco? New York? Kansas City? Toronto? Vancouver? Minneapolis?

Is RailsConf worth it?

So enough about Vegas. I get asked from time to time if RailsConf is worthwhile. My answer is: “Yes (I think).”

The first RailsConf was small(er), high energy, and novel. Rails was just on the brink of mainstream acceptance, which is a fun time in the life of a technology – it’s growing rapidly but is still edgy. The sessions were generally high quality, and the keynotes were excellent (Dave Thomas, Martin Fowler, David Heinemeier Hansson, Paul Graham, and _why).

RailsConf 2007 was a completely different conference, what with O’Reilly and 1600 attendees. Ze Frank was great, but the overall session quality was pretty weak. There were some great talks, to be sure; but there weren’t enough really deep technical talks, and some of the presenters didn’t seem to have really practiced. Things were probably set off on the wrong note from the beginning: Day 0’s 3-hour tutorials were mostly disappointing. Also, there seemed to be an abundance of non-programmer business folks, probably there to check out this new thing called Rails.

This year’s conference was quite a bit better. It had the same polished feel that O’Reilly brings, which is both good and bad. Keynotes were mixed. But mainly, the sessions were mostly really good. Whatever David Black, Chad Fowler, and Rich Kilmer did to improve the session quality, it worked. Interestingly enough, it seemed like the non-programmers were gone this year.

RubyConf 2007 (the only one I’ve attended) felt a lot like RailsConf 2006. It had a similar size, a similar venue, and somewhat similar atmosphere. Sessions were good, and I learned quite a bit. Most of the people there were Rails developers, but they were the ones who were interested in Ruby as a language and not just as the technology behind Rails.

So my answer is that RailsConf is worth it, as long as it follows the 2008 path. It will never look like 2006 again; big RailsConfs are here to stay. But 2008 was reasonably graceful for a big conference. I just hope it doesn’t fall into the 2007 trap, which will happen if it tries to cater to managers and the mainstream. It’s impossible to know what next year’s conference will look like, though I think the success of 2008 was a conscious rejection of some of the failures of 2007. And if you’re looking for something smaller and edgier, there’s always RubyConf, RubyFringe, and the regional conferences.

(Two updates.

First, the title of this article was ambiguous, so I changed it from “Just say ‘no’ to RailsConf Las Vegas” to “Just say ‘no’ to Vegas, RailsConf”. The title was supposed to say “Vegas is a bad idea for RailsConf,” not “Don’t attend if it is in Las Vegas.” If you decide not to attend because of the location, that’s fine – but I’ll consider being there either way.

Second, thanks to Chad Fowler and David A. Black for weighing in. Conferences are a lot of work, as Luke knows on a smaller scale, and I can’t imagine what kind of work Ruby Central puts into RailsConf each year. Especially when they’d probably be hacking. :) So thanks for the work, and the conferences that result.)

Lightning talk: How not to present at RailsConf, in 14 easy steps

Posted by Jon
on Friday, June 06

I gave this Lightning Talk on the last day of RailsConf.

Asynchronous Processing with Ruby on Rails (RailsConf 2008)

Posted by Jon
on Tuesday, June 03

As promised, here are the slides from my RailsConf talk on Asynchronous Processing. The slides don’t always make sense without the context, so I’ve added a few labels here and there. If something doesn’t make sense, feel free to ask about it.

Summary: if your web app has any sort of complexity, you’ll likely have some operations that are best processed asynchronously. In particular, any operation that takes a long time, is resource-intensive, or isn’t naturally triggered by http or by a human is a good candidate for asynchronous processing. You can build your own solution, but before you do, check out Bj for most uses, workling/starling for high performance or scalability, Amazon web services for distributed processing, and plain old cron for basic cases.

I also gave a lightning talk called “How not to present at RailsConf in 14 easy steps” – it’s just 2-3 minutes long, so I think I’ll turn it into a screencast. Look for that later this week.

Get more sleep

Posted by Luke Francl
on Saturday, May 31

There was one thing David Heinemeier Hansson said at his RailsConf keynote that I unequivocally agreed with. But besides the value of the 4 day work week, he also brought up the topic of sleep.

I’ve been reading the book Brain Rules by molecular biologist John Medina. It’s puts the latest research about how the brain works into a popularity accessible form, and makes recommendations for individuals, schools, and businesses to better harness human potential for learning.

Brain Rule #7 is Sleep well, think well. One of the interesting facts in the book is about the cumulative effects of sleep deprivation. After a few days of sleeping less than needed, human performance is equivaalent to going 48 hours without any sleep at all. Also, do you ever get the urge for an afternoon nap? According to Brain Rules, a nap can improve cognitive performance almost 30%.

So, yeah. Get more sleep.

Hello Portland!

Posted by Luke Francl
on Thursday, May 29

The whole Rail Spikes crew is here in Portland for RailsConf! Feel free to say “hi.”

Jon Dahl from Slantwise Design

Jon will be speaking about Asynchronous Processing with Rails at 11:45 on Saturday. Come check it out.

Tea time at Mariage Freres

I (Luke) am going to be checking out the Advanced Rails Recipes book signing during lunch on Friday. Come say hi and meet some other developers.

Eric Chapweske

This is Eric’s first RailsConf, so be gentle. He plans on taking up the Portland microbrew circuit.

See you at RailsConf '08

Posted by Jon
on Monday, February 25

It looks like I will be speaking at RailsConf again. This time, I’ll be discussing asynchronous processing.

If you’ve worked on a Rails app that strays beyond basic CRUD, you’ve probably had to think about running some actions in the background – that is, outside of the HTTP request/response cycle. I’ve had to worry about this on quite a few projects, and contributed a recipe to the forthcoming Advanced Rails Recipes book. The recipe discussed how to package a Ruby script as a daemon, which is a simple and powerful approach to background processing. But there are quite a few others, including (but not limited to): AP4R, cron, starling, workling, beanstalk, backgroundrb, spawn, background_fu, and Amazon SQS. And I’ve left out quite a few.

So as I prepare for the conference, I’d love your feedback. What options have you tried? What has worked well or worked poorly? What would you like to see at the conference?

RailsConf slides (video transcoding)

Posted by Jon
on Friday, May 18

Here are the slides from my RailsConf presentation today. I’ve taken out the video; if you’re really interested in seeing it, send me an email. I’ve also created a version without any images at all in order to save space.

Video transcoding slides, with images (4.5M)

Video transcoding slides, no images (207K)

I’d also love your feedback from the talk if you were there.

Slantwise goes global

Posted by Luke Francl
on Wednesday, May 02

Two of us from Slantwise (and Rail Spikes) will be going on the speaking circuit this summer.

Jon will be speaking about video transcoding and Rails at Rails Conf 2007. The series of blog posts here at Rail Spikes will form the basis for his talk. Jon will be speaking on the first day of the conference, May 18th.

My talk on mobile integration with Rails was accepted to Ostrava on Rails in the Czech Republic, June 22-23.

I’ll be speaking about how to connect your Rails application to mobile phones with MMS2R and some other techniques. It looks like I’ll have some good company, too!

Both of these presentations have grown out of some recent projects we’ve been working on that we’ll be making public shortly. So stay tuned!