Last weekend, I participated in my first Rails Rumble. Rails Rumble is a 48-hour app building contest. We started from scratch Friday evening – you can have concepts and notes on paper, but no code or digital UI assets – and stopped Sunday evening, after 48 hours. You can use open-source code and public web services, and we made liberal use of both.

Our team consisted of myself and three of the Sevenwire crew: @fowlduck, @brandonarbini, and @steveheffernan. That’s two developers (Nate and myself), one developer/UI combo (Brandon), and one UI guy (Steve). All in all, a really good mix for the app. We’re also the team behind two video encoding services: Zencoder and FlixCloud.

Check out our app (and the 21 other great finalists) and vote at http://r09.railsrumble.com/entries. Voting ends this weekend, so do it soon.

The App

Our project was ZenVDN, a video distribution network. In other words, a place to upload video that you want to publish, i.e. via your blog or website. Upload one or more videos, and they’re transcoded into web and mobile formats, and sent to a Content Delivery Network for distribution.

After that, you’re given a page to manage each video, with HTML embed code to plug the video directly into your blog or website. You can also link directly to the videos, if you want to use your own player. And finally, each video has a public page on the ZenVDN site if you want to share the video directly.

So it’s a complete start-to-finish video publishing platform. Let’s say you’re Ryan Bates of RailsCasts. You can compress, upload, and host your own video files manually, or you can use a service like ZenVDN to do that for you. (I emailed Ryan about this, by the way, and he prefers the manual route. ;)

Another way to look at it: a better YouTube for video publishers. YouTube and its peers were designed for wide-scale video sharing, not for video producers and content owners. If you don’t mind YouTube’s quality and watermark, and you don’t mind your video being shared publicly on YouTube, ZenVDN probably isn’t for you. But if you want better quality and to own distribution of your videos, check us out.

What’s cool? A multi-file uploader with progress; direct uploads to our CDN, for speed and scalability; video watermarking; video thumbnails; wide input video support; a Flash Player integrated into the embed code; and detailed statistics (by video, by date, by format).

What’s missing? Again, it’s a working end-to-end product, but we’d like to do a lot more. Examples: Ogg support (for HTML 5), an RSS feed for videos, more public/sales information, and better privacy controls.

And, of course, paid subscriptions. We hoped to get e-commerce done during the Rumble, probably using Spreedly, but we ran out of time. Maybe in a 72 hour Rumble. In the meantime, our Free level limits the number of videos you can upload, and the amount of video you can stream. A paid level would increase these limits and let you use your own watermark (instead of the ZenVDN watermark on free accounts).

All in all, we’re really happy with where we ended up. I’m proud to say that Obie briefly questioned whether we could build the whole thing in a weekend. That’s praise.

The experience

The Rumble was way more fun than I expected. I had just worked a hard week, and a part of me was dreading the prospect of a long weekend of work. But it was actually a blast.

Why? Development flow, I think. Development flow is a really fun experience. It’s probably why most of us are developers, after all. We like to build things; we like to solve problems; and we like to work effectively. I’ll sometimes go days, or even weeks, without experiencing concentrated development flow like I did during the Rumble. (Stupid meetings.) So the Rumble was a really great experience.

Our team really clicked. We had a great mix of skills: across the four of us, we had one designer, two front-end coders, and three back-end coders. Besides the initial design concepts, every task could have been handled by more than one person, so tasks rarely sat in the queue for long.

We tried really hard to avoid rushing at the end. We stopped development with 3 hours to go, and two of us started testing, while two others recorded the screencast for the homepage. But it didn’t work out quite so smoothly. The screencast wasn’t done until about T-30 minutes, and we were checking in fixes and refinements until about 6:45. Then a minor Git snafu, and panic ensued. Our final submission came down to the wire.

Finally: sleep and breaks. Call me weak, but I like to sleep. I got 8 hours/night during the Rumble, which definitely improved my experience (and the quality of my code). We ate lunch at our desks, but took a 90-minute dinner break on Saturday, and stopped several times for a game of darts.

Lessons learned

1. Blitzes can be fun and effective. I’m inclined to try a Rumble-like iteration every few months, to avoid project monotony, and to ship stuff quickly when necessary. I did ~3-4 days of work during the Rumble, so I figure a 3 day Rumble plus 2 days of vacation evens out to about a week of work.

2. Focus is essential. If I had three 30-minute meetings during the Rumble, my contribution would have been cut in half. Good reminder of makers’ schedules.

3. Don’t rush at the end. We left three hours for testing and padding. We should have left six.

4. Prioritize well. If we had tackled e-commerce on Day 2 (as I almost did), we wouldn’t have finished our core product. Build the Minimum Viable Product first, and then move on to concentric circles of improvement.

5. Small projects can work. I have a bias against small projects; 3 month gigs feel so much more comfortable to me as a consultant than 3 week gigs. But done properly, shorter projects can work fine. We did ~$15,000 worth of work over the course of the weekend. No reason that experience couldn’t translate into a client project.

Next steps

So what’s next for ZenVDN? We’d really like to get a few video publishers using it. (Talk to me if you want to be a beta customer.)

And we want to monetize the site, of course.

We think it complements our suite of video-related products well – Zencoder is the core software; FlixCloud makes it an easy web service; and ZenVDN brings video publishing one step closer to the producers.

We have some other ideas for ZenVDN. But if you have an interest in online video, or are a publisher/producer yourself, we’d love to talk more!

It’s been 2 months since I’ve posted about Zencoder. You may be glad to know that the project is still alive and is nearing completion. It has moved a little slower than we initially expected (we had hoped to finish in Nov/Dec), but the end is in sight. So if you are interested in a complete video transcoding solution for your web application, request more info at zencoder.com. And if you’ve inquired in the past, and you are interested in a demo, let us know: we’re starting our online demos this week.

But on to the sneak peak.

Scaling

Zencoder scales both horizontally and vertically to a nearly arbitrary limit. You can scale vertically (obviously) by increasing the speed of your machines: the faster the CPUs, the faster the transcoding, and any Linux or OS X machine should work. And you can scale horizontally by increasing your number of servers, as high as your licensing allows.

You can scale the same way on EC2, by moving from small instances to large instances, or by increasing your number of instances.

What’s important is that Zencoder works the same whether you run it with 2 servers or with 200 (or more). The system will happily distribute your jobs across however many servers you have. (Note that it will do this by distributing each job to a different server, not one job across many servers – the latter approach can finish each individual job more quickly, but the Zencoder approach should have a somewhat higher overall throughput, since it is a bit more efficient. And it is significantly simpler, of course.)

This means that Zencoder can grow with your application. If you underestimate your traffic, or your site grows exponentially, just add another server or fire up a new EC2 instance.

Finally, you don’t even need to restart Zencoder to add additional servers, so expansion is almost completely seamless. If you’re running on dedicated hardware, just configure Zencoder on a new server and start it up: it will begin working immediately. On EC2, things are even easier: click the big green “Launch new worker” button, and you’ll have a new server up and running in about 3 minutes.

Reliability

On the reliability front, Zencoder is built with three goals in mind.

First, errors should be few and far between. This goal is obvious, and is shared with most software (except for some software coming out of Redmond, WA). And Zencoder is a reliable system in this respect. The code is reasonably compact, and it is well tested (both by unit tests and by humans).

Second, if any (or all) pieces become unavailable, your website should continue to work. In other words, if a user uploads a video at your site, and Zencoder is unavailable for some reason, your website should continue functioning as-is. Similarly, if your site is down, and Zencoder finishes a job, your website should receive the job as soon as it comes back up. As a result, we have designed the system such that you can pull the plug on any piece – or multiple pieces – and as soon as everything is back online, the system picks up right where it left off.

Third, no job should ever get lost in the system. This is accomplished through integrity checks – between the queue and each transcoding node, and between the Zencoder system and your application. You can even wipe the Zencoder queue, and if your application is still waiting on any jobs, they will be automatically recreated.

The result is that Zencoder is almost a self-healing system. It isn’t perfect (yet), but it is robust, reliable, and scalable.

MinneDemo is a quarterly event in the Twin Cities where local startups and technologists demo their products.

Zencoder is distributed video transcoding software, built by Slantwise, that is scalable, reliable, and flexible.

The next MinneDemo event will be held December 6th at O’Gara’s Garage in St. Paul, and I’ll be presenting Zencoder there. This will be the first public showing of Zencoder, and I’m excited to be demoing the system after four months of development.

Whether or not you’re interested in video processing, if you’re reading this blog and you live within a 100 mile radius (or even 270), you should come. MinneDemo and MinneBar are the most worthwhile tech events going on in Minnesota.

MinneDemo works like this.

• Show up at 6:30 for a free beer/wine/soda or two.

• Meet interesting people: developers, designers, venture capitalists, tech lawyers, startup gurus, information architects, enterprise software providers, ISPs, and more. About 300 people typically show up.

• Have another beer/wine/soda.

• Watch five or six software demos, including Zencoder. The demos each last 15 minutes or less, and PowerPoint is not allowed, so you actually see working software, not marketing-speak.

• Continue to meet people, get ideas, land projects, and find partners. We met our attorneys, one business partner, two clients, and several friends at MinneDemo and MinneBar.

The event starts at 6:30pm on Thursday, December 6th (drinks and food), and the demos will start at 7:30pm. The demos will finish around 9pm, but some people will stick around for hours after that.

Many thanks to Dan Grigsby and our own Luke Francl for organizing MinneDemo, and Ben Edwards for starting MinneBar!

So how does Zencoder do its video transcoding? What codecs and formats does it support? And what about the critical issue of codec licensing?

Licensing

Let’s start with the question of licensing. Codecs are licensed in three main ways.

1. Commercial software (On2, Nellymoser)
2. Patents and standards requiring licenses (MPEG-2, AAC, MP3)
3. Free or unenforced formats and codecs (Ogg, Matroska, AVI)

If you’re going to transcode and distribute video, category 3 is easy. Category 1 is straightforward, and these commercial codecs work just fine with Zencoder.

Category 2 poses the real problem. In my estimation, three of the most important video codecs – MPEG-4, H.264, and MPEG-2 – and three of the most important audio codecs – AAC, MP3, and AMR – all require licensing direct from their patent holders and/or licensing authorities. Those six invaluable codecs are represented by four separate licensing bodies (representing hundreds of patent holders), each of which has separate terms, prices, minimum fees, etc. And when you move to the second tier of codecs, things get even more complicated.

Zencoder takes care of this for you. With Zencoder, you receive licenses to these and other critical codecs. We’re still working out the specifics, but when Zencoder launches, we plan on including licenses to 10-20 patented codecs and formats, covering all of the major ones and several in the second or third tier.

Compatibility

Zencoder can handle every major video codec, audio codec, and container format. And we do this legally, taking care of most licensing issues for you.

Here is a partial list of formats and codecs that we support.

• H.264
• FLV
• MP4
• 3GP
• VP6 (with commercial license)
• MP3
• AAC
• MPEG-4 Video, XviD
• MPEG-2
• AVI
• Ogg, Theora, Vorbis
• Nellymoser (with commercial license)
• And 100+ more

The goal is to decode everything that you can throw at Zencoder, and to encode everything that you have a good reason to encode.

Slantwise has been working on a video processing product called Zencoder for the last several months. Zencoder is a distributed multimedia processing system, written in Ruby, that is powerful, flexible, and scalable. It also integrates easily with any application that needs video transcoding. The product itself is in its final stages of testing, and will launch soon, so head over to http://zencoder.tv and fill out the contact form if you want more info.

In this first “sneak peek” post, we’ll look at Zencoder from a high level. Where does it run? How does it integrate with your application?

Two Hosting Options

Zencoder is a server, or collection of servers, that you host. The first server is the “mind” of Zencoder, which queues jobs and handles all communication with your application. The other servers are “workers” that do the actual video transcoding.

You can host Zencoder on your dedicated hardware or on Amazon EC2. Both work equally well, but we think that the EC2 approach is especially interesting, and Zencoder does some cool things with EC2. (More on that in a few weeks.) But you can mix-and-match EC2 instances and dedicated servers, so you don’t have to choose. You could use dedicated hardware for most of the transcoding work, and use EC2 for overflow work. Or use EC2 for most of your work, but one ultra-fast dedicated machine for high-priority jobs.

Easy Integration

Zencoder integrates via REST web services, so it is completely technology-agnostic. Your application can be written in Java, Ruby, PHP, .NET, Python, Erlang, Ada, or Assembly, as long as it can communicate via HTTP. It can be a web application, server software, or anything else that can send and receive HTTP requests.

To submit a job to Zencoder, you just need to send a POST request to your Zencoder server with the basic details of the job: input file, job id, and task/recipe. Zencoder will then do its work and send a request back to your application with the results of the job: status, output file attributes, output file location.

So your application only needs to do two things to integrate with Zencoder:

• Send a HTTP request to your Zencoder server when you want to trigger a new transcoding job
• Handle a HTTP request from Zencoder with the results of a job

We also provide a Ruby on Rails plugin that handles these things for you; so if you’re running Rails, this work is done for you. You just have to install the plugin and configure a few things, and your application is ready to go. Eventually, we plan on providing similar integration classes for Java and PHP. But even if you aren’t using one of these technologies, the integration work is pretty simple on your end, and we’ll help you through it.