It’s been 2 months since I’ve posted about Zencoder. You may be glad to know that the project is still alive and is nearing completion. It has moved a little slower than we initially expected (we had hoped to finish in Nov/Dec), but the end is in sight. So if you are interested in a complete video transcoding solution for your web application, request more info at Zencoder.tv. And if you’ve inquired in the past, and you are interested in a demo, let us know: we’re starting our online demos this week.
But on to the sneak peak.
Scaling
Zencoder scales both horizontally and vertically to a nearly arbitrary limit. You can scale vertically (obviously) by increasing the speed of your machines: the faster the CPUs, the faster the transcoding, and any Linux or OS X machine should work. And you can scale horizontally by increasing your number of servers, as high as your licensing allows.
You can scale the same way on EC2, by moving from small instances to large instances, or by increasing your number of instances.
What’s important is that Zencoder works the same whether you run it with 2 servers or with 200 (or more). The system will happily distribute your jobs across however many servers you have. (Note that it will do this by distributing each job to a different server, not one job across many servers – the latter approach can finish each individual job more quickly, but the Zencoder approach should have a somewhat higher overall throughput, since it is a bit more efficient. And it is significantly simpler, of course.)
This means that Zencoder can grow with your application. If you underestimate your traffic, or your site grows exponentially, just add another server or fire up a new EC2 instance.
Finally, you don’t even need to restart Zencoder to add additional servers, so expansion is almost completely seamless. If you’re running on dedicated hardware, just configure Zencoder on a new server and start it up: it will begin working immediately. On EC2, things are even easier: click the big green “Launch new worker” button, and you’ll have a new server up and running in about 3 minutes.
Reliability
On the reliability front, Zencoder is built with three goals in mind.
First, errors should be few and far between. This goal is obvious, and is shared with most software (except for some software coming out of Redmond, WA). And Zencoder is a reliable system in this respect. The code is reasonably compact, and it is well tested (both by unit tests and by humans).
Second, if any (or all) pieces become unavailable, your website should continue to work. In other words, if a user uploads a video at your site, and Zencoder is unavailable for some reason, your website should continue functioning as-is. Similarly, if your site is down, and Zencoder finishes a job, your website should receive the job as soon as it comes back up. As a result, we have designed the system such that you can pull the plug on any piece – or multiple pieces – and as soon as everything is back online, the system picks up right where it left off.
Third, no job should ever get lost in the system. This is accomplished through integrity checks – between the queue and each transcoding node, and between the Zencoder system and your application. You can even wipe the Zencoder queue, and if your application is still waiting on any jobs, they will be automatically recreated.
The result is that Zencoder is almost a self-healing system. It isn’t perfect (yet), but it is robust, reliable, and scalable.
