Update: stubbing a single integration point shaved 22 seconds off of my unit tests, reducing test time from 35 seconds to 13. See below.
The first step to faster tests is knowing what is slow. Fortunately, this is dead simple with the test_benchmark plugin by Tim Connor, and originally built by Geoffrey Groschenbach. Install the plugin, and when you run your tests via Rake, you’ll see handy output showing you the slowest tests, and the slowest test classes.
Step 1: Install the plugin.
script/plugin install git://github.com/timocratic/test_benchmark.gitStep 2: Run your tests
rake test
Here is a bit of output when I run the unit tests for FanChatter:
Finished in 34.838173 seconds.
Test Benchmark Times: Suite Totals:
25.393 MailReceiverTest
4.520 PhotoTest
1.429 REXMLTest
0.961 TeamTest
0.846 MessageTest
Pretty useful information. Almost 75% of our unit testing time is taken up in the MailReceiverTest. So if we want to speed up our tests, we need to make our MMS testing faster. Looking at that code, I see this line over and over:
MailReceiver.receive(fixture_mms(:fixture_name)) |
This method reads a test email message from the filesystem, and runs it through our mail parsing method. This is basically an integration test, hitting at least two integration points. So if we can remove these bottlenecks, we can reasonably expect a fairly large improvement in our unit test speed.
I think we could realistically reduce our unit testing time from 34 seconds to <15 seconds just by refactoring this one test method.
Other options
The test_benchmark plugin fires whenever you run your tests with rake. Tim recently patched the plugin to not fire when run with autotest, which is great. Personally, though, I don’t want to see this benchmark information every time I run my tests. So I added the following line to my test.rb environment file:
ENV['BENCHMARK'] ||= 'none'
Now, the benchmarks don’t run by default. If I want to see them, I call:
rake test BENCHMARK=true
And if to see full tests, showing the time it takes to run every test in the system, just call:
rake test BENCHMARK=full
That’s it. You still have to speed up your tests, and there are many ways to do that (from mocking to simply reducing the number of calls to expensive methods), but knowing what’s slow is half the battle.
The stirring conclusion (update)
I spent a few minutes optimizing these slow tests today. First, I tried rearranging the tests to reduce unnecessary calls to the slow method (MailReceiver.receive(message)). I was able to speed MailReceiverTest from about 25 seconds to 17. Not bad, but still slow.
The real problem is that this method saves a photo. It creates a Photo record that includes a file, treated sort of like an upload, like this:
1 |
photo.uploaded_data = mms.file |
This is what was slow. But my unit tests don’t actually deal with the file being saved to the filesystem; they test other things, like the right records being created, confirmation emails being sent, etc.
So I decided to try bypassing this file save/upload by stubbing the uploaded_data= method. I put the following at the top of my test class:
1 2 3 |
def setup Photo.any_instance.stubs(:uploaded_data=) end |
And voila! MailReciverTest went from 25 seconds to 17 seconds to 3 seconds.
Rspec has a similar function built in. You can get benchmark info on your 10 slowest test by adding:
to spec/spec.opts (or passing it as a command line option to the spec script).
@Sam, the problem though is if you have a lot of specs, only showing the 10 slowest might not be enough. You’ll have to write your own formatter but it’s easy to do.
Thanks for this article, Jon. Using the plugin, I’ve already killed two tests (and some related code) that were adding 8 seconds to test:units!
Thanks for the coverage, Jon. It’s Connor with an “o”, though. :D
@Will, that’s cool. Lately I’ve been loving the Rspec nested formatter for running small groups of tests. Good to know it’s so easy to create your own.
Sam and Will: thanks for the RSpec pointers.
Tony: glad this was able to save you some time!
It’s nice to see an old project get a second life!
Tim: fixed.
Geoffrey: thanks for writing the initial version!
I was just thinking yesterday about building something like this.
Thanks!
Nice one!
I get errors when I try this in Rails 2.1.2—
../vendor/plugins/test_benchmark/init.rb:3:in `evaluate_init_rb’: You have a nil object when you didn’t expect it! (NoMethodError) The error occurred while evaluating nil.strip from C:/tools/rails/ruby/lib/ruby/gems/1.8/gems/rails-2.1.2/lib/rails/plugin.rb:95:in `evaluate_init_rb’ from C:/tools/rails/ruby/lib/ruby/gems/1.8/gems/activesupport-2.1.2/lib/active_support/core_ext/kernel/reporting.rb:11:in `silence_warnings’ from C:/tools/rails/ruby/lib/ruby/gems/1.8/gems/rails-2.1.2/lib/rails/plugin.rb:91:in `evaluate_init_rb’
Can I assume this needs 2.2 or later?