Slow tests are a bug

Posted by Jon
on Tuesday, March 10

I’ve been doing TDD for about three years now. Once I figured out how to do it right, it became a natural part of how I program, and I can’t really imagine doing development without it. This isn’t to say that TDD is the only approach to writing quality software or that unit testing it the only kind of testing that matters. But it sure is useful.

The Ruby world talks a lot about TDD, moreso than many other developer communities. We have not one, not two, but at least half a dozen testing libraries that are actively being used and developed. For most Ruby developers, the question isn’t “Do you test?” but “BDD or TDD?” or even “RSpec, Shoulda, or Bacon?” We often use at least 2-3 layers of automated testing, and sometimes use different tools for each layer. Most Ruby conferences devote at least a few talks each day to testing-related topics. We’re test fanboys and -girls, for better or for worse.

But in spite of this, we rarely talk about test speed. Sure, there are purists who believe that unit tests shouldn’t touch the database because anything that touches the DB is actually an integration test. But few Ruby testers actually take this long and lonely road, and I personally prefer tests that talk to a database, at least some of the time.

And it’s true that others have written libraries to distribute their tests across multiple machines. But that’s the exception that proves the rule – the only reason to distribute your tests is that they’re too slow to begin with.

Most Rails projects I’ve worked on have ended up at around 3,000-15,000 lines of code, with a roughly as many lines of test code, and most have test suites that take a minute or more to run. Our test suite for Tumblon, for instance, churns along for 2.5 minutes. This is a too slow. And slow tests are a problem for at least two reasons: they slow down your development and decrease code quality.

1. Slow tests slow down development. If you’re practicing TDD, you want to see a test fail before you make it succeed. Two minutes is far too long for this feedback loop to be effective. Of course, you can (and should) just run the test classes that correspond to your code as you program – no need to run your entire test suite every time you write your failing tests. But even still, the test time bar should ideally be set quite low. Frequent 5-10 second delays are enough to break my concentration, and I find myself cmd-tabbing over to other programs if I have to wait more than a few seconds for a test to run. I don’t know of any hard-and-fast rules, but I know that as soon as my test suite runs longer than 30-45 seconds, and individual test classes take longer than 2-3 seconds, I’m less happy and less productive.

2. Slow tests decrease code quality. There are two simple reasons for this. First, if slow tests break your flow, you’re not only going to write code more slowly: you’re also going to write worse code. Second, if your tests are too slow, you’re not going to wait for them to finish before you move on to the next task. Or worse, you’re not going to run them at all.

So, how can I speed up my tests?

Fortunately, this problem can be addressed. There are plenty of ways to speed up tests. On a current project, we’ve managed to cut our test time substantially – a recent test refactoring cut test time from 129.45 seconds to 31.04 seconds, without removing any tests. That’s a 76% speedup. But we still have room for improvement.

Really quickly, here are at least five ways to speed up your test suite. I hope to post more on each of these over the next month or two.

1. Use a test database instead of fixtures/factories/etc.

2. Only touch the database when necessary

3. Organize your tests to avoid duplicate execution

4. Separate slow tests out into a lazier testing layer

5. Run a Rails test server

I’d love to see the Rails community devote more of its enthusiasm for testing to the question of test speed. There’s nothing wrong with improving our test frameworks, and let’s keep doing that. But let’s also make these frameworks fast.

Comments

Leave a response

  1. James LudlowMarch 10, 2009 @ 12:21 PM

    During code development I use a mixture of autotest and manually running single tests from TextMate. That focusses the testing, and the time required to run it, on just the files that you’re touching. autotest will run the entire suite in the background, where you can freely ignore it until it pops up with an error.

    Although I use fixtures heavily, I’ll agree that they are a godawful mess. Foxy Fixtures is a step in the right direction, but I’m going to try to dump fixtures entirely the next time I get to start a Rails app from scratch.

  2. Jon DahlMarch 10, 2009 @ 12:54 PM

    James: I use autotest too and love it. It helps with many of the problems of slow test suites by generally only running the tests you care about.

    But it doesn’t completely take care of the slow test suite problem. For example, if you have a routing error in one of your functional tests, and you change routes.rb, autotest will rerun your entire test suite (since routes.rb doesn’t match up to a single test class). And autotest can get “clogged up” when you have a lot of failures at the same time (e.g. a migration gone bad or a major refactoring).

  3. Daniel BergerMarch 10, 2009 @ 01:28 PM

    One of the great features added to Test::Unit 2.x are the startup & shutdown methods. These are similar to setup and teardown, but they only run once per test instead of once per test case.

    Moving code out of setup & teardown and into startup & shutdown could dramatically improve your test performance, depending on what you’re doing in your setup methods and how many test cases you have.

  4. DanMarch 10, 2009 @ 02:16 PM

    It is funny the community seems to go both ways on this. While we worked to keep our unit tests much smaller and faster on our most recent project. We also see that people are going away from simple single step functional tests to more stories the do full stack testing that uses webrat or cucumber. Saying that for functional the best thing is to actually log a user in through the whole site instead of just creating the session objects.

    It is great to keep test suites fast, but as a project grows it just isn’t possible to have good coverage and keep incredibly fast tests. That or you just start moving more and more of the test suite to later testing steps on CI and then begin to loose value of the immediate feedback that testing provides.

    Obviously, since I am building devver.net to distribute tests over a cluster of machines I am a bit biased. I did start building this to solve my own pain from seeing projects test suites always grow to large sizes and seeing that it hurts the entire development process.

  5. Software TesterMarch 10, 2009 @ 06:11 PM

    How about this. Run your tests at night. Then anything that finishes within 8 hours is a suitable test.

    - Software Tester http://verifcation-and-validation.blogspot.com/

  6. Sam GoldsteinMarch 10, 2009 @ 07:25 PM

    Just a tip for Rspec users that I find very useful: If you pass an -l <number> flag to the spec script it will only run the example on that line or in that describe block. This makes it really easy to just run one test and not break the flow.

    We add the following command to .vimrc (we work mostly in Vim): map !S :! script/spec % -f n -cl <c-r>=line(’.’)<cr><cr>

    Hitting !S with this defined executes the example or describe block under the cursor. There may be comparable aliases/commands that emacs or Textmate users can define.

  7. Luigi MontanezMarch 10, 2009 @ 10:42 PM

    I would love to see a HOWTO on profiling tests to determine exactly where your bottlenecks are. I assume it involves ruby-prof, but I’ve just never gotten around to it.

  8. Bryan HelmkampMarch 11, 2009 @ 02:35 AM

    I’ve come to disagree with this idea “the only reason to distribute your tests is that they’re too slow to begin with”.

    To be sure, plenty of people write unnecessarily slow tests, and they shouldn’t. More focus should be spent on avoiding that.

    But, with an application of significant size and complexity (codebase at work is >100KLOC of Ruby and ERB), even if you’re achieving an excellent speed / coverage ratio, eventually your tests will be too slow. We saw our full build reaching past the hour mark. We’ve got over 5,000 tests when you add our full stack tests and our unit specs.

    Because we want to be sure the app works all the time, and deploy to production regularly without a heavy manual testing process, we don’t want to sacrifice coverage for speed. So parallelizing the build is a good solution. In the future, I expect we’ll see better tooling in this area to help.

    All that said, great post. Slow tests are a big pet peeve of mine, and it deserves more discussion.

    Cheers,

    -Bryan

  9. Jon DahlMarch 11, 2009 @ 09:36 AM

    Thanks to everyone for the insightful comments.

    Bryan, testjour is a really impressive project, and I look forward to it running Test::Unit. (Is that in the works?)

    Bryan and Dan, you’re right that really big test suites are going to have unique challenges, and distributing tests is a lot more effective for huge test suites than incremental improvements. A 20% or even a 2x speedup isn’t going to make 5,000 tests run in 30 seconds, but distributing across an arbitrary number of machines might.

    On the other side, we’re an innovative community, and I see no reason why smart developers won’t be able to find ways to speed up tests by an order of magnitude or two.

    I think we need to address the problem from both ends. Speed up our test execution (distributing across multiple cores and multiple machines), and reduce total test suite time.

  10. Trevor TurkMarch 11, 2009 @ 02:54 PM

    Any techniques you can recommend for finding slow running tests would be greatly appreciated!

  11. Luke FranclMarch 11, 2009 @ 04:35 PM

    I always tell Jon that Rails developers don’t know how good they’ve got it. Two minutes is too slow? When I was working on a Java digital asset management system, our full test suite took over 30 minutes to run! You better believe we used continuous integration to run that baby.

    That said, anything you can do to speed up tests is a great benefit because of the reasons Jon has enumerated in this article.

  12. Tim ConnorMarch 11, 2009 @ 04:38 PM

    Trevor:

    I revamped Geoffrey Grosenbach’s test_benchmark plugin. It works as a rails plug-in or a ruby gem in general. Get it from the github home for test_benchmark and please fork or even just comment with what you would like to see in it.

  13. Jon DahlMarch 11, 2009 @ 07:56 PM

    Tim: thanks for the link. I installed and it worked well.

    I also use autotest, and it prints out these benchmark test times after each test is run. I don’t really want to see the benchmarks as I’m developing – they’re more useful, IMO, for occasional refactoring. But calling autotest like this silences the benchmarking:

    BENCHMARK=false autotest

  14. Mathias MeyerMarch 12, 2009 @ 04:12 AM

    I’m pretty much on board with what Luke said. The speed of Rails tests is still a luxury, and I altogether a big test suite will eventually get slower and slower. I’ve used to turn to mocks to speed it up, but that just feels wrong looking back. I’ve pretty much settled on trying to speed up tests as good as I can, but I’d rather have a decent test suite that takes a minute longer, but that gives me the assurance I need when working with the code.

    While I agree with the steps you can take to speed it up, I think the headline goes a little over board.

  15. Luke FranclMarch 12, 2009 @ 12:19 PM

    I forgot to add one thing: correctness is at all times preferable to speed.

    This is why I use mocks judiciously. It’s too easy to create a test that works perfectly with your mocks but doesn’t work in the real application.

    I’d rather have a slow test suite that is comprehensive and correct than a fast one that isn’t covering my code or is plain wrong. As I mentioned in my testing talk, “dirty” tests that check for correct responses to invalid data are especially important.

    But again, anything that can be done to speed up tests, while keeping them correct has my full support!

  16. Tim ConnorMarch 12, 2009 @ 07:21 PM

    Jon,

    It’d be trivial to do it that way, and probably preferable for the users of autotest. As my workplace was the only userbase so far, and I haven’t been using autotest for a while, I left it on by default to cram it down everyones throat. Please feel free to fork and modify as you’d like, or if I remember maybe I’ll flip the behavior.

  17. Tim ConnorMarch 12, 2009 @ 07:27 PM

    I guess my other thought was if you require it, you probably want to use it. I wonder if there is an easy way to check if you are being called from autotest and then require an explicit BENCHMARK=true

  18. bryanlMarch 16, 2009 @ 06:24 AM

    Good read. What we need is an agreed way to separate our test different test suites. (i actually think we have a good start on this). Just start out running your unit tests separate from your integration tests separate from your acceptance tests.

  19. Jon DahlMarch 17, 2009 @ 10:45 AM

    Bryan: we did something like that with our simple Shoulda model tests – should_belong_to :company, should_require_unique_attributes :name – which were really slow when we used factories to build our data. Just pulled them our of our unit tests and put them into a “secondary” test suite.

    Then we switched from factories to a test DB, and those tests sped up dramatically, so we’re thinking of pulling them back in.

    But moving slow tests (e.g. integration tests) to a separate layer can really help. They just need to be run frequently and not ignored. :)

    Tim: I found a simple solution – I put the following in test.rb

    ENV['BENCHMARK'] ||= 'none'

    Now the benchmarks don’t run unless I ask them to. :)

  20. Tim ConnorMarch 18, 2009 @ 03:57 PM

    Jon, yeah, I could have just coded it to not run by default. I guess users toggling wether it runs by default themselves might not be a bad idea (I could make a configurable, I suppose, but I stripped out some configuration for the sake of simplicity).

    If I ever need to hack it any more maybe I’ll add a TestBenchmark::run_by_default option. I still think it’d be interesting to see about making it not run in autotest, and stay on by default otherwise.

  21. Tim ConnorMarch 21, 2009 @ 09:17 PM

    Jon,

    It’s been forked, by mdm, and autotest exclusion added (and pulled back in). Try it out and let me know how it works for you.

  22. Toby MatejovskyMarch 27, 2009 @ 05:20 PM

    Autotest is the 80/20 solution for this problem of test speed. The amount of times I need to rerun the entire test suite (i.e. because routes.rb changed, like in your example), is rare enough that I can appreciate the 90 seconds break to look at lolcats or something.