I’m a big believer in unit testing. We unit test our Rails apps extensively, and we’ve done so for years. On some projects, we do both unit testing and integration testing using Cucumber. I preach unit testing to everyone I can. I’d probably turn down a project if the client wouldn’t let us write tests (though this has never come up, and I don’t think it would be a hard sell).

But for a long time, that’s all I did on my projects. Our clients and users would find the bugs that got past the developers. They were, in effect, our QA testers. (I think a lot of small/agile teams are the same way; based on my experience, I’d be surprised if more than 20% of Rails projects were comprehensively tested by a human.)

This is not right. A good QA tester is worth the surprisingly modest expense.

If I unit test, do I really need to hire a QA tester?

Keep on writing unit tests. But unit tests and human testing are two completely different things. They both aim to increase code quality and decrease bugs, but they do this in different ways.

Developer (unit) testing has three benefits. It:

• Makes refactoring possible. Don’t even try to refactor a large app without a test suite.

• Speeds up development. I know there are some haters who deny this, but they’ve either never really given unit testing a chance, or their experience has been 180º different than mine.

• Eliminates some bugs. Not all, but some.

Human testing has related, but somewhat different, benefits. It:

• Eliminates other bugs. Unit tests are great for certain categories of bugs, but not for others. When a human walks through an application with the express purpose of making things break, they’re going to find things that developer-written unit tests won’t find.

• Acts as a “practice run”. Before letting a client, boss, or user see a change, let a QA tester see it. You’d be surprised how many 500 errors and IE incompatibilities you can avoid.

• Gives you confidence before you deploy. After working with good QA testers, I can’t imagine deploying an app to production without having a QA tester walk through it.

• Saves you time. If you don’t have a QA role on your project, your developers will be defacto testers. They probably won’t do a good job at this, since they’ll be hoping things succeed (rather than making them fail). And their time is probably more expensive than a good tester’s time.

How to use a QA tester in an agile project

Agile testers should do four things.

First, they should verify or reject each story that is completed. Every time a developer indicates that a feature or bug is completed, whether you use a story tracker or index cards, a QA tester should verify this. Don’t deploy to production until the tester gives it a thumbs-up.

Second, they should do exploratory testing after every deploy. A few minutes clicking around in production can sniff out a lot of potential errors.

Third, they should test edge cases. What happens if a user types in a username that is 300 characters long? What they try to delete an item that is still processing? What if they upload a PDF file as an avatar? Testers are great at this sort of thing.

Fourth, they should test integrations. Unit tests can’t (and shouldn’t) test multi-step processes. Integration testing tools like Cucumber are OK, but don’t catch everything. Identify the main multi-step processes on your site, and have a human verify them every time they change.

Expect a tester to increase your development costs by 5%-10%. We find that 1 hour of testing for every 6 hours of developer time is a reasonable estimate. Our testers cost about 40% less than our developers. So on a typical invoice, testing services are about 10% of development services.

Bill separately for testing. Don’t just roll it into your developer rate. Clients are more likely to object to a 10% increase in your main hourly rate than a separate, lower testing line item.

Finding a good tester

There are two main ways to find a tester.

First, you can train one. Tech-savvy folks who aren’t programmers are a good option. They understand enough to fit in with your development process, but are happy testing and not coding. If you find the right person, they can be testing in no time, and won’t cost a ton of money.

Second, find one that understands agile development. There are plenty of professional testers out there, but most of them do waterfall testing: spend 3 weeks writing test cases, get release from developers, and spend 3 weeks testing. I can say, without hyperbole, that this is how exactly 0% of Rails development projects work. Look for the small number of testers that actually have experience with iterative development, flexible scope, and rapid turnaround. You can sometimes find these people at agile events (conferences or user groups). Otherwise, ask other developers. I found one via referral, and I’ve since referred him to others. This second category will probably be more expensive than the first, but if you want to ship the best code you can, go with this route. Just make sure you avoid a Zompire Dracularius.

If you ever need to test HTTP Authentication in your functional tests, here is how you do it:

1
2
3
4
5
6

def test_http_auth
  @request.env['HTTP_AUTHORIZATION'] = ActionController::HttpAuthentication::Basic.encode_credentials("quentin", "password")
  get :show, :id => @foobar.id

  assert_response :success
end

This is much like testing SSL.

Hat tip: Philipp Führer for Functional test for HTTP Basic Authentication in Rails 2.

I like to be extremely judicious with use of routes. Fewer routes means less memory consumption and fewer confusing magical methods.

I always delete the default route map.connect ':controller/:action/:id' (you should too, otherwise all your pretty RESTful routing is easily circumvented). Since Rails now has the ability to remove unneeded RESTful routes, I’ve been removing those, too.

However, this judiciousness recently painted me into a corner. I have a controller action that I would like to test and it’s wired up like this:

map.logout '/logout', :controller => 'user_sessions', :action => 'destroy', :method => 'delete'

I don’t have this mapped any other way, because why should I?

1
2
3
4
5
6
7
8

def test_logout_should_redirect_to_root_path
  UserSession.create(User.first)

  delete :destroy

  assert_match /logged out/, flash[:notice]
  assert_redirected_to root_path
end

Unfortunately, the test fails with ActionController::RoutingError: No route matches {:action=>"destroy", :controller=>"user_sessions"}! Huh?

The problem is that the delete (and get, post, etc.) method can’t find the route that I created.

Initially, I worked around this using with_routing to define a whole new set of routes just for that test.

1
2
3
4
5
6
7
8
9
10
11

with_routing do |set|
  set.draw do |map|
    map.resource :user_sessions, :only => [:destroy]
    map.root :controller => 'foobars', :action => 'index'
  end

  delete :destroy

  assert_match /logged out/, flash[:notice]
  assert_redirected_to root_path
end

But that was annoying. And after I had more than one route exhibiting this problem, it got really annoying.

Fortunately, I found Sam Ruby’s post Keeping Up With Rails about the challenge of Rails’ minor, quasi-documented API changes. Sam’s post has a bit about how you can add new routes without clearing the existing routes in Rails 2.3.2, which I knew was possible. Following Sam’s link to the commit (there’s no docs for this) showed how to do it.

Now, I’ve added this to test_helper.rb:

1
2
3
4

class ActionController::TestCase
  # add a catch-all route for the tests only.
 ActionController::Routing::Routes.draw { |map| map.connect ':controller/:action/:id' }
end

The downside to this is that real problems with broken routes may get swept under the rug. You could be more restrictive with the routes you are adding just for tests to overcome that problem.

Update: Thanks to Adam Cigánek in the comments for pointing out my error in why the route didn’t get picked up in the tests. I had the condition hash wrong!

Instead of:

map.logout '/logout', :controller => 'user_sessions', :action => 'destroy', :method => 'delete'

It should be:

map.logout '/logout', :controller => 'user_sessions', :action => 'destroy', :conditions => {:method => :delete}

The first way I had worked correctly when testing manually, but only because without :method, the route responds to all HTTP methods (still no clue why my test didn’t pick it up, though).

Interestingly enough, there’s another gotcha here. Notice that I specified :method => 'delete'. Even when put into the :conditions hash, that doesn’t work. You MUST pass a symbol (:delete) for the HTTP method.

This fixed my problem, but if I ever do need to add routes for tests, now I know how…

Update: stubbing a single integration point shaved 22 seconds off of my unit tests, reducing test time from 35 seconds to 13. See below.

The first step to faster tests is knowing what is slow. Fortunately, this is dead simple with the test_benchmark plugin by Tim Connor, and originally built by Geoffrey Groschenbach. Install the plugin, and when you run your tests via Rake, you’ll see handy output showing you the slowest tests, and the slowest test classes.

Step 1: Install the plugin.

script/plugin install git://github.com/timocratic/test_benchmark.git

Step 2: Run your tests

rake test

Here is a bit of output when I run the unit tests for FanChatter:

Finished in 34.838173 seconds.

Test Benchmark Times: Suite Totals:
25.393 MailReceiverTest
4.520 PhotoTest
1.429 REXMLTest
0.961 TeamTest
0.846 MessageTest

Pretty useful information. Almost 75% of our unit testing time is taken up in the MailReceiverTest. So if we want to speed up our tests, we need to make our MMS testing faster. Looking at that code, I see this line over and over:

MailReceiver.receive(fixture_mms(:fixture_name))

This method reads a test email message from the filesystem, and runs it through our mail parsing method. This is basically an integration test, hitting at least two integration points. So if we can remove these bottlenecks, we can reasonably expect a fairly large improvement in our unit test speed.

I think we could realistically reduce our unit testing time from 34 seconds to <15 seconds just by refactoring this one test method.

Other options

The test_benchmark plugin fires whenever you run your tests with rake. Tim recently patched the plugin to not fire when run with autotest, which is great. Personally, though, I don’t want to see this benchmark information every time I run my tests. So I added the following line to my test.rb environment file:

ENV['BENCHMARK'] ||= 'none'

Now, the benchmarks don’t run by default. If I want to see them, I call:

rake test BENCHMARK=true

And if to see full tests, showing the time it takes to run every test in the system, just call:

rake test BENCHMARK=full

That’s it. You still have to speed up your tests, and there are many ways to do that (from mocking to simply reducing the number of calls to expensive methods), but knowing what’s slow is half the battle.

The stirring conclusion (update)

I spent a few minutes optimizing these slow tests today. First, I tried rearranging the tests to reduce unnecessary calls to the slow method (MailReceiver.receive(message)). I was able to speed MailReceiverTest from about 25 seconds to 17. Not bad, but still slow.

The real problem is that this method saves a photo. It creates a Photo record that includes a file, treated sort of like an upload, like this:

1

photo.uploaded_data = mms.file

This is what was slow. But my unit tests don’t actually deal with the file being saved to the filesystem; they test other things, like the right records being created, confirmation emails being sent, etc.

So I decided to try bypassing this file save/upload by stubbing the uploaded_data= method. I put the following at the top of my test class:

1
2
3

def setup
    Photo.any_instance.stubs(:uploaded_data=)
  end

And voila! MailReciverTest went from 25 seconds to 17 seconds to 3 seconds.

I’ve been doing TDD for about three years now. Once I figured out how to do it right, it became a natural part of how I program, and I can’t really imagine doing development without it. This isn’t to say that TDD is the only approach to writing quality software or that unit testing it the only kind of testing that matters. But it sure is useful.

The Ruby world talks a lot about TDD, moreso than many other developer communities. We have not one, not two, but at least half a dozen testing libraries that are actively being used and developed. For most Ruby developers, the question isn’t “Do you test?” but “BDD or TDD?” or even “RSpec, Shoulda, or Bacon?” We often use at least 2-3 layers of automated testing, and sometimes use different tools for each layer. Most Ruby conferences devote at least a few talks each day to testing-related topics. We’re test fanboys and -girls, for better or for worse.

But in spite of this, we rarely talk about test speed. Sure, there are purists who believe that unit tests shouldn’t touch the database because anything that touches the DB is actually an integration test. But few Ruby testers actually take this long and lonely road, and I personally prefer tests that talk to a database, at least some of the time.

And it’s true that others have written libraries to distribute their tests across multiple machines. But that’s the exception that proves the rule – the only reason to distribute your tests is that they’re too slow to begin with.

Most Rails projects I’ve worked on have ended up at around 3,000-15,000 lines of code, with a roughly as many lines of test code, and most have test suites that take a minute or more to run. Our test suite for Tumblon, for instance, churns along for 2.5 minutes. This is a too slow. And slow tests are a problem for at least two reasons: they slow down your development and decrease code quality.

1. Slow tests slow down development. If you’re practicing TDD, you want to see a test fail before you make it succeed. Two minutes is far too long for this feedback loop to be effective. Of course, you can (and should) just run the test classes that correspond to your code as you program – no need to run your entire test suite every time you write your failing tests. But even still, the test time bar should ideally be set quite low. Frequent 5-10 second delays are enough to break my concentration, and I find myself cmd-tabbing over to other programs if I have to wait more than a few seconds for a test to run. I don’t know of any hard-and-fast rules, but I know that as soon as my test suite runs longer than 30-45 seconds, and individual test classes take longer than 2-3 seconds, I’m less happy and less productive.

2. Slow tests decrease code quality. There are two simple reasons for this. First, if slow tests break your flow, you’re not only going to write code more slowly: you’re also going to write worse code. Second, if your tests are too slow, you’re not going to wait for them to finish before you move on to the next task. Or worse, you’re not going to run them at all.

So, how can I speed up my tests?

Fortunately, this problem can be addressed. There are plenty of ways to speed up tests. On a current project, we’ve managed to cut our test time substantially – a recent test refactoring cut test time from 129.45 seconds to 31.04 seconds, without removing any tests. That’s a 76% speedup. But we still have room for improvement.

Really quickly, here are at least five ways to speed up your test suite. I hope to post more on each of these over the next month or two.

1. Use a test database instead of fixtures/factories/etc.

2. Only touch the database when necessary

3. Organize your tests to avoid duplicate execution

4. Separate slow tests out into a lazier testing layer

5. Run a Rails test server

I’d love to see the Rails community devote more of its enthusiasm for testing to the question of test speed. There’s nothing wrong with improving our test frameworks, and let’s keep doing that. But let’s also make these frameworks fast.

I’ve been following the recent blow up on the topic of test driven development with interest (perhaps a vested interest).

Basically what happened is Joel Spolsky said requiring 100% test coverage was doctrinaire and maybe mangled some testing gurus’ ideas a little bit. Which made them mad and there was this big harrumph about testing on blogs, Hacker News, and elsewhere.

Other people jumped into the discussion. Jay Fields offered a very well thought out post with a key point: if it hurts, you’re doing it wrong. Paul Buchheit said “unit tests are 20% useful engineering, and 80% fad”; and then Bill Moorier, the VP of Justin.TV, posted a followup to that discussing his dislike of unit tests.

Some people are trying to rip the test skeptics a new one. Giles Bowkett says it’s proof that the mainstream will never catch up, because they’re so totally clueless.

I find this all bemusing.

Here’s the thing.

Joel Spolsky created the original version of FogBugz, a pretty sweet piece of software. Paul Buchheit created GMail, a product that millions of people use and love every day. Justin.TV is also a very innovative product that’s popularized a new form of expression. Justin.TV is so “mainstream” that a guy committed suicide on it.

The testing advocates have a similarly impressive list of great software.

Maybe there’s more than one way to develop good software?

I tried to make this point explicitly in my Testing is Overrated talk: one of the reasons why test-driven design works is that it makes you think deeply about the problem you’re trying to solve. But it’s not the only way to think deeply about a problem! And it’s certainly not a guarantee for building successful software.

Here’s a quick tip for how to test that your application is using SSL correctly.

Enabling SSL in tests

You can turn SSL on in functional tests like this:

@request.env['HTTPS'] = 'on'

I have this turned on in my setup method and then override it for tests that don’t use SSL. To turn SSL off, use this:

@request.env['HTTPS'] = nil

(Via snippets)

Testing for SSL redirect

To test and see if you users will get redirected to SSL for particular actions, you can write a test like this. First, it turns off SSL. Then it makes a request to a method that should require SSL, and asserts that the request is redirected to the same URL, but using the https protocol.

1
2
3
4
5
6

def test_get_new_with_http_should_redirect_to_ssl
  @request.env['HTTPS'] = nil
  get :new
  assert_redirected_to "https://" + @request.host + @request.request_uri
end

Implementing the code

Having done this, you probably have a bunch of failing tests. To make them pass, get the SslRequirement plugin (I’m actually using Doug Johnson’s fork that adds support for different SSL domains, like secure.example.com).

Include SslRequirement in application.rb, and then in your secure controllers, add a line like this:

ssl_required :new, :create

Then run your tests.

Now the thing about this is that it is also going to try to send you to an HTTPS URL in development mode. Since most people don’t actually have SSL set up on their development environment, it makes sense to disable this in development.

ssl_required :new, :create if RAILS_ENV 'production' || RAILS_ENV ‘test’

Or for you fancy-pants Rails 2.1 types:

ssl_required :new, :create if Rails.env.production? || Rails.env.test?

(Yeah, you could also use unless Rails.env.development?, but I actually have a couple other environments configured that I do not want to enable SSL for.)

Photo by Darwin Bell.

Next week at RubyFringe, I’ll be taking on one of the programming world’s favorite topics: testing.

Hear me out. Like everyone who’s had their bacon saved by a unit test, I think testing is great. In a dynamic language like Ruby, tests are especially important to give us the confidence our code works. And once written, unit tests provide a regression framework that helps catch future errors.

However, testing is over-emphasized. If our goal is high-quality software, developer testing is not enough.

This is important because of what Steve McConnell calls The General Principle of Software Quality. Most development time is spent debugging. “Therefore, the most obvious method of shortening a development schedule is to improve the quality of the product.” (Code Complete 2, p. 474.)

Problems with developer testing

Developer testing has some limitations. Here are a few that I’ve noticed.

Testing is hard...and most developers aren’t very good at it!

Programmers tend write “clean” tests that verify the code works, not “dirty” tests that test error conditions. Steve McConnell reports, “Immature testing organizations tend to have about five clean tests for every dirty test. Mature testing organizations tend to have five dirty tests for every clean test. This ratio is not reversed by reducing the clean tests; it’s done by creating 25 times as many dirty tests.” (Code Complete 2, p. 504)

You can’t test code that isn’t there

Robert L. Glass discusses this several times in his book Facts and Fallacies of Software Engineering. Missing requirements are the hardest errors to correct, because often times only the customer can detect them. Unit tests with total code coverage (and even code inspections) can easily fail to detect missing code. Therefore, these errors can slip into production (or your iteration release).

Tests alone won’t solve this problem, but I have found that writing tests is often a good way to suss out missing requirements.

Tests are just as likely to contain bugs

Numerous studies have found that test cases are as likely to have errors as the code they’re testing (see Code Complete 2, p. 522).

So who tests the tests? Only review of the tests can find deficiencies in the tests themselves.

Developer testing isn’t very effective at finding defects

To cap it all off, developer testing isn’t all that effective at finding defects.

Don’t put all your eggs in one basket

The most interesting thing about these defect detection techniques is that they tend to find different errors. Unit testing finds certain errors; manual testing others; usability testing and code reviews still others.

Manual testing

As mentioned above, programmers tend to test the “clean” path through their code. A human tester can quickly make mincemeat of the developer’s fairy world.

Good QA testers are worth their weight in gold. I once worked with a guy who was incredibly skilled at finding the most obscure bugs. He could describe exactly how to replicate the problem, and he would dig into the log files for a better error report, and to get an indication of the location of the defect.

Joel Spolsky wrote a great article on the Top Five (Wrong) Reasons You Don’t Have Testers—and why you shouldn’t put developers on this task. We’re just not that good at it.

Code reviews

Code reviews and formal code inspections are incredibly effective at finding defects (studies show they are more effective at finding defects than developer testing, and cheaper too), and the peer pressure of knowing your code will be scrutinized helps ensure higher quality right off the bat.

I still remember my first code review. I was doing the ArsDigita Boot Camp which was a 2-week course on building web applications. At the end of the first week, we had to walk through our code in front of the group and face questions from the instructor. It was incredibly nerve-wracking! But I worked hard to make the code as good as I could.

This stresses the importance of what Robert L. Glass calls the “sociological aspects” of peer review. Reviewing code is a delicate activity. Remember to review the code…not the author.

Usability tests

Another huge problem with developer tests is that they won’t tell you if your software sucks. You can have 1500% test coverage and no known defects and your software can still be an unusable mess.

Jeff Atwood calls this the ultimate unit test failure:

I often get frustrated with the depth of our obsession over things like code coverage. Unit testing and code coverage are good things. But perfectly executed code coverage doesn’t mean users will use your program. Or that it’s even worth using in the first place. When users can’t figure out how to use your app, when users pass over your app in favor of something easier or simpler to use, that’s the ultimate unit test failure. That’s the problem you should be trying to solve.

Fortunately, usability tests are easy and cheap to run. Don’t Make Me Think is your Bible here (the chapters about usability testing are available online). For Tumblon, we’ve been conducting usability tests with screen recording software that costs $20. The problems we’ve found with usability tests have been amazing. It punctures your ego, while at the same time giving you the motivation to fix the problems.

Why testing works

Unit testing forces us to think about our code. Michael Feathers gets at this in his post The Flawed Theory Behind Unit Testing:

One very common theory about unit testing is that quality comes from removing the errors that your tests catch. Superficially, this makes sense….It’s a nice theory, but it’s wrong….

In the software industry, we’ve been chasing quality for years. The interesting thing is there are a number of things that work. Design by Contract works. Test Driven Development works. So do Clean Room, code inspections and the use of higher-level languages.

All of these techniques have been shown to increase quality. And, if we look closely we can see why: all of them force us to reflect on our code.

That’s the magic, and it’s why unit testing works also. When you write unit tests, TDD-style or after your development, you scrutinize, you think, and often you prevent problems without even encountering a test failure.

So: adapt practices that make you think about your code; and supplement them with other defect detection techniques.

Testing testing testing

Why do we developers read, hear, and write so much about (developer) testing?

I think it’s because it’s something that we can control. Most programmers can’t hire a QA person or conduct even a $50 usability test. And perhaps most places don’t have a culture of code reviews. But they can write tests. Unit tests! Specs! Mocks! Stubs! Integration tests! Fuzz tests!

But the truth is, no single technique is effective at detecting all defects. We need manual testing, peer reviews, usability testing and developer testing (and that’s just the start) if we want to produce high-quality software.

Resources

• Robert L. Glass, Facts and Fallacies of Software Engineering.
• Steve McConnell, Code Complete 2nd ed, Chapters 20-22.
• Steve Krug, Don’t Make Me Think.

I gave a presentation at RUM on Monday about code metrics. In particular, I showed tools for measuring two aspects of code: test coverage and complexity. Here are my slides.

Saikuro and Flog measure code complexity. Saikuro measures cyclomatic complexity, the number of independent paths through a method. Flog, on the other hand, parses your code and assigns a complexity value to assignments, branches, and calls. The goal, of course, is to minimize code complexity. This is an important goal, but I’m not sure yet what I think of these measurement tools. I haven’t used them enough to know if they have practical value.

Heckle and RCov on the other hand, are useful. I’m going to look at each in more detail here.

RCov

RCov measures C0 code coverage. That is, it runs your test suite, and looks at what lines of your application were run or not run. It then gives you a nice HTML report with red and green lines – red for lines of code that are not run, and green for lines that are run.

If your test suite doesn’t execute a line of your application code, it is safe to say that that line is not tested. On the other hand, if a line of your application is run, it is NOT safe to say that it IS tested. A test method with no asserts works just fine for RCov’s purposes, thank you very much. Take a look at this code.

def test_user_assignment
  User.assign
end

This test is enough to mark the User.assign method as tested. But nothing is asserted, and so nothing is tested. The problem is equally true even if you aren’t in the habit of writing tests without assertions; you may make assertions about some aspects of a method, but forget about other aspects. And RCov won’t tell you this.

Logically speaking, RCov tells you that if line_is_red, then !line_is_tested. From this, you can also infer the contrapositive: if line_is_tested, then !line_is_red. But that’s all you know. If a line is green, RCov tells you nothing at all. Saying if !line_is_red, then line_is_tested is a formal fallacy (denying the antecedent). And that’s bad.

So 100% RCov coverage is not equal to 100% test coverage. In fact, the two have nothing to do with each other. Your code could have 100% or 95% or 75% RCov coverage, and be extremely poorly tested.

In my experience, RCov is a one-time tool. That’s because green lines in RCov don’t tell you anything at all about your test coverage. Red lines provide the real value. If you run RCov, find an untested method, and write up a quick test hack that provides C0 coverage, RCov will never complain about that method again. It will be off your RCov radar. This is too bad, because it is really useful to know what is poorly tested. So whenever you see red in RCov, take the time to write comprehensive tests to cover the untested code.

Heckle

Heckle is a mutation tester that changes your code and checks to see whether your tests catch the changes. If Heckle is able to change instances of true to false (or 32 to nil, or remove method calls) in your application without creating a test failure, then your code isn’t tested well enough. To run it effectively, do this:

heckle Class method -t /test/units/class_test.rb -T 30

heckle is the tool, installed as a Ruby gem. Class is the name of the Ruby class you want to heckle. method is a method on the class; you can leave this out, but I don’t recommend it. -t /test/units/class_test.rb is the path to the unit test you want to use (also optional). Finally, -T 30 specifies a timeout for the test, in case your mutation creates an infinite loop.

You can leave out the last three options and just run Heckle with a class:

heckle Class

But I don’t recommend it.

First, it will take forever.

Second, you may run into infinite loops.

Third, heckle will unfortunately test EVERY method available to a class, including methods included by modules, superclasses, etc. So if you’re heckling an ActiveRecord class, you’re going to see dozens of Rails magic methods, not just the methods that you wrote.

Fourth, your UserTest should cover your User class on its own, if your code is well written and well tested; it shouldn’t rely on the ProductTest class (or another test). One problem with Heckle is that it doesn’t distinguish between well tested code and highly coupled code, where a small change somewhere causes the application to fall apart somewhere else. This problem can be minimized by only comparing a single method to a single test class.

I like Heckle and find it pretty useful. Unfortunately, it needs a little developer love. The -T timeout parameter is flaky; it doesn’t always play nice with its dependencies (especially ParseTree 2.0.x, the current version); and it would be more useful if by default it only heckled the methods directly added by a class, not methods brought in through parent classes, includes, or fancy metaprogramming. This is a shame, because it is really a great tool. Hopefully Kevin Clark and Ryan Davis have an update in the works.

ActiveRecord allows you to start transactions that will be rolled back in the event of an error.

A good example is importing records from a CSV file. If you want the entire import to roll back if any of the rows fail to import, you could write your code like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

def import_csv
  csv_file = params[:csv_file]
  
  begin
    Record.transaction do 
      fastercsv = FasterCSV.new( csv_file )
      while row = fastercsv.readline
        foo, bar = row
        Record.create!( :foo => foo, :bar => bar )
      end
    end
    redirect_to success_action_path
  rescue 
    # do something with the error
    flash[:error] = "CSV import failed"
    redirect_to import_path
  end
end

The Record.create! call will throw an ActiveRecord::InvalidRecord error if one of the rows can’t be saved. Then the rescue block catches the error and reports it to the user instead of showing them an ugly 500 error (or, worse, a corrupted import).

However, this doesn’t play nicely with your tests.

You’d like to do something like this:

1
2
3
4
5

def test_import_csv_failure
  assert_no_difference Record :count do 
    post :import_csv, :csv_file => fixture_file_upload('files/invalid.csv')
  end
end

But this won’t work, because running the test starts a transaction, and ActiveRecord doesn’t support nested transactions. There’s been a patch open on this problem for 9 months, but no action has been taken.

I was able to work around the problem by turning off transactional fixtures for the entire test case class.

1
2
3
4
5
6
7
8
9

class MyTest < Test::Unit::TestCase
  self.use_transactional_fixtures = false

  def test_import_csv_failure
    assert_no_difference Record :count do 
      post :import_csv, :csv_file => fixture_file_upload('files/invalid.csv')
    end
  end
end

This makes the test run slower, but now it passes. If you’re feeling adventurous, you can install the ActiveRecord nested transactions plugin.

Lots of people have hit this problem. Jerry Kuch blogged about it in January 2006 and ticket 5457 was filed back in June. But hopefully this post will help someone else figure out the problem.