Let a human test your app, not (just) unit tests

Posted by Jon
on Thursday, October 29

I’m a big believer in unit testing. We unit test our Rails apps extensively, and we’ve done so for years. On some projects, we do both unit testing and integration testing using Cucumber. I preach unit testing to everyone I can. I’d probably turn down a project if the client wouldn’t let us write tests (though this has never come up, and I don’t think it would be a hard sell).

But for a long time, that’s all I did on my projects. Our clients and users would find the bugs that got past the developers. They were, in effect, our QA testers. (I think a lot of small/agile teams are the same way; based on my experience, I’d be surprised if more than 20% of Rails projects were comprehensively tested by a human.)

This is not right. A good QA tester is worth the surprisingly modest expense.

If I unit test, do I really need to hire a QA tester?

Keep on writing unit tests. But unit tests and human testing are two completely different things. They both aim to increase code quality and decrease bugs, but they do this in different ways.

Developer (unit) testing has three benefits. It:

  • Makes refactoring possible. Don’t even try to refactor a large app without a test suite.
  • Speeds up development. I know there are some haters who deny this, but they’ve either never really given unit testing a chance, or their experience has been 180ยบ different than mine.
  • Eliminates some bugs. Not all, but some.

Human testing has related, but somewhat different, benefits. It:

  • Eliminates other bugs. Unit tests are great for certain categories of bugs, but not for others. When a human walks through an application with the express purpose of making things break, they’re going to find things that developer-written unit tests won’t find.
  • Acts as a “practice run”. Before letting a client, boss, or user see a change, let a QA tester see it. You’d be surprised how many 500 errors and IE incompatibilities you can avoid.
  • Gives you confidence before you deploy. After working with good QA testers, I can’t imagine deploying an app to production without having a QA tester walk through it.
  • Saves you time. If you don’t have a QA role on your project, your developers will be defacto testers. They probably won’t do a good job at this, since they’ll be hoping things succeed (rather than making them fail). And their time is probably more expensive than a good tester’s time.

How to use a QA tester in an agile project

Agile testers should do four things.

First, they should verify or reject each story that is completed. Every time a developer indicates that a feature or bug is completed, whether you use a story tracker or index cards, a QA tester should verify this. Don’t deploy to production until the tester gives it a thumbs-up.

Second, they should do exploratory testing after every deploy. A few minutes clicking around in production can sniff out a lot of potential errors.

Third, they should test edge cases. What happens if a user types in a username that is 300 characters long? What they try to delete an item that is still processing? What if they upload a PDF file as an avatar? Testers are great at this sort of thing.

Fourth, they should test integrations. Unit tests can’t (and shouldn’t) test multi-step processes. Integration testing tools like Cucumber are OK, but don’t catch everything. Identify the main multi-step processes on your site, and have a human verify them every time they change.

Expect a tester to increase your development costs by 5%-10%. We find that 1 hour of testing for every 6 hours of developer time is a reasonable estimate. Our testers cost about 40% less than our developers. So on a typical invoice, testing services are about 10% of development services.

Bill separately for testing. Don’t just roll it into your developer rate. Clients are more likely to object to a 10% increase in your main hourly rate than a separate, lower testing line item.

Finding a good tester

There are two main ways to find a tester.

First, you can train one. Tech-savvy folks who aren’t programmers are a good option. They understand enough to fit in with your development process, but are happy testing and not coding. If you find the right person, they can be testing in no time, and won’t cost a ton of money.

Second, find one that understands agile development. There are plenty of professional testers out there, but most of them do waterfall testing: spend 3 weeks writing test cases, get release from developers, and spend 3 weeks testing. I can say, without hyperbole, that this is how exactly 0% of Rails development projects work. Look for the small number of testers that actually have experience with iterative development, flexible scope, and rapid turnaround. You can sometimes find these people at agile events (conferences or user groups). Otherwise, ask other developers. I found one via referral, and I’ve since referred him to others. This second category will probably be more expensive than the first, but if you want to ship the best code you can, go with this route. Just make sure you avoid a Zompire Dracularius.


Leave a response

  1. Kurt WerleOctober 29, 2009 @ 04:07 PM

    “that got passed the developers” -> “that got past the developers”

    Just a pet peeve…

    Oh, and I totally agree about testers. Another great thing for projects that are not the only one in an organization is integration testing between your rails project and other applications that it interacts with. Your system may work, and theirs may work, but your tester will find where they fail together.

  2. Jon DahlOctober 29, 2009 @ 04:21 PM

    Kurt: Human testers are definitely useful in that situation. Assuming you’re on a project like this right now, have you tried also using automated tests via continuous integration? Both apps, ideally, could trigger a test suite that exercises the API.

    (Typo corrected. Thanks!)

  3. DannyOctober 30, 2009 @ 05:02 PM

    This is a great idea and is one too few developers acknowledge. Sure, the users are the ultimate testers because they have needs the application is supposed to provide, but it’s not fair to offload the bulk of the usability testing on them. But, as you mentioned, testers are valuable resources who cost time and money, and many project managers think there is never enough of either.

    We’ve tried to minimize the barriers of usability testing with openhallway.com . We allow a user to create a testing scenario and send a link (via email, IM, etc.) to a tester anywhere in the world, and we then record the tester’s screen and microphone as she follows the scenario instructions. The results are uploaded and the test creator gets that feedback: easy, cheap, and quick. This process fits perfectly into the agile process, allowing user feedback in an iterative manner.

  4. Luke FranclOctober 31, 2009 @ 02:01 PM

    Hey, I think this sounds familiar. :)