Why Design By Contract Does Not Replace a Test Suite

"Design By Contract" (DBC) usually refers both to very sophisticated assertion systems (for example in which assertions are inherited along with the methods to which they belong), and to the practice of using such assertions extensively, not only for quality assurance but also as a form of documentation.

When I was mostly programming in Eiffel some years ago, I liked DBC very much, and I still think that it's a very good idea, and that more programming language should offer good support for it.

However there's one comment that I've seen frequently on the web, in blogs and on IRC. Often DBC evangelists say something along these lines: "We have DBC, we don't need a test suite". I find such comments incredibly stupid, and here I want to write down why.

Code needs to run

If you want to verify that the code does what you want, you have to actually run it - otherwise the assertions won't be triggered, and are worthless as a verification tool.

You don't have to just run it, but should, when possible, cover every code path - just like you'd do it when you write tests. Doing that manually requires much work, so you still need a test suite that you can run to verify that some changes didn't break anything.

Examples are easy, general rules are hard

Test cases are just example input, paired with the expected output. Usually it's pretty easy to come with examples, so writing tests is also easy, even for corner cases.

On the other hand assertions are rules that have to hold for all possible input data, so to formulate them, you have to consider the general case - that's usually rather hard, so the lazy programmer leaves out the hard cases.

A simple example: suppose you've written a subroutine that adds two numbers (for example for a bignum library). Writing assertions for the general case of addition is quite hard if you can't trust your subtraction routine; so the only things you can really do is to check the signs (positive number plus positive number is positive etc.), but that won't catch any off-by-one errors.

So you should also write tests; tests like add(3, 4) == 7 are trivial to come up with, and catch potential errors.

Conclusions

Design by Contract and testing should go hand in hand so that the tests exercise as many code paths as possible, and should cover those areas that are hard to validate with assertions.

DBC should not be viewed as a replacement for tests.