A blog about the joys and perils of software development

Measure Your Code Un-coverage

Maximizing code coverage is not the way to maximize the benefits of unit testing. Instead (1) identify the most important user scenarios and (2) measure and analyze the code “un-coverage”.

Why do we unit test? There’s a multitude of reasons. We want to feel confident our code works and we want fast feedback. Other benefits are related more to the structure of the code. Good tests imply good testability which in turn implies proper decoupling, user-friendly APIs etc. But what do we unit test?

Code coverage is a metric often tied to unit testing. Code coverage can mean many things. The most commonly used metric is “line coverage”, which compares the code lines exercised by all your tests with the total number of lines (normally expressed in ELoCs, effective lines of code = code lines excluding blank lines, comments and curly braces). All coverage metrics work like this, comparing something (e.g. ELoCs) with the total number. Other coverage metrics are “function coverage”, “branch coverage” and “path coverage”.

Having a 60% line coverage means your test execution has touched 60% of your code base. It means close to nothing, however, in terms of quality assurance. You could hit 60% of the code lines and still miss the majority of your most important user scenarios. On the flip side, having a 60% line coverage also means 40% of the code lines have never been executed. Now this is useful information. If a line of code has never been executed, there is a chance the code cannot even be executed without crashing. Or formatting the hard drive, you never know.

So instead of focusing on covering your code with tests, follow this procedure:

  1. Identify the most important user scenarios and alternate paths and implement them as unit tests (or tests on other levels if more appropriate, e.g. component, integration or system level).
  2. Measure your code “un-coverage” and observe which parts of the code are never touched by your test cases. Come up with user scenarios that will exercise the un-covered code. Write tests.

You might still have uncovered code after this procedure. Obtaining 0% code un-coverage (100% code coverage) is expensive. All testing is a trade-off between the risk of releasing something broken and the cost of testing it. When you have covered what is important to your end-user and analyzed the remaining un-covered code, functionality-wise your tests are in great shape even if your coverage is nowhere near 100%.

As a side note: In the case of test-driven development, we will end up with close to 0% un-covered code. But with TDD, bullet (1) is more important than ever. It is easy to get carried away focusing on covering each line of production code with tests and lose focus on what’s important: testing the right thing, i.e. the things that are most important for your end-users. Happy testing.

This entry was posted in Software Development and tagged , , , , . Bookmark the permalink.

3 Responses to Measure Your Code Un-coverage

  1. Ulf Wiger says:

    A dilemma when working with more and more expressive languages, is that a line being covered doesn’t necessarily mean that that part of the program is correct.

    Here’s an example I used once when trying to explain the dangers of trusting too much in code coverage (fair warning: it’s Erlang code):


    -module(ex).
    -export([factorial/1]).

    factorial(N) ->
    N * factorial(N-1);
    factorial(0) ->
    1.

    Running ex:factorial(2), we get 100% code coverage, but the program is still buggy. Why? Since it doesn’t terminate for ex:factorial(-1). This program is in fact, (bugs and all) one of the first that people encounter when learning Erlang from scratch.

    On this particular occasion, I was arguing in favor of introducing random testing techniques, such as Quviq QuickCheck. I’m not sure how successful I was… ;-)

Leave a Reply

Your email address will not be published.