What is a Coverage Tool?

A coverage tool is a tool used when testing to determine exactly how much of the application you have tested. A good tool will also give you specific examples of what hasn't been tested. Coverage of a program is reported on the basis of metrics.

You can think of coverage in terms of a graph with vertices and edges. The vertices represent statements that can be executed, and the edges represent flow control within the program. There are three classes of coverage that can be divined from this model:

  • Statement Coverage - The percentage of vertices in the graph which have been traversed by a test.
  • Branch Coverage - The percentage of edges in the graph which have been traversed by a test.
  • Path Coverage - The percentage of all execution paths taken by a test.

There are other types of coverage which can be measured. These are primarily focused around where these metrics cannot easily reach. The two which come to mind first are:

  • Relational Coverage - Are all variations of <, = and > covered?
  • Loop Coverage - Have you skipped a loop, executed a loop exactly once and executed a loop more than once?

Coverage tools provide assurance that Unit tests and Integration tests are indeed exercising the class adequately. This information is vital to determine if enough testing is being done.

Coverage also provides valuable insight into the risk of software failure. The more a piece of code is executed under test, the less likely that a software failure will show up at the last minute, or even be found by a customer.

What is Quilt?

Quilt is a coverage analysis tool which operates on Java bytecode. It intercepts the bytecode before it is loaded into the JVM and instruments the class. The instruments in the class are then queried after the tests are run, and coverage is calculated. In the future, Quilt should also be able to provide insight into what tests can be added to improve coverage.

There are three ways of building a coverage tool, each with their own ups and downs. The three ways you can do it are:

  • Source Code Instrumentation - Change the source code before the software is compiled.
  • Byte Code Instrumentation - Change the binary before it is run.
  • Profiler Monitoring - Monitor a profiler, and report based on its results.
I pick byte code instrumentation for Quilt because of the ease in which it can be used. To use Quilt, you just load up the classes to be tested under a QuiltClassLoader, and they are instantly capable of being watched by Quilt. Thus if your code can be tested, it can be checked for coverage. If I used Source Instrumentation, a seperate compile phase would be necessary for the coverage.

Why Free?

There are a couple of reasons why Quilt is Open Source. First, its because Coverage tool providers charge an arm and a leg for their tools. I wanted to experiment with Code Coverage, and at the time I couldn't buy a tool for less that $500.

And of course, the idea of intercepting bytecode before it hits the JVM, and adding code to it is kind of fun. Why wouldn't anyone want to do it?