EzUnit — A New View on Unit Testing in Eclipse

New Version: EzUnit4

Purpose

a framework to link test failures to tested methods, and to report failures as problems analogous to compiler-generated errors and warnings

System requirements

A hands-on example

To show how EzUnit works we resort to the standard example of JUnit — class Money and test class MoneyTest. We use JUnit 4 and annotate test classes accordingly, basically because EzUnit also relies on annotations. EzUnit works with JUnit 3.8 also, but currently requires an @Test annotation to explicitly mark test methods (the materialization of test cases in JUnit).

Back to the example. Here is the code:

public class Money {
    private int fAmount;
    private String fCurrency;
    public Money(int amount, String currency) {
        fAmount= amount;
        fCurrency= currency;
    }
 
    public int amount() {
        return fAmount;
    }
   
    public Money add(Money m) {
        return new Money(amount()+m.amount(), currency());
    }
 
    public boolean equals(Object anObject) {
        if (anObject instanceof Money) {
            Money aMoney= (Money)anObject;
            return aMoney.currency().equals(currency())
                && amount() == aMoney.amount();
        }
        return false;
    }
 
    public String currency() {
        return fCurrency;
    }
}

and

import static org.junit.Assert.*;
import org.junit.Before;
import org.junit.Test;
 
public class MoneyTest {
    private Money f12CHF;
    private Money f14CHF;  
 
    @Before
    public void setUp() throws Exception {
        f12CHF= new Money(12, "CHF");
        f14CHF= new Money(14, "CHF");
    }
 
    @Test
    public void testSimpleAdd() {
        Money expected= new Money(26, "CHF");
        Money result= f12CHF.add(f14CHF);
        assertTrue(expected.equals(result));
    }
 
    @Test
    public void testEquals() {
        assertTrue(!f12CHF.equals(null));
        assertEquals(f12CHF, f12CHF);
        assertEquals(f12CHF, new Money(12, "CHF"));
        assertTrue(!f12CHF.equals(f14CHF));
    }
}

You can download an Eclipse project containing just the two classes here.

In unit testing jargon, Money is what is called a Class Under Test (CUT). We add to this notion that of a Method Under Test (MUT). The idea is that test cases call one or more methods of one or more CUTs, but not all of the called methods are necessarily test candidates, that is, are actually tested. Unfortunately, for EzUnit there is no way to tell which methods are, so you will have to name them.

For this, you have to add annotations to test cases that link them to the methods they test. Since this annotating is tedious, you don't have to do it manually — instead, you can select them from a list compiled from the call graph of the test method (test case) and have the annotations generated for you. Here is how it works:

  1. Add the @MUT annotation type to your project. For this, you have to enter the Properties page of your project and add "MUT" to the project's libraries. (A project's libraries are listed under Java Build Path > Libraries. Use the Add Library button and select MUT; if you haven't already done it, use it again to add JUnit 4).

  1. Right click on the test method you want to annotate (to which you want to assign its method(s) under test) and select "Select MUTs".

  1. In the following dialog

select "Money.add(QMoney;)" and confirm.

EzUnit then adds a corresponding annotation, as in

      @org.projectory.ezunit.MUT (
            methods = {
                  "Money.add(QMoney;)"
            }
      )
      @Test
      public void testSimpleAdd() {
            Money expected = new Money(26, "CHF");
            Money result = f12CHF.add(f14CHF);
            assertTrue(expected.equals(result));
      }

This tells EzUnit that if testSimpleAdd() fails, Money add(Money) is the only possible culprit. You could have added other methods as MUTs (and candidate culprits), but for the example one suffices.

You can now run JUnit as usual, and since everything is OK, it will pass both tests. To see what happens if a test fails, go to Money add(Money), for instance by selecting "Goto MUT… > Money.add" from the context menu of testSimpleAdd() as in

and seed an error, for instance by adding 1 to the sum. When you re-run JUnit, the following will happen:

As with compiler-generated errors, double clicking on the Problems entry leads you to the source of the problem, the MUT. Both the gutter annotation and the Problems entry lead you directly to the test case that caused the test run to fail: simply click on the gutter for a Quick Fix, or right click on the info and select Quick Fix.

Extensions

EzUnit is designed as a framework. This means that EzUnit is to be extended by plug-ins enhancing its functionality. An example can be found here.

Things to note

  1. In case you add more than one @MUT annotation to a test method (such as testSimpleAdd() above), upon test failure additional gutter annotations will appear in corresponding places, and corresponding entries will be added to the Problems tab. In most cases, only one of the MUTs will be the culprit, but for the EzUnit framework, there is no way to tell which one. More intelligent analyses of where exactly the error is located are the task of extensions to EzUnit.
  2. You can change the @MUT annotations of test cases anytime, either by hand editing the @MUT annotation or by calling "Select MUTs" from the context menu of a test method in an outline view.
  3. You can switch back and forth between test cases and MUTs by right clicking on a method in an outline view, and selecting "Goto Test Case…" or "Goto MUT…", respectively. This should allow a smoother process of getting MUTs and tests synchronized. Note that the lists of possible targets of the gotos respect the Preferences setting mentioned below.
  4. You can set EzUnit's preferences in the Window > Preferences > EzUnit Preferences page. Settings include packages to exclude from the call graph computation for a test method (and thus from the automatically generated list of possible @MUT annotation entries) and how possible targets of the above mentioned gotos are computed.
  5. If you create your JUnit test cases with Eclipse's wizard (New > JUnit Test Case), it adds a Javadoc tag to each test method that points to the method for which it was created. This is currently not considered in any way by EzUnit. However, a "Synchronize Javadoc with Annotation" feature might prove useful.
  6. We know of no way to include the @MUT annotation in new projects automatically, so you have to do it by hand (but only once per project).
  7. @MUT annotation entries for methods that do not (or do no longer) exist are not flagged in any way; also, these methods are (despite their absence) listed as possible goto MUT targets. Fixing this would mean to check annotations at compilation time, which we do not do.
  8. @MUT annotation entries that are not in the call graph of a test case are not flagged. This would require an analysis of annotations during compilation; also, it would not consider possible reflective calls.
  9. Continuous Testing seems to be an orthogonal approach to making unit testing more attractive.

Team