Testing the testers: code samples from a TDD framework comparison

Some while back I was preparing a presentation on mocking and testing frameworks for Java. As part of the aim was to demonstrate some real, running code, I ended up spending quite some time copying, pasting, extending and correcting various examples gleaned from readmes, Javadoc, Wiki pages and blog posts. Since then, this codebase has been extended with various new features I've come across, and I've often referred to it for experiments, as a helpful reference, and suchlike.

I imagine this kind of "live" reference could also be useful to others, so I thought I'd share it.

Mock Braindump

By way of introduction, here first a braindump of various points of interest related to the frameworks in the sample1. Don't expect a detailed overview or comparison - there are plenty out there - and for specific questions about the working of the frameworks please refer to their documentation2.

Mockito or EasyMock?

EasyMock and Mockito are the de facto standard mocking frameworks out there at present. Their feature sets are more-or-less identical, and if one of them comes up with something new you can be pretty sure it'll be in the next version of the other.

Personally, I have to say that I find Mockito's syntax just that little bit nicer and more intuitive. The fact that you don't explicitly have to switch to replay mode is handy, and for things like spy (for partial mocks) or InOrder (for ordered expectations) the syntax is simply more elegant.
More fundamentally, the emphasis on stubbing rather then laboriously verifying calls to mocks was, to me, a very useful and justified distinction.

Having said all that, it might well strike you as paradoxical that I still mainly use EasyMock on a day-to-day basis: force of habit and the fact that the projects I'm working on started out with EasyMock sees to that. The advantages I believe Mockito has are not sufficiently large to justify the switch.

Statics, locals and finals: TDD's final frontier?

"To boldly go...where no mocking framework has gone before" seems to be the mission of both Jmockit and JEasyTest.

public final class ServiceA {

  public void doBusinessOperationXyz(EntityX data)
      throws InvalidItemStatus {
    List<?> items = Database.find("select item from EntityY item where item.someProperty=?",
    BigDecimal total = new ServiceB().computeTotal(items);


public static final class ServiceB { ...

The challenge: to be able to unit test this class, specifically the find and save calls on Database and the computeTotal call on the new ServiceB instance. Using convential mocking this is nigh-on impossible:

  • find and save are static
  • ServiceB is a final class and thus can't be mocked
  • even if it could be, the ServiceB instance called is created in the code under test

My immediate reaction? If this is the kind of code you're supposed to test, you have other problems! Yes, of course it's an artificial example specifically chosen to highlight cases normal mocking can't deal with. But even if the real code you're having trouble with contains only one of these cases, I'd consider trying to refactor the code before looking for a different test framework.

Dependency Injection may have become a bit of a religion, but it is not so widespread for nothing. For code that's doing DI, the standard mocking frameworks are almost always sufficient.

But even if one perhaps shouldn't try to unit test the example, could one? Well, experience shows that there are few problems that cannot be solved with a sufficiently large dollop of bytecode manipulation.
JEasyTest works its brand of magic using a pre-test weaving step, which can be done by an
Eclipse plugin or by adding a plugin to your Maven build3. Jmockit uses the slightly more modern instrumentation approach and comes with a Java agent, which means that it only takes an additional VM argument to run in your IDE.

Usability issues aside, I found that I don't feel happy with the code to prepare a test fixture and register expectations in either framework; it's downright clumsy in some cases. Here's the JEasyTest test:

public void testBusinessOperation() throws InvalidItemStatus {
            arg("select item from EntityY item where item.someProperty=?"),
    assertEquals(total, entity.getTotal());

expectStaticNonVoidMethod? ARGH! Feels more like ASM than unit testing.

Jmockit's "expectations" mode comes closest to what Mockito/EasyMock users are likely to be familiar with4:

private final Database unused = null;
private ServiceB serviceB;

public void doBusinessOperationXyz() throws Exception {
  EntityX data = new EntityX();
  BigDecimal total = new BigDecimal("125.40");

  List<?> items = new ArrayList<Object>();
  Database.find(withSubstring("select"), withAny(""));
  new ServiceB().computeTotal(items);

  new ServiceA().doBusinessOperationXyz(data);
  assertEquals(total, data.getTotal());

Conceptually, this is reminiscent of an EasyMock test, with record and replay phases. The @MockField fields stand in for the creation of actual mock objects: the field declarations only indicate to Jmockit that mock of the given types are required when the test is run, cluttering the test class with unused properties.
In addition, the "mock management" methods (withAny, returns etc.) are not static, meaning they are not visually identified by e.g. being displayed in italics. I was surprised how much this seemingly minor discrepancy alienated me - it just doesn't look quite like a unit test.


From what I can see not much is happening around jMock anymore: the last release was in Aug 2008 and the last news posting over half a year ago. The syntax, which tries to mimic a pseudo-"natural language" DSL, is just a bit too cumbersome. jMock's support for multithreading prompted me to take a closer look, but it's actually simply a mechanism for ensuring that assertion errors thrown in other threads are actually registered by the test thread; there is no support for testing concurrent behaviour.

Testing concurrent code

I quite like MultithreadedTC, a small framework5 which aims to make it easy to start and coordinate multiple test threads. It does this by means of a global "clock" that moves forward whenever all threads are blocked - either "naturally" (e.g. during a call such as blockingQueue.take()
) or deliberately using a waitForTick(n) command.
As such, MultithreadedTC doesn't offer much more than can be achieved by "manual" latches as described in Iwein Fuld's recent blog post, but the clock metaphor does seem to make the test flow easier to understand, especially for longer tests.

Like latching, though, the main problem with MultithreadedTC is that you can't easily control the execution of code in the classes under test.

public void thread1() throws InterruptedException {

public void thread2() throws InterruptedException {

This code will go some way to ensuring that service.someMethod() and service.otherMethod() start at almost the same time, and will guarantee that neither thread will continue until both methods have completed. But what if you want to ensure that half of someMethod completes before otherMethod is called?

For that, you'll have to be able to get access to the implementations of someMethod and otherMethod, for instance by subclassing the service implementations, or using something like Byteman.

Ultimately, though, I think unit tests are just not the right way of going about testing concurrent code. "Choreographing" the carefully-chosen actions of a small number of test threads is a poor substitute for real concurrent usage, and the bugs you'll find, if any, aren't the kind of concurrency issues that end up causing nightmares.

For proper concurrency testing, there doesn't so far seem to be a good substitute for starting a whole bunch of threads - on as many cores as possible - and running them for a good while (see, for instance, the integration tests of Multiverse, the Java STM). If it's possible to inject a certain amount of randomness into the timing (using e.g. Byteman), all the better!

JUnit Rocks Rules!

Version 4.7 of JUnit introduced rules, sort-of around aspects that are called before and after the execution of a test. Some of the standard examples demonstrate "housekeeping" functionality such as opening and closing a resource or creating and cleaning up a temporary folder. Rules can also affect the result of a test, though, e.g. causing it to fail even if all the test's assertions were successful.

Whilst one can see how the concept of rules can be useful, they still have a bit of "v1" roughness about them. The "housekeeping" rules are essentially convenient replacements for @Before/@After logic, and the syntax of the "test-influencing" rules feels messy:

public class UsesErrorCollectorTwice {
    public ErrorCollector collector = new ErrorCollector();

    public void example() {
        collector.addError(new Throwable("first thing went wrong"));
        collector.addError(new Throwable("second thing went wrong"));
        collector.checkThat("ERROR", not("ERROR"));
        collector.checkThat("OK", not("ERROR"));
        System.out.println("Got here!");

Wouldn't that be nicer if the assertions were a little more, um, assert-like? Or take:

public class HasExpectedException {
    public ExpectedException thrown = ExpectedException.none();

    public void throwsNullPointerExceptionWithMessage() {
        throw new NullPointerException("What happened?");

I mean, it's certainly useful to be able to make more detailed assertions about exceptions, but could this not be integrated into either of the current exception-checking patterns6?

The potential power of rules also raises an question: is it wise to get into the habit of doing full-scale resource management (e.g. starting servers or DB connections) in a unit test?


  1. Sample source code here. Check out the project using svn checkout http://aphillips.googlecode.com/svn/mock-poc/trunk target-folder.
  2. See the sample code's POM.
  3. Which is not to say it's the best approach, of course!
  4. The original code is not longer being actively developed, but there has been some recent work aimed at better JUnit 4 integration.
  5. @Test
    public void tryCatchTestForException() {
      try {
        throw new NullPointerException();
      } catch (NullPointerException exception) {
        // expected
    @Test(expected = NullPointerException.class)
    public void annotationTestForException() {
      throw new NullPointerException();

Comments (13)

  1. Thomas - Reply

    November 26, 2009 at 12:29 pm

    What about Unitils?

  2. majson - Reply

    November 26, 2009 at 12:50 pm

    You should also take a look at PowerMock which could be used on top of EasyMock or Mockito.

  3. Andrew Phillips - Reply

    November 27, 2009 at 1:49 am

    @Thomas: I hadn't come across Unitils before; have just had a brief look at the website. Initial reactions: I'd be glad to hear about what it offers over and above (for instance) the Spring integration tests.
    Seeing that DbUnit is part of the suite brought back some not wholly pleasant memories of undocumented, strange code and an seemingly dormant project. I'm glad that it appears once again to be under active development.

    @majson: Thanks for that, I'll try to get round to adding the PowerMock examples to the project soon. The feature list reminds me of Jmockit and JEasyTest - I'm curious to see if PowerMock will turn out to be a bit less clunky.

  4. Rogério Liesenfeld - Reply

    November 27, 2009 at 2:10 am

    Concerning JMockit, I wrote a response to this article in the Javalobby post.

  5. Andrew Phillips - Reply

    November 27, 2009 at 2:06 pm

    @Rogério: Thanks for your response, the features and more recent changes you describe look interesting. For readers, I'll just repeat your link to the examples page you refer to.

    Over at the DZone reference to the article there's also a response from the jMock team. I'm glad that my impression that there wasn't much going on there anymore is unfounded - apparently, they have been busy preparing a book.

    There is also a mention of Thread Weaver, another multi-threaded unit testing framework. I haven't come across it before but from the user guide it appears to offer quite powerful interleaving functionality with code instrumentation.

  6. Rogério Liesenfeld - Reply

    November 27, 2009 at 4:37 pm

    Thanks Andrew, I appreciate it.

    I just added a response to the jMock one at DZone, to clarify the meaning of "record/replay", which for whatever reasons seems to be different according to who you ask.

    Of the several mocking tools you evaluated, I believe the only one "dead" right now would be JEasyTest, which hasn't seen any real activity since january of 2008.

  7. [...] Testing the testers: code samples from a TDD framework comparison Ich muss mir definitiv die Mocking Frameworks genauer ansehen. [...]

  8. majson - Reply

    November 28, 2009 at 1:11 am

    I've been using Unitils in my projects for some time, but for testing JPA. And this part works really good. There are some issues with DBUnit integration (flat xml data set) like null values and also case sensitiveness for column names which I simply cannot enforce to work with MySQL. However if you use DBUnit 2.4.3* it works perfectly fine for the first issue.

    This toolkit really simplifies testing infrastructure and is definitely worth checking if you want to test some non trivial logic relied on JPA/JPQL and don't want to use one of the embedded containers.

    * It seems that this is the only version from 2.4 branch that works with current version of Unitils for the time being. Issue has been already reported.

  9. Andrew Phillips - Reply

    November 28, 2009 at 1:24 pm

    @majson: Thanks for providing some details on Unitils - looks like you answered the question I posed to Thomas (see my first comment) ;-). From what you describe Unitils seems to fall in that contentious area between unit and integration testing, but a similar thing could probably also be said for JUnit Rules.

  10. Andrew Phillips - Reply

    November 28, 2009 at 1:33 pm

    PS: An interesting discussion regarding the suitability of the record-replay[-verify] model over at the DZone reference to this post. The "reposting gremlins" strike again...sigh.

  11. majson - Reply

    November 29, 2009 at 12:46 am

    @Andrew Phillips That's why I didn't call it "unit testing", just testing. You should give it a try when you will need to test some JPA stuff. I really like it 🙂 Keep on good writing.

  12. Rogério Liesenfeld - Reply

    November 30, 2009 at 9:13 pm

    Unitils has several "modules", each for a different purpose in the context of developer testing.

    The one relevant to this discussion is the "Mock" module, which happens to be similar to Mockito.

  13. Andrew Phillips - Reply

    December 6, 2009 at 2:45 pm

    A follow-up to the interesting discussion here and in comments to the DZone reference to this post appears here.

Add a Comment