Mocking the 'unmockable': too much of a good thing?

Static calls, final classes, objects created in test code: there are few things some of the current mocking frameworks cannot handle. Using powerful approaches like bytecode instrumentation or custom class loaders, these libraries make code that was previously a 'no go' area amenable to unit testing. This, moreover, in an elegant and convenient manner that will feel familiar to developers used to 'standard' mocking frameworks.
The question is: does such power perhaps come with hidden dangers? Might it be possible that the ability to test more could actually result in less code quality?

Recently, I published some code samples from a comparison of test frameworks I undertook some while ago, and wrote a blog post about some of my impressions and thoughts.

Crossing the Final Frontier

In one section, I mentioned a couple of libraries that aim to cross TDD's "Final Frontier", making it possible to test code such as the following, containing for instance static calls, final classes and instantiating collaborators in the code under test.

public final class ServiceA {

  public void doBusinessOperationXyz(EntityX data)
      throws InvalidItemStatus {
    List<?> items = Database.find("select item from EntityY item where item.someProperty=?",
                                  data.getSomeProperty());
    BigDecimal total = new ServiceB().computeTotal(items);
    data.setTotal(total);
    Database.save(data);
  }

}

public static final class ServiceB { ...

I remarked that I found the approaches of the two frameworks, JEasyTest and Jmockit, rather clunky. In the ensuing discussion I was happy to learn that my samples were rather out of date: Jmockit and PowerMock, a "mock extension" framework frequently mentioned in the comments, are now much more convenient, with a syntax very close to what one is used to from e.g. EasyMock or Mockito.1

Dangers of an undiscovered country?

Certainly, the existence of a Jmockit or a PowerMock is a tribute to the powerful techniques available via the JVM and Java language and, more so, the Java developer community. But equally I have a suspicion that, without carefully considered guidelines, the flexibiliy afforded by these "power mocking" frameworks could quickly lead to code that is as bad as it is (newly) testable.

Can ++testability = --quality?

Testability is frequently2 named as one of the ilities good software should aspire to, and I certainly agree with that3. Assuming for a minute that testability implies a high degree of unit testing4, two of the main benefits of a comprehensive test suite for me are:

  1. as a self-verifying functional description of the code
  2. ensuring code is unit testable helps avoid anti-patterns

From the perspective of the first point, power mocking frameworks are undoubtedly useful. The more code can be tested and thus described, the better - there will always be cases that standard mocks cannot address: calls to (for instance) Thread.currentThread(), legacy code5, invocations of final utility methods that may be expensive, such as cryptographic operations, etc.

It's the second aspect, however, that set me thinking: what is the relationship between testable and good code, aside from any potential "intrinsic" value of testability? Referring to Miško Hevery's frequently-cited list of testability nightmares, it is clear that there appears to be substantial overlap between patterns that make code hard to test and recognised anti-patterns that impact other aspects of software quality.
Whether it is static methods that may leak global state, or collaborators created in code that are potentially indicative of tightly-coupled systems, it (coincidentally?) looks as though code that used to be hard or impossible to attack with standard mocking6 was probably flawed in some more fundamental way.

Returning to our "canonical" example of hard-to-test code7:

public final class ServiceA {

  public void doBusinessOperationXyz(EntityX data)
      throws InvalidItemStatus {
    List<?> items = Database.find("select item from EntityY item where item.someProperty=?",
                                  data.getSomeProperty());
    BigDecimal total = new ServiceB().computeTotal(items);
    data.setTotal(total);
    Database.save(data);
  }

}

public static final class ServiceB { ...

In my opionion, creating a new collaborating service is not bad primarily because it's untestable, but because it introduces coupling and undermines polymorphism, and similarly for the static invocations of the Database class.

When less can be more

Succinctly put, the inability to accomplish certain things using standard mocking frameworks may be something of a blessing in disguise. Their constraints mean that aiming for testability effectively forces the developer to avoid common anti-patterns that are detrimental to scalability, reusability, maintainability and many of the other important aspects of code quality.
Testability as a proxy for other (perhaps more?) important aspects that are often more difficult to quantify and explain, if you will.

Having said that, I should make it clear that I consider Jmockit, PowerMock and the others to be good things! I simply feel their use will need to be accompanied by carefully considered best practices if our software development process is to reap the real benefit: better code, rather than simply 100% unit test coverage.

Footnotes

  1. There are most likely further interesting "power mocking" frameworks out there that aren't mentioned here. If you know of one please leave a comment!
  2. See here or here or here, to mention but a few.
  3. However, I don't feel "testability" necessarily has to mean "convenient to write unit tests for". My colleague Vincent Partington and I frequently get into discussions as to whether adding a package-level setter to a class to make the object more testable is justifiable, with me on the no side.
    Equally, whilst I find Google's @VisibleForTesting annotation a valuable hint to the developer, I feel it shouldn't really be necessary in the first place. But that's a topic for another blog post...
  4. A debatable assumption, of course!
  5. A comment to the previous post outlined the interesting case of "reverse-documentating" the dependencies and effects of un- or partially known legacy code using power mocking.
  6. This is not to imply that I believe power mocking frameworks are recommending the use of statics etc., just because they make it possible to mock them.
  7. There is a bit of a weak argument since the example is clearly not particularly realistic.

Comments (5)

  1. Steve Freeman - Reply

    December 6, 2009 at 12:41 pm

    The great discovery of TDD as a distinct technique is that writing tests first helps to drive better code structure. We've learned to what our tests are telling us about our designs, rather than finding new ways to work around them.

    If you're interested, I wrote a blog post on this topic some time ago. We expand on these ideas in our book Growing Object-Oriented Software Guided by Tests.

    Edited on 07.12.2009 to inline a couple of links.

  2. Peter Veentjer - Reply

    December 7, 2009 at 12:38 am

    Mocking is overrated anyway. There are 2 forms of unit testing: state based and behavior based verification. With the introduction of mocking frameworks making behavior based verification 'easy', it seems that most developers I meet only are able to use this type of test.

    Often leading to horrible testcode containing 30/40 lines of expectations and not providing any guarantee that the code works since the outcome is not verified. And mocking often is horrible when you are refactoring since you have to refactor the tests as well. And this is where I see the main problem with behavior-based verification: it is very easy to introduce bugs because developers tend to add extra recordings just to make the bar turn green without realizing what the algorithm is supposed to do. With state-based verification test code is much more independent of the code so instead of getting dragged down by the test code you get a speedup because you can refactor quicker because your tests will find problems sooner.

    I have seen this on quite a few projects with all kinds of teams. So although behavior-based verification is a powerfull technique it is just another tool in toolbox that only should be used under the right circumstances.

    Edited on 07.12.2009 to correct the typos Peter mentions in his next comment 😉

  3. Peter Veentjer - Reply

    December 7, 2009 at 12:40 am

    Sorry for the typos, I'm writing this on my iPhone

  4. Misko Hevery - Reply

    December 7, 2009 at 4:53 pm

    Could not agree more! Well said!

  5. Rogério Liesenfeld - Reply

    December 13, 2009 at 3:07 am

    "We’ve learned to [listen to] what our tests are telling us about our designs, rather than finding new ways to work around them."

    Well said. Work-arounds are never good, whether for design problems or for limited mocking tools.

Add a Comment