Conditionally ignoring JUnit tests

Barend Garvelink

A useful technique that I reinvent every once in a while is conditionally ignoring JUnit tests. Unit tests are supposed to be isolated, but occasionally you hit something that makes assumptions about the environment, such as code that executes a platform-specific shell command or (more commonly) an integration test that assumes the presence of a database. To keep such a test from breaking unsuspecting builds, you can @Ignore it, but that means you have to edit the code to run the test in a supported environment.

Proper Maven projects put their integration tests in a separate source folder called src/it/java and put an extra execution of the maven-surefire-plugin into their pom.xml, tied to the integration-test phase of the Maven build lifecycle. This is Maven's recommended way of setting these up. It ties in beautifully with the pre-integration-test and post-integration-test phases that can be used to set up and tear down the environmental dependencies of the integration test suite, such as initializing a database to a known state. There is nothing wrong with this approach, but it's a bit heavy handed for the simplest of cases.

In these simple situations it's easier to just keep the integration tests in the src/test/java directory and run them along with all your other tests. However, you still need a way to trigger them only when the right environment is present. This is easily dealt with by writing your own JUnit TestRunner and some custom annotations, as shown below.

The Entry Point

The default test runner in JUnit 4 is BlockJUnit4ClassRunner (javadoc, source code). From this class we need to override just one method:

protected void runChild(FrameworkMethod method, RunNotifier notifier)

This method is run once for each test and checks whether the @Ignore annotation is present. Let's add an annotation of our own, where we run a class only if a certain Java system property is set.

The Method Annotation

We'll define a simple annotation that targets the method level and defines the system property that must be set for that test method to run:

@Target( ElementType.METHOD )
@Retention(RetentionPolicy.RUNTIME)
public @interface SystemPropertyCondition {
    /** The name of a system property that must be set for the test to run. */
    String value();
}

The Test Runner

Writing the test runner is similarly easy:

public class ConditionalTestRunner extends BlockJUnit4ClassRunner {
    public ConditionalTestRunner(Class klass) {
        super(klass);
    }
    @Override
    public void runChild(FrameworkMethod method, RunNotifier notifier) {
        SystemPropertyCondition condition =
            method.getAnnotation(SystemPropertyCondition.class)

        if (condition != null && System.getProperty(condition.value()) != null) {
            super.runChild(method, notifier);
        } else {
            notifier.fireTestIgnored(describeChild(method));
        }
    }
}

The Usage Example

To use this test runner, annotate your test class with the @RunWith(Class) annotation. Any of the test methods can then be annotated to be conditional.

@RunWith(ConditionalTestRunner.class)
public class SomeExampleTest {

    @Test
    @SystemPropertyCondition("com.mycompany.includeConditionalTests")
    public void testMethodThatRunsConditionally() {
        // Normal test code goes here
    }
}

This approach can easily be extended for greater flexibility. For example, if you want to be able to annotate entire classes as well as individual methods, make the annotation target both ElementType.TYPE and ElementType.METHOD and, in your test runner, evaluate not only method.getAnnotations(), but also getTestClass().getAnnotations(). It's not difficult to add extra annotation types that check for things like Host OS, environment variables or an open TCP port on localhost.

The one thing to keep in mind is that each and every one of these conditionals that you add to your code violates the ideal situation where unit tests can run in any environment and any order. Apply judiciously.

Comments (8)

  1. Dridi Boukelmoune - Reply

    October 7, 2012 at 12:48 pm

    Hi,

    You are missing two points here:

    Maven comes by convention with the failsafe plugin for integration tests. The surefire plugin runs Test* and *Test classes, the failsafe plugin runs IT* and *IT classes.
    The standard usage of the src/it directory is for Maven plugin integration tests with the invoker plugin.

    As for jUnit 4, there is a @Rule annotation for conditional testing.

    I believe the runner API serves a different purpose.

    Dridi

    Sent from my smartphone.

  2. Dridi Boukelmoune - Reply

    October 7, 2012 at 1:52 pm

    Sorry for my mistake, it's the @Assume annotation you can use for conditional testing. The @Rule annotation is related to the subject though.

    • Dridi Boukelmoune - Reply

      October 8, 2012 at 11:55 am

      Two mistakes in a row...

      Assume is not an annotation but a class (it makes more sense actually).
      http://www.junit.org/apidocs/org/junit/Assume.html

  3. Pat Ludwig - Reply

    October 9, 2012 at 5:09 pm

    The issue with Assume is that affects your test statistics. If an Assume condition is found to be true, it treats the test as having passed even though it hasn't run. Annotations may offer the same conditional test execution but treat the skipped test as though it were @Ignore

  4. Karl - Reply

    July 22, 2013 at 8:30 am

    Thanks! Barend.

    I am looking for a solution, and it is right here! It is just what I searched for.

  5. mukund jindal - Reply

    June 5, 2014 at 11:40 am

    thanks a lot @barend it helped me a lot..i was caught into this complexity for a while. i highly appreciate your effort .
    thanks a lot. keep up the good work.

Add a Comment