Middleware integration testing with JUnit, Maven and VMware, part 3 (of 3)

Vincent Partington

Last year, before the Christmas holidays ;-), I described how we do middleware integration testing at XebiaLabs and I described the way we deploy test servlets by wrapping them in WAR and EAR files that get generated on the fly. There is only one thing left to explain; how do we integrate these tests into a continuous build using Maven and VMware?

Running the middleware integration tests

So let's start with the Maven configuration. As I mentioned in the first blog of this series, the integration tests are recognizable by the fact that the classnames end in Itest. That means they won't get picked up by the default configuration of the Maven Surefire plugin. And that is fortunate because we don't always want to run these tests. Firstly they require a very specific test setup (the application server configurations should be in an expected state, see below) and secondly they can take a long time to complete and that would get in the way of the quick turnaround we want from commit builds in our continuous integration system.

But when we do want to run the middleware integration tests, we can configure the Maven Surefire plugin with a Maven build profile like so:

<profiles>
    <profile>
        <id>itest</id>
        <build>
            <plugins>
                <plugin>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <includes>
                            <include>**/Test*.java</include>
                            <include>**/*Test.java</include>
                            <include>**/*TestCase.java</include>
                            <include>**/*Itest.java</include>
                        </includes>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
</profiles>

To run the middleware integration tests in addition to the regular tests, we can tell Maven to use this profile with the -P flag:

$ mvn -Pitest test

If we want this profile to run only the middleware integration tests, we need to leave out the lines that include the standard classnames (Test*.java, *Test.java and *TestCase.java).

Defining the expected state for the target middleware

A big challenge when performing middleware integration tests is to ensure that the middleware we test against is in a correct state. For example, the test might expect an application server cluster to be running but no application to be deployed on it. In a regular test we may set up a test fixture in a method marked with @Before, but in this case that might not be so simple. Confuguring the middleware to be in the expected state would be an integration in itself!

The first approach to address this issue is to write all middleware integration tests in such a way that they revert any changes they make. For example, a test that creates a datasource also removes it. In this way we combine the test for the creation-step with the test for the destruction-step. The disadvantage here is that these two steps can no longer be tested independently, but the advantage is that the combined test runs more quickly. For this approach to be successful, all tests using the same target middleware should agree on a common expected state. We configure the target middleware in that state once and all the tests return the middleware to that state after completion. With a setup like this we can quickly run one or more integration tests, either interactively from an IDE such as Eclipse, or from Maven.

But there is one problem with this. What if the test fails? In that case the middleware might not be left in the correct state. One could argue that the steps needed to revert the middleware should be placed in an @After method but they may still fail. More importantly, those steps are also part of the test so they don't really belong there!

Reverting the target middleware to the expected state with VMware

This is where VMware comes in. By installing the target middleware in a hypervisor such as VMware, we can make a snapshot when the middleware is in the expected state. Then we can revert to that state before executing a single test or a test suite. The simplest thing is to manually revert from the hypervisor administration console before running the tests. But we can also revert the images automatically, so that we can integrate these tests into our continuous integration builds.

The images we use for our middleware integration tests run on VMware vSphere 4 and we've found the vSphere SDK for Perl, and most specifically the provided vmrun.pl sample script, to be the easiest way to interface with VMware vSphere. As an example, this is the script we use to revert our JBoss image:

IMAGE_NAME="[datastore1] jboss-42/CentOS 5.3, JBoss 4.2.3-GA.vmx"
SNAPSHOT_NAME="Ready for itest"
SETTLE_TIME=60
VMRUN=/usr/lib/vmware-vix/lib/api/vix-perl/samples/vmrun.pl

echo "Reverting VMWare image $IMAGE_NAME to snapshot $SNAPSHOT_NAME"
$VMRUN -host $VMWARE_HOST -username $VMWARE_USERNAME -password $VMWARE_PASSWORD \
    revertToSnapshot "$IMAGE_NAME" "$SNAPSHOT_NAME"

echo "Waiting $SETTLE_TIME seconds for things to settle"
sleep $SETTLE_TIME

We do need to provide the proper values for the environment variables VMWARE_HOST, VMWARE_USERNAME and VMWARE_PASSWORD when running this script. After the image has been reverted we wait for 60 seconds to allow the image to settle. The amount of seconds needed here is hard to determine exactly but this value works well for us.

To have Maven run this script before the integration tests are executed, but only when a certain profile is selected, we have added this configuration to the POM:

<profiles>
    <profile>
        <id>revert-vmware-images</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>exec-maven-plugin</artifactId>
                    <version>1.1</version>
                    <executions>
                        <execution>
                            <!-- We really want this to execute right before the test phase. -->
                            <!-- This is the best we can do. -->
                            <phase>process-test-resources</phase>
                            <goals>
                                <goal>exec</goal>
                            </goals>
                        </execution>
                    </executions>
                    <configuration>
                        <executable>/bin/sh</executable>
                        <commandlineArgs>src/test/scripts/revert-vmware-images.sh</commandlineArgs>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
</profiles>

Putting it together

With the two profiles introduced in this blog, we tell Maven to revert the VMware images and execute all middleware integration tests by giving the following command:

$ mvn -Prevert-vmware-images -Pitest clean test

We have configured our continuous integration system to run this command once a day, as a secondary build, and configured it with the right values for the VMWARE_HOST, VMWARE_USERNAME and VMWARE_PASSWORD environment variables. In this way the middleware integration tests do not slow down the standard commit build (they take about 90 minutes to complete in our case!). As an added benefit, we know when the VMware images will be used by the tests so we can use them for manual experiments at other times.

Conclusion

In this blog and the first two blogs of this series I have explained how we use JUnit, Maven and VMware to write repeatable, manageable middleware integration tests. These tests have helped us gain a lot of confidence in the stability of Deployit's middleware integrations. And because this test-support code is part of Deployit's plugin API, Deployit plugin developers can use it to get the same confidence in their own code!

Comments (2)

  1. Eru - Reply

    January 7, 2010 at 9:40 pm

    Read the blog, although a lot of things went over my head but still found it interesting. everything sounds perfectly fine but i am not sure about two things

    first thing is Defining the expected state for the target middleware.
    This is not very difficult as far as you know what you want in your configuration.... i accept it will be very difficult when it comes to performance tuning coz that takes a lot of time to try different combination and then analyzing the results from those combinations, but if we are clear about the configuration we want for our testing environment we can easily (not so easy) automate this using UNIX scripting.

    Second thing is why are we investing so much effort in VMWare, why are we not taking a backup of master configuration of our App Server and in case our test fails, then revert to original configuration.

  2. Vincent Partington - Reply

    January 8, 2010 at 11:05 am

    @Eru: Setting up the expected state is indeed not that hard, by hand. As for scripting that part: automating the middleware to be in a certain state is exactly the thing we are trying to test! So it's actually not that easy. Which is we set it once manually and then have the tests clean up after themselves (which is basically the automation you are seeking). And if that fails, we use revert the VMware image.

    The reason we use VMware to revert the configuration is because we it is a generic way to do this for all the different middleware systems we integrate with (WebSphere, WebLogic, JBoss, Apache for now, but soon ALSB, WebSphere MQ, WebSphere Portal, etc.) without having to figure out how to backup and restore their configurations.

    There is not so much effort involved actually. Setting up a VMware image is surprisingly easy. Another advantage here is that only one person on our team has to spend time setting up that image and the rest can just copy it! At XebiaLabs we are also very excited about the possibilities of using VMware to ease deployment as my colleague Robert van Loghem has blogged about recently. Ease of deployment is one of the big advantages of cloud computing.

Add a Comment