Here, we explain how Deployit‘s Infrastructure and new Release Overview features help you achieve this goal.

Continuous Delivery & the Release Perspective

In today’s highly competitive economic environment, the need to bring new features to market quickly, flexibly and reliably is paramount – a goal that is ultimately the aim of the main IT trends Cloud, Agile and DevOps.

Continuous Delivery – extending Continuous Integration to automatically transition applications down the Dev-Test-Acc-Prod delivery pipeline – is a key component of this strategy. In order to be able to effectively implement this, your ARA solution needs to allow your developers – or, in larger organisations, release or DevOps teams, to quickly and efficiently answer questions such as:

• How far is MyApplication down the road to Production?
• When will MyApplication take the next step down the road?
• What do I still need to do before that next step can be taken²?

Ideally, this dashboard would also allow you to plan MyApplication‘s next step and calculate the estimated go-live data, perhaps even based on an analysis of previews versions of MyApplication.

(Virtual) Environment Management & the Operations Perspective

From an Operations point of view, an individual application is only a small part of the picture. Across your Dev-Test-Acc-Prod landscape, you will need to track all applications vying for these environments, to manage potentially conflicting resource requests, plan environment maintenance activities and the like.

Since these environments are often owned and managed by different teams and certainly have varying service levels, you will also want to limit your view to one or a subset of these environments at a time.

Your Operations or DevOps teams need to know:

• Which application versions are currently deployed to my environment(s), or were deployed at a certain point in time?
• Which components do these applications consist of? On which middleware and infrastructure systems are these components deployed?
• What are the current values of any properties or settings for these components? Which environment-specific customizations have been applied?

Cloud and the on-demand environments it enables will eventually replace the rigid Dev-Test-Acc-Prod distinction³. Nevertheless, the ability to present an environment-centric view will still be required, since virtual environments will still be owned by different groups or teams. Indeed, such a perspective will be even more important if you want to effectively combat “virtual sprawl”.

While the coming generations of “true” cloud architectures will hopefully reduce the shared resource conflicts that greatly complicate much of today’s Dev-Test-Acc-Prod management, databases, legacy systems and external payment providers are not likely to disappear anytime soon.

In fact, Facebook, Twitter and other social elements of your future business services may even increase the number of shared resources you need to manage!

Incorporating ARA Data in the Service Delivery Picture

Whilst your ARA solution should be your “go-to” platform for answers about how your applications and environments relate, it is equally important to consider when this data might be more effectively embedded in a broader service delivery picture.

For example, your ARA platform is not a good candidate for providing a release calendar, since it is not aware of much of the information that is relevant in this context, such as CAB⁴ meeting schedules, business sign-off dates or operational maintenance windows.

It is thus important to ensure that your ARA solution can make its data accessible via APIs such as RSS feeds, iCal calendars and other APIs, to enable effective integrations with the rest of your service delivery tooling.

Conclusion

The right Application Release Automation platform gives your Delivery and Operations teams fast, accurate insight into your application environments and delivery pipeline.

Choosing a solution like Deployit with focused Operations and Delivery overviews as well as open APIs for easy integration into your overall Service Delivery dashboards and reports greatly enhances the accessibility and effectiveness of your application release management.

What early adopters have learnt is that, at the enterprise scale, automating release execution quickly introduces a new bottleneck in today’s dynamic IT environments: continuous management of the deployment plan definition. A new generation of application release automation (ARA) tooling avoids this pitfall by leveraging intelligence to automate deployment planning as well as execution.

The State of ARA

Given how strongly our IT industry is dedicated to the automation of processes, it is nothing short of amazing how much of the deployment of our own solutions depends on manual actions coordinated more or less effectively between large groups of people.
Indeed, a recent analyst report noted that the majority of large enterprises were still relying on manual application release processes or on in-house scripting understood by only a small number of specialists, operated as a black box that – hopefully – will do its job and will most likely necessitate a painful troubleshooting session if it doesn’t.

With key IT trends such as Agile, Cloud and DevOps dramatically ramping up the frequency of application releases in order to increase responsiveness to business needs and provide more and better services to customers, it’s clear that this situation cannot continue.

Thinking of today’s common release processes, it is also hardly surprising that the initial drive has been to automate the actual rollout of the application itself: copying the files to the target machines, restarting the servers, running the SQL against the DB etc. Using a defined workflow to organise these activities makes lots of sense: fewer failures, no more missing steps or steps executed in the wrong order, no more typos, better visualization etc.

Lessons from the First Generation

Ironically, using one of these first generation ARA tools at an enterprise scale quickly made it obvious to early adopters how much effort is required to maintain the substantial number of workflow definitions that quickly accumulate to support full deployments, partial upgrades, rollbacks, environment scale-outs etc. across an enterprise application portfolio.

Of course, this is not a new challenge: ask anyone who has had to update 100 build job definitions in a continuous integration tool to change a compilation parameter, or 100 test plans in an automated testing setup to accommodate a different target browser, just how time-consuming and error-prone this type of maintenance is.

It’s not as though these modifications are unique per process. They tend to be systematic changes that reflect changes to the overall deployment strategy and/or context. The issue is that these first-generation tools, where all the deployment intelligence is stored in the power user’s brain, simply do not have enough internal knowledge of the structure of deployment to assist effectively.

The Next Level of Application Release Automation

A new generation of Application Release Automation includes this intelligence. These advanced tools² no longer require hand-holding by your power users every step of the way, and encode knowledge of deployment best practices and strategies to automate the planning and execution of deployments.³

No pre-provided strategy can be a 100% fit in an enterprise environment, so the strategies must be configurable, of course. Once your power users have fine-tuned them, however, all the individual deployment plans – initial installations, full and partial upgrades, downgrades, undeployments, scale-outs etc…easily hundreds across an application portfolio – are automatically tailored to your scenario.

This becomes even more efficient the fewer deployment strategies are in play, so these tools also motivate and reward increased standardisation of deployment procedures, in itself a valuable business goal. In fact, with a suitable interface and integrations into the development pipeline you essentially have an enterprise Platform as a Service, potentially on a private or hybrid cloud.

Conclusions

With adoption of Application Release Automation rapidly on the increase, a new generation of solutions are appearing that automate deployment planning as well as execution.
Based on the challenges experienced in scaling the first generation of ARA tools to enterprise levels, these next generation solutions are designed to eliminate “continuous expert hand-holding”, promote standardisation and allow organisations to create a “software factory” that continuously delivers business value.

A key part of the test harness for the original versions of Tweetstore that run on the Google App Engine is the neat GoogleDevServer class. Basically, it’s a clever wrapper around the KickStart class used under the covers by the GAE SDK tools that allows you to specify the SDK location, address, port and WAR file (or expanded WAR directory) to run.
Better still, it can programmatically be shut down cleanly, making it ideal for integration test runs¹.

For CloudBees, I was thus looking to put together a similar RunAtCloudServer. It proved more challenging than expected…²

Just spawn a container already?

Both the maven-gae-plugin and the bees-maven-plugin allow you to spawn a local GAE or RUN@cloud server simulation, of course. These can easily be integrated into your Maven build, for instance as part of your pre-integration-test setup. So why not just stick with that?

Well, for some scenarios that can be perfectly sufficient. For Tweetstore, having an inline server that cleanly shuts down allows us to:

• run integration tests alongside unit tests from an IDE without having to remember to start an external process
• debug the application running in the server on demand, without having to mess with remote debuggers or figure out a way to sometimes get the Maven plugins to start in debug mode
• work some conditional magic with the server arguments

If your project’s requirements are similar, an inline server might be a good option. And luckily, the code’s already been written!

Hangin’ around

The RUN@cloud SDK’s version of KickStart is the StaxSdkAppServer³. Figuring out the correct arguments for the launchServer factory method wasn’t too hard, so I hooked up a StaxSdkAppServer and kicked off the test.

Started like a charm! Then the test finished and…just hung on in there. Over in the trusty debugger⁴, I could see not one, but two zombie threads still running. The design was based on a “shutdown-by-JVM-kill” scenario, I guess.

Assassins and other Ninja Tricks

The first of the two threads is a Timer that something called the WebAppEngine uses to do…well, who knows? In any case, we can get a reference to it with a decent handful of reflection and cancel it. Easy enough.

Our remaining zombie is a bit more elusive, unfortunately. It’s a thread spawned by an instance of the RequestMonitorValve class in a manner analogous to

new Thread(idleTimer, "requestMonitor").start();

So no reference, and no way of interrupting it. However, the idleTimer Runnable is referenced and repeatedly calls a handler as part of its execution. How about injecting a handler that throws an exception to end the thread?

Nice idea, but it turns out the thread really doesn’t want to die that way:

while(true) {
    try {
        // do stuff
        if(callbackClient != null)
            callbackClient.updateStatus(state);
    } catch(Exception e) {
        // increment some error counters
    }

    try {
        Thread.sleep(statusIntervalSecs * 1000);
    } catch(InterruptedException interruptedexception) { }
}

So I’ve had to go one better and throw a ThreadDeath error instead. Urgh.

Now you see it, now you don’t?

The remaining gotcha involves injecting the catchily-named KillerCallback into the idleTimer. Even using reflection, this is a elementary concurrency failure because without a synchronized-with relationship⁵ between injecting the new handler and invocation of the handler by idleTimer, there is no guarantee the new handler will actually be visible to the thread we’re trying to stop.

Such a relationship could be established if the “assassin” was injected before the thread that runs idleTimer starts. Unless we want to override library classes, that would mean setting it after creating but before starting the StaxSdkAppServer⁶…which unfortunately doesn’t work either because the instances referencing idleTimer are only created when the server starts.

So?

So, for now the potential concurrency issue is documented and we hope for the best No problems observered so far…

Xebium creates a simple way to use Selenium IDE (low learning curve) and FitNesse (ease of maintenance) to it’s fullest when it comes to maintaining a web application test suites.

Xebium is using the same keywords as Selenium IDE. This has the huge advantage that no person should learn another DSL. Since tests are stated this way, they can be copied between Selenium IDE and FitNesse without a hassle (the FitNesse formatter for Selenium IDE is rather trivial). And to be honest: as long as there are XPath and Regular Expressions in the code, it makes no sense to come up with a substitute for verifyText.

So now it is possible to move our Selenium IDE test case in a FitNesse page. Nice. It’s easier to set up tests, since FitNesse is quite flexible in that. On the other hand, just playing back you recorded tests is not the most flexible way to work. There can be a huge maintenance burden on this. It would be a shame not to use the full power of FitNesse in this case. Scenario tables for example can make a nice abstraction where some common action, like logging in, only have to be defined (and maintained) in one place. By moving the login code out of the test and into a scenario, we add another level of indirection. The tests themselves become more readable, as they only state the business requirement and the low level details are directly editable by the tester. This is a lot more flexible than creating Selenium fixtures for every scenario.

| log in user | user | with password | password |
| open | http://example.com/login |
| type | name-field | @user |
| type | password-field | @password |
| click | login |

and in your tests, instead of repeating the above, just mention:

| script |
| log in user | john | with password | secret |
| ensure | do | verifyText | on | #userid | with | John |

Sure, we’re back at creating our own DSL, but we do it 100% in FitNesse. Selenium commands can be added where needed (e.g. assertions). No Java (fixture) code, no developers required to change one line because the templates have changed and, foremost, detailed feedback.

But FitNesse can help out even more. Consider data driven testing. Selenium has (by default) no option for that. Using the same scenario table fixture it’s easy to test logins for a bunch of users.

But we claimed you can copy stuff from FitNesse to Selenium IDE as well and that’s getting harder when you move everything inro scenarios. To overcome this issue the formatter in Selenium IDE can parse tab separated data as well. So after a test has run, the output as provided by FitNesse can be copied back. This is a useful aid for analyzing failed tests.

I tend to make my Xebium commands easely available. First of all the fixture needs to be imported. After that the fixture can be assigned as library:

| import |
| com.xebia.incubator.xebium |

| library |
| selenium driver fixture |

A note for Windows users, though. When performing a lot of tests, Windows tends to run out of sockets. There is a way to configure your machine to work around those silly limitations (at your own risk!).

When testing web interfaces, it’s convenient to use an intuitive tool like Selenium IDE, it’s easy to use and can be used by non-technical people, but it is solely meant for record and playback of test-scripts. One of its limitations is that it misses sufficient options for documenting and managing tests. Furthermore it misses an interface with the backend of the system under test (SUT), to setup preconditions for a test or for instance to manipulate or read from a database.
Fitnesse is a great tool to do just that, it has the Wiki to manage tests and it by default has a setup and teardown mechanism, it’s easy to add non invasive testfixtures to interface directly with your SUT. The downside is that it is incapable of doing webtests.

We now have the glue that combines the two, it’s called Xebium!

On several projects I’ve tried to find a way to combine the two, because both tools are free of use and both have their individual charm, but it seems combined they would master practically any situation. As a tester I miss the development skills to come up with a satisfying solution and I never convinced my team members to help me out in a satisfactory manner.

At our company we recently started app-incubator, a new form of knowledge exchange, in the form of mini projects, anyone can pitch a project idea and when enough colleagues sponsor the project, we can spend up to three days collaborating on it.  My pitch caught fire and with about five people we brainstormed on the solution. For me a prerequisite was that it should be possible to use the test scripts two directional, so record them with Selenium IDE, run and edit them in Fitnesse and playback from Selenium IDE again. During this first project session we came up with additional requirements, it should support data-driven testing and variable substitution. We wanted to keep it as simple as possible, no new DSL’s should be introduced and it should not require conversions between the two products. We explored some existing solutions such as Fitnium and Selenesse but came to the conclusion these (at the time) were fully based on their own DSL’s and did not support exchanging files between Selenium IDE and Fitnesse. So we decided to created something far more simple.

The Result

After several attempts we discovered that Webdriver (Selenium 2.0) would be the best engine for running our tests. It seems more future proof, turned out to be a lot more stable than Selenium Testrunner and its commands are more compatible with Selenium IDE than Selenium RC.
We introduced a Selenium IDE formatter which output can be directly copied into the Wiki. We created a Fitnesse fixture that is capable of running the pre-formatted scripts without additional interpretations of the Selenium commands.

The additional features we had in mind, the data-driven testing and variable substitution turned out to be standard Fitnesse features. What we did not realize when we started was that Fitnesse would introduce a mechanism which, in my opinion, added mind-blowing power to project: Scenario tables, which now enables us to create abstractions on top of our testscripts.

Conclusion

Xebium enables us to use both Selenium IDE (low learning curve) and FitNesse (ease of maintenance) to it’s fullest and provides a complete solution when it comes to automated web testing.

How can you package an application, including library dependencies nicely on a Mac? That’s the task at hand when I wanted to make Gaphor available on the Mac platform.

The idea has been there for quite some time, and it became a lot more concrete once I started to compile and install my open source software using Homebrew.

In this post I’ll outline the way Gaphor is packaged for this platform. I started Gaphor quite some years ago on Linux. It’s written in Python using the GTK+ gui library. After I switched to OS X as my main OS, I still work on this project. The X11 server on OS X works pretty well, so nothing holds me back. Except that it’s hard to share this app with my fellow Mac users.

Gaphor and most of the dependent modules are available as Python eggs from the Python Package Index, hence installing it is a simple task. GTK+ is not available on Mac OS by default and PyGTK (GTK+’s Python bindings) is not available as egg, so those have to be installed from source. I started off with compiling the dependencies (GTK+ and friends) using Homebrew. Homebrew has a big advantage over MacPorts and Fink in that it does not try to do everything itself (build everything from scratch). Instead it tries to use functionality available on the system already (like X11, Python and some basic libs). With a fork of Stein Magnus Jodal’s Homebrew branch I was able to get GTK+ and PyGTK compiled and installed.

Next step is to get this stuff packages in an application bundle in order to make it available to a broad audience. The de-facto packaging tool, py2app, is not able to handle eggs. Since Gaphor depends on eggs (the services for running the application are defined in the egg meta data), I had to figure out some other way to package the application, while retaining the egg structure.

I started of creating a simple application structure: a Gaphor.app folder, a Info.plist file and some additional directories. Next thing is to make this environment Pythonized. The way to to this is to use VirtualEnv. VirtualEnv is a Python tool that helps you set up isolated environments. This can be very handy for example for developing and testing software. A simple

PYVER=2.6
APP=Gaphor.app
INSTALLDIR=$APP/Contents
virtualenv --python=python$PYVER --no-site-packages $INSTALLDIR

did the trick.

I needed to install all PyGTK related libraries in the application folder.

LOCALDIR=/usr/local
SITEPACKAGES=$INSTALLDIR/lib/python$PYVER/site-packages
PYTHON_BREWS="pycairo pygobject pygtk pyrsvg"

for brew in $PYTHON_BREWS; do
  cp -r $LOCALDIR/Cellar/$brew/*/lib/python$PYVER/site-packages/* $SITEPACKAGES
done

Where $SITEPACKAGES is a directory in the folder that the `virtualenv-ed’ python installation will use to install the site-packages (non-default python modules).

Of course PyGTK won’t do a thing without the backing of GTK+ itself, so those libraries also had to be packages. To solve this a listing of dependencies for the freshly installed libraries will do. otool is able to list all dependencies. Those dependencies can be copied and the paths can be changed to relative paths. Relative paths on OS X, funny enough, relate to the location of the binary that was initially started. In this case Python (which is installed in Gaphor.app/Contents/bin).

function resolve_deps() {
  local lib=$1
  local dep
  otool -L $lib | grep -e "^.$LOCALDIR/" |\
      while read dep _; do
    echo $dep
  done
}

function fix_paths() {
  local lib=$1
  log Fixing $lib
  for dep in `resolve_deps $lib`; do
    #log Fixing `basename $lib`
    log "|  $dep"
    install_name_tool -change $dep @executable_path/../lib/`basename $dep` $lib
  done
}

binlibs=`find $INSTALLDIR -type f -name '*.so'`

for lib in $binlibs; do
  log Resolving $lib
  resolve_deps $lib
  fix_paths $lib
done | sort -u | while read lib; do
  log Copying $lib
  cp $lib $LIBDIR
  chmod u+w $LIBDIR/`basename $lib`
  fix_paths $LIBDIR/`basename $lib`
done

Some extra resource files had to be installed and that was it for GTK+. The launch script takes care of setting the environment so the application will work. Now the application itself:

$INSTALLDIR/bin/easy_install gaphor

That was easy! The latest version will be downloaded from the PyPI and easy_install is installing it. One of the benefits of using virtualenv is that easy_install as well as pip are installed by default. This makes it very, very easy to install python applications.

You can find the full script on GitHub.

It took a while to figure out all the details, but creating an installable app, Mac style, is rather trivial. Next is a Windows installer and something tells me it won’t be this easy .

Note, if you want to know why i chose GHUnit over OCUnit, just scroll down to the end of the post.

But wait….

Before I begin, I want to make one thing very clear, the difference between code unit testing and UI testing. Unfortunately, UI development can be hard to do in a TDD fashion. Especially when you want to test UI components. e.g. When I send a TouchEvent will the View respond and trigger my method in my controller.
My advice; don’t do UI testing with a Unit testing framework (OCUnit, JUnit, GHUnit), do it with for example the iOS automation API, which has been created specifically to test those UI components. I’ll also get back to you in a later post on how to do UI testing on Android.

What do you test with Unit testing frameworks? Well you test just the code, nothing more. Model and Controller code, not the View! You might need the help of a mocking framework to make it testable, because the view is missing and needs to be wired up for the controller and model to work properly, or it might need other controllers etc..

Let’s begin setting up!

With that in mind, let’s set up our own iPhone XCode 4 project! add GHunit, create a test and run it in the simulator or your own iOS device, iPhone, iPod Touch or iPad.

1. First of all download the GHUnitIOS version (0.4.28 at this time) at https://github.com/gabriel/gh-unit/archives/master
2. Unpack the downloaded zip file somewhere in your home directory, you should get the GHUnitIOS.framework directory

NOTE; I first placed it in /Developer/Library/Frameworks but XCode 4 didn’t like it and when compiling it could not find the header files, therefore I placed them somewhere in my home directory (e.g. /Users/rvanloghem/Development/Frameworks/GHUnitIOS.framework)

Right, you are now ready to set up your GHUnit ready XCode 4 project.

3. For example purposes, I’m choosing a normal, navigation-based application but you might have an existing project.
- NOTE; Don’t check the Include Unit Tests, because we are going to supply our own unit testing framework and not rely on OCUnit, which is supplied by default in XCode

4. In your XCode project settings (blue root icon in the tree browser) add a Target which you can call Tests, i usually base the project on a simple View-based Application (a Target named Tests will be added + a folder named Tests with all the Tests target files in them)

5. Next go back to the Tests target (click on the Tests Target) and add the GHUnitIOS.framework which you downloaded and unpacked in step 1 and 2. (click on the Build Phases tab, open up the Link Binary with Libraries, hit the + button, click Add Other and navigate+select the GHUnitIOS.framework directory on your filesystem)
6. Optional, but nice, move the GHUnitIOS.framework in your Tree to the Frameworks folder, to tidy things up

7. Set the -ObjC and -all_load in the other linker flags on the Tests Target (Select the Tests Target, select the Build Settings, search for other linker flags, and add the 2 flags)

8. Now you can delete some files which are not necessary, all the files in the Tests folder. (Note, Not! the Supporting Files folder as well, just the files!)
9. Delete the main.m file in the Supporting Files folder

10. In the Tests-Info.plist file (again, in the Supporting Files folder), clear out the Main nib file base name value

Time to create the GHUnit test runner, which will scan for our Unit Test Cases and run them.

11. Create an Objective-C class in the Tests folder named GHUnitIOSTestMain and make sure it is only! added to the Tests target

12. You can delete the GHUnitIOSTestMain.h file
13. Copy and paste source code from (http://github.com/gabriel/gh-unit/blob/master/Project-IPhone/GHUnitIOSTestMain.m) in your GHUnitIOSTestMain.m file

And now it’s time to create our own test case which will fail

14. Again create an Objective-C class in the Tests folder and as this is an example, name it ExampleTest, also make sure it is added to the Tests Target only!
15. You can delete the ExampleTest.h file
15. Copy and paste the next piece of code which is 99% copy/pasted from the example code from GHUnit (http://gabriel.github.com/gh-unit/_examples.html)

// For iOS
#import <GHUnitIOS/GHUnit.h>
// For Mac OS X
//#import <GHUnit/GHUnit.h>

@interface ExampleTest : GHTestCase { }
@end

@implementation ExampleTest

- (BOOL)shouldRunOnMainThread {
    // By default NO, but if you have a UI test or test dependent on running on the main thread return YES
    return NO;
}

- (void)setUpClass {
    // Run at start of all tests in the class
}

- (void)tearDownClass {
    // Run at end of all tests in the class
}

- (void)setUp {
    // Run before each test method
}

- (void)tearDown {
    // Run after each test method
}  

- (void)testFoo {
    NSString *a = @"foo";
    GHTestLog(@"I can log to the GHUnit test console: %@", a);

    // Assert a is not NULL, with no custom error description
    GHAssertNotNULL(a, nil);

    // Assert equal objects, add custom error description
    NSString *b = @"bar";
    GHAssertEqualObjects(a, b, @"A custom error message. a should be equal to: %@.", b);
}

- (void)testBar {
    GHAssertTrue(TRUE, @"Yes it worked");
}

@end

Right, ready to run!

16. Launch your Tests target (iTunes-like play button arrow) and run it against the simulator-scheme and your Unit test app should start

17. Now in the app in the simulator hit the blue run button and your tests will execute. (and testFoo will fail!, you can click on it to see why it failed)

And now it’s time to fix it…

18. Change the NSString *b = @"bar"; in the testFoo method to NSString *b = @"foo";
19. Run the Test app again and re-run the test and they should be green or in this case black, which means your tests are ok

Showing the power of GHUnit

20. Run the app against your own iOS device-scheme. (Select iOS-device-scheme and click the iTunes like run button)

Why i chose GHUnit over OCUnit

IMHO, this shows the real power of the GHUnit testing framework, not only does it run in the simulator but it also runs on your own iPhone, iPad, etc. Whereas OCUnit can only run as part of your build on your own machine, not on your phone and not in the simulator, this for me, is a big dealbreaker.
The closer you can get your unit tests running against a real-world environment the better it will be. Why? Because you are making use of the real devices processor (not the intel x86), real memory management, or lack there-of, the real API’s etc. If my Unit tests run on my phone, i’m 99.999% certain that the code under test will actually run on, yes, you guessed it, my phone.

There is of-course a downside to GHUnit, OCUnit (bundled with XCode) can be run automatically prior to you compiling your own app, this makes getting feedback about regression a lot faster, GHUnit is something which you have to run manually, in this case. But to solve that problem, or at least make it a whole lot better, we can use continuous integration aka build server to do the auto-running-of-unit-tests for us. There is a very nice blog post which compares various iOS unit testing frameworks.

So what is next? Well, the topic for my next blog and video in this series is hooking up a XCode project + GHUnit to Jenkins (or Hudson for the Oracle minded people out there).

Describe an exchange format

When the output XML producer does not have any mechanism to describe its node structure, its consumers have to choose a way to operate at its best. This can be done with an object graph equivalent to this XML output. Let’s see this with a delicious dessert example.

<recipe name="Pear compote" type="desert">
    <cooking duration="15">
    	<step optional="true">Leave aside a vanilla pod</step>
        <step>Peel and core pears</step>
        <step>Cut pears into slices</step>
        <step>Pour those slices in a casserole with water</step>
    </cooking>
    <menu>17-02-2011</menu>
    <menu>17-03-2011</menu>
</recipe>

JAXB identifies every node as an element with attributes. An element is a complex type with an element sequence (always first) and an attribute list. An element can have a simple text value if it has no sub-node. An attribute has only a text value. Every XML node is represented by an element and, with XJC, an object of the same name generated from the schema (XJC generation is explained in appendix).

Here is the XSD description of the previous XML:

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <xsd:element name="recipe">
        <xsd:complexType>
            <xsd:sequence>
                <xsd:element name="menu" type="xsd:string" maxOccurs="unbounded" />
                <xsd:element name="cooking" type="cooking" />
            </xsd:sequence>
            <xsd:attribute name="name" type="xsd:string" />
            <xsd:attribute name="type" type="xsd:string" />
        </xsd:complexType>
    </xsd:element>

    <xsd:complexType name="cooking">
        <xsd:sequence>
            <xsd:element name="step" type="step" maxOccurs="unbounded" />
        </xsd:sequence>
        <xsd:attribute name="duration" type="xsd:int" />
    </xsd:complexType>

    <xsd:complexType name="step" mixed="true">
        <xsd:attribute name="optional" type="xsd:boolean" />
    </xsd:complexType>
</xsd:schema>

These schema nodes are prefixed by « xsd » because their definition use a XML Namespace (xmlns) on the first line. Namespaces are a way to differentiate header declarations from one another, as a Java package would do.

Variables are typed following the w3c recommandation. When a primitive typed list is used (menu node), no new object is needed. However, an object is needed (step node) when a value and some attributes are both present. By default, it’s either one or another, but not both.

Here is the generated code from the schema:

@XmlAccessorType(XmlAccessType.FIELD)
@XmlRootElement(name = "recipe")
public class Recipe {
    protected List<String> menu;
    protected Cooking cooking;
    @XmlAttribute
    protected String name;
    @XmlAttribute
    protected String type;
}

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "cooking")
public class Cooking {
    protected List<Step> step;
    @XmlAttribute
    protected Integer duration;
}

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "step")
public class Step {
    @XmlValue
    protected String content;
    @XmlAttribute
    protected Boolean optional;
}

Only one class is annotated @XmlRootElement. It’s the only one (in this schema) that can be the first node of the output. Once the object graph is generated, 3 lines of code are enough to parse a XML string:

public class JaxbTest {
    @Test
    public void should_parse_recipe() throws JAXBException {
        URL xmlUrl = Resources.getResource("recipe.xml");
        Recipe recipe = parse(xmlUrl, Recipe.class);
        assertEquals(Integer.valueOf(15), recipe.getCooking().getDuration());
    }

    private <T> T parse(URL url, Class<T> clazz) throws JAXBException {
        Unmarshaller unmarsh = JAXBContext.newInstance(clazz).createUnmarshaller();
        return clazz.cast(unmarsh.unmarshal(url));
    }
}

Compared to low level API, no conversion is needed: declaring types in the schema is enough. In case of a conversion error (a string non convertible to a type) an appropriate IllegalArgumentException is thrown. When a node is missing, the corresponding variables are null.

Refine the exchange format

Once the data binding is successful, more control is needed:

1. restrict a value to a range of possibilities;
2. manipulate java date instead of XMLGregorianCalendar used by default by JAXB;
3. use a different class name to represent an element;
4. use object inheritance between elements;
5. annotate existing classes manually.

Restrict a value to a range of possibilities

Restrict attribute « type » to « starter, main, desert » can be done this way:

<xsd:element name="recipe">
    <xsd:attribute name="type" type="course" />
</xsd:element>

<xsd:simpleType name="course">
    <xsd:restriction base="xsd:string">
        <xsd:enumeration value="starter"></xsd:enumeration>
        <xsd:enumeration value="main"></xsd:enumeration>
        <xsd:enumeration value="desert"></xsd:enumeration>
    </xsd:restriction>
</xsd:simpleType>

Which gives us, when generated:

@XmlAccessorType(XmlAccessType.FIELD)
@XmlRootElement(name = "recipe")
public class Recipe {
    @XmlAttribute
    protected Course type;
}

@XmlEnum
public enum Course {
    @XmlEnumValue("starter")
    STARTER("starter"),
    @XmlEnumValue("main")
    MAIN("main"),
    @XmlEnumValue("desert")
    DESERT("desert");
    private final String value;
}

When binding happens, if the value does not match any of the listed possibilities, a null value is users by default. By convention, JAXB uses a null value when a problem occurs and throws an exception for typing problems.

Use java dates

To use a simple date instead of a XMLGregorianCalendar used by default in JAXB, a converter is required. Be careful, a new namespace has been declared on the root element. It allows us to redefine the date standard type.

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
            xmlns:jxb="http://java.sun.com/xml/ns/jaxb"
            jxb:version="2.0">

    <xsd:annotation><xsd:appinfo>
        <jxb:globalBindings>
            <jxb:javaType name="java.util.Date" xmlType="xsd:dateTime"
                          parseMethod="com.xebia.jaxb.JaxbDateConverter.parseDate" />
        </jxb:globalBindings>
    </xsd:appinfo></xsd:annotation>

    <xsd:element name="menu" type="xsd:dateTime" />
</xsd:schema>

The converter must adhere to JAXB conventions, if the value does not fit, no exception is thrown but a null value is returned instead.

public class JaxbDateConverter {

    public static Date parseDate(String s) {
        DateFormat formatter = new SimpleDateFormat("dd-MM-yyyy");
        try {
            return formatter.parse(s);
        } catch (ParseException e) {
            return null;
        }
    }
}

When generating classes, JAXB creates an adapter class binded to the static methods of the converter.

@XmlAccessorType(XmlAccessType.FIELD)
@XmlRootElement(name = "recipe")
public class Recipe {
    @XmlElement(type = String.class)
    @XmlJavaTypeAdapter(Adapter1.class)
    @XmlSchemaType(name = "dateTime")
    protected List<Date> menu;

Use a different class name to represent element

To name a class deferred to the represented element, modifying the schema in the following manner is enough (be careful to add both namespaces):

<xsd:element name="menu" type="xsd:dateTime" />
<xsd:annotation><xsd:appinfo>
    <jxb:class name="meal" />
</xsd:appinfo></xsd:annotation>

Use object inheritance

Using object inheritance is easy (generated objects will inherit one another, naturally):

<xsd:complexType name="menuxl">
    <xsd:complexContent>
        <xsd:extension base="menu" />
    </xsd:complexContent>
    <xsd:attribute name="cook" type="xsd:string" />
</xsd:complexType>

Annotate existing classes manually

So far, we have used only generated classes thanks to the schema. But we can also manually annotate a class to bind the XML. A schema can even be generated from the annotated source code.

Manual annotations give more control over objects, their type and the opportunity to use already existing ones that were used for other purposes (adding a non bindable attribute can be done with @XmlTransient).

Once again we have to adhere to JAXB convention. In its generated code JAXB overloads the getters to return empty lists instead of null. It is advised to do the same.

public List<String> getMenu() {
    if (menu == null) {
        menu = new ArrayList<String>();
    }
    return menu;
}

Epilogue: exchange format on the server

Until now, we have assumed that no mechanism describing nodes organisation was provided from producer to consumers; this was resolved eventually by exchange formats. It is better, when possible, to remove that constraint by giving consumers a schema with data.

To do that, it is wise to opt for a contract first development style. To limit coupling between the schema and its implementation, it is advised to generate objects on the server side rather than annotating them. To bear within the limits of a XSD schema types and organisation, guarantees the compatibility of the output to any kind of clients (more particularly to non java ones). The Spring reference documentation champions this idea.

Appendix: tools

In order to run the samples, you will need the following : JAXB maven dependency and the associated generation plugin. To ease testing, we also provide a simple plugin configuration (generation can be launched using mvn jaxb2:xjc):

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>jaxb2-maven-plugin</artifactId>
    <configuration>
    	<outputDirectory>${basedir}/src/main/java</outputDirectory>
    	<schemaDirectory>${basedir}/src/main/resources/xsd</schemaDirectory>
    	<packageName>com.xebia.jaxb.generated</packageName>
    	<schemaFiles>schema.xsd</schemaFiles>
    </configuration>
</plugin>

To dig further on data binding with JAXB, a well-stocked documentation is available here.

if (threadContextLoader != null) {
  loadFromContextLoader(className);
} else if (systemLoader != null) {
  loadFromSystemLoader(className);
} else {
  loadFromBootstrapLoader(className);
}

and realized that it’s not immediately evident how to do this in a uniform way. But actually, it’s quite simple…getting a ClassLoader object for the bootstrap loader is just a couple of lines of code.

So what’s the problem?

Well, if you’re happy with a literal implementation of the pseudo-code above, there obviously isn’t a problem.

ClassLoader loader = Thread.currentThread().getContextClassLoader();
if (loader != null) {
  return loader.loadClass(className);
}

loader = ClassLoader.getSystemClassLoader();
if (loader != null) {
  return loader.loadClass(className);
}

// null is the bootstrap loader
return Class.forName(className, true, null);

But this is very procedural code¹, and it would already be much more OO if we could do something like:

getLoader().loadClass(className);

ClassLoader getLoader() {
  ClassLoader loader = Thread.currentThread().getContextClassLoader();
  if (loader != null) {
    return loader;
  }

  loader = ClassLoader.getSystemClassLoader();
  if (loader != null) {
    return loader;
  }

  return ...???????
}

The problem is that there simply isn’t a ClassLoader.getBootstrapClassLoader() or equivalent method. You can try to walk up the system loader’s parent hierarchy, of course, but what you’re likely to find is that the bootstrap loader will be represented by null – not very useful in our circumstances.

Writing BootstrapClassLoader

So what we want is to come up with a ClassLoader implementation that delegates straight to the bootstrap loader. One option would be to override loadClass to simply call Class.forName with the above arguments, but there’s actually an even easier² way:

class BootstrapClassLoader extends ClassLoader {
    BootstrapClassLoader() {
        /*
         * The default classloader implementation will use the bootstrap loader
         * if it finds a null parent.
         */
        super(null);
    }
}

Now we can finish our getLoader method:

// or simply "new ClassLoader(null) {};"
final ClassLoader BOOTSTRAP_LOADER = new BootstrapClassLoader();

ClassLoader getLoader() {
  ClassLoader loader = Thread.currentThread().getContextClassLoader();
  if (loader != null) {
    return loader;
  }

  loader = ClassLoader.getSystemClassLoader();
  if (loader != null) {
    return loader;
  }

  return BOOTSTRAP_LOADER;
}

Unless we’re being really performance-conscious, we can even turn this into the almost functional³:

ClassLoader getLoader() {
  return Iterables.find(asList(Thread.currentThread().getContextClassLoader(),
      ClassLoader.getSystemClassLoader(), BOOTSTRAP_LOADER), Predicates.notNull());
}

There’s a demo snippet on PasteBin here.

At XebiaLabs, however, we’re not just the implementors of our plugin SPI. We also write the deployment engine that consumes these plugins. And unfortunately, there isn’t much out there to help you read, load and verify services. Hence MultiSPI.

What about ServiceLoader?

Evidently, the engineers at SunOracle – as prominent SPI consumers – shared this frustration. In Java 6, they introduced ServiceLoader to at least make “vanilla” SPI consumption easier. MultiSPI isn’t just a fully-compatible alternative for those still working with Java 5, however. It also supports a number of common additional use cases:

1. Inspecting the list of class names before they are loaded, for instance to remove or replace some items (think user-definable overrides), to use different class loaders for different packages (think OSGi) etc.
2. Inspecting the classes before they are instantiated, for instance to check for required annotations or implemented interfaces, to delay creation of instances etc.
3. Using non-vanilla instantiation strategies, for instance to support factory methods, or in conjunction with dependency injection etc.

Providing services in 2011

More importantly, though, MultiSPI doesn’t just support reading service implementations from META-INF/services files. Because let’s face it, if you’re designing a component-provider mechanism today, it probably won’t be based on the vanilla SPI specification.

MultiSPI supports annotation- and manifest-scanning out-of-the-box, but you can add providers for any scheme of your choice, be it XML descriptors, system properties, environment variables or whatever. Of course, you can also mix and match different providers, especially useful if (as in our case with Labs) you are supporting multiple versions of an SPI.

SPIs and dependency injection

Ultimately, of course, SPIs are just service factories, and if you’re working within a dependency injection context you probably don’t want to invoke an SPI loader explicitly, you just want the resulting services to “be available” for injection into your business logic.
Indeed, Spring added a ServiceLoader-backed factory in 2.5, and of course you can do the same with MultiSPI, whether you want the implementation class names, the classes or the instances themselves. There’s Spring and Guice examples; here’s the Spring context that prepares a set of class names, classes and instances, for example:

<bean id="multiSpi" class="com.qrmedia.commons.multispi.MultiSpi">
  <constructor-arg>
    <set>
      <bean class="com.qrmedia.commons.multispi.provider.MetaInfServicesProvider" />
      <bean class="com.qrmedia.commons.multispi.provider.AnnotationScanningProvider">
        <constructor-arg value="uk.gov.mi6.LicenseToKill" />
        <constructor-arg value="uk.gov" />
      </bean>
    </set>
  </constructor-arg>
</bean>

<bean id="agentNames" factory-bean="multiSpi" factory-method="findImplementationNames">
  <constructor-arg value="uk.gov.mi6.Agent" />
</bean>

<bean id="agentClasses" factory-bean="multiSpi" factory-method="findImplementations">
  <constructor-arg value="uk.gov.mi6.Agent" />
</bean>

<bean id="agents" factory-bean="multiSpi" factory-method="loadImplementations">
  <constructor-arg value="uk.gov.mi6.Agent" />
</bean>

Of course you can use many different MultiSpi instances, with different provider configurations, in your application at the same time.

Using MultiSPI

To use MultiSPI, just add the following dependency and repositories to your POM:

<dependency>
  <groupId>com.qrmedia.commons</groupId>
  <artifactId>multi-spi</artifactId>
  <version>1.0-SNAPSHOT</version>
</dependency> 

<repository>
  <id>qrmedia-releases</id>
  <url>http://aphillips.googlecode.com/svn/maven-repository/releases</url>
  <releases>
    <enabled>true</enabled>
  </releases>
  <snapshots>
    <enabled>false</enabled>
  </snapshots>
</repository>
<repository>
  <id>qrmedia-snapshots</id>
  <url>http://aphillips.googlecode.com/svn/maven-repository/snapshots</url>
  <releases>
    <enabled>false</enabled>
  </releases>
  <snapshots>
    <enabled>true</enabled>
  </snapshots>
</repository>

If you think you might run into conflicting dependencies and want something that just works without having to deal with <excludes>, you can use:

<dependency>
  <groupId>com.qrmedia.commons</groupId>
  <artifactId>multi-spi</artifactId>
  <version>1.0-SNAPSHOT</version>
  <classifier>jar-with-dependencies</classifier>
</dependency>

instead.

The Maven site and project reports are here.