I was so fortunate to be invited to present at the 10th edition of the QA&TEST. The conference focuses on QA and testing for embedded systems, an area I know little about. Though, when I first started testing in the late nineties, I was so lucky to work with a gentleman that had lost his job at Fokker (the airplane manufacturer had just gone bankrupt). At Fokker he used to test instruments used in the airplanes, like altitude meters, speedometers and other fine machinery. He knew very little about computers and was struggling with was to me basic stuff (being a rookie myself), but he was also one of the most structural and detail critical people I ever worked with. Besides loads of test techniques and responsibility for my work, what I learned from that experience is that testing is testing and the same rules apply in most fields. So after contemplating the above I happily accepted the invitation. What I experienced during the conference is that testing embedded systems is about software testing and lucky for me, in this field Agile methodologies and test automation can be applied just the same.

Automated Reliability Testing via hardware interfaces by Bryan Bakker

How test automation can successful be applied was highlighted by fellow speaker Bryan Bakker who did a great presentation of a case study on how his team. The team was requested to improve the reliability of a system and call it in an act of rebellion kept on increasing test automation rather than adding new functionality. This resulted in (if I remember the correct figures) a savior of over 1.2 million euro’s on damage that bugs would have otherwise caused in production. The result was so spectacular that they were granted extra budget to continue with their work, but also to apply the approach in other projects. Interesting takeaway from his presentation, that seems a bit more specific for embedded systems but might apply in other area’s as well, was a smart test-scheduler they had introduced. The product under test was a medical x-ray device that under circumstances would get an overheated engine, it took hours for it to cool down again which made it impossible to continue the test run. Whenever this was detected they would automatically switch to other types of tests, that were less dependent on the engine and could run until the engine had cooled down again. A simple but effective time-saver coming from a pragmatic mind.

Continuous Quality Improvement using Root Cause Analysis by Ben Linders

A session I unfortunately did not actually attend to, but that I had an interesting discussion about with the presenter, was the conference closing presentation by Ben Linders called “Continuous Quality Improvement using Root Cause Analysis”. He claims that a team can accurately predict the number of bugs it is going to make during a sprint and he has developed a method to help reduce this number using root cause analysis. I found this a fascinating and somewhat controversial idea, I have yet to meet a developer (specially the ones in Agile projects) that can admit let alone predict they’re making bugs. But as I understood it works in a similar way as predicting velocity, you get more accurate as you go, using historic data from previous sprints.

Runaway Test Automation Projects by Michael Stahl

The presentation of Michael Stahl titled “Runaway Test Automation Projects” seemed less relevant to Agile environments (at least the once I worked in), he however pointed out lots of valid risks that exist when doing test automation. Main take away for the audience in my opinion was the point that test automation should be treated as “regular” software, creating unit tests, applying quality standards and using version control. Things that in my experience sound like common sense, but never the less very true.

Pushing the Boundaries of User Experience by Julien Harty

The presentation that was probably of the most added value to me was called “Pushing the Boundaries of User Experience”, by Ebay’s Julien Harty. He had an enlightening story on automated user experience testing. With crawlers like Crawljax analysis of the dynamic (Ajax) behavior of a website can be done. With static analysis accessibility issues can be found, stuff that can be very important if you want to comply to WCAG guidelines, which hardly ever gets proper attention. Accessibility is often seen as something to facilitate a minority, but from a development point of view it helps to improve testability of the product What makes accessibility testing feasible from a business perspective is that it actually helps to improve search engine optimization, which increases the visibility of the site in search engines.
He explained how at Ebay they run automated tests for improving usability and accessibility, but also for finding layout issues and browser dependencies in a cost effective way.
Layout bugs can be found in an extremely simple but effective way using FightingLayoutBugs. Let me describe this in a little more detail.
How does this work? First you need to know which pixels belong to text, second you need to know which pixels belong to a horizontal or vertical edge, now if text pixels and edge pixels overlap you have a layout bug. Sounds pretty simple and it actually is!
How is text detected? All text of a page is set to a white font color and a snapshot is taken. All the text on a page is set to black and a snapshot is taken. Now when the two are compared, all different pixels are probably text.
How are horizontal and vertical lines detected? First set all the text on the page to transparent (btw setting the text color is done using jQuery), so now there are only graphics, take a screenshot. Now it’s identified which sequences of pixels have a certain minimal length and the same or very similar color, only those that have a high contrast to the left or to the right are selected. The same approach applies for horizontal lines. Now compare the outcome with the identified text. When text and lines overlap there’s a layout bug (which happens to be automatically reported).

During a life demo Julian ran some of these tools against the conference website. With web-accessibility-testing pointed out how bad the accessibility was for people that rely on tabbing to navigate through the screen. Static analysis of the site pointed out the page contained url’s with duplicate or no alt texts. Stuff that seems of minor severity but for disabled people essential. He concluded by revealing a security issue that caused some hilarity. He gave away the most lucrative reduction code for registration. It turned out all reduction codes for the conference were hardcoded in the Javascript in the page source.

All together it has been a valuable learning experience, beside the fact that a visit to Bilbao alone is definitely worth the trip!

The Deployit plugin for the new 3.3 release of Atlassian Bamboo adds the enterprise-scale deployment capabilities of XebiaLabs Deployit to Bamboo. This allows you to speed up your development process by adding automated deployment to your continuous integration setup and make the the first step towards continuous deployment and continuous delivery. Instead of deployment being a bottleneck to your development process, it will be be an integrated part of it. You can test your application on the target platform as soon as possible, find any platform incompatibility and deployment issues early on, and, when it’s time to deploy to the production environment, your deployment will be quick and reliable.

read more

I started out using the SoapUI REST functionality, but that still lacks a good implementation for my purpose of reacting on REST calls. I ended up with a 10-minute build-your-own REST mock using the Play framework. This blogs describes how this was accomplished.

The Play framework crossed my path earlier. It is a very simple yet powerful application framework for web applications. It is almost trivial to set it up for mocking REST services. First make sure you download and unpack the framework. Then execute the following commands to create a new project:

    play new RestMockServer
    play eclipsify RestMockServer
    play run RestMockServer

The project is running and now ready to be worked on in eclipse (see ‘play help’ for other IDEs). Changes in code and configuration will be picked up by the framework automatically. Just point your browser to http://localhost:9000/ to see the application running.

The next step is to add the REST URLs to the routing table. Let us say I ‘m creating a system for storing photos containing meta data and the image URL. Open the file ‘conf/routes’ and add the following lines before the ‘Catch all’ entry:

    PUT    /photo            PhotoController.create
    POST   /photo            PhotoController.update
    GET    /photo/{id}       PhotoController.read
    DELETE /photo/{id}       PhotoController.delete

This defines the CRUD actions on a photo object. First in line is the http method on which the framework should react. The second part is the URL with optional parameters which will be mapped onto the method indicated in the first part.

The following action is to define a controller handling the calls. The implementation uses a very simple response technique based on text. The mapping of the id to the actual response is left to the reader to implement. You could for example read a file based on the URL parameter and return the content of it. For now, we just return the identifier specified in the URL.

Save the file below as ‘PhotoController.java’ and call http://localhost:9000/photo/123 and see your favorite number showing on screen. Note that the name between curly braces should match the name of the parameter. The mapping on type will be done auto-magically by the framework.

package controllers;
import java.util.Random;
import com.google.gson.JsonObject;
import play.mvc.Controller;

public class PhotoController extends Controller {
    public static void read(int id) {
        try {
            response.contentType = "application/json";
            renderText(id);
        } catch (Exception e) {
            notFound(e.getMessage());
        }
    }
}

The next step is to implement the other methods. Since I’m using JSON as data format, I have to add a specific binder for the JsonObject type, used both by the create and update method. The ‘ok’ just returns a 200 status code. Upon creation of an object I need the identifier of the newly stored data object, so for test purposes I return a random integer.

Add the following method implementations to PhotoController.java:

    public static void delete(int id) {
        ok();
    }

    public static void update(JsonObject body) {
        ok();
    }

    public static void create(JsonObject body) {
        response.contentType = "application/json";
        renderText("\"id\":"+Math.abs(new Random().nextInt()));
    }

The binder is also automatically found by the framework. I’ve created a new package ‘util’ in which I placed the following class:

package util;
import java.lang.annotation.*;
import java.lang.reflect.Type;
import com.google.gson.*;
import play.data.binding.*;

@Global
public class GsonBinder implements TypeBinder<jsonobject> {
    @Override
    public Object bind(String name, Annotation[] a, String value, Class c, Type t) throws Exception
    {
        return new JsonParser().parse(value);
    }
}

And that is basically all to set up your mock server for REST services using the Play framework! From your favorite REST client, call the URL: http://localhost:9000/photo with a PUT request and set the entity of the request to some JSON content.

Happy development!

Robotium.org

What is Robotium & what does it do?

Robotium is a test framework created to make it easy to write powerful and robust automatic black-box test cases for Android applications. With the support of Robotium, test case developers can write function, system and acceptance test scenarios, spanning multiple Android activities.

In essence, a Robotium test is a subclass of junit.framework.TestCase (an ActivityInstrumentationTestCase2, in this “case”) in which, by use of the Robotium library, a Solo object is created which allows easy access to the views in your activities.
Through the solo object, you can set values in input fields, click on buttons and get results from other UI components. Methods of JUnits Assert class can then be used to check those results.

How to set up Robotium?

After creating an Android Test project in the usual way (I will assume you have done that before), you need to add the robotium JAR to the Libraries on the projects Build Path.

How to use Robotium?

In the setUp method of your testing class, you can create a Solo object and specify the Activity to be started.
The Solo object can then be used in your testXXX methods to get and set values in/from UI components and click buttons and such.
Many of Solos methods to access UI components take an int representing the index of the component in the UI. This can lead to brittle code so using getView (and the id defined in the layout xml) is a much safer alternative.
Do keep in mind that changing values and performing clicks is not allowed from outside of the thread that originally created the views, these things must still be done through Solos methods such as setXXX, EnterText and clickOnXXX.

The values to be asserted can then be retrieved through Solos methods isXXX, getCurrentXXX, searchText and others.
Solos finalize method should be called from the tests tearDown method.

An example:

package xebia.addapptesting;

import xebia.addapp.AddApp;
import xebia.addapp.R;
import android.test.ActivityInstrumentationTestCase2;
import android.widget.EditText;

import com.jayway.android.robotium.solo.Solo;

public class AddTest extends ActivityInstrumentationTestCase2<AddApp> {

    private Solo solo;

    public AddTest() {
        super("xebia.addapp", AddApp.class);
    }

    @Override
    protected void setUp() throws Exception {
        solo = new Solo(getInstrumentation(), getActivity());
    }

    public void testAdd() {
        int one = 1;
        int other = 2;

        solo.enterText((EditText) solo.getView(R.id.one), Integer.toString(one));
        solo.enterText((EditText) solo.getView(R.id.other), Integer.toString(other));
        solo.clickOnView(solo.getView(R.id.calcbutton));

        assertTrue(solo.searchText(Integer.toString(one + other)));
    }

    @Override
    public void tearDown() throws Exception {
        try {
            solo.finalize();
        } catch (Throwable e) {
            e.printStackTrace();
        }
        getActivity().finish();
        super.tearDown();
    }
}

When running the test case on the emulator or an actual device, you will see the values being entered into the UI components or see them being clicked as if you were doing it yourself. After the tests are completed, the JUnit view in Eclipse will show which tests have failed and passed.

Conclusion

I think it looks pretty cool to see the app being run and the values being entered. This kind black-box behavior testing on the emulator is much more realistic than just a unit test.

Stay tuned for my next post on a surprise topic, soon!

Driving back home, I decided not to give in so easily and to try and use some decent frameworks to overcome this challenge.

JavaScript testing: YUI

I turned to finding a testing framework first. Sure enough, most javascript testing frameworks focus on fancy UI stuff like asynchronous testing, integration testing or automated multi-browser testing. I was looking for none of that, but a simple xUnit like framework, that I could run in a browser to test the functionality of a number of JavaScript classes. After a few minutes I found YUI which seemed to fit the bill. Setting up was easy enough:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Testing TagCounter with YUI</title>
</head>
  <body class="yui-skin-sam">
    <div id="yui-main"><div id="testReport"></div></div>
    <script type="text/javascript" src="http://yui.yahooapis.com/3.0.0/build/yui/yui-min.js"></script>
    <script type="text/javascript" src="js_cols/base.js"></script>
    <script type="text/javascript" src="TagCounter.js"></script>
    <script type="text/javascript" src="TagCounter_Test.js"></script>
  </body>
 </html>

This static html file is all that is neede to run my tests. The tests themselves are written in the TagCounter_Test.js file:

/**
 * Unit Tests for TagCounter
 */
YUI({
  combine: true,
  timeout: 10000
}).use("node", "console", "test", function (Y) {
  var assert = Y.Assert;

  var TagCounterTestCase = new Y.Test.Case({
    // test case name - if not provided, one is generated
    name: "TagCounter Tests",

    "test should increment count": function () {
      var tc = new TagCounter();
      tc.add("my-tag");
      assert.areEqual(1, tc.map().get("my-tag"));
      tc.add("my-tag");
      assert.areEqual(2, tc.map().get("my-tag"));
    }
  });

  //create the console
  var r = new Y.Console({
    newestOnTop : false,
    style: 'block'
  });

  r.render("#testReport");
  Y.Test.Runner.add(TagCounterTestCase);
  Y.Test.Runner.run();
});

Opening the HTML file in FireFox shows the picture below.

It seems like I’m ready to implement the TagCounter TDD style!

Implementation: js_cols

To make this simple test pass, all I need is a simple HashMap with some basic features, like contains, get and remove, but… this is not java! After some googling, I found js_cols, a neat JavaScript collections framework, that basically had all the support that I wanted, including a LinkedHashMap implementation that can be used to implement a LRU cache, just like good old java! The LRU feature is going to help me to remove “old” tags, keeping the memory required to process the tags within reasonable limits.

To make the first test succeed, only a few lines of code are needed:

js_cols.require('js_cols.LinkedHashMap');
function TagCounter(maxSize) {
	this.tagCounts = new js_cols.LinkedHashMap(maxSize, true);
};
TagCounter.prototype.add = function (tag) {
	if (this.tagCounts.contains(tag)) {
		this.tagCounts.insert(tag, this.tagCounts.get(tag) + 1);
	} else {
		this.tagCounts.insert(tag,1);
	}
};
TagCounter.prototype.map = function () {
	return this.tagCounts;
};

In the end, however, we want the map method to return only the most popular tags, while our counter should keep track of a longer list. To implement this, we need two steps:

1. First we need to sort the tags by their number of appearances in descending order, keeping in mind that multiple tags can have the same number.
2. Then we need to take the top x most popular tags.

The test:

...
    "test should return limited sized map": function () {
    	var tc = new TagCounter(3);
    	tc.add("my-old-tag");
    	tc.add("my-old-tag");
    	tc.add("my-old-tag");
    	assertSame(tc.map(2), [["my-old-tag", 3]]);
    	tc.add("my-first-tag");
    	tc.add("my-first-tag");
    	assertSame(tc.map(2), [["my-old-tag", 3], ["my-first-tag", 2]]);
    	tc.add("my-second-tag");
    	assertSame(tc.map(2), [["my-old-tag", 3], ["my-first-tag", 2]]);
    	tc.add("my-third-tag");
    	assertSame(tc.map(2), [["my-first-tag", 2], ["my-third-tag", 1]]);
    }
  });

  function assertSame(map, expected) {
	  var keys = map.getKeys();
	  assert.areEqual(expected.length, keys.length);
	  for(var i=0,len=expected.length; value=expected[i], i<len; i++) {
		  assert.areEqual(value[0], keys[i]);
		  assert.areEqual(value[1], map.get(value[0]));
	  }
  };
...

In the test we now add a method assertSame to make it easier to express our expectations of the map method: The map should contain the expected number of tags with their counts in the right order. Therefor the expectation is a list of [tag, count]-tuples.

We can again use the collection library classes to implement this: The RedBlackMultiMap class implements a key-sorted map that allows multiple values for the same key.

TagCounter.prototype.map = function (opt_size) {
	var orderedMap = new js_cols.RedBlackMultiMap(function (a,b) {return b-a;});
	this.tagCounts.forEach(orderedMap.insert, orderedMap);
	var result = new js_cols.LinkedHashMap();
	if (opt_size) {
		var i = 0;
		orderedMap.forEach(function (value, key) {
			if (i < opt_size) {
				result.insert(value, key);
			}
			i++;
		});
	} else {
		orderedMap.forEach(result.insert, result);
	}
	return result;
};

On line 2 the RedBlackMultiMap constructor is used with a comparator function that will invert the natural ordering of numbers. This way we sort in descending order.
On line 3 all items of the tagCount map are inserted into the orderedMap reversing the key-value relation. The fact that the forEach method take a function that has the value as the first and the key as the second argument (which would usually be confusing) now appears a handy feature!
On lines 4 to 7, the key-value relation is again reversed, making sure to only take the top opt_size items from the map (when set).

Conclusion

Looking at the simplicity of the code and the tests, I would say that it is definitely possible to write well tested and documented JavaScript algorithms and data structures. I completely agree with my colleague, however, that such code is not often found in your run-of-the-mill web application project. Maybe it is time to start treating the JavaScript code in our projects more as Java code and make sure that it is properly structured and tested.

Around 2006 several initiatives to create open source performance monitoring tools for java production environments started to appear.

This was mainly because AOP (Aspect Oriented Programming), the technology used in most of these products, was getting attention in the market and there were quite some developments in that area at the time.

I am interested to see how the open source community around these kind of products is evolving. The outcome is quite surprising…

The main reason I am interested in tools in this area is to be able to pinpoint problems in an Acceptance or Production environments very quickly, because this saves (big) enterprises a lot of money.

The main reason I am interested in open source tools is to see if we are able to create a complete open source java hosting stack which is enterprise production ready.

Without adding a lot of extra license and/or support costs we don’t have to consolidate IT anymore. We will be able to do application hosting for each specific business unit separately and focus on the continuous improvement for that business unit alone. Without big boundaries between the business unit and IT. This kind of boundaries are most times created because of cost efficiency. Think about consolidation and architecture guidelines. Not having to think about licencing costs will help us in helping enterprises to become agile.

During a knowledge exchange session (XKE) at Xebia we first created a requirements list to select tools in the same area to be able to compare them.

The requirements so far:

1. The tool should measure production and acceptance environments with low (<= 5%) resource usage overhead.
2. The tool should be able to pinpoint (potential) problems in the running applications as quickly as possible.
3. The problems that should be detected: high resource consumption of particular code blocks, memory leaks and the location of the leak, blocking code blocks which block other threads, slow calls to back ends (i.e. web services, databases).
4. The environments the tools should run in are the following application servers: IBM Websphere, Oracle Weblogic, JBoss, Tomcat, Glassfish.
5. The problems should be made visible in such a way that the root cause can be found as quickly as possible. The most valuable will be to see complete business transaction flows over all JVM’s, with real-time flow states and measurement readings.
6. The tool should be easy to install, i.g. just use a simple agent that connects it to a server. The goal should be to leave the code base unchanged.

Before the session I did some homework and I searched the internet for these kind of monitoring tools. During our knowledge exchange session we had a brainstorm session and came to the following list of Open Source tools:

This list is as complete as we were able to come up with, if you know more and eventual better open source tools please let us know!

After evaluating all products on the lists (just a paper evaluation) we only see Glassbox and Infrared as a real Open Source solution in this area. But both (at least Glassbox) are not actively developed anymore. Of course because both products are open source it will be possible to restart development on one of the products.

It is interesting why both projects are not active anymore. I spent some time to find out why. This is what I found out:

What happened to Glassbox; the first developer mentioned is the VP of Engineering at Glassbox David Pickering. He has left the Glassbox company in 2009. And is now VP of Engineering at iWin, a company with a different focus. The second one mentioned is Ron Bodkin, he left Glassbox in 2007 and was the founder of the Glassbox company, he became VP of Engineering at Quantcast, so also his focus has moved. This looks like the main reason there is no active development on Glassbox anymore.

What happened to Infrared; the first developer mentioned on the project member list at SourceForge is Binil Thomas, he worked for Tavant Technologies at the time Infrared was created. Other members on the same list also worked for Tavant in that time. Most are now working for different companies. It looks that Tavant Technologies was no longer interested in this space. The last 2.6RC1 release from Infrared was created in 2010, at a time Binil was working for Ronin Capital. Since 2011 he is working for AppDynamics, this is a commercial vendor in this area and looks like the main reason the 2.6RC1 release never came to a full release, but this is something I am guessing of course.

What’s next

We will test both Infrared and Glassbox to see if one of them is still a good choice. We will start with Infrared because this one has been the most active project in the past and it has been maintained the longest, until 2010. After this we will see if one of these products can be a competitor for the commercial products in this area.

Many thanks to my colleagues Adriaan Thomes and Sander Hautvast for helping me with the brainstorm session.

Xebium creates a simple way to use Selenium IDE (low learning curve) and FitNesse (ease of maintenance) to it’s fullest when it comes to maintaining a web application test suites.

Xebium is using the same keywords as Selenium IDE. This has the huge advantage that no person should learn another DSL. Since tests are stated this way, they can be copied between Selenium IDE and FitNesse without a hassle (the FitNesse formatter for Selenium IDE is rather trivial). And to be honest: as long as there are XPath and Regular Expressions in the code, it makes no sense to come up with a substitute for verifyText.

So now it is possible to move our Selenium IDE test case in a FitNesse page. Nice. It’s easier to set up tests, since FitNesse is quite flexible in that. On the other hand, just playing back you recorded tests is not the most flexible way to work. There can be a huge maintenance burden on this. It would be a shame not to use the full power of FitNesse in this case. Scenario tables for example can make a nice abstraction where some common action, like logging in, only have to be defined (and maintained) in one place. By moving the login code out of the test and into a scenario, we add another level of indirection. The tests themselves become more readable, as they only state the business requirement and the low level details are directly editable by the tester. This is a lot more flexible than creating Selenium fixtures for every scenario.

| log in user | user | with password | password |
| open | http://example.com/login |
| type | name-field | @user |
| type | password-field | @password |
| click | login |

and in your tests, instead of repeating the above, just mention:

| script |
| log in user | john | with password | secret |
| ensure | do | verifyText | on | #userid | with | John |

Sure, we’re back at creating our own DSL, but we do it 100% in FitNesse. Selenium commands can be added where needed (e.g. assertions). No Java (fixture) code, no developers required to change one line because the templates have changed and, foremost, detailed feedback.

But FitNesse can help out even more. Consider data driven testing. Selenium has (by default) no option for that. Using the same scenario table fixture it’s easy to test logins for a bunch of users.

But we claimed you can copy stuff from FitNesse to Selenium IDE as well and that’s getting harder when you move everything inro scenarios. To overcome this issue the formatter in Selenium IDE can parse tab separated data as well. So after a test has run, the output as provided by FitNesse can be copied back. This is a useful aid for analyzing failed tests.

I tend to make my Xebium commands easely available. First of all the fixture needs to be imported. After that the fixture can be assigned as library:

| import |
| com.xebia.incubator.xebium |

| library |
| selenium driver fixture |

A note for Windows users, though. When performing a lot of tests, Windows tends to run out of sockets. There is a way to configure your machine to work around those silly limitations (at your own risk!).

When testing web interfaces, it’s convenient to use an intuitive tool like Selenium IDE, it’s easy to use and can be used by non-technical people, but it is solely meant for record and playback of test-scripts. One of its limitations is that it misses sufficient options for documenting and managing tests. Furthermore it misses an interface with the backend of the system under test (SUT), to setup preconditions for a test or for instance to manipulate or read from a database.
Fitnesse is a great tool to do just that, it has the Wiki to manage tests and it by default has a setup and teardown mechanism, it’s easy to add non invasive testfixtures to interface directly with your SUT. The downside is that it is incapable of doing webtests.

We now have the glue that combines the two, it’s called Xebium!

On several projects I’ve tried to find a way to combine the two, because both tools are free of use and both have their individual charm, but it seems combined they would master practically any situation. As a tester I miss the development skills to come up with a satisfying solution and I never convinced my team members to help me out in a satisfactory manner.

At our company we recently started app-incubator, a new form of knowledge exchange, in the form of mini projects, anyone can pitch a project idea and when enough colleagues sponsor the project, we can spend up to three days collaborating on it.  My pitch caught fire and with about five people we brainstormed on the solution. For me a prerequisite was that it should be possible to use the test scripts two directional, so record them with Selenium IDE, run and edit them in Fitnesse and playback from Selenium IDE again. During this first project session we came up with additional requirements, it should support data-driven testing and variable substitution. We wanted to keep it as simple as possible, no new DSL’s should be introduced and it should not require conversions between the two products. We explored some existing solutions such as Fitnium and Selenesse but came to the conclusion these (at the time) were fully based on their own DSL’s and did not support exchanging files between Selenium IDE and Fitnesse. So we decided to created something far more simple.

The Result

After several attempts we discovered that Webdriver (Selenium 2.0) would be the best engine for running our tests. It seems more future proof, turned out to be a lot more stable than Selenium Testrunner and its commands are more compatible with Selenium IDE than Selenium RC.
We introduced a Selenium IDE formatter which output can be directly copied into the Wiki. We created a Fitnesse fixture that is capable of running the pre-formatted scripts without additional interpretations of the Selenium commands.

The additional features we had in mind, the data-driven testing and variable substitution turned out to be standard Fitnesse features. What we did not realize when we started was that Fitnesse would introduce a mechanism which, in my opinion, added mind-blowing power to project: Scenario tables, which now enables us to create abstractions on top of our testscripts.

Conclusion

Xebium enables us to use both Selenium IDE (low learning curve) and FitNesse (ease of maintenance) to it’s fullest and provides a complete solution when it comes to automated web testing.

Note, if you want to know why i chose GHUnit over OCUnit, just scroll down to the end of the post.

But wait….

Before I begin, I want to make one thing very clear, the difference between code unit testing and UI testing. Unfortunately, UI development can be hard to do in a TDD fashion. Especially when you want to test UI components. e.g. When I send a TouchEvent will the View respond and trigger my method in my controller.
My advice; don’t do UI testing with a Unit testing framework (OCUnit, JUnit, GHUnit), do it with for example the iOS automation API, which has been created specifically to test those UI components. I’ll also get back to you in a later post on how to do UI testing on Android.

What do you test with Unit testing frameworks? Well you test just the code, nothing more. Model and Controller code, not the View! You might need the help of a mocking framework to make it testable, because the view is missing and needs to be wired up for the controller and model to work properly, or it might need other controllers etc..

Let’s begin setting up!

With that in mind, let’s set up our own iPhone XCode 4 project! add GHunit, create a test and run it in the simulator or your own iOS device, iPhone, iPod Touch or iPad.

1. First of all download the GHUnitIOS version (0.4.28 at this time) at https://github.com/gabriel/gh-unit/archives/master
2. Unpack the downloaded zip file somewhere in your home directory, you should get the GHUnitIOS.framework directory

NOTE; I first placed it in /Developer/Library/Frameworks but XCode 4 didn’t like it and when compiling it could not find the header files, therefore I placed them somewhere in my home directory (e.g. /Users/rvanloghem/Development/Frameworks/GHUnitIOS.framework)

Right, you are now ready to set up your GHUnit ready XCode 4 project.

3. For example purposes, I’m choosing a normal, navigation-based application but you might have an existing project.
- NOTE; Don’t check the Include Unit Tests, because we are going to supply our own unit testing framework and not rely on OCUnit, which is supplied by default in XCode

4. In your XCode project settings (blue root icon in the tree browser) add a Target which you can call Tests, i usually base the project on a simple View-based Application (a Target named Tests will be added + a folder named Tests with all the Tests target files in them)

5. Next go back to the Tests target (click on the Tests Target) and add the GHUnitIOS.framework which you downloaded and unpacked in step 1 and 2. (click on the Build Phases tab, open up the Link Binary with Libraries, hit the + button, click Add Other and navigate+select the GHUnitIOS.framework directory on your filesystem)
6. Optional, but nice, move the GHUnitIOS.framework in your Tree to the Frameworks folder, to tidy things up

7. Set the -ObjC and -all_load in the other linker flags on the Tests Target (Select the Tests Target, select the Build Settings, search for other linker flags, and add the 2 flags)

8. Now you can delete some files which are not necessary, all the files in the Tests folder. (Note, Not! the Supporting Files folder as well, just the files!)
9. Delete the main.m file in the Supporting Files folder

10. In the Tests-Info.plist file (again, in the Supporting Files folder), clear out the Main nib file base name value

Time to create the GHUnit test runner, which will scan for our Unit Test Cases and run them.

11. Create an Objective-C class in the Tests folder named GHUnitIOSTestMain and make sure it is only! added to the Tests target

12. You can delete the GHUnitIOSTestMain.h file
13. Copy and paste source code from (http://github.com/gabriel/gh-unit/blob/master/Project-IPhone/GHUnitIOSTestMain.m) in your GHUnitIOSTestMain.m file

And now it’s time to create our own test case which will fail

14. Again create an Objective-C class in the Tests folder and as this is an example, name it ExampleTest, also make sure it is added to the Tests Target only!
15. You can delete the ExampleTest.h file
15. Copy and paste the next piece of code which is 99% copy/pasted from the example code from GHUnit (http://gabriel.github.com/gh-unit/_examples.html)

// For iOS
#import <GHUnitIOS/GHUnit.h>
// For Mac OS X
//#import <GHUnit/GHUnit.h>

@interface ExampleTest : GHTestCase { }
@end

@implementation ExampleTest

- (BOOL)shouldRunOnMainThread {
    // By default NO, but if you have a UI test or test dependent on running on the main thread return YES
    return NO;
}

- (void)setUpClass {
    // Run at start of all tests in the class
}

- (void)tearDownClass {
    // Run at end of all tests in the class
}

- (void)setUp {
    // Run before each test method
}

- (void)tearDown {
    // Run after each test method
}  

- (void)testFoo {
    NSString *a = @"foo";
    GHTestLog(@"I can log to the GHUnit test console: %@", a);

    // Assert a is not NULL, with no custom error description
    GHAssertNotNULL(a, nil);

    // Assert equal objects, add custom error description
    NSString *b = @"bar";
    GHAssertEqualObjects(a, b, @"A custom error message. a should be equal to: %@.", b);
}

- (void)testBar {
    GHAssertTrue(TRUE, @"Yes it worked");
}

@end

Right, ready to run!

16. Launch your Tests target (iTunes-like play button arrow) and run it against the simulator-scheme and your Unit test app should start

17. Now in the app in the simulator hit the blue run button and your tests will execute. (and testFoo will fail!, you can click on it to see why it failed)

And now it’s time to fix it…

18. Change the NSString *b = @"bar"; in the testFoo method to NSString *b = @"foo";
19. Run the Test app again and re-run the test and they should be green or in this case black, which means your tests are ok

Showing the power of GHUnit

20. Run the app against your own iOS device-scheme. (Select iOS-device-scheme and click the iTunes like run button)

Why i chose GHUnit over OCUnit

IMHO, this shows the real power of the GHUnit testing framework, not only does it run in the simulator but it also runs on your own iPhone, iPad, etc. Whereas OCUnit can only run as part of your build on your own machine, not on your phone and not in the simulator, this for me, is a big dealbreaker.
The closer you can get your unit tests running against a real-world environment the better it will be. Why? Because you are making use of the real devices processor (not the intel x86), real memory management, or lack there-of, the real API’s etc. If my Unit tests run on my phone, i’m 99.999% certain that the code under test will actually run on, yes, you guessed it, my phone.

There is of-course a downside to GHUnit, OCUnit (bundled with XCode) can be run automatically prior to you compiling your own app, this makes getting feedback about regression a lot faster, GHUnit is something which you have to run manually, in this case. But to solve that problem, or at least make it a whole lot better, we can use continuous integration aka build server to do the auto-running-of-unit-tests for us. There is a very nice blog post which compares various iOS unit testing frameworks.

So what is next? Well, the topic for my next blog and video in this series is hooking up a XCode project + GHUnit to Jenkins (or Hudson for the Oracle minded people out there).

And to start off, here’s how I benefit from doing TDD:

1. Robust
2. Better code design, no really!
3. Find regression problems early on

and you can read more on TDD and Continuous testing here.

The thing is, writing a mobile app takes about 20% of the time it would take me to write a web+client+server based app. Which of-course is really nice, because I can write lots of apps. It also means that whenever i need to fix a bug or add new functionality, I need to have developed it in a TDD fashion, otherwise i cannot guarantee reliability. So let me explain why…

Previous experience

The last complex software piece I’ve worked on is a product called Deployit, which can deploy java applications to application servers, not just one, but many different type of servers and the deploy part is encapsulated in a piece called a plugin. For every plugin we have written lots and lots of tests and boy do they prove to be a lifesaver.

As a developer I’m not just working on a single plugin, I’m working on the entire product (partly, because we are a small team and i like to do so), that means that for a single week I’m adding new deployment logic to plugin A and next week fixing a bug in plugin B. I’m a good programmer with a good memory but working on so many plugins and quickly fixing problems or adding functionality is hard. I have to context switch between intricacies of plugin D and the complexity of plugin W, I am bound to make a mistake or two, which will delay the release or upset users because functionality that used to work is now broken.

So how does this compare with mobile apps? It’s simple, plugins, resemble for me, apps. In the mobile world, you are bound to work on multiple apps in a short amount of time, that means, you need help to keep those apps stable when you are working on them.

Translation of TDD+Continuous testing to mobile development

Did you ever update an app that you worked on 2 months ago and broke per accident some old functionality? Then you must have seen what happened with your user ratings, it went down, right? People expect functionality that have always worked to still keep working, otherwise you will get 1-star or worse, it gets deleted! New functionality can have some bugs, because it is new, existing functionality should just work, there is no excuse for it to not do so.

Unless your brain can keep track of what the code of every app that you might work on is doing, which I don’t think it can, if it would you would work at NASA on “Space Bus 3.0″. That is where your automated tests come in, they really help you, make a fix and guaranteeing that your old functionality does not break.

It also helps you make new functionality more robust, don’t just test the happy flow, also test the monkey flow, input some weird data, trigger some weird touch-events, see if you can make your app crash, and do this automated. Your app becomes much more robust and your users will love you for it.

What is next…

The above words are all fine, but how do we actually do this ? In the next couple of blog posts I will go into technical detail how to do;
1. Unit testing (cheap testing at code level, OCUnit, JUnit)
2. Continuous Integration (using Jenkins)
3. UI testing (not so cheap testing at the UI level, Automation API’s, mock framework)
and do this for iOS and Android platforms.