Xebia Blog » Maarten Winkels

Agile says: Nothing will ever be perfect

Maarten Winkels — Wed, 21 Sep 2011 07:00:59 +0000

Wouldn’t it be sweet if your whole life were perfect? Your wife would fulfill your every wish. Your children would be perfect examples of responsible happy people growing up. At work your colleagues are the nicest people and working with them is always fun. Your team would feel responsible for every action they (proactively) take and the software systems you produce and maintain are flawless and run like well oiled machines?…

You need to wake up! Nothing will ever be perfect and Agile knows it!

Perfection and Change

How does Agile know this? The answer is very simple: If something was perfect, changing it could only make it worse. Agile says embrace and welcome change and thus acknowledges that nothing will ever be perfect. There will always be room for improvement in (almost) every direction!

In an Agile software project this applies to all possible levels: The code, the design and the architecture will never be perfect. The business requirements will never be perfectly clear. The team and the process require constant attention. Collaboration can always be improved.
That is why Agile processes incorporate finding improvements into their very core. For example, the sole purpose of the retrospective meeting in a SCRUM process is to find concrete ways to improve yourself as a team in every possible direction. The scope of the improvements is limited only by the capabilities of the team: The team is allowed to change and improve every aspect of their way of working, be it how they write code, how they communicate with the rest of the organization or how they schedule their meetings.

Agile goes even further and says that everything should be barely good enough. As a software development practice Test Driven Development embodies this. This approach helps the developer to focus on writing only what is absolutely necessary.

The Agile answer: Learning by Doing

So, how do you build good systems in an imperfect world? The agile answer is to see the whole building process as a learning exercise on all levels. The business gradually learns what they really want, while the team learns to work together and how to apply the latest technology to the chose architecture and solve the technical problems. And where people are learning, mistakes¹ are being made. This is not to be avoided nor to be feared: Making mistakes is the only behavior that infallibly leads to learning.

As a SCRUM team you are in the perfect position to learn from your mistakes, if you allow yourself to make mistakes, dare to look them in the face and only then try to overcome them. SCRUM’s short iterative process and intensive inspect and adapt cycle will bring out the problems sharp and quickly. It is up to the SCRUM team to be brave and tackle the ever arising problems. And remember, even if you constantly tackle every problem that comes your way… nothing will.. ever… be perfect…

The one thing that is allowed to be perfect is the vision of where you are going. The vision of a beautiful system and a happy team. This vision lives in a perfect world. It will always be that one dot just over the horizon. The prefect image that -you will have to accept- is ever out of reach. The main challenge is to keep the team in the right direction and gently steer it closer and closer.

Agile pitfall: Single User Story Features

One approach that aims for perfection and ends with frustration is to try to complete every feature in one go. The iterative approach is meant to enable developers and users to learn. The more opportunities for learning, the more you will learn. So instead of building a feature in one go, and create only one learning moment, deliver in small steps and gradually improve the application.

A good agile planning incorporates this from the beginning and deliberately plans to pickup the same functional area multiple times during the development process. Don’t try to do the fancy search box all in one go: start with simple searching, add auto complete, add a time filter, add faceting. Make all of these simple improvements small User Stories that you plan over the course of the entire development process.

Conclusion

Be brave!
Deliberately take small steps!
Dare to accept the current reality as the only reality!
Dare to dream of the perfect end result!

If you start to take small steps, allow your team to make mistakes and stimulate them to learn from them, you’ll find that working with them becomes easier and more fun. The team will feel more empowered and more eager to take decisions. Eventually, you’ll also be happier at home, a better husband and after years and years of applying this rule, you’ll finally see how your children have grown into responsible adults

Footnotes

Mistakes here are not like bugs or coding errors, but rather like misinterpretations or misunderstandings.

Master-detail implementation for RESTful services with JQuery

Maarten Winkels — Thu, 15 Sep 2011 15:26:17 +0000

In two previous posts, we have seen how to develop RESTful application with JBoss AS 7. At the end of the second blog we used a generic REST client tool to execute some RESTful web services. Of course we would rather build a custom UI application as client for our services, so that a user can easily access and manipulate the data. In this blog we build a REST client that is based on the master-detail principle.

REST and Master-Detail

A RESTful web service, as developed in the previous posts is essentially an editable collection. This makes a Master-Detail design a prime candidate for the User Interface on top of it: The UI will display the entire collection in the master view and a single entry in the collection will be made editable in the detail view.

How do we implement this? Well, the web services produce data in the JSON format. A client will have to consume this data and display it in a User Interface. A JavaScript client is a very good choice for this, since JSON is a sort-of “native” format for JavaScript. Of course other options are possible, you could develop a client with Adobe Flex or JavaFX, for example. An advantage of using simple HTML and JavaScript would be that there are a lot of applications (web browsers) that understand these languages and can be used to run the REST client.
There are a number of frameworks that can be used to develop a REST client in JavaScript. A popular choice is BackBone.js with Mustache.
For this blog I have chosen to develop the client as a jQuery plugin. jQuery is a general purpose JavaScript library with a lot of features. We are going to use its templating feature to display the data.

Using jQuery template for the Master view

Let’s first look at the master view. This will be a table with a row for each book, displaying the id and the title.

What we need for this is a simple jQuery template that displays a single row for each book and an appropriate container in which the rows can be placed.

The jQuery template can be a script tag in the header of the page. It should have the type text/x-jquery-tmpl and an appropriate id for referencing. Properties of the objects we want to display (in this case the id and title of the book) can be inserted using ${…} notation.


	Books
	
		
			Id
			Title

The container element is in this case a tbody with id=”booksContainer”. The pattern in the ids used here will become more clear later.

Loading the data and filling out the template

Now how do we download the data and use the template to show it?

function loadList (options) {
	doAjaxCall('GET', options.url, null, function(data) {
		useTemplate(options.rowTemplate, data, options.rowsContainer);
	});
}

function doAjaxCall(type, url, data, callback) {
	$.ajax({
		type: type,
		url: url,
		dataType: "json",
		data: data,
		success: callback
	});
}

function useTemplate(template, data, container) {
	if (template) {
		container.empty();
		template.tmpl(data).appendTo(container);
	}
}

The loadList function loads the data and displays it in the master view. It uses the doAjaxCall function to start the asynchronous loading. This is merely a wrapper around the standard jQuery ajax function, but it will be used in many more places. When the ajax call returns successfully, the useTemplate function is used to display the returned data in the template.
The options object that is referenced in some places, is a container for configuration options for the client, that will be discussed later. The relevant configuration options here are:

var options = {
	url: 'books/',
	rowTemplate: $('#bookRowTemplate'),
	rowsContainer: $('#booksContainer')
};

The JSON that is loaded for display is shown below. In this case it is a list of objects, where each object is a book with an id and a title. Since it is a list, the template will be applied to each element and the resulting HTML is concatenated.

Selecting and deleting items from the list

The template for each row contains two links, one for selecting an item, the other for deleting it. How do these work?

// opts.selectLinks = $('.bookSelect');
opts.selectLinks.live('click', function (event) {
	var id = $(this).data('id');
	doAjaxCall('GET', url(opts.url, id), null, function(data) {
		useTemplate(opts.entryTemplate, data, opts.entryContainer);
	});
	event.preventDefault();
});

//opts.deleteLinks = $('.bookDelete');
opts.deleteLinks.live('click', function (event) {
	var id = $(this).data('id');
	doAjaxCall('DELETE', url(opts.url, id), null, function(data) {
		doResetAndReload(opts);
	});
	event.preventDefault();
});

function url(url, id) {
	return url + id + '/';
}

Handlers for the click events are attached to these links using jQueries live function. From the documentation

Attach a handler to the event for all elements which match the current selector, now and in the future.

In our case, the row template generates class attributes for the links that can be used for attaching the handler. For both operations, we need the id of the book we want to select or delete. For easy access, the template generates a data attribute (data-id) in the links. In the event handlers, the id is retrieved from the node and used to make another ajax call. For the delete links, a DELETE method is called and on success the UI is reset and the list reloaded. For the select links we need to display the detail data of the item in an entry form.

The detail view as entry form

The entry form should display the data for one item in the list (in our case a book) and it should allow the user to POST or PUT the data in the form to a Web Service. The static HTML is fairly simple:


Book:

The form has only an id attribute and no action or method. This will be handled by the javascript handler for submitting the form. The submit and cancel buttons are static and standard. The interesting part is the fieldset, that will contain the elements rendered by the template.

In this template there is a little logic to handle the cases for new and existing books. An existing book can be recognized by its id property. If the id property is filled, it should be set and send to the web service by adding the hidden input field.
The JavaScript handle the form events is a little bit more complicated.

opts.entryForm.submit(function (event) {
	var form = $(this);
	var data = form.serialize();
	var id = $('#'+opts.entryIdField).val();
	if (id) {
		doAjaxCall('PUT', url(opts.url, id), data, function(data) {
			doResetAndReload(opts);
		});
	} else {
		doAjaxCall('POST', opts.url, data, function(data) {
			doResetAndReload(opts);
		});
	}
	event.preventDefault();
});

opts.entryForm.bind('reset', function (event) {
	onSelect(opts, null);
	event.preventDefault();
});

When submitting the form, we want to send the contents as form data to through an ajax call. If the id is not set, a POST method is called to create a new instance. If the id is set, a PUT method should be called with the id appended to the URL. In both cases we want to reset the UI and reload the list on a successful return.

Icing on the cake: Expanding options

This is basically all the code for this master-detail based REST client with jQuery templates. With this code you can do simple CRUD operations on simple books, as shown in the screenshots below.

Now how about those options that you kept on hearing about? I designed the client to be able to use multiple REST clients for related collections on the same page. Therefore I wanted some mechanism that would allow me to do

new RestClient({name:'book'});
new RestClient({name:'author'});

and the client would figure out which templates, which containers and which forms to use for display and entry purposes. This is accomplished by using configurations that are expanded to include the value for other options.

RestClient = function (options) {
	var opts = $.extend({}, RestClient.defaults, options);
	expand(opts);
	...
};
RestClient.defaults = {
	plural: '{name}s',

	baseUrl: 'services',
	url: '{baseUrl}/{plural}/',

	rowTemplate: '#{name}RowTemplate',
	rowsContainer: '#{plural}Container',

	entryTemplate: '#{name}EntryTemplate',
	entryContainer: '#{name}EntryContainer',
	entryForm: '#{name}Form',
	entryIdField: 'idField',

	selectLinks: '.{name}Select',
	deleteLinks: '.{name}Delete',

	onSelect: $.noop,
	onReload: $.noop
};

The options passed into the constructor extend the default options, which allows for customizing the options anyway you would want. After the extension is applied, the values are expanded, basically replacing every occurrence of {…} by the option value referenced by that name.

function expand(options) {
	for (var field in options) {
		var val = options[field];
		if ($.type(val) == 'string') {
			while (val.match(/{([^{}]*)}/)) {
				val = val.replace(/{([^{}]*)}/, function (key, group) {
					return options[group];
				});
			}
			if (val.charAt(0) == '#' || val.charAt(0) == '.') {
				var obj = $(val);
				if (val.charAt(0) == '#') {
					if (obj.length == 0) {
						obj = undefined;
					}
				}
				val = obj;
			}
			options[field] = val;
		}
	}
}

The highlighted lines in this code fragment resolve any CSS selectors through jQuery after expanding them.
As an example (and a short preview) the screenshot below shows a page with two Master-Detail implementations, one for books and one for authors, both implemented using this plugin.

Conclusion

Writing a JavaScript client for RESTful web services is very simple with jQuery and jQuery template. The source code for this blog can be downloaded here:
books.html jquery.rest-client.js books.css
Since the backend is not deployed, the page is not functional

Developing a JPA application on JBoss AS 7

Maarten Winkels — Tue, 19 Jul 2011 07:00:23 +0000

In my previous post we setup a Maven/Eclipse project for developing RESTful web applications on JBoss AS 7. A RESTful web service that is not using a database is some what of an oddity. Therefor in this blog we’ll extend the project with JPA.

Configuration: Adding a JBoss Module

I want to use Postgres as the underlying database. To install the required JDBC driver for Postgres in the JBoss AS 7 container, we need to add a module. As I mentioned in the previous post, the module structure of JBoss is reflected on the file system.

The files in the ”main” directory that constitute the module are the folloing:

module.xml describes the module. The content follows below.
postgresql-9.0-801.jdbc4.jar is the postgres JDBC4 driver.
postgresql-9.0-801.jdbc4.jar.index is an index file generated by JBoss Annotation Indexer when the modules are scanned.

So in order to add a module to the JBoss container, we need to write an XML file:

Now we also need to enlist the datasource in the standalone.xml: the main configuration for JBoss. You’ll find it in the ”standalone/configuration” subdirectory of the JBoss installation. The file already has a datasource and driver for h2. So we add a new node:



...
            
                
...
                
                
                    
                        
                            org.postgresql.xa.PGXADataSource
                        
                    
                    
                        
                            org.h2.jdbcx.JdbcDataSource
                        
                    
                
            
...

Now, after restarting the server, we can finish the configuration using the admin console.

The ”New Datasource” button will open up a dialog that will guide you through a few simple steps to configure the datasource.
The changes will be persisted in the standalone.xml, which will now contain a fragment like this:

..
                
                    
                        jdbc:postgresql://localhost:5432/library
                    
                    
                        postgres
                    
                    
                        
                            library
                        
                        
                            library
                        
                    
                
...

Reading from the database

Now we need to configure our application to read from the database. To configure the database as a persistence context in JPA, we add a persistence.xml to our project.




	
		org.hibernate.ejb.HibernatePersistence
		java:/LibraryDS

The java code we have to write is quite trivial JPA code mixed in with some JAX-RS annotation. For completeness they are included here below (collapsed).

package com.xebia.library.model;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.SequenceGenerator;
import javax.ws.rs.FormParam;

@Entity
@SequenceGenerator(name="BOOK_SEQ", sequenceName="BOOK_SEQ")
public class Book {

	@Id
	@GeneratedValue(strategy=GenerationType.SEQUENCE, generator="BOOK_SEQ")
	@FormParam("id")
	private Long id;

	@FormParam("title")
	private String title;

	public Long getId() {
		return id;
	}

	public String getTitle() {
		return title;
	}
}

package com.xebia.library;

import static javax.ws.rs.core.MediaType.APPLICATION_JSON;

import java.util.List;

import javax.ejb.Local;
import javax.ws.rs.DELETE;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.PUT;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;

import org.jboss.resteasy.annotations.Form;

import com.xebia.library.model.Book;

@Local
@Path("books")
@Produces(APPLICATION_JSON)
public interface BookRepository {
	@GET
	@Path("/")
	List all();

	@POST
	@Path("/")
	Book create(@Form Book entity);

	@GET
	@Path("/{id}/")
	Book getById(@PathParam("id") long id);

	@PUT
	@Path("/{id}/")
	Book update(@Form Book entity);

	@DELETE
	@Path("/{id}/")
	void remove(@PathParam("id") long id);
}

package com.xebia.library.impl;

import java.util.List;

import javax.ejb.Stateless;
import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;
import javax.persistence.criteria.CriteriaBuilder;
import javax.persistence.criteria.CriteriaQuery;
import javax.persistence.criteria.Root;

import com.xebia.library.BookRepository;
import com.xebia.library.model.Book;

@Stateless
public class BookRepositoryBean implements BookRepository {

	@PersistenceContext
	private EntityManager em;

	@Override
	public List all() {
		CriteriaBuilder builder = em.getCriteriaBuilder();
		CriteriaQuery query = builder.createQuery(Book.class);
		Root root = query.from(Book.class);
		query.orderBy(builder.asc(root.get("id")));
		return em.createQuery(query).getResultList();
	}

	@Override
	public Book create(Book book) {
		em.persist(book);
		return book;
	}

	@Override
	public Book byId(long id) {
		return em.find(Book.class, id);
	}

	@Override
	public Book update(long id, Book book) {
		return em.merge(book);
	}

	@Override
	public void delete(long id) {
		em.remove(byId(id));
	}

}

A few things to notice:

The JPA entity is also used as a ”backing” object for Form posts. The @FormParam annotation is used to parameters posted to the fields of the object.
The Repository is now also used as a local interface for the stateless bean.

After deploying the application to the server, we can test it using this simple tool.

Conclusion

With just a few classes we were able to deploy a RESTful web service that enables users to maintain a (rudimentary) book list. The WAR that we build is really tiny: It only contains a few classes. The application relies on JEE specifications that are implemented in the JBoss container. A few remarks about this:

Developing a JEE application becomes much simpler and more fun! It feels almost like RAD web development alla Grails.
Testing an application like this is non-trivial: Much of the functionality is provided by the container. With a ‘thicker’ maven project with Hibernate and Spring/Seam as dependencies, we could mimic (parts of) the deployed application in an in-build testing environment. How do we do this now? One possible answer is to use Arquillian. A future blog will have to cover that…

Of course the application we have now is far from finished. To start with, it has no user interface…

Developing RESTful applications on JBoss AS 7

Maarten Winkels — Tue, 12 Jul 2011 07:00:25 +0000

JBoss AS 7 CR1 has been released recently. On the previous release it was pretty easy to develop RESTful applications with the build in JAX-RS support based on RESTeasy. In this blog I’ll look at how well the new version of JBoss keeps up with the rest of the field.

Changes: JBoss Modules

JBoss AS 7 is quite a different beast from its predecessors. The most striking difference is the usage of JBoss modules, which is a modular class loading system, comparable to Jigsaw, THE modular system for Java that will (one day) (probably) reach the JVM. Till that day however, JBoss modules can be used.

After downloading and unzipping the JBoss AS 7 CR1 release, you’ll find all the modules in the ”modules” subdirectory, grouped in a structure that somewhat resembles a maven repository: you’ll for example find directories for ”org”, ”javax” and ”net”, similar to the directory structure of group-ids and artifact-ids in a Maven2 repository layout. At some level, however, there will be a ”main” folder and that’s where the implementation details of the module are kept: a small XML file describing the module and some jar files that constitute the modules content. A very useful side effect of this is that finding out which versions of which jars the AS is using is rather simple: The jar files in the module are usually the maven artifacts and thus contain the version number in their name. See for example the javax.activation module below.

Another consequence of this change is that the usual JBoss structure with a ”server” directory with several profiles, like ”default”, ”web” and ”minimal” no longer exists. The main structure now has a ”standalone” directory for running a simple server, and a ”domain” directory for running a cluster.

Installation

Installation is as always straight forward. A development environment however is not ready for use without good integration between the IDE and the application server on the local system. IDE of choice is Eclipse, because of the nice JBoss Tools plugin suite. Following these instructions is already a large part of the installation.

To make things really easy, we include Maven integration for Eclipse with WTP from Sonatype into the mix. Especially finding the right version of the Maven integration for WTP was difficult. I finally found a copy that worked with the rest of the setup at this update site.

Starting the server can now be done from the Eclipse servers window. Browsing to the default port shows a nice welcome window and going to the admin console shows a simple but useful administrative interface.

Two small problems with the setup so far:
1. Eclipse doesn’t seem to detect the startup of the server. Which is kind of annoying, because Eclipse now wont compile anything since it is waiting for the process to end.

This can be worked around by setting the Startup Poller to ”Web Port Poller” in the Server overview screen of the JBoss 7 AS server.

2. Eclipse is not able to stop JBoss. At least not from the server view.

JBoss can still be stopped from the Console view, but that means restarting the server always takes at least 4 mouse clicks: (1) go to console (2) stop (3) go to servers (4) restart, which is kind of annoying.

Anyway, living with small annoyances is second nature for everybody in IT industry I suppose, so why would a simple JEE application developer be any different

Starting up the project

Now that everything is set up, we can continue to the more interesting part, the project and the Java code. With the help of M2Eclipse we create a maven project with the following pom:


	4.0.0
	com.xebia.library
	library
	0.9-SNAPSHOT
	war
	
		
			jboss
			https://repository.jboss.org/nexus/content/groups/public-jboss/
		
	
	
		
			
				maven-resources-plugin
				2.4.3
				
					UTF-8
				
			
			
				maven-compiler-plugin
				2.3.2
				
					1.6
					1.6
				
			
			
				maven-war-plugin
				2.1.1
				
					false
				
			
		
	
	
		
			org.jboss.spec
			jboss-javaee-6.0
			2.0.0.CR1
			pom
			provided
		
		
			org.jboss.resteasy
			resteasy-jaxrs
			2.2.1.GA
			provided

This pom will configure the maven project and the Eclipse-WTP project correctly for deployment on JBoss. The highlighted lines show the dependency on a pom with the Aggregated Java EE 6.0 APIs that JBoss provides. This pom is supposed to be used to import the right version numbers of all the Java EE APIs into your own pom. Since I am lazy, I’ll just depend on this pom, so all the APIs are already available in my project without having to specify them separately.

Now we have a well configured Maven Web project in Eclipse. The layout will look like the image below.

And now for some Java code, how do we configure RESTeasy framework?

package com.xebia.library;

import javax.ws.rs.ApplicationPath;
import javax.ws.rs.core.Application;

@ApplicationPath("services")
public class LibraryApplication extends Application {

}

That’s it! From the class RESTeasy will be configured and the RESTful web services will be available at ”/services” in the context of the project.

Let’s add a simple web service.

package com.xebia.library;

import java.util.List;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

import com.xebia.library.model.Book;

@Path("books")
public interface BookRepository {

	@GET
	@Path("/")
	@Produces(MediaType.APPLICATION_JSON)
	List all();
}

This interface defines the RESTful web service by means of the JAX-RS annotations. Adding an interface is useful, but not necessary.

package com.xebia.library.impl;

import java.util.Collections;
import java.util.List;

import com.xebia.library.BookRepository;
import com.xebia.library.model.Book;

public class BookRepositoryBean implements BookRepository {

	@Override
	public List all() {
		return Collections.singletonList(new Book("My Title"));
	}

}

The implementation of the web service is very simple (for now) and it uses a very simple POJO as return value.

package com.xebia.library.model;

public class Book {

	private String title;

	public Book() {
	}

	public Book(String title) {
		this.title = title;
	}

	public String getTitle() {
		return title;
	}
}

With these simple files, we have all we need to get a RESTful web service running on JBoss AS 7. Pointing the browser to the correct URL, we get a JSON string as shown below.

The URL is built up from a few parts:

http://localhost:8080 The JBoss web server
library The application context
services The path configured for the JAX-RS application in the @ApplicationPath annotation
books The path for the BookRepository resource as given in the @Path annotation

JBoss AS 7 comes with RESTeasy (an implementation of JAX-RS) and Jackson (a Java-JSON serializarion framework) configured and that is how the final result is obtained. This makes JBoss AS 7 an ideal platform for developing RESTful applications: it works out-of-the-box and is build on well known open source projects, making it easy to adapt the container to specific requirements in you application.

Conclusion

In this blog we have set up a development environment for JBoss AS 7 using Maven, Eclipse and WTP and we deployed a very simple RESTful application. The setup works pretty well.
Keep an eye out for new blogs, in which I’ll improve on the current project to show more aspects of JBoss 7 and RESTful applications.

Writing and testing data structures and algorithms in JavaScript

Maarten Winkels — Thu, 23 Jun 2011 17:07:07 +0000

Tonight in one of our knowledge exchange sessions, one of my colleagues challenged us to writing a TagCloud in JavaScript. He had prepared a nice setup with a server producing twitter hashtags over a WebSocket to the browser and using Processing.js to produce a graphical representation of the tags zooming by on twitter. Since he had already done all the heavy lifting in integrating all these fancy new frameworks, what was left to do, you might ask. Well, we still needed to implement the algorithm to count the number of tags on the continuous stream, sorting this list on the bases of the counts and making sure the system wouldn’t run out of memory by removing less used tags in some smart way. His point to all of this was, that although JavaScript is being prophesized in some circles as the new-old-new language of the future, writing and testing a non-trivial algorithm in it is a big challenge.

In the remainder of the hour, we tried in several pairs to implement the missing algorithm and it has to be said that we all failed miserably. It has to be said that none of us were JavaScript experts, but I (for one) felt (in the beginning…) that I should have enough experience in IT in general and a little in JavaScript to make this work. But alas…

Driving back home, I decided not to give in so easily and to try and use some decent frameworks to overcome this challenge.

JavaScript testing: YUI

I turned to finding a testing framework first. Sure enough, most javascript testing frameworks focus on fancy UI stuff like asynchronous testing, integration testing or automated multi-browser testing. I was looking for none of that, but a simple xUnit like framework, that I could run in a browser to test the functionality of a number of JavaScript classes. After a few minutes I found YUI which seemed to fit the bill. Setting up was easy enough:





Testing TagCounter with YUI

This static html file is all that is neede to run my tests. The tests themselves are written in the TagCounter_Test.js file:

/**
 * Unit Tests for TagCounter
 */
YUI({
  combine: true,
  timeout: 10000
}).use("node", "console", "test", function (Y) {
  var assert = Y.Assert;

  var TagCounterTestCase = new Y.Test.Case({
    // test case name - if not provided, one is generated
    name: "TagCounter Tests",

    "test should increment count": function () {
      var tc = new TagCounter();
      tc.add("my-tag");
      assert.areEqual(1, tc.map().get("my-tag"));
      tc.add("my-tag");
      assert.areEqual(2, tc.map().get("my-tag"));
    }
  });

  //create the console
  var r = new Y.Console({
    newestOnTop : false,
    style: 'block'
  });

  r.render("#testReport");
  Y.Test.Runner.add(TagCounterTestCase);
  Y.Test.Runner.run();
});

Opening the HTML file in FireFox shows the picture below.

It seems like I’m ready to implement the TagCounter TDD style!

Implementation: js_cols

To make this simple test pass, all I need is a simple HashMap with some basic features, like contains, get and remove, but… this is not java! After some googling, I found js_cols, a neat JavaScript collections framework, that basically had all the support that I wanted, including a LinkedHashMap implementation that can be used to implement a LRU cache, just like good old java! The LRU feature is going to help me to remove “old” tags, keeping the memory required to process the tags within reasonable limits.

To make the first test succeed, only a few lines of code are needed:

js_cols.require('js_cols.LinkedHashMap');
function TagCounter(maxSize) {
	this.tagCounts = new js_cols.LinkedHashMap(maxSize, true);
};
TagCounter.prototype.add = function (tag) {
	if (this.tagCounts.contains(tag)) {
		this.tagCounts.insert(tag, this.tagCounts.get(tag) + 1);
	} else {
		this.tagCounts.insert(tag,1);
	}
};
TagCounter.prototype.map = function () {
	return this.tagCounts;
};

In the end, however, we want the map method to return only the most popular tags, while our counter should keep track of a longer list. To implement this, we need two steps:

First we need to sort the tags by their number of appearances in descending order, keeping in mind that multiple tags can have the same number.
Then we need to take the top x most popular tags.

The test:

...
    "test should return limited sized map": function () {
    	var tc = new TagCounter(3);
    	tc.add("my-old-tag");
    	tc.add("my-old-tag");
    	tc.add("my-old-tag");
    	assertSame(tc.map(2), [["my-old-tag", 3]]);
    	tc.add("my-first-tag");
    	tc.add("my-first-tag");
    	assertSame(tc.map(2), [["my-old-tag", 3], ["my-first-tag", 2]]);
    	tc.add("my-second-tag");
    	assertSame(tc.map(2), [["my-old-tag", 3], ["my-first-tag", 2]]);
    	tc.add("my-third-tag");
    	assertSame(tc.map(2), [["my-first-tag", 2], ["my-third-tag", 1]]);
    }
  });

  function assertSame(map, expected) {
	  var keys = map.getKeys();
	  assert.areEqual(expected.length, keys.length);
	  for(var i=0,len=expected.length; value=expected[i], i
In the test we now add a method assertSame to make it easier to express our expectations of the map method: The map should contain the expected number of tags with their counts in the right order. Therefor the expectation is a list of [tag, count]-tuples.
We can again use the collection library classes to implement this: The RedBlackMultiMap class implements a key-sorted map that allows multiple values for the same key. 
TagCounter.prototype.map = function (opt_size) {
	var orderedMap = new js_cols.RedBlackMultiMap(function (a,b) {return b-a;});
	this.tagCounts.forEach(orderedMap.insert, orderedMap);
	var result = new js_cols.LinkedHashMap();
	if (opt_size) {
		var i = 0;
		orderedMap.forEach(function (value, key) {
			if (i < opt_size) {
				result.insert(value, key);
			}
			i++;
		});
	} else {
		orderedMap.forEach(result.insert, result);
	}
	return result;
};

On line 2 the RedBlackMultiMap constructor is used with a comparator function that will invert the natural ordering of numbers. This way we sort in descending order.

On line 3 all items of the tagCount map are inserted into the orderedMap reversing the key-value relation. The fact that the forEach method take a function that has the value as the first and the key as the second argument (which would usually be confusing) now appears a handy feature!

On lines 4 to 7, the key-value relation is again reversed, making sure to only take the top opt_size items from the map (when set).
Conclusion
Looking at the simplicity of the code and the tests, I would say that it is definitely possible to write well tested and documented JavaScript algorithms and data structures. I completely agree with my colleague, however, that such code is not often found in your run-of-the-mill web application project. Maybe it is time to start treating the JavaScript code in our projects more as Java code and make sure that it is properly structured and tested.

Posting complex forms with RESTEasy – Part 2

Maarten Winkels — Wed, 20 Apr 2011 10:30:48 +0000

As promised in a previous blog, I’ll devote this blog to how to extend the RESTEasy framework with support for mapping form fields on object-graphs with complex associations, like lists and maps.

These extensions have been reported to RESTEasy as two issues with patches. If you like these features, please vote for these issues.

Extending the existing framework (with the extensions from the previous blog) is not so hard and might be a lot easier then you think. The crux is finding the right mapping from names of the form fields to the complex structures in the Java objects.

A form with these fields and values:

firstName=Maarten
lastName=Winkels
emailAddresses[0].emailAddress=me@xebia.com
emailAddresses[1].emailAddress=me@gmail.com
phoneNumbers[home].number=0306324178
phoneNumbers[work].number=+31 35 2356785

Could be mapped to the Java class below easily.

public class Person {

	@FormParam("firstName")
	private String firstName;

	@FormParam("lastName")
	private String lastName;

	@NestedFormParams("emailAddresses")
	private List emailAddresses;

	@NestedFormParams("phoneNumbers")
	private Map phoneNumbers;
}

public class EmailAddress {

	@FormParam("emailAddress")
	private String emailAddress;
}

public class PhoneNumber {

	@FormParam("number")
	private String number;
}

The mapping is quite simple. A list is mapped with its prefix appended with an index number between square brackets. A maps prefix is appended with the key between square brackets. As soon as a prefix is found, the rest of the “deserialization” is the same as for the address in the previous blog: Properties can be appended to the prefix with a dot as separator.

Now when the RESTEasy framework encounters a @NestedFormParam, it can look at the type of the property to find out how to interpret the form fields. It also needs to figure out the types of the elements in the collection. This is done in the InjectorFactory or its extenion.

public class ExtendedInjectorFactory extends InjectorFactoryImpl {

	private final ResteasyProviderFactory providerFactory;

	public ExtendedInjectorFactory(ResteasyProviderFactory factory) {
		super(factory);
		this.providerFactory = factory;
	}

	public ValueInjector createParameterExtractor(Class injectTargetClass,
			AccessibleObject injectTarget, Class type, Type genericType,
			Annotation[] annotations, boolean useDefault) {
		NestedFormParams param = FindAnnotation.findAnnotation(annotations,
				NestedFormParams.class);
		if (param != null) {
			String prefix = param.value();
			if (genericType instanceof ParameterizedType) {
				ParameterizedType pType = (ParameterizedType) genericType;
				if (isA(List.class, pType)) {
					return new ListFormInjector(type,
							getTypeArgument(pType, 0), prefix, providerFactory);
				}
				if (isA(Map.class, pType)) {
					return new MapFormInjector(type, getArgumentType(pType, 0),
							getTypeArgument(pType, 1), prefix, providerFactory);
				}
			}
			return new NestedFormInjector(type, prefix, providerFactory);
		}
		return super.createParameterExtractor(injectTargetClass, injectTarget,
				type, genericType, annotations, useDefault);

	}

	private boolean isA(Class clazz, ParameterizedType pType) {
		return clazz.isAssignableFrom((Class) pType.getRawType());
	}

	private Class getTypeArgument(ParameterizedType pType, int index) {
		return (Class) pType.getActualTypeArguments()[index];
	}
}

By inspecting the genericType parameter, the InjectorFactory finds out which Injector to use to instantiate and populate the correct class. The isA method is a utility method to find out if a collection type has been encountered. To find out the type of the elements in the collection, the getTypeArgument method is used. For a map type property the key type is also needed. Note that this implementation requires the collection to be parameterized with the concrete type of its elements!

The implementations of ListFormInjector and MapFormInjector are very similar and have a common superclass.

public abstract class AbstractCollectionFormInjector extends NestedFormInjector {

	private final Class collectionType;

	private final Pattern pattern;

	protected AbstractCollectionFormInjector(Class collectionType, Class genericType,
				String prefix, Pattern pattern, ResteasyProviderFactory factory) {
		super(genericType, prefix, factory);
		this.collectionType = collectionType;
		this.pattern = pattern;
	}

	public Object inject(HttpRequest req, HttpResponse res) {
		T result = createInstance(collectionType);
		for (String prefix : findPrefixes(req.getDecodedFormParameters())) {
			Matcher matcher = pattern.matcher(prefix);
			matcher.matches();
			String key = matcher.group(1);
			addTo(result, key, super.doInject(prefix, req, res));
		}
		if (empty(result)) {
			return null;
		}
		return result;
	}

	private Set findPrefixes(MultivaluedMap parameters) {
		final HashSet result = new HashSet();
		for (String parameterName : parameters.keySet()) {
			final Matcher matcher = pattern.matcher(parameterName);
			if (matcher.lookingAt() && hasValue(parameters.get(parameterName))) {
				result.add(matcher.group(0));
			}
		}
		return result;
	}

	protected abstract T createInstance(Class collectionType);

	protected abstract void addTo(T collection, String key, Object value);

	protected abstract boolean empty(T result);
}

The idea of this class is quite simple:

First all matches in the form field names for the matcher are gathered. If the form contains fields like ‘phoneNumbers[home].number’, ‘phoneNumbers[home].extension’ and ‘phoneNumbers[work].number’ and the pattern is ‘phoneNumbers[(.*)]‘, then the prefixes ‘phoneNumbers[home]‘ and ‘phoneNumbers[work]‘ are found. This is done in the findPrefixes method.
Now for each prefix the key is extracted. This is the part of the prefix that matched the first group. In the above example the first key would be home and the second would be work.
For each prefix an instance of the required element type is constructed by using the functionality in the NestedFormInjector.
All instances are collected in a collection, which is the end result of this injector.

There are three parts of this procedure that need to be adapted for the specific collection type. Let’s look at the ListFormInjector.

public class ListFormInjector extends AbstractCollectionFormInjector {

	public ListFormInjector(Class collectionType, Class genericType,
				String prefix, ResteasyProviderFactory factory) {
		super(collectionType, genericType, prefix,
				Pattern.compile("^" + prefix + "\\[(\\d+)\\]"), factory);
	}

	protected List createInstance(Class collectionType) {
		return new ArrayList();
	}

	protected void addTo(List collection, String key, Object value) {
		collection.add(Integer.parseInt(key), value);
	}

	protected boolean empty(List result) {
		return result.isEmpty();
	}
}

This is a very simple implementation. The most important part is probably the construction of the pattern at line 6. The pattern consists of the prefix and a number between square brackets. This number is later used as index into the list at line 14, where a new element is added to the list.

The implementation for maps is a little more complicated, since the key value has to be converted to an object as well. For this we use the StringParameterInjector of RESTEasy. An instance of this class can be used to convert a String to an instance of the required type.

public class MapFormInjector extends AbstractCollectionFormInjector {

	private final StringParameterInjector keyInjector;

	public MapFormInjector(Class collectionType, Class keyType,
				Class valueType, String prefix, ResteasyProviderFactory factory) {
		super(collectionType, valueType, prefix,
				Pattern.compile("^" + prefix + "\\[([a-zA-Z_]+)\\]"), factory);
		keyInjector = new StringParameterInjector(keyType, keyType,
				null, Form.class, null, null, new Annotation[0], factory);
	}

	protected Map createInstance(Class collectionType) {
		if (collectionType.isAssignableFrom(LinkedHashMap.class)) {
			return new LinkedHashMap();
		}
		if (collectionType.isAssignableFrom(TreeMap.class)) {
			return new TreeMap();
		}
		throw new RuntimeException("Unsupported collectionType: " + collectionType);
	}

	protected void addTo(Map collection, String key, Object value) {
		collection.put(keyInjector.extractValue(key), value);
	}

	protected boolean empty(Map result) {
		return result.isEmpty();
	}
}

The pattern used here is quite strict: Only letters and underscores are allowed. This restricts the possibilities for the key values in the map. In some situations this might be to restrictive.

Conclusion

With a few classes we have extended the RESTEasy framework to be able to support mapping form fields to collections. Th required classes are attached to this blog in a zip file as a Maven project. Feel free to use it in your own projects.
I think this would be a good extension of RESTEasy, but so far the patches have not been accepted by the developers. If you like this extension, please vote for the issues mentioned in the beginning of this blog.

How to use annotations for configuration

Maarten Winkels — Fri, 15 Apr 2011 22:34:38 +0000

In the java world we have been using and getting used to annotations since Java 1.5. Although there were some critical voices at first, I think most of us have come around and are using annotations now quite extensively. In my experience annotations are mostly used on POJO domain classes to configure frameworks like Hibernate, Spring and Seam and many other frameworks to be able to handle the custom objects correctly.

There are as many different approaches to this as there are implementations. In this blog I try to identify a few of the better approaches and a few of the poorer ones. The blog is not so much meant as a critique on the frameworks that the examples are taken from, but more as a guide to designing your own annotations whenever you might be faced with that task.

Prime examples: JPA annotations

The best examples of using annotations for configuration – both good and bad – can be found in JPA Entity classes.

@Entity
@Table(name="person")
public class Person {

  @Id
  @GeneratedValue(strategy=SEQUENCE)
  @Column(name = "personId")
  private Long id;

  @Valid @NotEmpty
  @OneToMany(cascade = CascadeType.ALL)
  @JoinColumn(name = "personId", nullable = false)
  @org.hibernate.annotations.Cascade(CascadeType.DELETE_ORPHAN)
  @org.hibernate.annotations.IndexColumn(name = "position")
  private List addresses = new ArrayList();
  ...
}

What a wealth of annotations! And in such a few lines of code. Anybody that has dealt with this type of code before however, will testify that this example is not overly complex at all. You’ll find this ind of code all over.

To start with, let’s say that I think this type of coding has been extremely successful. So regardless whether we find the usage of annotations in this example recommendable or not, if you work with Java code, you’ll be reading and writing a lot of code like this, so understanding it is imperative. However success is just one of the indicators for quality and it is always good to think about how things can be improved.

A perfect annotation

Of course there is no such thing as a perfect annotation. But as a general rule of thumb a good annotation should describe the annotated element and be meaningful outside its intended context. This last requirement is a little bit vague. I hope it will become more clear once we look at some examples.

Annotations are very often used as a convenient means of configuration. This type of usage breaks the above rule for ‘perfect annotations’, but they are very convenient and very widely used. Drawbacks of this type of usage are:

To change the configuration the source code needs to be recompiled.
The annotations (and thus their containing packages and often all of the framework code) need to be on the classpath for compilation and running.
Only one configuration can (usually) be specified. Most frameworks allow for a different means of configuration to circumvent this.

…but these drawbacks do not mean that this type of usage has been less successful in the wild. On the contrary! There are however ways to design frameworks so that they can be configured by annotations that are not impacted by these drawbacks.

Let’s look at the example above in the light of these

The @Entity annotation specifies that instances of this class should contains data that is to be persisted. It describes how the class is intended. It also allows several frameworks to operate on it.
The @Table annotation is pure configuration. It does not describe anything about this class and is tightly bound to its context: Databases.
The @Valid and @NotEmpty annotations describe features of valid instances of this class. They are however quite tightly bound to their context: data validation.

For each of the annotations in the example such a analysis can be made. It is interesting to think about how useful the annotations are in relation how perfect they are: The @Entity annotation, although being almost perfect is, is actually superfluous, because it will never be used without the @Table annotation. The @Valid and @NotEmpty annotations are much more useful: They allow very powerful cross-cutting concerns to be implemented with very little code, configuration or dependency of application code on framework code, making it easy to test and reuse this code.

Configuration annotation patterns in the wild

The Worst

All this dribble about perfect annotations is nice and dandy, but where does it lead us? Maybe it is better to look at the other end of the spectrum, where we find the very worst of annotations:

  @XmlJavaTypeAdapter(CurrencyAdapter.class)
  private BigDecimal price;

This JAXB annotation, used to configure the way a value is represented in XML, ties the implementation of a framework interface to the field on the domain class we’re trying to configure the framework for. We could as well implement the interface in the class itself!

Better… Level of indirection

A better pattern can be found in the seam framework for configuring interceptors. The following code configures an interceptor around a business method to measure the time it takes to complete.

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Interceptors(SeamTimingInterceptor.class)
public @interface MeasureCalls {}

...
	@MeasureCalls
	public void someBusinessMethod() {
		...
	}
...

@Interceptor(around = {
	BijectionInterceptor.class,
	...
	SeamInterceptor.class,
})
public class SeamTimingInterceptor {

	@AroundInvoke
	public Object timeCall(InvocationContext invocation) throws Exception {
		long t0 = System.nanoTime();
		try {
			return invocation.proceed();
		} finally {
			long dt = System.nanoTime() - t0;
			// Log time.
		}
	}
}

The first annotation just defines a cross-cutting feature that we would like to implement in our application. We can easily use it in our application as shown by the second fragment.

Coming to the last code fragment, the actual implementation of the interceptor, i feel that the framework designers went a little overboard and were too annotation-happy: The interceptor could have simply implemented an interface:

public interface  AroundMethodInterceptor {
	Object aroundInvoke(InvocationContext invocation);
}

This would have made the contract for an interceptor much clearer.

It could also help solve the other problem with this implementation: Now, the business class needs the annotation to compile, the annotation needs the interceptor and the interceptor needs the framework classes. From this perspective, the extra level of indirection has not helped a bit.

Best… Classpath scanning and interfaces

Another approach that is quite elegant is the JAX-RS approach to extending the framework through Providers and the RESTEasy implementation for it. The code below automatically registers an exception mapper, which integrates with the framework and is used to handle EJB exceptions coming from resource methods.

@Provider
public class EJBExceptionMapper implements ExceptionMapper {

	@Context
	private Providers providers;

	@Override
	public Response toResponse(EJBException exception) {
		while (exception.getCause() != null
				&& exception.getCause() instanceof EJBException) {
			exception = (EJBException) exception.getCause();
		}
		Exception cause = exception.getCausedByException();
		ExceptionMapper delegate = providers.getExceptionMapper(cause.getClass());
		if (delegate != null) {
			return delegate.toResponse(cause);
		} else {
			return ExceptionUtil.serverError(cause.getMessage());
		}
	}
}

The good thing here is that the framework will scan for the annotation and then register the class according to its interface it implements.

Another interesting aspect is the @Context annotation, that basically works as an injector point.

Extensions for custom annotations

This mechanism can be used to configure the RESTEasy framework to handle custom annotations for custom serialization. Jackson has a similar mechanism like JAXB for configuring the way a field is seriaized with the @JsonSerializer annotation. Unfortunately this has the same drawbacks. A better approach can be implemented using the Provider mechanism on the Jackson framework itself.

@Provider
public class ObjectMapperResolver implements ContextResolver {

	private AnnotationIntrospector annotationIntrospector;

	public ObjectMapper getContext(Class type) {
		return new CustomObjectMapper();
	}

	class CustomObjectMapper extends ObjectMapper {

		/**
		 * Constructor.
		 */
		public CustomObjectMapper() {
			super(null, null, null,
					new SerializationConfig(
						DEFAULT_INTROSPECTOR,
						pair(annotationIntrospector,DEFAULT_ANNOTATION_INTROSPECTOR),
						STD_VISIBILITY_CHECKER, null),
					new DeserializationConfig(DEFAULT_INTROSPECTOR,
						pair(annotationIntrospector,DEFAULT_ANNOTATION_INTROSPECTOR),
						STD_VISIBILITY_CHECKER, null)
					);
		}
	}
}

The above class ‘provides’ the Jackson framework with a custom ObjectMapper. This is done automatically, because the class is annotated with @Provider and implements ContextResolver. Now the RESTEasy framework recognizes this as a part of the framework that will provide a ObjectMapper to inject into a @Context injection point. The Jackson framework uses the same injection point to find a ObjectMapper to use for serialization.

Now how do we get it to recognize our own annotations? The answer is given by the code below, which is used in the custom ObjectMapper.

public class CustomAnnotationInspector extends NopAnnotationIntrospector {

	private Map, String> serializerComponents =
		new HashMap, String>();

	@Override
	public Object findSerializer(Annotated am, BeanProperty bp) {
		if (property != null) {
			for (Annotation annotation : bp.getMember().getAnnotated().getAnnotations()) {
				String name = serializerComponents.get(annotation.annotationType());
				if (name != null) {
					return Component.getInstance(name);
				}
			}
		}
		return super.findSerializer(am, bp);
	}
}

This can be used as shown below, when a serializer for the @Currency is configured in the seriliazerComponents map. (We use this in combination with the Seam framework for configuration).

@Target({FIELD, METHOD})
@Retention(RUNTIME)
public @interface Currency {}

...
	@Currency
	BigInteger price;
...

Conclusion

In this blog I’ve shown a number of patterns for configuring frameworks using annotations. I’ve also shown some useful classes for extending the RESTEasy and Jackson frameworks with support for custom annotations. I hope these examples will inspire you to write your own perfect and useful annotations.

Posting complex forms with RESTEasy – Part 1

Maarten Winkels — Fri, 18 Mar 2011 21:18:24 +0000

RESTEasy is a Framework for building RESTful applications in Java. In this blog I will show how to easily build RESTful webservices that accept data from an HTML Form. We will also explore the possibilities to extend RESTEasy to handle more complex cases.

Note: All source code in this post is attached as a zipped Maven project.

Simple Example

So let’s look at a very simple example:

public class Person {
	private String firstName;
	private String lastName;
}

@Path("/person")
public class PersonResource {
	private PersonRepository repository;

	@POST
	@Consumes(MediaType.APPLICATION_FORM_URLENCODED)
	@Produces(MediaType.APPLICATION_JSON)
	public Person savePerson(Person person) {
		return repository.save(person);
	}
}

The intent of these two classes should be pretty clear: We want to expose the savePerson method as a WebService, accept data from a HTML form and produce JSON. The annotations used on the PersonResource are JAX-RS annotations.

Now notice that we both accept and return a Person object in the service method. We expect the RESTEasy framework to handle the unmarshalling from the Form fields and the marshalling to JSON. Luckily JAX-RS also comes with some handy annotations for this: The @FormParam annotation lets you define which Form field to map to which parameter or field. Unfortunately the JAX-RS specification does not talk about mapping forms to rich objects, so the signature for our service method would become

	public Person savePerson (
			@FormParam("firstName") String firstName,
			@FormParam("lastName") String lastName,
			...
			) {

which would become rather large and unwieldy rather soon, when the form has to include all the usual name and address fields.

This is where RESTEasy comes to the rescue with the @Form annotation. This annotation tells the RESTEasy framework to map the entire form onto a parameter object, relieving us from the task to specify all form fields in the parameter list of the service method. The code thus becomes:

public class Person {
	@FormParam("firstName") private String firstName;
	@FormParam("lastName") private String lastName;
}

@Path("/person")
public class PersonResource {
	private PersonRepository repository;

	@POST
	@Consumes(MediaType.APPLICATION_FORM_URLENCODED)
	@Produces(MediaType.APPLICATION_JSON)
	public Person savePerson(@Form Person person) {
		return repository.save(person);
	}
}

How to test this?

With these two simple classes, we’re done implementing our simple requirements. But how do we test this? RESTEasy comes with a JUnit extension that is very useful to test Resources.

@RunWith(MockitoJUnitRunner.class)
public class PersonResourceTest extends BaseResourceTest{

	private PersonResource resource;
	@Mock private PersonRepository repository;

	@Before
	public void setupResource() {
		MockitoAnnotations.initMocks(getClass());
		resource = new PersonResource();
		resource.setRepository(repository);
(1)		dispatcher.getRegistry().addSingletonResource(resource);
	}

	@After
	public void removeResource() {
		dispatcher.getRegistry().removeRegistrations(PersonResource.class);
	}	

	@Test
	public void shouldSavePerson() throws Exception {
(2)		MockHttpRequest request = post("/person");
		request.addFormHeader("firstName", "Maarten");
		request.addFormHeader("lastName", "Winkels");

(3)		ArgumentCaptor personCaptor = ArgumentCaptor.forClass(Person.class);
		when(repository.save(personCaptor.capture())).thenAnswer(new ArgumentAnswer(0));

		MockHttpResponse response = new MockHttpResponse();
(4)		dispatcher.invoke(request, response);

		verify(repository).save(any(Person.class));
(5)		Person person = capturePerson.getLastValue();
		assertEquals("Maarten", person.getFirstName());
		assertEquals("Winkels", person.getLastName());

(6)		assertEquals(200, response.getStatus());
		assertEquals("{\"firstName\":\"Maarten\",\"lastName\":\"Winkels\"}", response.getContentAsString());
	}
}

The test above is a true integration test as far as RESTEasy is concerned, in the sense that an HttpRequest is processed by the framework and triggers our PersonResource to produce a JSON response.

Mockito is used to be mock out the PersonRepository (which is not yet implemented), and to be able to check some behavior of the code in the test. Stepping through the code:

The PersonResource with the mock for the repository is registered with the framework in the test. We need to register it as a singleton here, wince we have to prepare the resource with a mock. Notice that the class is removed in the @After method to ensure that the tests have no side-effects.
A MockHttpRequest is prepared to be processed with certain values.
The mock repository is setup to capture the argument coming into the request and to return the same argument as response. The ArgumentAnswer is a small extension of the Mockito/Hamcrest framework to enable the mock to return the parameter passed into the call on the mock as the return value.
The request is executed on the dispatcher, capturing the response in a MockHttpResponse.
The Person we captured is inspected to see that the values are copied from the request.
The response is inspected for return value and content.

That was easy! We’ve integration tested our entire RESTful application without having to start a container!

A little more complex: nested object structures and forms

Let’s try something little more complex: Let’s assume the person has an address. We’ll code the test first:

	@Test
	public void shouldSavePersonWithAddress() throws Exception {
		MockHttpRequest request = post("/person");
		request.addFormHeader("firstName", "Maarten");
		request.addFormHeader("lastName", "Winkels");
		request.addFormHeader("address.street", "Dorpsstraat");
		request.addFormHeader("address.houseNumber", "19a");

		ArgumentCaptor personCaptor = ArgumentCaptor.forClass(Person.class);
		when(repository.save(personCaptor.capture())).thenAnswer(new ArgumentAnswer(0));

		MockHttpResponse response = new MockHttpResponse();
		dispatcher.invoke(request, response);

		verify(repository).save(any(Person.class));
		Person person = capturePerson.getLastValue();
		assertEquals("Maarten", person.getFirstName());
		assertEquals("Winkels", person.getLastName());
		assertEquals("Dorpsstraat", person.getAddress().getStreet());
		assertEquals("19a", person.getAddress().getHouseNumber());

		assertEquals(200, response.getStatus());
		assertEquals("{\"firstName\":\"Maarten\",\"lastName\":\"Winkels\",\"address\":{\"street\":\"Dorpsstraat\",\"houseNumber\":\"19a\"}}", response.getContentAsString());
	}

As you can see from the test, a person now has an address and we expect the form fields that start with “address.” to be mapped to the fields in the Address class. In a sense, the form is now mapped to the nested object structure of person and address. Unfortunately neither JAX-RS nor RESTEasy has support for this. (RESTEasy does support a @Form annotation in a @Form mapped parameter class, but it does not support specifying a prefix, thus only supporting a single nested object of the same type.)

So we introduce a new annotation to indicate that a field is mapped to a part of the form fields:

public class Person {
...
	@NestedFormParams("address")
	private Address address;
...
}

To the RESTEasy framework, this new annotation means a new way of injecting values into objects, which is rather similar to how values are injected into @Form annotated parameters. *Injectors are created by an InjectorFactory. The ExtendedInjectorFactory below extends the RESTEasy InjectorFactoryImpl to make RESTEasy aware of the new annotation.

public class ExtendedInjectorFactory extends InjectorFactoryImpl{

	private final ResteasyProviderFactory factory;

	public ExtendedInjectorFactory(ResteasyProviderFactory factory) {
		super(factory);
		this.factory = factory;
	}

	@Override
	public ValueInjector createParameterExtractor(Class injectTargetClass, AccessibleObject injectTarget, Class type, Type genericType, Annotation[] annotations, boolean useDefault) {
		NestedFormParams param = FindAnnotation.findAnnotation(annotations, NestedFormParams.class);
		if (param != null) {
			return new NestedFormInjector(type, factory, param.value());
		}
		return super.createParameterExtractor(injectTargetClass, injectTarget, type, genericType, annotations, useDefault);
	}

}

In the test, the new InjectorFactory is registered with the ProviderFactory:

	@Before
	public void setupResource() {
		getProviderFactory().setInjectorFactory(new ExtendedInjectorFactory(getProviderFactory()));
		...
	}

Whenever a NestedFormParams annotation is found, a NestedFormInjector is created to handle the injection. Since the functionality is so similar to the FormInjector, it simply is an extension of that class.

public class NestedFormInjector extends FormInjector {

	private final String prefix;

	public NestedFormInjector(Class type, ResteasyProviderFactory factory, String prefix) {
		super(type, factory);
		this.prefix = prefix;
	}

	@Override
	public Object inject(HttpRequest request, HttpResponse response) {
		if (!containsPrefixedName(request.getDecodedFormParameters().keySet())) {
			return null;
		}
		return super.inject(new PrefixedFormFieldsHttpRequest(prefix, request), response);
	}

	private boolean containsPrefixedName(Set keys) {
		for (String key : keys) {
			if (key.startsWith(prefix)) {
				return true;
			}
		}
		return false;
	}
}

To understand how the NestedFormInjector works, one has to understand how the FormInjector works. The form injector is created for a specific type. It uses a ConstructorInjector to create an instance of that type and a PropertyInjector to inject other values into this instance. The injection of these values is carried out by other injectors. If a field in the class is annotated with an @FormParam annotation, the FormInjector is used to inject the value from the form parameter into the field.

The NestedFormInjector works by restricting the access of any form injectors that are used for it’s properties to form fields that start with the given prefix (and a dot). This is achieved by wrapping the original HttpRequest in an PrefixedFormFieldHttpRequest. Objects of this class wrap the original form fields in a map that will add the prefix on each look up. The code can be found in the attached ZIP file.

Conclusion

By extending a few of the core classes of the RESTEasy framework, we have added the functionality to map a HTTP form to a complex association of two classes. This can be really useful when complex forms are posted.
A next step would be to map form fields to collections, lists or maps. I’ll write about that in a next post.

Regression Testing with an Agile Mindset

Maarten Winkels — Fri, 05 Nov 2010 14:07:09 +0000

When applying agile methods to existing software development practices, it is often useful to start from the basics. For regression testing this boils down to:

Regression testing ensures continuity of business functions.
In an agile context it helps the team focus on the (new) functionality to develop in the sprint while maintaining overall stability.
To maximally utilize regression testing in an agile context, test automation and continuous execution should be applied.
It is important to realize the difference between functional tests and regression tests to avoid some common pitfalls.

Regression testing ensures continuity of business functions
The target area of the regression test is the complete set of (end-to-end) business functions that a system encompasses. The same tests are executed repeatedly over time to ensure that the required behaviours of the system remain stable.
In an agile context this means that regression test results are of great value to the product owner at the end of each sprint, when he has to evaluate and accept not only the newly developed functionality, but the overall functionality of the system after the changes have been made. Ideally, there should be no regression in the functionality of the system. In real-life, complex systems however, a trade off can be made between the immediate value of the new functionality and regression bugs that have (unintentionally) been introduced.
But the Product Owner should be very careful! A zero-tolerance policy towards bugs is in the long run far more effective than accepting even small regressions for the sake of faster development. Therefore it is better to decrease the scope then to accept regression.

Of course expectations and sometimes entire tests have to be adapted as business functions change or new tests have to be added when additional functionality is developed. In an agile context, where change is continuous and gradual, changes to business functions should not be disruptive. Maintaining the regression test suites can be a relatively easy job.

Focus on new functionality
One of the main success factors of the agile approach (specifically SCRUM) is, that it allows the entire team to focus on the functionality that has been planned for the current sprint. This focus allows the team to aim for the best quality of the solution in every aspect, be it architecture, design, implementation or testing. In a complex environment, you cannot expect a team to focus on a particular part while at the same time understanding the consequences and risks for the entire system. Regression tests provide a safety net for the team in this situation. Any areas of the environment that might have been missed in the analysis of the problem or the proposed solution, will be shown by the regression test provided that the coverage of the regression test suites is sufficient.

Continuous automated regression tests give rapid feedback
To allow the team to respond efficiently to any findings from the regression tests, it is imperative that the regression tests are executed shortly after changes have been made. Late feedback means that it is harder to track the source of the unexpected behaviour, to analyse the error and to correct the implementation. The team might have moved on to implement other changes in other areas of the system, which means that it will take more effort to switch back to the area in which the problem was found. Since the problem did not surface during functional testing, it is very probable that the cause of the problem is of a more structural nature and the architecture and design of the solution have to be re-evaluated. Most likely new functional tests have to be added to ensure that the issues found are not just the tip of the iceberg.

The required rapid feedback can often only be achieved when the regression tests are executed fast and automatically. To this end regression tests need to be scripted and run on an automatic build environment.

Regression test suite != sum(Functional test suites)
The purpose of functional tests is to explore the behaviour of specific business functions and highlight corner cases. As discussed above, the purpose of regression testing is to give and overall overview of the entire system. These two purposes are distinct and therefore functional test suites should not simply be used as regression tests.

In an agile context, functional tests are used to prove that the team has delivered the functionality that is planned for a sprint. The development of the functional test cases is part of the sprint in which the functionality is developed. The functional test cases are thus a basis for the product owner to accept the new functionality. In many agile teams I have seen that functional tests that have been developed for previous sprints are run in succession as regression tests for the current and coming sprints. Although this is an easy and cheap approach to build a regression test suite, there are a number of problems with this approach:

Since the system is developed in iterative and evolutionary fashion, the same business functions are often changed and enhanced multiple times over many sprints. As a result many functional tests are written for the same business function. This means a lot of “test duplication”. Since the business function is changing, the old test scripts have to be adapted and the required effort will grow with every sprint.
At the same time, the value of the regression tests has not increased, since the same business function is being tested. The overall overview of the system has not improved, but rather been obfuscated by adding more data.
As a side effect it also takes longer to execute the regression tests and to analyse the results.

Therefore it is better to work with (or towards) a smaller, relatively stable set of regression tests. This set should be constructed on the basis of knowledge of the business functions and of the technical implementation, to ensure sufficient but not superfluous coverage. It is thus the responsibility of the entire team.

So how does the team develop this set of regression tests?

The product owner compiles a list of all business functions that should be included in the regression test. This can happen gradually over several sprints while the team learns what functions are important. Creating the tests can also be done gradually.
Functional tests in the sprint are basically “throw away” tests: They are not intended to be maintained for long. Perhaps you keep the tests so you can update and reuse some of them later.
Some functional tests can be upgraded to regression tests, when they cover certain business functions better then existing tests. The existing tests can then be taken out of the regression test suite.
Regression test scripts are updated in the sprint when necessary. This is part of the UserStory and should thus be taken into account when estimating.

Conclusions

Regression testing has an important role in Agile development. In addition to its usual purpose of proving stability, it helps the team focus on the functionality in the current sprint.
Although it is tempting to use the functional tests of the past sprints as regression tests for the current sprint, this will lead to an unwieldy regression test suite and will blur the purpose of the regression test suite by introducing a lot of duplication. In stead the regression test suite should be constructed to reflect the (most important) business processes.

Tweaking JAXB to generate better code

Maarten Winkels — Wed, 02 Sep 2009 21:03:07 +0000

JAXB can be a real time saver when working on a project that uses XSD to describe interfaces implemented in Java. Sometimes, however, the generated code is not up to standard. I ran into a problem that seems very common and found a very elegant solution to it on the web.

Yesterday, while working on the XML Schema for Document, I had to decide whether I would wrap collections in a surrounding element, like this:


...
  
    
      ...
    
    
      ...
    
    ...
  
  ...

or I would just have a sequence of -elements directly under Document. I think the general consensus is to use wrapping elements, but I decided to base the design on another measure: What would the (Java) code look like that would be generated from the Schema by JAXB.

After configuring the maven-jaxb-plugin into the project (which was really easy and it integrated out of the box with Eclipse through M2Eclipse), the Java code was a little disappointing: For every collection I would get the following structure:

class Document {

  @XmlElement(name = "Replies")
  private Replies replies;

  public Replies getReplies() {
    return replies;
  }

  public class Replies() {

    @XmlElement(name = "Reply")
    private List reply;

    public List getReply() {
      return reply;
    }
  }
}

This would lead to the following usage pattern:

  Document doc;
  ...
  doc.getReplies().getReply().add(new Reply(...));

which is in my opinion quite awful. I would expect there to be a simple collection structure:

  Document doc;
  ...
  doc.getReplies().add(new Reply(...));

But whether I would use the wrapping element or not, I would still get the additional class.

There is a JAXB annotation that will help in this situation:

  @XmlElementWrapper(name = "replies")
  @XmlElement(name = "reply")
  protected List replies;

And this is exactly what we want, but how to tell the JAXB code generator to use that annotation?

After some googling around, I found this blog posting, that provides a solution by using a custom plugin for XJB (the JAXB code generator). Configuring this in maven was quite easy:

Add the plugin jar (which was unfortunately not available from a Maven repo) to the maven-jaxb-plugin dependencies.
Add an -element to the plugin configuration with the -Xxew option to activate the plugin.

The final Maven configuration looks like this:


  com.sun.tools.xjc.maven2
  maven-jaxb-plugin
  
    
      jaxb-generate
      
        generate
      
    
  
  
    
      xsd/Document.xsd
    
    true
    true
    -Xxew
  
  
    
      xew
      xew
      1.0.0
      runtime

The generated code now uses a simple collection, so it looks much more usable.

One last problem is that the generated Replies class, that is now no longer used, is not removed from the code: A little clean up task that must have slipped the plugin developers mind.