It kind of snuck up on us, but when we recently checked the blog visitor statistics, we found that we had gone over 20000 unique visitors per month in april! So to all of you who've stayed with us through the past years, to all the Xebians and ex-Xebians who have been contributing posts, and to all who commented on the blog: a big thank you. We hope we can keep offering the content that will push us to, let's say, 50000!

A while ago, I compared Preon with Erlang's bit syntax. I looked at one one of the examples from "Programming Erlang" in particular; an example that illustrates how to decode MPEG headers using Erlang. However, this is not the only example in that chapter, so I decided to take a stab at one of the other examples as well.

The second example from the bit syntax chapter in "Programming Erlang" is about unpacking COFF data. The thing about COFF is that it doesn't have an IDL-alike language or anything for defining the data structures: all you have is the definition of C++ data structures, such as the one below:

typedef struct _IMAGE_RESOURCE_DIRECTORY {
DWORD Characteristics;
DWORD TimeDateStamp;
WORD MajorVersion;
WORD MinorVersion;
WORD NumberOfNamedEntries;
WORD NumberOfIdEntries;
} IMAGE_RESOURCE_DIRECTORY, *PIMAGE_RESOURCE_DIRECTORY;

In his book, Joe Armstrong explains that using Erlang's bit syntax and macro solutions, you would be able to unpack COFF data characterized by the C++ struct above using the Erlang code listed below.

unpack_image_resource_directory(Dir) ->
  <<Characteristics : ?DWORD,
    TimeDateStamp : ?DWORD,
    MajorVersion : ?WORD,
    MinorVersion : ?WORD,
    NumberOfNamedEntries : ?WORD,
    NumberOfIdEntries : ?WORD, _/binary>> = Dir,
...

The key message here is that Erlang not only allows you to unpack binary data easily, but that it also allows you to express that clearly maps to the only source of definition of the data structure: the C++ API.

Now, if you would use Preon only, the C++ data structure above would translate to this:

class ImageResourceDirectory {
  @BoundNumber(size="32") long characteristics;
  @BoundNumber(size="32") long timeDateStamp;
  @BoundNumber(size="16") int majorVersion;
  @BoundNumber(size="16") int minorVersion;
  @BoundNumber(size="16") int numberOfNamedEntries;
  @BoundNumber(size="16") int numberOfIdEntries;
}

... and you would be able to decode it by this:

Codec<ImageResourceDirectory> codec = Codecs.create(ImageResourceDirectory.class);
Codecs.decode(codec, ...);

Now, that's not bad, but it doesn't have that similarity to the original C++ API code the Erlang example has. However, using Andrew Philips' @Composite framework, you would actually be able to write this:

class ImageResourceDirectory {
  @DWORD long characteristics;
  @DWORD long timeDateStamp;
  @WORD int majorVersion;
  @WORD int minorVersion;
  @WORD int numberOfNamedEntries;
  @WORD int numberOfIdEntries;
}

... which is already a lot closer to the original C++ struct than what we had before.

Support for @Composite has not been included in Preon yet. At first sight, there appear to be two ways to deal with it. First of all, it could be build into the framework, by the AnnotatedElements interface all over the place. That should work, and it might actually be the most sensible thing to do.

However, there may be an alternative way to get it woven in. Preon already defines a CodecDecorator interface that allows you wrap Codec implementations around Codec instances created. Looking at that, I started to think that it might actually be quite attractive to also define a CodecFactoryDecorator, wrapping CodecFactories around other CodecFactories in the chain of responsibility.

public interface CodecFactory {
    <T> Codec<T> create(AnnotatedElement metadata, Class<T> type,
            ResolverContext context);
}

The wrappers created by the CodecFactoryDecorator would be able to intercept any reference to annotations passed down below down the chain, and replace it with an AnnotatedElement that uses AnnotatedElements to gain access to the annotations. As a consequence, CodecFactories responsible for creating the actual Codec from metadata passed in would get to see Preon annotations only, instead of the @DWORD and @WORD annotations.

It's just a thought. None of this has been implemented yet. Feel free to comment. Expect to see some more of this in the future.

Scala has become more and more popular over the recent months/years. Its hybrid nature of being an imperative as well as functional language attracts a crowd from the Java world as well as functional fundamentalists coming from the world where statements like x=x+1 are looked at with the utter disbelief. It has been stated that Scala is 'Java as it should have been', but there are also numerous complaints about the language and its features (like not being side effect free, overly complex, too much of everything, too much abstraction, having a weird syntax, etc). The latter might actually be a proof of its popularity, since people seem to be actually using the language instead of just looking at it briefly and stopping, tired but happy, after having written hello world with it.
In this blog post, I'll give you some (hopefully) useful tips how to best start if you want to learn this language, which is one of the candidates become 'our next big language' and surpass Java in this respect.

Pick an IDE, or not
The first problem you might encounter is that the various Scala IDE plugins have not yet reached the maturity of the Java support that you might have been accustomed to. Lots of hard work is in progress on this, but you still might experience hickups and freezes (as I have) of your favorite IDE. At this moment, the Netbeans plugin seems to be the most stable (this is the IDE that David Pollak, creator of Lift, seems to use). If you're an eclipse user, you'll probably want the nightly build version of the Scala plugin, since that's the only one actively under development. This means that you need the Scala trunk to work with, however.

There's also plenty of support for non-IDE's, and recently I've tried switching back to emacs again. For some, this brings back memories to good old university days, others may find it the utmost horror. I must say, I found it to be a rather soothing experience after years of development using Eclipse. You'll need to do some setup work to get a good Scala experience. An excellent blog post describing how to setup your emacs environment for Scala using Yasnippet, exuberant ctags and maven can be found here.

Try out the trunk
The latest released version of Scala is 2.7.5. The next version will be 2.8.0, but its release date is still not known. However, the invaluable Paul Philips has done many bug fixes and improvements in the Scala compiler and the REPL, all hanging out in the trunk. To take full advantage of this, check out the sources and build them. Note that the Scala collections API has been redesigned in the trunk, and differs from the 2.7.5 release. To get a good understanding of the new design, and the API in general, you're encouraged to read the new collection SIP (Scala improvement proposal) available here.

Read a book
The standard work is Programming in Scala, co-authored by Martin Odersky (creator of Scala) himself. This book is lengthy but very thorough, providing plenty of examples. Another book that has seen the light is Beginning Scala by David Pollak (already mentioned). I have not read this, but reviews suggest this is also an excellent start. More books will hit the market this year.

Check out Lift
Eventually it is possible that you might start to really like Scala and want to use it in the enterprise. Lift is the first web framework written in Scala, created by David Pollak. Downloading and creating a simple web application to work is a no-brainer. The distribution comes with plenty of maven archetypes to get a basic web application ready in under than five minutes. Lift has its own database mapping framework, but there's JPA support as well, if you like this. JTA integration is also being worked on.

Read some articles
There are loads of (both academic and slightly less academic) articles about Scala. For example, if you want to know the rationale behind Scala's actor library design, read the scala actor papers.
Martin Odersky's homepage also contains a long list of his publications.
It provides lots of material to provide insight in the design of Scala and its libraries. If you're not that academically inclined and from a Java background, try The busy java developer's guide to Scala, by Ted Neward. Provides a nice introduction.

Read some blogs.
Plenty of blogs available, here are some I like:

• Daniel Spiewak Lots of useful Scala examples.
• Tony morris. He's the creator of functional java and of Scalaz. Beware however, if you check this out you might end up in deep functional waters where Monads reign.
• Jonas BonerHe has lots of experience of using Scala in the real world, some of which he has blogged extensively about. Well worth the read.
• Planet Scala Perhaps you don't even need more than this. This aggregates numerous blog posts (including the one mentioned above) about Scala.

Join IRC
Scala has an IRC channel, which can be found here. Whenever you're stuck at some problem you're working at, there's always the mailing lists, but many knowledgeable people hang out on a daily basis on the IRC channel. Use pocoo to show the code you're stuck with, and you'll get an answer in no time.

Start coding.
Lift might be the choice if you really want to write some useful, practical real world web applications. If you're totally impractical like me, pick a few problems from project euler, or the 99 problems projects (solutions in Scala can be found here. Many excellent programmers have taken this path, it's fun and an excellent way to play around with the core Scala API.

To put into practice what I preach here, a few lines of Scala code to either whet your appetite or make you run away screaming. Note that my Scala knowledge is still in the pre-kindergarten stage, I'm barely able to speak, so laugh at will when viewing this code.

First, a randomly picked Euler problem, number 48, just because I like onliners:

(1 to 1000).map(x => new java.math.BigDecimal(x).pow(x)).reduceLeft((a,b) => a.add(b)).remainder(new java.math.BigDecimal(10).pow(10))

This one liner is not nearly as concise and neat as the Haskell version, but it will do. If nothing else, it has at least has the tiny merit of showing Scala - Java interoperability.

Another randomly picked and slightly more complex one, problem number 21:

object Euler21 {
  def sumOfDivisors(number: Int): Int = {
    List.range(1, number).filter{i => (number % i) == 0}.foldLeft(0){(a,b) => a+b}
  }

  def solver(): Int  = {
      (for (i <- 0 until 10000; di = sumOfDivisors(i); if(di > i && sumOfDivisors(di)  == i) ) yield (i+di)).foldLeft(0){(a,b) => a+b};
  }
}

Enjoy.

Last week I came across an interesting "coding kata" by Brett Schuchert on the Object Mentor blog. The trick of a kata is that you grow the program step-by-step using tests, just like a kata in karate is tought to a student. The problem of this kata was the Shunting Yard algorithm of Dijkstra. I wanted to see if I could implement this kata in Scala.

Instead of writing the algorithm as described, the trick in a coding kata is that you first write a test, and then make the test pass. And subsequently each time adding a test and keeping it all in the green. Brett described the test-cases in his blog entry.

The Shunting Yard algorithm is used to convert a mathematical function in infix notation to a reverse polish or postfix notation. For instance, it can convert an expression like 3+4 to 3 4 +.

In order to implement this algorithm, one needs to do string parsing to break up the infix string. Scala has the parser combinators that can do just this. Using a parser I build an Abstract Syntax Tree (AST) which is a representation of the formula in a tree form. An AST for 3 + 4 and (3 + 4) * 4 looks like:

The AST representation of the formula can then easily be printed in a reverse polish notation, as the code shows. The final Parser looks like this:

import scala.util.parsing.combinator.syntactical._

abstract class Expr {
  def rpn:String
}
case class BinaryOperator(lhs:Expr, op:String, rhs:Expr) extends Expr {
	def rpn:String = lhs.rpn + " " + rhs.rpn + " " + op
}
case class Number(v:String) extends Expr { def rpn:String = v }
case class Variable(v:String) extends Expr { def rpn:String = v }
case class Function(f:String, e:List[Expr]) extends Expr { def rpn:String = {
	var s = ""
	e.foreach { x => s += x.rpn + " " }
	s += f
	return s
  }
}
object ShuntingYard extends StandardTokenParsers {
    lexical.delimiters ++= List("+","-","*","/", "^","(",")",",")

    def value :Parser[Expr] = numericLit ^^ { s => Number(s) }
    def variable:Parser[Expr] =  ident ^^ { s => Variable(s) }
    def parens:Parser[Expr] = "(" ~> expr <~ ")"

    def argument:Parser[Expr] = expr <~ (","?)
    def func:Parser[Expr] = ( ident ~ "(" ~ (argument+) ~ ")" ^^ { case f ~ _ ~ e ~ _ => Function(f, e) })

    def term = (value | parens | func | variable)

    // Needed to define recursive because ^ is right-associative
    def pow :Parser[Expr] = ( term ~ "^" ~ pow ^^ {case left ~ _ ~ right => BinaryOperator(left, "^", right) }|
    			term)
    def factor = pow * ("*" ^^^ { (left:Expr, right:Expr) => BinaryOperator(left, "*", right) } |
                        "/" ^^^ { (left:Expr, right:Expr) => BinaryOperator(left, "/", right) } )
    def sum =  factor * ("+" ^^^ { (left:Expr, right:Expr) => BinaryOperator(left, "+", right) } |
    					"-" ^^^ { (left:Expr, right:Expr) => BinaryOperator(left, "-", right) } )
    def expr = ( sum | term )

    def parse(s:String) = {
        val tokens = new lexical.Scanner(s)
        phrase(expr)(tokens)
    }

    def shunt(exprstr: String) : String = exprstr match {
      case null => return ""
      case "" => return ""
      case _ =>
    	parse(exprstr) match {
            case Success(tree, _) =>
                println("Tree: "+tree)
                val v = tree.rpn
                println("RPN: "+v)
                return v
            case e: NoSuccess => Console.err.println(e)
            	return e.toString
        }
    }
}

Of course I tested this using the following unit test, of which I will show only a small part, one can easily complete this by looking at Brett's examples:

import org.junit.Test
import org.junit.Assert._

class ShuntingYardTest {
  @Test
  def test_11() {
    assertEquals("4 g f", ShuntingYard.shunt("f(g(4))"))
  }

  @Test
  def test_12() {
    assertEquals("3 4 19 f", ShuntingYard.shunt("f(3, 4, 19)"))
  }

  @Test
  def test_13() {
    assertEquals("3 4 2 * 1 5 - 2 3 ^ ^ / +", ShuntingYard.shunt("3 + 4 * 2 / ( 1 - 5 ) ^ 2 ^ 3"))
  }

  @Test
  def test_14 () {
    assertEquals("4 5 + 1 a 2 ^ + 8 b + 10 * f", ShuntingYard.shunt("f(4+5,1+a^2,(8+b)*10)"))
  }
}

Of all the tests that Brett defined, only one doesn't pass, can you spot which one? Of course as this is one of my first exercises in Scala, it probably isn't optimal yet, any improvements are welcome in the comments!

Apache Hadoop promises "a software platform that lets one easily write and run applications that process vast amounts of data". Sure enough, when reading the documentation, descriptions like:

(input) <k1, v1> -> map -> <k2, v2> -> combine -> <k2, v2> -> reduce -> <k3, v3> (output)

Are simple enough to read and understand, but how do you apply MapReduce to a problem you face in a real-life project?

This blog tries to give some insight into how to apply MapReduce with Hadoop.

What is Hadoop?

Hadoop is basically two things:

1. A distributed file system -- HDFS
2. A MapReduce framework that allows algorithms to work on data in the distributed file system in parallel

The Hadoop Distributed File System (HDFS) is really the heart of Hadoop. It provides scalability, reliability and performance at a low cost. The system is designed to run on commodity hardware. Although the system is written in Java, there are other ways to access and use it.

MapReduce is a software framework that allows computation to run on a cluster. It uses HDFS for data-proximity: The computation will be distributed and run in parallel on the cluster and each process will access and process data that is available locally on its node. This gives a major performance boost. Furthermore the framework provides reliability, because processes that fail will automatically restarted at other nodes.

When to apply MapReduce?

MapReduce is only useful for systems that process large amounts of data. There is a overhead for starting tasks and there is always the network overhead. When talking about large amounts of data we mean GBs and above rather then MBs.

Another important aspect is the data usage pattern in your application. If you need to 'randomly' read and write data, for example based on a certain request coming in, MapReduce cannot help you. MapReduce really shines when data is read in a batch-like streaming manner.

The final requirement for using MapReduce is that the algorithm can be described as a map-and-reduce process. In this blog I want to focus on this last aspect.

How to apply MapReduce? - Another example

So how do you describe an algorithm in a MapReduce manner? To illustrate, nothing works better than an example. As with every example that I have seen for Hadoop, it is a bit academic. What I'm trying to explain is how a MapReduce algorithm is different from a normal approach and how to go about designing that algorithm.

The main thing with a MapReduce algorithm is that it reasons about < key , value > pairs all along, from the input format to the output format, if necessary using synthetic keys. If your input is a simple flat file, it will by default break it up on line ends and provide the offset into the file as key and the line as value. The main strength of the algorithm lies with the fact that between the map and the reduce phase, it will sort the data by key. The framework will then provide all data with the same key to the same reducer instance. Any successful MapReduce algorithm should leverage this mechanism.

The problem - Finding Anagrams
Say you have to find Anagrams in a very large input file. How would implement this?

I think a first attempt would have some sort of function like this:

 
  public static boolean isAnagram(String first, String second) {
    // Checks that the two inputs are anagrams, by checking they have all the same characters.
    // Left as exercise for the user...
  }
 

The application would have to somehow execute this function on all pairs of words in the input. However fast this method would be, the overall execution would still take quite some time.

Hadoopifying...
How do you now design a MapReduce algorithm that will give the desired answer? The key lies in finding a function that will produce the same key for all words that are anagrams. Applying this in the map phase will use the power of the MapReduce framework to deliver all words that are anagrams to the same reducer. The solution, when found, is deceivingly simple as usual:

 
  public static String sortCharacters(String input) {
    char[] cs = input.toCharArray();
    Arrays.sort(cs);
    return new String(cs);
  }
 

By sorting all the characters in all the words in the input, all anagrams will have the same key:

  aspired -> adeiprs
  despair -> adeiprs

Now the list of characters to the right has no meaning, but all anagrams will have exactly the same result for this function.

Implementation
Once the algorithm is found the implementation using Hadoop is quite straightforward and simple (though pretty long...).

public class AnagramFinder extends Configured implements Tool {

  public static class Mapper extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, Text> {

    private Text sortedText = new Text();
    private Text outputValue = new Text();

    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
      StringTokenizer tokenizer = new StringTokenizer(value.toString(),
          " \t\n\r\f,.:()!?", false);
      while (tokenizer.hasMoreTokens()) {
        String token = tokenizer.nextToken().trim().toLowerCase();
        sortedText.set(sort(token));
        outputValue.set(token);
        context.write(sortedText, outputValue);
      }
    }

    protected String sort(String input) {
      char[] cs = input.toCharArray();
      Arrays.sort(cs);
      return new String(cs);
    }

  }

  public static class Combiner extends org.apache.hadoop.mapreduce.Reducer<Text, Text, Text, Text> {

    protected void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
      Set<Text> uniques = new HashSet<Text>();
      for (Text value : values) {
        if (uniques.add(value)) {
          context.write(key, value);
        }
      }
    }
  }

  public static class Reducer extends org.apache.hadoop.mapreduce.Reducer<Text, Text, IntWritable, Text> {

    private IntWritable count = new IntWritable();
    private Text outputValue = new Text();

    protected void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
      Set<Text> uniques = new HashSet<Text>();
      int size = 0;
      StringBuilder builder = new StringBuilder();
      for (Text value : values) {
        if (uniques.add(value)) {
          size++;
          builder.append(value.toString());
          builder.append(',');
        }
      }
      builder.setLength(builder.length() - 1);

      if (size > 1) {
        count.set(size);
        outputValue.set(builder.toString());
        context.write(count, outputValue);
      }
    }

  }

  public int run(String[] args) throws Exception {
    Path inputPath = new Path(args[0]);
    Path outputPath = new Path(args[1]);

    Job job = new Job(getConf(), "Anagram Finder");

    job.setJarByClass(AnagramFinder.class);

    FileInputFormat.setInputPaths(job, inputPath);
    FileOutputFormat.setOutputPath(job, outputPath);

    job.setMapperClass(Mapper.class);
    job.setCombinerClass(Combiner.class);
    job.setReducerClass(Reducer.class);

    job.setMapOutputKeyClass(Text.class);
    job.setMapOutputValueClass(Text.class);

    return job.waitForCompletion(false) ? 0 : -1;
  }

  public static void main(String[] args) throws Exception {
    System.exit(ToolRunner.run(new Configuration(), new AnagramFinder(), args));
  }
}

The main parts of the implementation are the following:

Mapper - Breaks up the input text in tokens (filtering some common punctuation marks) and applies the character sorting to arrive at the required key.
Combiner (optional) - Removes duplicate values from the input.
Reducer - Collects anagrams and outputs the number of anagrams (key) and all the words concatenated (value).
Main and Run - This code configures the job to run on the MapReduce framework.

The Combiner is used to do some preprocessing for the reducer. The main reason for this is that results from Mappers that are run on different nodes will be processed on the same node and will thus have to travel the network. To minimize network load, the Combiner might reduce the number of < key , value > pairs that will be processed, as shown here by filtering out duplicates. The process can however not rely on all < key , value > pairs to be processed by a single Combiner, so the Reducer will also have to remove duplicates.

It is interesting to see that the concept Anagram doesn't materialize anywhere in this code. The fact that the code finds anagrams follows from the fact that all anagrams will have the same value from the sort function and that is used as the map output key. This might be quite confusing for readers.

Conclusion

The main challenge posed by Hadoop is coming up with a good algorithm for MapReduce applications. The algorithm will mostly be the result of the whole MapReduce process and might not be easy to understand from the code. This is because some of the functionality that the framework provides might be key to the algorithm. Good documentation that describes the whole process is vital to overcome this problem. Once an algorithm is designed, impelementing it in Hadoop is quite straightforward.

Last week I co-organized an nlscrum event with a very special guest: Jeff Sutherland. After rushing with him from the airport to our Xebia office, Jeff gave a very inspiring presentation.

Jeff talked about the dramatic difference between those teams that take the red pill (that Morpheus offered Neo in the Matrix), and the blue pill that most people take. After his presentation, many attendees felt like they had just taken the red pill. I hope that this inspiration lasted long enough to have some effect at work, where countless small and big obstacles cause many people's pill to be blue.

After dinner, which was accompanied by a nice Italian ice cream booth, we had a question and answer/discussion session in which Jeff touched many Scrum related topics. Usually at nlscrum events, we host an OpenSpace to give members of the nlscrum community a chance to share experiences. This time, we decided to opt for a different format to give a maximum of attendees the opportunity to learn from and get inspired by Jeff.

The topic that triggered most debate was a discussion about Kanban versus Scrum. According to Jeff, Lean principles and techniques are important to do Agile software development projects successfully. In fact, last month Jeff gave a Deep Lean course with Henrik Kniberg and the Poppendiecks. For those interested, you can find the course material online, including a presentation about Kanban versus Scrum by Henrik Kniberg.

All in all, I'm very glad by the way this special nlscum event turned out to be. We had approximately 65 attendees and got a lot of positive feedback. I hope to see many of them again on our OpenSpaces. So, for all of you in the Netherlands who use Scrum or plan to do so, come to our nlscrum events to get inspired and learn from your peers!

Photos: Laurens Bonnema

Last time I blogged about the importance of benchmarking the architecture and new technology in a Proof of Concept for Performance. This time I’ll deal with the importance of representative performance testing.

Slowness of applications in development environments is often neglected with the rationale that faster hardware in the production environment will solve this problem. However, whether this is really true can only be predicted with a test on a representative environment and in a representative way. In such an environment, there needs to be more representative than just the hardware.

I have experienced multiple times that a database query on the test database with 1000 customers took only less than 10 ms., while on the production database with 100.000 customers this turned out to take tens of seconds, because of missing indexes. So, if the development team does not test with a full, complete database, going to production may lead to some surprises.

It is also important that the number of concurrent users and their behavior is well simulated in the test. Furthermore, care should be taken to take caching effects into account: if the test continuously requests for the same product by the same customer, this data will be in database or query cache the second and following times. This will speed up the request considerably and be much faster than with many customers and products. This test is therefore not representative for the real situation.

A suitable performance test tool and performance expertise is necessary to create a valuable test. The most popular open source performance test tool is Apache JMeter, see the next figure.

Figure: Screenshot of a run of a performance test in Apache JMeter.

This is a tool made by programmers, for programmers. Test scripts can be created with visual elements like a HTTP request, which can be recorded and configured. Many are available and if you need more, you can always fall back on a BeanShell element in which you can manipulate the request, response and various JMeter variables. If that even does not meet your needs yet, you can extend JMeter source code and develop your own elements. Because of its for-programmers nature, it is less suited for the average tester. Also reporting features and maintainability of the scripts are both not so great. Therefore, commercial tools like HP Mercury LoadRunner, Borland SilkPerformer or Neotys’ Neoload may be good alternatives for companies.

Next time I’ll blog about step 4: continuous performance testing.

Geertjan Wielenga has been trying to pull me back into the NetBeans community for a couple of years in a row now. I admire his perseverance; if this is typical for the whole NetBeans team, then Eclipse is going out of the window some day soon.

There is one thing - really, just one thing - that would make me drop Eclipse immediately in favor of NetBeans. That's having better support for fluent interfaces in the way the IDE formats source code.

Now, I've been working on a couple of fluent interfaces over the last couple of years, and it's just awesome. It will always result in code that is easier to read, and it doesn't cost you a dime; you get the benefits of Java 5 type safety , without sacrificing readability.

Let's take this Pecia example:

doc
    .section("Introduction")
        .para()
            .text("This is a document. Make sure you also check out the ")
            .emphasis("next").text(" section.")
        .end()
    .end()
    .section("Conclusion")
        .para("That's all folks.")
    .end()
.end();

Now, IMHO, this is pretty easy to read and understand. The layout of your code clearly reflects the structure of the underlying document model. However, if you press Command-Shift-F to format your code, this is what you get:

doc.section("Introduction").para().text("This is a document. Make sure you also check out the ").emphasis("next").text(" section.").end().end().section("Conclusion").para("That's all folks.").end().end();

Not quite as good as what we had before.

Now, I would love to have a solution that basically prevented this. I was thinking about this for a while, and I could imagine introducing a couple of annotations for it. Annotations for fluent interfaces. Annotations that basically tell your IDE how to treat different components in your fluent interfaces for formatting.

Maybe annotations on the type of object produced by the section() operation, informing the compiler to treat this object as a "code block", in terms of the indentation. And perhaps an annotation on the .end() method telling the IDE to consider it the end of the "code block".

So, this is to you Geertjan: I solemnly swear to erase Eclipse from my hard disk, as soon as something like this gets implemented in NetBeans. Maybe an annotation based approach is what is needed, maybe it isn't; I don't really care, as long as something gets done about this.

(If anyone else has some thoughts on this, I'd be happy to hear about it.)

1. Introduction

There is a chance that - after having read this article - you conclude that nobody in a sane state of mind would ever use what this article is going to suggest. Let me therefore start with disclaimer: I have never made any public claims regarding my state of mind.

Apart from that, I figure an article about a technology almost nobody is using, is still way more interesting than an article everyone is using. In fact, I guess the more senior you are, the more things you have already seen before, the less likely it is you will be you will be interested in something people already did many times before. Based on that, you might as well say that the most experienced people around will probably be interested in stuff that nobody is using. This article is for those people.

With that out of the way: Pecia is a new way of generating documentation from your Java applications. You will probably wonder why we need yet another way of generating documents from Java, and I have to admit that the Java world is not in a bad shape if it comes down to the number of frameworks allowing you to generate documents. However, Pecia takes another stab at it, and I just had to see if it would work. You be the judge whether it makes sense.

2. Background

I clearly remember the day on which I spend more time than I am willing to admit on finding out why my Maven report was not producing well-formed HTML. Definitely not one of my finest moments.

Maven reports are built using an API called Doxia. It's not a general purpose template-based text generating mechanism, like Velocity, FreeMarker or StringTemplate. In fact, there is no template at all. Instead, Doxia provides an API for building a document, abstracting the final representation of that document.

In a way, the most important interface in Doxia is the Sink interface. The Sink interface is basically the builder interface. It has operations to start the document body, to start a section, to generate text, to start a table, a table row, a table cell, and many other document chunks, and then a slew of operations to finish all of those. Example 1, “Doxia Sink usage” shows an excerpt of some of my code using Doxia.

Example 1. Doxia Sink usage

sink.body();
sink.sectionTitle1();
sink.text("Message Catalog");
sink.sectionTitle1_();
sink.table();
sink.tableRow();
sink.tableHeaderCell();
sink.text("Type");
sink.tableHeaderCell_();
sink.tableHeaderCell();
sink.text("Identifier");
sink.tableHeaderCell_();
sink.tableHeaderCell();
sink.text("Message");
sink.tableHeaderCell_();
sink.tableRow_();
...

Unmistakeably, there is a correspondence between the Sink interface and a subset of the HTML content model. However, HTML is not the only type of document that can be generated from Doxia. In fact, by abstracting the interface from the implementation, Doxia is capable of generating any type of output document. Because of that, it generates PDF just as easy as HTML.

Now, there is one limitation in the Doxia approach: you are never really sure if your code is building a valid document. The API does not prevent you from adding an image to a table row, or inserting text in a table row outside of a table cell. And since Doxia aims to abstract you from the target representation, it's quite hard to make any assumptions on what is or is not considered to be valid.

This was in fact the reason why I spend so much time in completing my Maven report. It turned out I was 'building' the wrong type of document elements at the wrong time. The API did not prevent me from doing it, and the framework did not warn me at runtime. Which made me wonder....

3. Pecia

It made me wonder if it would not be possible to enforce validation at compile time. Would it not be much nicer to have the API prevent me from adding images to table rows, or - for instance - from adding a fourth table cell to a three-column table? Would it not be much nicer if the API would prevent me from making any mistakes like these? And would it not be great if the API would be a fluent API^[]?

From my perspective, the answer to all of these questions was: yes, that would be much nicer. It would allow me to spot errors quickly, and moreover, it would make my IDE's code completion assistance actually become valuable. With Doxia's Sink interface, your IDE will offer you the choice of adding images to a table row. In a framework that enforces proper document structure through its API, your IDE would never consider that to be a viable option.

4. Pecia API Principles

4.1. Context-based

The API offered by Pecia depends on on your context. This is probably best explained with an example.

Example 2. Pecia API Usage

Article article = ...;
ItemizedList list = article.itemizedList();
list.item("first item");
list.item("second item");

So what happens in the example above? Well, first you create the target document. More on that later. From that point on, you can add content to the Article. Once you add an itemized list, the API returns an ItemizedList instance, an object representation of that new context. If you want to add content to the itemized list, you need to invoke operations on the ItemizedList instance. In this case, the example adds two items to the itemized list.

Just like Doxia, Pecia is also backed by a number of implementations of the API. At this stage, it supports both DocBook and HTML output. Given the sample code given above, a simple HTML implementation would generate HTML output like this:

<html>
  <body>
    <ul>
      <li>first item</li>
      <li>second item</li>
    </ul>
  </body>
</html>

4.2. Method Chaining

Now, the example given above is not really illustrative for the way you would write code with Pecia. With method chaining, you can create the document without declaring variables to hold every intermediate content model element, and create a interface that bears a greater similarity with the way you would normally create documents in markup languages in, say, HTML or DocBook. Let us just say, a more fluent interface. So instead of writing the code given above, in the previous chapter, you can write code like this:

Example 3. Method chaining

Article article = ...;
article.itemizedList()
  .item("first item")
  .item("second item");

4.3. Shorthand Notations

This is another sample document:

Example 4. Mixed 'Expanded' and Shorthand Notations

article
  .author("Wilfred Springer")
  .copyright("agilejava.com", 2008)
  .para()
    .text("This is the ")
    .emphasis("first")
    .text(" paragraph.")
  .end()
  .para("And this is the second.")
.end()

Which will generate something similar to this:

<html>
  <body>
    <p>This is the <em>first</em> paragraph.</p>
    <p>And this is the second.</p>
  </body>
</html>

The important principle illustrated here is that Pecia both has shorthand notations as well as more verbose notations for specifying content. The simple para(String) operation (illustrated by the second paragraph in the example) starts the paragraph, adds text to it and closes the paragraph. So, it basically expands to this:

.para()
  .text("And this is the second.")
.end()

The principle does not only apply to paragraphs. It also applies to other document elements, such as list items, table cells and footnotes. In all of these cases, you can add that document element using a simple operation accepting a String with the text to be embedded within that document element, or by calling an operation without any arguments, which will change the context into the context of that document element.

Let's take an API snippet as an example. Example 5, “Pecia API Snippet” shows the signature of some operations on Para, the interface implemented by paragraphs. As you can see, there are two different footnote operations. They are different in a number of ways.

First of all, the first one takes a String argument, and the second does not. The first operation will create footnote, add a paragraph, and add text to the paragraph in a single call. Once it is done, the entire footnote is considered to be done. The context is no longer the footnote that was just added the paragraph. The context is - again - the paragraph itself.

Example 5. Pecia API Snippet

interface Para<T> {
  ...
  Para<T> footnote(String text);
  Footnote<? extends Para<T>> footnote();
  ...
}

The other footnote operation does not take a String argument. The API assumes you are not interested in adding an empty footnote (why would you?), and changes the current context into the context of the footnote. From that point on, you can only invoke operations defined by the Footnote interface, until you finally consider yourself to be done with the footnote and call its end() operation, which will restore the original context.

4.4. Tables

Tables deserve some special attention. In order to preserve a valid document structure, you not only want to restrict table cells to table rows; you also need to make sure that every row contains exactly the same number of cells.

Enforcing this property of tables proved to be challenging. Before going into details, let us first look at an example:

Example 6. A table in Pecia

article.table2Cols()
  .header()
    .entry().para("col1")
    .entry().para("col2")
  .end()
  .row()
    .entry().para("foo")
    .entry().para("bar")
  .end()
  .row()
    .entry().para("foo")
    .entry().para("bar")
  .end()
.end();

Example 6, “A table in Pecia” illustrates how to build a table that more or less corresponds to this HTML table:

<table>
  <tr>
    <th><p>col1</p></th>
    <th><p>col2</p></th>
  </tr>
  <tr>
    <td><p>foo</p></td>
    <td><p>bar</p></td>
  </tr>
  <tr>
    <td><p>foo</p></td>
    <td><p>bar</p></td>
  </tr>
</table>

So what exactly is happening in Example 6, “A table in Pecia”? Well, first the table2Cols() operation constructs a table of two columns. The object getting created will allow only operations on tables of two columns.

The first thing we do after that, is adding a table header, by calling header() on the table. Since it is a two column table, the header accepts only two cells. Any attempt to add more or less then those two cells will give compilation errors.

Every table cell is getting constructed by calling entry(). The resulting context is a table cell. There are a number of things you can add to a table cell, such as paragraphs. Once you are done with the cell, you either call entry() or end(). Calling entry() will create the next table cell. Calling end() will mark the end of the current table header. And because of the way Pecia has been constructed, you can only call end() after the last table cell, and only call entry() before the last table cell.

Table header are added in exactly the same way as table rows; only in this case you call row() instead of header().

4.5. Metadata

Some document elements can have metadata associated to it; it often involves data that is not necessarily part of the main document flow. In cases like those, Pecia allows you to specify metadata at the start of the document element to which it is pertaining.

Let us take an article as an example. An article can have an author. At the beginning of an article, before adding any content to the article, you can add metadata like the author's name. Once you have started adding content to the article, it is impossible to add any more metadata. Example 7, “Article metadata” shows you a valid way of using it. Example 8, “Illegal article metadata” illustrates an invalid way of specifying metadata; the compiler will not accept any more metadata after content has been added to it.

Example 7. Article metadata

article
  .author()
    .firstname("Wilfred")
    .surname("Springer")
  .end()
  .para("This is the first paragraph.");

Example 8. Illegal article metadata

article
  .para("This is the first paragraph.")
  .author()
    .firstname("Wilfred")
    .surname("Springer")
 .end()

5. Using Pecia

In the previous section, you have seen most of the basic principles behind Pecia. However, you have not really seen how you actually make sure that some output document is generated as a result. That was done deliberately. The important thing here is the API outlined above. How you actually get your hands on an actual implementation, and how that implementation will treat handle the documents you are building is implementation specific.

Fortunately, Pecia does come with an implementation. So this is how you use the implementation:

Example 9. Producing HTML

// The standard implementation uses a wrapper around STaX to
// produce XML documents.
XmlWriter writer = new StreamingXmlWriter(...);

// The DocumentBuilder will actually produce the output.
DocumentBuilder builder = new HtmlDocumentBuilder(writer);

// But if we are building documents, we need to have an
// implementation of the interfaces mentioned above. Let's wrap
// the DocumentBuilder in an Article implementation. (The
// second argument is the Article's title.)
ArticleDocument document =
  new DefaultArticleDocument(builder, "Example");

// ... and now we can build the document.
document
  .section("First section")
    .section("First subsection")
    .end()
  .end()
.end();

The standard Pecia implementation will generate the output on the fly. Technically, there is nothing preventing you from creating the entire document in memory first, and generating output afterwards. So all of this is all just implementation. In fact, I suspect some significant changes in the implementation, somewhere in the future; use this implementation at your own risk.

6. Pecia State

After having read the previous section, you probably already guessed that Pecia is not done yet. It is usable, and it is actually in use in one of my projects, but there is still work left to be done. So this article is in a way covers an alpha version of the API.

7. Pecia Document Object Model

The document object model supported by Pecia today is fairly simple. In fact, it is probably way too simple. Which is another reason why Pecia there is no 1.0 version of Pecia yet.

Figure 1. Pecia Document Object Model

Figure 1, “Pecia Document Object Model” provides a schematic overview of the document object model supported. The arrows denote potential containment relationships: a list item can contain paragraphs, sections can contain tables, lists, verbatim content and other sections, etc. The document elements supported are pretty self-explanatory. The only exception may be xref, which represents an internal reference to another part of the document.

8. Summary & Conclusions

In this article, I have tried to justify the creation of yet another framework for generating documentation. It grew out of unease with the existing solutions, and then turned to have a couple of interesting side effects. Pecia not only prevents you from breaking the document structure in the documents you generate at compile time, but also supports the IDE in preventing you to make these errors altogether, at coding time.

In fact, there are some benefits that I haven't even covered yet. They might be less tangible, but nevertheless, very real. I started to see that benefit when applying Pecia in a project where I had 40-something small objects floating around, each of which needed to be represented differently in my document, depending on the context.

In situations like those, many of the existing frameworks commonly in use for generating documents will force you have all of these 40-something objects expose their state to the outside world, in order to be able to bind to it from an external template.

However, this violates one of the most important principles of object orientation: the encapsulation principle. By having the objects expose their entire state to the outer world, you have actually increased the dependencies between the outer world and the object's implementation, and instead of defining behaviour as part of the object, the behaviour (how to represent itself) is externalized entirely. Consequently, maintaining the templates becomes a nightmare; your templates need to have a deep understanding of the internals of all objects.

Doxia would already allow you to take a different approach: you could potentially define a common interface on all objects allowing each object to render itself using the Sink interface. However, how would you convey the context in which the content needs to be written? How would you make sure that your object understands that it needs to display itself as part of a paragraph. Or as a table? How do you make sure that the object does not write outside of the context expected by the calling program?

void render(Sink sink) {
  // doh, how would I know if I am in a paragraph or table context
}

In Doxia, there is no way of solving this. In Pecia, this is trivial. The common interface would simply define a single operation for each context in which the object needs to render itself:

void render(Para<?> para) {
  // ah, I need to render it as part of a paragraph
  // end there is no way of writing outside of that context.
}

Which is only to say: there does seem to be a case for frameworks like Pecia. True, the current incarnation of Pecia is still quite young, and there are definitely things that will change in due time, but I have come to believe that it has potential, and I hope to have convinced you about that too.

Pecia is currently hosted at SourceForge (http://pecia.sourceforge.net/).

^[] See http://www.martinfowler.com/bliki/FluentInterface.html.