This is not a rant against ESB. I am not saying that ESBs never have a purpose, nor suggest that it's all just a scam. If - after having read this post - you got the impression that I suspect a conspiracy behind ESB, then I want to tell you up front that this is certainly not what I intended to say.

The only claim I am actually making is that - if you are interested in implementing integration patterns - you can certainly achieve that with a whole lotta less. And as always, less clearly is more. You do not always need a monumental solution to solve an integration problem. Spring Integration and Camel are wonderful alternative solutions, and way less intrusive.

That doesn't mean that it's all just roses and sunshine. There are certainly things left to desire. One of the things that I find sort of missing in Spring Integration is an easy way to have guaranteed message delivery, so that's what this post is about.

Channels

One of Spring Integration's major abstractions is a channel. Think of it as a queue that is not necessarily backed by - well, a queue. It's just the place where you hand-of or receive a message, without the sender and receiver necessarily being aware of each other's existence. When the receiver reveives the message, then that message might be pulled from a buffer kept internally inside the channel (like in a QueueChannel), but it might as well be delivered in a synchronous manner, by the same thread that sent it to the channel (like in a DirectChannel). From a receiver's point of view, it's all the same.

Spring Integration Channel Abstractions

Now, if you're dealing with a channel that is backed by a an actual in-memory BlockingQueue, then you do have some degree of reliability. You know the message will be delivered at least once, at most once, and that order is preserved, but only if your system is never shut down. If your system is shut down, either because you stop it yourself, or because of unexpected error conditions, all of the messages that are inside those channels can essentially be considered lost: whenever the system reboots, all of those queues will be empty.

Message loss in regular channels, versus KahaChannel

Idempotency to the rescue?

You could argue that you could just replay all of your messages, and - as long as the receivers or intermediates are all idempotent receivers - everything will just work out fine, but then I tend to disagree. First of all, you do not want to replay your entire history of messages at system startup. So you probably need some mechanism to prevent messages that already traversed the entire chain of message handlers from being sent again. If idempotent receiver is the only trick up your sleeve, you end up doing a hell of a lot of work yourself, and it is not going to make your life any easier.

Apart from that, if you do to replay a fair selection of messages that entered your system in the past, then you still need the ability to reliably store the messages at the outer rim of your system. So even if you abstained from wanting reliable delivery guarantees inside your system, you still will not be able to avoid a reliable hand-off at the edge of your system. (That is, unless you convince your callers (business partners, external clients) to do all of this bookkeeping for you, but I bet you will not be able to pull that off.)

JMS to the rescue?

Now,  I said Spring Integration does not have support for guaranteed delivery, which actually was a lie. You can have reliable delivery, if you route your messages through JMS, and use a persistent queue. So, it can be done. However, why would you want the overhead of the JMS API and everything that sits behind it, if you could also implement the PollableChannel in a reliable way directly?

Kaha DB to the rescue!

So, that's what I did. I present to you, the KahaChannel. Kaha is ActiveMQ's data store backing persistent message queues. So essentially, all I did is rip out Kaha underneath ActiveMQ and implemented the PollableChannel interface on top of that. The result is a native PollableChannel that directly stores messages into ActiveMQ's highly optimized data store. And Kaha is file-based, so it does not require any other external datastore.

(Obviously, the PollableChannel interface could also be easily implemented on top of some other journalling system, such as Howl. However, Kaha seems to be way more active than any of the other journalling systems I am aware of.)

So now what?

Using the KahaChannel is fairly simple. It is just a drop-in replacement for any other PollableChannel. Unfortunately, it does not nicely blend in with the Spring Integration namespaces (yet), so you do need to construct an instance of a KahaChannel using vanilla Spring configuration.

So, if this used to be the configuration of your channel before:

    <si:channel id="inboxl">
        <si:queue capacity="1101"/>
    </si:channel>

Then the only thing to do to make your system a little bit more reliable is drop in this as a replacement:

    <bean id="inbox" class="nl.flotsam.spring.integration.kaha.KahaChannelFactory">
        <property name="directory" value="..."/>
    </bean>

The example above uses the KahaChannelFactory for constructing an instance. That's not a requirement. You can also construct it directly, by creating an instance of KahaChannel and passing in the parameters as constructor arguments, but then the name of the container constructed underneath will not automatically correspond to the id of the bean.

Performance

Replacing an in-memory buffer-based channel with a disk-based with reliable delivery guarantees obviously has its price. However, the performance penalty is not necessarily huge. It all depends on the size and complexity of your messages. If your messages cary hardly any data, then the penalty is fairly small: pumping 10.000 messages through an in-memory channel took 3.2 seconds on my system. Running the same messages through a KahaChannel took only 4.3 seconds. You do the math.

Encoding Messages

In order to make the KahaChannel really a reliable channel that will survive a crash, the messages obviously need to be stored. Unfortunately, Spring Integration does not define an abstraction for encoding and decoding messages yet.

The Kaha abstractions allow you to encode to DataOutput and decode from DataInput, so anything eventually needs to be mapped to something supporting these abstractions. Rather than trying to implement the ultimate way of encoding and decoding a message itself (if something like that would exist at all), the KahaChannel basically takes the stance that it is up to you to provide a sensible codec. And for that purpose, it defines its own message codec interface:

public interface MessageCodec<T> {
    void encode(Message<T> value, DataOutput out) throws IOException;
    Message<T> decode(DataInput in) throws IOException;
}

It does however also ship its own SimpleMessageCodec, which implements a ridiculously simple strategy for persisting your messages, only supporting basic types as int, boolean and String as types of the payload and headers.

Conclusions

Spring Integration itself does not provide a persistent message channel, guaranteeing that your messages will eventually arrive at the receiver, even in case of system restarts. However, if you leave that out, you end up with a toolbox that pretty much provides most of what you would expect from a fullblown ESB solution, at least in terms of its support for implementing integration patterns. A JMS Message Broker could be used to add durable message persistence to the mix.

However, despite the fact that Spring Integration has no support for persistent message channels built in, it does allow you to add your own. Implementing a persistent channel on top of those abstractions turns out to be fairly easy, if you rely on a data store such as Kaha. And having a lightweight native persistent message channel seems to be a more natural fit in Spring Integration's lightweight approach. In a sense, it is similar to the NMR in JBI implementations: you know your messages are safe once they are inside: it may be backed by a JMS Message Broker, but to the ESB developer, it's just a native component of the product.

Tags: esb, integration, messaging, SOA, spring integration
Filed under Java, SOA | 7 Comments »