Developing a SOA-based integration layer framework: introduction

Marco Fränkel

A few years ago I was asked by one of our customers to help them make better use of their integration layer. At that time it consisted of 2 or 3 integration platforms, some home-made software running on top of those, and a few minor rules about how to use it all. None of the benefits of a ‘real’ ESB were reached, even though that was always the idea.

Ever since then me and my team have been working on a framework that could fulfill the ‘business needs’:

  • Efficiency in business processes, such as automatically coupling sub processes into a whole.
  • Consistency in data representation, such as the representation of a customer in a CRM system and in a financial system.
  • Flexibility and time-to-market accelerated by the IT department.

It was decided that this framework should be build on top of the already existing platform layer:

Figure 1: Integration Layer layers.

The framework was designed in such a way that it could function as a lightweight ESB, although we tried avoiding the dreaded ‘E-word’ for as long as possible, as people tend to get a bit nervous when you use it.

Below are several subjects we needed to address while developing the framework. In future blogs I’ll elaborate on these; whenever such a more detailed blog is posted I’ll add a link here.

Goals & challenges

Partly based on the aforementioned business needs, several goals in areas like business centricity, federation and the appliance of SOA design principles were set by the team.

Of course no task is complete without a set of accompanying challenges. We’ve encountered our share of those as well, such as lack of knowledge on key topics, and resistance from within the organization. In a future blog I’ll dive deeper into those, and the ways we managed to handle them.

Building blocks

Like any decent framework ours consists of components & patterns, both conceptual as well as physical. For me as the architect the conceptual stuff was the most important; as I could not always find what I wanted ‘in the real world’ I had to come up with some of these building blocks myself.

Basis for both the other building blocks, as well as some of the products we deliver to the projects, are our so called framework services.  Most of these are designed in such a way that they can be reused time and time again. A few are not, merely functioning as proxies for the application services they call.

One of the most important aspects of the framework is that it provides generic features. Systems using the framework to transport messages to counter parties benefit from out of the box features in the areas of routing, robustness, security, persistence and transformation.

Another big part of our framework is the connection patterns, each providing a unique combination of a message exchange pattern and a feature-set.

Integration products

Having a nicely composed framework is fine and dandy, but in the end it’s all about working software (and the documentation that goes with it), or as we like to call it: our ‘integration products’.

Whenever we get a new assignment for an application to be exposed through our framework, we start with use case descriptions. After all, no connection exists without a business reason. For that we had to come up with our own type of architectural view model.

Probably most important are the integration layer connection points. The descriptions of these make clear how messages should look like in order to be transported, and what features a consumer can expect. Once they’re implemented and deployed in a specific environment, the application can be reached through the integration layer framework.

Conclusion

The framework has been running for quite some time in production now, with no real problems whatsoever. More and more applications are reachable through the federated endpoint layer we provide, meaning that using the framework has become the default way of connecting systems from different business domains, as well as exposing our product services to the outside world.

The goals have been met for the biggest part, although there is always room for improvement. Which is a good thing, because we’ve got a lot more up our sleeve, especially in the feature department. And implementation-wise we’ve learned a lot, so some of the older software solutions are eligible for refactoring.

Most important right now though is getting the framework better manageable: a lot of the configuration is still done by hand, and it being the center of the enterprise when it comes to the transportation of messages, there’s a need to get a better grasp on what’s going on.

That’s it for this intro; in the next blog I’ll go into more detail about the aforementioned goals.

Comments (4)

  1. Vivek - Reply

    September 20, 2012 at 7:13 am

    Nice article,

    Few years back I was working on a similar requirement, Although the client named it "Lightweight engine for micro orchestration" LEMO ??? :D

    We did it with Apache Camel as the backbone, We took apart all the parts we didnt need and changed few things here and there. Camel gives out of the box integration for almost every kind of endpoint. (although we ended up extending or rewritng the endpoint interfaces but it was not that bad !)

    Since then they have loaded it from like a few thousand orders to 100K orders a day and it works like a charm.

    Vivek
    geeksonjava.com

  2. Simon - Reply

    September 20, 2012 at 7:59 am

    Having worked at firms which have an ESB for years and done a number of systems integrations I am still a bit mystified by them. Message Oriented Middleware is great for point-to-point and publish-subscribe. Using a standard product makes sense, encouraging consistent payload format (xml, protobuff, json, whatever) seems nice to have. Yet none of these things actually seem to be that hard nor adding any value when compared to the overhead of having such fat "standards", MOM products, and an ESB team which costs a fortune.

    Agreed its a bad idea to have everyone talk to everything using ever product and every message format. Yet the idea of actually having a team of people (minimum three for holidays and sickness) actually look after an ESB beast (whilst never actually getting all system onto the ESB they only ever get as far as few systems getting on) seems, well, ridiculously excessive expense for negligible added value. Give the same resources to the dev teams who can delivery more value on a build rather than have the fixed ongoing overhead of the ESB team. That is going to cover the cost of an application build or three over the lifetime of the ESB.

    I sort of think of ESBs like mainframes; people who have them have them as a historic cost and people who don't never will because the cost is unjustified. Yes I work for big companies which have them and I smile politely and play nice with the ESB team; but if I ever make CTO the heavy weight ESB products and team are going into the same trash can as the mainframes...

    Part of the mystery to me is that they simply don't solve the complex part of systems integration out of the box; a shared domain model. Namely the politics, design, implementation, construction of having applications share data and shared business semantics and cross platform information flows. Thats really very hard and really very expensive. If you live that hard life then sure the cost of the "sunk cost" ESB vanity project the last CTO/CIO put in is a drop in the ocean compared to everything else; the architects talking SOAs without any understanding of business data flows can be mostly ignored. The delay in dealing with the bloaty ESB tech and its setup of a new application can mostly be ignored. Yet really the costs compared to the actual measurable benefits to the system integration cost is a very poor payoff.

    • Simon - Reply

      September 20, 2012 at 8:07 am

      Not to suggest thats what you built; as hinted by your statement about "avoiding the dreaded ‘E-word’ for as long as possible". It would be interested to hear from an agile shop like yours what was the actual overhead/benefit/costs compared to a big vendor solution or every integration rolling its own solution with opensource.

      • Marco Fränkel - Reply

        September 21, 2012 at 10:39 pm

        No, I don't think that's what we built ;) Still I recognize some of the points you address, and sure, they could form a potential danger but in our sitation the good outweighs the 'bad' by far. Plus. our team isn't that big in fact, most of us doing other stuff as well (my main role is SOA specialist for example, concentrating on SOA adoption & governance).

        The reason why we avoided calling our framework an ESB (apart from that I think it's more than that - as I'll explain in a future blog) is in fact all that negativity surrounding the term. I'm not really interested in defending the choice that was made (one that I didn't even have anything to do with in the first place - not that I think it's a bad one), I'm interested in providing the best solution possible.

        I'll come back to the agile influence in a later blog as well. To be continued, so to speak.

Add a Comment