Early in their book “Lean Software Development: An Agile Toolkit”, it says:

“However, lean production practices — specific guidelines on what to do — cannot be transplanted directly from a manufacturing plant to software development. Many attempts to apply lean production practices to software development have been unsuccessful because generating good software is not a production process; it is a development process.

Once a chef has developed a recipe, preparing the dish means following the recipe. This is equivalent to manufacturing, where the objective is to faithfully and repeatedly reproduce a “recipe” with a minimum of variation.”

Well, Tom and Mary, allow me to disagree! The main purpose of the modern assembly line is not to produce masses of the same good “with a minimum of variation”. Actually, for car manufactures this is about as far from the truth as it gets. The Mercedes-Benz assembly lines in Sindelfingen near Stuttgart for example, produce hundreds of thousands of variants of the C-, E- and S-class. There are about 8,000 cockpit variants and 10,000 seat variants for the E-class alone; there are almost certainly no two identical cars rolling of the lines on the same day. What starts of with me (if I were a Man of the Three-Pointed Star) ordering a car at the dealership results in an enormous logistic and organizational effort to get the right components to be available at the right place and at the right time on the assembly line (with suppliers supplying the right parts at the right time to minimize storage costs). Raise your hand if this fits with your idea of a “recipe with a minimum of variation.”

Software engineering lags more than a century in maturity behind manufacturing; it is really more like a cottage industry handcrafting one-of-a-kind solutions than proper engineering. The current practice in component-based software engineering amounts to getting just the individual parts necessary to assemble a single car. Some of these parts are not a perfect fit, and some cutting and filing is needed to make them fit (i.e. adapt them) and some parts are purpose-developed for this one car. And even if a library of elementary, designed-to-fit components is present (‘no cutting and filing required’), these still have to be assembled manually and there’s still a lot of detail to consider in the process.

Compare this to ordering a ready-made car by describing it in abstract, solution-oriented terms, stating only as much as is really needed: “Get me a SLK 55 AMG, shod on 18” wheels, with chromed exterior mirror housings.” And that’s what I would like as a programmer: stating what I want in abstract, solution-oriented terms and have the desired system or component produced in an automated fashion.

If this sounds all to magical, it is really a paradigm for software engineering based on (i) designing implementation components to fit a common architecture, (ii) the configuration knowledge linking abstract requirements to specific constellations of such components, and (iii) implementing this configuration knowledge in an automated fashion. These steps are what happened to car manufacturing as well: the principles of interchangeable parts (Hall, 1826) led to the introduction of the assembly line (Olds in 1901 and refined by Ford in 1913), and automation using industrial robots in the 1980s.

Not convinced yet, let me end with the inevitable referral to Ruby on Rails. It may not be on the level of true manufacture, but here’s too a way of specifying a system (‘family member’) in terms of constructs of a higher level of abstraction (‘has_many’, ‘validates_presence_of’), a set of implementation components (‘ApplicationController’) and generators to tie everything together (‘./script/generate’). And boy does it work. Beautifully. Just imagine what the next step will be like!

Lisp is, of course, the most powerful language available. That’s right, what looks like an odd syntax actually is a terse syntax that allows for a great deal of configurability. Where you probably are used to formulating a combination of keywords and punctuation to get your ideas across, in Lisp you directly code in what amounts to other languages’ compiler-generated parse trees. List programs are lists (of lists of lists…) and Lisp’s parse trees are lists too and generating parse trees in code therefore is very easy.

A program that generates and/or manipulates parse tree is effectively a program that (re-) writes programs. Now, this might seem like a gimmick that matters only in a game of geek trump stakes, but it actually opens up to a whole different style of programming. In Lisp you’re not limited to writing a program down towards the language, but you can also build the language up toward the program. If you’d ever think, “Gee, I wish I had this-or-that construct to make my problem go away”, you can go and write that construct to suit that particular problem. And inevitably, you will realize that using this new construct would simplify the design of another part of the program, and so on, and so on: language and program evolve together. You end up with a program that looks as if the language has been specifically designed for it. And having language and program fit one another optimally is what keeps the code as clear, small, and efficient as possible. Very few programming languages support this bottom-up design to the degree that Lisp does.

The phrase ‘design’ is there on purpose: to emphasize that this isn’t about writing the same program in a different order but that the resulting program is structured very, very differently. You end up with a larger language with more abstract (dare I say: more domain-specific?) constructs, and a smaller program written in it. And a smaller program doesn’t have to be divided into so many components. Fewer components means the program is easier to read and modify and that there’re fewer connections between components, and thus less chance for errors. Small is beautiful and beauty is what’s all about.

Of course, I haven’t ever (ever!) wielded Lisp on a paid job. And I reckon you haven’t either. Which means that you and I both missed out on something very important: the opportunities of server-side programming. When I started out programming, there wasn’t really much of a choice in the language to use for writing application programs. Back then, ‘applications’ generally meant writing software to run on desktop computers; and in desktop software there was a strong bias toward writing the application in the same language as the operating system. Fifteen years ago, in my world, applications were written in C/C++. Then, the Web happened and these things changed: in server-based applications it is far easier to get away with using the technology (language, OS, …) you like. Perl, PHP, Python, Ruby: they all got their place under the sun because people started using them on servers and not because people used them to do desktop stuff. Moving software onto servers means there’s just less pressure to use mainstream (or average, or middle-of-the-road) technologies.

If this sounds halfway reasonable, then only with 20/20 hindsight because for the past 7 years or so my diet has been building server-based applications using the very mainstream Java stack exclusively (and to a certain degree, inexcusably). I needed the Ruby on Rails wakeup call to ram the benefits of Lisp style home. Ruby of course too allows for the language to be extended in runtime from within. For Rails, the ‘base’ Ruby language has been raised in abstraction so that the problem of creating web apps effectively and the solution language fit one another better. Given rampant Rails’ success, the benefits (and joys!) that can be reaped from this are apparent to a significant amount of people. And it is the bottom-up design capabilities of Ruby that make it all possible. That’s why it’s called Ruby on Rails instead of Rails on Ruby (a fact that might have escaped the Trails project team).

Ruby and Groovy are no Lisp of course (Lisp’s macros really are something else), but they are effective enough in showcasing the fruits of Lisp style to make Java-the-language look, well, clunky. So clunky in fact that I could not bare programming Java without Eclipse’s editing and refactoring. I still do value Java-the-platform though, with its slew of great, proven and well-understood technologies for object/relational mapping (Hibernate, JPA), application configuration (Spring) and transactions (JTA) to name just an area’s. All of these technologies are of course perfectly addressable with Groovy and JRuby, which thus allows for superseding Java-the-language whilst leveraging Java-the-platform. What’s not to like, really?

Lisp. Ruby or Groovy… I’d say the future of server-side programming is funky!

Every new computing trend elicits debate and any sizeable debate provokes vendors to aggressively position their –often already existing– products to take advantage of the customer’s interest in the hot, new trend. Of course, the energy pumped into ever inflating expectations inevitably finds a way out in the guise of confusion and disillusionment. The vendor hype surrounding ESBs has been especially effective to the extent that organizations that want a quick-fix solution (and don’t they all?) are sold on the idea that an ESB is the yellow brick road to “instant SOA”.

Here’s the gentle introduction: an ESB is not an essential part of SOA. Actually, leave any product in their glitzy shrink-wraps for now: SOA is about analysis and design, and not about technology.

Of course, there are a few basic components you’ll probably want to use get kick-started. But an ESB is not one of these “basic components”; in fact – and here comes the shocker to some – you should absolutely not use an ESB when starting with SOA because an ESB does not encourage good SOA. ESBs are integration solutions and integration solutions reinforce application silos while SOA is about tearing down those silos.

An ESB is a useful component in a services infrastructure as it is especially good for bridging to legacy applications. Many ESBs also support reliable messaging, asynchronous messaging, and several message exchange patterns, which are all useful capabilities — just not in the initial stages of a SOA project. To put this in perspective, you might also want an orchestration engine at some stage in a SOA project — but that too isn’t where an organization should start a SOA initiative. All of this you just don’t need when first getting started; an ESB should be a later-stage purchase.

Some parting words for all of the kind folk that tried to convince me of the essential nature of an ESB by drawing hub-and-spoke diagrams: a bus, by definition, is not a hub but a physically distributed set of federated hubs. It may start out centralized, but as the implementation evolves the infrastructure should allow for more physical distribution. You might want (and appreciate) to keep the bus’ configuration and control centralized however.