Being hired as an Agile consultant or coach, you get to see a lot of organizations where the software development processes, or rather the whole chain between business idea and system implementation, is in serious trouble. They wouldn’t have needed a coach if they weren’t. Being experienced in this line of business, one can analyse and identify the main sources of problems quite quickly, and will come up with a concise plan to make big steps toward improvement.  Then we take the bull by the horns and want to implement the plan. RIGHT, GET TO IT DUDES, GO, GO, GO, AND MAKE iT SNAPPY!!! Agilists are roll-up-your-sleeves types by definition. Well, great, isn’t it? Get results fast, improve that process in weeks! Won’t we? Err…. Nope…

Coming in, finding the sore spots fast is great value. Moving too fast after this initial step is a HUGE pitfall for Agile Change Consultants. Why is that? There are several reasons why you shouldn’t move too fast, I’ll get to that. First  a disclaimer. When you arrive at a company that has a young, fresh, eager, willing team, that has a can-do mentality but is simply not structured, disciplined, focused or whatever enough, THEN it  can be a very good idea to do exactly what I described above. Get in, pump them up, do the right things and there you are, streamlined in weeks. Those are the “easy” ones, the exceptions to the rule.

 Usually you come into an organisation where ancient processes reign, where the culture has assimilated to having problems and to only allowing traditional solutions. Where the manoeuvering space of individuals has shrunk bit by bit, and where hierarchy, handovers, fingerpointing , ratty behaviour and lack of personal responsibility for the overall results (“not MY problem”) are present wherever you look.

 If you move too quickly there, you will leave everyone behind. Yes, I will say that again, if you move too quickly you will leave everyone behind. You will not change a damn thing, lose all the momentum and enthusiasm that you have managed to whip up and actually be counterproductive, producing only more disillusion and frustration when you have left.

The big question here is WHY? Wouldn’t one expect that once it becomes clear what to do to improve their situation, people would readily adopt the improvements?

Change is scary.

As good old Niccoló Machiavelli (“Il Principe”) astutely pointed out, all change has fierce adversaries and at best lukewarm supporters.  No matter how awful and inefficient the current process is, no matter how much frustration and superfluous work and effort is put in for minimal results, it is still THEIR PROCESS. This is what they currently do business with, it’s what they have got used to. So although it’s bad, it is at least “proven”. They will think “How do I know that this new approach will deliver our stuff AT ALL? We have lengthy cycles, but we do meet our deadlines now. Ok, the quality sucks, but the customer does accept it now. How can we forecast what will happen with all this new ideas?” Maybe they don’t even consciously think that, but they will feel it at least.

Moreover, in an environment of hierarchy and distrust there is an enormous risk to moving away from the Main Path, because if you fail within the path it is recognized as “standard failure” but if you fail taking an unconventional approach you will be seen as irresponsible because you didn’t follow procedures. Workers are not happy to take such risks just because some hotshot external Consultant comes shouting these crazy new ideas, even if they sound plausible. Or even proven someplace else.

Higher up

Then there are the higher layers. These people have been responsible for designing and implementing exactly the processes that you are burning to the ground. Not a message that will be received with big cheers., understandably, unless you ease it in. Inherent to agile is bottom up responsibility and empowerment, which is at first seen as a threat to manageability as well, so, lukewarm supporters at best, if at all.

Customers

And let’s not forget the customer end of the stick. Probably they will be the easiest to convince, because inherently they will experience the biggest problems with the quality of the existing process. For them there will have to be a lot of increased effort in that Product Owners (to use a Scrum term, it doesn’t matter which Agile method you employ) are expected to have a lot of direct involvement with the project, so it will cost them a lot of time and work. But it is not unlikely that they will go for it, if the problems of the current processes are big enough. Yet even so, if you go too fast in getting their support and enthusiasm, the slow response of the delivery organisation will annoy them and the customer satisfaction will go down even if you have slightly better results than before, because then it doesn’t go fast enough for them. Risky!

We could go on about the maintenance departments, the exposure of poor performers in self managing teams, the necessity for hugely increased discipline which is not always comfortable, etc. etc. but the picture is clear.

What to do.

Change will work best if it is experienced as something that comes forth from the people themselves. It must be their thing, or it will meet resistance.

So what you need to do is to talk to all of these stakeholders of the process (you already have while doing your analysis), and have them find out what their top two (yes only two) problems are. Then you help them find out the root cause (of course,  you may know what it is at a single glance, but you don’t necessarily TELL them just like that, you help them find out). If you help them find out it will be “their thing” and they will be much more involved. Once they’re involved they will be more susceptible to adopting a solution. You propose some small changes to improve their main problem. Of course you also have your own prioritized list based on your experience and analysis, and you will be tempted to nudge them towards your own top N issues. Don’t underestimated the force of addressing their top two however, even if you wouldn’t think their  road would be the most effective one towards implementing improvements. Take it seriously, and negotiate a bit perhaps. Then you give it some time to be implemented, not too much time because you will have selected the size of the change to be small enough not to disrupt too much of the current process, and big enough to make a measurable difference.

Don’t be smug. These people are not stupid, they have just evolved into using processes that can be improved. Try your very best, all the time, to think like them, to stand in their shoes, why does he behave like this and how can I help him realize that there is another way. See everyone with a problem as a friend with a problem.

PATIENCE and relentless energy at the same time. Sow the seeds. Water the plants. Nudge. Pet. Steer. Guide. Let the teams grow into the change. Make improvements visible. Except for the latter, this is contrary to our innate urge to get results quickly. But don’t be fooled. In this type of organizations “getting results quickly” means that you have implemented any change AT ALL in half a year, so instead of wanting to rearranging the whole process to an end to end Agile approach in eight weeks (“hey, come on we KNOW what we must do, what’s the big deal, just go and do it”) and be frustrated if everything comes to a grinding halt, rather be happy to improve productivity and quality measurably and to have whipped up the beginning of enthusiasm and acceptance for the Agile culture after, say, six months and counting. WHAT? SIX MONTHS?! Yes. Deal with it. It goes with the job.

Systems integration within limited timeframes is not mainly a technical art. It is a matter of social and organizational skills. Work on the touchy feely side, and tech will follow. “If you don’t manage the above factors well, you will be in for it anyway, no matter how cool, flexible, state of the art and gorgeous your technology.”

Introduction

In almost every software project that is done in a substantial company we need to integrate with other systems. When a team takes on such a software development project, there is often a tendency for developers and project leads to focus on the tasks that need to be done to create the software on their own side, and write scoping statements and non liability remarks to make sure they are not bothered with the integration risks. Especially when The Other System doesn’t have appropriate interfaces available that you can use, and “they” have to build new ones, then the relationship with the other system is kept at arm’s length in contracts.
Understandable, because in our projects we evidently cannot take on the responsibility for work outside our sphere of influence. However. If we want our project to really succeed, we cannot keep it at “they have to do their thing, and they have to deliver on time. If they don’t, then our project may fail but it will not be our fault!” Nope, not good. You will not deliver value to the customer like that, the only thing you do is cover your ass. Comfortable, but not outstanding craftsmanship.

Estimates

Now, suppose you really want to make sure that the system you are writing will be able to interact correctly with all of the necessary parties involved. Many projects fail because the temporal impact of interacting is seriously underestimated. Interacting can have a tremendous effect on the estimated total project time. Consider what has to be arranged:

• You need to communicate with the other party and define an interface between the systems.
• They have to build their end of the interface.
• You will have to stub out their system in order to be able to do your tests for as long as they don’t have anything you can talk to yet
• There will be unexpected changes to the interface. (No, don’t say that you can just finalize the design and be done with it: there WILL ALWAYS be unexpected changes to the interface)
• You will have to share an integration test environment. Stubbing is nice, but only up to a point.
• In the integration environment you will need “conditioned” test data that is meaningful to both systems. When your “other system” has already been around for a while they will probably have a predefined set of test case-data in their database, which of course is not compliant to your own test case-data. So you will have to arrange for new, shared, test data that is meaningful both to you AND the Other System.
• You will need access to their test system. If you are really lucky then the customer will have an integral acceptance test environment in place, with production-alike network infrastructure, many systems joining in, and with conditioned datasets. Not likely. Highly unlikely. So you will need to arrange for access to their test environment, which may be several hops away. Some companies shield their environments through subnetting: arrange access through the routers or try to get a machine of your own inside their test domain: often procedure-heavy activities.
• At integration time they will have interpreted your agreed specs differently (case sensitive, long instead of int, slightly different encryption or compression algorithm, what have you) and you both need to re-implement parts of what you had written.

So, all of this goes for one single system. Think of what happens when you need to integrate three or four, especially when the data goes to and fro between these systems. For a valid end- to-end roundtrip you will have to have all of the above in place for all of the systems in question.

Multiplication, sooner than addition

When you extrapolate this, it is not unrealistic to calculate with an almost exponential factor to the complete duration of your project. And I’m not talking “raw development time” here, no, I mean the time needed to get from project start to actual production. I have often seen estimates like “xxx weeks, plus an extra iteration for each system”. That will only be true if these systems are unrelated and if all conditioned data and integration environments are in place, if you can reach all systems from where you are developing AND everyone is at peace with the world. Again: unlikely.

Make sure you start requesting DTAP (Dev,Test, Accept, Production) environments immediately at the start of the project, the same goes for (network) access, etc. etc.

So the advice here is: TAKE HEED! Multiply, rather than add! They may not be as flexible as you are!

Agile approach

Now, if we were living in an agile world, then the odds would be a bit more optimistic. You will insist on early integration, they will not object to that, better yet, they will be enthusiastic to do so. You will gradually work together toward a cool interface, which will keep changing as needed to get to the most efficient, open solution. Test data will be created in unison, you will learn from each other, if one party has a problem and starts lagging a bit you offer to lend him one of your experts in exactly their problem field, who happens to be on the project anyway. Sigh. Love and understanding. Success. Profit. Happy Customers. Bliss.

Why not

Hm. If the company where you’re doing your project is large enough, this Arcadia is a rare find. The atmosphere among developers in traditional companies can sometimes be one of fear and loathing

• Fear of not being as good a developer as your competitor
• Fear of being punished for making mistakes
• Loathing of not being taken seriously as a craftsman
• Loathing of not being allowed to shine and make a difference

This is obviously not a great basis for collaboration. But as it is, sadly, not uncommon, you had better find a way to deal with it. As difficult as it is to change structurally, you can still do something to improve your chances of a successful integration.

Social calls

Make sure you have at least met once, face to face, with the Other System’s developers, project leads and with whomever else comes into play. If you are on opposite sides of the ocean, use video conferencing with even the crappiest of webcams if needs be. (No excuses like “we don’t have a video conferencing room”). Use direct messaging or irc to communicate. Of course, if the company doesn’t allow a streaming protocol through the firewall there’s not a lot you can do. But in that case send your photograph with your email-messages. Believe me, even knowing what the other party looks like can make a tremendous difference.

We’re in this together

Try to create an atmosphere of mutual interest. There will probably be a sense of togetherness around a system, a “three musketeers” feeling in the developer group. That’s great, because it improves team gelling and with that fun and productivity. You, as someone of the outside world, want something from them, you want to invade Their System and you will have an impact on their original plans. Evil Outsider! So what you want to do is overcome that and try to become accomplices, get them to realize that we work for a business process that needs you to integrate. “Yo, WE are going to make this business process work, guys, great isn’t it?” Even in traditional development environments you can gain a huge headstart by being sensitive to this sort of relationship management.

When there are more systems than one that you need to integrate with, try to initiate this same feeling of togetherness between all systems. Gather the teams, or at least the team members you will be working with, and have an integral meet and greet session somewhere. If politics prevail, hold the session on “neutral ground”, say in a café. Don’t hesitate for a second to pay for all drinks and bar food, the cost of that is absolutely neglectable relative to the enormous profit you will have in terms of risk mitigation.

Infrastructure

A special aside about infrastructure. There can be a huge difference in how slick the infra groups are organized. In some organizations a simple application form for an infra resource will be sufficient to get a nicely humming [machine/router/databaseaccess/...] way before you desperately need it. Whereas in other circumstances you will have to fight fierce battles for every inch of UTP. If the latter is the case, approach the infra groups like you would approach The Other System: try to make them your friends and allies.

Don’t squeeze it

It’s no big deal if you don’t agree on the definitive interface. Give them the feeling they are in the lead. Even if you might think the developers in The Other Systems are not the greatest, they will know a lot more about their system than you do. You don’t need to squeeze all possible brilliance out of the design. If it works well then it’s ok, as long as you can manage to give them a feeling of ownership and authority. Remember the fear and loathing: if you can give them the feeling they are in the lead and are taken seriously, they will be grateful for the opportunity.

Technology

“Hey, when will you start to talk about the Technology? Soap, Web services, Corba, RMI, OSF/DCE, vanilla XML over http, what should we use, synchronous, asynchronous, what?” you will ask. My answer is: All Of That Is Trivial. If you don’t manage the above factors well, you will be in for it anyway, no matter how cool, flexible, state of the art and gorgeous your technology.