For details on the calculation, refer to the book. It’s well worth the read and if you’d like to start off with a shorter version, there’s a whitepaper available here. The short summary is that simplicity trumps all other quality aspects of a system. Simple systems are easier to maintain, perform better, scale better, etc. Simple is the opposite of complex and using his approach you can calculate the relative complexity of different alternatives. Number of business functions and number of connections per system are the main input for the calculation. The one with the lowest complexity is the best alternative.

Start situation

Enough theory, what is the problem at hand? The picture on the right is the system landscape that is already in place. In reality there are more systems surrounding it, but I limited it to the systems that are considered in the alternatives. Systems A/B/C/D are all part of a fulfillment chain and as you can see they are all decoupled by EAI adapters. The datawarehouse (DWH) is fed with product details that are delivered by the fulfillment chain systems.

The functionality used down the chain from system A till D is now limited to order processing. However, system A now needs an information service that enables it to decide on certain properties for the order submitted to system B. This information service needs:

• Details of delivered products that are present in system B and C (and also fed to the DWH)
• Business rules that are only known in system C
• Business rules that are only known in system D

EAI coordination, no DWH

Alternative 1: EAI Coordination without DWH

In this alternative a composite EAI service is created that is invoked by system A and subsequently gets the required details from system B, then asks system C to enrich it and apply its business rules and finally asks system D to apply its business rules. After that it’s able to send the response back to system A.

Putting the number of business functions and connections per subsystem into the spreadsheet adds up to 422 SCUs. So adding one new component and 4 new connections almost tripled the complexity!

EAI Coordination with DWH

Alternative 2: EAI Coordination with DWH

In this alternative a composite EAI service is created that is invoked by system A and subsequently gets the required details from the DWH (instead of from system B), then asks system C to enrich it and apply its business rules and finally asks system D to apply its business rules. After that it’s able to sent the response back to system A.

Down the chain

Alternative 3: Down the chain

In this alternative there is no composite EAI service that does the orchestration, but the request is pushed down the fulfillment chain (incl. EAI adapters) just like the normal orders. Each system and EAI adapter exposes a new service and once system D produces the result, it traverses up the chain though all systems again before it’s handed over to system A. At first glimpse this may seem a nice solution because a similar pattern is used as for the order processing, but each system/EAI adapter now has one extra business function and has extra connections.

Conclusion

First pitfall is that many procurement departments and management still have the mindset that everything is clear at the start of a project, when there is a rough idea what problem must be solved and there is relative confidence that a positive business case can be made. To get a better picture of the costs service integrators are contacted such that the business case can be made. A preference for a COTS based solution is expressed and from there on the discussions focus on the features of the COTS product, license costs and hourly rates for professional services. Little attention is given to deeper analyze the problem to be solved, get core requirements clear and see if a COTS product is the appropriate solution. For example a real time ETL product may support Change Data Capture, but using it would create a really tight coupling to the internal data model of the source system and requires the ETL product to understand how to interpret that data model. Although technically possible, it is definitely not preferred from an architectural perspective.

Selecting a COTS product is a decision that cannot be taken lightly, it frames the architecture and design space and comes with a set of consequences. This decision is hard to revert, so to make this decision you should understand the problem you trying to solve, the constraints that apply, the pro’s and con’s of the product etc. Proof of concepts provide valuable information and it forces you to really think about the problem you’re solving. Domain and technical experts should be able to decide what the relevant requirements (both functional and non-functional) to be included in the proof of concept are. Don’t rely on experts of the service integrator or COTS vendor alone, ensure you have some neutral experts involved. Should you pay for execution of such a proof of concept? Definitely not for licenses involved, for hours spent you could. Costs of a proof of concept should be a fraction of the total costs and you may be saving yourself an ton of money by doing a serious proof of concept and learning from it. Remember, you get what you pay for.

When the decision for a specific COTS product is made next pitfalls come into play. Although the product is envisioned to work off the shelf, there is always that tiny bit of configuration to be done. More often than not this turns out to be a significant effort. Now the team starts to really dig into the requirements and understand them. Some can be realized easily with the COTS products, some not and that’s where trouble starts. If you feel like your pushing a square through a round hole, you know you got the wrong tool for the problem at hand. Deciding to no use the COTS product (for everything) often is a politically sensitive subject. Therefore, be open about this and always keep the option open to use other tools to if the COTS product becomes an impediment. The goal is not to use the COTS product XYZ, the goal is to solve the business problem at hand.

The next pitfall concerns the use of development practices. If significant configuration effort and/or custom coding is required it becomes a development project instead of just deploying a COTS product. If it is a development project, it should be treated like that and established practices like version and release management, continuous integration, automated testing, automated deployment, etc. must be applied. These practices build quality into the process and enable teams to identify and fix issues early instead of after go-live. Are these practices ignored? Yes, at least I’ve seen it happen. The question is “why?”.

I suspect that one cause is mix of skills in the team, or actually, the lack thereof. When running a project based on a COTS product the natural tendency is to search for team members that are trained for that product. Although they know the COTS product inside out, they may not have a software engineering background and/or are not hooked into the software engineering community. They are simply not aware of these practices. And if they are aware of the existence of these practices, they may not know how to execute them. For these reasons it’s important to have a mixed skills set in the team. Sure you need COTS product experts, but you also need all-round software engineers, a DBA, operating system experts, etc. They might not be full time team members during the whole project, but they must be available at short notice and stay informed about the current state of the project.

Another reason why these practices are not applied may be limitations of the COTS product. If the COTS product doesn’t support version management by itself and integration with a well known version management system like Subversion is not possible, then you’ll have to work out an alternative. This may involve some manual exporting etc, but the fact that it’s not supported out of the box, does not imply that these practices can be skipped without negative consequences.

A third reason to have a mixed team is to prevent the golden hammer syndrome. If the team consists only of COTS product experts, they’ll try to solve every problem with the COTS product. Even when a bit of scripting or the use of standard command line tools would do the trick much faster and with less effort. Generation of test data is an example, typically a bit of scripting is enough to generate test data. However, I’ve seen situations in which complete ETL workflows in the COTS product were developed to generate test data. That took significant effort and the speed at which the data was generated was low (which became a problem when the database had to be populated with larger amounts of data). An all-round software engineer must be able to generate that test data in a much more efficient way.

These are some pitfalls and suggestions how to deal with them that I cam across. They can be prevented by having the right mix of skills involved and by focussing on the problem to be solved instead of focussing on COTS product features. What pitfalls have you seen? How should we prevent them from happening?

Since I starting in IT I’ve spent 98% of my time on the software producing side of IT. I did development, design, architecture, switched back and forth a bit, but always in the area of building software. I was asked to take a role as technical implementation manager for an IT landscape consisting of 40+ systems and all kinds of integration adapters. Not really my cup of tea, but I took the challenge and hoped to learn a bit (although I was not sure what). The scope I had to deal with went from 1 system and it’s directly related systems, to 40+ systems that together form a whole chain with lots of interdependencies. Due to all these interdependencies it was impossible to upgrade just 1 system. Upgrading 1 system caused a snowball effect that required upgrades across the whole chain. Bad design, true, but a fact we could not change on the spot.

The main target of the implementation manager is to ensure that systems are taking into production in a reliable and predictable way without causing unplanned downtime of the chain. With 10s of systems to be updated in one go, that is not an trivial task. The last go-live of the chain had been a disaster and took weeks to get it “sort of” working. That’s most likely the reason why people offered their condolences when I took the role, there was no way we were going to be successful… The upside of this was that it was easy to do better than last time. To increase the pain of failure, at the time of the failed go-live there were hardly any end customers using the chain, however, when we’d go live there would be lots of end customers using the chain. So, failure was not an option.

In a couple of hours brainstorm we (Toine was my partner in crime) chewed on the goal we got from management and made a outline for how to get there. The goal was to upgrade the whole chain in less than 24 hours (so that we could do it in a weekend and leave room for rollback). To realize this we decided to prepare thoroughly and rehearse the whole exercise multiple times on a production alike environment with the people that would do the migration in the go-live weekend. The only difference being that rehearsals took place during normal office hours and not at night. The plan was summarized in 3 simple timeline diagrams that defined for each timeblock what goal must be achieved. This diagrams were the basis for many talks we had. For the months leading up to the go live, the timeblocks were pretty coarse (e.g. 2 weeks), for the last week and the weekend itself it got more fine grained into hours. Note that we deliberately did not state how the goals had to be achieved, we left that open for the various project teams such that they could define a approach that would work best for them. Requiring the team to prepare did put the bar high for lots of teams. We had to convince people over and over again that preparation was key and that it was worth the investment in hours of their scarce resources. We never drifted away from our vision and plan and the freedom with the how part did take away some resistance. What helped us was that the reputation of the department was at stake (especially after the last failure) and if we’d fail, the reputation of the company since the chain was used by a lot of customers. That helped to convince people, nobody wanted to repeat the last failure and with some successful mini-chain releases we proved our approach and built up some credit.

People that typically only focused on “their” phase in the project, were “invited” to help get the chain live. For example: architects and designers helped to work out implementation playbooks and were on-call or on-site during the go-live weekend. Quality assurance staff had most experience with running the new versions of the systems in the chain, therefore, they were on-site during the go-live to help validate the upgrades and analyze issues. Project team that typically got out of the loop when software was handed over to QA, now wrote the detailed implementation playbooks and had to coach the operational maintenance staff to execute the playbooks correctly. Operational maintenance started to realize that if they worked closely with the project team to rehearse the playbooks, they would have much less aftercare issues.

In the initial rehearsals of the playbooks we focused on sub-chains and later we’d let them all work together in a full chain rehearsal, this helped people to find each other and work together in a more efficient way. One of the biggest risks for meeting the 24 hour deadline were several data migrations that in some cases initially took days to run. A lot of effort was invested to optimize these and also test them on copies of production data. This highlighted numerous issues that would have lead to failure in the go-live weekend, but now we could tackle them upfront.

So what did I learn and what am I using (more) now in my current role as architect?

Communication is key
We had over a 100 people active in the weekend and the rehearsals smoothened the communication between persons. Often people that never really worked together now had to work together to analyze and resolve issues. This now started during the rehearsals when there was much less pressure compare to a go-live weekend. It makes it much easier for people to build relations (compared to a high pressure situation).

Get all stakeholders involved early on
Lots of project teams were involved and it proved really important to get all teams on board as soon as possible. After the initial brainstorm we started to communicate our vision to management, project teams, operation maintenance, the internal customer, etc. We made clear what we expected from them and asked them to start preparing. This was 6 months before go-live and going to production was not on top of their priority list. Still we wanted them to start preparing, make first playbook versions, think about dependencies and timelines, etc. We made sure we checked back with teams on their progress on a regular basis and kept communicating the goal and vision.

Create a vision and share it
We started of with a big hairy goal and a vision on how to get there. This goal and vision directed all our actions and provided clarity to all parties that we had to work with. It took some time to sink in, but by consistently repeating it and being almost religious about it, it became a shared goal of all teams involved. Initially many people had doubts, but as time progressed we could show results of the rehearsals and go-live of mini chains. That slowly got everybody to support the approach. Setting a dot on the horizon and making that a share goal proved to be really important.

A project does not end after the handover to quality assurance
The scope of projects should include go-live and aftercare. One of the hurdles we had to take was to convince PMs that their project team should spent time on preparing for the go-live. This was a hurdle because it was not considered to be part of their project deliverables and therefore, they had no resources to work on it. Luckily we’ve been able to convince all of them, but it did take some time. It would have been much easier if project scope includes go-live and does not stop at a sign off by QA.

Operational maintenance is your friend, involve them, always
Make sure you involve the operational maintenance teams early on and keep them involved. They will have to support the software on a daily basis, so it’s important that their needs are addressed and they must feel confident that the new software won’t make their life harder. Also, these are the guys and gals that have access to machines, they can actually check (e.g. connectivity) or fix things for you. Always bring some coffee when you need a favor from them.

Automate as much as possible
Deployment of software should be automated as much as possible. Main reason we spent a lot of time rehearsing the playbooks was that the upgrades of the systems were done manually. In addition to the long duration it proved to be error prone. Playbooks had to be really detailed and we had to convince people to actually follow the steps in the playbook. There is reason for these steps, so don’t skip them or do them in a different sequence. Unfortunately we’ve not been able to realize automated deployments for all systems, so this is an area for improvement.

Data migrations are a risk, validate them upfront with production data
As part of the go-live a number of large data migrations had to be done. Two risks to address here: the duration and the robustness. Both risks have been addressed by running the data migrations on copies of production data. This gave us insight in the duration, which was too long. So optimization was needed. Also it became clear that production data was not as clean as the test data used by QA to test the data migration. The dirty production data actually broke the migration and extra effort was needed to make the migration more robust and clean up the production data.

Give people freedom, but communicate the goals and explain them
We had to deal with 20+ project teams, departments, external suppliers, etc. There was no way we could align them on a detailed level. So we focused on communicating the goals, vision and plan. This explained what we altogether had to achieve and what the (mostly time) constraints were. Then we gave freedom to the various teams to come up with a solid solution for their system. Of course, during the rehearsals the had to prove that their approach worked and we had to check that it did fit into the overall plan, but giving them freedom and being clear about the goals empowered them to work out efficient solutions for their domain.

Working with internal IT departments can be challenging
Internal IT departments are typically focused on ongoing concern and organized by specialities (network, db, machines, OS). When infrastructure is running, they’ll make sure it keeps running. Supporting projects that require quick response times and change their requirements on short notice is not their cup of tea. This simple requires a different type of people and processes. This was a frustrating area and after several escalations we managed to get a dedicated PM that was integral part of our team and had 1 goal: get our change requests executed on time and cut as much procedural waste as possible. This did not mean that we could bypass all procedures, we now had someone that knew how to speed up things. He also forced us to do bit more upfront thinking and initiate change requests earlier. That was fair and we managed to get a workable situation. The only problem was that is was bound to that one person being, when he got replaced, we started all over again. So, if you want to get things done with these groups, involve them early, try to a 1 point of contact and use his/her network.

I’m glad to be on the ‘software producing’ side of IT again, but also very much appreciate the experience of working on the receiving side. It did teach me some new things and reinforced things I already knew but did not do enough. I’m sure I’ll do such a sabbatical again, not now but let’s see what challenges pop up in the future.

One of the earlier principles was “Incremental Development of Architecture“.  This explained the forward thinking part of our Lean Architecture mindset. Of course not as big up front design, but “Just in time, just enough“. In addition to this it is important learn from projects teams, operational maintenance, business and other stakeholders how architectural decisions work out during development and when the systems are in operation. The “Architecture emerging from Projects” principle address this important feedback loop. The idea is that there must be a constant feedback loop from the projects to the architecture role. The lessons learned are important input for next architectural decisions and other projects. In addition, these lessons may trigger adjustment of earlier decisions, guidelines or refinement of the architecture vision.

Not every lesson learned is related to an earlier architectural decision, guideline or vision. Project teams will encounter and solve problems that have not been foreseen as they move along. These lessons also have to be fed back to the rest of the organization and architects must cater for this spreading of the knowledge (and learn from it). The “Always Involved” principle helps to capture these lessons and when applied correct it facilitates both the harvesting of lessons learned and spreading of the information to other stakeholders.

The Cohesion of the 3 C’s of architecture is supported by this principle due to the constant reality check and alignment that is  facilitated by the “Architecture emerging from Projects” principle. Parallel running projects will learn from each other and the architecture stays in-sync with reality. The architects are aware what is being realized, what can be improved and can immediately feed this back into other projects. The short feedback loop also increases Changeability. It is much easier to change or revert a decision shortly after it is taken, compared to when significant time has past and lot’s of other decisions already assumed certain starting points.

This was the tenth in a series of blog posts on Lean Architecture principles, the next one will follow in about a week.

As a negative example: we’ve probably all seen situations where people start “investigating” the possibilities of a cool new technology without having a clue how it could contribute to achieving the business goals. Although this might be fun, interesting and good on the architects resume, it does not add value for the business and therefore no effort should be put into it.

The “business” is the most important stakeholder in any company and therefore also for the architecture role. The business defines vision and goals for the company and architecture must help to achieve the company vision and goals. It it crucial that architects understand the vision and goals and architects therefore will have to be in regular contact with the business. Make sure to actually talk with the business on a regular basis since it is hard to grasp the “real” essence of a vision from a document or slidepack. Understand what the business are aiming for and more importantly, “why” they are aiming for that. After understanding, architects - together with other stakeholders - can define the architecture vision, goals and short term actions. Capture this in a limited set a slides or few page document that can be discussed with and (very important) understood by the business. If you cannot explain to the business why you’re making certain decisions, then there’s more work to do.

Obviously, the Connection part of the 3 C’s of architecture is directly addressed by this principle. Applying this principle will ensure that architects are focussing on what adds values for the business. “Buy in” by the business for your plans is easier achieved, because they recognize their own vision and goals in the architecture vision and goals. Applying this principles also helps to achieve Changeability because you will know what the most likely areas for change are and therefore you’ll know where flexibility makes sense in the architecture.

This was the seventh in a series of blog posts on Lean Architecture principles, the next one will follow in about a week.

As an negative example: if the operations department is not included until a system goes into production, than, most likely the missing operational maintenance functionality will be noticed to late. Pressure to go live from the business will be high, the operations department will be overruled and must operate a system that is not well manageable. This will frustrate the (future) cooperation between operations and the other stakeholders.

“All hands on deck early on” means that at the start of a project all stakeholders are involved and contributing. Examples of stakeholders are: business, users, IT, operations, QA, hosting party, etc. They all will contribute by bringing in (non-) functional requirements and experience from other systems, by challenging decisions, and sometimes by setting the IT guys straight (although a function could be automated -and IT guys generally like to automate everything – the cost to automate it does not pay off because it would be rarely used in practice). Having all stakeholders involved in these early discussions can speed up decision making and makes clear what the main risks are that require further exploration.

When applying this principle there is a crucial role for the architect, he has to use his facilitation and social skills to keep the process progressing, prevent chaos from taking over and record, summarize, document collectively taken decisions. Not only his ‘soft skills’ are important, using his technical knowledge he must ensure that all required aspects get attention. If certains aspects are not thrown into the discussion by any of the stakeholders, then he must do that. A challenging role for the architect, but by empowering the collective brainpower of all stakeholders the architect can focus more on the process and have the stakeholders do most of the work.

A big benefit of this approach is that you will get instant buy-in and commitment from all stakeholders to make the project a success because all have been involved from the start. All stakeholders know what the vision is and why certain decisions are made. This will eliminate a lot of discussion of following project phases.

How does “all hands on deck early on” contribute to the 3 C’s of architecture? The system will be better connected to the business since the business is discussing their vision and requirements with all stakeholders. The vision will not be lost in translation as often happens when the “over the wall” anti pattern is applied. From the start on the project is viewed from many different viewpoints (by different stakeholders), this will gravitate the design and implementation decisions towards a cohesive solution that fits nicely in the environment of all stakeholders. The collective experience of all stakeholders will also enable you to determine which areas are likely to change frequently. It is then clear where to put some extra effort to be prepared for change.

This was the forth post in a series of blog posts on Lean Architecture principles, the next one will follow in about a week.

This is the second of a series of blog posts discussing Lean Architecture principles. Each post will discuss one principle and applying these principles will result in an architecture (process) that is better connected to the business, better able to deal with change and more cohesive. The first principle that we discuss applies to the architect role and is called “Always Involved“. The architect role is not limited to one project phase or even one project, a good architect takes a much broader perspective. The Lean Architect constantly communicates with all stakeholders (from business till operations), plays an active role in running projects, and ensures that lessons learned in projects are known and where applicable used in other projects.

We probably all know examples of architects that are not connected to the business objectives or to the projects being executed in an organization. The architects get an assignment from a project manager, isolate themselves in an office, work for months discussing with each other (but not with business, project, operational maintenance or other stakeholders) and produce a 100+ page document containing their perfect architecture. After delivering the document they abandon the ship and await a new assignment. This is an extreme case (although it does happen in reality), but variations in which the architect is only involved at the start of a project, or only communicates with one of the stakeholders are very common.

Always involved means that architects are involved during the whole lifecycle of a project, from the initial inception of ideas up until (and including) when the deliverables of a project are in production. The degree of involvement may vary depending on the project phase, but architects always stay in the loop. The architects feel responsible for the business goals and are committed to deliver value such that these goals are reached. They are supporting multiple projects and constantly create alignment between stakeholders of all projects and takes lessons learned in projects into account, ensuring that other can learn from it.

How does the principle contribute to the 3 C’s of architecture? A better cohesion is achieved since architects take an active role in all projects. This enables him to ensure that projects learn from each other and that similar problems are solved in a consistent way (e.g. process orchestration). Connection with organizational goals (both business and project) is improved since architects interact with the business and project side on a regular basis. This forces architects to understand these goals and enables him to translate these into IT goals. Architects also ensure that the other stakeholders understand the reasoning for these IT goals. Also, due to the constant involvement in projects, feedback, lessons learned, are immediately incorporated in the architecture vision. Changeability is improved because architects know what parts of the business are most likely to change, he can – together with other stakeholders – make decisions that are facilitating this change or at minimum not creating upfront barriers that would make responding to these changes hard.

What does it bring the organizations when the architects are always involved? First: Priorities are set correct. Both from business and project side architects are fed with priorities and they can now make a balanced decision: on what should they focus now? Impediments that prevent (or shortly will) projects from making progress are on the architects radar. So are the latest market or organizational developments from the business side. Because the architects are constantly fed with information from all stakeholders and results of choices made earlier by themselves, subsequent decisions will be based on real world facts/experience and not on theoretical assumptions.

This was the second in a series of blog posts on Lean Architecture principles, the next one will follow in about a week.

Putting all pitfalls together in one simple 10 item list quickly reveals a grouping of types pitfalls. Number #1 and #2 are both related to organizational aspect. If the culture, mindset and attitude are not right, these are typically the pitfalls that a SOA endeavor may run in to. The next group covers the items #3 till #7, these are all related to architectural/design skills. And the last group, numbers #8 till #10, relates to implementation issues (although proper design could help to prevent these pitfalls from manifesting themselves).

The complete Top 10 SOA pitfalls list is:

Implementation pitfalls

• #10 – Not Invented Here Syndrome
• #9 – Versioning
• #8 – Security

Architectural/design pitfalls

• #7 – Incorrect Granularity of Services
• #6 – SOA does not solve complexity automatically
• #5 – Big Design Upfront
• #4 – Incorrectly applied Canonical Data Model
• #3 – Missing skills

Organizational pitfalls:

• #2 – Unclear ownership/Project based funding
• #1 – Ignoring culture when introducing SOA

It was great fun to compile this list with the four of us and it also helped us to get a sharper view on why SOA projects or programs are challenging and hard to complete succesfully. Hopefully you’ve learned a thing or two and are able to prevent or at least recognize the pitfalls in your day-to-day work.

Thanks for having us,
Rik de Groot, Viktor Grgic, Vincent Partington, and Gero Vermaas

CDM is one of the silver bullets often fired in SOA projects. It should address miscommunication, ease integration and reduce integration costs. It surely can facilitate all of this, but attempts to use a CDM can also turn your SOA project into an endless discussion because one attempts to cover too much, because of a lack of alignment with business and because of a lack of design principles.

A Canonical Data Model (sometimes called CIM: Common Information Model) defines the business entities relevant for a specific integration domain, their relations and their semantics. What added value does a well defined and correctly used CDM bring to the table? First of all, it facilitates a common understanding of what a business entity really is. For example is the ‘Customer’ business entity a person or organization? Or is ‘Customer’ business entity a role that can be executed by a ‘Person’ or ‘Organization’ entity. In the same realm of “understanding”, it facilitates a common understanding of the relations between business entities. This common understanding eases communication between departments and on a broader scope between organizations as illustrated by the SID model of TM Forum. Lastly, integration costs can be significantly reduced if systems to be integrated speak the same language / use the same concepts (language in this case is not a programming language, but an understanding what a business entity is and what the relations between these entities are).

What are the pitfalls when attempting to create and use a CDM? CDM creators often try to boil the ocean and include each and every piece of information used in the organization. This explodes the amount of entities to be modeled and turns the CDM initiative into an endless exercise. A CDM is intended to be used in the integration domain and should therefore only include entities that are relevant in that domain. Another pitfall refers back to SOA Pitfall #10: Not Invented Here Syndrome and are from the ground up developed CDMs. Potential models that could be reused are ignored, while various potential reuseable domain models are available (SID, UDEF and AFD). Some are industry specific, but even then, definitions for customer, contracts, etc. can often be reused. The next pitfall is the big flat CDM without any structuring. This makes the model hard to use and understand, even when you only need to interact with a small part of it. It slows down adoption of the model. Adoption is also slowed down by inconsistencies in naming conventions and modeling patterns used. One of the biggest pitfalls is to not consult domain experts when defining the CDM entities and their relations. A CDM, just like any IT artifact, should support the business. Therefore it is crucial to ensure that the model reflects the business and it not a pure IT view. And lastly there is the pitfall of CDMs based on vendor models or current applications. A model like a CDM should model business concepts and therefore not be bound to vendors or current applications. Both vendors and systems come and go, your business hopefully survives these.

To prevent failure of the CDM the following guidelines can help:

• Develop a CDM in context of concrete projects and include entities that are needed for these projects. Sure, you need to think ahead a little, but this does not imply the model should include every entity you can think of. When looking ahead, consider where the model should be extensible and focus on the entities that are currently needed.
• A CDM is intended to be used for integration and therefore should only cover entities that are used on the integration layer. Entities that are not exchanged between systems should not be part of the CDM. The bright red area of the figure on the right illustrates the information that should be part of the CDM.
• Divide the model into a number of domains with strong cohesion between the entities in a domain and loose coupling between the various domains. This eases understanding and adoption of the model because readers can easily focus on the parts that are relevant for them.
• Check if there are models available that you can use as a base. There are industry specific models available and sometimes these models also include generic concepts like customers, products, contracts, prices, etc. that can perfectly be used outside that particular industry. Examples are SID, UDEF and AFD.
• Collaborate with domain experts, they know the business best and can help you ensure that the model reflects the business.
• Define a limited set of design principles to be used. This will lead to a more consistent and easier to understand model. Clear naming conventions for entities, attributes and relations also helps in this area.
• Provide examples that illustrate how the model should be used, how it should not be used and the reasoning behind it.
• When distributing the model, do it in a way that readers can easily navigate through it.

Defining a CDM is a challenging exercise, but following these guidelines should help you to win the challenge. Not using a CDM in an SOA can introduce extra complexity in the SOA because there will be many point-to-point connections on the data level. As stated in pitfall #6 – SOA does not solve complexity automatically, a CDM is one the items that can reduce complexity.

Next week Viktor will take us to #3…

Incorrect granularity could mean that a service covers too much functionality or too little functionality. Incorrect granularity of services in your SOA can lead to bad performance, low reuse possibilities, leaky abstractions and services without added business value. . Common causes for this are bottom-up and/or top-down design and taking a too narrow perspective (project instead of company scope). In this blog we’ll first take a closer look at the previously mentioned symptoms and their causes. And then we’ll explain why the solution lies in taking a business perspective when designing services.

First the symptoms that could indicate that services in a SOA are not of the right granularity:

• You’re not able to explain to business people (e.g. sales/marketing) what the service does. It simply does not fit in their understanding of the business the company is in. The services are at a level of detail that is irrelevant from a business perspective.
• Many different services are needed to achieve something of value to the business.
• Using the services in the SOA leads to an enormous amount of tiny interactions between systems and the performance of the overall process is dead slow.
• Services are variations of basic CRUD operations (Create, Read, Update, Delete) and do not add any functional value to the basic CRUD operations.
• Service definitions expose how the service is implemented. For example you can deduct from the service definition that a service uses a particular database.
• Of the available services hardly any service is used more then once. Each service seems to be bound to a particular application. This could mean services are too course grained because they encompass a set of functionality that is only used by one (or a few) application(s).
• There is no clear ownership of the service, multiple department feel they should own the service (or parts of it).
• Lot’s of layers of composite services. The top level services in this case may expose interfaces of the right granularity, but the are composed of services that have an incorrect granularity. This incorrect granularity is often caused by one of the above symptoms and extra unnecessary compositions are put in place to compensate for this.

So what types of pitfalls lead to these symptoms?

• A couple of pitfalls can be grouped into bottom-up design. For example, when existing systems are hooked up to the SOA it is tempting to take the API of this system and expose operations in this API as services in the SOA. Easy to do, but these service will have no business meaning, are potentially too fine grained (with CRUD services as the most obvious example) and tend to leak implementation details. Another example of bottom-up design happens when multiple departments (or projects) are dealing with similar functionality and are not aware of each other (or worse, they simply do not want to communicate, also see Not Invented Here syndrome). Each department or project is only considering their own perspective and that will lead to similar services being developed multiple times.
• On the opposite site, top-down design is not the cure to bottom-up design. When service are designed top down and decomposed to lower level services that are decomposed further down you often end up with a myriad of services that can only be used in that part of the decomposition tree. Different branches of the tree may have similar services defined and services in a branch are coupled to other services in that branch.
• Theoretic service design is another cause. Ivory tower architects or designers are designing services without being driven by concrete projects and/or business processes. This usually leads to services that are so generic that no concrete project can use then without modifications. Each project requires changes to the service definitions to make them useable which in turn leads to many variations of similar services.

What is the right level of service decomposition?

• Service should be decomposed to the smallest reusable unit while remaining be understandable by the business owners. Another way to define the optimal size if that the service should not be a grouping of finer grained services that are not available as separate services. Thus leaving open the possibility to compose services of other services.
• Furthermore it is important that one single role in the organization is the owner of the service (multiple roles may use the service of course). Services that are too course grained will be hard to assign to one owner, leading to unclear ownership (something we will discuss in more detail later in this top 10).
• And lastly, the service should be as autonomous as possible. If someone want to use a service he should not be forced to use other services also (note that the implementation of the service may use other services).

Finding the right granularity of services is not as simple as following a recipe. It’s a balancing act that requires knowledge, expertise and trying different options. Design your service using the guidelines above, run through a number of scenarios and see if you see any of the symptoms mentioned pop-up. If so, reconsider…

Next week, Rik de Groot will continue with pitfall #6.