Xebia is helping many organizations in the Netherlands, France, the United States and India with implementing an agile way of system development. In most of the cases the Scrum method is applied and very good results are achieved. Business and IT are working much closer together, resulting in more quality and much more customer satisfaction. However, lately we also see a trend in problems that seem to occur in (almost) every organization. Software is developed in a fast way with high quality, but it takes forever to get it in production. The more teams are being formed, the more interdependencies between the teams occur, disturbing the heartbeat of the teams. Teams have to wait for other teams to finish, or they have to work together to complete the work. Also it’s not always clear which team should do which job. Next to this product owners are being over asked. They have to supply the teams with user stories / epics, but also have to know upfront which team to ask it, have a clear understanding of the direction of the solution and at the same time define non-functional requirements like availability or system responsiveness.

The scrum masters in the teams are not able to solve all this. They have the interest of their team at hand and are mainly focused on the process to do a good job. They are less focused on the contents of the product backlog or how team deliverables are related to the broader context. However the real situation in many organizations is different. The environment of scrum teams is complex and often not very agile. Scrum teams need to work in the context of an existing organization, which has formed over years. To be able to do so, they need a context. Especially the context in the content in important. Here is where architecture comes in.

The picture below shows how Scrum synchronizes the business cycles with the IT project cycles. Without Scrum the time lines for supply and demand are out of sync. By delivering software faster, Scrum is able to follow the priority of the business and develop IT systems which are in line with current demand (not with a demand from last year). However, businesses also have a longer term strategy, which can make them give significant competitive advantages in the near future. To be able to direct the IT development towards this longer term goals, the business needs architecture. In this way the short terms Business cycles and IT development cycles are in sync, but they move forward towards reaching the ultimate business goals. The picture shows the three different stages from being out of sync, to being in sync and also be moving towards the strategic goals.

Therefore, architecture in the agile world has a very important role to play. However it has to adapt to the new way of working and the new way of looking at the world. In an earlier blog from Xebia, we talk about the 3 C’s of architecture. We defined three goals of the architecture function in IT organizations: The Three C’s of Architecture. These are: Connection, Cohesion and Changeability. Taking these as the prime principles of architecture provides focus on what to do and how to position architecture in the Agile organization. We deliberately do not speak of “the architect”, because architecture is created by many people: domain experts, (experienced) software developers, testers, operations people, software architects, enterprise architects, etc. Therefor we will talk about the architecture role.

The Connection with organizational goals is achieved by using enterprise architecture as a way to create a common understanding about the business and IT in the future. But also the architecture role plays an important part in helping the product owner(s) to specify their requirements and give directions to possible solutions (in line with the enterprise architecture). Cohesion between teams and cohesion between teams and their surroundings, are the main aspects the architecture role has to focus on, by helping the teams getting more productive. Finally the architecture role focuses on Changeability by continuously evolving the architecture and adjusting it to the real world. Architecture creates its own learning cycle. Next to this the modularity of the architecture has to be taken care of, so software is easily adapted without too much interference with other elements in the architecture.

In the coming months we will write a series of blogs, about several aspects of architecture in the agile world. First of all we will write about experiences in the field. How things are done. What can be done better and which lessons are learned. Next to this we would like to present out view about the architecture role in an Agile environment. Which tasks should be done, who contributes to creating architecture. Also we would like to write some blogs, about tools and methodologies used to be able to fulfill this new role in the best way possible. Finally we would like to talk about the architecture itself. How can we make architecture agile? How do we modularize the application architecture, or in what way does virtualization, provisioning and automated deployment help? With all these blogs we intend to redefine the world of architecture in the agile world. Comments and opinions are most welcome, so the architecture and agile communities merge together with a way of thinking on architecture.

But this time I was really impressed. The device is a possible replacement for IBM Datapower XML Security Gateway. But the way they designed the device is totally different.

What CrossCheck networks did was creating a device with just security as main use case. First of all it was an XML gateway, nowadays is does support HTML, XML, SOAP, FTP, JMS and others.
It also translates different flavors of JMS to each other, it can even convert from IBM MQ to JBoss MQ directly.

The Forum Sentry XML Gatewat comes in two flavors, as an appliance and as an VMWare image.
This is useful for the Development and Test environments.

The main use-case for the device is as a security device. It will stand in the DMZ and acts as a level 7 security gateway. This is the same use-case as the IBM Datapower, only the marketing is a little bit different, CrossCheck Networks markets the device mainly as a security appliance. IBM did market the Datapower mainly as a ESB in the past. Nowadays they also market it mainly as a security appliance.

The main differences I spotted between those two where:

My conclusion after this short demo
The Forum sentry has some advantages when you compare it to the IBM Datapower XML Security Gateway XS40. The main difference is that you can do more whith only one appliance. You can replace an IBM Webseal, a virus scanner and an IBM Datapower XS40 with only one device.
My advice is to take this device in considerations where you have to choose for an XML firewall/ hardware ESB.

The focus in this blog is on the solution architect or application architect. The way the Enterprise architect deals with Scrum will be explored more in detail in a later blog. This blog combined with the previous 3 blogs can be also downloaded as a whitepaper from the Xebia website: http://www.xebia.com/architects_scrum

What is the role of the architect?
Last blog I presented the illustration as shown below. In this blog I will focus on the parts of this illustration in which the solution architect / application architect plays a role

Product backlog
Traditionally, architects have been the chief designers in software development. Projects often start with an extensive project architecture document, in which the solution has been written down using many words and lots of paper. In today’s Scrum practice, the focus is on the most prioritized epics and user stories in the product backlog. Epics and user stories are usually short. Teams take epics and user stories from the backlog and design the best solution for them. So what is the role of architects at this stage?

In fact, architects have a very important role during this stage, especially in larger organizations with many software systems. They discuss the requirements with the organization and make the first global design decisions for each epic on the product backlog in line with the overall architecture. In large organizations with multi-layered SOA architectures, the number of software components used can be very high. Ten to twenty agile teams working on new features is not an exception. Architects make the first global design decisions because they are the ones who oversee the whole picture. Questions have to be answered, such as ”do we need extra components?”, ”do we adjust existing components?” and “in which layer of the architecture do we want to implement the change?” These decisions are not made by architects alone. They discuss them with product owners to determine their real needs, and they discuss them with the agile teams to determine how the software really works.

The design decisions made at this stage are normally written down on no more than one-half page. If this is not the case, then the architects could lose themselves in details. These details will be discussed in the teams. The architects can also make an initial estimation of the complexity of the solution, which can be used when epics are picked up by the teams.

Architects help to narrow the cone of uncertainty. The cone of uncertainty describes the evolution of uncertainty during a project. At the beginning of a project, comparatively little is known about the product or work results, so estimates have a high level of uncertainty. Architects help the product owner reduce the level of uncertainty, making it easier for the product owner to set priorities and for teams to focus on the specific solution within their areas of expertise.

Agile teamwork

Architects are present daily on agile teams. They have to discuss the upcoming epics/user stories in the next sprints. Additionally, daily design decisions are made together with the teams on a lower level. This provides feedback to the architects about the positive and negative sides of the global design decisions they have made. Architects without experience in the teams will lose contact with the real world.

Architects also have an important role in monitoring the architectural principles through all layers of the architecture. This means, for example, that they have to know whether all business rules are implemented within the business layer or whether customer information is always stored in the CRM component. Being on the team can help architects make architectural principles more practical; the architectural principles evolve from a practical standpoint.

The architectural principles also include more technical aspects, such as the use of middleware and hardware. Ideally, the “definition of done” within a team includes the deployment to production. In this respect, the DEVOPS movement has recently become stronger. The architects can provide guidance in performance issues or security issues. This brings these aspects of software development closer to software developers and analysts.

Cooperation in teams between developers, testers, performance specialists, hardware specialists and others will help to create and capture new ideas. These new ideas might not always be implemented directly. New ideas can also impact other teams, or software redesign may require extra time or expertise that is not available to the team, which means the team cannot implement the ideas. So how can architects deal with these ideas?

Dealing with design ideas

Architects who are helping to groom the product backlog and who are working in agile teams are often very busy with daily problems. But there is a second, equally important part to their role: to envision the future. They step out of the daily problem-solving activities to evaluate the big picture. What new ideas are desirable and possible? How does current progress influence the overall architecture? This thinking results in what we call design ideas.

First of all, new design ideas can emerge from grooming the product backlog and working in the teams. Ideas for new components or ideas to change existing components emerge. The focus in agile teams is often on the speed of delivery, so the business can adapt quickly to changes in its environment. Although this is a major strength of Scrum, it also makes architectural deficiencies very likely. Software redesign is not done properly because of lack of time, or is not done at all. It is important that these shortcomings are acknowledged and written down as design ideas for future sprints. A design idea with a valid business case of its own should also be placed on the product backlog.

Design ideas arise not only from software architecture; technical and business areas are also a rich source of design ideas. Virtualization of the hardware can save a lot of money and reduce complexity. Regulatory compliance demands more control over traceability. These types of ideas are often executed by the IT department, without a direct connection to business goals. However, these design ideas should be linked to business requirements and find their way to the product backlog. If necessary, new agile teams — guided by product owners with help from the architects — can be formed to handle the implementation of these ideas.

New ideas also emerge from the strategic alignment of the business and IT. This is the domain of enterprise architecture and IT strategy. IT can become a critical success factor for the business and a catalyst for future business change. Enterprise architects or chief information officers are closely related to and cooperate with organization management. Their goal is to see how IT can be employed as an important production factor in the organization. Enterprise architects translate strategic choices into new design ideas. These design ideas will also find their way to the product backlog.

Traditionally, architects decide on the architecture and on the conformance of software to the enterprise architecture. But a transparent assessment of the real business value is rarely made. Conformance is king. This dynamic has changed in the new reality of Scrum. We employ a different approach for translating enterprise architectural requirements into concrete software artifacts. The main purpose of design ideas is to select the right ideas or architectural features to implement, based on actual value. The first step is to organize design ideas in a backlog. Architects use that backlog while collaborating with product owners in evolving the product backlog. This collaboration is based on a partnership aiming towards business value, now and in the future.

Business value does not necessarily equal cost savings. It is also found in continuity planning (e.g. when the number of servers needs to be increased to preserve system stability) or in software adaptability. The choice of which design ideas to implement is ultimately made by the product owner. Architects have an important role in explaining and promoting these ideas.

Balancing the work

In companies with a complex multi-system environment and multiple agile teams, architects should be very close to the product owner, helping to create maximum business value out of the requirements on the product backlog. By sketching the solution and the complexity, architects help the product owner to set even better priorities, and help the support teams with the outcome of these global design discussions. Also, new design ideas will be discussed with the product owner and possibly placed on the product backlog. An effective relationship between the product owner and the architects can create more business value with less effort.

If we follow Chandler’s principles, the adoption of Scrum should include an alignment of structure, process, and strategy — or it will fail. At best, it results in a sub-optimization within the overall enterprise. In this white paper we explored the alignment of Scrum with the architectural processes and showed how the role of architects within an agile context takes shape in order to create more business value.

Agile architects focus on the most important outcomes of their work. They go back to basics and chose their tools deliberately. By focusing less on documentation and much more on communication, they can become an important accelerator for sprint teams and help the organization to build business value by fostering new design ideas.

But architects not only focus on daily business activities and on strengthening the heartbeat of the Scrum teams. The second pillar of their work is envisioning the future and clarifying the enterprise architecture. All architects therefore should claim time for this and communicate the benefits for business and IT.

Agile architects do not live in ivory towers. They are pragmatic and involved in daily business, while not forgetting their role in envisioning the future.

Architects add vision

Strengthening the heartbeat of Scrum teams is an overly limited view of the contribution that architects can make in an agile environment. They also help develop new design ideas, assist the organization in pursuing its strategy with the aid of IT, identify IT opportunities which could promote business value and assist in their implementation. Architects envision the future. In this way architects can also become catalysts for change. New IT developments can have a major impact on the business strategy and on the priorities for IT development.

This part of the work of the architect is often forgotten in organizations that are adopting Scrum. But is a very important part of the architects work. The following illustration shows the two primary aspects of the architects’ role. On one side they play a role in strengthening the heartbeat. On the other side, they play a role in envisioning the future.

As can be seen in the illustration, there are of course the necessary connections between the two parts of the job. The concept of “design ideas” is helping a lot to be able to connect the two worlds. In my next post I will go into more detail about the way the architects can connect both worlds and add more value to the organization.

Architects and the agile values

Most of the literature concerning the role of architects in an agile context focuses on the Agile flow itself and how architects can avoid disturbing that flow. Mike Cohn, in his book “succeeding with agile” makes the distinction between coding & non-coding architects. In where he states that the coding architects will have less trouble finding their new role in de Agile development process.

An architect within a team has to be able to code himself. He is a team member, who has more experience in structuring the application being build compared to other team members.  By using that experience he can add value to the team.  Scrum has no particular role for non-coding architects. The question rises if this is totally true.

The role of architects in an agile environment should at least follow the agile values from the agile manifesto. Examining these values leads to guide lines that architects need to follow in able to be of optimal value for the organization.

1. Individuals and interactions over processes and tools
   The first agile value states that we are dealing with people and the way they communicate. Processes and tools can be helpful in certain situations, but should never become overemphasized. Obligatory thick architecture documents at the start of a project or the lengthy review procedures of documents, are examples where processes and tools have gotten too much attention. In a Scrum environment this is out of the question. Therefor Architects have to change their focus from written to oral communication. They have to communicate in person as much as they can with all of the stakeholders in the organization. Architects need to be able to communicate equally  well with the product owner as with individual members of the teams. Communication has to be two-way, enabling the architect be fed with ideas from others. Architectural frameworks like TOGAF can be used, but only if it helps the architect to get the message across. The method can never prescribe the activities the architects should do.
2. Working software over comprehensive documentation
   Architects in the past have been doing this from a distant perspective from the builders. Communicating through extensive design documents proofed to be not the most effective one.  Although architects themselves can be very proud of the documents they produced, this will not work within an environment with Scrum. Implementing Scrum in an organization urges the architects to focus much more on the communicative part of their job.An architect has to focus more on drawing sketches. He has to talk about the sketches to all the stakeholders. Adjust the sketches to remarks that customers en suppliers make.  Adjust the sketches to ideas given by the development team. But also the sketches to the impediments found during implementation, which makes them more in line with the real world.  Using sketches helps the architect to create a common view about the design of the software with all stakeholders, which helps to raise the quality of the software developed.
3. Customer collaboration over contract negotiation
   Architects have a big role to play in aligning IT and business. In companies with a complex multi-system environment, and multiple agile teams, the architects should be very close to the product owner, helping him to create maximal business value out of the requirements on the product backlog. By sketching the solution and giving insight in the complexity of this solution, the architect can help the product owner in settings its priorities even better, and support teams with the outcome of these global design discussions. An effective relationship between the product owner and the architect can create more business value with less effort.
4. Responding to change over following a plan
   The architect has to be able to respond to daily change. The best way to experience daily difficulties which can lead to changes, is to be close to the development team. The architect should be there to be consulted about all kind of practical design issues. The architect has to adjust his own sketches of the future continuously. The architect should be able to let go of some of his ideas if they are very difficult to execute.

Following these agile principles the architect has to communicate & interact more than ever with all stakeholders. In his communication he should make use of sketches instead of large documents. His working desk should be close to the teams and he should daily talk to the product owner. Most of this is focused on “strengthening the heartbeat” of the Scrum teams. He can add more quality to the software and speed up teams by smart designs.

However in my opinion there is missing something. So what is the missing ingredient what could make architects really successful?  In next week blog I give you my answers to that.

The forgotten questions of  Scrum

In the 1960’s Alfred Chandler already wrote that the organization structure of an organization is tightly related to its strategy and based on its organizational processes.  In the optimal world according to Chandler: Structure follows processes follows strategy.

But what if a process change occurs in the organization? Should that also not at least influence the organizational structure and the strategy? This blog is the first of an intended series of more, where I will try to find answers to this question when the new process that is implemented is a Scrum process.

A Scrum implementation in any organization is a clear process implementation in a specific part of the organization, namely software development. A new way of working, that starts by having business requirements and ends by having working software. A clear scope of a process. However having the statement of Chandler in mind three questions arise, which have not been given many thoughts in the Agile world:

1. Was there a new strategy from the organization that made Scrum happen? Or will the strategy be influenced by it?
   Implementing a process changes without a clear justifiable goal will fail in the end. That’s why the reason Scrum is implemented in organizations should be examined carefully. Is it really there to support the business strategy towards the future or is is just a nice hobby from a well intended IT-manager?
2. Are new organization departments being formed now at the IT-side and business side?
   Implementing Scrum is forming self organizing teams in the organization. But what does that mean for the old functional departments? Are they ceasing to exist? Does the role of the line manager change? and how? And what about the more supporting departments like architecture, quality and operations?
3. What about the influence this process change has on processes in the organization that are linked to this Scrum process?
   Processes in an organization are linked together. They need each other to work optimally. A scrum process should have clearly influence on the business requirements process. But also all business processes should be able to adopt changes more quickly then ever before. Next to this the more supporting IT-processes should find their new way of working in line with the Scrum development process.

In the next upcoming blogs I will first focus on the third question. Specifically I will examine the architecture processes in an organization and the way they are influenced or even incorporated in the Scrum process.  Next week more…

The posts are, in order:

1. Lean Architecture Principle #1: Always involved
2. Lean Architecture Principle #2: Travel light
3. Lean Architecture Principle #3: Think Big, Act Small
4. Lean Architecture Principle #4: All Hands on Deck early on
5. Lean Architecture Principle #5: Just in time, just enough
6. Lean Architecture Principle #6: Iterative Architecture Development
7. Lean Architecture Principle #7: Architecture Initiated by Business Goals
8. Lean Architecture Principle #8: Focus on the Value Stream
9. Lean Architecture Principle #9: Comprehensible over comprehensiveness
10. Lean Architecture Principle #10: Architecture emerging from Projects
11. Lean Architecture Principle #11: Freedom where possible, standardize where needed

These posts provide a complete overview of the mindset of a lean architect. The principles were derived from our daily jobs as architects and of experience of our colleagues.

After all this writing we became very curious. Do you recognize the added value of the principles we stated? Do you apply them in your work as an architect in an agile environment? Did you discover any principles yourself? Did you like our posts?

We would love to read and reply to your comments!

Out of fear to lose control over projects and in an attempt to maintain cohesion on IT systems , tight controls and thus tight restrictions are often placed on IT projects. These restrictions, or standards and regulations as they are also called, have evolved historically in organizations. Ideally these standards result from best practices and experiences during projects, but often the are based on personal preferences, wishes to gain some experience or other non rational reasons. Updating these standards frequently is left behind resulting in outdated standards which as a result hinder projects in creating the solutions the business needs.

Standards should be only concerned with reducing complexity. Suppose that your team is in the business of enterprise application integration, and your company has tens of dozens of dedicated systems for separate departments to be integrated. Your mission statement could be that this team has to solve all integration problems between all these systems, and thus that you shun no integration problem, however difficult and intricate. In a narrow perspective this would be correct, however in perspective of the larger picture solving every possible integration problem on your way is undesirable. This will result in tens of dozens of dedicated solutions with a nightmare in administration and maintenance as a result. So in this case standardization of both proces and techniques is clearly called for.

Examples of relevant standards in this environment are:

• Use JMS for asynchronous communication.
• Use SAML2, OpenId, or some other open standard for authentication company wide.
• Use one central directory for storing authentication information.
• And so on…

On the other hand, continuing in the field of enterprise application integration, suppose the standards and regulations in your company dictate the use of web services using the WS-* stack. A lot has been written on the pro’s and con’s of the WS-* stack in comparison to REST for instance, so this is clearly a case of a disputable standard. Now suppose you need to solve a publish and subscribe integration problem. In this case using JMS instead of WS-* or REST is a viable, and much cleaner and easier way to implement solution. Being limited to the WS-* stack in this case is clearly not in the benefit of the business, which should be the most significant factor in the end.

So what are the requirements for good standards ?

• For every standard there should be a clear connection with the business goals. If a standard is not endorsed by the business, then it probably does not help the business reaching their goals.
• Standards should leave room for innovative idea’s. Very tight standards kill creative new idea’s not thought of at the time the standards were devised.
• To get support and endorsement for the standards, these standards should be accompanied by the argumentation behind the standard. This makes the argument on following standards in projects much easier.
• Standards should be continually updated to comply with the current best practices in the field.

The Cohesion of the 3 C’s of architecture is supported by this principle due to the fact that standards are prescribed where needed, thus creating the cohesion between IT solutions where applicable, and limiting complexity. Connection is accomplished by being able to create the solutions the business really needs, within the burden of to constraining standards.

This was the eleventh and last in a series of blog posts on Lean Architecture principles. Next week will show a short summary of the discussed Lean Architecture principles.

One of the earlier principles was “Incremental Development of Architecture“.  This explained the forward thinking part of our Lean Architecture mindset. Of course not as big up front design, but “Just in time, just enough“. In addition to this it is important learn from projects teams, operational maintenance, business and other stakeholders how architectural decisions work out during development and when the systems are in operation. The “Architecture emerging from Projects” principle address this important feedback loop. The idea is that there must be a constant feedback loop from the projects to the architecture role. The lessons learned are important input for next architectural decisions and other projects. In addition, these lessons may trigger adjustment of earlier decisions, guidelines or refinement of the architecture vision.

Not every lesson learned is related to an earlier architectural decision, guideline or vision. Project teams will encounter and solve problems that have not been foreseen as they move along. These lessons also have to be fed back to the rest of the organization and architects must cater for this spreading of the knowledge (and learn from it). The “Always Involved” principle helps to capture these lessons and when applied correct it facilitates both the harvesting of lessons learned and spreading of the information to other stakeholders.

The Cohesion of the 3 C’s of architecture is supported by this principle due to the constant reality check and alignment that is  facilitated by the “Architecture emerging from Projects” principle. Parallel running projects will learn from each other and the architecture stays in-sync with reality. The architects are aware what is being realized, what can be improved and can immediately feed this back into other projects. The short feedback loop also increases Changeability. It is much easier to change or revert a decision shortly after it is taken, compared to when significant time has past and lot’s of other decisions already assumed certain starting points.

This was the tenth in a series of blog posts on Lean Architecture principles, the next one will follow in about a week.

Documentation is important in architecture. Architecture is primarily a long-term thing. Even though in a lean-style architecture the focus is on adding value as early as posible, the results of our actions last for the lifetime of the architecture itself. This usually spans years. If we take a step back and reflect on architecture and value creation, we can ask ourself why we write documentation. The prime objective of writing documentation is communication. We want to communicate why we made certain choices, why we standardized on a certain infrastructure. We write things down because someday, we will no longer be there to communicate these things! The goal of (architectural-)documentation is to create some kind of “organisational memory”. Documentation describes the knowledge that should be retained within
an organisation.

The focus of documentation is communication. Communication is about delivering a message to a public. Each public has their own way of approaching. To create documentation that will actually be read, we have to take the audience into account. Documentation needs to be comprehensible for the targeted audience, or it will become TAGRI (They Aren’t Gonna Read It) documentation (thanks to
Scott Ambler). For some audiences it may be important to be as comprehensive as possible, however, just like the agile manifesto, we prefer comprehensible over comprehensiveness.

Documentation should also be just good enough, see our earlier principles of ”Just in time, just enough” and “Travel Light“. Any architectural document should provide just enough content to “get the job done” and nothing more. The hard part is determining what is good enough. Also, artifacts will become less
“good enough” over time. This implies that artifacts are also living things, that should be updated and expanded incrementally.

For example, in my not to recent past, I worked in a software company that wrote specifications for (software)components. These specifications were written in a slightly formalized form of natural language. Naturally this introduced a lack of clarity, leading to questions and new versions of the
specification. In order to prevent this tedious and time-consuming feedback-loop, the descission was made to create new specifications using UML and OCL in a way as formal as possible. This led to very detailed specifications, almost mathematically sound and complete. However, these specifications
were meant to be used in tenders. The audience of the specifications were accountmanagers, businessanalist and other business focused people. These people had little to none experience using and reading formal languages, therefore the specifcation was completely unusable. Had the specification been comprehensible, communication would have been easier.

So how does this match the 3 C’s of architecture? We obviously gain better connection to the business if we communicate in a manner that is comprehensible.This was the ninth post in a series of blog posts on Lean Architecture principles, the next one will follow in about a week.