In this blog post we will give a description of an open source Java Application Platform as a Service that we created for our customer, using VMware, Redhat Enterprise Linux, Apache WebServer, JBoss Enterprise Application Platform, JBoss Operations Network, Puppet, Deployit,  F5 Load Balancer and  a Layer7 SecureSpan gateway.

Data Center Quality Platform

The customer wanted a data center quality Java Application Platform with the following features:

• Standard configuration
• Standardized provisioning
• Standardized deployment
• Centralized monitoring
• Centralized access control
• Virtual environment
• Proving technology

Current situation

As the current Java application platform was based on HP-UX on Itanium, the customer was facing high cost for hardware, software licenses and fading support from software vendors.  As all applications ran on a HP Superdome, it was very difficult to add resources to individual applications. In addition, development teams spend too much time taking their software through the development, test and acceptance environments, resulting in slow delivery of software into production. Finally, it was difficult to provide 24×7 availability because all applications are running on a single machine.

Java Application Platform

The following figure illustrate the solution architecture of the java application platform.

In the following paragraphs we will describe the purpose of the most important components.

Dual Data Center – HP

Not shown in the figure, is the hardware setup of the platform. It consists of HP blades setup in two data centers on two different locations. This provides the basic infrastructure for 24×7 availability and fault tolerance.

VMware ESX

VMware ESX is deployed on top of the hardware in the dual data center. This provides us with the ability to create virtual machines and provide high availability in case of single server of single site failures. It also allows us the quickly scale up virtual machines and increase the resources assigned to individual virtual machines.

For all machines in the platform we use a single VMware template image. This image is installed with RedHat Enterprise Linux and a puppet client.

Puppet

Puppet fully automates system management. It is used for the installation of software packages, conformity tests and day to day system administration tasks.  For every type of node, we have a puppet plan. When the machines boots, the puppet agent provisions the machine with all the necessary software and configuration according to the plan for that machine.

The use of Puppet completely automates and standardizes the configuration, ensures 100% reproducibility of the configuration and is fast. Provisioning of a new machine from the template to full operational mode is done in a matter of minutes.

JBoss EAP

JBoss Enterprise Application Platform is the Enterprise Java applications server for all java applications.  The installation and configuration is done by Puppet and uses the official RedHat RPMs.  Puppet configures JBoss to ensure that :

• JBoss management applications authenticate users against Active Directory, providing a single point of authorization for operations.
• A JBoss Oracle database schema is automatically provisioned for that specific instance of JBoss, providing persistence for the JBoss server system state.
• All Business Applications can authenticate users using SAML against the Layer7 Identity provider, providing a single point of authentication and authorization for their customers.
• The JBoss instance is added to the pool in the F5 Load balancer
• The application server is added to the Deployit infrastructure inventory, providing the tenants of the platform with the ability to deploy applications to the server.

JBoss application servers are always deployed in multiples of two, where each server of a pair is assigned to a physically different data center location by VMware.

The use of puppet provides us with a fast and reproducible way of provisioning JBoss application servers, allowing for a fast and reliable scale out mechanism for the applications.

JBoss Operations Network

JBoss Operations Network (JON) is used for monitoring all the resources in the platform.   By default, Puppet installs a JON agent on every machine. This agent scans the inventory of the machine and reports it to the JON server.

JON has a very good support for high availability and fail over. By simply adding a JON server machine, agents will automatically distribute themselves across the servers and failover if necessary. Each JON server also runs a JON agent, making sure that unavailability of a JON server is also covered.

In JON we created a number of alert templates for different resource types (os, apache, jboss, jon, puppet, etc.)  that will monitor and report critical conditions on the system.  All error messages from the JBoss servers logs are reported as incidents.

All alerts and clearing conditions from JBoss Operations network are reported via SNMP to TNG Unicenter.

Through the  use of JBoss Operations Network all machines, servers and resources in the platform are automatically added to the centralize monitoring system.

Deployit

Deployit is used for the automated deployment of applications onto the platform. It automatically deploys all the application components in a stack to the appropriate containers.  Deployit :

• deploys static content and proxy configuration to the apache webservers,
• deploys enterprise java application components to all individual JBoss servers in the farm,
• executes SQL scripts to the database,
• configures the F5 loadbalancers to add or remove servers or applications to the pool,
• applies environment specific changes to the application configuration.

The deployment plan for a specific application is prepared in close cooperation between the application developer and platform management staff. When the deployment plan is finished, developers can deploy new versions of the application themselves, directly from a build tool or manually.This ensures solving any installation or configuration problem isn’t postponed until the application is installed for production use, but rather is solved at the early stage of any development.

The same deployment plan is used for all environments. Authorization can be configured per enviroment and per application. LDAP is used to authorize software developers to deploy and configure an application for development and testing purposes, while integration specialist can deploy the application in production.

The use of Deployit provides the platform with a fully automated and standardized deployment mechanism, improving the speed of deployment of applications through the development, test and acceptance environments while reducing the number of staff involved and lowering the number of configuration errors.

F5 Load Balancer

The F5 Load balancers is used to support scalability and fail over for the JBoss Application Server farm.

The pools are configured to use a sticky session protocol based upon the JSESSIONID session cookie. If the cookie is not present, round-robin load balancing of the HTTP requests is performed.

Puppet adds the JBoss servers to the  pool in the F5 Load Balancer.

When a server is scheduled for a restart, the server is taken out of the pool. This ensures that this server does not get any new request, but will still be servicing existing sessions. When the session count in JBoss drops to zero, the server is restarted and restored to the pool.

The use of the F5 Load Balancer provides us with the ability to increase and decrease the number of servers in the farm, provide load balancing, fail over and graceful decommissioning of servers in the farm.

Layer7 XML Gateway / Identity Provider

The Layer7 SecureSpan Gateway is used as centralized security policy enforcement point and SAML identity provider.

Layer7 supports multiple authentication methods, Kerberos, digital certificates, username+password and is able to use multiple identity stores.

Puppet configures all JBoss application servers with SAML support and configures Layer7 as the identity provider: JBoss receives authentication (identity) and authorization (roles) information as a SAML-token. The information contained in the token is translated to a standard JEE-principal user (using a tiny layer of custom code), so all JEE applications can access the authentication and authorization information in a standard way.  Whether the JEE application is a web application or provides webservices, from a security there’s no distinction. All application designers have to do is declare the application security roles conform the JEE standard.

The use of Layer7 standardizes the authentication and authorization for all business applications and centralizes access control.

Conclusion

The customer wanted a modern data center quality Java Application Platform to ensure that java applications could be deployed with lower cost and with high availability and easy scalability.

VMware, the dual data center, Layer7, the F5 Load balancer and JBoss provide the infrastructure for a  high availability and scalability for any java application. The combination of VMware, Puppet and Deployit are the fabric to enable continuous delivery of enterprise java applications.

Through virtualization and automated provisioning and deployment it has become possible to add a completely new, correctly configured machine to a cluster in a matter of minutes, completely secure and under full monitoring.

For details on the calculation, refer to the book. It’s well worth the read and if you’d like to start off with a shorter version, there’s a whitepaper available here. The short summary is that simplicity trumps all other quality aspects of a system. Simple systems are easier to maintain, perform better, scale better, etc. Simple is the opposite of complex and using his approach you can calculate the relative complexity of different alternatives. Number of business functions and number of connections per system are the main input for the calculation. The one with the lowest complexity is the best alternative.

Start situation

Enough theory, what is the problem at hand? The picture on the right is the system landscape that is already in place. In reality there are more systems surrounding it, but I limited it to the systems that are considered in the alternatives. Systems A/B/C/D are all part of a fulfillment chain and as you can see they are all decoupled by EAI adapters. The datawarehouse (DWH) is fed with product details that are delivered by the fulfillment chain systems.

The functionality used down the chain from system A till D is now limited to order processing. However, system A now needs an information service that enables it to decide on certain properties for the order submitted to system B. This information service needs:

• Details of delivered products that are present in system B and C (and also fed to the DWH)
• Business rules that are only known in system C
• Business rules that are only known in system D

EAI coordination, no DWH

Alternative 1: EAI Coordination without DWH

In this alternative a composite EAI service is created that is invoked by system A and subsequently gets the required details from system B, then asks system C to enrich it and apply its business rules and finally asks system D to apply its business rules. After that it’s able to send the response back to system A.

Putting the number of business functions and connections per subsystem into the spreadsheet adds up to 422 SCUs. So adding one new component and 4 new connections almost tripled the complexity!

EAI Coordination with DWH

Alternative 2: EAI Coordination with DWH

In this alternative a composite EAI service is created that is invoked by system A and subsequently gets the required details from the DWH (instead of from system B), then asks system C to enrich it and apply its business rules and finally asks system D to apply its business rules. After that it’s able to sent the response back to system A.

Down the chain

Alternative 3: Down the chain

In this alternative there is no composite EAI service that does the orchestration, but the request is pushed down the fulfillment chain (incl. EAI adapters) just like the normal orders. Each system and EAI adapter exposes a new service and once system D produces the result, it traverses up the chain though all systems again before it’s handed over to system A. At first glimpse this may seem a nice solution because a similar pattern is used as for the order processing, but each system/EAI adapter now has one extra business function and has extra connections.

Conclusion

First pitfall is that many procurement departments and management still have the mindset that everything is clear at the start of a project, when there is a rough idea what problem must be solved and there is relative confidence that a positive business case can be made. To get a better picture of the costs service integrators are contacted such that the business case can be made. A preference for a COTS based solution is expressed and from there on the discussions focus on the features of the COTS product, license costs and hourly rates for professional services. Little attention is given to deeper analyze the problem to be solved, get core requirements clear and see if a COTS product is the appropriate solution. For example a real time ETL product may support Change Data Capture, but using it would create a really tight coupling to the internal data model of the source system and requires the ETL product to understand how to interpret that data model. Although technically possible, it is definitely not preferred from an architectural perspective.

Selecting a COTS product is a decision that cannot be taken lightly, it frames the architecture and design space and comes with a set of consequences. This decision is hard to revert, so to make this decision you should understand the problem you trying to solve, the constraints that apply, the pro’s and con’s of the product etc. Proof of concepts provide valuable information and it forces you to really think about the problem you’re solving. Domain and technical experts should be able to decide what the relevant requirements (both functional and non-functional) to be included in the proof of concept are. Don’t rely on experts of the service integrator or COTS vendor alone, ensure you have some neutral experts involved. Should you pay for execution of such a proof of concept? Definitely not for licenses involved, for hours spent you could. Costs of a proof of concept should be a fraction of the total costs and you may be saving yourself an ton of money by doing a serious proof of concept and learning from it. Remember, you get what you pay for.

When the decision for a specific COTS product is made next pitfalls come into play. Although the product is envisioned to work off the shelf, there is always that tiny bit of configuration to be done. More often than not this turns out to be a significant effort. Now the team starts to really dig into the requirements and understand them. Some can be realized easily with the COTS products, some not and that’s where trouble starts. If you feel like your pushing a square through a round hole, you know you got the wrong tool for the problem at hand. Deciding to no use the COTS product (for everything) often is a politically sensitive subject. Therefore, be open about this and always keep the option open to use other tools to if the COTS product becomes an impediment. The goal is not to use the COTS product XYZ, the goal is to solve the business problem at hand.

The next pitfall concerns the use of development practices. If significant configuration effort and/or custom coding is required it becomes a development project instead of just deploying a COTS product. If it is a development project, it should be treated like that and established practices like version and release management, continuous integration, automated testing, automated deployment, etc. must be applied. These practices build quality into the process and enable teams to identify and fix issues early instead of after go-live. Are these practices ignored? Yes, at least I’ve seen it happen. The question is “why?”.

I suspect that one cause is mix of skills in the team, or actually, the lack thereof. When running a project based on a COTS product the natural tendency is to search for team members that are trained for that product. Although they know the COTS product inside out, they may not have a software engineering background and/or are not hooked into the software engineering community. They are simply not aware of these practices. And if they are aware of the existence of these practices, they may not know how to execute them. For these reasons it’s important to have a mixed skills set in the team. Sure you need COTS product experts, but you also need all-round software engineers, a DBA, operating system experts, etc. They might not be full time team members during the whole project, but they must be available at short notice and stay informed about the current state of the project.

Another reason why these practices are not applied may be limitations of the COTS product. If the COTS product doesn’t support version management by itself and integration with a well known version management system like Subversion is not possible, then you’ll have to work out an alternative. This may involve some manual exporting etc, but the fact that it’s not supported out of the box, does not imply that these practices can be skipped without negative consequences.

A third reason to have a mixed team is to prevent the golden hammer syndrome. If the team consists only of COTS product experts, they’ll try to solve every problem with the COTS product. Even when a bit of scripting or the use of standard command line tools would do the trick much faster and with less effort. Generation of test data is an example, typically a bit of scripting is enough to generate test data. However, I’ve seen situations in which complete ETL workflows in the COTS product were developed to generate test data. That took significant effort and the speed at which the data was generated was low (which became a problem when the database had to be populated with larger amounts of data). An all-round software engineer must be able to generate that test data in a much more efficient way.

These are some pitfalls and suggestions how to deal with them that I cam across. They can be prevented by having the right mix of skills involved and by focussing on the problem to be solved instead of focussing on COTS product features. What pitfalls have you seen? How should we prevent them from happening?

From their homepages:

Node.js: Evented I/O for V8 JavaScript. (A framework for building completely non-blocking servers in Javascript)
NPM: A package manager for node.
CoffeeScript: A little language that compiles into JavaScript
Express: High performance, high class web development for Node.js
Jade: Node Template Engine
Redis: An open source, advanced key-value store

In this guide I will take very small steps so that you can verify that you are check whether you are still on track.
The result is an extremely performant, scalable and lightweight alternative for web development.

So I installed a brand new Ubuntu 11.4 virtual machine and got started. After following some tutorials, googling and some tweaking this is the step-by-step guide I came up with.

Installing Node.js

$ sudo apt-get install python-software-properties
$ sudo add-apt-repository ppa:jerome-etienne/neoip
$ sudo apt-get update
$ sudo apt-get install nodejs


Installing npm

$ apt-get install curl
$ curl http://npmjs.org/install.sh | sudo sh


Node.js Hello World

$ mkdir helloworld
$ cd helloworld
$ vi helloworld.js


insert

var http = require('http');
http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
}).listen(1337);
console.log('Server running at http://127.0.0.1:1337/');

$ node helloworld.js

check http://127.0.0.1:1337

Add Coffeescript

Writing all that javascript is going to lead to lots and lots of brackets, semi-colons and parantheses. Let’s use Coffeescript to make our code readable again.

$ sudo npm install -g coffee-script
$ cd ..
$ mkdir hellocoffee
$ cd hellocoffee
$ vi hellocoffee.coffee


insert

http = require "http"
http.createServer( (req, res) ->
  res.writeHead 200, {"Content-Type": "text/plain"}
  res.end "Hello Coffee!!"
).listen 1337

console.log 'Server running at http://127.0.0.1:1337/'

$ coffee -c hellocoffee.coffee
$ node hellocoffee.js

And check http://127.0.0.1:1337 again.

Adding Express

Node.js is a very powerful, but it still requires quite a bit of boilerplating to use it for web development. That’s where Express comes in. It is build on top of another middleware layer called Connect and gives you much better support in handling requests, rendering, templates and responses.

$ cd ..
$ sudo npm install -g mime qs connect express
$ mkdir helloexpress
$ cd helloexpress
$ express
$ npm install -d
$ node app.js


check http://127.0.0.1:3000

vi app2.coffee

insert:

express = require('express')app = express.createServer()

# Setup configuration
app.use express.static(__dirname + '/public')
app.set 'view engine', 'jade'

# App Routes
app.get '/', (request, response) ->
  response.render 'index', { title: 'Express with Coffee' }

# Listen
app.listen 3000
console.log "Express server listening on port %d", app.address().port

$ coffee -c app2.coffee
$ node app2.js


And check http://127.0.0.1:3000 again.
Now that we have Coffeescript working with Express let’s add sessions.

Add session support

$ vi app2.coffee

insert after app.use express.static(__dirname + ‘/public’)

app.use express.cookieParser()
app.use express.session {secret: "Coffeebreak" }

insert after app.get ‘/’, (request, response) ->

request.session.views++

replace

response.render 'index', { title: request.session.views + ': Express with Coffee and sessions' }

$ coffee -c app2.coffee
$ node app2.js

Check http://127.0.0.1:3000 and refresh a few times. You should see the counter increase.

Install Redis

Open a new terminal.

$ wget http://redis.googlecode.com/files/redis-2.2.11.tar.gz
$ tar -zxvf redis-2.2.11.tar.gz
$ cd redis-2.2.11/
$ sudo apt-get install build-essential
$ make
$ ./src/redis-server


Add redis session to node

$ npm install -d redis connect-redis
vi app2.coffee

insert after express = require ‘express’

RedisStore = require('connect-redis')(express)

change

app.use express.session {secret: "Coffeebreak", store: new RedisStore, cookie: { maxAge: 60000 } }

$ coffee -c app2.coffee
$ node app2.js


In the Redis terminal you should now see something like:
[25489] 24 Jun 04:57:11 - DB 0: 1 keys (1 volatile) in 4 slots HT.
[25489] 24 Jun 04:57:11 - 1 clients connected (0 slaves), 799032 bytes in use


Check http://127.0.0.1:3000 and refresh a bunch of times
kill and restart node

$ node app2.js

Check http://127.0.0.1:3000 again and refresh a few more times
Note that you started where you left off.

And that brings us to the end of this guide. As you can see the stack is very powerful and extremely small, lightweight and clean. With all the session information in the Redis store the availability of the entire systems comes down to the availability of Redis. And with these guys working on a proper clustering solution it will be very interesting to see where all this is going in the near future.

Recently I worked at a classical, hierarchal organization with all kinds of rigid procedures. The organization probably has (or at least had) very good reasons to have these procedures: when something goes wrong it will be headline news. For that reason they are quite anxious for things to go wrong. This has resulted in a standard waterfall development process. Because of this, it takes a long time to develop new applications. In some cases it takes even longer to deploy a new application in the production environment due to a complex and document heavy handover to the operations department.

About 5 years ago this organization decided to change to a flexible architecture¹⁾ by means of a SOA. The reason for this was mainly to be able to respond more quickly to business requests. In general the key benefits of a SOA are among others: IT-business alignment, improved development speed and re-usability of services. This should lead to a lower Time to Market, hence the decision for the organization to choose this architecture.
Noble benefits, but what are the added values of a flexible architecture to this kind of static organizations when time related business requirements are involved?

In my opinion there are none. The flexibility of the architecture is outbalanced by the rigidity of the design, development and deployment process. A (very) small project would probably take 4-5 months to design, 2-3 weeks to build and test and something like 3-4 months to deploy. That includes all the paper work. A flexible architecture would probably shorten this by a few weeks. Instead of 9 months it would perhaps take 8.5 months. The processes involved are simply too time consuming to take a reasonable profit of a flexible architecture.

A flexible architecture can therefore only come to its full potential when the organization itself is flexible, for example by a more iterative and incremental way of working.

More about this, in relation to Conway’s law in a next blog.

¹⁾ Flexible architecture = an architecture which meets changing business demands without major changes to the architecture itself first.

Xebia is helping many organizations in the Netherlands, France, the United States and India with implementing an agile way of system development. In most of the cases the Scrum method is applied and very good results are achieved. Business and IT are working much closer together, resulting in more quality and much more customer satisfaction. However, lately we also see a trend in problems that seem to occur in (almost) every organization. Software is developed in a fast way with high quality, but it takes forever to get it in production. The more teams are being formed, the more interdependencies between the teams occur, disturbing the heartbeat of the teams. Teams have to wait for other teams to finish, or they have to work together to complete the work. Also it’s not always clear which team should do which job. Next to this product owners are being over asked. They have to supply the teams with user stories / epics, but also have to know upfront which team to ask it, have a clear understanding of the direction of the solution and at the same time define non-functional requirements like availability or system responsiveness.

The scrum masters in the teams are not able to solve all this. They have the interest of their team at hand and are mainly focused on the process to do a good job. They are less focused on the contents of the product backlog or how team deliverables are related to the broader context. However the real situation in many organizations is different. The environment of scrum teams is complex and often not very agile. Scrum teams need to work in the context of an existing organization, which has formed over years. To be able to do so, they need a context. Especially the context in the content in important. Here is where architecture comes in.

The picture below shows how Scrum synchronizes the business cycles with the IT project cycles. Without Scrum the time lines for supply and demand are out of sync. By delivering software faster, Scrum is able to follow the priority of the business and develop IT systems which are in line with current demand (not with a demand from last year). However, businesses also have a longer term strategy, which can make them give significant competitive advantages in the near future. To be able to direct the IT development towards this longer term goals, the business needs architecture. In this way the short terms Business cycles and IT development cycles are in sync, but they move forward towards reaching the ultimate business goals. The picture shows the three different stages from being out of sync, to being in sync and also be moving towards the strategic goals.

Therefore, architecture in the agile world has a very important role to play. However it has to adapt to the new way of working and the new way of looking at the world. In an earlier blog from Xebia, we talk about the 3 C’s of architecture. We defined three goals of the architecture function in IT organizations: The Three C’s of Architecture. These are: Connection, Cohesion and Changeability. Taking these as the prime principles of architecture provides focus on what to do and how to position architecture in the Agile organization. We deliberately do not speak of “the architect”, because architecture is created by many people: domain experts, (experienced) software developers, testers, operations people, software architects, enterprise architects, etc. Therefor we will talk about the architecture role.

The Connection with organizational goals is achieved by using enterprise architecture as a way to create a common understanding about the business and IT in the future. But also the architecture role plays an important part in helping the product owner(s) to specify their requirements and give directions to possible solutions (in line with the enterprise architecture). Cohesion between teams and cohesion between teams and their surroundings, are the main aspects the architecture role has to focus on, by helping the teams getting more productive. Finally the architecture role focuses on Changeability by continuously evolving the architecture and adjusting it to the real world. Architecture creates its own learning cycle. Next to this the modularity of the architecture has to be taken care of, so software is easily adapted without too much interference with other elements in the architecture.

In the coming months we will write a series of blogs, about several aspects of architecture in the agile world. First of all we will write about experiences in the field. How things are done. What can be done better and which lessons are learned. Next to this we would like to present out view about the architecture role in an Agile environment. Which tasks should be done, who contributes to creating architecture. Also we would like to write some blogs, about tools and methodologies used to be able to fulfill this new role in the best way possible. Finally we would like to talk about the architecture itself. How can we make architecture agile? How do we modularize the application architecture, or in what way does virtualization, provisioning and automated deployment help? With all these blogs we intend to redefine the world of architecture in the agile world. Comments and opinions are most welcome, so the architecture and agile communities merge together with a way of thinking on architecture.

The focus in this blog is on the solution architect or application architect. The way the Enterprise architect deals with Scrum will be explored more in detail in a later blog. This blog combined with the previous 3 blogs can be also downloaded as a whitepaper from the Xebia website: http://www.xebia.com/architects_scrum

What is the role of the architect?
Last blog I presented the illustration as shown below. In this blog I will focus on the parts of this illustration in which the solution architect / application architect plays a role

Product backlog
Traditionally, architects have been the chief designers in software development. Projects often start with an extensive project architecture document, in which the solution has been written down using many words and lots of paper. In today’s Scrum practice, the focus is on the most prioritized epics and user stories in the product backlog. Epics and user stories are usually short. Teams take epics and user stories from the backlog and design the best solution for them. So what is the role of architects at this stage?

In fact, architects have a very important role during this stage, especially in larger organizations with many software systems. They discuss the requirements with the organization and make the first global design decisions for each epic on the product backlog in line with the overall architecture. In large organizations with multi-layered SOA architectures, the number of software components used can be very high. Ten to twenty agile teams working on new features is not an exception. Architects make the first global design decisions because they are the ones who oversee the whole picture. Questions have to be answered, such as ”do we need extra components?”, ”do we adjust existing components?” and “in which layer of the architecture do we want to implement the change?” These decisions are not made by architects alone. They discuss them with product owners to determine their real needs, and they discuss them with the agile teams to determine how the software really works.

The design decisions made at this stage are normally written down on no more than one-half page. If this is not the case, then the architects could lose themselves in details. These details will be discussed in the teams. The architects can also make an initial estimation of the complexity of the solution, which can be used when epics are picked up by the teams.

Architects help to narrow the cone of uncertainty. The cone of uncertainty describes the evolution of uncertainty during a project. At the beginning of a project, comparatively little is known about the product or work results, so estimates have a high level of uncertainty. Architects help the product owner reduce the level of uncertainty, making it easier for the product owner to set priorities and for teams to focus on the specific solution within their areas of expertise.

Agile teamwork

Architects are present daily on agile teams. They have to discuss the upcoming epics/user stories in the next sprints. Additionally, daily design decisions are made together with the teams on a lower level. This provides feedback to the architects about the positive and negative sides of the global design decisions they have made. Architects without experience in the teams will lose contact with the real world.

Architects also have an important role in monitoring the architectural principles through all layers of the architecture. This means, for example, that they have to know whether all business rules are implemented within the business layer or whether customer information is always stored in the CRM component. Being on the team can help architects make architectural principles more practical; the architectural principles evolve from a practical standpoint.

The architectural principles also include more technical aspects, such as the use of middleware and hardware. Ideally, the “definition of done” within a team includes the deployment to production. In this respect, the DEVOPS movement has recently become stronger. The architects can provide guidance in performance issues or security issues. This brings these aspects of software development closer to software developers and analysts.

Cooperation in teams between developers, testers, performance specialists, hardware specialists and others will help to create and capture new ideas. These new ideas might not always be implemented directly. New ideas can also impact other teams, or software redesign may require extra time or expertise that is not available to the team, which means the team cannot implement the ideas. So how can architects deal with these ideas?

Dealing with design ideas

Architects who are helping to groom the product backlog and who are working in agile teams are often very busy with daily problems. But there is a second, equally important part to their role: to envision the future. They step out of the daily problem-solving activities to evaluate the big picture. What new ideas are desirable and possible? How does current progress influence the overall architecture? This thinking results in what we call design ideas.

First of all, new design ideas can emerge from grooming the product backlog and working in the teams. Ideas for new components or ideas to change existing components emerge. The focus in agile teams is often on the speed of delivery, so the business can adapt quickly to changes in its environment. Although this is a major strength of Scrum, it also makes architectural deficiencies very likely. Software redesign is not done properly because of lack of time, or is not done at all. It is important that these shortcomings are acknowledged and written down as design ideas for future sprints. A design idea with a valid business case of its own should also be placed on the product backlog.

Design ideas arise not only from software architecture; technical and business areas are also a rich source of design ideas. Virtualization of the hardware can save a lot of money and reduce complexity. Regulatory compliance demands more control over traceability. These types of ideas are often executed by the IT department, without a direct connection to business goals. However, these design ideas should be linked to business requirements and find their way to the product backlog. If necessary, new agile teams — guided by product owners with help from the architects — can be formed to handle the implementation of these ideas.

New ideas also emerge from the strategic alignment of the business and IT. This is the domain of enterprise architecture and IT strategy. IT can become a critical success factor for the business and a catalyst for future business change. Enterprise architects or chief information officers are closely related to and cooperate with organization management. Their goal is to see how IT can be employed as an important production factor in the organization. Enterprise architects translate strategic choices into new design ideas. These design ideas will also find their way to the product backlog.

Traditionally, architects decide on the architecture and on the conformance of software to the enterprise architecture. But a transparent assessment of the real business value is rarely made. Conformance is king. This dynamic has changed in the new reality of Scrum. We employ a different approach for translating enterprise architectural requirements into concrete software artifacts. The main purpose of design ideas is to select the right ideas or architectural features to implement, based on actual value. The first step is to organize design ideas in a backlog. Architects use that backlog while collaborating with product owners in evolving the product backlog. This collaboration is based on a partnership aiming towards business value, now and in the future.

Business value does not necessarily equal cost savings. It is also found in continuity planning (e.g. when the number of servers needs to be increased to preserve system stability) or in software adaptability. The choice of which design ideas to implement is ultimately made by the product owner. Architects have an important role in explaining and promoting these ideas.

Balancing the work

In companies with a complex multi-system environment and multiple agile teams, architects should be very close to the product owner, helping to create maximum business value out of the requirements on the product backlog. By sketching the solution and the complexity, architects help the product owner to set even better priorities, and help the support teams with the outcome of these global design discussions. Also, new design ideas will be discussed with the product owner and possibly placed on the product backlog. An effective relationship between the product owner and the architects can create more business value with less effort.

If we follow Chandler’s principles, the adoption of Scrum should include an alignment of structure, process, and strategy — or it will fail. At best, it results in a sub-optimization within the overall enterprise. In this white paper we explored the alignment of Scrum with the architectural processes and showed how the role of architects within an agile context takes shape in order to create more business value.

Agile architects focus on the most important outcomes of their work. They go back to basics and chose their tools deliberately. By focusing less on documentation and much more on communication, they can become an important accelerator for sprint teams and help the organization to build business value by fostering new design ideas.

But architects not only focus on daily business activities and on strengthening the heartbeat of the Scrum teams. The second pillar of their work is envisioning the future and clarifying the enterprise architecture. All architects therefore should claim time for this and communicate the benefits for business and IT.

Agile architects do not live in ivory towers. They are pragmatic and involved in daily business, while not forgetting their role in envisioning the future.

Architects add vision

Strengthening the heartbeat of Scrum teams is an overly limited view of the contribution that architects can make in an agile environment. They also help develop new design ideas, assist the organization in pursuing its strategy with the aid of IT, identify IT opportunities which could promote business value and assist in their implementation. Architects envision the future. In this way architects can also become catalysts for change. New IT developments can have a major impact on the business strategy and on the priorities for IT development.

This part of the work of the architect is often forgotten in organizations that are adopting Scrum. But is a very important part of the architects work. The following illustration shows the two primary aspects of the architects’ role. On one side they play a role in strengthening the heartbeat. On the other side, they play a role in envisioning the future.

As can be seen in the illustration, there are of course the necessary connections between the two parts of the job. The concept of “design ideas” is helping a lot to be able to connect the two worlds. In my next post I will go into more detail about the way the architects can connect both worlds and add more value to the organization.

The forgotten questions of  Scrum

In the 1960’s Alfred Chandler already wrote that the organization structure of an organization is tightly related to its strategy and based on its organizational processes.  In the optimal world according to Chandler: Structure follows processes follows strategy.

But what if a process change occurs in the organization? Should that also not at least influence the organizational structure and the strategy? This blog is the first of an intended series of more, where I will try to find answers to this question when the new process that is implemented is a Scrum process.

A Scrum implementation in any organization is a clear process implementation in a specific part of the organization, namely software development. A new way of working, that starts by having business requirements and ends by having working software. A clear scope of a process. However having the statement of Chandler in mind three questions arise, which have not been given many thoughts in the Agile world:

1. Was there a new strategy from the organization that made Scrum happen? Or will the strategy be influenced by it?
   Implementing a process changes without a clear justifiable goal will fail in the end. That’s why the reason Scrum is implemented in organizations should be examined carefully. Is it really there to support the business strategy towards the future or is is just a nice hobby from a well intended IT-manager?
2. Are new organization departments being formed now at the IT-side and business side?
   Implementing Scrum is forming self organizing teams in the organization. But what does that mean for the old functional departments? Are they ceasing to exist? Does the role of the line manager change? and how? And what about the more supporting departments like architecture, quality and operations?
3. What about the influence this process change has on processes in the organization that are linked to this Scrum process?
   Processes in an organization are linked together. They need each other to work optimally. A scrum process should have clearly influence on the business requirements process. But also all business processes should be able to adopt changes more quickly then ever before. Next to this the more supporting IT-processes should find their new way of working in line with the Scrum development process.

In the next upcoming blogs I will first focus on the third question. Specifically I will examine the architecture processes in an organization and the way they are influenced or even incorporated in the Scrum process.  Next week more…

We are sure the Agile hot topics for 2011 are:

Agile Management

The managers finally find their place within an agile organization. In 2010 a lot of managers struggled with their position, in 2011 they find their spot. It’s called facilitation island and it smells like paradise and still feels like hard work.

Agile Management Consultancy

The management consultancy world invites itself to the party. The consultancies bring new (project)managementmodels around Agile with them to establish their position in the agile world. This is a mixed blessing: it’s both confusing and useful at the same time. Confusing because Agile is put in models that oppose the Agile values and put more emphasis on process tracking, Big Up Front Plans and management control. Useful as it attracts senior management attention to Agile and help accelarate agile thinking on typical board management topics as Agile strategy implementation, Agile KPI management, Agile risk management, Agile budget control and Agile organization structures.

Agile at Scale

Corporations shift from scaling agile to agile at scale. Instead of using Agile only for projects, agile is implemented at corporate level with a stronger focus on product portfolio management to deliver more value. Senior management gets seriously involved in the prioritisation of what’s really important. The discussion is no longer about prioritizing projects, it’s about what delivers most value to the company in the long run.

Agile Maturity

Many companies start their journey with agile. They seek to learn from the past experiences of those that went before them. Those trailblazers on the other hand are ready for the ‘beyond Agile’. Agile maturity with a focus on the journey from just starting to completely incorporating agile is the subject of many publications and presentations in 2011. Agile maturity is how the different groups share their experiences and learn from each other.

Agile Metrics

Now the use of Agile has become more and more serious, the need for good metrics has become even more real. The Agile world rids itself of its cowboy stigma by creating more transparancy in a way that is useful for everyone in an organization. Putting things on a wall is no longer enough, you can’t expect everybody in a 30,000 people company to visit all the teams daily or even weekly. The big agile adoptions embrace metrics to enable running agile at scale.

Risk Management

Given these other trends risk management becomes more important. The IT risk management so far has been the laughing stock of real risk management. 2011 is the year IT risk management matures and agile leads the way.

Agile Results Only Work Spheres

The Results Only Work Environment (ROWE) or ‘new way of working’ (Dutch: “Het nieuwe werken”) is here and implemented by many corporations. While corporations want the benefits from working with Agile, there is also a strong desire to apply ROWE. An answer is found to combine distributed working and teameffort into a new way of working called: Agile Result Only Work Spheres also known as AROWS.

We wish you all a pleasant 2011 and may it be a MoreAgile year !