But the question (and the initial answers) kept nagging at me all day. And while I sat down with a glass of good whisky in the evening I got back to the question. Here is what I came up with:

• Agile is simple, not easy
• Experience bootstraps learning
• Organizational gravity

Agile is simple, not easy

Any agile methodology is usually very light on rules and have-to-dos. That makes it simple, but not easy.
Behind and because of those simple rules there are still very complicated issues.
“Features must be broken down into chunks that can be implemented in x days” is very simple, but very hard to do if you have no experience. (and sometimes even if you do have experience.)
“At the end of every sprint you need to have a retrospective”. Again very simple. But actually creating a team and environment that can really learn and improve their own process is hard.

So yes Agile is simple, but hard

Experience bootstraps learning

One of the most important things about Agile methodologies is that they make it possible and encourage learning teams.
Having someone with experience however allows a team to accelerate learning in the beginning. Someone who has been in a similar situation before can pick lessons learned and share those with the team so some wheels do not need to be reinvented. As long as the team only uses those ideas as a starting point for their own learning it can accelerate the initial learning.

Organization gravity

Lyssa Adkins has a brilliant quote in the book “Coaching Agile Teams: A Companion for ScrumMasters, Agile Coaches, and Project Managers in Transition”.

“Gravity Works”. Yes it does. Rock climbers know this and plan for it. So do Agile coaches.

Agile transitions are not limited to the teams doing the actual development. They are part of much bigger picture. Other departments need to at least be aware of the transition and probably have to change considerably.
For example walk into the operation department of a big company that is doing waterfall projects and tell them that you are going to release every project every month. They’ll laugh at you.
Most managers actually really like the fake certainty of waterfall projects..
Agile does not solve any problems by itself, it just makes them painfully clear. Not all organizations are mature enough to realize that and will usually blame Agile for all those problems that are suddenly surfacing.

Those are only some examples where organizational gravity will play up. They will constantly pull your Agile transition towards the old status quo.
An experienced Agile coach will recognize those signals and work with the clients to keep an Agile transition on track towards the goals that were set.

Just like a regular coach.

The game is designed to let people experience the how to plan with stable teams and let them experience the advantages of planning with agile projects. It does this by giving a group a somewhat simplified portfolio wall and challenge them to optimize the value generated by the teams.
After they have made the perfect 3 year plan, we of course hit them with random events to mess up that plan.
The explanation of the game is up at tastycupcakes.org

From their homepages:

Node.js: Evented I/O for V8 JavaScript. (A framework for building completely non-blocking servers in Javascript)
NPM: A package manager for node.
CoffeeScript: A little language that compiles into JavaScript
Express: High performance, high class web development for Node.js
Jade: Node Template Engine
Redis: An open source, advanced key-value store

In this guide I will take very small steps so that you can verify that you are check whether you are still on track.
The result is an extremely performant, scalable and lightweight alternative for web development.

So I installed a brand new Ubuntu 11.4 virtual machine and got started. After following some tutorials, googling and some tweaking this is the step-by-step guide I came up with.

Installing Node.js

$ sudo apt-get install python-software-properties
$ sudo add-apt-repository ppa:jerome-etienne/neoip
$ sudo apt-get update
$ sudo apt-get install nodejs


Installing npm

$ apt-get install curl
$ curl http://npmjs.org/install.sh | sudo sh


Node.js Hello World

$ mkdir helloworld
$ cd helloworld
$ vi helloworld.js


insert

var http = require('http');
http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
}).listen(1337);
console.log('Server running at http://127.0.0.1:1337/');

$ node helloworld.js

check http://127.0.0.1:1337

Add Coffeescript

Writing all that javascript is going to lead to lots and lots of brackets, semi-colons and parantheses. Let’s use Coffeescript to make our code readable again.

$ sudo npm install -g coffee-script
$ cd ..
$ mkdir hellocoffee
$ cd hellocoffee
$ vi hellocoffee.coffee


insert

http = require "http"
http.createServer( (req, res) ->
  res.writeHead 200, {"Content-Type": "text/plain"}
  res.end "Hello Coffee!!"
).listen 1337

console.log 'Server running at http://127.0.0.1:1337/'

$ coffee -c hellocoffee.coffee
$ node hellocoffee.js

And check http://127.0.0.1:1337 again.

Adding Express

Node.js is a very powerful, but it still requires quite a bit of boilerplating to use it for web development. That’s where Express comes in. It is build on top of another middleware layer called Connect and gives you much better support in handling requests, rendering, templates and responses.

$ cd ..
$ sudo npm install -g mime qs connect express
$ mkdir helloexpress
$ cd helloexpress
$ express
$ npm install -d
$ node app.js


check http://127.0.0.1:3000

vi app2.coffee

insert:

express = require('express')app = express.createServer()

# Setup configuration
app.use express.static(__dirname + '/public')
app.set 'view engine', 'jade'

# App Routes
app.get '/', (request, response) ->
  response.render 'index', { title: 'Express with Coffee' }

# Listen
app.listen 3000
console.log "Express server listening on port %d", app.address().port

$ coffee -c app2.coffee
$ node app2.js


And check http://127.0.0.1:3000 again.
Now that we have Coffeescript working with Express let’s add sessions.

Add session support

$ vi app2.coffee

insert after app.use express.static(__dirname + ‘/public’)

app.use express.cookieParser()
app.use express.session {secret: "Coffeebreak" }

insert after app.get ‘/’, (request, response) ->

request.session.views++

replace

response.render 'index', { title: request.session.views + ': Express with Coffee and sessions' }

$ coffee -c app2.coffee
$ node app2.js

Check http://127.0.0.1:3000 and refresh a few times. You should see the counter increase.

Install Redis

Open a new terminal.

$ wget http://redis.googlecode.com/files/redis-2.2.11.tar.gz
$ tar -zxvf redis-2.2.11.tar.gz
$ cd redis-2.2.11/
$ sudo apt-get install build-essential
$ make
$ ./src/redis-server


Add redis session to node

$ npm install -d redis connect-redis
vi app2.coffee

insert after express = require ‘express’

RedisStore = require('connect-redis')(express)

change

app.use express.session {secret: "Coffeebreak", store: new RedisStore, cookie: { maxAge: 60000 } }

$ coffee -c app2.coffee
$ node app2.js


In the Redis terminal you should now see something like:
[25489] 24 Jun 04:57:11 - DB 0: 1 keys (1 volatile) in 4 slots HT.
[25489] 24 Jun 04:57:11 - 1 clients connected (0 slaves), 799032 bytes in use


Check http://127.0.0.1:3000 and refresh a bunch of times
kill and restart node

$ node app2.js

Check http://127.0.0.1:3000 again and refresh a few more times
Note that you started where you left off.

And that brings us to the end of this guide. As you can see the stack is very powerful and extremely small, lightweight and clean. With all the session information in the Redis store the availability of the entire systems comes down to the availability of Redis. And with these guys working on a proper clustering solution it will be very interesting to see where all this is going in the near future.

What I learned during this week is just how powerful stories are. And that they do not need to be big or elaborate. Let me explain.

My girlfriend had just gotten her scuba diving certification when it was slightly warmer in the Netherlands. But she had not seen more than 8 fish of 3 different species while getting the certification.
So it was quite a culture shock for her to be diving in the Red Sea, which has hundreds of different species and always tens of different types of fish to see at any given moment.

With this overload of information it was hard for her to learn any of the new fishes. Trying to point out a type of fish in the book and then underwater did not prove very effective.
That wasn’t until I realized how I remember the different types of fishes and that was through , sometimes extremely short, stories, usually about a fish behavior.

Some examples:

Seahorse
This is the story of how the male seahorse has a womb-like organ in which he carries the babies.

Butterfly fish
Most butterfly fishes mate for life and are almost always found in pairs.

Sergeant Major
Sergeant Majors are very territorial, especially when they have just deposited their eggs. Which can be seen a purple spots on rocks.
You will often find that a picture of a Sergeant Major is completely blurred and out of focus, because it decided to attack the camera at the last moment.

Goby Symbioses
There are actually 2 ways Gobies work together with other animals. The most common is that they clean other fish. They stay in an area called a cleaning station. Whenever a fish wants to be cleaned, it swims to a cleaning station and it will be cleaned by the Gobies and other cleaning fish. The most remarkable is that these fish would normally consider these fish prey. It is remarkable to see a goby swim not just in the mouth of a huge grouper, but also come out alive.

And the other great story about Gobies is their symbioses with a type of shrimp. The shrimps will burrow a hole in the sand and the Goby will keep watch. They communicate with their antenna and fins. Amazing to see when it happens.

When I started sharing these and other stories behind the fishes the underwater world suddenly came alive.

So how will that benefit you when you are at the office doing serious work? Stories matter; They allow you to connect with other people much better. Dry facts are a good way to communicate information, but stories will make people remember or act. Maybe even inspire.

Today I saw a brilliant TED talk by Tom Chatfield called “7 ways games engage the brain”. While watching the presentation and going through these 7 ways, I realized that while I have seen these playing games, I have also seen these happen in a good Scrum.

The 7 ways are:

1. Experience bars measuring progress
2. Multiple long and short-term aims
3. Rewards for effort
4. Rapid, frequent and clear feedback
5. An element of Uncertainty
6. Windows of enhanced attention
7. Other people

I will go through each of the points comparing World of Warcraft to a Scrum.

Experience bars measuring progress

World of Warcraft:

In World of Warcraft you have a very literal experience bar when you level a character from level 1. But even when you get to the maximum level the game changes from focus around gaining experience and levels and shifts to gaining better and better gear. To make this process more visible people have actually created ways to distill the level of someone’s gear into a single number. Just so they can see their own improvements and compare theirs to others.

Scrum:

In Scrum, all progress is tracked by a burndown chart.

Multiple long and short-term aims

World of Warcraft:

Tom compares killing creatures in World of Warcraft with opening boxes looking for pie. Finding 5000 pies is boring; finding 15 is fun. World of Warcraft makes it fun by having a whole lot of different quests and challenges to overcome on a wide range of levels. From the killing of monster hoping he will drop item X to vanquishing the big dragon who is threatening to destroy the entire world.
Which is probably why I have killed 85592 creatures since they started keeping track without being bored out of my skull yet.

Scrum:

Scrum (Or Agile Modeling, or whatever you want to call it) does pretty much the same by breaking long term goals and visions down into smaller and smaller pieces. At the top are visions and we work our way down through themes, epics, user stories. And even those are broken into tasks, the equivalent of “kill 10 bears” quests.

Reward for effort

World of Warcraft:

In World of Warcraft you always get some reward for doing anything. Kill any monster and you will get a little bit of loot, possibly some experience. Gathering resources and crafting obviously gives rewards too. The interesting bit here is that even the mighty Blizzard messed up in this department, but they fixed it later in the game.
As I mentioned before, at maximum level the game changes from leveling a character to improving his gear. But because of point 5, the need for uncertainty, and the desire to extend the playing experience, the best bossed can only be killed once a week and drop random items from a fixed list. This meant that someone who was either unlucky or already had all the items from a boss would get no reward. This was changed a few years ago with the introduction of points to buy items from when you killed any boss, giving everybody at least some points to spend on stuff.

Scrum:

Anything you do in Scrum is related to some task that is being tracked. These tasks are small, typically between 4 and 10 hours of work. After every task you get rewarded with the feeling of having completed something. This is usually done with the ceremony of moving your task on the Scrum Board from In Progress to Done.

Rapid, frequent and clear feedback

World of Warcraft:

In World of Warcraft all feedback is instant and clear. Kill a monster, get experience and items; Craft an item and receive it; Hand in a quest and you can pick a quest reward; Level up and you will receive new abilities; Fail at killing a boss and you will die and have to run back to your corpse, have bad tactics and you lose a match. You get a reward for doing something good, and get punished when you do something bad.

Scrum:
Scrum (with some borrowing from XP) also has multiple feedback cycles. The first is Continuous Integration and automatic testing. On every checkin of code the CI application will check out your code, compile it, run all your tests and report the outcome. This gives you near instant feedback that your code is still in decent shape.
Then there is the burndown chart feedback. Are we still on track to make our Sprint goals? How about the epic goals?
At the end of every sprint we demonstrate our changes to the Product Owner and other stakeholders. This allows them to give feedback on what we did the past weeks.
And of course we have the retrospective, where we can give feedback on what went well and what could be improved in delivering working software.
All initiatives to give rapid, frequent and clear feedback.

Element of uncertainty

World of Warcraft:

The rewards in World of Warcraft are a mix of certain and uncertain awards. Will this monster drop my quest item? will I receive an extra skill when I make this item?
Bosses are known to drop certain items, but what they drop will be random. This adds to the excitement of killing bosses to find out if they drop the item you want from them.

Scrum:

All of Software Development is all about uncertainty. And Scrum is no exception to that. A lot of things are uncertain during a sprint.

• Will my code still compile and all the test run?
• Will we make our sprint goal?
• Will the demo work when it is supposed to?
• Will the stakeholders like what they see when they get to see it?

Window of enhanced attention

World of Warcraft:

Have you ever tried to start a conversation with someone who is in the middle of trying to kill a boss he has never killed before? That is immersion. The person is completely focussed on achieving her goal and will not be disturbed by the game until after the encounter.

Scrum:

In Scrum we also have the notion of not changing or disturbing a team during a Sprint. Between Sprints stakeholders can change whatever they want, but during a Spring the team is in control of how and when. This allows a team to be immersed in the problem at hand and solve it.

Other people

World of Warcraft:

Being a Massively Online Roleplaying Game, World of Warcraft is for a large part about player interaction.
There is cooperation because you need to work together to kill the tougher bosses, but there is also competition between players. Both in who can kill the toughest bosses first or has the best gear, but also all out warfare in player versus player combat.

Scrum:

Is obviously all about working together. It is a tool for people to collaborate and produce great quality software that delivers real business value. A lot of people have to be working together to make that happen.

To wrap it all up

A good working Scrum is eerily similar to World of Warcraft, which is interesting, because we can now ask ourselves the question: “Would I rather work, or be playing World of Warcraft?”. If the answer is the latter, you might want to take a look into which above areas you are not addressing properly at work.

Now I do not know what they are going to do to make things right in the future, but if we go by past experience most people will want to enforce even stricter entrance criteria.
There are a couple of problems with this approach:

• Having to test every component for these extra criteria is a lot of work
• The longer the list of criteria is, the harder it will be to guarantee that every component does adhere to all of them
• When integrating multiple components it is the interaction where the complexity is, not the separate components
• You will only prevent foreseen errors

Especially the last two make it very unlikely that having stricter entrance criteria will work. So what can we do?

All of this reminds me of something in ancient IT history. I was not around myself, but I have heard much about the Punch Card days. Just to refresh everyone’s memory. This was an age when if you programmed something you made a punch card. You then took a stack of these cards to an operator who at some point in time ran your program on a big computer with a fraction of the computing power that you currently have in your microwave.

Now these programmers were in pretty much the same boat. They had to get it right the first time around. Any typo or mistake and at best a couple of hours were wasted, potentially days.
Currently however there are very few programmers around that care about typos before running a program. Let alone trying to figure out more complex mistakes. Why is that?
The answer is of course the compiler. A compiler can check your code for these mistakes much more quickly then you can. Why would you spend hours trying to find errors in your code manually when the compiler can find most syntactic errors for you in minutes or seconds?

The next great invention was the invention of the Unit Test. It was a lot more involved than just running a compiler on your code, but it did allow you to very quickly find not just syntactic errors, but also most of your logical errors.

Then came Continuous Integration. No longer content in finding errors in a single persons code we are now able to find another class of logical errors, which are introduced by the complexity of having multiple people work on the same application and relying on external components.

So what do these three measures have in common? They are very cheap, certainly compared to the cost of not detecting the error and they are very quick. These properties allow you to use them whenever you want, which means you might as well use them after every significant change, just in case you introduced an error.

In the complex environments we are currently working in there is no way we can get it right the first time all the time.
So next time you are saying “We just have to get this right first time in the future” or hear someone else say it, make a slight adjustment. “We have to go get this right the first time around when we try this for real”. And then go find a cheap and quick way of finding out if it works.

It has the same characteristics as a project in that once you get started it just keeps on going; There are other projects (or skiers) in your way, environments change and conditions might not be what you expected them to be.

Let’s see how it works out:

Walter the waterfall skier.

Walter is standing on top of the mountain, having just left the ski lift. During the requirements phase he and Adrian had decided that they were going to have lunch in a restaurant not too far from here at the bottom of the slope. The only thing between Walter and a bratwurst and beer is a couple hundred meters of completely empty ski slope.
Looking down he decides it is time to make the design for the decent. He takes out his notepad from his backpack, sits down and starts drawing the slope with the estimated distances and angles on it.
He can not really see all that well past one of the turns, but luckily he is well prepared and he takes out some of the photographs of the last part he found last night and continues drawing.
Happy that the drawing of the slope is accurate enough Walter starts drawing in his planned decent. This involves drawing in every turn his is going to make to make sure he slows down enough so that he stays in control, that he goes quickly enough to go over that bump and that he avoids the pillars in the middle of the slope.

Finally he is happy with the result, stands up, and puts on his skis and goes..

Adrian the agile boarder

A little bit earlier Adrian had realized he was due to meet Walter at the restaurant in 30 min. The restaurant was just over the hill and if he left now he would make it there in time.

On the ride in the ski lift Adrian takes a look at the map. The slope on the other side is a red slope, which means it would not be very easy, but certainly doable for Adrian. He also needs to make sure he does not miss one of the turns to the right to stay on the right slope.
Leaving the lift he walks to the crest of the slope and looks down. The whole slope is nice and empty and he fastens his board.

The decent

Together with the other 5 people from the lift Adrian starts his decent when suddenly out of nowhere a skier appears.
It is Walter who is intently studying the map he drew and only glances up every now and then to check his location. One of Adrian’s fellow skiers only managed to avoid a collision with Walter by letting himself fall and tumble down.
A little further Adrian notices an ice patch further down and adjusts his speed accordingly. Walter however is still studying his map and fails to see the ice and falls hard on his face.
Not discouraged by this however he sets up, checks his map and continues to the first junction.
When Adrian and Walter get there however they notice the slope is closed because of possible avalanches. Walter starts to curse, the last bit of the slope had taken him the better part of half an hour to plan properly. He finds a spot to sit, takes out his notepad and map again starts drawing again.
Adrian noticed nothing of this however. Remembering there was a second turn to take he just continued down when he noticed the closure of the first slope.

Adrian was on his second beer by the time Walter arrived, completely covered in snow. When asked what happened Walter explained that because of the change in requirements a redesign was needed. But because of the time pressure he did not have time to do a full redesign, so he reused his design for the other slope. They seemed very similar at first. It wasn’t until later he realized the pillars were spaced differently and he only barely managed to avoid one.

Does it sound like any project you have seen?

When I look back at my most successful debugging sessions there are three recurring themes:

• Triage
• Partitioning
• Assumptions

Triage

When something goes wrong, there are usually a lot of other issues that are either caused by that failure or are related. My first priority when faced with multiple issues is to figure out which issue is most important to solve right now. Those are usually either issues that prevent all flows, or the most important happy flow, from functioning properly, or likely root causes of multiple issues.
This focus is needed because when you start debugging you usually run into errors in log files, stacktraces on web pages etc. Whenever you see anything that indicates a problem you can now ask yourself if it is related to the problem you are trying to solve or not and prevent being distracted by unrelated issues.

Partitioning

The next step is isolating the issue. Difficult debugging problems usually arise when things disappear or there is an unexpected flow in the logic. What I usually do is create an overview of the entire application and its related systems either in my head or scribbled on a piece of paper. With this overview in place you can find system or module boundaries where you can usually quickly check whether the problem is present at that location and zoom in on that.
So first figure out what application the unexpected happens, then what module, class and method level. Do not guess what method it will be and verify that.

Assumptions

There are two famous sayings related to assumptions. “Assumptions are the mother of all f*ckups” and “When you assume things, you make an ass out of u and me” and those are very true. Especially when debugging you need to be extremely careful to not make any assumptions. When 2 requests in a log file look the same, copy and paste them and manually verify them line by line or better yet, run a diff on them. Do not only look whether the request is sent, but is it a valid request? Assumptions are most likely the reasons you are in the mess in the first place, the only way to get out of it is to be mindful of them.

Forcing myself to answer the question and write this blog post has forced me to really think about debugging and going from unconscious competent to conscious competent, which is always a good thing. Please leave a comment if you think I have missed something, or if you have any practices that have worked for you.

In the end we settled on “Incentivizing (is that a word?) parts of the chain instead of the whole”.

Let me explain.

A typical waterfall project has a few distinct phases that are always something along the lines of Business Analyses, Detailed Specification, Construction and Testing. They all depend on each other and nothing can start before the previous phase is finished. You can not start Constructing if you do not know what the Detailed Specifications are for example. Nothing really spectacular wrong with this yet. The problem arises when you consider that all phases are usually impacted and budgeted separately. You can not know how long Construction will take until the Specification guys are done.
But in the real world you have to manage your resources, so once your impact and estimates for the Specification phase are done, you have to make reservations for the Construction guys to actually code your project.

With that reservation in place, there is now a very big incentive to have that phase finished by that time, because the penalty for not finishing in time are that a whole development team is twiddling its thumbs. When it comes to making decisions that might delay the current phase, even though it might save hundred of hours later down the line are now surprisingly hard. A project manager’s first priority is delivering his particular part of the chain on time.
This is hard to spot though, because the next phase will look at the deliverables of the previous phase and if there are any defects or inefficiencies they will just plan and budget accordingly. This hides (most of) the real pain. Usually the only performance metrics that are used are if you delivered your part in time. Not if the overall project went smoothly.

With no focus on the total project costs, including maintenance, it is no surprise that a lot of these projects eventually run very late and get more and more expensive as they go along. So if you absolutely have to do waterfall, at least make sure that if ever a decision comes up about when to fix or improve something the answer can not be anything but: “Right now!“

All these environments have a couple things in common in no particular order.

• They have experienced and passionate teams and especially team leaders.
• There is a very strong focus on keeping things simple and getting results.
• The entire organization has an culture of excellence.
• An open culture.

I will discuss those points one by one:

Experienced teams and team leaders

Xebia has very strict recruiting requirements and as part of the recruitment procedure, besides the usual interview, we have assignments where candidates can (and must) show they have experience in the field. At my previous employer and my customer there were Gertjan van Oosten and Andre Kampert, easily some of the best team leads I have come across in my career. These people not only know the rules to writing good software, but also when to break them. All those people are passionate about creating quality working software.

Simplicity

But maybe the most important thing that happened in all the environments is a very strict focus on keeping things simple and getting results. However do not confuse simplicity with using simple code. The best example on how not to do it came from an application I had to quickly fix a couple of bugs in. The most complex construct in the entire application was a case statement with 4 cases and a default. However there were methods with 6 deep nested if statements. Some complexity, in this case in the form of polymorphism for example, would have gone a long way to make the application more simple.
The focus on results makes sure the application never strays too far away from working code. If we have something working now and we have to have something working in one week, we better make sure it keeps working in between as well.

Culture of excellence

This one is very visible within Xebia, but also in the other organizations. One of our core values in Quality without Compromise. While that is of course utopian, it does mean that if we ever make a compromise on quality, we have at least thought about it very carefully. It also means I do not have to defend each and every decision to project managers or line managers. If I can do something dirty in 2 hours and do it properly in a day, I can easily say it is going to take a day. We do things right or we do not do them at all.

Open culture

Software quality is never black and white. There is a great deal of grey. What these environments have is a culture in which it is ok to discuss quality. No single person ‘owns’ a single piece of code and the entire thing is a team effort. A discussion about the quality of the code that someone had written is perceived as a way to improve both the code and the person having written the code.

So to sum it all up, if you are having code quality issue you should contemplate throwing out all quality procedures, hire a passionate, experienced team lead, keep a focus on simple, working code, and give your team the assurance that while it is ok to make mistakes, it is not ok to take shortcuts and not to do things properly.