In this post (an extended version of which is available as a whitepaper), we will outline today’s Release Management challenges and discuss the need for Release Automation.

We’ll identify key considerations for successful solutions and highlight why “Zero-Maintenance” is a critical requirement for Release Automation that provides the scalability required in an agile landscape and enables the delivery of continuous business value.

(more…)

In this first post, we’ll focus on some of the changes and trends across the industry that have brought such increased business attention to the area of release, deployment and management of applications.

As companies focus on tuning their IT environment on rapid delivery of business value, more and more projects and initiatives within organisations are looking at the entire value chain of software production.

Whether under the banner of Devops, via integrated project teams or as part of the introduction of new development methodologies like Agile¹ or infrastructure technologies such as cloud², there is a growing awareness of the need for automated, reliable and flexible deployment procedures.

Whilst it’s clear that the generation of customer features takes place within the development, testing and QA teams, the business value inherent in these features is only unlocked once the application is actually running in a target environment, accessible to users. And it’s not just the final release to production that requires a deployment: every UAT, performance or integration test generally needs an application running in a “real” environment, not a developer’s local machine. One test, one deployment.

Given the well-known costs and adverse effects of faulty software in production, the ability to raise quality, usability and performance through a greatly increased number of testing cycles represents significant added value in itself.

One of the consequences of this increased attention is that the importance of build, release and deployment professionals as deliverers of business value is being recognized more and more. The heightened focus also means, though, that many companies are realizing that the effectivity, simplicity and overhead of their release and deployment processes lag far behind that of build and continuous integration.

As my role at XebiaLabs involves studying, automating and improving deployment processes across the industry and helping developing our vision for deployment into the future, the question of why deployment is so different from build in today’s enterprises was bound to raise its head sooner or later. The resulting discussions and reflections formed the basis for this series.

What’s in a word?

One of the challenges surrounding the discussion of deployment in relation to build, release, provisioning and other tasks in the ALM space is the – decreasing, thankfully – lack of a clear shared definition for deployment. Without wanting to promulgate this as the correct definition, for the context of this discussion I’d like to treat deployment as the process that

takes the components that make up a release (typically a specific version of an application) and getting them correctly set up in an infrastructure environment so that the release is accessible to (end) users

This would differentiate it from build and release in that it assumes that the application components have already been created, and from provisioning and other infrastructure tasks in that the target infrastructure is already assumed to be present.

On-demand virtual or cloud environments, or virtual appliances, put a bit of a different spin on the issue, and will be a topic for a subsequent blog.

Blast from the past

Taking the above as our working definition, the sobering picture is that, with very few exceptions, deployment now is pretty much at the stage where build was in the days of the make guru: a black box put together and operated by a specialist that somehow works. With luck, this precious resource is still employed and around to fix or extend things when required, but if he or she is out to lunch you’re simply out of luck.

To be fair, there are quite a few places where there is least tooling or automation in place that tries to map out the sequence of steps or actions required to carry out a deployment, to at least bring some visibility and traceability and shine some light on the ‘magic’. But this is still a long way away from the push-button experience that build is nowadays. Laboriously visualizing and walking through a process step-by-step is still a sign ultimately of a lack of trust; a truly mature process doesn’t display its internals any more, it “just works”. Much like build today.

The road to “just works”

Of course, build didn’t start out as a “just works” process either, so what actually were the steps that made this transition possible? Looking at the evolution of Java build tooling over the past decade or so, there have been at least three main developments:

1. Reusable commands
2. Models
3. Conventions++

We’ll dive into the details of these three concepts in the next blog…

A couple of weeks ago, I overheard a discussion between a couple of people involved in the music business. They concluded that the days of the megastars are officially over; they didn’t think there would ever be another artist rising to the same levels of stardom as Madonna, Elvis, Michael Jackson or the Beatles.

The Music Industry

The reason why these people from the music industry figured there’d never be anything as big as the artists mentioned, ever again, is because of the Internet. It just has never been as easy as it is today to share music. I can put something together in an afternoon, and have it available on Bandcamp or Facebook that same night. And if it’s any good, it will spread like wildfire.

But, hold on, it has not only become way easier to put something out there on the Internet. It also became much easier to pull something down from the Internet. I recently switched over to Spotify. Even though I have a convenient setup that allows me to listen to my (legally acquired) MP3 all over my place, I haven’t touched it since then. And before that, I tried Last.fm for a while and found a load of new artists. The collection of music I listen to has grown significantly, last year. (As an example, I found about the “Wifi?” song through Facebook. A song right from the heart for many of us, I’m sure.)

Back in the days, both the artist and the listener depended on the record labels to tie supply to demand. That basically acted as a big funnel. In went all artists, out came the ones that were most likely to generate a lot of money. For a record label, it made way more sense to have one world-famous artist, than having several targeting a smaller audience. Because of that, some artists got all the exposure (radio, TV) and others hardly any at all.

The Internet has changed that. It basically short circuited this relationship, and boosted the supply.

The Software Industry

Let’s assume for a while that there is some truth in all of this. I can imagine that marketing funding has contributed a lot to putting some artists in the spotlight, and keeping others out in the dark. That seems to be exactly what happened to Java as well; only this time it wasn’t one single vendor, but a lot of them; and the same goes for SQL. (IBM, Sun, Hewlett Packard, Sybase, Oracle, etc.)

That doesn’t mean Java is awful or SQL is a piece of crap. Au contraire. It just means that because of critical mass of these big corporations, Java got more visibility, and therefore more momentum than any of its competitors.

It’s interesting that Java may actually both have profited and ‘suffered’ from the Internet. Java’s appearance coincided with the rise of the Internet. The first people enthusiastic about Java liked it because of its Internet potential. (I remember my boss at Sun saying that, one day, his friend passed by and showed him Java tooth tumbling inside an Applet; that’s the day he applied for a job at Sun.)

These were probably also the people who started to talk about their ideas on how to use Java in new ways. Java dates from 1995. Sourceforge from 1999; it now provides a home for 45,186 Java projects. (And remember, they got some competition since then: Github, Java.net, Codehaus, Apache, etc.) At OOPSLA 2004, in Vancouver, all posters where about Java, with the exception of the ones that were not about programming at all. Sharing has been the basic attitude of the Java world, and that – I feel – contributed a lot to its popularity.

The same also goes for having other languages running on the Java VM. And in that sense Java (the language) also ‘suffered’ from the Internet. The first other language running on the Java VM might have been Ada (back in 1996), but now it’s hard to keep up with the numbers: Clojure, Scala, Jerlang, JRuby, Mirah, JavaScript, and I could go on for a while.

So, the genie is out of the bottle. People got used to polyglot programming, and no marketing budget will ever be able to make any more noise than the buzz coming from the community directly. Again, it’s not unlike the demise of the music industry. The Internet short circuited supply-demand both in the music industry as well as in the software industry.

And alternatives to Java on the Java VM are not the exception. The same goes for storage solutions. Back in the days, the only real option for storing your data used to be a SQL database. (In fact, in many cases, the only option used to be Oracle or DB2.) But since then, people started to scratch their head, wondering how on earth SQL solutions would solve their scalability problems, and came up with other, sometimes simpler ways to store their data. And then, they spread it through the Internet.

Conclusion

In summary, it seems our world is coming to a joint conclusion that Java and SQL are no longer the only options out there. Phew! That’s a breeze of fresh air! I therefore declare this year to be the year to celebrate the “No” movement: NoSQL (Not only SQL), NoJAVA (Not only Java) and – well – basically No[you name it] (Not only ‘you name it’). Back in the days, all we had was NoCHOICE. But now we have. Let’s take advantage of that.

The image that pops up in my head when I think about NoSQL and alternative languages on the Java VM (and the music industry) is one of a lava lamp. Down at the bottom, you see blobs of wax trying to cut loose to make their escape. Often they get pulled back, only to settle back at the bottom and wait for a new opportunity to make their way. It sort of reminds me of the big corporations and monumental industry standards preventing creative ideas to escape. But now, the old industry seems to loose its grip. The knot is slipping. Many bright ideas managed to cut themselves loose and have crossed the chasm. It’s unlikely that anything will be able to pull them back now. Welcome to 2011. Up, up, and away!

(And for that reason, it might also be good to mention the NoSQL NL meetup scheduled for January 24, the Dutch Scala Enthusiasts, and the Clojure meetup. If you want to learn more about your options, it might be good to drop by somewhere soon.)

Recently I published an article on testing in a Scrum team for the Eurostar 2010 newsletters. It’s about the mindset of an Agile tester. This blog post summarizes the core of that article.

To get there, together!

Good interaction is the key to good results. An agile mindset and working in an agile process facilitates a cooperative spirit: “To get there, together !”. To have that ‘Agile Mindset’ means to respect the balance that is presented by the Agile Manifesto and take all decisions you make ( big or small) in the spirit of these principles.

A good Scrum Team feels responsible for the delivery of the desired software on time and according to the agreed quality standards. From a testing perspective some modern test approaches like SmarTEST are designed for ‘agility’ in the testing process. Although SmarTEST has a strong focus on people and interaction, the big change for the tester may still not be fully perceived. Full adaptation of agile principles means that the agile tester is not responsible for testing! How about that?! The team as a whole is responsible for the collective result, which includes testing as one of the steps.

What does this mean for the role of the tester?

For a tester to be truly successful in an agile team he needs to change in four way. These four points describe the heart of the mindset of an agile tester.

1. The tester is responsible for delivering what the business needs (not for finding bugs).
2. The tester is focussed on delivering business value (not on just writing, executing and reporting tests)
3. The tester is a full-time member of the team (not just at the start and end of a sprint)
4. The tester is no longer a tester, he’s a team member with a talent for testing

Of course this is only the top of the agile testers mindset.

How about independent testing ?

Independent testing in an agile environment doesn’t differ much from the more traditional approach. Some of the testing is moved to the team’s responsibility. Things like integration testing and user acceptance testing allow the team to learn. They need to learn from executing these tasks themselves so that they can understand their customers and their environment better.
Choose independent testing for cross team testing, regression testing and non functional testing.

Conclusion
The agile tester feels responsible for the whole, is fulltime committed to the result, is focussed on delivering business value and acts like a true member of the team.

Independent testing is still desired. It’s important to leave integration testing and user acceptance testing to the responsibilities of the Team so they can keep on learning from what they deliver.

Parting thoughts

Not every tester will be able to change his mindset and become team member first and tester second. These testers can provide most value when working in an independent test team.
Finding those really great testers that are able to become part of the Scrum Team is a challenge. Having them in the teams is a critical factor the success of your teams.

Dimensional planning
Dimensional planning (story mapping) can help with this. Usually interfaces and prototyping are put in the first slices. The features that can be addressed later on should be in later slices.
Most of the times the different dimensions of the team work from separate backlogs, but are aligned in the dimensional planning. Although they work from a different backlog they can discuss the problems together. Often this leads to better solutions due to the different views on the problem. In some cases this leads to backlog items for the different dimensions.

Freeze
Somewhere in time the electronics development is frozen in order to start the hardware production on time. From this moment on only small changes are possible. For instance to lower the radiation levels or things that need to be done in order to get an CE certification. Meanwhile the other dimension still continues to develop software for the device. Sometimes solving some hardware issues. Depending of the requirements of the CE certification the main software features needs to be frozen too. Fortunately software updates can be applied in a late stadia.

Conclusion
In conclusion a cross dimensional team can work in an agile way. Using dimensional planning brings focus to the cross dimensional teams. Important is to freeze the interfaces between the hardware and software in an early stage. This will reduce the dependencies of the dimensions. Problems that occur during the prototyping can be solved by working closely together in an early stage.

Part 1:
Automated code reviews with Checkstyle, Part 1

Part 2:
Automated code reviews with Checkstyle, Part 2

This article series attempts to bridge the gap of code review with applying automated Checkstyle checks in a complete and proactive way. First goal is to make the task of custom Checkstyle rules creation so simple so that any enterprise IT team could create new custom rules suiting to their project (IT standards) needs.

Second goal is to apply these rules in PROCTIVE fashion. Instead of waiting the build to fail or waiting for rule violation reports and working on them in a reactive way, the idea is to apply these checks proactively with Checkstyle Eclipse plugin or applying them at SVN level itself. Irrespective of which IDE you are using, if your code contains some of the high severity violations, you will not be able to commit the code in SVN. You will see the same kind errors and location on SVN console as you see with Eclipse plugin. This is achieved using SVN pre-commit hooks.

The article describes how the combination of Flex and Grails leads to a highly productive platform for writing JEE applications. It discusses the problems one faces when integrating Flex as client technology and Grails as server technology and details solutions for each of these problems.

The article can be used as a tutorial for writing simple client-server applications with Grails and Flex.

Agile 2008

At Agile2008 in Toronto Jeff Sutherland and myself presented our article outlining how to achieve hyperproductivity in distributed Scrum when working in an offshore situation. InfoQ recorded our presentation and will publish it online in November as the end of a series of Agile2008 talks.

Download article    Download presentation

Also see this InfoQ article

Agile and Offshoring, oil and water?

If you are reading the Xebia blog chances are that you are already familiar with the benefits of Agile development. Practicing Agile (in our case Scrum combined with XP) delivers hyperproductivity combined with very high quality. The promise of offshoring in the modern IT industry is also clear: more available talent, scaling up and down without local layoffs or knowledge drain, and of course cost reduction. Together they make a killer combo!

However, Agile and offshoring seem like oil and water, they don’t seem to mix. How to get a focus on individuals and interactions when your people are distributed across the globe? What is the secret sauce to use to get it running smoothly?

Distributed Outsourcing Styles

There are different models that you can consider when distributing Scrum. The most primitive model that is often applied is to create fully separate Scrum teams that work on unrelated work. Slightly more advanced is the usage of isolated Scrum teams that connect regularly via a distributed Scrum of Scrums. This is the model that the Scrum Alliance currently advocates.

Much more effective is a Fully Distributed Scrum model where each team consists of members on both locations. In this model there is no division of teams based on geography, teams span the globe! At Xebia we came to this model simply by reasoning from the Agile manifesto. The main issue in offshoring is communication. When creating a geographical division between teams you create another communication barrier that you need to take. People tend to take the road of the least resistance and end up with all sorts of processes and tools.

When creating a distributed team, with members on both locations, the team members have no choice but to resolve their communication problems.

Once these are resolved you can operate on a basis of equality, providing your offshore team members with responsible intelligent work, thus giving them the job that they deserve as 21st century knowledge workers.

Shared mindspace

Setting the team up for success has a lot of it has to do with getting into a shared mindspace across locations. This means developing a shared ownership of the code and architecture, a shared domain context and a shared value system and culture. In order to set this up you need to think carefully about traveling, about building personal relationships, about finding the commonalities that Agile developers the world over share. Setting this up and cherishing it is vital to a smooth operation

The presentation

Presenting these concepts that I am so passionate about at a conference like Agile2008 is a fantastic experience. Distributed Agile is a topic that is obviously very much in the spotlight. The audience had a lot of questions and it was great to share ideas after the presentation.

When talking to the people that attended our presentation I noticed that a lot of people are in the process of training their offshore partner to become more Agile. Usually first through a Scrum of Scrums model and eventually aiming for a Fully Distributed Scrum. Setting it up in this way, by changing your suppliers habits, is a very slow and costly process.

Inspired by the success of projects like this one we are building distributed teams now for partners, with a match on Agile and quality. If you have an Agile business where you are looking to include the benefits of offshoring then we can help by hooking you up to Xebia India. Have a look at Xebia Global Services for more information.

Next step is JAOO in Aarhus, Denmark where Jeff and I will present about this topic.

Guido Schoonheim, CTO

However more frequently than not getting an SCA violation is the start of the story. If you start associating the question “Why’ with every SCA violation found then the real reasons start unfolding.

This is similar to the way we resolve impediments on an Scrum project. The impediments rarely represent the isolated incidences of inefficiency. Rather, most of the times they are a part of a larger problem. The way to work out an impediment is fix it so that the team can work effectively and then to look at the root cause which caused the impediment so that the main cause can be fixed. This is called “Bottom-up process re-engineering.”

Similarly the way to work out an SCA violation is to remove it so that the code looks clean and good and then to hunt for the real cause.

I got first hand insight into this when I was doing an audit for a large system which is being used online by millions of people. While doing the analysis my co-auditor suggested that let us look for hot spots which are highlighted by the SCA tools. Then, let us dive into the code at those places to find out bigger issues. At that time I was skeptical if that approach would work however now I am a firm believer of his philosophy.

The SCA tools do a great job of pointing out the hot spots. You can either patch the hot spots to make the SCA tool happy or you could look at the bigger issues so that the root cause of the problem is eliminated.

I would take some examples from my audit to make it more clear. Let me warn you that these problems look too simple for any deeper analysis but believe me once you do the analysis you do manage to uncover some unexpected thought provoking findings. And of course these are just a few examples for illustration, you could apply this to ideally every violation that you get.

1.Size violation: We found that there were servlets in the application which were more than 3000 lines, having methods more than 800 lines each. A quick fix to this might have been to reduce the method size by splitting the methods and moving some methods out of the class to a helper class so that the class size and method size violations are taken care of.

However, when we looked deeper to answer the question “Why do the servlets have so much lines and such big methods?” we realized that all of the business logic was present in the servlets. There was a bigger violation of Single Responsibility Principle where all the logic was present in the single class. The presentation logic, business logic and data access logic were all clubbed together thus leading to fragile design which is hard to change without breaking. There was no layering and no separation of concerns.

The design of the system was a culprit here and probably none of the developers or the architect on the project felt so.

2.Methods with large number of parameters: A quick way to solve this and make the SCA tool happy would be to introduce an object and populate the object with the required parameters.

However, when we took a deep dive we realized that the system did not have any abstraction. There were no domain objects in the system either. When data had to be passed between method calls, all of that would be passed as primitive data, mostly strings. There was no focus on domain driven design. The methods were overloaded with every extra parameter that would be required in a different situation which suggested that the system was open for modification and closed for extension. For any small change a new method had to be written. This violated the Open Close Principle.

Again design was the culprit here.

3.Large Cyclomatic complexity and NPath: We were baffled to see CC numbers > 50 for some methods and NPath figures of > 1.7 million.

When we dived deep we found that the methods in question had a lot of nested if..else logic. So far so good, but what is the reason for the nested logic? We found that the method contained all the dispatch logic for the request. If the request has these attributes do this else do that and so on. This also explained why the system had low code coverage by unit tests. A unit test to test 1.7 million possible paths would take weeks if not months to write. Clearly nobody had thought about introducing one of the many readily available MVC frameworks. When we questioned the team on why they did not use a MVC framework we were told that they did not feel the need in the beginning because the system was too small. Once it started growing they never had the time to refactor and introduce a framework.

This could have several causes ranging from lack of communication between the client and the vendor about realistic time lines, refactoring not being a part of the standard development process, lack of interest of the team in project, lack of interest in achieving the right quality and focusing on just getting the work done.

4.Missing braces and wrong variable naming: This is probably one of those violations which is immediately reported by a SCA tool and is easy to correct.

However, when we started looking at it deeply we found that certain modules of the application reported huge violations as compared to others. When we compared two modules with highest number of violations we found that both of them were using very different naming strategies. Further investigation revealed that there were no coding standards which were used on the project. Every module and each developer was free to use a nomenclature which went well with his taste. There was no conceptual integrity and no uniformity in which the entire software was built.

Improper software development process along with lack of coordinated communication between the teams were the obvious culprits in this case.

5.Large number of unused imports, variables and methods: This is again something which is readily reported and is easy to fix.

However, when we questioned the developers about the reason to have unused code, we were told that they never removed the unused code lest they should need it some day. It would save them time in the future.
Obviously nobody was thinking about the time lost when we had to scroll through tons of unused code and probably they were never educated on the principle of YAGNI.

Again improper development process and lack of developer education were the culprits here.

You would concur with me that all these violations were very small and could have easily been fixed without resolving to deeper analysis. However, if you try to dig deeper then most of them would point to bigger issues which need a more focussed approach to fix. We saw that relatively small problems reported by the SCA tools could be attributed to bigger issues like improper design, sub standard development process, lack of communication and lack of proper education towards effective software development for the development team.

Hence though doing a root cause analysis for small violations might sound like an overkill but a lot of times the small violation is just the tip of the iceberg!

Next time when you are resolving the SCA tool reported violation just pause for a moment and ask “Why do I have this violation in the first place?”

It is another great day, sun shining bright, traffic snarls continuing on roads, the team getting ready for another productive agile day with droopy faces! Droopy faces , why ???

Well because they have to get over the stand-up ritual first before they can get started with some real work.

But is the stand up a ritual??? Aren’t stand-ups supposed to be exciting and energizing???

If this a common question disturbing you for a while then it is time to stand up for the stand-up. The details that follow are meant for any team practicing any Agile methodology like Scrum and who has started thinking that stand-ups are no more than an empty ritual which has to be pushed out at the start of the day.

If you are a stand up team member a.k.a pig then you must commit to the list below to make sure that you do your part for a successful stand-up. If you are not a stand up team member a.k.a chicken then the list below would let you know what not to do and what to expect.

Say aloud, I am committed to the success of the project and I make the following commitments

1. I commit to make sure that the stand up is a lively huddle and not a formal status meeting.
2. I commit never to say that “I do not remember what I did yesterday” nor “I am not sure what I would do today”, I shall come prepared with my points for the stand up on a post it or in my notebook after looking at the Sprint Backlog or the story board.
3. I commit to focus on the sprint backlog during stand up, it is easy to wander in all directions during the stand up but I commit to bring the team back to the backlog.
4. I commit to raise impediments and blocking issues, ensure that the impediments are added to the impediment list and are taken care of in a timely fashion, this might require my taking initiative to point the impediments to the scrum master.
5. I commit that I would attentively listen to what everyone has to say and not just listen to the person standing next to me just before my turn.
6. I commit that on hearing an issue to which I know the solution I would not resort to problem solving, I would state that I know of a possible solution which we can discuss after the stand up.
7. I commit that I shall not address my points to the scrum master but would talk to the whole team, since I am not reporting my status but I am stating facts and commitments to my team.
8. I commit that I would not use the stand-up for socializing even though I or a team member came back from an exotic vacation.
9. I commit that I would be loud and clear with my information so that everyone can listen and high energy is maintained.
10. I commit that keeping the length of meeting in mind, my information in a stand up would be to the point but still informative enough, I would not just say that I am working on issue number x but would also give a brief description of what it is about.
11. I commit that if I am in a distributed stand-up across continents, I would make sure that the tools and machines used for stand-up are ready before the stand-up starts.
12. I commit that if I cannot come to a stand-up I would make sure that my input information is sent or announced via an email or a telephone call with a colleague.
13. I commit that I would not delay a stand-up because it is not just my time but the entire teams time.
14. I commit to be involved only if I am committed to the project else I would just observe and not interrupt / disrupt the proceedings.
15. Last but not the least I commit that whenever I see a deviation from these commitments from anybody I would openly debate that and initiate corrective action.

Conclusion: The daily stand-up meeting is meant to be an opportunity for the team members to communicate to the team on their progress and obstacles. It is an opportunity to share commitment and set direction for the team. Most of all it is a medium to build a closely knit team. Let us commit to the power of this simple opportunity and stand up for stand-ups.