The Top 10 SOA Pitfalls countdown hit #1 last week with Rik de Groot's post on "Ignoring culture when introducing SOA", time for a wrap-up.

Putting all pitfalls together in one simple 10 item list quickly reveals a grouping of types pitfalls. Number #1 and #2 are both related to organizational aspect. If the culture, mindset and attitude are not right, these are typically the pitfalls that a SOA endeavor may run in to. The next group covers the items #3 till #7, these are all related to architectural/design skills. And the last group, numbers #8 till #10, relates to implementation issues (although proper design could help to prevent these pitfalls from manifesting themselves).

The complete Top 10 SOA pitfalls list is:

Implementation pitfalls

• #10 - Not Invented Here Syndrome
• #9 - Versioning
• #8 - Security

Architectural/design pitfalls

• #7 - Incorrect Granularity of Services
• #6 - SOA does not solve complexity automatically
• #5 - Big Design Upfront
• #4 - Incorrectly applied Canonical Data Model
• #3 - Missing skills

Organizational pitfalls:

• #2 - Unclear ownership/Project based funding
• #1 - Ignoring culture when introducing SOA

It was great fun to compile this list with the four of us and it also helped us to get a sharper view on why SOA projects or programs are challenging and hard to complete succesfully. Hopefully you've learned a thing or two and are able to prevent or at least recognize the pitfalls in your day-to-day work.

Thanks for having us,
Rik de Groot, Viktor Grgic, Vincent Partington, and Gero Vermaas

Working abroad has been a wish of mine for some time now. Xebia offers me the opportunity to live and work in India. Through this blog series I will keep you informed of the progress and challenges of this project.

It has been quite a while since I wrote about our plans to go and work in India. A lot of people have asked me whether I had already returned so soon, assuming that I had already left! The truth is that our plans are still very much alive, but the past period has not been very exciting, so there was nothing much to blog about. Currently we are in the final stages of drafting the employment contracts and I would like to inform you about the structure we have in mind for this.

Contract triangle

Currently I have an employment contract with Xebia in the Netherlands. For me it would be easiest if this contract would remain in use and I would simply be posted in India under a deputation contract. Since I would be getting salary in India for a longer period, I would have to pay taxes to the Indian government. This would mean that the financial department in the Netherlands, responsible for paying my salary, would have to get in contact with the Indian governmental organization for taxes.

It seems more fit to sign an employment contract with Xebia in India. That way my salary would be paid in the same country as my taxes. The problem with signing a new contract for me is that the rights that I have built having a contract with Xebia over the last few years would have been forfeited. Also the Dutch laws on employment give a lot more protection to the employee than the Indian laws. We were looking for a way to retain these rights and protective rules, whilst signing a new contract.

After some consulting we decided upon keeping the original employment contract with Xebia in the Netherlands and putting this contract in a non-active state through an “unpaid leave” construction. This is a known construction in the Dutch juridical system, that practically suspends the execution of an employment contract for a certain period of time. The employment relation remains intact. During this period an employment contract between Xebia in India and me will be active, under which salary will be payable. To govern the relation between these two contracts, which are standard contracts, and to make some additional arrangements, an additional contract will be signed between Xebia in the Netherlands and me.

Additional arrangements

The new contract between Xebia in the Netherlands and me will contain the following arrangements:

• The duration of our stay in India is set to 2 years. Extension is negotiable. We will be able to end the stay earlier, after discussing this with Xebia.
• Additional cost are for Xebia. These cost are for example medical cost of vaccination, insurance, moving and travelling cost.
• Pension.

Pension is under Dutch law an arrangement for postponed salary. It is practically a special type of saving. Taxes over this salary are payable when the pension is paid, thus after the employee ends his working live. As such it is mostly a fiscal arrangement: A part of the salary is saved and over this part no taxes are paid at that time. In addition, the employer also pays a part of the monthly pension amount.

The Indian fiscal system doesn’t have such an arrangement, as far as I know. The amount of money that would be paid for pension while working in India would be taxable. That would mean that taxes would be paid twice over this amount: Once to the Indian government (when the salary is paid) and once to the Dutch government (when the pension is paid). This could eventually cost a lot of money.

Under the Dutch fiscal law, an extra payment to a pension-fund can also be made when you have not been able to save for pension for a period of time due to working abroad. As additional arrangement for pension, Xebia will pay the amount that it would pay to my pension-fund over this period as a one-time bonus when the unpaid-leave period ends. I can then put this bonus into my pension-fund and use the fiscal advantages at that time.

Financial picture

The cost of my current contract to Xebia in the Netherlands will be the same as the cost of my future contract with Xebia in India. This means that I will get a lot of salary in India, compared to other employees. Most of my Dutch colleagues are quick in spotting this. They are definitely right (and I don’t understand why they don’t try to do the same… ;-)), but I do feel a bit offended by this slight suggestion of greediness and feel the need to explain.

For starters, life in India is definitely cheaper than live in India. Nonetheless, life in India for a foreigner that stays for a relatively short period, will definitely be more expensive than life for a native Indian. You have to do some investments for a rather short period, like furniture and cooking equipment. Those kind of things are all not very expensive, but you’ll be needing them for a rather short time, which makes them expensive in comparison. Some services will also be more expensive for foreigners and you need to be very careful not to be taken advantage of when doing any kind of financial
transaction, from paying your taxi to doing your grocery or renting a car. That’s just normal live for a ‘rich westerner’ in India.

The other explanation is that I offer Xebia the opportunity to have a native Dutch employee in India for free! I could have just stayed in the Netherlands and gotten the same salary, without taking this risk of leaving everything behind and going to India. Of course I don’t see it this way, and neither does my girlfriend. We think of this as a great opportunity and adventure. From the perspective of Xebia though, it makes a lot of sense.

Actually, I don’t mind the gloomy, smiling faces of the people that make these kind of remarks. They are right and I don’t mind the lower cost of living in India. We’ll try to feed some of it back into the community by doing some social, voluntary work and maybe helping out one good initiative or another.

The next step in this project will be to get visa from the Indian Embassy. It appears that this is a bit more difficult than we had anticipated. I’ll also be trying to get a better idea of my tasks in the Indian team and to get a grasp on the expectations of management in the Netherlands. This will also be the main purpose of a visit to India that is planned for August. The big move will be around October 1th.

Recently held workshop on Agile Awareness in Xebia India was a great learning experience for all  the newbies into Agile way of working. The delegates who made up for the event  were from a wide range of portfolios starting from Team Lead, Project Manager to Software Developer. So, this made the whole stage quite interesting.

It started from "The Problems that we face generally in Software Development". And the answers that we had were quite unanimous.

• Unrealistic Deadlines
• Poor Estimation
• Surplus Documentation
• Inadequate Testing

and the most stressed point by all of us was "Changing Requirements".

After expressing our major 'Blockers' it seems  we were ready to take a plunge  into the Agile World. The buzz word here is "Iterative Incremental". This simply means that Iteratively you keep on making progress by adding something to the structure, however small the increments may be.This is a lot different from what "Traditional" methods ask us to do. Simply put,in traditional methods,the peak comes only when you are at "All-at-once-Delivery Stage" which is when your project completes.

Nicely Stated by Craig Larman :"Usually a partial system grows incrementally with new features,iteration by iteration,in other words,incremental development."

 The dictionary defines Agile as "Nimbleness or Flexibility". Agile in context of software development paradigm is all about being Adaptive than Predictive or Rigid. To change is law of nature. So we should not be afraid of changes. Rather we should plan how to act according to the change. The ones who crib over the changes required are rigid in thier approach. And Agile calls for being flexible in accepting any change made or to be made.

Most of the delegates have already experienced the pain of changing requirements, thus we were excited to see how Agile would come to our rescue in this area

This was explained quite well .It goes like:

In Software development it is impossible to have "Frozen Requirements". And this is because we simply cannot freeze time. So we generally have bulk of requirements from the client  which can be modified , updated or scrapped by him in later stages of time. . Out of these  we first focus onto the "High Risk" and "High value" requirements. High Risks need to be tackled early in the development phase and High Value means more profits to the business. Apart from this we also have MoSCoW techniques to prioritize our requirements. For those who are thinking what is MoSCoW city to do with Agile and software..Take a look:

• MoSCoW

  MoSCoW represents a way of prioritizing items. In the context of DSDM the MoSCoW technique is used to prioritize requirements. It is an acronym that stands for:

MUST have this requirement to meet the business needs.

SHOULD have this requirement if at all possible, but the project success does not rely on this.

COULD have this requirement if it does not affect the fitness of business needs of the project.

WOULD have this requirement at later date if there is some time left (or in the future development of the system).

A brief mention of The Agile Manifesto was also made which is followed by  the four points:

1. Working software over Comprehensive Documents.. (Simply Said is: All that matters is what Works!!)
2. Customer Collaboration over Contract Negotiation..(Collaborates definitely sounds positive than Negotiates)
3. Responding to change(Definately with a plan!)
4. Individuals and interactions over processes and tools.

Short Iterations, TDD, Potentially Shippable (or Working Software) product at the end of each iteration, individuals and interactions over processes---Is all what we need to do.

To summarize Agile Way of working ill put it like this:

PRIORTIZE --> DO SOMETHING (ITERATION) --> GET FEEDBACK -->IMPROVE N GET GOING.

The two things that personally strike me are:

• Getting Feedback
• Failing Early.

I was amazed to see how "Real" is this way of development. Feedback was something we ve always taken/given. Be it shopping with friends or Areas of Improvements Column in magazines or newspapers.We all want to improve.

Another thing is to fail early. It's always appreciable if you fail early ,improve and then win your task. And as a developer this is what we should look for too.This Calls for the use of Test Driven Environment.

After opening up the umbrella of Agile in the" Why Agile?" session we were now braced to take a peek at various technologies comprising it.

Our first destination was SCRUM Techniques. Had heard the word "SCRUM" before which had to do with the Rugby team. It is when the team huddles together to discuss their plan of action for the next few minutes of the game. Was figuring out as to how it fits in here?  To this we were told that to work Agile way the teams should be self organised and cross functional. When we say cross functional it means that a team should be adept and complete in every sense. This can fuel a very common debate of Generalists over Specialists that is persisting in the industry since long. I came across a very useful blog on this topic which happened to be written by one of the speakers of this event.  The other key aspect is self discipline in the team. And I think this is what makes this simple concept a difficult one to follow. We don't like to be disciplined ..that's the general word..

After this we were introduced to various roles existing in Scrum. To give a short trip ..Product Backlog,Scrum Master,Sprint and Sprint backlog,Committed Backlog, Burndown chart. Another very important point that caught my attention was "Responsibilities are never given but taken". I agree.Responsibility "given" is more of an instruction. As someone has aptly said : "You cannot instruct to innovate". Later in the session we talked more about the iteration details.An iteration is a short period of time say 2-4 weeks in which a team goes for a complete development cycle.For traditional developers we can put as a whole waterfall within an iteration. In Agile starting from taking requirements into Sprint backlog to Sprint retrospectives -all becomes part of an iteration.In this whole discusssion it was little difficult to think in terms of velocity and complexity points though. Probably cause brain was busy receiving hunger signals!!

After breaking for snacks,it was eXtremeProgramming..second flavour of Agile. This name was little tough to comprehend than the earlier ones. It started on the note that everything done should have some business value and should make sense. So it was clear that its from the developer's perspective. Out of the four levers of a project namely:

1. Time
2. Scope
3. Quality
4. Cost

agile team plays with the lever of functionality i.e.Scope.  Its very common to see that about 60% of the functionalities of a product are never used and about 20% are intensively used.

Again coming back to real life association of agile..I believe that not everybody is a born genious. But two ordinary minds working together can definately come up with extra ordinary results. And simply put this is Pair Programming in Agile. No doubt it is a win-win situation if both of you pair well. But i was stuck with a question..What if you and your pair dont gel well? It could be disastrous to project's productivity. We then discussed number of intelligent ways to overcome such situations within the team. In this course of discussion one of the speakers came up with a really interesting Solution to this called "CC and BCC". Those who are wondering..CC is communication over coffee and BCC is bullshit communication over coffee. Indeed, a lot can happen over coffee

Here I would like to quote one of the speaker ..You need to have a lot of "Courage" to work Agile way. I guess that's absolutely true. When working agile way you need to shed all your inhibitions . Come out with any or all of your problems . Discuss with the team and move ahead. How many of us are really comfortable saying "I'm stuck with this problem , Dunno what to do?"Not many of us can really tell our weakness to others. This in particular calls for lot of "Courage"as agile encourages full interaction among the team.

Till the end of this session i was happy as i could now make this point clear that XP talks at developer level whereas SCRUM talks at project management level..

As we say in software industry there is nothing like theory sessions.. So here also a hands on session was awaiting. It was real fun to now imply the "simple and realistic rules of Agile".We went through two iterations..where we were given user story . We had to plan and assign complexity points to each user story. After this planning we had to commit to those stories which we thought were achievable in stipulated amount of time. And it was not points that made a team win but how much you learnt from  the mistakes.. We had a sprint retrospective after each iteration. This was the real learning of the day. At the end of this hands on we were now really comfortable with complexity points and velocity concept..

In the end to summarize i'll say that Agile way of development is not a revolution but evolution. It's realising your roots ,infact, coming back to your roots. Taking feedback , working in baby steps and improving, failing early is all we have been doing in our early days but then had kept them in oblivion. 

Agile is not about revolutionizing software development but  breaking the ice of rigid mindsets!! Its time that we break free and appreciate the beauty of Agile way of development. I congratulate the speakers for their efforts which definitely ignited these sparks in most of us.

Last week Viktor Grgic explained Unclear ownership / Project based funding. This week we’ll continue with #1 - Ignoring culture when introducing SOA.

SOA is an approach. The culture aspect of introducing a SOA is important, but it seems that companies want to invest in tools and not in people. In order of making this SOA to work they force their employees into this new way of thinking/acting. Often this leads to resistance which undermines the SOA goals. In this part we will look into ignoring culture when introducing SOA.

Culture
First of all: what is a culture? An organizational culture compromises the attitudes, experiences, beliefs and values of an organization. Hofstede identified five dimensions in his study :

• Small vs Large Power distance; degree of equality/inequality between people.
• Uncertainty avoidance; the level of acceptance for uncertainty and risks.
• Individualism vs collectivism; the contrast between individual or collective achievement and interpersonal relationships.
• Masculinity vs femininity; is your organization based on male or female values.
• Long vs short term orientation; contrast between relation values orientation.

When introducing a SOA some of these dimensions are likely to change in order to improve the performance of an organization. But wait a minute, a cultural change? It was just an architectural style! Culture is certainly relevant for SOA. For instance, when you look at ownership the first dimension can change. The ownership shifts from IT to the business. When you look at the second dimension, a SOA introduces an uncertainty, because people are confronted with a new way of thinking/acting. In the third dimension the way of working together (business and IT) changes.

Ivory tower
The way SOA is introduced in companies varies, but often SOA introduction is technology driven. From an ivory tower this new way of thinking is figured out and dumped into the organization (see also missing skills). Sometimes with a few practical guidelines, which are often very abstract. The guidelines are misunderstood by the business and IT people which makes it more complex. The main problem here is that the SOA vision is not shared between business, IT and CxOs, (and ivory tower). You can document guidelines, but this does not imply that involved people share the same ideas.

Awareness
In companies the adequate awareness is not filtered down to an enterprise level. Adoption of SOA might become a roadblock. The awareness starts with the concept of SOA. What is a SOA? In most cases people think they know what a SOA is, but have a different vision on the topic. Even people who do know what a SOA is, have different views on SOA. Therefore it is important to have the same vision at enterprise level.

Resistance
Although SOA should not be used to change a culture, the introduction will probably change some cultural dimensions. Resistance to these changes needs to be handled with care. The success of the SOA depends on the acceptance of the SOA, reducing/removing of resistance is crucial to increase acceptance

The best way to introduce a SOA is to follow the following steps:

1. Make the SOA vision a shared vision. Only a shared vision could lead to a successful SOA implementation. Without this first step a SOA is doomed.
2. Define a strategy of how to change. What is your strategy of changing to a SOA. Included cultural aspects and awareness.
3. Define clear tasks, roles and responsibilities.

When using these steps the acceptance of a SOA will increase.

The cultural aspect is often forgotten when introducing a SOA. People that introduce a SOA, often change a culture without knowing. When you are not aware of this, this might lead to an adoption roadblock. Be careful with a large cultural change. Start small, think big.

Next week Vincent Partington and Gero Vermaas will wrap up the SOA pitfall series.

Saturday, June 14, 2008: Xebia India organized an Agile Awareness Workshop that aimed at helping people very new to the Agile world of software development to understand the concept, appreciate the difference and know the jargon.

We plan to keep the momentum going, by following this up with:
1. Certified Scrum Master Training by Pete Deemer: June 25, 26
2. Agile Toolkit - June 27, 28
3. Seminar with The Poppendiecks (Organized by ASCII, sponsored by Xebia) - July 06.

Presenting "token of thanks" to the speakers

Last week Viktor Grgic explained the Missing skills en this week we’ll continue with #2 - Unclear ownership / Project based funding

In the world of standalone applications, there is typically a clear sponsorship and ownership of an application. There is also a single project with one project manager. The systems could be small or big, but the pattern remains the same. Funding is based on a business case and can be easily defended.
In SOA world, the story is different. There are the usual projects, each having their own objectives and often reluctant to work on generic services or enterprise components. If the ownership and funding for these components and aspects are unclear then chances are high that nothing happens on enterprise level or that it's not according to enterprise architecture or nobody feels responsible when things on enterprise level go wrong (e.g. security).
Several projects working together is not a bad situation, but there should also be a SOA steering committee and SOA competence center well funded and supported by company board.

The problems related to the unclear ownership / project based funding pitfall are:

Many projects with own interests
There is usually an Enterprise Architect or someone else expecting separate projects to deliver reusable services and deliver according to the enterprise architecture. The projects are business-driven, but often projects only target one aspect of the business strategy. Exposing the services to others and making them reusable conflicts with direct business interests. It requires additional effort for a project to generalize services and expose them to others; as apposed to just using the functionality of the services in the project itself only. Especially waterfall projects have this problem, because this reusability aspect is quite difficult to plan in advance.

Single large SOA project
Other companies define a single large SOA project which delivers "SOA" more or less as a product. This is strange, because SOA is an architecture style and not a product. Defining a business case for SOA is difficult. The usual consequence of an unclear business case is that SOA itself becomes the goal. The real goal is achievement of business requirements like faster time-to-market or cost reduction by better information and IT resources reusability.

Everything generic
In companies where several projects are delivering business products in SOA fashion everybody is talking about generic services and many projects claim to build these. In practice, the projects are building these services for themselves and don't really gather requirements from other projects or departments. Again, why should they? It costs significant effort / money and creates project risks.

Centralized services
It is a good practice to centralize services, which are not directly bound to one particular domain or department. These are often technical services. For example, security (authentication, authorization, audit log), message routing, reference data. But, what about transformation services specific to one business domain? In SOA projects, there is a tendency to centralize as much as possible. However centralization could cause more troubles. As an example, one problem is that it becomes very difficult to define how  responsibilities are divided. A shared responsibility mostly results in no responsibility. The main question is who is paying for development and maintenance.

The root cause of these problems can be found in unclear ownership and project based funding. It can get very messy if this is not organized well. The following are possible solutions:

SOA competency center
With any SOA initiative, there should always be something like a SOA competency center. Eric Roch gives an explaination on how to organize such competency center. This is a team of people from different area's (technical integration specialists, developers, architects, business consultants) which are pro-actively solving the mentioned problems. This team has independent funding and has a short term delivery of results in short iterations. This team does not only deliver underlying infrastructure and middleware, but also generic, reusable, domain-independent services. Domain-specific services will still be delivered by normal projects. In addition this SOA competency center can also support projects in variety of other ways (making services reusable, middleware integration).

Create market within the company
The projects need to invest in Enterprise aspects. The investment can be covered by having other departments/projects pay for it when services are needed. This is usually a single payment for requirements gathering and development effort.

Steering committee
The participants of the steering committee are representatives from departments. They discuss and make the decisions about enterprise subjects. The budget is determined by company management. The steering committee is also the sponsor and budget-holder for the services delivered by the SOA competence center.

Guerilla Tactics (from CIBIT-DNV) if you are an Enterprise Architect
If the projects / departments are reluctant to cooperate, you have always guerilla tactics:

• Create a gang of architects from all these projects and make them do the dirty job. Project managers often don't really understand what architects do anyway. In this ways enterprise aspects are funded by the projects secretly.
• Threat the projects with external legislation, security aspects, and anything else that you can think of.
• Steal the enterprise subjects / components (core services and entities, logging, security) and hold them hostage. The projects will have to work with you and deliver these with their own budget.

Having clear ownership and funding for enterprise components and aspects like security, logging, message routing, message transformation, generic services, ESB, and so on solves not only the problems during development, but especially in production when operational aspects become essential.

Next week Rik de Groot will take us to #1...

Last tuesday I attended the “RubyEnRails2008 dag”, which is the Dutch RubyAndRails conference. It was the third time this conference was organized and this year it was located at the Hoge school in Amsterdam. The line up was really impressive. A number of the Ruby Rockstars gave presentations and at the end of the day David Heinemeier Hansson joined from Chicago for a Q&A session

EaRing

The day started with a talk from Zed A. Shaw. He is the creator of Mongrel and became really famous with his rant against the Rails community. His presentation was very entertaining but not so much about Ruby than about how great a programmer he is (which he is btw.). The project he presented was the Meta-Literal Machine which makes learning ASM fun... It was written in 2 weeks and has support for some great features like: fully dynamic functions, access to .so files, dynamic constants(!) and it has even better error messages than Ruby interpreters. He used a Fibonacci number calculation program to show how fast the Meta-Literal Machine was. Calculating 96 Fibonacci numbers 5.000.000 times took on his laptop about 1.2 sec. There was some ranting against the idea to scale up application speed by parallelization. The arguments he used were quite solid and based on an extensive analysis with Amdahl’s law. So, to make programs run faster, don’t bet on parallelization but write faster programs...

The worst Rails code you’ve ever seen

This talk, by Obie Fernandez, was about bad coding practices in Rails projects. Obie Fernandez is a well known figure in the Ruby community and is the author of the book ‘The Rails Way’. In his talk he explained that worst (from the talk title) doesn’t mean stupid, but means programming a Rails application not in the Rails way. He gave some pointers on how to code the Rails way. One of the most important things to do is to learn Ruby. An other excellent advice he gave was that it is very useful to pair program with more experienced programmers. And never forget good OO principles when writing Rails code.

Phusion Passenger™ (aka mod_rails)

A better way of deploying Rails applications on apache was presented by Ninh Bui & Hongli Lai. The apache module for running Rails apps was very outdated, not any more! The guys from Phusion created a great product for deploying Rails apps on Apache using Phusion Passenger and their Ruby Enterprise Edition. They announced version 2 of Passenger which includes many improvements like: fair load balancing (as opposite to a round robin strategy), parallel file upload (freeing up the Rails instance to handle new incoming requests) and a big reduction of the VM size. The architecture allows for running not only Rails, but also Merb, Camping, ... and even Django apps using there Rack and WSGI support. The benchmarks they presented were very impressive. Comparing different deployment strategies (not JRuby on Glassfish), the Apache plus Passenger setup wins hands down with respect to speed and memory consumption. I would have liked to see a deployment setup with JRuby on Glassfish in the benchmark though.

JRuby

A talk not to miss was given by Charles Olivier Nutter about running Ruby on the Java VM. The JRuby project is production ready for some time now, and on average apps run even faster on the JVM than when running on the standard Ruby Interpreter (MSI). He mentioned that Java for some people was surrounded with negativity due to emotional arguments (I don’t like the languages) and sometimes the arrogant attitude of the Java vendors (“the answer is Java. What was the question?”). He continued by praising the JVM and showed some really impressive demos of ruby programs running on the JVM and leveraging the very rich collection of Java libraries (e.g. processing.org/ and www.the-shoebox.org/apps/44).
He then continued on showing how to run Rails apps on the JVM. The ease of deployment is really impressive. One can use any Java app server to run a Rails app as a WAR file build with the Warbler tool. Besides that, there is even an easier way provided by the guys from Glassfish using the Glassfish gem:
> gem install glassfish
> cd $your_rails_project_dir
> glassfish_rails
And its done!

A number of companies already use JRuby in there production environment: Oracle, Sun, Thoughtworks (Mingle), ...

Q&A with David Heinemeier Hansson

David Heinemeier Hansson is the creator of Ruby on Rails. There was a live video connection with him in Chicago giving us the possibility to ask questions. A lot of the questions were more about the business side of 37signals (his company) than actually on Rails and its future, but it was kinda nice to see him “live” I give you some random notes I wrote up during this Q&A:

• Rails is not equal to mainstream. The mainstream should bend to Rails.
• No 5 year plan: Goal numbers are bull...
• Flame wars are not leading to much
• Localization should not be part of Rails (everybody has its own idea about what localization means)
• 37signals has 10TB of data on Amazon S3

Scalability?

• the twitter rant is very loud, lots of noise
• twitter might not be a good fit for rails (not really a traditional web app)
• Yellow Pages performs very well on Rails

In the end a very fun and inspiring day at RubyEnRails 2008!

Last week Gero Vermaas explained the Incorrectly Applied CDM en this week we’ll continue with #3 - Missing skills

Just like any other paradigm, a level of new knowledge and experience is required. Unfortunately, SOA requires lots of new knowledge and experience. It requires a different way of thinking of more or less everyone involved. People are used to closed environments on both organisational and technical level; largely well protected from influences and unwanted dependencies with the outside world. It's all in their area of influence which makes achieving short term results relatively easy. I'm referring to silo applications where the world is complicated enough. From their view, SOA makes things even more complex. There should be awareness that there is lack of knowledge, experience and attitude and something should be done about this first. There is no real solution, except for the obvious one: educate everyone involved. Also, agile methodologies have proven to be effective in building up knowledge and experience.

Companies we come across introduce SOA in different ways:

• External consultant / architect advises the company to follow this new paradigm. If she is wise enough, she will first focus on the business case for SOA and make people understand why the company needs this architecture style. At the end, this makes things less confusing for people who don't understand SOA. At the same time, making an understandable business case for SOA is a daunting task. But very much worth the effort. At the same time, companies tend to approach SOA in their business cases as a thing which directly delivers value. They don't realize that SOA is just an architectural style and not a thing. Creating business case for SOA means making the match between the real business values and advantages of this architectural style.
• Product vendor has an ESB (Enterprise Service Bus), which seems to match with wanted features, which seems to be backed by real business needs. In the end, everybody is playing with the toy. Playing becomes difficult over time, because knowledge development is often driven by the tool. Non-IT people tends to feel left out and see SOA as something technical. Strangely enough, even they seem to be happy with the big toy without the understanding how the toy solves their problems. Another proof we all have something childish in ourselves.
• A group of people from within the company read the books, attend courses, conferences and so on and somehow realise SOA will solve many of their problems. They get together, start brainstorming and soon realize that this thing called SOA is huge and that there are many area's "where no man has gone before". According to them, if you want to do it well, you should spend a lot of time and effort in thinking and defining it. They act like researchers but without real plan covering the outline of this research, principles, results, criteria when the research is finished and so on. Another proof of childish behaviour. Children love to create something big, are satisfied if they learn something a long the way and be completely equanimous if the whole thing is not used or replaced with something else.

Let me be clear that there are many ways to introduce SOA and that it very much depends on the concrete situation. Even the toy-based approach is in some situations the best possible approach. But, each of them suffers from a lack of knowledge and skills in many area's:

Business Architecture
The term itself is quite new for many organisations and becomes important in SOA world. Creating business architecture in SOA means crossing the departmental boundaries, integrated business process modelling, creation of canonical data models, definition of business services (which are later translated into concrete services), defining different kind of security architecture, gathering business requirements and concerns of many stakeholders, and thinking in SOA design patterns.

Software Requirements Engineering
The toolbox of an requirements engineer is often limited to Use Cases. This methodology is not very suitable in SOA environment because it covers a very limited area. Quality (deprecated term: non-functional) requirements become more important because the complexity of a SOA architecture has larger impact on performance, availability, application management.

Project management
As a project manager, your project has budget and scope. It gets really fuzzy when people start requesting to build generic services and take care of their concerns. Your budget holder could also be reluctant to pay extra for something like this. Especially when you tell her what this means when the system is in production mode. A project manager should aim for two goals: usual delivery of the business product and "readiness" for SOA. These goals could conflict and project manager should be able to manage these conflicts. We see in practice that project managers simply ignore the SOA goals if the conflicts occur.

Software Architects and Developers
They require skills on EAI/SOA patterns, webservices, messaging, ESB's, BPM engines. And particularly difficult to find are specialists with experience in XSLT, Xquery, ebXML and WS-* stack.

Testing
Testers will also need to understand SOA and the exact implications of it on testing. Testing services independently of each other is relatively easy. But what about a complete business process, supply chain testing, transaction handling over multiple services when a transaction must be rolled back, asynchronous messaging, automated testing and so on.

IT management
An IT manager who does not really understand SOA sees it as introduction of some large and expensive infrastructure components which magically solves the integration problems. In large organisations there are several IT managers, one for each department. It proves almost impossible to introduce collective middleware. The problem is the assigned responsibility and ownership. One of the solutions is creation of a the separate Middleware Competence Center or Shared Service Center. This often is a complete new department. If you're lucky, the IT manager and her staff have SOA skills.

Collaborative attitude
Separate departments can have a protective attitude. Although everybody in the company will tell you that collaboration is good for everyone, they will first protect interests of the department before they address overall company interests. The root cause is the unclear advantages of collaboration. People tend to see disadvantages, e.g. additional dependencies.

If a company lacks knowledge in only one these areas, the whole SOA undertaking is in danger. The symptoms are long discussions, sometimes for a year or longer and the only results are large documents and/or some unused middleware infrastructure. The discussions focus on who should do what and how are responsibilities divided. The root problem is that these people have many years of experience, but don't realize they are lacking the skills needed for SOA and they should be doing something about this first.
Hiring experienced consultants to the job only solves the problem if everybody actively participates in the learning process and consultants have coaching role.
Lack of skills can be partially compensated by focusing on short term results and learning from mistakes, which is an Agile way of thinking.

Next week Viktor Grgic will take us to #2...

A thing I was playing with today was many-to-many relationships in Grails to create a Tag Cloud. To create a Tag Cloud, I must have a set of key/value pairs, each with a label and a value of the label, which could look like this:

['Java': 5, 'Grails': 16, 'Groovy': 12]

But to query this, I need to query a many to many relationship and produce the above result. This blog will describe how to do this with HQL, Criteria and the HibernateCriteriaBuilder.

Domain

My current domain looks like this:

(This picture is unfortunately missing a Tag attribute in the Snippet class).

As you can see, a Snippet can have multiple Tags, while a Tag can belong to multiple Snippets. How can we query this?

Options

We have three options in this case

• HQL
• Hibernate Criteria
• Hibernate Criteria using the HibernateCriteriaBuilder

HQL
The HQL version is not so hard (and neither are the other ones), but we have to start somewhere, so let's start with this one.

To start at the end: the query looks like this:

select tag.text, count(snippet.id)
from Tag as tag
inner join tag.snippets as snippet
group by tag.text


To test this code, I find it easiest to start up the Grails console:
grails console
and run the HQL there. You can do so by obtaining the SessionFactory from the ApplicationContext, which is done by accessing the implicit 'ctx' variable in the console. The complete code looks like this:

 
def sessionFactory = ctx.sessionFactory
def session = sessionFactory.getCurrentSession()
def query = session.createQuery("select tag.text, count(snippet.id) from Tag as tag inner join tag.snippets as snippet group by tag.text")
def results = query.list()
 

Which, in my case, returns this (I have two snippets, one with 3 tags and one with 1 tag, which is a duplicate tag of the first snippet):

[["groovy", 1], ["io", 2], ["testing", 1]]

Hibernate Criteria
Since Grails is built on Groovy/Java, Hibernate and Spring, it's easy to use that power while coming up with a solution to your problem. If you write Java code all day, and writing Hibernate Criteria is something you do daily, this won't be a problem at all. Just copy and paste your existing Java code into Grails, and you'll produce something like this:

 
def sessionFactory = ctx.sessionFactory
def session = sessionFactory.getCurrentSession()
List results = session.createCriteria(Tag.class)
  .setProjection ( Projections.projectionList()
     .add (Projections.groupProperty("text"))
     .add (Projections.rowCount() ) )
  .createCriteria("snippets")
  .list()
 

Hibernate Criteria using the HibernateCriteriaBuilder
If you, however feel that the above is lacking some style, you might want to check out the HibernateCriteriaBuilder, which is a DSL for building Hibernate Criteria.

When using the DSL, we first need to retrieve the HibernateCriteriaBuilder, which can be obtained from any domain class. After that, you'll have the full power of Hibernate contained in an easy, readable DSL.

 
def c = Tag.createCriteria()
def results = c.list {
   projections {
      groupProperty("text")
      rowCount()
   }
   snippets { }
}
 

Conclusion

As you can see, Grails provides a lot of options to query your domain. Whether you prefer HQL, the safe and sound Criteria, or the new HibernateCriteriaBuilder, the choice is up to you. By leveraging the power of existing frameworks, Grails provides an easy and powerful way to quickly develop an application with a minimal learning curve!

Consider the following senario: we want to display some data in a table like manner, and we want it to update when we click on a link or button. We do not want to do a complete page refresh, we want it in an ajax way. Also we would like the modify the css for each cell.

In wicket you can use a ListView iterate over a List of Objects and display them in a table like manner.
This blog describes how you can update ListView data and modify the css for each cell.

So how do we start?

Well first we need to create a ListView. Lets call it view.
But before we do that we need a List of objects we want to display. So lets create it.
I use a List of TestObject's.

As you can see below, this isn't very hard.

 
private ListView view;
private List<TestObject> objects;
 
objects = Arrays.asList(new TestObject[] {
    new TestObject("one", true),
    new TestObject("two", false),
    new TestObject("three", true)});
 
view = new ListView("view", objects) {
    @Override
    protected void populateItem(ListItem item) {
        Label label = new Label("label", new PropertyModel(item.getModel(), "name"));
        label.setOutputMarkupId(true);
        item.add(label);
    }
};
 

Next we will create an IndicatingAjaxFallbackLink which is basically a AjaxLink that shows an indicator.

 
private IndicatingAjaxFallbackLink link;
 
link = new IndicatingAjaxFallbackLink("link") {
    @Override
    public void onClick(AjaxRequestTarget target) {
        objects = Arrays.asList(new TestObject[] {
            new TestObject("four", false),
            new TestObject("five", true),
            new TestObject("six", false)});
        view.setList(objects);
        target.addChildren(view, Label.class);
    }
};
 

Thats all. Now see that we add a new List of objects to the ListView. To make sure the ListView is updated we add

target.addChildren(view, Label.class);

instead of

target.addComponent(view);

. As a ListView is a repeater, we cannot updated it, but we can update components in a ListView.

So now we can update a ListView, but how do we change the css for each field?

We add an AttributeModifier. In the populateItem method show above we just add the following

 
label.add(new AttributeModifier("class", true, new PropertyModel(item.getModel(), "odd") {
    @Override
    public Object getObject() {
        return ((Boolean) super.getObject()) ? "odd" : "even";
    }
}));
 

The getObject method is overridden, because the method getOdd() on the TestObject returns a boolean. So instead of returning true or false, it now returns "odd" or "even" which are css class styles.

That's it.

I have attached the example project here so you can look into the code yourself.