Xebia Software Development builds Posterous clone in a day!

Iwein Fuld

Sharing knowledge is one of our core values and as lot's of research confirms knowledge transfer is best done between peers. We have a great knowledge sharing platform at Xebia through bi-weekly evening sessions, where we do some experimental coding and some presentations. Once in a while we take it to the max and organise a tech rally. One of those happened last Friday and it was a total blast. I'll give you some of the highlights. More detailed posts on the technical details will follow and I'll update the list below as they do:

We started out with a picture and a maven skeleton and starting code for some tricky parts. Graciously prepared by Albert Sikkema.
This image was the only visual aid in the presentation of the case, causing much skepticism and questions about lack of preparation. These early moments of uncertainty were casually ignored during development.

From there we split up in 4 teams and chose common sense and utter anarchy as a methodology.

Javascript flamewars

Prior to the tech rally we had a good old flamewar between javascript zealots and heretics. One of the best moments was when Gerbrand, who isn't a big fan of javascript turned up as the first volunteer to work on the javascript needed for the front end. The end result was cuddly like a baby bear, I'm really curious to hear if Gerbrand has been converted to the dark arts. I couldn't find the JSUnit testcases, but that's a topic for another day.

Spring REST

To render JSON to a client it is tempting to try out a REST framework. To keep things simple we went for Spring REST and the result was as expected. There were a few hick-ups related to setting the content type or facing the 406, but luckily no fairies got killed. The details of this little trick have been described already on the Spring blog, but if you need a working sample you can also have a look at our web project.

Reading email with Spring Integration

Through no influence of yours truly (boy scouts honor) we ended up using Spring Integration to read mails from a gmail account. I avoided joining that team, but I had the pleasure of joining in one particularly interesting discussion on preferred granularity of pipes and filters. The answer I came up with could fill another blog, but the question is more interesting anyway, so I'll just give you that.

When should we use a synchronous channel and when is it better to just call the method directly?

In other words: how big should endpoints become before you split them up? Think on that and let me know if you have figured it out.

Dealing with javax.mail, even through a framework is a daunting exercise. Andrew Snare (who recently joined us) found a really tricky one, Albert Sikkema found a couple more during the preparations. Because of this, intelligent folder management was not one of our achievements during the day. We did however read the mail successfully and store it in our database. Good enough for a first prototype.

Storing the posts in MongoDB

At Xebia we're involved in a number of initiatives around NoSQL, most notably the NoSQL Meetups organized by Wilfred. Obviously we cannot store data in a traditional database for fun anymore. Even though the MongoDB part of the application might be the coolest bit on paper the work there was minimal. MongoDB is just too easy to use.
Being in the MongoDB team was pretty cool nonetheless, because being the main integration point of a 20 developer 4 team commit frenzy is the closest thing to being in a warzone I ever hope to experience.

The coolest thing was that at the end, the application actually worked. Nobody got hurt and we could show an end to end demo of the whole thing at five in the afternoon.

This rally was exactly the kind of experience I was looking for when I joined Xebia, so I was totally thrilled. The only thing left to do is find a customer that allows us to really go crazy like that. Any takers?

Comments (3)

  1. Peter Veentjer - Reply

    September 21, 2010 at 12:32 am

    Hi Iwein,

    one of the things that concerns me when people are using NoSQL databases is at how well they understand the relaxed guarantees most NoSQL databases provide (especially in the area of consistency and atomicity). Depending on the implementation used you don't always get distributed failure atomicity, so a change could partially enter the system (the older versions of Cassandra are suffering from this problem.. haven't checked out the newest releases).

    Another problem I see is with consistency; consistency is a very hard topic and I think I can say I know the bare basics since I deal with it on a daily basis with STM. Dealing with all these relaxed consistency models that form the heart of most NoSQL databases sounds like a gift from heaven from a scalability point of view, but what it essentially means is that the burden is now placed on the developer and my experience is that most developers don't have the time or understanding to get it right.

    I find relaxed consistency models very hard to reason about, so how should most developers do it. I see a similar problem when multi-threading is used; a lot of developers write code that looks alright in their eyes, but if you look closer, you can see all kinds of problems ranging from traditional problems like: liveness, deadlocks and data-races to all kinds of less traditional problems like instruction reorderings or visibility problems.

    My bet is that most applications can be designed quickly and scale reasonable well (much better than a central databases because that just doesn't scale), but under contention or when the system or parts of the system crash, that the integrity of the data could be violated.

    Peter Veentjer
    Multiverse: Software Transactional Memory for Java

  2. Iwein Fuld - Reply

    September 21, 2010 at 9:19 am

    Consistency is indeed a tricky topic and using MongoDB doesn't simplify the discussion at all. In this particular case we had one writing process, that was reading from mail. Consistency across the storage and email client is not feasable, at least not in this scope. So the only problem left is to go for consistency inside the storage itself.

    All BlogItems were stored as a single object in a single write operation to MongoDB, so the consistency there is fine. The readers will see the posts some time after they have been stored. Since there are no concurrent edits possible (yet) the discussion on consistency is pretty simple in this particular case.

    If we would go for an editable solution, all edits would happen through PUT operations. The risk of overwriting undoing changes because of dirty reads in concurrent scenarios is one that I'd be willing to take. To avoid it I'd rather merge eagerly on the client (like sync.in does) then to postpone the merge and fail on the server.

    That said, it would be awesome to put multiverse into this app, it would have been fun to tackle that during the rally, but somehow it didn't come up. I'm looking forward to your pull request though 😉

  3. [...] This post was mentioned on Twitter by Iwein Fuld and Rossen Stoyanchev, evanderkoogh. evanderkoogh said: RT @iweinfuld: Summary of the Xebia techrally last friday http://bit.ly/techrally #springintegration #mongo #springrest #nojsp [...]

Add a Comment