As an example we’ll analyze tweets from the Twitter Streaming logs and calculate the top 5 hashtags per day which are associated with positive sentiment signals (smileys).

You can imagine how this can be expand this to simple sentiment analysis on your (potential) customer feedback.

Gather the data from Twitter streaming API
The JSON log lines from the Twitter Streaming API look like these:

    {
      "created_at": "Sat Sep 10 22:23:38 +0000 2011",
      "id_str": "112652479837110273",
      "text": "@twitter meets @seepicturely at #tcdisrupt cc.@boscomonkey @episod http://t.co/6J2EgYM",
      ...
      "user": {
        "name": "Eoin McMillan ",
        "screen_name": "imeoin",
        ...
      }
      ...
    }

For now we only care about the “created_at” and “text” attributes. See detailed information on all available attributes at https://dev.twitter.com/docs/platform-objects/tweets

Import raw data into Hadoop & Hive
Now from the Hive console import the logs into a Hive table.

create table raw_tweets (json string);
load data local inpath 'sample.json' into table raw_tweets;

With Hadoop it is a best practice to always preserve the raw source data. It is pretty common that you detect obscure parse errors or missing information days later, keeping the source information allows you to correct this without much hassle.

Parse JSON tweets and extract relevant information
From this raw data, we parse and extract the actual information that we care about. Using the get_json_object function we can access the “text” and “created_at” attributes using an XPath like query on the JSON object. The timestamp needs to be converted into Unix epoch format for later formatting.

create table tweets as
    select get_json_object(json, "$.text") as text,
           unix_timestamp(get_json_object(json, "$.created_at"),
                "EEE MMM d HH:mm:ss Z yyyy") as ts_created
    from raw_tweets;

Text parsing, extract hashtags and sentiment identification
Now that we got the source data in a format we can deal with (text, timestamp), it is time to identify sentiment information. For this case, we’ll compose a regular expression that matches some positive smileys: and . Feel free to expand this to your taste. Also we split the text on whitespaces and identify terms which look like a #hashtag. These matching hashtags we emit together with the date in YYYY-MM-DD format allowing us to do daily aggregations afterwards.

create table positive_hashtags_per_day as
    select from_unixtime(ts_created, 'yyyy-MM-dd') as dt,
        lower(hashtag) as hashtag from tweets
            lateral view explode(split(text, ' ')) b as hashtag
        where ts_created is not null
            and hashtag rlike "^#[a-zA-Z0-9]+$"
            and text rlike "^.*[\;:]-?\\).*$";

Aggregate occurrences per day
Aggregation is straight forward, just a matter of counting the occurrences per day. Now if you want to do multiple aggregations (per week, per month, etc), you might want to move the date string creation to this step.

create table count_positive_hashtags_per_day as
    select dt, hashtag, count(*) as cnt from positive_hashtags_per_day
        group by dt, hashtag;

Limit top 5 results per day using external reducer
Finally we only want the top 5 results per day. Now this is a bit tricky in Hive, as this requires a user defined function or streaming reducer call. We do the latter using a little piece of Python code that only returns the first 5 results per keyword. Because the input to the reducer call is already secondary sorted on the count in descending order, this will return the top 5 results; just how we want it.

add file topN.py;
create table top5_positive_hashtags_per_day as
    reduce dt, hashtag, cnt
    using 'topN.py 5' as dt, hashtag, cnt
    from
    (select dt, hashtag, cnt from count_positive_hashtags_per_day
        distribute by dt sort by dt, cnt desc) cnts;

The Python reduce code looks like this. Due to the way the Hadoop Streaming API works, you need to detect key boundaries yourself.

#!/usr/bin/env python
# Reducer that returns the top N results per keyword
import sys

maxN = int(sys.argv[1])
last_key = None
count = 0
for line in sys.stdin:
    (key, value) = line.strip().split("\t", 1)
    if key != last_key:
        count = 0
        last_key = key;
    if count < maxN:
        print "%s\t%s" % (key, value)
    count += 1

Results
The results of this exercise on my tiny sample set (can’t redistribute the source according to the Twitter TOS):

hive> select * from top5_positive_hashtags_per_day;
OK
2011-02-25 #ff 8
2011-02-25 #happy 3
2011-02-25 #followfriday 2
2011-02-25 #teamzeeti 2
2011-02-25 #baensv 1

2011-02-27 #1 2
2011-02-27 #12 1
2011-02-27 #27f 1
2011-02-27 #dvdmeb 1
2011-02-27 #fail 1
Time taken: 4.231 seconds


Follow Fridays make people happy… who would have thought

Download the full code and Follow @mids106

Background:
The Story Tree is loosely based on the KANO model, which looks like this:

The model shows that a product can be built up by a mix of features falling in one of the below categories below.

Features in category 1 are not really missed when absent. But if they are executed well they can delight your customer. These could very well be your products unique selling point (USP) or unique buying reason (UBR).

Category 2 features are the linear performers. These are like hotel beds. If you get a small bed with a crap matrass, you will probably be dissatisfied. The better the bed, the more satisfied you would be. A unique mix of these properties could provide the base for your products positioning relative to your competitors.

Category 3 features are threshold features you would expect from a product in a certain category. If this is left out, the theory indicates you would be less satisfied, because you would expect it from a product in this product-category.

Category 4 features leave you indifferent. I translate these in my exercise as stuff you have to do that nobody will notice, but is still necessary. I call this sprint 0 stuff.

Now here’s what the story tree looks like (sorry for the Dutch terms in here, they will be explained later on in the post):

Silent Story Tree®

The Silent Story Tree shows us different levels of features loosely linked to the KANO model described above.

The roots of the tree symbolize the foundation of our product. The framework it’s built upon, the environments and all that kind of stuff. Invisible to the end-user, but never the less valuable to start sprinting.

Next comes the stem of the tree. This represents the threshold features, as a tree is not considered a tree without a stem of course.

The low hanging fruit represents linear performance features. These are mostly clearer features than the delighters in the category of sustainable USP/UBR’s and easier to reach targets.

Most of the time the top of a rich and well leafed tree can’t be seen from the ground up, and that’s why we also have a top level in the story tree for ideas that are still to unclear to put anywhere else at this stage.

When the Story Tree is filled with stories you can actually get a group feel of what the ideas are around the product it’s and proposition. You can group features and reprioritize them on the story map according to your implementation strategy (bottom up, or validating your value (USP/UBR) first).

So here’s the process:

Step 0: prepare and explain

Grab 3 flip-over sheets and stick them together using tape. Draw the tree outline and the specific areas in which the stories and feature groups will be placed. Fun up the tree by drawing in apples, birds or even a squirrel?! Next hang up the tree drawing somewhere close by the place where you will be creating your initial storymap.
Next, grab another flip-sheet and draw a box with some apples in it. Write “disputable fruit” on the side of the box. This is your parking lot for shuffled stories from “step 3” below.
Before you start this exercise explain to the team the aim of the exercise, and further proceedings. The most important thing to explain is that the order picked now, can be changed as things proceed. The whole thing is aimed to get a common understanding of the starting point and get ideas into the open. The group will have to understand that things will change along the way.

Step 1: create stories

Get some stickies and a wall to create your stoymap. Only create the fully functional axis together and let the group write as many stories as possible. For now, use the standard template; as a…. I want…. so I can…. Note the user function the story belongs to in the top right-hand corner of the sticky, so you can place these back afterwards (not necessarily the column it came from).

I will not go into this step any further in this blog post as the sorting of stories is the subject here.

When the group is finished hang the stories underneath the functions. You should have a great wall decoration by know that’s screaming to be made sensible.

Step 2: moving the stories to the right spot

Now here starts the silent part and it works just like silent prioritization. Line up your team in a row. Let every team-member pick a story of the map and place this in a section of the tree. Continue without talking until all stories are gone from the map. You cannot pass at this stage.
Finish by examining the result and any anomalies (chances are high the roots section will be empty if you haven’t spotted this before). Compliment the group and explain the next round. Let them have five minutes to silently prepare for it, as there could be quite a lot of information hanging in the tree. The photo below shows the Story Tree in action. See how relaxed these people are ?

Step 3: shuffling stories around

Repeat step 2, but now people can take a card they disagree with positioning wise from the tree and place it somewhere else. If people can’t think of anything they want to shuffle they are allowed to pass. If an item shuffles more than once, remove it from the tree and hang it on the disputable fruit list to discuss in the next step. At the point you feel energy is going down and more people pass, you should stop and move on.

Step 4: the disputable fruit box

The exercise is silent. Since things change along the way anyway and the exact features to start still need to be decided, that’s OK. Keep it that way. It’s waste to spend a lot of time on this stage.
Only discuss the fruit in the disputable fruit box as a group. You should timebox this part to max. 15 minutes.

Go try it!

The whole process from step 2 to 4 should not take more than an hour to do. The remaining work is to place the stories back on the story map in the right order depending on the implementation strategy and general ideas on winning in today’s marketplace. But that’s an entirely different story.

This exercise works like a charm in groups up to 6-7 people and makes use of the collective body of knowledge in this group. My final tip would therefor be to create a divers group to increase creativity.

Have fun growing your tree! Should you wish to add to this idea, just leave your comment below.

But lately you have noticed  that simple new features take more and more time to implement. That despite of your test coverage, bugs have crept in your production code which can take days to solve. And that solving the highest prioritized issues is sometimes delayed because specific knowledge is located at a single person, who can only handle one task at a time.

What’s going on?
Automated testing and metrics are certainly useful, but they don’t catch everything. For instance, it is perfectly possible to have meaningless unit tests which cover 100% of your code.
You could write code with splendid metrics but which is impossible to understand by your colleagues, because method- or variable names are badly chosen.
Maybe you have written code while a colleague has already written something similar, thus missing an opportunity for reuse.
And last but not least, while there are many tools for measuring the quality of Java code, the choice of tools for non-Java code is much more limited; Think of JSF or JSP pages or new languages for which tools have not had enough time yet to develop.

Solution: Code reviews!
Code is not only written for computers, but also for humans. So why not let humans take their part in the code quality process?
Your colleague can see immediately if your code is difficult to understand or not. He can check coding standards in those areas where tools are still missing. And there’s more:

• He could spot bugs in your code
• You might have misinterpreted or missed a functional requirement
• Your colleague might know of a cool third party library you could have used

On top of that, by letting colleagues read each other’s code, knowledge of the code base will be spread thoughout the team.
And by discussing suggested improvements of the reviewed code, general coding knowledge is spread through the team as well.

Tips for how to do it
In my experience, there are a few things which work well when integrating code reviews into your software development process:

First, arrange a brain storm session for your whole team, and agree on a set of review criteria.
By agreeing as a team on a set of review criteria, you prevent discussions later on.
Think of such things as:

• descriptive variable- and method names
• meaningful unit tests
• conformance to gui standards
• non functional requirements such as performance or security

When assigning reviews, make sure they are assigned to a random colleague. After all you have agreed on a set of standards to review on, so a junior colleague might just as well review your code as a senior. And, different people have different points of view, so it’s good to have many different colleagues review your code.

Look for integration with your bug tracking tool. Code reviews become much easier if commits can be traced back from a bug report or feature request.
For instance, Jira has a nice integration with Crucible which enables you to assign reviews to teammembers straight from a Jira issue.

Alternatives
The major disadvantage of code reviews is that they take extra time. It is my personal opinion that this investment in time will pay itself back with interest.
However, if you find it impossible to convince your manager or colleagues to allocate the extra time, then doing selective reviews might be a good alternative; for instance, you could subscribe to each other’s commit e-mails and check those.
But, if time is not an issue, an alternative that goes a even bit further might be pair programming.

A product manufacturer joined us in our idea. In 4 months we changed their product development department, as well as all connected departments. We discovered that Scrum as a framework is compatible with non-IT environments. So we discovered our way of working within the framework. While looking at the results, we asked ourselves: what are the key values in this company, that made this change successful?

At first glance, the Agile manifesto does not give an answer. Though very concrete, The Agile manifesto has some hidden values in it. We made a first step to discover them.

• Feedback: gain feedback in order to learn and make your product better. Feedback is needed when you want to embrace change, is the output of interaction and de purpose of customer collaboration. Feedback is the foundation of Agile and for example Lean startups. Lean startups is not about building a product as soon as possible, it is about gaining feedback as soon as possible so you can valid your business case.
  Gaining feedback is possible in various ways. Architects (the ones who are designing houses) gain feedback by a drawing, the butcher gives you a piece so you can taste the product you want to buy. Car manufacturers gives demonstrations to the car press and sometimes even involve them in the development of the car. Be creative in gaining feedback and the way you demonstrate your product.
• Iterative: Don’t try to be perfect at once. Or even try to solve the whole puzzle at once. At the start of building a new product, you do not know what the outcome will be, nor your client. So try to solve the puzzle while gaining feedback with the product you have. Only with the feedback of your customer, you are able to be most valuable for your client – and your company. Iterations help with gaining feedback, and force you to split things up. While timeboxing yourself, you enforce ourself to be concrete and work on real products instead of paper.
  Small sized products are required for working iterative. But their is no objection to work in even smaller sizes, such as user stories. If you cannot create user stories (sometimes teams just couldn’t) you can always try to add a new feature or make an improvement to your product within 2-4 weeks. You can even do an experiment in order to validate your (technical) assumptions or make a proof-of-concept (what’s in a word).
• Valuable: Everything you do, should have value for the customer (customer value), the company (business value) or required by law (regulatory value). This is written in order. Ask your customer what he needs the most, he is the only one who can tell you!
  Don’t accept the product value that is stated initially without asking why. Too often there are hidden assumptions in it (the customer likes this, so it has lots of value) that are not true. Asking five times why helps to find the real reason. Try to prove your assumptions, while using Lean startup techniques or by asking the customer. Quantitative arguments of a business case have extra value to the qualitative arguments.
• Cooperation: You can cooperate at two levels: within and outside our team. Cooperation within your team creates synergy and helps people to get the best out of them. Cooperation outside your team helps to get the customer heard. And oh, he wants to be heard. Really. Interaction -and even more: collaboration- is not only valid within your team. It is even more valuable outside your team.
  You can cooperate at various ways. Prefer the one that gives you the most layers of interaction: verbal, non-verbal, written, realtime are examples of such layers. Standing together at a whiteboard includes all layers of interaction and is the most valuable one.

We discovered that these values are applicable for all sorts of environments, no matter what kind of products are made. That is not a surprise: most agilists agree that the Agile philosophy is a major step forward in the history of business. Agile is able to not only change IT, but also the world outside IT. With the Agile manifesto and these values in our toolkit, we think lots of non-IT companies are able to become Agile too.

Easiest and quickest way is to just add it on the view’s layer:

-(void)loadView {
    CGRect frame = [[UIScreen mainScreen] applicationFrame];

    self.view = [[UIView alloc] initWithFrame:frame];
    self.view.autoresizingMask = UIViewAutoresizingFlexibleHeight | UIViewAutoresizingFlexibleWidth;
    self.view.backgroundColor = [UIColor blackColor];

    UIView *glowView = [[UIView alloc] initWithFrame:CGRectMake(frame.size.width /2 -10, frame.size.height /2 -30, 20, 60)];
    glowView.autoresizingMask = UIViewAutoresizingFlexibleBottomMargin |
                                    UIViewAutoresizingFlexibleTopMargin |
                                    UIViewAutoresizingFlexibleLeftMargin |
                                    UIViewAutoresizingFlexibleRightMargin;

    glowView.backgroundColor = [UIColor whiteColor];

    [self.view addSubview:glowView];

    //Setup the shadow on the view's CALayer.
    CALayer *viewLayer = glowView.layer;
    viewLayer.shadowOffset = CGSizeMake(0, 0);
    viewLayer.shadowColor = [[UIColor yellowColor] CGColor];
    viewLayer.shadowPath = [UIBezierPath bezierPathWithRect:glowView.bounds].CGPath;
    viewLayer.shadowRadius = 10.0f;
    viewLayer.shadowOpacity = 1.0f;

    //Let's animate it while we're at it.
    CABasicAnimation *animation = [CABasicAnimation animationWithKeyPath:@"shadowOpacity"];
    animation.duration = 0.5f;
    animation.timingFunction = [CAMediaTimingFunction functionWithName:kCAMediaTimingFunctionLinear];
    animation.fromValue = [NSNumber numberWithFloat:1.0];
    animation.toValue = [NSNumber numberWithFloat:0.0];
    animation.autoreverses = YES;
    animation.repeatCount = HUGE_VALF;
    [viewLayer addAnimation:animation forKey:@"shadowOpacity"];
}

I must admit, this is kind of basic stuff. But it seems a lot of people actually forget about the fact that all UIView subclasses are based on Core Animation CALayers.

See the source code https://github.com/xebia/ios-DemoForBlog and try the example, the interesting bits about this example are all contained within the fourth tab of the application, it’s the XSDFourthViewController in the code.

Software development processes

Continuous is the word. Continuous integration, development and deployment. Software should not be production ready, software should be used in production. That’s also the main idea behind the lean startup movement: release something viable as soon as possible, so you can get feedback, but not just at (lean) startups, at any enterprise. More and more organizations finally realize the extremely long time it takes between idea and actual realization is wasteful all the way.

With that regard, fixed price, fixed scope projects, fixed time are out. Creating elaborate specifications consisting of long documents by management consultancy companies and then creating software based on those documents by the cheapest bidder. Fixed price has never worked, will never work and it won’t be to long for even the government to realize that. Or at least the people that don’t retire. But as many organizations can’t handle truly agile development yet, we may see the rise of fixed price, fixed time projects with a single goal rather than a fixed set of requirements. That requires both customer and vendor communicate openly with each-other, in word and in person. So traditional contract based tenders (aanbestedingen in Dutch) don’t work here. Fortunately, they’re are good alternatives, agile and lean contracts are worth an entire blog posting if not a book.

Local Cloud

The cloud is hip. Not everyone agrees on the definition, but one I use in this posting is using resources of connected computers, and getting billed only for usage. When everyone can use those connected computers, you’re on the public cloud and you not only have to trust Amazon, Google, Rackspace or Microsoft or other vendor thats maintaining those computers, you’ll also have to trust the government where the actual hardware is located. For this reason, most of these vendors have the option to have your data and software be hosted only in specific regions (like US, Europe, Switzerland).
Better then trusting a vendor, is to host your own local cloud. Multiple solutions exist for that too: Redhat, VMWare (Spring) and various other companies provide cloud solutions that you can install on your own data-center. Many organizations see the benefit of that, so lots of organizations will want to have they’re own cloud.

Local cloud means your application just uses up capacity on demand of your entire data-center, both in terms of hardware resources as in people. Since all of your applications share the same data-center so resources and people are used more efficient. When capacity isn’t enough, you just order new hardware. The cloud solution like Redhat or VMWare will make sure those new resources are used. The system operators (ops) can focus on having run you’re cloud smoothly, rather then individual applications or servers.

Privacy

Speaking of clouds: people that worry about privacy issues on the data being stored at Google, Facebook, Apple and lots of other companies will be listened too. The EU will enforce companies to allow people to fully access and optionally remove any personal data companies have gathered on them. With a bit of luck and some sanity will be able to move our data from one cloud to another, truly owning our data.

Security

Security is also an important aspect of clouds and software in general. Quite a lot of security incidents have occurred over the last year. Often, if not all the time, the root cause is emphasis on procedures and procurements rather then the mathematical and technical aspect of security. Notable the Diginotar disaster in our country, The Netherlands. Diginotar was the royal provider of digital certificates. The company had trustworthy name (Notar comes from the Dutch word Notaris – Notary) and a business consultancy company PwC (Dutch article), have verified they had proper checklists in place!

People will realize to have truly security, you’ll have to understand the software and the encryption methods you use, rather then policies, brochures and buzz-words. Also something which is worth a few blog postings.

Let’s take a closer look at this comparison and see if it makes any sense.

For my brother, to successfully manage a construction project from start to finish, he needs to fill in at least 4 blanks.

1. Have a goal and the ability to monitor progress

My brother needs to know what his goals are at the start of the build and make sure he is able to measure progress to see if he is getting any closer towards those goals. In construction, goals can be unclear and contradictious. Is our goal to make profit, to stay in budget and time boundaries, be as eco-friendly as possible, to create structures that people love or to show off your design skills and taste…?
The same goes for me. Organizations need goals to justify change. Change just for the sake of change means nothing. There needs to be something to fix and a sense of urgency to take responsibility and change.
The second element here is measuring progress. The problem in construction is that the process involved is often linear. So if a builder sees he is not on track, can he actually change the plan or the situation? The truth is, most of the time he can’t. My brother cannot inspect, adapt and experiment like me. Yes, if he should see he is not on target, he could use other materials, but only if cancelling the existing orders is still possible. Lucky for me, my outcome is not shaped from bricks and mortar. Only the goal is fixed. The means to reach our goals is subject to change and inherently unpredictable surpassing pattern levels due to the self-organizing abilities and continues improvement cycles we try to advocate.

2. Know the soil and what to build on top of it

The builder also needs to make sure the ground he is building on is suited for building purposes. The different build-ups of the soil have impact on it’s suitability. What if a building sinks and all kinds of cracks appear? Also he needs to check whether the ground could be polluted from prior use. The build can’t take place if there are health hazards involved!
This is the same for me as a change agent. I need to get a feel for what I am working with. I have a look at the facts and get some sample data from the part of the system we would like to change. I have to do so, because I need to be able to judge the realism and ambition of our goals.
Some parts of organizations are easier to change than others and some aspects, like culture for example, change slower than others. Slow changing parts take longer to feedback the results of our efforts. If culture needs change to reach a short-term goal, this will not work.
The second part is to have a design to illustrate the future situation. The construction expert needs these designs to get a mandate from local building authority, to explain the vision to all other stakeholders and to get a buy-in from end-users. This is also the case for change agents. The only difference is; we can let go of the design. We create a new design for the system once, as it gives room to discuss with each other how to proceed. After that we consciously try to experiment our way towards our goals, more or less updating the final design as we go. This gives room for learning. There is next no room for learning in the build though. It is plan driven, mistakes are only made for correction according to the plan.

3. Knowledge of the surrounding area

We shouldn’t forget that our structure would be part of an existing eco-system. I live in a rural part of the Netherlands. If there is new construction going on, I’m always afraid for how the build will affect the existing landscape. My brother will have some vision on how the new structure will enhance the landscape in some way, but the most important thing is to keep the balance and harmony in the system.
I am sure that this is exactly the same in construction as in agile consultancy. When I change one part of the organization, I need to check whether or not the surroundings of the new form are still in balance with each other. If we preach co-operation within a business unit and we overlook that the companywide HR system only rewards individual achievements, we will not likely get the results we need. So if the extended system is not aligned with the part we changed, we need to inspect and adapt further. Either implementing further changes within the system in scope, or try to involve the surrounding systems more and adjust over there. Some call this organizational gravity by the way. I just call it common sense.

4. Involve the right people

During the whole process of the build, the right people need to be involved: the one who ordered the build (of course), government inspectors, sub-contracters, customers and other stakeholders. My brother is full of stories of construction projectmanagers not paying enough attention to having the right people at the table. The consequences of these mistakes are often severe.
The same goes for consulting. You need a group with the right mandates (with regards to the system you are changing) to form the change along with you. This also includes the people doing the actual work on the work-floor, as they have natural mandates over how they think their role is best performed from day to day. This group should be focused on their tasks and basically adhere to all other things we know to work for agile teams; dedication, co-location, working from a prioritized change list etcetera. Implementing agile and lean as a means to reach a higher goal, is not something you do in between meetings, it takes serious effort, focus and determination to get it right.
The change effort is much like a large construction project in the sense that it can be highly complex, both in following what is actually happening and processing all the data and feedback at the same time. This is why in both construction and consultancy this process should be facilitated as much as possible through the above measures.

Conclusion

Oh…. ok, maybe I am trying to compare apples with oranges, and maybe I really like the idea of sharing professional commonalities with my brother, but it Looks like my brother and I at least have some similarities in our work. The most fundamental difference to me is the fact that construction is hugely plan-driven. In my line of work we also need some sort of plan or future state picture, but only at the start of the change. This picture itself will change anyway, because we will find ever-improving ways of organizing and executing our work. We will only need to update our future state as we go along, finding those elements that add the most in our quest of reaching our goals and try to implement them more.

Were my brother remains working on a change, the center of my work shifts from the first change to perpetual changing of the organization itself.

Thanks for inspiring me bro!

Because we were working in short iterations (1 week Sprints), we had awesome focus and we could deal with most discrepancies between dreams and reality quickly.

Making the deal

At the end of last year I got an email of a consultant I knew from a previous client. He was helping a guy start a business and he thought he needed a good developer to build his product for him. It shouldn’t be very complicated, 3 months of work should do it. After a couple of emails, me and Peter Petit met with Reyndert, the man with the plan. He brought with him a spreadsheet with requirements, a flow chart of user interactions and a bunch of ideas. It turned out that he wanted to build a video recruiting product where recruiters could publish video interviews for jobs and people could apply for those jobs using videos of their own. Cool!

Video I thought is a solved problem and I found several articles about solutions to this type of problem, even without doing your own flash coding. I felt safe and comfortable and I knew at Xebia we could build stuff fast. We made a deal to build a first version for a client demo in February (6 weeks after starting the project).

Obviously I knew this was ambitious, but knowing what our guys could do in a day I wasn’t worried that we could work our way out of this. We decided to build this thing with a team of me and two apprentices and use all the help we could get from our colleagues when they were in between assignments. We started off with a team of five, which was more than I had hoped for.

Video streaming, how hard can it be?

The biggest challenge in the project was video streaming, so we attacked that first. It turns out that recording video from the browser is not as easy as I assumed. By this time, the client expected a full featured video recorder and a scalable hosting solution for video too. Woops. We found Nimbb and made it work with our application. Nimbb is a service that allows you to record videos with a flash player that you can talk to from javascript and then hosts the video for you at a fee. This saved us a ton of time, because we didn’t have to set up a scalable video hosting platform or build the recorder ourselves. I thought that was a clever trick, and I believe it made a lot of sense business wise, but it also meant some interesting discussions about the scope of the project, the operational costs, and all that. Lesson learned here is that if you’re going to be smart and think lean, make sure the client is on board with that before you start cheating. You don’t want the guy paying the bills to feel cheated. You should be cheating reality together with the owner, then things are great.

During this time we knew we were on a tight deadline and we pressured the client to deliver designs to us so that we could make our application presentable. The design firm he used delivered a very nice logo, but no acceptable UI designs. Woops. As we worried and pressured, it started to dawn on us that we would have to do the UI and UX designs ourselves too. Getting this right is not an easy job and we had little expertise in the team.

By this time our six weeks were almost over. The first demo to a prospect of our client was upon us. We polished the application somewhat and sat down with our client to look at the damage. By this time the amount of work we had in our backlog had almost doubled. As you know great idea’s tend to have offspring. We knew very well we wouldn’t deliver all of it. It was time to prioritize ideas and slim down the scope.

The client appreciated what we had done so far and agreed that new work had surfaced along the way. But he rather have all of it than go live too soon. I disagreed. Going live soon means early feedback and many of the features we had thought up might not make sense to clients. Discussions got heated, but together we managed to keep things on the constructive side.

The demo was a success, but the application was not finished. After some tough negotiations we agreed that we would spend six more weeks and make sure we could go live with an application that was ready for business then.

Video uploads, and still no design

The client wanted to allow recruiters to upload prepared videos as well as record them, and Nimbb doesn’t support video uploads. We proceeded to integrate with Vimeo to support video uploads. Vimeo supports many formats and has an awesome player that even works on mobile devices. Now with two subscription accounts and some glue code we had solved the hardest problems of our challenge.

In the mean time we had also built most of the website and the work left was to finish up the detailed features of recruiters rating the applications coming in, and the sysadmin interface to support recruitment companies in setting up accounts, removing stale applications and adding labels for the different languages. We were supposed to support multiple languages in the first release, did I mention that?

By this time we had decided to do all the UI and UX design ourselves. We learned a lot and came up with some nice solutions, but as we expected this was hard work.

Production ready software

Since we also promised to host the application we had to come up with some smart tricks to have a production ready setup for our MongoDB and web the nodes. We wanted to have good performance and scalability and also redundancy. We wouldn’t be able to set all that up in such a short time, so we used Heroku and Mongolabs to get it for a (very modest) fee. Forced by Heroku, we removed all use of HttpSession from our application and made it commandline executable with Jetty. After some tweaking, we were quite pleased with the results. In fact I’m so pleased with the result now that you’d have to put a gun to my head to make me use HttpSession ever again. The pleasure of scaling on a PaaS cloud and maintaining truly stateless web code is such a relief.

Champagne

At the end of our twelve weeks we had a demo. The demo went well and by that time the owner had demoed the application to a handful of prospects successfully. Sipping champagne we applied the last small bits of polish and went on to our next project. Of course there are things that we would do differently if we would do it again, or if we had had more time, but I don’t believe that the ROI on “more time” is much smaller than many developers think.

It’s hard to believe that we actually went from an idea to a a real product in 12 weeks, but I’m quite sure it actually happened. Hah!

As part of those initiatives, many of our customers are also virtualizing their infrastructure. The functionality that makes Deployit ideal for deploying new releases also make it a perfect companion to an on-demand infrastructure strategy. When spikes in demand for applications hit, virtualized infrastructure makes it possible to scale up quickly and automatically. But this infrastructure is not terribly useful without an application running on it. Deployit ensures that the newly provisioned servers run the right version of the desired application (configuring loadbalancers, static HTML, Java or .NET applications and databases) and join in shouldering the increased load.
read more

Imagine you are selling your house and a potential buyer is coming to check out your property within one hour.  Your home is a complete mess and you really need to clean the house. You start straight away and work hard to get the job done. You’re vacuuming every corner and you’re polishing every vase on the table. Quality without compromise keeps popping in your head. Then your real estate agent comes in and checks the place out. You hear him mumble while he’s investigating your home. You’ve done the best you can and then he starts telling you what you’ve done wrong. He tells you that the potential buyer is totally into Feng Shui and he thinks the place doesn’t look in balance at all. It’s kind of frustrating isn’t it? How could you possibly know? Think about how you would you react to your agent.

Some people avoid seeing or bringing bad news altogether. They put them off as long as possible and sometimes dodge them. If you really want to be successful and competent at what you do, you should not be afraid of receiving and communicating bad news. Real winners actually try to spot them as soon as possible. Winners are up for challenges and they deal with them.

In traditional waterfall projects testers work together testing the product. But there is no challenge in communicating bugs through Excel sheets or other electronic tools. It does not talk back or show emotions. Communicating them with your testing teammates isn’t hard either. You are all on the same “testing” side. Which is totally a bad practice in my opinion. In Software Development there should not be any sides. We build software together. We are all involved in changing people’s lives. I’ve met some testers who really didn’t care about faults or bugs in the product. They did not show any engagement with the product. Some of those testers were actually happy about finding a bug and were excited to show how it’s broken. This is bad…

There are many ways to deal with bad news. But the most important thing is we should stop avoiding bad news. Maybe it’s a no brainer, but stop finding excuses! Force yourself out of this habit. Yes, it’s painful, that sinking feeling in your stomach when you find out that things aren’t going like it supposed to. And yes, such pain is difficult to invite into your process. But in this situation it’s absolutely necessary. Bring on the bad news. Seek it out. Speculate on its possible sources.

Garbage In, Garbage Out

When you want to communicate an issue to a developer, try to understand that there might be a possibility that the required functionality is not well explained or understood. Garbage In, Garbage Out… Developers are already struggling with writing code. Don’t blame anyone for not knowing every little thing. Also communicate your finding with a concrete example. Being abstract confuses the situation even more, so show (or write down) the steps to reproduce the issue.

Everyone is responsible

Make sure you are not the only one reporting bad news or issues. Testing should be a team effort. Everyone should be involved with testing in order to get more insight of the product. Automating your tests will help, but try to build them with the whole team so everyone is engaged getting the testing/checking right. When something bad happens the whole team, including the tester, is responsible for not knowing this upfront.

It is not the symptoms of the problem that must be treated, but the cause

When you receive a bug from your teammate. Don’t fix it straight away. Try to find out where it came from and why you it occured this late. Most of the times it’s because of some obvious assumption you’ve made during your work. Make sure you verify and communicate any assumption with your teammates.

Help developers test

Recurring issues should be avoided. But when it happens don’t be too negative about it when you communicate this issue (again). When you see an issue coming back after a bug fix, try to help the developers build an automated (unit) test. Explain to your teammate why it’s important to make sure that this problem does not happen again.

Overcoming bad news

One of the things I’ve noticed working with “winning” Scrum teams is that they overcame any bad news / problems. Call them Agile or not, I don’t care. The team (including testers, developers, business analysts, etc) knew that there would be a large number of obstacles to be thrown into their path. Ignoring bugs was unacceptable. They tried to know as much as they could about the problem and formulate a strategy to “defeat” the situation. Communicating bad news lead them to stop wasting resources on a less important effort. They preferred to know about problems standing in the way of key users sooner rather than later.