Of course installing this application in the production environment should have been as easy as the setup on our own laptop’s right? Wrong! On our laptops it was a easy git clone followed by a npm install and voila we have a running application. So how hard could it be to do this on a server at the client. Let me tell you….

This server is not connected to the internet, so the git clone wouldn’t work in the first place. Not really a problem because it’s a small app and we could just make a tarball and ship it to the server.
Next thing was all our dependencies. We used a few modules which were mentioned as dependencies in our package.json file so npm with the install command would do it’s magic.

The npm install magic consists, among other things, of getting the modules from the npm registry and that fails if you’re not connected to the internet. Searching for a way to do this differently I figured out that npm had a cache directory and thought I could get the stuff from there. This might have worked, but with that solution I would miss the dependencies where these modules depended on. And it would be a messy kind of script that I needed to make.

Browsing the internet didn’t provide me with the right answer but it led me on the path to the npm pack function. This is used to pack your module together with all dependencies. The only thing you need to configure it correctly, is a separate array containing the dependencies to bundle with your app.

So far so good, so I went on and added a bundleDependencies section in my package.json and ran npm pack.
The result was a nice .tgz file containing all the files needed for the application together with all the modules it depended on. At least the main modules it depended on. My fellow programmers in crime from which I got these modules hadn’t bothered to add this extra section, so npm had no notice of the modules they depended on.

This was easy to solve. Just add a correct section of bundleDependencies to all package.json files.
Sounded like a boring task to do this manually and because I love my programming job, I decided to write a program for it.

My obvious choice of programming languages was: awk, grep and sed. Why? Because I can.
Without further ado, here it is:

    awk '/dependencies/,/]|}/' $file |
    grep -o '\".*\".*:' |
    sed 's/^.*{//g' |
    sed 's/\"dependencies.*\://g' |
    grep -v -e '^$' |
    uniq |
    sed 's/\"[ ^I]\:/\",/g' |
    sed 's/\"\:/\",/g' |
    sed '$ s/,/ ]/' |
    sed '1 s/\"/\"bundleDependencies\" \: [ \"/' |
    sed 's/\"/\\"/g' |
    tr -d '\n'

What it does, you ask?
I'll explain line by line:

line1: get the dependencies part (an array or json object) from the file (the package.json file)
ex: "dependencies" : { "express": "0.2.2", "findit" : "0.0.1" }

line2: get only the stuff from that object between the double quotes before the colon (removing the version number part of the dependency)
ex: "express":
"findit":

line3: remove anything preceding the {
line4: remove the original dependencies text
line5: remove any empty lines
line6: remove duplicates

line7 and line8: replace the ": by ", so we can create an array from it
ex: "express",
"findit",

line9: replace the last , by an ] to close the array
ex: "express",
"findit"]

line10: replace the first " by "bundleDependencies" : [ "
ex: "bundleDependencies" : [ "express",
"findit"]

line11: precede all quotes by a backslash so it can be safely used in the sed command to add it to the file

line12: remove all newlines
ex: "bundleDependencies" : [ "express", "findit"]

This is added to the package.json file we are currently processing and if we have finished doing this for all package.json files we can use npm pack to create our tarball.
Works pretty well I might say. But I'm the first to admit this isn't the most readable program ever written.

Of course when building a node application you might have node around to help you do this so I also created a javascript version to do this:

var fs=require('fs')
var findit=require('findit')

findit.find('.', function(name) {
  if (endsWith(name,'package.json')) {
    handleFile(name)
  }
}).on('end', bundleApp)

function handleFile(file) {
  var data = fs.readFile(file, function(err, data) {
    if (err) {
      console.log('Not processesed '+file+' bo the following error: '+err)
    } else {
      var arr = []
      var packageFile = JSON.parse(data)
      if (packageFile.bundleDependencies) {
        console.log('Bundledeps already present. Skipping')
      } else {
        for (var d in packageFile.dependencies) {
          arr.push(d+"")
        }
        if (arr.length > 0) {
          packageFile['bundleDependencies'] = arr
          fs.writeFile(file, JSON.stringify(packageFile, null, 4))
        }
      }
    }
  })
}

function bundleApp() {
  console.log('Finished. preparing package.json files for packaging. Now run npm pack to create the fullblown tarball')
  //exercise left for the reader to require('npm') and run the pack command
}

function endsWith(str, suffix) {
  return str.indexOf(suffix, str.length - suffix.length) !== -1;
}

Hope somebody can benefit from this in the future.

After the first few issues the so called ‘Transaction timeout’ issue raised it’s ugly head.
Every now and then the application threw an exception due to a transaction timeout.
This was very strange since the timeout was set to 30 seconds and the complete processing of the whole
application was done in less than 2 seconds (spread over more than 1 transaction).

The first thing to try was to setup a production like environment to try and reproduce this.
We spent a few hours to make our development environment as much production like as we could.
With a lot of stubbing on the outside boundaries of the system and a live feed of data to process, our
test was started on Friday afternoon.

We where very surprised to see 0 timeouts when we arrived at the office on Monday morning.
On to plan B. The original application consists of 4 parts so the plan was to strip our environment to only run a small part of the original application.
We also changed the transaction timeout to 1 second to raise the chances of seeing timeouts.
This way we might predict in which part of the application the search needed to be continued.

1 != 1

The title of this paragraph might sound a bit strange but hold on, an explanation will follow soon.

After our testrun there were some transaction timeouts. We also noticed that this part of the application did the normal processing in about 13 milliseconds.
Fast enough to beat the timeout you would think.
But then we discovered that the reason for these timeouts was not the same as the timeouts we experienced in our live environment.

The reason we got timeouts in this setup was because of a somewhat different meaning of the timeout setting than we expected.
We expected the timeout monitor to start when the transaction started and raise an exception after 1 second. But…..

Let’s take a look at the code

in the class org.hibernate.jdbc.AbstractBatcher there are two methods involved

public void setTransactionTimeout(int seconds) {
    isTransactionTimeoutSet = true;
    transactionTimeout = System.currentTimeMillis() / 1000 + seconds;
}

and

private void setTimeout(PreparedStatement result) throws SQLException {
    if ( isTransactionTimeoutSet ) {
        int timeout = (int) ( transactionTimeout - ( System.currentTimeMillis() / 1000 ) );
        if (timeout<=0) {
            throw new TransactionException("transaction timeout expired");
        }
        else {
            result.setQueryTimeout(timeout);
        }
    }
}

If you set the transaction timeout to 1 second, Hibernate raises the timeout exception at the switch of the second exactly.
When you start your transaction at 13:45:21.990 the exact timeout is 10 milliseconds! No wonder we saw these timeouts.
So if you want the lower boundary of your transaction timeout to be 1 second you need to set the Hibernate transaction timeout setting to 2 seconds!

And yes, after setting it to 2 seconds, no more timeouts!

Silly idea!

During the testing I stumbled upon some threaddumps (made for a different reason). The strange thing I noticed was a lot of threads waiting for a lock on apache log4j related classes.
We do some extensive logging in our application so this wasn’t completely strange.
But somewhere in my gray matter a connection was made between this phenomena and the transaction timeout we were investigating.

So I shared my thoughts with the team. I told them that I suspected that the logging done in the application might take such a long time that the transaction was not finished within the 30 seconds.
And then I was laughed at! How on earth could something as fast as writing a line to a logfile take this long.

This silly idea triggered some more silly ideas (although not as silly as mine), thus the team bet was born!

Team bet

The team bet is a bet that emerged inside the team. All ideas about a problem, which is investigated inside the sprint, are put on the whiteboard. People can come up with their own, or support the idea of another team member.
The only strange thing is that the winner (and his/her supporters) need to pay for the cake. Eternal fame is the price you win of course.

For a few days my silly idea was on the whiteboard with the sole support of the tester in the team.
Because it was the most silly idea on the board, the other ideas where investigated first.
But after a few days the only thing left was this silly idea. In the days that went by my feeling for this silly idea got stronger.

In my spare time I went through the code and pinpointed the strange behavior to the log rotate functionality.
Whenever a new logfile was created the oldest one needed to be removed to keep the directory clean.
But a few tests with a large amount of files showed that this was done in under 30 milliseconds on my local system.

Pigheaded me

Pigheaded as I am, I was still convinced that this was the problem. Because at this time everybody from the team was willing to prove me wrong we decided to
empty the logfile directory in the production environment (because that was the only place where the timeouts were experienced).

Before this action we experienced about 10 to 15 timeouts per hour. And yes you guessed right, after emptying the log directory 0 timeouts for a period of several weeks.
So this was beginning to sound like a duck, but to be sure it was a duck we needed to do a reverse test.
But hey! don’t touch a good working system (because that is what we created by removing the large amount of files), so we weren’t allowed to experiment anymore in our live environment.

And because all was running smooth regarding the transaction timeouts, our focus switched to the next most important issue.
To keep a little progress on the transaction timeout issue we asked the testers who do the acceptance testing to do this reverse test for us.

And after a day we got the final word: transaction timeouts were caused by the large amount of logfiles in the directory combined with a log rotate and cleanup of the oldest file.

End result of this quest was that me and my supporter took off to the nearest bakery to buy cake for the whole department.

And this is why the quest had cost me money and gave me eternal fame in return.
And this also makes me wonder about how free open source software actually is…..

The application is running in a browser (currently IE7) and browsers are generally limited to do not more then 2 parallel request to the same domain.(this has improved a bit in later versions of the browsers). In this post i will describe the quest for solutions.

A simplified description of the application will help to understand the problem better i think. The application is used to track movements of cars around the country so a lot of data is gathered every N seconds with the help of XMLHttpRequest. For every type of car there is a separate request done to the server. Currently about 5 requests are issued every 5 seconds.

There wouldn’t have been a noticeable problem if all the queries had performed well and every request would take less then say 1 second, but one of the queries was performing very slow with a response time of over 20 seconds. This query was used in 2 out of the 5 requests. Also the app was build in a way that if a new request was triggered the current request was aborted. So the problem was that at least one of the 2 parallel requests was never able to deliver an answer and was holding up other requests.

Using sub domains

So even after improving the query performance there was a need to do more then the 2 parallel request at a time to speed things up even more. For normal http requests a widely used solution is to set up sub domains in your DNS server pointing to the same domain which lets the browser believe the request is made to a different domain. This technique is used with Google Maps for example to get the images from multiple domains for a faster user experience.

I wanted to implement the same kind of functionality for my XMLHttpRequests so i went off and implemented a solution to select an sub domain from an array and send the request there. Unfortunately this didn’t work because of the security restrictions on cross domain XMLHttpRequest. Even if i know better then the browser that this sub domain is essentially the same domain the browser was restricting me here. That’s what you get for tricking the browser to do more in parallel right?

Searching a bit more i found all kind of workaround-ish solutions using frames and stuff to do cross domain XMLHttpRequests but the most useful solution was limited to the newer generation of browsers (Firefox 3.5+, IE8 etc.)

Using response header Access-Control-Allow-Origin

Setting the response header Access-control-Allow-Origin to the exact (including http://) value of the origin your web application is loaded from made it possible to do cross domain XMLHttpRequests.The easiest way i implemented this was to load the module headers_module (modules/mod_headers.so) in my Apache http.conf and add the following line to the configuration file:

Header always set Access-Control-Allow-Origin: http://maindomain.sample.org

this made the browser send an OPTION request on the XMLHttpRequest.open call to check if the call was allowed before sending the data with the POST request.

And what for the older browsers?

But he! I’m still running on IE7 right? Right so back to the drawing board.

So currently I’m trying to combine as much of the requests into one request and filter out the data on return. In the meantime I found that for IE you can influence the amount of parallel requests done to the same domain with an adjustment of your registry settings. Since the app is running in a closed and controlled environment this might be the way to go if my programming attempts are unsuccessful.

Especially the Birds Of a Feather session yesterday made me really enthusiastic about the next level of OpenLayers.

Changes to come?

The discussion in the Birds of a Feather session were mostly about the following subjects:

Documentation

Documentation is always a big issue within open source projects. Its always difficult to write good documentation without any clear objective or guideline from the people using the documentation. In an opensource project you not only have to deal with the users (who you don’t know) but also a lot of developers with mixed experience.

Examples

There are a lot of examples on how to use OpenLayers. The only problem is that these are not all written with the same style. Some of them are still using some old school way of dealing with the OpenLayers code. It would be great if there could be a set of examples using the latest and greatest way of handling different use cases, written in the same style of coding. This prevents new time users to build of a deprecated example and asking all kinds of questions on the mailing list.

Separation of UI look and feel

The current control elements are containing both functionality and representation stuff so it is considered a good thing to separate those two concerns. The control object would be modified to only contain the specific functionality and launch events on which the UI layer could base its presentation. This way it would be a lot easier to substitute the current basic looks of OpenLayers whit something fancier with the help of Ext, GeoExt or JQueryUI.

HTML5

With the new html5 functionality around the corner it would be great if we can start using the geolocation api to for example center the map on your current position, use a canvas for the map and its layers to minimize the dom manipulation which causes big memory problems (at least in IE).

This will be a big change from the current API since i imagine the Map to be a 3d (or 2d) canvas where you can add your image tiles and features from different layers. Would be awesome to be able to use 3d and create a globe with your tiles on it!!

Summary

I personally want to work on the HTML5 stuff (2d at first) creating a new Map object and adding tiles and features to it would be the first go i think.

I know some people are afraid this would break the current way OpenLayers is working. So one thing i want to know from whoever is reading this: Is this really breaking the current API or is it just adding new and cool stuff?

Part I

The need for using Spring JMS opposed to ‘normal’ MessageDrivenBeans came out of the following requirement:

On receiving messages from the Queue we need to send these messages to another system through a Socket connection.

This means that we cannot start processing the messages from the queue until the socket connection is available. This socket connection is created on application startup in  a spring bean so it isn’t possible to guarantee that the connection is available. And also the application will be deployed inside a WebSphere cluster so one of the two servers in the cluster will never get a connection to the socket.

So we need some kind of programmatic control over the starting of the message listener.

Luckily this is possible in Spring with the start (and stop) methods of the AbstractJmsListeningContainer or the GenericMessageEndpointManager. Since we already had an implementation with MessageDrivenbeans and ActivationSpecifications to distribute the messages we decided to go with the JmsMessageEndpointManager because we could reuse our ActivationSpecifications configured in WebSphere (well we thought we could do that…..)

Reusing the configured ActivationSpecification was not possible see this topic on the springsource forum so we configured the ActivationSpecification in the spring config files together with the message endpoint handler the resource adapter and the workmanager as described in the using Spring JMS on WebSphere article.

No messages were picked up and no error messages were found to guide us in the right direction. So without more information it became some kind of trial and error exercise (which i hate, and become to hate even more because of the long /try to fix/deploy/test cycle).

So after a few tries i decided to take a closer look at the sources i had used to develop the configuration and after reading over it more than once my eyes stopped at the following piece of text hidden in the using Spring JMS on WebSphere article:

…Be aware that no other Spring JMS MessageListenerContainer types are supported…

Did this mean…..? Yes it did. By making the decision to use the message endpoint handler i had used the wrong type. So after a few minutes of rewriting the configuration to use the DefaultMessageListenerContainer (together with a messageselector to replace the activationspecification), setting the autoStartup property to false and injecting it into my connection manager bean to programmatically start the listener after the connection is created, one more long /try to fix/deploy/test cycle………..

Part II

and then it almost worked. A ClassCastException in the onMessage method stating that class A could not be cast to class A. Why isn’t it possible to cast something to itself. Ok i admit it seems a bit strange to want to do this and i didn’t do it on purpose really. The onMessage method was supposed to receive a Serialized message as input parameter so i thought i needed to cast it. But it turned out to be a class A already. So i changed the method signature to take a class A as input parameter.

So we got rid of the ClassCastException this way but now we got an NoSuchMethodError (i’m not exactly sure what i prefer). This is going to look like a ClassLoader issue and if it looks like a ClassLoader issue and behaves like a ClassLoader issue it is a ClassLoader issue!

But how can that be? I have an ear file with a war file which loads all the spring bean with the contextLoaderListener and nothing else. Well almost nothing else. On the same server a different ear is deployed which also makes use of class A. therefore class A is in the core module (its called core module for a reason!). Could it be that?

Yes it can be that! The Websphere server can decide to use a ApplicationClassloader for multiple applications so on server level we need to set the classloading policy to single to prevent the ClassCastException or the NoSuchMethodError.

Hope this helps anyone out there to avoid these pitfalls.

The case i ran into was a defined database field of NUMBER called status with the possible values of 0,100 and 200 meaning INITIAL, ACTIVE and INACTIVE. Using the default conversion with the below mentioned enum would mean that the values 0, 1 and 2 where used to insert into the database.

public enum Status {
  INITIAL, ACTIVE, INACTIVE;
}

and using the enum in the entity class

private Status status

And what if someone decides to add an extra status called CHECKED with a value of 10?

Should i create an Enum like this to be able to use the default?

public enum Status {
  INITIAL, UNUSED_1, …, UNUSED_9, CHECKED, UNUSED_11 .. UNUSED_99, ACTIVE, DO_YOU_GET_THE_FEELING?;
}

So i looked further and found that with EclipseLink (the JPA provider we use) it’s possible to define a Converter and use that in your Entities.  So let’s try.

First we need an Enum which contains the correct value inside: (See also a previous blog on Enums)

public enum Status {
  INITIAL(0), CHECKED(10), ACTIVE(100), INACTIVE(200);

  private final byte status;

  private Status(int value) {
    status = (byte) value;
  }

  public int getValue() {
    return status;
  }
}

Next we need a Converter:

public class StatusEnumConverter implements Converter {
  public Object convertDataValueToObjectValue(Object data, Session session) {
    //TODO implement converting database value to Enum
    return null;
  }

  public Object convertObjectValueToDataValue(Object data, Session session) {
    if (data instanceof Status) {
      return BigDecimal.valueOf(((Status) data).getValue());
    }
    return null;
  }

  public void initialize(DatabaseMapping dbMap, Session session) {
    // No need for database mapping
  }

  public boolean isMutable() {
    return false;
  }
}

That’s one way traffic now. How do we get form a value to the correct item of the enum?

Looking on the internet i found the Enum Inversion problem which supplied me with a solution for this. Also i wanted the StatusEnumConverter to be more generic so the converter could be easily reused. I create a ReverseEnumMap class to aid to the problem of getting the right Enum value and a convertable interface to generify (is that a word?) the Converter. The end result looks like this:

public class ReverseEnumMap & Convertable> {
  private Map map = new HashMap();

  public ReverseEnumMap(Class valueType) {
    for (V v : valueType.getEnumConstants()) {
      map.put(v.convert(), v);
    }
  }

  public V get(Object obj) {
    return map.get(obj);
  }
}

public interface Convertable & Convertable> {
  Object convert();
  E getFromValue(Object value);
}

public enum Status implements Convertable {
  INITIAL(0), CHECKED(10), ACTIVE(100), INACTIVE(200);

  private static ReverseEnumMap map = new ReverseEnumMap(Status.class);

  private final byte status;

  private Status(int value) {
    status = (byte) value;
  }

  public Object convert() {
    return BigDecimal.valueOf(status);
  }

  public Status getFromValue(Object obj) {
    return map.get(obj);
  }
}

public abstract class AbstractConverter implements Converter {

  public abstract Convertable getConvertableEnum();

  public Object convertDataValueToObjectValue(Object data, Session session) {
    if (data == null) {
      return getConvertableEnum();
    }
    Convertable convertableEnum = getConvertableEnum().getFromValue(data);
    if (convertableEnum == null) {
      throw new IllegalArgumentException(
          "Data not with a value suitable got [" + data.getClass() +" : "+data
          + "] expected a valid value of ["
          + getConvertableEnum().getClass() + "]");
    } else {
      return convertableEnum;
    }
  }

  public Object convertObjectValueToDataValue(Object data, Session session) {
    if (data == null) {
      return getConvertableEnum().convert();
    }
    if (data instanceof Convertable) {
      return ((Convertable) data).convert();
    }
    throw new IllegalArgumentException("Data not of correct type got ["
      + data.getClass() + "] expected [Convertable]");
  }
  //left out the other methods for readibility
}

public class StatusConverter extends AbstractConverter {

  private static final long serialVersionUID = 6209909216228257358L;

  @Override
  public Convertable getConvertableEnum() {
    return Status.INITIAL;
  }
}

And this is how you will use the converter in your entity class. Annotating the status field. Note that you only have to define the @Converter annotation once to be able to use the @Convert in other entities also!

@Converter(name="status", converterClass=com.xebia.enum.converters.StatusConverter.class)
@Convert("status")
private Status status;

So with this i think we have a pretty generic solution to avoid using ordinal with enums and JPA.

In this blog I will describe the tools used, a few of the problems I ran into and of course the solutions to the problems which involve coding and communication

The tools used where a Java based server called Geoserver and a client side JavaScript library called OpenLayers.

Geoserver

GeoServer is a server that allows users to share and edit geospatial data. It is the implementation of the Open Geospatial Consortium (OGC) Web Feature Service (WFS) and Web Map Service (WMS) standards.

WMS

A Web Map Server publishes maps. In short it is used to create pictures from all kinds of different spatial sources like databases (for example postgis, oracle spatial), files (like esri shapefiles, GML (Geography Markup Language), MapInfo) etc. It can also be used to get some extra information stored with the geographical data.

WFS

A Web Feature Server is an interface to get, supply or change geographical vector data. It uses the GML format to communicate between client and server.

OpenLayers

OpenLayers is an open source JavaScript library for displaying geographical data in a rich web environment, the API can be used to create Google Maps like applications.

Challenges

In my current assignment I faced a few challenges in different parts of the application. In the next part of this blog I’d like to explain the solutions I used. I hope someone will benefit from this. With each solution i will also explain some of the surrounding techniques used within the GIS domain.

Reprojection

One of the problems in the application showing the position of the mobile caller was the accuracy of the position. The position data was provided in WGS84 (world geodetic system) This coordinate reference system is also used for GPS coordinates. But because the coordinates needed to be put on a map of the Netherlands which was in a different coordinate reference (RD) the coordinates needed to be transformed (this is called reprojection).

Transforming between two coordinate systems involves difficult mathematic formulas, but OpenLayers is prepared to use the Proj4js library which simplifies this for you a lot.

So with a simple statement like this you can transform between 2 different projections:

  //transform from wgs84 to rijksdriehoeks  28992
  markerLocation =new OpenLayers.LonLat(x,y);
  markerLocation.transform(new OpenLayers.Projection('EPSG:4326'),
                           new OpenLayers.Projection('EPSG:28992'));

The catch here was that the definition for the EPSG:28992 projection needed to be accurate to be able to put the marker at the correct possition. In our case the +to_wgs part of the definition mattered a lot!

Proj4js.defs['EPSG:28992'] = '+title=Amersfoort / RD New EPSG:28992
                              +proj=sterea
                              +lat_0=52.15616055555555 +lon_0=5.38763888888889
                              +k=1.0
                              +x_0=155000 +y_0=463000
                              +ellps=bessel
                              +towgs84=565.,49.9,465.8,-0.409,0.36,-1.869,4.08
                              +units=m'; 

Showing current position

Another application was developed to show the position of cars on the map of the Netherlands. With the right configuration of Geoserver and a WFS Layer in OpenLayers this was not that difficult. At least not at first sight.

Problem here was that the positions where update in the database every 5 seconds so the view from the browser needed to be refreshed every 5 seconds also.

layer.refresh({force: true});

Not much code for such functionality, and it worked. At least for about 30 minutes until the browser crashed on me. What?

It seemed that the browser was taking more and more memory after each refresh. So somewhere there was a Memory Leak. It turned out to be because of the way OpenLayers draws its elements on the page. It is done by DOM manipulation (element.appendChild and element.removeChild).

The problem is that the removeChild method removes the child from the visual space but keeps a reference in memory. This reference can be used to retrieve the element again and show it (because you accidently removed it??). This memory space is only garbage collected when a new page is opened. But nowadays Ajax like applications don’t open a new page so hence the memory leak.

For IE (the browser used) this is by design so hope that a fix will be around in newer versions is zero.

Luckily a workaround exists which helps to limit the Memory Leak.

Workaround

OpenLayers.Util.discardElement = function(element) {
  var garbageBin = document.getElementById('IELeakGarbageBin');
  if (!garbageBin) {
    garbageBin = document.createElement('DIV');
    garbageBin.id = 'IELeakGarbageBin';
    garbageBin.style.display = 'none';
    document.body.appendChild(garbageBin);
  }

  // move the element to the garbage bin
  try {
    garbageBin.appendChild(element);
  }
  catch(e) {
    //do nothing
  }
  garbageBin.innerHTML = '';

  if (element.removeNode) element.removeNode(false);
  //delete element;
};

Distance calculation

On to the next challenge. One of the requirements was that all markers within a certain distance of a region would be visible. Imagine such a region consisting of 40.000+ points. So to determine if a point lies within say 10kilometers from that region is not an easy calculation. The Oracle Spatial database has a function to do this but to do this for all markers was not performing fast enough (remember that the client was refreshed every 5 seconds!)

The solution here was to change the requirement a bit and to show only the markers within a certain distance of a point. This was an easy calculation and could be done with supplying a query to the geoserver request

complex types

Next problem was the use of complex geometries. oracle spatial has support for curves and stuff, but geoserver can only read simple geometry types like points, linestrings, polygons, multilinestrings, multipolygons.

Lucky i was to find that within the definition of a complex geometry also is encapsulated a simple geometry. All that was needed was a little code to extract that simple geometry from the complex one and voila:

Heres the code snippet:

        if ((L == 0) && (TT == 01) && (point != null) && (elemInfo == null)) {
            // Single Point Type Optimization
            coords = SDO.coordinates(gf.getCoordinateSequenceFactory(), GTYPE,
                    point);
            elemInfo = new int[] { 1, ETYPE.POINT, 1 };
        } else {
            int element = 0;
            int etype = ETYPE(elemInfo, element);
            if (etype == 0) {
               // complex type, search for encapsulated simpletype (with etype != 0)
               int startpointCoordinates = 0;

               // look for a simple one
               while (etype == 0) {
                   element++;
                   etype = ETYPE(elemInfo, element);
                   startpointCoordinates = STARTING_OFFSET(elemInfo, element);
               }

               // if we found the simple fallback, read it
               if (etype != -1) {
                   int ol = ordinates.length;
                   int elemsToCopy = ol - (startpointCoordinates - 1);
                   double[] newOrdinates = new double[elemsToCopy];
                   System.arraycopy(ordinates, startpointCoordinates - 1, newOrdinates, 0, elemsToCopy);
                   elemInfo = new int[] { 1, etype, INTERPRETATION(elemInfo, element) };
                   ordinates = newOrdinates;
               }
            }
            coords = SDO.coordinates(gf.getCoordinateSequenceFactory(), GTYPE,
                     ordinates);
        }

        return create(gf, GTYPE, SRID, elemInfo, 0, coords, -1);

So far a few of the challenges i came across using OpenSource GIS software. What I’ve noticed is that both the openLayers and Geoserver community are very helpful and that both projects are maturing every release.

Let’s see what the future of this project brings when we need to have the client be available for at least 8 hours in a row. I guess the browser (at least IE) is out of the question. Looking forward to investigate the possibilities of a flex client application.

Monday 8th of December 2008 was the start of a week full of information. After attending the complete conference (including the University sessions) last year I felt it would be a good thing to do the same this year.

The university sessions give me a change to get more in-depth knowledge on some of the subjects. For this first day I had chosen the sessions on Scala and Java Power Tools.

Scala

The session about Scala has got me really interested in this (for me) new language. The combination of Object Oriented and Functional programming, the tight integration with Java (in the end its all Java bytecode) and the conciseness makes it worth my while to have a closer look. As Ted Neward mentioned in his talk: ‘Today start with Scala to experiment and prototype, so next year you’ll have the advantage of Scala knowledge to be able to use it in production systems.’

Java Power Tools

The Java Power Tools title got me a little confused. I expected a session about some of the tools in the jdk that can help you to program more powerful, but instead it was a session about automated builds, code coverage and quality metrics. These things are common practice in the projects done by Xebia. The only things that where new to me where the part of the session about the Hamcrest Asserts and the Behavior Driven Development framework easyB.

Hamcrest Asserts

JUnit 4.4 introduces the assertThat method in which assertions can be expressed in a more readable and understandable manner. Instead of saying

  assertEquals(10,x)

you can say

  assertThat(x, is(10))

which to english speaking people makes a lot more sense.

easyB

By using a specification based Domain Specific Language, easyb aims to enable executable, yet readable documentation.

Describing a zipcodevalidation can be done like this:

  given "an invalid zip code",

{ invalidzipcode = "221o1" }

and "given the zipcodevalidator is initialized",

{ zipvalidate = new ZipCodeValidator() }

when "validate is invoked with the invalid zip code",

{ value = zipvalidate.validate(invalidzipcode) }

then "the validator instance should return false",

{ value.shouldBe false }

Later in the week a session about the Feel of Scala described a similar approach used in the ScalaTest framework. With this approach the existence of a key in a map can be expressed like:

map should have key 'a'// because everything in Scala is a method call and an expression
// this results in a java like call
map.should(have).key('a')
// the should method of course doesn't exist so this is a potential compiler error.
// Scala's use of implicit conversions allows you to create a wrapper method
// which add's the should method to a class.

For the second day of the University sessions I chose Java Performance Tooling and Filthy Rich Clients beyond Java.

Java Performance Tooling

This session was a mix of IBM promotional talk about IBM Monitoring and Diagnostic Tools – Health Center (which looks good by the way). And a general introduction on how to analyze performance problems and the tools you can use with that. Overall the session gave me a better understanding of memory management and garbage collection and the difficulties you run into when investigating performance issues.

The most interesting statement was from Kirk Pepperdine saying: The purpose of using Queue’s is, not getting work done!

Filthy Rich Clients (beyond java)

This session was an introduction to Android and Flex and how these two can create Rich Clients. The Filthy Rich Client book uses Java examples and with this session Chet Haase an Romain Guy wanted to show some examples in Android and Flex.

Android

Android is not a Java platform but it uses the Java programming language. The Java bytecode is compiled into dex bytecode which runs on the Dalvik VM. This VM interprets the dex bytecode, but there’s no JIT compilation yet. Furthermore the garbage collection is of a simple mark-sweep type. Every application runs in it’s own process with it’s own virtual machine and as a separate user for security reasons.

From the little coding examples I’ve seen it feels a lot like Swing programming but I will need to further investigate that to be sure. It was a nice first introduction but for me personally not something to dive into.

Flex

I already played a bit with Flex since i got the first introduction on last year’s conference, so the only new things were about the work being done on Flex 4 and some interesting examples about the Pixelbender and Filtering possibilities in Flex.

Some of the other Flex sessions this week make me look forward to the release of Flex 4 and it’s improved possibilities.

Conference

In this section I will highlight some of the sessions I attended.

JavaFX

As JavaFX 1.0 was released just before the conference there were a lot of sessions about this subject. Also the keynote of the first conference day was mainly about JavaFX (As well as a really commercial talk by IBM on how they implemented RFID to monitor the conference)

The most interesting parts of the presentation were about the plugins available for Adobe Photoshop and Illustrator to export design and timelines to a format that can be imported into your JavaFX application. The demonstration showed a somewhat plug and play like experience – Really cool stuff!

The biggest problem in my opinion is the non existence of a UI Toolkit with widgets to support building enterprise applications with a rich user experience. Which leads me to believe that flex is still miles ahead.

Concurrency

The new Fork-Join framework introduced in Java 7 can help in programming for multiple cores. With the increasing amount of cores in the next few years it won’t be enough anymore to separate on the user request level. This means applications must decompose into small units of work that can be run separately. Most promising candidates are sorting and searching.

The new ParallelArray class abstracts the parallelism from you.

The fork-join framework reduces contention for the work queue by using a technique known as work stealing. Each worker thread has its own work queue, which is implemented using a double-ended queue. When a task forks a new thread, it pushes it onto the head of its own deque. When a task executes a join operation with another task that has not yet completed, rather than sleeping until the target task is complete (as Thread.join() would), it pops another task off the head of its deque and executes that. In the event the thread’s task queue is empty, it then tries to steal another task off the tail of another thread’s deque.

Future of RIA

With the introduction of Flash player 10 a lot of new possibilities are available for Flex applications like: 3d (or at least 2.5D), the Pixelbender, Flash text engine and the text layout framework and Sound.

With AIR 1.5 released all the Flash player 10 features are also available on the desktop including encrypted local storage.

The new Flex release (codename Gumbo) will support Flash Catalyst (the integration between Photoshop/Illustrator en FlexBuilder), a new States Model, a new Animation Model and more…

Also Adobe is working with springsource to get a better integration between Spring and Flex.

An interesting demo was shown about the Model Driven Architecture capabilities of the new LiveCycleDataServices suite.

Later in the week James Ward showed a very impressive demo where a PDF was created from a Flex application without losing any functionality of the flex app. All rich user experience stuff was still working (a flash movie can be easily integrated in the PDF) and also the update of the data could still be performed. Wow! Speechless.

Effective Java

This keynote talk was all about the second edition of the book with rewritten examples to reflect the current best practices. The most interesting part was about Enums.

First thing was about adding a constructor to the enum type to be able to add extra information about the enum element. This can help you avoid using the ordinal function which is a bad practice anyway since changing the enum can influence your application if the ordinal function is used.

public Enum Ensemble {

SOLO(1), DUET(2), TRIO(3), QUARTET(4), QUINTET(5), SEXTET(6), SEPTET(7),
OCTET(8), DOUBLE_QUARTET(8), NONET(9), DECTET(10), TRIPLE_QUARTET(12);  private final int numberOfMusicians;

Ensemble(int size) {
numberOfMusicians = size;
}

public int getNumberOfMusicians() {
return numberOfMusicians;
}

}

Second was about singletons which have the need to be serialized. Using the readResolve method to return the instance variable is open for carefully crafted attacks. This way it becomes possible two have multiple instances of that singleton if not all fields are transient.

The easiest way to create a singleton is by using an Enum class. All enum’s are serializable and the JVM guarantees Enum is a singleton.

public Enum Elvis {
INSTANCE;
private String[] favoriteSongs = { "Hound Dog", "Heartbreak Hotel" };

public String[] getFavoriteSongs() {
return favoriteSongs;
}
}

Java 7

Java 7 will contain:

• modularity
• JSR 292: VM support for dynamic languages
• JSR 203: more new IO API’s
• some small language changes like safe rethrows, better type inference and multi catch.
• JSR 296: Swing app framework
• forward-port of jdk6_u10 features
• Swing updates
• SCTP (Stream Control Transmission)
• SDP (Sockets direct protocol)
• upgrade classloader architecture
• Unicode 5 support
• Hotspot runtime compile enhancements
• G1 garbage collection
• JSR 308: Annotations on java types
• JSR 166: Concurrency and Collection updates like the fork-join framework.

But that’s about it. And the most disappointing about this talk was that Java 7 is expected in 2010. Yes, that’s more then a year from now!

Be Smart

This session by Ivar Jacobson was the most entertaining session. In front of a fullyfilled room, he had a very good talk about Being Smart (subtitled: What they don’t teach you about software at school). After OO, UML, Rup & CMMI and XP it’s al about Scrum now. All are good but none is all you need.

Smart means, things should be done as simple as possible (but not simpler). Agile provides simple and lightweight starting points, being Smart is about knowing when to break the rules. Smart is Agile++.

It was very good to hear all the things we already practice within Xebia being called Smart.

Conclusion

This Devoxx conference was a good conference overall. The minor thing was that there was nothing really new that was mentioned. For me the items I will work on in the near future are Scala and the new Flex release. JavaFX will be a good thing to have a look at after that I guess.

This blog will be about the second part: Me experiencing India

Off we go

The most difficult part of the trip started at the airport immediately. Saying goodbye to my wife and kids (32, 1 and 3 years old) has never been one of my favorites and this being the first time leaving them for a two week period , didn’t make it easier. The rest of the journey was smooth and I had a lot of time thinking about what to expect. Preparation wasn’t that good so I decided to keep an open mind (the easiest way to prepare).

In the Xebia Knowledge Exchange we had before my departure a discussion was raised about the difficulties that can arise in the communication so I took that as one of the goals to investigate during my stay.

Arrival

The arrival at New Delhi airport was kind of what I expected. A small ‘organized’ chaos of people gathering around the newly arrived waving the signs with the names of all foreigners they should pick up. Our driver (I forgot to mention my colleague who joined me) had a nice sign with the Xebia name on it, so finding him was not too difficult. Leaving the parking lot was a much bigger challenge. Living in the Netherlands made me get used to some kind of structure (either voluntary or enforced by government rules), but structure was completely lacking at the parking lot of the airport. The driver had parked his car on a place I would have called being ‘under construction’.

The arrival at the guesthouse was fine and after a good night sleep and some breakfast a taxi picked us up to go to the office. The charm of the guesthouse being a little bit away from the office building is that you could experience the traffic (which I was warned for). To give a small clue, imagine yourself a highway, 4 lanes (one way). In the Netherlands there will be at most 4 cars driving alongside. In India they manage to get 6 to 7 cars driving together. This is of course until a bus stops at the second lane to let the approximately 50 people in who are waiting on the first lane (of the highway!). And I must not forget to mention the people crossing the total of 8 lanes, risking their lives, to hop on this bus. Oh and what about the cows which need to be left alone and take a rest on one of the lanes. Do you get the picture?

Now I understand why some time ago, one of the team members sent an email that he couldn’t come to the office because of the rain. With my simple Dutch mind I thought that because of rain you could be 15 minutes late, but not being able to come at all?? Now I know that it can be impossible to come to the office. On a normal day the traffic is chaotic enough. Add some rain, and keep in mind that the roads are not tilted like in the Netherlands to let the water flow away and ZOAB (a kind of open road that let’s the water go through) is not used, so you cannot see if the road is still there. This all together makes a simple trip to the office a big effort.

The arrival at the Xebia India office felt the same as arriving in any Xebia office. They did a very nice job to give the place a same look and feel.

The biggest difference compared to the Dutch Xebia offices is that in Xebia India the whole company is working in the same office. This results in a great group of people working together. Other differences come more from a cultural perspective. To me it felt a little awkward having a guard at the front door of the office who opened the door every time I needed to go to the toilet. And having a sort of liftoperator pushing the buttons for me. But from a cultural point this is a way to help people earn a living which sounds good.

Working in the India Office

Working in the India office proved to be some kind of a struggle. Not only due to my stomach not able to handle the spiciness of the food (or was it the Kingfisher) but also the ‘problems’ with the internet connection speed. I call it problems because I relate it to the speed at our Dutch office. Even at home I’m able to get a much faster connection for a much smaller amount of money. Working with 30 people in the office all using exchange, jira, confluence, skype, svn etc. is not helping the maximum speed provided by the ISP. All day I’m having the idea of driving with the handbrake still on.

This explains some of the communication struggles we face in working together. Refreshing jira will take minutes so you cannot quickly see if somebody else has assigned that issue which you want to work on already.

Communication

A few of the struggles in communication I already mentioned. The other thing is that the connection is not always so reliable. This is mainly caused by the many power outages that occur on a daily basis. Inside the office a power backup is present but for the connection we depend on external parties. Also the meetings are more difficult if the connection is not optimal. In a standup meeting things are relatively simple because one person speaks at a time, but with a planning meeting when a discussion is launched into space it becomes very hard to follow the people further away from the mic and interruption is almost impossible because of delay issues.

To improve the planning meeting we used a good conference mic and we tried to "plan the planning meeting". In previous meetings the whole team was waiting for the connection to be set up and the right sound settings were enabled. In the planning of the planning meeting two persons (preferably one in NL and one in India) prepare the connection and sound settings and make sure both parties are looking at the same view of the product backlog. This way the meeting has a clear start. Also the environment for the meeting is important, as with the conference mic everybody needs to be in range of the mic as much as possible to be able to follow a discussion more easily.

Last but not least the availability of the product owner or at least someone who is able to answer questions about user stories and issues, is a must for everybody in the team but especially for the distributed part of the team. That access is hard by not being able to contact directly and the time difference. The start of the day in India takes place in the terribly early morning here so no explanation can be done in that period.

Getting help

The biggest difference in culture I experienced in the weekend. An Indian colleague took me with him the whole weekend to visit the Taj Mahal and all kind of inspiring places in Delhi while his family was over to visit him for the weekend only. That made me wonder why in the Netherlands we seem to plan our time full with work related and private appointments. So full that the only thing I have done for our Indian colleagues who visited the Netherlands was point them the right way to the train station. And yes I think here you can more easily manage on your own, but that’s no real excuse.

Value

I really got to know the Indian colleagues as a close group of people really enjoying the openness of the Xebia culture, working in agile projects. I got a better understanding of the way they work and live, and they absolutely earned my great respect for the hospitality they showed during my stay in India.

For me it has been a really valuable experience and I hope I have shared a bit of that with you. So I’m looking forward to my next visit which I will plan when my family has recovered from my absence.