Picking the right ORM-framework

I didn’t do any extensive research, just some googling. A framework called Scweery popped up. The name Scweery sounded very nice and it has a Twitter account with a funny avatar, so why look any further? Well, it seems the project isn’t that active anymore (no updates for over a year and that’s a long time with Scala).
Finally Stackoverflow came to the rescue. I found a question called Examples of Scala database persistence . I picked Squeryl, the syntax looked nice and there was useful documentation. I’m not sure whether I picked the right framework, but it survived my initial selection, so let’s find out.

Setting up the environment

For this article I assume the following:

-You have version 2.9.0 of Scala installed
-Git is on your system
-SBT is available on the command line (I used version 0.7.7)

I believe the links above explain everything pretty well, so let’s get started!

Getting started

I used the ‘minimal example’ project to get started:

git clone https://github.com/pbrant/squeryl_sbt_minimal_example
cd squeryl_sbt_minimal_example
squeryl_sbt_minimal_example>sbt
....
[info]    using ExampleProject with sbt 0.7.7 and Scala 2.9.0
squeryl_sbt_minimal_example>update
.....
squeryl_sbt_minimal_example>compile
squeryl_sbt_minimal_example>run

I updated the Scala version to 2.9.0. This can be done by changing the version number project/build/build.properties. For this article it doesn’t matter much, but for your real project this newer version has lots of improvements.

That went pretty smooth. Let’s rip everything out and use Postgres. Open the file Main.scala (/src/main/scala/code). First remove drop, create and printDdl. We don’t want our valuable production database to be destroyed!
For my database testing I use my good old movie database. With ancient column names (in Dutch) and the ID column is a String. This might sound like a bad idea, but real databases are like this and I found out it’s a good way to test a new framework.

Schema definition

The first section to change is Library extends Schema. This is the section where the tables are defined. My table is called movies.

val movies = table[Movies]

When running the example (with some additional code, so don’t try it yet) I got the following error:
org.postgresql.util.PSQLException: ERROR: relation “Movies” does not exist

It seems the queries are executed with the table names surrounded with double quotes, which means case sensitive. To solve this append (“MOVIES”) to the table definition. Now we can also solve the problem that a class is singular (unlike a database table). Now I can call the class Movie instead of Movies:

val movies = table[Movie]("MOVIES")

Table definition

Our next step is the table definition. (In Dutch, “jaar” means “year” and “naam” means “name”) This time we have to use annotations to use other column names (to get rid of typos and underscores):

class Movie(
 @Column("imdb_id")
 val imdbId: String,
 @Column("jaar")
 val year: Int,
 @Column("naam_film")
 val name: String,
 @Column("genre")
 val genres: String
)

Note that I used val for the columns, when the columns are mutable it’s allowed to use var.

Connecting to Postgresql

To connect to to a Postgres database replace the H2 dependency in /project/build/ExampleProject.scala with

val pg = "postgresql" % "postgresql" % "9.0-801.jdbc4"

Now go to the main method in Main.scala and replace the driver and connection parameters:

Class.forName("org.postgresql.Driver");
SessionFactory.concreteFactory = Some(() =>
 Session.create(
 java.sql.DriverManager.getConnection("jdbc:postgresql://localhost/moviedb", "bill", "s3cr3tp455w0rdth4t15nt5053cr3t4nym0r3n0w"),
 new PostgreSqlAdapter)
)

At a first glance it just looks like JDBC, but the Some(()=>..) notation might be a bit odd. This is by-name-parameter, just see it as a function that creates a session whenever needed.

A select * from

The final section is doing the actual query:

inTransaction {
 val movies = from(Library.movie)(select(_))
 for (movie <- movies){
 println (movie.name)
 }
}

To execute queries you need a transaction (also for read only queries). A transaction is available after the initialization of the SessionFactory.concreteFactory. To run code in a transaction wrap it in an inTransaction block (you can also use ‘transaction’, this always starts a new transaction, inTransaction only does when there is no transaction in progress).

The first line in the inTransaction block does a select * from movie.

Scala might be a bit confusing when you’re new to it, like me. The scaladocs might help. From can be found here and select here.

The statement should read like this: from Library.movie apply each row to the select function. The second set of parameters for from (the so called Queryable) can be expanded with where and order by clauses. For more information read the page about selects on the Squeryl website.

The for loop of course prints all the movie names.

‘Advanced’ queries

The previous paragraph was just a simple query you’ll probably never execute in real life. So let’s try something a little bit more difficult:

inTransaction {
 val movies = from(Library.movie)(s=>
 where(s.year === 1994)
 select(s) )
 println(movies)
 for (movie <- movies){
 println (movie.name)
 }
}

The operator === is a method that is added to Int (this construction is called ‘implicit’) to create the actual where-statement in SQL. I also added a println(movies), this prints the actual SQL-statement, very useful!

Conclusion

I really like Squeryl, it was easy to set up and the basic documentation is sufficient. Of course a next step is inserting and adding more expressions (and for me, handling Postgis geometry objects).
I hope this article gives you a head start.

Sources

-Stackoverflow – Examples of Scala database persistence
-Scala installation
-Git
-Sbt setup
-Scweery
-Squeryl
-Squeryl on github
-Squeryl google group

In this article I will show you how to collect the data, parse and convert it to a usable format and show it on a map. This is just a first prototype and proving my theory will probably take a lot more time which I don’t have because I’m in traffic jams all the time .
If you don’t like all the technical mumbo jumbo, just scroll to the maps, they’re cool to look at.

Preparation

Since I drive a Peugeot I had some trouble with GPS reception on my phone, because of the infrared coating on the windscreen. To solve this problem I used an external Bluetooth antenna and told Android to use that antenna. Bluetooth GPS is a nice solution for that.
Other reasons to use an external antenna are that they are much more accurate, save the battery of your phone and that you can place the antenna where the reception is best (the black rectangle behind your rear view mirror is a perfect place and this is one of the reasons this rectangle exists).

When you have trouble firing up the Bluetooth GPS try disabling and enabling the GPS and/or Bluetooth on your phone. I had some trouble with my car kit, but when you connect the GPS first and then the car kit you can still use both. You have to check the connection with the GPS when someone calls you, a while ago I lost my GPS connection when my colleague Mischa called me to tell me that the left lane was much faster

For the logging I used GPS Logger for Android, a simple application that writes .gpx and/or .kml to the /mnt/sdcard/GPSLogger directory. You can even set the log frequency. Note that the logging is not done a the set interval, it’s just a suggestion. The antenna spits out a stream of data, when your phone is busy it just skips data. So be aware of this when processing the data and don’t hardcode the interval, but use the timestamp from the antenna.
There are other logging applications available, some even draw a trace. I prefer a very fast application, I don’t want to wait for an application to start up when I start my car.

Parsing the data

I decided to use the .gpx data and skip .kml. I already have some experience with gpx, but kml shouldn’t be a problem. The fields I’m interested in are the coordinates (x,y) and the timestamp. Course and speed are also very useful, but that’s derived data for which I’m going to use GeoTools. Since GPS doesn’t really know when you’re stationary you get a lot of garbage at low speeds. For now it’s good enough to assume that you’re stationary when the speed drops below 5 kph. Another thing is that when you remove trackpoints from the log the speed field becomes invalid, I want the average speed on a leg of the track, not at the trackpoint itself.

To parse the data I used Scala. This sample chaper explains how to parse xml in Scala. The structure of a trackpoint in the gpx is as follows:

<trkpt lat="52.131231666666665" lon="5.011798333333333">
 <ele>49.4</ele>
 <course>329.36</course>
 <speed>29.8172</speed>
 <src>gps</src>
 <time>2010-12-23T07:23:21+01:00</time>
</trkpt>

I’m interested in lat (y), lon (x) and time. With these fields I can calculate the speed and create my map.

Displaying speed

I thought it would be nice to display each leg of the track in a color that represents the speed. To display the maps I use qGis, a desktop application to display GIS-data. QGis can load all kinds of formats (shapefiles, simple .csv’s and there are lots of plugins available).
After trying to generate shapefiles with GeoTools for a while I decided points were good enough (instead of legs of a track). After all I’m interested in the time spent at traffic lights.
In the nightly versions of qGis 1.7 there is a new and improved delimited text plugin. The guys at Linfiniti even made an instruction video for it! With version 1.7 it is possible to import well known text. This immediately solves my problem, unfortunately it is not yet available for Mac Os yet.

So, for now I have to show a map with points:

To import the data I converted the parsed xml to .csv first. A snippet from the output:

lonX, latY, speed, dateTime
5.074956666666667, 52.09170666666667, 37.266029160336, 2011-01-04T09:07:35.000+01:00
5.07473, 52.091948333333335, 31.436951983284487, 2011-01-04T09:07:38.000+01:00
5.0746, 52.092083333333335, 26.498677863536287, 2011-01-04T09:07:40.000+01:00

Calculating the speed

To calculate the speed I used GeoTools in Scala:

 /**
 * @param start (starting point (lon, lat))
 * @param destination (end point (lon, lat))
 * @return Distance in meters
 */
def calcDistance(start: (Double, Double), destination: (Double, Double)): Double = {
  val geodeticCalculator = new GeodeticCalculator();
  geodeticCalculator.setStartingGeographicPoint(start._1, start._2);
  geodeticCalculator.setDestinationGeographicPoint(destination._1, destination._2);

  // To be computed only once for ever for a given GeodeticCalculator
  val ellipsoid = geodeticCalculator.getEllipsoid();
  val sourceUnit = ellipsoid.getAxisUnit();
  val targetUnit = SI.METER;

  val converterTo = sourceUnit.getConverterTo(targetUnit);

  // To apply after each distance calculation
  converterTo.convert(geodeticCalculator.getOrthodromicDistance());
}

Speed is calculated the following way: (distanceInMeters / deltaMillis) * 3600

Delimited text layer

The csv file can be imported into qGis with the delimited text plugin. QGis is smart enough to see that lonX and latY are the x and y coordinates. When the layer is imported you will see a map with points, all in the same color. To change the colors double click on the layer in the legend on the left. Pick symbology and graduated symbol as legend type. I increased the number of classes to 7 and hit the classify button. QGis generates some kind of gradient for you, which can be changed easily. With save and load style you can save and load these settings for later use.

OpenStreetMap raster layer (georeferencing)

To create a base layer I downloaded a map from OpenStreetMap. When you go to the site and have a nice view of a map you want to use in qGis you can click on the Export tab. This opens a dock in which you can export an image. Remember the coordinates, you need them in qGis!

In qGis open the Georeferencer plugin. Click the first button on the toolbar and open the image you saved. Zoom in to the upper left corner of the image and add dot (the three dots in the toolbar) and enter the coordinates you just saved. Do the same for the lower right corner. Click the green play button and click on the folder right of output raster. You now have a georeferenced tiff image which can be added as a raster layer in qGis.

Of course this map has only one zoom level, which looks a bit distorted (see the screenshot in the next paragraph).

I’m not sure if the plugins I mentioned are installed by default, if they’re not and you have trouble please let me know, then I’ll add a section to this blog. Usually plugins can be enabled/disabled und the menu plugins, manage plugins.

Displaying stationary time

To display the stationary time I used the interval the speed drops below 5 kph. This time is included as a field in the csv, which we can use in qGis.
In qGis create a delimited text layer. At symbology pick the traffic light symbol and increase its size (I used 5 points). Now add a label. ‘Field containing label’ should be the field that has the delay in it. The label displays on top of the symbol by default, below right is a better place. Increase the font size and color (ie. 24 and black) and add a buffer (2.0, white).
Now there will appear traffic light symbols on the map with a delay next to it. Of course we also want to display a trace of the driven route. This can be achieved with a gpx layer. Go to the menu Plugins, Gps, Gps tools.
Click on the tab Load GPX…, browse for the gpx and select the Tracks at Feature types.
At the properties the line size and color can be adjusted.

The purple lines represent two different trips. The trip I analyzed starts at the traffic light with number 17, via 18 and ending with 15. On OpenStreetMap:  Exit 17 on highway A12.

Conclusion/Future

I always thought I was waiting minutes in front of traffic lights, it’s not that bad. It rarely happens that you have to wait more than one minute. Guess I have to work on my impatience

I display the stationary time and this doesn’t necessarily have to be at a traffic light. So a stop symbol might be more suitable

The next step is ‘snapping’ the gps data to the OpenStreetMap roads and calculate the average speed. I’m also working on getting nicer looking maps (now the maps are distorted and only suitable for one zoom level), it probably has something to do with coordinate systems and I might give OpenLayers a shot.

Androgeoid is a very nice site with reviews of Android apps, it really helped me to pick the right tools for the job.

Sources

GPS Logger for Android
Bluetooth GPS
Androgeoid
GeoTools
OpenStreetMap
Delimted text plugin video
Herding XML in Scala

Introduction

I want to skip the introduction because a lot has been written about websockets and you probably want to get started as soon as possible. Check out the sources section for more information. For now you only have to know that it’s fast, it has almost no overhead, keeps a permanent connection open (so the server can push data to the client, no polling required!) and best of all: it’s very easy.

Installation

When you’re using maven 2+ you don’t have to download anything. Just start a maven war project and add the jetty-maven-plugin (we used version 7.2.1.v20101111), add a dependency to jetty-websocket (inside the project element) and you’re ready to get started.
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-websocket</artifactId>
<version>7.2.1.v20101111</version>
</dependency>

Dissecting the WebSocketChatServlet example

A sample application is included with Jetty. On this blog this sample is explained and in the end we used it to see what was happening and of course see some communication between client and server. After that we stripped everything and started over again to see what was actually happening.

Creating your own servlet

Since I used Struts and Spring MVC in the past and always copied the servlet configuration it was a while ago I added my own servlet to web.xml, so here is a litte snippet to help you (this probably was one of the most difficult parts ):
<servlet>
<servlet-name>hw</servlet-name>
<servlet-class>com.xebia.ws.HelloWorldServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>

<servlet-mapping>
<servlet-name>hw</servlet-name>
<url-pattern>hw.do</url-pattern>
</servlet-mapping>
The HelloWorldServlet extends WebSocketServlet and you have to implement the doWebSocketConnect method. This method returns a WebSocket. For now it’s enough to do a new HelloWorldWebSocket.

HelloWorldWebSocket implements WebSocket. The onConnect method receives an Outbound object. With this object it is possible to send data back to the client. The Outbound object is a field on the WebSocket object so we can use it in the onMessage method.

The onMessage method:

public void onMessage(byte opcode, String data) {
    outbound.sendMessage("pong");
}

When we send data in javascript (with send()) the server will respond with “pong”.

The Javascript side

The Javascript is also very simple. I added the jQuery plugin to add data to a div with id bla, but your free to do it the old-fashioned way.

var ws=new WebSocket("ws://10.0.0.230:8080/hw.do");
ws.onmessage = onMessage;
ws.onopen = onOpen;

function onMessage(m) {
    $("#bla").append(m.data + "<br/>");
}

function onOpen() {
    console.log("connection open")
}

function send(){
console.log("sending ping");
    ws.send("ping");
}

For testing we’re using Safari, it has a really nice console (available under the menu Develop, Show Error Console). By typing send() in the console we’re sending a ping which will respond with a pong. The response is logged to the div with id bla.

It’s even possible to write a simple websocket client with nothing more than the console:

var m=new WebSocket("ws://10.0.0.230:8080")
m.onmessage=function(m){ console.log(m.data); };

This piece of code prints the payload of any received message. Sending is done with m.send(“data to send”);

Problems

Thinking http is the protocol of websockets

We used the browser to test whether the WebSocketServlet was working. Of course websockets uses the ws:// or ws:// prefix and we used http://. I made this mistake a couple of times.

WebSocket is not ready

We tried to send data over the socket, but got an INVALID_STATE_ERR. It appeared the WebSocket isn’t ready immediately after creating. After the WebSocket is ready the onopen function is called and then it’s ready to send and receive. The readyState field on a WebSocket is a useful field to check the status of the socket (0=connecting, 1=open, 2=closing, 3=closed). So don’t try to do anything with the websocket before onopen is called.

WebSocket disabled in Firefox 4

While we were coding away Dan North paid us a visit. In a very cool improvisation style presentation he pointed us to an article about some security issues with websockets.

Conclusion

Websockets is a very cool technology and I’m very happy it was a subject of the techrally. Feel free to ask any questions. At least 5 people at Xebia are now websockets experts and even more listened to two presentations about it.

Sources

http://blogs.webtide.com/gregw/entry/jetty_websocket_server
http://dev.w3.org/html5/websockets/
http://blog.dannorth.net/
http://www.slideshare.net/spagalloco/introduction-to-websockets
http://docs.codehaus.org/display/JETTY/Maven+Jetty+Plugin
http://hacks.mozilla.org/2010/12/websockets-disabled-in-firefox-4/
HTML5 Series: Part 6: WebSockets

I have no experience with MongoDB at all, it just sounds cool and people tell me it’s cool. That’s enough for me to do some research. I recently finished my bachelor thesis about collecting gps points and discovering running routes from those points. The main bottleneck in my application was counting the number of points within a certain radius of a point. You basically have to check all the points to see whether they’re inside the radius. Let’s say you have 10^6 points (and I have), then (10^6)*(10^6) steps are needed.
I advised to use a geospatial index with PostGIS in my conclusion, but that’s not really new and cool, so let’s give it a shot with MongoDB.

What is MongoDB?

“MongoDB (from ‘humongous’) is a scalable, high-performance, open source, schema-free, document-oriented database.”
Well that’s quite a proper description I think. The idea is to insert documents (JSON objects) into a collection (comparable to a table in an RDBMS) in a database. Queries don’t use SQL, they are a bit comparable with QBE.

Inserting and querying can be done with a plethora of languages. There is official support for C, C++, Java, Javascript, Perl, PHP, Python and Ruby. The community supports another bunch of languages which are listed here).
Installing MonogDB is very easy. Just extract a file and start the database with mongod. Since it is different amongst the various operating systems I have to refer to the quickstart guide (and I can’t explain it any clearer anyways).
With the command mongo a CLI is started. Let’s insert an object and find that object.
> db.foo.save( { a : 1 } )
> db.foo.find()
The first line of code above saves a document in the collection foo with key/value a/1. When you do a find (second line) the same object is displayed, now with a unique id (the key _id is used for this)

There is a lot to tell about MongoDB, but many people did that already, so let’s focus on the geospatial indexing in combination with Scala.

How a geospatial index works in MongoDB

A geospatial index is just like a normal database index, but in two dimensions (or sometimes three when you account for height). The index in MongoDB is a so called grid index (there are other indexes like r-tree and quadtree, but that might be too much info for a blog).
The geo-index of MongoDB encodes a geohash on top of a standard MongoDB b-tree. Geohashing (not to be confused with geocaching) is a way to divide a coordinate system into hierarchical buckets of grid shape.
It basically divides an area in two for every bit and so on. An elaborate description can be found on the wikipedia page.
A drawback of geohashing is the wrapping (-180 and 179 are very close for example), this can be solved by doing a grid search after the initial scan. It’s a bit comparable by how google maps loads the map images, it starts with the center of the viewport and then the surrounding areas are downloaded.

Our Laapersveld office in a geohash: u179jbzdexru

Using the java driver (in Scala)

Since the Scala driver gave me too much trouble I tried the java driver. It can be downloaded from a github repo. Just include the jar in your Scala project and you’re ready to go.

To query everything from collection foo, the following code is needed:
val mongo = new Mongo( "localhost" , 27017 )
val db = mongo.getDB( "test" )
val coll = db.getCollection("foo")
val cursor = db.find()
while (cursor.hasNext) {
println(cursor.next)
}
A big warning from the API which is important enough to repeat is that toArray and length on cursor convert the cursor to an array, which can be a very expensive operation, especially on humongous databases!

Inserting the data

Let’s take some arbritary data from my data set:
{ "routeId" : "9327014.tcx", "id" : "1105409", "loc" : { "x" : 5.1400218, "y" : 52.0614276 } }
As you can see I use a nested object, we need this for the index later on and it’s to show you how to do nested objects (plain objects are just too easy ).
val doc = new BasicDBObject()
doc.put("routeId", "9327014.tcx")
doc.put("id", "1105409")
val loc=new BasicDBObject()
loc.put("x", 5.1400218)
loc.put("y", 52.0614276)
doc.put("loc", loc)
coll.insert(doc) /* coll is a DBCollection */
That’s all! No transactions to commit or connections to close.

Creating the index

Creating the index is quite easy, it can be done from the java driver or on the command line.

The objects in the database look like this:
{ "routeId" : "activity_123456.tcx", "id" : " 1105402", "loc" : { "x" : 5.1415501, "y" : 52.0616409 } }
The x,y coordinates are wrapped in a loc key, because that’s the way MongoDB likes it. The first two columns in the loc object are treated as x and y coordinates.
On the command line execute db.foo.ensureIndex( { loc: “2d” } ) and your index is created.
Creating the index in Scala is also possible:
val index=new BasicDBObject()
index.put("loc","2d")
coll.ensureIndex(index) /* coll is a DBCollection */
Forgetting to create an index and doing spatial queries might work, but sometimes I got strange crashes (and bad performance). So check this before throwing MongoDB out of your window

Querying the data and comparing the speed

It’s now possible to do a geospatial query:
db.foo.find( { loc : { $near : [5.14,52.06] } } )
This will display all the points near 5.14, 52.06 sorted by distance from that point. Note that by default a maximum of 100 points is returned! This can be solved by appending the query with .limit(500), for a maximum of 500 results. Note that the higher the number the slower your query gets, so be careful with increasing this number.

I wanted to count the number of points within a certain radius. For this you need to add bounds criteria to the query:
db.foo.find({"loc" : {"$within" : {"$center" : [[5.14,52.06], 0.0005]}}})
Note that the bounds criteria were added two weeks ago (version 1.3.4) and I’m not sure whether this already is ready for production. It works fine for me, but you might be a bit careful when using it for real.

In Scala building the query is a bit more verbose. You have to construct an object and query a collection for that object (note that this is the same as the first query):
val query = new BasicDBObject()
val loc = new BasicDBObject()
val near = new BasicDBList()
near.put( "0", 5.15 )
near.put( "1", 52.069 )
loc.put("near", near)
query.put("loc",loc)
/* coll is a DBCollection */
coll.find(query).limit(50)

I wanted to do a speed comparison, but I decided it isn’t that important yet. Since MongoDB and especially the geo-index is improving every day it’s enough to say that it’s a lot faster on large data sets (for me > 400,000 points). The small sets ( < 200,000 ) have comparable speed.

Conclusion

MongoDB is cool and the people who told me that were right. Besides cool I really loved it. I got my database running within a few minutes. The documentation is very well written and updated regularly (the new features in the geo-index were updated in the documentation instantly and that’s quite unique).
Scamongo was a bit of a disappointment (probably also thanks to my perseverance and wanting quick results ), it looks promising so I might give it another shot within a few months.
The geo-index is really cool. When I started playing around a few weeks ago some features were missing, but within days these features were added (and I had to rewrite some parts of this blog ). Kudo’s to the MongoDB guys for giving me cool stuff, updating it regularly and even without breaking!

Sources

MongoDB
Geohashing
MongoDB Java Driver
MongoDB Java Driver API

The prerequisites for a java callout are simple. Any static method in a jar-file will do. You can even return your home made objects (they don’t even have to be serializable). One downside is that you cannot do anything with these objects on the bus, they’re just an object with an id. You can however pass these objects to another java callout and use them as your regular java objects.

The static method you want to call should be in a jar file that is loaded on the alsb (so it is NOT a library in WebLogic). It is allowed to put dependencies in this jar file, so in theory everything is possible. But idea is to keep it small and simple, otherwise go with EJB’s and JMS with a library or application on WebLogic.

It is very inconvenient when you just can pass the objects around. There are however some type of objects that can be handled on the bus. Those objects  are primitives, String, BigDecimal, BigInteger, byte[], String[] and XmlObject. XmlObject is an object provided by the Apache Xml Beans framework and is basically an xml-representation of a java object. I know that converting an object to xml should happen on the bus and it would be nice when a java object can be passed to the bus, but this just isn’t the fact and this is the cleanest solution I came up with.

Now that you know the theory let’s get our hands dirty:

Preparation

Create a class with a static method and pacakge it in a jar file. I added the following maven dependency to an empty pom.xml:

<dependencies>
 <dependency>
 <groupId>xmlbeans</groupId>
 <artifactId>xmlbeans</artifactId>
 <version>1.0.3</version>
 <scope>compile</scope>
 </dependency>
</dependencies>

This will give you the Apache Xml Beans library at compile time. This library is excluded in the packaged jar because it lives in WebLogic.

The java class :

public class CalloutDemo {
    public static String getString(String input) {
		System.out.println("input: " + input);
		return "Buy a bicycle from " + input + "?";
    }

    public static XmlObject getCode(String input) throws XmlException {
		System.out.println("input: " + input);

		String xmlString = String.format("<root><mainCode>%s</mainCode><subCode>%s</subCode></root>", "mc", input);
		XmlObject instance = XmlObject.Factory.parse(xmlString);

		return instance;
    }

    public static Code getCode() {
		return new Code("hoofdCode", "subCode");
    }

    public static void printCode(Code code) {
		System.out.println("code: " + code);
    }
}

This class returns a main code and a sub code for an input String or just a new String based on an input String.
Apache Xml Beans is ued to return ‘complex objects’. In this example I’ll return a String and an XmlObject.

When you return a normal java object (ie. a domain object) and print this on the bus you will see something like this:
<con:java-content ref=”jcid:5333178f:11fa2a0451c:-7fda” xmlns:con=”http://www.bea.com/wli/sb/context”/>
It’s just a pointer to an object. When you pass this pointer to another java callout it becomes an object again. It’s not possible to do transformations or even view the java object on the bus, you can only pass it around (and probably only on the same VM instance)

No rocket science, just for the example. When you understand the code displayed above it’s time to put everything on the bus. But first run the command ‘mvn package’ to compile the project and create a jar.

On the bus

Open the AquaLogic service bus console (I’m using the web interface by the way).
Create a session with the create button. Then create a new project in the Project Explorer (I called mine callout-test)
Click on callout-test in the project tree and create a jar resource.
Give the resource a name (ie. callout-test-jar) and upload the jar maven delivered to you (from the target folder).
When the jar is uploaded it’s possible to browse the jar. You will see your brand new class with al its methods.
Now create a new Proxy Service (callout-ps), just give it a name, click last and then save.
Edit the message flow and add a pipeline pair. Add a stage at the request pipeline. Edit that stage and add an action:
Message processing, Java callout.
When you click on <method> it’s possible to pick your jar and choose a method. The first method I want to show is  public static java.lang.String getString(java.lang.String). For the input expression enter xs:string(‘Harrie’).
In real life this is a value somewhere from the bus of course.
Give the result String the name result, we can use that value later on.
Click on the icon left of Java Callout and add an Alert action. At the expression enter $result and leave the summary empty.
Make sure you save everything and activate the session.

Putting it to the test

When everything is saved it’s time to test the proxy service. Find the proxy service in your project explorer and click on the debug icon (the little bug icon in the action colum). Since we created an any xml-service it’s okay enter anything in the payload field as long as it’s xml. Click the execute button and you’ll get the response you entered in the payload field. Of course we want to see what happened internally.
Click on home and then pipeline alerts. There probably is a new alert named ‘AquaLogic Service Bus Alert’ with a timestamp of a few seconds ago. Click on this alert and you’ll see the new String in the Description field:

Since we used System.out.println (I know it’s evil, but I wanted no dependencies in my little jar) the server console shows:
input: Harrie

The next step

Now of course it’s time to use the ‘advanced’ xml.
Re-open the flow of the proxy service, click on Stage1, click Edit, click on the getString method and change it to getCode (the one that returns XmlObject).
Enter xs:string(‘Arie’) at the expression. Since the method is returning xml we can print the part of the message we want. Enter $result/subCode at the Alert message (or $result/subCode/text() if you just want the text without xml tags, but alerting the text you just entered isn’t cool )
Save and activate and repeat the steps from ‘Putting it to the test’

It’s easy to predict what happens now.

Unfortunately we didn’t find use for the java callout at our project, but it’s pretty cool nonetheless.

Disclaimer

I used the term alsb and AquaLogic Service Bus in this article, it’s probably called Oracle Service Bus. But everyone I know still talks about the AquaLogic Service bus.
I started this article a while ago and since Oracle removed edocs.bea.com I can’t show you the sources where I got my information from.

How to install WebLogic on a Mac is explained in this blog:
Running WebLogic Server 10g Release 3 on a Mac
Make sure you also read the comments on the blog when you get errors, they contain valuable information that’ll save you a lot of time.

Go to the Oracle download page and pick the HP-UX Itanium version. I picked the HP-UX version because it’s in fact a generic version.

Run the installer on the command line (don’t run the jar directly!):

java -Xmx1024m -jar -Dos.name=unix osb103_wls103_generic.jar

Since the WebLogic installer had some memory problems on a Mac I added the -Xmx1024m paramter. The os.name parameter is added to prevent the “Insufficient disk space!” error.

Follow the default installation instructions. When the screen “Choose Products and Components” appears I also included the Service Bus Examples, they might come in handy.

After installing the OSB you have to extend your domain with the Oracle Service Bus (or just create a new domain).
Start the Configuration Wizard ($BEA_HOME/wlserver_10.3/common/bin/config.sh) and pick Extend an existing WebLogic domain (assuming you already have a domain running)
Search for the directory where your domain is running (ie. $BEA_HOME/user_projects/domain/base_domain)

I chose to replace all components when asked for and picked the default choice at every wizard screen.

The changes you made to startWebLogic.sh (see the blog by Yusuke Yamamoto I mentioned earlier) also have to be reapplied.

When you start your domain (with startWebLogic.sh) you should be able to access the Service Bus Console on http://localhost:7001/sbconsole

To check whether things are working I followed the first tutorial.

Sources

• Running WebLogic Server 10g Release 3 on a Mac
• Introduction to the BEA AquaLogic Service Bus Tutorials

My unit test is extending AbstractTransactionalJUnit4SpringContextTests and is annotated with @RunWith(SpringJUnit4ClassRunner.class) and @ContextConfiguration(locations = { … })

The code I’m about to show should be put after initialzing the MockServletContext. I recommend using a method annotated with @Before where you do the initialization (I called mine setUpMockServletContext)
Since the unit tests run before the Spring config files are put in the WEB-INF directory, you have to tell Stripes where to find the configuration files:

context.addInitParameter("contextConfigLocation",
"classpath:dataSourceContext.xml\nclasspath:daoContext.xml\nclasspath:applicationContext.xml");

Stripes isn’t smart enough (yet?) to read the @ContextConfiguration annotation, but this file will probably included somewhere in a parent-unit test, so it isn’t a big deal.

The final step is telling Spring to initialize its context with your MockServletContext (note that this is a Stripes MockServletContext, not the one used by Spring!):

ContextLoaderListener springContextLoader = new ContextLoaderListener();
springContextLoader.contextInitialized(new ServletContextEvent(context));

That’s all there is to it. Let’s hope google picks up this blog and other people can handle this tiny problem quicker than I did.

Sources

* Unit testing with Spring
* Unit testing Stripes
* Using Spring with Stripes
* Stripes mailing list

Door B:
-id: 2
-size: small
-color: blue and green

Now lets select all red, green or blue doors, that will be two doors right?

Criteria crit = session.createCriteria(Door.class);
crit.createAlias("colors", "c");
crit.add(Restrictions.in("c.name", new Object[]{ "red" , "green" , "blue" }));

Because of the colors alias an INNER JOIN is used and the result might not be what you expect.
The size of the list is 4, 4 Door objects are returned! That is very strange. When you have a look at the SQL you will see the following query:
select this_.id as id2_1_, this_.size as size2_1_, colors3_.DOOR_ID as DOOR1_, c1_.id as COLOR2_, c1_.id as id3_0_, c1_.name as name3_0_
from doors this_ inner join DOORS_COLORS colors3_ on this_.id=colors3_.DOOR_ID inner join colors c1_ on colors3_.COLOR_ID=c1_.id
where c1_.name in (?, ?, ?)
Translated to readable SQL:
SELECT * FROM doors d
INNER JOIN doors_colors dc on d.id=dc.door_id
INNER JOIN colors c ON c.id=dc.color_id
WHERE c.name in (?, ?, ?)
This query will indeed return 4 results, but we’re selecting Door objects, not SQL-rows. Looking at the java code I expected Door object, not query result rows wrapped in a Door object
To solve this problem you can add a ResultTransformer to the crit-object:

crit.setResultTransformer(CriteriaSpecification.DISTINCT_ROOT_ENTITY);

Now only two doors are selected, problem solved.
Well, not so fast, let’s see what happens with many doors. Our database now has 16 doors. When we select the red, green and blue doors again, 11 results appears. Suppose we want to add paging to the results, because it’s just too many doors to handle at a time. 5 doors per page is enough.
We can achieve it by adding the following two lines:

crit.setFirstResult(0);
crit.setMaxResults(5);

11 doors found, 5 results per page and display the results starting from the first result (which is 0-indexed).
How many results do you expect on the first page?
Not the 3 results returned probably. The ResultTransformer is still active by the way. What happens is that the paging is applied before the duplicate entities are filtered out.

When you start thinking in SQL you might come up with the idea to use a subquery. This is exactly what we’re are going to do in Hibernate.
A subquery in pseudocode:
select * from doors d where d.id in (select id from doors, colors where colors.name in (‘red’,'green’,'blue’))
As you can see the subquery only selects the id column and in my previous example a complete Door object was selected.
To select other columns than the columns in the Entity object you can use Projections. With multiple columns use the ProjectionList object. For now we need an id-projection, which has is own method:

crit.setProjection(Projections.id());

Now a list with int’s is selected. This result is the input for the next query. You can create Criteria objects without a Hibernate Session for later use. These objects are called DetachedCriteria.

When we rename the crit to dc and create it as a DetachedCriteria you will have the following lines of code:

DetachedCriteria dc = DetachedCriteria.forClass(Door.class);
dc.createAlias( "colors" , "c");
dc.add(Restrictions.in( "c.name" , new Object[]{ "red" , "green" , "blue" }));
dc.setResultTransformer(CriteriaSpecification.DISTINCT_ROOT_ENTITY);
dc.setProjection(Projections.id());

The next step is creating an outer query around the previous query.

Criteria outer = session.createCriteria(Door.class);
outer.add(Subqueries.propertyIn("id", dc));
outer.setFirstResult(0);
outer.setMaxResults(5);

With Subqueries.propertyIn you can add the previous query. When you add paging and sorting to the new outer query object all your problems are solved. You can even get rid of the ResultTransformer and get the same result. When you’re tuning performance it might be a nice test to see whether it makes any difference to do the DISTINCT before you pass the results to the outer query.

How you can solve DISTINCT problemns is explained reasonably in the Hibernate FAQ:
Hibernate does not return distinct results for a query with outer join fetching enabled for a collection (even if I use the distinct keyword)?

The lesson I learned with Hibernate is that you still have to know SQL and don’t trust the results blindly. If you don’t know why you have to use DISTINCT and how a JOIN presents its results you’ll get lost very soon.

With the DetachedCriteria object you can split your queries in a very readable way and when using projections, the SQL under the hood only selects the needed columns, which improves performance.

sources

http://floledermann.blogspot.com/2007/10/solving-hibernate-criterias-distinct.html
http://www.hibernate.org/117.html#A12

It’s is quite easy to get normal url’s in Wicket. In this article I’ll show you several solutions, the first two with plain Wicket and the final solution is with wicketstuff-annotations.

Recently I read an article called “Wicket Creating RESTful URLs”. It’s a useful article but it still didn’t feel right. It should be easier and in the right place.
In Stripes for example you can just put @UrlBinding(“/helloWorld.action”) on your ActionBean and it works. That’s what I want!

After some searching I found wicketstuff-annotation. Wicketstuff provides a large set of additions to Wicket to make it even better. wicketstuff-annotation is a small part of it.

Start from scratch

To demonstrate wsa (Wicketstuff annotation) I will let Maven 2 generate a simple Wicket application from scratch. To generate this application visit http://wicket.apache.org/quickstart.html

Execute this statement on the command line and you’ll have a working Wicket application within seconds.

Add a link to HomePage.html so you can see what url’s Wicket generates and how to improve them:

<wicket:link>
<a href="HomePage2.html?page=133&paragraph=25">
Link to wicket document HomePage2.html
</a>
</wicket:link>

Make a copy of HomePage.java and HomePage.html and call the files HomePage2.java and HomePage2.html. Change the href to HomePage2.html in the file HomePage2.html to HomePage.html. When you redeploy the application you will see the unreadable url I was talking about earlier:

http://localhost:8080
/wicketstuff-test/?wicket:bookmarkablePage=:com.xebia.HomePage2&paragraph=25&page=133

I added the parameters page and paragraph to show that even these parameters can be made more readable.

Another bad thing is that we expose our package structure. End users don’t care in what package the page is and you shouldn’t give potential hackers any information.

The plain Wicket solution

The simplest solution is mounting a package to a path. We can map all of the package com.xebia to http://localhost:8080/wicketstuff-test/web for example. Unfortunately we cannot map to the root (/) of the application. To mount a package add the following line to the constructor of WicketApplication:

mount("web",PackageName.forClass(HomePage.class));

The ugly url now looks much better:

http://localhost:8080/wicketstuff-test/web/HomePage2/paragraph/25/page/133/

Even the parameters page and paragraph are improved.

Suppose you don’t want to expose the name of your WebPage (HomePage2) and also hide the parameter names. This can be achieved by mounting a MixedParamurlCodingStrategy to the application:

public WicketApplication() {

    String[] parameterNames = new String[] { "page", "paragraph"};

    MixedParamUrlCodingStrategy mixedParamUrlCodingStrategy =
new MixedParamUrlCodingStrategy(

            "h2", HomePage2.class, parameterNames);

    mount(mixedParamUrlCodingStrategy);

}

The problem with this solution is that you have to pass all parameters and put all class files in the Application file. The url structure and page parameters are related to the page, not the application, so they should be on the WebPage. But the result is a very nice url without  having to add an extra dependency to wsa:

http://localhost:8080/wicketstuff-test/h2/133/25/

The Wicketstuff Annotation solution

To add wsa suppport to your application you have to add a dependency to your pom.xml:

<dependency>

     <groupId>org.wicketstuff</groupId>

     <artifactId>wicketstuff-annotation</artifactId>

     <version>1.1</version>

 </dependency>

The next step is clearing the old solution from the constructor of WicketApplication.
Add the folllowing line:
new AnnotatedMountScanner().scanPackage(“com.xebia”).mount(this);
This line will search the package com.xebia for annotations to mount url’s. In a production environment it is probably smarter to limit this to the package where all the WebPage classes are (ie. com.xebia.web).

Now we can finally add annotations to HomePage2 :

@MountPath(path = "h2")

@MountMixedParam(parameterNames = { "page", "paragraph" })

public class HomePage2 extends WebPage {

...

}

This works exactly the same as before, only in the right place.

There are several variations on the second annotation :

@MountPath (and no second annotation)

http://localhost:8080/wicketstuff-test/h2/paragraph/25/page/133/

@MountPath and @MountMixedParam

http://localhost:8080/wicketstuff-test/h2/133/25/

@MountPath and @MountQueryString

http://localhost:8080/wicketstuff-test/h2?paragraph=25&page=133

There are several other encoding strategies you can use. On the wsa javadoc there isn’t much information. When you want to know more you should consult the Wicket javadocs and look for the package org.wicket.request.target.coding. The names used here are similar to the ones for wsa.

Conclusion

Wicketstuff annotation is a very nice addition to Wicket. It helps you to encode your url’s in the right place. The only small drawback is the dependency on Spring core, such a small library as wsa should work without Spring (not that Spring is a bad thing, I just don’t like many dependencies). So don’t be alarmed when you see a Spring jar pop up between your libraries.
Also be aware that the mount(path, packageName) solution is the safest when you don’t want to expose the package structure. When you forget to add annotations you end up with the ugly url’s and still exposing the package structure. I tried to add multiple mount points, but this wasn’t possible.

Sources

http://wicketstuff.org/confluence/display/STUFFWIKI/wicketstuff-annotation

http://css.dzone.com/news/wicket-creating-restful-urls

http://cwiki.apache.org/WICKET/url-coding-strategies.html