My search led me to Dispatch, which is a highly compact Scala DSL wrapper around Apache’s reliable HttpClient. This DSL, however, is not very well documented and rather hard to decipher due to it’s heavy usage of symbolic method names but nevertheless highly appealing when understood.

In this blog I’ll decipher it for you and show how easy it is to test restful services with mere oneliners.

My Prerequisites

Before we dive into Dispatch’s mysterious DSL let’s look at the prerequisites I had for testing my restful service. These will most likely apply for many other rest services as well:

• Support for CRUD operations with http’s POST, GET, PUT and DELETE methods
• Support for XML and Json as input and output payloads
• Support for reading http status codes to test results of erroneous responses like 404 (Not Found)
• Support for security aspects such as https and basic/digest authentication

Begin with the End in Mind

The goal of this blog is it to create a RestClientHelper that offers a set of generic restful client methods and explain how they’re implemented using Dispatch. With these methods in place we will be able to call and therefore test restful service APIs very easily.

The RestClientHelper will be a trait that offers the following client side CRUD methods:

trait RestClientHelper {

  //Create (POST)
  def create[T](target: String, reqBody: String)(fromRespStr: String => T): T
  def create[T](target: String, reqBody: Elem)(fromRespXml: Elem => T): T 

  //Read (GET)
  def query[T](target: String, params: Seq[(String, String)])( fromRespStr: String => T): (Int, Option[T])
  def query[T](target: String, params: Seq[(String, String)])( fromRespXml: Elem => T): (Int, Option[T]) 

  //Update (PUT)
  def update[T](target: String, reqBody: String)(fromRespStr: String => T): T
  def update[T](target: String, reqBody: Elem)(fromRespXml: Elem => T): T
  def update[T](target: String, reqBody: String): Int
  def update[T](target: String, reqBody: Elem): Int 

  //Delete (DELETE)
  def delete(target: String) :Int
}

As you can see, most methods consist of two parameter lists. The first parameter list represents the rest call input such as target uri and request body or parameter map. The second parameter list is a function that converts the output of a rest service into the desired type. The return type of these methods are of type T, Int or Tuple2[Int, Option[T]]. The Int always represents the http status code, the Option[T] represents the converted output if the service call was successful.

How to use the RestClientHelper

To get an understanding how the RestClientHelper trait is supposed to be used let’s look at an example. Assuming we have a domain object, Person, that provides serialization and deserialization methods for xml a CRUD call sequence with the RestClientHelper trait would look as follows:

val person = Person("John Doe")

//POST /add.xml
val p = create("add.xml",person.toXml){Person.fromXml(_)}
assert(p.id != None)

//GET /search.xml?q=John+Doe
val (status, personOpt) = query("search.xml", Seq("q" -> "John Doe")) { Person.fromXml(_) }
assert(status == 200)

//PUT /update.xml
val changedPerson = p.copy(_.name = "John Who")
val status2  = update("update.xml", changedPerson.toXml)
assert(status2 == 200)

//DELETE /delete/4
val status3 = delete("delete/" + changedPerson.id.get)
assert(status3 == 200)

//GET /search.xml?q=John+Who
1
val (status4, personOpt2) = query("search.xml", Seq("q" -> "John Who")) { Person.fromXml(_) }
assert(status4 == 404)
assert(personOpt2 == None)

Looking under the hood

Now let’s dig a little deeper and discover how one of these rest client helper methods are actually implemented using Dispatch.

def create[T](target: String, reqBody: String)(fromRespStr: String => T): T = {
  executer(:/(host, port) / target << reqBody >- { fromRespStr })
}

Well, this is all that is needed for invoking restful service by means of a POST request. Being new to Dispatch it can help untangling this highly compact DSL for a better understanding.

The core classes participating in an http call as shown above are the following:

• A HttpExecuter (dispatch.HttpExecuter) responsible for executing the http request. A common implementation would be dispatch.Http. In this example it is respresented by the executer object.
• A Request (dispatch.Request) and its DSL wrapper dispatch.RequestVerbs, which are responsible for creating a specific kind of request, e.g. a gzip http post with content-type application/x-www-form-urlencoded.
  The class RequestVerbs contains DSL-ish symbolic methods names, which mostly start with a ‘<’ character indicating that something is ‘added’ to a request (left arrow).
  The method << is one of those. It adds a request body to the request and therefore automatically transforms it into a POST request.
• A Handler (dispatch.Handler) and its DSL wrapper dispatch.HandlerVerbs, which is responsible for handling the result returned by the http call, such as converting it to xml or json.
  The class HandlerVerbs contains as well DSL-ish symbolic method names, which mostly start with a ‘>’ character indicating that something is done with the ‘output’ of the http call (right arrow).
  The method >- is one of those. It converts the response received as a String into the desired type.
• For both, the Request and Handler, implicit conversions allows them to be converted into their corresponding DSL wrapper RequestVerbs or HandlerVerbs respectively.

With those key classes in mind we can rewrite the above http call in a more verbose manner in order to understand the parts the DSL is made of:

def create[T](target: String, reqBody: String)(fromRespStr: String => T): T = {
    val emptyReq:Request = :/(host, port)

    //implicit conversion applied explicitely
    val reqVerbs:RequestVerbs = Request.toRequestVerbs(emptyReq) 

    val configuredReq:Request = reqVerbs./(target).<<(reqBody)

    //implicit conversion applied explicitely
    val handlerVerbs:HandlerVerbs = Request.toHandlerVerbs(configuredReq)

    //The http executer always needs to be called with a handler
    val handler:Handler[T] = handlerVerbs.>-(fromRespStr)

    executer.apply(handler)
}

The executer needs further explanation. As said, Dispatch requires you to use an executer to finally execute an http request. Dispatch offers several types of executers, such as thread and non-thread safe ones. A thread-safe one that makes use of a shared connection pool could be provided as follows:

  protected lazy val executer = new Http with thread.Safety

The non-thread counterpart could be accessed like this:

  protected def executer = new Http

Various other executers are available. The ones above will most likely be sufficient for most purposes.

To summarize we can conclude that most of the power of Dispatch from a usage point of view lies within the RequestVerbs and HandlerVerbs classes, which allow us to compose a request and decompose its response the way we want it. There is an excellent periodic table of dispatch operators available that lists all the possible Verb methods with a short description.

…and now the nitty-gritty details

With the knowledge we have gained about Dispatch the remaining method implementations of the RestClientHelper should be straight forward. I’ll show each implementation with a short explanation of the most notable Dispatch methods:

Create (POST) with String in- and output

 def create[T](target: String, reqBody: String)(fromRespStr: String => T): T = {
    executer(:/(host, port).POST /  target << reqBody >- { fromRespStr })
  }

• RequestVars: POST
  Create a POST request (the POST method is optional because of the ‘<<<' method, which forces the request to be a Post)
• RequestVars: <<(body:String)
  Post the given String value with text/plain content-type
• RequestVars: >-[T](block:(String) => T)
  Convert the response body from a String to the desired type

Create (POST) with XML in- and output

  def create[T](target: String, reqBody: Elem)(fromRespXml: Elem => T): T = {
    executer(:/(host, port) /  target << reqBody.toString <> { fromRespXml })
  }

• RequestVerbs: <<(body:String)
  Post the given String value with text/plain content-type
• HandlerVerbs: <>[T](block:(Elem) => T)
  Convert the response body from an Elem to the desired type

Read (GET)
The read method is probably the most intriguing one from the whole series. In order to understand it fully, I will provide a detailed explanation:

 def query[T](target: String, params: Seq[(String, String)])(fromRespStr: String => T): (Int, Option[T]) = {
    executer x (:/(host, port) /  target <<? params >:> identity) {
      case (200, _, Some(entity), _) => {
     	   val respBodyStr = fromInputStream(entity.getContent()).getLines.mkString
           (200, Some(fromRespStr(respBodyStr)))
      }
      case (status, _, _, _) => (status, None)
    }
 }

Now let’s look at the most important Dispatch ingredients for this GET request:

Input processing
The method <<? from RequestVerbs: <<?(params:Traversable[(String, String)]) simply adds query parameters to the request url.

Output processing
To process the query result we are interested in two things: the response code and response body. Both are returned in the form of a Tuple2 (Int, Option[T]), where Int represents the status code and Option[T] the converted object in case the query yielded a result. The question is how can we retrieve both of them, since Dispatch does not offer a symbolic method that does that for us.

First let’s take a closer look at what is actually called: Instead of providing the executer with a handler directly we use the method ‘x’.

  executer x (…)

Why is that? By calling the executer’s apply method (that’s what finally happens under the hood) the handler block is only called in case the response status code is 200 – 204. In all other cases an Exception is thrown. So if we want to intercept all response codes we need to use the Executer’s x[T](hanlder:Handler[T]) method, which executes the handler no matter which response code is returned.

The next question is what kind of handler is passed to the executer’s x method? If we explicitly assigned the first part of the DSL construct to a handler it would look as follows:

  val intermediateHandler = (:/(host, port) /  target <<? params >:> identity)

The ‘magic’ lies in the >:> identity construct. According to the API the HandlerVerb method >:> accepts a function, by which the response headers can be processed. By passing Scala’s Predef identity method to the >:> method, nothing is processed but the resulting handler of type Handler[Map[String, Set[String]]] is returned.

As said, the handler above does not process the result itself. The real processing is done by the case statements, which are the last arguments that are passed to the DSL construct. How do we need to interpret that? For a better understanding, the last statement could be rewritten as follows:

 val realHandler = intermediateHandler.apply {
    case (200, _, Some(entity), _) => {
       val respBodyStr = fromInputStream(entity.getContent()).getLines.mkString
       (200, Some(fromRespStr(respBodyStr)))
    }
    case (status, _, _, _) => (status, None)
 }
 executer.x(realHandler)

What happens is that we construct another handler by calling the apply method of the previously created one. The apply method of all the handlers accept a function with the following signature:

    apply(next:(Int, HttpResponse, Option[HttpEntity], () => T) => R)

The Int stands for the response code, the HttpResponse and Option[HttpEntity] give us access to the underlying Apache HttpClient implementation and the argument () => T represents the transformation function, which transforms the response to type T, the type of the Handler itself.

As you probably know a case statement IS a function (PartialFunction), which can be chained and passed – even if chained – as a single PartialFunction to a method. So by providing the handler’s apply method with the above case statements we’re able to define in detail how the low-level result of the http call needs to be processed. In our case this means: retrieve and convert the response body when the response code is 200, otherwise simply return the response code.

Even though this construct might look rather complicated it is a very powerful way to access and process the raw result that the http call returns.

Update (PUT) with String in- and output

def update[T](target: String, reqBody: String)(fromRespStr: String => T): T = {
  executer(:/(host, port).PUT /  target <<< reqBody >- { fromRespStr })
}

• RequestVerbs: PUT
  Create a PUT request (the PUT method is optional because the <<< method forces the request to be a PUT)
• RequestVerbs: <<<(body:String)
  Put the given String value with text/plain content-type
• HandlerVerbs: >-[T](block:(String) => T)
  Convert the response body from a String to the desired type

Update (PUT) with XML in- and output

 def update[T](target: String, reqBody: Elem)(fromRespXml: Elem => T): T = {
  executer(:/(host, port) /  target <<< reqBody.toString <> { fromRespXml })
 }

• RequestVerbs: <<<(body:String)
  Put the given String value with text/plain content-type
• HandlerVerbs: <>[T](block:(Elem) => T)
  Convert the response body from an Elem to the desired type

Update (PUT) without response body

def update[T](target: String, reqBody: String): Int = {
    executer x ((:/(host, port) /  target <<< reqBody >:> identity) {
      case (status, _, _, _) => status
    })
}

If we want to send a PUT without expecting a response body but are still interested in the response code we use the same construct as described above in the read sample. The only difference compared to the read example is that we do not retrieve the response body, but simply return the response code.

Delete (DELETE)

def delete(target: String):Int = {
   executer x ((:/(host, port)).DELETE /  target >:> identity)  {
      case (status, _, _, _) => status
   }
}

The delete method does not process a response body. In order to know whether the deletion was successful we are however interested in the response code. Therefore, we again use the construct as in the read example.

Security & Authentication

Finally, let’s explain what we needed to do if our restful service was secured with https and/or basic/digest authentication.

For https the RequestVars’ secure method should be called on the request, for authentication the as(“username”, “pwd”) method. Therefore, a secure create (POST) call that uses basic authentication would look as follows:

 def create[T](target: String, reqBody: String)(fromRespStr: String => T): T = {
    executer(:/(host, port).secure.as(“user”, “pwd”) /  target << reqBody >- { fromRespStr })
 }

In case security combined with authentication is used it is self evident that the RestClientHelper trait would provide a method that returns a preconfigured request in order to by DRY:

trait RestClientHelper {
 val host: String
 val port: Int
 protected def username:String = "unkown"
 protected def pwd:String = "unkown"
 private def req = :/(host, port).secure.as(username, pwd)

 def create[T](target: String, reqBody: String)(fromRespStr: String => T): T = {
    http(req / target << reqBody >- { fromRespStr })
 }
...

Wait, there is more

Dispatch offers various other features that are not mentioned in this blog. E.g.: Dispatch can directly convert a response body into a Lift Json object by means of the ># method (a good example is provided on the Dispatch site), it can execute http requests in a background thread, there are various HandlerVerbs and RequestVers, which we have not covered yet like using gzip mode, chaining handlers etc. to mention some of them. (check the periodic table for further reference). For testing restful services however, the stuff covered in this blog should suffice.

The RestClientHelper ready to use

To conclude this blog as follows the source of the RestClientHelper trait, which hopefully makes restful service testing for you a piece of cake and fun (after having Dispatch set up)!

import scala.io.Source._
import scala.xml._
import org.apache.http._
import dispatch._

trait RestClientHelper {
  val host: String
  val port: Int
  val contextRoot: String
  protected val ssl = false
  protected val username = "notdefined"
  protected val pwd = "notdefined"
  protected val executer = new Http with thread.Safety
  private def req = {
    val req = :/(host, port).as(username, pwd) / contextRoot
    if (ssl) req.secure else req
  }

  def query[T](target: String, params: Seq[(String, String)])(fromRespStr: String => T): (Int, Option[T]) = {
    executer x (req / target <<? params >:> identity) {
      case (200, _, Some(entity), _) => {
        val respStr = fromInputStream(entity.getContent()).getLines.mkString
        (200, Some(fromRespStr(respStr)))
      }
      case (status, _, _, _) => (status, None)
    }
  }
  def query[T](target: String)(fromRespStr: String => T): (Int, Option[T]) = {
    query(target, List())(fromRespStr)
  }
  def queryXml[T](target: String)(fromRespXml: Elem => T): (Int, Option[T]) = {
    queryXml(target, List())(fromRespXml)
  }
  def queryXml[T](target: String, params: Seq[(String, String)])(fromRespXml: Elem => T): (Int, Option[T]) = {
    val convertOutput = (s: String) => fromRespXml(XML.loadString(s))
    query(target, params)(convertOutput(_))
  }

  def create[T](target: String, reqBody: String)(fromRespStr: String => T): T = {
    executer(req / target << reqBody >- { fromRespStr })
  }
  def create[T](target: String, reqBody: Elem)(fromRespXml: Elem => T): T = {
    executer(req / target << reqBody.toString <> { fromRespXml })
  }

  def update[T](target: String, reqBody: String)(fromRespStr: String => T): T = {
    executer(req / target <<< reqBody >- { fromRespStr })
  }
  def update[T](target: String, reqBody: Elem)(fromRespXml: Elem => T): T = {
    executer(req / target <<< reqBody.toString <> { fromRespXml })
  }
  def update[T](target: String, reqBody: String): Int = {
    executer x ((req / target <<< reqBody >:> identity) {
      case (status, _, _, _) => status
    })
  }
  def update[T](target: String, reqBody: Elem): Int = {
    update(target, reqBody.toString)
  }

  def delete(target: String): Int = {
    executer x ((req).DELETE / target >:> identity) {
      case (status, _, _, _) => status
    }
  }
}

Possible usage in an object or any other class you want the RestClientHelper to be mixed in:

object MyRestClientHelper extends RestClientHelper {
    val host = "rest.service.host"
    val port = 443
    val contextRoot = "v1"
    override val ssl = true
    override val username = "myusername"
    override val pwd = "secret"
}

I’m still trying to get rid of old habits, to shake of my winter hide, so to speak, and create some real Scala in stead of ScaVa (i.e. Java with a Scala syntax). If you’re interested you can bear witness to my struggle on GitHub (ShoppingList on GitHub). This story came about because I asked some colleagues for help. We ended up rewriting loops in several ways.
What I’ll show you is some alternatives to classic loops over collections.

The code is attached here. The example shows how to summarize items in a list. The objects in the list are all instances of class Stuff, a simple value container for a String and an Int. The idea is to summarize Stuffs with the same key to produce a new list:

  val items = List(Stuff("A", 1), Stuff("A", 2), Stuff("B", 3), Stuff("B", 4))

should become

  val expectedResult = List(Stuff("A", 3), Stuff("B", 7))

where Stuff is a simple case class with two attributes:

case class Stuff(val label: String, val number: Int)

Because I’m a long time Java programmer, my first solution looks like this:

  def classicSum = {
    var result: List[Stuff] = List()
    for (item <- items) {
      if (result.size > 0 && item.label == result.head.label) {
        result = Stuff(result.head.label, result.head.number + item.number) :: result.drop(1)
      } else {
        result = item :: result
      }
    }
    assertEquals(expectedResult, result.reverse)
  }
 

Good old looping and a var to collect the result. This works, but it doesn’t say what’s happening. All you see is a loop and some fancy list processing. Note the call to drop(1) allowing you to replace the head of the list with a new instance. I like the drop function and I’ve used it in other programs but here it just obfuscates matters.

The next solution is to go recursive. If you check out the version of ShoppingList as of October/November 2011, you’ll find lots of recursion. My goal at the time was to jam recursion into my head by banning all other forms of looping.
Applied to the list of Stuff instances and the problem at hand, the result is tragic:

  def recursiveSum = {
    @tailrec def sum(listOfPairs: List[Stuff], result: List[Stuff]): List[Stuff] = {
      listOfPairs match {
        case Nil => result
        case head :: tail =>
          {
            val currentHead = if (result.size == 0) { Stuff(head.label, 0) } else { result.head }
            val newResult =
              if (currentHead.label == head.label) {
                Stuff(currentHead.label, currentHead.number + head.number) :: result.drop(1)
              } else { head :: result }
            sum(tail, newResult)
          }
      }
    }
    val result = sum(items, List())
    assertEquals(expectedResult, result.reverse)
  }
 

My next version leverages Scala’s collections to drastically reduce the amount of code. The first attempt passes the test but it is cryptic. In the spirit of the red-green-refactor mantra I’ll show it to you anyway:

  def sumByGroupSolution1 = {
    val groupedByLabel = items.groupBy(_.label)
    val result = groupedByLabel map { t => Stuff(t._1, t._2 map { _.number } sum) }
    assertEquals(expectedResult, result)
  }
 

There! Go and parse that if you need to change this code sometime next year (note: having worked with this code for some time now while writing this blog, I must admit that it grows on you; after a while it doesn’t hurt all that much, sort of like a new pair of shoes). This cryptic piece of mal-ware illustrates the use of three powerfull methods named groupBy, map and sum.
groupBy is in a sense comparable to SQLs group by clause. The Scala version takes a collection and returns a Map. The map’s key is the element used to group by (Stuff.label in my case). The value is a List of elements that have the same key. In this case the items collection contains Stuff instances. I’ve grouped by the label field so the type of groupedByLabel is Map[String, List[Stuff]]. On this Map we have to apply a sum function to add up all Stuff instances in the List of Stuffs. Sum however works on Int’s, so before sum can be applied we have to extract the number field of each Stuff instance. This is done by a map: map {_.number}. The mapping is applied to each Stuff instance in the list of Stuffs that was returned by groupedByLabel.

By factoring out part of the code on line 3 in the example above, we can improve readability:

  def sumByGroupSolution2 = {
	val stuffsGroupedByLabel = items.groupBy(_.label)
    def sumOfStuffsWithTheSameLabel(stuffs: List[Stuff]): Int = stuffs map { _.number } sum
    val result = stuffsGroupedByLabel map { t => Stuff(t._1, sumOfStuffsWithTheSameLabel(t._2)) }
    assertEquals(expectedResult, result)
  }
 

I think this is better because now the result of part of the computation is named. Naming a thing makes it easier to see the algorithm.

The next solution uses more intermediate results. The results of each map or sum operation are stored in a variable that gets a meaningful name. This clarifies the meaning of intermediate results but the amount of code grows.

  @Test
  def sumByGroupSolution5 = {
    val calcSumOfStuff = (stuffs: Seq[Stuff]) => stuffs map { _.number } sum
    val groupedByLabel = items.groupBy { _.label }
    val resultGroupedByLabel = groupedByLabel mapValues { calcSumOfStuff }
    val result = resultGroupedByLabel map { t => Stuff(t._1, t._2) }
    assertEquals(expectedResult, result)
  }

The code shows another Map function named mapValues.

 val resultGroupedByLabel = groupedByLabel mapValues { calcSumOfStuff }

mapValues works on the values of a map only, ignoring the keys. In my case the keys don’t really matter (they’re an artifact of the groupBy call) so I can get away with mapValues rather than map.

Naming things was also the inspiration for the next solution. Now I’m not extracting intermediate results but in this case I’ve named the variables being manipulated:

  def sumByGroupSolution4 = {
    val groupedByLabel = items.groupBy(_.label)
    val result = groupedByLabel map { case (label, stuffs) => Stuff(label, stuffs map { _.number } sum) }
    assertEquals(expectedResult, result)
  }

The case statement as argument to the map makes it possible to introduce two variables to identify the things we’re manipulating. Label and stuffs mean more to me than _1 and _2.

Looking back on my struggle I like this solution best. I think it is both concise and easy to read.

My quest was to write code, which enables me to compare different kinds of currencies as elegant as possible. What I wanted was a very simple DSL with which I could do the following:

2.dollar > 1.euro

The very beginning

That’s where I started out:

trait Money extends Ordered[Money] {
   val unit:String
   val amount:Double
   def compare(that:Money) = if(amount > that.amount) 1 else -1
}

case class Euro(amount: Double) with Money {
    val unit = “EUR”
}

case class Dollar(amount:Double) with Money {
    val unit = “USD”
}

You probably see already which problem we’ll encounter. When we want to compare the same currencies this code will work fine. However, comparing two different kinds of currencies will compile but will yield a wrong result:

//works as expected
assert(Euro(200) > Euro(100))

//compiles but yields wrong result
assert(Euro(99) > Dollar(100))

So my question was how to cope with this situation. Having a trait Money that does the comparison is definitely not the way to go because it does not take the conversion of currencies into considering. This design allows me to compare Euros with Dollars whereas we’re only comparing amounts.

No real option for the problem

One second I considered to add the following check to the Money’s compare method:

def compare(that:Money) = {
    require(unit == that.unit)
    if(amount > that.amount) 1 else -1
}

Needless to say that this cannot be considered a viable solution. Especially with regard to Scala’s rich type system, which should help you to exactly solve such problems. So how to cope with it?

Abstract types to the rescue

Minutes later I remembered abstract types, which provided the solution I was exactly looking for. Instead of letting the compare method accept a type of Money, I define an abstract type alias called Currency, which extends from Money. This abstract type is then used as input for the compare method.

trait Money  {
  type Currency <: Money
  val unit: String
  val amount: Double
  def compare(that:Currency):Int = if(amount > that.amount) 1 else -1
} 

Next I can define an OrderedMoney trait in order to introduce the Ordered trait, for which the compare method is needed in the first place.

trait OrderedMoney[T <: Money] extends Money with Ordered[T]

Because the super trait, Money, already contains an implementation of the compare method the OrderedMoney trait does not need to implement it. Obviously, we also could have provided the implementation of the compare method in the OrderedMoney trait. However the effect is the same.

The OrderedMoney trait needs to be mixed in for every concrete currency implementation as follows:

case class Euro(val amount: Double) extends OrderedMoney[Euro]{
  type Currency = Euro
  val unit = "EUR"
}

case class Dollar(val amount: Double) extends OrderedMoney[Dollar]{
  type Currency = Dollar
  val unit = "USD"
}

From this point on the compiler makes sure that I only compare apples to apples and not apples to pears:

//works as expected
assert(Euro(200) > Euro(100))

//does not compile anymore, as intended
assert(Dollar(100) > Euro(100))
scala: Diverging implicit conversion…

Making it compare apples to pears

In the end we would like to compare apples to pears or here Dollars to Euros. In order to achieve that we simply have to add some implicit conversion logic and our money DSL is (nearly) complete:

object Conversions {
  implicit def fromEuroToDollar(d: Euro) = new Dollar(d.amount * 1.2)
  implicit def fromDollarToEuro(d: Dollar) = new Euro(d.amount * 0.85)
}

For the sake of simplicity we use a hard-coded value to convert from one currency to another. Ideally such functionality would have to be placed in a separate class, like a CurrencyConverter. With these conversions in place apples can finally be compared to pears:

//works as expected
assert(Euro(200) > Euro(100))

//compiles AND works as expected
assert(Dollar(100) > Euro(100))

Finishing touch

For a user of this API it would be convenient to use natural syntax like 2.euro instead of Euro(2). For this to happen only another simple piece of conversion logic needs to be added:

object Conversions {
...
implicit def fromDoubleToCurrency(d: Double) = new {
    def euro =  Euro(d)
    def dollar = Dollar(d)
  }
}

This implicit method converts a Double into an anonymous object that contains a euro and a dollar method. With this in place every Double is pimped with these methods, which makes this DSL as smooth to use as initially intended:

2.euro > 2.dollar

Round up: Adding basic arithmetic operations

Are we there yet? Well, we came quite far but it would be nice if we were able to calculate with currencies, preferably with different types of currencies. The following operations would be quite useful:

//calculate with same types of currencies
2.euro + 10.euro 

//calculations with different types of currencies
1.euro + 20.dollar – 5.pounds

The question is how to implement this requirement with as little impact as possible. With some lines of code this additional requirement can be satisfied quite easily:

trait Money  {
  ...
  def create(amount:Double):Currency
  def +(that:Currency) = create(amount + that.amount)
  def -(that:Currency) = create(amount - that.amount)
}

We’ve added a + and – method to Money and an abstract create method, which is needed to instantiate a currency with the calculated amount. For every type of currency the only thing we have to do is implementing the create method as follows:

case class Euro(val amount: Double) extends OrderedMoney[Euro]{
  type Currency = Euro
  val unit = "EUR"
  def create(amount:Double) = Euro(amount)
}

And from then on we not only can compare different currencies but also calculate with them:

//calculations and comparisons with different types of currencies
1.euro + 20.euro > 15.dollar – 3.euro

All at once

So to round up here the whole code, which enables you to compare apples to pears and perform some basic arithmetic operations all in a typesafe way:

trait Money  {
  type Currency <: Money
  val unit: String
  val amount: Double
  def compare(that:Currency):Int =  if(amount > that.amount) 1 else -1
  def +(that:Currency) = create(amount + that.amount)
  def -(that:Currency) = create(amount - that.amount)
  protected def create(amount:Double):Currency
}

trait OrderedMoney[T <: Money] extends Money with Ordered[T] 

case class Euro(amount: Double) extends OrderedMoney[Euro]{
  type Currency = Euro
  val unit = "EUR"
  def create(amount:Double) = Euro(amount)
}

case class Dollar(amount: Double) extends OrderedMoney[Dollar]{
  type Currency = Dollar
  val unit = "USD"
  def create(amount:Double) = Dollar(amount)
}

object Conversions {
  implicit def fromEuroToDollar(d: Euro) =  Dollar(d.amount * 1.2)
  implicit def fromDollarToEuro(d: Dollar) = Euro(d.amount * 0.85)
  implicit def fromDoubleToCurrency(d: Double) = new {
    def euro =  Euro(d)
    def dollar = Dollar(d)
  }
}

//usage samples:
assert(2.dollar + 3.euro >= 1.dollar + 1.dollar + 1.euro)

After writing this code, I finally understood why Martin Odersky ‘s new company was baptized ‘typesafe’ : that’s what Scala for a ‘big part’ is ‘all’ about

Acknowledgments

This blog was partly inspired on an example in the book Programming in Scala, Chapter 20, Case study currencies

Picking the right ORM-framework

I didn’t do any extensive research, just some googling. A framework called Scweery popped up. The name Scweery sounded very nice and it has a Twitter account with a funny avatar, so why look any further? Well, it seems the project isn’t that active anymore (no updates for over a year and that’s a long time with Scala).
Finally Stackoverflow came to the rescue. I found a question called Examples of Scala database persistence . I picked Squeryl, the syntax looked nice and there was useful documentation. I’m not sure whether I picked the right framework, but it survived my initial selection, so let’s find out.

Setting up the environment

For this article I assume the following:

-You have version 2.9.0 of Scala installed
-Git is on your system
-SBT is available on the command line (I used version 0.7.7)

I believe the links above explain everything pretty well, so let’s get started!

Getting started

I used the ‘minimal example’ project to get started:

git clone https://github.com/pbrant/squeryl_sbt_minimal_example
cd squeryl_sbt_minimal_example
squeryl_sbt_minimal_example>sbt
....
[info]    using ExampleProject with sbt 0.7.7 and Scala 2.9.0
squeryl_sbt_minimal_example>update
.....
squeryl_sbt_minimal_example>compile
squeryl_sbt_minimal_example>run

I updated the Scala version to 2.9.0. This can be done by changing the version number project/build/build.properties. For this article it doesn’t matter much, but for your real project this newer version has lots of improvements.

That went pretty smooth. Let’s rip everything out and use Postgres. Open the file Main.scala (/src/main/scala/code). First remove drop, create and printDdl. We don’t want our valuable production database to be destroyed!
For my database testing I use my good old movie database. With ancient column names (in Dutch) and the ID column is a String. This might sound like a bad idea, but real databases are like this and I found out it’s a good way to test a new framework.

Schema definition

The first section to change is Library extends Schema. This is the section where the tables are defined. My table is called movies.

val movies = table[Movies]

When running the example (with some additional code, so don’t try it yet) I got the following error:
org.postgresql.util.PSQLException: ERROR: relation “Movies” does not exist

It seems the queries are executed with the table names surrounded with double quotes, which means case sensitive. To solve this append (“MOVIES”) to the table definition. Now we can also solve the problem that a class is singular (unlike a database table). Now I can call the class Movie instead of Movies:

val movies = table[Movie]("MOVIES")

Table definition

Our next step is the table definition. (In Dutch, “jaar” means “year” and “naam” means “name”) This time we have to use annotations to use other column names (to get rid of typos and underscores):

class Movie(
 @Column("imdb_id")
 val imdbId: String,
 @Column("jaar")
 val year: Int,
 @Column("naam_film")
 val name: String,
 @Column("genre")
 val genres: String
)

Note that I used val for the columns, when the columns are mutable it’s allowed to use var.

Connecting to Postgresql

To connect to to a Postgres database replace the H2 dependency in /project/build/ExampleProject.scala with

val pg = "postgresql" % "postgresql" % "9.0-801.jdbc4"

Now go to the main method in Main.scala and replace the driver and connection parameters:

Class.forName("org.postgresql.Driver");
SessionFactory.concreteFactory = Some(() =>
 Session.create(
 java.sql.DriverManager.getConnection("jdbc:postgresql://localhost/moviedb", "bill", "s3cr3tp455w0rdth4t15nt5053cr3t4nym0r3n0w"),
 new PostgreSqlAdapter)
)

At a first glance it just looks like JDBC, but the Some(()=>..) notation might be a bit odd. This is by-name-parameter, just see it as a function that creates a session whenever needed.

A select * from

The final section is doing the actual query:

inTransaction {
 val movies = from(Library.movie)(select(_))
 for (movie <- movies){
 println (movie.name)
 }
}

To execute queries you need a transaction (also for read only queries). A transaction is available after the initialization of the SessionFactory.concreteFactory. To run code in a transaction wrap it in an inTransaction block (you can also use ‘transaction’, this always starts a new transaction, inTransaction only does when there is no transaction in progress).

The first line in the inTransaction block does a select * from movie.

Scala might be a bit confusing when you’re new to it, like me. The scaladocs might help. From can be found here and select here.

The statement should read like this: from Library.movie apply each row to the select function. The second set of parameters for from (the so called Queryable) can be expanded with where and order by clauses. For more information read the page about selects on the Squeryl website.

The for loop of course prints all the movie names.

‘Advanced’ queries

The previous paragraph was just a simple query you’ll probably never execute in real life. So let’s try something a little bit more difficult:

inTransaction {
 val movies = from(Library.movie)(s=>
 where(s.year === 1994)
 select(s) )
 println(movies)
 for (movie <- movies){
 println (movie.name)
 }
}

The operator === is a method that is added to Int (this construction is called ‘implicit’) to create the actual where-statement in SQL. I also added a println(movies), this prints the actual SQL-statement, very useful!

Conclusion

I really like Squeryl, it was easy to set up and the basic documentation is sufficient. Of course a next step is inserting and adding more expressions (and for me, handling Postgis geometry objects).
I hope this article gives you a head start.

Sources

-Stackoverflow – Examples of Scala database persistence
-Scala installation
-Git
-Sbt setup
-Scweery
-Squeryl
-Squeryl on github
-Squeryl google group

I love the Option type in Scala. Options provide a very concise way to express whether a variable can have a value or not. Whereas a programmer can express his/her intent in Scala in an extremely concise and compact way I must admit that this is not always true when dealing with Options.
Depending whether I’m dealing with a Some() a want to convert the value of type X to a value of another type Y. Otherwise, in case it’s a None, I want to provide a default value of the target type Y. When writing Scala code I find myself using this construct quite often. To illustrate this let’s consider the following snippet:

val myOpt:Option[Int] = Some(1234)
val converted = myOpt match {
case Some(v) => v.toString.reverse.mkString("-")
case None => "unknown"
}
assert("4-3-2-1" == converted)


Basically, what we want to achieve here is a conversion from Int to a String depending whether the Option is of type Some(Int). Otherwise we want to provide a default value of “unkown”.

This is the only case I can think of where I can associate Scala with boilerplate code. Four lines of code to perform a simple conversion or provide a default value depending whether we are dealing with a Some or a None is quite a lot. The good news is, that there is an alternative. By using other methods of the Option API I could convert this into a one-liner as follows:


val myOpt:Option[Int] = Some(1234)
val converted = if(myOpt.isDefined) myOpt.get.toString.reverse.mkString("-") else "unknown"
assert("4-3-2-1" == converted)


Apparently, it’s not that hard to put the desired conversion in one line of code, however, using conditional logic to achieve the desired result tempers my enthusiasm considerably. What I actually want is the following:


val myOpt:Option[Int] = Some(1234)
val converted = myOpt some_? (_.toString.reverse.mkString("-")) orNone "unknown"
assert("4-3-2-1" == converted)


Personally, I would find such an API for Option very intuitive to use, concise, compact and fully OO. So, since it’s not there (yet) we have to come up with a solution ourselves. Since we want to add new functionality to an existing type without being able to modify the source code, the way to go are implicits. A possible solution looks as follows:

implicit def fromOptionToConvertedVal[T](o:Option[T]) = new {
def some_?[R] (doWithSomeVal:(T) => R) = new {
def orNone(handleNone: => R) = o match {
case Some(value) => doWithSomeVal(value)
case None => handleNone
}
}
}


What is happening here? We first define an implicit function, which takes an Option as parameter. The conversion returns an anonymous object, which contains one function with the name some_?. Any name would do of course. The some_? function is a higher order function that takes a function parameter of type T, which is the same as the one of the Option. The function parameter of some_? then returns a new type of R, since we want to convert the initial value of type T to R.

Instead of doing something within the function body, we return another anonymous object with the counterpart of some_? called orNone. The orNone function takes a call-by-name argument. The body of the orNone does the actual work. Based on whether the Option is a Some the function parameter of some_? is executed otherwise the call-by-name argument of the orNone, which leads to the desired result.

Bottom line: if you are missing something in any library you use in Scala, implicits provide you the perfect means to add everything you like so that it works for you. That is the real ‘option’ Scala offers you

    val configData: Elem = XML.loadFile(jmsConfigFileName)

configData now holds a parsed version of the file whose name is stored in jmsConfigFileName. This file contains stuff like queue and topic definitions like the fragment below:

<queue name="queueTo">
    <sub-deployment-name>domain</sub-deployment-name>
    <jndi-name>domain/jms/qto</jndi-name>
  </queue>
  <topic name="domainJMSTopic">
    <sub-deployment-name>domain</sub-deployment-name>
    <jndi-name>domain/jms/topic</jndi-name>
  </topic>

My first attempt at retrieving all topic definitions from this config file was this:

  def findTopics(configData: Elem): Set[JmsObject] = {
    val jmsObjects = for (topic <- (configData \\ "topic"))
      yield (new JmsTopic((topic \ "@name").text, (topic \ "jndi-name").text))
    jmsObjects.toSet
  }

I felt really happy about having written this powerful and concise loop. Scala’s for construct is powerful, as well as its XML parsing. To retrieve the queues I added the method below:

  def findQueues(configData: Elem): Set[JmsObject] = {
    val jmsObjects = for (queue <- (configData \\ "queue"))
      yield (new JmsQueue((queue \ "@name").text, (queue \ "jndi-name").text))
    jmsObjects.toSet
  }

The duplication was obvious, so I felt a little less happy with my solution but I couldn’t see an easy way out. Enter my colleague and Scala trainer Urs Peter. He showed me two ways to improve the code.
Our first attempt looks like this:

def buildListOfJmsObjectsFromConfigData[T]
           (configData:Elem, clazz:Class[T], startNode:String): Set[T] = {
    val jmsObjects = for (node <- (configData \\ startNode))
      yield ( clazz.getConstructors.apply(0).newInstance(
                             (node \ "@name").text
                           , (node \ "jndi-name").text).asInstanceOf[T])
    jmsObjects.toSet
  }

Which you can then call like this:

buildListOfJmsObjectsFromConfigData (configData, classOf[JmsConnectionFactory]
           , "connection-factory")

This looked quite Java-ish to me, as well as complex and obfuscated.

Our second attempt removes the loop and uses currying. It looks like this:

def buildListOfJmsObjectsFromConfigData[T]
            (configData:Elem, startNode:String) (f:(NodeSeq) => T): Set[T] = {
     (configData \\ startNode).map(f).toSet
}

This second variant of buildListOfJmsObjectsFromConfigData[T] can be called like this:

      buildListOfJmsObjectsFromConfigData (configData, "topic") {
        node => new JmsTopic((node \ "@name").text, (node \ "jndi-name").text)
      }

This works because the (configData \\ startNode) yields a set of objects on which we call map. Map takes a function as a parameter. The function shown above extracts a JmsTopic object, so we end up with a collection of JmsTopic instances.

I’m happy with our final version. It is concise and clear as well as extensible; way better than my first Java-ish version.

The code is attached here.
I created three versions that are exercised in a unit test. JmsConfigReaderV1 is the first Java-ish version, JmsConfigReaderV2 is the compact but obfuscated solution and JmsConfigReaderV3 is my final attempt.
I’m sure there are lots of opportunities to improve, so I welcome your opinions ans comments.

This week I spent two days at EPF in Lausanne (EPFL), where I taught a module of Martin Odersky’s Scala course in order to certify for ‘official’ Scala trainer. My teaching was followed and evaluated by the master himself, which was a unique experience. It definitely sounds more scary than it actually was. Nevertheless, it certainly was a once in a lifetime opportunity, great fun to do and fortunately with a positive outcome.

Being within the walls where Scala first has seen the light had more impact on me than I initially imagined. Realizing that all the highly modular and extendable building blocks of Scala were invented within these walls I started to grasp that this was just the beginning. Martin directs a great deal of research on new features for Scala, where he has the powerful position to cherry-pick the best of breed and integrate it into new versions of Scala.

While talking I learned that just some blocks away there was the research group who was porting Scala to the .Net platform. Martin’s assistant, Iulian Dragos, who taught another module of the Scala course wrote his thesis a.o. about the @specialized annotation, which is now part of Scala 2.8. Whereas parallel collections planned for 2.9 reached already an advanced stage I heard him talking about language virtualization (using the Scala compiler to compile to other languages), a ‘certified’ enterprise Scala stack, comparable to what RedHat does with JBoss, to ease the adoption for the enterprise, the beta eclipse Scala IDE, which is about to be released and so on, and so on.

Posing with Martin Odersky and the Swiss mountains surrounding EPFL

However, what I found most interesting to hear – being a consultant – is that many major Banks mostly in the States and Europe have started to embrace Scala, so far mostly for their business critical trading platforms. When besides innovative startups like Twitter and LinkedIn rather conservative enterprises dare to make such a move, I can hear a bell ring.

During the course it also became evident, that really a lot is going on. Martin had to delegate parts of the course to his assistant Iulian because of his myriads of current activities. Most of his free time during the course he spent mailing, reviewing papers etc. to manage the increasingly demanding Scala empire. He told me that because of lack of time he will delegate the Scala trainings to an associate, so that he can focus on the direction Scala is heading to. It became evident to me that Scala has grown far beyond the academic walls and is gaining serious ground in the industry. So all in all that is REALLY good news.

At the end of the first day there was another pleasing surprise waiting for me. Two local participants of the Scala course told me that at this very evening there is a meeting of the Scala user group of Lausanne and they asked me to join. Half an hour later I stood face to face with Ceki Gülcü, the inventor of log4j and other logging frameworks who is leading the Scala user group in Lausanne. Needless to say that it was a great evening, with lots of Scala live coding and geeky discussions. The next day we told Martin Odersky about our whereabouts, and because I mentioned that the amount of people joining the user group meeting wasn’t that impressive he answered: “Right, I think I should send them some Scala people from EPFL, to help them start up”. I wish our Scala user group in the Netherlands had such a highly elaborate Scala resource pool within reach…

Some people of the Scala user group Lausanne with Ceki Gülcü on the left.

There is another noteworthy anecdote. After lunch at the canteen of the EPFL Iulian pointed me to the other end of the aula, asking me whether I recognize ‘it’. What I saw was a fancy designed stairway, which triggered me at second glance. Have a look by yourself:

THE stairway to Scala

So far this blog has revealed everything else than fancy nuts and bolts of Scala. So to round up lets look at a neat little expression, which might find its way to Scala 2.9:

def ??? = throw new IllegalArgumentException(“Not implemented yet”)

What we’ve defined is a method with the name ??? which throws an IllegalArgumentException. What’s so special about it? The interesting thing is that you can use the ??? method for all methods you have not implemented yet, no matter what type they return (a value type, Unit, a Function, Nothing etc.). Handy for prototyping:

def myMethod(i:Int):String = ???
def myMethod(s:String, i:Int):Unit = ???
def myMethod:Int => Int= ???

So why does that compile? Because of Scala’s fundamental OO nature every method returns something. In case of void the returntype is Unit, in case a method only throws an exception the returntype is Nothing. Therefore, the returntype of the ??? method is of type Nothing. Without making use of Scala’s type inference I would have to write:

def ???():Nothing = throw new IllegalArgumentException(“not implemented yet”)

When you look at the class hierarchy of Scala you see that Nothing is the sub type of every type. Since you always can return a sub type of the actually defined type you return in a method, the Nothing type is a valid return type for anything. As you probably assumed, the Nothing type is added by the compiler. There is no way and need to deal with the Nothing type explicitly. Personally, I find this a smart and powerful construct, stressing once more Scala’s flawless class hierarchy setup.

All in all, it was a great and special experience to visit the breeding grounds of Scala, whose gifted creator is leading this impressive language to new heights, for years of fun and productivity to come for all daring to step over the threshold.