Easy breezy restful service testing with Dispatch in Scala

Urs Peter

For testing a restful service API I was looking for a lean library, which would allow me to test CRUD operations of rest services with as little code as possible.

My search led me to Dispatch, which is a highly compact Scala DSL wrapper around Apache’s reliable HttpClient. This DSL, however, is not very well documented and rather hard to decipher due to it's heavy usage of symbolic method names but nevertheless highly appealing when understood.

In this blog I'll decipher it for you and show how easy it is to test restful services with mere oneliners.

My Prerequisites

Before we dive into Dispatch’s mysterious DSL let’s look at the prerequisites I had for testing my restful service. These will most likely apply for many other rest services as well:

  • Support for CRUD operations with http’s POST, GET, PUT and DELETE methods
  • Support for XML and Json as input and output payloads
  • Support for reading http status codes to test results of erroneous responses like 404 (Not Found)
  • Support for security aspects such as https and basic/digest authentication

Begin with the End in Mind

The goal of this blog is it to create a RestClientHelper that offers a set of generic restful client methods and explain how they’re implemented using Dispatch. With these methods in place we will be able to call and therefore test restful service APIs very easily.

The RestClientHelper will be a trait that offers the following client side CRUD methods:

trait RestClientHelper {
	
  //Create (POST) 
  def create[T](target: String, reqBody: String)(fromRespStr: String => T): T 
  def create[T](target: String, reqBody: Elem)(fromRespXml: Elem => T): T 

  //Read (GET)
  def query[T](target: String, params: Seq[(String, String)])( fromRespStr: String => T): (Int, Option[T]) 
  def query[T](target: String, params: Seq[(String, String)])( fromRespXml: Elem => T): (Int, Option[T]) 

  //Update (PUT)
  def update[T](target: String, reqBody: String)(fromRespStr: String => T): T 
  def update[T](target: String, reqBody: Elem)(fromRespXml: Elem => T): T 
  def update[T](target: String, reqBody: String): Int 
  def update[T](target: String, reqBody: Elem): Int 

  //Delete (DELETE)
  def delete(target: String) :Int
}

As you can see, most methods consist of two parameter lists. The first parameter list represents the rest call input such as target uri and request body or parameter map. The second parameter list is a function that converts the output of a rest service into the desired type. The return type of these methods are of type T, Int or Tuple2[Int, Option[T]]. The Int always represents the http status code, the Option[T] represents the converted output if the service call was successful.

How to use the RestClientHelper

To get an understanding how the RestClientHelper trait is supposed to be used let’s look at an example. Assuming we have a domain object, Person, that provides serialization and deserialization methods for xml a CRUD call sequence with the RestClientHelper trait would look as follows:

val person = Person("John Doe")

//POST /add.xml 
val p = create("add.xml",person.toXml){Person.fromXml(_)}
assert(p.id != None)

//GET /search.xml?q=John+Doe 
val (status, personOpt) = query("search.xml", Seq("q" -> "John Doe")) { Person.fromXml(_) }
assert(status == 200)

//PUT /update.xml 
val changedPerson = p.copy(_.name = "John Who")
val status2  = update("update.xml", changedPerson.toXml)
assert(status2 == 200)

//DELETE /delete/4 
val status3 = delete("delete/" + changedPerson.id.get)
assert(status3 == 200)

//GET /search.xml?q=John+Who 
1
val (status4, personOpt2) = query("search.xml", Seq("q" -> "John Who")) { Person.fromXml(_) }
assert(status4 == 404)
assert(personOpt2 == None)

Looking under the hood

Now let’s dig a little deeper and discover how one of these rest client helper methods are actually implemented using Dispatch.

def create[T](target: String, reqBody: String)(fromRespStr: String => T): T = {
  executer(:/(host, port) / target << reqBody >- { fromRespStr })
}

Well, this is all that is needed for invoking restful service by means of a POST request. Being new to Dispatch it can help untangling this highly compact DSL for a better understanding.

The core classes participating in an http call as shown above are the following:

  • A HttpExecuter (dispatch.HttpExecuter) responsible for executing the http request. A common implementation would be dispatch.Http. In this example it is respresented by the executer object.
  • A Request (dispatch.Request) and its DSL wrapper dispatch.RequestVerbs, which are responsible for creating a specific kind of request, e.g. a gzip http post with content-type application/x-www-form-urlencoded.
    The class RequestVerbs contains DSL-ish symbolic methods names, which mostly start with a ‘<’ character indicating that something is ‘added’ to a request (left arrow).
    The method << is one of those. It adds a request body to the request and therefore automatically transforms it into a POST request.
  • A Handler (dispatch.Handler) and its DSL wrapper dispatch.HandlerVerbs, which is responsible for handling the result returned by the http call, such as converting it to xml or json.
    The class HandlerVerbs contains as well DSL-ish symbolic method names, which mostly start with a ‘>’ character indicating that something is done with the ‘output’ of the http call (right arrow).
    The method >- is one of those. It converts the response received as a String into the desired type.
  • For both, the Request and Handler, implicit conversions allows them to be converted into their corresponding DSL wrapper RequestVerbs or HandlerVerbs respectively.

With those key classes in mind we can rewrite the above http call in a more verbose manner in order to understand the parts the DSL is made of:

def create[T](target: String, reqBody: String)(fromRespStr: String => T): T = {
    val emptyReq:Request = :/(host, port)
    
    //implicit conversion applied explicitely
    val reqVerbs:RequestVerbs = Request.toRequestVerbs(emptyReq) 

    val configuredReq:Request = reqVerbs./(target).<<(reqBody)
    
    //implicit conversion applied explicitely
    val handlerVerbs:HandlerVerbs = Request.toHandlerVerbs(configuredReq)

    //The http executer always needs to be called with a handler
    val handler:Handler[T] = handlerVerbs.>-(fromRespStr)

    executer.apply(handler)
}

The executer needs further explanation. As said, Dispatch requires you to use an executer to finally execute an http request. Dispatch offers several types of executers, such as thread and non-thread safe ones. A thread-safe one that makes use of a shared connection pool could be provided as follows:

  protected lazy val executer = new Http with thread.Safety

The non-thread counterpart could be accessed like this:

  protected def executer = new Http

Various other executers are available. The ones above will most likely be sufficient for most purposes.

To summarize we can conclude that most of the power of Dispatch from a usage point of view lies within the RequestVerbs and HandlerVerbs classes, which allow us to compose a request and decompose its response the way we want it. There is an excellent periodic table of dispatch operators available that lists all the possible Verb methods with a short description.

…and now the nitty-gritty details

With the knowledge we have gained about Dispatch the remaining method implementations of the RestClientHelper should be straight forward. I’ll show each implementation with a short explanation of the most notable Dispatch methods:

Create (POST) with String in- and output

 def create[T](target: String, reqBody: String)(fromRespStr: String => T): T = {
    executer(:/(host, port).POST /  target << reqBody >- { fromRespStr })
  }
  • RequestVars: POST
    Create a POST request (the POST method is optional because of the '<<<' method, which forces the request to be a Post)
  • RequestVars: <<(body:String)
    Post the given String value with text/plain content-type
  • RequestVars: >-[T](block:(String) => T)
    Convert the response body from a String to the desired type

Create (POST) with XML in- and output

  def create[T](target: String, reqBody: Elem)(fromRespXml: Elem => T): T = {
    executer(:/(host, port) /  target << reqBody.toString <> { fromRespXml })
  }
  • RequestVerbs: <<(body:String)
    Post the given String value with text/plain content-type
  • HandlerVerbs: <>[T](block:(Elem) => T)
    Convert the response body from an Elem to the desired type

Read (GET)
The read method is probably the most intriguing one from the whole series. In order to understand it fully, I will provide a detailed explanation:

 def query[T](target: String, params: Seq[(String, String)])(fromRespStr: String => T): (Int, Option[T]) = {
    executer x (:/(host, port) /  target <<? params >:> identity) {
      case (200, _, Some(entity), _) => {
     	   val respBodyStr = fromInputStream(entity.getContent()).getLines.mkString
           (200, Some(fromRespStr(respBodyStr)))
      }
      case (status, _, _, _) => (status, None)
    }
 } 

Now let’s look at the most important Dispatch ingredients for this GET request:

Input processing
The method <<? from RequestVerbs: <<?(params:Traversable[(String, String)]) simply adds query parameters to the request url.

Output processing
To process the query result we are interested in two things: the response code and response body. Both are returned in the form of a Tuple2 (Int, Option[T]), where Int represents the status code and Option[T] the converted object in case the query yielded a result. The question is how can we retrieve both of them, since Dispatch does not offer a symbolic method that does that for us.

First let’s take a closer look at what is actually called: Instead of providing the executer with a handler directly we use the method ‘x’.

  executer x (…)

Why is that? By calling the executer’s apply method (that’s what finally happens under the hood) the handler block is only called in case the response status code is 200 – 204. In all other cases an Exception is thrown. So if we want to intercept all response codes we need to use the Executer’s x[T](hanlder:Handler[T]) method, which executes the handler no matter which response code is returned.

The next question is what kind of handler is passed to the executer’s x method? If we explicitly assigned the first part of the DSL construct to a handler it would look as follows:

  val intermediateHandler = (:/(host, port) /  target <<? params >:> identity)

The ‘magic’ lies in the >:> identity construct. According to the API the HandlerVerb method >:> accepts a function, by which the response headers can be processed. By passing Scala’s Predef identity method to the >:> method, nothing is processed but the resulting handler of type Handler[Map[String, Set[String]]] is returned.

As said, the handler above does not process the result itself. The real processing is done by the case statements, which are the last arguments that are passed to the DSL construct. How do we need to interpret that? For a better understanding, the last statement could be rewritten as follows:

 val realHandler = intermediateHandler.apply {
    case (200, _, Some(entity), _) => {
       val respBodyStr = fromInputStream(entity.getContent()).getLines.mkString
       (200, Some(fromRespStr(respBodyStr)))
    }
    case (status, _, _, _) => (status, None)
 }
 executer.x(realHandler)

What happens is that we construct another handler by calling the apply method of the previously created one. The apply method of all the handlers accept a function with the following signature:

    apply(next:(Int, HttpResponse, Option[HttpEntity], () => T) => R)

The Int stands for the response code, the HttpResponse and Option[HttpEntity] give us access to the underlying Apache HttpClient implementation and the argument () => T represents the transformation function, which transforms the response to type T, the type of the Handler itself.

As you probably know a case statement IS a function (PartialFunction), which can be chained and passed - even if chained - as a single PartialFunction to a method. So by providing the handler’s apply method with the above case statements we’re able to define in detail how the low-level result of the http call needs to be processed. In our case this means: retrieve and convert the response body when the response code is 200, otherwise simply return the response code.

Even though this construct might look rather complicated it is a very powerful way to access and process the raw result that the http call returns.

Update (PUT) with String in- and output

def update[T](target: String, reqBody: String)(fromRespStr: String => T): T = {
  executer(:/(host, port).PUT /  target <<< reqBody >- { fromRespStr })
}
  • RequestVerbs: PUT
    Create a PUT request (the PUT method is optional because the <<< method forces the request to be a PUT)
  • RequestVerbs: <<<(body:String)
    Put the given String value with text/plain content-type
  • HandlerVerbs: >-[T](block:(String) => T)
    Convert the response body from a String to the desired type

Update (PUT) with XML in- and output

 def update[T](target: String, reqBody: Elem)(fromRespXml: Elem => T): T = {
  executer(:/(host, port) /  target <<< reqBody.toString <> { fromRespXml })
 }
  • RequestVerbs: <<<(body:String)
    Put the given String value with text/plain content-type
  • HandlerVerbs: <>[T](block:(Elem) => T)
    Convert the response body from an Elem to the desired type

Update (PUT) without response body

def update[T](target: String, reqBody: String): Int = {
    executer x ((:/(host, port) /  target <<< reqBody >:> identity) {
      case (status, _, _, _) => status
    })
}

If we want to send a PUT without expecting a response body but are still interested in the response code we use the same construct as described above in the read sample. The only difference compared to the read example is that we do not retrieve the response body, but simply return the response code.

Delete (DELETE)

def delete(target: String):Int = {
   executer x ((:/(host, port)).DELETE /  target >:> identity)  {
      case (status, _, _, _) => status
   }
}

The delete method does not process a response body. In order to know whether the deletion was successful we are however interested in the response code. Therefore, we again use the construct as in the read example.

Security & Authentication

Finally, let’s explain what we needed to do if our restful service was secured with https and/or basic/digest authentication.

For https the RequestVars' secure method should be called on the request, for authentication the as(“username”, “pwd”) method. Therefore, a secure create (POST) call that uses basic authentication would look as follows:

 def create[T](target: String, reqBody: String)(fromRespStr: String => T): T = {
    executer(:/(host, port).secure.as(“user”, “pwd”) /  target << reqBody >- { fromRespStr })
 }

In case security combined with authentication is used it is self evident that the RestClientHelper trait would provide a method that returns a preconfigured request in order to by DRY:

trait RestClientHelper {
 val host: String
 val port: Int
 protected def username:String = "unkown"
 protected def pwd:String = "unkown"
 private def req = :/(host, port).secure.as(username, pwd)
 
 def create[T](target: String, reqBody: String)(fromRespStr: String => T): T = {
    http(req / target << reqBody >- { fromRespStr })
 }
...

Wait, there is more

Dispatch offers various other features that are not mentioned in this blog. E.g.: Dispatch can directly convert a response body into a Lift Json object by means of the ># method (a good example is provided on the Dispatch site), it can execute http requests in a background thread, there are various HandlerVerbs and RequestVers, which we have not covered yet like using gzip mode, chaining handlers etc. to mention some of them. (check the periodic table for further reference). For testing restful services however, the stuff covered in this blog should suffice.

The RestClientHelper ready to use

To conclude this blog as follows the source of the RestClientHelper trait, which hopefully makes restful service testing for you a piece of cake and fun (after having Dispatch set up)!

import scala.io.Source._
import scala.xml._
import org.apache.http._
import dispatch._

trait RestClientHelper {
  val host: String
  val port: Int
  val contextRoot: String
  protected val ssl = false
  protected val username = "notdefined"
  protected val pwd = "notdefined"
  protected val executer = new Http with thread.Safety
  private def req = {
    val req = :/(host, port).as(username, pwd) / contextRoot
    if (ssl) req.secure else req
  }

  def query[T](target: String, params: Seq[(String, String)])(fromRespStr: String => T): (Int, Option[T]) = {
    executer x (req / target <<? params >:> identity) {
      case (200, _, Some(entity), _) => {
        val respStr = fromInputStream(entity.getContent()).getLines.mkString
        (200, Some(fromRespStr(respStr)))
      }
      case (status, _, _, _) => (status, None)
    }
  }
  def query[T](target: String)(fromRespStr: String => T): (Int, Option[T]) = {
    query(target, List())(fromRespStr)
  }
  def queryXml[T](target: String)(fromRespXml: Elem => T): (Int, Option[T]) = {
    queryXml(target, List())(fromRespXml)
  }
  def queryXml[T](target: String, params: Seq[(String, String)])(fromRespXml: Elem => T): (Int, Option[T]) = {
    val convertOutput = (s: String) => fromRespXml(XML.loadString(s))
    query(target, params)(convertOutput(_))
  }

  def create[T](target: String, reqBody: String)(fromRespStr: String => T): T = {
    executer(req / target << reqBody >- { fromRespStr })
  }
  def create[T](target: String, reqBody: Elem)(fromRespXml: Elem => T): T = {
    executer(req / target << reqBody.toString <> { fromRespXml })
  }
  
  def update[T](target: String, reqBody: String)(fromRespStr: String => T): T = {
    executer(req / target <<< reqBody >- { fromRespStr })
  }
  def update[T](target: String, reqBody: Elem)(fromRespXml: Elem => T): T = {
    executer(req / target <<< reqBody.toString <> { fromRespXml })
  }
  def update[T](target: String, reqBody: String): Int = {
    executer x ((req / target <<< reqBody >:> identity) {
      case (status, _, _, _) => status
    })
  }
  def update[T](target: String, reqBody: Elem): Int = {
    update(target, reqBody.toString)
  }

  def delete(target: String): Int = {
    executer x ((req).DELETE / target >:> identity) {
      case (status, _, _, _) => status
    }
  }
}

Possible usage in an object or any other class you want the RestClientHelper to be mixed in:

object MyRestClientHelper extends RestClientHelper {
    val host = "rest.service.host"
    val port = 443
    val contextRoot = "v1"
    override val ssl = true
    override val username = "myusername"
    override val pwd = "secret"
}

Comments (6)

  1. Into the weeds « Tim Gilbert's Blog - Reply

    December 8, 2011 at 8:51 am

    [...] a very small client / Scala wrapper.  Here’s a REST client example in Dispatch, here’s another focused on testing with some explanatory apparatus, and another Dispatch-related example. Share [...]

  2. Liam Clarke - Reply

    January 31, 2012 at 8:43 am

    Interesting blog post, I'm going to have to look further into Predef.identity, thanks. :)

  3. Tue - Reply

    February 24, 2012 at 12:39 pm

    Super article, Thanks!

    It looks like dispatch crue didn't make a UPDATE function with the option to set the content_type
    unless you are using a File to do the update (which seems a bit impractical in most cases). Thus I
    have made the following modification to the RestClientHelper presented in this article.
    =====================================================
    def update[T](target: String, reqBody: String)(fromRespStr: String => T): (Int, Option[T]) = {
    executer x(<<<:> identity) {
    case (200, _, Some(entity), _) => {
    val respStr = fromInputStream(entity.getContent).getLines().mkString
    (200, Some(fromRespStr(respStr)))
    }
    case (status, _, _, _) => {
    (status, None)
    }
    }
    }
    def <<<< (request: Request, reqBody: String): Request = request.copy(method = "PUT", body = Some(new RefStringEntity(reqBody, "application/json", req.defaultCharset)))
    =====================================================
    Usage example:

    @BeanInfo case class VMaction(action: String)
    val actionStop = new VMaction("stop")
    val jsonActionStop: String = new String(serializer.out(actionStop))
    System.out.println("JSON Action Stop: " + jsonActionStop); // See the output below
    val (stat, resp) = update("servers/" + "1fdf2bc4-031a-428d-bd8d-dd37427370d8", jsonActionStop) { parseResponse(_) }
    if (resp.isDefined) {
    System.out.println("HTTP Body: " + resp)
    }
    else {
    System.out.println("HTTP Status code: " + stat)
    }

    ===============================================================

    It would be super nice if the dispatch team would provide a function like:

    def <<< (stringbody: String, contenttype: String): Request = PUT.copy(
    body=Some(new RefStringEntity(
    stringbody, contenttype, subject.defaultCharset))
    )

    ==============================================
    Try to take a look at the implementation of the different functions
    in dispatch.requests.scala

    Thanks again :-)

  4. Tue - Reply

    February 24, 2012 at 12:49 pm

    Hi again,

    It seem like the text is formatted when you leave a reply. In the above reply (by me) i made a modification
    to the RestClientHelper, but it is not what I wrote. In the second line I wrote:

    execute x
    (
    <<<:> (this should be: right arrow colon right arrow)
    identity
    )
    =====================================================
    Reproduced modification to RestClientHelper:

    def update[T](target: String, reqBody: String)(fromRespStr: String => T): (Int, Option[T]) = {
    executer x(<<<:> identity) {
    case (200, _, Some(entity), _) => {
    val respStr = fromInputStream(entity.getContent).getLines().mkString
    (200, Some(fromRespStr(respStr)))
    }
    case (status, _, _, _) => {
    (status, None)
    }
    }
    }
    def <<<< (request: Request, reqBody: String): Request = request.copy(method = "PUT", body = Some(new RefStringEntity(reqBody, "application/json", req.defaultCharset)))
    =====================================================

  5. Tue - Reply

    February 24, 2012 at 1:00 pm

    Hi again,

    I'm going to try again in words.

    In the second line (executer x ...) that you see is wrong. It should be the following all in one line:

    execute x followed by an opening parenthesis followed by four left arrows (we are calling the function shown correctly above) followed by an opening parenthesis followed by req followed by a slash followed by target followed by a comma followed by reqBody followed by a closing parenthesis followed by a right arrow followed by a colon followed by a right arrow followed by identity followed by a closing parenthesis followed by an opening curly bracket.

    WOW

  6. Lawrence Wagerfield - Reply

    June 4, 2013 at 7:06 pm

    Great article on Dispatch - look forward to getting my hands messy with this soon :)

    However, the perception that 'REST is CRUD' is incorrect:
    http://blog.punchbarrel.com/2008/10/31/rest-is-not-crud-and-heres-why/

Add a Comment