LogStore Project Report 1

This is the first update since we started development on the LogStore. Despite the bumpy ride, we have quite a lot accomplished.

To summarize, the LogStore is a storage solution for logging statements. It’s based on CouchDB (document oriented database) and Elastic Search (schemaless search engine). The view on the log statements stored is built on CouchApp, a tool on CouchDB for creating HTML pages that are served directly from CouchDB instead of a separate application server.

.0pt; line-height: 115%; font-family: Calibri,sans-serif;">As I said we’ve had a bumpy ride. It has taken quite some time to get even a single page on the browser. While we were wrestling with copy and paste actions with javascript snippets from the CouchApp sample project Sofa, we were not really learning the idea behind CouchApp. What we did learn at the time was that there is a large amount of sample applications with all different ways of implementing the functionality, templating etc. We also learnt that our knowledge of javascript and associated technologies is not really up to par. The lack of a tutorial or documentation for javascript noobs was also not very contributing to success.

In some of my spare time, fuelled by frustration, I made a plan for getting to know CouchApp and its relation to CouchDB. I took the online version of CouchDB, The Definitive Guide and started reading. There I learnt (again, as I remember now) that CouchApp is ‘just’ a script that makes it easy to write a design document of a CouchDB database. The layout of a CouchApp represents the internal structure of a design document. At deployment time, CouchApp bundles scripts like list and show functions into the design document and copies it to a CouchDB instance, together with other resources like client side javascript and images.

With the help of this piece of incomplete, but vital information, I came to producing a search result page based on a property of a log item and a log item detail page. The search result page consists of a CouchDB view, a controller script (CouchDB list function) and a Mustache template. The detail page consists of a controller script (CouchDB show function) and a Mustache template. For importing data we developed two small Java applications, based on Ektorp. Ektorp is a persistence API for CouchDB that uses Jackson for JSON-object mapping. One application reads a (log)file and sends the log items one by one to CouchDB and the other reads a (log)file in memory and performs a bulk insert of the log items into CouchDB. It appeared, logically, that the latter application was by far the fastest one.

So what’s next? Since we did not dive into one of our main investigation topics yet, Elastic Search will be the topic of the next session, next to adding some more flow and functionality to the “CouchApp”.

If you want to check out our progress, see https://github.com/xebia/logstore.


| Leave a comment  »