Recently for our project we started using Kettle for ETL purposes. Pentaho Kettle provides UI based tool. Initially it takes quite some time to get used to Kettle UI as it becomes difficult to visualize how to orchestrate available Kettle Steps to solve a business problem. As you know how to use it, it's all about drag and drop a step and configuring it with available UI. With our experience we observed that it's pretty easy to design 90% stuff easily but rest 10% involves a lot of research and at the end involved some hacks which we never liked.
(more...)

Tags: kettle integration testing
Filed under kettle | 3 Comments »

For my current project we use Kettle to process data from a number of sources and store it in a shared database. Kettle has great support for parsing data from a variety of sources, transforming it and writing it to a variety of destinations. One problem that often arises when inserting data in a relational database is the need for a syntactic, unique key that will be generated when a new row is inserted and later in the process used for other rows that refer the primary row. There are many solutions for this problem, both in the RDBMS and in the Java space. This blog reports on a search through several code bases to come up with a good solution in kettle which, unfortunately, still evades me.
(more...)

Tags: generated keys, Hibernate, hsqldb, kettle, postgresql
Filed under kettle | 2 Comments »