Comments on: Pentaho Kettle and Integration Testing http://blog.xebia.com/2009/09/30/pentaho-kettle-and-integration-testing/ Software development done right! Wed, 08 Feb 2012 14:41:38 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Shrikant Vashishtha http://blog.xebia.com/2009/09/30/pentaho-kettle-and-integration-testing/#comment-99382 Shrikant Vashishtha Mon, 17 Jan 2011 08:20:55 +0000 http://blog.xebia.com/?p=3196#comment-99382 Hi Max, We felt that it was much more difficult to do that and would require a lot of time and resources as there was not much available documentation. Also we didn't feel very confident that we are under control while implementing enterprise level solution with Kettle because on and off we kept on seeing problems coming. We actually left using Kettle and adopted a new strategy based on Spring Integration implementation and it worked for us. Hi Max,

We felt that it was much more difficult to do that and would require a lot of time and resources as there was not much available documentation. Also we didn’t feel very confident that we are under control while implementing enterprise level solution with Kettle because on and off we kept on seeing problems coming. We actually left using Kettle and adopted a new strategy based on Spring Integration implementation and it worked for us.

]]> By: Max Hofer http://blog.xebia.com/2009/09/30/pentaho-kettle-and-integration-testing/#comment-99319 Max Hofer Sat, 15 Jan 2011 18:51:47 +0000 http://blog.xebia.com/?p=3196#comment-99319 ShriKant, have you made any progress in this direction? I'm also trying to figure out how to test transformations/jobs in an automated way. ShriKant, have you made any progress in this direction?

I’m also trying to figure out how to test transformations/jobs in an automated way.

]]> By: Shrikant Vashishtha http://blog.xebia.com/2009/09/30/pentaho-kettle-and-integration-testing/#comment-92619 Shrikant Vashishtha Thu, 01 Oct 2009 05:45:35 +0000 http://blog.xebia.com/?p=3196#comment-92619 @Matt - We already used a different properties file and passed it through using TransMeta.injectVariables(). It works fine until we have a sub-transformation underneath. Somehow transformation is not able to pass properties to sub-transformation. If you use EnvUtil.environmentInit(), it overrides the properties passed with the ones existing in kettle.properties (should have been other way round). I may have been a bit harsh in calling the Kettle code "legacy code" but I could see source code of 6000+ lines which is hard to understand and certainly not designed for testability. While working with Kettle I found following roadblocks for which may be solutions exist but I could not find them in available resources: 1. Manual restart after failure (the ability to restart from where it failed) 2. Transaction over multiple insert steps 3. Automatic retry (for instance HTTP service or web service) and recovery for items that have exhausted their retry count. 4. Integration testing with database independence (using actual db instead of in-memory db right now) 5. Web services portability. For certain standards it doesn't work. It's difficult to ask a web-service vendor to change the web-service itself. Many a times we used to reach to a wall from where we had to find some workaround. Integration testing (using Continuous Integration) example I mentioned in the blog is one of them. @Matt – We already used a different properties file and passed it through using TransMeta.injectVariables(). It works fine until we have a sub-transformation underneath. Somehow transformation is not able to pass properties to sub-transformation. If you use EnvUtil.environmentInit(), it overrides the properties passed with the ones existing in kettle.properties (should have been other way round).

I may have been a bit harsh in calling the Kettle code “legacy code” but I could see source code of 6000+ lines which is hard to understand and certainly not designed for testability.

While working with Kettle I found following roadblocks for which may be solutions exist but I could not find them in available resources:
1. Manual restart after failure (the ability to restart from where it failed)
2. Transaction over multiple insert steps
3. Automatic retry (for instance HTTP service or web service) and recovery for items that have exhausted their retry count.
4. Integration testing with database independence (using actual db instead of in-memory db right now)
5. Web services portability. For certain standards it doesn’t work. It’s difficult to ask a web-service vendor to change the web-service itself.

Many a times we used to reach to a wall from where we had to find some workaround. Integration testing (using Continuous Integration) example I mentioned in the blog is one of them.

]]> By: Brad http://blog.xebia.com/2009/09/30/pentaho-kettle-and-integration-testing/#comment-92615 Brad Wed, 30 Sep 2009 14:58:32 +0000 http://blog.xebia.com/?p=3196#comment-92615 Great post. We are currently looking into the same issue -- the ability to test Kettle transforms in isolation -- and this is a better explanation than I've seen anywhere else (including the Pentaho wiki). Matt - your solution of not using kettle.properties is a good one, and I'd agree that Kettle is far more testable than most other ETL tools... no need to get so defensive, though. The poster pointed out some legitimate issues, and this is good feedback for your community. Great post. We are currently looking into the same issue — the ability to test Kettle transforms in isolation — and this is a better explanation than I’ve seen anywhere else (including the Pentaho wiki).

Matt – your solution of not using kettle.properties is a good one, and I’d agree that Kettle is far more testable than most other ETL tools… no need to get so defensive, though. The poster pointed out some legitimate issues, and this is good feedback for your community.

]]> By: Matt Casters http://blog.xebia.com/2009/09/30/pentaho-kettle-and-integration-testing/#comment-92612 Matt Casters Wed, 30 Sep 2009 11:16:13 +0000 http://blog.xebia.com/?p=3196#comment-92612 It's always good fun to hear about hard-core programmers that try to solve business intelligence issues. If you don't want to load the information in the kettle.properties file, here's some advice: don't put anything in there! The Kettle variable or named parameter system does indeed NOT put anything in java.lang.System. Not testable? You got to be kidding me. Just because you have problems grasping a few basic concepts, that doesn't mean you have the right to call Kettle "legacy code" or throw around other insults. Try to find another way to vent your frustrations. It’s always good fun to hear about hard-core programmers that try to solve business intelligence issues.

If you don’t want to load the information in the kettle.properties file, here’s some advice: don’t put anything in there! The Kettle variable or named parameter system does indeed NOT put anything in java.lang.System.

Not testable? You got to be kidding me.

Just because you have problems grasping a few basic concepts, that doesn’t mean you have the right to call Kettle “legacy code” or throw around other insults. Try to find another way to vent your frustrations.

]]>