JMeter vs NeoLoad

Erwin van der Koogh

Recently I was asked by a client to introduce performance testing into their development process.
They have 1 week iterations where they integrate the different components of their system and they wanted to performance test the end-result every other week.
Starting out small and simple I downloaded JMeter from apache, fired up their recording proxy, recorded a scenario, changed all the relevant form fields into regex variables and off I went.
It worked great, we had some interesting data with some nice graphs to show performance was okish.
But the happiness didn't last long..

Things quickly went down-hill from there.

  • We had a couple of peaks in the performance data; with no hope of understanding why those happened.
  • We had a need for a slightly different graph; But no way to generate it, except to copy and paste all the raw data into excel and generate the graph from there. The latest version of course, because earlier versions of excel can not handle all the data.
  • We wanted to scale up the load to the point where 1 machine was not able to generate the load anymore; We divided up the data sets, started 3 separate JMeters at roughly the same time and integrated the XML files later. This took between half a day and a day to accomplish
  • We needed information on pageviews, not HTTP requests; There seems to be no way in JMeter to group a couple of HTTP request in one pageview.
  • We could not get the ramp up period to work in JMeter. Our servers were getting hammered in the first couple of minutes until everything settled down and we got a normal amount of traffic going
  • It was very hard to compare the data from one run to the data on a previous run. It was very hard to tell whether the performance had gotten better or worse, or if there was something else going on.

But the major issue did not surface until a couple of weeks later. When someone changed something in the flow of one of the scenarios we tested. We had a failing test with hardly a chance of figuring out what to do about it. After hours of trying to find the problem we gave up and rerecorded the entire thing, redid the regular expressions etc etc.

It was around this time I decided to look for another solution and was recommended NeoLoad by a fellow Xebian. It is a commercial product with what we would call reasonable prices. While definitively more expensive than free, it does solve all the problems we were having with JMeter. You can add what they call monitors that monitor your database statistics, webserver load etc. This gives you the tool to find the root of 'strange' behaviour in the problem areas. You can distribute the load over multiple machines just by installing an agent on another machine and checking a checkbox in the controller. It allows you to view data on HTTP request level *and* pageview level. And best of all, comparing a report with an earlier report is done in 30 seconds. And bestest of all is that you can compare current responses with the responses you got when you recorded them. This allows you to see if someone added/changed/removed form fields for example. If that happens you find the error quickly and do not have to rerecord and prepare the entire scenario.

So the bottom line is it took us almost 3 extra mandays to prepare JMeter and post-process the reports into graphs. With NeoLoad that time is down to about 30 minutes. We have better looking reports with more information than we did before, which we can easily compare with previous runs.
And we can now distribute the load over more machines than we did before.

Now I am not saying JMeter is useless, far from it. It allowed me to quickly get started and get me results really quickly. It supports a lot more different type of requests than NeoLoad does. It can do JMS messages, database calls etc. I have successfully used JMeter to performance tune applications. But for big, serious performance tests, that are part of acceptance testing for example, I really think you should consider NeoLoad.

As a disclaimer, I or Xebia is in no way affiliated with Neotys in any shape or form, except that we got a trial license when I asked for one. Something you can and should do as well.

Comments (15)

  1. anjan bacchu - Reply

    October 29, 2008 at 9:08 pm

    hi there,

    good to know about this tool.

    Have you heard about sun's open source faban ? Here's Scott Oaks(java security book fame) talking about a benchmarking scenario : http://weblogs.java.net/blog/sdo/archive/2007/05/how_to_test_con.html -- very cool.

    Good to know about NeoLoad -- it sure is interesting to know that it can save you good amount of time for something like Acceptance testing.

    BR,
    ~A

  2. Erwin van der Koogh - Reply

    October 30, 2008 at 12:47 am

    Hey Anjan,

    Thanks for the link. Reading his blogs and the Faban site it looks pretty promising. Seeing as Faban is pretty new, it probably lacks the shininess that is NeoLoad. But I am definitely going to give it a try next time I need a free tool!

  3. Paul Bakker - Reply

    October 30, 2008 at 5:44 pm

    Looks really good, thanks for the post. It looks much easier to use than JMeter. I will definitely try it out.

  4. Kai - Reply

    November 2, 2008 at 1:12 am

    Hi!

    I've been using JMeter with success and can say it's surely possible. If you want to group a bunch of HTTP requests (samplers) together, just use a transaction controller. This way you can get more meaningful results.

    Cheers,

    Kai

  5. LV2k - Reply

    March 14, 2009 at 10:09 pm

    I would like to do Performance testing for 250 users. My application is built using applets, basically a gaming application. Can anyone pls suggest some open source tool that supports java applets. I need only in Open source

  6. Oliver - Reply

    March 29, 2009 at 9:17 pm

    Hi

    The report above is partly right. The issues you have had with JMeter are partly due to a lack of know-how around JMeter. JMeter's graphing functions are abysmal. I've written my own graphing functions.

    And a little hint, never use Excel to plot anything. It can only show 32000 resultsets. Use OpenOffice. That doesn't seem to have a limit.

    As for scalability and rampup, both work excellently. Don't know what happened with your rampup.

    I think if you want an Open Source performance testing tool JMeter is the way to go (there are other good ones but they are generally more complex to use). NeoLoad looks good for a commercial tool and is definitely cheaper than most. Personally I'd still think it's money down the drain, that is spent better in manpower to learn to use your tool.

    All commercisl performance testing tools have one drawback. You cannot discredit the tool from your time measurements. How do you know the tool is giving you the right response times? We have analysed JMeter on a code level to check that it does what we expect. then you can guarantee that it's the program under test and not your tool. Don't underestimate this fact!

    Cheers
    Oliver

  7. Erwin van der Koogh - Reply

    April 2, 2009 at 12:18 pm

    Hey Oliver,

    Of course your mileage will vary. For me having to write my own graphing functions means I would rather shell out money for a tool that does it out of the box and will keep working 3 or 4 years down the road.

    I am not saying JMeter is a bad tool, it certainly isn't. But in my situation at the time it was not sufficient for our needs.

  8. Ravi Gill - Reply

    April 21, 2009 at 1:13 am

    Hi Erwin,

    I have used neoload, load runner, siege etc.

    And i have been using jmeter for about 2 years now.

    There is absolutely nothing that you cant do with jmeter that you can with neoload.

    some of the points:

    * We had a couple of peaks in the performance data; with no hope of understanding why those happened.

    - Did you monitor the targeted servers, and looked in the log files, i am sure you can sync the times and see what happend during those peaks.

    * We had a need for a slightly different graph; But no way to generate it, except to copy and paste all the raw data into excel and generate the graph from there. The latest version of course, because earlier versions of excel can not handle all the data.

    -An simple perl script (available on the net) and or a java class(also available on the net) can be used to parse those results and gnu-plot and or any other plotting tool can be used to generate graphs. once automated it makes life pretty eay :).

    * We wanted to scale up the load to the point where 1 machine was not able to generate the load anymore; We divided up the data sets, started 3 separate JMeter instances at roughly the same time and integrated the XML files later. This took between half a day and a day to accomplish

    Jmeter can be used for distributed testing, which means the master jmeter gui(even non-gui) can use extra servers to generate load and save the results from all the slave+master server on the master server itself. starting 3 different instances of jmeter and then combining the xml files is not advised.

    * We needed information on pageviews, not HTTP requests; There seems to be no way in JMeter to group a couple of HTTP request in one pageview.
    - you can easily accomplish this with "simple controller", indi Http requests can be stacked as child under simple controllers.

    * We could not get the ramp up period to work in JMeter. Our servers were getting hammered in the first couple of minutes until everything settled and we got a normal amount of traffic going
    -this is odd behaviour, i hope all the givne variables were correct.

    * It was very hard to compare the data from one run to the data on a previous run. It was very hard to tell whether the performance had gotten better or worse, or if there was something else going on.
    -Are you sure that the servers were not shared with any other parties/apps that were generating load of their own and or the testdata was same for all the tests?, different variables can also interfere here like available network bandwidth etc, the jmeterclient should ideally be under the same subnet and close to the targeted environment.

    For any questions i would like to forward you to your fellow colleague from the netherlandse called Levent, he seems to be an expert at jmeter aswell :).

    with kind regards,

    Ravi Gill,

  9. Erwin van der Koogh - Reply

    April 21, 2009 at 6:23 am

    Hey Ravi,

    Thanks for your suggestions. They certainly help if I ever have to use JMeter again!

    What I meant with the no hope of understanding what happened is that there was no way to automatically correlate the information from JMeter with the monitoring data from all the different servers and backend systems that we had. It turned out one of the servers wasn't even time synced properly so all timestamps were also out of whack by a bit. So of course it was possible, just very very time consuming.
    With NeoLoad you can gather all your monitoring data together with your performance data and compare them together, zoom in together. That was extremely helpful.

    I have found no way to distribute the load over multiple machines. Either it was not possible at the time, it wasn't very obvious or my and my team was blind at the time 🙂
    But it is good to know it is possible, I'll try harder next time.

    About the ramp up meter. As far as I could see all the variables were properly set up. We tried changing every variable that could conceivably be tied to the ramp up with little to no effect.

    What I meant was that it was hard to compare is that you could look at the tables and graphs and note the differences. What I really liked about NeoLoad is that I could select 2 different runs and it would compare them and create a change report.

    Funny you should mention Levent.. If I am not mistaken it was him that suggested I give NeoLoad a try 😉

    Thanks for the comments, they will make at least my next JMeter job a bit easier.

  10. Ravi gill - Reply

    April 22, 2009 at 4:07 pm

    Hi Erwin,

    I Completely agree with you here that performance testing with jmeter can be really time consuming as the inbuilt jmeter reporting is not very reliable.

    Neoload is a pretty impressive tool and definitely competes with the market lead loadrunner in both pricing and value.

    Regarding Jmeter i remember that in the previous versions the distributed testing was not working and the simple controller was not available either, but both of these problems have been resolved in the latest release version 2.3.2, you should definitely check it out :).

    Please do give my greetings to levent the next time you see him.

    With kind regards,

    Ravi.

  11. Shiva Kumar - Reply

    November 4, 2009 at 12:43 pm

    Hi,

    Please help me comparision between loadrunner,Jmeter,Webload

  12. Erwin van der Koogh - Reply

    November 5, 2009 at 7:36 pm

    Shiva,

    I am afraid I can not help you much in that regard. First you need to figure out what your requirements are from a load testing point of view.
    Loadrunner can do pretty much everything under the performance test sun, but is expensive and not very easy to use.
    Jmeter is open source, has quite a bit of features, but I find it pretty clunky to use and not very straightforward.
    Neoload is a good mix between the two. Not free, but a lot cheaper than loadrunner, a lot of loadrunner's features and a better user experience than both Loadrunner and JMeter.

  13. Gerind - Reply

    August 13, 2012 at 11:39 pm

    This is quite an interesting article, even though i put - "i have been there and done that" across JMeter and Neoload for a very high volume based testing using AWS, as part of our geographical load distribution.

    Here are my 2 cents of info:
    a. JMeter can be used for small scale load testing with limited features and if you can live with it - then its fine. If anybody wants to know the reliability or scalabilit of the applicaiton - then this isnt the best choice. what else can you expect of a 'free-tool'!!!
    b. i have tried to conduct a high volume testing using JMeter for more than 50,000 vus - but, couldnt scale more than 20,000 users - even on a 64-bit-Linux basd OS
    c. usually perf testing / scalability / reliability / stress testing are conducted within a shorter time frame, to unearth performance issues / bottleneck; the team should be more judicious in conducting these tests rather than building utilities aroung JMeter and maintaining the same.
    d. at the end of the perf testing - all eyes are on the issues and a good analytical report - to culmincate all the metrics of back end servers, infra, apps metrics. so need a tool which shudnt burn the budget and at the same time to be more flexible and stable to scale upto high volumes....

    being this, when we tried with Neoload - it was quite ok adopt, scale up to 1 Million user session with 75,000 vus over AWS without any glitches and at the same time, we spent very few minutes in extracting data and put it into a good graphicall and easy-to-understand-for-non-techines-and-managment-team report, with a small spend.

    Anecdote: About 1.5 yrs back, we had done an extensive experiment of conducting a load of 1 Million Virtual Users PER SECOND, just by using 4 OPEN SOURCE tools in our lab on a very sample applicaiton with tomcat and MySQL, using a EC2 environment - dont ask me about the consolidation of reports from all these open source tools....it took days....!!!!with manual co-relation.

  14. kirti - Reply

    July 15, 2013 at 3:14 pm

    Can I use JMeter for testing PHP based applications.

  15. RUDRA GANESH SUBBARAYULU - Reply

    March 22, 2016 at 2:19 pm

    I really don't get how the functionality of TCP Sampler,BSF Sampler in jmeter is used in Neoload. Neoload functionality is hard to define and understand.

Add a Comment