If an application turns out to be too slow, tuning can provide a solution. Tuning can take place on multiple levels. Adding hardware can be a cheap solution. However, when you add hardware at a place where the bottleneck is not located, this has little use.

Important steps of tuning are therefore identifying which pages or services do not meet stated requirements and isolating the problem: where is it located, in which layer, in which component. This can be made clear with testing and monitoring on application parts. The next step is diagnosing. In fact, this comes down to making up a hypothesis why this component is so slow.

Figure 1. Performance tune cycle.

This can for instance be a missing or wrong index on a database table or the invocation of too many small queries. Next, the component is improved based on this hypothesis. Finally, one needs to verify whether the improvement actually brings the expected speedup. If so, then the proposed hypothesis is true and the speedup is the result. If not, then there is something wrong with the hypothesis and we need an alternative hypothesis. As soon as the performance of the system meets its requirements, tuning is finished.

Figure 2. Finding evidence.

The right tools are indispensable: performance test tool, enterprise profiler, heap monitor, etc. I have seen developers work on assumed performance improvements which turned out not to help at all, or even slowed down the application and also deteriorate the maintainability and flexibility. This is caused by the fact that developers are used to mould functionality from source code and therefore work from source code to improve performance. What is missing here is measure, don’t guess. This is something developers learn in my performance training. Experience also has taught me to judge every proposed improvement separately and to only implement the improvement when we have proven that it really helps. Without this systematic approach, a handful of steps together may take you backwards at the end of the day instead of forwards.

Next time I’ll talk about Sharing responsibility for the whole chain.

When a new version of the software is released into the production environment, the question always is: will it actually perform like we saw in testing and acceptance environments? And we keep our fingers crossed.
It is therefore important in such cases to monitor carefully what happens with the performance and availability of the application.

There are all sorts of tools and services available to monitor your web site for availability and response times of web pages, like Uptrends, Site24x7 and Dotcom-monitor. They look at the application as a black box and measure once in several minutes. This is very useful, however, to be able to take the right measures in case of a calamity, it is necessary to be able to pin-point the problem. It is therefore essential to monitor on multiple levels and on multiple internal application parts. For levels, think of hardware, OS, app server, web server, database and application. Measuring internal Java application parts can be achieved with JAMon. JAMon is an open source timing API and basically works like a stopwatch with a start() and stop() call. Every method which you want to measure gets its own stopwatch (or counter) . We deal with JAMon as one of the tools to measure time in the first day of our Speeding up Java Applications course.

Figure: JAMon API start() and stop() calls in a Spring interceptor

Each counter maintains statistics like the number of calls, average, maximum, standard deviation, etc. , and this information can be requested for. The individual calls are not stored. This approach results in low memory usage and a low performance overhead, at the cost of some information loss. Recently, a new competitor of JAMon appeared: Simon. It claims to be JAMon’s successor, although it has (had) some infancy issues.

Then there is the question: where to measure? The answer is that it makes most sense to measure all incoming calls like web requests and outgoing calls to for instance the database. Furthermore, parts like Spring beans, EJB’s and DAO’s. Measuring these parts is not only relevant with new releases, but also trends and usage spikes are useful to monitor in order to solve quickly and prevent various problems. Open source tool JARep offers the possibility to store JAMon data from a cluster in a database and monitor trends and changes graphically.

Figure: JARep shows the increasing response time trend starting October 15, on two of the four production JVMs.

Customer story

We had the following situation at our customer. Processing an order slowly took more and more time over a period of several weeks. This happened while no new release was introduced and no other page became slower. This behavior was a complete mystery, until we looked deeper in our JARep monitoring tool. The troublemaker turned out to be a DAO executing a prepared statement with only part of the variables being bind-variables. With help of JARep, we could look back to where the trend of increasing response time started so when the problems started. We could also see that this problem was only present at one of the two machines. With this knowledge and his log book, the operator could remember that on the start date he had experimented with a new JDBC driver to try to solve a memory leak. This seemed not to change anything concerning performance, what actually was the case in the beginning. Problems only appeared slowly during the following weeks. They had left the new driver in place, manifesting itself as a time bomb later. When we put back the old driver, the problem disappeared. This real life experience shows the usefulness of monitoring and trend analyses on application internals.

Next time I’ll blog about evidence based tuning.

Continuous performance testing
With a representative test as one of the last steps before going live we prevent that expensive bad-performance surprises will pop up in production. However, the same surprises will pop-up, only earlier and with less impact. To save costs and prevent large architectural refactoring, it is crucial to test for performance as soon as possible. This is just like any other software defects and Quality Assurance: the later in the development process defects are detected, the more costly these defects are.

At a popular web shop I had the following challenge: we wrote the performance tests only at the end of the six-weekly release period, after functional testing had taken place and functional defects were corrected. In case serious performance defects popped up, a crisis team was gathered and we found ourselves in a stressful situation. There was usually not enough time to fix the defect before the release date, so my recommendation at times was to defer the release date. However, deferring the release date often just was not possible, because TV or radio time was bought to promote the new functionality. So, how to solve this dilemma? We found the solution in applying agile principles: test features as early as possible and the team is responsible.

We included meeting performance requirements in the definition of done in each new or changed feature individually. The development process already included an automatic build, quite common these days. Unit tests of a feature were written as usual by the developer. We now introduced performance tests to the spectrum: the developer writes the performance test script for his feature (service or web page) in JMeter, side-by-side to his unit tests on the classes. When the nightly build with Maven has taken place, the application is deployed on WebSphere and the performance tests are run by the JMeter Ant script. This script generates a report which is e-mailed to stakeholders. In this way, the IT department gets early insight into new and changed features, it can adapt its course more quickly, back-off early from an unfortunate architecture or approach, minimize surprises and have lower costs as well. Additional benefit is that writing test scripts gets done more quickly than before, because the developer has all details of the new feature still fresh in his memory. These details are for instance the conditions under which the service may be called and by which parameters, in which variations and special cases. The usual communication overhead between a performance tester and a developer on these details is drastically reduced, thereby further improving productivity.

Continuous performance testing is too often underestimated and un recognized in my opinion, it really has a whole lot of advantages and it is not that hard to achieve.

Next time I’ll blog about Step 5: Monitoring and diagnostics.

Slowness of applications in development environments is often neglected with the rationale that faster hardware in the production environment will solve this problem. However, whether this is really true can only be predicted with a test on a representative environment and in a representative way. In such an environment, there needs to be more representative than just the hardware.

I have experienced multiple times that a database query on the test database with 1000 customers took only less than 10 ms., while on the production database with 100.000 customers this turned out to take tens of seconds, because of missing indexes. So, if the development team does not test with a full, complete database, going to production may lead to some surprises.

It is also important that the number of concurrent users and their behavior is well simulated in the test. Furthermore, care should be taken to take caching effects into account: if the test continuously requests for the same product by the same customer, this data will be in database or query cache the second and following times. This will speed up the request considerably and be much faster than with many customers and products. This test is therefore not representative for the real situation.

A suitable performance test tool and performance expertise is necessary to create a valuable test. The most popular open source performance test tool is Apache JMeter, see the next figure.

Figure: Screenshot of a run of a performance test in Apache JMeter.

We deal with this tool, and how to performance test properly in our Speeding up Java Applications course. This is a tool made by programmers, for programmers. Test scripts can be created with visual elements like a HTTP request, which can be recorded and configured. Many are available and if you need more, you can always fall back on a BeanShell element in which you can manipulate the request, response and various JMeter variables. If that even does not meet your needs yet, you can extend JMeter source code and develop your own elements. Because of its for-programmers nature, it is less suited for the average tester. Also reporting features and maintainability of the scripts are both not so great. Therefore, commercial tools like HP Mercury LoadRunner, Borland SilkPerformer or Neotys’ Neoload may be good alternatives for companies.

Performance testing from the cloud

The emergence of cloud computing adds new possibilities for performance testing. An elastic compute cloud like Amazon EC2 provides the ability to scale up quickly with the number of application deployments because of increasing load. For performance testing, the cloud can be used the other way around: for temporary use of many load generating test clients to generate expected and peak loads for your application. This saves you from having to buy many servers to run the load generating clients and if you run these performance tests say only a couple of days in a release cycle, this can be an economically attractive solution. Quite some information is available how to test with various performance tools from the cloud.

Next time I’ll blog about step 4: continuous performance testing.

The IT world is very sensitive to trends. Having been around in the IT industry for 15 years, I’ve seen a few. A technology is hot for a while, and then quickly becomes out-of-fashion and yesterdays news. It will be replaced by something which is much better and what everyone follows almost blindly.

Such fashion items are for example CORBA, CGI, applets, EJB, Struts, Spring, Server Faces, XML, SOA, OMT, UML and RIA to name a few. Often new, bleeding edge technology is used in a project just for the sake of being trendy. In addition, each technology or framework comes with its own teething troubles and largely uses more resources than its predecessor. Goal of such a new technology is generally improvement of flexibility, productivity or maintainability, and performance usually has no priority or has not even been considered at all.

Therefore, it is questionable if the chosen new technology and architecture will meet the specified performance requirements. In practice, this regularly becomes only evident in a late stage of the project: when it has already slipped beyond the planned production date. Only then it may become clear that the chosen technology or architecture is just not sufficient. And switching to a different technology or architecture usually results in high cost and long delays. Therefore it is essential to execute a Proof of Concept for performance in which all technology and architecture components are touched, in a vertical slice of the application. It is important that this test is performed in a sufficiently representative manner, on which I will elaborate in my next post. By executing this PoC and understanding and using the results of it, the project can early be corrected in the right direction to prevent excessive cost and delay.

Next time I’ll blog about step 3: representative performance testing.

It can be a valid choice to run the risk of performance problems in production and deal with them in a re-active manner. However, it is usually wiser to be pro-active and prevent them. This approach brings more certainty, peace of mind and also saves money. It consists of seven steps. Step 1: Define performance requirements.

Defining the performance requirements well usually is of under-estimated importance. Most of the time the requirement is formulated as: it just has to be fast or: at least as fast as the previous platform. With such vague definitions the confusion starts. The goal is unclear and is typically explained very differently by the business and the IT department. To prevent this, the goals should be formulated in a SMART way and be prioritized. Speed will be more important for a shop homepage than for a page where a customer can change his profile. By defining priorities, this order of importance is made explicit and clear. From SMART, the A stands for Attainable and the R for Realistic. These aspects are often ignored by the non-technical contributors to these requirements. In that case, a short response time will lead to an extended development time or expensive hardware. Half a second slower during peak hours can be acceptable if this saves tons of money. On the other hand, reducing the response time of an important page from 4 to 2 seconds can lead to a substantial growth in revenue. So, a solid analysis of the impact of performance on the business is needed to be able to clearly define the performance requirements in a SMART way, prioritized and be able to balance the cost and benefits.

Next time we’ll deal with step 2: Execute a performance PoC.

More and more Internet users buy in web shops these days. Research shows that the part of European Internet users that buys on-line has grown from 40% in 2004 to 80% in 2008. Additionally, large web retailers in The Netherlands see their revenue grow just as if the recession has never materialized. Business seems to be flourishing.

I also like to shop on the web. Now and then I enter a web shop where I have to wait before pages appear fully. Most of the time I’ll move away: with just one click I’m off to the competitor. The increased comparison possibilities and freedom of choice offered by the Internet are not only valid for the products, but also for the web shops themselves. Therefore, it has become crucial for the success of the web shop to have a responsive web site.

With only a few concurrent visitors, it is usually not so hard to have a quick website. However, with the growing trend of Internet sales, the increasing integration and complexity of back-end systems and the by marketing demanded ever increasing richness of the user experience, this often becomes a big challenge for developers and operators. This may result in systems blacking out under high load or responding too slowly.

So the question is: how can we prevent these performance and availability problems and how can we assure a web site is always quick and available?

On the basis of real life, trial and error experience we’ve come to an approach which can be described as: measure, don’t guess; seven steps to performance success. I’ll explain this approach in the follow up blogs, and next time I’ll first describe in which forms performance difficulties can present themselves. Stay tuned.

Recently, I was called in by a company with a website in trouble. And because they make all their money on-line, it was evident that they really wanted to have the issue solved. The day I came in, the site had gone down about 5 times the last 24 hours. Because of this they got less traffic, which directly meant less revenue. The log files showed periods of long response times and OutOfMemoryErrors. Their questions were: Why do we get this behavior? and How do we fix it? My short answer turned out to be: because of too many loitering objects; and this can be fixed by not holding on to them in the HTTP session. I’ll explain.

Troubleshooting assignment
As often is the case in this kind of troubleshooting assignments, once you have found exactly what is wrong, you’ve almost solved it already. The difficulty is in doing the detective work of nailing down the root cause. Or differently put, finding the needle in the haystack.

VisualVM
The tool they already used and actually was new to me was VisualVM.

Figure 1. VisualVM About window

It is similar to jdk’s jconsole in that you can visually look into the JVM. It actually has a plug-in architecture by which for instance parts of jconsole are plugged into visualVm to see heap and thread usage, loaded classes and MBeans. For lower overhead than jconsole, it however has less detail in sections like heap. It also has a profiler plug-in which is the Netbeans profiler. So you can actually CPU or memory profile local applications from it. And since the recent version 1.6.0_07 the JDK has this great tool bundled with it now. In case you cannot find it in the JDK: they have renamed it to jvisualvm.

Figure 2. VisualVM monitor example

VisualVM helped in that we could see in the generated stack dumps that threads were often waiting for some log4j lock. Reason being the log level was set to DEBUG. Setting it to INFO improved performance somewhat. However, in this case visualVM did not bring me the crucial facts. Reason for this was twofold. The first was the lack of detail I mentioned and the second was the type of tool, being one having to be a witness of the problem situation to be able to analyze it.

Finding the crucial information
The toolset that gave me the first half of the crucial information was the -Xloggc:gc.log flag combined with HPJmeter. This gives the opportunity to analyze after the facts. That night, traffic was rather low and the site did not go down, but it was very slow during some hours. The next day, the log file showed at the time when performance became bad:
15045.703: [GC 1786342K->1481712K(1856512K), 0.1639990 secs]
15053.886: [GC 1790320K->1489985K(1858880K), 0.1722250 secs]
15065.930: [GC 1800257K->1493876K(1859968K), 0.1601460 secs]
15066.090: [Full GC 1493876K->1467212K(1859968K), 3.5031310 secs]
15071.966: [Full GC 1708415K->1471029K(1859968K), 3.5039160 secs]
15078.215: [Full GC 1708277K->1476287K(1859968K), 3.5326720 secs]
15087.847: [Full GC 1708386K->1408698K(1859968K), 4.5038550 secs]

We see here the fall-over from doing mainly GC’s (young space) to mainly Full GC’s, so from short pauses to long pauses. Actually every 6 seconds or so, and a collection takes 3.5 s. So a lot of time is spent in gc. As you can see, however, some memory is still reclaimed. With HPJMeter I could see this in nice graphics to draw conclusions:

Figure 3. HPJmeter showing percentage of time spent in GC.

So, it became clear to me that the site was so slow because up to 95% of the CPU cycles were spent on garbage collection, which left only 5% to the application itself.

Need for more details
What however the reason for this excessive gc was, was not clear to me, yet. I needed more details on the memory usage. So, we switched on the -XX:+PrintGCDetails flag which gave us the second half of the crucial information.
Unfortunately, HPJMeter could not deal with the produced log format, but we got lines at the start of the problems like:
21270.421: [GC [PSYoungGen: 436938K->112249K(481664K)] 1808785K->1489945K(1879808K), 0.1790760 secs]
21277.559: [GC [PSYoungGen: 443385K->111206K(486528K)] 1821081K->1497781K(1884672K), 0.1815330 secs]
21284.659: [GC [PSYoungGen: 442406K->107401K(489536K)] 1828981K->1504035K(1887680K), 0.1780120 secs]
21284.837: [Full GC [PSYoungGen: 107401K->79553K(489536K)] [PSOldGen: 1396634K->1398143K(1398144K)] 1504035K->1477697K(1887680K) [PSPermGen: 79038K->79038K(262144K)], 3.5320210 secs]
21292.969: [Full GC [PSYoungGen: 331200K->83102K(489536K)] [PSOldGen: 1398143K->1398143K(1398144K)] 1729343K->1481246K(1887680K) [PSPermGen: 79047K->79047K(262144K)], 3.5302490 secs]
21302.663: [Full GC [PSYoungGen: 331200K->85665K(489536K)] [PSOldGen: 1398143K->1398143K(1398144K)] 1729343K->1483809K(1887680K) [PSPermGen: 79055K->79055K(262144K)], 3.4508870 secs]
21311.177: [Full GC [PSYoungGen: 330681K->21303K(489536K)] [PSOldGen: 1398143K->1398144K(1398144K)] 1728825K->1419447K(1887680K) [PSPermGen: 79065K->79027K(262144K)], 4.4884740 secs]
Again, here we see the fall-over. If you look at the old space: the first full gc puts objects from young in old making it almost full, there is only 1K or less of unused space left shown in the full GC lines. The confusion comes from the young space usage, where most garbage still can be collected. This makes the total heap not full, while the old space is at its limits. So, some things get stuck in old space. The remarkable thing is that the situation recovers after an hour or so and an OOME does not occur.

Heap analyses with Netbeans
So, now it is time to do more heap analyses. Netbeans has a great built-in profiler with a useful heap analyzer. We found that most of the memory was used by Category objects, which was not intended as such. By using the heap walker we could find that references to it were mostly held from Tomcat session objects. So, each HTTP session had its own copy of these Categories and they were kept hanging around for half an hour: the default HTTP session timeout. The term for this kind of objects is: loitering objects.

The quick and the real fix
At this point we could devise a quick remedy: decrease the HTTP session timeout from 30 to 5 minutes. This reduced most pressure on us fixers and gave us some air to breath. Then we created some tests and were able to reproduce the behavior in a test environment. Looking at jconsole also clearly showed the growing usage of old space.

Figure 4. JConsole showing memory usage for each memory pool.

In the source code we could easily find the place where all category objects were loaded from the database, put in a cache once and, along the way, also a copy in each user HTTP session. For no reason really. And with 1700 user sessions this took most of the memory.

So, this was a no-brainer to solve in the code. And after deploying and setting back the session timeout to the default 30 minutes, problems were solved. I love it when a plan comes together.

Learn more
If you want to learn more about this stuff and get hands-on experience, come to my class of “Speeding up Java applications” in April or November 2009.