Around 2006 several initiatives to create open source performance monitoring tools for java production environments started to appear.

This was mainly because AOP (Aspect Oriented Programming), the technology used in most of these products, was getting attention in the market and there were quite some developments in that area at the time.

I am interested to see how the open source community around these kind of products is evolving. The outcome is quite surprising…

The main reason I am interested in tools in this area is to be able to pinpoint problems in an Acceptance or Production environments very quickly, because this saves (big) enterprises a lot of money.

The main reason I am interested in open source tools is to see if we are able to create a complete open source java hosting stack which is enterprise production ready.

Without adding a lot of extra license and/or support costs we don’t have to consolidate IT anymore. We will be able to do application hosting for each specific business unit separately and focus on the continuous improvement for that business unit alone. Without big boundaries between the business unit and IT. This kind of boundaries are most times created because of cost efficiency. Think about consolidation and architecture guidelines. Not having to think about licencing costs will help us in helping enterprises to become agile.

During a knowledge exchange session (XKE) at Xebia we first created a requirements list to select tools in the same area to be able to compare them.

The requirements so far:

1. The tool should measure production and acceptance environments with low (<= 5%) resource usage overhead.
2. The tool should be able to pinpoint (potential) problems in the running applications as quickly as possible.
3. The problems that should be detected: high resource consumption of particular code blocks, memory leaks and the location of the leak, blocking code blocks which block other threads, slow calls to back ends (i.e. web services, databases).
4. The environments the tools should run in are the following application servers: IBM Websphere, Oracle Weblogic, JBoss, Tomcat, Glassfish.
5. The problems should be made visible in such a way that the root cause can be found as quickly as possible. The most valuable will be to see complete business transaction flows over all JVM’s, with real-time flow states and measurement readings.
6. The tool should be easy to install, i.g. just use a simple agent that connects it to a server. The goal should be to leave the code base unchanged.

Before the session I did some homework and I searched the internet for these kind of monitoring tools. During our knowledge exchange session we had a brainstorm session and came to the following list of Open Source tools:

This list is as complete as we were able to come up with, if you know more and eventual better open source tools please let us know!

After evaluating all products on the lists (just a paper evaluation) we only see Glassbox and Infrared as a real Open Source solution in this area. But both (at least Glassbox) are not actively developed anymore. Of course because both products are open source it will be possible to restart development on one of the products.

It is interesting why both projects are not active anymore. I spent some time to find out why. This is what I found out:

What happened to Glassbox; the first developer mentioned is the VP of Engineering at Glassbox David Pickering. He has left the Glassbox company in 2009. And is now VP of Engineering at iWin, a company with a different focus. The second one mentioned is Ron Bodkin, he left Glassbox in 2007 and was the founder of the Glassbox company, he became VP of Engineering at Quantcast, so also his focus has moved. This looks like the main reason there is no active development on Glassbox anymore.

What happened to Infrared; the first developer mentioned on the project member list at SourceForge is Binil Thomas, he worked for Tavant Technologies at the time Infrared was created. Other members on the same list also worked for Tavant in that time. Most are now working for different companies. It looks that Tavant Technologies was no longer interested in this space. The last 2.6RC1 release from Infrared was created in 2010, at a time Binil was working for Ronin Capital. Since 2011 he is working for AppDynamics, this is a commercial vendor in this area and looks like the main reason the 2.6RC1 release never came to a full release, but this is something I am guessing of course.

What’s next

We will test both Infrared and Glassbox to see if one of them is still a good choice. We will start with Infrared because this one has been the most active project in the past and it has been maintained the longest, until 2010. After this we will see if one of these products can be a competitor for the commercial products in this area.

Many thanks to my colleagues Adriaan Thomes and Sander Hautvast for helping me with the brainstorm session.

But this time I was really impressed. The device is a possible replacement for IBM Datapower XML Security Gateway. But the way they designed the device is totally different.

What CrossCheck networks did was creating a device with just security as main use case. First of all it was an XML gateway, nowadays is does support HTML, XML, SOAP, FTP, JMS and others.
It also translates different flavors of JMS to each other, it can even convert from IBM MQ to JBoss MQ directly.

The Forum Sentry XML Gatewat comes in two flavors, as an appliance and as an VMWare image.
This is useful for the Development and Test environments.

The main use-case for the device is as a security device. It will stand in the DMZ and acts as a level 7 security gateway. This is the same use-case as the IBM Datapower, only the marketing is a little bit different, CrossCheck Networks markets the device mainly as a security appliance. IBM did market the Datapower mainly as a ESB in the past. Nowadays they also market it mainly as a security appliance.

The main differences I spotted between those two where:

My conclusion after this short demo
The Forum sentry has some advantages when you compare it to the IBM Datapower XML Security Gateway XS40. The main difference is that you can do more whith only one appliance. You can replace an IBM Webseal, a virus scanner and an IBM Datapower XS40 with only one device.
My advice is to take this device in considerations where you have to choose for an XML firewall/ hardware ESB.

My conclusion for now is that JavaFX is an interesting new technology, I think it can be used for small applications or forms (think about internet adds). But for serious applications it is not ready yet.
I am really missing Datagrids, the way you have to deal with Session cookies for the Rest service is far to low level. I hope someone can point me to a place where I can find better solutions for those problems (if there are).

Requirements
The basic functional and technical requirements I described in my previous post, but the most important requirements for the front end are:

• Have an tree data table (in Flex it is called an advanced datagrid) for visualizing an hierarchical data structure.
• Be able to show adds in the front-end to be able to earn the costs of the hosting.

The back-end service
The service I used in this review is an simple service for which I have to logon via Basic Authentication an can get User information (username) via an Rest call.

The javaFX client
First I created a new FavaFX project and added some fields to the form.

After this I created some code to do the login action and show the username.  (FYI: I now that this is no good OO-design but for this small test it is the best way to show it.)

public class User {

    var sessionCookie: String;
    var username: String;
}
var user: User = User {};

public class Main {
    // <editor-fold defaultstate="collapsed" desc="Generated Code">

    var userJsonInput;
    var userParser = PullParser {
                documentType: PullParser.JSON;
                input: bind userJsonInput;
                onEvent: function (event: Event) { // parse the JSON user data and populate the user object
                    if (event.type == PullParser.END_VALUE) {
                        if (event.name == "username") {
                            user.username = event.text;
                        }
                    }
                }
            }

    //todo make this a function to add the headers! Make ik possible to logout/ relogin
    function buttonOnMouseClicked(event: javafx.scene.input.MouseEvent): Void {
        var httpHeader = HttpHeader.basicAuth(usernameTextbox.text, passwordTextbox.text);
        var sessionHeader = HttpHeader {
            name: "cookie"
            value: user.sessionCookie
        };

        def request: HttpRequest = HttpRequest {
            location: "http://localhost:8080/pfserver/services/private/account/show";
            headers: [ httpHeader,sessionHeader ]
            onInput: function(input: java.io.InputStream) {
                userJsonInput = input;
                userParser.parse();
                if(request.getResponseHeaderValue("set-cookie") != null){
                    user.sessionCookie = request.getResponseHeaderValue("set-cookie");
                }

           }
           onException: function(ex: java.lang.Exception) {
              println("onException - exception: {ex.getClass()} {ex.getMessage()}");
          }
        }
        request.start();

    }
}

The first thing you see is the PullParser, this parser is an parser for JSON data. I fetched the event of an new field an added this to a global value (ugly, I know ).
Using this was quite simple, but in my opinion to low level for big JSON data structures.
Getting the data is done with the HttpRequest, searching for this function did costs me 30 minutes because this is changed in each new version of JavaFX. First there was some async package for this, but now we only have the HttpRequest. Using the HttpRequest was not complex but this time again to complex for big applications, simple things as a cookieManager are missing, of course I can write one myself, but this type of things will cost me to much time if I compare it to Flex.

The first requirement, having a tree data table is also not met, even a simple data table does not exists for now.

Conclusion
I will not use the current version of Flex for my new front-end. It’s to low level at the moment compared to Flex. It is slowly getting there if I compare it to previous releases but it still has a long way to go.

In the first part of this review (the JavaEE6 back-end) I tested a small application which is a JSON REST service to be used as back-end for a JavaFX front-end. My conclusion for now is that JavaEE6 has a lot of new features which makes it a lot easier to use Java EE without extra libraries like Spring, Seam or Resteasy. I was able to make a back-end application which was noticeable fast with a low overhead in bandwidth and CPU usage.

Introduction

In a search for the current best technology platform I am building a small real-world application for personal use in different languages and frameworks. First up is Java EE 6 and JavaFX 1.3.

I think this review can be helpful for others as well. If you just want to see the implementation you can skip the functional and technical requirements. If you are interested in the application and want to help creating the new version; please send me a reply:-).

Functional requirements

Currently I am using an application for managing my personal finances. This application is created in Java Swing with a Hsql database beneath it. The application functions well, but is only usable for one person at a time an the Hsql database is not ideal.

I created this application because I could not find any other application to manage my personal finances which has the following features (and more):

• Register coming transactions on budgets
  • One time transactions
  • Repeated transactions (currently with a cron syntax)
• Be able to report on self defined periods
  • The period can be defined by a start date and and Unix cron syntax
  • The period reports shows
    • The budget for that period (based on all coming transactions) for each budget entry
    • The total amount spent for that period
    • The amount to spent for that period
    • The difference between the budget and the reality
  • The multi period report shows
    • For the defined period the totals shown in the period reports
    • This report can be used to know your financial situation for the coming months.
  • Be able to register transactions
  • Be able to import transactions (currently only from one bank)

To be able to let more users make use of this application I am thinking about rebuilding it.

Technology requirements

I have some specific requirements for the technologies which I am going to use for this new version:

• Have a tree data table (in Flex it is called an advanced datagrid) for visualizing an hierarchical data structure.
• Low resource utilization on server side to reduce the cost of running the server
• Be able to run on mobile phones as well (maybe a basic version)
• Use an extremely scalable backend to service lots of users
• Be able to show ads in the front-end to be able to earn the costs of the hosting.

Java EE6 and JavaFX

Since JEE6 is coming and the first beta releases of Glassfish and Netbeans are out I thought I try to use only Java EE6 technologies and try to make a basic test setup including client server communication and basic security. This will also give me the ability to deliver a small review of the new Java EE Edition.

In the past I have used a lot of different technologies to be able to use some kind of Dependency Injection (DI):

• Self build DI framework (2003)
• Spring Framework(2006)
• Seam Framework (2007)

JEE6 is the first Java EE edition to deliver a DI framework which appear to not need additional frameworks. So I will use the Injection framework of JEE6 for Dependency Injection. For the client I choose JavaFX. Currently JavaFX 1.3 is in beta, but it at first glance it seems to fits my requirements. For the communication between JavaFX and JavaEE I used JSON with JAX-RS (new in EE6). This way I hope to get low bandwidth and CPU overhead. I used JSON before but at that time the protocol was brand new. Now it is integrated in JavaEE.

The back-end application

For creating the back-end application I started a Netbeans 6.9 beta web project. I only gave the project a name, selected the internal Glassfish 3.0.1b14 server and selected the JEE6 profile. First I created an User object which I can use to transfer data from the server to the client.

@XmlRootElement
public class User {
     private String username;
     private String password;

     public User(){} //needed for Jaxb objects

     public User(String username, String password){
         this.username = username;
         this.password = password;
     }
     ...getters and setters for the username and password
}

The only thing I needed to do to use this object in JSON communication is adding the @XmlRootElement annotation.
This is quite simple and saves a lot of work. The next thing I created was a service to look up account information

@SessionScoped
public class UserService implements Serializable{
    public UserService() {}

    private User user;

    public boolean login(String username,String password) {
       this.user = new User(username,password);
       return true;
    }

    public boolean isLoggedIn(){
        return(this.user != null);
    }

    public User getUser(){
        return user;
    }
}

Here I only needed to give the bean a scope. Since this bean is used for logging in and holding the account information, I make it a Session scoped bean. As you can see I did not connect it to a real back-end because I am going to choose the back-end later. I now have a login service I have to connect to a security mechanism. Since I do not want to use external frameworks I used Basic authentication. This works well because it integrates well with JavaFX. To be more secure I can use transport layer security (https). For the authentication I created a Servlet Filter

@WebFilter("/services/private/*")
public class SecurityFilter implements Filter {

    @Inject UserService userService;

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
        HttpServletRequest req = (HttpServletRequest) request;
        if (userService.isLoggedIn()) {
            chain.doFilter(request, response);
            return;
        } else {
            String username = null;
            String password = null;
            // Get the Authorization header, if one was supplied
            String authHeader = req.getHeader("Authorization");
            if (authHeader != null) {
                java.util.StringTokenizer st = new java.util.StringTokenizer(authHeader);
                if (st.hasMoreTokens()) {
                    String basic = st.nextToken();
                    if (basic.equalsIgnoreCase("Basic")) {
                        String credentials = st.nextToken();
                        sun.misc.BASE 64 Decoder decoder = new sun.misc.BASE 64 Decoder();
                        String userPass = new String(decoder.decodeBuffer(credentials));
                        int p = userPass.indexOf(":");
                        if (p != -1) {
                            username = userPass.sub string(0, p);
                            password = userPass.sub string(p + 1);
                        }
                    }
                }
            }
            if (username != null && password != null && !"".equals(username) && !"".equals(password)) {
                if (userService.login(username, password)) {
                    chain.doFilter(request, response);
                    return;
                }
            }

        }
        HttpServletResponse res = (HttpServletResponse) response;
        String s = "Basic realm=\"Login Rest Services - PersonalFinance\"";
        res.setHeader("WWW-Authenticate", s);
        res.setStatus(401);
    }
}

As you can see most of the code is quite simple and is only needed for decoding the basic authentication realm. The interesting JavaEE6 part is the @WebFilter(“/services/private/*”) annotation. With this annotation you can define where the filter will be used. It is no longer needed to define the filter in an web.xml file. The other interesting annotation is the @Inject UserService userService; with this annotation the UserService service is injected. Since the UserService has an @SessionScoped annotation this service will always be one instance for each session. The only part of the back-end that is left is the Rest Service itself. This service gives the current User bean back via JSON over REST.

@Path("private/account/")
@RequestScoped
public class Accounts implements Serializable{
    @Inject UserService userService;

    @GET
    @Path("show")
    @Produces("application/json")
    public User show(){
        return userService.getUser();
    }
}

As we can see implementing JSON over REST is easy.
The implementation of REST is done by using the @Path(“private/account/”) ,@GET and @Path(“show”) annotations. This produces an url /private/account/show for the show function. Which will accept the HTTP GET method.
To deliver the JSON output all I had to do was adding the @Produces(“application/json”) annotation and be sure that the User object is annotated with the @XmlRootElement annotation.

To test this simple back-end I deployed the application to the build in Glassfish server by clicking with the right mouse button on the project and selecting deploy.
After the deployment we can directly test the back-end:

As you can see we have to login. After typing an Username and password we see the JSON output.
The requirement to have a minimum of bandwidth overhead will be met. The JSON output is very small.
To test the performance I created a simple Jmeter test which just sends a correct REALM to login and shows the JSON output. The results where quite good.

I got 307 requests per second with 90% of the requests within 37 milliseconds without tuning anything and with 10 concurrent threads in Jmeter. The CPU load was 40% at that moment (Core i7 QC820). This proves for me that the overhead is low. With this case I simulated an worse case scenario where every request is a new session because I did not use an cookie manager in Jmeter.

Conclusions
The use of JavaEE6 for the back-end will be an appropriate choice. It is possible to develop very fast (I created the whole demo without any knowledge of the new features of JEE6 in less than 2 hours). And it meets the technical requirements in terms of overhead.
In the next part I will blog about the front-end in JavaFX 1.3.

Currently I am working at a big Enterprise where they use Tivoli Access Manager as authorization and authentication source for a lot of there applications.

This Enterprise is using JBoss as open source application server platform and is using this more and more. When they began using JBoss they got a TAM plug-in for JBoss from IBM. This plug-in did the complete authorization and authentication by implementing JAAS and registering all the used security roles in TAM. This is done during deployment time.

If you have an application with a lot of roles this is very frustrating because it can take a lot of extra time to start up (think of 30 minutes per application) because TAM is synchronizing all the new roles.

Most applications at this customer are using JAAS but do not have special method level authorizations implemented by using TAM. So only the roles are important.

After realizing this I thought is could be a good idea to create a simpler solution for integration TAM and JBoss. For this I wrote some custom code (only 250 lines).

Things to keep in mind

This method is using the iv-user and iv-groups headers at http transport level. Without transport level security (i.e. ssl) it is possible to fake this by using a man in the middle attack.

If you have proper firewall rules and security zones this does not have to be a problem.

When using this method you can use a mapping file for mapping TAM roles to application roles. With the TAM module this can be centrally managed. But since these roles normally only change at deployment time I don’t think this is a real problem.

The steps to use this solution

In the following overview you see the steps you need to take, after this overview I will go into detail for each step.

Step 0

Download the plugin including all sources here.

Step 1

Add the jboss_sso_tam.jar file to the server classpath (server/[profile]/lib)

Step 2

Set the authorization method to BASIC in the web.xml inside the application (application.war/WEB-INF/web.xml).

Set the roles in the web.xml (or in annotations). See http://java.sun.com/javaee/5/docs/tutorial/doc/bncav.html for more information about security roles in Java.

web.xml

Step 3

Add the TAM JBoss authorization Valve to the web deployer (apache). This can be done by editing jboss.web-deployer/META-INF/jboss-service.xml inside the deploy directory inside a server profile.

This authorization Valve is called before the servlet or static content is called for each request. It is important that the key is set to the same value as the authorization method in the web.xml inside the application (see step 2).

Jboss-service.xml

Step 4

Edit the login-config.xml file inside the server/[profle]/conf directory. Here you have to add the login-module to the application-policy which is used. In this example case I edited the jmx-console policy. The security domain for an application defaults to the name of the war or ear file. (i.e. jmx-console.war à jmx-console).

You can override the security domain in a jboss-web.xml file in the WEB-INF directory of an application.

login-config.xml

jboss-web.xml

After editing the login-config.xml file it is possible to use a mapping file for mapping the tam roles to application roles. Default you can use the server/[profile]/conf/props/tamroles.properties file for this.

The name of this file can be overridden with a module-option tag with the name tamrolesProperties.

tamroles.properties