There has been much cool stuff lately about devops and devs and ops working together in one team, like at sky.com. The uncool reality for a lot of companies is that dev and ops are separated in different departments and don´t communicate well. Immature applications, at least from a middleware perspective, are what you get.

One of the aims of middleware is to provide standard components to application programmers. Two of the nicest and widespread examples of this are JDBC and the datasource. JDBC, being an industry standard, is a powerful abstraction delivered by software vendors to enable communication to their database product. The datasource is another abstraction, delivered by application server administrators, to enable communication to the database that their company uses. The latter hides features like connection pooling, statement caching and database specific properties. In the early days of J2EE, though, you were likely to find applications that did not use these features. Instead the application programmers had decided to build them from scratch. A lot of Xebians have in the past removed these homegrown “frameworks” and replaced custom constructs with datasources and the like. These days are over by now….or, so it seems.

It is no longer the homegrown datasources that should worry us. Application reinvents platform, again. Why? Because developers don´t know the platform their applications run on. The reason: middleware products have become vastly complicated, their range expanded into remote galaxies, their depth unknown as the depths of the ocean. Am I exaggerating?
Architects can no longer focus on the mere coding part of an application. The days of just servlets and EJB´s are over. We are confronted with ESB´s, BPM, J2EE, CMS´s and growing integration needs with all kinds of back-ends. An immature application would be one that reinvents an ESB. Or consider the company in which the middleware department has bought a BPM platform (like Oracle SOA suite, or Websphere process server) but at the same time is implementing web services choreography using plain java, because no architectural guidelines (that dictate the use of BPM) have yet been presented to them. The term “immature” is relative to the “maturity” of the middleware stack at hand.

I´m not saying that you are obliged to use the middleware vendor solution for any given problem. Take web services. There are plenty mature open source stacks that will enable you to build them and not use the (outdated) vendor solution. It’s up to the architect to make such decisions. (What is the fundamental difference between a library and middleware? Both should make your coding life easier).

All enterprisy applications need a minimum level of quality regarding deployability and maintainability:

Developers should know and use these rules. Administrators should know and enforce them.
All companies that want to use their middleware effectively need someone who can act as a stakeholder for the administrative department. He, she, or they must actively spread the middleware knowledge among development teams, architects, designers and administrators. Analysis must result in the identification of generic services that can and will be reused. The architecture and the working software should be implemented accordingly. If necessary, a set of criteria must be issued that serves as a barrier against immaturity. Envision a tool that inspects developers deliverables (EAR files) and generates a report that clearly states if an application passes or not (AFAIK there is no such thing).

The following example is loosely based on the situation at one of our recent customers. They have a pretty impressive set of applications and services that together implement mission-critical functions for various types of end-users. The backend consists of pre-web technology that is accessed via web services. The backend application is roughly ten years old, has had its fair share of problems in the past but is now quite stable. The team has developed a release schedule they feel is vital for ensuring stability. Every three months after rigorous testing in the Development environment, the new release is put in Test. From there on, the application is eventually rolled out in Production. It has been like this for years.

The web teams have a Development environment as well (not being their desktops), but it is populated with odd developer data, which makes it unusable for testers. So the testers maintain their test-data in the Test databases. They all work in Scrum teams (unlike the backend team), so testing here is a continuous activity. You can by now guess what the problem is. The Scrum testers will have to wait three months to get their features tested, because they very much depend on the backend system.

To add to this, there´s also a set of user stories that need interaction with external companies. The network guys have decided that their firewalls will only be really opened in the Acceptance environment. So, only after all functional testing has been done in Test, and in a stage of the project where the ultimate deadline approaches (you know the deal, marketing has already started a radio campaign, stating that on that day the website will be totally “new and improved“), only then you are able to test the b2b functions. Of course all project planning has turned out a little too optimistic, but so far, everything is going pretty smoothly.

Less than a week before going live someone in the web team discovers that his application in Acceptance is actually connected to the security component in Test. Acceptance and Production is 100% equal (as it should) but Test differs in unclear ways. So the Acceptance component has never been tested. Now suddenly, there´s no guarantee that Production will work! One day later: more horror: security in Test and Acceptance is totally different. In Acceptance, it does not work!

You can imagine the stresslevels at this time. Luckily the story ends well, although it cost them a lot of hard work.

The problems exist at different levels.
• Organizational: different teams have different views and practices concerning the test environments. In the example this resulted in testing too late.

• Procedural: Changes to the environments are not evaluated and recorded (or only in production). You will have spaghetti!

• Conscious decisions: The firewall policy in the example. Not bad in itself, but results in late testing.

• Errors: unexpected or wrong connections between components in different test environments.

• Documentation debt: the lack of adequate up-to-date technical documentation (diagrams).

• Negligence: changes to one environment are not replicated in other environments.

• Mindset: “this error will probably not show up in acceptance and production”. This belief originating in existing DTAP differences will open the door for mister Murphy, who is sure to enter the room.

• Scaling differences: the Development environment has no clustering. This could implicate errors later on, due to applications not being able to be clustered.

The longer companies allow DTAP problems to exist, the harder they will be to correct. That is no rocket science. Probably everyone in IT knows it, including the responsible managers. It might be a little bit difficult to explain to the financial guy, the product owner or the customer. It´s like code refactoring. You will have to explain the need for internal housekeeping in order to get the necessary budget.

So what to do? Again this is no rocket science. The main point of this blog is that you need the awareness that things are wrong and that this will result in delayed projects, having more bugs than necessary, rising stress levels, frustration or anger between developers, testers, and administrators.

Some words of advice:
• Turn the right (amount of) knobs. All changes must be approved and coordinated by a change manager. Replicate the changes. Document them. Test the test environment after every change

• Keep it simple. Do not introduce too many differences, because of lower costs. It will increase administrative burden. So create that cluster in Development, but scale to a lower number of cluster members.

• Get your teams aligned. They must share the way in which they treat a test environment, including release cycles.

• Be defensive. Be pessimistic. Pessimistic people are often right. An administrators’ error often has far bigger impact than a programmers’. Test everything.

Imagine a typical middleware team in a fairly sized company. John is the manager of this team. The middleware team is continuously struggling to keep up with all the new products that are introduced by development teams and their solutions. The team members and Unix administrators Leo and Lloyd are maintaining the J2EE application servers, web servers and a queuing product to connect to the old company’s mainframe. They are fairly successful in their jobs, at least the number of incidents has subsided, performance is reasonably good, they know how to deploy new software versions, and they do middleware upgrades in the evening hours.

The Unix gurus Leo and Lloyd both have a long history with the company and will probably stay with the company until their retirement. The new guy Dave (originally also a Unix administrator) has undertaken the effort of getting to know all this new fancy Java stuff. When Dave is not in the office, Leo and Lloyd are a bit helpless as far as the application server is concerned.

Dave recently improved the availability of the company’s website by introducing clustering, has written deployment scripts to improve on deployment speed and quality, and worked with developers to get better log statements from the application. He shows ambition. He works late hours, seemingly ignoring the continuing stream of incidents and the nagging customers at the helpdesk. Over and over John praises Dave, provides him with raises and bonuses, persuades him with some success to spread his knowledge and hires new people to assist him (they all left by the way).

John has become dependent on Dave for his day-to-day operations and middleware development surrounding the growing middleware management stack and this situation worries John. If Dave ever decides to leave, the company will be in big trouble. A new project wants to introduce BPM tooling and Dave is already reading the product documentation. Leo and Lloyd are not able to keep up with Dave, and by the way, BPM tooling is way out of their league. Hiring an expensive external consultant could ease the pain here, but only temporarily. Some knowledge of the BPM product should be secured in the organization. John has a hard time guaranteeing website stability and supporting innovation in new projects.
Dave is willing, but cannot share his intimate knowledge of the application server, because there’s no one able to understand anything beyond the basics. Dave himself likes his job, but might have other goals and dreams as well, like an occasional holiday or raising a family, things that interfere with the weekend and midnight work. Moreover he has just found out that he is actually tired of the multitude of repetitive tasks he has to deal with every day. He’d rather be a developer, or even an architect. How did he end up here?
John has a “dependency problem”, because he has become “addicted” to Dave who is the “hero” that keeps the middleware operation and middleware development going. This is a risk to the availability of the company’s website and the projects that need work done in test environments. The company may lose customers and projects might run late.
There is a growing need for experts in the middleware field that have more skills than the average administrator, but do understand the administrator’s problems and worries. Often middleware experts are external consultants who assist in development projects and move on to new ones instead of sticking to work they find mindless. So what can John do to maintain a healthy team of middleware oriented professionals?
Above all, John needs to find a person that Dave can team up with. John could try and motivate Leo or Lloyd (or both) to learn the skills needed to get to a certain level of expertise that Dave recognizes. The problem here is that Leo and Lloyd are excellent Unix administrators, Java and BPM are out-of-scope. Another problem might be the (lack of) ability of Leo and Lloyd to learn this new fancy Java stuff and the BPM middleware.

If John can prove to his manager that the middleware team is not properly scaled with regard to the amount of work (the team is undersized) then John could try and find one or more skilled middleware professionals and expand the team. John might be facing a big challenge here because skilled middleware professionals are pretty hard to come by these days, unless you are willing to pay.

Dave’s assignment will be to “share knowledge”. But if Dave does not want to cooperate then John needs to persuade him to change his behavior. He also needs to stimulate Dave to share his knowledge, by stating his expertise in the matter, and by emphasizing the negative effects of his behavior (like his own fatigue or the risk of downtime if he’s on holiday). If that does not help then John might want to dig deeper to find out Dave’s personal motives. Is it competitiveness, prestige, perfectionism or lack of social skills? This might become a psychological exercise.

The problem could also be in the technology itself. If John is not able to support the system with his team members, he could try to eliminate complexity, standardize as much as possible, take out error-prone procedures and automate boring repetitive tasks. Or switch to other technologies that are widely supported.

As it turns out, quite a number of us have been in hero-like situations ourselves. Looking back I ask myself: “What kept me there in the first place? Was it the sense of being needed that felt pleasant? It stopped me from growing as a person and as a professional. I’m happy I left in the end and guess what, they still exist…”

as [a] software layer [which role] is to make application development easier,
by providing common programming abstractions,
by masking the heterogeneity and the distribution of the underlying hardware and operating systems,
and by hiding low-level programming details

This makes sense, considering the fact that writing an http server nowadays or a servlet-container is not considered sane anymore, given the multitude of commercial and open source products that have already proven themselves. Over the years a range of products and standards has emerged that to a growing extent hide the low-level intricacies and provide the application programmer with easy yet powerful abstractions. They range from webservers, databases and application servers to EBS´s and BPM platforms. They form the IT landscape that enable modern business. And it´s their heterogeneity and distribution that is at the heart of the emerging problems that we will address in a top-10 style series of blogs in the coming weeks.

Middleware adds a new level of complexity

Application servers (for instance) solve a lot of problems regarding web-based applications. With it come higher level problems, regarding enterprise wide infrastructure and integration. Middleware problems occur typically in a medium to large enterprises that have seen their primary business become more and more dependent on internet based applications. The problems have their origins and solutions in technology, as well as in organizational psychology. It’s about people, their knowledge, span of control and their being involved in other processes in the company.

The 10 issues have formed our frame of mind that we use to assess companies that ask us to audit their software development and operation processes. They all have had to find answers to problems that application servers have not solved satisfactorily like deployment or synchronizing test and production environments. They have come to realize the fact that middleware is (often) the battleground where developers and administrators meet. And they have trouble finding people with the right technical skills and at the same time have the ability to mediate between these two groups.

Middleware is here to stay

Writing software is to a growing degree about combining existing components using code and configuration. Integrating the wide range of infrastructural components requires detailed knowledge of the platform and the ability to guide developers. This typically is an architect’s job . This architect should also be in contact with business consultants, to be able to wield the power of middleware to underpin the success of business initiatives in the long run. While projects deal with immediate needs, the middleware is a long-term investment, requiring revision over time. Failing to understand this is a recipe for trouble.

Middleware requires management

Middleware management is not solely about application development, or application lifecycle management, nor is it part of a helpdesk or just another form of systems management. It´s about all of that and more. It´s at the heart of an IT organization. This means that middleware people should be involved in decision making processes, which is often not the case. On the other hand, the middleware people should streamline their work, stay updated on technology and work along principles and best practices, or they will be an impediment for the projects they support.

Number 10 on the list is the “dependency problem”, meaning: systems availability should not depend on that one expert alone.

Part1 Starting your own threads.

As long as I worked with application servers, people have always told me not to start my own threads, because the j2ee specification states that this is forbidden. These threads are also referred to as ‘naked’ and ‘unmanaged’. The danger they pose is doing things that the application server knows nothing about. It could cause resource leaks, no debugging, failing to stop a server or security problems.

Yet there is a number of open source frameworks that do just this, and no one seems to object. Think of quartz, or Log4j (the watchdog that monitors changes in log4j settings). And even the jdk itself is guilty: use of java.util.Timer also causes so called unmanaged threads.

And until now the application servers I know haven´t objected either. So applications were actually free to spawn whatever they want and get away with it.

There was also a reason not to change this situation. The way in which it worked made the application dependent on application server internals. In the case of websphere, the method is fully supported by IBM and didn´t change in websphere 6 for instance; yet developers are reluctant to build software that won´t run on other application servers. This certainly makes migration easier.  A disadvantage (of creating your won threads) is that you´re on your own when it comes to transaction management and concurrency control.

The way it worked was:

implement com.ibm.websphere.asynchbeans.Work, obtain a workmanager from jndi and submit the Work class to the manager.

For some time now there is commonj. This is a set of interfaces that have to be implemented by j2ee vendors.
Now you can implement commonj.work.Work, obtain a workmanager from jndi and submit the Work class

The main advantage of this is that commonj can be supported by other application servers. Weblogic 9.0+ does it, geronimo, and sadly no other, until now (afaik). So your application is now only a bit more portable. I think commonj could be made to work with jboss AS. You´d have to implement the workmanager (and other classes) yourself, as an Mbean.

The spring javadoc mentions the lack of official vendor support for commonj with some regret. And I have not been able to find any information about the origins and the future for commonj.
Talking of Spring, they provide a couple of very useful classes that simplify even further the above procedure:

create a org.springframework.scheduling.commonj.WorkManagerTaskExecutor instance, providing a jndi name and submit any Runnable class. You only need commonj on your build path, which is in the spring dependencies.

If you happen to run in websphere 6.1 you´ll have a WorkManager by default using wm/default for the jndi lookup. At lookup a log entry will appear stating that you didn´t use a resource reference. Yet it will all work just fine.

Add the resource reference by putting the next lines in your web.xml


<resource-ref>
   <res-ref-name>wm/WorkManager</res-ref-name>
   <res-type>commonj.work.WorkManager</res-type>
   <res-auth>Container</res-auth>
   <res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>


The deployer will have to map this to the actual WorkManager at deploy time.

Spring will add java:comp/env to your jndi name if you set the flag resourceRef to true.

Both commonj and spring also support scheduling of jobs. This could be a safer way than using quartz, especially if your jobs need database access.

Configuring the WorkManager is another thing. Your administrator will have to know how many threads you expect. In Websphere you can also specify if the JAAS subject of the scheduling thread will have to be passed on to the scheduled thread. If disabled, the scheduled thread will have no security context. It will not be able to call EJB´s or other services (unless they do their own login).

The growable property makes the threadpool maximum obsolete and any number of threads can be started. This is all right as long as there are enough resources. It’s probably a good setting for finding out the number of threads in certain situations (like under load). Once you´ve established this number, apply it as the maximum for the threadpool and disable growable.

Conclusion

There is no reason to create naked threads. Aapache and two major vendors support commonj. Existing applications and frameworks that run in a j2ee container should be adjusted to at least try to obtain a commonj WorkManager, before resorting to creating threads itself. Spring accomodates for this nicely. Talk to your administrator to sort out the details of the threadpool.

Recomended reading

http://www.devx.com/Java/Article/28815/1763

http://download.oracle.com/docs/cd/E12840_01/wls/docs103/config_wls/self_tuned.html

http://www.devwebsphere.com/devwebsphere/2005/06/async_beans_pro.html

There´s another problem: JCL  needs to be told that it has to use log4j. By default the JDK logger is used and every line ends up in SystemOut.log.  The IBM documentation on this is clear: simply put a file called META-INF/services/org.apache.commons.logging.LogFactory on the classpath and you´re ready. The contents of this file turned out to be the puzzling part for me. I wrongly assumed that the file should be the same as the commons-logging.properties:

org.apache.commons.logging.LogFactory=org.apache.commons.logging.impl.Log4jFactory

I did what (I thought) the documentation  told me, redeployed my application and everything seemed fine. Until I changed classloading mode to PARENT_FIRST. Websphere applications can have this setting or APPLICATION_FIRST. In the latter mode jar files bundled with the application will be used first, before loading standard websphere libraries.

When you´re dealing  with an application you did not write and cannot change (say you´re the deployer),  PARENT_FIRST is your friend in the face of evil developers. I mean evil  as in: including  j2ee.jar in the earfile. It happens!! And not just this file, but anything you can find in a Maven repository today, containing lowlevel functionality (like xml processing, or j2ee api’s ). They should not be in the earfile!! Using PARENT_FIRST all correct versions of the platform classes will be loaded from the right location. Use this option and then nag the developers.
Actually the logging behavior in my case was very inconsistent. Sometimes APPLICATION_FIRST  didn´t work either. And sometimes a server restart changed the logging. It was a frustrating experience, and in those moments, you´re happy to fall back on java decompiling using good old JAD.

The source for org.apache.commons.LogFactory.class made it clear that the above configuration line is  wrong, and that you will NEVER get a warning if you fail to do it right. The class loads the file and the complete line is then interpreted as the classname for the LogFactoryImpl. So it´s not a properties file, and the correct contents for META-INF/services/ org.apache.commons.logging.LogFactory are:

org.apache.commons.logging.impl.Log4jFactory

And that´s all there is to it. Thanks Apache, for swallowing the  ClassNotFoundException !

Actually after some searching I came across a less well known part of the official SUN jarfile specification which states that service interfaces can have certain implementation classes, which are specified in a file that has the name of the interface. It´s a really rudimentary IOC implementation, that must have been fashionable back in the days of jdk 1.3.

I really didn´t know all this. It turned out to be a learning experience after all.

So what do you do? You´ll probably need a local websphere 6.1 installation, because it´s easier to use a websphere JVM to connect to a server. In this example I have done this.
First go to WAS_HOME/profiles/dmgr01/properties/sas.client.props and set the following properties (assuming you have security enabled):

com.ibm.CORBA.loginSource=properties
com.ibm.CORBA.loginUserid=<your websphere username>
com.ibm.CORBA.loginPassword=<your password>

Then start WAS_HOME/java/bin/jconsole using the following options:

-J-Djava.class.path=WAS_HOME/java/lib/tools.jar;WAS_HOME/runtimes/com.ibm.ws.admin.client_6.1.0.jar
-J-Djava.naming.factory.initial=com.ibm.websphere.naming.WsnInitialContextFactory
-J-Dcom.ibm.CORBA.ConfigURL=file:WAS_HOME/profiles/dmgr01/properties/sas.client.props
-J-Dcom.ibm.SSL.ConfigURL=file:WAS_HOME/profiles/dmgr01/properties/ssl.client.props

Fill in WAS_HOME your self and mind the semicolon on unix!
dmgr01 is my deployment manager profile
The -J gives you the opportunity to enter any standard JVM option, so jconsole -J-Xmx500M
would be the way to set the heap to 500Mb.

Then the gui pops up. Go to the advanced tab and enter service:jmx:iiop://[host]:[port]/jndi/JMXConnector where port is the bootstrap port for the deployment manager. Connect without entering credentials (they´re already in the file)

NB. You´ll have to watch your command shell, because it may prompt you to add the SSL certificate signer to the trust store.

What you then get is a slightly different jconsole. It only shows you the tab containing the available mbeans instead of all monitoring like options a Sun jvm offers out of the box (like cpu heap and usage counters). Instead you have all the mbeans needed to manage the cell. Every function in be it in the web console or wsadmin can be found here. This is all really neat, but not actually helpful, because you already had administrative tools. Let´s think of something useful.

Managing the SI Bus.

Through Mbeans you can also do things not available in the console. Like manage messages in queues.
So I set up a Service Integration Bus (using the webconsole), defined a queue and set up JMS resources: a queue reference to the bus queue and a queue connection factory. Bus member was my application server.

Next I connected the console to the application server (being the bus member). Look up the bootstrap port (typically 2809) and replace it in the JMXConnector url. The bus is not a separate process in websphere. Instead the messaging engine becomes part of a server or cluster when it is added as a busmember.

In the server1JMSBasicfunction Mbean there´s an enqueue operation that I used to put the message “hi world” on the queue. Another Mbean called server1BasicAdministration contains a browse operation. Et voila.

All this would be a a quick way to set up tests for Message Driven Beans for example. You could write also unit tests that call the Mbean to do the same.