Sometime in the bright future, you will be able to deploy the same virtual appliance containing your application to all your target environments without adjustments. For the time being, however, deployments to traditional DTAP¹ landscapes almost always mean "tweaking" the application and associated configuration and resources to match the target environment - think endpoints, properties files or datasource usernames and passwords, to name but a few.

In the absence of any established standards or even guidelines in this area, many different solutions to this problem of deployment package customization have been employed, from fairly elegant approaches such as JMX to crude string search-and-replace.
Furthermore, different types of middleware platforms have varying degrees of support for customizations: typically, portals, ESBs and process servers offer some "native" solution to the problem, whereas application servers tend to leave users to fend for themselves.

More often than not, the result is a chaotic mix of customization approaches across projects, target platforms and departments². Here, we'll look at some of these approaches, classify them and examine some drawbacks and benefits.

Important considerations: So what are we looking for..?

When comparing different approaches to customization, there are a number of things to bear in mind:

1. Convenience: The customization procedure should be easy to set up and quick to carry out. This is mainly a concern for developers that might have to execute a customization whenever the application is deployed to the development environment.
2. Visibility: It should be easy for appropriately authorized users to view both the customization points (i.e. which parts of the application can be customized) and the values assigned to them in a given environment.
3. Fail-safety: If an application is deployed with missing or invalid values for customization points (think forgetting to set a timeout or using the test endpoint for the production environment) this should be detected quickly. Preferably, missing or incorrect values would be detected at deployment-time, rather than becoming apparent only due to spurious runtime behaviour.
4. Revisioning and access control: Only appropriate users should be able to view and edit the values assigned to an application's customization points. Preferably, a history of changes to these values would be maintained. This aids the comparison of the values for an application across versions, as well as allowing users to compare the values for the same version of a deployed application across target environments.

Wait for it... Do this at runtime

As ever, a good way to approach a problem is to try to avoid it. In this case, you can side-step the need to tweak the application at deployment-time by deferring the lookup of application properties to runtime, using JMX (which, after all, was designed also for that purpose) with suitable default values. You get all the access control, history management etc. of the JMX implementation for free, and can update the environment-specific values without having to edit application files.

Of course, this only works if your container provides a suitably convenient JMX implementation - if not, getting your development environment set up is unlikely to be a lot of fun. More fundamentally, this will not work for attributes that are set before the application is started (e.g. the context root) or for properties of middleware resources (such as a queue retry timeout).

1. Convenience: Usually moderate, as the developer's environment tends not to be easy to set up, but this depends strongly on the target platform's support for JMX or whichever lookup mechanism is used. Needs to be balanced against the advantage of being able to change the behaviour of the running application.
2. Visibility: In the case of JMX, good, since it is easy to see all the MBeans and their properties, especially if a sensible naming convention is used.
3. Fail-safety: JMX will usually require "safe" default values (which should not be environment-specific).
4. Revisioning and access control: Again, depends on the target platform's implementation. But generally good, especially access control, as it is usually integrated with the management console of the middleware stack.

Replace me: Token-based replacement

Token-based replacement basically means placeholders – the deployment package contains (in artifacts, resource definitions etc.) special symbols, and at deployment time these symbols are replaced by values supplied for them. Token-based customization requires the provider, usually developers, to prepare the deployment artifacts specifically.

This means that, firstly, all the tokens that need to be replaced are³ known at the time of delivery, which has the added advantage that the application now has a well-defined set of customization points. The special syntax of tokens also makes it relatively easy to verify that values for all the tokens have been supplied.
Even if this verification fails, the application will presumably break due to syntax errors – tokens are usually not valid values – which provides an extra "fail-safe" mechanism.

Of course, from a developer's perspective this fail-safe mechanism can be a nuisance, too. Since the application doesn't work until the tokens have been replaced, build processes have to be set up to carry this out, and quickly, certainly in development environments.

1. Convenience: Moderate, since the build process has to be set up to correctly process files containing tokens. If you're working in a development environment like Eclipse with WTP it can be worse, especially if it interferes with "hot-update" capabilities.
2. Visibility: Wherever there is a token there is a customization points, so they are clear and self-documenting. If you're using JNDI, arguably the "right" way of doing token-based replacement in JEE, the assigned values are usually also very easy to see in the console.
   Gaining insight on the can be trickier if you're working with file-based search-and-replace as is usual with properties files, as you have to not only have the final artifact to see the values but also need to look at the original to see where the placeholders were.
3. Fail-safety: High, which in the context of a mission-critical activity such as deployment is crucial. Tokens are almost never valid values, so forgetting to assign a value will usually result in a failed deployment. Of course, there is still a risk of replacing the tokens with values for the wrong environment.
4. Revisioning and access control: Depends on how the values that are to be assigned to the customization points are provided. With JNDI, good revisioning and access control normally come "for free" with the console. As for the common case of properties files, we've seen a bit of everything: from files that are not versioned and shared with everyone as part of a "template project", to ones that are maintained in a dedicated version control system.

Replace that: Pointer-based replacement

Pointer-based replacement is essentially a fancy way of referring to "search-and-replace" and its slightly more advanced cousin, XPath-based replacement, commonly found in portal or ESB environments. This is fragile because the deployment package contains valid values, so there is a strong risk of silent failure. Further, the customization points of the package are essentially invisible. Ironically, this can be useful in order to "patch" packages that were not written in a customizable way.

It is generally true that it is much easier to prepare a deployment package for pointer-based customization – it is simply a matter of exporting, or making a snapshot of, the settings and artifacts of the development or other "authoring" environment. Of course, such an export can be "tokenized" (by replacing the values that need to be customizable with tokens), but this is error-prone and can involve prohibitive manual overhead, especially a problem during development when such exports are made frequently.

Obviously, pointer-based customization is only possible if the artifacts or definitions that need to be customized are structured in some way – otherwise, it is not possible to construct a pointer to refer to the item to be modified. XML and resource definitions (i.e. properties files) fall into this category, but e.g. plain text files do not.

1. Convenience: High. The source application contains valid values, so can run "as is".
2. Visibility: Customization points are not visible in the source files, although they can be deduced (with more or less ease) from the specification of the pointers that define where values need to be replaced. The values to be assigned are usually defined alongside the pointer specifications.
3. Fail-safety: Weak, since the source application contains (syntactically) valid values, even if the replacement is not or only partially carried out. This means it can take quite a while before the problem manifests itself, often in spurious ways - a maintenance nightmare.
4. Revisioning and access control: Depends again on how the pointers and values are specified and stored. Because the pointer specifications are usually contained in the same file as the values to be assigned, it is usually not possible to separate access to customization point (i.e. pointer) definitions - usually a developer activity - from access to the assigned values, which often should only be known to Operations.

So..?

If the customizations required for your application can be carried out after the application has started, it's certainly worth considering designing your application to be runtime-configurable. This makes management of the application convenient and secure, but of course adds some extra complexity at development time. And in some cases⁴, customization needs to be carried out before the application is loaded.

Whilst pointer-based replacement is convenient and can be used even with applications that were not designed to be customized, token-based replacement offers significant advantages in terms of (fail-)safety and visibility. Since deployments are often mission-critical, these are substantial benefits and should lead to tokens being preferred wherever possible.

Unfortunately, many of the existing middleware platforms have neglected to properly address this issue, yet another reason why a deployment automation solution like Deployit that is designed to support customization in a consistent and secure way can be a big benefit.

A word about environment-specific builds

Anyone who has tried, in the middle of a deployment, to follow vague instructions on how to customize the application, has probably wondered why this cannot be done in a less error-prone manner. As few people have suitable deployment automation in place, a common alternative is to try to integrate customization in the (continuous) build process.
From a technical perspective, this can certainly look like an easy option: there are great open-source and commercial continuous build products out there, and most organizations already have one in place. Many of the tools support the notion of "profiles" or similar customization mechanisms, and if not they all offer hooks to add in your own search-and-replace functionality.

The key disadvantage to this approach is procedural: environment-specific details need to be accessible during the build process. This shouldn't simple feel "wrong", some of this information can be highly sensitive - think passwords for the production database - making this solution infeasible from a security perspective. In addition, continuous build tools generally do not provide suitable repositories⁵ for these environment specific values, and env.properties files are notoriously prone to copy-paste and other errors.

And from painful experience: with environment-specific builds, it's only a matter of time before you have the test build of your application running, by accident, on the production environment.

Tags: customization, deployment package, DTAP
Filed under Deployment, Middleware, Xebia Labs | No Comments »