Customize This: Tailoring deployment packages to your target environments

Andrew Phillips

Sometime in the bright future, you will be able to deploy the same virtual appliance containing your application to all your target environments without adjustments. For the time being, however, deployments to traditional DTAP1 landscapes almost always mean "tweaking" the application and associated configuration and resources to match the target environment - think endpoints, properties files or datasource usernames and passwords, to name but a few.

In the absence of any established standards or even guidelines in this area, many different solutions to this problem of deployment package customization have been employed, from fairly elegant approaches such as JMX to crude string search-and-replace.
Furthermore, different types of middleware platforms have varying degrees of support for customizations: typically, portals, ESBs and process servers offer some "native" solution to the problem, whereas application servers tend to leave users to fend for themselves.

More often than not, the result is a chaotic mix of customization approaches across projects, target platforms and departments2. Here, we'll look at some of these approaches, classify them and examine some drawbacks and benefits.

Important considerations: So what are we looking for..?

When comparing different approaches to customization, there are a number of things to bear in mind:

  1. Convenience: The customization procedure should be easy to set up and quick to carry out. This is mainly a concern for developers that might have to execute a customization whenever the application is deployed to the development environment.
  2. Visibility: It should be easy for appropriately authorized users to view both the customization points (i.e. which parts of the application can be customized) and the values assigned to them in a given environment.
  3. Fail-safety: If an application is deployed with missing or invalid values for customization points (think forgetting to set a timeout or using the test endpoint for the production environment) this should be detected quickly. Preferably, missing or incorrect values would be detected at deployment-time, rather than becoming apparent only due to spurious runtime behaviour.
  4. Revisioning and access control: Only appropriate users should be able to view and edit the values assigned to an application's customization points. Preferably, a history of changes to these values would be maintained. This aids the comparison of the values for an application across versions, as well as allowing users to compare the values for the same version of a deployed application across target environments.

Wait for it... Do this at runtimecustomization-runtime-lookup

As ever, a good way to approach a problem is to try to avoid it. In this case, you can side-step the need to tweak the application at deployment-time by deferring the lookup of application properties to runtime, using JMX (which, after all, was designed also for that purpose) with suitable default values. You get all the access control, history management etc. of the JMX implementation for free, and can update the environment-specific values without having to edit application files.

Of course, this only works if your container provides a suitably convenient JMX implementation - if not, getting your development environment set up is unlikely to be a lot of fun. More fundamentally, this will not work for attributes that are set before the application is started (e.g. the context root) or for properties of middleware resources (such as a queue retry timeout).

  1. Convenience: Usually moderate, as the developer's environment tends not to be easy to set up, but this depends strongly on the target platform's support for JMX or whichever lookup mechanism is used. Needs to be balanced against the advantage of being able to change the behaviour of the running application.
  2. Visibility: In the case of JMX, good, since it is easy to see all the MBeans and their properties, especially if a sensible naming convention is used.
  3. Fail-safety: JMX will usually require "safe" default values (which should not be environment-specific).
  4. Revisioning and access control: Again, depends on the target platform's implementation. But generally good, especially access control, as it is usually integrated with the management console of the middleware stack.

Replace me: Token-based replacementcustomization-token

Token-based replacement basically means placeholders – the deployment package contains (in artifacts, resource definitions etc.) special symbols, and at deployment time these symbols are replaced by values supplied for them. Token-based customization requires the provider, usually developers, to prepare the deployment artifacts specifically.

This means that, firstly, all the tokens that need to be replaced are3 known at the time of delivery, which has the added advantage that the application now has a well-defined set of customization points. The special syntax of tokens also makes it relatively easy to verify that values for all the tokens have been supplied.
Even if this verification fails, the application will presumably break due to syntax errors – tokens are usually not valid values – which provides an extra "fail-safe" mechanism.

Of course, from a developer's perspective this fail-safe mechanism can be a nuisance, too. Since the application doesn't work until the tokens have been replaced, build processes have to be set up to carry this out, and quickly, certainly in development environments.

  1. Convenience: Moderate, since the build process has to be set up to correctly process files containing tokens. If you're working in a development environment like Eclipse with WTP it can be worse, especially if it interferes with "hot-update" capabilities.
  2. Visibility: Wherever there is a token there is a customization points, so they are clear and self-documenting. If you're using JNDI, arguably the "right" way of doing token-based replacement in JEE, the assigned values are usually also very easy to see in the console.
    Gaining insight on the can be trickier if you're working with file-based search-and-replace as is usual with properties files, as you have to not only have the final artifact to see the values but also need to look at the original to see where the placeholders were.
  3. Fail-safety: High, which in the context of a mission-critical activity such as deployment is crucial. Tokens are almost never valid values, so forgetting to assign a value will usually result in a failed deployment. Of course, there is still a risk of replacing the tokens with values for the wrong environment.
  4. Revisioning and access control: Depends on how the values that are to be assigned to the customization points are provided. With JNDI, good revisioning and access control normally come "for free" with the console. As for the common case of properties files, we've seen a bit of everything: from files that are not versioned and shared with everyone as part of a "template project", to ones that are maintained in a dedicated version control system.

Replace that: Pointer-based replacementcustomization-pointer

Pointer-based replacement is essentially a fancy way of referring to "search-and-replace" and its slightly more advanced cousin, XPath-based replacement, commonly found in portal or ESB environments. This is fragile because the deployment package contains valid values, so there is a strong risk of silent failure. Further, the customization points of the package are essentially invisible. Ironically, this can be useful in order to "patch" packages that were not written in a customizable way.

It is generally true that it is much easier to prepare a deployment package for pointer-based customization – it is simply a matter of exporting, or making a snapshot of, the settings and artifacts of the development or other "authoring" environment. Of course, such an export can be "tokenized" (by replacing the values that need to be customizable with tokens), but this is error-prone and can involve prohibitive manual overhead, especially a problem during development when such exports are made frequently.

Obviously, pointer-based customization is only possible if the artifacts or definitions that need to be customized are structured in some way – otherwise, it is not possible to construct a pointer to refer to the item to be modified. XML and resource definitions (i.e. properties files) fall into this category, but e.g. plain text files do not.

  1. Convenience: High. The source application contains valid values, so can run "as is".
  2. Visibility: Customization points are not visible in the source files, although they can be deduced (with more or less ease) from the specification of the pointers that define where values need to be replaced. The values to be assigned are usually defined alongside the pointer specifications.
  3. Fail-safety: Weak, since the source application contains (syntactically) valid values, even if the replacement is not or only partially carried out. This means it can take quite a while before the problem manifests itself, often in spurious ways - a maintenance nightmare.
  4. Revisioning and access control: Depends again on how the pointers and values are specified and stored. Because the pointer specifications are usually contained in the same file as the values to be assigned, it is usually not possible to separate access to customization point (i.e. pointer) definitions - usually a developer activity - from access to the assigned values, which often should only be known to Operations.

So..?

If the customizations required for your application can be carried out after the application has started, it's certainly worth considering designing your application to be runtime-configurable. This makes management of the application convenient and secure, but of course adds some extra complexity at development time. And in some cases4, customization needs to be carried out before the application is loaded.

Whilst pointer-based replacement is convenient and can be used even with applications that were not designed to be customized, token-based replacement offers significant advantages in terms of (fail-)safety and visibility. Since deployments are often mission-critical, these are substantial benefits and should lead to tokens being preferred wherever possible.

Unfortunately, many of the existing middleware platforms have neglected to properly address this issue, yet another reason why a deployment automation solution like Deployit that is designed to support customization in a consistent and secure way can be a big benefit.

A word about environment-specific buildscustomization-build-process

Anyone who has tried, in the middle of a deployment, to follow vague instructions on how to customize the application, has probably wondered why this cannot be done in a less error-prone manner. As few people have suitable deployment automation in place, a common alternative is to try to integrate customization in the (continuous) build process.
From a technical perspective, this can certainly look like an easy option: there are great open-source and commercial continuous build products out there, and most organizations already have one in place. Many of the tools support the notion of "profiles" or similar customization mechanisms, and if not they all offer hooks to add in your own search-and-replace functionality.

The key disadvantage to this approach is procedural: environment-specific details need to be accessible during the build process. This shouldn't simple feel "wrong", some of this information can be highly sensitive - think passwords for the production database - making this solution infeasible from a security perspective. In addition, continuous build tools generally do not provide suitable repositories5 for these environment specific values, and env.properties files are notoriously prone to copy-paste and other errors.

And from painful experience: with environment-specific builds, it's only a matter of time before you have the test build of your application running, by accident, on the production environment.

Footnotes

  1. Development, Test, Acceptance, Production
  2. Chatting about customization during a WebSphere Process Server 'expert session' at Impact 2010, one of the WPS developers mentioned that there must be "oh, probably 20 ways to customize a deployment". And he wasn't even joking!
  3. Should be ;-)
  4. Context root, data source username etc.
  5. Versioned, secured etc.

Comments (4)

  1. Mark van Holsteijn - Reply

    September 1, 2010 at 4:33 pm

    Andrew,

    Nice article and good to see that you have named the different styles of customizing deployments.

    In a solution that I created for a customer I introduced the concept of configurable deployment units. A configuration deployment unit contains all the relevant instructions to tailor the deployment unit to a specific environment. A important feature of this deployment unit is that the configuration operation itself is idempotent. Repeated configuration of the same deployment unit delivers the same result (unless the parameter values change).

    A self contained configurable deployment unit is a specialization which also contains all the parameter values for the different environment (D, T, A and P).

    The deployment configurator executes the actual instructions, provides the visibility and safety features that you claim are inherent disadvantages of the pointer based replacement pattern. You could even improve the tooling surrounding the configurable deployment unit to provide more insight into the configuration process to the developer and deployer alike (For instance through an eclipse plugin).

    In addition, the configurator does support token based replacement as well but you have to explicit instruct it and provide a source and target name to ensure idempotency of the configuration.

    The reasons I prefer pointer based above token based are:

    1. Pointer based is precise. You will NOT have any accidental replacements and if it performs a incorrect replacement, it is consistently incorrect ;-)
    2. Pointer based is non-intrusive: No changes are required to the source files themselves.
    3. It supports strict separation of responsibility between developer and deployer, although best results are achieved when they cooperate.
    4. Above two features allows you to configure any deployment unit, not just the once that were prepared by the developers.

    Cheers,

    Mark

  2. Andrew Phillips - Reply

    September 6, 2010 at 1:51 pm

    @Mark: the tool you describe sounds like a flexible and quite powerful customization solution. If I understand correctly, it is based around a "customization descriptor" file that contains the pointer definitions (the actual values are also specified in the package or are supplied externally at "customization time")?

    I can see how the customization descriptor, assuming it is written in a fairly human-readable way, can provide some visibility into the customization process, especially if the customizations for the entire package can be reviewed in one file. Pointer syntax (think XPath) can be hard to interpret, but that be mitigated by including a plain text description of what is actually being pointed to.

    I'd be more curious to know how this approach provides safety, however. Idempotency prevents you from screwing up if you run the customization multiple times, but that doesn't address the issue of customizations not being applied at all. Of course, your tool may somehow integrated with the app server...

    Regarding the other points:

    > 1. Pointer based is precise. You will NOT have any accidental replacements and
    > if it performs a incorrect replacement, it is consistently incorrect

    Agreed, although with a suitable placeholder syntax (${...} or whatever) accidental replacements should not happen either. Certainly, plain string search'n'replace is something to avoid if at all possible.

    > 2. Pointer based is non-intrusive: No changes are required to the source
    > files themselves.

    Indeed - perhaps the main reason why pointer-based replacement scores 'High' for convenience.

    > 3. It supports strict separation of responsibility between developer and
    > deployer, although best results are achieved when they cooperate.

    That depends largely on whether or how the customization descriptor splits the pointer specification from the specification of the actual values. This also determines, to a large extent, how well the system can deal with sensitive data that should only be seen by deployers (the production database password, for instance).

    > 4. Above two features allows you to configure any deployment unit, not just the
    > once that were prepared by the developers.

    Indeed - that's what I meant by "Ironically, this can be useful in order to "patch" packages that were not written in a customizable way." ;-) It shouldn't be forgotton, though, that pointer-based replacement only works if the targeted files have a suitable structure where elements can reliably be pointed to.

    For .properties files this is fairly trivial, and XML supports powerful, fine-grained pointers via XPath (although constructing these for e.g. a piece of WebSphere configuration XML can be a non-trivial exercise!), but modifying an SQL script via pointers can present some interesting challenges.

  3. Mark van Holsteijn - Reply

    September 17, 2010 at 7:36 pm

    When a file does not have a particular structure, the configuration instructions work off a template file that contains references to configuration variables. In this case, you have to specify a source and a destination in the deployment unit. e.g.

    replace-properties from sql/.create-table.sql to sql/create-table.sql

    Configuration variables do not have to be included and can be specified at configuration time. This allows the administrator to keep the security sensitive properties apart from the general settings.

    Cheers,

    Mark

  4. [...] for a single CI build that delivers your "gold standard" deployment package which can be customized (e.g. using configuration files with placeholders) for the target environment at deployment time. [...]

Add a Comment