Four weeks ago my colleague Robert van Loghem wrote about what the full scope of a deployment in an enterprise Java environment. The scenario that Robert presented included the configuration of databases, web servers, firewalls and Java EE resources. All are just as important to get your application up and running as the deployment of an EAR file.

But even though Robert's procedure was more complicated than the simple installation of an EAR file, it did not cover an even more common scenario; upgrading an application to a new version. While the initial deployment of an application is important, I think that upgrades are even more interesting for a number of reasons:

• The most obvious reason is that upgrades simply happen more often than initial deployments.
• Upgrades (in production) happen when users are already using the application, so the application should remain accessible during the deployment.
• If the upgrade fails, the old version of the application should still be there.
• And finally, but this is a more people oriented aspect of it, upgrades can be tricky because the focus on the application may have disippated over time. When the application is deployed for the first time, everybody is paying attention: the developers, the middleware administrators, the architects, the project managers, the end users, etc. So when a problem arrises, it is usually addressed quite quickly. But when version 2.1.54 comes around and an administrator is asked to install it on a Friday afternoon, that is when things can go horribly wrong and ruin the poor admin's weekend. Paying attention to the complexity of application upgrades, can mitigate this.

Because the upgrade scenario is the most important deployment scenario, it is something that comes up often when talking to clients about Deployit, our deployment automation product.

Let's go back to Robert's scenario. The application that was deployed in that scenario consisted of an EAR file, some static content (HTML files, images, etc.), and a number of SQL DDL scripts. So how should we upgrade that application to the next version, say v1.1? For the sake of simplicity, let us assume that the database schema hasn't changed between v1.0 and v1.1.Basically there are two scenarios imaginable. Either we undo the whole deployment of v1.0 and do a clean deployment of v1.0, something we like to call a full redeployment, or we do the minimal amount work to do the upgrade, an incremental deployment.

Incremental deployment scenario

An incremental deployment would look like this:

1. Remove static content for v1.0 from the web server.
2. Stop v1.0 of the EAR file.
3. Undeploy v1.0 of the EAR file.
4. Deploy v1.1 of the EAR file.
5. Start v1.1 of the EAR file.
6. Copy static content for v1.1 to the web server.

Full redeployment scenario

And a full redeployment would look like this:

1. Stop the web server.
2. Remove static content for v1.0 from the web server.
3. Remove the web server configuration for v1.0.
4. Stop the application server.
5. Undeploy v1.0 of the EAR file.
6. Destroy the application server configuration needed for v1.0; datasource, queues, server definitions, etc.
7. Create the application server configuration needed for v1.1; datasource, queues, server definitions, etc.
8. Deploy v1.1 of the EAR file.
9. Start the application server.
10. Create the web server configuration for v1.1.
11. Copy static content for v1.1 to the web server.
12. Start the web server.

If you compare these two scenarios, you will notice a number of differences:

• The web server is stopped at the start of scenario and only started as the very last step (steps 1 and 12 respectively).
• The web server configuration is removed and recreated (steps 3 and 10 respectively).
• Instead of just stopping and then starting just the EAR file, the whole application server is stopped and then started (steps 4 and 9 respectively).
• The application server configuration is removed and recreated (steps 6 and 7 respectively).

By removing and then recreating the web server and application server configuration in this deployment scenario, we will know exactly what the results will be. Any changes done to the configuration that are not documented in the deployment scenario will be removed and will not be recreated. To make it possible to do this, we have to stop the web server and application server and then start them again near the end of the scenario. This might negatively impact the availability of your application, but you know you need to set up a cluster to make your application highly available anyway, don't you?

What's also very interesting about the full redeployment scenario is that steps 7 through 12 look exactly like the initial deployment scenario for this application. The only things that are missing are the steps needed to create the database schema. Because the database keeps state it is not feasible to drop and recreate the schema. And looking at the full redeployment scenario some more, we see that steps 1 though 6 are what needs to be done to undeploy the application for good. Let's sum up the advantages and disadvantages of both scenarios:

Pros and cons of incremental deployments

• Pro: Incremental deployments execute very fast because they do the minimal amount of work.
• Con: Incremental deployments leave any manual modifications made intact. This is bad because it means there are magical, probably undocumented, configuration changes necessary to get the application to work. If you needed to migrate to new hardware, you would forget about this option again. And if the modification was done to one of the deployable artifacts, well... do I really need to explain?
• Pro: Incremental deployments don't touch systems that are already OK. Then again, if you are afraid to touch your systems because you fear they might otherwise break, you have work to do. You need to be able to trust your middleware environment so much so that you can make any change and restart any server without being afraid it might not come up again.
• Con: Incremental deployments are hard to undo. You need to define a separate undo scenario for every incremental deployment because you can't just redo the previous deployment as that will be incremental too. The only way to get back to v1.3 is to the clean deploment of v1.0 and then apply the incremental deployments for v1.1, v1.2, and v1.3.

Pros and cons of full redeployments

• Pro: Full redeployments wipe out any manual modifications, leaving only documented configurations in place.
• Pro: Full redeployments are repeatable and predictable. Because the procedure is the same no matter what part of the deployable artifacts has changed and no matter what the current state of the target environment, it will always yield the same results.
• Pro: Full redeployment allow for a very simple undo strategy. When the deployment of a new version of your application fails, or the new version just does not work, just redo the deployment for the previous version.
• Con: Full redeployments do not work for system that have state, such as a database. These systems need to be kept intact.
• Con: Keeping the application available while performing a full redeployment is harder to do than with an incremental deployment. Then again, as mentioned before, to really have a highly available application you will need to set up clusters anyway.
• Con: Full redeployments can execute slowly if the target middleware is not very fast because of the large number of middleware configuration actions to perform.

Conclusion

While full redeployment can be sometimes be hard to get working properly, the fact that they are repeatable and predictable is such a major benefit that I think it far outweighs any disadvantages. Incremental deployments might seem like a leaner and meaner approach to deploying upgrades, but in fact you are sacrificing that thing that is essential to a deployment scenario; that it is a repeatable, predictable and undoable process. Only when state is kept (we are talking about databases here), are incremental deployments really necessary. In all other circumstances I recommend to implement a full redeployment scenario for upgrades.

Filed under Deployment, Xebia Labs | 1 Comment »