The Deployit plugin for the new 3.3 release of Atlassian Bamboo adds the enterprise-scale deployment capabilities of XebiaLabs Deployit to Bamboo. This allows you to speed up your development process by adding automated deployment to your continuous integration setup and make the the first step towards continuous deployment and continuous delivery. Instead of deployment being a bottleneck to your development process, it will be be an integrated part of it. You can test your application on the target platform as soon as possible, find any platform incompatibility and deployment issues early on, and, when it’s time to deploy to the production environment, your deployment will be quick and reliable.

read more

Workload Deployer V3 and Deployit from XebiaLabs both “deploy” things, but they deploy different things. Deployit deploys application artifacts and resources, such as EAR files and data sources, to middleware systems like IBM WebSphere Application Server (but also HTML to web servers, IBM WebSphere MQ configurations to queue managers, and so on). IBM Workload Deployer, on the other hand, deploys patterns (or topologies) of virtual images to hypervisors — but not just any kind of virtual images. IBM Workload Deployer is especially geared toward deploying middleware topologies.

IBM Workload Deployer V3 is an updated and enhanced version of the WebSphere CloudBurst Appliance, renamed to reflect the expanded scope of workloads it can deploy, which are no longer limited to only WebSphere workloads. The content for this article (including screen captures) was created using a WebSphere CloudBurst Appliance, but everything noted here is equally applicable to IBM Workload Deployer V3. However, for the sake of consistency with the images presented, “WebSphere CloudBurst” is used throughout this article to refer to both products.

In other words, IBM Workload Deployer deploys the middleware systems and Deployit deploys applications to those middleware systems — complementary functionalities that form a perfect fit.

At XebiaLabs, we have been working on two exciting new integrations for Deployit. We created a Deployit plugin that enables you to deploy EAR files directly to virtual systems created by IBM Workload Deployer V3 or its predecessor, IBM WebSphere CloudBurst Appliance V2. We also created a WebSphere CloudBurst script package to deploy application artifacts and resources on newly created virtual systems.

This article explores two integrations between WebSphere CloudBurst and Deployit as a way of showing how you can leverage the WebSphere CloudBurst command line interface and script packages to integrate cloud deployment with your application deployment automation solution.

(more…)

IBM’s WebSphere DataPower appliances are a family of appliances that provide valuable services for SOA architectures such as XML acceleration (XA35), XML security (XS40) and data integration/ESB (XI50 ). While the DataPower appliances provide a powerful web-based management GUI, they are not easy to automate. The only command line available is an interactive command line that requires you to telnet into the appliance and the other way to automate the system is a SOAP/XML based API that requires quite a lot of coding.

(more…)

But a small piece of background first. Deployit and WebSphere CloudBurst Appliance both “deploy” things. But they deploy different things. Deployit deploys application artifacts and resources such as EAR files and data sources to middleware systems like WebSphere Application Server (but also HTML to web servers, MQ configuration to queue managers, etc.). The WebSphere CloudBurst Appliance on the other hand, deploys patterns (topologies) of virtual images to hypervisors. But not just any kind of virtual images. It is especially geared towards deploying middleware topologies. In other words, the software Deployit wants to deploy to! This means that the functionalities of WCA and Deployit are a perfect fit; have WCA deploy the middleware systems and have Deployit deploy applications to those middleware systems.
(more…)

Let’s start by defining server provisioning. We can look at the ubiquitous Wikipedia definition or at the one from wordIQ. They tell a similar story; Server provisioning is about making a server ready for service. It usually involves activities such as:
(more…)

In a previous blog, I compared deployment automation to build automation. I wrote about the differences between the build and the deployment process and I explained why different features are required from the respective automation tools. In this blog I will explain the difference between release management and deployment and why release management tools that claim they do deployment automation are actually doing something different. And why that is a good thing.

Let’s start by defining release management. While Wikipedia might define release management as a relatively new discipline in the application lifecycle management space, it has actually been a part of ITIL v2 since its release in 2000. It concerns itself with the management of software releases. Courtesy of the ITIL Open Guide, the key activities of release management are:

(more…)

At XebiaLabs we have developed Deployit and that has given us a lot of insight into the deployment automation domain. This is the first blog in a series that will explore that domain by comparing deployment automation with a number of related topics.

(more…)

There are organizational and technical consequences to introducing a deployment automation product. But let’s focus on the organizational aspects first. These pointers will help you get started with implementing Deployit in your organization in the right way.

Involve the right people

Because application deployment is in essence a cross-departmental issue, you will need to find and involve everybody that it concerns. Depending on the organization of your IT department, that might be developers, application (deployment) administrators, system administrators, database administrators, infrastructure experts, etc. Each of these parties stands to gain from the automation of their part of the deployment process so it is good to get their involvement early in the project.

Standardize your deployment process

A very obvious thing to do before automating your deployment process is to standardize it. If each application uses a separate deployment process, it will be a lot of work to automate all of those. But more importantly, this is a good moment to no longer have a “mixed bag” of processes. With some analysis it should be possible to find one or maybe two different deployment processes that will work for all your Java EE application deployments. When standardizing your deployment process, have a look at the ones that come standard with Deployit. They are based on the recommendations of the middleware vendors (such as IBM’s redbooks) combined with XebiaLabs’ experience setting up deployment automation at a lot of sites.

Rethink ownership of (parts of) the process and the environment

While you are standardizing your deployment process, take a good look at why certain parts of the process and certain environments are the responsible of certain groups and possibly change that. For example, until now, test environments have mostly been owned by the system administration group which made it hard for the development teams to verify their work on this environment. To prevent the development teams from making incompatible changes on the test environment, most sysadmin groups have come up with a pretty tightly controlled process to allow developers access to this environment. This has usually led to a high overhead process with a lot of handover moments and waiting time. Because of Deployit’s security features, the system administrators can give development teams permission to deploy new versions of their application without having the ability to touch other applications or the other parts of the configuration.

Question any derivations from the standard

During the standardization of the deployment process, you might encounter certain areas where some special action is considered absolutely necessary. While that may be valid in some cases, we have seen quite a few cases where that is not the case (anymore). For example, some sites still prefer to keep static content separate from the WAR files and host it on the webserver. The reasoning behind this is that the Java EE application server would not have enough processing power to serve that static content. This has not been true for quite a while thanks to advances such as Java NIO and Edge Side Include caching. Always ask the 5 why’s.

Automate the quick wins first: deployment process steps

Deployit can automate every part of your application deployment process. But when you start out with Deployit it is a good idea to take it step by step. Identify the parts of the deployment process where Deployit will give you the most benefit, the most easily. For example, you might want to incorporate updating SQL schema’s into the deployment process but unless you perform a lot of SQL schema updates there is less benefit to be gained here. So you might leave that part of your deployment process a manual step for now. Then again, if you have a lot of problems with SQL schema updates starting here would be the ideal thing to do. Also, the plugin API of Deployit makes it possible to extend Deployit to support custom middleware systems (there are still a lot of ESB-like systems built on top of standard message queueing software out there!) but you will already gain an advantage when starting with the automation of your standard Java EE application deployments.

Automate the quick wins first: applications

Ultimately you will want to automate the deployment of all your applications, but you’ve gotta start with one of those applications. There are two tactics here: choose a simple application or choose the one that is deployed most often. Usually these are different applications. Our suggestion would be to start with that most simple application and deploy it to a number of different environment so you will be able to find out what configuration changes might be necessary there (SSH access, firewalls, etc.). Changing those things might take a while, giving you time to deploy the more complex applications.

As I had promised, this blog has mainly focused on the organizational aspect of implementing Deployit. Part 2 will focus on the technical side of implementing Deployit. We will cover such subjects as “picking the right unit of deployment”, “choosing a machine to install Deployit on”, and “how to import your current environment into Deployit”. In the meantime, don’t forget to join us for on the episodes of the Deployit webinar series (and don’t worry; you don’t have to be there for all of the webinars ).

If you don’t know what Deployit is yet, have a look at the movie below!

To summarize; Deployit will automate your Java EE application deployments and, because of the overview it offers and the history it keeps, it also allows you to manage and optimize your deployments.

What’s in Deployit v1.2?

We have just released version 1.2 of Deployit. This version has support for the major three application servers; IBM WebSphere AS, Oracle WebLogic Server and JBoss AS. And Deployit now has an open plugin API that allows you to extend the existing plugins or write your own. In this way you can support your custom middleware systems, custom procedures, or even add support for middleware we do not support (yet). The full list of features is as follows:

• Support for IBM WebSphere Application Server 6 and 7;
• Support for Oracle WebLogic Server 10 and 11g;
• Support for JBoss Application Server 4 and 5;
• Support for Apache HTTPD including the management of virtual hosts and generation of WebSphere and WebLogic plugin configurations;
• Support for the creation and configuration of clusters and servers on heterogenous middleware environments;
• Support for a broad range of artifacts, JDBC data sources, JMS resources, shared libraries (JARs), application configurations, firewalls and static content;
• Extensible plugin API framework to include custom middleware stacks and applications;
• Middleware integration test framework and a TCK.

Get started with the Personal Edition!

Together with the release of version 1.2 we have made a Personal Edition available for download. The Personal Edition has the same features as the full Enterprise Edition except for the security features. Also, the license is only valid for a single user. For a full comparison of the editions see the download page.

When you have downloaded the Personal Edition of Deployit, you can see how it works in your own environment! There’s a Getting Started manual to get you up and running and doing deployments within minutes. When you are stuck you can use our email support address or use our telephone support. And you can use the Deployit forums and browse (and add to) the Deployit JIRA. When you’ve done all that, have a look at the plugin API to see how easy it is to customize and extend Deployit. A couple of people from the Deployit community (Hi Benoit and Emmanuel! ) have already written new plugins with the Deployit framework.

We are very happy with the release of Deployit v1.2 and the Personal Edition and we look forward to your questions, new ideas, feature requests and even your bug reports. So, if you haven’t done so already, start downloading now!

Running the middleware integration tests

So let’s start with the Maven configuration. As I mentioned in the first blog of this series, the integration tests are recognizable by the fact that the classnames end in Itest. That means they won’t get picked up by the default configuration of the Maven Surefire plugin. And that is fortunate because we don’t always want to run these tests. Firstly they require a very specific test setup (the application server configurations should be in an expected state, see below) and secondly they can take a long time to complete and that would get in the way of the quick turnaround we want from commit builds in our continuous integration system.

But when we do want to run the middleware integration tests, we can configure the Maven Surefire plugin with a Maven build profile like so:

<profiles>
    <profile>
        <id>itest</id>
        <build>
            <plugins>
                <plugin>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <configuration>
                        <includes>
                            <include>**/Test*.java</include>
                            <include>**/*Test.java</include>
                            <include>**/*TestCase.java</include>
                            <include>**/*Itest.java</include>
                        </includes>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
</profiles>

To run the middleware integration tests in addition to the regular tests, we can tell Maven to use this profile with the -P flag:

$ mvn -Pitest test

If we want this profile to run only the middleware integration tests, we need to leave out the lines that include the standard classnames (Test*.java, *Test.java and *TestCase.java).

Defining the expected state for the target middleware

A big challenge when performing middleware integration tests is to ensure that the middleware we test against is in a correct state. For example, the test might expect an application server cluster to be running but no application to be deployed on it. In a regular test we may set up a test fixture in a method marked with @Before, but in this case that might not be so simple. Confuguring the middleware to be in the expected state would be an integration in itself!

The first approach to address this issue is to write all middleware integration tests in such a way that they revert any changes they make. For example, a test that creates a datasource also removes it. In this way we combine the test for the creation-step with the test for the destruction-step. The disadvantage here is that these two steps can no longer be tested independently, but the advantage is that the combined test runs more quickly. For this approach to be successful, all tests using the same target middleware should agree on a common expected state. We configure the target middleware in that state once and all the tests return the middleware to that state after completion. With a setup like this we can quickly run one or more integration tests, either interactively from an IDE such as Eclipse, or from Maven.

But there is one problem with this. What if the test fails? In that case the middleware might not be left in the correct state. One could argue that the steps needed to revert the middleware should be placed in an @After method but they may still fail. More importantly, those steps are also part of the test so they don’t really belong there!

Reverting the target middleware to the expected state with VMware

This is where VMware comes in. By installing the target middleware in a hypervisor such as VMware, we can make a snapshot when the middleware is in the expected state. Then we can revert to that state before executing a single test or a test suite. The simplest thing is to manually revert from the hypervisor administration console before running the tests. But we can also revert the images automatically, so that we can integrate these tests into our continuous integration builds.

The images we use for our middleware integration tests run on VMware vSphere 4 and we’ve found the vSphere SDK for Perl, and most specifically the provided vmrun.pl sample script, to be the easiest way to interface with VMware vSphere. As an example, this is the script we use to revert our JBoss image:

IMAGE_NAME="[datastore1] jboss-42/CentOS 5.3, JBoss 4.2.3-GA.vmx"
SNAPSHOT_NAME="Ready for itest"
SETTLE_TIME=60
VMRUN=/usr/lib/vmware-vix/lib/api/vix-perl/samples/vmrun.pl

echo "Reverting VMWare image $IMAGE_NAME to snapshot $SNAPSHOT_NAME"
$VMRUN -host $VMWARE_HOST -username $VMWARE_USERNAME -password $VMWARE_PASSWORD \
    revertToSnapshot "$IMAGE_NAME" "$SNAPSHOT_NAME"

echo "Waiting $SETTLE_TIME seconds for things to settle"
sleep $SETTLE_TIME

We do need to provide the proper values for the environment variables VMWARE_HOST, VMWARE_USERNAME and VMWARE_PASSWORD when running this script. After the image has been reverted we wait for 60 seconds to allow the image to settle. The amount of seconds needed here is hard to determine exactly but this value works well for us.

To have Maven run this script before the integration tests are executed, but only when a certain profile is selected, we have added this configuration to the POM:

<profiles>
    <profile>
        <id>revert-vmware-images</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>exec-maven-plugin</artifactId>
                    <version>1.1</version>
                    <executions>
                        <execution>
                            <!-- We really want this to execute right before the test phase. -->
                            <!-- This is the best we can do. -->
                            <phase>process-test-resources</phase>
                            <goals>
                                <goal>exec</goal>
                            </goals>
                        </execution>
                    </executions>
                    <configuration>
                        <executable>/bin/sh</executable>
                        <commandlineArgs>src/test/scripts/revert-vmware-images.sh</commandlineArgs>
                    </configuration>
                </plugin>
            </plugins>
        </build>
    </profile>
</profiles>

Putting it together

With the two profiles introduced in this blog, we tell Maven to revert the VMware images and execute all middleware integration tests by giving the following command:

$ mvn -Prevert-vmware-images -Pitest clean test

We have configured our continuous integration system to run this command once a day, as a secondary build, and configured it with the right values for the VMWARE_HOST, VMWARE_USERNAME and VMWARE_PASSWORD environment variables. In this way the middleware integration tests do not slow down the standard commit build (they take about 90 minutes to complete in our case!). As an added benefit, we know when the VMware images will be used by the tests so we can use them for manual experiments at other times.

Conclusion

In this blog and the first two blogs of this series I have explained how we use JUnit, Maven and VMware to write repeatable, manageable middleware integration tests. These tests have helped us gain a lot of confidence in the stability of Deployit‘s middleware integrations. And because this test-support code is part of Deployit’s plugin API, Deployit plugin developers can use it to get the same confidence in their own code!