Here, we explain how Deployit‘s Infrastructure and new Release Overview features help you achieve this goal.

Continuous Delivery & the Release Perspective

In today’s highly competitive economic environment, the need to bring new features to market quickly, flexibly and reliably is paramount – a goal that is ultimately the aim of the main IT trends Cloud, Agile and DevOps.

Continuous Delivery – extending Continuous Integration to automatically transition applications down the Dev-Test-Acc-Prod delivery pipeline – is a key component of this strategy. In order to be able to effectively implement this, your ARA solution needs to allow your developers – or, in larger organisations, release or DevOps teams, to quickly and efficiently answer questions such as:

• How far is MyApplication down the road to Production?
• When will MyApplication take the next step down the road?
• What do I still need to do before that next step can be taken²?

Ideally, this dashboard would also allow you to plan MyApplication‘s next step and calculate the estimated go-live data, perhaps even based on an analysis of previews versions of MyApplication.

(Virtual) Environment Management & the Operations Perspective

From an Operations point of view, an individual application is only a small part of the picture. Across your Dev-Test-Acc-Prod landscape, you will need to track all applications vying for these environments, to manage potentially conflicting resource requests, plan environment maintenance activities and the like.

Since these environments are often owned and managed by different teams and certainly have varying service levels, you will also want to limit your view to one or a subset of these environments at a time.

Your Operations or DevOps teams need to know:

• Which application versions are currently deployed to my environment(s), or were deployed at a certain point in time?
• Which components do these applications consist of? On which middleware and infrastructure systems are these components deployed?
• What are the current values of any properties or settings for these components? Which environment-specific customizations have been applied?

Cloud and the on-demand environments it enables will eventually replace the rigid Dev-Test-Acc-Prod distinction³. Nevertheless, the ability to present an environment-centric view will still be required, since virtual environments will still be owned by different groups or teams. Indeed, such a perspective will be even more important if you want to effectively combat “virtual sprawl”.

While the coming generations of “true” cloud architectures will hopefully reduce the shared resource conflicts that greatly complicate much of today’s Dev-Test-Acc-Prod management, databases, legacy systems and external payment providers are not likely to disappear anytime soon.

In fact, Facebook, Twitter and other social elements of your future business services may even increase the number of shared resources you need to manage!

Incorporating ARA Data in the Service Delivery Picture

Whilst your ARA solution should be your “go-to” platform for answers about how your applications and environments relate, it is equally important to consider when this data might be more effectively embedded in a broader service delivery picture.

For example, your ARA platform is not a good candidate for providing a release calendar, since it is not aware of much of the information that is relevant in this context, such as CAB⁴ meeting schedules, business sign-off dates or operational maintenance windows.

It is thus important to ensure that your ARA solution can make its data accessible via APIs such as RSS feeds, iCal calendars and other APIs, to enable effective integrations with the rest of your service delivery tooling.

Conclusion

The right Application Release Automation platform gives your Delivery and Operations teams fast, accurate insight into your application environments and delivery pipeline.

Choosing a solution like Deployit with focused Operations and Delivery overviews as well as open APIs for easy integration into your overall Service Delivery dashboards and reports greatly enhances the accessibility and effectiveness of your application release management.

• Software-as-a-Service (SaaS)
• Platform-as-a-Service (PaaS)
• Infrastructure-as-a-Service (IaaS)

While SaaS has been around for some time (Salesforce.com started in 1999!), we are seeing an increase in adoption of IaaS and some heavy development in the PaaS world.

Now that 2011 is coming to an end, this is also the time for lists. So here are my 3 top 3’s of cloud computing.

Software-as-a-Service

3. GitHub

Distributed version control systems are gaining track rapidly, with Git leading the way. GitHub provides Git as a service, and offers unlimited repositories for open source software. It provides an excellent interface that stimulates social coding. This means it is a great incentive for open source development.

2. Dropbox

Well, who doesn’t use Dropbox? Dropbox offers a filesharing service that is easy to use. There are two reasons for the success of Dropbox; it is easy to start with 2GB of free space, and it provides clients for almost all platforms.

1. Google Apps

Google is trying hard to get us all to work in the cloud, and their Google Apps service is their way to do so. They even have a free service for private use. Google Apps provides a full application suite including e-mail, calendar and docs. GMail is massively adopted and has been a game-changer since introduction in 2004.

Infrastructure-as-a-Service

3. OpenStack

When talking about IaaS, people immediately think about the Amazon AWS platform. But what if you don’t like their terms-of-service, or simply want to create something similar in your own data center? Enter OpenStack. If there is one DIY IaaS framework that has momentum, it is OpenStack. It is backed by no less than 144 companies, and best of all, it’s open source.

2. jclouds

If all the IaaS frameworks and providers and all their different APIs are giving you an headache, jclouds is the framework for you. The jclouds API provides an abstraction of the different cloud-specific implementations. Currently over 30 providers are supported, including all the usual suspects (Amazon AWS, OpenStack, Azure, etc).

1. Amazon AWS

The undisputed number 1 of IaaS is of course Amazon AWS. Ever since the introduction in 2002, Amazon AWS is the reference implementation for IaaS. It sets the standard and the rest of the IaaS providers are merely trying to catch up. Amazon also doesn’t sit still, it constantly adds new services to its platform (and is slowly growing into a PaaS). It is available around the globe, with data centers in almost every continent.

Platform-as-a-Service

3. Heroku

Heroku is a fully hosted PaaS platform. It supports lots of languages, and it completely hides the infrastructure (servers, instances, etc) from your applications. It has a partnership with Facebook, creating the Heroku Facebook App Package, which enables quick development of Facebook apps. I think it is one of the best examples of a public PaaS.

2. CloudFoundry

CloudFoundry is being developed by VMware. After acquiring SpringSource back in 2009, this is the next logical step for them. CloudFoundy is positioned as the Open PaaS. While most PaaS solutions limit the choice of frameworks and infrastructure services, CloudFoundry tries to be open and extensible. And best of all, you can use the micro edition for development, the private (open source) edition in your own data center and the hosted edition as a public PaaS (or even a hybrid setup).

1. ???

We are seeing lots of development in this area with all different flavors of PaaS platforms and services. We have even built custom PaaS platforms for our customers based on the traditional application servers (JBoss, WebLogic, etc). But there is still lots of work to be done, before full stack solutions will be readily available. So, I think there is no number 1… yet.

And beyond…

2012 looks like it will be a good year for the cloud. I am very curious to see what the PaaS providers are going to bring to the table. Projects like OpenShift, CloudFoundry and Stratos are looking very promising, and I can’t wait to dive into them.

What are your top cloud services? Or which ones do you think that will become the next best thing? Feel free to add them to the comments below.

Introduction

One of the most common challenges of managing the configuration of servers in your typical DTAP environment is, in my opinion, keeping all the involved hosts at the same level of configuration in terms of installed operating system packages and their configuration files. It really can be a pain to keep all the systems at the same configuration level. Faillure to do so can lead to interesting situations where software produced by the project team does not run or perform on the acceptance and/or production environment while it was running perfectly on the development and/or test servers.

Ofcourse, there is the possibility of creating one golden virtualized image and pass it around your DTAP environment. However, this can introduce serious issues. For example, when the company hosting your acceptance or production environment does not accept, for obvious reasons, an alien virtualized image to be installed on their precious server farm. By that time, the project has already been running for several months, the engineer who developed the golden virtualized images has left the project and the documentation turned out to be not sufficient to reproduce the golden image.

This is where a Linux systems management solution like Red Hat Satellite can help you out. Since you need a Red Hat subscription for Satellite, this article will discuss the open source alternative called Spacewalk. Spacewalk is an open source Linux systems management solution. It is the upstream community project from which the Red Hat Network Satellite product is derived. Spacewalk manages software content updates for Red Hat derived distributions such as Fedora, CentOS, and Scientific Linux.

With spacewalk you can deploy linux systems, over and over again and always the same way (using kickstart). Centrally manage the packages to be installed on a system and last but not least centrally manage configuration files for each deployed system.

Sounds cool, i want this too!

So, enough about the theory, how does this actually work? To demonstrate this, i have compiled the following cookbook. At the end of this cookbook you will have:

• A 64 bit CentOS 5.7 server running spacewalk 1.5
• Deployed a base 64 bit CentOS 6.1 vm using spacewalk
• Deployed packages on the deployed system using spacewalk
• Deployed configuration files managed by spacewalk to the deployed server

Prerequisites

• Create a vm with the following properties:
• 1 cpu
• 1 GB of memory
• Size of harddisk = 6 GB (excluding swap) + (number of distributions x 6 GB)
• vm must have internet access during installation
• Make sure you have the following installation binaries available:
• CentOS-5.7-x86_64-netinstall.iso
• oracle-xe-11.2.0-1.0.x86_64.rpm
• oracle-instantclient11.2-basic-11.2.0.2.0.x86_64.rpm
• oracle-instantclient11.2-sqlplus-11.2.0.2.0.x86_64.rpm

Getting the VM up and running

I prefer to keep things lean and mean. For this blogpost a minimal 64 bit CentOS 5.7 will be installed using the net-installer. The following walkthrough provides you with vm ready for spacewalk to be installed.

Start your empty vm booting from the attached CentOS net installer iso. During installation select the defaults or change it to whatever suits your environment for language and keyboard-type. The installation-method is, of course, http. tcp/ip configuration: whatever suits your local network needs for internet access.

Select a mirror service from the CentOS website.

Provide the web site name: my.fast.mirror.com
CentOS directory: path/to/5.7/os/x86_64

Click next on the welcome screen, choose to do a fresh install of CentOS.

Partition your disk to suit your needs. Important note regarding partitioning: This blog article assumes some defaults, based on those defaults you should be aware that there are two locations which need sufficient disk space, you may want to keep this in mind while partitioning:

• /var/satellite (5GB per distro)
• /u01/app/oracle/oradata/XE (1GB per distro)

Network setup: configure as needed for your vm to fit in your network and to have internet connection.

Finalize the installation by selecting your timezone, entering your root password and unselecting all installation tasks including the default selected “Desktop – Gnome”. Let the installer do it’s job, once the system is rebooted you have a fresh base 64 bit centos 5.7 vm available.

Preparing the system for Spacewalk

Spacewalk uses a database for it’s back-end administration, this can be eighter a Oracle (XE) or PostgreSQL database. In this article we are going to use the Oracle 11g Express Edition (XE) database together with the Oracle 11g instant client. Transfer the rpm’s to your vm and install them (as user root) using the following commands:
yum install --nogpgcheck oracle-xe-11.2.0-1.0.x86_64.rpm
yum install --nogpgcheck oracle-instantclient11.2-basic-11.2.0.2.0.x86_64.rpm
yum install --nogpgcheck oracle-instantclient11.2-sqlplus-11.2.0.2.0.x86_64.rpm
After installation start configuration by:
/etc/init.d/oracle-xe configure
After accepting the defaults (to avoid port conflicts later on in the article, it may be a good idea to specify an other http port then suggested by default. This article assumes you use port 8888), choosing passwords and specifing oracle-xe to start at boot you should have a running oracle XE available. This can be checked by executing the following command:
ps -ef | grep pmon
which should be returning something like this:
[root@spacewalk ~]# ps -ef | grep pmon
oracle    1763     1  0 16:21 ?        00:00:00 xe_pmon_XE
root      3739  1957  0 16:56 pts/0    00:00:00 grep pmon
[root@spacewalk ~]#
Next step is to create a tablespace for spacewalk to store it’s data. Start by loading the Oracle XE environment settings (note the space between th ‘.’ and ‘/’):
. /u01/app/oracle/product/11.2.0/xe/bin/oracle_env.sh
Next, start an sqlplus session.
sqlplus sys as sysdba
Create a tablespace as follows:
create bigfile tablespace spacewalk datafile '/u01/app/oracle/oradata/XE/spacewalk.dbf' size 1G autoextend on;
Create a spacewalk database user and grant it the required privileges:
create user spacewalk identified by spacewalk default tablespace spacewalk;
grant dba to spacewalk;
Oracle XE comes with an apex based management console which can be reached at:
http://hostnameOfYourSpacewalkServer:8888/apex/f?p=4950
Navigate your browser to the url mentioned above and check if management console shows up. For future reference: Oracle XE can be stopped or started using the following command:
service oracle-xe stop
service oracle-xe start

Install Spacewalk

Finally we have arrived at the point where Spacewalk is going to be installed. As user root, perform the following commands to acquire the required repositories:
rpm -Uvh http://spacewalk.redhat.com/yum/1.5/RHEL/5/x86_64/spacewalk-repo-1.5-1.el5.noarch.rpm
rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
rpm -Uvh http://spacewalk.redhat.com/yum/1.5-client/RHEL/5/x86_64/spacewalk-client-repo-1.5-1.el5.noarch.rpm
Next step is to actually install spacewalk (note: due to the speed of the spacewalk repo’s this step may take up to 30 minutes to complete).
yum install spacewalk-oracle
Next, configure spacewalk by issuing the following command:
spacewalk-setup -disconnected
After providing the setup program with the Oracle SID (XE), spacewalk db username en password the database is populated. Mostly the defaults can be accepted and/or obvious data can be provided during the rest of the setup program.

For future reference: Spacewalk can be started and stopped using the following commands:
/usr/sbin/spacewalk-service stop
/usr/sbin/spacewalk-service start
Check if the spacewalk server is up and running using the following url (note: you may get a certificate exception upon opening this page):
https://hostnameOfYourSpacewalkServer/
The first time this url is selected the following screen appears allowing you to create an administrative user.

Populate Spacewalk

The goal is to deploy a new machine with an os, this means the obvious next step is to populate spacewalk with a Red Hat derived Linux distribution of your choice. In this article the 64 bit version of CentOS 6.1 is used.

First step is to mount the CentOS 6.1 iso’s somewhere on your spacewalk server. Make sure to get a full distro iso, this means that required directories like, for example, images/pxeboot do exist. A minimal or netinst iso of a distribution, in general, does not contain these directories. These directories are used later on in this article, most important at this stage is the content and location of the Packages directory of your distribution’s iso.

The packages which belong to a distribution are administered in Spacewalk as software channels, so we first have to create a software channel before we can add/upload packages to it.

Create a new software channel by opening the spacewalk console and navigate to:

“Channels” -> “Manage Software Channels” -> “Create New Channel”

Enter a reasonable channel name (this is for display only, this article uses: “CentOS 6.1 – 64 Bit”), a channel label (remember this name for later use, this article uses: “centos6.1-x86_64″) and select the correct architecture (x86_64).

Next step is to populate spacewalk with the CentOS packages, this proces is started by issuing the following command:

rhnpush -v --channel=centos6.1-x86_64 --server=http://localhost –dir=/path/to/Packages

where “/path/to/Packages” is the absolute path of the Packages directory of the mounted iso.

CentOS 6.1 consists of two dvd’s, execute above step for both dvd’s.

The rhnpush process uploads all packages and registers them in spacewalk. On average, rhnpush processes packages at a rate of around 2000 packages per 30 minutes (ofcourse depending on the configuration of your host and vm). CentOS 6.1 contains almost 6200 packages so, it will take around one and a half hour to upload all packages from dvd1 and dvd2 to spacewalk.

Since we want the deployed linux system to be able to connect to the spacewalk server and use it’s package and configuration management facilities it is recommended to include the spacewalk client packages in a spacewalk channel as well. In this article we will upload the packages directly from the online repository into a child channel of the just created CentOS channel.

In the spacewalk console navigate to:

“Channels” -> “Manage Software Channels” -> “Create New Channel”

Enter a reasonable channel name (this is for display only, this article uses: “Spacewalk Client 1.5 – el6 – 64 Bit”), a channel label (remember this name for later use, this article uses: “swclnt1.5-el6-x86_64″), the correct architecture (x86_64) and the correct parent channel (this article uses: “CentOS 6.1 – 64 Bit”).

Populating spacewalk with the spacewalk client packages directly from the online repository is started by issuing the following command:

spacewalk-repo-sync -c swclnt1.5-el6-x86_64 --url http://spacewalk.redhat.com/yum/1.5-client/RHEL/6/x86_64

The spacewalk client has a dependency on the python-hwdata-1.2-1.el6.noarch.rpm package from the epel repository. Download the python-hwdata-1.2-1.el6.noarch.rpm package from the epel repository ( http://download.fedora.redhat.com/pub/epel/6/x86_64/ ) and upload it to the spacewalk client child channel using the command (assuming you downloaded the rpm to a folder named epel):

rhnpush -v --channel=swclnt1.5-el6-x86_64 --server=http://localhost -dir=epel

Create a distribution

For automating the installation of a Linux system a method called kickstart can be used. First, we have to setup a directory structure on the spacewalk server based on content of the CentOS dvd1 iso. From your CentOS 6.1 dvd1, copy the following directories:

• images
• isolinux
• repodata

to:
/var/distro-trees/centos6.1-x86_64

Next, open the spacewalk console and navigate to the following location:

systems -> kickstart -> distributions -> new distribution

Enter the following parameters for the new distribution:

• Distribution label: centos6.1-x86_64
• tree path: /var/distro-trees/centos6.1-x86_64
• Base Channel: CentOS 6.1 – 64 Bit
• Installer Generation: Red Hat Enterprise Linux 6

Next step is to create a kickstart profile for the channel and distribution. Open the spacewalk console and navigate to the following location:

systems -> kickstart -> create new kickstart profile

Enter the following parameters for the new kickstart profile:

• Label: centos61-minimal
• Base channel: CentOS 6.1 – 64 Bit
• Kickstartable tree: centos6.1-x86_64
• Virtualization type: none

To make sure the spacewalk client repository is used during kickstart, navigate to the following location:

systems -> kickstart -> profiles -> centos61-minimal -> operating system

Make sure the child channel swclnt1.5-el6-x86_64 is checked.

Also, have a look at the other tabs to have an idea of the configuration options which are available. Possible interesting area’s are:

• Software: Adding extra packages or package groups in addition to the base installation. Add the package just by adding it on a new line, package groups can be added by an @-sign followed by the group name. A package can be excluded by an hyphen (-) followed by the package name
• Kickstart details -> Details -> Kernel options: Adding and removing kernel options. You can add a kernel option, just by adding it’s key/value pair to the input field. Removal is done by just mentioning the kernel option preceded by an ! and giving it ~ as a value. For example, the value “!text=~ resolution=800×600″ in the kernel option box forces the use of the graphical installer (remove the text kernel option) and sets screen resolution to 800×600.
• Kickstart details -> Advanced options: Allows detailed configuration of the kickstarted system. For example, to add an user,during installation, named weblogic with password weblogic01, tick the “user” checkbox and add the value “–name=weblogic –password=weblogic01 –plaintext” to the input field.
• Kickstart details -> Variables: the usage of variables can be done by adding a key/value pair and refer to it in another tab. For example (might be a bad example but it is just to demonstrate the usage), to define the hostname during kickstart, add a key/value pair (hostname=appsrvr1) in the variables tab and refer to it in the Advanced options by adding “–hostname $hostname” to the network text box.

Let’s cobbler

Next step is to create an iso image to boot a new vm from. Important note: In the next couple of steps we are going to deploy a new linux virtual machine. If your virtualization network setup supports a dns where the spacewalk server can be found by it’s hostname you can skip the next step. In other words, your newly created vm must be able to find the spacewalk server using it’s hostname during boot/initial setup. If this is not the case or if you are unsure, please perform the following step to change the spacewalk hostname to it’s ip-address, if you are sure dns is in place you can skip this step:

In /etc/rhn/rhn.conf change the value of the parameter cobbler.host to the ip address of the spacewalk server.
In /etc/cobbler/settings change the value of the parameters server and redhat_management_server to the ip-address of the spacewalk server.

On the spacewalk server, run the command (this only needs to be done once):

cobbler get-loaders

Next, start building the iso using the command:

cobbler buildiso

The result of the buildiso command is a file named generated.iso in the directory from where you issued the command.

Let’s kickstart

On your host, create a new virtual machine and provide it with the generated.iso file to boot from. Upon boot you will see a menu allowing you to specify the centos61-minimal setup to be installed.

Select this entry and the setup will install a base 64 bit CentOS 6.1 Linux system. If all goes well, this will happen completely automated, without any user intervention whatsoever. If, during install, you receive messages like “Error downloading kickstart file”, this probably means you have to look into dns issues as described earlier in the article.

Verify that the system registered itself in spacewalk, it should appear in the system tab on the main screen of the spacewalk web console.

Configuring the client

Now that we have installed a fresh 64 bit CentOS 6.1 Linux vm we have to configure it as a client for spacewalk. Open an ssh session to the newly deployed CentOS 6.1 vm and install the packages rhncfg-client and rhn-check using yum.
yum install -y --nogpgcheck rhncfg-client rhn-check

Managing the configuration of this newly created vm can be done in the following two ways:

• Deploy new packages to the client
• Deploy (configuration) files to the client

Deploy new packages to the client

To install a new package from the repository to the new server, go to the spacewalk web console and navigate to the following location:

system -> “your system” -> Software -> Packages -> Install

Select the required package from the repository (for example xauth) and click on “Install Selected Packages”

Next, select “Schedule action as soon as possible” at the confirmation screen and click on “Confirm”

Now, log on to the client and verify the software channels it is subscribed to by executing:

rhn-channel --list

Check if the channel where you made the pending change is in the list. Next, verify if the selected package is not installed yet by executing, on the client:

[root@appsrvr1 ~]# rpm -qa | grep -i xauth
[root@appsrvr1 ~]#

If the package is not installed yet, apply the pending change (installation of the package) by executing:

rhn_check

The server will check for any pending actions (in this case installing the selected package) and execute (install the package) them. Now, check again to verify that the (xauth) package was installed by executing:

[root@appsrvr1 ~]# rpm -qa | grep -i xauth
xorg-x11-xauth-1.0.2-7.1.el6.x86_64
[root@appsrvr1 ~]#

Deploy (configuration) files to the client

In case of managing the configuration files of a linux system through spacewalk, this can be done through configuration channels.

First of all, create a new configuration channel. Open the spacewalk web console and navigate to the following location:

Configuration -> Configuration channels -> create new config channel

Enter information to identify the config channel:

Name: My Config Channel
Label: myConfigChannel
Description: My Config Channel

Next step is to populate this channel with files and directories by navigating to the following location:

Select the configuration channel -> add files -> create file

Now you can create files, directories and symlinks, set ownerships and file permissions. In case of creating a file it is possible to add the actual content of the file in the inline editor. Click on “Create Configuration File” to finalize this action.

To deploy this file to the managed linux system, this system must first be subscribed to the config channel. In the spacewalk web console, navigate to the following location:

systems -> “your system” -> configuration -> manage configuration channels -> subscribe to channels

Next, verify if the client is successfully subscribed to the newly created config channel by executing the following command on the client:
[root@appsrvr1 ~]# rhncfg-client channels
Using server name spacewalk
Config channels:
Label Name
----- ----
myConfigChannel My Config Channel
[root@appsrvr1 ~]#

If the channel appears in the output of the previous command you can get those files (or directories) by issuing:

[root@appsrvr1 ~]# rhncfg-client get
Using server name spacewalk
Deploying /opt/oracle
Deploying /opt/oracle/middleware
Deploying /opt/oracle/middleware/jrockit
[root@appsrvr1 ~]#


If you want to verify if there is a delta between your system and the config channel you can do so by executing
rhncfg-client diff

Conclusion

As usual with this kind of systems, it takes a lot of effort upfront to set it all up. With this article i hope i will help the reader by setting up a spacewalk system relatively easy and fast. Hopefully, the reader will realize pretty soon that managing Linux systems now really is a breeze and all effort for setting it up was worth it. In my opinion, as of version 1.5, which is current at the time of writing this article, stability and functionality has increased a lot since I started working with Spacewalk. If you’re looking for a way to manage your Red Hat derived Linux systems, i highly recommend taking a look at Spacewalk.

In this blog post we will give a description of an open source Java Application Platform as a Service that we created for our customer, using VMware, Redhat Enterprise Linux, Apache WebServer, JBoss Enterprise Application Platform, JBoss Operations Network, Puppet, Deployit,  F5 Load Balancer and  a Layer7 SecureSpan gateway.

Data Center Quality Platform

The customer wanted a data center quality Java Application Platform with the following features:

• Standard configuration
• Standardized provisioning
• Standardized deployment
• Centralized monitoring
• Centralized access control
• Virtual environment
• Proving technology

Current situation

As the current Java application platform was based on HP-UX on Itanium, the customer was facing high cost for hardware, software licenses and fading support from software vendors.  As all applications ran on a HP Superdome, it was very difficult to add resources to individual applications. In addition, development teams spend too much time taking their software through the development, test and acceptance environments, resulting in slow delivery of software into production. Finally, it was difficult to provide 24×7 availability because all applications are running on a single machine.

Java Application Platform

The following figure illustrate the solution architecture of the java application platform.

In the following paragraphs we will describe the purpose of the most important components.

Dual Data Center – HP

Not shown in the figure, is the hardware setup of the platform. It consists of HP blades setup in two data centers on two different locations. This provides the basic infrastructure for 24×7 availability and fault tolerance.

VMware ESX

VMware ESX is deployed on top of the hardware in the dual data center. This provides us with the ability to create virtual machines and provide high availability in case of single server of single site failures. It also allows us the quickly scale up virtual machines and increase the resources assigned to individual virtual machines.

For all machines in the platform we use a single VMware template image. This image is installed with RedHat Enterprise Linux and a puppet client.

Puppet

Puppet fully automates system management. It is used for the installation of software packages, conformity tests and day to day system administration tasks.  For every type of node, we have a puppet plan. When the machines boots, the puppet agent provisions the machine with all the necessary software and configuration according to the plan for that machine.

The use of Puppet completely automates and standardizes the configuration, ensures 100% reproducibility of the configuration and is fast. Provisioning of a new machine from the template to full operational mode is done in a matter of minutes.

JBoss EAP

JBoss Enterprise Application Platform is the Enterprise Java applications server for all java applications.  The installation and configuration is done by Puppet and uses the official RedHat RPMs.  Puppet configures JBoss to ensure that :

• JBoss management applications authenticate users against Active Directory, providing a single point of authorization for operations.
• A JBoss Oracle database schema is automatically provisioned for that specific instance of JBoss, providing persistence for the JBoss server system state.
• All Business Applications can authenticate users using SAML against the Layer7 Identity provider, providing a single point of authentication and authorization for their customers.
• The JBoss instance is added to the pool in the F5 Load balancer
• The application server is added to the Deployit infrastructure inventory, providing the tenants of the platform with the ability to deploy applications to the server.

JBoss application servers are always deployed in multiples of two, where each server of a pair is assigned to a physically different data center location by VMware.

The use of puppet provides us with a fast and reproducible way of provisioning JBoss application servers, allowing for a fast and reliable scale out mechanism for the applications.

JBoss Operations Network

JBoss Operations Network (JON) is used for monitoring all the resources in the platform.   By default, Puppet installs a JON agent on every machine. This agent scans the inventory of the machine and reports it to the JON server.

JON has a very good support for high availability and fail over. By simply adding a JON server machine, agents will automatically distribute themselves across the servers and failover if necessary. Each JON server also runs a JON agent, making sure that unavailability of a JON server is also covered.

In JON we created a number of alert templates for different resource types (os, apache, jboss, jon, puppet, etc.)  that will monitor and report critical conditions on the system.  All error messages from the JBoss servers logs are reported as incidents.

All alerts and clearing conditions from JBoss Operations network are reported via SNMP to TNG Unicenter.

Through the  use of JBoss Operations Network all machines, servers and resources in the platform are automatically added to the centralize monitoring system.

Deployit

Deployit is used for the automated deployment of applications onto the platform. It automatically deploys all the application components in a stack to the appropriate containers.  Deployit :

• deploys static content and proxy configuration to the apache webservers,
• deploys enterprise java application components to all individual JBoss servers in the farm,
• executes SQL scripts to the database,
• configures the F5 loadbalancers to add or remove servers or applications to the pool,
• applies environment specific changes to the application configuration.

The deployment plan for a specific application is prepared in close cooperation between the application developer and platform management staff. When the deployment plan is finished, developers can deploy new versions of the application themselves, directly from a build tool or manually.This ensures solving any installation or configuration problem isn’t postponed until the application is installed for production use, but rather is solved at the early stage of any development.

The same deployment plan is used for all environments. Authorization can be configured per enviroment and per application. LDAP is used to authorize software developers to deploy and configure an application for development and testing purposes, while integration specialist can deploy the application in production.

The use of Deployit provides the platform with a fully automated and standardized deployment mechanism, improving the speed of deployment of applications through the development, test and acceptance environments while reducing the number of staff involved and lowering the number of configuration errors.

F5 Load Balancer

The F5 Load balancers is used to support scalability and fail over for the JBoss Application Server farm.

The pools are configured to use a sticky session protocol based upon the JSESSIONID session cookie. If the cookie is not present, round-robin load balancing of the HTTP requests is performed.

Puppet adds the JBoss servers to the  pool in the F5 Load Balancer.

When a server is scheduled for a restart, the server is taken out of the pool. This ensures that this server does not get any new request, but will still be servicing existing sessions. When the session count in JBoss drops to zero, the server is restarted and restored to the pool.

The use of the F5 Load Balancer provides us with the ability to increase and decrease the number of servers in the farm, provide load balancing, fail over and graceful decommissioning of servers in the farm.

Layer7 XML Gateway / Identity Provider

The Layer7 SecureSpan Gateway is used as centralized security policy enforcement point and SAML identity provider.

Layer7 supports multiple authentication methods, Kerberos, digital certificates, username+password and is able to use multiple identity stores.

Puppet configures all JBoss application servers with SAML support and configures Layer7 as the identity provider: JBoss receives authentication (identity) and authorization (roles) information as a SAML-token. The information contained in the token is translated to a standard JEE-principal user (using a tiny layer of custom code), so all JEE applications can access the authentication and authorization information in a standard way.  Whether the JEE application is a web application or provides webservices, from a security there’s no distinction. All application designers have to do is declare the application security roles conform the JEE standard.

The use of Layer7 standardizes the authentication and authorization for all business applications and centralizes access control.

Conclusion

The customer wanted a modern data center quality Java Application Platform to ensure that java applications could be deployed with lower cost and with high availability and easy scalability.

VMware, the dual data center, Layer7, the F5 Load balancer and JBoss provide the infrastructure for a  high availability and scalability for any java application. The combination of VMware, Puppet and Deployit are the fabric to enable continuous delivery of enterprise java applications.

Through virtualization and automated provisioning and deployment it has become possible to add a completely new, correctly configured machine to a cluster in a matter of minutes, completely secure and under full monitoring.

What early adopters have learnt is that, at the enterprise scale, automating release execution quickly introduces a new bottleneck in today’s dynamic IT environments: continuous management of the deployment plan definition. A new generation of application release automation (ARA) tooling avoids this pitfall by leveraging intelligence to automate deployment planning as well as execution.

The State of ARA

Given how strongly our IT industry is dedicated to the automation of processes, it is nothing short of amazing how much of the deployment of our own solutions depends on manual actions coordinated more or less effectively between large groups of people.
Indeed, a recent analyst report noted that the majority of large enterprises were still relying on manual application release processes or on in-house scripting understood by only a small number of specialists, operated as a black box that – hopefully – will do its job and will most likely necessitate a painful troubleshooting session if it doesn’t.

With key IT trends such as Agile, Cloud and DevOps dramatically ramping up the frequency of application releases in order to increase responsiveness to business needs and provide more and better services to customers, it’s clear that this situation cannot continue.

Thinking of today’s common release processes, it is also hardly surprising that the initial drive has been to automate the actual rollout of the application itself: copying the files to the target machines, restarting the servers, running the SQL against the DB etc. Using a defined workflow to organise these activities makes lots of sense: fewer failures, no more missing steps or steps executed in the wrong order, no more typos, better visualization etc.

Lessons from the First Generation

Ironically, using one of these first generation ARA tools at an enterprise scale quickly made it obvious to early adopters how much effort is required to maintain the substantial number of workflow definitions that quickly accumulate to support full deployments, partial upgrades, rollbacks, environment scale-outs etc. across an enterprise application portfolio.

Of course, this is not a new challenge: ask anyone who has had to update 100 build job definitions in a continuous integration tool to change a compilation parameter, or 100 test plans in an automated testing setup to accommodate a different target browser, just how time-consuming and error-prone this type of maintenance is.

It’s not as though these modifications are unique per process. They tend to be systematic changes that reflect changes to the overall deployment strategy and/or context. The issue is that these first-generation tools, where all the deployment intelligence is stored in the power user’s brain, simply do not have enough internal knowledge of the structure of deployment to assist effectively.

The Next Level of Application Release Automation

A new generation of Application Release Automation includes this intelligence. These advanced tools² no longer require hand-holding by your power users every step of the way, and encode knowledge of deployment best practices and strategies to automate the planning and execution of deployments.³

No pre-provided strategy can be a 100% fit in an enterprise environment, so the strategies must be configurable, of course. Once your power users have fine-tuned them, however, all the individual deployment plans – initial installations, full and partial upgrades, downgrades, undeployments, scale-outs etc…easily hundreds across an application portfolio – are automatically tailored to your scenario.

This becomes even more efficient the fewer deployment strategies are in play, so these tools also motivate and reward increased standardisation of deployment procedures, in itself a valuable business goal. In fact, with a suitable interface and integrations into the development pipeline you essentially have an enterprise Platform as a Service, potentially on a private or hybrid cloud.

Conclusions

With adoption of Application Release Automation rapidly on the increase, a new generation of solutions are appearing that automate deployment planning as well as execution.
Based on the challenges experienced in scaling the first generation of ARA tools to enterprise levels, these next generation solutions are designed to eliminate “continuous expert hand-holding”, promote standardisation and allow organisations to create a “software factory” that continuously delivers business value.

Android apps are not monolithic but rather a collection of components of different kinds. I suspect android took inspiration from the concept of midlet suites in j2me and believe it expanded on that quite well.
These components (except for provider) can be exposed through intent filters in the package’s manifest and can be used by other components in different packages. This allows apps to accomplish tasks together which a single app could never do.

The most important type of component is the Activity and it can be used in several ways:

• Leaf: view data specified or referred to in the extras of the calling intent.
• Put: a pre-filled form with data from the extras. performs an action then stops or returns.
• Pick: select data from a source owned by another app and return it (or a reference to it) to the calling activity.

As activities might start other activities in other tasks, care must be taken to maintain a consistent activity back stack.
Image Shortcut and Send Text demonstrate usage of these intents (but an abnormal use of content provider)

The type of component most suitable for increasing synergy is perhaps the Content Provider.
A Content Provider is accessed through a content resolver and can be used to search, read and update data belonging to a different package. A component can also register to the Content Provider to be notified of changes to its data set.

To send a message to a different package, a Broadcast Receiver is used.
The system also sends a number of messages which apps can listen to.
Sending a broadcast intent can be done asynchronously or synchronously so you can get a result for each matching receiver and have them called in order of priority.
In either case a Broadcast Receiver is not the place to do a lot of processing. Usually either a service is started to handle any data or a notification is raised so the user can take action.

The final type of component is a Service. As the name suggests it runs in the background, and comes in the varieties “run once and die” and “create once then run as demanded”. A service can be bound to through a Binder and communicates through aidl.

I think that, apart from selfishness, fear of abuse is a factor holding back synergy.
Although you can create permissions and set those on your components so other apps need to request those permissions before they can use your components, some users might not review the requested permissions when installing a potentially malicious app.

I am currently developing an Android app at a bank which has a component (currently not exposed) that allows a user to transfer money from his account.
If the activity was exposed it could be used as part of the payment process in any shopping app. The transfer activity would receive the amount and the destination account pre-filled through intent extras set by the shopping app. The transfer activity could return a cookie to the shopping app (similar to pick) which it can verify against the bank’s server independently.

Of course the shopping app shouldn’t have to scan the phone for each banking app so there needs to be defined a common profile of supported component interactions for each type of task. This would allow the shopping app to show an iDeal button calling an intent that matches the filter of any installed banking app’s transfer component in drive-through mode.

‘The whole is greater than the sum of its parts’ Aristotle, Metaphysica

The Deployit plugin for the new 3.3 release of Atlassian Bamboo adds the enterprise-scale deployment capabilities of XebiaLabs Deployit to Bamboo. This allows you to speed up your development process by adding automated deployment to your continuous integration setup and make the the first step towards continuous deployment and continuous delivery. Instead of deployment being a bottleneck to your development process, it will be be an integrated part of it. You can test your application on the target platform as soon as possible, find any platform incompatibility and deployment issues early on, and, when it’s time to deploy to the production environment, your deployment will be quick and reliable.

read more

Workload Deployer V3 and Deployit from XebiaLabs both “deploy” things, but they deploy different things. Deployit deploys application artifacts and resources, such as EAR files and data sources, to middleware systems like IBM WebSphere Application Server (but also HTML to web servers, IBM WebSphere MQ configurations to queue managers, and so on). IBM Workload Deployer, on the other hand, deploys patterns (or topologies) of virtual images to hypervisors — but not just any kind of virtual images. IBM Workload Deployer is especially geared toward deploying middleware topologies.

IBM Workload Deployer V3 is an updated and enhanced version of the WebSphere CloudBurst Appliance, renamed to reflect the expanded scope of workloads it can deploy, which are no longer limited to only WebSphere workloads. The content for this article (including screen captures) was created using a WebSphere CloudBurst Appliance, but everything noted here is equally applicable to IBM Workload Deployer V3. However, for the sake of consistency with the images presented, “WebSphere CloudBurst” is used throughout this article to refer to both products.

In other words, IBM Workload Deployer deploys the middleware systems and Deployit deploys applications to those middleware systems — complementary functionalities that form a perfect fit.

At XebiaLabs, we have been working on two exciting new integrations for Deployit. We created a Deployit plugin that enables you to deploy EAR files directly to virtual systems created by IBM Workload Deployer V3 or its predecessor, IBM WebSphere CloudBurst Appliance V2. We also created a WebSphere CloudBurst script package to deploy application artifacts and resources on newly created virtual systems.

This article explores two integrations between WebSphere CloudBurst and Deployit as a way of showing how you can leverage the WebSphere CloudBurst command line interface and script packages to integrate cloud deployment with your application deployment automation solution.

(more…)

In this post (an extended version of which is available as a whitepaper), we will outline today’s Release Management challenges and discuss the need for Release Automation.

We’ll identify key considerations for successful solutions and highlight why “Zero-Maintenance” is a critical requirement for Release Automation that provides the scalability required in an agile landscape and enables the delivery of continuous business value.

(more…)

A key part of the test harness for the original versions of Tweetstore that run on the Google App Engine is the neat GoogleDevServer class. Basically, it’s a clever wrapper around the KickStart class used under the covers by the GAE SDK tools that allows you to specify the SDK location, address, port and WAR file (or expanded WAR directory) to run.
Better still, it can programmatically be shut down cleanly, making it ideal for integration test runs¹.

For CloudBees, I was thus looking to put together a similar RunAtCloudServer. It proved more challenging than expected…²

Just spawn a container already?

Both the maven-gae-plugin and the bees-maven-plugin allow you to spawn a local GAE or RUN@cloud server simulation, of course. These can easily be integrated into your Maven build, for instance as part of your pre-integration-test setup. So why not just stick with that?

Well, for some scenarios that can be perfectly sufficient. For Tweetstore, having an inline server that cleanly shuts down allows us to:

• run integration tests alongside unit tests from an IDE without having to remember to start an external process
• debug the application running in the server on demand, without having to mess with remote debuggers or figure out a way to sometimes get the Maven plugins to start in debug mode
• work some conditional magic with the server arguments

If your project’s requirements are similar, an inline server might be a good option. And luckily, the code’s already been written!

Hangin’ around

The RUN@cloud SDK’s version of KickStart is the StaxSdkAppServer³. Figuring out the correct arguments for the launchServer factory method wasn’t too hard, so I hooked up a StaxSdkAppServer and kicked off the test.

Started like a charm! Then the test finished and…just hung on in there. Over in the trusty debugger⁴, I could see not one, but two zombie threads still running. The design was based on a “shutdown-by-JVM-kill” scenario, I guess.

Assassins and other Ninja Tricks

The first of the two threads is a Timer that something called the WebAppEngine uses to do…well, who knows? In any case, we can get a reference to it with a decent handful of reflection and cancel it. Easy enough.

Our remaining zombie is a bit more elusive, unfortunately. It’s a thread spawned by an instance of the RequestMonitorValve class in a manner analogous to

new Thread(idleTimer, "requestMonitor").start();

So no reference, and no way of interrupting it. However, the idleTimer Runnable is referenced and repeatedly calls a handler as part of its execution. How about injecting a handler that throws an exception to end the thread?

Nice idea, but it turns out the thread really doesn’t want to die that way:

while(true) {
    try {
        // do stuff
        if(callbackClient != null)
            callbackClient.updateStatus(state);
    } catch(Exception e) {
        // increment some error counters
    }

    try {
        Thread.sleep(statusIntervalSecs * 1000);
    } catch(InterruptedException interruptedexception) { }
}

So I’ve had to go one better and throw a ThreadDeath error instead. Urgh.

Now you see it, now you don’t?

The remaining gotcha involves injecting the catchily-named KillerCallback into the idleTimer. Even using reflection, this is a elementary concurrency failure because without a synchronized-with relationship⁵ between injecting the new handler and invocation of the handler by idleTimer, there is no guarantee the new handler will actually be visible to the thread we’re trying to stop.

Such a relationship could be established if the “assassin” was injected before the thread that runs idleTimer starts. Unless we want to override library classes, that would mean setting it after creating but before starting the StaxSdkAppServer⁶…which unfortunately doesn’t work either because the instances referencing idleTimer are only created when the server starts.

So?

So, for now the potential concurrency issue is documented and we hope for the best No problems observered so far…