Introduction

One of the most common challenges of managing the configuration of servers in your typical DTAP environment is, in my opinion, keeping all the involved hosts at the same level of configuration in terms of installed operating system packages and their configuration files. It really can be a pain to keep all the systems at the same configuration level. Faillure to do so can lead to interesting situations where software produced by the project team does not run or perform on the acceptance and/or production environment while it was running perfectly on the development and/or test servers.

Ofcourse, there is the possibility of creating one golden virtualized image and pass it around your DTAP environment. However, this can introduce serious issues. For example, when the company hosting your acceptance or production environment does not accept, for obvious reasons, an alien virtualized image to be installed on their precious server farm. By that time, the project has already been running for several months, the engineer who developed the golden virtualized images has left the project and the documentation turned out to be not sufficient to reproduce the golden image.

This is where a Linux systems management solution like Red Hat Satellite can help you out. Since you need a Red Hat subscription for Satellite, this article will discuss the open source alternative called Spacewalk. Spacewalk is an open source Linux systems management solution. It is the upstream community project from which the Red Hat Network Satellite product is derived. Spacewalk manages software content updates for Red Hat derived distributions such as Fedora, CentOS, and Scientific Linux.

With spacewalk you can deploy linux systems, over and over again and always the same way (using kickstart). Centrally manage the packages to be installed on a system and last but not least centrally manage configuration files for each deployed system.

Sounds cool, i want this too!

So, enough about the theory, how does this actually work? To demonstrate this, i have compiled the following cookbook. At the end of this cookbook you will have:

  • A 64 bit CentOS 5.7 server running spacewalk 1.5
  • Deployed a base 64 bit CentOS 6.1 vm using spacewalk
  • Deployed packages on the deployed system using spacewalk
  • Deployed configuration files managed by spacewalk to the deployed server

Prerequisites

Getting the VM up and running

I prefer to keep things lean and mean. For this blogpost a minimal 64 bit CentOS 5.7 will be installed using the net-installer. The following walkthrough provides you with vm ready for spacewalk to be installed.

Start your empty vm booting from the attached CentOS net installer iso. During installation select the defaults or change it to whatever suits your environment for language and keyboard-type. The installation-method is, of course, http. tcp/ip configuration: whatever suits your local network needs for internet access.

Select a mirror service from the CentOS website.

Provide the web site name: my.fast.mirror.com
CentOS directory: path/to/5.7/os/x86_64

Click next on the welcome screen, choose to do a fresh install of CentOS.

Partition your disk to suit your needs. Important note regarding partitioning: This blog article assumes some defaults, based on those defaults you should be aware that there are two locations which need sufficient disk space, you may want to keep this in mind while partitioning:

  • /var/satellite (5GB per distro)
  • /u01/app/oracle/oradata/XE (1GB per distro)

Network setup: configure as needed for your vm to fit in your network and to have internet connection.

Finalize the installation by selecting your timezone, entering your root password and unselecting all installation tasks including the default selected "Desktop – Gnome". Let the installer do it's job, once the system is rebooted you have a fresh base 64 bit centos 5.7 vm available.

Preparing the system for Spacewalk

Spacewalk uses a database for it's back-end administration, this can be eighter a Oracle (XE) or PostgreSQL database. In this article we are going to use the Oracle 11g Express Edition (XE) database together with the Oracle 11g instant client. Transfer the rpm's to your vm and install them (as user root) using the following commands:
yum install --nogpgcheck oracle-xe-11.2.0-1.0.x86_64.rpm
yum install --nogpgcheck oracle-instantclient11.2-basic-11.2.0.2.0.x86_64.rpm
yum install --nogpgcheck oracle-instantclient11.2-sqlplus-11.2.0.2.0.x86_64.rpm

After installation start configuration by:
/etc/init.d/oracle-xe configure
After accepting the defaults (to avoid port conflicts later on in the article, it may be a good idea to specify an other http port then suggested by default. This article assumes you use port 8888), choosing passwords and specifing oracle-xe to start at boot you should have a running oracle XE available. This can be checked by executing the following command:
ps -ef | grep pmon
which should be returning something like this:
[root@spacewalk ~]# ps -ef | grep pmon
oracle    1763     1  0 16:21 ?        00:00:00 xe_pmon_XE
root      3739  1957  0 16:56 pts/0    00:00:00 grep pmon
[root@spacewalk ~]#

Next step is to create a tablespace for spacewalk to store it's data. Start by loading the Oracle XE environment settings (note the space between th '.' and '/'):
. /u01/app/oracle/product/11.2.0/xe/bin/oracle_env.sh
Next, start an sqlplus session.
sqlplus sys as sysdba
Create a tablespace as follows:
create bigfile tablespace spacewalk datafile '/u01/app/oracle/oradata/XE/spacewalk.dbf' size 1G autoextend on;
Create a spacewalk database user and grant it the required privileges:
create user spacewalk identified by spacewalk default tablespace spacewalk;
grant dba to spacewalk;

Oracle XE comes with an apex based management console which can be reached at:
http://hostnameOfYourSpacewalkServer:8888/apex/f?p=4950
Navigate your browser to the url mentioned above and check if management console shows up. For future reference: Oracle XE can be stopped or started using the following command:
service oracle-xe stop
service oracle-xe start

Install Spacewalk

Finally we have arrived at the point where Spacewalk is going to be installed. As user root, perform the following commands to acquire the required repositories:
rpm -Uvh http://spacewalk.redhat.com/yum/1.5/RHEL/5/x86_64/spacewalk-repo-1.5-1.el5.noarch.rpm
rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
rpm -Uvh http://spacewalk.redhat.com/yum/1.5-client/RHEL/5/x86_64/spacewalk-client-repo-1.5-1.el5.noarch.rpm

Next step is to actually install spacewalk (note: due to the speed of the spacewalk repo's this step may take up to 30 minutes to complete).
yum install spacewalk-oracle
Next, configure spacewalk by issuing the following command:
spacewalk-setup -disconnected
After providing the setup program with the Oracle SID (XE), spacewalk db username en password the database is populated. Mostly the defaults can be accepted and/or obvious data can be provided during the rest of the setup program.

For future reference: Spacewalk can be started and stopped using the following commands:
/usr/sbin/spacewalk-service stop
/usr/sbin/spacewalk-service start

Check if the spacewalk server is up and running using the following url (note: you may get a certificate exception upon opening this page):
https://hostnameOfYourSpacewalkServer/
The first time this url is selected the following screen appears allowing you to create an administrative user.

Populate Spacewalk

The goal is to deploy a new machine with an os, this means the obvious next step is to populate spacewalk with a Red Hat derived Linux distribution of your choice. In this article the 64 bit version of CentOS 6.1 is used.

First step is to mount the CentOS 6.1 iso's somewhere on your spacewalk server. Make sure to get a full distro iso, this means that required directories like, for example, images/pxeboot do exist. A minimal or netinst iso of a distribution, in general, does not contain these directories. These directories are used later on in this article, most important at this stage is the content and location of the Packages directory of your distribution's iso.

The packages which belong to a distribution are administered in Spacewalk as software channels, so we first have to create a software channel before we can add/upload packages to it.

Create a new software channel by opening the spacewalk console and navigate to:

"Channels" -> "Manage Software Channels" -> "Create New Channel"

Enter a reasonable channel name (this is for display only, this article uses: "CentOS 6.1 - 64 Bit"), a channel label (remember this name for later use, this article uses: "centos6.1-x86_64") and select the correct architecture (x86_64).

Next step is to populate spacewalk with the CentOS packages, this proces is started by issuing the following command:

rhnpush -v --channel=centos6.1-x86_64 --server=http://blog.xebia.com –dir=/path/to/Packages

where "/path/to/Packages" is the absolute path of the Packages directory of the mounted iso.

CentOS 6.1 consists of two dvd's, execute above step for both dvd's.

The rhnpush process uploads all packages and registers them in spacewalk. On average, rhnpush processes packages at a rate of around 2000 packages per 30 minutes (ofcourse depending on the configuration of your host and vm). CentOS 6.1 contains almost 6200 packages so, it will take around one and a half hour to upload all packages from dvd1 and dvd2 to spacewalk.

Since we want the deployed linux system to be able to connect to the spacewalk server and use it's package and configuration management facilities it is recommended to include the spacewalk client packages in a spacewalk channel as well. In this article we will upload the packages directly from the online repository into a child channel of the just created CentOS channel.

In the spacewalk console navigate to:

"Channels" -> "Manage Software Channels" -> "Create New Channel"

Enter a reasonable channel name (this is for display only, this article uses: "Spacewalk Client 1.5 - el6 - 64 Bit"), a channel label (remember this name for later use, this article uses: "swclnt1.5-el6-x86_64"), the correct architecture (x86_64) and the correct parent channel (this article uses: "CentOS 6.1 - 64 Bit").

Populating spacewalk with the spacewalk client packages directly from the online repository is started by issuing the following command:

spacewalk-repo-sync -c swclnt1.5-el6-x86_64 --url http://spacewalk.redhat.com/yum/1.5-client/RHEL/6/x86_64

The spacewalk client has a dependency on the python-hwdata-1.2-1.el6.noarch.rpm package from the epel repository. Download the python-hwdata-1.2-1.el6.noarch.rpm package from the epel repository ( http://download.fedora.redhat.com/pub/epel/6/x86_64/ ) and upload it to the spacewalk client child channel using the command (assuming you downloaded the rpm to a folder named epel):

rhnpush -v --channel=swclnt1.5-el6-x86_64 --server=http://blog.xebia.com -dir=epel

Create a distribution

For automating the installation of a Linux system a method called kickstart can be used. First, we have to setup a directory structure on the spacewalk server based on content of the CentOS dvd1 iso. From your CentOS 6.1 dvd1, copy the following directories:

  • images
  • isolinux
  • repodata

to:
/var/distro-trees/centos6.1-x86_64

Next, open the spacewalk console and navigate to the following location:

systems -> kickstart -> distributions -> new distribution

Enter the following parameters for the new distribution:

  • Distribution label: centos6.1-x86_64
  • tree path: /var/distro-trees/centos6.1-x86_64
  • Base Channel: CentOS 6.1 - 64 Bit
  • Installer Generation: Red Hat Enterprise Linux 6

Next step is to create a kickstart profile for the channel and distribution. Open the spacewalk console and navigate to the following location:

systems -> kickstart -> create new kickstart profile

Enter the following parameters for the new kickstart profile:

  • Label: centos61-minimal
  • Base channel: CentOS 6.1 - 64 Bit
  • Kickstartable tree: centos6.1-x86_64
  • Virtualization type: none

To make sure the spacewalk client repository is used during kickstart, navigate to the following location:

systems -> kickstart -> profiles -> centos61-minimal -> operating system

Make sure the child channel swclnt1.5-el6-x86_64 is checked.

Also, have a look at the other tabs to have an idea of the configuration options which are available. Possible interesting area's are:

  • Software: Adding extra packages or package groups in addition to the base installation. Add the package just by adding it on a new line, package groups can be added by an @-sign followed by the group name. A package can be excluded by an hyphen (-) followed by the package name
  • Kickstart details -> Details -> Kernel options: Adding and removing kernel options. You can add a kernel option, just by adding it's key/value pair to the input field. Removal is done by just mentioning the kernel option preceded by an ! and giving it ~ as a value. For example, the value "!text=~ resolution=800x600" in the kernel option box forces the use of the graphical installer (remove the text kernel option) and sets screen resolution to 800x600.
  • Kickstart details -> Advanced options: Allows detailed configuration of the kickstarted system. For example, to add an user,during installation, named weblogic with password weblogic01, tick the "user" checkbox and add the value "--name=weblogic --password=weblogic01 --plaintext" to the input field.
  • Kickstart details -> Variables: the usage of variables can be done by adding a key/value pair and refer to it in another tab. For example (might be a bad example but it is just to demonstrate the usage), to define the hostname during kickstart, add a key/value pair (hostname=appsrvr1) in the variables tab and refer to it in the Advanced options by adding "--hostname $hostname" to the network text box.

Let's cobbler

Next step is to create an iso image to boot a new vm from. Important note: In the next couple of steps we are going to deploy a new linux virtual machine. If your virtualization network setup supports a dns where the spacewalk server can be found by it's hostname you can skip the next step. In other words, your newly created vm must be able to find the spacewalk server using it's hostname during boot/initial setup. If this is not the case or if you are unsure, please perform the following step to change the spacewalk hostname to it's ip-address, if you are sure dns is in place you can skip this step:

In /etc/rhn/rhn.conf change the value of the parameter cobbler.host to the ip address of the spacewalk server.
In /etc/cobbler/settings change the value of the parameters server and redhat_management_server to the ip-address of the spacewalk server.

On the spacewalk server, run the command (this only needs to be done once):

cobbler get-loaders

Next, start building the iso using the command:

cobbler buildiso

The result of the buildiso command is a file named generated.iso in the directory from where you issued the command.

Let's kickstart

On your host, create a new virtual machine and provide it with the generated.iso file to boot from. Upon boot you will see a menu allowing you to specify the centos61-minimal setup to be installed.

Select this entry and the setup will install a base 64 bit CentOS 6.1 Linux system. If all goes well, this will happen completely automated, without any user intervention whatsoever. If, during install, you receive messages like "Error downloading kickstart file", this probably means you have to look into dns issues as described earlier in the article.

Verify that the system registered itself in spacewalk, it should appear in the system tab on the main screen of the spacewalk web console.

Configuring the client

Now that we have installed a fresh 64 bit CentOS 6.1 Linux vm we have to configure it as a client for spacewalk. Open an ssh session to the newly deployed CentOS 6.1 vm and install the packages rhncfg-client and rhn-check using yum.
yum install -y --nogpgcheck rhncfg-client rhn-check

Managing the configuration of this newly created vm can be done in the following two ways:

  • Deploy new packages to the client
  • Deploy (configuration) files to the client

Deploy new packages to the client

To install a new package from the repository to the new server, go to the spacewalk web console and navigate to the following location:

system -> "your system" -> Software -> Packages -> Install

Select the required package from the repository (for example xauth) and click on "Install Selected Packages"

Next, select "Schedule action as soon as possible" at the confirmation screen and click on "Confirm"

Now, log on to the client and verify the software channels it is subscribed to by executing:

rhn-channel --list

Check if the channel where you made the pending change is in the list. Next, verify if the selected package is not installed yet by executing, on the client:

[root@appsrvr1 ~]# rpm -qa | grep -i xauth
[root@appsrvr1 ~]#

If the package is not installed yet, apply the pending change (installation of the package) by executing:

rhn_check

The server will check for any pending actions (in this case installing the selected package) and execute (install the package) them. Now, check again to verify that the (xauth) package was installed by executing:

[root@appsrvr1 ~]# rpm -qa | grep -i xauth
xorg-x11-xauth-1.0.2-7.1.el6.x86_64
[root@appsrvr1 ~]#

Deploy (configuration) files to the client

In case of managing the configuration files of a linux system through spacewalk, this can be done through configuration channels.

First of all, create a new configuration channel. Open the spacewalk web console and navigate to the following location:

Configuration -> Configuration channels -> create new config channel

Enter information to identify the config channel:

Name: My Config Channel
Label: myConfigChannel
Description: My Config Channel

Next step is to populate this channel with files and directories by navigating to the following location:

Select the configuration channel -> add files -> create file

Now you can create files, directories and symlinks, set ownerships and file permissions. In case of creating a file it is possible to add the actual content of the file in the inline editor. Click on "Create Configuration File" to finalize this action.

To deploy this file to the managed linux system, this system must first be subscribed to the config channel. In the spacewalk web console, navigate to the following location:

systems -> "your system" -> configuration -> manage configuration channels -> subscribe to channels

Next, verify if the client is successfully subscribed to the newly created config channel by executing the following command on the client:
[root@appsrvr1 ~]# rhncfg-client channels
Using server name spacewalk
Config channels:
Label Name
----- ----
myConfigChannel My Config Channel
[root@appsrvr1 ~]#

If the channel appears in the output of the previous command you can get those files (or directories) by issuing:

[root@appsrvr1 ~]# rhncfg-client get
Using server name spacewalk
Deploying /opt/oracle
Deploying /opt/oracle/middleware
Deploying /opt/oracle/middleware/jrockit
[root@appsrvr1 ~]#

If you want to verify if there is a delta between your system and the config channel you can do so by executing
rhncfg-client diff

Conclusion

As usual with this kind of systems, it takes a lot of effort upfront to set it all up. With this article i hope i will help the reader by setting up a spacewalk system relatively easy and fast. Hopefully, the reader will realize pretty soon that managing Linux systems now really is a breeze and all effort for setting it up was worth it. In my opinion, as of version 1.5, which is current at the time of writing this article, stability and functionality has increased a lot since I started working with Spacewalk. If you’re looking for a way to manage your Red Hat derived Linux systems, i highly recommend taking a look at Spacewalk.