How to walk with spacewalk

Maarten Kennis

Introduction

One of the most common challenges of managing the configuration of servers in your typical DTAP environment is, in my opinion, keeping all the involved hosts at the same level of configuration in terms of installed operating system packages and their configuration files. It really can be a pain to keep all the systems at the same configuration level. Faillure to do so can lead to interesting situations where software produced by the project team does not run or perform on the acceptance and/or production environment while it was running perfectly on the development and/or test servers.

Ofcourse, there is the possibility of creating one golden virtualized image and pass it around your DTAP environment. However, this can introduce serious issues. For example, when the company hosting your acceptance or production environment does not accept, for obvious reasons, an alien virtualized image to be installed on their precious server farm. By that time, the project has already been running for several months, the engineer who developed the golden virtualized images has left the project and the documentation turned out to be not sufficient to reproduce the golden image.

This is where a Linux systems management solution like Red Hat Satellite can help you out. Since you need a Red Hat subscription for Satellite, this article will discuss the open source alternative called Spacewalk. Spacewalk is an open source Linux systems management solution. It is the upstream community project from which the Red Hat Network Satellite product is derived. Spacewalk manages software content updates for Red Hat derived distributions such as Fedora, CentOS, and Scientific Linux.

With spacewalk you can deploy linux systems, over and over again and always the same way (using kickstart). Centrally manage the packages to be installed on a system and last but not least centrally manage configuration files for each deployed system.

Sounds cool, i want this too!

So, enough about the theory, how does this actually work? To demonstrate this, i have compiled the following cookbook. At the end of this cookbook you will have:

  • A 64 bit CentOS 5.7 server running spacewalk 1.5
  • Deployed a base 64 bit CentOS 6.1 vm using spacewalk
  • Deployed packages on the deployed system using spacewalk
  • Deployed configuration files managed by spacewalk to the deployed server

Prerequisites

Getting the VM up and running

I prefer to keep things lean and mean. For this blogpost a minimal 64 bit CentOS 5.7 will be installed using the net-installer. The following walkthrough provides you with vm ready for spacewalk to be installed.

Start your empty vm booting from the attached CentOS net installer iso. During installation select the defaults or change it to whatever suits your environment for language and keyboard-type. The installation-method is, of course, http. tcp/ip configuration: whatever suits your local network needs for internet access.

Select a mirror service from the CentOS website.

Provide the web site name: my.fast.mirror.com
CentOS directory: path/to/5.7/os/x86_64

Click next on the welcome screen, choose to do a fresh install of CentOS.

Partition your disk to suit your needs. Important note regarding partitioning: This blog article assumes some defaults, based on those defaults you should be aware that there are two locations which need sufficient disk space, you may want to keep this in mind while partitioning:

  • /var/satellite (5GB per distro)
  • /u01/app/oracle/oradata/XE (1GB per distro)

Network setup: configure as needed for your vm to fit in your network and to have internet connection.

Finalize the installation by selecting your timezone, entering your root password and unselecting all installation tasks including the default selected "Desktop – Gnome". Let the installer do it's job, once the system is rebooted you have a fresh base 64 bit centos 5.7 vm available.

Preparing the system for Spacewalk

Spacewalk uses a database for it's back-end administration, this can be eighter a Oracle (XE) or PostgreSQL database. In this article we are going to use the Oracle 11g Express Edition (XE) database together with the Oracle 11g instant client. Transfer the rpm's to your vm and install them (as user root) using the following commands:
yum install --nogpgcheck oracle-xe-11.2.0-1.0.x86_64.rpm
yum install --nogpgcheck oracle-instantclient11.2-basic-11.2.0.2.0.x86_64.rpm
yum install --nogpgcheck oracle-instantclient11.2-sqlplus-11.2.0.2.0.x86_64.rpm

After installation start configuration by:
/etc/init.d/oracle-xe configure
After accepting the defaults (to avoid port conflicts later on in the article, it may be a good idea to specify an other http port then suggested by default. This article assumes you use port 8888), choosing passwords and specifing oracle-xe to start at boot you should have a running oracle XE available. This can be checked by executing the following command:
ps -ef | grep pmon
which should be returning something like this:
[root@spacewalk ~]# ps -ef | grep pmon
oracle    1763     1  0 16:21 ?        00:00:00 xe_pmon_XE
root      3739  1957  0 16:56 pts/0    00:00:00 grep pmon
[root@spacewalk ~]#

Next step is to create a tablespace for spacewalk to store it's data. Start by loading the Oracle XE environment settings (note the space between th '.' and '/'):
. /u01/app/oracle/product/11.2.0/xe/bin/oracle_env.sh
Next, start an sqlplus session.
sqlplus sys as sysdba
Create a tablespace as follows:
create bigfile tablespace spacewalk datafile '/u01/app/oracle/oradata/XE/spacewalk.dbf' size 1G autoextend on;
Create a spacewalk database user and grant it the required privileges:
create user spacewalk identified by spacewalk default tablespace spacewalk;
grant dba to spacewalk;

Oracle XE comes with an apex based management console which can be reached at:
http://hostnameOfYourSpacewalkServer:8888/apex/f?p=4950
Navigate your browser to the url mentioned above and check if management console shows up. For future reference: Oracle XE can be stopped or started using the following command:
service oracle-xe stop
service oracle-xe start

Install Spacewalk

Finally we have arrived at the point where Spacewalk is going to be installed. As user root, perform the following commands to acquire the required repositories:
rpm -Uvh http://spacewalk.redhat.com/yum/1.5/RHEL/5/x86_64/spacewalk-repo-1.5-1.el5.noarch.rpm
rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
rpm -Uvh http://spacewalk.redhat.com/yum/1.5-client/RHEL/5/x86_64/spacewalk-client-repo-1.5-1.el5.noarch.rpm

Next step is to actually install spacewalk (note: due to the speed of the spacewalk repo's this step may take up to 30 minutes to complete).
yum install spacewalk-oracle
Next, configure spacewalk by issuing the following command:
spacewalk-setup -disconnected
After providing the setup program with the Oracle SID (XE), spacewalk db username en password the database is populated. Mostly the defaults can be accepted and/or obvious data can be provided during the rest of the setup program.

For future reference: Spacewalk can be started and stopped using the following commands:
/usr/sbin/spacewalk-service stop
/usr/sbin/spacewalk-service start

Check if the spacewalk server is up and running using the following url (note: you may get a certificate exception upon opening this page):
https://hostnameOfYourSpacewalkServer/
The first time this url is selected the following screen appears allowing you to create an administrative user.

Populate Spacewalk

The goal is to deploy a new machine with an os, this means the obvious next step is to populate spacewalk with a Red Hat derived Linux distribution of your choice. In this article the 64 bit version of CentOS 6.1 is used.

First step is to mount the CentOS 6.1 iso's somewhere on your spacewalk server. Make sure to get a full distro iso, this means that required directories like, for example, images/pxeboot do exist. A minimal or netinst iso of a distribution, in general, does not contain these directories. These directories are used later on in this article, most important at this stage is the content and location of the Packages directory of your distribution's iso.

The packages which belong to a distribution are administered in Spacewalk as software channels, so we first have to create a software channel before we can add/upload packages to it.

Create a new software channel by opening the spacewalk console and navigate to:

"Channels" -> "Manage Software Channels" -> "Create New Channel"

Enter a reasonable channel name (this is for display only, this article uses: "CentOS 6.1 - 64 Bit"), a channel label (remember this name for later use, this article uses: "centos6.1-x86_64") and select the correct architecture (x86_64).

Next step is to populate spacewalk with the CentOS packages, this proces is started by issuing the following command:

rhnpush -v --channel=centos6.1-x86_64 --server=http://localhost –dir=/path/to/Packages

where "/path/to/Packages" is the absolute path of the Packages directory of the mounted iso.

CentOS 6.1 consists of two dvd's, execute above step for both dvd's.

The rhnpush process uploads all packages and registers them in spacewalk. On average, rhnpush processes packages at a rate of around 2000 packages per 30 minutes (ofcourse depending on the configuration of your host and vm). CentOS 6.1 contains almost 6200 packages so, it will take around one and a half hour to upload all packages from dvd1 and dvd2 to spacewalk.

Since we want the deployed linux system to be able to connect to the spacewalk server and use it's package and configuration management facilities it is recommended to include the spacewalk client packages in a spacewalk channel as well. In this article we will upload the packages directly from the online repository into a child channel of the just created CentOS channel.

In the spacewalk console navigate to:

"Channels" -> "Manage Software Channels" -> "Create New Channel"

Enter a reasonable channel name (this is for display only, this article uses: "Spacewalk Client 1.5 - el6 - 64 Bit"), a channel label (remember this name for later use, this article uses: "swclnt1.5-el6-x86_64"), the correct architecture (x86_64) and the correct parent channel (this article uses: "CentOS 6.1 - 64 Bit").

Populating spacewalk with the spacewalk client packages directly from the online repository is started by issuing the following command:

spacewalk-repo-sync -c swclnt1.5-el6-x86_64 --url http://spacewalk.redhat.com/yum/1.5-client/RHEL/6/x86_64

The spacewalk client has a dependency on the python-hwdata-1.2-1.el6.noarch.rpm package from the epel repository. Download the python-hwdata-1.2-1.el6.noarch.rpm package from the epel repository ( http://download.fedora.redhat.com/pub/epel/6/x86_64/ ) and upload it to the spacewalk client child channel using the command (assuming you downloaded the rpm to a folder named epel):

rhnpush -v --channel=swclnt1.5-el6-x86_64 --server=http://localhost -dir=epel

Create a distribution

For automating the installation of a Linux system a method called kickstart can be used. First, we have to setup a directory structure on the spacewalk server based on content of the CentOS dvd1 iso. From your CentOS 6.1 dvd1, copy the following directories:

  • images
  • isolinux
  • repodata

to:
/var/distro-trees/centos6.1-x86_64

Next, open the spacewalk console and navigate to the following location:

systems -> kickstart -> distributions -> new distribution

Enter the following parameters for the new distribution:

  • Distribution label: centos6.1-x86_64
  • tree path: /var/distro-trees/centos6.1-x86_64
  • Base Channel: CentOS 6.1 - 64 Bit
  • Installer Generation: Red Hat Enterprise Linux 6

Next step is to create a kickstart profile for the channel and distribution. Open the spacewalk console and navigate to the following location:

systems -> kickstart -> create new kickstart profile

Enter the following parameters for the new kickstart profile:

  • Label: centos61-minimal
  • Base channel: CentOS 6.1 - 64 Bit
  • Kickstartable tree: centos6.1-x86_64
  • Virtualization type: none

To make sure the spacewalk client repository is used during kickstart, navigate to the following location:

systems -> kickstart -> profiles -> centos61-minimal -> operating system

Make sure the child channel swclnt1.5-el6-x86_64 is checked.

Also, have a look at the other tabs to have an idea of the configuration options which are available. Possible interesting area's are:

  • Software: Adding extra packages or package groups in addition to the base installation. Add the package just by adding it on a new line, package groups can be added by an @-sign followed by the group name. A package can be excluded by an hyphen (-) followed by the package name
  • Kickstart details -> Details -> Kernel options: Adding and removing kernel options. You can add a kernel option, just by adding it's key/value pair to the input field. Removal is done by just mentioning the kernel option preceded by an ! and giving it ~ as a value. For example, the value "!text=~ resolution=800x600" in the kernel option box forces the use of the graphical installer (remove the text kernel option) and sets screen resolution to 800x600.
  • Kickstart details -> Advanced options: Allows detailed configuration of the kickstarted system. For example, to add an user,during installation, named weblogic with password weblogic01, tick the "user" checkbox and add the value "--name=weblogic --password=weblogic01 --plaintext" to the input field.
  • Kickstart details -> Variables: the usage of variables can be done by adding a key/value pair and refer to it in another tab. For example (might be a bad example but it is just to demonstrate the usage), to define the hostname during kickstart, add a key/value pair (hostname=appsrvr1) in the variables tab and refer to it in the Advanced options by adding "--hostname $hostname" to the network text box.

Let's cobbler

Next step is to create an iso image to boot a new vm from. Important note: In the next couple of steps we are going to deploy a new linux virtual machine. If your virtualization network setup supports a dns where the spacewalk server can be found by it's hostname you can skip the next step. In other words, your newly created vm must be able to find the spacewalk server using it's hostname during boot/initial setup. If this is not the case or if you are unsure, please perform the following step to change the spacewalk hostname to it's ip-address, if you are sure dns is in place you can skip this step:

In /etc/rhn/rhn.conf change the value of the parameter cobbler.host to the ip address of the spacewalk server.
In /etc/cobbler/settings change the value of the parameters server and redhat_management_server to the ip-address of the spacewalk server.

On the spacewalk server, run the command (this only needs to be done once):

cobbler get-loaders

Next, start building the iso using the command:

cobbler buildiso

The result of the buildiso command is a file named generated.iso in the directory from where you issued the command.

Let's kickstart

On your host, create a new virtual machine and provide it with the generated.iso file to boot from. Upon boot you will see a menu allowing you to specify the centos61-minimal setup to be installed.

Select this entry and the setup will install a base 64 bit CentOS 6.1 Linux system. If all goes well, this will happen completely automated, without any user intervention whatsoever. If, during install, you receive messages like "Error downloading kickstart file", this probably means you have to look into dns issues as described earlier in the article.

Verify that the system registered itself in spacewalk, it should appear in the system tab on the main screen of the spacewalk web console.

Configuring the client

Now that we have installed a fresh 64 bit CentOS 6.1 Linux vm we have to configure it as a client for spacewalk. Open an ssh session to the newly deployed CentOS 6.1 vm and install the packages rhncfg-client and rhn-check using yum.
yum install -y --nogpgcheck rhncfg-client rhn-check

Managing the configuration of this newly created vm can be done in the following two ways:

  • Deploy new packages to the client
  • Deploy (configuration) files to the client

Deploy new packages to the client

To install a new package from the repository to the new server, go to the spacewalk web console and navigate to the following location:

system -> "your system" -> Software -> Packages -> Install

Select the required package from the repository (for example xauth) and click on "Install Selected Packages"

Next, select "Schedule action as soon as possible" at the confirmation screen and click on "Confirm"

Now, log on to the client and verify the software channels it is subscribed to by executing:

rhn-channel --list

Check if the channel where you made the pending change is in the list. Next, verify if the selected package is not installed yet by executing, on the client:

[root@appsrvr1 ~]# rpm -qa | grep -i xauth
[root@appsrvr1 ~]#

If the package is not installed yet, apply the pending change (installation of the package) by executing:

rhn_check

The server will check for any pending actions (in this case installing the selected package) and execute (install the package) them. Now, check again to verify that the (xauth) package was installed by executing:

[root@appsrvr1 ~]# rpm -qa | grep -i xauth
xorg-x11-xauth-1.0.2-7.1.el6.x86_64
[root@appsrvr1 ~]#

Deploy (configuration) files to the client

In case of managing the configuration files of a linux system through spacewalk, this can be done through configuration channels.

First of all, create a new configuration channel. Open the spacewalk web console and navigate to the following location:

Configuration -> Configuration channels -> create new config channel

Enter information to identify the config channel:

Name: My Config Channel
Label: myConfigChannel
Description: My Config Channel

Next step is to populate this channel with files and directories by navigating to the following location:

Select the configuration channel -> add files -> create file

Now you can create files, directories and symlinks, set ownerships and file permissions. In case of creating a file it is possible to add the actual content of the file in the inline editor. Click on "Create Configuration File" to finalize this action.

To deploy this file to the managed linux system, this system must first be subscribed to the config channel. In the spacewalk web console, navigate to the following location:

systems -> "your system" -> configuration -> manage configuration channels -> subscribe to channels

Next, verify if the client is successfully subscribed to the newly created config channel by executing the following command on the client:
[root@appsrvr1 ~]# rhncfg-client channels
Using server name spacewalk
Config channels:
Label Name
----- ----
myConfigChannel My Config Channel
[root@appsrvr1 ~]#

If the channel appears in the output of the previous command you can get those files (or directories) by issuing:

[root@appsrvr1 ~]# rhncfg-client get
Using server name spacewalk
Deploying /opt/oracle
Deploying /opt/oracle/middleware
Deploying /opt/oracle/middleware/jrockit
[root@appsrvr1 ~]#

If you want to verify if there is a delta between your system and the config channel you can do so by executing
rhncfg-client diff

Conclusion

As usual with this kind of systems, it takes a lot of effort upfront to set it all up. With this article i hope i will help the reader by setting up a spacewalk system relatively easy and fast. Hopefully, the reader will realize pretty soon that managing Linux systems now really is a breeze and all effort for setting it up was worth it. In my opinion, as of version 1.5, which is current at the time of writing this article, stability and functionality has increased a lot since I started working with Spacewalk. If you’re looking for a way to manage your Red Hat derived Linux systems, i highly recommend taking a look at Spacewalk.

Comments (16)

  1. habib - Reply

    December 19, 2011 at 1:31 pm

    Hi,
    Thanks for your nice information. I like it.

    Thanks.

  2. akashalo - Reply

    December 20, 2011 at 8:04 am

    Hi,
    Thank you for your nice article . It will help me.
    Thanks

  3. Francois - Reply

    December 22, 2011 at 12:26 am

    Hi,

    What are the main advantage of something like spacewalk compared to traditionnal configuration or drift assessment tools like Chef (http://www.opscode.com/chef/), CFEngine 3 (http://cfengine.com/), Puppet (http://puppetlabs.com/) or Rudder (www.rudder-project.org/) ?

    And thanks for the nice article !

  4. salim - Reply

    February 9, 2012 at 2:40 pm

    dosnt work .. :(

    after successive installation not registering with spacewalk server ..

    am using centos 5

  5. salim - Reply

    February 14, 2012 at 11:37 am

    Haii

    could you get success result after kicktart usiing this same step .. i can kickstart with out client channell and i need to mannuly register with Space walk ..

  6. salim - Reply

    February 21, 2012 at 4:14 pm

    Thanks Brother :)

  7. Raj - Reply

    March 18, 2012 at 9:41 pm

    Hi,

    Thanks for the excellent article. I have followed your instructions, but am getting an error with osa-dispatcher (when starting the spacewalk service), and then te spacewalk page is blank. Would appreciate any assistance you could provide. Any chance I can send you the exact error via email.

    Thanks,
    Raj

  8. Auro - Reply

    September 6, 2012 at 6:53 am

    hi.

    to use the rhncfg-client you should install SSL Cert to client

    this quote from official doc RH (https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Network_Satellite/5.4/pdf/Reference_Guide/Red_Hat_Network_Satellite-5.4-Reference_Guide-en-US.pdf):

    12.3.2. Deploying Client SSL Certificates
    To ensure secure data transfer, Red Hat strongly recommends the use of SSL. The RHN Satellite eases
    implementation of SSL by generating the necessary certificates during its installation. The server-side
    certificate is automatically installed on the Satellite itself, while the client certificate is placed in the /pub/
    directory of the Satellite's Web server.
    To install the certificate, follow these steps for each client:
    1. Download the SSL certificate from the /var/www/html/pub/ directory of the RHN Satellite onto
    the client system. The certificate will be named something similar to RHN-ORG-TRUSTED-SSLCERT. It is accessible via the web at the following URL: https://yoursatellite.example.com/pub/RHN-ORG-TRUSTED-SSL-CERT.
    2. Move the client SSL certificate to the RHN-specific directory for your UNIX variant. For Solaris, this
    can be accomplished with a command similar to:
    mv /path/to/RHN-ORG-TRUSTED-SSL-CERT
    /opt/redhat/rhn/solaris/usr/share/rhn/
    When finished, the new client certificate will be installed in the appropriate directory for your UNIX
    system. If you have a large number of systems to prepare for RHN management, you may script this
    entire process.

  9. Auro - Reply

    September 6, 2012 at 7:22 am

    And, if you have a problem with Cobbler, read this:
    https://github.com/cobbler/cobbler/wiki/Selinux

    Quote:
    Fedora 16 / RHEL6 / CentOS6 - Python MemoryError
    Obscure error message for which a solution is unknown. The workaround is to disable SELinux.

    s**t...

  10. Derek - Reply

    February 19, 2013 at 12:27 am

    Thanks for the great walk-through. I had had a horrible time getting this thing installed and working, and it's still not 100%. I just wanted to help anyone else who had the same problems I had so that you don't give up too soon. I almost did...

    First, there is a point during the spacewalk-setup operation where it needs to be able to resolve the local hostname to its IP. If you don't have a DNS server running (which I didn't), it's too stupid to look at localhost. You need to add a line to the hosts file to keep it from failing toward the end of the configuration process. Here's what my hosts file looks like:
    192.168.88.6 localhost spacewalk.localdomain
    127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

    The second problem was the missing site certificate, which won't allow rhnpush to work. It can be easily installed with this:
    rpm -i /var/www/html/pub/rhn-org-trusted-ssl-cert-1.0-1.noarch.rpm

    Third, when experimenting on a VM with non-RAID DAS, the drive churn introduced enough lag to cause the channel population operation to repeatedly timeout and fail. Simply changing the DEFAULT_TIMEOUT value in /usr/lib/python2.6/site-packages/rhn/SSL.py from 120 to 240 fixed that problem for me.

    The last problem I haven't figured out... The kickstart ISO built perfectly with only the base channel selected. If I enable the child channel for the client, in the kickstart profile (and rebuild the ISO), it fails saying that it can't read the repository metadata. I must be missing something. Was I supposed to create a repodata directory for the client packages?

  11. mahesh - Reply

    April 1, 2013 at 10:31 pm

    While creating, getting the following message. Need help..

    ==================================
    The server experienced a problem which prevented your request from being filled out. It may not be possible to execute this action at this time.

    Please help us correct this problem by contacting us with details of how you received this message.

  12. dIPESH - Reply

    May 4, 2013 at 7:36 am

    hey,

    when i run "rhnpush -v --channel=centos6.1-x86_64 --server=http://localhost –dir=/media/CentOS_6.4_Final" i got the following error message -

    ERROR: unhandled exception occurred: (Cannot read from directory /root/ir=/media/CentOS_6.4_Final).

    please help me to fix this.
    by the way thanks for nice explanation.

    • Bas - Reply

      February 16, 2014 at 3:14 pm

      Copy and paste

      "rhnpush -v --channel=centos6.1-x86_64 --server=http://localhost -–dir=/media/CentOS_6.4_Final"

  13. Jonson Johnson (NSA agent) - Reply

    June 30, 2013 at 11:38 pm

    Hi All,

    Short addition. For Fedora 18 (and probably others) you also need ot copy the LiveOS in to your chosen location (/var/distro-trees/ above)

    Also with Fed 18, some muppet has put the cobbler 2.4.0 beta3 in to the approved updates channel (as opposed to updates-testing), so I lost hours working out why my "stable configuration" was not generating KS files. Simple yum downgrade resolved that.

    Other than those "update" issues, a big thanks to Maarten for this.

    PS. I am not an NSA agent.

  14. Bas - Reply

    February 16, 2014 at 8:49 pm

    Hi All,

    The article above says:

    ¨Select this entry and the setup will install a base 64 bit CentOS 6.1 Linux system. If all goes well, this will happen completely automated, without any user intervention whatsoever. If, during install, you receive messages like "Error downloading kickstart file", this probably means you have to look into dns issues as described earlier in the article.¨

    I´am using the manual for a CentOS-6.5 KVM on the same os. The KVM boots and tells me ¨Cannot retieve repository metadata (repomd.xml) for repository: --and then the name I gave to the repo ¨swclnt2.0-el6-x86_64¨ Please verify its path and try again.

    My question is : How can I see which code is running and of course how it is configured ?

    Thnx in advance,
    Bas

  15. Bas - Reply

    February 17, 2014 at 12:49 pm

    And when clicking Next the system respond with

    Unable to read group information from repositories. This is a problem with the generation of your installer tree.

    Anyone ?

Add a Comment