my examined life

I have a gumstix Connex 200 sitting under my desk. Aparently, it’s been sitting there for a long time.

Uptime:
15:43:34 up 678 days, 21:16, load average: 0.00, 0.00, 0.00

So far, I’ve done a couple of hard drive images and swaps on devices that run Linux. The latest one was on my laptop. It’s quite easy once you know the steps, and frankly, I’m writing them down here, so I don’t forget.

1. Image the whole hard drive using what ever imaging software you desire
2. Place new drive in system, and boot from linux rescue disk
3. Mount the new hard drive and chroot it, i.e. chroot, mount /proc, mount /sys, etc.
4. Edit grub files with new hard drive. These would be device.map and menu.lst
5. Reinstall grub
6. Edit fstab to boot new hard drive
7. Reinstall mkinitrd with the proper device drivers to boot new hard drive

That’s about it. See, I told you it was pretty simple.

I reinstalled Lotus Notes 8.0.2 today to fix some issues with the Preferences pages not displaying and remembered one more caveat about running Lotus Notes under Linux. For the longest time, I was having issues about email attachments not opening, so I would have to save every attachment and open it with its respective application. Finally, out of frustration, I searched IBM’s website and came across an known issue in which the openwith application, found at: /opt/ibm/lotus/notes/openwith, doesn’t correctly detect magic numbers. The work around is to rename the openwith application to something like openwith.original and create a soft link to gnome-open or the similar KDE application.

• mv /opt/ibm/lotus/notes/openwith /opt/ibm/lotus/notes/openwith.origina
• ln -s /usr/bin/gnome-open /opt/ibm/louts/notes/openwith

A few weeks ago, at work, we decommissioned a few websites, and the business wanted traffic from the old websites to redirect to a specific page on our main website. That’s a pretty simple task, and there’s a number of ways to complish it: use a CGI script, “Refresh” directive in the webpage, mod_alias, mod_rewrite, or .htaccess.

We decided on using mod_alias since at first glance that was the easiest. It works fine as long as you are trying to go to the root document of the website, i.e. trying to go to www.website.com will get redirected to www.newwebsite.com. But the problem arose that www.website.com/some/path/here.html would end up redirecting to www.newwebsite.com/some/path/here.html and of course that page didn’t exist on the new website. This was a major issue for the business because they were concerned about bookmarked links to sub-pages, or links in old email fliers.

In order to work around this we had to go with mod_rewrite to redirect any and all request to the old website and forward them on to the appropriate page on the new website. We setup a vhost on one of our apache servers for the old website and added the following:

RewriteEngine On
RewriteRule /.* http://new_url [R=301]

Here’s what our vhost entry looked like:

<VirtualHost XXX.XXX.XXX.XXX>
ServerName oldwebsite.com
ServerAlias www.oldwebsite.com
RewriteEngine On
RewriteRule /.* http://new_url [R=301]
</VirtualHost>

It was that simple, too bad we didn’t think of that from the start.

We use Lotus Notes email clients at work, and for those of you who use the Notes client for Linux know what a pain it has been. I recently switched to the the new 8.0.2 client and am impressed with the increased performance and decreased memory footprint. My biggest complaint with the Linux client has always been how much slower it is compared to the Windows client, and so far each release has been getting better and better.

The upgrade process is still a pain, mainly because the java installer doesn’t show what it’s doing the the background. In my case, I had installed 8.0.1 client on a clean install of openSUSE 11, making sure to only install the client. I opted not to install Sametime or any of that other crud since we don’t use it here. The problem arose when doing the upgrade to 8.0.2 because the installer detected that Notes was already installed and showed that nothing was checked on the optional software screen, but decided to install Sametime along with the client. Since I had unchecked Sametime during the installed of 8.0.1, those installation settings were still on my PC somewhere and were being displayed leaving me without the ability to uncheck, and uninstall Sametime.

Frustratingly, I tried doing the install a number of times, checking Sametime and then trying to uncheck it, moving /opt/ibm to /opt/ibm_old, etc. but nothing worked. Finally, I went searching on my hard drive for what exactly was going on during the install and found the following four directories: /tmp, /opt, /etc, and /root. The stuff in /tmp gets deleted immediately after you close the InstallShield window, but the stuff in the other three say for a while. The program binaries are installed in /opt/ibm/lotus/notes, an InstallShield directory gets created in /root containing who knows what, and a bunch of shared files get created in /etc/lotus/notes.

Interestingly enought, the file /etc/lotus/notes/notesrc is a log of Notes installations that InstallShield reads to determine if the client has already been installed, and the stuff in /root/InstallShield are where the settings for the installer are located. Once I removed /root/InstallShield the installer still said that Notes was already installed, but displayed the correct Sametime checkbox and allowed me to uncheck it. Finally, I was able to rid myself of it.

There are a few Dell servers that I administer at work. these servers are running SLES 9 SP2, but they don’t have the Dell hardware management agents installed, so given that these are production servers we won’t see any hardware issues until it’s too late. But, I couldn’t install the management agents because the running kernel is a security update version pushed out by Novell and I need the kernel source files which, of course weren’t installed along with the kernel.

The kernel source on the SP2 CD is too old, and the one on the SP3 CD is too new, so for a long while the only way to get the agent installed would be for me to upgrade to SP3 since I couldn’t find the source file for the kernel anywhere. This wouldn’t be a problem except that upgrading to SP3 would require a reboot to start running the newer kernel, and a reboot means downtime which means scheduling the reboot for the middle of the night, or a Saturday, which happen to be times that I’d rather not be working.

All that changed today, when I came across the source rpm for the running kernel on Novell’s website. That gave me the ability to rebuild the source rpm into a binary rpm and install the binary rpm on all the servers running said kernel. Now, I can installed the dell management agents without any downtime, or discomfort to my personal life.

Here is the relevant command:

rpmbuild --nodeps --rebuild --clean source.src.rpm

You can then find the binary rpm in /usr/src/packages/RPMS/ on a SuSE machine.

Because I cut my Linux teeth on Slackware I’m not that familiar with rpm. But now for work and pleasure, I am using rpm based distros, SuSE and CentOS, so it’s time to learn a bit more.

Here’s something simple to display installed packaged and their sizes:

rpm -qa --queryformat '%{NAME}-%{version}-%{release} %{SIZE}\n'

A few weeks back, I was just about to walk out the door for the day when I got an alarm message that our website had gone down. After feverishly try to ssh into the sever only to have it hang after I entered my password, I ran to the server room to get to the console. Now, the person who I replaced setup these SLES servers to boot into runlevel 5, which means that the console has a nifty graphical login prompt. In my opinion, that’s great for a laptop/desktop, but not for a server, especially a production server.

After trying to login only to have it hang, I killed the X session (ctrl-alt-backspace) and noticed a message about /tmp being out of space, and right away I knew that the hard drive had filled up for some reason or another.

But at times like these thank goodness for single user mode. I’ve never actually had a harddrive fill up on my before, so this was a good learning experience. To boot into single user mode in GRUB you type single at the prompt, and for LILO you type linux single. Then the system boots up, mounts the file system as read-only and you enter the root password. What I didn’t know at the time was that the filesystem is mounted read-only, so after a minute of googleing I found this command to remount as read-write: mount -o remount,rw /

Once the file system was remounted read-write, all that was left was the trivial task of freeing up enough space to bring up the server and go home. Luckily, I found an old 22GB tar file to delete.

After analysis the next day, I discovered over 90GB of heapdump/javacore files from WebSphere Commerce that was the cause of the hard drive filling up. Apparently, whenever WebSphere runs out of memory or has problems it dumps the contents of memory into a heapdump file on the hard drive, and there were close to 1000 heapdumps at around 120MB a pop.

At work, I recently took over the linux administration of some 30 servers. I’ve been using linux off and on since 2000, so to be able to do linux admin work in a corporate setting has been quite enjoyable and it’s a career path I’ve been wanting to head down for a long time. There’s been a little bit of a learning curve, since I’m used to Slackware and these servers are running various releases (7,9,10) of SuSE Linux Enterprise Server, but all in all it hasn’t been too bad.

Currently, we have two machines running under VMware, six Lintel boxes, and about 20 or so servers running under z/VM on our big mainframe. The one thing that I still haven’t wrapped my head around is how almost all of the hard disks on the z/VM guests are almost entirely filled up. For example, the other day, the websphere admin came to me and asked why the root partition of the big money making websphere server was 100% full. I wasn’t surprised, since this was probably the seventh or eight time in a month that I’ve run across this.

This prompted me to google a bash script that would search the file system for files bigger than a given value, and since google never does me wrong, I came across this site.

That was a nifty find, but for some reason the web software Jarrod is using to power his website wouldn’t display a much needed \ and the script wouldn’t work. That caused me to dig deep into some of my linux reference manuals to get it working, and I decided to improve it a little. This is what I came up with.

#!/bin/bash
# if nothing is passed to the script, show usage and exit
[[ -n "$1" && -n "$2" ]] || { echo "Usage: findlarge [PATHNAME] [SIZE in KB]"; exit 0 ; }
# simple using find, $1 is the first variable passed to the script
find $1 -type f -size +$2k -exec ls -lh {} \; | awk '{ print $8 ": " $5 }'