my examined life

So far, I’ve done a couple of hard drive images and swaps on devices that run Linux. The latest one was on my laptop. It’s quite easy once you know the steps, and frankly, I’m writing them down here, so I don’t forget.

1. Image the whole hard drive using what ever imaging software you desire
2. Place new drive in system, and boot from linux rescue disk
3. Mount the new hard drive and chroot it, i.e. chroot, mount /proc, mount /sys, etc.
4. Edit grub files with new hard drive. These would be device.map and menu.lst
5. Reinstall grub
6. Edit fstab to boot new hard drive
7. Reinstall mkinitrd with the proper device drivers to boot new hard drive

That’s about it. See, I told you it was pretty simple.

A few weeks ago, at work, we decommissioned a few websites, and the business wanted traffic from the old websites to redirect to a specific page on our main website. That’s a pretty simple task, and there’s a number of ways to complish it: use a CGI script, “Refresh” directive in the webpage, mod_alias, mod_rewrite, or .htaccess.

We decided on using mod_alias since at first glance that was the easiest. It works fine as long as you are trying to go to the root document of the website, i.e. trying to go to www.website.com will get redirected to www.newwebsite.com. But the problem arose that www.website.com/some/path/here.html would end up redirecting to www.newwebsite.com/some/path/here.html and of course that page didn’t exist on the new website. This was a major issue for the business because they were concerned about bookmarked links to sub-pages, or links in old email fliers.

In order to work around this we had to go with mod_rewrite to redirect any and all request to the old website and forward them on to the appropriate page on the new website. We setup a vhost on one of our apache servers for the old website and added the following:

RewriteEngine On
RewriteRule /.* http://new_url [R=301]

Here’s what our vhost entry looked like:

<VirtualHost XXX.XXX.XXX.XXX>
ServerName oldwebsite.com
ServerAlias www.oldwebsite.com
RewriteEngine On
RewriteRule /.* http://new_url [R=301]
</VirtualHost>

It was that simple, too bad we didn’t think of that from the start.

We use Lotus Notes email clients at work, and for those of you who use the Notes client for Linux know what a pain it has been. I recently switched to the the new 8.0.2 client and am impressed with the increased performance and decreased memory footprint. My biggest complaint with the Linux client has always been how much slower it is compared to the Windows client, and so far each release has been getting better and better.

The upgrade process is still a pain, mainly because the java installer doesn’t show what it’s doing the the background. In my case, I had installed 8.0.1 client on a clean install of openSUSE 11, making sure to only install the client. I opted not to install Sametime or any of that other crud since we don’t use it here. The problem arose when doing the upgrade to 8.0.2 because the installer detected that Notes was already installed and showed that nothing was checked on the optional software screen, but decided to install Sametime along with the client. Since I had unchecked Sametime during the installed of 8.0.1, those installation settings were still on my PC somewhere and were being displayed leaving me without the ability to uncheck, and uninstall Sametime.

Frustratingly, I tried doing the install a number of times, checking Sametime and then trying to uncheck it, moving /opt/ibm to /opt/ibm_old, etc. but nothing worked. Finally, I went searching on my hard drive for what exactly was going on during the install and found the following four directories: /tmp, /opt, /etc, and /root. The stuff in /tmp gets deleted immediately after you close the InstallShield window, but the stuff in the other three say for a while. The program binaries are installed in /opt/ibm/lotus/notes, an InstallShield directory gets created in /root containing who knows what, and a bunch of shared files get created in /etc/lotus/notes.

Interestingly enought, the file /etc/lotus/notes/notesrc is a log of Notes installations that InstallShield reads to determine if the client has already been installed, and the stuff in /root/InstallShield are where the settings for the installer are located. Once I removed /root/InstallShield the installer still said that Notes was already installed, but displayed the correct Sametime checkbox and allowed me to uncheck it. Finally, I was able to rid myself of it.

A few weeks back, I was just about to walk out the door for the day when I got an alarm message that our website had gone down. After feverishly try to ssh into the sever only to have it hang after I entered my password, I ran to the server room to get to the console. Now, the person who I replaced setup these SLES servers to boot into runlevel 5, which means that the console has a nifty graphical login prompt. In my opinion, that’s great for a laptop/desktop, but not for a server, especially a production server.

After trying to login only to have it hang, I killed the X session (ctrl-alt-backspace) and noticed a message about /tmp being out of space, and right away I knew that the hard drive had filled up for some reason or another.

But at times like these thank goodness for single user mode. I’ve never actually had a harddrive fill up on my before, so this was a good learning experience. To boot into single user mode in GRUB you type single at the prompt, and for LILO you type linux single. Then the system boots up, mounts the file system as read-only and you enter the root password. What I didn’t know at the time was that the filesystem is mounted read-only, so after a minute of googleing I found this command to remount as read-write: mount -o remount,rw /

Once the file system was remounted read-write, all that was left was the trivial task of freeing up enough space to bring up the server and go home. Luckily, I found an old 22GB tar file to delete.

After analysis the next day, I discovered over 90GB of heapdump/javacore files from WebSphere Commerce that was the cause of the hard drive filling up. Apparently, whenever WebSphere runs out of memory or has problems it dumps the contents of memory into a heapdump file on the hard drive, and there were close to 1000 heapdumps at around 120MB a pop.

At work, I recently took over the linux administration of some 30 servers. I’ve been using linux off and on since 2000, so to be able to do linux admin work in a corporate setting has been quite enjoyable and it’s a career path I’ve been wanting to head down for a long time. There’s been a little bit of a learning curve, since I’m used to Slackware and these servers are running various releases (7,9,10) of SuSE Linux Enterprise Server, but all in all it hasn’t been too bad.

Currently, we have two machines running under VMware, six Lintel boxes, and about 20 or so servers running under z/VM on our big mainframe. The one thing that I still haven’t wrapped my head around is how almost all of the hard disks on the z/VM guests are almost entirely filled up. For example, the other day, the websphere admin came to me and asked why the root partition of the big money making websphere server was 100% full. I wasn’t surprised, since this was probably the seventh or eight time in a month that I’ve run across this.

This prompted me to google a bash script that would search the file system for files bigger than a given value, and since google never does me wrong, I came across this site.

That was a nifty find, but for some reason the web software Jarrod is using to power his website wouldn’t display a much needed \ and the script wouldn’t work. That caused me to dig deep into some of my linux reference manuals to get it working, and I decided to improve it a little. This is what I came up with.

#!/bin/bash
# if nothing is passed to the script, show usage and exit
[[ -n "$1" && -n "$2" ]] || { echo "Usage: findlarge [PATHNAME] [SIZE in KB]"; exit 0 ; }
# simple using find, $1 is the first variable passed to the script
find $1 -type f -size +$2k -exec ls -lh {} \; | awk '{ print $8 ": " $5 }'