The kernel source on the SP2 CD is too old, and the one on the SP3 CD is too new, so for a long while the only way to get the agent installed would be for me to upgrade to SP3 since I couldn't find the source file for the kernel anywhere. This wouldn't be a problem except that upgrading to SP3 would require a reboot to start running the newer kernel, and a reboot means downtime which means scheduling the reboot for the middle of the night, or a Saturday, which happen to be times that I'd rather not be working.

All that changed today, when I came across the source rpm for the running kernel on Novell's website. That gave me the ability to rebuild the source rpm into a binary rpm and install the binary rpm on all the servers running said kernel. Now, I can installed the dell management agents without any downtime, or discomfort to my personal life.

Here is the relevant command:

rpmbuild --nodeps --rebuild --clean source.src.rpm

You can then find the binary rpm in /usr/src/packages/RPMS/ on a SuSE machine.

After trying to login only to have it hang, I killed the X session (ctrl-alt-backspace) and noticed a message about /tmp being out of space, and right away I knew that the hard drive had filled up for some reason or another.

But at times like these thank goodness for single user mode. I've never actually had a harddrive fill up on my before, so this was a good learning experience. To boot into single user mode in GRUB you type single at the prompt, and for LILO you type linux single. Then the system boots up, mounts the file system as read-only and you enter the root password. What I didn't know at the time was that the filesystem is mounted read-only, so after a minute of googleing I found this command to remount as read-write: mount -o remount,rw /

Once the file system was remounted read-write, all that was left was the trivial task of freeing up enough space to bring up the server and go home. Luckily, I found an old 22GB tar file to delete.

After analysis the next day, I discovered over 90GB of heapdump/javacore files from WebSphere Commerce that was the cause of the hard drive filling up. Apparently, whenever WebSphere runs out of memory or has problems it dumps the contents of memory into a heapdump file on the hard drive, and there were close to 1000 heapdumps at around 120MB a pop.

Currently, we have two machines running under VMware, six Lintel boxes, and about 20 or so servers running under z/VM on our big mainframe. The one thing that I still haven't wrapped my head around is how almost all of the hard disks on the z/VM guests are almost entirely filled up. For example, the other day, the websphere admin came to me and asked why the root partition of the big money making websphere server was 100% full. I wasn't surprised, since this was probably the seventh or eight time in a month that I've run across this.

This prompted me to google a bash script that would search the file system for files bigger than a given value, and since google never does me wrong, I came across this site.

That was a nifty find, but for some reason the web software Jarrod is using to power his website wouldn't display a much needed \ and the script wouldn't work. That caused me to dig deep into some of my linux reference manuals to get it working, and I decided to improve it a little. This is what I came up with.

#!/bin/bash
# if nothing is passed to the script, show usage and exit
[[ -n "$1" && -n "$2" ]] || { echo "Usage: findlarge [PATHNAME] [SIZE in KB]"; exit 0 ; }
# simple using find, $1 is the first variable passed to the script
find $1 -type f -size +$2k -exec ls -lh {} \; | awk '{ print $8 ": " $5 }'