my examined life

A few weeks back, I was just about to walk out the door for the day when I got an alarm message that our website had gone down. After feverishly try to ssh into the sever only to have it hang after I entered my password, I ran to the server room to get to the console. Now, the person who I replaced setup these SLES servers to boot into runlevel 5, which means that the console has a nifty graphical login prompt. In my opinion, that’s great for a laptop/desktop, but not for a server, especially a production server.

After trying to login only to have it hang, I killed the X session (ctrl-alt-backspace) and noticed a message about /tmp being out of space, and right away I knew that the hard drive had filled up for some reason or another.

But at times like these thank goodness for single user mode. I’ve never actually had a harddrive fill up on my before, so this was a good learning experience. To boot into single user mode in GRUB you type single at the prompt, and for LILO you type linux single. Then the system boots up, mounts the file system as read-only and you enter the root password. What I didn’t know at the time was that the filesystem is mounted read-only, so after a minute of googleing I found this command to remount as read-write: mount -o remount,rw /

Once the file system was remounted read-write, all that was left was the trivial task of freeing up enough space to bring up the server and go home. Luckily, I found an old 22GB tar file to delete.

After analysis the next day, I discovered over 90GB of heapdump/javacore files from WebSphere Commerce that was the cause of the hard drive filling up. Apparently, whenever WebSphere runs out of memory or has problems it dumps the contents of memory into a heapdump file on the hard drive, and there were close to 1000 heapdumps at around 120MB a pop.