Friday, September 18, 2009

No Space Left on Device!

About two weeks ago, my colleague and I had an issue with the /var partition on a seldom used Sun Blade 2500 workstation. Frankly speaking, I thought a simple fsck would suffice while in single user mode. Not quite. Anyway, we executed fsck on the partition and it "fixed" whatever errors reported. And while in single user mode, we wrote test files to the partition without incident. However, once we booted to init 3 run level, we still could not write to the /var partition, not even as root. That was a bit strange because df -k reported only 20% of the partition was used. hmmm...inode problem?? Then we looked through the logs and noticed a message about not having enough space on the device – in short, we ran out of inodes. Almost immediately, we suspected a problem with mail. We checked the mqueue directory and it contained almost a 1,000,000 small extraneous/useless messages from cron. We deleted most of them (after a few hours) and it resolved the issue. And yes, we got a handle on the cron that was generating those messages.