This got me to thinking that HN probably has a good trove of debugging war stories. What are your favorites? Bonus points if they happened to you.
At this point, your mail spool begins to fill quickly. You can't process mails and get rid of them in the spool and with 80-90k mails a day combined with qmails primitive handling of bodies, your disk is going low quickly.
The key observation was that the qmail disk-log service used plenty of CPU-time. Restarting it didn't help. At this point, the next key observation is that most of the time is spent in the kernel. strace(1)! We are using blocking disk-writes to sync log changes to disk. Hmmm. It turned out that the log file was so big that many indirections in the inodes were needed. Qmail logs an insane number of lines per mail, so this combined with a hefty disk fragmentation killed the system. rotating the log file made the system jump to 20 processes sending again and we cleared the some 400k mails in the queue in a matter of 80 minutes.
http://research.microsoft.com/~mbj/Mars_Pathfinder/Mars_Path...