A) Running the code on another system to double check that the systems memory / OS has not been corrupted. I once wasted ~6 hours on a corrupted production box, so now the new rule is when system call fail double check it's not just the machine.
B) Double check that none of the memory allocation / book keeping memory had been over written, memory had not been freed twice etc.
Honestly, I usually try to do that type of stuff by direct code inspection. If it's failed once I am probably going to need to debug related code in the not to distant future so really understanding what's going on is important, but hack and slash debugging can be fun.
Then you can either litter break points and start stepping or load the core file in gdb, and start looking closer.
Is the not asshole answer to be stopping right before the malloc and taking note of memory consumption and determining how much malloc is trying to allocate?
When you start asking about how much memory malloc is asking for, that's a tip-off in the wrong direction for me too; malloc handles "out of memory" pretty gracefully.