Why did you not see that on metrics / dashboards ... Tracing an app should be last resort, a proper setup would have told that you had memory issues right away.
It would be nice to language and os your talking about.
Also, to the writer, I think you want to turn off swap?
It will give you the option of swap when OOM would otherwise kill process, but you can decide exactly how much, to prevent swapping from hurting performance -- say have a 10% extra.
Assuming you have a significant number of identical servers, turning off swap should be reserved for after you have done your tuning of the daemons and their memory use, and you are confident nothing will run out (still, you should monitor dmesg) to maximize bang for bugs.
If however you need swap a good old trick- don't put the swap on LVM or RAID. Just have separate partition on each physical device. The kernel will always know to use them in parallel, and you get to squeeze out a bit of extra performance
Turning off swapping has been a heated discussion, and I am on the negative side of it. Swapping is good; not all memory is important and some of it should not occupy precious physical space. Not all processes are important though. What I would rather have is a per-process swapping flag, allowing me to disable swapping for one or two essential processes, but swap the rest.
Using kernel panics as an alerting mechanism is a brutal solution.
Why not track page faults? Most orgs that I interact with track paging across the whole fleet.