There's a userspace daemon that does that: https://github.com/rfjakob/earlyoom (I am the author)