On Linux, I use cgroups to restrict memory usage of certain applications to prevent them from crashing the whole system.