Hint: disable or minimize overcommit and you're good to go.
malloc failing seems like a much simpler indication of memory pressure than the alternatives like PSI.
this would work if there was a cgroup virtual address space controller, but I think those have been proposed but never merged.