It seems like this would be easiest to do with a cgroup memory limit and supervisor process. In systemd or kubernetes, this should be pretty easy, IIRC.