In Linux, you use cgroups to confine a process to a fixed memory limit that won't cause an OOM in adjacent processes. You can create a separate memory cgroup with a different allowance for each customer depending how much memory is allotted to them per the contract, for example.

You can also use ulimit with a hard memory limit, but cgroups are more fine grained and flexible.

Also do note that the point about the read-only mmap still holds – it can't cause an OOM, it is mmap'd pages pointing to dirty pages that have not been written out to the file system yet that can – on the condition their number is growing uncontrollably and/or fast.

P.S. But allowing a database to cause an OOM is never a good idea to start off with whether memory limits have been set or not – the data loss is unacceptable for a database, and databases have been optimised to operate within pre-configured space constraints that do not even require ulimit/cgroup in the first place. Perhaps fixing the database configuration is a better starting point rather than going down the rabbit hole with mmap(2).