If you are a DBMS kernel developer, the most urgent need is how to control memory more precisely to prevent out-of-memory (OOM) situations. Clearly, MMAP cannot achieve this, which is also why most databases still use their own memory pool + direct-IO.
Clearly you have no idea what you're talking about. With a read-only mmap you don't need to do anything, OOM won't happen*. All of the pages of the map are clean; whenever there's memory pressure the OS can just reclaim those pages from the page cache and use them to meet other demands. That page re-use is zero cost; since the pages are clean they don't require a flush or pageout before they can be reused. Using a read-only mmap means the DB will never be the cause of an OOM situation, and will never be the cause of swap usage.
*on Linux you must lower /proc/sys/vm/swappiness from its default of 60, otherwise the kernel won't reclaim FS cache pages.
You can also use ulimit with a hard memory limit, but cgroups are more fine grained and flexible.
Also do note that the point about the read-only mmap still holds – it can't cause an OOM, it is mmap'd pages pointing to dirty pages that have not been written out to the file system yet that can – on the condition their number is growing uncontrollably and/or fast.
P.S. But allowing a database to cause an OOM is never a good idea to start off with whether memory limits have been set or not – the data loss is unacceptable for a database, and databases have been optimised to operate within pre-configured space constraints that do not even require ulimit/cgroup in the first place. Perhaps fixing the database configuration is a better starting point rather than going down the rabbit hole with mmap(2).
We manage memory directly within the DBMS, for instance, by setting MySQL's innodb_buffer_pool_size to 1GB. This precise control method ensures that each cloud-based DBMS instance on our large EC2 instances remains within its memory limit without adversely affecting others.
When the internal memory pool limit of the DBMS is reached, we understand that it will spill dirty pages to disk and utilize the write-ahead log to guarantee ACID properties in the event of a crash. This process is predictable and manageable. In contrast, the effects on the DBMS when cgroup memory limits are reached are less clear, introducing uncertainty into system behavior under memory pressure. This unpredictability strongly supports our preference for managing memory within the DBMS itself.
Our experience over a decade with cloud DBMS products has shown that mmap doesn't suit our needs. We prioritize direct memory management methods within the DBMS to ensure system reliability and optimal performance.
And if you'd read the reliability references linked in the article, you'd see that MariaDB/InnoDB's reliability is the worst of many tested systems. LMDB's is flawless.
https://lists.openldap.org/hyperkitty/list/openldap-devel@op...
The one fault they found for LMDB, caused by fdatasync not updating the DB filesize, was actually a kernel/filesystem bug (fdatasync is documented to update the filesize) that had already been fixed two years before these guys did their research. Not an LMDB flaw.