hyc_symas

I walked through the design at Monerokon in 2019 here https://www.youtube.com/watch?v=4Hkd-n1W_e4

Ah yes, the ever popular "mongoDB's developers were incompetent therefore mmap is bad" paper.

Pure tripe. https://www.symas.com/post/are-you-sure-you-want-to-use-mmap...

How often does anyone care about using data on a different system than it was created on?

These days, any C struct you built on amd64 will work identically on arm64. There really aren't any other architectures that matter.

And yes, managing concurrent access to shared resources requires care and cooperation. That has always been true, and has nothing specific to do with mmap.

On BSD, read() was already implemented in the kernel by page-faulting in the desired pages of the file, to then be copied into the user-supplied buffer. So from the first time mmap was ever implemented, it was always the fastest input mechanism. (First deployed implementation was in SunOS btw, 4.2BSD specified and documented it but didn't implement it.) Anyway there's no magic to get data off a device into memory faster, io_uring just lets you hide the delay in some other thread's time.

Network data and most serialization formats are big endian because it's easiest to shift bits in and out of a shift register onto a serial comm channel in that order. If you used little endian, the shifter on output would have to operate in reverse direction relative to the shifter on input, which just causes stupid inconsistency headaches.

In-memory DBs were always a dead-end waste of time. There will always be bigger slower cheaper secondary storage, no matter how cheap main memory gets. And workloads always grow to exceed the available main memory (unless a system is dying). And secondary storage will always be paged, because it's too inefficient to address it at smaller granularity. That's just reality, and anyone who ignores reality isn't going to get very far.

Fun footnote: SQLite only got on board with mmap after I demonstrated how slow their code was without it. I.e., getting a 22x speedup by replacing SQLite's btree code with LMDB https://github.com/LMDB/sqlightning

Andy's critiques are only valid on dedicated database servers.

https://www.symas.com/post/are-you-sure-you-want-to-use-mmap...

LMDB uses mmap and Andy recommends LMDB, in the very article this thread is about.

A lot of potential treatments are too easily available and can't be patented. If a big pharma company can't make massive profit from it, they won't bother bringing it to market. Consider that a not-good reason.

Other treatments may eventually prove to have too many serious negative side effects. That's a good reason to abandon them.

You don't have to take my word for it. Plenty of other developers know. https://www.youtube.com/watch?v=CfiQ0h4bGWM

Not every.

LMDB is optimized for read-heavy workloads. I make no particular claims about write-heavy workloads.

Because it's so efficient, it can retain more useful data in-memory than other DBs for a given RAM size. For DBs much larger than RAM it will get more useful work done with the available RAM than other DBs. You can examine the benchmark reports linked above, they provide not just the data but also the analysis of why the results are as they are.

Plenty of proof. http://www.lmdb.tech/bench/

Identical hardware, same RAM size, same data volume.

It has a single writer transaction mutex, yes. But it's a process-shared mutex, so it will serialize write transactions across an arbitrary number of processes. And of course, read transactions are completely lockfree/waitfree across arbitrarily many processes.

As for working set size, that is always merely the height of the B+tree. Scans won't change that. It will always be far more efficient than any other DB under the same conditions.

Wrong, LMDB fully supports multiprocess concurrency as well as DBs multiple orders of magnitude larger than RAM. Wherever you got your info from is dead wrong.

Among embedded key/value stores, only LMDB and BerkeleyDB support multiprocess access. RocksDB, LevelDB, etc. are all single process.

> So much C library code is documented in ad-hoc ways - often through doxygen, which is a disaster. Eg here's the documentation for LMDB. LMDB is one of the most thoroughly documented C APIs I've seen, but I find this almost totally unusable. I often find myself reading the source instead. There's not even any links to the source from here:

> http://www.lmdb.tech/doc/group__mdb.html

How is doxygen a disaster?

Why do we need links to the source code? Doxygen is already embedded in the source, you should already be reading the source code on your local machine. It makes no sense to go searching across the web for information that's already stored on your local machine. Especially since you have no idea if the version you find on the web matches the version you're using locally.

That's such an obvious error in their benchmark code. In my benchmark code I make sure to touch the data so at least the 1st page is actually paged in from disk.

https://github.com/LMDB/dbbench/blob/1281588b7fdf119bcba65ce...

Thanks for that. It doesn't look like there is an open bug report for this yet.

I understand your commit description but I'm still a bit puzzled by it. cursor_to_update shouldn't have a deeper stack than cursor, even if it was a cursor on a subdatabase. All the references to the subdatabase are in the subcursor, and that has no bearing on the main cursor's stack.

The original code with the bug was added for ITS#8406, commit 37081325f7356587c5e6ce4c1f36c3b303fa718c on 2016-04-18. Definitely a rare occurrence since it's uncommon to write from multiple cursors on the same DB. Fixed now in ITS#10396.

What's the bug ID, have you submitted the fix back?

LMDB is for read-heavy workloads. The opposite of RocksDB.

RocksDB can use thousands of file descriptors at once, on larger DBs. Makes it unsuitable for servers that may also need to manage thousands of client connections at once.

LMDB uses 2 file descriptors at most; just 1 if you don't use its lock management, or if you're serving static data from a readonly filesystem.

RocksDB requires extensive configuration to tune properly. LMDB doesn't require any tuning.

A system built on LMDB will work better for this use case than RocksDB. And OSM Express already uses it. https://wiki.openstreetmap.org/wiki/OSM_Express

> LMDB has MDB_PANIC, documented as "Update of meta page failed or environment had fatal error".

Yes. That doesn't mean there was anything bad in the program logic. It most likely means your storage device had a fatal I/O error. It means there's something physically wrong with your system. Not that there was any bug in any code.

Welp, back in college I wrote a virtual memory demand pager for IBM 370 mainframe. I also read the entire 4.3BSD kernel source, as well as the Minix kernel source, and hacked on both of them. Also ported NFS server and client to the mainframe. After college, wrote VMEbus device drivers for Sun4 SPARC/Solaris, and a couple drivers for Linux and Windows as well. I'm pretty familiar with what any given kernel does.

> That link claims the test is 16 cores but my hunch given that the benchmark is from 2013 is that it’s 8+HT.

We tell no lies. Those tests were run on an HP DL585 G5 server with 128GB of RAM and 4 quad core AMD Opteron 8354 CPUs. If cross-TLB shootdowns were actually as big a problem as they claim, there should be massive latency spikes due to cross-socket communication, far worse than for inter-core single socket.

You're definitely misreading the graphs, the Y axis is clearly labeled "msec". You're going to have to pay better attention if you want to discuss this further. Otherwise we're just wasting our time.

> Keep in mind that the original paper looked at raw mmap vs raw io api issues regardless of the surrounding DB

Yes, but that's exactly why the paper is utterly worthless and never should have made it past peer review. Their initial claim is this:

> Unfortunately, mmap has a hidden dark side with many sordid problems that make it undesirable for file I/O in a DBMS. As we describe in this paper, these problems involve both data safety and system performance concerns. We contend that the engineering steps required to overcome them negate the purported simplicity of working with mmap. For these reasons, we believe that mmap adds too much complexity with no commensurate performance benefit and strongly urge DBMS developers to avoid using mmap as a replacement for a traditional buffer pool.

To prove their claims they would have to have demonstrated that a DBMS using mmap was more complex and less reliable than one using a traditional buffer pool. They also had to demonstrate that a DBMS using mmap had no performance benefit vs a DBMS using a traditional buffer pool. They didn't demonstrate any of these things, and LMDB is concrete proof that all of these claims are wrong: at only 7KLOCs it is simpler than every other traditional DBMS, its reliability is literally perfect, and its read performance is always orders of magnitude than all traditional designs.

The reviewers should have known just from reading the abstract that this paper was a dud:

> Such problems make it difficult, if not impossible, to use mmap correctly and efficiently in a modern DBMS.

Their claim required them to prove a negative, which obviously defies the rules of logic. Everything after that was just noise.

If there's any effect (like frame stuttering) then it will impact the max and 99th percentile latencies. Given that LMDB's latencies are always the tightest of any DB, it still boils down to "even if the effect is real, it's unmeasurable and therefore negligible." http://www.lmdb.tech/bench/memcache/

And of course, none of this is working in a realtime environment, so a couple nanoseconds here or there just doesn't matter.

Yes, you can use raw block devices in Linux. LMDB works on raw block devices as well. You get write-through cache behavior; writes are immediate but everything still goes into the page cache.

I've not yet bothered to check how things work with raw devices in Windows. I believe only Admins can even operate on them so it would be kind of limited usefulness.

Replacing the NIC sounds expensive, on a college student budget. You could've just reprogrammed the MAC EEPROM, no?

It took many years before mobile carriers in the US supported IPv6. I don't think any carriers here in Ireland do.

My gigabit fiber provider only supports IPv4. Granted, they're just a VNO, using PPPoE on top of Eircom's network. Eir supports IPv6 if you subscribe to them directly, but their customer service is nonexistent and you'll be waiting over a year just for them to hook you up. For now I stay with the VNO because they have excellent customer service, except for not having an answer for me every time I ask them when they'll support v6.

What you call optimal performance with InnoDB, I call pathetically slow. http://www.lmdb.tech/bench/memcache/

And if you'd read the reliability references linked in the article, you'd see that MariaDB/InnoDB's reliability is the worst of many tested systems. LMDB's is flawless.

https://lists.openldap.org/hyperkitty/list/openldap-devel@op...

The one fault they found for LMDB, caused by fdatasync not updating the DB filesize, was actually a kernel/filesystem bug (fdatasync is documented to update the filesize) that had already been fixed two years before these guys did their research. Not an LMDB flaw.

If this was a significant factor, then it should have a greater impact in high concurrency workloads, getting progressively worse across multi-socket machines.

Benchmarks all show that LMDB performance scales perfectly linearly across arbitrarily many CPUs. If this effect were an actual issue then the performance scaling should be significantly sub-linear. It's not. See e.g. http://www.lmdb.tech/bench/inmem/scaling.html or even the redb benchmark https://mastodon.social/@hyc/111887593549586249

The notion that "maybe their performance floor was higher" is laughable, since no other DBMS delivers anywhere near LMDB's read performance.