Hacker News Reader

> There is a trade off obviously where the kernel can evict I/O buffers more efficiently. But to me all that says is that the interface for managing I/O buffers in user space is really immature and needs some better performing API so that we have something in between mmap and buffer management.

This was essentially the argument against adding the raw device support to Linux, which delayed the Oracle database Linux port for years, with Linus himself being adamant that operating system's own VMM and page cache management was superior to that of the database (except it was not), which was proven to be a fallacy as the transaction and error handling database requires completely different guarantees, and the VMM pursuing different design objectives that most of the time are incompatible with the database expectations.

The raw devices have been used precisely for that reason: «let's bypass the VMM and the page cache for this device, and we will take it from there». That is what the Oracle database did on Solaris – it wanted an entire raw device that it would claim for the table space. Granted, Oracle has had its own VMM and page cache (so to speak) implementation within the DB, an OS mini-kernel if you like.

The argument was eventually settled with adding the O_DIRECT flag to open(2) accomplishing the same outcome, and the raw devices have never made into Linux.

O_DiRECT is insufficient because you have no way to coordinate with the kernel to drop io buffers when there’s memory pressure

This is commonly avoided by not putting memory pressure on your O_DIRECT application, which is probably an RDBMS on a dedicated single-purpose host.

Not saying more work in the space wouldn't be welcome and interesting, just that it's generally the assumption whenever I've seen O_DIRECT (since the Solaris days and through today) that what's using it is the only thing of substance on the box.

I think that’s a flawed assumption. Databases show up in many spots besides dedicated RDBMS (eg SQLite and RocksDB power all sorts of things and likely coexist with compute nodes because even compute nodes can need some amount of local storage).

I did not imply that O_DIRECT was a full equivalent of raw devices, it was rather a compromise as Linus would not concede with the idea of raw devices making it into Linux.

Besides, O_DIRECT can be used at the file level (and the file system level as well via chattr – if the file system supports it) whereas the raw devices are, well, device level only.

I’m pretty sure you can open disks raw in Linux. But then the disk is owned by the RDBMS. Ceph can do this in its BlueStore engine.

But yeah, I understand you weren’t talking about buffers, I was just highlighting that O_DIRECT and raw devices are totally separate problems from having a user space cache that can be evicted by the kernel memory and reconstructed by the application lazily.

Yes, you can use raw block devices in Linux. LMDB works on raw block devices as well. You get write-through cache behavior; writes are immediate but everything still goes into the page cache.

I've not yet bothered to check how things work with raw devices in Windows. I believe only Admins can even operate on them so it would be kind of limited usefulness.

I don’t believe the page cache is accessed if you open the raw block device via o_direct.

When I was writing a game engine and doing a multithreaded resource loading system using job queues, I was concerned about TLB shootdowns, but was unable to really experience the issues with them.

I tried both regular IO and mmap and was unable to make regular IO as fast as mmap.

This is on a machine with 32 real cores running 64 job threads, saturating all of them with jobs. The jobs are a mixture of small (metadata) and big (textures and models) files. In theory, this should be a pretty bad situation for the TLB shootdown, and I was concerned about them.

But in reality it seemed to be fine, and a fair amount faster than with regular IO due to being able to have one less extra copy of the data.

You’d be surprised how difficult it can be to create such a scenario and detect when you do, especially using a synthetic benchmark.

In terms of regular io, did you remember to do O_DIRECT? Also yes, I/O paths are slower than mmap for pure throughput in most cases. Databases are a special snowflake though. In some ways you’d want the I/O layer in the kernel like with a filesystem which is why databases typically go the other way and try to bypass the kernel to keep their code in user space

> buffers in user space is really immature and needs some better performing API so that we have something in between mmap and buffer management.

If you don't know about it already, you should look at Viktor Leis' et al's work on this -- kernel module + supporting buffer mgmt code to do the kind of thing you're talking about. The code is not production quality, but the idea is nifty:

https://github.com/viktorleis/vmcache

https://github.com/tuhhosg/exmap

https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/_my_direct_up...

I've personally experimented a bit with userfaultfd and this kind of thing, but it's got issues.

Yeah I’m aware work is kind of being done in the space. Not pretending it’s my idea. Just a more formal official API would be supremely interesting and not just for databases - there’s lots of user space caches that should be migrated to that if they’re recomputable.

Relatedly there’s an effort going on in the filesystem space for a long time to add transaction support which would also be useful. Some day.

Transactions in the filesystem API is misguided and useless. All you need is to expose command tagging to user level, probably with some fcntl:

https://www.spinics.net/lists/linux-fsdevel/msg70047.html

Trying to implement actual transactions can't work because it requires the FS to maintain unbounded state. https://lore.kernel.org/all/5283E6DC.8000801@symas.com/

By “transcations” I mean the ability to do a bunch of “invisible” writes to a file that are only published through an fsync (and also not made visible just because a sibling application invoked sync). Full transactions aren’t necessary and I agree would be overkill. The work is primarily driven by XFS developers if I recall correctly (too lazy to lookup the exact links). But there are clever workarounds in user space although they’re not particularly confidence inducing.

You can say the performance analysis is wrong, when you find any benchmarks showing that LMDB performance suffers because of it.

Just like those paper's authors can claim using mmap increases complexity when they can find any DBMS with LMDB's features, in fewer lines of code.

The original paper literally shows the effect of cross tlb shootdowns. I don’t recall LMDB specifically so maybe their performance floor was just naturally much higher anyway. I’ve also seen the impact of cross tlb flushes and synthetic benchmarks are the wrong mechanism because of all the secondary effects. It really is a kind of global mutex for memory operations on the machine where a memory allocation in an entirely different process can take longer as a result. The real world results showing it were inconsistent and impossible to reproduce in synthetic benchmarks and took a lot of smart people a long time to narrow down the cause when I saw this (not a DB application but it was cross tlb shootdowns slowing down the machine).

If this was a significant factor, then it should have a greater impact in high concurrency workloads, getting progressively worse across multi-socket machines.

Benchmarks all show that LMDB performance scales perfectly linearly across arbitrarily many CPUs. If this effect were an actual issue then the performance scaling should be significantly sub-linear. It's not. See e.g. http://www.lmdb.tech/bench/inmem/scaling.html or even the redb benchmark https://mastodon.social/@hyc/111887593549586249

The notion that "maybe their performance floor was higher" is laughable, since no other DBMS delivers anywhere near LMDB's read performance.

> The notion that "maybe their performance floor was higher" is laughable, since no other DBMS delivers anywhere near LMDB's read performance.

What I said was that lmdb’s read performance may be so good, that at its worst when you’re seeing cross tlb shootdowns, it’s still better than the other DBs. You seem to contradict me and then restate the exact same thing.

> Benchmarks all show that LMDB performance scales perfectly linearly across arbitrarily many CPUs

As I said, synthetic benchmarks may not reproduce the cross tlb shootdown scenario frequently enough for you to notice it. It’s like frame stuttering - you can’t argue that it’s not happening by saying the average frame rate was 70fps. The data you’re using to support your argument isn’t a relevant measure. You’d have to actually measure how frequently cross tlb shootdowns are happening (via dtrace probably) and ideally compare it against a robust lmdb read path written against iouring to get an accurate measure of the issue.

If there's any effect (like frame stuttering) then it will impact the max and 99th percentile latencies. Given that LMDB's latencies are always the tightest of any DB, it still boils down to "even if the effect is real, it's unmeasurable and therefore negligible." http://www.lmdb.tech/bench/memcache/

And of course, none of this is working in a realtime environment, so a couple nanoseconds here or there just doesn't matter.

Except even your link shows a substantial jump in the log graph from ~500ms->1s going p99 to pmax. And cross tlb shoot down times can vary wildly but nanoseconds is the wrong order of magnitude. They also get significantly worse with the number of cores. That link claims the test is 16 cores but my hunch given that the benchmark is from 2013 is that it’s 8+HT.

I really don’t understand this defensiveness and doubling down on something that’s pretty well studied and documented. Keep in mind that the original paper looked at raw mmap vs raw io api issues regardless of the surrounding DB: https://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf

It’s a really well written and accessible paper and this is not the hill to die on if you’re disagreeing. I agree mmap for the read path only is a weakness in their paper and that idea I agree has some merit. But cross tlb shootdowns are a real performance downside of this approach.

Are we looking at the same benchmark? All of LMDB's MAX read perf is under 1ms.

Him doubling down makes sense, given that LMDB continues to offer the best read performance in any benchmark, regardless of who performed it.

If tlb shoot down times are such a problem, why aren't we seeing it being represented in any DB benchmark?

> That link claims the test is 16 cores but my hunch given that the benchmark is from 2013 is that it’s 8+HT.

We tell no lies. Those tests were run on an HP DL585 G5 server with 128GB of RAM and 4 quad core AMD Opteron 8354 CPUs. If cross-TLB shootdowns were actually as big a problem as they claim, there should be massive latency spikes due to cross-socket communication, far worse than for inter-core single socket.

You're definitely misreading the graphs, the Y axis is clearly labeled "msec". You're going to have to pay better attention if you want to discuss this further. Otherwise we're just wasting our time.

> Keep in mind that the original paper looked at raw mmap vs raw io api issues regardless of the surrounding DB

Yes, but that's exactly why the paper is utterly worthless and never should have made it past peer review. Their initial claim is this:

> Unfortunately, mmap has a hidden dark side with many sordid problems that make it undesirable for file I/O in a DBMS. As we describe in this paper, these problems involve both data safety and system performance concerns. We contend that the engineering steps required to overcome them negate the purported simplicity of working with mmap. For these reasons, we believe that mmap adds too much complexity with no commensurate performance benefit and strongly urge DBMS developers to avoid using mmap as a replacement for a traditional buffer pool.

To prove their claims they would have to have demonstrated that a DBMS using mmap was more complex and less reliable than one using a traditional buffer pool. They also had to demonstrate that a DBMS using mmap had no performance benefit vs a DBMS using a traditional buffer pool. They didn't demonstrate any of these things, and LMDB is concrete proof that all of these claims are wrong: at only 7KLOCs it is simpler than every other traditional DBMS, its reliability is literally perfect, and its read performance is always orders of magnitude than all traditional designs.

The reviewers should have known just from reading the abstract that this paper was a dud:

> Such problems make it difficult, if not impossible, to use mmap correctly and efficiently in a modern DBMS.

Their claim required them to prove a negative, which obviously defies the rules of logic. Everything after that was just noise.