Hacker News Reader

The endianness answer is "we thought about it, and it's a non-problem 99% of the time, and not worth further thought." You can't use a DB concurrently from multiple hosts, so it's unnecessary to support both endiannesses at once. If you need to copy a DB between different endian hosts, export it with mdb_dump and reimport it with mdb_load on the other machine. Don't be an ass.

There are no limitations on 32bit systems using MDB_VL32 in the 1.0 branch, you're spewing crap without any idea what you're talking about. Whoever's writing on stackoverflow doesn't know what they're talking about either.

> Moreover, there are use-case scenarios where the memory mapped file approach can suffer from problems of unnecessarily reading the page that will anyway be completely overwritten.

That's only true if you use a writable mmap, and the DB is larger than RAM. The default for LMDB is not to use a writable mmap, so this isn't an issue that affects most LMDB users. It is also already documented, so you choose it at your own risk.

> Again, it seems that for LMDB is again "solving" the problem by ignoring it. Which is OK if it fits your use case... But shouldn't Firefox work properly on 32-bit systems? Or did they completely decide that they don't want to target any 32-bit platform anymore?

Yes and no, respectively. Yes, Firefox should work properly on 32-bit systems; and no, we didn't completely decide that we don't want to target 32-bit platforms anymore.

Rather, Firefox plans to use LMDB where it fits one of its many use cases for persistent storage (and not use it where it doesn't).

The StackOverflow thread you referenced describes a blockchain program that expects to use "a few GB of lmdb diskspace." Whereas Firefox's use cases for LMDB are sized in the range of a few KiB to a few MiB. Firefox has no plans to start storing a blockchain.

Claimer: I'm the engineer integrating LMDB into Firefox.

The last issue is merely one of performance, not capability.

I’ve successfully used LMDB in 32-bit land, though it takes some effort. I had to scan available virtual memory for the largest contiguous chunk and use that.

Bigger issues are growing the database size and performance in low memory. No transactions can be running to increase the map size, which is usually hard to coordinate.

Also, in low memory situations, LMDB’s performance suffers tremendously. It can be 100x slower, and commits can take seconds. You won’t run into it unless you are really hammering it, and the developer usually won’t notice because they usually have lots of RAM.

re: 32bit land, You should look into MDB_VL32 in mdb.master as other posters have suggested.

If any single commit takes multiple seconds, you've probably written too much data in a single transaction. Above about 512MB it starts preemptively flushing intermediate pages out, to keep the in-RAM footprint limited. Some of those intermediate pages will need to be read back in if your transactions are hitting all over the DB. This is where the main slowdown comes.

Fwiw, we've benchmarked on Raspberry Pis with slow SDcards. You want to talk about low RAM situations and slow I/Os. The reality is still the same though - every other DB engine is many times slower under the same conditions.

Even when the DB is 5x or 50x larger than RAM... http://www.lmdb.tech/bench/hyperdex/