And what about the limitations on the 32-bit system? Isn't there also needed to use memory space of RAM for the complete size of the database, that's how that database works if I understood? Which makes LMDB effectively unsuitable for 32-bit systems:
https://stackoverflow.com/questions/52862176/lmdb-open-large...
"your user will need to either enable PAE on their system, or upgrade to 64-bit CPU. If neither of these is an option in your application, then you cannot use a memory mapped file larger than your available address space"
In short it still looks that LMDB is designed effectively only for one-endianness and only for 64-bit systems, which is still limiting for many use cases.
Again, it seems that for LMDB is again "solving" the problem by ignoring it. Which is OK if it fits your use case... But shouldn't Firefox work properly on 32-bit systems? Or did they completely decide that they don't want to target any 32-bit platform anymore?
Moreover, there are use-case scenarios where the memory mapped file approach can suffer from problems of unnecessarily reading the page that will anyway be completely overwritten. My conclusion is... if LMDB "works for you" fine, but do properly your research first. I still believe SQLite covers much more use cases and is safer starting point for many of them, including Firefox use cases, until I read that they really decided to reduce these.
There are no limitations on 32bit systems using MDB_VL32 in the 1.0 branch, you're spewing crap without any idea what you're talking about. Whoever's writing on stackoverflow doesn't know what they're talking about either.
> Moreover, there are use-case scenarios where the memory mapped file approach can suffer from problems of unnecessarily reading the page that will anyway be completely overwritten.
That's only true if you use a writable mmap, and the DB is larger than RAM. The default for LMDB is not to use a writable mmap, so this isn't an issue that affects most LMDB users. It is also already documented, so you choose it at your own risk.
I’ve successfully used LMDB in 32-bit land, though it takes some effort. I had to scan available virtual memory for the largest contiguous chunk and use that.
Bigger issues are growing the database size and performance in low memory. No transactions can be running to increase the map size, which is usually hard to coordinate.
Also, in low memory situations, LMDB’s performance suffers tremendously. It can be 100x slower, and commits can take seconds. You won’t run into it unless you are really hammering it, and the developer usually won’t notice because they usually have lots of RAM.
If any single commit takes multiple seconds, you've probably written too much data in a single transaction. Above about 512MB it starts preemptively flushing intermediate pages out, to keep the in-RAM footprint limited. Some of those intermediate pages will need to be read back in if your transactions are hitting all over the DB. This is where the main slowdown comes.
Even when the DB is 5x or 50x larger than RAM... http://www.lmdb.tech/bench/hyperdex/
Yes and no, respectively. Yes, Firefox should work properly on 32-bit systems; and no, we didn't completely decide that we don't want to target 32-bit platforms anymore.
Rather, Firefox plans to use LMDB where it fits one of its many use cases for persistent storage (and not use it where it doesn't).
The StackOverflow thread you referenced describes a blockchain program that expects to use "a few GB of lmdb diskspace." Whereas Firefox's use cases for LMDB are sized in the range of a few KiB to a few MiB. Firefox has no plans to start storing a blockchain.
Claimer: I'm the engineer integrating LMDB into Firefox.