Hacker News Reader

Comment by qwertyuiop924 | Hacker News Reader

Can somebody get us an on disk, small, fast, in-process, low-footprint NoSQL DB? Thus far, I've been pressing SQLite into use for a lot of monotyped data that would have been better served by a NoSQL DB, but I couldn't find one suitable.

Apparently, in-memory DBs in now. They're fast and all, that's nice, but I'm still running on a machine with 8 Gigs of ram, and when the dataset starts to exceed that, I want an option other than "grab some more DDR3." I remember the fate of TinyMUD: on-disk operation must at least be an option for any DB I'd use for datasets of a non-fixed size.

View on HN

There are several key-value stores you can choose from. RocksDB, LevelDB, lmdb, Berkeley DB, Tokyo Cabinet, sophia, and others. What about those? You can always build abstractions on top.

BoltDB if you'd like to embed it into a Go program.

BoltDB can be corrupted on crash (kill -9 or hard reset) at least this was the case when I gave it a try. Any database can but in that case that was easy to achieve.

did you file a bug for this? I thought it was pretty resilient in such cases

I've found several submitted issues that described this problem. All this issues are closed now so maybe I was wrong about Bolt.

Thanks for the suggestions. Some of those might work. The only problem with k/v stores is the serialization/deserialization cost, as most of them can only store strings.

lmdb has some special cases where you can store capnproto/flat-buffers objects and read without copying/deserializing!

Really? that could be useful...

LMDB can hand you back a pointer to your object in the memory map, and we can easily use that pointer with Cap'n Proto without any calls to malloc(). source: https://news.ycombinator.com/item?id=12394385

> The only problem with k/v stores is the serialization/deserialization cost, as most of them can only store strings.

I store Cap'n Proto objects directly into LMDB, and can read them without any memory allocation or deserialization cost. Freaking rocks.

You can try LMBD - https://symas.com/products/lightning-memory-mapped-database/ In my opinion there is exactly two production ready embedded databases - SQLite and LMDB. At least this two databases have decent crash recovery mechanisms (I've tested both). I haven't tested all of them, maybe ForestDB and BerkleyDB are good as well.

The only question is this: Is the dataset required to fit into RAM? Because if it is, than I haven't fixed the problem, now have I...

> The only question is this: Is the dataset required to fit into RAM?

I run an LMDB dataset many times the size of RAM, works great. Most data isn't recent, so the virtual memory cache works extremely well.

Okay. Good.

http://engine.so/ -- quite a nice list of small, fast, in-process databases.

> Can somebody get us an on disk, small, fast, in-process, low-footprint NoSQL DB?

That's LMDB, as others have noted. I have used it in production for years, never once lost data. Insanely fast, low footprint, crash-proof.

(P.s. I have seen SQLite get corrupted regularly on mobile devices, which is why I no longer use it there.)

Bizarre. SQLite is known for being hard to corrupt. How did it happen?

No idea, but with about 50,000 regular users hitting the local SQLite DB on their mobile devices hard, I'd get a corrupted DB report a few times a week.

It's possible my SQLite configuration wasn't as crash-proof as the default install. I moved onto another project before I could investigate further.

Seems possible you were just seeing the rate at which people's storage hardware gets corrupted rather than an issue with SQLite.

I use LevelDB, it is a key-value database, from Google.

I am using it with Nodejs, on t2.micro, handling 10+ million requests / day

Never had an issue. in production for one year already. Really good :-)

Level could work, but it depends on how expensive serialization is.

There's also a lot of higher level abstractions over leveldb at this point.

Mnesia built into Elixir/Erlang potentially?

Have you heard about TieDot?

make a gigantic swap file?

Reply on news.ycombinator.com