What do people typically store in KV stores that do not have a database built on top of them? Terabytes of what? Accessed by what kind of application logic?
One use case is maintaining node-level state for stream processing systems. This gives you a scaleable way to do stateful computations (such as aggregations) without the complexity and performance cost of hitting a remote data store. Such support is built into Samza using RocksDB. [1]
I currently use leveldb in my project as a way to handle an intermediate data processing step that chews on much-larger-than-RAM datasets. And I didn't feel like this processing step warranted the overhead of a full-blown database sitting on another box, or even the same box. Works quite nicely thus far.
Would you have used SQlite if it had an LSM backend so the write performance would be as good as leveldb?
I considered it, but for what I was doing, a K/V store that also has fast key prefix lookup was perfect. Plus, yeah, what I'm doing is currently super write-heavy.
It's often a storage layer for some more complex thing - e.g. riak had a leveldb "backend". Facebook uses rocksdb similarly, plus there's the myrocks storage engine for using rocksdb with mysql on top.