Hacker News Reader

Comment by ot | Hacker News Reader

> (and each key has, on the order of, 20 bits of overhead, IIRC)

20 bits? Really? Less than an integer? Or did you mean bytes? (Not nitpicking here, I'm just curious)

I believe the limit using the current bitcask backend is (40 bytes + average key size) * (replication factor) / (number of cluster nodes * memory capacity of the smallest node). If that factor grows above 1, you can't store any more.

IIRC correctly a 64-node cluster with 24gb of ram per node will handle a few billion 32-byte keys, replicated to three nodes. For larger keyspaces, the current recommendation is to use innodb, which doesn't need to keep keys in memory.

Given that Riak seems to implement buckets "for free" by essentially making them key prefixes under-the-hood, does the bucket name size need to be considered as part of the key size? e.g., if my bucket name is 32 chars and my keys in that bucket are 32 chars, should I be using 64 bytes for average key size?

This is a question I've gotten conflicting answers to.

I think 20 bits is too small. I've been looking for a source without success; the closest I've found[0] is 32-75 bytes per key which is something like 11 million keys per GB of RAM.

[0] http://www.quora.com/Matt-Heitzenroder

20 bits can support a million records. As an average, per node or cluster, it seems reasonable to me.

Only a million records? That's waaaay too small.

Reply on news.ycombinator.com