Nubmq: A high performant key-value store engine built from first principles
not able to post the link in url for some reason, here it is: https://github.com/nubskr/nubmq
It's not clear from the readme physically where the data is stored, nor where in the storage process the "congestion" is coming from.
I'm surprised there's no range scan. Range scans enable a whole swathe of functionality that make kv stores punch above their weight. I suppose that's more rocksdb/dynamo/bigtable/cassandra than redis/memcached, though.
Yeah, something weird happened with the image rendering; I'll fix that.
> *"It's not clear from the readme physically where the data is stored, nor where in the storage process the 'congestion' is coming from."*
The data is *fully in-memory*, distributed across dynamically growing shards (think of an adaptive hashtable that resizes itself). There’s no external storage layer like RocksDB or disk persistence—this is meant to be *pure cache-speed KV storage.*
Congestion happens when a shard starts getting too many keys relative to the rest of the system. The engine constantly tracks *contention per shard*, and when it crosses a threshold, we trigger an upgrade (new shards added, old ones redistributed). Migration is *zero-downtime*, but at very high write rates, there’s a brief moment where some writes are directed to the old store while the new one warms up.
> *"I'm surprised there's no range scan."*
Yeah, that’s an intentional design choice—this is meant to be a *high-speed cache*, closer to Redis than a full database like RocksDB or BigTable. Range queries would need an ordered structure (e.g., skip lists or B-trees), which add overhead. But I’m definitely considering implementing *prefix scans* (e.g., `SCAN user:*` style queries) since that’d be useful for a lot of real-world use cases.
Redis:
Write Latency ~1.1ms
Read Latency ~700µs
Max Throughput ~85,000 ops/sec
Write Latency 900µs
Read Latency 500µs
Max Throughput 115,809 ops/sec
a few reasons for being fast that come to my mind now are:
1. reads are direct lookups, so distributing that across goroutines will result in them being faster 2. it is set requests where it gets complicated, if we're simply updating some key's value, it's essentially negligible, but if we're creating a new key value pair, that can increase per shard load under scale, which would trigger an store resizing, to avoid just stopping everything when that happens, the engine recognises when the per shard load starts to gets too high, in the backgroud creates a bigger store, and then essentially switches writes from the old engine to the new(bigger) one, while the old one keeps processing reads, and the older engine migrates it's keys to the newer one in backgroud, once it's done, we just dereference the older engine to be collected by GC :) this essentially makes sure that incoming requests keep getting served (oh shit, I spilled the secret sauce)
atleast on my machine,with the default setting, under that concurrent load, Redis starts slowing down due to single-threaded execution and Lua overhead.
For benchmarking, I used the same test script (included in the repo) for both Redis and nubmq—100 concurrent clients hammering the server with requests. The core execution models remain unchanged; only the endpoints differ. So if you're asking whether the comparison reflects actual engine throughput rather than just connection handling—yes, it does.
The real ‘secret sauce’ is how nubmq prevents bottlenecks under load. When per-shard contention spikes, it spins up a larger store in the background, shifts new writes to it, and lazily migrates keys over. No pauses, no blocking. Meanwhile, Redis has no way to redistribute the load—it just starts choking when contention builds.
Also, Redis carries Lua scripting overhead in some cases. nubmq skips all that—pure Golang, no dependencies, no interpreter overhead
No threads waiting on locks, no hard pauses for resizes, no conventional polling mechanisms. Instead, nubmq's entire architecture revolves around one relentless obsession: never let a client wait. Everything else—adaptive sharding, real-time migrations, soft expiry handling—is just engineering fallout from aggressively challenging every assumption Redis and Memcached made decades ago.
Would love to see which of these breaks with convention you're most curious (or skeptical) about!
In terms of therapeutic approaches I can't blame them but you are right that a lot of the current technology is too old and doesn't even make good use of all the good innovation that has been happening for now decades.
Case in point: so many programs are still single-threaded.
feel free to open any issues :)
Are you thinking about a specific use case?