Hacker News Reader

Comment by etaioinshrdlu | Hacker News Reader

It looks like it drops the single-threaded limitation of redis to achieve much better performance.

Could this architecture be extended to scale across multiple machines? What would be the benefits and costs of this?

Unfortunately unlikely. Shared nothing architecture works with messaging. So threads send each other messages like "give me the contents of that key" or "update this key". Operations like SET/GET will require a single message hop. Operations like RENAME require 2 hops. Transactions and Lua scripts will require number of hops as number of operations in the transactions. When it's in the same process - the latency penalty in negligible. But between different machines - you will feel it. But who knows, AWS already have very cool UDP-like protocol with very low latency... if this will become the standard inside cloud environments maybe we can create a truly distributed memory store that spans multiple machines transparently.

The latency would be worse, but would the throughput still scale pretty much linearly? It might still be super useful.

Yes, VLL algorithm kinda solves the throughput problem. You will still have some issues with fault-taulerance - i.e. what you do if a machine does not answer you.

For intra-process framework it's not an issue (as long as we do not have deadlocks).

It kind of sounds like simple sharing but across all cores in a machine. Anyone know if this is an accurate take?

It's not a simple sharing. It's actually share-nothing architecture and doing multi-key operations atomically is pretty complicated thing. I used paper from 2014 to solve this problem

Maybe simple is the wrong word (perhaps I should have said familiar) but Redis sharding is similarly shared-nothing, is it not?

In Redis cluster the client needs to be connected to all shards and manage those connections. Multi-key operation on different slots are not supported etc... Maintaining a cluster is not a fun responsibility. DF saves you in most cases from the need to grow horizontly which should be much simpler to maintain and work with.

So is it architecturally different or does DF just have a much better UX?

it is architecturally different. DF has the orchestrator underneath. The general. The guy that handles all the incoming requests and makes sure they execute atomically. Redis cluster is an orchestra without the orchestrator. Everyone to himself. It's share-nothing that is exposed outside. To actually hide the complexity in a *transparent way* is a difficult problem. Researches write articles on how to solve this problem efficiently.

Reply on news.ycombinator.com