Hacker News Reader

Comment by atombender | Hacker News Reader

It's fascinating how Raft has democratized distributed computing -- writing a consistent, distributed state machine now can be done in a few lines of code, assuming you have a Raft implementation lying around, like Hashicorp's excellent Go library.

What this project (and similar projects such as Rqlite) doesn't address is the hard problem -- sharding. Raft makes it quite trivial to write a master/slave system where every node is guaranteed to be identical, but it won't scale writes and won't distribute storage across your cluster.

You can build sharding on top of Raft, but in an SQL setting this would require distributed transactions, something that's difficult enough that relatively few projects have tackled it so far, the notable ones being CockroachDB and TiDB. (Google Spanner, being proprietary, is not relevant in this context.)

View on HN

An excellent interactive explanation of Raft can be found here: http://thesecretlivesofdata.com/raft/. It would be interesting as hell to build a small implementation in the language of choice (elixir?, pony?).

That was good to see. Thank you.

rqlite author here. I am considering adding distributed transaction support to rqlite (it already has a form of transaction support), if I can come up with a solid implementation, while keeping rqlite simple to deploy and operate (which is a primary goal of rqlite):

https://groups.google.com/forum/#!topic/raft-dev/BaS5Z2NkDxA

https://github.com/rqlite/rqlite/issues/266

dqlite is very interesting. If its patches to SQLite are accepted upstream, dqlite could become the storage engine for rqlite. But right now rqlite is deliberately built on vanilla-SQLite, so it can offer the same correctness guarantees as SQLite.

  > It's fascinating how Raft has democratized distributed computing

First I thought you probably ment commoditized but then I realized how democratized actually applies perfectly well to consensus algorithms because by definition they apply decisions via voting by all participants :)

But anyways, I agree very much with you that Raft has transformed the industry and we can't be thankful enough for it. But like you said, it's just one of the core building blocks of a distributed system that also shards data. Sharding is more use-case specific and so it can be valid for different databases to have different sharding logics.

It's the second meaning of democratize, meaning "to make something accessible to everyone", that I intended, but I suppose Raft also has commoditized the technology, since there are now off-the-shelf libraries you can pick and choose from somewhat interchangeably.

FWIW, the CockroachDB people developed a modified version they call MultiRaft [1] that deals with the scaling challenges they have around shards.

[1] https://www.cockroachlabs.com/blog/scaling-raft/

If you require SQL, sharding and concurrent writes, yes do use CockroachDB or other equivalent solution.

If you don't have those requirements, dqlite or rqlite might get the job done with less moving parts and operational overhead.

It's pretty much the same argument of using SQLite vs (say) PostgreSQL, but translated into distributed systems. See:

https://www.sqlite.org/whentouse.html

Check out http://www.actordb.com/ for a massively sharded sqlite system.

ActorDB is very interesting. It uses two-phase commit, similar to CockroachDB, TiDB and Spanner, but as I understand it, relies on locking to update the participating shards. I'm not sure how well ActorDB performs in a large cluster with lots of distributed transactions, but I've never tried it.

IIUC these are micro shards - e.g. one sqlite instance per user. Depending on the key, distributed transactions may be rare.

The size of the shards depends on the data. ActorDB wants you to design your application around explicit containment boundaries (the actors). For example, a social networking site might collect all of a single user profile's data in one actor. Or there might be one actor for each grouping of a user's stuff (one actor for all of user X's photos, one actor for all of their tweets, etc.). So the size of those actors might not be "micro". You could decide to shard it further, of course, e.g. shard a user's photos by date. Since ActorDB supports cross-actor queries (though not joins), you can still query multiple shards at the same time.

But ActorDB does seem to promote inter-actor transactions, and a pattern I've seen encouraged is where you use shared actors (especially with the key-value store functionality) to keep some kind of central lookup table that your app then can use to find actors. For example, if you're modeling HN with ActorDB, you'd have a global list of stories, but each story/comment tree could be a separate actor, and each user would be a separate actor. Posting a story only needs to access the story actor, but to post a comment transactionally, you have to do a transaction that inserts the comment and updates the user's comment history together.

> The size of the shards depends on the data.

You're right. What I meant was that a 'shard' in actordb isn't meant to be 'all data on one machine/node' - which is how many other databases view sharding, but is expected to be much more fine-grained. Thanks for the other explanations.

Another interesting implementation aspect is its use of LMDB for storing the SQLite pages.

> What this project (and similar projects such as rqlite) doesn't address is the hard problem -- sharding.

Yes, this is technically correct. However to meet its goals rqlite (and dqlites) does not require sharding. The point of rqlite is to provide fault-tolerance and high-availability. Again, you are technically correct that rqlite doesn't support sharding, but sharding simply isn't required for rqlite to do what it wants to do. Sharding is a solution to a different type of problem.

is raft a fancy name for distinguished leader multipaxos?

Raft does the things that Paxos does, using a mechanism that is much simpler for humans to understand and therefore implement correctly.

if it does the things paxos does, it's provably isomorphic to paxos. what I saw in the demo was a flavor of paxos that's been around since the early 90's, I'm pretty sure that's where the credit is due.

You can't think of any case, for anything, in which you can do something in two (or more!) different ways, but one of them is simpler to explain, understand, and implement?

> writing a consistent, distributed state machine now can be done in a few lines of code

Well, if you run it on a very fast and reliable local network without much load, not multi dc deployments over public internet, than sure, it can sort of work. Not without problems though once a fault occurs.

I don't agree. The whole point of putting a solid Raft implementation at the center of your system is deal with the faults. I myself built a simple distributed state machine which deals with faults perfectly well. Of course I built on top of a good Raft implementation, which certainly is not a few lines of code.

https://github.com/otoolep/hraftd

No, Raft doesn't deal with the faults, it deals with consensus and can help tolerate certain faults. You then deal with the faults by other means. And even on a fast and reliable local network where Raft can work well, once you have a significant amount of data replacing failing nodes, healing broken records, resyncing, rebalancing all without affecting operations is not trivial and unlikely to be done properly.

> Raft doesn't deal with the faults

I'm not sure what this means. The Raft paper explicitly states that Raft is a fault-tolerant system -- and by definition this means it deals with faults. To quote the paper:

"Replicated state machines are used to solve a variety of fault tolerance problems in distributed systems."

Raft is a type of replicated state machine. To say that "Raft doesn't deal with faults" is not correct. Perhaps you mean that there is other work to be done to take the fault tolerance offered by Raft and build a functioning application, and a system that will stay up in the real world -- and deal with an even wider range of faults. That I definitely agree with.

Well, if you’re being smart - do please tell us how Raft solves the problem of online faulty node replacement: how how does it re-sync the current, up-to-date state to the new node, how does it make sure that when one of the nodes fails - that all of the clients do not overwhelm the rest of the system.

As far as Google went with their paper; they were incredibly correct by saying that actually implementing a distributed system is hard because of the failure modes; rather than the success modes of the running system.

>Well, if you’re being smart - do please tell us how Raft solves the problem of online faulty node replacement:

I never said that it did. That is what I mean by my statement that more work is required to build a real system in the real world. Much of the code of rqlite is about cluster management, built on a Raft substrate.

>how how does it re-sync the current, up-to-date state to the new node

I suggest you read the Raft paper, and study the Hashicorp Go implementation (https://github.com/hashicorp/raft) to see how this is done. It's all there.

>As far as Google went with their paper; they were incredibly correct by saying that actually implementing a distributed system is hard because of the failure modes; rather than the success modes of the running system.

I couldn't agree more.

If you read the raft paper, or just watch presentations of it, you'll see that all the issues you mentioned are covered:

- faulty node replacement: you can take any node off (or any node can crash) at any time. As long as there are enough nodes left to reach a quorum, your system will be available. If there are not enough nodes left, your system will be unavailable (but keep consistency)

- re-syncs, snapshots and the rest are all covered in the raft paper

- clients wanting to perform writes always talk to the node that is currently the leader, that node fails, clients will look for next leader (which will be eventually elected as long as there's a quorum of surviving nodes)

- overwhelming a system is a different concern, raft writes are serialized so the goal is usually not throughput (for that you might look at AP/AC storage solutions in the CAP spectrum, raft is CP). Designs that need high write throughput might still use raft as internal building block for coordination.

> how does it re-sync the current, up-to-date state to the new node

it is a core feature of the protocol.

> how does it make sure that when one of the nodes fails - that all of the clients do not overwhelm the rest of the system.

for write access (proposals), in a typical implementation (defined below [1]), one failed nodes actually slightly speed up the system with the cost of reduced reliability. Consider a typical setup with 3 nodes, the leader normally replicate state to 2 followers, the replication cost get cut in half once your cluster loses one member.

given that you can implement linearizable read by going through the write procedure, one can argue the above "speed up" can obviously be achieved on reads as well.

[1] independent replication to followers, for N followers, the same entry will be serialised and sent N times by the leader.

again - just talking about typical implementation and the described "speed up" comes with degraded reliability.

Rereading this, I see what you mean in the sense that if dealing == "fixing the fault", but tolerating == "keep going in the face of faults". So yes, you are right -- Raft doesn't fix the faults. One must still fix the network, replace the node etc. When I wrote "deal" I meant "tolerate the fault", but I think "deal" meant "fix the fault" to you.

Reply on news.ycombinator.com