Show HN: A reference use of Hashicorp's Raft implementation | Hacker News Reader

Show HN: A reference use of Hashicorp's Raft implementation

github.com

60 pointsotoolep14 comments

For those unfamiliar with how the Raft protocol works (as I was) this site[1] provides a great step by step visualization.

1. http://thesecretlivesofdata.com/raft/

https://raft.github.io/ does at better job at giving an overview.

Which itself says: The Secret Lives[1] of Data is a different visualization of Raft. It's more guided and less interactive, so it may be a gentler starting point. :)

1: The original link.

>Because any node will answer a GET request, and nodes may "fall behind" updates, stale reads are possible. Again, hraftd is a simple program, for the purpose of demonstrating a distributed key-value store. These shortcomings can be addressed by enhancements to the existing source.

Late to the conversation, but would you mind updating the README to elaborate upon this? I'm looking for the simplest implementation of Raft but need to be certain that I'm getting the "most recent" value for a key (in other words the last value that passed consensus).

I'd like to be able to either:

1) turn on that functionality without modifying the code

2) see a code snippet that achieves it by reading from some number of nodes or whatever

3) be directed to a link explaining if/why it's not possible

Otherwise, cool project, congrats!

Sure, I'll add some details, but hraft would require a chunk of work to support other modes of operation.

You might like to read a fuller description of this issue for another project based on Raft, called rqlite:

https://github.com/rqlite/rqlite/blob/master/doc/CONSISTENCY...

https://github.com/rqlite/rqlite/issues/5

I'll include a link to this from the README.

Nice work. What prompted you to write this? Are you using it anywhere?

Not OP but I am pretty sure OP has worked with/on InfluxDB which used it as part of the clustering component (which I am not sure if exists in the open source release anymore).

The hashicorp/raft library actually seems to have a bunch of production users, namely Consul ofcourse but also Flynn (used in service discovery and metadata store called discoverd) and I would assume still InfluxDB.

It's currently seeing a resurgence in development as the author of the Raft paper has begun contributing to the library also.

Disclaimer: I work on the Flynn project.

Do you think Hashicorp's Raft implementation is the best among the Go implementations? Just curious what is the perception among Raft users.

Best? I don't know. The etcd implementation is pretty well known too: https://github.com/coreos/etcd/tree/master/raft

I'm just more familiar with the one from Hashicorp, and it's used in a few production systems.

I think so yes. I haven't used the others to the same degree as the Hashicorp implementation but I have done some research and played around with toy KV stores and have formed the opinion that it's the easiest to get going with and provides the highest performance out of the box due to pipelining. These things may have changed but were my experience when learning Raft.

I also expect it to become much better now that development has picked up pace again.

After taking a quick glance, this implementation seems quite large, and cluttered with details, such as dealing with tcp connections. Wouldn't it be possible to have a much smaller reference implementation?

This is the smallest possible implementation in terms of TCP connections. Those connections are required for the nodes to communicate, and achieve consensus. The HTTP interface is technically optional, but you need some way to interact with the system as a user.

You could probably multiplex the use of the TCP connections to do everything, but that would require even more complexity.

I'd really prefer it if the TCP connections were not in the main code, so people could add their own network protocols.

In fact, I think the code should not even call any kernel function. Raft is an algorithm after all.

You can add whatever network connection you like (within reason), as the basis of an implementation. All the Hashicorp implementation requires is that it implements a certain interface. Check the docs: https://godoc.org/github.com/hashicorp/raft

Reply on news.ycombinator.com