Hacker News Reader

Comment by tdrhq | Hacker News Reader

Raft does do persistence and crash recovery, at least of the transaction logs.

What you need from your side (and there are libraries that already do this): a) A mechanism to snapshot all the data b) An easy in-memory mechanism to create indexes on fields--not strictly needed, but definitely makes things a lot more easier to work with.

Bespoke data structures are just simple classes, so if you're familiar with traversing simple objects in the language of your choice, you're all set. You might be over-estimating the benefits of a query engine (and I have worked at multiple places that used MySQL extensively, and used MySQL to build heavily scaled software in the past).

View on HN

> Raft does do persistence and crash recovery, at least of the transaction logs.

It simply does not. The paper that definitionally is Raft doesn't tell you how to interact with durable storage. The raft protocol handles crash recovery in so far as it allows one or more nodes to rebuild state after a crash, but Raft doesn't talk about serialization or WAL or any of the other things you inevitably have to do for reliability and performance. It gives you a way to go from some existing state to the state of the leader (even if that means downloading the full state from scratch), but it doesn't give you a way to go from a pile of bits on a disk to that existing state.

If you have a library that implements Raft and gives you those things, that's not Raft giving you things. And that library could just be SQLite.

> You might be over-estimating the benefits of a query engine

No, I'm not. It's great to describe the data I want and get, say, an array of strings back without having to crawl some Btrees by hand.

> The paper that definitionally is Raft doesn't tell you how to interact with durable storage.

That's being a bit pedantic. Yeah, I did mean that any respectable library implementing Raft would handle all of this correctly.

> without having to crawl some Btrees by hand.

This is not how I query an index. First, we don't even use Btrees, most of the times it's just hash-tables, and otherwise it's a simpler form of binary search trees. But in both cases, it's completely abstracted away in library I'm using. So if I'm trying to search for companies with a given name, in my code it looks like '(company-with-name "foobar")'. If I'm looking for users that belong to a specific company, it'll look like '(users-for-company company)'.

So I still think you're overestimating the benefits of a query engine.

Reply on news.ycombinator.com